<html><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">Basically, when a new OSS server was added, I followed this procedure (I'm not sure that all the steps are needed but this works with our installation): <div><br></div><div>quotainv</div><div><font class="Apple-style-span" color="#000000" size="3"><span class="Apple-style-span" style="background-color: transparent; font-size: 13px;">quotacheck</span></font></div><div><div apple-content-edited="true"><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: 'Lucida Grande'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0; "><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: 'Lucida Grande'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; "><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: 'Lucida Grande'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; "><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: 'Lucida Grande'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; "><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: 'Lucida Grande'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; "><div><span class="Apple-style-span" style="font-family: Helvetica; "><div><font class="Apple-style-span" color="#000000" face="'Lucida Grande'"><span class="Apple-style-span" style="background-color: transparent;">lfs setquota -u donvito --block-softlimit 0 --block-hardlimit 0 /lustre</span></font></div><div><div><font class="Apple-style-span" color="#000000" face="'Lucida Grande'"><span class="Apple-style-span" style="background-color: transparent;">lfs setquota -u donvito --block-softlimit 2000000 --block-hardlimit 1000000 /lustre</span></font></div><div><br></div><div>after this all the OSTs have the quota showed correctly. </div><div><br></div><div>I hope that this will be useful for you too.</div><div><br></div><div>Cheers,</div><div>Giacinto</div></div><div><br></div><div>-- </div><div>-- </div><div>~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~</div><div>Giacinto Donvito LIBI -- EGEE3 SA1 INFN - Bari ITALY</div><div>------------------------------------------------------------------</div><div><a href="mailto:giacinto.donvito@ba.infn.it">giacinto.donvito@ba.infn.it</a> | GTalk/GMail: <a href="mailto:donvito.giacinto@gmail.com">donvito.giacinto@gmail.com</a></div><div>tel. +39 080 5443244 Fax +39 0805442470 | Skype: giacinto_it</div><div>VOIP: +41225481596 | MSN: <a href="mailto:donvito.giacinto@hotmail.it">donvito.giacinto@hotmail.it</a></div><div>AIM/iChat: gdonvito1 | Yahoo: eric1_it </div><div>------------------------------------------------------------------</div><div>"Not everything that can be counted counts, and not everything that counts can be counted."</div><div>- Albert Einstein (1879-1955)</div><div><br class="webkit-block-placeholder"></div></span></div></span></div></span><br class="Apple-interchange-newline"></div></span><br class="Apple-interchange-newline"></div></span><br class="Apple-interchange-newline"></div></span><br class="Apple-interchange-newline"> </div><br><div><div>Il giorno 01/lug/09, alle ore 16:59, Robert LeBlanc ha scritto:</div><br class="Apple-interchange-newline"><blockquote type="cite">How did you solve this, we will be implementing quotas on our system soon and don't want to fall into the same trap.<br><br>Thanks,<br clear="all"><br>Robert LeBlanc<br>Life Sciences & Undergraduate Education Computer Support<br> Brigham Young University<br> <br><br><div class="gmail_quote">On Wed, Jul 1, 2009 at 5:53 AM, Giacinto Donvito <span dir="ltr"><<a href="mailto:giacinto.donvito@ba.infn.it">giacinto.donvito@ba.infn.it</a>></span> wrote:<br><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"> Thank you Zhiyong,<br> <br> with this hint I was able to find a way to solve the problem.<br> <br> Cheers,<br> Giacinto<br> <br> -- -- ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~<br> Giacinto Donvito LIBI -- EGEE3 SA1 INFN - Bari ITALY<br> ------------------------------------------------------------------<br> <a href="mailto:giacinto.donvito@ba.infn.it" target="_blank">giacinto.donvito@ba.infn.it</a> | GTalk/GMail: <a href="mailto:donvito.giacinto@gmail.com" target="_blank">donvito.giacinto@gmail.com</a><br> tel. +39 080 5443244 Fax +39 0805442470 | Skype: giacinto_it<div class="im"><br> VOIP: +41225481596 | MSN: <a href="mailto:donvito.giacinto@hotmail.it" target="_blank">donvito.giacinto@hotmail.it</a><br></div><div class="im"> AIM/iChat: gdonvito1 | Yahoo: eric1_it<br> ------------------------------------------------------------------<br></div> Life is something that everyone should try at least once.<br> Henry J. Tillman<br> <br> <br> <br> <br> <br> Il giorno 01/lug/09, alle ore 12:24, Zhiyong Landen tian ha scritto:<div><div></div><div class="h5"><br> <br> <blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"> <br> <blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"> Hi All,<br> <br> I'm experiencing some problem in a test installation of lustre 1.8.0.X<br> <br> The installation is composed by one server hosting the MDS, and two servers hosting the OSTs.<br> One of the servers has 12x2.7TB devices and the other has 16x2.3TB devices.<br> <br> All the devices were configured with:<br> <br> "tunefs.lustre --ost --mgsnode=lustre01@tcp0 --param ost.quota_type=ug --writeconf /dev/sdxx"<br> <br> on the admin node I issued the "lfs quotacheck -ug /lustre" (I see read operation occurring on the both disk servers) that ends without error.<br> <br> I was able to set-up quotas per user on the admin node and it seems successfully registered by checking with: "lfs quota -u donvito /lustre"<br> <br> The problem that I see is that it is possible for a user to overfill the quota as the two server behave differently: one of the two deny writing while the other not.<br> I tried with both lustre rpms and vanilla (2.6.22) patched kernel and the result is the same. It is not related to the physical server as both of them sometimes has the same behaviour (But only one of the server at the time). I have tried with both 1.8.0 and 1.8.0.1 and the same behaviour is observed.<br> <br> As you can see the system is correctly accounting the used space but the server do not deny writing:<br> <br> [root@lustre01 ~]# lfs quota -u donvito /lustre<br> Disk quotas for user donvito (uid 501):<br> Filesystem kbytes quota limit grace files quota limit grace<br> /lustre 124927244* 12000000 15000000 24 100 100<br> </blockquote> <br> (hint)You can use "lfs quota -v ..." to get how much quota grant every quota slave has.<br> <br> <blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"> <br> The "messages" log on the MDS and both the OSS servers are clean and nothing strange is noticeable.<br> Do you have any ideas of where I can look in order to understand the problem?<br> </blockquote> It is expected by the current lquota design of lustre. For lustre quota, there are two kinds of roles: quota master(mds) and quota slaves(osts). When you set quota, the limitation is recorded on quota master. When data is written on osts, osts will get some quota grant from quota master if remained quota on osts isn't enough. But, at the same time, osts will get some kinds of "quota grant cache" so that quota slaves won't ask quota from quota master every time when they write(if so, performance will be hurted). Then every quota slave will judge if the request it received will trigger out of quota based upon the grant quota it got from mds _respectively_. Then you get what you saw.<br> <blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"> Do you have any other suggestion of tests that I can do?<br> </blockquote> For causes of performance, lquota of lustre is distributed and isn't exactly like local quota(e.g. ext3). So what you saw is normal to lquota, currently you can only change you application to adapt it if it hurt you<br> <blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"> <br> Thank you very much.<br> <br> Best Regards,<br> Giacinto<br> <br> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~<br> Giacinto Donvito LIBI -- EGEE2 SA1 INFN - Bari ITALY<br> ------------------------------------------------------------------<br> <a href="mailto:giacinto.donvito@ba.infn.it" target="_blank">giacinto.donvito@ba.infn.it</a> <mailto:<a href="mailto:giacinto.donvito@ba.infn.it" target="_blank">giacinto.donvito@ba.infn.it</a>> | GTalk/GMail: <a href="mailto:donvito.giacinto@gmail.com" target="_blank">donvito.giacinto@gmail.com</a> <mailto:<a href="mailto:donvito.giacinto@gmail.com" target="_blank">donvito.giacinto@gmail.com</a>><br> tel. +39 080 5443244 Fax +39 0805442470 VOIP: +41225481596 | MSN: <a href="mailto:donvito.giacinto@hotmail.it" target="_blank">donvito.giacinto@hotmail.it</a> <mailto:<a href="mailto:donvito.giacinto@hotmail.it" target="_blank">donvito.giacinto@hotmail.it</a>><br> Skype: giacinto_it | AIM/iChat: gdonvito1 | Yahoo: eric1_it<br> ------------------------------------------------------------------<br> "A simple design always takes less time to finish than a complex one.<br> So always do the simplest thing that could possibly work."<br> Don <a href="mailto:Wells@www.extremeprogramming.org" target="_blank">Wells@www.extremeprogramming.org</a> <mailto:<a href="mailto:Wells@www.extremeprogramming.org" target="_blank">Wells@www.extremeprogramming.org</a>><br> <br> "Writing about music is like dancing about architecture." - Frank Zappa <<a href="http://feedproxy.google.com/%7Er/randomquotes/%7E3/G2PjcLJ0ONI/" target="_blank">http://feedproxy.google.com/%7Er/randomquotes/%7E3/G2PjcLJ0ONI/</a>><br> ------------------------------------------------------------------------<br> <br> _______________________________________________<br> Lustre-discuss mailing list<br> <a href="mailto:Lustre-discuss@lists.lustre.org" target="_blank">Lustre-discuss@lists.lustre.org</a><br> <a href="http://lists.lustre.org/mailman/listinfo/lustre-discuss" target="_blank">http://lists.lustre.org/mailman/listinfo/lustre-discuss</a><br> <br> </blockquote> <br> </blockquote> <br> </div></div><br>_______________________________________________<br> Lustre-discuss mailing list<br> <a href="mailto:Lustre-discuss@lists.lustre.org">Lustre-discuss@lists.lustre.org</a><br> <a href="http://lists.lustre.org/mailman/listinfo/lustre-discuss" target="_blank">http://lists.lustre.org/mailman/listinfo/lustre-discuss</a><br> <br></blockquote></div><br></blockquote></div><br></div></body></html>