[Lustre-discuss] quota problems in lustre 1.8
Giacinto Donvito
giacinto.donvito at ba.infn.it
Wed Jul 1 04:53:48 PDT 2009
Thank you Zhiyong,
with this hint I was able to find a way to solve the problem.
Cheers,
Giacinto
--
--
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Giacinto Donvito LIBI -- EGEE3 SA1 INFN - Bari ITALY
------------------------------------------------------------------
giacinto.donvito at ba.infn.it | GTalk/GMail: donvito.giacinto at gmail.com
tel. +39 080 5443244 Fax +39 0805442470 | Skype: giacinto_it
VOIP: +41225481596 | MSN: donvito.giacinto at hotmail.it
AIM/iChat: gdonvito1 | Yahoo: eric1_it
------------------------------------------------------------------
Life is something that everyone should try at least once.
Henry J. Tillman
Il giorno 01/lug/09, alle ore 12:24, Zhiyong Landen tian ha scritto:
>
>> Hi All,
>>
>> I'm experiencing some problem in a test installation of lustre
>> 1.8.0.X
>>
>> The installation is composed by one server hosting the MDS, and two
>> servers hosting the OSTs.
>> One of the servers has 12x2.7TB devices and the other has 16x2.3TB
>> devices.
>>
>> All the devices were configured with:
>>
>> "tunefs.lustre --ost --mgsnode=lustre01 at tcp0 --param
>> ost.quota_type=ug --writeconf /dev/sdxx"
>>
>> on the admin node I issued the "lfs quotacheck -ug /lustre" (I see
>> read operation occurring on the both disk servers) that ends
>> without error.
>>
>> I was able to set-up quotas per user on the admin node and it seems
>> successfully registered by checking with: "lfs quota -u donvito /
>> lustre"
>>
>> The problem that I see is that it is possible for a user to
>> overfill the quota as the two server behave differently: one of the
>> two deny writing while the other not.
>> I tried with both lustre rpms and vanilla (2.6.22) patched kernel
>> and the result is the same. It is not related to the physical
>> server as both of them sometimes has the same behaviour (But only
>> one of the server at the time). I have tried with both 1.8.0 and
>> 1.8.0.1 and the same behaviour is observed.
>>
>> As you can see the system is correctly accounting the used space
>> but the server do not deny writing:
>>
>> [root at lustre01 ~]# lfs quota -u donvito /lustre
>> Disk quotas for user donvito (uid 501):
>> Filesystem kbytes quota limit grace files quota
>> limit grace
>> /lustre 124927244* 12000000 15000000 24
>> 100 100
>
> (hint)You can use "lfs quota -v ..." to get how much quota grant
> every quota slave has.
>
>>
>> The "messages" log on the MDS and both the OSS servers are clean
>> and nothing strange is noticeable.
>> Do you have any ideas of where I can look in order to understand
>> the problem?
> It is expected by the current lquota design of lustre. For lustre
> quota, there are two kinds of roles: quota master(mds) and quota
> slaves(osts). When you set quota, the limitation is recorded on
> quota master. When data is written on osts, osts will get some
> quota grant from quota master if remained quota on osts isn't
> enough. But, at the same time, osts will get some kinds of "quota
> grant cache" so that quota slaves won't ask quota from quota master
> every time when they write(if so, performance will be hurted). Then
> every quota slave will judge if the request it received will trigger
> out of quota based upon the grant quota it got from mds
> _respectively_. Then you get what you saw.
>> Do you have any other suggestion of tests that I can do?
> For causes of performance, lquota of lustre is distributed and isn't
> exactly like local quota(e.g. ext3). So what you saw is normal to
> lquota, currently you can only change you application to adapt it if
> it hurt you
>>
>> Thank you very much.
>>
>> Best Regards,
>> Giacinto
>>
>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>> Giacinto Donvito LIBI -- EGEE2 SA1 INFN - Bari ITALY
>> ------------------------------------------------------------------
>> giacinto.donvito at ba.infn.it
>> <mailto:giacinto.donvito at ba.infn.it> | GTalk/
>> GMail: donvito.giacinto at gmail.com <mailto:donvito.giacinto at gmail.com>
>> tel. +39 080 5443244 Fax +39 0805442470 VOIP: +41225481596 |
>> MSN: donvito.giacinto at hotmail.it <mailto:donvito.giacinto at hotmail.it>
>> Skype: giacinto_it | AIM/iChat: gdonvito1 | Yahoo: eric1_it
>> ------------------------------------------------------------------
>> "A simple design always takes less time to finish than a complex one.
>> So always do the simplest thing that could possibly work."
>> Don Wells at www.extremeprogramming.org <mailto:Wells at www.extremeprogramming.org
>> >
>>
>> "Writing about music is like dancing about architecture." - Frank
>> Zappa <http://feedproxy.google.com/%7Er/randomquotes/%7E3/G2PjcLJ0ONI/
>> >
>> ------------------------------------------------------------------------
>>
>> _______________________________________________
>> Lustre-discuss mailing list
>> Lustre-discuss at lists.lustre.org
>> http://lists.lustre.org/mailman/listinfo/lustre-discuss
>>
>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 1760 bytes
Desc: not available
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20090701/6ee815e0/attachment.bin>
More information about the lustre-discuss
mailing list