[Lustre-discuss] More detail regarding soft lockup error

[Andreas Dilger]

On 2010-08-19, at 00:41, Fourie Joubert wrote:
> Many thanks for the suggestion - I seem to have indeed run out of inodes!

That is what I had asked previously, but I wasn't very clear.

> We have a massive amount of very small files :-(
> 
> Does anyone know if it is possible increase the inodes, or do I need to 
> redo the MDT?

If you increase the size of the MDT (via resize2fs) it will increase the number of inodes as well.

> Ben Evans wrote:
>> I've found MDT utilization is best measured using "df -i" rather than
>> "df -k", since you'll run out of inodes before you run out of space.  No
>> file data is stored on the MDT and the metadata is small.
>> 
>> -----Original Message-----
>> From: lustre-discuss-bounces at lists.lustre.org
>> [mailto:lustre-discuss-bounces at lists.lustre.org] On Behalf Of Fourie
>> Joubert
>> Sent: Wednesday, August 18, 2010 2:01 PM
>> To: Andreas Dilger; lustre-discuss at lists.lustre.org
>> Subject: Re: [Lustre-discuss] More detail regarding soft lockup error
>> 
>>  Hi
>> 
>> Many thanks for the reply!
>> 
>> The MDT is only 5% used.
>> 
>> OST's are all around 30%.
>> 
>> I have run the full series of e2fsck's.
>> 
>> Have been running 1.8.1 as it was built with weak dependencies to 
>> Infiniband, and I needed OFED 1.5 for my hardware....
>> 
>> But, I have recently been told by Marco Gomes (based on a recent forum 
>> discussion?) that 1.8.3 seems to work with OFED 1.5.2-rc2.
>> 
>> Will try to upgrade and see...
>> 
>> Thanks and kindest regards!
>> 
>> Fourie
>> 
>> 
>> 
>> 
>> On 18/08/2010 16:27, Andreas Dilger wrote:
>> 
>>> On 2010-08-18, at 4:14, Fourie Joubert<fourie.joubert at up.ac.za>
>>> 
>> wrote:
>> 
>>>> Just reporting some more detail about the soft lockup error I have
>>>> 
>> been
>> 
>>>> getting:
>>>> 
>>>> I am running Lustre 1.8.1, kernels are from the Lustre distro.
>>>> 
>>> Firstly, there is a known corruption bug in 1.8.1, you should ar
>>> 
>> minimum upgrade to 1.8.1.1, but it may be that this problem is fixed in
>> a newer release.
>> 
>>> 
>>>> Aug 18 12:14:11 wonkofs kernel:  [<ffffffff88b7007f>]
>>>> 
>> :ldiskfs:ldiskfs_get_group_desc+0x3b/0xb0
>> 
>>>> Aug 18 12:14:11 wonkofs kernel:  [<ffffffff88b7007f>]
>>>> :ldiskfs:ldiskfs_find_reverse+0x2f/0xa0
>>>> Aug 18 12:14:11 wonkofs kernel:  [<ffffffff88b78f26>]
>>>> :ldiskfs:ldiskfs_new_inode_wantedi+0x46/0x80
>>>> 
>>> This is getting stuck looking for a free inode. Is there free space in
>>> 
>> the MDT filesystem? Have you run a full e2fsck?
>> 
>>> I do recall some changes is this code a while ago, but I don't recall
>>> 
>> what version it was in offhand.
>> 
>>> Cheers, Andreas
>>> 
>>> 
>> 
>> 
>> 
> 
> 
> -- 
> --------------
> Prof Fourie Joubert
> Associate Professor
> Bioinformatics and Computational Biology Unit
> Department of Biochemistry
> University of Pretoria
> fourie.joubert at up.ac.za
> http://www.bi.up.ac.za
> Tel. +27-12-420-5802
> Fax. +27-12-420-5800
> 
> -------------------------------------------------------------------------
> This message and attachments are subject to a disclaimer. Please refer
> to www.it.up.ac.za/documentation/governance/disclaimer/ for full details.
> 
> _______________________________________________
> Lustre-discuss mailing list
> Lustre-discuss at lists.lustre.org
> http://lists.lustre.org/mailman/listinfo/lustre-discuss


Cheers, Andreas
--
Andreas Dilger
Lustre Technical Lead
Oracle Corporation Canada Inc.