[lustre-discuss] Directory striping not working in Lustre 2.8?

Patrick Farrell paf at cray.com
Mon May 16 06:10:30 PDT 2016


Juan,

A first thought is that you've probably hit a bug, but also that more information would be good - For starters, do you have stack traces for any of these stuck threads?

- Patrick
________________________________________
From: lustre-discuss <lustre-discuss-bounces at lists.lustre.org> on behalf of Juan PC <piernas at ditec.um.es>
Sent: Monday, May 16, 2016 3:33:23 AM
To: lustre-discuss at lists.lustre.org
Subject: [lustre-discuss] Directory striping not working in Lustre 2.8?

Hi,

I have put MDTs and OSTs on different servers (either one MDT or one OST
per server), and Lustre becomes unresponsive where there are more than
one MDT, directories are striped, and workload is metadata-intensive.
Error messages in one of the MDTs (which is also the MGS) are:

 BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpc_hr01_000:2382]
 BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpc_hr01_000:2382]
 BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpc_hr01_002:2384]
 BUG: soft lockup - CPU#3 stuck for 23s! [ptlrpc_hr01_002:2384]
 BUG: soft lockup - CPU#4 stuck for 22s! [jbd2/sdb1-8:4498]
 BUG: soft lockup - CPU#4 stuck for 23s! [jbd2/sdb1-8:4498]
 BUG: soft lockup - CPU#5 stuck for 22s! [ptlrpc_hr01_001:2383]
 BUG: soft lockup - CPU#5 stuck for 23s! [ptlrpc_hr01_001:2383]
 BUG: soft lockup - CPU#6 stuck for 22s! [jbd2/sdb1-8:4225]
 BUG: soft lockup - CPU#6 stuck for 22s! [jbd2/sdb1-8:4498]
 BUG: soft lockup - CPU#6 stuck for 23s! [jbd2/sdb1-8:4498]
 BUG: soft lockup - CPU#7 stuck for 23s! [ptlrpc_hr01_003:2385]

In the other MDT, error messages are similar:

 BUG: soft lockup - CPU#6 stuck for 22s! [jbd2/sdb1-8:4421]
 BUG: soft lockup - CPU#6 stuck for 23s! [jbd2/sdb1-8:4421]
 BUG: soft lockup - CPU#7 stuck for 22s! [jbd2/sdb1-8:4421]
 BUG: soft lockup - CPU#7 stuck for 23s! [jbd2/sdb1-8:4421]

Workload is generated by scenarios 9, 10, 11 and 12 of the HPCS-IO
suite, which stat thousands of empty files.

With a single MDT, and one or more OSTs, no problem appears.

Any ideas or things I can try?

Regards,

        Juan

El 11/05/16 a las 12:19, Juan PC escribió:
> Hello Olaf:
>
> Thanks for replying. In my setup, each MDS also has a single MDT, and
> each OSS has a single OST too. The difference, maybe, is that I have a
> MDT and an OST running in the same server, and this seems to cause some
> problems. I will try other configurations.
>
> Regards,
>
>       Juan
>
> El 10/05/16 a las 20:41, Faaland, Olaf P. escribió:
>> Hello Juan,
>>
>> No, I haven't seen the problem you describe.  Our testing configuration has only a single target per server - each MDS has a single MDT, and each OSS has a single OST.
>>
>> We've encountered some issues, but so far stability has been good.  Our testing has been on relatively small scale, though.
>>
>> Olaf P. Faaland
>> Livermore Computing
>>
>> ________________________________________
>> From: Juan PC [piernas at ditec.um.es]
>> Sent: Monday, May 09, 2016 4:25 AM
>> To: Faaland, Olaf P.; lustre-discuss at lists.lustre.org
>> Subject: Re: [lustre-discuss] default directory striping with Lustre 2.8
>>
>> Dear Olaf,
>>
>> I would like to hear about your experience with stripped directories in
>> Lustre 2.8. Mine is that this feature is still unstable and a lot of
>> "BUG: soft lockup..." errors start appearing. Maybe the problem is the
>> setup that I use: as many OSTs as MDTs, with a pair OST-MDT per server.
>>
>> Have you faced the same problem?
>>
>> Regards,
>>
>>         Juan
>>
>>
>> El 05/05/16 a las 01:04, Faaland, Olaf P. escribió:
>>> Hi,
>>>
>>> Suppose you have m MDTs in your filesystem, and create a new directory
>>> and set default directory striping using
>>>
>>> lfs mkdir --count=c --index=k <path> && lfs setdirstripe --default
>>> --count=c <path>
>>>
>>> Suppose that c < m and m > 2.
>>>
>>> Then you make subdirectories, like
>>>
>>> mkdir <path>/child.{1,2,3,...}
>>>
>>> a) By design, do the child directories have the same starting index as
>>> <path>?
>>> b) By design, are the child directories all striped across the same set
>>> of MDTs as <path>?
>>>
>>> I didn't see that specified one way or the other in the DNE phase 2 high
>>> level design document at
>>> http://wiki.opensfs.org/DNE_StripedDirectories_HighLevelDesign_wiki_version.
>>> If I should look elsewhere, let me know.
>>>
>>> In a test I was doing today, I noticed that neither (a) nor (b) were
>>> true in practice.  I'm wondering whether that's a bug or a feature.
>>> Here's partial output from my test.
>>>
>>> $ lfs mkdir --count=6 --index=2 /p/lustre/faaland1/count6_index2
>>> $ lfs setdirstripe -D --count=6 /p/lustre/faaland1/count6_index2
>>> $ mkdir
>>> /p/lustre/faaland1/count6_index2/subdir.{1,2,3,4,5,6,7,8,9,10,11,12,13,14}
>>> $ lfs getdirstripe /p/lustre/faaland1/count6_index2
>>> /p/lustre/faaland1/count6_index2
>>> lmv_stripe_count: 6 lmv_stripe_offset: 2
>>> mdtidx           FID[seq:oid:ver]
>>>      2           [0x280000400:0x33f3:0x0]
>>>      3           [0x2c0000404:0x33f3:0x0]
>>>      4           [0x300000402:0x33f2:0x0]
>>>      5           [0x340000407:0x33f1:0x0]
>>>      6           [0x380000406:0x33f0:0x0]
>>>      7           [0x3c0000404:0x33ef:0x0]
>>> /p/lustre/faaland1/count6_index2/subdir.4
>>> lmv_stripe_count: 6 lmv_stripe_offset: 2
>>> mdtidx           FID[seq:oid:ver]
>>>      2           [0x280000400:0x33f5:0x0]
>>>      3           [0x2c0000404:0x33f5:0x0]
>>>      4           [0x300000402:0x33f4:0x0]
>>>      5           [0x340000407:0x33f3:0x0]
>>>      6           [0x380000406:0x33f2:0x0]
>>>      7           [0x3c0000404:0x33f1:0x0]
>>> /p/lustre/faaland1/count6_index2/subdir.9
>>> lmv_stripe_count: 6 lmv_stripe_offset: 5
>>> mdtidx           FID[seq:oid:ver]
>>>      5           [0x340000400:0x37a1:0x0]
>>>      6           [0x380000405:0x37a1:0x0]
>>>      7           [0x3c0000402:0x37a0:0x0]
>>>      8           [0x40000040e:0x379f:0x0]
>>>      9           [0x440000403:0x379e:0x0]
>>>      0           [0x200000405:0x379d:0x0]
>>> /p/lustre/faaland1/count6_index2/subdir.3
>>> lmv_stripe_count: 6 lmv_stripe_offset: 5
>>> mdtidx           FID[seq:oid:ver]
>>>      5           [0x340000400:0x37a0:0x0]
>>>      6           [0x380000405:0x37a0:0x0]
>>>      7           [0x3c0000402:0x379f:0x0]
>>>      8           [0x40000040e:0x379e:0x0]
>>>      9           [0x440000403:0x379d:0x0]
>>>      0           [0x200000405:0x379c:0x0]
>>> /p/lustre/faaland1/count6_index2/subdir.14
>>> lmv_stripe_count: 6 lmv_stripe_offset: 7
>>> mdtidx           FID[seq:oid:ver]
>>>      7           [0x3c0000400:0x30d4:0x0]
>>>      8           [0x400000403:0x30d4:0x0]
>>>      9           [0x440000405:0x30d3:0x0]
>>>      0           [0x200000407:0x30d2:0x0]
>>>      1           [0x240000407:0x30d1:0x0]
>>>      2           [0x280000407:0x30d0:0x0]
>>> ...
>>>
>>>
>>> Olaf P. Faaland
>>> Livermore Computing
>>> phone : 925-422-2263
>>>
>>>
>>> _______________________________________________
>>> lustre-discuss mailing list
>>> lustre-discuss at lists.lustre.org
>>> http://secure-web.cisco.com/1b6O1t0NM65SMDLd0VNnGkO8_2o0mWu7q_SotZhqVF-Z73mleDDzd45s2PwtZKJUD3Ztk-yckmuoGMq16E0Fw-LUq6hUtZjrPDpumyab9NGBeDZW21yP4CD9mliUhSCZwbhRAGnVnx6TJbxY-lfFJNAdY6Pdfv1hIx4bNQDu5cYl-aIXJWj15Y_Lum0LphN60mITae7Uf-w6xbGKd5g1bEyUK5SHoQ-NIJdGLF215kUypB_BB-96iWRj3tsoPdUp_ClSF7fBDItzoBV5jgOeHam7Ne6ztUVj7zqKJ8wQEEwJAC9CV_u7N7rLrxVaOVuFO2drWjpFe8tnmgn8tlPCjaH8Q4Y85Unr_I3dcOI4xZyo/http%3A%2F%2Flists.lustre.org%2Flistinfo.cgi%2Flustre-discuss-lustre.org
>>>
>>
>>
>> --
>> D. Juan Piernas Cánovas
>> Departamento de Ingeniería y Tecnología de Computadores
>> Facultad de Informática. Universidad de Murcia
>> Campus de Espinardo - 30080 Murcia (SPAIN)
>> Tel.: +34868887657    Fax: +34868884151
>> email: piernas at ditec.um.es
>> PGP public key:
>> http://pgp.rediris.es:11371/pks/lookup?search=piernas%40ditec.um.es&op=index
>>
>> *** Por favor, envíeme sus documentos en formato texto, HTML, PDF o
>> PostScript :-) ***
>>
>
>


--
D. Juan Piernas Cánovas
Departamento de Ingeniería y Tecnología de Computadores
Facultad de Informática. Universidad de Murcia
Campus de Espinardo - 30080 Murcia (SPAIN)
Tel.: +34868887657    Fax: +34868884151
email: piernas at ditec.um.es
PGP public key:
http://pgp.rediris.es:11371/pks/lookup?search=piernas%40ditec.um.es&op=index

*** Por favor, envíeme sus documentos en formato texto, HTML, PDF o
PostScript :-) ***
_______________________________________________
lustre-discuss mailing list
lustre-discuss at lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org


More information about the lustre-discuss mailing list