[lustre-discuss] Directory striping not working in Lustre 2.8?
Juan PC
piernas at ditec.um.es
Mon May 16 08:28:47 PDT 2016
Hi Patrick,
If you mean something like the attached file, yes, I can collect that
information from the different servers. After that, what should I do?
Thanks,
Juan
El 16/05/16 a las 15:10, Patrick Farrell escribió:
>
> Juan,
>
> A first thought is that you've probably hit a bug, but also that more information would be good - For starters, do you have stack traces for any of these stuck threads?
>
> - Patrick
> ________________________________________
> From: lustre-discuss <lustre-discuss-bounces at lists.lustre.org> on behalf of Juan PC <piernas at ditec.um.es>
> Sent: Monday, May 16, 2016 3:33:23 AM
> To: lustre-discuss at lists.lustre.org
> Subject: [lustre-discuss] Directory striping not working in Lustre 2.8?
>
> Hi,
>
> I have put MDTs and OSTs on different servers (either one MDT or one OST
> per server), and Lustre becomes unresponsive where there are more than
> one MDT, directories are striped, and workload is metadata-intensive.
> Error messages in one of the MDTs (which is also the MGS) are:
>
> BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpc_hr01_000:2382]
> BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpc_hr01_000:2382]
> BUG: soft lockup - CPU#3 stuck for 22s! [ptlrpc_hr01_002:2384]
> BUG: soft lockup - CPU#3 stuck for 23s! [ptlrpc_hr01_002:2384]
> BUG: soft lockup - CPU#4 stuck for 22s! [jbd2/sdb1-8:4498]
> BUG: soft lockup - CPU#4 stuck for 23s! [jbd2/sdb1-8:4498]
> BUG: soft lockup - CPU#5 stuck for 22s! [ptlrpc_hr01_001:2383]
> BUG: soft lockup - CPU#5 stuck for 23s! [ptlrpc_hr01_001:2383]
> BUG: soft lockup - CPU#6 stuck for 22s! [jbd2/sdb1-8:4225]
> BUG: soft lockup - CPU#6 stuck for 22s! [jbd2/sdb1-8:4498]
> BUG: soft lockup - CPU#6 stuck for 23s! [jbd2/sdb1-8:4498]
> BUG: soft lockup - CPU#7 stuck for 23s! [ptlrpc_hr01_003:2385]
>
> In the other MDT, error messages are similar:
>
> BUG: soft lockup - CPU#6 stuck for 22s! [jbd2/sdb1-8:4421]
> BUG: soft lockup - CPU#6 stuck for 23s! [jbd2/sdb1-8:4421]
> BUG: soft lockup - CPU#7 stuck for 22s! [jbd2/sdb1-8:4421]
> BUG: soft lockup - CPU#7 stuck for 23s! [jbd2/sdb1-8:4421]
>
> Workload is generated by scenarios 9, 10, 11 and 12 of the HPCS-IO
> suite, which stat thousands of empty files.
>
> With a single MDT, and one or more OSTs, no problem appears.
>
> Any ideas or things I can try?
>
> Regards,
>
> Juan
>
> El 11/05/16 a las 12:19, Juan PC escribió:
>> Hello Olaf:
>>
>> Thanks for replying. In my setup, each MDS also has a single MDT, and
>> each OSS has a single OST too. The difference, maybe, is that I have a
>> MDT and an OST running in the same server, and this seems to cause some
>> problems. I will try other configurations.
>>
>> Regards,
>>
>> Juan
>>
>> El 10/05/16 a las 20:41, Faaland, Olaf P. escribió:
>>> Hello Juan,
>>>
>>> No, I haven't seen the problem you describe. Our testing configuration has only a single target per server - each MDS has a single MDT, and each OSS has a single OST.
>>>
>>> We've encountered some issues, but so far stability has been good. Our testing has been on relatively small scale, though.
>>>
>>> Olaf P. Faaland
>>> Livermore Computing
>>>
>>> ________________________________________
>>> From: Juan PC [piernas at ditec.um.es]
>>> Sent: Monday, May 09, 2016 4:25 AM
>>> To: Faaland, Olaf P.; lustre-discuss at lists.lustre.org
>>> Subject: Re: [lustre-discuss] default directory striping with Lustre 2.8
>>>
>>> Dear Olaf,
>>>
>>> I would like to hear about your experience with stripped directories in
>>> Lustre 2.8. Mine is that this feature is still unstable and a lot of
>>> "BUG: soft lockup..." errors start appearing. Maybe the problem is the
>>> setup that I use: as many OSTs as MDTs, with a pair OST-MDT per server.
>>>
>>> Have you faced the same problem?
>>>
>>> Regards,
>>>
>>> Juan
>>>
>>>
>>> El 05/05/16 a las 01:04, Faaland, Olaf P. escribió:
>>>> Hi,
>>>>
>>>> Suppose you have m MDTs in your filesystem, and create a new directory
>>>> and set default directory striping using
>>>>
>>>> lfs mkdir --count=c --index=k <path> && lfs setdirstripe --default
>>>> --count=c <path>
>>>>
>>>> Suppose that c < m and m > 2.
>>>>
>>>> Then you make subdirectories, like
>>>>
>>>> mkdir <path>/child.{1,2,3,...}
>>>>
>>>> a) By design, do the child directories have the same starting index as
>>>> <path>?
>>>> b) By design, are the child directories all striped across the same set
>>>> of MDTs as <path>?
>>>>
>>>> I didn't see that specified one way or the other in the DNE phase 2 high
>>>> level design document at
>>>> http://wiki.opensfs.org/DNE_StripedDirectories_HighLevelDesign_wiki_version.
>>>> If I should look elsewhere, let me know.
>>>>
>>>> In a test I was doing today, I noticed that neither (a) nor (b) were
>>>> true in practice. I'm wondering whether that's a bug or a feature.
>>>> Here's partial output from my test.
>>>>
>>>> $ lfs mkdir --count=6 --index=2 /p/lustre/faaland1/count6_index2
>>>> $ lfs setdirstripe -D --count=6 /p/lustre/faaland1/count6_index2
>>>> $ mkdir
>>>> /p/lustre/faaland1/count6_index2/subdir.{1,2,3,4,5,6,7,8,9,10,11,12,13,14}
>>>> $ lfs getdirstripe /p/lustre/faaland1/count6_index2
>>>> /p/lustre/faaland1/count6_index2
>>>> lmv_stripe_count: 6 lmv_stripe_offset: 2
>>>> mdtidx FID[seq:oid:ver]
>>>> 2 [0x280000400:0x33f3:0x0]
>>>> 3 [0x2c0000404:0x33f3:0x0]
>>>> 4 [0x300000402:0x33f2:0x0]
>>>> 5 [0x340000407:0x33f1:0x0]
>>>> 6 [0x380000406:0x33f0:0x0]
>>>> 7 [0x3c0000404:0x33ef:0x0]
>>>> /p/lustre/faaland1/count6_index2/subdir.4
>>>> lmv_stripe_count: 6 lmv_stripe_offset: 2
>>>> mdtidx FID[seq:oid:ver]
>>>> 2 [0x280000400:0x33f5:0x0]
>>>> 3 [0x2c0000404:0x33f5:0x0]
>>>> 4 [0x300000402:0x33f4:0x0]
>>>> 5 [0x340000407:0x33f3:0x0]
>>>> 6 [0x380000406:0x33f2:0x0]
>>>> 7 [0x3c0000404:0x33f1:0x0]
>>>> /p/lustre/faaland1/count6_index2/subdir.9
>>>> lmv_stripe_count: 6 lmv_stripe_offset: 5
>>>> mdtidx FID[seq:oid:ver]
>>>> 5 [0x340000400:0x37a1:0x0]
>>>> 6 [0x380000405:0x37a1:0x0]
>>>> 7 [0x3c0000402:0x37a0:0x0]
>>>> 8 [0x40000040e:0x379f:0x0]
>>>> 9 [0x440000403:0x379e:0x0]
>>>> 0 [0x200000405:0x379d:0x0]
>>>> /p/lustre/faaland1/count6_index2/subdir.3
>>>> lmv_stripe_count: 6 lmv_stripe_offset: 5
>>>> mdtidx FID[seq:oid:ver]
>>>> 5 [0x340000400:0x37a0:0x0]
>>>> 6 [0x380000405:0x37a0:0x0]
>>>> 7 [0x3c0000402:0x379f:0x0]
>>>> 8 [0x40000040e:0x379e:0x0]
>>>> 9 [0x440000403:0x379d:0x0]
>>>> 0 [0x200000405:0x379c:0x0]
>>>> /p/lustre/faaland1/count6_index2/subdir.14
>>>> lmv_stripe_count: 6 lmv_stripe_offset: 7
>>>> mdtidx FID[seq:oid:ver]
>>>> 7 [0x3c0000400:0x30d4:0x0]
>>>> 8 [0x400000403:0x30d4:0x0]
>>>> 9 [0x440000405:0x30d3:0x0]
>>>> 0 [0x200000407:0x30d2:0x0]
>>>> 1 [0x240000407:0x30d1:0x0]
>>>> 2 [0x280000407:0x30d0:0x0]
>>>> ...
>>>>
>>>>
>>>> Olaf P. Faaland
>>>> Livermore Computing
>>>> phone : 925-422-2263
>>>>
>>>>
>>>> _______________________________________________
>>>> lustre-discuss mailing list
>>>> lustre-discuss at lists.lustre.org
>>>> http://secure-web.cisco.com/1b6O1t0NM65SMDLd0VNnGkO8_2o0mWu7q_SotZhqVF-Z73mleDDzd45s2PwtZKJUD3Ztk-yckmuoGMq16E0Fw-LUq6hUtZjrPDpumyab9NGBeDZW21yP4CD9mliUhSCZwbhRAGnVnx6TJbxY-lfFJNAdY6Pdfv1hIx4bNQDu5cYl-aIXJWj15Y_Lum0LphN60mITae7Uf-w6xbGKd5g1bEyUK5SHoQ-NIJdGLF215kUypB_BB-96iWRj3tsoPdUp_ClSF7fBDItzoBV5jgOeHam7Ne6ztUVj7zqKJ8wQEEwJAC9CV_u7N7rLrxVaOVuFO2drWjpFe8tnmgn8tlPCjaH8Q4Y85Unr_I3dcOI4xZyo/http%3A%2F%2Flists.lustre.org%2Flistinfo.cgi%2Flustre-discuss-lustre.org
>>>>
>>>
>>>
>>> --
>>> D. Juan Piernas Cánovas
>>> Departamento de Ingeniería y Tecnología de Computadores
>>> Facultad de Informática. Universidad de Murcia
>>> Campus de Espinardo - 30080 Murcia (SPAIN)
>>> Tel.: +34868887657 Fax: +34868884151
>>> email: piernas at ditec.um.es
>>> PGP public key:
>>> http://pgp.rediris.es:11371/pks/lookup?search=piernas%40ditec.um.es&op=index
>>>
>>> *** Por favor, envíeme sus documentos en formato texto, HTML, PDF o
>>> PostScript :-) ***
>>>
>>
>>
>
>
> --
> D. Juan Piernas Cánovas
> Departamento de Ingeniería y Tecnología de Computadores
> Facultad de Informática. Universidad de Murcia
> Campus de Espinardo - 30080 Murcia (SPAIN)
> Tel.: +34868887657 Fax: +34868884151
> email: piernas at ditec.um.es
> PGP public key:
> http://pgp.rediris.es:11371/pks/lookup?search=piernas%40ditec.um.es&op=index
>
> *** Por favor, envíeme sus documentos en formato texto, HTML, PDF o
> PostScript :-) ***
> _______________________________________________
> lustre-discuss mailing list
> lustre-discuss at lists.lustre.org
> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
>
--
D. Juan Piernas Cánovas
Departamento de Ingeniería y Tecnología de Computadores
Facultad de Informática. Universidad de Murcia
Campus de Espinardo - 30080 Murcia (SPAIN)
Tel.: +34868887657 Fax: +34868884151
email: piernas at ditec.um.es
PGP public key:
http://pgp.rediris.es:11371/pks/lookup?search=piernas%40ditec.um.es&op=index
*** Por favor, envíeme sus documentos en formato texto, HTML, PDF o
PostScript :-) ***
-------------- next part --------------
May 15 20:25:52 localhost kernel: BUG: soft lockup - CPU#6 stuck for 22s! [jbd2/sdb1-8:4421]
May 15 20:25:52 localhost kernel: Modules linked in: mdd(OE) lod(OE) mdt(OE) osp(OE) ofd(OE) lfsck(OE) ost(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ksocklnd(OE) ptlrpc(OE) obdc
lass(OE) lnet(OE) sha512_generic crypto_null libcfs(OE) ldiskfs(OE) mbcache jbd2 rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache xprtrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_trans
port_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_sa iTCO_wdt iTCO_vendor_support lpc_ich mfd_core ipmi_
ssif i5400_edac ipmi_si ipmi_msghandler coretemp ioatdma edac_core pcspkr sg i2c_i801 dca shpchp i5k_amb kvm nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sd_mod crc_t1
0dif crct10dif_generic crct10dif_common ata_generic pata_acpi radeon
May 15 20:25:52 localhost kernel: i2c_algo_bit drm_kms_helper ttm ata_piix e1000e drm ptp ib_mthca libata serio_raw ib_mad i2c_core ib_core pps_core ib_addr dm_mirror dm_region_hash dm_log d
m_mod
May 15 20:25:52 localhost kernel: CPU: 6 PID: 4421 Comm: jbd2/sdb1-8 Tainted: G IOEL ------------ 3.10.0-327.3.1.el7_lustre.x86_64 #1
May 15 20:25:52 localhost kernel: Hardware name: Supermicro X7DWT/X7DWT, BIOS 1.2c 11/19/2010
May 15 20:25:52 localhost kernel: task: ffff88008bc88000 ti: ffff88006a540000 task.ti: ffff88006a540000
May 15 20:25:52 localhost kernel: RIP: 0010:[<ffffffffa0d36ff1>] [<ffffffffa0d36ff1>] ptlrpc_commit_replies+0xd1/0x380 [ptlrpc]
May 15 20:25:52 localhost kernel: RSP: 0018:ffff88006a543b90 EFLAGS: 00000202
May 15 20:25:52 localhost kernel: RAX: ffff880055f17030 RBX: 0008000000000001 RCX: 000000010009d6c0
May 15 20:25:52 localhost kernel: RDX: ffff880055f17000 RSI: ffff8801019e5af0 RDI: ffff880118258538
May 15 20:25:52 localhost kernel: RBP: ffff88006a543c00 R08: ffff880090719000 R09: 000000010020000c
May 15 20:25:52 localhost kernel: R10: ffffea00047e1b80 R11: ffffffffa0f32975 R12: ffff88006a543bb0
May 15 20:25:52 localhost kernel: R13: ffff88012b003600 R14: ffff8800512a1400 R15: ffffea000144a800
May 15 20:25:52 localhost kernel: FS: 0000000000000000(0000) GS:ffff88012fd80000(0000) knlGS:0000000000000000
May 15 20:25:52 localhost kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
May 15 20:25:52 localhost kernel: CR2: 00007fc46efd1b40 CR3: 00000000c6784000 CR4: 00000000000407e0
May 15 20:25:52 localhost kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
May 15 20:25:52 localhost kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
May 15 20:25:52 localhost kernel: Stack:
May 15 20:25:52 localhost kernel: ffff880118258538 ffffffffa0add8e1 ffff88006a543ba0 ffff88006a543ba0
May 15 20:25:52 localhost kernel: 0000000000000000 0000000000000000 0000000000000028 000000009f820f30
May 15 20:25:52 localhost kernel: ffff88006a543bf8 ffff880090719000 ffff88000eefce00 0000000000000000
May 15 20:25:52 localhost kernel: Call Trace:
May 15 20:25:52 localhost kernel: [<ffffffffa0add8e1>] ? keys_fini+0xb1/0x1d0 [obdclass]
May 15 20:25:52 localhost kernel: [<ffffffffa0d7d71c>] tgt_cb_last_committed+0x16c/0x3d0 [ptlrpc]
May 15 20:25:52 localhost kernel: [<ffffffffa0f328cb>] osd_trans_commit_cb+0xbb/0x380 [osd_ldiskfs]
May 15 20:25:52 localhost kernel: [<ffffffffa09588e4>] ldiskfs_journal_commit_callback+0x84/0xc0 [ldiskfs]
May 15 20:25:52 localhost kernel: [<ffffffffa0838364>] jbd2_journal_commit_transaction+0x1494/0x19a0 [jbd2]
May 15 20:25:52 localhost kernel: [<ffffffff81013588>] ? __switch_to+0xf8/0x4b0
May 15 20:25:52 localhost kernel: [<ffffffffa083cd79>] kjournald2+0xc9/0x260 [jbd2]
May 15 20:25:52 localhost kernel: [<ffffffff810a6ae0>] ? wake_up_atomic_t+0x30/0x30
May 15 20:25:52 localhost kernel: [<ffffffffa083ccb0>] ? commit_timeout+0x10/0x10 [jbd2]
May 15 20:25:52 localhost kernel: [<ffffffff810a5aef>] kthread+0xcf/0xe0
May 15 20:25:52 localhost kernel: [<ffffffff810a5a20>] ? kthread_create_on_node+0x140/0x140
May 15 20:25:52 localhost kernel: [<ffffffff81645a58>] ret_from_fork+0x58/0x90
May 15 20:25:52 localhost kernel: [<ffffffff810a5a20>] ? kthread_create_on_node+0x140/0x140
May 15 20:25:52 localhost kernel: Code: 00 00 49 83 7d 38 00 75 39 e9 8d 02 00 00 0f 1f 40 00 48 8b 43 30 4c 8d 6b 30 4d 39 f5 48 8d 50 d0 0f 84 13 01 00 00 f6 43 44 01 <0f> 84 38 02 00 00 48 83 7b 68 00 0f 84 5f 02 00 00 49 89 df 48
More information about the lustre-discuss
mailing list