[lustre-discuss] MDS kernel panics - 2.5.3 - waking for gap in transno

Steve Barnet barnet at icecube.wisc.edu
Mon Sep 14 12:06:28 PDT 2015


Hi all,

   Earlier today, one of our MDSes kernel panicked. I was not able
to grab the stack trace, but it appeared to be in the locking code
path.

The last entries prior to the crash on our remote syslog server
looked like this:

Sep 14 10:17:19 lfs-us-mds kernel: LustreError: 
6864:0:(mdt_handler.c:1444:mdt_getattr_name_lock()) ASSERTION( lock != 
NULL ) failed: Invalid lock handle 0x55755bac33ca291d
Sep 14 10:17:19 lfs-us-mds kernel: LustreError: 
6864:0:(mdt_handler.c:1444:mdt_getattr_name_lock()) LBUG


After a reboot and e2fsck of the MDT, we saw many errors that look
like this:


Sep 14 12:40:35 lfs-us-mds kernel: LustreError: 
3307:0:(ldlm_lib.c:1751:check_for_next_transno()) lfs3-MDT0000: waking 
for gap in transno, VBR is OFF (skip: 66055727404, ql: 15, comp: 172, 
conn: 187, next: 66055727409, last_committed: 66055720625)


The output from the e2fsck is attached, but the quick summary is
that it was entirely QUOTA WARNINGs. Shortly after recovery
completed, the machine panicked again (trace attached).

After a reboot, it appears that the system has stabilized, but
as you might expect, it does not leave me with a warm, fuzzy
feeling.

Some of my searching indicates that an lfsck may be needed, but
before I start too far down that path, I'd like to have some idea
that this is indeed reasonable.

We are using 2.5.3 on all servers and clients:

lfs-us-mds ~ # cat /proc/fs/lustre/version
lustre: 2.5.3
kernel: patchless_client
build:  2.5.3-RC1--PRISTINE-2.6.32-431.23.3.el6_lustre.x86_64


OS is Scientific Linux 6.4:

Linux lfs-us-mds 2.6.32-431.23.3.el6_lustre.x86_64 #1 SMP Thu Aug 28 
20:20:13 PDT 2014 x86_64 x86_64 x86_64 GNU/Linux


Any advice?

Thanks much, in advance!

Best,

---Steve

--
Steve Barnet
UW-Madison - IceCube
barnet at icecube.wisc.edu


-------------- next part --------------
lfs-us-mds ~ # e2fsck -fp /dev/sdb
lfs3-MDT0000: recovering journal
[QUOTA WARNING] Usage inconsistent for ID 0:actual (15023759360, 226810) != expected (19607052288, 245133)
[QUOTA WARNING] Usage inconsistent for ID 10210:actual (78381056, 230125) != expected (75276288, 192681)
[QUOTA WARNING] Usage inconsistent for ID 33480:actual (449388544, 724085) != expected (449351680, 724040)
[QUOTA WARNING] Usage inconsistent for ID 50330:actual (458280960, 312174) != expected (458280960, 312173)
[QUOTA WARNING] Usage inconsistent for ID 33416:actual (4321316864, 3795704) != expected (4044886016, 3506498)
[QUOTA WARNING] Usage inconsistent for ID 17437:actual (339566592, 487621) != expected (1359523840, 8241900)
[QUOTA WARNING] Usage inconsistent for ID 15329:actual (318918656, 657583) != expected (269213696, 496130)
[QUOTA WARNING] Usage inconsistent for ID 37701:actual (1103691776, 1146625) != expected (971554816, 969972)
[QUOTA WARNING] Usage inconsistent for ID 23129:actual (579133440, 4448732) != expected (579379200, 4449891)
[QUOTA WARNING] Usage inconsistent for ID 31251:actual (17772544, 213487) != expected (17752064, 215409)
[QUOTA WARNING] Usage inconsistent for ID 50225:actual (189779968, 216721) != expected (189005824, 234420)
[QUOTA WARNING] Usage inconsistent for ID 41003:actual (550715392, 3005115) != expected (323002368, 1068804)
[QUOTA WARNING] Usage inconsistent for ID 51509:actual (739606528, 575118) != expected (739540992, 574880)
[QUOTA WARNING] Usage inconsistent for ID 41279:actual (76873728, 192292) != expected (20619264, 50497)
[QUOTA WARNING] Usage inconsistent for ID 1823:actual (135487488, 175342) != expected (134180864, 167683)
[QUOTA WARNING] Usage inconsistent for ID 46376:actual (25743360, 35712) != expected (25350144, 31629)
[QUOTA WARNING] Usage inconsistent for ID 501:actual (9920512, 37723) != expected (9920512, 37713)
[QUOTA WARNING] Usage inconsistent for ID 37735:actual (69378048, 897135) != expected (65445888, 864621)
[QUOTA WARNING] Usage inconsistent for ID 6204:actual (27136000, 29948) != expected (27136000, 29924)
[QUOTA WARNING] Usage inconsistent for ID 13092:actual (4096, 20178) != expected (4096, 15903)
[QUOTA WARNING] Usage inconsistent for ID 17657:actual (244539392, 234381) != expected (244539392, 234368)
[QUOTA WARNING] Usage inconsistent for ID 5427:actual (27328512, 214542) != expected (27324416, 214507)
[QUOTA WARNING] Usage inconsistent for ID 47558:actual (303357952, 713320) != expected (303357952, 714298)
[QUOTA WARNING] Usage inconsistent for ID 27572:actual (218345472, 996620) != expected (218345472, 996619)
[QUOTA WARNING] Usage inconsistent for ID 46314:actual (457404416, 3944252) != expected (0, 8709)
[QUOTA WARNING] Usage inconsistent for ID 26165:actual (22102016, 189712) != expected (22102016, 206743)
[QUOTA WARNING] Usage inconsistent for ID 40893:actual (1303056384, 3147456) != expected (1210523648, 3677756)
[QUOTA WARNING] Usage inconsistent for ID 26992:actual (1749823488, 5134252) != expected (1749684224, 5133980)
[QUOTA WARNING] Usage inconsistent for ID 5408:actual (151408640, 204432) != expected (151339008, 203525)
[QUOTA WARNING] Usage inconsistent for ID 15116:actual (190898176, 223552) != expected (190898176, 223547)
[QUOTA WARNING] Usage inconsistent for ID 9741:actual (109424640, 109499) != expected (108670976, 99550)
[QUOTA WARNING] Usage inconsistent for ID 54750:actual (52523008, 325010) != expected (52183040, 321355)
[QUOTA WARNING] Usage inconsistent for ID 48626:actual (33587200, 50554) != expected (33587200, 45546)
[QUOTA WARNING] Usage inconsistent for ID 34114:actual (33488896, 36116) != expected (33579008, 36374)
[QUOTA WARNING] Usage inconsistent for ID 37328:actual (541990912, 438449) != expected (537448448, 414919)
[QUOTA WARNING] Usage inconsistent for ID 7262:actual (341319680, 382436) != expected (341299200, 382381)
[QUOTA WARNING] Usage inconsistent for ID 633:actual (1353412608, 1513917) != expected (1348395008, 1449381)
[QUOTA WARNING] Usage inconsistent for ID 10412:actual (8458240, 82179) != expected (8142848, 79195)
[QUOTA WARNING] Usage inconsistent for ID 6889:actual (564248576, 600723) != expected (385892352, 306481)
[QUOTA WARNING] Usage inconsistent for ID 17388:actual (304844800, 1031030) != expected (307908608, 1059027)
[QUOTA WARNING] Usage inconsistent for ID 32530:actual (1516093440, 1476546) != expected (1485680640, 1363904)
[QUOTA WARNING] Usage inconsistent for ID 40548:actual (107249664, 86034) != expected (58286080, 55384)
[QUOTA WARNING] Usage inconsistent for ID 1371:actual (49795072, 177363) != expected (15421440, 86341)
[QUOTA WARNING] Usage inconsistent for ID 22059:actual (194437120, 1814834) != expected (193912832, 1806816)
[QUOTA WARNING] Usage inconsistent for ID 54633:actual (79728640, 122957) != expected (18124800, 66136)
[QUOTA WARNING] Usage inconsistent for ID 18702:actual (189685760, 694166) != expected (126910464, 461027)
[QUOTA WARNING] Usage inconsistent for ID 680:actual (114266112, 934734) != expected (114053120, 931364)
[QUOTA WARNING] Usage inconsistent for ID 50142:actual (318468096, 370482) != expected (291581952, 325945)
[QUOTA WARNING] Usage inconsistent for ID 9490:actual (74825728, 702056) != expected (35409920, 300309)
[QUOTA WARNING] Usage inconsistent for ID 8827:actual (38006784, 39115) != expected (37130240, 37407)
[QUOTA WARNING] Usage inconsistent for ID 806:actual (260120576, 516204) != expected (260112384, 516200)
[QUOTA WARNING] Usage inconsistent for ID 29092:actual (252956672, 1078188) != expected (110682112, 692250)
[QUOTA WARNING] Usage inconsistent for ID 36962:actual (14741504, 109636) != expected (14372864, 116021)
[QUOTA WARNING] Usage inconsistent for ID 22243:actual (78413824, 666482) != expected (78303232, 665363)
[QUOTA WARNING] Usage inconsistent for ID 783:actual (692224, 1261) != expected (180224, 767)
[QUOTA WARNING] Usage inconsistent for ID 26260:actual (159744, 616) != expected (159744, 617)
[QUOTA WARNING] Usage inconsistent for ID 25363:actual (34902016, 324567) != expected (34729984, 322371)
[QUOTA WARNING] Usage inconsistent for ID 40834:actual (1759961088, 5182256) != expected (629850112, 2424847)
[QUOTA WARNING] Usage inconsistent for ID 42446:actual (162107392, 1206130) != expected (158162944, 1184231)
[QUOTA WARNING] Usage inconsistent for ID 5110:actual (39157760, 57315) != expected (34295808, 50458)
[QUOTA WARNING] Usage inconsistent for ID 13183:actual (1149480960, 873249) != expected (1137983488, 824649)
[QUOTA WARNING] Usage inconsistent for ID 13893:actual (97591296, 734172) != expected (97591296, 734150)
[QUOTA WARNING] Usage inconsistent for ID 38188:actual (586682368, 1394204) != expected (510181376, 1322923)
[QUOTA WARNING] Usage inconsistent for ID 39623:actual (781099008, 2110157) != expected (965722112, 3958607)
[QUOTA WARNING] Usage inconsistent for ID 12659:actual (26103808, 22849) != expected (25972736, 24725)
[QUOTA WARNING] Usage inconsistent for ID 3113:actual (11833344, 12510) != expected (11489280, 12503)
[QUOTA WARNING] Usage inconsistent for ID 25260:actual (82071552, 254416) != expected (82071552, 254414)
[QUOTA WARNING] Usage inconsistent for ID 28276:actual (8183808, 102108) != expected (1884160, 20597)
[QUOTA WARNING] Usage inconsistent for ID 30438:actual (482426880, 451376) != expected (482418688, 451240)
[QUOTA WARNING] Usage inconsistent for ID 35337:actual (57479168, 51448) != expected (56856576, 41380)
[QUOTA WARNING] Usage inconsistent for ID 26507:actual (882724864, 1604474) != expected (806420480, 1213485)
[QUOTA WARNING] Usage inconsistent for ID 44499:actual (514334720, 707806) != expected (504791040, 612218)
[QUOTA WARNING] Usage inconsistent for ID 24313:actual (139292672, 594390) != expected (139288576, 594390)
[QUOTA WARNING] Usage inconsistent for ID 41267:actual (666238976, 1306210) != expected (660647936, 1236733)
[QUOTA WARNING] Usage inconsistent for ID 6940:actual (51060736, 66167) != expected (54132736, 46303)
lfs3-MDT0000: Update quota info for quota type 0.
[QUOTA WARNING] Usage inconsistent for ID 0:actual (15023988736, 228485) != expected (19607281664, 246808)
[QUOTA WARNING] Usage inconsistent for ID 33430:actual (78381056, 230125) != expected (75276288, 192681)
[QUOTA WARNING] Usage inconsistent for ID 15453:actual (449388544, 724085) != expected (449351680, 724040)
[QUOTA WARNING] Usage inconsistent for ID 24573:actual (339566592, 487621) != expected (1359523840, 8241900)
[QUOTA WARNING] Usage inconsistent for ID 27406:actual (318918656, 657583) != expected (269213696, 496130)
[QUOTA WARNING] Usage inconsistent for ID 50128:actual (1103691776, 1146606) != expected (971554816, 969723)
[QUOTA WARNING] Usage inconsistent for ID 30816:actual (458280960, 312174) != expected (458280960, 312173)
[QUOTA WARNING] Usage inconsistent for ID 29987:actual (727601152, 569453) != expected (727572480, 569216)
[QUOTA WARNING] Usage inconsistent for ID 3777:actual (76873728, 192292) != expected (20619264, 50497)
[QUOTA WARNING] Usage inconsistent for ID 21510:actual (135487488, 175342) != expected (134180864, 167683)
[QUOTA WARNING] Usage inconsistent for ID 22858:actual (541990912, 438449) != expected (537448448, 414919)
[QUOTA WARNING] Usage inconsistent for ID 34734:actual (341319680, 382436) != expected (341299200, 382381)
[QUOTA WARNING] Usage inconsistent for ID 29386:actual (109424640, 109499) != expected (108670976, 99550)
[QUOTA WARNING] Usage inconsistent for ID 41557:actual (303357952, 713320) != expected (303357952, 714298)
[QUOTA WARNING] Usage inconsistent for ID 36935:actual (4321595392, 3797313) != expected (4072370176, 3457670)
[QUOTA WARNING] Usage inconsistent for ID 25851:actual (78417920, 666483) != expected (78307328, 665364)
[QUOTA WARNING] Usage inconsistent for ID 41206:actual (318468096, 370482) != expected (291581952, 325945)
[QUOTA WARNING] Usage inconsistent for ID 8111:actual (25743360, 35712) != expected (25350144, 31629)
[QUOTA WARNING] Usage inconsistent for ID 49013:actual (550715392, 3005113) != expected (323002368, 1068802)
[QUOTA WARNING] Usage inconsistent for ID 3113:actual (1303056384, 3147456) != expected (1210523648, 3677756)
[QUOTA WARNING] Usage inconsistent for ID 13340:actual (1749823488, 5132513) != expected (1749684224, 5132241)
[QUOTA WARNING] Usage inconsistent for ID 1058:actual (9920512, 37723) != expected (9920512, 37713)
[QUOTA WARNING] Usage inconsistent for ID 13101:actual (304844800, 1031030) != expected (307908608, 1059027)
[QUOTA WARNING] Usage inconsistent for ID 3074:actual (17772544, 213487) != expected (17752064, 215409)
[QUOTA WARNING] Usage inconsistent for ID 21455:actual (189779968, 216721) != expected (189005824, 234420)
[QUOTA WARNING] Usage inconsistent for ID 1940:actual (52523008, 325010) != expected (52183040, 321355)
[QUOTA WARNING] Usage inconsistent for ID 5571:actual (1149480960, 873249) != expected (1137983488, 824649)
[QUOTA WARNING] Usage inconsistent for ID 14425:actual (97591296, 734172) != expected (97591296, 734150)
[QUOTA WARNING] Usage inconsistent for ID 28625:actual (151408640, 204432) != expected (151339008, 203525)
[QUOTA WARNING] Usage inconsistent for ID 47559:actual (244539392, 234381) != expected (244539392, 234368)
[QUOTA WARNING] Usage inconsistent for ID 26896:actual (27328512, 214542) != expected (27324416, 214507)
[QUOTA WARNING] Usage inconsistent for ID 30155:actual (1759961088, 5182256) != expected (629850112, 2424847)
[QUOTA WARNING] Usage inconsistent for ID 41865:actual (162107392, 1206130) != expected (158162944, 1184231)
[QUOTA WARNING] Usage inconsistent for ID 32244:actual (194437120, 1814834) != expected (193912832, 1806816)
[QUOTA WARNING] Usage inconsistent for ID 29960:actual (27136000, 29948) != expected (27136000, 29924)
[QUOTA WARNING] Usage inconsistent for ID 50850:actual (218345472, 996620) != expected (218345472, 996619)
[QUOTA WARNING] Usage inconsistent for ID 45955:actual (579133440, 4448732) != expected (579379200, 4449891)
[QUOTA WARNING] Usage inconsistent for ID 8208:actual (74825728, 702056) != expected (35409920, 300309)
[QUOTA WARNING] Usage inconsistent for ID 28825:actual (190898176, 223552) != expected (190898176, 223547)
[QUOTA WARNING] Usage inconsistent for ID 37347:actual (69378048, 897135) != expected (65445888, 864621)
[QUOTA WARNING] Usage inconsistent for ID 15615:actual (22102016, 189712) != expected (22102016, 206743)
[QUOTA WARNING] Usage inconsistent for ID 19967:actual (33587200, 50554) != expected (33587200, 45546)
[QUOTA WARNING] Usage inconsistent for ID 27866:actual (33488896, 36116) != expected (33579008, 36374)
[QUOTA WARNING] Usage inconsistent for ID 46265:actual (1516093440, 1476545) != expected (1485680640, 1363903)
[QUOTA WARNING] Usage inconsistent for ID 1184:actual (1353412608, 1513915) != expected (1348395008, 1449379)
[QUOTA WARNING] Usage inconsistent for ID 8266:actual (8458240, 82179) != expected (8142848, 79195)
[QUOTA WARNING] Usage inconsistent for ID 38369:actual (4096, 20178) != expected (4096, 15903)
[QUOTA WARNING] Usage inconsistent for ID 38395:actual (107249664, 86034) != expected (58286080, 55384)
[QUOTA WARNING] Usage inconsistent for ID 1531:actual (49795072, 177363) != expected (15421440, 86341)
[QUOTA WARNING] Usage inconsistent for ID 11806:actual (79728640, 122957) != expected (18124800, 66136)
[QUOTA WARNING] Usage inconsistent for ID 20674:actual (189685760, 694166) != expected (126910464, 461027)
[QUOTA WARNING] Usage inconsistent for ID 1231:actual (111190016, 928883) != expected (110977024, 925513)
[QUOTA WARNING] Usage inconsistent for ID 48598:actual (564248576, 600723) != expected (385892352, 306481)
[QUOTA WARNING] Usage inconsistent for ID 12529:actual (252956672, 1078188) != expected (110682112, 692250)
[QUOTA WARNING] Usage inconsistent for ID 43875:actual (14741504, 109636) != expected (14372864, 116021)
[QUOTA WARNING] Usage inconsistent for ID 38931:actual (38006784, 39115) != expected (37130240, 37407)
[QUOTA WARNING] Usage inconsistent for ID 29383:actual (260120576, 516204) != expected (260112384, 516200)
[QUOTA WARNING] Usage inconsistent for ID 45601:actual (34902016, 324567) != expected (34729984, 322371)
[QUOTA WARNING] Usage inconsistent for ID 41806:actual (586682368, 1394204) != expected (510181376, 1322923)
[QUOTA WARNING] Usage inconsistent for ID 20941:actual (781094912, 2110156) != expected (965718016, 3958606)
[QUOTA WARNING] Usage inconsistent for ID 39898:actual (39157760, 57315) != expected (34295808, 50458)
[QUOTA WARNING] Usage inconsistent for ID 21900:actual (159744, 616) != expected (159744, 617)
[QUOTA WARNING] Usage inconsistent for ID 51132:actual (26103808, 22849) != expected (25972736, 24725)
[QUOTA WARNING] Usage inconsistent for ID 46391:actual (11833344, 12510) != expected (11489280, 12503)
[QUOTA WARNING] Usage inconsistent for ID 28420:actual (82071552, 254416) != expected (82071552, 254414)
[QUOTA WARNING] Usage inconsistent for ID 5792:actual (8183808, 102108) != expected (1884160, 20597)
[QUOTA WARNING] Usage inconsistent for ID 6175:actual (692224, 1261) != expected (180224, 767)
[QUOTA WARNING] Usage inconsistent for ID 14093:actual (482426880, 451376) != expected (482418688, 451240)
[QUOTA WARNING] Usage inconsistent for ID 46136:actual (57479168, 51448) != expected (56856576, 41380)
[QUOTA WARNING] Usage inconsistent for ID 6194:actual (882724864, 1604474) != expected (806420480, 1213485)
[QUOTA WARNING] Usage inconsistent for ID 18336:actual (514334720, 707806) != expected (504791040, 612218)
[QUOTA WARNING] Usage inconsistent for ID 31982:actual (139292672, 594390) != expected (139288576, 594390)
[QUOTA WARNING] Usage inconsistent for ID 46859:actual (666238976, 1306210) != expected (660647936, 1236733)
[QUOTA WARNING] Usage inconsistent for ID 47690:actual (51060736, 66167) != expected (54132736, 46303)
lfs3-MDT0000: Update quota info for quota type 1.
lfs3-MDT0000: 229159492/878247936 files (0.1% non-contiguous), 139178446/439123968 blocks
lfs-us-mds ~ # 
-------------- next part --------------
Sep 14 12:50:58 lfs-us-mds kernel: ------------[ cut here ]------------
Sep 14 12:50:58 lfs-us-mds kernel: kernel BUG at fs/jbd2/transaction.c:1033!
Sep 14 12:50:58 lfs-us-mds kernel: invalid opcode: 0000 [#1] SMP 
Sep 14 12:50:58 lfs-us-mds kernel: last sysfs file: /sys/module/ipv6/initstate
Sep 14 12:50:58 lfs-us-mds kernel: CPU 11 
Sep 14 12:50:58 lfs-us-mds kernel: Modules linked in: vfat fat usb_storage mpt3sas mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu osp(U) mdd(U) lfsck(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) lquota(U) ldiskfs(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ksocklnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) sha512_generic sha256_generic crc32c_intel libcfs(U) ipmi_devintf bonding 8021q garp stp llc ipv6 xt_comment iptable_filter ip_tables power_meter iTCO_wdt iTCO_vendor_support sg ixgbe dca mdio shpchp tg3 ptp pps_core dcdbas sb_edac edac_core lpc_ich mfd_core ext4 jbd2 mbcache sd_mod crc_t10dif wmi ahci megaraid_sas dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan]
Sep 14 12:50:58 lfs-us-mds kernel: 
Sep 14 12:50:58 lfs-us-mds kernel: Pid: 8334, comm: mdt_rdpg03_016 Not tainted 2.6.32-431.23.3.el6_lustre.x86_64 #1 Dell Inc. PowerEdge R720/0C4Y3R
Sep 14 12:50:58 lfs-us-mds kernel: RIP: 0010:[<ffffffffa00748ad>]  [<ffffffffa00748ad>] jbd2_journal_dirty_metadata+0x10d/0x150 [jbd2]
Sep 14 12:50:58 lfs-us-mds kernel: RSP: 0018:ffff88078c957670  EFLAGS: 00010246
Sep 14 12:50:58 lfs-us-mds kernel: RAX: ffff8807f5e0fdc0 RBX: ffff881014578b28 RCX: ffff8807ea4502d0
Sep 14 12:50:58 lfs-us-mds kernel: RDX: 0000000000000000 RSI: ffff8807ea4502d0 RDI: 0000000000000000
Sep 14 12:50:58 lfs-us-mds kernel: RBP: ffff88078c957690 R08: a010000000000000 R09: f7f7bcf7c138b402
Sep 14 12:50:58 lfs-us-mds kernel: R10: 0000000000000001 R11: 0000000000000000 R12: ffff88100b114588
Sep 14 12:50:58 lfs-us-mds kernel: R13: ffff8807ea4502d0 R14: ffff8808237d4800 R15: 0000000000000c00
Sep 14 12:50:58 lfs-us-mds kernel: FS:  0000000000000000(0000) GS:ffff88084c4a0000(0000) knlGS:0000000000000000
Sep 14 12:50:58 lfs-us-mds kernel: CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
Sep 14 12:50:58 lfs-us-mds kernel: CR2: 0000000001033468 CR3: 0000000001a85000 CR4: 00000000000407e0
Sep 14 12:50:58 lfs-us-mds kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Sep 14 12:50:58 lfs-us-mds kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Sep 14 12:50:58 lfs-us-mds kernel: Process mdt_rdpg03_016 (pid: 8334, threadinfo ffff88078c956000, task ffff88078b4f2ae0)
Sep 14 12:50:58 lfs-us-mds kernel: Stack:
Sep 14 12:50:58 lfs-us-mds kernel: ffff881014578b28 ffffffffa0a196f0 ffff8807ea4502d0 0000000000000000
Sep 14 12:50:58 lfs-us-mds kernel: <d> ffff88078c9576d0 ffffffffa09d700b ffff88078c9576c0 ffffffff8109af8f
Sep 14 12:50:58 lfs-us-mds kernel: <d> ffff88082482b980 ffff881014578b28 0000000000000400 ffff8807ea4502d0
Sep 14 12:50:58 lfs-us-mds kernel: Call Trace:
Sep 14 12:50:58 lfs-us-mds kernel: [<ffffffffa09d700b>] __ldiskfs_handle_dirty_metadata+0x7b/0x100 [ldiskfs]
Sep 14 12:50:58 lfs-us-mds kernel: [<ffffffff8109af8f>] ? wake_up_bit+0x2f/0x40
Sep 14 12:50:58 lfs-us-mds kernel: [<ffffffffa0a0c7c5>] ldiskfs_quota_write+0x165/0x210 [ldiskfs]
Sep 14 12:50:58 lfs-us-mds kernel: [<ffffffff811ef59e>] write_blk+0x2e/0x30
Sep 14 12:50:58 lfs-us-mds kernel: [<ffffffff811f03fc>] do_insert_tree+0x21c/0x3d0
Sep 14 12:50:58 lfs-us-mds kernel: [<ffffffff811f0465>] do_insert_tree+0x285/0x3d0
Sep 14 12:50:58 lfs-us-mds kernel: [<ffffffff811f0465>] do_insert_tree+0x285/0x3d0
Sep 14 12:50:58 lfs-us-mds kernel: [<ffffffff811f06a8>] qtree_write_dquot+0xf8/0x150
Sep 14 12:50:58 lfs-us-mds kernel: [<ffffffff811eedf0>] v2_write_dquot+0x30/0x40
Sep 14 12:50:58 lfs-us-mds kernel: [<ffffffff811eafa0>] dquot_acquire+0xc0/0x140
Sep 14 12:50:58 lfs-us-mds kernel: [<ffffffff811ef91e>] ? qtree_read_dquot+0x5e/0x200
Sep 14 12:50:58 lfs-us-mds kernel: [<ffffffffa0a0b956>] ldiskfs_acquire_dquot+0x66/0xb0 [ldiskfs]
Sep 14 12:50:58 lfs-us-mds kernel: [<ffffffff811ecf8c>] dqget+0x2ac/0x390
Sep 14 12:50:58 lfs-us-mds kernel: [<ffffffff811ed51b>] dquot_initialize+0x7b/0x240
Sep 14 12:50:58 lfs-us-mds kernel: [<ffffffffa0a0bbb3>] ldiskfs_dquot_initialize+0x83/0xd0 [ldiskfs]
Sep 14 12:50:58 lfs-us-mds kernel: [<ffffffffa0ad6baf>] osd_attr_set+0x12f/0x540 [osd_ldiskfs]
Sep 14 12:50:58 lfs-us-mds kernel: [<ffffffffa0cbe96b>] lod_attr_set+0x12b/0x450 [lod]
Sep 14 12:50:58 lfs-us-mds kernel: [<ffffffffa0d1e9b1>] mdd_attr_set_internal+0x151/0x230 [mdd]
Sep 14 12:50:58 lfs-us-mds kernel: [<ffffffffa0d2130a>] mdd_attr_set+0x117a/0x1470 [mdd]
Sep 14 12:50:58 lfs-us-mds kernel: [<ffffffffa0bfc69c>] mdt_mfd_close+0x7ac/0x1bc0 [mdt]
Sep 14 12:50:58 lfs-us-mds kernel: [<ffffffffa053de95>] ? lustre_msg_buf+0x55/0x60 [ptlrpc]
Sep 14 12:50:58 lfs-us-mds kernel: [<ffffffffa0564e56>] ? __req_capsule_get+0x166/0x710 [ptlrpc]
Sep 14 12:50:58 lfs-us-mds kernel: [<ffffffffa053eeae>] ? lustre_pack_reply_flags+0xae/0x1f0 [ptlrpc]
Sep 14 12:50:58 lfs-us-mds kernel: [<ffffffffa03910d5>] ? class_handle2object+0x95/0x190 [obdclass]
Sep 14 12:50:58 lfs-us-mds kernel: [<ffffffffa0bfee42>] mdt_close+0x642/0xa80 [mdt]
Sep 14 12:50:58 lfs-us-mds kernel: [<ffffffffa0bd3aba>] mdt_handle_common+0x52a/0x1470 [mdt]
Sep 14 12:50:58 lfs-us-mds kernel: [<ffffffffa0c10965>] mds_readpage_handle+0x15/0x20 [mdt]
Sep 14 12:50:58 lfs-us-mds kernel: [<ffffffffa054ecf5>] ptlrpc_server_handle_request+0x385/0xc00 [ptlrpc]
Sep 14 12:50:58 lfs-us-mds kernel: [<ffffffffa024a4ce>] ? cfs_timer_arm+0xe/0x10 [libcfs]
Sep 14 12:50:58 lfs-us-mds kernel: [<ffffffffa05463d9>] ? ptlrpc_wait_event+0xa9/0x2d0 [ptlrpc]
Sep 14 12:50:58 lfs-us-mds kernel: [<ffffffffa025b3cf>] ? lc_watchdog_touch+0x6f/0x170 [libcfs]
Sep 14 12:50:58 lfs-us-mds kernel: [<ffffffff810546b9>] ? __wake_up_common+0x59/0x90
Sep 14 12:50:58 lfs-us-mds kernel: [<ffffffffa055005d>] ptlrpc_main+0xaed/0x1740 [ptlrpc]
Sep 14 12:50:58 lfs-us-mds kernel: [<ffffffffa054f570>] ? ptlrpc_main+0x0/0x1740 [ptlrpc]
Sep 14 12:50:58 lfs-us-mds kernel: [<ffffffff8109abf6>] kthread+0x96/0xa0
Sep 14 12:50:58 lfs-us-mds kernel: [<ffffffff8100c20a>] child_rip+0xa/0x20
Sep 14 12:50:58 lfs-us-mds kernel: [<ffffffff8109ab60>] ? kthread+0x0/0xa0
Sep 14 12:50:58 lfs-us-mds kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Sep 14 12:50:58 lfs-us-mds kernel: Code: c6 9c 03 00 00 4c 89 f7 e8 c1 71 4b e1 48 8b 33 ba 01 00 00 00 4c 89 e7 e8 11 ec ff ff 4c 89 f0 66 ff 00 66 66 90 e9 73 ff ff ff <0f> 0b eb fe 0f 0b eb fe 0f 0b 66 0f 1f 84 00 00 00 00 00 eb f5 
Sep 14 12:50:58 lfs-us-mds kernel: RIP  [<ffffffffa00748ad>] jbd2_journal_dirty_metadata+0x10d/0x150 [jbd2]
Sep 14 12:50:58 lfs-us-mds kernel: RSP <ffff88078c957670>
Sep 14 12:50:58 lfs-us-mds kernel: ---[ end trace f6214844c1d51fe7 ]---


More information about the lustre-discuss mailing list