Hi Michael,<br><br>This looks like the problem we had some time ago after upgrading to 1.8.3<br><br><a href="https://bugzilla.lustre.org/show_bug.cgi?id=22889">https://bugzilla.lustre.org/show_bug.cgi?id=22889</a><br><br>Best regards<br>
<br>Wojciech<br><div class="gmail_quote">On 20 July 2010 00:00, Michael Sternberg <span dir="ltr"><<a href="mailto:sternberg@anl.gov">sternberg@anl.gov</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">
Hello,<br>
<br>
I use OSSs with external journal partitions and since lustre-1.8.1 about one to two times a week I get frustrating panics on OSSs as follows:<br>
<br>
:libcfs:cfs_alloc ...<br>
:lvfs:lprocfs_counter_add ...<br>
...<br>
<br>
RIP [<ffffffff88031e64>] :jbd:journal_dirty_metadata+0x7f/0x1e3<br>
RSP <ffff8101f99c3670><br>
<0>Kernel panic - not syncing: Fatal exception<br>
<br>
(full graphical screenshot attached, hoping it passes through)<br>
<br>
Clients sometimes report:<br>
<br>
Message from syslogd@ at Mon Jul 19 04:11:46 2010 ...<br>
login4 kernel: journal commit I/O error<br>
<br>
<br>
I have recently updated to 1.8.3, where I e2fsck'd and re-initialized the external journals, but still get those panics. I use 2 OSS with heartbeat failover, each one "owns" and normally serves 2 OSTs (4 OST total), coming from 4 LUNs on a RAID unit with dual controllers. All OSTs use ldiskfs (pre-ext4 proper, if I understand correctly) with the journals located on partitions of 2 further LUNs. I use a variant of the script at bug 20807 to account for different device numbers of the external journals on the two OSSs. Failover usually works, eventually, after a load peak of up to 100 on the OSS taking over, and messages about hung threads (see below).<br>
<br>
<br>
Is there anything I could do besides giving up on external journals? My data stores are RAID1, and the journal disks are a single pair of disks also in RAID1.<br>
<br>
I had difficulties locating further information on googling "journal_dirty_metadata" pertaining to lustre/ldiskfs specifically. There are old discussions at:<br>
<br>
<a href="https://bugzilla.redhat.com/show_bug.cgi?id=183119" target="_blank">https://bugzilla.redhat.com/show_bug.cgi?id=183119</a> 2007/2008 - kernel 2.4.7<br>
<a href="http://oss.oracle.com/pipermail/ocfs2-users/2010-January/004113.html" target="_blank">http://oss.oracle.com/pipermail/ocfs2-users/2010-January/004113.html</a> (ahem)<br>
<br>
<br>
With best regards,<br>
Michael<br>
<br>
<br>
[root@mds01 ~]# cat /proc/fs/lustre/version<br>
lustre: 1.8.3<br>
kernel: patchless_client<br>
build: 1.8.3-20100409182943-PRISTINE-2.6.18-164.11.1.el5_lustre.1.8.3<br>
<br>
[root@mds01 ~]# uname -r<br>
2.6.18-164.11.1.el5_lustre.1.8.3<br>
<br>
[root@mds01 ~]# lctl dl<br>
0 UP mgs MGS MGS 725<br>
1 UP mgc MGC172.17.120.1@o2ib 9642cdcd-4955-ca05-4e85-8a9f6d10c027 5<br>
2 UP mdt MDS MDS_uuid 3<br>
7 UP lov sandbox-mdtlov sandbox-mdtlov_UUID 4<br>
8 UP mds sandbox-MDT0000 sandbox-MDT0000_UUID 719<br>
9 UP osc sandbox-OST0000-osc sandbox-mdtlov_UUID 5<br>
10 UP osc sandbox-OST0001-osc sandbox-mdtlov_UUID 5<br>
<br>
[root@mds01 ~]# ssh mds02 lctl dl<br>
0 UP mgc MGC172.17.120.1@o2ib 12c07f8c-f1e7-f739-9983-3c3aa2ec492a 5<br>
1 UP mdt MDS MDS_uuid 3<br>
2 UP lov carbonfs-mdtlov carbonfs-mdtlov_UUID 4<br>
3 UP mds carbonfs-MDT0000 carbonfs-MDT0000_UUID 719<br>
4 UP osc carbonfs-OST0001-osc carbonfs-mdtlov_UUID 5<br>
5 UP osc carbonfs-OST0000-osc carbonfs-mdtlov_UUID 5<br>
<br>
[root@oss01 ~]# lctl dl<br>
0 UP mgc MGC172.17.120.1@o2ib a89ba7f9-2a12-ff77-0321-151a1addf043 5<br>
1 UP ost OSS OSS_uuid 3<br>
2 UP obdfilter sandbox-OST0000 sandbox-OST0000_UUID 721<br>
3 UP obdfilter carbonfs-OST0000 carbonfs-OST0000_UUID 721<br>
4 UP obdfilter sandbox-OST0001 sandbox-OST0001_UUID 721<br>
5 UP obdfilter carbonfs-OST0001 carbonfs-OST0001_UUID 721<br>
<br>
[client]# lctl dl<br>
0 UP mgc MGC172.17.120.1@o2ib dadd88bf-fbad-d933-b02a-a539fd8abfea 5<br>
1 UP lov sandbox-clilov-ffff8101da93c400 30094b3a-b246-e667-ef8a-f6690e4d051c 4<br>
2 UP mdc sandbox-MDT0000-mdc-ffff8101da93c400 30094b3a-b246-e667-ef8a-f6690e4d051c 5<br>
3 UP osc sandbox-OST0000-osc-ffff8101da93c400 30094b3a-b246-e667-ef8a-f6690e4d051c 5<br>
4 UP osc sandbox-OST0001-osc-ffff8101da93c400 30094b3a-b246-e667-ef8a-f6690e4d051c 5<br>
5 UP lov carbonfs-clilov-ffff81041dc0f800 4aaa2ddd-eee8-564f-ea48-b9c36a428eb9 4<br>
6 UP mdc carbonfs-MDT0000-mdc-ffff81041dc0f800 4aaa2ddd-eee8-564f-ea48-b9c36a428eb9 5<br>
7 UP osc carbonfs-OST0001-osc-ffff81041dc0f800 4aaa2ddd-eee8-564f-ea48-b9c36a428eb9 5<br>
8 UP osc carbonfs-OST0000-osc-ffff81041dc0f800 4aaa2ddd-eee8-564f-ea48-b9c36a428eb9 5<br>
<br>
<br>
--------------------------------------------------<br>
"thread hung" messages:<br>
--------------------------------------------------<br>
Jul 19 04:01:05 oss01 kernel: Lustre: carbonfs-OST0001: Recovery period over after 1:05, of 359 clients 358 recovered and 0 were evicted.<br>
Jul 19 04:01:05 oss01 kernel: Lustre: carbonfs-OST0001: sending delayed replies to recovered clientsJul 19 04:01:05 oss01 kernel: Lustre: carbonfs-OST0001: received MDS connection from 172.17.120.2@o2ib<br>
Jul 19 04:03:36 oss01 kernel: LustreError: 0:0:(ldlm_lockd.c:305:waiting_locks_callback()) ### lock callback timer expired after 151s: evicting client at 172.17.0.235@o2ib ns: filter-carbonfs-OST0001_UUID loc<br>
k: ffff810178b8d600/0x80a4d28c4aff67ec lrc: 3/0,0 mode: PW/PW res: 152472/0 rrc: 5 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x10020 remote: 0x4701b34288b619f7 expref: 12 pid: 68<br>
06 timeout 4356934523<br>
Jul 19 04:04:16 oss01 kernel: Lustre: 6757:0:(ldlm_lib.c:804:target_handle_connect()) carbonfs-OST0001: exp ffff8101ecf94e00 already connectingJul 19 04:04:16 oss01 kernel: Lustre: 6757:0:(ldlm_lib.c:804:target_handle_connect()) Skipped 38 previous similar messages<br>
Jul 19 04:04:25 oss01 kernel: Lustre: Service thread pid 6145 was inactive for 200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:<br>
Jul 19 04:04:25 oss01 kernel: Lustre: Skipped 2 previous similar messages<br>
Jul 19 04:04:25 oss01 kernel: Lustre: 0:0:(linux-debug.c:264:libcfs_debug_dumpstack()) showing stack for process 6145<br>
Jul 19 04:04:25 oss01 kernel: Lustre: 0:0:(linux-debug.c:264:libcfs_debug_dumpstack()) Skipped 2 previous similar messagesJul 19 04:04:25 oss01 kernel: ll_ost_io_38 D 0000000000000000 0 6145 1 6146 6144 (L-TLB)<br>
Jul 19 04:04:25 oss01 kernel: ffff810214dcb730 0000000000000046 ffff81021158b220 ffff81017fe44cc0<br>
Jul 19 04:04:25 oss01 kernel: ffff81017fe44cc0 000000000000000a ffff810214d4a7e0 ffff810214c0f7e0<br>
Jul 19 04:04:25 oss01 kernel: 000038708ddb5ba3 0000000000001a95 ffff810214d4a9c8 000000042fe56000<br>
Jul 19 04:04:25 oss01 kernel: Call Trace:Jul 19 04:04:25 oss01 kernel: [<ffffffff88036769>] :jbd:log_wait_commit+0xa3/0xf5<br>
Jul 19 04:04:25 oss01 kernel: [<ffffffff800a00be>] autoremove_wake_function+0x0/0x2e<br>
Jul 19 04:04:25 oss01 kernel: [<ffffffff88cc900b>] :fsfilt_ldiskfs:fsfilt_ldiskfs_commit_wait+0xab/0xd0<br>
Jul 19 04:04:25 oss01 kernel: [<ffffffff88d0798f>] :obdfilter:filter_commitrw_write+0x19ef/0x2980<br>
Jul 19 04:04:25 oss01 kernel: [<ffffffff88ca4ed5>] :ost:ost_checksum_bulk+0x385/0x5b0Jul 19 04:04:26 oss01 kernel: [<ffffffff88ca4c38>] :ost:ost_checksum_bulk+0xe8/0x5b0<br>
Jul 19 04:04:26 oss01 kernel: [<ffffffff88cab8da>] :ost:ost_brw_write+0x1c1a/0x23b0<br>
Jul 19 04:04:26 oss01 kernel: [<ffffffff889ec238>] :ptlrpc:ptlrpc_send_reply+0x5c8/0x5e0<br>
Jul 19 04:04:26 oss01 kernel: [<ffffffff889b7b10>] :ptlrpc:target_committed_to_req+0x40/0x120<br>
Jul 19 04:04:26 oss01 kernel: [<ffffffff88ca781d>] :ost:ost_brw_read+0x189d/0x1a70Jul 19 04:04:26 oss01 kernel: [<ffffffff889f06c5>] :ptlrpc:lustre_msg_get_version+0x35/0xf0<br>
Jul 19 04:04:26 oss01 kernel: [<ffffffff8008c86b>] default_wake_function+0x0/0xe<br>
Jul 19 04:04:26 oss01 kernel: [<ffffffff889f0788>] :ptlrpc:lustre_msg_check_version_v2+0x8/0x20<br>
Jul 19 04:04:26 oss01 kernel: [<ffffffff88caec1e>] :ost:ost_handle+0x2bae/0x53d0<br>
Jul 19 04:04:26 oss01 kernel: [<ffffffff889cde89>] :ptlrpc:__ldlm_add_waiting_lock+0x189/0x1c0Jul 19 04:04:26 oss01 kernel: [<ffffffff889cf10e>] :ptlrpc:ldlm_refresh_waiting_lock+0xbe/0x110<br>
Jul 19 04:04:26 oss01 kernel: [<ffffffff88ca2571>] :ost:ost_prolong_locks_iter+0x151/0x180<br>
Jul 19 04:04:26 oss01 kernel: [<ffffffff889fd9b5>] :ptlrpc:ptlrpc_server_handle_request+0xaa5/0x1140<br>
Jul 19 04:04:26 oss01 kernel: [<ffffffff80062ff8>] thread_return+0x62/0xfe<br>
Jul 19 04:04:26 oss01 kernel: [<ffffffff8003dafe>] lock_timer_base+0x1b/0x3cJul 19 04:04:26 oss01 kernel: [<ffffffff8001ca74>] __mod_timer+0xb0/0xbe<br>
Jul 19 04:04:26 oss01 kernel: [<ffffffff88a01408>] :ptlrpc:ptlrpc_main+0x1258/0x1420<br>
Jul 19 04:04:26 oss01 kernel: [<ffffffff8008c86b>] default_wake_function+0x0/0xe<br>
Jul 19 04:04:26 oss01 kernel: [<ffffffff800b7076>] audit_syscall_exit+0x336/0x362<br>
Jul 19 04:04:26 oss01 kernel: [<ffffffff8005dfb1>] child_rip+0xa/0x11Jul 19 04:04:26 oss01 kernel: [<ffffffff88a001b0>] :ptlrpc:ptlrpc_main+0x0/0x1420<br>
Jul 19 04:04:26 oss01 kernel: [<ffffffff8005dfa7>] child_rip+0x0/0x11<br>
Jul 19 04:04:26 oss01 kernel:<br>
--------------------------------------------------<br>
<br>
<br>_______________________________________________<br>
Lustre-discuss mailing list<br>
<a href="mailto:Lustre-discuss@lists.lustre.org">Lustre-discuss@lists.lustre.org</a><br>
<a href="http://lists.lustre.org/mailman/listinfo/lustre-discuss" target="_blank">http://lists.lustre.org/mailman/listinfo/lustre-discuss</a><br>
<br></blockquote></div><br><br clear="all"><br>-- <br>--<br>Wojciech Turek<br><br>Assistant System Manager<br><br>High Performance Computing Service<br>University of Cambridge<br>Email: <a href="mailto:wjt27@cam.ac.uk">wjt27@cam.ac.uk</a><br>
Tel: (+)44 1223 763517 <br>