[Lustre-discuss] freezing
Papp Tamás
tompos at martos.bme.hu
Sun May 18 03:16:32 PDT 2008
Dear All,
Our small but hardly used cluster was running for weeks without any
problems, but yesterday 3 of 8 nodes froze up in 1 hour.
This is from messages log, there was many of this in it before this last
one:
May 17 22:00:31 node8 kernel: LustreError: dumping log to
/tmp/lustre-log.1211054431.7807
May 17 22:00:33 node8 kernel: LustreError:
7807:0:(service.c:668:ptlrpc_server_handle_request()) request 3062525
opc 4 from 12345-192.168.0.63 at tcp processed in 101s trans 2522291 rc 0/0
May 17 22:00:33 node8 kernel: Lustre:
7807:0:(watchdog.c:312:lcw_update_time()) Expired watchdog for pid 7807
disabled after 101.5718s
May 17 22:00:43 node8 kernel: Lustre:
7641:0:(ldlm_lib.c:519:target_handle_reconnect()) cubefs-OST0008:
9737b5de-8c67-99be-94ae-12f66fe80edf reconnecting
May 17 22:00:43 node8 kernel: Lustre:
7641:0:(ldlm_lib.c:519:target_handle_reconnect()) Skipped 1 previous
similar message
May 17 22:01:05 node8 kernel: Lustre:
7627:0:(ldlm_lib.c:747:target_handle_connect()) cubefs-OST0008: refuse
reconnection from 1a535b99-47d7-7f5f-f3cb-277793
0c9217 at 192.168.0.65@tcp to 0xffff810010759000; still busy with 2 active
RPCs
May 17 22:01:05 node8 kernel: LustreError:
7627:0:(ldlm_lib.c:1442:target_send_reply_msg()) @@@ processing error
(-16) req at ffff8100664aec00 x2179620/t0
o8->1a535b99-47d7-7f5f-f3cb-2777930c9217 at NET_0x20000c0a80041_UUID:-1
lens 304/200 ref 0 fl Interpret:/0/0 rc -16/0
May 17 22:01:05 node8 kernel: Lustre: 0:0:(watchdog.c:130:lcw_cb())
Watchdog triggered for pid 7738: it was inactive for 100s
May 17 22:01:05 node8 kernel: Lustre: 0:0:(watchdog.c:130:lcw_cb())
Skipped 1 previous similar message
May 17 22:01:05 node8 kernel: Lustre:
0:0:(linux-debug.c:168:libcfs_debug_dumpstack()) showing stack for
process 7738
May 17 22:01:05 node8 kernel: Lustre:
0:0:(linux-debug.c:168:libcfs_debug_dumpstack()) Skipped 1 previous
similar message
May 17 22:01:05 node8 kernel: ll_ost_io_13 D 000000000000712e 0
7738 1 7739 7737 (L-TLB)
May 17 22:01:05 node8 kernel: ffff8100655273f0 0000000000000046
ffff810003d18c58 ffffffff80012c28
May 17 22:01:05 node8 kernel: 000000000000000a ffff810065511820
ffff810063e1f0c0 0007d730dc4a35a5
May 17 22:01:05 node8 kernel: 0000000000000213 ffff810065511a08
ffff810000000001 ffff810063c0a7e0
May 17 22:01:05 node8 kernel: Call Trace:
May 17 22:01:05 node8 kernel: [<ffffffff80012c28>] get_request+0x1a6/0x36b
May 17 22:01:05 node8 kernel: [<ffffffff883354ac>] :drbd:lc_find+0x30/0x51
May 17 22:01:05 node8 kernel: [<ffffffff88334e23>]
:drbd:drbd_al_begin_io+0x160/0x24c
May 17 22:01:05 node8 kernel: [<ffffffff80022d47>] mempool_alloc+0x24/0xda
May 17 22:01:05 node8 kernel: [<ffffffff8009b3d0>]
autoremove_wake_function+0x0/0x2e
May 17 22:01:05 node8 kernel: [<ffffffff800d7ad5>] __bio_clone+0x71/0x8a
May 17 22:01:05 node8 kernel: [<ffffffff883320d9>]
:drbd:drbd_make_request_common+0x593/0x8c5
May 17 22:01:05 node8 kernel: [<ffffffff8001b998>]
generic_make_request+0x204/0x21b
May 17 22:01:05 node8 kernel: [<ffffffff80022d47>] mempool_alloc+0x24/0xda
May 17 22:01:05 node8 kernel: [<ffffffff80032b85>] submit_bio+0xcd/0xd4
May 17 22:01:05 node8 kernel: [<ffffffff883fa686>]
:obdclass:lprocfs_oh_tally+0x26/0x50
May 17 22:01:05 node8 kernel: [<ffffffff886ba31c>]
:fsfilt_ldiskfs:fsfilt_ldiskfs_send_bio+0xc/0x20
May 17 22:01:05 node8 kernel: [<ffffffff886ee45e>]
:obdfilter:filter_do_bio+0x53e/0xac0
May 17 22:01:05 node8 kernel: [<ffffffff8803374e>]
:jbd:journal_callback_set+0x2d/0x47
May 17 22:01:05 node8 kernel: [<ffffffff886ba630>]
:fsfilt_ldiskfs:fsfilt_ldiskfs_commit_async+0xd0/0x150
May 17 22:01:05 node8 kernel: [<ffffffff886ef694>]
:obdfilter:filter_direct_io+0xcb4/0xce0
May 17 22:01:05 node8 kernel: [<ffffffff886f1535>]
:obdfilter:filter_commitrw_write+0x1855/0x2570
May 17 22:01:05 node8 kernel: [<ffffffff88476eca>]
:ptlrpc:ldlm_resource_foreach+0x11a/0x390
May 17 22:01:05 node8 kernel: [<ffffffff8003ceb4>]
lock_timer_base+0x1b/0x3c
May 17 22:01:05 node8 kernel: [<ffffffff8869eb46>]
:ost:ost_brw_write+0x21b6/0x28c0
May 17 22:01:05 node8 kernel: [<ffffffff884942d0>]
:ptlrpc:ptlrpc_send_reply+0x370/0x380
May 17 22:01:05 node8 kernel: [<ffffffff88493db0>]
:ptlrpc:ptl_send_buf+0x4a0/0x650
May 17 22:01:05 node8 kernel: [<ffffffff80088436>]
default_wake_function+0x0/0xe
May 17 22:01:05 node8 kernel: [<ffffffff886a283e>]
:ost:ost_handle+0x2a8e/0x58d8
May 17 22:01:05 node8 kernel: [<ffffffff800d1003>]
kmem_freepages+0xe6/0x110
May 17 22:01:05 node8 kernel: [<ffffffff88496c58>]
:ptlrpc:lustre_unpack_msg_v1+0x118/0x520
May 17 22:01:05 node8 kernel: [<ffffffff88400cc2>]
:obdclass:class_handle2object+0xd2/0x160
May 17 22:01:05 node8 kernel: [<ffffffff8849c220>]
:ptlrpc:lustre_swab_ptlrpc_body+0x0/0x90
May 17 22:01:05 node8 kernel: [<ffffffff88499de5>]
:ptlrpc:lustre_swab_buf+0xc5/0xf0
May 17 22:01:05 node8 kernel: [<ffffffff884a1a2b>]
:ptlrpc:ptlrpc_server_handle_request+0xb0b/0x1270
May 17 22:01:05 node8 kernel: [<ffffffff80060f29>] thread_return+0x0/0xeb
May 17 22:01:05 node8 kernel: [<ffffffff8006b6c9>]
do_gettimeofday+0x50/0x92
May 17 22:01:05 node8 kernel: [<ffffffff8835a066>]
:libcfs:lcw_update_time+0x16/0x100
May 17 22:01:05 node8 kernel: [<ffffffff8003ceb4>]
lock_timer_base+0x1b/0x3c
May 17 22:01:05 node8 kernel: [<ffffffff884a446c>]
:ptlrpc:ptlrpc_main+0x7dc/0x950
May 17 22:01:05 node8 kernel: [<ffffffff80088436>]
default_wake_function+0x0/0xe
May 17 22:01:05 node8 kernel: [<ffffffff8005bfb1>] child_rip+0xa/0x11
May 17 22:01:05 node8 kernel: [<ffffffff884a3c90>]
:ptlrpc:ptlrpc_main+0x0/0x950
May 17 22:01:05 node8 kernel: [<ffffffff8005bfa7>] child_rip+0x0/0x11
May 17 22:01:05 node8 kernel:
May 17 22:01:05 node8 kernel: LustreError: dumping log to
/tmp/lustre-log.1211054465.7738
May 17 22:01:08 node8 kernel: LustreError:
7738:0:(filter_io_26.c:763:filter_commitrw_write()) cubefs-OST0008: slow
direct_io 102s
May 17 22:01:08 node8 kernel: LustreError:
7738:0:(filter_io_26.c:763:filter_commitrw_write()) Skipped 24 previous
similar messages
May 17 22:01:08 node8 kernel: LustreError:
7738:0:(filter_io_26.c:776:filter_commitrw_write()) cubefs-OST0008: slow
commitrw commit 102s
May 17 22:01:08 node8 kernel: LustreError:
7738:0:(filter_io_26.c:776:filter_commitrw_write()) Skipped 24 previous
similar messages
May 17 22:01:08 node8 kernel: LustreError:
7738:0:(service.c:668:ptlrpc_server_handle_request()) request 2179029
opc 4 from 12345-192.168.0.65 at tcp processed in 102s trans 2522315 rc 0/0
May 17 22:01:08 node8 kernel: LustreError:
7738:0:(service.c:668:ptlrpc_server_handle_request()) Skipped 2 previous
similar messages
May 17 22:01:08 node8 kernel: Lustre:
7738:0:(watchdog.c:312:lcw_update_time()) Expired watchdog for pid 7738
disabled after 102.7567s
May 17 22:01:08 node8 kernel: Lustre:
7738:0:(watchdog.c:312:lcw_update_time()) Skipped 2 previous similar
messages
May 17 22:01:30 node8 kernel: Lustre:
7647:0:(ldlm_lib.c:519:target_handle_reconnect()) cubefs-OST0008:
1a535b99-47d7-7f5f-f3cb-2777930c9217 reconnecting
May 17 22:01:30 node8 kernel: Lustre:
7647:0:(ldlm_lib.c:519:target_handle_reconnect()) Skipped 2 previous
similar messages
May 18 10:10:17 node8 syslogd 1.4.1: restart.
I also attach the log dump and if it neccessary I can send more logs.
What could be the problem? Lustre, linux or hw?
The other node was node1, but it has no messages in logs like this, but
before I restarted it, I saw the same on it's screen.
In tme the first one was node3. It has in message log this:
May 17 19:57:55 node3 kernel: Lustre: cubefs-OST0003: haven't heard from
client d955fc8d-934e-2fef-9614-cd5b90f10298 (at 192.168.0.236 at tcp) in
227 seconds. I think it's dead, and I am evicting it.
May 17 19:57:55 node3 kernel: Lustre: Skipped 1 previous similar message
May 17 19:59:33 node3 kernel: LustreError:
7650:0:(client.c:975:ptlrpc_expire_one_request()) @@@ timeout (sent at
1211047153, 20s ago) req at ffff810005dea800
x454190/t0 o104->@NET_0x20000c0a80089_UUID:15 lens 232/128 ref 1 fl
Rpc:N/0/0 rc 0/-22
May 17 19:59:33 node3 kernel: LustreError: 138-a: cubefs-OST0003: A
client on nid 192.168.0.137 at tcp was evicted due to a lock blocking
callback to 192.168.0.137 at tcp timed out: rc -107
May 17 20:03:08 node3 kernel: Lustre:
7109:0:(filter_io_26.c:763:filter_commitrw_write()) cubefs-OST0003: slow
direct_io 31s
May 17 20:03:08 node3 kernel: Lustre:
7109:0:(filter_io_26.c:763:filter_commitrw_write()) Skipped 78 previous
similar messages
May 17 20:03:08 node3 kernel: Lustre:
7109:0:(filter_io_26.c:776:filter_commitrw_write()) cubefs-OST0003: slow
commitrw commit 31s
May 17 20:03:08 node3 kernel: Lustre:
7109:0:(filter_io_26.c:776:filter_commitrw_write()) Skipped 79 previous
similar messages
May 17 20:03:11 node3 kernel: Lustre:
7722:0:(filter_io_26.c:763:filter_commitrw_write()) cubefs-OST0003: slow
direct_io 31s
May 17 20:03:11 node3 kernel: Lustre:
7722:0:(filter_io_26.c:776:filter_commitrw_write()) cubefs-OST0003: slow
commitrw commit 31s
May 17 20:03:17 node3 kernel: Lustre:
7749:0:(filter_io_26.c:763:filter_commitrw_write()) cubefs-OST0003: slow
direct_io 31s
May 17 20:03:17 node3 kernel: Lustre:
7749:0:(filter_io_26.c:776:filter_commitrw_write()) cubefs-OST0003: slow
commitrw commit 31s
May 17 20:03:29 node3 kernel: Lustre:
23971:0:(filter_io_26.c:763:filter_commitrw_write()) cubefs-OST0003:
slow direct_io 48s
May 17 20:03:29 node3 kernel: Lustre:
23971:0:(filter_io_26.c:776:filter_commitrw_write()) cubefs-OST0003:
slow commitrw commit 48s
May 17 20:03:29 node3 kernel: LustreError:
7690:0:(filter_io_26.c:763:filter_commitrw_write()) cubefs-OST0003: slow
direct_io 50s
May 17 20:03:29 node3 kernel: LustreError:
7690:0:(filter_io_26.c:776:filter_commitrw_write()) cubefs-OST0003: slow
commitrw commit 50s
May 17 20:03:38 node3 kernel: LustreError:
7117:0:(filter_io_26.c:763:filter_commitrw_write()) cubefs-OST0003: slow
direct_io 53s
May 17 20:03:38 node3 kernel: LustreError:
7117:0:(filter_io_26.c:776:filter_commitrw_write()) cubefs-OST0003: slow
commitrw commit 53s
May 17 20:03:49 node3 kernel: LustreError:
12335:0:(filter_io_26.c:763:filter_commitrw_write()) cubefs-OST0003:
slow direct_io 52s
May 17 20:03:49 node3 kernel: LustreError:
12335:0:(filter_io_26.c:776:filter_commitrw_write()) cubefs-OST0003:
slow commitrw commit 52s
May 17 20:03:55 node3 kernel: LustreError:
7727:0:(filter_io_26.c:763:filter_commitrw_write()) cubefs-OST0003: slow
direct_io 53s
May 17 20:03:55 node3 kernel: LustreError:
7727:0:(filter_io_26.c:776:filter_commitrw_write()) cubefs-OST0003: slow
commitrw commit 53s
May 17 20:03:57 node3 kernel: Lustre:
7735:0:(filter_io_26.c:763:filter_commitrw_write()) cubefs-OST0003: slow
direct_io 41s
May 17 20:03:57 node3 kernel: Lustre:
7735:0:(filter_io_26.c:776:filter_commitrw_write()) cubefs-OST0003: slow
commitrw commit 41s
May 17 20:04:00 node3 kernel: LustreError:
29950:0:(filter_io_26.c:763:filter_commitrw_write()) cubefs-OST0003:
slow direct_io 60s
May 17 20:04:00 node3 kernel: LustreError:
29950:0:(filter_io_26.c:776:filter_commitrw_write()) cubefs-OST0003:
slow commitrw commit 60s
May 17 20:04:06 node3 kernel: LustreError:
7169:0:(filter_io_26.c:763:filter_commitrw_write()) cubefs-OST0003: slow
direct_io 66s
May 17 20:04:06 node3 kernel: LustreError:
7169:0:(filter_io_26.c:763:filter_commitrw_write()) Skipped 1 previous
similar message
May 17 20:04:06 node3 kernel: LustreError:
7169:0:(filter_io_26.c:776:filter_commitrw_write()) cubefs-OST0003: slow
commitrw commit 66s
May 17 20:04:06 node3 kernel: LustreError:
7169:0:(filter_io_26.c:776:filter_commitrw_write()) Skipped 1 previous
similar message
May 17 20:04:15 node3 kernel: LustreError:
7124:0:(filter_io_26.c:763:filter_commitrw_write()) cubefs-OST0003: slow
direct_io 67s
May 17 20:04:15 node3 kernel: LustreError:
7124:0:(filter_io_26.c:763:filter_commitrw_write()) Skipped 2 previous
similar messages
May 17 20:04:15 node3 kernel: LustreError:
7124:0:(filter_io_26.c:776:filter_commitrw_write()) cubefs-OST0003: slow
commitrw commit 67s
May 17 20:04:15 node3 kernel: LustreError:
7124:0:(filter_io_26.c:776:filter_commitrw_write()) Skipped 2 previous
similar messages
May 17 20:46:50 node3 syslogd 1.4.1: restart.
Servers are Intel mainboards with Adaptec ASR-2820SA .
Linux node3 2.6.18-53.1.13.el5_lustre.1.6.4.3custom #2 SMP Tue Mar 11
12:57:23 CET 2008 x86_64 x86_64 x86_64 GNU/Linux
And we use drbd 0.7.25 .
Please give me some help.
Thank you,
tamas
ps.: I' don't know, it is relevant or not, but after this whole thing I
restarted the whole cluster and node2 was cannot be mounted. it said,
"File exists"
This is the log for it:
May 18 11:16:29 node2 kernel: Lustre: OBD class driver, info at clusterfs.com
May 18 11:16:29 node2 kernel: Lustre Version: 1.6.4.3
May 18 11:16:29 node2 kernel: Build Version:
1.6.4.3-19700101010000-PRISTINE-.usr.src.linux-2.6.18-53.1.13.el5_lustre.1.6.4.3-2.6.18-53.1.13.el5_lustre.1.6.4.3-cube1
May 18 11:16:29 node2 kernel: Lustre: Added LNI 10.1.1.12 at tcp [8/256]
May 18 11:16:29 node2 kernel: Lustre: Accept secure, port 988
May 18 11:16:29 node2 kernel: Lustre: Lustre Client File System;
info at clusterfs.com
May 18 11:16:30 node2 kernel: kjournald starting. Commit interval 5 seconds
May 18 11:16:30 node2 kernel: LDISKFS FS on drbd1, internal journal
May 18 11:16:30 node2 kernel: LDISKFS-fs: mounted filesystem with
ordered data mode.
May 18 11:16:30 node2 kernel: kjournald starting. Commit interval 5 seconds
May 18 11:16:30 node2 kernel: LDISKFS FS on drbd1, internal journal
May 18 11:16:30 node2 kernel: LDISKFS-fs: mounted filesystem with
ordered data mode.
May 18 11:16:30 node2 kernel: LDISKFS-fs: file extents enabled
May 18 11:16:30 node2 kernel: LDISKFS-fs: mballoc enabled
May 18 11:16:30 node2 kernel: LustreError:
3952:0:(socklnd_cb.c:2167:ksocknal_recv_hello()) Error -104 reading
HELLO from 10.1.1.1
May 18 11:16:30 node2 kernel: LustreError: 11b-b: Connection to
10.1.1.1 at tcp at host 10.1.1.1 on port 988 was reset: is it running a
compatible version of Lustre and is 10.1.1.1 at tcp one of its NIDs?
May 18 11:16:33 node2 kernel: LustreError: 137-5: UUID
'cubefs-OST0002_UUID' is not available for connect (no target)
May 18 11:16:33 node2 kernel: LustreError:
4023:0:(ldlm_lib.c:1442:target_send_reply_msg()) @@@ processing error
(-19) req at ffff8100668bc000 x79617/t0 o8-><?>@<?>:-1 lens 304/0 ref 0 fl
Interpret:/0/0 rc -19/0
May 18 11:16:35 node2 kernel: LustreError:
3956:0:(client.c:975:ptlrpc_expire_one_request()) @@@ timeout (sent at
1211102190, 5s ago) req at ffff81006aa9f000 x1/t0
o250->MGS at MGC10.1.1.1@tcp_0:26 lens 240/272 ref 1 fl Rpc:/0/0 rc 0/-22
May 18 11:16:35 node2 kernel: Lustre: Changing connection for
MGC10.1.1.1 at tcp to MGC10.1.1.1 at tcp_1/10.1.1.2 at tcp
May 18 11:16:35 node2 kernel: Lustre: Binding irq 169 to CPU 0 with cmd:
echo 1 > /proc/irq/169/smp_affinity
May 18 11:16:35 node2 kernel: Lustre: Filtering OBD driver;
info at clusterfs.com
May 18 11:16:37 node2 kernel: LustreError:
3916:0:(recov_thread.c:473:llog_start_commit_thread()) error starting
thread #1: -513
May 18 11:16:37 node2 kernel: LustreError:
3916:0:(llog_obd.c:392:llog_cat_initialize()) rc: -513
May 18 11:16:37 node2 kernel: LustreError:
3916:0:(filter.c:1717:filter_common_setup()) failed to setup llogging
subsystems
May 18 11:16:37 node2 kernel: LustreError:
3916:0:(obd_config.c:325:class_setup()) setup cubefs-OST0002 failed (-513)
May 18 11:16:37 node2 kernel: LustreError:
3916:0:(obd_config.c:1062:class_config_llog_handler()) Err -513 on cfg
command:
May 18 11:16:37 node2 kernel: Lustre: cmd=cf003 0:cubefs-OST0002
1:dev 2:type 3:f
May 18 11:16:37 node2 kernel: LustreError: 15c-8: MGC10.1.1.1 at tcp: The
configuration from log 'cubefs-OST0002' failed (-513). This may be the
result of communication errors between this node and the MGS, a bad
configuration, or other errors. See the syslog for more information.
May 18 11:16:37 node2 kernel: LustreError:
3916:0:(obd_mount.c:1080:server_start_targets()) failed to start server
cubefs-OST0002: -513
May 18 11:16:37 node2 kernel: LustreError:
3916:0:(obd_mount.c:1570:server_fill_super()) Unable to start targets: -513
May 18 11:16:37 node2 kernel: LustreError:
3916:0:(obd_config.c:392:class_cleanup()) Device 2 not setup
May 18 11:16:37 node2 kernel: LDISKFS-fs: mballoc: 1 blocks 1 reqs (0
success)
May 18 11:16:37 node2 kernel: LDISKFS-fs: mballoc: 1 extents scanned, 0
goal hits, 1 2^N hits, 0 breaks, 0 lost
May 18 11:16:37 node2 kernel: LDISKFS-fs: mballoc: 2 generated and it
took 107912
May 18 11:16:37 node2 kernel: LDISKFS-fs: mballoc: 256 preallocated, 0
discarded
May 18 11:16:37 node2 kernel: Lustre: server umount cubefs-OST0002 complete
May 18 11:16:37 node2 kernel: LustreError:
3916:0:(obd_mount.c:1924:lustre_fill_super()) Unable to mount (-513)
May 18 11:16:38 node2 kernel: kjournald starting. Commit interval 5 seconds
May 18 11:16:38 node2 kernel: LDISKFS FS on drbd1, internal journal
May 18 11:16:38 node2 kernel: LDISKFS-fs: mounted filesystem with
ordered data mode.
May 18 11:16:38 node2 kernel: LustreError: 137-5: UUID
'cubefs-OST0002_UUID' is not available for connect (not set up)
May 18 11:16:38 node2 kernel: LustreError:
4024:0:(ldlm_lib.c:1442:target_send_reply_msg()) @@@ processing error
(-19) req at ffff81006abf6c00 x85843/t0 o8-><?>@<?>:-1 lens 304/0 ref 0 fl
Interpret:/0/0 rc -19/0
I restarted the node, and I could mount it. At this moment it's working,
but I see this in logs again:
May 18 12:05:13 node2 kernel: Lustre:
3507:0:(filter_io_26.c:698:filter_commitrw_write()) cubefs-OST0002: slow
i_mutex 46s
May 18 12:05:13 node2 kernel: Lustre:
3507:0:(filter_io_26.c:711:filter_commitrw_write()) cubefs-OST0002: slow
brw_start 46s
May 18 12:05:13 node2 kernel: Lustre:
3478:0:(lustre_fsfilt.h:240:fsfilt_brw_start_log()) cubefs-OST0002: slow
journal start 39s
May 18 12:05:13 node2 kernel: Lustre:
3483:0:(lustre_fsfilt.h:205:fsfilt_start_log()) cubefs-OST0002: slow
journal start 41s
May 18 12:05:13 node2 kernel: Lustre:
3507:0:(filter_io_26.c:711:filter_commitrw_write()) Skipped 1 previous
similar message
May 18 12:05:13 node2 kernel: Lustre:
3515:0:(filter_io_26.c:698:filter_commitrw_write()) cubefs-OST0002: slow
i_mutex 46s
May 18 12:05:13 node2 kernel: Lustre:
3515:0:(filter_io_26.c:711:filter_commitrw_write()) cubefs-OST0002: slow
brw_start 46s
May 18 12:05:13 node2 kernel: Lustre:
3506:0:(lustre_fsfilt.h:296:fsfilt_commit_wait()) cubefs-OST0002: slow
journal start 46s
May 18 12:05:13 node2 kernel: Lustre:
3506:0:(filter_io_26.c:776:filter_commitrw_write()) cubefs-OST0002: slow
commitrw commit 46s
May 18 12:05:13 node2 kernel: Lustre:
3478:0:(filter_io_26.c:763:filter_commitrw_write()) cubefs-OST0002: slow
direct_io 39s
May 18 12:05:13 node2 kernel: Lustre:
3478:0:(filter_io_26.c:776:filter_commitrw_write()) cubefs-OST0002: slow
commitrw commit 39s
May 18 12:05:13 node2 kernel: Lustre:
3478:0:(filter_io_26.c:776:filter_commitrw_write()) Skipped 1 previous
similar message
May 18 12:05:13 node2 kernel: Lustre:
3515:0:(filter_io_26.c:763:filter_commitrw_write()) cubefs-OST0002: slow
direct_io 47s
May 18 12:05:13 node2 kernel: Lustre:
3515:0:(filter_io_26.c:763:filter_commitrw_write()) Skipped 1 previous
similar message
More information about the lustre-discuss
mailing list