[Lustre-discuss] Lustre Issue
Nauman Yousuf
nauman.yousuf at gmail.com
Wed Jan 26 09:24:33 PST 2011
Dear All
I am getting these errors in my dmesg file. My lustre client which is my
mail storage is generating lot of system load due to which all operations
went slow and server average load is above 60% which delayed my whole email
server logs are attached. can any one guide me what the prb is?
--
Regards
Nauman Yousuf
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20110126/9791ac17/attachment.htm>
-------------- next part --------------
primary MDS
bnx2: eth0 NIC Link is Up, 1000 Mbps full duplex
bnx2: eth1 NIC Link is Up, 1000 Mbps full duplex
bonding: bond0: link status up for interface eth0, enabling it in 1000 ms.
bonding: bond0: link status up for interface eth1, enabling it in 1000 ms.
bonding: bond0: link status definitely up for interface eth0.
bonding: bond0: link status definitely up for interface eth1.
bonding: bond0: making interface eth0 the new active one.
drbd: initialised. Version: 0.7.23 (api:79/proto:74)
drbd: SVN Revision: 2686 build by root at satyrs, 2007-04-24 14:43:41
drbd: registered as block device major 147
drbd0: Adjusting my ra_pages to backing device's (32 -> 1024)
drbd0: resync bitmap: bits=158692071 words=4959128
drbd0: size = 605 GB (634768281 KB)
drbd0: 605 GB marked out-of-sync by on disk bit-map.
drbd0: Found 6 transactions (324 active extents) in activity log.
drbd0: drbdsetup [1125]: cstate Unconfigured --> StandAlone
drbd0: drbdsetup [1138]: cstate StandAlone --> Unconnected
drbd0: drbd0_receiver [1139]: cstate Unconnected --> WFConnection
eth0: no IPv6 routers present
bond0: no IPv6 routers present
eth1: no IPv6 routers present
drbd0: Secondary/Unknown --> Primary/Unknown
Lustre: 1502:0:(module.c:382:init_libcfs_module()) maximum lustre stack 8192
Lustre: OBD class driver Build Version: 1.4.9.1-19691231170000-PRISTINE-.testsuite.tmp.lbuild-boulder.lbuild-v1_4_9_1-2.6-rhel4-i686.lbuild.BUILD.lustre-kerne
l-2.6.9.lustre.linux-2.6.9-42.0.8.EL_lustre.1.4.9.1smp, info at clusterfs.com
Lustre: Added LNI 10.65.200.6 at tcp [8/256]
Lustre: Accept secure, port 988
kjournald starting. Commit interval 5 seconds
LDISKFS-fs warning: maximal mount count reached, running e2fsck is recommended
LDISKFS FS on drbd0, internal journal
LDISKFS-fs: mounted filesystem with ordered data mode.
Lustre: 1679:0:(llog_obd.c:160:cat_cancel_cb()) processing log 0x8ab8005:4732d8ff at index 356 of catalog 0x8ab8002
Lustre: 1679:0:(llog_obd.c:160:cat_cancel_cb()) processing log 0x8ab8004:472cceab at index 345 of catalog 0x8ab8003
Lustre: 1679:0:(llog_obd.c:160:cat_cancel_cb()) processing log 0x8ab8006:472ed7dc at index 346 of catalog 0x8ab8003
Lustre: 1679:0:(llog_obd.c:160:cat_cancel_cb()) processing log 0x8ab8008:47468c57 at index 358 of catalog 0x8ab8003
Lustre: MDT mds01 now serving /dev/drbd0 (f74ccccb-80c7-443c-be33-1c2ec53f4b0c) with recovery enabled
Lustre: MDS mds01: All OSCs now active, resetting orphans
drbd0: drbd0_receiver [1139]: cstate WFConnection --> WFReportParams
drbd0: Handshake successful: DRBD Network Protocol version 74
drbd0: Connection established.
drbd0: I am(P): 1:00000004:00000001:00000041:00000002:10
drbd0: Peer(S): 0:00000003:00000001:00000040:00000002:00
drbd0: drbd0_receiver [1139]: cstate WFReportParams --> WFBitMapS
drbd0: Primary/Unknown --> Primary/Secondary
drbd0: drbd0_receiver [1139]: cstate WFBitMapS --> SyncSource
drbd0: Resync started as SyncSource (need to sync 634747844 KB [158686961 bits set]).
drbd0: Resync done (total 97313 sec; paused 0 sec; 6520 K/sec)
drbd0: drbd0_worker [1126]: cstate SyncSource --> Connected
Lustre: 1588:0:(lustre_fsfilt.h:283:fsfilt_setattr()) mds01: slow setattr 31s
Lustre: 1595:0:(lustre_fsfilt.h:182:fsfilt_start_log()) mds01: slow journal start 33s
Lustre: 1720:0:(lustre_fsfilt.h:182:fsfilt_start_log()) mds01: slow journal start 32s
Lustre: 1602:0:(lustre_fsfilt.h:182:fsfilt_start_log()) mds01: slow journal start 38s
Lustre: 1602:0:(lustre_fsfilt.h:182:fsfilt_start_log()) Skipped 2 previous similar messages
Lustre: 1656:0:(lustre_fsfilt.h:182:fsfilt_start_log()) mds01: slow journal start 34s
Lustre: 1741:0:(lustre_fsfilt.h:182:fsfilt_start_log()) mds01: slow journal start 34s
Lustre: 1601:0:(lustre_fsfilt.h:182:fsfilt_start_log()) mds01: slow journal start 46s
Lustre: 1667:0:(lustre_fsfilt.h:182:fsfilt_start_log()) mds01: slow journal start 46s
Lustre: 1601:0:(lustre_fsfilt.h:182:fsfilt_start_log()) Skipped 1 previous similar message
Lustre: 1737:0:(lustre_fsfilt.h:182:fsfilt_start_log()) mds01: slow journal start 37s
Lustre: 1737:0:(lustre_fsfilt.h:182:fsfilt_start_log()) Skipped 1 previous similar message
Lustre: 1604:0:(lustre_fsfilt.h:182:fsfilt_start_log()) mds01: slow journal start 34s
Lustre: 1604:0:(lustre_fsfilt.h:182:fsfilt_start_log()) Skipped 2 previous similar messages
Lustre: 1719:0:(lustre_fsfilt.h:182:fsfilt_start_log()) mds01: slow journal start 31s
Lustre: 1719:0:(lustre_fsfilt.h:182:fsfilt_start_log()) Skipped 5 previous similar messages
Lustre: 1716:0:(lustre_fsfilt.h:182:fsfilt_start_log()) mds01: slow journal start 31s
Lustre: 1716:0:(lustre_fsfilt.h:182:fsfilt_start_log()) Skipped 3 previous similar messages
Lustre: 1592:0:(lustre_fsfilt.h:182:fsfilt_start_log()) mds01: slow journal start 31s
Lustre: 1592:0:(lustre_fsfilt.h:182:fsfilt_start_log()) Skipped 3 previous similar messages
Lustre: 1643:0:(lustre_fsfilt.h:182:fsfilt_start_log()) mds01: slow journal start 41s
Lustre: 1643:0:(lustre_fsfilt.h:182:fsfilt_start_log()) Skipped 1 previous similar message
Lustre: 1741:0:(lustre_fsfilt.h:182:fsfilt_start_log()) mds01: slow journal start 40s
Lustre: 1741:0:(lustre_fsfilt.h:182:fsfilt_start_log()) Skipped 2 previous similar messages
Lustre: 1736:0:(lustre_fsfilt.h:182:fsfilt_start_log()) mds01: slow journal start 32s
Lustre: 1736:0:(lustre_fsfilt.h:182:fsfilt_start_log()) Skipped 6 previous similar messages
Lustre: 1593:0:(lustre_fsfilt.h:182:fsfilt_start_log()) mds01: slow journal start 32s
Lustre: 1593:0:(lustre_fsfilt.h:182:fsfilt_start_log()) Skipped 2 previous similar messages
Lustre: 1651:0:(lustre_fsfilt.h:182:fsfilt_start_log()) mds01: slow journal start 31s
Lustre: 1588:0:(lustre_fsfilt.h:182:fsfilt_start_log()) mds01: slow journal start 46s
Lustre: 1740:0:(lustre_fsfilt.h:182:fsfilt_start_log()) mds01: slow journal start 46s
Lustre: 1740:0:(lustre_fsfilt.h:182:fsfilt_start_log()) Skipped 2 previous similar messages
Lustre: 1671:0:(lustre_fsfilt.h:182:fsfilt_start_log()) mds01: slow journal start 39s
Lustre: 1607:0:(lustre_fsfilt.h:182:fsfilt_start_log()) mds01: slow journal start 39s
Lustre: 1607:0:(lustre_fsfilt.h:182:fsfilt_start_log()) Skipped 3 previous similar messages
Lustre: 1743:0:(lustre_fsfilt.h:182:fsfilt_start_log()) mds01: slow journal start 36s
Lustre: 1743:0:(lustre_fsfilt.h:182:fsfilt_start_log()) Skipped 6 previous similar messages
Lustre: 1581:0:(lustre_fsfilt.h:182:fsfilt_start_log()) mds01: slow journal start 32s
Lustre: 1670:0:(lustre_fsfilt.h:182:fsfilt_start_log()) mds01: slow journal start 32s
Lustre: 1735:0:(lustre_fsfilt.h:182:fsfilt_start_log()) mds01: slow journal start 32s
LustreError: 1593:0:(client.c:940:ptlrpc_expire_one_request()) @@@ timeout (sent at 1276502817, 6s ago) req at c229d200 x1425932/t0 o104->@NET_0x200000a41c82d_U
UID:15 lens 176/64 ref 1 fl Rpc:/0/0 rc 0/0
LustreError: A client on nid 10.65.200.45 at tcp was evicted from service mds01.
LustreError: 1593:0:(ldlm_lockd.c:427:ldlm_failed_ast()) ### blocking AST failed (-110): evicting client b59fa_MNT_client_5e85c4f00a at NET_0x200000a41c82d_UUID
NID 10.65.200.45 at tcp (10.65.200.45 at tcp) ns: mds-mds01_UUID lock: e24e6440/0x7bfc427d0b5b000e lrc: 2/0,0 mode: CR/CR res: 99848313/2526868437 bits 0x3 rrc: 4 t
ype: IBT flags: 20 remote: 0x54fc0cf21abf3df8 expref: 4296 pid 1586
Lustre: 7:0:(linux-debug.c:98:libcfs_run_upcall()) Invoked LNET upcall /usr/lib/lustre/lnet_upcall ROUTER_NOTIFY,10.65.200.45 at tcp,down,1276502808
drbd0: sock was shut down by peer
drbd0: drbd0_receiver [1139]: cstate Connected --> BrokenPipe
drbd0: short read expecting header on sock: r=0
drbd0: meta connection shut down by peer.
drbd0: asender terminated
drbd0: worker terminated
drbd0: drbd0_receiver [1139]: cstate BrokenPipe --> Unconnected
drbd0: Connection lost.
drbd0: drbd0_receiver [1139]: cstate Unconnected --> WFConnection
drbd0: drbd0_receiver [1139]: cstate WFConnection --> WFReportParams
drbd0: Handshake successful: DRBD Network Protocol version 74
drbd0: Connection established.
drbd0: I am(P): 1:00000004:00000001:00000042:00000002:10
drbd0: Peer(S): 1:00000004:00000001:00000041:00000002:00
drbd0: drbd0_receiver [1139]: cstate WFReportParams --> WFBitMapS
drbd0: Primary/Unknown --> Primary/Secondary
drbd0: drbd0_receiver [1139]: cstate WFBitMapS --> SyncSource
drbd0: Resync started as SyncSource (need to sync 33752 KB [8438 bits set]).
drbd0: Resync done (total 9 sec; paused 0 sec; 3748 K/sec)
drbd0: drbd0_worker [15901]: cstate SyncSource --> Connected
bnx2: eth0 NIC Link is Down
bonding: bond0: link status down for idle interface eth0, disabling it in 1000 ms.
bonding: bond0: link status definitely down for interface eth0, disabling it
bonding: bond0: making interface eth1 the new active one.
drbd0: PingAck did not arrive in time.
drbd0: drbd0_asender [15902]: cstate Connected --> NetworkFailure
drbd0: asender terminated
drbd0: drbd0_receiver [1139]: cstate NetworkFailure --> BrokenPipe
drbd0: short read expecting header on sock: r=-512
drbd0: worker terminated
drbd0: drbd0_receiver [1139]: cstate BrokenPipe --> Unconnected
drbd0: Connection lost.
drbd0: drbd0_receiver [1139]: cstate Unconnected --> WFConnection
bnx2: eth1 NIC Link is Down
bonding: bond0: link status down for idle interface eth1, disabling it in 1000 ms.
bonding: bond0: link status definitely down for interface eth1, disabling it
bonding: bond0: now running without any active interface !
LustreError: OSC_satyrs_ost1_mds01: Connection to service ost1 via nid 10.65.200.21 at tcp was lost; in progress operations using this service will wait for reco
very to complete.
Lustre: Changing connection for OSC_satyrs_ost1_mds01 to cyclops_UUID/10.65.200.30 at tcp
Lustre: Changing connection for OSC_satyrs_ost1_mds01 to typhoon_UUID/10.65.200.21 at tcp
Lustre: Skipped 1 previous similar message
Lustre: Changing connection for OSC_satyrs_ost1_mds01 to cyclops_UUID/10.65.200.30 at tcp
Lustre: Skipped 1 previous similar message
Lustre: Changing connection for OSC_satyrs_ost1_mds01 to typhoon_UUID/10.65.200.21 at tcp
Lustre: Skipped 1 previous similar message
Lustre: Changing connection for OSC_satyrs_ost1_mds01 to cyclops_UUID/10.65.200.30 at tcp
Lustre: Skipped 1 previous similar message
Lustre: Changing connection for OSC_satyrs_ost1_mds01 to typhoon_UUID/10.65.200.21 at tcp
Lustre: Skipped 1 previous similar message
Lustre: Changing connection for OSC_satyrs_ost1_mds01 to cyclops_UUID/10.65.200.30 at tcp
Lustre: Skipped 1 previous similar message
Lustre: Changing connection for OSC_satyrs_ost1_mds01 to typhoon_UUID/10.65.200.21 at tcp
Lustre: Skipped 1 previous similar message
Lustre: Changing connection for OSC_satyrs_ost1_mds01 to cyclops_UUID/10.65.200.30 at tcp
Lustre: Skipped 1 previous similar message
LustreError: 1780:0:(events.c:53:request_out_callback()) @@@ type 4, status -5 req at c22afe00 x8555234/t0 o8->ost1_UUID at cyclops_UUID:28 lens 240/272 ref 2 fl R
pc:/0/0 rc 0/0
Lustre: Changing connection for OSC_satyrs_ost1_mds01 to typhoon_UUID/10.65.200.21 at tcp
Lustre: Skipped 1 previous similar message
Lustre: Changing connection for OSC_satyrs_ost1_mds01 to cyclops_UUID/10.65.200.30 at tcp
Lustre: Skipped 1 previous similar message
LustreError: 1780:0:(events.c:53:request_out_callback()) @@@ type 4, status -5 req at f7cd3800 x8555238/t0 o8->ost1_UUID at cyclops_UUID:28 lens 240/272 ref 2 fl R
pc:/0/0 rc 0/0
LustreError: 1780:0:(events.c:53:request_out_callback()) Skipped 1 previous similar message
Lustre: Changing connection for OSC_satyrs_ost1_mds01 to typhoon_UUID/10.65.200.21 at tcp
Lustre: Skipped 1 previous similar message
Lustre: Changing connection for OSC_satyrs_ost1_mds01 to cyclops_UUID/10.65.200.30 at tcp
Lustre: Skipped 1 previous similar message
LustreError: 1780:0:(events.c:53:request_out_callback()) @@@ type 4, status -5 req at c2259c00 x8555242/t0 o8->ost1_UUID at cyclops_UUID:28 lens 240/272 ref 2 fl R
pc:/0/0 rc 0/0
LustreError: 1780:0:(events.c:53:request_out_callback()) Skipped 1 previous similar message
Lustre: Changing connection for OSC_satyrs_ost1_mds01 to typhoon_UUID/10.65.200.21 at tcp
Lustre: Skipped 1 previous similar message
Lustre: Changing connection for OSC_satyrs_ost1_mds01 to cyclops_UUID/10.65.200.30 at tcp
Lustre: Skipped 1 previous similar message
LustreError: 1780:0:(events.c:53:request_out_callback()) @@@ type 4, status -5 req at e2a7ca00 x8555246/t0 o8->ost1_UUID at cyclops_UUID:28 lens 240/272 ref 2 fl R
pc:/0/0 rc 0/0
LustreError: 1780:0:(events.c:53:request_out_callback()) Skipped 1 previous similar message
Lustre: Changing connection for OSC_satyrs_ost1_mds01 to typhoon_UUID/10.65.200.21 at tcp
Lustre: Skipped 1 previous similar message
Lustre: Changing connection for OSC_satyrs_ost1_mds01 to cyclops_UUID/10.65.200.30 at tcp
Lustre: Skipped 1 previous similar message
LustreError: 1780:0:(events.c:53:request_out_callback()) @@@ type 4, status -5 req at f7ccda00 x8555250/t0 o8->ost1_UUID at cyclops_UUID:28 lens 240/272 ref 2 fl R
pc:/0/0 rc 0/0
LustreError: 1780:0:(events.c:53:request_out_callback()) Skipped 1 previous similar message
Lustre: Changing connection for OSC_satyrs_ost1_mds01 to cyclops_UUID/10.65.200.30 at tcp
Lustre: Skipped 3 previous similar messages
LustreError: 1780:0:(events.c:53:request_out_callback()) @@@ type 4, status -5 req at e1f40800 x8555254/t0 o8->ost1_UUID at cyclops_UUID:28 lens 240/272 ref 2 fl R
pc:/0/0 rc 0/0
LustreError: 1780:0:(events.c:53:request_out_callback()) Skipped 1 previous similar message
Lustre: Changing connection for OSC_satyrs_ost1_mds01 to cyclops_UUID/10.65.200.30 at tcp
Lustre: Skipped 3 previous similar messages
LustreError: 1780:0:(events.c:53:request_out_callback()) @@@ type 4, status -5 req at d150d800 x8555258/t0 o8->ost1_UUID at cyclops_UUID:28 lens 240/272 ref 2 fl R
pc:/0/0 rc 0/0
LustreError: 1780:0:(events.c:53:request_out_callback()) Skipped 1 previous similar message
LustreError: 1780:0:(events.c:53:request_out_callback()) @@@ type 4, status -5 req at c22ae200 x8555262/t0 o8->ost1_UUID at cyclops_UUID:28 lens 240/272 ref 2 fl R
pc:/0/0 rc 0/0
LustreError: 1780:0:(events.c:53:request_out_callback()) Skipped 1 previous similar message
Lustre: Changing connection for OSC_satyrs_ost1_mds01 to cyclops_UUID/10.65.200.30 at tcp
Lustre: Skipped 7 previous similar messages
LustreError: 1780:0:(events.c:53:request_out_callback()) @@@ type 4, status -5 req at e18ad200 x8555266/t0 o8->ost1_UUID at cyclops_UUID:28 lens 240/272 ref 2 fl R
pc:/0/0 rc 0/0
LustreError: 1780:0:(events.c:53:request_out_callback()) Skipped 1 previous similar message
LustreError: 1780:0:(events.c:53:request_out_callback()) @@@ type 4, status -5 req at c2264400 x8555270/t0 o8->ost1_UUID at cyclops_UUID:28 lens 240/272 ref 2 fl R
pc:/0/0 rc 0/0
LustreError: 1780:0:(events.c:53:request_out_callback()) Skipped 1 previous similar message
LustreError: 1780:0:(events.c:53:request_out_callback()) @@@ type 4, status -5 req at c22b2600 x8555274/t0 o8->ost1_UUID at cyclops_UUID:28 lens 240/272 ref 2 fl R
pc:/0/0 rc 0/0
LustreError: 1780:0:(events.c:53:request_out_callback()) Skipped 1 previous similar message
LustreError: 1780:0:(events.c:53:request_out_callback()) @@@ type 4, status -5 req at c228b200 x8555278/t0 o8->ost1_UUID at cyclops_UUID:28 lens 240/272 ref 2 fl R
pc:/0/0 rc 0/0
LustreError: 1780:0:(events.c:53:request_out_callback()) Skipped 1 previous similar message
Lustre: Changing connection for OSC_satyrs_ost1_mds01 to cyclops_UUID/10.65.200.30 at tcp
Lustre: Skipped 15 previous similar messages
LustreError: 1780:0:(events.c:53:request_out_callback()) @@@ type 4, status -5 req at e18aca00 x8555282/t0 o8->ost1_UUID at cyclops_UUID:28 lens 240/272 ref 2 fl R
pc:/0/0 rc 0/0
LustreError: 1780:0:(events.c:53:request_out_callback()) Skipped 1 previous similar message
LustreError: 1780:0:(events.c:53:request_out_callback()) @@@ type 4, status -5 req at c22b2200 x8555286/t0 o8->ost1_UUID at cyclops_UUID:28 lens 240/272 ref 2 fl R
pc:/0/0 rc 0/0
LustreError: 1780:0:(events.c:53:request_out_callback()) Skipped 1 previous similar message
LustreError: 1780:0:(events.c:53:request_out_callback()) @@@ type 4, status -5 req at c2289600 x8555290/t0 o8->ost1_UUID at cyclops_UUID:28 lens 240/272 ref 2 fl R
pc:/0/0 rc 0/0
LustreError: 1780:0:(events.c:53:request_out_callback()) Skipped 1 previous similar message
LustreError: 1780:0:(events.c:53:request_out_callback()) @@@ type 4, status -5 req at e18a7400 x8555294/t0 o8->ost1_UUID at cyclops_UUID:28 lens 240/272 ref 2 fl R
pc:/0/0 rc 0/0
LustreError: 1780:0:(events.c:53:request_out_callback()) Skipped 1 previous similar message
LustreError: 1780:0:(events.c:53:request_out_callback()) @@@ type 4, status -5 req at c229c200 x8555298/t0 o8->ost1_UUID at cyclops_UUID:28 lens 240/272 ref 2 fl R
pc:/0/0 rc 0/0
LustreError: 1780:0:(events.c:53:request_out_callback()) Skipped 1 previous similar message
LustreError: 1780:0:(events.c:53:request_out_callback()) @@@ type 4, status -5 req at f7ce4600 x8555302/t0 o8->ost1_UUID at cyclops_UUID:28 lens 240/272 ref 2 fl R
pc:/0/0 rc 0/0
LustreError: 1780:0:(events.c:53:request_out_callback()) Skipped 1 previous similar message
LustreError: 1780:0:(events.c:53:request_out_callback()) @@@ type 4, status -5 req at e1f4ea00 x8555306/t0 o8->ost1_UUID at cyclops_UUID:28 lens 240/272 ref 2 fl R
pc:/0/0 rc 0/0
LustreError: 1780:0:(events.c:53:request_out_callback()) Skipped 1 previous similar message
Lustre: Changing connection for OSC_satyrs_ost1_mds01 to cyclops_UUID/10.65.200.30 at tcp
Lustre: Skipped 27 previous similar messages
LustreError: 1780:0:(events.c:53:request_out_callback()) @@@ type 4, status -5 req at c22b2800 x8555314/t0 o8->ost1_UUID at cyclops_UUID:28 lens 240/272 ref 2 fl R
pc:/0/0 rc 0/0
LustreError: 1780:0:(events.c:53:request_out_callback()) Skipped 3 previous similar messages
LustreError: 1780:0:(events.c:53:request_out_callback()) @@@ type 4, status -5 req at c22a1c00 x8555330/t0 o8->ost1_UUID at cyclops_UUID:28 lens 240/272 ref 2 fl R
pc:/0/0 rc 0/0
LustreError: 1780:0:(events.c:53:request_out_callback()) Skipped 7 previous similar messages
Lustre: Changing connection for OSC_satyrs_ost1_mds01 to typhoon_UUID/10.65.200.21 at tcp
Lustre: Skipped 33 previous similar messages
LustreError: 1780:0:(events.c:53:request_out_callback()) @@@ type 4, status -5 req at c22b2e00 x8555358/t0 o8->ost1_UUID at cyclops_UUID:28 lens 240/272 ref 2 fl R
pc:/0/0 rc 0/0
LustreError: 1780:0:(events.c:53:request_out_callback()) Skipped 13 previous similar messages
Lustre: Changing connection for OSC_satyrs_ost1_mds01 to cyclops_UUID/10.65.200.30 at tcp
Lustre: Skipped 33 previous similar messages
LustreError: 1780:0:(events.c:53:request_out_callback()) @@@ type 4, status -5 req at c228ee00 x8555394/t0 o8->ost1_UUID at cyclops_UUID:28 lens 240/272 ref 2 fl R
pc:/0/0 rc 0/0
LustreError: 1780:0:(events.c:53:request_out_callback()) Skipped 17 previous similar messages
Lustre: Changing connection for OSC_satyrs_ost1_mds01 to typhoon_UUID/10.65.200.21 at tcp
Lustre: Skipped 33 previous similar messages
LustreError: 1780:0:(events.c:53:request_out_callback()) @@@ type 4, status -5 req at c228e400 x8555430/t0 o8->ost1_UUID at cyclops_UUID:28 lens 240/272 ref 2 fl R
pc:/0/0 rc 0/0
LustreError: 1780:0:(events.c:53:request_out_callback()) Skipped 17 previous similar messages
Lustre: Changing connection for OSC_satyrs_ost1_mds01 to cyclops_UUID/10.65.200.30 at tcp
Lustre: Skipped 33 previous similar messages
LustreError: 1780:0:(events.c:53:request_out_callback()) @@@ type 4, status -5 req at c228ee00 x8555466/t0 o8->ost1_UUID at cyclops_UUID:28 lens 240/272 ref 2 fl R
pc:/0/0 rc 0/0
LustreError: 1780:0:(events.c:53:request_out_callback()) Skipped 17 previous similar messages
Lustre: Changing connection for OSC_satyrs_ost1_mds01 to cyclops_UUID/10.65.200.30 at tcp
Lustre: Skipped 31 previous similar messages
LustreError: 1780:0:(events.c:53:request_out_callback()) @@@ type 4, status -5 req at f7cce600 x8555498/t0 o8->ost1_UUID at cyclops_UUID:28 lens 240/272 ref 2 fl R
pc:/0/0 rc 0/0
LustreError: 1780:0:(events.c:53:request_out_callback()) Skipped 15 previous similar messages
Lustre: Changing connection for OSC_satyrs_ost1_mds01 to cyclops_UUID/10.65.200.30 at tcp
Lustre: Skipped 31 previous similar messages
LustreError: 1780:0:(events.c:53:request_out_callback()) @@@ type 4, status -5 req at e18bf800 x8555534/t0 o8->ost1_UUID at cyclops_UUID:28 lens 240/272 ref 2 fl R
pc:/0/0 rc 0/0
LustreError: 1780:0:(events.c:53:request_out_callback()) Skipped 17 previous similar messages
Lustre: Changing connection for OSC_satyrs_ost1_mds01 to typhoon_UUID/10.65.200.21 at tcp
Lustre: Skipped 33 previous similar messages
LustreError: 1780:0:(events.c:53:request_out_callback()) @@@ type 4, status -5 req at f7ce4600 x8555570/t0 o8->ost1_UUID at cyclops_UUID:28 lens 240/272 ref 2 fl R
pc:/0/0 rc 0/0
LustreError: 1780:0:(events.c:53:request_out_callback()) Skipped 17 previous similar messages
Lustre: Changing connection for OSC_satyrs_ost1_mds01 to cyclops_UUID/10.65.200.30 at tcp
Lustre: Skipped 33 previous similar messages
LustreError: 1780:0:(events.c:53:request_out_callback()) @@@ type 4, status -5 req at e18ada00 x8555606/t0 o8->ost1_UUID at cyclops_UUID:28 lens 240/272 ref 2 fl R
pc:/0/0 rc 0/0
LustreError: 1780:0:(events.c:53:request_out_callback()) Skipped 17 previous similar messages
Lustre: Changing connection for OSC_satyrs_ost1_mds01 to typhoon_UUID/10.65.200.21 at tcp
Lustre: Skipped 33 previous similar messages
LustreError: 1780:0:(events.c:53:request_out_callback()) @@@ type 4, status -5 req at e182ae00 x8555638/t0 o8->ost1_UUID at cyclops_UUID:28 lens 240/272 ref 2 fl R
LustreError: 1780:0:(events.c:53:request_out_callback()) Skipped 15 previous similar messages
Lustre: Changing connection for OSC_satyrs_ost1_mds01 to typhoon_UUID/10.65.200.21 at tcp
Lustre: Skipped 31 previous similar messages
bnx2: eth1 NIC Link is Up, 1000 Mbps full duplex
bonding: bond0: link status up for interface eth1, enabling it in 1000 ms.
bonding: bond0: link status definitely up for interface eth1.
bonding: bond0: making interface eth1 the new active one.
bnx2: eth0 NIC Link is Up, 1000 Mbps full duplex
bonding: bond0: link status up for interface eth0, enabling it in 1000 ms.
bonding: bond0: link status definitely up for interface eth0.
drbd0: drbd0_receiver [1139]: cstate WFConnection --> WFReportParams
drbd0: Handshake successful: DRBD Network Protocol version 74
drbd0: Connection established.
drbd0: I am(P): 1:00000004:00000001:00000043:00000002:10
drbd0: Peer(S): 1:00000004:00000001:00000042:00000002:01
drbd0: drbd0_receiver [1139]: cstate WFReportParams --> WFBitMapS
drbd0: Primary/Unknown --> Primary/Secondary
drbd0: drbd0_receiver [1139]: cstate WFBitMapS --> SyncSource
drbd0: Resync started as SyncSource (need to sync 12 KB [3 bits set]).
drbd0: Resync done (total 1 sec; paused 0 sec; 12 K/sec)
drbd0: drbd0_worker [28899]: cstate SyncSource --> Connected
Lustre: 1606:0:(ldlm_lib.c:486:target_handle_reconnect()) mds01: 92e82_MNT_client_8029478b2f reconnecting
Lustre: OSC_satyrs_ost1_mds01: Connection restored to service ost1 using nid 10.65.200.21 at tcp.
Lustre: MDS mds01: ost1_UUID now active, resetting orphans
Lustre: mds01: haven't heard from 10.65.200.37 at tcp in 8538 seconds. Last request was at 1280642838. I think it's dead, and I am evicting it.
LustreError: 1643:0:(mds_open.c:1455:mds_close()) @@@ no handle for file close ino 99713500: cookie 0x7bfc427e3595552c req at e1f4e400 x456756486/t0 o35->252fb_
MNT_client_3715c9a62f at NET_0x200000a41c825_UUID:-1 lens 240/392 ref 0 fl Interpret:/0/0 rc 0/0
LustreError: 1643:0:(ldlm_lib.c:1318:target_send_reply_msg()) @@@ processing error (-116) req at e1f4e400 x456756486/t0 o35->252fb_MNT_client_3715c9a62f at NET_0x2
00000a41c825_UUID:-1 lens 240/392 ref 0 fl Interpret:/0/0 rc -116/0
Lustre: 1714:0:(lustre_fsfilt.h:182:fsfilt_start_log()) mds01: slow journal start 30s
Lustre: 1714:0:(lustre_fsfilt.h:182:fsfilt_start_log()) Skipped 7 previous similar messages
Lustre: 1650:0:(lustre_fsfilt.h:182:fsfilt_start_log()) mds01: slow journal start 30s
Lustre: 1650:0:(lustre_fsfilt.h:182:fsfilt_start_log()) Skipped 9 previous similar messages
Lustre: 1734:0:(lustre_fsfilt.h:182:fsfilt_start_log()) mds01: slow journal start 35s
Lustre: 1652:0:(lustre_fsfilt.h:182:fsfilt_start_log()) mds01: slow journal start 37s
Lustre: 1652:0:(lustre_fsfilt.h:182:fsfilt_start_log()) Skipped 1 previous similar message
Lustre: 1665:0:(lustre_fsfilt.h:182:fsfilt_start_log()) mds01: slow journal start 37s
Lustre: 1665:0:(lustre_fsfilt.h:182:fsfilt_start_log()) Skipped 1 previous similar message
Lustre: 1655:0:(lustre_fsfilt.h:182:fsfilt_start_log()) mds01: slow journal start 33s
Lustre: 1730:0:(lustre_fsfilt.h:182:fsfilt_start_log()) mds01: slow journal start 32s
Lustre: 1730:0:(lustre_fsfilt.h:182:fsfilt_start_log()) Skipped 4 previous similar messages
Lustre: 1590:0:(lustre_fsfilt.h:182:fsfilt_start_log()) mds01: slow journal start 46s
Lustre: 1590:0:(lustre_fsfilt.h:182:fsfilt_start_log()) Skipped 2 previous similar messages
Lustre: 1645:0:(lustre_fsfilt.h:182:fsfilt_start_log()) mds01: slow journal start 46s
Lustre: 1733:0:(lustre_fsfilt.h:182:fsfilt_start_log()) mds01: slow journal start 46s
LustreError: 1725:0:(lustre_fsfilt.h:182:fsfilt_start_log()) mds01: slow journal start 66s
LustreError: 1648:0:(lustre_fsfilt.h:182:fsfilt_start_log()) mds01: slow journal start 64s
LustreError: 1714:0:(lustre_fsfilt.h:182:fsfilt_start_log()) mds01: slow journal start 58s
LustreError: 1714:0:(lustre_fsfilt.h:182:fsfilt_start_log()) Skipped 2 previous similar messages
Lustre: 1724:0:(lustre_fsfilt.h:182:fsfilt_start_log()) mds01: slow journal start 40s
Lustre: 1724:0:(lustre_fsfilt.h:182:fsfilt_start_log()) Skipped 9 previous similar messages
LustreError: 1658:0:(lustre_fsfilt.h:182:fsfilt_start_log()) mds01: slow journal start 66s
Lustre: 1714:0:(lustre_fsfilt.h:182:fsfilt_start_log()) mds01: slow journal start 36s
Lustre: 1714:0:(lustre_fsfilt.h:182:fsfilt_start_log()) Skipped 1 previous similar message
Lustre: 1653:0:(lustre_fsfilt.h:182:fsfilt_start_log()) mds01: slow journal start 36s
Lustre: 1593:0:(lustre_fsfilt.h:182:fsfilt_start_log()) mds01: slow journal start 37s
Lustre: 1593:0:(lustre_fsfilt.h:182:fsfilt_start_log()) Skipped 2 previous similar messages
Lustre: 1662:0:(lustre_fsfilt.h:182:fsfilt_start_log()) mds01: slow journal start 30s
Lustre: 1662:0:(lustre_fsfilt.h:182:fsfilt_start_log()) Skipped 1 previous similar message
Lustre: 1657:0:(lustre_fsfilt.h:182:fsfilt_start_log()) mds01: slow journal start 30s
Lustre: 1743:0:(lustre_fsfilt.h:182:fsfilt_start_log()) mds01: slow journal start 30s
Lustre: 1743:0:(lustre_fsfilt.h:182:fsfilt_start_log()) Skipped 1 previous similar message
Lustre: 1657:0:(lustre_fsfilt.h:182:fsfilt_start_log()) Skipped 1 previous similar message
Lustre: 1598:0:(lustre_fsfilt.h:182:fsfilt_start_log()) mds01: slow journal start 46s
Lustre: 1598:0:(lustre_fsfilt.h:182:fsfilt_start_log()) Skipped 1 previous similar message
Lustre: 1671:0:(lustre_fsfilt.h:182:fsfilt_start_log()) mds01: slow journal start 46s
Lustre: 1744:0:(lustre_fsfilt.h:182:fsfilt_start_log()) mds01: slow journal start 46s
Lustre: 1648:0:(lustre_fsfilt.h:182:fsfilt_start_log()) mds01: slow journal start 30s
Lustre: 1610:0:(lustre_fsfilt.h:182:fsfilt_start_log()) mds01: slow journal start 30s
Lustre: 1610:0:(lustre_fsfilt.h:182:fsfilt_start_log()) Skipped 2 previous similar messages
Lustre: 1610:0:(lustre_fsfilt.h:182:fsfilt_start_log()) mds01: slow journal start 47s
Lustre: 1610:0:(lustre_fsfilt.h:182:fsfilt_start_log()) Skipped 1 previous similar message
Lustre: 1718:0:(lustre_fsfilt.h:182:fsfilt_start_log()) mds01: slow journal start 47s
Lustre: 1718:0:(lustre_fsfilt.h:182:fsfilt_start_log()) Skipped 2 previous similar messages
Lustre: 1610:0:(lustre_fsfilt.h:182:fsfilt_start_log()) mds01: slow journal start 40s
Lustre: 1610:0:(lustre_fsfilt.h:182:fsfilt_start_log()) Skipped 3 previous similar messages
Lustre: 1729:0:(lustre_fsfilt.h:182:fsfilt_start_log()) mds01: slow journal start 38s
Lustre: 1729:0:(lustre_fsfilt.h:182:fsfilt_start_log()) Skipped 1 previous similar message
LustreError: 1665:0:(lustre_fsfilt.h:182:fsfilt_start_log()) mds01: slow journal start 68s
LustreError: 1665:0:(lustre_fsfilt.h:182:fsfilt_start_log()) Skipped 1 previous similar message
LustreError: 1731:0:(lustre_fsfilt.h:182:fsfilt_start_log()) mds01: slow journal start 63s
LustreError: 1731:0:(lustre_fsfilt.h:182:fsfilt_start_log()) Skipped 2 previous similar messages
LustreError: 1738:0:(lustre_fsfilt.h:182:fsfilt_start_log()) mds01: slow journal start 52s
Lustre: 1724:0:(lustre_fsfilt.h:182:fsfilt_start_log()) mds01: slow journal start 46s
LustreError: 1727:0:(lustre_fsfilt.h:182:fsfilt_start_log()) mds01: slow journal start 70s
LustreError: 1595:0:(lustre_fsfilt.h:182:fsfilt_start_log()) mds01: slow journal start 70s
LustreError: 1595:0:(lustre_fsfilt.h:182:fsfilt_start_log()) Skipped 1 previous similar message
Lustre: 1591:0:(lustre_fsfilt.h:182:fsfilt_start_log()) mds01: slow journal start 45s
Lustre: 1591:0:(lustre_fsfilt.h:182:fsfilt_start_log()) Skipped 3 previous similar messages
Lustre: 1718:0:(lustre_fsfilt.h:182:fsfilt_start_log()) mds01: slow journal start 31s
Lustre: 1718:0:(lustre_fsfilt.h:182:fsfilt_start_log()) Skipped 3 previous similar messages
LustreError: 1578:0:(lustre_fsfilt.h:182:fsfilt_start_log()) mds01: slow journal start 59s
LustreError: 1578:0:(lustre_fsfilt.h:182:fsfilt_start_log()) Skipped 1 previous similar message
LustreError: 1724:0:(lustre_fsfilt.h:182:fsfilt_start_log()) mds01: slow journal start 58s
LustreError: 1724:0:(lustre_fsfilt.h:182:fsfilt_start_log()) Skipped 1 previous similar message
Lustre: 1714:0:(lustre_fsfilt.h:182:fsfilt_start_log()) mds01: slow journal start 48s
Lustre: 1714:0:(lustre_fsfilt.h:182:fsfilt_start_log()) Skipped 4 previous similar messages
LustreError: 1606:0:(lustre_fsfilt.h:182:fsfilt_start_log()) mds01: slow journal start 58s
LustreError: 1606:0:(lustre_fsfilt.h:182:fsfilt_start_log()) Skipped 3 previous similar messages
LustreError: 1670:0:(lustre_fsfilt.h:182:fsfilt_start_log()) mds01: slow journal start 58s
LustreError: 1742:0:(lustre_fsfilt.h:182:fsfilt_start_log()) mds01: slow journal start 58s
LustreError: 1742:0:(lustre_fsfilt.h:182:fsfilt_start_log()) Skipped 5 previous similar messages
Lustre: 1727:0:(lustre_fsfilt.h:182:fsfilt_start_log()) mds01: slow journal start 44s
Lustre: 1661:0:(lustre_fsfilt.h:182:fsfilt_start_log()) mds01: slow journal start 30s
Lustre: 1661:0:(lustre_fsfilt.h:182:fsfilt_start_log()) Skipped 3 previous similar messages
Lustre: 1594:0:(lustre_fsfilt.h:182:fsfilt_start_log()) mds01: slow journal start 46s
Lustre: 1594:0:(lustre_fsfilt.h:182:fsfilt_start_log()) Skipped 1 previous similar message
Lustre: 1659:0:(lustre_fsfilt.h:182:fsfilt_start_log()) mds01: slow journal start 46s
Lustre: 1717:0:(lustre_fsfilt.h:182:fsfilt_start_log()) mds01: slow journal start 42s
Lustre: 1717:0:(lustre_fsfilt.h:182:fsfilt_start_log()) Skipped 1 previous similar message
Lustre: 1602:0:(lustre_fsfilt.h:182:fsfilt_start_log()) mds01: slow journal start 49s
Lustre: 1602:0:(lustre_fsfilt.h:182:fsfilt_start_log()) Skipped 1 previous similar message
LustreError: 1671:0:(lustre_fsfilt.h:182:fsfilt_start_log()) mds01: slow journal start 65s
LustreError: 1671:0:(lustre_fsfilt.h:182:fsfilt_start_log()) Skipped 1 previous similar message
Lustre: 1727:0:(lustre_fsfilt.h:182:fsfilt_start_log()) mds01: slow journal start 49s
Lustre: 1727:0:(lustre_fsfilt.h:182:fsfilt_start_log()) Skipped 4 previous similar messages
Lustre: 1659:0:(lustre_fsfilt.h:283:fsfilt_setattr()) mds01: slow setattr 31s
LustreError: 1735:0:(llog_server.c:433:llog_origin_handle_cancel()) cancel 125 llog-records failed: -22
Lustre: 1582:0:(lustre_fsfilt.h:182:fsfilt_start_log()) mds01: slow journal start 33s
Lustre: 1582:0:(lustre_fsfilt.h:182:fsfilt_start_log()) Skipped 4 previous similar messages
Lustre: 1714:0:(lustre_fsfilt.h:182:fsfilt_start_log()) mds01: slow journal start 32s
Lustre: 1714:0:(lustre_fsfilt.h:182:fsfilt_start_log()) Skipped 8 previous similar messages
Lustre: 1658:0:(lustre_fsfilt.h:182:fsfilt_start_log()) mds01: slow journal start 45s
Lustre: 1658:0:(lustre_fsfilt.h:182:fsfilt_start_log()) Skipped 1 previous similar message
Lustre: 1729:0:(lustre_fsfilt.h:182:fsfilt_start_log()) mds01: slow journal start 43s
Lustre: 1729:0:(lustre_fsfilt.h:182:fsfilt_start_log()) Skipped 3 previous similar messages
LustreError: 1595:0:(lov_request.c:666:lov_update_create_set()) error creating fid 0x5f4b399 sub-object on OST idx 0/1: rc = -28
LustreError: 1582:0:(lov_request.c:666:lov_update_create_set()) error creating fid 0x8747f6d sub-object on OST idx 0/1: rc = -28
LustreError: 1609:0:(lov_request.c:666:lov_update_create_set()) error creating fid 0x3fa8002 sub-object on OST idx 0/1: rc = -28
LustreError: 1609:0:(lov_request.c:666:lov_update_create_set()) error creating fid 0x87504c3 sub-object on OST idx 0/1: rc = -28
LustreError: 1589:0:(lov_request.c:666:lov_update_create_set()) error creating fid 0x8750510 sub-object on OST idx 0/1: rc = -28
LustreError: 1610:0:(lov_request.c:666:lov_update_create_set()) error creating fid 0x5f7cf48 sub-object on OST idx 0/1: rc = -28
LustreError: 1588:0:(lov_request.c:666:lov_update_create_set()) error creating fid 0x8750c4d sub-object on OST idx 0/1: rc = -28
LustreError: 1589:0:(lov_request.c:666:lov_update_create_set()) error creating fid 0x5f38fae sub-object on OST idx 0/1: rc = -28
LustreError: 1589:0:(lov_request.c:666:lov_update_create_set()) Skipped 1 previous similar message
LustreError: 1591:0:(lov_request.c:666:lov_update_create_set()) error creating fid 0x5f38fae sub-object on OST idx 0/1: rc = -28
LustreError: 1607:0:(lov_request.c:666:lov_update_create_set()) error creating fid 0x87516a7 sub-object on OST idx 0/1: rc = -28
LustreError: 1595:0:(lov_request.c:666:lov_update_create_set()) error creating fid 0x5230001 sub-object on OST idx 0/1: rc = -28
LustreError: 1580:0:(lov_request.c:666:lov_update_create_set()) error creating fid 0x875276c sub-object on OST idx 0/1: rc = -28
LustreError: 1578:0:(lov_request.c:666:lov_update_create_set()) error creating fid 0x8752eb5 sub-object on OST idx 0/1: rc = -28
LustreError: 1607:0:(lov_request.c:666:lov_update_create_set()) error creating fid 0x8752ef5 sub-object on OST idx 0/1: rc = -28
LustreError: 1592:0:(lov_request.c:666:lov_update_create_set()) error creating fid 0x5f2b2fd sub-object on OST idx 0/1: rc = -28
LustreError: 1578:0:(lov_request.c:666:lov_update_create_set()) error creating fid 0x8753772 sub-object on OST idx 0/1: rc = -28
LustreError: 1595:0:(lov_request.c:666:lov_update_create_set()) error creating fid 0x8753844 sub-object on OST idx 0/1: rc = -28
LustreError: 1588:0:(lov_request.c:666:lov_update_create_set()) error creating fid 0x875389c sub-object on OST idx 0/1: rc = -28
LustreError: 1587:0:(lov_request.c:666:lov_update_create_set()) error creating fid 0x8753cca sub-object on OST idx 0/1: rc = -28
LustreError: 1583:0:(lov_request.c:666:lov_update_create_set()) error creating fid 0x87548ab sub-object on OST idx 0/1: rc = -28
LustreError: 1600:0:(lov_request.c:666:lov_update_create_set()) error creating fid 0x875499b sub-object on OST idx 0/1: rc = -28
LustreError: 1582:0:(lov_request.c:666:lov_update_create_set()) error creating fid 0x8754aba sub-object on OST idx 0/1: rc = -28
LustreError: 1591:0:(lov_request.c:666:lov_update_create_set()) error creating fid 0x8754b5a sub-object on OST idx 0/1: rc = -28
LustreError: 1603:0:(lov_request.c:666:lov_update_create_set()) error creating fid 0x8754bd9 sub-object on OST idx 0/1: rc = -28
LustreError: 1604:0:(lov_request.c:666:lov_update_create_set()) error creating fid 0x8755bd1 sub-object on OST idx 0/1: rc = -28
LustreError: 1600:0:(lov_request.c:666:lov_update_create_set()) error creating fid 0x8755d06 sub-object on OST idx 0/1: rc = -28
LustreError: 1580:0:(lov_request.c:666:lov_update_create_set()) error creating fid 0x8755e91 sub-object on OST idx 0/1: rc = -28
LustreError: 1580:0:(lov_request.c:666:lov_update_create_set()) error creating fid 0x8756425 sub-object on OST idx 0/1: rc = -28
LustreError: 1608:0:(lov_request.c:666:lov_update_create_set()) error creating fid 0x8756995 sub-object on OST idx 0/1: rc = -28
LustreError: 1590:0:(lov_request.c:666:lov_update_create_set()) error creating fid 0x8756a0a sub-object on OST idx 0/1: rc = -28
LustreError: 1599:0:(lov_request.c:666:lov_update_create_set()) error creating fid 0x8756c09 sub-object on OST idx 0/1: rc = -28
LustreError: 1578:0:(lov_request.c:666:lov_update_create_set()) error creating fid 0x8756c49 sub-object on OST idx 0/1: rc = -28
LustreError: 1586:0:(lov_request.c:666:lov_update_create_set()) error creating fid 0x8756c69 sub-object on OST idx 0/1: rc = -28
LustreError: 1600:0:(lov_request.c:666:lov_update_create_set()) error creating fid 0x87571ab sub-object on OST idx 0/1: rc = -28
LustreError: 1583:0:(lov_request.c:666:lov_update_create_set()) error creating fid 0x87700ab sub-object on OST idx 0/1: rc = -28
LustreError: 1583:0:(lov_request.c:666:lov_update_create_set()) error creating fid 0x87702b5 sub-object on OST idx 0/1: rc = -28
LustreError: 1609:0:(lov_request.c:666:lov_update_create_set()) error creating fid 0x8770330 sub-object on OST idx 0/1: rc = -28
LustreError: 1578:0:(lov_request.c:666:lov_update_create_set()) error creating fid 0x5f186b9 sub-object on OST idx 0/1: rc = -28
LustreError: 1608:0:(lov_request.c:666:lov_update_create_set()) error creating fid 0x877034c sub-object on OST idx 0/1: rc = -28
LustreError: 1585:0:(lov_request.c:666:lov_update_create_set()) error creating fid 0x87703c5 sub-object on OST idx 0/1: rc = -28
LustreError: 1585:0:(lov_request.c:666:lov_update_create_set()) Skipped 1 previous similar message
LustreError: 1610:0:(lov_request.c:666:lov_update_create_set()) error creating fid 0x8770d69 sub-object on OST idx 0/1: rc = -28
LustreError: 1599:0:(lov_request.c:666:lov_update_create_set()) error creating fid 0x5f99c50 sub-object on OST idx 0/1: rc = -28
LustreError: 1609:0:(lov_request.c:666:lov_update_create_set()) error creating fid 0x8771c20 sub-object on OST idx 0/1: rc = -28
LustreError: 1606:0:(lov_request.c:666:lov_update_create_set()) error creating fid 0x8771ded sub-object on OST idx 0/1: rc = -28
LustreError: 1586:0:(lov_request.c:666:lov_update_create_set()) error creating fid 0x8742c64 sub-object on OST idx 0/1: rc = -28
LustreError: 1592:0:(lov_request.c:666:lov_update_create_set()) error creating fid 0x8754719 sub-object on OST idx 0/1: rc = -28
LustreError: 1588:0:(lov_request.c:666:lov_update_create_set()) error creating fid 0x87566ef sub-object on OST idx 0/1: rc = -28
LustreError: 1583:0:(lov_request.c:666:lov_update_create_set()) error creating fid 0x8773897 sub-object on OST idx 0/1: rc = -28
LustreError: 1578:0:(lov_request.c:666:lov_update_create_set()) error creating fid 0x8773d33 sub-object on OST idx 0/1: rc = -28
LustreError: 1593:0:(lov_request.c:666:lov_update_create_set()) error creating fid 0x87740ca sub-object on OST idx 0/1: rc = -28
LustreError: 1590:0:(lov_request.c:666:lov_update_create_set()) error creating fid 0x8774d26 sub-object on OST idx 0/1: rc = -28
LustreError: 1588:0:(lov_request.c:666:lov_update_create_set()) error creating fid 0x87754e0 sub-object on OST idx 0/1: rc = -28
LustreError: 1586:0:(lov_request.c:666:lov_update_create_set()) error creating fid 0x877559f sub-object on OST idx 0/1: rc = -28
LustreError: 1586:0:(lov_request.c:666:lov_update_create_set()) error creating fid 0x877575d sub-object on OST idx 0/1: rc = -28
LustreError: 1609:0:(lov_request.c:666:lov_update_create_set()) error creating fid 0x87757b0 sub-object on OST idx 0/1: rc = -28
LustreError: 1594:0:(lov_request.c:666:lov_update_create_set()) error creating fid 0x877580e sub-object on OST idx 0/1: rc = -28
LustreError: 1587:0:(lov_request.c:666:lov_update_create_set()) error creating fid 0x877631e sub-object on OST idx 0/1: rc = -28
LustreError: 1586:0:(lov_request.c:666:lov_update_create_set()) error creating fid 0x8776b90 sub-object on OST idx 0/1: rc = -28
LustreError: 1582:0:(lov_request.c:666:lov_update_create_set()) error creating fid 0x8776e7a sub-object on OST idx 0/1: rc = -28
LustreError: 1583:0:(lov_request.c:666:lov_update_create_set()) error creating fid 0x8777bfe sub-object on OST idx 0/1: rc = -28
LustreError: 1596:0:(lov_request.c:666:lov_update_create_set()) error creating fid 0x8777cbc sub-object on OST idx 0/1: rc = -28
LustreError: 1586:0:(lov_request.c:666:lov_update_create_set()) error creating fid 0x87b0396 sub-object on OST idx 0/1: rc = -28
LustreError: 1590:0:(lov_request.c:666:lov_update_create_set()) error creating fid 0x87b0473 sub-object on OST idx 0/1: rc = -28
LustreError: 1592:0:(lov_request.c:666:lov_update_create_set()) error creating fid 0x87b0820 sub-object on OST idx 0/1: rc = -28
LustreError: 1586:0:(lov_request.c:666:lov_update_create_set()) error creating fid 0x87b095e sub-object on OST idx 0/1: rc = -28
LustreError: 1604:0:(lov_request.c:666:lov_update_create_set()) error creating fid 0x87b1326 sub-object on OST idx 0/1: rc = -28
LustreError: 1596:0:(lov_request.c:666:lov_update_create_set()) error creating fid 0x87b1346 sub-object on OST idx 0/1: rc = -28
LustreError: 1610:0:(lov_request.c:666:lov_update_create_set()) error creating fid 0x87b216a sub-object on OST idx 0/1: rc = -28
LustreError: 1589:0:(lov_request.c:666:lov_update_create_set()) error creating fid 0x87b2362 sub-object on OST idx 0/1: rc = -28
bnx2: eth0 NIC Link is Down
bonding: bond0: link status down for idle interface eth0, disabling it in 1000 ms.
bonding: bond0: link status definitely down for interface eth0, disabling it
drbd0: PingAck did not arrive in time.
drbd0: drbd0_asender [28919]: cstate Connected --> NetworkFailure
drbd0: asender terminated
drbd0: drbd0_receiver [1139]: cstate NetworkFailure --> BrokenPipe
drbd0: short read expecting header on sock: r=-512
drbd0: worker terminated
drbd0: drbd0_receiver [1139]: cstate BrokenPipe --> Unconnected
drbd0: Connection lost.
drbd0: drbd0_receiver [1139]: cstate Unconnected --> WFConnection
bnx2: eth0 NIC Link is Up, 1000 Mbps full duplex
bonding: bond0: link status up for interface eth0, enabling it in 1000 ms.
bonding: bond0: link status definitely up for interface eth0.
drbd0: drbd0_receiver [1139]: cstate WFConnection --> WFReportParams
drbd0: Handshake successful: DRBD Network Protocol version 74
drbd0: Connection established.
drbd0: I am(P): 1:00000004:00000001:00000044:00000002:10
drbd0: Peer(S): 1:00000004:00000001:00000043:00000002:01
drbd0: drbd0_receiver [1139]: cstate WFReportParams --> WFBitMapS
drbd0: Primary/Unknown --> Primary/Secondary
drbd0: drbd0_receiver [1139]: cstate WFBitMapS --> SyncSource
drbd0: Resync started as SyncSource (need to sync 936 KB [234 bits set]).
drbd0: Resync done (total 1 sec; paused 0 sec; 936 K/sec)
drbd0: drbd0_worker [6997]: cstate SyncSource --> Connected
Lustre: mds01: haven't heard from 10.65.200.45 at tcp in 253 seconds. Last request was at 1294236586. I think it's dead, and I am evicting it.
LustreError: 1665:0:(mds_open.c:1455:mds_close()) @@@ no handle for file close ino 100242438: cookie 0x7bfc4281d33d729e req at e18acc00 x936095341/t0 o35->92e82
_MNT_client_8029478b2f at NET_0x200000a41c82d_UUID:-1 lens 240/392 ref 0 fl Interpret:/0/0 rc 0/0
LustreError: 1665:0:(mds_open.c:1455:mds_close()) Skipped 1 previous similar message
LustreError: 1665:0:(ldlm_lib.c:1318:target_send_reply_msg()) @@@ processing error (-116) req at e18acc00 x936095341/t0 o35->92e82_MNT_client_8029478b2f at NET_0x2
00000a41c82d_UUID:-1 lens 240/392 ref 0 fl Interpret:/0/0 rc -116/0
LustreError: 1665:0:(ldlm_lib.c:1318:target_send_reply_msg()) Skipped 1 previous similar message
LustreError: 1650:0:(mds_open.c:1455:mds_close()) @@@ no handle for file close ino 99550273: cookie 0x7bfc4281d33d778a req at c2289e00 x936095393/t0 o35->92e82_
MNT_client_8029478b2f at NET_0x200000a41c82d_UUID:-1 lens 240/392 ref 0 fl Interpret:/0/0 rc 0/0
LustreError: 1650:0:(mds_open.c:1455:mds_close()) Skipped 1 previous similar message
LustreError: 1650:0:(ldlm_lib.c:1318:target_send_reply_msg()) @@@ processing error (-116) req at c2289e00 x936095393/t0 o35->92e82_MNT_client_8029478b2f at NET_0x2
00000a41c82d_UUID:-1 lens 240/392 ref 0 fl Interpret:/0/0 rc -116/0
LustreError: 1650:0:(ldlm_lib.c:1318:target_send_reply_msg()) Skipped 1 previous similar message
LustreError: 1658:0:(mds_open.c:1455:mds_close()) @@@ no handle for file close ino 99714850: cookie 0x7bfc4281d33d7b57 req at c221842c x936095417/t0 o35->92e82_
MNT_client_8029478b2f at NET_0x200000a41c82d_UUID:-1 lens 240/392 ref 0 fl Interpret:/0/0 rc 0/0
LustreError: 1658:0:(mds_open.c:1455:mds_close()) Skipped 1 previous similar message
LustreError: 1658:0:(ldlm_lib.c:1318:target_send_reply_msg()) @@@ processing error (-116) req at c221842c x936095417/t0 o35->92e82_MNT_client_8029478b2f at NET_0x2
00000a41c82d_UUID:-1 lens 240/392 ref 0 fl Interpret:/0/0 rc -116/0
LustreError: 1658:0:(ldlm_lib.c:1318:target_send_reply_msg()) Skipped 1 previous similar message
LustreError: 1661:0:(mds_open.c:1455:mds_close()) @@@ no handle for file close ino 100139209: cookie 0x7bfc4281d33d7bff req at e1f4ec00 x936095444/t0 o35->92e82
_MNT_client_8029478b2f at NET_0x200000a41c82d_UUID:-1 lens 240/392 ref 0 fl Interpret:/0/0 rc 0/0
LustreError: 1661:0:(mds_open.c:1455:mds_close()) Skipped 1 previous similar message
LustreError: 1661:0:(ldlm_lib.c:1318:target_send_reply_msg()) @@@ processing error (-116) req at e1f4ec00 x936095444/t0 o35->92e82_MNT_client_8029478b2f at NET_0x2
00000a41c82d_UUID:-1 lens 240/392 ref 0 fl Interpret:/0/0 rc -116/0
LustreError: 1661:0:(ldlm_lib.c:1318:target_send_reply_msg()) Skipped 1 previous similar message
LustreError: 1646:0:(mds_open.c:1455:mds_close()) @@@ no handle for file close ino 100143301: cookie 0x7bfc4281d33d7c14 req at e1832200 x936095446/t0 o35->92e82
_MNT_client_8029478b2f at NET_0x200000a41c82d_UUID:-1 lens 240/392 ref 0 fl Interpret:/0/0 rc 0/0
LustreError: 1646:0:(mds_open.c:1455:mds_close()) Skipped 1 previous similar message
LustreError: 1646:0:(ldlm_lib.c:1318:target_send_reply_msg()) @@@ processing error (-116) req at e1832200 x936095446/t0 o35->92e82_MNT_client_8029478b2f at NET_0x2
00000a41c82d_UUID:-1 lens 240/392 ref 0 fl Interpret:/0/0 rc -116/0
LustreError: 1646:0:(ldlm_lib.c:1318:target_send_reply_msg()) Skipped 1 previous similar message
LustreError: 1646:0:(mds_open.c:1455:mds_close()) @@@ no handle for file close ino 100238688: cookie 0x7bfc4281d33d7e05 req at f7cd5200 x936095461/t0 o35->92e82
_MNT_client_8029478b2f at NET_0x200000a41c82d_UUID:-1 lens 240/392 ref 0 fl Interpret:/0/0 rc 0/0
LustreError: 1646:0:(ldlm_lib.c:1318:target_send_reply_msg()) @@@ processing error (-116) req at f7cd5200 x936095461/t0 o35->92e82_MNT_client_8029478b2f at NET_0x2
00000a41c82d_UUID:-1 lens 240/392 ref 0 fl Interpret:/0/0 rc -116/0
LustreError: 1671:0:(mds_open.c:1455:mds_close()) @@@ no handle for file close ino 100336631: cookie 0x7bfc4281d33d6e29 req at f7cc8400 x936095478/t0 o35->92e82
_MNT_client_8029478b2f at NET_0x200000a41c82d_UUID:-1 lens 240/392 ref 0 fl Interpret:/0/0 rc 0/0
LustreError: 1671:0:(mds_open.c:1455:mds_close()) Skipped 1 previous similar message
LustreError: 1671:0:(ldlm_lib.c:1318:target_send_reply_msg()) @@@ processing error (-116) req at f7cc8400 x936095478/t0 o35->92e82_MNT_client_8029478b2f at NET_0x2
00000a41c82d_UUID:-1 lens 240/392 ref 0 fl Interpret:/0/0 rc -116/0
LustreError: 1671:0:(ldlm_lib.c:1318:target_send_reply_msg()) Skipped 1 previous similar message
LustreError: 1663:0:(mds_open.c:1455:mds_close()) @@@ no handle for file close ino 99781569: cookie 0x7bfc4281d33d7cb5 req at e18bb800 x936095491/t0 o35->92e82_
MNT_client_8029478b2f at NET_0x200000a41c82d_UUID:-1 lens 240/392 ref 0 fl Interpret:/0/0 rc 0/0
LustreError: 1663:0:(mds_open.c:1455:mds_close()) Skipped 1 previous similar message
LustreError: 1663:0:(ldlm_lib.c:1318:target_send_reply_msg()) @@@ processing error (-116) req at e18bb800 x936095491/t0 o35->92e82_MNT_client_8029478b2f at NET_0x2
00000a41c82d_UUID:-1 lens 240/392 ref 0 fl Interpret:/0/0 rc -116/0
LustreError: 1663:0:(ldlm_lib.c:1318:target_send_reply_msg()) Skipped 1 previous similar message
LustreError: 1646:0:(mds_open.c:1455:mds_close()) @@@ no handle for file close ino 99793390: cookie 0x7bfc4281d33d7cca req at e18b5400 x936095499/t0 o35->92e82_
MNT_client_8029478b2f at NET_0x200000a41c82d_UUID:-1 lens 240/392 ref 0 fl Interpret:/0/0 rc 0/0
LustreError: 1646:0:(mds_open.c:1455:mds_close()) Skipped 3 previous similar messages
LustreError: 1646:0:(ldlm_lib.c:1318:target_send_reply_msg()) @@@ processing error (-116) req at e18b5400 x936095499/t0 o35->92e82_MNT_client_8029478b2f at NET_0x2
00000a41c82d_UUID:-1 lens 240/392 ref 0 fl Interpret:/0/0 rc -116/0
LustreError: 1657:0:(mds_open.c:1455:mds_close()) @@@ no handle for file close ino 100336520: cookie 0x7bfc4281d33d7a07 req at e1832200 x936095531/t0 o35->92e82
_MNT_client_8029478b2f at NET_0x200000a41c82d_UUID:-1 lens 240/392 ref 0 fl Interpret:/0/0 rc 0/0
LustreError: 1657:0:(ldlm_lib.c:1318:target_send_reply_msg()) @@@ processing error (-116) req at e1832200 x936095531/t0 o35->92e82_MNT_client_8029478b2f at NET_0x2
00000a41c82d_UUID:-1 lens 240/392 ref 0 fl Interpret:/0/0 rc -116/0
LustreError: 1662:0:(mds_open.c:1455:mds_close()) @@@ no handle for file close ino 99784193: cookie 0x7bfc4281d33d7584 req at c2296e00 x936095549/t0 o35->92e82_
MNT_client_8029478b2f at NET_0x200000a41c82d_UUID:-1 lens 240/392 ref 0 fl Interpret:/0/0 rc 0/0
LustreError: 1662:0:(mds_open.c:1455:mds_close()) Skipped 1 previous similar message
LustreError: 1662:0:(ldlm_lib.c:1318:target_send_reply_msg()) @@@ processing error (-116) req at c2296e00 x936095549/t0 o35->92e82_MNT_client_8029478b2f at NET_0x2
00000a41c82d_UUID:-1 lens 240/392 ref 0 fl Interpret:/0/0 rc -116/0
LustreError: 1662:0:(ldlm_lib.c:1318:target_send_reply_msg()) Skipped 1 previous similar message
LustreError: 1662:0:(mds_open.c:1455:mds_close()) @@@ no handle for file close ino 99946358: cookie 0x7bfc4281d33d6f79 req at e18bb200 x936095566/t0 o35->92e82_
MNT_client_8029478b2f at NET_0x200000a41c82d_UUID:-1 lens 240/392 ref 0 fl Interpret:/0/0 rc 0/0
LustreError: 1662:0:(mds_open.c:1455:mds_close()) Skipped 1 previous similar message
LustreError: 1662:0:(ldlm_lib.c:1318:target_send_reply_msg()) @@@ processing error (-116) req at e18bb200 x936095566/t0 o35->92e82_MNT_client_8029478b2f at NET_0x2
00000a41c82d_UUID:-1 lens 240/392 ref 0 fl Interpret:/0/0 rc -116/0
LustreError: 1662:0:(ldlm_lib.c:1318:target_send_reply_msg()) Skipped 1 previous similar message
LustreError: 1649:0:(mds_open.c:1455:mds_close()) @@@ no handle for file close ino 99782610: cookie 0x7bfc4281d33d7036 req at c22af600 x936095603/t0 o35->92e82_
MNT_client_8029478b2f at NET_0x200000a41c82d_UUID:-1 lens 240/392 ref 0 fl Interpret:/0/0 rc 0/0
LustreError: 1649:0:(mds_open.c:1455:mds_close()) Skipped 3 previous similar messages
LustreError: 1649:0:(ldlm_lib.c:1318:target_send_reply_msg()) @@@ processing error (-116) req at c22af600 x936095603/t0 o35->92e82_MNT_client_8029478b2f at NET_0x2
00000a41c82d_UUID:-1 lens 240/392 ref 0 fl Interpret:/0/0 rc -116/0
LustreError: 1649:0:(ldlm_lib.c:1318:target_send_reply_msg()) Skipped 3 previous similar messages
LustreError: 1674:0:(mds_open.c:1455:mds_close()) @@@ no handle for file close ino 99586302: cookie 0x7bfc4281d33d7426 req at c2289e00 x936095620/t0 o35->92e82_
MNT_client_8029478b2f at NET_0x200000a41c82d_UUID:-1 lens 240/392 ref 0 fl Interpret:/0/0 rc 0/0
LustreError: 1674:0:(mds_open.c:1455:mds_close()) Skipped 1 previous similar message
LustreError: 1674:0:(ldlm_lib.c:1318:target_send_reply_msg()) @@@ processing error (-116) req at c2289e00 x936095620/t0 o35->92e82_MNT_client_8029478b2f at NET_0x2
00000a41c82d_UUID:-1 lens 240/392 ref 0 fl Interpret:/0/0 rc -116/0
LustreError: 1674:0:(ldlm_lib.c:1318:target_send_reply_msg()) Skipped 1 previous similar message
LustreError: 1643:0:(mds_open.c:1455:mds_close()) @@@ no handle for file close ino 99883919: cookie 0x7bfc4281d33d8552 req at c2264c00 x936095638/t0 o35->92e82_
MNT_client_8029478b2f at NET_0x200000a41c82d_UUID:-1 lens 240/392 ref 0 fl Interpret:/0/0 rc 0/0
LustreError: 1643:0:(mds_open.c:1455:mds_close()) Skipped 2 previous similar messages
LustreError: 1643:0:(ldlm_lib.c:1318:target_send_reply_msg()) @@@ processing error (-116) req at c2264c00 x936095638/t0 o35->92e82_MNT_client_8029478b2f at NET_0x2
00000a41c82d_UUID:-1 lens 240/392 ref 0 fl Interpret:/0/0 rc -116/0
LustreError: 1643:0:(ldlm_lib.c:1318:target_send_reply_msg()) Skipped 2 previous similar messages
LustreError: 1669:0:(mds_open.c:1455:mds_close()) @@@ no handle for file close ino 99850165: cookie 0x7bfc4281d33d6bdd req at c2264e00 x936095798/t0 o35->92e82_
MNT_client_8029478b2f at NET_0x200000a41c82d_UUID:-1 lens 240/392 ref 0 fl Interpret:/0/0 rc 0/0
LustreError: 1669:0:(mds_open.c:1455:mds_close()) Skipped 4 previous similar messages
LustreError: 1669:0:(ldlm_lib.c:1318:target_send_reply_msg()) @@@ processing error (-116) req at c2264e00 x936095798/t0 o35->92e82_MNT_client_8029478b2f at NET_0x2
00000a41c82d_UUID:-1 lens 240/392 ref 0 fl Interpret:/0/0 rc -116/0
LustreError: 1669:0:(ldlm_lib.c:1318:target_send_reply_msg()) Skipped 4 previous similar messages
LustreError: 1664:0:(mds_open.c:1455:mds_close()) @@@ no handle for file close ino 100208138: cookie 0x7bfc4281d16f1663 req at e18a7800 x936098100/t0 o35->92e82
_MNT_client_8029478b2f at NET_0x200000a41c82d_UUID:-1 lens 240/392 ref 0 fl Interpret:/0/0 rc 0/0
LustreError: 1664:0:(mds_open.c:1455:mds_close()) Skipped 3 previous similar messages
LustreError: 1664:0:(ldlm_lib.c:1318:target_send_reply_msg()) @@@ processing error (-116) req at e18a7800 x936098100/t0 o35->92e82_MNT_client_8029478b2f at NET_0x2
00000a41c82d_UUID:-1 lens 240/392 ref 0 fl Interpret:/0/0 rc -116/0
LustreError: 1664:0:(ldlm_lib.c:1318:target_send_reply_msg()) Skipped 3 previous similar messages
LustreError: 1645:0:(mds_open.c:1455:mds_close()) @@@ no handle for file close ino 100340007: cookie 0x7bfc4281d17d91ed req at c2259200 x936107868/t0 o35->92e82
_MNT_client_8029478b2f at NET_0x200000a41c82d_UUID:-1 lens 240/392 ref 0 fl Interpret:/0/0 rc 0/0
LustreError: 1645:0:(ldlm_lib.c:1318:target_send_reply_msg()) @@@ processing error (-116) req at c2259200 x936107868/t0 o35->92e82_MNT_client_8029478b2f at NET_0x2
00000a41c82d_UUID:-1 lens 240/392 ref 0 fl Interpret:/0/0 rc -116/0
LustreError: 1674:0:(mds_open.c:1455:mds_close()) @@@ no handle for file close ino 99550113: cookie 0x7bfc4281d16a1a87 req at e1f4ee00 x936108706/t0 o35->92e82_
MNT_client_8029478b2f at NET_0x200000a41c82d_UUID:-1 lens 240/392 ref 0 fl Interpret:/0/0 rc 0/0
LustreError: 1674:0:(ldlm_lib.c:1318:target_send_reply_msg()) @@@ processing error (-116) req at e1f4ee00 x936108706/t0 o35->92e82_MNT_client_8029478b2f at NET_0x2
00000a41c82d_UUID:-1 lens 240/392 ref 0 fl Interpret:/0/0 rc -116/0
LustreError: 1538:0:(acceptor.c:422:lnet_acceptor()) Refusing connection from 127.0.0.1: insecure port 37757
LustreError: 1657:0:(mds_open.c:1455:mds_close()) @@@ no handle for file close ino 99794102: cookie 0x7bfc4281d0b4b154 req at f7d42200 x941376814/t0 o35->92e82_
MNT_client_8029478b2f at NET_0x200000a41c82d_UUID:-1 lens 240/392 ref 0 fl Interpret:/0/0 rc 0/0
LustreError: 1657:0:(ldlm_lib.c:1318:target_send_reply_msg()) @@@ processing error (-116) req at f7d42200 x941376814/t0 o35->92e82_MNT_client_8029478b2f at NET_0x2
00000a41c82d_UUID:-1 lens 240/392 ref 0 fl Interpret:/0/0 rc -116/0
LustreError: 1658:0:(mds_open.c:1455:mds_close()) @@@ no handle for file close ino 99713568: cookie 0x7bfc4281d15a5658 req at f7ce4c00 x943537900/t0 o35->92e82_
MNT_client_8029478b2f at NET_0x200000a41c82d_UUID:-1 lens 240/392 ref 0 fl Interpret:/0/0 rc 0/0
LustreError: 1658:0:(ldlm_lib.c:1318:target_send_reply_msg()) @@@ processing error (-116) req at f7ce4c00 x943537900/t0 o35->92e82_MNT_client_8029478b2f at NET_0x2
00000a41c82d_UUID:-1 lens 240/392 ref 0 fl Interpret:/0/0 rc -116/0
Primary OSS
LustreError: dumping log to /tmp/lustre-log-typhoon.exampe.com.1296015719.1704
Lustre: 0:0:(watchdog.c:130:lcw_cb()) Watchdog triggered for pid 1702: it was inactive for 100000ms
Lustre: 0:0:(watchdog.c:130:lcw_cb()) Skipped 1 previous similar message
Lustre: 0:0:(linux-debug.c:166:libcfs_debug_dumpstack()) showing stack for process 1702
Lustre: 0:0:(linux-debug.c:166:libcfs_debug_dumpstack()) Skipped 1 previous similar message
ll_ost_io_30 S 00000001 4156 1702 1 1703 1701 (L-TLB)
dbbf5c10 00000046 00000000 00000001 ffffffff ffffffff 00000000 00000000
00000002 00000000 d42bf7c0 c201ade0 00000000 00000000 d5e3f580 0056cb5d
c0331a80 dbbd6cf0 dbbd6e5c 00000000 00000246 b02ca346 b02ca346 ffffffff
Call Trace:
[<c02dfaa9>] schedule_timeout+0x139/0x154
[<c01296a4>] process_timeout+0x0/0x5
[<c011f2f9>]ll_ost_io_03 S 00000001 4028 1675 1 1676 1674 (L-TLB)
add_wait_queue+0x12/0x30
[<f930fb61>] ldlm_completion_ast+0x4bd/0x993 [ptlrpc]
[<f930f045>] ldlm_process_extent_lock+0x4df/0x63b [ptlrpc]
[<c011d6f8>]de2cdc10 00000046 00000000 00000001 ffffffff ffffffff 00000000 default_wake_function+0x0/0xc
[<f92f815f>] l_unlock+0xab/0xc4 [ptlrpc]
[<f930f42d>] ldlm_expired_completion_wait+0x0/0x277 [ptlrpc]
[<f930f42c>] interrupted_completion_wait+0x0/0x1 [ptlrpc]
[<f930f6a4>] ldlm_completion_ast+0x0/0x993 [ptlrpc]
[<f931091c>] ldlm_cli_enqueue_local+0x4b4/0x5ce [ptlrpc]
[<f9c9fcb6>] filter_prepare_destroy+0x11a/0x1c7 [obdfilter]
[<f9310037>] ldlm_blocking_ast+0x0/0x42b [ptlrpc]
[<f930f6a4>] ldlm_completion_ast+0x0/0x993 [ptlrpc]
[<f9cac988>] filter_destroy+0x36d/0x184a [obdfilter]
[<f8c2d62f>] lnet_ni_send+0x70/0x88 [lnet]
[<f8c31c93>] LNetPut+0x838/0x8ea [lnet]
[<f93a0704>] obd_destroy+0x3ef/0x484 [ost]
[<f93a0280>] ost_destroy+0x236/0x2cb [ost]
[<f93accaf>] ost_handle+0xfeb/0x383f [ost]
[<f9339841>] ptlrpc_update_export_timer+0x233/0x454 [ptlrpc]
[<f933a4d4>] ptlrpc_server_handle_request+0xa72/0x1204 [ptlrpc]
[<f933bb8a>] ptlrpc_main+0x827/0x9e9 [ptlrpc]
[<c011d6f8>] default_wake_function+0x0/0xc
[<f933b356>] ptlrpc_retry_rqbds+0x0/0xd [ptlrpc]
[<c02e123e>]00000000
00000002 00000000 ee7deac0 c2022de0 00000001 00000000 d5f337c0 0056cb5d
c22710b0 de2c36b0 de2c381c 00000000 00000246 b02ca347 b02ca347 ffffffff
Call Trace:
[<c02dfaa9>] ret_from_fork+0x6/0x14
[<f933b356>] ptlrpc_retry_rqbds+0x0/0xd [ptlrpc]
[<f933b363>] ptlrpc_main+0x0/0x9e9 [ptlrpc]
[<c01041f5>] kernel_thread_helper+0x5/0xb
LustreError: 1702:0:(ldlm_request.c:59:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1296015621, 100s ago); not entering recovery in server code, just going back to sleep ns: filter-ost1_UUID lock: d42bf7c0/0x64a0224008967602 lrc: 3/0,1 mode: --/PW res: 28303634/0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 80004000 remote: 0x0 expref: -99 pid: 1702
LustreError: 1702:0:(ldlm_request.c:59:ldlm_expired_completion_wait()) Skipped 1 previous similar message
schedule_timeout+0x139/0x154
[<c01296a4>] process_timeout+0x0/0x5
[<c011f2f9>] add_wait_queue+0x12/0x30
[<f930fb61>] ldlm_completion_ast+0x4bd/0x993 [ptlrpc]
[<f930f045>] ldlm_process_extent_lock+0x4df/0x63b [ptlrpc]
[<c011d6f8>] default_wake_function+0x0/0xc
[<f92f815f>] l_unlock+0xab/0xc4 [ptlrpc]
[<f930f42d>] ldlm_expired_completion_wait+0x0/0x277 [ptlrpc]
[<f930f42c>] interrupted_completion_wait+0x0/0x1 [ptlrpc]
[<f930f6a4>] ldlm_completion_ast+0x0/0x993 [ptlrpc]
[<f931091c>] ldlm_cli_enqueue_local+0x4b4/0x5ce [ptlrpc]
[<f9c9fcb6>] filter_prepare_destroy+0x11a/0x1c7 [obdfilter]
[<f9310037>] ldlm_blocking_ast+0x0/0x42b [ptlrpc]
[<f930f6a4>] ldlm_completion_ast+0x0/0x993 [ptlrpc]
[<f9cac988>] filter_destroy+0x36d/0x184a [obdfilter]
[<f8c2d62f>] lnet_ni_send+0x70/0x88 [lnet]
[<f8c31c93>] LNetPut+0x838/0x8ea [lnet]
[<f93a0704>] obd_destroy+0x3ef/0x484 [ost]
[<f93a0280>] ost_destroy+0x236/0x2cb [ost]
[<f93accaf>] ost_handle+0xfeb/0x383f [ost]
[<f9339841>] ptlrpc_update_export_timer+0x233/0x454 [ptlrpc]
[<f933a4d4>] ptlrpc_server_handle_request+0xa72/0x1204 [ptlrpc]
[<f933bb8a>] ptlrpc_main+0x827/0x9e9 [ptlrpc]
[<c011d6f8>] default_wake_function+0x0/0xc
[<f933b356>] ptlrpc_retry_rqbds+0x0/0xd [ptlrpc]
[<c02e123e>]<1>LustreError: dumping log to /tmp/lustre-log-typhoon.exampe.com.1296015721.1702
ret_from_fork+0x6/0x14
[<f933b356>] ptlrpc_retry_rqbds+0x0/0xd [ptlrpc]
ll_ost_io_11 S 00000001 4196 1683 1 1684 1682 (L-TLB)
de317c10 00000046 00000000 00000001 ffffffff ffffffff 00000000 00000000
00000002 00000000 dead5d40 c201ade0 00000000 00000000 d64ec540 0056cb5d
c0331a80 de2f6670 de2f67dc 00000000 00000246 b02ca34d b02ca34d ffffffff
Call Trace:
[<c02dfaa9>] [<f933b363>] ptlrpc_main+0x0/0x9e9 [ptlrpc]
[<c01041f5>] kernel_thread_helper+0x5/0xb
schedule_timeout+0x139/0x154
[<c01296a4>] process_timeout+0x0/0x5
[<c011f2f9>] add_wait_queue+0x12/0x30
[<f930fb61>] ldlm_completion_ast+0x4bd/0x993 [ptlrpc]
[<f930f045>] ldlm_process_extent_lock+0x4df/0x63b [ptlrpc]
[<c011d6f8>] default_wake_function+0x0/0xc
[<f92f815f>] l_unlock+0xab/0xc4 [ptlrpc]
[<f930f42d>] ldlm_expired_completion_wait+0x0/0x277 [ptlrpc]
[<f930f42c>] interrupted_completion_wait+0x0/0x1 [ptlrpc]
[<f930f6a4>] ldlm_completion_ast+0x0/0x993 [ptlrpc]
[<f931091c>] ldlm_cli_enqueue_local+0x4b4/0x5ce [ptlrpc]
[<f9c9fcb6>] filter_prepare_destroy+0x11a/0x1c7 [obdfilter]
[<f9310037>] ldlm_blocking_ast+0x0/0x42b [ptlrpc]
[<f930f6a4>] ldlm_completion_ast+0x0/0x993 [ptlrpc]
[<f9cac988>] filter_destroy+0x36d/0x184a [obdfilter]
[<f8c2d62f>] lnet_ni_send+0x70/0x88 [lnet]
[<f8c31c93>] LNetPut+0x838/0x8ea [lnet]
[<f93a0704>] obd_destroy+0x3ef/0x484 [ost]
[<f93a0280>] ost_destroy+0x236/0x2cb [ost]
[<f93accaf>] ost_handle+0xfeb/0x383f [ost]
[<f9339841>] ptlrpc_update_export_timer+0x233/0x454 [ptlrpc]
[<f933a4d4>] ptlrpc_server_handle_request+0xa72/0x1204 [ptlrpc]
[<f933bb8a>] ptlrpc_main+0x827/0x9e9 [ptlrpc]
[<c011d6f8>] default_wake_function+0x0/0xc
[<f933b356>] ptlrpc_retry_rqbds+0x0/0xd [ptlrpc]
[<c02e123e>] ret_from_fork+0x6/0x14
[<f933b356>] ptlrpc_retry_rqbds+0x0/0xd [ptlrpc]
[<f933b363>] ptlrpc_main+0x0/0x9e9 [ptlrpc]
[<c01041f5>] kernel_thread_helper+0x5/0xb
LustreError: dumping log to /tmp/lustre-log-typhoon.exampe.com.1296015721.1683
LustreError: dumping log to /tmp/lustre-log-typhoon.exampe.com.1296015721.1675
LustreError: 1655:0:(ldlm_lib.c:1318:target_send_reply_msg()) @@@ processing error (-16) req at e9cb2600 x226166541/t0 o8->c2914_lov1_153086c94b at NET_0x200000a41c825_UUID:-1 lens 240/144 ref 0 fl Interpret:/0/0 rc -16/0
LustreError: 1655:0:(ldlm_lib.c:1318:target_send_reply_msg()) Skipped 1 previous similar message
LustreError: 1659:0:(ldlm_lib.c:557:target_handle_connect()) @@@ UUID 'ost2_UUID' is not available for connect (no target) req at f182e600 x226166544/t0 o8-><?>@<?>:-1 lens 240/0 ref 0 fl Interpret:/0/0 rc 0/0
LustreError: 0:0:(ldlm_lockd.c:205:waiting_locks_callback()) ### lock callback timer expired: evicting client c2914_lov1_153086c94b at NET_0x200000a41c825_UUID nid 10.65.200.37 at tcp ns: filter-ost1_UUID lock: e9a1c180/0x64a0224008966edf lrc: 1/0,0 mode: PW/PW res: 28303634/0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) flags: 10020 remote: 0xfbc2b603d4a9afd2 expref: 1976 pid: 1625
LustreError: 1704:0:(service.c:654:ptlrpc_server_handle_request()) request 226164089 opc 6 from 12345-10.65.200.37 at tcp processed in 106s trans 49298371 rc 0/0
Lustre: 1704:0:(watchdog.c:311:lcw_update_time()) Expired watchdog for pid 1704 disabled after 106098642.0001s
Lustre: 1704:0:(watchdog.c:311:lcw_update_time()) Skipped 1 previous similar message
LustreError: 1556:0:(ldlm_lockd.c:1056:ldlm_handle_cancel()) received cancel for unknown lock cookie 0x64a0224008966f72 from client c2914_lov1_153086c94b id 12345-10.65.200.37 at tcp
LustreError: 1556:0:(ldlm_lockd.c:1056:ldlm_handle_cancel()) Skipped 1 previous similar message
LustreError: 1722:0:(filter_io.c:532:filter_preprw_write()) ost1: trying to BRW to non-existent file 28303634
LustreError: 1538:0:(ldlm_lockd.c:1056:ldlm_handle_cancel()) received cancel for unknown lock cookie 0x64a0224008966afd from client c2914_lov1_153086c94b id 12345-10.65.200.37 at tcp
LustreError: 1546:0:(ldlm_lockd.c:1056:ldlm_handle_cancel()) received cancel for unknown lock cookie 0x64a0224008966edf from client c2914_lov1_153086c94b id 12345-10.65.200.37 at tcp
LustreError: 1546:0:(ldlm_lockd.c:1056:ldlm_handle_cancel()) Skipped 1 previous similar message
LustreError: 1609:0:(ldlm_lib.c:557:target_handle_connect()) @@@ UUID 'ost2_UUID' is not available for connect (no target) req at de92ba00 x226305667/t0 o8-><?>@<?>:-1 lens 240/0 ref 0 fl Interpret:/0/0 rc 0/0
LustreError: 1609:0:(ldlm_lib.c:1318:target_send_reply_msg()) @@@ processing error (-19) req at de92ba00 x226305667/t0 o8-><?>@<?>:-1 lens 240/0 ref 0 fl Interpret:/0/0 rc -19/0
LustreError: 1609:0:(ldlm_lib.c:1318:target_send_reply_msg()) Skipped 1 previous similar message
Lustre: 1470:0:(lib-move.c:1644:lnet_parse_put()) Dropping PUT from 12345-10.65.200.37 at tcp portal 16 match 10625151 offset 0 length 64: 2
Lustre: 1470:0:(lib-move.c:1644:lnet_parse_put()) Dropping PUT from 12345-10.65.200.37 at tcp portal 16 match 10625153 offset 0 length 64: 2
LustreError: 1636:0:(ldlm_lib.c:557:target_handle_connect()) @@@ UUID 'ost2_UUID' is not available for connect (no target) req at c5ef1200 x226306267/t0 o8-><?>@<?>:-1 lens 240/0 ref 0 fl Interpret:/0/0 rc 0/0
LustreError: 1636:0:(ldlm_lib.c:1318:target_send_reply_msg()) @@@ processing error (-19) req at c5ef1200 x226306267/t0 o8-><?>@<?>:-1 lens 240/0 ref 0 fl Interpret:/0/0 rc -19/0
LustreError: 1563:0:(ldlm_lockd.c:584:ldlm_server_completion_ast()) ### enqueue wait took 162499891us from 1296018348 ns: filter-ost1_UUID lock: c4248040/0x64a0224008993a36 lrc: 2/0,0 mode: PW/PW res: 28132994/0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0 remote: 0xfbc2b603d54bef71 expref: 2511 pid: 1640
LustreError: 1566:0:(ldlm_lockd.c:584:ldlm_server_completion_ast()) ### enqueue wait took 117534968us from 1296025387 ns: filter-ost1_UUID lock: e5233bc0/0x64a02240089dd1f0 lrc: 2/0,0 mode: PR/PR res: 28145386/0 rrc: 8 type: EXT [0->18446744073709551615] (req 0->8191) flags: 0 remote: 0x943ed9a0643337b0 expref: 2553 pid: 1626
LustreError: 1566:0:(ldlm_lockd.c:584:ldlm_server_completion_ast()) ### enqueue wait took 118528501us from 1296025386 ns: filter-ost1_UUID lock: ceb2a640/0x64a02240089dd1db lrc: 2/0,0 mode: PR/PR res: 28145386/0 rrc: 8 type: EXT [0->18446744073709551615] (req 4096->8191) flags: 0 remote: 0x943ed9a064330123 expref: 2553 pid: 1614
LustreError: 1566:0:(ldlm_lockd.c:584:ldlm_server_completion_ast()) ### enqueue wait took 161988593us from 1296025342 ns: filter-ost1_UUID lock: c46ba3c0/0x64a02240089dcb4b lrc: 2/0,0 mode: PR/PR res: 28145386/0 rrc: 8 type: EXT [0->18446744073709551615] (req 4096->8191) flags: 0 remote: 0xc4d09fb494319a1e expref: 2549 pid: 1654
LustreError: 1566:0:(ldlm_lockd.c:584:ldlm_server_completion_ast()) Skipped 3 previous similar messages
LustreError: 1551:0:(ldlm_lockd.c:584:ldlm_server_completion_ast()) ### enqueue wait took 103243702us from 1296025522 ns: filter-ost1_UUID lock: ef5628c0/0x64a02240089dede9 lrc: 2/0,0 mode: PW/PW res: 28327121/0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) flags: 0 remote: 0x943ed9a0643d689c expref: 2549 pid: 1609
LustreError: 1550:0:(ldlm_lockd.c:584:ldlm_server_completion_ast()) ### enqueue wait took 111169666us from 1296025557 ns: filter-ost1_UUID lock: e2926b40/0x64a02240089df376 lrc: 2/0,0 mode: PR/PR res: 28145386/0 rrc: 11 type: EXT [0->18446744073709551615] (req 0->8191) flags: 0 remote: 0xc4d09fb4943eb436 expref: 2538 pid: 1626
Lustre: 0:0:(watchdog.c:130:lcw_cb()) Watchdog triggered for pid 1691: it was inactive for 100000ms
Lustre: 0:0:(watchdog.c:130:lcw_cb()) Skipped 2 previous similar messages
Lustre: 0:0:(linux-debug.c:166:libcfs_debug_dumpstack()) showing stack for process 1691
Lustre: 0:0:(linux-debug.c:166:libcfs_debug_dumpstack()) Skipped 2 previous similar messages
ll_ost_io_19 S 00000001 4180 1691 1 1692 1690 (L-TLB)
f73ebc10 00000046 00000000 00000001 ffffffff ffffffff 00000000 00000000
00000002 00000000 eaa9f480 c2022de0 00000001 00000000 a504d000 0056d473
c22710b0 f73df1f0 f73df35c 00000000 00000246 b0c50fe0 b0c50fe0 ffffffff
Call Trace:
[<c02dfaa9>] schedule_timeout+0x139/0x154
[<c01296a4>] process_timeout+0x0/0x5
[<c011f2f9>] add_wait_queue+0x12/0x30
[<f930fb61>] ldlm_completion_ast+0x4bd/0x993 [ptlrpc]
[<f930f045>] ldlm_process_extent_lock+0x4df/0x63b [ptlrpc]
[<c011d6f8>] default_wake_function+0x0/0xc
[<f92f815f>] l_unlock+0xab/0xc4 [ptlrpc]
[<f930f42d>] ldlm_expired_completion_wait+0x0/0x277 [ptlrpc]
[<f930f42c>] interrupted_completion_wait+0x0/0x1 [ptlrpc]
[<f930f6a4>] ldlm_completion_ast+0x0/0x993 [ptlrpc]
[<f931091c>] ldlm_cli_enqueue_local+0x4b4/0x5ce [ptlrpc]
[<f9c9fcb6>] filter_prepare_destroy+0x11a/0x1c7 [obdfilter]
[<f9310037>] ldlm_blocking_ast+0x0/0x42b [ptlrpc]
[<f930f6a4>] ldlm_completion_ast+0x0/0x993 [ptlrpc]
[<f9cac988>] filter_destroy+0x36d/0x184a [obdfilter]
[<f93306ec>] ptlrpc_send_reply+0x38b/0x392 [ptlrpc]
[<f93a8cbd>] ost_brw_write+0x224a/0x2418 [ost]
[<f93a0704>] obd_destroy+0x3ef/0x484 [ost]
[<f93a0280>] ost_destroy+0x236/0x2cb [ost]
[<f93accaf>] ost_handle+0xfeb/0x383f [ost]
[<f9339841>] ptlrpc_update_export_timer+0x233/0x454 [ptlrpc]
[<f933a4d4>] ptlrpc_server_handle_request+0xa72/0x1204 [ptlrpc]
[<f933bb8a>] ptlrpc_main+0x827/0x9e9 [ptlrpc]
[<c011d6f8>] default_wake_function+0x0/0xc
[<f933b356>] ptlrpc_retry_rqbds+0x0/0xd [ptlrpc]
[<c02e123e>] ret_from_fork+0x6/0x14
[<f933b356>] ptlrpc_retry_rqbds+0x0/0xd [ptlrpc]
[<f933b363>] ptlrpc_main+0x0/0x9e9 [ptlrpc]
[<c01041f5>] kernel_thread_helper+0x5/0xb
LustreError: dumping log to /tmp/lustre-log-typhoon.exampe.com.1296025710.1691
LustreError: 1691:0:(ldlm_request.c:59:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1296025610, 100s ago); not entering recovery in server code, just going back to sleep ns: filter-ost1_UUID lock: eaa9f480/0x64a02240089dfe97 lrc: 3/0,1 mode: --/PW res: 28327587/0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 80004000 remote: 0x0 expref: -99 pid: 1691
LustreError: 1691:0:(ldlm_request.c:59:ldlm_expired_completion_wait()) Skipped 2 previous similar messages
LustreError: 1654:0:(ldlm_lib.c:557:target_handle_connect()) @@@ UUID 'ost2_UUID' is not available for connect (no target) req at e22b0e00 x904040/t0 o8-><?>@<?>:-1 lens 240/0 ref 0 fl Interpret:/0/0 rc 0/0
LustreError: 1654:0:(ldlm_lib.c:1318:target_send_reply_msg()) @@@ processing error (-19) req at e22b0e00 x904040/t0 o8-><?>@<?>:-1 lens 240/0 ref 0 fl Interpret:/0/0 rc -19/0
LustreError: 1644:0:(client.c:940:ptlrpc_expire_one_request()) @@@ timeout (sent at 1296025713, 20s ago) req at cee31800 x10419493/t0 o104->@NET_0x200000a41c825_UUID:15 lens 176/64 ref 1 fl Rpc:/0/0 rc 0/0
LustreError: A client on nid 10.65.200.37 at tcp was evicted from service ost1.
LustreError: 1644:0:(ldlm_lockd.c:427:ldlm_failed_ast()) ### blocking AST failed (-110): evicting client 7c2f0_lov1_34edb7f8f7 at NET_0x200000a41c825_UUID NID 10.65.200.37 at tcp (10.65.200.37 at tcp) ns: filter-ost1_UUID lock: ed9131c0/0x64a02240089dfbd4 lrc: 2/0,0 mode: PW/PW res: 28177721/0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 20 remote: 0x943ed9a06440dfce expref: 2519 pid: 1662
LustreError: 1691:0:(service.c:654:ptlrpc_server_handle_request()) request 876972 opc 6 from 12345-10.65.200.37 at tcp processed in 122s trans 49349204 rc 0/0
LustreError: 1691:0:(service.c:654:ptlrpc_server_handle_request()) Skipped 6 previous similar messages
Lustre: 1691:0:(watchdog.c:311:lcw_update_time()) Expired watchdog for pid 1691 disabled after 122643196.0001s
Lustre: 1691:0:(watchdog.c:311:lcw_update_time()) Skipped 5 previous similar messages
Lustre: 1470:0:(lib-move.c:1644:lnet_parse_put()) Dropping PUT from 12345-10.65.200.37 at tcp portal 16 match 10663839 offset 0 length 112: 2
LustreError: 1638:0:(ldlm_lib.c:557:target_handle_connect()) @@@ UUID 'ost2_UUID' is not available for connect (no target) req at d9df1400 x904744/t0 o8-><?>@<?>:-1 lens 240/0 ref 0 fl Interpret:/0/0 rc 0/0
LustreError: 1638:0:(ldlm_lib.c:1318:target_send_reply_msg()) @@@ processing error (-19) req at d9df1400 x904744/t0 o8-><?>@<?>:-1 lens 240/0 ref 0 fl Interpret:/0/0 rc -19/0
Lustre: 1470:0:(lib-move.c:1644:lnet_parse_put()) Dropping PUT from 12345-10.65.200.37 at tcp portal 16 match 10663856 offset 0 length 112: 2
LustreError: 1543:0:(ldlm_lockd.c:1056:ldlm_handle_cancel()) received cancel for unknown lock cookie 0x64a02240089dfbd4 from client 7c2f0_lov1_34edb7f8f7 id 12345-10.65.200.37 at tcp
LustreError: 1543:0:(ldlm_lockd.c:1056:ldlm_handle_cancel()) Skipped 1 previous similar message
Lustre: 1470:0:(lib-move.c:1644:lnet_parse_put()) Dropping PUT from 12345-10.65.200.37 at tcp portal 16 match 10663860 offset 0 length 64: 2
Lustre: 1470:0:(lib-move.c:1644:lnet_parse_put()) Dropping PUT from 12345-10.65.200.37 at tcp portal 16 match 10663863 offset 0 length 112: 2
LustreError: 1730:0:(filter_io.c:532:filter_preprw_write()) ost1: trying to BRW to non-existent file 28331257
LustreError: 1730:0:(filter_io.c:532:filter_preprw_write()) Skipped 1 previous similar message
LustreError: 0:0:(ldlm_lockd.c:205:waiting_locks_callback()) ### lock callback timer expired: evicting client 7c2f0_lov1_34edb7f8f7 at NET_0x200000a41c825_UUID nid 10.65.200.37 at tcp ns: filter-ost1_UUID lock: d089b080/0x64a02240089fdf2f lrc: 1/0,0 mode: PW/PW res: 28049109/0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 20 remote: 0x943ed9a064e092a6 expref: 2547 pid: 1650
LustreError: 1708:0:(ldlm_lib.c:1318:target_send_reply_msg()) @@@ processing error (-107) req at d1b25200 x1184444/t0 o4-><?>@<?>:-1 lens 328/0 ref 0 fl Interpret:/0/0 rc -107/0
LustreError: 1542:0:(ldlm_lockd.c:1056:ldlm_handle_cancel()) received cancel for unknown lock cookie 0x64a02240089f6002 from client 7c2f0_lov1_34edb7f8f7 id 12345-10.65.200.37 at tcp
LustreError: 1638:0:(ldlm_lib.c:557:target_handle_connect()) @@@ UUID 'ost2_UUID' is not available for connect (no target) req at d0c7da00 x1209963/t0 o8-><?>@<?>:-1 lens 240/0 ref 0 fl Interpret:/0/0 rc 0/0
LustreError: 1638:0:(ldlm_lib.c:1318:target_send_reply_msg()) @@@ processing error (-19) req at d0c7da00 x1209963/t0 o8-><?>@<?>:-1 lens 240/0 ref 0 fl Interpret:/0/0 rc -19/0
LustreError: 1638:0:(ldlm_lib.c:1318:target_send_reply_msg()) Skipped 2 previous similar messages
Clinet loogs
LustreError: 6561:0:(file.c:703:ll_pgcache_remove_extent()) Skipped 27 previous similar messages
LustreError: 6370:0:(namei.c:941:ll_objects_destroy()) obd destroy objid 0xbc84e6 error -5
LustreError: 6370:0:(namei.c:941:ll_objects_destroy()) Skipped 3 previous similar messages
LustreError: 6561:0:(file.c:703:ll_pgcache_remove_extent()) writepage of page c1b33ea0 failed: -5
LustreError: 6561:0:(file.c:703:ll_pgcache_remove_extent()) Skipped 10 previous similar messages
LustreError: 6561:0:(ldlm_resource.c:365:ldlm_namespace_cleanup()) Namespace OSC_hades.exampe.com_ost1_MNT_client resource refcount 1 after lock cleanup; forcing cleanup.
LustreError: 6370:0:(namei.c:941:ll_objects_destroy()) obd destroy objid 0xbc8654 error -5
LustreError: 6370:0:(namei.c:941:ll_objects_destroy()) Skipped 11 previous similar messages
LustreError: 6383:0:(file.c:1003:ll_glimpse_size()) obd_enqueue returned rc -5, returning -EIO
LustreError: 6383:0:(file.c:1003:ll_glimpse_size()) Skipped 3 previous similar messages
LustreError: 6561:0:(file.c:703:ll_pgcache_remove_extent()) writepage of page c1225440 failed: -5
LustreError: 2169:0:(client.c:511:ptlrpc_import_delay_req()) @@@ IMP_INVALID req at f7d8d400 x906093/t0 o4->ost1_UUID at typhoon_UUID:28 lens 328/288 ref 2 fl Rpc:/0/0 rc 0/0
LustreError: 2169:0:(client.c:511:ptlrpc_import_delay_req()) Skipped 505 previous similar messages
LustreError: 6561:0:(file.c:703:ll_pgcache_remove_extent()) writepage of page c12f00e0 failed: -5
LustreError: 6561:0:(file.c:703:ll_pgcache_remove_extent()) Skipped 236 previous similar messages
LustreError: 6561:0:(file.c:703:ll_pgcache_remove_extent()) writepage of page c124bb80 failed: -5
LustreError: 6561:0:(file.c:703:ll_pgcache_remove_extent()) Skipped 430 previous similar messages
LustreError: 6419:0:(namei.c:941:ll_objects_destroy()) obd destroy objid 0xbc8776 error -5
LustreError: 6419:0:(namei.c:941:ll_objects_destroy()) Skipped 14 previous similar messages
LustreError: 6561:0:(file.c:703:ll_pgcache_remove_extent()) writepage of page c14d3bc0 failed: -5
LustreError: 6561:0:(file.c:703:ll_pgcache_remove_extent()) Skipped 269 previous similar messages
LustreError: 6370:0:(namei.c:941:ll_objects_destroy()) obd destroy objid 0xbc8732 error -5
LustreError: 6370:0:(namei.c:941:ll_objects_destroy()) Skipped 42 previous similar messages
LustreError: 6561:0:(file.c:703:ll_pgcache_remove_extent()) writepage of page c179c260 failed: -5
LustreError: 6561:0:(ldlm_resource.c:365:ldlm_namespace_cleanup()) Namespace OSC_hades.exampe.com_ost1_MNT_client resource refcount 1 after lock cleanup; forcing cleanup.
LustreError: 6561:0:(ldlm_resource.c:365:ldlm_namespace_cleanup()) Namespace OSC_hades.exampe.com_ost1_MNT_client resource refcount 2 after lock cleanup; forcing cleanup.
LustreError: 2169:0:(client.c:511:ptlrpc_import_delay_req()) @@@ IMP_INVALID req at f7d8d800 x907615/t0 o4->ost1_UUID at typhoon_UUID:28 lens 328/288 ref 2 fl Rpc:/0/0 rc 0/0
LustreError: 2169:0:(client.c:511:ptlrpc_import_delay_req()) Skipped 997 previous similar messages
LustreError: 6561:0:(file.c:703:ll_pgcache_remove_extent()) writepage of page c1094f20 failed: -5
LustreError: 6561:0:(file.c:703:ll_pgcache_remove_extent()) Skipped 712 previous similar messages
LustreError: 5723:0:(file.c:1003:ll_glimpse_size()) obd_enqueue returned rc -5, returning -EIO
LustreError: 5723:0:(file.c:1003:ll_glimpse_size()) Skipped 4 previous similar messages
LustreError: 6564:0:(llite_lib.c:1334:ll_setattr_raw()) obd_setattr_async fails: rc=-5
LustreError: 6561:0:(file.c:703:ll_pgcache_remove_extent()) writepage of page c13206e0 failed: -5
LustreError: 6561:0:(file.c:703:ll_pgcache_remove_extent()) Skipped 2057 previous similar messages
Lustre: OSC_hades.exampe.com_ost1_MNT_client: Connection restored to service ost1 using nid 10.65.200.21 at tcp.
LustreError: 3036:0:(lov_request.c:181:lov_update_enqueue_set()) enqueue objid 0x5f7a008 subobj 0x1ad4682 on OST idx 0: rc -5
LustreError: 2182:0:(ldlm_request.c:746:ldlm_cli_cancel()) client/server (nid 10.65.200.21 at tcp) out of sync -- not fatal, flags 322c90
LustreError: 2182:0:(file.c:754:ll_extent_lock_callback()) ldlm_cli_cancel failed: 116
Lustre: 2118:0:(lib-move.c:1644:lnet_parse_put()) Dropping PUT from 12345-10.65.200.30 at tcp portal 4 match 877070 offset 0 length 280: 2
Lustre: Changing connection for OSC_hades.exampe.com_ost2_MNT_client to cyclops_UUID/10.65.200.30 at tcp
LustreError: This client was evicted by ost2; in progress operations using this service will fail.
LustreError: 2183:0:(ldlm_request.c:752:ldlm_cli_cancel()) Got rc -5 from cancel RPC: canceling anyway
LustreError: 6445:0:(lov_request.c:181:lov_update_enqueue_set()) enqueue objid 0x7670003 subobj 0x1ab6cc4 on OST idx 1: rc -5
LustreError: 6445:0:(lov_request.c:181:lov_update_enqueue_set()) Skipped 1 previous similar message
LustreError: 6445:0:(file.c:1003:ll_glimpse_size()) obd_enqueue returned rc -5, returning -EIO
LustreError: 6445:0:(file.c:1003:ll_glimpse_size()) Skipped 1 previous similar message
LustreError: 5590:0:(rw.c:966:ll_issue_page_read()) page c12300e0 map eab7c3f0 index 0 flags 20001023 count 4 priv e9ee5dc0: read queue failed: rc -5
LustreError: 2183:0:(file.c:754:ll_extent_lock_callback()) ldlm_cli_cancel failed: -5
LustreError: 2183:0:(file.c:754:ll_extent_lock_callback()) Skipped 1 previous similar message
LustreError: 2169:0:(client.c:511:ptlrpc_import_delay_req()) @@@ IMP_INVALID req at c229d200 x911586/t0 o4->ost2_UUID at cyclops_UUID:28 lens 328/288 ref 2 fl Rpc:/0/0 rc 0/0
LustreError: 2169:0:(client.c:511:ptlrpc_import_delay_req()) Skipped 1141 previous similar messages
LustreError: 3252:0:(llite_lib.c:1334:ll_setattr_raw()) obd_setattr_async fails: rc=-5
LustreError: 6332:0:(rw.c:966:ll_issue_page_read()) page c1793ba0 map e746f630 index 0 flags 40001123 count 5 priv d5c0a980: read queue failed: rc -5
LustreError: 6435:0:(rw.c:966:ll_issue_page_read()) page c1793ba0 map e746f630 index 0 flags 40001123 count 4 priv d5c0a980: read queue failed: rc -5
LustreError: 6435:0:(rw.c:966:ll_issue_page_read()) page c1793ba0 map e746f630 index 0 flags 40001123 count 4 priv d5c0a980: read queue failed: rc -5
LustreError: 6275:0:(lov_request.c:181:lov_update_enqueue_set()) enqueue objid 0x6728007 subobj 0x1ab6cc8 on OST idx 1: rc -5
LustreError: 6275:0:(lov_request.c:181:lov_update_enqueue_set()) Skipped 7 previous similar messages
LustreError: 2191:0:(file.c:703:ll_pgcache_remove_extent()) writepage of page c1363380 failed: -5
LustreError: 2191:0:(file.c:703:ll_pgcache_remove_extent()) Skipped 3493 previous similar messages
LustreError: 5672:0:(rw.c:966:ll_issue_page_read()) page c1402980 map d706c870 index 1 flags 20001023 count 7 priv e92c1380: read queue failed: rc -5
LustreError: 5747:0:(rw.c:966:ll_issue_page_read()) page c1402980 map d706c870 index 1 flags 20001023 count 6 priv e92c1380: read queue failed: rc -5
LustreError: 5747:0:(rw.c:966:ll_issue_page_read()) page c1402980 map d706c870 index 1 flags 20001023 count 8 priv e92c1380: read queue failed: rc -5
LustreError: 5834:0:(rw.c:966:ll_issue_page_read()) page c1402980 map d706c870 index 1 flags 20001023 count 7 priv e92c1380: read queue failed: rc -5
LustreError: 5834:0:(rw.c:966:ll_issue_page_read()) page c1402980 map d706c870 index 1 flags 20001023 count 7 priv e92c1380: read queue failed: rc -5
LustreError: 6579:0:(rw.c:966:ll_issue_page_read()) page c1ad5900 map e3420ab0 index 0 flags 40001123 count 4 priv e3910100: read queue failed: rc -5
LustreError: 6579:0:(rw.c:966:ll_issue_page_read()) page c1ad5900 map e3420ab0 index 0 flags 40001123 count 4 priv e3910100: read queue failed: rc -5
LustreError: 6538:0:(rw.c:966:ll_issue_page_read()) page c15db2a0 map f5273630 index 0 flags 20001023 count 5 priv da791f00: read queue failed: rc -5
LustreError: 2918:0:(rw.c:966:ll_issue_page_read()) page c15db2a0 map f5273630 index 0 flags 20001023 count 4 priv da791f00: read queue failed: rc -5
LustreError: 6579:0:(rw.c:966:ll_issue_page_read()) page c1ad5900 map e3420ab0 index 0 flags 40001123 count 4 priv e3910100: read queue failed: rc -5
LustreError: 6517:0:(rw.c:966:ll_issue_page_read()) page c1ad5900 map e3420ab0 index 0 flags 40001123 count 4 priv e3910100: read queue failed: rc -5
LustreError: 6517:0:(rw.c:966:ll_issue_page_read()) page c1ad5900 map e3420ab0 index 0 flags 40001123 count 4 priv e3910100: read queue failed: rc -5
LustreError: 5726:0:(rw.c:966:ll_issue_page_read()) page c1402980 map d706c870 index 1 flags 20001023 count 6 priv e92c1380: read queue failed: rc -5
LustreError: 5993:0:(rw.c:966:ll_issue_page_read()) page c1402980 map d706c870 index 1 flags 20001023 count 5 priv e92c1380: read queue failed: rc -5
LustreError: 5672:0:(rw.c:966:ll_issue_page_read()) page c1402980 map d706c870 index 1 flags 20001023 count 4 priv e92c1380: read queue failed: rc -5
LustreError: 3050:0:(rw.c:966:ll_issue_page_read()) page c1257680 map e5c73870 index 0 flags 20001023 count 4 priv d8ef9180: read queue failed: rc -5
LustreError: 5993:0:(rw.c:966:ll_issue_page_read()) page c1402980 map d706c870 index 1 flags 20001023 count 4 priv e92c1380: read queue failed: rc -5
LustreError: 6594:0:(rw.c:966:ll_issue_page_read()) page c1638a20 map f50903f0 index 0 flags 20001023 count 4 priv f4c60100: read queue failed: rc -5
LustreError: 6538:0:(rw.c:966:ll_issue_page_read()) page c15db2a0 map f5273630 index 0 flags 20001023 count 4 priv da791f00: read queue failed: rc -5
LustreError: 6596:0:(llite_lib.c:1334:ll_setattr_raw()) obd_setattr_async fails: rc=-5
LustreError: 6596:0:(llite_lib.c:1334:ll_setattr_raw()) Skipped 3 previous similar messages
LustreError: 6412:0:(namei.c:941:ll_objects_destroy()) obd destroy objid 0x74005ef error -5
LustreError: 6412:0:(namei.c:941:ll_objects_destroy()) Skipped 134 previous similar messages
LustreError: 6412:0:(file.c:106:ll_close_inode_openhandle()) inode 99614226 ll_objects destroy: rc = -5
LustreError: 6597:0:(ldlm_resource.c:365:ldlm_namespace_cleanup()) Namespace OSC_hades.exampe.com_ost2_MNT_client resource refcount 2 after lock cleanup; forcing cleanup.
LustreError: 6601:0:(file.c:1003:ll_glimpse_size()) obd_enqueue returned rc -5, returning -EIO
LustreError: 6601:0:(file.c:1003:ll_glimpse_size()) Skipped 13 previous similar messages
Lustre: OSC_hades.exampe.com_ost2_MNT_client: Connection restored to service ost2 using nid 10.65.200.30 at tcp.
LustreError: 6536:0:(lov_request.c:181:lov_update_enqueue_set()) enqueue objid 0x5efcea5 subobj 0x1a88749 on OST idx 1: rc -5
LustreError: 2169:0:(client.c:576:ptlrpc_check_status()) @@@ type == PTL_RPC_MSG_ERR, err == -2 req at f7d8f400 x1081227/t0 o4->ost1_UUID at typhoon_UUID:28 lens 328/288 ref 2 fl Rpc:R/0/0 rc 0/-2
LustreError: 2169:0:(client.c:576:ptlrpc_check_status()) Skipped 1 previous similar message
LustreError: 2169:0:(client.c:576:ptlrpc_check_status()) @@@ type == PTL_RPC_MSG_ERR, err == -107 req at f7d8dc00 x1184444/t0 o4->ost1_UUID at typhoon_UUID:28 lens 328/288 ref 2 fl Rpc:R/0/0 rc 0/-107
LustreError: OSC_hades.exampe.com_ost1_MNT_client: Connection to service ost1 via nid 10.65.200.21 at tcp was lost; in progress operations using this service will wait for recovery to complete.
Lustre: Changing connection for OSC_hades.exampe.com_ost1_MNT_client to cyclops_UUID/10.65.200.30 at tcp
Lustre: Changing connection for OSC_hades.exampe.com_ost1_MNT_client to typhoon_UUID/10.65.200.21 at tcp
LustreError: This client was evicted by ost1; in progress operations using this service will fail.
LustreError: 7673:0:(lov_request.c:181:lov_update_enqueue_set()) enqueue objid 0x6c08150 subobj 0x1b055ab on OST idx 0: rc -5
LustreError: 2944:0:(file.c:1003:ll_glimpse_size()) obd_enqueue returned rc -5, returning -EIO
LustreError: 7673:0:(lov_request.c:181:lov_update_enqueue_set()) Skipped 2 previous similar messages
LustreError: 2169:0:(client.c:511:ptlrpc_import_delay_req()) @@@ IMP_INVALID req at f6878a00 x1186234/t0 o3->ost1_UUID at typhoon_UUID:28 lens 328/280 ref 2 fl Rpc:/0/0 rc 0/0
LustreError: 2169:0:(client.c:511:ptlrpc_import_delay_req()) Skipped 657 previous similar messages
LustreError: 6363:0:(rw.c:1446:ll_readpage()) page c1a9d160 map f48753f0 index 0 flags 40001123 count 5 priv e33a2d00: lock match failed: rc -5
LustreError: 6227:0:(rw.c:1446:ll_readpage()) page c1a9d160 map f48753f0 index 0 flags 40001123 count 4 priv e33a2d00: lock match failed: rc -5
LustreError: 7722:0:(rw.c:1446:ll_readpage()) page c13a5240 map f23e6630 index 1 flags 20001023 count 3 priv f4198e80: lock match failed: rc -5
LustreError: 7736:0:(rw.c:1446:ll_readpage()) page c1a9d160 map f48753f0 index 0 flags 40001123 count 3 priv e33a2d00: lock match failed: rc -5
LustreError: 6949:0:(rw.c:1446:ll_readpage()) page c15ee600 map d79073f0 index 1 flags 20001023 count 14 priv dc8db080: lock match failed: rc -5
LustreError: 6637:0:(rw.c:1446:ll_readpage()) page c15ee600 map d79073f0 index 1 flags 20001023 count 12 priv dc8db080: lock match failed: rc -5
LustreError: 5723:0:(rw.c:1446:ll_readpage()) page c1abd4c0 map f6a4ef30 index 0 flags 40001123 count 3 priv d33c85c0: lock match failed: rc -5
LustreError: 7615:0:(llite_lib.c:1334:ll_setattr_raw()) obd_setattr_async fails: rc=-5
LustreError: 6790:0:(rw.c:1446:ll_readpage()) page c15ee600 map d79073f0 index 1 flags 20001023 count 11 priv dc8db080: lock match failed: rc -5
LustreError: 7467:0:(rw.c:1446:ll_readpage()) page c15ee600 map d79073f0 index 1 flags 20001023 count 10 priv dc8db080: lock match failed: rc -5
LustreError: 6689:0:(rw.c:1446:ll_readpage()) page c15ee600 map d79073f0 index 1 flags 20001023 count 9 priv dc8db080: lock match failed: rc -5
LustreError: 6547:0:(rw.c:1446:ll_readpage()) page c15ee600 map d79073f0 index 1 flags 20001023 count 8 priv dc8db080: lock match failed: rc -5
LustreError: 7149:0:(rw.c:1446:ll_readpage()) page c15ee600 map d79073f0 index 1 flags 20001023 count 7 priv dc8db080: lock match failed: rc -5
LustreError: 6997:0:(rw.c:1446:ll_readpage()) page c15ee600 map d79073f0 index 1 flags 20001023 count 6 priv dc8db080: lock match failed: rc -5
LustreError: 6734:0:(rw.c:1446:ll_readpage()) page c15ee600 map d79073f0 index 1 flags 20001023 count 5 priv dc8db080: lock match failed: rc -5
LustreError: 7134:0:(rw.c:1446:ll_readpage()) page c15ee600 map d79073f0 index 1 flags 20001023 count 4 priv dc8db080: lock match failed: rc -5
LustreError: 7597:0:(rw.c:1446:ll_readpage()) page c1add3a0 map e3a961b0 index 1 flags 40001123 count 4 priv f60943c0: lock match failed: rc -5
LustreError: 7513:0:(rw.c:1446:ll_readpage()) page c15ee600 map d79073f0 index 1 flags 20001023 count 3 priv dc8db080: lock match failed: rc -5
LustreError: 7744:0:(rw.c:1446:ll_readpage()) page c12cf300 map f3e5ecf0 index 0 flags 20001023 count 3 priv e4c4aa00: lock match failed: rc -5
LustreError: 7741:0:(llite_lib.c:1334:ll_setattr_raw()) obd_setattr_async fails: rc=-5
LustreError: 7778:0:(file.c:703:ll_pgcache_remove_extent()) writepage of page c1bd2380 failed: -5
LustreError: 7778:0:(file.c:703:ll_pgcache_remove_extent()) Skipped 7533 previous similar messages
LustreError: 7741:0:(namei.c:941:ll_objects_destroy()) obd destroy objid 0xbcac42 error -5
LustreError: 7741:0:(namei.c:941:ll_objects_destroy()) Skipped 493 previous similar messages
LustreError: 6689:0:(llite_lib.c:1334:ll_setattr_raw()) obd_setattr_async fails: rc=-5
LustreError: 6689:0:(llite_lib.c:1334:ll_setattr_raw()) Skipped 1 previous similar message
LustreError: 7452:0:(file.c:106:ll_close_inode_openhandle()) inode 100175813 ll_objects destroy: rc = -5
LustreError: 7650:0:(file.c:1003:ll_glimpse_size()) obd_enqueue returned rc -5, returning -EIO
LustreError: 7650:0:(file.c:1003:ll_glimpse_size()) Skipped 4 previous similar messages
LustreError: 7650:0:(llite_lib.c:1334:ll_setattr_raw()) obd_setattr_async fails: rc=-5
LustreError: 7650:0:(llite_lib.c:1334:ll_setattr_raw()) Skipped 6 previous similar messages
LustreError: 7650:0:(namei.c:941:ll_objects_destroy()) obd destroy objid 0x429000a error -5
LustreError: 7650:0:(namei.c:941:ll_objects_destroy()) Skipped 5 previous similar messages
LustreError: 2169:0:(client.c:511:ptlrpc_import_delay_req()) @@@ IMP_INVALID req at f7d8d600 x1187047/t0 o4->ost1_UUID at typhoon_UUID:28 lens 328/288 ref 2 fl Rpc:/0/0 rc 0/0
LustreError: 2169:0:(client.c:511:ptlrpc_import_delay_req()) Skipped 620 previous similar messages
LustreError: 7778:0:(file.c:703:ll_pgcache_remove_extent()) writepage of page c12edce0 failed: -5
LustreError: 7778:0:(file.c:703:ll_pgcache_remove_extent()) Skipped 1711 previous similar messages
LustreError: 7778:0:(ldlm_resource.c:365:ldlm_namespace_cleanup()) Namespace OSC_hades.exampe.com_ost1_MNT_client resource refcount 1 after lock cleanup; forcing cleanup.
LustreError: 7778:0:(ldlm_resource.c:365:ldlm_namespace_cleanup()) Namespace OSC_hades.exampe.com_ost1_MNT_client resource refcount 1 after lock cleanup; forcing cleanup.
LustreError: 6689:0:(namei.c:941:ll_objects_destroy()) obd destroy objid 0xbacaf0 error -5
LustreError: 6689:0:(namei.c:941:ll_objects_destroy()) Skipped 79 previous similar messages
LustreError: 7778:0:(ldlm_resource.c:365:ldlm_namespace_cleanup()) Namespace OSC_hades.exampe.com_ost1_MNT_client resource refcount 1 after lock cleanup; forcing cleanup.
LustreError: 7513:0:(file.c:1003:ll_glimpse_size()) obd_enqueue returned rc -5, returning -EIO
LustreError: 7513:0:(file.c:1003:ll_glimpse_size()) Skipped 15 previous similar messages
LustreError: 7778:0:(file.c:703:ll_pgcache_remove_extent()) writepage of page c1a28b40 failed: -5
LustreError: 7778:0:(file.c:703:ll_pgcache_remove_extent()) Skipped 5271 previous similar messages
LustreError: 7778:0:(ldlm_resource.c:365:ldlm_namespace_cleanup()) Namespace OSC_hades.exampe.com_ost1_MNT_client resource refcount 1 after lock cleanup; forcing cleanup.
LustreError: 7778:0:(ldlm_resource.c:365:ldlm_namespace_cleanup()) Namespace OSC_hades.exampe.com_ost1_MNT_client resource refcount 1 after lock cleanup; forcing cleanup.
Lustre: OSC_hades.exampe.com_ost1_MNT_client: Connection restored to service ost1 using nid 10.65.200.21 at tcp.
LustreError: 5978:0:(lov_request.c:181:lov_update_enqueue_set()) enqueue objid 0x5f9b8ed subobj 0x1adf539 on OST idx 0: rc -5
LustreError: 5978:0:(lov_request.c:181:lov_update_enqueue_set()) Skipped 5 previous similar messages
LustreError: 2206:0:(ldlm_request.c:746:ldlm_cli_cancel()) client/server (nid 10.65.200.21 at tcp) out of sync -- not fatal, flags 322c90
LustreError: 2206:0:(file.c:754:ll_extent_lock_callback()) ldlm_cli_cancel failed: 116
LustreError: 2206:0:(file.c:754:ll_extent_lock_callback()) Skipped 2 previous similar messages
LustreError: 2169:0:(client.c:576:ptlrpc_check_status()) @@@ type == PTL_RPC_MSG_ERR, err == -107 req at c229d400 x1209948/t0 o4->ost2_UUID at cyclops_UUID:28 lens 328/288 ref 2 fl Rpc:R/0/0 rc 0/-107
LustreError: 2169:0:(client.c:576:ptlrpc_check_status()) Skipped 3 previous similar messages
LustreError: OSC_hades.exampe.com_ost2_MNT_client: Connection to service ost2 via nid 10.65.200.30 at tcp was lost; in progress operations using this service will wait for recovery to complete.
Lustre: Changing connection for OSC_hades.exampe.com_ost2_MNT_client to typhoon_UUID/10.65.200.21 at tcp
LustreError: 2169:0:(client.c:576:ptlrpc_check_status()) @@@ type == PTL_RPC_MSG_ERR, err == -107 req at f7e0ca00 x1209953/t0 o4->ost2_UUID at typhoon_UUID:28 lens 328/288 ref 2 fl Rpc:R/0/0 rc 0/-107
LustreError: 2173:0:(ldlm_request.c:752:ldlm_cli_cancel()) Got rc -107 from cancel RPC: canceling anyway
LustreError: 2173:0:(ldlm_request.c:752:ldlm_cli_cancel()) Skipped 3 previous similar messages
LustreError: 2173:0:(file.c:754:ll_extent_lock_callback()) ldlm_cli_cancel failed: -107
LustreError: 2169:0:(client.c:576:ptlrpc_check_status()) Skipped 5 previous similar messages
LustreError: 2209:0:(ldlm_request.c:752:ldlm_cli_cancel()) Got rc -107 from cancel RPC: canceling anyway
Lustre: Changing connection for OSC_hades.exampe.com_ost2_MNT_client to cyclops_UUID/10.65.200.30 at tcp
LustreError: This client was evicted by ost2; in progress operations using this service will fail.
LustreError: 2169:0:(client.c:511:ptlrpc_import_delay_req()) @@@ IMP_INVALID req at c22a3200 x1219312/t0 o4->ost2_UUID at cyclops_UUID:28 lens 328/288 ref 2 fl Rpc:/0/0 rc 0/0
LustreError: 2169:0:(client.c:511:ptlrpc_import_delay_req()) Skipped 514 previous similar messages
LustreError: 7819:0:(file.c:1003:ll_glimpse_size()) obd_enqueue returned rc -5, returning -EIO
LustreError: 7819:0:(file.c:1003:ll_glimpse_size()) Skipped 4 previous similar messages
LustreError: 7832:0:(ldlm_resource.c:365:ldlm_namespace_cleanup()) Namespace OSC_hades.exampe.com_ost2_MNT_client resource refcount 1 after lock cleanup; forcing cleanup.
LustreError: 7832:0:(ldlm_resource.c:365:ldlm_namespace_cleanup()) Skipped 1 previous similar message
LustreError: 2176:0:(file.c:703:ll_pgcache_remove_extent()) writepage of page c17af000 failed: -5
LustreError: 2176:0:(file.c:703:ll_pgcache_remove_extent()) Skipped 1080 previous similar messages
LustreError: 7786:0:(rw.c:1446:ll_readpage()) page c1245840 map f1cb7870 index 1 flags 20001023 count 3 priv d8334b80: lock match failed: rc -5
LustreError: 7819:0:(namei.c:941:ll_objects_destroy()) obd destroy objid 0x5f90f20 error -5
LustreError: 7819:0:(namei.c:941:ll_objects_destroy()) Skipped 218 previous similar messages
LustreError: 7832:0:(ldlm_resource.c:365:ldlm_namespace_cleanup()) Namespace OSC_hades.exampe.com_ost2_MNT_client resource refcount 1 after lock cleanup; forcing cleanup.
LustreError: 7832:0:(ldlm_resource.c:365:ldlm_namespace_cleanup()) Skipped 1 previous similar message
LustreError: 7832:0:(ldlm_resource.c:365:ldlm_namespace_cleanup()) Namespace OSC_hades.exampe.com_ost2_MNT_client resource refcount 1 after lock cleanup; forcing cleanup.
LustreError: 7832:0:(ldlm_resource.c:365:ldlm_namespace_cleanup()) Skipped 17 previous similar messages
Lustre: OSC_hades.exampe.com_ost2_MNT_client: Connection restored to service ost2 using nid 10.65.200.30 at tcp.
LustreError: 2220:0:(ldlm_request.c:746:ldlm_cli_cancel()) client/server (nid 10.65.200.30 at tcp) out of sync -- not fatal, flags 332c90
LustreError: 2220:0:(file.c:754:ll_extent_lock_callback()) ldlm_cli_cancel failed: 116
LustreError: 2220:0:(file.c:754:ll_extent_lock_callback()) Skipped 6 previous similar messages
LustreError: 2203:0:(ldlm_request.c:746:ldlm_cli_cancel()) client/server (nid 10.65.200.30 at tcp) out of sync -- not fatal, flags 332c90
LustreError: 2203:0:(ldlm_request.c:746:ldlm_cli_cancel()) Skipped 4 previous similar messages
LustreError: 2169:0:(client.c:576:ptlrpc_check_status()) @@@ type == PTL_RPC_MSG_ERR, err == -2 req at c229d400 x1219419/t0 o4->ost2_UUID at cyclops_UUID:28 lens 328/288 ref 2 fl Rpc:R/0/0 rc 0/-2
LustreError: 2169:0:(client.c:576:ptlrpc_check_status()) Skipped 2 previous similar messages
LustreError: 7781:0:(lov_request.c:181:lov_update_enqueue_set()) enqueue objid 0x5ef90fd subobj 0x1a5e593 on OST idx 1: rc -5
LustreError: 7781:0:(lov_request.c:181:lov_update_enqueue_set()) Skipped 1 previous similar message
LustreError: 2227:0:(file.c:754:ll_extent_lock_callback()) ldlm_cli_cancel failed: 116
LustreError: 2227:0:(file.c:754:ll_extent_lock_callback()) Skipped 12 previous similar messages
LustreError: 2195:0:(ldlm_request.c:746:ldlm_cli_cancel()) client/server (nid 10.65.200.30 at tcp) out of sync -- not fatal, flags 332c90
LustreError: 2195:0:(ldlm_request.c:746:ldlm_cli_cancel()) Skipped 10 previous similar messages
LustreError: 7813:0:(lov_request.c:181:lov_update_enqueue_set()) enqueue objid 0x7108001 subobj 0x1a76e0d on OST idx 1: rc -5
LustreError: 7813:0:(lov_request.c:181:lov_update_enqueue_set()) Skipped 1 previous similar message
LustreError: 2169:0:(client.c:576:ptlrpc_check_status()) @@@ type == PTL_RPC_MSG_ERR, err == -2 req at c22a3a00 x1219453/t0 o4->ost2_UUID at cyclops_UUID:28 lens 328/288 ref 2 fl Rpc:R/0/0 rc 0/-2
LustreError: 2169:0:(client.c:576:ptlrpc_check_status()) Skipped 4 previous similar messages
LustreError: 2199:0:(file.c:754:ll_extent_lock_callback()) ldlm_cli_cancel failed: 116
LustreError: 2199:0:(file.c:754:ll_extent_lock_callback()) Skipped 3 previous similar messages
LustreError: 2169:0:(client.c:576:ptlrpc_check_status()) @@@ type == PTL_RPC_MSG_ERR, err == -2 req at c229d200 x1219484/t0 o4->ost2_UUID at cyclops_UUID:28 lens 328/288 ref 2 fl Rpc:R/0/0 rc 0/-2
LustreError: 2169:0:(client.c:576:ptlrpc_check_status()) Skipped 17 previous similar messages
LustreError: 2208:0:(ldlm_request.c:746:ldlm_cli_cancel()) client/server (nid 10.65.200.30 at tcp) out of sync -- not fatal, flags 332c90
LustreError: 2208:0:(ldlm_request.c:746:ldlm_cli_cancel()) Skipped 1 previous similar message
LustreError: 2208:0:(file.c:754:ll_extent_lock_callback()) ldlm_cli_cancel failed: 116
LustreError: 2169:0:(client.c:576:ptlrpc_check_status()) @@@ type == PTL_RPC_MSG_ERR, err == -2 req at c229bc00 x1219552/t0 o4->ost2_UUID at cyclops_UUID:28 lens 328/288 ref 2 fl Rpc:R/0/0 rc 0/-2
LustreError: 2169:0:(client.c:576:ptlrpc_check_status()) Skipped 40 previous similar messages
LustreError: 2188:0:(ldlm_request.c:746:ldlm_cli_cancel()) client/server (nid 10.65.200.30 at tcp) out of sync -- not fatal, flags 332c90
LustreError: 2188:0:(file.c:754:ll_extent_lock_callback()) ldlm_cli_cancel failed: 116
LustreError: 2169:0:(client.c:576:ptlrpc_check_status()) @@@ type == PTL_RPC_MSG_ERR, err == -2 req at c22a3a00 x1219666/t0 o4->ost2_UUID at cyclops_UUID:28 lens 328/288 ref 2 fl Rpc:R/0/0 rc 0/-2
LustreError: 2169:0:(client.c:576:ptlrpc_check_status()) Skipped 88 previous similar messages
LustreError: 2231:0:(ldlm_request.c:746:ldlm_cli_cancel()) client/server (nid 10.65.200.30 at tcp) out of sync -- not fatal, flags 332c90
LustreError: 2231:0:(ldlm_request.c:746:ldlm_cli_cancel()) Skipped 2 previous similar messages
LustreError: 2231:0:(file.c:754:ll_extent_lock_callback()) ldlm_cli_cancel failed: 116
LustreError: 2231:0:(file.c:754:ll_extent_lock_callback()) Skipped 2 previous similar messages
More information about the lustre-discuss
mailing list