[Lustre-discuss] e2fsck mdsdb: DB_NOTFOUND

Karen M. Fernsler fernsler at ncsa.uiuc.edu
Fri Mar 14 09:19:45 PDT 2008


We've been able to verify that it is not a hardware issue, but we've
managed to narrow down that the problem seems to involve osth4_1 and
osth4_2.

We disabled recovery on ost start to see if it would behave
differently to no avail.  The results are attatched.

Anyone have any idea?

thanks,
-k

On Thu, Mar 13, 2008 at 03:51:22PM -0500, Fernsler; Karen wrote:
> 2.6.9-42.0.10.EL_lustre-1.4.10.1smp
> 
> This is a 2.6.9-42.0.10.E kernel with lustre-1.4.10.1.
> 
> This has been working ok for almost a year.  We did try to
> export this filesystem to another cluster over nfs before
> we started seeing problems, but I don't know how related if
> at all that is.
> 
> We are now trying to dissect the problem by inspecting
> the switch logs these nodes are connected to.
> 
> thanks,
> -k
> 
> On Thu, Mar 13, 2008 at 04:50:04PM -0400, Aaron Knister wrote:
> > What version of lustre/kernel is running on the problematic server?
> > 
> > On Mar 13, 2008, at 11:02 AM, Michelle Butler wrote:
> > 
> > >We got past that point by e2fsck the individual partitions first.
> > >
> > >But we are still having problems.. I'm sorry to
> > >say.   we have an I/O server that is fine until
> > >we start Lustre.  It starts spewing lustre call traces :
> > >
> > >Call
> > >Trace:<ffffffffa02fa089>{:libcfs:lcw_update_time+22}
> > ><ffffffffa03e06e3>{:ptlrpc:ptlrpc_main+1408}
> > >       <ffffffff8013327d>{default_wake_function+0}
> > ><ffffffffa03e0156>{:ptlrpc:ptlrpc_retry_rqbds+0}
> > >       <ffffffffa03e0156>{:ptlrpc:ptlrpc_retry_rqbds+0}
> > ><ffffffff80110ebb>{child_rip+8}
> > >       <ffffffffa03e0163>{:ptlrpc:ptlrpc_main+0}
> > ><ffffffff80110eb3>{child_rip+0}
> > >
> > >ll_ost_io_232 S 000001037d6bbee8     0 26764      1         26765  
> > >26763 (L-TLB)
> > >000001037d6bbe58 0000000000000046 0000000100000246 0000000000000003
> > >       0000000000000016 0000000000000001 00000104100bcb20  
> > >0000000300000246
> > >       00000103f5470030 000000000001d381
> > >Call
> > >Trace:<ffffffffa02fa089>{:libcfs:lcw_update_time+22}
> > ><ffffffffa03e06e3>{:ptlrpc:ptlrpc_main+1408}
> > >       <ffffffff8013327d>{default_wake_function+0}
> > ><ffffffffa03e0156>{:ptlrpc:ptlrpc_retry_rqbds+0}
> > >       <ffffffffa03e0156>{:ptlrpc:ptlrpc_retry_rqbds+0}
> > ><ffffffff80110ebb>{child_rip+8}
> > >       <ffffffffa03e0163>{:ptlrpc:ptlrpc_main+0}
> > ><ffffffff80110eb3>{child_rip+0}
> > >
> > >ll_ost_io_233 S 00000103de847ee8     0 26765      1         26766  
> > >26764 (L-TLB)
> > >00000103de847e58 0000000000000046 0000000100000246 0000000000000001
> > >       0000000000000016 0000000000000001 000001040f83c620  
> > >0000000100000246
> > >       00000103e627e030 000000000001d487
> > >Call
> > >Trace:<ffffffffa02fa089>{:libcfs:lcw_update_time+22}
> > ><ffffffffa03e06e3>{:ptlrpc:ptlrpc_main+1408}
> > >       <ffffffff8013327d>{default_wake_function+0}
> > ><ffffffffa03e0156>{:ptlrpc:ptlrpc_retry_rqbds+0}
> > >       <ffffffffa03e0156>{:ptlrpc:ptlrpc_retry_rqbds+0}
> > ><ffffffff80110ebb>{child_rip+8}
> > >       <ffffffffa03e0163>{:ptlrpc:ptlrpc_main+0}
> > ><ffffffff80110eb3>{child_rip+0}
> > >
> > >ll_ost_io_234 S 00000100c4353ee8     0 26766      1         26767  
> > >26765 (L-TLB)
> > >00000100c4353e58 0000000000000046 0000000100000246 0000000000000003
> > >       0000000000000016 0000000000000001 00000104100bcc60  
> > >0000000300000246
> > >       00000103de81b810 000000000001d945
> > >Call
> > >Trace:<ffffffffa02fa089>{:libcfs:lcw_update_time+22}
> > ><ffffffffa03e06e3>{:ptlrpc:ptlrpc_main+1408}
> > >       <ffffffff8013327d>{default_wake_function+0}
> > ><ffffffffa03e0156>{:ptlrpc:ptlrpc_retry_rqbds+0}
> > >        
> > ><ffffffffa03e0156>{:ptlrpc:ptlrpc_retr�f���c���c��
> > >                                                          
> > >Ks[F����
> > ><ffffffff8013327d>{default_wake_function+0}
> > ><ffffffffa03e0156>{:ptlrpc:ptlrpc_retry_rqbds+0}
> > >       <ffffffffa03e0156>{:ptl
> > >
> > >It then panic's the kernel.. ??
> > >
> > >Michelle Butler
> > >
> > >At 02:39 AM 3/13/2008, Andreas Dilger wrote:
> > >>On Mar 12, 2008  06:44 -0500, Karen M. Fernsler wrote:
> > >>>I'm running:
> > >>>
> > >>>e2fsck -y -v --mdsdb mdsdb --ostdb osth3_1 /dev/mapper/27l4
> > >>>
> > >>>and getting:
> > >>>
> > >>>Pass 6: Acquiring information for lfsck
> > >>>error getting mds_hdr (3685469441:8) in
> > >>/post/cfg/mdsdb: DB_NOTFOUND: No matching key/data pair found
> > >>>e2fsck: aborted
> > >>>
> > >>>Any ideas how to get around this?
> > >>
> > >>Does "mdsdb" actually exist?  This should be created by first  
> > >>running:
> > >>
> > >>e2fsck --mdsdb mdsdb /dev/{mdsdevicename}
> > >>
> > >>before running your above command on the OST.
> > >>
> > >>Please also try specifying the absolute pathname for the mdsdb and  
> > >>ostdb
> > >>files.
> > >>
> > >>Cheers, Andreas
> > >>--
> > >>Andreas Dilger
> > >>Sr. Staff Engineer, Lustre Group
> > >>Sun Microsystems of Canada, Inc.
> > >
> > >
> > >_______________________________________________
> > >Lustre-discuss mailing list
> > >Lustre-discuss at lists.lustre.org
> > >http://lists.lustre.org/mailman/listinfo/lustre-discuss
> > 
> > Aaron Knister
> > Associate Systems Analyst
> > Center for Ocean-Land-Atmosphere Studies
> > 
> > (301) 595-7000
> > aaron at iges.org
> > 
> > 
> > 
> 
> -- 
> Karen Fernsler Systems Engineer
> National Center for Supercomputing Applications
> ph: (217) 265 5249
> email: fernsler at ncsa.uiuc.edu
> _______________________________________________
> Lustre-discuss mailing list
> Lustre-discuss at lists.lustre.org
> http://lists.lustre.org/mailman/listinfo/lustre-discuss

-- 
Karen Fernsler Systems Engineer
National Center for Supercomputing Applications
ph: (217) 265 5249
email: fernsler at ncsa.uiuc.edu
-------------- next part --------------
LDISKFS-fs: file extents enabled
LDISKFS-fs: mballoc enabled
Lustre: 14674:0:(filter.c:730:filter_init_server_data()) RECOVERY: service osth2_1, 3 recoverable
clients, last_rcvd 21428009
Lustre: OST osth2_1 now serving /dev/mapper/24l4 (044e7efd-496b-43cc-b794-3a9e9c48bc78), but will be
inrecovery until 3 clients reconnect, or if no clients reconnect for 4:10; during that time new
clients will not be allowed to connect. Recovery progress can be monitored by watching
/proc/fs/lustre/obdfilter/osth2_1/recovery_status.
Lustre: 14603:0:(lib-move.c:1644:lnet_parse_put()) Dropping PUT from 12345-141.142.71.12 at o2ib portal
28 match 115715421 offset 0 length 240: 2
Lustre: 14600:0:(lib-move.c:1644:lnet_parse_put()) Dropping PUT from 12345-141.142.71.12 at o2ib portal
28 match 115715558 offset 0 length 240: 2
Lustre: 14600:0:(lib-move.c:1644:lnet_parse_put()) Skipped 23 previous similar messages
LustreError: 14872:0:(filter.c:3257:filter_iocontrol()) aborting recovery for device osth2_1
LustreError: 14872:0:(ldlm_lib.c:969:target_abort_recovery()) osth2_1: recovery period over;
disconnecting unfinished clients.
LustreError: 14872:0:(genops.c:908:class_disconnect_stale_exports()) osth2_1: disconnecting 3 stale
clients
Lustre: 14872:0:(ldlm_lib.c:867:target_finish_recovery()) osth2_1: sending delayed replies to
recovered clients
Lustre: 14872:0:(recover.c:77:ptlrpc_run_recovery_over_upcall()) Invoked upcall DEFAULT
RECOVERY_OVER osth2_1_UUID
LDISKFS-fs: file extents enabled
LDISKFS-fs: mballoc enabled
Lustre: 15907:0:(filter.c:730:filter_init_server_data()) RECOVERY: service osth2_2, 3 recoverable
clients, last_rcvd 23278178
Lustre: OST osth2_2 now serving /dev/mapper/2al4 (a4bbdf80-f96d-4bc1-84e0-fcd42ca14eb4), but will be
inrecovery until 3 clients reconnect, or if no clients reconnect for 4:10; during that time new
clients will not be allowed to connect. Recovery progress can be monitored by watching
/proc/fs/lustre/obdfilter/osth2_2/recovery_status.
LustreError: 15909:0:(filter.c:3257:filter_iocontrol()) aborting recovery for device osth2_2
LustreError: 15909:0:(ldlm_lib.c:969:target_abort_recovery()) osth2_2: recovery period over;
disconnecting unfinished clients.
LustreError: 15909:0:(genops.c:908:class_disconnect_stale_exports()) osth2_2: disconnecting 3 stale
clients
Lustre: 15909:0:(ldlm_lib.c:867:target_finish_recovery()) osth2_2: sending delayed replies to
recovered clients
Lustre: 15909:0:(recover.c:77:ptlrpc_run_recovery_over_upcall()) Invoked upcall DEFAULT
RECOVERY_OVER osth2_2_UUID
LDISKFS-fs: file extents enabled
LDISKFS-fs: mballoc enabled
Lustre: 15915:0:(filter.c:730:filter_init_server_data()) RECOVERY: service osth2_3, 3 recoverable
clients, last_rcvd 22572894
Lustre: OST osth2_3 now serving /dev/mapper/26l5 (ae63c80f-43dc-46f2-828b-ebe4fadf9ebc), but will be
inrecovery until 3 clients reconnect, or if no clients reconnect for 4:10; during that time new
clients will not be allowed to connect. Recovery progress can be monitored by watching
/proc/fs/lustre/obdfilter/osth2_3/recovery_status.
LustreError: 15917:0:(filter.c:3257:filter_iocontrol()) aborting recovery for device osth2_3
LustreError: 15917:0:(ldlm_lib.c:969:target_abort_recovery()) osth2_3: recovery period over;
disconnecting unfinished clients.
LustreError: 15917:0:(genops.c:908:class_disconnect_stale_exports()) osth2_3: disconnecting 3 stale
clients
Lustre: 15917:0:(ldlm_lib.c:867:target_finish_recovery()) osth2_3: sending delayed replies to
recovered clients
Lustre: 15917:0:(recover.c:77:ptlrpc_run_recovery_over_upcall()) Invoked upcall DEFAULT
RECOVERY_OVER osth2_3_UUID
LDISKFS-fs: file extents enabled
LDISKFS-fs: mballoc enabled
Lustre: 15923:0:(filter.c:730:filter_init_server_data()) RECOVERY: service osth2_4, 3 recoverable
clients, last_rcvd 23889626
Lustre: OST osth2_4 now serving /dev/mapper/22l6 (2f906cea-8413-4821-b92c-0637230959c7), but will be
inrecovery until 3 clients reconnect, or if no clients reconnect for 4:10; during that time new
clients will not be allowed to connect. Recovery progress can be monitored by watching
/proc/fs/lustre/obdfilter/osth2_4/recovery_status.
LustreError: 15925:0:(filter.c:3257:filter_iocontrol()) aborting recovery for device osth2_4
LustreError: 15925:0:(ldlm_lib.c:969:target_abort_recovery()) osth2_4: recovery period over;
disconnecting unfinished clients.
LustreError: 15925:0:(genops.c:908:class_disconnect_stale_exports()) osth2_4: disconnecting 3 stale
clients
Lustre: 15925:0:(ldlm_lib.c:867:target_finish_recovery()) osth2_4: sending delayed replies to
recovered clients
Lustre: 15925:0:(recover.c:77:ptlrpc_run_recovery_over_upcall()) Invoked upcall DEFAULT
RECOVERY_OVER osth2_4_UUID
LDISKFS-fs: file extents enabled
LDISKFS-fs: mballoc enabled
Lustre: 15931:0:(filter.c:730:filter_init_server_data()) RECOVERY: service osth2_5, 3 recoverable
clients, last_rcvd 21923408
Lustre: OST osth2_5 now serving /dev/mapper/28l6 (f237633e-b2e8-4372-998f-2d7c6f2913bd), but will be
inrecovery until 3 clients reconnect, or if no clients reconnect for 4:10; during that time new
clients will not be allowed to connect. Recovery progress can be monitored by watching
/proc/fs/lustre/obdfilter/osth2_5/recovery_status.
LustreError: 15933:0:(filter.c:3257:filter_iocontrol()) aborting recovery for device osth2_5
LustreError: 15933:0:(ldlm_lib.c:969:target_abort_recovery()) osth2_5: recovery period over;
disconnecting unfinished clients.
LustreError: 15933:0:(genops.c:908:class_disconnect_stale_exports()) osth2_5: disconnecting 3 stale
clients
Lustre: 15933:0:(ldlm_lib.c:867:target_finish_recovery()) osth2_5: sending delayed replies to
recovered clients
Lustre: 15933:0:(recover.c:77:ptlrpc_run_recovery_over_upcall()) Invoked upcall DEFAULT
RECOVERY_OVER osth2_5_UUID
LDISKFS-fs: file extents enabled
LDISKFS-fs: mballoc enabled
Lustre: 15963:0:(filter.c:730:filter_init_server_data()) RECOVERY: service osth4_1, 6 recoverable
clients, last_rcvd 41799396
Lustre: OST osth4_1 now serving /dev/mapper/26l4 (a2babbd1-a8c1-40a5-9d4d-c0139d8ecc9b), but will be
inrecovery until 6 clients reconnect, or if no clients reconnect for 4:10; during that time new
clients will not be allowed to connect. Recovery progress can be monitored by watching
/proc/fs/lustre/obdfilter/osth4_1/recovery_status.
LustreError: 15965:0:(filter.c:3257:filter_iocontrol()) aborting recovery for device osth4_1
LustreError: 15965:0:(ldlm_lib.c:969:target_abort_recovery()) osth4_1: recovery period over;
disconnecting unfinished clients.
LustreError: 15965:0:(genops.c:908:class_disconnect_stale_exports()) osth4_1: disconnecting 6 stale
clients
Lustre: 15965:0:(ldlm_lib.c:867:target_finish_recovery()) osth4_1: sending delayed replies to
recovered clients
Lustre: 15965:0:(recover.c:77:ptlrpc_run_recovery_over_upcall()) Invoked upcall DEFAULT
RECOVERY_OVER osth4_1_UUID
LDISKFS-fs: file extents enabled
LDISKFS-fs: mballoc enabled
----------- [cut here ] --------- [please bite here ] ---------
Kernel BUG at extents:301
invalid operand: 0000 [1] SMP
CPU 2
Modules linked in: obdfilter(U) fsfilt_ldiskfs(U) ost(U) lquota(U) ko2iblnd(U) rdma_cm(U) iw_cm(U)
ib_addr(U) ib_local_sa(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ldiskfs(U)
ipmi_devintf(U) ipmi_si(U) ipmi_msghandler(U) dell_rbu(U) netconsole(U) netdump(U) autofs4(U)
i2c_dev(U) i2c_core(U) nfs(U) lockd(U) nfs_acl(U) sunrpc(U) ds(U) yenta_socket(U) pcmcia_core(U)
dm_round_robin(U) dm_multipath(U) button(U) battery(U) ac(U) uhci_hcd(U) ehci_hcd(U) hw_random(U)
ib_mthca(U) ib_ipoib(U) ib_umad(U) ib_ucm(U) ib_uverbs(U) ib_cm(U) ib_sa(U) ib_mad(U) ib_core(U)
md5(U) ipv6(U) e1000(U) floppy(U) qla2300(U) qla2xxx(U) scsi_transport_fc(U) sg(U) dm_snapshot(U)
dm_zero(U) dm_mirror(U) ext3(U) jbd(U) dm_mod(U) mptscsih(U) mptsas(U) mptspi(U) mptfc(U) mptscsi(U)
mptbase(U) sd_mod(U) scsi_mod(U)
Pid: 15971, comm: lctl Tainted: GF     2.6.9-42.0.10.EL_lustre-1.4.10.1smp
RIP: 0010:[<ffffffffa030647f>] <ffffffffa030647f>{:ldiskfs:ldiskfs_ext_find_extent+323}
RSP: 0018:0000010235dc5a88  EFLAGS: 00010246
RAX: 00000101b15cc520 RBX: 00000101b15cc520 RCX: 0000000000000000
RDX: 0000000000000000 RSI: 000000000000080f RDI: 00000101dbb446d0
RBP: 00000101dbb44680 R08: 0000000000000000 R09: 00000101dbb44680
R10: 0000000000000000 R11: 0000000000001000 R12: 0000000000000001
R13: 00000101b15cc520 R14: 0000010235dc5ae8 R15: 0000000000000000
FS:  0000002a95574360(0000) GS:ffffffff80479d00(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00000032c658fae0 CR3: 000000000817e000 CR4: 00000000000006e0
Process lctl (pid: 15971, threadinfo 0000010235dc4000, task 0000010229c0e810)
Stack: 0000000000000001 00000101b15cc618 0000000000000000 00000101b15cc638
       0000010235dc5b88 0000000000000000 0000000000000000 ffffffffa0308919
       00000101b15cc638 00000001b15cc638
Call Trace:<ffffffffa0308919>{:ldiskfs:ldiskfs_ext_get_block+256}
       <ffffffff8018578e>{may_open+88} <ffffffffa02f8a3c>{:ldiskfs:ldiskfs_getblk+155}
       <ffffffffa02f8c0a>{:ldiskfs:ldiskfs_bread+15}
<ffffffffa04bc059>{:fsfilt_ldiskfs:fsfilt_ldiskfs_read_record+466}
       <ffffffffa04c95d2>{:obdfilter:filter_prep+1857} <ffffffff8017da00>{do_kern_mount+304}
       <ffffffffa04cc56c>{:obdfilter:filter_common_setup+2002}
       <ffffffffa04ccfda>{:obdfilter:filter_setup+367}
<ffffffffa03b707e>{:obdclass:class_setup+1561}
       <ffffffffa03bab44>{:obdclass:class_process_config+3366}
       <ffffffffa03a613a>{:obdclass:class_handle_ioctl+1954}
       <ffffffff80143631>{do_sigaction+490} <ffffffff80189265>{sys_ioctl+853}
       <ffffffff8011029a>{system_call+126}

Code: 0f 0b aa 25 31 a0 ff ff ff ff 2d 01 48 8d 48 0c 49 89 49 10
RIP <ffffffffa030647f>{:ldiskfs:ldiskfs_ext_find_extent+323} RSP <0000010235dc5a88>
CPU#0 is frozen.
CPU#1 is frozen.
CPU#2 is executing netdump.
CPU#3 is frozen.
< netdump activated - performing handshake with the server. >
NETDUMP START!
< handshake completed - listening for dump requests. >
0(4296190029)\
Modules linked in: obdfilter(U) fsfilt_ldiskfs(U) ost(U) lquota(U) ko2iblnd(U) rdma_cm(U) iw_cm(U)
ib_addr(U) ib_local_sa(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ldiskfs(U)
ipmi_devintf(U) ipmi_si(U) ipmi_msghandler(U) dell_rbu(U) netconsole(U) netdump(U) autofs4(U)
i2c_dev(U) i2c_core(U) nfs(U) lockd(U) nfs_acl(U) sunrpc(U) ds(U) yenta_socket(U) pcmcia_core(U)
dm_round_robin(U) dm_multipath(U) button(U) battery(U) ac(U) uhci_hcd(U) ehci_hcd(U) hw_random(U)
ib_mthca(U) ib_ipoib(U) ib_umad(U) ib_ucm(U) ib_uverbs(U) ib_cm(U) ib_sa(U) ib_mad(U) ib_core(U)
md5(U) ipv6(U) e1000(U) floppy(U) qla2300(U) qla2xxx(U) scsi_transport_fc(U) sg(U) dm_snapshot(U)
dm_zero(U) dm_mirror(U) ext3(U) jbd(U) dm_mod(U) mptscsih(U) mptsas(U) mptspi(U) mptfc(U) mptscsi(U)
mptbase(U) sd_mod(U) scsi_mod(U)
Pid: 15971, comm: lctl Tainted: GF     2.6.9-42.0.10.EL_lustre-1.4.10.1smp
RIP: 0010:[<ffffffffa030647f>] <ffffffffa030647f>{:ldiskfs:ldiskfs_ext_find_extent+323}
RSP: 0018:0000010235dc5a88  EFLAGS: 00010246
RAX: 00000101b15cc520 RBX: 00000101b15cc520 RCX: 0000000000000000
RDX: 0000000000000000 RSI: 000000000000080f RDI: 00000101dbb446d0
RBP: 00000101dbb44680 R08: 0000000000000000 R09: 00000101dbb44680
R10: 0000000000000000 R11: 0000000000001000 R12: 0000000000000001
R13: 00000101b15cc520 R14: 0000010235dc5ae8 R15: 0000000000000000
FS:  0000002a95574360(0000) GS:ffffffff80479d00(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00000032c658fae0 CR3: 000000000817e000 CR4: 00000000000006e0

Call Trace:<ffffffffa030640a>{:ldiskfs:ldiskfs_ext_find_extent+206}
       <ffffffffa0308919>{:ldiskfs:ldiskfs_ext_get_block+256}
       <ffffffff8018578e>{may_open+88} <ffffffffa02f8a3c>{:ldiskfs:ldiskfs_getblk+155}
       <ffffffffa02f8c0a>{:ldiskfs:ldiskfs_bread+15}
<ffffffffa04bc059>{:fsfilt_ldiskfs:fsfilt_ldiskfs_read_record+466}
       <ffffffffa04c95d2>{:obdfilter:filter_prep+1857} <ffffffff8017da00>{do_kern_mount+304}
       <ffffffffa04cc56c>{:obdfilter:filter_common_setup+2002}
       <ffffffffa04ccfda>{:obdfilter:filter_setup+367}
<ffffffffa03b707e>{:obdclass:class_setup+1561}
       <ffffffffa03bab44>{:obdclass:class_process_config+3366}
       <ffffffffa03a613a>{:obdclass:class_handle_ioctl+1954}
       <ffffffff80143631>{do_sigaction+490} <ffffffff80189265>{sys_ioctl+853}
       <ffffffff8011029a>{system_call+126}

                                                       sibling
  task                 PC          pid father child younger older
init          S 000000000000000b     0     1      0     2               (NOTLB)
00000101fffe1d78 0000000000000046 00000102ffc40d68 00000102fffe8e00
       000000000000000b 0000000000000400 000000d000000001 0000000300000246
       00000102fffd0810 0000000000000dd8
Call Trace:<ffffffff8013f210>{__mod_timer+293} <ffffffff80309ea4>{schedule_timeout+367}
       <ffffffff8013fc4c>{process_timeout+0} <ffffffff80189c5f>{do_select+939}
       <ffffffff801897f9>{__pollwait+0} <ffffffff80189fde>{sys_select+820}
       <ffffffff8011029a>{system_call+126}
migration/0   S 00000100010b87e0     0     2      1             3       (L-TLB)
00000102fff89ec8 0000000000000046 00000100010b87e0 0000001900000076
       0000010229c0e810 0000000000000076 00000100010a1a40 0000000000000001
       00000102fffd4810 000000000000018a
Call Trace:<ffffffff8013443e>{migration_thread+324} <ffffffff801342fa>{migration_thread+0}
       <ffffffff8014aa37>{kthread+200} <ffffffff80110ebb>{child_rip+8}
       <ffffffff8014a96f>{kthread+0} <ffffffff80110eb3>{child_rip+0}

ksoftirqd/0   S 0000000000000000     0     3      1             4     2 (L-TLB)
00000102fff8df08 0000000000000046 00000102fffd4810 0000001900000077
       0000010400fb7030 0000000000000077 00000100010a1a40 0000000000000000
       00000102fffd4030 00000000000000f5
Call Trace:<ffffffff8013c1ec>{ksoftirqd+0} <ffffffff8013c228>{ksoftirqd+60}
       <ffffffff8014aa37>{kthread+200} <ffffffff80110ebb>{child_rip+8}
       <ffffffff8014a96f>{kthread+0} <ffffffff80110eb3>{child_rip+0}

migration/1   S 00000100010b87e0     0     4      1             5     3 (L-TLB)
00000102fff8fec8 0000000000000046 00000100010b87e0 0000001900000077
       0000010229c0e810 0000000000000077 00000100010a9a40 0000000100000001
       00000102fffd5810 00000000000001cf
Call Trace:<ffffffff8013443e>{migration_thread+324} <ffffffff801342fa>{migration_thread+0}
       <ffffffff8014aa37>{kthread+200} <ffffffff80110ebb>{child_rip+8}
       <ffffffff8014a96f>{kthread+0} <ffffffff80110eb3>{child_rip+0}

ksoftirqd/1   S 0000000000000000     0     5      1             6     4 (L-TLB)
00000101fff91f08 0000000000000046 000001040c6e7810 0000001900000079
       000001040f8c0810 0000000000000079 00000100010a9a40 0000000100000000
       00000102fffd5030 0000000000000158
Call Trace:<ffffffff8013c1ec>{ksoftirqd+0} <ffffffff8013c228>{ksoftirqd+60}
       <ffffffff8014aa37>{kthread+200} <ffffffff80110ebb>{child_rip+8}
       <ffffffff8014a96f>{kthread+0} <ffffffff80110eb3>{child_rip+0}

migration/2   S 00000100010a87e0     0     6      1             7     5 (L-TLB)
00000101fff95ec8 0000000000000046 00000100010a87e0 0000001900000074
       000001023761f030 0000000000000074 00000100010b1a40 0000000200000001
       00000102fffa3810 00000000000001b0
Call Trace:<ffffffff8013443e>{migration_thread+324} <ffffffff801342fa>{migration_thread+0}
       <ffffffff8014aa37>{kthread+200} <ffffffff80110ebb>{child_rip+8}
       <ffffffff8014a96f>{kthread+0} <ffffffff80110eb3>{child_rip+0}

ksoftirqd/2   R  running task       0     7      1             8     6 (L-TLB)
migration/3   S 00000100010a07e0     0     8      1             9     7 (L-TLB)
00000100dff89ec8 0000000000000046 00000100010a07e0 0000001900000077
       000001023761f030 0000000000000077 00000100010b9a40 0000000300000001
       00000101fff9f810 00000000000001cf
Call Trace:<ffffffff8013443e>{migration_thread+324} <ffffffff801342fa>{migration_thread+0}
       <ffffffff8014aa37>{kthread+200} <ffffffff80110ebb>{child_rip+8}
       <ffffffff8014a96f>{kthread+0} <ffffffff80110eb3>{child_rip+0}

ksoftirqd/3   S 0000000000000000     0     9      1            10     8 (L-TLB)
00000100dffbdf08 0000000000000046 00000101a7aaa030 0000000000000246
       00000100dffbc000 0000000000000246 00000100010bbcc0 0000000300000000
       00000101fff9f030 00000000000001a4
Call Trace:<ffffffff8013c1ec>{ksoftirqd+0} <ffffffff8013c228>{ksoftirqd+60}
       <ffffffff8014aa37>{kthread+200} <ffffffff80110ebb>{child_rip+8}
       <ffffffff8014a96f>{kthread+0} <ffffffff80110eb3>{child_rip+0}

events/0      S ffffffff80160fc6     0    10      1    14      11     9 (L-TLB)
00000103ffe43e68 0000000000000046 00000103ffe43da8 0000000000000246
       0000000000000246 0000000000000246 000000010012a1f8 0000000000000246
       00000103ffe13810 0000000000001c0a
Call Trace:<ffffffff80160fc6>{cache_reap+0} <ffffffff80146d01>{worker_thread+226}
       <ffffffff8013327d>{default_wake_function+0} <ffffffff801332ce>{__wake_up_common+67}
       <ffffffff8013327d>{default_wake_function+0} <ffffffff80146c1f>{worker_thread+0}
       <ffffffff8014aa37>{kthread+200} <ffffffff80110ebb>{child_rip+8}
       <ffffffff8014a96f>{kthread+0} <ffffffff80110eb3>{child_rip+0}

events/1      S ffffffffa011d1d8     0    11      1   399      12    10 (L-TLB)
00000103ffe45e68 0000000000000046 00000000182a3c00 0000000000000000
       00000104101a1738 ffffffffa012100a 00000104101a1738 0000000100000246
       00000103ffe13030 00000000000025b9
Call Trace:<ffffffffa012100a>{:e1000:e1000_read_phy_reg+510}
<ffffffffa011d1d8>{:e1000:e1000_watchdog_task+0}
       <ffffffff80146d01>{worker_thread+226} <ffffffff8013327d>{default_wake_function+0}
       <ffffffff801332ce>{__wake_up_common+67} <ffffffff8013327d>{default_wake_function+0}
       <ffffffff80146c1f>{worker_thread+0} <ffffffff8014aa37>{kthread+200}
       <ffffffff80110ebb>{child_rip+8} <ffffffff8014a96f>{kthread+0}
       <ffffffff80110eb3>{child_rip+0}
events/2      S ffffffff80160fc6     0    12      1  8059      13    11 (L-TLB)
00000103ffe49e68 0000000000000046 00000102fffd3810 0000001900000086
       000001040bc48030 0000000000000246 0000000100129861 0000000200000246
       00000103ffe14810 00000000000011b8
Call Trace:<ffffffff80160fc6>{cache_reap+0} <ffffffff80146d01>{worker_thread+226}
       <ffffffff8013327d>{default_wake_function+0} <ffffffff801332ce>{__wake_up_common+67}
       <ffffffff8013327d>{default_wake_function+0} <ffffffff80146c1f>{worker_thread+0}
       <ffffffff8014aa37>{kthread+200} <ffffffff80110ebb>{child_rip+8}
       <ffffffff8014a96f>{kthread+0} <ffffffff80110eb3>{child_rip+0}

events/3      S ffffffff80160fc6     0    13



More information about the lustre-discuss mailing list