[Lustre-discuss] Inactive Service Threads

Charles Taylor taylor at hpc.ufl.edu
Tue Dec 27 06:28:09 PST 2011


Lustre Version:
lustre-1.8.6-wc1_2.6.18_238.12.1.el5_lustre.1.8.6.x86_64

OSS Configuration:
------------------
Dual E5620 Processors (8 2.4 GHz cores)
24 GB RAM:
8 OSTs ( two per controller)
   4 x Adaptec 51245
   2 x RAID-6 LUN's per controller
   7200 RPM Hitachi Drives (SATA HUA722020ALA330))
   128 KB Stripe Size
   512 KB RPCs


We've tested the configuration extensively and know that we can sustain 2.4 GB/sec to the OSSs for large-block sequential I/O for long periods of time with no issues.  The problem comes in production under more typical client work loads where we see far too many of the messages below - even when the load on the servers is not all that great (8 - 10).   Five minutes for an IOP to complete seems like a looooong time.   Seems like we must be either hitting a bug or running out of some resource (locks?).  Iostat tends to show fairly typical service, queue, and wait times which further suggests that there is more going on here than just busy disks.

We have about 600 clients with the following settings...

   lctl set_param ldlm.namespaces.*osc*.lru_size=600
   lctl set_param ldlm.namespaces.*mdc*.lru_size=600
   max_rpcs_in_flight=32
   max_pages_per_rpc=128


Note that we have tuned down the number of ost threads from the default to 96 but it has had little impact.  If we are to believe the messages, we should probably reduce the thread count further but it feels like something else is wrong.   Perhaps someone else has encountered this or can see an obvious problem in our setup.

Any ideas or suggestions are welcome.

Charlie Taylor
UF HPC Center


Dec 26 15:05:10 hpcoss8 kernel: Lustre: Service thread pid 26929 was inactive for 320.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Dec 26 15:05:10 hpcoss8 kernel: Pid: 26929, comm: ll_ost_io_36
Dec 26 15:05:10 hpcoss8 kernel:
Dec 26 15:05:10 hpcoss8 kernel: Call Trace:
Dec 26 15:05:10 hpcoss8 kernel:  [<ffffffff80047144>] try_to_wake_up+0x472/0x484
Dec 26 15:05:10 hpcoss8 kernel:  [<ffffffff8008c871>] __wake_up_common+0x3e/0x68
Dec 26 15:05:10 hpcoss8 kernel:  [<ffffffff8028882e>] __down_trylock+0x39/0x4e
Dec 26 15:05:10 hpcoss8 kernel:  [<ffffffff8006472d>] __down_failed_trylock+0x35/0x3a
Dec 26 15:05:10 hpcoss8 kernel:  [<ffffffff800646b9>] __down_failed+0x35/0x3a
Dec 26 15:05:10 hpcoss8 kernel:  [<ffffffff88b491e6>] .text.lock.ldlm_resource+0x7d/0x87 [ptlrpc]
Dec 26 15:05:10 hpcoss8 kernel:  [<ffffffff88b6c337>] ldlm_pools_shrink+0x247/0x2f0 [ptlrpc]
Dec 26 15:05:10 hpcoss8 kernel:  [<ffffffff80064604>] __down_read+0x12/0x92
Dec 26 15:05:10 hpcoss8 kernel:  [<ffffffff8002231e>] __up_read+0x19/0x7f
Dec 26 15:05:10 hpcoss8 kernel:  [<ffffffff8003f6c0>] shrink_slab+0x60/0x153
Dec 26 15:05:10 hpcoss8 kernel:  [<ffffffff800cdd0a>] zone_reclaim+0x235/0x2cd
Dec 26 15:05:10 hpcoss8 kernel:  [<ffffffff800ca13d>] __rmqueue+0x44/0xc7
Dec 26 15:05:10 hpcoss8 kernel:  [<ffffffff8000a919>] get_page_from_freelist+0xbf/0x43a
Dec 26 15:05:10 hpcoss8 kernel:  [<ffffffff8000f41a>] __alloc_pages+0x78/0x308
Dec 26 15:05:10 hpcoss8 kernel:  [<ffffffff80025d41>] find_or_create_page+0x32/0x72
Dec 26 15:05:10 hpcoss8 kernel:  [<ffffffff88e694e5>] filter_get_page+0x35/0x70 [obdfilter]
Dec 26 15:05:10 hpcoss8 kernel:  [<ffffffff88e6b72a>] filter_preprw+0x14da/0x1e00 [obdfilter]
Dec 26 15:05:10 hpcoss8 kernel:  [<ffffffff88a41a54>] kiblnd_init_tx_msg+0x154/0x1d0 [ko2iblnd]
Dec 26 15:05:10 hpcoss8 kernel:  [<ffffffff88ad4dc0>] class_handle2object+0xe0/0x170 [obdclass]
Dec 26 15:05:11 hpcoss8 kernel:  [<ffffffff88a49f2d>] kiblnd_send+0x86d/0x8b0 [ko2iblnd]
Dec 26 15:05:11 hpcoss8 kernel:  [<ffffffff88e1600c>] ost_brw_write+0xf9c/0x2480 [ost]
Dec 26 15:05:11 hpcoss8 kernel:  [<ffffffff889fe111>] LNetMDBind+0x301/0x450 [lnet]
Dec 26 15:05:11 hpcoss8 kernel:  [<ffffffff88b88c65>] lustre_msg_set_limit+0x35/0xf0 [ptlrpc]
Dec 26 15:05:11 hpcoss8 kernel:  [<ffffffff88b7eac8>] ptlrpc_send_reply+0x5e8/0x600 [ptlrpc]
Dec 26 15:05:11 hpcoss8 kernel:  [<ffffffff88b82fe5>] lustre_msg_get_version+0x35/0xf0 [ptlrpc]
Dec 26 15:05:11 hpcoss8 kernel:  [<ffffffff88b82ef5>] lustre_msg_get_opc+0x35/0xf0 [ptlrpc]
Dec 26 15:05:11 hpcoss8 kernel:  [<ffffffff88b830a8>] lustre_msg_check_version_v2+0x8/0x20 [ptlrpc]
Dec 26 15:05:11 hpcoss8 kernel:  [<ffffffff88e1a09e>] ost_handle+0x2bae/0x55b0 [ost]
Dec 26 15:05:11 hpcoss8 kernel:  [<ffffffff80153e70>] __next_cpu+0x19/0x28
Dec 26 15:05:11 hpcoss8 kernel:  [<ffffffff8008dc31>] dequeue_task+0x18/0x37
Dec 26 15:05:11 hpcoss8 kernel:  [<ffffffff88b926d9>] ptlrpc_server_handle_request+0x989/0xe00 [ptlrpc]
Dec 26 15:05:11 hpcoss8 kernel:  [<ffffffff88b92e35>] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc]
Dec 26 15:05:11 hpcoss8 kernel:  [<ffffffff8008e435>] default_wake_function+0x0/0xe
Dec 26 15:05:11 hpcoss8 kernel:  [<ffffffff88b93dc6>] ptlrpc_main+0xf66/0x1120 [ptlrpc]
Dec 26 15:05:11 hpcoss8 kernel:  [<ffffffff8005dfb1>] child_rip+0xa/0x11
Dec 26 15:05:11 hpcoss8 kernel:  [<ffffffff88b92e60>] ptlrpc_main+0x0/0x1120 [ptlrpc]
Dec 26 15:05:11 hpcoss8 kernel:  [<ffffffff8005dfa7>] child_rip+0x0/0x11





More information about the lustre-discuss mailing list