<div dir="ltr">I don't to start a new thread. But something fishy is going on. The client I write you about exports Lustre via samba for some windows users. The client has a 10Gigabit Ethernet NIC. Usually it works fine, but sometimes this happens :<div>
<br></div><div>Sep 4 16:07:22 cache kernel: LustreError: Skipped 1 previous similar message</div><div>Sep 4 16:07:22 cache kernel: LustreError: 167-0: lustre0-MDT0000-mdc-ffff88062edb0c00: This client was evicted by lustre0-MDT0000; in progress operations using this service will fail.</div>
<div>Sep 4 16:07:22 cache kernel: LustreError: 7411:0:(mdc_locks.c:840:mdc_enqueue()) ldlm_cli_enqueue: -5</div><div>Sep 4 16:07:22 cache kernel: LustreError: 7411:0:(file.c:159:ll_close_inode_openhandle()) inode 144115380745406215 mdc close failed: rc = -108</div>
<div>Sep 4 16:07:22 cache smbd[7580]: [2013/09/04 16:07:22.451276, 0] smbd/process.c:2440(keepalive_fn)</div><div>Sep 4 16:07:22 cache kernel: LustreError: 7411:0:(file.c:159:ll_close_inode_openhandle()) inode 144115307009623849 mdc close failed: rc = -108</div>
<div>Sep 4 16:07:22 cache kernel: LustreError: 7411:0:(file.c:159:ll_close_inode_openhandle()) Skipped 1 previous similar message</div><div>Sep 4 16:07:22 cache kernel: LustreError: 6553:0:(mdc_locks.c:840:mdc_enqueue()) ldlm_cli_enqueue: -108</div>
<div>Sep 4 16:07:22 cache kernel: LustreError: 6553:0:(mdc_locks.c:840:mdc_enqueue()) Skipped 21 previous similar messages</div><div>Sep 4 16:07:22 cache kernel: LustreError: 6517:0:(vvp_io.c:1228:vvp_io_init()) lustre0: refresh file layout [0x200002b8a:0x119dd:0x0] error -108.</div>
<div>Sep 4 16:07:22 cache kernel: LustreError: 6517:0:(vvp_io.c:1228:vvp_io_init()) lustre0: refresh file layout [0x200002b8a:0x119dd:0x0] error -108.</div><div>Sep 4 16:07:22 cache kernel: LustreError: 6519:0:(dir.c:389:ll_get_dir_page()) lock enqueue: [0x200000007:0x1:0x0] at 0: rc -108</div>
<div>Sep 4 16:07:22 cache kernel: LustreError: 6519:0:(dir.c:595:ll_dir_read()) error reading dir [0x200000007:0x1:0x0] at 0: rc -108</div><div>Sep 4 16:07:22 cache kernel: LustreError: 6553:0:(dir.c:389:ll_get_dir_page()) lock enqueue: [0x200000007:0x1:0x0] at 0: rc -108</div>
<div>Sep 4 16:07:22 cache kernel: LustreError: 6553:0:(dir.c:595:ll_dir_read()) error reading dir [0x200000007:0x1:0x0] at 0: rc -108</div><div>Sep 4 16:07:22 cache smbd[6517]: [2013/09/04 16:07:22.469729, 0] smbd/dfree.c:137(sys_disk_free)</div>
<div>Sep 4 16:07:22 cache smbd[6517]: disk_free: sys_fsusage() failed. Error was : Cannot send after transport endpoint shutdown</div><div>Sep 4 16:07:22 cache kernel: LustreError: 6517:0:(lmv_obd.c:1289:lmv_statfs()) can't stat MDS #0 (lustre0-MDT0000-mdc-ffff88062edb0c00), error -108</div>
<div>Sep 4 16:07:22 cache kernel: LustreError: 6517:0:(llite_lib.c:1610:ll_statfs_internal()) md_statfs fails: rc = -108</div><div>Sep 4 16:07:22 cache smbd[6517]: [2013/09/04 16:07:22.470531, 0] smbd/dfree.c:137(sys_disk_free)</div>
<div>Sep 4 16:07:22 cache smbd[6517]: disk_free: sys_fsusage() failed. Error was : Cannot send after transport endpoint shutdown</div><div>Sep 4 16:07:22 cache smbd[6517]: [2013/09/04 16:07:22.470922, 0] smbd/dfree.c:137(sys_disk_free)</div>
<div>Sep 4 16:07:22 cache smbd[6517]: disk_free: sys_fsusage() failed. Error was : Cannot send after transport endpoint shutdown</div><div>Sep 4 16:07:22 cache kernel: LustreError: 6517:0:(lmv_obd.c:1289:lmv_statfs()) can't stat MDS #0 (lustre0-MDT0000-mdc-ffff88062edb0c00), error -108</div>
<div>Sep 4 16:07:22 cache kernel: LustreError: 6553:0:(statahead.c:1397:is_first_dirent()) error reading dir [0x200000007:0x1:0x0] at 0: [rc -108] [parent 6553]<br></div><div>Sep 4 16:07:22 cache kernel: LustreError: 6553:0:(statahead.c:1397:is_first_dirent()) error reading dir [0x200000007:0x1:0x0] at 0: [rc -108] [parent 6553]</div>
<div>Sep 4 16:07:22 cache smbd[8958]: [2013/09/04 16:07:22.517112, 0] smbd/dfree.c:137(sys_disk_free)</div><div>Sep 4 16:07:22 cache smbd[8958]: disk_free: sys_fsusage() failed. Error was : Cannot send after transport endpoint shutdown</div>
<div>Sep 4 16:07:22 cache smbd[8958]: [2013/09/04 16:07:22.517496, 0] smbd/dfree.c:137(sys_disk_free)</div><div> </div><div>Eventually it connects again to the MDS:</div><div><br></div><div><div>Sep 4 16:07:31 cache kernel: LustreError: 6582:0:(ldlm_resource.c:811:ldlm_resource_complain()) Resource: ffff880a125f7e40 (8589945618/117308/0/0) (rc: 1)</div>
<div>Sep 4 16:07:31 cache kernel: LustreError: 6582:0:(ldlm_resource.c:1423:ldlm_resource_dump()) --- Resource: ffff880a125f7e40 (8589945618/117308/0/0) (rc: 2)</div><div>Sep 4 16:07:31 cache kernel: Lustre: lustre0-MDT0000-mdc-ffff88062edb0c00: Connection restored to lustre0-MDT0000 (at 192.168.11.23@tcp)</div>
</div><div><br></div><div>But now the load average hits to high heaven ... 25 or even 50 for a 16 CPU machine. And shortly after that :</div><div><br></div><div><div>Sep 4 16:08:14 cache kernel: INFO: task ptlrpcd_6:2425 blocked for more than 120 seconds.</div>
<div>Sep 4 16:08:14 cache kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.</div><div>Sep 4 16:08:14 cache kernel: ptlrpcd_6 D 000000000000000c 0 2425 2 0x00000080</div>
<div>Sep 4 16:08:14 cache kernel: ffff880c34ab7a10 0000000000000046 0000000000000000 ffffffffa0a01736</div><div>Sep 4 16:08:14 cache kernel: ffff880c34ab79d0 ffffffffa09fc199 ffff88062e274000 ffff880c34cf8000</div><div>
Sep 4 16:08:14 cache kernel: ffff880c34aae638 ffff880c34ab7fd8 000000000000fb88 ffff880c34aae638</div><div>Sep 4 16:08:14 cache kernel: Call Trace:</div><div>Sep 4 16:08:14 cache kernel: [<ffffffffa0a01736>] ? ksocknal_queue_tx_locked+0x136/0x530 [ksocklnd]</div>
<div>Sep 4 16:08:14 cache kernel: [<ffffffffa09fc199>] ? ksocknal_find_conn_locked+0x159/0x290 [ksocklnd]</div><div>Sep 4 16:08:14 cache kernel: [<ffffffff8150f1ee>] __mutex_lock_slowpath+0x13e/0x180</div><div>
Sep 4 16:08:14 cache kernel: [<ffffffff8150f08b>] mutex_lock+0x2b/0x50</div><div>Sep 4 16:08:14 cache kernel: [<ffffffffa0751ecf>] cl_lock_mutex_get+0x6f/0xd0 [obdclass]</div><div>Sep 4 16:08:14 cache kernel: [<ffffffffa0b5b469>] lovsub_parent_lock+0x49/0x120 [lov]</div>
<div>Sep 4 16:08:14 cache kernel: [<ffffffffa0b5c60f>] lovsub_lock_modify+0x7f/0x1e0 [lov]</div><div>Sep 4 16:08:14 cache kernel: [<ffffffffa07514d8>] cl_lock_modify+0x98/0x310 [obdclass]</div><div>Sep 4 16:08:14 cache kernel: [<ffffffffa0748dae>] ? cl_object_attr_unlock+0xe/0x20 [obdclass]</div>
<div>Sep 4 16:08:14 cache kernel: [<ffffffffa0ac1e52>] ? osc_lock_lvb_update+0x1a2/0x470 [osc]</div><div>Sep 4 16:08:14 cache kernel: [<ffffffffa0ac2302>] osc_lock_granted+0x1e2/0x2b0 [osc]</div><div>Sep 4 16:08:14 cache kernel: [<ffffffffa0ac30b0>] osc_lock_upcall+0x3f0/0x5e0 [osc]</div>
<div>Sep 4 16:08:14 cache kernel: [<ffffffffa0ac2cc0>] ? osc_lock_upcall+0x0/0x5e0 [osc]</div><div>Sep 4 16:08:14 cache kernel: [<ffffffffa0aa3876>] osc_enqueue_fini+0x106/0x240 [osc]</div><div>Sep 4 16:08:14 cache kernel: [<ffffffffa0aa82c2>] osc_enqueue_interpret+0xe2/0x1e0 [osc]</div>
<div>Sep 4 16:08:14 cache kernel: [<ffffffffa0884d2c>] ptlrpc_check_set+0x2ac/0x1b20 [ptlrpc]</div><div>Sep 4 16:08:14 cache kernel: [<ffffffffa08b1c7b>] ptlrpcd_check+0x53b/0x560 [ptlrpc]</div><div>Sep 4 16:08:14 cache kernel: [<ffffffffa08b21a3>] ptlrpcd+0x233/0x390 [ptlrpc]</div>
<div>Sep 4 16:08:14 cache kernel: [<ffffffff81063310>] ? default_wake_function+0x0/0x20</div><div>Sep 4 16:08:14 cache kernel: [<ffffffffa08b1f70>] ? ptlrpcd+0x0/0x390 [ptlrpc]</div><div>Sep 4 16:08:14 cache kernel: [<ffffffff8100c0ca>] child_rip+0xa/0x20</div>
<div>Sep 4 16:08:14 cache kernel: [<ffffffffa08b1f70>] ? ptlrpcd+0x0/0x390 [ptlrpc]</div><div>Sep 4 16:08:14 cache kernel: [<ffffffffa08b1f70>] ? ptlrpcd+0x0/0x390 [ptlrpc]</div><div>Sep 4 16:08:14 cache kernel: [<ffffffff8100c0c0>] ? child_rip+0x0/0x20</div>
</div><div><br></div><div>Sometimes its the smbd daemon that spits similar call trace. I`m using lustre 2.4.0-2.6.32_358.6.2.el6.x86_64_gd3f91c4.x86_64. This is the only client I use for exporting lustre via samba. No other client is having errors or issues like that. Sometimes I can see that this particular client disconnects from some of the OSTs too. I`ll test the NIC to see if there are any hardware problems. but besides that, does any one have any clues or hints that want to share with me ?</div>
<div><br></div><div><br></div><div>Cheers,</div><div><br></div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Tue, Jun 25, 2013 at 8:55 AM, Nikolay Kvetsinski <span dir="ltr"><<a href="mailto:nkvecinski@gmail.com" target="_blank">nkvecinski@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Thank you all for your quick responses. Unfortunately my own stupidity played me this time ... the MDS server was not a domain member. After joining it it works OK.<div>
<br></div><div>Cheers,</div><div>:(</div>
</div><div class="HOEnZb"><div class="h5"><div class="gmail_extra"><br><br><div class="gmail_quote">On Mon, Jun 24, 2013 at 8:48 PM, Michael Watters <span dir="ltr"><<a href="mailto:wattersmt@gmail.com" target="_blank">wattersmt@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Does your uid in Windows match the uid on the samba server? Does the samba account exist? I ran into similar issues with NFS.</div>
<div class="gmail_extra"><br><br><div class="gmail_quote"><div>On Mon, Jun 24, 2013 at 5:59 AM, Nikolay Kvetsinski <span dir="ltr"><<a href="mailto:nkvecinski@gmail.com" target="_blank">nkvecinski@gmail.com</a>></span> wrote:<br>
</div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div><div dir="ltr">Hello guys,<div><br></div><div>I`m using the latest feature release (lustre-2.4.0-2.6.32_358.6.2.el6_lustre.g230b174.x86_64_gd3f91c4.x86_64.rpm) + centos 6.4. Lustre itself is working fine, but when I export it with samba and try to connect with a windows 7 client I get :</div>
<div><br></div><div><div>Jun 24 12:53:14 R-82L kernel: LustreError: 2326:0:(mdc_locks.c:840:mdc_enqueue()) ldlm_cli_enqueue: -13</div><div>Jun 24 12:53:16 R-82L kernel: LustreError: 2326:0:(mdc_locks.c:840:mdc_enqueue()) ldlm_cli_enqueue: -13</div>
<div>Jun 24 12:53:16 R-82L kernel: LustreError: 2326:0:(mdc_locks.c:840:mdc_enqueue()) Skipped 7 previous similar messages</div><div>Jun 24 12:53:16 R-82L kernel: LustreError: 2326:0:(mdc_locks.c:840:mdc_enqueue()) ldlm_cli_enqueue: -13</div>
<div>Jun 24 12:53:16 R-82L kernel: LustreError: 2326:0:(mdc_locks.c:840:mdc_enqueue()) Skipped 7 previous similar messages</div><div><br></div><div>And on the windows client I get "You don't have permissions to access ....." error. Permissions are 777. I created a local share just to test the samba server and its working. The error pops up only when trying to access samba share with lustre backend storage.</div>
<div><br></div><div>Any help will be greatly appreciated.</div><div><br></div><div>Cheers,</div></div></div>
<br></div></div><div>_______________________________________________<br>
Lustre-discuss mailing list<br>
<a href="mailto:Lustre-discuss@lists.lustre.org" target="_blank">Lustre-discuss@lists.lustre.org</a><br>
<a href="http://lists.lustre.org/mailman/listinfo/lustre-discuss" target="_blank">http://lists.lustre.org/mailman/listinfo/lustre-discuss</a><br>
<br></div></blockquote></div><br></div>
</blockquote></div><br></div>
</div></div></blockquote></div><br></div>