<font><span style="background-color:rgba(255,255,255,0)">Eli,</span></font><div><font><span style="background-color:rgba(255,255,255,0)"><br></span></font></div><div><font><span style="background-color:rgba(255,255,255,0)">LNET is bound to a particular device at the time the Lustre modules are loaded. Lustre modules need to be unloaded prior to any unloading of a device to which it is bound. This can be done w/ Lustre init scripts, manually or by using lustre_rmmod. </span></font></div><div><font><span style="background-color:rgba(255,255,255,0)"><br></span></font></div><div><font><span style="background-color:rgba(255,255,255,0)">I can't speak to whether or not this will be fixed as I don't know that the developer community sees this as being broken. I'm sure someone will speak to that.</span></font></div><div><font><span style="background-color:rgba(255,255,255,0)"><br></span></font></div><div><font><span style="background-color:rgba(255,255,255,0)">--Jeff</span></font></div><br>On Wednesday, November 26, 2014, Eli Cohen <<a href="mailto:eli@dev.mellanox.co.il">eli@dev.mellanox.co.il</a>> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi,<br>
we installed Lustre over rdma_cm on our system. When we tried to<br>
unload the inifinband drivers we got this call trace:<br>
<br>
LNetError: 131-3: Received notification of device removal<br>
Please shutdown LNET to allow this to proceed<br>
INFO: task modprobe:6236 blocked for more than 120 seconds.<br>
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this<br>
message.<br>
modprobe D 000000000000000a 0 6236 6189 0x00000000<br>
ffff881fb14b5c18 0000000000000086 0000000000000000 ffffffffa03c3f33<br>
ffff881fb14b5c38 ffff881fb14b5c08 ffff881fb14b5ba8 ffff882028df5c70<br>
ffff882013637ab8 ffff881fb14b5fd8 000000000000fb88 ffff882013637ab8<br>
Call Trace:<br>
[<ffffffffa03c3f33>] ? libcfs_debug_vmsg2+0x5d3/0xbc0 [libcfs]<br>
[<ffffffff8150e555>] schedule_timeout+0x215/0x2e0<br>
[<ffffffff8150e1d3>] wait_for_common+0x123/0x180<br>
[<ffffffff81063310>] ? default_wake_function+0x0/0x20<br>
[<ffffffff8150e2ed>] wait_for_completion+0x1d/0x20<br>
[<ffffffffa02c10be>] cma_remove_one+0x18e/0x210 [rdma_cm]<br>
[<ffffffffa024660f>] ib_unregister_device+0x4f/0x100 [ib_core]<br>
[<ffffffff81063310>] ? default_wake_function+0x0/0x20<br>
[<ffffffffa0316689>] mlx5_ib_remove+0x19/0x50 [mlx5_ib]<br>
[<ffffffffa02f4245>] mlx5_remove_device+0x75/0x90 [mlx5_core]<br>
[<ffffffffa02f4633>] mlx5_unregister_interface+0x43/0x80 [mlx5_core]<br>
[<ffffffffa0328955>] __exit_compat+0x15/0xe2 [mlx5_ib]<br>
[<ffffffff810b4814>] sys_delete_module+0x194/0x260<br>
[<ffffffff8151311e>] ? do_page_fault+0x3e/0xa0<br>
[<ffffffff8100b072>] system_call_fastpath+0x16/0x1b<br>
<br>
I saw this peace of code in the callback function in<br>
kiblnd_cm_callback:<br>
<br>
case RDMA_CM_EVENT_DEVICE_REMOVAL:<br>
LCONSOLE_ERROR_MSG(0x131,<br>
"Received notification of device removal\n"<br>
"Please shutdown LNET to allow this to proceed\n");<br>
/* Can't remove network from underneath LNET for now, * so I have<br>
* to ignore this */<br>
return 0;<br>
<br>
which suggests that device removal events are not handled. Is there a<br>
plan to fix this?<br>
_______________________________________________<br>
Lustre-discuss mailing list<br>
<a href="javascript:;" onclick="_e(event, 'cvml', 'Lustre-discuss@lists.lustre.org')">Lustre-discuss@lists.lustre.org</a><br>
<a href="http://lists.lustre.org/mailman/listinfo/lustre-discuss" target="_blank">http://lists.lustre.org/mailman/listinfo/lustre-discuss</a><br>
</blockquote><br><br>-- <br><div dir="ltr">------------------------------<br>Jeff Johnson<br>Co-Founder<br>Aeon Computing<br><br><a href="mailto:jeff.johnson@aeoncomputing.com" target="_blank">jeff.johnson@aeoncomputing.com</a><br><a href="http://www.aeoncomputing.com" target="_blank">www.aeoncomputing.com</a><br>t: 858-412-3810 x1001 f: 858-412-3845<br>m: 619-204-9061<br><br>4170 Morena Boulevard, Suite D - San Diego, CA 92117<div><br></div><div>High-Performance Computing / Lustre Filesystems / Scale-out Storage</div></div><br>