<p>Hi all,</p>
<p>The Lustre FS has crashed after the entire system was rebooted. <br>Here are some error messages as follows:</p>
<p><br>On n01-ib0 (one of the clients)<br>-------------------------------</p>
<p># /usr/sbin/lconf --node n01-ib0 /etc/lustre/config.xml<br>MDC: MDC_n01.local_mds_master2_MNT_n01-ib0_2 <br> 23503_MNT_n01-ib0_2_a7896cb070 mds_master2_UUID<br>MDC: MDC_n01.local_mds_master2_MNT_n01-ib0_2 <br> 23503_MNT_n01-ib0_2_a7896cb070
<br>! /usr/sbin/lctl (255): IOC_PORTAL_DEL_UUID failed: Invalid argument</p>
<p># dmesg<br>ERROR : IPOIB_UD : ipoib_ud_find_dev_by_dst:(ipoib_ud_arp.c)<br> :ip_route_output_key(<a href="http://127.0.0.1">127.0.0.1</a>) failed<br>... ...<br>LustreError: 5208:0:(client.c:947:ptlrpc_expire_one_request())
<br> @@@ timeout (sent at 1200501512, 5s ago) <a href="mailto:req@0000010117c34600">req@0000010117c34600</a> <br> x1/t0 o38->mds_master2_UUID@s03-ib0_UUID:12 lens 240/272 ref 1 fl <br> Rpc:/0/0 rc 0/0<br>Lustre: Changing connection for MDC_n01.local_mds_master2_MNT_n01-ib0_2
<br> to <a href="mailto:s04-ib0_UUID/11.0.0.249@vib">s04-ib0_UUID/11.0.0.249@vib</a><br>LustreError: 5208:0:(client.c:947:ptlrpc_expire_one_request()) <br> @@@ timeout (sent at 1200501517, 5s ago) <a href="mailto:req@0000010117c34600">
req@0000010117c34600</a> <br> x3/t0 o38->mds_master2_UUID@s04-ib0_UUID:12 lens 240/272 ref 1 fl <br> Rpc:/0/0 rc 0/0<br>Lustre: Changing connection for MDC_n01.local_mds_master2_MNT_n01-ib0_2 <br> to <a href="mailto:s03-ib0_UUID/11.0.0.250@vib">
s03-ib0_UUID/11.0.0.250@vib</a><br>... ...<br>Lustre: Skipped 39 previous similar messages<br>ERROR : AD_TAVOR : vvi_mlx_poll_for_completion:(adaptor_tavor.c):VLT: <br> completion_status: 10 (MLX: 12, syndrom: 129), total err num: 5
<br> (not print flush errors)<br>LustreError: 4941:0:(events.c:53:request_out_callback()) @@@ type 4, status <br> -113 <a href="mailto:req@000001011294f200">req@000001011294f200</a> x2794/t0 o38->mds_master2_UUID@s03-ib0
_UUID:12 <br> lens 240/272 ref 2 fl Rpc:/0/0 rc 0/0<br>LustreError: 4941:0:(events.c:53:request_out_callback()) <br> Skipped 14 previous similar messages<br>LustreError: 23731:0:(obd_config.c:333:class_cleanup()) OBD <br>
MDC_n01.local_mds_master2_MNT_n01-ib0_2 is still busy with 5 references<br>You should stop active file system users, or use the --force option to cleanup.<br>LustreError: 23731:0:(obd_config.c:234:class_detach()) OBD device 2 still set up
<br>LustreError: 23732:0:(lustre_peer.c:148:class_del_uuid()) delete non-existent <br> uuid s03-ib0_UUID</p>
<p><br>On s03-ib0 (failover MDS with s04-ib0)<br>--------------------------------------</p>
<p># traceroute <a href="http://11.0.0.1">11.0.0.1</a><br>traceroute to <a href="http://11.0.0.1">11.0.0.1</a> (<a href="http://11.0.0.1">11.0.0.1</a>), 30 hops max, 46 byte packets<br> 1 n01-ib0 (<a href="http://11.0.0.1">
11.0.0.1</a>) 0.149 ms 0.086 ms 0.088 ms</p>
<p># dmesg<br>ERROR : IPOIB_UD : ipoib_ud_find_dev_by_dst:(ipoib_ud_arp.c):<br> ip_route_output_key(<a href="http://127.0.0.1">127.0.0.1</a>) failed<br>new: ipoib_allow_arp_joins: 1<br>Linux Kernel Card Services<br> options: [pci] [cardbus] [pm]
<br>ERROR : IPOIB_UD : ipoib_ud_find_dev_by_dst:(ipoib_ud_arp.c):<br> ip_route_output_key(<a href="http://11.0.0.4">11.0.0.4</a>) failed<br>... ...<br>Lustre: Added LNI <a href="mailto:11.0.0.250@vib">11.0.0.250@vib</a>
[8/128]<br>Lustre: 4362:0:(lib-move.c:1644:lnet_parse_put()) Dropping PUT from <br> <a href="mailto:12345-11.0.0.3@vib">12345-11.0.0.3@vib</a> portal 12 match 3734 offset 0 length 240: 2<br>Lustre: 4362:0:(lib-move.c:1644:lnet_parse_put()) Dropping PUT from
<br> <a href="mailto:12345-11.0.0.15@vib">12345-11.0.0.15@vib</a> portal 12 match 3736 offset 0 length 240: 2</p>
<p>The error messages like 'ip_route_output_key(*) failed' means <br>there is probably wrong routing IPOIB interface configuration.<br>But both IPOIB Interface configuration and node routing table<br>seem to be OK. Any help would be greatly appreciated.
<br clear="all"><br>-- <br>Regards,<br>Changer </p>