<div dir="ltr"><div>Hi Raj,<br><br>[root@pg-gpu01 ~]# lustre_rmmod<br><br>[root@pg-gpu01 ~]# modprobe -v lustre<br>insmod /lib/modules/2.6.32-642.6.2.el6.x86_64/weak-updates/kernel/net/lustre/libcfs.ko <br>insmod /lib/modules/2.6.32-642.6.2.el6.x86_64/weak-updates/kernel/fs/lustre/lvfs.ko <br>insmod /lib/modules/2.6.32-642.6.2.el6.x86_64/weak-updates/kernel/net/lustre/lnet.ko networks=o2ib(ib0)<br>insmod /lib/modules/2.6.32-642.6.2.el6.x86_64/weak-updates/kernel/fs/lustre/obdclass.ko <br>insmod /lib/modules/2.6.32-642.6.2.el6.x86_64/weak-updates/kernel/fs/lustre/ptlrpc.ko <br>insmod /lib/modules/2.6.32-642.6.2.el6.x86_64/weak-updates/kernel/fs/lustre/fid.ko <br>insmod /lib/modules/2.6.32-642.6.2.el6.x86_64/weak-updates/kernel/fs/lustre/mdc.ko <br>insmod /lib/modules/2.6.32-642.6.2.el6.x86_64/weak-updates/kernel/fs/lustre/osc.ko <br>insmod /lib/modules/2.6.32-642.6.2.el6.x86_64/weak-updates/kernel/fs/lustre/lov.ko <br>insmod /lib/modules/2.6.32-642.6.2.el6.x86_64/weak-updates/kernel/fs/lustre/lustre.ko <br><br></div>dmesg:<br><div><br>LNet: HW CPU cores: 24, npartitions: 4<br>alg: No test for crc32 (crc32-table)<br>alg: No test for adler32 (adler32-zlib)<br>alg: No test for crc32 (crc32-pclmul)<br>Lustre: Lustre: Build Version: 2.5.3-RC1--PRISTINE-2.6.32-573.el6.x86_64<br>LNet: Added LNI 172.23.54.51@o2ib [8/256/0/180]<br><br></div><div>But no luck,<br></div><div><br>[root@pg-gpu01 ~]# lctl ping 172.23.55.211@o2ib<br>failed to ping 172.23.55.211@o2ib: Input/output error<br><br>[root@pg-gpu01 ~]# mount /home<br>mount.lustre: mount 172.23.55.211@o2ib:172.23.55.212@o2ib:/pghome01 at /home failed: Input/output error<br>Is the MGS running?<br><br><br><br><br><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Apr 24, 2017 at 7:53 PM, Raj <span dir="ltr"><<a href="mailto:rajgautam@gmail.com" target="_blank">rajgautam@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div>Yes, this is strange. Normally, I have seen that credits mismatch results this scenario but it doesn't look like this is the case. <br></div><div><br></div><div>You wouldn't want to put mgs into capture debug messages as there will be a lot of data. </div><div><br></div><div>I guess you already tried removing the lustre drivers and adding it again ? </div><div>lustre_rmmod </div><div>modprobe -v lustre</div><div><br></div><div>And check dmesg for any errors...</div><div class="HOEnZb"><div class="h5"><div><br></div><div><br></div><div><div class="gmail_quote"><div>On Mon, Apr 24, 2017 at 12:43 PM Strikwerda, Ger <<a href="mailto:g.j.c.strikwerda@rug.nl" target="_blank">g.j.c.strikwerda@rug.nl</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div><div>Hi Raj,<br><br></div>When i do a lctl ping on a MGS server i do not see any logs at all. Also not when i do a sucessfull ping from a working node. Is there a way to verbose the Lustre logging to see more detail on the LNET level?<br><br></div>It is very strange that a rebooted node is able to lctl ping compute nodes, but fails to lctl ping metadata and storage nodes. <br><br><br><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Apr 24, 2017 at 7:35 PM, Raj <span><<a href="mailto:rajgautam@gmail.com" target="_blank">rajgautam@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Ger,<br>It looks like default configuration of lustre. <br><br>Do you see any error message on the MGS side while you are doing lctl ping from the rebooted clients? <br><div class="m_-2824780214394110366m_-4169833405554715758m_-1807384931616815595HOEnZb"><div class="m_-2824780214394110366m_-4169833405554715758m_-1807384931616815595h5"><div class="gmail_quote"><div>On Mon, Apr 24, 2017 at 12:27 PM Strikwerda, Ger <<a href="mailto:g.j.c.strikwerda@rug.nl" target="_blank">g.j.c.strikwerda@rug.nl</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div><div>Hi Eli,<br><br></div>Nothing can be mounted on the Lustre filesystems so the output is:<br><br>[root@pg-gpu01 ~]# lfs df /home/ger/<br>[root@pg-gpu01 ~]# <br><br></div>Empty..<br><br><br></div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Apr 24, 2017 at 7:24 PM, E.S. Rosenberg <span><<a href="mailto:esr@cs.huji.ac.il" target="_blank">esr@cs.huji.ac.il</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><br><div class="gmail_extra"><br><div class="gmail_quote"><span>On Mon, Apr 24, 2017 at 8:19 PM, Strikwerda, Ger <span><<a href="mailto:g.j.c.strikwerda@rug.nl" target="_blank">g.j.c.strikwerda@rug.nl</a>></span> wrote:<br><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote"><div><div>Hallo Eli,<br><br></div>Logfile/syslog on the client-side:<span class="m_-2824780214394110366m_-4169833405554715758m_-1807384931616815595m_6244912658785255702m_2516206972536374282m_-8670976717830459019gmail-"><br><br>Lustre: Lustre: Build Version: 2.5.3-RC1--PRISTINE-2.6.32-<wbr>573.el6.x86_64<br>LNet: Added LNI 172.23.54.51@o2ib [8/256/0/180]<br></span>LNetError: 2878:0:(o2iblnd_cb.c:2587:<wbr>kiblnd_rejected()) 172.23.55.211@o2ib rejected: consumer defined fatal error<br></div></blockquote><div><br></div></span>lctl df /path/to/some/file<br><div><br></div><div>gives nothing useful? (the second one will dump *a lot*)<br></div><div><div class="m_-2824780214394110366m_-4169833405554715758m_-1807384931616815595m_6244912658785255702m_2516206972536374282h5"><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote"><div><br><br><br></div><div class="m_-2824780214394110366m_-4169833405554715758m_-1807384931616815595m_6244912658785255702m_2516206972536374282m_-8670976717830459019gmail-HOEnZb"><div class="m_-2824780214394110366m_-4169833405554715758m_-1807384931616815595m_6244912658785255702m_2516206972536374282m_-8670976717830459019gmail-h5"><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Apr 24, 2017 at 7:16 PM, E.S. Rosenberg <span><<a href="mailto:esr+lustre@mail.hebrew.edu" target="_blank">esr+lustre@mail.hebrew.edu</a>></span> wrote:<br><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote"><div><br><div class="gmail_extra"><br><div class="gmail_quote"><span>On Mon, Apr 24, 2017 at 8:13 PM, Strikwerda, Ger <span><<a href="mailto:g.j.c.strikwerda@rug.nl" target="_blank">g.j.c.strikwerda@rug.nl</a>></span> wrote:<br><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote"><div><div><div>Hi Raj (and others),<br><br></div>In which file should i state the credits/peer_credits stuff? <br><br></div>Perhaps relevant config-files:<br><div><br>[root@pg-gpu01 ~]# cd /etc/modprobe.d/<br><br>[root@pg-gpu01 modprobe.d]# ls<br>anaconda.conf blacklist-kvm.conf dist-alsa.conf dist-oss.conf ib_ipoib.conf lustre.conf openfwwf.conf<br>blacklist.conf blacklist-nouveau.conf dist.conf freeipmi-modalias.conf ib_sdp.conf mlnx.conf truescale.conf<br><br>[root@pg-gpu01 modprobe.d]# cat ./ib_ipoib.conf <br>alias netdev-ib* ib_ipoib<br><br>[root@pg-gpu01 modprobe.d]# cat ./mlnx.conf <br># Module parameters for MLNX_OFED kernel modules<br><br>[root@pg-gpu01 modprobe.d]# cat ./lustre.conf <br>options lnet networks=o2ib(ib0)<br><br></div><div>Are there more Lustre/LNET options that could help in this situation?<br></div></div></blockquote><div><br></div></span><div>What about the logfiles?<br></div><div>Any error messages in syslog? lctl debug options?<br></div><div>Veel geluk,<br></div><div>Eli <br></div><div><div class="m_-2824780214394110366m_-4169833405554715758m_-1807384931616815595m_6244912658785255702m_2516206972536374282m_-8670976717830459019gmail-m_-6199673089827782171h5"><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote"><div><div><br><br></div><div><br></div></div><div class="m_-2824780214394110366m_-4169833405554715758m_-1807384931616815595m_6244912658785255702m_2516206972536374282m_-8670976717830459019gmail-m_-6199673089827782171m_-954142030923126256HOEnZb"><div class="m_-2824780214394110366m_-4169833405554715758m_-1807384931616815595m_6244912658785255702m_2516206972536374282m_-8670976717830459019gmail-m_-6199673089827782171m_-954142030923126256h5"><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Apr 24, 2017 at 7:02 PM, Raj <span><<a href="mailto:rajgautam@gmail.com" target="_blank">rajgautam@gmail.com</a>></span> wrote:<br><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote">May be worth checking your lnet credits and peer_credits in /etc/modprobe.d ? <br>You can compare between working hosts and non working hosts. <br>Thanks<span class="m_-2824780214394110366m_-4169833405554715758m_-1807384931616815595m_6244912658785255702m_2516206972536374282m_-8670976717830459019gmail-m_-6199673089827782171m_-954142030923126256m_-4950728271923496188HOEnZb"><font color="#888888"><br>_Raj</font></span><div class="m_-2824780214394110366m_-4169833405554715758m_-1807384931616815595m_6244912658785255702m_2516206972536374282m_-8670976717830459019gmail-m_-6199673089827782171m_-954142030923126256m_-4950728271923496188HOEnZb"><div class="m_-2824780214394110366m_-4169833405554715758m_-1807384931616815595m_6244912658785255702m_2516206972536374282m_-8670976717830459019gmail-m_-6199673089827782171m_-954142030923126256m_-4950728271923496188h5"><br><div class="gmail_quote"><div>On Mon, Apr 24, 2017 at 10:10 AM Strikwerda, Ger <<a href="mailto:g.j.c.strikwerda@rug.nl" target="_blank">g.j.c.strikwerda@rug.nl</a>> wrote:<br></div><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote"><div><div>Hi Rick,<br><br></div>Even without iptables rules and loading the correct modules afterwards, we get the same results:<br><br>[root@pg-gpu01 sysconfig]# iptables --list<br>Chain INPUT (policy ACCEPT)<br>target prot opt source destination <br><br>Chain FORWARD (policy ACCEPT)<br>target prot opt source destination <br><br>Chain OUTPUT (policy ACCEPT)<br>target prot opt source destination <br><br>Chain LOGDROP (0 references)<br>target prot opt source destination <br>LOG all -- anywhere anywhere LOG level warning <br>DROP all -- anywhere anywhere <br><br>[root@pg-gpu01 sysconfig]# modprobe lnet<br><br>[root@pg-gpu01 sysconfig]# modprobe lustre<br><br>[root@pg-gpu01 sysconfig]# lctl ping 172.23.55.211@o2ib</div><div><br>failed to ping 172.23.55.211@o2ib: Input/output error<br><br><br><br><br><br><br></div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Apr 24, 2017 at 4:59 PM, Mohr Jr, Richard Frank (Rick Mohr) <span><<a href="mailto:rmohr@utk.edu" target="_blank">rmohr@utk.edu</a>></span> wrote:<br><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote">This might be a long shot, but have you checked for possible firewall rules that might be causing the issue? I\u2019m wondering if there is a chance that some rules were added after the nodes were up to allow Lustre access, and when a node got rebooted, it lost the rules.<br>
<br>
--<br>
Rick Mohr<br>
Senior HPC System Administrator<br>
National Institute for Computational Sciences<br>
<a rel="noreferrer" href="http://www.nics.tennessee.edu" target="_blank">http://www.nics.tennessee.edu</a><br>
<div class="m_-2824780214394110366m_-4169833405554715758m_-1807384931616815595m_6244912658785255702m_2516206972536374282m_-8670976717830459019gmail-m_-6199673089827782171m_-954142030923126256m_-4950728271923496188m_-2221363250000481128m_-72022338792146547HOEnZb"><div class="m_-2824780214394110366m_-4169833405554715758m_-1807384931616815595m_6244912658785255702m_2516206972536374282m_-8670976717830459019gmail-m_-6199673089827782171m_-954142030923126256m_-4950728271923496188m_-2221363250000481128m_-72022338792146547h5"><br>
<br>
> On Apr 24, 2017, at 10:19 AM, Strikwerda, Ger <<a href="mailto:g.j.c.strikwerda@rug.nl" target="_blank">g.j.c.strikwerda@rug.nl</a>> wrote:<br>
><br>
> Hi Russell,<br>
><br>
> Thanks for the IB subnet clues:<br>
><br>
> [root@pg-gpu01 ~]# ibv_devinfo<br>
> hca_id: mlx4_0<br>
> transport: InfiniBand (0)<br>
> fw_ver: 2.32.5100<br>
> node_guid: f452:1403:00f5:4620<br>
> sys_image_guid: f452:1403:00f5:4623<br>
> vendor_id: 0x02c9<br>
> vendor_part_id: 4099<br>
> hw_ver: 0x1<br>
> board_id: MT_1100120019<br>
> phys_port_cnt: 1<br>
> port: 1<br>
> state: PORT_ACTIVE (4)<br>
> max_mtu: 4096 (5)<br>
> active_mtu: 4096 (5)<br>
> sm_lid: 1<br>
> port_lid: 185<br>
> port_lmc: 0x00<br>
> link_layer: InfiniBand<br>
><br>
> [root@pg-gpu01 ~]# sminfo<br>
> sminfo: sm lid 1 sm guid 0xf452140300f62320, activity count 80878098 priority 0 state 3 SMINFO_MASTER<br>
><br>
> Looks like the rebooted node is able to connect/contact IB/IB subnetmanager<br>
><br>
><br>
><br>
><br>
> On Mon, Apr 24, 2017 at 4:14 PM, Russell Dekema <<a href="mailto:dekemar@umich.edu" target="_blank">dekemar@umich.edu</a>> wrote:<br>
> At first glance, this sounds like your Infiniband subnet manager may<br>
> be down or malfunctioning. In this case, nodes which were already up<br>
> when the subnet manager was working will continue to be able to<br>
> communicate over IB, but nodes which reboot after the SM goes down<br>
> will not.<br>
><br>
> You can test this theory by running the 'ibv_devinfo' command on one<br>
> of your rebooted nodes. If the relevant IB port is in state PORT_INIT,<br>
> this confirms there is a problem with your subnet manager.<br>
><br>
> Sincerely,<br>
> Rusty Dekema<br>
><br>
><br>
><br>
><br>
> On Mon, Apr 24, 2017 at 9:57 AM, Strikwerda, Ger<br>
> <<a href="mailto:g.j.c.strikwerda@rug.nl" target="_blank">g.j.c.strikwerda@rug.nl</a>> wrote:<br>
> > Hi everybody,<br>
> ><br>
> > Here at the university of Groningen we are now experiencing a strange Lustre<br>
> > error. If a client reboots, it fails to mount the Lustre storage. The client<br>
> > is not able to reach the MSG service. The storage and nodes are<br>
> > communicating over IB and unitil now without any problems. It looks like an<br>
> > issue inside LNET. Clients cannot LNET ping/connect the metadata and or<br>
> > storage. But the clients are able to LNET ping each other. Clients which not<br>
> > have been rebooted, are working fine and have their mounts on our Lustre<br>
> > filesystem.<br>
> ><br>
> > Lustre client log:<br>
> ><br>
> > Lustre: Lustre: Build Version: 2.5.3-RC1--PRISTINE-2.6.32-<wbr>573.el6.x86_64<br>
> > LNet: Added LNI 172.23.54.51@o2ib [8/256/0/180]<br>
> ><br>
> > LustreError: 15c-8: MGC172.23.55.211@o2ib: The configuration from log<br>
> > 'pgdata01-client' failed (-5). This may be the result of communication<br>
> > errors between this node and the MGS, a bad configuration, or other errors.<br>
> > See the syslog for more information.<br>
> > LustreError: 3812:0:(llite_lib.c:1046:ll_<wbr>fill_super()) Unable to process<br>
> > log: -5<br>
> > Lustre: Unmounted pgdata01-client<br>
> > LustreError: 3812:0:(obd_mount.c:1325:<wbr>lustre_fill_super()) Unable to mount<br>
> > (-5)<br>
> > LNetError: 2882:0:(o2iblnd_cb.c:2587:<wbr>kiblnd_rejected()) 172.23.55.212@o2ib<br>
> > rejected: consumer defined fatal error<br>
> > LNetError: 2882:0:(o2iblnd_cb.c:2587:<wbr>kiblnd_rejected()) Skipped 1 previous<br>
> > similar message<br>
> > Lustre: 3765:0:(client.c:1918:ptlrpc_<wbr>expire_one_request()) @@@ Request sent<br>
> > has failed due to network error: [sent 1492789626/real 1492789626]<br>
> > req@ffff88105af2cc00 x1565303228072004/t0(0)<br>
> > o250->MGC172.23.55.211@o2ib@<wbr>172.23.55.212@o2ib:26/25 lens 400/544 e 0 to 1<br>
> > dl 1492789631 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1<br>
> > Lustre: 3765:0:(client.c:1918:ptlrpc_<wbr>expire_one_request()) Skipped 1<br>
> > previous similar message<br>
> > LustreError: 3826:0:(client.c:1083:ptlrpc_<wbr>import_delay_req()) @@@ send limit<br>
> > expired req@ffff882041ffc000 x1565303228071996/t0(0)<br>
> > o101->MGC172.23.55.211@o2ib@<wbr>172.23.55.211@o2ib:26/25 lens 328/344 e 0 to 0<br>
> > dl 0 ref 2 fl Rpc:W/0/ffffffff rc 0/-1<br>
> > LustreError: 3826:0:(client.c:1083:ptlrpc_<wbr>import_delay_req()) Skipped 2<br>
> > previous similar messages<br>
> > LustreError: 15c-8: MGC172.23.55.211@o2ib: The configuration from log<br>
> > 'pghome01-client' failed (-5). This may be the result of communication<br>
> > errors between this node and the MGS, a bad configuration, or other errors.<br>
> > See the syslog for more information.<br>
> > LustreError: 3826:0:(llite_lib.c:1046:ll_<wbr>fill_super()) Unable to process<br>
> > log: -5<br>
> ><br>
> > LNetError: 2882:0:(o2iblnd_cb.c:2587:<wbr>kiblnd_rejected()) 172.23.55.212@o2ib<br>
> > rejected: consumer defined fatal error<br>
> > LNetError: 2882:0:(o2iblnd_cb.c:2587:<wbr>kiblnd_rejected()) Skipped 1 previous<br>
> > similar message<br>
> > LNet: 3755:0:(o2iblnd_cb.c:475:<wbr>kiblnd_rx_complete()) Rx from<br>
> > 172.23.55.211@o2ib failed: 5<br>
> > LNetError: 2882:0:(o2iblnd_cb.c:2587:<wbr>kiblnd_rejected()) 172.23.55.211@o2ib<br>
> > rejected: consumer defined fatal error<br>
> > LNetError: 2882:0:(o2iblnd_cb.c:2587:<wbr>kiblnd_rejected()) Skipped 1 previous<br>
> > similar message<br>
> > LNet: 2882:0:(o2iblnd_cb.c:2072:<wbr>kiblnd_peer_connect_failed()) Deleting<br>
> > messages for 172.23.55.211@o2ib: connection failed<br>
> > LNet: 2882:0:(o2iblnd_cb.c:2072:<wbr>kiblnd_peer_connect_failed()) Deleting<br>
> > messages for 172.23.55.212@o2ib: connection failed<br>
> > LNet: 3754:0:(o2iblnd_cb.c:475:<wbr>kiblnd_rx_complete()) Rx from<br>
> > 172.23.55.212@o2ib failed: 5<br>
> > LNet: 3754:0:(o2iblnd_cb.c:475:<wbr>kiblnd_rx_complete()) Skipped 17 previous<br>
> > similar messages<br>
> > LNet: 2882:0:(o2iblnd_cb.c:2072:<wbr>kiblnd_peer_connect_failed()) Deleting<br>
> > messages for 172.23.55.211@o2ib: connection failed<br>
> > LNet: 3754:0:(o2iblnd_cb.c:475:<wbr>kiblnd_rx_complete()) Rx from<br>
> > 172.23.55.212@o2ib failed: 5<br>
> > LNet: 2882:0:(o2iblnd_cb.c:2072:<wbr>kiblnd_peer_connect_failed()) Deleting<br>
> > messages for 172.23.55.212@o2ib: connection failed<br>
> ><br>
> > LNET ping of a metadata-node:<br>
> ><br>
> > [root@pg-gpu01 ~]# lctl ping 172.23.55.211@o2ib<br>
> > failed to ping 172.23.55.211@o2ib: Input/output error<br>
> ><br>
> > LNET ping of the number 2 metadata-node:<br>
> ><br>
> > [root@pg-gpu01 ~]# lctl ping 172.23.55.212@o2ib<br>
> > failed to ping 172.23.55.212@o2ib: Input/output error<br>
> ><br>
> > LNET ping of a random compute-node:<br>
> ><br>
> > [root@pg-gpu01 ~]# lctl ping 172.23.52.5@o2ib<br>
> > 12345-0@lo<br>
> > 12345-172.23.52.5@o2ib<br>
> ><br>
> > LNET to OST01:<br>
> ><br>
> > [root@pg-gpu01 ~]# lctl ping 172.23.55.201@o2ib<br>
> > failed to ping 172.23.55.201@o2ib: Input/output error<br>
> ><br>
> > LNET to OST02:<br>
> ><br>
> > [root@pg-gpu01 ~]# lctl ping 172.23.55.202@o2ib<br>
> > failed to ping 172.23.55.202@o2ib: Input/output error<br>
> ><br>
> > 'normal' pings (on ip level) works fine:<br>
> ><br>
> > [root@pg-gpu01 ~]# ping 172.23.55.201<br>
> > PING 172.23.55.201 (172.23.55.201) 56(84) bytes of data.<br>
> > 64 bytes from <a rel="noreferrer" href="http://172.23.55.201" target="_blank">172.23.55.201</a>: icmp_seq=1 ttl=64 time=0.741 ms<br>
> ><br>
> > [root@pg-gpu01 ~]# ping 172.23.55.202<br>
> > PING 172.23.55.202 (172.23.55.202) 56(84) bytes of data.<br>
> > 64 bytes from <a rel="noreferrer" href="http://172.23.55.202" target="_blank">172.23.55.202</a>: icmp_seq=1 ttl=64 time=0.704 ms<br>
> ><br>
> > lctl on a rebooted node:<br>
> ><br>
> > [root@pg-gpu01 ~]# lctl dl<br>
> ><br>
> > lctl on a not rebooted node:<br>
> ><br>
> > [root@pg-node005 ~]# lctl dl<br>
> > 0 UP mgc MGC172.23.55.211@o2ib 94bd1c8a-512f-b920-9a4e-<wbr>a6aced3d386d 5<br>
> > 1 UP lov pgtemp01-clilov-<wbr>ffff88206906d400<br>
> > 281c441f-8aa3-ab56-8812-<wbr>e459d308f47c 4<br>
> > 2 UP lmv pgtemp01-clilmv-<wbr>ffff88206906d400<br>
> > 281c441f-8aa3-ab56-8812-<wbr>e459d308f47c 4<br>
> > 3 UP mdc pgtemp01-MDT0000-mdc-<wbr>ffff88206906d400<br>
> > 281c441f-8aa3-ab56-8812-<wbr>e459d308f47c 5<br>
> > 4 UP osc pgtemp01-OST0001-osc-<wbr>ffff88206906d400<br>
> > 281c441f-8aa3-ab56-8812-<wbr>e459d308f47c 5<br>
> > 5 UP osc pgtemp01-OST0003-osc-<wbr>ffff88206906d400<br>
> > 281c441f-8aa3-ab56-8812-<wbr>e459d308f47c 5<br>
> > 6 UP osc pgtemp01-OST0005-osc-<wbr>ffff88206906d400<br>
> > 281c441f-8aa3-ab56-8812-<wbr>e459d308f47c 5<br>
> > 7 UP osc pgtemp01-OST0007-osc-<wbr>ffff88206906d400<br>
> > 281c441f-8aa3-ab56-8812-<wbr>e459d308f47c 5<br>
> > 8 UP osc pgtemp01-OST0009-osc-<wbr>ffff88206906d400<br>
> > 281c441f-8aa3-ab56-8812-<wbr>e459d308f47c 5<br>
> > 9 UP osc pgtemp01-OST000b-osc-<wbr>ffff88206906d400<br>
> > 281c441f-8aa3-ab56-8812-<wbr>e459d308f47c 5<br>
> > 10 UP osc pgtemp01-OST000d-osc-<wbr>ffff88206906d400<br>
> > 281c441f-8aa3-ab56-8812-<wbr>e459d308f47c 5<br>
> > 11 UP osc pgtemp01-OST000f-osc-<wbr>ffff88206906d400<br>
> > 281c441f-8aa3-ab56-8812-<wbr>e459d308f47c 5<br>
> > 12 UP osc pgtemp01-OST0011-osc-<wbr>ffff88206906d400<br>
> > 281c441f-8aa3-ab56-8812-<wbr>e459d308f47c 5<br>
> > 13 UP osc pgtemp01-OST0002-osc-<wbr>ffff88206906d400<br>
> > 281c441f-8aa3-ab56-8812-<wbr>e459d308f47c 5<br>
> > 14 UP osc pgtemp01-OST0004-osc-<wbr>ffff88206906d400<br>
> > 281c441f-8aa3-ab56-8812-<wbr>e459d308f47c 5<br>
> > 15 UP osc pgtemp01-OST0006-osc-<wbr>ffff88206906d400<br>
> > 281c441f-8aa3-ab56-8812-<wbr>e459d308f47c 5<br>
> > 16 UP osc pgtemp01-OST0008-osc-<wbr>ffff88206906d400<br>
> > 281c441f-8aa3-ab56-8812-<wbr>e459d308f47c 5<br>
> > 17 UP osc pgtemp01-OST000a-osc-<wbr>ffff88206906d400<br>
> > 281c441f-8aa3-ab56-8812-<wbr>e459d308f47c 5<br>
> > 18 UP osc pgtemp01-OST000c-osc-<wbr>ffff88206906d400<br>
> > 281c441f-8aa3-ab56-8812-<wbr>e459d308f47c 5<br>
> > 19 UP osc pgtemp01-OST000e-osc-<wbr>ffff88206906d400<br>
> > 281c441f-8aa3-ab56-8812-<wbr>e459d308f47c 5<br>
> > 20 UP osc pgtemp01-OST0010-osc-<wbr>ffff88206906d400<br>
> > 281c441f-8aa3-ab56-8812-<wbr>e459d308f47c 5<br>
> > 21 UP osc pgtemp01-OST0012-osc-<wbr>ffff88206906d400<br>
> > 281c441f-8aa3-ab56-8812-<wbr>e459d308f47c 5<br>
> > 22 UP osc pgtemp01-OST0013-osc-<wbr>ffff88206906d400<br>
> > 281c441f-8aa3-ab56-8812-<wbr>e459d308f47c 5<br>
> > 23 UP osc pgtemp01-OST0015-osc-<wbr>ffff88206906d400<br>
> > 281c441f-8aa3-ab56-8812-<wbr>e459d308f47c 5<br>
> > 24 UP osc pgtemp01-OST0017-osc-<wbr>ffff88206906d400<br>
> > 281c441f-8aa3-ab56-8812-<wbr>e459d308f47c 5<br>
> > 25 UP osc pgtemp01-OST0014-osc-<wbr>ffff88206906d400<br>
> > 281c441f-8aa3-ab56-8812-<wbr>e459d308f47c 5<br>
> > 26 UP osc pgtemp01-OST0016-osc-<wbr>ffff88206906d400<br>
> > 281c441f-8aa3-ab56-8812-<wbr>e459d308f47c 5<br>
> > 27 UP osc pgtemp01-OST0018-osc-<wbr>ffff88206906d400<br>
> > 281c441f-8aa3-ab56-8812-<wbr>e459d308f47c 5<br>
> > 28 UP lov pgdata01-clilov-<wbr>ffff88204bab6400<br>
> > 996b1742-82eb-281c-c322-<wbr>e244672d5225 4<br>
> > 29 UP lmv pgdata01-clilmv-<wbr>ffff88204bab6400<br>
> > 996b1742-82eb-281c-c322-<wbr>e244672d5225 4<br>
> > 30 UP mdc pgdata01-MDT0000-mdc-<wbr>ffff88204bab6400<br>
> > 996b1742-82eb-281c-c322-<wbr>e244672d5225 5<br>
> > 31 UP osc pgdata01-OST0001-osc-<wbr>ffff88204bab6400<br>
> > 996b1742-82eb-281c-c322-<wbr>e244672d5225 5<br>
> > 32 UP osc pgdata01-OST0003-osc-<wbr>ffff88204bab6400<br>
> > 996b1742-82eb-281c-c322-<wbr>e244672d5225 5<br>
> > 33 UP osc pgdata01-OST0005-osc-<wbr>ffff88204bab6400<br>
> > 996b1742-82eb-281c-c322-<wbr>e244672d5225 5<br>
> > 34 UP osc pgdata01-OST0007-osc-<wbr>ffff88204bab6400<br>
> > 996b1742-82eb-281c-c322-<wbr>e244672d5225 5<br>
> > 35 UP osc pgdata01-OST0009-osc-<wbr>ffff88204bab6400<br>
> > 996b1742-82eb-281c-c322-<wbr>e244672d5225 5<br>
> > 36 UP osc pgdata01-OST000b-osc-<wbr>ffff88204bab6400<br>
> > 996b1742-82eb-281c-c322-<wbr>e244672d5225 5<br>
> > 37 UP osc pgdata01-OST000d-osc-<wbr>ffff88204bab6400<br>
> > 996b1742-82eb-281c-c322-<wbr>e244672d5225 5<br>
> > 38 UP osc pgdata01-OST000f-osc-<wbr>ffff88204bab6400<br>
> > 996b1742-82eb-281c-c322-<wbr>e244672d5225 5<br>
> > 39 UP osc pgdata01-OST0002-osc-<wbr>ffff88204bab6400<br>
> > 996b1742-82eb-281c-c322-<wbr>e244672d5225 5<br>
> > 40 UP osc pgdata01-OST0004-osc-<wbr>ffff88204bab6400<br>
> > 996b1742-82eb-281c-c322-<wbr>e244672d5225 5<br>
> > 41 UP osc pgdata01-OST0006-osc-<wbr>ffff88204bab6400<br>
> > 996b1742-82eb-281c-c322-<wbr>e244672d5225 5<br>
> > 42 UP osc pgdata01-OST0008-osc-<wbr>ffff88204bab6400<br>
> > 996b1742-82eb-281c-c322-<wbr>e244672d5225 5<br>
> > 43 UP osc pgdata01-OST000a-osc-<wbr>ffff88204bab6400<br>
> > 996b1742-82eb-281c-c322-<wbr>e244672d5225 5<br>
> > 44 UP osc pgdata01-OST000c-osc-<wbr>ffff88204bab6400<br>
> > 996b1742-82eb-281c-c322-<wbr>e244672d5225 5<br>
> > 45 UP osc pgdata01-OST000e-osc-<wbr>ffff88204bab6400<br>
> > 996b1742-82eb-281c-c322-<wbr>e244672d5225 5<br>
> > 46 UP osc pgdata01-OST0010-osc-<wbr>ffff88204bab6400<br>
> > 996b1742-82eb-281c-c322-<wbr>e244672d5225 5<br>
> > 47 UP osc pgdata01-OST0013-osc-<wbr>ffff88204bab6400<br>
> > 996b1742-82eb-281c-c322-<wbr>e244672d5225 5<br>
> > 48 UP osc pgdata01-OST0015-osc-<wbr>ffff88204bab6400<br>
> > 996b1742-82eb-281c-c322-<wbr>e244672d5225 5<br>
> > 49 UP osc pgdata01-OST0017-osc-<wbr>ffff88204bab6400<br>
> > 996b1742-82eb-281c-c322-<wbr>e244672d5225 5<br>
> > 50 UP osc pgdata01-OST0014-osc-<wbr>ffff88204bab6400<br>
> > 996b1742-82eb-281c-c322-<wbr>e244672d5225 5<br>
> > 51 UP osc pgdata01-OST0016-osc-<wbr>ffff88204bab6400<br>
> > 996b1742-82eb-281c-c322-<wbr>e244672d5225 5<br>
> > 52 UP osc pgdata01-OST0018-osc-<wbr>ffff88204bab6400<br>
> > 996b1742-82eb-281c-c322-<wbr>e244672d5225 5<br>
> > 53 UP osc pgdata01-OST0019-osc-<wbr>ffff88204bab6400<br>
> > 996b1742-82eb-281c-c322-<wbr>e244672d5225 5<br>
> > 54 UP osc pgdata01-OST001a-osc-<wbr>ffff88204bab6400<br>
> > 996b1742-82eb-281c-c322-<wbr>e244672d5225 5<br>
> > 55 UP osc pgdata01-OST001b-osc-<wbr>ffff88204bab6400<br>
> > 996b1742-82eb-281c-c322-<wbr>e244672d5225 5<br>
> > 56 UP lov pghome01-clilov-<wbr>ffff88204bb50000<br>
> > 9ae8f2a9-1cdf-901f-160c-<wbr>66f70e4c10d1 4<br>
> > 57 UP lmv pghome01-clilmv-<wbr>ffff88204bb50000<br>
> > 9ae8f2a9-1cdf-901f-160c-<wbr>66f70e4c10d1 4<br>
> > 58 UP mdc pghome01-MDT0000-mdc-<wbr>ffff88204bb50000<br>
> > 9ae8f2a9-1cdf-901f-160c-<wbr>66f70e4c10d1 5<br>
> > 59 UP osc pghome01-OST0011-osc-<wbr>ffff88204bb50000<br>
> > 9ae8f2a9-1cdf-901f-160c-<wbr>66f70e4c10d1 5<br>
> > 60 UP osc pghome01-OST0012-osc-<wbr>ffff88204bb50000<br>
> > 9ae8f2a9-1cdf-901f-160c-<wbr>66f70e4c10d1 5<br>
> ><br>
> > Please help, any clues/advice/hints/tips are appricated<br>
> ><br>
> > --<br>
> ><br>
> > Vriendelijke groet,<br>
> ><br>
> > Ger Strikwerda<br>
> > Chef Special<br>
> > Rijksuniversiteit Groningen<br>
> > Centrum voor Informatie Technologie<br>
> > Unit Pragmatisch Systeembeheer<br>
> ><br>
> > Smitsborg<br>
> > Nettelbosje 1<br>
> > 9747 AJ Groningen<br>
> > Tel. 050 363 9276<br>
> ><br>
> > "God is hard, God is fair<br>
> > some men he gave brains, others he gave hair"<br>
> ><br>
> ><br>
> > ______________________________<wbr>_________________<br>
> > lustre-discuss mailing list<br>
> > <a href="mailto:lustre-discuss@lists.lustre.org" target="_blank">lustre-discuss@lists.lustre.<wbr>org</a><br>
> > <a rel="noreferrer" href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org" target="_blank">http://lists.lustre.org/<wbr>listinfo.cgi/lustre-discuss-<wbr>lustre.org</a><br>
> ><br>
><br>
><br>
><br>
> --<br>
> Vriendelijke groet,<br>
><br>
> Ger Strikwerda<br>
><br>
> Chef Special<br>
> Rijksuniversiteit Groningen<br>
> Centrum voor Informatie Technologie<br>
> Unit Pragmatisch Systeembeheer<br>
><br>
> Smitsborg<br>
> Nettelbosje 1<br>
> 9747 AJ Groningen<br>
> Tel. 050 363 9276<br>
><br>
><br>
> "God is hard, God is fair<br>
> some men he gave brains, others he gave hair"<br>
> ______________________________<wbr>_________________<br>
> lustre-discuss mailing list<br>
> <a href="mailto:lustre-discuss@lists.lustre.org" target="_blank">lustre-discuss@lists.lustre.<wbr>org</a><br>
> <a rel="noreferrer" href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org" target="_blank">http://lists.lustre.org/<wbr>listinfo.cgi/lustre-discuss-<wbr>lustre.org</a><br>
<br>
<br>
</div></div></blockquote></div><br><br clear="all"><br>-- <br><div class="m_-2824780214394110366m_-4169833405554715758m_-1807384931616815595m_6244912658785255702m_2516206972536374282m_-8670976717830459019gmail-m_-6199673089827782171m_-954142030923126256m_-4950728271923496188m_-2221363250000481128m_-72022338792146547gmail_signature"><div><div><div><div><div><div><div><div><div><div><div><pre cols="72"><span style="font-family:arial,helvetica,sans-serif"><font size="2">Vriendelijke groet, <br><br>Ger Strikwerda
</font></span><span style="font-family:arial,helvetica,sans-serif"><font size="2">Chef Special
Rijksuniversiteit Groningen
Centrum voor Informatie Technologie
Unit Pragmatisch Systeembeheer
Smitsborg
Nettelbosje 1
9747 AJ Groningen
Tel. 050 363 9276
<br>"God is hard, God is fair<br> some men he gave brains, others he gave hair"<br></font></span></pre></div></div></div></div></div></div></div></div></div></div></div></div>
</div>
______________________________<wbr>_________________<br>
lustre-discuss mailing list<br>
<a href="mailto:lustre-discuss@lists.lustre.org" target="_blank">lustre-discuss@lists.lustre.<wbr>org</a><br>
<a rel="noreferrer" href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org" target="_blank">http://lists.lustre.org/<wbr>listinfo.cgi/lustre-discuss-<wbr>lustre.org</a><br>
</blockquote></div>
</div></div></blockquote></div><br><br clear="all"><br>-- <br><div class="m_-2824780214394110366m_-4169833405554715758m_-1807384931616815595m_6244912658785255702m_2516206972536374282m_-8670976717830459019gmail-m_-6199673089827782171m_-954142030923126256m_-4950728271923496188gmail_signature"><div><div><div><div><div><div><div><div><div><div><div><pre cols="72"><span style="font-family:arial,helvetica,sans-serif"><font size="2">Vriendelijke groet, <br><br>Ger Strikwerda
</font></span><span style="font-family:arial,helvetica,sans-serif"><font size="2">Chef Special
Rijksuniversiteit Groningen
Centrum voor Informatie Technologie
Unit Pragmatisch Systeembeheer
Smitsborg
Nettelbosje 1
9747 AJ Groningen
Tel. 050 363 9276
<br>"God is hard, God is fair<br> some men he gave brains, others he gave hair"<br></font></span></pre></div></div></div></div></div></div></div></div></div></div></div></div>
</div>
</div></div><br>______________________________<wbr>_________________<br>
lustre-discuss mailing list<br>
<a href="mailto:lustre-discuss@lists.lustre.org" target="_blank">lustre-discuss@lists.lustre.<wbr>org</a><br>
<a rel="noreferrer" href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org" target="_blank">http://lists.lustre.org/<wbr>listinfo.cgi/lustre-discuss-<wbr>lustre.org</a><br>
<br></blockquote></div></div></div><br></div></div>
</blockquote></div><br><br clear="all"><br>-- <br><div class="m_-2824780214394110366m_-4169833405554715758m_-1807384931616815595m_6244912658785255702m_2516206972536374282m_-8670976717830459019gmail-m_-6199673089827782171gmail_signature"><div><div><div><div><div><div><div><div><div><div><div><pre cols="72"><span style="font-family:arial,helvetica,sans-serif"><font size="2">Vriendelijke groet, <br><br>Ger Strikwerda
</font></span><span style="font-family:arial,helvetica,sans-serif"><font size="2">Chef Special
Rijksuniversiteit Groningen
Centrum voor Informatie Technologie
Unit Pragmatisch Systeembeheer
Smitsborg
Nettelbosje 1
9747 AJ Groningen
Tel. 050 363 9276
<br>"God is hard, God is fair<br> some men he gave brains, others he gave hair"<br></font></span></pre></div></div></div></div></div></div></div></div></div></div></div></div>
</div>
</div></div></blockquote></div></div></div><br></div></div>
</blockquote></div><br><br clear="all"><br>-- <br><div class="m_-2824780214394110366m_-4169833405554715758m_-1807384931616815595m_6244912658785255702m_2516206972536374282gmail_signature" data-smartmail="gmail_signature"><div><div><div><div><div><div><div><div><div><div><div><pre cols="72"><span style="font-family:arial,helvetica,sans-serif"><font size="2">Vriendelijke groet, <br><br>Ger Strikwerda
</font></span><span style="font-family:arial,helvetica,sans-serif"><font size="2">Chef Special
Rijksuniversiteit Groningen
Centrum voor Informatie Technologie
Unit Pragmatisch Systeembeheer
Smitsborg
Nettelbosje 1
9747 AJ Groningen
Tel. 050 363 9276
<br>"God is hard, God is fair<br> some men he gave brains, others he gave hair"<br></font></span></pre></div></div></div></div></div></div></div></div></div></div></div></div>
</div>
</blockquote></div>
</div></div></blockquote></div><br><br clear="all"><br>-- <br><div class="m_-2824780214394110366m_-4169833405554715758m_-1807384931616815595gmail_signature" data-smartmail="gmail_signature"><div><div><div><div><div><div><div><div><div><div><div><pre cols="72"><span style="font-family:arial,helvetica,sans-serif"><font size="2">Vriendelijke groet, <br><br>Ger Strikwerda
</font></span><span style="font-family:arial,helvetica,sans-serif"><font size="2">Chef Special
Rijksuniversiteit Groningen
Centrum voor Informatie Technologie
Unit Pragmatisch Systeembeheer
Smitsborg
Nettelbosje 1
9747 AJ Groningen
Tel. 050 363 9276
<br>"God is hard, God is fair<br> some men he gave brains, others he gave hair"<br></font></span></pre></div></div></div></div></div></div></div></div></div></div></div></div>
</div>
</blockquote></div></div>
</div></div></blockquote></div><br><br clear="all"><br>-- <br><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><pre cols="72"><span style="font-family:arial,helvetica,sans-serif"><font size="2">Vriendelijke groet, <br><br>Ger Strikwerda
</font></span><span style="font-family:arial,helvetica,sans-serif"><font size="2">Chef Special
Rijksuniversiteit Groningen
Centrum voor Informatie Technologie
Unit Pragmatisch Systeembeheer
Smitsborg
Nettelbosje 1
9747 AJ Groningen
Tel. 050 363 9276
<br>"God is hard, God is fair<br> some men he gave brains, others he gave hair"<br></font></span></pre></div></div></div></div></div></div></div></div></div></div></div></div>
</div>