<div dir="ltr"><div><div><div><div><div><div>WRT Subject: lctl ping node28@o2ib report Input/output error<br><br></div>Hello Yu,<br><br></div>Just to check the obvious,<br></div>-- the recipient system (node28) is running lnet (an "lsmod | grep lnet" returns the appropriate modules, for example)<br></div>-- there is nothing along the path which might be blocking Lustre port 998<br><br></div>Cheers,<br></div>megan<br></div><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Jun 29, 2018 at 4:19 PM, <span dir="ltr"><<a href="mailto:lustre-discuss-request@lists.lustre.org" target="_blank">lustre-discuss-request@lists.lustre.org</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Send lustre-discuss mailing list submissions to<br>
<a href="mailto:lustre-discuss@lists.lustre.org">lustre-discuss@lists.lustre.<wbr>org</a><br>
<br>
To subscribe or unsubscribe via the World Wide Web, visit<br>
<a href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org" rel="noreferrer" target="_blank">http://lists.lustre.org/<wbr>listinfo.cgi/lustre-discuss-<wbr>lustre.org</a><br>
or, via email, send a message with subject or body 'help' to<br>
<a href="mailto:lustre-discuss-request@lists.lustre.org">lustre-discuss-request@lists.<wbr>lustre.org</a><br>
<br>
You can reach the person managing the list at<br>
<a href="mailto:lustre-discuss-owner@lists.lustre.org">lustre-discuss-owner@lists.<wbr>lustre.org</a><br>
<br>
When replying, please edit your Subject line so it is more specific<br>
than "Re: Contents of lustre-discuss digest..."<br>
<br>
<br>
Today's Topics:<br>
<br>
1. Re: lctl ping node28@o2ib report Input/output error (Cory Spitz)<br>
<br>
<br>
------------------------------<wbr>------------------------------<wbr>----------<br>
<br>
Message: 1<br>
Date: Fri, 29 Jun 2018 16:14:18 +0000<br>
From: Cory Spitz <<a href="mailto:spitzcor@cray.com">spitzcor@cray.com</a>><br>
To: Andreas Dilger <<a href="mailto:adilger@whamcloud.com">adilger@whamcloud.com</a>>, yu sun<br>
<<a href="mailto:sunyu1949@gmail.com">sunyu1949@gmail.com</a>><br>
Cc: "<a href="mailto:lustre-discuss@lists.lustre.org">lustre-discuss@lists.lustre.<wbr>org</a>"<br>
<<a href="mailto:lustre-discuss@lists.lustre.org">lustre-discuss@lists.lustre.<wbr>org</a>><br>
Subject: Re: [lustre-discuss] lctl ping node28@o2ib report<br>
Input/output error<br>
Message-ID: <<a href="mailto:AC964404-78C4-4F0F-B894-7619464AFF90@cray.com">AC964404-78C4-4F0F-B894-<wbr>7619464AFF90@cray.com</a>><br>
Content-Type: text/plain; charset="utf-8"<br>
<br>
FYI, there is a helpful guide to LNet setup at <a href="http://wiki.lustre.org/LNet_Router_Config_Guide" rel="noreferrer" target="_blank">http://wiki.lustre.org/LNet_<wbr>Router_Config_Guide</a>. Despite the title, it is applicable to non-routed cases as well.<br>
-Cory<br>
<br>
-- <br>
<br>
?On 6/29/18, 1:06 AM, "lustre-discuss on behalf of Andreas Dilger" <<a href="mailto:lustre-discuss-bounces@lists.lustre.org">lustre-discuss-bounces@lists.<wbr>lustre.org</a> on behalf of <a href="mailto:adilger@whamcloud.com">adilger@whamcloud.com</a>> wrote:<br>
<br>
On Jun 28, 2018, at 21:14, yu sun <<a href="mailto:sunyu1949@gmail.com">sunyu1949@gmail.com</a>> wrote:<br>
> <br>
> all server and client that fore-mentioned is using netmasks 255.255.255.224. and they can ping with each other, for example:<br>
> <br>
> root@ml-gpu-ser200.nmg01:~$ ping node28<br>
> PING node28 (10.82.143.202) 56(84) bytes of data.<br>
> 64 bytes from node28 (10.82.143.202): icmp_seq=1 ttl=61 time=0.047 ms<br>
> 64 bytes from node28 (10.82.143.202): icmp_seq=2 ttl=61 time=0.028 ms<br>
> <br>
> --- node28 ping statistics ---<br>
> 2 packets transmitted, 2 received, 0% packet loss, time 999ms<br>
> rtt min/avg/max/mdev = 0.028/0.037/0.047/0.011 ms<br>
> root@ml-gpu-ser200.nmg01:~$ lctl ping node28@o2ib1<br>
> failed to ping 10.82.143.202@o2ib1: Input/output error<br>
> root@ml-gpu-ser200.nmg01:~$<br>
> <br>
> and we also have hundreds of GPU machines with different IP Subnet, they are in service and it's difficulty to change the network structure. so any material or document can guide me solve this by don't change network structure.<br>
<br>
The regular IP "ping" is being routed by an IP router, but that doesn't<br>
work with IB networks, AFAIK. The IB interfaces need to be on the same<br>
subnet, you need to have an IB interface on each subnet configured on<br>
each subnet (which might get ugly if you have a large number of subnets)<br>
or you need to use LNet routers that are connected to each IB subnet to<br>
do the routing (each subnet would be a separate LNet network, for example<br>
10.82.142.202@o2ib23 or whatever).<br>
<br>
The other option would be to use the IPoIB layer with socklnd (e.g.<br>
10.82.142.202@tcp) but this would not run as fast as native verbs.<br>
<br>
Cheers, Andreas<br>
<br>
<br>
> Mohr Jr, Richard Frank (Rick Mohr) <<a href="mailto:rmohr@utk.edu">rmohr@utk.edu</a>> ?2018?6?29??? ??3:30???<br>
> <br>
> > On Jun 27, 2018, at 4:44 PM, Mohr Jr, Richard Frank (Rick Mohr) <<a href="mailto:rmohr@utk.edu">rmohr@utk.edu</a>> wrote:<br>
> ><br>
> ><br>
> >> On Jun 27, 2018, at 3:12 AM, yu sun <<a href="mailto:sunyu1949@gmail.com">sunyu1949@gmail.com</a>> wrote:<br>
> >><br>
> >> client:<br>
> >> root@ml-gpu-ser200.nmg01:~$ mount -t lustre node28@o2ib1:node29@o2ib1:/<wbr>project /mnt/lustre_data<br>
> >> mount.lustre: mount node28@o2ib1:node29@o2ib1:/<wbr>project at /mnt/lustre_data failed: Input/output error<br>
> >> Is the MGS running?<br>
> >> root@ml-gpu-ser200.nmg01:~$ lctl ping node28@o2ib1<br>
> >> failed to ping 10.82.143.202@o2ib1: Input/output error<br>
> >> root@ml-gpu-ser200.nmg01:~$<br>
> ><br>
> > In your previous email, you said that you could mount lustre on the client ml-gpu-ser200.nmg01. Was that not accurate, or did something change in the meantime?<br>
> <br>
> (Note: Received out-of-band reply from Yu stating that there was a typo in the previous email, and that client ml-gpu-ser200.nmg01 could not mount lustre. Continuing discussion here so others on list can follow/benefit.)<br>
> <br>
> Yu,<br>
> <br>
> For the IPoIB addresses used on your nodes, what are the subnets (and netmasks) that you are using? It looks like servers use 10.82.143.X and clients use 10.82.141.X. If you are using a 255.255.0.0 netmask, you should be fine. But if you are using 255.255.255.0, then you will run into problems. Lustre expects that all nodes on the same lnet network (o2ib1 in your case) will also be on the same IP subnet.<br>
> <br>
> Have you tried running a regular ?ping <IPoIB_address>? command between clients and servers to make sure that part is working?<br>
> <br>
> --<br>
> Rick Mohr<br>
> Senior HPC System Administrator<br>
> National Institute for Computational Sciences<br>
> <a href="http://www.nics.tennessee.edu" rel="noreferrer" target="_blank">http://www.nics.tennessee.edu</a><br>
> <br>
> ______________________________<wbr>_________________<br>
> lustre-discuss mailing list<br>
> <a href="mailto:lustre-discuss@lists.lustre.org">lustre-discuss@lists.lustre.<wbr>org</a><br>
> <a href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org" rel="noreferrer" target="_blank">http://lists.lustre.org/<wbr>listinfo.cgi/lustre-discuss-<wbr>lustre.org</a><br>
<br>
Cheers, Andreas<br>
---<br>
Andreas Dilger<br>
Principal Lustre Architect<br>
Whamcloud<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
------------------------------<br>
<br>
Subject: Digest Footer<br>
<br>
______________________________<wbr>_________________<br>
lustre-discuss mailing list<br>
<a href="mailto:lustre-discuss@lists.lustre.org">lustre-discuss@lists.lustre.<wbr>org</a><br>
<a href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org" rel="noreferrer" target="_blank">http://lists.lustre.org/<wbr>listinfo.cgi/lustre-discuss-<wbr>lustre.org</a><br>
<br>
<br>
------------------------------<br>
<br>
End of lustre-discuss Digest, Vol 147, Issue 43<br>
******************************<wbr>*****************<br>
</blockquote></div><br></div>