<div dir="ltr"><div>thanks robin, sorry for my later reply.</div><div><br></div><div>[root@bigdata-dlp-server00 ~]# salt "ml-storage-ser2[0-9].nmg01" cmd.run "lctl list_nids"</div><div>ml-storage-ser28.nmg01: (node28)</div><div> 10.82.143.202@o2ib1</div><div> 10.83.162.19@tcp1</div><div>ml-storage-ser25.nmg01: (node25)</div><div> 10.83.162.16@tcp1</div><div> 10.82.143.199@o2ib1</div><div>ml-storage-ser20.nmg01: (node20)</div><div> 10.82.143.194@o2ib1</div><div> 10.83.162.11@tcp1</div><div>ml-storage-ser24.nmg01:<span style="background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">(node24)</span></div><div> 10.82.143.198@o2ib1</div><div> 10.83.162.15@tcp1</div><div>ml-storage-ser29.nmg01:<span style="background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">(node29)</span></div><div> 10.83.162.20@tcp1</div><div> 10.82.143.203@o2ib1</div><div>ml-storage-ser22.nmg01: <span style="background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">(node22)</span></div><div> 10.82.143.196@o2ib1</div><div> 10.83.162.13@tcp1</div><div>ml-storage-ser27.nmg01: <span style="background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">(node27)</span></div><div> 10.83.162.18@tcp1</div><div> 10.82.143.201@o2ib1</div><div>ml-storage-ser23.nmg01: <span style="background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">(node23)</span></div><div> 10.83.162.14@tcp1</div><div> 10.82.143.197@o2ib1</div><div>ml-storage-ser26.nmg01: <span style="background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">(node26)</span></div><div> 10.82.143.200@o2ib1</div><div> 10.83.162.17@tcp1</div><div>ml-storage-ser21.nmg01: <span style="background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">(node21)</span></div><div> 10.83.162.12@tcp1</div><div> 10.82.143.195@o2ib1</div><div><br></div><div><div>root@ml-gpu-ser200.nmg01:~$ lctl list_nids</div><div>10.82.141.208@o2ib1</div><div>10.83.152.55@tcp1</div><div>root@ml-gpu-ser200.nmg01:~$ lctl ping node28@o2ib1</div><div>failed to ping 10.82.143.202@o2ib1: Input/output error</div><div>root@ml-gpu-ser200.nmg01:~$</div></div><div><br></div><div>I have create file /etc/modprobe.d/lustre.conf with content on all mdt ost and client:</div><div><div>root@ml-gpu-ser200.nmg01:~$ cat /etc/modprobe.d/lustre.conf</div><div>options lnet networks="o2ib1(eth3.2)"</div></div><div>and I exec command line : lnetctl lnet configure --all to make my static lnet configuration take effect. but i still can't ping node28 from my client <span style="background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">ml-gpu-ser200.nmg01. I can mount as well as access lustre on <span style="text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"> client </span><span style="text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">ml-gpu-ser200.nmg01. </span></span></div><div><br></div><div>And I can lctl ping node28@o2ib successfully from other mdt or ost nodes, such as:</div><div><div>root@ml-storage-ser26.nmg01:/home/odin/sunyuyusun$ lctl ping node28@o2ib1</div><div>12345-0@lo</div><div>12345-10.82.143.202@o2ib1</div><div>12345-10.83.162.19@tcp1</div><div>root@ml-storage-ser26.nmg01:/home/odin/sunyuyusun$ lctl ping node20@o2ib1</div><div>12345-0@lo</div><div>12345-10.82.143.194@o2ib1</div><div>12345-10.83.162.11@tcp1</div><div>root@ml-storage-ser26.nmg01:/home/odin/sunyuyusun$ lctl ping node21@o2ib1</div><div>12345-0@lo</div><div>12345-10.83.162.12@tcp1</div><div>12345-10.82.143.195@o2ib1</div><div>root@ml-storage-ser26.nmg01:/home/odin/sunyuyusun$ lctl ping node22@o2ib1<br></div><div>12345-0@lo</div><div>12345-10.82.143.196@o2ib1</div><div>12345-10.83.162.13@tcp1</div></div><div><br></div><div>so what lnet configuration should I set to solve this problem?</div><div><br></div><div>Thanks very much .</div><div>Yours</div><div>Yu</div></div><br><div class="gmail_quote"><div dir="ltr">Robin Humble <<a href="mailto:rjh%2Blustre@cita.utoronto.ca">rjh+lustre@cita.utoronto.ca</a>> 于2018年6月26日周二 下午10:48写道：<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">On Tue, Jun 26, 2018 at 04:05:14PM +0800, yu sun wrote:<br>
>hi all:<br>
> I want to build a lustre storage system, and I found not all of the<br>
>machine in the same sub-network, and they cant lctl ping with each other.<br>
>the details list as below:<br>
><br>
>root@ml-storage-ser30.nmg01:~$ lctl list_nids<br>
>10.82.145.2@o2ib<br>
>root@ml-storage-ser30.nmg01:~$ lctl ping node28@o2ib<br>
>failed to ping 10.82.143.202@o2ib: Input/output error<br>
>root@ml-storage-ser30.nmg01:~$<br>
<br>
what does 'lctl list_nids' say on node28?<br>
also disable iptables everywhere.<br>
<br>
cheers,<br>
robin<br>
</blockquote></div>