<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=us-ascii">
<meta name=Generator content="Microsoft Word 11 (filtered medium)">
<style>
<!--
/* Font Definitions */
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman";}
a:link, span.MsoHyperlink
{color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:Arial;
color:windowtext;}
@page Section1
{size:8.5in 11.0in;
margin:1.0in 1.25in 1.0in 1.25in;}
div.Section1
{page:Section1;}
-->
</style>
</head>
<body lang=EN-US link=blue vlink=purple>
<div class=Section1>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>Hello There,<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>I am trying to wet my feet by diving into the waters of
Lustre File System and I am having some problems that I need help with. I have
3 physical servers and this is what I have installed on all 3 servers.
Let’s name them S1, S2, S3 for now.<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 color=red face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:red'>S1<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color=red face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:red'>I have dual port IB card and here is
network config for each port<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color=red face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:red'>ib0 – 192.168.100.100<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color=red face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:red'>ib1 – 172.16.100.100<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color=red face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:red'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 color=red face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:red'>Kernel-2.6.18-194.3.1.el5_lustre<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color=red face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:red'>Lustre-modules<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color=red face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:red'>Lustre-ldiskfs<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color=red face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:red'>Lustre-1.8.4-<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color=red face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:red'>E2fsprogrs<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color=red face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:red'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 color=red face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:red'>Here is the /etc/modprobe.conf file<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color=red face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:red'>options lnet
forwarding=”enabled”<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color=red face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:red'>options lnet accept=all<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color=red face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:red'>options lnet
networks=”o2ib0(ib0),o2ib1(ib1)”<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color=red face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:red'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 color=red face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:red'>I have partitioned /dev/sda3 and /dev/sda4
on this server as mgs/mdt and ost filesystem respectively.<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 color=blue face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:blue'>S2<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color=blue face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:blue'>I have one port IB card and here is the
network config for that port. I have connected this port directly to ib0 of S1
server.<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color=blue face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:blue'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 color=blue face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:blue'>ib0 – 192.168.100.101<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color=blue face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:blue'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 color=blue face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:blue'>Here is the /etc/modprobe.conf file<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color=blue face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:blue'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 color=blue face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:blue'>options lnet
networks=”o2ib0(ib0)”<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color=blue face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:blue'>options lnet routes=”o2ib1 <a
href="mailto:192.168.100.100@o2ib0"
title="blocked::mailto:192.168.100.100@o2ib0">192.168.100.100 at o2ib0</a>”<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color=blue face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:blue'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 color=blue face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:blue'>When I run cat /proc/sys/lnet/routers I
get following output<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color=blue face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:blue'>ref
rtr_ref
alive_cnt
state
last_ping ping_sent
deadline
down_ni
router<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color=blue face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:blue'>3
1
4
up
4303108
1
NA
-2
<a href="mailto:192.168.100.100@o2ib"
title="blocked::mailto:192.168.100.100@o2ib">192.168.100.100 at o2ib</a><o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color=blue face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:blue'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 color=blue face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:blue'>When I run lctl ping <a
href="mailto:192.168.100.100@o2ib0"
title="blocked::mailto:192.168.100.100@o2ib0">192.168.100.100 at o2ib0</a> , I
get following output<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color=blue face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:blue'>12345-0@lo<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color=blue face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:blue'><a href="mailto:12345-192.168.100.100@o2ib"
title="blocked::mailto:12345-192.168.100.100@o2ib">12345-192.168.100.100@o2ib</a><o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color=blue face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:blue'><a href="mailto:12345-172.16.100.100@o2ib1"
title="blocked::mailto:12345-172.16.100.100@o2ib1">12345-172.16.100.100@o2ib1</a><o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 color="#993366" face=Arial><span
style='font-size:10.0pt;font-family:Arial;color:#993366'>S3<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color="#993366" face=Arial><span
style='font-size:10.0pt;font-family:Arial;color:#993366'>I have one port IB
card and here is the network config for that port. I have connected this port
directly to ib1 of S1 server<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color="#993366" face=Arial><span
style='font-size:10.0pt;font-family:Arial;color:#993366'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 color="#993366" face=Arial><span
style='font-size:10.0pt;font-family:Arial;color:#993366'>ib0 –
172.16.100.101<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color="#993366" face=Arial><span
style='font-size:10.0pt;font-family:Arial;color:#993366'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 color="#993366" face=Arial><span
style='font-size:10.0pt;font-family:Arial;color:#993366'>Here is the
/etc/modprobe.conf file<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color="#993366" face=Arial><span
style='font-size:10.0pt;font-family:Arial;color:#993366'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 color="#993366" face=Arial><span
style='font-size:10.0pt;font-family:Arial;color:#993366'>options lnet
networks=”o2ib1(ib0)”<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color="#993366" face=Arial><span
style='font-size:10.0pt;font-family:Arial;color:#993366'>options lnet
routes=”o2ib0 172.16.100.100 at o2ib1”<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color="#993366" face=Arial><span
style='font-size:10.0pt;font-family:Arial;color:#993366'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 color="#993366" face=Arial><span
style='font-size:10.0pt;font-family:Arial;color:#993366'>When I run cat /proc/sys/lnet/routers
I get following output<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color="#993366" face=Arial><span
style='font-size:10.0pt;font-family:Arial;color:#993366'>ref
rtr_ref
alive_cnt
state
last_ping ping_sent
deadline
down_ni
router<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color="#993366" face=Arial><span
style='font-size:10.0pt;font-family:Arial;color:#993366'>3
1
2
up
4297593
1
NA
-2
<a href="mailto:192.168.100.100@o2ib"
title="blocked::mailto:192.168.100.100@o2ib"><font color="#993366"
title="blocked::mailto:192.168.100.100@o2ib"><span
title="blocked::mailto:192.168.100.100@o2ib"><span style='color:#993366'>172.16.100.100
at o2ib</span></span></font></a>1<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color="#993366" face=Arial><span
style='font-size:10.0pt;font-family:Arial;color:#993366'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 color="#993366" face=Arial><span
style='font-size:10.0pt;font-family:Arial;color:#993366'>When I run lctl ping
172.16.100.100 at o2ib1 , I get following output<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color="#993366" face=Arial><span
style='font-size:10.0pt;font-family:Arial;color:#993366'>12345-0@lo<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color="#993366" face=Arial><span
style='font-size:10.0pt;font-family:Arial;color:#993366'><a
href="mailto:12345-192.168.100.100@o2ib"
title="blocked::mailto:12345-192.168.100.100@o2ib"><font color="#993366"
title="blocked::mailto:12345-192.168.100.100@o2ib"><span
title="blocked::mailto:12345-192.168.100.100@o2ib"><span style='color:#993366'>12345-192.168.100.100@o2ib</span></span></font></a><o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color="#993366" face=Arial><span
style='font-size:10.0pt;font-family:Arial;color:#993366'><a
href="mailto:12345-172.16.100.100@o2ib1"
title="blocked::mailto:12345-172.16.100.100@o2ib1"><font color="#993366"
title="blocked::mailto:12345-172.16.100.100@o2ib1"><span
title="blocked::mailto:12345-172.16.100.100@o2ib1"><span style='color:#993366'>12345-172.16.100.100@o2ib1</span></span></font></a><o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>Now my problem is to run some network tests from S2 </span></font><font
size=2 face=Wingdings><span style='font-size:10.0pt;font-family:Wingdings'>à</span></font><font
size=2 face=Arial><span style='font-size:10.0pt;font-family:Arial'> S3 and S3 </span></font><font
size=2 face=Wingdings><span style='font-size:10.0pt;font-family:Wingdings'>à</span></font><font
size=2 face=Arial><span style='font-size:10.0pt;font-family:Arial'> S2 to
measure the bandwidth but somehow both S2 and S3 complain that network is
unreachable. What am I doing wrong?<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>Thanks<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>Nihir<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'><o:p> </o:p></span></font></p>
</div>
</body>
</html>