<strong>Hi, I get this:</strong><br>***************************************************************************<br>[root@x-math20 ~]# lctl list_nids<br>132.66.176.211@tcp<br>[root@x-math20 ~]# ifconfig -a<br>eth0 Link encap:Ethernet HWaddr 00:02:B3:2D:A6:BF
<br> inet addr:<a href="http://132.66.176.211">132.66.176.211</a> Bcast:<a href="http://132.66.255.255">132.66.255.255</a> Mask:<a href="http://255.255.0.0">255.255.0.0</a><br> inet6 addr: fe80::202:b3ff:fe2d:a6bf/64 Scope:Link
<br> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1<br> RX packets:9448397 errors:0 dropped:0 overruns:0 frame:0<br> TX packets:194259 errors:0 dropped:0 overruns:0 carrier:0<br> collisions:0 txqueuelen:1000
<br> RX bytes:1171910501 (1.0 GiB) TX bytes:40500450 (38.6 MiB)<br><br>lo Link encap:Local Loopback<br> inet addr:<a href="http://127.0.0.1">127.0.0.1</a> Mask:<a href="http://255.0.0.0">255.0.0.0
</a><br> inet6 addr: ::1/128 Scope:Host<br> UP LOOPBACK RUNNING MTU:16436 Metric:1<br> RX packets:8180 errors:0 dropped:0 overruns:0 frame:0<br> TX packets:8180 errors:0 dropped:0 overruns:0 carrier:0
<br> collisions:0 txqueuelen:0<br> RX bytes:3335243 (3.1 MiB) TX bytes:3335243 (3.1 MiB)<br><br>sit0 Link encap:IPv6-in-IPv4<br> NOARP MTU:1480 Metric:1<br> RX packets:0 errors:0 dropped:0 overruns:0 frame:0
<br> TX packets:0 errors:0 dropped:0 overruns:0 carrier:0<br> collisions:0 txqueuelen:0<br> RX bytes:0 (0.0 b) TX bytes:0 (0.0 b)<br><br>[root@x-math20 ~]# cat /etc/modprobe.conf<br>alias eth0 e100
<br>alias usb-controller uhci-hcd<br>alias scsi_hostadapter ata_piix<br>alias lustre llite<br>options lnet networks=tcp0<br>[root@x-math20 ~]#<br><br>***********************************************************************************************************8
<br><br><div><span class="gmail_quote">On 1/2/08, <b class="gmail_sendername">Aaron Knister</b> <<a href="mailto:aaron@iges.org">aaron@iges.org</a>> wrote:</span><blockquote class="gmail_quote" style="margin-top: 0; margin-right: 0; margin-bottom: 0; margin-left: 0; margin-left: 0.80ex; border-left-color: #cccccc; border-left-width: 1px; border-left-style: solid; padding-left: 1ex">
<div><div>On the host x-math20 could you run an "lctl list_nids" and also an "ifconfig -a". I want to see if lnet is listening on the correct interface. Oh could you also post the contents of your /etc/modprobe.conf.
</div><div><br></div><div>Thanks!</div><div><br></div><div>-Aaron</div><div><span class="e" id="q_1173aab74eb630e0_1"><br><div><div>On Jan 2, 2008, at 4:42 AM, Avi Gershon wrote:</div><br><blockquote type="cite">Hello to every one and happy new year..
<br>I think I have reduce my problem to this: lctl ping 132.66.176.211@tcp0 don't work for me for some strange reason <br>as you can see:<br>***********************************************************************************
<br>[root@x-math20 ~]# lctl ping 132.66.176.211@tcp0<br>failed to ping 132.66.176.211@tcp: Input/output error<br>[root@x-math20 ~]# ping <a href="http://132.66.176.211" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)">
132.66.176.211</a><br>PING <a href="http://132.66.176.211" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)"> 132.66.176.211</a> (<a href="http://132.66.176.211" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)">
132.66.176.211</a>) 56(84) bytes of data.<br>64 bytes from <a href="http://132.66.176.211" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)">132.66.176.211</a>: icmp_seq=0 ttl=64 time=0.152 ms<br>64 bytes from
<a href="http://132.66.176.211" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)"> 132.66.176.211</a>: icmp_seq=1 ttl=64 time=0.130 ms<br>64 bytes from <a href="http://132.66.176.211" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)">
132.66.176.211</a>: icmp_seq=2 ttl=64 time=0.131 m<br>--- <a href="http://132.66.176.211" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)">132.66.176.211</a> ping statistics --- <br>3 packets transmitted, 3 received, 0% packet loss, time 2018ms
<br>rtt min/avg/max/mdev = 0.130/0.137/0.152/0.016 ms, pipe 2<br>[root@x-math20 ~]#<br>***************************************************************************************** <br><br><br><div><span class="gmail_quote">On 12/24/07,
<b class="gmail_sendername">Avi Gershon</b> <<a href="mailto:gershonavi@gmail.com" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)">gershonavi@gmail.com</a>> wrote:</span><blockquote class="gmail_quote" style="margin-top: 0; margin-right: 0; margin-bottom: 0; margin-left: 0; margin-left: 0.80ex; border-left-color: #cccccc; border-left-width: 1px; border-left-style: solid; padding-left: 1ex">
<div><font color="#ff0000">Hi, </font></div><div><font color="#ff0000">here is the "iptables -L " results:</font><br> </div><div><font style="background-color: #ffff33"> NODE 1 </font><a href="mailto:root@132.66.176.212" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)">
<font style="background-color: #ffff33">132.66.176.212</font></a><span><br>Scientific Linux CERN SLC release 4.6 (Beryllium)<br></span><span><a href="mailto:root@132.66.176.212's" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)">
root@132.66.176.212's</a> password:<br></span>Last login: Sun Dec 23 22:01:18 2007 from <a href="http://x-fishelov.tau.ac.il" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)">x-fishelov.tau.ac.il
</a><br>[root@localhost ~]#<br>[root@localhost ~]#<span><br>[root@localhost ~]# iptables -L <br>Chain INPUT (policy ACCEPT)<br>target prot opt source destination</span></div><span><p> Chain FORWARD (policy ACCEPT)
<br>target prot opt source destination</p><div>Chain OUTPUT (policy ACCEPT)<br>target prot opt source destination</div></span><div>************************************************************************************************
<br><font style="background-color: #ffff33"> MDT </font><font style="background-color: #ffff33"><a href="http://132.66.176.211" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)"> 132.66.176.211</a></font>
</div><div><br>Last login: Mon Dec 24 11:51:57 2007 from <a href="http://dynamic136-91.tau.ac.il" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)">dynamic136-91.tau.ac.il</a><br>[root@x-math20 ~]# iptables -L
<span><br>Chain INPUT (policy ACCEPT)<br>target prot opt source destination </span></div><span><p>Chain FORWARD (policy ACCEPT)<br>target prot opt source destination </p></span><div><span>
Chain OUTPUT (policy ACCEPT)<br>target prot opt source destination<br></span>*************************************************************************</div><div><br><font style="background-color: #ffff33">
NODE 2 </font><a href="mailto:root@132.66.176.215" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)"><font style="background-color: #ffff33">132.66.176.215</font></a><br>Last login: Mon Dec 24 11:01:22 2007 from
<a href="http://erezlab.tau.ac.il" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)"> erezlab.tau.ac.il</a><br>[root@x-mathr11 ~]# iptables -L<div><span><br>Chain INPUT (policy ACCEPT) <br>target prot opt source destination
<br>RH-Firewall-1-INPUT all -- anywhere anywhere</span></div></div><div><span><p>Chain FORWARD (policy ACCEPT)<br> target prot opt source destination<br>RH-Firewall-1-INPUT all -- anywhere anywhere
</p><p>Chain OUTPUT (policy ACCEPT)<br>target prot opt source destination</p><div>Chain RH-Firewall-1-INPUT (2 references) <br>target prot opt source destination<br>ACCEPT all -- anywhere anywhere
<br>ACCEPT icmp -- anywhere anywhere icmp any <br>ACCEPT ipv6-crypt-- anywhere anywhere <br>ACCEPT ipv6-auth-- anywhere anywhere<br>ACCEPT udp -- anywhere
<a href="http://224.0.0.251" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)">224.0.0.251</a> udp dpt:5353 <br>ACCEPT udp -- anywhere anywhere udp dpt:ipp<br>ACCEPT all -- anywhere anywhere state RELATED,ESTAB
<br>LISHED<br>ACCEPT tcp -- anywhere anywhere state NEW tcp dpts: <br>30000:30101<br>ACCEPT tcp -- anywhere anywhere state NEW tcp dpt:s<br>sh<br>ACCEPT udp -- anywhere anywhere state NEW udp dpt:a
<br>fs3-callback<br>REJECT all -- anywhere anywhere reject-with icmp-ho <br>st-prohibited<br>[root@x-mathr11 ~]#<br><br>************************************************************</div></span>
</div><div><font color="#ff0000">one more thing....</font></div><div><font color="#ff0000">Do you use TCP protocol? or do you use UDP? </font></div><div> </div><div><font color="#ff0000">Regards Avi, </font></div><div><font color="#ff0000">
P.S I think a beginning of a beautiful friendship.. :-)</font></div><div><span><div><br><br><br></div><div class="gmail_quote">On Dec 24, 2007 5:29 PM, Aaron Knister <<a href="mailto:aaron@iges.org" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)">
aaron@iges.org</a>> wrote: <br><blockquote class="gmail_quote" style="padding-left: 1ex; margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0.80ex; border-left-color: #cccccc; border-left-width: 1px; border-left-style: solid">
That sounds like quite a task! Could you show me the contents of your <br>firewall rules on the systems mentioned below? (iptables -L) on each. <br>That would help to diagnose the problem further.<br><font color="#888888">
<br>-Aaron<br></font><div><div><br>On Dec 24, 2007, at 1:21 AM, Yan Benhammou wrote: <br><br>> Hi Aaron and thank you for you fast answwers.<br>> We are working (Avi,Meny and me) on the israeli GRID and we need to<br>
> create a single huge file system for this GRID. <br>> cheers<br>> Yan <br>><br>> ________________________________<br>><br>> From: Aaron Knister [mailto:<a href="mailto:aaron@iges.org" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)">
aaron@iges.org</a>]<br>> Sent: Sun 12/23/2007 8:27 PM <br>> To: Avi Gershon<br>> Cc: <a href="mailto:lustre-discuss@clusterfs.com" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)">lustre-discuss@clusterfs.com
</a>; Yan Benhammou; Meny Ben moshe<br> > Subject: Re: [Lustre-discuss] help needed.<br>><br>><br>> Can you check the firewall on each of those machines ( iptables -L ) <br>> and paste that here. Also, is this network dedicated to Lustre?
<br>> Lustre can easily saturate a network interface under load to the <br>> point it becomes difficult to login to a node if it only has one<br> > interface. I'd recommend using a different interface if you can.
<br>><br>> On Dec 23, 2007, at 11:03 AM, Avi Gershon wrote:<br>> <br>><br>> node 1 <a href="http://132.66.176.212/" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)"> 132.66.176.212
</a> <<a href="http://132.66.176.212/" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)"> http://132.66.176.212/</a>><br>> node 2 <a href="http://132.66.176.215/" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)">
132.66.176.215</a> <<a href="http://132.66.176.215/" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)"> http://132.66.176.215/</a>><br>><br>> [root@x-math20 ~]# ssh <a href="http://132.66.176.215/" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)">
132.66.176.215</a> <<a href="http://132.66.176.215/" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)"> http://132.66.176.215/</a> ><br>> <a href="mailto:root@132.66.176.215" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)">
root@132.66.176.215</a>'s password:<br>> ssh(21957) Permission denied, please try again. <br>> <a href="mailto:root@132.66.176.215" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)">
root@132.66.176.215 </a>'s password:<br>> Last login: Sun Dec 23 14:32:51 2007 from <a href="http://x-math20.tau.ac.il/" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)"> x-math20.tau.ac.il
</a> <<a href="http://x-math20.tau.ac.il/" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)">http://x-math20.tau.ac.il/ </a><br>> ><br>> [root@x-mathr11 ~]# lctl ping 132.66.176.211@tcp0
<br>> failed to ping 132.66.176.211@tcp: Input/output error<br>> [root@x-mathr11 ~]# lctl list_nids<br>> 132.66.176.215@tcp <br>> [root@x-mathr11 ~]# ssh <a href="http://132.66.176.212/" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)">
132.66.176.212</a> <<a href="http://132.66.176.212/" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)">http://132.66.176.212/</a>><br>> The authenticity of host ' <a href="http://132.66.176.212/" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)">
132.66.176.212</a> <<a href="http://132.66.176.212/" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)">http://132.66.176.212/</a>><br>> (<a href="http://132.66.176.212/" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)">
132.66.176.212 </a> <<a href="http://132.66.176.212/" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)">http://132.66.176.212/</a>> )' can't be established.<br>> RSA1 key fingerprint is 85:2a:c1:47:84:b7:b5:a6:cd:c4:57:86:af:ce:
<br>> 7e:74.<br> > Are you sure you want to continue connecting (yes/no)? yes<br>> ssh(11526) Warning: Permanently added '<a href="http://132.66.176.212/" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)">
132.66.176.212</a> <<a href="http://132.66.176.212/" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)"> http://132.66.176.212/</a><br>> > ' (RSA1) to the list of kno<br>> wn hosts.
<br>> <a href="mailto:root@132.66.176.212" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)">root@132.66.176.212</a>'s password:<br>> Last login: Sun Dec 23 15:24:41 2007 from <a href="http://x-math20.tau.ac.il/" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)">
x-math20.tau.ac.il</a> <<a href="http://x-math20.tau.ac.il/" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)">http://x-math20.tau.ac.il/</a><br>> ><br>> [root@localhost ~]# lctl ping
132.66.176.211@tcp0<br>> failed to ping 132.66.176.211@tcp: Input/output error<br>> [root@localhost ~]# lctl list_nids<br>> 132.66.176.212@tcp<br>> [root@localhost ~]#<br>><br>><br>
> thanks for helping!!<br>> Avi<br>><br>><br>> On Dec 23, 2007 5:32 PM, Aaron Knister <<a href="mailto:aaron@iges.org" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)">
aaron@iges.org</a>> wrote:<br>><br>><br>> On the oss can you ping the mds/mgs using this command-- <br>><br>> lctl ping 132.66.176.211@tcp0<br>><br>> If it doesn't ping, list the nids on each node by running
<br>><br>> lctl list_nids<br>><br>> and tell me what comes back. <br>><br>> -Aaron<br>><br>><br>> On Dec 23, 2007, at 9:22 AM, Avi Gershon wrote:
<br>><br>><br>> HI I could use some help.<br>> I installed lustre on 3 computers <br>> mdt/mgs :<br>><br>><br>> ************************************************************************************8
<br>> [root@x-math20 ~]#mkfs.lustre --reformat --fsname spfs --mdt -- <br>> mgs /dev/hdb<br>><br>> Permanent disk data:<br>> Target: spfs-MDTffff
<br>> Index: unassigned<br>> Lustre FS: spfs <br>> Mount type: ldiskfs<br>> Flags: 0x75<br>> (MDT MGS needs_index first_time update )
<br>> Persistent mount opts: errors=remount-ro,iopen_nopriv,user_xattr <br>> Parameters:<br>><br>> device size = 19092MB<br>> formatting backing filesystem ldiskfs on /dev/hdb
<br>> target name spfs-MDTffff <br>> 4k blocks 0<br>> options -J size=400 -i 4096 -I 512 -q -O dir_index<br>> -F
<br>> mkfs_cmd = mkfs.ext2 -j -b 4096 -L spfs-MDTffff -J size=400 -i <br>> 4096 -I 512 -q -O dir_index -F /dev/hdb<br>> Writing CONFIGS/mountdata<br>> [
root@x-math20 ~]# df<br>> Filesystem 1K-blocks Used Available Use% Mounted on <br>> /dev/hda1 19228276 4855244 13396284 27% /<br>> none 127432 0 127432 0% /dev/shm
<br>> /dev/hdb 17105436 455152 15672728 3% /mnt/test/ <br>> mdt<br>> [root@x-math20 ~]# cat /proc/fs/lustre/devices<br>> 0 UP mgs MGS MGS 5
<br>> 1 UP mgc MGC132.66.176.211@tcp<br>> 5f5ba729-6412-3843-2229-1310a0b48f71 5 <br>> 2 UP mdt MDS MDS_uuid 3<br>> 3 UP lov spfs-mdtlov spfs-mdtlov_UUID 4
<br>> 4 UP mds spfs-MDT0000 spfs-MDT0000_UUID 3<br>> [ root@x-math20 ~]#<br>> *************************************************************end<br>
> mdt******************************8<br>> so you can see that the MGS is up<br> > ond on the ost's I get an error!! plz help...<br>><br>> ost:
<br>><br>> **********************************************************************<br>> [ root@x-mathr11 ~]# mkfs.lustre --reformat --fsname spfs --ost --<br>> mgsnode=132.66. 176.211@tcp0
/dev/hdb1<br>><br>> Permanent disk data:<br>> Target: spfs-OSTffff <br>> Index: unassigned<br>> Lustre FS: spfs
<br>> Mount type: ldiskfs<br>> Flags: 0x72<br>> (OST needs_index first_time update ) <br>> Persistent mount opts: errors=remount-ro,extents,mballoc
<br>> Parameters: mgsnode=132.66.176.211@tcp<br>><br>> device size = 19594MB<br>> formatting backing filesystem ldiskfs on /dev/hdb1 <br>> target name spfs-OSTffff
<br>> 4k blocks 0<br>> options -J size=400 -i 16384 -I 256 -q -O<br>> dir_index -F <br>> mkfs_cmd = mkfs.ext2 -j -b 4096 -L spfs-OSTffff -J size=400 -i
<br>> 16384 -I 256 -q -O dir_index -F /dev/hdb1<br>> Writing CONFIGS/mountdata<br>> [ root@x-mathr11 ~]# /CONFIGS/mountdata<br>> -bash: /CONFIGS/mountdata: No such file or directory
<br>> [root@x-mathr11 ~]# mount -t lustre /dev/hdb1 /mnt/test/ost1<br>> mount.lustre: mount /dev/hdb1 at /mnt/test/ost1 failed: Input/<br>> output error<br>> Is the MGS running?
<br>> ***********************************************end<br>> ost******************************** <br>><br>> can any one point out the problem?<br>> thanks Avi.
<br>><br>><br>><br>> _______________________________________________<br> > Lustre-discuss mailing list<br>> <a href="mailto:Lustre-discuss@clusterfs.com" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)">
Lustre-discuss@clusterfs.com</a><br>> <a href="https://mail.clusterfs.com/mailman/listinfo/lustre-discuss" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)"> https://mail.clusterfs.com/mailman/listinfo/lustre-discuss
</a><br>><br>><br>><br>><br>><br>> Aaron Knister<br>> Associate Systems Administrator/Web Designer<br>> Center for Research on Environment and Water <br>>
<br>> (301) 595-7001<br>> <a href="mailto:aaron@iges.org" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)">aaron@iges.org</a><br>><br>><br>><br>><br>>
<br>><br>> Aaron Knister<br>> Associate Systems Administrator/Web Designer <br>> Center for Research on Environment and Water<br>><br>> (301) 595-7001<br>> <a href="mailto:aaron@iges.org" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)">
aaron@iges.org</a><br>><br>><br>><br><br>Aaron Knister<br>Associate Systems Administrator/Web Designer <br>Center for Research on Environment and Water<br><br>(301) 595-7001<br><a href="mailto:aaron@iges.org" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)">
aaron@iges.org</a><br><br><br><br></div></div></blockquote></div><br></span></div></blockquote></div><br> _______________________________________________<br>Lustre-discuss mailing list<br><a href="mailto:Lustre-discuss@clusterfs.com" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)">
Lustre-discuss@clusterfs.com</a><br><a href="https://mail.clusterfs.com/mailman/listinfo/lustre-discuss" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)">https://mail.clusterfs.com/mailman/listinfo/lustre-discuss
</a><br></blockquote></div><br></span></div><div><span style="border-collapse: separate; color: #000000; font-size: 12px; font-style: normal; font-variant: normal; font-weight: 400; letter-spacing: normal; line-height: normal; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px">
<div><span style="border-collapse: separate; color: #000000; font-size: 12px; font-style: normal; font-variant: normal; font-weight: 400; letter-spacing: normal; line-height: normal; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px">
<div><span style="border-collapse: separate; color: #000000; font-size: 12px; font-style: normal; font-variant: normal; font-weight: 400; letter-spacing: normal; line-height: normal; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px">
<div>Aaron Knister</div><div>Associate Systems Analyst</div><div><div><div>Center for Ocean-Land-Atmosphere Studies</div></div></div><div><br></div><div>(301) 595-7000</div><div><a href="mailto:aaron@iges.org" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)">
aaron@iges.org</a></div><div><br></div><br></span></div></span></div></span><br></div><br></div></blockquote></div><br>