Got it working. The firewall was blocking lustre traffic. :( After disabling it, it works.<br><br>Thanks all for the help!<br><br><div class="gmail_quote">On Sat, Jan 16, 2010 at 9:57 AM, Dusty Marks <span dir="ltr"><<a href="mailto:dustynmarks@gmail.com">dustynmarks@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"><span><span></span></span>I've posted my /var/log/messages here before, but here it is again:<br>
<br><br>--------------------------------------- /var/log/messages -----------------------------------------------------------<br>
<div><div class="im">Jan 14 22:41:05 oss kernel: Lustre: OBD class driver, <a href="http://www.lustre.org/" target="_blank">http://www.lustre.org/</a><br>
Jan 14 22:41:05 oss kernel: Lustre: Lustre Version: 1.8.1.1<br>Jan 14 22:41:05 oss kernel: Lustre: Build Version: 1.8.1.1-20091009075116-PRISTINE-2.6.18-128.7.1.el5_lustre.1.8.1.1.20091003130007<br>Jan 14 22:41:06 oss kernel: Lustre: Added LNI 192.168.0.3@tcp [8/256/0/0]<br>
Jan 14 22:41:06 oss kernel: Lustre: Accept secure, port 988<br>Jan 14 22:41:06 oss kernel: Lustre: Lustre Client File System; <a href="http://www.lustre.org/" target="_blank">http://www.lustre.org/</a><br>Jan 14 22:41:07 oss kernel: kjournald starting. Commit interval 5 seconds<br>
Jan 14 22:41:07 oss kernel: LDISKFS FS on dm-2, internal journal<br>Jan 14 22:41:07 oss kernel: LDISKFS-fs: mounted filesystem with ordered data mode.<br>Jan 14 22:41:07 oss kernel: kjournald starting. Commit interval 5 seconds<br>
Jan 14 22:41:07 oss kernel: LDISKFS FS on dm-2, internal journal<br>Jan 14 22:41:07 oss kernel: LDISKFS-fs: mounted filesystem with ordered data mode.<br></div><div class="im">Jan 14 22:41:07 oss kernel: LDISKFS-fs: file extents enabled<br>
</div>Jan 14 22:41:07 oss kernel: LDISKFS-fs: mballoc enabled<div class="im"><br>
Jan 14 22:41:07 oss kernel: Lustre: 2846:0:(linux-tcpip.c:688:libcfs_sock_connect()) Error -113 connecting <a href="http://0.0.0.0/1023" target="_blank">0.0.0.0/1023</a> -> <a href="http://192.168.0.2/988" target="_blank">192.168.0.2/988</a><br>
Jan 14 22:41:07 oss kernel: Lustre: 2846:0:(acceptor.c:95:lnet_connect_console_error())
Connection to 192.168.0.2@tcp at host 192.168.0.2 was unreachable: the
network or that node may be down, or Lustre may be misconfigured.<br></div><div class="im">
Jan 14 22:41:07 oss kernel: Lustre: 2846:0:(socklnd_cb.c:421:ksocknal_txlist_done()) Deleting packet type 1 len 368 192.168.0.3@tcp->192.168.0.2@tcp<br>Jan 14 22:41:12 oss kernel: Lustre: 2853:0:(client.c:1383:ptlrpc_expire_one_request())
@@@ Request x1324907721916417 sent from MGC192.168.0.2@tcp to NID
192.168.0.2@tcp 5s ago has timed out (limit 5s).<br>
Jan 14 22:41:12 oss kernel: req@f5d7fe00 x1324907721916417/t0 o250->MGS@MGC192.168.0.2@tcp_0:26/25 lens 368/584 e 0 to 1 dl 1263530472 ref 1 fl Rpc:N/0/0 rc 0/0<br>Jan 14 22:41:12 oss kernel: LustreError: 2819:0:(obd_mount.c:1085:server_start_targets()) Required registration failed for datafs-OSTffff: -5<br>
Jan 14 22:41:12 oss kernel: LustreError: 15f-b: Communication error with the MGS. Is the MGS running?<br>Jan 14 22:41:12 oss kernel: LustreError: 2819:0:(obd_mount.c:1629:server_fill_super()) Unable to start targets: -5<br>
Jan 14 22:41:12 oss kernel: LustreError: 2819:0:(obd_mount.c:1412:server_put_super()) no obd datafs-OSTffff<br>Jan 14 22:41:12 oss kernel: LustreError: 2819:0:(obd_mount.c:136:server_deregister_mount()) datafs-OSTffff not registered<br>
Jan 14 22:41:12 oss kernel: LDISKFS-fs: mballoc: 0 blocks 0 reqs (0 success)<br>Jan 14 22:41:12 oss kernel: LDISKFS-fs: mballoc: 0 extents scanned, 0 goal hits, 0 2^N hits, 0 breaks, 0 lost<br>Jan 14 22:41:12 oss kernel: LDISKFS-fs: mballoc: 0 generated and it took 0<br>
Jan 14 22:41:12 oss kernel: LDISKFS-fs: mballoc: 0 preallocated, 0 discarded<br>Jan 14 22:41:12 oss kernel: Lustre: server umount datafs-OSTffff complete<br>Jan 14 22:41:12 oss kernel: LustreError: 2819:0:(obd_mount.c:1997:lustre_fill_super()) Unable to mount (-5)</div>
</div><div><div></div><div class="h5">
<br><br><div class="gmail_quote">On Sat, Jan 16, 2010 at 5:14 AM, Christopher J. Walker <span dir="ltr"><<a href="mailto:C.J.Walker@qmul.ac.uk" target="_blank">C.J.Walker@qmul.ac.uk</a>></span> wrote:<br><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<div>Wojciech Turek wrote:<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
Hi,<br>
<br>
Could you please post output of the 'lctl list_nids' command on OSS<br>
system and on MDS system. This will show us which network was<br>
configured to work with lustre.<br>
<br>
Regarding entries in the modprobe.conf, they tell lnet module which<br>
NIC or multiple NICs will be configured to work with lustre. <br>
</blockquote>
<br></div>
There's a gotcha here which I've been meaning to write up. We have a 10Gig card as eth2 assigned a different IP address on the same subnet as eth0, a 1Gig card. Whilst lustre correctly bound to the ip address of eth2, the kernel decided (correctly) it could route packets via eth0. This worked, but gave poor performance (partly due to a bottleneck on that art of the network). The solution was to ensure that packets from eth2's IP address were routed out of eth2.<br>
<br>
Chris<div><div></div><div><br>
<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
If your<br>
modprobe.conf doesn't have lnet options line, by default Lustre will<br>
configure the first NIC which is usually eth0.<br>
Below is a modprobe.conf entry from my lustre setup.<br>
My OSS(s) and MDS(s) have 2 NICs eth0 and eth1 and an Infiniband NIC<br>
ib0. The IB is set to work as IPoIB so lustre treats it as an ordinary<br>
Ethernet NIC<br>
options lnet networks=tcp0(ib0),tcp1(eth1),tcp2(eth1:0)<br>
So the line above means that:<br>
first lustre network tcp0 is configured on interface ib0<br>
second lustre network tcp1 is configured on interface eth1<br>
third lustre network tcp2 is confiured on alias interface eth1:0<br>
<br>
eth0 is not mentioned on this line because I have chosen not to<br>
configure it to work with lustre.<br>
<br>
<br>
Once lnet module is loaded you can check which network or networks are<br>
configured to work with Lustre using 'lctl list_nids' command<br>
<br>
Cheers<br>
<br>
Wojciech<br>
2010/1/15 Dusty Marks <<a href="mailto:dustynmarks@gmail.com" target="_blank">dustynmarks@gmail.com</a>>:<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
I did some googling and i found the command lctl ping. So i went on the oss<br>
and typed in "lctl ping 192.168.0.2@tcp". This errored out with an I/O<br>
error.<br>
<br>
It is quite obvious that i've simply misconfigured the network. Could<br>
someone explain how to properly configure it?<br>
<br>
I don't understand what the entry in modprobe actually means, so i cannot<br>
say what should be entered.<br>
<br>
Each one of my machines has one NIC (eth0). What do i enter in<br>
modprobe.conf? To make this work correctly? if i update the entry in<br>
modprobe.conf, do i have to redo anything? or does lustre pickup on the<br>
changes without restarting anything?<br>
<br>
Thanks all for the help so far.<br>
<br>
- Dusty<br>
<br>
On Fri, Jan 15, 2010 at 10:36 AM, Dusty Marks <<a href="mailto:dustynmarks@gmail.com" target="_blank">dustynmarks@gmail.com</a>> wrote:<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
I searched through the manual, and the only section i could find dealing<br>
with networking configuration is section 4.1.0.2 titled "Module Setup" in<br>
the Lustre 1.8 operations manual.<br>
<br>
It tells me to run the command modprobe -v lustre "networks=tcp0(eth0)",<br>
and i did such on the MDS, however it errored out with:<br>
<br>
[root@mds ~]# modprobe -v lustre "networks=tcp0(eth0)"<br>
insmod<br>
/lib/modules/2.6.18-128.7.1.el5_lustre.1.8.1.1.20091003130007/kernel/fs/lustre/lustre.ko<br>
networks=tcp0(eth0)<br>
FATAL: Error inserting lustre<br>
(/lib/modules/2.6.18-128.7.1.el5_lustre.1.8.1.1.20091003130007/kernel/fs/lustre/lustre.ko):<br>
Unknown symbol in module, or unknown parameter (see dmesg)<br>
<br>
dmesg says nothing, but message says this:<br>
Jan 15 10:27:48 mds kernel: lustre: Unknown parameter `networks'<br>
<br>
I even tried adding "options lnet networks=tcp0(eth0)" however that didn't<br>
work either<br>
<br>
I'm terribly sorry for my incompetence, but i'm having a difficult time<br>
understanding lustre's abstractions.<br>
<br>
Each one of my nodes have a single ethernet card (eth0)<br>
<br>
<br>
On Thu, Jan 14, 2010 at 11:32 PM, Andreas Dilger <<a href="mailto:adilger@sun.com" target="_blank">adilger@sun.com</a>> wrote:<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
On 2010-01-15, at 00:21, Arden Wiebe wrote:<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
Your mount command is wrong - try this format.<br>
<br>
mount -t lustre 192.168.0.7@tcp0:/ioio /mnt/ioio<br>
<br>
So by substitution for supplied your mount line should<br>
read:<br>
<br>
mount -t datafs 192.168.0.2@tcp0:/datafs /mnt/datafs<br>
</blockquote>
No, that isn't correct. You are showing the mount command for a<br>
client. It is the OST that is failing to mount, likely because<br>
the network is not configured correctly, and the OST needs to<br>
contact the MGS node always on the first mount in order to join<br>
the filesystem.<br>
<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
Enjoy the required reading and testing. I found by<br>
naming things uniquely helped me clarify what was actually<br>
required. Try calling your filesystem "Dusty" or<br>
"Mark" and that should make things clearer for you.<br>
<br>
--- On Thu, 1/14/10, Andreas Dilger <<a href="mailto:adilger@sun.com" target="_blank">adilger@sun.com</a>> wrote:<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
On 2010-01-14, at 23:51, Dusty Marks wrote:<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
You are correct, there is information in messages. Following are the<br>
entries related the lustre. The line that says 192.168.0.2@tcp is<br>
unreachable makes sense, but what exactly is the problem? I entered<br>
the line "options lnet networks=tcp" in modprobe.conf on the oss and<br>
mds. The only difference was, i entered that line AFTER i setup<br>
lustre on the OSS. Could that be the problem? I don't see why that<br>
would be the problem, as the oss is trying to reach the MDS/MGS,<br>
which is 192.168.0.2.<br>
<br>
---------------------------------------<br>
Jan 14 22:41:07 oss kernel: Lustre: 2846:0:(linux-tcpip.c:<br>
688:libcfs_sock_connect()) Error -113 connecting <a href="http://0.0.0.0/1023" target="_blank">0.0.0.0/1023</a> -><br>
<a href="http://192.168.0.2/988" target="_blank">192.168.0.2/988</a><br>
Jan 14 22:41:07 oss kernel: Lustre: 2846:0:(acceptor.c:<br>
95:lnet_connect_console_error()) Connection to 192.168.0.2@tcp at<br>
host 192.168.0.2 was unreachable: the network or that node may be<br>
down, or Lustre may be misconfigured.<br>
</blockquote>
<br>
Please read the chapter in the manual about network configuration. I<br>
suspect the .0.2 network is not your eth0 network interface, and your<br>
modprobe.conf needs to be fixed.<br>
</blockquote></blockquote>
<br>
Cheers, Andreas<br>
--<br>
Andreas Dilger<br>
Sr. Staff Engineer, Lustre Group<br>
Sun Microsystems of Canada, Inc.<br>
<br>
</blockquote>
<br>
<br>
--<br>
The graduate with a Science degree asks, "Why does it work?" The graduate<br>
with an Engineering degree asks, "How does it work?" The graduate with an<br>
Accounting degree asks, "How much will it cost?" The graduate with an Arts<br>
degree asks, "Do you want fries with that?"<br>
</blockquote>
<br>
<br>
--<br>
The graduate with a Science degree asks, "Why does it work?" The graduate<br>
with an Engineering degree asks, "How does it work?" The graduate with an<br>
Accounting degree asks, "How much will it cost?" The graduate with an Arts<br>
degree asks, "Do you want fries with that?"<br>
<br>
_______________________________________________<br>
Lustre-discuss mailing list<br>
<a href="mailto:Lustre-discuss@lists.lustre.org" target="_blank">Lustre-discuss@lists.lustre.org</a><br>
<a href="http://lists.lustre.org/mailman/listinfo/lustre-discuss" target="_blank">http://lists.lustre.org/mailman/listinfo/lustre-discuss</a><br>
<br>
<br>
</blockquote>
<br>
<br>
<br>
</blockquote>
<br>
</div></div></blockquote></div><br><br clear="all"><br></div></div>-- <br><div><div></div><div class="h5">The graduate with a Science degree asks, "Why does it work?" The graduate with an Engineering degree asks, "How does it work?" The graduate with an Accounting degree asks, "How much will it cost?" The graduate with an Arts degree asks, "Do you want fries with that?"<br>
</div></div></blockquote></div><br><br clear="all"><br>-- <br>The graduate with a Science degree asks, "Why does it work?" The graduate with an Engineering degree asks, "How does it work?" The graduate with an Accounting degree asks, "How much will it cost?" The graduate with an Arts degree asks, "Do you want fries with that?"<br>