the output of ltcl list_nids on the oss is<br><br>[root@oss ~]# lctl list_nids<br>192.168.0.3@tcp<br><br>and from the mds<br><br>[root@mds ~]# lctl list_nids<br>192.168.0.2@tcp<br><br>Thanks,<br>Dusty<br><br><div class="gmail_quote">
On Fri, Jan 15, 2010 at 5:39 PM, Wojciech Turek <span dir="ltr"><<a href="mailto:wjt27@cam.ac.uk">wjt27@cam.ac.uk</a>></span> wrote:<br><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
Hi,<br>
<br>
Could you please post output of the 'lctl list_nids' command on OSS<br>
system and on MDS system. This will show us which network was<br>
configured to work with lustre.<br>
<br>
Regarding entries in the modprobe.conf, they tell lnet module which<br>
NIC or multiple NICs will be configured to work with lustre. If your<br>
modprobe.conf doesn't have lnet options line, by default Lustre will<br>
configure the first NIC which is usually eth0.<br>
Below is a modprobe.conf entry from my lustre setup.<br>
My OSS(s) and MDS(s) have 2 NICs eth0 and eth1 and an Infiniband NIC<br>
ib0. The IB is set to work as IPoIB so lustre treats it as an ordinary<br>
Ethernet NIC<br>
options lnet networks=tcp0(ib0),tcp1(eth1),tcp2(eth1:0)<br>
So the line above means that:<br>
first lustre network tcp0 is configured on interface ib0<br>
second lustre network tcp1 is configured on interface eth1<br>
third lustre network tcp2 is confiured on alias interface eth1:0<br>
<br>
eth0 is not mentioned on this line because I have chosen not to<br>
configure it to work with lustre.<br>
<br>
<br>
Once lnet module is loaded you can check which network or networks are<br>
configured to work with Lustre using 'lctl list_nids' command<br>
<br>
Cheers<br>
<br>
Wojciech<br>
2010/1/15 Dusty Marks <<a href="mailto:dustynmarks@gmail.com">dustynmarks@gmail.com</a>>:<br>
<div><div></div><div class="h5">> I did some googling and i found the command lctl ping. So i went on the oss<br>
> and typed in "lctl ping 192.168.0.2@tcp". This errored out with an I/O<br>
> error.<br>
><br>
> It is quite obvious that i've simply misconfigured the network. Could<br>
> someone explain how to properly configure it?<br>
><br>
> I don't understand what the entry in modprobe actually means, so i cannot<br>
> say what should be entered.<br>
><br>
> Each one of my machines has one NIC (eth0). What do i enter in<br>
> modprobe.conf? To make this work correctly? if i update the entry in<br>
> modprobe.conf, do i have to redo anything? or does lustre pickup on the<br>
> changes without restarting anything?<br>
><br>
> Thanks all for the help so far.<br>
><br>
> - Dusty<br>
><br>
> On Fri, Jan 15, 2010 at 10:36 AM, Dusty Marks <<a href="mailto:dustynmarks@gmail.com">dustynmarks@gmail.com</a>> wrote:<br>
>><br>
>> I searched through the manual, and the only section i could find dealing<br>
>> with networking configuration is section 4.1.0.2 titled "Module Setup" in<br>
>> the Lustre 1.8 operations manual.<br>
>><br>
>> It tells me to run the command modprobe -v lustre "networks=tcp0(eth0)",<br>
>> and i did such on the MDS, however it errored out with:<br>
>><br>
>> [root@mds ~]# modprobe -v lustre "networks=tcp0(eth0)"<br>
>> insmod<br>
>> /lib/modules/2.6.18-128.7.1.el5_lustre.1.8.1.1.20091003130007/kernel/fs/lustre/lustre.ko<br>
>> networks=tcp0(eth0)<br>
>> FATAL: Error inserting lustre<br>
>> (/lib/modules/2.6.18-128.7.1.el5_lustre.1.8.1.1.20091003130007/kernel/fs/lustre/lustre.ko):<br>
>> Unknown symbol in module, or unknown parameter (see dmesg)<br>
>><br>
>> dmesg says nothing, but message says this:<br>
>> Jan 15 10:27:48 mds kernel: lustre: Unknown parameter `networks'<br>
>><br>
>> I even tried adding "options lnet networks=tcp0(eth0)" however that didn't<br>
>> work either<br>
>><br>
>> I'm terribly sorry for my incompetence, but i'm having a difficult time<br>
>> understanding lustre's abstractions.<br>
>><br>
>> Each one of my nodes have a single ethernet card (eth0)<br>
>><br>
>><br>
>> On Thu, Jan 14, 2010 at 11:32 PM, Andreas Dilger <<a href="mailto:adilger@sun.com">adilger@sun.com</a>> wrote:<br>
>>><br>
>>> On 2010-01-15, at 00:21, Arden Wiebe wrote:<br>
>>>><br>
>>>> Your mount command is wrong - try this format.<br>
>>>><br>
>>>> mount -t lustre 192.168.0.7@tcp0:/ioio /mnt/ioio<br>
>>>><br>
>>>> So by substitution for supplied your mount line should<br>
>>>> read:<br>
>>>><br>
>>>> mount -t datafs 192.168.0.2@tcp0:/datafs /mnt/datafs<br>
>>><br>
>>> No, that isn't correct. You are showing the mount command for a<br>
>>> client. It is the OST that is failing to mount, likely because<br>
>>> the network is not configured correctly, and the OST needs to<br>
>>> contact the MGS node always on the first mount in order to join<br>
>>> the filesystem.<br>
>>><br>
>>>> Enjoy the required reading and testing. I found by<br>
>>>> naming things uniquely helped me clarify what was actually<br>
>>>> required. Try calling your filesystem "Dusty" or<br>
>>>> "Mark" and that should make things clearer for you.<br>
>>>><br>
>>>> --- On Thu, 1/14/10, Andreas Dilger <<a href="mailto:adilger@sun.com">adilger@sun.com</a>> wrote:<br>
>>>>><br>
>>>>> On 2010-01-14, at 23:51, Dusty Marks wrote:<br>
>>>>>><br>
>>>>>> You are correct, there is information in messages. Following are the<br>
>>>>>> entries related the lustre. The line that says 192.168.0.2@tcp is<br>
>>>>>> unreachable makes sense, but what exactly is the problem? I entered<br>
>>>>>> the line "options lnet networks=tcp" in modprobe.conf on the oss and<br>
>>>>>> mds. The only difference was, i entered that line AFTER i setup<br>
>>>>>> lustre on the OSS. Could that be the problem? I don't see why that<br>
>>>>>> would be the problem, as the oss is trying to reach the MDS/MGS,<br>
>>>>>> which is 192.168.0.2.<br>
>>>>>><br>
>>>>>> ---------------------------------------<br>
>>>>>> Jan 14 22:41:07 oss kernel: Lustre: 2846:0:(linux-tcpip.c:<br>
>>>>>> 688:libcfs_sock_connect()) Error -113 connecting <a href="http://0.0.0.0/1023" target="_blank">0.0.0.0/1023</a> -><br>
>>>>>> <a href="http://192.168.0.2/988" target="_blank">192.168.0.2/988</a><br>
>>>>>> Jan 14 22:41:07 oss kernel: Lustre: 2846:0:(acceptor.c:<br>
>>>>>> 95:lnet_connect_console_error()) Connection to 192.168.0.2@tcp at<br>
>>>>>> host 192.168.0.2 was unreachable: the network or that node may be<br>
>>>>>> down, or Lustre may be misconfigured.<br>
>>>>><br>
>>>>><br>
>>>>> Please read the chapter in the manual about network configuration. I<br>
>>>>> suspect the .0.2 network is not your eth0 network interface, and your<br>
>>>>> modprobe.conf needs to be fixed.<br>
>>><br>
>>><br>
>>> Cheers, Andreas<br>
>>> --<br>
>>> Andreas Dilger<br>
>>> Sr. Staff Engineer, Lustre Group<br>
>>> Sun Microsystems of Canada, Inc.<br>
>>><br>
>><br>
>><br>
>><br>
>> --<br>
>> The graduate with a Science degree asks, "Why does it work?" The graduate<br>
>> with an Engineering degree asks, "How does it work?" The graduate with an<br>
>> Accounting degree asks, "How much will it cost?" The graduate with an Arts<br>
>> degree asks, "Do you want fries with that?"<br>
><br>
><br>
><br>
> --<br>
> The graduate with a Science degree asks, "Why does it work?" The graduate<br>
> with an Engineering degree asks, "How does it work?" The graduate with an<br>
> Accounting degree asks, "How much will it cost?" The graduate with an Arts<br>
> degree asks, "Do you want fries with that?"<br>
><br>
</div></div>> _______________________________________________<br>
> Lustre-discuss mailing list<br>
> <a href="mailto:Lustre-discuss@lists.lustre.org">Lustre-discuss@lists.lustre.org</a><br>
> <a href="http://lists.lustre.org/mailman/listinfo/lustre-discuss" target="_blank">http://lists.lustre.org/mailman/listinfo/lustre-discuss</a><br>
><br>
><br>
<br>
<br>
<br>
--<br>
--<br>
Wojciech Turek<br>
<br>
Assistant System Manager<br>
<br>
High Performance Computing Service<br>
University of Cambridge<br>
Email: <a href="mailto:wjt27@cam.ac.uk">wjt27@cam.ac.uk</a><br>
Tel: (+)44 1223 763517<br>
</blockquote></div><br><br clear="all"><br>-- <br>The graduate with a Science degree asks, "Why does it work?" The graduate with an Engineering degree asks, "How does it work?" The graduate with an Accounting degree asks, "How much will it cost?" The graduate with an Arts degree asks, "Do you want fries with that?"<br>