[Lustre-discuss] lustre performance degrade on SuSE 11

Tharindu Rukshan Bamunuarachchi tharindub at millenniumit.com
Wed Aug 12 05:20:54 PDT 2009


there is only one NIC in each machine. Does lustre need more than one nic in each machine ?



suse11-202:~ # ifconfig
eth0      Link encap:Ethernet  HWaddr 00:14:4F:EB:47:8C
          inet addr:172.25.88.202  Bcast:172.25.88.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:2672766 errors:0 dropped:0 overruns:0 frame:0
          TX packets:2479712 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:3217001573 (3067.9 Mb)  TX bytes:1817419819 (1733.2 Mb)

lustre-4200:~ # ifconfig
eth0      Link encap:Ethernet  HWaddr 00:14:4F:3F:B8:38
          inet addr:172.25.88.220  Bcast:172.25.91.255  Mask:255.255.252.0
          inet6 addr: fe80::214:4fff:fe3f:b838/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:73757596 errors:0 dropped:0 overruns:0 frame:0
          TX packets:20095469 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:110072295131 (104973.1 Mb)  TX bytes:1525481556 (1454.8 Mb)
          Base address:0x8c00 Memory:fbde0000-fbe00000

lustre-2100:~ # ifconfig
eth0      Link encap:Ethernet  HWaddr 00:E0:81:5D:F7:BE
          inet addr:172.25.88.120  Bcast:172.25.88.255  Mask:255.255.255.0
          inet6 addr: fe80::2e0:81ff:fe5d:f7be/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:20975604 errors:0 dropped:0 overruns:0 frame:0
          TX packets:15997617 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:2317735053 (2210.3 Mb)  TX bytes:105271680726 (100394.8 Mb)
          Interrupt:58 Base address:0xe000


cheers,
__
tharindu


-----Original Message-----
From: Mag Gam [mailto:magawake at gmail.com]
Sent: Wednesday, August 12, 2009 5:21 PM
To: Tharindu Rukshan Bamunuarachchi
Cc: lustre-discuss at lists.lustre.org
Subject: Re: [Lustre-discuss] lustre performance degrade on SuSE 11

How many NICs do you have on your client, MDS, OSS?

I am suspecting its a routing issue. Do you have more than 1 NIC on
all of these boxes?

options lnet 'networks="tcp0(eth0,eth1),elan0"'

I advise you to take a look at 5.1.1 in the Lustre manual,
manual.lustre.org/images/4/48/LustreManual_1.6_man_v19.pdf




On Wed, Aug 12, 2009 at 7:31 AM, Tharindu Rukshan
Bamunuarachchi<tharindub at millenniumit.com> wrote:
>
> I can SCP from/to client machine at 50 MB/s rate.
> Interface is up at 1000Full.
> No packet loss.
>
>
> Just like to emphasize ... Client is SuSE 11 (1.8.1) and OSS/MDS are SuSE 10 (1.8.0)
>
>
>
> suse11-202:~ # scp SLES-11-DVD-x86_64-GM-DVD1.iso 172.25.88.220:
> Password:
> SLES-11-DVD-x86_64-GM-DVD1.iso                100%  539MB  49.0MB/s   00:11
>
>
> Kernel Interface table
> Iface   MTU Met   RX-OK RX-ERR RX-DRP RX-OVR   TX-OK TX-ERR TX-DRP TX-OVR Flg
> eth0   1500   0 1741587      0      0      0 2002121      0      0      0 BMRU
>
>
> cheers,
> __
> tharindu
>
> -----Original Message-----
> From: Mag Gam [mailto:magawake at gmail.com]
> Sent: Wednesday, August 12, 2009 4:40 PM
> To: Tharindu Rukshan Bamunuarachchi
> Cc: lustre-discuss at lists.lustre.org
> Subject: Re: [Lustre-discuss] lustre performance degrade on SuSE 11
>
> Is your network setup properly? Can you scp/ftp file to your OSS as a
> test? Check if your network interfaces are properly connected
> (autoneg, 1000Full). make sure you aren't getting any packet loss
> (netstat -i). Also First then that and see how it goes.
>
>
> On Wed, Aug 12, 2009 at 3:37 AM, Tharindu Rukshan
> Bamunuarachchi<tharindub at millenniumit.com> wrote:
>> hi All,
>>
>>
>>
>> I have following lustre installation.
>>
>>
>>
>> Node A : OSS (SuSE 10) – 1.8.0
>>
>> Node B : MDS (SuSE 10) – 1.8.0
>>
>>
>>
>> Node C: Client (SuSE 11) – 1.8.1
>>
>>
>>
>> When try to run “dd” command on lustre mount, it is only writing at 2.1 MB/s
>>
>> But If mount same file system on Node B, I can achive 45 MB/s performance.
>>
>>
>>
>> All nodes are connected to same 3560G cisco switch.
>>
>>
>>
>> I saw following error in MDS server message file. Is this related to
>> performance of lustre file system ?
>>
>>
>>
>> LustreError: 3039:0:(socklnd_cb.c:2228:ksocknal_recv_hello()) Unknown
>> protocol version (2.x expected) from 172.25.88.202
>>
>>
>>
>>
>>
>> cheers,
>>
>> __
>>
>> tharindu
>>
>>
>>
>> *******************************************************************************************************************************************************************
>>
>> "The information contained in this email including in any attachment is
>> confidential and is meant to be read only by the person to whom it is
>> addressed. If you are not the intended recipient(s), you are prohibited from
>> printing, forwarding, saving or copying this email. If you have received
>> this e-mail in error, please immediately notify the sender and delete this
>> e-mail and its attachments from your computer."
>>
>> *******************************************************************************************************************************************************************
>>
>> _______________________________________________
>> Lustre-discuss mailing list
>> Lustre-discuss at lists.lustre.org
>> http://lists.lustre.org/mailman/listinfo/lustre-discuss
>>
>>
>
>
> *******************************************************************************************************************************************************************
>
> "The information contained in this email including in any attachment is confidential and is meant to be read only by the person to whom it is addressed. If you are not the intended recipient(s), you are prohibited from printing, forwarding, saving or copying this email. If you have received this e-mail in error, please immediately notify the sender and delete this e-mail and its attachments from your computer."
>
> *******************************************************************************************************************************************************************
>


*******************************************************************************************************************************************************************

"The information contained in this email including in any attachment is confidential and is meant to be read only by the person to whom it is addressed. If you are not the intended recipient(s), you are prohibited from printing, forwarding, saving or copying this email. If you have received this e-mail in error, please immediately notify the sender and delete this e-mail and its attachments from your computer."

*******************************************************************************************************************************************************************



More information about the lustre-discuss mailing list