[Lustre-discuss] Modifying Lustre network (good practices)

[Balagopal Pillai]

If you use etherchannel/isl on the switch side, wouldn't it be better 
performing if
the bonding is mode 0 on the server side rather than use mode 4 for 
lacp? As far as i know, mode 0 is the only mode that is capable of 
providing bandwidth more than a single port  could provide, if there is 
switch support, like cisco etherchannel. I use mode 6 on several hpc 
clusters and the performance is not bad, as long as the compute nodes 
pick up
different mac addresses of the available interfaces on the storage 
boxes. That is worth a try. It needs no switch support, but works only 
if all clients are on the same subnet.

          On the other hand, i have also seen some weird behavior with 
procurve switches before, when two switches have 4 interfaces on a trunk 
with lacp and then i put the storage server also on mode 4 and lacp 
connected to one of the switches. No matter what i tried,
there was significant traffic on just one of the interfaces. So i went 
back to mode 6 on the server and lacp between switches. Please see the 
interface statistics of each bonded interface on the server and see if 
there is an imbalance, like one of the interfaces having bulk of the 
traffic and others get nothing. When looking at performance, there might 
also be the issue of iowaits and see if dirty pages are piling up and 
the server is hitting the dirty_ratio limit on the storage server. That 
could cause temporary lockups and performance issues under heavy load. 
More aggressive flushing and increasing the diry_ratio limit might help 
if that is the case. Hope this helps!



On 10-05-20 12:39 PM, Olivier Hargoaa wrote:
> Nate Pearlstein a écrit :
>> Which bonding method are you using?  Has the performance always been
>> this way?  Depending on which bonding type you are using and the network
>> hardware involved you might see the behavior you are describing.
>>
>
> Hi,
>
> Here is our bonding configuration :
>
> On linux side :
>
> mode=4	          - to use 802.3ad
> miimon=100	 - to set the link check interval (ms)
> xmit_hash_policy=layer2+3	- to set XOR hashing method
> lacp_rate=fast	 - to set LCAPDU tx rate to request (slow=20s, fast=1s)
>
> Onethernet switch side, load balancing is configured as:
> # port-channel load-balance src-dst-mac
>
> thanks
>
>>
>> On Thu, 2010-05-20 at 16:27 +0200, Olivier Hargoaa wrote:
>>> Dear All,
>>>
>>> We have a cluster with lustre critical data. On this cluster there are
>>> three networks on each Lustre server and client : one ethernet network
>>> for administration (eth0), and two other ethernet networks configured in
>>> bonding (bond0: eth1&  eth2). On Lustre we get poor read performances
>>> and good write performances so we decide to modify Lustre network in
>>> order to see if problems comes from network layer.
>>>
>>> Currently Lustre network is bond0. We want to set it as eth0, then eth1,
>>> then eth2 and finally back to bond0 in order to compare performances.
>>>
>>> Therefore, we'll perform the following steps: we will umount the
>>> filesystem, reformat the mgs, change lnet options in modprobe file,
>>> start new mgs server, and finally modify our ost and mdt with
>>> tunefs.lustre with failover and mgs new nids using "--erase-params" and
>>> "--writeconf" options.
>>>
>>> We tested it successfully on a test filesystem but we read in the manual
>>> that this can be really dangerous. Do you agree with this procedure? Do
>>> you have some advice or practice on this kind of requests? What's the
>>> danger?
>>>
>>> Regards.
>>>
>>
>
> _______________________________________________
> Lustre-discuss mailing list
> Lustre-discuss at lists.lustre.org
> http://lists.lustre.org/mailman/listinfo/lustre-discuss