[Lustre-discuss] Preferred network for OST data transfer

David Boyes d.boyes at ieee.org
Sun Jan 24 13:04:07 PST 2010


I've just been tuning a small Lustre installation, with two Ethernet 
physical networks and have noticed that the bulk of the network traffic 
is not going through the intended network.  The two networks are 
labelled tcp0 and tcp1, and tcp1 is in principle the best network for 
OST traffic, because of hardware and other traffic on tcp0.

The system is running 1.8.1.1 on the server and 1.8.0.1 on the clients, 
and Centos 5.3.  It was running 1.6 on the server previously, but the 
upgrade did not seem to have any effect.  Originally - and briefly - 
there was only tcp0.

So modprobe.conf now has tcp1 specified first, and lctl list_nids and 
pings all look good.  The mgsnode parameter in the (for now) single OST 
specifies the tcp1 network.  Then tunefs.lustre --writeconf has been 
run.  Client mounts works fine, using the tcp1 NID.  However, as shown 
by the transfer speed and also by iptraf, the actual file data is being 
transferred over tcp0.

This makes some sort of sense, as I suppose that client to MDS contact 
takes place over the network specified in the mount command, while 
actual data transfer can take place by another route.  On the other hand 
the manual does state that the first network given in the module options 
is preferred.

So I guess there are two questions - the general one of what is 
happening and the specific one of is there another parameter than can be 
set to tell the OSC which network to use?

Thanks in advance,
David Boyes




More information about the lustre-discuss mailing list