[Lustre-discuss] Preferred network for OST data transfer

[David Boyes]

As far as I know IP packets are not exempted from normal kernel routing, 
but I may be wrong and would be happy to learn something here.  I should 
have mentioned that the two physical networks are on two separate IPv4 
subnetworks, without routers, and that both of these networks seem to be 
functioning correctly.

But in any case I shut down the Lustre system at a quiet moment, removed 
all mention of the tcp0 network and rebooted several nodes including the 
MDS server.  The mount requests of course remained with tcp1.  Hey 
presto, the traffic was taking the route we wanted, viz. tcp1.

Then, mainly out of curiosity, I restored tcp0 to the MDS and to a 
client node with no effect - that is lctl list_nids looks OK (with both 
NID's) and traffic keeps going via tcp1.  This is not quite the mirror 
image of the original setup, as mount requests have always been by tcp1, 
but the modprobe.conf parameters on the MDS and on the selected client 
are exactly as at the start.

So we've got the traffic going the way we want.  However it does seem 
that something in the system is retaining a preference for a specificc 
NID for OST traffic, and that this is not reflected in the modprobe.conf 
or the documented filesystem parameters.

Best regards,
David Boyes