[Lustre-discuss] Clients getting incorrect network information for one of two MDT servers (active/passive)

[McKee, Shawn]

Hi Everyone,

We are having a problem with Lustre v1.8.3/x86_64 (ext4 flavor if it matters).  We are very new to using Lustre so our problem may be trivial to those with experience. 

We have setup a separate MGS server and we have an HA setup for our MDT.  There are two servers with a backend iSCSI storage area for the MDT on lmd01.aglt2.org/lmd02.aglt2.org (active/passive using RedHat clustering).  All nodes are dual-homed (private and public networks).  Failover works without a problem modulo the issue we are asking about.

The primary problem is that one of the MDT nodes (LMD01) seems to be unreachable from the clients.  We have configured lnet to use the private network to mount/access Lustre.  The lnet line in /etc/modprobe.conf looks like this on an MDT server:

options lnet networks=tcp0(bond0.4010) routes="tcp2 10.10.1.[50-52]@tcp0"

(We also have some routing for an external public network to allow clients there to mount...not sure it is relevant to our problem.  I can provide details if it is useful)

The 'bond0.4010' is the private network.  The clients on this  private network look similar:

options lnet networks=tcp0(eth0)

The relevant IPs:   lmd01 has 10.10.1.48 (private) and 192.41.230.48 (public)
                    lmd02 has 10.10.1.49 (private) and 192.41.230.49 (public)

The problem we have is shown in the 'lctl --net tcp0 peer_list' output:

[root at bl-11-1 ~]# lctl --net tcp0  peer_list
12345-10.10.1.26 at tcp [1]bl-11-1.local->umfs06.local:988 #3
12345-10.10.1.36 at tcp [1]bl-11-1.local->umfs16.local:988 #3
12345-10.10.1.140 at tcp [1]bl-11-1.local->mgs.local:988 #3
12345-10.10.1.49 at tcp [2]bl-11-1.local->lmd02.local:988 #6
12345-192.41.230.48 at tcp [1116]0.0.0.0->lmd01.aglt2.org:988 #0
12345-10.10.1.25 at tcp [1]bl-11-1.local->umfs05.local:988 #3

Notice the "public" address 192.41.230.48 showing up on the 'tcp' ('tcp0') network?   This seems to be the problem.  If LMD01 takes over actively serving the MDT we see things like the following in the logs:

2010-05-10T12:21:01-04:00 lmd01.aglt2.org kernel: [272846.750287] LustreError: 120-3: Refusing connection from 192.41.237.235 for 192.41.230.48 at tcp: No matching NI
2010-05-10T12:23:46-04:00 lmd01.aglt2.org kernel: [273011.595403] LustreError: 120-3: Refusing connection from 192.41.237.235 for 192.41.230.48 at tcp: No matching NI
2010-05-10T12:29:01-04:00 lmd01.aglt2.org kernel: [273326.290186] LustreError: 120-3: Refusing connection from 192.41.230.203 for 192.41.230.48 at tcp: No matching NI
2010-05-10T12:48:11-04:00 lmd01.aglt2.org kernel: [274475.351001] LustreError: 120-3: Refusing connection from 192.41.230.168 for 192.41.230.48 at tcp: No matching NI

This makes sense because LMD01 is NOT supposed to be using its public IP for Lustre.   The strange thing is the LMD02 (setup almost exactly the same way as LMD01) doesn't have this problem and always works fine on the private network.  Deleting the "bad" peer address on the client doesn't help since it just re-appears as soon as the client tries to access Lustre.  Any ideas about what could be providing this "bad" IP and how we can remove it?   

FYI, I even tried "adding" tcp1 (for the public NIC) to the lnet options on LMD01/LMD02 but clients still fail since the request is coming in as '192.41.230.48 at tcp'  and not as '192.41.230.48 at tcp1'.

Thanks for any help or pointers to what might be wrong.

Shawn McKee/University of Michigan Physics