[Lustre-discuss] Configuring Lustre routring between two tcp networks

[Andreas Dilger]

On Jun 03, 2009  17:45 -0400, Erik Froese wrote:
> I'm trying to configure a lustre router so I can mount a test lustre FS over
> our standard network here (NYU federated ethernet).
> We have a small rocks cluster with one MDS/MGS and 3 OSSs on a private
> switch. Its a pretty standard rocks configuration.
> The cluster network is 10.1.255.0/24.
> 
> == OSS / Router ==
> One of the OSS (oss-0-2) is configured as follows:
> eth0 - 10.1.255.247
> eth1 - 128.122.x.y
> 
> In its /etc/modprobe.conf I have the following
> options lnet forwarding="enabled"
> options lnet accept=all
> options lnet networks="tcp0(eth0),tcp1(eth1)"
> 
> [root at oss-0-2 ~]# lctl list_nids
> 10.1.255.247 at tcp
> 128.122.x.y at tcp1

I'm not a routing expert, but I think I can see what is wrong here.
To reiterate - your private network is tcp0 (10.x), and your external
network is tcp1 (128.x).

> == Routed Client ==
> Then I have another client on the 128.122.x.* network. Let's call it
> 128.122.x.z It just has eth0 configured as 128.122.x.z
> 
> and in its modprobe.conf
> options lnet networks=tcp0(eth0) routes="tcp1 128.122.x.y at tcp0"

Here you are configuring your external client to use tcp0 as 128.x,
which does NOT match what you have configured on your router.  You
need to have (I think):

options lnet networks=tcp1(eth0) routes="tcp0 128.122.x.y at tcp1"

> Now should I be able to mount the lustre fs as such?
> mount.lustre 10.1.255.252 at tcp0:/scratch /scratch
> mount.lustre: mount 10.1.255.252 at tcp:/scratch at /scratch failed: Cannot
> send after transport endpoint shutdown

This should be fine, once your routing is working.  You could try
"lctl ping 128.122.x.z at tcp1" to verify you can communicate with your
router/OSS and "lctl ping 10.1.255.252 at tcp0" for the MDS behind it.

Cheers, Andreas
--
Andreas Dilger
Sr. Staff Engineer, Lustre Group
Sun Microsystems of Canada, Inc.