[Lustre-discuss] tcp0 for maximum effect

[Peter Grandi]

> I have two boxes that have this:

> [root at lustrethree Desktop]# ifconfig
> eth0      Link encap:Ethernet  HWaddr 00:1B:21:2A:17:76
>           inet addr:192.168.0.19  Bcast:192.168.0.255  Mask:255.255.255.0
>           RX bytes:120168321 (114.6 MiB)  TX bytes:5300070662 (4.9 GiB)
> [ ... ]
> eth1      Link encap:Ethernet  HWaddr 00:1B:21:2A:1C:DC
>           inet addr:192.168.0.20  Bcast:192.168.0.255  Mask:255.255.255.0
>           RX bytes:55673426 (53.0 MiB)  TX bytes:846 (846.0 b)
> [ ... another 4 like that, 192.168.0.21-24 ... ]

That's a very bizarre network configuration, you have 5
interfaces on the same subnet (presumably all plugged into the
same switch) with no load balancing, as all the outgoing traffic
goes via 'eth0'.

You have some better alternatives:

* Use bonding (if the switch supports to ties together the 5
  interfaces as one virtual interface with a single IP address.

* Use something like 'nexthop' routing (and a couple other
  tricks) to split the load across the several interfaces. This
  is easier for the outgoing traffic than the incoming traffic,
  but it seems you have a lot more outgoing traffic.

* Use 1 10Gb/s card per server and a 1Gb/s switch with 2 10Gb/s
  ports. 10Gb/s cards and switches have fallen in price a lot
  recently (check Myri.com), and a server that can do several
  hundred MB/s really deserves a nice 10Gb/s interface.

IIRC 'lnet' has something like bonding built in, but I am not
sure that it handles multiple addresses in the same subnet well.

> Would it be better to have these two boxes as OSS's or as MDT
> or MGS machines?  Currently they are configured 1 as a MGS and
> the other as the MDT.

If these are the two servers with gigantic disk arrays, I'd have
on each both MDS and OSS. Possibly with the OSTs replicated
across both machines in an active/passive configuration.

> The question is does LNET use the available tcp0 connections
> different from the OSS perspective as opposed to the MDT or
> MGS perspective?

Not sure that the question means.