[lustre-discuss] Multi-cluster (multi-rail) setup

[Thrash Er]

Hi Chris !!

On Fri, Jun 12, 2015 at 6:47 PM, Chris Horn <hornc at cray.com> wrote:
> Hello and welcome to Lustre :)

Thanks!

>> 3.- configure /etc/modprobe.d/lustre.conf on each node of each cluster
>> like this:
>>
>>     Nodes con Cluster A:      options lnet networks="o2ib0(ib0)"
>>
>>     Nodes con Cluster B:      options lnet networks="o2ib1(ib1)"
>>
>>     Nodes con Cluster C:      options lnet networks="o2ib2(ib2)"
>>
>>     Nodes con Cluster D:      options lnet networks="o2ib3(ib3)”
>
> The “(ibX)” portion of that string should correspond to the local IB interface that the clients in those clusters are actually using. i.e which port on the clients is active, not the port that is used by servers on that LNet. My guess is that the clients have a single IB HCA with a cable plugged into port 0, so that what you probably want is:
>
>     Nodes con Cluster A:      options lnet networks="o2ib0(ib0)"
>
>     Nodes con Cluster B:      options lnet networks="o2ib1(ib0)"
>
>     Nodes con Cluster C:      options lnet networks="o2ib2(ib0)"
>
>     Nodes con Cluster D:      options lnet networks="o2ib3(ib0)”
>
> Again, that’s just a guess on how these things are typically configured. You’ll want to check if that is actually case for your clusters.


OK, gotcha. Yup, the compute nodes only have 1 IB port.

Thanks !!


>> On Jun 12, 2015, at 2:37 AM, Thrash Er <mingorrubioer at gmail.com> wrote:
>>
>> New to Lustre O:)
>>
>> I have to install and configure a Lustre storage for 4 small clusters
>> (4 different departments). Each cluster has its own IB QDR
>> interconnect for MPI (and now Lustre) and its own 1 GigE management
>> network. IB networks would be something like:
>>     Cluster A      192.168.1.0      o2ib0(ib0)
>>     Cluster B      192.168.2.0      o2ib1(ib1)
>>     Cluster C      192.168.3.0      o2ib2(ib2)
>>     Cluster D      192.168.4.0      o2ib3(ib3)
>>
>> I've gone through the Lustre Operations Manual 2.x and, from what I
>> understood, I would have to:
>>
>> 1.- add 4 IB ports to each OSS and MDS/MGT and cable them like this:
>>     IB Port 0 -> cluster A
>>     IB Port 1 -> cluster B
>>     IB Port 2 -> cluster C
>>     IB Port 3 -> cluster D
>>
>> 2.- configure /etc/modprobe.d/lustre.conf on the OSS and MDS like this:
>>
>>     options lnet networks="o2ib0(ib0),o2ib1(ib1),o2ib2(ib2),o2ib3(ib3)"
>>
>> 3.- configure /etc/modprobe.d/lustre.conf on each node of each cluster
>> like this:
>>
>>     Nodes con Cluster A:      options lnet networks="o2ib0(ib0)"
>>
>>     Nodes con Cluster B:      options lnet networks="o2ib1(ib1)"
>>
>>     Nodes con Cluster C:      options lnet networks="o2ib2(ib2)"
>>
>>     Nodes con Cluster D:      options lnet networks="o2ib3(ib3)"
>>
>>
>> Soooo, questions:
>>   1.- Are my assumptions correct?
>>   2.- No need for LNET routers, right?
>>   3.- Am I missing something?
>>
>> Thanks !!!!!!
>> _______________________________________________
>> lustre-discuss mailing list
>> lustre-discuss at lists.lustre.org
>> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
>