<div>Hello List,</div><div><br></div><div>I wanted to outline our proposed configuration and hope for some feedback from the list on whether the LNET config is sound.</div><div><br></div><div>We're planning a site-wide (HPC clusters) Lustre filesystem at NYU to be installed in the next few weeks.Lustre version 1.8.1.</div>
<div>The MDS, OSS, and routers will be running Redhat 5.3 with Lustre Kernels.</div><div>Our cluster compute/login nodes will be running Rocks/Redhat 5.1 with lustre kernels.</div><div><br></div><div>We've installed a small test cluster (1 MDS/MGS, 2 OSS, and 4 compute clients) with the same versions and it works well.</div>
<div><br></div><div>Note: Sun is going to be onsite to install the MDS failover pair as part of their "Lustre Building Blocks" (Hi Hung-Sheng)</div><div><br></div><div>Here goes.</div><div><br></div><div>Core Lustre Network:</div>
<div>We have two Mellanox MTS3600 36-Port QDR switches. I thought we'd configure one as o2ib0, the other o2ib1.</div><div>There's also the possibility of combining them and connecting each dual port HCA to each switch.</div>
<div><br></div><div>Servers on the "Core Lustre Network" will be known as $SERVER_TYPE-$SERVER_NAME-$IB_RANK, so we have:</div><div><br></div><div>2 MDS/MGS servers configured as a failover pair. Each has a dual port IB QDR HCA.</div>
<div>MDS servers would be on both core networks, o2ib0 and o2ib1.</div><div>mds-0-0 #o2ib0</div><div>mds-0-1 #o2ib1</div><div>mds-1-0 #o2ib0</div><div>mds-1-1 #o2ib1</div><div><br></div><div>4 OSS servers will be configured as 2 sets of failover pairs. Each has a dual port IB QDR HCA.</div>
<div>OSS servers would be on both core networks, o2ib0 and o2ib1.</div><div>oss-0-0 #o2ib0</div><div>oss-0-1 #o2ib1</div><div>...</div><div>oss-3-0</div><div>oss-3-1</div><div><br></div><div>Each cluster (currently 3) will have 2 routers on the "Lustre Core" network(s) and the "Private Cluster IB" network.</div>
<div>2 of the clusters have DDR private IB networks. The other cluster has a QDR private IB network. I know the 2 QDR switches are overkill</div><div>but they were relatively cheap and should survive adding more clients and storage.</div>
<div><br></div><div>The DDR cluster's routers will have 2 routers, each with 1 dual port QDR HCA (core) and 1 single-port DDR HCA (private).</div><div>The QDR cluster's routers will have 2 dual port QDR HCAs, 1 core, 1 private.</div>
<div><br></div><div>Here's where I'm not 100000% sure about the proper LNET config.</div><div>Let's assume the cluster we're talking about is o2ib4</div><div><br></div><div>1. Each cluster sees only one core network o2ib0 OR o2ib1.</div>
<div>This roughly corresponds to the multi-rail config in the manual but balancing by cluster (not perfect I know).</div><div><br></div><div>The routers would be configured with 1 address on the "Lustre Core Network" and 1 address on the "Private Cluster IB".</div>
<div>Clients would specify mds-0-0:mds-1-0 or mds-0-1:mds-1-1 as the metadata failover pair when mounting the cluster. Both servers on o2ib0.</div><div>options lnet="o2ib4(ib0)" routes="o2ib0 local.router.ip.[1-2]@o2ib4;"</div>
<div>or</div><div>options lnet="o2ib4(ib0)" routes="o2ib1 local.router.ip.[1-2]@o2ib4;"</div><div><br></div><div>2. Each cluster uses both core networks</div><div>The routers would be configured with 1 address on o2ib0, 1 address on o2ib1, and 1 address on the "Private Cluster IB".</div>
<div>Compute clients would specify mds-0-0,mds-0-1:mds-1-0,mds-0-1</div><div>Compute clients would use:</div><div>options lnet="o2ib4(ib0)" routes="o2ib0 local.router.ip.[1-2]@o2ib4; o2ib1 local.router.ip.[1-2]@o2ib4;"</div>
<div><br></div><div>Will that work?</div><div>If once switch fails, clients should fail over to the other mds address pair as both addresses become unreachable.</div><div>If an MDS fails, the clients should stay on the same o2ib network but fail over to the other MDS.</div>
<div><br></div><div>Is this possible? I would think that even in the second configuration we'd want to manually balance the traffic.</div><div>Some clients would specify o2ib0 first while other specify o2ib1 first.</div>
<div><br></div><div>Erik</div>