<HTML><BODY style="word-wrap: break-word; -khtml-nbsp-mode: space; -khtml-line-break: after-white-space; "><DIV><BR class="khtml-block-placeholder"></DIV><DIV><FONT class="Apple-style-span" face="Courier">There are a number of LNET routing examples in the Lustre documentation but the actual LNET topology requirements don't seem to be spelled out. In the simplest terms, what are the actual network topology requirements for the MDS, OSS's, and clients in order for lustre to work correctly?</FONT></DIV><DIV><FONT class="Apple-style-span" face="Courier"><BR class="khtml-block-placeholder"></FONT></DIV><DIV><FONT class="Apple-style-span" face="Courier">MDS NIDS</FONT></DIV><DIV><FONT class="Apple-style-span" face="Courier">-------------------------</FONT></DIV><DIV><FONT class="Apple-style-span" face="Courier">10.13.24.40@o2ib</FONT></DIV><DIV><FONT class="Apple-style-span" face="Courier">10.13.16.40@tcp (ip/ethernet) ----------------------------------------</FONT></DIV><DIV><FONT class="Apple-style-span" face="Courier"> |</FONT></DIV><DIV><FONT class="Apple-style-span" face="Courier">OSS NIDS |</FONT></DIV><DIV><FONT class="Apple-style-span" face="Courier">----------------------- |</FONT></DIV><DIV><FONT class="Apple-style-span" face="Courier">10.13.24.42@o2ib |</FONT></DIV><DIV><FONT class="Apple-style-span" face="Courier">10.13.31.229@tcp (ipoib subnet) --------------------------- ipoib gateway</FONT></DIV><DIV><FONT class="Apple-style-span" face="Courier"> |</FONT></DIV><DIV><FONT class="Apple-style-span" face="Courier">IB Client NIDS |</FONT></DIV><DIV><FONT class="Apple-style-span" face="Courier">----------------------- |</FONT></DIV><DIV><FONT class="Apple-style-span" face="Courier">10.13.25.150@o2ib |</FONT></DIV><DIV><FONT class="Apple-style-span" face="Courier"> |</FONT></DIV><DIV><FONT class="Apple-style-span" face="Courier">Ethernet Client NIDS |</FONT></DIV><DIV><FONT class="Apple-style-span" face="Courier">---------------------------- |</FONT></DIV><DIV><FONT class="Apple-style-span" face="Courier">10.13.18.152@tcp ip/ethernet -------------------------------------------</FONT></DIV><DIV><FONT class="Apple-style-span" face="Courier"><BR class="khtml-block-placeholder"></FONT></DIV><DIV><FONT class="Apple-style-span" face="Courier"><BR class="khtml-block-placeholder"></FONT></DIV><DIV><FONT class="Apple-style-span" face="Courier">There is no problem with the o2ib side of things. The IB network is 10.13.24.0/21 so the clients, mds, and oss's are all on the same "network". </FONT></DIV><DIV><FONT class="Apple-style-span" face="Courier"><BR class="khtml-block-placeholder"></FONT></DIV><DIV><FONT class="Apple-style-span" face="Courier">The ethernet clients have no access to the 10.13.24.0 network. They communicate with the MDS directly on the 10.13.16.0/22 network. However, the OSSs for the ethernet side all sit on their own subnetted /30 network each of which is bridged by an IPoIB gateway between the OSSs and the ethernet clients. So the MDS and the clients talk to the OSSs through the IPoIB gateway and the OSSs can talk to each other through the IPoIB Gateway (though on separate subnets - there is a good reason for this, trust me :) ). So there is complete connectivity in the "IP" sense.</FONT></DIV><DIV><FONT class="Apple-style-span" face="Courier"><BR class="khtml-block-placeholder"></FONT></DIV><DIV><FONT class="Apple-style-span" face="Courier">However, after mounting the lustre file system on the ethernet clients (which succeeds), the clients are always evicted immediately following the obd_timeout period with a message such as...</FONT></DIV><DIV><FONT class="Apple-style-span" face="Courier"><BR class="khtml-block-placeholder"></FONT></DIV><DIV><FONT class="Apple-style-span" face="Courier">Lustre: ufhpc-MDT0000: haven't heard from client f6c6db8a-6fbc-6464-6261-af9ecfc0cb60 (at 10.13.18.152@tcp) in 2462 seconds. I think it's dead, and I am evicting it.</FONT></DIV><DIV><FONT class="Apple-style-span" face="Courier"><BR class="khtml-block-placeholder"></FONT></DIV><DIV><FONT class="Apple-style-span" face="Courier">Of course, once that happens, the client can no longer write to the file system although it works fine for the "obd_timeout" period. We don't completely understand this because all the players can talk to each other via IP. Why would the MDT not be "hearing" from the ethernet client? They are on the same IP network. It seems like the problem is in the lnet topology? Do we really have to introduce an lnet router even though we have complete IP connectivity among the various components (mds, oss, clients)?</FONT></DIV><DIV><FONT class="Apple-style-span" face="Courier"><BR class="khtml-block-placeholder"></FONT></DIV><DIV><FONT class="Apple-style-span" face="Courier">I'm hoping that someone more familiar with the LNET abstraction layer can help us understand what the problem is.</FONT></DIV><DIV><FONT class="Apple-style-span" face="Courier"><BR class="khtml-block-placeholder"></FONT></DIV><DIV><FONT class="Apple-style-span" face="Courier">Thanks,</FONT></DIV><DIV><FONT class="Apple-style-span" face="Courier"><BR class="khtml-block-placeholder"></FONT></DIV><DIV><FONT class="Apple-style-span" face="Courier">charlie taylor</FONT></DIV><DIV><FONT class="Apple-style-span" face="Courier">uf hpc center</FONT></DIV><DIV><BR class="khtml-block-placeholder"></DIV><DIV><BR class="khtml-block-placeholder"></DIV></BODY></HTML>