[Lustre-discuss] odd mount behavior

[Heald, Nathan T.]

While I've never used Lustre on IB, I have seen clients with very similar symptoms before when the network appears to be functioning properly. You should verify that jumbo frames are getting passed properly between your networks (or switch ports if both hosts are on the same network segment). If your MDS has MTU set for 9000 but the client has MTU at the default of 1500 then the MDS will send a jumbo frame at the very last moment in the mount process and it never shows up on the client. The symptom is repeated timeout errors in syslog during the mount on both the client and MDS and a hanging mount command.

Assuming you are using jumbo frames send a packet larger than MTU1500 from both hosts, if it fails it's an MTU issue:
ping -s 8000 10.4.200.6

If there's a router between your hosts you might try sending a jumbo frame with Don't-Fragment set, sometimes routers can silently drop these:
ping -s 8000 -M do 10.4.200.6

Again, if there's a router between your hosts you can double check with the "tracepath" command which sets the DF bit. If it stops returning output at some point between your hosts then Path-MTU-Discovery will not work and hence the client on that network segment will get this issue while clients on other networks may work fine. The next hop beyond the last IP which returns PMTU packets in the tracepath output is the one silently dropping DF packets on you, use traceroute to compare and locate the offender.

If both hosts are on the same network segment (no router between them) then this should not be an issue anyway as it's assumed both hosts are configured for the proper MTU of their local network. I haven't tested this on IB, but on Myricom 10GB nics we found that if one host has the wrong MTU set you may see the error count for the interface increment during the failing mount attempt. Since the network stack on the client with the smaller MTU can't handle the jumbo frames, it tosses them and increments the error counter (visible with "netstat -I"). Your mileage may vary with IB.

Hope this helps.
-Nathan


On 3/28/10 3:18 AM, "John White" <jwhite at lbl.gov> wrote:

Unfortunately for this case, there is a clear network path, no firewalls.  A quick telnet test at least confirms something is listening on the port and closes the connection pretty quickly.  If networking were the case, wouldn't I still see connection errors for the @tcp NID?
----------------
John White
High Performance Computing Services (HPCS)
(510) 486-7307
One Cyclotron Rd, MS: 50B-3209C
Lawrence Berkeley National Lab
Berkeley, CA 94720








On Mar 26, 2010, at 9:49 PM, Andreas Dilger wrote:

> On 2010-03-26, at 17:45, John White wrote:
>>      We've got a new client we're trying to get to mount an existing file system.  The host cluster is set up with 2 NIDs for the MDT (o2ib, tcp), same with the client.  When I try mounting via tcp (mount -t lustre -o flock n0006.lustre at tcp:/vulcan /clusterfs/vulcan/pscratch), it just hangs there indefinitely with repeated messages like:
>>
>> Lustre: Request x264 sent from vulcan-MDT0000-mdc-ffff8101e39bb400 to NID 10.4.200.6 at o2ib 5s ago has timed out (limit 5s).
>>
>> I'm confused why it's attempting the o2ib NID repeatedly and never tries the tcp NID... Ideas?
>
>
> A common cause for newly-installed systems is hosts.deny or firewall rules that are preventing connections on port 988.
>
> Cheers, Andreas
> --
> Andreas Dilger
> Sr. Staff Engineer, Lustre Group
> Sun Microsystems of Canada, Inc.
>

_______________________________________________
Lustre-discuss mailing list
Lustre-discuss at lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20100329/e6d995d9/attachment.htm>