[Lustre-discuss] [HPDD-discuss] Failure to connect to some OST from a client machine

[Bob Ball]

There are very few clients right now on the tcp2 network.  AFAIK, they 
are happy.  No one is complaining, but they may not be using the Lustre 
storage right now, either.

bob

On 9/5/2013 4:22 PM, Kris Howard wrote:
> If you lctl ping 10.10.X.XX at tcp from both sides it should bring the 
> route up.
> With all of those down routes all is happy?
> hmm.
>
>
> On Thu, Sep 5, 2013 at 1:14 PM, Bob Ball <ball at umich.edu 
> <mailto:ball at umich.edu>> wrote:
>
>     That is an interesting mix.  Nothing shows up at all on the
>     clients, even on those 3 that route to a second NIC.  On the OSS,
>     it is quite the mix of up/down on the 3 routers, with no obvious
>     pattern.
>
>     Most of our traffic is on the 10.10 network, with the 3 machines
>     shown below routing to a small number of clients on a more public
>     network.
>
>     FYI, the current situation is one in which all machines are happy,
>     as far as I can tell.
>
>     bob
>
>     Running lctl show_route on all machines in lustre_fss.txt
>     On umdist05.local
>     net               tcp2 hops 1 gw 10.10.1.52 at tcp down
>     net               tcp2 hops 1 gw 10.10.1.51 at tcp down
>     net               tcp2 hops 1 gw 10.10.1.50 at tcp up
>              Succeeded
>     On umfs06.local
>     net               tcp2 hops 1 gw 10.10.1.51 at tcp down
>     net               tcp2 hops 1 gw 10.10.1.50 at tcp up
>     net               tcp2 hops 1 gw 10.10.1.52 at tcp up
>              Succeeded
>     On umdist01.local
>     net               tcp2 hops 1 gw 10.10.1.52 at tcp down
>     net               tcp2 hops 1 gw 10.10.1.51 at tcp down
>     net               tcp2 hops 1 gw 10.10.1.50 at tcp up
>              Succeeded
>     On umdist02.local
>     net               tcp2 hops 1 gw 10.10.1.52 at tcp down
>     net               tcp2 hops 1 gw 10.10.1.51 at tcp down
>     net               tcp2 hops 1 gw 10.10.1.50 at tcp up
>              Succeeded
>     On umdist03.local
>     net               tcp2 hops 1 gw 10.10.1.51 at tcp down
>     net               tcp2 hops 1 gw 10.10.1.52 at tcp up
>     net               tcp2 hops 1 gw 10.10.1.50 at tcp up
>              Succeeded
>     On umdist04.local
>     net               tcp2 hops 1 gw 10.10.1.52 at tcp down
>     net               tcp2 hops 1 gw 10.10.1.51 at tcp down
>     net               tcp2 hops 1 gw 10.10.1.50 at tcp up
>              Succeeded
>     On umdist07.local
>     net               tcp2 hops 1 gw 10.10.1.50 at tcp down
>     net               tcp2 hops 1 gw 10.10.1.52 at tcp down
>     net               tcp2 hops 1 gw 10.10.1.51 at tcp down
>              Succeeded
>     On umdist08.local
>     net               tcp2 hops 1 gw 10.10.1.50 at tcp down
>     net               tcp2 hops 1 gw 10.10.1.52 at tcp down
>     net               tcp2 hops 1 gw 10.10.1.51 at tcp down
>              Succeeded
>
>
>     On 9/5/2013 4:01 PM, Kris Howard wrote:
>>     Might check lctl show_route and look for downed routes.
>>
>>
>>     On Thu, Sep 5, 2013 at 12:56 PM, Bob Ball <ball at umich.edu
>>     <mailto:ball at umich.edu>> wrote:
>>
>>         We are running Lustre 2.1.6 on Scientific Linux 6.4, kernel
>>         2.6.32-358.11.1.el6.x86_64.  This was an upgrade from Lustre
>>         1.8.4 on SL5.
>>
>>         We have had a few situations lately where a client stops
>>         talking to some subset of the OST (about 58 of these total on
>>         8 OSS, nearly 500TB in total).  I have a couple of questions.
>>
>>         1. "lctl dl"  on the OSS shows a smaller count on the
>>         affected servers; on the client, all OSS showed UP in "lctl
>>         dl".  Today, I first tried rebooting this OSS, but that did
>>         not change the situation.  I ended up rebooting the client
>>         before I could get full connectivity.  Is there any way from
>>         the client to get the reconnect, short of rebooting that client?
>>
>>         2. It used to be the case under Lustre 1.8.4 that I could run
>>         "lfs df -h" on the client, and see all OST, even those where
>>         the connection was not working, for whatever reason.  That is
>>         no longer the case, now the lfs command stops at the first,
>>         non-talking OST. This seems more like a bug than a feature.
>>          Is there some other way to see a list of non-communicating
>>         OST on a client?
>>
>>         Thanks in advance for any help offered.
>>
>>         bob
>>
>>
>>
>>         _______________________________________________
>>         HPDD-discuss mailing list
>>         HPDD-discuss at lists.01.org <mailto:HPDD-discuss at lists.01.org>
>>         https://lists.01.org/mailman/listinfo/hpdd-discuss
>>
>>
>
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20130905/ce45e6e9/attachment.htm>