[Lustre-discuss] files/directories are temporarily unavailable on patchless clients

[Harald van Pee]

On Monday 21 January 2008 05:44 pm, you wrote:
> On Monday 21 January 2008 17:33:31 Harald van Pee wrote:
> > Hi,
> >
> > this is lustre 1.6.0.1 on redhat 4 update 4.
> > With debian etch patchless clients and debian kernels 2.6.18.
> > All are AMD64 bit.
> > We use 1 MGS, 1 MDT and 5 OSS with together 10 OST.
> >
> > The cluster is running since roughly 6 monthes, no problems on the server
> > side up to know!
> > The only problem on client side was, that two clients lost connection to
> > mdt or ost and after recovering some files can't be accessed on this
> > clients. After reboot of this clients everything was o.k.!
> >
> > But now we have a much more serious problem! It seems that it exists
> > since a couple of days, but we just have found out what the reason is! A
> > directory of one user with a couple of directories becomes temporarily
> > unavailable on some clients.
> > After a while one can access the directory again, but for me its not
> > clear what happens and it vanishes also after a while.
>
> What does this mean in detail? Does access to the directory just hang or is
> the directory not there?

The directory is just not there! Directory or file not found.

>
> > Reboot of the client does not change the situation.
> >
> > Now I have to ask for help, unfortunately its a production system and its
> > needed as soon as possible. An update is planned, but most likely not
> > realistic bevor middle of next month.
> >
> > What should I do next to solve or investigate the problem?
>
> Lustre writes lots of debug messages. For the beginning logs of a client
> would help.
in my opinion there is no error message on the clients which is directly 
related to the problem 
on our node0010 today I have seen this problem a several time.
Mostly the directory is not seen! Probably all of the other directories can be 
accessed at the same time.

and here all lustre related messages from the last days (others are mostly 
timestamps!)



Jan 17 07:41:16 node0010 kernel: Lustre: 5723:0:
(namei.c:235:ll_mdc_blocking_ast()) More than 1 alias dir 133798800 alias 2
Jan 17 07:41:16 node0010 kernel: Lustre: 5723:0:
(namei.c:235:ll_mdc_blocking_ast()) Skipped 2 previous similar messages
Jan 17 07:41:16 node0010 kernel: Lustre: 5716:0:
(namei.c:235:ll_mdc_blocking_ast()) More than 1 alias dir 133799550 alias 2
Jan 17 07:41:16 node0010 kernel: Lustre: 5716:0:
(namei.c:235:ll_mdc_blocking_ast()) Skipped 1 previous similar message
Jan 17 07:41:28 node0010 kernel: Lustre: 5721:0:
(namei.c:235:ll_mdc_blocking_ast()) More than 1 alias dir 133891907 alias 2
Jan 17 07:41:28 node0010 kernel: Lustre: 5721:0:
(namei.c:235:ll_mdc_blocking_ast()) More than 1 alias dir 133891908 alias 2
Jan 18 14:37:58 node0010 kernel: Lustre: 5716:0:
(namei.c:235:ll_mdc_blocking_ast()) More than 1 alias dir 134059689 alias 2
Jan 18 14:52:48 node0010 kernel: Lustre: 5723:0:
(namei.c:235:ll_mdc_blocking_ast()) More than 1 alias dir 134092437 alias 2
Jan 18 14:54:04 node0010 kernel: Lustre: 5717:0:
(namei.c:235:ll_mdc_blocking_ast()) More than 1 alias dir 134092815 alias 2
Jan 18 14:54:37 node0010 kernel: Lustre: 5719:0:
(namei.c:235:ll_mdc_blocking_ast()) More than 1 alias dir 134092988 alias 2
Jan 18 15:02:34 node0010 kernel: Lustre: 5723:0:
(namei.c:235:ll_mdc_blocking_ast()) More than 1 alias dir 134093893 alias 2
Jan 18 15:05:33 node0010 kernel: Lustre: 5722:0:
(namei.c:235:ll_mdc_blocking_ast()) More than 1 alias dir 134094054 alias 2
Jan 18 15:06:04 node0010 kernel: Lustre: 5721:0:
(namei.c:235:ll_mdc_blocking_ast()) More than 1 alias dir 134094179 alias 2
Jan 18 15:10:53 node0010 kernel: Lustre: 5723:0:
(namei.c:235:ll_mdc_blocking_ast()) More than 1 alias dir 134094366 alias 2
Jan 18 15:14:59 node0010 kernel: Lustre: 5718:0:
(namei.c:235:ll_mdc_blocking_ast()) More than 1 alias dir 134094555 alias 2
Jan 18 15:15:44 node0010 kernel: Lustre: 5723:0:
(namei.c:235:ll_mdc_blocking_ast()) More than 1 alias dir 134094694 alias 2
Jan 18 15:22:55 node0010 kernel: Lustre: 5721:0:
(namei.c:235:ll_mdc_blocking_ast()) More than 1 alias dir 134094871 alias 2
Jan 18 15:23:30 node0010 kernel: Lustre: 5719:0:
(namei.c:235:ll_mdc_blocking_ast()) More than 1 alias dir 134119425 alias 2
Jan 18 15:28:37 node0010 kernel: Lustre: 5721:0:
(namei.c:235:ll_mdc_blocking_ast()) More than 1 alias dir 134119833 alias 2
Jan 18 16:15:58 node0010 kernel: Lustre: 5718:0:
(namei.c:235:ll_mdc_blocking_ast()) More than 1 alias dir 134059687 alias 2
Jan 18 22:27:36 node0010 kernel: Lustre: 5723:0:
(namei.c:235:ll_mdc_blocking_ast()) More than 1 alias dir 134092437 alias 2
Jan 18 22:27:36 node0010 kernel: Lustre: 5723:0:
(namei.c:235:ll_mdc_blocking_ast()) Skipped 2 previous similar messages
Jan 18 22:27:49 node0010 kernel: Lustre: 5723:0:
(namei.c:235:ll_mdc_blocking_ast()) More than 1 alias dir 134092648 alias 2
Jan 18 22:28:45 node0010 kernel: Lustre: 5721:0:
(namei.c:235:ll_mdc_blocking_ast()) More than 1 alias dir 134092815 alias 2
Jan 18 22:29:46 node0010 kernel: Lustre: 5720:0:
(namei.c:235:ll_mdc_blocking_ast()) More than 1 alias dir 134092988 alias 2
Jan 18 22:29:53 node0010 kernel: Lustre: 5722:0:
(namei.c:235:ll_mdc_blocking_ast()) More than 1 alias dir 134093165 alias 2
Jan 18 22:30:03 node0010 kernel: Lustre: 5718:0:
(namei.c:235:ll_mdc_blocking_ast()) More than 1 alias dir 134093358 alias 2
Jan 18 22:30:39 node0010 kernel: Lustre: 5722:0:
(namei.c:235:ll_mdc_blocking_ast()) More than 1 alias dir 134093732 alias 2
Jan 18 22:31:40 node0010 kernel: Lustre: 5718:0:
(namei.c:235:ll_mdc_blocking_ast()) More than 1 alias dir 134093893 alias 2
Jan 18 22:32:46 node0010 kernel: Lustre: 5721:0:
(namei.c:235:ll_mdc_blocking_ast()) More than 1 alias dir 134094054 alias 2
Jan 18 22:34:27 node0010 kernel: Lustre: 5716:0:
(namei.c:235:ll_mdc_blocking_ast()) More than 1 alias dir 134094179 alias 2
Jan 18 22:35:37 node0010 kernel: Lustre: 5720:0:
(namei.c:235:ll_mdc_blocking_ast()) More than 1 alias dir 134094555 alias 2
Jan 18 22:35:37 node0010 kernel: Lustre: 5720:0:
(namei.c:235:ll_mdc_blocking_ast()) Skipped 1 previous similar message
Jan 18 22:37:47 node0010 kernel: Lustre: 5716:0:
(namei.c:235:ll_mdc_blocking_ast()) More than 1 alias dir 134094694 alias 2
Jan 19 08:30:49 node0010 kernel: Lustre: 5717:0:
(namei.c:235:ll_mdc_blocking_ast()) More than 1 alias dir 134059690 alias 2
Jan 19 08:30:49 node0010 kernel: Lustre: 5717:0:
(namei.c:235:ll_mdc_blocking_ast()) Skipped 3 previous similar messages
Jan 19 08:32:33 node0010 kernel: Lustre: 5721:0:
(namei.c:235:ll_mdc_blocking_ast()) More than 1 alias dir 134060498 alias 2
Jan 19 08:34:02 node0010 kernel: Lustre: 5716:0:
(namei.c:235:ll_mdc_blocking_ast()) More than 1 alias dir 134061258 alias 2
Jan 19 08:35:21 node0010 kernel: Lustre: 5716:0:
(namei.c:235:ll_mdc_blocking_ast()) More than 1 alias dir 134091392 alias 2
Jan 19 08:35:21 node0010 kernel: Lustre: 5716:0:
(namei.c:235:ll_mdc_blocking_ast()) Skipped 1 previous similar message
Jan 19 08:37:41 node0010 kernel: Lustre: 5722:0:
(namei.c:235:ll_mdc_blocking_ast()) More than 1 alias dir 134091529 alias 2
Jan 19 08:37:41 node0010 kernel: Lustre: 5722:0:
(namei.c:235:ll_mdc_blocking_ast()) Skipped 1 previous similar message
Jan 19 08:43:42 node0010 kernel: Lustre: 5718:0:
(namei.c:235:ll_mdc_blocking_ast()) More than 1 alias dir 134091822 alias 2
Jan 19 08:43:42 node0010 kernel: Lustre: 5718:0:
(namei.c:235:ll_mdc_blocking_ast()) Skipped 2 previous similar messages
Jan 19 08:53:04 node0010 kernel: Lustre: 5721:0:
(namei.c:235:ll_mdc_blocking_ast()) More than 1 alias dir 134059688 alias 2
Jan 19 08:53:04 node0010 kernel: Lustre: 5721:0:
(namei.c:235:ll_mdc_blocking_ast()) Skipped 2 previous similar messages
Jan 19 09:05:22 node0010 kernel: Lustre: 5722:0:
(namei.c:235:ll_mdc_blocking_ast()) More than 1 alias dir 134093165 alias 2
Jan 19 09:05:22 node0010 kernel: Lustre: 5722:0:
(namei.c:235:ll_mdc_blocking_ast()) Skipped 4 previous similar messages
Jan 19 09:15:42 node0010 kernel: Lustre: 5721:0:
(namei.c:235:ll_mdc_blocking_ast()) More than 1 alias dir 134094179 alias 2
Jan 19 09:15:42 node0010 kernel: Lustre: 5721:0:
(namei.c:235:ll_mdc_blocking_ast()) Skipped 4 previous similar messages
Jan 19 09:50:20 node0010 kernel: Lustre: 5719:0:
(namei.c:235:ll_mdc_blocking_ast()) More than 1 alias dir 134094871 alias 2
Jan 19 09:50:20 node0010 kernel: Lustre: 5719:0:
(namei.c:235:ll_mdc_blocking_ast()) Skipped 3 previous similar messages
Jan 19 09:56:23 node0010 kernel: Lustre: 5717:0:
(namei.c:235:ll_mdc_blocking_ast()) More than 1 alias dir 134119425 alias 2
Jan 19 10:01:28 node0010 kernel: Lustre: 5718:0:
(namei.c:235:ll_mdc_blocking_ast()) More than 1 alias dir 134119630 alias 2
Jan 19 10:11:43 node0010 kernel: Lustre: 5723:0:
(namei.c:235:ll_mdc_blocking_ast()) More than 1 alias dir 134119833 alias 2
Jan 20 00:58:03 node0010 kernel: Lustre: 5717:0:
(namei.c:235:ll_mdc_blocking_ast()) More than 1 alias dir 134094366 alias 2
Jan 20 00:58:03 node0010 kernel: Lustre: 5717:0:
(namei.c:235:ll_mdc_blocking_ast()) Skipped 3 previous similar messages
Jan 20 01:03:04 node0010 kernel: Lustre: 5722:0:
(namei.c:235:ll_mdc_blocking_ast()) More than 1 alias dir 134094555 alias 2
Jan 20 01:59:56 node0010 kernel: Lustre: 5722:0:
(namei.c:235:ll_mdc_blocking_ast()) More than 1 alias dir 134060498 alias 2
Jan 20 02:00:18 node0010 kernel: Lustre: 5721:0:
(namei.c:235:ll_mdc_blocking_ast()) More than 1 alias dir 134059690 alias 2
Jan 20 02:01:31 node0010 kernel: Lustre: 5717:0:
(namei.c:235:ll_mdc_blocking_ast()) More than 1 alias dir 134059688 alias 2
Jan 20 02:47:29 node0010 kernel: Lustre: 5716:0:
(namei.c:235:ll_mdc_blocking_ast()) More than 1 alias dir 134059687 alias 2
Jan 20 02:47:29 node0010 kernel: Lustre: 5716:0:
(namei.c:235:ll_mdc_blocking_ast()) Skipped 1 previous similar message
Jan 20 22:28:27 node0010 kernel: Lustre: 5721:0:
(namei.c:235:ll_mdc_blocking_ast()) More than 1 alias dir 134092396 alias 2
Jan 20 22:28:27 node0010 kernel: Lustre: 5721:0:
(namei.c:235:ll_mdc_blocking_ast()) Skipped 1 previous similar message
Jan 20 22:28:59 node0010 kernel: Lustre: 5718:0:
(namei.c:235:ll_mdc_blocking_ast()) More than 1 alias dir 134092437 alias 2
Jan 20 23:19:09 node0010 kernel: Lustre: 5722:0:
(namei.c:235:ll_mdc_blocking_ast()) More than 1 alias dir 134093165 alias 2
Jan 20 23:39:11 node0010 kernel: Lustre: 5716:0:
(namei.c:235:ll_mdc_blocking_ast()) More than 1 alias dir 134093358 alias 2
Jan 21 11:10:39 node0010 kernel: Lustre: 5720:0:
(namei.c:235:ll_mdc_blocking_ast()) More than 1 alias dir 134059689 alias 2
Jan 21 14:22:44 node0010 kernel: Lustre: 5721:0:
(namei.c:235:ll_mdc_blocking_ast()) More than 1 alias dir 134091343 alias 3
Jan 21 14:38:43 node0010 kernel: Lustre: 5720:0:
(namei.c:235:ll_mdc_blocking_ast()) More than 1 alias dir 134091486 alias 3
Jan 21 15:08:20 node0010 kernel: Lustre: 5716:0:
(namei.c:235:ll_mdc_blocking_ast()) More than 1 alias dir 134091822 alias 3
Jan 21 15:31:26 node0010 kernel: Lustre: 5723:0:
(namei.c:235:ll_mdc_blocking_ast()) More than 1 alias dir 134059690 alias 3
Jan 21 15:32:42 node0010 kernel: Lustre: 5719:0:
(namei.c:235:ll_mdc_blocking_ast()) More than 1 alias dir 134092333 alias 3
Jan 21 15:37:00 node0010 kernel: Lustre: 5719:0:
(namei.c:235:ll_mdc_blocking_ast()) More than 1 alias dir 134060498 alias 3
Jan 21 15:42:40 node0010 kernel: Lustre: 5718:0:
(namei.c:235:ll_mdc_blocking_ast()) More than 1 alias dir 134061258 alias 3
Jan 21 15:57:33 node0010 kernel: Lustre: 5721:0:
(namei.c:235:ll_mdc_blocking_ast()) More than 1 alias dir 134091529 alias 3
Jan 21 16:32:29 node0010 kernel: Lustre: 5719:0:
(namei.c:235:ll_mdc_blocking_ast()) More than 1 alias dir 134091683 alias 3
Jan 21 17:11:16 node0010 kernel: Lustre: 5720:0:
(namei.c:235:ll_mdc_blocking_ast()) More than 1 alias dir 134091985 alias 3
Jan 21 17:32:25 node0010 kernel: Lustre: 5717:0:
(namei.c:235:ll_mdc_blocking_ast()) More than 1 alias dir 134059688 alias 3
Jan 21 17:32:25 node0010 kernel: Lustre: 5721:0:
(namei.c:235:ll_mdc_blocking_ast()) More than 1 alias dir 134092182 alias 3
Jan 21 18:12:51 node0010 kernel: Lustre: 5717:0:
(namei.c:235:ll_mdc_blocking_ast()) More than 1 alias dir 134059687 alias 3
Jan 21 18:12:51 node0010 kernel: Lustre: 5722:0:
(namei.c:235:ll_mdc_blocking_ast()) More than 1 alias dir 134094694 alias 2
Jan 21 18:12:51 node0010 kernel: Lustre: 5722:0:
(namei.c:235:ll_mdc_blocking_ast()) Skipped 4 previous similar messages
Jan 21 18:12:51 node0010 kernel: Lustre: 5717:0:
(namei.c:235:ll_mdc_blocking_ast()) More than 1 alias dir 134120476 alias 2
Jan 21 18:12:51 node0010 kernel: Lustre: 5717:0:
(namei.c:235:ll_mdc_blocking_ast()) Skipped 6 previous similar messages


>
>
> Cheers,
> Bernd

Helmholtz-Institut fuer Strahlen- und Kernphysik der Universitaet Bonn