[Lustre-discuss] Large directories optimization

[Brian J. Murrell]

On Thu, 2009-09-17 at 13:28 +0200, Lukas Hejtmanek wrote:
> Hello,

Hi,

> is it possible to optimize Lustre so that is supports really large directories
> (with 30k small files in it)?

We already have optimizations for large directories (i.e. HTREE
indexing, etc.).

> We have 8 physical clients which process jpeg
> files stored on Lustre volume and I get sooner or later client freezes - ls in
> Lustre directory waits forever. I there something I could do to improve
> performance?

I suspect you don't really have a "performance" issue.

> I got the following messages on the client:
> Lustre: stable-MDT0000-mdc-ffff8802855b7800: Connection to service
> stable-MDT0000 via nid x.x.x.x at tcp was lost; in progress operations using
> this service will wait for recovery to complete.

So this immediately points to the MDT.  The above message is saying that
the connection to the MDT was lost.  The question becomes "why?".

What did the MDS log around the time the above message was logged on the
client?  FWIW, it's much better to look at the syslog for this sort of
thing than dmesg as the syslog provides timing context.

b.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 197 bytes
Desc: This is a digitally signed message part
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20090917/37896fce/attachment.pgp>