[Lustre-discuss] slow journal/commitrw on OSTs lead to crash

[Daniel Leaberry]

Hendelman, Rob wrote:
> Hello,

<snip>

> 
> Our OST's are almost completely full (85-90% on each) and our MDT is
> almost entirely empty.
> 


I ran into a similar scenario with lustre when I hit 80% full as well.
Exact same problem with journal commits and disks seemingly unusable.
iostat on the disks (DDN 9500 array) shows huge numbers of small reads.
Almost like the disk is being scanned.

I eventually decided it must be something in the filesystem code that
changes how it allocates free blocks when the disk is ~80% full.

I have no confirmation about this other than I've been through the exact
same thing you're seeing and now we consider our 90TB of lustre full
when we hit 77% usage.

Daniel Leaberry
Senior Systems Administrator