[Lustre-discuss] Performance dropoff for a nearly full Lustre file system

[Dilger, Andreas]

Note that due to the nature of spinning disks, you will always get a performance loss as the disk fills up, because the outside of the disk (low sector numbers) can store more data more quickly than the inside of the disk (high sector numbers).  From graphs that I've seen, the low sector numbers can do IO about twice as fast as high sector numbers. (This obviously doesn't apply to SSDs.)

The underlying ldiskfs filesystem will bias allocations of inodes, and in turn the blocks, to the low sector numbers  for best performance.  As the filesystem fills, that means it will start allocating these 1/2 as fast sectors.

Of course, fragmentation also plays a role, which is why ldiskfs will reserve 5% of the disk by default to avoid permanent performance loss caused by fragmentation if the filesystem gets totally full.

Cheers, Andreas

On Jan 14, 2015, at 12:43, Mike Selway <mselway at cray.com<mailto:mselway at cray.com>> wrote:

Hello,
               I’m looking for experiences for what has been observed to happen (performance drop offs, severity of drops, partial/full failures, …) when an operational Lustre File System has been almost “filled”… percentages of interest are in the range from say 80% to 99%.  Multiple responses appreciated.

Also, comments from anyone who has implemented a Robin Hood approach, about how they worked to avoid performance drop offs of a “near full” file system by “archiving and releasing data blocks” to auto-reconstruct continuous data areas.

Thanks!
Mike

Mike Selway | Sr. Storage Architect (TAS) | Cray Inc.
Work +1-301-332-4116 | mselway at cray.com<mailto:mselway at cray.com>
146 Castlemaine Ct,   Castle Rock,  CO  80104|   Check out Tiered Adaptive Storage (TAS)!<http://www.cray.com/Products/Storage/Tiered-Adaptive-Storage.aspx>

<image001.png><http://www.cray.com/>
<image002.jpg>

_______________________________________________
Lustre-discuss mailing list
Lustre-discuss at lists.lustre.org<mailto:Lustre-discuss at lists.lustre.org>
http://lists.lustre.org/mailman/listinfo/lustre-discuss
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 5290 bytes
Desc: image001.png
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20150115/ed13287e/attachment.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image002.jpg
Type: image/jpeg
Size: 2329 bytes
Desc: image002.jpg
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20150115/ed13287e/attachment.jpg>