<div dir="ltr"><div><div><div><div>Greetings,<br><br></div>I concur with Mr. Ball that running a Lustre file system in excess of 90% full can be problematic. In my personal experience numbers above 92% have caused slow response times for users especially for write activity. Depending upon your Lustre stripe set-up, the system takes a longer time using the Lustre default stripe of one in its attempt to locate enough space to store a file. It a larger stripe number is used, the problem does not go away but it is lessened. I have had a few experiences in which if one OST completely fills to 100% then nothing else may be written anywhere on that single-mount-point Lustre file system. Man oh man! Have I heard user complaints about that! "What do you mean no more space? A df shows me another 800Gb (on a 100Tb files system)".<br><br></div>That said, I have had success with creating a folder with a specified stripe size of two or so less than the total number of OSTs in the file system and putting files into that striped folder until I can re-balance the file system either by a clean-up of deleting files, moving them to tape or some other archive system, or until I can add more OSTs (I like that grow feature!).<br><br></div>Someone once said that files will grow to consume all available space. I forget the attribution.<br><br></div>Cheers,<br>megan<br></div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Jan 14, 2015 at 3:55 PM, <span dir="ltr"><<a href="mailto:lustre-discuss-request@lists.lustre.org" target="_blank">lustre-discuss-request@lists.lustre.org</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Send Lustre-discuss mailing list submissions to<br>
<a href="mailto:lustre-discuss@lists.lustre.org">lustre-discuss@lists.lustre.org</a><br>
<br>
To subscribe or unsubscribe via the World Wide Web, visit<br>
<a href="http://lists.lustre.org/mailman/listinfo/lustre-discuss" target="_blank">http://lists.lustre.org/mailman/listinfo/lustre-discuss</a><br>
or, via email, send a message with subject or body 'help' to<br>
<a href="mailto:lustre-discuss-request@lists.lustre.org">lustre-discuss-request@lists.lustre.org</a><br>
<br>
You can reach the person managing the list at<br>
<a href="mailto:lustre-discuss-owner@lists.lustre.org">lustre-discuss-owner@lists.lustre.org</a><br>
<br>
When replying, please edit your Subject line so it is more specific<br>
than "Re: Contents of Lustre-discuss digest..."<br>
<br>
<br>
Today's Topics:<br>
<br>
1. Performance dropoff for a nearly full Lustre file system<br>
(Mike Selway)<br>
2. Re: Performance dropoff for a nearly full Lustre file system<br>
(Bob Ball)<br>
<br>
<br>
----------------------------------------------------------------------<br>
<br>
Message: 1<br>
Date: Wed, 14 Jan 2015 19:43:02 +0000<br>
From: Mike Selway <<a href="mailto:mselway@cray.com">mselway@cray.com</a>><br>
To: "<a href="mailto:lustre-discuss@lists.lustre.org">lustre-discuss@lists.lustre.org</a>"<br>
<<a href="mailto:lustre-discuss@lists.lustre.org">lustre-discuss@lists.lustre.org</a>><br>
Subject: [Lustre-discuss] Performance dropoff for a nearly full Lustre<br>
file system<br>
Message-ID:<br>
<<a href="mailto:5073651DB6C02643B8739403BE96A0E27BCE4D@CFWEX01.americas.cray.com">5073651DB6C02643B8739403BE96A0E27BCE4D@CFWEX01.americas.cray.com</a>><br>
Content-Type: text/plain; charset="us-ascii"<br>
<br>
Hello,<br>
I'm looking for experiences for what has been observed to happen (performance drop offs, severity of drops, partial/full failures, ...) when an operational Lustre File System has been almost "filled"... percentages of interest are in the range from say 80% to 99%. Multiple responses appreciated.<br>
<br>
Also, comments from anyone who has implemented a Robin Hood approach, about how they worked to avoid performance drop offs of a "near full" file system by "archiving and releasing data blocks" to auto-reconstruct continuous data areas.<br>
<br>
Thanks!<br>
Mike<br>
<br>
Mike Selway | Sr. Storage Architect (TAS) | Cray Inc.<br>
Work <a href="tel:%2B1-301-332-4116" value="+13013324116">+1-301-332-4116</a> | <a href="mailto:mselway@cray.com">mselway@cray.com</a><br>
146 Castlemaine Ct, Castle Rock, CO 80104| Check out Tiered Adaptive Storage (TAS)!<<a href="http://www.cray.com/Products/Storage/Tiered-Adaptive-Storage.aspx" target="_blank">http://www.cray.com/Products/Storage/Tiered-Adaptive-Storage.aspx</a>><br>
<br>
[cid:image001.png@01CF36E5.85AF42A0]<<a href="http://www.cray.com/" target="_blank">http://www.cray.com/</a>><br>
[cid:image002.jpg@01D02FF7.A1B0F1E0]<br>
<br>
-------------- next part --------------<br>
An HTML attachment was scrubbed...<br>
URL: <<a href="http://lists.lustre.org/pipermail/lustre-discuss/attachments/20150114/65e3298d/attachment-0001.html" target="_blank">http://lists.lustre.org/pipermail/lustre-discuss/attachments/20150114/65e3298d/attachment-0001.html</a>><br>
-------------- next part --------------<br>
A non-text attachment was scrubbed...<br>
Name: image001.png<br>
Type: image/png<br>
Size: 5290 bytes<br>
Desc: image001.png<br>
URL: <<a href="http://lists.lustre.org/pipermail/lustre-discuss/attachments/20150114/65e3298d/attachment-0001.png" target="_blank">http://lists.lustre.org/pipermail/lustre-discuss/attachments/20150114/65e3298d/attachment-0001.png</a>><br>
-------------- next part --------------<br>
A non-text attachment was scrubbed...<br>
Name: image002.jpg<br>
Type: image/jpeg<br>
Size: 2329 bytes<br>
Desc: image002.jpg<br>
URL: <<a href="http://lists.lustre.org/pipermail/lustre-discuss/attachments/20150114/65e3298d/attachment-0001.jpg" target="_blank">http://lists.lustre.org/pipermail/lustre-discuss/attachments/20150114/65e3298d/attachment-0001.jpg</a>><br>
<br>
------------------------------<br>
<br>
Message: 2<br>
Date: Wed, 14 Jan 2015 15:55:13 -0500<br>
From: Bob Ball <<a href="mailto:ball@umich.edu">ball@umich.edu</a>><br>
To: Mike Selway <<a href="mailto:mselway@cray.com">mselway@cray.com</a>>, "<a href="mailto:lustre-discuss@lists.lustre.org">lustre-discuss@lists.lustre.org</a>"<br>
<<a href="mailto:lustre-discuss@lists.lustre.org">lustre-discuss@lists.lustre.org</a>><br>
Subject: Re: [Lustre-discuss] Performance dropoff for a nearly full<br>
Lustre file system<br>
Message-ID: <<a href="mailto:54B6D7B1.7020403@umich.edu">54B6D7B1.7020403@umich.edu</a>><br>
Content-Type: text/plain; charset="windows-1252"; Format="flowed"<br>
<br>
In my memory, it is not recommended to run Lustre more than 90% full.<br>
<br>
bob<br>
<br>
On 1/14/2015 2:43 PM, Mike Selway wrote:<br>
><br>
> Hello,<br>
><br>
> I?m looking for experiences for what has been observed<br>
> to happen (performance drop offs, severity of drops, partial/full<br>
> failures, ?) when an operational Lustre File System has been almost<br>
> ?filled?? percentages of interest are in the range from say 80% to<br>
> 99%. Multiple responses appreciated.<br>
><br>
> Also, comments from anyone who has implemented a Robin Hood approach,<br>
> about how they worked to avoid performance drop offs of a ?near full?<br>
> file system by ?archiving and releasing data blocks? to<br>
> auto-reconstruct continuous data areas.<br>
><br>
> Thanks!<br>
> Mike<br>
><br>
> *Mike Selway****|** Sr. Storage Architect (TAS) | Cray Inc.*<br>
><br>
> Work<a href="tel:%2B1-301-332-4116" value="+13013324116">+1-301-332-4116</a> | <a href="mailto:mselway@cray.com">mselway@cray.com</a><br>
><br>
> 146 Castlemaine Ct, Castle Rock, CO 80104| Check out Tiered<br>
> Adaptive Storage (TAS)!<br>
> <<a href="http://www.cray.com/Products/Storage/Tiered-Adaptive-Storage.aspx" target="_blank">http://www.cray.com/Products/Storage/Tiered-Adaptive-Storage.aspx</a>><br>
><br>
> cid:image001.png@01CF36E5.85AF42A0 <<a href="http://www.cray.com/" target="_blank">http://www.cray.com/</a>><br>
><br>
><br>
><br>
> _______________________________________________<br>
> Lustre-discuss mailing list<br>
> <a href="mailto:Lustre-discuss@lists.lustre.org">Lustre-discuss@lists.lustre.org</a><br>
> <a href="http://lists.lustre.org/mailman/listinfo/lustre-discuss" target="_blank">http://lists.lustre.org/mailman/listinfo/lustre-discuss</a><br>
<br>
-------------- next part --------------<br>
An HTML attachment was scrubbed...<br>
URL: <<a href="http://lists.lustre.org/pipermail/lustre-discuss/attachments/20150114/8618f787/attachment.html" target="_blank">http://lists.lustre.org/pipermail/lustre-discuss/attachments/20150114/8618f787/attachment.html</a>><br>
-------------- next part --------------<br>
A non-text attachment was scrubbed...<br>
Name: not available<br>
Type: image/png<br>
Size: 5290 bytes<br>
Desc: not available<br>
URL: <<a href="http://lists.lustre.org/pipermail/lustre-discuss/attachments/20150114/8618f787/attachment.png" target="_blank">http://lists.lustre.org/pipermail/lustre-discuss/attachments/20150114/8618f787/attachment.png</a>><br>
-------------- next part --------------<br>
A non-text attachment was scrubbed...<br>
Name: not available<br>
Type: image/jpeg<br>
Size: 2329 bytes<br>
Desc: not available<br>
URL: <<a href="http://lists.lustre.org/pipermail/lustre-discuss/attachments/20150114/8618f787/attachment.jpe" target="_blank">http://lists.lustre.org/pipermail/lustre-discuss/attachments/20150114/8618f787/attachment.jpe</a>><br>
<br>
------------------------------<br>
<br>
_______________________________________________<br>
Lustre-discuss mailing list<br>
<a href="mailto:Lustre-discuss@lists.lustre.org">Lustre-discuss@lists.lustre.org</a><br>
<a href="http://lists.lustre.org/mailman/listinfo/lustre-discuss" target="_blank">http://lists.lustre.org/mailman/listinfo/lustre-discuss</a><br>
<br>
<br>
End of Lustre-discuss Digest, Vol 106, Issue 18<br>
***********************************************<br>
</blockquote></div><br></div>