<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
Many thanks, Simon and Andreas. I agree the behaviour makes sense. Apologies for not checking the documentation.</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
<br>
</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
Kind regards,</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
Frank</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
<br>
</div>
<div id="Signature">
<div style="font-family: Consolas, Courier, monospace; font-size: 9pt; color: rgb(0, 0, 0);">
<span style="line-height: normal;">-- </span></div>
<div style="font-family: Consolas, Courier, monospace; font-size: 9pt;"><span style="line-height: normal;">Dr. Frank Otto<br>
</span>Senior Research Infrastructure Developer<br>
<span style="line-height: normal;">UCL Centre for Advanced Research Computing<br>
Tel: 020 7679 1506</span></div>
</div>
<div id="appendonsend"></div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> lustre-discuss <lustre-discuss-bounces@lists.lustre.org> on behalf of Andreas Dilger via lustre-discuss <lustre-discuss@lists.lustre.org><br>
<b>Sent:</b> 29 April 2024 19:29<br>
<b>To:</b> Simon Guilbault <simon.guilbault@calculquebec.ca><br>
<b>Cc:</b> lustre-discuss@lists.lustre.org <lustre-discuss@lists.lustre.org><br>
<b>Subject:</b> Re: [lustre-discuss] [EXTERNAL] [BULK] Files created in append mode don't obey directory default stripe count</font>
<div> </div>
</div>
<div class="" style="word-wrap:break-word; line-break:after-white-space">
<div style="background-color:#FFEFD5; padding:1px">
<p style="font-size:11pt; line-height:10pt; font-family:'Arial','Helvetica',sans-serif">
⚠ Caution: External sender</p>
</div>
<br>
<div>Simon is exactly correct. This is expected behavior for files opened with O_APPEND, at least until LU-12738 is implemented. Since O_APPEND writes are (by definition) entirely serialized, having multiple stripes on such files is mostly useless and just
adds overhead.
<div class=""><br class="">
</div>
<div class="">Feel free to read <a href="https://jira.whamcloud.com/browse/LU-9341" originalsrc="https://jira.whamcloud.com/browse/LU-9341" shash="NFz1kM++ZtE5MxkUU1z8XzFrMqxC1B5IMdMqxtn7XKhJn1PYpwy/7bzRXNCsD4YIr6W2inHZwczjm0X9tAmobJn5ZzU5trFa7jwHo52Vqygdiy3yLEGKRpCasNzoqVUrUicLIZvKWrQWdFEd8Y1NuOoAuVMHPsdl6Q92b/3UeOs=" class="">https://jira.whamcloud.com/browse/LU-9341</a>
for the very lengthy saga on the history of this behavior.</div>
<div class=""><br class="">
</div>
<div class="">Cheers, Andreas<br class="">
<div><br class="">
<blockquote type="cite" class="">
<div class="">On Apr 29, 2024, at 10:42, Simon Guilbault <<a href="mailto:simon.guilbault@calculquebec.ca" class="">simon.guilbault@calculquebec.ca</a>> wrote:</div>
<br class="x_Apple-interchange-newline">
<div class="">
<div dir="ltr" class="">This is the expected behaviour. In the original implementation of PFL, when a file was open in append mode, the lock from 0 to EOF was initializing all stripes of the PFL file. We have a PFL layout on our system with 1 stripe up to 1
GB, then it increased to 4 and then 32 stripes when the file was getting very large. This was a problem with software that was creating 4kb log files (like slurm.out) because they were creating files with > 32 stripes because of the append mode. This was patched
a few releases ago, that behaviour can be changed, but I would recommend keeping 1 stripe for files that are using append mode.<br class="">
<br class="">
From the manual:<br class="">
O_APPEND mode. When files are opened for append, they instantiate all uninitialized components expressed in the layout. Typically, log files are opened for append, and complex layouts can be inefficient.<br class="">
Note<br class="">
The mdd.*.append_stripe_count and mdd.*.append_pool options can be used to specify special default striping for files created with O_APPEND.<br class="">
</div>
<br class="">
<div class="x_gmail_quote">
<div dir="ltr" class="x_gmail_attr">On Mon, Apr 29, 2024 at 11:21 AM Vicker, Darby J. (JSC-EG111)[Jacobs Technology, Inc.] via lustre-discuss <<a href="mailto:lustre-discuss@lists.lustre.org" class="">lustre-discuss@lists.lustre.org</a>> wrote:<br class="">
</div>
<blockquote class="x_gmail_quote" style="margin:0px 0px 0px 0.8ex; border-left:1px solid rgb(204,204,204); padding-left:1ex">
<div class="x_msg-6154978525767147563">
<div lang="EN-US" class="" style="">
<div class="x_m_7725801462833266643WordSection1">
<p class="x_MsoNormal"><span class="" style="font-size:11pt">Wow, I would say that is definitely not expected. I can recreate this on both of our LFS’s. One is community lustre 2.14, the other is a DDN Exascalar. Shown below is our community lustre but we
also have a 3-segment PFL on our Exascalar and the behavor is the same there. <u class="">
</u><u class=""></u></span></p>
<p class="x_MsoNormal"><span class="" style="font-size:11pt"><u class=""></u> <u class=""></u></span></p>
<p class="x_MsoNormal"><span class="" style="font-size:11pt">$ echo > aaa<u class=""></u><u class=""></u></span></p>
<p class="x_MsoNormal"><span class="" style="font-size:11pt">$ echo >> bbb<u class=""></u><u class=""></u></span></p>
<p class="x_MsoNormal"><span class="" style="font-size:11pt">$ lfs getstripe aaa bbb<u class=""></u><u class=""></u></span></p>
<p class="x_MsoNormal"><span class="" style="font-size:11pt">aaa<u class=""></u><u class=""></u></span></p>
<p class="x_MsoNormal"><span class="" style="font-size:11pt"> lcm_layout_gen: 3<u class=""></u><u class=""></u></span></p>
<p class="x_MsoNormal"><span class="" style="font-size:11pt"> lcm_mirror_count: 1<u class=""></u><u class=""></u></span></p>
<p class="x_MsoNormal"><span class="" style="font-size:11pt"> lcm_entry_count: 3<u class=""></u><u class=""></u></span></p>
<p class="x_MsoNormal"><span class="" style="font-size:11pt"> lcme_id: 1<u class=""></u><u class=""></u></span></p>
<p class="x_MsoNormal"><span class="" style="font-size:11pt"> lcme_mirror_id: 0<u class=""></u><u class=""></u></span></p>
<p class="x_MsoNormal"><span class="" style="font-size:11pt"> lcme_flags: init<u class=""></u><u class=""></u></span></p>
<p class="x_MsoNormal"><span class="" style="font-size:11pt"> lcme_extent.e_start: 0<u class=""></u><u class=""></u></span></p>
<p class="x_MsoNormal"><span class="" style="font-size:11pt"> lcme_extent.e_end: 33554432<u class=""></u><u class=""></u></span></p>
<p class="x_MsoNormal"><span class="" style="font-size:11pt"> lmm_stripe_count: 1<u class=""></u><u class=""></u></span></p>
<p class="x_MsoNormal"><span class="" style="font-size:11pt"> lmm_stripe_size: 4194304<u class=""></u><u class=""></u></span></p>
<p class="x_MsoNormal"><span class="" style="font-size:11pt"> lmm_pattern: raid0<u class=""></u><u class=""></u></span></p>
<p class="x_MsoNormal"><span class="" style="font-size:11pt"> lmm_layout_gen: 0<u class=""></u><u class=""></u></span></p>
<p class="x_MsoNormal"><span class="" style="font-size:11pt"> lmm_stripe_offset: 6<u class=""></u><u class=""></u></span></p>
<p class="x_MsoNormal"><span class="" style="font-size:11pt"> lmm_objects:<u class=""></u><u class=""></u></span></p>
<p class="x_MsoNormal"><span class="" style="font-size:11pt"> - 0: { l_ost_idx: 6, l_fid: [0x100060000:0xace8112:0x0] }<u class=""></u><u class=""></u></span></p>
<p class="x_MsoNormal"><span class="" style="font-size:11pt"><u class=""></u> <u class=""></u></span></p>
<p class="x_MsoNormal"><span class="" style="font-size:11pt"> lcme_id: 2<u class=""></u><u class=""></u></span></p>
<p class="x_MsoNormal"><span class="" style="font-size:11pt"> lcme_mirror_id: 0<u class=""></u><u class=""></u></span></p>
<p class="x_MsoNormal"><span class="" style="font-size:11pt"> lcme_flags: 0<u class=""></u><u class=""></u></span></p>
<p class="x_MsoNormal"><span class="" style="font-size:11pt"> lcme_extent.e_start: 33554432<u class=""></u><u class=""></u></span></p>
<p class="x_MsoNormal"><span class="" style="font-size:11pt"> lcme_extent.e_end: 10737418240<u class=""></u><u class=""></u></span></p>
<p class="x_MsoNormal"><span class="" style="font-size:11pt"> lmm_stripe_count: 4<u class=""></u><u class=""></u></span></p>
<p class="x_MsoNormal"><span class="" style="font-size:11pt"> lmm_stripe_size: 4194304<u class=""></u><u class=""></u></span></p>
<p class="x_MsoNormal"><span class="" style="font-size:11pt"> lmm_pattern: raid0<u class=""></u><u class=""></u></span></p>
<p class="x_MsoNormal"><span class="" style="font-size:11pt"> lmm_layout_gen: 0<u class=""></u><u class=""></u></span></p>
<p class="x_MsoNormal"><span class="" style="font-size:11pt"> lmm_stripe_offset: -1<u class=""></u><u class=""></u></span></p>
<p class="x_MsoNormal"><span class="" style="font-size:11pt"><u class=""></u> <u class=""></u></span></p>
<p class="x_MsoNormal"><span class="" style="font-size:11pt"> lcme_id: 3<u class=""></u><u class=""></u></span></p>
<p class="x_MsoNormal"><span class="" style="font-size:11pt"> lcme_mirror_id: 0<u class=""></u><u class=""></u></span></p>
<p class="x_MsoNormal"><span class="" style="font-size:11pt"> lcme_flags: 0<u class=""></u><u class=""></u></span></p>
<p class="x_MsoNormal"><span class="" style="font-size:11pt"> lcme_extent.e_start: 10737418240<u class=""></u><u class=""></u></span></p>
<p class="x_MsoNormal"><span class="" style="font-size:11pt"> lcme_extent.e_end: EOF<u class=""></u><u class=""></u></span></p>
<p class="x_MsoNormal"><span class="" style="font-size:11pt"> lmm_stripe_count: 8<u class=""></u><u class=""></u></span></p>
<p class="x_MsoNormal"><span class="" style="font-size:11pt"> lmm_stripe_size: 4194304<u class=""></u><u class=""></u></span></p>
<p class="x_MsoNormal"><span class="" style="font-size:11pt"> lmm_pattern: raid0<u class=""></u><u class=""></u></span></p>
<p class="x_MsoNormal"><span class="" style="font-size:11pt"> lmm_layout_gen: 0<u class=""></u><u class=""></u></span></p>
<p class="x_MsoNormal"><span class="" style="font-size:11pt"> lmm_stripe_offset: -1<u class=""></u><u class=""></u></span></p>
<p class="x_MsoNormal"><span class="" style="font-size:11pt"><u class=""></u> <u class=""></u></span></p>
<p class="x_MsoNormal"><span class="" style="font-size:11pt">bbb<u class=""></u><u class=""></u></span></p>
<p class="x_MsoNormal"><span class="" style="font-size:11pt">lmm_stripe_count: 1<u class=""></u><u class=""></u></span></p>
<p class="x_MsoNormal"><span class="" style="font-size:11pt">lmm_stripe_size: 2097152<u class=""></u><u class=""></u></span></p>
<p class="x_MsoNormal"><span class="" style="font-size:11pt">lmm_pattern: raid0<u class=""></u><u class=""></u></span></p>
<p class="x_MsoNormal"><span class="" style="font-size:11pt">lmm_layout_gen: 0<u class=""></u><u class=""></u></span></p>
<p class="x_MsoNormal"><span class="" style="font-size:11pt">lmm_stripe_offset: 3<u class=""></u><u class=""></u></span></p>
<p class="x_MsoNormal"><span class="" style="font-size:11pt"> obdidx objid objid group<u class=""></u><u class=""></u></span></p>
<p class="x_MsoNormal"><span class="" style="font-size:11pt"> 3 179773949 0xab721fd 0<u class=""></u><u class=""></u></span></p>
<p class="x_MsoNormal"><span class="" style="font-size:11pt"><u class=""></u> <u class=""></u></span></p>
<p class="x_MsoNormal"><span class="" style="font-size:11pt"><u class=""></u> <u class=""></u></span></p>
<div id="x_m_7725801462833266643mail-editor-reference-message-container" class="">
<div class="">
<div class="" style="border-right:none; border-bottom:none; border-left:none; border-top:1pt solid rgb(181,196,223); padding:3pt 0in 0in">
<p class="x_MsoNormal" style="margin-bottom:12pt"><b class=""><span class="" style="font-size:12pt">From:
</span></b><span class="" style="font-size:12pt">lustre-discuss <<a href="mailto:lustre-discuss-bounces@lists.lustre.org" target="_blank" class="">lustre-discuss-bounces@lists.lustre.org</a>> on behalf of Otto, Frank via lustre-discuss <<a href="mailto:lustre-discuss@lists.lustre.org" target="_blank" class="">lustre-discuss@lists.lustre.org</a>><br class="">
<b class="">Date: </b>Monday, April 29, 2024 at 8:33 AM<br class="">
<b class="">To: </b><a href="mailto:lustre-discuss@lists.lustre.org" target="_blank" class="">lustre-discuss@lists.lustre.org</a> <<a href="mailto:lustre-discuss@lists.lustre.org" target="_blank" class="">lustre-discuss@lists.lustre.org</a>><br class="">
<b class="">Subject: </b>[EXTERNAL] [BULK] [lustre-discuss] Files created in append mode don't obey directory default stripe count<u class=""></u><u class=""></u></span></p>
</div>
<table border="1" cellspacing="0" cellpadding="0" align="left" class="" style="border:1.5pt solid black">
<tbody class="">
<tr class="">
<td width="100%" class="" style="width:100%; border:none; background:rgb(255,235,156); padding:3.75pt">
<p class="x_MsoNormal"><b class=""><span class="" style="">CAUTION:</span></b><span class="" style="font-size:12pt">
</span><span class="" style="">This email originated from outside of NASA. Please take care when clicking links or opening attachments. Use the "Report Message" button to report suspicious messages to the NASA SOC.</span><span class="" style="font-size:12pt">
</span><span class="" style="font-size:12pt"><u class=""></u><u class=""></u></span></p>
</td>
</tr>
</tbody>
</table>
<p class="x_MsoNormal" style="margin-bottom:12pt"><span class="" style="font-size:12pt"><br class="">
<br class="">
<u class=""></u><u class=""></u></span></p>
<div class="">
<div class="">
<p class="x_MsoNormal"><span class="" style="font-size:11pt">See subject. Is it a known issue? Is it expected? Easy to reproduce:<u class=""></u><u class=""></u></span></p>
</div>
<div class="">
<p class="x_MsoNormal"><span class="" style="font-size:11pt"><u class=""></u> <u class=""></u></span></p>
</div>
<div class="">
<p class="x_MsoNormal"><span class="" style="font-size:11pt"><u class=""></u> <u class=""></u></span></p>
</div>
<div class="">
<p class="x_MsoNormal"><span class="" style="font-size:11pt"># lfs getstripe .<u class=""></u><u class=""></u></span></p>
</div>
<div class="">
<p class="x_MsoNormal"><span class="" style="font-size:11pt">.<u class=""></u><u class=""></u></span></p>
</div>
<div class="">
<p class="x_MsoNormal"><span class="" style="font-size:11pt">stripe_count: 4 stripe_size: 1048576 pattern: raid0 stripe_offset: -1<u class=""></u><u class=""></u></span></p>
</div>
<div class="">
<p class="x_MsoNormal"><span class="" style="font-size:11pt"><u class=""></u> <u class=""></u></span></p>
</div>
<div class="">
<p class="x_MsoNormal"><span class="" style="font-size:11pt"># echo > aaa<u class=""></u><u class=""></u></span></p>
</div>
<div class="">
<p class="x_MsoNormal"><span class="" style="font-size:11pt"># echo >> bbb<u class=""></u><u class=""></u></span></p>
</div>
<div class="">
<p class="x_MsoNormal"><span class="" style="font-size:11pt"># lfs getstripe .<u class=""></u><u class=""></u></span></p>
</div>
<div class="">
<p class="x_MsoNormal"><span class="" style="font-size:11pt">.<u class=""></u><u class=""></u></span></p>
</div>
<div class="">
<p class="x_MsoNormal"><span class="" style="font-size:11pt">stripe_count: 4 stripe_size: 1048576 pattern: raid0 stripe_offset: -1<u class=""></u><u class=""></u></span></p>
</div>
<div class="">
<p class="x_MsoNormal"><span class="" style="font-size:11pt"><u class=""></u> <u class=""></u></span></p>
</div>
<div class="">
<p class="x_MsoNormal"><span class="" style="font-size:11pt">./aaa<u class=""></u><u class=""></u></span></p>
</div>
<div class="">
<p class="x_MsoNormal"><span class="" style="font-size:11pt">lmm_stripe_count: 4<u class=""></u><u class=""></u></span></p>
</div>
<div class="">
<p class="x_MsoNormal"><span class="" style="font-size:11pt">lmm_stripe_size: 1048576<u class=""></u><u class=""></u></span></p>
</div>
<div class="">
<p class="x_MsoNormal"><span class="" style="font-size:11pt">lmm_pattern: raid0<u class=""></u><u class=""></u></span></p>
</div>
<div class="">
<p class="x_MsoNormal"><span class="" style="font-size:11pt">lmm_layout_gen: 0<u class=""></u><u class=""></u></span></p>
</div>
<div class="">
<p class="x_MsoNormal"><span class="" style="font-size:11pt">lmm_stripe_offset: 0<u class=""></u><u class=""></u></span></p>
</div>
<div class="">
<p class="x_MsoNormal"><span class="" style="font-size:11pt"> obdidx objid objid group<u class=""></u><u class=""></u></span></p>
</div>
<div class="">
<p class="x_MsoNormal"><span class="" style="font-size:11pt"> 0 2830 0xb0e 0<u class=""></u><u class=""></u></span></p>
</div>
<div class="">
<p class="x_MsoNormal"><span class="" style="font-size:11pt"> 1 2894 0xb4e 0<u class=""></u><u class=""></u></span></p>
</div>
<div class="">
<p class="x_MsoNormal"><span class="" style="font-size:11pt"> 2 2831 0xb0f 0<u class=""></u><u class=""></u></span></p>
</div>
<div class="">
<p class="x_MsoNormal"><span class="" style="font-size:11pt"> 3 2895 0xb4f 0<u class=""></u><u class=""></u></span></p>
</div>
<div class="">
<p class="x_MsoNormal"><span class="" style="font-size:11pt"><u class=""></u> <u class=""></u></span></p>
</div>
<div class="">
<p class="x_MsoNormal"><span class="" style="font-size:11pt">./bbb<u class=""></u><u class=""></u></span></p>
</div>
<div class="">
<p class="x_MsoNormal"><span class="" style="font-size:11pt">lmm_stripe_count: 1<u class=""></u><u class=""></u></span></p>
</div>
<div class="">
<p class="x_MsoNormal"><span class="" style="font-size:11pt">lmm_stripe_size: 1048576<u class=""></u><u class=""></u></span></p>
</div>
<div class="">
<p class="x_MsoNormal"><span class="" style="font-size:11pt">lmm_pattern: raid0<u class=""></u><u class=""></u></span></p>
</div>
<div class="">
<p class="x_MsoNormal"><span class="" style="font-size:11pt">lmm_layout_gen: 0<u class=""></u><u class=""></u></span></p>
</div>
<div class="">
<p class="x_MsoNormal"><span class="" style="font-size:11pt">lmm_stripe_offset: 4<u class=""></u><u class=""></u></span></p>
</div>
<div class="">
<p class="x_MsoNormal"><span class="" style="font-size:11pt"> obdidx objid objid group<u class=""></u><u class=""></u></span></p>
</div>
<div class="">
<p class="x_MsoNormal"><span class="" style="font-size:11pt"> 4 2831 0xb0f 0<u class=""></u><u class=""></u></span></p>
</div>
<div class="">
<p class="x_MsoNormal"><span class="" style="font-size:11pt"><u class=""></u> <u class=""></u></span></p>
</div>
<div class="">
<p class="x_MsoNormal"><span class="" style="font-size:11pt"><br class="">
<br class="">
As you see, file "bbb" is created with stripe count 1 instead of 4.<u class=""></u><u class=""></u></span></p>
</div>
<div class="">
<p class="x_MsoNormal"><span class="" style="font-size:11pt">Observed in Lustre 2.12.x and Lustre 2.15.4.<u class=""></u><u class=""></u></span></p>
</div>
<div class="">
<p class="x_MsoNormal"><span class="" style="font-size:11pt"><u class=""></u> <u class=""></u></span></p>
</div>
<div class="">
<p class="x_MsoNormal"><span class="" style="font-size:11pt">Thanks,<u class=""></u><u class=""></u></span></p>
</div>
<div class="">
<p class="x_MsoNormal"><span class="" style="font-size:11pt">Frank<u class=""></u><u class=""></u></span></p>
</div>
<div class="">
<p class="x_MsoNormal"><span class="" style="font-size:11pt"><u class=""></u> <u class=""></u></span></p>
</div>
<div id="x_m_7725801462833266643Signature" class="">
<div class="">
<p class="x_MsoNormal"><span class="" style="font-size:9pt; font-family:Consolas">-- <u class=""></u><u class=""></u></span></p>
</div>
<div class="">
<p class="x_MsoNormal"><span class="" style="font-size:9pt; font-family:Consolas">Dr. Frank Otto<br class="">
Senior Research Infrastructure Developer<br class="">
UCL Centre for Advanced Research Computing<br class="">
Tel: 020 7679 1506<u class=""></u><u class=""></u></span></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
_______________________________________________<br class="">
lustre-discuss mailing list<br class="">
<a href="mailto:lustre-discuss@lists.lustre.org" target="_blank" class="">lustre-discuss@lists.lustre.org</a><br class="">
<a href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org" originalsrc="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org" shash="UNzujYZjepnFbt/iTkWEIUO38Gtl/Fa8s+uynxO6g2CsWS/rbdRAdYNyP8ynX/zDlKAqMZiIfOAY+3p8tGtl0LjowCHXsnGgmaSVvXFoKV68ucakac/DC1ZQotS41IW5tuKuhJiPLYFcK0SuopG2UVxVeZPrKKV/mblEVlXcUqU=" rel="noreferrer" target="_blank" class="">http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org</a><br class="">
</div>
</blockquote>
</div>
_______________________________________________<br class="">
lustre-discuss mailing list<br class="">
<a href="mailto:lustre-discuss@lists.lustre.org" class="">lustre-discuss@lists.lustre.org</a><br class="">
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org<br class="">
</div>
</blockquote>
</div>
<br class="">
<div class="">
<div dir="auto" class="" style="color:rgb(0,0,0); letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none; word-wrap:break-word; line-break:after-white-space">
<div dir="auto" class="" style="color:rgb(0,0,0); letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none; word-wrap:break-word; line-break:after-white-space">
<div dir="auto" class="" style="color:rgb(0,0,0); letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none; word-wrap:break-word; line-break:after-white-space">
<div dir="auto" class="" style="color:rgb(0,0,0); letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none; word-wrap:break-word; line-break:after-white-space">
<div dir="auto" class="" style="color:rgb(0,0,0); letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none; word-wrap:break-word; line-break:after-white-space">
<div dir="auto" class="" style="color:rgb(0,0,0); letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none; word-wrap:break-word; line-break:after-white-space">
<div>Cheers, Andreas</div>
<div>--</div>
<div>Andreas Dilger</div>
<div>Lustre Principal Architect</div>
<div>Whamcloud</div>
<div><br class="">
</div>
<div><br class="">
</div>
<div><br class="">
</div>
</div>
</div>
</div>
</div>
</div>
<br class="x_Apple-interchange-newline">
</div>
<br class="x_Apple-interchange-newline">
<br class="x_Apple-interchange-newline">
</div>
<br class="">
</div>
</div>
</div>
</body>
</html>