<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
It is worthwhile to note that the proposed scenario is racy even for local filesystems, regardless of whether recovery is involved or not.
<div class=""><br class="">
</div>
<div class="">If (1) and (3) are happening on two clients at the same time:</div>
<div class="">- if (1) happens first, then (3) will fail with ENOENT ("no such file or directory") because "dir1" is gone.</div>
<div class="">- if (3) happens first, then (1) may delete "file100", but that is not the filesystem's fault, the user on cli1 asked for everything in "dir1" to be deleted.</div>
<div class="">- in some cases, "file100" may be created after (1) has passed that part of the directory traversal (it depends on how large the tree under "dir1" is), and then (3) will fail the final rmdir("dir1") with EBUSY ("directory is not empty") because
"file100" still exists there. This is the classic "TOCTOU" race ("Time of Creation/Time of Use").</div>
<div class=""><br class="">
</div>
<div class="">Cheers, Andreas<br class="">
<div><br class="">
<blockquote type="cite" class="">
<div class="">On Feb 1, 2021, at 11:18, Spitz, Cory James <<a href="mailto:cory.spitz@hpe.com" class="">cory.spitz@hpe.com</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div class="WordSection1" style="page: WordSection1; caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
> I am trying to get a 30,000 ft overview of how lustre replay/recovery works<o:p class=""></o:p></div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""> </o:p></div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
This old slide deck might be useful to you:<o:p class=""></o:p></div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<a href="https://wiki.lustre.org/images/0/00/A_Deep_Dive_into_Lustre_Recovery_Mechanisms.pdf" style="color: blue; text-decoration: underline;" class="">https://wiki.lustre.org/images/0/00/A_Deep_Dive_into_Lustre_Recovery_Mechanisms.pdf</a><o:p class=""></o:p></div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""> </o:p></div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
Granted, it may not be 100% correct any longer.<o:p class=""></o:p></div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""> </o:p></div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
-Cory<o:p class=""></o:p></div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""> </o:p></div>
<div class="">
<div style="margin: 0in 0in 0in 0.5in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
On 1/30/21, 11:08 AM, "lustre-devel" <<a href="mailto:lustre-devel-bounces@lists.lustre.org" style="color: blue; text-decoration: underline;" class="">lustre-devel-bounces@lists.lustre.org</a>> wrote:<o:p class=""></o:p></div>
</div>
<div style="margin: 0in 0in 0in 0.5in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""> </o:p></div>
<div class="">
<div style="margin: 0in 0in 0in 0.5in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
Thank you for the explanation on LLOG and changelog. With respect to the following statement :<o:p class=""></o:p></div>
<div class="">
<div style="margin: 0in 0in 0in 0.5in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""> </o:p></div>
</div>
<div class="">
<div style="margin: 0in 0in 0in 0.5in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<b class="">>> Lustre has its own mechanisms to guarantee transaction are committed to disk and handle crash. Basicly, I/O are not acknowledge to Lustre clients before the data is actually on disk. In case of server crash, the Lustre client will replay all
non-acknowledge I/Os to ensure none of them are lost.</b><o:p class=""></o:p></div>
</div>
<div class="">
<div style="margin: 0in 0in 0in 0.5in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""> </o:p></div>
</div>
<div class="">
<div style="margin: 0in 0in 0in 0.5in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
For example:<o:p class=""></o:p></div>
</div>
<div class="">
<div style="margin: 0in 0in 0in 0.5in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""> </o:p></div>
</div>
<div class="">
<div style="margin: 0in 0in 0in 0.5in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
Let us say that I have 4 clients (cli1, cli2, cli3 and cli4) and all are writing and reading data. I have 1 host with 4 disks (2 OSTs, 1 MDT, 1 MGT). <o:p class=""></o:p></div>
</div>
<div class="">
<div style="margin: 0in 0in 0in 1in; font-size: 11pt; font-family: Calibri, sans-serif; text-indent: -0.25in;" class="">
<span class="">1.<span style="font-style: normal; font-variant-caps: normal; font-weight: normal; font-stretch: normal; font-size: 7pt; line-height: normal; font-family: "Times New Roman";" class=""> <span class="Apple-converted-space"> </span></span></span>cli1
issues a directory remove (rm -rf /mnt/lustre/dir1)<o:p class=""></o:p></div>
<div style="margin: 0in 0in 0in 1in; font-size: 11pt; font-family: Calibri, sans-serif; text-indent: -0.25in;" class="">
<span class="">2.<span style="font-style: normal; font-variant-caps: normal; font-weight: normal; font-stretch: normal; font-size: 7pt; line-height: normal; font-family: "Times New Roman";" class=""> <span class="Apple-converted-space"> </span></span></span>cli1
loses connection with Lustre targets.<o:p class=""></o:p></div>
<div style="margin: 0in 0in 0in 1in; font-size: 11pt; font-family: Calibri, sans-serif; text-indent: -0.25in;" class="">
<span class="">3.<span style="font-style: normal; font-variant-caps: normal; font-weight: normal; font-stretch: normal; font-size: 7pt; line-height: normal; font-family: "Times New Roman";" class=""> <span class="Apple-converted-space"> </span></span></span>cli2
wants to now create a file under /mnt/lustre/dir1/file100 and write some data to file100<o:p class=""></o:p></div>
<div class="">
<div style="margin: 0in 0in 0in 0.5in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
All of these are happening in parallel. <o:p class=""></o:p></div>
</div>
<div class="">
<div style="margin: 0in 0in 0in 1in; font-size: 11pt; font-family: Calibri, sans-serif; text-indent: -0.25in;" class="">
<span style="font-size: 10pt; font-family: Symbol;" class=""><span class="">·<span style="font-style: normal; font-variant-caps: normal; font-weight: normal; font-stretch: normal; font-size: 7pt; line-height: normal; font-family: "Times New Roman";" class=""> <span class="Apple-converted-space"> </span></span></span></span>Does
cli2 get an error that /mnt/lustre/dir1 has been removed and it has to first issue additional I/O to create /mnt/lustre/dir1 before reissuing the I/O to write file100 ?<o:p class=""></o:p></div>
<div style="margin: 0in 0in 0in 1in; font-size: 11pt; font-family: Calibri, sans-serif; text-indent: -0.25in;" class="">
<span style="font-size: 10pt; font-family: Symbol;" class=""><span class="">·<span style="font-style: normal; font-variant-caps: normal; font-weight: normal; font-stretch: normal; font-size: 7pt; line-height: normal; font-family: "Times New Roman";" class=""> <span class="Apple-converted-space"> </span></span></span></span>If
a transaction from cli2 happens before cli1, then this would lead to data lost situation for cli2, if cli2 tries to read/write data from/to file100 after sometime.<o:p class=""></o:p></div>
<div style="margin: 0in 0in 0in 1in; font-size: 11pt; font-family: Calibri, sans-serif; text-indent: -0.25in;" class="">
<span style="font-size: 10pt; font-family: Symbol;" class=""><span class="">·<span style="font-style: normal; font-variant-caps: normal; font-weight: normal; font-stretch: normal; font-size: 7pt; line-height: normal; font-family: "Times New Roman";" class=""> <span class="Apple-converted-space"> </span></span></span></span>What
is the role of last_rcvd file in this entire picture ?<o:p class=""></o:p></div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div style="margin: 0in 0in 0in 0.5in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
I am trying to get a 30,000 ft overview of how lustre replay/recovery works.<o:p class=""></o:p></div>
</div>
</div>
</div>
<div class="">
<div style="margin: 0in 0in 0in 0.5in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""> </o:p></div>
</div>
<div class="">
<div style="margin: 0in 0in 0in 0.5in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
Thanks again and appreciate your timely response.<o:p class=""></o:p></div>
</div>
</div>
<div style="margin: 0in 0in 0in 0.5in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""> </o:p></div>
<div class="">
<div class="">
<div style="margin: 0in 0in 0in 0.5in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
On Fri, Jan 29, 2021 at 1:22 AM Degremont, Aurelien <<a href="mailto:degremoa@amazon.com" style="color: blue; text-decoration: underline;" class="">degremoa@amazon.com</a>> wrote:<o:p class=""></o:p></div>
</div>
<blockquote style="border-style: none none none solid; border-left-width: 1pt; border-left-color: rgb(204, 204, 204); padding: 0in 0in 0in 6pt; margin-left: 4.8pt; margin-right: 0in;" class="">
<div class="">
<div class="">
<div style="margin: 0in 0in 0in 0.5in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
Hi,<span lang="FR" class=""><o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0in 0.5in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span lang="FR" class=""><o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0in 0.5in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
This is not totally correct.<span lang="FR" class=""><o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0in 0.5in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span lang="FR" class=""><o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0in 0.5in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
First, LLOG is the underlying technology used to store and handle Lustre Changelogs. But LLOG is used for other Lustre mechanisms, like lustre configuration.<span lang="FR" class=""><o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0in 0.5in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
Second, Changelog is similar to an audit feature. Changelog only logs different filesystem change, mostly metadata change, but definitely not the file content change. They don't play a role at all in transaction or failure recovery. This is only an admin feature.<span lang="FR" class=""><o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0in 0.5in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span lang="FR" class=""><o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0in 0.5in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
At the end, indeed ZIL cannot be used and Lustre has its own mechanisms to guarantee transaction are committed to disk and handle crash. Basicly, I/O are not acknowledge to Lustre clients before the data is actually on disk. In case of server crash, the Lustre
client will replay all non-acknowledge I/Os to ensure none of them are lost.<span lang="FR" class=""><o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0in 0.5in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span lang="FR" class=""><o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0in 0.5in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
Changelog is not needed in your case.<span lang="FR" class=""><o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0in 0.5in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span lang="FR" class=""><o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0in 0.5in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
Aurélien<span lang="FR" class=""><o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0in 0.5in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span lang="FR" class=""><o:p class=""></o:p></span></div>
<div style="border-style: solid none none; border-top-width: 1pt; border-top-color: rgb(181, 196, 223); padding: 3pt 0in 0in;" class="">
<div style="margin: 0in 0in 0in 71.4pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<b class=""><span lang="FR" style="font-size: 12pt;" class="">De :<span class="Apple-converted-space"> </span></span></b><span lang="FR" style="font-size: 12pt;" class="">lustre-devel <<a href="mailto:lustre-devel-bounces@lists.lustre.org" target="_blank" style="color: blue; text-decoration: underline;" class="">lustre-devel-bounces@lists.lustre.org</a>>
au nom de Sudheendra Sampath <<a href="mailto:sudheendra.sampath@gmail.com" target="_blank" style="color: blue; text-decoration: underline;" class="">sudheendra.sampath@gmail.com</a>><br class="">
<b class="">Date :<span class="Apple-converted-space"> </span></b>jeudi 28 janvier 2021 à 21:43<br class="">
<b class="">À :<span class="Apple-converted-space"> </span></b>"<a href="mailto:lustre-devel@lists.lustre.org" target="_blank" style="color: blue; text-decoration: underline;" class="">lustre-devel@lists.lustre.org</a>" <<a href="mailto:lustre-devel@lists.lustre.org" target="_blank" style="color: blue; text-decoration: underline;" class="">lustre-devel@lists.lustre.org</a>><br class="">
<b class="">Objet :<span class="Apple-converted-space"> </span></b>[EXTERNAL] [lustre-devel] Lustre log question(s)</span><span lang="FR" class=""><o:p class=""></o:p></span></div>
</div>
<div class="">
<div style="margin: 0in 0in 0in 71.4pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span lang="FR" class=""> <o:p class=""></o:p></span></div>
</div>
<div class="">
<table class="MsoNormalTable" border="0" cellspacing="0" cellpadding="0" width="625" style="width: 468.8pt; margin-left: 71.4pt; border-collapse: collapse;">
<tbody class="">
<tr style="height: 15.25pt;" class="">
<td width="625" valign="top" style="width: 468.8pt; border: 1.5pt solid rgb(237, 125, 49); padding: 0in 5.4pt; height: 15.25pt;" class="">
<p class=""><strong class=""><span style="font-family: Calibri, sans-serif; background-color: rgb(255, 255, 153);" class="">CAUTION</span></strong><span style="background-color: rgb(255, 255, 153);" class="">: This email originated from outside of the organization.
Do not click links or open attachments unless you can confirm the sender and know the content is safe.</span><o:p class=""></o:p></p>
</td>
</tr>
</tbody>
</table>
</div>
<div style="margin: 0in 0in 0in 71.4pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span lang="FR" style="color: white;" class=""> </span><span lang="FR" class=""><o:p class=""></o:p></span></div>
<div class="">
<div class="">
<div style="margin: 0in 0in 0in 71.4pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span lang="FR" class="">Hi,<span class="Apple-converted-space"> </span><o:p class=""></o:p></span></div>
<div class="">
<div style="margin: 0in 0in 0in 71.4pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span lang="FR" class=""> <o:p class=""></o:p></span></div>
</div>
<div class="">
<div style="margin: 0in 0in 0in 71.4pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span lang="FR" class="">I am trying to evaluate osd-zfs based MDS and OST deployment on a 2 node setup.<o:p class=""></o:p></span></div>
</div>
<div class="">
<div style="margin: 0in 0in 0in 71.4pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span lang="FR" class=""> <o:p class=""></o:p></span></div>
</div>
<div class="">
<div style="margin: 0in 0in 0in 71.4pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span lang="FR" class="">I have the following questions about Lustre log:<o:p class=""></o:p></span></div>
</div>
<div class="">
<div style="margin: 0in 0in 0in 107.4pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span lang="FR" class="">1.</span><span lang="FR" style="font-size: 7pt; font-family: "Times New Roman", serif;" class=""> <span class="Apple-converted-space"> </span></span><b class=""><span lang="FR" class="">Is changelog and llog both the same, in the
sense are they synonymous with each other?</span></b><span lang="FR" class=""><o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0in 107.4pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span lang="FR" class="">2.</span><span lang="FR" style="font-size: 7pt; font-family: "Times New Roman", serif;" class=""> <span class="Apple-converted-space"> </span></span><span lang="FR" class="">I understand that ZIL is currently not supported in Lustre
version 2.12.2. My question is : <o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0in 143.4pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span lang="FR" class="">1.</span><span lang="FR" style="font-size: 7pt; font-family: "Times New Roman", serif;" class=""> <span class="Apple-converted-space"> </span></span><span lang="FR" class="">My understanding is that transactions (in general) need
some logging mechanism for it to work in 'all or none' scenarios. Please correct me if my understanding is incorrect. I understand that changelog has to be enabled so that filesystem changes are recorded to be replayed after a crash. <span class="Apple-converted-space"> </span><b class="">How
does Lustre transactions work if there is no intent log/changelog ?</b><o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0in 143.4pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span lang="FR" class="">2.</span><span lang="FR" style="font-size: 7pt; font-family: "Times New Roman", serif;" class=""> <span class="Apple-converted-space"> </span></span><span lang="FR" class="">Does it mean that if changelog is NOT enabled and there
is a crash, we risk losing all changes/updates to the filesystem ?<o:p class=""></o:p></span></div>
<div class="">
<div style="margin: 0in 0in 0in 71.4pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span lang="FR" class="">Appreciate your timely response and Thank you for your help.<o:p class=""></o:p></span></div>
</div>
</div>
<div class="">
<div class="">
<div style="margin: 0in 0in 0in 71.4pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span lang="FR" class=""> <o:p class=""></o:p></span></div>
</div>
<div style="margin: 0in 0in 0in 71.4pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span lang="FR" class="">--<span class="Apple-converted-space"> </span><o:p class=""></o:p></span></div>
<div class="">
<div style="margin: 0in 0in 0in 71.4pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span lang="FR" class="">Regards<br class="">
<br class="">
Sudheendra Sampath<o:p class=""></o:p></span></div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<div style="margin: 0in 0in 0in 0.5in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<br clear="all" class="">
<o:p class=""></o:p></div>
<div class="">
<div style="margin: 0in 0in 0in 0.5in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""> </o:p></div>
</div>
<div style="margin: 0in 0in 0in 0.5in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
--<span class="Apple-converted-space"> </span><o:p class=""></o:p></div>
<div class="">
<div style="margin: 0in 0in 0in 0.5in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
Regards<br class="">
<br class="">
Sudheendra Sampath<o:p class=""></o:p></div>
</div>
</div>
<span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">_______________________________________________</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">
<span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">lustre-devel
mailing list</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">
<a href="mailto:lustre-devel@lists.lustre.org" style="color: blue; text-decoration: underline; font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px;" class="">lustre-devel@lists.lustre.org</a><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">
<a href="http://lists.lustre.org/listinfo.cgi/lustre-devel-lustre.org" style="color: blue; text-decoration: underline; font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px;" class="">http://lists.lustre.org/listinfo.cgi/lustre-devel-lustre.org</a></div>
</blockquote>
</div>
<br class="">
<div class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div>Cheers, Andreas</div>
<div>--</div>
<div>Andreas Dilger</div>
<div>Principal Lustre Architect</div>
<div>Whamcloud</div>
<div><br class="">
</div>
<div><br class="">
</div>
<div><br class="">
</div>
</div>
</div>
</div>
</div>
</div>
<br class="Apple-interchange-newline">
<br class="Apple-interchange-newline">
</div>
<br class="">
</div>
</body>
</html>