<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<span style="font-size: 14px;" class="">Patrick,</span>
<div class=""><span style="font-size: 14px;" class="">once upon a time there was "file join" functionality in Lustre that was ancient and complex, and was finally removed in 2009. There are still a few remnants of this like "MDS_OPEN_JOIN_FILE" and "LOV_MAGIC_JOIN_V1"
defined, but unused. That functionality long predated composite file layouts (PFL, FLR), and used an external llog file *per file* to declare a series of other files that described the layout. It was extremely fragile and complex and thankfully never got
into widespread usage.</span></div>
<div class=""><span style="font-size: 14px;" class=""><br class="">
</span></div>
<div class=""><span style="font-size: 14px;" class="">I think with the advent of composite file layout that it should be _possible_ to implement this kind of functionality purely with layout changes, similar to "lfs migrate" doing layout swap, or "lfs mirror
extend" merging the layout of a victim file into another file to create a mirror.</span></div>
<div class=""><span style="font-size: 14px;" class=""><br class="">
</span></div>
<div class=""><span style="font-size: 14px;" class="">My expectation is that "join" of two files would be handled at the file EOF and *not* at the layout boundary. Based on the original description from Sven, I'd think that small gaps in the file (e.g. 4KB
for page alignment, 64KB for minimum layout alignment, or 1MB for stripe alignment) would be OK, but tens or hundreds of MB holes would be inefficient for processing.</span></div>
<div class=""><span style="font-size: 14px;" class=""><br class="">
</span></div>
<div class=""><span style="font-size: 14px;" class="">My guess, based on similar requests I've seen previously, and Sven's email address, is that this relates to merging video streams from different files into a single file?</span></div>
<div class=""><span style="font-size: 14px;" class=""><br class="">
</span></div>
<div class=""><span style="font-size: 14px;" class="">Sven,</span></div>
<div class=""><span style="font-size: 14px;" class="">while I think it is possible to implement this in Lustre, I'd have to ask what requirements are driving your request? Is this just something you want to test, or is there some real-world usage demand for
this (e.g. specific application workload, usage in some popular library, etc)?</span></div>
<div class=""><span style="font-size: 14px;" class=""><br class="">
</span></div>
<div class=""><span style="font-size: 14px;" class="">It seems possible to do this with layout manipulation similar to "lfs mirror extend -f" (i.e. a kind of "super file append" mechanism) but would be similarly destructive to the "victim" files appended to
the original one, and would definitely not be something that could be done while the "original" file was actively in use. Essentially, instead of "lfs mirror extend" just appending the victim layout to the existing file, it would need to also modify the original
layout to truncate the layout at EOF, then offset the extent ranges in the victim layout by the current file size (rounded up to at least 64KB multiples, but preferably 1MB multiples to maintain RAID alignment).</span></div>
<div class=""><br class="">
</div>
<div class=""><span style="font-size: 14px;" class="">Is this something that you would be willing to work on with guidance for the implementation details, or a feature request that you hope someone else will implement?</span></div>
<div class=""><span style="font-size: 14px;" class=""><br class="">
</span></div>
<div class=""><span style="font-size: 14px;" class="">Cheers, Andreas<br class="">
</span>
<div><br class="">
<blockquote type="cite" class="">
<div class="">On Mar 29, 2023, at 07:41, Patrick Farrell via lustre-discuss <<a href="mailto:lustre-discuss@lists.lustre.org" class="">lustre-discuss@lists.lustre.org</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div class="elementToProof" style="font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; background-color: rgb(255, 255, 255);">
Sven,</div>
<div class="elementToProof" style="font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; background-color: rgb(255, 255, 255);">
<br class="">
</div>
<div class="elementToProof" style="font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; background-color: rgb(255, 255, 255);">
The "combining layouts without any data movement" part isn't currently possible. It's probably possible in theory, but it's never been implemented. (I'm curious what your use case is?)</div>
<div class="elementToProof" style="font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; background-color: rgb(255, 255, 255);">
<br class="">
</div>
<div class="elementToProof" style="font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; background-color: rgb(255, 255, 255);">
Even allowing for data movement, there's no tool to do this for you. Depending what you mean by combining, it's possible to do this with Linux tools (see the end of my note), but you're going to have data copying.</div>
<div class="elementToProof" style="font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; background-color: rgb(255, 255, 255);">
<br class="">
</div>
<div class="elementToProof" style="font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; background-color: rgb(255, 255, 255);">
It's a bit of an odd requirement, with some inherent questions - For example, file layouts generally go to infinity, because if they don't, you will get IO errors when you 'run off the end', ie, go past the defined layout, so the last component is usually defined
to go to infinity.</div>
<div class="elementToProof" style="font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; background-color: rgb(255, 255, 255);">
<br class="">
</div>
<div class="elementToProof" style="font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; background-color: rgb(255, 255, 255);">
That poses obvious questions when combining files.</div>
<div class="elementToProof" style="font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; background-color: rgb(255, 255, 255);">
<br class="">
</div>
<div class="elementToProof" style="font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; background-color: rgb(255, 255, 255);">
If you're looking to combine files with layouts that do not go to infinity, then it's at least straightforward to see how you'd concatenate them. But presumably the data in each file doesn't go to the very end of the layout? So do you want the empty parts
of the layout included?</div>
<div class="elementToProof" style="font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; background-color: rgb(255, 255, 255);">
<br class="">
</div>
<div class="elementToProof" style="font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; background-color: rgb(255, 255, 255);">
Say file 1 is 10 MiB in size but the layout goes to 20 MiB (again, layouts normally should go to infinity) and file 2 is also 10 MiB in size but the layout goes to, say, 15 MiB. Should the result look like this?</div>
<div class="elementToProof" style="font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; background-color: rgb(255, 255, 255);">
<br class="">
</div>
<div class="elementToProof" style="font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; background-color: rgb(255, 255, 255);">
Layout: 1 1 1 1 1 1 1 ... 20 MiB 2 2 2 2 2 2 .... 35 MiB<br class="">
<br class="">
</div>
<div class="elementToProof" style="font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; background-color: rgb(255, 255, 255);">
With data from 0-10 MiB and 20 - 30 MiB.</div>
<div class="elementToProof" style="font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; background-color: rgb(255, 255, 255);">
<br class="">
</div>
<div class="elementToProof" style="font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; background-color: rgb(255, 255, 255);">
That's something you'd have to write a tool for, so it could write the data at your specified offset for putting in the second file (and third, etc...). You could also do something like:<br class="">
<br class="">
lfs setstripe [your layout] combined file; cat file 1 > combined file; truncate [combined file] 20 MiB (the end of the file 1 layout); cat file 2 > combined_file", etc.</div>
<div class="elementToProof" style="font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; background-color: rgb(255, 255, 255);">
<br class="">
</div>
<div class="elementToProof" style="font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; background-color: rgb(255, 255, 255);">
So, you definitely can't avoid data copying here. But that's how you could do it with simple Linux tools (which you could probably have drawn up yourself :)).</div>
<div class="elementToProof" style="font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; background-color: rgb(255, 255, 255);">
<br class="">
</div>
<div class="elementToProof" style="font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; background-color: rgb(255, 255, 255);">
-Patrick</div>
<div class="elementToProof" style="font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; background-color: rgb(255, 255, 255);">
</div>
<div class="elementToProof" style="font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; background-color: rgb(255, 255, 255);">
</div>
<div class="elementToProof" style="font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; background-color: rgb(255, 255, 255);">
</div>
<div id="appendonsend" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">
</div>
<div style="font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;" class="">
<br class="">
</div>
<hr tabindex="-1" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; display: inline-block; width: 1275.953125px;" class="">
<span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class=""></span>
<div id="divRplyFwdMsg" dir="ltr" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">
<font face="Calibri, sans-serif" style="font-size: 11pt;" class=""><b class="">From:</b><span class="Apple-converted-space"> </span>lustre-discuss <<a href="mailto:lustre-discuss-bounces@lists.lustre.org" class="">lustre-discuss-bounces@lists.lustre.org</a>>
on behalf of Sven Willner <<a href="mailto:sven.willner@mpimet.mpg.de" class="">sven.willner@mpimet.mpg.de</a>><br class="">
<b class="">Sent:</b><span class="Apple-converted-space"> </span>Wednesday, March 29, 2023 7:58 AM<br class="">
<b class="">To:</b><span class="Apple-converted-space"> </span><a href="mailto:lustre-discuss@lists.lustre.org" class="">lustre-discuss@lists.lustre.org</a> <<a href="mailto:lustre-discuss@lists.lustre.org" class="">lustre-discuss@lists.lustre.org</a>><br class="">
<b class="">Subject:</b><span class="Apple-converted-space"> </span>[lustre-discuss] Joining files</font>
<div class=""> </div>
</div>
<div class="BodyFragment" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;">
<font size="2" class=""><span style="font-size: 11pt;" class="">
<div class="PlainText">[You don't often get email from <a href="mailto:sven.willner@mpimet.mpg.de" class="">
sven.willner@mpimet.mpg.de</a>. Learn why this is important at<span class="Apple-converted-space"> </span><a href="https://aka.ms/LearnAboutSenderIdentification" data-auth="NotApplicable" class="">https://aka.ms/LearnAboutSenderIdentification</a><span class="Apple-converted-space"> </span>]<br class="">
<br class="">
Dear all,<br class="">
<br class="">
I am looking for a way to join/merge/concatenate several files into one, whose layout is just the concatenation of the layouts of the respective files - ideally without any copying/moving on the data side (even if this would result in "holes" in the joined
file).<br class="">
<br class="">
I would very much appreciate any hints to tools or ideas of how to achieve such a join. As I understand that has been a `join` command for `lfs`, which is now deprecated (however, I am not sure if a use case like mine has been its purpose or why it has been
deprecated).<br class="">
<br class="">
Thanks a lot!<br class="">
Best regards,<br class="">
Sven<br class="">
<br class="">
--<br class="">
Dr. Sven Willner<br class="">
Scientific Computing Lab (SCLab)<br class="">
Max Planck Institute for Meteorology<br class="">
Bundesstraße 53, D-20146 Hamburg, Germany<br class="">
</div>
</span></font></div>
<span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">_______________________________________________</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">
<span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">lustre-discuss
mailing list</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">
<span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class=""><a href="mailto:lustre-discuss@lists.lustre.org" class="">lustre-discuss@lists.lustre.org</a></span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">
<span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class=""><a href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org" class="">http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org</a></span></div>
</blockquote>
</div>
<br class="">
<div class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div>Cheers, Andreas</div>
<div>--</div>
<div>Andreas Dilger</div>
<div>Lustre Principal Architect</div>
<div>Whamcloud</div>
<div><br class="">
</div>
<div><br class="">
</div>
<div><br class="">
</div>
</div>
</div>
</div>
</div>
</div>
<br class="Apple-interchange-newline">
</div>
<br class="Apple-interchange-newline">
<br class="Apple-interchange-newline">
</div>
<br class="">
</div>
</body>
</html>