<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
On Jun 15, 2022, at 06:56, John Bauer <<a href="mailto:bauerj@iodoctors.com" class="">bauerj@iodoctors.com</a>> wrote:<br class="">
<div>
<blockquote type="cite" class=""><br class="Apple-interchange-newline">
<div class="">
<div class="">
<div class=""><br class="webkit-block-placeholder">
</div>
<div class="moz-text-html" lang="x-unicode">
<p class="">Andreas,</p>
<p class="">Thanks for the info. It got me a lot farther down the road. A few comments:</p>
<p class="">1) It appears that the values returned in the poollist argument to llapi_get_poollist() are temporary. I used the values in poollist in the call to
<font face="Courier New, Courier,
monospace" class="">llapi_get_poolmembers( poollist[i],...).</font> Works great on the first call to get_poolmembers(), but subsequent calls fail and it appears the poollist has been overwritten. If I make my own
copy of the strings in poollist before calling get_poolmembers, everythings works well.</p>
</div>
</div>
</div>
</blockquote>
The pool names are stored in the passed "buffer" so if this is re-used then the old pointers will be invalid. The reason to have a pointer array to point into the buffer is to avoid the need to alloc/free a separate pointer for each pool name on each call.
<br class="">
<blockquote type="cite" class="">
<div class="">
<div class="">
<div class="moz-text-html" lang="x-unicode">
<p class="">If this is indeed the case, it should be noted.</p>
<p class="">2) There are prints to stdout or stderr resulting from calls to these llapi functions.
<br class="">
</p>
<p class="">llapi_get_poollist generates a print of:Pools from nbp17:</p>
<p class="">llapi_get_poolmembers generates a print of:Pool: nbp17.1_ssd</p>
<p class="">Seems undesirable to have llapi doing unrequested prints.</p>
</div>
</div>
</div>
</blockquote>
Yes, this is a bug - the llapi_printf() calls should be moved to the callers that care about it instead of being in the library function. In the meantime, you can hook llapi_printf() and llapi_error() with llapi_error_callback_set() and llapi_info_callback_set()
respectively, so that they do not print to the console.<br class="">
<blockquote type="cite" class="">
<div class="">
<div class="moz-text-html" lang="x-unicode">
<p class="">3) What is the buffer argument good for ( last argument for each of the functions )? It appears to be populated with the first value in the array.</p>
</div>
</div>
</blockquote>
See above. The buffer actually holds the pool names, the pool array is just pointers into the buffer.<br class="">
<blockquote type="cite" class="">
<div class="">
<div class="moz-text-html" lang="x-unicode">
<p class="">4) Here is output from my test program. There are 10 poolNames. All works great until I get to the pool that has 64 members ( which I know from other paths ). The 2nd argument to llapi_get_poolmembers() is a 'char *poolMembers[256]'. Plenty
of room, but the call fails</p>
<p class="">with a return value of -75. Why?</p>
</div>
</div>
</blockquote>
-75 = -EOVERFLOW. Probably the buffer is too small to hold all of the pool names.</div>
<div><br class="">
</div>
<div>Cheers, Andreas<br class="">
<blockquote type="cite" class="">
<div class="">
<div class="moz-text-html" lang="x-unicode">
<p class=""></p>
<p class="">Pools from nbp17:<br class="">
<font face="Courier New, Courier, monospace" class="">poollist_get() nPool=10 buffer=nbp17.1_hdd<br class="">
Pool: nbp17.1_hdd<br class="">
poollist_get() poolNames[0]=nbp17.1_hdd 8 members buffer=nbp17-OST0000_UUID<br class="">
poollist_get() poolNames[0]=nbp17.1_hdd members[0]=nbp17-OST0000_UUID<br class="">
poollist_get() poolNames[0]=nbp17.1_hdd members[1]=nbp17-OST0001_UUID<br class="">
poollist_get() poolNames[0]=nbp17.1_hdd members[2]=nbp17-OST0002_UUID<br class="">
poollist_get() poolNames[0]=nbp17.1_hdd members[3]=nbp17-OST0003_UUID<br class="">
poollist_get() poolNames[0]=nbp17.1_hdd members[4]=nbp17-OST0004_UUID<br class="">
poollist_get() poolNames[0]=nbp17.1_hdd members[5]=nbp17-OST0005_UUID<br class="">
poollist_get() poolNames[0]=nbp17.1_hdd members[6]=nbp17-OST0006_UUID<br class="">
poollist_get() poolNames[0]=nbp17.1_hdd members[7]=nbp17-OST0007_UUID<br class="">
Pool: nbp17.1_ssd<br class="">
poollist_get() poolNames[1]=nbp17.1_ssd 8 members buffer=nbp17-OST0064_UUID<br class="">
poollist_get() poolNames[1]=nbp17.1_ssd members[0]=nbp17-OST0064_UUID<br class="">
poollist_get() poolNames[1]=nbp17.1_ssd members[1]=nbp17-OST0065_UUID<br class="">
poollist_get() poolNames[1]=nbp17.1_ssd members[2]=nbp17-OST0066_UUID<br class="">
poollist_get() poolNames[1]=nbp17.1_ssd members[3]=nbp17-OST0067_UUID<br class="">
poollist_get() poolNames[1]=nbp17.1_ssd members[4]=nbp17-OST0068_UUID<br class="">
poollist_get() poolNames[1]=nbp17.1_ssd members[5]=nbp17-OST0069_UUID<br class="">
poollist_get() poolNames[1]=nbp17.1_ssd members[6]=nbp17-OST006a_UUID<br class="">
poollist_get() poolNames[1]=nbp17.1_ssd members[7]=nbp17-OST006b_UUID<br class="">
Pool: nbp17.2_hdd<br class="">
poollist_get() poolNames[2]=nbp17.2_hdd 8 members buffer=nbp17-OST0008_UUID<br class="">
poollist_get() poolNames[2]=nbp17.2_hdd members[0]=nbp17-OST0008_UUID<br class="">
poollist_get() poolNames[2]=nbp17.2_hdd members[1]=nbp17-OST0009_UUID<br class="">
poollist_get() poolNames[2]=nbp17.2_hdd members[2]=nbp17-OST000a_UUID<br class="">
poollist_get() poolNames[2]=nbp17.2_hdd members[3]=nbp17-OST000b_UUID<br class="">
poollist_get() poolNames[2]=nbp17.2_hdd members[4]=nbp17-OST000c_UUID<br class="">
poollist_get() poolNames[2]=nbp17.2_hdd members[5]=nbp17-OST000d_UUID<br class="">
poollist_get() poolNames[2]=nbp17.2_hdd members[6]=nbp17-OST000e_UUID<br class="">
poollist_get() poolNames[2]=nbp17.2_hdd members[7]=nbp17-OST000f_UUID<br class="">
Pool: nbp17.2_ssd<br class="">
poollist_get() poolNames[3]=nbp17.2_ssd 8 members buffer=nbp17-OST006c_UUID<br class="">
poollist_get() poolNames[3]=nbp17.2_ssd members[0]=nbp17-OST006c_UUID<br class="">
poollist_get() poolNames[3]=nbp17.2_ssd members[1]=nbp17-OST006d_UUID<br class="">
poollist_get() poolNames[3]=nbp17.2_ssd members[2]=nbp17-OST006e_UUID<br class="">
poollist_get() poolNames[3]=nbp17.2_ssd members[3]=nbp17-OST006f_UUID<br class="">
poollist_get() poolNames[3]=nbp17.2_ssd members[4]=nbp17-OST0070_UUID<br class="">
poollist_get() poolNames[3]=nbp17.2_ssd members[5]=nbp17-OST0071_UUID<br class="">
poollist_get() poolNames[3]=nbp17.2_ssd members[6]=nbp17-OST0072_UUID<br class="">
poollist_get() poolNames[3]=nbp17.2_ssd members[7]=nbp17-OST0073_UUID<br class="">
Pool: nbp17.3_hdd<br class="">
poollist_get() poolNames[4]=nbp17.3_hdd 8 members buffer=nbp17-OST0010_UUID<br class="">
poollist_get() poolNames[4]=nbp17.3_hdd members[0]=nbp17-OST0010_UUID<br class="">
poollist_get() poolNames[4]=nbp17.3_hdd members[1]=nbp17-OST0011_UUID<br class="">
poollist_get() poolNames[4]=nbp17.3_hdd members[2]=nbp17-OST0012_UUID<br class="">
poollist_get() poolNames[4]=nbp17.3_hdd members[3]=nbp17-OST0013_UUID<br class="">
poollist_get() poolNames[4]=nbp17.3_hdd members[4]=nbp17-OST0014_UUID<br class="">
poollist_get() poolNames[4]=nbp17.3_hdd members[5]=nbp17-OST0015_UUID<br class="">
poollist_get() poolNames[4]=nbp17.3_hdd members[6]=nbp17-OST0016_UUID<br class="">
poollist_get() poolNames[4]=nbp17.3_hdd members[7]=nbp17-OST0017_UUID<br class="">
Pool: nbp17.3_ssd<br class="">
poollist_get() poolNames[5]=nbp17.3_ssd 8 members buffer=nbp17-OST0074_UUID<br class="">
poollist_get() poolNames[5]=nbp17.3_ssd members[0]=nbp17-OST0074_UUID<br class="">
poollist_get() poolNames[5]=nbp17.3_ssd members[1]=nbp17-OST0075_UUID<br class="">
poollist_get() poolNames[5]=nbp17.3_ssd members[2]=nbp17-OST0076_UUID<br class="">
poollist_get() poolNames[5]=nbp17.3_ssd members[3]=nbp17-OST0077_UUID<br class="">
poollist_get() poolNames[5]=nbp17.3_ssd members[4]=nbp17-OST0078_UUID<br class="">
poollist_get() poolNames[5]=nbp17.3_ssd members[5]=nbp17-OST0079_UUID<br class="">
poollist_get() poolNames[5]=nbp17.3_ssd members[6]=nbp17-OST007a_UUID<br class="">
poollist_get() poolNames[5]=nbp17.3_ssd members[7]=nbp17-OST007b_UUID<br class="">
Pool: nbp17.4_hdd<br class="">
poollist_get() poolNames[6]=nbp17.4_hdd 8 members buffer=nbp17-OST0018_UUID<br class="">
poollist_get() poolNames[6]=nbp17.4_hdd members[0]=nbp17-OST0018_UUID<br class="">
poollist_get() poolNames[6]=nbp17.4_hdd members[1]=nbp17-OST0019_UUID<br class="">
poollist_get() poolNames[6]=nbp17.4_hdd members[2]=nbp17-OST001a_UUID<br class="">
poollist_get() poolNames[6]=nbp17.4_hdd members[3]=nbp17-OST001b_UUID<br class="">
poollist_get() poolNames[6]=nbp17.4_hdd members[4]=nbp17-OST001c_UUID<br class="">
poollist_get() poolNames[6]=nbp17.4_hdd members[5]=nbp17-OST001d_UUID<br class="">
poollist_get() poolNames[6]=nbp17.4_hdd members[6]=nbp17-OST001e_UUID<br class="">
poollist_get() poolNames[6]=nbp17.4_hdd members[7]=nbp17-OST001f_UUID<br class="">
Pool: nbp17.4_ssd<br class="">
poollist_get() poolNames[7]=nbp17.4_ssd 8 members buffer=nbp17-OST007c_UUID<br class="">
poollist_get() poolNames[7]=nbp17.4_ssd members[0]=nbp17-OST007c_UUID<br class="">
poollist_get() poolNames[7]=nbp17.4_ssd members[1]=nbp17-OST007d_UUID<br class="">
poollist_get() poolNames[7]=nbp17.4_ssd members[2]=nbp17-OST007e_UUID<br class="">
poollist_get() poolNames[7]=nbp17.4_ssd members[3]=nbp17-OST007f_UUID<br class="">
poollist_get() poolNames[7]=nbp17.4_ssd members[4]=nbp17-OST0080_UUID<br class="">
poollist_get() poolNames[7]=nbp17.4_ssd members[5]=nbp17-OST0081_UUID<br class="">
poollist_get() poolNames[7]=nbp17.4_ssd members[6]=nbp17-OST0082_UUID<br class="">
poollist_get() poolNames[7]=nbp17.4_ssd members[7]=nbp17-OST0083_UUID<br class="">
Pool: nbp17.hdd-pool<br class="">
poollist_get() poolNames[8]=nbp17.hdd-pool -75 members buffer=nbp17-OST0000_UUID<br class="">
Pool: nbp17.ssd-pool<br class="">
poollist_get() poolNames[9]=nbp17.ssd-pool -75 members buffer=nbp17-OST0064_UUID</font><br class="">
<br class="">
</p>
<p class="">John<br class="">
</p>
<div class="moz-cite-prefix">On 6/15/22 02:08, Andreas Dilger wrote:<br class="">
</div>
<blockquote type="cite" cite="mid:19C08E40-B54A-485D-99E8-9258866085E0@ddn.com" class="">
On Jun 14, 2022, at 05:32, John Bauer <<a href="mailto:bauerj@iodoctors.com" class="moz-txt-link-freetext">bauerj@iodoctors.com</a>> wrote:<br class="">
<div class="">
<blockquote type="cite" class=""><br class="Apple-interchange-newline">
<div class="">
<div class="">I have had little success in my search for documentation on pool functions in llapi. I've looked in:<br class="">
<br class="">
<a href="https://wiki.lustre.org/PFL2_High_Level_Design" class="moz-txt-link-freetext">https://wiki.lustre.org/PFL2_High_Level_Design</a><br class="">
<br class="">
<a class="moz-txt-link-freetext" href="https://doc.lustre.org/lustre_manual.xhtml#managingstripingfreespace">https://doc.lustre.org/lustre_manual.xhtml#managingstripingfreespace</a><br class="">
<br class="">
<br class="">
I'm looking for info on llapi_get_poollist() and llapi_get_poolmembers(). I've found the prototype in /usr/include/lustre/lustreapi.h, but that's about it.<br class="">
<br class="">
Can anyone point me to some documentation?<br class="">
</div>
</div>
</blockquote>
<br class="">
</div>
<div class="">it looks like there aren't man pages for these functions, just the function comment blocks</div>
<div class="">in the code, and their usage internally:</div>
<div class=""><br class="">
</div>
<div class="">/**<br class="">
* Get the list of pools in a filesystem.<br class="">
* \param name filesystem name or path<br class="">
* \param poollist caller-allocated array of char*<br class="">
* \param list_size size of the poollist array<br class="">
* \param buffer caller-allocated buffer for storing pool names<br class="">
* \param buffer_size size of the buffer<br class="">
*<br class="">
* \return number of pools retrieved for this filesystem<br class="">
* \retval -error failure<br class="">
*/<br class="">
int llapi_get_poollist(const char *name, char **poollist, int list_size,<br class="">
char *buffer, int buffer_size)</div>
<div class=""><br class="">
</div>
<div class="">/**<br class="">
* Get the list of pool members.<br class="">
* \param poolname string of format \<fsname\>.\<poolname\><br class="">
* \param members caller-allocated array of char*<br class="">
* \param list_size size of the members array<br class="">
* \param buffer caller-allocated buffer for storing OST names<br class="">
* \param buffer_size size of the buffer<br class="">
*<br class="">
* \return number of members retrieved for this pool<br class="">
* \retval -error failure<br class="">
*/<br class="">
int llapi_get_poolmembers(const char *poolname, char **members,<br class="">
int list_size, char *buffer, int buffer_size)</div>
<div class=""><br class="">
</div>
<div class="">Patches to add llapi_get_poollist.3 and llapi_get_poolmembers.3 (and related) man</div>
<div class="">pages welcome. The pool related functions should probably be moved into a new</div>
<div class="">liblustreapi_pool.c file to reduce the size of liblustreapi.c.</div>
<br class="">
<div class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div class="">Cheers, Andreas</div>
<div class="">--</div>
<div class="">Andreas Dilger</div>
<div class="">Lustre Principal Architect</div>
<div class="">Whamcloud</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
</div>
</div>
</div>
</div>
</div>
<br class="Apple-interchange-newline">
</div>
<br class="Apple-interchange-newline">
<br class="Apple-interchange-newline">
</div>
<br class="">
</blockquote>
</div>
<div class="moz-cite-prefix">On 6/15/22 02:08, Andreas Dilger wrote:<br class="">
</div>
<blockquote type="cite" cite="mid:19C08E40-B54A-485D-99E8-9258866085E0@ddn.com" class="">
On Jun 14, 2022, at 05:32, John Bauer <<a href="mailto:bauerj@iodoctors.com" class="moz-txt-link-freetext" moz-do-not-send="true">bauerj@iodoctors.com</a>> wrote:<br class="">
<div class="">
<blockquote type="cite" class=""><br class="Apple-interchange-newline">
<div class="">
<div class="">I have had little success in my search for documentation on pool functions in llapi. I've looked in:<br class="">
<br class="">
<a href="https://wiki.lustre.org/PFL2_High_Level_Design" class="moz-txt-link-freetext" moz-do-not-send="true">https://wiki.lustre.org/PFL2_High_Level_Design</a><br class="">
<br class="">
<a class="moz-txt-link-freetext" href="https://doc.lustre.org/lustre_manual.xhtml#managingstripingfreespace">https://doc.lustre.org/lustre_manual.xhtml#managingstripingfreespace</a><br class="">
<br class="">
<br class="">
I'm looking for info on llapi_get_poollist() and llapi_get_poolmembers(). I've found the prototype in /usr/include/lustre/lustreapi.h, but that's about it.<br class="">
<br class="">
Can anyone point me to some documentation?<br class="">
</div>
</div>
</blockquote>
<br class="">
</div>
<div class="">it looks like there aren't man pages for these functions, just the function comment blocks</div>
<div class="">in the code, and their usage internally:</div>
<div class=""><br class="">
</div>
<div class="">/**<br class="">
* Get the list of pools in a filesystem.<br class="">
* \param name filesystem name or path<br class="">
* \param poollist caller-allocated array of char*<br class="">
* \param list_size size of the poollist array<br class="">
* \param buffer caller-allocated buffer for storing pool names<br class="">
* \param buffer_size size of the buffer<br class="">
*<br class="">
* \return number of pools retrieved for this filesystem<br class="">
* \retval -error failure<br class="">
*/<br class="">
int llapi_get_poollist(const char *name, char **poollist, int list_size,<br class="">
char *buffer, int buffer_size)</div>
<div class=""><br class="">
</div>
<div class="">/**<br class="">
* Get the list of pool members.<br class="">
* \param poolname string of format \<fsname\>.\<poolname\><br class="">
* \param members caller-allocated array of char*<br class="">
* \param list_size size of the members array<br class="">
* \param buffer caller-allocated buffer for storing OST names<br class="">
* \param buffer_size size of the buffer<br class="">
*<br class="">
* \return number of members retrieved for this pool<br class="">
* \retval -error failure<br class="">
*/<br class="">
int llapi_get_poolmembers(const char *poolname, char **members,<br class="">
int list_size, char *buffer, int buffer_size)</div>
<div class=""><br class="">
</div>
<div class="">Patches to add llapi_get_poollist.3 and llapi_get_poolmembers.3 (and related) man</div>
<div class="">pages welcome. The pool related functions should probably be moved into a new</div>
<div class="">liblustreapi_pool.c file to reduce the size of liblustreapi.c.</div>
<br class="">
<div class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div class="">Cheers, Andreas</div>
<div class="">--</div>
<div class="">Andreas Dilger</div>
<div class="">Lustre Principal Architect</div>
<div class="">Whamcloud</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
</div>
</div>
</div>
</div>
</div>
<br class="Apple-interchange-newline">
</div>
<br class="Apple-interchange-newline">
<br class="Apple-interchange-newline">
</div>
<br class="">
</blockquote>
</div>
</blockquote>
</div>
<br class="">
<div class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div>Cheers, Andreas</div>
<div>--</div>
<div>Andreas Dilger</div>
<div>Lustre Principal Architect</div>
<div>Whamcloud</div>
<div><br class="">
</div>
<div><br class="">
</div>
<div><br class="">
</div>
</div>
</div>
</div>
</div>
</div>
<br class="Apple-interchange-newline">
</div>
<br class="Apple-interchange-newline">
<br class="Apple-interchange-newline">
</div>
<br class="">
</body>
</html>