<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">
Thanks, Colin. Let me do a sanity check on the process and I’ll see about updating the Lustre with ZFS page [1].
<div class=""><br class="">
</div>
<div class="">—Rick</div>
<div class=""><br class="">
</div>
<div class="">[1] <a href="http://wiki.lustre.org/Lustre_with_ZFS_Install" class="">
http://wiki.lustre.org/Lustre_with_ZFS_Install</a></div>
<div class=""><br class="">
<div>
<blockquote type="cite" class="">
<div class="">On Nov 2, 2015, at 1:07 PM, Colin Faber <<a href="mailto:colin.faber@seagate.com" class="">colin.faber@seagate.com</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div dir="ltr" class="">Hi Rick,
<div class=""><br class="">
</div>
<div class="">This is fantastic, if you haven't already would you mind adding this to the
<a href="http://wiki.lustre.org/" class="">wiki.lustre.org</a> wiki site?</div>
<div class=""><br class="">
</div>
<div class="">-cf</div>
</div>
<div class="gmail_extra"><br class="">
<div class="gmail_quote">On Mon, Nov 2, 2015 at 1:25 PM, Rick Wagner <span dir="ltr" class="">
<<a href="mailto:rpwagner@sdsc.edu" target="_blank" class="">rpwagner@sdsc.edu</a>></span> wrote:<br class="">
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div style="word-wrap:break-word" class="">
<div class="">We’ve run ZFS-backed Lustre servers using a patchless (i.e., without ldiskfs) Linux 3 kernel under CentOS 6. While I don’t recommend this for everyone I can say that the mechanics of building the kernel modules (ZFS and Lustre) and standing up
the file system all worked well. We’re looking to move back to a stock CentOS 6 kernel, but are sticking with the patchless model. If you’re willing to try ZFS I suspect the build will be fine, and then you should test test test.</div>
<div class=""><br class="">
</div>
<div class="">I’ve included the outline of commands we use to build our server RPMs. You’ll want to adjust the versions of SPL, ZFS, & Lustre, along with the --with-linux and --with-linux-config parameters. I apologize for any typos and subsequent frustration.
What level of dependency hell you find yourself in is is largely dependent on your choice of distro.</div>
<div class=""><br class="">
</div>
<div class="">—Rick</div>
<div class=""><br class="">
</div>
<div class="">git clone <a href="https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_zfsonlinux_spl.git&d=CwMF-g&c=IGDlg0lD0b-nebmJJ0Kp8A&r=x9pM59OqndbWw-lPPdr8w1Vud29EZigcxcNkz0uw5oQ&m=CjVqTjggbFA1F-2WDW87aU9UmdOsvkSSdbrx6n_jVLg&s=tHup8osKBWRBPj9rC-qu--lua4itTMknj0eiUjMPJMg&e=" target="_blank" class="">
https://github.com/zfsonlinux/spl.git</a></div>
<div class="">cd spl<br class="">
git checkout spl-0.6.5.3<br class="">
./autogen.sh<br class="">
./configure</div>
<div class="">make pkg</div>
<div class="">rpm -ivh *x86_64.rpm</div>
<div class=""><br class="">
</div>
<div class="">cd ../</div>
<div class="">
<div class="">git clone <a href="https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_zfsonlinux_zfs.git&d=CwMF-g&c=IGDlg0lD0b-nebmJJ0Kp8A&r=x9pM59OqndbWw-lPPdr8w1Vud29EZigcxcNkz0uw5oQ&m=CjVqTjggbFA1F-2WDW87aU9UmdOsvkSSdbrx6n_jVLg&s=QijKf5IjRvVDun-yIB7EWmvWA7NX8KTdBzyCR5b2PsA&e=" target="_blank" class="">
https://github.com/zfsonlinux/zfs.git</a></div>
<div class="">cd zfs<br class="">
git checkout zfs-0.6.5.3<br class="">
./autogen.sh<br class="">
./configure</div>
<div class="">make pkg</div>
<div class="">rpm -ivh *x86_64.rpm</div>
</div>
<div class=""><br class="">
</div>
<div class="">cd ../</div>
<div class="">git clone <a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__git.whamcloud.com_fs_lustre-2Drelease.git&d=CwMF-g&c=IGDlg0lD0b-nebmJJ0Kp8A&r=x9pM59OqndbWw-lPPdr8w1Vud29EZigcxcNkz0uw5oQ&m=CjVqTjggbFA1F-2WDW87aU9UmdOsvkSSdbrx6n_jVLg&s=-Na1KZ7h6BNtaEVI_N0DRtTl1xhhwsqJ7YlqvXYxQNg&e=" target="_blank" class="">
http://git.whamcloud.com/fs/lustre-release.git</a></div>
<div class="">cd lustre-release</div>
<div class="">git checkout 2.7.62</div>
<div class="">sh autogen.sh<br class="">
./configure --with-zfs=/usr/src/zfs-0.6.5.3 --with-spl=/usr/src/spl-0.6.5.3 --with-linux=/usr/src/linux-3.10 --without-ldiskfs --with-linux-config=/boot/config-3.10.73-1.el6.elrepo.x86_64 --with-o2ib=no<br class="">
make rpms</div>
<div class="">
<div class="h5">
<div class=""><br class="">
</div>
<br class="">
<div class="">
<blockquote type="cite" class="">
<div class="">On Nov 2, 2015, at 11:45 AM, Patrick Farrell <<a href="mailto:paf@cray.com" target="_blank" class="">paf@cray.com</a>> wrote:</div>
<br class="">
<div class=""><span style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);float:none;display:inline!important" class="">For
ZFS:</span><br style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255)" class="">
<br style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255)" class="">
<span style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);float:none;display:inline!important" class="">The
pitfall is primarily that it's less familiar than ldiskfs, which basically uses the standard Linux fs tools, like ext. ZFS is its own world, which will probably have a learning curve for someone who has not used it before, and it must be built for the version
of the kernel you're interested in.</span><br style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255)" class="">
<br style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255)" class="">
<span style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);float:none;display:inline!important" class="">But,
Ben is right, that's likely to be easier than porting ldiskfs.</span><br style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255)" class="">
<br style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255)" class="">
<span style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);float:none;display:inline!important" class="">-
Patrick</span><br style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255)" class="">
<br style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255)" class="">
<div style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255)" class="">
On 11/02/2015 01:35 PM, Smith, Stan wrote:<br class="">
</div>
<blockquote type="cite" style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255)" class="">
<div class="">
<div style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif" class="">
<b class=""><span style="color:windowtext" class="">From:</span></b><span style="color:windowtext" class=""><span class=""> </span>lustre-devel [<a href="mailto:lustre-devel-bounces@lists.lustre.org" style="color:rgb(149,79,114);text-decoration:underline" target="_blank" class="">mailto:lustre-devel-bounces@lists.lustre.org</a>]<span class=""> </span><b class="">On
Behalf Of<span class=""> </span></b>Patrick Farrell<br class="">
<b class="">Sent:</b><span class=""> </span>Monday, November 2, 2015 11:13 AM<br class="">
<b class="">To:</b><span class=""> </span><a href="mailto:lustre-devel@lists.lustre.org" style="color:rgb(149,79,114);text-decoration:underline" target="_blank" class="">lustre-devel@lists.lustre.org</a><br class="">
<b class="">Subject:</b><span class=""> </span>Re: [lustre-devel] porting Lustre server to RHEL6.7 distro running a
<a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__kernel.org&d=CwMF-g&c=IGDlg0lD0b-nebmJJ0Kp8A&r=x9pM59OqndbWw-lPPdr8w1Vud29EZigcxcNkz0uw5oQ&m=CjVqTjggbFA1F-2WDW87aU9UmdOsvkSSdbrx6n_jVLg&s=JNDXqBVf-dJrXGvONRXY6Ow6ONNvfCJCHdkedQYE1WA&e=" target="_blank" class="">
kernel.org</a> linux-3.12.39<u class=""></u><u class=""></u></span></div>
<div style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif" class="">
<u class=""></u> <u class=""></u></div>
<div style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif" class="">
Stan,<br class="">
<br class="">
Why not work with the CentOS 7 kernel on the server anyway? Is something your LND needs not available? CentOS 7 is freely available and isn't far off in terms of kernel version from 3.12.<span style="color:rgb(31,73,125)" class=""><u class=""></u><u class=""></u></span></div>
<div style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif" class="">
<span style="color:rgb(31,73,125)" class=""> </span></div>
<div style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif" class="">
<span style="color:rgb(31,73,125)" class="">Yes –I’m somewhat stuck on RHEL 6.x series as I use warewulf as a cluster provisioning system (diskless compute nodes), so far have not found a functional warewulf for RHEL7.<u class=""></u><u class=""></u></span></div>
<div style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif" class="">
<span style="color:rgb(31,73,125)" class="">I did choose linux-3.12.39 as a kernel base believing it to be somewhat close to RHEL7; although I’m finding this to less so than hoped for….<u class=""></u><u class=""></u></span></div>
<div style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif" class="">
<span style="color:rgb(31,73,125)" class="">Additionally the RD drivers I’m using have not been ported to RHEL7;<u class=""></u><u class=""></u></span></div>
<div style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif" class="">
<span style="color:rgb(31,73,125)" class="">In examining lustre-2_7.62 I found support for sles12 based on 3.12.39 kernel.; thinking the lustre patches would be close…<u class=""></u><u class=""></u></span></div>
<div style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif" class="">
<span style="color:rgb(31,73,125)" class="">So now I’m wedged between RHEL7 ldiskfs patches and SLES12 ldiskfs patches – figure out distro specific enhancements vs lustre enhancements.</span><br class="">
<br class="">
You may be hard pressed to get useful advice on changing the ldiskfs patch series, that has the potential to be a pretty big job.<span style="color:rgb(31,73,125)" class=""><u class=""></u><u class=""></u></span></div>
<div style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif" class="">
<span style="color:rgb(31,73,125)" class=""> </span></div>
<div style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif" class="">
<span style="color:rgb(31,73,125)" class="">I’m starting to understand the magnitude of the ldiskfs task…….<u class=""></u><u class=""></u></span></div>
<div style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif" class="">
<span style="color:rgb(31,73,125)" class=""> </span></div>
<div style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif" class="">
<span style="color:rgb(31,73,125)" class="">I’m ignorant about zfs*, would zfs be a better (read that as path-of-least-resistance) approach ? What are the zfs pitfalls?<u class=""></u><u class=""></u></span></div>
<div style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif" class="">
<span style="color:rgb(31,73,125)" class=""> </span></div>
<div style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif" class="">
<span style="color:rgb(31,73,125)" class="">Thanks,<u class=""></u><u class=""></u></span></div>
<div style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif" class="">
<span style="color:rgb(31,73,125)" class=""> </span></div>
<div style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif" class="">
<span style="color:rgb(31,73,125)" class="">Stan.<u class=""></u><u class=""></u></span></div>
<div style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif" class="">
<br class="">
<br class="">
Regards,<br class="">
- Patrick Farrell<span style="font-size:12pt" class=""><u class=""></u><u class=""></u></span></div>
<div class="">
<div style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif" class="">
On 11/02/2015 01:05 PM, Smith, Stan wrote:<u class=""></u><u class=""></u></div>
</div>
<blockquote style="margin-top:5pt;margin-bottom:5pt" class="">
<div style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif" class="">
<span style="font-size:12pt" class="">Hello,</span><u class=""></u><u class=""></u></div>
<div style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif" class="">
<span style="font-size:12pt" class=""> I’m currently unable to search the lustre-devel archives + 1<sup class="">st</sup><span class=""> </span>post to this list.</span><u class=""></u><u class=""></u></div>
<div style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif" class="">
<span style="font-size:12pt" class=""> </span><u class=""></u><u class=""></u></div>
<div style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif" class="">
<span style="font-size:12pt" class="">Having read lustre documentation I find only references to building lustre server for stock ‘distro’ kernels.</span><u class=""></u><u class=""></u></div>
<div style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif" class="">
<span style="font-size:12pt" class=""> </span><u class=""></u><u class=""></u></div>
<div style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif" class="">
<span style="font-size:12pt" class="">Does someone have suggestions on how to patch ldiskfs for a
<a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__kernel.org&d=CwMF-g&c=IGDlg0lD0b-nebmJJ0Kp8A&r=x9pM59OqndbWw-lPPdr8w1Vud29EZigcxcNkz0uw5oQ&m=CjVqTjggbFA1F-2WDW87aU9UmdOsvkSSdbrx6n_jVLg&s=JNDXqBVf-dJrXGvONRXY6Ow6ONNvfCJCHdkedQYE1WA&e=" target="_blank" class="">
kernel.org</a> kernel…..specifically weeding out RHEL/SLES/* distro FS enhancements from a
<a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__kernel.org&d=CwMF-g&c=IGDlg0lD0b-nebmJJ0Kp8A&r=x9pM59OqndbWw-lPPdr8w1Vud29EZigcxcNkz0uw5oQ&m=CjVqTjggbFA1F-2WDW87aU9UmdOsvkSSdbrx6n_jVLg&s=JNDXqBVf-dJrXGvONRXY6Ow6ONNvfCJCHdkedQYE1WA&e=" target="_blank" class="">
kernel.org</a> base?</span><u class=""></u><u class=""></u></div>
<div style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif" class="">
<span style="font-size:12pt" class=""> </span><u class=""></u><u class=""></u></div>
<div style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif" class="">
<span style="font-size:12pt" class="">I have written a Reliable Datagram LND which passes the LNet selftests (client lustre build) now entering the world of lustre server building.</span><u class=""></u><u class=""></u></div>
<div style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif" class="">
<span style="font-size:12pt" class="">Patching the <a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__kernel.org&d=CwMF-g&c=IGDlg0lD0b-nebmJJ0Kp8A&r=x9pM59OqndbWw-lPPdr8w1Vud29EZigcxcNkz0uw5oQ&m=CjVqTjggbFA1F-2WDW87aU9UmdOsvkSSdbrx6n_jVLg&s=JNDXqBVf-dJrXGvONRXY6Ow6ONNvfCJCHdkedQYE1WA&e=" target="_blank" class="">
kernel.org</a> kernel for lustre server was fairly straightforward, creating the patch series for ldiskfs is another story.</span><u class=""></u><u class=""></u></div>
<div style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif" class="">
<span style="font-size:12pt" class=""> </span><u class=""></u><u class=""></u></div>
<div style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif" class="">
<span style="font-size:12pt" class="">Thanks,</span><u class=""></u><u class=""></u></div>
<div style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif" class="">
<span style="font-size:12pt" class=""> </span><u class=""></u><u class=""></u></div>
<div style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif" class="">
<span style="font-size:12pt" class="">Stan.</span><u class=""></u><u class=""></u></div>
<div style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif" class="">
<span style="font-size:12pt" class=""><br class="">
<br class="">
<br class="">
<u class=""></u><u class=""></u></span></div>
<pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:'Courier New'" class="">_______________________________________________<u class=""></u><u class=""></u></pre>
<pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:'Courier New'" class="">lustre-devel mailing list<u class=""></u><u class=""></u></pre>
<pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:'Courier New'" class=""><a href="mailto:lustre-devel@lists.lustre.org" style="color:rgb(149,79,114);text-decoration:underline" target="_blank" class="">lustre-devel@lists.lustre.org</a><u class=""></u><u class=""></u></pre>
<pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:'Courier New'" class=""><a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.lustre.org_listinfo.cgi_lustre-2Ddevel-2Dlustre.org&d=CwMF-g&c=IGDlg0lD0b-nebmJJ0Kp8A&r=x9pM59OqndbWw-lPPdr8w1Vud29EZigcxcNkz0uw5oQ&m=CjVqTjggbFA1F-2WDW87aU9UmdOsvkSSdbrx6n_jVLg&s=A6GC4M4a2RZsUUrt-VAKwS0MmRRdPLYYgVDeJ5RDF7Q&e=" style="color:rgb(149,79,114);text-decoration:underline" target="_blank" class="">http://lists.lustre.org/listinfo.cgi/lustre-devel-lustre.org</a><u class=""></u><u class=""></u></pre>
</blockquote>
<div style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif" class="">
<span style="font-size:12pt" class=""> </span></div>
</div>
</blockquote>
<br style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255)" class="">
<span style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);float:none;display:inline!important" class="">_______________________________________________</span><br style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255)" class="">
<span style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);float:none;display:inline!important" class="">lustre-devel
mailing list</span><br style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255)" class="">
<span style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);float:none;display:inline!important" class=""><a href="mailto:lustre-devel@lists.lustre.org" target="_blank" class="">lustre-devel@lists.lustre.org</a></span><br style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255)" class="">
<span style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);float:none;display:inline!important" class=""><a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.lustre.org_listinfo.cgi_lustre-2Ddevel-2Dlustre.org&d=CwMF-g&c=IGDlg0lD0b-nebmJJ0Kp8A&r=x9pM59OqndbWw-lPPdr8w1Vud29EZigcxcNkz0uw5oQ&m=CjVqTjggbFA1F-2WDW87aU9UmdOsvkSSdbrx6n_jVLg&s=A6GC4M4a2RZsUUrt-VAKwS0MmRRdPLYYgVDeJ5RDF7Q&e=" target="_blank" class="">http://lists.lustre.org/listinfo.cgi/lustre-devel-lustre.org</a></span><br style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255)" class="">
</div>
</blockquote>
</div>
<br class="">
</div>
</div>
</div>
<br class="">
_______________________________________________<br class="">
lustre-devel mailing list<br class="">
<a href="mailto:lustre-devel@lists.lustre.org" class="">lustre-devel@lists.lustre.org</a><br class="">
<a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.lustre.org_listinfo.cgi_lustre-2Ddevel-2Dlustre.org&d=CwICAg&c=IGDlg0lD0b-nebmJJ0Kp8A&r=x9pM59OqndbWw-lPPdr8w1Vud29EZigcxcNkz0uw5oQ&m=CjVqTjggbFA1F-2WDW87aU9UmdOsvkSSdbrx6n_jVLg&s=A6GC4M4a2RZsUUrt-VAKwS0MmRRdPLYYgVDeJ5RDF7Q&e=" rel="noreferrer" target="_blank" class="">https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.lustre.org_listinfo.cgi_lustre-2Ddevel-2Dlustre.org&d=CwICAg&c=IGDlg0lD0b-nebmJJ0Kp8A&r=x9pM59OqndbWw-lPPdr8w1Vud29EZigcxcNkz0uw5oQ&m=CjVqTjggbFA1F-2WDW87aU9UmdOsvkSSdbrx6n_jVLg&s=A6GC4M4a2RZsUUrt-VAKwS0MmRRdPLYYgVDeJ5RDF7Q&e=</a><br class="">
<br class="">
</blockquote>
</div>
<br class="">
</div>
</div>
</blockquote>
</div>
<br class="">
</div>
</body>
</html>