<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">
<div class="">We’ve run ZFS-backed Lustre servers using a patchless (i.e., without ldiskfs) Linux 3 kernel under CentOS 6. While I don’t recommend this for everyone I can say that the mechanics of building the kernel modules (ZFS and Lustre) and standing up
the file system all worked well. We’re looking to move back to a stock CentOS 6 kernel, but are sticking with the patchless model. If you’re willing to try ZFS I suspect the build will be fine, and then you should test test test.</div>
<div class=""><br class="">
</div>
<div class="">I’ve included the outline of commands we use to build our server RPMs. You’ll want to adjust the versions of SPL, ZFS, & Lustre, along with the --with-linux and --with-linux-config parameters. I apologize for any typos and subsequent frustration.
What level of dependency hell you find yourself in is is largely dependent on your choice of distro.</div>
<div class=""><br class="">
</div>
<div class="">—Rick</div>
<div class=""><br class="">
</div>
<div class="">git clone <a href="https://github.com/zfsonlinux/spl.git" class="">
https://github.com/zfsonlinux/spl.git</a></div>
<div class="">cd spl<br class="">
git checkout spl-0.6.5.3<br class="">
./autogen.sh<br class="">
./configure</div>
<div class="">make pkg</div>
<div class="">rpm -ivh *x86_64.rpm</div>
<div class=""><br class="">
</div>
<div class="">cd ../</div>
<div class="">
<div class="">git clone <a href="https://github.com/zfsonlinux/zfs.git" class="">
https://github.com/zfsonlinux/zfs.git</a></div>
<div class="">cd zfs<br class="">
git checkout zfs-0.6.5.3<br class="">
./autogen.sh<br class="">
./configure</div>
<div class="">make pkg</div>
<div class="">rpm -ivh *x86_64.rpm</div>
</div>
<div class=""><br class="">
</div>
<div class="">cd ../</div>
<div class="">git clone <a href="http://git.whamcloud.com/fs/lustre-release.git" class="">
http://git.whamcloud.com/fs/lustre-release.git</a></div>
<div class="">cd lustre-release</div>
<div class="">git checkout 2.7.62</div>
<div class="">sh autogen.sh<br class="">
./configure --with-zfs=/usr/src/zfs-0.6.5.3 --with-spl=/usr/src/spl-0.6.5.3 --with-linux=/usr/src/linux-3.10 --without-ldiskfs --with-linux-config=/boot/config-3.10.73-1.el6.elrepo.x86_64 --with-o2ib=no<br class="">
make rpms</div>
<div class=""><br class="">
</div>
<br class="">
<div>
<blockquote type="cite" class="">
<div class="">On Nov 2, 2015, at 11:45 AM, Patrick Farrell <<a href="mailto:paf@cray.com" class="">paf@cray.com</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255); float: none; display: inline !important;" class="">For
ZFS:</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255);" class="">
<br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255);" class="">
<span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255); float: none; display: inline !important;" class="">The
pitfall is primarily that it's less familiar than ldiskfs, which basically uses the standard Linux fs tools, like ext. ZFS is its own world, which will probably have a learning curve for someone who has not used it before, and it must be built for the version
of the kernel you're interested in.</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255);" class="">
<br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255);" class="">
<span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255); float: none; display: inline !important;" class="">But,
Ben is right, that's likely to be easier than porting ldiskfs.</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255);" class="">
<br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255);" class="">
<span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255); float: none; display: inline !important;" class="">-
Patrick</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255);" class="">
<br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255);" class="">
<div class="moz-cite-prefix" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255);">
On 11/02/2015 01:35 PM, Smith, Stan wrote:<br class="">
</div>
<blockquote cite="mid:2DDE0F15AAFF91448E6F813C42EF4CBEA195FE29@ORSMSX105.amr.corp.intel.com" type="cite" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255);" class="">
<div class="WordSection1" style="page: WordSection1;">
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<b class=""><span style="color: windowtext;" class="">From:</span></b><span style="color: windowtext;" class=""><span class="Apple-converted-space"> </span>lustre-devel [<a class="moz-txt-link-freetext" href="mailto:lustre-devel-bounces@lists.lustre.org" style="color: rgb(149, 79, 114); text-decoration: underline;">mailto:lustre-devel-bounces@lists.lustre.org</a>]<span class="Apple-converted-space"> </span><b class="">On
Behalf Of<span class="Apple-converted-space"> </span></b>Patrick Farrell<br class="">
<b class="">Sent:</b><span class="Apple-converted-space"> </span>Monday, November 2, 2015 11:13 AM<br class="">
<b class="">To:</b><span class="Apple-converted-space"> </span><a class="moz-txt-link-abbreviated" href="mailto:lustre-devel@lists.lustre.org" style="color: rgb(149, 79, 114); text-decoration: underline;">lustre-devel@lists.lustre.org</a><br class="">
<b class="">Subject:</b><span class="Apple-converted-space"> </span>Re: [lustre-devel] porting Lustre server to RHEL6.7 distro running a
<a href="http://kernel.org" class="">kernel.org</a> linux-3.12.39<o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""> </o:p></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
Stan,<br class="">
<br class="">
Why not work with the CentOS 7 kernel on the server anyway? Is something your LND needs not available? CentOS 7 is freely available and isn't far off in terms of kernel version from 3.12.<span style="color: rgb(31, 73, 125);" class=""><o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="color: rgb(31, 73, 125);" class=""> </span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="color: rgb(31, 73, 125);" class="">Yes –I’m somewhat stuck on RHEL 6.x series as I use warewulf as a cluster provisioning system (diskless compute nodes), so far have not found a functional warewulf for RHEL7.<o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="color: rgb(31, 73, 125);" class="">I did choose linux-3.12.39 as a kernel base believing it to be somewhat close to RHEL7; although I’m finding this to less so than hoped for….<o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="color: rgb(31, 73, 125);" class="">Additionally the RD drivers I’m using have not been ported to RHEL7;<o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="color: rgb(31, 73, 125);" class="">In examining lustre-2_7.62 I found support for sles12 based on 3.12.39 kernel.; thinking the lustre patches would be close…<o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="color: rgb(31, 73, 125);" class="">So now I’m wedged between RHEL7 ldiskfs patches and SLES12 ldiskfs patches – figure out distro specific enhancements vs lustre enhancements.</span><br class="">
<br class="">
You may be hard pressed to get useful advice on changing the ldiskfs patch series, that has the potential to be a pretty big job.<span style="color: rgb(31, 73, 125);" class=""><o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="color: rgb(31, 73, 125);" class=""> </span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="color: rgb(31, 73, 125);" class="">I’m starting to understand the magnitude of the ldiskfs task…….<o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="color: rgb(31, 73, 125);" class=""> </span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="color: rgb(31, 73, 125);" class="">I’m ignorant about zfs*, would zfs be a better (read that as path-of-least-resistance) approach ? What are the zfs pitfalls?<o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="color: rgb(31, 73, 125);" class=""> </span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="color: rgb(31, 73, 125);" class="">Thanks,<o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="color: rgb(31, 73, 125);" class=""> </span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="color: rgb(31, 73, 125);" class="">Stan.<o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<br class="">
<br class="">
Regards,<br class="">
- Patrick Farrell<span style="font-size: 12pt;" class=""><o:p class=""></o:p></span></div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
On 11/02/2015 01:05 PM, Smith, Stan wrote:<o:p class=""></o:p></div>
</div>
<blockquote style="margin-top: 5pt; margin-bottom: 5pt;" class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 12pt;" class="">Hello,</span><o:p class=""></o:p></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 12pt;" class=""> I’m currently unable to search the lustre-devel archives + 1<sup class="">st</sup><span class="Apple-converted-space"> </span>post to this list.</span><o:p class=""></o:p></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 12pt;" class=""> </span><o:p class=""></o:p></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 12pt;" class="">Having read lustre documentation I find only references to building lustre server for stock ‘distro’ kernels.</span><o:p class=""></o:p></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 12pt;" class=""> </span><o:p class=""></o:p></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 12pt;" class="">Does someone have suggestions on how to patch ldiskfs for a
<a href="http://kernel.org" class="">kernel.org</a> kernel…..specifically weeding out RHEL/SLES/* distro FS enhancements from a
<a href="http://kernel.org" class="">kernel.org</a> base?</span><o:p class=""></o:p></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 12pt;" class=""> </span><o:p class=""></o:p></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 12pt;" class="">I have written a Reliable Datagram LND which passes the LNet selftests (client lustre build) now entering the world of lustre server building.</span><o:p class=""></o:p></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 12pt;" class="">Patching the <a href="http://kernel.org" class="">
kernel.org</a> kernel for lustre server was fairly straightforward, creating the patch series for ldiskfs is another story.</span><o:p class=""></o:p></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 12pt;" class=""> </span><o:p class=""></o:p></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 12pt;" class="">Thanks,</span><o:p class=""></o:p></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 12pt;" class=""> </span><o:p class=""></o:p></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 12pt;" class="">Stan.</span><o:p class=""></o:p></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 12pt;" class=""><br class="">
<br class="">
<br class="">
<o:p class=""></o:p></span></div>
<pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: 'Courier New';" class="">_______________________________________________<o:p class=""></o:p></pre>
<pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: 'Courier New';" class="">lustre-devel mailing list<o:p class=""></o:p></pre>
<pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: 'Courier New';" class=""><a moz-do-not-send="true" href="mailto:lustre-devel@lists.lustre.org" style="color: rgb(149, 79, 114); text-decoration: underline;" class="">lustre-devel@lists.lustre.org</a><o:p class=""></o:p></pre>
<pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: 'Courier New';" class=""><a moz-do-not-send="true" href="http://lists.lustre.org/listinfo.cgi/lustre-devel-lustre.org" style="color: rgb(149, 79, 114); text-decoration: underline;" class="">http://lists.lustre.org/listinfo.cgi/lustre-devel-lustre.org</a><o:p class=""></o:p></pre>
</blockquote>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 12pt;" class=""> </span></div>
</div>
</blockquote>
<br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255);" class="">
<span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255); float: none; display: inline !important;" class="">_______________________________________________</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255);" class="">
<span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255); float: none; display: inline !important;" class="">lustre-devel
mailing list</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255);" class="">
<span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255); float: none; display: inline !important;" class=""><a href="mailto:lustre-devel@lists.lustre.org" class="">lustre-devel@lists.lustre.org</a></span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255);" class="">
<span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255); float: none; display: inline !important;" class=""><a href="http://lists.lustre.org/listinfo.cgi/lustre-devel-lustre.org" class="">http://lists.lustre.org/listinfo.cgi/lustre-devel-lustre.org</a></span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255);" class="">
</div>
</blockquote>
</div>
<br class="">
</body>
</html>