<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Helvetica;
panose-1:2 11 6 4 2 2 2 2 2 4;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
pre
{mso-style-priority:99;
mso-style-link:"HTML Preformatted Char";
margin:0in;
margin-bottom:.0001pt;
font-size:10.0pt;
font-family:"Courier New";}
span.apple-converted-space
{mso-style-name:apple-converted-space;}
span.HTMLPreformattedChar
{mso-style-name:"HTML Preformatted Char";
mso-style-priority:99;
mso-style-link:"HTML Preformatted";
font-family:Consolas;}
span.EmailStyle20
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> Rick Wagner [mailto:rpwagner@sdsc.edu]
<br>
<b>Sent:</b> Monday, November 2, 2015 12:25 PM<br>
<b>To:</b> Patrick Farrell <paf@cray.com><br>
<b>Cc:</b> Smith, Stan <stan.smith@intel.com>; lustre-devel@lists.lustre.org<br>
<b>Subject:</b> Re: [lustre-devel] porting Lustre server to RHEL6.7 distro running a kernel.org linux-3.12.39<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">We’ve run ZFS-backed Lustre servers using a patchless (i.e., without ldiskfs) Linux 3 kernel under CentOS 6. While I don’t recommend this for everyone I can say that the mechanics of building the kernel modules (ZFS and Lustre) and standing
up the file system all worked well. We’re looking to move back to a stock CentOS 6 kernel, but are sticking with the patchless model. If you’re willing to try ZFS I suspect the build will be fine, and then you should test test test.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">I’ve included the outline of commands we use to build our server RPMs. You’ll want to adjust the versions of SPL, ZFS, & Lustre, along with the --with-linux and --with-linux-config parameters. I apologize for any typos and subsequent frustration.
What level of dependency hell you find yourself in is is largely dependent on your choice of distro.<o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Wow – A great assist! Thank you much!!<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
</div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">And thank you Patrick, Ben & Rick for the timely sage advice on making forward progress in the land of lustre.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Let the ZFS adventure begin……will write if I find work… ;-)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Stan.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">—Rick<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">git clone <a href="https://github.com/zfsonlinux/spl.git">https://github.com/zfsonlinux/spl.git</a><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">cd spl<br>
git checkout spl-0.6.5.3<br>
./autogen.sh<br>
./configure<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">make pkg<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">rpm -ivh *x86_64.rpm<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">cd ../<o:p></o:p></p>
</div>
<div>
<div>
<p class="MsoNormal">git clone <a href="https://github.com/zfsonlinux/zfs.git">https://github.com/zfsonlinux/zfs.git</a><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">cd zfs<br>
git checkout zfs-0.6.5.3<br>
./autogen.sh<br>
./configure<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">make pkg<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">rpm -ivh *x86_64.rpm<o:p></o:p></p>
</div>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">cd ../<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">git clone <a href="http://git.whamcloud.com/fs/lustre-release.git">
http://git.whamcloud.com/fs/lustre-release.git</a><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">cd lustre-release<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">git checkout 2.7.62<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">sh autogen.sh<br>
./configure --with-zfs=/usr/src/zfs-0.6.5.3 --with-spl=/usr/src/spl-0.6.5.3 --with-linux=/usr/src/linux-3.10 --without-ldiskfs --with-linux-config=/boot/config-3.10.73-1.el6.elrepo.x86_64 --with-o2ib=no<br>
make rpms<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<p class="MsoNormal">On Nov 2, 2015, at 11:45 AM, Patrick Farrell <<a href="mailto:paf@cray.com">paf@cray.com</a>> wrote:<o:p></o:p></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Helvetica",sans-serif;background:white">For ZFS:</span><span style="font-size:9.0pt;font-family:"Helvetica",sans-serif"><br>
<br>
<span style="background:white">The pitfall is primarily that it's less familiar than ldiskfs, which basically uses the standard Linux fs tools, like ext. ZFS is its own world, which will probably have a learning curve for someone who has not used it before,
and it must be built for the version of the kernel you're interested in.</span><br>
<br>
<span style="background:white">But, Ben is right, that's likely to be easier than porting ldiskfs.</span><br>
<br>
<span style="background:white">- Patrick</span><br style="orphans: auto;text-align:start;widows: auto;-webkit-text-stroke-width: 0px;word-spacing:0px">
<br>
</span><o:p></o:p></p>
<div>
<p class="MsoNormal" style="background:white"><span style="font-size:9.0pt;font-family:"Helvetica",sans-serif">On 11/02/2015 01:35 PM, Smith, Stan wrote:<o:p></o:p></span></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt;orphans: auto;text-align:start;widows: auto;-webkit-text-stroke-width: 0px;word-spacing:0px">
<div>
<p class="MsoNormal" style="background:white"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span class="apple-converted-space"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> </span></span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">lustre-devel
[<a href="mailto:lustre-devel-bounces@lists.lustre.org"><span style="color:#954F72">mailto:lustre-devel-bounces@lists.lustre.org</span></a>]<span class="apple-converted-space"> </span><b>On Behalf Of<span class="apple-converted-space"> </span></b>Patrick Farrell<br>
<b>Sent:</b><span class="apple-converted-space"> </span>Monday, November 2, 2015 11:13 AM<br>
<b>To:</b><span class="apple-converted-space"> </span><a href="mailto:lustre-devel@lists.lustre.org"><span style="color:#954F72">lustre-devel@lists.lustre.org</span></a><br>
<b>Subject:</b><span class="apple-converted-space"> </span>Re: [lustre-devel] porting Lustre server to RHEL6.7 distro running a
<a href="http://kernel.org">kernel.org</a> linux-3.12.39<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="background:white"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="background:white"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">Stan,<br>
<br>
Why not work with the CentOS 7 kernel on the server anyway? Is something your LND needs not available? CentOS 7 is freely available and isn't far off in terms of kernel version from 3.12.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="background:white"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> </span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="background:white"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Yes –I’m somewhat stuck on RHEL 6.x series as I use warewulf as a cluster provisioning system (diskless compute nodes), so far have not
found a functional warewulf for RHEL7.</span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="background:white"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">I did choose linux-3.12.39 as a kernel base believing it to be somewhat close to RHEL7; although I’m finding this to less so than hoped
for….</span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="background:white"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Additionally the RD drivers I’m using have not been ported to RHEL7;</span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="background:white"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">In examining lustre-2_7.62 I found support for sles12 based on 3.12.39 kernel.; thinking the lustre patches would be close…</span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="background:white"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">So now I’m wedged between RHEL7 ldiskfs patches and SLES12 ldiskfs patches – figure out distro specific enhancements vs lustre enhancements.</span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><br>
<br>
You may be hard pressed to get useful advice on changing the ldiskfs patch series, that has the potential to be a pretty big job.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="background:white"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> </span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="background:white"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">I’m starting to understand the magnitude of the ldiskfs task…….</span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="background:white"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> </span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="background:white"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">I’m ignorant about zfs*, would zfs be a better (read that as path-of-least-resistance) approach ? What are the zfs pitfalls?</span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="background:white"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> </span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="background:white"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Thanks,</span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="background:white"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> </span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="background:white"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Stan.</span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="background:white"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><br>
<br>
Regards,<br>
- Patrick Farrell<o:p></o:p></span></p>
</div>
<div>
<div>
<p class="MsoNormal" style="background:white"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">On 11/02/2015 01:05 PM, Smith, Stan wrote:<o:p></o:p></span></p>
</div>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<p class="MsoNormal" style="background:white"><span style="font-family:"Calibri",sans-serif">Hello,</span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="background:white"><span style="font-family:"Calibri",sans-serif"> I’m currently unable to search the lustre-devel archives + 1<sup>st</sup><span class="apple-converted-space"> </span>post to this list.</span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="background:white"><span style="font-family:"Calibri",sans-serif"> </span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="background:white"><span style="font-family:"Calibri",sans-serif">Having read lustre documentation I find only references to building lustre server for stock ‘distro’ kernels.</span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="background:white"><span style="font-family:"Calibri",sans-serif"> </span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="background:white"><span style="font-family:"Calibri",sans-serif">Does someone have suggestions on how to patch ldiskfs for a
<a href="http://kernel.org">kernel.org</a> kernel…..specifically weeding out RHEL/SLES/* distro FS enhancements from a
<a href="http://kernel.org">kernel.org</a> base?</span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="background:white"><span style="font-family:"Calibri",sans-serif"> </span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="background:white"><span style="font-family:"Calibri",sans-serif">I have written a Reliable Datagram LND which passes the LNet selftests (client lustre build) now entering the world of lustre server building.</span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="background:white"><span style="font-family:"Calibri",sans-serif">Patching the
<a href="http://kernel.org">kernel.org</a> kernel for lustre server was fairly straightforward, creating the patch series for ldiskfs is another story.</span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="background:white"><span style="font-family:"Calibri",sans-serif"> </span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="background:white"><span style="font-family:"Calibri",sans-serif">Thanks,</span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="background:white"><span style="font-family:"Calibri",sans-serif"> </span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="background:white"><span style="font-family:"Calibri",sans-serif">Stan.</span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="background:white"><span style="font-family:"Calibri",sans-serif"><br>
<br>
<br>
<br>
</span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p></o:p></span></p>
</div>
<pre style="background:white">_______________________________________________<o:p></o:p></pre>
<pre style="background:white">lustre-devel mailing list<o:p></o:p></pre>
<pre style="background:white"><a href="mailto:lustre-devel@lists.lustre.org"><span style="color:#954F72">lustre-devel@lists.lustre.org</span></a><o:p></o:p></pre>
<pre style="background:white"><a href="http://lists.lustre.org/listinfo.cgi/lustre-devel-lustre.org"><span style="color:#954F72">http://lists.lustre.org/listinfo.cgi/lustre-devel-lustre.org</span></a><o:p></o:p></pre>
</blockquote>
<div>
<p class="MsoNormal" style="background:white"><span style="font-family:"Calibri",sans-serif"> </span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p></o:p></span></p>
</div>
</blockquote>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Helvetica",sans-serif"><br>
<span style="background:white">_______________________________________________</span><br>
<span style="background:white">lustre-devel mailing list</span><br>
<span style="background:white"><a href="mailto:lustre-devel@lists.lustre.org">lustre-devel@lists.lustre.org</a></span><br>
<span style="background:white"><a href="http://lists.lustre.org/listinfo.cgi/lustre-devel-lustre.org">http://lists.lustre.org/listinfo.cgi/lustre-devel-lustre.org</a></span></span><o:p></o:p></p>
</div>
</blockquote>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</body>
</html>