<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:x="urn:schemas-microsoft-com:office:excel" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns:mv="http://macVmlSchemaUri" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Title" content="">
<meta name="Keywords" content="">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal;
font-family:"Consolas",sans-serif;
color:windowtext;
mso-ligatures:none;}
span.EmailStyle18
{mso-style-type:personal-compose;
font-family:"Consolas",sans-serif;
color:windowtext;
mso-ligatures:none;}
span.msoIns
{mso-style-type:export-only;
mso-style-name:"";
text-decoration:underline;
color:teal;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;
font-family:"Calibri",sans-serif;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1027"/>
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1"/>
</o:shapelayout></xml><![endif]-->
</head>
<body bgcolor="white" lang="EN-US" link="#0563C1" vlink="#954F72">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Consolas",sans-serif">Hi All,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Consolas",sans-serif"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Consolas",sans-serif">I\u2019m curious if anyone has gotten Lustre working on top of native ZFS encryption. I realize I\u2019m stretching the bounds of compatibility, but I\u2019m wondering if someone has gotten
it to work. I assumed that Lustre would just sit on top of the encryption layer of ZFS, but
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Consolas",sans-serif">it doesn\u2019t seem to work for me. I\u2019m able to run the \u201cmkfs.lustre\u201d with the ZFS encryption options added to the mkfs, but when I try to mount the OST, the mount command hangs.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Consolas",sans-serif"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Consolas",sans-serif">What does work:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Consolas",sans-serif">- I can create Lustre OSTs without encryption options, and the OSTs gets created and can be mounted as expected.
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Consolas",sans-serif">- I can create encrypted ZFS filesystems, and the ZFS filesystem works as expected.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Consolas",sans-serif">- I can use LUKS to create encrypted devices, build a zpool on top of those LUKS encrypted devices, then run \u201cmkfs.lustre\u201d (without encryption options), and the Lustre filesystem
mounts and works as expected.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Consolas",sans-serif"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Consolas",sans-serif">What doesn\u2019t work:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Consolas",sans-serif">- Creating a Lustre filesystem with encryption options (mkfs.lustre ... --mkfsoptions="encryption=on ...\u201d), and then mounting the Lustre filesystem.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Consolas",sans-serif">- Building a zpool with encryption on the pool, running mkfs.lustre on top of the encrypted zpool.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Consolas",sans-serif">- I also tried building an MDT with encryption, but that also hung while trying to mount the filesystem.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Consolas",sans-serif"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Consolas",sans-serif">I\u2019m on Centos 7.4 (3.10.0-693.21.1.el7.x86_64), and using zfs and spl compiled from
<a href="https://github.com/zfsonlinux">https://github.com/zfsonlinux</a> (which has the encryption code from Tom Caputi built into it). For the Lustre versions, I\u2019ve tried using the version 2.10.3 rpms from
<a href="https://downloads.hpdd.intel.com/public/lustre/latest-release/el7/server">
https://downloads.hpdd.intel.com/public/lustre/latest-release/el7/server</a> as well as the 2.11.50_74 version compiled from git://git.hpdd.intel.com/fs/lustre-release.git, and the results are the same on both versions of Lustre.<o:p></o:p></span></p>
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Consolas",sans-serif"><o:p> </o:p></span></b></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Consolas",sans-serif"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Consolas",sans-serif">Here is an example of what happens when I try to use Lustre with ZFS encryption:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Consolas",sans-serif"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Consolas",sans-serif">[root@oss1 lustre-release]# zpool</span><span style="font-size:10.0pt;font-family:"Consolas",sans-serif"> create ost02-pool raidz2 disk0 disk1 disk2 disk3<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Consolas",sans-serif">[root@oss1 lustre-release]# mkfs.lustre --ost --backfstype=zfs --fsname=lustre01 --index=3 --mgsnode=192.168.56.131@tcp1 --mkfsoptions="encryption=on -o keyformat=passphrase
-o keylocation=file:///tmp/key" --servicenode=192.168.56.121@tcp1 ost02-pool/ost3<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Consolas",sans-serif"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Consolas",sans-serif"> Permanent disk data:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Consolas",sans-serif">Target: lustre01:OST0003<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Consolas",sans-serif">Index: 3<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Consolas",sans-serif">Lustre FS: lustre01<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Consolas",sans-serif">Mount type: zfs<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Consolas",sans-serif">Flags: 0x1062<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Consolas",sans-serif"> (OST first_time update no_primnode )<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Consolas",sans-serif">Persistent mount opts: <o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Consolas",sans-serif">Parameters: mgsnode=192.168.56.131@tcp1 failover.node=192.168.56.121@tcp1<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Consolas",sans-serif">checking for existing Lustre data: not found<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Consolas",sans-serif">mkfs_cmd = zfs create -o canmount=off -o encryption=on -o keyformat=passphrase -o keylocation=file:///tmp/key ost02-pool/ost3<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Consolas",sans-serif"> xattr=sa<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Consolas",sans-serif"> dnodesize=auto<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Consolas",sans-serif"> recordsize=1M<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Consolas",sans-serif">Writing ost02-pool/ost3 properties<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Consolas",sans-serif"> lustre:mgsnode=192.168.56.131@tcp1<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Consolas",sans-serif"> lustre:failover.node=192.168.56.121@tcp1<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Consolas",sans-serif"> lustre:version=1<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Consolas",sans-serif"> lustre:flags=4194<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Consolas",sans-serif"> lustre:index=3<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Consolas",sans-serif"> lustre:fsname=lustre01<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Consolas",sans-serif"> lustre:svname=lustre01:OST0003<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Consolas",sans-serif"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Consolas",sans-serif"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Consolas",sans-serif">But then when I run:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Consolas",sans-serif"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Consolas",sans-serif">[root@oss1 lustre-release]# mount -t lustre ost02-pool/ost3 /mnt/lustre/local/oss03<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Consolas",sans-serif"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Consolas",sans-serif">the command hangs. After I reboot and do an strace of the mount command, it hangs at:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Consolas",sans-serif"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Consolas",sans-serif">. . .<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Consolas",sans-serif"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Consolas",sans-serif">ioctl(3, _IOC(0, 0x5a, 0x16, 0x00), 0x7ffe87468b70) = 0<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Consolas",sans-serif">ioctl(3, _IOC(0, 0x5a, 0x12, 0x00), 0x7ffe87465550) = 0<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Consolas",sans-serif">ioctl(3, _IOC(0, 0x5a, 0x16, 0x00), 0x7ffe87468bd0) = 0<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Consolas",sans-serif">ioctl(3, _IOC(0, 0x5a, 0x12, 0x00), 0x7ffe874655b0) = 0<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Consolas",sans-serif">ioctl(3, _IOC(0, 0x5a, 0x16, 0x00), 0x7ffe87468bd0) = 0<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Consolas",sans-serif">ioctl(3, _IOC(0, 0x5a, 0x12, 0x00), 0x7ffe874655b0) = 0<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Consolas",sans-serif">mount("ost02-pool/ost3", "/mnt/lustre/local/oss03", "lustre", MS_STRICTATIME, "osd=osd-zfs,mgsnode=192.168.56.131@tcp1,virgin,update,noprimnode,param=mgsnode=192.168.56.131@tcp1,param=failover.node=192.168.56.121@tcp1,svname=lustre01-OST0003,device=ost02-pool/ost3"<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Consolas",sans-serif"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Consolas",sans-serif"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Consolas",sans-serif"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Consolas",sans-serif">The LUKS encryption does work to provide encryption at rest, but there would be benefits to having native ZFS encryption working. Thanks for any insights, and thanks for
all of the work that this community has done on Lustre and ZFS.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Consolas",sans-serif"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Consolas",sans-serif;color:black;mso-contextual-alternates:yes">Mark<o:p></o:p></span></p>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Consolas",sans-serif;color:black;mso-contextual-alternates:yes">--<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Consolas",sans-serif;color:black;mso-contextual-alternates:yes">Mark Miller \u2013 JHPCE Cluster Technology Manager<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Consolas",sans-serif;color:black;mso-contextual-alternates:yes">Johns Hopkins Bloomberg School of Public Health<br>
Office E2530, 615 N. Wolfe St., Baltimore, MD, 21205<o:p></o:p></span></p>
</div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Consolas",sans-serif;color:black;mso-contextual-alternates:yes">443-287-2774 | </span><a href="https://jhpce.jhu.edu"><span style="font-size:10.0pt;font-family:"Consolas",sans-serif;color:purple;mso-contextual-alternates:yes">https://jhpce.jhu.edu</span></a><span style="font-size:10.0pt;color:black;mso-contextual-alternates:yes">/</span><o:p></o:p></p>
</div>
</body>
</html>