[lustre-discuss] Lustre on native ZFS encryption

[Mark Miller]

Hi All,

I’m curious if anyone has gotten Lustre working on top of native ZFS encryption.  I realize I’m stretching the bounds of compatibility, but I’m wondering if someone has gotten it to work.  I assumed that Lustre would just sit on top of the encryption layer of ZFS, but
it doesn’t seem to work for me.  I’m able to run the “mkfs.lustre” with the ZFS encryption options added to the mkfs, but when I try to mount the OST, the mount command hangs.

What does work:
- I can create Lustre OSTs without encryption options, and the OSTs gets created and can be mounted as expected.
- I can create encrypted ZFS filesystems, and the ZFS filesystem works as expected.
- I can use LUKS to create encrypted devices, build a zpool on top of those LUKS encrypted devices, then run “mkfs.lustre” (without encryption options), and the Lustre filesystem mounts and works as expected.

What doesn’t work:
- Creating a Lustre filesystem with encryption options (mkfs.lustre ... --mkfsoptions="encryption=on ...”), and then mounting the Lustre filesystem.
- Building a zpool with encryption on the pool, running mkfs.lustre on top of the encrypted zpool.
- I also tried building an MDT with encryption, but that also hung while trying to mount the filesystem.

I’m on Centos 7.4 (3.10.0-693.21.1.el7.x86_64), and using zfs and spl compiled from https://github.com/zfsonlinux (which has the encryption code from Tom Caputi built into it).  For the Lustre versions, I’ve tried using the version 2.10.3 rpms from https://downloads.hpdd.intel.com/public/lustre/latest-release/el7/server as well as the 2.11.50_74 version compiled from git://git.hpdd.intel.com/fs/lustre-release.git, and the results are the same on both versions of Lustre.


Here is an example of what happens when I try to use Lustre with ZFS encryption:

[root at oss1 lustre-release]# zpool create ost02-pool raidz2 disk0 disk1 disk2 disk3
[root at oss1 lustre-release]# mkfs.lustre --ost --backfstype=zfs --fsname=lustre01 --index=3 --mgsnode=192.168.56.131 at tcp1 --mkfsoptions="encryption=on -o keyformat=passphrase -o keylocation=file:///tmp/key" --servicenode=192.168.56.121 at tcp1 ost02-pool/ost3

   Permanent disk data:
Target:     lustre01:OST0003
Index:      3
Lustre FS:  lustre01
Mount type: zfs
Flags:      0x1062
              (OST first_time update no_primnode )
Persistent mount opts:
Parameters: mgsnode=192.168.56.131 at tcp1 failover.node=192.168.56.121 at tcp1
checking for existing Lustre data: not found
mkfs_cmd = zfs create -o canmount=off  -o encryption=on -o keyformat=passphrase -o keylocation=file:///tmp/key ost02-pool/ost3
  xattr=sa
  dnodesize=auto
  recordsize=1M
Writing ost02-pool/ost3 properties
  lustre:mgsnode=192.168.56.131 at tcp1
  lustre:failover.node=192.168.56.121 at tcp1
  lustre:version=1
  lustre:flags=4194
  lustre:index=3
  lustre:fsname=lustre01
  lustre:svname=lustre01:OST0003


But then when I run:

[root at oss1 lustre-release]# mount -t lustre ost02-pool/ost3 /mnt/lustre/local/oss03

the command hangs. After I reboot and do an strace of the mount command, it hangs at:

. . .

ioctl(3, _IOC(0, 0x5a, 0x16, 0x00), 0x7ffe87468b70) = 0
ioctl(3, _IOC(0, 0x5a, 0x12, 0x00), 0x7ffe87465550) = 0
ioctl(3, _IOC(0, 0x5a, 0x16, 0x00), 0x7ffe87468bd0) = 0
ioctl(3, _IOC(0, 0x5a, 0x12, 0x00), 0x7ffe874655b0) = 0
ioctl(3, _IOC(0, 0x5a, 0x16, 0x00), 0x7ffe87468bd0) = 0
ioctl(3, _IOC(0, 0x5a, 0x12, 0x00), 0x7ffe874655b0) = 0
mount("ost02-pool/ost3", "/mnt/lustre/local/oss03", "lustre", MS_STRICTATIME, "osd=osd-zfs,mgsnode=192.168.56.131 at tcp1,virgin,update,noprimnode,param=mgsnode=192.168.56.131 at tcp1,param=failover.node=192.168.56.121 at tcp1,svname=lustre01-OST0003,device=ost02-pool/ost3"



The LUKS encryption does work to provide encryption at rest, but there would be benefits to having native ZFS encryption working.  Thanks for any insights, and thanks for all of the work that this community has done on Lustre and ZFS.

Mark
--
Mark Miller – JHPCE Cluster Technology Manager
Johns Hopkins Bloomberg School of Public Health
Office E2530, 615 N. Wolfe St., Baltimore, MD, 21205
443-287-2774 |  https://jhpce.jhu.edu/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20180501/88f50631/attachment-0001.html>