[lustre-discuss] Lustre on native ZFS encryption

Dilger, Andreas andreas.dilger at intel.com
Thu May 3 15:32:16 PDT 2018


If your mount is hanging, you could "modprobe libcfs" to load the debugging
infrastructure, then "lctl set_param debug=-1" to enable full debugging before
you try to mount the ZFS filesystem.  This will tell you where the code is
getting stuck.  Also, "echo t > /proc/sysrq-trigger" to dump the kernel
thread stacks would be useful...

It's probably worthwhile to start putting this info into a Jira ticket so
that it is available for future reference.  Since ZFS 0.8 is not released
yet we still have some time to debug this before there are any real users.
I don't recommend anyone to use ZFS 0.8 in production yet, because the disk
formats may still change during development.  However, testing it before
release is definitely valuable so that it works reasonably well _before_ it
is released and doesn't need 5 bugs fixed immediately after the release.

Cheers, Andreas

On May 2, 2018, at 07:59, Mark Miller <mmill116 at jhu.edu> wrote:
> 
> Hi Andreas,
> 
> You are correct in that there is a separate key management step in the zfs encryption, but in the testing I was doing, the key was already loaded.  The key is loaded either a) when the “zfs create” is done, b) when a “zfs mount -l . .” is done, or c) separately via the “zfs load-key ...” command (the benefit of this third method is that key management can be done by a non-root key owner).  In my testing, the key had been loaded when I either created the zfs filesystem via mkfs.lustre, or, after a reboot, by running the “zfs load-key” just prior to the “mount -t lustre ...” command.
> 
> Since I have the Lustre source code, I can start looking through it to see if I can find where the Lustre mount system call may be getting hung up.  I have no idea... but it feels like the Lustre mount may be trying to read something from the zfs filesystem “under the hood”, and it’s getting the result back in its encrypted state.  We always have LUKS encryption to fall back to.
> 
> Mark
> --
> Mark Miller – JHPCE Cluster Technology Manager
> Johns Hopkins Bloomberg School of Public Health
> Office E2530, 615 N. Wolfe St., Baltimore, MD, 21205
> 443-287-2774 |  https://jhpce.jhu.edu/
> 
> On 5/1/18, 7:11 PM, "Dilger, Andreas" <andreas.dilger at intel.com> wrote:
> 
>    On May 1, 2018, at 11:10, Mark Miller <mmill116 at jhu.edu> wrote:
>> 
>> Hi All,
>> 
>> I’m curious if anyone has gotten Lustre working on top of native ZFS encryption. 
> 
>    Yes, I think this would be quite interesting and useful for a number of environments.  Since Lustre already has over-the-network encryption (Kerberos or SSK "privacy" mode), this would provide a reasonably secure solution, at the expense of doing extra crypto operations on the server.  Not quite as awesome as end-to-end encryption handled entirely on the client, but definitely better than no disk encryption at all.
> 
>> I realize I’m stretching the bounds of compatibility, but I’m wondering if someone has gotten it to work.  I assumed that Lustre would just sit on top of the encryption layer of ZFS, but
>> it doesn’t seem to work for me.  I’m able to run the “mkfs.lustre” with the ZFS encryption options added to the mkfs, but when I try to mount the OST, the mount command hangs.
>> 
>> What does work:
>> - I can create Lustre OSTs without encryption options, and the OSTs gets created and can be mounted as expected.
>> - I can create encrypted ZFS filesystems, and the ZFS filesystem works as expected.
>> - I can use LUKS to create encrypted devices, build a zpool on top of those LUKS encrypted devices, then run “mkfs.lustre” (without encryption options), and the Lustre filesystem mounts and works as expected.
>> 
>> What doesn’t work:
>> - Creating a Lustre filesystem with encryption options (mkfs.lustre ... --mkfsoptions="encryption=on ...”), and then mounting the Lustre filesystem.
>> - Building a zpool with encryption on the pool, running mkfs.lustre on top of the encrypted zpool.
>> - I also tried building an MDT with encryption, but that also hung while trying to mount the filesystem.
> 
>    I haven't looked at the ZFS encryption code, but I suspect that the Lustre osd-zfs code would need to pass an encryption key when it opens the pool, or at least before it is accessible.  That is likely what is preventing the filesystem access when Lustre mounts the new MDT.
> 
>    I don't know the mechanism by which the key would be passed to the dataset.  It might be a parameter that is passed to the dataset at open time, or it might be passed out-of-band from userspace with a "zfs" or "zpool" command?
> 
>    Cheers, Andreas
> 
>> I’m on Centos 7.4 (3.10.0-693.21.1.el7.x86_64), and using zfs and spl compiled from https://github.com/zfsonlinux (which has the encryption code from Tom Caputi built into it).  For the Lustre versions, I’ve tried using the version 2.10.3 rpms from https://downloads.hpdd.intel.com/public/lustre/latest-release/el7/server as well as the 2.11.50_74 version compiled from git://git.hpdd.intel.com/fs/lustre-release.git, and the results are the same on both versions of Lustre.
>> 
>> 
>> Here is an example of what happens when I try to use Lustre with ZFS encryption:
>> 
>> [root at oss1 lustre-release]# zpool create ost02-pool raidz2 disk0 disk1 disk2 disk3
>> [root at oss1 lustre-release]# mkfs.lustre --ost --backfstype=zfs --fsname=lustre01 --index=3 --mgsnode=192.168.56.131 at tcp1 --mkfsoptions="encryption=on -o keyformat=passphrase -o keylocation=file:///tmp/key" --servicenode=192.168.56.121 at tcp1 ost02-pool/ost3
>> 
>>   Permanent disk data:
>> Target:     lustre01:OST0003
>> Index:      3
>> Lustre FS:  lustre01
>> Mount type: zfs
>> Flags:      0x1062
>>              (OST first_time update no_primnode )
>> Persistent mount opts: 
>> Parameters: mgsnode=192.168.56.131 at tcp1 failover.node=192.168.56.121 at tcp1
>> checking for existing Lustre data: not found
>> mkfs_cmd = zfs create -o canmount=off  -o encryption=on -o keyformat=passphrase -o keylocation=file:///tmp/key ost02-pool/ost3
>>  xattr=sa
>>  dnodesize=auto
>>  recordsize=1M
>> Writing ost02-pool/ost3 properties
>>  lustre:mgsnode=192.168.56.131 at tcp1
>>  lustre:failover.node=192.168.56.121 at tcp1
>>  lustre:version=1
>>  lustre:flags=4194
>>  lustre:index=3
>>  lustre:fsname=lustre01
>>  lustre:svname=lustre01:OST0003
>> 
>> 
>> But then when I run:
>> 
>> [root at oss1 lustre-release]# mount -t lustre ost02-pool/ost3 /mnt/lustre/local/oss03
>> 
>> the command hangs. After I reboot and do an strace of the mount command, it hangs at:
>> 
>> . . .
>> 
>> ioctl(3, _IOC(0, 0x5a, 0x16, 0x00), 0x7ffe87468b70) = 0
>> ioctl(3, _IOC(0, 0x5a, 0x12, 0x00), 0x7ffe87465550) = 0
>> ioctl(3, _IOC(0, 0x5a, 0x16, 0x00), 0x7ffe87468bd0) = 0
>> ioctl(3, _IOC(0, 0x5a, 0x12, 0x00), 0x7ffe874655b0) = 0
>> ioctl(3, _IOC(0, 0x5a, 0x16, 0x00), 0x7ffe87468bd0) = 0
>> ioctl(3, _IOC(0, 0x5a, 0x12, 0x00), 0x7ffe874655b0) = 0
>> mount("ost02-pool/ost3", "/mnt/lustre/local/oss03", "lustre", MS_STRICTATIME, "osd=osd-zfs,mgsnode=192.168.56.131 at tcp1,virgin,update,noprimnode,param=mgsnode=192.168.56.131 at tcp1,param=failover.node=192.168.56.121 at tcp1,svname=lustre01-OST0003,device=ost02-pool/ost3"
>> 
>> 
>> 
>> The LUKS encryption does work to provide encryption at rest, but there would be benefits to having native ZFS encryption working.  Thanks for any insights, and thanks for all of the work that this community has done on Lustre and ZFS.
>> 
>> Mark
>> --
>> Mark Miller – JHPCE Cluster Technology Manager
>> Johns Hopkins Bloomberg School of Public Health
>> Office E2530, 615 N. Wolfe St., Baltimore, MD, 21205
>> 443-287-2774 |  https://jhpce.jhu.edu/
>> _______________________________________________
>> lustre-discuss mailing list
>> lustre-discuss at lists.lustre.org
>> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
> 
>    Cheers, Andreas
>    --
>    Andreas Dilger
>    Lustre Principal Architect
>    Intel Corporation
> 
> 
> 
> 
> 
> 
> 
> 
> 

Cheers, Andreas
--
Andreas Dilger
Lustre Principal Architect
Intel Corporation









More information about the lustre-discuss mailing list