[lustre-discuss] SSK configuration

Fri Jun 22 12:41:00 PDT 2018

Hi Lustre Admins,

I am hoping someone can help me understand what I'm doing wrong with SSK
setup. I have set up a lustre 2.11 server and worked through the steps to
use shared secret keys (SSKs) to encrypt data in transit between client
nodes and the MDT and OSS.  I followed the manual instructions here:
http://doc.lustre.org/lustre_manual.xhtml#idm140687075065344

Before enabling the encryption settings on the MDT, I can mount the FS on
the client node.  After I turn on the encryption I get back an encryption
refused error and cannot mount:

mount.lustre: mount 172.31.46.245 at tcp:/scratch at /scratch failed:
Connection refused

The keys are definitely distributed to client nodes and server nodes and
the settings have all been made as instruct4red in the manual (I did this a
few times from scratch to make sure).  I can manually load the keys into
the keyring and see them by running `keyctl show`, I can compare the key
files on client and server nodes with the command `lgss_sk --read
/secure_directory/scratch.client.key` and validate that they all match and
that the client has a prime.

The commands I'm using to enable the encryption are:

  mdt# sudo lctl conf_param scratch.srpc.flavor.tcp.cli2mdt=skpi
  mdt# sudo lctl conf_param scratch.srpc.flavor.tcp.cli2ost=skpi
I tried tailing /var/log/messages and am not able to interpret the output,
I'm wondering - does anyone have a hypothesis about what might be wrong or
instructions to debug?

Log output is below!  Many thanks to anyone who can help!

Mark

Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22250]:TRACE:main(): start
parsing parameters
Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22250]:INFO:main(): key
428863463, desc 0 at 26, ugid 0:0, sring 46159405, coinfo
38:sk:0:0:m:p:2:0x20000ac1f2109:scratch-OST1cd0-osc-MDT0000:0x20000ac1f2ef5:1
Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring:
[22250]:TRACE:parse_callout_info(): components:
38,sk,0,0,m,p,2,0x20000ac1f2109,scratch-OST1cd0-osc-MDT0000,0x20000ac1f2ef5,1
Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring:
[22250]:DEBUG:parse_callout_info(): parse call out info: secid 38, mech sk,
ugid 0:0, is_root 0, is_mdt 1, is_ost 0, svc type p, svc 2, nid
0x20000ac1f2109, tgt scratch-OST1cd0-osc-MDT0000, self nid 0x20000ac1f2ef5,
pid 1
Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22250]:TRACE:main():
parsing parameters OK
Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring:
[22250]:TRACE:lgss_mech_initialize(): initialize mech sk
Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring:
[22250]:TRACE:lgss_create_cred(): create a sk cred at 0x1ecc2e0
Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22250]:TRACE:main():
caller's namespace is the same
Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring:
[22250]:TRACE:lgss_prepare_cred(): preparing sk cred 0x1ecc2e0
Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring:
[22250]:INFO:sk_create_cred(): Creating credentials for target:
scratch-OST1cd0-osc-MDT0000 with nodemap: (null)
Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring:
[22250]:INFO:sk_create_cred(): Searching for key with description:
lustre:scratch
Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring:
[22250]:TRACE:prepare_and_instantiate(): instantiated kernel key 198fefe7
Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22250]:TRACE:main(): forked
child 22251
Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring:
[22251]:TRACE:lgssc_kr_negotiate(): child start on behalf of key 198fefe7:
cred 0x1ecc2e0, uid 0, svc 2, nid 20000ac1f2109, uids: 0:0/0:0
Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring:
[22251]:INFO:ipv4_nid2hostname(): SOCKLND: net 0x20000, addr 0x9211fac =>
ip-172-31-33-9.us-west-2.compute.internal
Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring:
[22251]:DEBUG:lgss_get_service_str(): constructed service string:
lustre_oss at ip-172-31-33-9.us-west-2.compute.internal
Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring:
[22251]:TRACE:lgss_using_cred(): using sk cred 0x1ecc2e0
Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22253]:TRACE:main(): start
parsing parameters
Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22253]:INFO:main(): key
189483693, desc 0 at 25, ugid 0:0, sring 46159405, coinfo
37:sk:0:0:m:p:2:0x20000ac1f2687:scratch-OST2b9d-osc-MDT0000:0x20000ac1f2ef5:1
Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring:
[22253]:TRACE:parse_callout_info(): components:
37,sk,0,0,m,p,2,0x20000ac1f2687,scratch-OST2b9d-osc-MDT0000,0x20000ac1f2ef5,1
Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring:
[22253]:DEBUG:parse_callout_info(): parse call out info: secid 37, mech sk,
ugid 0:0, is_root 0, is_mdt 1, is_ost 0, svc type p, svc 2, nid
0x20000ac1f2687, tgt scratch-OST2b9d-osc-MDT0000, self nid 0x20000ac1f2ef5,
pid 1
Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22253]:TRACE:main():
parsing parameters OK
Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring:
[22253]:TRACE:lgss_mech_initialize(): initialize mech sk
Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring:
[22253]:TRACE:lgss_create_cred(): create a sk cred at 0x21b02e0
Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22253]:TRACE:main():
caller's namespace is the same
Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring:
[22253]:TRACE:lgss_prepare_cred(): preparing sk cred 0x21b02e0
Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring:
[22253]:INFO:sk_create_cred(): Creating credentials for target:
scratch-OST2b9d-osc-MDT0000 with nodemap: (null)
Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring:
[22253]:INFO:sk_create_cred(): Searching for key with description:
lustre:scratch
Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring:
[22253]:TRACE:prepare_and_instantiate(): instantiated kernel key 0b4b4aad
Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22253]:TRACE:main(): forked
child 22254
Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring:
[22254]:TRACE:lgssc_kr_negotiate(): child start on behalf of key 0b4b4aad:
cred 0x21b02e0, uid 0, svc 2, nid 20000ac1f2687, uids: 0:0/0:0
Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring:
[22254]:INFO:ipv4_nid2hostname(): SOCKLND: net 0x20000, addr 0x87261fac =>
ip-172-31-38-135.us-west-2.compute.internal
Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring:
[22254]:DEBUG:lgss_get_service_str(): constructed service string:
lustre_oss at ip-172-31-38-135.us-west-2.compute.internal
Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring:
[22254]:TRACE:lgss_using_cred(): using sk cred 0x21b02e0
Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring:
[22251]:INFO:sk_encode_netstring(): Encoded netstring of 647 bytes
Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring:
[22251]:INFO:lgss_sk_using_cred(): Created netstring of 647 bytes
Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring:
[22251]:TRACE:lgssc_negotiation_manual(): starting gss negotation
Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22251]:TRACE:do_nego_rpc():
start negotiation rpc
Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring:
[22251]:TRACE:gss_do_ioctl(): to open
/proc/fs/lustre/sptlrpc/gss/init_channel
Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring:
[22251]:TRACE:gss_do_ioctl(): to down-write
Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring:
[22254]:INFO:sk_encode_netstring(): Encoded netstring of 647 bytes
Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring:
[22254]:INFO:lgss_sk_using_cred(): Created netstring of 647 bytes
Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring:
[22254]:TRACE:lgssc_negotiation_manual(): starting gss negotation
Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22254]:TRACE:do_nego_rpc():
start negotiation rpc
Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring:
[22254]:TRACE:gss_do_ioctl(): to open
/proc/fs/lustre/sptlrpc/gss/init_channel
Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring:
[22254]:TRACE:gss_do_ioctl(): to down-write
Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22251]:TRACE:do_nego_rpc():
do_nego_rpc: to parse reply
Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22251]:DEBUG:do_nego_rpc():
do_nego_rpc: receive handle len 0, token len 0, res 0
Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring:
[22251]:ERROR:lgssc_negotiation_manual(): negotiation gss error 60000
Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring:
[22251]:ERROR:lgssc_kr_negotiate_manual(): key 198fefe7: failed to negotiate
Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring:
[22251]:TRACE:error_kernel_key(): revoking kernel key 198fefe7
Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring:
[22251]:INFO:error_kernel_key(): key 198fefe7: revoked
Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring:
[22251]:TRACE:lgss_release_cred(): releasing sk cred 0x1ecc2e0
Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22254]:TRACE:do_nego_rpc():
do_nego_rpc: to parse reply
Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22254]:DEBUG:do_nego_rpc():
do_nego_rpc: receive handle len 0, token len 0, res 0
Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring:
[22254]:ERROR:lgssc_negotiation_manual(): negotiation gss error 60000
Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring:
[22254]:ERROR:lgssc_kr_negotiate_manual(): key 0b4b4aad: failed to negotiate
Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring:
[22254]:TRACE:error_kernel_key(): revoking kernel key 0b4b4aad
Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring:
[22254]:INFO:error_kernel_key(): key 0b4b4aad: revoked
Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring:
[22254]:TRACE:lgss_release_cred(): releasing sk cred 0x21b02e0
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20180622/98cff815/attachment-0001.html>