[lustre-discuss] SSK configuration

Jeremy Filizetti jeremy.filizetti at gmail.com
Sun Jun 24 13:02:11 PDT 2018


GSS error 0x60000 is GSS bad signature which would mean the HMAC was
invalid.  Can you verify your key file's have the same shared key?  Do you
have any logs for the server side as well?  You can increase server
verbosity by adding some extra v's to LSVCGSSDARGS in
/etc/sysconfig/lsvcgss.

Jeremy

On Fri, Jun 22, 2018 at 3:41 PM, Mark Roper <markroper at gmail.com> wrote:

> Hi Lustre Admins,
>
> I am hoping someone can help me understand what I'm doing wrong with SSK
> setup. I have set up a lustre 2.11 server and worked through the steps to
> use shared secret keys (SSKs) to encrypt data in transit between client
> nodes and the MDT and OSS.  I followed the manual instructions here:
> http://doc.lustre.org/lustre_manual.xhtml#idm140687075065344
>
> Before enabling the encryption settings on the MDT, I can mount the FS on
> the client node.  After I turn on the encryption I get back an encryption
> refused error and cannot mount:
>
> mount.lustre: mount 172.31.46.245 at tcp:/scratch at /scratch failed:
> Connection refused
>
> The keys are definitely distributed to client nodes and server nodes and
> the settings have all been made as instruct4red in the manual (I did this a
> few times from scratch to make sure).  I can manually load the keys into
> the keyring and see them by running `keyctl show`, I can compare the key
> files on client and server nodes with the command `lgss_sk --read
> /secure_directory/scratch.client.key` and validate that they all match
> and that the client has a prime.
>
> The commands I'm using to enable the encryption are:
>
>   mdt# sudo lctl conf_param scratch.srpc.flavor.tcp.cli2mdt=skpi
>   mdt# sudo lctl conf_param scratch.srpc.flavor.tcp.cli2ost=skpi
> I tried tailing /var/log/messages and am not able to interpret the output,
> I'm wondering - does anyone have a hypothesis about what might be wrong or
> instructions to debug?
>
> Log output is below!  Many thanks to anyone who can help!
>
> Mark
>
> Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22250]:TRACE:main(): start
> parsing parameters
> Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22250]:INFO:main(): key
> 428863463, desc 0 at 26, ugid 0:0, sring 46159405, coinfo 38:sk:0:0:m:p:2:
> 0x20000ac1f2109:scratch-OST1cd0-osc-MDT0000:0x20000ac1f2ef5:1
> Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22250]:TRACE:parse_callout_info():
> components: 38,sk,0,0,m,p,2,0x20000ac1f2109,scratch-OST1cd0-osc-MDT0000,
> 0x20000ac1f2ef5,1
> Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22250]:DEBUG:parse_callout_info():
> parse call out info: secid 38, mech sk, ugid 0:0, is_root 0, is_mdt 1,
> is_ost 0, svc type p, svc 2, nid 0x20000ac1f2109, tgt
> scratch-OST1cd0-osc-MDT0000, self nid 0x20000ac1f2ef5, pid 1
> Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22250]:TRACE:main():
> parsing parameters OK
> Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22250]:TRACE:lgss_mech_initialize():
> initialize mech sk
> Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22250]:TRACE:lgss_create_cred():
> create a sk cred at 0x1ecc2e0
> Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22250]:TRACE:main():
> caller's namespace is the same
> Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22250]:TRACE:lgss_prepare_cred():
> preparing sk cred 0x1ecc2e0
> Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring:
> [22250]:INFO:sk_create_cred(): Creating credentials for target:
> scratch-OST1cd0-osc-MDT0000 with nodemap: (null)
> Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring:
> [22250]:INFO:sk_create_cred(): Searching for key with description:
> lustre:scratch
> Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22250]:TRACE:prepare_and_instantiate():
> instantiated kernel key 198fefe7
> Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22250]:TRACE:main():
> forked child 22251
> Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22251]:TRACE:lgssc_kr_negotiate():
> child start on behalf of key 198fefe7: cred 0x1ecc2e0, uid 0, svc 2, nid
> 20000ac1f2109, uids: 0:0/0:0
> Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22251]:INFO:ipv4_nid2hostname():
> SOCKLND: net 0x20000, addr 0x9211fac => ip-172-31-33-9.us-west-2.
> compute.internal
> Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22251]:DEBUG:lgss_get_service_str():
> constructed service string: lustre_oss at ip-172-31-33-9.us-
> west-2.compute.internal
> Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring:
> [22251]:TRACE:lgss_using_cred(): using sk cred 0x1ecc2e0
> Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22253]:TRACE:main(): start
> parsing parameters
> Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22253]:INFO:main(): key
> 189483693, desc 0 at 25, ugid 0:0, sring 46159405, coinfo 37:sk:0:0:m:p:2:
> 0x20000ac1f2687:scratch-OST2b9d-osc-MDT0000:0x20000ac1f2ef5:1
> Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22253]:TRACE:parse_callout_info():
> components: 37,sk,0,0,m,p,2,0x20000ac1f2687,scratch-OST2b9d-osc-MDT0000,
> 0x20000ac1f2ef5,1
> Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22253]:DEBUG:parse_callout_info():
> parse call out info: secid 37, mech sk, ugid 0:0, is_root 0, is_mdt 1,
> is_ost 0, svc type p, svc 2, nid 0x20000ac1f2687, tgt
> scratch-OST2b9d-osc-MDT0000, self nid 0x20000ac1f2ef5, pid 1
> Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22253]:TRACE:main():
> parsing parameters OK
> Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22253]:TRACE:lgss_mech_initialize():
> initialize mech sk
> Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22253]:TRACE:lgss_create_cred():
> create a sk cred at 0x21b02e0
> Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22253]:TRACE:main():
> caller's namespace is the same
> Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22253]:TRACE:lgss_prepare_cred():
> preparing sk cred 0x21b02e0
> Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring:
> [22253]:INFO:sk_create_cred(): Creating credentials for target:
> scratch-OST2b9d-osc-MDT0000 with nodemap: (null)
> Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring:
> [22253]:INFO:sk_create_cred(): Searching for key with description:
> lustre:scratch
> Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22253]:TRACE:prepare_and_instantiate():
> instantiated kernel key 0b4b4aad
> Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22253]:TRACE:main():
> forked child 22254
> Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22254]:TRACE:lgssc_kr_negotiate():
> child start on behalf of key 0b4b4aad: cred 0x21b02e0, uid 0, svc 2, nid
> 20000ac1f2687, uids: 0:0/0:0
> Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22254]:INFO:ipv4_nid2hostname():
> SOCKLND: net 0x20000, addr 0x87261fac => ip-172-31-38-135.us-west-2.
> compute.internal
> Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22254]:DEBUG:lgss_get_service_str():
> constructed service string: lustre_oss at ip-172-31-38-135.
> us-west-2.compute.internal
> Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring:
> [22254]:TRACE:lgss_using_cred(): using sk cred 0x21b02e0
> Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22251]:INFO:sk_encode_netstring():
> Encoded netstring of 647 bytes
> Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22251]:INFO:lgss_sk_using_cred():
> Created netstring of 647 bytes
> Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22251]:TRACE:lgssc_negotiation_manual():
> starting gss negotation
> Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring:
> [22251]:TRACE:do_nego_rpc(): start negotiation rpc
> Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring:
> [22251]:TRACE:gss_do_ioctl(): to open /proc/fs/lustre/sptlrpc/gss/
> init_channel
> Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring:
> [22251]:TRACE:gss_do_ioctl(): to down-write
> Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22254]:INFO:sk_encode_netstring():
> Encoded netstring of 647 bytes
> Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22254]:INFO:lgss_sk_using_cred():
> Created netstring of 647 bytes
> Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22254]:TRACE:lgssc_negotiation_manual():
> starting gss negotation
> Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring:
> [22254]:TRACE:do_nego_rpc(): start negotiation rpc
> Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring:
> [22254]:TRACE:gss_do_ioctl(): to open /proc/fs/lustre/sptlrpc/gss/
> init_channel
> Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring:
> [22254]:TRACE:gss_do_ioctl(): to down-write
> Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring:
> [22251]:TRACE:do_nego_rpc(): do_nego_rpc: to parse reply
> Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring:
> [22251]:DEBUG:do_nego_rpc(): do_nego_rpc: receive handle len 0, token len
> 0, res 0
> Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22251]:ERROR:lgssc_negotiation_manual():
> negotiation gss error 60000
> Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22251]:ERROR:lgssc_kr_negotiate_manual():
> key 198fefe7: failed to negotiate
> Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22251]:TRACE:error_kernel_key():
> revoking kernel key 198fefe7
> Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring:
> [22251]:INFO:error_kernel_key(): key 198fefe7: revoked
> Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22251]:TRACE:lgss_release_cred():
> releasing sk cred 0x1ecc2e0
> Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring:
> [22254]:TRACE:do_nego_rpc(): do_nego_rpc: to parse reply
> Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring:
> [22254]:DEBUG:do_nego_rpc(): do_nego_rpc: receive handle len 0, token len
> 0, res 0
> Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22254]:ERROR:lgssc_negotiation_manual():
> negotiation gss error 60000
> Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22254]:ERROR:lgssc_kr_negotiate_manual():
> key 0b4b4aad: failed to negotiate
> Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22254]:TRACE:error_kernel_key():
> revoking kernel key 0b4b4aad
> Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring:
> [22254]:INFO:error_kernel_key(): key 0b4b4aad: revoked
> Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22254]:TRACE:lgss_release_cred():
> releasing sk cred 0x21b02e0
>
> _______________________________________________
> lustre-discuss mailing list
> lustre-discuss at lists.lustre.org
> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20180624/f2abd852/attachment-0001.html>


More information about the lustre-discuss mailing list