<div dir="ltr"><div>GSS error 0x60000 is GSS bad signature which would mean the HMAC was invalid. Can you verify your key file's have the same shared key? Do you have any logs for the server side as well? You can increase server verbosity by adding some extra v's to LSVCGSSDARGS in /etc/sysconfig/lsvcgss.<br><br></div>Jeremy<br></div><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Jun 22, 2018 at 3:41 PM, Mark Roper <span dir="ltr"><<a href="mailto:markroper@gmail.com" target="_blank">markroper@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hi Lustre Admins,<div><br></div><div>I am hoping someone can help me understand what I'm doing wrong with SSK setup. I have set up a lustre 2.11 server and worked through the steps to use shared secret keys (SSKs) to encrypt data in transit between client nodes and the MDT and OSS. I followed the manual instructions here: <a href="http://doc.lustre.org/lustre_manual.xhtml#idm140687075065344" target="_blank">http://doc.lustre.org/<wbr>lustre_manual.xhtml#<wbr>idm140687075065344</a></div><div><br></div><div>Before enabling the encryption settings on the MDT, I can mount the FS on the client node. After I turn on the encryption I get back an encryption refused error and cannot mount:</div><div><br></div><div><font size="1">mount.lustre: mount 172.31.46.245@tcp:/scratch at /scratch failed: Connection refused</font><br><p class="m_-15220743430933092inbox-inbox-p1">The keys are definitely distributed to client nodes and server nodes and the settings have all been made as instruct4red in the manual (I did this a few times from scratch to make sure). I can manually load the keys into the keyring and see them by running `keyctl show`, I can compare the key files on client and server nodes with the command `lgss_sk --read /secure_directory/scratch.<wbr>client.key` and validate that they all match and that the client has a prime.</p><p class="m_-15220743430933092inbox-inbox-p1">The commands I'm using to enable the encryption are:</p>
<p class="m_-15220743430933092inbox-inbox-p1"><font size="1"> mdt# sudo lctl conf_param scratch.srpc.flavor.tcp.<wbr>cli2mdt=skpi <br> mdt# sudo lctl conf_param scratch.srpc.flavor.tcp.<wbr>cli2ost=skpi</font></p></div><div>I tried tailing /var/log/messages and am not able to interpret the output, I'm wondering - does anyone have a hypothesis about what might be wrong or instructions to debug? </div><div><br></div><div>Log output is below! Many thanks to anyone who can help!</div><div><br></div><div>Mark</div><div><br></div><div><div><font size="1">Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22250]:TRACE:main(): start parsing parameters</font></div><div><font size="1">Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22250]:INFO:main(): key 428863463, desc 0@26, ugid 0:0, sring 46159405, coinfo 38:sk:0:0:m:p:2:<wbr>0x20000ac1f2109:scratch-<wbr>OST1cd0-osc-MDT0000:<wbr>0x20000ac1f2ef5:1</font></div><div><font size="1">Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22250]:TRACE:parse_callout_<wbr>info(): components: 38,sk,0,0,m,p,2,<wbr>0x20000ac1f2109,scratch-<wbr>OST1cd0-osc-MDT0000,<wbr>0x20000ac1f2ef5,1</font></div><div><font size="1">Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22250]:DEBUG:parse_callout_<wbr>info(): parse call out info: secid 38, mech sk, ugid 0:0, is_root 0, is_mdt 1, is_ost 0, svc type p, svc 2, nid 0x20000ac1f2109, tgt scratch-OST1cd0-osc-MDT0000, self nid 0x20000ac1f2ef5, pid 1</font></div><div><font size="1">Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22250]:TRACE:main(): parsing parameters OK</font></div><div><font size="1">Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22250]:TRACE:lgss_mech_<wbr>initialize(): initialize mech sk</font></div><div><font size="1">Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22250]:TRACE:lgss_create_<wbr>cred(): create a sk cred at 0x1ecc2e0</font></div><div><font size="1">Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22250]:TRACE:main(): caller's namespace is the same</font></div><div><font size="1">Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22250]:TRACE:lgss_prepare_<wbr>cred(): preparing sk cred 0x1ecc2e0</font></div><div><font size="1">Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22250]:INFO:sk_create_cred(): Creating credentials for target: scratch-OST1cd0-osc-MDT0000 with nodemap: (null)</font></div><div><font size="1">Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22250]:INFO:sk_create_cred(): Searching for key with description: lustre:scratch</font></div><div><font size="1">Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22250]:TRACE:prepare_and_<wbr>instantiate(): instantiated kernel key 198fefe7</font></div><div><font size="1">Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22250]:TRACE:main(): forked child 22251</font></div><div><font size="1">Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22251]:TRACE:lgssc_kr_<wbr>negotiate(): child start on behalf of key 198fefe7: cred 0x1ecc2e0, uid 0, svc 2, nid 20000ac1f2109, uids: 0:0/0:0</font></div><div><font size="1">Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22251]:INFO:ipv4_<wbr>nid2hostname(): SOCKLND: net 0x20000, addr 0x9211fac => ip-172-31-33-9.us-west-2.<wbr>compute.internal</font></div><div><font size="1">Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22251]:DEBUG:lgss_get_<wbr>service_str(): constructed service string: lustre_oss@ip-172-31-33-9.us-<wbr>west-2.compute.internal</font></div><div><font size="1">Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22251]:TRACE:lgss_using_cred(<wbr>): using sk cred 0x1ecc2e0</font></div><div><font size="1">Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22253]:TRACE:main(): start parsing parameters</font></div><div><font size="1">Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22253]:INFO:main(): key 189483693, desc 0@25, ugid 0:0, sring 46159405, coinfo 37:sk:0:0:m:p:2:<wbr>0x20000ac1f2687:scratch-<wbr>OST2b9d-osc-MDT0000:<wbr>0x20000ac1f2ef5:1</font></div><div><font size="1">Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22253]:TRACE:parse_callout_<wbr>info(): components: 37,sk,0,0,m,p,2,<wbr>0x20000ac1f2687,scratch-<wbr>OST2b9d-osc-MDT0000,<wbr>0x20000ac1f2ef5,1</font></div><div><font size="1">Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22253]:DEBUG:parse_callout_<wbr>info(): parse call out info: secid 37, mech sk, ugid 0:0, is_root 0, is_mdt 1, is_ost 0, svc type p, svc 2, nid 0x20000ac1f2687, tgt scratch-OST2b9d-osc-MDT0000, self nid 0x20000ac1f2ef5, pid 1</font></div><div><font size="1">Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22253]:TRACE:main(): parsing parameters OK</font></div><div><font size="1">Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22253]:TRACE:lgss_mech_<wbr>initialize(): initialize mech sk</font></div><div><font size="1">Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22253]:TRACE:lgss_create_<wbr>cred(): create a sk cred at 0x21b02e0</font></div><div><font size="1">Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22253]:TRACE:main(): caller's namespace is the same</font></div><div><font size="1">Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22253]:TRACE:lgss_prepare_<wbr>cred(): preparing sk cred 0x21b02e0</font></div><div><font size="1">Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22253]:INFO:sk_create_cred(): Creating credentials for target: scratch-OST2b9d-osc-MDT0000 with nodemap: (null)</font></div><div><font size="1">Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22253]:INFO:sk_create_cred(): Searching for key with description: lustre:scratch</font></div><div><font size="1">Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22253]:TRACE:prepare_and_<wbr>instantiate(): instantiated kernel key 0b4b4aad</font></div><div><font size="1">Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22253]:TRACE:main(): forked child 22254</font></div><div><font size="1">Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22254]:TRACE:lgssc_kr_<wbr>negotiate(): child start on behalf of key 0b4b4aad: cred 0x21b02e0, uid 0, svc 2, nid 20000ac1f2687, uids: 0:0/0:0</font></div><div><font size="1">Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22254]:INFO:ipv4_<wbr>nid2hostname(): SOCKLND: net 0x20000, addr 0x87261fac => ip-172-31-38-135.us-west-2.<wbr>compute.internal</font></div><div><font size="1">Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22254]:DEBUG:lgss_get_<wbr>service_str(): constructed service string: lustre_oss@ip-172-31-38-135.<wbr>us-west-2.compute.internal</font></div><div><font size="1">Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22254]:TRACE:lgss_using_cred(<wbr>): using sk cred 0x21b02e0</font></div><div><font size="1">Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22251]:INFO:sk_encode_<wbr>netstring(): Encoded netstring of 647 bytes</font></div><div><font size="1">Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22251]:INFO:lgss_sk_using_<wbr>cred(): Created netstring of 647 bytes</font></div><div><font size="1">Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22251]:TRACE:lgssc_<wbr>negotiation_manual(): starting gss negotation</font></div><div><font size="1">Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22251]:TRACE:do_nego_rpc(): start negotiation rpc</font></div><div><font size="1">Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22251]:TRACE:gss_do_ioctl(): to open /proc/fs/lustre/sptlrpc/gss/<wbr>init_channel</font></div><div><font size="1">Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22251]:TRACE:gss_do_ioctl(): to down-write</font></div><div><font size="1">Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22254]:INFO:sk_encode_<wbr>netstring(): Encoded netstring of 647 bytes</font></div><div><font size="1">Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22254]:INFO:lgss_sk_using_<wbr>cred(): Created netstring of 647 bytes</font></div><div><font size="1">Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22254]:TRACE:lgssc_<wbr>negotiation_manual(): starting gss negotation</font></div><div><font size="1">Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22254]:TRACE:do_nego_rpc(): start negotiation rpc</font></div><div><font size="1">Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22254]:TRACE:gss_do_ioctl(): to open /proc/fs/lustre/sptlrpc/gss/<wbr>init_channel</font></div><div><font size="1">Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22254]:TRACE:gss_do_ioctl(): to down-write</font></div><div><font size="1">Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22251]:TRACE:do_nego_rpc(): do_nego_rpc: to parse reply</font></div><div><font size="1">Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22251]:DEBUG:do_nego_rpc(): do_nego_rpc: receive handle len 0, token len 0, res 0</font></div><div><font size="1">Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22251]:ERROR:lgssc_<wbr>negotiation_manual(): negotiation gss error 60000</font></div><div><font size="1">Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22251]:ERROR:lgssc_kr_<wbr>negotiate_manual(): key 198fefe7: failed to negotiate</font></div><div><font size="1">Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22251]:TRACE:error_kernel_<wbr>key(): revoking kernel key 198fefe7</font></div><div><font size="1">Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22251]:INFO:error_kernel_key(<wbr>): key 198fefe7: revoked</font></div><div><font size="1">Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22251]:TRACE:lgss_release_<wbr>cred(): releasing sk cred 0x1ecc2e0</font></div><div><font size="1">Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22254]:TRACE:do_nego_rpc(): do_nego_rpc: to parse reply</font></div><div><font size="1">Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22254]:DEBUG:do_nego_rpc(): do_nego_rpc: receive handle len 0, token len 0, res 0</font></div><div><font size="1">Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22254]:ERROR:lgssc_<wbr>negotiation_manual(): negotiation gss error 60000</font></div><div><font size="1">Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22254]:ERROR:lgssc_kr_<wbr>negotiate_manual(): key 0b4b4aad: failed to negotiate</font></div><div><font size="1">Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22254]:TRACE:error_kernel_<wbr>key(): revoking kernel key 0b4b4aad</font></div><div><font size="1">Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22254]:INFO:error_kernel_key(<wbr>): key 0b4b4aad: revoked</font></div><div><font size="1">Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22254]:TRACE:lgss_release_<wbr>cred(): releasing sk cred 0x21b02e0</font></div></div></div>
<br>______________________________<wbr>_________________<br>
lustre-discuss mailing list<br>
<a href="mailto:lustre-discuss@lists.lustre.org">lustre-discuss@lists.lustre.<wbr>org</a><br>
<a href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org" rel="noreferrer" target="_blank">http://lists.lustre.org/<wbr>listinfo.cgi/lustre-discuss-<wbr>lustre.org</a><br>
<br></blockquote></div><br></div>