<div dir="ltr">Hi Andreas,<br><br>Thanks again for the help, "SINGLEMDS=mds1" does the trick !<br>We have hit some issues with tainted kernel modules and hanging now (working on RHEL8 on ARM64),<br>but we need to update to latest kernel update, and install latest Lustre master and we'll see !<br><br>Sorry for the late reply, was off/away from Lustre !<br><br>Cheers,<br><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, 2 Aug 2019 at 13:04, Andreas Dilger <<a href="mailto:adilger@whamcloud.com">adilger@whamcloud.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="auto">
I thought I replied to this email, but maybe it was lost.
<div><br>
</div>
<div>It looks like you have "$SINGLEMDS" unset in your test config. It should</div>
<div>just be "mds1". That is causing the error:</div>
<div><br>
</div>
<div>
<div dir="ltr">
<div dir="ltr"><font color="#000000"><span style="background-color:rgba(255,255,255,0)"> MDS: No host defined for facet /usr/sbin/lctl</span></font></div>
</div>
</div>
<div><br>
</div>
<div>I don't know if that is causing your other problem or something else,</div>
<div>but may as well fix it and see. </div>
<div><br>
</div>
<div>You could also run with "sh -vx" to get all the gory details from bash</div>
<div>to see what is being executed. </div>
<div><br>
<div id="gmail-m_-2936280308203384609AppleMailSignature" dir="ltr">Cheers, Andreas</div>
<div dir="ltr"><br>
On Jul 23, 2019, at 02:33, Baptiste Gerondeau <<a href="mailto:baptiste.gerondeau@linaro.org" target="_blank">baptiste.gerondeau@linaro.org</a>> wrote:<br>
<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div dir="ltr">After testing it out on an ARM64 client (hostname : lustrerhel, running RHEL8, compiled from master), it seems it has the same problem.<br>
<br>
I can <b>successfully</b> llmount.sh and llmountcleanup.sh and write and read files from the client.<br>
That said, sanity.sh is <b>not</b> working for me : it never gets to the tests part, it just stops at 'cat /proc/mounts on OSS'.<br>
dmesg says nothing more, and I can't seem to get a more info (an error) from the logs.<br>
I have confirmed that I can 'cat /proc/mounts' just fine on all the machines.<br>
<br>
Client: Lustre version: 2.12.0<br>
MDS: No host defined for facet /usr/sbin/lctl<br>
OSS: Lustre version: 2.12.0<br>
CMD: lustrerhel,x8602 PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/utils/gss:/usr/lib64/lustre/utils:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin:/sbin::/sbin:/bin:/usr/sbin:
NAME=local bash rpc.sh check_config_client /lustre <br>
x8602: x8602: executing check_config_client /lustre<br>
lustrerhel: CMD: lustrerhel /usr/sbin/lctl get_param -n version 2>/dev/null ||<br>
lustrerhel: /usr/sbin/lctl lustre_build_version 2>/dev/null ||<br>
lustrerhel: /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2<br>
lustrerhel: CMD: lustrerhel /usr/sbin/lctl get_param -n version 2>/dev/null ||<br>
lustrerhel: /usr/sbin/lctl lustre_build_version 2>/dev/null ||<br>
lustrerhel: /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2<br>
lustrerhel: CMD: lustrerhel /usr/sbin/lctl get_param -n version 2>/dev/null ||<br>
lustrerhel: /usr/sbin/lctl lustre_build_version 2>/dev/null ||<br>
lustrerhel: /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2<br>
lustrerhel: CMD: lustrerhel /usr/sbin/lctl get_param -n version 2>/dev/null ||<br>
lustrerhel: /usr/sbin/lctl lustre_build_version 2>/dev/null ||<br>
lustrerhel: /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2<br>
x8602: Checking config lustre mounted on /lustre<br>
lustrerhel: lustrerhel: executing check_config_client /lustre<br>
lustrerhel: Checking config lustre mounted on /lustre<br>
Checking servers environments<br>
[...]<br>
CMD: x86ohpc e2label /dev/sda2 2>/dev/null<br>
x86ohpc: Warning: Permanently added 'x86ohpc,10.40.24.210' (ECDSA) to the list of known hosts.<br>
CMD: x86ohpc cat /proc/mounts<br>
x86ohpc: Warning: Permanently added 'x86ohpc,10.40.24.210' (ECDSA) to the list of known hosts.<br>
CMD: x8601 e2label /dev/sda2 2>/dev/null<br>
CMD: x8601 cat /proc/mounts<br>
<br>
Thanks a lot for your support,<br>
Best regards,</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Thu, 18 Jul 2019 at 20:56, Andreas Dilger <<a href="mailto:adilger@whamcloud.com" target="_blank">adilger@whamcloud.com</a>> wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
On Jul 18, 2019, at 04:29, Baptiste Gerondeau <<a href="mailto:baptiste.gerondeau@linaro.org" target="_blank">baptiste.gerondeau@linaro.org</a>> wrote:<br>
> <br>
> Thank you very much for your quick help !<br>
> I reformatted and remounted everything from scratch and can confirm that mounting works, and that the client can communicate with the MDS (210, OSS is 211 and client 212):<br>
[snip]<br>
> [root@x8602 tests]# lctl which_nid 10.40.24.210@tcp<br>
> 10.40.24.210@tcp<br>
> [root@x8602 tests]# lfs df -ih<br>
> UUID Inodes IUsed IFree IUse% Mounted on<br>
> test-MDT0000_UUID 4.0M 272 4.0M 1% /lustre[MDT:0]<br>
> test-OST0000_UUID 640.0K 267 639.7K 0% /lustre[OST:0]<br>
> <br>
> filesystem_summary: 640.0K 272 639.7K 0% /lustre<br>
> <br>
> [root@x8602 tests]# ls -lsah /lustre/<br>
> total 12K<br>
> 4.0K drwxr-xr-x 3 root root 4.0K Jul 18 11:03 .<br>
> 4.0K dr-xr-xr-x. 19 root root 4.0K Jun 28 11:43 ..<br>
> 4.0K -rw-r--r-- 1 root root 14 Jul 18 11:03 test.txt<br>
> <br>
> I get the same output from auster though:<br>
> Client: Lustre version: 2.12.0<br>
> MDS: No host defined for facet /usr/sbin/lctl<br>
<br>
This looks like some kind of problem with the test configuration file, where an environment variable is not set (e.g. mds_HOST) and it is interpreting the next argument (the lctl command) as the target facet when calling do_facet() or similar?<br>
<br>
If "llmount.sh" works, then you are also able to run tests directly like:<br>
<br>
client# cd lustre/tests<br>
client# sh sanity.sh<br>
<br>
I don't use auster myself (it is just a wrapper around lower-level scripts), so I can't really comment where the problem might be.<br>
<br>
Cheers, Andreas<br>
<br>
> OSS: Lustre version: 2.12.0<br>
> <br>
> From the client I can ssh into the other nodes (and from each node I can ssh into the others).<br>
> I had tried to debug the scripts behind the above auster output but was unable to track down where it failed...<br>
> <br>
> On Tue, 16 Jul 2019 at 23:09, Andreas Dilger <<a href="mailto:adilger@whamcloud.com" target="_blank">adilger@whamcloud.com</a>> wrote:<br>
> On Jul 16, 2019, at 06:11, Baptiste Gerondeau <<a href="mailto:baptiste.gerondeau@linaro.org" target="_blank">baptiste.gerondeau@linaro.org</a>> wrote:<br>
> > <br>
> > Hi,<br>
> > <br>
> > I'm currently in the process of bringing up the "3 node" x86 cluster and running "verbose=true ./auster -f multinode -rsv runtests" (on CentOS 7.6 x86 client & server, installed from repos), I keep getting "MDS: No host defined for facet /usr/sbin/lctl".<br>
> > <br>
> > Auster then prints out some pdsh stuff, "Failures : 0" and exits after 16s obviously without running any tests.<br>
> > <br>
> > Any suggestions?<br>
> > Thanks a lot,<br>
> > <br>
> > <br>
> > PS : My multinode config is attached<br>
> > PPS: I posted to the devel list because it concerned auster, if I need to post it elsewhere please let me know<br>
> <br>
> Before running auster, which tries to launch a lot of tests, start with just a plain mount to see if that is working:<br>
> <br>
> master.sh:<br>
> > MOUNT=/mnt/lustre<br>
> > MOUNT2=/mnt/master2<br>
> <br>
> This is a bit odd for tests, which normally have e.g. /mnt/master and /mnt/master2, but I'm<br>
> not sure i there will be a problem or not.<br>
> <br>
> ### assume modules/utils are built<br>
> ### modules/utils are installed or you are running out of the build directory<br>
> ### ssh to the MDS and OSS nodes works without a password<br>
> ### if you are not using @tcp0 for LNet, /etc/modprobe.d/lnet.conf is correct<br>
> <br>
> all# modprobe ptlrpc ### on client and OSS and MDS to start LNet<br>
> x8602# lctl ping x86ohpc ### should print NID(s) of x860hpc<br>
> x8602# lctl ping x8601 ### should print NID(s) of x8601<br>
> x8602# export NAME=master ### get config from lustre/tests/cfg/master.sh<br>
> x8602# sh llmount.sh ### should format x86ohpc:/dev/sda2 and x8601:/dev/sda2<br>
> x8602# lfs df ### should show master-MDT0000 and master-OST0000<br>
> <br>
> Cheers, Andreas<br>
> --<br>
> Andreas Dilger<br>
> Principal Lustre Architect<br>
> Whamcloud<br>
> <br>
> <br>
> <br>
> <br>
> <br>
> <br>
> <br>
> <br>
> -- <br>
> Baptiste Gerondeau<br>
> Engineer - HPC SIG - LDCG - Linaro<br>
> #irc : BaptisteGer<br>
<br>
Cheers, Andreas<br>
--<br>
Andreas Dilger<br>
Principal Lustre Architect<br>
Whamcloud<br>
<br>
<br>
<br>
<br>
<br>
<br>
</blockquote>
</div>
<br clear="all">
<div><br>
</div>
-- <br>
<div dir="ltr" class="gmail-m_-2936280308203384609gmail_signature">
<div dir="ltr"><font face="monospace, monospace">Baptiste Gerondeau<br>
Engineer - HPC SIG - LDCG - Linaro<br>
#irc : BaptisteGer</font><br>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr"><font face="monospace, monospace">Baptiste Gerondeau<br>Engineer - HPC SIG - LDCG - Linaro<br>#irc : BaptisteGer</font><br></div></div>