<div dir="auto">We encountered this in testing done time ago and already have a bug filed (don't recall the number right now) and should have a patch soonish if not already. The gist of the problem is changelog registration limits (interger type) and some padding resulting in an artificially low limit.</div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Jul 4, 2019, 6:42 AM Matt Rásó-Barnett <<a href="mailto:matt@rasobarnett.com">matt@rasobarnett.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">I just tried out this configuration and was able to reproduce what Scott <br>
saw on 2.12.2.<br>
<br>
I couldn't see a Jira ticket for this though so I've opened one a new <br>
one: <a href="https://jira.whamcloud.com/browse/LU-12506" rel="noreferrer noreferrer" target="_blank">https://jira.whamcloud.com/browse/LU-12506</a><br>
<br>
Cheers,<br>
-- <br>
Matt Rásó-Barnett<br>
University of Cambridge<br>
<br>
On Wed, May 22, 2019 at 08:02:59AM +0000, Andreas Dilger wrote:<br>
>Scott, if you haven't already done so, it is probably best to file a <br>
>ticket in Jira with the details. Please include the client <br>
>syslog/dmesg as well as a Lustre debug log ("lctl dk /tmp/debug") so <br>
>that the problem can be isolated.<br>
><br>
>During DNE development we tested with up to 128 MDTs in AWS, but <br>
>haven't tested that many MDTs in some time.<br>
><br>
>Cheers, Andreas<br>
><br>
>On May 8, 2019, at 12:28, White, Scott F <<a href="mailto:sfpwhite@lanl.gov" target="_blank" rel="noreferrer">sfpwhite@lanl.gov</a>> wrote:<br>
>><br>
>> We\u2019ve been testing DNE Phase II and tried scaling the number of <br>
>> MDSes(one MDT each for all of our tests) very high, but when we did <br>
>> that, we couldn\u2019t mount the filesystem on a client. After trial and <br>
>> error, we discovered that we were unable to mount the filesystem when <br>
>> there were 56 MDSes. 55 MDSes mounted without issue, and it appears <br>
>> any number below that will mount. This failure at 56 MDSes was <br>
>> replicable across different nodes being used for the MDSes, all of <br>
>> which were tested with working configurations, so it doesn\u2019t seem to <br>
>> be a bad server.<br>
>><br>
>> Here\u2019s the error info we saw in dmesg on the client:<br>
>><br>
>> LustreError: 28880:0:(obd_config.c:559:class_setup()) setup <br>
>> lustre-MDT0037-mdc-ffff95923d31b000 failed (-16)<br>
>> LustreError: 28880:0:(obd_config.c:1836:class_config_llog_handler()) <br>
>> MGCx.x.x.x@o2ib: cfg command failed: rc = -16<br>
>> Lustre: cmd=cf003 0:lustre-MDT0037-mdc 1:lustre-MDT0037_UUID <br>
>> 2:x.x.x.x@o2ib<br>
>> LustreError: 15c-8: MGCx.x.x.x@o2ib: The configuration from log <br>
>> 'lustre-client' failed (-16). This may be the result of communication <br>
>> errors between this node and the MGS, a bad configuration, or other <br>
>> errors. See the syslog for more information.<br>
>> LustreError: 28858:0:(obd_config.c:610:class_cleanup()) Device 58 not <br>
>> setup<br>
>> Lustre: Unmounted lustre-client<br>
>> LustreError: 28858:0:(obd_mount.c:1608:lustre_fill_super()) Unable to <br>
>> mount (-16)<br>
>><br>
>> OS: CentOS 7.6.1810<br>
>> Kernel: 3.10.0-957.5.1.el7.x86_64<br>
>> Lustre: 2.12.1<br>
>> Network card: Qlogic InfiniPath_QLE7340<br>
>><br>
>> Other things to note for completeness\u2019 sake: this happened with both <br>
>> ldiskfs and zfs backfstypes, and these tests were using files in <br>
>> memory as the backing devices.<br>
>><br>
>> Is there something I\u2019m missing as to why more than 56 MDSes won\u2019t <br>
>> mount?<br>
>><br>
>> Thanks,<br>
>> Scott White<br>
>> Scientist, HPC<br>
>> Los Alamos National Laboratory<br>
>><br>
>> _______________________________________________<br>
>> lustre-discuss mailing list<br>
>> <a href="mailto:lustre-discuss@lists.lustre.org" target="_blank" rel="noreferrer">lustre-discuss@lists.lustre.org</a><br>
>> <a href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org" rel="noreferrer noreferrer" target="_blank">http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org</a><br>
><br>
>Cheers, Andreas<br>
>--<br>
>Andreas Dilger<br>
>Principal Lustre Architect<br>
>Whamcloud<br>
><br>
>_______________________________________________<br>
>lustre-discuss mailing list<br>
><a href="mailto:lustre-discuss@lists.lustre.org" target="_blank" rel="noreferrer">lustre-discuss@lists.lustre.org</a><br>
><a href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org" rel="noreferrer noreferrer" target="_blank">http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org</a><br>
_______________________________________________<br>
lustre-discuss mailing list<br>
<a href="mailto:lustre-discuss@lists.lustre.org" target="_blank" rel="noreferrer">lustre-discuss@lists.lustre.org</a><br>
<a href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org" rel="noreferrer noreferrer" target="_blank">http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org</a><br>
</blockquote></div>