[lustre-discuss] nodemap exports

[Thomas Roth]

Thanks Sebastien,
this put all clients into the correct LNet. And nodemap.exports looks as expected.

(Had to add 'options lnet lnet_peer_discovery_disabled=1' for ‘-o network=<lnet>’ to work).


Regards
Thomas


On 6/19/24 08:31, Sebastien Buisson wrote:
> Hi,
> 
> I am not sure the NIDs displayed under ‘exports’ and in the logs are the actual NIDs used to communicate with, maybe they are just identifiers to describe the peers.
> 
> If you want to restrict a client to a given LNet network for a given mount point, you can use the ‘-o network=<lnet>’ mount option, like ‘-o network=o2ib8’ in your case.
> 
> Cheers,
> Sebastien.
> 
>> Le 18 juin 2024 à 18:01, Thomas Roth via lustre-discuss <lustre-discuss at lists.lustre.org> a écrit :
>>
>> OK, client 247 is the good one ;-)
>>
>> It is rather an LNET issue.
>>
>> All three clients have:
>>
>>> options lnet networks="o2ib8(ib0),o2ib6(ib0)"
>>
>> I can mount my nodemapped Lustre, with its  MGS on 10.20.6.63 at o2ib8, and another Lustre on o2ib6.
>>
>> However, the log of the MGS at 10.20.6.63 at o2ib8 remarks
>>> MGS: Connection restored to 15da687f-7a11-4913-9f9d-3c764c97a9cb (at 10.20.3.246 at o2ib6)
>>> MGS: Connection restored to 059da3e0-b277-4767-8678-e83730769fb8 (at 10.20.3.248 at o2ib6)
>> and then
>>> MGS: Connection restored to a0ee6b8c-48b9-46e8-ba2c-9448889c77ed (at 10.20.3.247 at o2ib8)
>>
>>
>> I can see the "alien" nids also in
>> mgs # ls /proc/fs/lustre/mgs/MGS/exports
>> ... 10.20.3.246 at o2ib6  10.20.3.247 at o2ib8  10.20.3.248 at o2ib6
>>
>> Question is: Why would an MGS on "o2ib8" accept connections from a client on "o2ib6"?
>>
>> Obviously, this would not happen if the client had only o2ib6. So the MGS is somewhat confused, since the LNET connection is actually via o2ib8, but the labels and the nodemapping use o2ib6.
>>
>>
>>
>> This is boot-resistant, the MGS has these wrong nids stored somewhere - can I erase them and start again with correct nids?
>>
>>
>> Regards
>> Thomas
>>
>>
>>
>> On 6/18/24 17:33, Thomas Roth via lustre-discuss wrote:
>>> Hi all,
>>> what is the meaning of the "exports" property/parameter of a nodemap?
>>> I have
>>> mgs ]# lctl nodemap_info newclients
>>> ...
>>> There are three clients:
>>> mgs # lctl get_param nodemap.newclients.ranges
>>> nodemap.newclients.ranges=
>>> [
>>>   { id: 13, start_nid: 10.20.3.246 at o2ib8, end_nid: 10.20.3.246 at o2ib8 },
>>>   { id: 12, start_nid: 10.20.3.247 at o2ib8, end_nid: 10.20.3.247 at o2ib8 },
>>>   { id: 9, start_nid: 10.20.3.248 at o2ib8, end_nid: 10.20.3.248 at o2ib8 }
>>> This nodemap has
>>> nodemap.newclients.admin_nodemap=0
>>> nodemap.newclients.trusted_nodemap=0
>>> nodemap.newclients.deny_unknown=1
>>> and
>>> mgs # lctl get_param nodemap.newclients.exports
>>> nodemap.newclients.exports=
>>> [
>>>   { nid: 10.20.3.247 at o2ib8, uuid: 5d9964f9-81eb-4ea5-93dc-a145534f9e74 },
>>> ]
>>> _This_ client, 10.20.3.247, behaves differently: No access for root (ok!), no access for a regular user.
>>> The other two clients, 10.20.3.246/248, show no access for root (ok!), while a regular users sees the squashed (uid 99) directories of the top level and his own directories/files with correct uid/gid beneath.
>>> And the only difference between these clients seems to be these 'exports' (totally absent from the manual, btw).
>>> Regards,
>>> Thomas
>>
>> -- 
>> --------------------------------------------------------------------
>> Thomas Roth
>> Department: Informationstechnologie
>> Location: SB3 2.291
>>
>>
>> GSI Helmholtzzentrum für Schwerionenforschung GmbH
>> Planckstraße 1, 64291 Darmstadt, Germany, www.gsi.de
>>
>> Commercial Register / Handelsregister: Amtsgericht Darmstadt, HRB 1528
>> Managing Directors / Geschäftsführung:
>> Professor Dr. Paolo Giubellino, Dr. Katharina Stummeyer, Jörg Blaurock
>> Chairman of the Supervisory Board / Vorsitzender des GSI-Aufsichtsrats:
>> Ministerialdirigent Dr. Volkmar Dietz
>>
>> _______________________________________________
>> lustre-discuss mailing list
>> lustre-discuss at lists.lustre.org
>> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
> 

-- 
--------------------------------------------------------------------
Thomas Roth
Department: Informationstechnologie
Location: SB3 2.291


GSI Helmholtzzentrum für Schwerionenforschung GmbH
Planckstraße 1, 64291 Darmstadt, Germany, www.gsi.de

Commercial Register / Handelsregister: Amtsgericht Darmstadt, HRB 1528
Managing Directors / Geschäftsführung:
Professor Dr. Paolo Giubellino, Dr. Katharina Stummeyer, Jörg Blaurock
Chairman of the Supervisory Board / Vorsitzender des GSI-Aufsichtsrats:
Ministerialdirigent Dr. Volkmar Dietz