[lustre-discuss] Nodemap and setreuid/setregid
Hans Henrik Happe
happe at nbi.dk
Sat Feb 29 14:44:25 PST 2020
Hi,
Sorry for the delay. I had to spend some time nursing the glusterfs that
this lustre fs will replace :-)
Anyway, I've created a procedure to reproduce the issue. It's attached
together with the testing program.
Basically, its a simple single mgs,mdt,oss setup, with a nodemap, that
maps a client to a fileset. This works fine. However, when turning on
SSK for cli2mdt the issue appears.
This was for 2.12.3, I will move on to 2.12.4 just to check.
Cheers,
Hans Henrik
On 06.02.2020 23.08, Hans Henrik Happe wrote:
> Hi Sebastien,
>
> Thanks for looking into this.
>
> You are right that nodemap deactivation didn't affect the outcome. I
> must have made a mistake and cannot reproduce.
>
> The uid/gid are on the mds. I can do a sudo to the user and run the test
> program successfully.
>
> I forgot to mention that I use SSK in ski mode.
>
> I think I will start from scratch and see if I can reproduce and find
> out at what point it stops working.
>
> Cheers,
> Hans Henrik
>
> On 06.02.2020 18.19, Sebastien Buisson wrote:
>> Hi,
>>
>> I am not able to reproduce your issue. I compiled your C program, in all cases I am not getting Permission Denied.
>>
>> You say that it works when you deactivate the nodemap. But given that you have a fileset on your nodemap entry « sif », when you deactivate it you might end up doing IOs in a different directory. So you might compare different things.
>> Also, does the uid/gid 20501 exist on server side?
>>
>> Cheers,
>> Sebastien.
>>
>>> Le 6 févr. 2020 à 14:29, Hans Henrik Happe <happe at nbi.dk> a écrit :
>>>
>>> Hi,
>>>
>>> Thanks for a very quick reply :-) Here are the map:
>>>
>>> # lctl get_param nodemap.sif.*
>>> nodemap.sif.admin_nodemap=1
>>> nodemap.sif.audit_mode=1
>>> nodemap.sif.deny_unknown=0
>>> nodemap.sif.exports=
>>> [
>>> { nid: 172.25.10.51 at tcp, uuid: 56bb9b04-9bb5-d7b5-3f50-d62804690db1 },
>>> ]
>>> nodemap.sif.fileset=/sif
>>> nodemap.sif.id=2
>>> nodemap.sif.idmap=
>>> [
>>> { idtype: uid, client_id: 501, fs_id: 20501 },
>>> { idtype: gid, client_id: 501, fs_id: 20501 }
>>> ]
>>> nodemap.sif.map_mode=both
>>> nodemap.sif.ranges=
>>> [
>>> { id: 11, start_nid: 172.25.1.28 at tcp, end_nid: 172.25.1.28 at tcp },
>>> { id: 10, start_nid: 172.25.1.27 at tcp, end_nid: 172.25.1.27 at tcp },
>>> { id: 9, start_nid: 172.25.10.51 at tcp, end_nid: 172.25.10.51 at tcp }
>>> ]
>>> nodemap.sif.sepol=
>>>
>>> nodemap.sif.squash_gid=20000
>>> nodemap.sif.squash_uid=20000
>>> nodemap.sif.trusted_nodemap=0
>>>
>>> Cheers,
>>> Hans Henrik
>>>
>>> On 06.02.2020 14.17, Sebastien Buisson wrote:
>>>> Hi,
>>>>
>>>> It might be due to a property on the nodemap you defined.
>>>> Could you please dump your nodemap definition?
>>>>
>>>> Thanks,
>>>> Sebastien.
>>>>
>>>>
>>>>> Le 6 févr. 2020 à 14:14, Hans Henrik Happe <happe at nbi.dk>
>>>>> a écrit :
>>>>>
>>>>> Hi,
>>>>>
>>>>> Has anyone had success with gocryptfs 1.7.x on top of a Lustre nodemap?
>>>>>
>>>>> I've tested with Lustre 2.12.3.
>>>>>
>>>>> I found that gocryptfs 1.6 worked. However, with 1.7.x I got a lot of
>>>>> "Permission denied". I tried all permutations of trusted and admin on
>>>>> the nodemap.
>>>>>
>>>>> By stracing a bit, I've created a small peace of code provoking the issue:
>>>>>
>>>>> ---
>>>>>
>>>>> #include <unistd.h>
>>>>> #include <sys/types.h>
>>>>> #include <fcntl.h>
>>>>> #include <stdio.h>
>>>>>
>>>>> int main() {
>>>>> int r;
>>>>>
>>>>> setregid(-1, 501);
>>>>> setreuid(-1, 501);
>>>>>
>>>>> r = open("foo", O_CREAT, S_IRWXU);
>>>>> if (r < 0) {
>>>>> perror("open");
>>>>> }
>>>>> return 0;
>>>>> }
>>>>>
>>>>> ---
>>>>>
>>>>>
>>>>>
>>>>> When run as root in a directory owned by uid=501 and gid=501 in a
>>>>> nodemap based Lustre fs it returns:
>>>>>
>>>>> open: Permission denied
>>>>>
>>>>> Works when I deactivate nodemap (lctl nodemap_activate 0) or just use a
>>>>> plain local fs.
>>>>>
>>>>> I don't think this is intended behavior for nodemaps, but I might be wrong.
>>>>>
>>>>> Cheers,
>>>>> Hans Henrik
>>>>> _______________________________________________
>>>>> lustre-discuss mailing list
>>>>>
>>>>> lustre-discuss at lists.lustre.org
>>>>> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
>>> _______________________________________________
>>> lustre-discuss mailing list
>>> lustre-discuss at lists.lustre.org
>>> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
> _______________________________________________
> lustre-discuss mailing list
> lustre-discuss at lists.lustre.org
> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20200229/7e39693b/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: test.c
Type: text/x-csrc
Size: 239 bytes
Desc: not available
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20200229/7e39693b/attachment.c>
-------------- next part --------------
*Create*
mds:
mkfs.lustre --backfstype=zfs --mgs --servicenode=172.25.10.50 at tcp0 sci-mds00p0/mgs-test
mkfs.lustre --backfstype=zfs --mdt --fsname test --index=0 --mgsnode=172.25.10.50 at tcp0 --servicenode=172.25.10.50 at tcp0 sci-mds00p0/test0
oss:
mkfs.lustre --backfstype=zfs --ost --fsname test --index=0 --mgsnode=172.25.10.50 at tcp0 --servicenode=172.25.10.111 at tcp0 sci-oss11p0/test0
/etc/ldev.conf:
sci-mds00.science - MGS zfs:sci-mds00p0/mgs-test
sci-mds00.science - test-MDT0000 zfs:sci-mds00p0/test0
sci-oss11.science - test-OST0000 zfs:sci-oss11p0/test0
keys:
Server: lgss_sk -t server -f test -n map -w test.server.key
Client: (Copy key to test.client.key)
Cleint: lgss_sk -t client -m test.client.key
*Prepare nodemap*
mds:
mount -t lustre sci-mds00.science:/test /mnt
mkdir /mnt/map
lctl nodemap_add map
lctl nodemap_add_range --name map --range 172.25.10.51 at tcp
lctl nodemap_modify --name map --property trusted --value 1
lctl nodemap_modify --name map --property admin --value 1
lctl nodemap_set_fileset --name map --fileset '/map'
lctl nodemap_activate 1
*Test with no SSK*
client (172.25.10.51 at tcp):
mount -t lustre sci-mds00:/test /mnt
mkdir /mnt/userdir
chown 501:501 /mnt/userdir
cd /mnt/userdir
(run test program. Works)
rm -f foo
cd
umount /mnt
*Add SSK to the mdt*
mds:
lctl conf_param test.srpc.flavor.default.cli2mdt=ski
*Test with SSK on mdt*
client (172.25.10.51 at tcp):
mount -o skpath=/<path>/test.client.key -t lustre sci-mds00:/test /mnt
cd /mnt/userdir
(run test program => open: Permission denied)
More information about the lustre-discuss
mailing list