[Lustre-discuss] Can't put file on specific device or see it it in lfs df -h

[Katya Tutlyaeva]

Hi everybody!

I've just configured Lustre Failover Active-Active configuration with 4 
OST's, 6 Tb each. (there are two RAID massives partitioned with "parted" 
utility. First raid - OST0 and OST1, second - OST2 and OST3).

I have met some problem with second RAID (Accordingly, with OST 2 and OST3).
It just absent on client machine when I use lfs df -h command:
[client]# lfs df -h
UUID                     bytes      Used Available  Use% Mounted on
lustre-MDT0000_UUID     814.8G    471.8M    767.8G    0% /mnt/lustre[MDT:0]
lustre-OST0000_UUID       5.4T    466.1M      5.1T    0% /mnt/lustre[OST:0]
lustre-OST0001_UUID       5.4T    458.1M      5.1T    0% /mnt/lustre[OST:1]

filesystem summary:      10.7T    924.3M     10.2T    0% /mnt/lustre

But I can see it Up and Active in lctl device list:
[client]# lctl dl
  0 UP mgc MGC192.168.11.151 at o2ib 928d4945-4799-86b7-73e2-2fe9fe686ad1 5
  1 UP lov lustre-clilov-ffff8806334cf400 
85a0c4c1-c194-26e3-d266-ef6fde47d182 4
  2 UP mdc lustre-MDT0000-mdc-ffff8806334cf400 
85a0c4c1-c194-26e3-d266-ef6fde47d182 5
  3 UP osc lustre-OST0000-osc-ffff8806334cf400 
85a0c4c1-c194-26e3-d266-ef6fde47d182 5
  4 UP osc lustre-OST0001-osc-ffff8806334cf400 
85a0c4c1-c194-26e3-d266-ef6fde47d182 5
*  5 UP osc lustre-OST0002-osc-ffff8806334cf400 
85a0c4c1-c194-26e3-d266-ef6fde47d182 5
  6 UP osc lustre-OST0003-osc-ffff8806334cf400 
85a0c4c1-c194-26e3-d266-ef6fde47d182 5*

Additional information:
I also can't put file directly on this devices, when I use following 
command:
lfs setstripe --count 1 --index 3 Only3.txt

I have got file on 0-th OSDS, but not on 3-th, as expected:
[client]# lfs getstripe Only3.txt
OBDS:
0: lustre-OST0000_UUID ACTIVE
1: lustre-OST0001_UUID ACTIVE
2: lustre-OST0002_UUID ACTIVE
3: lustre-OST0003_UUID ACTIVE
Only3.txt
    obdidx         objid        objid         group
         0                67             0x43                 0
But lfs getstripe also shows that 2th and 3th devices are ACTIVE.


I also formatted all OST's with the same FS-type, as examples my 
formatting command for 0 (working) and 2 (N/A) devices:

mkfs.lustre --ost --fsname=lustre --mgsnode=192.168.11.11 at o2ib,192.168.11.12 at o2ib --failover=192.168.11.12 at o2ib /dev/disk/ost0

mkfs.lustre --ost --fsname=lustre --mgsnode=192.168.11.11 at o2ib,192.168.11.12 at o2ib --failover=192.168.11.11 at o2ib /dev/disk/ost2

Of course, these devices are successfully mounted on OSS, when I move 
them using hb_takeover on another OSS (even if I move all devices, 
include mdt on second OSS or move these unworking devices on first OSS) 
first two OST's remains up and accessible, second two still N/A in df -h 
and for file striping.
Please tell me if I missing something..

Looking forward to your advices!

Best regards,
Katya