[Lustre-discuss] OST: inactive device, but why?

Sebastian Reitenbach sebastia at l00-bugdead-prods.de
Mon Feb 1 04:32:11 PST 2010


Hi,

I am running lustre in my test setup, and I saw inactive OST device, when I do 
what I describe below:
I run lustre-1.8.1.1 on SLES11 x86_64, on servers and clients.

On the MGS/MDS (10.0.0.81) server, I created one mgs and two mdt partitions:
mkfs.lustre --mgs --mdt --fsname=foo --reformat --mkfsoptions="-N 500000" 
/dev/xvdb1
mkfs.lustre --mdt --fsname=bar --mgsnode=10.0.0.81 at tcp --reformat /dev/xvdb2
mount -t lustre /dev/xvdb1 /lustre/foo-mgs-mdt
mount -t lustre /dev/xvdb2 /lustre/bar-mdt

On the two OSS hosts, I created those storages:
ON OST1 (10.0.0.85)host:
mke2fs -O journal_dev -b 4096 /dev/xvdd
mke2fs -O journal_dev -b 4096 /dev/xvde
mkfs.lustre --fsname=foo --param="failover.node=10.0.0.86 at tcp" --
mgsnode=10.0.0.81 at tcp --ost --reformat --mkfsoptions "-j -J device=/dev/xvdd -
E stride=32" /dev/xvdb1
mkfs.lustre --fsname=bar --param="failover.node=10.0.0.86 at tcp" --
mgsnode=10.0.0.81 at tcp --ost --reformat --mkfsoptions "-j -J device=/dev/xvde -
E stride=32" /dev/xvdb2
ON OST2 (10.0.0.86)host:
mke2fs -O journal_dev -b 4096 /dev/xvdf
mke2fs -O journal_dev -b 4096 /dev/xvdg
mkfs.lustre --fsname=foo --param="failover.node=10.0.0.85 at tcp" --
mgsnode=10.0.0.81 at tcp --ost --reformat --mkfsoptions "-j -J device=/dev/xvdf -
E stride=32" /dev/xvdc1
mkfs.lustre --fsname=bar --param="failover.node=10.0.0.85 at tcp" --
mgsnode=10.0.0.81 at tcp --ost --reformat --mkfsoptions "-j -J device=/dev/xvdg -
E stride=32" /dev/xvdc2

The devices are shared storage via SAN. I configured pacemaker, using the 
lustre resource script, to mount and unmount the partitions on the OST hosts. 
I had both OST hosts in standby mode. I put OST2 (10.0.0.86) in active mode. 
Then all four partitions got successfully mounted on OST2. Then I put OST1 in 
active mode too. The shares that belong to OST1, were automatically unmounted 
from OST2, and successfully mounted on OST1. That happened due to configured 
constraints in pacemaker.

Then I mounted the client filesystems:
mount -t lustre 10.0.0.81 at tcp:/foo /lustre/foo
mount -t lustre 10.0.0.81 at tcp:/bar /lustre/bar


LUSTRE-CLIENT:/lustre # lfs df -h
UUID                     bytes      Used Available  Use% Mounted on
foo-MDT0000_UUID        265.5M     19.1M    220.9M    7% foo[MDT:0]
foo-OST0000_UUID    : inactive device
foo-OST0001_UUID          9.8G     22.7M      9.3G    0% foo[OST:1]

filesystem summary:       9.8G     22.7M      9.3G    0% foo

LUSTRE-CLIENT:/lustre # cat /proc/fs/lustre/osc/foo-OST0000-osc-
ffff88003e695800/ost_conn_uuid                                                                                                                         
10.0.0.86 at tcp                                                                                                                                                                                                          
LUSTRE-CLIENT:/lustre # cat /proc/fs/lustre/osc/foo-OST0001-osc-
ffff88003e695800/ost_conn_uuid 
10.0.0.86 at tcp

And then I got that foo-OST0000_UUID    : inactive device
I unmounted the filesystem from the client, and stopped the OSTs, and MDT/MGS, 
and remounted everything, starting with the MGS, MDT, OSTs and then on the 
client. However, I still have the inactive device.

Therefore I reformatted everything on the servers like above. Then mounting 
the MGS/MDTs and then the OSTs. The only difference was how I started the 
OSTs:
Instead of taking them from standby mode online, both were already online, and 
I only started the lustre resource one after each other. Then the lustre 
filesystems got mounted on the server where they are intended to run.


LUSTRE-CLIENT:/lustre # lfs df -h
UUID                     bytes      Used Available  Use% Mounted on
foo-MDT0000_UUID        265.5M     19.1M    220.9M    7% foo[MDT:0]
foo-OST0000_UUID          9.8G     22.7M      9.3G    0% foo[OST:0]
foo-OST0001_UUID          9.8G     22.7M      9.3G    0% foo[OST:1]

filesystem summary:      19.7G     45.4M     18.6G    0% foo


LUSTRE-CLIENT:~ # cat /proc/fs/lustre/osc/foo-OST0000-osc-
ffff88003e5ca000/ost_conn_uuid
10.0.0.85 at tcp
LUSTRE-CLIENT:~ # cat /proc/fs/lustre/osc/foo-OST0001-osc-
ffff88003e5ca000/ost_conn_uuid
10.0.0.86 at tcp
LUSTRE-CLIENT:~ #

Afterwards, I can also put one of the OST servers into standby mode, and put 
it back online, without problem, e.g. after putting OST1 into standby mode:
LUSTRE-CLIENT:/lustre/foo # lfs df -h
UUID                     bytes      Used Available  Use% Mounted on
foo-MDT0000_UUID        265.5M     19.1M    220.9M    7% /lustre/foo[MDT:0]
foo-OST0000_UUID          9.8G     22.7M      9.3G    0% /lustre/foo[OST:0]
foo-OST0001_UUID          9.8G     22.7M      9.3G    0% /lustre/foo[OST:1]

filesystem summary:      19.7G     45.4M     18.6G    0% /lustre/foo

LUSTRE-CLIENT:/lustre/foo # cat /proc/fs/lustre/osc/foo-OST0001-osc-
ffff88003e5ca000/ost_conn_uuid
10.0.0.86 at tcp
LUSTRE-CLIENT:/lustre/foo # cat /proc/fs/lustre/osc/foo-OST0000-osc-
ffff88003e5ca000/ost_conn_uuid
10.0.0.86 at tcp

and taking it back online:
LUSTRE-CLIENT:/lustre/foo # lfs df -h
UUID                     bytes      Used Available  Use% Mounted on
foo-MDT0000_UUID        265.5M     19.1M    220.9M    7% /lustre/foo[MDT:0]
foo-OST0001_UUID          9.8G     22.7M      9.3G    0% /lustre/foo[OST:1]

filesystem summary:       9.8G     22.7M      9.3G    0% /lustre/foo

LUSTRE-CLIENT:/lustre/foo # cat /proc/fs/lustre/osc/foo-OST0001-osc-
ffff88003e5ca000/ost_conn_uuid
10.0.0.86 at tcp
LUSTRE-CLIENT:/lustre/foo # cat /proc/fs/lustre/osc/foo-OST0000-osc-
ffff88003e5ca000/ost_conn_uuid
10.0.0.85 at tcp

Therefore, some questions came up:
Is this expected behaviour when the OST filesystems are mounted the first 
time, but unfortunately on the wrong server?
How can I make the inactive OST active, when such things happened, without 
reformatting everything?
Is there a way how I can prevent this from happening accidentally? 

regards,
Sebastian



More information about the lustre-discuss mailing list