[Lustre-discuss] striping files to different OSTs

[Sebastian Reitenbach]

Hi,

I am new to lustre, just did a test setup with a MGS/MGS and two OST servers.
I run lustre 1.8.1 on SLES 11 x86_64.
reading chapter 1.4.1 of the Lustre manual, I see that files should get 
striped to different OSTs, and that files can become larger than the size of 
one OST.

However, when I use dd if=/dev/zero of=outfile.img then I get an error 
dd: writing `outfile.img': No space left on device
after the first OST is full, the second one is still empty, see below:

srv3:/lustre/foo # lfs df -h
UUID                     bytes      Used Available  Use% Mounted on
foo-MDT0000_UUID        446.1M     16.6M    404.0M    3% /lustre/foo[MDT:0]
foo-OST0000_UUID          9.8G     22.7M      9.3G    0% /lustre/foo[OST:0]
foo-OST0001_UUID          9.8G     22.7M      9.3G    0% /lustre/foo[OST:1]

filesystem summary:      19.7G     45.4M     18.6G    0% /lustre/foo

UUID                     bytes      Used Available  Use% Mounted on
bar-MDT0000_UUID        425.5M     16.4M    384.7M    3% /lustre/bar[MDT:0]
bar-OST0000_UUID          9.8G     22.7M      9.3G    0% /lustre/bar[OST:0]
bar-OST0001_UUID          9.8G     22.7M      9.3G    0% /lustre/bar[OST:1]

filesystem summary:      19.7G     45.4M     18.6G    0% /lustre/bar

srv3:/lustre/foo # dd if=/dev/zero of=outfile.img bs=4M
dd: writing `outfile.img': No space left on device     
2388+0 records in                                      
2387+0 records out                                     
10012561408 bytes (10 GB) copied, 221.014 s, 45.3 MB/s 
srv3:/lustre/foo # lfs df -h
UUID                     bytes      Used Available  Use% Mounted on
foo-MDT0000_UUID        446.1M     16.6M    404.0M    3% /lustre/foo[MDT:0]
foo-OST0000_UUID          9.8G      9.3G      4.0K   94% /lustre/foo[OST:0]
foo-OST0001_UUID          9.8G     22.7M      9.3G    0% /lustre/foo[OST:1]

filesystem summary:      19.7G      9.4G      9.3G   47% /lustre/foo

UUID                     bytes      Used Available  Use% Mounted on
bar-MDT0000_UUID        425.5M     16.4M    384.7M    3% /lustre/bar[MDT:0]
bar-OST0000_UUID          9.8G     22.7M      9.3G    0% /lustre/bar[OST:0]
bar-OST0001_UUID          9.8G     22.7M      9.3G    0% /lustre/bar[OST:1]

filesystem summary:      19.7G     45.4M     18.6G    0% /lustre/bar

srv3:/lustre/foo # dd if=/dev/zero of=outfile2.img bs=4M
dd: writing `outfile2.img': No space left on device
2388+0 records in
2387+0 records out
10012565504 bytes (10 GB) copied, 331.793 s, 30.2 MB/s
srv3:/lustre/foo # lfs df -h
UUID                     bytes      Used Available  Use% Mounted on
foo-MDT0000_UUID        446.1M     16.6M    404.0M    3% /lustre/foo[MDT:0]
foo-OST0000_UUID          9.8G      9.3G      4.0K   94% /lustre/foo[OST:0]
foo-OST0001_UUID          9.8G      9.3G      4.0K   94% /lustre/foo[OST:1]

filesystem summary:      19.7G     18.7G      8.0K   94% /lustre/foo

UUID                     bytes      Used Available  Use% Mounted on
bar-MDT0000_UUID        425.5M     16.4M    384.7M    3% /lustre/bar[MDT:0]
bar-OST0000_UUID          9.8G     22.7M      9.3G    0% /lustre/bar[OST:0]
bar-OST0001_UUID          9.8G     22.7M      9.3G    0% /lustre/bar[OST:1]

filesystem summary:      19.7G     45.4M     18.6G    0% /lustre/bar


The filesystems were created like this on the MGS/MDS:
mkfs.lustre --mgs --mdt --fsname=foo --reformat /dev/xvdb1
mkfs.lustre --mdt --fsname=bar --mgsnode=10.0.0.81 at tcp --reformat /dev/xvdb2
mount -t lustre /dev/xvdb1 /lustre/foo-mgs-mdt
mount -t lustre /dev/xvdb2 /lustre/bar-mdt

on OST1:
mke2fs -O journal_dev -b 4096 /dev/xvdd
mke2fs -O journal_dev -b 4096 /dev/xvde
mkfs.lustre --fsname=foo --param="failover.node=10.0.0.86 at tcp" --
mgsnode=10.0.0.81 at tcp --ost --reformat --mkfsoptions "-j -J device=/dev/xvdd -
E stride=32" /dev/xvdb1
mkfs.lustre --fsname=bar --param="failover.node=10.0.0.86 at tcp" --
mgsnode=10.0.0.81 at tcp --ost --reformat --mkfsoptions "-j -J device=/dev/xvde -
E stride=32" /dev/xvdb2

on OST2:
mke2fs -O journal_dev -b 4096 /dev/xvdf
mke2fs -O journal_dev -b 4096 /dev/xvdg
mkfs.lustre --fsname=foo --param="failover.node=10.0.0.85 at tcp" --
mgsnode=10.0.0.81 at tcp --ost --reformat --mkfsoptions "-j -J device=/dev/xvdf -
E stride=32" /dev/xvdc1
mkfs.lustre --fsname=bar --param="failover.node=10.0.0.85 at tcp" --
mgsnode=10.0.0.81 at tcp --ost --reformat --mkfsoptions "-j -J device=/dev/xvdg -
E stride=32" /dev/xvdc2
On the OST hosts I use pacemaker to mout the filesystems.
On the clients I mounted the filesystems:
mount -t lustre 10.0.0.81 at tcp:/foo /lustre/foo
mount -t lustre 10.0.0.81 at tcp:/bar /lustre/bar

Anything I did wrong or misunderstood? Maybe I need to tune a parameter of the 
file system to activate that striping?

regards,
Sebastian