[Lustre-discuss] max file size on lustre - No space left on device after 37GB!

[Alex]

Hello,

Reading Lustre Manual (section 33. System Limits) i could not find an answer 
to my problem.

I am tring to create a 50GB testfile on my lustre file system using dd. After 
37GB (which is the max size of one of my OST), dd is stopping complaining 
about FREE SPACE!!!

[root at rs1 ~]# uname -rm
2.6.18-53.1.14.el5_lustre.1.6.5.1smp i686
[root at rs1 ~]#

Before dd usage:

[root at rs1 ~]# lfs df -h
UUID                     bytes      Used Available  Use% Mounted on
testfs-MDT0000_UUID     130.4G    460.0M    122.5G    0% /mnt/lustre[MDT:0]
testfs-OST0000_UUID      18.3G      1.4G     16.0G    7% /mnt/lustre[OST:0]
testfs-OST0001_UUID      18.3G    444.5M     17.0G    2% /mnt/lustre[OST:1]
testfs-OST0002_UUID      36.7G    448.5M     34.4G    1% /mnt/lustre[OST:2]
testfs-OST0003_UUID      36.7G    448.5M     34.4G    1% /mnt/lustre[OST:3]
filesystem summary:     110.0G      2.7G    101.7G    2% /mnt/lustre
[root at rs1 ~]#

So, i have 101GB free -> start dd and create a 50GB file on /mnt/lustre:

[root at rs1 ~]# dd if=/dev/zero of=/mnt/lustre/test2.img bs=100M count=500
dd: writing `/mnt/lustre/test2.img': No space left on device
352+0 records in
351+0 records out
36909240320 bytes (37 GB) copied, 3496.8 seconds, 10.6 MB/s
[root at rs1 ~]#

list again free space usage:

[root at rs1 ~]# lfs df -h
UUID                     bytes      Used Available  Use% Mounted on
testfs-MDT0000_UUID     130.4G    460.0M    122.5G    0% /mnt/lustre[MDT:0]
testfs-OST0000_UUID      18.3G      1.4G     16.0G    7% /mnt/lustre[OST:0]
testfs-OST0001_UUID      18.3G    444.5M     17.0G    2% /mnt/lustre[OST:1]
testfs-OST0002_UUID      36.7G     34.8G      4.0M   94% /mnt/lustre[OST:2]
testfs-OST0003_UUID      36.7G    448.5M     34.4G    1% /mnt/lustre[OST:3]
filesystem summary:     110.0G     37.1G     67.4G   33% /mnt/lustre
[root at rs1 ~]#

As you see, testfs-OST0002_UUID is full, but on /mnt/lustre I have enough free 
space - 67GB (on other OSTs) in order to allow dd to continue and create my 
file. How can i avoid dd to stop and allow it to continue to complete on 
other OSTs?

Regards,
Alx