[Lustre-discuss] no file creation on not yet full Lustre

[Thomas Roth]

Hi all,

on a Lustre FS v1.6.5 with Debian Etch kernel 2.6.22 that is 94% full, I 
can't create any more files.
Each OST has still room for  ~ 2GB:

#>  lfs df
UUID                 1K-blocks      Used Available  Use% Mounted on
MDT0000_UUID 495497804    669376 466511924    0% /lustre[MDT:0]
OST0000_UUID 2440596776 2316586108   1942896   94% /lustre[OST:0]
OST0001_UUID 2928715484 2779554112   2680056   94% /lustre[OST:1]
OST0002_UUID 2440596776 2316342644   2186360   94% /lustre[OST:2]
OST0003_UUID 2928715484 2779758712   2475456   94% /lustre[OST:3]
OST0004_UUID: inactive device
OST0005_UUID: inactive device
OST0006_UUID 2440596776 2316400736   2128268   94% /lustre[OST:6]
OST0007_UUID 2928715484 2779972968   2261196   94% /lustre[OST:7]
OST0008_UUID 2440596776 2316306052   2214696   94% /lustre[OST:8]
OST0009_UUID 2928715484 2779558976   2666004   94% /lustre[OST:9]

filesystem summary:  21477249040 20384480308  18554932   94% /lustre


And it's not the amount of inodes, either:

#> lfs df -i
UUID                    Inodes     IUsed     IFree IUse% Mounted on
MDT0000_UUID 128657164   4950058 123707106    3% /lustre[MDT:0]
OST0000_UUID   2384256    452839   1931417   18% /lustre[OST:0]
OST0001_UUID   2861056    543317   2317739   18% /lustre[OST:1]
OST0002_UUID   2384256    452336   1931920   18% /lustre[OST:2]
OST0003_UUID   2861056    542826   2318230   18% /lustre[OST:3]
OST0004_UUID: inactive device
OST0005_UUID: inactive device
OST0006_UUID   2384256    452735   1931521   18% /lustre[OST:6]
OST0007_UUID   2861056    543414   2317642   18% /lustre[OST:7]
OST0008_UUID   2384256    452335   1931921   18% /lustre[OST:8]
OST0009_UUID   2861056    542851   2318205   18% /lustre[OST:9]

filesystem summary:  128657164   4950058 123707106    3% /lustre

The MDT has a log message (possibly) on this:

LustreError: 4586:0:(mds_open.c:416:mds_create_objects()) error creating 
objects for inode 110854263: rc = -5

Later on, there are more Errors like
LustreError: 4581:0:(mds_open.c:710:mds_finish_open()) 
mds_create_objects: rc = -28

There is also rc = -22. By the way, does anybody know what these numbers 
mean/where to look them up? In my search through the net I once again 
stumbled over  ras.lustre.org and error.lustre.org but  without success. 
It's a pity, I must by now have Gigabytes of LustreError messages but I 
have never found what part to paste there...

The error while trying to write was:
~# dd if=/dev/zero of=/lustre/Null
dd: opening `/lustre/Null': Input/output error

I tried to make a new file go to a specific OST:
~# lfs setstripe /lustre/a -i 3
error on ioctl 0x4008669a for '/lustre/a' (3): No space left on device
error: setstripe: create stripe file failed

On the OSTs I don't see any corresponding entry in the logs.

There is an old bug that might be related: 
bugzilla.lustre.org/show_bug.cgi?id=12768 .  At least it left similar 
traces in the log. But it has been  resolved  a year ago.

So, anybody has got an idea what goes wrong here?

Many thanks,
Thomas