[Lustre-discuss] Question about No space left on device

[Chad Kerner]

Hello,

      We are running lustre 1.6.6 and are seeing a weird error on space 
usage.  The filesystem is not anywhere near full, but writes are failing 
if they hit OST 23.

if I do an lfs setstripe -i 23 chad, and then do
# dd if=/dev/zero of=chad
dd: writing to `chad': No space left on device
26+0 records in
25+0 records out
#

The actual device is fairly full.
# df /lustre/home/ost_h_24
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/mapper/ost_h_24 564172088 509452368  26061444  96% 
/lustre/home/ost_h_24
#

The striping is set as:
stripe_count: 2 stripe_size: 1048576 stripe_offset: -1

So there should be enough space to still create that stripe.  Was 
wondering if anyone else sees this type of error and if so, how can you 
get around it?

# lfs df /u
UUID                 1K-blocks      Used Available  Use% Mounted on
home-MDT0000_UUID    250741088   7437416 228974536    2% /u[MDT:0]
home-OST0000_UUID    564172088 465752856  69742520   82% /u[OST:0]
home-OST0001_UUID    564172088 481188596  54304660   85% /u[OST:1]
home-OST0002_UUID    564172088 476474964  59007024   84% /u[OST:2]
home-OST0003_UUID    564172088 477867116  57604028   84% /u[OST:3]
home-OST0004_UUID    564172088 467160140  68338168   82% /u[OST:4]
home-OST0005_UUID    564172088 460648340  74822768   81% /u[OST:5]
home-OST0006_UUID    564172088 461577612  73936192   81% /u[OST:6]
home-OST0007_UUID    564172088 463904716  71582888   82% /u[OST:7]
home-OST0008_UUID    564172088 490786452  67629072   86% /u[OST:8]
home-OST0009_UUID    564172088 494448544  41036872   87% /u[OST:9]
home-OST000a_UUID    564172088 469107600  66406208   83% /u[OST:10]
home-OST000b_UUID    564172088 471937184  63550148   83% /u[OST:11]
home-OST000c_UUID    564172088 470872456  64614456   83% /u[OST:12]
home-OST000d_UUID    564172088 468447640  67066156   83% /u[OST:13]
home-OST000e_UUID    564172088 461661040  73823812   81% /u[OST:14]
home-OST000f_UUID    564172088 472427452  63086356   83% /u[OST:15]
home-OST0010_UUID    564172088 465760292  69712320   82% /u[OST:16]
home-OST0011_UUID    564172088 463973452  71515716   82% /u[OST:17]
home-OST0012_UUID    564172088 478922600  56568572   84% /u[OST:18]
home-OST0013_UUID    564172088 488618676  46890016   86% /u[OST:19]
home-OST0014_UUID    564172088 472108048  63382768   83% /u[OST:20]
home-OST0015_UUID    564172088 489621560  45874056   86% /u[OST:21]
home-OST0016_UUID    564172088 475357600  83082828   84% /u[OST:22]
home-OST0017_UUID    564172088 509437736  26069524   90% /u[OST:23]

filesystem summary:  13540130112 11398062672 1499647128   84% /u


Thanks,
Chad