[Lustre-discuss] Problem with lustre 2.0.0.1, ext3/4 and big OSTs (>8Tb)

Kevin Van Maren kevin.van.maren at oracle.com
Tue Mar 15 08:22:58 PDT 2011


Joan J. Piles wrote:
> Hi,
>
> We are trying to set up a lustre 2.0.0.1 (the most recent one 
> downladable from the offiecial site) installation. We plan to have some 
> big OSTs (~ 12Tb), using ScientificLinux 5.5 (which should be a RHEL 
> clone for all purposes).
>
> However, when we try to format the OSTs, we get the following error:
>
>   
>> [root at oss01 ~]# mkfs.lustre --ost --fsname=extra 
>> --mgsnode=172.16.4.4 at tcp0 --mkfsoptions '-i 262144 -E 
>> stride=32,stripe_width=192 ' /dev/sde
>>
>>    Permanent disk data:
>> Target:     extra-OSTffff
>> Index:      unassigned
>> Lustre FS:  extra
>> Mount type: ldiskfs
>> Flags:      0x72
>>               (OST needs_index first_time update )
>> Persistent mount opts: errors=remount-ro,extents,mballoc
>> Parameters: mgsnode=172.16.4.4 at tcp
>>
>> checking for existing Lustre data: not found
>> device size = 11427830MB
>> formatting backing filesystem ldiskfs on /dev/sde
>>     target name  extra-OSTffff
>>     4k blocks     2925524480
>>     options       -i 262144 -E stride=32,stripe_width=192  -J size=400 
>> -I 256 -q -O dir_index,extents,uninit_bg -F
>> mkfs_cmd = mke2fs -j -b 4096 -L extra-OSTffff -i 262144 -E 
>> stride=32,stripe_width=192  -J size=400 -I 256 -q -O 
>> dir_index,extents,uninit_bg -F /dev/sde 2925524480
>> mkfs.lustre: Unable to mount /dev/sde: Invalid argument
>>
>> mkfs.lustre FATAL: failed to write local files
>> mkfs.lustre: exiting with 22 (Invalid argument)
>>     
>
>
> In the dmesg log, we find the following line:
>
>   
>> LDISKFS-fs does not support filesystems greater than 8TB and can cause 
>> data corruption.Use "force_over_8tb" mount option to override.
>>     
>
> After some investigation, we find it is related to the use of ext3 
> instead of ext4, 

Correct.

> even though we should be using ext4, proven by the fact 
> that the file systems created are actually ext4:
>
>   
>> [root at oss01 ~]# file -s /dev/sde
>> /dev/sde: Linux rev 1.0 ext4 filesystem data (extents) (large files)
>>     

No, these are "ldiskfs" filesystems.  ext3+ldiskfs looks a bit like ext4 
(ext4 is largely based on the
enhancements done for Lustre's ldiskfs), but is not the same as 
ext4+ldiskfs.  In particular, file system
size is limited to 8TB, not 16TB.

> Further, we made a test with an ext3 filesystem in the same machine, and 
> the difference is found:
>
>   
>> [root at oss01 ~]# file -s /dev/sda1
>> /dev/sda1: Linux rev 1.0 ext3 filesystem data (large files)
>>     
>
> Everything we found in the net about this problem seems to refer to 
> lustre 1.8.5. However, we would not expect such a regression in lustre 
> 2. Is this actually a problem with lustre 2? Has ext4 to be enabled 
> either at compile time or with a parameter somewhere (we found no 
> documentation about it)?
>   

Lustre 2.0 did not enable ext4 by default, due to known issues.  You can 
rebuild the Lustre server,
with "--enable-ext4" on the configure line, to enable it.  But if you 
are going to use 12TB LUNs,
you should either sick with v1.8.5 (stable), or pull a newer version 
from git (experimental).

Kevin




More information about the lustre-discuss mailing list