[Lustre-discuss] lustre1.6.7: inodes problem

[Shigeru Sugimoto]

Hello,

So to fix the situation below, I can switch the MGS's HDD from 29GB to
750GB, which I guess would simply give me 25 times larger space for
inodes.
But before taking that way, I would like to make sure if there's any
other(better) solution or configuration to fix my problem below.

Thanks,

Shigeru

2009/3/23 Shigeru Sugimoto <lustre.shigeru at gmail.com>
>
> Hello,
>
> I am having a problem regarding inodes. My storage has more than 36TB space and I tried to copy less than 1TB data(total size is less than 1TB but more than 200,000 files, each of those are around 1MB-10MB).
> In the middle of the rsync process to copy the data, it suddenly started to display "No space left on device", though there are almost 99% of the storage is still free.
> I have checked the inodes with df -i option and got the result below.
>
> [root at lustre-client1 mnt]# lfs df -i
> UUID                    Inodes     IUsed     IFree IUse% Mounted on
> user-MDT0000_UUID       393216    393216         0  100% /mnt/lustre[MDT:0]
> user-OST0000_UUID     91570176     14999  91555177    0% /mnt/lustre[OST:0]
> user-OST0001_UUID     91570176     14999  91555177    0% /mnt/lustre[OST:1]
> user-OST0002_UUID     91570176     14999  91555177    0% /mnt/lustre[OST:2]
> user-OST0003_UUID     91570176     14999  91555177    0% /mnt/lustre[OST:3]
> user-OST0004_UUID     91570176     14999  91555177    0% /mnt/lustre[OST:4]
> user-OST0005_UUID     91570176     14999  91555177    0% /mnt/lustre[OST:5]
> user-OST0006_UUID     91570176     15000  91555176    0% /mnt/lustre[OST:6]
> user-OST0007_UUID     91570176     15000  91555176    0% /mnt/lustre[OST:7]
> user-OST0008_UUID     91570176     15000  91555176    0% /mnt/lustre[OST:8]
> user-OST0009_UUID     91570176     15000  91555176    0% /mnt/lustre[OST:9]
> user-OST000a_UUID     91570176     15000  91555176    0% /mnt/lustre[OST:10]
> user-OST000b_UUID     91570176     15000  91555176    0% /mnt/lustre[OST:11]
> user-OST000c_UUID     91570176     15000  91555176    0% /mnt/lustre[OST:12]
> user-OST000d_UUID     91570176     15000  91555176    0% /mnt/lustre[OST:13]
> user-OST000e_UUID     91570176     15000  91555176    0% /mnt/lustre[OST:14]
> user-OST000f_UUID     91570176     14968  91555208    0% /mnt/lustre[OST:15]
> user-OST0010_UUID     91570176     14968  91555208    0% /mnt/lustre[OST:16]
> user-OST0011_UUID     91570176     14968  91555208    0% /mnt/lustre[OST:17]
> user-OST0012_UUID     91570176     14968  91555208    0% /mnt/lustre[OST:18]
> user-OST0013_UUID     91570176     14968  91555208    0% /mnt/lustre[OST:19]
> user-OST0014_UUID     91570176     14968  91555208    0% /mnt/lustre[OST:20]
> user-OST0015_UUID     91570176     14968  91555208    0% /mnt/lustre[OST:21]
> user-OST0016_UUID     91570176     14968  91555208    0% /mnt/lustre[OST:22]
> user-OST0017_UUID     91570176     14968  91555208    0% /mnt/lustre[OST:23]
>
> My configurations are below.
> - Lustre version: 1.6.7
> - One physical MGS and one physical OSS.
> - The MGS is doing MDT role, cofigured by "mkfs.lustre --mgs --mdt --fsname=user --reformat /dev/VolGroup00/mdt"
> - The OSS consists of 24 OSTs, configured by "mkfs.lustre --ost --msgnode=servername at tcp --fsname=user --reformat /dev/sda" ( the same command executed for /dev/sdb to /dev/sdx)
> So I'm using the default setting for inodes (i.e. --mkfsoptions=-i 4096, 1 inode per 4096 bytes of filesystem space).
> - Available storage size of MGS is 29GB, as below.
>
> [root at mgs ~]# df -h
> Filesystem            Size  Used Avail Use% Mounted on
> /dev/mapper/VolGroup00-LogVol00
>                        29G  2.0G   26G   8% /
> /dev/cciss/c0d0p1      99M   16M   78M  17% /boot
> none                 1014M     0 1014M   0% /dev/shm
> /dev/VolGroup00/mdt   1.4G   63M  1.2G   5% /mnt/mdt
>
> I was trying to change the size of inode but the minimum size is 1024bytes/inode, so it would just get four times bigger number of inodes compared to the current configuration, with the current hardware.
>
> Would anyone here help me to correct my configuration in order to handle thousands of files with the current MGS?
>
> Thanks for your time,
>
> Shigeru
>