[Lustre-discuss] lustre1.6.7: inodes problem

[Robin Humble]

On Mon, Mar 23, 2009 at 04:10:16PM +0900, Shigeru Sugimoto wrote:
>I am having a problem regarding inodes. My storage has more than 36TB space
>and I tried to copy less than 1TB data(total size is less than 1TB but more
>than 200,000 files, each of those are around 1MB-10MB).
>I have checked the inodes with df -i option and got the result below.
>
>[root at lustre-client1 mnt]# lfs df -i
>UUID                    Inodes     IUsed     IFree IUse% Mounted on
>user-MDT0000_UUID       393216    393216         0  100% /mnt/lustre[MDT:0]
...

as you have worked out, you have run out of inodes on your MDT.

>- Available storage size of MGS is 29GB, as below.

>[root at mgs ~]# df -h
>Filesystem            Size  Used Avail Use% Mounted on
>/dev/mapper/VolGroup00-LogVol00
>                       29G  2.0G   26G   8% /
>/dev/cciss/c0d0p1      99M   16M   78M  17% /boot
>none                 1014M     0 1014M   0% /dev/shm
>/dev/VolGroup00/mdt   1.4G   63M  1.2G   5% /mnt/mdt

your MDT is only 1.4G in size. you have 29G in that LVM setup (which
isn't much), but you have allocated it mostly to the root partition. if
you are constrained not to add any more storage to this machine then I
suggest you reallocate it so that ~5g is for root and the majority if
for the MDT.
most likely this requires rebuilding your MDS node and your Lustre
filesystem from scratch.

sorry if that's not much help to you.

the Lustre manual has words about MDT inode sizes if you want to tweak
that, but as you say, that might get you 4x better density, but overall
it looks like you need ~30x the inodes you have now if you are going to
fill up your whole fs with files of the same size. so you'll need more
than a 1.4G of MDT storage for that.

cheers,
robin