[Lustre-discuss] lustre1.6.7: inodes problem
Shigeru Sugimoto
lustre.shigeru at gmail.com
Mon Mar 23 00:10:16 PDT 2009
Hello,
I am having a problem regarding inodes. My storage has more than 36TB space
and I tried to copy less than 1TB data(total size is less than 1TB but more
than 200,000 files, each of those are around 1MB-10MB).
In the middle of the rsync process to copy the data, it suddenly started to
display "No space left on device", though there are almost 99% of the
storage is still free.
I have checked the inodes with df -i option and got the result below.
[root at lustre-client1 mnt]# lfs df -i
UUID Inodes IUsed IFree IUse% Mounted on
user-MDT0000_UUID 393216 393216 0 100% /mnt/lustre[MDT:0]
user-OST0000_UUID 91570176 14999 91555177 0% /mnt/lustre[OST:0]
user-OST0001_UUID 91570176 14999 91555177 0% /mnt/lustre[OST:1]
user-OST0002_UUID 91570176 14999 91555177 0% /mnt/lustre[OST:2]
user-OST0003_UUID 91570176 14999 91555177 0% /mnt/lustre[OST:3]
user-OST0004_UUID 91570176 14999 91555177 0% /mnt/lustre[OST:4]
user-OST0005_UUID 91570176 14999 91555177 0% /mnt/lustre[OST:5]
user-OST0006_UUID 91570176 15000 91555176 0% /mnt/lustre[OST:6]
user-OST0007_UUID 91570176 15000 91555176 0% /mnt/lustre[OST:7]
user-OST0008_UUID 91570176 15000 91555176 0% /mnt/lustre[OST:8]
user-OST0009_UUID 91570176 15000 91555176 0% /mnt/lustre[OST:9]
user-OST000a_UUID 91570176 15000 91555176 0% /mnt/lustre[OST:10]
user-OST000b_UUID 91570176 15000 91555176 0% /mnt/lustre[OST:11]
user-OST000c_UUID 91570176 15000 91555176 0% /mnt/lustre[OST:12]
user-OST000d_UUID 91570176 15000 91555176 0% /mnt/lustre[OST:13]
user-OST000e_UUID 91570176 15000 91555176 0% /mnt/lustre[OST:14]
user-OST000f_UUID 91570176 14968 91555208 0% /mnt/lustre[OST:15]
user-OST0010_UUID 91570176 14968 91555208 0% /mnt/lustre[OST:16]
user-OST0011_UUID 91570176 14968 91555208 0% /mnt/lustre[OST:17]
user-OST0012_UUID 91570176 14968 91555208 0% /mnt/lustre[OST:18]
user-OST0013_UUID 91570176 14968 91555208 0% /mnt/lustre[OST:19]
user-OST0014_UUID 91570176 14968 91555208 0% /mnt/lustre[OST:20]
user-OST0015_UUID 91570176 14968 91555208 0% /mnt/lustre[OST:21]
user-OST0016_UUID 91570176 14968 91555208 0% /mnt/lustre[OST:22]
user-OST0017_UUID 91570176 14968 91555208 0% /mnt/lustre[OST:23]
My configurations are below.
- Lustre version: 1.6.7
- One physical MGS and one physical OSS.
- The MGS is doing MDT role, cofigured by "mkfs.lustre --mgs --mdt
--fsname=user --reformat /dev/VolGroup00/mdt"
- The OSS consists of 24 OSTs, configured by "mkfs.lustre --ost
--msgnode=servername at tcp --fsname=user --reformat /dev/sda" ( the same
command executed for /dev/sdb to /dev/sdx)
So I'm using the default setting for inodes (i.e. --mkfsoptions=-i 4096, 1
inode per 4096 bytes of filesystem space).
- Available storage size of MGS is 29GB, as below.
[root at mgs ~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
29G 2.0G 26G 8% /
/dev/cciss/c0d0p1 99M 16M 78M 17% /boot
none 1014M 0 1014M 0% /dev/shm
/dev/VolGroup00/mdt 1.4G 63M 1.2G 5% /mnt/mdt
I was trying to change the size of inode but the minimum size is
1024bytes/inode, so it would just get four times bigger number of inodes
compared to the current configuration, with the current hardware.
Would anyone here help me to correct my configuration in order to handle
thousands of files with the current MGS?
Thanks for your time,
Shigeru
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20090323/f3fcca82/attachment.htm>
More information about the lustre-discuss
mailing list