[Lustre-discuss] Size of MDT, used space
Thomas Roth
t.roth at gsi.de
Wed May 14 07:41:50 UTC 2008
Andreas Dilger wrote:
> On May 13, 2008 19:41 +0200, Thomas Roth wrote:
>> I'm still in trouble with numbers: the available, used and necessary
>> space on my MDT:
>> According to "lfs df", I have now filled my file system with 115.3 TB.
>> All of these files are sized 5 MB. That should be roughly 24 million files.
>> For the MDT, "lfs df" reports 28.2 GB used.
>>
>> Now I believed that creating a file on Lustre means using one inode on
>> the MDT. Since all of my Lustre partitions were formatted with the
>> default options (all of this is running Lustre v. 1.6.4.3, btw), an
>> inode should eat up 4kB on the MDT partition. Of course, 24 million
>> files times 4 kB gives you 91 GB rather than 28GB.
>> Obviously, there is something I missed completely. Perhaps somebody
>> could illuminate me here?
>>
>> This issue could also be phrased as "How large should my MDT be to
>> accommodate n TB storage space?" The manual's answer boils down to "=
>> number of files * 4 kB " (*2 per recommendation). That's how I
>> calculated above - maybe my test system is broken? I can't check on the
>> content of these files, it's just 5MB test files created with the
>> 'stress' utility.
>
> Please provide output of "lfs df" and "lfs df -i" so we can see the
> actual numbers.
>
> Cheers, Andreas
> --
> Andreas Dilger
> Sr. Staff Engineer, Lustre Group
> Sun Microsystems of Canada, Inc.
>
Ok, I'm attaching these two files with the output. You'll notice some
OSTs with a lower fill level: these OSS had some hardware problems
during the production of my (now counted) 22.7 million files.
Greetings,
Thomas
-------------- next part --------------
# lfs df
UUID 1K-blocks Used Available Use% Mounted on
gsilust1-MDT0000_UUID 495497804 29575416 465922388 5% /lustre[MDT:0]
gsilust1-OST0000_UUID 2403044744 2208409576 194635168 91% /lustre[OST:0]
gsilust1-OST0001_UUID 2883653852 2648332660 235321192 91% /lustre[OST:1]
gsilust1-OST0002_UUID 2403044744 2131046760 271997984 88% /lustre[OST:2]
gsilust1-OST0003_UUID 2883653852 2557946316 325707536 88% /lustre[OST:3]
gsilust1-OST0004_UUID 2403044744 2206128516 196916228 91% /lustre[OST:4]
gsilust1-OST0005_UUID 2883653852 2647658256 235995596 91% /lustre[OST:5]
gsilust1-OST0006_UUID 2403044744 2197323104 205721640 91% /lustre[OST:6]
gsilust1-OST0007_UUID 2883653852 2636058844 247595008 91% /lustre[OST:7]
gsilust1-OST0008_UUID 2403044744 2207045972 195998772 91% /lustre[OST:8]
gsilust1-OST0009_UUID 2883653852 2648436060 235217792 91% /lustre[OST:9]
gsilust1-OST000a_UUID 2403044744 2205203944 197840800 91% /lustre[OST:10]
gsilust1-OST000b_UUID 2883653852 2648915248 234738604 91% /lustre[OST:11]
gsilust1-OST000c_UUID 2403044744 2206725516 196319228 91% /lustre[OST:12]
gsilust1-OST000d_UUID 2883653852 1449668056 1433985796 50% /lustre[OST:13]
gsilust1-OST000e_UUID 2403044744 2206148640 196896104 91% /lustre[OST:14]
gsilust1-OST000f_UUID 2883653852 2648466960 235186892 91% /lustre[OST:15]
gsilust1-OST0010_UUID 2403044744 2205981044 197063700 91% /lustre[OST:16]
gsilust1-OST0011_UUID 2883653852 2648466180 235187672 91% /lustre[OST:17]
gsilust1-OST0012_UUID 2403044744 2197113560 205931184 91% /lustre[OST:18]
gsilust1-OST0013_UUID 2883653852 2638146024 245507828 91% /lustre[OST:19]
gsilust1-OST0014_UUID 2403044744 2206067992 196976752 91% /lustre[OST:20]
gsilust1-OST0015_UUID 2883653852 2647881580 235772272 91% /lustre[OST:21]
gsilust1-OST0016_UUID 2403044744 2205237108 197807636 91% /lustre[OST:22]
gsilust1-OST0017_UUID 2883653852 2646574192 237079660 91% /lustre[OST:23]
gsilust1-OST0018_UUID 2403044744 2205484856 197559888 91% /lustre[OST:24]
gsilust1-OST0019_UUID 2883653852 2645678088 237975764 91% /lustre[OST:25]
gsilust1-OST001a_UUID 2403044744 2206248696 196796048 91% /lustre[OST:26]
gsilust1-OST001b_UUID 2883653852 2647041328 236612524 91% /lustre[OST:27]
gsilust1-OST001c_UUID 2403044744 2207710496 195334248 91% /lustre[OST:28]
gsilust1-OST001d_UUID 2883653852 2649304508 234349344 91% /lustre[OST:29]
gsilust1-OST001e_UUID 2403044744 2206033636 197011108 91% /lustre[OST:30]
gsilust1-OST001f_UUID 2883653852 2648367388 235286464 91% /lustre[OST:31]
gsilust1-OST0020_UUID 2403044744 2207774816 195269928 91% /lustre[OST:32]
gsilust1-OST0021_UUID 2883653852 2649212840 234441012 91% /lustre[OST:33]
gsilust1-OST0022_UUID 2403044744 2203817988 199226756 91% /lustre[OST:34]
gsilust1-OST0023_UUID 2883653852 2646795736 236858116 91% /lustre[OST:35]
gsilust1-OST0024_UUID 2403044744 1868659544 534385200 77% /lustre[OST:36]
gsilust1-OST0025_UUID 2883653852 2243676508 639977344 77% /lustre[OST:37]
gsilust1-OST0026_UUID 2403044744 2205775772 197268972 91% /lustre[OST:38]
gsilust1-OST0027_UUID 2883653852 2647277724 236376128 91% /lustre[OST:39]
gsilust1-OST0028_UUID 2403044744 2208664348 194380396 91% /lustre[OST:40]
gsilust1-OST0029_UUID 2883653852 2647154096 236499756 91% /lustre[OST:41]
gsilust1-OST002a_UUID 2403044744 2206053488 196991256 91% /lustre[OST:42]
gsilust1-OST002b_UUID 2883653852 2647567536 236086316 91% /lustre[OST:43]
gsilust1-OST002c_UUID 2403044744 2197169140 205875604 91% /lustre[OST:44]
gsilust1-OST002d_UUID 2883653852 2636726236 246927616 91% /lustre[OST:45]
gsilust1-OST002e_UUID 2403044744 2196736784 206307960 91% /lustre[OST:46]
gsilust1-OST002f_UUID 2883653852 2636158280 247495572 91% /lustre[OST:47]
gsilust1-OST0030_UUID 2403044744 2127152780 275891964 88% /lustre[OST:48]
gsilust1-OST0031_UUID 2883653852 2555100512 328553340 88% /lustre[OST:49]
gsilust1-OST0032_UUID 2403044744 2205456928 197587816 91% /lustre[OST:50]
gsilust1-OST0033_UUID 2883653852 2650130140 233523712 91% /lustre[OST:51]
filesystem summary: 137454163496 123851912300 13602251196 90% /lustre
-------------- next part --------------
# lfs df -i
UUID Inodes IUsed IFree IUse% Mounted on
gsilust1-MDT0000_UUID 141590528 22816486 118774042 16% /lustre[MDT:0]
gsilust1-OST0000_UUID 79583043 407308 79175735 0% /lustre[OST:0]
gsilust1-OST0001_UUID 95939053 488426 95450627 0% /lustre[OST:1]
gsilust1-OST0002_UUID 98908655 392216 98516439 0% /lustre[OST:2]
gsilust1-OST0003_UUID 118518048 470835 118047213 0% /lustre[OST:3]
gsilust1-OST0004_UUID 80152929 406929 79746000 0% /lustre[OST:4]
gsilust1-OST0005_UUID 96107538 488310 95619228 0% /lustre[OST:5]
gsilust1-OST0006_UUID 82352633 405280 81947353 0% /lustre[OST:6]
gsilust1-OST0007_UUID 99005270 486189 98519081 0% /lustre[OST:7]
gsilust1-OST0008_UUID 79923681 407045 79516636 0% /lustre[OST:8]
gsilust1-OST0009_UUID 95913223 488446 95424777 0% /lustre[OST:9]
gsilust1-OST000a_UUID 80383880 406737 79977143 0% /lustre[OST:10]
gsilust1-OST000b_UUID 95793561 488581 95304980 0% /lustre[OST:11]
gsilust1-OST000c_UUID 80003809 407059 79596750 0% /lustre[OST:12]
gsilust1-OST000d_UUID 183107584 254523 182853061 0% /lustre[OST:13]
gsilust1-OST000e_UUID 80147843 406874 79740969 0% /lustre[OST:14]
gsilust1-OST000f_UUID 95905520 488468 95417052 0% /lustre[OST:15]
gsilust1-OST0010_UUID 80189759 406891 79782868 0% /lustre[OST:16]
gsilust1-OST0011_UUID 95905738 488491 95417247 0% /lustre[OST:17]
gsilust1-OST0012_UUID 82404960 405221 81999739 0% /lustre[OST:18]
gsilust1-OST0013_UUID 98483893 486607 97997286 0% /lustre[OST:19]
gsilust1-OST0014_UUID 80168024 406893 79761131 0% /lustre[OST:20]
gsilust1-OST0015_UUID 96051801 488404 95563397 0% /lustre[OST:21]
gsilust1-OST0016_UUID 80375578 406726 79968852 0% /lustre[OST:22]
gsilust1-OST0017_UUID 96378384 488140 95890244 0% /lustre[OST:23]
gsilust1-OST0018_UUID 80313665 406750 79906915 0% /lustre[OST:24]
gsilust1-OST0019_UUID 96602213 487943 96114270 0% /lustre[OST:25]
gsilust1-OST001a_UUID 80122878 406923 79715955 0% /lustre[OST:26]
gsilust1-OST001b_UUID 96261703 488243 95773460 0% /lustre[OST:27]
gsilust1-OST001c_UUID 79757758 407253 79350505 0% /lustre[OST:28]
gsilust1-OST001d_UUID 95696293 488628 95207665 0% /lustre[OST:29]
gsilust1-OST001e_UUID 80176610 406890 79769720 0% /lustre[OST:30]
gsilust1-OST001f_UUID 95930453 488508 95441945 0% /lustre[OST:31]
gsilust1-OST0020_UUID 79741629 407204 79334425 0% /lustre[OST:32]
gsilust1-OST0021_UUID 95719241 488659 95230582 0% /lustre[OST:33]
gsilust1-OST0022_UUID 80730098 406466 80323632 0% /lustre[OST:34]
gsilust1-OST0023_UUID 96323031 488173 95834858 0% /lustre[OST:35]
gsilust1-OST0024_UUID 152592384 341011 152251373 0% /lustre[OST:36]
gsilust1-OST0025_UUID 183107584 409411 182698173 0% /lustre[OST:37]
gsilust1-OST0026_UUID 80241000 406814 79834186 0% /lustre[OST:38]
gsilust1-OST0027_UUID 96202598 488237 95714361 0% /lustre[OST:39]
gsilust1-OST0028_UUID 79519409 407367 79112042 0% /lustre[OST:40]
gsilust1-OST0029_UUID 96233492 488224 95745268 0% /lustre[OST:41]
gsilust1-OST002a_UUID 80171622 406865 79764757 0% /lustre[OST:42]
gsilust1-OST002b_UUID 96130241 488333 95641908 0% /lustre[OST:43]
gsilust1-OST002c_UUID 82391112 405268 81985844 0% /lustre[OST:44]
gsilust1-OST002d_UUID 98838567 486334 98352233 0% /lustre[OST:45]
gsilust1-OST002e_UUID 82499023 405090 82093933 0% /lustre[OST:46]
gsilust1-OST002f_UUID 98980406 486184 98494222 0% /lustre[OST:47]
gsilust1-OST0030_UUID 99881387 391453 99489934 0% /lustre[OST:48]
gsilust1-OST0031_UUID 119228929 470265 118758664 0% /lustre[OST:49]
gsilust1-OST0032_UUID 80320717 406820 79913897 0% /lustre[OST:50]
gsilust1-OST0033_UUID 95490008 488751 95001257 0% /lustre[OST:51]
filesystem summary: 141590528 22816486 118774042 16% /lustre
More information about the lustre-discuss
mailing list