[Lustre-devel] [Lustre-discuss] Research on filesystem metadata operation distribution
Mark Nelson
mark at msi.umn.edu
Thu Apr 21 14:47:22 PDT 2011
Hi Andreas,
Here are the statistics for our current lustre storage as requested.
This filesystem is used for scratch on a ~8700 core cluster.
> [root at mds ~]# lctl get_param mds.*.stats | egrep "open|close|rename|link|attr|sync"
> open 52365106 samples [reqs]
> close 26309366 samples [reqs]
> link 164 samples [reqs]
> unlink 4199061 samples [reqs]
> rename 605499 samples [reqs]
> getattr 204467294 samples [reqs]
> setattr 6947314 samples [reqs]
> getxattr 1211618173 samples [reqs]
> sync 2399 samples [reqs]
> mark at node1083:~> lfs df
> UUID 1K-blocks Used Available Use% Mounted on
> lustre-MDT0000_UUID 2446151592 7290552 2299069836 0% /scratch1[MDT:0]
> lustre-OST0000_UUID 2850686364 2260170480 445708680 79% /scratch1[OST:0]
> lustre-OST0001_UUID 2850686364 2360436928 345415236 82% /scratch1[OST:1]
> lustre-OST0002_UUID 2850686364 2301852200 404026800 80% /scratch1[OST:2]
> lustre-OST0003_UUID 2850686364 2297681284 408197928 80% /scratch1[OST:3]
> lustre-OST0004_UUID 2850686364 2268630672 437220700 79% /scratch1[OST:4]
> lustre-OST0005_UUID 2850686364 2352652832 353226340 82% /scratch1[OST:5]
> lustre-OST0006_UUID 2850686364 2256628780 449250944 79% /scratch1[OST:6]
> lustre-OST0007_UUID 2850686364 2239905176 465895280 78% /scratch1[OST:7]
> lustre-OST0008_UUID 2850686364 2256707044 449145120 79% /scratch1[OST:8]
> lustre-OST0009_UUID 2850686364 2318356564 387467160 81% /scratch1[OST:9]
> lustre-OST000a_UUID 2850686364 2246170292 459654224 78% /scratch1[OST:10]
> lustre-OST000b_UUID 2850686364 2289467904 416357844 80% /scratch1[OST:11]
> lustre-OST000c_UUID 2850686364 2267773568 438077736 79% /scratch1[OST:12]
> lustre-OST000d_UUID 2850686364 2370072892 335806104 83% /scratch1[OST:13]
> lustre-OST000e_UUID 2850686364 2373615788 332262332 83% /scratch1[OST:14]
> lustre-OST000f_UUID 2850686364 2268974064 436784684 79% /scratch1[OST:15]
> lustre-OST0010_UUID 2850686364 2254083524 451795496 79% /scratch1[OST:16]
> lustre-OST0011_UUID 2850686364 2305220652 400636828 80% /scratch1[OST:17]
> lustre-OST0012_UUID 2850686364 2289688148 416189404 80% /scratch1[OST:18]
> lustre-OST0013_UUID 2850686364 2248032400 457846596 78% /scratch1[OST:19]
> lustre-OST0014_UUID 2850686364 2348553416 357297956 82% /scratch1[OST:20]
> lustre-OST0015_UUID 2850686364 2336745344 369133676 81% /scratch1[OST:21]
> lustre-OST0016_UUID 2850686364 2286710092 419095976 80% /scratch1[OST:22]
> lustre-OST0017_UUID 2850686364 2258930636 446948364 79% /scratch1[OST:23]
> lustre-OST0018_UUID 2850686364 2394833760 311045260 84% /scratch1[OST:24]
> lustre-OST0019_UUID 2850686364 2315306792 390571460 81% /scratch1[OST:25]
> lustre-OST001a_UUID 2850686364 2300589812 405289220 80% /scratch1[OST:26]
> lustre-OST001b_UUID 2850686364 2296248192 409542792 80% /scratch1[OST:27]
> lustre-OST001c_UUID 2850686364 2306921556 398957656 80% /scratch1[OST:28]
> lustre-OST001d_UUID 2850686364 2277275716 428541804 79% /scratch1[OST:29]
> lustre-OST001e_UUID 2850686364 2290738376 415141384 80% /scratch1[OST:30]
> lustre-OST001f_UUID 2850686364 2358853432 347025588 82% /scratch1[OST:31]
> lustre-OST0020_UUID 2850686364 2164837628 541041392 75% /scratch1[OST:32]
> lustre-OST0021_UUID 2850686364 2250094264 455757108 78% /scratch1[OST:33]
> lustre-OST0022_UUID 2850686364 2318377076 387474296 81% /scratch1[OST:34]
> lustre-OST0023_UUID 2850686364 2287996640 417827084 80% /scratch1[OST:35]
> lustre-OST0024_UUID 2850686364 2266655604 439213268 79% /scratch1[OST:36]
> lustre-OST0025_UUID 2850686364 2196015760 509804940 77% /scratch1[OST:37]
> lustre-OST0026_UUID 2850686364 2326710900 379141252 81% /scratch1[OST:38]
> lustre-OST0027_UUID 2850686364 2263182244 442696788 79% /scratch1[OST:39]
> lustre-OST0028_UUID 2850686364 2281411520 424466108 80% /scratch1[OST:40]
> lustre-OST0029_UUID 2850686364 2253024788 452854156 79% /scratch1[OST:41]
> lustre-OST002a_UUID 2850686364 2265263264 440614844 79% /scratch1[OST:42]
> lustre-OST002b_UUID 2850686364 2358109672 347769300 82% /scratch1[OST:43]
> lustre-OST002c_UUID 2850686364 2246612120 459253300 78% /scratch1[OST:44]
> lustre-OST002d_UUID 2850686364 2315081464 390671376 81% /scratch1[OST:45]
> lustre-OST002e_UUID 2850686364 2151456488 554423308 75% /scratch1[OST:46]
> lustre-OST002f_UUID 2850686364 2276389044 429401272 79% /scratch1[OST:47]
>
> filesystem summary: 136832945472 109819046792 20061966364 80% /scratch1
> mark at node1083:~> lfs df -i
> UUID Inodes IUsed IFree IUse% Mounted on
> lustre-MDT0000_UUID 623065492 13350254 609715238 2% /scratch1[MDT:0]
> lustre-OST0000_UUID 147951373 356246 147595127 0% /scratch1[OST:0]
> lustre-OST0001_UUID 122874520 348299 122526221 0% /scratch1[OST:1]
> lustre-OST0002_UUID 137525462 310049 137215413 0% /scratch1[OST:2]
> lustre-OST0003_UUID 138550809 340738 138210071 0% /scratch1[OST:3]
> lustre-OST0004_UUID 145867558 363872 145503686 0% /scratch1[OST:4]
> lustre-OST0005_UUID 124846336 353476 124492860 0% /scratch1[OST:5]
> lustre-OST0006_UUID 148769303 334806 148434497 0% /scratch1[OST:6]
> lustre-OST0007_UUID 153043111 338572 152704539 0% /scratch1[OST:7]
> lustre-OST0008_UUID 148836688 343526 148493162 0% /scratch1[OST:8]
> lustre-OST0009_UUID 133406354 332875 133073479 0% /scratch1[OST:9]
> lustre-OST000a_UUID 151428889 318410 151110479 0% /scratch1[OST:10]
> lustre-OST000b_UUID 140643612 370513 140273099 0% /scratch1[OST:11]
> lustre-OST000c_UUID 146024187 317395 145706792 0% /scratch1[OST:12]
> lustre-OST000d_UUID 120508125 334975 120173150 0% /scratch1[OST:13]
> lustre-OST000e_UUID 119591403 335426 119255977 0% /scratch1[OST:14]
> lustre-OST000f_UUID 145755615 354062 145401553 0% /scratch1[OST:15]
> lustre-OST0010_UUID 149476374 334318 149142056 0% /scratch1[OST:16]
> lustre-OST0011_UUID 136596942 304167 136292775 0% /scratch1[OST:17]
> lustre-OST0012_UUID 140548883 326959 140221924 0% /scratch1[OST:18]
> lustre-OST0013_UUID 150984278 329164 150655114 0% /scratch1[OST:19]
> lustre-OST0014_UUID 125850214 335954 125514260 0% /scratch1[OST:20]
> lustre-OST0015_UUID 128803314 340503 128462811 0% /scratch1[OST:21]
> lustre-OST0016_UUID 141332100 356012 140976088 0% /scratch1[OST:22]
> lustre-OST0017_UUID 148282098 332665 147949433 0% /scratch1[OST:23]
> lustre-OST0018_UUID 114305646 341206 113964440 0% /scratch1[OST:24]
> lustre-OST0019_UUID 134162367 320108 133842259 0% /scratch1[OST:25]
> lustre-OST001a_UUID 137846405 346482 137499923 0% /scratch1[OST:26]
> lustre-OST001b_UUID 138954376 354556 138599820 0% /scratch1[OST:27]
> lustre-OST001c_UUID 136261269 314506 135946763 0% /scratch1[OST:28]
> lustre-OST001d_UUID 143665519 346163 143319356 0% /scratch1[OST:29]
> lustre-OST001e_UUID 140292861 310260 139982601 0% /scratch1[OST:30]
> lustre-OST001f_UUID 123343915 374877 122969038 0% /scratch1[OST:31]
> lustre-OST0020_UUID 171817157 349887 171467270 0% /scratch1[OST:32]
> lustre-OST0021_UUID 150499898 352977 150146921 0% /scratch1[OST:33]
> lustre-OST0022_UUID 133410450 335784 133074666 0% /scratch1[OST:34]
> lustre-OST0023_UUID 140965613 325669 140639944 0% /scratch1[OST:35]
> lustre-OST0024_UUID 146302421 339776 145962645 0% /scratch1[OST:36]
> lustre-OST0025_UUID 163987206 343710 163643496 0% /scratch1[OST:37]
> lustre-OST0026_UUID 131322985 361117 130961868 0% /scratch1[OST:38]
> lustre-OST0027_UUID 147155706 296609 146859097 0% /scratch1[OST:39]
> lustre-OST0028_UUID 142674349 344009 142330340 0% /scratch1[OST:40]
> lustre-OST0029_UUID 149723181 332111 149391070 0% /scratch1[OST:41]
> lustre-OST002a_UUID 146671256 320469 146350787 0% /scratch1[OST:42]
> lustre-OST002b_UUID 123435148 350587 123084561 0% /scratch1[OST:43]
> lustre-OST002c_UUID 151317772 343224 150974548 0% /scratch1[OST:44]
> lustre-OST002d_UUID 134212577 353558 133859019 0% /scratch1[OST:45]
> lustre-OST002e_UUID 175115293 326239 174789054 0% /scratch1[OST:46]
> lustre-OST002f_UUID 143864174 326506 143537668 0% /scratch1[OST:47]
>
> filesystem summary: 623065492 13350254 609715238 2% /scratch1
--
Mark Nelson, HPC Systems Administrator
Minnesota Supercomputing Institute
Phone: (612)626-4479
Email: mark at msi.umn.edu
More information about the lustre-devel
mailing list