[lustre-discuss] getting without inodes
Carlos Adean
carlosadean at linea.org.br
Mon Aug 14 10:34:54 PDT 2023
First of all, thank you for the support.
The targets are LDdiskFS and below you can see some outputs.
I have no idea why these large inodes were chosen and I'm understanding
that the only option to fix it is the process of migrating and reformatting
OSTs. Sounds easy, but I've never done this before.
I was looking for the log of installation and I found some commands like
this. From mpatha to mpathp the command is the same and only the index is
incremented +1.
mkfs.lustre --reformat --ost --fsname t1 --index 0 --mgsnode
10.148.0.20 at o2ib1 --mgsnode 10.148.0.21 at o2ib1 --servicenode
10.148.0.22 at o2ib1 --servicenode 10.148.0.23 at o2ib1 --backfstype=ldiskfs
/dev/mapper/mpatha
Permanent disk data:
Target: t1:OST0000
Index: 0
Lustre FS: t1
Mount type: ldiskfs
Flags: 0x1062
(OST first_time update no_primnode )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=10.148.0.20 at o2ib1:10.148.0.21 at o2ib1
failover.node=10.148.0.22 at o2ib1:10.148.0.23 at o2ib1
device size = 30492028MB
formatting backing filesystem ldiskfs on /dev/mapper/mpatha
target name t1:OST0000
kilobytes 31223836672
options -J size=400 -I 512 -i 1048576 -q -O
extents,uninit_bg,mmp,dir_nlink,quota,huge_file,64bit,flex_bg -G 256 -E
lazy_journal_init -F
mkfs_cmd = mke2fs -j -b 4096 -L t1:OST0000 -J size=400 -I 512 -i 1048576
-q -O extents,uninit_bg,mmp,dir_nlink,quota,huge_file,64bit,flex_bg -G 256
-E lazy_journal_init -F /dev/mapper/mpatha 31223836672k
Writing CONFIGS/mountdata
$ lfs df
UUID 1K-blocks Used Available Use% Mounted on
t0-MDT0000_UUID 577014664 655944 527530716 1%
/lustre/t0[MDT:0]
t0-OST0000_UUID 3713444212 909645492 2616245424 26%
/lustre/t0[OST:0]
t0-OST0001_UUID 3713444212 899171460 2626719456 26%
/lustre/t0[OST:1]
t0-OST0002_UUID 3713444212 876364492 2649526424 25%
/lustre/t0[OST:2]
t0-OST0003_UUID 3713444212 885720364 2640170552 26%
/lustre/t0[OST:3]
t0-OST0004_UUID 3713444212 888714516 2637176400 26%
/lustre/t0[OST:4]
t0-OST0005_UUID 3713444212 894227612 2631663304 26%
/lustre/t0[OST:5]
t0-OST0006_UUID 3713444212 899596028 2626294888 26%
/lustre/t0[OST:6]
t0-OST0007_UUID 3713444212 886533188 2639357728 26%
/lustre/t0[OST:7]
t0-OST0008_UUID 3713444212 897605864 2628285052 26%
/lustre/t0[OST:8]
t0-OST0009_UUID 3713444212 882908908 2642982008 26%
/lustre/t0[OST:9]
t0-OST000a_UUID 3713444212 890232056 2635658860 26%
/lustre/t0[OST:10]
t0-OST000b_UUID 3713444212 736843200 2789047716 21%
/lustre/t0[OST:11]
t0-OST000c_UUID 3713444212 740349224 2785541692 21%
/lustre/t0[OST:12]
t0-OST000d_UUID 3713444212 734080364 2791810552 21%
/lustre/t0[OST:13]
t0-OST000e_UUID 3713444212 743070588 2782820328 22%
/lustre/t0[OST:14]
t0-OST000f_UUID 3713444212 736546656 2789344260 21%
/lustre/t0[OST:15]
t0-OST0010_UUID 3713444212 733151712 2792739204 21%
/lustre/t0[OST:16]
t0-OST0011_UUID 3713444212 727708628 2798182288 21%
/lustre/t0[OST:17]
t0-OST0012_UUID 3713444212 731904760 2793986156 21%
/lustre/t0[OST:18]
t0-OST0013_UUID 3713444212 726399548 2799491368 21%
/lustre/t0[OST:19]
t0-OST0014_UUID 3713444212 735012680 2790878236 21%
/lustre/t0[OST:20]
t0-OST0015_UUID 3713444212 730765704 2795125212 21%
/lustre/t0[OST:21]
filesystem_summary: 81695772664 17886553044 59683047108 24% /lustre/t0
UUID 1K-blocks Used Available Use% Mounted on
t1-MDT0000_UUID 2196097232 23183624 1988074236 2%
/lustre/t1[MDT:0]
t1-OST0000_UUID 30962092008 8846988896 20553894896 31%
/lustre/t1[OST:0]
t1-OST0001_UUID 30962092008 7991470000 21409413792 28%
/lustre/t1[OST:1]
t1-OST0002_UUID 30962092008 7637904380 21762979412 26%
/lustre/t1[OST:2]
t1-OST0003_UUID 30962092008 10149520884 19251362908 35%
/lustre/t1[OST:3]
t1-OST0004_UUID 30962092008 8144756200 21256127592 28%
/lustre/t1[OST:4]
t1-OST0005_UUID 30962092008 7779247180 21621636612 27%
/lustre/t1[OST:5]
t1-OST0006_UUID 30962092008 7622756520 21778127272 26%
/lustre/t1[OST:6]
t1-OST0007_UUID 30962092008 7643234260 21757649532 26%
/lustre/t1[OST:7]
t1-OST0008_UUID 30962092008 10553974584 18846909208 36%
/lustre/t1[OST:8]
t1-OST0009_UUID 30962092008 10375327460 19025556332 36%
/lustre/t1[OST:9]
t1-OST000a_UUID 30962092008 10299326580 19101557212 36%
/lustre/t1[OST:10]
t1-OST000b_UUID 30962092008 10350395396 19050488396 36%
/lustre/t1[OST:11]
t1-OST000c_UUID 30962092008 10334395612 19066488180 36%
/lustre/t1[OST:12]
t1-OST000d_UUID 30962092008 10330851228 19070032564 36%
/lustre/t1[OST:13]
t1-OST000e_UUID 30962092008 10306664120 19094219672 36%
/lustre/t1[OST:14]
t1-OST000f_UUID 30962092008 10282787736 19118096056 35%
/lustre/t1[OST:15]
filesystem_summary: 495393472128 148649601036 321764539636 32% /lustre/t1
Cheers,
--
Carlos Adean
www.linea.org.br
Em sex., 11 de ago. de 2023 às 18:05, Andreas Dilger <adilger at whamcloud.com>
escreveu:
> The t0 filesystem OSTs are formatted for an average file size of 70TB /
> 300M inodes = 240KB/inode.
>
> The t1 filesystem OSTs are formatted for an average file size of 500TB /
> 65M inodes = 7.7MB/inode.
>
> So not only are the t1 OSTs larger, but they have fewer inodes (by a
> factor of 32x). This must have been done with specific formatting options
> since the default inode ratio is 1MiB/inode for the OSTs.
>
> There isn't any information about the actual space usage (eg. "lfs df"),
> so I can't calculate whether the default 1MiB/inode would be appropriate
> for your filesystem, but definitely it was formatted with the expectation
> that the average file size would become larger as they were copied to t1
> (eg. combined in a tarfile or something).
>
> Unfortunately, there is no way to "fix" this in place, since the inode
> ratio for ldiskfs/ext4 filesystems is decided at format time.
>
> One option is to use "lfs find" to find files on an OST (eg. OST0003 which
> is the least used), disable creates on that OST, and use "lfs migrate" to
> migrate all of the files to other OSTs, and then reformat the OST with more
> inodes and repeat this process for all of the OSTs.
>
> Unfortunately, the t1 filesystem only has 8.5M free inodes and there are
> 27M inodes on OST0003, so it couldn't be drained completely to perform this
> process. You would need to delete enough files from t1 to free up inodes to
> do the migration, or eg. tar them up into larger files to reduce the inode
> count.
>
> The OST migration/replacement process is described in the Lustre
> Operations Manual.
>
> Cheers, Andreas
>
> On Aug 11, 2023, at 01:17, Carlos Adean via lustre-discuss <
> lustre-discuss at lists.lustre.org> wrote:
>
>
> Hello experts,
>
> We have a Lustre with two tiers T0(SSD) and T1(HDD), the first with 70TB
> and the second one with ~500TB.
>
> I'm experiencing a problem that the T1 has much less inodes than T0 and
> that is getting without inodes in the OSTs, so I'd like to understand the
> source of this and how to fix that.
>
>
> Thanks in advance.
>
>
>
> === T0
>
> $ lfs df -i /lustre/t0
> UUID Inodes IUsed IFree IUse% Mounted on
> t0-MDT0000_UUID 390627328 1499300 389128028 1%
> /lustre/t0[MDT:0]
> t0-OST0000_UUID 14651392 1097442 13553950 8%
> /lustre/t0[OST:0]
> t0-OST0001_UUID 14651392 1097492 13553900 8%
> /lustre/t0[OST:1]
> t0-OST0002_UUID 14651392 1097331 13554061 8%
> /lustre/t0[OST:2]
> t0-OST0003_UUID 14651392 1097563 13553829 8%
> /lustre/t0[OST:3]
> t0-OST0004_UUID 14651392 1097576 13553816 8%
> /lustre/t0[OST:4]
> t0-OST0005_UUID 14651392 1097505 13553887 8%
> /lustre/t0[OST:5]
> t0-OST0006_UUID 14651392 1097524 13553868 8%
> /lustre/t0[OST:6]
> t0-OST0007_UUID 14651392 1097596 13553796 8%
> /lustre/t0[OST:7]
> t0-OST0008_UUID 14651392 1097442 13553950 8%
> /lustre/t0[OST:8]
> t0-OST0009_UUID 14651392 1097563 13553829 8%
> /lustre/t0[OST:9]
> t0-OST000a_UUID 14651392 1097515 13553877 8%
> /lustre/t0[OST:10]
> t0-OST000b_UUID 14651392 1096524 13554868 8%
> /lustre/t0[OST:11]
> t0-OST000c_UUID 14651392 1096608 13554784 8%
> /lustre/t0[OST:12]
> t0-OST000d_UUID 14651392 1096524 13554868 8%
> /lustre/t0[OST:13]
> t0-OST000e_UUID 14651392 1096641 13554751 8%
> /lustre/t0[OST:14]
> t0-OST000f_UUID 14651392 1096647 13554745 8%
> /lustre/t0[OST:15]
> t0-OST0010_UUID 14651392 1096705 13554687 8%
> /lustre/t0[OST:16]
> t0-OST0011_UUID 14651392 1096616 13554776 8%
> /lustre/t0[OST:17]
> t0-OST0012_UUID 14651392 1096520 13554872 8%
> /lustre/t0[OST:18]
> t0-OST0013_UUID 14651392 1096598 13554794 8%
> /lustre/t0[OST:19]
> t0-OST0014_UUID 14651392 1096669 13554723 8%
> /lustre/t0[OST:20]
> t0-OST0015_UUID 14651392 1096570 13554822 8%
> /lustre/t0[OST:21]
>
> filesystem_summary: 299694753 1499300 298195453 1% /lustre/t0
>
>
> === T1
>
> $ lfs df -i /lustre/t1
> UUID Inodes IUsed IFree IUse% Mounted on
> t1-MDT0000_UUID 1478721536 56448788 1422272748 4% /lustre/t1[MDT:0]
> t1-OST0000_UUID 30492032 30491899 133 100% /lustre/t1[OST:0]
> t1-OST0001_UUID 30492032 30491990 42 100% /lustre/t1[OST:1]
> t1-OST0002_UUID 30492032 30491916 116 100% /lustre/t1[OST:2]
> t1-OST0003_UUID 30492032 27471050 3020982 91% /lustre/t1[OST:3]
> t1-OST0004_UUID 30492032 30491989 43 100% /lustre/t1[OST:4]
> t1-OST0005_UUID 30492032 30491960 72 100% /lustre/t1[OST:5]
> t1-OST0006_UUID 30492032 30491948 84 100% /lustre/t1[OST:6]
> t1-OST0007_UUID 30492032 30491939 93 100% /lustre/t1[OST:7]
> t1-OST0008_UUID 30492032 29811803 680229 98% /lustre/t1[OST:8]
> t1-OST0009_UUID 30492032 29808261 683771 98% /lustre/t1[OST:9]
> t1-OST000a_UUID 30492032 29809919 682113 98% /lustre/t1[OST:10]
> t1-OST000b_UUID 30492032 29807585 684447 98% /lustre/t1[OST:11]
> t1-OST000c_UUID 30492032 29809171 682861 98% /lustre/t1[OST:12]
> t1-OST000d_UUID 30492032 29804206 687826 98% /lustre/t1[OST:13]
> t1-OST000e_UUID 30492032 29806399 685633 98% /lustre/t1[OST:14]
> t1-OST000f_UUID 30492032 29802857 689175 98% /lustre/t1[OST:15]
>
> filesystem_summary: 64946408 56448788 8497620 87% /lustre/t1
>
>
>
>
> --
> Carlos Adean
> www.linea.org.br
>
>
> Site <https://www.linea.org.br/> | Twitter
> <https://twitter.com/LIneA_org> | Instagram
> <https://www.instagram.com/linea_org/> | Facebook
> <https://www.facebook.com/linea.org>
> *Laboratório Interinstitucional de e-Astronomia *
> _______________________________________________
> lustre-discuss mailing list
> lustre-discuss at lists.lustre.org
> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20230814/32996ce2/attachment-0001.htm>
More information about the lustre-discuss
mailing list