[lustre-discuss] How to speed up Lustre

[Marion Hakanson]

We started with DoM on our new Lustre system a couple years ago.
  - Converting to DoM (or away from it) is a full-reformat operation.
  - DoM uses a fixed amount of metadata space (64k minimum for us) for every file, even those smaller than 64k.

Basically, DoM uses a lot of flash metadata space, more than we planned for, and more than we could afford.

We ended up switching to a PFL arrangement, where the first 64k lives on flash OST's (mounted on our metadata servers), and the remainder of larger files lives on HDD OST's.  This is working very well for our small-file workloads, and uses less flash space than the DoM configuration did.

Since you don't already have DoM in effect, it may be possible that you could add flash OST's, configure a PFL, and then use "lfs migrate" to re-layout existing files into the new OST's.  Your mileage may vary, so be safe!

Regards,

Marion


On Feb 14, 2022, at 03:32, Taner KARAGÖL via lustre-discuss <lustre-discuss at lists.lustre.org> wrote:


UNCLASSIFIED

Hi Everybody;

We have a performance problem with small files on our HPC system (120 compute nodes). Our all OSS targets are classic spinning HDDs. To speed up, I want to configure Data on Metadata. Our metadata target has SDD disks.

Underlying file systems are ZFS (for OSS and Meta)
Lustre version: 2.12.5
ZFS version: .0.7.13

Our Lustre file system size is 720TB (2 OSS servers, 1 enclosure with 6 zpools), Metadata file system size is 2.1TB(1 enclosure and 1 metadata target).

What is your opinions to speed up this setup? I want to configure DoM but I am concerning about Metadata size. My questions:

  1.  How can I increase Medatadata size? Metadata enclosure has a empty slots. Is there a way to increase size online/offline?
  2.  Is it possible to migrate big files from DoM to OSS targets completely? Off course online migration. (So I think I can free Metadata for new small files).

Best Regards;
Taner

________________________________
Dikkat:

Bu elektronik posta mesaji kisisel ve ozeldir. Eger size gonderilmediyse lutfen gondericiyi bilgilendirip mesaji siliniz. Firmamiza gelen ve giden mesajlar virus taramasindan gecirilmekte, guvenlik nedeni ile kontrol edilerek saklanmaktadir. Mesajdaki gorusler ve bakis acisi gondericiye ait olup Aselsan A.S. resmi gorusu olmak zorunda degildir.

________________________________
Attention:

This e-mail message is privileged and confidential. If you are not the intended recipient please delete the message and notify the sender. E-mails to and from the company are monitored for operational reasons and in accordance with lawful business practices. Any views or opinions presented are solely those of the author and do not necessarily represent the views of the company.

________________________________



_______________________________________________
lustre-discuss mailing list
lustre-discuss at lists.lustre.org
https://urldefense.com/v3/__http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org__;!!Mi0JBg!bW2FnSTRNdX7DpkjIiMayeexmYJ3D5Xt7wtneny2zgGi1ZXPcy7QMRlM3mno-HWR$
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20220217/56d4cb42/attachment-0001.html>