[lustre-discuss] lustre for home directories

[Einar Næss Jensen]

Hello Rick

On Wed, Apr 25, 2018 at 09:09:44AM -0700, Riccardo Veraldi wrote:
> Hello,
> just wondering if who is using lustre for home directories with several
> users is happy or not.

lustre: 2.5.42.8

I can give you my (our) two cents. 
We have some experience with current setup:
10.158.253.21 at o2ib:10.158.253.22 at o2ib:/work  395T  297T   95T  76% /work
10.158.253.12 at o2ib:10.158.253.13 at o2ib:/prod  198T  157T   39T  81% /prod2
10.158.253.12 at o2ib:10.158.253.13 at o2ib:/home  113T   36T   77T  32% /home

/work is the "scratch" file system for general useres
/home is home directory
/prod2 is a scratch/home directory specifically for weather forecast/climate codes

/home and /prod2 share mds and oss servers (but not mdt or ost)

/home and /prod is on DDN sfa10K config
and /work on separate sfa10k config

We have had very few complaints from regular users, but weather forecast runtimes have, from time to time been suffering when too many users doing compilations, rsyncs recursive copy/rm/mv etc.

So my short advice would be: Don't mix usage on same servers.


> I am considering to move home directories from NFS to Lustre/ZFS.
> it is quite easy to send the NFS server in troubles with just a few
> users copying files around.
> What special tuning is needed to optimize Lustre usage with small files
> ? I guess 1M record size wold not be a good choice anymore.

We did not change any settings except setting default stripe count = 1. We were bold and tried 4, but it was not a good choice. Turned it was not a good option for the other file systems either, as we have mixed workload, and anything else than 1 was close to disaster for applications creating millions of small files.


I'm currently also considering switching from NFS to Lustre home directory on another (smaller) system we have, as I'm having trouble with NFS over (mellanox) infiniband.

ps: if you're at LUG18 we could have a chat if you want.



Best regards
Einar


-- 
-- 
Einar Næss Jensen
NTNU HPC Section
Norwegian University of Science and Technoloy
Address: Høgskoleringen 7i
         N-7491 Trondheim, NORWAY
tlf:     +47 90990249
email:   einar.nass.jensen at ntnu.no