[Lustre-discuss] Lustre - usage / best practices

Francois Chassaing fch at weborama.com
Thu Nov 8 03:36:59 PST 2012

about your DB usage of the Lustre FS, I would strongly advise you to follow Andreas advice and test first.
Note also that you must use a recent Percona-released MySQL to avoid a nasty POSIX-related bug in Innodb that crashes MySQL...

In my case, performance showed to be dramatically insufficent. small random I/O and meta-data performances are a killer for MySQL.

I could probably optimze in a number of other ways that what I did, or throw a whole lot of additional hardware at it, but after several month of tweaking (both lustre and mysql), we've just got away from it and returned to plain direct-attached storage (and sharding). 

Nevertheless, I took the time to run post-mortem benchs with mysql+lustre (for other untested optimzations, notably RAID configs of the underlying storage). And also with standard tools (dd, sgp_dd, bonnie++) on other DFS. 
To make a long story short : 
1/ our usage of MySQL is not compatible with Lustre. You should check your is !
2/ Lustre is the best-performing DFS I could test, given then hardware we had (tested PVFS2/OrangeFS, GlusterFS, FHGFS/FraunhoferFS)
(NB : FHGFS is second-best by far. OrangeFS showed stability issues, Gluster is stable but is a joke...)

Hardware : 
- a pair of HA MDS/MGS with a shared MDT on a iscsi array of RAID10 15k disks (note : dedicated gigE switch)
- 3 OSSes with 2 RAID1 'os' disks + 6 RAIDed 7.2K SAS disks (originally was a RAID10)
- connected with DDR IB (+ gigE for 'legacy' clients)

Hope that helps.

François Chassaing Directeur Technique - CTO 
weborama.com - fch at weborama.com 

----- Mail Original -----
De: "Andreas Dilger" <adilger at whamcloud.com>
À: "Jon Yeargers" <yeargers at ohsu.edu>
Cc: lustre-discuss at lists.lustre.org
Envoyé: Jeudi 8 Novembre 2012 02h19:30 GMT +01:00 Amsterdam / Berlin / Berne / Rome / Stockholm / Vienne
Objet: Re: [Lustre-discuss] Lustre - usage / best practices

On 2012-11-04, at 8:54 AM, Jon Yeargers wrote:
> I’ll soon be retiring a number of old file servers and have the opportunity to move to a DFS based system. Im wondering about using Lustre for ‘mixed purposes’ and what the recommended setup is.
> More specifically: lets say there are three groups that want to use the space. One group is primarily storing Tb of data in large numbers of flat files.

So, perfect use case for Lustre.

> A second group is using a DB like MySQL that has fewer but larger files.

This winds up being small random IO to the file.  Not a common workload for Lustre, so I would really recommend testing it out on a small system first.

> A third may have some other process. Would I divide up my DFS space somehow or is it ok for all three groups to comingle? Would I do something on the Lustre clients to force them apart or just rely on Linux security to restrict access to the client folders (on the lustre client)?

The Lustre servers have essentially the same security model as NFS today, so they depend on the clients to enforce user identification.  If malicious users can configure their own client nodes and connect them to the filesystem, then they could access any files in the filesystem.  If the client nodes are secure, then Lustre will properly enforce user access permissions/ACLs to the files.

> I can imagine creating a single Lustre client and then NIS mounting the various volumes on separate machines for each group. This seems like adding an extra step but perhaps it’s the normal pattern?

Do you mean "NFS" when you wrote "NIS"?  I'm not sure why this would be better, but I can definitely think of reasons why it would be worse.

Cheers, Andreas
Andreas Dilger                       Whamcloud, Inc.
Principal Lustre Engineer            http://www.whamcloud.com/

Lustre-discuss mailing list
Lustre-discuss at lists.lustre.org

More information about the lustre-discuss mailing list