[Lustre-discuss] Anybody actually using Flash (Fusion IO specifically) for meta data?

[Dardo D Kleiner - CONTRACTOR]

Short answer: of course it works - they're just block devices after all - but you'll find that you won't realize the performance gains you might expect (at least not for an MDT).

Aside from simply being fast OSTs, there are several areas that would allow Lustre to take advantage of these kinds of devices:

1) SMP scaling for the MDS - the problem right now is that the low latency of these devices really shines best when you have many threads scattering small I/O.  The current (1.8.x) Lustre MDS doesn't 
do this.
2) Flashcache/bcache over traditional disk storage (OST or MDT) - this can be done today, of course.  There's some interop issues in my testing, but when it works it does what it says it does.  It 
still won't really help an MDT though.
3) Targeted device mapping of the metadata portions of an OST on traditional disk (e.g. extent lists) onto flash.

#1 is substantial work (ongoing I believe).  #2 is pretty nifty, basically grow your local page cache beyond RAM - helps when "hot" working set is large.  #3 is trickier and though I haven't tried it 
I understand there's real effort ongoing in this regard.

Filesystem size in this discussion is mostly irrelevant for an MDT, its just whether or not the device is big enough for the number of objects (a few million is *not* many).  A huge number of clients 
thrashing about creating/modifying/deleting is where these things have the most potential.

- Dardo

On 5/16/11 2:58 PM, Carlson, Timothy S wrote:
>
> Folks,
>
> I know that flash based technology gets talked about from time to time on the list, but I was wondering if anybody has actually implemented FusionIO devices for metadata. The last thread I can find on the mailing list that relates to this topic dates from 3 years ago. The software driving the Fusion cards has come quite a ways since then and I've got good experience using the device as a raw disk. I'm just fishing around to see if anybody has implemented one of these devices in a reasonably sized Lustre config where "reasonably" is left open to interpretation. I'm thinking>500T and a few million files.
>
> Thanks!
>
> Tim
>
>
> _______________________________________________
> Lustre-discuss mailing list
> Lustre-discuss at lists.lustre.org
> http://lists.lustre.org/mailman/listinfo/lustre-discuss
>
>