[Lustre-discuss] ioDrive from Fusion-IO as an MDT?

[Craig Prescott]

Thanks for the replies and insights, Chris and Klaus!

What had attracted us to the ioDrive, particularly in the role
of MDT, was the extremely high IOPS figures it claims for small
I/Os.

With our current mirrored-SATA disk MDTs, we find that when
pdflush runs on the MDS during metadata-intense activities, the
MDT becomes 100% utilized with tiny I/Os, and await skyrockets
while the disks are swamped by all the requests pdflush is pushing
to disk.  While this is happening, the metadata-intense processes
grind to a halt, as do interactive activities like opening files
with editors, etc.  Everything gets moving again once the pdflush
is done, but this can last for tens of seconds, and our users do
notice this.  We've tuned the VM some on our MDS to mitigate this,
but we'd like to make this behavior go away completely.  If we can.

The metadata-intense activities I refer to are usually due to our
users managing their data interactively (i.e., removing enormous
numbers of dinky files).  Some of the applications they run do
I/O inefficiently, too (lots of
stat()/open()/write(tinyamount)/close()).  It seems like a more
capable MDT would help out such "bad" jobs, as well.

Anyway, that is the source of our interest in the ioDrive as an MDT.
During pdflush, we see less than 10MB/s of I/O being done to the MDT;
if the ioDrive can do lots more than that at a much higher IOPS rating,
it would seem that it would make a very nice MDT.

Cheers,
Craig

Chris Worley wrote:
> I've visited their headquarters, and they're a very much Linux-savvy
> organization.
> 
> As an ext3 or XFS fs, they get 700MB/s out of one ioDrive... with
> three in a single system MD raided together they were getting 2GB/s
> out of IOZone atop an XFS fs.  It is just presented as a block device.
> 
> It's an associative memory, so they have a special fs tailored to
> using it (and bypass the fs unnecessarily turning names into block
> addresses).  That fs gets 800MB/s under IOZone (with one "disk").
> Since the fs layer is somewhat independent in Lustre, I thought that
> might be a great optimization for this hardware.  They probably won't
> focus on the HPC market, as they are probably going after bigger
> fish... so this might be something the community has to do if it's
> going to get done.
> 
> Big issues w/ SSD's include bottlenecks in controllers meant for
> spinning drives, and extra device driver layers that do unnecessary
> steps like elevator code (which SSD or the ioDrive don't need as seek
> time is consistent).  The ioDrive doesn't carry or need this extra
> baggage.
> 
> I'd like to see some experiment w/ Lustre and ioDrives... especially
> in the area of minimizing stripe sizes... since these drives don't
> need to work w/ big blocks to get optimal performance.
> 
> Chris
> On Fri, Aug 8, 2008 at 12:53 PM, Klaus Steden <klaus.steden at thomson.net> wrote:
>> Hi Craig,
>>
>> Assuming it presents itself as a regular LUN to the OS, I don't think it
>> would appear to operate any differently than a physical drive. I've always
>> been extremely curious to try solid-state disks in a high-performance
>> environment, so if anyone's used one of these, I'm dying to hear how it
>> worked for you.
>>
>> cheers,
>> Klaus
>>
>> On 8/8/08 9:31 AM, "Craig Prescott" <prescott at hpc.ufl.edu>did etch on stone
>> tablets:
>>
>>> Hi all;
>>>
>>> Has anyone ever tried using a Fusion-IO ioDrive as
>>> a Lustre MDT?
>>>
>>> http://www.fusionio.com/products.aspx
>>>
>>> If so, I'd love to hear your experiences, particularly
>>> if you've used one in a production filesystem.  We're using
>>> a mirrored pair of SATA disks at present for our MDTs,
>>> and interactive response can get pretty crummy when
>>> our users perform metadata-heavy workloads.  It seems
>>> this product could help us out in that area.
>>>
>>> Thanks,
>>> Craig Prescott
>>> UF HPC Center
>>> _______________________________________________
>>> Lustre-discuss mailing list
>>> Lustre-discuss at lists.lustre.org
>>> http://lists.lustre.org/mailman/listinfo/lustre-discuss
>> _______________________________________________
>> Lustre-discuss mailing list
>> Lustre-discuss at lists.lustre.org
>> http://lists.lustre.org/mailman/listinfo/lustre-discuss
>>