[Lustre-discuss] Applications of Lustre - streaming?

[Jon Yeargers]

If it weren’t for the positive aspects of HDFS I wouldn’t really be considering HBase (over Cassandra).

Any notion of the merits of Lustre’s kernel-based mounts vs a FUSE-based mount (HDFS)? Whichever filesystem I go with I will need to store ‘flat files’ in.

From: Jason Brooks
Sent: Friday, December 07, 2012 9:38 AM
To: Jon Yeargers; lustre-discuss at lists.lustre.org
Subject: Re: [Lustre-discuss] Applications of Lustre - streaming?

Hello,

The question of hdfs storage via lustre has been in the foreground of my thinking.  the hadoop hdfs processes are not aware of block devices: they only know of a filesystem mount point to begin storing data in hdfs.  THUS…

If we provide a filesystem interface (say a lustre mount point) whose latencies and throughput approach that of local disk storage (say, via infiniband), could we not have the various hadoop nodes store their data in the lustre filesystem?  would hadoop even care?

I realize that this may not be a good place to bring it up.  But there you go…

One of these days, (with all of my ample spare time), I will benchmark it.  and report of course…

--jason

From: Jon Yeargers <yeargers at ohsu.edu<mailto:yeargers at ohsu.edu>>
Date: Friday, December 7, 2012 9:26 AM
To: "lustre-discuss at lists.lustre.org<mailto:lustre-discuss at lists.lustre.org>" <lustre-discuss at lists.lustre.org<mailto:lustre-discuss at lists.lustre.org>>
Subject: [Lustre-discuss] Applications of Lustre - streaming?

Can Lustre be used to store data like streaming audio / video? I’ve been scolded about considering it for DB storage but I’m looking at the relative merits of Lustre vs HDFS.

I’m moving to a clustered DB setup and wondering about Cassandra / Lustre vs Hadoop (IE HBase / HDFS). One offers flexibility in terms of mixing hardware components while the other is a ‘one stop shop’.

Not trying to elicit a religious war – and yes, I’ve been reading as much as I can find about this. Just hoping for the opinion(s) of this side of the table.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20121207/a43e1e02/attachment.htm>