[Lustre-discuss] Applications of Lustre - streaming?

[Jason Brooks]

It is the question of how to handle redundancy that stops me from
immediately testing this idea of mine.  Well, that and time, werewithal,
etc...

Hadoop is great because it uses the speed and latency of local disks to
work with data, and does not require systems be homogeneous.  With the
data replicated, it not only has the storage redundancy, but also x-1
hosts that can work with the data if a host goes down.

The down side appears to be that hadoop can't really handle many nodes
going down ungracefully.  My personal goal with my idea is to make a host
a "dumb compute node" that I can shutdown with impunity.


On 12/7/12 9:37 AM, "Jon Yeargers" <yeargers at ohsu.edu> wrote:

>The redundancy of HDFS is very appealing. I've been weighing the merits
>of this vs a RAID-6 / server on Lustre. HDFS recommends avoiding RAID for
>the very reason that the data is (typically) saved in several locations.
>
>-----Original Message-----
>From: Dilger, Andreas [mailto:andreas.dilger at intel.com]
>Sent: Friday, December 07, 2012 9:35 AM
>To: Jon Yeargers
>Cc: lustre-discuss at lists.lustre.org
>Subject: Re: [Lustre-discuss] Applications of Lustre - streaming?
>
>On 2012-12-07, at 10:26, Jon Yeargers
><yeargers at ohsu.edu<mailto:yeargers at ohsu.edu>> wrote:
>Can Lustre be used to store data like streaming audio / video? I¹ve been
>scolded about considering it for DB storage but I¹m looking at the
>relative merits of Lustre vs HDFS.
>
>I've been using Lustre for years with my home MythTV (Linux PVR) setup.
>The only major change I made was to reduce the readahead window size so
>that there wasn't lag when videos first start playing due to the large
>readahead window being filled.
>
>Of course, the suitability for a given workload depends on the hardware
>being used. Lustre will definitely give you better performance for the
>same hardware than HDFS, but if you need highly available data, the
>storage needs to be able to failover between servers.
>
>Cheers, Andreas
>
>I¹m moving to a clustered DB setup and wondering about Cassandra / Lustre
>vs Hadoop (IE HBase / HDFS). One offers flexibility in terms of mixing
>hardware components while the other is a Œone stop shop¹.
>
>Not trying to elicit a religious war ­ and yes, I¹ve been reading as much
>as I can find about this. Just hoping for the opinion(s) of this side of
>the table.
>_______________________________________________
>Lustre-discuss mailing list
>Lustre-discuss at lists.lustre.org<mailto:Lustre-discuss at lists.lustre.org>
>http://lists.lustre.org/mailman/listinfo/lustre-discuss
>_______________________________________________
>Lustre-discuss mailing list
>Lustre-discuss at lists.lustre.org
>http://lists.lustre.org/mailman/listinfo/lustre-discuss