[Lustre-discuss] Can lustre be trusted to keep my data safe?

[Mike Berg]

I should added to my original response, that being a customer once  
many years ago, it is always a good idea to have a second copy of your  
most important/expensive data no matter what a vendor says :-). That  
said, I understand in the large data world, this can be costly.


On May 14, 2008, at 3:21 PM, jrs wrote:

> Thanks for the insight, Jim (and Mike and Aaron),
>
> Unfortunately, I've now gotten contradictory views (not terribly
> surprising: people have different views and experiences, etc...).
>
> Mike (who posted earlier) implied that, if the underlying storage
> and network were solid and if failover is done right that it
> can be trusted.
>
> Jim, would having a support contract change your view?  Or, might
> the progression toward finding that right version/right hardware
> be dangerous even with support?  Is this something related to
> the codes immaturity?  Or just a complex problem?
>
> thanks much,
> John
>
>
> Jim Garlick wrote:
>> John,
>>
>> Lustre can be damn robust if you get the right version on the right
>> hardware.  Also, I think the new engineering practices and future
>> architecture that uses ZFS on the back end will only improve this.
>>
>> That said, your predicament is troubling.  As a general rule I  
>> would not
>> trust any parallel file system that I know of with mission critical  
>> data.
>> Failures do happen; indeed we have lost data in Lustre on several  
>> occasions.
>>
>> In some sense we're in a similar position.  The data we put in Lustre
>> is important to our mission (well some of it anyway), costly to  
>> regenerate,
>> and impractical to back up with a general backup policy.
>>
>> What we do is basically advertise Lustre as temporary scratch space  
>> and
>> provide an HPSS tape archive for users to copy their most critical  
>> data to.
>> That may not work in your case, but if I were you I would at least  
>> have
>> some sort of disaster plan for recovering or regenerating your data.
>> In short, don't trust Lustre or any parallel file system as the sole
>> repository for your mission critical data.
>>
>> Jim
>>
>> On Wed, May 14, 2008 at 02:21:02PM -0400, jrs wrote:
>>> Greetings all,
>>>
>>> I just spoke with someone at a large computing company who
>>> has a close relationship with lustre/sun (a reseller, I guess).
>>> This person described lustre as being something that Sun
>>> "would not recommend for mission critical use."
>>>
>>> Can this be true?
>>>
>>> I work for a small/medium company that does image processing.
>>> We have about 700TB of data presently and might be at 2PB within
>>> the next couple of years.  Owing to the amount of data we don't
>>> make backups for most of it and trust raid 6 on our hardware raid
>>> boxes (nexsan Satabeast) to fail more slowly than we can replace
>>> disks.  Over the last couple of years we've had great luck and,
>>> I believe, have never lost data owing to a failure with this
>>> hardware (software or human error is another matter ;-).
>>> However, the unbacked up data is "mission critical."  Though
>>> it can, probably, all be reconstructed or reacquired, as a practical
>>> matter losing a significant quantity of this data could be
>>> catastrophic for our business.
>>>
>>> So, what do you think, can lustre be trusted to keep our
>>> data safe at our company?  Assume in answering that we have
>>> failover working properly.  We can also withstand some blocking
>>> of the filesystem while a failover event completes, i.e., not
>>> having the filesystem available for some amount of time is
>>> not a problem, but having directory important-data/ disappear
>>> is a HUGE problem.
>>>
>>> Thanks for any help or guidance,
>>>
>>> John
>>> _______________________________________________
>>> Lustre-discuss mailing list
>>> Lustre-discuss at lists.lustre.org
>>> http://lists.lustre.org/mailman/listinfo/lustre-discuss
> _______________________________________________
> Lustre-discuss mailing list
> Lustre-discuss at lists.lustre.org
> http://lists.lustre.org/mailman/listinfo/lustre-discuss