[Lustre-discuss] Can lustre be trusted to keep my data safe?

[jrs]

Greetings all,

Thanks to everyone who offered comments on my original question
about lustre's trustworthiness.

I apologize if I seem to be flogging this issue to death but
I have to make a decision in the next day or two about whether
to move ahead with deploying lustre and I'm probably more
uncertain now about what to do than I was 2 days ago ;-)

A couple of things.

Some people commented on backups.  I'd like to skip the entire
issue of backups by noting that our backup policy will be the
same if we're using lustre as if we were using any other file
system.  Similarly, I'd like to bracket away the issue of the
backend storage.  We are going to rely on the integrity
of our raid 6 systems, whether we're using lustre or another
filesystem.  Also, the dreaded 'rm -rf /lustre/*' issue is
a non-issue for us (it is, at least, not a lustre-specific
issue).

I'm just looking for a direct assessment of whether:

- lustre can be as safe as having a set of linux NFS servers
   (where the filesystem is ext3)?
- lustre can be as safe as a set of windows servers (which is what
   we presently use)?

Note that I'm assuming:
- no striping (every file resides, wholley, on only a single OST)
- failover is configured properly
- our applications can withstand having to block the 7 minutes
   (or whatever similiar time) it takes for lustre to
   make a failed over OST available on a new OSS)
- we rarely have multiple clients writing to the same file
- Our traffic tends to be: client copies 10 large files to
   local disk, computes, then writes output to filesystem.
- We'll be using bonded NICs and, if we adopted IB, it would
   be some time out ...

We just want to use lustre as a fast, scalable, single name
space general purpose filesystem and we want to know whether
LUSTRE itself (not the hardware) will ever lose our data.

A conversation with my Sun reseller went something like this:

  Reseller:
  "lustre is fast scratch space; it's good enough for that; it
   performs extremely well in that role, but it's not a general
   purpose file system and you shouldn't trust it."

  Me:
  "but some of the jobs being run by those who use it as fast
   scratch space run for days, sometimes weeks, right?  And
   some of that data is very important, weather prediction,
   oil and gas, etc..., right?"

  Reseller:
  "yup.  yup."

  Me:
  "and the data is uncorrupted and trusted during the time that
  the application is running, right?"

  Reseller:
  "Sure."

  Me:
  "Well, if the data is trusted at 2 weeks, why wouldn't it be
  trusted for 2 years (again hardware error excepted)?  Are there
  bugs that will most likely be exposed with additional usage that
  might threaten the integrity of the data?"

  Reseller:
  "Dunno."

  Me:
  "or, if we lose a LUN, will the loss of data be greater with
  lustre than with, say, losing a ext3 file system being exposed
  via NFS?  I can't see how this would be but ..."

  Reseller:
  "seems unlikely to be worse than any other filesystem"

  Me:
  "or, in the event of the loss of a LUN will I have fewer opportunities
  to do a low level recover of the data."

  Reseller:
  "well, it's just ext3 (or ext4) so you should be able to it whatever
  you do to a regular ext3 volume."



Anyway, thanks for your time and thanks in advance for any further
advice,

John

Joe Georger wrote:
 > I think you should be more worried about your hardware.  We have
 > several Nexsan units including dual controller Satabeasts running
 > Raid6.  A few months ago one of them had a "glitch" and it incorrectly
 > marked 3 disks as bad within 6 seconds.  So it had started the rebuild
 > process then stopped in an unsynchronized state.  We are running Ibrix
 > (I read this list in case we ever want to switch) and it corrupted the
 > file system.  We lost data.  Even after spending 3 days running fsck
 > on a 160 TB filesystem.  Fortunately Ibrix does not stripe files
 > across OST's so the loss had minimal impact and we were able to
 > restore the 4 TB from backup.   Maybe Lustre would handle this better,
 > I'm not sure....
 >
 > So the glitch was eventually traced to some bad ECC....  It seemed
 > like a one-off failure, but the lesson was important.  You really need
 > a 2nd copy if your data is critical.  I've also been told than Nexsan
 > is more like a "Tier 2" storage vendor.  If the 2nd copy is not
 > feasible, perhaps consider more expensive "Tier 1" like Compellant, etc.
 >
 > Joe
 >
 > jrs wrote:
 >> Greetings all,
 >>
 >> I just spoke with someone at a large computing company who
 >> has a close relationship with lustre/sun (a reseller, I guess).
 >> This person described lustre as being something that Sun
 >> "would not recommend for mission critical use."
 >>
 >> Can this be true?
 >>
 >> I work for a small/medium company that does image processing.
 >> We have about 700TB of data presently and might be at 2PB within
 >> the next couple of years.  Owing to the amount of data we don't
 >> make backups for most of it and trust raid 6 on our hardware raid
 >> boxes (nexsan Satabeast) to fail more slowly than we can replace
 >> disks.  Over the last couple of years we've had great luck and,
 >> I believe, have never lost data owing to a failure with this
 >> hardware (software or human error is another matter ;-).
 >> However, the unbacked up data is "mission critical."  Though
 >> it can, probably, all be reconstructed or reacquired, as a practical
 >> matter losing a significant quantity of this data could be
 >> catastrophic for our business.
 >>
 >> So, what do you think, can lustre be trusted to keep our
 >> data safe at our company?  Assume in answering that we have
 >> failover working properly.  We can also withstand some blocking
 >> of the filesystem while a failover event completes, i.e., not
 >> having the filesystem available for some amount of time is
 >> not a problem, but having directory important-data/ disappear
 >> is a HUGE problem.
 >>
 >> Thanks for any help or guidance,
 >>
 >> John
 >> _______________________________________________
 >> Lustre-discuss mailing list
 >> Lustre-discuss at lists.lustre.org
 >> http://lists.lustre.org/mailman/listinfo/lustre-discuss
 >>
 >>
 > _______________________________________________
 > Lustre-discuss mailing list
 > Lustre-discuss at lists.lustre.org
 > http://lists.lustre.org/mailman/listinfo/lustre-discuss