[Lustre-discuss] MDS

[Cliff White]

Cliff White wrote:
> Mag Gam wrote:
>> Also, what is the best way to test the backup? Other than really
>> remove my MGS and restore it. Is there a better way to test this?
> 
> If you really care about the backups, you need to be brave. If you can't
> remove the MDS and restore it, then something is wrong with your backup
> process. Many people seem to focus on the backup part and ignore the 
> 'restore' bit, so I definately reccomend a live test.
> 
> That said, if you can bring up your backup MDT image on a separate node, 
> you could configure that node as a failover MDS - this would require you 
> to tunefs.lustre all the servers, and remount all the clients. Then you 
> can test restore using a 'manual failover' - and once you made the mount 
> changes, you could repeat this test at will, without even halting the 
> filesystem. Also, you would not have to 'remove' your primary MDS, just 
> stop that node.
> 
> If your MDS _does_ die, the failover config will cause a slightly longer 
> timeout (everybody will retry the alternate) but otherwise won't impact 
> you.

Just to be clear, there is a potential data loss issue due to the time 
delta between the backup and the live system. Any transactions in play
that miss the snapshot could result in lost data, as the MDS will replay 
transaction logs and delete orphans on startup. So testing on your live 
system definately is for the brave.
cliffw

> 
> cliffw
>>
>> TIA
>>
>>
>> On Tue, Aug 5, 2008 at 6:37 PM, Mag Gam <magawake at gmail.com> wrote:
>>> Brian:
>>>
>>> Thanks for the response. I actually seen this response before and was
>>> wondering if my technique would simply work. I guess not.
>>>
>>> I guess another question will be, if I take a snapshot every 10 mins
>>> and back it up. If I have a failure at 15th minute. Can I just simply
>>> restore my MDS to the previous snapshot and be with it? Ofcourse I
>>> will lose my 5 minutes of data, correct?
>>>
>>> TIA
>>>
>>>
>>> On Tue, Aug 5, 2008 at 12:08 PM, Brian J. Murrell 
>>> <Brian.Murrell at sun.com> wrote:
>>>> On Tue, 2008-08-05 at 01:12 -0400, Mag Gam wrote:
>>>>> What is a good MGS/MDT backup strategy if there is one?
>>>>>
>>>>> I was thinking of  mounting the MGS/MDT partition on the MDS as ext3
>>>>> and rsync it every 10 mins to another server. Would this work? What
>>>>> would happen in the 9th minute I lose my MDS, would I still be able to
>>>>> have a good copy? Any thoughts or ideas?
>>>> Peter Braam answered a similar question and of course, the answer is in
>>>> the archives.  It was the second google hit on a search for "lustre mds
>>>> backup".  The answer is at:
>>>>
>>>> http://lists.lustre.org/pipermail/lustre-discuss/2006-June/001655.html
>>>>
>>>> Backup of the MDT is also covered in the manual in section 15 at
>>>>
>>>> http://manual.lustre.org/manual/LustreManual16_HTML/BackupAndRestore.html#50544703_pgfId-5529 
>>>>
>>>>
>>>> Now, as for mounting the MDT as ext3 (you should actually use ldiskfs,
>>>> not ext3) every 10 minutes, that means you are going to make your
>>>> filesystem unavailable every 10 minutes as you CANNOT mount the MDT
>>>> partition on more than one machine and we have not tested multiple
>>>> mounting on a single machine with any degree of confidence.
>>>>
>>>> Of course Peter's LVM snapshotting technique will allow you to mount
>>>> snapshots which you can backup as you describe.
>>>>
>>>> But if you are going to have a whole separate machine with enough
>>>> storage to mirror your MDT why not use something more active like DRBD
>>>> and have a fully functional active/passive MDT failover strategy?  
>>>> While
>>>> nobody in the Lustre Group has done any extensive testing of Lustre on
>>>> DRBD, there have been a number of reports of success with it here on
>>>> this list.
>>>>
>>>> b.
>>>>
>>>>
>>>> _______________________________________________
>>>> Lustre-discuss mailing list
>>>> Lustre-discuss at lists.lustre.org
>>>> http://lists.lustre.org/mailman/listinfo/lustre-discuss
>>>>
>>>>
>> _______________________________________________
>> Lustre-discuss mailing list
>> Lustre-discuss at lists.lustre.org
>> http://lists.lustre.org/mailman/listinfo/lustre-discuss
> 
>