[Lustre-devel] changelog for whole filesystem?

Andreas Dilger andreas.dilger at oracle.com
Fri Nov 12 15:58:17 PST 2010


On 2010-11-12, at 16:41, Robert Read wrote:
> It seems simpler to do the iteration on  a snapshot, instead of a live filesystem, and allow post-snapshot changes to accumulate on the regular changelog for processing once the "from empty" iteration was complete.  

Just need to implement snapshot support for Lustre. :-)

> On Nov 11, 2010, at 10:10 AM, Eric Barton wrote:
>> Nathan wrote...
>>> This same "initial population of a database from objects" problem
>>> occurs when trying to replicate a Lustre filesystem using the
>>> changelog.
>> 
>> That must be a great test case.  If we can reliably reconstruct an
>> exact replica using a "from empty" changelog, we must have got it
>> right :)
>> 
>>> The problem is actually more complicated even for a single changelog
>>> consumer: since iterating through the virtual changelog takes non-0
>>> time, you're not sure if a virtual record was created before or
>>> after an actual changelog entry.  E.g. you might try to resolve the
>>> name of a file for a virtual record either before or after a rename,
>>> and then later you would see the rename in the changelog, leading to
>>> an inconsistent view of the namespace.
>>> 
>>> If you don't care about having the exact right name, then it's easy
>>> enough to ignore inapplicable changelog records.
>> 
>> I was trying to hint at the need to filter changes for individual
>> consumers depending on how far through the initial "from empty"
>> iteration they are in my original post.  Andreas stated it more
>> explicitly.  I think you just need to enumerate the cases - e.g. for
>> rename...
>> 
>> Iterated over yet?               Record
>> Source Inode   Target Inode      emitted
>> no             no                none
>> no             yes               delete target
>> yes            yes               rename source + delete target
>> yes            no                rename source
>> 
>> The final case just looks like a rename where the target name didn't
>> exist already.  In any case, the only real requirement on the stream
>> of changelog records constructed in the initial iteration is that
>> consistent filesystem state be reconstructed after the last record
>> is consumed.
>> 
>> -- 
>> 
>>               Cheers,
>>                       Eric
>> 
>> Eric Barton
>> CTO Whamcloud, Inc.
>> Tel: +44 117 330 1575
>> Mob: +44 7920 797 273
>> 
>> _______________________________________________
>> Lustre-devel mailing list
>> Lustre-devel at lists.lustre.org
>> http://lists.lustre.org/mailman/listinfo/lustre-devel
> 


Cheers, Andreas
--
Andreas Dilger
Lustre Technical Lead
Oracle Corporation Canada Inc.




More information about the lustre-devel mailing list