[Lustre-devel] Lustre client I/O

[Nikita Danilov]

[Lustre-devel copied.]

Hello,

see comments below.

2009/12/4 jay <Jinshan.Xiong at sun.com>:
> Oleg Drokin wrote:
>>
>> Hello!
>>
>> On Dec 3, 2009, at 5:32 PM, Nikita Danilov wrote:
>>
>>
>>>
>>> Hello Oleg,
>>>
>>> Peter Braam told me you pointed him to some issues with CLIO. Could
>>> you provide details? I am probably owing an explanation of why things
>>> were done in this or that way. May be on lustre-devel?
>>>
>>
>>   Actually I did not dig too much into details myself yet.
>> Jay and WangDi are the best people to ask about specific deficiencies,
>> I think I remember there were mentioned some missed state machine states,
>> then clio takes locks in random order for the same file, so it was
>> deadlock prone.
>> See bug 19906 for some of that. There are many more.
>>
>
> Mostly implementation defects. For 19906, it turns out to be lack of error
> handling at some point, for example, if lock wait fails, we have to
> de-referenced the enqueued sublocks.
>
>> I did some measurements and it is very slow too, over 30% slower than b1_8
>> code
>> for the case of local access.
>>
>
> We've never done any performance tune against clio.
>>
>> Certain questions are raised by Eric Barton if we ever need another cache
>> for compound locks if we already have ldlm cache for locks, since that
>> other
>> cache adds a lot of complications too. And I sort of agree the current
>> clio code
>> looks like a total overkill and very complicated.
>>
>
> Yes, the two-level cached lock mechanism makes cl_lock extremely complex and
> fragile - it can be simplified by removing the top level cache.

I agree that caching top- and bottom- locks adds considerable
complexity. Performance advantages of caching alone probably cannot
justify it. On the other hand, removing top-lock caching only makes
sense when layering is fixed: in a general IO stack layers must have
uniform caching behaviour.

>
> CLIO is still in its childhood, it has never been verified at customer's
> side. It's quite common that we meet new problems when running a new test
> suite. As a matter of fact, most clio bugs were found in new test suites -
> we don't have new issues in these days, because no new test suite is
> introduced.
>
> Before diving into the good and bad side of clio, let's check up what's the
> initiatives of having clio, and whether we have reached those targets or not
> in current implementation. We don't need to focus on those defects in
> implementation.
> This is the design goals of clio:
>
> clear layering;
> controlled state sharing;
> simplified layer interface;
> real stacking;
> reuse of mdt server layering;
> improving portability.

Those are mostly achieved. "Mostly" because few things, like
read-ahead, are still at a wrong level. As an example of advantages of
improved layering, implementation of lock-less IO in CLIO re-uses the
same code paths as a normal caching IO: DLM interaction details are
encapsulated within osc and it is possible to substitute "surrogate
locks" there without rest of the stack noticing. (As a note, CLIO
lock-less IO doesn't currently handle sub-page lockless read as
efficiently as 1.8, because it fetches the whole page from the server
first, but this is easy to fix or maybe it is already fixed.)

> support for:
>  SNS;
>  read-only p2p caching;
>  lock-less IO and ost intents;
>
>> Feel free to add lustre-devel at any point.

One of the more confusing parts of CLIO is its file and stripe lock
implementation, even putting top-lock caching aside for a moment.
There are, I think, two main reasons for this:

    * locks are implemented as non-blocking state machines. This is
rather unusual and definitely not common programming style looking
somewhat inside-out-ish at first sight. The justification for this is
support for a "parallel IO", i.e., a mode where a write, instead of
blocking on a full per-OST cache, proceeds to the next stripe. The
upside is that once non-blocking infrastructure is in place in cl_lock
and cl_io, many other interesting things, like concurrent copy_*_user
and "lock-ahead", are easier to do;

    * concurrency control for cl_lock is fiendishly difficult. I now
think it was a mistake to strive for finer-grained locking. Eric
Barton noted that a lock on a top-object could protect state of all
locks on the object and its sub-objects. This would greatly simplify
things without intolerable decrease in concurrency.

>>
>> Bye,
>>    Oleg
>>

Thank you,
Nikita.

>
>