[Lustre-devel] Improving create rate with ZFS backed servers

Prakash Surya surya1 at llnl.gov
Fri Jul 26 09:45:24 PDT 2013

On Fri, Jul 26, 2013 at 11:26:06AM +0400, Alexey Zhuravlev wrote:
> On Jul 25, 2013, at 9:50 PM, Prakash Surya <surya1 at llnl.gov> wrote:
> > so I started by trying to figure out where the time was being spent in
> > the precreate call path. On one of the OSS nodes I used ftrace to
> > profile {{ofd_precreate_objects()}} and saw that it took a whopping 22
> > seconds to complete for the specific call I was looking at! Digging a
> > little deeper I saw many calls to {{dt_object_find()}} which would
> > degenerate into {{fzap_lookup()}}'s and those into {{dbuf_read()}}
> > calls. Some of these {{dbuf_read()}} calls took over 600ms to complete
> > in the trace I was looking at. If we do a {{dt_object_find()}} for each
> > object we're precreating, and each of those causes a {{dbuf_read()}}
> > from disk which could take over half a second to complete, that's a
> > recipe for terrible create rates.
> > 
> > Thus, the two solutions seem to be to either not do the
> > {{fzap_lookup()}} altogether, or make the {{fzap_lookup()}} faster. One
> > way to make it faster is to ensure the call is a ARC hit, rather than it
> > reading from disk.
> even if we remove lookups (which is possible), I don't think it will help much
> because we still need to modify that leaf anyway (to add just created object
> into OI), so read will be done few cycles later at declare.

Yes, AFAIK, you are right. Even if we remove the lookup in Lustre, ZFS
does essentially the same operation when inserting into the ZAP. So I
don't think this is a practical option for the time being.

Cheers, Prakash

> thanks, Alex
> --------------------------------------------------------------------
> Closed Joint Stock Company Intel A/O
> Registered legal address: Krylatsky Hills Business Park, 
> 17 Krylatskaya Str., Bldg 4, Moscow 121614, 
> Russian Federation
> This e-mail and any attachments may contain confidential material for
> the sole use of the intended recipient(s). Any review or distribution
> by others is strictly prohibited. If you are not the intended
> recipient, please contact the sender and delete all copies.

More information about the lustre-devel mailing list