[Lustre-devel] Improving create rate with ZFS backed servers

[Prakash Surya]

On Fri, Jul 26, 2013 at 11:26:06AM +0400, Alexey Zhuravlev wrote:
> 
> On Jul 25, 2013, at 9:50 PM, Prakash Surya <surya1 at llnl.gov> wrote:
> 
> > so I started by trying to figure out where the time was being spent in
> > the precreate call path. On one of the OSS nodes I used ftrace to
> > profile {{ofd_precreate_objects()}} and saw that it took a whopping 22
> > seconds to complete for the specific call I was looking at! Digging a
> > little deeper I saw many calls to {{dt_object_find()}} which would
> > degenerate into {{fzap_lookup()}}'s and those into {{dbuf_read()}}
> > calls. Some of these {{dbuf_read()}} calls took over 600ms to complete
> > in the trace I was looking at. If we do a {{dt_object_find()}} for each
> > object we're precreating, and each of those causes a {{dbuf_read()}}
> > from disk which could take over half a second to complete, that's a
> > recipe for terrible create rates.
> > 
> > Thus, the two solutions seem to be to either not do the
> > {{fzap_lookup()}} altogether, or make the {{fzap_lookup()}} faster. One
> > way to make it faster is to ensure the call is a ARC hit, rather than it
> > reading from disk.
> 
> even if we remove lookups (which is possible), I don't think it will help much
> because we still need to modify that leaf anyway (to add just created object
> into OI), so read will be done few cycles later at declare.
>

Yes, AFAIK, you are right. Even if we remove the lookup in Lustre, ZFS
does essentially the same operation when inserting into the ZAP. So I
don't think this is a practical option for the time being.

-- 
Cheers, Prakash


> 
> 
> thanks, Alex
> 
> 
> --------------------------------------------------------------------
> Closed Joint Stock Company Intel A/O
> Registered legal address: Krylatsky Hills Business Park, 
> 17 Krylatskaya Str., Bldg 4, Moscow 121614, 
> Russian Federation
> 
> This e-mail and any attachments may contain confidential material for
> the sole use of the intended recipient(s). Any review or distribution
> by others is strictly prohibited. If you are not the intended
> recipient, please contact the sender and delete all copies.