[Lustre-devel] Question on iopen_lookup()

Andreas Dilger adilger at sun.com
Fri Sep 26 23:15:02 PDT 2008


On Sep 24, 2008  10:43 -0500, Nathan Roberts wrote:
> I've got a quick question on the open_by_inode support that ldiskfs
> is using.
> 
> Since there is only a single magic ".inode" directory inode, AND
> real_lookup() locks the directory mutex when performing the lookup
> of the actual file inode, I'm seeing major contention for this lock.
> The contention makes it impossible for the io scheduler to do anything
> intelligent when there are lots of opens occurring at the same time,
> thus killing small file performance.

We're always interested in improving the metadata performance of
Lustre, and it sounds like you have an interesting patch.

> Since this directory isn't real I don't think there is a reason
> we need to hold the lock during this type of lookup. I added
> a couple of lines to drop and re-aquire this lock around the
> expensive iget() just to see if it removes the bottleneck. My
> read request rate went up from 96 ops/second to 565 so it does
> seem to help.
> 
> Can you see any danger in this type of approach? If it is a completely
> broken approach, any suggestions on how to go about removing this
> contention?

> +       mutex_unlock(&dir->i_mutex);
> 	inode = iget(dir->i_sb, ino);
> +	mutex_lock(&dir->i_mutex);

It looks reasonable to make the change you are proposing.  In fact,
we are also discussing to replace __iopen__ with ext3_fh_to_dentry(),
which calls ext3_nfs_get_inode() internally and it also does iget()
without any locking from looking at these functions and callers.

The benefit of replacing __iopen__ with the ext3_export_ops() is that
we don't have to maintain the twisty mazes of the __iopen__ code in
the face of constantly changing kernel APIs.  This would also be one
step closer to having a patchless server kernel.

We haven't had anyone working on this previously, but if you'd like
to take a crack at it I think we could get both the performance and
reduction in code complexity at the same time.

Cheers, Andreas
--
Andreas Dilger
Sr. Staff Engineer, Lustre Group
Sun Microsystems of Canada, Inc.




More information about the lustre-devel mailing list