[Lustre-discuss] copy_user_generic_c ?

[Andreas Dilger]

On Jul 07, 2008  17:42 -0400, Roger Spellman wrote:
> I'm running an OST (not a client on an OST) with SW RAID 5.  My
> interconnect is IB, and I'm using OFED 1.3.1.  I added OPROFILE to my
> kernel, to see if I could find a bottleneck.  The biggest CPU user, at
> 25%, was copy_user_generic_c.  
>
> Grepping through the linux, ofed, and lustre code, I cannot find where
> this is being called.  Can anyone suggest where this is being called,
> and why?

This is a well-known problem - this kernel function is copying data
from userspace to the kernel buffers on a write, and vice versa on
a read.  The way to avoid this is by using O_DIRECT, but as a result
you will not get cached data on the client, and this means you will
not able to do cached writes (i.e. write behind) and will wait for
IO completion for each write (i.e. sync writes).

If you are doing enough IO to hit a bottleneck with copy_{to,from}_user()
then you can probably also be doing large enough IOs to make the sync IO
performance hit of O_DIRECT negligible.

We are looking at how to spread the load of copy_{to,from}_user() over
more CPUs, but that is not likely to make it into a Lustre release for
some time yet.  Completely avoiding the copy while allowing a cache on
the client would require major VFS/VM surgery (e.g. ensuring the buffers
are aligned and marking the pages read-only, forcing the client to fault
them if it changes the buffer again).

Cheers, Andreas
--
Andreas Dilger
Sr. Staff Engineer, Lustre Group
Sun Microsystems of Canada, Inc.