[Lustre-discuss] strange performance with POSIX file capabilities

[Ralf Utermann]

Andreas Dilger wrote:
> On Oct 06, 2009  15:13 +0200, Ralf Utermann wrote:
>> with newer vanilla kernels we  saw strange performance
>> data with iozone on patchless clients: some OSTs had a lower write
>> bandwith in the iozone benchmark, getting worse with record
>> sizes below 1024.
>> After lots of kernel builds, it looks like the kernel config entry
>> CONFIG_SECURITY_FILE_CAPABILITIES is the one, wich
>> introduces this problem. If CONFIG_SECURITY_FILE_CAPABILITIES
>> is not set, iozone data look good, if it's compiled into the
>> kernel, we see the problem:
>> http://www.physik.uni-augsburg.de/~ralfu/LustreTest/Lustre_with_file_caps.html
> 
> Just to clarify, you are reporting the above config option affects
> write performance when changed on the client, correct?  It appears

Hi Andreas,

Yes, this option has only been used on the client side. The servers
are running a 2.6.22 kernel and it looks like this option has been
introduced with 2.6.24.

> that this option is off by default in the upstream kernels, so I
> suspect it doesn't get tested much.

This option is set on by default in the Debian kernels, and that's the config
I usually start with. I think, recent fedora kernels would also have this set,
and also RHEL6.

> 
>> Any idea, why file capabilities should affect the write
>> performance on Lustre, and why it should only affect some OSTs?
> 
> I can imagine that if this is adding some significant overhead on a
> per-system-call basis that it would hurt performance.
> 
> It is definitely odd that it would affect the performance of only some
> of the OSTs.  I assume they are otherwise identical?  The only thing

the OSTs are either 4 or 8 data disks on Sun 6140 systems; the 4 with
problems are on 2 OSS, the 3 without problems are on the other 2 OSS.

> I can imagine is that this option is related to SELinux and has some
> overhead in getting extended attributes, but even then the xattrs are
> only stored on the MDS so this would hurt all OSTs uniformly.

As I don't need this option anyway, I will just build my kernels now
with this option off. Of course an unpleasant feeling remains, not
knowing what really happens ...

As off vanilla kernel 2.6.29 there should be a no_file_caps kernel boot
parameter. I would like to test this setup, but b1_8 only builds fine with
vanilla 2.6.28, I cannot get it running with vanilla 2.6.[29|30] -- but
this should be different thread ...

Bye, Ralf
-- 
        Ralf Utermann
_____________________________________________________________________
        Universität Augsburg, Institut für Physik   --   EDV-Betreuer
        Universitätsstr.1             
        D-86135 Augsburg                     Phone:  +49-821-598-3231
        SMTP: Ralf.Utermann at Physik.Uni-Augsburg.DE         Fax: -3411