[Lustre-discuss] socknal_sd00 100% lower?

[Andreas Dilger]

On Mar 07, 2008  13:27 -0500, Brock Palen wrote:
> On Mar 7, 2008, at 1:23 PM, Maxim V. Patlasov wrote:
> > Brock,
> >> Notice the amount of cpu time given to sd00  and how sd01 has done  
> >> nothing.  What could cause this?
> > Please try Isaac's recommendation:
> >> So if you have multiple CPUs and a single NIC (or more precisely  
> >> Lustre only uses a single NIC) I'd suggest to try:
> >> options ksocklnd enable_irq_affinity=0
> 
> How do you do this on a live system?  Taking away the filesystem is  
> 'bad'

On relatively modern kernels you can change SOME module parameters:

echo 0 > /sys/modules/ksocklnd/{parameters}/enable_irq_affinity

The "{parameters}" subdir is present or not, depending on the kernel
version (yay consistency!).

That said, it is implementation dependent on whether changing the
parameter after the module is loaded has any effect.  I suspect in
this case it will not have any effect because the kernel threads
are already bound to their CPUs, but this isn't my strongest area.

Maxim, Isaac, what are your thoughts about disabling IRQ affinity
by default?  In the past this was important for maximizing performance
with N CPUs and N ethernet NICs, but the CPUs have gotten much faster
and more cores and I believe other customers have found better performance
with irq_affinity disabled.

Cheers, Andreas
--
Andreas Dilger
Sr. Staff Engineer, Lustre Group
Sun Microsystems of Canada, Inc.