[Lustre-discuss] high CPU load limits bandwidth?
Andreas Dilger
andreas.dilger at oracle.com
Wed Oct 20 09:53:27 PDT 2010
On 2010-10-20, at 10:40, Michael Kluge <Michael.Kluge at tu-dresden.de> wrote:
> It is the CPU load on the client. The dd/IOR process is using one core completely. The clients and the servers are connected via DDR IB. LNET bandwidth is at 1.8 GB/s. Servers have 1.8.3, the client has 1.8.3 patchless.
If you only have a single threaded write, then this is somewhat unavoidable to saturate a CPU due to copy_from_user(). O_DIRECT will avoid this.
Also, disabling data checksums and debugging can help considerably. There is a patch in bugzilla to add support for h/w crc32c on Nehalem CPUs to reduce this overhead, but still not as fast as no checksum at all.
Cheers, Andreas
> Am 20.10.2010 18:15, schrieb Andreas Dilger:
>> Is this client CPU or server CPU? If you are using Ethernet it will definitely be CPU hungry and can easily saturate a single core.
>>
>> Cheers, Andreas
>>
>> On 2010-10-20, at 8:41, Michael Kluge<Michael.Kluge at tu-dresden.de> wrote:
>>
>>> Hi list,
>>>
>>> is it normal, that a 'dd' or an 'IOR' pushing 10MB blocks to a lustre
>>> file system shows up with a 100% CPU load within 'top'? The reason why I
>>> am asking this is that I can write from one client to one OST with 500
>>> MB/s. The CPU load will be at 100% in this case. If I stripe over two
>>> OSTs (which use different OSS servers and different RAID controllers) I
>>> will get 500 as well (seeing 2x250 MB/s on the OSTs). The CPU load will
>>> be at 100% again.
>>>
>>> A 'dd' on my desktop pushing 10M blocks to the local disk shows 7-10%
>>> CPU load.
>>>
>>> Are there ways to tune this behavior? Changing max_rpcs_in_flight and
>>> max_dirty_mb did not help.
>>>
>>>
>>> Regards, Michael
>>>
>>> --
>>>
>>> Michael Kluge, M.Sc.
>>>
>>> Technische Universität Dresden
>>> Center for Information Services and
>>> High Performance Computing (ZIH)
>>> D-01062 Dresden
>>> Germany
>>>
>>> Contact:
>>> Willersbau, Room A 208
>>> Phone: (+49) 351 463-34217
>>> Fax: (+49) 351 463-37773
>>> e-mail: michael.kluge at tu-dresden.de
>>> WWW: http://www.tu-dresden.de/zih
>>> _______________________________________________
>>> Lustre-discuss mailing list
>>> Lustre-discuss at lists.lustre.org
>>> http://lists.lustre.org/mailman/listinfo/lustre-discuss
>>
>
>
> --
> Michael Kluge, M.Sc.
>
> Technische Universität Dresden
> Center for Information Services and
> High Performance Computing (ZIH)
> D-01062 Dresden
> Germany
>
> Contact:
> Willersbau, Room WIL A 208
> Phone: (+49) 351 463-34217
> Fax: (+49) 351 463-37773
> e-mail: michael.kluge at tu-dresden.de
> WWW: http://www.tu-dresden.de/zih
More information about the lustre-discuss
mailing list