[Lustre-devel] COS performance issues
Alex Zhuravlev
Alex.Zhuravlev at Sun.COM
Mon Oct 13 08:04:51 PDT 2008
cool! do you have profile with COS disabled? how is it different?
I guess ldlm_resorce_find() is result of COS as now we've got 10K times
more locks and resorces, but what about try_to_wake_up() and __wake_up_common() ?
thanks, Alex
Alexander Zarochentsev wrote:
> On 12 October 2008 23:12:10 Alex Zhuravlev wrote:
>> would be good to look at profiles as the next one was
>> ldlm_resource_get()
>
> Alex, look, ptlrpc_server_handle_reply has gone:
>
> CPU: P4 / Xeon, speed 2800.25 MHz (estimated)
> Counted GLOBAL_POWER_EVENTS events (time during which processor is not stopped
> with a unit mask of 0x01 (mandatory) count
> 100000
> samples % image name app name symbol name
> 312318 4.7857 vmlinux vmlinux try_to_wake_up
> 259690 3.9792 ptlrpc.ko ptlrpc ldlm_resource_find
> 175916 2.6956 vmlinux vmlinux __wake_up_common
> 160912 2.4657 e1000.ko e1000 e1000_irq_enable
> 157031 2.4062 obdclass.ko obdclass htable_lookup
> 134309 2.0580 e1000.ko e1000 e1000_intr_msi
> 121261 1.8581 lvfs.ko lvfs lprocfs_counter_add
> 87613 1.3425 vmlinux vmlinux __find_get_block
> 87562 1.3417 oprofiled oprofiled (no symbols)
> 77926 1.1941 vmlinux vmlinux memset
> 74561 1.1425 vmlinux vmlinux __kmalloc
> 70952 1.0872 vmlinux vmlinux __switch_to
> 67288 1.0311 mds.ko mds mds_lov_dump_objids
> 67085 1.0279 vmlinux vmlinux memmove
> 66521 1.0193 vmlinux vmlinux kfree
> 55112 0.8445 ptlrpc.ko ptlrpc ptlrpc_main
> 54357 0.8329 vmlinux vmlinux schedule
> 53501 0.8198 vmlinux vmlinux find_get_page
>
>> thanks, Alex
>>
>> Alexander Zarochentsev wrote:
>>> On 8 October 2008 15:48:50 Alex Zhuravlev wrote:
>>>> try to profile with single CPU? you'll probably get an idea how
>>>> "per-cpu" approach can help.
>>> I booted the MDS server with maxcpus=1 kernel parameter and here
>>> are the results:
>>>
>>> cos=0
>>> 2039.31 creates/sec (total: 2 threads 611794 creates 300 secs)
>>> 2037.80 creates/sec (total: 2 threads 611341 creates 300 secs)
>>> 2076.21 creates/sec (total: 2 threads 622864 creates 300 secs)
>>>
>>> cos=1
>>> 1874.93 creates/sec (total: 2 threads 564354 creates 301 secs)
>>> 1923.97 creates/sec (total: 2 threads 577191 creates 300 secs)
>>> 1892.61 creates/sec (total: 2 threads 567783 creates 300 secs)
>>> 1874.74 creates/sec (total: 2 threads 562421 creates 300 secs)
>>>
>>> unfortunately profiling info isn't available yet, the results are
>>> done with SLES10 which can boot with maxcpus=1 but has no oprofile
>>> installed.
>>>
>>>> Alexander Zarochentsev wrote:
>>>>> I have a patch to avoid using of obd_uncommitted_replies_lock
>>>>> in ptlrpc_server_handle_reply but it has minimal effect,
>>>>> ptlrpc_server_handle_reply still the most cpu consuming function
>>>>> because of svc->srv_lock contention.
>>>>>
>>>>> I think the problem is that COS defers processing of replies to
>>>>> transaction commit time. When commit happens, MDS has to process
>>>>> thousands of replies (about 14k replies per commit in the test
>>>>> 3.a) in short period of time. I guess the mdt service threads all
>>>>> woken up and spin trying to get the service svr_lock. Processing
>>>>> of new requests may also suffer of this.
>>>>>
>>>>> I ran the tests with with CONFIG_DEBUG_SPINLOCK_SLEEP debugging
>>>>> compiled into a kernel, it found no sleep under spinlock bugs.
>>>>>
>>>>> Further optimization may include
>>>>> 1. per-reply spin locks.
>>>>> 2. per-cpu structures and threads to process reply queues.
>>>>>
>>>>> Any comments?
>>>>>
>>>>> Thanks.
>>>>>
>>>>> PS. the test results are much better when MDS server is sata20
>>>>> machine with 4 cores (the MDS from Washie1 has 2 cores), COS=0
>>>>> and COS=1 have only %3 difference:
>>>>>
>>>>> COS=1
>>>>> Rate: 3101.77 creates/sec (total: 2 threads 930530 creates 300
>>>>> secs) Rate: 3096.94 creates/sec (total: 2 threads 929083 creates
>>>>> 300 secs)
>>>>>
>>>>> COS=0
>>>>> Rate: 3184.01 creates/sec (total: 2 threads 958388 creates 301
>>>>> secs) Rate: 3152.89 creates/sec (total: 2 threads 945868 creates
>>>>> 300 secs)
>>>> _______________________________________________
>>>> Lustre-devel mailing list
>>>> Lustre-devel at lists.lustre.org
>>>> http://lists.lustre.org/mailman/listinfo/lustre-devel
>
More information about the lustre-devel
mailing list