[Lustre-discuss] high CPU usage on MDS
Liang Zhen
liang at whamcloud.com
Tue Jul 19 21:20:21 PDT 2011
Hi there,
I can't see any unexpected information in your oprofile, although a few functions have high values in oprofile, but I think they are still in acceptable range.
Since you are using GigE, which means data copy might contribute to cpu-usage. Although ksocklnd can support zero-copy, but it's not used for small messages which is plenty on MDS. The way to change it is adding this to modprobe.conf of MDS:
"ksocklnd zc_min_payload=1"
It will enable zero-copy for payload larger than 4K (1 page), although I'm not sure how much this can help.
Regards
Liang
On Jul 19, 2011, at 2:33 PM, Ekaterina Popova wrote:
> On 07/07/2011 06:10 PM, Colin Faber wrote:
>>
>> Agreed however more details are still needed.
>>
>> -cf
>> -----Original message-----
>> From: Isaac Huang
>> Sent: 07/07/2011, 12:31 am
>> To: Colin Faber
>> Cc: Ekaterina Popova; lustre-discuss at lists.lustre.org
>> Subject: Re: [Lustre-discuss] high CPU usage on MDS
>>
>>
>>
>> I think it's TCP/IP according to the process list. It'd help to find
>> out where the CPU time was spent, e.g. by oprofile.
>>
>> - Isaac
>>
>> On Wed, Jul 06, 2011 at 12:14:54PM -0600, Colin Faber wrote:
>>
>>> Hi,
>>>
>>> More details are needed here. What type of interconnect are you using?
>>> What are your clients doing? etc.
>>>
>>> -cf
>>>
>>>
>>> On 07/04/2011 11:08 PM, Ekaterina Popova wrote:
>>>
>>>> Hello!
>>>>
>>>> We've got one MDS (Lustre 2.0) with 4 processors (Intel(R) Core(TM)2
>>>> Quad CPU Q9500 @ 2.83GHz).
>>>> Our CPU usage is HIGH - 80-100% .
>>>> With top command we see:
>>>> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
>>>> 2973 root 15 0 0 0 0 R 35.0 0.0 1509:09 socknal_sd02
>>>> 2972 root 15 0 0 0 0 S 33.5 0.0 1132:13 socknal_sd01
>>>> 2971 root 15 0 0 0 0 S 19.8 0.0 1129:12 socknal_sd00
>>>> 2974 root 16 0 0 0 0 R 13.7 0.0 830:47.46 socknal_sd03
>>>>
>>>> Is it normal situation? Can we improve performance at mds ? Maybe
>>>> setting any parameters?
>>>> (Only about 70 clients mount to Lustre and not all of them work with it.)
>>>>
>>>> With regards,
>>>> Popova Ekaterina
>>>>
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> Lustre-discuss mailing list
>>>> Lustre-discuss at lists.lustre.org
>>>> http://lists.lustre.org/mailman/listinfo/lustre-discuss
>>>>
>>> _______________________________________________
>>> Lustre-discuss mailing list
>>> Lustre-discuss at lists.lustre.org
>>> http://lists.lustre.org/mailman/listinfo/lustre-discuss
>>>
>> ______________________________________________________________________
>> This email may contain privileged or confidential information, which should only be used for the purpose for which it was sent by Xyratex. No further rights or licenses are granted to use such information. If you are not the intended recipient of this message, please notify the sender by return and delete it. You may not use, copy, disclose or rely on the information contained in it.
>>
>> Internet email is susceptible to data corruption, interception and unauthorised amendment for which Xyratex does not accept liability. While we have taken reasonable precautions to ensure that this email is free of viruses, Xyratex does not accept liability for the presence of any computer viruses in this email, nor for any losses caused as a result of viruses.
>>
>> Xyratex Technology Limited (03134912), Registered in England & Wales, Registered Office, Langstone Road, Havant, Hampshire, PO9 1SA.
>>
>> The Xyratex group of companies also includes, Xyratex Ltd, registered in Bermuda, Xyratex International Inc, registered in California, Xyratex (Malaysia) Sdn Bhd registered in Malaysia, Xyratex Technology (Wuxi) Co Ltd registered in The People's Republic of China and Xyratex Japan Limited registered in Japan.
>> ______________________________________________________________________
>>
>>
>
> Hi,
> We have two 1Gb Ethernet cards, one card is using for internal cluster, another
> one is for outer network (it is almost not used). We see large network traffic
> in MDS on internal network card (~200M/s in case of 150 jobs running in grid
> cluster). Memory using is low (total mem - 4G, used mem - 1G). At changelog on
> client we have such records:
> ...
> 5463915 06UNLNK 06:21:17.11299887 2011.07.05 0x0 t=[0x2000061c4:0x13af5:0x0] p=
> [0x200005233:0x12fd3:0x0] app.upmf.properties285631923419962629.TMP
> 5463916 01CREAT 06:31:17.30895460 2011.07.05 0x0 t=[0x2000061c4:0x13b4d:0x0] p=
> [0x200005233:0x12fd3:0x0] app.upmf.properties5508069877142384203.TMP
> 5463917 06UNLNK 06:31:47.60574777 2011.07.05 0x0 t=[0x2000061c4:0x13b4d:0x0] p=
> [0x200005233:0x12fd3:0x0] app.upmf.properties5508069877142384203.TMP
> 5463918 01CREAT 06:31:47.77574595 2011.07.05 0x0 t=[0x2000061c4:0x13b55:0x0] p=
> [0x200005233:0x12fd3:0x0] app.upmf.properties5753247530971764727.TMP
> 5463919 06UNLNK 06:31:47.219573079 2011.07.05 0x0 t=[0x2000061c4:0x13b55:0x0] p=
> [0x200005233:0x12fd3:0x0] app.upmf.properties5753247530971764727.TMP
>
> On MDS:
>
> opreport --symbols -p /lib/modules/2.6.18-prep/
>
> CPU: Core 2, speed 2833 MHz (estimated)
> Counted CPU_CLK_UNHALTED events (Clock cycles when not halted) with a unit mask
> of 0x00 (Unhalted core cycles) count 100000
> samples % image name app name symbol name
> 3445413 3.1207 lnet.ko lnet.ko
> lnet_finalize
> 3367306 3.0499 lvfs.ko lvfs.ko
> lprocfs_counter_add
> 3224905 2.9210 ptlrpc.ko ptlrpc.ko
> ptlrpc_main_check_event
> 3207810 2.9055 vmlinux vmlinux task_rq_lock
> 3029057 2.7436 vmlinux vmlinux memset
> 2890069 2.6177 vmlinux vmlinux schedule
> 2292712 2.0766 ptlrpc.ko ptlrpc.ko ptlrpc_main
> 1997346 1.8091 lnet.ko lnet.ko lnet_parse
> 1777756 1.6102 vmlinux vmlinux kfree
> 1769944 1.6031 obdclass.ko obdclass.ko key_fini
> 1745169 1.5807 lnet.ko lnet.ko LNetMDBind
> 1646427 1.4913 vmlinux vmlinux __wake_up
> 1573871 1.4255 obdclass.ko obdclass.ko keys_fill
> 1556594 1.4099 atl1e.ko atl1e.ko atl1e_intr
> 1546613 1.4008 vmlinux vmlinux __list_add
> 1512194 1.3697 ptlrpc.ko ptlrpc.ko
> ptlrpc_server_handle_request
> 1398966 1.2671 vmlinux vmlinux list_del
> 1254363 1.1361 lvfs.ko lvfs.ko
> lprocfs_counter_sub
> 1206541 1.0928 vmlinux vmlinux
> try_to_wake_up
> 1188367 1.0764 obdclass.ko obdclass.ko keys_fini
> 1158540 1.0494 obdclass.ko obdclass.ko
> class_handle2object
> 1032197 0.9349 ptlrpc.ko ptlrpc.ko
> ptlrpc_server_drop_request
> 1027855 0.9310 vmlinux vmlinux
> thread_return
> 1013642 0.9181 vmlinux vmlinux tcp_sendmsg
> 978190 0.8860 vmlinux vmlinux
> copy_user_generic_unrolled
> 961920 0.8713 lnet.ko lnet.ko LNetPut
> 929284 0.8417 obdclass.ko obdclass.ko
> lu_context_init
> 911345 0.8255 vmlinux vmlinux
> kmem_cache_free
> 879200 0.7963 vmlinux vmlinux mwait_idle
> 843759 0.7642 atl1e.ko atl1e.ko
> atl1e_xmit_frame
> 808896 0.7327 vmlinux vmlinux __kmalloc
> 806832 0.7308 ksocklnd.ko ksocklnd.ko
> ksocknal_receive
> 762878 0.6910 ptlrpc.ko ptlrpc.ko
> ptlrpc_at_add_timed
> 753020 0.6821 ptlrpc.ko ptlrpc.ko
> lustre_msg_buf_v2
> 731792 0.6628 vmlinux vmlinux
> __wake_up_common
> 680931 0.6168 vmlinux vmlinux tcp_recvmsg
> 674703 0.6111 vmlinux vmlinux
> remove_wait_queue
> 649026 0.5879 vmlinux vmlinux
> skb_release_head_state
> 648489 0.5874 mdt.ko mdt.ko
> mdt_handle_common
> 620536 0.5621 ptlrpc.ko ptlrpc.ko
> request_in_callback
> 616810 0.5587 vmlinux vmlinux memcpy
> 605268 0.5482 vmlinux vmlinux
> find_busiest_group
> 588503 0.5330 vmlinux vmlinux __switch_to
> 584848 0.5297 obdclass.ko obdclass.ko
> lu_context_key_get
> 566552 0.5132 atl1e.ko atl1e.ko
> atl1e_clean_tx_irq
> 527686 0.4780 vmlinux vmlinux tcp_v4_rcv
> 524791 0.4753 ksocklnd.ko ksocklnd.ko
> ksocknal_find_conn_locked
> 495425 0.4487 ptlrpc.ko ptlrpc.ko
> reply_out_callback
> 482759 0.4373 vmlinux vmlinux
> kmem_cache_alloc
> 459759 0.4164 ptlrpc.ko ptlrpc.ko
> ptlrpc_send_reply
> 457681 0.4145 obdclass.ko obdclass.ko
> htable_lookup
> 438283 0.3970 ksocklnd.ko ksocklnd.ko
> ksocknal_transmit
> 432296 0.3916 ksocklnd.ko ksocklnd.ko
> ksocknal_scheduler
> 426023 0.3859 lnet.ko lnet.ko
> lnet_md_unlink
> 422453 0.3826 vmlinux vmlinux __down_read
> 416777 0.3775 vmlinux vmlinux
> dev_queue_xmit
> 390667 0.3538 ptlrpc.ko ptlrpc.ko
> __req_capsule_get
> 389829 0.3531 lnet.ko lnet.ko lnet_send
> 385353 0.3490 ptlrpc.ko ptlrpc.ko
> lustre_pack_reply_v2
> 379540 0.3438 vmlinux vmlinux
> prepare_to_wait
> 375104 0.3398 vmlinux vmlinux
> tcp_transmit_skb
> 365540 0.3311 ksocklnd.ko ksocklnd.ko
> ksocknal_alloc_tx
> 363096 0.3289 vmlinux vmlinux memmove
> 349697 0.3167 lnet.ko lnet.ko
> lnet_complete_msg_locked
> 344320 0.3119 ptlrpc.ko ptlrpc.ko
> sptlrpc_wireflavor2policy
> 334530 0.3030 obdclass.ko obdclass.ko
> lu_context_exit
> 333776 0.3023 vmlinux vmlinux
> effective_prio
> 332232 0.3009 ptlrpc.ko ptlrpc.ko at_measured
> 328450 0.2975 vmlinux vmlinux
> skb_release_data
> 326971 0.2962 obdclass.ko obdclass.ko
> lu_object_put
> 323959 0.2934 ptlrpc.ko ptlrpc.ko
> ptlrpc_server_free_request
> 322772 0.2924 vmlinux vmlinux tcp_ack
> 322667 0.2923 lnet.ko lnet.ko LNetMEAttach
> 313380 0.2838 mdt.ko mdt.ko
> mdt_getattr_internal
> 311371 0.2820 vmlinux vmlinux __alloc_skb
> 308274 0.2792 ptlrpc.ko ptlrpc.ko
> ldlm_resource_find
> 306783 0.2779 lnet.ko lnet.ko
> lnet_return_credits_locked
> 303357 0.2748 vmlinux vmlinux
> ip_route_input
> 302618 0.2741 obdclass.ko obdclass.ko
> lu_context_fini
> 297917 0.2698 vmlinux vmlinux enqueue_task
> 291245 0.2638 lnet.ko lnet.ko
> lnet_recv_put
> 289224 0.2620 vmlinux vmlinux __qdisc_run
> 288297 0.2611 osd_ldiskfs.ko osd_ldiskfs.ko osd_attr_get
> 283552 0.2568 ptlrpc.ko ptlrpc.ko
> lock_res_and_lock
> 282665 0.2560 ptlrpc.ko ptlrpc.ko
> target_pack_pool_reply
> 270775 0.2453 vmlinux vmlinux
> ip_queue_xmit
> 269148 0.2438 lnet.ko lnet.ko
> lnet_enq_event_locked
> 266185 0.2411 vmlinux vmlinux
> __find_get_block
> 262071 0.2374 vmlinux vmlinux resched_task
> 261675 0.2370 lnet.ko lnet.ko
> lnet_match_md
> 260547 0.2360 ip_tables.ko ip_tables.ko ipt_do_table
> 257705 0.2334 vmlinux vmlinux
> tcp_rcv_established
> 250976 0.2273 ksocklnd.ko ksocklnd.ko
> ksocknal_queue_tx_locked
> 249668 0.2261 vmlinux vmlinux
> local_bh_enable_ip
> 248794 0.2253 vmlinux vmlinux ip_output
> 245507 0.2224 ksocklnd.ko ksocklnd.ko
> ksocknal_read_callback
> 241913 0.2191 vmlinux vmlinux put_page
> 238653 0.2162 vmlinux vmlinux
> recalc_task_prio
> 232245 0.2104 lnet.ko lnet.ko
> lnet_lookup_cookie
> 230806 0.2091 ptlrpc.ko ptlrpc.ko null_accept
> 228674 0.2071 obdclass.ko obdclass.ko
> class_handle_hash
> 228161 0.2067 ptlrpc.ko ptlrpc.ko
> sptlrpc_svc_alloc_rs
> 228019 0.2065 ptlrpc.ko ptlrpc.ko
> sptlrpc_target_export_check
> 225989 0.2047 obdclass.ko obdclass.ko
> lu_object_find_at
> 221477 0.2006 oprofiled oprofiled
> odb_update_node_with_offset
> 213570 0.1934 ptlrpc.ko ptlrpc.ko
> null_alloc_rs
> 213379 0.1933 ptlrpc.ko ptlrpc.ko
> req_capsule_has_field
> 212791 0.1927 vmlinux vmlinux
> tcp_cleanup_rbuf
> 212646 0.1926 vmlinux vmlinux list_add
> 211718 0.1918 ksocklnd.ko ksocklnd.ko
> ksocknal_process_receive
> 211447 0.1915 vmlinux vmlinux sock_wfree
> 210107 0.1903 vmlinux vmlinux sched_clock
> 207013 0.1875 ptlrpc.ko ptlrpc.ko
> __req_capsule_offset
> 206226 0.1868 ksocklnd.ko ksocklnd.ko
> ksocknal_recv_iov
> 204788 0.1855 vmlinux vmlinux
> local_bh_disable
> 202609 0.1835 libcfs.ko libcfs.ko
> cfs_alloc_flags_to_gfp
> 201355 0.1824 vmlinux vmlinux
> do_gettimeofday
> 200403 0.1815 ptlrpc.ko ptlrpc.ko
> sptlrpc_svc_unwrap_request
> 200133 0.1813 ksocklnd.ko ksocklnd.ko
> ksocknal_data_ready
> 196847 0.1783 ksocklnd.ko ksocklnd.ko
> ksocknal_free_tx
> 194325 0.1760 ptlrpc.ko ptlrpc.ko
> ptlrpc_master_callback
> 191462 0.1734 ptlrpc.ko ptlrpc.ko
> ptlrpc_check_req
> 191407 0.1734 lnet.ko lnet.ko
> lnet_commit_md
> 191155 0.1731 libcfs.ko libcfs.ko
> lc_watchdog_touch
> 190496 0.1725 mdt.ko mdt.ko mdt_getattr
> 189965 0.1721 obdclass.ko obdclass.ko
> class_export_put
> 186893 0.1693 atl1e.ko atl1e.ko
> atl1e_clean_rx_irq
> 186683 0.1691 lnet.ko lnet.ko
> lnet_find_peer_locked
> 186143 0.1686 ldiskfs.ko ldiskfs.ko
> ldiskfs_find_entry
> 185352 0.1679 iptable_filter.ko iptable_filter.ko ipt_hook
> 184570 0.1672 ptlrpc.ko ptlrpc.ko
> ldlm_resource_get
> 184553 0.1672 ksocklnd.ko ksocklnd.ko
> ksocknal_find_peer_locked
> 182066 0.1649 vmlinux vmlinux
> add_wait_queue_exclusive
> 179692 0.1628 vmlinux vmlinux cond_resched
> 177681 0.1609 ksocklnd.ko ksocklnd.ko
> ksocknal_tx_prep
> 172136 0.1559 ptlrpc.ko ptlrpc.ko null_free_rs
> 168889 0.1530 libstdc++.so.6.0.8 opreport /usr/lib64/
> libstdc++.so.6.0.8
> 167529 0.1517 ptlrpc.ko ptlrpc.ko
> lustre_msg_get_opc
> 166144 0.1505 ksocklnd.ko ksocklnd.ko
> ksocknal_lib_recv_iov
> 162940 0.1476 vmlinux vmlinux
> recalc_sigpending_tsk
> 161651 0.1464 ptlrpc.ko ptlrpc.ko
> sptlrpc_svc_ctx_decref
> 161047 0.1459 oprofiled oprofiled
> find_kernel_image
> 160557 0.1454 lnet.ko lnet.ko
> lnet_nid2ni_locked
> 159886 0.1448 ptlrpc.ko ptlrpc.ko
> lustre_shrink_msg_v2
> 158071 0.1432 obdclass.ko obdclass.ko
> class_export_get
> 157994 0.1431 vmlinux vmlinux __up_read
> 157418 0.1426 ldiskfs.ko ldiskfs.ko
> __ldiskfs_get_inode_loc
> 157222 0.1424 libcfs.ko libcfs.ko
> cfs_block_sigs
> 154417 0.1399 vmlinux vmlinux csum_partial
> 153571 0.1391 lnet.ko lnet.ko LNetMDAttach
> 149419 0.1353 vmlinux vmlinux
> __activate_task
> 149345 0.1353 vmlinux vmlinux
> __tcp_push_pending_frames
> 148491 0.1345 ksocklnd.ko ksocklnd.ko
> ksocknal_launch_packet
> 147683 0.1338 vmlinux vmlinux sock_recvmsg
> 144704 0.1311 ptlrpc.ko ptlrpc.ko
> ptlrpc_connection_addref
> 144254 0.1307 obdclass.ko obdclass.ko
> lu_object_find
> 139774 0.1266 vmlinux vmlinux
> pfifo_fast_enqueue
> 139234 0.1261 lnet.ko lnet.ko
> lnet_try_match_md
> 138095 0.1251 vmlinux vmlinux
> do_gettimeoffset_tsc
> 137704 0.1247 ptlrpc.ko ptlrpc.ko
> req_capsule_shrink
> 137273 0.1243 vmlinux vmlinux
> lock_timer_base
> 137021 0.1241 vmlinux vmlinux release_sock
> 135984 0.1232 mdt.ko mdt.ko
> mdt_unpack_req_pack_rep
> 135442 0.1227 vmlinux vmlinux
> generic_getxattr
> 132666 0.1202 obdclass.ko obdclass.ko
> class_connected_export
> 131849 0.1194 ptlrpc.ko ptlrpc.ko
> ptlrpc_buf_need_swab
> 131417 0.1190 libcfs.ko libcfs.ko cfs_alloc
> 131039 0.1187 ptlrpc.ko ptlrpc.ko
> target_send_reply
> 131009 0.1187 libcfs.ko libcfs.ko cfs_hash_add
> 130974 0.1186 vmlinux vmlinux __mod_timer
> 128345 0.1162 osd_ldiskfs.ko osd_ldiskfs.ko
> osd_object_read_lock
> 126437 0.1145 vmlinux vmlinux netif_rx
> 125451 0.1136 vmlinux vmlinux
> sock_common_recvmsg
> 124448 0.1127 ptlrpc.ko ptlrpc.ko
> ldlm_lock_put
> 123629 0.1120 lnet.ko lnet.ko
> lnet_post_send_locked
> 120179 0.1089 ptlrpc.ko ptlrpc.ko
> lustre_free_reply_state
> 119279 0.1080 ptlrpc.ko ptlrpc.ko ptl_send_buf
> 118334 0.1072 vmlinux vmlinux
> default_wake_function
> 117874 0.1068 ptlrpc.ko ptlrpc.ko
> sptlrpc_svc_wrap_reply
> 117338 0.1063 vmlinux vmlinux dequeue_task
> 116492 0.1055 opreport opreport
> extra_images::matcher::operator()(std::string const&) const
> 115787 0.1049 vmlinux vmlinux
> tcp_current_mss
> 114841 0.1040 mdt.ko mdt.ko
> mdt_regular_handle
> 114397 0.1036 vmlinux vmlinux
> __copy_skb_header
> 113796 0.1031 ptlrpc.ko ptlrpc.ko
> ptlrpc_server_post_idle_rqbds
> 112548 0.1019 vmlinux vmlinux
> _atomic_dec_and_lock
> 112275 0.1017 vmlinux vmlinux
> find_first_bit
> 112062 0.1015 vmlinux vmlinux
> find_next_bit
> 109904 0.0995 vmlinux vmlinux skb_clone
> 109041 0.0988 libcfs.ko libcfs.ko
> lc_watchdog_disable
> 107246 0.0971 ptlrpc.ko ptlrpc.ko
> __lustre_unpack_msg
> 106395 0.0964 ptlrpc.ko ptlrpc.ko
> __req_format_is_sane
> 105582 0.0956 mdt.ko mdt.ko
> mdt_shrink_reply
> 104623 0.0948 ptlrpc.ko ptlrpc.ko
> ptlrpc_update_export_timer
> 103974 0.0942 vmlinux vmlinux
> __rcu_process_callbacks
> 102272 0.0926 lnet.ko lnet.ko lib_md_build
> 100840 0.0913 lnet.ko lnet.ko
> lnet_md_deconstruct
> 100757 0.0913 ptlrpc.ko ptlrpc.ko
> lustre_msg_size_v2
> 99939 0.0905 vmlinux vmlinux ip_rcv
> 99033 0.0897 ptlrpc.ko ptlrpc.ko
> ldlm_lock_create
> 98275 0.0890 vmlinux vmlinux mod_timer
> 97811 0.0886 vmlinux vmlinux kfree_skbmem
> 96558 0.0875 mdd.ko mdd.ko mdd_get_md
> 96390 0.0873 ptlrpc.ko ptlrpc.ko
> req_capsule_init_area
> 95672 0.0867 mdt.ko mdt.ko
> mdt_check_ucred
> 95263 0.0863 lnet.ko lnet.ko lnet_ni_recv
> 94561 0.0856 mdt.ko mdt.ko
> mdt_reint_open
> 93791 0.0850 mdd.ko mdd.ko
> mdd_read_lock
> 93714 0.0849 osd_ldiskfs.ko osd_ldiskfs.ko
> osd_object_auth
> 90762 0.0822 vmlinux vmlinux
> IRQ0x4a_interrupt
> 90128 0.0816 ksocklnd.ko ksocklnd.ko
> ksocknal_recv
> 89651 0.0812 obdclass.ko obdclass.ko
> lu_time_key_exit
> 89647 0.0812 vmlinux vmlinux
> netif_receive_skb
> 88498 0.0802 mdt.ko mdt.ko
> mdt_pack_attr2body
> 87280 0.0791 ksocklnd.ko ksocklnd.ko
> ksocknal_new_packet
> 86200 0.0781 mdt.ko mdt.ko
> mdt_lock_handle_init
> 84561 0.0766 ptlrpc.ko ptlrpc.ko
> req_capsule_filled_sizes
> 84364 0.0764 ksocklnd.ko ksocklnd.ko
> ksocknal_send
> 83359 0.0755 vmlinux vmlinux nf_iterate
> 82758 0.0750 ptlrpc.ko ptlrpc.ko
> ldlm_inodebits_compat_queue
> 82041 0.0743 oprofiled oprofiled sfile_find
> 81178 0.0735 ksocklnd.ko ksocklnd.ko
> ksocknal_match_tx_v3
> 80916 0.0733 vmlinux vmlinux
> xattr_resolve_name
> 80849 0.0732 mdd.ko mdd.ko
> mdd_ucred_key_fini
> 80835 0.0732 osd_ldiskfs.ko osd_ldiskfs.ko
> osd_xattr_get
> 80829 0.0732 lustre.ko lustre.ko
> vvp_session_key_fini
> 80522 0.0729 mdt.ko mdt.ko
> mdt_object_lock
> 80471 0.0729 vmlinux vmlinux
> process_backlog
> 80218 0.0727 ptlrpc.ko ptlrpc.ko __req_msg
> 80150 0.0726 vmlinux vmlinux
> alternate_node_alloc
> 79896 0.0724 lvfs.ko lvfs.ko
> upcall_cache_get_entry
> 78040 0.0707 ptlrpc.ko ptlrpc.ko
> ldlm_handle_enqueue0
> 77957 0.0706 ptlrpc.ko ptlrpc.ko
> ldlm_lock_remove_from_lru
> 77256 0.0700 vmlinux vmlinux
> tcp_event_data_recv
> 77006 0.0697 vmlinux vmlinux lock_sock
> 75922 0.0688 mdd.ko mdd.ko
> mdd_attr_get_internal
> 74796 0.0677 ptlrpc.ko ptlrpc.ko
> target_committed_to_req
> 74788 0.0677 ksocklnd.ko ksocklnd.ko
> ksocknal_process_transmit
> 74656 0.0676 cmm.ko cmm.ko cml_attr_get
> 74576 0.0675 ksocklnd.ko ksocklnd.ko
> ksocknal_next_tx_carrier
> 73666 0.0667 lnet.ko lnet.ko lnet_ni_send
> 73296 0.0664 ksocklnd.ko ksocklnd.ko
> ksocknal_tx_done
> 73023 0.0661 lnet.ko lnet.ko
> lnet_setpayloadbuffer
> 72641 0.0658 osd_ldiskfs.ko osd_ldiskfs.ko
> lu_device_is_osd
> 72051 0.0653 vmlinux vmlinux
> skb_copy_datagram_iovec
> 71925 0.0651 ptlrpc.ko ptlrpc.ko
> ptlrpc_server_allow_normal
> 70839 0.0642 vmlinux vmlinux
> local_bh_enable
> 70244 0.0636 vmlinux vmlinux sock_sendmsg
> 70131 0.0635 vmlinux vmlinux get_task_mm
> 69625 0.0631 opreport opreport (anonymous
> namespace)::module_matcher::operator()(std::string const&) const
> 69573 0.0630 vmlinux vmlinux
> main_timer_handler
> 68370 0.0619 mdd.ko mdd.ko
> mdd_iattr_get
> 67955 0.0616 ptlrpc.ko ptlrpc.ko
> ptlrpc_connection_put
> 67875 0.0615 ptlrpc.ko ptlrpc.ko
> ldlm_pool_add
> 66775 0.0605 mdd.ko mdd.ko
> __mdd_lmm_get
> 66361 0.0601 vmlinux vmlinux
> __sk_dst_check
> 66330 0.0601 ksocklnd.ko ksocklnd.ko
> ksocknal_pack_msg_v2
> 66036 0.0598 oprofiled oprofiled
> pop_buffer_value
> 65776 0.0596 mdd.ko mdd.ko
> mdd_capainfo_key_fini
> 64314 0.0583 lnet.ko lnet.ko
> lnet_match_blocked_msg
> 64257 0.0582 vmlinux vmlinux
> recalc_sigpending
> 64184 0.0581 ptlrpc.ko ptlrpc.ko
> unlock_res_and_lock
> 64009 0.0580 vmlinux vmlinux
> tcp_check_space
> 63851 0.0578 libc-2.5.so top _IO_vfscanf
> 62692 0.0568 vmlinux vmlinux
> tcp_rcv_space_adjust
> 62667 0.0568 ldiskfs.ko ldiskfs.ko
> ldiskfs_xattr_get
> 62548 0.0567 ptlrpc.ko ptlrpc.ko
> req_capsule_extend
> 61313 0.0555 mdd.ko mdd.ko md_capainfo
> 61159 0.0554 libcfs.ko libcfs.ko
> cfs_waitq_active
> 59511 0.0539 vmlinux vmlinux __do_IRQ
> 59201 0.0536 ptlrpc.ko ptlrpc.ko
> req_capsule_init
> 58802 0.0533 vmlinux vmlinux
> interleave_nodes
> 58475 0.0530 ptlrpc.ko ptlrpc.ko
> lustre_shrink_msg
> 58425 0.0529 mdt.ko mdt.ko
> mdt_object_find
> 57699 0.0523 ksocklnd.ko ksocklnd.ko
> ksocknal_lib_send_iov
> 57284 0.0519 ptlrpc.ko ptlrpc.ko
> sptlrpc_svc_free_rs
> 57257 0.0519 oprofile.ko oprofile.ko sync_buffer
> 57155 0.0518 lnet.ko lnet.ko
> lnet_extract_iov
> 56945 0.0516 lnet.ko lnet.ko
> lnet_peer_alive_locked
> 56904 0.0515 osc.ko osc.ko
> osc_session_init
> 56737 0.0514 mdd.ko mdd.ko mdd_la_get
> 56521 0.0512 mdt.ko mdt.ko
> mdt_body_reverse_idmap
> 56498 0.0512 osd_ldiskfs.ko osd_ldiskfs.ko osd_dev
> 56151 0.0509 lov.ko lov.ko
> lov_session_key_init
> 55877 0.0506 vmlinux vmlinux
> __write_lock_failed
> 55655 0.0504 ldiskfs.ko ldiskfs.ko
> ldiskfs_get_blocks_handle
> 55472 0.0502 ptlrpc.ko ptlrpc.ko
> ldlm_cli_enqueue_local
> 54412 0.0493 vmlinux vmlinux
> ip_local_deliver
> 52606 0.0476 ptlrpc.ko ptlrpc.ko
> ptlrpc_server_request_pending
> 51767 0.0469 vmlinux vmlinux __do_softirq
> 51562 0.0467 ptlrpc.ko ptlrpc.ko
> req_capsule_field_present
> 51549 0.0467 vmlinux vmlinux inet_sendmsg
> 51414 0.0466 mdt.ko mdt.ko
> mdt_intent_policy
> 51296 0.0465 ksocklnd.ko ksocklnd.ko
> ksocknal_queue_tx_msg_v2
> 51009 0.0462 libcfs.ko libcfs.ko
> cfs_hash_rlock
> 50977 0.0462 vmlinux vmlinux __brelse
> 50811 0.0460 oprofile.ko oprofile.ko
> add_event_entry
> ...
>
> With regards,
> Popova Ekaterina
> _______________________________________________
> Lustre-discuss mailing list
> Lustre-discuss at lists.lustre.org
> http://lists.lustre.org/mailman/listinfo/lustre-discuss
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20110720/081a2cfc/attachment.htm>
More information about the lustre-discuss
mailing list