[Lustre-devel] COS performance issues

Alexander Zarochentsev Alexander.Zarochentsev at Sun.COM
Sat Oct 18 03:34:54 PDT 2008


On 18 October 2008 00:55:03 Robert Read wrote:
> On Oct 17, 2008, at 01:07 , Alexander Zarochentsev wrote:
> > The massive create tests shows the following results:
> >
> > COS=1
> > Rate: 2085.73 creates/sec (total: 2 threads 625719 creates 300
> > secs) Rate: 2072.38 creates/sec (total: 2 threads 621713 creates
> > 300 secs) Rate: 2053.80 creates/sec (total: 2 threads 616141
> > creates 300 secs)
> >
> > COS=0
> > Rate: 2128.22 creates/sec (total: 2 threads 638465 creates 300
> > secs) Rate: 2151.05 creates/sec (total: 2 threads 645316 creates
> > 300 secs) Rate: 2169.91 creates/sec (total: 2 threads 650974
> > creates 300 secs)
> >
> > Clean HEAD
> >
> > Rate: 2199.67 creates/sec (total: 2 threads 659900 creates 300
> > secs) Rate: 2213.95 creates/sec (total: 2 threads 664184 creates
> > 300 secs) Rate: 2186.74 creates/sec (total: 2 threads 658209
> > creates 301 secs) Rate: 2206.10 creates/sec (total: 2 threads
> > 661831 creates 300 secs) Rate: 2217.49 creates/sec (total: 2
> > threads 665247 creates 300 secs) Rate: 2230.86 creates/sec (total:
> > 2 threads 669258 creates 300 secs)
>
> Any idea why COS=0 was consistently a bit slower than HEAD?

I only suspect the ENTRY/EXIT stuff I added to several ptlrpc functions.
next time I will try to build Lustre with CDEBUG_ENTRY_EXIT=0.

> robert

The latest results I got after increasing of ldlm resource
hashtable size are:

zam at lts-head:~$ zgrep Rate: m-ht-improved-WASHIE3.4-20081017.log

COS = 1
Rate: 1887.57 creates/sec (total: 2 threads 566271 creates 300 secs)
Rate: 1980.18 creates/sec (total: 2 threads 594053 creates 300 secs)
Rate: 1923.76 creates/sec (total: 2 threads 577129 creates 300 secs)

COS = 0
Rate: 1804.92 creates/sec (total: 2 threads 541475 creates 300 secs)
Rate: 1915.29 creates/sec (total: 2 threads 574587 creates 300 secs)
Rate: 2025.70 creates/sec (total: 2 threads 607711 creates 300 secs)


CLEAN HEAD:
zam at lts-head:~$ zgrep Rate: m-cleanhead-WASHIE3.4-20081017.log
Rate: 1986.13 creates/sec (total: 2 threads 595839 creates 300 secs)
Rate: 2009.84 creates/sec (total: 2 threads 602952 creates 300 secs)
Rate: 1887.05 creates/sec (total: 2 threads 566115 creates 300 secs)
Rate: 2060.59 creates/sec (total: 2 threads 618178 creates 300 secs)
Rate: 1868.89 creates/sec (total: 2 threads 560666 creates 300 secs)
Rate: 1976.95 creates/sec (total: 2 threads 594084 creates 301 secs)
zam at lts-head:~$

average values:

COS=1 : 1930.5
COS=0 : 1915.3
HEAD  : 1964.9

The results are done on another test cluster 
so they are not directly comparable with the previous ones.

The profiling chart for COS=1, there is no ldlm_resource_find anymore in top20:

198152    2.9052  e1000.ko                 e1000                    e1000_irq_enable
158535    2.3243  e1000.ko                 e1000                    e1000_intr_msi
152375    2.2340  vmlinux                  vmlinux                  schedule
148709    2.1803  obdclass.ko              obdclass                 htable_lookup
109440    1.6045  lvfs.ko                  lvfs                     lprocfs_counter_add
102773    1.5068  vmlinux                  vmlinux                  __switch_to
95607     1.4017  vmlinux                  vmlinux                  __find_get_block
90447     1.3261  vmlinux                  vmlinux                  kfree
88713     1.3006  vmlinux                  vmlinux                  mwait_idle
84511     1.2390  vmlinux                  vmlinux                  try_to_wake_up
83801     1.2286  vmlinux                  vmlinux                  memset
79735     1.1690  vmlinux                  vmlinux                  __kmalloc
78594     1.1523  oprofiled                oprofiled                (no symbols)
71184     1.0437  ksocklnd.ko              ksocklnd                 ksocknal_scheduler
65613     0.9620  vmlinux                  vmlinux                  kmem_cache_free
64525     0.9460  mds.ko                   mds                      mds_lov_dump_objids
64208     0.9414  ptlrpc.ko                ptlrpc                   ptlrpc_main
63260     0.9275  vmlinux                  vmlinux                  memmove
56148     0.8232  vmlinux                  vmlinux                  find_get_page
55011     0.8065  vmlinux                  vmlinux                  kmem_cache_alloc

I attached all performance enhancement patches 
(except the latest patch of ldlm resource hash table size increase) 
to the COS bug (Bug 15393).

Thanks,
-- 
Alexander "Zam" Zarochentsev
Staff Engineer
Lustre Group, Sun Microsystems



More information about the lustre-devel mailing list