[Lustre-discuss] some bonnie benchmark numbers

[Ms. Megan Larko]

Hi,

Just sharing some numbers I got running bonnie on a CentOS 5 box using
lustre kernel 2.6.18-53.1.13.el5_lustre.1.6.4.3smp.  The box has four
dual-core Intel Xeon 2.66GHz CPUs (all 8 cores recognized) and 16Gb
RAM.   The test hardware is one Xstore 16 bay JBOD filled with Hitachi
1Tb Ultrastar sATA drives.   The JBOD is connected to the server via
an LSI 8888ELP card (PCI-e slot).  My two test subjects are a RAID5
(R5) and RAID6 (R6) 5.4Tb array  (RAID done on LSI 8888ELP and
exported to box as one big disk).  They were partitioned using parted
type GFI (not 83 linux) and formatted using mk.fs -j -m 0 (no inode
options to mkfs were given).

I don't have any raw numbers because I was never able to get the
lustre IOkit program sgpdd-survey to read or write to any of the
disks; I think it somehow just couldn't get past the LSI controller at
some raw level....

My numbers are not quite as good as those of Robin Humble on Lustre
1.6.4.3 (reprinted below).

My bonnie results ext3 only R5 and R6:
/media/TEST
              -------Sequential Output-------- ---Sequential Input-- --Random--
              -Per Char- --Block--- -Rewrite-- -Per Char- --Block--- --Seeks---
Machine    MB K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU  /sec %CPU
R5       16384 72198 88.9 276205 61.5 113861 19.2 68358 74.0 499889
27.3 4885.9  6.5

              -Per Char- --Block--- -Rewrite-- -Per Char- --Block--- --Seeks---
Machine    MB K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU  /sec %CPU
R5       32768 79577 96.1 218047 52.0 94666 17.6 67140 73.8 258427
18.5 380.9  0.8

             -------Sequential Output-------- ---Sequential Input-- --Random--
              -Per Char- --Block--- -Rewrite-- -Per Char- --Block--- --Seeks---
Machine    MB K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU  /sec %CPU
R6       16384 77530 93.5 363416 82.0 140333 23.6 72707 78.4 626962
33.6 5079.7  5.2

              -------Sequential Output-------- ---Sequential Input-- --Random--
              -Per Char- --Block--- -Rewrite-- -Per Char- --Block--- --Seeks---
Machine    MB K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU  /sec %CPU
R6       32768 80428 96.9 286178 68.8 111837 21.0 70525 77.6 333741
24.0 369.0  0.8

Again after formatting as Lustre disks using the format command:
mkfs.lustre --fsname crew5 --ost --reformat --mgsnode=ic-mds1 at o2ib
/dev/sdg (and /dev/sdh as --fsname crew6).

The bonnie test was run on the same hw (CPU and RAM) on a different
physical machine to separate the client from the server as it would be
actually used.  Note:  On the mgs, the mgt for both R5 /sdg and R6
/sdh was a 2Gb USB stick.  I don't know if that matters either good or
bad.

              -------Sequential Output-------- ---Sequential Input-- --Random--
              -Per Char- --Block--- -Rewrite-- -Per Char- --Block--- --Seeks---
Machine    MB K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU  /sec %CPU
R5       16384 77362 100.0 277523 58.2 146215 56.8 69092 98.4 407877
99.9 1467.7  4.6

              -------Sequential Output-------- ---Sequential Input-- --Random--
              -Per Char- --Block--- -Rewrite-- -Per Char- --Block--- --Seeks---
Machine    MB K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU  /sec %CPU
R5       32768 76785 100.0 270494 58.0 147043 57.1 69446 98.4 411353
99.9 470.9  2.1

              -------Sequential Output-------- ---Sequential Input-- --Random--
              -Per Char- --Block--- -Rewrite-- -Per Char- --Block--- --Seeks---
Machine    MB K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU  /sec %CPU
R6       16384 77924 100.0 300328 62.8 140733 55.5 69669 98.4 407137
99.9 1476.3  3.7

              -------Sequential Output-------- ---Sequential Input-- --Random--
              -Per Char- --Block--- -Rewrite-- -Per Char- --Block--- --Seeks---
Machine    MB K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU  /sec %CPU
R6       32768 74561 98.4 298559 64.6 142042 56.2 69391 98.4 400980
98.7 480.5  2.5


---------------------------------   Robin Humble's results
------------------------------------------------------------
                     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
> Machine   Size:chnk K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
> rhel4 oss  16G:256k 84624  99 842138 92 310044 91 77675  99 491239 96 285.8  10
> rhel5 oss  16G:256k 86085  99 827731 95 327007 97 79639 100 495487 98 456.2  18
-------------------------------------------------------------------------------------------------------------------------------

The reader is free to form his/her own conclusions.

megan