[Lustre-discuss] Slow Copy (Small Files) 1 RPC In Flight?

[Lee, Brett]

"test between 2 virtual machines from MDT/OST server to client's local disk."

Andrew,

I'm confused by the description of your test.  Can you clarify?

--
Brett Lee
Sr. Systems Engineer
Intel High Performance Data Division

From: lustre-discuss-bounces at lists.lustre.org [mailto:lustre-discuss-bounces at lists.lustre.org] On Behalf Of Andrew Mast
Sent: Friday, June 21, 2013 3:42 PM
To: lustre-discuss at lists.lustre.org
Subject: [Lustre-discuss] Slow Copy (Small Files) 1 RPC In Flight?

Hello, I am new to Lustre and wanted to run a small simple small copy test between 2 virtual machines from MDT/OST server to client's local disk.

I realize small file performance is never fast, but this seems particularly slow considering the data is all buffered in memory with little to no disk activity.

Setup Info
Version is 2.4.50
Average file size is small. < 10KB
The amount of data being copied is about 250MB.
The VMs are on separate hosts.

Performance
7 minutes over a gigabit network.
NFS takes only 3 minutes.

Observations
iostat on the OST/MDT is usually 0% during the copy. Assuming all buffered.
Additional network traffic is minimal.
CPU load on the VMs is 15-20% during copy.

RPC stats on the client shows only 1 RPC in flight at a time. max inflight is set to 64. Is that expected behavior for a copy?

Here is a snapshot of rpc_stats early during the copy:

   read                                    write
pages per rpc         rpcs   % cum % |       rpcs   % cum %
1:                         1653  90  90   |          0   0   0
2:                          164   8  98   |          0   0   0
4:                            7   0  99   |          0   0   0
8:                            3   0  99   |          0   0   0
16:                          3   0  99   |          0   0   0
32:                          5   0  99   |          0   0   0
64:                          0   0  99   |          0   0   0
128:                        1   0 100   |          0   0   0

                                  read                             write
rpcs in flight        rpcs   % cum % |       rpcs   % cum %
0:                            0   0   0   |          0   0   0
1:                         1836 100 100   |          0   0   0

                                  read                             write
offset                rpcs   % cum % |       rpcs   % cum %
0:                         1836 100 100   |          0   0   0

As I am new, any suggestions for what to look for or improve would be greatly appreciated.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20130621/26353b26/attachment.htm>