[Lustre-discuss] Slow Copy (Small Files) 1 RPC In Flight?

[Andrew Mast]

Hi Brett,

Sorry, I think my choice in wording is not correct.

One VM is holding the metadata and the objects. I guess that would mean ti
is the OSS and MDS?
Another VM is the client.It has mounted the lusture filesystem and also has
some local disks. The test is to just to use cp to read data to local disk.

Thanks,
Andy



On Fri, Jun 21, 2013 at 3:22 PM, Lee, Brett <brett.lee at intel.com> wrote:

>  “test between 2 virtual machines from MDT/OST server to client's local
> disk.”****
>
> ** **
>
> Andrew,****
>
> ** **
>
> I’m confused by the description of your test.  Can you clarify?****
>
> ** **
>
> --****
>
> Brett Lee****
>
> Sr. Systems Engineer****
>
> Intel High Performance Data Division****
>
> ** **
>
> *From:* lustre-discuss-bounces at lists.lustre.org [mailto:
> lustre-discuss-bounces at lists.lustre.org] *On Behalf Of *Andrew Mast
> *Sent:* Friday, June 21, 2013 3:42 PM
> *To:* lustre-discuss at lists.lustre.org
> *Subject:* [Lustre-discuss] Slow Copy (Small Files) 1 RPC In Flight?****
>
> ** **
>
> Hello, I am new to Lustre and wanted to run a small simple small copy test
> between 2 virtual machines from MDT/OST server to client's local disk.****
>
> ** **
>
> I realize small file performance is never fast, but this seems
> particularly slow considering the data is all buffered in memory with
> little to no disk activity.****
>
> ** **
>
> Setup Info****
>
> Version is 2.4.50****
>
> Average file size is small. < 10KB****
>
> The amount of data being copied is about 250MB.****
>
> The VMs are on separate hosts.****
>
> ** **
>
> Performance****
>
> 7 minutes over a gigabit network. ****
>
> NFS takes only 3 minutes.****
>
> ** **
>
> Observations****
>
> iostat on the OST/MDT is usually 0% during the copy. Assuming all buffered.
> ****
>
> Additional network traffic is minimal. ****
>
> CPU load on the VMs is 15-20% during copy.****
>
> ** **
>
> RPC stats on the client shows only 1 RPC in flight at a time. max inflight
> is set to 64. Is that expected behavior for a copy?****
>
> ** **
>
> Here is a snapshot of rpc_stats early during the copy:****
>
> ** **
>
>    read                                    write****
>
> pages per rpc         rpcs   % cum % |       rpcs   % cum %****
>
> 1:                         1653  90  90   |          0   0   0****
>
> 2:                          164   8  98   |          0   0   0****
>
> 4:                            7   0  99   |          0   0   0****
>
> 8:                            3   0  99   |          0   0   0****
>
> 16:                          3   0  99   |          0   0   0****
>
> 32:                          5   0  99   |          0   0   0****
>
> 64:                          0   0  99   |          0   0   0****
>
> 128:                        1   0 100   |          0   0   0****
>
> ** **
>
>                                   read                             write**
> **
>
> rpcs in flight        rpcs   % cum % |       rpcs   % cum %****
>
> 0:                            0   0   0   |          0   0   0****
>
> 1:                         1836 100 100   |          0   0   0****
>
> ** **
>
>                                   read                             write**
> **
>
> offset                rpcs   % cum % |       rpcs   % cum %****
>
> 0:                         1836 100 100   |          0   0   0****
>
> ** **
>
> As I am new, any suggestions for what to look for or improve would be
> greatly appreciated.****
>
> _______________________________________________
> Lustre-discuss mailing list
> Lustre-discuss at lists.lustre.org
> http://lists.lustre.org/mailman/listinfo/lustre-discuss
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20130621/790a4355/attachment.htm>