[Lustre-discuss] Lustre read performance decay when OSSes are assigned in two different subnet

[zhengfeng]

Dear all,
We met one problem about Lustre read performance decay when OSSes are assigned in two different subnet.
Describing that in the following diagram:
diagram 1, OSS in different subnets:
Client (subnet 10.0.1.2)
  |
  |
  |
                Switch 
|          | 
|          |
|          |
   OSS1       OSS2
(10.0.2.2)   (10.0.3.2)
For diagram 1, we made the CLient OSS1 and OSS2 in 3 different subnets. the switch used is able forward all packages. 
Use dd cmd to test r/w performance， write/rad data to/from to OSS1 and OSS2 at the same time:
test result:
[root at client client]# time dd if=test2 of=/dev/null bs=1M count=2000
2000+0 records in
2000+0 records out
2097152000 bytes (2.1 GB) copied, 53.5922 seconds, 39.1 MB/s

real 0m53.796s
user 0m0.005s
sys 0m2.914s


diagram 2, OSS in same subnet:
Client (subnet 10.0.1.2)
  |
  |
  |
                Switch 
|          | 
|          |
|          |
   OSS1       OSS2
(10.0.2.2, 10.0.2.3, at same subnet)

for diagram 2, we assigned OSS1 and OSS2 at the same subnet, then test:
test result:
[root at client219 client]# time dd of=/dev/null if=test1 bs=1M 
10000+0 records in
10000+0 records out
10485760000 bytes (10 GB) copied, 193.07 seconds, 54.3 MB/s


conclusion:
In different subnets, the OSS read performance is 39.1 MB/s, while OSS in
same subnet, the read performance is 54.3 MB/s. the performance decays so much.

Question:
Why using different subnets in lustre, the performance decayed?
Anyone had met such problems? Many thanks for your answers and advice.




B.R.
Feng
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20120315/488d8e78/attachment.htm>