[lustre-discuss] weird issue w. lnet routers

[John Casu]

just built a system w. lnet routers that bridge Infiniband & 100GbE, using Centos built in Infiniband support
servers are Infiniband, clients are 100GbE (connectx-4 cards)

my direct write performance from clients over Infiniband is around 15GB/s

When I introduced the lnet routers, performance dropped to 10GB/s

Thought the problem was an MTU of 1500, but when I changed the MTUs to 9000
performance dropped to 3GB/s.

When I tuned according to John Fragella's LUG slides, things went even slower (1.5GB/s write)

does anyone have any ideas on what I'm doing wrong??

thanks,
-john c.