[lustre-discuss] Lustre Clients on KVM Virtual Guests on CentOS 7

[Martin Hecht]

to my knowledge the virtio driver adds less overhead than the other ones
that try to emulate some specific behavior of a hardware device.
However, for RDMA I think you should try to hand through the physical
device to the vm rather than adding both to a bridge.

An lnet router is another possibility. You would define another tcp
network in addition to the o2ib lnet. The tcp network has an interface
on the kvm host and the guests are attached to it (it could be the
internal default network that you usually have with kvm when you set up
nat for you vms. You have to enable forwarding on the kvm host and you
have to add routes to the tcp network on all lustre servers and add a
route to the o2ib network on the virtualized clients.

Lnet routing costs almost nothing, but the virtual network in kvm would
probably be the bottleneck. I'd expect that a device handed through to
the guest offers much more performance. Some cards are advertised to
support being split up into several virtual devices which can be handed
through to different vms.


On 10/2/19 5:30 PM, Ault, James R (Contractor) wrote:
> Does anyone on this list have experience running lustre clients on Virtual Guests running in a QEMU/KVM environment (using CentOS 7)?
> The desired configuration for the base host is Mellanox Connect-X 5 for 100GB Ethernet, but the KVM Guests are running over a Bridged Ethernet using the virtio from KVM.
>
> The device shows on the Virtual Guest as eth0, and we have not been successful enabling RDMA over this Linux Ethernet Bridge device so far.
>
> Other KVM networking options for this network device displayed in virt-manager are "e1000", "rtl8139" and "Hypervisor default" instead of "virtio".
>
> Another possibility could be to use an LNET router to allow these Virtual Guests to talk to a Lustre server over the TCP driver.
>
> Servers: CentOS 7.6, Lustre 2.12.1, ZFS devices
> Clients: CentOS 7.6, Lustre 2.12.2
>
>
> Error messages displayed include:
>
> # lnetctl add --net o2ib --if eth0
> add:
>    net:
>       errno: -100
>       descr: "cannot add network: Network is down"
> # lnetctl net show
> net:
>    - net type: lo
>      local NI(s):
>          - nid: 0 at lo
>           status: up
>
> and nothing else.
>
> Thanks in advance for any helpful recommendations.