[lustre-discuss] Lustre rpm install creating a file that breaks lustre

[Andreas Dilger]

The "alias ko2iblnd-opa" line in ko2iblnd.conf doesn't do anything unless OPA (or older QLogic) interfaces are detected on your system via the /usr/sbin/ko2iblnd-probe script.

This indirection is used to allow better default module parameters between OPA and MLX devices, which can't easily be determined inside the kernel, and would be hard to change after the fact.  In theory, if there were substantially better ko2iblnd module parameters for new MLX or RoCE devices, then this same mechanism could be used to do similar interface-specific tunings in userspace.

You could run that script with the last line commented out (or replace "exec->echo" on the last line) to see what it is doing.

Cheers, Andreas

On Oct 2, 2019, at 12:55, Kurt Strosahl <strosahl at jlab.org<mailto:strosahl at jlab.org>> wrote:



Good Afternoon,



    While getting lustre 2.10.8 running on a RHEL 7.7 system I found that the RPM install was putting a file in /etc/modprobe.d that was preventing lnet from starting properly.



the file is ko2iblnd.conf, which contains the following...



alias ko2iblnd-opa ko2iblnd
options ko2iblnd-opa peer_credits=128 peer_credits_hiw=64 credits=1024 concurrent_sends=256 ntx=2048 map_on_demand=32 fmr_pool_size=2048 fmr_flush_trigger=512 fmr_cache=1 conns_per_peer=4



install ko2iblnd /usr/sbin/ko2iblnd-probe



Our system is running infiniband, not omnipath.  So I'm mot sure why this file is being put in place.  Removing the file allows lnet to start properly.

Cheers, Andreas
--
Andreas Dilger
Principal Lustre Architect
Whamcloud






-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20191005/b067f8e4/attachment.html>