[lustre-discuss] Lustre poor performance

[Dilger, Andreas]

On Aug 23, 2017, at 08:39, Mohr Jr, Richard Frank (Rick Mohr) <rmohr at utk.edu> wrote:
> 
> 
>> On Aug 22, 2017, at 7:14 PM, Riccardo Veraldi <Riccardo.Veraldi at cnaf.infn.it> wrote:
>> 
>> On 8/22/17 9:22 AM, Mannthey, Keith wrote:
>>> Younot expected.
>>> 
>> yes they are automatically used on my Mellanox and the script ko2iblnd-probe seems like not working properly.
> 
> The ko2iblnd-probe script looks in /sys/class/infiniband for device names starting with “hfi” or “qib”.  If it detects those, it decides that the “profile” it should use is “opa” so then it basically invokes the ko2iblnd-opa modprobe line.  But the script has no logic to detect other types of card (i.e. - mellanox), so in those cases, no ko2iblnd options are used and you end up with the default module parameters being used.
> 
> If you want to use the script, you will need to modify ko2iblnd-probe to add a new case for your brand of HCA and then add an appropriate ko2iblnd-<profile> line to ko2iblnd.conf.
> 
> Or just do what I did and comment out all the lines in ko2iblnd.conf and add your own lines.

If there are significantly different options needed for newer Mellanox HCAs (e.g. as between Qlogic/OPA and MLX) it would be great to get a patch to ko2iblnd-probe and ko2iblnd.conf that adds those options as the default for the new type of card, so that Lustre works better out of the box.  That helps transfer the experience of veteran IB users to users that may not have the background to get the best LNet IB performance.

Cheers, Andreas
--
Andreas Dilger
Lustre Principal Architect
Intel Corporation