[lustre-discuss] ko2iblnd optimizations for EDR
Riccardo.Veraldi at cnaf.infn.it
Thu Nov 8 08:24:28 PST 2018
thanks for sharing your experience.
I am running peer credits value 63 as it was for mlx4.
I did not notice problems with these settings anyway to saturate the EDR
bandwith writing and reading using Lustre I have to do it from 2
so that I can reach up to 9GB/s per OSS. Otherwise from a single client
I cannot reach more than 5GB/s. it is not clear to me why.
On 11/8/18 4:58 AM, Martin Hecht wrote:
> On 11/7/18 9:44 PM, Riccardo Veraldi wrote:
>> Anyway I Was wondering if something different is needed for mlx5 and
>> what are the suggested values in that case ?
>> Anyone has experience with mlx5 LNET performance tunings ?
> Hi Riccardo,
> We have recently integrated mlx5 nodes into our fabric, and we had to
> reduce the values to
> peer_credits = 16
> concurrent_sends = 16
> because mlx5 doesn't support larger values for some reason. The peer_credits must have the same value in all connected lnets, even across routers (at least it used to be like this. I believe we are currently running some Lustre 2.5.x derivates on the server side, and newer versions on the various clients).
> kind regards,
> lustre-discuss mailing list
> lustre-discuss at lists.lustre.org
More information about the lustre-discuss