[lustre-devel] [PATCH 16/33] lnet: improve numeric NID to CPT hashing

[James Simmons]

From: Chris Horn <chris.horn at hpe.com>

Testing shows that the sum-by-multiplication of nid bytes method for
hashing a NID to a CPT does not have good distribution for gnilnd
and kfilnd NIDs. For these LNDs, use the old hash_long() method.

Also modify lustre_lnet_calc_cpt_of_nid() to ensure the specified
number of CPTs is greater than 0. This avoids a divide by zero.

HPE-bug-id: LUS-11576
WC-bug-id: https://jira.whamcloud.com/browse/LU-16797
Lustre-commit: bd12731f2f15b3903 ("LU-16797 lnet: improve numeric NID to CPT hashing")
Signed-off-by: Chris Horn <chris.horn at hpe.com>
Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/50862
Reviewed-by: Andreas Dilger <adilger at whamcloud.com>
Reviewed-by: Serguei Smirnov <ssmirnov at whamcloud.com>
Reviewed-by: Oleg Drokin <green at whamcloud.com>
Signed-off-by: James Simmons <jsimmons at infradead.org>
---
 net/lnet/lnet/api-ni.c | 31 +++++++++++++++++++++----------
 1 file changed, 21 insertions(+), 10 deletions(-)

diff --git a/net/lnet/lnet/api-ni.c b/net/lnet/lnet/api-ni.c
index 585148b4b6aa..fa42cfd99aa7 100644
--- a/net/lnet/lnet/api-ni.c
+++ b/net/lnet/lnet/api-ni.c
@@ -1532,20 +1532,31 @@ static unsigned int
 lnet_nid4_cpt_hash(lnet_nid_t nid, unsigned int number)
 {
 	u64 key = nid;
-	u64 pair_bits = 0x0001000100010001LLU;
-	u64 mask = pair_bits * 0xFF;
-	u64 pair_sum;
+	u16 lnd = LNET_NETTYP(LNET_NIDNET(nid));
+	unsigned int cpt;
 
-	/* Use (sum-by-multiplication of nid bytes) mod (number of CPTs)
-	 * to match nid to a CPT.
-	 */
-	pair_sum = (key & mask) + ((key >> 8) & mask);
-	pair_sum = (pair_sum * pair_bits) >> 48;
+	if (lnd == KFILND || lnd == GNILND) {
+		cpt = hash_long(key, LNET_CPT_BITS);
+
+		/* NB: The number of CPTs needn't be a power of 2 */
+		if (cpt >= number)
+			cpt = (key + cpt + (cpt >> 1)) % number;
+	} else {
+		u64 pair_bits = 0x0001000100010001LLU;
+		u64 mask = pair_bits * 0xFF;
+		u64 pair_sum;
+		/* For ipv4 NIDs, use (sum-by-multiplication of nid bytes) mod
+		 * (number of CPTs) to match nid to a CPT.
+		 */
+		pair_sum = (key & mask) + ((key >> 8) & mask);
+		pair_sum = (pair_sum * pair_bits) >> 48;
+		cpt = (unsigned int)(pair_sum) % number;
+	}
 
 	CDEBUG(D_NET, "Match nid %s to cpt %u\n",
-	       libcfs_nid2str(nid), (unsigned int)(pair_sum) % number);
+	       libcfs_nid2str(nid), cpt);
 
-	return (unsigned int)(pair_sum) % number;
+	return cpt;
 }
 
 unsigned int
-- 
2.39.3