From charlie at whamcloud.com  Thu Nov  1 16:55:06 2018
From: charlie at whamcloud.com (Charlie Olmstead)
Date: Thu, 1 Nov 2018 16:55:06 +0000
Subject: [lustre-devel] lustre-reviews - new optional ARM test session
Message-ID: <E163B855-AF62-4C92-83F2-ADD178E36DC5@ddn.com>

Hello all,

We are adding an optional test session to lustre-reviews builds for testing ARM architecture. The session will run review-ldiskfs with 2 ARM clients. It does not need to pass in order to get a +1 so there’s no reason to retest it unless you want to. This optional session will only run if the Autotest instance has the capacity to run it immediately (it will not queue up and wait for resources).

Let me know if you have any questions,

Charlie Olmstead
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-devel_lists.lustre.org/attachments/20181101/b66f6b0c/attachment.html>

From jsimmons at infradead.org  Thu Nov  1 20:55:37 2018
From: jsimmons at infradead.org (James Simmons)
Date: Thu, 1 Nov 2018 20:55:37 +0000 (GMT)
Subject: [lustre-devel] [PATCH RFC] lustre: obd: convert obd_nid_hash to
 rhashtable
Message-ID: <alpine.LFD.2.21.1811011955300.11470@casper.infradead.org>


This patch converts the struct obd_export obd_nid_hash used by the servr
to use rhltables. The reason is that the NID hash can have multiple obd 
exports using the same NID key. In the process we gain lockless lookup 
which  should improve performance. This should also address the rare 
crashes:

[<ffffffffc0913388>] ? cfs_hash_bd_from_key+0x38/0xb0 [libcfs]
[74844.507416]  [<ffffffffc0913425>] cfs_hash_bd_get+0x25/0x70 [libcfs]
[74844.516384]  [<ffffffffc09166d2>] cfs_hash_add+0x52/0x1a0 [libcfs]
[74844.525211]  [<ffffffffc0d8a765>] target_handle_connect+0x1fe5/0x29b0 [ptlrpc]

Pre 4.8 kernels do not support rhltables so wrappers have been created.

***
My testing has been positive so far but I do need to figure out the 
debugfs file mapping from the old libcfs hash to rhashtables.

***

Signed-off-by: James Simmons <uja.ornl at yahoo.com>
---
 libcfs/autoconf/lustre-libcfs.m4         |  21 +++
 libcfs/include/libcfs/linux/linux-hash.h |  88 ++++++++++++
 lustre/include/lustre_export.h           |   2 +-
 lustre/include/obd.h                     |   8 +-
 lustre/include/obd_support.h             |   3 -
 lustre/ldlm/ldlm_flock.c                 |  18 ++-
 lustre/ldlm/ldlm_lib.c                   |  14 +-
 lustre/mdt/mdt_lproc.c                   |  21 ++-
 lustre/obdclass/genops.c                 |  76 ++++++-----
 lustre/obdclass/lprocfs_status_server.c  | 133 +++++++++++-------
 lustre/obdclass/obd_config.c             | 225 ++++++++++++++++++-------------
 11 files changed, 394 insertions(+), 215 deletions(-)

diff --git a/libcfs/autoconf/lustre-libcfs.m4 b/libcfs/autoconf/lustre-libcfs.m4
index d437331..147ecb3 100644
--- a/libcfs/autoconf/lustre-libcfs.m4
+++ b/libcfs/autoconf/lustre-libcfs.m4
@@ -761,6 +761,26 @@ LB_CHECK_LINUX_HEADER([linux/stringhash.h], [
 ]) # LIBCFS_STRINGHASH
 
 #
+# LIBCFS_RHLTABLE
+# Kernel version 4.8 commit ca26893f05e86497a86732768ec53cd38c0819ca
+# created the rhlist interface to allow inserting duplicate objects
+# into the same table.
+#
+AC_DEFUN([LIBCFS_RHLTABLE], [
+LB_CHECK_COMPILE([if 'struct rhltable' exist],
+rhtable, [
+	#include <linux/rhashtable.h>
+],[
+	struct rhltable *hlt = NULL;
+
+	rhltable_destroy(hlt);
+],[
+	AC_DEFINE(HAVE_RHLTABLE, 1,
+		  [struct rhltable exist])
+])
+]) # LIBCFS_RHLTABLE
+
+#
 # LIBCFS_STACKTRACE_OPS
 #
 # Kernel version 4.8 commit c8fe4609827aedc9c4b45de80e7cdc8ccfa8541b
@@ -999,6 +1019,7 @@ LIBCFS_STACKTRACE_OPS_ADDRESS_RETURN_INT
 LIBCFS_GET_USER_PAGES_6ARG
 LIBCFS_STRINGHASH
 # 4.8
+LIBCFS_RHLTABLE
 LIBCFS_STACKTRACE_OPS
 # 4.9
 LIBCFS_GET_USER_PAGES_GUP_FLAGS
diff --git a/libcfs/include/libcfs/linux/linux-hash.h b/libcfs/include/libcfs/linux/linux-hash.h
index 1227ec8..0453cd9 100644
--- a/libcfs/include/libcfs/linux/linux-hash.h
+++ b/libcfs/include/libcfs/linux/linux-hash.h
@@ -38,6 +38,94 @@ u64 cfs_hashlen_string(const void *salt, const char *name);
 #endif
 #endif /* !HAVE_STRINGHASH */
 
+#ifndef HAVE_RHLTABLE
+struct rhlist_head {
+	struct rhash_head		rhead;
+	struct rhlist_head __rcu	*next;
+};
+
+struct rhltable {
+	struct rhashtable ht;
+};
+
+#define rhl_for_each_entry_rcu(tpos, pos, list, member)                 \
+	for (pos = list; pos && rht_entry(tpos, pos, member);           \
+		pos = rcu_dereference_raw(pos->next))
+
+static inline int rhltable_init(struct rhltable *hlt, const struct rhashtable_params *params)
+{
+	return rhashtable_init(&hlt->ht, params);
+}
+
+static inline struct rhlist_head *rhltable_lookup(
+	struct rhltable *hlt, const void *key,
+	const struct rhashtable_params params)
+{
+	struct rhashtable *ht = &hlt->ht;
+	struct rhashtable_compare_arg arg = {
+		.ht = ht,
+		.key = key,
+	};
+	struct bucket_table *tbl;
+	struct rhash_head *he;
+	unsigned int hash;
+
+	tbl = rht_dereference_rcu(ht->tbl, ht);
+restart:
+	hash = rht_key_hashfn(ht, tbl, key, params);
+	rht_for_each_rcu(he, tbl, hash) {
+		if (params.obj_cmpfn ?
+		    params.obj_cmpfn(&arg, rht_obj(ht, he)) :
+		    rhashtable_compare(&arg, rht_obj(ht, he)))
+			continue;
+		return he ? container_of(he, struct rhlist_head, rhead) : NULL;
+	}
+
+	/* Ensure we see any new tables. */
+	smp_rmb();
+
+	tbl = rht_dereference_rcu(tbl->future_tbl, ht);
+	if (unlikely(tbl))
+		goto restart;
+
+	return NULL;
+}
+
+static inline int rhltable_insert_key(
+	struct rhltable *hlt, const void *key, struct rhlist_head *list,
+	const struct rhashtable_params params)
+{
+	return PTR_ERR(__rhashtable_insert_fast(&hlt->ht, key, &list->rhead,
+						params));
+}
+
+static inline int rhltable_remove(
+	struct rhltable *hlt, struct rhlist_head *list,
+	const struct rhashtable_params params)
+{
+	return rhashtable_remove_fast(&hlt->ht, &list->rhead, params);
+}
+
+static inline void rhltable_free_and_destroy(struct rhltable *hlt,
+					     void (*free_fn)(void *ptr,
+							     void *arg),
+					     void *arg)
+{
+	return rhashtable_free_and_destroy(&hlt->ht, free_fn, arg);
+}
+
+static inline void rhltable_destroy(struct rhltable *hlt)
+{
+	return rhltable_free_and_destroy(hlt, NULL, NULL);
+}
+
+static inline void rhltable_walk_enter(struct rhltable *hlt,
+				       struct rhashtable_iter *iter)
+{
+	rhashtable_walk_init(&hlt->ht, iter);
+}
+#endif /* !HAVE_RHLTABLE */
+
 #ifndef HAVE_RHASHTABLE_LOOKUP_GET_INSERT_FAST
 /**
  * rhashtable_lookup_get_insert_fast - lookup and insert object into hash table
diff --git a/lustre/include/lustre_export.h b/lustre/include/lustre_export.h
index 5ead593..54887c9 100644
--- a/lustre/include/lustre_export.h
+++ b/lustre/include/lustre_export.h
@@ -209,7 +209,7 @@ struct obd_export {
 	/* Unlinked export list */
 	struct list_head	exp_stale_list;
 	struct hlist_node	exp_uuid_hash;	/** uuid-export hash*/
-	struct hlist_node	exp_nid_hash;	/** nid-export hash */
+	struct rhlist_head	exp_nid_hash;	/** nid-export hash */
 	struct hlist_node	exp_gen_hash;   /** last_rcvd clt gen hash */
         /**
          * All exports eligible for ping evictor are linked into a list
diff --git a/lustre/include/obd.h b/lustre/include/obd.h
index 1fcf0a2..8219710 100644
--- a/lustre/include/obd.h
+++ b/lustre/include/obd.h
@@ -639,7 +639,7 @@ struct obd_device {
         /* uuid-export hash body */
 	struct cfs_hash             *obd_uuid_hash;
         /* nid-export hash body */
-	struct cfs_hash             *obd_nid_hash;
+	struct rhltable			obd_nid_hash;
 	/* nid stats body */
 	struct cfs_hash             *obd_nid_stats_hash;
 	/* client_generation-export hash body */
@@ -750,6 +750,12 @@ struct obd_device {
 	struct completion		obd_kobj_unregister;
 };
 
+int obd_nid_export_for_each(struct obd_device *obd, lnet_nid_t nid,
+			    int cb(struct obd_export *exp, void *data),
+			    void *data);
+int obd_nid_add(struct obd_device *obd, struct obd_export *exp);
+void obd_nid_del(struct obd_device *obd, struct obd_export *exp);
+
 /* get/set_info keys */
 #define KEY_ASYNC               "async"
 #define KEY_CHANGELOG_CLEAR     "changelog_clear"
diff --git a/lustre/include/obd_support.h b/lustre/include/obd_support.h
index e9dd33e..0175cf8 100644
--- a/lustre/include/obd_support.h
+++ b/lustre/include/obd_support.h
@@ -78,9 +78,6 @@ extern char obd_jobid_var[];
 #define HASH_UUID_BKT_BITS 5
 #define HASH_UUID_CUR_BITS 7
 #define HASH_UUID_MAX_BITS 12
-#define HASH_NID_BKT_BITS 5
-#define HASH_NID_CUR_BITS 7
-#define HASH_NID_MAX_BITS 12
 #define HASH_NID_STATS_BKT_BITS 5
 #define HASH_NID_STATS_CUR_BITS 7
 #define HASH_NID_STATS_MAX_BITS 12
diff --git a/lustre/ldlm/ldlm_flock.c b/lustre/ldlm/ldlm_flock.c
index f848a36..4c1603a 100644
--- a/lustre/ldlm/ldlm_flock.c
+++ b/lustre/ldlm/ldlm_flock.c
@@ -161,20 +161,18 @@ ldlm_flock_destroy(struct ldlm_lock *lock, enum ldlm_mode mode, __u64 flags)
  */
 
 struct ldlm_flock_lookup_cb_data {
-	__u64 *bl_owner;
+	u64 *bl_owner;
 	struct ldlm_lock *lock;
 	struct obd_export *exp;
 };
 
-static int ldlm_flock_lookup_cb(struct cfs_hash *hs, struct cfs_hash_bd *bd,
-				struct hlist_node *hnode, void *data)
+static int ldlm_flock_lookup_cb(struct obd_export *exp, void *data)
 {
 	struct ldlm_flock_lookup_cb_data *cb_data = data;
-	struct obd_export *exp = cfs_hash_object(hs, hnode);
 	struct ldlm_lock *lock;
 
 	lock = cfs_hash_lookup(exp->exp_flock_hash, cb_data->bl_owner);
-	if (lock == NULL)
+	if (!lock)
 		return 0;
 
 	/* Stop on first found lock. Same process can't sleep twice */
@@ -206,13 +204,13 @@ ldlm_flock_deadlock(struct ldlm_lock *req, struct ldlm_lock *bl_lock)
 		struct ldlm_lock *lock = NULL;
 		struct ldlm_flock *flock;
 
-		if (bl_exp->exp_flock_hash != NULL) {
-			cfs_hash_for_each_key(bl_exp->exp_obd->obd_nid_hash,
-				&bl_exp->exp_connection->c_peer.nid,
-				ldlm_flock_lookup_cb, &cb_data);
+		if (bl_exp->exp_flock_hash) {
+			obd_nid_export_for_each(bl_exp->exp_obd,
+						bl_exp->exp_connection->c_peer.nid,
+						ldlm_flock_lookup_cb, &cb_data);
 			lock = cb_data.lock;
 		}
-		if (lock == NULL)
+		if (!lock)
 			break;
 
 		class_export_put(bl_exp);
diff --git a/lustre/ldlm/ldlm_lib.c b/lustre/ldlm/ldlm_lib.c
index de66aa3..b62e785 100644
--- a/lustre/ldlm/ldlm_lib.c
+++ b/lustre/ldlm/ldlm_lib.c
@@ -1337,11 +1337,9 @@ dont_check_exports:
 
 	if (export->exp_connection != NULL) {
 		/* Check to see if connection came from another NID. */
-		if ((export->exp_connection->c_peer.nid != req->rq_peer.nid) &&
-		    !hlist_unhashed(&export->exp_nid_hash))
-			cfs_hash_del(export->exp_obd->obd_nid_hash,
-				     &export->exp_connection->c_peer.nid,
-				     &export->exp_nid_hash);
+		if (export->exp_connection->c_peer.nid != req->rq_peer.nid &&
+		    export != export->exp_obd->obd_self_export)
+			obd_nid_del(export->exp_obd, export);
 
 		ptlrpc_connection_put(export->exp_connection);
 	}
@@ -1349,10 +1347,8 @@ dont_check_exports:
 	export->exp_connection = ptlrpc_connection_get(req->rq_peer,
 						       req->rq_self,
 						       &cluuid);
-	if (hlist_unhashed(&export->exp_nid_hash))
-		cfs_hash_add(export->exp_obd->obd_nid_hash,
-			     &export->exp_connection->c_peer.nid,
-			     &export->exp_nid_hash);
+	if (export != export->exp_obd->obd_self_export)
+		obd_nid_add(export->exp_obd, export);
 
 	lustre_msg_set_handle(req->rq_repmsg, &conn);
 
diff --git a/lustre/mdt/mdt_lproc.c b/lustre/mdt/mdt_lproc.c
index b41eeee..61b11ad 100644
--- a/lustre/mdt/mdt_lproc.c
+++ b/lustre/mdt/mdt_lproc.c
@@ -1009,19 +1009,17 @@ static struct lprocfs_vars lprocfs_mdt_obd_vars[] = {
 };
 
 static int
-lprocfs_mdt_print_open_files(struct cfs_hash *hs, struct cfs_hash_bd *bd,
-			     struct hlist_node *hnode, void *v)
+lprocfs_mdt_print_open_files(struct obd_export *exp, void *v)
 {
-	struct obd_export	*exp = cfs_hash_object(hs, hnode);
-	struct seq_file		*seq = v;
+	struct seq_file	*seq = v;
 
-	if (exp->exp_lock_hash != NULL) {
-		struct mdt_export_data  *med = &exp->exp_mdt_data;
-		struct mdt_file_data	*mfd;
+	if (exp->exp_lock_hash) {
+		struct mdt_export_data *med = &exp->exp_mdt_data;
+		struct mdt_file_data *mfd;
 
 		spin_lock(&med->med_open_lock);
 		list_for_each_entry(mfd, &med->med_open_head, mfd_list) {
-			seq_printf(seq, DFID"\n",
+			seq_printf(seq, DFID "\n",
 				   PFID(mdt_object_fid(mfd->mfd_object)));
 		}
 		spin_unlock(&med->med_open_lock);
@@ -1033,12 +1031,9 @@ lprocfs_mdt_print_open_files(struct cfs_hash *hs, struct cfs_hash_bd *bd,
 static int lprocfs_mdt_open_files_seq_show(struct seq_file *seq, void *v)
 {
 	struct nid_stat *stats = seq->private;
-	struct obd_device *obd = stats->nid_obd;
 
-	cfs_hash_for_each_key(obd->obd_nid_hash, &stats->nid,
-			      lprocfs_mdt_print_open_files, seq);
-
-	return 0;
+	return obd_nid_export_for_each(stats->nid_obd, stats->nid,
+				       lprocfs_mdt_print_open_files, seq);
 }
 
 int lprocfs_mdt_open_files_seq_open(struct inode *inode, struct file *file)
diff --git a/lustre/obdclass/genops.c b/lustre/obdclass/genops.c
index 84a8ce1..d7a65ec 100644
--- a/lustre/obdclass/genops.c
+++ b/lustre/obdclass/genops.c
@@ -1114,7 +1114,6 @@ struct obd_export *__class_new_export(struct obd_device *obd,
 	spin_lock_init(&export->exp_lock);
 	spin_lock_init(&export->exp_rpc_lock);
 	INIT_HLIST_NODE(&export->exp_uuid_hash);
-	INIT_HLIST_NODE(&export->exp_nid_hash);
 	INIT_HLIST_NODE(&export->exp_gen_hash);
 	spin_lock_init(&export->exp_bl_list_lock);
 	INIT_LIST_HEAD(&export->exp_bl_list);
@@ -1518,20 +1517,17 @@ int class_disconnect(struct obd_export *export)
 	export->exp_disconnected = 1;
 	/*  We hold references of export for uuid hash
 	 *  and nid_hash and export link at least. So
-	 *  it is safe to call cfs_hash_del in there.  */
-	if (!hlist_unhashed(&export->exp_nid_hash))
-		cfs_hash_del(export->exp_obd->obd_nid_hash,
-			     &export->exp_connection->c_peer.nid,
-			     &export->exp_nid_hash);
+	 *  it is safe to call cfs_hash_del in there.
+	 */
+	if (export != export->exp_obd->obd_self_export)
+		obd_nid_del(export->exp_obd, export);
 	spin_unlock(&export->exp_lock);
 
         /* class_cleanup(), abort_recovery(), and class_fail_export()
          * all end up in here, and if any of them race we shouldn't
          * call extra class_export_puts(). */
-        if (already_disconnected) {
-		LASSERT(hlist_unhashed(&export->exp_nid_hash));
+	if (already_disconnected)
                 GOTO(no_disconn, already_disconnected);
-        }
 
 	CDEBUG(D_IOCTL, "disconnect: cookie %#llx\n",
                export->exp_handle.h_cookie);
@@ -1719,11 +1715,10 @@ EXPORT_SYMBOL(class_fail_export);
 
 int obd_export_evict_by_nid(struct obd_device *obd, const char *nid)
 {
-	struct cfs_hash *nid_hash;
-	struct obd_export *doomed_exp = NULL;
-	int exports_evicted = 0;
-
 	lnet_nid_t nid_key = libcfs_str2nid((char *)nid);
+	struct obd_export *doomed_exp;
+	struct rhashtable_iter iter;
+	int exports_evicted = 0;
 
 	spin_lock(&obd->obd_dev_lock);
 	/* umount has run already, so evict thread should leave
@@ -1732,36 +1727,45 @@ int obd_export_evict_by_nid(struct obd_device *obd, const char *nid)
 		spin_unlock(&obd->obd_dev_lock);
 		return exports_evicted;
 	}
-	nid_hash = obd->obd_nid_hash;
-	cfs_hash_getref(nid_hash);
 	spin_unlock(&obd->obd_dev_lock);
 
-	do {
-		doomed_exp = cfs_hash_lookup(nid_hash, &nid_key);
-                if (doomed_exp == NULL)
-                        break;
+	rhltable_walk_enter(&obd->obd_nid_hash, &iter);
+	rhashtable_walk_start(&iter);
+	while ((doomed_exp = rhashtable_walk_next(&iter)) != NULL) {
+		if (IS_ERR(doomed_exp))
+			continue;
 
-                LASSERTF(doomed_exp->exp_connection->c_peer.nid == nid_key,
-                         "nid %s found, wanted nid %s, requested nid %s\n",
-                         obd_export_nid2str(doomed_exp),
-                         libcfs_nid2str(nid_key), nid);
-                LASSERTF(doomed_exp != obd->obd_self_export,
-                         "self-export is hashed by NID?\n");
-                exports_evicted++;
-		LCONSOLE_WARN("%s: evicting %s (at %s) by administrative "
-			      "request\n", obd->obd_name,
+		if (!doomed_exp->exp_connection ||
+		    doomed_exp->exp_connection->c_peer.nid != nid_key)
+			continue;
+
+		rhashtable_walk_stop(&iter);
+
+		LASSERTF(doomed_exp != obd->obd_self_export,
+			 "self-export is hashed by NID?\n");
+
+		LCONSOLE_WARN("%s: evicting %s (at %s) by administrative request\n",
+			      obd->obd_name,
 			      obd_uuid2str(&doomed_exp->exp_client_uuid),
 			      obd_export_nid2str(doomed_exp));
-                class_fail_export(doomed_exp);
-                class_export_put(doomed_exp);
-        } while (1);
 
-	cfs_hash_putref(nid_hash);
+		class_fail_export(doomed_exp);
+		class_export_put(doomed_exp);
+		obd_nid_del(obd, doomed_exp);
+		exports_evicted++;
 
-        if (!exports_evicted)
-                CDEBUG(D_HA,"%s: can't disconnect NID '%s': no exports found\n",
-                       obd->obd_name, nid);
-        return exports_evicted;
+		rhashtable_walk_start(&iter);
+	}
+
+	rhashtable_walk_stop(&iter);
+	rhashtable_walk_exit(&iter);
+
+	if (!exports_evicted) {
+		CDEBUG(D_HA,
+		      "%s: can't disconnect NID '%s': no exports found\n",
+		       obd->obd_name, nid);
+	}
+	return exports_evicted;
 }
 EXPORT_SYMBOL(obd_export_evict_by_nid);
 
diff --git a/lustre/obdclass/lprocfs_status_server.c b/lustre/obdclass/lprocfs_status_server.c
index f878de0..38f74b7 100644
--- a/lustre/obdclass/lprocfs_status_server.c
+++ b/lustre/obdclass/lprocfs_status_server.c
@@ -172,12 +172,10 @@ static int obd_export_flags2str(struct obd_export *exp, struct seq_file *m)
 }
 
 static int
-lprocfs_exp_print_export_seq(struct cfs_hash *hs, struct cfs_hash_bd *bd,
-			     struct hlist_node *hnode, void *cb_data)
+lprocfs_exp_print_export_seq(struct obd_export *exp, void *cb_data)
 {
-	struct seq_file		*m = cb_data;
-	struct obd_export	*exp = cfs_hash_object(hs, hnode);
-	struct obd_device	*obd;
+	struct seq_file	*m = cb_data;
+	struct obd_device *obd;
 	struct obd_connect_data	*ocd;
 
 	LASSERT(exp != NULL);
@@ -231,11 +229,9 @@ out:
 static int lprocfs_exp_export_seq_show(struct seq_file *m, void *data)
 {
 	struct nid_stat *stats = m->private;
-	struct obd_device *obd = stats->nid_obd;
 
-	cfs_hash_for_each_key(obd->obd_nid_hash, &stats->nid,
-			      lprocfs_exp_print_export_seq, m);
-	return 0;
+	return obd_nid_export_for_each(stats->nid_obd, stats->nid,
+				       lprocfs_exp_print_export_seq, m);
 }
 LPROC_SEQ_FOPS_RO(lprocfs_exp_export);
 
@@ -281,64 +277,103 @@ void lprocfs_free_per_client_stats(struct obd_device *obd)
 EXPORT_SYMBOL(lprocfs_free_per_client_stats);
 
 static int
-lprocfs_exp_print_uuid_seq(struct cfs_hash *hs, struct cfs_hash_bd *bd,
-			   struct hlist_node *hnode, void *cb_data)
+lprocfs_exp_print_nodemap_seq(struct obd_export *exp, void *cb_data)
 {
+	struct lu_nodemap *nodemap = exp->exp_target_data.ted_nodemap;
 	struct seq_file *m = cb_data;
-	struct obd_export *exp = cfs_hash_object(hs, hnode);
 
-	if (exp->exp_nid_stats != NULL)
-		seq_printf(m, "%s\n", obd_uuid2str(&exp->exp_client_uuid));
+	if (nodemap)
+		seq_printf(m, "%s\n", nodemap->nm_name);
 	return 0;
 }
 
 static int
-lprocfs_exp_print_nodemap_seq(struct cfs_hash *hs, struct cfs_hash_bd *bd,
-			      struct hlist_node *hnode, void *cb_data)
+lprocfs_exp_nodemap_seq_show(struct seq_file *m, void *data)
 {
-	struct seq_file *m = cb_data;
-	struct obd_export *exp = cfs_hash_object(hs, hnode);
-	struct lu_nodemap *nodemap = exp->exp_target_data.ted_nodemap;
+	struct nid_stat *stats = m->private;
 
-	if (nodemap != NULL)
-		seq_printf(m, "%s\n", nodemap->nm_name);
-	return 0;
+	return obd_nid_export_for_each(stats->nid_obd, stats->nid,
+				       lprocfs_exp_print_nodemap_seq, m);
 }
+LPROC_SEQ_FOPS_RO(lprocfs_exp_nodemap);
 
 static int
-lprocfs_exp_nodemap_seq_show(struct seq_file *m, void *data)
+lprocfs_exp_print_uuid_seq(struct obd_export *exp, void *cb_data)
 {
-	struct nid_stat *stats = m->private;
-	struct obd_device *obd = stats->nid_obd;
+	struct seq_file *m = cb_data;
 
-	cfs_hash_for_each_key(obd->obd_nid_hash, &stats->nid,
-			      lprocfs_exp_print_nodemap_seq, m);
+	if (exp->exp_nid_stats)
+		seq_printf(m, "%s\n", obd_uuid2str(&exp->exp_client_uuid));
 	return 0;
 }
-LPROC_SEQ_FOPS_RO(lprocfs_exp_nodemap);
 
 static int lprocfs_exp_uuid_seq_show(struct seq_file *m, void *data)
 {
 	struct nid_stat *stats = m->private;
-	struct obd_device *obd = stats->nid_obd;
 
-	cfs_hash_for_each_key(obd->obd_nid_hash, &stats->nid,
-				lprocfs_exp_print_uuid_seq, m);
-	return 0;
+	return obd_nid_export_for_each(stats->nid_obd, stats->nid,
+				       lprocfs_exp_print_uuid_seq, m);
 }
 LPROC_SEQ_FOPS_RO(lprocfs_exp_uuid);
 
+static void ldebugfs_rhash_seq_show(const char *name, struct rhashtable *ht,
+				    struct seq_file *m)
+{
+/*	int dist[8] = { 0, };
+	int maxdep = -1;
+	int maxdepb = -1;
+	int total = 0;
+	int theta;
+	int i;
+
+	theta = __cfs_hash_theta(hs);
+
+*/
+	seq_printf(m, "%-*s %5d %5d %5d\n",
+		   CFS_HASH_BIGNAME_LEN, name, atomic_read(&ht->nelems),
+		   ht->p.min_size, ht->p.max_size);
+/*
+	seq_printf(m, "%-*s %5d %5d %5d %d.%03d %d.%03d %d.%03d  0x%02x %6d ",
+		   CFS_HASH_BIGNAME_LEN, hs->hs_name,
+		   1 << hs->hs_cur_bits, 1 << hs->hs_min_bits,
+		   1 << hs->hs_max_bits,
+		   __cfs_hash_theta_int(theta), __cfs_hash_theta_frac(theta),
+		   __cfs_hash_theta_int(hs->hs_min_theta),
+		   __cfs_hash_theta_frac(hs->hs_min_theta),
+		   __cfs_hash_theta_int(hs->hs_max_theta),
+		   __cfs_hash_theta_frac(hs->hs_max_theta),
+		   hs->hs_flags, hs->hs_rehash_count);
+
+	for (i = 0; i < cfs_hash_full_nbkt(hs); i++) {
+		struct cfs_hash_bd bd;
+
+		bd.bd_bucket = cfs_hash_full_bkts(hs)[i];
+		cfs_hash_bd_lock(hs, &bd, 0);
+		if (maxdep < bd.bd_bucket->hsb_depmax) {
+			maxdep  = bd.bd_bucket->hsb_depmax;
+			maxdepb = ffz(~maxdep);
+		}
+		total += bd.bd_bucket->hsb_count;
+		dist[min(fls(bd.bd_bucket->hsb_count / max(theta, 1)), 7)]++;
+		cfs_hash_bd_unlock(hs, &bd, 0);
+	}
+
+	seq_printf(m, "%7d %7d %7d ", total, maxdep, maxdepb);
+	for (i = 0; i < 8; i++)
+		seq_printf(m, "%d%c",  dist[i], (i == 7) ? '\n' : '/');
+*/
+}
+
 static int
-lprocfs_exp_print_hash_seq(struct cfs_hash *hs, struct cfs_hash_bd *bd,
-			   struct hlist_node *hnode, void *cb_data)
+lprocfs_exp_print_hash_seq(struct obd_export *exp, void *cb_data)
 
 {
+	struct obd_device *obd = exp->exp_obd;
 	struct seq_file *m = cb_data;
-	struct obd_export *exp = cfs_hash_object(hs, hnode);
 
-	if (exp->exp_lock_hash != NULL) {
+	if (exp->exp_lock_hash) {
 		cfs_hash_debug_header(m);
-		cfs_hash_debug_str(hs, m);
+		ldebugfs_rhash_seq_show("NID_HASH", &obd->obd_nid_hash.ht, m);
 	}
 	return 0;
 }
@@ -346,26 +381,22 @@ lprocfs_exp_print_hash_seq(struct cfs_hash *hs, struct cfs_hash_bd *bd,
 static int lprocfs_exp_hash_seq_show(struct seq_file *m, void *data)
 {
 	struct nid_stat *stats = m->private;
-	struct obd_device *obd = stats->nid_obd;
 
-	cfs_hash_for_each_key(obd->obd_nid_hash, &stats->nid,
-				lprocfs_exp_print_hash_seq, m);
-	return 0;
+	return obd_nid_export_for_each(stats->nid_obd, stats->nid,
+				       lprocfs_exp_print_hash_seq, m);
 }
 LPROC_SEQ_FOPS_RO(lprocfs_exp_hash);
 
-int lprocfs_exp_print_replydata_seq(struct cfs_hash *hs, struct cfs_hash_bd *bd,
-				    struct hlist_node *hnode, void *cb_data)
+int lprocfs_exp_print_replydata_seq(struct obd_export *exp, void *cb_data)
 
 {
-	struct obd_export *exp = cfs_hash_object(hs, hnode);
 	struct seq_file *m = cb_data;
 	struct tg_export_data *ted = &exp->exp_target_data;
 
 	seq_printf(m, "reply_cnt: %d\n"
-		   "reply_max: %d\n"
-		   "reply_released_by_xid: %d\n"
-		   "reply_released_by_tag: %d\n\n",
+		      "reply_max: %d\n"
+		      "reply_released_by_xid: %d\n"
+		      "reply_released_by_tag: %d\n\n",
 		   ted->ted_reply_cnt,
 		   ted->ted_reply_max,
 		   ted->ted_release_xid,
@@ -376,11 +407,9 @@ int lprocfs_exp_print_replydata_seq(struct cfs_hash *hs, struct cfs_hash_bd *bd,
 int lprocfs_exp_replydata_seq_show(struct seq_file *m, void *data)
 {
 	struct nid_stat *stats = m->private;
-	struct obd_device *obd = stats->nid_obd;
 
-	cfs_hash_for_each_key(obd->obd_nid_hash, &stats->nid,
-				lprocfs_exp_print_replydata_seq, m);
-	return 0;
+	return obd_nid_export_for_each(stats->nid_obd, stats->nid,
+				       lprocfs_exp_print_replydata_seq, m);
 }
 LPROC_SEQ_FOPS_RO(lprocfs_exp_replydata);
 
@@ -624,7 +653,7 @@ int lprocfs_hash_seq_show(struct seq_file *m, void *data)
 
 	cfs_hash_debug_header(m);
 	cfs_hash_debug_str(obd->obd_uuid_hash, m);
-	cfs_hash_debug_str(obd->obd_nid_hash, m);
+	ldebugfs_rhash_seq_show("NID_HASH", &obd->obd_nid_hash.ht, m);
 	cfs_hash_debug_str(obd->obd_nid_stats_hash, m);
 	return 0;
 }
diff --git a/lustre/obdclass/obd_config.c b/lustre/obdclass/obd_config.c
index c4a20e6..448e7d8 100644
--- a/lustre/obdclass/obd_config.c
+++ b/lustre/obdclass/obd_config.c
@@ -50,10 +50,132 @@
 #include "llog_internal.h"
 
 static struct cfs_hash_ops uuid_hash_ops;
-static struct cfs_hash_ops nid_hash_ops;
 static struct cfs_hash_ops nid_stat_hash_ops;
 static struct cfs_hash_ops gen_hash_ops;
 
+/*
+ * nid<->export hash operations
+ */
+static u32 nid_keyhash(const void *data, u32 key_len, u32 seed)
+{
+	const struct obd_export *exp = data;
+	void *key;
+
+	if (!exp->exp_connection)
+		return 0;
+
+	key = &exp->exp_connection->c_peer.nid;
+	return jhash2(key, key_len / sizeof(u32), seed);
+}
+
+/*
+ * NOTE: It is impossible to find an export that is in failed
+ *       state with this function
+ */
+static int
+nid_keycmp(struct rhashtable_compare_arg *arg, const void *obj)
+{
+	const lnet_nid_t *nid = arg->key;
+	const struct obd_export *exp = obj;
+
+	if (exp->exp_connection->c_peer.nid == *nid && !exp->exp_failed)
+		return 0;
+
+	return -ESRCH;
+}
+
+static void
+nid_export_exit(void *vexport, void *data)
+{
+	struct obd_export *exp = vexport;
+
+	class_export_put(exp);
+}
+
+const struct rhashtable_params nid_hash_params = {
+	.key_len		= sizeof(lnet_nid_t),
+	.head_offset		= offsetof(struct obd_export, exp_nid_hash),
+	.obj_hashfn		= nid_keyhash,
+	.obj_cmpfn		= nid_keycmp,
+	.min_size		= 128,
+	.max_size		= 4096,
+	.automatic_shrinking	= true,
+};
+
+int obd_nid_add(struct obd_device *obd, struct obd_export *exp)
+{
+	struct rhlist_head *exp_list;
+	int rc;
+
+	rcu_read_lock();
+	exp_list = rhltable_lookup(&obd->obd_nid_hash,
+				   &exp->exp_connection->c_peer.nid,
+				   nid_hash_params);
+	if (exp_list) {
+		struct rhlist_head *pos;
+		struct obd_export *tmp;
+
+		rhl_for_each_entry_rcu(tmp, pos, exp_list, exp_nid_hash) {
+			if (tmp == exp) {
+				rcu_read_unlock();
+				return -EALREADY;
+			}
+		}
+	}
+	rcu_read_unlock();
+
+	rc = rhltable_insert_key(&obd->obd_nid_hash,
+				 &exp->exp_connection->c_peer.nid,
+				 &exp->exp_nid_hash,
+				 nid_hash_params);
+	if (rc == 0)
+		class_export_get(exp);
+	else
+		/* map obscure error codes to -ENOMEM */
+		rc = -ENOMEM;
+	return rc;
+}
+EXPORT_SYMBOL(obd_nid_add);
+
+void obd_nid_del(struct obd_device *obd, struct obd_export *exp)
+{
+	int rc;
+
+	rc = rhltable_remove(&obd->obd_nid_hash, &exp->exp_nid_hash,
+			     nid_hash_params);
+	if (rc == 0)
+		class_export_put(exp);
+}
+EXPORT_SYMBOL(obd_nid_del);
+
+int obd_nid_export_for_each(struct obd_device *obd, lnet_nid_t nid,
+			    int cb(struct obd_export *exp, void *data),
+			    void *data)
+{
+	struct rhlist_head *exports, *tmp;
+	struct obd_export *exp;
+	int err_cnt = 0;
+
+	rcu_read_lock();
+	exports = rhltable_lookup(&obd->obd_nid_hash, &nid, nid_hash_params);
+	if (!exports) {
+		err_cnt = -ENODEV;
+		goto out_unlock;
+	}
+
+	rhl_for_each_entry_rcu(exp,  tmp, exports, exp_nid_hash) {
+		if (cb(exp, data)) {
+			err_cnt++;
+			continue;
+		}
+	}
+
+out_unlock:
+	rcu_read_unlock();
+	return err_cnt;
+}
+EXPORT_SYMBOL(obd_nid_export_for_each);
+
 /*********** string parsing utils *********/
 
 /* returns 0 if we find this key in the buffer, else 1 */
@@ -474,7 +596,6 @@ int class_setup(struct obd_device *obd, struct lustre_cfg *lcfg)
            other fns check that status, and we're not actually set up yet. */
         obd->obd_starting = 1;
         obd->obd_uuid_hash = NULL;
-        obd->obd_nid_hash = NULL;
         obd->obd_nid_stats_hash = NULL;
 	obd->obd_gen_hash = NULL;
 	spin_unlock(&obd->obd_dev_lock);
@@ -490,16 +611,10 @@ int class_setup(struct obd_device *obd, struct lustre_cfg *lcfg)
         if (!obd->obd_uuid_hash)
 		GOTO(err_exit, err = -ENOMEM);
 
-        /* create a nid-export lustre hash */
-        obd->obd_nid_hash = cfs_hash_create("NID_HASH",
-                                            HASH_NID_CUR_BITS,
-                                            HASH_NID_MAX_BITS,
-                                            HASH_NID_BKT_BITS, 0,
-                                            CFS_HASH_MIN_THETA,
-                                            CFS_HASH_MAX_THETA,
-                                            &nid_hash_ops, CFS_HASH_DEFAULT);
-        if (!obd->obd_nid_hash)
-		GOTO(err_exit, err = -ENOMEM);
+	/* create a nid-export lustre hash */
+	err = rhltable_init(&obd->obd_nid_hash, &nid_hash_params);
+	if (err)
+		GOTO(err_exit, err);
 
         /* create a nid-stats lustre hash */
         obd->obd_nid_stats_hash = cfs_hash_create("NID_STATS",
@@ -543,10 +658,9 @@ err_exit:
                 cfs_hash_putref(obd->obd_uuid_hash);
                 obd->obd_uuid_hash = NULL;
         }
-        if (obd->obd_nid_hash) {
-                cfs_hash_putref(obd->obd_nid_hash);
-                obd->obd_nid_hash = NULL;
-        }
+
+	rhltable_destroy(&obd->obd_nid_hash);
+
         if (obd->obd_nid_stats_hash) {
                 cfs_hash_putref(obd->obd_nid_stats_hash);
                 obd->obd_nid_stats_hash = NULL;
@@ -673,10 +787,7 @@ int class_cleanup(struct obd_device *obd, struct lustre_cfg *lcfg)
 	}
 
 	/* destroy a nid-export hash body */
-	if (obd->obd_nid_hash) {
-		cfs_hash_putref(obd->obd_nid_hash);
-		obd->obd_nid_hash = NULL;
-	}
+	rhltable_free_and_destroy(&obd->obd_nid_hash, nid_export_exit, NULL);
 
 	/* destroy a nid-stats hash body */
 	if (obd->obd_nid_stats_hash) {
@@ -2220,81 +2331,15 @@ static struct cfs_hash_ops uuid_hash_ops = {
         .hs_put_locked  = uuid_export_put_locked,
 };
 
-
 /*
- * nid<->export hash operations
+ * nid<->nidstats hash operations
  */
-
 static unsigned
-nid_hash(struct cfs_hash *hs, const void *key, unsigned mask)
-{
-        return cfs_hash_djb2_hash(key, sizeof(lnet_nid_t), mask);
-}
-
-static void *
-nid_key(struct hlist_node *hnode)
-{
-        struct obd_export *exp;
-
-	exp = hlist_entry(hnode, struct obd_export, exp_nid_hash);
-
-        RETURN(&exp->exp_connection->c_peer.nid);
-}
-
-/*
- * NOTE: It is impossible to find an export that is in failed
- *       state with this function
- */
-static int
-nid_kepcmp(const void *key, struct hlist_node *hnode)
-{
-        struct obd_export *exp;
-
-        LASSERT(key);
-	exp = hlist_entry(hnode, struct obd_export, exp_nid_hash);
-
-        RETURN(exp->exp_connection->c_peer.nid == *(lnet_nid_t *)key &&
-               !exp->exp_failed);
-}
-
-static void *
-nid_export_object(struct hlist_node *hnode)
-{
-	return hlist_entry(hnode, struct obd_export, exp_nid_hash);
-}
-
-static void
-nid_export_get(struct cfs_hash *hs, struct hlist_node *hnode)
+nidstats_hash(struct cfs_hash *hs, const void *key, unsigned int mask)
 {
-        struct obd_export *exp;
-
-	exp = hlist_entry(hnode, struct obd_export, exp_nid_hash);
-        class_export_get(exp);
+	return cfs_hash_djb2_hash(key, sizeof(lnet_nid_t), mask);
 }
 
-static void
-nid_export_put_locked(struct cfs_hash *hs, struct hlist_node *hnode)
-{
-        struct obd_export *exp;
-
-	exp = hlist_entry(hnode, struct obd_export, exp_nid_hash);
-        class_export_put(exp);
-}
-
-static struct cfs_hash_ops nid_hash_ops = {
-        .hs_hash        = nid_hash,
-        .hs_key         = nid_key,
-        .hs_keycmp      = nid_kepcmp,
-        .hs_object      = nid_export_object,
-        .hs_get         = nid_export_get,
-        .hs_put_locked  = nid_export_put_locked,
-};
-
-
-/*
- * nid<->nidstats hash operations
- */
-
 static void *
 nidstats_key(struct hlist_node *hnode)
 {
@@ -2336,7 +2381,7 @@ nidstats_put_locked(struct cfs_hash *hs, struct hlist_node *hnode)
 }
 
 static struct cfs_hash_ops nid_stat_hash_ops = {
-        .hs_hash        = nid_hash,
+	.hs_hash	= nidstats_hash,
         .hs_key         = nidstats_key,
         .hs_keycmp      = nidstats_keycmp,
         .hs_object      = nidstats_object,
-- 
1.8.3.1


From neilb at suse.com  Fri Nov  2 02:36:43 2018
From: neilb at suse.com (NeilBrown)
Date: Fri, 02 Nov 2018 13:36:43 +1100
Subject: [lustre-devel] [PATCH RFC] lustre: obd: convert obd_nid_hash to
 rhashtable
In-Reply-To: <alpine.LFD.2.21.1811011955300.11470@casper.infradead.org>
References: <alpine.LFD.2.21.1811011955300.11470@casper.infradead.org>
Message-ID: <87tvl019z8.fsf@notabene.neil.brown.name>

On Thu, Nov 01 2018, James Simmons wrote:

> This patch converts the struct obd_export obd_nid_hash used by the servr
> to use rhltables. The reason is that the NID hash can have multiple obd 
> exports using the same NID key. In the process we gain lockless lookup 
> which  should improve performance. This should also address the rare 
> crashes:
>
> [<ffffffffc0913388>] ? cfs_hash_bd_from_key+0x38/0xb0 [libcfs]
> [74844.507416]  [<ffffffffc0913425>] cfs_hash_bd_get+0x25/0x70 [libcfs]
> [74844.516384]  [<ffffffffc09166d2>] cfs_hash_add+0x52/0x1a0 [libcfs]
> [74844.525211]  [<ffffffffc0d8a765>] target_handle_connect+0x1fe5/0x29b0 [ptlrpc]
>
> Pre 4.8 kernels do not support rhltables so wrappers have been created.
>
> ***
> My testing has been positive so far but I do need to figure out the 
> debugfs file mapping from the old libcfs hash to rhashtables.
>
> ***
>
> Signed-off-by: James Simmons <uja.ornl at yahoo.com>
> ---
>  libcfs/autoconf/lustre-libcfs.m4         |  21 +++
>  libcfs/include/libcfs/linux/linux-hash.h |  88 ++++++++++++
>  lustre/include/lustre_export.h           |   2 +-
>  lustre/include/obd.h                     |   8 +-
>  lustre/include/obd_support.h             |   3 -
>  lustre/ldlm/ldlm_flock.c                 |  18 ++-
>  lustre/ldlm/ldlm_lib.c                   |  14 +-
>  lustre/mdt/mdt_lproc.c                   |  21 ++-
>  lustre/obdclass/genops.c                 |  76 ++++++-----
>  lustre/obdclass/lprocfs_status_server.c  | 133 +++++++++++-------
>  lustre/obdclass/obd_config.c             | 225 ++++++++++++++++++-------------
>  11 files changed, 394 insertions(+), 215 deletions(-)
>
> diff --git a/libcfs/autoconf/lustre-libcfs.m4 b/libcfs/autoconf/lustre-libcfs.m4
> index d437331..147ecb3 100644
> --- a/libcfs/autoconf/lustre-libcfs.m4
> +++ b/libcfs/autoconf/lustre-libcfs.m4
> @@ -761,6 +761,26 @@ LB_CHECK_LINUX_HEADER([linux/stringhash.h], [
>  ]) # LIBCFS_STRINGHASH
>  
>  #
> +# LIBCFS_RHLTABLE
> +# Kernel version 4.8 commit ca26893f05e86497a86732768ec53cd38c0819ca
> +# created the rhlist interface to allow inserting duplicate objects
> +# into the same table.
> +#
> +AC_DEFUN([LIBCFS_RHLTABLE], [
> +LB_CHECK_COMPILE([if 'struct rhltable' exist],
> +rhtable, [
> +	#include <linux/rhashtable.h>
> +],[
> +	struct rhltable *hlt = NULL;
> +
> +	rhltable_destroy(hlt);
> +],[
> +	AC_DEFINE(HAVE_RHLTABLE, 1,
> +		  [struct rhltable exist])
> +])
> +]) # LIBCFS_RHLTABLE
> +
> +#
>  # LIBCFS_STACKTRACE_OPS
>  #
>  # Kernel version 4.8 commit c8fe4609827aedc9c4b45de80e7cdc8ccfa8541b
> @@ -999,6 +1019,7 @@ LIBCFS_STACKTRACE_OPS_ADDRESS_RETURN_INT
>  LIBCFS_GET_USER_PAGES_6ARG
>  LIBCFS_STRINGHASH
>  # 4.8
> +LIBCFS_RHLTABLE
>  LIBCFS_STACKTRACE_OPS
>  # 4.9
>  LIBCFS_GET_USER_PAGES_GUP_FLAGS
> diff --git a/libcfs/include/libcfs/linux/linux-hash.h b/libcfs/include/libcfs/linux/linux-hash.h
> index 1227ec8..0453cd9 100644
> --- a/libcfs/include/libcfs/linux/linux-hash.h
> +++ b/libcfs/include/libcfs/linux/linux-hash.h
> @@ -38,6 +38,94 @@ u64 cfs_hashlen_string(const void *salt, const char *name);
>  #endif
>  #endif /* !HAVE_STRINGHASH */
>  
> +#ifndef HAVE_RHLTABLE
> +struct rhlist_head {
> +	struct rhash_head		rhead;
> +	struct rhlist_head __rcu	*next;
> +};
> +
> +struct rhltable {
> +	struct rhashtable ht;
> +};
> +
> +#define rhl_for_each_entry_rcu(tpos, pos, list, member)                 \
> +	for (pos = list; pos && rht_entry(tpos, pos, member);           \
> +		pos = rcu_dereference_raw(pos->next))
> +
> +static inline int rhltable_init(struct rhltable *hlt, const struct rhashtable_params *params)
> +{
> +	return rhashtable_init(&hlt->ht, params);
> +}
> +
> +static inline struct rhlist_head *rhltable_lookup(
> +	struct rhltable *hlt, const void *key,
> +	const struct rhashtable_params params)
> +{
> +	struct rhashtable *ht = &hlt->ht;
> +	struct rhashtable_compare_arg arg = {
> +		.ht = ht,
> +		.key = key,
> +	};
> +	struct bucket_table *tbl;
> +	struct rhash_head *he;
> +	unsigned int hash;
> +
> +	tbl = rht_dereference_rcu(ht->tbl, ht);
> +restart:
> +	hash = rht_key_hashfn(ht, tbl, key, params);
> +	rht_for_each_rcu(he, tbl, hash) {
> +		if (params.obj_cmpfn ?
> +		    params.obj_cmpfn(&arg, rht_obj(ht, he)) :
> +		    rhashtable_compare(&arg, rht_obj(ht, he)))
> +			continue;
> +		return he ? container_of(he, struct rhlist_head, rhead) : NULL;
> +	}
> +
> +	/* Ensure we see any new tables. */
> +	smp_rmb();
> +
> +	tbl = rht_dereference_rcu(tbl->future_tbl, ht);
> +	if (unlikely(tbl))
> +		goto restart;
> +
> +	return NULL;
> +}
> +
> +static inline int rhltable_insert_key(
> +	struct rhltable *hlt, const void *key, struct rhlist_head *list,
> +	const struct rhashtable_params params)
> +{
> +	return PTR_ERR(__rhashtable_insert_fast(&hlt->ht, key, &list->rhead,
> +						params));
> +}
> +
> +static inline int rhltable_remove(
> +	struct rhltable *hlt, struct rhlist_head *list,
> +	const struct rhashtable_params params)
> +{
> +	return rhashtable_remove_fast(&hlt->ht, &list->rhead, params);
> +}
> +
> +static inline void rhltable_free_and_destroy(struct rhltable *hlt,
> +					     void (*free_fn)(void *ptr,
> +							     void *arg),
> +					     void *arg)
> +{
> +	return rhashtable_free_and_destroy(&hlt->ht, free_fn, arg);
> +}
> +
> +static inline void rhltable_destroy(struct rhltable *hlt)
> +{
> +	return rhltable_free_and_destroy(hlt, NULL, NULL);
> +}
> +
> +static inline void rhltable_walk_enter(struct rhltable *hlt,
> +				       struct rhashtable_iter *iter)
> +{
> +	rhashtable_walk_init(&hlt->ht, iter);
> +}
> +#endif /* !HAVE_RHLTABLE */
> +
>  #ifndef HAVE_RHASHTABLE_LOOKUP_GET_INSERT_FAST
>  /**
>   * rhashtable_lookup_get_insert_fast - lookup and insert object into hash table
> diff --git a/lustre/include/lustre_export.h b/lustre/include/lustre_export.h
> index 5ead593..54887c9 100644
> --- a/lustre/include/lustre_export.h
> +++ b/lustre/include/lustre_export.h
> @@ -209,7 +209,7 @@ struct obd_export {
>  	/* Unlinked export list */
>  	struct list_head	exp_stale_list;
>  	struct hlist_node	exp_uuid_hash;	/** uuid-export hash*/
> -	struct hlist_node	exp_nid_hash;	/** nid-export hash */
> +	struct rhlist_head	exp_nid_hash;	/** nid-export hash */
>  	struct hlist_node	exp_gen_hash;   /** last_rcvd clt gen hash */
>          /**
>           * All exports eligible for ping evictor are linked into a list
> diff --git a/lustre/include/obd.h b/lustre/include/obd.h
> index 1fcf0a2..8219710 100644
> --- a/lustre/include/obd.h
> +++ b/lustre/include/obd.h
> @@ -639,7 +639,7 @@ struct obd_device {
>          /* uuid-export hash body */
>  	struct cfs_hash             *obd_uuid_hash;
>          /* nid-export hash body */
> -	struct cfs_hash             *obd_nid_hash;
> +	struct rhltable			obd_nid_hash;
>  	/* nid stats body */
>  	struct cfs_hash             *obd_nid_stats_hash;
>  	/* client_generation-export hash body */
> @@ -750,6 +750,12 @@ struct obd_device {
>  	struct completion		obd_kobj_unregister;
>  };
>  
> +int obd_nid_export_for_each(struct obd_device *obd, lnet_nid_t nid,
> +			    int cb(struct obd_export *exp, void *data),
> +			    void *data);
> +int obd_nid_add(struct obd_device *obd, struct obd_export *exp);
> +void obd_nid_del(struct obd_device *obd, struct obd_export *exp);
> +
>  /* get/set_info keys */
>  #define KEY_ASYNC               "async"
>  #define KEY_CHANGELOG_CLEAR     "changelog_clear"
> diff --git a/lustre/include/obd_support.h b/lustre/include/obd_support.h
> index e9dd33e..0175cf8 100644
> --- a/lustre/include/obd_support.h
> +++ b/lustre/include/obd_support.h
> @@ -78,9 +78,6 @@ extern char obd_jobid_var[];
>  #define HASH_UUID_BKT_BITS 5
>  #define HASH_UUID_CUR_BITS 7
>  #define HASH_UUID_MAX_BITS 12
> -#define HASH_NID_BKT_BITS 5
> -#define HASH_NID_CUR_BITS 7
> -#define HASH_NID_MAX_BITS 12
>  #define HASH_NID_STATS_BKT_BITS 5
>  #define HASH_NID_STATS_CUR_BITS 7
>  #define HASH_NID_STATS_MAX_BITS 12
> diff --git a/lustre/ldlm/ldlm_flock.c b/lustre/ldlm/ldlm_flock.c
> index f848a36..4c1603a 100644
> --- a/lustre/ldlm/ldlm_flock.c
> +++ b/lustre/ldlm/ldlm_flock.c
> @@ -161,20 +161,18 @@ ldlm_flock_destroy(struct ldlm_lock *lock, enum ldlm_mode mode, __u64 flags)
>   */
>  
>  struct ldlm_flock_lookup_cb_data {
> -	__u64 *bl_owner;
> +	u64 *bl_owner;

Arrrggg.   Please don't do this.

>  	lock = cfs_hash_lookup(exp->exp_flock_hash, cb_data->bl_owner);
> -	if (lock == NULL)
> +	if (!lock)

Or this.

If you want to fix up this stuff, do it in a separate patch.
A patch should just do one thing.
I would actually prefer that the "add compatibilty code so we can use
rhltables in old kernels" was a separate patch from "use rhltables for
obd_nid_hash", but at least those two are conceptually related.

It is much harder to read a patch if I keep having to say to myself "Oh,
that change is irrelevant here,  I can ignore it".

Thanks,
NeilBrown
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 832 bytes
Desc: not available
URL: <http://lists.lustre.org/pipermail/lustre-devel_lists.lustre.org/attachments/20181102/8d11f411/attachment.sig>

From jsimmons at infradead.org  Sun Nov  4 20:52:17 2018
From: jsimmons at infradead.org (James Simmons)
Date: Sun, 4 Nov 2018 20:52:17 +0000 (GMT)
Subject: [lustre-devel] [PATCH 16/28] lustre: statahead: missing barrier
 before wake_up
In-Reply-To: <87efci8w4q.fsf@notabene.neil.brown.name>
References: <1539543498-29105-1-git-send-email-jsimmons@infradead.org>
 <1539543498-29105-17-git-send-email-jsimmons@infradead.org>
 <87d0s8c8tp.fsf@notabene.neil.brown.name>
 <alpine.LFD.2.21.1810212348370.6249@casper.infradead.org>
 <87efci8w4q.fsf@notabene.neil.brown.name>
Message-ID: <alpine.LFD.2.21.1811042048170.27308@casper.infradead.org>


> On Sun, Oct 21 2018, James Simmons wrote:
> 
> >> On Sun, Oct 14 2018, James Simmons wrote:
> >> 
> >> > From: Lai Siyao <lai.siyao at whamcloud.com>
> >> >
> >> > A barrier is missing before wake_up() in ll_statahead_interpret(),
> >> > which may cause 'ls' hang. Under the right conditions a basic 'ls'
> >> > can fail. The debug logs show:
> >> >
> >> > statahead.c:683:ll_statahead_interpret()) sa_entry software rc -13
> >> > statahead.c:1666:ll_statahead()) revalidate statahead software: -11.
> >> >
> >> > Obviously statahead failure didn't notify 'ls' process in time.
> >> > The mi_cbdata can be stale so add a barrier before calling
> >> > wake_up().
> >> >
> >> > Signed-off-by: Lai Siyao <lai.siyao at whamcloud.com>
> >> > Signed-off-by: Bob Glossman <bob.glossman at intel.com>
> >> > WC-bug-id: https://jira.whamcloud.com/browse/LU-9210
> >> > Reviewed-on: https://review.whamcloud.com/27330
> >> > Reviewed-by: Nathaniel Clark <nclark at whamcloud.com>
> >> > Reviewed-by: Oleg Drokin <green at whamcloud.com>
> >> > Signed-off-by: James Simmons <jsimmons at infradead.org>
> >> > ---
> >> >  drivers/staging/lustre/lustre/llite/statahead.c | 8 +++++++-
> >> >  1 file changed, 7 insertions(+), 1 deletion(-)
> >> >
> >> > diff --git a/drivers/staging/lustre/lustre/llite/statahead.c b/drivers/staging/lustre/lustre/llite/statahead.c
> >> > index 1ad308c..0174a4c 100644
> >> > --- a/drivers/staging/lustre/lustre/llite/statahead.c
> >> > +++ b/drivers/staging/lustre/lustre/llite/statahead.c
> >> > @@ -680,8 +680,14 @@ static int ll_statahead_interpret(struct ptlrpc_request *req,
> >> >  
> >> >  	spin_lock(&lli->lli_sa_lock);
> >> >  	if (rc) {
> >> > -		if (__sa_make_ready(sai, entry, rc))
> >> > +		if (__sa_make_ready(sai, entry, rc)) {
> >> > +			/* LU-9210 : Under the right conditions even 'ls'
> >> > +			 * can cause the statahead to fail. Using a memory
> >> > +			 * barrier resolves this issue.
> >> > +			 */
> >> > +			smp_mb();
> >> >  			wake_up(&sai->sai_waitq);
> >> > +		}
> >> >  	} else {
> >> >  		int first = 0;
> >> >  		entry->se_minfo = minfo;
> >> > -- 
> >> > 1.8.3.1
> >> 
> >> Again, this is a fairly lame comment to justify the smp_mb().
> >> It appears to me that the issue is most likely the value of
> >> entry->se_state.
> >> __sa_make_ready() sets this and revalidate_statahead_dentry tests it
> >> after waiting on sai_waitq.
> >> So I think it would be best if we changed __sa_make_ready() to
> >> 
> >> 	smp_store_release(&entry->se_state, ret < 0 ? SA_ENTRY_INVA : SA_ENTRY_SUCC)
> >> 
> >> and in ll_statahead_interpret() have
> >> 
> >> 	if (smp_load_acquire(&entry->se_state) == SA_ENTRY_SUCC &&
> >>             entry->se_inode) {
> >> 
> >> This would make it obvious which variable was important, and would show
> >> the paired synchronization points.
> >
> > If you think this is lame you should be the JIRA ticket and the original 
> > patch. It had zero info so I attempted to extract what I could out of the
> > ticket. Hopefully Lai can fill in the details. I have no problems fixing 
> > this another way. I don't see a way in the ticket to easily reproduce this
> > problem to see the new approach would fix it :-(
> 
> I have imposed the version that I think is correct.  See below.

I opened a ticket - https://jira.whamcloud.com/browse/LU-11616 and pushed 
this to the OpenSFS branch for full testing. The patch is at:

https://review.whamcloud.com/#/c/33571

BTW I did not know about (total store order) TSO platforms and how some 
architectures don't support this property. An audit of the smp barriers
might be a good idea for Lustre. For people not aware of TSO a good
article on this is at:

https://lwn.net/Articles/576486
 
> Thanks,
> NeilBrown
> 
> --- a/drivers/staging/lustre/lustre/llite/statahead.c
> +++ b/drivers/staging/lustre/lustre/llite/statahead.c
> @@ -322,7 +322,11 @@ __sa_make_ready(struct ll_statahead_info *sai, struct sa_entry *entry, int ret)
>  		}
>  	}
>  	list_add(&entry->se_list, pos);
> -	entry->se_state = ret < 0 ? SA_ENTRY_INVA : SA_ENTRY_SUCC;
> +	/*
> +	 * LU-9210: ll_statahead_interpet must be able to see this before
> +	 * we wake it up
> +	 */
> +	smp_store_release(&entry->se_state, ret < 0 ? SA_ENTRY_INVA : SA_ENTRY_SUCC);
>  
>  	return (index == sai->sai_index_wait);
>  }
> @@ -1390,7 +1394,12 @@ static int revalidate_statahead_dentry(struct inode *dir,
>  		}
>  	}
>  
> -	if (entry->se_state == SA_ENTRY_SUCC && entry->se_inode) {
> +	/*
> +	 * We need to see the value that was set immediately before we
> +	 * were woken up.
> +	 */
> +	if (smp_load_acquire(&entry->se_state) == SA_ENTRY_SUCC &&
> +	    entry->se_inode) {
>  		struct inode *inode = entry->se_inode;
>  		struct lookup_intent it = { .it_op = IT_GETATTR,
>  					    .it_lock_handle = entry->se_handle };
> 

From jsimmons at infradead.org  Sun Nov  4 20:53:35 2018
From: jsimmons at infradead.org (James Simmons)
Date: Sun, 4 Nov 2018 20:53:35 +0000 (GMT)
Subject: [lustre-devel] [PATCH 12/28] lustre: ptlrpc: do not wakeup
 every second
In-Reply-To: <87pnvt325a.fsf@notabene.neil.brown.name>
References: <1539543498-29105-1-git-send-email-jsimmons@infradead.org>
 <1539543498-29105-13-git-send-email-jsimmons@infradead.org>
 <87va5l39hl.fsf@notabene.neil.brown.name>
 <MWHPR11MB148802D76D49E06343462260CBF30@MWHPR11MB1488.namprd11.prod.outlook.com>
 <87pnvt325a.fsf@notabene.neil.brown.name>
Message-ID: <alpine.LFD.2.21.1811042052360.27308@casper.infradead.org>


> > Neil,
> >
> > Does your statement imply this would spin?  It definitely doesn’t just
> > spin (that behavior in a main “wait for work” spot of a (depending on
> > settings) ~per-CPU daemon would render systems unusable and this patch
> > has been in testing for a while).  So what is the detailed behavior of
> > a “timeout that expires immediately”?
> 
> Hi Patrick,
>  it definitely spins for me.
> 
>  I should have clarified that the SFS patch
> 
>    e81847bd0651 LU-9660 ptlrpc: do not wakeup every second
> 
>  is correct, as __l_wait_event() treats a timeout value of 0 as meaning an
>  indefinite timeout.
>  The error was in the conversion to wait_event_idle_timeout().  The
>  various wait_event*timeout() functions treat 0 as 1 less than 1.
>  If you want to not have a timeout, you need to not use the *_timeout()
>  version.
>  If a timeout is undesirable rather than fatal, then
>  MAX_SCHEDULE_TIMEOUT can be used.  In this case, that seemed best.
> 
> Thanks,
> NeilBrown
> 
> 
> >
> > - Patrick
> >
> >
> > ________________________________
> > From: lustre-devel <lustre-devel-bounces at lists.lustre.org> on behalf of NeilBrown <neilb at suse.com>
> > Sent: Sunday, October 28, 2018 7:03:02 PM
> > To: James Simmons; Andreas Dilger; Oleg Drokin
> > Cc: Lustre Development List
> > Subject: Re: [lustre-devel] [PATCH 12/28] lustre: ptlrpc: do not wakeup every second
> >
> > On Sun, Oct 14 2018, James Simmons wrote:
> >
> >> From: Alex Zhuravlev <bzzz at whamcloud.com>
> >>
> >> Even if there are no RPC requests on the set, there is no need to
> >> wake up every second. The thread is woken up when a request is added
> >> to the set or when the STOP bit is set, so it is sufficient to only
> >> wake up when there are requests on the set to worry about.
> >>
> >> Signed-off-by: Alex Zhuravlev <bzzz at whamcloud.com>
> >> WC-bug-id: https://jira.whamcloud.com/browse/LU-9660
> >> Reviewed-on: https://review.whamcloud.com/28776
> >> Reviewed-by: Andreas Dilger <adilger at whamcloud.com>
> >> Reviewed-by: Patrick Farrell <paf at cray.com>
> >> Reviewed-by: Oleg Drokin <green at whamcloud.com>
> >> Signed-off-by: James Simmons <jsimmons at infradead.org>
> >> ---
> >>  drivers/staging/lustre/lustre/ptlrpc/ptlrpcd.c | 4 ++--
> >>  1 file changed, 2 insertions(+), 2 deletions(-)
> >>
> >> diff --git a/drivers/staging/lustre/lustre/ptlrpc/ptlrpcd.c b/drivers/staging/lustre/lustre/ptlrpc/ptlrpcd.c
> >> index c201a88..5b4977b 100644
> >> --- a/drivers/staging/lustre/lustre/ptlrpc/ptlrpcd.c
> >> +++ b/drivers/staging/lustre/lustre/ptlrpc/ptlrpcd.c
> >> @@ -371,7 +371,7 @@ static int ptlrpcd_check(struct lu_env *env, struct ptlrpcd_ctl *pc)
> >>                }
> >>        }
> >>
> >> -     return rc;
> >> +     return rc || test_bit(LIOD_STOP, &pc->pc_flags);
> >>  }
> >>
> >>  /**
> >> @@ -441,7 +441,7 @@ static int ptlrpcd(void *arg)
> >>                lu_context_enter(env.le_ses);
> >>                if (wait_event_idle_timeout(set->set_waitq,
> >>                                            ptlrpcd_check(&env, pc),
> >> -                                         (timeout ? timeout : 1) * HZ) == 0)
> >> +                                         timeout * HZ) == 0)
> >>                        ptlrpc_expired_set(set);
> >
> > This is incorrect.
> > A timeout of zero means the timeout happens after zero jiffies
> > (immediately), it doesn't mean there is no timeout.
> > If we want a "timeout" of zero to mean "Wait forever", we need something
> > like:
> >
> >   wait_event_idle_timeout(.....,
> >                           timeout ? (timeout * HZ) : MAX_SCHEDULE_TIMEOUT) == 0
> >
> > I've updated the patch accordingly.

I did that change locally as well and my CPU load problem went away. 
Thanks for figuring it out. Will be more careful in the future.

From jsimmons at infradead.org  Sun Nov  4 21:29:47 2018
From: jsimmons at infradead.org (James Simmons)
Date: Sun, 4 Nov 2018 21:29:47 +0000 (GMT)
Subject: [lustre-devel] [PATCH 04/28] lustre: ptlrpc: Do not assert when
 bd_nob_transferred != 0
In-Reply-To: <87pnw28xvx.fsf@notabene.neil.brown.name>
References: <1539543498-29105-1-git-send-email-jsimmons@infradead.org>
 <1539543498-29105-5-git-send-email-jsimmons@infradead.org>
 <87va60cgjx.fsf@notabene.neil.brown.name>
 <alpine.LFD.2.21.1810212339590.6249@casper.infradead.org>
 <87pnw28xvx.fsf@notabene.neil.brown.name>
Message-ID: <alpine.LFD.2.21.1811042054180.27308@casper.infradead.org>


> >> On Sun, Oct 14 2018, James Simmons wrote:
> >> 
> >> > From: Doug Oucharek <dougso at me.com>
> >> >
> >> > There is a case in the routine ptlrpc_register_bulk() where we were
> >> > asserting if bd_nob_transferred != 0 when not resending.  There is
> >> > evidence that network errors can create a situation where
> >> > this does happen. So we should not be asserting!
> >> >
> >> > This patch changes that assert to an error return code of -EIO.
> >> >
> >> > Signed-off-by: Doug Oucharek <dougso at me.com>
> >> > WC-bug-id: https://jira.whamcloud.com/browse/LU-9828
> >> > Reviewed-on: https://review.whamcloud.com/28491
> >> > Reviewed-by: Dmitry Eremin <dmitry.eremin at intel.com>
> >> > Reviewed-by: Sonia Sharma <sharmaso at whamcloud.com>
> >> > Reviewed-by: Oleg Drokin <green at whamcloud.com>
> >> > Signed-off-by: James Simmons <jsimmons at infradead.org>
> >> > ---
> >> >  drivers/staging/lustre/lustre/ptlrpc/niobuf.c | 8 ++++++--
> >> >  1 file changed, 6 insertions(+), 2 deletions(-)
> >> >
> >> > diff --git a/drivers/staging/lustre/lustre/ptlrpc/niobuf.c b/drivers/staging/lustre/lustre/ptlrpc/niobuf.c
> >> > index 27eb1c0..7e7db24 100644
> >> > --- a/drivers/staging/lustre/lustre/ptlrpc/niobuf.c
> >> > +++ b/drivers/staging/lustre/lustre/ptlrpc/niobuf.c
> >> > @@ -139,8 +139,12 @@ static int ptlrpc_register_bulk(struct ptlrpc_request *req)
> >> >  	/* cleanup the state of the bulk for it will be reused */
> >> >  	if (req->rq_resend || req->rq_send_state == LUSTRE_IMP_REPLAY)
> >> >  		desc->bd_nob_transferred = 0;
> >> > -	else
> >> > -		LASSERT(desc->bd_nob_transferred == 0);
> >> > +	else if (desc->bd_nob_transferred != 0)
> >> > +		/* If the network failed after an RPC was sent, this condition
> >> > +		 * could happen.  Rather than assert (was here before), return
> >> > +		 * an EIO error.
> >> > +		 */
> >> > +		return -EIO;
> >> 
> >> This looks weird, and the justification is rather lame.
> >> I wonder if this is an attempt to fix the same problem that the smp_mb()
> >> in the previous patch was attempting to fix (and I'm not yet convinced
> >> that either is the correct fix).
> >
> > When the above condition happens the LASSERT ends up taking out the 
> > node with a panic which in turn kills the application running on the cluster.
> > When replaced with reporting an EIO error the node survives as well as the 
> > job. The job might fail at its IO but it wouldn't fail performing its work 
> > flow which is way more important.
> 
> Yes, a meaningless error is better than a crash, but a proper fix is
> better still.  As I said, my guess is that the memory barrier in the
> previous patch might have fixed the bug, so the LASSERT can remain.
> 
> Doug: is there any chance that this might be the case?

I got a hold of Doug and discussed this issue. So the answer is that the
original logs to track down the original problem no longer exist. So 
finding the original source of the problem can't be done at this point.
Would you be okay with a version of this patch with dump_stack() and
treat it as a debug patch. We really need to collect logs to figure out
the real problem. I will push a debug patch to OpenSFS branch since it
is more widely used.

From jsimmons at infradead.org  Sun Nov  4 21:31:27 2018
From: jsimmons at infradead.org (James Simmons)
Date: Sun, 4 Nov 2018 21:31:27 +0000 (GMT)
Subject: [lustre-devel] [PATCH 18/28] lustre: mdc: expose changelog
 through char devices
In-Reply-To: <87efc82ayj.fsf@notabene.neil.brown.name>
References: <1539543498-29105-1-git-send-email-jsimmons@infradead.org>
 <1539543498-29105-19-git-send-email-jsimmons@infradead.org>
 <87efc82ayj.fsf@notabene.neil.brown.name>
Message-ID: <alpine.LFD.2.21.1811042130370.27308@casper.infradead.org>


> On Sun, Oct 14 2018, James Simmons wrote:
> 
> > From: Henri Doreau <henri.doreau at cea.fr>
> >
> > Register one character device per MDT in order to allow non-llapi to
> > read them and to make delivery more efficient.
> >
> > - open() spawns a thread to prefetch records and enqueue them into a
> >   local buffer (unless the device is open in write-only mode).
> > - lseek() can be used to jump to a specific record, in which case the
> >   offset is a record number (with SEEK_SET) or a number of records to
> >   skip (SEEK_CUR). Movement can only be done forward.
> > - read() copies records to userland. No truncation happens, so short
> >   reads are likely.
> > - write() is used to transmit control commands to the device.
> >   The only available one is changelog_clear, which is done by writing
> >   "clear:cl<user>:<recno>" into the device.
> > - close() terminates the prefetch thread if any, and releases resources.
> >
> > It is possible to poll() on the device to get notified when new records
> > are available for read.
> >
> > Signed-off-by: Henri Doreau <henri.doreau at cea.fr>
> > WC-bug-id: https://jira.whamcloud.com/browse/LU-7659
> > Reviewed-on: https://review.whamcloud.com/18900
> > Reviewed-by: Andreas Dilger <adilger at whamcloud.com>
> > Reviewed-by: John L. Hammond <jhammond at whamcloud.com>
> > Reviewed-by: Oleg Drokin <green at whamcloud.com>
> > Signed-off-by: James Simmons <jsimmons at infradead.org>
> 
> This patches causes problems around sanity test 161a.
> If you run -only '160h 160i 161a' it hangs.
> 
> Adding
> Commit: 89e52326b5bd ("LU-10166 mdc: invalid free in changelog reader")
> seems to fix the problem, so I'll port that into the series immediately
> after this patch.

I was planning to push this in the next batch. I wouldn't push it in that
case and just wait for it to show up in lustre-testing.


> >  .../include/uapi/linux/lustre/lustre_kernelcomm.h  |   3 -
> >  .../lustre/include/uapi/linux/lustre/lustre_user.h |   7 -
> >  drivers/staging/lustre/lustre/include/obd.h        |   2 +
> >  drivers/staging/lustre/lustre/ldlm/ldlm_lib.c      |   2 +
> >  drivers/staging/lustre/lustre/llite/dir.c          |   8 -
> >  drivers/staging/lustre/lustre/lmv/lmv_obd.c        |  13 -
> >  drivers/staging/lustre/lustre/mdc/Makefile         |   2 +-
> >  drivers/staging/lustre/lustre/mdc/mdc_changelog.c  | 722 +++++++++++++++++++++
> >  drivers/staging/lustre/lustre/mdc/mdc_internal.h   |   4 +
> >  drivers/staging/lustre/lustre/mdc/mdc_request.c    | 198 +-----
> >  11 files changed, 745 insertions(+), 218 deletions(-)
> >  create mode 100644 drivers/staging/lustre/lustre/mdc/mdc_changelog.c
> >
> > diff --git a/drivers/staging/lustre/include/uapi/linux/lustre/lustre_ioctl.h b/drivers/staging/lustre/include/uapi/linux/lustre/lustre_ioctl.h
> > index 6e4e109..098b6451 100644
> > --- a/drivers/staging/lustre/include/uapi/linux/lustre/lustre_ioctl.h
> > +++ b/drivers/staging/lustre/include/uapi/linux/lustre/lustre_ioctl.h
> > @@ -172,7 +172,7 @@ static inline __u32 obd_ioctl_packlen(struct obd_ioctl_data *data)
> >  #define OBD_GET_VERSION		_IOWR('f', 144, OBD_IOC_DATA_TYPE)
> >  /*	OBD_IOC_GSS_SUPPORT	_IOWR('f', 145, OBD_IOC_DATA_TYPE) */
> >  /*	OBD_IOC_CLOSE_UUID	_IOWR('f', 147, OBD_IOC_DATA_TYPE) */
> > -#define OBD_IOC_CHANGELOG_SEND	_IOW('f', 148, OBD_IOC_DATA_TYPE)
> > +/*	OBD_IOC_CHANGELOG_SEND	_IOW('f', 148, OBD_IOC_DATA_TYPE) */
> >  #define OBD_IOC_GETDEVICE	_IOWR('f', 149, OBD_IOC_DATA_TYPE)
> >  #define OBD_IOC_FID2PATH	_IOWR('f', 150, OBD_IOC_DATA_TYPE)
> >  /*	lustre/lustre_user.h	151-153 */
> > diff --git a/drivers/staging/lustre/include/uapi/linux/lustre/lustre_kernelcomm.h b/drivers/staging/lustre/include/uapi/linux/lustre/lustre_kernelcomm.h
> > index 94dadbe..d84a8fc 100644
> > --- a/drivers/staging/lustre/include/uapi/linux/lustre/lustre_kernelcomm.h
> > +++ b/drivers/staging/lustre/include/uapi/linux/lustre/lustre_kernelcomm.h
> > @@ -54,15 +54,12 @@ struct kuc_hdr {
> >  	__u16 kuc_msglen;
> >  } __aligned(sizeof(__u64));
> >  
> > -#define KUC_CHANGELOG_MSG_MAXSIZE (sizeof(struct kuc_hdr) + CR_MAXSIZE)
> > -
> >  #define KUC_MAGIC		0x191C /*Lustre9etLinC */
> >  
> >  /* kuc_msgtype values are defined in each transport */
> >  enum kuc_transport_type {
> >  	KUC_TRANSPORT_GENERIC	= 1,
> >  	KUC_TRANSPORT_HSM	= 2,
> > -	KUC_TRANSPORT_CHANGELOG	= 3,
> >  };
> >  
> >  enum kuc_generic_message_type {
> > diff --git a/drivers/staging/lustre/include/uapi/linux/lustre/lustre_user.h b/drivers/staging/lustre/include/uapi/linux/lustre/lustre_user.h
> > index b8525e5..715f1c5 100644
> > --- a/drivers/staging/lustre/include/uapi/linux/lustre/lustre_user.h
> > +++ b/drivers/staging/lustre/include/uapi/linux/lustre/lustre_user.h
> > @@ -967,13 +967,6 @@ static inline void changelog_remap_rec(struct changelog_rec *rec,
> >  	rec->cr_flags = (rec->cr_flags & CLF_FLAGMASK) | crf_wanted;
> >  }
> >  
> > -struct ioc_changelog {
> > -	__u64 icc_recno;
> > -	__u32 icc_mdtindex;
> > -	__u32 icc_id;
> > -	__u32 icc_flags;
> > -};
> > -
> >  enum changelog_message_type {
> >  	CL_RECORD = 10, /* message is a changelog_rec */
> >  	CL_EOF    = 11, /* at end of current changelog */
> > diff --git a/drivers/staging/lustre/lustre/include/obd.h b/drivers/staging/lustre/lustre/include/obd.h
> > index 11e7ae8..76ae0b3 100644
> > --- a/drivers/staging/lustre/lustre/include/obd.h
> > +++ b/drivers/staging/lustre/lustre/include/obd.h
> > @@ -345,6 +345,8 @@ struct client_obd {
> >  	void			*cl_lru_work;
> >  	/* hash tables for osc_quota_info */
> >  	struct rhashtable	cl_quota_hash[MAXQUOTAS];
> > +	/* Links to the global list of registered changelog devices */
> > +	struct list_head	cl_chg_dev_linkage;
> >  };
> >  
> >  #define obd2cli_tgt(obd) ((char *)(obd)->u.cli.cl_target_uuid.uuid)
> > diff --git a/drivers/staging/lustre/lustre/ldlm/ldlm_lib.c b/drivers/staging/lustre/lustre/ldlm/ldlm_lib.c
> > index 32eda4f..732ef3a 100644
> > --- a/drivers/staging/lustre/lustre/ldlm/ldlm_lib.c
> > +++ b/drivers/staging/lustre/lustre/ldlm/ldlm_lib.c
> > @@ -395,6 +395,8 @@ int client_obd_setup(struct obd_device *obddev, struct lustre_cfg *lcfg)
> >  	init_waitqueue_head(&cli->cl_mod_rpcs_waitq);
> >  	cli->cl_mod_tag_bitmap = NULL;
> >  
> > +	INIT_LIST_HEAD(&cli->cl_chg_dev_linkage);
> > +
> >  	if (connect_op == MDS_CONNECT) {
> >  		cli->cl_max_mod_rpcs_in_flight = cli->cl_max_rpcs_in_flight - 1;
> >  		cli->cl_mod_tag_bitmap = kcalloc(BITS_TO_LONGS(OBD_MAX_RIF_MAX),
> > diff --git a/drivers/staging/lustre/lustre/llite/dir.c b/drivers/staging/lustre/lustre/llite/dir.c
> > index 19c5e9c..36cea8d 100644
> > --- a/drivers/staging/lustre/lustre/llite/dir.c
> > +++ b/drivers/staging/lustre/lustre/llite/dir.c
> > @@ -1481,14 +1481,6 @@ static long ll_dir_ioctl(struct file *file, unsigned int cmd, unsigned long arg)
> >  		return obd_iocontrol(cmd, sbi->ll_md_exp, 0, NULL,
> >  				     (void __user *)arg);
> >  	}
> > -	case OBD_IOC_CHANGELOG_SEND:
> > -	case OBD_IOC_CHANGELOG_CLEAR:
> > -		if (!capable(CAP_SYS_ADMIN))
> > -			return -EPERM;
> > -
> > -		rc = copy_and_ioctl(cmd, sbi->ll_md_exp, (void __user *)arg,
> > -				    sizeof(struct ioc_changelog));
> > -		return rc;
> >  	case OBD_IOC_FID2PATH:
> >  		return ll_fid2path(inode, (void __user *)arg);
> >  	case LL_IOC_GETPARENT:
> > diff --git a/drivers/staging/lustre/lustre/lmv/lmv_obd.c b/drivers/staging/lustre/lustre/lmv/lmv_obd.c
> > index 952c68e..32bb9fc 100644
> > --- a/drivers/staging/lustre/lustre/lmv/lmv_obd.c
> > +++ b/drivers/staging/lustre/lustre/lmv/lmv_obd.c
> > @@ -951,19 +951,6 @@ static int lmv_iocontrol(unsigned int cmd, struct obd_export *exp,
> >  		kfree(oqctl);
> >  		break;
> >  	}
> > -	case OBD_IOC_CHANGELOG_SEND:
> > -	case OBD_IOC_CHANGELOG_CLEAR: {
> > -		struct ioc_changelog *icc = karg;
> > -
> > -		if (icc->icc_mdtindex >= count)
> > -			return -ENODEV;
> > -
> > -		tgt = lmv->tgts[icc->icc_mdtindex];
> > -		if (!tgt || !tgt->ltd_exp || !tgt->ltd_active)
> > -			return -ENODEV;
> > -		rc = obd_iocontrol(cmd, tgt->ltd_exp, sizeof(*icc), icc, NULL);
> > -		break;
> > -	}
> >  	case LL_IOC_GET_CONNECT_FLAGS: {
> >  		tgt = lmv->tgts[0];
> >  
> > diff --git a/drivers/staging/lustre/lustre/mdc/Makefile b/drivers/staging/lustre/lustre/mdc/Makefile
> > index 64cf49e..5f48e91 100644
> > --- a/drivers/staging/lustre/lustre/mdc/Makefile
> > +++ b/drivers/staging/lustre/lustre/mdc/Makefile
> > @@ -2,4 +2,4 @@ ccflags-y += -I$(srctree)/drivers/staging/lustre/include
> >  ccflags-y += -I$(srctree)/drivers/staging/lustre/lustre/include
> >  
> >  obj-$(CONFIG_LUSTRE_FS) += mdc.o
> > -mdc-y := mdc_request.o mdc_reint.o mdc_lib.o mdc_locks.o lproc_mdc.o
> > +mdc-y := mdc_changelog.o mdc_request.o mdc_reint.o mdc_lib.o mdc_locks.o lproc_mdc.o
> > diff --git a/drivers/staging/lustre/lustre/mdc/mdc_changelog.c b/drivers/staging/lustre/lustre/mdc/mdc_changelog.c
> > new file mode 100644
> > index 0000000..a5f3c64
> > --- /dev/null
> > +++ b/drivers/staging/lustre/lustre/mdc/mdc_changelog.c
> > @@ -0,0 +1,722 @@
> > +// SPDX-License-Identifier: GPL-2.0
> > +/*
> > + * GPL HEADER START
> > + *
> > + * DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
> > + *
> > + * This program is free software; you can redistribute it and/or modify
> > + * it under the terms of the GNU General Public License version 2 only,
> > + * as published by the Free Software Foundation.
> > + *
> > + * This program is distributed in the hope that it will be useful, but
> > + * WITHOUT ANY WARRANTY; without even the implied warranty of
> > + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
> > + * General Public License version 2 for more details (a copy is included
> > + * in the LICENSE file that accompanied this code).
> > + *
> > + * You should have received a copy of the GNU General Public License
> > + * version 2 along with this program; If not, see
> > + * http://www.gnu.org/licenses/gpl-2.0.html
> > + *
> > + * GPL HEADER END
> > + */
> > +/*
> > + * Copyright (c) 2017, Commissariat a l'Energie Atomique et aux Energies
> > + *                     Alternatives.
> > + *
> > + * Author: Henri Doreau <henri.doreau at cea.fr>
> > + */
> > +
> > +#define DEBUG_SUBSYSTEM S_MDC
> > +
> > +#include <linux/init.h>
> > +#include <linux/kthread.h>
> > +#include <linux/poll.h>
> > +#include <linux/miscdevice.h>
> > +
> > +#include <lustre_log.h>
> > +
> > +#include "mdc_internal.h"
> > +
> > +/*
> > + * -- Changelog delivery through character device --
> > + */
> > +
> > +/**
> > + * Mutex to protect chlg_registered_devices below
> > + */
> > +static DEFINE_MUTEX(chlg_registered_dev_lock);
> > +
> > +/**
> > + * Global linked list of all registered devices (one per MDT).
> > + */
> > +static LIST_HEAD(chlg_registered_devices);
> > +
> > +struct chlg_registered_dev {
> > +	/* Device name of the form "changelog-{MDTNAME}" */
> > +	char			ced_name[32];
> > +	/* Misc device descriptor */
> > +	struct miscdevice	ced_misc;
> > +	/* OBDs referencing this device (multiple mount point) */
> > +	struct list_head	ced_obds;
> > +	/* Reference counter for proper deregistration */
> > +	struct kref		ced_refs;
> > +	/* Link within the global chlg_registered_devices */
> > +	struct list_head	ced_link;
> > +};
> > +
> > +struct chlg_reader_state {
> > +	/* Shortcut to the corresponding OBD device */
> > +	struct obd_device	*crs_obd;
> > +	/* An error occurred that prevents from reading further */
> > +	bool			 crs_err;
> > +	/* EOF, no more records available */
> > +	bool			 crs_eof;
> > +	/* Userland reader closed connection */
> > +	bool			 crs_closed;
> > +	/* Desired start position */
> > +	u64			 crs_start_offset;
> > +	/* Wait queue for the catalog processing thread */
> > +	wait_queue_head_t	 crs_waitq_prod;
> > +	/* Wait queue for the record copy threads */
> > +	wait_queue_head_t	 crs_waitq_cons;
> > +	/* Mutex protecting crs_rec_count and crs_rec_queue */
> > +	struct mutex		 crs_lock;
> > +	/* Number of item in the list */
> > +	u64			 crs_rec_count;
> > +	/* List of prefetched enqueued_record::enq_linkage_items */
> > +	struct list_head	 crs_rec_queue;
> > +};
> > +
> > +struct chlg_rec_entry {
> > +	/* Link within the chlg_reader_state::crs_rec_queue list */
> > +	struct list_head	enq_linkage;
> > +	/* Data (enq_record) field length */
> > +	u64			enq_length;
> > +	/* Copy of a changelog record (see struct llog_changelog_rec) */
> > +	struct changelog_rec	enq_record[];
> > +};
> > +
> > +enum {
> > +	/* Number of records to prefetch locally. */
> > +	CDEV_CHLG_MAX_PREFETCH = 1024,
> > +};
> > +
> > +/**
> > + * ChangeLog catalog processing callback invoked on each record.
> > + * If the current record is eligible to userland delivery, push
> > + * it into the crs_rec_queue where the consumer code will fetch it.
> > + *
> > + * @param[in]     env  (unused)
> > + * @param[in]     llh  Client-side handle used to identify the llog
> > + * @param[in]     hdr  Header of the current llog record
> > + * @param[in,out] data chlg_reader_state passed from caller
> > + *
> > + * @return 0 or LLOG_PROC_* control code on success, negated error on failure.
> > + */
> > +static int chlg_read_cat_process_cb(const struct lu_env *env,
> > +				    struct llog_handle *llh,
> > +				    struct llog_rec_hdr *hdr, void *data)
> > +{
> > +	struct llog_changelog_rec *rec;
> > +	struct chlg_reader_state *crs = data;
> > +	struct chlg_rec_entry *enq;
> > +	size_t len;
> > +	int rc;
> > +
> > +	LASSERT(crs);
> > +	LASSERT(hdr);
> > +
> > +	rec = container_of(hdr, struct llog_changelog_rec, cr_hdr);
> > +
> > +	if (rec->cr_hdr.lrh_type != CHANGELOG_REC) {
> > +		rc = -EINVAL;
> > +		CERROR("%s: not a changelog rec %x/%d in llog : rc = %d\n",
> > +		       crs->crs_obd->obd_name, rec->cr_hdr.lrh_type,
> > +		       rec->cr.cr_type, rc);
> > +		return rc;
> > +	}
> > +
> > +	/* Skip undesired records */
> > +	if (rec->cr.cr_index < crs->crs_start_offset)
> > +		return 0;
> > +
> > +	CDEBUG(D_HSM, "%llu %02d%-5s %llu 0x%x t=" DFID " p=" DFID " %.*s\n",
> > +	       rec->cr.cr_index, rec->cr.cr_type,
> > +	       changelog_type2str(rec->cr.cr_type), rec->cr.cr_time,
> > +	       rec->cr.cr_flags & CLF_FLAGMASK,
> > +	       PFID(&rec->cr.cr_tfid), PFID(&rec->cr.cr_pfid),
> > +	       rec->cr.cr_namelen, changelog_rec_name(&rec->cr));
> > +
> > +	wait_event_idle(crs->crs_waitq_prod,
> > +			(crs->crs_rec_count < CDEV_CHLG_MAX_PREFETCH ||
> > +			 crs->crs_closed));
> > +
> > +	if (crs->crs_closed)
> > +		return LLOG_PROC_BREAK;
> > +
> > +	len = changelog_rec_size(&rec->cr) + rec->cr.cr_namelen;
> > +	enq = kzalloc(sizeof(*enq) + len, GFP_KERNEL);
> > +	if (!enq)
> > +		return -ENOMEM;
> > +
> > +	INIT_LIST_HEAD(&enq->enq_linkage);
> > +	enq->enq_length = len;
> > +	memcpy(enq->enq_record, &rec->cr, len);
> > +
> > +	mutex_lock(&crs->crs_lock);
> > +	list_add_tail(&enq->enq_linkage, &crs->crs_rec_queue);
> > +	crs->crs_rec_count++;
> > +	mutex_unlock(&crs->crs_lock);
> > +
> > +	wake_up_all(&crs->crs_waitq_cons);
> > +
> > +	return 0;
> > +}
> > +
> > +/**
> > + * Remove record from the list it is attached to and free it.
> > + */
> > +static void enq_record_delete(struct chlg_rec_entry *rec)
> > +{
> > +	list_del(&rec->enq_linkage);
> > +	kfree(rec);
> > +}
> > +
> > +/**
> > + * Release resources associated to a changelog_reader_state instance.
> > + *
> > + * @param  crs  CRS instance to release.
> > + */
> > +static void crs_free(struct chlg_reader_state *crs)
> > +{
> > +	struct chlg_rec_entry *rec;
> > +	struct chlg_rec_entry *tmp;
> > +
> > +	list_for_each_entry_safe(rec, tmp, &crs->crs_rec_queue, enq_linkage)
> > +		enq_record_delete(rec);
> > +
> > +	kfree(crs);
> > +}
> > +
> > +/**
> > + * Record prefetch thread entry point. Opens the changelog catalog and starts
> > + * reading records.
> > + *
> > + * @param[in,out]  args  chlg_reader_state passed from caller.
> > + * @return 0 on success, negated error code on failure.
> > + */
> > +static int chlg_load(void *args)
> > +{
> > +	struct chlg_reader_state *crs = args;
> > +	struct obd_device *obd = crs->crs_obd;
> > +	struct llog_ctxt *ctx = NULL;
> > +	struct llog_handle *llh = NULL;
> > +	int rc;
> > +
> > +	ctx = llog_get_context(obd, LLOG_CHANGELOG_REPL_CTXT);
> > +	if (!ctx) {
> > +		rc = -ENOENT;
> > +		goto err_out;
> > +	}
> > +
> > +	rc = llog_open(NULL, ctx, &llh, NULL, CHANGELOG_CATALOG,
> > +		       LLOG_OPEN_EXISTS);
> > +	if (rc) {
> > +		CERROR("%s: fail to open changelog catalog: rc = %d\n",
> > +		       obd->obd_name, rc);
> > +		goto err_out;
> > +	}
> > +
> > +	rc = llog_init_handle(NULL, llh, LLOG_F_IS_CAT | LLOG_F_EXT_JOBID,
> > +			      NULL);
> > +	if (rc) {
> > +		CERROR("%s: fail to init llog handle: rc = %d\n",
> > +		       obd->obd_name, rc);
> > +		goto err_out;
> > +	}
> > +
> > +	rc = llog_cat_process(NULL, llh, chlg_read_cat_process_cb, crs, 0, 0);
> > +	if (rc < 0) {
> > +		CERROR("%s: fail to process llog: rc = %d\n",
> > +		       obd->obd_name, rc);
> > +		goto err_out;
> > +	}
> > +
> > +err_out:
> > +	crs->crs_err = true;
> > +	wake_up_all(&crs->crs_waitq_cons);
> > +
> > +	if (llh)
> > +		llog_cat_close(NULL, llh);
> > +
> > +	if (ctx)
> > +		llog_ctxt_put(ctx);
> > +
> > +	wait_event_idle(crs->crs_waitq_prod, crs->crs_closed);
> > +	crs_free(crs);
> > +	return rc;
> > +}
> > +
> > +/**
> > + * Read handler, dequeues records from the chlg_reader_state if any.
> > + * No partial records are copied to userland so this function can return less
> > + * data than required (short read).
> > + *
> > + * @param[in]   file   File pointer to the character device.
> > + * @param[out]  buff   Userland buffer where to copy the records.
> > + * @param[in]   count  Userland buffer size.
> > + * @param[out]  ppos   File position, updated with the index number of the next
> > + *		       record to read.
> > + * @return number of copied bytes on success, negated error code on failure.
> > + */
> > +static ssize_t chlg_read(struct file *file, char __user *buff, size_t count,
> > +			 loff_t *ppos)
> > +{
> > +	struct chlg_reader_state *crs = file->private_data;
> > +	struct chlg_rec_entry *rec;
> > +	struct chlg_rec_entry *tmp;
> > +	ssize_t  written_total = 0;
> > +	LIST_HEAD(consumed);
> > +
> > +	if (file->f_flags & O_NONBLOCK && crs->crs_rec_count == 0)
> > +		return -EAGAIN;
> > +
> > +	wait_event_idle(crs->crs_waitq_cons,
> > +			crs->crs_rec_count > 0 || crs->crs_eof || crs->crs_err);
> > +
> > +	mutex_lock(&crs->crs_lock);
> > +	list_for_each_entry_safe(rec, tmp, &crs->crs_rec_queue, enq_linkage) {
> > +		if (written_total + rec->enq_length > count)
> > +			break;
> > +
> > +		if (copy_to_user(buff, rec->enq_record, rec->enq_length)) {
> > +			if (written_total == 0)
> > +				written_total = -EFAULT;
> > +			break;
> > +		}
> > +
> > +		buff += rec->enq_length;
> > +		written_total += rec->enq_length;
> > +
> > +		crs->crs_rec_count--;
> > +		list_move_tail(&rec->enq_linkage, &consumed);
> > +
> > +		crs->crs_start_offset = rec->enq_record->cr_index + 1;
> > +	}
> > +	mutex_unlock(&crs->crs_lock);
> > +
> > +	if (written_total > 0)
> > +		wake_up_all(&crs->crs_waitq_prod);
> > +
> > +	list_for_each_entry_safe(rec, tmp, &consumed, enq_linkage)
> > +		enq_record_delete(rec);
> > +
> > +	*ppos = crs->crs_start_offset;
> > +
> > +	return written_total;
> > +}
> > +
> > +/**
> > + * Jump to a given record index. Helper for chlg_llseek().
> > + *
> > + * @param[in,out]  crs     Internal reader state.
> > + * @param[in]      offset  Desired offset (index record).
> > + * @return 0 on success, negated error code on failure.
> > + */
> > +static int chlg_set_start_offset(struct chlg_reader_state *crs, u64 offset)
> > +{
> > +	struct chlg_rec_entry *rec;
> > +	struct chlg_rec_entry *tmp;
> > +
> > +	mutex_lock(&crs->crs_lock);
> > +	if (offset < crs->crs_start_offset) {
> > +		mutex_unlock(&crs->crs_lock);
> > +		return -ERANGE;
> > +	}
> > +
> > +	crs->crs_start_offset = offset;
> > +	list_for_each_entry_safe(rec, tmp, &crs->crs_rec_queue, enq_linkage) {
> > +		struct changelog_rec *cr = rec->enq_record;
> > +
> > +		if (cr->cr_index >= crs->crs_start_offset)
> > +			break;
> > +
> > +		crs->crs_rec_count--;
> > +		enq_record_delete(rec);
> > +	}
> > +
> > +	mutex_unlock(&crs->crs_lock);
> > +	wake_up_all(&crs->crs_waitq_prod);
> > +	return 0;
> > +}
> > +
> > +/**
> > + * Move read pointer to a certain record index, encoded as an offset.
> > + *
> > + * @param[in,out] file   File pointer to the changelog character device
> > + * @param[in]	  off    Offset to skip, actually a record index, not byte count
> > + * @param[in]	  whence Relative/Absolute interpretation of the offset
> > + * @return the resulting position on success or negated error code on failure.
> > + */
> > +static loff_t chlg_llseek(struct file *file, loff_t off, int whence)
> > +{
> > +	struct chlg_reader_state *crs = file->private_data;
> > +	loff_t pos;
> > +	int rc;
> > +
> > +	switch (whence) {
> > +	case SEEK_SET:
> > +		pos = off;
> > +		break;
> > +	case SEEK_CUR:
> > +		pos = file->f_pos + off;
> > +		break;
> > +	case SEEK_END:
> > +	default:
> > +		return -EINVAL;
> > +	}
> > +
> > +	/* We cannot go backward */
> > +	if (pos < file->f_pos)
> > +		return -EINVAL;
> > +
> > +	rc = chlg_set_start_offset(crs, pos);
> > +	if (rc != 0)
> > +		return rc;
> > +
> > +	file->f_pos = pos;
> > +	return pos;
> > +}
> > +
> > +/**
> > + * Clear record range for a given changelog reader.
> > + *
> > + * @param[in]  crs     Current internal state.
> > + * @param[in]  reader  Changelog reader ID (cl1, cl2...)
> > + * @param[in]  record  Record index up which to clear
> > + * @return 0 on success, negated error code on failure.
> > + */
> > +static int chlg_clear(struct chlg_reader_state *crs, u32 reader, u64 record)
> > +{
> > +	struct obd_device *obd = crs->crs_obd;
> > +	struct changelog_setinfo cs  = {
> > +		.cs_recno = record,
> > +		.cs_id    = reader
> > +	};
> > +
> > +	return obd_set_info_async(NULL, obd->obd_self_export,
> > +				  strlen(KEY_CHANGELOG_CLEAR),
> > +				  KEY_CHANGELOG_CLEAR, sizeof(cs), &cs, NULL);
> > +}
> > +
> > +/** Maximum changelog control command size */
> > +#define CHLG_CONTROL_CMD_MAX	64
> > +
> > +/**
> > + * Handle writes() into the changelog character device. Write() can be used
> > + * to request special control operations.
> > + *
> > + * @param[in]  file  File pointer to the changelog character device
> > + * @param[in]  buff  User supplied data (written data)
> > + * @param[in]  count Number of written bytes
> > + * @param[in]  off   (unused)
> > + * @return number of written bytes on success, negated error code on failure.
> > + */
> > +static ssize_t chlg_write(struct file *file, const char __user *buff,
> > +			  size_t count, loff_t *off)
> > +{
> > +	struct chlg_reader_state *crs = file->private_data;
> > +	char *kbuf;
> > +	u64 record;
> > +	u32 reader;
> > +	int rc = 0;
> > +
> > +	if (count > CHLG_CONTROL_CMD_MAX)
> > +		return -EINVAL;
> > +
> > +	kbuf = kzalloc(CHLG_CONTROL_CMD_MAX, GFP_KERNEL);
> > +	if (!kbuf)
> > +		return -ENOMEM;
> > +
> > +	if (copy_from_user(kbuf, buff, count)) {
> > +		rc = -EFAULT;
> > +		goto out_kbuf;
> > +	}
> > +
> > +	kbuf[CHLG_CONTROL_CMD_MAX - 1] = '\0';
> > +
> > +	if (sscanf(kbuf, "clear:cl%u:%llu", &reader, &record) == 2)
> > +		rc = chlg_clear(crs, reader, record);
> > +	else
> > +		rc = -EINVAL;
> > +
> > +out_kbuf:
> > +	kfree(kbuf);
> > +	return rc < 0 ? rc : count;
> > +}
> > +
> > +/**
> > + * Find the OBD device associated to a changelog character device.
> > + * @param[in]  cdev  character device instance descriptor
> > + * @return corresponding OBD device or NULL if none was found.
> > + */
> > +static struct obd_device *chlg_obd_get(dev_t cdev)
> > +{
> > +	int minor = MINOR(cdev);
> > +	struct obd_device *obd = NULL;
> > +	struct chlg_registered_dev *curr;
> > +
> > +	mutex_lock(&chlg_registered_dev_lock);
> > +	list_for_each_entry(curr, &chlg_registered_devices, ced_link) {
> > +		if (curr->ced_misc.minor == minor) {
> > +			/* take the first available OBD device attached */
> > +			obd = list_first_entry(&curr->ced_obds,
> > +					       struct obd_device,
> > +					       u.cli.cl_chg_dev_linkage);
> > +			break;
> > +		}
> > +	}
> > +	mutex_unlock(&chlg_registered_dev_lock);
> > +	return obd;
> > +}
> > +
> > +/**
> > + * Open handler, initialize internal CRS state and spawn prefetch thread if
> > + * needed.
> > + * @param[in]  inode  Inode struct for the open character device.
> > + * @param[in]  file   Corresponding file pointer.
> > + * @return 0 on success, negated error code on failure.
> > + */
> > +static int chlg_open(struct inode *inode, struct file *file)
> > +{
> > +	struct chlg_reader_state *crs;
> > +	struct obd_device *obd = chlg_obd_get(inode->i_rdev);
> > +	struct task_struct *task;
> > +	int rc;
> > +
> > +	if (!obd)
> > +		return -ENODEV;
> > +
> > +	crs = kzalloc(sizeof(*crs), GFP_KERNEL);
> > +	if (!crs)
> > +		return -ENOMEM;
> > +
> > +	crs->crs_obd = obd;
> > +	crs->crs_err = false;
> > +	crs->crs_eof = false;
> > +	crs->crs_closed = false;
> > +
> > +	mutex_init(&crs->crs_lock);
> > +	INIT_LIST_HEAD(&crs->crs_rec_queue);
> > +	init_waitqueue_head(&crs->crs_waitq_prod);
> > +	init_waitqueue_head(&crs->crs_waitq_cons);
> > +
> > +	if (file->f_mode & FMODE_READ) {
> > +		task = kthread_run(chlg_load, crs, "chlg_load_thread");
> > +		if (IS_ERR(task)) {
> > +			rc = PTR_ERR(task);
> > +			CERROR("%s: cannot start changelog thread: rc = %d\n",
> > +			       obd->obd_name, rc);
> > +			goto err_crs;
> > +		}
> > +	}
> > +
> > +	file->private_data = crs;
> > +	return 0;
> > +
> > +err_crs:
> > +	kfree(crs);
> > +	return rc;
> > +}
> > +
> > +/**
> > + * Close handler, release resources.
> > + *
> > + * @param[in]  inode  Inode struct for the open character device.
> > + * @param[in]  file   Corresponding file pointer.
> > + * @return 0 on success, negated error code on failure.
> > + */
> > +static int chlg_release(struct inode *inode, struct file *file)
> > +{
> > +	struct chlg_reader_state *crs = file->private_data;
> > +
> > +	if (file->f_mode & FMODE_READ) {
> > +		crs->crs_closed = true;
> > +		wake_up_all(&crs->crs_waitq_prod);
> > +	} else {
> > +		/* No producer thread, release resource ourselves */
> > +		crs_free(crs);
> > +	}
> > +	return 0;
> > +}
> > +
> > +/**
> > + * Poll handler, indicates whether the device is readable (new records) and
> > + * writable (always).
> > + *
> > + * @param[in]  file   Device file pointer.
> > + * @param[in]  wait   (opaque)
> > + * @return combination of the poll status flags.
> > + */
> > +static unsigned int chlg_poll(struct file *file, poll_table *wait)
> > +{
> > +	struct chlg_reader_state *crs  = file->private_data;
> > +	unsigned int mask = 0;
> > +
> > +	mutex_lock(&crs->crs_lock);
> > +	poll_wait(file, &crs->crs_waitq_cons, wait);
> > +	if (crs->crs_rec_count > 0)
> > +		mask |= POLLIN | POLLRDNORM;
> > +	if (crs->crs_err)
> > +		mask |= POLLERR;
> > +	if (crs->crs_eof)
> > +		mask |= POLLHUP;
> > +	mutex_unlock(&crs->crs_lock);
> > +	return mask;
> > +}
> > +
> > +static const struct file_operations chlg_fops = {
> > +	.owner		= THIS_MODULE,
> > +	.llseek		= chlg_llseek,
> > +	.read		= chlg_read,
> > +	.write		= chlg_write,
> > +	.open		= chlg_open,
> > +	.release	= chlg_release,
> > +	.poll		= chlg_poll,
> > +};
> > +
> > +/**
> > + * This uses obd_name of the form: "testfs-MDT0000-mdc-ffff88006501600"
> > + * and returns a name of the form: "changelog-testfs-MDT0000".
> > + */
> > +static void get_chlg_name(char *name, size_t name_len, struct obd_device *obd)
> > +{
> > +	int i;
> > +
> > +	snprintf(name, name_len, "changelog-%s", obd->obd_name);
> > +
> > +	/* Find the 2nd '-' from the end and truncate on it */
> > +	for (i = 0; i < 2; i++) {
> > +		char *p = strrchr(name, '-');
> > +
> > +		if (!p)
> > +			return;
> > +		*p = '\0';
> > +	}
> > +}
> > +
> > +/**
> > + * Find a changelog character device by name.
> > + * All devices registered during MDC setup are listed in a global list with
> > + * their names attached.
> > + */
> > +static struct chlg_registered_dev *
> > +chlg_registered_dev_find_by_name(const char *name)
> > +{
> > +	struct chlg_registered_dev *dit;
> > +
> > +	list_for_each_entry(dit, &chlg_registered_devices, ced_link)
> > +		if (strcmp(name, dit->ced_name) == 0)
> > +			return dit;
> > +	return NULL;
> > +}
> > +
> > +/**
> > + * Find chlg_registered_dev structure for a given OBD device.
> > + * This is bad O(n^2) but for each filesystem:
> > + *   - N is # of MDTs times # of mount points
> > + *   - this only runs at shutdown
> > + */
> > +static struct chlg_registered_dev *
> > +chlg_registered_dev_find_by_obd(const struct obd_device *obd)
> > +{
> > +	struct chlg_registered_dev *dit;
> > +	struct obd_device *oit;
> > +
> > +	list_for_each_entry(dit, &chlg_registered_devices, ced_link)
> > +		list_for_each_entry(oit, &dit->ced_obds,
> > +				    u.cli.cl_chg_dev_linkage)
> > +			if (oit == obd)
> > +				return dit;
> > +	return NULL;
> > +}
> > +
> > +/**
> > + * Changelog character device initialization.
> > + * Register a misc character device with a dynamic minor number, under a name
> > + * of the form: 'changelog-fsname-MDTxxxx'. Reference this OBD device with it.
> > + *
> > + * @param[in] obd  This MDC obd_device.
> > + * @return 0 on success, negated error code on failure.
> > + */
> > +int mdc_changelog_cdev_init(struct obd_device *obd)
> > +{
> > +	struct chlg_registered_dev *exist;
> > +	struct chlg_registered_dev *entry;
> > +	int rc;
> > +
> > +	entry = kzalloc(sizeof(*entry), GFP_KERNEL);
> > +	if (!entry)
> > +		return -ENOMEM;
> > +
> > +	get_chlg_name(entry->ced_name, sizeof(entry->ced_name), obd);
> > +
> > +	entry->ced_misc.minor = MISC_DYNAMIC_MINOR;
> > +	entry->ced_misc.name  = entry->ced_name;
> > +	entry->ced_misc.fops  = &chlg_fops;
> > +
> > +	kref_init(&entry->ced_refs);
> > +	INIT_LIST_HEAD(&entry->ced_obds);
> > +	INIT_LIST_HEAD(&entry->ced_link);
> > +
> > +	mutex_lock(&chlg_registered_dev_lock);
> > +	exist = chlg_registered_dev_find_by_name(entry->ced_name);
> > +	if (exist) {
> > +		kref_get(&exist->ced_refs);
> > +		list_add_tail(&obd->u.cli.cl_chg_dev_linkage, &exist->ced_obds);
> > +		rc = 0;
> > +		goto out_unlock;
> > +	}
> > +
> > +	/* Register new character device */
> > +	rc = misc_register(&entry->ced_misc);
> > +	if (rc != 0)
> > +		goto out_unlock;
> > +
> > +	list_add_tail(&obd->u.cli.cl_chg_dev_linkage, &entry->ced_obds);
> > +	list_add_tail(&entry->ced_link, &chlg_registered_devices);
> > +
> > +	entry = NULL;	/* prevent it from being freed below */
> > +
> > +out_unlock:
> > +	mutex_unlock(&chlg_registered_dev_lock);
> > +	kfree(entry);
> > +	return rc;
> > +}
> > +
> > +/**
> > + * Deregister a changelog character device whose refcount has reached zero.
> > + */
> > +static void chlg_dev_clear(struct kref *kref)
> > +{
> > +	struct chlg_registered_dev *entry = container_of(kref,
> > +							 struct chlg_registered_dev,
> > +							 ced_refs);
> > +	list_del(&entry->ced_link);
> > +	misc_deregister(&entry->ced_misc);
> > +	kfree(entry);
> > +}
> > +
> > +/**
> > + * Release OBD, decrease reference count of the corresponding changelog device.
> > + */
> > +void mdc_changelog_cdev_finish(struct obd_device *obd)
> > +{
> > +	struct chlg_registered_dev *dev = chlg_registered_dev_find_by_obd(obd);
> > +
> > +	mutex_lock(&chlg_registered_dev_lock);
> > +	list_del_init(&obd->u.cli.cl_chg_dev_linkage);
> > +	kref_put(&dev->ced_refs, chlg_dev_clear);
> > +	mutex_unlock(&chlg_registered_dev_lock);
> > +}
> > diff --git a/drivers/staging/lustre/lustre/mdc/mdc_internal.h b/drivers/staging/lustre/lustre/mdc/mdc_internal.h
> > index 941a896..6da9046 100644
> > --- a/drivers/staging/lustre/lustre/mdc/mdc_internal.h
> > +++ b/drivers/staging/lustre/lustre/mdc/mdc_internal.h
> > @@ -129,6 +129,10 @@ enum ldlm_mode mdc_lock_match(struct obd_export *exp, __u64 flags,
> >  			      enum ldlm_mode mode,
> >  			      struct lustre_handle *lockh);
> >  
> > +int mdc_changelog_cdev_init(struct obd_device *obd);
> > +
> > +void mdc_changelog_cdev_finish(struct obd_device *obd);
> > +
> >  static inline int mdc_prep_elc_req(struct obd_export *exp,
> >  				   struct ptlrpc_request *req, int opc,
> >  				   struct list_head *cancels, int count)
> > diff --git a/drivers/staging/lustre/lustre/mdc/mdc_request.c b/drivers/staging/lustre/lustre/mdc/mdc_request.c
> > index 8f8e3d2..3692b1c 100644
> > --- a/drivers/staging/lustre/lustre/mdc/mdc_request.c
> > +++ b/drivers/staging/lustre/lustre/mdc/mdc_request.c
> > @@ -35,7 +35,6 @@
> >  
> >  # include <linux/module.h>
> >  # include <linux/pagemap.h>
> > -# include <linux/miscdevice.h>
> >  # include <linux/init.h>
> >  # include <linux/utsname.h>
> >  # include <linux/file.h>
> > @@ -1810,174 +1809,6 @@ static int mdc_ioc_hsm_request(struct obd_export *exp,
> >  	return rc;
> >  }
> >  
> > -static struct kuc_hdr *changelog_kuc_hdr(char *buf, size_t len, u32 flags)
> > -{
> > -	struct kuc_hdr *lh = (struct kuc_hdr *)buf;
> > -
> > -	LASSERT(len <= KUC_CHANGELOG_MSG_MAXSIZE);
> > -
> > -	lh->kuc_magic = KUC_MAGIC;
> > -	lh->kuc_transport = KUC_TRANSPORT_CHANGELOG;
> > -	lh->kuc_flags = flags;
> > -	lh->kuc_msgtype = CL_RECORD;
> > -	lh->kuc_msglen = len;
> > -	return lh;
> > -}
> > -
> > -struct changelog_show {
> > -	__u64		cs_startrec;
> > -	enum changelog_send_flag	cs_flags;
> > -	struct file	*cs_fp;
> > -	char		*cs_buf;
> > -	struct obd_device *cs_obd;
> > -};
> > -
> > -static inline char *cs_obd_name(struct changelog_show *cs)
> > -{
> > -	return cs->cs_obd->obd_name;
> > -}
> > -
> > -static int changelog_kkuc_cb(const struct lu_env *env, struct llog_handle *llh,
> > -			     struct llog_rec_hdr *hdr, void *data)
> > -{
> > -	struct changelog_show *cs = data;
> > -	struct llog_changelog_rec *rec = (struct llog_changelog_rec *)hdr;
> > -	struct kuc_hdr *lh;
> > -	size_t len;
> > -	int rc;
> > -
> > -	if (rec->cr_hdr.lrh_type != CHANGELOG_REC) {
> > -		rc = -EINVAL;
> > -		CERROR("%s: not a changelog rec %x/%d: rc = %d\n",
> > -		       cs_obd_name(cs), rec->cr_hdr.lrh_type,
> > -		       rec->cr.cr_type, rc);
> > -		return rc;
> > -	}
> > -
> > -	if (rec->cr.cr_index < cs->cs_startrec) {
> > -		/* Skip entries earlier than what we are interested in */
> > -		CDEBUG(D_HSM, "rec=%llu start=%llu\n",
> > -		       rec->cr.cr_index, cs->cs_startrec);
> > -		return 0;
> > -	}
> > -
> > -	CDEBUG(D_HSM, "%llu %02d%-5s %llu 0x%x t=" DFID " p=" DFID
> > -		" %.*s\n", rec->cr.cr_index, rec->cr.cr_type,
> > -		changelog_type2str(rec->cr.cr_type), rec->cr.cr_time,
> > -		rec->cr.cr_flags & CLF_FLAGMASK,
> > -		PFID(&rec->cr.cr_tfid), PFID(&rec->cr.cr_pfid),
> > -		rec->cr.cr_namelen, changelog_rec_name(&rec->cr));
> > -
> > -	len = sizeof(*lh) + changelog_rec_size(&rec->cr) + rec->cr.cr_namelen;
> > -
> > -	/* Set up the message */
> > -	lh = changelog_kuc_hdr(cs->cs_buf, len, cs->cs_flags);
> > -	memcpy(lh + 1, &rec->cr, len - sizeof(*lh));
> > -
> > -	rc = libcfs_kkuc_msg_put(cs->cs_fp, lh);
> > -	CDEBUG(D_HSM, "kucmsg fp %p len %zu rc %d\n", cs->cs_fp, len, rc);
> > -
> > -	return rc;
> > -}
> > -
> > -static int mdc_changelog_send_thread(void *csdata)
> > -{
> > -	enum llog_flag flags = LLOG_F_IS_CAT;
> > -	struct changelog_show *cs = csdata;
> > -	struct llog_ctxt *ctxt = NULL;
> > -	struct llog_handle *llh = NULL;
> > -	struct kuc_hdr *kuch;
> > -	int rc;
> > -
> > -	CDEBUG(D_HSM, "changelog to fp=%p start %llu\n",
> > -	       cs->cs_fp, cs->cs_startrec);
> > -
> > -	cs->cs_buf = kzalloc(KUC_CHANGELOG_MSG_MAXSIZE, GFP_NOFS);
> > -	if (!cs->cs_buf) {
> > -		rc = -ENOMEM;
> > -		goto out;
> > -	}
> > -
> > -	/* Set up the remote catalog handle */
> > -	ctxt = llog_get_context(cs->cs_obd, LLOG_CHANGELOG_REPL_CTXT);
> > -	if (!ctxt) {
> > -		rc = -ENOENT;
> > -		goto out;
> > -	}
> > -	rc = llog_open(NULL, ctxt, &llh, NULL, CHANGELOG_CATALOG,
> > -		       LLOG_OPEN_EXISTS);
> > -	if (rc) {
> > -		CERROR("%s: fail to open changelog catalog: rc = %d\n",
> > -		       cs_obd_name(cs), rc);
> > -		goto out;
> > -	}
> > -
> > -	if (cs->cs_flags & CHANGELOG_FLAG_JOBID)
> > -		flags |= LLOG_F_EXT_JOBID;
> > -
> > -	rc = llog_init_handle(NULL, llh, flags, NULL);
> > -	if (rc) {
> > -		CERROR("llog_init_handle failed %d\n", rc);
> > -		goto out;
> > -	}
> > -
> > -	rc = llog_cat_process(NULL, llh, changelog_kkuc_cb, cs, 0, 0);
> > -
> > -	/* Send EOF no matter what our result */
> > -	kuch = changelog_kuc_hdr(cs->cs_buf, sizeof(*kuch), cs->cs_flags);
> > -	kuch->kuc_msgtype = CL_EOF;
> > -	libcfs_kkuc_msg_put(cs->cs_fp, kuch);
> > -
> > -out:
> > -	fput(cs->cs_fp);
> > -	if (llh)
> > -		llog_cat_close(NULL, llh);
> > -	if (ctxt)
> > -		llog_ctxt_put(ctxt);
> > -	kfree(cs->cs_buf);
> > -	kfree(cs);
> > -	return rc;
> > -}
> > -
> > -static int mdc_ioc_changelog_send(struct obd_device *obd,
> > -				  struct ioc_changelog *icc)
> > -{
> > -	struct changelog_show *cs;
> > -	struct task_struct *task;
> > -	int rc;
> > -
> > -	/* Freed in mdc_changelog_send_thread */
> > -	cs = kzalloc(sizeof(*cs), GFP_NOFS);
> > -	if (!cs)
> > -		return -ENOMEM;
> > -
> > -	cs->cs_obd = obd;
> > -	cs->cs_startrec = icc->icc_recno;
> > -	/* matching fput in mdc_changelog_send_thread */
> > -	cs->cs_fp = fget(icc->icc_id);
> > -	cs->cs_flags = icc->icc_flags;
> > -
> > -	/*
> > -	 * New thread because we should return to user app before
> > -	 * writing into our pipe
> > -	 */
> > -	task = kthread_run(mdc_changelog_send_thread, cs,
> > -			   "mdc_clg_send_thread");
> > -	if (IS_ERR(task)) {
> > -		rc = PTR_ERR(task);
> > -		CERROR("%s: can't start changelog thread: rc = %d\n",
> > -		       cs_obd_name(cs), rc);
> > -		kfree(cs);
> > -	} else {
> > -		rc = 0;
> > -		CDEBUG(D_HSM, "%s: started changelog thread\n",
> > -		       cs_obd_name(cs));
> > -	}
> > -
> > -	CERROR("Failed to start changelog thread: %d\n", rc);
> > -	return rc;
> > -}
> > -
> >  static int mdc_ioc_hsm_ct_start(struct obd_export *exp,
> >  				struct lustre_kernelcomm *lk);
> >  
> > @@ -2087,21 +1918,6 @@ static int mdc_iocontrol(unsigned int cmd, struct obd_export *exp, int len,
> >  		return -EINVAL;
> >  	}
> >  	switch (cmd) {
> > -	case OBD_IOC_CHANGELOG_SEND:
> > -		rc = mdc_ioc_changelog_send(obd, karg);
> > -		goto out;
> > -	case OBD_IOC_CHANGELOG_CLEAR: {
> > -		struct ioc_changelog *icc = karg;
> > -		struct changelog_setinfo cs = {
> > -			.cs_recno = icc->icc_recno,
> > -			.cs_id = icc->icc_id
> > -		};
> > -
> > -		rc = obd_set_info_async(NULL, exp, strlen(KEY_CHANGELOG_CLEAR),
> > -					KEY_CHANGELOG_CLEAR, sizeof(cs), &cs,
> > -					NULL);
> > -		goto out;
> > -	}
> >  	case OBD_IOC_FID2PATH:
> >  		rc = mdc_ioc_fid2path(exp, karg);
> >  		goto out;
> > @@ -2670,12 +2486,22 @@ static int mdc_setup(struct obd_device *obd, struct lustre_cfg *cfg)
> >  
> >  	rc = mdc_llog_init(obd);
> >  	if (rc) {
> > -		CERROR("failed to setup llogging subsystems\n");
> > +		CERROR("%s: failed to setup llogging subsystems: rc = %d\n",
> > +		       obd->obd_name, rc);
> >  		goto err_llog_cleanup;
> >  	}
> >  
> > +	rc = mdc_changelog_cdev_init(obd);
> > +	if (rc) {
> > +		CERROR("%s: failed to setup changelog char device: rc = %d\n",
> > +		       obd->obd_name, rc);
> > +		goto err_changelog_cleanup;
> > +	}
> > +
> >  	return 0;
> >  
> > +err_changelog_cleanup:
> > +	mdc_llog_finish(obd);
> >  err_llog_cleanup:
> >  	ldebugfs_free_md_stats(obd);
> >  	ptlrpc_lprocfs_unregister_obd(obd);
> > @@ -2714,6 +2540,8 @@ static int mdc_precleanup(struct obd_device *obd)
> >  	if (obd->obd_type->typ_refcnt <= 1)
> >  		libcfs_kkuc_group_rem(0, KUC_GRP_HSM);
> >  
> > +	mdc_changelog_cdev_finish(obd);
> > +
> >  	obd_cleanup_client_import(obd);
> >  	ptlrpc_lprocfs_unregister_obd(obd);
> >  	lprocfs_obd_cleanup(obd);
> > -- 
> > 1.8.3.1
> 

From jsimmons at infradead.org  Sun Nov  4 21:34:04 2018
From: jsimmons at infradead.org (James Simmons)
Date: Sun, 4 Nov 2018 21:34:04 +0000 (GMT)
Subject: [lustre-devel] [PATCH v2] lustre: mdc: fix possible deadlock in
 chlg_open()
In-Reply-To: <875zxh3elu.fsf@notabene.neil.brown.name>
References: <87bm7a3nue.fsf@notabene.neil.brown.name>
 <7db27e7d-e685-af8f-80d8-891a0d8db4d5@cea.fr>
 <875zxh3elu.fsf@notabene.neil.brown.name>
Message-ID: <alpine.LFD.2.21.1811042131490.27308@casper.infradead.org>


> Lockdep reports a possible deadlock between chlg_open() and
> mdc_changelog_cdev_init()
> 
> mdc_changelog_cdev_init() takes chlg_registered_dev_lock and then
> calls misc_register() which takes misc_mtx.
> chlg_open() is called while misc_mtx is held, and tries to take
> chlg_registered_dev_lock.
> If these two functions race, a deadlock can occur as each thread will
> hold one of the locks while trying to take the other.
> 
> chlg_open() does not need to take a lock.  It only uses the
> lock to stablize a list while looking for the matching
> chlg_registered_dev, and this can be found directly by examining
> file->private_data.
> 
> So remove chlg_obd_get(), and use file->private_data to find the
> obd_device.
> Also ensure the device is fully initialized before calling
> misc_register().  This means setting up some list linkage before the
> call, and tearing it down if there is an error.

I have been testing this but I'm no HSM expert. I pushed this patch
to OpenSFS branch as well.

https://jira.whamcloud.com/browse/LU-11617
https://review.whamcloud.com/#/c/33572/

Reviewed-by: James Simmons <jsimmons at infradead.org>

> Signed-off-by: NeilBrown <neilb at suse.com>
> ---
> 
> This is the revised version with the problem identified by Quentin
> fixed.
> 
>  drivers/staging/lustre/lustre/mdc/mdc_changelog.c | 46 +++++++----------------
>  1 file changed, 14 insertions(+), 32 deletions(-)
> 
> diff --git a/drivers/staging/lustre/lustre/mdc/mdc_changelog.c b/drivers/staging/lustre/lustre/mdc/mdc_changelog.c
> index d83507cbf95c..af29ea73c48a 100644
> --- a/drivers/staging/lustre/lustre/mdc/mdc_changelog.c
> +++ b/drivers/staging/lustre/lustre/mdc/mdc_changelog.c
> @@ -444,31 +444,6 @@ static ssize_t chlg_write(struct file *file, const char __user *buff,
>  	return rc < 0 ? rc : count;
>  }
>  
> -/**
> - * Find the OBD device associated to a changelog character device.
> - * @param[in]  cdev  character device instance descriptor
> - * @return corresponding OBD device or NULL if none was found.
> - */
> -static struct obd_device *chlg_obd_get(dev_t cdev)
> -{
> -	int minor = MINOR(cdev);
> -	struct obd_device *obd = NULL;
> -	struct chlg_registered_dev *curr;
> -
> -	mutex_lock(&chlg_registered_dev_lock);
> -	list_for_each_entry(curr, &chlg_registered_devices, ced_link) {
> -		if (curr->ced_misc.minor == minor) {
> -			/* take the first available OBD device attached */
> -			obd = list_first_entry(&curr->ced_obds,
> -					       struct obd_device,
> -					       u.cli.cl_chg_dev_linkage);
> -			break;
> -		}
> -	}
> -	mutex_unlock(&chlg_registered_dev_lock);
> -	return obd;
> -}
> -
>  /**
>   * Open handler, initialize internal CRS state and spawn prefetch thread if
>   * needed.
> @@ -479,12 +454,16 @@ static struct obd_device *chlg_obd_get(dev_t cdev)
>  static int chlg_open(struct inode *inode, struct file *file)
>  {
>  	struct chlg_reader_state *crs;
> -	struct obd_device *obd = chlg_obd_get(inode->i_rdev);
> +	struct miscdevice *misc = file->private_data;
> +	struct chlg_registered_dev *dev;
> +	struct obd_device *obd;
>  	struct task_struct *task;
>  	int rc;
>  
> -	if (!obd)
> -		return -ENODEV;
> +	dev = container_of(misc, struct chlg_registered_dev, ced_misc);
> +	obd = list_first_entry(&dev->ced_obds,
> +			       struct obd_device,
> +			       u.cli.cl_chg_dev_linkage);
>  
>  	crs = kzalloc(sizeof(*crs), GFP_KERNEL);
>  	if (!crs)
> @@ -669,13 +648,16 @@ int mdc_changelog_cdev_init(struct obd_device *obd)
>  		goto out_unlock;
>  	}
>  
> +	list_add_tail(&obd->u.cli.cl_chg_dev_linkage, &entry->ced_obds);
> +	list_add_tail(&entry->ced_link, &chlg_registered_devices);
> +
>  	/* Register new character device */
>  	rc = misc_register(&entry->ced_misc);
> -	if (rc != 0)
> +	if (rc != 0) {
> +		list_del_init(&obd->u.cli.cl_chg_dev_linkage);
> +		list_del(&entry->ced_link);
>  		goto out_unlock;
> -
> -	list_add_tail(&obd->u.cli.cl_chg_dev_linkage, &entry->ced_obds);
> -	list_add_tail(&entry->ced_link, &chlg_registered_devices);
> +	}
>  
>  	entry = NULL;	/* prevent it from being freed below */
>  
> -- 
> 2.14.0.rc0.dirty
> 
> 

From jsimmons at infradead.org  Sun Nov  4 21:35:30 2018
From: jsimmons at infradead.org (James Simmons)
Date: Sun, 4 Nov 2018 21:35:30 +0000 (GMT)
Subject: [lustre-devel] [PATCH 15/28] lustre: llite: fix for stat under
 kthread and X86_X32
In-Reply-To: <87h8he8wfq.fsf@notabene.neil.brown.name>
References: <1539543498-29105-1-git-send-email-jsimmons@infradead.org>
 <1539543498-29105-16-git-send-email-jsimmons@infradead.org>
 <87ftx4c9e6.fsf@notabene.neil.brown.name>
 <87h8he8wfq.fsf@notabene.neil.brown.name>
Message-ID: <alpine.LFD.2.21.1811042134360.27308@casper.infradead.org>


> On Thu, Oct 18 2018, NeilBrown wrote:
> 
> > On Sun, Oct 14 2018, James Simmons wrote:
> >
> >> From: Frank Zago <fzago at cray.com>
> >>
> >> Under the following conditions, ll_getattr will flatten the inode
> >> number when it shouldn't:
> >>
> >>  - the X86_X32 architecture is defined CONFIG_X86_X32, and not even
> >>    used,
> >>  - ll_getattr is called from a kernel thread (though vfs_getattr for
> >>    instance.)
> >>
> >> This has the result that inode numbers are different whether the same
> >> file is stat'ed from a kernel thread, or from a syscall. For instance,
> >> 4198401 vs. 144115205272502273.
> >>
> >> ll_getattr calls ll_need_32bit_api to determine whether the task is 32
> >> bits. When the combination is kthread+X86_X32, that function returns
> >> that the task is 32 bits, which is incorrect, as the kernel is 64
> >> bits.
> >>
> >> The solution is to check whether the call is from a kernel thread
> >> (which is 64 bits) and act consequently.
> >>
> >> Signed-off-by: Frank Zago <fzago at cray.com>
> >> WC-bug-id: https://jira.whamcloud.com/browse/LU-9468
> >> Reviewed-on: https://review.whamcloud.com/26992
> >> Reviewed-by: Andreas Dilger <adilger at whamcloud.com>
> >> Reviewed-by: Dmitry Eremin <dmitry.eremin at intel.com>
> >> Signed-off-by: James Simmons <jsimmons at infradead.org>
> >> ---
> >>  drivers/staging/lustre/lustre/llite/dir.c          |  6 +++---
> >>  drivers/staging/lustre/lustre/llite/lcommon_cl.c   |  2 +-
> >>  .../staging/lustre/lustre/llite/llite_internal.h   | 22 +++++++++++++++++-----
> >>  3 files changed, 21 insertions(+), 9 deletions(-)
> >>
> >> diff --git a/drivers/staging/lustre/lustre/llite/dir.c b/drivers/staging/lustre/lustre/llite/dir.c
> >> index 231b351..19c5e9c 100644
> >> --- a/drivers/staging/lustre/lustre/llite/dir.c
> >> +++ b/drivers/staging/lustre/lustre/llite/dir.c
> >> @@ -202,7 +202,7 @@ int ll_dir_read(struct inode *inode, __u64 *ppos, struct md_op_data *op_data,
> >>  {
> >>  	struct ll_sb_info    *sbi	= ll_i2sbi(inode);
> >>  	__u64		   pos		= *ppos;
> >> -	int		   is_api32 = ll_need_32bit_api(sbi);
> >> +	bool is_api32 = ll_need_32bit_api(sbi);
> >>  	int		   is_hash64 = sbi->ll_flags & LL_SBI_64BIT_HASH;
> >>  	struct page	  *page;
> >>  	bool		   done = false;
> >> @@ -296,7 +296,7 @@ static int ll_readdir(struct file *filp, struct dir_context *ctx)
> >>  	struct ll_sb_info	*sbi	= ll_i2sbi(inode);
> >>  	__u64 pos = lfd ? lfd->lfd_pos : 0;
> >>  	int			hash64	= sbi->ll_flags & LL_SBI_64BIT_HASH;
> >> -	int			api32	= ll_need_32bit_api(sbi);
> >> +	bool api32 = ll_need_32bit_api(sbi);
> >>  	struct md_op_data *op_data;
> >>  	int			rc;
> >>  
> >> @@ -1674,7 +1674,7 @@ static loff_t ll_dir_seek(struct file *file, loff_t offset, int origin)
> >>  	struct inode *inode = file->f_mapping->host;
> >>  	struct ll_file_data *fd = LUSTRE_FPRIVATE(file);
> >>  	struct ll_sb_info *sbi = ll_i2sbi(inode);
> >> -	int api32 = ll_need_32bit_api(sbi);
> >> +	bool api32 = ll_need_32bit_api(sbi);
> >>  	loff_t ret = -EINVAL;
> >>  
> >>  	switch (origin) {
> >> diff --git a/drivers/staging/lustre/lustre/llite/lcommon_cl.c b/drivers/staging/lustre/lustre/llite/lcommon_cl.c
> >> index 30f17ea..20a3c74 100644
> >> --- a/drivers/staging/lustre/lustre/llite/lcommon_cl.c
> >> +++ b/drivers/staging/lustre/lustre/llite/lcommon_cl.c
> >> @@ -267,7 +267,7 @@ void cl_inode_fini(struct inode *inode)
> >>  /**
> >>   * build inode number from passed @fid
> >>   */
> >> -__u64 cl_fid_build_ino(const struct lu_fid *fid, int api32)
> >> +u64 cl_fid_build_ino(const struct lu_fid *fid, bool api32)
> >>  {
> >>  	if (BITS_PER_LONG == 32 || api32)
> >>  		return fid_flatten32(fid);
> >> diff --git a/drivers/staging/lustre/lustre/llite/llite_internal.h b/drivers/staging/lustre/lustre/llite/llite_internal.h
> >> index dcb2fed..796a8ae 100644
> >> --- a/drivers/staging/lustre/lustre/llite/llite_internal.h
> >> +++ b/drivers/staging/lustre/lustre/llite/llite_internal.h
> >> @@ -651,13 +651,25 @@ static inline struct inode *ll_info2i(struct ll_inode_info *lli)
> >>  __u32 ll_i2suppgid(struct inode *i);
> >>  void ll_i2gids(__u32 *suppgids, struct inode *i1, struct inode *i2);
> >>  
> >> -static inline int ll_need_32bit_api(struct ll_sb_info *sbi)
> >> +static inline bool ll_need_32bit_api(struct ll_sb_info *sbi)
> >>  {
> >>  #if BITS_PER_LONG == 32
> >> -	return 1;
> >> +	return true;
> >>  #elif defined(CONFIG_COMPAT)
> >> -	return unlikely(in_compat_syscall() ||
> >> -			(sbi->ll_flags & LL_SBI_32BIT_API));
> >> +	if (unlikely(sbi->ll_flags & LL_SBI_32BIT_API))
> >> +		return true;
> >> +
> >> +#ifdef CONFIG_X86_X32
> >> +	/* in_compat_syscall() returns true when called from a kthread
> >> +	 * and CONFIG_X86_X32 is enabled, which is wrong. So check
> >> +	 * whether the caller comes from a syscall (ie. not a kthread)
> >> +	 * before calling in_compat_syscall().
> >> +	 */
> >> +	if (current->flags & PF_KTHREAD)
> >> +		return false;
> >> +#endif
> >
> > This is wrong.  We should fix in_compat_syscall(), not work around it
> > here.
> > (and then there is that fact that the patch changes 'int' to 'bool'
> > without explaining that in the change description).
> >
> > I've sent a query to some relevant people (Cc:ed to James) to ask about
> > fixing in_compat_syscall().
> 
> Upstream say in_compat_syscall() should only be called from a syscall,
> so I have change the patch to the below.
> 
> We probably need to remove this in_compat_syscall() before going
> mainline.

Do you know the progress of this work?
 
> NeilBrown
> 
> -static inline int ll_need_32bit_api(struct ll_sb_info *sbi)
> +static inline bool ll_need_32bit_api(struct ll_sb_info *sbi)
>  {
>  #if BITS_PER_LONG == 32
> -	return 1;
> -#elif defined(CONFIG_COMPAT)
> -	return unlikely(in_compat_syscall() ||
> -			(sbi->ll_flags & LL_SBI_32BIT_API));
> +	return true;
> +#else
> +	if (unlikely(sbi->ll_flags & LL_SBI_32BIT_API))
> +		return true;
> +
> +#if defined(CONFIG_COMPAT)
> +	/* in_compat_syscall() is only meaningful inside a syscall.
> +	 * As this can be called from a kthread (e.g. nfsd), we
> +	 * need to catch that case first.  kthreads never need the
> +	 * 32bit api.
> +	 */
> +	if (current->flags & PF_KTHREAD)
> +		return false;
> +
> +	return unlikely(in_compat_syscall());
>  #else
> -	return unlikely(sbi->ll_flags & LL_SBI_32BIT_API);
> +	return false;
> +#endif
>  #endif
>  }
> 

From jsimmons at infradead.org  Sun Nov  4 21:37:22 2018
From: jsimmons at infradead.org (James Simmons)
Date: Sun, 4 Nov 2018 21:37:22 +0000 (GMT)
Subject: [lustre-devel] Limitations of kernel read ahead
In-Reply-To: <E8183EB7-961B-4FC0-B223-7671E9502F87@whamcloud.com>
References: <alpine.LFD.2.21.1810291532100.21751@casper.infradead.org>
 <62D5855B-8FE1-4157-858C-01B182D759AD@ddn.com>
 <E8183EB7-961B-4FC0-B223-7671E9502F87@whamcloud.com>
Message-ID: <alpine.LFD.2.21.1811042136490.27308@casper.infradead.org>


> One other enhancement that would be good to make for the readahead is to add opportunistic readahead for random access of smaller files (as determined by file size and client RAM) as discussed in LU-11416.  This has shown significant improvement for random access (about 40x speedup when doing random read IOPS on a 1GB file).

Has anyone created patches for the kernels read ahead to get this work 
started?
 
> On Oct 29, 2018, at 20:00, Li Xi <lixi at ddn.com> wrote:
> > 
> > Thank you for summarize this, James!
> > 
> > I think everyone agrees that the current readahead algorithm of Lustre needs to be improved. And evidences show that the readahead algorithm of Linux kernel would not suitable for Lustre either. There are several reasons for this. In general, the readahead algorithm of kernel is designed for local file system with small readahead window. It is single thread, synchronous readahead, only usable for sequential read. Because the read operation of Lustre is has longer latency than local file system, while its bandwidth is typically higher than local file system, we need totally different algorithm for Lustre readahead. The readahead algorithm needs to be 1) asynchronous to hide latency for application 2) multiple threaded to utilize the high bandwidth 3) use big readahead window to align with the big RPC size 4) work for sequential read, stride read and potentially small & random read.
> > 
> > The work of LU-8709 was started with these targets and got pretty good numbers even without detailed tuning. We (the Whamcloud team) would like to rework on it with a goal of merging it in the next releases of Lustre.
> > 
> > Regards,
> > Li Xi
> > 
> > 在 2018/10/30 上午2:06，“James Simmons”<jsimmons at infradead.org> 写入:
> >> 
> >> 
> >>    Currently the lustre client has its own read ahead handling in the CLIO 
> >>    layer. The reason for this is due to some limitations in the read ahead
> >>    code for the linux kernel. Some work to use the kernel's read ahead was 
> >>    attempted for the LU-8964 work but the general work for LU-8964 had other
> >>    issues. Alternative work to LU-8964 has emerged under ticket
> >> 
> >>    https://jira.whamcloud.com/browse/LU-8709
> >> 
> >>    with early code at:
> >> 
> >>    https://review.whamcloud.com/#/c/23552
> >> 
> >>    Also I have included a link to a presentation of this work and it gives
> >>    insight on how lustre does its own read ahead.
> >> 
> >>    https://www.eofs.eu/_media/events/lad16/19_parallel_readahead_framework_li_xi.pdf
> >> 
> >>    Now that this seems to be the targeted work for read ahead the discussion
> >>    has come up about why this new work doesn't use the kernel read ahead 
> >>    again. I wasn't involved in the discussion about the limitations but I 
> >>    have included the people interested in this work so progress can be done
> >>    to imporve the linux kernels version of read ahead.
> >> 
> >> 
> 
> Cheers, Andreas
> ---
> Andreas Dilger
> CTO Whamcloud
> 
> 
> 
> 
> 

From jsimmons at infradead.org  Sun Nov  4 21:40:45 2018
From: jsimmons at infradead.org (James Simmons)
Date: Sun, 4 Nov 2018 21:40:45 +0000 (GMT)
Subject: [lustre-devel] [PATCH RFC] lustre: obd: convert obd_nid_hash to
 rhashtable
In-Reply-To: <87tvl019z8.fsf@notabene.neil.brown.name>
References: <alpine.LFD.2.21.1811011955300.11470@casper.infradead.org>
 <87tvl019z8.fsf@notabene.neil.brown.name>
Message-ID: <alpine.LFD.2.21.1811042138000.27308@casper.infradead.org>


> > This patch converts the struct obd_export obd_nid_hash used by the servr
> > to use rhltables. The reason is that the NID hash can have multiple obd 
> > exports using the same NID key. In the process we gain lockless lookup 
> > which  should improve performance. This should also address the rare 
> > crashes:
> >
> > [<ffffffffc0913388>] ? cfs_hash_bd_from_key+0x38/0xb0 [libcfs]
> > [74844.507416]  [<ffffffffc0913425>] cfs_hash_bd_get+0x25/0x70 [libcfs]
> > [74844.516384]  [<ffffffffc09166d2>] cfs_hash_add+0x52/0x1a0 [libcfs]
> > [74844.525211]  [<ffffffffc0d8a765>] target_handle_connect+0x1fe5/0x29b0 [ptlrpc]
> >
> > Pre 4.8 kernels do not support rhltables so wrappers have been created.
> >
> > ***
> > My testing has been positive so far but I do need to figure out the 
> > debugfs file mapping from the old libcfs hash to rhashtables.
> >
> > ***
> >
> > Signed-off-by: James Simmons <uja.ornl at yahoo.com>
> > ---
> >  libcfs/autoconf/lustre-libcfs.m4         |  21 +++
> >  libcfs/include/libcfs/linux/linux-hash.h |  88 ++++++++++++
> >  lustre/include/lustre_export.h           |   2 +-
> >  lustre/include/obd.h                     |   8 +-
> >  lustre/include/obd_support.h             |   3 -
> >  lustre/ldlm/ldlm_flock.c                 |  18 ++-
> >  lustre/ldlm/ldlm_lib.c                   |  14 +-
> >  lustre/mdt/mdt_lproc.c                   |  21 ++-
> >  lustre/obdclass/genops.c                 |  76 ++++++-----
> >  lustre/obdclass/lprocfs_status_server.c  | 133 +++++++++++-------
> >  lustre/obdclass/obd_config.c             | 225 ++++++++++++++++++-------------
> >  11 files changed, 394 insertions(+), 215 deletions(-)
> >
> > diff --git a/libcfs/autoconf/lustre-libcfs.m4 b/libcfs/autoconf/lustre-libcfs.m4
> > index d437331..147ecb3 100644
> > --- a/libcfs/autoconf/lustre-libcfs.m4
> > +++ b/libcfs/autoconf/lustre-libcfs.m4
> > @@ -761,6 +761,26 @@ LB_CHECK_LINUX_HEADER([linux/stringhash.h], [
> >  ]) # LIBCFS_STRINGHASH
> >  
> >  #
> > +# LIBCFS_RHLTABLE
> > +# Kernel version 4.8 commit ca26893f05e86497a86732768ec53cd38c0819ca
> > +# created the rhlist interface to allow inserting duplicate objects
> > +# into the same table.
> > +#
> > +AC_DEFUN([LIBCFS_RHLTABLE], [
> > +LB_CHECK_COMPILE([if 'struct rhltable' exist],
> > +rhtable, [
> > +	#include <linux/rhashtable.h>
> > +],[
> > +	struct rhltable *hlt = NULL;
> > +
> > +	rhltable_destroy(hlt);
> > +],[
> > +	AC_DEFINE(HAVE_RHLTABLE, 1,
> > +		  [struct rhltable exist])
> > +])
> > +]) # LIBCFS_RHLTABLE
> > +
> > +#
> >  # LIBCFS_STACKTRACE_OPS
> >  #
> >  # Kernel version 4.8 commit c8fe4609827aedc9c4b45de80e7cdc8ccfa8541b
> > @@ -999,6 +1019,7 @@ LIBCFS_STACKTRACE_OPS_ADDRESS_RETURN_INT
> >  LIBCFS_GET_USER_PAGES_6ARG
> >  LIBCFS_STRINGHASH
> >  # 4.8
> > +LIBCFS_RHLTABLE
> >  LIBCFS_STACKTRACE_OPS
> >  # 4.9
> >  LIBCFS_GET_USER_PAGES_GUP_FLAGS
> > diff --git a/libcfs/include/libcfs/linux/linux-hash.h b/libcfs/include/libcfs/linux/linux-hash.h
> > index 1227ec8..0453cd9 100644
> > --- a/libcfs/include/libcfs/linux/linux-hash.h
> > +++ b/libcfs/include/libcfs/linux/linux-hash.h
> > @@ -38,6 +38,94 @@ u64 cfs_hashlen_string(const void *salt, const char *name);
> >  #endif
> >  #endif /* !HAVE_STRINGHASH */
> >  
> > +#ifndef HAVE_RHLTABLE
> > +struct rhlist_head {
> > +	struct rhash_head		rhead;
> > +	struct rhlist_head __rcu	*next;
> > +};
> > +
> > +struct rhltable {
> > +	struct rhashtable ht;
> > +};
> > +
> > +#define rhl_for_each_entry_rcu(tpos, pos, list, member)                 \
> > +	for (pos = list; pos && rht_entry(tpos, pos, member);           \
> > +		pos = rcu_dereference_raw(pos->next))
> > +
> > +static inline int rhltable_init(struct rhltable *hlt, const struct rhashtable_params *params)
> > +{
> > +	return rhashtable_init(&hlt->ht, params);
> > +}
> > +
> > +static inline struct rhlist_head *rhltable_lookup(
> > +	struct rhltable *hlt, const void *key,
> > +	const struct rhashtable_params params)
> > +{
> > +	struct rhashtable *ht = &hlt->ht;
> > +	struct rhashtable_compare_arg arg = {
> > +		.ht = ht,
> > +		.key = key,
> > +	};
> > +	struct bucket_table *tbl;
> > +	struct rhash_head *he;
> > +	unsigned int hash;
> > +
> > +	tbl = rht_dereference_rcu(ht->tbl, ht);
> > +restart:
> > +	hash = rht_key_hashfn(ht, tbl, key, params);
> > +	rht_for_each_rcu(he, tbl, hash) {
> > +		if (params.obj_cmpfn ?
> > +		    params.obj_cmpfn(&arg, rht_obj(ht, he)) :
> > +		    rhashtable_compare(&arg, rht_obj(ht, he)))
> > +			continue;
> > +		return he ? container_of(he, struct rhlist_head, rhead) : NULL;
> > +	}
> > +
> > +	/* Ensure we see any new tables. */
> > +	smp_rmb();
> > +
> > +	tbl = rht_dereference_rcu(tbl->future_tbl, ht);
> > +	if (unlikely(tbl))
> > +		goto restart;
> > +
> > +	return NULL;
> > +}
> > +
> > +static inline int rhltable_insert_key(
> > +	struct rhltable *hlt, const void *key, struct rhlist_head *list,
> > +	const struct rhashtable_params params)
> > +{
> > +	return PTR_ERR(__rhashtable_insert_fast(&hlt->ht, key, &list->rhead,
> > +						params));
> > +}
> > +
> > +static inline int rhltable_remove(
> > +	struct rhltable *hlt, struct rhlist_head *list,
> > +	const struct rhashtable_params params)
> > +{
> > +	return rhashtable_remove_fast(&hlt->ht, &list->rhead, params);
> > +}
> > +
> > +static inline void rhltable_free_and_destroy(struct rhltable *hlt,
> > +					     void (*free_fn)(void *ptr,
> > +							     void *arg),
> > +					     void *arg)
> > +{
> > +	return rhashtable_free_and_destroy(&hlt->ht, free_fn, arg);
> > +}
> > +
> > +static inline void rhltable_destroy(struct rhltable *hlt)
> > +{
> > +	return rhltable_free_and_destroy(hlt, NULL, NULL);
> > +}
> > +
> > +static inline void rhltable_walk_enter(struct rhltable *hlt,
> > +				       struct rhashtable_iter *iter)
> > +{
> > +	rhashtable_walk_init(&hlt->ht, iter);
> > +}
> > +#endif /* !HAVE_RHLTABLE */
> > +
> >  #ifndef HAVE_RHASHTABLE_LOOKUP_GET_INSERT_FAST
> >  /**
> >   * rhashtable_lookup_get_insert_fast - lookup and insert object into hash table
> > diff --git a/lustre/include/lustre_export.h b/lustre/include/lustre_export.h
> > index 5ead593..54887c9 100644
> > --- a/lustre/include/lustre_export.h
> > +++ b/lustre/include/lustre_export.h
> > @@ -209,7 +209,7 @@ struct obd_export {
> >  	/* Unlinked export list */
> >  	struct list_head	exp_stale_list;
> >  	struct hlist_node	exp_uuid_hash;	/** uuid-export hash*/
> > -	struct hlist_node	exp_nid_hash;	/** nid-export hash */
> > +	struct rhlist_head	exp_nid_hash;	/** nid-export hash */
> >  	struct hlist_node	exp_gen_hash;   /** last_rcvd clt gen hash */
> >          /**
> >           * All exports eligible for ping evictor are linked into a list
> > diff --git a/lustre/include/obd.h b/lustre/include/obd.h
> > index 1fcf0a2..8219710 100644
> > --- a/lustre/include/obd.h
> > +++ b/lustre/include/obd.h
> > @@ -639,7 +639,7 @@ struct obd_device {
> >          /* uuid-export hash body */
> >  	struct cfs_hash             *obd_uuid_hash;
> >          /* nid-export hash body */
> > -	struct cfs_hash             *obd_nid_hash;
> > +	struct rhltable			obd_nid_hash;
> >  	/* nid stats body */
> >  	struct cfs_hash             *obd_nid_stats_hash;
> >  	/* client_generation-export hash body */
> > @@ -750,6 +750,12 @@ struct obd_device {
> >  	struct completion		obd_kobj_unregister;
> >  };
> >  
> > +int obd_nid_export_for_each(struct obd_device *obd, lnet_nid_t nid,
> > +			    int cb(struct obd_export *exp, void *data),
> > +			    void *data);
> > +int obd_nid_add(struct obd_device *obd, struct obd_export *exp);
> > +void obd_nid_del(struct obd_device *obd, struct obd_export *exp);
> > +
> >  /* get/set_info keys */
> >  #define KEY_ASYNC               "async"
> >  #define KEY_CHANGELOG_CLEAR     "changelog_clear"
> > diff --git a/lustre/include/obd_support.h b/lustre/include/obd_support.h
> > index e9dd33e..0175cf8 100644
> > --- a/lustre/include/obd_support.h
> > +++ b/lustre/include/obd_support.h
> > @@ -78,9 +78,6 @@ extern char obd_jobid_var[];
> >  #define HASH_UUID_BKT_BITS 5
> >  #define HASH_UUID_CUR_BITS 7
> >  #define HASH_UUID_MAX_BITS 12
> > -#define HASH_NID_BKT_BITS 5
> > -#define HASH_NID_CUR_BITS 7
> > -#define HASH_NID_MAX_BITS 12
> >  #define HASH_NID_STATS_BKT_BITS 5
> >  #define HASH_NID_STATS_CUR_BITS 7
> >  #define HASH_NID_STATS_MAX_BITS 12
> > diff --git a/lustre/ldlm/ldlm_flock.c b/lustre/ldlm/ldlm_flock.c
> > index f848a36..4c1603a 100644
> > --- a/lustre/ldlm/ldlm_flock.c
> > +++ b/lustre/ldlm/ldlm_flock.c
> > @@ -161,20 +161,18 @@ ldlm_flock_destroy(struct ldlm_lock *lock, enum ldlm_mode mode, __u64 flags)
> >   */
> >  
> >  struct ldlm_flock_lookup_cb_data {
> > -	__u64 *bl_owner;
> > +	u64 *bl_owner;
> 
> Arrrggg.   Please don't do this.
> 
> >  	lock = cfs_hash_lookup(exp->exp_flock_hash, cb_data->bl_owner);
> > -	if (lock == NULL)
> > +	if (!lock)
> 
> Or this.
> 
> If you want to fix up this stuff, do it in a separate patch.
> A patch should just do one thing.
> I would actually prefer that the "add compatibilty code so we can use
> rhltables in old kernels" was a separate patch from "use rhltables for
> obd_nid_hash", but at least those two are conceptually related.
> 
> It is much harder to read a patch if I keep having to say to myself "Oh,
> that change is irrelevant here,  I can ignore it".

This is a pretty big patch for email review so I will break it up. I think 
it can be more than 2 and it doesn't matter for buildable from patch to
patch since this us just for review. 

From neilb at suse.com  Sun Nov  4 23:43:25 2018
From: neilb at suse.com (NeilBrown)
Date: Mon, 05 Nov 2018 10:43:25 +1100
Subject: [lustre-devel] [PATCH RFC] lustre: obd: convert obd_nid_hash to
 rhashtable
In-Reply-To: <alpine.LFD.2.21.1811042138000.27308@casper.infradead.org>
References: <alpine.LFD.2.21.1811011955300.11470@casper.infradead.org>
 <87tvl019z8.fsf@notabene.neil.brown.name>
 <alpine.LFD.2.21.1811042138000.27308@casper.infradead.org>
Message-ID: <874lcw1k9u.fsf@notabene.neil.brown.name>

On Sun, Nov 04 2018, James Simmons wrote:

>> > This patch converts the struct obd_export obd_nid_hash used by the servr
>> > to use rhltables. The reason is that the NID hash can have multiple obd 
>> > exports using the same NID key. In the process we gain lockless lookup 
>> > which  should improve performance. This should also address the rare 
>> > crashes:
>> >
>> > [<ffffffffc0913388>] ? cfs_hash_bd_from_key+0x38/0xb0 [libcfs]
>> > [74844.507416]  [<ffffffffc0913425>] cfs_hash_bd_get+0x25/0x70 [libcfs]
>> > [74844.516384]  [<ffffffffc09166d2>] cfs_hash_add+0x52/0x1a0 [libcfs]
>> > [74844.525211]  [<ffffffffc0d8a765>] target_handle_connect+0x1fe5/0x29b0 [ptlrpc]
>> >
>> > Pre 4.8 kernels do not support rhltables so wrappers have been created.
>> >
>> > ***
>> > My testing has been positive so far but I do need to figure out the 
>> > debugfs file mapping from the old libcfs hash to rhashtables.
>> >
>> > ***
>> >
>> > Signed-off-by: James Simmons <uja.ornl at yahoo.com>
>> > ---
>> >  libcfs/autoconf/lustre-libcfs.m4         |  21 +++
>> >  libcfs/include/libcfs/linux/linux-hash.h |  88 ++++++++++++
>> >  lustre/include/lustre_export.h           |   2 +-
>> >  lustre/include/obd.h                     |   8 +-
>> >  lustre/include/obd_support.h             |   3 -
>> >  lustre/ldlm/ldlm_flock.c                 |  18 ++-
>> >  lustre/ldlm/ldlm_lib.c                   |  14 +-
>> >  lustre/mdt/mdt_lproc.c                   |  21 ++-
>> >  lustre/obdclass/genops.c                 |  76 ++++++-----
>> >  lustre/obdclass/lprocfs_status_server.c  | 133 +++++++++++-------
>> >  lustre/obdclass/obd_config.c             | 225 ++++++++++++++++++-------------
>> >  11 files changed, 394 insertions(+), 215 deletions(-)
>> >
>> > diff --git a/libcfs/autoconf/lustre-libcfs.m4 b/libcfs/autoconf/lustre-libcfs.m4
>> > index d437331..147ecb3 100644
>> > --- a/libcfs/autoconf/lustre-libcfs.m4
>> > +++ b/libcfs/autoconf/lustre-libcfs.m4
>> > @@ -761,6 +761,26 @@ LB_CHECK_LINUX_HEADER([linux/stringhash.h], [
>> >  ]) # LIBCFS_STRINGHASH
>> >  
>> >  #
>> > +# LIBCFS_RHLTABLE
>> > +# Kernel version 4.8 commit ca26893f05e86497a86732768ec53cd38c0819ca
>> > +# created the rhlist interface to allow inserting duplicate objects
>> > +# into the same table.
>> > +#
>> > +AC_DEFUN([LIBCFS_RHLTABLE], [
>> > +LB_CHECK_COMPILE([if 'struct rhltable' exist],
>> > +rhtable, [
>> > +	#include <linux/rhashtable.h>
>> > +],[
>> > +	struct rhltable *hlt = NULL;
>> > +
>> > +	rhltable_destroy(hlt);
>> > +],[
>> > +	AC_DEFINE(HAVE_RHLTABLE, 1,
>> > +		  [struct rhltable exist])
>> > +])
>> > +]) # LIBCFS_RHLTABLE
>> > +
>> > +#
>> >  # LIBCFS_STACKTRACE_OPS
>> >  #
>> >  # Kernel version 4.8 commit c8fe4609827aedc9c4b45de80e7cdc8ccfa8541b
>> > @@ -999,6 +1019,7 @@ LIBCFS_STACKTRACE_OPS_ADDRESS_RETURN_INT
>> >  LIBCFS_GET_USER_PAGES_6ARG
>> >  LIBCFS_STRINGHASH
>> >  # 4.8
>> > +LIBCFS_RHLTABLE
>> >  LIBCFS_STACKTRACE_OPS
>> >  # 4.9
>> >  LIBCFS_GET_USER_PAGES_GUP_FLAGS
>> > diff --git a/libcfs/include/libcfs/linux/linux-hash.h b/libcfs/include/libcfs/linux/linux-hash.h
>> > index 1227ec8..0453cd9 100644
>> > --- a/libcfs/include/libcfs/linux/linux-hash.h
>> > +++ b/libcfs/include/libcfs/linux/linux-hash.h
>> > @@ -38,6 +38,94 @@ u64 cfs_hashlen_string(const void *salt, const char *name);
>> >  #endif
>> >  #endif /* !HAVE_STRINGHASH */
>> >  
>> > +#ifndef HAVE_RHLTABLE
>> > +struct rhlist_head {
>> > +	struct rhash_head		rhead;
>> > +	struct rhlist_head __rcu	*next;
>> > +};
>> > +
>> > +struct rhltable {
>> > +	struct rhashtable ht;
>> > +};
>> > +
>> > +#define rhl_for_each_entry_rcu(tpos, pos, list, member)                 \
>> > +	for (pos = list; pos && rht_entry(tpos, pos, member);           \
>> > +		pos = rcu_dereference_raw(pos->next))
>> > +
>> > +static inline int rhltable_init(struct rhltable *hlt, const struct rhashtable_params *params)
>> > +{
>> > +	return rhashtable_init(&hlt->ht, params);
>> > +}
>> > +
>> > +static inline struct rhlist_head *rhltable_lookup(
>> > +	struct rhltable *hlt, const void *key,
>> > +	const struct rhashtable_params params)
>> > +{
>> > +	struct rhashtable *ht = &hlt->ht;
>> > +	struct rhashtable_compare_arg arg = {
>> > +		.ht = ht,
>> > +		.key = key,
>> > +	};
>> > +	struct bucket_table *tbl;
>> > +	struct rhash_head *he;
>> > +	unsigned int hash;
>> > +
>> > +	tbl = rht_dereference_rcu(ht->tbl, ht);
>> > +restart:
>> > +	hash = rht_key_hashfn(ht, tbl, key, params);
>> > +	rht_for_each_rcu(he, tbl, hash) {
>> > +		if (params.obj_cmpfn ?
>> > +		    params.obj_cmpfn(&arg, rht_obj(ht, he)) :
>> > +		    rhashtable_compare(&arg, rht_obj(ht, he)))
>> > +			continue;
>> > +		return he ? container_of(he, struct rhlist_head, rhead) : NULL;
>> > +	}
>> > +
>> > +	/* Ensure we see any new tables. */
>> > +	smp_rmb();
>> > +
>> > +	tbl = rht_dereference_rcu(tbl->future_tbl, ht);
>> > +	if (unlikely(tbl))
>> > +		goto restart;
>> > +
>> > +	return NULL;
>> > +}
>> > +
>> > +static inline int rhltable_insert_key(
>> > +	struct rhltable *hlt, const void *key, struct rhlist_head *list,
>> > +	const struct rhashtable_params params)
>> > +{
>> > +	return PTR_ERR(__rhashtable_insert_fast(&hlt->ht, key, &list->rhead,
>> > +						params));
>> > +}
>> > +
>> > +static inline int rhltable_remove(
>> > +	struct rhltable *hlt, struct rhlist_head *list,
>> > +	const struct rhashtable_params params)
>> > +{
>> > +	return rhashtable_remove_fast(&hlt->ht, &list->rhead, params);
>> > +}
>> > +
>> > +static inline void rhltable_free_and_destroy(struct rhltable *hlt,
>> > +					     void (*free_fn)(void *ptr,
>> > +							     void *arg),
>> > +					     void *arg)
>> > +{
>> > +	return rhashtable_free_and_destroy(&hlt->ht, free_fn, arg);
>> > +}
>> > +
>> > +static inline void rhltable_destroy(struct rhltable *hlt)
>> > +{
>> > +	return rhltable_free_and_destroy(hlt, NULL, NULL);
>> > +}
>> > +
>> > +static inline void rhltable_walk_enter(struct rhltable *hlt,
>> > +				       struct rhashtable_iter *iter)
>> > +{
>> > +	rhashtable_walk_init(&hlt->ht, iter);
>> > +}
>> > +#endif /* !HAVE_RHLTABLE */
>> > +
>> >  #ifndef HAVE_RHASHTABLE_LOOKUP_GET_INSERT_FAST
>> >  /**
>> >   * rhashtable_lookup_get_insert_fast - lookup and insert object into hash table
>> > diff --git a/lustre/include/lustre_export.h b/lustre/include/lustre_export.h
>> > index 5ead593..54887c9 100644
>> > --- a/lustre/include/lustre_export.h
>> > +++ b/lustre/include/lustre_export.h
>> > @@ -209,7 +209,7 @@ struct obd_export {
>> >  	/* Unlinked export list */
>> >  	struct list_head	exp_stale_list;
>> >  	struct hlist_node	exp_uuid_hash;	/** uuid-export hash*/
>> > -	struct hlist_node	exp_nid_hash;	/** nid-export hash */
>> > +	struct rhlist_head	exp_nid_hash;	/** nid-export hash */
>> >  	struct hlist_node	exp_gen_hash;   /** last_rcvd clt gen hash */
>> >          /**
>> >           * All exports eligible for ping evictor are linked into a list
>> > diff --git a/lustre/include/obd.h b/lustre/include/obd.h
>> > index 1fcf0a2..8219710 100644
>> > --- a/lustre/include/obd.h
>> > +++ b/lustre/include/obd.h
>> > @@ -639,7 +639,7 @@ struct obd_device {
>> >          /* uuid-export hash body */
>> >  	struct cfs_hash             *obd_uuid_hash;
>> >          /* nid-export hash body */
>> > -	struct cfs_hash             *obd_nid_hash;
>> > +	struct rhltable			obd_nid_hash;
>> >  	/* nid stats body */
>> >  	struct cfs_hash             *obd_nid_stats_hash;
>> >  	/* client_generation-export hash body */
>> > @@ -750,6 +750,12 @@ struct obd_device {
>> >  	struct completion		obd_kobj_unregister;
>> >  };
>> >  
>> > +int obd_nid_export_for_each(struct obd_device *obd, lnet_nid_t nid,
>> > +			    int cb(struct obd_export *exp, void *data),
>> > +			    void *data);
>> > +int obd_nid_add(struct obd_device *obd, struct obd_export *exp);
>> > +void obd_nid_del(struct obd_device *obd, struct obd_export *exp);
>> > +
>> >  /* get/set_info keys */
>> >  #define KEY_ASYNC               "async"
>> >  #define KEY_CHANGELOG_CLEAR     "changelog_clear"
>> > diff --git a/lustre/include/obd_support.h b/lustre/include/obd_support.h
>> > index e9dd33e..0175cf8 100644
>> > --- a/lustre/include/obd_support.h
>> > +++ b/lustre/include/obd_support.h
>> > @@ -78,9 +78,6 @@ extern char obd_jobid_var[];
>> >  #define HASH_UUID_BKT_BITS 5
>> >  #define HASH_UUID_CUR_BITS 7
>> >  #define HASH_UUID_MAX_BITS 12
>> > -#define HASH_NID_BKT_BITS 5
>> > -#define HASH_NID_CUR_BITS 7
>> > -#define HASH_NID_MAX_BITS 12
>> >  #define HASH_NID_STATS_BKT_BITS 5
>> >  #define HASH_NID_STATS_CUR_BITS 7
>> >  #define HASH_NID_STATS_MAX_BITS 12
>> > diff --git a/lustre/ldlm/ldlm_flock.c b/lustre/ldlm/ldlm_flock.c
>> > index f848a36..4c1603a 100644
>> > --- a/lustre/ldlm/ldlm_flock.c
>> > +++ b/lustre/ldlm/ldlm_flock.c
>> > @@ -161,20 +161,18 @@ ldlm_flock_destroy(struct ldlm_lock *lock, enum ldlm_mode mode, __u64 flags)
>> >   */
>> >  
>> >  struct ldlm_flock_lookup_cb_data {
>> > -	__u64 *bl_owner;
>> > +	u64 *bl_owner;
>> 
>> Arrrggg.   Please don't do this.
>> 
>> >  	lock = cfs_hash_lookup(exp->exp_flock_hash, cb_data->bl_owner);
>> > -	if (lock == NULL)
>> > +	if (!lock)
>> 
>> Or this.
>> 
>> If you want to fix up this stuff, do it in a separate patch.
>> A patch should just do one thing.
>> I would actually prefer that the "add compatibilty code so we can use
>> rhltables in old kernels" was a separate patch from "use rhltables for
>> obd_nid_hash", but at least those two are conceptually related.
>> 
>> It is much harder to read a patch if I keep having to say to myself "Oh,
>> that change is irrelevant here,  I can ignore it".
>
> This is a pretty big patch for email review so I will break it up. I think 
> it can be more than 2 and it doesn't matter for buildable from patch to
> patch since this us just for review. 

I don't accept that there is a distinction between "For review" and "for
use".  The patches as reviewed much be exactly the patches that get
applied.
It does take a little more work to make sure that you can build and test
after each patch, but it is only a little.  Often the early patches
introduce functionality that is not immediately used, then the later
patches use it.  Very rarely you might need to leave off a "static"
declaration, so the code will build even though the function isn't used.

Thanks,
NeilBrown
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 832 bytes
Desc: not available
URL: <http://lists.lustre.org/pipermail/lustre-devel_lists.lustre.org/attachments/20181105/f28fd5c1/attachment.sig>

From neilb at suse.com  Sun Nov  4 23:59:17 2018
From: neilb at suse.com (NeilBrown)
Date: Mon, 05 Nov 2018 10:59:17 +1100
Subject: [lustre-devel] [PATCH 04/28] lustre: ptlrpc: Do not assert when
 bd_nob_transferred != 0
In-Reply-To: <alpine.LFD.2.21.1811042054180.27308@casper.infradead.org>
References: <1539543498-29105-1-git-send-email-jsimmons@infradead.org>
 <1539543498-29105-5-git-send-email-jsimmons@infradead.org>
 <87va60cgjx.fsf@notabene.neil.brown.name>
 <alpine.LFD.2.21.1810212339590.6249@casper.infradead.org>
 <87pnw28xvx.fsf@notabene.neil.brown.name>
 <alpine.LFD.2.21.1811042054180.27308@casper.infradead.org>
Message-ID: <871s801jje.fsf@notabene.neil.brown.name>

On Sun, Nov 04 2018, James Simmons wrote:

>> >> On Sun, Oct 14 2018, James Simmons wrote:
>> >> 
>> >> > From: Doug Oucharek <dougso at me.com>
>> >> >
>> >> > There is a case in the routine ptlrpc_register_bulk() where we were
>> >> > asserting if bd_nob_transferred != 0 when not resending.  There is
>> >> > evidence that network errors can create a situation where
>> >> > this does happen. So we should not be asserting!
>> >> >
>> >> > This patch changes that assert to an error return code of -EIO.
>> >> >
>> >> > Signed-off-by: Doug Oucharek <dougso at me.com>
>> >> > WC-bug-id: https://jira.whamcloud.com/browse/LU-9828
>> >> > Reviewed-on: https://review.whamcloud.com/28491
>> >> > Reviewed-by: Dmitry Eremin <dmitry.eremin at intel.com>
>> >> > Reviewed-by: Sonia Sharma <sharmaso at whamcloud.com>
>> >> > Reviewed-by: Oleg Drokin <green at whamcloud.com>
>> >> > Signed-off-by: James Simmons <jsimmons at infradead.org>
>> >> > ---
>> >> >  drivers/staging/lustre/lustre/ptlrpc/niobuf.c | 8 ++++++--
>> >> >  1 file changed, 6 insertions(+), 2 deletions(-)
>> >> >
>> >> > diff --git a/drivers/staging/lustre/lustre/ptlrpc/niobuf.c b/drivers/staging/lustre/lustre/ptlrpc/niobuf.c
>> >> > index 27eb1c0..7e7db24 100644
>> >> > --- a/drivers/staging/lustre/lustre/ptlrpc/niobuf.c
>> >> > +++ b/drivers/staging/lustre/lustre/ptlrpc/niobuf.c
>> >> > @@ -139,8 +139,12 @@ static int ptlrpc_register_bulk(struct ptlrpc_request *req)
>> >> >  	/* cleanup the state of the bulk for it will be reused */
>> >> >  	if (req->rq_resend || req->rq_send_state == LUSTRE_IMP_REPLAY)
>> >> >  		desc->bd_nob_transferred = 0;
>> >> > -	else
>> >> > -		LASSERT(desc->bd_nob_transferred == 0);
>> >> > +	else if (desc->bd_nob_transferred != 0)
>> >> > +		/* If the network failed after an RPC was sent, this condition
>> >> > +		 * could happen.  Rather than assert (was here before), return
>> >> > +		 * an EIO error.
>> >> > +		 */
>> >> > +		return -EIO;
>> >> 
>> >> This looks weird, and the justification is rather lame.
>> >> I wonder if this is an attempt to fix the same problem that the smp_mb()
>> >> in the previous patch was attempting to fix (and I'm not yet convinced
>> >> that either is the correct fix).
>> >
>> > When the above condition happens the LASSERT ends up taking out the 
>> > node with a panic which in turn kills the application running on the cluster.
>> > When replaced with reporting an EIO error the node survives as well as the 
>> > job. The job might fail at its IO but it wouldn't fail performing its work 
>> > flow which is way more important.
>> 
>> Yes, a meaningless error is better than a crash, but a proper fix is
>> better still.  As I said, my guess is that the memory barrier in the
>> previous patch might have fixed the bug, so the LASSERT can remain.
>> 
>> Doug: is there any chance that this might be the case?
>
> I got a hold of Doug and discussed this issue. So the answer is that the
> original logs to track down the original problem no longer exist. So 
> finding the original source of the problem can't be done at this point.
> Would you be okay with a version of this patch with dump_stack() and
> treat it as a debug patch. We really need to collect logs to figure out
> the real problem. I will push a debug patch to OpenSFS branch since it
> is more widely used.

Yes.
   else if (WARN_ON(desc->nb_nob_transferred != 0))
       /* comment explaining what we know 8/
       return -EIO

would be perfectly appropriate.

Thanks,
NeilBrown
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 832 bytes
Desc: not available
URL: <http://lists.lustre.org/pipermail/lustre-devel_lists.lustre.org/attachments/20181105/c9315ca8/attachment.sig>

From neilb at suse.com  Mon Nov  5 00:03:13 2018
From: neilb at suse.com (NeilBrown)
Date: Mon, 05 Nov 2018 11:03:13 +1100
Subject: [lustre-devel] [PATCH 15/28] lustre: llite: fix for stat under
 kthread and X86_X32
In-Reply-To: <alpine.LFD.2.21.1811042134360.27308@casper.infradead.org>
References: <1539543498-29105-1-git-send-email-jsimmons@infradead.org>
 <1539543498-29105-16-git-send-email-jsimmons@infradead.org>
 <87ftx4c9e6.fsf@notabene.neil.brown.name>
 <87h8he8wfq.fsf@notabene.neil.brown.name>
 <alpine.LFD.2.21.1811042134360.27308@casper.infradead.org>
Message-ID: <87y3a8z8zi.fsf@notabene.neil.brown.name>

On Sun, Nov 04 2018, James Simmons wrote:

>> On Thu, Oct 18 2018, NeilBrown wrote:
>> 
>> > On Sun, Oct 14 2018, James Simmons wrote:
>> >
>> >> From: Frank Zago <fzago at cray.com>
>> >>
>> >> Under the following conditions, ll_getattr will flatten the inode
>> >> number when it shouldn't:
>> >>
>> >>  - the X86_X32 architecture is defined CONFIG_X86_X32, and not even
>> >>    used,
>> >>  - ll_getattr is called from a kernel thread (though vfs_getattr for
>> >>    instance.)
>> >>
>> >> This has the result that inode numbers are different whether the same
>> >> file is stat'ed from a kernel thread, or from a syscall. For instance,
>> >> 4198401 vs. 144115205272502273.
>> >>
>> >> ll_getattr calls ll_need_32bit_api to determine whether the task is 32
>> >> bits. When the combination is kthread+X86_X32, that function returns
>> >> that the task is 32 bits, which is incorrect, as the kernel is 64
>> >> bits.
>> >>
>> >> The solution is to check whether the call is from a kernel thread
>> >> (which is 64 bits) and act consequently.
>> >>
>> >> Signed-off-by: Frank Zago <fzago at cray.com>
>> >> WC-bug-id: https://jira.whamcloud.com/browse/LU-9468
>> >> Reviewed-on: https://review.whamcloud.com/26992
>> >> Reviewed-by: Andreas Dilger <adilger at whamcloud.com>
>> >> Reviewed-by: Dmitry Eremin <dmitry.eremin at intel.com>
>> >> Signed-off-by: James Simmons <jsimmons at infradead.org>
>> >> ---
>> >>  drivers/staging/lustre/lustre/llite/dir.c          |  6 +++---
>> >>  drivers/staging/lustre/lustre/llite/lcommon_cl.c   |  2 +-
>> >>  .../staging/lustre/lustre/llite/llite_internal.h   | 22 +++++++++++++++++-----
>> >>  3 files changed, 21 insertions(+), 9 deletions(-)
>> >>
>> >> diff --git a/drivers/staging/lustre/lustre/llite/dir.c b/drivers/staging/lustre/lustre/llite/dir.c
>> >> index 231b351..19c5e9c 100644
>> >> --- a/drivers/staging/lustre/lustre/llite/dir.c
>> >> +++ b/drivers/staging/lustre/lustre/llite/dir.c
>> >> @@ -202,7 +202,7 @@ int ll_dir_read(struct inode *inode, __u64 *ppos, struct md_op_data *op_data,
>> >>  {
>> >>  	struct ll_sb_info    *sbi	= ll_i2sbi(inode);
>> >>  	__u64		   pos		= *ppos;
>> >> -	int		   is_api32 = ll_need_32bit_api(sbi);
>> >> +	bool is_api32 = ll_need_32bit_api(sbi);
>> >>  	int		   is_hash64 = sbi->ll_flags & LL_SBI_64BIT_HASH;
>> >>  	struct page	  *page;
>> >>  	bool		   done = false;
>> >> @@ -296,7 +296,7 @@ static int ll_readdir(struct file *filp, struct dir_context *ctx)
>> >>  	struct ll_sb_info	*sbi	= ll_i2sbi(inode);
>> >>  	__u64 pos = lfd ? lfd->lfd_pos : 0;
>> >>  	int			hash64	= sbi->ll_flags & LL_SBI_64BIT_HASH;
>> >> -	int			api32	= ll_need_32bit_api(sbi);
>> >> +	bool api32 = ll_need_32bit_api(sbi);
>> >>  	struct md_op_data *op_data;
>> >>  	int			rc;
>> >>  
>> >> @@ -1674,7 +1674,7 @@ static loff_t ll_dir_seek(struct file *file, loff_t offset, int origin)
>> >>  	struct inode *inode = file->f_mapping->host;
>> >>  	struct ll_file_data *fd = LUSTRE_FPRIVATE(file);
>> >>  	struct ll_sb_info *sbi = ll_i2sbi(inode);
>> >> -	int api32 = ll_need_32bit_api(sbi);
>> >> +	bool api32 = ll_need_32bit_api(sbi);
>> >>  	loff_t ret = -EINVAL;
>> >>  
>> >>  	switch (origin) {
>> >> diff --git a/drivers/staging/lustre/lustre/llite/lcommon_cl.c b/drivers/staging/lustre/lustre/llite/lcommon_cl.c
>> >> index 30f17ea..20a3c74 100644
>> >> --- a/drivers/staging/lustre/lustre/llite/lcommon_cl.c
>> >> +++ b/drivers/staging/lustre/lustre/llite/lcommon_cl.c
>> >> @@ -267,7 +267,7 @@ void cl_inode_fini(struct inode *inode)
>> >>  /**
>> >>   * build inode number from passed @fid
>> >>   */
>> >> -__u64 cl_fid_build_ino(const struct lu_fid *fid, int api32)
>> >> +u64 cl_fid_build_ino(const struct lu_fid *fid, bool api32)
>> >>  {
>> >>  	if (BITS_PER_LONG == 32 || api32)
>> >>  		return fid_flatten32(fid);
>> >> diff --git a/drivers/staging/lustre/lustre/llite/llite_internal.h b/drivers/staging/lustre/lustre/llite/llite_internal.h
>> >> index dcb2fed..796a8ae 100644
>> >> --- a/drivers/staging/lustre/lustre/llite/llite_internal.h
>> >> +++ b/drivers/staging/lustre/lustre/llite/llite_internal.h
>> >> @@ -651,13 +651,25 @@ static inline struct inode *ll_info2i(struct ll_inode_info *lli)
>> >>  __u32 ll_i2suppgid(struct inode *i);
>> >>  void ll_i2gids(__u32 *suppgids, struct inode *i1, struct inode *i2);
>> >>  
>> >> -static inline int ll_need_32bit_api(struct ll_sb_info *sbi)
>> >> +static inline bool ll_need_32bit_api(struct ll_sb_info *sbi)
>> >>  {
>> >>  #if BITS_PER_LONG == 32
>> >> -	return 1;
>> >> +	return true;
>> >>  #elif defined(CONFIG_COMPAT)
>> >> -	return unlikely(in_compat_syscall() ||
>> >> -			(sbi->ll_flags & LL_SBI_32BIT_API));
>> >> +	if (unlikely(sbi->ll_flags & LL_SBI_32BIT_API))
>> >> +		return true;
>> >> +
>> >> +#ifdef CONFIG_X86_X32
>> >> +	/* in_compat_syscall() returns true when called from a kthread
>> >> +	 * and CONFIG_X86_X32 is enabled, which is wrong. So check
>> >> +	 * whether the caller comes from a syscall (ie. not a kthread)
>> >> +	 * before calling in_compat_syscall().
>> >> +	 */
>> >> +	if (current->flags & PF_KTHREAD)
>> >> +		return false;
>> >> +#endif
>> >
>> > This is wrong.  We should fix in_compat_syscall(), not work around it
>> > here.
>> > (and then there is that fact that the patch changes 'int' to 'bool'
>> > without explaining that in the change description).
>> >
>> > I've sent a query to some relevant people (Cc:ed to James) to ask about
>> > fixing in_compat_syscall().
>> 
>> Upstream say in_compat_syscall() should only be called from a syscall,
>> so I have change the patch to the below.
>> 
>> We probably need to remove this in_compat_syscall() before going
>> mainline.
>
> Do you know the progress of this work?
>  

Below is the code that I currently have.
It avoids making a special case of X86_X32 - I should update
the comment to reflect that.

I've haven't given much thought yet to removing the need for
in_compat_syscall().  I need to make sure I understand exactly why
it is currently needed first.

NeilBrown


commit c69633550256d1a68306caf4f67a7d58ba8763e8
Author: Frank Zago <fzago at cray.com>
Date:   Sun Oct 14 14:58:05 2018 -0400

    lustre: llite: fix for stat under kthread and X86_X32
    
    Under the following conditions, ll_getattr will flatten the inode
    number when it shouldn't:
    
     - the X86_X32 architecture is defined CONFIG_X86_X32, and not even
       used,
     - ll_getattr is called from a kernel thread (though vfs_getattr for
       instance.)
    
    This has the result that inode numbers are different whether the same
    file is stat'ed from a kernel thread, or from a syscall. For instance,
    4198401 vs. 144115205272502273.
    
    ll_getattr calls ll_need_32bit_api to determine whether the task is 32
    bits. When the combination is kthread+X86_X32, that function returns
    that the task is 32 bits, which is incorrect, as the kernel is 64
    bits.
    
    The solution is to check whether the call is from a kernel thread
    (which is 64 bits) and act consequently.
    
    Signed-off-by: Frank Zago <fzago at cray.com>
    WC-bug-id: https://jira.whamcloud.com/browse/LU-9468
    Reviewed-on: https://review.whamcloud.com/26992
    Reviewed-by: Andreas Dilger <adilger at whamcloud.com>
    Reviewed-by: Dmitry Eremin <dmitry.eremin at intel.com>
    Signed-off-by: James Simmons <jsimmons at infradead.org>
    Signed-off-by: NeilBrown <neilb at suse.com>

diff --git a/drivers/staging/lustre/lustre/llite/dir.c b/drivers/staging/lustre/lustre/llite/dir.c
index 231b351536bf..19c5e9cee3f9 100644
--- a/drivers/staging/lustre/lustre/llite/dir.c
+++ b/drivers/staging/lustre/lustre/llite/dir.c
@@ -202,7 +202,7 @@ int ll_dir_read(struct inode *inode, __u64 *ppos, struct md_op_data *op_data,
 {
 	struct ll_sb_info    *sbi	= ll_i2sbi(inode);
 	__u64		   pos		= *ppos;
-	int		   is_api32 = ll_need_32bit_api(sbi);
+	bool is_api32 = ll_need_32bit_api(sbi);
 	int		   is_hash64 = sbi->ll_flags & LL_SBI_64BIT_HASH;
 	struct page	  *page;
 	bool		   done = false;
@@ -296,7 +296,7 @@ static int ll_readdir(struct file *filp, struct dir_context *ctx)
 	struct ll_sb_info	*sbi	= ll_i2sbi(inode);
 	__u64 pos = lfd ? lfd->lfd_pos : 0;
 	int			hash64	= sbi->ll_flags & LL_SBI_64BIT_HASH;
-	int			api32	= ll_need_32bit_api(sbi);
+	bool api32 = ll_need_32bit_api(sbi);
 	struct md_op_data *op_data;
 	int			rc;
 
@@ -1674,7 +1674,7 @@ static loff_t ll_dir_seek(struct file *file, loff_t offset, int origin)
 	struct inode *inode = file->f_mapping->host;
 	struct ll_file_data *fd = LUSTRE_FPRIVATE(file);
 	struct ll_sb_info *sbi = ll_i2sbi(inode);
-	int api32 = ll_need_32bit_api(sbi);
+	bool api32 = ll_need_32bit_api(sbi);
 	loff_t ret = -EINVAL;
 
 	switch (origin) {
diff --git a/drivers/staging/lustre/lustre/llite/lcommon_cl.c b/drivers/staging/lustre/lustre/llite/lcommon_cl.c
index 30f17eaa6b2c..20a3c749f085 100644
--- a/drivers/staging/lustre/lustre/llite/lcommon_cl.c
+++ b/drivers/staging/lustre/lustre/llite/lcommon_cl.c
@@ -267,7 +267,7 @@ void cl_inode_fini(struct inode *inode)
 /**
  * build inode number from passed @fid
  */
-__u64 cl_fid_build_ino(const struct lu_fid *fid, int api32)
+u64 cl_fid_build_ino(const struct lu_fid *fid, bool api32)
 {
 	if (BITS_PER_LONG == 32 || api32)
 		return fid_flatten32(fid);
diff --git a/drivers/staging/lustre/lustre/llite/llite_internal.h b/drivers/staging/lustre/lustre/llite/llite_internal.h
index dcb2fed7a350..26c35f5d28a6 100644
--- a/drivers/staging/lustre/lustre/llite/llite_internal.h
+++ b/drivers/staging/lustre/lustre/llite/llite_internal.h
@@ -651,15 +651,27 @@ static inline struct inode *ll_info2i(struct ll_inode_info *lli)
 __u32 ll_i2suppgid(struct inode *i);
 void ll_i2gids(__u32 *suppgids, struct inode *i1, struct inode *i2);
 
-static inline int ll_need_32bit_api(struct ll_sb_info *sbi)
+static inline bool ll_need_32bit_api(struct ll_sb_info *sbi)
 {
 #if BITS_PER_LONG == 32
-	return 1;
-#elif defined(CONFIG_COMPAT)
-	return unlikely(in_compat_syscall() ||
-			(sbi->ll_flags & LL_SBI_32BIT_API));
+	return true;
+#else
+	if (unlikely(sbi->ll_flags & LL_SBI_32BIT_API))
+		return true;
+
+#if defined(CONFIG_COMPAT)
+	/* in_compat_syscall() is only meaningful inside a syscall.
+	 * As this can be called from a kthread (e.g. nfsd), we
+	 * need to catch that case first.  kthreads never need the
+	 * 32bit api.
+	 */
+	if (current->flags & PF_KTHREAD)
+		return false;
+
+	return unlikely(in_compat_syscall());
 #else
-	return unlikely(sbi->ll_flags & LL_SBI_32BIT_API);
+	return false;
+#endif
 #endif
 }
 
@@ -1353,7 +1365,7 @@ extern u16 cl_inode_fini_refcheck;
 int cl_file_inode_init(struct inode *inode, struct lustre_md *md);
 void cl_inode_fini(struct inode *inode);
 
-__u64 cl_fid_build_ino(const struct lu_fid *fid, int api32);
+u64 cl_fid_build_ino(const struct lu_fid *fid, bool api32);
 __u32 cl_fid_build_gen(const struct lu_fid *fid);
 
 #endif /* LLITE_INTERNAL_H */
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 832 bytes
Desc: not available
URL: <http://lists.lustre.org/pipermail/lustre-devel_lists.lustre.org/attachments/20181105/d9e78cd5/attachment.sig>

From neilb at suse.com  Mon Nov  5 00:13:18 2018
From: neilb at suse.com (NeilBrown)
Date: Mon, 05 Nov 2018 11:13:18 +1100
Subject: [lustre-devel] [PATCH 18/28] lustre: mdc: expose changelog
 through char devices
In-Reply-To: <alpine.LFD.2.21.1811042130370.27308@casper.infradead.org>
References: <1539543498-29105-1-git-send-email-jsimmons@infradead.org>
 <1539543498-29105-19-git-send-email-jsimmons@infradead.org>
 <87efc82ayj.fsf@notabene.neil.brown.name>
 <alpine.LFD.2.21.1811042130370.27308@casper.infradead.org>
Message-ID: <87va5cz8ip.fsf@notabene.neil.brown.name>

On Sun, Nov 04 2018, James Simmons wrote:

>> On Sun, Oct 14 2018, James Simmons wrote:
>> 
>> > From: Henri Doreau <henri.doreau at cea.fr>
>> >
>> > Register one character device per MDT in order to allow non-llapi to
>> > read them and to make delivery more efficient.
>> >
>> > - open() spawns a thread to prefetch records and enqueue them into a
>> >   local buffer (unless the device is open in write-only mode).
>> > - lseek() can be used to jump to a specific record, in which case the
>> >   offset is a record number (with SEEK_SET) or a number of records to
>> >   skip (SEEK_CUR). Movement can only be done forward.
>> > - read() copies records to userland. No truncation happens, so short
>> >   reads are likely.
>> > - write() is used to transmit control commands to the device.
>> >   The only available one is changelog_clear, which is done by writing
>> >   "clear:cl<user>:<recno>" into the device.
>> > - close() terminates the prefetch thread if any, and releases resources.
>> >
>> > It is possible to poll() on the device to get notified when new records
>> > are available for read.
>> >
>> > Signed-off-by: Henri Doreau <henri.doreau at cea.fr>
>> > WC-bug-id: https://jira.whamcloud.com/browse/LU-7659
>> > Reviewed-on: https://review.whamcloud.com/18900
>> > Reviewed-by: Andreas Dilger <adilger at whamcloud.com>
>> > Reviewed-by: John L. Hammond <jhammond at whamcloud.com>
>> > Reviewed-by: Oleg Drokin <green at whamcloud.com>
>> > Signed-off-by: James Simmons <jsimmons at infradead.org>
>> 
>> This patches causes problems around sanity test 161a.
>> If you run -only '160h 160i 161a' it hangs.
>> 
>> Adding
>> Commit: 89e52326b5bd ("LU-10166 mdc: invalid free in changelog reader")
>> seems to fix the problem, so I'll port that into the series immediately
>> after this patch.
>
> I was planning to push this in the next batch. I wouldn't push it in that
> case and just wait for it to show up in lustre-testing.
>

It turns out that it didn't fix my problem - I don't know why it seemed
to.
Still, I'll keep it - and hope to push out a new lustre-testing today.

For now, I've disabled 160h and 160i as they always lead to problems
at about 161a.  Messages like

[  226.614732] Lustre: 4149:0:(client.c:2064:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1540794670/real 1540794670]  req at 000000008eaae33d x1615640091691072/t0(0) o250->MGC192.168.20.11 at tcp@192.168.20.11 at tcp:26/25 lens 520/544 e 0 to 1 dl 1540794676 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1

keep appearing, and no progress is made.
It is definitely related to the new changelog code, but I have no ideas
beyond that.

Thanks,
NeilBrown

>
>> >  .../include/uapi/linux/lustre/lustre_kernelcomm.h  |   3 -
>> >  .../lustre/include/uapi/linux/lustre/lustre_user.h |   7 -
>> >  drivers/staging/lustre/lustre/include/obd.h        |   2 +
>> >  drivers/staging/lustre/lustre/ldlm/ldlm_lib.c      |   2 +
>> >  drivers/staging/lustre/lustre/llite/dir.c          |   8 -
>> >  drivers/staging/lustre/lustre/lmv/lmv_obd.c        |  13 -
>> >  drivers/staging/lustre/lustre/mdc/Makefile         |   2 +-
>> >  drivers/staging/lustre/lustre/mdc/mdc_changelog.c  | 722 +++++++++++++++++++++
>> >  drivers/staging/lustre/lustre/mdc/mdc_internal.h   |   4 +
>> >  drivers/staging/lustre/lustre/mdc/mdc_request.c    | 198 +-----
>> >  11 files changed, 745 insertions(+), 218 deletions(-)
>> >  create mode 100644 drivers/staging/lustre/lustre/mdc/mdc_changelog.c
>> >
>> > diff --git a/drivers/staging/lustre/include/uapi/linux/lustre/lustre_ioctl.h b/drivers/staging/lustre/include/uapi/linux/lustre/lustre_ioctl.h
>> > index 6e4e109..098b6451 100644
>> > --- a/drivers/staging/lustre/include/uapi/linux/lustre/lustre_ioctl.h
>> > +++ b/drivers/staging/lustre/include/uapi/linux/lustre/lustre_ioctl.h
>> > @@ -172,7 +172,7 @@ static inline __u32 obd_ioctl_packlen(struct obd_ioctl_data *data)
>> >  #define OBD_GET_VERSION		_IOWR('f', 144, OBD_IOC_DATA_TYPE)
>> >  /*	OBD_IOC_GSS_SUPPORT	_IOWR('f', 145, OBD_IOC_DATA_TYPE) */
>> >  /*	OBD_IOC_CLOSE_UUID	_IOWR('f', 147, OBD_IOC_DATA_TYPE) */
>> > -#define OBD_IOC_CHANGELOG_SEND	_IOW('f', 148, OBD_IOC_DATA_TYPE)
>> > +/*	OBD_IOC_CHANGELOG_SEND	_IOW('f', 148, OBD_IOC_DATA_TYPE) */
>> >  #define OBD_IOC_GETDEVICE	_IOWR('f', 149, OBD_IOC_DATA_TYPE)
>> >  #define OBD_IOC_FID2PATH	_IOWR('f', 150, OBD_IOC_DATA_TYPE)
>> >  /*	lustre/lustre_user.h	151-153 */
>> > diff --git a/drivers/staging/lustre/include/uapi/linux/lustre/lustre_kernelcomm.h b/drivers/staging/lustre/include/uapi/linux/lustre/lustre_kernelcomm.h
>> > index 94dadbe..d84a8fc 100644
>> > --- a/drivers/staging/lustre/include/uapi/linux/lustre/lustre_kernelcomm.h
>> > +++ b/drivers/staging/lustre/include/uapi/linux/lustre/lustre_kernelcomm.h
>> > @@ -54,15 +54,12 @@ struct kuc_hdr {
>> >  	__u16 kuc_msglen;
>> >  } __aligned(sizeof(__u64));
>> >  
>> > -#define KUC_CHANGELOG_MSG_MAXSIZE (sizeof(struct kuc_hdr) + CR_MAXSIZE)
>> > -
>> >  #define KUC_MAGIC		0x191C /*Lustre9etLinC */
>> >  
>> >  /* kuc_msgtype values are defined in each transport */
>> >  enum kuc_transport_type {
>> >  	KUC_TRANSPORT_GENERIC	= 1,
>> >  	KUC_TRANSPORT_HSM	= 2,
>> > -	KUC_TRANSPORT_CHANGELOG	= 3,
>> >  };
>> >  
>> >  enum kuc_generic_message_type {
>> > diff --git a/drivers/staging/lustre/include/uapi/linux/lustre/lustre_user.h b/drivers/staging/lustre/include/uapi/linux/lustre/lustre_user.h
>> > index b8525e5..715f1c5 100644
>> > --- a/drivers/staging/lustre/include/uapi/linux/lustre/lustre_user.h
>> > +++ b/drivers/staging/lustre/include/uapi/linux/lustre/lustre_user.h
>> > @@ -967,13 +967,6 @@ static inline void changelog_remap_rec(struct changelog_rec *rec,
>> >  	rec->cr_flags = (rec->cr_flags & CLF_FLAGMASK) | crf_wanted;
>> >  }
>> >  
>> > -struct ioc_changelog {
>> > -	__u64 icc_recno;
>> > -	__u32 icc_mdtindex;
>> > -	__u32 icc_id;
>> > -	__u32 icc_flags;
>> > -};
>> > -
>> >  enum changelog_message_type {
>> >  	CL_RECORD = 10, /* message is a changelog_rec */
>> >  	CL_EOF    = 11, /* at end of current changelog */
>> > diff --git a/drivers/staging/lustre/lustre/include/obd.h b/drivers/staging/lustre/lustre/include/obd.h
>> > index 11e7ae8..76ae0b3 100644
>> > --- a/drivers/staging/lustre/lustre/include/obd.h
>> > +++ b/drivers/staging/lustre/lustre/include/obd.h
>> > @@ -345,6 +345,8 @@ struct client_obd {
>> >  	void			*cl_lru_work;
>> >  	/* hash tables for osc_quota_info */
>> >  	struct rhashtable	cl_quota_hash[MAXQUOTAS];
>> > +	/* Links to the global list of registered changelog devices */
>> > +	struct list_head	cl_chg_dev_linkage;
>> >  };
>> >  
>> >  #define obd2cli_tgt(obd) ((char *)(obd)->u.cli.cl_target_uuid.uuid)
>> > diff --git a/drivers/staging/lustre/lustre/ldlm/ldlm_lib.c b/drivers/staging/lustre/lustre/ldlm/ldlm_lib.c
>> > index 32eda4f..732ef3a 100644
>> > --- a/drivers/staging/lustre/lustre/ldlm/ldlm_lib.c
>> > +++ b/drivers/staging/lustre/lustre/ldlm/ldlm_lib.c
>> > @@ -395,6 +395,8 @@ int client_obd_setup(struct obd_device *obddev, struct lustre_cfg *lcfg)
>> >  	init_waitqueue_head(&cli->cl_mod_rpcs_waitq);
>> >  	cli->cl_mod_tag_bitmap = NULL;
>> >  
>> > +	INIT_LIST_HEAD(&cli->cl_chg_dev_linkage);
>> > +
>> >  	if (connect_op == MDS_CONNECT) {
>> >  		cli->cl_max_mod_rpcs_in_flight = cli->cl_max_rpcs_in_flight - 1;
>> >  		cli->cl_mod_tag_bitmap = kcalloc(BITS_TO_LONGS(OBD_MAX_RIF_MAX),
>> > diff --git a/drivers/staging/lustre/lustre/llite/dir.c b/drivers/staging/lustre/lustre/llite/dir.c
>> > index 19c5e9c..36cea8d 100644
>> > --- a/drivers/staging/lustre/lustre/llite/dir.c
>> > +++ b/drivers/staging/lustre/lustre/llite/dir.c
>> > @@ -1481,14 +1481,6 @@ static long ll_dir_ioctl(struct file *file, unsigned int cmd, unsigned long arg)
>> >  		return obd_iocontrol(cmd, sbi->ll_md_exp, 0, NULL,
>> >  				     (void __user *)arg);
>> >  	}
>> > -	case OBD_IOC_CHANGELOG_SEND:
>> > -	case OBD_IOC_CHANGELOG_CLEAR:
>> > -		if (!capable(CAP_SYS_ADMIN))
>> > -			return -EPERM;
>> > -
>> > -		rc = copy_and_ioctl(cmd, sbi->ll_md_exp, (void __user *)arg,
>> > -				    sizeof(struct ioc_changelog));
>> > -		return rc;
>> >  	case OBD_IOC_FID2PATH:
>> >  		return ll_fid2path(inode, (void __user *)arg);
>> >  	case LL_IOC_GETPARENT:
>> > diff --git a/drivers/staging/lustre/lustre/lmv/lmv_obd.c b/drivers/staging/lustre/lustre/lmv/lmv_obd.c
>> > index 952c68e..32bb9fc 100644
>> > --- a/drivers/staging/lustre/lustre/lmv/lmv_obd.c
>> > +++ b/drivers/staging/lustre/lustre/lmv/lmv_obd.c
>> > @@ -951,19 +951,6 @@ static int lmv_iocontrol(unsigned int cmd, struct obd_export *exp,
>> >  		kfree(oqctl);
>> >  		break;
>> >  	}
>> > -	case OBD_IOC_CHANGELOG_SEND:
>> > -	case OBD_IOC_CHANGELOG_CLEAR: {
>> > -		struct ioc_changelog *icc = karg;
>> > -
>> > -		if (icc->icc_mdtindex >= count)
>> > -			return -ENODEV;
>> > -
>> > -		tgt = lmv->tgts[icc->icc_mdtindex];
>> > -		if (!tgt || !tgt->ltd_exp || !tgt->ltd_active)
>> > -			return -ENODEV;
>> > -		rc = obd_iocontrol(cmd, tgt->ltd_exp, sizeof(*icc), icc, NULL);
>> > -		break;
>> > -	}
>> >  	case LL_IOC_GET_CONNECT_FLAGS: {
>> >  		tgt = lmv->tgts[0];
>> >  
>> > diff --git a/drivers/staging/lustre/lustre/mdc/Makefile b/drivers/staging/lustre/lustre/mdc/Makefile
>> > index 64cf49e..5f48e91 100644
>> > --- a/drivers/staging/lustre/lustre/mdc/Makefile
>> > +++ b/drivers/staging/lustre/lustre/mdc/Makefile
>> > @@ -2,4 +2,4 @@ ccflags-y += -I$(srctree)/drivers/staging/lustre/include
>> >  ccflags-y += -I$(srctree)/drivers/staging/lustre/lustre/include
>> >  
>> >  obj-$(CONFIG_LUSTRE_FS) += mdc.o
>> > -mdc-y := mdc_request.o mdc_reint.o mdc_lib.o mdc_locks.o lproc_mdc.o
>> > +mdc-y := mdc_changelog.o mdc_request.o mdc_reint.o mdc_lib.o mdc_locks.o lproc_mdc.o
>> > diff --git a/drivers/staging/lustre/lustre/mdc/mdc_changelog.c b/drivers/staging/lustre/lustre/mdc/mdc_changelog.c
>> > new file mode 100644
>> > index 0000000..a5f3c64
>> > --- /dev/null
>> > +++ b/drivers/staging/lustre/lustre/mdc/mdc_changelog.c
>> > @@ -0,0 +1,722 @@
>> > +// SPDX-License-Identifier: GPL-2.0
>> > +/*
>> > + * GPL HEADER START
>> > + *
>> > + * DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
>> > + *
>> > + * This program is free software; you can redistribute it and/or modify
>> > + * it under the terms of the GNU General Public License version 2 only,
>> > + * as published by the Free Software Foundation.
>> > + *
>> > + * This program is distributed in the hope that it will be useful, but
>> > + * WITHOUT ANY WARRANTY; without even the implied warranty of
>> > + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
>> > + * General Public License version 2 for more details (a copy is included
>> > + * in the LICENSE file that accompanied this code).
>> > + *
>> > + * You should have received a copy of the GNU General Public License
>> > + * version 2 along with this program; If not, see
>> > + * http://www.gnu.org/licenses/gpl-2.0.html
>> > + *
>> > + * GPL HEADER END
>> > + */
>> > +/*
>> > + * Copyright (c) 2017, Commissariat a l'Energie Atomique et aux Energies
>> > + *                     Alternatives.
>> > + *
>> > + * Author: Henri Doreau <henri.doreau at cea.fr>
>> > + */
>> > +
>> > +#define DEBUG_SUBSYSTEM S_MDC
>> > +
>> > +#include <linux/init.h>
>> > +#include <linux/kthread.h>
>> > +#include <linux/poll.h>
>> > +#include <linux/miscdevice.h>
>> > +
>> > +#include <lustre_log.h>
>> > +
>> > +#include "mdc_internal.h"
>> > +
>> > +/*
>> > + * -- Changelog delivery through character device --
>> > + */
>> > +
>> > +/**
>> > + * Mutex to protect chlg_registered_devices below
>> > + */
>> > +static DEFINE_MUTEX(chlg_registered_dev_lock);
>> > +
>> > +/**
>> > + * Global linked list of all registered devices (one per MDT).
>> > + */
>> > +static LIST_HEAD(chlg_registered_devices);
>> > +
>> > +struct chlg_registered_dev {
>> > +	/* Device name of the form "changelog-{MDTNAME}" */
>> > +	char			ced_name[32];
>> > +	/* Misc device descriptor */
>> > +	struct miscdevice	ced_misc;
>> > +	/* OBDs referencing this device (multiple mount point) */
>> > +	struct list_head	ced_obds;
>> > +	/* Reference counter for proper deregistration */
>> > +	struct kref		ced_refs;
>> > +	/* Link within the global chlg_registered_devices */
>> > +	struct list_head	ced_link;
>> > +};
>> > +
>> > +struct chlg_reader_state {
>> > +	/* Shortcut to the corresponding OBD device */
>> > +	struct obd_device	*crs_obd;
>> > +	/* An error occurred that prevents from reading further */
>> > +	bool			 crs_err;
>> > +	/* EOF, no more records available */
>> > +	bool			 crs_eof;
>> > +	/* Userland reader closed connection */
>> > +	bool			 crs_closed;
>> > +	/* Desired start position */
>> > +	u64			 crs_start_offset;
>> > +	/* Wait queue for the catalog processing thread */
>> > +	wait_queue_head_t	 crs_waitq_prod;
>> > +	/* Wait queue for the record copy threads */
>> > +	wait_queue_head_t	 crs_waitq_cons;
>> > +	/* Mutex protecting crs_rec_count and crs_rec_queue */
>> > +	struct mutex		 crs_lock;
>> > +	/* Number of item in the list */
>> > +	u64			 crs_rec_count;
>> > +	/* List of prefetched enqueued_record::enq_linkage_items */
>> > +	struct list_head	 crs_rec_queue;
>> > +};
>> > +
>> > +struct chlg_rec_entry {
>> > +	/* Link within the chlg_reader_state::crs_rec_queue list */
>> > +	struct list_head	enq_linkage;
>> > +	/* Data (enq_record) field length */
>> > +	u64			enq_length;
>> > +	/* Copy of a changelog record (see struct llog_changelog_rec) */
>> > +	struct changelog_rec	enq_record[];
>> > +};
>> > +
>> > +enum {
>> > +	/* Number of records to prefetch locally. */
>> > +	CDEV_CHLG_MAX_PREFETCH = 1024,
>> > +};
>> > +
>> > +/**
>> > + * ChangeLog catalog processing callback invoked on each record.
>> > + * If the current record is eligible to userland delivery, push
>> > + * it into the crs_rec_queue where the consumer code will fetch it.
>> > + *
>> > + * @param[in]     env  (unused)
>> > + * @param[in]     llh  Client-side handle used to identify the llog
>> > + * @param[in]     hdr  Header of the current llog record
>> > + * @param[in,out] data chlg_reader_state passed from caller
>> > + *
>> > + * @return 0 or LLOG_PROC_* control code on success, negated error on failure.
>> > + */
>> > +static int chlg_read_cat_process_cb(const struct lu_env *env,
>> > +				    struct llog_handle *llh,
>> > +				    struct llog_rec_hdr *hdr, void *data)
>> > +{
>> > +	struct llog_changelog_rec *rec;
>> > +	struct chlg_reader_state *crs = data;
>> > +	struct chlg_rec_entry *enq;
>> > +	size_t len;
>> > +	int rc;
>> > +
>> > +	LASSERT(crs);
>> > +	LASSERT(hdr);
>> > +
>> > +	rec = container_of(hdr, struct llog_changelog_rec, cr_hdr);
>> > +
>> > +	if (rec->cr_hdr.lrh_type != CHANGELOG_REC) {
>> > +		rc = -EINVAL;
>> > +		CERROR("%s: not a changelog rec %x/%d in llog : rc = %d\n",
>> > +		       crs->crs_obd->obd_name, rec->cr_hdr.lrh_type,
>> > +		       rec->cr.cr_type, rc);
>> > +		return rc;
>> > +	}
>> > +
>> > +	/* Skip undesired records */
>> > +	if (rec->cr.cr_index < crs->crs_start_offset)
>> > +		return 0;
>> > +
>> > +	CDEBUG(D_HSM, "%llu %02d%-5s %llu 0x%x t=" DFID " p=" DFID " %.*s\n",
>> > +	       rec->cr.cr_index, rec->cr.cr_type,
>> > +	       changelog_type2str(rec->cr.cr_type), rec->cr.cr_time,
>> > +	       rec->cr.cr_flags & CLF_FLAGMASK,
>> > +	       PFID(&rec->cr.cr_tfid), PFID(&rec->cr.cr_pfid),
>> > +	       rec->cr.cr_namelen, changelog_rec_name(&rec->cr));
>> > +
>> > +	wait_event_idle(crs->crs_waitq_prod,
>> > +			(crs->crs_rec_count < CDEV_CHLG_MAX_PREFETCH ||
>> > +			 crs->crs_closed));
>> > +
>> > +	if (crs->crs_closed)
>> > +		return LLOG_PROC_BREAK;
>> > +
>> > +	len = changelog_rec_size(&rec->cr) + rec->cr.cr_namelen;
>> > +	enq = kzalloc(sizeof(*enq) + len, GFP_KERNEL);
>> > +	if (!enq)
>> > +		return -ENOMEM;
>> > +
>> > +	INIT_LIST_HEAD(&enq->enq_linkage);
>> > +	enq->enq_length = len;
>> > +	memcpy(enq->enq_record, &rec->cr, len);
>> > +
>> > +	mutex_lock(&crs->crs_lock);
>> > +	list_add_tail(&enq->enq_linkage, &crs->crs_rec_queue);
>> > +	crs->crs_rec_count++;
>> > +	mutex_unlock(&crs->crs_lock);
>> > +
>> > +	wake_up_all(&crs->crs_waitq_cons);
>> > +
>> > +	return 0;
>> > +}
>> > +
>> > +/**
>> > + * Remove record from the list it is attached to and free it.
>> > + */
>> > +static void enq_record_delete(struct chlg_rec_entry *rec)
>> > +{
>> > +	list_del(&rec->enq_linkage);
>> > +	kfree(rec);
>> > +}
>> > +
>> > +/**
>> > + * Release resources associated to a changelog_reader_state instance.
>> > + *
>> > + * @param  crs  CRS instance to release.
>> > + */
>> > +static void crs_free(struct chlg_reader_state *crs)
>> > +{
>> > +	struct chlg_rec_entry *rec;
>> > +	struct chlg_rec_entry *tmp;
>> > +
>> > +	list_for_each_entry_safe(rec, tmp, &crs->crs_rec_queue, enq_linkage)
>> > +		enq_record_delete(rec);
>> > +
>> > +	kfree(crs);
>> > +}
>> > +
>> > +/**
>> > + * Record prefetch thread entry point. Opens the changelog catalog and starts
>> > + * reading records.
>> > + *
>> > + * @param[in,out]  args  chlg_reader_state passed from caller.
>> > + * @return 0 on success, negated error code on failure.
>> > + */
>> > +static int chlg_load(void *args)
>> > +{
>> > +	struct chlg_reader_state *crs = args;
>> > +	struct obd_device *obd = crs->crs_obd;
>> > +	struct llog_ctxt *ctx = NULL;
>> > +	struct llog_handle *llh = NULL;
>> > +	int rc;
>> > +
>> > +	ctx = llog_get_context(obd, LLOG_CHANGELOG_REPL_CTXT);
>> > +	if (!ctx) {
>> > +		rc = -ENOENT;
>> > +		goto err_out;
>> > +	}
>> > +
>> > +	rc = llog_open(NULL, ctx, &llh, NULL, CHANGELOG_CATALOG,
>> > +		       LLOG_OPEN_EXISTS);
>> > +	if (rc) {
>> > +		CERROR("%s: fail to open changelog catalog: rc = %d\n",
>> > +		       obd->obd_name, rc);
>> > +		goto err_out;
>> > +	}
>> > +
>> > +	rc = llog_init_handle(NULL, llh, LLOG_F_IS_CAT | LLOG_F_EXT_JOBID,
>> > +			      NULL);
>> > +	if (rc) {
>> > +		CERROR("%s: fail to init llog handle: rc = %d\n",
>> > +		       obd->obd_name, rc);
>> > +		goto err_out;
>> > +	}
>> > +
>> > +	rc = llog_cat_process(NULL, llh, chlg_read_cat_process_cb, crs, 0, 0);
>> > +	if (rc < 0) {
>> > +		CERROR("%s: fail to process llog: rc = %d\n",
>> > +		       obd->obd_name, rc);
>> > +		goto err_out;
>> > +	}
>> > +
>> > +err_out:
>> > +	crs->crs_err = true;
>> > +	wake_up_all(&crs->crs_waitq_cons);
>> > +
>> > +	if (llh)
>> > +		llog_cat_close(NULL, llh);
>> > +
>> > +	if (ctx)
>> > +		llog_ctxt_put(ctx);
>> > +
>> > +	wait_event_idle(crs->crs_waitq_prod, crs->crs_closed);
>> > +	crs_free(crs);
>> > +	return rc;
>> > +}
>> > +
>> > +/**
>> > + * Read handler, dequeues records from the chlg_reader_state if any.
>> > + * No partial records are copied to userland so this function can return less
>> > + * data than required (short read).
>> > + *
>> > + * @param[in]   file   File pointer to the character device.
>> > + * @param[out]  buff   Userland buffer where to copy the records.
>> > + * @param[in]   count  Userland buffer size.
>> > + * @param[out]  ppos   File position, updated with the index number of the next
>> > + *		       record to read.
>> > + * @return number of copied bytes on success, negated error code on failure.
>> > + */
>> > +static ssize_t chlg_read(struct file *file, char __user *buff, size_t count,
>> > +			 loff_t *ppos)
>> > +{
>> > +	struct chlg_reader_state *crs = file->private_data;
>> > +	struct chlg_rec_entry *rec;
>> > +	struct chlg_rec_entry *tmp;
>> > +	ssize_t  written_total = 0;
>> > +	LIST_HEAD(consumed);
>> > +
>> > +	if (file->f_flags & O_NONBLOCK && crs->crs_rec_count == 0)
>> > +		return -EAGAIN;
>> > +
>> > +	wait_event_idle(crs->crs_waitq_cons,
>> > +			crs->crs_rec_count > 0 || crs->crs_eof || crs->crs_err);
>> > +
>> > +	mutex_lock(&crs->crs_lock);
>> > +	list_for_each_entry_safe(rec, tmp, &crs->crs_rec_queue, enq_linkage) {
>> > +		if (written_total + rec->enq_length > count)
>> > +			break;
>> > +
>> > +		if (copy_to_user(buff, rec->enq_record, rec->enq_length)) {
>> > +			if (written_total == 0)
>> > +				written_total = -EFAULT;
>> > +			break;
>> > +		}
>> > +
>> > +		buff += rec->enq_length;
>> > +		written_total += rec->enq_length;
>> > +
>> > +		crs->crs_rec_count--;
>> > +		list_move_tail(&rec->enq_linkage, &consumed);
>> > +
>> > +		crs->crs_start_offset = rec->enq_record->cr_index + 1;
>> > +	}
>> > +	mutex_unlock(&crs->crs_lock);
>> > +
>> > +	if (written_total > 0)
>> > +		wake_up_all(&crs->crs_waitq_prod);
>> > +
>> > +	list_for_each_entry_safe(rec, tmp, &consumed, enq_linkage)
>> > +		enq_record_delete(rec);
>> > +
>> > +	*ppos = crs->crs_start_offset;
>> > +
>> > +	return written_total;
>> > +}
>> > +
>> > +/**
>> > + * Jump to a given record index. Helper for chlg_llseek().
>> > + *
>> > + * @param[in,out]  crs     Internal reader state.
>> > + * @param[in]      offset  Desired offset (index record).
>> > + * @return 0 on success, negated error code on failure.
>> > + */
>> > +static int chlg_set_start_offset(struct chlg_reader_state *crs, u64 offset)
>> > +{
>> > +	struct chlg_rec_entry *rec;
>> > +	struct chlg_rec_entry *tmp;
>> > +
>> > +	mutex_lock(&crs->crs_lock);
>> > +	if (offset < crs->crs_start_offset) {
>> > +		mutex_unlock(&crs->crs_lock);
>> > +		return -ERANGE;
>> > +	}
>> > +
>> > +	crs->crs_start_offset = offset;
>> > +	list_for_each_entry_safe(rec, tmp, &crs->crs_rec_queue, enq_linkage) {
>> > +		struct changelog_rec *cr = rec->enq_record;
>> > +
>> > +		if (cr->cr_index >= crs->crs_start_offset)
>> > +			break;
>> > +
>> > +		crs->crs_rec_count--;
>> > +		enq_record_delete(rec);
>> > +	}
>> > +
>> > +	mutex_unlock(&crs->crs_lock);
>> > +	wake_up_all(&crs->crs_waitq_prod);
>> > +	return 0;
>> > +}
>> > +
>> > +/**
>> > + * Move read pointer to a certain record index, encoded as an offset.
>> > + *
>> > + * @param[in,out] file   File pointer to the changelog character device
>> > + * @param[in]	  off    Offset to skip, actually a record index, not byte count
>> > + * @param[in]	  whence Relative/Absolute interpretation of the offset
>> > + * @return the resulting position on success or negated error code on failure.
>> > + */
>> > +static loff_t chlg_llseek(struct file *file, loff_t off, int whence)
>> > +{
>> > +	struct chlg_reader_state *crs = file->private_data;
>> > +	loff_t pos;
>> > +	int rc;
>> > +
>> > +	switch (whence) {
>> > +	case SEEK_SET:
>> > +		pos = off;
>> > +		break;
>> > +	case SEEK_CUR:
>> > +		pos = file->f_pos + off;
>> > +		break;
>> > +	case SEEK_END:
>> > +	default:
>> > +		return -EINVAL;
>> > +	}
>> > +
>> > +	/* We cannot go backward */
>> > +	if (pos < file->f_pos)
>> > +		return -EINVAL;
>> > +
>> > +	rc = chlg_set_start_offset(crs, pos);
>> > +	if (rc != 0)
>> > +		return rc;
>> > +
>> > +	file->f_pos = pos;
>> > +	return pos;
>> > +}
>> > +
>> > +/**
>> > + * Clear record range for a given changelog reader.
>> > + *
>> > + * @param[in]  crs     Current internal state.
>> > + * @param[in]  reader  Changelog reader ID (cl1, cl2...)
>> > + * @param[in]  record  Record index up which to clear
>> > + * @return 0 on success, negated error code on failure.
>> > + */
>> > +static int chlg_clear(struct chlg_reader_state *crs, u32 reader, u64 record)
>> > +{
>> > +	struct obd_device *obd = crs->crs_obd;
>> > +	struct changelog_setinfo cs  = {
>> > +		.cs_recno = record,
>> > +		.cs_id    = reader
>> > +	};
>> > +
>> > +	return obd_set_info_async(NULL, obd->obd_self_export,
>> > +				  strlen(KEY_CHANGELOG_CLEAR),
>> > +				  KEY_CHANGELOG_CLEAR, sizeof(cs), &cs, NULL);
>> > +}
>> > +
>> > +/** Maximum changelog control command size */
>> > +#define CHLG_CONTROL_CMD_MAX	64
>> > +
>> > +/**
>> > + * Handle writes() into the changelog character device. Write() can be used
>> > + * to request special control operations.
>> > + *
>> > + * @param[in]  file  File pointer to the changelog character device
>> > + * @param[in]  buff  User supplied data (written data)
>> > + * @param[in]  count Number of written bytes
>> > + * @param[in]  off   (unused)
>> > + * @return number of written bytes on success, negated error code on failure.
>> > + */
>> > +static ssize_t chlg_write(struct file *file, const char __user *buff,
>> > +			  size_t count, loff_t *off)
>> > +{
>> > +	struct chlg_reader_state *crs = file->private_data;
>> > +	char *kbuf;
>> > +	u64 record;
>> > +	u32 reader;
>> > +	int rc = 0;
>> > +
>> > +	if (count > CHLG_CONTROL_CMD_MAX)
>> > +		return -EINVAL;
>> > +
>> > +	kbuf = kzalloc(CHLG_CONTROL_CMD_MAX, GFP_KERNEL);
>> > +	if (!kbuf)
>> > +		return -ENOMEM;
>> > +
>> > +	if (copy_from_user(kbuf, buff, count)) {
>> > +		rc = -EFAULT;
>> > +		goto out_kbuf;
>> > +	}
>> > +
>> > +	kbuf[CHLG_CONTROL_CMD_MAX - 1] = '\0';
>> > +
>> > +	if (sscanf(kbuf, "clear:cl%u:%llu", &reader, &record) == 2)
>> > +		rc = chlg_clear(crs, reader, record);
>> > +	else
>> > +		rc = -EINVAL;
>> > +
>> > +out_kbuf:
>> > +	kfree(kbuf);
>> > +	return rc < 0 ? rc : count;
>> > +}
>> > +
>> > +/**
>> > + * Find the OBD device associated to a changelog character device.
>> > + * @param[in]  cdev  character device instance descriptor
>> > + * @return corresponding OBD device or NULL if none was found.
>> > + */
>> > +static struct obd_device *chlg_obd_get(dev_t cdev)
>> > +{
>> > +	int minor = MINOR(cdev);
>> > +	struct obd_device *obd = NULL;
>> > +	struct chlg_registered_dev *curr;
>> > +
>> > +	mutex_lock(&chlg_registered_dev_lock);
>> > +	list_for_each_entry(curr, &chlg_registered_devices, ced_link) {
>> > +		if (curr->ced_misc.minor == minor) {
>> > +			/* take the first available OBD device attached */
>> > +			obd = list_first_entry(&curr->ced_obds,
>> > +					       struct obd_device,
>> > +					       u.cli.cl_chg_dev_linkage);
>> > +			break;
>> > +		}
>> > +	}
>> > +	mutex_unlock(&chlg_registered_dev_lock);
>> > +	return obd;
>> > +}
>> > +
>> > +/**
>> > + * Open handler, initialize internal CRS state and spawn prefetch thread if
>> > + * needed.
>> > + * @param[in]  inode  Inode struct for the open character device.
>> > + * @param[in]  file   Corresponding file pointer.
>> > + * @return 0 on success, negated error code on failure.
>> > + */
>> > +static int chlg_open(struct inode *inode, struct file *file)
>> > +{
>> > +	struct chlg_reader_state *crs;
>> > +	struct obd_device *obd = chlg_obd_get(inode->i_rdev);
>> > +	struct task_struct *task;
>> > +	int rc;
>> > +
>> > +	if (!obd)
>> > +		return -ENODEV;
>> > +
>> > +	crs = kzalloc(sizeof(*crs), GFP_KERNEL);
>> > +	if (!crs)
>> > +		return -ENOMEM;
>> > +
>> > +	crs->crs_obd = obd;
>> > +	crs->crs_err = false;
>> > +	crs->crs_eof = false;
>> > +	crs->crs_closed = false;
>> > +
>> > +	mutex_init(&crs->crs_lock);
>> > +	INIT_LIST_HEAD(&crs->crs_rec_queue);
>> > +	init_waitqueue_head(&crs->crs_waitq_prod);
>> > +	init_waitqueue_head(&crs->crs_waitq_cons);
>> > +
>> > +	if (file->f_mode & FMODE_READ) {
>> > +		task = kthread_run(chlg_load, crs, "chlg_load_thread");
>> > +		if (IS_ERR(task)) {
>> > +			rc = PTR_ERR(task);
>> > +			CERROR("%s: cannot start changelog thread: rc = %d\n",
>> > +			       obd->obd_name, rc);
>> > +			goto err_crs;
>> > +		}
>> > +	}
>> > +
>> > +	file->private_data = crs;
>> > +	return 0;
>> > +
>> > +err_crs:
>> > +	kfree(crs);
>> > +	return rc;
>> > +}
>> > +
>> > +/**
>> > + * Close handler, release resources.
>> > + *
>> > + * @param[in]  inode  Inode struct for the open character device.
>> > + * @param[in]  file   Corresponding file pointer.
>> > + * @return 0 on success, negated error code on failure.
>> > + */
>> > +static int chlg_release(struct inode *inode, struct file *file)
>> > +{
>> > +	struct chlg_reader_state *crs = file->private_data;
>> > +
>> > +	if (file->f_mode & FMODE_READ) {
>> > +		crs->crs_closed = true;
>> > +		wake_up_all(&crs->crs_waitq_prod);
>> > +	} else {
>> > +		/* No producer thread, release resource ourselves */
>> > +		crs_free(crs);
>> > +	}
>> > +	return 0;
>> > +}
>> > +
>> > +/**
>> > + * Poll handler, indicates whether the device is readable (new records) and
>> > + * writable (always).
>> > + *
>> > + * @param[in]  file   Device file pointer.
>> > + * @param[in]  wait   (opaque)
>> > + * @return combination of the poll status flags.
>> > + */
>> > +static unsigned int chlg_poll(struct file *file, poll_table *wait)
>> > +{
>> > +	struct chlg_reader_state *crs  = file->private_data;
>> > +	unsigned int mask = 0;
>> > +
>> > +	mutex_lock(&crs->crs_lock);
>> > +	poll_wait(file, &crs->crs_waitq_cons, wait);
>> > +	if (crs->crs_rec_count > 0)
>> > +		mask |= POLLIN | POLLRDNORM;
>> > +	if (crs->crs_err)
>> > +		mask |= POLLERR;
>> > +	if (crs->crs_eof)
>> > +		mask |= POLLHUP;
>> > +	mutex_unlock(&crs->crs_lock);
>> > +	return mask;
>> > +}
>> > +
>> > +static const struct file_operations chlg_fops = {
>> > +	.owner		= THIS_MODULE,
>> > +	.llseek		= chlg_llseek,
>> > +	.read		= chlg_read,
>> > +	.write		= chlg_write,
>> > +	.open		= chlg_open,
>> > +	.release	= chlg_release,
>> > +	.poll		= chlg_poll,
>> > +};
>> > +
>> > +/**
>> > + * This uses obd_name of the form: "testfs-MDT0000-mdc-ffff88006501600"
>> > + * and returns a name of the form: "changelog-testfs-MDT0000".
>> > + */
>> > +static void get_chlg_name(char *name, size_t name_len, struct obd_device *obd)
>> > +{
>> > +	int i;
>> > +
>> > +	snprintf(name, name_len, "changelog-%s", obd->obd_name);
>> > +
>> > +	/* Find the 2nd '-' from the end and truncate on it */
>> > +	for (i = 0; i < 2; i++) {
>> > +		char *p = strrchr(name, '-');
>> > +
>> > +		if (!p)
>> > +			return;
>> > +		*p = '\0';
>> > +	}
>> > +}
>> > +
>> > +/**
>> > + * Find a changelog character device by name.
>> > + * All devices registered during MDC setup are listed in a global list with
>> > + * their names attached.
>> > + */
>> > +static struct chlg_registered_dev *
>> > +chlg_registered_dev_find_by_name(const char *name)
>> > +{
>> > +	struct chlg_registered_dev *dit;
>> > +
>> > +	list_for_each_entry(dit, &chlg_registered_devices, ced_link)
>> > +		if (strcmp(name, dit->ced_name) == 0)
>> > +			return dit;
>> > +	return NULL;
>> > +}
>> > +
>> > +/**
>> > + * Find chlg_registered_dev structure for a given OBD device.
>> > + * This is bad O(n^2) but for each filesystem:
>> > + *   - N is # of MDTs times # of mount points
>> > + *   - this only runs at shutdown
>> > + */
>> > +static struct chlg_registered_dev *
>> > +chlg_registered_dev_find_by_obd(const struct obd_device *obd)
>> > +{
>> > +	struct chlg_registered_dev *dit;
>> > +	struct obd_device *oit;
>> > +
>> > +	list_for_each_entry(dit, &chlg_registered_devices, ced_link)
>> > +		list_for_each_entry(oit, &dit->ced_obds,
>> > +				    u.cli.cl_chg_dev_linkage)
>> > +			if (oit == obd)
>> > +				return dit;
>> > +	return NULL;
>> > +}
>> > +
>> > +/**
>> > + * Changelog character device initialization.
>> > + * Register a misc character device with a dynamic minor number, under a name
>> > + * of the form: 'changelog-fsname-MDTxxxx'. Reference this OBD device with it.
>> > + *
>> > + * @param[in] obd  This MDC obd_device.
>> > + * @return 0 on success, negated error code on failure.
>> > + */
>> > +int mdc_changelog_cdev_init(struct obd_device *obd)
>> > +{
>> > +	struct chlg_registered_dev *exist;
>> > +	struct chlg_registered_dev *entry;
>> > +	int rc;
>> > +
>> > +	entry = kzalloc(sizeof(*entry), GFP_KERNEL);
>> > +	if (!entry)
>> > +		return -ENOMEM;
>> > +
>> > +	get_chlg_name(entry->ced_name, sizeof(entry->ced_name), obd);
>> > +
>> > +	entry->ced_misc.minor = MISC_DYNAMIC_MINOR;
>> > +	entry->ced_misc.name  = entry->ced_name;
>> > +	entry->ced_misc.fops  = &chlg_fops;
>> > +
>> > +	kref_init(&entry->ced_refs);
>> > +	INIT_LIST_HEAD(&entry->ced_obds);
>> > +	INIT_LIST_HEAD(&entry->ced_link);
>> > +
>> > +	mutex_lock(&chlg_registered_dev_lock);
>> > +	exist = chlg_registered_dev_find_by_name(entry->ced_name);
>> > +	if (exist) {
>> > +		kref_get(&exist->ced_refs);
>> > +		list_add_tail(&obd->u.cli.cl_chg_dev_linkage, &exist->ced_obds);
>> > +		rc = 0;
>> > +		goto out_unlock;
>> > +	}
>> > +
>> > +	/* Register new character device */
>> > +	rc = misc_register(&entry->ced_misc);
>> > +	if (rc != 0)
>> > +		goto out_unlock;
>> > +
>> > +	list_add_tail(&obd->u.cli.cl_chg_dev_linkage, &entry->ced_obds);
>> > +	list_add_tail(&entry->ced_link, &chlg_registered_devices);
>> > +
>> > +	entry = NULL;	/* prevent it from being freed below */
>> > +
>> > +out_unlock:
>> > +	mutex_unlock(&chlg_registered_dev_lock);
>> > +	kfree(entry);
>> > +	return rc;
>> > +}
>> > +
>> > +/**
>> > + * Deregister a changelog character device whose refcount has reached zero.
>> > + */
>> > +static void chlg_dev_clear(struct kref *kref)
>> > +{
>> > +	struct chlg_registered_dev *entry = container_of(kref,
>> > +							 struct chlg_registered_dev,
>> > +							 ced_refs);
>> > +	list_del(&entry->ced_link);
>> > +	misc_deregister(&entry->ced_misc);
>> > +	kfree(entry);
>> > +}
>> > +
>> > +/**
>> > + * Release OBD, decrease reference count of the corresponding changelog device.
>> > + */
>> > +void mdc_changelog_cdev_finish(struct obd_device *obd)
>> > +{
>> > +	struct chlg_registered_dev *dev = chlg_registered_dev_find_by_obd(obd);
>> > +
>> > +	mutex_lock(&chlg_registered_dev_lock);
>> > +	list_del_init(&obd->u.cli.cl_chg_dev_linkage);
>> > +	kref_put(&dev->ced_refs, chlg_dev_clear);
>> > +	mutex_unlock(&chlg_registered_dev_lock);
>> > +}
>> > diff --git a/drivers/staging/lustre/lustre/mdc/mdc_internal.h b/drivers/staging/lustre/lustre/mdc/mdc_internal.h
>> > index 941a896..6da9046 100644
>> > --- a/drivers/staging/lustre/lustre/mdc/mdc_internal.h
>> > +++ b/drivers/staging/lustre/lustre/mdc/mdc_internal.h
>> > @@ -129,6 +129,10 @@ enum ldlm_mode mdc_lock_match(struct obd_export *exp, __u64 flags,
>> >  			      enum ldlm_mode mode,
>> >  			      struct lustre_handle *lockh);
>> >  
>> > +int mdc_changelog_cdev_init(struct obd_device *obd);
>> > +
>> > +void mdc_changelog_cdev_finish(struct obd_device *obd);
>> > +
>> >  static inline int mdc_prep_elc_req(struct obd_export *exp,
>> >  				   struct ptlrpc_request *req, int opc,
>> >  				   struct list_head *cancels, int count)
>> > diff --git a/drivers/staging/lustre/lustre/mdc/mdc_request.c b/drivers/staging/lustre/lustre/mdc/mdc_request.c
>> > index 8f8e3d2..3692b1c 100644
>> > --- a/drivers/staging/lustre/lustre/mdc/mdc_request.c
>> > +++ b/drivers/staging/lustre/lustre/mdc/mdc_request.c
>> > @@ -35,7 +35,6 @@
>> >  
>> >  # include <linux/module.h>
>> >  # include <linux/pagemap.h>
>> > -# include <linux/miscdevice.h>
>> >  # include <linux/init.h>
>> >  # include <linux/utsname.h>
>> >  # include <linux/file.h>
>> > @@ -1810,174 +1809,6 @@ static int mdc_ioc_hsm_request(struct obd_export *exp,
>> >  	return rc;
>> >  }
>> >  
>> > -static struct kuc_hdr *changelog_kuc_hdr(char *buf, size_t len, u32 flags)
>> > -{
>> > -	struct kuc_hdr *lh = (struct kuc_hdr *)buf;
>> > -
>> > -	LASSERT(len <= KUC_CHANGELOG_MSG_MAXSIZE);
>> > -
>> > -	lh->kuc_magic = KUC_MAGIC;
>> > -	lh->kuc_transport = KUC_TRANSPORT_CHANGELOG;
>> > -	lh->kuc_flags = flags;
>> > -	lh->kuc_msgtype = CL_RECORD;
>> > -	lh->kuc_msglen = len;
>> > -	return lh;
>> > -}
>> > -
>> > -struct changelog_show {
>> > -	__u64		cs_startrec;
>> > -	enum changelog_send_flag	cs_flags;
>> > -	struct file	*cs_fp;
>> > -	char		*cs_buf;
>> > -	struct obd_device *cs_obd;
>> > -};
>> > -
>> > -static inline char *cs_obd_name(struct changelog_show *cs)
>> > -{
>> > -	return cs->cs_obd->obd_name;
>> > -}
>> > -
>> > -static int changelog_kkuc_cb(const struct lu_env *env, struct llog_handle *llh,
>> > -			     struct llog_rec_hdr *hdr, void *data)
>> > -{
>> > -	struct changelog_show *cs = data;
>> > -	struct llog_changelog_rec *rec = (struct llog_changelog_rec *)hdr;
>> > -	struct kuc_hdr *lh;
>> > -	size_t len;
>> > -	int rc;
>> > -
>> > -	if (rec->cr_hdr.lrh_type != CHANGELOG_REC) {
>> > -		rc = -EINVAL;
>> > -		CERROR("%s: not a changelog rec %x/%d: rc = %d\n",
>> > -		       cs_obd_name(cs), rec->cr_hdr.lrh_type,
>> > -		       rec->cr.cr_type, rc);
>> > -		return rc;
>> > -	}
>> > -
>> > -	if (rec->cr.cr_index < cs->cs_startrec) {
>> > -		/* Skip entries earlier than what we are interested in */
>> > -		CDEBUG(D_HSM, "rec=%llu start=%llu\n",
>> > -		       rec->cr.cr_index, cs->cs_startrec);
>> > -		return 0;
>> > -	}
>> > -
>> > -	CDEBUG(D_HSM, "%llu %02d%-5s %llu 0x%x t=" DFID " p=" DFID
>> > -		" %.*s\n", rec->cr.cr_index, rec->cr.cr_type,
>> > -		changelog_type2str(rec->cr.cr_type), rec->cr.cr_time,
>> > -		rec->cr.cr_flags & CLF_FLAGMASK,
>> > -		PFID(&rec->cr.cr_tfid), PFID(&rec->cr.cr_pfid),
>> > -		rec->cr.cr_namelen, changelog_rec_name(&rec->cr));
>> > -
>> > -	len = sizeof(*lh) + changelog_rec_size(&rec->cr) + rec->cr.cr_namelen;
>> > -
>> > -	/* Set up the message */
>> > -	lh = changelog_kuc_hdr(cs->cs_buf, len, cs->cs_flags);
>> > -	memcpy(lh + 1, &rec->cr, len - sizeof(*lh));
>> > -
>> > -	rc = libcfs_kkuc_msg_put(cs->cs_fp, lh);
>> > -	CDEBUG(D_HSM, "kucmsg fp %p len %zu rc %d\n", cs->cs_fp, len, rc);
>> > -
>> > -	return rc;
>> > -}
>> > -
>> > -static int mdc_changelog_send_thread(void *csdata)
>> > -{
>> > -	enum llog_flag flags = LLOG_F_IS_CAT;
>> > -	struct changelog_show *cs = csdata;
>> > -	struct llog_ctxt *ctxt = NULL;
>> > -	struct llog_handle *llh = NULL;
>> > -	struct kuc_hdr *kuch;
>> > -	int rc;
>> > -
>> > -	CDEBUG(D_HSM, "changelog to fp=%p start %llu\n",
>> > -	       cs->cs_fp, cs->cs_startrec);
>> > -
>> > -	cs->cs_buf = kzalloc(KUC_CHANGELOG_MSG_MAXSIZE, GFP_NOFS);
>> > -	if (!cs->cs_buf) {
>> > -		rc = -ENOMEM;
>> > -		goto out;
>> > -	}
>> > -
>> > -	/* Set up the remote catalog handle */
>> > -	ctxt = llog_get_context(cs->cs_obd, LLOG_CHANGELOG_REPL_CTXT);
>> > -	if (!ctxt) {
>> > -		rc = -ENOENT;
>> > -		goto out;
>> > -	}
>> > -	rc = llog_open(NULL, ctxt, &llh, NULL, CHANGELOG_CATALOG,
>> > -		       LLOG_OPEN_EXISTS);
>> > -	if (rc) {
>> > -		CERROR("%s: fail to open changelog catalog: rc = %d\n",
>> > -		       cs_obd_name(cs), rc);
>> > -		goto out;
>> > -	}
>> > -
>> > -	if (cs->cs_flags & CHANGELOG_FLAG_JOBID)
>> > -		flags |= LLOG_F_EXT_JOBID;
>> > -
>> > -	rc = llog_init_handle(NULL, llh, flags, NULL);
>> > -	if (rc) {
>> > -		CERROR("llog_init_handle failed %d\n", rc);
>> > -		goto out;
>> > -	}
>> > -
>> > -	rc = llog_cat_process(NULL, llh, changelog_kkuc_cb, cs, 0, 0);
>> > -
>> > -	/* Send EOF no matter what our result */
>> > -	kuch = changelog_kuc_hdr(cs->cs_buf, sizeof(*kuch), cs->cs_flags);
>> > -	kuch->kuc_msgtype = CL_EOF;
>> > -	libcfs_kkuc_msg_put(cs->cs_fp, kuch);
>> > -
>> > -out:
>> > -	fput(cs->cs_fp);
>> > -	if (llh)
>> > -		llog_cat_close(NULL, llh);
>> > -	if (ctxt)
>> > -		llog_ctxt_put(ctxt);
>> > -	kfree(cs->cs_buf);
>> > -	kfree(cs);
>> > -	return rc;
>> > -}
>> > -
>> > -static int mdc_ioc_changelog_send(struct obd_device *obd,
>> > -				  struct ioc_changelog *icc)
>> > -{
>> > -	struct changelog_show *cs;
>> > -	struct task_struct *task;
>> > -	int rc;
>> > -
>> > -	/* Freed in mdc_changelog_send_thread */
>> > -	cs = kzalloc(sizeof(*cs), GFP_NOFS);
>> > -	if (!cs)
>> > -		return -ENOMEM;
>> > -
>> > -	cs->cs_obd = obd;
>> > -	cs->cs_startrec = icc->icc_recno;
>> > -	/* matching fput in mdc_changelog_send_thread */
>> > -	cs->cs_fp = fget(icc->icc_id);
>> > -	cs->cs_flags = icc->icc_flags;
>> > -
>> > -	/*
>> > -	 * New thread because we should return to user app before
>> > -	 * writing into our pipe
>> > -	 */
>> > -	task = kthread_run(mdc_changelog_send_thread, cs,
>> > -			   "mdc_clg_send_thread");
>> > -	if (IS_ERR(task)) {
>> > -		rc = PTR_ERR(task);
>> > -		CERROR("%s: can't start changelog thread: rc = %d\n",
>> > -		       cs_obd_name(cs), rc);
>> > -		kfree(cs);
>> > -	} else {
>> > -		rc = 0;
>> > -		CDEBUG(D_HSM, "%s: started changelog thread\n",
>> > -		       cs_obd_name(cs));
>> > -	}
>> > -
>> > -	CERROR("Failed to start changelog thread: %d\n", rc);
>> > -	return rc;
>> > -}
>> > -
>> >  static int mdc_ioc_hsm_ct_start(struct obd_export *exp,
>> >  				struct lustre_kernelcomm *lk);
>> >  
>> > @@ -2087,21 +1918,6 @@ static int mdc_iocontrol(unsigned int cmd, struct obd_export *exp, int len,
>> >  		return -EINVAL;
>> >  	}
>> >  	switch (cmd) {
>> > -	case OBD_IOC_CHANGELOG_SEND:
>> > -		rc = mdc_ioc_changelog_send(obd, karg);
>> > -		goto out;
>> > -	case OBD_IOC_CHANGELOG_CLEAR: {
>> > -		struct ioc_changelog *icc = karg;
>> > -		struct changelog_setinfo cs = {
>> > -			.cs_recno = icc->icc_recno,
>> > -			.cs_id = icc->icc_id
>> > -		};
>> > -
>> > -		rc = obd_set_info_async(NULL, exp, strlen(KEY_CHANGELOG_CLEAR),
>> > -					KEY_CHANGELOG_CLEAR, sizeof(cs), &cs,
>> > -					NULL);
>> > -		goto out;
>> > -	}
>> >  	case OBD_IOC_FID2PATH:
>> >  		rc = mdc_ioc_fid2path(exp, karg);
>> >  		goto out;
>> > @@ -2670,12 +2486,22 @@ static int mdc_setup(struct obd_device *obd, struct lustre_cfg *cfg)
>> >  
>> >  	rc = mdc_llog_init(obd);
>> >  	if (rc) {
>> > -		CERROR("failed to setup llogging subsystems\n");
>> > +		CERROR("%s: failed to setup llogging subsystems: rc = %d\n",
>> > +		       obd->obd_name, rc);
>> >  		goto err_llog_cleanup;
>> >  	}
>> >  
>> > +	rc = mdc_changelog_cdev_init(obd);
>> > +	if (rc) {
>> > +		CERROR("%s: failed to setup changelog char device: rc = %d\n",
>> > +		       obd->obd_name, rc);
>> > +		goto err_changelog_cleanup;
>> > +	}
>> > +
>> >  	return 0;
>> >  
>> > +err_changelog_cleanup:
>> > +	mdc_llog_finish(obd);
>> >  err_llog_cleanup:
>> >  	ldebugfs_free_md_stats(obd);
>> >  	ptlrpc_lprocfs_unregister_obd(obd);
>> > @@ -2714,6 +2540,8 @@ static int mdc_precleanup(struct obd_device *obd)
>> >  	if (obd->obd_type->typ_refcnt <= 1)
>> >  		libcfs_kkuc_group_rem(0, KUC_GRP_HSM);
>> >  
>> > +	mdc_changelog_cdev_finish(obd);
>> > +
>> >  	obd_cleanup_client_import(obd);
>> >  	ptlrpc_lprocfs_unregister_obd(obd);
>> >  	lprocfs_obd_cleanup(obd);
>> > -- 
>> > 1.8.3.1
>> 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 832 bytes
Desc: not available
URL: <http://lists.lustre.org/pipermail/lustre-devel_lists.lustre.org/attachments/20181105/8e402c5f/attachment.sig>

From neilb at suse.com  Mon Nov  5 05:52:53 2018
From: neilb at suse.com (NeilBrown)
Date: Mon, 05 Nov 2018 16:52:53 +1100
Subject: [lustre-devel] drivers/staging updated to 4.20-rc1
Message-ID: <87k1lsyssq.fsf@notabene.neil.brown.name>


Hi,
 I've just pushed out new lustre/lustre, lustre/lustre-testing, lustre/lustre-wip
 to github.com:neilbrown/linux.git

 lustre is updated to 4.20-rc1 and doesn't compile because of some
 changes.
 The patch below makes it compile and (probably) work.

 I'd appreciate a review, especially of the iov_for_each change.

Thanks,
NeilBrown


Subject: [PATCH] lustre: fixes for 4.20-rc1

Most are trivial.
iov_for_each() is broken upstream and there are no users in mainline,
so it seem to make sense to just open-code it in the one place
we use it.

Signed-off-by: NeilBrown <neilb at suse.com>
---
 drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.h | 4 ++--
 drivers/staging/lustre/lustre/llite/vvp_io.c        | 5 ++++-
 drivers/staging/lustre/lustre/mdc/mdc_request.c     | 2 +-
 3 files changed, 7 insertions(+), 4 deletions(-)

diff --git a/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.h b/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.h
index c882345226e1..7f81fe24c5eb 100644
--- a/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.h
+++ b/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.h
@@ -478,7 +478,7 @@ struct kib_rx {					/* receive message */
 	enum ib_wc_status      rx_status;     /* completion status */
 	struct kib_msg		*rx_msg;	/* message buffer (host vaddr) */
 	__u64                  rx_msgaddr;    /* message buffer (I/O addr) */
-	DECLARE_PCI_UNMAP_ADDR(rx_msgunmap);  /* for dma_unmap_single() */
+	DEFINE_DMA_UNMAP_ADDR(rx_msgunmap);  /* for dma_unmap_single() */
 	struct ib_recv_wr      rx_wrq;        /* receive work item... */
 	struct ib_sge          rx_sge;        /* ...and its memory */
 };
@@ -501,7 +501,7 @@ struct kib_tx {					/* transmit message */
 	struct lnet_msg		*tx_lntmsg[2];	/* lnet msgs to finalize on completion */
 	struct kib_msg	      *tx_msg;        /* message buffer (host vaddr) */
 	__u64                 tx_msgaddr;     /* message buffer (I/O addr) */
-	DECLARE_PCI_UNMAP_ADDR(tx_msgunmap);  /* for dma_unmap_single() */
+	DEFINE_DMA_UNMAP_ADDR(tx_msgunmap);  /* for dma_unmap_single() */
 	/** sge for tx_msgaddr */
 	struct ib_sge		tx_msgsge;
 	int                   tx_nwrq;        /* # send work items */
diff --git a/drivers/staging/lustre/lustre/llite/vvp_io.c b/drivers/staging/lustre/lustre/llite/vvp_io.c
index 70d23874b674..524f3f4feec8 100644
--- a/drivers/staging/lustre/lustre/llite/vvp_io.c
+++ b/drivers/staging/lustre/lustre/llite/vvp_io.c
@@ -388,7 +388,10 @@ static int vvp_mmap_locks(const struct lu_env *env,
 	if (!mm)
 		return 0;
 
-	iov_for_each(iov, i, *vio->vui_iter) {
+	for (i = *vio->vui_iter;
+	     i.count;
+	     iov_iter_advance(&i, iov.iov_len)) {
+		iov = iov_iter_iovec(&i);
 		addr = (unsigned long)iov.iov_base;
 		count = iov.iov_len;
 		if (count == 0)
diff --git a/drivers/staging/lustre/lustre/mdc/mdc_request.c b/drivers/staging/lustre/lustre/mdc/mdc_request.c
index 8f8e3d220dc8..2242204f3c9e 100644
--- a/drivers/staging/lustre/lustre/mdc/mdc_request.c
+++ b/drivers/staging/lustre/lustre/mdc/mdc_request.c
@@ -993,7 +993,7 @@ static struct page *mdc_page_locate(struct address_space *mapping, __u64 *hash,
 	xa_lock_irq(&mapping->i_pages);
 	found = radix_tree_gang_lookup(&mapping->i_pages,
 				       (void **)&page, offset, 1);
-	if (found > 0 && !radix_tree_exceptional_entry(page)) {
+	if (found > 0 && !xa_is_value(page)) {
 		struct lu_dirpage *dp;
 
 		get_page(page);
-- 
2.14.0.rc0.dirty

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 832 bytes
Desc: not available
URL: <http://lists.lustre.org/pipermail/lustre-devel_lists.lustre.org/attachments/20181105/0932c5fa/attachment.sig>

From quentin.bouget at cea.fr  Mon Nov  5 13:37:33 2018
From: quentin.bouget at cea.fr (quentin.bouget at cea.fr)
Date: Mon, 5 Nov 2018 14:37:33 +0100
Subject: [lustre-devel] [PATCH v2] lustre: mdc: fix possible deadlock in
 chlg_open()
In-Reply-To: <alpine.LFD.2.21.1811042131490.27308@casper.infradead.org>
References: <87bm7a3nue.fsf@notabene.neil.brown.name>
 <7db27e7d-e685-af8f-80d8-891a0d8db4d5@cea.fr>
 <875zxh3elu.fsf@notabene.neil.brown.name>
 <alpine.LFD.2.21.1811042131490.27308@casper.infradead.org>
Message-ID: <88ccd141-4778-e3ab-3ce0-e7cbd5c290fc@cea.fr>

Le 04/11/2018 à 22:34, James Simmons a écrit :
>> Lockdep reports a possible deadlock between chlg_open() and
>> mdc_changelog_cdev_init()
>>
>> mdc_changelog_cdev_init() takes chlg_registered_dev_lock and then
>> calls misc_register() which takes misc_mtx.
>> chlg_open() is called while misc_mtx is held, and tries to take
>> chlg_registered_dev_lock.
>> If these two functions race, a deadlock can occur as each thread will
>> hold one of the locks while trying to take the other.
>>
>> chlg_open() does not need to take a lock.  It only uses the
>> lock to stablize a list while looking for the matching
>> chlg_registered_dev, and this can be found directly by examining
>> file->private_data.
>>
>> So remove chlg_obd_get(), and use file->private_data to find the
>> obd_device.
>> Also ensure the device is fully initialized before calling
>> misc_register().  This means setting up some list linkage before the
>> call, and tearing it down if there is an error.
> I have been testing this but I'm no HSM expert. I pushed this patch
> to OpenSFS branch as well.
>
> https://jira.whamcloud.com/browse/LU-11617
> https://review.whamcloud.com/#/c/33572/
>
> Reviewed-by: James Simmons <jsimmons at infradead.org>

Reviewed-by: Quentin Bouget <quentin.bouget at cea.fr>

>
>> Signed-off-by: NeilBrown <neilb at suse.com>
>> ---
>>
>> This is the revised version with the problem identified by Quentin
>> fixed.
>>
>>   drivers/staging/lustre/lustre/mdc/mdc_changelog.c | 46 +++++++----------------
>>   1 file changed, 14 insertions(+), 32 deletions(-)
>>
>> diff --git a/drivers/staging/lustre/lustre/mdc/mdc_changelog.c b/drivers/staging/lustre/lustre/mdc/mdc_changelog.c
>> index d83507cbf95c..af29ea73c48a 100644
>> --- a/drivers/staging/lustre/lustre/mdc/mdc_changelog.c
>> +++ b/drivers/staging/lustre/lustre/mdc/mdc_changelog.c
>> @@ -444,31 +444,6 @@ static ssize_t chlg_write(struct file *file, const char __user *buff,
>>   	return rc < 0 ? rc : count;
>>   }
>>   
>> -/**
>> - * Find the OBD device associated to a changelog character device.
>> - * @param[in]  cdev  character device instance descriptor
>> - * @return corresponding OBD device or NULL if none was found.
>> - */
>> -static struct obd_device *chlg_obd_get(dev_t cdev)
>> -{
>> -	int minor = MINOR(cdev);
>> -	struct obd_device *obd = NULL;
>> -	struct chlg_registered_dev *curr;
>> -
>> -	mutex_lock(&chlg_registered_dev_lock);
>> -	list_for_each_entry(curr, &chlg_registered_devices, ced_link) {
>> -		if (curr->ced_misc.minor == minor) {
>> -			/* take the first available OBD device attached */
>> -			obd = list_first_entry(&curr->ced_obds,
>> -					       struct obd_device,
>> -					       u.cli.cl_chg_dev_linkage);
>> -			break;
>> -		}
>> -	}
>> -	mutex_unlock(&chlg_registered_dev_lock);
>> -	return obd;
>> -}
>> -
>>   /**
>>    * Open handler, initialize internal CRS state and spawn prefetch thread if
>>    * needed.
>> @@ -479,12 +454,16 @@ static struct obd_device *chlg_obd_get(dev_t cdev)
>>   static int chlg_open(struct inode *inode, struct file *file)
>>   {
>>   	struct chlg_reader_state *crs;
>> -	struct obd_device *obd = chlg_obd_get(inode->i_rdev);
>> +	struct miscdevice *misc = file->private_data;
>> +	struct chlg_registered_dev *dev;
>> +	struct obd_device *obd;
>>   	struct task_struct *task;
>>   	int rc;
>>   
>> -	if (!obd)
>> -		return -ENODEV;
>> +	dev = container_of(misc, struct chlg_registered_dev, ced_misc);
>> +	obd = list_first_entry(&dev->ced_obds,
>> +			       struct obd_device,
>> +			       u.cli.cl_chg_dev_linkage);
>>   
>>   	crs = kzalloc(sizeof(*crs), GFP_KERNEL);
>>   	if (!crs)
>> @@ -669,13 +648,16 @@ int mdc_changelog_cdev_init(struct obd_device *obd)
>>   		goto out_unlock;
>>   	}
>>   
>> +	list_add_tail(&obd->u.cli.cl_chg_dev_linkage, &entry->ced_obds);
>> +	list_add_tail(&entry->ced_link, &chlg_registered_devices);
>> +
>>   	/* Register new character device */
>>   	rc = misc_register(&entry->ced_misc);
>> -	if (rc != 0)
>> +	if (rc != 0) {
>> +		list_del_init(&obd->u.cli.cl_chg_dev_linkage);
>> +		list_del(&entry->ced_link);
>>   		goto out_unlock;
>> -
>> -	list_add_tail(&obd->u.cli.cl_chg_dev_linkage, &entry->ced_obds);
>> -	list_add_tail(&entry->ced_link, &chlg_registered_devices);
>> +	}
>>   
>>   	entry = NULL;	/* prevent it from being freed below */
>>   
>> -- 
>> 2.14.0.rc0.dirty
>>
>>


From jsimmons at infradead.org  Mon Nov  5 18:18:07 2018
From: jsimmons at infradead.org (James Simmons)
Date: Mon,  5 Nov 2018 13:18:07 -0500
Subject: [lustre-devel] [PATCH] lustre: remove iter type from
 iov_iter_[b|k]vec()
Message-ID: <1541441887-15064-1-git-send-email-jsimmons@infradead.org>

The linux commit aa563d7bca6e ("iov_iter: Separate type from
direction and use accessor functions) removed the iter type
from both iov_iter_bvec() and iov_iter_kvec(). Update lustre
to this change. Without it we see in testing:

WARNING: CPU: 2 PID: 3088 at lib/iov_iter.c:1082 iov_iter_kvec+0x25/0x30

Signed-off-by: James Simmons <jsimmons at infradead.org>
---
 drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd_cb.c  | 4 ++--
 drivers/staging/lustre/lnet/klnds/socklnd/socklnd_cb.c  | 8 ++++----
 drivers/staging/lustre/lnet/klnds/socklnd/socklnd_lib.c | 4 ++--
 drivers/staging/lustre/lnet/lnet/lib-move.c             | 4 ++--
 drivers/staging/lustre/lnet/lnet/lib-socket.c           | 4 ++--
 5 files changed, 12 insertions(+), 12 deletions(-)

diff --git a/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd_cb.c b/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd_cb.c
index 23ce59e..57fe037 100644
--- a/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd_cb.c
+++ b/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd_cb.c
@@ -1548,11 +1548,11 @@ static int kiblnd_resolve_addr(struct rdma_cm_id *cmid,
 	LASSERT(!(payload_kiov && payload_iov));
 
 	if (payload_kiov)
-		iov_iter_bvec(&from, ITER_BVEC | WRITE,
+		iov_iter_bvec(&from, WRITE,
 			      payload_kiov, payload_niov,
 			      payload_nob + payload_offset);
 	else
-		iov_iter_kvec(&from, ITER_KVEC | WRITE,
+		iov_iter_kvec(&from, WRITE,
 			      payload_iov, payload_niov,
 			      payload_nob + payload_offset);
 
diff --git a/drivers/staging/lustre/lnet/klnds/socklnd/socklnd_cb.c b/drivers/staging/lustre/lnet/klnds/socklnd/socklnd_cb.c
index c401896..4abf0eb 100644
--- a/drivers/staging/lustre/lnet/klnds/socklnd/socklnd_cb.c
+++ b/drivers/staging/lustre/lnet/klnds/socklnd/socklnd_cb.c
@@ -1001,7 +1001,7 @@ struct ksock_route *
 			kvec->iov_base = &conn->ksnc_msg;
 			kvec->iov_len = offsetof(struct ksock_msg, ksm_u);
 			conn->ksnc_rx_nob_left = offsetof(struct ksock_msg, ksm_u);
-			iov_iter_kvec(&conn->ksnc_rx_to, READ|ITER_KVEC, kvec,
+			iov_iter_kvec(&conn->ksnc_rx_to, READ, kvec,
 					1, offsetof(struct ksock_msg, ksm_u));
 			break;
 
@@ -1011,7 +1011,7 @@ struct ksock_route *
 			kvec->iov_base = &conn->ksnc_msg.ksm_u.lnetmsg;
 			kvec->iov_len = sizeof(struct lnet_hdr);
 			conn->ksnc_rx_nob_left = sizeof(struct lnet_hdr);
-			iov_iter_kvec(&conn->ksnc_rx_to, READ|ITER_KVEC, kvec,
+			iov_iter_kvec(&conn->ksnc_rx_to, READ, kvec,
 					1, sizeof(struct lnet_hdr));
 			break;
 
@@ -1043,7 +1043,7 @@ struct ksock_route *
 	} while (nob_to_skip &&    /* mustn't overflow conn's rx iov */
 		 niov < sizeof(conn->ksnc_rx_iov_space) / sizeof(struct iovec));
 
-	iov_iter_kvec(&conn->ksnc_rx_to, READ|ITER_KVEC, kvec, niov, skipped);
+	iov_iter_kvec(&conn->ksnc_rx_to, READ, kvec, niov, skipped);
 	return 0;
 }
 
@@ -1157,7 +1157,7 @@ struct ksock_route *
 		kvec->iov_base = &conn->ksnc_msg.ksm_u.lnetmsg;
 		kvec->iov_len = sizeof(struct ksock_lnet_msg);
 
-		iov_iter_kvec(&conn->ksnc_rx_to, READ|ITER_KVEC, kvec,
+		iov_iter_kvec(&conn->ksnc_rx_to, READ, kvec,
 				1, sizeof(struct ksock_lnet_msg));
 
 		goto again;     /* read lnet header now */
diff --git a/drivers/staging/lustre/lnet/klnds/socklnd/socklnd_lib.c b/drivers/staging/lustre/lnet/klnds/socklnd/socklnd_lib.c
index 33847b9..686c2d3 100644
--- a/drivers/staging/lustre/lnet/klnds/socklnd/socklnd_lib.c
+++ b/drivers/staging/lustre/lnet/klnds/socklnd/socklnd_lib.c
@@ -92,7 +92,7 @@
 	    nob < tx->tx_resid)
 		msg.msg_flags |= MSG_MORE;
 
-	iov_iter_kvec(&msg.msg_iter, WRITE | ITER_KVEC,
+	iov_iter_kvec(&msg.msg_iter, WRITE,
 		      tx->tx_iov, tx->tx_niov, nob);
 	return sock_sendmsg(sock, &msg);
 }
@@ -140,7 +140,7 @@
 		    nob < tx->tx_resid)
 			msg.msg_flags |= MSG_MORE;
 
-		iov_iter_bvec(&msg.msg_iter, WRITE | ITER_BVEC,
+		iov_iter_bvec(&msg.msg_iter, WRITE,
 			      kiov, tx->tx_nkiov, nob);
 		rc = sock_sendmsg(sock, &msg);
 	}
diff --git a/drivers/staging/lustre/lnet/lnet/lib-move.c b/drivers/staging/lustre/lnet/lnet/lib-move.c
index 5694d85..eaa1dfa 100644
--- a/drivers/staging/lustre/lnet/lnet/lib-move.c
+++ b/drivers/staging/lustre/lnet/lnet/lib-move.c
@@ -498,10 +498,10 @@ void lnet_usr_translate_stats(struct lnet_ioctl_element_msg_stats *msg_stats,
 	}
 
 	if (iov) {
-		iov_iter_kvec(&to, ITER_KVEC | READ, iov, niov, mlen + offset);
+		iov_iter_kvec(&to, READ, iov, niov, mlen + offset);
 		iov_iter_advance(&to, offset);
 	} else {
-		iov_iter_bvec(&to, ITER_BVEC | READ, kiov, niov, mlen + offset);
+		iov_iter_bvec(&to, READ, kiov, niov, mlen + offset);
 		iov_iter_advance(&to, offset);
 	}
 	rc = ni->ni_net->net_lnd->lnd_recv(ni, private, msg, delayed, &to, rlen);
diff --git a/drivers/staging/lustre/lnet/lnet/lib-socket.c b/drivers/staging/lustre/lnet/lnet/lib-socket.c
index d9c62d3..62a742e 100644
--- a/drivers/staging/lustre/lnet/lnet/lib-socket.c
+++ b/drivers/staging/lustre/lnet/lnet/lib-socket.c
@@ -58,7 +58,7 @@
 	 * Caller may pass a zero timeout if she thinks the socket buffer is
 	 * empty enough to take the whole message immediately
 	 */
-	iov_iter_kvec(&msg.msg_iter, WRITE | ITER_KVEC, &iov, 1, nob);
+	iov_iter_kvec(&msg.msg_iter, WRITE, &iov, 1, nob);
 	for (;;) {
 		msg.msg_flags = !timeout ? MSG_DONTWAIT : 0;
 		if (timeout) {
@@ -113,7 +113,7 @@
 	LASSERT(nob > 0);
 	LASSERT(jiffies_left > 0);
 
-	iov_iter_kvec(&msg.msg_iter, READ | ITER_KVEC, &iov, 1, nob);
+	iov_iter_kvec(&msg.msg_iter, READ, &iov, 1, nob);
 
 	for (;;) {
 		/* Set receive timeout to remaining time */
-- 
1.8.3.1


From adilger at whamcloud.com  Mon Nov  5 19:50:19 2018
From: adilger at whamcloud.com (Andreas Dilger)
Date: Mon, 5 Nov 2018 19:50:19 +0000
Subject: [lustre-devel] [PATCH] lustre: remove iter type from
 iov_iter_[b|k]vec()
In-Reply-To: <1541441887-15064-1-git-send-email-jsimmons@infradead.org>
References: <1541441887-15064-1-git-send-email-jsimmons@infradead.org>
Message-ID: <91C563BF-78C7-4409-B0BC-5048D689A309@whamcloud.com>

On Nov 5, 2018, at 11:18, James Simmons <jsimmons at infradead.org> wrote:
> 
> The linux commit aa563d7bca6e ("iov_iter: Separate type from
> direction and use accessor functions) removed the iter type
> from both iov_iter_bvec() and iov_iter_kvec(). Update lustre
> to this change. Without it we see in testing:
> 
> WARNING: CPU: 2 PID: 3088 at lib/iov_iter.c:1082 iov_iter_kvec+0x25/0x30

I'm no LNet expert, but it would seem that just removing the ITER_KVEC
flag would cause problems with the code?  As recently as the 4.17 kernel
it looks like removing this flag would cause a BUG_ON():

void iov_iter_kvec(struct iov_iter *i, int direction,
                        const struct kvec *kvec, unsigned long nr_segs,
                        size_t count)
{
        BUG_ON(!(direction & ITER_KVEC));
        i->type = direction;
        i->kvec = kvec;
        i->nr_segs = nr_segs;
        i->iov_offset = 0;
        i->count = count;
}

I suspect what you need to do is something like:

#if <something>
#define LNET_ITER_KVEC 0
#else
#define LNET_ITER_KVEC ITER_KVEC
#endif

so that it works on both old and new kernels, at least for the version
that is included in the master repo.

Cheers, Andreas


> Signed-off-by: James Simmons <jsimmons at infradead.org>
> ---
> drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd_cb.c  | 4 ++--
> drivers/staging/lustre/lnet/klnds/socklnd/socklnd_cb.c  | 8 ++++----
> drivers/staging/lustre/lnet/klnds/socklnd/socklnd_lib.c | 4 ++--
> drivers/staging/lustre/lnet/lnet/lib-move.c             | 4 ++--
> drivers/staging/lustre/lnet/lnet/lib-socket.c           | 4 ++--
> 5 files changed, 12 insertions(+), 12 deletions(-)
> 
> diff --git a/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd_cb.c b/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd_cb.c
> index 23ce59e..57fe037 100644
> --- a/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd_cb.c
> +++ b/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd_cb.c
> @@ -1548,11 +1548,11 @@ static int kiblnd_resolve_addr(struct rdma_cm_id *cmid,
> 	LASSERT(!(payload_kiov && payload_iov));
> 
> 	if (payload_kiov)
> -		iov_iter_bvec(&from, ITER_BVEC | WRITE,
> +		iov_iter_bvec(&from, WRITE,
> 			      payload_kiov, payload_niov,
> 			      payload_nob + payload_offset);
> 	else
> -		iov_iter_kvec(&from, ITER_KVEC | WRITE,
> +		iov_iter_kvec(&from, WRITE,
> 			      payload_iov, payload_niov,
> 			      payload_nob + payload_offset);
> 
> diff --git a/drivers/staging/lustre/lnet/klnds/socklnd/socklnd_cb.c b/drivers/staging/lustre/lnet/klnds/socklnd/socklnd_cb.c
> index c401896..4abf0eb 100644
> --- a/drivers/staging/lustre/lnet/klnds/socklnd/socklnd_cb.c
> +++ b/drivers/staging/lustre/lnet/klnds/socklnd/socklnd_cb.c
> @@ -1001,7 +1001,7 @@ struct ksock_route *
> 			kvec->iov_base = &conn->ksnc_msg;
> 			kvec->iov_len = offsetof(struct ksock_msg, ksm_u);
> 			conn->ksnc_rx_nob_left = offsetof(struct ksock_msg, ksm_u);
> -			iov_iter_kvec(&conn->ksnc_rx_to, READ|ITER_KVEC, kvec,
> +			iov_iter_kvec(&conn->ksnc_rx_to, READ, kvec,
> 					1, offsetof(struct ksock_msg, ksm_u));
> 			break;
> 
> @@ -1011,7 +1011,7 @@ struct ksock_route *
> 			kvec->iov_base = &conn->ksnc_msg.ksm_u.lnetmsg;
> 			kvec->iov_len = sizeof(struct lnet_hdr);
> 			conn->ksnc_rx_nob_left = sizeof(struct lnet_hdr);
> -			iov_iter_kvec(&conn->ksnc_rx_to, READ|ITER_KVEC, kvec,
> +			iov_iter_kvec(&conn->ksnc_rx_to, READ, kvec,
> 					1, sizeof(struct lnet_hdr));
> 			break;
> 
> @@ -1043,7 +1043,7 @@ struct ksock_route *
> 	} while (nob_to_skip &&    /* mustn't overflow conn's rx iov */
> 		 niov < sizeof(conn->ksnc_rx_iov_space) / sizeof(struct iovec));
> 
> -	iov_iter_kvec(&conn->ksnc_rx_to, READ|ITER_KVEC, kvec, niov, skipped);
> +	iov_iter_kvec(&conn->ksnc_rx_to, READ, kvec, niov, skipped);
> 	return 0;
> }
> 
> @@ -1157,7 +1157,7 @@ struct ksock_route *
> 		kvec->iov_base = &conn->ksnc_msg.ksm_u.lnetmsg;
> 		kvec->iov_len = sizeof(struct ksock_lnet_msg);
> 
> -		iov_iter_kvec(&conn->ksnc_rx_to, READ|ITER_KVEC, kvec,
> +		iov_iter_kvec(&conn->ksnc_rx_to, READ, kvec,
> 				1, sizeof(struct ksock_lnet_msg));
> 
> 		goto again;     /* read lnet header now */
> diff --git a/drivers/staging/lustre/lnet/klnds/socklnd/socklnd_lib.c b/drivers/staging/lustre/lnet/klnds/socklnd/socklnd_lib.c
> index 33847b9..686c2d3 100644
> --- a/drivers/staging/lustre/lnet/klnds/socklnd/socklnd_lib.c
> +++ b/drivers/staging/lustre/lnet/klnds/socklnd/socklnd_lib.c
> @@ -92,7 +92,7 @@
> 	    nob < tx->tx_resid)
> 		msg.msg_flags |= MSG_MORE;
> 
> -	iov_iter_kvec(&msg.msg_iter, WRITE | ITER_KVEC,
> +	iov_iter_kvec(&msg.msg_iter, WRITE,
> 		      tx->tx_iov, tx->tx_niov, nob);
> 	return sock_sendmsg(sock, &msg);
> }
> @@ -140,7 +140,7 @@
> 		    nob < tx->tx_resid)
> 			msg.msg_flags |= MSG_MORE;
> 
> -		iov_iter_bvec(&msg.msg_iter, WRITE | ITER_BVEC,
> +		iov_iter_bvec(&msg.msg_iter, WRITE,
> 			      kiov, tx->tx_nkiov, nob);
> 		rc = sock_sendmsg(sock, &msg);
> 	}
> diff --git a/drivers/staging/lustre/lnet/lnet/lib-move.c b/drivers/staging/lustre/lnet/lnet/lib-move.c
> index 5694d85..eaa1dfa 100644
> --- a/drivers/staging/lustre/lnet/lnet/lib-move.c
> +++ b/drivers/staging/lustre/lnet/lnet/lib-move.c
> @@ -498,10 +498,10 @@ void lnet_usr_translate_stats(struct lnet_ioctl_element_msg_stats *msg_stats,
> 	}
> 
> 	if (iov) {
> -		iov_iter_kvec(&to, ITER_KVEC | READ, iov, niov, mlen + offset);
> +		iov_iter_kvec(&to, READ, iov, niov, mlen + offset);
> 		iov_iter_advance(&to, offset);
> 	} else {
> -		iov_iter_bvec(&to, ITER_BVEC | READ, kiov, niov, mlen + offset);
> +		iov_iter_bvec(&to, READ, kiov, niov, mlen + offset);
> 		iov_iter_advance(&to, offset);
> 	}
> 	rc = ni->ni_net->net_lnd->lnd_recv(ni, private, msg, delayed, &to, rlen);
> diff --git a/drivers/staging/lustre/lnet/lnet/lib-socket.c b/drivers/staging/lustre/lnet/lnet/lib-socket.c
> index d9c62d3..62a742e 100644
> --- a/drivers/staging/lustre/lnet/lnet/lib-socket.c
> +++ b/drivers/staging/lustre/lnet/lnet/lib-socket.c
> @@ -58,7 +58,7 @@
> 	 * Caller may pass a zero timeout if she thinks the socket buffer is
> 	 * empty enough to take the whole message immediately
> 	 */
> -	iov_iter_kvec(&msg.msg_iter, WRITE | ITER_KVEC, &iov, 1, nob);
> +	iov_iter_kvec(&msg.msg_iter, WRITE, &iov, 1, nob);
> 	for (;;) {
> 		msg.msg_flags = !timeout ? MSG_DONTWAIT : 0;
> 		if (timeout) {
> @@ -113,7 +113,7 @@
> 	LASSERT(nob > 0);
> 	LASSERT(jiffies_left > 0);
> 
> -	iov_iter_kvec(&msg.msg_iter, READ | ITER_KVEC, &iov, 1, nob);
> +	iov_iter_kvec(&msg.msg_iter, READ, &iov, 1, nob);
> 
> 	for (;;) {
> 		/* Set receive timeout to remaining time */
> -- 
> 1.8.3.1
> 

Cheers, Andreas
---
Andreas Dilger
CTO Whamcloud





From jsimmons at infradead.org  Mon Nov  5 19:56:30 2018
From: jsimmons at infradead.org (James Simmons)
Date: Mon, 5 Nov 2018 19:56:30 +0000 (GMT)
Subject: [lustre-devel] [PATCH] lustre: remove iter type from
 iov_iter_[b|k]vec()
In-Reply-To: <91C563BF-78C7-4409-B0BC-5048D689A309@whamcloud.com>
References: <1541441887-15064-1-git-send-email-jsimmons@infradead.org>
 <91C563BF-78C7-4409-B0BC-5048D689A309@whamcloud.com>
Message-ID: <alpine.LFD.2.21.1811051953160.11915@casper.infradead.org>


> On Nov 5, 2018, at 11:18, James Simmons <jsimmons at infradead.org> wrote:
> > 
> > The linux commit aa563d7bca6e ("iov_iter: Separate type from
> > direction and use accessor functions) removed the iter type
> > from both iov_iter_bvec() and iov_iter_kvec(). Update lustre
> > to this change. Without it we see in testing:
> > 
> > WARNING: CPU: 2 PID: 3088 at lib/iov_iter.c:1082 iov_iter_kvec+0x25/0x30
> 
> I'm no LNet expert, but it would seem that just removing the ITER_KVEC
> flag would cause problems with the code?  As recently as the 4.17 kernel
> it looks like removing this flag would cause a BUG_ON():
> 
> void iov_iter_kvec(struct iov_iter *i, int direction,
>                         const struct kvec *kvec, unsigned long nr_segs,
>                         size_t count)
> {
>         BUG_ON(!(direction & ITER_KVEC));
>         i->type = direction;
>         i->kvec = kvec;
>         i->nr_segs = nr_segs;
>         i->iov_offset = 0;
>         i->count = count;
> }
> 
> I suspect what you need to do is something like:
> 
> #if <something>
> #define LNET_ITER_KVEC 0
> #else
> #define LNET_ITER_KVEC ITER_KVEC
> #endif
> 
> so that it works on both old and new kernels, at least for the version
> that is included in the master repo.
> 
> Cheers, Andreas

For the OpenSFS branch yes we will have to do something like that to 
support various kernel releases. The change was just landed in the 
4.20-rc1 cycle. iov_iter_kvec is now:

void iov_iter_kvec(struct iov_iter *i, unsigned int direction,
                        const struct kvec *kvec, unsigned long nr_segs,
                        size_t count)
{
        WARN_ON(direction & ~(READ | WRITE));
        i->type = ITER_KVEC | (direction & (READ | WRITE));
        i->kvec = kvec;
        i->nr_segs = nr_segs;
        i->iov_offset = 0;
        i->count = count;
}
EXPORT_SYMBOL(iov_iter_kvec);

With LNet the WARN_ON was causing so many back traces to appear that it 
cause my testing to grind to a halt :-( 

> > Signed-off-by: James Simmons <jsimmons at infradead.org>
> > ---
> > drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd_cb.c  | 4 ++--
> > drivers/staging/lustre/lnet/klnds/socklnd/socklnd_cb.c  | 8 ++++----
> > drivers/staging/lustre/lnet/klnds/socklnd/socklnd_lib.c | 4 ++--
> > drivers/staging/lustre/lnet/lnet/lib-move.c             | 4 ++--
> > drivers/staging/lustre/lnet/lnet/lib-socket.c           | 4 ++--
> > 5 files changed, 12 insertions(+), 12 deletions(-)
> > 
> > diff --git a/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd_cb.c b/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd_cb.c
> > index 23ce59e..57fe037 100644
> > --- a/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd_cb.c
> > +++ b/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd_cb.c
> > @@ -1548,11 +1548,11 @@ static int kiblnd_resolve_addr(struct rdma_cm_id *cmid,
> > 	LASSERT(!(payload_kiov && payload_iov));
> > 
> > 	if (payload_kiov)
> > -		iov_iter_bvec(&from, ITER_BVEC | WRITE,
> > +		iov_iter_bvec(&from, WRITE,
> > 			      payload_kiov, payload_niov,
> > 			      payload_nob + payload_offset);
> > 	else
> > -		iov_iter_kvec(&from, ITER_KVEC | WRITE,
> > +		iov_iter_kvec(&from, WRITE,
> > 			      payload_iov, payload_niov,
> > 			      payload_nob + payload_offset);
> > 
> > diff --git a/drivers/staging/lustre/lnet/klnds/socklnd/socklnd_cb.c b/drivers/staging/lustre/lnet/klnds/socklnd/socklnd_cb.c
> > index c401896..4abf0eb 100644
> > --- a/drivers/staging/lustre/lnet/klnds/socklnd/socklnd_cb.c
> > +++ b/drivers/staging/lustre/lnet/klnds/socklnd/socklnd_cb.c
> > @@ -1001,7 +1001,7 @@ struct ksock_route *
> > 			kvec->iov_base = &conn->ksnc_msg;
> > 			kvec->iov_len = offsetof(struct ksock_msg, ksm_u);
> > 			conn->ksnc_rx_nob_left = offsetof(struct ksock_msg, ksm_u);
> > -			iov_iter_kvec(&conn->ksnc_rx_to, READ|ITER_KVEC, kvec,
> > +			iov_iter_kvec(&conn->ksnc_rx_to, READ, kvec,
> > 					1, offsetof(struct ksock_msg, ksm_u));
> > 			break;
> > 
> > @@ -1011,7 +1011,7 @@ struct ksock_route *
> > 			kvec->iov_base = &conn->ksnc_msg.ksm_u.lnetmsg;
> > 			kvec->iov_len = sizeof(struct lnet_hdr);
> > 			conn->ksnc_rx_nob_left = sizeof(struct lnet_hdr);
> > -			iov_iter_kvec(&conn->ksnc_rx_to, READ|ITER_KVEC, kvec,
> > +			iov_iter_kvec(&conn->ksnc_rx_to, READ, kvec,
> > 					1, sizeof(struct lnet_hdr));
> > 			break;
> > 
> > @@ -1043,7 +1043,7 @@ struct ksock_route *
> > 	} while (nob_to_skip &&    /* mustn't overflow conn's rx iov */
> > 		 niov < sizeof(conn->ksnc_rx_iov_space) / sizeof(struct iovec));
> > 
> > -	iov_iter_kvec(&conn->ksnc_rx_to, READ|ITER_KVEC, kvec, niov, skipped);
> > +	iov_iter_kvec(&conn->ksnc_rx_to, READ, kvec, niov, skipped);
> > 	return 0;
> > }
> > 
> > @@ -1157,7 +1157,7 @@ struct ksock_route *
> > 		kvec->iov_base = &conn->ksnc_msg.ksm_u.lnetmsg;
> > 		kvec->iov_len = sizeof(struct ksock_lnet_msg);
> > 
> > -		iov_iter_kvec(&conn->ksnc_rx_to, READ|ITER_KVEC, kvec,
> > +		iov_iter_kvec(&conn->ksnc_rx_to, READ, kvec,
> > 				1, sizeof(struct ksock_lnet_msg));
> > 
> > 		goto again;     /* read lnet header now */
> > diff --git a/drivers/staging/lustre/lnet/klnds/socklnd/socklnd_lib.c b/drivers/staging/lustre/lnet/klnds/socklnd/socklnd_lib.c
> > index 33847b9..686c2d3 100644
> > --- a/drivers/staging/lustre/lnet/klnds/socklnd/socklnd_lib.c
> > +++ b/drivers/staging/lustre/lnet/klnds/socklnd/socklnd_lib.c
> > @@ -92,7 +92,7 @@
> > 	    nob < tx->tx_resid)
> > 		msg.msg_flags |= MSG_MORE;
> > 
> > -	iov_iter_kvec(&msg.msg_iter, WRITE | ITER_KVEC,
> > +	iov_iter_kvec(&msg.msg_iter, WRITE,
> > 		      tx->tx_iov, tx->tx_niov, nob);
> > 	return sock_sendmsg(sock, &msg);
> > }
> > @@ -140,7 +140,7 @@
> > 		    nob < tx->tx_resid)
> > 			msg.msg_flags |= MSG_MORE;
> > 
> > -		iov_iter_bvec(&msg.msg_iter, WRITE | ITER_BVEC,
> > +		iov_iter_bvec(&msg.msg_iter, WRITE,
> > 			      kiov, tx->tx_nkiov, nob);
> > 		rc = sock_sendmsg(sock, &msg);
> > 	}
> > diff --git a/drivers/staging/lustre/lnet/lnet/lib-move.c b/drivers/staging/lustre/lnet/lnet/lib-move.c
> > index 5694d85..eaa1dfa 100644
> > --- a/drivers/staging/lustre/lnet/lnet/lib-move.c
> > +++ b/drivers/staging/lustre/lnet/lnet/lib-move.c
> > @@ -498,10 +498,10 @@ void lnet_usr_translate_stats(struct lnet_ioctl_element_msg_stats *msg_stats,
> > 	}
> > 
> > 	if (iov) {
> > -		iov_iter_kvec(&to, ITER_KVEC | READ, iov, niov, mlen + offset);
> > +		iov_iter_kvec(&to, READ, iov, niov, mlen + offset);
> > 		iov_iter_advance(&to, offset);
> > 	} else {
> > -		iov_iter_bvec(&to, ITER_BVEC | READ, kiov, niov, mlen + offset);
> > +		iov_iter_bvec(&to, READ, kiov, niov, mlen + offset);
> > 		iov_iter_advance(&to, offset);
> > 	}
> > 	rc = ni->ni_net->net_lnd->lnd_recv(ni, private, msg, delayed, &to, rlen);
> > diff --git a/drivers/staging/lustre/lnet/lnet/lib-socket.c b/drivers/staging/lustre/lnet/lnet/lib-socket.c
> > index d9c62d3..62a742e 100644
> > --- a/drivers/staging/lustre/lnet/lnet/lib-socket.c
> > +++ b/drivers/staging/lustre/lnet/lnet/lib-socket.c
> > @@ -58,7 +58,7 @@
> > 	 * Caller may pass a zero timeout if she thinks the socket buffer is
> > 	 * empty enough to take the whole message immediately
> > 	 */
> > -	iov_iter_kvec(&msg.msg_iter, WRITE | ITER_KVEC, &iov, 1, nob);
> > +	iov_iter_kvec(&msg.msg_iter, WRITE, &iov, 1, nob);
> > 	for (;;) {
> > 		msg.msg_flags = !timeout ? MSG_DONTWAIT : 0;
> > 		if (timeout) {
> > @@ -113,7 +113,7 @@
> > 	LASSERT(nob > 0);
> > 	LASSERT(jiffies_left > 0);
> > 
> > -	iov_iter_kvec(&msg.msg_iter, READ | ITER_KVEC, &iov, 1, nob);
> > +	iov_iter_kvec(&msg.msg_iter, READ, &iov, 1, nob);
> > 
> > 	for (;;) {
> > 		/* Set receive timeout to remaining time */
> > -- 
> > 1.8.3.1
> > 
> 
> Cheers, Andreas
> ---
> Andreas Dilger
> CTO Whamcloud
> 
> 
> 
> 
> 

From jsimmons at infradead.org  Mon Nov  5 20:59:23 2018
From: jsimmons at infradead.org (James Simmons)
Date: Mon, 5 Nov 2018 20:59:23 +0000 (GMT)
Subject: [lustre-devel] drivers/staging updated to 4.20-rc1
In-Reply-To: <87k1lsyssq.fsf@notabene.neil.brown.name>
References: <87k1lsyssq.fsf@notabene.neil.brown.name>
Message-ID: <alpine.LFD.2.21.1811052058590.11915@casper.infradead.org>

> 
> Hi,
>  I've just pushed out new lustre/lustre, lustre/lustre-testing, lustre/lustre-wip
>  to github.com:neilbrown/linux.git
> 
>  lustre is updated to 4.20-rc1 and doesn't compile because of some
>  changes.
>  The patch below makes it compile and (probably) work.
> 
>  I'd appreciate a review, especially of the iov_for_each change.

Almost. I see the following errors while testing.

[ 7111.002667] RIP: 0010:sanity+0xb2/0xb9
[ 7111.002668] Code: 63 c5 48 c7 c7 a1 7c e8 81 ff c5 48 6b c0 28 48 03 43 
78 8b 48 08 48 8b 70 10 44 8b 40 0c 48 8b 10 31 c0 e8 4c 67 d6 ff  c
[ 7111.002669] RSP: 0018:ffffc900072d3a60 EFLAGS: 00010246
[ 7111.002670] RAX: 000000000000002a RBX: ffff8808502126c0 RCX: 
0000000000000006
[ 7111.002671] RDX: 0000000000000000 RSI: ffffffff81e770e3 RDI: 
00000000ffffffff
[ 7111.002672] RBP: 0000000000000010 R08: 0000000000000004 R09: 
ffffffff82683e8e
[ 7111.002673] R10: 00000000000015a9 R11: 0000000000000000 R12: 
ffffea003f9523c0
[ 7111.002674] R13: ffff8808502126c0 R14: ffffea003f9523c0 R15: 
0000000000001000
[ 7111.002676] FS:  00007f140a193740(0000) GS:ffff88107fd80000(0000) 
knlGS:0000000000000000
[ 7111.002677] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 7111.002678] CR2: 000000000067a198 CR3: 00000008305dc001 CR4: 
00000000000606e0
[ 7111.002679] Call Trace:
[ 7111.002682]  copy_page_to_iter+0x245/0x2f0
[ 7111.002686]  generic_file_buffered_read+0x345/0x7c0
[ 7111.002689]  ? touch_atime+0x33/0xd0
[ 7111.002701]  vvp_io_read_start+0x22a/0x3a0 [lustre]
[ 7111.002715]  cl_io_start+0x6d/0x130 [obdclass]
[ 7111.002727]  cl_io_loop+0xae/0x220 [obdclass]
[ 7111.002736]  ll_file_io_generic+0x4ee/0x880 [lustre]
[ 7111.002740]  ? generic_file_buffered_read+0x591/0x7c0
[ 7111.002749]  ll_file_read_iter+0x13e/0x1b0 [lustre]
[ 7111.002751]  generic_file_splice_read+0xe1/0x190
[ 7111.002755]  splice_direct_to_actor+0xea/0x200
[ 7111.002757]  ? generic_pipe_buf_nosteal+0x10/0x10
[ 7111.002759]  do_splice_direct+0x87/0xd0
[ 7111.002762]  do_sendfile+0x1c6/0x3c0
[ 7111.002765]  __x64_sys_sendfile64+0x5c/0xb0
[ 7111.002768]  do_syscall_64+0x68/0x38f
[ 7111.002770]  ? do_page_fault+0x2d/0x130
[ 7111.002773]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
[ 7111.002775] RIP: 0033:0x7f1409ca71ca


 
> Thanks,
> NeilBrown
> 
> 
> Subject: [PATCH] lustre: fixes for 4.20-rc1
> 
> Most are trivial.
> iov_for_each() is broken upstream and there are no users in mainline,
> so it seem to make sense to just open-code it in the one place
> we use it.
> 
> Signed-off-by: NeilBrown <neilb at suse.com>
> ---
>  drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.h | 4 ++--
>  drivers/staging/lustre/lustre/llite/vvp_io.c        | 5 ++++-
>  drivers/staging/lustre/lustre/mdc/mdc_request.c     | 2 +-
>  3 files changed, 7 insertions(+), 4 deletions(-)
> 
> diff --git a/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.h b/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.h
> index c882345226e1..7f81fe24c5eb 100644
> --- a/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.h
> +++ b/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.h
> @@ -478,7 +478,7 @@ struct kib_rx {					/* receive message */
>  	enum ib_wc_status      rx_status;     /* completion status */
>  	struct kib_msg		*rx_msg;	/* message buffer (host vaddr) */
>  	__u64                  rx_msgaddr;    /* message buffer (I/O addr) */
> -	DECLARE_PCI_UNMAP_ADDR(rx_msgunmap);  /* for dma_unmap_single() */
> +	DEFINE_DMA_UNMAP_ADDR(rx_msgunmap);  /* for dma_unmap_single() */
>  	struct ib_recv_wr      rx_wrq;        /* receive work item... */
>  	struct ib_sge          rx_sge;        /* ...and its memory */
>  };
> @@ -501,7 +501,7 @@ struct kib_tx {					/* transmit message */
>  	struct lnet_msg		*tx_lntmsg[2];	/* lnet msgs to finalize on completion */
>  	struct kib_msg	      *tx_msg;        /* message buffer (host vaddr) */
>  	__u64                 tx_msgaddr;     /* message buffer (I/O addr) */
> -	DECLARE_PCI_UNMAP_ADDR(tx_msgunmap);  /* for dma_unmap_single() */
> +	DEFINE_DMA_UNMAP_ADDR(tx_msgunmap);  /* for dma_unmap_single() */
>  	/** sge for tx_msgaddr */
>  	struct ib_sge		tx_msgsge;
>  	int                   tx_nwrq;        /* # send work items */
> diff --git a/drivers/staging/lustre/lustre/llite/vvp_io.c b/drivers/staging/lustre/lustre/llite/vvp_io.c
> index 70d23874b674..524f3f4feec8 100644
> --- a/drivers/staging/lustre/lustre/llite/vvp_io.c
> +++ b/drivers/staging/lustre/lustre/llite/vvp_io.c
> @@ -388,7 +388,10 @@ static int vvp_mmap_locks(const struct lu_env *env,
>  	if (!mm)
>  		return 0;
>  
> -	iov_for_each(iov, i, *vio->vui_iter) {
> +	for (i = *vio->vui_iter;
> +	     i.count;
> +	     iov_iter_advance(&i, iov.iov_len)) {
> +		iov = iov_iter_iovec(&i);
>  		addr = (unsigned long)iov.iov_base;
>  		count = iov.iov_len;
>  		if (count == 0)
> diff --git a/drivers/staging/lustre/lustre/mdc/mdc_request.c b/drivers/staging/lustre/lustre/mdc/mdc_request.c
> index 8f8e3d220dc8..2242204f3c9e 100644
> --- a/drivers/staging/lustre/lustre/mdc/mdc_request.c
> +++ b/drivers/staging/lustre/lustre/mdc/mdc_request.c
> @@ -993,7 +993,7 @@ static struct page *mdc_page_locate(struct address_space *mapping, __u64 *hash,
>  	xa_lock_irq(&mapping->i_pages);
>  	found = radix_tree_gang_lookup(&mapping->i_pages,
>  				       (void **)&page, offset, 1);
> -	if (found > 0 && !radix_tree_exceptional_entry(page)) {
> +	if (found > 0 && !xa_is_value(page)) {
>  		struct lu_dirpage *dp;
>  
>  		get_page(page);
> -- 
> 2.14.0.rc0.dirty
> 
> 

From neilb at suse.com  Tue Nov  6 01:59:52 2018
From: neilb at suse.com (NeilBrown)
Date: Tue, 06 Nov 2018 12:59:52 +1100
Subject: [lustre-devel] [PATCH] lustre: remove iter type from
 iov_iter_[b|k]vec()
In-Reply-To: <1541441887-15064-1-git-send-email-jsimmons@infradead.org>
References: <1541441887-15064-1-git-send-email-jsimmons@infradead.org>
Message-ID: <87zhunx8x3.fsf@notabene.neil.brown.name>

On Mon, Nov 05 2018, James Simmons wrote:

> The linux commit aa563d7bca6e ("iov_iter: Separate type from
> direction and use accessor functions) removed the iter type
> from both iov_iter_bvec() and iov_iter_kvec(). Update lustre
> to this change. Without it we see in testing:
>
> WARNING: CPU: 2 PID: 3088 at lib/iov_iter.c:1082 iov_iter_kvec+0x25/0x30
>
> Signed-off-by: James Simmons <jsimmons at infradead.org>

Looks good to me - applied.
Thanks,
NeilBrown


> ---
>  drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd_cb.c  | 4 ++--
>  drivers/staging/lustre/lnet/klnds/socklnd/socklnd_cb.c  | 8 ++++----
>  drivers/staging/lustre/lnet/klnds/socklnd/socklnd_lib.c | 4 ++--
>  drivers/staging/lustre/lnet/lnet/lib-move.c             | 4 ++--
>  drivers/staging/lustre/lnet/lnet/lib-socket.c           | 4 ++--
>  5 files changed, 12 insertions(+), 12 deletions(-)
>
> diff --git a/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd_cb.c b/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd_cb.c
> index 23ce59e..57fe037 100644
> --- a/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd_cb.c
> +++ b/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd_cb.c
> @@ -1548,11 +1548,11 @@ static int kiblnd_resolve_addr(struct rdma_cm_id *cmid,
>  	LASSERT(!(payload_kiov && payload_iov));
>  
>  	if (payload_kiov)
> -		iov_iter_bvec(&from, ITER_BVEC | WRITE,
> +		iov_iter_bvec(&from, WRITE,
>  			      payload_kiov, payload_niov,
>  			      payload_nob + payload_offset);
>  	else
> -		iov_iter_kvec(&from, ITER_KVEC | WRITE,
> +		iov_iter_kvec(&from, WRITE,
>  			      payload_iov, payload_niov,
>  			      payload_nob + payload_offset);
>  
> diff --git a/drivers/staging/lustre/lnet/klnds/socklnd/socklnd_cb.c b/drivers/staging/lustre/lnet/klnds/socklnd/socklnd_cb.c
> index c401896..4abf0eb 100644
> --- a/drivers/staging/lustre/lnet/klnds/socklnd/socklnd_cb.c
> +++ b/drivers/staging/lustre/lnet/klnds/socklnd/socklnd_cb.c
> @@ -1001,7 +1001,7 @@ struct ksock_route *
>  			kvec->iov_base = &conn->ksnc_msg;
>  			kvec->iov_len = offsetof(struct ksock_msg, ksm_u);
>  			conn->ksnc_rx_nob_left = offsetof(struct ksock_msg, ksm_u);
> -			iov_iter_kvec(&conn->ksnc_rx_to, READ|ITER_KVEC, kvec,
> +			iov_iter_kvec(&conn->ksnc_rx_to, READ, kvec,
>  					1, offsetof(struct ksock_msg, ksm_u));
>  			break;
>  
> @@ -1011,7 +1011,7 @@ struct ksock_route *
>  			kvec->iov_base = &conn->ksnc_msg.ksm_u.lnetmsg;
>  			kvec->iov_len = sizeof(struct lnet_hdr);
>  			conn->ksnc_rx_nob_left = sizeof(struct lnet_hdr);
> -			iov_iter_kvec(&conn->ksnc_rx_to, READ|ITER_KVEC, kvec,
> +			iov_iter_kvec(&conn->ksnc_rx_to, READ, kvec,
>  					1, sizeof(struct lnet_hdr));
>  			break;
>  
> @@ -1043,7 +1043,7 @@ struct ksock_route *
>  	} while (nob_to_skip &&    /* mustn't overflow conn's rx iov */
>  		 niov < sizeof(conn->ksnc_rx_iov_space) / sizeof(struct iovec));
>  
> -	iov_iter_kvec(&conn->ksnc_rx_to, READ|ITER_KVEC, kvec, niov, skipped);
> +	iov_iter_kvec(&conn->ksnc_rx_to, READ, kvec, niov, skipped);
>  	return 0;
>  }
>  
> @@ -1157,7 +1157,7 @@ struct ksock_route *
>  		kvec->iov_base = &conn->ksnc_msg.ksm_u.lnetmsg;
>  		kvec->iov_len = sizeof(struct ksock_lnet_msg);
>  
> -		iov_iter_kvec(&conn->ksnc_rx_to, READ|ITER_KVEC, kvec,
> +		iov_iter_kvec(&conn->ksnc_rx_to, READ, kvec,
>  				1, sizeof(struct ksock_lnet_msg));
>  
>  		goto again;     /* read lnet header now */
> diff --git a/drivers/staging/lustre/lnet/klnds/socklnd/socklnd_lib.c b/drivers/staging/lustre/lnet/klnds/socklnd/socklnd_lib.c
> index 33847b9..686c2d3 100644
> --- a/drivers/staging/lustre/lnet/klnds/socklnd/socklnd_lib.c
> +++ b/drivers/staging/lustre/lnet/klnds/socklnd/socklnd_lib.c
> @@ -92,7 +92,7 @@
>  	    nob < tx->tx_resid)
>  		msg.msg_flags |= MSG_MORE;
>  
> -	iov_iter_kvec(&msg.msg_iter, WRITE | ITER_KVEC,
> +	iov_iter_kvec(&msg.msg_iter, WRITE,
>  		      tx->tx_iov, tx->tx_niov, nob);
>  	return sock_sendmsg(sock, &msg);
>  }
> @@ -140,7 +140,7 @@
>  		    nob < tx->tx_resid)
>  			msg.msg_flags |= MSG_MORE;
>  
> -		iov_iter_bvec(&msg.msg_iter, WRITE | ITER_BVEC,
> +		iov_iter_bvec(&msg.msg_iter, WRITE,
>  			      kiov, tx->tx_nkiov, nob);
>  		rc = sock_sendmsg(sock, &msg);
>  	}
> diff --git a/drivers/staging/lustre/lnet/lnet/lib-move.c b/drivers/staging/lustre/lnet/lnet/lib-move.c
> index 5694d85..eaa1dfa 100644
> --- a/drivers/staging/lustre/lnet/lnet/lib-move.c
> +++ b/drivers/staging/lustre/lnet/lnet/lib-move.c
> @@ -498,10 +498,10 @@ void lnet_usr_translate_stats(struct lnet_ioctl_element_msg_stats *msg_stats,
>  	}
>  
>  	if (iov) {
> -		iov_iter_kvec(&to, ITER_KVEC | READ, iov, niov, mlen + offset);
> +		iov_iter_kvec(&to, READ, iov, niov, mlen + offset);
>  		iov_iter_advance(&to, offset);
>  	} else {
> -		iov_iter_bvec(&to, ITER_BVEC | READ, kiov, niov, mlen + offset);
> +		iov_iter_bvec(&to, READ, kiov, niov, mlen + offset);
>  		iov_iter_advance(&to, offset);
>  	}
>  	rc = ni->ni_net->net_lnd->lnd_recv(ni, private, msg, delayed, &to, rlen);
> diff --git a/drivers/staging/lustre/lnet/lnet/lib-socket.c b/drivers/staging/lustre/lnet/lnet/lib-socket.c
> index d9c62d3..62a742e 100644
> --- a/drivers/staging/lustre/lnet/lnet/lib-socket.c
> +++ b/drivers/staging/lustre/lnet/lnet/lib-socket.c
> @@ -58,7 +58,7 @@
>  	 * Caller may pass a zero timeout if she thinks the socket buffer is
>  	 * empty enough to take the whole message immediately
>  	 */
> -	iov_iter_kvec(&msg.msg_iter, WRITE | ITER_KVEC, &iov, 1, nob);
> +	iov_iter_kvec(&msg.msg_iter, WRITE, &iov, 1, nob);
>  	for (;;) {
>  		msg.msg_flags = !timeout ? MSG_DONTWAIT : 0;
>  		if (timeout) {
> @@ -113,7 +113,7 @@
>  	LASSERT(nob > 0);
>  	LASSERT(jiffies_left > 0);
>  
> -	iov_iter_kvec(&msg.msg_iter, READ | ITER_KVEC, &iov, 1, nob);
> +	iov_iter_kvec(&msg.msg_iter, READ, &iov, 1, nob);
>  
>  	for (;;) {
>  		/* Set receive timeout to remaining time */
> -- 
> 1.8.3.1
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 832 bytes
Desc: not available
URL: <http://lists.lustre.org/pipermail/lustre-devel_lists.lustre.org/attachments/20181106/1a27805c/attachment.sig>

From adilger at whamcloud.com  Tue Nov  6 02:06:18 2018
From: adilger at whamcloud.com (Andreas Dilger)
Date: Tue, 6 Nov 2018 02:06:18 +0000
Subject: [lustre-devel] [PATCH] lustre: remove iter type from
 iov_iter_[b|k]vec()
In-Reply-To: <alpine.LFD.2.21.1811051953160.11915@casper.infradead.org>
References: <1541441887-15064-1-git-send-email-jsimmons@infradead.org>
 <91C563BF-78C7-4409-B0BC-5048D689A309@whamcloud.com>,
 <alpine.LFD.2.21.1811051953160.11915@casper.infradead.org>
Message-ID: <15DDFA38-2C46-40E7-AA71-166468E4C789@whamcloud.com>

The WARN_ON() should probably be changed to WARN_ON_ONCE()?

Cheers, Andreas

> On Nov 5, 2018, at 13:12, James Simmons <jsimmons at infradead.org> wrote:
> 
> 
>>> On Nov 5, 2018, at 11:18, James Simmons <jsimmons at infradead.org> wrote:
>>> 
>>> The linux commit aa563d7bca6e ("iov_iter: Separate type from
>>> direction and use accessor functions) removed the iter type
>>> from both iov_iter_bvec() and iov_iter_kvec(). Update lustre
>>> to this change. Without it we see in testing:
>>> 
>>> WARNING: CPU: 2 PID: 3088 at lib/iov_iter.c:1082 iov_iter_kvec+0x25/0x30
>> 
>> I'm no LNet expert, but it would seem that just removing the ITER_KVEC
>> flag would cause problems with the code?  As recently as the 4.17 kernel
>> it looks like removing this flag would cause a BUG_ON():
>> 
>> void iov_iter_kvec(struct iov_iter *i, int direction,
>>                        const struct kvec *kvec, unsigned long nr_segs,
>>                        size_t count)
>> {
>>        BUG_ON(!(direction & ITER_KVEC));
>>        i->type = direction;
>>        i->kvec = kvec;
>>        i->nr_segs = nr_segs;
>>        i->iov_offset = 0;
>>        i->count = count;
>> }
>> 
>> I suspect what you need to do is something like:
>> 
>> #if <something>
>> #define LNET_ITER_KVEC 0
>> #else
>> #define LNET_ITER_KVEC ITER_KVEC
>> #endif
>> 
>> so that it works on both old and new kernels, at least for the version
>> that is included in the master repo.
>> 
>> Cheers, Andreas
> 
> For the OpenSFS branch yes we will have to do something like that to 
> support various kernel releases. The change was just landed in the 
> 4.20-rc1 cycle. iov_iter_kvec is now:
> 
> void iov_iter_kvec(struct iov_iter *i, unsigned int direction,
>                        const struct kvec *kvec, unsigned long nr_segs,
>                        size_t count)
> {
>        WARN_ON(direction & ~(READ | WRITE));
>        i->type = ITER_KVEC | (direction & (READ | WRITE));
>        i->kvec = kvec;
>        i->nr_segs = nr_segs;
>        i->iov_offset = 0;
>        i->count = count;
> }
> EXPORT_SYMBOL(iov_iter_kvec);
> 
> With LNet the WARN_ON was causing so many back traces to appear that it 
> cause my testing to grind to a halt :-( 
> 
>>> Signed-off-by: James Simmons <jsimmons at infradead.org>
>>> ---
>>> drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd_cb.c  | 4 ++--
>>> drivers/staging/lustre/lnet/klnds/socklnd/socklnd_cb.c  | 8 ++++----
>>> drivers/staging/lustre/lnet/klnds/socklnd/socklnd_lib.c | 4 ++--
>>> drivers/staging/lustre/lnet/lnet/lib-move.c             | 4 ++--
>>> drivers/staging/lustre/lnet/lnet/lib-socket.c           | 4 ++--
>>> 5 files changed, 12 insertions(+), 12 deletions(-)
>>> 
>>> diff --git a/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd_cb.c b/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd_cb.c
>>> index 23ce59e..57fe037 100644
>>> --- a/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd_cb.c
>>> +++ b/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd_cb.c
>>> @@ -1548,11 +1548,11 @@ static int kiblnd_resolve_addr(struct rdma_cm_id *cmid,
>>>    LASSERT(!(payload_kiov && payload_iov));
>>> 
>>>    if (payload_kiov)
>>> -        iov_iter_bvec(&from, ITER_BVEC | WRITE,
>>> +        iov_iter_bvec(&from, WRITE,
>>>                  payload_kiov, payload_niov,
>>>                  payload_nob + payload_offset);
>>>    else
>>> -        iov_iter_kvec(&from, ITER_KVEC | WRITE,
>>> +        iov_iter_kvec(&from, WRITE,
>>>                  payload_iov, payload_niov,
>>>                  payload_nob + payload_offset);
>>> 
>>> diff --git a/drivers/staging/lustre/lnet/klnds/socklnd/socklnd_cb.c b/drivers/staging/lustre/lnet/klnds/socklnd/socklnd_cb.c
>>> index c401896..4abf0eb 100644
>>> --- a/drivers/staging/lustre/lnet/klnds/socklnd/socklnd_cb.c
>>> +++ b/drivers/staging/lustre/lnet/klnds/socklnd/socklnd_cb.c
>>> @@ -1001,7 +1001,7 @@ struct ksock_route *
>>>            kvec->iov_base = &conn->ksnc_msg;
>>>            kvec->iov_len = offsetof(struct ksock_msg, ksm_u);
>>>            conn->ksnc_rx_nob_left = offsetof(struct ksock_msg, ksm_u);
>>> -            iov_iter_kvec(&conn->ksnc_rx_to, READ|ITER_KVEC, kvec,
>>> +            iov_iter_kvec(&conn->ksnc_rx_to, READ, kvec,
>>>                    1, offsetof(struct ksock_msg, ksm_u));
>>>            break;
>>> 
>>> @@ -1011,7 +1011,7 @@ struct ksock_route *
>>>            kvec->iov_base = &conn->ksnc_msg.ksm_u.lnetmsg;
>>>            kvec->iov_len = sizeof(struct lnet_hdr);
>>>            conn->ksnc_rx_nob_left = sizeof(struct lnet_hdr);
>>> -            iov_iter_kvec(&conn->ksnc_rx_to, READ|ITER_KVEC, kvec,
>>> +            iov_iter_kvec(&conn->ksnc_rx_to, READ, kvec,
>>>                    1, sizeof(struct lnet_hdr));
>>>            break;
>>> 
>>> @@ -1043,7 +1043,7 @@ struct ksock_route *
>>>    } while (nob_to_skip &&    /* mustn't overflow conn's rx iov */
>>>         niov < sizeof(conn->ksnc_rx_iov_space) / sizeof(struct iovec));
>>> 
>>> -    iov_iter_kvec(&conn->ksnc_rx_to, READ|ITER_KVEC, kvec, niov, skipped);
>>> +    iov_iter_kvec(&conn->ksnc_rx_to, READ, kvec, niov, skipped);
>>>    return 0;
>>> }
>>> 
>>> @@ -1157,7 +1157,7 @@ struct ksock_route *
>>>        kvec->iov_base = &conn->ksnc_msg.ksm_u.lnetmsg;
>>>        kvec->iov_len = sizeof(struct ksock_lnet_msg);
>>> 
>>> -        iov_iter_kvec(&conn->ksnc_rx_to, READ|ITER_KVEC, kvec,
>>> +        iov_iter_kvec(&conn->ksnc_rx_to, READ, kvec,
>>>                1, sizeof(struct ksock_lnet_msg));
>>> 
>>>        goto again;     /* read lnet header now */
>>> diff --git a/drivers/staging/lustre/lnet/klnds/socklnd/socklnd_lib.c b/drivers/staging/lustre/lnet/klnds/socklnd/socklnd_lib.c
>>> index 33847b9..686c2d3 100644
>>> --- a/drivers/staging/lustre/lnet/klnds/socklnd/socklnd_lib.c
>>> +++ b/drivers/staging/lustre/lnet/klnds/socklnd/socklnd_lib.c
>>> @@ -92,7 +92,7 @@
>>>        nob < tx->tx_resid)
>>>        msg.msg_flags |= MSG_MORE;
>>> 
>>> -    iov_iter_kvec(&msg.msg_iter, WRITE | ITER_KVEC,
>>> +    iov_iter_kvec(&msg.msg_iter, WRITE,
>>>              tx->tx_iov, tx->tx_niov, nob);
>>>    return sock_sendmsg(sock, &msg);
>>> }
>>> @@ -140,7 +140,7 @@
>>>            nob < tx->tx_resid)
>>>            msg.msg_flags |= MSG_MORE;
>>> 
>>> -        iov_iter_bvec(&msg.msg_iter, WRITE | ITER_BVEC,
>>> +        iov_iter_bvec(&msg.msg_iter, WRITE,
>>>                  kiov, tx->tx_nkiov, nob);
>>>        rc = sock_sendmsg(sock, &msg);
>>>    }
>>> diff --git a/drivers/staging/lustre/lnet/lnet/lib-move.c b/drivers/staging/lustre/lnet/lnet/lib-move.c
>>> index 5694d85..eaa1dfa 100644
>>> --- a/drivers/staging/lustre/lnet/lnet/lib-move.c
>>> +++ b/drivers/staging/lustre/lnet/lnet/lib-move.c
>>> @@ -498,10 +498,10 @@ void lnet_usr_translate_stats(struct lnet_ioctl_element_msg_stats *msg_stats,
>>>    }
>>> 
>>>    if (iov) {
>>> -        iov_iter_kvec(&to, ITER_KVEC | READ, iov, niov, mlen + offset);
>>> +        iov_iter_kvec(&to, READ, iov, niov, mlen + offset);
>>>        iov_iter_advance(&to, offset);
>>>    } else {
>>> -        iov_iter_bvec(&to, ITER_BVEC | READ, kiov, niov, mlen + offset);
>>> +        iov_iter_bvec(&to, READ, kiov, niov, mlen + offset);
>>>        iov_iter_advance(&to, offset);
>>>    }
>>>    rc = ni->ni_net->net_lnd->lnd_recv(ni, private, msg, delayed, &to, rlen);
>>> diff --git a/drivers/staging/lustre/lnet/lnet/lib-socket.c b/drivers/staging/lustre/lnet/lnet/lib-socket.c
>>> index d9c62d3..62a742e 100644
>>> --- a/drivers/staging/lustre/lnet/lnet/lib-socket.c
>>> +++ b/drivers/staging/lustre/lnet/lnet/lib-socket.c
>>> @@ -58,7 +58,7 @@
>>>     * Caller may pass a zero timeout if she thinks the socket buffer is
>>>     * empty enough to take the whole message immediately
>>>     */
>>> -    iov_iter_kvec(&msg.msg_iter, WRITE | ITER_KVEC, &iov, 1, nob);
>>> +    iov_iter_kvec(&msg.msg_iter, WRITE, &iov, 1, nob);
>>>    for (;;) {
>>>        msg.msg_flags = !timeout ? MSG_DONTWAIT : 0;
>>>        if (timeout) {
>>> @@ -113,7 +113,7 @@
>>>    LASSERT(nob > 0);
>>>    LASSERT(jiffies_left > 0);
>>> 
>>> -    iov_iter_kvec(&msg.msg_iter, READ | ITER_KVEC, &iov, 1, nob);
>>> +    iov_iter_kvec(&msg.msg_iter, READ, &iov, 1, nob);
>>> 
>>>    for (;;) {
>>>        /* Set receive timeout to remaining time */
>>> -- 
>>> 1.8.3.1
>>> 
>> 
>> Cheers, Andreas
>> ---
>> Andreas Dilger
>> CTO Whamcloud
>> 
>> 
>> 
>> 
>> 

From neilb at suse.com  Tue Nov  6 03:11:54 2018
From: neilb at suse.com (NeilBrown)
Date: Tue, 06 Nov 2018 14:11:54 +1100
Subject: [lustre-devel] [PATCH v2] lustre: mdc: fix possible deadlock in
 chlg_open()
In-Reply-To: <88ccd141-4778-e3ab-3ce0-e7cbd5c290fc@cea.fr>
References: <87bm7a3nue.fsf@notabene.neil.brown.name>
 <7db27e7d-e685-af8f-80d8-891a0d8db4d5@cea.fr>
 <875zxh3elu.fsf@notabene.neil.brown.name>
 <alpine.LFD.2.21.1811042131490.27308@casper.infradead.org>
 <88ccd141-4778-e3ab-3ce0-e7cbd5c290fc@cea.fr>
Message-ID: <87wopqyk5h.fsf@notabene.neil.brown.name>

On Mon, Nov 05 2018, quentin.bouget at cea.fr wrote:

> Le 04/11/2018 à 22:34, James Simmons a écrit :
>>> Lockdep reports a possible deadlock between chlg_open() and
>>> mdc_changelog_cdev_init()
>>>
>>> mdc_changelog_cdev_init() takes chlg_registered_dev_lock and then
>>> calls misc_register() which takes misc_mtx.
>>> chlg_open() is called while misc_mtx is held, and tries to take
>>> chlg_registered_dev_lock.
>>> If these two functions race, a deadlock can occur as each thread will
>>> hold one of the locks while trying to take the other.
>>>
>>> chlg_open() does not need to take a lock.  It only uses the
>>> lock to stablize a list while looking for the matching
>>> chlg_registered_dev, and this can be found directly by examining
>>> file->private_data.
>>>
>>> So remove chlg_obd_get(), and use file->private_data to find the
>>> obd_device.
>>> Also ensure the device is fully initialized before calling
>>> misc_register().  This means setting up some list linkage before the
>>> call, and tearing it down if there is an error.
>> I have been testing this but I'm no HSM expert. I pushed this patch
>> to OpenSFS branch as well.
>>
>> https://jira.whamcloud.com/browse/LU-11617
>> https://review.whamcloud.com/#/c/33572/
>>
>> Reviewed-by: James Simmons <jsimmons at infradead.org>
>
> Reviewed-by: Quentin Bouget <quentin.bouget at cea.fr>
>

Thanks to you both for the review.

NeilBrown

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 832 bytes
Desc: not available
URL: <http://lists.lustre.org/pipermail/lustre-devel_lists.lustre.org/attachments/20181106/3a987263/attachment.sig>

From quentin.bouget at cea.fr  Tue Nov  6 10:41:56 2018
From: quentin.bouget at cea.fr (quentin.bouget at cea.fr)
Date: Tue, 6 Nov 2018 11:41:56 +0100
Subject: [lustre-devel] [PATCH v2] lustre: mdc: fix possible deadlock in
 chlg_open()
In-Reply-To: <87wopqyk5h.fsf@notabene.neil.brown.name>
References: <87bm7a3nue.fsf@notabene.neil.brown.name>
 <7db27e7d-e685-af8f-80d8-891a0d8db4d5@cea.fr>
 <875zxh3elu.fsf@notabene.neil.brown.name>
 <alpine.LFD.2.21.1811042131490.27308@casper.infradead.org>
 <88ccd141-4778-e3ab-3ce0-e7cbd5c290fc@cea.fr>
 <87wopqyk5h.fsf@notabene.neil.brown.name>
Message-ID: <a513884a-046f-a28c-3fbb-dd112abd4b26@cea.fr>

Le 06/11/2018 à 04:11, NeilBrown a écrit :
> On Mon, Nov 05 2018, quentin.bouget at cea.fr wrote:
>
>> Le 04/11/2018 à 22:34, James Simmons a écrit :
>>>> Lockdep reports a possible deadlock between chlg_open() and
>>>> mdc_changelog_cdev_init()
>>>>
>>>> mdc_changelog_cdev_init() takes chlg_registered_dev_lock and then
>>>> calls misc_register() which takes misc_mtx.
>>>> chlg_open() is called while misc_mtx is held, and tries to take
>>>> chlg_registered_dev_lock.
>>>> If these two functions race, a deadlock can occur as each thread will
>>>> hold one of the locks while trying to take the other.
>>>>
>>>> chlg_open() does not need to take a lock.  It only uses the
>>>> lock to stablize a list while looking for the matching
>>>> chlg_registered_dev, and this can be found directly by examining
>>>> file->private_data.
>>>>
>>>> So remove chlg_obd_get(), and use file->private_data to find the
>>>> obd_device.
>>>> Also ensure the device is fully initialized before calling
>>>> misc_register().  This means setting up some list linkage before the
>>>> call, and tearing it down if there is an error.
>>> I have been testing this but I'm no HSM expert. I pushed this patch
>>> to OpenSFS branch as well.
>>>
>>> https://jira.whamcloud.com/browse/LU-11617
>>> https://review.whamcloud.com/#/c/33572/
>>>
>>> Reviewed-by: James Simmons <jsimmons at infradead.org>
>> Reviewed-by: Quentin Bouget <quentin.bouget at cea.fr>
>>
> Thanks to you both for the review.
>
> NeilBrown
>
Wait! I just realised there might be another issue!
I think there is now a race between chlg_open() and 
mdc_changelog_cdev_finish().

Wait! I just realised there might be another bigger issue!
The whole "take the first obd you can find" is broken! I opened a ticket 
<https://jira.whamcloud.com/browse/LU-11626> on whamcloud's JIRA about it.

Quentin Bouget

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-devel_lists.lustre.org/attachments/20181106/bad9cdb2/attachment.html>

From paf at cray.com  Tue Nov  6 17:38:14 2018
From: paf at cray.com (Patrick Farrell)
Date: Tue, 6 Nov 2018 17:38:14 +0000
Subject: [lustre-devel] Apparent bug in LU-8130 ptlrpc: convert conn_hash to
 rhashtable / Linux-commit: ac2370ac2bc5215daf78546cd8d925510065bb7f
Message-ID: <3E0E82E2-0EF0-4F05-9C33-282002000D31@cray.com>

Neil, James,

Sent this earlier, but with nasty formatting that got it rejected.  Whoops.
 
It looks like the patch we landed as:
LU-8130 ptlrpc: convert conn_hash to rhashtable
 
Linux has a resizeable hashtable implementation in lib,
so we should use that instead of having one in libcfs.
 
This patch converts the ptlrpc conn_hash to use rhashtable.
In the process we gain lockless lookup.
 
As connections are never deleted until the hash table is destroyed,
there is no need to count the reference in the hash table. There
is also no need to enable automatic_shrinking.
 
Linux-commit: ac2370ac2bc5215daf78546cd8d925510065bb7f
 
Introduced a bug.  Ihara-san opened something to track it here:
https://jira.whamcloud.com/browse/LU-11624
 
It’s a null pointer in nid_hash(); there are some more details at that link.
 
We’re seeing it at Cray as well, when testing the current WhamCloud branch.
 
Basically, when we fail over an MDS(/MDT) under load (ie with real activity on the file system) we hit this panic about 30-50% of the time right now.  I assume it’s possible on OSSes as well but we haven’t seen it there.
 
I haven’t done any detailed investigation, but I thought I’d bring it to your attention.  Per Ihara-san in LU-11624, the crash does not happen without the commit listed above.
 
• Patrick



From paf at cray.com  Tue Nov  6 17:19:55 2018
From: paf at cray.com (Patrick Farrell)
Date: Tue, 6 Nov 2018 17:19:55 +0000
Subject: [lustre-devel] Apparent bug in LU-8130 ptlrpc: convert conn_hash to
 rhashtable / Linux-commit: ac2370ac2bc5215daf78546cd8d925510065bb7f
Message-ID: <86CBD270-5FA9-482D-8AA4-C6309096BD81@cray.com>

Neil, James,

It looks like the patch we landed as:

LU-8130 ptlrpc: convert conn_hash to rhashtable

Linux has a resizeable hashtable implementation in lib,
so we should use that instead of having one in libcfs.

This patch converts the ptlrpc conn_hash to use rhashtable.
In the process we gain lockless lookup.

As connections are never deleted until the hash table is destroyed,
there is no need to count the reference in the hash table. There
is also no need to enable automatic_shrinking.

Linux-commit: ac2370ac2bc5215daf78546cd8d925510065bb7f

Introduced a bug.  Ihara-san opened something to track it here:
https://jira.whamcloud.com/browse/LU-11624

It’s a null pointer in nid_hash(); there are some more details at that link.

We’re seeing it at Cray as well, when testing the current WhamCloud branch.

Basically, when we fail over an MDS(/MDT) under load (ie with real activity on the file system) we hit this panic about 30-50% of the time right now.  I assume it’s possible on OSSes as well but we haven’t seen it there.

I haven’t done any detailed investigation, but I thought I’d bring it to your attention.  Per Ihara-san in LU-11624, the crash does not happen without the commit listed above.


  *   Patrick
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-devel_lists.lustre.org/attachments/20181106/103b69a0/attachment.html>

From neilb at suse.com  Tue Nov  6 22:23:32 2018
From: neilb at suse.com (NeilBrown)
Date: Wed, 07 Nov 2018 09:23:32 +1100
Subject: [lustre-devel] Apparent bug in LU-8130 ptlrpc: convert
 conn_hash to rhashtable / Linux-commit:
 ac2370ac2bc5215daf78546cd8d925510065bb7f
In-Reply-To: <3E0E82E2-0EF0-4F05-9C33-282002000D31@cray.com>
References: <3E0E82E2-0EF0-4F05-9C33-282002000D31@cray.com>
Message-ID: <87r2fxyhej.fsf@notabene.neil.brown.name>

On Tue, Nov 06 2018, Patrick Farrell wrote:

> Neil, James,
>
> Sent this earlier, but with nasty formatting that got it rejected.  Whoops.
>  
> It looks like the patch we landed as:
> LU-8130 ptlrpc: convert conn_hash to rhashtable
>  
> Linux has a resizeable hashtable implementation in lib,
> so we should use that instead of having one in libcfs.
>  
> This patch converts the ptlrpc conn_hash to use rhashtable.
> In the process we gain lockless lookup.
>  
> As connections are never deleted until the hash table is destroyed,
> there is no need to count the reference in the hash table. There
> is also no need to enable automatic_shrinking.
>  
> Linux-commit: ac2370ac2bc5215daf78546cd8d925510065bb7f
>  
> Introduced a bug.  Ihara-san opened something to track it here:
> https://jira.whamcloud.com/browse/LU-11624
>  
> It’s a null pointer in nid_hash(); there are some more details at that link.

That commit changed ptlrpc_connection_get() so that it can return NULL.

This is actually one of the really annoying things about rhashtable.  I
cannot convince the maintainers that sometimes you need guaranteed
success (which is quite possible with a hash table).  They come from the
networking community were they drop packets on errors all the time - no
big deal.

Anyway, all users of ptlrpc_connection_get() in the client code (in
drivers/staging) test the return value against NULL, so this was safe.

In SFS lustre, the call in target_handle_connect() assigned the
result to export->exp_connection without checking against NULL.
That leads to the oops.

ptlrpc_connection_get() will return NULL when
rhashtable_lookup_get_insert_fast() returns an error.
The errors are mostly weird internal implementation details leaking out
the API.  A retry, possibly after a short delay, will usually succeed.
(see https://lwn.net/Articles/751974/, section "Failure during insertion)
If this happens a lot, it might suggest that the hash function is poor.

Ahhhh... this was built against a 3.10 kernel.
Prior to 4.6, hash_64() was not so much "poor" as "appalling".
See https://lwn.net/Articles/687494/

For SFS we should avoid using hash_64() (at least before 4.6) and
instead use something like
  hash_32((foo>>32) ^ hash_32(foo & 0xFFFFFFFF))

NeilBrown

>  
> We’re seeing it at Cray as well, when testing the current WhamCloud branch.
>  
> Basically, when we fail over an MDS(/MDT) under load (ie with real activity on the file system) we hit this panic about 30-50% of the time right now.  I assume it’s possible on OSSes as well but we haven’t seen it there.
>  
> I haven’t done any detailed investigation, but I thought I’d bring it to your attention.  Per Ihara-san in LU-11624, the crash does not happen without the commit listed above.
>  
> • Patrick
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 832 bytes
Desc: not available
URL: <http://lists.lustre.org/pipermail/lustre-devel_lists.lustre.org/attachments/20181107/9aae45cc/attachment.sig>

From neilb at suse.com  Wed Nov  7 00:33:25 2018
From: neilb at suse.com (NeilBrown)
Date: Wed, 07 Nov 2018 11:33:25 +1100
Subject: [lustre-devel] [PATCH v2] lustre: mdc: fix possible deadlock in
 chlg_open()
In-Reply-To: <a513884a-046f-a28c-3fbb-dd112abd4b26@cea.fr>
References: <87bm7a3nue.fsf@notabene.neil.brown.name>
 <7db27e7d-e685-af8f-80d8-891a0d8db4d5@cea.fr>
 <875zxh3elu.fsf@notabene.neil.brown.name>
 <alpine.LFD.2.21.1811042131490.27308@casper.infradead.org>
 <88ccd141-4778-e3ab-3ce0-e7cbd5c290fc@cea.fr>
 <87wopqyk5h.fsf@notabene.neil.brown.name>
 <a513884a-046f-a28c-3fbb-dd112abd4b26@cea.fr>
Message-ID: <87d0rhybe2.fsf@notabene.neil.brown.name>

On Tue, Nov 06 2018, quentin.bouget at cea.fr wrote:

> Le 06/11/2018 à 04:11, NeilBrown a écrit :
>> On Mon, Nov 05 2018, quentin.bouget at cea.fr wrote:
>>
>>> Le 04/11/2018 à 22:34, James Simmons a écrit :
>>>>> Lockdep reports a possible deadlock between chlg_open() and
>>>>> mdc_changelog_cdev_init()
>>>>>
>>>>> mdc_changelog_cdev_init() takes chlg_registered_dev_lock and then
>>>>> calls misc_register() which takes misc_mtx.
>>>>> chlg_open() is called while misc_mtx is held, and tries to take
>>>>> chlg_registered_dev_lock.
>>>>> If these two functions race, a deadlock can occur as each thread will
>>>>> hold one of the locks while trying to take the other.
>>>>>
>>>>> chlg_open() does not need to take a lock.  It only uses the
>>>>> lock to stablize a list while looking for the matching
>>>>> chlg_registered_dev, and this can be found directly by examining
>>>>> file->private_data.
>>>>>
>>>>> So remove chlg_obd_get(), and use file->private_data to find the
>>>>> obd_device.
>>>>> Also ensure the device is fully initialized before calling
>>>>> misc_register().  This means setting up some list linkage before the
>>>>> call, and tearing it down if there is an error.
>>>> I have been testing this but I'm no HSM expert. I pushed this patch
>>>> to OpenSFS branch as well.
>>>>
>>>> https://jira.whamcloud.com/browse/LU-11617
>>>> https://review.whamcloud.com/#/c/33572/
>>>>
>>>> Reviewed-by: James Simmons <jsimmons at infradead.org>
>>> Reviewed-by: Quentin Bouget <quentin.bouget at cea.fr>
>>>
>> Thanks to you both for the review.
>>
>> NeilBrown
>>
> Wait! I just realised there might be another issue!
> I think there is now a race between chlg_open() and 
> mdc_changelog_cdev_finish().

Hmmm.. yes.  Also another deadlock possibility as the locks can be taken
in the wrong order here too.

>
> Wait! I just realised there might be another bigger issue!
> The whole "take the first obd you can find" is broken! I opened a ticket 
> <https://jira.whamcloud.com/browse/LU-11626> on whamcloud's JIRA about it.

Yep...
My guess is that chlg_load(), chlg_clear() and (possibly)
chlg_read_cat_process_cb() should take the mutex, choose an obd, used
it, and release the mutex.

I might post an RFC patch.

NeilBrown


>
> Quentin Bouget
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 832 bytes
Desc: not available
URL: <http://lists.lustre.org/pipermail/lustre-devel_lists.lustre.org/attachments/20181107/df58b631/attachment.sig>

From neilb at suse.com  Wed Nov  7 00:43:02 2018
From: neilb at suse.com (NeilBrown)
Date: Wed, 07 Nov 2018 11:43:02 +1100
Subject: [lustre-devel] [PATCH v2] lustre: mdc: fix possible deadlock in
 chlg_open()
In-Reply-To: <87d0rhybe2.fsf@notabene.neil.brown.name>
References: <87bm7a3nue.fsf@notabene.neil.brown.name>
 <7db27e7d-e685-af8f-80d8-891a0d8db4d5@cea.fr>
 <875zxh3elu.fsf@notabene.neil.brown.name>
 <alpine.LFD.2.21.1811042131490.27308@casper.infradead.org>
 <88ccd141-4778-e3ab-3ce0-e7cbd5c290fc@cea.fr>
 <87wopqyk5h.fsf@notabene.neil.brown.name>
 <a513884a-046f-a28c-3fbb-dd112abd4b26@cea.fr>
 <87d0rhybe2.fsf@notabene.neil.brown.name>
Message-ID: <87a7mlyay1.fsf@notabene.neil.brown.name>

On Wed, Nov 07 2018, NeilBrown wrote:

> On Tue, Nov 06 2018, quentin.bouget at cea.fr wrote:
>
>> Le 06/11/2018 à 04:11, NeilBrown a écrit :
>>> On Mon, Nov 05 2018, quentin.bouget at cea.fr wrote:
>>>
>>>> Le 04/11/2018 à 22:34, James Simmons a écrit :
>>>>>> Lockdep reports a possible deadlock between chlg_open() and
>>>>>> mdc_changelog_cdev_init()
>>>>>>
>>>>>> mdc_changelog_cdev_init() takes chlg_registered_dev_lock and then
>>>>>> calls misc_register() which takes misc_mtx.
>>>>>> chlg_open() is called while misc_mtx is held, and tries to take
>>>>>> chlg_registered_dev_lock.
>>>>>> If these two functions race, a deadlock can occur as each thread will
>>>>>> hold one of the locks while trying to take the other.
>>>>>>
>>>>>> chlg_open() does not need to take a lock.  It only uses the
>>>>>> lock to stablize a list while looking for the matching
>>>>>> chlg_registered_dev, and this can be found directly by examining
>>>>>> file->private_data.
>>>>>>
>>>>>> So remove chlg_obd_get(), and use file->private_data to find the
>>>>>> obd_device.
>>>>>> Also ensure the device is fully initialized before calling
>>>>>> misc_register().  This means setting up some list linkage before the
>>>>>> call, and tearing it down if there is an error.
>>>>> I have been testing this but I'm no HSM expert. I pushed this patch
>>>>> to OpenSFS branch as well.
>>>>>
>>>>> https://jira.whamcloud.com/browse/LU-11617
>>>>> https://review.whamcloud.com/#/c/33572/
>>>>>
>>>>> Reviewed-by: James Simmons <jsimmons at infradead.org>
>>>> Reviewed-by: Quentin Bouget <quentin.bouget at cea.fr>
>>>>
>>> Thanks to you both for the review.
>>>
>>> NeilBrown
>>>
>> Wait! I just realised there might be another issue!
>> I think there is now a race between chlg_open() and 
>> mdc_changelog_cdev_finish().
>
> Hmmm.. yes.  Also another deadlock possibility as the locks can be taken
> in the wrong order here too.

Sorry, I got that wrong - there is no other deadlock.
mdc_changelog_cdev_finish() can call misc_deregister() while holding
chlg_registered_dev_lock, and misc_deregister() will take misc_mtx, but
that is the right way around, not deadlock-causing.

NeilBrown

>
>>
>> Wait! I just realised there might be another bigger issue!
>> The whole "take the first obd you can find" is broken! I opened a ticket 
>> <https://jira.whamcloud.com/browse/LU-11626> on whamcloud's JIRA about it.
>
> Yep...
> My guess is that chlg_load(), chlg_clear() and (possibly)
> chlg_read_cat_process_cb() should take the mutex, choose an obd, used
> it, and release the mutex.
>
> I might post an RFC patch.
>
> NeilBrown
>
>
>>
>> Quentin Bouget
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 832 bytes
Desc: not available
URL: <http://lists.lustre.org/pipermail/lustre-devel_lists.lustre.org/attachments/20181107/fe44f28c/attachment.sig>

From neilb at suse.com  Wed Nov  7 01:09:53 2018
From: neilb at suse.com (NeilBrown)
Date: Wed, 07 Nov 2018 12:09:53 +1100
Subject: [lustre-devel] [PATCH/RFC] lustre: changelog_cdev need to find obd
 on each access.
In-Reply-To: <87d0rhybe2.fsf@notabene.neil.brown.name>
References: <87bm7a3nue.fsf@notabene.neil.brown.name>
 <7db27e7d-e685-af8f-80d8-891a0d8db4d5@cea.fr>
 <875zxh3elu.fsf@notabene.neil.brown.name>
 <alpine.LFD.2.21.1811042131490.27308@casper.infradead.org>
 <88ccd141-4778-e3ab-3ce0-e7cbd5c290fc@cea.fr>
 <87wopqyk5h.fsf@notabene.neil.brown.name>
 <a513884a-046f-a28c-3fbb-dd112abd4b26@cea.fr>
 <87d0rhybe2.fsf@notabene.neil.brown.name>
Message-ID: <877ehpy9pa.fsf@notabene.neil.brown.name>


A changelog cdev can be held open indefinitely, and obd structures can
come and go while it is open.  So it isn't safe to choose and obd
at open time, and continue to use it.

Instead, we need to choose and obd on each access that needs it, and
hold the chlg_registered_dev_lock mutex while using the obd.  This
prevents the obd from being freed.

To help with this, we store a link to the chlg_registered_dev
in the chlg_reader_state, and use the name of the dev (instead
of the name of the obd) in some error messages.

Reported-by: Quentin Bouget <quentin.bouget at cea.fr>
Signed-off-by: NeilBrown <neilb at suse.com>
---

This is and RFC - I have only compile-tested it.
It seems sensible to me, but I know little about what is happening
here so I could easily be missing something important.

See also https://jira.whamcloud.com/browse/LU-11626

Thanks,
NeilBrown



 drivers/staging/lustre/lustre/mdc/mdc_changelog.c | 50 ++++++++++++++++-------
 1 file changed, 36 insertions(+), 14 deletions(-)

diff --git a/drivers/staging/lustre/lustre/mdc/mdc_changelog.c b/drivers/staging/lustre/lustre/mdc/mdc_changelog.c
index becdee86f4d2..ee4b1b95408d 100644
--- a/drivers/staging/lustre/lustre/mdc/mdc_changelog.c
+++ b/drivers/staging/lustre/lustre/mdc/mdc_changelog.c
@@ -66,8 +66,8 @@ struct chlg_registered_dev {
 };
 
 struct chlg_reader_state {
-	/* Shortcut to the corresponding OBD device */
-	struct obd_device	*crs_obd;
+	/* Device this state is associated with */
+	struct chlg_registered_dev *crs_dev;
 	/* Producer thread (if any) */
 	struct task_struct	*crs_prod_task;
 	/* An error occurred that prevents from reading further */
@@ -132,7 +132,7 @@ static int chlg_read_cat_process_cb(const struct lu_env *env,
 	if (rec->cr_hdr.lrh_type != CHANGELOG_REC) {
 		rc = -EINVAL;
 		CERROR("%s: not a changelog rec %x/%d in llog : rc = %d\n",
-		       crs->crs_obd->obd_name, rec->cr_hdr.lrh_type,
+		       crs->crs_dev->ced_name, rec->cr_hdr.lrh_type,
 		       rec->cr.cr_type, rc);
 		return rc;
 	}
@@ -183,6 +183,17 @@ static void enq_record_delete(struct chlg_rec_entry *rec)
 	kfree(rec);
 }
 
+/*
+ * Find any OBD device associated with this reader
+ * chlg_registered_dev_lock is held.
+ */
+static inline struct obd_device *chlg_obd_get(struct chlg_registered_dev *dev)
+{
+	return list_first_entry_or_null(&dev->ced_obds,
+					struct obd_device,
+					u.cli.cl_chg_dev_linkage);
+}
+
 /**
  * Record prefetch thread entry point. Opens the changelog catalog and starts
  * reading records.
@@ -193,11 +204,17 @@ static void enq_record_delete(struct chlg_rec_entry *rec)
 static int chlg_load(void *args)
 {
 	struct chlg_reader_state *crs = args;
-	struct obd_device *obd = crs->crs_obd;
+	struct obd_device *obd;
 	struct llog_ctxt *ctx = NULL;
 	struct llog_handle *llh = NULL;
 	int rc;
 
+	mutex_lock(&chlg_registered_dev_lock);
+	obd = chlg_obd_get(crs->crs_dev);
+	if (!obd) {
+		rc = -ENOENT;
+		goto err_out;
+	}
 	ctx = llog_get_context(obd, LLOG_CHANGELOG_REPL_CTXT);
 	if (!ctx) {
 		rc = -ENOENT;
@@ -230,6 +247,7 @@ static int chlg_load(void *args)
 	crs->crs_eof = true;
 
 err_out:
+	mutex_unlock(&chlg_registered_dev_lock);
 	if (rc < 0)
 		crs->crs_err = true;
 
@@ -387,15 +405,23 @@ static loff_t chlg_llseek(struct file *file, loff_t off, int whence)
  */
 static int chlg_clear(struct chlg_reader_state *crs, u32 reader, u64 record)
 {
-	struct obd_device *obd = crs->crs_obd;
+	struct obd_device *obd;
 	struct changelog_setinfo cs  = {
 		.cs_recno = record,
 		.cs_id    = reader
 	};
+	int ret;
 
-	return obd_set_info_async(NULL, obd->obd_self_export,
-				  strlen(KEY_CHANGELOG_CLEAR),
-				  KEY_CHANGELOG_CLEAR, sizeof(cs), &cs, NULL);
+	mutex_lock(&chlg_registered_dev_lock);
+	obd = chlg_obd_get(crs->crs_dev);
+	if (!obd)
+		ret = -ENOENT;
+	else
+		ret = obd_set_info_async(NULL, obd->obd_self_export,
+					 strlen(KEY_CHANGELOG_CLEAR),
+					 KEY_CHANGELOG_CLEAR, sizeof(cs), &cs, NULL);
+	mutex_unlock(&chlg_registered_dev_lock);
+	return ret;
 }
 
 /** Maximum changelog control command size */
@@ -456,20 +482,16 @@ static int chlg_open(struct inode *inode, struct file *file)
 	struct chlg_reader_state *crs;
 	struct miscdevice *misc = file->private_data;
 	struct chlg_registered_dev *dev;
-	struct obd_device *obd;
 	struct task_struct *task;
 	int rc;
 
 	dev = container_of(misc, struct chlg_registered_dev, ced_misc);
-	obd = list_first_entry(&dev->ced_obds,
-			       struct obd_device,
-			       u.cli.cl_chg_dev_linkage);
 
 	crs = kzalloc(sizeof(*crs), GFP_KERNEL);
 	if (!crs)
 		return -ENOMEM;
 
-	crs->crs_obd = obd;
+	crs->crs_dev = dev;
 	crs->crs_err = false;
 	crs->crs_eof = false;
 
@@ -483,7 +505,7 @@ static int chlg_open(struct inode *inode, struct file *file)
 		if (IS_ERR(task)) {
 			rc = PTR_ERR(task);
 			CERROR("%s: cannot start changelog thread: rc = %d\n",
-			       obd->obd_name, rc);
+			       dev->ced_name, rc);
 			goto err_crs;
 		}
 		crs->crs_prod_task = task;
-- 
2.14.0.rc0.dirty

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 832 bytes
Desc: not available
URL: <http://lists.lustre.org/pipermail/lustre-devel_lists.lustre.org/attachments/20181107/c50ffabf/attachment.sig>

From quentin.bouget at cea.fr  Wed Nov  7 10:05:34 2018
From: quentin.bouget at cea.fr (quentin.bouget at cea.fr)
Date: Wed, 7 Nov 2018 11:05:34 +0100
Subject: [lustre-devel] [PATCH/RFC] lustre: changelog_cdev need to find
 obd on each access.
In-Reply-To: <877ehpy9pa.fsf@notabene.neil.brown.name>
References: <87bm7a3nue.fsf@notabene.neil.brown.name>
 <7db27e7d-e685-af8f-80d8-891a0d8db4d5@cea.fr>
 <875zxh3elu.fsf@notabene.neil.brown.name>
 <alpine.LFD.2.21.1811042131490.27308@casper.infradead.org>
 <88ccd141-4778-e3ab-3ce0-e7cbd5c290fc@cea.fr>
 <87wopqyk5h.fsf@notabene.neil.brown.name>
 <a513884a-046f-a28c-3fbb-dd112abd4b26@cea.fr>
 <87d0rhybe2.fsf@notabene.neil.brown.name>
 <877ehpy9pa.fsf@notabene.neil.brown.name>
Message-ID: <bb0e1f58-37a1-35e7-4ec2-dc8a496d2d99@cea.fr>

Le 07/11/2018 à 02:09, NeilBrown a écrit :
> A changelog cdev can be held open indefinitely, and obd structures can
> come and go while it is open.  So it isn't safe to choose and obd
> at open time, and continue to use it.
>
> Instead, we need to choose and obd on each access that needs it, and
> hold the chlg_registered_dev_lock mutex while using the obd.  This
> prevents the obd from being freed.
`struct llog_ctxt' has a reference on that obd (and `struct llog_handle' 
has a reference on a `struct llog_ctxt').
I don't think letting the obd go away is safe.

> @@ -193,11 +204,17 @@ static void enq_record_delete(struct chlg_rec_entry *rec)
>   static int chlg_load(void *args)
>   {
>   	struct chlg_reader_state *crs = args;
> -	struct obd_device *obd = crs->crs_obd;
> +	struct obd_device *obd;
>   	struct llog_ctxt *ctx = NULL;
>   	struct llog_handle *llh = NULL;
>   	int rc;
>   
> +	mutex_lock(&chlg_registered_dev_lock);
> +	obd = chlg_obd_get(crs->crs_dev);
> +	if (!obd) {
> +		rc = -ENOENT;
> +		goto err_out;
> +	}
>   	ctx = llog_get_context(obd, LLOG_CHANGELOG_REPL_CTXT);
 From there the obd is irrevocably referenced. I suspect it is used for 
more than just its name.
Maybe trying to unmount the (wrong) mountpoint will fail after that, but 
I am not confident.

Quentin

From jsimmons at infradead.org  Wed Nov  7 19:24:27 2018
From: jsimmons at infradead.org (James Simmons)
Date: Wed, 7 Nov 2018 19:24:27 +0000 (GMT)
Subject: [lustre-devel] Apparent bug in LU-8130 ptlrpc: convert
 conn_hash to rhashtable / Linux-commit:
 ac2370ac2bc5215daf78546cd8d925510065bb7f
In-Reply-To: <87r2fxyhej.fsf@notabene.neil.brown.name>
References: <3E0E82E2-0EF0-4F05-9C33-282002000D31@cray.com>
 <87r2fxyhej.fsf@notabene.neil.brown.name>
Message-ID: <alpine.LFD.2.21.1811071922400.3887@casper.infradead.org>


> > Neil, James,
> >
> > Sent this earlier, but with nasty formatting that got it rejected.  Whoops.
> >  
> > It looks like the patch we landed as:
> > LU-8130 ptlrpc: convert conn_hash to rhashtable
> >  
> > Linux has a resizeable hashtable implementation in lib,
> > so we should use that instead of having one in libcfs.
> >  
> > This patch converts the ptlrpc conn_hash to use rhashtable.
> > In the process we gain lockless lookup.
> >  
> > As connections are never deleted until the hash table is destroyed,
> > there is no need to count the reference in the hash table. There
> > is also no need to enable automatic_shrinking.
> >  
> > Linux-commit: ac2370ac2bc5215daf78546cd8d925510065bb7f
> >  
> > Introduced a bug.  Ihara-san opened something to track it here:
> > https://jira.whamcloud.com/browse/LU-11624
> >  
> > It’s a null pointer in nid_hash(); there are some more details at that link.
> 
> That commit changed ptlrpc_connection_get() so that it can return NULL.
> 
> This is actually one of the really annoying things about rhashtable.  I
> cannot convince the maintainers that sometimes you need guaranteed
> success (which is quite possible with a hash table).  They come from the
> networking community were they drop packets on errors all the time - no
> big deal.
> 
> Anyway, all users of ptlrpc_connection_get() in the client code (in
> drivers/staging) test the return value against NULL, so this was safe.
> 
> In SFS lustre, the call in target_handle_connect() assigned the
> result to export->exp_connection without checking against NULL.
> That leads to the oops.
> 
> ptlrpc_connection_get() will return NULL when
> rhashtable_lookup_get_insert_fast() returns an error.
> The errors are mostly weird internal implementation details leaking out
> the API.  A retry, possibly after a short delay, will usually succeed.
> (see https://lwn.net/Articles/751974/, section "Failure during insertion)
> If this happens a lot, it might suggest that the hash function is poor.
> 
> Ahhhh... this was built against a 3.10 kernel.
> Prior to 4.6, hash_64() was not so much "poor" as "appalling".
> See https://lwn.net/Articles/687494/
> 
> For SFS we should avoid using hash_64() (at least before 4.6) and
> instead use something like
>   hash_32((foo>>32) ^ hash_32(foo & 0xFFFFFFFF))

The libcfs hash is not much better :-( 

I have a purposed fix. Let me know if its correct. Thank you.

>From 32df018c1af8fdbc833c537aafeea081ab3d8d7e Mon Sep 17 00:00:00 2001
From: James Simmons <uja.ornl at yahoo.com>
Date: Wed, 7 Nov 2018 14:06:37 -0500
Subject: [PATCH] LU-11624 ptlrpc: handle no ptlrpc connection case

With the move of ptlrpc connections hash to rhashtable it is now
possible for ptlrpc_connection_get() to return NULL. The reason
this can happen is due to conditions. One is that the hash has no
more free slots which results in an E2BIG error. By default the
maximum size for rhashtables is 2^31 so this is very unlikely
going to happen. The second case returns a -ENOMEM which can
happen when the bucket can be resized. For this case test the
return value of rhashtable_lookup_get_insert_fast() and if it is
-ENOMEM try again inserting the new connection into the hash
table. Just as an extra level of protection against a failed
connection attempt in target_handle_connect() exit out with
-ENOTCONN.

Change-Id: I0537564ba544ed06be42ba243606a884a1290f20
Signed-off-by: James Simmons <uja.ornl at yahoo.com>
---
 lustre/ldlm/ldlm_lib.c     | 3 +++
 lustre/ptlrpc/connection.c | 6 +++++-
 2 files changed, 8 insertions(+), 1 deletion(-)

diff --git a/lustre/ldlm/ldlm_lib.c b/lustre/ldlm/ldlm_lib.c
index de66aa3..463c6c6 100644
--- a/lustre/ldlm/ldlm_lib.c
+++ b/lustre/ldlm/ldlm_lib.c
@@ -1349,6 +1349,9 @@ dont_check_exports:
 	export->exp_connection = ptlrpc_connection_get(req->rq_peer,
 						       req->rq_self,
 						       &cluuid);
+	if (!export->exp_connection)
+		GOTO(out, rc = -ENOTCONN);
+
 	if (hlist_unhashed(&export->exp_nid_hash))
 		cfs_hash_add(export->exp_obd->obd_nid_hash,
 			     &export->exp_connection->c_peer.nid,
diff --git a/lustre/ptlrpc/connection.c b/lustre/ptlrpc/connection.c
index 21a24a2..045b3ee 100644
--- a/lustre/ptlrpc/connection.c
+++ b/lustre/ptlrpc/connection.c
@@ -103,13 +103,17 @@ ptlrpc_connection_get(struct lnet_process_id peer, lnet_nid_t self,
 	 * connection.	The object which exists in the hash will be
 	 * returned,otherwise NULL is returned on success.
 	 */
+try_again:
 	conn2 = rhashtable_lookup_get_insert_fast(&conn_hash, &conn->c_hash,
 						  conn_hash_params);
 	if (conn2) {
 		/* insertion failed */
 		OBD_FREE_PTR(conn);
-		if (IS_ERR(conn2))
+		if (IS_ERR(conn2)) {
+			if (PTR_ERR(conn2) == -ENOMEM)
+				goto try_again;
 			return NULL;
+		}
 		conn = conn2;
 		ptlrpc_connection_addref(conn);
 	}
-- 
1.8.3.1

From paf at cray.com  Wed Nov  7 19:56:12 2018
From: paf at cray.com (Patrick Farrell)
Date: Wed, 7 Nov 2018 19:56:12 +0000
Subject: [lustre-devel] Apparent bug in LU-8130 ptlrpc: convert
 conn_hash to rhashtable / Linux-commit:
 ac2370ac2bc5215daf78546cd8d925510065bb7f
In-Reply-To: <alpine.LFD.2.21.1811071922400.3887@casper.infradead.org>
References: <3E0E82E2-0EF0-4F05-9C33-282002000D31@cray.com>
 <87r2fxyhej.fsf@notabene.neil.brown.name>,
 <alpine.LFD.2.21.1811071922400.3887@casper.infradead.org>
Message-ID: <BN6PR11MB1475520E7E3172593CEB8138CBC40@BN6PR11MB1475.namprd11.prod.outlook.com>


Thanks much, both of you.  I’ll take a look.

________________________________
From: James Simmons <jsimmons at infradead.org>
Sent: Wednesday, November 7, 2018 1:24:27 PM
To: NeilBrown
Cc: Patrick Farrell; Lustre Developement; James Simmons; sihara at whamcloud.com
Subject: Re: Apparent bug in LU-8130 ptlrpc: convert conn_hash to rhashtable / Linux-commit: ac2370ac2bc5215daf78546cd8d925510065bb7f


> > Neil, James,
> >
> > Sent this earlier, but with nasty formatting that got it rejected.  Whoops.
> >
> > It looks like the patch we landed as:
> > LU-8130 ptlrpc: convert conn_hash to rhashtable
> >
> > Linux has a resizeable hashtable implementation in lib,
> > so we should use that instead of having one in libcfs.
> >
> > This patch converts the ptlrpc conn_hash to use rhashtable.
> > In the process we gain lockless lookup.
> >
> > As connections are never deleted until the hash table is destroyed,
> > there is no need to count the reference in the hash table. There
> > is also no need to enable automatic_shrinking.
> >
> > Linux-commit: ac2370ac2bc5215daf78546cd8d925510065bb7f
> >
> > Introduced a bug.  Ihara-san opened something to track it here:
> > https://jira.whamcloud.com/browse/LU-11624
> >
> > It’s a null pointer in nid_hash(); there are some more details at that link.
>
> That commit changed ptlrpc_connection_get() so that it can return NULL.
>
> This is actually one of the really annoying things about rhashtable.  I
> cannot convince the maintainers that sometimes you need guaranteed
> success (which is quite possible with a hash table).  They come from the
> networking community were they drop packets on errors all the time - no
> big deal.
>
> Anyway, all users of ptlrpc_connection_get() in the client code (in
> drivers/staging) test the return value against NULL, so this was safe.
>
> In SFS lustre, the call in target_handle_connect() assigned the
> result to export->exp_connection without checking against NULL.
> That leads to the oops.
>
> ptlrpc_connection_get() will return NULL when
> rhashtable_lookup_get_insert_fast() returns an error.
> The errors are mostly weird internal implementation details leaking out
> the API.  A retry, possibly after a short delay, will usually succeed.
> (see https://lwn.net/Articles/751974/, section "Failure during insertion)
> If this happens a lot, it might suggest that the hash function is poor.
>
> Ahhhh... this was built against a 3.10 kernel.
> Prior to 4.6, hash_64() was not so much "poor" as "appalling".
> See https://lwn.net/Articles/687494/
>
> For SFS we should avoid using hash_64() (at least before 4.6) and
> instead use something like
>   hash_32((foo>>32) ^ hash_32(foo & 0xFFFFFFFF))

The libcfs hash is not much better :-(

I have a purposed fix. Let me know if its correct. Thank you.

>From 32df018c1af8fdbc833c537aafeea081ab3d8d7e Mon Sep 17 00:00:00 2001
From: James Simmons <uja.ornl at yahoo.com>
Date: Wed, 7 Nov 2018 14:06:37 -0500
Subject: [PATCH] LU-11624 ptlrpc: handle no ptlrpc connection case

With the move of ptlrpc connections hash to rhashtable it is now
possible for ptlrpc_connection_get() to return NULL. The reason
this can happen is due to conditions. One is that the hash has no
more free slots which results in an E2BIG error. By default the
maximum size for rhashtables is 2^31 so this is very unlikely
going to happen. The second case returns a -ENOMEM which can
happen when the bucket can be resized. For this case test the
return value of rhashtable_lookup_get_insert_fast() and if it is
-ENOMEM try again inserting the new connection into the hash
table. Just as an extra level of protection against a failed
connection attempt in target_handle_connect() exit out with
-ENOTCONN.

Change-Id: I0537564ba544ed06be42ba243606a884a1290f20
Signed-off-by: James Simmons <uja.ornl at yahoo.com>
---
 lustre/ldlm/ldlm_lib.c     | 3 +++
 lustre/ptlrpc/connection.c | 6 +++++-
 2 files changed, 8 insertions(+), 1 deletion(-)

diff --git a/lustre/ldlm/ldlm_lib.c b/lustre/ldlm/ldlm_lib.c
index de66aa3..463c6c6 100644
--- a/lustre/ldlm/ldlm_lib.c
+++ b/lustre/ldlm/ldlm_lib.c
@@ -1349,6 +1349,9 @@ dont_check_exports:
         export->exp_connection = ptlrpc_connection_get(req->rq_peer,
                                                        req->rq_self,
                                                        &cluuid);
+       if (!export->exp_connection)
+               GOTO(out, rc = -ENOTCONN);
+
         if (hlist_unhashed(&export->exp_nid_hash))
                 cfs_hash_add(export->exp_obd->obd_nid_hash,
                              &export->exp_connection->c_peer.nid,
diff --git a/lustre/ptlrpc/connection.c b/lustre/ptlrpc/connection.c
index 21a24a2..045b3ee 100644
--- a/lustre/ptlrpc/connection.c
+++ b/lustre/ptlrpc/connection.c
@@ -103,13 +103,17 @@ ptlrpc_connection_get(struct lnet_process_id peer, lnet_nid_t self,
          * connection.   The object which exists in the hash will be
          * returned,otherwise NULL is returned on success.
          */
+try_again:
         conn2 = rhashtable_lookup_get_insert_fast(&conn_hash, &conn->c_hash,
                                                   conn_hash_params);
         if (conn2) {
                 /* insertion failed */
                 OBD_FREE_PTR(conn);
-               if (IS_ERR(conn2))
+               if (IS_ERR(conn2)) {
+                       if (PTR_ERR(conn2) == -ENOMEM)
+                               goto try_again;
                         return NULL;
+               }
                 conn = conn2;
                 ptlrpc_connection_addref(conn);
         }
--
1.8.3.1
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-devel_lists.lustre.org/attachments/20181107/007ac0d2/attachment.html>

From robl at mcs.anl.gov  Wed Nov  7 20:32:37 2018
From: robl at mcs.anl.gov (Latham, Robert J.)
Date: Wed, 7 Nov 2018 20:32:37 +0000
Subject: [lustre-devel] Limitations of kernel read ahead
In-Reply-To: <62D5855B-8FE1-4157-858C-01B182D759AD@ddn.com>
References: <alpine.LFD.2.21.1810291532100.21751@casper.infradead.org>
 <62D5855B-8FE1-4157-858C-01B182D759AD@ddn.com>
Message-ID: <4d9ebc734ecef37322c89cfa171d987cf111c316.camel@mcs.anl.gov>

On Tue, 2018-10-30 at 02:00 +0000, Li Xi wrote:
> Thank you for summarize this, James!
> 
> I think everyone agrees that the current readahead algorithm of
> Lustre needs to be improved. And evidences show that the readahead
> algorithm of Linux kernel would not suitable for Lustre either. There
> are several reasons for this. In general, the readahead algorithm of
> kernel is designed for local file system with small readahead window.
> It is single thread, synchronous readahead, only usable for
> sequential read. Because the read operation of Lustre is has longer
> latency than local file system, while its bandwidth is typically
> higher than local file system, we need totally different algorithm
> for Lustre readahead. The readahead algorithm needs to be 1)
> asynchronous to hide latency for application 2) multiple threaded to
> utilize the high bandwidth 3) use big readahead window to align with
> the big RPC size 4) work for sequential read, stride read and
> potentially small & random read.

Please don't forget that HPC workloads are likely to fall into category
4.

==rob

> The work of LU-8709 was started with these targets and got pretty
> good numbers even without detailed tuning. We (the Whamcloud team)
> would like to rework on it with a goal of merging it in the next
> releases of Lustre.
> 
> Regards,
> Li Xi
> 
> 在 2018/10/30 上午2:06，“James Simmons”<jsimmons at infradead.org> 写入:
> 
>     
>     Currently the lustre client has its own read ahead handling in
> the CLIO 
>     layer. The reason for this is due to some limitations in the read
> ahead
>     code for the linux kernel. Some work to use the kernel's read
> ahead was 
>     attempted for the LU-8964 work but the general work for LU-8964
> had other
>     issues. Alternative work to LU-8964 has emerged under ticket
>     
>     https://jira.whamcloud.com/browse/LU-8709
>     
>     with early code at:
>     
>     https://review.whamcloud.com/#/c/23552
>     
>     Also I have included a link to a presentation of this work and it
> gives
>     insight on how lustre does its own read ahead.
>     
>     
> 
https://www.eofs.eu/_media/events/lad16/19_parallel_readahead_framework_li_xi.pdf
>     
>     Now that this seems to be the targeted work for read ahead the
> discussion
>     has come up about why this new work doesn't use the kernel read
> ahead 
>     again. I wasn't involved in the discussion about the limitations
> but I 
>     have included the people interested in this work so progress can
> be done
>     to imporve the linux kernels version of read ahead.
>     
> 
> _______________________________________________
> lustre-devel mailing list
> lustre-devel at lists.lustre.org
> http://lists.lustre.org/listinfo.cgi/lustre-devel-lustre.org


From neilb at suse.com  Thu Nov  8 01:15:16 2018
From: neilb at suse.com (NeilBrown)
Date: Thu, 08 Nov 2018 12:15:16 +1100
Subject: [lustre-devel] Apparent bug in LU-8130 ptlrpc: convert
 conn_hash to rhashtable / Linux-commit:
 ac2370ac2bc5215daf78546cd8d925510065bb7f
In-Reply-To: <alpine.LFD.2.21.1811071922400.3887@casper.infradead.org>
References: <3E0E82E2-0EF0-4F05-9C33-282002000D31@cray.com>
 <87r2fxyhej.fsf@notabene.neil.brown.name>
 <alpine.LFD.2.21.1811071922400.3887@casper.infradead.org>
Message-ID: <87y3a4wesb.fsf@notabene.neil.brown.name>

On Wed, Nov 07 2018, James Simmons wrote:

>> > Neil, James,
>> >
>> > Sent this earlier, but with nasty formatting that got it rejected.  Whoops.
>> >  
>> > It looks like the patch we landed as:
>> > LU-8130 ptlrpc: convert conn_hash to rhashtable
>> >  
>> > Linux has a resizeable hashtable implementation in lib,
>> > so we should use that instead of having one in libcfs.
>> >  
>> > This patch converts the ptlrpc conn_hash to use rhashtable.
>> > In the process we gain lockless lookup.
>> >  
>> > As connections are never deleted until the hash table is destroyed,
>> > there is no need to count the reference in the hash table. There
>> > is also no need to enable automatic_shrinking.
>> >  
>> > Linux-commit: ac2370ac2bc5215daf78546cd8d925510065bb7f
>> >  
>> > Introduced a bug.  Ihara-san opened something to track it here:
>> > https://jira.whamcloud.com/browse/LU-11624
>> >  
>> > It’s a null pointer in nid_hash(); there are some more details at that link.
>> 
>> That commit changed ptlrpc_connection_get() so that it can return NULL.
>> 
>> This is actually one of the really annoying things about rhashtable.  I
>> cannot convince the maintainers that sometimes you need guaranteed
>> success (which is quite possible with a hash table).  They come from the
>> networking community were they drop packets on errors all the time - no
>> big deal.
>> 
>> Anyway, all users of ptlrpc_connection_get() in the client code (in
>> drivers/staging) test the return value against NULL, so this was safe.
>> 
>> In SFS lustre, the call in target_handle_connect() assigned the
>> result to export->exp_connection without checking against NULL.
>> That leads to the oops.
>> 
>> ptlrpc_connection_get() will return NULL when
>> rhashtable_lookup_get_insert_fast() returns an error.
>> The errors are mostly weird internal implementation details leaking out
>> the API.  A retry, possibly after a short delay, will usually succeed.
>> (see https://lwn.net/Articles/751974/, section "Failure during insertion)
>> If this happens a lot, it might suggest that the hash function is poor.
>> 
>> Ahhhh... this was built against a 3.10 kernel.
>> Prior to 4.6, hash_64() was not so much "poor" as "appalling".
>> See https://lwn.net/Articles/687494/
>> 
>> For SFS we should avoid using hash_64() (at least before 4.6) and
>> instead use something like
>>   hash_32((foo>>32) ^ hash_32(foo & 0xFFFFFFFF))
>
> The libcfs hash is not much better :-( 
>
> I have a purposed fix. Let me know if its correct. Thank you.
>
> From 32df018c1af8fdbc833c537aafeea081ab3d8d7e Mon Sep 17 00:00:00 2001
> From: James Simmons <uja.ornl at yahoo.com>
> Date: Wed, 7 Nov 2018 14:06:37 -0500
> Subject: [PATCH] LU-11624 ptlrpc: handle no ptlrpc connection case
>
> With the move of ptlrpc connections hash to rhashtable it is now
> possible for ptlrpc_connection_get() to return NULL. The reason
> this can happen is due to conditions. One is that the hash has no
                           ^two ??

Actually there are three.  -EBUSY is also possible, and is the one that
might be caused by a bad hash function.

I would retry on both -ENOMEM and -EBUSY.
Maybe give a warning if -EBUSY.

Otherwise it looks OK.

Thanks,
NeilBrown


> more free slots which results in an E2BIG error. By default the
> maximum size for rhashtables is 2^31 so this is very unlikely
> going to happen. The second case returns a -ENOMEM which can
> happen when the bucket can be resized. For this case test the
> return value of rhashtable_lookup_get_insert_fast() and if it is
> -ENOMEM try again inserting the new connection into the hash
> table. Just as an extra level of protection against a failed
> connection attempt in target_handle_connect() exit out with
> -ENOTCONN.
>
> Change-Id: I0537564ba544ed06be42ba243606a884a1290f20
> Signed-off-by: James Simmons <uja.ornl at yahoo.com>
> ---
>  lustre/ldlm/ldlm_lib.c     | 3 +++
>  lustre/ptlrpc/connection.c | 6 +++++-
>  2 files changed, 8 insertions(+), 1 deletion(-)
>
> diff --git a/lustre/ldlm/ldlm_lib.c b/lustre/ldlm/ldlm_lib.c
> index de66aa3..463c6c6 100644
> --- a/lustre/ldlm/ldlm_lib.c
> +++ b/lustre/ldlm/ldlm_lib.c
> @@ -1349,6 +1349,9 @@ dont_check_exports:
>  	export->exp_connection = ptlrpc_connection_get(req->rq_peer,
>  						       req->rq_self,
>  						       &cluuid);
> +	if (!export->exp_connection)
> +		GOTO(out, rc = -ENOTCONN);
> +
>  	if (hlist_unhashed(&export->exp_nid_hash))
>  		cfs_hash_add(export->exp_obd->obd_nid_hash,
>  			     &export->exp_connection->c_peer.nid,
> diff --git a/lustre/ptlrpc/connection.c b/lustre/ptlrpc/connection.c
> index 21a24a2..045b3ee 100644
> --- a/lustre/ptlrpc/connection.c
> +++ b/lustre/ptlrpc/connection.c
> @@ -103,13 +103,17 @@ ptlrpc_connection_get(struct lnet_process_id peer, lnet_nid_t self,
>  	 * connection.	The object which exists in the hash will be
>  	 * returned,otherwise NULL is returned on success.
>  	 */
> +try_again:
>  	conn2 = rhashtable_lookup_get_insert_fast(&conn_hash, &conn->c_hash,
>  						  conn_hash_params);
>  	if (conn2) {
>  		/* insertion failed */
>  		OBD_FREE_PTR(conn);
> -		if (IS_ERR(conn2))
> +		if (IS_ERR(conn2)) {
> +			if (PTR_ERR(conn2) == -ENOMEM)
> +				goto try_again;
>  			return NULL;
> +		}
>  		conn = conn2;
>  		ptlrpc_connection_addref(conn);
>  	}
> -- 
> 1.8.3.1
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 832 bytes
Desc: not available
URL: <http://lists.lustre.org/pipermail/lustre-devel_lists.lustre.org/attachments/20181108/f426fd66/attachment.sig>

From paf at cray.com  Fri Nov 16 18:06:41 2018
From: paf at cray.com (Patrick Farrell)
Date: Fri, 16 Nov 2018 18:06:41 +0000
Subject: [lustre-devel] Compact layouts
Message-ID: <3600968B-94D3-4C3F-857A-D00A5C3BC601@cray.com>

All,

There is an old idea for reducing the data required to describe file striping by using a bitmap to record which OSTs are in use.  As best I can tell, this was most recently described here:
http://wiki.lustre.org/Layout_Enhancement_Solution_Architecture#Compact_Layouts_2

I’m curious if this has been pursued any further, if there’s a JIRA or other place that might have more info or be tracking the idea.  I poked around and didn’t find anything.

In particular, this comment:
“with enough data that for each OST index set in the bitmap, a corresponding OST object FID may be computed”
Points at the difficult part of implementing this.

So, before I get too far considering this problem - Is there more out there somewhere?  Hoping to avoid duplicating work!

Thanks,

  *   Patrick
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-devel_lists.lustre.org/attachments/20181116/cb32f66b/attachment.html>

From adilger at whamcloud.com  Wed Nov 21 23:53:03 2018
From: adilger at whamcloud.com (Andreas Dilger)
Date: Wed, 21 Nov 2018 23:53:03 +0000
Subject: [lustre-devel] Compact layouts
In-Reply-To: <3600968B-94D3-4C3F-857A-D00A5C3BC601@cray.com>
References: <3600968B-94D3-4C3F-857A-D00A5C3BC601@cray.com>
Message-ID: <E2B38E8A-55A5-481D-AF44-FF0B4CD8472E@whamcloud.com>

On Nov 16, 2018, at 11:06, Patrick Farrell <paf at cray.com> wrote:
> 
> All,
>  
> There is an old idea for reducing the data required to describe file striping by using a bitmap to record which OSTs are in use.  As best I can tell, this was most recently described here:
> http://wiki.lustre.org/Layout_Enhancement_Solution_Architecture#Compact_Layouts_2
>  
> I’m curious if this has been pursued any further, if there’s a JIRA or other place that might have more info or be tracking the idea.  I poked around and didn’t find anything.
>  
> In particular, this comment:
> “with enough data that for each OST index set in the bitmap, a corresponding OST object FID may be computed”
> Points at the difficult part of implementing this.
>  
> So, before I get too far considering this problem - Is there more out there somewhere?  Hoping to avoid duplicating work!

Patrick,
as you mention above, the tricky part is that there would need to be sequential FID sequence allocation across all of the OSTs.  Then, each of the compact files would allocate/reserve the same OID in each of the sequences so that the mapping could be compact.  I don't think that is insurmountable - we already have a good mechanism for allocating FID sequences to different targets, but it would need to be extended so that compact layouts would allocate sequences from a different range of values from regular layouts.

It would also likely need to implement "OST object create on write" so that there aren't large numbers of unused objects on each OST (one for each OID that isn't used on a particular file).

The other issue is that anything like migrating any single object to another OST (e.g. for mirror resync, tiering, etc) would potentially break the compact layout.

I guess the question is what the need for compact layouts is?  To handle more than 2000 stripes, to reduce the xattr size/RPC size, to allow more complex PFL layouts to fit into the layout size limit?

In the past we discussed compressing the layout with gzip, which might be quite effective since large parts of it are zero-filled and repetitive.  This would help the xattr/RPC size, and I think even with compact layouts that they would still be expanded in RAM to allow easier processing.

Cheers, Andreas
---
Andreas Dilger
Principal Lustre Architect
Whamcloud








From paf at cray.com  Thu Nov 22 02:27:42 2018
From: paf at cray.com (Patrick Farrell)
Date: Thu, 22 Nov 2018 02:27:42 +0000
Subject: [lustre-devel] Compact layouts
In-Reply-To: <E2B38E8A-55A5-481D-AF44-FF0B4CD8472E@whamcloud.com>
References: <3600968B-94D3-4C3F-857A-D00A5C3BC601@cray.com>,
 <E2B38E8A-55A5-481D-AF44-FF0B4CD8472E@whamcloud.com>
Message-ID: <MWHPR11MB1488BCF770411CE094423321CBDB0@MWHPR11MB1488.namprd11.prod.outlook.com>

Andreas,


Thanks for the informative reply.


You raise an interesting and nasty point about breaking the compact layout with movement.  It's not possible today to move an individual OST object/stripe, though it's certainly something I've heard people ask for.  So it wouldn't be an issue if all such operations must address whole components, as is required today.


If we did add the ability to switch out an individual OST object/stripe (which would be pretty easy to implement - data copy, layout swap, rm now-unused object), we could add those modifications as additional "traditional" layout info "atop" the compact layout.  So just the usual layout format, with OST IDs, and where present, it supersedes the relevant part of the compact layout.  This implicitly assumes we don't do a ton of this to a particular layout.


But as to reasons, it's a few things.


The primary concern is improving the open performance of very widely striped files, which means your second case - reduce the xattr and rpc size.


The same things that motivate this would also motivate raising the count limit, but my understanding from comments in the code is that 2000 is arbitrary, and the actual max could be quite a bit higher.  The first limit I'm aware of - I'm not sure if this is right? - is 1 MiB of extended attribute.  That's a little over 5000 stripes.  (Obviously, 1 MiB of layout is probably a non-starter...)


Your suggestion of gzip is very intriguing.  Ideally, I'd pick something available in kernel and with good performance.  A bit of experimentation is probably in order if we go that route.  Thanks for the pointer there.  I'd probably start with extracting the binary xattr and seeing how it compresses.


- Patrick

________________________________
From: Andreas Dilger <adilger at whamcloud.com>
Sent: Wednesday, November 21, 2018 5:53:03 PM
To: Patrick Farrell
Cc: Lustre Developement
Subject: Re: [lustre-devel] Compact layouts

On Nov 16, 2018, at 11:06, Patrick Farrell <paf at cray.com> wrote:
>
> All,
>
> There is an old idea for reducing the data required to describe file striping by using a bitmap to record which OSTs are in use.  As best I can tell, this was most recently described here:
> http://wiki.lustre.org/Layout_Enhancement_Solution_Architecture#Compact_Layouts_2
>
> I’m curious if this has been pursued any further, if there’s a JIRA or other place that might have more info or be tracking the idea.  I poked around and didn’t find anything.
>
> In particular, this comment:
> “with enough data that for each OST index set in the bitmap, a corresponding OST object FID may be computed”
> Points at the difficult part of implementing this.
>
> So, before I get too far considering this problem - Is there more out there somewhere?  Hoping to avoid duplicating work!

Patrick,
as you mention above, the tricky part is that there would need to be sequential FID sequence allocation across all of the OSTs.  Then, each of the compact files would allocate/reserve the same OID in each of the sequences so that the mapping could be compact.  I don't think that is insurmountable - we already have a good mechanism for allocating FID sequences to different targets, but it would need to be extended so that compact layouts would allocate sequences from a different range of values from regular layouts.

It would also likely need to implement "OST object create on write" so that there aren't large numbers of unused objects on each OST (one for each OID that isn't used on a particular file).

The other issue is that anything like migrating any single object to another OST (e.g. for mirror resync, tiering, etc) would potentially break the compact layout.

I guess the question is what the need for compact layouts is?  To handle more than 2000 stripes, to reduce the xattr size/RPC size, to allow more complex PFL layouts to fit into the layout size limit?

In the past we discussed compressing the layout with gzip, which might be quite effective since large parts of it are zero-filled and repetitive.  This would help the xattr/RPC size, and I think even with compact layouts that they would still be expanded in RAM to allow easier processing.

Cheers, Andreas
---
Andreas Dilger
Principal Lustre Architect
Whamcloud







-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-devel_lists.lustre.org/attachments/20181122/16669351/attachment.html>

From johnbent at gmail.com  Thu Nov 22 02:30:49 2018
From: johnbent at gmail.com (John Bent)
Date: Wed, 21 Nov 2018 19:30:49 -0700
Subject: [lustre-devel] Compact layouts
In-Reply-To: <MWHPR11MB1488BCF770411CE094423321CBDB0@MWHPR11MB1488.namprd11.prod.outlook.com>
References: <3600968B-94D3-4C3F-857A-D00A5C3BC601@cray.com>
 <E2B38E8A-55A5-481D-AF44-FF0B4CD8472E@whamcloud.com>
 <MWHPR11MB1488BCF770411CE094423321CBDB0@MWHPR11MB1488.namprd11.prod.outlook.com>
Message-ID: <F5BAE559-E5F9-47C5-BB8C-8D2553F7BD43@gmail.com>

As HW latencies shrink to zero, does it not make you nervous to suggest adding compression into the metadata critical path?

> On Nov 21, 2018, at 7:27 PM, Patrick Farrell <paf at cray.com> wrote:
> 
> Andreas,
> 
> Thanks for the informative reply.
> 
> You raise an interesting and nasty point about breaking the compact layout with movement.  It's not possible today to move an individual OST object/stripe, though it's certainly something I've heard people ask for.  So it wouldn't be an issue if all such operations must address whole components, as is required today.
> 
> If we did add the ability to switch out an individual OST object/stripe (which would be pretty easy to implement - data copy, layout swap, rm now-unused object), we could add those modifications as additional "traditional" layout info "atop" the compact layout.  So just the usual layout format, with OST IDs, and where present, it supersedes the relevant part of the compact layout.  This implicitly assumes we don't do a ton of this to a particular layout.
> 
> But as to reasons, it's a few things.
> 
> The primary concern is improving the open performance of very widely striped files, which means your second case - reduce the xattr and rpc size.
> 
> The same things that motivate this would also motivate raising the count limit, but my understanding from comments in the code is that 2000 is arbitrary, and the actual max could be quite a bit higher.  The first limit I'm aware of - I'm not sure if this is right? - is 1 MiB of extended attribute.  That's a little over 5000 stripes.  (Obviously, 1 MiB of layout is probably a non-starter...)
> 
> Your suggestion of gzip is very intriguing.  Ideally, I'd pick something available in kernel and with good performance.  A bit of experimentation is probably in order if we go that route.  Thanks for the pointer there.  I'd probably start with extracting the binary xattr and seeing how it compresses.
> 
> - Patrick
> From: Andreas Dilger <adilger at whamcloud.com>
> Sent: Wednesday, November 21, 2018 5:53:03 PM
> To: Patrick Farrell
> Cc: Lustre Developement
> Subject: Re: [lustre-devel] Compact layouts
>  
> On Nov 16, 2018, at 11:06, Patrick Farrell <paf at cray.com> wrote:
> > 
> > All,
> >  
> > There is an old idea for reducing the data required to describe file striping by using a bitmap to record which OSTs are in use.  As best I can tell, this was most recently described here:
> > http://wiki.lustre.org/Layout_Enhancement_Solution_Architecture#Compact_Layouts_2
> >  
> > I’m curious if this has been pursued any further, if there’s a JIRA or other place that might have more info or be tracking the idea.  I poked around and didn’t find anything.
> >  
> > In particular, this comment:
> > “with enough data that for each OST index set in the bitmap, a corresponding OST object FID may be computed”
> > Points at the difficult part of implementing this.
> >  
> > So, before I get too far considering this problem - Is there more out there somewhere?  Hoping to avoid duplicating work!
> 
> Patrick,
> as you mention above, the tricky part is that there would need to be sequential FID sequence allocation across all of the OSTs.  Then, each of the compact files would allocate/reserve the same OID in each of the sequences so that the mapping could be compact.  I don't think that is insurmountable - we already have a good mechanism for allocating FID sequences to different targets, but it would need to be extended so that compact layouts would allocate sequences from a different range of values from regular layouts.
> 
> It would also likely need to implement "OST object create on write" so that there aren't large numbers of unused objects on each OST (one for each OID that isn't used on a particular file).
> 
> The other issue is that anything like migrating any single object to another OST (e.g. for mirror resync, tiering, etc) would potentially break the compact layout.
> 
> I guess the question is what the need for compact layouts is?  To handle more than 2000 stripes, to reduce the xattr size/RPC size, to allow more complex PFL layouts to fit into the layout size limit?
> 
> In the past we discussed compressing the layout with gzip, which might be quite effective since large parts of it are zero-filled and repetitive.  This would help the xattr/RPC size, and I think even with compact layouts that they would still be expanded in RAM to allow easier processing.
> 
> Cheers, Andreas
> ---
> Andreas Dilger
> Principal Lustre Architect
> Whamcloud
> 
> 
> 
> 
> 
> 
> 
> _______________________________________________
> lustre-devel mailing list
> lustre-devel at lists.lustre.org
> http://lists.lustre.org/listinfo.cgi/lustre-devel-lustre.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-devel_lists.lustre.org/attachments/20181121/4e2ae26c/attachment.html>

From paf at cray.com  Thu Nov 22 02:41:37 2018
From: paf at cray.com (Patrick Farrell)
Date: Thu, 22 Nov 2018 02:41:37 +0000
Subject: [lustre-devel] Compact layouts
In-Reply-To: <F5BAE559-E5F9-47C5-BB8C-8D2553F7BD43@gmail.com>
References: <3600968B-94D3-4C3F-857A-D00A5C3BC601@cray.com>
 <E2B38E8A-55A5-481D-AF44-FF0B4CD8472E@whamcloud.com>
 <MWHPR11MB1488BCF770411CE094423321CBDB0@MWHPR11MB1488.namprd11.prod.outlook.com>,
 <F5BAE559-E5F9-47C5-BB8C-8D2553F7BD43@gmail.com>
Message-ID: <MWHPR11MB148817404AF6C0B690EEB770CBDB0@MWHPR11MB1488.namprd11.prod.outlook.com>

It's an issue, certainly, but as an interim solution, a little bit of compression (which could be limited to layouts over a certain size) is a lot better than sending around large globs of data.  (Which in the case of layout are A) highly compressible [we suspect], and B) must be sent to every client.)

Also, while I'm a huge fan of Lustre, it is not really designed for the sort of hyper-low latency hardware (basically, persistent memory tech) you're describing.

- Patrick
________________________________
From: John Bent <johnbent at gmail.com>
Sent: Wednesday, November 21, 2018 8:30:49 PM
To: Patrick Farrell
Cc: Andreas Dilger; Lustre Developement
Subject: Re: [lustre-devel] Compact layouts

As HW latencies shrink to zero, does it not make you nervous to suggest adding compression into the metadata critical path?

On Nov 21, 2018, at 7:27 PM, Patrick Farrell <paf at cray.com<mailto:paf at cray.com>> wrote:


Andreas,


Thanks for the informative reply.


You raise an interesting and nasty point about breaking the compact layout with movement.  It's not possible today to move an individual OST object/stripe, though it's certainly something I've heard people ask for.  So it wouldn't be an issue if all such operations must address whole components, as is required today.


If we did add the ability to switch out an individual OST object/stripe (which would be pretty easy to implement - data copy, layout swap, rm now-unused object), we could add those modifications as additional "traditional" layout info "atop" the compact layout.  So just the usual layout format, with OST IDs, and where present, it supersedes the relevant part of the compact layout.  This implicitly assumes we don't do a ton of this to a particular layout.


But as to reasons, it's a few things.


The primary concern is improving the open performance of very widely striped files, which means your second case - reduce the xattr and rpc size.


The same things that motivate this would also motivate raising the count limit, but my understanding from comments in the code is that 2000 is arbitrary, and the actual max could be quite a bit higher.  The first limit I'm aware of - I'm not sure if this is right? - is 1 MiB of extended attribute.  That's a little over 5000 stripes.  (Obviously, 1 MiB of layout is probably a non-starter...)


Your suggestion of gzip is very intriguing.  Ideally, I'd pick something available in kernel and with good performance.  A bit of experimentation is probably in order if we go that route.  Thanks for the pointer there.  I'd probably start with extracting the binary xattr and seeing how it compresses.


- Patrick

________________________________
From: Andreas Dilger <adilger at whamcloud.com<mailto:adilger at whamcloud.com>>
Sent: Wednesday, November 21, 2018 5:53:03 PM
To: Patrick Farrell
Cc: Lustre Developement
Subject: Re: [lustre-devel] Compact layouts

On Nov 16, 2018, at 11:06, Patrick Farrell <paf at cray.com<mailto:paf at cray.com>> wrote:
>
> All,
>
> There is an old idea for reducing the data required to describe file striping by using a bitmap to record which OSTs are in use.  As best I can tell, this was most recently described here:
> http://wiki.lustre.org/Layout_Enhancement_Solution_Architecture#Compact_Layouts_2
>
> I’m curious if this has been pursued any further, if there’s a JIRA or other place that might have more info or be tracking the idea.  I poked around and didn’t find anything.
>
> In particular, this comment:
> “with enough data that for each OST index set in the bitmap, a corresponding OST object FID may be computed”
> Points at the difficult part of implementing this.
>
> So, before I get too far considering this problem - Is there more out there somewhere?  Hoping to avoid duplicating work!

Patrick,
as you mention above, the tricky part is that there would need to be sequential FID sequence allocation across all of the OSTs.  Then, each of the compact files would allocate/reserve the same OID in each of the sequences so that the mapping could be compact.  I don't think that is insurmountable - we already have a good mechanism for allocating FID sequences to different targets, but it would need to be extended so that compact layouts would allocate sequences from a different range of values from regular layouts.

It would also likely need to implement "OST object create on write" so that there aren't large numbers of unused objects on each OST (one for each OID that isn't used on a particular file).

The other issue is that anything like migrating any single object to another OST (e.g. for mirror resync, tiering, etc) would potentially break the compact layout.

I guess the question is what the need for compact layouts is?  To handle more than 2000 stripes, to reduce the xattr size/RPC size, to allow more complex PFL layouts to fit into the layout size limit?

In the past we discussed compressing the layout with gzip, which might be quite effective since large parts of it are zero-filled and repetitive.  This would help the xattr/RPC size, and I think even with compact layouts that they would still be expanded in RAM to allow easier processing.

Cheers, Andreas
---
Andreas Dilger
Principal Lustre Architect
Whamcloud







_______________________________________________
lustre-devel mailing list
lustre-devel at lists.lustre.org<mailto:lustre-devel at lists.lustre.org>
http://lists.lustre.org/listinfo.cgi/lustre-devel-lustre.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-devel_lists.lustre.org/attachments/20181122/bb683788/attachment.html>

From paf at cray.com  Thu Nov 22 02:53:48 2018
From: paf at cray.com (Patrick Farrell)
Date: Thu, 22 Nov 2018 02:53:48 +0000
Subject: [lustre-devel] Compact layouts
In-Reply-To: <MWHPR11MB148817404AF6C0B690EEB770CBDB0@MWHPR11MB1488.namprd11.prod.outlook.com>
References: <3600968B-94D3-4C3F-857A-D00A5C3BC601@cray.com>
 <E2B38E8A-55A5-481D-AF44-FF0B4CD8472E@whamcloud.com>
 <MWHPR11MB1488BCF770411CE094423321CBDB0@MWHPR11MB1488.namprd11.prod.outlook.com>,
 <F5BAE559-E5F9-47C5-BB8C-8D2553F7BD43@gmail.com>,
 <MWHPR11MB148817404AF6C0B690EEB770CBDB0@MWHPR11MB1488.namprd11.prod.outlook.com>
Message-ID: <MWHPR11MB1488F239E19293AED1B35882CBDB0@MWHPR11MB1488.namprd11.prod.outlook.com>

By the way, an update:

The 1 MiB xattr limit I mentioned is incorrect.  If you raise the arbitrary stripe count limit in the code, the limit does appear to be 65532 (which was documented as the theoretical max when wide striping was implemented).  However, my VM started getting soft lockups around 30,000 stripes, so I'm not 100% sure.  Nothing is exactly broken, but some areas of the code (understandably) do not scale well to 15x the current upper limit.  Especially not on a single node VM.


- Patrick

________________________________
From: lustre-devel <lustre-devel-bounces at lists.lustre.org> on behalf of Patrick Farrell <paf at cray.com>
Sent: Wednesday, November 21, 2018 8:41:37 PM
To: John Bent
Cc: Lustre Developement
Subject: Re: [lustre-devel] Compact layouts

It's an issue, certainly, but as an interim solution, a little bit of compression (which could be limited to layouts over a certain size) is a lot better than sending around large globs of data.  (Which in the case of layout are A) highly compressible [we suspect], and B) must be sent to every client.)

Also, while I'm a huge fan of Lustre, it is not really designed for the sort of hyper-low latency hardware (basically, persistent memory tech) you're describing.

- Patrick
________________________________
From: John Bent <johnbent at gmail.com>
Sent: Wednesday, November 21, 2018 8:30:49 PM
To: Patrick Farrell
Cc: Andreas Dilger; Lustre Developement
Subject: Re: [lustre-devel] Compact layouts

As HW latencies shrink to zero, does it not make you nervous to suggest adding compression into the metadata critical path?

On Nov 21, 2018, at 7:27 PM, Patrick Farrell <paf at cray.com<mailto:paf at cray.com>> wrote:


Andreas,


Thanks for the informative reply.


You raise an interesting and nasty point about breaking the compact layout with movement.  It's not possible today to move an individual OST object/stripe, though it's certainly something I've heard people ask for.  So it wouldn't be an issue if all such operations must address whole components, as is required today.


If we did add the ability to switch out an individual OST object/stripe (which would be pretty easy to implement - data copy, layout swap, rm now-unused object), we could add those modifications as additional "traditional" layout info "atop" the compact layout.  So just the usual layout format, with OST IDs, and where present, it supersedes the relevant part of the compact layout.  This implicitly assumes we don't do a ton of this to a particular layout.


But as to reasons, it's a few things.


The primary concern is improving the open performance of very widely striped files, which means your second case - reduce the xattr and rpc size.


The same things that motivate this would also motivate raising the count limit, but my understanding from comments in the code is that 2000 is arbitrary, and the actual max could be quite a bit higher.  The first limit I'm aware of - I'm not sure if this is right? - is 1 MiB of extended attribute.  That's a little over 5000 stripes.  (Obviously, 1 MiB of layout is probably a non-starter...)


Your suggestion of gzip is very intriguing.  Ideally, I'd pick something available in kernel and with good performance.  A bit of experimentation is probably in order if we go that route.  Thanks for the pointer there.  I'd probably start with extracting the binary xattr and seeing how it compresses.


- Patrick

________________________________
From: Andreas Dilger <adilger at whamcloud.com<mailto:adilger at whamcloud.com>>
Sent: Wednesday, November 21, 2018 5:53:03 PM
To: Patrick Farrell
Cc: Lustre Developement
Subject: Re: [lustre-devel] Compact layouts

On Nov 16, 2018, at 11:06, Patrick Farrell <paf at cray.com<mailto:paf at cray.com>> wrote:
>
> All,
>
> There is an old idea for reducing the data required to describe file striping by using a bitmap to record which OSTs are in use.  As best I can tell, this was most recently described here:
> http://wiki.lustre.org/Layout_Enhancement_Solution_Architecture#Compact_Layouts_2
>
> I’m curious if this has been pursued any further, if there’s a JIRA or other place that might have more info or be tracking the idea.  I poked around and didn’t find anything.
>
> In particular, this comment:
> “with enough data that for each OST index set in the bitmap, a corresponding OST object FID may be computed”
> Points at the difficult part of implementing this.
>
> So, before I get too far considering this problem - Is there more out there somewhere?  Hoping to avoid duplicating work!

Patrick,
as you mention above, the tricky part is that there would need to be sequential FID sequence allocation across all of the OSTs.  Then, each of the compact files would allocate/reserve the same OID in each of the sequences so that the mapping could be compact.  I don't think that is insurmountable - we already have a good mechanism for allocating FID sequences to different targets, but it would need to be extended so that compact layouts would allocate sequences from a different range of values from regular layouts.

It would also likely need to implement "OST object create on write" so that there aren't large numbers of unused objects on each OST (one for each OID that isn't used on a particular file).

The other issue is that anything like migrating any single object to another OST (e.g. for mirror resync, tiering, etc) would potentially break the compact layout.

I guess the question is what the need for compact layouts is?  To handle more than 2000 stripes, to reduce the xattr size/RPC size, to allow more complex PFL layouts to fit into the layout size limit?

In the past we discussed compressing the layout with gzip, which might be quite effective since large parts of it are zero-filled and repetitive.  This would help the xattr/RPC size, and I think even with compact layouts that they would still be expanded in RAM to allow easier processing.

Cheers, Andreas
---
Andreas Dilger
Principal Lustre Architect
Whamcloud







_______________________________________________
lustre-devel mailing list
lustre-devel at lists.lustre.org<mailto:lustre-devel at lists.lustre.org>
http://lists.lustre.org/listinfo.cgi/lustre-devel-lustre.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-devel_lists.lustre.org/attachments/20181122/bf331f61/attachment.html>

From adilger at whamcloud.com  Thu Nov 22 03:29:02 2018
From: adilger at whamcloud.com (Andreas Dilger)
Date: Thu, 22 Nov 2018 03:29:02 +0000
Subject: [lustre-devel] Compact layouts
In-Reply-To: <MWHPR11MB1488BCF770411CE094423321CBDB0@MWHPR11MB1488.namprd11.prod.outlook.com>
References: <3600968B-94D3-4C3F-857A-D00A5C3BC601@cray.com>
 <E2B38E8A-55A5-481D-AF44-FF0B4CD8472E@whamcloud.com>
 <MWHPR11MB1488BCF770411CE094423321CBDB0@MWHPR11MB1488.namprd11.prod.outlook.com>
Message-ID: <9E901853-E6EA-4CFF-825F-21275488E3FC@whamcloud.com>

On Nov 21, 2018, at 19:27, Patrick Farrell <paf at cray.com> wrote:
> 
> Andreas,
> 
> Thanks for the informative reply.
> 
> You raise an interesting and nasty point about breaking the compact layout with movement.  It's not possible today to move an individual OST object/stripe, though it's certainly something I've heard people ask for.  So it wouldn't be an issue if all such operations must address whole components, as is required today.

Yes, the ability to replace a single stripe of a file (e.g. in case of OST failure) is needed to make resync more efficient.  I agree that the current migration/resync implementation doesn't allow this, but it would be unfortunate to limit it as it is a feature that some users have already asked for.

> If we did add the ability to switch out an individual OST object/stripe (which would be pretty easy to implement - data copy, layout swap, rm now-unused object), we could add those modifications as additional "traditional" layout info "atop" the compact layout.  So just the usual layout format, with OST IDs, and where present, it supersedes the relevant part of the compact layout.  This implicitly assumes we don't do a ton of this to a particular layout.

Sure.

> But as to reasons, it's a few things.
> 
> The primary concern is improving the open performance of very widely striped files, which means your second case - reduce the xattr and rpc size.

I was thinking that a better approach would be to implement open-by-handle, so that e.g. MPI rank 0 can open the file by path, generate a short-lived file handle to send to the other ranks, and they open by handle (the kernel interface is limited to root-only, but we could add our own user interface if we had secure handles that were resistent to guessing).  However, I don't think  that helps for your case because the other clients still need to have a local copy of the layout in order to do IO properly.  Passing in the file layout from  userspace doesn't reduce the overall network traffic, and opens the window to severe data integrity issues.

> The same things that motivate this would also motivate raising the count limit, but my understanding from comments in the code is that 2000 is arbitrary, and the actual max could be quite a bit higher.  The first limit I'm aware of - I'm not sure if this is right? - is 1 MiB of extended attribute.  That's a little over 5000 stripes.  (Obviously, 1 MiB of layout is probably a non-starter...)

For a 1MB layout, this would be over 43k stripes (not sure where you got 5000 stripes from, as we have 2000 stripes in about 48KB).  The basic layout format has each stripe using 24 bytes to hold struct lov_ost_data_v1.  We could make that more compact these days by only storing the 16-byte object FID, since that also contains the OST index (directly in an IDIF FID, or indirectly via FLDB for a normal FID).  That would get us up to 64k stripes, and since we are adding a new layout type we should strongly consider this.

Note that the 1MB layout size limit on disk is also arbitrary.  There would be a larger issue with the kernel xattr interface, since that is currently limited to 64KB in size (about 4k stripes if we stored only the 16-byte FID).

The bigger problem than the xattr disk size is the network transfer and in-kernel memory needed to handle such a large layout.  We would likely need to go to a separate RDMA transfer for the layout to avoid bloating the reply buffer for the common (small layout) case. 

> Your suggestion of gzip is very intriguing.  Ideally, I'd pick something available in kernel and with good performance.  A bit of experimentation is probably in order if we go that route.  Thanks for the pointer there.  I'd probably start with extracting the binary xattr and seeing how it compresses.

I believe gzip is available for both compression and decompression in the kernel, and even has hardware acceleration (QAT) available from Intel.  It has the best compression ratio, though the performance is lower.  I think we'd prefer a higher compression ratio since this isn't a huge volume of data we are working with.

Cheers, Andreas

> From: Andreas Dilger <adilger at whamcloud.com>
> Sent: Wednesday, November 21, 2018 5:53:03 PM
> To: Patrick Farrell
> Cc: Lustre Developement
> Subject: Re: [lustre-devel] Compact layouts
>  
> On Nov 16, 2018, at 11:06, Patrick Farrell <paf at cray.com> wrote:
> > 
> > All,
> >  
> > There is an old idea for reducing the data required to describe file striping by using a bitmap to record which OSTs are in use.  As best I can tell, this was most recently described here:
> > http://wiki.lustre.org/Layout_Enhancement_Solution_Architecture#Compact_Layouts_2
> >  
> > I’m curious if this has been pursued any further, if there’s a JIRA or other place that might have more info or be tracking the idea.  I poked around and didn’t find anything.
> >  
> > In particular, this comment:
> > “with enough data that for each OST index set in the bitmap, a corresponding OST object FID may be computed”
> > Points at the difficult part of implementing this.
> >  
> > So, before I get too far considering this problem - Is there more out there somewhere?  Hoping to avoid duplicating work!
> 
> Patrick,
> as you mention above, the tricky part is that there would need to be sequential FID sequence allocation across all of the OSTs.  Then, each of the compact files would allocate/reserve the same OID in each of the sequences so that the mapping could be compact.  I don't think that is insurmountable - we already have a good mechanism for allocating FID sequences to different targets, but it would need to be extended so that compact layouts would allocate sequences from a different range of values from regular layouts.
> 
> It would also likely need to implement "OST object create on write" so that there aren't large numbers of unused objects on each OST (one for each OID that isn't used on a particular file).
> 
> The other issue is that anything like migrating any single object to another OST (e.g. for mirror resync, tiering, etc) would potentially break the compact layout.
> 
> I guess the question is what the need for compact layouts is?  To handle more than 2000 stripes, to reduce the xattr size/RPC size, to allow more complex PFL layouts to fit into the layout size limit?
> 
> In the past we discussed compressing the layout with gzip, which might be quite effective since large parts of it are zero-filled and repetitive.  This would help the xattr/RPC size, and I think even with compact layouts that they would still be expanded in RAM to allow easier processing.
> 
> Cheers, Andreas
> ---
> Andreas Dilger
> Principal Lustre Architect
> Whamcloud

Cheers, Andreas
---
Andreas Dilger
CTO Whamcloud





From mail at gmelikov.ru  Thu Nov 22 06:53:22 2018
From: mail at gmelikov.ru (George Melikov)
Date: Thu, 22 Nov 2018 09:53:22 +0300
Subject: [lustre-devel] Compact layouts
In-Reply-To: <9E901853-E6EA-4CFF-825F-21275488E3FC@whamcloud.com>
References: <3600968B-94D3-4C3F-857A-D00A5C3BC601@cray.com>
 <E2B38E8A-55A5-481D-AF44-FF0B4CD8472E@whamcloud.com>
 <MWHPR11MB1488BCF770411CE094423321CBDB0@MWHPR11MB1488.namprd11.prod.outlook.com>
 <9E901853-E6EA-4CFF-825F-21275488E3FC@whamcloud.com>
Message-ID: <5929481542869602@myt3-964eae3a5b05.qloud-c.yandex.net>



22.11.2018, 06:29, "Andreas Dilger" <adilger at whamcloud.com>:
> On Nov 21, 2018, at 19:27, Patrick Farrell <paf at cray.com> wrote:
>>  Andreas,
>>
>>  Thanks for the informative reply.
>>
>>  You raise an interesting and nasty point about breaking the compact layout with movement. It's not possible today to move an individual OST object/stripe, though it's certainly something I've heard people ask for. So it wouldn't be an issue if all such operations must address whole components, as is required today.
>
> Yes, the ability to replace a single stripe of a file (e.g. in case of OST failure) is needed to make resync more efficient. I agree that the current migration/resync implementation doesn't allow this, but it would be unfortunate to limit it as it is a feature that some users have already asked for.
>
>>  If we did add the ability to switch out an individual OST object/stripe (which would be pretty easy to implement - data copy, layout swap, rm now-unused object), we could add those modifications as additional "traditional" layout info "atop" the compact layout. So just the usual layout format, with OST IDs, and where present, it supersedes the relevant part of the compact layout. This implicitly assumes we don't do a ton of this to a particular layout.
>
> Sure.
>
>>  But as to reasons, it's a few things.
>>
>>  The primary concern is improving the open performance of very widely striped files, which means your second case - reduce the xattr and rpc size.
>
> I was thinking that a better approach would be to implement open-by-handle, so that e.g. MPI rank 0 can open the file by path, generate a short-lived file handle to send to the other ranks, and they open by handle (the kernel interface is limited to root-only, but we could add our own user interface if we had secure handles that were resistent to guessing). However, I don't think that helps for your case because the other clients still need to have a local copy of the layout in order to do IO properly. Passing in the file layout from userspace doesn't reduce the overall network traffic, and opens the window to severe data integrity issues.
>
>>  The same things that motivate this would also motivate raising the count limit, but my understanding from comments in the code is that 2000 is arbitrary, and the actual max could be quite a bit higher. The first limit I'm aware of - I'm not sure if this is right? - is 1 MiB of extended attribute. That's a little over 5000 stripes. (Obviously, 1 MiB of layout is probably a non-starter...)
>
> For a 1MB layout, this would be over 43k stripes (not sure where you got 5000 stripes from, as we have 2000 stripes in about 48KB). The basic layout format has each stripe using 24 bytes to hold struct lov_ost_data_v1. We could make that more compact these days by only storing the 16-byte object FID, since that also contains the OST index (directly in an IDIF FID, or indirectly via FLDB for a normal FID). That would get us up to 64k stripes, and since we are adding a new layout type we should strongly consider this.
>
> Note that the 1MB layout size limit on disk is also arbitrary. There would be a larger issue with the kernel xattr interface, since that is currently limited to 64KB in size (about 4k stripes if we stored only the 16-byte FID).
>
> The bigger problem than the xattr disk size is the network transfer and in-kernel memory needed to handle such a large layout. We would likely need to go to a separate RDMA transfer for the layout to avoid bloating the reply buffer for the common (small layout) case.
>
>>  Your suggestion of gzip is very intriguing. Ideally, I'd pick something available in kernel and with good performance. A bit of experimentation is probably in order if we go that route. Thanks for the pointer there. I'd probably start with extracting the binary xattr and seeing how it compresses.
>
> I believe gzip is available for both compression and decompression in the kernel, and even has hardware acceleration (QAT) available from Intel. It has the best compression ratio, though the performance is lower. I think we'd prefer a higher compression ratio since this isn't a huge volume of data we are working with.

ZSTD with dictionary may be very interesting specially for this case https://facebook.github.io/zstd/#small-data

It's better than gzip in compression and speed, and it's included in kernel since 4.14 (but you may want not to use it, see discussion in ZFSonLinux ZSTD integration https://github.com/zfsonlinux/zfs/pull/8044 )

>
> Cheers, Andreas
>
>>  From: Andreas Dilger <adilger at whamcloud.com>
>>  Sent: Wednesday, November 21, 2018 5:53:03 PM
>>  To: Patrick Farrell
>>  Cc: Lustre Developement
>>  Subject: Re: [lustre-devel] Compact layouts
>>
>>  On Nov 16, 2018, at 11:06, Patrick Farrell <paf at cray.com> wrote:
>>  >
>>  > All,
>>  >
>>  > There is an old idea for reducing the data required to describe file striping by using a bitmap to record which OSTs are in use. As best I can tell, this was most recently described here:
>>  > http://wiki.lustre.org/Layout_Enhancement_Solution_Architecture#Compact_Layouts_2
>>  >
>>  > I’m curious if this has been pursued any further, if there’s a JIRA or other place that might have more info or be tracking the idea. I poked around and didn’t find anything.
>>  >
>>  > In particular, this comment:
>>  > “with enough data that for each OST index set in the bitmap, a corresponding OST object FID may be computed”
>>  > Points at the difficult part of implementing this.
>>  >
>>  > So, before I get too far considering this problem - Is there more out there somewhere? Hoping to avoid duplicating work!
>>
>>  Patrick,
>>  as you mention above, the tricky part is that there would need to be sequential FID sequence allocation across all of the OSTs. Then, each of the compact files would allocate/reserve the same OID in each of the sequences so that the mapping could be compact. I don't think that is insurmountable - we already have a good mechanism for allocating FID sequences to different targets, but it would need to be extended so that compact layouts would allocate sequences from a different range of values from regular layouts.
>>
>>  It would also likely need to implement "OST object create on write" so that there aren't large numbers of unused objects on each OST (one for each OID that isn't used on a particular file).
>>
>>  The other issue is that anything like migrating any single object to another OST (e.g. for mirror resync, tiering, etc) would potentially break the compact layout.
>>
>>  I guess the question is what the need for compact layouts is? To handle more than 2000 stripes, to reduce the xattr size/RPC size, to allow more complex PFL layouts to fit into the layout size limit?
>>
>>  In the past we discussed compressing the layout with gzip, which might be quite effective since large parts of it are zero-filled and repetitive. This would help the xattr/RPC size, and I think even with compact layouts that they would still be expanded in RAM to allow easier processing.
>>
>>  Cheers, Andreas
>>  ---
>>  Andreas Dilger
>>  Principal Lustre Architect
>>  Whamcloud
>
> Cheers, Andreas
> ---
> Andreas Dilger
> CTO Whamcloud
>
> _______________________________________________
> lustre-devel mailing list
> lustre-devel at lists.lustre.org
> http://lists.lustre.org/listinfo.cgi/lustre-devel-lustre.org

____________________________________
Sincerely,
George Melikov


From neilb at suse.com  Fri Nov 23 04:44:58 2018
From: neilb at suse.com (NeilBrown)
Date: Fri, 23 Nov 2018 15:44:58 +1100
Subject: [lustre-devel] [RPC] parallel directory operations for mainline
 Linux
Message-ID: <8736rsbdx1.fsf@notabene.neil.brown.name>


One of the remaining features of ldiskfs which is not in ext4fs is
parallel directory ops.
It would not be possible to get this upstream without VFS support
for parallel directory ops.  Lustre doesn't use the VFS interfaces
so this lack is not an immediate problem for lustre, but it is a
real problem for upstreaming.

This patch (which seems to work in my testing so far, but is probably
still buggy) adds VFS support for parallel dir ops - create and remove.
I haven't attempted rename - it would be complex for various reasons and
while I'm sure it is possible, I'm not sure it is worth the effort.

With this patch a filesystem can indicate that it supports parallel ops
by setting a flag on a directory.  The VFS will then get exclusive
access to the dentry - instead of the whole directory - when
performing the operation.

A filesystem which supports this much have its own locking to ensure
that lookup, readdir, create, unlink can all happen in parallel.
For NFS this is easy as the server takes care of those details, so
this patch also adds parallel-ops support for NFS.
For a filesystem like ext4 it would mean adding some locking to
the internal data structures.

I've had a bit of a look at the parallel-ops patch for ldiskfs and I
think it is over-engineered.  We don't need a new locking primitive.

I suspect I would start by adding a seqlock to each htree node.
This allows reads to proceed locklessly when no changes are happening
(if they are careful not to get confused by an inconsistent node).
A modification would normally find the relevant leaf with a similar
lockless walk, then lock the leaf, verify the seq-lock on the parent
hasn't changed, and perform the update.
In the rarer case when a leaf needs to split or merge something more
heavy handed would be needed - possibly lock the whole tree - possibly
just lock a higher node.

I don't expect to look at ext4 parallel ops in more detail in the
immediate future, and I don't plan to post this upstream until we have
credible support in ext4.  So I'm just posting it here now in case
anyone else want to explore how to make ext4 work with this.

NeilBrown


From 827c01aee1cb74b72e5dbb2f40c01666b914bc15 Mon Sep 17 00:00:00 2001
From: NeilBrown <neilb at suse.com>
Date: Fri, 16 Nov 2018 19:58:53 +1100
Subject: [PATCH] VFS: support parallel updates in the one directory.

Some filesystems can support parallel modifications to a directory,
either because the modification happen on a remote server which does
its own locking (e.g. NFS) or because they can internally lock just
a part of a directory (e.g. many local filesystems, with a bit of
work).

To support these, we introduce support for parallel modification:
unlink (including rmdir) and create.

If a filesystem supports parallel modification in a given directory,
it sets S_PAR_UNLINK on the inode for that directory.
lookup_open() and the new lookup_hash_modify() (similar to
__lookup_hash()) notice the flag and take a shared
lock on the directory.

Once a dentry for the target name has been obtained,
DCACHE_PAR_UPDATE is set on it, waiting if necessary.
Once this is set, the thread has exclusive access to the
name and can call into the filesystem to perform
the required action.

Some files do *not* complete the lookup that precedes
a create, but leave the dentry d_in_lookup() and unhashed,
so often a dentry will have both DCACHE_PAR_LOOKUP and
DCACHE_PAR_UPDATE set at the same time.  To allow
for this, we need the 'wq' that is used when DCACHE_PAR_LOOKUP is
cleared, to exist until the creation is complete.  We also
need to re-initialize it if it might get re-used.

As NFS trivially supports parallel unlinks, this patch also adds the
flag to all NFS directories.

Signed-off-by: NeilBrown <neilb at suse.com>
---
 fs/dcache.c            |  37 ++++++++++
 fs/namei.c             | 189 ++++++++++++++++++++++++++++++++++++++++++-------
 fs/nfs/dir.c           |   2 +-
 fs/nfs/inode.c         |   2 +
 fs/nfs/unlink.c        |   4 +-
 include/linux/dcache.h |  43 +++++++++++
 include/linux/fs.h     |   1 +
 7 files changed, 249 insertions(+), 29 deletions(-)

diff --git a/fs/dcache.c b/fs/dcache.c
index 2593153471cf..3821ce0bc37f 100644
--- a/fs/dcache.c
+++ b/fs/dcache.c
@@ -3030,6 +3030,43 @@ void d_tmpfile(struct dentry *dentry, struct inode *inode)
 }
 EXPORT_SYMBOL(d_tmpfile);
 
+/*
+ * Lock a dentry to unlink or create in an S_PAR_UPDATE directory.
+ * After a successful return the dentry will not be modified by any other
+ * thread, and still has the given name and parent.
+ * On unsuccessful return it is not locked, because it was either unlinked
+ * or renamed in the mean-time.  If it was instantiated or created,
+ * we still return success, so caller might need to test if the dentry
+ * is negative or positive.
+ */
+bool d_lock_modify(struct dentry *dentry,
+		   struct dentry *base, const struct qstr *name)
+{
+	bool ret = true;
+
+	spin_lock(&dentry->d_lock);
+	if (dentry->d_flags & DCACHE_PAR_UPDATE)
+		___wait_var_event(&dentry->d_flags,
+				  !(dentry->d_flags & DCACHE_PAR_UPDATE),
+				  TASK_UNINTERRUPTIBLE, 0, 0,
+				  (spin_unlock(&dentry->d_lock),
+				   schedule(),
+				   spin_lock(&dentry->d_lock))
+			);
+	if (d_unhashed(dentry) && d_is_positive(dentry))
+		/* name was unlinked while we waited */
+		ret = false;
+	else if (dentry->d_parent != base ||
+		 dentry->d_name.hash != name->hash ||
+		 !d_same_name(dentry, base, name))
+		/* dentry was renamed - possibly silly-rename */
+		ret = false;
+	else
+		dentry->d_flags |= DCACHE_PAR_UPDATE;
+	spin_unlock(&dentry->d_lock);
+	return ret;
+}
+
 static __initdata unsigned long dhash_entries;
 static int __init set_dhash_entries(char *str)
 {
diff --git a/fs/namei.c b/fs/namei.c
index 0cab6494978c..ab6ccc03b9f4 100644
--- a/fs/namei.c
+++ b/fs/namei.c
@@ -1551,6 +1551,91 @@ static struct dentry *__lookup_hash(const struct qstr *name,
 	return dentry;
 }
 
+/*
+ * Parent directory is not locked.  We take either an exclusive
+ * or shared lock depending on the fs preference, and then
+ * get the DCACHE_PAR_UPDATE bit.
+ */
+static struct dentry *lookup_hash_modify(const struct qstr *name,
+					 struct dentry *base, unsigned int flags,
+					 wait_queue_head_t *wq)
+{
+	struct dentry *dentry;
+	struct inode *dir = base->d_inode;
+	bool shared = (dir->i_flags & S_PAR_UPDATE) && wq;
+	int err;
+
+	if (shared)
+		inode_lock_shared_nested(dir, I_MUTEX_PARENT);
+	else
+		inode_lock_nested(dir, I_MUTEX_PARENT);
+
+retry:
+	dentry = lookup_dcache(name, base, flags);
+	if (!dentry) {
+		/* Don't create child dentry for a dead directory. */
+		err = -ENOENT;
+		if (unlikely(IS_DEADDIR(dir)))
+			goto out_err;
+
+		if (shared)
+			dentry = d_alloc_parallel(base, name, wq);
+		else
+			dentry = d_alloc(base, name);
+
+		if (!IS_ERR(dentry) &&
+		    (!shared || d_in_lookup(dentry))) {
+			struct dentry *old;
+
+			old = dir->i_op->lookup(dir, dentry, flags);
+			/*
+			 * Note: dentry might still be d_unhashed() and
+			 * d_in_lookup() if the fs will do the lookup
+			 * at 'create' time.
+			 */
+			if (unlikely(old)) {
+				d_lookup_done(dentry);
+				dput(dentry);
+				dentry = old;
+			}
+		}
+	}
+	if (IS_ERR(dentry)) {
+		err = PTR_ERR(dentry);
+		goto out_err;
+	}
+	if (!shared || d_lock_modify(dentry, base, name))
+		return dentry;
+
+	/* Failed to get lock due to race with unlink or rename */
+	d_lookup_done(dentry);
+	init_waitqueue_head(wq);
+	dput(dentry);
+	goto retry;
+
+out_err:
+	if (shared)
+		inode_unlock_shared(dir);
+	else
+		inode_unlock(dir);
+	return ERR_PTR(err);
+}
+
+static void lookup_done_modify(struct path *path, struct dentry *dentry,
+			       wait_queue_head_t *wq)
+{
+	struct inode *dir = path->dentry->d_inode;
+	bool shared = (dir->i_flags & S_PAR_UPDATE) && wq;
+
+	if (shared) {
+		d_lookup_done(dentry);
+		d_unlock_modify(dentry);
+		inode_unlock_shared(dir);
+	} else {
+		inode_unlock(dir);
+	}
+}
+
 static int lookup_fast(struct nameidata *nd,
 		       struct path *path, struct inode **inode,
 		       unsigned *seqp)
@@ -3136,6 +3221,7 @@ static int lookup_open(struct nameidata *nd, struct path *path,
 	int error, create_error = 0;
 	umode_t mode = op->mode;
 	DECLARE_WAIT_QUEUE_HEAD_ONSTACK(wq);
+	bool have_par_update = false;
 
 	if (unlikely(IS_DEADDIR(dir_inode)))
 		return -ENOENT;
@@ -3201,10 +3287,22 @@ static int lookup_open(struct nameidata *nd, struct path *path,
 	}
 
 	if (dir_inode->i_op->atomic_open) {
+		/* dentry is negative or d_in_lookup().  If this is a shared-lock
+		 * create we need to get DCACHE_PAR_UPDATE to ensure exclusion
+		 */
+		if ((open_flag & O_CREAT) &&
+		    (dir->d_inode->i_flags & S_PAR_UPDATE)) {
+			if (!d_lock_create(dentry))
+				/* already exists, non-atomic open */
+				goto out_no_open;
+			have_par_update = true;
+		}
 		error = atomic_open(nd, dentry, path, file, op, open_flag,
 				    mode);
 		if (unlikely(error == -ENOENT) && create_error)
 			error = create_error;
+		if (have_par_update)
+			d_unlock_modify(dentry);
 		return error;
 	}
 
@@ -3222,6 +3320,13 @@ static int lookup_open(struct nameidata *nd, struct path *path,
 			dentry = res;
 		}
 	}
+	/* If dentry is negative and this is a shared-lock
+	 * create we need to get DCACHE_PAR_UPDATE to ensure exclusion
+	 */
+	if ((open_flag & O_CREAT) &&
+	    !dentry->d_inode &&
+	    (dir->d_inode->i_flags & S_PAR_UPDATE))
+		have_par_update = d_lock_create(dentry);
 
 	/* Negative dentry, just create the file */
 	if (!dentry->d_inode && (open_flag & O_CREAT)) {
@@ -3242,11 +3347,15 @@ static int lookup_open(struct nameidata *nd, struct path *path,
 		goto out_dput;
 	}
 out_no_open:
+	if (have_par_update)
+		d_unlock_modify(dentry);
 	path->dentry = dentry;
 	path->mnt = nd->path.mnt;
 	return 0;
 
 out_dput:
+	if (have_par_update)
+		d_unlock_modify(dentry);
 	dput(dentry);
 	return error;
 }
@@ -3266,6 +3375,7 @@ static int do_last(struct nameidata *nd,
 	struct inode *inode;
 	struct path path;
 	int error;
+	bool shared;
 
 	nd->flags &= ~LOOKUP_PARENT;
 	nd->flags |= op->intent;
@@ -3317,12 +3427,13 @@ static int do_last(struct nameidata *nd,
 		 * dropping this one anyway.
 		 */
 	}
-	if (open_flag & O_CREAT)
+	shared = !!(dir->d_inode->i_flags & S_PAR_UPDATE);
+	if ((open_flag & O_CREAT) && !shared)
 		inode_lock(dir->d_inode);
 	else
 		inode_lock_shared(dir->d_inode);
 	error = lookup_open(nd, &path, file, op, got_write);
-	if (open_flag & O_CREAT)
+	if ((open_flag & O_CREAT) && !shared)
 		inode_unlock(dir->d_inode);
 	else
 		inode_unlock_shared(dir->d_inode);
@@ -3600,7 +3711,8 @@ struct file *do_file_open_root(struct dentry *dentry, struct vfsmount *mnt,
 }
 
 static struct dentry *filename_create(int dfd, struct filename *name,
-				struct path *path, unsigned int lookup_flags)
+				      struct path *path, unsigned int lookup_flags,
+				      wait_queue_head_t *wq)
 {
 	struct dentry *dentry = ERR_PTR(-EEXIST);
 	struct qstr last;
@@ -3632,8 +3744,7 @@ static struct dentry *filename_create(int dfd, struct filename *name,
 	 * Do the final lookup.
 	 */
 	lookup_flags |= LOOKUP_CREATE | LOOKUP_EXCL;
-	inode_lock_nested(path->dentry->d_inode, I_MUTEX_PARENT);
-	dentry = __lookup_hash(&last, path->dentry, lookup_flags);
+	dentry = lookup_hash_modify(&last, path->dentry, lookup_flags, wq);
 	if (IS_ERR(dentry))
 		goto unlock;
 
@@ -3658,10 +3769,10 @@ static struct dentry *filename_create(int dfd, struct filename *name,
 	putname(name);
 	return dentry;
 fail:
+	lookup_done_modify(path, dentry, wq);
 	dput(dentry);
 	dentry = ERR_PTR(error);
 unlock:
-	inode_unlock(path->dentry->d_inode);
 	if (!err2)
 		mnt_drop_write(path->mnt);
 out:
@@ -3674,23 +3785,34 @@ struct dentry *kern_path_create(int dfd, const char *pathname,
 				struct path *path, unsigned int lookup_flags)
 {
 	return filename_create(dfd, getname_kernel(pathname),
-				path, lookup_flags);
+			       path, lookup_flags, NULL);
 }
 EXPORT_SYMBOL(kern_path_create);
 
-void done_path_create(struct path *path, struct dentry *dentry)
+void __done_path_create(struct path *path, struct dentry *dentry,
+			wait_queue_head_t *wq)
 {
+	lookup_done_modify(path, dentry, wq);
 	dput(dentry);
-	inode_unlock(path->dentry->d_inode);
 	mnt_drop_write(path->mnt);
 	path_put(path);
 }
+void done_path_create(struct path *path, struct dentry *dentry)
+{
+	__done_path_create(path, dentry, NULL);
+}
 EXPORT_SYMBOL(done_path_create);
 
+inline struct dentry *user_path_create_wq(int dfd, const char __user *pathname,
+					  struct path *path, unsigned int lookup_flags,
+					  wait_queue_head_t *wq)
+{
+	return filename_create(dfd, getname(pathname), path, lookup_flags, wq);
+}
 inline struct dentry *user_path_create(int dfd, const char __user *pathname,
-				struct path *path, unsigned int lookup_flags)
+				       struct path *path, unsigned int lookup_flags)
 {
-	return filename_create(dfd, getname(pathname), path, lookup_flags);
+	return filename_create(dfd, getname(pathname), path, lookup_flags, NULL);
 }
 EXPORT_SYMBOL(user_path_create);
 
@@ -3747,12 +3869,13 @@ long do_mknodat(int dfd, const char __user *filename, umode_t mode,
 	struct path path;
 	int error;
 	unsigned int lookup_flags = 0;
+	DECLARE_WAIT_QUEUE_HEAD_ONSTACK(wq);
 
 	error = may_mknod(mode);
 	if (error)
 		return error;
 retry:
-	dentry = user_path_create(dfd, filename, &path, lookup_flags);
+	dentry = user_path_create_wq(dfd, filename, &path, lookup_flags, &wq);
 	if (IS_ERR(dentry))
 		return PTR_ERR(dentry);
 
@@ -3776,7 +3899,7 @@ long do_mknodat(int dfd, const char __user *filename, umode_t mode,
 			break;
 	}
 out:
-	done_path_create(&path, dentry);
+	__done_path_create(&path, dentry, &wq);
 	if (retry_estale(error, lookup_flags)) {
 		lookup_flags |= LOOKUP_REVAL;
 		goto retry;
@@ -3827,9 +3950,10 @@ long do_mkdirat(int dfd, const char __user *pathname, umode_t mode)
 	struct path path;
 	int error;
 	unsigned int lookup_flags = LOOKUP_DIRECTORY;
+	DECLARE_WAIT_QUEUE_HEAD_ONSTACK(wq);
 
 retry:
-	dentry = user_path_create(dfd, pathname, &path, lookup_flags);
+	dentry = user_path_create_wq(dfd, pathname, &path, lookup_flags, &wq);
 	if (IS_ERR(dentry))
 		return PTR_ERR(dentry);
 
@@ -3838,7 +3962,7 @@ long do_mkdirat(int dfd, const char __user *pathname, umode_t mode)
 	error = security_path_mkdir(&path, dentry, mode);
 	if (!error)
 		error = vfs_mkdir(path.dentry->d_inode, dentry, mode);
-	done_path_create(&path, dentry);
+	__done_path_create(&path, dentry, &wq);
 	if (retry_estale(error, lookup_flags)) {
 		lookup_flags |= LOOKUP_REVAL;
 		goto retry;
@@ -3904,6 +4028,7 @@ long do_rmdir(int dfd, const char __user *pathname)
 	struct qstr last;
 	int type;
 	unsigned int lookup_flags = 0;
+	DECLARE_WAIT_QUEUE_HEAD_ONSTACK(wq);
 retry:
 	name = filename_parentat(dfd, getname(pathname), lookup_flags,
 				&path, &last, &type);
@@ -3926,8 +4051,7 @@ long do_rmdir(int dfd, const char __user *pathname)
 	if (error)
 		goto exit1;
 
-	inode_lock_nested(path.dentry->d_inode, I_MUTEX_PARENT);
-	dentry = __lookup_hash(&last, path.dentry, lookup_flags);
+	dentry = lookup_hash_modify(&last, path.dentry, lookup_flags, &wq);
 	error = PTR_ERR(dentry);
 	if (IS_ERR(dentry))
 		goto exit2;
@@ -3940,9 +4064,9 @@ long do_rmdir(int dfd, const char __user *pathname)
 		goto exit3;
 	error = vfs_rmdir(path.dentry->d_inode, dentry);
 exit3:
+	lookup_done_modify(&path, dentry, &wq);
 	dput(dentry);
 exit2:
-	inode_unlock(path.dentry->d_inode);
 	mnt_drop_write(path.mnt);
 exit1:
 	path_put(&path);
@@ -3965,7 +4089,8 @@ SYSCALL_DEFINE1(rmdir, const char __user *, pathname)
  * @dentry:	victim
  * @delegated_inode: returns victim inode, if the inode is delegated.
  *
- * The caller must hold dir->i_mutex.
+ * The caller must either hold a write-lock on dir->i_rwsem, or
+ * a readlock having atomically cleared DCACHE_PAR_UNLINK.
  *
  * If vfs_unlink discovers a delegation, it will return -EWOULDBLOCK and
  * return a reference to the inode in delegated_inode.  The caller
@@ -4022,6 +4147,11 @@ EXPORT_SYMBOL(vfs_unlink);
  * directory's i_mutex.  Truncate can take a long time if there is a lot of
  * writeout happening, and we don't want to prevent access to the directory
  * while waiting on the I/O.
+ * If both the directory and the target dentry have DCACHE_PAR_UNLINK set,
+ * we do the unlink with a shared lock on the directory while clearing
+ * DCACHE_PAR_UNLINK on the target.  IF the flags is not set, then either
+ * the filesystem doesn't support parallel unlinks, or we are racing with
+ * another thread unlinking the same name.
  */
 long do_unlinkat(int dfd, struct filename *name)
 {
@@ -4033,6 +4163,7 @@ long do_unlinkat(int dfd, struct filename *name)
 	struct inode *inode = NULL;
 	struct inode *delegated_inode = NULL;
 	unsigned int lookup_flags = 0;
+	DECLARE_WAIT_QUEUE_HEAD_ONSTACK(wq);
 retry:
 	name = filename_parentat(dfd, name, lookup_flags, &path, &last, &type);
 	if (IS_ERR(name))
@@ -4045,9 +4176,9 @@ long do_unlinkat(int dfd, struct filename *name)
 	error = mnt_want_write(path.mnt);
 	if (error)
 		goto exit1;
+
 retry_deleg:
-	inode_lock_nested(path.dentry->d_inode, I_MUTEX_PARENT);
-	dentry = __lookup_hash(&last, path.dentry, lookup_flags);
+	dentry = lookup_hash_modify(&last, path.dentry, lookup_flags, &wq);
 	error = PTR_ERR(dentry);
 	if (!IS_ERR(dentry)) {
 		/* Why not before? Because we want correct error value */
@@ -4062,14 +4193,15 @@ long do_unlinkat(int dfd, struct filename *name)
 			goto exit2;
 		error = vfs_unlink(path.dentry->d_inode, dentry, &delegated_inode);
 exit2:
+		lookup_done_modify(&path, dentry, &wq);
 		dput(dentry);
 	}
-	inode_unlock(path.dentry->d_inode);
 	if (inode)
 		iput(inode);	/* truncate the inode here */
 	inode = NULL;
 	if (delegated_inode) {
 		error = break_deleg_wait(&delegated_inode);
+		init_waitqueue_head(&wq);
 		if (!error)
 			goto retry_deleg;
 	}
@@ -4079,6 +4211,7 @@ long do_unlinkat(int dfd, struct filename *name)
 	if (retry_estale(error, lookup_flags)) {
 		lookup_flags |= LOOKUP_REVAL;
 		inode = NULL;
+		init_waitqueue_head(&wq);
 		goto retry;
 	}
 	putname(name);
@@ -4139,12 +4272,13 @@ long do_symlinkat(const char __user *oldname, int newdfd,
 	struct dentry *dentry;
 	struct path path;
 	unsigned int lookup_flags = 0;
+	DECLARE_WAIT_QUEUE_HEAD_ONSTACK(wq);
 
 	from = getname(oldname);
 	if (IS_ERR(from))
 		return PTR_ERR(from);
 retry:
-	dentry = user_path_create(newdfd, newname, &path, lookup_flags);
+	dentry = user_path_create_wq(newdfd, newname, &path, lookup_flags, &wq);
 	error = PTR_ERR(dentry);
 	if (IS_ERR(dentry))
 		goto out_putname;
@@ -4152,7 +4286,7 @@ long do_symlinkat(const char __user *oldname, int newdfd,
 	error = security_path_symlink(&path, dentry, from->name);
 	if (!error)
 		error = vfs_symlink(path.dentry->d_inode, dentry, from->name);
-	done_path_create(&path, dentry);
+	__done_path_create(&path, dentry, &wq);
 	if (retry_estale(error, lookup_flags)) {
 		lookup_flags |= LOOKUP_REVAL;
 		goto retry;
@@ -4270,6 +4404,7 @@ int do_linkat(int olddfd, const char __user *oldname, int newdfd,
 	struct inode *delegated_inode = NULL;
 	int how = 0;
 	int error;
+	DECLARE_WAIT_QUEUE_HEAD_ONSTACK(wq);
 
 	if ((flags & ~(AT_SYMLINK_FOLLOW | AT_EMPTY_PATH)) != 0)
 		return -EINVAL;
@@ -4291,8 +4426,8 @@ int do_linkat(int olddfd, const char __user *oldname, int newdfd,
 	if (error)
 		return error;
 
-	new_dentry = user_path_create(newdfd, newname, &new_path,
-					(how & LOOKUP_REVAL));
+	new_dentry = user_path_create_wq(newdfd, newname, &new_path,
+				      (how & LOOKUP_REVAL), &wq);
 	error = PTR_ERR(new_dentry);
 	if (IS_ERR(new_dentry))
 		goto out;
@@ -4308,7 +4443,7 @@ int do_linkat(int olddfd, const char __user *oldname, int newdfd,
 		goto out_dput;
 	error = vfs_link(old_path.dentry, new_path.dentry->d_inode, new_dentry, &delegated_inode);
 out_dput:
-	done_path_create(&new_path, new_dentry);
+	__done_path_create(&new_path, new_dentry, &wq);
 	if (delegated_inode) {
 		error = break_deleg_wait(&delegated_inode);
 		if (!error) {
diff --git a/fs/nfs/dir.c b/fs/nfs/dir.c
index 71b2e390becf..9a3c51f3040b 100644
--- a/fs/nfs/dir.c
+++ b/fs/nfs/dir.c
@@ -57,7 +57,7 @@ static void nfs_readdir_clear_array(struct page*);
 const struct file_operations nfs_dir_operations = {
 	.llseek		= nfs_llseek_dir,
 	.read		= generic_read_dir,
-	.iterate	= nfs_readdir,
+	.iterate_shared	= nfs_readdir,
 	.open		= nfs_opendir,
 	.release	= nfs_closedir,
 	.fsync		= nfs_fsync_dir,
diff --git a/fs/nfs/inode.c b/fs/nfs/inode.c
index 5b1eee4952b7..9a6dac08cc79 100644
--- a/fs/nfs/inode.c
+++ b/fs/nfs/inode.c
@@ -453,6 +453,8 @@ nfs_fhget(struct super_block *sb, struct nfs_fh *fh, struct nfs_fattr *fattr, st
 
 		/* We can't support update_atime(), since the server will reset it */
 		inode->i_flags |= S_NOATIME|S_NOCMTIME;
+		/* Parallel updates to directory are trivial */
+		inode->i_flags |= S_PAR_UPDATE;
 		inode->i_mode = fattr->mode;
 		nfsi->cache_validity = 0;
 		if ((fattr->valid & NFS_ATTR_FATTR_MODE) == 0
diff --git a/fs/nfs/unlink.c b/fs/nfs/unlink.c
index fd61bf0fce63..2ac7a7be10e6 100644
--- a/fs/nfs/unlink.c
+++ b/fs/nfs/unlink.c
@@ -467,6 +467,7 @@ nfs_sillyrename(struct inode *dir, struct dentry *dentry)
 	sdentry = NULL;
 	do {
 		int slen;
+		d_unlock_modify(sdentry);
 		dput(sdentry);
 		sillycounter++;
 		slen = scnprintf(silly, sizeof(silly),
@@ -484,7 +485,7 @@ nfs_sillyrename(struct inode *dir, struct dentry *dentry)
 		 */
 		if (IS_ERR(sdentry))
 			goto out;
-	} while (d_inode(sdentry) != NULL); /* need negative lookup */
+	} while (!d_lock_create(sdentry)); /* need negative lookup */
 
 	ihold(inode);
 
@@ -529,6 +530,7 @@ nfs_sillyrename(struct inode *dir, struct dentry *dentry)
 	rpc_put_task(task);
 out_dput:
 	iput(inode);
+	d_unlock_modify(sdentry);
 	dput(sdentry);
 out:
 	return error;
diff --git a/include/linux/dcache.h b/include/linux/dcache.h
index ef4b70f64f33..befa52ec4f6b 100644
--- a/include/linux/dcache.h
+++ b/include/linux/dcache.h
@@ -12,7 +12,9 @@
 #include <linux/rcupdate.h>
 #include <linux/lockref.h>
 #include <linux/stringhash.h>
+#include <linux/sched.h>
 #include <linux/wait.h>
+#include <linux/wait_bit.h>
 
 struct path;
 struct vfsmount;
@@ -216,6 +218,7 @@ struct dentry_operations {
 
 #define DCACHE_PAR_LOOKUP		0x10000000 /* being looked up (with parent locked shared) */
 #define DCACHE_DENTRY_CURSOR		0x20000000
+#define DCACHE_PAR_UPDATE		0x40000000 /* Being created or unlinked with shared lock */
 
 extern seqlock_t rename_lock;
 
@@ -599,4 +602,44 @@ struct name_snapshot {
 void take_dentry_name_snapshot(struct name_snapshot *, struct dentry *);
 void release_dentry_name_snapshot(struct name_snapshot *);
 
+/*
+ * Lock a dentry in an S_PAR_UPDATE directory prior to creating
+ * an object with that name.  Will fail if the object is created
+ * or instantiated while we waited.
+ */
+static inline bool d_lock_create(struct dentry *dentry)
+{
+	spin_lock(&dentry->d_lock);
+	if (dentry->d_flags & DCACHE_PAR_UPDATE)
+		___wait_var_event(&dentry->d_flags,
+				  !(dentry->d_flags & DCACHE_PAR_UPDATE),
+				  TASK_UNINTERRUPTIBLE, 0, 0,
+				  (spin_unlock(&dentry->d_lock),
+				   schedule(),
+				   spin_lock(&dentry->d_lock))
+			);
+	if (d_inode(dentry) == NULL) {
+		dentry->d_flags |= DCACHE_PAR_UPDATE;
+		spin_unlock(&dentry->d_lock);
+		return true;
+	} else {
+		spin_unlock(&dentry->d_lock);
+		return false;
+	}
+}
+bool d_lock_modify(struct dentry *dentry,
+		   struct dentry *base, const struct qstr *name);
+
+static inline void d_unlock_modify(struct dentry *dentry)
+{
+	if (IS_ERR_OR_NULL(dentry))
+		return;
+	if (dentry->d_flags & DCACHE_PAR_UPDATE) {
+		spin_lock(&dentry->d_lock);
+		dentry->d_flags &= ~DCACHE_PAR_UPDATE;
+		spin_unlock(&dentry->d_lock);
+		wake_up_var(&dentry->d_flags);
+	}
+}
+
 #endif	/* __LINUX_DCACHE_H */
diff --git a/include/linux/fs.h b/include/linux/fs.h
index c95c0807471f..15b1c3223438 100644
--- a/include/linux/fs.h
+++ b/include/linux/fs.h
@@ -1947,6 +1947,7 @@ struct super_operations {
 #define S_DAX		0	/* Make all the DAX code disappear */
 #endif
 #define S_ENCRYPTED	16384	/* Encrypted file (using fs/crypto/) */
+#define S_PAR_UPDATE	32768	/* Parallel directory operations supported */
 
 /*
  * Note that nosuid etc flags are inode-specific: setting some file-system
-- 
2.14.0.rc0.dirty

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 832 bytes
Desc: not available
URL: <http://lists.lustre.org/pipermail/lustre-devel_lists.lustre.org/attachments/20181123/8ded6d1e/attachment.sig>

From neilb at suse.com  Fri Nov 23 07:15:27 2018
From: neilb at suse.com (NeilBrown)
Date: Fri, 23 Nov 2018 18:15:27 +1100
Subject: [lustre-devel] [PATCH 0/9] Assorted lustre patches - mostly from
 OpenSFS
Message-ID: <154295730810.2850.961218355189474016.stgit@noble>

This is a miscellaneous set of patches for lustre-in-linux.
Apart from the first, they are (bases on) patches from OpenSFS lustre.
I've been trawling through OpenSFS looking for patches that aren't
in linux-lustre.  Most of what I find are only relevant on the
server, but these seem to be appropriate for the client.
In most cases, I don't really know what is going on, so they
might be inappropriate for the client.  I'm hoping that more
knowledgeable people can put be straight.

This are just the first few - I think there are quite a few more
(before we get to e.g. pfl which has been attempted yet).  If
I've got these mostly right, I'll continue looking through
the rest of my list.

Thanks,
NeilBrown

---

Bruno Faccini (2):
      lustre: llite: clear LLIF_FILE_RESTORING when done
      lustre: obdclass: health_check to report unhealthy upon LBUG

Doug Oucharek (1):
      lustre: lnet: Stop MLX5 triggering a dump_cqe

Fan Yong (1):
      lustre: statahead: skip agl for the file in restoring

James Simmons (1):
      lustre: remove EIOCBRETRY handling

Lai Siyao (2):
      lustre: rename: DNE2 should return -EXDEV upon remote rename
      lustre: statahead: add smp_mb() to serialize ops

NeilBrown (1):
      lustre: obdclass: fix formating of connection flags

Vladimir Saveliev (1):
      lustre: ptlrpc: use smp unsafe at_init only for initialization


 .../staging/lustre/lnet/klnds/o2iblnd/o2iblnd.c    |    7 ++++-
 .../staging/lustre/lustre/include/lustre_errno.h   |    1 -
 .../staging/lustre/lustre/include/lustre_import.h  |   19 ++++++++++++--
 drivers/staging/lustre/lustre/llite/llite_lib.c    |    6 ++++
 drivers/staging/lustre/lustre/llite/statahead.c    |   28 ++++++++++++++++++--
 drivers/staging/lustre/lustre/lmv/lmv_obd.c        |    2 +
 .../lustre/lustre/obdclass/lprocfs_status.c        |    4 +--
 drivers/staging/lustre/lustre/obdclass/obd_sysfs.c |    4 ++-
 drivers/staging/lustre/lustre/ptlrpc/import.c      |    2 +
 9 files changed, 61 insertions(+), 12 deletions(-)

--
Signature


From neilb at suse.com  Fri Nov 23 07:15:27 2018
From: neilb at suse.com (NeilBrown)
Date: Fri, 23 Nov 2018 18:15:27 +1100
Subject: [lustre-devel] [PATCH 1/9] lustre: obdclass: fix formating of
 connection flags
In-Reply-To: <154295730810.2850.961218355189474016.stgit@noble>
References: <154295730810.2850.961218355189474016.stgit@noble>
Message-ID: <154295732787.2850.4431010272997855706.stgit@noble>

The current code puts the separator *only*
at the start rather than *never* at the start.

Signed-off-by: NeilBrown <neilb at suse.com>
---
 .../lustre/lustre/obdclass/lprocfs_status.c        |    4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/drivers/staging/lustre/lustre/obdclass/lprocfs_status.c b/drivers/staging/lustre/lustre/obdclass/lprocfs_status.c
index feba2ef5a3bc..acfea7a44350 100644
--- a/drivers/staging/lustre/lustre/obdclass/lprocfs_status.c
+++ b/drivers/staging/lustre/lustre/obdclass/lprocfs_status.c
@@ -709,14 +709,14 @@ static void obd_connect_seq_flags2str(struct seq_file *m, u64 flags,
 	for (i = 0, mask = 1; i < 64; i++, mask <<= 1) {
 		if (flags & mask) {
 			seq_printf(m, "%s%s",
-				   first ? sep : "", obd_connect_names[i]);
+				   first ? "" : sep, obd_connect_names[i]);
 			first = false;
 		}
 	}
 
 	if (flags & ~(mask - 1)) {
 		seq_printf(m, "%sunknown flags %#llx",
-			   first ? sep : "", flags & ~(mask - 1));
+			   first ? "" : sep, flags & ~(mask - 1));
 		first = false;
 	}
 



From neilb at suse.com  Fri Nov 23 07:15:27 2018
From: neilb at suse.com (NeilBrown)
Date: Fri, 23 Nov 2018 18:15:27 +1100
Subject: [lustre-devel] [PATCH 2/9] lustre: remove EIOCBRETRY handling
In-Reply-To: <154295730810.2850.961218355189474016.stgit@noble>
References: <154295730810.2850.961218355189474016.stgit@noble>
Message-ID: <154295732791.2850.17753016627908700130.stgit@noble>

From: James Simmons <uja.ornl at yahoo.com>

With linux commit 41003a7bcfed ("aio: remove retry-based AIO")
AIO retry handling was removed due to it being buggy and no
one using it, including lustre. Since this is the case
remove EIOCBRETRY since it no longer in the kernel starting
with version 3.18.

Signed-off-by: James Simmons <uja.ornl at yahoo.com>
Reviewed-on: http://review.whamcloud.com/14507
WC-bug-id: https://jira.whamcloud.com/browse/LU-6426
Reviewed-by: Bob Glossman <bob.glossman at intel.com>
Reviewed-by: Dmitry Eremin <dmitry.eremin at intel.com>
Reviewed-by: Blake Caldwell <blakec at ornl.gov>
Reviewed-by: Thomas Stibor <thomas at stibor.net>
Reviewed-by: Oleg Drokin <oleg.drokin at intel.com>
Signed-off-by: NeilBrown <neilb at suse.com>
---
 .../staging/lustre/lustre/include/lustre_errno.h   |    1 -
 1 file changed, 1 deletion(-)

diff --git a/drivers/staging/lustre/lustre/include/lustre_errno.h b/drivers/staging/lustre/lustre/include/lustre_errno.h
index 59fbb9f47ff1..806e79672482 100644
--- a/drivers/staging/lustre/lustre/include/lustre_errno.h
+++ b/drivers/staging/lustre/lustre/include/lustre_errno.h
@@ -181,7 +181,6 @@
 #define LUSTRE_EBADTYPE		527	/* Type not supported by server */
 #define LUSTRE_EJUKEBOX		528	/* Request won't finish until timeout */
 #define LUSTRE_EIOCBQUEUED	529	/* iocb queued await completion event */
-#define LUSTRE_EIOCBRETRY	530	/* iocb queued, will trigger a retry */
 
 /*
  * Translations are optimized away on x86.  Host errnos that shouldn't be put



From neilb at suse.com  Fri Nov 23 07:15:27 2018
From: neilb at suse.com (NeilBrown)
Date: Fri, 23 Nov 2018 18:15:27 +1100
Subject: [lustre-devel] [PATCH 3/9] lustre: ptlrpc: use smp unsafe at_init
 only for initialization
In-Reply-To: <154295730810.2850.961218355189474016.stgit@noble>
References: <154295730810.2850.961218355189474016.stgit@noble>
Message-ID: <154295732794.2850.5519811101627910956.stgit@noble>

From: Vladimir Saveliev <vladimir.saveliev at seagate.com>

at_init() is not smp safe, so it is not supposed to be used anywhere
but in at initialization.
Add at_reinit() - safe version of at_init().

Signed-off-by: Vladimir Saveliev <vladimir.saveliev at seagate.com>
WC-bug-id: https://jira.whamcloud.com/browse/LU-6805
Reviewed-on: http://review.whamcloud.com/15522
Reviewed-by: Andreas Dilger <andreas.dilger at intel.com>
Reviewed-by: Chris Horn <hornc at cray.com>
Signed-off-by: NeilBrown <neilb at suse.com>
---
 .../staging/lustre/lustre/include/lustre_import.h  |   19 +++++++++++++++++--
 drivers/staging/lustre/lustre/ptlrpc/import.c      |    2 +-
 2 files changed, 18 insertions(+), 3 deletions(-)

diff --git a/drivers/staging/lustre/lustre/include/lustre_import.h b/drivers/staging/lustre/lustre/include/lustre_import.h
index a629f6bba814..8a8a125bd130 100644
--- a/drivers/staging/lustre/lustre/include/lustre_import.h
+++ b/drivers/staging/lustre/lustre/include/lustre_import.h
@@ -331,12 +331,17 @@ static inline unsigned int at_timeout2est(unsigned int val)
 	return (max((val << 2) / 5, 5U) - 4);
 }
 
-static inline void at_reset(struct adaptive_timeout *at, int val)
+static inline void at_reset_nolock(struct adaptive_timeout *at, int val)
 {
-	spin_lock(&at->at_lock);
 	at->at_current = val;
 	at->at_worst_ever = val;
 	at->at_worst_time = ktime_get_real_seconds();
+}
+
+static inline void at_reset(struct adaptive_timeout *at, int val)
+{
+	spin_lock(&at->at_lock);
+	at_reset_nolock(at, val);
 	spin_unlock(&at->at_lock);
 }
 
@@ -348,6 +353,16 @@ static inline void at_init(struct adaptive_timeout *at, int val, int flags)
 	at_reset(at, val);
 }
 
+static inline void at_reinit(struct adaptive_timeout *at, int val, int flags)
+{
+	spin_lock(&at->at_lock);
+	at->at_binstart = 0;
+	memset(at->at_hist, 0, sizeof(at->at_hist));
+	at->at_flags = flags;
+	at_reset_nolock(at, val);
+	spin_unlock(&at->at_lock);
+}
+
 extern unsigned int at_min;
 static inline int at_get(struct adaptive_timeout *at)
 {
diff --git a/drivers/staging/lustre/lustre/ptlrpc/import.c b/drivers/staging/lustre/lustre/ptlrpc/import.c
index 07dc87d9513e..480c860d066e 100644
--- a/drivers/staging/lustre/lustre/ptlrpc/import.c
+++ b/drivers/staging/lustre/lustre/ptlrpc/import.c
@@ -1036,7 +1036,7 @@ static int ptlrpc_connect_interpret(const struct lu_env *env,
 	 * The net statistics after (re-)connect is not valid anymore,
 	 * because may reflect other routing, etc.
 	 */
-	at_init(&imp->imp_at.iat_net_latency, 0, 0);
+	at_reinit(&imp->imp_at.iat_net_latency, 0, 0);
 	ptlrpc_at_adj_net_latency(request,
 				  lustre_msg_get_service_time(request->rq_repmsg));
 



From neilb at suse.com  Fri Nov 23 07:15:28 2018
From: neilb at suse.com (NeilBrown)
Date: Fri, 23 Nov 2018 18:15:28 +1100
Subject: [lustre-devel] [PATCH 4/9] lustre: rename: DNE2 should return
 -EXDEV upon remote rename
In-Reply-To: <154295730810.2850.961218355189474016.stgit@noble>
References: <154295730810.2850.961218355189474016.stgit@noble>
Message-ID: <154295732797.2850.16990175697450002727.stgit@noble>

From: Lai Siyao <lai.siyao at intel.com>

DNE2 MDS should return -EXDEV upon remote rename, so that old
client can do rename with copy and delete, instead of fail
with -EREMOTE.

Signed-off-by: Lai Siyao <lai.siyao at intel.com>
Change-Id: I68e8e99259065922f31bee5343be309380715674
WC-bug-id: https://jira.whamcloud.com/browse/LU-6660
Reviewed-on: http://review.whamcloud.com/15323
Reviewed-by: Andreas Dilger <andreas.dilger at intel.com>
Reviewed-by: wangdi <di.wang at intel.com>
Reviewed-by: Fan Yong <fan.yong at intel.com>
Signed-off-by: NeilBrown <neilb at suse.com>
---
 drivers/staging/lustre/lustre/lmv/lmv_obd.c |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/staging/lustre/lustre/lmv/lmv_obd.c b/drivers/staging/lustre/lustre/lmv/lmv_obd.c
index 32bb9fca88c9..7e4ffeb15a63 100644
--- a/drivers/staging/lustre/lustre/lmv/lmv_obd.c
+++ b/drivers/staging/lustre/lustre/lmv/lmv_obd.c
@@ -1945,7 +1945,7 @@ static int lmv_rename(struct obd_export *exp, struct md_op_data *op_data,
 	}
 
 	rc = md_rename(target_exp, op_data, old, oldlen, new, newlen, request);
-	if (rc && rc != -EREMOTE)
+	if (rc && rc != -EXDEV)
 		return rc;
 
 	body = req_capsule_server_get(&(*request)->rq_pill, &RMF_MDT_BODY);



From neilb at suse.com  Fri Nov 23 07:15:28 2018
From: neilb at suse.com (NeilBrown)
Date: Fri, 23 Nov 2018 18:15:28 +1100
Subject: [lustre-devel] [PATCH 5/9] lustre: llite: clear LLIF_FILE_RESTORING
 when done
In-Reply-To: <154295730810.2850.961218355189474016.stgit@noble>
References: <154295730810.2850.961218355189474016.stgit@noble>
Message-ID: <154295732800.2850.8303806531168774380.stgit@noble>

From: Bruno Faccini <bruno.faccini at intel.com>

Clear LLIF_FILE_RESTORING if restore done to ensure to start again
to glimpse new attrs.

Signed-off-by: Bruno Faccini <bruno.faccini at intel.com>
WC-bug-id: https://jira.whamcloud.com/browse/LU-6560
Reviewed-on: http://review.whamcloud.com/14609
Reviewed-by: John L. Hammond <john.hammond at intel.com>
Reviewed-by: Jinshan Xiong <jinshan.xiong at intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin at intel.com>
Signed-off-by: NeilBrown <neilb at suse.com>
---
 drivers/staging/lustre/lustre/llite/llite_lib.c |    6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/drivers/staging/lustre/lustre/llite/llite_lib.c b/drivers/staging/lustre/lustre/llite/llite_lib.c
index fac65845526a..b8693049f3b6 100644
--- a/drivers/staging/lustre/lustre/llite/llite_lib.c
+++ b/drivers/staging/lustre/lustre/llite/llite_lib.c
@@ -1905,8 +1905,14 @@ int ll_update_inode(struct inode *inode, struct lustre_md *md)
 	}
 
 	if (body->mbo_valid & OBD_MD_TSTATE) {
+		/* Set LLIF_FILE_RESTORING if restore ongoing and
+		 * clear it when done to ensure to start again
+		 * glimpsing updated attrs
+		 */
 		if (body->mbo_t_state & MS_RESTORE)
 			set_bit(LLIF_FILE_RESTORING, &lli->lli_flags);
+		else
+			clear_bit(LLIF_FILE_RESTORING, &lli->lli_flags);
 	}
 
 	return 0;



From neilb at suse.com  Fri Nov 23 07:15:28 2018
From: neilb at suse.com (NeilBrown)
Date: Fri, 23 Nov 2018 18:15:28 +1100
Subject: [lustre-devel] [PATCH 6/9] lustre: obdclass: health_check to report
 unhealthy upon LBUG
In-Reply-To: <154295730810.2850.961218355189474016.stgit@noble>
References: <154295730810.2850.961218355189474016.stgit@noble>
Message-ID: <154295732803.2850.16198081335816276527.stgit@noble>

From: Bruno Faccini <bruno.faccini at intel.com>

When a LBUG has occurred, without panic_on_lbug being set, health_check
/proc file must return an unhealthy state.

Signed-off-by: Bruno Faccini <bruno.faccini at intel.com>
WC-bug-id: https://jira.whamcloud.com/browse/LU-7486
Reviewed-on: http://review.whamcloud.com/17981
Reviewed-by: Bobi Jam <bobijam at hotmail.com>
Reviewed-by: Niu Yawei <yawei.niu at intel.com>
Reviewed-by: James Simmons <uja.ornl at yahoo.com>
Reviewed-by: Oleg Drokin <oleg.drokin at intel.com>
Signed-off-by: NeilBrown <neilb at suse.com>
---
 drivers/staging/lustre/lustre/obdclass/obd_sysfs.c |    4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/drivers/staging/lustre/lustre/obdclass/obd_sysfs.c b/drivers/staging/lustre/lustre/obdclass/obd_sysfs.c
index 6669c235dd51..5fd30a8e2b44 100644
--- a/drivers/staging/lustre/lustre/obdclass/obd_sysfs.c
+++ b/drivers/staging/lustre/lustre/obdclass/obd_sysfs.c
@@ -173,8 +173,10 @@ health_check_show(struct kobject *kobj, struct attribute *attr, char *buf)
 	int i;
 	size_t len = 0;
 
-	if (libcfs_catastrophe)
+	if (libcfs_catastrophe) {
 		return sprintf(buf, "LBUG\n");
+		healthy = false;
+	}
 
 	read_lock(&obd_dev_lock);
 	for (i = 0; i < class_devno_max(); i++) {



From neilb at suse.com  Fri Nov 23 07:15:28 2018
From: neilb at suse.com (NeilBrown)
Date: Fri, 23 Nov 2018 18:15:28 +1100
Subject: [lustre-devel] [PATCH 7/9] lustre: lnet: Stop MLX5 triggering a
 dump_cqe
In-Reply-To: <154295730810.2850.961218355189474016.stgit@noble>
References: <154295730810.2850.961218355189474016.stgit@noble>
Message-ID: <154295732806.2850.603181458106225374.stgit@noble>

From: Doug Oucharek <doug.s.oucharek at intel.com>

We have found that MLX5 will trigger a dump_cqe if we don't
invalidate the rkey on a newly allocated MR for FastReg usage.

This fix just tags the MR as invalid on its creation if we are
using FastReg and that will force it to do an invalidate of the
rkey on first usage.

Signed-off-by: Doug Oucharek <doug.s.oucharek at intel.com>
WC-bug-id: https://jira.whamcloud.com/browse/LU-8752
Reviewed-on: https://review.whamcloud.com/24306
Reviewed-by: Dmitry Eremin <dmitry.eremin at intel.com>
Reviewed-by: Amir Shehata <amir.shehata at intel.com>
Reviewed-by: James Simmons <uja.ornl at yahoo.com>
Reviewed-by: Oleg Drokin <oleg.drokin at intel.com>
Signed-off-by: NeilBrown <neilb at suse.com>
---
 .../staging/lustre/lnet/klnds/o2iblnd/o2iblnd.c    |    7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)

diff --git a/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.c b/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.c
index ecdf4dee533d..a5eada8ee354 100644
--- a/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.c
+++ b/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.c
@@ -1483,7 +1483,12 @@ static int kiblnd_alloc_freg_pool(struct kib_fmr_poolset *fps,
 			goto out_middle;
 		}
 
-		frd->frd_valid = true;
+		/*
+		 * There appears to be a bug in MLX5 code where you must
+		 * invalidate the rkey of a new FastReg pool before first
+		 * using it. Thus, I am marking the FRD invalid here.
+		 */
+		frd->frd_valid = false;
 
 		list_add_tail(&frd->frd_list, &fpo->fast_reg.fpo_pool_list);
 		fpo->fast_reg.fpo_pool_size++;



From neilb at suse.com  Fri Nov 23 07:15:28 2018
From: neilb at suse.com (NeilBrown)
Date: Fri, 23 Nov 2018 18:15:28 +1100
Subject: [lustre-devel] [PATCH 8/9] lustre: statahead: skip agl for the file
 in restoring
In-Reply-To: <154295730810.2850.961218355189474016.stgit@noble>
References: <154295730810.2850.961218355189474016.stgit@noble>
Message-ID: <154295732808.2850.16445000176556176904.stgit@noble>

From: Fan Yong <fan.yong at intel.com>

In case of restore, the MDT has the right size and has already sent
it back without granting the layout lock, inode is up-to-date. Then
AGL (async glimpse lock) is useless.

Also to glimpse we need the layout, in case of a running restore the
MDT holds the layout lock so the glimpse will block up to the end of
restore (statahead/agl will block).

Signed-off-by: Fan Yong <fan.yong at intel.com>
WC-bug-id: https://jira.whamcloud.com/browse/LU-9319
Reviewed-by: Lai Siyao <lai.siyao at intel.com>
Reviewed-by: Jinshan Xiong <jinshan.xiong at intel.com>
Reviewed-by: John L. Hammond <john.hammond at intel.com>
Reviewed-on: https://review.whamcloud.com/26501
Reviewed-by: Oleg Drokin <oleg.drokin at intel.com>
Signed-off-by: NeilBrown <neilb at suse.com>
---
 drivers/staging/lustre/lustre/llite/statahead.c |   13 +++++++++++++
 1 file changed, 13 insertions(+)

diff --git a/drivers/staging/lustre/lustre/llite/statahead.c b/drivers/staging/lustre/lustre/llite/statahead.c
index 28e85bfb9b82..3d71322aa1c7 100644
--- a/drivers/staging/lustre/lustre/llite/statahead.c
+++ b/drivers/staging/lustre/lustre/llite/statahead.c
@@ -504,6 +504,19 @@ static void ll_agl_trigger(struct inode *inode, struct ll_statahead_info *sai)
 		return;
 	}
 
+	/* In case of restore, the MDT has the right size and has already
+	 * sent it back without granting the layout lock, inode is up-to-date.
+	 * Then AGL (async glimpse lock) is useless.
+	 * Also to glimpse we need the layout, in case of a runninh restore
+	 * the MDT holds the layout lock so the glimpse will block up to the
+	 * end of restore (statahead/agl will block)
+	 */
+	if (test_bit(LLIF_FILE_RESTORING, &lli->lli_flags)) {
+		lli->lli_agl_index = 0;
+		iput(inode);
+		return;
+	}
+
 	/* Someone is in glimpse (sync or async), do nothing. */
 	rc = down_write_trylock(&lli->lli_glimpse_sem);
 	if (rc == 0) {



From neilb at suse.com  Fri Nov 23 07:15:28 2018
From: neilb at suse.com (NeilBrown)
Date: Fri, 23 Nov 2018 18:15:28 +1100
Subject: [lustre-devel] [PATCH 9/9] lustre: statahead: add smp_mb() to
 serialize ops
In-Reply-To: <154295730810.2850.961218355189474016.stgit@noble>
References: <154295730810.2850.961218355189474016.stgit@noble>
Message-ID: <154295732811.2850.6059970126126914084.stgit@noble>

From: Lai Siyao <lai.siyao at intel.com>

In ll_deauthorize_statahead(), it set thread stop flag, and then
wake up thread, however wakeup is called inside spinlock in case
ll_statahead_info is released, then we need to call smp_mb() to
serialize setting and wakeup.

Signed-off-by: Lai Siyao <lai.siyao at intel.com>
WC-bug-id: https://jira.whamcloud.com/browse/LU-7994
Reviewed-on: https://review.whamcloud.com/23040
Reviewed-by: Fan Yong <fan.yong at intel.com>
Reviewed-by: Bobi Jam <bobijam at hotmail.com>
Reviewed-by: Oleg Drokin <oleg.drokin at intel.com>
Signed-off-by: NeilBrown <neilb at suse.com>
---
 drivers/staging/lustre/lustre/llite/statahead.c |   15 ++++++++++++---
 1 file changed, 12 insertions(+), 3 deletions(-)

diff --git a/drivers/staging/lustre/lustre/llite/statahead.c b/drivers/staging/lustre/lustre/llite/statahead.c
index 3d71322aa1c7..24c2335c70a7 100644
--- a/drivers/staging/lustre/lustre/llite/statahead.c
+++ b/drivers/staging/lustre/lustre/llite/statahead.c
@@ -1110,8 +1110,9 @@ static int ll_statahead_thread(void *arg)
 		sa_handle_callback(sai);
 
 		set_current_state(TASK_IDLE);
+		/* ensure we see the NULL stored by ll_deauthorize_statahead() */
 		if (!sa_has_callback(sai) &&
-		    sai->sai_task)
+		    smp_load_acquire(&sai->sai_task))
 			schedule();
 		__set_current_state(TASK_RUNNING);
 	}
@@ -1191,9 +1192,17 @@ void ll_deauthorize_statahead(struct inode *dir, void *key)
 		/*
 		 * statahead thread may not quit yet because it needs to cache
 		 * entries, now it's time to tell it to quit.
+		 *
+		 * In case sai is released, wake_up() is called inside spinlock,
+		 * so we use smp_store_release() to serialize ops.
 		 */
-		wake_up_process(sai->sai_task);
-		sai->sai_task = NULL;
+		struct task_struct *task = sai->sai_task;
+
+		/* ensure ll_statahead_thread sees the NULL before
+		 * calling schedule() again.
+		 */
+		smp_store_release(&sai->sai_task, NULL);
+		wake_up_process(task);
 	}
 	spin_unlock(&lli->lli_sa_lock);
 }



From jsimmons at infradead.org  Mon Nov 26 01:30:01 2018
From: jsimmons at infradead.org (James Simmons)
Date: Mon, 26 Nov 2018 01:30:01 +0000 (GMT)
Subject: [lustre-devel] [PATCH 1/9] lustre: obdclass: fix formating of
 connection flags
In-Reply-To: <154295732787.2850.4431010272997855706.stgit@noble>
References: <154295730810.2850.961218355189474016.stgit@noble>
 <154295732787.2850.4431010272997855706.stgit@noble>
Message-ID: <alpine.LFD.2.21.1811260128280.32255@casper.infradead.org>


> The current code puts the separator *only*
> at the start rather than *never* at the start.

Reviewed-by: James Simmons <jsimmons at infradead.org>y
 
> Signed-off-by: NeilBrown <neilb at suse.com>
> ---
>  .../lustre/lustre/obdclass/lprocfs_status.c        |    4 ++--
>  1 file changed, 2 insertions(+), 2 deletions(-)
> 
> diff --git a/drivers/staging/lustre/lustre/obdclass/lprocfs_status.c b/drivers/staging/lustre/lustre/obdclass/lprocfs_status.c
> index feba2ef5a3bc..acfea7a44350 100644
> --- a/drivers/staging/lustre/lustre/obdclass/lprocfs_status.c
> +++ b/drivers/staging/lustre/lustre/obdclass/lprocfs_status.c
> @@ -709,14 +709,14 @@ static void obd_connect_seq_flags2str(struct seq_file *m, u64 flags,
>  	for (i = 0, mask = 1; i < 64; i++, mask <<= 1) {
>  		if (flags & mask) {
>  			seq_printf(m, "%s%s",
> -				   first ? sep : "", obd_connect_names[i]);
> +				   first ? "" : sep, obd_connect_names[i]);
>  			first = false;
>  		}
>  	}
>  
>  	if (flags & ~(mask - 1)) {
>  		seq_printf(m, "%sunknown flags %#llx",
> -			   first ? sep : "", flags & ~(mask - 1));
> +			   first ? "" : sep, flags & ~(mask - 1));
>  		first = false;
>  	}
>  
> 
> 
> 

From jsimmons at infradead.org  Mon Nov 26 01:30:25 2018
From: jsimmons at infradead.org (James Simmons)
Date: Mon, 26 Nov 2018 01:30:25 +0000 (GMT)
Subject: [lustre-devel] [PATCH 2/9] lustre: remove EIOCBRETRY handling
In-Reply-To: <154295732791.2850.17753016627908700130.stgit@noble>
References: <154295730810.2850.961218355189474016.stgit@noble>
 <154295732791.2850.17753016627908700130.stgit@noble>
Message-ID: <alpine.LFD.2.21.1811260130160.32255@casper.infradead.org>


> From: James Simmons <uja.ornl at yahoo.com>
> 
> With linux commit 41003a7bcfed ("aio: remove retry-based AIO")
> AIO retry handling was removed due to it being buggy and no
> one using it, including lustre. Since this is the case
> remove EIOCBRETRY since it no longer in the kernel starting
> with version 3.18.

Reviewed-by: James Simmons <jsimmons at infradead.org>
 
> Signed-off-by: James Simmons <uja.ornl at yahoo.com>
> Reviewed-on: http://review.whamcloud.com/14507
> WC-bug-id: https://jira.whamcloud.com/browse/LU-6426
> Reviewed-by: Bob Glossman <bob.glossman at intel.com>
> Reviewed-by: Dmitry Eremin <dmitry.eremin at intel.com>
> Reviewed-by: Blake Caldwell <blakec at ornl.gov>
> Reviewed-by: Thomas Stibor <thomas at stibor.net>
> Reviewed-by: Oleg Drokin <oleg.drokin at intel.com>
> Signed-off-by: NeilBrown <neilb at suse.com>
> ---
>  .../staging/lustre/lustre/include/lustre_errno.h   |    1 -
>  1 file changed, 1 deletion(-)
> 
> diff --git a/drivers/staging/lustre/lustre/include/lustre_errno.h b/drivers/staging/lustre/lustre/include/lustre_errno.h
> index 59fbb9f47ff1..806e79672482 100644
> --- a/drivers/staging/lustre/lustre/include/lustre_errno.h
> +++ b/drivers/staging/lustre/lustre/include/lustre_errno.h
> @@ -181,7 +181,6 @@
>  #define LUSTRE_EBADTYPE		527	/* Type not supported by server */
>  #define LUSTRE_EJUKEBOX		528	/* Request won't finish until timeout */
>  #define LUSTRE_EIOCBQUEUED	529	/* iocb queued await completion event */
> -#define LUSTRE_EIOCBRETRY	530	/* iocb queued, will trigger a retry */
>  
>  /*
>   * Translations are optimized away on x86.  Host errnos that shouldn't be put
> 
> 
> 

From jsimmons at infradead.org  Mon Nov 26 01:31:32 2018
From: jsimmons at infradead.org (James Simmons)
Date: Mon, 26 Nov 2018 01:31:32 +0000 (GMT)
Subject: [lustre-devel] [PATCH 4/9] lustre: rename: DNE2 should return
 -EXDEV upon remote rename
In-Reply-To: <154295732797.2850.16990175697450002727.stgit@noble>
References: <154295730810.2850.961218355189474016.stgit@noble>
 <154295732797.2850.16990175697450002727.stgit@noble>
Message-ID: <alpine.LFD.2.21.1811260130360.32255@casper.infradead.org>



On Fri, 23 Nov 2018, NeilBrown wrote:

> From: Lai Siyao <lai.siyao at intel.com>
> 
> DNE2 MDS should return -EXDEV upon remote rename, so that old
> client can do rename with copy and delete, instead of fail
> with -EREMOTE.

Let me guess you were debugging the migration failures and fould this :-)
I was doing the same thing.

Reviewed-by: James Simmons <jsimmons at infradead.org>
 
> Signed-off-by: Lai Siyao <lai.siyao at intel.com>
> Change-Id: I68e8e99259065922f31bee5343be309380715674
> WC-bug-id: https://jira.whamcloud.com/browse/LU-6660
> Reviewed-on: http://review.whamcloud.com/15323
> Reviewed-by: Andreas Dilger <andreas.dilger at intel.com>
> Reviewed-by: wangdi <di.wang at intel.com>
> Reviewed-by: Fan Yong <fan.yong at intel.com>
> Signed-off-by: NeilBrown <neilb at suse.com>
> ---
>  drivers/staging/lustre/lustre/lmv/lmv_obd.c |    2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/drivers/staging/lustre/lustre/lmv/lmv_obd.c b/drivers/staging/lustre/lustre/lmv/lmv_obd.c
> index 32bb9fca88c9..7e4ffeb15a63 100644
> --- a/drivers/staging/lustre/lustre/lmv/lmv_obd.c
> +++ b/drivers/staging/lustre/lustre/lmv/lmv_obd.c
> @@ -1945,7 +1945,7 @@ static int lmv_rename(struct obd_export *exp, struct md_op_data *op_data,
>  	}
>  
>  	rc = md_rename(target_exp, op_data, old, oldlen, new, newlen, request);
> -	if (rc && rc != -EREMOTE)
> +	if (rc && rc != -EXDEV)
>  		return rc;
>  
>  	body = req_capsule_server_get(&(*request)->rq_pill, &RMF_MDT_BODY);
> 
> 
> 

From jsimmons at infradead.org  Mon Nov 26 01:32:53 2018
From: jsimmons at infradead.org (James Simmons)
Date: Mon, 26 Nov 2018 01:32:53 +0000 (GMT)
Subject: [lustre-devel] [PATCH 3/9] lustre: ptlrpc: use smp unsafe
 at_init only for initialization
In-Reply-To: <154295732794.2850.5519811101627910956.stgit@noble>
References: <154295730810.2850.961218355189474016.stgit@noble>
 <154295732794.2850.5519811101627910956.stgit@noble>
Message-ID: <alpine.LFD.2.21.1811260132240.32255@casper.infradead.org>


> From: Vladimir Saveliev <vladimir.saveliev at seagate.com>
> 
> at_init() is not smp safe, so it is not supposed to be used anywhere
> but in at initialization.
> Add at_reinit() - safe version of at_init().

Reviewed-by: James Simmons <jsimmons at infradead.org>
 
> Signed-off-by: Vladimir Saveliev <vladimir.saveliev at seagate.com>
> WC-bug-id: https://jira.whamcloud.com/browse/LU-6805
> Reviewed-on: http://review.whamcloud.com/15522
> Reviewed-by: Andreas Dilger <andreas.dilger at intel.com>
> Reviewed-by: Chris Horn <hornc at cray.com>
> Signed-off-by: NeilBrown <neilb at suse.com>
> ---
>  .../staging/lustre/lustre/include/lustre_import.h  |   19 +++++++++++++++++--
>  drivers/staging/lustre/lustre/ptlrpc/import.c      |    2 +-
>  2 files changed, 18 insertions(+), 3 deletions(-)
> 
> diff --git a/drivers/staging/lustre/lustre/include/lustre_import.h b/drivers/staging/lustre/lustre/include/lustre_import.h
> index a629f6bba814..8a8a125bd130 100644
> --- a/drivers/staging/lustre/lustre/include/lustre_import.h
> +++ b/drivers/staging/lustre/lustre/include/lustre_import.h
> @@ -331,12 +331,17 @@ static inline unsigned int at_timeout2est(unsigned int val)
>  	return (max((val << 2) / 5, 5U) - 4);
>  }
>  
> -static inline void at_reset(struct adaptive_timeout *at, int val)
> +static inline void at_reset_nolock(struct adaptive_timeout *at, int val)
>  {
> -	spin_lock(&at->at_lock);
>  	at->at_current = val;
>  	at->at_worst_ever = val;
>  	at->at_worst_time = ktime_get_real_seconds();
> +}
> +
> +static inline void at_reset(struct adaptive_timeout *at, int val)
> +{
> +	spin_lock(&at->at_lock);
> +	at_reset_nolock(at, val);
>  	spin_unlock(&at->at_lock);
>  }
>  
> @@ -348,6 +353,16 @@ static inline void at_init(struct adaptive_timeout *at, int val, int flags)
>  	at_reset(at, val);
>  }
>  
> +static inline void at_reinit(struct adaptive_timeout *at, int val, int flags)
> +{
> +	spin_lock(&at->at_lock);
> +	at->at_binstart = 0;
> +	memset(at->at_hist, 0, sizeof(at->at_hist));
> +	at->at_flags = flags;
> +	at_reset_nolock(at, val);
> +	spin_unlock(&at->at_lock);
> +}
> +
>  extern unsigned int at_min;
>  static inline int at_get(struct adaptive_timeout *at)
>  {
> diff --git a/drivers/staging/lustre/lustre/ptlrpc/import.c b/drivers/staging/lustre/lustre/ptlrpc/import.c
> index 07dc87d9513e..480c860d066e 100644
> --- a/drivers/staging/lustre/lustre/ptlrpc/import.c
> +++ b/drivers/staging/lustre/lustre/ptlrpc/import.c
> @@ -1036,7 +1036,7 @@ static int ptlrpc_connect_interpret(const struct lu_env *env,
>  	 * The net statistics after (re-)connect is not valid anymore,
>  	 * because may reflect other routing, etc.
>  	 */
> -	at_init(&imp->imp_at.iat_net_latency, 0, 0);
> +	at_reinit(&imp->imp_at.iat_net_latency, 0, 0);
>  	ptlrpc_at_adj_net_latency(request,
>  				  lustre_msg_get_service_time(request->rq_repmsg));
>  
> 
> 
> 

From jsimmons at infradead.org  Mon Nov 26 01:34:19 2018
From: jsimmons at infradead.org (James Simmons)
Date: Mon, 26 Nov 2018 01:34:19 +0000 (GMT)
Subject: [lustre-devel] [PATCH 5/9] lustre: llite: clear
 LLIF_FILE_RESTORING when done
In-Reply-To: <154295732800.2850.8303806531168774380.stgit@noble>
References: <154295730810.2850.961218355189474016.stgit@noble>
 <154295732800.2850.8303806531168774380.stgit@noble>
Message-ID: <alpine.LFD.2.21.1811260133540.32255@casper.infradead.org>


> From: Bruno Faccini <bruno.faccini at intel.com>
> 
> Clear LLIF_FILE_RESTORING if restore done to ensure to start again
> to glimpse new attrs.

Reviewed-by: James Simmons <jsimmons at infradead.org>
 
> Signed-off-by: Bruno Faccini <bruno.faccini at intel.com>
> WC-bug-id: https://jira.whamcloud.com/browse/LU-6560
> Reviewed-on: http://review.whamcloud.com/14609
> Reviewed-by: John L. Hammond <john.hammond at intel.com>
> Reviewed-by: Jinshan Xiong <jinshan.xiong at intel.com>
> Reviewed-by: Oleg Drokin <oleg.drokin at intel.com>
> Signed-off-by: NeilBrown <neilb at suse.com>
> ---
>  drivers/staging/lustre/lustre/llite/llite_lib.c |    6 ++++++
>  1 file changed, 6 insertions(+)
> 
> diff --git a/drivers/staging/lustre/lustre/llite/llite_lib.c b/drivers/staging/lustre/lustre/llite/llite_lib.c
> index fac65845526a..b8693049f3b6 100644
> --- a/drivers/staging/lustre/lustre/llite/llite_lib.c
> +++ b/drivers/staging/lustre/lustre/llite/llite_lib.c
> @@ -1905,8 +1905,14 @@ int ll_update_inode(struct inode *inode, struct lustre_md *md)
>  	}
>  
>  	if (body->mbo_valid & OBD_MD_TSTATE) {
> +		/* Set LLIF_FILE_RESTORING if restore ongoing and
> +		 * clear it when done to ensure to start again
> +		 * glimpsing updated attrs
> +		 */
>  		if (body->mbo_t_state & MS_RESTORE)
>  			set_bit(LLIF_FILE_RESTORING, &lli->lli_flags);
> +		else
> +			clear_bit(LLIF_FILE_RESTORING, &lli->lli_flags);
>  	}
>  
>  	return 0;
> 
> 
> 

From jsimmons at infradead.org  Mon Nov 26 01:46:10 2018
From: jsimmons at infradead.org (James Simmons)
Date: Mon, 26 Nov 2018 01:46:10 +0000 (GMT)
Subject: [lustre-devel] [PATCH 6/9] lustre: obdclass: health_check to
 report unhealthy upon LBUG
In-Reply-To: <154295732803.2850.16198081335816276527.stgit@noble>
References: <154295730810.2850.961218355189474016.stgit@noble>
 <154295732803.2850.16198081335816276527.stgit@noble>
Message-ID: <alpine.LFD.2.21.1811260134310.32255@casper.infradead.org>


> From: Bruno Faccini <bruno.faccini at intel.com>
> 
> When a LBUG has occurred, without panic_on_lbug being set, health_check
> /proc file must return an unhealthy state.

I pushed this one to Greg which was disliked since it breaks the one item 
per sysfs rule. See 

https://lore.kernel.org/patchwork/patch/755571

I did start a proper port to sysfs at 
https://review.whamcloud.com/#/c/25631

but it needs to be updated. I do like Andreas idea of a sysfs and debugfs
file since lctl get_param will return the results from both together.
We could land it as is and update the sysfs handling at a latter date 
(shouldn't be too far down the road). Here is my review in case you want
to land it.y

> Signed-off-by: Bruno Faccini <bruno.faccini at intel.com>
> WC-bug-id: https://jira.whamcloud.com/browse/LU-7486
> Reviewed-on: http://review.whamcloud.com/17981
> Reviewed-by: Bobi Jam <bobijam at hotmail.com>
> Reviewed-by: Niu Yawei <yawei.niu at intel.com>
> Reviewed-by: James Simmons <uja.ornl at yahoo.com>
> Reviewed-by: Oleg Drokin <oleg.drokin at intel.com>
> Signed-off-by: NeilBrown <neilb at suse.com>
> ---
>  drivers/staging/lustre/lustre/obdclass/obd_sysfs.c |    4 +++-
>  1 file changed, 3 insertions(+), 1 deletion(-)
> 
> diff --git a/drivers/staging/lustre/lustre/obdclass/obd_sysfs.c b/drivers/staging/lustre/lustre/obdclass/obd_sysfs.c
> index 6669c235dd51..5fd30a8e2b44 100644
> --- a/drivers/staging/lustre/lustre/obdclass/obd_sysfs.c
> +++ b/drivers/staging/lustre/lustre/obdclass/obd_sysfs.c
> @@ -173,8 +173,10 @@ health_check_show(struct kobject *kobj, struct attribute *attr, char *buf)
>  	int i;
>  	size_t len = 0;
>  
> -	if (libcfs_catastrophe)
> +	if (libcfs_catastrophe) {
>  		return sprintf(buf, "LBUG\n");
> +		healthy = false;
> +	}
>  
>  	read_lock(&obd_dev_lock);
>  	for (i = 0; i < class_devno_max(); i++) {
> 
> 
> 

From jsimmons at infradead.org  Mon Nov 26 01:46:42 2018
From: jsimmons at infradead.org (James Simmons)
Date: Mon, 26 Nov 2018 01:46:42 +0000 (GMT)
Subject: [lustre-devel] [PATCH 6/9] lustre: obdclass: health_check to
 report unhealthy upon LBUG
In-Reply-To: <154295732803.2850.16198081335816276527.stgit@noble>
References: <154295730810.2850.961218355189474016.stgit@noble>
 <154295732803.2850.16198081335816276527.stgit@noble>
Message-ID: <alpine.LFD.2.21.1811260146120.32255@casper.infradead.org>


> From: Bruno Faccini <bruno.faccini at intel.com>
> 
> When a LBUG has occurred, without panic_on_lbug being set, health_check
> /proc file must return an unhealthy state.

I pushed this one to Greg which was disliked since it breaks the one item 
per sysfs rule. See 

https://lore.kernel.org/patchwork/patch/755571

I did start a proper port to sysfs at 
https://review.whamcloud.com/#/c/25631

but it needs to be updated. I do like Andreas idea of a sysfs and debugfs
file since lctl get_param will return the results from both together.
We could land it as is and update the sysfs handling at a latter date 
(shouldn't be too far down the road). Here is my review in case you want
to land it.

Reviewed-by: James Simmons <jsimmons at infradead.org>

> Signed-off-by: Bruno Faccini <bruno.faccini at intel.com>
> WC-bug-id: https://jira.whamcloud.com/browse/LU-7486
> Reviewed-on: http://review.whamcloud.com/17981
> Reviewed-by: Bobi Jam <bobijam at hotmail.com>
> Reviewed-by: Niu Yawei <yawei.niu at intel.com>
> Reviewed-by: James Simmons <uja.ornl at yahoo.com>
> Reviewed-by: Oleg Drokin <oleg.drokin at intel.com>
> Signed-off-by: NeilBrown <neilb at suse.com>
> ---
>  drivers/staging/lustre/lustre/obdclass/obd_sysfs.c |    4 +++-
>  1 file changed, 3 insertions(+), 1 deletion(-)
> 
> diff --git a/drivers/staging/lustre/lustre/obdclass/obd_sysfs.c b/drivers/staging/lustre/lustre/obdclass/obd_sysfs.c
> index 6669c235dd51..5fd30a8e2b44 100644
> --- a/drivers/staging/lustre/lustre/obdclass/obd_sysfs.c
> +++ b/drivers/staging/lustre/lustre/obdclass/obd_sysfs.c
> @@ -173,8 +173,10 @@ health_check_show(struct kobject *kobj, struct attribute *attr, char *buf)
>  	int i;
>  	size_t len = 0;
>  
> -	if (libcfs_catastrophe)
> +	if (libcfs_catastrophe) {
>  		return sprintf(buf, "LBUG\n");
> +		healthy = false;
> +	}
>  
>  	read_lock(&obd_dev_lock);
>  	for (i = 0; i < class_devno_max(); i++) {
> 
> 
> 

From jsimmons at infradead.org  Mon Nov 26 01:49:01 2018
From: jsimmons at infradead.org (James Simmons)
Date: Mon, 26 Nov 2018 01:49:01 +0000 (GMT)
Subject: [lustre-devel] [PATCH 7/9] lustre: lnet: Stop MLX5 triggering a
 dump_cqe
In-Reply-To: <154295732806.2850.603181458106225374.stgit@noble>
References: <154295730810.2850.961218355189474016.stgit@noble>
 <154295732806.2850.603181458106225374.stgit@noble>
Message-ID: <alpine.LFD.2.21.1811260146540.32255@casper.infradead.org>


> From: Doug Oucharek <doug.s.oucharek at intel.com>
> 
> We have found that MLX5 will trigger a dump_cqe if we don't
> invalidate the rkey on a newly allocated MR for FastReg usage.
> 
> This fix just tags the MR as invalid on its creation if we are
> using FastReg and that will force it to do an invalidate of the
> rkey on first usage.

I pushed this one already, see https://lkml.org/lkml/2018/3/16/1410.
Dan felt this was more a infiniband layer bug that needed to be fixed.
It could be fixed already upstream or if it is not once this problem
is reported we will need to work the rdma group to fix it.
 
> Signed-off-by: Doug Oucharek <doug.s.oucharek at intel.com>
> WC-bug-id: https://jira.whamcloud.com/browse/LU-8752
> Reviewed-on: https://review.whamcloud.com/24306
> Reviewed-by: Dmitry Eremin <dmitry.eremin at intel.com>
> Reviewed-by: Amir Shehata <amir.shehata at intel.com>
> Reviewed-by: James Simmons <uja.ornl at yahoo.com>
> Reviewed-by: Oleg Drokin <oleg.drokin at intel.com>
> Signed-off-by: NeilBrown <neilb at suse.com>
> ---
>  .../staging/lustre/lnet/klnds/o2iblnd/o2iblnd.c    |    7 ++++++-
>  1 file changed, 6 insertions(+), 1 deletion(-)
> 
> diff --git a/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.c b/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.c
> index ecdf4dee533d..a5eada8ee354 100644
> --- a/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.c
> +++ b/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.c
> @@ -1483,7 +1483,12 @@ static int kiblnd_alloc_freg_pool(struct kib_fmr_poolset *fps,
>  			goto out_middle;
>  		}
>  
> -		frd->frd_valid = true;
> +		/*
> +		 * There appears to be a bug in MLX5 code where you must
> +		 * invalidate the rkey of a new FastReg pool before first
> +		 * using it. Thus, I am marking the FRD invalid here.
> +		 */
> +		frd->frd_valid = false;
>  
>  		list_add_tail(&frd->frd_list, &fpo->fast_reg.fpo_pool_list);
>  		fpo->fast_reg.fpo_pool_size++;
> 
> 
> 

From jsimmons at infradead.org  Mon Nov 26 02:09:40 2018
From: jsimmons at infradead.org (James Simmons)
Date: Mon, 26 Nov 2018 02:09:40 +0000 (GMT)
Subject: [lustre-devel] [PATCH 8/9] lustre: statahead: skip agl for the
 file in restoring
In-Reply-To: <154295732808.2850.16445000176556176904.stgit@noble>
References: <154295730810.2850.961218355189474016.stgit@noble>
 <154295732808.2850.16445000176556176904.stgit@noble>
Message-ID: <alpine.LFD.2.21.1811260209230.32255@casper.infradead.org>


> From: Fan Yong <fan.yong at intel.com>
> 
> In case of restore, the MDT has the right size and has already sent
> it back without granting the layout lock, inode is up-to-date. Then
> AGL (async glimpse lock) is useless.
> 
> Also to glimpse we need the layout, in case of a running restore the
> MDT holds the layout lock so the glimpse will block up to the end of
> restore (statahead/agl will block).

Reviewed-by: James Simmons <jsimmons at infradead.org>
 
> Signed-off-by: Fan Yong <fan.yong at intel.com>
> WC-bug-id: https://jira.whamcloud.com/browse/LU-9319
> Reviewed-by: Lai Siyao <lai.siyao at intel.com>
> Reviewed-by: Jinshan Xiong <jinshan.xiong at intel.com>
> Reviewed-by: John L. Hammond <john.hammond at intel.com>
> Reviewed-on: https://review.whamcloud.com/26501
> Reviewed-by: Oleg Drokin <oleg.drokin at intel.com>
> Signed-off-by: NeilBrown <neilb at suse.com>
> ---
>  drivers/staging/lustre/lustre/llite/statahead.c |   13 +++++++++++++
>  1 file changed, 13 insertions(+)
> 
> diff --git a/drivers/staging/lustre/lustre/llite/statahead.c b/drivers/staging/lustre/lustre/llite/statahead.c
> index 28e85bfb9b82..3d71322aa1c7 100644
> --- a/drivers/staging/lustre/lustre/llite/statahead.c
> +++ b/drivers/staging/lustre/lustre/llite/statahead.c
> @@ -504,6 +504,19 @@ static void ll_agl_trigger(struct inode *inode, struct ll_statahead_info *sai)
>  		return;
>  	}
>  
> +	/* In case of restore, the MDT has the right size and has already
> +	 * sent it back without granting the layout lock, inode is up-to-date.
> +	 * Then AGL (async glimpse lock) is useless.
> +	 * Also to glimpse we need the layout, in case of a runninh restore
> +	 * the MDT holds the layout lock so the glimpse will block up to the
> +	 * end of restore (statahead/agl will block)
> +	 */
> +	if (test_bit(LLIF_FILE_RESTORING, &lli->lli_flags)) {
> +		lli->lli_agl_index = 0;
> +		iput(inode);
> +		return;
> +	}
> +
>  	/* Someone is in glimpse (sync or async), do nothing. */
>  	rc = down_write_trylock(&lli->lli_glimpse_sem);
>  	if (rc == 0) {
> 
> 
> 

From jsimmons at infradead.org  Mon Nov 26 02:09:58 2018
From: jsimmons at infradead.org (James Simmons)
Date: Mon, 26 Nov 2018 02:09:58 +0000 (GMT)
Subject: [lustre-devel] [PATCH 8/9] lustre: statahead: skip agl for the
 file in restoring
In-Reply-To: <154295732808.2850.16445000176556176904.stgit@noble>
References: <154295730810.2850.961218355189474016.stgit@noble>
 <154295732808.2850.16445000176556176904.stgit@noble>
Message-ID: <alpine.LFD.2.21.1811260209420.32255@casper.infradead.org>

> From: Fan Yong <fan.yong at intel.com>
> 
> In case of restore, the MDT has the right size and has already sent
> it back without granting the layout lock, inode is up-to-date. Then
> AGL (async glimpse lock) is useless.
> 
> Also to glimpse we need the layout, in case of a running restore the
> MDT holds the layout lock so the glimpse will block up to the end of
> restore (statahead/agl will block).

Reviewed-by: James Simmons <jsimmons at infradead.org>
 
> Signed-off-by: Fan Yong <fan.yong at intel.com>
> WC-bug-id: https://jira.whamcloud.com/browse/LU-9319
> Reviewed-by: Lai Siyao <lai.siyao at intel.com>
> Reviewed-by: Jinshan Xiong <jinshan.xiong at intel.com>
> Reviewed-by: John L. Hammond <john.hammond at intel.com>
> Reviewed-on: https://review.whamcloud.com/26501
> Reviewed-by: Oleg Drokin <oleg.drokin at intel.com>
> Signed-off-by: NeilBrown <neilb at suse.com>
> ---
>  drivers/staging/lustre/lustre/llite/statahead.c |   13 +++++++++++++
>  1 file changed, 13 insertions(+)
> 
> diff --git a/drivers/staging/lustre/lustre/llite/statahead.c b/drivers/staging/lustre/lustre/llite/statahead.c
> index 28e85bfb9b82..3d71322aa1c7 100644
> --- a/drivers/staging/lustre/lustre/llite/statahead.c
> +++ b/drivers/staging/lustre/lustre/llite/statahead.c
> @@ -504,6 +504,19 @@ static void ll_agl_trigger(struct inode *inode, struct ll_statahead_info *sai)
>  		return;
>  	}
>  
> +	/* In case of restore, the MDT has the right size and has already
> +	 * sent it back without granting the layout lock, inode is up-to-date.
> +	 * Then AGL (async glimpse lock) is useless.
> +	 * Also to glimpse we need the layout, in case of a runninh restore
> +	 * the MDT holds the layout lock so the glimpse will block up to the
> +	 * end of restore (statahead/agl will block)
> +	 */
> +	if (test_bit(LLIF_FILE_RESTORING, &lli->lli_flags)) {
> +		lli->lli_agl_index = 0;
> +		iput(inode);
> +		return;
> +	}
> +
>  	/* Someone is in glimpse (sync or async), do nothing. */
>  	rc = down_write_trylock(&lli->lli_glimpse_sem);
>  	if (rc == 0) {
> 
> 
> 

From jsimmons at infradead.org  Mon Nov 26 02:10:26 2018
From: jsimmons at infradead.org (James Simmons)
Date: Mon, 26 Nov 2018 02:10:26 +0000 (GMT)
Subject: [lustre-devel] [PATCH 9/9] lustre: statahead: add smp_mb() to
 serialize ops
In-Reply-To: <154295732811.2850.6059970126126914084.stgit@noble>
References: <154295730810.2850.961218355189474016.stgit@noble>
 <154295732811.2850.6059970126126914084.stgit@noble>
Message-ID: <alpine.LFD.2.21.1811260210170.32255@casper.infradead.org>


> From: Lai Siyao <lai.siyao at intel.com>
> 
> In ll_deauthorize_statahead(), it set thread stop flag, and then
> wake up thread, however wakeup is called inside spinlock in case
> ll_statahead_info is released, then we need to call smp_mb() to
> serialize setting and wakeup.

Reviewed-by: James Simmons <jsimmons at infradead.org>
 
> Signed-off-by: Lai Siyao <lai.siyao at intel.com>
> WC-bug-id: https://jira.whamcloud.com/browse/LU-7994
> Reviewed-on: https://review.whamcloud.com/23040
> Reviewed-by: Fan Yong <fan.yong at intel.com>
> Reviewed-by: Bobi Jam <bobijam at hotmail.com>
> Reviewed-by: Oleg Drokin <oleg.drokin at intel.com>
> Signed-off-by: NeilBrown <neilb at suse.com>
> ---
>  drivers/staging/lustre/lustre/llite/statahead.c |   15 ++++++++++++---
>  1 file changed, 12 insertions(+), 3 deletions(-)
> 
> diff --git a/drivers/staging/lustre/lustre/llite/statahead.c b/drivers/staging/lustre/lustre/llite/statahead.c
> index 3d71322aa1c7..24c2335c70a7 100644
> --- a/drivers/staging/lustre/lustre/llite/statahead.c
> +++ b/drivers/staging/lustre/lustre/llite/statahead.c
> @@ -1110,8 +1110,9 @@ static int ll_statahead_thread(void *arg)
>  		sa_handle_callback(sai);
>  
>  		set_current_state(TASK_IDLE);
> +		/* ensure we see the NULL stored by ll_deauthorize_statahead() */
>  		if (!sa_has_callback(sai) &&
> -		    sai->sai_task)
> +		    smp_load_acquire(&sai->sai_task))
>  			schedule();
>  		__set_current_state(TASK_RUNNING);
>  	}
> @@ -1191,9 +1192,17 @@ void ll_deauthorize_statahead(struct inode *dir, void *key)
>  		/*
>  		 * statahead thread may not quit yet because it needs to cache
>  		 * entries, now it's time to tell it to quit.
> +		 *
> +		 * In case sai is released, wake_up() is called inside spinlock,
> +		 * so we use smp_store_release() to serialize ops.
>  		 */
> -		wake_up_process(sai->sai_task);
> -		sai->sai_task = NULL;
> +		struct task_struct *task = sai->sai_task;
> +
> +		/* ensure ll_statahead_thread sees the NULL before
> +		 * calling schedule() again.
> +		 */
> +		smp_store_release(&sai->sai_task, NULL);
> +		wake_up_process(task);
>  	}
>  	spin_unlock(&lli->lli_sa_lock);
>  }
> 
> 
> 

From jsimmons at infradead.org  Mon Nov 26 02:48:16 2018
From: jsimmons at infradead.org (James Simmons)
Date: Sun, 25 Nov 2018 21:48:16 -0500
Subject: [lustre-devel] [PATCH 00/12] lustre: new patches to address
 previous reviews
Message-ID: <1543200508-6838-1-git-send-email-jsimmons@infradead.org>

New patches from myself to address issues with the last batch I sent.
Found a bug in the llite debugfs stats registeration code while
testing. Updated the TODO since many of the task have been done.
The patch for LU-5461 has been ported which makes sanity test 244 to
pass. The patch ported from LU-10218 now makes it possible to run
the sanity 160 changelog test without locking up the client node.
Several patches independent of PFL have been ported from lustre
2.10 LTS release.

Alexander Boyko (1):
  lustre: obdclass: obd_device improvement

Dmitry Eremin (1):
  lustre: clio: Introduce parallel tasks framework

James Simmons (6):
  lustre: llite: move CONFIG_SECURITY handling to llite_internal.h
  lustre: lnd: create enum kib_dev_caps
  lustre: lnd: test fpo_fmr_pool pointer instead of special bool
  lustre: llite: remove llite_loop left overs
  lustre: llite: avoid duplicate stats debugfs registration
  lustre: update TODO lustre list

John L. Hammond (2):
  lustre: mdc: propagate changelog errors to readers
  lustre: mdc: use large xattr buffers for old servers

Lai Siyao (1):
  lustre: mdc: don't add to page cache upon failure

Patrick Farrell (1):
  lustre: ldlm: No -EINVAL for canceled != unused

 drivers/staging/lustre/TODO                        |  85 ----
 .../staging/lustre/lnet/klnds/o2iblnd/o2iblnd.c    |  17 +-
 .../staging/lustre/lnet/klnds/o2iblnd/o2iblnd.h    |  13 +-
 drivers/staging/lustre/lustre/include/cl_ptask.h   | 145 ++++++
 drivers/staging/lustre/lustre/include/obd_class.h  |   9 +-
 drivers/staging/lustre/lustre/ldlm/ldlm_resource.c |  16 +-
 drivers/staging/lustre/lustre/llite/file.c         | 107 +----
 .../staging/lustre/lustre/llite/llite_internal.h   |  81 +---
 drivers/staging/lustre/lustre/llite/llite_lib.c    |  11 +-
 drivers/staging/lustre/lustre/llite/lproc_llite.c  |   2 +-
 drivers/staging/lustre/lustre/llite/rw26.c         |  51 +--
 drivers/staging/lustre/lustre/mdc/mdc_changelog.c  |  45 +-
 drivers/staging/lustre/lustre/mdc/mdc_locks.c      |  14 +
 drivers/staging/lustre/lustre/mdc/mdc_request.c    |   5 +-
 drivers/staging/lustre/lustre/obdclass/Makefile    |   3 +-
 drivers/staging/lustre/lustre/obdclass/cl_ptask.c  | 501 +++++++++++++++++++++
 drivers/staging/lustre/lustre/obdclass/genops.c    | 284 ++++++++----
 .../staging/lustre/lustre/obdclass/obd_config.c    | 143 ++----
 drivers/staging/lustre/lustre/obdclass/obd_mount.c |   6 +-
 19 files changed, 1007 insertions(+), 531 deletions(-)
 create mode 100644 drivers/staging/lustre/lustre/include/cl_ptask.h
 create mode 100644 drivers/staging/lustre/lustre/obdclass/cl_ptask.c

-- 
1.8.3.1


From jsimmons at infradead.org  Mon Nov 26 02:48:19 2018
From: jsimmons at infradead.org (James Simmons)
Date: Sun, 25 Nov 2018 21:48:19 -0500
Subject: [lustre-devel] [PATCH 03/12] lustre: lnd: test fpo_fmr_pool pointer
 instead of special bool
In-Reply-To: <1543200508-6838-1-git-send-email-jsimmons@infradead.org>
References: <1543200508-6838-1-git-send-email-jsimmons@infradead.org>
Message-ID: <1543200508-6838-4-git-send-email-jsimmons@infradead.org>

For the ko2iblnd driver it sets a fpo_is_fmr bool to tell use if
a pool was allocated. The name fpo_is_fmr is very misleading to
its function and its a weak test to tell us if a pool was allocated
in the FMR case. It is much easier to test the actually FMR pool
pointer then manually setting a bool flag to tell us if the FMR
pool is valide.

Signed-off-by: James Simmons <uja.ornl at yahoo.com>
WC-bug-id: https://jira.whamcloud.com/browse/LU-11152
Reviewed-on: https://review.whamcloud.com/33408
Reviewed-by: Amir Shehata <ashehata at whamcloud.com>
Reviewed-by: Sonia Sharma <sharmaso at whamcloud.com>
Reviewed-by: Oleg Drokin <green at whamcloud.com>
Signed-off-by: James Simmons <jsimmons at infradead.org>
---
 drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.c | 12 ++++--------
 drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.h |  1 -
 2 files changed, 4 insertions(+), 9 deletions(-)

diff --git a/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.c b/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.c
index 281004a..5394c1a 100644
--- a/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.c
+++ b/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.c
@@ -1356,9 +1356,8 @@ static void kiblnd_destroy_fmr_pool(struct kib_fmr_pool *fpo)
 {
 	LASSERT(!fpo->fpo_map_count);
 
-	if (fpo->fpo_is_fmr) {
-		if (fpo->fmr.fpo_fmr_pool)
-			ib_destroy_fmr_pool(fpo->fmr.fpo_fmr_pool);
+	if (!IS_ERR_OR_NULL(fpo->fmr.fpo_fmr_pool)) {
+		ib_destroy_fmr_pool(fpo->fmr.fpo_fmr_pool);
 	} else {
 		struct kib_fast_reg_descriptor *frd;
 		int i = 0;
@@ -1435,7 +1434,6 @@ static int kiblnd_alloc_fmr_pool(struct kib_fmr_poolset *fps, struct kib_fmr_poo
 		else
 			CERROR("FMRs are not supported\n");
 	}
-	fpo->fpo_is_fmr = true;
 
 	return rc;
 }
@@ -1447,8 +1445,6 @@ static int kiblnd_alloc_freg_pool(struct kib_fmr_poolset *fps,
 	struct kib_fast_reg_descriptor *frd;
 	int i, rc;
 
-	fpo->fpo_is_fmr = false;
-
 	INIT_LIST_HEAD(&fpo->fast_reg.fpo_pool_list);
 	fpo->fast_reg.fpo_pool_size = 0;
 	for (i = 0; i < fps->fps_pool_size; i++) {
@@ -1646,7 +1642,7 @@ void kiblnd_fmr_pool_unmap(struct kib_fmr *fmr, int status)
 		return;
 
 	fps = fpo->fpo_owner;
-	if (fpo->fpo_is_fmr) {
+	if (!IS_ERR_OR_NULL(fpo->fmr.fpo_fmr_pool)) {
 		if (fmr->fmr_pfmr) {
 			rc = ib_fmr_pool_unmap(fmr->fmr_pfmr);
 			LASSERT(!rc);
@@ -1708,7 +1704,7 @@ int kiblnd_fmr_pool_map(struct kib_fmr_poolset *fps, struct kib_tx *tx,
 		fpo->fpo_deadline = ktime_get_seconds() + IBLND_POOL_DEADLINE;
 		fpo->fpo_map_count++;
 
-		if (fpo->fpo_is_fmr) {
+		if (!IS_ERR_OR_NULL(fpo->fmr.fpo_fmr_pool)) {
 			struct ib_pool_fmr *pfmr;
 
 			spin_unlock(&fps->fps_lock);
diff --git a/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.h b/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.h
index 0994fae..2ddd83b 100644
--- a/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.h
+++ b/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.h
@@ -288,7 +288,6 @@ struct kib_fmr_pool {
 	time64_t		fpo_deadline;	/* deadline of this pool */
 	int                   fpo_failed;          /* fmr pool is failed */
 	int                   fpo_map_count;       /* # of mapped FMR */
-	bool			fpo_is_fmr;	/* True if FMR pools allocated */
 };
 
 struct kib_fmr {
-- 
1.8.3.1


From jsimmons at infradead.org  Mon Nov 26 02:48:21 2018
From: jsimmons at infradead.org (James Simmons)
Date: Sun, 25 Nov 2018 21:48:21 -0500
Subject: [lustre-devel] [PATCH 05/12] lustre: llite: avoid duplicate stats
 debugfs registration
In-Reply-To: <1543200508-6838-1-git-send-email-jsimmons@infradead.org>
References: <1543200508-6838-1-git-send-email-jsimmons@infradead.org>
Message-ID: <1543200508-6838-6-git-send-email-jsimmons@infradead.org>

The unwinding of debugfs handling in llite introduced a bug. Two
different debugfs files are currently being registered with the
same name "stats". Change the registration of the ll_ra_stats
debugfs file to its proper name "read_ahead_stats".

Fixes: cd514eac8029 ("staging: lustre: remove ldebugfs_register_stats() wrapper function")
Signed-off-by: James Simmons <jsimmons at infradead.org>
---
 drivers/staging/lustre/lustre/llite/lproc_llite.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/staging/lustre/lustre/llite/lproc_llite.c b/drivers/staging/lustre/lustre/llite/lproc_llite.c
index 10dc7a8..8139f84 100644
--- a/drivers/staging/lustre/lustre/llite/lproc_llite.c
+++ b/drivers/staging/lustre/lustre/llite/lproc_llite.c
@@ -1381,7 +1381,7 @@ int ll_debugfs_register_super(struct super_block *sb, const char *name)
 		lprocfs_counter_init(sbi->ll_ra_stats, id, 0,
 				     ra_stat_string[id], "pages");
 
-	debugfs_create_file("stats", 0644, sbi->ll_debugfs_entry,
+	debugfs_create_file("read_ahead_stats", 0644, sbi->ll_debugfs_entry,
 			    sbi->ll_ra_stats, &lprocfs_stats_seq_fops);
 out_ll_kset:
 	/* Yes we also register sysfs mount kset here as well */
-- 
1.8.3.1


From jsimmons at infradead.org  Mon Nov 26 02:48:17 2018
From: jsimmons at infradead.org (James Simmons)
Date: Sun, 25 Nov 2018 21:48:17 -0500
Subject: [lustre-devel] [PATCH 01/12] lustre: llite: move CONFIG_SECURITY
 handling to llite_internal.h
In-Reply-To: <1543200508-6838-1-git-send-email-jsimmons@infradead.org>
References: <1543200508-6838-1-git-send-email-jsimmons@infradead.org>
Message-ID: <1543200508-6838-2-git-send-email-jsimmons@infradead.org>

For the linux kernel its recommended to keep CONFIG_* wrapped code
in a header file instead of the source files to avoid making the
code more difficulty to read. Move CONFIG_SECURITY wrapped code
to llite_internal.h in this case.

Signed-off-by: James Simmons <uja.ornl at yahoo.com>
WC-bug-id: https://jira.whamcloud.com/browse/LU-6142
Reviewed-on: https://review.whamcloud.com/33410
Reviewed-by: Alex Zhuravlev <bzzz at whamcloud.com>
Reviewed-by: Andreas Dilger <adilger at whamcloud.com>
Reviewed-by: John L. Hammond <jhammond at whamcloud.com>
Signed-off-by: James Simmons <jsimmons at infradead.org>
---
 drivers/staging/lustre/lustre/llite/llite_internal.h | 17 +++++++++++++++++
 drivers/staging/lustre/lustre/llite/llite_lib.c      | 11 ++---------
 2 files changed, 19 insertions(+), 9 deletions(-)

diff --git a/drivers/staging/lustre/lustre/llite/llite_internal.h b/drivers/staging/lustre/lustre/llite/llite_internal.h
index a55b568..8c703e6 100644
--- a/drivers/staging/lustre/lustre/llite/llite_internal.h
+++ b/drivers/staging/lustre/lustre/llite/llite_internal.h
@@ -269,6 +269,23 @@ static inline void ll_layout_version_set(struct ll_inode_info *lli, __u32 gen)
 int ll_xattr_cache_get(struct inode *inode, const char *name,
 		       char *buffer, size_t size, __u64 valid);
 
+static inline bool obd_connect_has_secctx(struct obd_connect_data *data)
+{
+#ifdef CONFIG_SECURITY
+	return data->ocd_connect_flags & OBD_CONNECT_FLAGS2 &&
+	       data->ocd_connect_flags2 & OBD_CONNECT2_FILE_SECCTX;
+#else
+       return false;
+#endif
+}
+
+static inline void obd_connect_set_secctx(struct obd_connect_data *data)
+{
+#ifdef CONFIG_SECURITY
+	data->ocd_connect_flags2 |= OBD_CONNECT2_FILE_SECCTX;
+#endif
+}
+
 int ll_dentry_init_security(struct dentry *dentry, int mode, struct qstr *name,
 			    const char **secctx_name, void **secctx,
 			    u32 *secctx_size);
diff --git a/drivers/staging/lustre/lustre/llite/llite_lib.c b/drivers/staging/lustre/lustre/llite/llite_lib.c
index fac6584..2dfeab4 100644
--- a/drivers/staging/lustre/lustre/llite/llite_lib.c
+++ b/drivers/staging/lustre/lustre/llite/llite_lib.c
@@ -148,12 +148,6 @@ static void ll_free_sbi(struct super_block *sb)
 	kfree(sbi);
 }
 
-static inline int obd_connect_has_secctx(struct obd_connect_data *data)
-{
-	return data->ocd_connect_flags & OBD_CONNECT_FLAGS2 &&
-	       data->ocd_connect_flags2 & OBD_CONNECT2_FILE_SECCTX;
-}
-
 static int client_common_fill_super(struct super_block *sb, char *md, char *dt)
 {
 	struct inode *root = NULL;
@@ -244,9 +238,8 @@ static int client_common_fill_super(struct super_block *sb, char *md, char *dt)
 	if (sbi->ll_flags & LL_SBI_ALWAYS_PING)
 		data->ocd_connect_flags &= ~OBD_CONNECT_PINGLESS;
 
-#ifdef CONFIG_SECURITY
-	data->ocd_connect_flags2 |= OBD_CONNECT2_FILE_SECCTX;
-#endif
+	obd_connect_set_secctx(data);
+
 	data->ocd_brw_size = MD_MAX_BRW_SIZE;
 
 	err = obd_connect(NULL, &sbi->ll_md_exp, sbi->ll_md_obd,
-- 
1.8.3.1


From jsimmons at infradead.org  Mon Nov 26 02:48:22 2018
From: jsimmons at infradead.org (James Simmons)
Date: Sun, 25 Nov 2018 21:48:22 -0500
Subject: [lustre-devel] [PATCH 06/12] lustre: mdc: don't add to page cache
 upon failure
In-Reply-To: <1543200508-6838-1-git-send-email-jsimmons@infradead.org>
References: <1543200508-6838-1-git-send-email-jsimmons@infradead.org>
Message-ID: <1543200508-6838-7-git-send-email-jsimmons@infradead.org>

From: Lai Siyao <lai.siyao at whamcloud.com>

Reading directory pages may fail on MDS, in this case client should
not cache a non-up-to-date directory page, because it will cause
a later read on the same page fail.

Signed-off-by: Lai Siyao <lai.siyao at whamcloud.com>
WC-bug-id: https://jira.whamcloud.com/browse/LU-5461
Reviewed-on: http://review.whamcloud.com/11450
Reviewed-by: Fan Yong <fan.yong at intel.com>
Reviewed-by: John L. Hammond <jhammond at whamcloud.com>
Reviewed-by: Oleg Drokin <green at whamcloud.com>
Signed-off-by: James Simmons <jsimmons at infradead.org>
---
 drivers/staging/lustre/lustre/mdc/mdc_request.c | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/drivers/staging/lustre/lustre/mdc/mdc_request.c b/drivers/staging/lustre/lustre/mdc/mdc_request.c
index 1072b66..09b30ef 100644
--- a/drivers/staging/lustre/lustre/mdc/mdc_request.c
+++ b/drivers/staging/lustre/lustre/mdc/mdc_request.c
@@ -1212,7 +1212,10 @@ static int mdc_read_page_remote(void *data, struct page *page0)
 	}
 
 	rc = mdc_getpage(rp->rp_exp, fid, rp->rp_off, page_pool, npages, &req);
-	if (!rc) {
+	if (rc < 0) {
+		/* page0 is special, which was added into page cache early */
+		delete_from_page_cache(page0);
+	} else {
 		int lu_pgs = req->rq_bulk->bd_nob_transferred;
 
 		rd_pgs = (lu_pgs + PAGE_SIZE - 1) >> PAGE_SHIFT;
-- 
1.8.3.1


From jsimmons at infradead.org  Mon Nov 26 02:48:28 2018
From: jsimmons at infradead.org (James Simmons)
Date: Sun, 25 Nov 2018 21:48:28 -0500
Subject: [lustre-devel] [PATCH 12/12] lustre: update TODO lustre list
In-Reply-To: <1543200508-6838-1-git-send-email-jsimmons@infradead.org>
References: <1543200508-6838-1-git-send-email-jsimmons@infradead.org>
Message-ID: <1543200508-6838-13-git-send-email-jsimmons@infradead.org>

Many of the items in TODO have been completed so remove them from
the list.

Signed-off-by: James Simmons <jsimmons at infradead.org>
---
 drivers/staging/lustre/TODO | 85 ---------------------------------------------
 1 file changed, 85 deletions(-)

diff --git a/drivers/staging/lustre/TODO b/drivers/staging/lustre/TODO
index 942280b..1df7697 100644
--- a/drivers/staging/lustre/TODO
+++ b/drivers/staging/lustre/TODO
@@ -35,91 +35,6 @@ So move to time64_t and ktime_t as much as possible.
 ******************************************************************************
 * Proper IB support for ko2iblnd
 ******************************************************************************
-https://jira.hpdd.intel.com/browse/LU-9179
-
-Poor performance for the ko2iblnd driver. This is related to many of the
-patches below that are missing from the linux client.
-------------------------------------------------------------------------------
-
-https://jira.hpdd.intel.com/browse/LU-10394 / LU-10526 / LU-10089
-
-Default to using MEM_REG
-------------------------------------------------------------------------------
-
-https://jira.hpdd.intel.com/browse/LU-10459
-
-throttle tx based on queue depth
-------------------------------------------------------------------------------
-
-https://jira.hpdd.intel.com/browse/LU-9943
-
-correct WR fast reg accounting
-------------------------------------------------------------------------------
-
-https://jira.hpdd.intel.com/browse/LU-10291
-
-remove concurrent_sends tunable
-------------------------------------------------------------------------------
-
-https://jira.hpdd.intel.com/browse/LU-10213
-
-calculate qp max_send_wrs properly
-------------------------------------------------------------------------------
-
-https://jira.hpdd.intel.com/browse/LU-9810
-
-use less CQ entries for each connection
-------------------------------------------------------------------------------
-
-https://jira.hpdd.intel.com/browse/LU-10129 / LU-9180
-
-rework map_on_demand behavior
-------------------------------------------------------------------------------
-
-https://jira.hpdd.intel.com/browse/LU-10129
-
-query device capabilities
-------------------------------------------------------------------------------
-
-https://jira.hpdd.intel.com/browse/LU-9983
-
-allow for discontiguous fragments
-------------------------------------------------------------------------------
-
-https://jira.hpdd.intel.com/browse/LU-9500
-
-Don't Page Align remote_addr with FastReg
-------------------------------------------------------------------------------
-
-https://jira.hpdd.intel.com/browse/LU-9448
-
-handle empty CPTs
-------------------------------------------------------------------------------
-
-https://jira.hpdd.intel.com/browse/LU-9507
-
-Don't Assert On Reconnect with MultiQP
-------------------------------------------------------------------------------
-
-https://jira.hpdd.intel.com/browse/LU-9425
-
-Turn on 2 sges by default
-------------------------------------------------------------------------------
-
-https://jira.hpdd.intel.com/browse/LU-5718
-
-multiple sges for work request
-------------------------------------------------------------------------------
-
-https://jira.hpdd.intel.com/browse/LU-9094
-
-kill timedout txs from ibp_tx_queue
-------------------------------------------------------------------------------
-
-https://jira.hpdd.intel.com/browse/LU-9094
-
-reconnect peer for REJ_INVALID_SERVICE_ID
-------------------------------------------------------------------------------
 
 https://jira.hpdd.intel.com/browse/LU-8752
 
-- 
1.8.3.1


From jsimmons at infradead.org  Mon Nov 26 02:48:18 2018
From: jsimmons at infradead.org (James Simmons)
Date: Sun, 25 Nov 2018 21:48:18 -0500
Subject: [lustre-devel] [PATCH 02/12] lustre: lnd: create enum kib_dev_caps
In-Reply-To: <1543200508-6838-1-git-send-email-jsimmons@infradead.org>
References: <1543200508-6838-1-git-send-email-jsimmons@infradead.org>
Message-ID: <1543200508-6838-3-git-send-email-jsimmons@infradead.org>

Cleanup IBLND_DEV_CAPS_* by creating enum kib_dev_caps and using
the BIT() macros.

Signed-off-by: James Simmons <uja.ornl at yahoo.com>
WC-bug-id: https://jira.whamcloud.com/browse/LU-6142
Reviewed-on: https://review.whamcloud.com/33409
Reviewed-by: Amir Shehata <ashehata at whamcloud.com>
Reviewed-by: Doug Oucharek <dougso at me.com>
Reviewed-by: Oleg Drokin <green at whamcloud.com>
Signed-off-by: James Simmons <jsimmons at infradead.org>
---
 drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.c |  5 +++--
 drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.h | 12 +++++++-----
 2 files changed, 10 insertions(+), 7 deletions(-)

diff --git a/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.c b/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.c
index ecdf4de..281004a 100644
--- a/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.c
+++ b/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.c
@@ -656,7 +656,7 @@ static unsigned int kiblnd_send_wrs(struct kib_conn *conn)
 	 * One WR for the LNet message
 	 * And ibc_max_frags for the transfer WRs
 	 */
-	u32 dev_caps = conn->ibc_hdev->ibh_dev->ibd_dev_caps;
+	enum kib_dev_caps dev_caps = conn->ibc_hdev->ibh_dev->ibd_dev_caps;
 	unsigned int ret = 1 + conn->ibc_max_frags;
 
 	/* FastReg needs two extra WRs for map and invalidate */
@@ -1441,7 +1441,8 @@ static int kiblnd_alloc_fmr_pool(struct kib_fmr_poolset *fps, struct kib_fmr_poo
 }
 
 static int kiblnd_alloc_freg_pool(struct kib_fmr_poolset *fps,
-				  struct kib_fmr_pool *fpo, u32 dev_caps)
+				  struct kib_fmr_pool *fpo,
+				  enum kib_dev_caps dev_caps)
 {
 	struct kib_fast_reg_descriptor *frd;
 	int i, rc;
diff --git a/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.h b/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.h
index 7f81fe2..0994fae 100644
--- a/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.h
+++ b/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.h
@@ -73,10 +73,6 @@
 #define IBLND_N_SCHED			2
 #define IBLND_N_SCHED_HIGH		4
 
-#define IBLND_DEV_CAPS_FASTREG_ENABLED		0x1
-#define IBLND_DEV_CAPS_FASTREG_GAPS_SUPPORT	0x2
-#define IBLND_DEV_CAPS_FMR_ENABLED		0x4
-
 struct kib_tunables {
 	int *kib_dev_failover;           /* HCA failover */
 	unsigned int *kib_service;       /* IB service number */
@@ -152,6 +148,12 @@ struct kib_tunables {
 #define KIB_IFNAME_SIZE	      256
 #endif
 
+enum kib_dev_caps {
+	IBLND_DEV_CAPS_FASTREG_ENABLED		= BIT(0),
+	IBLND_DEV_CAPS_FASTREG_GAPS_SUPPORT	= BIT(1),
+	IBLND_DEV_CAPS_FMR_ENABLED		= BIT(2),
+};
+
 struct kib_dev {
 	struct list_head   ibd_list;            /* chain on kib_devs */
 	struct list_head   ibd_fail_list;       /* chain on kib_failed_devs */
@@ -167,7 +169,7 @@ struct kib_dev {
 	unsigned int ibd_can_failover; /* IPoIB interface is a bonding master */
 	struct list_head   ibd_nets;
 	struct kib_hca_dev *ibd_hdev;
-	u32			ibd_dev_caps;
+	enum kib_dev_caps	ibd_dev_caps;
 };
 
 struct kib_hca_dev {
-- 
1.8.3.1


From jsimmons at infradead.org  Mon Nov 26 02:48:20 2018
From: jsimmons at infradead.org (James Simmons)
Date: Sun, 25 Nov 2018 21:48:20 -0500
Subject: [lustre-devel] [PATCH 04/12] lustre: llite: remove llite_loop left
 overs
In-Reply-To: <1543200508-6838-1-git-send-email-jsimmons@infradead.org>
References: <1543200508-6838-1-git-send-email-jsimmons@infradead.org>
Message-ID: <1543200508-6838-5-git-send-email-jsimmons@infradead.org>

With the removal of llite_loop several pieces of code are still
present in the llite layer that were only used by the lloop device.
We can remove these no longer used pieces.

Signed-off-by: James Simmons <uja.ornl at yahoo.com>
WC-bug-id: https://jira.whamcloud.com/browse/LU-8958
Reviewed-on: https://review.whamcloud.com/26795
Reviewed-by: Dmitry Eremin <dmitry.eremin at intel.com>
Reviewed-by: John L. Hammond <jhammond at whamcloud.com>
Reviewed-by: Oleg Drokin <green at whamcloud.com>
Signed-off-by: James Simmons <jsimmons at infradead.org>
---
 drivers/staging/lustre/lustre/llite/file.c         | 107 +--------------------
 .../staging/lustre/lustre/llite/llite_internal.h   |  64 ------------
 drivers/staging/lustre/lustre/llite/rw26.c         |  51 +++-------
 3 files changed, 15 insertions(+), 207 deletions(-)

diff --git a/drivers/staging/lustre/lustre/llite/file.c b/drivers/staging/lustre/lustre/llite/file.c
index 68f88cf..15910ff 100644
--- a/drivers/staging/lustre/lustre/llite/file.c
+++ b/drivers/staging/lustre/lustre/llite/file.c
@@ -56,10 +56,6 @@
 static int ll_lease_close(struct obd_client_handle *och, struct inode *inode,
 			  bool *lease_broken);
 
-static enum llioc_iter
-ll_iocontrol_call(struct inode *inode, struct file *file,
-		  unsigned int cmd, unsigned long arg, int *rcp);
-
 static struct ll_file_data *ll_file_data_get(void)
 {
 	struct ll_file_data *fd;
@@ -2620,17 +2616,10 @@ int ll_ioctl_fssetxattr(struct inode *inode, unsigned int cmd,
 		return ll_ioctl_fssetxattr(inode, cmd, arg);
 	case BLKSSZGET:
 		return put_user(PAGE_SIZE, (int __user *)arg);
-	default: {
-		int err;
-
-		if (ll_iocontrol_call(inode, file, cmd, arg, &err) ==
-		     LLIOC_STOP)
-			return err;
-
+	default:
 		return obd_iocontrol(cmd, ll_i2dtexp(inode), 0, NULL,
 				     (void __user *)arg);
 	}
-	}
 }
 
 static loff_t ll_file_seek(struct file *file, loff_t offset, int origin)
@@ -3543,100 +3532,6 @@ int ll_inode_permission(struct inode *inode, int mask)
 	.get_acl	= ll_get_acl,
 };
 
-/* dynamic ioctl number support routines */
-static struct llioc_ctl_data {
-	struct rw_semaphore	ioc_sem;
-	struct list_head	      ioc_head;
-} llioc = {
-	__RWSEM_INITIALIZER(llioc.ioc_sem),
-	LIST_HEAD_INIT(llioc.ioc_head)
-};
-
-struct llioc_data {
-	struct list_head	      iocd_list;
-	unsigned int	    iocd_size;
-	llioc_callback_t	iocd_cb;
-	unsigned int	    iocd_count;
-	unsigned int	    iocd_cmd[0];
-};
-
-void *ll_iocontrol_register(llioc_callback_t cb, int count, unsigned int *cmd)
-{
-	unsigned int size;
-	struct llioc_data *in_data = NULL;
-
-	if (!cb || !cmd || count > LLIOC_MAX_CMD || count < 0)
-		return NULL;
-
-	size = sizeof(*in_data) + count * sizeof(unsigned int);
-	in_data = kzalloc(size, GFP_KERNEL);
-	if (!in_data)
-		return NULL;
-
-	in_data->iocd_size = size;
-	in_data->iocd_cb = cb;
-	in_data->iocd_count = count;
-	memcpy(in_data->iocd_cmd, cmd, sizeof(unsigned int) * count);
-
-	down_write(&llioc.ioc_sem);
-	list_add_tail(&in_data->iocd_list, &llioc.ioc_head);
-	up_write(&llioc.ioc_sem);
-
-	return in_data;
-}
-EXPORT_SYMBOL(ll_iocontrol_register);
-
-void ll_iocontrol_unregister(void *magic)
-{
-	struct llioc_data *tmp;
-
-	if (!magic)
-		return;
-
-	down_write(&llioc.ioc_sem);
-	list_for_each_entry(tmp, &llioc.ioc_head, iocd_list) {
-		if (tmp == magic) {
-			list_del(&tmp->iocd_list);
-			up_write(&llioc.ioc_sem);
-
-			kfree(tmp);
-			return;
-		}
-	}
-	up_write(&llioc.ioc_sem);
-
-	CWARN("didn't find iocontrol register block with magic: %p\n", magic);
-}
-EXPORT_SYMBOL(ll_iocontrol_unregister);
-
-static enum llioc_iter
-ll_iocontrol_call(struct inode *inode, struct file *file,
-		  unsigned int cmd, unsigned long arg, int *rcp)
-{
-	enum llioc_iter ret = LLIOC_CONT;
-	struct llioc_data *data;
-	int rc = -EINVAL, i;
-
-	down_read(&llioc.ioc_sem);
-	list_for_each_entry(data, &llioc.ioc_head, iocd_list) {
-		for (i = 0; i < data->iocd_count; i++) {
-			if (cmd != data->iocd_cmd[i])
-				continue;
-
-			ret = data->iocd_cb(inode, file, cmd, arg, data, &rc);
-			break;
-		}
-
-		if (ret == LLIOC_STOP)
-			break;
-	}
-	up_read(&llioc.ioc_sem);
-
-	if (rcp)
-		*rcp = rc;
-	return ret;
-}
-
 int ll_layout_conf(struct inode *inode, const struct cl_object_conf *conf)
 {
 	struct ll_inode_info *lli = ll_i2info(inode);
diff --git a/drivers/staging/lustre/lustre/llite/llite_internal.h b/drivers/staging/lustre/lustre/llite/llite_internal.h
index 8c703e6..48424a4 100644
--- a/drivers/staging/lustre/lustre/llite/llite_internal.h
+++ b/drivers/staging/lustre/lustre/llite/llite_internal.h
@@ -1234,73 +1234,9 @@ static inline int ll_glimpse_size(struct inode *inode)
 	return true;
 }
 
-/* llite ioctl register support routine */
-enum llioc_iter {
-	LLIOC_CONT = 0,
-	LLIOC_STOP
-};
-
-#define LLIOC_MAX_CMD	   256
-
-/*
- * Rules to write a callback function:
- *
- * Parameters:
- *  @magic: Dynamic ioctl call routine will feed this value with the pointer
- *      returned to ll_iocontrol_register.  Callback functions should use this
- *      data to check the potential collasion of ioctl cmd. If collasion is
- *      found, callback function should return LLIOC_CONT.
- *  @rcp: The result of ioctl command.
- *
- *  Return values:
- *      If @magic matches the pointer returned by ll_iocontrol_data, the
- *      callback should return LLIOC_STOP; return LLIOC_STOP otherwise.
- */
-typedef enum llioc_iter (*llioc_callback_t)(struct inode *inode,
-		struct file *file, unsigned int cmd, unsigned long arg,
-		void *magic, int *rcp);
-
-/* export functions */
-/* Register ioctl block dynamatically for a regular file.
- *
- * @cmd: the array of ioctl command set
- * @count: number of commands in the @cmd
- * @cb: callback function, it will be called if an ioctl command is found to
- *      belong to the command list @cmd.
- *
- * Return value:
- *      A magic pointer will be returned if success;
- *      otherwise, NULL will be returned.
- */
-void *ll_iocontrol_register(llioc_callback_t cb, int count, unsigned int *cmd);
-void ll_iocontrol_unregister(void *magic);
-
 int cl_sync_file_range(struct inode *inode, loff_t start, loff_t end,
 		       enum cl_fsync_mode mode, int ignore_layout);
 
-/** direct write pages */
-struct ll_dio_pages {
-	/** page array to be written. we don't support
-	 * partial pages except the last one.
-	 */
-	struct page **ldp_pages;
-	/* offset of each page */
-	loff_t       *ldp_offsets;
-	/** if ldp_offsets is NULL, it means a sequential
-	 * pages to be written, then this is the file offset
-	 * of the first page.
-	 */
-	loff_t	ldp_start_offset;
-	/** how many bytes are to be written. */
-	size_t	ldp_size;
-	/** # of pages in the array. */
-	int	   ldp_nr;
-};
-
-ssize_t ll_direct_rw_pages(const struct lu_env *env, struct cl_io *io,
-			   int rw, struct inode *inode,
-			   struct ll_dio_pages *pv);
-
 static inline int ll_file_nolock(const struct file *file)
 {
 	struct ll_file_data *fd = LUSTRE_FPRIVATE(file);
diff --git a/drivers/staging/lustre/lustre/llite/rw26.c b/drivers/staging/lustre/lustre/llite/rw26.c
index 722e5ea..9843c9e 100644
--- a/drivers/staging/lustre/lustre/llite/rw26.c
+++ b/drivers/staging/lustre/lustre/llite/rw26.c
@@ -172,32 +172,27 @@ static void ll_free_user_pages(struct page **pages, int npages, int do_dirty)
 	kvfree(pages);
 }
 
-ssize_t ll_direct_rw_pages(const struct lu_env *env, struct cl_io *io,
-			   int rw, struct inode *inode,
-			   struct ll_dio_pages *pv)
+static ssize_t ll_direct_IO_seg(const struct lu_env *env, struct cl_io *io,
+				int rw, struct inode *inode, size_t size,
+				loff_t file_offset, struct page **pages,
+				int page_count)
 {
 	struct cl_page    *clp;
 	struct cl_2queue  *queue;
 	struct cl_object  *obj = io->ci_obj;
 	int i;
 	ssize_t rc = 0;
-	loff_t file_offset  = pv->ldp_start_offset;
-	size_t size = pv->ldp_size;
-	int page_count      = pv->ldp_nr;
-	struct page **pages = pv->ldp_pages;
 	size_t page_size = cl_page_size(obj);
+	size_t orig_size = size;
 	bool do_io;
-	int  io_pages       = 0;
+	int io_pages = 0;
 
 	queue = &io->ci_queue;
 	cl_2queue_init(queue);
 	for (i = 0; i < page_count; i++) {
-		if (pv->ldp_offsets)
-			file_offset = pv->ldp_offsets[i];
-
 		LASSERT(!(file_offset & (page_size - 1)));
 		clp = cl_page_find(env, obj, cl_index(obj, file_offset),
-				   pv->ldp_pages[i], CPT_TRANSIENT);
+				   pages[i], CPT_TRANSIENT);
 		if (IS_ERR(clp)) {
 			rc = PTR_ERR(clp);
 			break;
@@ -274,31 +269,13 @@ ssize_t ll_direct_rw_pages(const struct lu_env *env, struct cl_io *io,
 				       queue, 0);
 	}
 	if (rc == 0)
-		rc = pv->ldp_size;
+		rc = orig_size;
 
 	cl_2queue_discard(env, io, queue);
 	cl_2queue_disown(env, io, queue);
 	cl_2queue_fini(env, queue);
 	return rc;
 }
-EXPORT_SYMBOL(ll_direct_rw_pages);
-
-static ssize_t ll_direct_IO_26_seg(const struct lu_env *env, struct cl_io *io,
-				   int rw, struct inode *inode,
-				   struct address_space *mapping,
-				   size_t size, loff_t file_offset,
-				   struct page **pages, int page_count)
-{
-	struct ll_dio_pages pvec = {
-		.ldp_pages	= pages,
-		.ldp_nr		= page_count,
-		.ldp_size	= size,
-		.ldp_offsets	= NULL,
-		.ldp_start_offset = file_offset
-	};
-
-	return ll_direct_rw_pages(env, io, rw, inode, &pvec);
-}
 
 /* This is the maximum size of a single O_DIRECT request, based on the
  * kmalloc limit.  We need to fit all of the brw_page structs, each one
@@ -308,7 +285,8 @@ static ssize_t ll_direct_IO_26_seg(const struct lu_env *env, struct cl_io *io,
  */
 #define MAX_DIO_SIZE ((KMALLOC_MAX_SIZE / sizeof(struct brw_page) *	  \
 		       PAGE_SIZE) & ~(DT_MAX_BRW_SIZE - 1))
-static ssize_t ll_direct_IO_26(struct kiocb *iocb, struct iov_iter *iter)
+
+static ssize_t ll_direct_IO(struct kiocb *iocb, struct iov_iter *iter)
 {
 	struct ll_cl_context *lcc;
 	const struct lu_env *env;
@@ -362,10 +340,9 @@ static ssize_t ll_direct_IO_26(struct kiocb *iocb, struct iov_iter *iter)
 		if (likely(result > 0)) {
 			int n = DIV_ROUND_UP(result + offs, PAGE_SIZE);
 
-			result = ll_direct_IO_26_seg(env, io, iov_iter_rw(iter),
-						     inode, file->f_mapping,
-						     result, file_offset, pages,
-						     n);
+			result = ll_direct_IO_seg(env, io, iov_iter_rw(iter),
+						  inode, result, file_offset,
+						  pages, n);
 			ll_free_user_pages(pages, n, iov_iter_rw(iter) == READ);
 		}
 		if (unlikely(result <= 0)) {
@@ -627,7 +604,7 @@ static int ll_migratepage(struct address_space *mapping,
 
 const struct address_space_operations ll_aops = {
 	.readpage	= ll_readpage,
-	.direct_IO      = ll_direct_IO_26,
+	.direct_IO      = ll_direct_IO,
 	.writepage      = ll_writepage,
 	.writepages     = ll_writepages,
 	.set_page_dirty = __set_page_dirty_nobuffers,
-- 
1.8.3.1


From jsimmons at infradead.org  Mon Nov 26 02:48:23 2018
From: jsimmons at infradead.org (James Simmons)
Date: Sun, 25 Nov 2018 21:48:23 -0500
Subject: [lustre-devel] [PATCH 07/12] lustre: ldlm: No -EINVAL for canceled
 != unused
In-Reply-To: <1543200508-6838-1-git-send-email-jsimmons@infradead.org>
References: <1543200508-6838-1-git-send-email-jsimmons@infradead.org>
Message-ID: <1543200508-6838-8-git-send-email-jsimmons@infradead.org>

From: Patrick Farrell <paf at cray.com>

If any locks are removed from or added to the lru, the
check of "number unused vs number cancelled" may be wrong.
This is fine - do not return an error or print debug in
this case.

Signed-off-by: Patrick Farrell <paf at cray.com>
WC-bug-id: https://jira.whamcloud.com/browse/LU-7802
Reviewed-on: https://review.whamcloud.com/28560
Reviewed-by: James Simmons <uja.ornl at yahoo.com>
Reviewed-by: Andreas Dilger <adilger at whamcloud.com>
Reviewed-by: Oleg Drokin <green at whamcloud.com>
Signed-off-by: James Simmons <jsimmons at infradead.org>
---
 drivers/staging/lustre/lustre/ldlm/ldlm_resource.c | 16 +++-------------
 1 file changed, 3 insertions(+), 13 deletions(-)

diff --git a/drivers/staging/lustre/lustre/ldlm/ldlm_resource.c b/drivers/staging/lustre/lustre/ldlm/ldlm_resource.c
index 5028db7..11c0b88 100644
--- a/drivers/staging/lustre/lustre/ldlm/ldlm_resource.c
+++ b/drivers/staging/lustre/lustre/ldlm/ldlm_resource.c
@@ -193,19 +193,9 @@ static ssize_t lru_size_store(struct kobject *kobj, struct attribute *attr,
 		       "dropping all unused locks from namespace %s\n",
 		       ldlm_ns_name(ns));
 		if (ns_connect_lru_resize(ns)) {
-			int canceled, unused  = ns->ns_nr_unused;
-
-			/* Try to cancel all @ns_nr_unused locks. */
-			canceled = ldlm_cancel_lru(ns, unused, 0,
-						   LDLM_LRU_FLAG_PASSED |
-						   LDLM_LRU_FLAG_CLEANUP);
-			if (canceled < unused) {
-				CDEBUG(D_DLMTRACE,
-				       "not all requested locks are canceled, requested: %d, canceled: %d\n",
-				       unused,
-				       canceled);
-				return -EINVAL;
-			}
+			ldlm_cancel_lru(ns, ns->ns_nr_unused, 0,
+					LDLM_LRU_FLAG_PASSED |
+					LDLM_LRU_FLAG_CLEANUP);
 		} else {
 			tmp = ns->ns_max_unused;
 			ns->ns_max_unused = 0;
-- 
1.8.3.1


From jsimmons at infradead.org  Mon Nov 26 02:48:24 2018
From: jsimmons at infradead.org (James Simmons)
Date: Sun, 25 Nov 2018 21:48:24 -0500
Subject: [lustre-devel] [PATCH 08/12] lustre: mdc: propagate changelog
 errors to readers
In-Reply-To: <1543200508-6838-1-git-send-email-jsimmons@infradead.org>
References: <1543200508-6838-1-git-send-email-jsimmons@infradead.org>
Message-ID: <1543200508-6838-9-git-send-email-jsimmons@infradead.org>

From: "John L. Hammond" <jhammond at whamcloud.com>

Store errors encountered by the changelog llog reader thread in the
crs_err field of struct changelog_reader_state so that they can be
propageted to changelog readers. In chlg_read() improve the error and
EOF reporting. Return -ERESTARTSYS when the blocked reader is
interrupted. Replace uses of wait_event_idle() with
ait_event_interruptible().

Signed-off-by: John L. Hammond <jhammond at whamcloud.com>
WC-bug-id: https://jira.whamcloud.com/browse/LU-10218
Reviewed-on: https://review.whamcloud.com/30040
Reviewed-by: Quentin Bouget <quentin.bouget at cea.fr>
Reviewed-by: Henri Doreau <henri.doreau at cea.fr>
Reviewed-by: Oleg Drokin <green at whamcloud.com>
Signed-off-by: James Simmons <jsimmons at infradead.org>
---
 drivers/staging/lustre/lustre/mdc/mdc_changelog.c | 45 ++++++++++++++---------
 1 file changed, 28 insertions(+), 17 deletions(-)

diff --git a/drivers/staging/lustre/lustre/mdc/mdc_changelog.c b/drivers/staging/lustre/lustre/mdc/mdc_changelog.c
index becdee8..811a36a 100644
--- a/drivers/staging/lustre/lustre/mdc/mdc_changelog.c
+++ b/drivers/staging/lustre/lustre/mdc/mdc_changelog.c
@@ -71,7 +71,7 @@ struct chlg_reader_state {
 	/* Producer thread (if any) */
 	struct task_struct	*crs_prod_task;
 	/* An error occurred that prevents from reading further */
-	bool			 crs_err;
+	int			 crs_err;
 	/* EOF, no more records available */
 	bool			 crs_eof;
 	/* Desired start position */
@@ -148,9 +148,9 @@ static int chlg_read_cat_process_cb(const struct lu_env *env,
 	       PFID(&rec->cr.cr_tfid), PFID(&rec->cr.cr_pfid),
 	       rec->cr.cr_namelen, changelog_rec_name(&rec->cr));
 
-	wait_event_idle(crs->crs_waitq_prod,
-			(crs->crs_rec_count < CDEV_CHLG_MAX_PREFETCH ||
-			 kthread_should_stop()));
+	wait_event_interruptible(crs->crs_waitq_prod,
+				 crs->crs_rec_count < CDEV_CHLG_MAX_PREFETCH ||
+				 kthread_should_stop());
 
 	if (kthread_should_stop())
 		return LLOG_PROC_BREAK;
@@ -231,7 +231,7 @@ static int chlg_load(void *args)
 
 err_out:
 	if (rc < 0)
-		crs->crs_err = true;
+		crs->crs_err = rc;
 
 	wake_up_all(&crs->crs_waitq_cons);
 
@@ -241,7 +241,7 @@ static int chlg_load(void *args)
 	if (ctx)
 		llog_ctxt_put(ctx);
 
-	wait_event_idle(crs->crs_waitq_prod, kthread_should_stop());
+	wait_event_interruptible(crs->crs_waitq_prod, kthread_should_stop());
 
 	return rc;
 }
@@ -264,13 +264,20 @@ static ssize_t chlg_read(struct file *file, char __user *buff, size_t count,
 	struct chlg_reader_state *crs = file->private_data;
 	struct chlg_rec_entry *rec;
 	struct chlg_rec_entry *tmp;
-	ssize_t  written_total = 0;
+	ssize_t written_total = 0;
 	LIST_HEAD(consumed);
+	size_t rc;
+
+	if (file->f_flags & O_NONBLOCK && crs->crs_rec_count == 0) {
+		if (crs->crs_err < 0)
+			return crs->crs_err;
+		else if (crs->crs_eof)
+			return 0;
+		else
+			return -EAGAIN;
+	}
 
-	if (file->f_flags & O_NONBLOCK && crs->crs_rec_count == 0)
-		return -EAGAIN;
-
-	wait_event_idle(crs->crs_waitq_cons,
+	rc = wait_event_interruptible(crs->crs_waitq_cons,
 			crs->crs_rec_count > 0 || crs->crs_eof || crs->crs_err);
 
 	mutex_lock(&crs->crs_lock);
@@ -279,8 +286,7 @@ static ssize_t chlg_read(struct file *file, char __user *buff, size_t count,
 			break;
 
 		if (copy_to_user(buff, rec->enq_record, rec->enq_length)) {
-			if (written_total == 0)
-				written_total = -EFAULT;
+			rc = -EFAULT;
 			break;
 		}
 
@@ -294,15 +300,19 @@ static ssize_t chlg_read(struct file *file, char __user *buff, size_t count,
 	}
 	mutex_unlock(&crs->crs_lock);
 
-	if (written_total > 0)
+	if (written_total > 0) {
+		rc = written_total;
 		wake_up_all(&crs->crs_waitq_prod);
+	} else if (rc == 0) {
+		rc = crs->crs_err;
+	}
 
 	list_for_each_entry_safe(rec, tmp, &consumed, enq_linkage)
 		enq_record_delete(rec);
 
 	*ppos = crs->crs_start_offset;
 
-	return written_total;
+	return rc;
 }
 
 /**
@@ -509,15 +519,16 @@ static int chlg_release(struct inode *inode, struct file *file)
 	struct chlg_reader_state *crs = file->private_data;
 	struct chlg_rec_entry *rec;
 	struct chlg_rec_entry *tmp;
+	int rc = 0;
 
 	if (crs->crs_prod_task)
-		kthread_stop(crs->crs_prod_task);
+		rc = kthread_stop(crs->crs_prod_task);
 
 	list_for_each_entry_safe(rec, tmp, &crs->crs_rec_queue, enq_linkage)
 		enq_record_delete(rec);
 
 	kfree(crs);
-	return 0;
+	return rc;
 }
 
 /**
-- 
1.8.3.1


From jsimmons at infradead.org  Mon Nov 26 02:48:25 2018
From: jsimmons at infradead.org (James Simmons)
Date: Sun, 25 Nov 2018 21:48:25 -0500
Subject: [lustre-devel] [PATCH 09/12] lustre: obdclass: obd_device
 improvement
In-Reply-To: <1543200508-6838-1-git-send-email-jsimmons@infradead.org>
References: <1543200508-6838-1-git-send-email-jsimmons@infradead.org>
Message-ID: <1543200508-6838-10-git-send-email-jsimmons@infradead.org>

From: Alexander Boyko <c17825 at cray.com>

The patch removes self exports from obd's reference counting which
allows to avoid freeing of self exports by zombie thread.
A pair of functions class_register_device()/class_unregister_device()
is to make sure that an obd can not be referenced again once its
refcount reached 0.

Signed-off-by: Vladimir Saveliev Vladimir Saveliev <c17830 at cray.com>
Signed-off-by: Alexey Lyashkov <c17817 at cray.com>
Signed-off-by: Alexander Boyko <c17825 at cray.com>
Signed-off-by: Yang Sheng <ys at whamcloud.com>
WC-bug-id: https://jira.whamcloud.com/browse/LU-4134
Seagate-bug-id: MRP-2139 MRP-3267
Reviewed-on: https://review.whamcloud.com/8045
Reviewed-on: https://review.whamcloud.com/29967
Reviewed-by: James Simmons <uja.ornl at yahoo.com>
Reviewed-by: Alexey Lyashkov <c17817 at cray.com>
Reviewed-by: Oleg Drokin <green at whamcloud.com>
Signed-off-by: James Simmons <jsimmons at infradead.org>
---
 drivers/staging/lustre/lustre/include/obd_class.h  |   9 +-
 drivers/staging/lustre/lustre/obdclass/genops.c    | 284 +++++++++++++++------
 .../staging/lustre/lustre/obdclass/obd_config.c    | 143 ++++-------
 drivers/staging/lustre/lustre/obdclass/obd_mount.c |   6 +-
 4 files changed, 261 insertions(+), 181 deletions(-)

diff --git a/drivers/staging/lustre/lustre/include/obd_class.h b/drivers/staging/lustre/lustre/include/obd_class.h
index 567189c..cc00915 100644
--- a/drivers/staging/lustre/lustre/include/obd_class.h
+++ b/drivers/staging/lustre/lustre/include/obd_class.h
@@ -65,8 +65,11 @@ int class_register_type(struct obd_ops *dt_ops, struct md_ops *md_ops,
 			const char *name, struct lu_device_type *ldt);
 int class_unregister_type(const char *name);
 
-struct obd_device *class_newdev(const char *type_name, const char *name);
-void class_release_dev(struct obd_device *obd);
+struct obd_device *class_newdev(const char *type_name, const char *name,
+				const char *uuid);
+int class_register_device(struct obd_device *obd);
+void class_unregister_device(struct obd_device *obd);
+void class_free_dev(struct obd_device *obd);
 
 int class_name2dev(const char *name);
 struct obd_device *class_name2obd(const char *name);
@@ -230,6 +233,8 @@ void __class_export_del_lock_ref(struct obd_export *exp,
 void class_export_put(struct obd_export *exp);
 struct obd_export *class_new_export(struct obd_device *obddev,
 				    struct obd_uuid *cluuid);
+struct obd_export *class_new_export_self(struct obd_device *obd,
+					 struct obd_uuid *uuid);
 void class_unlink_export(struct obd_export *exp);
 
 struct obd_import *class_import_get(struct obd_import *imp);
diff --git a/drivers/staging/lustre/lustre/obdclass/genops.c b/drivers/staging/lustre/lustre/obdclass/genops.c
index 59891a8..cdd44f7 100644
--- a/drivers/staging/lustre/lustre/obdclass/genops.c
+++ b/drivers/staging/lustre/lustre/obdclass/genops.c
@@ -38,6 +38,7 @@
 
 #define DEBUG_SUBSYSTEM S_CLASS
 #include <obd_class.h>
+#include <lustre_log.h>
 #include <lprocfs_status.h>
 #include <lustre_kernelcomm.h>
 
@@ -273,21 +274,20 @@ int class_unregister_type(const char *name)
 /**
  * Create a new obd device.
  *
- * Find an empty slot in ::obd_devs[], create a new obd device in it.
+ * Allocate the new obd_device and initialize it.
  *
  * \param[in] type_name obd device type string.
  * \param[in] name      obd device name.
+ * @uuid		obd device UUID.
  *
- * \retval NULL if create fails, otherwise return the obd device
- *	 pointer created.
+ * RETURN newdev	 pointer to created obd_device
+ * RETURN ERR_PTR(errno) on error
  */
-struct obd_device *class_newdev(const char *type_name, const char *name)
+struct obd_device *class_newdev(const char *type_name, const char *name,
+				const char *uuid)
 {
-	struct obd_device *result = NULL;
 	struct obd_device *newdev;
 	struct obd_type *type = NULL;
-	int i;
-	int new_obd_minor = 0;
 
 	if (strlen(name) >= MAX_OBD_NAME) {
 		CERROR("name/uuid must be < %u bytes long\n", MAX_OBD_NAME);
@@ -302,87 +302,167 @@ struct obd_device *class_newdev(const char *type_name, const char *name)
 
 	newdev = obd_device_alloc();
 	if (!newdev) {
-		result = ERR_PTR(-ENOMEM);
-		goto out_type;
+		class_put_type(type);
+		return ERR_PTR(-ENOMEM);
 	}
 
 	LASSERT(newdev->obd_magic == OBD_DEVICE_MAGIC);
+	strncpy(newdev->obd_name, name, sizeof(newdev->obd_name) - 1);
+	newdev->obd_type = type;
+	newdev->obd_minor = -1;
+
+	rwlock_init(&newdev->obd_pool_lock);
+	newdev->obd_pool_limit = 0;
+	newdev->obd_pool_slv = 0;
+
+	INIT_LIST_HEAD(&newdev->obd_exports);
+	INIT_LIST_HEAD(&newdev->obd_unlinked_exports);
+	INIT_LIST_HEAD(&newdev->obd_delayed_exports);
+	spin_lock_init(&newdev->obd_nid_lock);
+	spin_lock_init(&newdev->obd_dev_lock);
+	mutex_init(&newdev->obd_dev_mutex);
+	spin_lock_init(&newdev->obd_osfs_lock);
+	/* newdev->obd_osfs_age must be set to a value in the distant
+	 * past to guarantee a fresh statfs is fetched on mount.
+	 */
+	newdev->obd_osfs_age = get_jiffies_64() - 1000 * HZ;
 
-	write_lock(&obd_dev_lock);
-	for (i = 0; i < class_devno_max(); i++) {
-		struct obd_device *obd = class_num2obd(i);
+	/* XXX belongs in setup not attach  */
+	init_rwsem(&newdev->obd_observer_link_sem);
+	/* recovery data */
+	init_waitqueue_head(&newdev->obd_evict_inprogress_waitq);
 
-		if (obd && (strcmp(name, obd->obd_name) == 0)) {
-			CERROR("Device %s already exists at %d, won't add\n",
-			       name, i);
-			if (result) {
-				LASSERTF(result->obd_magic == OBD_DEVICE_MAGIC,
-					 "%p obd_magic %08x != %08x\n", result,
-					 result->obd_magic, OBD_DEVICE_MAGIC);
-				LASSERTF(result->obd_minor == new_obd_minor,
-					 "%p obd_minor %d != %d\n", result,
-					 result->obd_minor, new_obd_minor);
-
-				obd_devs[result->obd_minor] = NULL;
-				result->obd_name[0] = '\0';
-			 }
-			result = ERR_PTR(-EEXIST);
-			break;
-		}
-		if (!result && !obd) {
-			result = newdev;
-			result->obd_minor = i;
-			new_obd_minor = i;
-			result->obd_type = type;
-			strncpy(result->obd_name, name,
-				sizeof(result->obd_name) - 1);
-			obd_devs[i] = result;
-		}
-	}
-	write_unlock(&obd_dev_lock);
+	llog_group_init(&newdev->obd_olg);
+	/* Detach drops this */
+        atomic_set(&newdev->obd_refcount, 1);
+        lu_ref_init(&newdev->obd_reference);
+        lu_ref_add(&newdev->obd_reference, "newdev", newdev);
 
-	if (!result && i >= class_devno_max()) {
-		CERROR("all %u OBD devices used, increase MAX_OBD_DEVICES\n",
-		       class_devno_max());
-		result = ERR_PTR(-EOVERFLOW);
-		goto out;
-	}
+	newdev->obd_conn_inprogress = 0;
 
-	if (IS_ERR(result))
-		goto out;
+	strncpy(newdev->obd_uuid.uuid, uuid, strlen(uuid));
 
-	CDEBUG(D_IOCTL, "Adding new device %s (%p)\n",
-	       result->obd_name, result);
+	CDEBUG(D_IOCTL, "Allocate new device %s (%p)\n",
+	       newdev->obd_name, newdev);
 
-	return result;
-out:
-	obd_device_free(newdev);
-out_type:
-	class_put_type(type);
-	return result;
+	return newdev;
 }
 
-void class_release_dev(struct obd_device *obd)
+/**
+ * Free obd device.
+ *
+ * @obd obd_device to be freed
+ *
+ * RETURN none
+ */
+void class_free_dev(struct obd_device *obd)
 {
 	struct obd_type *obd_type = obd->obd_type;
 
 	LASSERTF(obd->obd_magic == OBD_DEVICE_MAGIC, "%p obd_magic %08x != %08x\n",
 		 obd, obd->obd_magic, OBD_DEVICE_MAGIC);
-	LASSERTF(obd == obd_devs[obd->obd_minor], "obd %p != obd_devs[%d] %p\n",
+	LASSERTF(obd->obd_minor == -1 || obd == obd_devs[obd->obd_minor],
+		 "obd %p != obd_devs[%d] %p\n",
 		 obd, obd->obd_minor, obd_devs[obd->obd_minor]);
+	LASSERTF(atomic_read(&obd->obd_refcount) == 0,
+		 "obd_refcount should be 0, not %d\n",
+		 atomic_read(&obd->obd_refcount));
 	LASSERT(obd_type);
 
-	CDEBUG(D_INFO, "Release obd device %s at %d obd_type name =%s\n",
-	       obd->obd_name, obd->obd_minor, obd->obd_type->typ_name);
+	CDEBUG(D_INFO, "Release obd device %s obd_type name =%s\n",
+	       obd->obd_name, obd->obd_type->typ_name);
+
+	CDEBUG(D_CONFIG, "finishing cleanup of obd %s (%s)\n",
+	       obd->obd_name, obd->obd_uuid.uuid);
+	if (obd->obd_stopping) {
+		int err;
+
+		/* If we're not stopping, we were never set up */
+		err = obd_cleanup(obd);
+		if (err)
+			CERROR("Cleanup %s returned %d\n",
+			       obd->obd_name, err);
+	}
 
-	write_lock(&obd_dev_lock);
-	obd_devs[obd->obd_minor] = NULL;
-	write_unlock(&obd_dev_lock);
 	obd_device_free(obd);
 
 	class_put_type(obd_type);
 }
 
+/**
+ * Unregister obd device.
+ *
+ * Free slot in obd_dev[] used by \a obd.
+ *
+ * @new_obd	obd_device to be unregistered
+ *
+ * RETURN	none
+ */
+void class_unregister_device(struct obd_device *obd)
+{
+	write_lock(&obd_dev_lock);
+	if (obd->obd_minor >= 0) {
+		LASSERT(obd_devs[obd->obd_minor] == obd);
+		obd_devs[obd->obd_minor] = NULL;
+		obd->obd_minor = -1;
+	}
+	write_unlock(&obd_dev_lock);
+}
+
+/**
+ * Register obd device.
+ *
+ * Find free slot in obd_devs[], fills it with \a new_obd.
+ *
+ * @new_obd	obd_device to be registered
+ *
+ * RETURN	0		success
+ *		-EEXIST		device with this name is registered
+ *		-EOVERFLOW	obd_devs[] is full
+ */
+int class_register_device(struct obd_device *new_obd)
+{
+	int new_obd_minor = 0;
+	bool minor_assign = false;
+	int ret = 0;
+	int i;
+
+	write_lock(&obd_dev_lock);
+	for (i = 0; i < class_devno_max(); i++) {
+		struct obd_device *obd = class_num2obd(i);
+
+		if (obd && (strcmp(new_obd->obd_name, obd->obd_name) == 0)) {
+			CERROR("%s: already exists, won't add\n",
+			       obd->obd_name);
+			/* in case we found a free slot before duplicate */
+			minor_assign = false;
+			ret = -EEXIST;
+			break;
+		}
+		if (!minor_assign && !obd) {
+			new_obd_minor = i;
+			minor_assign = true;
+		}
+	}
+
+	if (minor_assign) {
+		new_obd->obd_minor = new_obd_minor;
+		LASSERTF(!obd_devs[new_obd_minor], "obd_devs[%d] %p\n",
+			 new_obd_minor, obd_devs[new_obd_minor]);
+		obd_devs[new_obd_minor] = new_obd;
+	} else {
+		if (ret == 0) {
+			ret = -EOVERFLOW;
+			CERROR("%s: all %u/%u devices used, increase MAX_OBD_DEVICES: rc = %d\n",
+			       new_obd->obd_name, i, class_devno_max(), ret);
+		}
+	}
+	write_unlock(&obd_dev_lock);
+
+	return ret;
+}
+
+
 int class_name2dev(const char *name)
 {
 	int i;
@@ -677,7 +757,11 @@ static void class_export_destroy(struct obd_export *exp)
 	LASSERT(list_empty(&exp->exp_req_replay_queue));
 	LASSERT(list_empty(&exp->exp_hp_rpcs));
 	obd_destroy_export(exp);
-	class_decref(obd, "export", exp);
+	/* self export doesn't hold a reference to an obd, although it
+	 * exists until freeing of the obd
+	 */
+	if (exp != obd->obd_self_export)
+		class_decref(obd, "export", exp);
 
 	OBD_FREE_RCU(exp, sizeof(*exp), &exp->exp_handle);
 }
@@ -708,11 +792,27 @@ void class_export_put(struct obd_export *exp)
 	       atomic_read(&exp->exp_refcount) - 1);
 
 	if (atomic_dec_and_test(&exp->exp_refcount)) {
-		LASSERT(!list_empty(&exp->exp_obd_chain));
+		struct obd_device *obd = exp->exp_obd;
+
 		CDEBUG(D_IOCTL, "final put %p/%s\n",
 		       exp, exp->exp_client_uuid.uuid);
 
-		obd_zombie_export_add(exp);
+		if (exp == obd->obd_self_export) {
+			/* self export should be destroyed without
+			 * zombie thread as it doesn't hold a
+			 * reference to obd and doesn't hold any
+			 * resources
+			 */
+			class_export_destroy(exp);
+			/* self export is destroyed, no class
+			 * references exist and it is safe to free
+			 * obd
+			 */
+			class_free_dev(obd);
+		} else {
+			LASSERT(!list_empty(&exp->exp_obd_chain));
+			obd_zombie_export_add(exp);
+		}
 	}
 }
 EXPORT_SYMBOL(class_export_put);
@@ -728,8 +828,9 @@ static void obd_zombie_exp_cull(struct work_struct *ws)
  * pointer to it. The refcount is 2: one for the hash reference, and
  * one for the pointer returned by this function.
  */
-struct obd_export *class_new_export(struct obd_device *obd,
-				    struct obd_uuid *cluuid)
+static struct obd_export *__class_new_export(struct obd_device *obd,
+					     struct obd_uuid *cluuid,
+					     bool is_self)
 {
 	struct obd_export *export;
 	int rc = 0;
@@ -739,6 +840,7 @@ struct obd_export *class_new_export(struct obd_device *obd,
 		return ERR_PTR(-ENOMEM);
 
 	export->exp_conn_cnt = 0;
+	/* 2 = class_handle_hash + last */
 	atomic_set(&export->exp_refcount, 2);
 	atomic_set(&export->exp_rpc_count, 0);
 	atomic_set(&export->exp_cb_count, 0);
@@ -767,41 +869,65 @@ struct obd_export *class_new_export(struct obd_device *obd,
 	export->exp_client_uuid = *cluuid;
 	obd_init_export(export);
 
-	spin_lock(&obd->obd_dev_lock);
-	/* shouldn't happen, but might race */
-	if (obd->obd_stopping) {
-		rc = -ENODEV;
-		goto exit_unlock;
-	}
-
 	if (!obd_uuid_equals(cluuid, &obd->obd_uuid)) {
+		spin_lock(&obd->obd_dev_lock);
+		/* shouldn't happen, but might race */
+		if (obd->obd_stopping) {
+			rc = -ENODEV;
+			goto exit_unlock;
+		}
+		spin_unlock(&obd->obd_dev_lock);
+
 		rc = obd_uuid_add(obd, export);
 		if (rc) {
 			LCONSOLE_WARN("%s: denying duplicate export for %s, %d\n",
 				      obd->obd_name, cluuid->uuid, rc);
-			goto exit_unlock;
+			goto exit_err;
 		}
 	}
 
-	class_incref(obd, "export", export);
-	list_add(&export->exp_obd_chain, &export->exp_obd->obd_exports);
-	export->exp_obd->obd_num_exports++;
+	spin_lock(&obd->obd_dev_lock);
+	if (!is_self) {
+		class_incref(obd, "export", export);
+		list_add(&export->exp_obd_chain, &export->exp_obd->obd_exports);
+		obd->obd_num_exports++;
+	} else {
+		INIT_LIST_HEAD(&export->exp_obd_chain);
+	}
 	spin_unlock(&obd->obd_dev_lock);
 	return export;
 
 exit_unlock:
 	spin_unlock(&obd->obd_dev_lock);
+exit_err:
 	class_handle_unhash(&export->exp_handle);
 	obd_destroy_export(export);
 	kfree(export);
 	return ERR_PTR(rc);
 }
+
+struct obd_export *class_new_export(struct obd_device *obd,
+				    struct obd_uuid *uuid)
+{
+	return __class_new_export(obd, uuid, false);
+}
 EXPORT_SYMBOL(class_new_export);
 
+struct obd_export *class_new_export_self(struct obd_device *obd,
+					 struct obd_uuid *uuid)
+{
+	return __class_new_export(obd, uuid, true);
+}
+
 void class_unlink_export(struct obd_export *exp)
 {
 	class_handle_unhash(&exp->exp_handle);
 
+	if (exp->exp_obd->obd_self_export == exp) {
+		class_export_put(exp);
+		return;
+	}
+
 	spin_lock(&exp->exp_obd->obd_dev_lock);
 	/* delete an uuid-export hashitem from hashtables */
 	if (exp != exp->exp_obd->obd_self_export)
diff --git a/drivers/staging/lustre/lustre/obdclass/obd_config.c b/drivers/staging/lustre/lustre/obdclass/obd_config.c
index 9e46eb2..8be8751 100644
--- a/drivers/staging/lustre/lustre/obdclass/obd_config.c
+++ b/drivers/staging/lustre/lustre/obdclass/obd_config.c
@@ -280,6 +280,7 @@ static int class_attach(struct lustre_cfg *lcfg)
 {
 	struct obd_device *obd = NULL;
 	char *typename, *name, *uuid;
+	struct obd_export *exp;
 	int rc, len;
 
 	if (!LUSTRE_CFG_BUFLEN(lcfg, 1)) {
@@ -298,78 +299,50 @@ static int class_attach(struct lustre_cfg *lcfg)
 		CERROR("No UUID passed!\n");
 		return -EINVAL;
 	}
-	uuid = lustre_cfg_string(lcfg, 2);
 
-	CDEBUG(D_IOCTL, "attach type %s name: %s uuid: %s\n",
-	       typename, name, uuid);
+	uuid = lustre_cfg_string(lcfg, 2);
+	len = strlen(uuid);
+	if (len >= sizeof(obd->obd_uuid)) {
+		CERROR("uuid must be < %d bytes long\n",
+		       (int)sizeof(obd->obd_uuid));
+		return -EINVAL;
+	}
 
-	obd = class_newdev(typename, name);
+	obd = class_newdev(typename, name, uuid);
 	if (IS_ERR(obd)) {
 		/* Already exists or out of obds */
 		rc = PTR_ERR(obd);
-		obd = NULL;
 		CERROR("Cannot create device %s of type %s : %d\n",
 		       name, typename, rc);
-		goto out;
+		return rc;
 	}
-	LASSERTF(obd, "Cannot get obd device %s of type %s\n",
-		 name, typename);
 	LASSERTF(obd->obd_magic == OBD_DEVICE_MAGIC,
 		 "obd %p obd_magic %08X != %08X\n",
 		 obd, obd->obd_magic, OBD_DEVICE_MAGIC);
 	LASSERTF(strncmp(obd->obd_name, name, strlen(name)) == 0,
 		 "%p obd_name %s != %s\n", obd, obd->obd_name, name);
 
-	rwlock_init(&obd->obd_pool_lock);
-	obd->obd_pool_limit = 0;
-	obd->obd_pool_slv = 0;
-
-	INIT_LIST_HEAD(&obd->obd_exports);
-	INIT_LIST_HEAD(&obd->obd_unlinked_exports);
-	INIT_LIST_HEAD(&obd->obd_delayed_exports);
-	spin_lock_init(&obd->obd_nid_lock);
-	spin_lock_init(&obd->obd_dev_lock);
-	mutex_init(&obd->obd_dev_mutex);
-	spin_lock_init(&obd->obd_osfs_lock);
-	/* obd->obd_osfs_age must be set to a value in the distant
-	 * past to guarantee a fresh statfs is fetched on mount.
-	 */
-	obd->obd_osfs_age = get_jiffies_64() - 1000 * HZ;
-
-	/* XXX belongs in setup not attach  */
-	init_rwsem(&obd->obd_observer_link_sem);
-	/* recovery data */
-	init_waitqueue_head(&obd->obd_evict_inprogress_waitq);
-
-	llog_group_init(&obd->obd_olg);
+	exp = class_new_export_self(obd, &obd->obd_uuid);
+	if (IS_ERR(exp)) {
+		rc = PTR_ERR(exp);
+		class_free_dev(obd);
+		return rc;
+	}
 
-	obd->obd_conn_inprogress = 0;
+	obd->obd_self_export = exp;
+	class_export_put(exp);
 
-	len = strlen(uuid);
-	if (len >= sizeof(obd->obd_uuid)) {
-		CERROR("uuid must be < %d bytes long\n",
-		       (int)sizeof(obd->obd_uuid));
-		rc = -EINVAL;
-		goto out;
+	rc = class_register_device(obd);
+	if (rc) {
+		class_decref(obd, "newdev", obd);
+		return rc;
 	}
-	memcpy(obd->obd_uuid.uuid, uuid, len);
-
-	/* Detach drops this */
-	spin_lock(&obd->obd_dev_lock);
-	atomic_set(&obd->obd_refcount, 1);
-	spin_unlock(&obd->obd_dev_lock);
-	lu_ref_init(&obd->obd_reference);
-	lu_ref_add(&obd->obd_reference, "attach", obd);
 
 	obd->obd_attached = 1;
 	CDEBUG(D_IOCTL, "OBD: dev %d attached type %s with refcount %d\n",
 	       obd->obd_minor, typename, atomic_read(&obd->obd_refcount));
-	return 0;
- out:
-	if (obd)
-		class_release_dev(obd);
 
-	return rc;
+	return 0;
 }
 
 /** Create hashes, self-export, and call type-specific setup.
@@ -378,7 +351,6 @@ static int class_attach(struct lustre_cfg *lcfg)
 static int class_setup(struct obd_device *obd, struct lustre_cfg *lcfg)
 {
 	int err = 0;
-	struct obd_export *exp;
 
 	LASSERT(obd);
 	LASSERTF(obd == class_num2obd(obd->obd_minor),
@@ -420,18 +392,9 @@ static int class_setup(struct obd_device *obd, struct lustre_cfg *lcfg)
 	if (err)
 		goto err_hash;
 
-	exp = class_new_export(obd, &obd->obd_uuid);
-	if (IS_ERR(exp)) {
-		err = PTR_ERR(exp);
-		goto err_new;
-	}
-
-	obd->obd_self_export = exp;
-	class_export_put(exp);
-
 	err = obd_setup(obd, lcfg);
 	if (err)
-		goto err_exp;
+		goto err_setup;
 
 	obd->obd_set_up = 1;
 
@@ -444,12 +407,7 @@ static int class_setup(struct obd_device *obd, struct lustre_cfg *lcfg)
 	       obd->obd_name, obd->obd_uuid.uuid);
 
 	return 0;
-err_exp:
-	if (obd->obd_self_export) {
-		class_unlink_export(obd->obd_self_export);
-		obd->obd_self_export = NULL;
-	}
-err_new:
+err_setup:
 	rhashtable_destroy(&obd->obd_uuid_hash);
 err_hash:
 	obd->obd_starting = 0;
@@ -476,10 +434,13 @@ static int class_detach(struct obd_device *obd, struct lustre_cfg *lcfg)
 	obd->obd_attached = 0;
 	spin_unlock(&obd->obd_dev_lock);
 
+	/* cleanup in progress. we don't like to find this device after now */
+	class_unregister_device(obd);
+
 	CDEBUG(D_IOCTL, "detach on obd %s (uuid %s)\n",
 	       obd->obd_name, obd->obd_uuid.uuid);
 
-	class_decref(obd, "attach", obd);
+	class_decref(obd, "newdev", obd);
 	return 0;
 }
 
@@ -507,6 +468,10 @@ static int class_cleanup(struct obd_device *obd, struct lustre_cfg *lcfg)
 	}
 	/* Leave this on forever */
 	obd->obd_stopping = 1;
+	/* function can't return error after that point, so clear setup flag
+	 * as early as possible to avoid finding via obd_devs / hash
+	 */
+	obd->obd_set_up = 0;
 	spin_unlock(&obd->obd_dev_lock);
 
 	while (obd->obd_conn_inprogress > 0)
@@ -567,43 +532,27 @@ struct obd_device *class_incref(struct obd_device *obd,
 
 void class_decref(struct obd_device *obd, const char *scope, const void *source)
 {
-	int err;
-	int refs;
+	int last;
 
-	spin_lock(&obd->obd_dev_lock);
-	atomic_dec(&obd->obd_refcount);
-	refs = atomic_read(&obd->obd_refcount);
-	spin_unlock(&obd->obd_dev_lock);
+	CDEBUG(D_INFO, "Decref %s (%p) now %d - %s\n", obd->obd_name, obd,
+	       atomic_read(&obd->obd_refcount), scope);
+
+	LASSERT(obd->obd_num_exports >= 0);
+	last = atomic_dec_and_test(&obd->obd_refcount);
 	lu_ref_del(&obd->obd_reference, scope, source);
 
-	CDEBUG(D_INFO, "Decref %s (%p) now %d\n", obd->obd_name, obd, refs);
+	if (last) {
+		struct obd_export *exp;
 
-	if ((refs == 1) && obd->obd_stopping) {
+		LASSERT(!obd->obd_attached);
 		/* All exports have been destroyed; there should
 		 * be no more in-progress ops by this point.
 		 */
-
-		spin_lock(&obd->obd_self_export->exp_lock);
-		obd->obd_self_export->exp_flags |= exp_flags_from_obd(obd);
-		spin_unlock(&obd->obd_self_export->exp_lock);
-
-		/* note that we'll recurse into class_decref again */
-		class_unlink_export(obd->obd_self_export);
-		return;
-	}
-
-	if (refs == 0) {
-		CDEBUG(D_CONFIG, "finishing cleanup of obd %s (%s)\n",
-		       obd->obd_name, obd->obd_uuid.uuid);
-		LASSERT(!obd->obd_attached);
-		if (obd->obd_stopping) {
-			/* If we're not stopping, we were never set up */
-			err = obd_cleanup(obd);
-			if (err)
-				CERROR("Cleanup %s returned %d\n",
-				       obd->obd_name, err);
+		exp = obd->obd_self_export;
+		if (exp) {
+			exp->exp_flags |= exp_flags_from_obd(obd);
+			class_unlink_export(exp);
 		}
-		class_release_dev(obd);
 	}
 }
 EXPORT_SYMBOL(class_decref);
diff --git a/drivers/staging/lustre/lustre/obdclass/obd_mount.c b/drivers/staging/lustre/lustre/obdclass/obd_mount.c
index 5ed1758..db5e1b5 100644
--- a/drivers/staging/lustre/lustre/obdclass/obd_mount.c
+++ b/drivers/staging/lustre/lustre/obdclass/obd_mount.c
@@ -214,7 +214,7 @@ int lustre_start_mgc(struct super_block *sb)
 	struct lustre_sb_info *lsi = s2lsi(sb);
 	struct obd_device *obd;
 	struct obd_export *exp;
-	struct obd_uuid *uuid;
+	struct obd_uuid *uuid = NULL;
 	class_uuid_t uuidc;
 	lnet_nid_t nid;
 	char nidstr[LNET_NIDSTR_SIZE];
@@ -342,7 +342,6 @@ int lustre_start_mgc(struct super_block *sb)
 	rc = lustre_start_simple(mgcname, LUSTRE_MGC_NAME,
 				 (char *)uuid->uuid, LUSTRE_MGS_OBDNAME,
 				 niduuid, NULL, NULL);
-	kfree(uuid);
 	if (rc)
 		goto out_free;
 
@@ -404,7 +403,7 @@ int lustre_start_mgc(struct super_block *sb)
 	    lsi->lsi_lmd->lmd_flags & LMD_FLG_NOIR)
 		data->ocd_connect_flags &= ~OBD_CONNECT_IMP_RECOV;
 	data->ocd_version = LUSTRE_VERSION_CODE;
-	rc = obd_connect(NULL, &exp, obd, &obd->obd_uuid, data, NULL);
+	rc = obd_connect(NULL, &exp, obd, uuid, data, NULL);
 	if (rc) {
 		CERROR("connect failed %d\n", rc);
 		goto out;
@@ -420,6 +419,7 @@ int lustre_start_mgc(struct super_block *sb)
 out_free:
 	mutex_unlock(&mgc_start_lock);
 
+	kfree(uuid);
 	kfree(data);
 	kfree(mgcname);
 	kfree(niduuid);
-- 
1.8.3.1


From jsimmons at infradead.org  Mon Nov 26 02:48:26 2018
From: jsimmons at infradead.org (James Simmons)
Date: Sun, 25 Nov 2018 21:48:26 -0500
Subject: [lustre-devel] [PATCH 10/12] lustre: clio: Introduce parallel tasks
 framework
In-Reply-To: <1543200508-6838-1-git-send-email-jsimmons@infradead.org>
References: <1543200508-6838-1-git-send-email-jsimmons@infradead.org>
Message-ID: <1543200508-6838-11-git-send-email-jsimmons@infradead.org>

From: Dmitry Eremin <dmitry.eremin at intel.com>

In this patch new API for parallel tasks execution is introduced.
This API based on Linux kernel padata API which is used to perform
encryption and decryption on large numbers of packets without
reordering those packets.

It was adopted for general use in Lustre for parallelization of
various functionality. The first place of its usage is parallel I/O
implementation.

The first step in using it is to set up a cl_ptask structure to
control of how this task are to be run:

    #include <cl_ptask.h>

    int cl_ptask_init(struct cl_ptask *ptask, cl_ptask_cb_t cbfunc,
                      void *cbdata, unsigned int flags, int cpu);

The cbfunc function with cbdata argument will be called in the process
of getting the task done. The cpu specifies which CPU will be used for
the final callback when the task is done.

The submission of task is done with:

    int cl_ptask_submit(struct cl_ptask *ptask,
                        struct cl_ptask_engine *engine);

The task is submitted to the engine for execution.

In order to wait for result of task execution you should call:

   int cl_ptask_wait_for(struct cl_ptask *ptask);

The tasks with flag PTF_ORDERED are executed in parallel but complete
into submission order. So, waiting for last ordered task you can be sure
that all previous tasks were done before this task complete.

This patch differs from the OpenSFS tree by adding this functional
to the clio layer instead of libcfs.

Signed-off-by: Dmitry Eremin <dmitry.eremin at intel.com>
WC-bug-id: https://jira.whamcloud.com/browse/LU-8964
Reviewed-on: https://review.whamcloud.com/24474
Reviewed-by: Jinshan Xiong <jinshan.xiong at intel.com>
Reviewed-by: James Simmons <uja.ornl at yahoo.com>
Reviewed-by: Oleg Drokin <green at whamcloud.com>
Signed-off-by: James Simmons <jsimmons at infradead.org>
---
 drivers/staging/lustre/lustre/include/cl_ptask.h  | 145 +++++++
 drivers/staging/lustre/lustre/obdclass/Makefile   |   3 +-
 drivers/staging/lustre/lustre/obdclass/cl_ptask.c | 501 ++++++++++++++++++++++
 3 files changed, 648 insertions(+), 1 deletion(-)
 create mode 100644 drivers/staging/lustre/lustre/include/cl_ptask.h
 create mode 100644 drivers/staging/lustre/lustre/obdclass/cl_ptask.c

diff --git a/drivers/staging/lustre/lustre/include/cl_ptask.h b/drivers/staging/lustre/lustre/include/cl_ptask.h
new file mode 100644
index 0000000..02abd69
--- /dev/null
+++ b/drivers/staging/lustre/lustre/include/cl_ptask.h
@@ -0,0 +1,145 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * GPL HEADER START
+ *
+ * DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 only,
+ * as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+ * General Public License version 2 for more details (a copy is included
+ * in the LICENSE file that accompanied this code).
+ *
+ * You should have received a copy of the GNU General Public License
+ * version 2 along with this program; If not, see
+ * http://www.gnu.org/licenses/gpl-2.0.html
+ *
+ * GPL HEADER END
+ */
+/*
+ * Copyright (c) 2017, Intel Corporation.
+ * Use is subject to license terms.
+ */
+/*
+ * This file is part of Lustre, http://www.lustre.org/
+ *
+ * parallel task interface
+ */
+#ifndef __CL_LUSTRE_PTASK_H__
+#define __CL_LUSTRE_PTASK_H__
+
+#include <linux/types.h>
+#include <linux/bitops.h>
+#include <linux/kernel.h>
+#include <linux/cpumask.h>
+#include <linux/uaccess.h>
+#include <linux/notifier.h>
+#include <linux/workqueue.h>
+#include <linux/completion.h>
+#ifdef CONFIG_PADATA
+#include <linux/padata.h>
+#else
+struct padata_priv {};
+struct padata_instance {};
+#endif
+
+#define PTF_COMPLETE	BIT(0)
+#define PTF_AUTOFREE	BIT(1)
+#define PTF_ORDERED	BIT(2)
+#define PTF_USER_MM	BIT(3)
+#define PTF_ATOMIC	BIT(4)
+#define PTF_RETRY	BIT(5)
+
+struct cl_ptask_engine {
+	struct padata_instance	*pte_pinst;
+	struct workqueue_struct	*pte_wq;
+	struct notifier_block	 pte_notifier;
+	int			 pte_weight;
+};
+
+struct cl_ptask;
+typedef int (*cl_ptask_cb_t)(struct cl_ptask *);
+
+struct cl_ptask {
+	struct padata_priv	 pt_padata;
+	struct completion	 pt_completion;
+	mm_segment_t		 pt_fs;
+	struct mm_struct	*pt_mm;
+	unsigned int		 pt_flags;
+	int			 pt_cbcpu;
+	cl_ptask_cb_t		 pt_cbfunc;
+	void			*pt_cbdata;
+	int			 pt_result;
+};
+
+static inline
+struct padata_priv *cl_ptask2padata(struct cl_ptask *ptask)
+{
+	return &ptask->pt_padata;
+}
+
+static inline
+struct cl_ptask *cl_padata2ptask(struct padata_priv *padata)
+{
+	return container_of(padata, struct cl_ptask, pt_padata);
+}
+
+static inline
+bool cl_ptask_need_complete(struct cl_ptask *ptask)
+{
+	return ptask->pt_flags & PTF_COMPLETE;
+}
+
+static inline
+bool cl_ptask_is_autofree(struct cl_ptask *ptask)
+{
+	return ptask->pt_flags & PTF_AUTOFREE;
+}
+
+static inline
+bool cl_ptask_is_ordered(struct cl_ptask *ptask)
+{
+	return ptask->pt_flags & PTF_ORDERED;
+}
+
+static inline
+bool cl_ptask_use_user_mm(struct cl_ptask *ptask)
+{
+	return ptask->pt_flags & PTF_USER_MM;
+}
+
+static inline
+bool cl_ptask_is_atomic(struct cl_ptask *ptask)
+{
+	return ptask->pt_flags & PTF_ATOMIC;
+}
+
+static inline
+bool cl_ptask_is_retry(struct cl_ptask *ptask)
+{
+	return ptask->pt_flags & PTF_RETRY;
+}
+
+static inline
+int cl_ptask_result(struct cl_ptask *ptask)
+{
+	return ptask->pt_result;
+}
+
+struct cl_ptask_engine *cl_ptengine_init(const char *name,
+					 const struct cpumask *cpumask);
+void cl_ptengine_fini(struct cl_ptask_engine *engine);
+int cl_ptengine_set_cpumask(struct cl_ptask_engine *engine,
+			    const struct cpumask *cpumask);
+int cl_ptengine_weight(struct cl_ptask_engine *engine);
+
+int cl_ptask_submit(struct cl_ptask *ptask,  struct cl_ptask_engine *engine);
+int cl_ptask_wait_for(struct cl_ptask *ptask);
+int cl_ptask_init(struct cl_ptask *ptask, cl_ptask_cb_t cbfunc, void *cbdata,
+		  unsigned int flags, int cpu);
+
+#endif /* __CL_LUSTRE_PTASK_H__ */
diff --git a/drivers/staging/lustre/lustre/obdclass/Makefile b/drivers/staging/lustre/lustre/obdclass/Makefile
index b1fac48..a705aa0 100644
--- a/drivers/staging/lustre/lustre/obdclass/Makefile
+++ b/drivers/staging/lustre/lustre/obdclass/Makefile
@@ -8,4 +8,5 @@ obdclass-y := llog.o llog_cat.o llog_obd.o llog_swab.o class_obd.o debug.o \
 	      genops.o obd_sysfs.o lprocfs_status.o lprocfs_counters.o \
 	      lustre_handles.o lustre_peer.o statfs_pack.o linkea.o \
 	      obdo.o obd_config.o obd_mount.o lu_object.o lu_ref.o \
-	      cl_object.o cl_page.o cl_lock.o cl_io.o kernelcomm.o
+	      cl_object.o cl_page.o cl_lock.o cl_io.o cl_ptask.o \
+	      kernelcomm.o
diff --git a/drivers/staging/lustre/lustre/obdclass/cl_ptask.c b/drivers/staging/lustre/lustre/obdclass/cl_ptask.c
new file mode 100644
index 0000000..b0df3c4
--- /dev/null
+++ b/drivers/staging/lustre/lustre/obdclass/cl_ptask.c
@@ -0,0 +1,501 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * GPL HEADER START
+ *
+ * DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 only,
+ * as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+ * General Public License version 2 for more details (a copy is included
+ * in the LICENSE file that accompanied this code).
+ *
+ * You should have received a copy of the GNU General Public License
+ * version 2 along with this program; If not, see
+ * http://www.gnu.org/licenses/gpl-2.0.html
+ *
+ * GPL HEADER END
+ */
+/*
+ * Copyright (c) 2017, Intel Corporation.
+ * Use is subject to license terms.
+ */
+/*
+ * This file is part of Lustre, http://www.lustre.org/
+ *
+ * parallel task interface
+ */
+#include <linux/errno.h>
+#include <linux/kernel.h>
+#include <linux/module.h>
+#include <linux/cpumask.h>
+#include <linux/cpu.h>
+#include <linux/slab.h>
+#include <linux/sched.h>
+#include <linux/sched/signal.h>
+#include <linux/sched/mm.h>
+#include <linux/moduleparam.h>
+#include <linux/mmu_context.h>
+
+#define DEBUG_SUBSYSTEM S_UNDEFINED
+
+#include <linux/libcfs/libcfs.h>
+#include <cl_ptask.h>
+
+/**
+ * This API based on Linux kernel padada API which is used to perform
+ * encryption and decryption on large numbers of packets without
+ * reordering those packets.
+ *
+ * It was adopted for general use in Lustre for parallelization of
+ * various functionality.
+ *
+ * The first step in using it is to set up a cl_ptask structure to
+ * control of how this task are to be run:
+ *
+ * #include <cl_ptask.h>
+ *
+ * int cl_ptask_init(struct cl_ptask *ptask, cl_ptask_cb_t cbfunc,
+ *                    void *cbdata, unsigned int flags, int cpu);
+ *
+ * The cbfunc function with cbdata argument will be called in the process
+ * of getting the task done. The cpu specifies which CPU will be used for
+ * the final callback when the task is done.
+ *
+ * The submission of task is done with:
+ *
+ * int cl_ptask_submit(struct cl_ptask *ptask, struct cl_ptask_engine *engine);
+ *
+ * The task is submitted to the engine for execution.
+ *
+ * In order to wait for result of task execution you should call:
+ *
+ * int cl_ptask_wait_for(struct cl_ptask *ptask);
+ *
+ * The tasks with flag PTF_ORDERED are executed in parallel but complete
+ * into submission order. So, waiting for last ordered task you can be sure
+ * that all previous tasks were done before this task complete.
+ */
+#ifdef CONFIG_PADATA
+static void cl_ptask_complete(struct padata_priv *padata)
+{
+	struct cl_ptask *ptask = cl_padata2ptask(padata);
+
+	if (cl_ptask_need_complete(ptask)) {
+		if (cl_ptask_is_ordered(ptask))
+			complete(&ptask->pt_completion);
+	} else if (cl_ptask_is_autofree(ptask)) {
+		kfree(ptask);
+	}
+}
+
+static void cl_ptask_execute(struct padata_priv *padata)
+{
+	struct cl_ptask *ptask = cl_padata2ptask(padata);
+	mm_segment_t old_fs = get_fs();
+	bool bh_enabled = false;
+
+	if (!cl_ptask_is_atomic(ptask)) {
+		local_bh_enable();
+		bh_enabled = true;
+	}
+
+	if (cl_ptask_use_user_mm(ptask) && ptask->pt_mm) {
+		use_mm(ptask->pt_mm);
+		set_fs(ptask->pt_fs);
+	}
+
+	if (ptask->pt_cbfunc)
+		ptask->pt_result = ptask->pt_cbfunc(ptask);
+	else
+		ptask->pt_result = -ENXIO;
+
+	if (cl_ptask_use_user_mm(ptask) && ptask->pt_mm) {
+		set_fs(old_fs);
+		unuse_mm(ptask->pt_mm);
+		mmput(ptask->pt_mm);
+		ptask->pt_mm = NULL;
+	}
+
+	if (cl_ptask_need_complete(ptask) && !cl_ptask_is_ordered(ptask))
+		complete(&ptask->pt_completion);
+
+	if (bh_enabled)
+		local_bh_disable();
+
+	padata_do_serial(padata);
+}
+
+static int cl_do_parallel(struct cl_ptask_engine *engine,
+			  struct padata_priv *padata)
+{
+	struct cl_ptask *ptask = cl_padata2ptask(padata);
+	int rc;
+
+	if (cl_ptask_need_complete(ptask))
+		reinit_completion(&ptask->pt_completion);
+
+	if (cl_ptask_use_user_mm(ptask)) {
+		ptask->pt_mm = get_task_mm(current);
+		ptask->pt_fs = get_fs();
+	}
+	ptask->pt_result = -EINPROGRESS;
+
+retry:
+	rc = padata_do_parallel(engine->pte_pinst, padata, ptask->pt_cbcpu);
+	if (rc == -EBUSY && cl_ptask_is_retry(ptask)) {
+		/* too many tasks already in queue */
+		schedule_timeout_uninterruptible(1);
+		goto retry;
+	}
+
+	if (rc) {
+		if (cl_ptask_use_user_mm(ptask) && ptask->pt_mm) {
+			mmput(ptask->pt_mm);
+			ptask->pt_mm = NULL;
+		}
+		ptask->pt_result = rc;
+	}
+
+	return rc;
+}
+
+/**
+ * This function submit initialized task for async execution
+ * in engine with specified id.
+ */
+int cl_ptask_submit(struct cl_ptask *ptask, struct cl_ptask_engine *engine)
+{
+	struct padata_priv *padata = cl_ptask2padata(ptask);
+
+	if (IS_ERR_OR_NULL(engine))
+		return -EINVAL;
+
+	memset(padata, 0, sizeof(*padata));
+
+	padata->parallel = cl_ptask_execute;
+	padata->serial   = cl_ptask_complete;
+
+	return cl_do_parallel(engine, padata);
+}
+
+#else  /* !CONFIG_PADATA */
+
+/**
+ * If CONFIG_PADATA is not defined this function just execute
+ * the initialized task in current thread. (emulate async execution)
+ */
+int cl_ptask_submit(struct cl_ptask *ptask, struct cl_ptask_engine *engine)
+{
+	if (IS_ERR_OR_NULL(engine))
+		return -EINVAL;
+
+	if (ptask->pt_cbfunc)
+		ptask->pt_result = ptask->pt_cbfunc(ptask);
+	else
+		ptask->pt_result = -ENXIO;
+
+	if (cl_ptask_need_complete(ptask))
+		complete(&ptask->pt_completion);
+	else if (cl_ptask_is_autofree(ptask))
+		kfree(ptask);
+
+	return 0;
+}
+#endif /* CONFIG_PADATA */
+EXPORT_SYMBOL(cl_ptask_submit);
+
+/**
+ * This function waits when task complete async execution.
+ * The tasks with flag PTF_ORDERED are executed in parallel but completes
+ * into submission order. So, waiting for last ordered task you can be sure
+ * that all previous tasks were done before this task complete.
+ */
+int cl_ptask_wait_for(struct cl_ptask *ptask)
+{
+	if (!cl_ptask_need_complete(ptask))
+		return -EINVAL;
+
+	wait_for_completion(&ptask->pt_completion);
+
+	return 0;
+}
+EXPORT_SYMBOL(cl_ptask_wait_for);
+
+/**
+ * This function initialize internal members of task and prepare it for
+ * async execution.
+ */
+int cl_ptask_init(struct cl_ptask *ptask, cl_ptask_cb_t cbfunc, void *cbdata,
+		  unsigned int flags, int cpu)
+{
+	memset(ptask, 0, sizeof(*ptask));
+
+	ptask->pt_flags  = flags;
+	ptask->pt_cbcpu  = cpu;
+	ptask->pt_mm     = NULL; /* will be set in cl_do_parallel() */
+	ptask->pt_fs     = get_fs();
+	ptask->pt_cbfunc = cbfunc;
+	ptask->pt_cbdata = cbdata;
+	ptask->pt_result = -EAGAIN;
+
+	if (cl_ptask_need_complete(ptask)) {
+		if (cl_ptask_is_autofree(ptask))
+			return -EINVAL;
+
+		init_completion(&ptask->pt_completion);
+	}
+
+	if (cl_ptask_is_atomic(ptask) && cl_ptask_use_user_mm(ptask))
+		return -EINVAL;
+
+	return 0;
+}
+EXPORT_SYMBOL(cl_ptask_init);
+
+/**
+ * This function set the mask of allowed CPUs for parallel execution
+ * for engine with specified id.
+ */
+int cl_ptengine_set_cpumask(struct cl_ptask_engine *engine,
+			    const struct cpumask *cpumask)
+{
+	int rc = 0;
+
+#ifdef CONFIG_PADATA
+	cpumask_var_t serial_mask;
+	cpumask_var_t parallel_mask;
+
+	if (IS_ERR_OR_NULL(engine))
+		return -EINVAL;
+
+	if (!alloc_cpumask_var(&serial_mask, GFP_KERNEL))
+		return -ENOMEM;
+
+	if (!alloc_cpumask_var(&parallel_mask, GFP_KERNEL)) {
+		free_cpumask_var(serial_mask);
+		return -ENOMEM;
+	}
+
+	cpumask_copy(parallel_mask, cpumask);
+	cpumask_copy(serial_mask, cpu_online_mask);
+
+	rc = padata_set_cpumask(engine->pte_pinst, PADATA_CPU_PARALLEL,
+				parallel_mask);
+	free_cpumask_var(parallel_mask);
+	if (rc)
+		goto out_failed_mask;
+
+	rc = padata_set_cpumask(engine->pte_pinst, PADATA_CPU_SERIAL,
+				serial_mask);
+out_failed_mask:
+	free_cpumask_var(serial_mask);
+#endif /* CONFIG_PADATA */
+
+	return rc;
+}
+EXPORT_SYMBOL(cl_ptengine_set_cpumask);
+
+/**
+ * This function returns the count of allowed CPUs for parallel execution
+ * for engine with specified id.
+ */
+int cl_ptengine_weight(struct cl_ptask_engine *engine)
+{
+	if (IS_ERR_OR_NULL(engine))
+		return -EINVAL;
+
+	return engine->pte_weight;
+}
+EXPORT_SYMBOL(cl_ptengine_weight);
+
+#ifdef CONFIG_PADATA
+static int cl_ptask_cpumask_change_notify(struct notifier_block *self,
+					  unsigned long val, void *data)
+{
+	struct padata_cpumask *padata_cpumask = data;
+	struct cl_ptask_engine *engine;
+
+	engine = container_of(self, struct cl_ptask_engine, pte_notifier);
+
+	if (val & PADATA_CPU_PARALLEL)
+		engine->pte_weight = cpumask_weight(padata_cpumask->pcpu);
+
+	return 0;
+}
+
+static int cl_ptengine_padata_init(struct cl_ptask_engine *engine,
+				   const char *name,
+				   const struct cpumask *cpumask)
+{
+	unsigned int wq_flags = WQ_MEM_RECLAIM | WQ_CPU_INTENSIVE;
+	char *pa_mask_buff, *cb_mask_buff;
+	cpumask_var_t all_mask;
+	cpumask_var_t par_mask;
+	int rc;
+
+	get_online_cpus();
+
+	engine->pte_wq = alloc_workqueue(name, wq_flags, 1);
+	if (!engine->pte_wq) {
+		rc = -ENOMEM;
+		goto err;
+	}
+
+	if (!alloc_cpumask_var(&all_mask, GFP_KERNEL)) {
+		rc = -ENOMEM;
+		goto err_destroy_workqueue;
+	}
+
+	if (!alloc_cpumask_var(&par_mask, GFP_KERNEL)) {
+		rc = -ENOMEM;
+		goto err_free_all_mask;
+	}
+
+	cpumask_copy(par_mask, cpumask);
+	if (cpumask_empty(par_mask) ||
+	    cpumask_equal(par_mask, cpu_online_mask)) {
+		cpumask_copy(all_mask, cpu_online_mask);
+		cpumask_clear(par_mask);
+		while (!cpumask_empty(all_mask)) {
+			int cpu = cpumask_first(all_mask);
+
+			cpumask_set_cpu(cpu, par_mask);
+			cpumask_andnot(all_mask, all_mask,
+				       topology_sibling_cpumask(cpu));
+		}
+	}
+
+	cpumask_copy(all_mask, cpu_online_mask);
+
+	pa_mask_buff = (char *)__get_free_page(GFP_KERNEL);
+	if (!pa_mask_buff) {
+		rc = -ENOMEM;
+		goto err_free_par_mask;
+	}
+
+	cb_mask_buff = (char *)__get_free_page(GFP_KERNEL);
+	if (!cb_mask_buff) {
+		free_page((unsigned long)pa_mask_buff);
+		rc = -ENOMEM;
+		goto err_free_par_mask;
+	}
+
+	cpumap_print_to_pagebuf(true, pa_mask_buff, par_mask);
+	pa_mask_buff[PAGE_SIZE - 1] = '\0';
+	cpumap_print_to_pagebuf(true, cb_mask_buff, all_mask);
+	cb_mask_buff[PAGE_SIZE - 1] = '\0';
+
+	CDEBUG(D_INFO, "%s weight=%u plist='%s' cblist='%s'\n",
+	       name, cpumask_weight(par_mask),
+	       pa_mask_buff, cb_mask_buff);
+
+	free_page((unsigned long)cb_mask_buff);
+	free_page((unsigned long)pa_mask_buff);
+
+	engine->pte_weight = cpumask_weight(par_mask);
+	engine->pte_pinst  = padata_alloc_possible(engine->pte_wq);
+	if (!engine->pte_pinst) {
+		rc = -ENOMEM;
+		goto err_free_par_mask;
+	}
+
+	engine->pte_notifier.notifier_call = cl_ptask_cpumask_change_notify;
+	rc = padata_register_cpumask_notifier(engine->pte_pinst,
+					      &engine->pte_notifier);
+	if (rc)
+		goto err_free_padata;
+
+	rc = cl_ptengine_set_cpumask(engine, par_mask);
+	if (rc)
+		goto err_unregister;
+
+	rc = padata_start(engine->pte_pinst);
+	if (rc)
+		goto err_unregister;
+
+	free_cpumask_var(par_mask);
+	free_cpumask_var(all_mask);
+
+	put_online_cpus();
+	return 0;
+
+err_unregister:
+	padata_unregister_cpumask_notifier(engine->pte_pinst,
+					   &engine->pte_notifier);
+err_free_padata:
+	padata_free(engine->pte_pinst);
+err_free_par_mask:
+	free_cpumask_var(par_mask);
+err_free_all_mask:
+	free_cpumask_var(all_mask);
+err_destroy_workqueue:
+	destroy_workqueue(engine->pte_wq);
+err:
+	put_online_cpus();
+	return rc;
+}
+
+static void cl_ptengine_padata_fini(struct cl_ptask_engine *engine)
+{
+	padata_stop(engine->pte_pinst);
+	padata_unregister_cpumask_notifier(engine->pte_pinst,
+					   &engine->pte_notifier);
+	padata_free(engine->pte_pinst);
+	destroy_workqueue(engine->pte_wq);
+}
+
+#else  /* !CONFIG_PADATA */
+
+static int cl_ptengine_padata_init(struct cl_ptask_engine *engine,
+				   const char *name,
+				   const struct cpumask *cpumask)
+{
+	engine->pte_weight = 1;
+
+	return 0;
+}
+
+static void cl_ptengine_padata_fini(struct cl_ptask_engine *engine)
+{
+}
+#endif /* CONFIG_PADATA */
+
+struct cl_ptask_engine *cl_ptengine_init(const char *name,
+					 const struct cpumask *cpumask)
+{
+	struct cl_ptask_engine *engine;
+	int rc;
+
+	engine = kzalloc(sizeof(*engine), GFP_KERNEL);
+	if (!engine) {
+		rc = -ENOMEM;
+		goto err;
+	}
+
+	rc = cl_ptengine_padata_init(engine, name, cpumask);
+	if (rc)
+		goto err_free_engine;
+
+	return engine;
+
+err_free_engine:
+	kfree(engine);
+err:
+	return ERR_PTR(rc);
+}
+EXPORT_SYMBOL(cl_ptengine_init);
+
+void cl_ptengine_fini(struct cl_ptask_engine *engine)
+{
+	if (IS_ERR_OR_NULL(engine))
+		return;
+
+	cl_ptengine_padata_fini(engine);
+	kfree(engine);
+}
+EXPORT_SYMBOL(cl_ptengine_fini);
-- 
1.8.3.1


From jsimmons at infradead.org  Mon Nov 26 02:48:27 2018
From: jsimmons at infradead.org (James Simmons)
Date: Sun, 25 Nov 2018 21:48:27 -0500
Subject: [lustre-devel] [PATCH 11/12] lustre: mdc: use large xattr buffers
 for old servers
In-Reply-To: <1543200508-6838-1-git-send-email-jsimmons@infradead.org>
References: <1543200508-6838-1-git-send-email-jsimmons@infradead.org>
Message-ID: <1543200508-6838-12-git-send-email-jsimmons@infradead.org>

From: "John L. Hammond" <jhammond at whamcloud.com>

Pre 2.10.1 MDTs will crash when they receive a listxattr (MDS_GETXATTR
with OBD_MD_FLXATTRLS) RPC for an orphan or dead object. So for
clients connected to these older MDTs, try to avoid sending listxattr
RPCs by making the bulk getxattr (MDS_GETXATTR with OBD_MD_FLXATTRALL)
more likely to succeed and thereby reducing the chances of falling
ack to listxattr.

Signed-off-by: John L. Hammond <jhammond at whamcloud.com>
WC-bug-id: https://jira.whamcloud.com/browse/LU-10912
Reviewed-on: https://review.whamcloud.com/31990
Reviewed-by: Andreas Dilger <adilger at whamcloud.com>
Reviewed-by: Fan Yong <fan.yong at intel.com>
Reviewed-by: Oleg Drokin <green at whamcloud.com>
Signed-off-by: James Simmons <jsimmons at infradead.org>
---
 drivers/staging/lustre/lustre/mdc/mdc_locks.c | 14 ++++++++++++++
 1 file changed, 14 insertions(+)

diff --git a/drivers/staging/lustre/lustre/mdc/mdc_locks.c b/drivers/staging/lustre/lustre/mdc/mdc_locks.c
index 2cc2378..7d4ba9c 100644
--- a/drivers/staging/lustre/lustre/mdc/mdc_locks.c
+++ b/drivers/staging/lustre/lustre/mdc/mdc_locks.c
@@ -352,6 +352,20 @@ static void mdc_realloc_openmsg(struct ptlrpc_request *req,
 	lit = req_capsule_client_get(&req->rq_pill, &RMF_LDLM_INTENT);
 	lit->opc = IT_GETXATTR;
 
+	/* If the supplied buffer is too small then the server will
+	 * return -ERANGE and llite will fallback to using non cached
+	 * xattr operations. On servers before 2.10.1 a (non-cached)
+	 * listxattr RPC for an orphan or dead file causes an oops. So
+	 * let's try to avoid sending too small a buffer to too old a
+	 * server. This is effectively undoing the memory conservation
+	 * of LU-9417 when it would be *more* likely to crash the
+	 * server. See LU-9856.
+	 */
+	BUILD_BUG_ON(OBD_OCD_VERSION(3, 0, 53, 0) <= LUSTRE_VERSION_CODE);
+	if (exp->exp_connect_data.ocd_version < OBD_OCD_VERSION(2, 10, 1, 0))
+		ea_vals_buf_size = max_t(u32, ea_vals_buf_size,
+					 exp->exp_connect_data.ocd_max_easize);
+
 	/* pack the intended request */
 	mdc_pack_body(req, &op_data->op_fid1, op_data->op_valid,
 		      ea_vals_buf_size, -1, 0);
-- 
1.8.3.1


From neilb at suse.com  Mon Nov 26 03:00:59 2018
From: neilb at suse.com (NeilBrown)
Date: Mon, 26 Nov 2018 14:00:59 +1100
Subject: [lustre-devel] [PATCH 4/9] lustre: rename: DNE2 should return
 -EXDEV upon remote rename
In-Reply-To: <alpine.LFD.2.21.1811260130360.32255@casper.infradead.org>
References: <154295730810.2850.961218355189474016.stgit@noble>
 <154295732797.2850.16990175697450002727.stgit@noble>
 <alpine.LFD.2.21.1811260130360.32255@casper.infradead.org>
Message-ID: <87sgzor190.fsf@notabene.neil.brown.name>

On Mon, Nov 26 2018, James Simmons wrote:

> On Fri, 23 Nov 2018, NeilBrown wrote:
>
>> From: Lai Siyao <lai.siyao at intel.com>
>> 
>> DNE2 MDS should return -EXDEV upon remote rename, so that old
>> client can do rename with copy and delete, instead of fail
>> with -EREMOTE.
>
> Let me guess you were debugging the migration failures and fould this :-)
> I was doing the same thing.

Nope - I was just looking though all the missing patches to see if
anything was relevant. :-)

Thanks for all the "Reviewed-by"s !

NeilBrown

>
> Reviewed-by: James Simmons <jsimmons at infradead.org>
>  
>> Signed-off-by: Lai Siyao <lai.siyao at intel.com>
>> Change-Id: I68e8e99259065922f31bee5343be309380715674
>> WC-bug-id: https://jira.whamcloud.com/browse/LU-6660
>> Reviewed-on: http://review.whamcloud.com/15323
>> Reviewed-by: Andreas Dilger <andreas.dilger at intel.com>
>> Reviewed-by: wangdi <di.wang at intel.com>
>> Reviewed-by: Fan Yong <fan.yong at intel.com>
>> Signed-off-by: NeilBrown <neilb at suse.com>
>> ---
>>  drivers/staging/lustre/lustre/lmv/lmv_obd.c |    2 +-
>>  1 file changed, 1 insertion(+), 1 deletion(-)
>> 
>> diff --git a/drivers/staging/lustre/lustre/lmv/lmv_obd.c b/drivers/staging/lustre/lustre/lmv/lmv_obd.c
>> index 32bb9fca88c9..7e4ffeb15a63 100644
>> --- a/drivers/staging/lustre/lustre/lmv/lmv_obd.c
>> +++ b/drivers/staging/lustre/lustre/lmv/lmv_obd.c
>> @@ -1945,7 +1945,7 @@ static int lmv_rename(struct obd_export *exp, struct md_op_data *op_data,
>>  	}
>>  
>>  	rc = md_rename(target_exp, op_data, old, oldlen, new, newlen, request);
>> -	if (rc && rc != -EREMOTE)
>> +	if (rc && rc != -EXDEV)
>>  		return rc;
>>  
>>  	body = req_capsule_server_get(&(*request)->rq_pill, &RMF_MDT_BODY);
>> 
>> 
>> 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 832 bytes
Desc: not available
URL: <http://lists.lustre.org/pipermail/lustre-devel_lists.lustre.org/attachments/20181126/7da77410/attachment.sig>

From jsimmons at infradead.org  Mon Nov 26 03:47:26 2018
From: jsimmons at infradead.org (James Simmons)
Date: Mon, 26 Nov 2018 03:47:26 +0000 (GMT)
Subject: [lustre-devel] [PATCH 0/9] Assorted lustre patches - mostly
 from OpenSFS
In-Reply-To: <154295730810.2850.961218355189474016.stgit@noble>
References: <154295730810.2850.961218355189474016.stgit@noble>
Message-ID: <alpine.LFD.2.21.1811260344030.32255@casper.infradead.org>


> This is a miscellaneous set of patches for lustre-in-linux.
> Apart from the first, they are (bases on) patches from OpenSFS lustre.
> I've been trawling through OpenSFS looking for patches that aren't
> in linux-lustre.  Most of what I find are only relevant on the
> server, but these seem to be appropriate for the client.
> In most cases, I don't really know what is going on, so they
> might be inappropriate for the client.  I'm hoping that more
> knowledgeable people can put be straight.
> 
> This are just the first few - I think there are quite a few more
> (before we get to e.g. pfl which has been attempted yet).  If
> I've got these mostly right, I'll continue looking through
> the rest of my list.

To let you know I have ported the PFL patches for the linux client.
I'm currently doing testing to make sure it works correctly. Need
to work out some bugs but the major porting is done :-)

From jsimmons at infradead.org  Mon Nov 26 03:54:26 2018
From: jsimmons at infradead.org (James Simmons)
Date: Mon, 26 Nov 2018 03:54:26 +0000 (GMT)
Subject: [lustre-devel] lustre migration bug
Message-ID: <alpine.LFD.2.21.1811260347400.32255@casper.infradead.org>


Most of the major bugs outstanding are for migration functionality.
I did some tracing in the code (ll_migrate -> lmv_rename -> mdc_rename)
and found the failures is due to errors returned from after_reply() in
the ptlrpc layer. From that I see that the server is reporting that
the RPC packets are invalid. So far I don't see why exactly this is
the case. I will need to debug on the server side to figure it out.


From jsimmons at infradead.org  Mon Nov 26 04:52:34 2018
From: jsimmons at infradead.org (James Simmons)
Date: Mon, 26 Nov 2018 04:52:34 +0000 (GMT)
Subject: [lustre-devel] Do we need LOOKUP_CONTINUE in ll_revalidate_dentry()
Message-ID: <alpine.LFD.2.21.1811260404040.32255@casper.infradead.org>


Doing a compare to the OpenSFS branch I noticed this difference:

diff --git a/drivers/staging/lustre/lustre/llite/dcache.c 
b/drivers/staging/lustre/lustre/llite/dcache.c
index 11b82c63..6ee0ec9 100644
--- a/drivers/staging/lustre/lustre/llite/dcache.c
+++ b/drivers/staging/lustre/lustre/llite/dcache.c
@@ -254,7 +254,7 @@ static int ll_revalidate_dentry(struct dentry *dentry,
         * to this dentry, then its lock has not been revoked and the
         * path component is valid.
         */
-       if (lookup_flags & LOOKUP_PARENT)
+       if (lookup_flags & (LOOKUP_CONTINUE | LOOKUP_PARENT))
                return 1;

        /* Symlink - always valid as long as the dentry was found */

Is that needed for newer kernels?

From jsimmons at infradead.org  Mon Nov 26 05:06:25 2018
From: jsimmons at infradead.org (James Simmons)
Date: Mon, 26 Nov 2018 05:06:25 +0000 (GMT)
Subject: [lustre-devel] [PATCH RFC] lustre: llite: add LL_IOC_FUTIMES_3
Message-ID: <alpine.LFD.2.21.1811260455120.32255@casper.infradead.org>


Add a new regular file ioctl LL_IOC_FUTIMES_3 similar to futimes() but
which allows setting of all three inode timestamps. Use this ioctl
during HSM restore to ensure that the volatile file has the same
timestamps as the file to be restored. Strengthen sanity-hsm test_24a
to check that archive, release, and restore do not change a file's
ctime. Add sanity-hsm test_24e to check that tar will succeed when it
encounters a HSM released file.

*******************************************************************

Original pushed this to staging but Greg discussed about making this
a syscall instead. Andreas brought you this might be of use to other
file systems like XFS.

*******************************************************************

Signed-off-by: John L. Hammond <jhammond at whamcloud.com>
Intel-bug-id: https://jira.hpdd.intel.com/browse/LU-6213
Reviewed-on: http://review.whamcloud.com/13665
Reviewed-by: Andreas Dilger <adilger at whamcloud.com>
Reviewed-by: Jinshan Xiong <jinshan.xiong at gmail.com>
Reviewed-by: frank zago <fzago at cray.com>
Reviewed-by: Oleg Drokin <green at whamcloud.com>
Signed-off-by: James Simmons <jsimmons at infradead.org>
---
 .../lustre/include/uapi/linux/lustre/lustre_user.h | 10 +++++
 drivers/staging/lustre/lustre/llite/file.c         | 46 ++++++++++++++++++++++
 2 files changed, 56 insertions(+)

diff --git a/drivers/staging/lustre/include/uapi/linux/lustre/lustre_user.h b/drivers/staging/lustre/include/uapi/linux/lustre/lustre_user.h
index 9d553ce6..6904d6d 100644
--- a/drivers/staging/lustre/include/uapi/linux/lustre/lustre_user.h
+++ b/drivers/staging/lustre/include/uapi/linux/lustre/lustre_user.h
@@ -215,6 +215,15 @@ struct ost_id {
 #define DOSTID "%#llx:%llu"
 #define POSTID(oi) ostid_seq(oi), ostid_id(oi)
 
+struct ll_futimes_3 {
+	__u64 lfu_atime_sec;
+	__u64 lfu_atime_nsec;
+	__u64 lfu_mtime_sec;
+	__u64 lfu_mtime_nsec;
+	__u64 lfu_ctime_sec;
+	__u64 lfu_ctime_nsec;
+};
+
 /*
  * The ioctl naming rules:
  * LL_*     - works on the currently opened filehandle instead of parent dir
@@ -251,6 +260,7 @@ struct ost_id {
 #define LL_IOC_PATH2FID		 _IOR('f', 173, long)
 #define LL_IOC_GET_CONNECT_FLAGS	_IOWR('f', 174, __u64 *)
 #define LL_IOC_GET_MDTIDX	       _IOR('f', 175, int)
+#define LL_IOC_FUTIMES_3		_IOWR('f', 176, struct ll_futimes_3)
 
 /*	lustre_ioctl.h			177-210 */
 #define LL_IOC_HSM_STATE_GET		_IOR('f', 211, struct hsm_user_state)
diff --git a/drivers/staging/lustre/lustre/llite/file.c b/drivers/staging/lustre/lustre/llite/file.c
index 06789ec..aa54aad 100644
--- a/drivers/staging/lustre/lustre/llite/file.c
+++ b/drivers/staging/lustre/lustre/llite/file.c
@@ -2098,6 +2098,42 @@ static inline long ll_lease_type_from_fmode(fmode_t fmode)
 	       ((fmode & FMODE_WRITE) ? LL_LEASE_WRLCK : 0);
 }
 
+static int ll_file_futimes_3(struct file *file, const struct ll_futimes_3 *lfu)
+{
+	struct inode *inode = file_inode(file);
+	struct iattr ia = {
+		.ia_valid = ATTR_ATIME | ATTR_ATIME_SET |
+			    ATTR_MTIME | ATTR_MTIME_SET |
+			    ATTR_CTIME,
+		.ia_atime = {
+			.tv_sec = lfu->lfu_atime_sec,
+			.tv_nsec = lfu->lfu_atime_nsec,
+		},
+		.ia_mtime = {
+			.tv_sec = lfu->lfu_mtime_sec,
+			.tv_nsec = lfu->lfu_mtime_nsec,
+		},
+		.ia_ctime = {
+			.tv_sec = lfu->lfu_ctime_sec,
+			.tv_nsec = lfu->lfu_ctime_nsec,
+		},
+	};
+	int rc;
+
+	if (!capable(CAP_SYS_ADMIN))
+		return -EPERM;
+
+	if (!S_ISREG(inode->i_mode))
+		return -EINVAL;
+
+	inode_lock(inode);
+	rc = ll_setattr_raw(file_dentry(file), &ia, OP_XVALID_CTIME_SET,
+			    false);
+	inode_unlock(inode);
+
+	return rc;
+}
+
 /*
  * Give file access advices
  *
@@ -2552,6 +2588,16 @@ int ll_ioctl_fssetxattr(struct inode *inode, unsigned int cmd,
 		kfree(hui);
 		return rc;
 	}
+	case LL_IOC_FUTIMES_3: {
+		const struct ll_futimes_3 __user *lfu_user;
+		struct ll_futimes_3 lfu;
+
+		lfu_user = (const struct ll_futimes_3 __user *)arg;
+		if (copy_from_user(&lfu, lfu_user, sizeof(lfu)))
+			return -EFAULT;
+
+		return ll_file_futimes_3(file, &lfu);
+        }
 	case LL_IOC_LADVISE: {
 		struct llapi_ladvise_hdr *ladvise_hdr;
 		int alloc_size = sizeof(*ladvise_hdr);
-- 
1.8.3.1


From neilb at suse.com  Mon Nov 26 07:25:50 2018
From: neilb at suse.com (NeilBrown)
Date: Mon, 26 Nov 2018 18:25:50 +1100
Subject: [lustre-devel] Do we need LOOKUP_CONTINUE in
 ll_revalidate_dentry()
In-Reply-To: <alpine.LFD.2.21.1811260404040.32255@casper.infradead.org>
References: <alpine.LFD.2.21.1811260404040.32255@casper.infradead.org>
Message-ID: <87pnusqozl.fsf@notabene.neil.brown.name>

On Mon, Nov 26 2018, James Simmons wrote:

> Doing a compare to the OpenSFS branch I noticed this difference:
>
> diff --git a/drivers/staging/lustre/lustre/llite/dcache.c 
> b/drivers/staging/lustre/lustre/llite/dcache.c
> index 11b82c63..6ee0ec9 100644
> --- a/drivers/staging/lustre/lustre/llite/dcache.c
> +++ b/drivers/staging/lustre/lustre/llite/dcache.c
> @@ -254,7 +254,7 @@ static int ll_revalidate_dentry(struct dentry *dentry,
>          * to this dentry, then its lock has not been revoked and the
>          * path component is valid.
>          */
> -       if (lookup_flags & LOOKUP_PARENT)
> +       if (lookup_flags & (LOOKUP_CONTINUE | LOOKUP_PARENT))
>                 return 1;
>
>         /* Symlink - always valid as long as the dentry was found */
>
> Is that needed for newer kernels?

LOOKUP_CONTINUE disappeared in 2011

Commit: 49084c3bb205 ("kill LOOKUP_CONTINUE")

LOOKUP_PARENT is the new LOOKUP_CONTINUE.

NeilBrown
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 832 bytes
Desc: not available
URL: <http://lists.lustre.org/pipermail/lustre-devel_lists.lustre.org/attachments/20181126/1d212677/attachment.sig>

From jsimmons at infradead.org  Mon Nov 26 19:51:16 2018
From: jsimmons at infradead.org (James Simmons)
Date: Mon, 26 Nov 2018 19:51:16 +0000 (GMT)
Subject: [lustre-devel] Do we need LOOKUP_CONTINUE in
 ll_revalidate_dentry()
In-Reply-To: <87pnusqozl.fsf@notabene.neil.brown.name>
References: <alpine.LFD.2.21.1811260404040.32255@casper.infradead.org>
 <87pnusqozl.fsf@notabene.neil.brown.name>
Message-ID: <alpine.LFD.2.21.1811261950090.25858@casper.infradead.org>


> On Mon, Nov 26 2018, James Simmons wrote:
> 
> > Doing a compare to the OpenSFS branch I noticed this difference:
> >
> > diff --git a/drivers/staging/lustre/lustre/llite/dcache.c 
> > b/drivers/staging/lustre/lustre/llite/dcache.c
> > index 11b82c63..6ee0ec9 100644
> > --- a/drivers/staging/lustre/lustre/llite/dcache.c
> > +++ b/drivers/staging/lustre/lustre/llite/dcache.c
> > @@ -254,7 +254,7 @@ static int ll_revalidate_dentry(struct dentry *dentry,
> >          * to this dentry, then its lock has not been revoked and the
> >          * path component is valid.
> >          */
> > -       if (lookup_flags & LOOKUP_PARENT)
> > +       if (lookup_flags & (LOOKUP_CONTINUE | LOOKUP_PARENT))
> >                 return 1;
> >
> >         /* Symlink - always valid as long as the dentry was found */
> >
> > Is that needed for newer kernels?
> 
> LOOKUP_CONTINUE disappeared in 2011
> 
> Commit: 49084c3bb205 ("kill LOOKUP_CONTINUE")
> 
> LOOKUP_PARENT is the new LOOKUP_CONTINUE.

So its really

if (lookup_flags & (LOOKUP_PARENT | LOOKUP_PARENT))
	return  1;

in OpenSFS branch :-/

From adilger at whamcloud.com  Mon Nov 26 20:53:17 2018
From: adilger at whamcloud.com (Andreas Dilger)
Date: Mon, 26 Nov 2018 20:53:17 +0000
Subject: [lustre-devel] Do we need LOOKUP_CONTINUE in
 ll_revalidate_dentry()
In-Reply-To: <alpine.LFD.2.21.1811261950090.25858@casper.infradead.org>
References: <alpine.LFD.2.21.1811260404040.32255@casper.infradead.org>
 <87pnusqozl.fsf@notabene.neil.brown.name>
 <alpine.LFD.2.21.1811261950090.25858@casper.infradead.org>
Message-ID: <F28BADFF-E5B5-4E3F-81B7-46B33A951C64@whamcloud.com>

On Nov 26, 2018, at 12:51, James Simmons <jsimmons at infradead.org> wrote:
> 
> 
>> On Mon, Nov 26 2018, James Simmons wrote:
>> 
>>> Doing a compare to the OpenSFS branch I noticed this difference:
>>> 
>>> diff --git a/drivers/staging/lustre/lustre/llite/dcache.c 
>>> b/drivers/staging/lustre/lustre/llite/dcache.c
>>> index 11b82c63..6ee0ec9 100644
>>> --- a/drivers/staging/lustre/lustre/llite/dcache.c
>>> +++ b/drivers/staging/lustre/lustre/llite/dcache.c
>>> @@ -254,7 +254,7 @@ static int ll_revalidate_dentry(struct dentry *dentry,
>>>         * to this dentry, then its lock has not been revoked and the
>>>         * path component is valid.
>>>         */
>>> -       if (lookup_flags & LOOKUP_PARENT)
>>> +       if (lookup_flags & (LOOKUP_CONTINUE | LOOKUP_PARENT))
>>>                return 1;
>>> 
>>>        /* Symlink - always valid as long as the dentry was found */
>>> 
>>> Is that needed for newer kernels?
>> 
>> LOOKUP_CONTINUE disappeared in 2011
>> 
>> Commit: 49084c3bb205 ("kill LOOKUP_CONTINUE")
>> 
>> LOOKUP_PARENT is the new LOOKUP_CONTINUE.
> 
> So its really
> 
> if (lookup_flags & (LOOKUP_PARENT | LOOKUP_PARENT))
> 	return  1;
> 
> in OpenSFS branch :-/

/* Kernel 3.1 kills LOOKUP_CONTINUE, LOOKUP_PARENT is equivalent to it.
 * seem kernel commit 49084c3bb2055c401f3493c13edae14d49128ca0 */
#ifndef LOOKUP_CONTINUE
#define LOOKUP_CONTINUE LOOKUP_PARENT
#endif

That commit is in the 3.0 release, so LOOKUP_CONTINUE is needed for older
kernels.  Having the same value specified twice for newer kernels is not harmful, but this isn't needed for the upstream client.

Cheers, Andreas
---
Andreas Dilger
CTO Whamcloud





From neilb at suse.com  Mon Nov 26 21:15:47 2018
From: neilb at suse.com (NeilBrown)
Date: Tue, 27 Nov 2018 08:15:47 +1100
Subject: [lustre-devel] [PATCH RFC] lustre: llite: add LL_IOC_FUTIMES_3
In-Reply-To: <alpine.LFD.2.21.1811260455120.32255@casper.infradead.org>
References: <alpine.LFD.2.21.1811260455120.32255@casper.infradead.org>
Message-ID: <87mupvr14s.fsf@notabene.neil.brown.name>

On Mon, Nov 26 2018, James Simmons wrote:

> Add a new regular file ioctl LL_IOC_FUTIMES_3 similar to futimes() but
> which allows setting of all three inode timestamps. Use this ioctl
> during HSM restore to ensure that the volatile file has the same
> timestamps as the file to be restored. Strengthen sanity-hsm test_24a
> to check that archive, release, and restore do not change a file's
> ctime. Add sanity-hsm test_24e to check that tar will succeed when it
> encounters a HSM released file.

What is this used for?  What is the minimum functionality that would be
acceptable?

I'm guessing that it is used to migrate files between different levels
in the HSM storage hierarchy.  Why is it important that the ctime not
change?  How is the ctime used in a way that would be confused?
Would it be sufficient, for example, to be able to set the ctime to the
same as the mtime?

I'm imagining adding a flag to utimesat which is privileged and causes
ctime to be set to mtime, instead of to current time.  The justification
would be that it is for restoring from backups in a way that the
restored file doesn't look like it needs backup up immediately - not
even the metadata.

Thanks,
NeilBrown


>
> *******************************************************************
>
> Original pushed this to staging but Greg discussed about making this
> a syscall instead. Andreas brought you this might be of use to other
> file systems like XFS.
>
> *******************************************************************
>
> Signed-off-by: John L. Hammond <jhammond at whamcloud.com>
> Intel-bug-id: https://jira.hpdd.intel.com/browse/LU-6213
> Reviewed-on: http://review.whamcloud.com/13665
> Reviewed-by: Andreas Dilger <adilger at whamcloud.com>
> Reviewed-by: Jinshan Xiong <jinshan.xiong at gmail.com>
> Reviewed-by: frank zago <fzago at cray.com>
> Reviewed-by: Oleg Drokin <green at whamcloud.com>
> Signed-off-by: James Simmons <jsimmons at infradead.org>
> ---
>  .../lustre/include/uapi/linux/lustre/lustre_user.h | 10 +++++
>  drivers/staging/lustre/lustre/llite/file.c         | 46 ++++++++++++++++++++++
>  2 files changed, 56 insertions(+)
>
> diff --git a/drivers/staging/lustre/include/uapi/linux/lustre/lustre_user.h b/drivers/staging/lustre/include/uapi/linux/lustre/lustre_user.h
> index 9d553ce6..6904d6d 100644
> --- a/drivers/staging/lustre/include/uapi/linux/lustre/lustre_user.h
> +++ b/drivers/staging/lustre/include/uapi/linux/lustre/lustre_user.h
> @@ -215,6 +215,15 @@ struct ost_id {
>  #define DOSTID "%#llx:%llu"
>  #define POSTID(oi) ostid_seq(oi), ostid_id(oi)
>  
> +struct ll_futimes_3 {
> +	__u64 lfu_atime_sec;
> +	__u64 lfu_atime_nsec;
> +	__u64 lfu_mtime_sec;
> +	__u64 lfu_mtime_nsec;
> +	__u64 lfu_ctime_sec;
> +	__u64 lfu_ctime_nsec;
> +};
> +
>  /*
>   * The ioctl naming rules:
>   * LL_*     - works on the currently opened filehandle instead of parent dir
> @@ -251,6 +260,7 @@ struct ost_id {
>  #define LL_IOC_PATH2FID		 _IOR('f', 173, long)
>  #define LL_IOC_GET_CONNECT_FLAGS	_IOWR('f', 174, __u64 *)
>  #define LL_IOC_GET_MDTIDX	       _IOR('f', 175, int)
> +#define LL_IOC_FUTIMES_3		_IOWR('f', 176, struct ll_futimes_3)
>  
>  /*	lustre_ioctl.h			177-210 */
>  #define LL_IOC_HSM_STATE_GET		_IOR('f', 211, struct hsm_user_state)
> diff --git a/drivers/staging/lustre/lustre/llite/file.c b/drivers/staging/lustre/lustre/llite/file.c
> index 06789ec..aa54aad 100644
> --- a/drivers/staging/lustre/lustre/llite/file.c
> +++ b/drivers/staging/lustre/lustre/llite/file.c
> @@ -2098,6 +2098,42 @@ static inline long ll_lease_type_from_fmode(fmode_t fmode)
>  	       ((fmode & FMODE_WRITE) ? LL_LEASE_WRLCK : 0);
>  }
>  
> +static int ll_file_futimes_3(struct file *file, const struct ll_futimes_3 *lfu)
> +{
> +	struct inode *inode = file_inode(file);
> +	struct iattr ia = {
> +		.ia_valid = ATTR_ATIME | ATTR_ATIME_SET |
> +			    ATTR_MTIME | ATTR_MTIME_SET |
> +			    ATTR_CTIME,
> +		.ia_atime = {
> +			.tv_sec = lfu->lfu_atime_sec,
> +			.tv_nsec = lfu->lfu_atime_nsec,
> +		},
> +		.ia_mtime = {
> +			.tv_sec = lfu->lfu_mtime_sec,
> +			.tv_nsec = lfu->lfu_mtime_nsec,
> +		},
> +		.ia_ctime = {
> +			.tv_sec = lfu->lfu_ctime_sec,
> +			.tv_nsec = lfu->lfu_ctime_nsec,
> +		},
> +	};
> +	int rc;
> +
> +	if (!capable(CAP_SYS_ADMIN))
> +		return -EPERM;
> +
> +	if (!S_ISREG(inode->i_mode))
> +		return -EINVAL;
> +
> +	inode_lock(inode);
> +	rc = ll_setattr_raw(file_dentry(file), &ia, OP_XVALID_CTIME_SET,
> +			    false);
> +	inode_unlock(inode);
> +
> +	return rc;
> +}
> +
>  /*
>   * Give file access advices
>   *
> @@ -2552,6 +2588,16 @@ int ll_ioctl_fssetxattr(struct inode *inode, unsigned int cmd,
>  		kfree(hui);
>  		return rc;
>  	}
> +	case LL_IOC_FUTIMES_3: {
> +		const struct ll_futimes_3 __user *lfu_user;
> +		struct ll_futimes_3 lfu;
> +
> +		lfu_user = (const struct ll_futimes_3 __user *)arg;
> +		if (copy_from_user(&lfu, lfu_user, sizeof(lfu)))
> +			return -EFAULT;
> +
> +		return ll_file_futimes_3(file, &lfu);
> +        }
>  	case LL_IOC_LADVISE: {
>  		struct llapi_ladvise_hdr *ladvise_hdr;
>  		int alloc_size = sizeof(*ladvise_hdr);
> -- 
> 1.8.3.1
>
> _______________________________________________
> lustre-devel mailing list
> lustre-devel at lists.lustre.org
> http://lists.lustre.org/listinfo.cgi/lustre-devel-lustre.org
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 832 bytes
Desc: not available
URL: <http://lists.lustre.org/pipermail/lustre-devel_lists.lustre.org/attachments/20181127/43e379e2/attachment.sig>

From adilger at whamcloud.com  Mon Nov 26 21:46:01 2018
From: adilger at whamcloud.com (Andreas Dilger)
Date: Mon, 26 Nov 2018 21:46:01 +0000
Subject: [lustre-devel] [PATCH RFC] lustre: llite: add LL_IOC_FUTIMES_3
In-Reply-To: <87mupvr14s.fsf@notabene.neil.brown.name>
References: <alpine.LFD.2.21.1811260455120.32255@casper.infradead.org>
 <87mupvr14s.fsf@notabene.neil.brown.name>
Message-ID: <075A6FB1-D778-46F7-B780-FC9229902690@whamcloud.com>

On Nov 26, 2018, at 14:15, NeilBrown <neilb at suse.com> wrote:
> 
> On Mon, Nov 26 2018, James Simmons wrote:
> 
>> Add a new regular file ioctl LL_IOC_FUTIMES_3 similar to futimes() but
>> which allows setting of all three inode timestamps. Use this ioctl
>> during HSM restore to ensure that the volatile file has the same
>> timestamps as the file to be restored. Strengthen sanity-hsm test_24a
>> to check that archive, release, and restore do not change a file's
>> ctime. Add sanity-hsm test_24e to check that tar will succeed when it
>> encounters a HSM released file.
> 
> What is this used for?  What is the minimum functionality that would be
> acceptable?

Correct, this is for restoring files from HSM archive or migrating between
different tiers of storage within the filesystem.  This is transparent to the user, and we don't want the migration/restore to cause the file to be backed up again.

> I'm guessing that it is used to migrate files between different levels
> in the HSM storage hierarchy.  Why is it important that the ctime not
> change?  How is the ctime used in a way that would be confused?
> Would it be sufficient, for example, to be able to set the ctime to the
> same as the mtime?

There are lots of different kinds of backup tools out there, I can't
comment on whether ctime == mtime is sufficient to avoid triggering
another backup cycle.  Keeping the same timestamps over this process
is safest.  It seems that XFS has a similar requirement, and it is
using "FMODE_NOCMTIME", but that only avoids updating the timestamps,
it doesn't allow setting them.

> I'm imagining adding a flag to utimesat which is privileged and causes
> ctime to be set to mtime, instead of to current time.  The justification
> would be that it is for restoring from backups in a way that the
> restored file doesn't look like it needs backup up immediately - not
> even the metadata.
> 
> Thanks,
> NeilBrown
> 
> 
>> 
>> *******************************************************************
>> 
>> Original pushed this to staging but Greg discussed about making this
>> a syscall instead. Andreas brought you this might be of use to other
>> file systems like XFS.
>> 
>> *******************************************************************
>> 
>> Signed-off-by: John L. Hammond <jhammond at whamcloud.com>
>> Intel-bug-id: https://jira.hpdd.intel.com/browse/LU-6213
>> Reviewed-on: http://review.whamcloud.com/13665
>> Reviewed-by: Andreas Dilger <adilger at whamcloud.com>
>> Reviewed-by: Jinshan Xiong <jinshan.xiong at gmail.com>
>> Reviewed-by: frank zago <fzago at cray.com>
>> Reviewed-by: Oleg Drokin <green at whamcloud.com>
>> Signed-off-by: James Simmons <jsimmons at infradead.org>
>> ---
>> .../lustre/include/uapi/linux/lustre/lustre_user.h | 10 +++++
>> drivers/staging/lustre/lustre/llite/file.c         | 46 ++++++++++++++++++++++
>> 2 files changed, 56 insertions(+)
>> 
>> diff --git a/drivers/staging/lustre/include/uapi/linux/lustre/lustre_user.h b/drivers/staging/lustre/include/uapi/linux/lustre/lustre_user.h
>> index 9d553ce6..6904d6d 100644
>> --- a/drivers/staging/lustre/include/uapi/linux/lustre/lustre_user.h
>> +++ b/drivers/staging/lustre/include/uapi/linux/lustre/lustre_user.h
>> @@ -215,6 +215,15 @@ struct ost_id {
>> #define DOSTID "%#llx:%llu"
>> #define POSTID(oi) ostid_seq(oi), ostid_id(oi)
>> 
>> +struct ll_futimes_3 {
>> +	__u64 lfu_atime_sec;
>> +	__u64 lfu_atime_nsec;
>> +	__u64 lfu_mtime_sec;
>> +	__u64 lfu_mtime_nsec;
>> +	__u64 lfu_ctime_sec;
>> +	__u64 lfu_ctime_nsec;
>> +};
>> +
>> /*
>>  * The ioctl naming rules:
>>  * LL_*     - works on the currently opened filehandle instead of parent dir
>> @@ -251,6 +260,7 @@ struct ost_id {
>> #define LL_IOC_PATH2FID		 _IOR('f', 173, long)
>> #define LL_IOC_GET_CONNECT_FLAGS	_IOWR('f', 174, __u64 *)
>> #define LL_IOC_GET_MDTIDX	       _IOR('f', 175, int)
>> +#define LL_IOC_FUTIMES_3		_IOWR('f', 176, struct ll_futimes_3)
>> 
>> /*	lustre_ioctl.h			177-210 */
>> #define LL_IOC_HSM_STATE_GET		_IOR('f', 211, struct hsm_user_state)
>> diff --git a/drivers/staging/lustre/lustre/llite/file.c b/drivers/staging/lustre/lustre/llite/file.c
>> index 06789ec..aa54aad 100644
>> --- a/drivers/staging/lustre/lustre/llite/file.c
>> +++ b/drivers/staging/lustre/lustre/llite/file.c
>> @@ -2098,6 +2098,42 @@ static inline long ll_lease_type_from_fmode(fmode_t fmode)
>> 	       ((fmode & FMODE_WRITE) ? LL_LEASE_WRLCK : 0);
>> }
>> 
>> +static int ll_file_futimes_3(struct file *file, const struct ll_futimes_3 *lfu)
>> +{
>> +	struct inode *inode = file_inode(file);
>> +	struct iattr ia = {
>> +		.ia_valid = ATTR_ATIME | ATTR_ATIME_SET |
>> +			    ATTR_MTIME | ATTR_MTIME_SET |
>> +			    ATTR_CTIME,
>> +		.ia_atime = {
>> +			.tv_sec = lfu->lfu_atime_sec,
>> +			.tv_nsec = lfu->lfu_atime_nsec,
>> +		},
>> +		.ia_mtime = {
>> +			.tv_sec = lfu->lfu_mtime_sec,
>> +			.tv_nsec = lfu->lfu_mtime_nsec,
>> +		},
>> +		.ia_ctime = {
>> +			.tv_sec = lfu->lfu_ctime_sec,
>> +			.tv_nsec = lfu->lfu_ctime_nsec,
>> +		},
>> +	};
>> +	int rc;
>> +
>> +	if (!capable(CAP_SYS_ADMIN))
>> +		return -EPERM;
>> +
>> +	if (!S_ISREG(inode->i_mode))
>> +		return -EINVAL;
>> +
>> +	inode_lock(inode);
>> +	rc = ll_setattr_raw(file_dentry(file), &ia, OP_XVALID_CTIME_SET,
>> +			    false);
>> +	inode_unlock(inode);
>> +
>> +	return rc;
>> +}
>> +
>> /*
>>  * Give file access advices
>>  *
>> @@ -2552,6 +2588,16 @@ int ll_ioctl_fssetxattr(struct inode *inode, unsigned int cmd,
>> 		kfree(hui);
>> 		return rc;
>> 	}
>> +	case LL_IOC_FUTIMES_3: {
>> +		const struct ll_futimes_3 __user *lfu_user;
>> +		struct ll_futimes_3 lfu;
>> +
>> +		lfu_user = (const struct ll_futimes_3 __user *)arg;
>> +		if (copy_from_user(&lfu, lfu_user, sizeof(lfu)))
>> +			return -EFAULT;
>> +
>> +		return ll_file_futimes_3(file, &lfu);
>> +        }
>> 	case LL_IOC_LADVISE: {
>> 		struct llapi_ladvise_hdr *ladvise_hdr;
>> 		int alloc_size = sizeof(*ladvise_hdr);
>> -- 
>> 1.8.3.1
>> 
>> _______________________________________________
>> lustre-devel mailing list
>> lustre-devel at lists.lustre.org
>> http://lists.lustre.org/listinfo.cgi/lustre-devel-lustre.org
> _______________________________________________
> lustre-devel mailing list
> lustre-devel at lists.lustre.org
> http://lists.lustre.org/listinfo.cgi/lustre-devel-lustre.org

Cheers, Andreas
---
Andreas Dilger
CTO Whamcloud





From neilb at suse.com  Mon Nov 26 22:28:35 2018
From: neilb at suse.com (NeilBrown)
Date: Tue, 27 Nov 2018 09:28:35 +1100
Subject: [lustre-devel] [PATCH RFC] lustre: llite: add LL_IOC_FUTIMES_3
In-Reply-To: <075A6FB1-D778-46F7-B780-FC9229902690@whamcloud.com>
References: <alpine.LFD.2.21.1811260455120.32255@casper.infradead.org>
 <87mupvr14s.fsf@notabene.neil.brown.name>
 <075A6FB1-D778-46F7-B780-FC9229902690@whamcloud.com>
Message-ID: <87k1kzqxrg.fsf@notabene.neil.brown.name>

On Mon, Nov 26 2018, Andreas Dilger wrote:

> On Nov 26, 2018, at 14:15, NeilBrown <neilb at suse.com> wrote:
>> 
>> On Mon, Nov 26 2018, James Simmons wrote:
>> 
>>> Add a new regular file ioctl LL_IOC_FUTIMES_3 similar to futimes() but
>>> which allows setting of all three inode timestamps. Use this ioctl
>>> during HSM restore to ensure that the volatile file has the same
>>> timestamps as the file to be restored. Strengthen sanity-hsm test_24a
>>> to check that archive, release, and restore do not change a file's
>>> ctime. Add sanity-hsm test_24e to check that tar will succeed when it
>>> encounters a HSM released file.
>> 
>> What is this used for?  What is the minimum functionality that would be
>> acceptable?
>
> Correct, this is for restoring files from HSM archive or migrating between
> different tiers of storage within the filesystem.  This is transparent to the user, and we don't want the migration/restore to cause the file to be backed up again.
>
>> I'm guessing that it is used to migrate files between different levels
>> in the HSM storage hierarchy.  Why is it important that the ctime not
>> change?  How is the ctime used in a way that would be confused?
>> Would it be sufficient, for example, to be able to set the ctime to the
>> same as the mtime?
>
> There are lots of different kinds of backup tools out there, I can't
> comment on whether ctime == mtime is sufficient to avoid triggering
> another backup cycle.  Keeping the same timestamps over this process
> is safest.  It seems that XFS has a similar requirement, and it is
> using "FMODE_NOCMTIME", but that only avoids updating the timestamps,
> it doesn't allow setting them.

Interesting.. FMODE_NOCMTIME was added 10 years ago (next month) with
the comment

+ * Currently a special hack for the XFS open_by_handle ioctl, but we'll
+ * hopefully graduate it to a proper O_CMTIME flag supported by open(2) soon.

"soon" is an imprecise term of course...

But as you say, there is no way to set the ctime, even at file creation.
I wonder how this is used.

Maybe it would make sense to add a flag to utimensat() to expect a third
time-stamp and use that for ctime.  That would be a fairly simple and
safe API change.
It would be nice to creae an O_NOCMTIME and plumb it through properly
too.
Possibly utimensat(AT_3TIMES) would required an fd that was opened with
N_NOCMTIME...

NeilBrown
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 832 bytes
Desc: not available
URL: <http://lists.lustre.org/pipermail/lustre-devel_lists.lustre.org/attachments/20181127/f72b03d2/attachment.sig>

From neilb at suse.com  Tue Nov 27 02:21:08 2018
From: neilb at suse.com (NeilBrown)
Date: Tue, 27 Nov 2018 13:21:08 +1100
Subject: [lustre-devel] [PATCH 7/9] lustre: lnet: Stop MLX5 triggering a
 dump_cqe
In-Reply-To: <alpine.LFD.2.21.1811260146540.32255@casper.infradead.org>
References: <154295730810.2850.961218355189474016.stgit@noble>
 <154295732806.2850.603181458106225374.stgit@noble>
 <alpine.LFD.2.21.1811260146540.32255@casper.infradead.org>
Message-ID: <87bm6bqmzv.fsf@notabene.neil.brown.name>

On Mon, Nov 26 2018, James Simmons wrote:

>> From: Doug Oucharek <doug.s.oucharek at intel.com>
>> 
>> We have found that MLX5 will trigger a dump_cqe if we don't
>> invalidate the rkey on a newly allocated MR for FastReg usage.
>> 
>> This fix just tags the MR as invalid on its creation if we are
>> using FastReg and that will force it to do an invalidate of the
>> rkey on first usage.
>
> I pushed this one already, see https://lkml.org/lkml/2018/3/16/1410.
> Dan felt this was more a infiniband layer bug that needed to be fixed.
> It could be fixed already upstream or if it is not once this problem
> is reported we will need to work the rdma group to fix it.

Thanks.  I've dropped it for now.

If I had any idea about infiniband, I might look at the MLX driver - but
I don't :-(

NeilBrown
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 832 bytes
Desc: not available
URL: <http://lists.lustre.org/pipermail/lustre-devel_lists.lustre.org/attachments/20181127/90c3732c/attachment.sig>

From neilb at suse.com  Tue Nov 27 02:32:48 2018
From: neilb at suse.com (NeilBrown)
Date: Tue, 27 Nov 2018 13:32:48 +1100
Subject: [lustre-devel] [PATCH 6/9] lustre: obdclass: health_check to
 report unhealthy upon LBUG
In-Reply-To: <alpine.LFD.2.21.1811260146120.32255@casper.infradead.org>
References: <154295730810.2850.961218355189474016.stgit@noble>
 <154295732803.2850.16198081335816276527.stgit@noble>
 <alpine.LFD.2.21.1811260146120.32255@casper.infradead.org>
Message-ID: <878t1fqmgf.fsf@notabene.neil.brown.name>

On Mon, Nov 26 2018, James Simmons wrote:

>> From: Bruno Faccini <bruno.faccini at intel.com>
>> 
>> When a LBUG has occurred, without panic_on_lbug being set, health_check
>> /proc file must return an unhealthy state.
>
> I pushed this one to Greg which was disliked since it breaks the one item 
> per sysfs rule. See 
>
> https://lore.kernel.org/patchwork/patch/755571
>
> I did start a proper port to sysfs at 
> https://review.whamcloud.com/#/c/25631
>
> but it needs to be updated. I do like Andreas idea of a sysfs and debugfs
> file since lctl get_param will return the results from both together.
> We could land it as is and update the sysfs handling at a latter date 
> (shouldn't be too far down the road). Here is my review in case you want
> to land it.
>
> Reviewed-by: James Simmons <jsimmons at infradead.org>
>

Thanks, but the patch as it stands it totally broken - I add code
immediately after a 'return'. :-(

I'll just discard this patch.

Thanks,
NeilBrown


>> Signed-off-by: Bruno Faccini <bruno.faccini at intel.com>
>> WC-bug-id: https://jira.whamcloud.com/browse/LU-7486
>> Reviewed-on: http://review.whamcloud.com/17981
>> Reviewed-by: Bobi Jam <bobijam at hotmail.com>
>> Reviewed-by: Niu Yawei <yawei.niu at intel.com>
>> Reviewed-by: James Simmons <uja.ornl at yahoo.com>
>> Reviewed-by: Oleg Drokin <oleg.drokin at intel.com>
>> Signed-off-by: NeilBrown <neilb at suse.com>
>> ---
>>  drivers/staging/lustre/lustre/obdclass/obd_sysfs.c |    4 +++-
>>  1 file changed, 3 insertions(+), 1 deletion(-)
>> 
>> diff --git a/drivers/staging/lustre/lustre/obdclass/obd_sysfs.c b/drivers/staging/lustre/lustre/obdclass/obd_sysfs.c
>> index 6669c235dd51..5fd30a8e2b44 100644
>> --- a/drivers/staging/lustre/lustre/obdclass/obd_sysfs.c
>> +++ b/drivers/staging/lustre/lustre/obdclass/obd_sysfs.c
>> @@ -173,8 +173,10 @@ health_check_show(struct kobject *kobj, struct attribute *attr, char *buf)
>>  	int i;
>>  	size_t len = 0;
>>  
>> -	if (libcfs_catastrophe)
>> +	if (libcfs_catastrophe) {
>>  		return sprintf(buf, "LBUG\n");
>> +		healthy = false;
>> +	}
>>  
>>  	read_lock(&obd_dev_lock);
>>  	for (i = 0; i < class_devno_max(); i++) {
>> 
>> 
>> 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 832 bytes
Desc: not available
URL: <http://lists.lustre.org/pipermail/lustre-devel_lists.lustre.org/attachments/20181127/a713a21f/attachment.sig>

From neilb at suse.com  Tue Nov 27 03:01:30 2018
From: neilb at suse.com (NeilBrown)
Date: Tue, 27 Nov 2018 14:01:30 +1100
Subject: [lustre-devel] [PATCH 06/12] lustre: mdc: don't add to page
 cache upon failure
In-Reply-To: <1543200508-6838-7-git-send-email-jsimmons@infradead.org>
References: <1543200508-6838-1-git-send-email-jsimmons@infradead.org>
 <1543200508-6838-7-git-send-email-jsimmons@infradead.org>
Message-ID: <875zwjql4l.fsf@notabene.neil.brown.name>

On Sun, Nov 25 2018, James Simmons wrote:

> From: Lai Siyao <lai.siyao at whamcloud.com>
>
> Reading directory pages may fail on MDS, in this case client should
> not cache a non-up-to-date directory page, because it will cause
> a later read on the same page fail.
>
> Signed-off-by: Lai Siyao <lai.siyao at whamcloud.com>
> WC-bug-id: https://jira.whamcloud.com/browse/LU-5461
> Reviewed-on: http://review.whamcloud.com/11450
> Reviewed-by: Fan Yong <fan.yong at intel.com>
> Reviewed-by: John L. Hammond <jhammond at whamcloud.com>
> Reviewed-by: Oleg Drokin <green at whamcloud.com>
> Signed-off-by: James Simmons <jsimmons at infradead.org>
> ---
>  drivers/staging/lustre/lustre/mdc/mdc_request.c | 5 ++++-
>  1 file changed, 4 insertions(+), 1 deletion(-)
>
> diff --git a/drivers/staging/lustre/lustre/mdc/mdc_request.c b/drivers/staging/lustre/lustre/mdc/mdc_request.c
> index 1072b66..09b30ef 100644
> --- a/drivers/staging/lustre/lustre/mdc/mdc_request.c
> +++ b/drivers/staging/lustre/lustre/mdc/mdc_request.c
> @@ -1212,7 +1212,10 @@ static int mdc_read_page_remote(void *data, struct page *page0)
>  	}
>  
>  	rc = mdc_getpage(rp->rp_exp, fid, rp->rp_off, page_pool, npages, &req);
> -	if (!rc) {
> +	if (rc < 0) {
> +		/* page0 is special, which was added into page cache early */
> +		delete_from_page_cache(page0);

This looks wrong to me.  We shouldn't need to delete the page from the
page-cache.
It won't be marked up-to-date, so why will that cause an error on a
later read???

Well, because mdc_page_locate() finds a page and if it isn't up-to-date,
it returns -EIO.  Why does it do that?  If it found a PageError() page,
then it might be reasonable to return -EIO.

Why not just return the page that was found, and let the caller check if
it is Uptodate?

Well, because mdc_read_page() handles a successful page return from
mdc_page_locate() as a hash collision.

I guess I need to understand how lustre maps a hash to a directory
block, and then how it handles collisions...

The reason this jumped out at me is that it looks like it might be
racy.  Adding a page and then removing it might leave a window where
some other thread can find the page.  That is not a problem is a
non-up-to-date page just means we should wait for it.  But if it can
cause an error, then maybe the race is a real problem.

But maybe there is some higher level locking...

NeilBrown
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 832 bytes
Desc: not available
URL: <http://lists.lustre.org/pipermail/lustre-devel_lists.lustre.org/attachments/20181127/5e6433ff/attachment.sig>

From neilb at suse.com  Tue Nov 27 03:13:17 2018
From: neilb at suse.com (NeilBrown)
Date: Tue, 27 Nov 2018 14:13:17 +1100
Subject: [lustre-devel] [PATCH 08/12] lustre: mdc: propagate changelog
 errors to readers
In-Reply-To: <1543200508-6838-9-git-send-email-jsimmons@infradead.org>
References: <1543200508-6838-1-git-send-email-jsimmons@infradead.org>
 <1543200508-6838-9-git-send-email-jsimmons@infradead.org>
Message-ID: <8736rnqkky.fsf@notabene.neil.brown.name>

On Sun, Nov 25 2018, James Simmons wrote:

> From: "John L. Hammond" <jhammond at whamcloud.com>
>
> Store errors encountered by the changelog llog reader thread in the
> crs_err field of struct changelog_reader_state so that they can be
> propageted to changelog readers. In chlg_read() improve the error and
> EOF reporting. Return -ERESTARTSYS when the blocked reader is
> interrupted. Replace uses of wait_event_idle() with
> ait_event_interruptible().
>
> Signed-off-by: John L. Hammond <jhammond at whamcloud.com>
> WC-bug-id: https://jira.whamcloud.com/browse/LU-10218
> Reviewed-on: https://review.whamcloud.com/30040
> Reviewed-by: Quentin Bouget <quentin.bouget at cea.fr>
> Reviewed-by: Henri Doreau <henri.doreau at cea.fr>
> Reviewed-by: Oleg Drokin <green at whamcloud.com>
> Signed-off-by: James Simmons <jsimmons at infradead.org>
> ---
>  drivers/staging/lustre/lustre/mdc/mdc_changelog.c | 45 ++++++++++++++---------
>  1 file changed, 28 insertions(+), 17 deletions(-)
>
> diff --git a/drivers/staging/lustre/lustre/mdc/mdc_changelog.c b/drivers/staging/lustre/lustre/mdc/mdc_changelog.c
> index becdee8..811a36a 100644
> --- a/drivers/staging/lustre/lustre/mdc/mdc_changelog.c
> +++ b/drivers/staging/lustre/lustre/mdc/mdc_changelog.c
> @@ -71,7 +71,7 @@ struct chlg_reader_state {
>  	/* Producer thread (if any) */
>  	struct task_struct	*crs_prod_task;
>  	/* An error occurred that prevents from reading further */
> -	bool			 crs_err;
> +	int			 crs_err;
>  	/* EOF, no more records available */
>  	bool			 crs_eof;
>  	/* Desired start position */
> @@ -148,9 +148,9 @@ static int chlg_read_cat_process_cb(const struct lu_env *env,
>  	       PFID(&rec->cr.cr_tfid), PFID(&rec->cr.cr_pfid),
>  	       rec->cr.cr_namelen, changelog_rec_name(&rec->cr));
>  
> -	wait_event_idle(crs->crs_waitq_prod,
> -			(crs->crs_rec_count < CDEV_CHLG_MAX_PREFETCH ||
> -			 kthread_should_stop()));
> +	wait_event_interruptible(crs->crs_waitq_prod,
> +				 crs->crs_rec_count < CDEV_CHLG_MAX_PREFETCH ||
> +				 kthread_should_stop());

This is wrong.  Not harmful, but wrong.
This is in a kernel thread, so it doesn't expect to receive signals.
So allowing an abort on a signal is pointless and misleading.

So I've removed this chunk from the patch, and also a later chunk which
uses wait_event_interruptible() in a kthread.

Other places where wait_event_interruptible() are used make sense.

Thanks,
NeilBrown


>  
>  	if (kthread_should_stop())
>  		return LLOG_PROC_BREAK;
> @@ -231,7 +231,7 @@ static int chlg_load(void *args)
>  
>  err_out:
>  	if (rc < 0)
> -		crs->crs_err = true;
> +		crs->crs_err = rc;
>  
>  	wake_up_all(&crs->crs_waitq_cons);
>  
> @@ -241,7 +241,7 @@ static int chlg_load(void *args)
>  	if (ctx)
>  		llog_ctxt_put(ctx);
>  
> -	wait_event_idle(crs->crs_waitq_prod, kthread_should_stop());
> +	wait_event_interruptible(crs->crs_waitq_prod, kthread_should_stop());
>  
>  	return rc;
>  }
> @@ -264,13 +264,20 @@ static ssize_t chlg_read(struct file *file, char __user *buff, size_t count,
>  	struct chlg_reader_state *crs = file->private_data;
>  	struct chlg_rec_entry *rec;
>  	struct chlg_rec_entry *tmp;
> -	ssize_t  written_total = 0;
> +	ssize_t written_total = 0;
>  	LIST_HEAD(consumed);
> +	size_t rc;
> +
> +	if (file->f_flags & O_NONBLOCK && crs->crs_rec_count == 0) {
> +		if (crs->crs_err < 0)
> +			return crs->crs_err;
> +		else if (crs->crs_eof)
> +			return 0;
> +		else
> +			return -EAGAIN;
> +	}
>  
> -	if (file->f_flags & O_NONBLOCK && crs->crs_rec_count == 0)
> -		return -EAGAIN;
> -
> -	wait_event_idle(crs->crs_waitq_cons,
> +	rc = wait_event_interruptible(crs->crs_waitq_cons,
>  			crs->crs_rec_count > 0 || crs->crs_eof || crs->crs_err);
>  
>  	mutex_lock(&crs->crs_lock);
> @@ -279,8 +286,7 @@ static ssize_t chlg_read(struct file *file, char __user *buff, size_t count,
>  			break;
>  
>  		if (copy_to_user(buff, rec->enq_record, rec->enq_length)) {
> -			if (written_total == 0)
> -				written_total = -EFAULT;
> +			rc = -EFAULT;
>  			break;
>  		}
>  
> @@ -294,15 +300,19 @@ static ssize_t chlg_read(struct file *file, char __user *buff, size_t count,
>  	}
>  	mutex_unlock(&crs->crs_lock);
>  
> -	if (written_total > 0)
> +	if (written_total > 0) {
> +		rc = written_total;
>  		wake_up_all(&crs->crs_waitq_prod);
> +	} else if (rc == 0) {
> +		rc = crs->crs_err;
> +	}
>  
>  	list_for_each_entry_safe(rec, tmp, &consumed, enq_linkage)
>  		enq_record_delete(rec);
>  
>  	*ppos = crs->crs_start_offset;
>  
> -	return written_total;
> +	return rc;
>  }
>  
>  /**
> @@ -509,15 +519,16 @@ static int chlg_release(struct inode *inode, struct file *file)
>  	struct chlg_reader_state *crs = file->private_data;
>  	struct chlg_rec_entry *rec;
>  	struct chlg_rec_entry *tmp;
> +	int rc = 0;
>  
>  	if (crs->crs_prod_task)
> -		kthread_stop(crs->crs_prod_task);
> +		rc = kthread_stop(crs->crs_prod_task);
>  
>  	list_for_each_entry_safe(rec, tmp, &crs->crs_rec_queue, enq_linkage)
>  		enq_record_delete(rec);
>  
>  	kfree(crs);
> -	return 0;
> +	return rc;
>  }
>  
>  /**
> -- 
> 1.8.3.1
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 832 bytes
Desc: not available
URL: <http://lists.lustre.org/pipermail/lustre-devel_lists.lustre.org/attachments/20181127/82e37de0/attachment.sig>

From neilb at suse.com  Tue Nov 27 04:01:03 2018
From: neilb at suse.com (NeilBrown)
Date: Tue, 27 Nov 2018 15:01:03 +1100
Subject: [lustre-devel] [PATCH 09/12] lustre: obdclass: obd_device
 improvement
In-Reply-To: <1543200508-6838-10-git-send-email-jsimmons@infradead.org>
References: <1543200508-6838-1-git-send-email-jsimmons@infradead.org>
 <1543200508-6838-10-git-send-email-jsimmons@infradead.org>
Message-ID: <87zhtvp3sw.fsf@notabene.neil.brown.name>

On Sun, Nov 25 2018, James Simmons wrote:

> From: Alexander Boyko <c17825 at cray.com>
>
> The patch removes self exports from obd's reference counting which
> allows to avoid freeing of self exports by zombie thread.
> A pair of functions class_register_device()/class_unregister_device()
> is to make sure that an obd can not be referenced again once its
> refcount reached 0.
>
> Signed-off-by: Vladimir Saveliev Vladimir Saveliev <c17830 at cray.com>
> Signed-off-by: Alexey Lyashkov <c17817 at cray.com>
> Signed-off-by: Alexander Boyko <c17825 at cray.com>
> Signed-off-by: Yang Sheng <ys at whamcloud.com>
> WC-bug-id: https://jira.whamcloud.com/browse/LU-4134
> Seagate-bug-id: MRP-2139 MRP-3267
> Reviewed-on: https://review.whamcloud.com/8045
> Reviewed-on: https://review.whamcloud.com/29967
> Reviewed-by: James Simmons <uja.ornl at yahoo.com>
> Reviewed-by: Alexey Lyashkov <c17817 at cray.com>
> Reviewed-by: Oleg Drokin <green at whamcloud.com>
> Signed-off-by: James Simmons <jsimmons at infradead.org>
> ---
>  drivers/staging/lustre/lustre/include/obd_class.h  |   9 +-
>  drivers/staging/lustre/lustre/obdclass/genops.c    | 284 +++++++++++++++------
>  .../staging/lustre/lustre/obdclass/obd_config.c    | 143 ++++-------
>  drivers/staging/lustre/lustre/obdclass/obd_mount.c |   6 +-
>  4 files changed, 261 insertions(+), 181 deletions(-)
>
> diff --git a/drivers/staging/lustre/lustre/include/obd_class.h b/drivers/staging/lustre/lustre/include/obd_class.h
> index 567189c..cc00915 100644
> --- a/drivers/staging/lustre/lustre/include/obd_class.h
> +++ b/drivers/staging/lustre/lustre/include/obd_class.h
> @@ -65,8 +65,11 @@ int class_register_type(struct obd_ops *dt_ops, struct md_ops *md_ops,
>  			const char *name, struct lu_device_type *ldt);
>  int class_unregister_type(const char *name);
>  
> -struct obd_device *class_newdev(const char *type_name, const char *name);
> -void class_release_dev(struct obd_device *obd);
> +struct obd_device *class_newdev(const char *type_name, const char *name,
> +				const char *uuid);
> +int class_register_device(struct obd_device *obd);
> +void class_unregister_device(struct obd_device *obd);
> +void class_free_dev(struct obd_device *obd);
>  
>  int class_name2dev(const char *name);
>  struct obd_device *class_name2obd(const char *name);
> @@ -230,6 +233,8 @@ void __class_export_del_lock_ref(struct obd_export *exp,
>  void class_export_put(struct obd_export *exp);
>  struct obd_export *class_new_export(struct obd_device *obddev,
>  				    struct obd_uuid *cluuid);
> +struct obd_export *class_new_export_self(struct obd_device *obd,
> +					 struct obd_uuid *uuid);
>  void class_unlink_export(struct obd_export *exp);
>  
>  struct obd_import *class_import_get(struct obd_import *imp);
> diff --git a/drivers/staging/lustre/lustre/obdclass/genops.c b/drivers/staging/lustre/lustre/obdclass/genops.c
> index 59891a8..cdd44f7 100644
> --- a/drivers/staging/lustre/lustre/obdclass/genops.c
> +++ b/drivers/staging/lustre/lustre/obdclass/genops.c
> @@ -38,6 +38,7 @@
>  
>  #define DEBUG_SUBSYSTEM S_CLASS
>  #include <obd_class.h>
> +#include <lustre_log.h>
>  #include <lprocfs_status.h>
>  #include <lustre_kernelcomm.h>
>  
> @@ -273,21 +274,20 @@ int class_unregister_type(const char *name)
>  /**
>   * Create a new obd device.
>   *
> - * Find an empty slot in ::obd_devs[], create a new obd device in it.
> + * Allocate the new obd_device and initialize it.
>   *
>   * \param[in] type_name obd device type string.
>   * \param[in] name      obd device name.
> + * @uuid		obd device UUID.
>   *
> - * \retval NULL if create fails, otherwise return the obd device
> - *	 pointer created.
> + * RETURN newdev	 pointer to created obd_device
> + * RETURN ERR_PTR(errno) on error
>   */
> -struct obd_device *class_newdev(const char *type_name, const char *name)
> +struct obd_device *class_newdev(const char *type_name, const char *name,
> +				const char *uuid)
>  {
> -	struct obd_device *result = NULL;
>  	struct obd_device *newdev;
>  	struct obd_type *type = NULL;
> -	int i;
> -	int new_obd_minor = 0;
>  
>  	if (strlen(name) >= MAX_OBD_NAME) {
>  		CERROR("name/uuid must be < %u bytes long\n", MAX_OBD_NAME);
> @@ -302,87 +302,167 @@ struct obd_device *class_newdev(const char *type_name, const char *name)
>  
>  	newdev = obd_device_alloc();
>  	if (!newdev) {
> -		result = ERR_PTR(-ENOMEM);
> -		goto out_type;
> +		class_put_type(type);
> +		return ERR_PTR(-ENOMEM);
>  	}
>  
>  	LASSERT(newdev->obd_magic == OBD_DEVICE_MAGIC);
> +	strncpy(newdev->obd_name, name, sizeof(newdev->obd_name) - 1);
> +	newdev->obd_type = type;
> +	newdev->obd_minor = -1;
> +
> +	rwlock_init(&newdev->obd_pool_lock);
> +	newdev->obd_pool_limit = 0;
> +	newdev->obd_pool_slv = 0;
> +
> +	INIT_LIST_HEAD(&newdev->obd_exports);
> +	INIT_LIST_HEAD(&newdev->obd_unlinked_exports);
> +	INIT_LIST_HEAD(&newdev->obd_delayed_exports);
> +	spin_lock_init(&newdev->obd_nid_lock);
> +	spin_lock_init(&newdev->obd_dev_lock);
> +	mutex_init(&newdev->obd_dev_mutex);
> +	spin_lock_init(&newdev->obd_osfs_lock);
> +	/* newdev->obd_osfs_age must be set to a value in the distant
> +	 * past to guarantee a fresh statfs is fetched on mount.
> +	 */
> +	newdev->obd_osfs_age = get_jiffies_64() - 1000 * HZ;
>  
> -	write_lock(&obd_dev_lock);
> -	for (i = 0; i < class_devno_max(); i++) {
> -		struct obd_device *obd = class_num2obd(i);
> +	/* XXX belongs in setup not attach  */
> +	init_rwsem(&newdev->obd_observer_link_sem);
> +	/* recovery data */
> +	init_waitqueue_head(&newdev->obd_evict_inprogress_waitq);
>  
> -		if (obd && (strcmp(name, obd->obd_name) == 0)) {
> -			CERROR("Device %s already exists at %d, won't add\n",
> -			       name, i);
> -			if (result) {
> -				LASSERTF(result->obd_magic == OBD_DEVICE_MAGIC,
> -					 "%p obd_magic %08x != %08x\n", result,
> -					 result->obd_magic, OBD_DEVICE_MAGIC);
> -				LASSERTF(result->obd_minor == new_obd_minor,
> -					 "%p obd_minor %d != %d\n", result,
> -					 result->obd_minor, new_obd_minor);
> -
> -				obd_devs[result->obd_minor] = NULL;
> -				result->obd_name[0] = '\0';
> -			 }
> -			result = ERR_PTR(-EEXIST);
> -			break;
> -		}
> -		if (!result && !obd) {
> -			result = newdev;
> -			result->obd_minor = i;
> -			new_obd_minor = i;
> -			result->obd_type = type;
> -			strncpy(result->obd_name, name,
> -				sizeof(result->obd_name) - 1);
> -			obd_devs[i] = result;
> -		}
> -	}
> -	write_unlock(&obd_dev_lock);
> +	llog_group_init(&newdev->obd_olg);
> +	/* Detach drops this */
> +        atomic_set(&newdev->obd_refcount, 1);
> +        lu_ref_init(&newdev->obd_reference);
> +        lu_ref_add(&newdev->obd_reference, "newdev", newdev);
>  
> -	if (!result && i >= class_devno_max()) {
> -		CERROR("all %u OBD devices used, increase MAX_OBD_DEVICES\n",
> -		       class_devno_max());
> -		result = ERR_PTR(-EOVERFLOW);
> -		goto out;
> -	}
> +	newdev->obd_conn_inprogress = 0;
>  
> -	if (IS_ERR(result))
> -		goto out;
> +	strncpy(newdev->obd_uuid.uuid, uuid, strlen(uuid));
>  
> -	CDEBUG(D_IOCTL, "Adding new device %s (%p)\n",
> -	       result->obd_name, result);
> +	CDEBUG(D_IOCTL, "Allocate new device %s (%p)\n",
> +	       newdev->obd_name, newdev);
>  
> -	return result;
> -out:
> -	obd_device_free(newdev);
> -out_type:
> -	class_put_type(type);
> -	return result;
> +	return newdev;
>  }
>  
> -void class_release_dev(struct obd_device *obd)
> +/**
> + * Free obd device.
> + *
> + * @obd obd_device to be freed
> + *
> + * RETURN none
> + */
> +void class_free_dev(struct obd_device *obd)
>  {
>  	struct obd_type *obd_type = obd->obd_type;
>  
>  	LASSERTF(obd->obd_magic == OBD_DEVICE_MAGIC, "%p obd_magic %08x != %08x\n",
>  		 obd, obd->obd_magic, OBD_DEVICE_MAGIC);
> -	LASSERTF(obd == obd_devs[obd->obd_minor], "obd %p != obd_devs[%d] %p\n",
> +	LASSERTF(obd->obd_minor == -1 || obd == obd_devs[obd->obd_minor],
> +		 "obd %p != obd_devs[%d] %p\n",
>  		 obd, obd->obd_minor, obd_devs[obd->obd_minor]);
> +	LASSERTF(atomic_read(&obd->obd_refcount) == 0,
> +		 "obd_refcount should be 0, not %d\n",
> +		 atomic_read(&obd->obd_refcount));
>  	LASSERT(obd_type);
>  
> -	CDEBUG(D_INFO, "Release obd device %s at %d obd_type name =%s\n",
> -	       obd->obd_name, obd->obd_minor, obd->obd_type->typ_name);
> +	CDEBUG(D_INFO, "Release obd device %s obd_type name =%s\n",
> +	       obd->obd_name, obd->obd_type->typ_name);
> +
> +	CDEBUG(D_CONFIG, "finishing cleanup of obd %s (%s)\n",
> +	       obd->obd_name, obd->obd_uuid.uuid);
> +	if (obd->obd_stopping) {
> +		int err;
> +
> +		/* If we're not stopping, we were never set up */
> +		err = obd_cleanup(obd);
> +		if (err)
> +			CERROR("Cleanup %s returned %d\n",
> +			       obd->obd_name, err);
> +	}
>  
> -	write_lock(&obd_dev_lock);
> -	obd_devs[obd->obd_minor] = NULL;
> -	write_unlock(&obd_dev_lock);
>  	obd_device_free(obd);
>  
>  	class_put_type(obd_type);
>  }
>  
> +/**
> + * Unregister obd device.
> + *
> + * Free slot in obd_dev[] used by \a obd.
> + *
> + * @new_obd	obd_device to be unregistered
> + *
> + * RETURN	none
> + */
> +void class_unregister_device(struct obd_device *obd)
> +{
> +	write_lock(&obd_dev_lock);
> +	if (obd->obd_minor >= 0) {
> +		LASSERT(obd_devs[obd->obd_minor] == obd);
> +		obd_devs[obd->obd_minor] = NULL;
> +		obd->obd_minor = -1;
> +	}
> +	write_unlock(&obd_dev_lock);
> +}
> +
> +/**
> + * Register obd device.
> + *
> + * Find free slot in obd_devs[], fills it with \a new_obd.
> + *
> + * @new_obd	obd_device to be registered
> + *
> + * RETURN	0		success
> + *		-EEXIST		device with this name is registered
> + *		-EOVERFLOW	obd_devs[] is full
> + */
> +int class_register_device(struct obd_device *new_obd)
> +{
> +	int new_obd_minor = 0;
> +	bool minor_assign = false;
> +	int ret = 0;
> +	int i;
> +
> +	write_lock(&obd_dev_lock);
> +	for (i = 0; i < class_devno_max(); i++) {
> +		struct obd_device *obd = class_num2obd(i);
> +
> +		if (obd && (strcmp(new_obd->obd_name, obd->obd_name) == 0)) {
> +			CERROR("%s: already exists, won't add\n",
> +			       obd->obd_name);
> +			/* in case we found a free slot before duplicate */
> +			minor_assign = false;
> +			ret = -EEXIST;
> +			break;
> +		}
> +		if (!minor_assign && !obd) {
> +			new_obd_minor = i;
> +			minor_assign = true;
> +		}
> +	}
> +
> +	if (minor_assign) {
> +		new_obd->obd_minor = new_obd_minor;
> +		LASSERTF(!obd_devs[new_obd_minor], "obd_devs[%d] %p\n",
> +			 new_obd_minor, obd_devs[new_obd_minor]);
> +		obd_devs[new_obd_minor] = new_obd;
> +	} else {
> +		if (ret == 0) {
> +			ret = -EOVERFLOW;
> +			CERROR("%s: all %u/%u devices used, increase MAX_OBD_DEVICES: rc = %d\n",
> +			       new_obd->obd_name, i, class_devno_max(), ret);
> +		}
> +	}
> +	write_unlock(&obd_dev_lock);
> +
> +	return ret;
> +}
> +
> +
>  int class_name2dev(const char *name)
>  {
>  	int i;
> @@ -677,7 +757,11 @@ static void class_export_destroy(struct obd_export *exp)
>  	LASSERT(list_empty(&exp->exp_req_replay_queue));
>  	LASSERT(list_empty(&exp->exp_hp_rpcs));
>  	obd_destroy_export(exp);
> -	class_decref(obd, "export", exp);
> +	/* self export doesn't hold a reference to an obd, although it
> +	 * exists until freeing of the obd
> +	 */
> +	if (exp != obd->obd_self_export)
> +		class_decref(obd, "export", exp);
>  
>  	OBD_FREE_RCU(exp, sizeof(*exp), &exp->exp_handle);
>  }
> @@ -708,11 +792,27 @@ void class_export_put(struct obd_export *exp)
>  	       atomic_read(&exp->exp_refcount) - 1);
>  
>  	if (atomic_dec_and_test(&exp->exp_refcount)) {
> -		LASSERT(!list_empty(&exp->exp_obd_chain));
> +		struct obd_device *obd = exp->exp_obd;
> +
>  		CDEBUG(D_IOCTL, "final put %p/%s\n",
>  		       exp, exp->exp_client_uuid.uuid);
>  
> -		obd_zombie_export_add(exp);
> +		if (exp == obd->obd_self_export) {
> +			/* self export should be destroyed without
> +			 * zombie thread as it doesn't hold a
> +			 * reference to obd and doesn't hold any
> +			 * resources
> +			 */
> +			class_export_destroy(exp);
> +			/* self export is destroyed, no class
> +			 * references exist and it is safe to free
> +			 * obd
> +			 */
> +			class_free_dev(obd);
> +		} else {
> +			LASSERT(!list_empty(&exp->exp_obd_chain));
> +			obd_zombie_export_add(exp);
> +		}
>  	}
>  }
>  EXPORT_SYMBOL(class_export_put);
> @@ -728,8 +828,9 @@ static void obd_zombie_exp_cull(struct work_struct *ws)
>   * pointer to it. The refcount is 2: one for the hash reference, and
>   * one for the pointer returned by this function.
>   */
> -struct obd_export *class_new_export(struct obd_device *obd,
> -				    struct obd_uuid *cluuid)
> +static struct obd_export *__class_new_export(struct obd_device *obd,
> +					     struct obd_uuid *cluuid,
> +					     bool is_self)
>  {
>  	struct obd_export *export;
>  	int rc = 0;
> @@ -739,6 +840,7 @@ struct obd_export *class_new_export(struct obd_device *obd,
>  		return ERR_PTR(-ENOMEM);
>  
>  	export->exp_conn_cnt = 0;
> +	/* 2 = class_handle_hash + last */
>  	atomic_set(&export->exp_refcount, 2);
>  	atomic_set(&export->exp_rpc_count, 0);
>  	atomic_set(&export->exp_cb_count, 0);
> @@ -767,41 +869,65 @@ struct obd_export *class_new_export(struct obd_device *obd,
>  	export->exp_client_uuid = *cluuid;
>  	obd_init_export(export);
>  
> -	spin_lock(&obd->obd_dev_lock);
> -	/* shouldn't happen, but might race */
> -	if (obd->obd_stopping) {
> -		rc = -ENODEV;
> -		goto exit_unlock;
> -	}
> -
>  	if (!obd_uuid_equals(cluuid, &obd->obd_uuid)) {
> +		spin_lock(&obd->obd_dev_lock);
> +		/* shouldn't happen, but might race */
> +		if (obd->obd_stopping) {
> +			rc = -ENODEV;
> +			goto exit_unlock;
> +		}
> +		spin_unlock(&obd->obd_dev_lock);

This is wrong.  The lock previously protected obd_uuid_add(), now it
doesn't.
This probably happened because the locking was simplified when I
converted to rhashtables so the OpenSFS patch did quite different things
with locking.

I've applied the following incremental patch to fix up the locking.

Thanks,
NeilBrown

diff --git a/drivers/staging/lustre/lustre/obdclass/genops.c b/drivers/staging/lustre/lustre/obdclass/genops.c
index cdd44f72403f..76bc73fd79c8 100644
--- a/drivers/staging/lustre/lustre/obdclass/genops.c
+++ b/drivers/staging/lustre/lustre/obdclass/genops.c
@@ -869,24 +869,22 @@ static struct obd_export *__class_new_export(struct obd_device *obd,
 	export->exp_client_uuid = *cluuid;
 	obd_init_export(export);
 
+	spin_lock(&obd->obd_dev_lock);
 	if (!obd_uuid_equals(cluuid, &obd->obd_uuid)) {
-		spin_lock(&obd->obd_dev_lock);
 		/* shouldn't happen, but might race */
 		if (obd->obd_stopping) {
 			rc = -ENODEV;
 			goto exit_unlock;
 		}
-		spin_unlock(&obd->obd_dev_lock);
 
 		rc = obd_uuid_add(obd, export);
 		if (rc) {
 			LCONSOLE_WARN("%s: denying duplicate export for %s, %d\n",
 				      obd->obd_name, cluuid->uuid, rc);
-			goto exit_err;
+			goto exit_unlock;
 		}
 	}
 
-	spin_lock(&obd->obd_dev_lock);
 	if (!is_self) {
 		class_incref(obd, "export", export);
 		list_add(&export->exp_obd_chain, &export->exp_obd->obd_exports);
@@ -899,7 +897,6 @@ static struct obd_export *__class_new_export(struct obd_device *obd,
 
 exit_unlock:
 	spin_unlock(&obd->obd_dev_lock);
-exit_err:
 	class_handle_unhash(&export->exp_handle);
 	obd_destroy_export(export);
 	kfree(export);
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 832 bytes
Desc: not available
URL: <http://lists.lustre.org/pipermail/lustre-devel_lists.lustre.org/attachments/20181127/d8edc8f4/attachment.sig>

From neilb at suse.com  Tue Nov 27 04:20:19 2018
From: neilb at suse.com (NeilBrown)
Date: Tue, 27 Nov 2018 15:20:19 +1100
Subject: [lustre-devel] [PATCH 10/12] lustre: clio: Introduce parallel
 tasks framework
In-Reply-To: <1543200508-6838-11-git-send-email-jsimmons@infradead.org>
References: <1543200508-6838-1-git-send-email-jsimmons@infradead.org>
 <1543200508-6838-11-git-send-email-jsimmons@infradead.org>
Message-ID: <87woozp2ws.fsf@notabene.neil.brown.name>

On Sun, Nov 25 2018, James Simmons wrote:

> From: Dmitry Eremin <dmitry.eremin at intel.com>
>
> In this patch new API for parallel tasks execution is introduced.
> This API based on Linux kernel padata API which is used to perform
> encryption and decryption on large numbers of packets without
> reordering those packets.
>
> It was adopted for general use in Lustre for parallelization of
> various functionality. The first place of its usage is parallel I/O
> implementation.
>
> The first step in using it is to set up a cl_ptask structure to
> control of how this task are to be run:
>
>     #include <cl_ptask.h>
>
>     int cl_ptask_init(struct cl_ptask *ptask, cl_ptask_cb_t cbfunc,
>                       void *cbdata, unsigned int flags, int cpu);
>
> The cbfunc function with cbdata argument will be called in the process
> of getting the task done. The cpu specifies which CPU will be used for
> the final callback when the task is done.
>
> The submission of task is done with:
>
>     int cl_ptask_submit(struct cl_ptask *ptask,
>                         struct cl_ptask_engine *engine);
>
> The task is submitted to the engine for execution.
>
> In order to wait for result of task execution you should call:
>
>    int cl_ptask_wait_for(struct cl_ptask *ptask);
>
> The tasks with flag PTF_ORDERED are executed in parallel but complete
> into submission order. So, waiting for last ordered task you can be sure
> that all previous tasks were done before this task complete.
>
> This patch differs from the OpenSFS tree by adding this functional
> to the clio layer instead of libcfs.

While you are right that it shouldn't be in libcfs, it actually
shouldn't exist at all.
cfs_ptask_init() is used precisely once in OpenSFS.  There is no point
creating a generic API wrapper like this that is only used once.

cl_oi needs to use padata API calls directly.

Thanks,
NeilBrown
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 832 bytes
Desc: not available
URL: <http://lists.lustre.org/pipermail/lustre-devel_lists.lustre.org/attachments/20181127/94e4c201/attachment.sig>

From adilger at whamcloud.com  Tue Nov 27 05:08:45 2018
From: adilger at whamcloud.com (Andreas Dilger)
Date: Tue, 27 Nov 2018 05:08:45 +0000
Subject: [lustre-devel] [PATCH 10/12] lustre: clio: Introduce parallel
 tasks framework
In-Reply-To: <87woozp2ws.fsf@notabene.neil.brown.name>
References: <1543200508-6838-1-git-send-email-jsimmons@infradead.org>
 <1543200508-6838-11-git-send-email-jsimmons@infradead.org>
 <87woozp2ws.fsf@notabene.neil.brown.name>
Message-ID: <7ED31E7C-2657-4FB8-B083-E451504EE62F@whamcloud.com>

On Nov 26, 2018, at 21:20, NeilBrown <neilb at suse.com> wrote:
> 
> On Sun, Nov 25 2018, James Simmons wrote:
> 
>> From: Dmitry Eremin <dmitry.eremin at intel.com>
>> 
>> In this patch new API for parallel tasks execution is introduced.
>> This API based on Linux kernel padata API which is used to perform
>> encryption and decryption on large numbers of packets without
>> reordering those packets.
>> 
>> It was adopted for general use in Lustre for parallelization of
>> various functionality. The first place of its usage is parallel I/O
>> implementation.
>> 
>> The first step in using it is to set up a cl_ptask structure to
>> control of how this task are to be run:
>> 
>>    #include <cl_ptask.h>
>> 
>>    int cl_ptask_init(struct cl_ptask *ptask, cl_ptask_cb_t cbfunc,
>>                      void *cbdata, unsigned int flags, int cpu);
>> 
>> The cbfunc function with cbdata argument will be called in the process
>> of getting the task done. The cpu specifies which CPU will be used for
>> the final callback when the task is done.
>> 
>> The submission of task is done with:
>> 
>>    int cl_ptask_submit(struct cl_ptask *ptask,
>>                        struct cl_ptask_engine *engine);
>> 
>> The task is submitted to the engine for execution.
>> 
>> In order to wait for result of task execution you should call:
>> 
>>   int cl_ptask_wait_for(struct cl_ptask *ptask);
>> 
>> The tasks with flag PTF_ORDERED are executed in parallel but complete
>> into submission order. So, waiting for last ordered task you can be sure
>> that all previous tasks were done before this task complete.
>> 
>> This patch differs from the OpenSFS tree by adding this functional
>> to the clio layer instead of libcfs.
> 
> While you are right that it shouldn't be in libcfs, it actually
> shouldn't exist at all.
> cfs_ptask_init() is used precisely once in OpenSFS.  There is no point
> creating a generic API wrapper like this that is only used once.
> 
> cl_oi needs to use padata API calls directly.

This infrastructure was also going to be used for parallel readahead, but the patch that implemented that was never landed because the expected performance gains didn't materialize.

Cheers, Andreas
---
Andreas Dilger
Principal Lustre Architect
Whamcloud








From paf at cray.com  Tue Nov 27 13:51:02 2018
From: paf at cray.com (Patrick Farrell)
Date: Tue, 27 Nov 2018 13:51:02 +0000
Subject: [lustre-devel] [PATCH 10/12] lustre: clio: Introduce parallel
 tasks framework
In-Reply-To: <7ED31E7C-2657-4FB8-B083-E451504EE62F@whamcloud.com>
References: <1543200508-6838-1-git-send-email-jsimmons@infradead.org>
 <1543200508-6838-11-git-send-email-jsimmons@infradead.org>
 <87woozp2ws.fsf@notabene.neil.brown.name>,
 <7ED31E7C-2657-4FB8-B083-E451504EE62F@whamcloud.com>
Message-ID: <MWHPR11MB1488A82DFA620A251C2EEC70CBD00@MWHPR11MB1488.namprd11.prod.outlook.com>

Two notes coming, first about padata.

A major reason is actually the infrastructure itself - it’s inappropriate to our kinds of tasks.  I did a quick talk on it a while back, intending then to fix it, but never got the chance (and since had better ideas to improve write performance):

https://www.eofs.eu/_media/events/devsummit17/patrick_farrell_laddevsummit_pio.pdf

padata basically bakes in a set of assumptions that amount to “functionally infinite amount of small work units and a dedicated machine”, which fit well with its role in packet encryption but don’t sit well for other kinds of paralelliziation.  (For example, all work is strictly and explicitly bound to a CPU.  No scheduler.  One more as a bonus - it distributes work across all allowed CPUs, but that means if you have a small number of work items (which splitting I/O tends to be because you have to make relatively big chunks) that effectively every work unit starts a worker thread for itself.)

The recent discussion of a new parallel inaction framework on LWN looked intriguing for future work.  it’s expected to fix a number of the limitations.
https://lwn.net/Articles/771169/

________________________________
From: lustre-devel <lustre-devel-bounces at lists.lustre.org> on behalf of Andreas Dilger <adilger at whamcloud.com>
Sent: Monday, November 26, 2018 11:08:45 PM
To: NeilBrown
Cc: Lustre Development List
Subject: Re: [lustre-devel] [PATCH 10/12] lustre: clio: Introduce parallel tasks framework

On Nov 26, 2018, at 21:20, NeilBrown <neilb at suse.com> wrote:
>
> On Sun, Nov 25 2018, James Simmons wrote:
>
>> From: Dmitry Eremin <dmitry.eremin at intel.com>
>>
>> In this patch new API for parallel tasks execution is introduced.
>> This API based on Linux kernel padata API which is used to perform
>> encryption and decryption on large numbers of packets without
>> reordering those packets.
>>
>> It was adopted for general use in Lustre for parallelization of
>> various functionality. The first place of its usage is parallel I/O
>> implementation.
>>
>> The first step in using it is to set up a cl_ptask structure to
>> control of how this task are to be run:
>>
>>    #include <cl_ptask.h>
>>
>>    int cl_ptask_init(struct cl_ptask *ptask, cl_ptask_cb_t cbfunc,
>>                      void *cbdata, unsigned int flags, int cpu);
>>
>> The cbfunc function with cbdata argument will be called in the process
>> of getting the task done. The cpu specifies which CPU will be used for
>> the final callback when the task is done.
>>
>> The submission of task is done with:
>>
>>    int cl_ptask_submit(struct cl_ptask *ptask,
>>                        struct cl_ptask_engine *engine);
>>
>> The task is submitted to the engine for execution.
>>
>> In order to wait for result of task execution you should call:
>>
>>   int cl_ptask_wait_for(struct cl_ptask *ptask);
>>
>> The tasks with flag PTF_ORDERED are executed in parallel but complete
>> into submission order. So, waiting for last ordered task you can be sure
>> that all previous tasks were done before this task complete.
>>
>> This patch differs from the OpenSFS tree by adding this functional
>> to the clio layer instead of libcfs.
>
> While you are right that it shouldn't be in libcfs, it actually
> shouldn't exist at all.
> cfs_ptask_init() is used precisely once in OpenSFS.  There is no point
> creating a generic API wrapper like this that is only used once.
>
> cl_oi needs to use padata API calls directly.

This infrastructure was also going to be used for parallel readahead, but the patch that implemented that was never landed because the expected performance gains didn't materialize.

Cheers, Andreas
---
Andreas Dilger
Principal Lustre Architect
Whamcloud







_______________________________________________
lustre-devel mailing list
lustre-devel at lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-devel-lustre.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-devel_lists.lustre.org/attachments/20181127/06cc8d7e/attachment.html>

From paf at cray.com  Tue Nov 27 14:01:53 2018
From: paf at cray.com (Patrick Farrell)
Date: Tue, 27 Nov 2018 14:01:53 +0000
Subject: [lustre-devel] [PATCH 10/12] lustre: clio: Introduce parallel
 tasks framework
In-Reply-To: <MWHPR11MB1488A82DFA620A251C2EEC70CBD00@MWHPR11MB1488.namprd11.prod.outlook.com>
References: <1543200508-6838-1-git-send-email-jsimmons@infradead.org>
 <1543200508-6838-11-git-send-email-jsimmons@infradead.org>
 <87woozp2ws.fsf@notabene.neil.brown.name>,
 <7ED31E7C-2657-4FB8-B083-E451504EE62F@whamcloud.com>,
 <MWHPR11MB1488A82DFA620A251C2EEC70CBD00@MWHPR11MB1488.namprd11.prod.outlook.com>
Message-ID: <MWHPR11MB148834CE6B90A81CE7D63D91CBD00@MWHPR11MB1488.namprd11.prod.outlook.com>


Second, about pio.

I believe that long term it’s headed out of Lustre.  It only improves performance in a limited way in certain circumstances, and harms it in various others.  So it’s off by default, and, I suspect, remains completely unused.  A while back I noticed its test framework test didn’t activate it correctly, and once fixed, it sometimes deadlocks (race with truncate). There’s a patch to fix that, but a problem was found in it and it has since languished.

I would still suggest you take it, Neil, as othewise you’ll complicate a bunch of potentially nasty porting working in the CLIO stack, as you apply the years of patches written with it there.  Instead, I’d suggest we pull it in the open sfs branch (Sorry!  It was a promising idea but it hasn’t panned out, and the current parallel readahead work isn’t going to use it.) and then eventually you could pick that up.

Curious how folks feel about this.  I’d be willing to take a stab at writing a removal patch for 2.13.  It pains me a bit to suggest giving up on it, but Jinshan and I want to do write container type work to improve writes, and there’s the older/new again DDN parallel readahead work for reads.

________________________________
From: lustre-devel <lustre-devel-bounces at lists.lustre.org> on behalf of Patrick Farrell <paf at cray.com>
Sent: Tuesday, November 27, 2018 7:51:02 AM
To: Andreas Dilger; NeilBrown
Cc: Lustre Development List
Subject: Re: [lustre-devel] [PATCH 10/12] lustre: clio: Introduce parallel tasks framework

Two notes coming, first about padata.

A major reason is actually the infrastructure itself - it’s inappropriate to our kinds of tasks.  I did a quick talk on it a while back, intending then to fix it, but never got the chance (and since had better ideas to improve write performance):

https://www.eofs.eu/_media/events/devsummit17/patrick_farrell_laddevsummit_pio.pdf

padata basically bakes in a set of assumptions that amount to “functionally infinite amount of small work units and a dedicated machine”, which fit well with its role in packet encryption but don’t sit well for other kinds of paralelliziation.  (For example, all work is strictly and explicitly bound to a CPU.  No scheduler.  One more as a bonus - it distributes work across all allowed CPUs, but that means if you have a small number of work items (which splitting I/O tends to be because you have to make relatively big chunks) that effectively every work unit starts a worker thread for itself.)

The recent discussion of a new parallel inaction framework on LWN looked intriguing for future work.  it’s expected to fix a number of the limitations.
https://lwn.net/Articles/771169/

________________________________
From: lustre-devel <lustre-devel-bounces at lists.lustre.org> on behalf of Andreas Dilger <adilger at whamcloud.com>
Sent: Monday, November 26, 2018 11:08:45 PM
To: NeilBrown
Cc: Lustre Development List
Subject: Re: [lustre-devel] [PATCH 10/12] lustre: clio: Introduce parallel tasks framework

On Nov 26, 2018, at 21:20, NeilBrown <neilb at suse.com> wrote:
>
> On Sun, Nov 25 2018, James Simmons wrote:
>
>> From: Dmitry Eremin <dmitry.eremin at intel.com>
>>
>> In this patch new API for parallel tasks execution is introduced.
>> This API based on Linux kernel padata API which is used to perform
>> encryption and decryption on large numbers of packets without
>> reordering those packets.
>>
>> It was adopted for general use in Lustre for parallelization of
>> various functionality. The first place of its usage is parallel I/O
>> implementation.
>>
>> The first step in using it is to set up a cl_ptask structure to
>> control of how this task are to be run:
>>
>>    #include <cl_ptask.h>
>>
>>    int cl_ptask_init(struct cl_ptask *ptask, cl_ptask_cb_t cbfunc,
>>                      void *cbdata, unsigned int flags, int cpu);
>>
>> The cbfunc function with cbdata argument will be called in the process
>> of getting the task done. The cpu specifies which CPU will be used for
>> the final callback when the task is done.
>>
>> The submission of task is done with:
>>
>>    int cl_ptask_submit(struct cl_ptask *ptask,
>>                        struct cl_ptask_engine *engine);
>>
>> The task is submitted to the engine for execution.
>>
>> In order to wait for result of task execution you should call:
>>
>>   int cl_ptask_wait_for(struct cl_ptask *ptask);
>>
>> The tasks with flag PTF_ORDERED are executed in parallel but complete
>> into submission order. So, waiting for last ordered task you can be sure
>> that all previous tasks were done before this task complete.
>>
>> This patch differs from the OpenSFS tree by adding this functional
>> to the clio layer instead of libcfs.
>
> While you are right that it shouldn't be in libcfs, it actually
> shouldn't exist at all.
> cfs_ptask_init() is used precisely once in OpenSFS.  There is no point
> creating a generic API wrapper like this that is only used once.
>
> cl_oi needs to use padata API calls directly.

This infrastructure was also going to be used for parallel readahead, but the patch that implemented that was never landed because the expected performance gains didn't materialize.

Cheers, Andreas
---
Andreas Dilger
Principal Lustre Architect
Whamcloud







_______________________________________________
lustre-devel mailing list
lustre-devel at lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-devel-lustre.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-devel_lists.lustre.org/attachments/20181127/c7df3662/attachment.html>

From charlie at whamcloud.com  Tue Nov 27 15:31:27 2018
From: charlie at whamcloud.com (Charlie Olmstead)
Date: Tue, 27 Nov 2018 15:31:27 +0000
Subject: [lustre-devel] Autotest: review-ldiskfs-arm session
Message-ID: <AF1EA0FC-3824-4D4C-B12B-910D7284B09A@contoso.com>

With the landing of https://review.whamcloud.com/#/c/33652/ the review-ldiskfs-arm session is now passing. Starting tomorrow (11/27) it will change over to a required session.

Charlie Olmstead
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-devel_lists.lustre.org/attachments/20181127/c01a8cc4/attachment.html>

From neilb at suse.com  Tue Nov 27 22:27:06 2018
From: neilb at suse.com (NeilBrown)
Date: Wed, 28 Nov 2018 09:27:06 +1100
Subject: [lustre-devel] [PATCH 10/12] lustre: clio: Introduce parallel
 tasks framework
In-Reply-To: <MWHPR11MB148834CE6B90A81CE7D63D91CBD00@MWHPR11MB1488.namprd11.prod.outlook.com>
References: <1543200508-6838-1-git-send-email-jsimmons@infradead.org>
 <1543200508-6838-11-git-send-email-jsimmons@infradead.org>
 <87woozp2ws.fsf@notabene.neil.brown.name>
 <7ED31E7C-2657-4FB8-B083-E451504EE62F@whamcloud.com>
 <MWHPR11MB1488A82DFA620A251C2EEC70CBD00@MWHPR11MB1488.namprd11.prod.outlook.com>
 <MWHPR11MB148834CE6B90A81CE7D63D91CBD00@MWHPR11MB1488.namprd11.prod.outlook.com>
Message-ID: <87sgzmp35x.fsf@notabene.neil.brown.name>

On Tue, Nov 27 2018, Patrick Farrell wrote:

> Second, about pio.
>
> I believe that long term it’s headed out of Lustre.  It only improves performance in a limited way in certain circumstances, and harms it in various others.  So it’s off by default, and, I suspect, remains completely unused.  A while back I noticed its test framework test didn’t activate it correctly, and once fixed, it sometimes deadlocks (race with truncate). There’s a patch to fix that, but a problem was found in it and it has since languished.
>
> I would still suggest you take it, Neil, as othewise you’ll complicate a bunch of potentially nasty porting working in the CLIO stack, as you apply the years of patches written with it there.  Instead, I’d suggest we pull it in the open sfs branch (Sorry!  It was a promising idea but it hasn’t panned out, and the current parallel readahead work isn’t going to use it.) and then eventually you could pick that up.

Thanks so much for this background and context - really helpful.

I looked though your slides and got the impression that a simple
work-queue would probably be the best approach - no need to create your
own pool of kthreads as I think you said you had trialed.

As for the suggestion that I take it anyway, and then remove it later
after it gets removed from OpenSFS, I remain unconvinced.
You mention "years of patches written with it there"  but the first
usage of the cfs_ptask_init only landed in March 2017 (less than 2 years
ago).  libcfs_ptask is only use in lustre/obdclass/ lustre/llite/
lustre/lov/ and the total patches in these directories since it was
introduced in 319.  I suspect most of them aren't related to ptask.

So I see no evidence that there will be much "nasty porting work".  I
suspect there will be some, but porting code is what I spend a lot of my
time doing, and doing it helps force me to understand the code.

So what this isn't a "no way, never", it is "I'm not convinced".

Thanks,
NeilBrown


>
> Curious how folks feel about this.  I’d be willing to take a stab at writing a removal patch for 2.13.  It pains me a bit to suggest giving up on it, but Jinshan and I want to do write container type work to improve writes, and there’s the older/new again DDN parallel readahead work for reads.
>
> ________________________________
> From: lustre-devel <lustre-devel-bounces at lists.lustre.org> on behalf of Patrick Farrell <paf at cray.com>
> Sent: Tuesday, November 27, 2018 7:51:02 AM
> To: Andreas Dilger; NeilBrown
> Cc: Lustre Development List
> Subject: Re: [lustre-devel] [PATCH 10/12] lustre: clio: Introduce parallel tasks framework
>
> Two notes coming, first about padata.
>
> A major reason is actually the infrastructure itself - it’s inappropriate to our kinds of tasks.  I did a quick talk on it a while back, intending then to fix it, but never got the chance (and since had better ideas to improve write performance):
>
> https://www.eofs.eu/_media/events/devsummit17/patrick_farrell_laddevsummit_pio.pdf
>
> padata basically bakes in a set of assumptions that amount to “functionally infinite amount of small work units and a dedicated machine”, which fit well with its role in packet encryption but don’t sit well for other kinds of paralelliziation.  (For example, all work is strictly and explicitly bound to a CPU.  No scheduler.  One more as a bonus - it distributes work across all allowed CPUs, but that means if you have a small number of work items (which splitting I/O tends to be because you have to make relatively big chunks) that effectively every work unit starts a worker thread for itself.)
>
> The recent discussion of a new parallel inaction framework on LWN looked intriguing for future work.  it’s expected to fix a number of the limitations.
> https://lwn.net/Articles/771169/
>
> ________________________________
> From: lustre-devel <lustre-devel-bounces at lists.lustre.org> on behalf of Andreas Dilger <adilger at whamcloud.com>
> Sent: Monday, November 26, 2018 11:08:45 PM
> To: NeilBrown
> Cc: Lustre Development List
> Subject: Re: [lustre-devel] [PATCH 10/12] lustre: clio: Introduce parallel tasks framework
>
> On Nov 26, 2018, at 21:20, NeilBrown <neilb at suse.com> wrote:
>>
>> On Sun, Nov 25 2018, James Simmons wrote:
>>
>>> From: Dmitry Eremin <dmitry.eremin at intel.com>
>>>
>>> In this patch new API for parallel tasks execution is introduced.
>>> This API based on Linux kernel padata API which is used to perform
>>> encryption and decryption on large numbers of packets without
>>> reordering those packets.
>>>
>>> It was adopted for general use in Lustre for parallelization of
>>> various functionality. The first place of its usage is parallel I/O
>>> implementation.
>>>
>>> The first step in using it is to set up a cl_ptask structure to
>>> control of how this task are to be run:
>>>
>>>    #include <cl_ptask.h>
>>>
>>>    int cl_ptask_init(struct cl_ptask *ptask, cl_ptask_cb_t cbfunc,
>>>                      void *cbdata, unsigned int flags, int cpu);
>>>
>>> The cbfunc function with cbdata argument will be called in the process
>>> of getting the task done. The cpu specifies which CPU will be used for
>>> the final callback when the task is done.
>>>
>>> The submission of task is done with:
>>>
>>>    int cl_ptask_submit(struct cl_ptask *ptask,
>>>                        struct cl_ptask_engine *engine);
>>>
>>> The task is submitted to the engine for execution.
>>>
>>> In order to wait for result of task execution you should call:
>>>
>>>   int cl_ptask_wait_for(struct cl_ptask *ptask);
>>>
>>> The tasks with flag PTF_ORDERED are executed in parallel but complete
>>> into submission order. So, waiting for last ordered task you can be sure
>>> that all previous tasks were done before this task complete.
>>>
>>> This patch differs from the OpenSFS tree by adding this functional
>>> to the clio layer instead of libcfs.
>>
>> While you are right that it shouldn't be in libcfs, it actually
>> shouldn't exist at all.
>> cfs_ptask_init() is used precisely once in OpenSFS.  There is no point
>> creating a generic API wrapper like this that is only used once.
>>
>> cl_oi needs to use padata API calls directly.
>
> This infrastructure was also going to be used for parallel readahead, but the patch that implemented that was never landed because the expected performance gains didn't materialize.
>
> Cheers, Andreas
> ---
> Andreas Dilger
> Principal Lustre Architect
> Whamcloud
>
>
>
>
>
>
>
> _______________________________________________
> lustre-devel mailing list
> lustre-devel at lists.lustre.org
> http://lists.lustre.org/listinfo.cgi/lustre-devel-lustre.org
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 832 bytes
Desc: not available
URL: <http://lists.lustre.org/pipermail/lustre-devel_lists.lustre.org/attachments/20181128/3a9f4f4b/attachment.sig>

From paf at cray.com  Tue Nov 27 22:50:12 2018
From: paf at cray.com (Patrick Farrell)
Date: Tue, 27 Nov 2018 22:50:12 +0000
Subject: [lustre-devel] [PATCH 10/12] lustre: clio: Introduce parallel
 tasks framework
In-Reply-To: <87sgzmp35x.fsf@notabene.neil.brown.name>
References: <1543200508-6838-1-git-send-email-jsimmons@infradead.org>
 <1543200508-6838-11-git-send-email-jsimmons@infradead.org>
 <87woozp2ws.fsf@notabene.neil.brown.name>
 <7ED31E7C-2657-4FB8-B083-E451504EE62F@whamcloud.com>
 <MWHPR11MB1488A82DFA620A251C2EEC70CBD00@MWHPR11MB1488.namprd11.prod.outlook.com>
 <MWHPR11MB148834CE6B90A81CE7D63D91CBD00@MWHPR11MB1488.namprd11.prod.outlook.com>
 <87sgzmp35x.fsf@notabene.neil.brown.name>
Message-ID: <5CFCC897-3F70-4925-B445-965189ED943D@cray.com>

Starting from the top:
Yes, a simple work queue would probably work OK.  I had good luck with a simple kthread_run, actually (in a later pass at it).

But if you're thinking of improving it, there are a number of issues with it today, which are non-trivial to resolve.  Not sure which I mentioned in my presentation, but here's a quick attempt:
1. It only works on > 1 stripe files, which isn't ideal
2. It has no limit on the number of threads it will use to do I/O to one file.  In reality, 2-4 threads or so is the maximum which gets a benefit.  More than that actually hurts.
3. I believe it hurts read performance (or maybe it's off for reads even when on for writes?  Can't remember)
4. It has a deadlock with truncate which is not easy to fix.  My attempt to fix it creates a *different* lock inversion and since pio isn't (AFAIK) being used, I gave up.  No one has complained about the deadlock and it happens fairly easily with 'dd', so...

RE: Porting difficulties.  Sorry - Not the *ptask* part.  The changes in the CLIO stack to allow the actual parallel I/O use, which are in another patch.  I tend to run all the LU-8964 patches together in my mind.

This is the change I was suggesting you might not want to skip:

"commit db59ecb5d1d0284fb918def6348a11e0966d7767
Author: Dmitry Eremin <dmitry.eremin at intel.com>
Date:   Thu Mar 30 22:38:56 2017 +0300

    LU-8964 clio: Parallelize generic I/O

    Add parallel version of cl_io_loop() function which use information
    about stripes from LOV layer and process them in parallel.
    This feature is disabled by default. To enable it you should run
    "lctl set_param llite.*.pio=1" command."

" lustre/include/cl_object.h     |  49 ++++++---
 lustre/include/lustre_compat.h | 119 +++++++++++++++++++++
 lustre/include/obd_support.h   |   1 +
 lustre/llite/file.c            | 201 ++++++++++++++++++++++++++++-------
 lustre/llite/llite_internal.h  | 123 +--------------------
 lustre/llite/lproc_llite.c     |  39 ++++++-
 lustre/llite/rw26.c            |   4 +-
 lustre/llite/vvp_internal.h    |   9 +-
 lustre/llite/vvp_io.c          | 235 +++++++++++++++++++++--------------------
 lustre/lov/lov_io.c            |  91 ++++++++++------
 lustre/obdclass/cl_io.c        | 233 +++++++++++++++++++++++++++++++---------
 lustre/obdclass/cl_object.c    |  13 +++
 lustre/osc/osc_io.c            |   4 +-
 lustre/osc/osc_lock.c          |   6 +-
 lustre/tests/sanity.sh         |  11 ++"

It is, of course, up to you, and you are *really* good at porting code.  But as I assume you see, this one is significantly scarier.  The ptask patch itself is no big deal.

- Patrick


On 11/27/18, 4:27 PM, "NeilBrown" <neilb at suse.com> wrote:

    On Tue, Nov 27 2018, Patrick Farrell wrote:
    
    > Second, about pio.
    >
    > I believe that long term it’s headed out of Lustre.  It only improves performance in a limited way in certain circumstances, and harms it in various others.  So it’s off by default, and, I suspect, remains completely unused.  A while back I noticed its test framework test didn’t activate it correctly, and once fixed, it sometimes deadlocks (race with truncate). There’s a patch to fix that, but a problem was found in it and it has since languished.
    >
    > I would still suggest you take it, Neil, as othewise you’ll complicate a bunch of potentially nasty porting working in the CLIO stack, as you apply the years of patches written with it there.  Instead, I’d suggest we pull it in the open sfs branch (Sorry!  It was a promising idea but it hasn’t panned out, and the current parallel readahead work isn’t going to use it.) and then eventually you could pick that up.
    
    Thanks so much for this background and context - really helpful.
    
    I looked though your slides and got the impression that a simple
    work-queue would probably be the best approach - no need to create your
    own pool of kthreads as I think you said you had trialed.
    
    As for the suggestion that I take it anyway, and then remove it later
    after it gets removed from OpenSFS, I remain unconvinced.
    You mention "years of patches written with it there"  but the first
    usage of the cfs_ptask_init only landed in March 2017 (less than 2 years
    ago).  libcfs_ptask is only use in lustre/obdclass/ lustre/llite/
    lustre/lov/ and the total patches in these directories since it was
    introduced in 319.  I suspect most of them aren't related to ptask.
    
    So I see no evidence that there will be much "nasty porting work".  I
    suspect there will be some, but porting code is what I spend a lot of my
    time doing, and doing it helps force me to understand the code.
    
    So what this isn't a "no way, never", it is "I'm not convinced".
    
    Thanks,
    NeilBrown
    
    
    >
    > Curious how folks feel about this.  I’d be willing to take a stab at writing a removal patch for 2.13.  It pains me a bit to suggest giving up on it, but Jinshan and I want to do write container type work to improve writes, and there’s the older/new again DDN parallel readahead work for reads.
    >
    > ________________________________
    > From: lustre-devel <lustre-devel-bounces at lists.lustre.org> on behalf of Patrick Farrell <paf at cray.com>
    > Sent: Tuesday, November 27, 2018 7:51:02 AM
    > To: Andreas Dilger; NeilBrown
    > Cc: Lustre Development List
    > Subject: Re: [lustre-devel] [PATCH 10/12] lustre: clio: Introduce parallel tasks framework
    >
    > Two notes coming, first about padata.
    >
    > A major reason is actually the infrastructure itself - it’s inappropriate to our kinds of tasks.  I did a quick talk on it a while back, intending then to fix it, but never got the chance (and since had better ideas to improve write performance):
    >
    > https://www.eofs.eu/_media/events/devsummit17/patrick_farrell_laddevsummit_pio.pdf
    >
    > padata basically bakes in a set of assumptions that amount to “functionally infinite amount of small work units and a dedicated machine”, which fit well with its role in packet encryption but don’t sit well for other kinds of paralelliziation.  (For example, all work is strictly and explicitly bound to a CPU.  No scheduler.  One more as a bonus - it distributes work across all allowed CPUs, but that means if you have a small number of work items (which splitting I/O tends to be because you have to make relatively big chunks) that effectively every work unit starts a worker thread for itself.)
    >
    > The recent discussion of a new parallel inaction framework on LWN looked intriguing for future work.  it’s expected to fix a number of the limitations.
    > https://lwn.net/Articles/771169/
    >
    > ________________________________
    > From: lustre-devel <lustre-devel-bounces at lists.lustre.org> on behalf of Andreas Dilger <adilger at whamcloud.com>
    > Sent: Monday, November 26, 2018 11:08:45 PM
    > To: NeilBrown
    > Cc: Lustre Development List
    > Subject: Re: [lustre-devel] [PATCH 10/12] lustre: clio: Introduce parallel tasks framework
    >
    > On Nov 26, 2018, at 21:20, NeilBrown <neilb at suse.com> wrote:
    >>
    >> On Sun, Nov 25 2018, James Simmons wrote:
    >>
    >>> From: Dmitry Eremin <dmitry.eremin at intel.com>
    >>>
    >>> In this patch new API for parallel tasks execution is introduced.
    >>> This API based on Linux kernel padata API which is used to perform
    >>> encryption and decryption on large numbers of packets without
    >>> reordering those packets.
    >>>
    >>> It was adopted for general use in Lustre for parallelization of
    >>> various functionality. The first place of its usage is parallel I/O
    >>> implementation.
    >>>
    >>> The first step in using it is to set up a cl_ptask structure to
    >>> control of how this task are to be run:
    >>>
    >>>    #include <cl_ptask.h>
    >>>
    >>>    int cl_ptask_init(struct cl_ptask *ptask, cl_ptask_cb_t cbfunc,
    >>>                      void *cbdata, unsigned int flags, int cpu);
    >>>
    >>> The cbfunc function with cbdata argument will be called in the process
    >>> of getting the task done. The cpu specifies which CPU will be used for
    >>> the final callback when the task is done.
    >>>
    >>> The submission of task is done with:
    >>>
    >>>    int cl_ptask_submit(struct cl_ptask *ptask,
    >>>                        struct cl_ptask_engine *engine);
    >>>
    >>> The task is submitted to the engine for execution.
    >>>
    >>> In order to wait for result of task execution you should call:
    >>>
    >>>   int cl_ptask_wait_for(struct cl_ptask *ptask);
    >>>
    >>> The tasks with flag PTF_ORDERED are executed in parallel but complete
    >>> into submission order. So, waiting for last ordered task you can be sure
    >>> that all previous tasks were done before this task complete.
    >>>
    >>> This patch differs from the OpenSFS tree by adding this functional
    >>> to the clio layer instead of libcfs.
    >>
    >> While you are right that it shouldn't be in libcfs, it actually
    >> shouldn't exist at all.
    >> cfs_ptask_init() is used precisely once in OpenSFS.  There is no point
    >> creating a generic API wrapper like this that is only used once.
    >>
    >> cl_oi needs to use padata API calls directly.
    >
    > This infrastructure was also going to be used for parallel readahead, but the patch that implemented that was never landed because the expected performance gains didn't materialize.
    >
    > Cheers, Andreas
    > ---
    > Andreas Dilger
    > Principal Lustre Architect
    > Whamcloud
    >
    >
    >
    >
    >
    >
    >
    > _______________________________________________
    > lustre-devel mailing list
    > lustre-devel at lists.lustre.org
    > http://lists.lustre.org/listinfo.cgi/lustre-devel-lustre.org
    


From charlie at whamcloud.com  Tue Nov 27 23:10:41 2018
From: charlie at whamcloud.com (Charlie Olmstead)
Date: Tue, 27 Nov 2018 23:10:41 +0000
Subject: [lustre-devel] Autotest: review-ldiskfs-arm session
Message-ID: <88C22267-CA3B-44C8-84BD-13DBFD7F7493@contoso.com>

Converting review-ldiskfs-arm will be delayed until some issues are resolved.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-devel_lists.lustre.org/attachments/20181127/ca452954/attachment.html>

From neilb at suse.com  Wed Nov 28 01:34:14 2018
From: neilb at suse.com (NeilBrown)
Date: Wed, 28 Nov 2018 12:34:14 +1100
Subject: [lustre-devel] drivers/staging updated to 4.20-rc1
In-Reply-To: <alpine.LFD.2.21.1811052058590.11915@casper.infradead.org>
References: <87k1lsyssq.fsf@notabene.neil.brown.name>
 <alpine.LFD.2.21.1811052058590.11915@casper.infradead.org>
Message-ID: <87h8g2oui1.fsf@notabene.neil.brown.name>

On Mon, Nov 05 2018, James Simmons wrote:

>> 
>> Hi,
>>  I've just pushed out new lustre/lustre, lustre/lustre-testing, lustre/lustre-wip
>>  to github.com:neilbrown/linux.git
>> 
>>  lustre is updated to 4.20-rc1 and doesn't compile because of some
>>  changes.
>>  The patch below makes it compile and (probably) work.
>> 
>>  I'd appreciate a review, especially of the iov_for_each change.
>
> Almost. I see the following errors while testing.
>
> [ 7111.002667] RIP: 0010:sanity+0xb2/0xb9
> [ 7111.002668] Code: 63 c5 48 c7 c7 a1 7c e8 81 ff c5 48 6b c0 28 48 03 43 
> 78 8b 48 08 48 8b 70 10 44 8b 40 0c 48 8b 10 31 c0 e8 4c 67 d6 ff  c
> [ 7111.002669] RSP: 0018:ffffc900072d3a60 EFLAGS: 00010246
> [ 7111.002670] RAX: 000000000000002a RBX: ffff8808502126c0 RCX: 
> 0000000000000006
> [ 7111.002671] RDX: 0000000000000000 RSI: ffffffff81e770e3 RDI: 
> 00000000ffffffff
> [ 7111.002672] RBP: 0000000000000010 R08: 0000000000000004 R09: 
> ffffffff82683e8e
> [ 7111.002673] R10: 00000000000015a9 R11: 0000000000000000 R12: 
> ffffea003f9523c0
> [ 7111.002674] R13: ffff8808502126c0 R14: ffffea003f9523c0 R15: 
> 0000000000001000
> [ 7111.002676] FS:  00007f140a193740(0000) GS:ffff88107fd80000(0000) 
> knlGS:0000000000000000
> [ 7111.002677] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [ 7111.002678] CR2: 000000000067a198 CR3: 00000008305dc001 CR4: 
> 00000000000606e0
> [ 7111.002679] Call Trace:
> [ 7111.002682]  copy_page_to_iter+0x245/0x2f0
> [ 7111.002686]  generic_file_buffered_read+0x345/0x7c0
> [ 7111.002689]  ? touch_atime+0x33/0xd0
> [ 7111.002701]  vvp_io_read_start+0x22a/0x3a0 [lustre]
> [ 7111.002715]  cl_io_start+0x6d/0x130 [obdclass]
> [ 7111.002727]  cl_io_loop+0xae/0x220 [obdclass]
> [ 7111.002736]  ll_file_io_generic+0x4ee/0x880 [lustre]
> [ 7111.002740]  ? generic_file_buffered_read+0x591/0x7c0
> [ 7111.002749]  ll_file_read_iter+0x13e/0x1b0 [lustre]
> [ 7111.002751]  generic_file_splice_read+0xe1/0x190
> [ 7111.002755]  splice_direct_to_actor+0xea/0x200
> [ 7111.002757]  ? generic_pipe_buf_nosteal+0x10/0x10
> [ 7111.002759]  do_splice_direct+0x87/0xd0
> [ 7111.002762]  do_sendfile+0x1c6/0x3c0
> [ 7111.002765]  __x64_sys_sendfile64+0x5c/0xb0
> [ 7111.002768]  do_syscall_64+0x68/0x38f
> [ 7111.002770]  ? do_page_fault+0x2d/0x130
> [ 7111.002773]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
> [ 7111.002775] RIP: 0033:0x7f1409ca71ca
>

Thanks for testing, and sorry that I didn't respond at the time - I
managed to lose track of it somehow.
Anyway, I hit testing problems that were quite different but in
a generally similar area.
This fixes them, and might fix yours too.  It'll appear in
lustre-testing shortly.

Thanks,
NeilBrown

From 77e9a84550bce7a77c0bc2e24455f11c72d1cc26 Mon Sep 17 00:00:00 2001
From: NeilBrown <neilb at suse.com>
Date: Wed, 28 Nov 2018 12:28:34 +1100
Subject: [PATCH] lustre: fix iov_for_each() replaced in vvp_mmap_locks()

iov_for_each() only makes sense of ITER_IOVEC and ITER_KVEC, not - for
exmaple - on ITER_PIPE.
When I replaced the call with an open-coded version I incorrectly left
that check out - resulting in problems when testing.
So add it back.

Fixes: 85bdcd0ea139 ("lustre: fixes for 4.20-rc1")
Signed-off-by: NeilBrown <neilb at suse.com>
---
 drivers/staging/lustre/lustre/llite/vvp_io.c | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/drivers/staging/lustre/lustre/llite/vvp_io.c b/drivers/staging/lustre/lustre/llite/vvp_io.c
index 524f3f4feec8..d6b27bae8226 100644
--- a/drivers/staging/lustre/lustre/llite/vvp_io.c
+++ b/drivers/staging/lustre/lustre/llite/vvp_io.c
@@ -388,6 +388,9 @@ static int vvp_mmap_locks(const struct lu_env *env,
 	if (!mm)
 		return 0;
 
+	if (iov_iter_type(vio->vui_iter) != ITER_IOVEC &&
+	    iov_iter_type(vio->vui_iter) != ITER_KVEC)
+		return 0;
 	for (i = *vio->vui_iter;
 	     i.count;
 	     iov_iter_advance(&i, iov.iov_len)) {
-- 
2.14.0.rc0.dirty

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 832 bytes
Desc: not available
URL: <http://lists.lustre.org/pipermail/lustre-devel_lists.lustre.org/attachments/20181128/04aa99cd/attachment.sig>

From jsimmons at infradead.org  Fri Nov 30 02:59:51 2018
From: jsimmons at infradead.org (James Simmons)
Date: Fri, 30 Nov 2018 02:59:51 +0000 (GMT)
Subject: [lustre-devel] reviving whamcloud linux lustre client
Message-ID: <alpine.LFD.2.21.1811300245590.16646@casper.infradead.org>


Hello :-)

    With the linux lustre client getting into shape its time to revive
the whamcloud linux tree for testing. Currently 

ssh://user at review.whamcloud.com:29418/fs/linux-staging

points to an outdated staging tree hosted by Greg. Can you migrate
the tree hosting to the following:

https://github.com/neilbrown/linux
    branches:
	remotes/origin/lustre/lustre
        remotes/origin/lustre/lustre-testing
        remotes/origin/lustre/lustre-wip

Can you use the current lustre 2.10 LTS test suite for testing and enabled
all the test!!!

From lai.siyao at whamcloud.com  Thu Nov 29 12:06:12 2018
From: lai.siyao at whamcloud.com (Siyao Lai)
Date: Thu, 29 Nov 2018 12:06:12 +0000
Subject: [lustre-devel] [PATCH 06/12] lustre: mdc: don't add to page
 cache upon failure
In-Reply-To: <875zwjql4l.fsf@notabene.neil.brown.name>
References: <1543200508-6838-1-git-send-email-jsimmons@infradead.org>
 <1543200508-6838-7-git-send-email-jsimmons@infradead.org>
 <875zwjql4l.fsf@notabene.neil.brown.name>
Message-ID: <DFD6BB83-222B-42E1-98DA-9DF41C8F5755@ddn.com>

You're right, I'll fix it later.

Thanks,
Lai

On 2018/11/27, 11:01 AM, "NeilBrown" <neilb at suse.com> wrote:

    On Sun, Nov 25 2018, James Simmons wrote:
    
    > From: Lai Siyao <lai.siyao at whamcloud.com>
    >
    > Reading directory pages may fail on MDS, in this case client should
    > not cache a non-up-to-date directory page, because it will cause
    > a later read on the same page fail.
    >
    > Signed-off-by: Lai Siyao <lai.siyao at whamcloud.com>
    > WC-bug-id: https://jira.whamcloud.com/browse/LU-5461
    > Reviewed-on: http://review.whamcloud.com/11450
    > Reviewed-by: Fan Yong <fan.yong at intel.com>
    > Reviewed-by: John L. Hammond <jhammond at whamcloud.com>
    > Reviewed-by: Oleg Drokin <green at whamcloud.com>
    > Signed-off-by: James Simmons <jsimmons at infradead.org>
    > ---
    >  drivers/staging/lustre/lustre/mdc/mdc_request.c | 5 ++++-
    >  1 file changed, 4 insertions(+), 1 deletion(-)
    >
    > diff --git a/drivers/staging/lustre/lustre/mdc/mdc_request.c b/drivers/staging/lustre/lustre/mdc/mdc_request.c
    > index 1072b66..09b30ef 100644
    > --- a/drivers/staging/lustre/lustre/mdc/mdc_request.c
    > +++ b/drivers/staging/lustre/lustre/mdc/mdc_request.c
    > @@ -1212,7 +1212,10 @@ static int mdc_read_page_remote(void *data, struct page *page0)
    >  	}
    >  
    >  	rc = mdc_getpage(rp->rp_exp, fid, rp->rp_off, page_pool, npages, &req);
    > -	if (!rc) {
    > +	if (rc < 0) {
    > +		/* page0 is special, which was added into page cache early */
    > +		delete_from_page_cache(page0);
    
    This looks wrong to me.  We shouldn't need to delete the page from the
    page-cache.
    It won't be marked up-to-date, so why will that cause an error on a
    later read???
    
    Well, because mdc_page_locate() finds a page and if it isn't up-to-date,
    it returns -EIO.  Why does it do that?  If it found a PageError() page,
    then it might be reasonable to return -EIO.
    
    Why not just return the page that was found, and let the caller check if
    it is Uptodate?
    
    Well, because mdc_read_page() handles a successful page return from
    mdc_page_locate() as a hash collision.
    
    I guess I need to understand how lustre maps a hash to a directory
    block, and then how it handles collisions...
    
    The reason this jumped out at me is that it looks like it might be
    racy.  Adding a page and then removing it might leave a window where
    some other thread can find the page.  That is not a problem is a
    non-up-to-date page just means we should wait for it.  But if it can
    cause an error, then maybe the race is a real problem.
    
    But maybe there is some higher level locking...
    
    NeilBrown