From dmitriy at oss-tech.org  Wed Jul  5 22:03:21 2017
From: dmitriy at oss-tech.org (Dmitriy Cherkasov)
Date: Wed, 5 Jul 2017 22:03:21 +0000
Subject: [lustre-devel] Dead code cleanup
Message-ID: <2d7d67d4-3716-c521-e7dd-b8f934330c7e@oss-tech.org>

Folks,

I've found one last #if 0 block in the code, but when attempting to investigate 
its origin I ran into a dead end.

The code in question [1] has not been touched since 2013 [2], and seems to 
originate from one big patch [3] with a non-public commit history. I found the 
relevant bugzilla entry [4], but no history beyond that.

Although I was unable to find the original intent behind it, I think it's safe 
to assume it is not used and if there are no objections I'll post a patch to 
remove it.

Thanks,
Dmitriy

[1] 
https://git.hpdd.intel.com/?p=fs/lustre-release.git;a=blob;f=lustre/lov/lov_io.c;h=2339ea56;hb=HEAD#l1067
[2] https://git.hpdd.intel.com/?p=fs/lustre-release.git;a=commit;h=d2589286
[3] https://git.hpdd.intel.com/?p=fs/lustre-release.git;a=commit;h=fbf5870b
[4] https://projectlava.xyratex.com/show_bug.cgi?id=14166

From fransklaver at gmail.com  Thu Jul  6 07:23:53 2017
From: fransklaver at gmail.com (Frans Klaver)
Date: Thu, 6 Jul 2017 09:23:53 +0200
Subject: [lustre-devel] [PATCH 8/8] Staging: lustre :lustre: include
 :lustre_compat.h: Prefer using the BIT macro
In-Reply-To: <1499325195-17466-1-git-send-email-rjdurga@gmail.com>
References: <1499325195-17466-1-git-send-email-rjdurga@gmail.com>
Message-ID: <CAH6sp9PuJ_zxK1D=t_i2aAeeNzaoUP+UqnGQC_Pin-QNgjJ7hg@mail.gmail.com>

On Thu, Jul 6, 2017 at 9:13 AM, Jaya Durga <rjdurga at gmail.com> wrote:
> Subject: Staging: lustre :lustre: include :lustre_compat.h: Prefer using the BIT macro

Don't overdo it ;-).

Subject: staging: lustre: lustre_compat.h: Prefer using the BIT macro

> Replace all instances of (1 << 27) with BIT(27) to fix
> checkpatch check messages

While it may technically be true that this one instance is every
instance of (1<<27) there is in lustre_compat.h, I must say I expected
a bigger patch when I saw "replace all instances".


> Signed-off-by: Jaya Durga <rjdurga at gmail.com>
> ---
>  drivers/staging/lustre/lustre/include/lustre_compat.h | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/drivers/staging/lustre/lustre/include/lustre_compat.h b/drivers/staging/lustre/lustre/include/lustre_compat.h
> index da9ce19..686a251 100644
> --- a/drivers/staging/lustre/lustre/include/lustre_compat.h
> +++ b/drivers/staging/lustre/lustre/include/lustre_compat.h
> @@ -43,7 +43,7 @@
>   * set ATTR_BLOCKS to a high value to avoid any risk of collision with other
>   * ATTR_* attributes (see bug 13828)
>   */
> -#define ATTR_BLOCKS    (1 << 27)
> +#define ATTR_BLOCKS    BIT(27)
>
>  #define current_ngroups current_cred()->group_info->ngroups
>  #define current_groups current_cred()->group_info->small_block
> --
> 1.9.1
>
> _______________________________________________
> devel mailing list
> devel at linuxdriverproject.org
> http://driverdev.linuxdriverproject.org/mailman/listinfo/driverdev-devel

From jlayton at kernel.org  Wed Jul  5 16:37:44 2017
From: jlayton at kernel.org (Jeff Layton)
Date: Wed,  5 Jul 2017 12:37:44 -0400
Subject: [lustre-devel] [PATCH] lustre: don't set f_version in ll_readdir
Message-ID: <20170705163744.3228-1-jlayton@kernel.org>

From: Jeff Layton <jlayton at redhat.com>

f_version is only ever used by filesystem-specific code. Generic
VFS code never uses it.

Nothing in lustre ever looks at it, so just remove this.

Signed-off-by: Jeff Layton <jlayton at redhat.com>
---
 drivers/staging/lustre/lustre/llite/dir.c | 3 ---
 1 file changed, 3 deletions(-)

diff --git a/drivers/staging/lustre/lustre/llite/dir.c b/drivers/staging/lustre/lustre/llite/dir.c
index 03a72c07f57c..ab53500007bf 100644
--- a/drivers/staging/lustre/lustre/llite/dir.c
+++ b/drivers/staging/lustre/lustre/llite/dir.c
@@ -367,8 +367,6 @@ static int ll_readdir(struct file *filp, struct dir_context *ctx)
 	}
 	ctx->pos = pos;
 	ll_finish_md_op_data(op_data);
-	filp->f_version = inode->i_version;
-
 out:
 	if (!rc)
 		ll_stats_ops_tally(sbi, LPROC_LL_READDIR, 1);
@@ -1675,7 +1673,6 @@ static loff_t ll_dir_seek(struct file *file, loff_t offset, int origin)
 			else
 				fd->lfd_pos = offset;
 			file->f_pos = offset;
-			file->f_version = 0;
 		}
 		ret = offset;
 	}
-- 
2.13.0

From rjdurga at gmail.com  Thu Jul  6 07:13:15 2017
From: rjdurga at gmail.com (Jaya Durga)
Date: Thu,  6 Jul 2017 12:43:15 +0530
Subject: [lustre-devel] [PATCH 8/8] Staging: lustre :lustre: include
	:lustre_compat.h: Prefer using the BIT macro
Message-ID: <1499325195-17466-1-git-send-email-rjdurga@gmail.com>

Replace all instances of (1 << 27) with BIT(27) to fix
checkpatch check messages

Signed-off-by: Jaya Durga <rjdurga at gmail.com>
---
 drivers/staging/lustre/lustre/include/lustre_compat.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/staging/lustre/lustre/include/lustre_compat.h b/drivers/staging/lustre/lustre/include/lustre_compat.h
index da9ce19..686a251 100644
--- a/drivers/staging/lustre/lustre/include/lustre_compat.h
+++ b/drivers/staging/lustre/lustre/include/lustre_compat.h
@@ -43,7 +43,7 @@
  * set ATTR_BLOCKS to a high value to avoid any risk of collision with other
  * ATTR_* attributes (see bug 13828)
  */
-#define ATTR_BLOCKS    (1 << 27)
+#define ATTR_BLOCKS    BIT(27)
 
 #define current_ngroups current_cred()->group_info->ngroups
 #define current_groups current_cred()->group_info->small_block
-- 
1.9.1

From dmitriy at oss-tech.org  Thu Jul  6 19:14:04 2017
From: dmitriy at oss-tech.org (Dmitriy Cherkasov)
Date: Thu, 6 Jul 2017 12:14:04 -0700
Subject: [lustre-devel] [PATCH v4] staging: lustre: lnet: remove dead
 code and crc32_le() wrapper
In-Reply-To: <1498801841-5799-1-git-send-email-dmitriy@oss-tech.org>
References: <1498794766-12298-1-git-send-email-dmitriy@oss-tech.org>
	<1498801841-5799-1-git-send-email-dmitriy@oss-tech.org>
Message-ID: <71e755f4-eac5-f74f-7add-399355141b16@oss-tech.org>

On 06/29/2017 10:50 PM, Dmitriy Cherkasov wrote:
> After removing code which was permanently disabled with ifdefs, the
> function ksocknal_csum() becomes just a wrapper for crc32_le(). Remove
> this useless wrapper and instead call crc32_le() directly.
> 

Any feedback on v4 please?

From Craig at craiginches.com  Fri Jul  7 01:46:26 2017
From: Craig at craiginches.com (Craig Inches)
Date: Fri, 07 Jul 2017 01:46:26 +0000 (UTC)
Subject: [lustre-devel] [PATCH 0/4] Staging: Lustre Style Fixes
Message-ID: <cover.1499388016.git.Craig@craiginches.com>

This series fixes a few style issues in lustre. All where picked up
by checkpatch. 

Craig Inches (4):
  Staging: Luster: Clean up line over 80Char in lib-lnet.h
  Staging: Lustre Fix up multiple Block Comments in lib-types.h
  Staging: Lustre Fixing multiline block comments in lnetst.h
  Staging: Lustre Fix block statement style issue

 .../staging/lustre/include/linux/lnet/lib-lnet.h   |   3 +-
 .../staging/lustre/include/linux/lnet/lib-types.h  |  46 +++++---
 drivers/staging/lustre/include/linux/lnet/lnetst.h | 129 +++++++++++++--------
 .../staging/lustre/include/linux/lnet/socklnd.h    |   3 +-
 4 files changed, 115 insertions(+), 66 deletions(-)

-- 
2.13.0

From Craig at craiginches.com  Fri Jul  7 01:46:42 2017
From: Craig at craiginches.com (Craig Inches)
Date: Fri, 07 Jul 2017 01:46:42 +0000 (UTC)
Subject: [lustre-devel] [PATCH 1/4] Staging: Luster: Clean up line over
	80Char in lib-lnet.h
In-Reply-To: <cover.1499388016.git.Craig@craiginches.com>
References: <cover.1499388016.git.Craig@craiginches.com>
Message-ID: <64eeec691ca65e0fa68f2fcc5e7a04518f4971b8.1499388016.git.Craig@craiginches.com>

This patch fixes a warning generated by checkpatch for
a line over 80 characters.

Signed-off-by: Craig Inches <Craig at craiginches.com>
---
 drivers/staging/lustre/include/linux/lnet/lib-lnet.h | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/drivers/staging/lustre/include/linux/lnet/lib-lnet.h b/drivers/staging/lustre/include/linux/lnet/lib-lnet.h
index 8ae7423b4543..f534115d402a 100644
--- a/drivers/staging/lustre/include/linux/lnet/lib-lnet.h
+++ b/drivers/staging/lustre/include/linux/lnet/lib-lnet.h
@@ -453,7 +453,8 @@ extern int portal_rotor;
 int lnet_lib_init(void);
 void lnet_lib_exit(void);
 
-int lnet_notify(struct lnet_ni *ni, lnet_nid_t peer, int alive, unsigned long when);
+int lnet_notify(struct lnet_ni *ni, lnet_nid_t peer, int alive,
+		unsigned long when);
 void lnet_notify_locked(struct lnet_peer *lp, int notifylnd, int alive,
 			unsigned long when);
 int lnet_add_route(__u32 net, __u32 hops, lnet_nid_t gateway_nid,
-- 
2.13.0

From Craig at craiginches.com  Fri Jul  7 01:46:54 2017
From: Craig at craiginches.com (Craig Inches)
Date: Fri, 07 Jul 2017 01:46:54 +0000 (UTC)
Subject: [lustre-devel] [PATCH 2/4] Staging: Lustre Fix up multiple Block
 Comments in lib-types.h
In-Reply-To: <cover.1499388016.git.Craig@craiginches.com>
References: <cover.1499388016.git.Craig@craiginches.com>
Message-ID: <e58fb4521c39f2858c16e20757979383a008ed0a.1499388016.git.Craig@craiginches.com>

This patch fixes some multiline comment blocks which didnt conform
to the style guide, found by checkpatch.

Signed-off-by: Craig Inches <Craig at craiginches.com>
---
 .../staging/lustre/include/linux/lnet/lib-types.h  | 46 ++++++++++++++--------
 1 file changed, 30 insertions(+), 16 deletions(-)

diff --git a/drivers/staging/lustre/include/linux/lnet/lib-types.h b/drivers/staging/lustre/include/linux/lnet/lib-types.h
index 321752dfe58b..ddb808ed5d0b 100644
--- a/drivers/staging/lustre/include/linux/lnet/lib-types.h
+++ b/drivers/staging/lustre/include/linux/lnet/lib-types.h
@@ -308,9 +308,11 @@ struct lnet_rc_data {
 struct lnet_peer {
 	struct list_head	 lp_hashlist;	/* chain on peer hash */
 	struct list_head	 lp_txq;	/* messages blocking for
-						   tx credits */
+						 * tx credits
+						 */
 	struct list_head	 lp_rtrq;	/* messages blocking for
-						   router credits */
+						 * router credits
+						 */
 	struct list_head	 lp_rtr_list;	/* chain on router list */
 	int			 lp_txcredits;	/* # tx credits available */
 	int			 lp_mintxcredits;  /* low water mark */
@@ -319,23 +321,31 @@ struct lnet_peer {
 	unsigned int		 lp_alive:1;	   /* alive/dead? */
 	unsigned int		 lp_notify:1;	/* notification outstanding? */
 	unsigned int		 lp_notifylnd:1;/* outstanding notification
-						   for LND? */
+						 * for LND?
+						 */
 	unsigned int		 lp_notifying:1; /* some thread is handling
-						    notification */
+						  * notification
+						  */
 	unsigned int		 lp_ping_notsent;/* SEND event outstanding
-						    from ping */
+						  * from ping
+						  */
 	int			 lp_alive_count; /* # times router went
-						    dead<->alive */
-	long			 lp_txqnob;	 /* bytes queued for sending */
+						  * dead<->alive
+						  */
+	long			 lp_txqnob;	 /* ytes queued for sending */
 	unsigned long		 lp_timestamp;	 /* time of last aliveness
-						    news */
+						  * news
+						  */
 	unsigned long		 lp_ping_timestamp;/* time of last ping
-						      attempt */
+						    * attempt
+						    */
 	unsigned long		 lp_ping_deadline; /* != 0 if ping reply
-						      expected */
+						    * expected
+						    */
 	unsigned long		 lp_last_alive;	/* when I was last alive */
 	unsigned long		 lp_last_query;	/* when lp_ni was queried
-						   last time */
+						 * last time
+						 */
 	struct lnet_ni		*lp_ni;		/* interface peer is on */
 	lnet_nid_t		 lp_nid;	/* peer's NID */
 	int			 lp_refcount;	/* # refs */
@@ -386,7 +396,8 @@ struct lnet_route {
 
 struct lnet_remotenet {
 	struct list_head	lrn_list;	/* chain on
-						   ln_remote_nets_hash */
+						 * ln_remote_nets_hash
+						 */
 	struct list_head	lrn_routes;	/* routes to me */
 	__u32			lrn_net;	/* my net number */
 };
@@ -399,14 +410,16 @@ struct lnet_remotenet {
 struct lnet_rtrbufpool {
 	struct list_head	rbp_bufs;	/* my free buffer pool */
 	struct list_head	rbp_msgs;	/* messages blocking
-						   for a buffer */
+						 * for a buffer
+						 */
 	int			rbp_npages;	/* # pages in each buffer */
 	/* requested number of buffers */
 	int			rbp_req_nbuffers;
 	/* # buffers actually allocated */
 	int			rbp_nbuffers;
-	int			rbp_credits;	/* # free buffers /
-						     blocked messages */
+	int			rbp_credits;	/* # free buffers
+						 * blocked messages
+						 */
 	int			rbp_mincredits;	/* low water mark */
 };
 
@@ -442,7 +455,8 @@ enum lnet_match_flags {
 #define LNET_PTL_LAZY		(1 << 0)
 #define LNET_PTL_MATCH_UNIQUE	(1 << 1)	/* unique match, for RDMA */
 #define LNET_PTL_MATCH_WILDCARD	(1 << 2)	/* wildcard match,
-						   request portal */
+						 * request portal
+						 */
 
 /* parameter for matching operations (GET, PUT) */
 struct lnet_match_info {
-- 
2.13.0

From Craig at craiginches.com  Fri Jul  7 01:47:04 2017
From: Craig at craiginches.com (Craig Inches)
Date: Fri, 07 Jul 2017 01:47:04 +0000 (UTC)
Subject: [lustre-devel] [PATCH 3/4] Staging: Lustre Fixing multiline block
 comments in lnetst.h
In-Reply-To: <cover.1499388016.git.Craig@craiginches.com>
References: <cover.1499388016.git.Craig@craiginches.com>
Message-ID: <dea8ec86c52eb0f89a75774953af127b7e7a7e64.1499388016.git.Craig@craiginches.com>

This fixes multiple block statements found not to match
style as per checkpatch

Signed-off-by: Craig Inches <Craig at craiginches.com>
---
 drivers/staging/lustre/include/linux/lnet/lnetst.h | 129 +++++++++++++--------
 1 file changed, 81 insertions(+), 48 deletions(-)

diff --git a/drivers/staging/lustre/include/linux/lnet/lnetst.h b/drivers/staging/lustre/include/linux/lnet/lnetst.h
index ea736f8d5231..a4f9ff01d458 100644
--- a/drivers/staging/lustre/include/linux/lnet/lnetst.h
+++ b/drivers/staging/lustre/include/linux/lnet/lnetst.h
@@ -54,7 +54,8 @@
 #define LSTIO_GROUP_ADD		0xC10	/* add group */
 #define LSTIO_GROUP_LIST	0xC11	/* list all groups in session */
 #define LSTIO_GROUP_INFO	0xC12	/* query default information of
-					 * specified group */
+					 * specified group
+					 */
 #define LSTIO_GROUP_DEL		0xC13	/* delete group */
 #define LSTIO_NODES_ADD		0xC14	/* add nodes to specified group */
 #define LSTIO_GROUP_UPDATE      0xC15	/* update group */
@@ -102,27 +103,32 @@ struct lstcon_test_ent {
 	int	tse_type;       /* test type */
 	int	tse_loop;       /* loop count */
 	int	tse_concur;     /* concurrency of test */
-};				/*** test summary entry, for
-				 *** list_batch command */
+};				/* test summary entry, for
+				 * list_batch command
+				 */
 
 struct lstcon_batch_ent {
 	int	bae_state;	/* batch status */
 	int	bae_timeout;	/* batch timeout */
 	int	bae_ntest;	/* # of tests in the batch */
-};				/*** batch summary entry, for
-				 *** list_batch command */
+};				/* batch summary entry, for
+				 * list_batch command
+				 */
 
 struct lstcon_test_batch_ent {
 	struct lstcon_ndlist_ent   tbe_cli_nle;	/* client (group) node_list
-						 * entry */
+						 * entry
+						 */
 	struct lstcon_ndlist_ent   tbe_srv_nle;	/* server (group) node_list
-						 * entry */
+						 * entry
+						 */
 	union {
 		struct lstcon_test_ent	tbe_test; /* test entry */
 		struct lstcon_batch_ent tbe_batch;/* batch entry */
 	} u;
-};				/*** test/batch verbose information entry,
-				 *** for list_batch command */
+};				/* test/batch verbose information entry,
+				 * for list_batch command
+				 */
 
 struct lstcon_rpc_ent {
 	struct list_head	rpe_link;	/* link chain */
@@ -138,10 +144,10 @@ struct lstcon_rpc_ent {
 };
 
 struct lstcon_trans_stat {
-	int	trs_rpc_stat[4];	/* RPCs stat (0: total
-						      1: failed
-						      2: finished
-						      4: reserved */
+	int	trs_rpc_stat[4];	/* RPCs stat (0: total 1: failed
+					 * 2: finished
+					 * 4: reserved
+					 */
 	int	trs_rpc_errno;		/* RPC errno */
 	int	trs_fwk_stat[8];	/* framework stat */
 	int	trs_fwk_errno;		/* errno of the first remote error */
@@ -275,22 +281,28 @@ struct lstio_session_end_args {
 struct lstio_debug_args {
 	int			 lstio_dbg_key;		/* IN: session key */
 	int			 lstio_dbg_type;	/* IN: debug
-								session|batch|
-								group|nodes
-								list */
+							 * session|batch|
+							 * group|nodes list
+							 */
 	int			 lstio_dbg_flags;	/* IN: reserved debug
-							       flags */
+							 * flags
+							 */
 	int			 lstio_dbg_timeout;	/* IN: timeout of
-							       debug */
+							 * debug
+							 */
 	int			 lstio_dbg_nmlen;	/* IN: len of name */
 	char __user		*lstio_dbg_namep;	/* IN: name of
-							       group|batch */
+							 * group|batch
+							 */
 	int			 lstio_dbg_count;	/* IN: # of test nodes
-							       to debug */
+							 * to debug
+							 */
 	struct lnet_process_id __user *lstio_dbg_idsp;	/* IN: id of test
-							       nodes */
+							 * nodes
+							 */
 	struct list_head __user	*lstio_dbg_resultp;	/* OUT: list head of
-								result buffer */
+							 * result buffer
+							 */
 };
 
 struct lstio_group_add_args {
@@ -307,7 +319,8 @@ struct lstio_group_del_args {
 
 #define LST_GROUP_CLEAN		1	/* remove inactive nodes in the group */
 #define LST_GROUP_REFRESH	2	/* refresh inactive nodes
-					 * in the group */
+					 * in the group
+					 */
 #define LST_GROUP_RMND		3	/* delete nodes from the group */
 
 struct lstio_group_update_args {
@@ -319,7 +332,8 @@ struct lstio_group_update_args {
 	int			 lstio_grp_count;	/* IN: # of nodes id */
 	struct lnet_process_id __user *lstio_grp_idsp;	/* IN: array of nodes */
 	struct list_head __user	*lstio_grp_resultp;	/* OUT: list head of
-								result buffer */
+							 * result buffer
+							 */
 };
 
 struct lstio_group_nodes_args {
@@ -331,7 +345,8 @@ struct lstio_group_nodes_args {
 	unsigned int __user	*lstio_grp_featp;
 	struct lnet_process_id __user *lstio_grp_idsp;	/* IN: nodes */
 	struct list_head __user	*lstio_grp_resultp;	/* OUT: list head of
-								result buffer */
+							 * result buffer
+							 */
 };
 
 struct lstio_group_list_args {
@@ -345,8 +360,9 @@ struct lstio_group_info_args {
 	int			 lstio_grp_key;		/* IN: session key */
 	int			 lstio_grp_nmlen;	/* IN: name len */
 	char __user		*lstio_grp_namep;	/* IN: name */
-	struct lstcon_ndlist_ent __user *lstio_grp_entp;/* OUT: description of
-							   group */
+	struct lstcon_ndlist_ent __user *lstio_grp_entp;/* OUT: description
+							 * of group
+							 */
 	int __user		*lstio_grp_idxp;	/* IN/OUT: node index */
 	int __user		*lstio_grp_ndentp;	/* IN/OUT: # of nodent */
 	struct lstcon_node_ent __user *lstio_grp_dentsp;/* OUT: nodent array */
@@ -369,34 +385,41 @@ struct lstio_batch_del_args {
 struct lstio_batch_run_args {
 	int			 lstio_bat_key;		/* IN: session key */
 	int			 lstio_bat_timeout;	/* IN: timeout for
-							       the batch */
+							 * the batch
+							 */
 	int			 lstio_bat_nmlen;	/* IN: name length */
 	char __user		*lstio_bat_namep;	/* IN: batch name */
 	struct list_head __user	*lstio_bat_resultp;	/* OUT: list head of
-								result buffer */
+							 * result buffer
+							 */
 };
 
 struct lstio_batch_stop_args {
 	int			 lstio_bat_key;		/* IN: session key */
 	int			 lstio_bat_force;	/* IN: abort unfinished
-							       test RPC */
+							 * test RPC
+							 */
 	int			 lstio_bat_nmlen;	/* IN: name length */
 	char __user		*lstio_bat_namep;	/* IN: batch name */
 	struct list_head __user	*lstio_bat_resultp;	/* OUT: list head of
-								result buffer */
+							 * result buffer
+							 */
 };
 
 struct lstio_batch_query_args {
 	int			 lstio_bat_key;		/* IN: session key */
 	int			 lstio_bat_testidx;	/* IN: test index */
 	int			 lstio_bat_client;	/* IN: we testing
-							       client? */
+							 * client?
+							 */
 	int			 lstio_bat_timeout;	/* IN: timeout for
-							       waiting */
+							 * waiting
+							 */
 	int			 lstio_bat_nmlen;	/* IN: name length */
 	char __user		*lstio_bat_namep;	/* IN: batch name */
 	struct list_head __user	*lstio_bat_resultp;	/* OUT: list head of
-								result buffer */
+							 * result buffer
+							 */
 };
 
 struct lstio_batch_list_args {
@@ -411,7 +434,8 @@ struct lstio_batch_info_args {
 	int			 lstio_bat_nmlen;	/* IN: name length */
 	char __user		*lstio_bat_namep;	/* IN: name */
 	int			 lstio_bat_server;	/* IN: query server
-							       or not */
+							 * or not
+							 */
 	int			 lstio_bat_testidx;	/* IN: test index */
 	struct lstcon_test_batch_ent __user *lstio_bat_entp;/* OUT: batch ent */
 
@@ -424,14 +448,17 @@ struct lstio_batch_info_args {
 struct lstio_stat_args {
 	int			 lstio_sta_key;		/* IN: session key */
 	int			 lstio_sta_timeout;	/* IN: timeout for
-							       stat request */
+							 * stat request
+							 */
 	int			 lstio_sta_nmlen;	/* IN: group name
-							       length */
+							 * length
+							 */
 	char __user		*lstio_sta_namep;	/* IN: group name */
 	int			 lstio_sta_count;	/* IN: # of pid */
 	struct lnet_process_id __user *lstio_sta_idsp;	/* IN: pid */
 	struct list_head __user	*lstio_sta_resultp;	/* OUT: list head of
-								result buffer */
+							 * result buffer
+							 */
 };
 
 enum lst_test_type {
@@ -452,26 +479,32 @@ struct lstio_test_args {
 	int		  lstio_tes_concur;	/* IN: concurrency */
 
 	int		  lstio_tes_dist;	/* IN: node distribution in
-						       destination groups */
+						 * destination groups
+						 */
 	int		  lstio_tes_span;	/* IN: node span in
-						       destination groups */
+						 * destination groups
+						 */
 	int		  lstio_tes_sgrp_nmlen;	/* IN: source group
-						       name length */
+						 * name length
+						 */
 	char __user	 *lstio_tes_sgrp_name;	/* IN: group name */
 	int		  lstio_tes_dgrp_nmlen;	/* IN: destination group
-						       name length */
+						 * name length
+						 */
 	char __user	 *lstio_tes_dgrp_name;	/* IN: group name */
 
 	int		  lstio_tes_param_len;	/* IN: param buffer len */
 	void __user	 *lstio_tes_param;	/* IN: parameter for specified
-						       test:
-						       lstio_bulk_param_t,
-						       lstio_ping_param_t,
-						       ... more */
+						 * test: lstio_bulk_param_t,
+						 * lstio_ping_param_t,
+						 * ... more
+						 */
 	int __user	 *lstio_tes_retp;	/* OUT: private returned
-							value */
+						 * value
+						 */
 	struct list_head __user *lstio_tes_resultp;/* OUT: list head of
-							result buffer */
+						    * result buffer
+						    */
 };
 
 enum lst_brw_type {
-- 
2.13.0

From Craig at craiginches.com  Fri Jul  7 01:47:20 2017
From: Craig at craiginches.com (Craig Inches)
Date: Fri, 07 Jul 2017 01:47:20 +0000 (UTC)
Subject: [lustre-devel] [PATCH 4/4] Staging: Lustre Fix block statement
	style issue
In-Reply-To: <cover.1499388016.git.Craig@craiginches.com>
References: <cover.1499388016.git.Craig@craiginches.com>
Message-ID: <62fcfe7c2367fb58e4ef0c8d5963739bf4c48a58.1499388016.git.Craig@craiginches.com>

This fixes a block statement which didnt end with */

Signed-off-by: Craig Inches <Craig at craiginches.com>
---
 drivers/staging/lustre/include/linux/lnet/socklnd.h | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/drivers/staging/lustre/include/linux/lnet/socklnd.h b/drivers/staging/lustre/include/linux/lnet/socklnd.h
index dd5bc0e46560..a1ae66ede7a8 100644
--- a/drivers/staging/lustre/include/linux/lnet/socklnd.h
+++ b/drivers/staging/lustre/include/linux/lnet/socklnd.h
@@ -76,7 +76,8 @@ struct ksock_msg {
 	__u64	ksm_zc_cookies[2];	/* Zero-Copy request/ACK cookie */
 	union {
 		struct ksock_lnet_msg lnetmsg; /* lnet message, it's empty if
-					  * it's NOOP */
+						* it's NOOP
+						*/
 	} WIRE_ATTR ksm_u;
 } WIRE_ATTR;
 
-- 
2.13.0

From dmitriy at oss-tech.org  Fri Jul  7 23:03:42 2017
From: dmitriy at oss-tech.org (Dmitriy Cherkasov)
Date: Fri,  7 Jul 2017 16:03:42 -0700
Subject: [lustre-devel] [PATCH v5] staging: lustre: lnet: remove dead code
	and useless wrapper
In-Reply-To: <1498801841-5799-1-git-send-email-dmitriy@oss-tech.org>
References: <1498801841-5799-1-git-send-email-dmitriy@oss-tech.org>
Message-ID: <1499468622-10276-1-git-send-email-dmitriy@oss-tech.org>

After removing commented out code, ksocknal_csum() becomes a useless 
wrapper for crc32_le(). Remove it, and instead call crc32_le() directly.

Fixes the following checkpatch warning:

WARNING: space prohibited before semicolon

Signed-off-by: Dmitriy Cherkasov <dmitriy at oss-tech.org>
---
v5:
	- fix incomplete patch changelog
	- improve code alignment
	- reword commit message and summary to be more concise
v4:
	- fix spacing issues
	- fix typo in commit message
	- add patch changelog
v3:
	- fix spacing and alignment issues 
	  (suggested by Joe Perches)
v2:
	- remove ksocknal_csum() and call crc32_le() directly
	  (suggested by Greg KH)
	- update commit message and summary to reflect these changes

 drivers/staging/lustre/lnet/klnds/socklnd/socklnd.h  | 11 -----------
 .../staging/lustre/lnet/klnds/socklnd/socklnd_lib.c  | 20 ++++++++++----------
 2 files changed, 10 insertions(+), 21 deletions(-)

diff --git a/drivers/staging/lustre/lnet/klnds/socklnd/socklnd.h b/drivers/staging/lustre/lnet/klnds/socklnd/socklnd.h
index 5540de6..9eb169d 100644
--- a/drivers/staging/lustre/lnet/klnds/socklnd/socklnd.h
+++ b/drivers/staging/lustre/lnet/klnds/socklnd/socklnd.h
@@ -519,17 +519,6 @@ struct ksock_proto {
 #define CPU_MASK_NONE   0UL
 #endif
 
-static inline __u32 ksocknal_csum(__u32 crc, unsigned char const *p, size_t len)
-{
-#if 1
-	return crc32_le(crc, p, len);
-#else
-	while (len-- > 0)
-		crc = ((crc + 0x100) & ~0xff) | ((crc + *p++) & 0xff) ;
-	return crc;
-#endif
-}
-
 static inline int
 ksocknal_route_mask(void)
 {
diff --git a/drivers/staging/lustre/lnet/klnds/socklnd/socklnd_lib.c b/drivers/staging/lustre/lnet/klnds/socklnd/socklnd_lib.c
index 8a036f4..9c328dc 100644
--- a/drivers/staging/lustre/lnet/klnds/socklnd/socklnd_lib.c
+++ b/drivers/staging/lustre/lnet/klnds/socklnd/socklnd_lib.c
@@ -201,9 +201,9 @@
 			if (fragnob > sum)
 				fragnob = sum;
 
-			conn->ksnc_rx_csum = ksocknal_csum(conn->ksnc_rx_csum,
-							   iov[i].iov_base,
-							   fragnob);
+			conn->ksnc_rx_csum = crc32_le(conn->ksnc_rx_csum,
+						      iov[i].iov_base,
+						      fragnob);
 		}
 		conn->ksnc_msg.ksm_csum = saved_csum;
 	}
@@ -243,8 +243,8 @@
 			if (fragnob > sum)
 				fragnob = sum;
 
-			conn->ksnc_rx_csum = ksocknal_csum(conn->ksnc_rx_csum,
-							   base, fragnob);
+			conn->ksnc_rx_csum = crc32_le(conn->ksnc_rx_csum,
+						      base, fragnob);
 
 			kunmap(kiov[i].bv_page);
 		}
@@ -265,22 +265,22 @@
 
 	tx->tx_msg.ksm_csum = 0;
 
-	csum = ksocknal_csum(~0, tx->tx_iov[0].iov_base,
-			     tx->tx_iov[0].iov_len);
+	csum = crc32_le(~0, tx->tx_iov[0].iov_base,
+			tx->tx_iov[0].iov_len);
 
 	if (tx->tx_kiov) {
 		for (i = 0; i < tx->tx_nkiov; i++) {
 			base = kmap(tx->tx_kiov[i].bv_page) +
 			       tx->tx_kiov[i].bv_offset;
 
-			csum = ksocknal_csum(csum, base, tx->tx_kiov[i].bv_len);
+			csum = crc32_le(csum, base, tx->tx_kiov[i].bv_len);
 
 			kunmap(tx->tx_kiov[i].bv_page);
 		}
 	} else {
 		for (i = 1; i < tx->tx_niov; i++)
-			csum = ksocknal_csum(csum, tx->tx_iov[i].iov_base,
-					     tx->tx_iov[i].iov_len);
+			csum = crc32_le(csum, tx->tx_iov[i].iov_base,
+					tx->tx_iov[i].iov_len);
 	}
 
 	if (*ksocknal_tunables.ksnd_inject_csum_error) {
-- 
1.9.1

From arnd at arndb.de  Mon Jul 10 13:08:03 2017
From: arnd at arndb.de (Arnd Bergmann)
Date: Mon, 10 Jul 2017 15:08:03 +0200
Subject: [lustre-devel] [PATCH] lustre: check copy_from_iter/copy_to_iter
	return code
Message-ID: <20170710130833.1834210-1-arnd@arndb.de>

We now get a helpful warning for code that calls copy_{from,to}_iter
without checking the return value, introduced by commit aa28de275a24
("iov_iter/hardening: move object size checks to inlined part").

drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd_cb.c: In function 'kiblnd_send':
drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd_cb.c:1643:2: error: ignoring return value of 'copy_from_iter', declared with attribute warn_unused_result [-Werror=unused-result]
drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd_cb.c: In function 'kiblnd_recv':
drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd_cb.c:1744:3: error: ignoring return value of 'copy_to_iter', declared with attribute warn_unused_result [-Werror=unused-result]

In case we get short copies here, we may get incorrect behavior.
I've added failure handling for both rx and tx now, returning
-EFAULT as expected.

Cc: stable at vger.kernel.org
Signed-off-by: Arnd Bergmann <arnd at arndb.de>
---
This warning now shows up in 'allmodconfig' builds, so it would be
good to get it fixed quickly for 4.13, but my patch should not go
in without careful review since I'm not familiar with with code
and the error handling is a bit tricky here.

I added 'Cc: stable' since this is a preexisting condition that we
only started warning about now.
---
 drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd_cb.c | 15 +++++++++++++--
 1 file changed, 13 insertions(+), 2 deletions(-)

diff --git a/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd_cb.c b/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd_cb.c
index 85b242ec5f9b..70256a0ffd2e 100644
--- a/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd_cb.c
+++ b/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd_cb.c
@@ -1640,8 +1640,13 @@ kiblnd_send(struct lnet_ni *ni, void *private, struct lnet_msg *lntmsg)
 	ibmsg = tx->tx_msg;
 	ibmsg->ibm_u.immediate.ibim_hdr = *hdr;
 
-	copy_from_iter(&ibmsg->ibm_u.immediate.ibim_payload, IBLND_MSG_SIZE,
+	rc = copy_from_iter(&ibmsg->ibm_u.immediate.ibim_payload, IBLND_MSG_SIZE,
 		       &from);
+	if (rc != IBLND_MSG_SIZE) {
+		kiblnd_pool_free_node(&tx->tx_pool->tpo_pool, &tx->tx_list);
+		return -EFAULT;
+	}
+
 	nob = offsetof(struct kib_immediate_msg, ibim_payload[payload_nob]);
 	kiblnd_init_tx_msg(ni, tx, IBLND_MSG_IMMEDIATE, nob);
 
@@ -1741,8 +1746,14 @@ kiblnd_recv(struct lnet_ni *ni, void *private, struct lnet_msg *lntmsg,
 			break;
 		}
 
-		copy_to_iter(&rxmsg->ibm_u.immediate.ibim_payload,
+		rc = copy_to_iter(&rxmsg->ibm_u.immediate.ibim_payload,
 			     IBLND_MSG_SIZE, to);
+		if (rc != IBLND_MSG_SIZE) {
+			rc = -EFAULT;
+			break;
+		}
+
+		rc = 0;
 		lnet_finalize(ni, lntmsg, 0);
 		break;
 
-- 
2.9.0

From gregkh at linuxfoundation.org  Tue Jul 11 17:08:18 2017
From: gregkh at linuxfoundation.org (Greg KH)
Date: Tue, 11 Jul 2017 19:08:18 +0200
Subject: [lustre-devel] [PATCH 8/8] Staging: lustre :lustre: include
 :lustre_compat.h: Prefer using the BIT macro
In-Reply-To: <1499325195-17466-1-git-send-email-rjdurga@gmail.com>
References: <1499325195-17466-1-git-send-email-rjdurga@gmail.com>
Message-ID: <20170711170818.GA5013@kroah.com>

On Thu, Jul 06, 2017 at 12:43:15PM +0530, Jaya Durga wrote:
> Replace all instances of (1 << 27) with BIT(27) to fix
> checkpatch check messages
> 
> Signed-off-by: Jaya Durga <rjdurga at gmail.com>
> ---
>  drivers/staging/lustre/lustre/include/lustre_compat.h | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/drivers/staging/lustre/lustre/include/lustre_compat.h b/drivers/staging/lustre/lustre/include/lustre_compat.h
> index da9ce19..686a251 100644
> --- a/drivers/staging/lustre/lustre/include/lustre_compat.h
> +++ b/drivers/staging/lustre/lustre/include/lustre_compat.h
> @@ -43,7 +43,7 @@
>   * set ATTR_BLOCKS to a high value to avoid any risk of collision with other
>   * ATTR_* attributes (see bug 13828)
>   */
> -#define ATTR_BLOCKS    (1 << 27)
> +#define ATTR_BLOCKS    BIT(27)

Isn't this used in lustre's userspace code?  If so, you can't use the
BIT() macro there :(

Please check before you redo this.

thanks,

greg k-h

From gregkh at linuxfoundation.org  Tue Jul 11 17:13:40 2017
From: gregkh at linuxfoundation.org (Greg Kroah-Hartman)
Date: Tue, 11 Jul 2017 19:13:40 +0200
Subject: [lustre-devel] [PATCH 1/4] Staging: Luster: Clean up line over
 80Char in lib-lnet.h
In-Reply-To: <64eeec691ca65e0fa68f2fcc5e7a04518f4971b8.1499388016.git.Craig@craiginches.com>
References: <cover.1499388016.git.Craig@craiginches.com>
	<64eeec691ca65e0fa68f2fcc5e7a04518f4971b8.1499388016.git.Craig@craiginches.com>
Message-ID: <20170711171340.GB11838@kroah.com>

On Fri, Jul 07, 2017 at 01:46:42AM +0000, Craig Inches wrote:
> This patch fixes a warning generated by checkpatch for
> a line over 80 characters.
> 
> Signed-off-by: Craig Inches <Craig at craiginches.com>
> ---
>  drivers/staging/lustre/include/linux/lnet/lib-lnet.h | 3 ++-
>  1 file changed, 2 insertions(+), 1 deletion(-)

Your subject line mispelled the filesystem's name :(

From gregkh at linuxfoundation.org  Tue Jul 11 17:14:49 2017
From: gregkh at linuxfoundation.org (Greg Kroah-Hartman)
Date: Tue, 11 Jul 2017 19:14:49 +0200
Subject: [lustre-devel] [PATCH 3/4] Staging: Lustre Fixing multiline
 block comments in lnetst.h
In-Reply-To: <dea8ec86c52eb0f89a75774953af127b7e7a7e64.1499388016.git.Craig@craiginches.com>
References: <cover.1499388016.git.Craig@craiginches.com>
	<dea8ec86c52eb0f89a75774953af127b7e7a7e64.1499388016.git.Craig@craiginches.com>
Message-ID: <20170711171449.GC11838@kroah.com>

On Fri, Jul 07, 2017 at 01:47:04AM +0000, Craig Inches wrote:
> This fixes multiple block statements found not to match
> style as per checkpatch
> 
> Signed-off-by: Craig Inches <Craig at craiginches.com>
> ---
>  drivers/staging/lustre/include/linux/lnet/lnetst.h | 129 +++++++++++++--------
>  1 file changed, 81 insertions(+), 48 deletions(-)
> 
> diff --git a/drivers/staging/lustre/include/linux/lnet/lnetst.h b/drivers/staging/lustre/include/linux/lnet/lnetst.h
> index ea736f8d5231..a4f9ff01d458 100644
> --- a/drivers/staging/lustre/include/linux/lnet/lnetst.h
> +++ b/drivers/staging/lustre/include/linux/lnet/lnetst.h
> @@ -54,7 +54,8 @@
>  #define LSTIO_GROUP_ADD		0xC10	/* add group */
>  #define LSTIO_GROUP_LIST	0xC11	/* list all groups in session */
>  #define LSTIO_GROUP_INFO	0xC12	/* query default information of
> -					 * specified group */
> +					 * specified group
> +					 */
>  #define LSTIO_GROUP_DEL		0xC13	/* delete group */
>  #define LSTIO_NODES_ADD		0xC14	/* add nodes to specified group */
>  #define LSTIO_GROUP_UPDATE      0xC15	/* update group */
> @@ -102,27 +103,32 @@ struct lstcon_test_ent {
>  	int	tse_type;       /* test type */
>  	int	tse_loop;       /* loop count */
>  	int	tse_concur;     /* concurrency of test */
> -};				/*** test summary entry, for
> -				 *** list_batch command */
> +};				/* test summary entry, for
> +				 * list_batch command
> +				 */

That's odd, what was the *** stuff for?

I'd like to get a lustre maintainer's ack for all of these before I
apply them...

thanks,

greg k-h

From andreas.dilger at intel.com  Wed Jul 12 05:09:16 2017
From: andreas.dilger at intel.com (Dilger, Andreas)
Date: Wed, 12 Jul 2017 05:09:16 +0000
Subject: [lustre-devel] [PATCH 3/4] Staging: Lustre Fixing multiline
 block comments in lnetst.h
In-Reply-To: <20170711171449.GC11838@kroah.com>
References: <cover.1499388016.git.Craig@craiginches.com>
	<dea8ec86c52eb0f89a75774953af127b7e7a7e64.1499388016.git.Craig@craiginches.com>
	<20170711171449.GC11838@kroah.com>
Message-ID: <80BA3F43-DFB0-45B2-A36A-83628E731A88@intel.com>

On Jul 11, 2017, at 11:14, Greg Kroah-Hartman <gregkh at linuxfoundation.org> wrote:
> 
> On Fri, Jul 07, 2017 at 01:47:04AM +0000, Craig Inches wrote:
>> This fixes multiple block statements found not to match
>> style as per checkpatch
>> 
>> Signed-off-by: Craig Inches <Craig at craiginches.com>
>> ---
>> drivers/staging/lustre/include/linux/lnet/lnetst.h | 129 +++++++++++++--------
>> 1 file changed, 81 insertions(+), 48 deletions(-)
>> 
>> diff --git a/drivers/staging/lustre/include/linux/lnet/lnetst.h b/drivers/staging/lustre/include/linux/lnet/lnetst.h
>> index ea736f8d5231..a4f9ff01d458 100644
>> --- a/drivers/staging/lustre/include/linux/lnet/lnetst.h
>> +++ b/drivers/staging/lustre/include/linux/lnet/lnetst.h
>> @@ -54,7 +54,8 @@
>> #define LSTIO_GROUP_ADD		0xC10	/* add group */
>> #define LSTIO_GROUP_LIST	0xC11	/* list all groups in session */
>> #define LSTIO_GROUP_INFO	0xC12	/* query default information of
>> -					 * specified group */
>> +					 * specified group
>> +					 */
>> #define LSTIO_GROUP_DEL		0xC13	/* delete group */
>> #define LSTIO_NODES_ADD		0xC14	/* add nodes to specified group */
>> #define LSTIO_GROUP_UPDATE      0xC15	/* update group */
>> @@ -102,27 +103,32 @@ struct lstcon_test_ent {
>> 	int	tse_type;       /* test type */
>> 	int	tse_loop;       /* loop count */
>> 	int	tse_concur;     /* concurrency of test */
>> -};				/*** test summary entry, for
>> -				 *** list_batch command */
>> +};				/* test summary entry, for
>> +				 * list_batch command
>> +				 */
> 
> That's odd, what was the *** stuff for?
> 
> I'd like to get a lustre maintainer's ack for all of these before I
> apply them...

It's nothing that I know about.  We used DOxygen to comment the code and generate docs,
but the "***" isn't any markup that I'm familiar with.

Reviewed-by: Andreas Dilger <andreas.dilger at intel.com>

Cheers, Andreas
--
Andreas Dilger
Lustre Principal Architect
Intel Corporation







From andreas.dilger at intel.com  Wed Jul 12 05:13:11 2017
From: andreas.dilger at intel.com (Dilger, Andreas)
Date: Wed, 12 Jul 2017 05:13:11 +0000
Subject: [lustre-devel] [PATCH 8/8] Staging: lustre :lustre: include
 :lustre_compat.h: Prefer using the BIT macro
In-Reply-To: <20170711170818.GA5013@kroah.com>
References: <1499325195-17466-1-git-send-email-rjdurga@gmail.com>
	<20170711170818.GA5013@kroah.com>
Message-ID: <326250A0-4785-40C4-BA11-2D708815039B@intel.com>


> On Jul 11, 2017, at 11:08, Greg KH <gregkh at linuxfoundation.org> wrote:
> 
> On Thu, Jul 06, 2017 at 12:43:15PM +0530, Jaya Durga wrote:
>> Replace all instances of (1 << 27) with BIT(27) to fix
>> checkpatch check messages
>> 
>> Signed-off-by: Jaya Durga <rjdurga at gmail.com>
>> ---
>> drivers/staging/lustre/lustre/include/lustre_compat.h | 2 +-
>> 1 file changed, 1 insertion(+), 1 deletion(-)
>> 
>> diff --git a/drivers/staging/lustre/lustre/include/lustre_compat.h b/drivers/staging/lustre/lustre/include/lustre_compat.h
>> index da9ce19..686a251 100644
>> --- a/drivers/staging/lustre/lustre/include/lustre_compat.h
>> +++ b/drivers/staging/lustre/lustre/include/lustre_compat.h
>> @@ -43,7 +43,7 @@
>>  * set ATTR_BLOCKS to a high value to avoid any risk of collision with other
>>  * ATTR_* attributes (see bug 13828)
>>  */
>> -#define ATTR_BLOCKS    (1 << 27)
>> +#define ATTR_BLOCKS    BIT(27)
> 
> Isn't this used in lustre's userspace code?  If so, you can't use the
> BIT() macro there :(
> 
> Please check before you redo this.

The "lustre_compat.h" header was previously used for compatibility between different kernel
versions, which is why it is now basically empty.  It isn't used for userspace interfaces
as other "compat" headers are in the kernel.

Cheers, Andreas
--
Andreas Dilger
Lustre Principal Architect
Intel Corporation







From Craig at craiginches.com  Wed Jul 12 09:02:28 2017
From: Craig at craiginches.com (Craig Inches)
Date: Wed, 12 Jul 2017 09:02:28 +0000 (UTC)
Subject: [lustre-devel] [PATCH v2 0/4] Staging: Lustre Style Fixes
Message-ID: <cover.1499849566.git.Craig@craiginches.com>

This series fixes a few style issues in lustre. All where picked up                                                                                                                                               
by checkpatch. 

v2 Changes:
- Fixed typo in patch subject s/luster/lustre/

Craig Inches (4):
  Staging: Lustre Clean up line over 80Char in lib-lnet.h
  Staging: Lustre Fix up multiple Block Comments in lib-types.h
  Staging: Lustre Fixing multiline block comments in lnetst.h
  Staging: Lustre Fix block statement style issue

 .../staging/lustre/include/linux/lnet/lib-lnet.h   |   3 +-
 .../staging/lustre/include/linux/lnet/lib-types.h  |  46 +++++---
 drivers/staging/lustre/include/linux/lnet/lnetst.h | 129 +++++++++++++--------
 .../staging/lustre/include/linux/lnet/socklnd.h    |   3 +-
 4 files changed, 115 insertions(+), 66 deletions(-)

-- 
2.13.0

From Craig at craiginches.com  Wed Jul 12 09:02:40 2017
From: Craig at craiginches.com (Craig Inches)
Date: Wed, 12 Jul 2017 09:02:40 +0000 (UTC)
Subject: [lustre-devel] [PATCH v2 1/4] Staging: Lustre Clean up line over
 80Char in lib-lnet.h
In-Reply-To: <cover.1499849566.git.Craig@craiginches.com>
References: <cover.1499849566.git.Craig@craiginches.com>
Message-ID: <fabf2712145dae43b5538d2c5f9e1a6f89dec4a0.1499849566.git.Craig@craiginches.com>

This patch fixes a warning generated by checkpatch for
a line over 80 characters.

Signed-off-by: Craig Inches <Craig at craiginches.com>
---
 drivers/staging/lustre/include/linux/lnet/lib-lnet.h | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/drivers/staging/lustre/include/linux/lnet/lib-lnet.h b/drivers/staging/lustre/include/linux/lnet/lib-lnet.h
index 8ae7423b4543..f534115d402a 100644
--- a/drivers/staging/lustre/include/linux/lnet/lib-lnet.h
+++ b/drivers/staging/lustre/include/linux/lnet/lib-lnet.h
@@ -453,7 +453,8 @@ extern int portal_rotor;
 int lnet_lib_init(void);
 void lnet_lib_exit(void);
 
-int lnet_notify(struct lnet_ni *ni, lnet_nid_t peer, int alive, unsigned long when);
+int lnet_notify(struct lnet_ni *ni, lnet_nid_t peer, int alive,
+		unsigned long when);
 void lnet_notify_locked(struct lnet_peer *lp, int notifylnd, int alive,
 			unsigned long when);
 int lnet_add_route(__u32 net, __u32 hops, lnet_nid_t gateway_nid,
-- 
2.13.0

From Craig at craiginches.com  Wed Jul 12 09:02:53 2017
From: Craig at craiginches.com (Craig Inches)
Date: Wed, 12 Jul 2017 09:02:53 +0000 (UTC)
Subject: [lustre-devel] [PATCH v2 2/4] Staging: Lustre Fix up multiple Block
 Comments in lib-types.h
In-Reply-To: <cover.1499849566.git.Craig@craiginches.com>
References: <cover.1499849566.git.Craig@craiginches.com>
Message-ID: <f99ea9e68fa0dcf8b843c2b7926ab20f3697d896.1499849566.git.Craig@craiginches.com>

This patch fixes some multiline comment blocks which didnt conform
to the style guide, found by checkpatch.

Signed-off-by: Craig Inches <Craig at craiginches.com>
---
 .../staging/lustre/include/linux/lnet/lib-types.h  | 46 ++++++++++++++--------
 1 file changed, 30 insertions(+), 16 deletions(-)

diff --git a/drivers/staging/lustre/include/linux/lnet/lib-types.h b/drivers/staging/lustre/include/linux/lnet/lib-types.h
index 321752dfe58b..ddb808ed5d0b 100644
--- a/drivers/staging/lustre/include/linux/lnet/lib-types.h
+++ b/drivers/staging/lustre/include/linux/lnet/lib-types.h
@@ -308,9 +308,11 @@ struct lnet_rc_data {
 struct lnet_peer {
 	struct list_head	 lp_hashlist;	/* chain on peer hash */
 	struct list_head	 lp_txq;	/* messages blocking for
-						   tx credits */
+						 * tx credits
+						 */
 	struct list_head	 lp_rtrq;	/* messages blocking for
-						   router credits */
+						 * router credits
+						 */
 	struct list_head	 lp_rtr_list;	/* chain on router list */
 	int			 lp_txcredits;	/* # tx credits available */
 	int			 lp_mintxcredits;  /* low water mark */
@@ -319,23 +321,31 @@ struct lnet_peer {
 	unsigned int		 lp_alive:1;	   /* alive/dead? */
 	unsigned int		 lp_notify:1;	/* notification outstanding? */
 	unsigned int		 lp_notifylnd:1;/* outstanding notification
-						   for LND? */
+						 * for LND?
+						 */
 	unsigned int		 lp_notifying:1; /* some thread is handling
-						    notification */
+						  * notification
+						  */
 	unsigned int		 lp_ping_notsent;/* SEND event outstanding
-						    from ping */
+						  * from ping
+						  */
 	int			 lp_alive_count; /* # times router went
-						    dead<->alive */
-	long			 lp_txqnob;	 /* bytes queued for sending */
+						  * dead<->alive
+						  */
+	long			 lp_txqnob;	 /* ytes queued for sending */
 	unsigned long		 lp_timestamp;	 /* time of last aliveness
-						    news */
+						  * news
+						  */
 	unsigned long		 lp_ping_timestamp;/* time of last ping
-						      attempt */
+						    * attempt
+						    */
 	unsigned long		 lp_ping_deadline; /* != 0 if ping reply
-						      expected */
+						    * expected
+						    */
 	unsigned long		 lp_last_alive;	/* when I was last alive */
 	unsigned long		 lp_last_query;	/* when lp_ni was queried
-						   last time */
+						 * last time
+						 */
 	struct lnet_ni		*lp_ni;		/* interface peer is on */
 	lnet_nid_t		 lp_nid;	/* peer's NID */
 	int			 lp_refcount;	/* # refs */
@@ -386,7 +396,8 @@ struct lnet_route {
 
 struct lnet_remotenet {
 	struct list_head	lrn_list;	/* chain on
-						   ln_remote_nets_hash */
+						 * ln_remote_nets_hash
+						 */
 	struct list_head	lrn_routes;	/* routes to me */
 	__u32			lrn_net;	/* my net number */
 };
@@ -399,14 +410,16 @@ struct lnet_remotenet {
 struct lnet_rtrbufpool {
 	struct list_head	rbp_bufs;	/* my free buffer pool */
 	struct list_head	rbp_msgs;	/* messages blocking
-						   for a buffer */
+						 * for a buffer
+						 */
 	int			rbp_npages;	/* # pages in each buffer */
 	/* requested number of buffers */
 	int			rbp_req_nbuffers;
 	/* # buffers actually allocated */
 	int			rbp_nbuffers;
-	int			rbp_credits;	/* # free buffers /
-						     blocked messages */
+	int			rbp_credits;	/* # free buffers
+						 * blocked messages
+						 */
 	int			rbp_mincredits;	/* low water mark */
 };
 
@@ -442,7 +455,8 @@ enum lnet_match_flags {
 #define LNET_PTL_LAZY		(1 << 0)
 #define LNET_PTL_MATCH_UNIQUE	(1 << 1)	/* unique match, for RDMA */
 #define LNET_PTL_MATCH_WILDCARD	(1 << 2)	/* wildcard match,
-						   request portal */
+						 * request portal
+						 */
 
 /* parameter for matching operations (GET, PUT) */
 struct lnet_match_info {
-- 
2.13.0

From Craig at craiginches.com  Wed Jul 12 09:03:04 2017
From: Craig at craiginches.com (Craig Inches)
Date: Wed, 12 Jul 2017 09:03:04 +0000 (UTC)
Subject: [lustre-devel] [PATCH v2 3/4] Staging: Lustre Fixing multiline
 block comments in lnetst.h
In-Reply-To: <cover.1499849566.git.Craig@craiginches.com>
References: <cover.1499849566.git.Craig@craiginches.com>
Message-ID: <4645128edf267dcef75f58d23e9a00efe684e76a.1499849566.git.Craig@craiginches.com>

This fixes multiple block statements found not to match
style as per checkpatch

Signed-off-by: Craig Inches <Craig at craiginches.com>
---
 drivers/staging/lustre/include/linux/lnet/lnetst.h | 129 +++++++++++++--------
 1 file changed, 81 insertions(+), 48 deletions(-)

diff --git a/drivers/staging/lustre/include/linux/lnet/lnetst.h b/drivers/staging/lustre/include/linux/lnet/lnetst.h
index ea736f8d5231..a4f9ff01d458 100644
--- a/drivers/staging/lustre/include/linux/lnet/lnetst.h
+++ b/drivers/staging/lustre/include/linux/lnet/lnetst.h
@@ -54,7 +54,8 @@
 #define LSTIO_GROUP_ADD		0xC10	/* add group */
 #define LSTIO_GROUP_LIST	0xC11	/* list all groups in session */
 #define LSTIO_GROUP_INFO	0xC12	/* query default information of
-					 * specified group */
+					 * specified group
+					 */
 #define LSTIO_GROUP_DEL		0xC13	/* delete group */
 #define LSTIO_NODES_ADD		0xC14	/* add nodes to specified group */
 #define LSTIO_GROUP_UPDATE      0xC15	/* update group */
@@ -102,27 +103,32 @@ struct lstcon_test_ent {
 	int	tse_type;       /* test type */
 	int	tse_loop;       /* loop count */
 	int	tse_concur;     /* concurrency of test */
-};				/*** test summary entry, for
-				 *** list_batch command */
+};				/* test summary entry, for
+				 * list_batch command
+				 */
 
 struct lstcon_batch_ent {
 	int	bae_state;	/* batch status */
 	int	bae_timeout;	/* batch timeout */
 	int	bae_ntest;	/* # of tests in the batch */
-};				/*** batch summary entry, for
-				 *** list_batch command */
+};				/* batch summary entry, for
+				 * list_batch command
+				 */
 
 struct lstcon_test_batch_ent {
 	struct lstcon_ndlist_ent   tbe_cli_nle;	/* client (group) node_list
-						 * entry */
+						 * entry
+						 */
 	struct lstcon_ndlist_ent   tbe_srv_nle;	/* server (group) node_list
-						 * entry */
+						 * entry
+						 */
 	union {
 		struct lstcon_test_ent	tbe_test; /* test entry */
 		struct lstcon_batch_ent tbe_batch;/* batch entry */
 	} u;
-};				/*** test/batch verbose information entry,
-				 *** for list_batch command */
+};				/* test/batch verbose information entry,
+				 * for list_batch command
+				 */
 
 struct lstcon_rpc_ent {
 	struct list_head	rpe_link;	/* link chain */
@@ -138,10 +144,10 @@ struct lstcon_rpc_ent {
 };
 
 struct lstcon_trans_stat {
-	int	trs_rpc_stat[4];	/* RPCs stat (0: total
-						      1: failed
-						      2: finished
-						      4: reserved */
+	int	trs_rpc_stat[4];	/* RPCs stat (0: total 1: failed
+					 * 2: finished
+					 * 4: reserved
+					 */
 	int	trs_rpc_errno;		/* RPC errno */
 	int	trs_fwk_stat[8];	/* framework stat */
 	int	trs_fwk_errno;		/* errno of the first remote error */
@@ -275,22 +281,28 @@ struct lstio_session_end_args {
 struct lstio_debug_args {
 	int			 lstio_dbg_key;		/* IN: session key */
 	int			 lstio_dbg_type;	/* IN: debug
-								session|batch|
-								group|nodes
-								list */
+							 * session|batch|
+							 * group|nodes list
+							 */
 	int			 lstio_dbg_flags;	/* IN: reserved debug
-							       flags */
+							 * flags
+							 */
 	int			 lstio_dbg_timeout;	/* IN: timeout of
-							       debug */
+							 * debug
+							 */
 	int			 lstio_dbg_nmlen;	/* IN: len of name */
 	char __user		*lstio_dbg_namep;	/* IN: name of
-							       group|batch */
+							 * group|batch
+							 */
 	int			 lstio_dbg_count;	/* IN: # of test nodes
-							       to debug */
+							 * to debug
+							 */
 	struct lnet_process_id __user *lstio_dbg_idsp;	/* IN: id of test
-							       nodes */
+							 * nodes
+							 */
 	struct list_head __user	*lstio_dbg_resultp;	/* OUT: list head of
-								result buffer */
+							 * result buffer
+							 */
 };
 
 struct lstio_group_add_args {
@@ -307,7 +319,8 @@ struct lstio_group_del_args {
 
 #define LST_GROUP_CLEAN		1	/* remove inactive nodes in the group */
 #define LST_GROUP_REFRESH	2	/* refresh inactive nodes
-					 * in the group */
+					 * in the group
+					 */
 #define LST_GROUP_RMND		3	/* delete nodes from the group */
 
 struct lstio_group_update_args {
@@ -319,7 +332,8 @@ struct lstio_group_update_args {
 	int			 lstio_grp_count;	/* IN: # of nodes id */
 	struct lnet_process_id __user *lstio_grp_idsp;	/* IN: array of nodes */
 	struct list_head __user	*lstio_grp_resultp;	/* OUT: list head of
-								result buffer */
+							 * result buffer
+							 */
 };
 
 struct lstio_group_nodes_args {
@@ -331,7 +345,8 @@ struct lstio_group_nodes_args {
 	unsigned int __user	*lstio_grp_featp;
 	struct lnet_process_id __user *lstio_grp_idsp;	/* IN: nodes */
 	struct list_head __user	*lstio_grp_resultp;	/* OUT: list head of
-								result buffer */
+							 * result buffer
+							 */
 };
 
 struct lstio_group_list_args {
@@ -345,8 +360,9 @@ struct lstio_group_info_args {
 	int			 lstio_grp_key;		/* IN: session key */
 	int			 lstio_grp_nmlen;	/* IN: name len */
 	char __user		*lstio_grp_namep;	/* IN: name */
-	struct lstcon_ndlist_ent __user *lstio_grp_entp;/* OUT: description of
-							   group */
+	struct lstcon_ndlist_ent __user *lstio_grp_entp;/* OUT: description
+							 * of group
+							 */
 	int __user		*lstio_grp_idxp;	/* IN/OUT: node index */
 	int __user		*lstio_grp_ndentp;	/* IN/OUT: # of nodent */
 	struct lstcon_node_ent __user *lstio_grp_dentsp;/* OUT: nodent array */
@@ -369,34 +385,41 @@ struct lstio_batch_del_args {
 struct lstio_batch_run_args {
 	int			 lstio_bat_key;		/* IN: session key */
 	int			 lstio_bat_timeout;	/* IN: timeout for
-							       the batch */
+							 * the batch
+							 */
 	int			 lstio_bat_nmlen;	/* IN: name length */
 	char __user		*lstio_bat_namep;	/* IN: batch name */
 	struct list_head __user	*lstio_bat_resultp;	/* OUT: list head of
-								result buffer */
+							 * result buffer
+							 */
 };
 
 struct lstio_batch_stop_args {
 	int			 lstio_bat_key;		/* IN: session key */
 	int			 lstio_bat_force;	/* IN: abort unfinished
-							       test RPC */
+							 * test RPC
+							 */
 	int			 lstio_bat_nmlen;	/* IN: name length */
 	char __user		*lstio_bat_namep;	/* IN: batch name */
 	struct list_head __user	*lstio_bat_resultp;	/* OUT: list head of
-								result buffer */
+							 * result buffer
+							 */
 };
 
 struct lstio_batch_query_args {
 	int			 lstio_bat_key;		/* IN: session key */
 	int			 lstio_bat_testidx;	/* IN: test index */
 	int			 lstio_bat_client;	/* IN: we testing
-							       client? */
+							 * client?
+							 */
 	int			 lstio_bat_timeout;	/* IN: timeout for
-							       waiting */
+							 * waiting
+							 */
 	int			 lstio_bat_nmlen;	/* IN: name length */
 	char __user		*lstio_bat_namep;	/* IN: batch name */
 	struct list_head __user	*lstio_bat_resultp;	/* OUT: list head of
-								result buffer */
+							 * result buffer
+							 */
 };
 
 struct lstio_batch_list_args {
@@ -411,7 +434,8 @@ struct lstio_batch_info_args {
 	int			 lstio_bat_nmlen;	/* IN: name length */
 	char __user		*lstio_bat_namep;	/* IN: name */
 	int			 lstio_bat_server;	/* IN: query server
-							       or not */
+							 * or not
+							 */
 	int			 lstio_bat_testidx;	/* IN: test index */
 	struct lstcon_test_batch_ent __user *lstio_bat_entp;/* OUT: batch ent */
 
@@ -424,14 +448,17 @@ struct lstio_batch_info_args {
 struct lstio_stat_args {
 	int			 lstio_sta_key;		/* IN: session key */
 	int			 lstio_sta_timeout;	/* IN: timeout for
-							       stat request */
+							 * stat request
+							 */
 	int			 lstio_sta_nmlen;	/* IN: group name
-							       length */
+							 * length
+							 */
 	char __user		*lstio_sta_namep;	/* IN: group name */
 	int			 lstio_sta_count;	/* IN: # of pid */
 	struct lnet_process_id __user *lstio_sta_idsp;	/* IN: pid */
 	struct list_head __user	*lstio_sta_resultp;	/* OUT: list head of
-								result buffer */
+							 * result buffer
+							 */
 };
 
 enum lst_test_type {
@@ -452,26 +479,32 @@ struct lstio_test_args {
 	int		  lstio_tes_concur;	/* IN: concurrency */
 
 	int		  lstio_tes_dist;	/* IN: node distribution in
-						       destination groups */
+						 * destination groups
+						 */
 	int		  lstio_tes_span;	/* IN: node span in
-						       destination groups */
+						 * destination groups
+						 */
 	int		  lstio_tes_sgrp_nmlen;	/* IN: source group
-						       name length */
+						 * name length
+						 */
 	char __user	 *lstio_tes_sgrp_name;	/* IN: group name */
 	int		  lstio_tes_dgrp_nmlen;	/* IN: destination group
-						       name length */
+						 * name length
+						 */
 	char __user	 *lstio_tes_dgrp_name;	/* IN: group name */
 
 	int		  lstio_tes_param_len;	/* IN: param buffer len */
 	void __user	 *lstio_tes_param;	/* IN: parameter for specified
-						       test:
-						       lstio_bulk_param_t,
-						       lstio_ping_param_t,
-						       ... more */
+						 * test: lstio_bulk_param_t,
+						 * lstio_ping_param_t,
+						 * ... more
+						 */
 	int __user	 *lstio_tes_retp;	/* OUT: private returned
-							value */
+						 * value
+						 */
 	struct list_head __user *lstio_tes_resultp;/* OUT: list head of
-							result buffer */
+						    * result buffer
+						    */
 };
 
 enum lst_brw_type {
-- 
2.13.0

From Craig at craiginches.com  Wed Jul 12 09:03:28 2017
From: Craig at craiginches.com (Craig Inches)
Date: Wed, 12 Jul 2017 09:03:28 +0000 (UTC)
Subject: [lustre-devel] [PATCH v2 4/4] Staging: Lustre Fix block statement
	style issue
In-Reply-To: <cover.1499849566.git.Craig@craiginches.com>
References: <cover.1499849566.git.Craig@craiginches.com>
Message-ID: <774a2f25b839305f100e7fa99ebf835fb2ab7d44.1499849566.git.Craig@craiginches.com>

This fixes a block statement which didnt end with */

Signed-off-by: Craig Inches <Craig at craiginches.com>
---
 drivers/staging/lustre/include/linux/lnet/socklnd.h | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/drivers/staging/lustre/include/linux/lnet/socklnd.h b/drivers/staging/lustre/include/linux/lnet/socklnd.h
index dd5bc0e46560..a1ae66ede7a8 100644
--- a/drivers/staging/lustre/include/linux/lnet/socklnd.h
+++ b/drivers/staging/lustre/include/linux/lnet/socklnd.h
@@ -76,7 +76,8 @@ struct ksock_msg {
 	__u64	ksm_zc_cookies[2];	/* Zero-Copy request/ACK cookie */
 	union {
 		struct ksock_lnet_msg lnetmsg; /* lnet message, it's empty if
-					  * it's NOOP */
+						* it's NOOP
+						*/
 	} WIRE_ATTR ksm_u;
 } WIRE_ATTR;
 
-- 
2.13.0

From andreas.dilger at intel.com  Thu Jul 13 00:27:03 2017
From: andreas.dilger at intel.com (Dilger, Andreas)
Date: Thu, 13 Jul 2017 00:27:03 +0000
Subject: [lustre-devel] [PATCH v5] staging: lustre: lnet: remove dead
 code and useless wrapper
In-Reply-To: <1499468622-10276-1-git-send-email-dmitriy@oss-tech.org>
References: <1498801841-5799-1-git-send-email-dmitriy@oss-tech.org>
	<1499468622-10276-1-git-send-email-dmitriy@oss-tech.org>
Message-ID: <58312993-66C3-4B40-A8B7-A2BFC07069E8@intel.com>

On Jul 7, 2017, at 16:03, Dmitriy Cherkasov <dmitriy at oss-tech.org> wrote:
> 
> After removing commented out code, ksocknal_csum() becomes a useless 
> wrapper for crc32_le(). Remove it, and instead call crc32_le() directly.
> 
> Fixes the following checkpatch warning:
> 
> WARNING: space prohibited before semicolon
> 
> Signed-off-by: Dmitriy Cherkasov <dmitriy at oss-tech.org>

Reviewed-by: Andreas Dilger <andreas.dilger at intel.com>


> ---
> v5:
> 	- fix incomplete patch changelog
> 	- improve code alignment
> 	- reword commit message and summary to be more concise
> v4:
> 	- fix spacing issues
> 	- fix typo in commit message
> 	- add patch changelog
> v3:
> 	- fix spacing and alignment issues 
> 	  (suggested by Joe Perches)
> v2:
> 	- remove ksocknal_csum() and call crc32_le() directly
> 	  (suggested by Greg KH)
> 	- update commit message and summary to reflect these changes
> 
> drivers/staging/lustre/lnet/klnds/socklnd/socklnd.h  | 11 -----------
> .../staging/lustre/lnet/klnds/socklnd/socklnd_lib.c  | 20 ++++++++++----------
> 2 files changed, 10 insertions(+), 21 deletions(-)
> 
> diff --git a/drivers/staging/lustre/lnet/klnds/socklnd/socklnd.h b/drivers/staging/lustre/lnet/klnds/socklnd/socklnd.h
> index 5540de6..9eb169d 100644
> --- a/drivers/staging/lustre/lnet/klnds/socklnd/socklnd.h
> +++ b/drivers/staging/lustre/lnet/klnds/socklnd/socklnd.h
> @@ -519,17 +519,6 @@ struct ksock_proto {
> #define CPU_MASK_NONE   0UL
> #endif
> 
> -static inline __u32 ksocknal_csum(__u32 crc, unsigned char const *p, size_t len)
> -{
> -#if 1
> -	return crc32_le(crc, p, len);
> -#else
> -	while (len-- > 0)
> -		crc = ((crc + 0x100) & ~0xff) | ((crc + *p++) & 0xff) ;
> -	return crc;
> -#endif
> -}
> -
> static inline int
> ksocknal_route_mask(void)
> {
> diff --git a/drivers/staging/lustre/lnet/klnds/socklnd/socklnd_lib.c b/drivers/staging/lustre/lnet/klnds/socklnd/socklnd_lib.c
> index 8a036f4..9c328dc 100644
> --- a/drivers/staging/lustre/lnet/klnds/socklnd/socklnd_lib.c
> +++ b/drivers/staging/lustre/lnet/klnds/socklnd/socklnd_lib.c
> @@ -201,9 +201,9 @@
> 			if (fragnob > sum)
> 				fragnob = sum;
> 
> -			conn->ksnc_rx_csum = ksocknal_csum(conn->ksnc_rx_csum,
> -							   iov[i].iov_base,
> -							   fragnob);
> +			conn->ksnc_rx_csum = crc32_le(conn->ksnc_rx_csum,
> +						      iov[i].iov_base,
> +						      fragnob);
> 		}
> 		conn->ksnc_msg.ksm_csum = saved_csum;
> 	}
> @@ -243,8 +243,8 @@
> 			if (fragnob > sum)
> 				fragnob = sum;
> 
> -			conn->ksnc_rx_csum = ksocknal_csum(conn->ksnc_rx_csum,
> -							   base, fragnob);
> +			conn->ksnc_rx_csum = crc32_le(conn->ksnc_rx_csum,
> +						      base, fragnob);
> 
> 			kunmap(kiov[i].bv_page);
> 		}
> @@ -265,22 +265,22 @@
> 
> 	tx->tx_msg.ksm_csum = 0;
> 
> -	csum = ksocknal_csum(~0, tx->tx_iov[0].iov_base,
> -			     tx->tx_iov[0].iov_len);
> +	csum = crc32_le(~0, tx->tx_iov[0].iov_base,
> +			tx->tx_iov[0].iov_len);
> 
> 	if (tx->tx_kiov) {
> 		for (i = 0; i < tx->tx_nkiov; i++) {
> 			base = kmap(tx->tx_kiov[i].bv_page) +
> 			       tx->tx_kiov[i].bv_offset;
> 
> -			csum = ksocknal_csum(csum, base, tx->tx_kiov[i].bv_len);
> +			csum = crc32_le(csum, base, tx->tx_kiov[i].bv_len);
> 
> 			kunmap(tx->tx_kiov[i].bv_page);
> 		}
> 	} else {
> 		for (i = 1; i < tx->tx_niov; i++)
> -			csum = ksocknal_csum(csum, tx->tx_iov[i].iov_base,
> -					     tx->tx_iov[i].iov_len);
> +			csum = crc32_le(csum, tx->tx_iov[i].iov_base,
> +					tx->tx_iov[i].iov_len);
> 	}
> 
> 	if (*ksocknal_tunables.ksnd_inject_csum_error) {
> -- 
> 1.9.1
> 

Cheers, Andreas
--
Andreas Dilger
Lustre Principal Architect
Intel Corporation







From andreas.dilger at intel.com  Thu Jul 13 00:27:03 2017
From: andreas.dilger at intel.com (Dilger, Andreas)
Date: Thu, 13 Jul 2017 00:27:03 +0000
Subject: [lustre-devel] [PATCH v5] staging: lustre: lnet: remove dead
 code and useless wrapper
In-Reply-To: <1499468622-10276-1-git-send-email-dmitriy@oss-tech.org>
References: <1498801841-5799-1-git-send-email-dmitriy@oss-tech.org>
	<1499468622-10276-1-git-send-email-dmitriy@oss-tech.org>
Message-ID: <E6268FDE-5BC3-4D38-BD7B-A69E5470EBCD@intel.com>

On Jul 7, 2017, at 16:03, Dmitriy Cherkasov <dmitriy at oss-tech.org> wrote:
> 
> After removing commented out code, ksocknal_csum() becomes a useless 
> wrapper for crc32_le(). Remove it, and instead call crc32_le() directly.
> 
> Fixes the following checkpatch warning:
> 
> WARNING: space prohibited before semicolon
> 
> Signed-off-by: Dmitriy Cherkasov <dmitriy at oss-tech.org>

Reviewed-by: Andreas Dilger <andreas.dilger at intel.com>


> ---
> v5:
> 	- fix incomplete patch changelog
> 	- improve code alignment
> 	- reword commit message and summary to be more concise
> v4:
> 	- fix spacing issues
> 	- fix typo in commit message
> 	- add patch changelog
> v3:
> 	- fix spacing and alignment issues 
> 	  (suggested by Joe Perches)
> v2:
> 	- remove ksocknal_csum() and call crc32_le() directly
> 	  (suggested by Greg KH)
> 	- update commit message and summary to reflect these changes
> 
> drivers/staging/lustre/lnet/klnds/socklnd/socklnd.h  | 11 -----------
> .../staging/lustre/lnet/klnds/socklnd/socklnd_lib.c  | 20 ++++++++++----------
> 2 files changed, 10 insertions(+), 21 deletions(-)
> 
> diff --git a/drivers/staging/lustre/lnet/klnds/socklnd/socklnd.h b/drivers/staging/lustre/lnet/klnds/socklnd/socklnd.h
> index 5540de6..9eb169d 100644
> --- a/drivers/staging/lustre/lnet/klnds/socklnd/socklnd.h
> +++ b/drivers/staging/lustre/lnet/klnds/socklnd/socklnd.h
> @@ -519,17 +519,6 @@ struct ksock_proto {
> #define CPU_MASK_NONE   0UL
> #endif
> 
> -static inline __u32 ksocknal_csum(__u32 crc, unsigned char const *p, size_t len)
> -{
> -#if 1
> -	return crc32_le(crc, p, len);
> -#else
> -	while (len-- > 0)
> -		crc = ((crc + 0x100) & ~0xff) | ((crc + *p++) & 0xff) ;
> -	return crc;
> -#endif
> -}
> -
> static inline int
> ksocknal_route_mask(void)
> {
> diff --git a/drivers/staging/lustre/lnet/klnds/socklnd/socklnd_lib.c b/drivers/staging/lustre/lnet/klnds/socklnd/socklnd_lib.c
> index 8a036f4..9c328dc 100644
> --- a/drivers/staging/lustre/lnet/klnds/socklnd/socklnd_lib.c
> +++ b/drivers/staging/lustre/lnet/klnds/socklnd/socklnd_lib.c
> @@ -201,9 +201,9 @@
> 			if (fragnob > sum)
> 				fragnob = sum;
> 
> -			conn->ksnc_rx_csum = ksocknal_csum(conn->ksnc_rx_csum,
> -							   iov[i].iov_base,
> -							   fragnob);
> +			conn->ksnc_rx_csum = crc32_le(conn->ksnc_rx_csum,
> +						      iov[i].iov_base,
> +						      fragnob);
> 		}
> 		conn->ksnc_msg.ksm_csum = saved_csum;
> 	}
> @@ -243,8 +243,8 @@
> 			if (fragnob > sum)
> 				fragnob = sum;
> 
> -			conn->ksnc_rx_csum = ksocknal_csum(conn->ksnc_rx_csum,
> -							   base, fragnob);
> +			conn->ksnc_rx_csum = crc32_le(conn->ksnc_rx_csum,
> +						      base, fragnob);
> 
> 			kunmap(kiov[i].bv_page);
> 		}
> @@ -265,22 +265,22 @@
> 
> 	tx->tx_msg.ksm_csum = 0;
> 
> -	csum = ksocknal_csum(~0, tx->tx_iov[0].iov_base,
> -			     tx->tx_iov[0].iov_len);
> +	csum = crc32_le(~0, tx->tx_iov[0].iov_base,
> +			tx->tx_iov[0].iov_len);
> 
> 	if (tx->tx_kiov) {
> 		for (i = 0; i < tx->tx_nkiov; i++) {
> 			base = kmap(tx->tx_kiov[i].bv_page) +
> 			       tx->tx_kiov[i].bv_offset;
> 
> -			csum = ksocknal_csum(csum, base, tx->tx_kiov[i].bv_len);
> +			csum = crc32_le(csum, base, tx->tx_kiov[i].bv_len);
> 
> 			kunmap(tx->tx_kiov[i].bv_page);
> 		}
> 	} else {
> 		for (i = 1; i < tx->tx_niov; i++)
> -			csum = ksocknal_csum(csum, tx->tx_iov[i].iov_base,
> -					     tx->tx_iov[i].iov_len);
> +			csum = crc32_le(csum, tx->tx_iov[i].iov_base,
> +					tx->tx_iov[i].iov_len);
> 	}
> 
> 	if (*ksocknal_tunables.ksnd_inject_csum_error) {
> -- 
> 1.9.1
> 

Cheers, Andreas
--
Andreas Dilger
Lustre Principal Architect
Intel Corporation







From oleg.drokin at intel.com  Thu Jul 13 04:29:44 2017
From: oleg.drokin at intel.com (Oleg Drokin)
Date: Thu, 13 Jul 2017 00:29:44 -0400
Subject: [lustre-devel] [PATCH] stating: lustre: fix sparse error:
	incompatible types in comparison expression
In-Reply-To: <20170713021017.5368-1-rui.teng@linux.vnet.ibm.com>
References: <20170713021017.5368-1-rui.teng@linux.vnet.ibm.com>
Message-ID: <C93C23FD-C96C-4EF0-A622-AE060EB87203@intel.com>


On Jul 12, 2017, at 10:10 PM, Rui Teng wrote:

> Comparing two user space addresses to avoid sparse error:
> 
> drivers/staging//lustre/lnet/selftest/conrpc.c:490:30: error:
> incompatible types in comparison expression (different address spaces)
> 
> Signed-off-by: Rui Teng <rui.teng at linux.vnet.ibm.com>
> ---
> drivers/staging/lustre/lnet/selftest/conrpc.c | 5 ++---
> 1 file changed, 2 insertions(+), 3 deletions(-)
> 
> diff --git a/drivers/staging/lustre/lnet/selftest/conrpc.c b/drivers/staging/lustre/lnet/selftest/conrpc.c
> index da36c55b86d3..ae7c2772825e 100644
> --- a/drivers/staging/lustre/lnet/selftest/conrpc.c
> +++ b/drivers/staging/lustre/lnet/selftest/conrpc.c
> @@ -487,10 +487,9 @@ lstcon_rpc_trans_interpreter(struct lstcon_rpc_trans *trans,
> 				   sizeof(struct list_head)))
> 			return -EFAULT;
> 
> -		if (tmp.next == head_up)
> -			return 0;
> -
> 		next = tmp.next;

So the assignment is fine, but comparison is not? Strange.

I guess this is fine by me if that makes the warning go away.

Acked-by: Oleg Drokin <oleg.drokin at intel.com>

> +		if (next == head_up)
> +			return 0;
> 
> 		ent = list_entry(next, struct lstcon_rpc_ent, rpe_link);
> 
> -- 
> 2.11.0

From rui.teng at linux.vnet.ibm.com  Thu Jul 13 02:10:17 2017
From: rui.teng at linux.vnet.ibm.com (Rui Teng)
Date: Thu, 13 Jul 2017 10:10:17 +0800
Subject: [lustre-devel] [PATCH] stating: lustre: fix sparse error:
	incompatible types in comparison expression
Message-ID: <20170713021017.5368-1-rui.teng@linux.vnet.ibm.com>

Comparing two user space addresses to avoid sparse error:

drivers/staging//lustre/lnet/selftest/conrpc.c:490:30: error:
incompatible types in comparison expression (different address spaces)

Signed-off-by: Rui Teng <rui.teng at linux.vnet.ibm.com>
---
 drivers/staging/lustre/lnet/selftest/conrpc.c | 5 ++---
 1 file changed, 2 insertions(+), 3 deletions(-)

diff --git a/drivers/staging/lustre/lnet/selftest/conrpc.c b/drivers/staging/lustre/lnet/selftest/conrpc.c
index da36c55b86d3..ae7c2772825e 100644
--- a/drivers/staging/lustre/lnet/selftest/conrpc.c
+++ b/drivers/staging/lustre/lnet/selftest/conrpc.c
@@ -487,10 +487,9 @@ lstcon_rpc_trans_interpreter(struct lstcon_rpc_trans *trans,
 				   sizeof(struct list_head)))
 			return -EFAULT;
 
-		if (tmp.next == head_up)
-			return 0;
-
 		next = tmp.next;
+		if (next == head_up)
+			return 0;
 
 		ent = list_entry(next, struct lstcon_rpc_ent, rpe_link);
 
-- 
2.11.0

From jsimmons at infradead.org  Thu Jul 13 17:07:07 2017
From: jsimmons at infradead.org (James Simmons)
Date: Thu, 13 Jul 2017 18:07:07 +0100 (BST)
Subject: [lustre-devel] [PATCH] lustre: check
	copy_from_iter/copy_to_iter return code
In-Reply-To: <20170710130833.1834210-1-arnd@arndb.de>
References: <20170710130833.1834210-1-arnd@arndb.de>
Message-ID: <alpine.LFD.2.20.1707131806080.14535@casper.infradead.org>


> We now get a helpful warning for code that calls copy_{from,to}_iter
> without checking the return value, introduced by commit aa28de275a24
> ("iov_iter/hardening: move object size checks to inlined part").
> 
> drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd_cb.c: In function 'kiblnd_send':
> drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd_cb.c:1643:2: error: ignoring return value of 'copy_from_iter', declared with attribute warn_unused_result [-Werror=unused-result]
> drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd_cb.c: In function 'kiblnd_recv':
> drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd_cb.c:1744:3: error: ignoring return value of 'copy_to_iter', declared with attribute warn_unused_result [-Werror=unused-result]
> 
> In case we get short copies here, we may get incorrect behavior.
> I've added failure handling for both rx and tx now, returning
> -EFAULT as expected.
> 
> Cc: stable at vger.kernel.org
> Signed-off-by: Arnd Bergmann <arnd at arndb.de>
> ---
> This warning now shows up in 'allmodconfig' builds, so it would be
> good to get it fixed quickly for 4.13, but my patch should not go
> in without careful review since I'm not familiar with with code
> and the error handling is a bit tricky here.
> 
> I added 'Cc: stable' since this is a preexisting condition that we
> only started warning about now.
> ---
>  drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd_cb.c | 15 +++++++++++++--
>  1 file changed, 13 insertions(+), 2 deletions(-)
> 
> diff --git a/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd_cb.c b/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd_cb.c
> index 85b242ec5f9b..70256a0ffd2e 100644
> --- a/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd_cb.c
> +++ b/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd_cb.c
> @@ -1640,8 +1640,13 @@ kiblnd_send(struct lnet_ni *ni, void *private, struct lnet_msg *lntmsg)
>  	ibmsg = tx->tx_msg;
>  	ibmsg->ibm_u.immediate.ibim_hdr = *hdr;
>  
> -	copy_from_iter(&ibmsg->ibm_u.immediate.ibim_payload, IBLND_MSG_SIZE,
> +	rc = copy_from_iter(&ibmsg->ibm_u.immediate.ibim_payload, IBLND_MSG_SIZE,
>  		       &from);

I have to Nak this to prevent it from landing but for some reason in my 
testing its returning zero from copy_from_iter.


> +	if (rc != IBLND_MSG_SIZE) {
> +		kiblnd_pool_free_node(&tx->tx_pool->tpo_pool, &tx->tx_list);
> +		return -EFAULT;
> +	}
> +
>  	nob = offsetof(struct kib_immediate_msg, ibim_payload[payload_nob]);
>  	kiblnd_init_tx_msg(ni, tx, IBLND_MSG_IMMEDIATE, nob);
>  
> @@ -1741,8 +1746,14 @@ kiblnd_recv(struct lnet_ni *ni, void *private, struct lnet_msg *lntmsg,
>  			break;
>  		}
>  
> -		copy_to_iter(&rxmsg->ibm_u.immediate.ibim_payload,
> +		rc = copy_to_iter(&rxmsg->ibm_u.immediate.ibim_payload,
>  			     IBLND_MSG_SIZE, to);
> +		if (rc != IBLND_MSG_SIZE) {
> +			rc = -EFAULT;
> +			break;
> +		}
> +
> +		rc = 0;
>  		lnet_finalize(ni, lntmsg, 0);
>  		break;
>  
> -- 
> 2.9.0
> 
> 

From arnd at arndb.de  Thu Jul 13 20:57:59 2017
From: arnd at arndb.de (Arnd Bergmann)
Date: Thu, 13 Jul 2017 22:57:59 +0200
Subject: [lustre-devel] [PATCH] lustre: check
	copy_from_iter/copy_to_iter return code
In-Reply-To: <alpine.LFD.2.20.1707131806080.14535@casper.infradead.org>
References: <20170710130833.1834210-1-arnd@arndb.de>
	<alpine.LFD.2.20.1707131806080.14535@casper.infradead.org>
Message-ID: <CAK8P3a3nJRN2QQO4AGVB3V=jKafqvjdwoJ5u4ntEMsEugzJ_+Q@mail.gmail.com>

On Thu, Jul 13, 2017 at 7:07 PM, James Simmons <jsimmons at infradead.org> wrote:
>
>> We now get a helpful warning for code that calls copy_{from,to}_iter
>> without checking the return value, introduced by commit aa28de275a24
>> ("iov_iter/hardening: move object size checks to inlined part").
>>
>> drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd_cb.c: In function 'kiblnd_send':
>> drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd_cb.c:1643:2: error: ignoring return value of 'copy_from_iter', declared with attribute warn_unused_result [-Werror=unused-result]
>> drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd_cb.c: In function 'kiblnd_recv':
>> drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd_cb.c:1744:3: error: ignoring return value of 'copy_to_iter', declared with attribute warn_unused_result [-Werror=unused-result]
>>
>> In case we get short copies here, we may get incorrect behavior.
>> I've added failure handling for both rx and tx now, returning
>> -EFAULT as expected.
>>
>> Cc: stable at vger.kernel.org
>> Signed-off-by: Arnd Bergmann <arnd at arndb.de>
>> ---
>> This warning now shows up in 'allmodconfig' builds, so it would be
>> good to get it fixed quickly for 4.13, but my patch should not go
>> in without careful review since I'm not familiar with with code
>> and the error handling is a bit tricky here.
>>
>> I added 'Cc: stable' since this is a preexisting condition that we
>> only started warning about now.
>> ---
>>  drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd_cb.c | 15 +++++++++++++--
>>  1 file changed, 13 insertions(+), 2 deletions(-)
>>
>> diff --git a/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd_cb.c b/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd_cb.c
>> index 85b242ec5f9b..70256a0ffd2e 100644
>> --- a/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd_cb.c
>> +++ b/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd_cb.c
>> @@ -1640,8 +1640,13 @@ kiblnd_send(struct lnet_ni *ni, void *private, struct lnet_msg *lntmsg)
>>       ibmsg = tx->tx_msg;
>>       ibmsg->ibm_u.immediate.ibim_hdr = *hdr;
>>
>> -     copy_from_iter(&ibmsg->ibm_u.immediate.ibim_payload, IBLND_MSG_SIZE,
>> +     rc = copy_from_iter(&ibmsg->ibm_u.immediate.ibim_payload, IBLND_MSG_SIZE,
>>                      &from);
>
> I have to Nak this to prevent it from landing but for some reason in my
> testing its returning zero from copy_from_iter.

Thanks for testing it!

That means we did not copy any data and the kernel continues with
an uninitialized buffer, right? The problem may be the definition of

struct kib_immediate_msg {
        struct lnet_hdr ibim_hdr;        /* portals header */
        char         ibim_payload[0]; /* piggy-backed payload */
} WIRE_ATTR;

The check that Al added will try to ensure that we don't write
beyond the size of the ibim_payload[] array, which unfortunately
is defined as a zero-byte array, so I can see why it will now
fail. However, it's already broken in mainline now, with or without
my patch.

Are you able to come up with a fix that avoids the warning in
'allmodconfig' and makes the function do something reasonable
again?

        Arnd

From peter.a.jones at intel.com  Thu Jul 13 21:25:42 2017
From: peter.a.jones at intel.com (Jones, Peter A)
Date: Thu, 13 Jul 2017 21:25:42 +0000
Subject: [lustre-devel] Lustre 2.10.0 released
Message-ID: <D17F6C2E-90D0-4C61-94AE-79C13DB08F75@intel.com>

We are pleased to announce that the Lustre 2.10.0 Release has been declared GA and is available for <https://downloads.hpdd.intel.com/public/lustre/latest-feature-release/> download<https://downloads.hpdd.intel.com/public/lustre/lustre-2.10.0/> <https://downloads.hpdd.intel.com/public/lustre/lustre-2.10.0/> . You can also grab the source from <http://git.whamcloud.com/fs/lustre-release.git/commit/ea79df5af4d9b034e39caa396cefc2a075e572ba> git<https://git.hpdd.intel.com/?p=fs/lustre-release.git;a=commit;h=58fd06e4d9fcee95f5c60bc8120d782b61f0823c>

This major release includes new features:

Progressive File Layouts  Enables file layouts to automatically adjust as size of files grow thus optimizing performance for diverse workloads    (LU-8898<https://jira.hpdd.intel.com/browse/LU-8998>)

Multi-Rail LNet Allows LNet to utilize multiple network interfaces on a node in parallel, aggregating their performance (LU-7734<https://jira.hpdd.intel.com/browse/LU-7734>)

Project Quotas  Extension to the Lustre quotas feature to provide the option to place quotas on a per project basis rather than just per user or per group (LU-4017<https://jira.hpdd.intel.com/browse/LU-4017>)

Simplified Userspace Snapshots Provides a mechanism to leverage the snapshot capability in OpenZFS to take a coordinated snapshot of a Lustre filesystem. (LU-8900<https://jira.hpdd.intel.com/browse/LU-8900>)

NRS Delay Policy Simulates high server load as a way of validating the resilience of Lustre under load . (LU-6283<https://jira.hpdd.intel.com/browse/LU-6283>)

Fuller details can be found in the 2.10 wiki page<http://wiki.lustre.org/Release_2.10.0> (including the ch<http://wiki.lustre.org/Lustre_2.10.0_Changelog>ange log<http://wiki.lustre.org/Lustre_2.10.0_Changelog> and test mat<http://wiki.lustre.org/Release_2.10.0#Test_Matrix>rix<http://wiki.lustre.org/Release_2.10.0#Test_Matrix>)

Please log any issues found in the issue tracking system<https://jira.hpdd.intel.com/>

Thanks to all those who have contributed to the creation of this release.

This is the first release of the 2.10.x LTS release stream. We are planning to release a freely available Lustre 2.10.1 in the coming weeks

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-devel_lists.lustre.org/attachments/20170713/6eb64259/attachment.html>

From viro at ZenIV.linux.org.uk  Fri Jul 14 01:50:34 2017
From: viro at ZenIV.linux.org.uk (Al Viro)
Date: Fri, 14 Jul 2017 02:50:34 +0100
Subject: [lustre-devel] [PATCH] lustre: check
	copy_from_iter/copy_to_iter return code
In-Reply-To: <CAK8P3a3nJRN2QQO4AGVB3V=jKafqvjdwoJ5u4ntEMsEugzJ_+Q@mail.gmail.com>
References: <20170710130833.1834210-1-arnd@arndb.de>
	<alpine.LFD.2.20.1707131806080.14535@casper.infradead.org>
	<CAK8P3a3nJRN2QQO4AGVB3V=jKafqvjdwoJ5u4ntEMsEugzJ_+Q@mail.gmail.com>
Message-ID: <20170714015034.GE13102@ZenIV.linux.org.uk>

On Thu, Jul 13, 2017 at 10:57:59PM +0200, Arnd Bergmann wrote:

> Thanks for testing it!
> 
> That means we did not copy any data and the kernel continues with
> an uninitialized buffer, right? The problem may be the definition of
> 
> struct kib_immediate_msg {
>         struct lnet_hdr ibim_hdr;        /* portals header */
>         char         ibim_payload[0]; /* piggy-backed payload */
> } WIRE_ATTR;
> 
> The check that Al added will try to ensure that we don't write
> beyond the size of the ibim_payload[] array, which unfortunately
> is defined as a zero-byte array, so I can see why it will now
> fail. However, it's already broken in mainline now, with or without
> my patch.
> 
> Are you able to come up with a fix that avoids the warning in
> 'allmodconfig' and makes the function do something reasonable
> again?

Might make sense to try and use valid C99 for "array of indefinite
size as the last member", i.e.
struct kib_immediate_msg {
         struct lnet_hdr ibim_hdr;        /* portals header */
         char         ibim_payload[]; /* piggy-backed payload */
} WIRE_ATTR;

	Zero-sized array as the last member is gcc hack predating that;
looks like gcc gets confused into deciding that it knows the distance
from the end of object...

	Said that, are we really guaranteed the IBLND_MSG_SIZE bytes
in there?

From colin.king at canonical.com  Fri Jul 14 13:26:02 2017
From: colin.king at canonical.com (Colin King)
Date: Fri, 14 Jul 2017 14:26:02 +0100
Subject: [lustre-devel] [PATCH] staging: lustre: fix spelling mistake,
	"grranted" -> "granted"
Message-ID: <20170714132602.27713-1-colin.king@canonical.com>

From: Colin Ian King <colin.king at canonical.com>

Trivial fix to spelling mistake in CERROR error message

Signed-off-by: Colin Ian King <colin.king at canonical.com>
---
 drivers/staging/lustre/lustre/ptlrpc/import.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/staging/lustre/lustre/ptlrpc/import.c b/drivers/staging/lustre/lustre/ptlrpc/import.c
index 52cb1f0c9c94..99877aa10d6f 100644
--- a/drivers/staging/lustre/lustre/ptlrpc/import.c
+++ b/drivers/staging/lustre/lustre/ptlrpc/import.c
@@ -1026,7 +1026,7 @@ static int ptlrpc_connect_interpret(const struct lu_env *env,
 	/* check that server granted subset of flags we asked for. */
 	if ((ocd->ocd_connect_flags & imp->imp_connect_flags_orig) !=
 	    ocd->ocd_connect_flags) {
-		CERROR("%s: Server didn't granted asked subset of flags: asked=%#llx grranted=%#llx\n",
+		CERROR("%s: Server didn't granted asked subset of flags: asked=%#llx granted=%#llx\n",
 		       imp->imp_obd->obd_name, imp->imp_connect_flags_orig,
 		       ocd->ocd_connect_flags);
 		rc = -EPROTO;
-- 
2.11.0

From oleg.drokin at intel.com  Fri Jul 14 20:43:10 2017
From: oleg.drokin at intel.com (Oleg Drokin)
Date: Fri, 14 Jul 2017 16:43:10 -0400
Subject: [lustre-devel] [PATCH] staging: lustre: fix spelling mistake,
	"grranted" -> "granted"
In-Reply-To: <20170714132602.27713-1-colin.king@canonical.com>
References: <20170714132602.27713-1-colin.king@canonical.com>
Message-ID: <6FB4ABAB-3CB3-470C-BFD1-A8DC02862D3F@intel.com>


On Jul 14, 2017, at 9:26 AM, Colin King wrote:

> From: Colin Ian King <colin.king at canonical.com>
> 
> Trivial fix to spelling mistake in CERROR error message
> 
> Signed-off-by: Colin Ian King <colin.king at canonical.com>
> ---
> drivers/staging/lustre/lustre/ptlrpc/import.c | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/drivers/staging/lustre/lustre/ptlrpc/import.c b/drivers/staging/lustre/lustre/ptlrpc/import.c
> index 52cb1f0c9c94..99877aa10d6f 100644
> --- a/drivers/staging/lustre/lustre/ptlrpc/import.c
> +++ b/drivers/staging/lustre/lustre/ptlrpc/import.c
> @@ -1026,7 +1026,7 @@ static int ptlrpc_connect_interpret(const struct lu_env *env,
> 	/* check that server granted subset of flags we asked for. */
> 	if ((ocd->ocd_connect_flags & imp->imp_connect_flags_orig) !=
> 	    ocd->ocd_connect_flags) {
> -		CERROR("%s: Server didn't granted asked subset of flags: asked=%#llx grranted=%#llx\n",
> +		CERROR("%s: Server didn't granted asked subset of flags: asked=%#llx granted=%#llx\n",

While we are at it can also address grammar problem: "didn't granted" -> "didn't grant"?

Thanks!

> 		       imp->imp_obd->obd_name, imp->imp_connect_flags_orig,
> 		       ocd->ocd_connect_flags);
> 		rc = -EPROTO;
> -- 
> 2.11.0

From colin.king at canonical.com  Fri Jul 14 21:33:54 2017
From: colin.king at canonical.com (Colin King)
Date: Fri, 14 Jul 2017 22:33:54 +0100
Subject: [lustre-devel] [PATCH][V2] staging: lustre: fix spelling mistake,
	"grranted" -> "granted"
Message-ID: <20170714213354.25101-1-colin.king@canonical.com>

From: Colin Ian King <colin.king at canonical.com>

Trivial fix to spelling mistake in CERROR error message. Also
clean up the grammar.

Signed-off-by: Colin Ian King <colin.king at canonical.com>
---
 drivers/staging/lustre/lustre/ptlrpc/import.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/staging/lustre/lustre/ptlrpc/import.c b/drivers/staging/lustre/lustre/ptlrpc/import.c
index 52cb1f0c9c94..b19dac15e901 100644
--- a/drivers/staging/lustre/lustre/ptlrpc/import.c
+++ b/drivers/staging/lustre/lustre/ptlrpc/import.c
@@ -1026,7 +1026,7 @@ static int ptlrpc_connect_interpret(const struct lu_env *env,
 	/* check that server granted subset of flags we asked for. */
 	if ((ocd->ocd_connect_flags & imp->imp_connect_flags_orig) !=
 	    ocd->ocd_connect_flags) {
-		CERROR("%s: Server didn't granted asked subset of flags: asked=%#llx grranted=%#llx\n",
+		CERROR("%s: Server didn't grant the asked for subset of flags: asked=%#llx granted=%#llx\n",
 		       imp->imp_obd->obd_name, imp->imp_connect_flags_orig,
 		       ocd->ocd_connect_flags);
 		rc = -EPROTO;
-- 
2.11.0

From oleg.drokin at intel.com  Fri Jul 14 21:46:29 2017
From: oleg.drokin at intel.com (Oleg Drokin)
Date: Fri, 14 Jul 2017 17:46:29 -0400
Subject: [lustre-devel] [PATCH][V2] staging: lustre: fix spelling
	mistake, "grranted" -> "granted"
In-Reply-To: <20170714213354.25101-1-colin.king@canonical.com>
References: <20170714213354.25101-1-colin.king@canonical.com>
Message-ID: <D03F15A6-04AE-46C3-9564-55CA055C08DF@intel.com>


On Jul 14, 2017, at 5:33 PM, Colin King wrote:

> From: Colin Ian King <colin.king at canonical.com>
> 
> Trivial fix to spelling mistake in CERROR error message. Also
> clean up the grammar.
> 
> Signed-off-by: Colin Ian King <colin.king at canonical.com>

Reviewed-by: Oleg Drokin <oleg.drokin at intel.com>

> ---
> drivers/staging/lustre/lustre/ptlrpc/import.c | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/drivers/staging/lustre/lustre/ptlrpc/import.c b/drivers/staging/lustre/lustre/ptlrpc/import.c
> index 52cb1f0c9c94..b19dac15e901 100644
> --- a/drivers/staging/lustre/lustre/ptlrpc/import.c
> +++ b/drivers/staging/lustre/lustre/ptlrpc/import.c
> @@ -1026,7 +1026,7 @@ static int ptlrpc_connect_interpret(const struct lu_env *env,
> 	/* check that server granted subset of flags we asked for. */
> 	if ((ocd->ocd_connect_flags & imp->imp_connect_flags_orig) !=
> 	    ocd->ocd_connect_flags) {
> -		CERROR("%s: Server didn't granted asked subset of flags: asked=%#llx grranted=%#llx\n",
> +		CERROR("%s: Server didn't grant the asked for subset of flags: asked=%#llx granted=%#llx\n",
> 		       imp->imp_obd->obd_name, imp->imp_connect_flags_orig,
> 		       ocd->ocd_connect_flags);
> 		rc = -EPROTO;
> -- 
> 2.11.0

From jsimmons at infradead.org  Sat Jul 15 14:40:29 2017
From: jsimmons at infradead.org (James Simmons)
Date: Sat, 15 Jul 2017 15:40:29 +0100 (BST)
Subject: [lustre-devel] [PATCH] lustre: check
	copy_from_iter/copy_to_iter return code
In-Reply-To: <20170714015034.GE13102@ZenIV.linux.org.uk>
References: <20170710130833.1834210-1-arnd@arndb.de>
	<alpine.LFD.2.20.1707131806080.14535@casper.infradead.org>
	<CAK8P3a3nJRN2QQO4AGVB3V=jKafqvjdwoJ5u4ntEMsEugzJ_+Q@mail.gmail.com>
	<20170714015034.GE13102@ZenIV.linux.org.uk>
Message-ID: <alpine.LFD.2.20.1707151520150.24574@casper.infradead.org>



On Fri, 14 Jul 2017, Al Viro wrote:

> On Thu, Jul 13, 2017 at 10:57:59PM +0200, Arnd Bergmann wrote:
> 
> > Thanks for testing it!
> > 
> > That means we did not copy any data and the kernel continues with
> > an uninitialized buffer, right? The problem may be the definition of
> > 
> > struct kib_immediate_msg {
> >         struct lnet_hdr ibim_hdr;        /* portals header */
> >         char         ibim_payload[0]; /* piggy-backed payload */
> > } WIRE_ATTR;
> > 
> > The check that Al added will try to ensure that we don't write
> > beyond the size of the ibim_payload[] array, which unfortunately
> > is defined as a zero-byte array, so I can see why it will now
> > fail. However, it's already broken in mainline now, with or without
> > my patch.
> > 
> > Are you able to come up with a fix that avoids the warning in
> > 'allmodconfig' and makes the function do something reasonable
> > again?

Yes, I'm testing a fix right now which I will merge with the original
patch. Greg this patch will need to be sent to Linus as well so the
kernel release isn't broken for users.
 
> Might make sense to try and use valid C99 for "array of indefinite
> size as the last member", i.e.
> struct kib_immediate_msg {
>          struct lnet_hdr ibim_hdr;        /* portals header */
>          char         ibim_payload[]; /* piggy-backed payload */
> } WIRE_ATTR;
> 
> 	Zero-sized array as the last member is gcc hack predating that;
> looks like gcc gets confused into deciding that it knows the distance
> from the end of object...

I did some profiling and found gcc was doing the right thing. That
should be updated to a C99 flexable array in a latter patch. 

> 	Said that, are we really guaranteed the IBLND_MSG_SIZE bytes
> in there?

This is what the real bug was. In the current code we are telling
copy_from_iter and copy_to_iter that the number of bytes are always
IBLND_MSG_SIZE. Arnd thought this was always the size so in his
patch he was testing the returned result of copy_[from|to]_iter to 
IBLND_MSG_SIZE. This nearly always failed since variable sized messages 
are being created. The zero size I initially saw was from doing pings. 
When I later tested with pushing I/O packets of other sizes were
observed but none of them were IBLND_MSG_SIZE in size so they failed to 
transmit. As soon as I'm done testing I will send a patch.

From jsimmons at infradead.org  Sat Jul 15 15:32:08 2017
From: jsimmons at infradead.org (James Simmons)
Date: Sat, 15 Jul 2017 11:32:08 -0400
Subject: [lustre-devel] [PATCH] staging: lustre: ko2iblnd: check
	copy_from_iter/copy_to_iter return code
Message-ID: <1500132728-28511-1-git-send-email-jsimmons@infradead.org>

From: Arnd Bergmann <arnd at arndb.de>

We now get a helpful warning for code that calls copy_{from,to}_iter
without checking the return value, introduced by commit aa28de275a24
("iov_iter/hardening: move object size checks to inlined part").

drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd_cb.c: In function 'kiblnd_send':
drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd_cb.c:1643:2: error: ignoring return value of 'copy_from_iter', declared with attribute warn_unused_result [-Werror=unused-result]
drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd_cb.c: In function 'kiblnd_recv':
drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd_cb.c:1744:3: error: ignoring return value of 'copy_to_iter', declared with attribute warn_unused_result [-Werror=unused-result]

In case we get short copies here, we may get incorrect behavior.
I've added failure handling for both rx and tx now, returning
-EFAULT as expected.

Cc: stable at vger.kernel.org
Signed-off-by: Arnd Bergmann <arnd at arndb.de>
Signed-off-by: James Simmons <jsimmons at infradead.org>
---
 .../staging/lustre/lnet/klnds/o2iblnd/o2iblnd_cb.c    | 19 +++++++++++++++----
 1 file changed, 15 insertions(+), 4 deletions(-)

diff --git a/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd_cb.c b/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd_cb.c
index 85b242e..8fc191d 100644
--- a/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd_cb.c
+++ b/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd_cb.c
@@ -1640,8 +1640,13 @@ static int kiblnd_resolve_addr(struct rdma_cm_id *cmid,
 	ibmsg = tx->tx_msg;
 	ibmsg->ibm_u.immediate.ibim_hdr = *hdr;
 
-	copy_from_iter(&ibmsg->ibm_u.immediate.ibim_payload, IBLND_MSG_SIZE,
-		       &from);
+	rc = copy_from_iter(&ibmsg->ibm_u.immediate.ibim_payload, payload_nob,
+			    &from);
+	if (rc != payload_nob) {
+		kiblnd_pool_free_node(&tx->tx_pool->tpo_pool, &tx->tx_list);
+		return -EFAULT;
+	}
+
 	nob = offsetof(struct kib_immediate_msg, ibim_payload[payload_nob]);
 	kiblnd_init_tx_msg(ni, tx, IBLND_MSG_IMMEDIATE, nob);
 
@@ -1741,8 +1746,14 @@ static int kiblnd_resolve_addr(struct rdma_cm_id *cmid,
 			break;
 		}
 
-		copy_to_iter(&rxmsg->ibm_u.immediate.ibim_payload,
-			     IBLND_MSG_SIZE, to);
+		rc = copy_to_iter(&rxmsg->ibm_u.immediate.ibim_payload, rlen,
+				  to);
+		if (rc != rlen) {
+			rc = -EFAULT;
+			break;
+		}
+
+		rc = 0;
 		lnet_finalize(ni, lntmsg, 0);
 		break;
 
-- 
1.8.3.1

From jsimmons at infradead.org  Sat Jul 15 15:39:22 2017
From: jsimmons at infradead.org (James Simmons)
Date: Sat, 15 Jul 2017 11:39:22 -0400
Subject: [lustre-devel] [PATCH] staging: lustre: lustre: fix all braces
	issues reported by checkpatch
Message-ID: <1500133162-29310-1-git-send-email-jsimmons@infradead.org>

Cleanup all braces that was reported by checkpatch. The only
issue not fixed up is in mdc_lock.c. Removing the braces in
the case of mdc_lock.c will break the build.

Signed-off-by: James Simmons <jsimmons at infradead.org>
------------------------------------------------------------------
v1) Initial patch
v2) Rebased against latest staging-next tree
---
 drivers/staging/lustre/lustre/fld/fld_cache.c  | 3 ++-
 drivers/staging/lustre/lustre/ldlm/ldlm_lock.c | 9 +++++----
 drivers/staging/lustre/lustre/llite/vvp_dev.c  | 5 +++--
 3 files changed, 10 insertions(+), 7 deletions(-)

diff --git a/drivers/staging/lustre/lustre/fld/fld_cache.c b/drivers/staging/lustre/lustre/fld/fld_cache.c
index b852fed..adaa094 100644
--- a/drivers/staging/lustre/lustre/fld/fld_cache.c
+++ b/drivers/staging/lustre/lustre/fld/fld_cache.c
@@ -348,9 +348,10 @@ static void fld_cache_overlap_handle(struct fld_cache *cache,
 
 		f_curr->fce_range.lsr_end = new_start;
 		fld_cache_entry_add(cache, f_new, &f_curr->fce_list);
-	} else
+	} else {
 		CERROR("NEW range =" DRANGE " curr = " DRANGE "\n",
 		       PRANGE(range), PRANGE(&f_curr->fce_range));
+	}
 }
 
 struct fld_cache_entry
diff --git a/drivers/staging/lustre/lustre/ldlm/ldlm_lock.c b/drivers/staging/lustre/lustre/ldlm/ldlm_lock.c
index ddb4642..181025d 100644
--- a/drivers/staging/lustre/lustre/ldlm/ldlm_lock.c
+++ b/drivers/staging/lustre/lustre/ldlm/ldlm_lock.c
@@ -1029,11 +1029,11 @@ void ldlm_grant_lock(struct ldlm_lock *lock, struct list_head *work_list)
 	if (work_list && lock->l_completion_ast)
 		ldlm_add_ast_work_item(lock, NULL, work_list);
 
-	if (res->lr_type == LDLM_PLAIN || res->lr_type == LDLM_IBITS)
+	if (res->lr_type == LDLM_PLAIN || res->lr_type == LDLM_IBITS) {
 		ldlm_grant_lock_with_skiplist(lock);
-	else if (res->lr_type == LDLM_EXTENT)
+	} else if (res->lr_type == LDLM_EXTENT) {
 		ldlm_extent_add_lock(res, lock);
-	else if (res->lr_type == LDLM_FLOCK) {
+	} else if (res->lr_type == LDLM_FLOCK) {
 		/*
 		 * We should not add locks to granted list in the following cases:
 		 * - this is an UNLOCK but not a real lock;
@@ -1045,8 +1045,9 @@ void ldlm_grant_lock(struct ldlm_lock *lock, struct list_head *work_list)
 		    ldlm_is_test_lock(lock) || ldlm_is_flock_deadlock(lock))
 			return;
 		ldlm_resource_add_lock(res, &res->lr_granted, lock);
-	} else
+	} else {
 		LBUG();
+	}
 
 	ldlm_pool_add(&ldlm_res_to_ns(res)->ns_pool, lock);
 }
diff --git a/drivers/staging/lustre/lustre/llite/vvp_dev.c b/drivers/staging/lustre/lustre/llite/vvp_dev.c
index 8e45672..2b60699 100644
--- a/drivers/staging/lustre/lustre/llite/vvp_dev.c
+++ b/drivers/staging/lustre/lustre/llite/vvp_dev.c
@@ -591,9 +591,10 @@ static void *vvp_pgcache_start(struct seq_file *f, loff_t *pos)
 	env = cl_env_get(&refcheck);
 	if (!IS_ERR(env)) {
 		sbi = f->private;
-		if (sbi->ll_site->ls_obj_hash->hs_cur_bits > 64 - PGC_OBJ_SHIFT)
+		if (sbi->ll_site->ls_obj_hash->hs_cur_bits >
+		    64 - PGC_OBJ_SHIFT) {
 			pos = ERR_PTR(-EFBIG);
-		else {
+		} else {
 			*pos = vvp_pgcache_find(env, &sbi->ll_cl->cd_lu_dev,
 						*pos);
 			if (*pos == ~0ULL)
-- 
1.8.3.1

From joe at perches.com  Sat Jul 15 15:52:54 2017
From: joe at perches.com (Joe Perches)
Date: Sat, 15 Jul 2017 08:52:54 -0700
Subject: [lustre-devel] [PATCH] staging: lustre: lustre: fix all braces
 issues reported by checkpatch
In-Reply-To: <1500133162-29310-1-git-send-email-jsimmons@infradead.org>
References: <1500133162-29310-1-git-send-email-jsimmons@infradead.org>
Message-ID: <1500133974.4457.79.camel@perches.com>

On Sat, 2017-07-15 at 11:39 -0400, James Simmons wrote:
> Cleanup all braces that was reported by checkpatch. The only
> issue not fixed up is in mdc_lock.c. Removing the braces in
> the case of mdc_lock.c will break the build.

what checkpatch warning in mdc_locks.c is that?

$ ./scripts/checkpatch.pl -f --terse --nosummary drivers/staging/lustre/lustre/mdc/mdc_locks.c 
drivers/staging/lustre/lustre/mdc/mdc_locks.c:590: WARNING: line over 80 characters
drivers/staging/lustre/lustre/mdc/mdc_locks.c:600: WARNING: line over 80 characters
drivers/staging/lustre/lustre/mdc/mdc_locks.c:637: WARNING: line over 80 characters

From jsimmons at infradead.org  Sat Jul 15 23:00:31 2017
From: jsimmons at infradead.org (James Simmons)
Date: Sun, 16 Jul 2017 00:00:31 +0100 (BST)
Subject: [lustre-devel] [PATCH] staging: lustre: lustre: fix all braces
 issues reported by checkpatch
In-Reply-To: <1500133974.4457.79.camel@perches.com>
References: <1500133162-29310-1-git-send-email-jsimmons@infradead.org>
	<1500133974.4457.79.camel@perches.com>
Message-ID: <alpine.LFD.2.20.1707152359420.15994@casper.infradead.org>


> On Sat, 2017-07-15 at 11:39 -0400, James Simmons wrote:
> > Cleanup all braces that was reported by checkpatch. The only
> > issue not fixed up is in mdc_lock.c. Removing the braces in
> > the case of mdc_lock.c will break the build.
> 
> what checkpatch warning in mdc_locks.c is that?
> 
> $ ./scripts/checkpatch.pl -f --terse --nosummary drivers/staging/lustre/lustre/mdc/mdc_locks.c 
> drivers/staging/lustre/lustre/mdc/mdc_locks.c:590: WARNING: line over 80 characters
> drivers/staging/lustre/lustre/mdc/mdc_locks.c:600: WARNING: line over 80 characters
> drivers/staging/lustre/lustre/mdc/mdc_locks.c:637: WARNING: line over 80 characters

For 4.11-xxx kernels I was seeing

WARNING: braces {} are not necessary for any arm of this statement
#914: FILE: drivers/staging/lustre/lustre/mdc/mdc_locks.c:914:
+       if (it->it_op & IT_CREAT) {
[...]
+       } else if (it->it_op == IT_OPEN) {
[...]
+       } else {
[...]

Now it doesn't show up.

From lkp at intel.com  Mon Jul 17 07:15:35 2017
From: lkp at intel.com (kbuild test robot)
Date: Mon, 17 Jul 2017 15:15:35 +0800
Subject: [lustre-devel] [PATCH 1/4] staging: lustre: constify
	attribute_group structures.
In-Reply-To: <1500270540-5928-2-git-send-email-arvind.yadav.cs@gmail.com>
Message-ID: <201707171554.Ee9bmEKo%fengguang.wu@intel.com>

Hi Arvind,

[auto build test ERROR on staging/staging-testing]
[also build test ERROR on v4.13-rc1 next-20170717]
[if your patch is applied to the wrong git tree, please drop us a note to help improve the system]

url:    https://github.com/0day-ci/linux/commits/Arvind-Yadav/constify-lustre-attribute_group-structures/20170717-142957
config: sparc64-allyesconfig (attached as .config)
compiler: sparc64-linux-gnu-gcc (Debian 6.1.1-9) 6.1.1 20160705
reproduce:
        wget https://raw.githubusercontent.com/01org/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross
        chmod +x ~/bin/make.cross
        # save the attached .config to linux build tree
        make.cross ARCH=sparc64 

Note: the linux-review/Arvind-Yadav/constify-lustre-attribute_group-structures/20170717-142957 HEAD a6df15f5fc00a12dffbac94e0ccc52155118946e builds fine.
      It only hurts bisectibility.

All error/warnings (new ones prefixed by >>):

>> drivers/staging/lustre/lustre/obdclass/lprocfs_status.c:1033:5: error: conflicting types for 'lprocfs_obd_setup'
    int lprocfs_obd_setup(struct obd_device *obd, struct lprocfs_vars *list,
        ^~~~~~~~~~~~~~~~~
   In file included from drivers/staging/lustre/lustre/obdclass/../include/obd_support.h:41:0,
                    from drivers/staging/lustre/lustre/obdclass/../include/obd_class.h:35,
                    from drivers/staging/lustre/lustre/obdclass/lprocfs_status.c:39:
   drivers/staging/lustre/lustre/obdclass/../include/lprocfs_status.h:470:5: note: previous declaration of 'lprocfs_obd_setup' was here
    int lprocfs_obd_setup(struct obd_device *obd, struct lprocfs_vars *list,
        ^~~~~~~~~~~~~~~~~
   In file included from include/linux/linkage.h:6:0,
                    from include/linux/kernel.h:6,
                    from include/asm-generic/bug.h:15,
                    from arch/sparc/include/asm/bug.h:20,
                    from include/linux/bug.h:4,
                    from include/linux/mmdebug.h:4,
                    from include/linux/gfp.h:4,
                    from include/linux/slab.h:14,
                    from drivers/staging/lustre/lustre/obdclass/../include/obd_support.h:36,
                    from drivers/staging/lustre/lustre/obdclass/../include/obd_class.h:35,
                    from drivers/staging/lustre/lustre/obdclass/lprocfs_status.c:39:
   drivers/staging/lustre/lustre/obdclass/lprocfs_status.c:1066:19: error: conflicting types for 'lprocfs_obd_setup'
    EXPORT_SYMBOL_GPL(lprocfs_obd_setup);
                      ^
   include/linux/export.h:65:21: note: in definition of macro '___EXPORT_SYMBOL'
     extern typeof(sym) sym;      \
                        ^~~
>> drivers/staging/lustre/lustre/obdclass/lprocfs_status.c:1066:1: note: in expansion of macro 'EXPORT_SYMBOL_GPL'
    EXPORT_SYMBOL_GPL(lprocfs_obd_setup);
    ^~~~~~~~~~~~~~~~~
   In file included from drivers/staging/lustre/lustre/obdclass/../include/obd_support.h:41:0,
                    from drivers/staging/lustre/lustre/obdclass/../include/obd_class.h:35,
                    from drivers/staging/lustre/lustre/obdclass/lprocfs_status.c:39:
   drivers/staging/lustre/lustre/obdclass/../include/lprocfs_status.h:470:5: note: previous declaration of 'lprocfs_obd_setup' was here
    int lprocfs_obd_setup(struct obd_device *obd, struct lprocfs_vars *list,
        ^~~~~~~~~~~~~~~~~

vim +/lprocfs_obd_setup +1033 drivers/staging/lustre/lustre/obdclass/lprocfs_status.c

9b8013023 Oleg Drokin   2015-05-21  1032  
9b8013023 Oleg Drokin   2015-05-21 @1033  int lprocfs_obd_setup(struct obd_device *obd, struct lprocfs_vars *list,
9b8013023 Oleg Drokin   2015-05-21  1034  		      struct attribute_group *attrs)
d7e09d039 Peng Tao      2013-05-02  1035  {
d7e09d039 Peng Tao      2013-05-02  1036  	int rc = 0;
d7e09d039 Peng Tao      2013-05-02  1037  
9b8013023 Oleg Drokin   2015-05-21  1038  	init_completion(&obd->obd_kobj_unregister);
9b8013023 Oleg Drokin   2015-05-21  1039  	rc = kobject_init_and_add(&obd->obd_kobj, &obd_ktype,
9b8013023 Oleg Drokin   2015-05-21  1040  				  obd->obd_type->typ_kobj,
9b8013023 Oleg Drokin   2015-05-21  1041  				  "%s", obd->obd_name);
9b8013023 Oleg Drokin   2015-05-21  1042  	if (rc)
9b8013023 Oleg Drokin   2015-05-21  1043  		return rc;
9b8013023 Oleg Drokin   2015-05-21  1044  
9b8013023 Oleg Drokin   2015-05-21  1045  	if (attrs) {
9b8013023 Oleg Drokin   2015-05-21  1046  		rc = sysfs_create_group(&obd->obd_kobj, attrs);
9b8013023 Oleg Drokin   2015-05-21  1047  		if (rc) {
9b8013023 Oleg Drokin   2015-05-21  1048  			kobject_put(&obd->obd_kobj);
9b8013023 Oleg Drokin   2015-05-21  1049  			return rc;
9b8013023 Oleg Drokin   2015-05-21  1050  		}
9b8013023 Oleg Drokin   2015-05-21  1051  	}
9b8013023 Oleg Drokin   2015-05-21  1052  
61e87ab0f Dmitry Eremin 2015-05-21  1053  	obd->obd_debugfs_entry = ldebugfs_register(obd->obd_name,
61e87ab0f Dmitry Eremin 2015-05-21  1054  						   obd->obd_type->typ_debugfs_entry,
d7e09d039 Peng Tao      2013-05-02  1055  						   list, obd);
61e87ab0f Dmitry Eremin 2015-05-21  1056  	if (IS_ERR_OR_NULL(obd->obd_debugfs_entry)) {
61e87ab0f Dmitry Eremin 2015-05-21  1057  		rc = obd->obd_debugfs_entry ? PTR_ERR(obd->obd_debugfs_entry)
61e87ab0f Dmitry Eremin 2015-05-21  1058  					    : -ENOMEM;
1d8cb70c7 Greg Donald   2014-08-25  1059  		CERROR("error %d setting up lprocfs for %s\n",
1d8cb70c7 Greg Donald   2014-08-25  1060  		       rc, obd->obd_name);
61e87ab0f Dmitry Eremin 2015-05-21  1061  		obd->obd_debugfs_entry = NULL;
d7e09d039 Peng Tao      2013-05-02  1062  	}
9b8013023 Oleg Drokin   2015-05-21  1063  
d7e09d039 Peng Tao      2013-05-02  1064  	return rc;
d7e09d039 Peng Tao      2013-05-02  1065  }
da33f1dd9 Oleg Drokin   2016-02-06 @1066  EXPORT_SYMBOL_GPL(lprocfs_obd_setup);
d7e09d039 Peng Tao      2013-05-02  1067  

:::::: The code at line 1033 was first introduced by commit
:::::: 9b8013023cb62360b56c04313687e93a1c2bf3d6 staging/lustre/obdclass: Prepare for procfs to sysfs migration

:::::: TO: Oleg Drokin <green at linuxhacker.ru>
:::::: CC: Greg Kroah-Hartman <gregkh at linuxfoundation.org>

---
0-DAY kernel test infrastructure                Open Source Technology Center
https://lists.01.org/pipermail/kbuild-all                   Intel Corporation
-------------- next part --------------
A non-text attachment was scrubbed...
Name: .config.gz
Type: application/gzip
Size: 51477 bytes
Desc: not available
URL: <http://lists.lustre.org/pipermail/lustre-devel_lists.lustre.org/attachments/20170717/00fbe1e3/attachment.gz>

From gregkh at linuxfoundation.org  Mon Jul 17 12:27:29 2017
From: gregkh at linuxfoundation.org (Greg Kroah-Hartman)
Date: Mon, 17 Jul 2017 14:27:29 +0200
Subject: [lustre-devel] [PATCH] staging: lustre: lustre: fix all braces
 issues reported by checkpatch
In-Reply-To: <1500133162-29310-1-git-send-email-jsimmons@infradead.org>
References: <1500133162-29310-1-git-send-email-jsimmons@infradead.org>
Message-ID: <20170717122729.GC24503@kroah.com>

On Sat, Jul 15, 2017 at 11:39:22AM -0400, James Simmons wrote:
> Cleanup all braces that was reported by checkpatch. The only
> issue not fixed up is in mdc_lock.c. Removing the braces in
> the case of mdc_lock.c will break the build.
> 
> Signed-off-by: James Simmons <jsimmons at infradead.org>
> ------------------------------------------------------------------
> v1) Initial patch
> v2) Rebased against latest staging-next tree

Those lines go below:

> ---

That line.

I'll edit it by hand this time...

From arvind.yadav.cs at gmail.com  Mon Jul 17 05:48:59 2017
From: arvind.yadav.cs at gmail.com (Arvind Yadav)
Date: Mon, 17 Jul 2017 11:18:59 +0530
Subject: [lustre-devel] [PATCH 3/4] staging: lustre: obdclass: linux:
	constify attribute_group structures.
In-Reply-To: <1500270540-5928-1-git-send-email-arvind.yadav.cs@gmail.com>
References: <1500270540-5928-1-git-send-email-arvind.yadav.cs@gmail.com>
Message-ID: <1500270540-5928-4-git-send-email-arvind.yadav.cs@gmail.com>

attribute_groups are not supposed to change at runtime. All functions
working with attribute_groups provided by <linux/sysfs.h> work
with const attribute_group. So mark the non-const structs as const.

Signed-off-by: Arvind Yadav <arvind.yadav.cs at gmail.com>
---
 drivers/staging/lustre/lustre/obdclass/linux/linux-module.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/staging/lustre/lustre/obdclass/linux/linux-module.c b/drivers/staging/lustre/lustre/obdclass/linux/linux-module.c
index 9f5e829..eb88bd9 100644
--- a/drivers/staging/lustre/lustre/obdclass/linux/linux-module.c
+++ b/drivers/staging/lustre/lustre/obdclass/linux/linux-module.c
@@ -405,7 +405,7 @@ static int obd_device_list_open(struct inode *inode, struct file *file)
 struct kobject *lustre_kobj;
 EXPORT_SYMBOL_GPL(lustre_kobj);
 
-static struct attribute_group lustre_attr_group = {
+static const struct attribute_group lustre_attr_group = {
 	.attrs = lustre_attrs,
 };
 
-- 
1.9.1

From arvind.yadav.cs at gmail.com  Mon Jul 17 05:48:57 2017
From: arvind.yadav.cs at gmail.com (Arvind Yadav)
Date: Mon, 17 Jul 2017 11:18:57 +0530
Subject: [lustre-devel] [PATCH 1/4] staging: lustre: constify
	attribute_group structures.
In-Reply-To: <1500270540-5928-1-git-send-email-arvind.yadav.cs@gmail.com>
References: <1500270540-5928-1-git-send-email-arvind.yadav.cs@gmail.com>
Message-ID: <1500270540-5928-2-git-send-email-arvind.yadav.cs@gmail.com>

attribute_groups are not supposed to change at runtime. All functions
working with attribute_groups provided by <linux/sysfs.h> work
with const attribute_group. So mark the non-const structs as const.

File size before:
   text	   data	    bss	    dec	    hex	filename
   9489	    992	     40	  10521	   2919	lustre/lustre/osc/lproc_osc.o
   1289	    288	      0	   1577	    629	lustre/lustre/lmv/lproc_lmv.o
   3794	    928	     40	   4762	   129a	lustre/lustre/lov/lproc_lov.o
   3802	    576	     40	   4418	   1142	lustre/lustre/mdc/lproc_mdc.o

File size After adding 'const':
   text	   data	    bss	    dec	    hex	filename
   9553	    928	     40	  10521	   2919	lustre/lustre/osc/lproc_osc.o
   1353	    224	      0	   1577	    629	lustre/lustre/lmv/lproc_lmv.o
   3858	    864	     40	   4762	   129a	lustre/lustre/lov/lproc_lov.o
   3866	    512	     40	   4418	   1142	lustre/lustre/mdc/lproc_mdc.o

Signed-off-by: Arvind Yadav <arvind.yadav.cs at gmail.com>
---
 drivers/staging/lustre/lustre/include/lprocfs_status.h | 4 ++--
 drivers/staging/lustre/lustre/lmv/lproc_lmv.c          | 2 +-
 drivers/staging/lustre/lustre/lov/lproc_lov.c          | 2 +-
 drivers/staging/lustre/lustre/mdc/lproc_mdc.c          | 2 +-
 drivers/staging/lustre/lustre/osc/lproc_osc.c          | 2 +-
 5 files changed, 6 insertions(+), 6 deletions(-)

diff --git a/drivers/staging/lustre/lustre/include/lprocfs_status.h b/drivers/staging/lustre/lustre/include/lprocfs_status.h
index 915283c..9054d37 100644
--- a/drivers/staging/lustre/lustre/include/lprocfs_status.h
+++ b/drivers/staging/lustre/lustre/include/lprocfs_status.h
@@ -59,7 +59,7 @@ struct lprocfs_vars {
 
 struct lprocfs_static_vars {
 	struct lprocfs_vars *obd_vars;
-	struct attribute_group *sysfs_vars;
+	const struct attribute_group *sysfs_vars;
 };
 
 /* if we find more consumers this could be generalized */
@@ -468,7 +468,7 @@ struct dentry *ldebugfs_register(const char *name,
 void ldebugfs_remove(struct dentry **entryp);
 
 int lprocfs_obd_setup(struct obd_device *obd, struct lprocfs_vars *list,
-		      struct attribute_group *attrs);
+		      const struct attribute_group *attrs);
 int lprocfs_obd_cleanup(struct obd_device *obd);
 
 int ldebugfs_seq_create(struct dentry *parent,
diff --git a/drivers/staging/lustre/lustre/lmv/lproc_lmv.c b/drivers/staging/lustre/lustre/lmv/lproc_lmv.c
index bf25f88..4c13e39 100644
--- a/drivers/staging/lustre/lustre/lmv/lproc_lmv.c
+++ b/drivers/staging/lustre/lustre/lmv/lproc_lmv.c
@@ -161,7 +161,7 @@ static int lmv_target_seq_open(struct inode *inode, struct file *file)
 	NULL,
 };
 
-static struct attribute_group lmv_attr_group = {
+static const struct attribute_group lmv_attr_group = {
 	.attrs = lmv_attrs,
 };
 
diff --git a/drivers/staging/lustre/lustre/lov/lproc_lov.c b/drivers/staging/lustre/lustre/lov/lproc_lov.c
index eb6d30d..ce46821 100644
--- a/drivers/staging/lustre/lustre/lov/lproc_lov.c
+++ b/drivers/staging/lustre/lustre/lov/lproc_lov.c
@@ -279,7 +279,7 @@ static int lov_target_seq_open(struct inode *inode, struct file *file)
 	NULL,
 };
 
-static struct attribute_group lov_attr_group = {
+static const struct attribute_group lov_attr_group = {
 	.attrs = lov_attrs,
 };
 
diff --git a/drivers/staging/lustre/lustre/mdc/lproc_mdc.c b/drivers/staging/lustre/lustre/mdc/lproc_mdc.c
index 9021c46..9fea535 100644
--- a/drivers/staging/lustre/lustre/mdc/lproc_mdc.c
+++ b/drivers/staging/lustre/lustre/mdc/lproc_mdc.c
@@ -219,7 +219,7 @@ static ssize_t max_pages_per_rpc_show(struct kobject *kobj,
 	NULL,
 };
 
-static struct attribute_group mdc_attr_group = {
+static const struct attribute_group mdc_attr_group = {
 	.attrs = mdc_attrs,
 };
 
diff --git a/drivers/staging/lustre/lustre/osc/lproc_osc.c b/drivers/staging/lustre/lustre/osc/lproc_osc.c
index 86f252d..6e0fd15 100644
--- a/drivers/staging/lustre/lustre/osc/lproc_osc.c
+++ b/drivers/staging/lustre/lustre/osc/lproc_osc.c
@@ -831,7 +831,7 @@ int lproc_osc_attach_seqstat(struct obd_device *dev)
 	NULL,
 };
 
-static struct attribute_group osc_attr_group = {
+static const struct attribute_group osc_attr_group = {
 	.attrs = osc_attrs,
 };
 
-- 
1.9.1

From arvind.yadav.cs at gmail.com  Mon Jul 17 07:41:04 2017
From: arvind.yadav.cs at gmail.com (Arvind Yadav)
Date: Mon, 17 Jul 2017 13:11:04 +0530
Subject: [lustre-devel] [PATCH 4/4] staging: lustre: obdclass: constify
 attribute_group structures.
In-Reply-To: <1500270540-5928-5-git-send-email-arvind.yadav.cs@gmail.com>
References: <1500270540-5928-1-git-send-email-arvind.yadav.cs@gmail.com>
	<1500270540-5928-5-git-send-email-arvind.yadav.cs@gmail.com>
Message-ID: <84171e90-fd40-43f8-46b4-8645a60aab6e@gmail.com>

Sorry for noise. Please ignore this. This change should not be separate.

~arvind

On Monday 17 July 2017 11:19 AM, Arvind Yadav wrote:
> attribute_groups are not supposed to change at runtime. All functions
> working with attribute_groups provided by <linux/sysfs.h> work
> with const attribute_group. So mark the non-const structs as const.
>
> Signed-off-by: Arvind Yadav <arvind.yadav.cs at gmail.com>
> ---
>   drivers/staging/lustre/lustre/obdclass/lprocfs_status.c | 2 +-
>   1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/drivers/staging/lustre/lustre/obdclass/lprocfs_status.c b/drivers/staging/lustre/lustre/obdclass/lprocfs_status.c
> index bc19f19..ba41983 100644
> --- a/drivers/staging/lustre/lustre/obdclass/lprocfs_status.c
> +++ b/drivers/staging/lustre/lustre/obdclass/lprocfs_status.c
> @@ -1031,7 +1031,7 @@ static void obd_sysfs_release(struct kobject *kobj)
>   };
>   
>   int lprocfs_obd_setup(struct obd_device *obd, struct lprocfs_vars *list,
> -		      struct attribute_group *attrs)
> +		      const struct attribute_group *attrs)
>   {
>   	int rc = 0;
>   

From arvind.yadav.cs at gmail.com  Mon Jul 17 05:49:00 2017
From: arvind.yadav.cs at gmail.com (Arvind Yadav)
Date: Mon, 17 Jul 2017 11:19:00 +0530
Subject: [lustre-devel] [PATCH 4/4] staging: lustre: obdclass: constify
	attribute_group structures.
In-Reply-To: <1500270540-5928-1-git-send-email-arvind.yadav.cs@gmail.com>
References: <1500270540-5928-1-git-send-email-arvind.yadav.cs@gmail.com>
Message-ID: <1500270540-5928-5-git-send-email-arvind.yadav.cs@gmail.com>

attribute_groups are not supposed to change at runtime. All functions
working with attribute_groups provided by <linux/sysfs.h> work
with const attribute_group. So mark the non-const structs as const.

Signed-off-by: Arvind Yadav <arvind.yadav.cs at gmail.com>
---
 drivers/staging/lustre/lustre/obdclass/lprocfs_status.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/staging/lustre/lustre/obdclass/lprocfs_status.c b/drivers/staging/lustre/lustre/obdclass/lprocfs_status.c
index bc19f19..ba41983 100644
--- a/drivers/staging/lustre/lustre/obdclass/lprocfs_status.c
+++ b/drivers/staging/lustre/lustre/obdclass/lprocfs_status.c
@@ -1031,7 +1031,7 @@ static void obd_sysfs_release(struct kobject *kobj)
 };
 
 int lprocfs_obd_setup(struct obd_device *obd, struct lprocfs_vars *list,
-		      struct attribute_group *attrs)
+		      const struct attribute_group *attrs)
 {
 	int rc = 0;
 
-- 
1.9.1

From arvind.yadav.cs at gmail.com  Mon Jul 17 07:42:29 2017
From: arvind.yadav.cs at gmail.com (Arvind Yadav)
Date: Mon, 17 Jul 2017 13:12:29 +0530
Subject: [lustre-devel] [PATCH 2/4] staging: lustre: ldlm: constify
 attribute_group structures.
In-Reply-To: <1500270540-5928-3-git-send-email-arvind.yadav.cs@gmail.com>
References: <1500270540-5928-1-git-send-email-arvind.yadav.cs@gmail.com>
	<1500270540-5928-3-git-send-email-arvind.yadav.cs@gmail.com>
Message-ID: <2032ba34-5f7b-259b-d224-a9eb8c73a4a5@gmail.com>

Sorry for noise. Please ignore this.

~arvind


On Monday 17 July 2017 11:18 AM, Arvind Yadav wrote:
> attribute_groups are not supposed to change at runtime. All functions
> working with attribute_groups provided by <linux/sysfs.h> work
> with const attribute_group. So mark the non-const structs as const.
>
> Signed-off-by: Arvind Yadav <arvind.yadav.cs at gmail.com>
> ---
>   drivers/staging/lustre/lustre/ldlm/ldlm_lockd.c | 2 +-
>   1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/drivers/staging/lustre/lustre/ldlm/ldlm_lockd.c b/drivers/staging/lustre/lustre/ldlm/ldlm_lockd.c
> index fff930f..e0c3e5d 100644
> --- a/drivers/staging/lustre/lustre/ldlm/ldlm_lockd.c
> +++ b/drivers/staging/lustre/lustre/ldlm/ldlm_lockd.c
> @@ -926,7 +926,7 @@ static ssize_t cancel_unused_locks_before_replay_store(struct kobject *kobj,
>   	NULL,
>   };
>   
> -static struct attribute_group ldlm_attr_group = {
> +static const struct attribute_group ldlm_attr_group = {
>   	.attrs = ldlm_attrs,
>   };
>   

From arvind.yadav.cs at gmail.com  Mon Jul 17 07:45:28 2017
From: arvind.yadav.cs at gmail.com (Arvind Yadav)
Date: Mon, 17 Jul 2017 13:15:28 +0530
Subject: [lustre-devel] [PATCH 1/4] staging: lustre: constify
	attribute_group structures.
In-Reply-To: <201707171554.Ee9bmEKo%fengguang.wu@intel.com>
References: <201707171554.Ee9bmEKo%fengguang.wu@intel.com>
Message-ID: <ac92cdae-f135-6707-323c-1e9e44703b2f@gmail.com>

Sorry for noise. Please ignore this. I will fix this error and push again.

~arvind


On Monday 17 July 2017 12:45 PM, kbuild test robot wrote:
> Hi Arvind,
>
> [auto build test ERROR on staging/staging-testing]
> [also build test ERROR on v4.13-rc1 next-20170717]
> [if your patch is applied to the wrong git tree, please drop us a note to help improve the system]
>
> url:    https://github.com/0day-ci/linux/commits/Arvind-Yadav/constify-lustre-attribute_group-structures/20170717-142957
> config: sparc64-allyesconfig (attached as .config)
> compiler: sparc64-linux-gnu-gcc (Debian 6.1.1-9) 6.1.1 20160705
> reproduce:
>          wget https://raw.githubusercontent.com/01org/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross
>          chmod +x ~/bin/make.cross
>          # save the attached .config to linux build tree
>          make.cross ARCH=sparc64
>
> Note: the linux-review/Arvind-Yadav/constify-lustre-attribute_group-structures/20170717-142957 HEAD a6df15f5fc00a12dffbac94e0ccc52155118946e builds fine.
>        It only hurts bisectibility.
>
> All error/warnings (new ones prefixed by >>):
>
>>> drivers/staging/lustre/lustre/obdclass/lprocfs_status.c:1033:5: error: conflicting types for 'lprocfs_obd_setup'
>      int lprocfs_obd_setup(struct obd_device *obd, struct lprocfs_vars *list,
>          ^~~~~~~~~~~~~~~~~
>     In file included from drivers/staging/lustre/lustre/obdclass/../include/obd_support.h:41:0,
>                      from drivers/staging/lustre/lustre/obdclass/../include/obd_class.h:35,
>                      from drivers/staging/lustre/lustre/obdclass/lprocfs_status.c:39:
>     drivers/staging/lustre/lustre/obdclass/../include/lprocfs_status.h:470:5: note: previous declaration of 'lprocfs_obd_setup' was here
>      int lprocfs_obd_setup(struct obd_device *obd, struct lprocfs_vars *list,
>          ^~~~~~~~~~~~~~~~~
>     In file included from include/linux/linkage.h:6:0,
>                      from include/linux/kernel.h:6,
>                      from include/asm-generic/bug.h:15,
>                      from arch/sparc/include/asm/bug.h:20,
>                      from include/linux/bug.h:4,
>                      from include/linux/mmdebug.h:4,
>                      from include/linux/gfp.h:4,
>                      from include/linux/slab.h:14,
>                      from drivers/staging/lustre/lustre/obdclass/../include/obd_support.h:36,
>                      from drivers/staging/lustre/lustre/obdclass/../include/obd_class.h:35,
>                      from drivers/staging/lustre/lustre/obdclass/lprocfs_status.c:39:
>     drivers/staging/lustre/lustre/obdclass/lprocfs_status.c:1066:19: error: conflicting types for 'lprocfs_obd_setup'
>      EXPORT_SYMBOL_GPL(lprocfs_obd_setup);
>                        ^
>     include/linux/export.h:65:21: note: in definition of macro '___EXPORT_SYMBOL'
>       extern typeof(sym) sym;      \
>                          ^~~
>>> drivers/staging/lustre/lustre/obdclass/lprocfs_status.c:1066:1: note: in expansion of macro 'EXPORT_SYMBOL_GPL'
>      EXPORT_SYMBOL_GPL(lprocfs_obd_setup);
>      ^~~~~~~~~~~~~~~~~
>     In file included from drivers/staging/lustre/lustre/obdclass/../include/obd_support.h:41:0,
>                      from drivers/staging/lustre/lustre/obdclass/../include/obd_class.h:35,
>                      from drivers/staging/lustre/lustre/obdclass/lprocfs_status.c:39:
>     drivers/staging/lustre/lustre/obdclass/../include/lprocfs_status.h:470:5: note: previous declaration of 'lprocfs_obd_setup' was here
>      int lprocfs_obd_setup(struct obd_device *obd, struct lprocfs_vars *list,
>          ^~~~~~~~~~~~~~~~~
>
> vim +/lprocfs_obd_setup +1033 drivers/staging/lustre/lustre/obdclass/lprocfs_status.c
>
> 9b8013023 Oleg Drokin   2015-05-21  1032
> 9b8013023 Oleg Drokin   2015-05-21 @1033  int lprocfs_obd_setup(struct obd_device *obd, struct lprocfs_vars *list,
> 9b8013023 Oleg Drokin   2015-05-21  1034  		      struct attribute_group *attrs)
> d7e09d039 Peng Tao      2013-05-02  1035  {
> d7e09d039 Peng Tao      2013-05-02  1036  	int rc = 0;
> d7e09d039 Peng Tao      2013-05-02  1037
> 9b8013023 Oleg Drokin   2015-05-21  1038  	init_completion(&obd->obd_kobj_unregister);
> 9b8013023 Oleg Drokin   2015-05-21  1039  	rc = kobject_init_and_add(&obd->obd_kobj, &obd_ktype,
> 9b8013023 Oleg Drokin   2015-05-21  1040  				  obd->obd_type->typ_kobj,
> 9b8013023 Oleg Drokin   2015-05-21  1041  				  "%s", obd->obd_name);
> 9b8013023 Oleg Drokin   2015-05-21  1042  	if (rc)
> 9b8013023 Oleg Drokin   2015-05-21  1043  		return rc;
> 9b8013023 Oleg Drokin   2015-05-21  1044
> 9b8013023 Oleg Drokin   2015-05-21  1045  	if (attrs) {
> 9b8013023 Oleg Drokin   2015-05-21  1046  		rc = sysfs_create_group(&obd->obd_kobj, attrs);
> 9b8013023 Oleg Drokin   2015-05-21  1047  		if (rc) {
> 9b8013023 Oleg Drokin   2015-05-21  1048  			kobject_put(&obd->obd_kobj);
> 9b8013023 Oleg Drokin   2015-05-21  1049  			return rc;
> 9b8013023 Oleg Drokin   2015-05-21  1050  		}
> 9b8013023 Oleg Drokin   2015-05-21  1051  	}
> 9b8013023 Oleg Drokin   2015-05-21  1052
> 61e87ab0f Dmitry Eremin 2015-05-21  1053  	obd->obd_debugfs_entry = ldebugfs_register(obd->obd_name,
> 61e87ab0f Dmitry Eremin 2015-05-21  1054  						   obd->obd_type->typ_debugfs_entry,
> d7e09d039 Peng Tao      2013-05-02  1055  						   list, obd);
> 61e87ab0f Dmitry Eremin 2015-05-21  1056  	if (IS_ERR_OR_NULL(obd->obd_debugfs_entry)) {
> 61e87ab0f Dmitry Eremin 2015-05-21  1057  		rc = obd->obd_debugfs_entry ? PTR_ERR(obd->obd_debugfs_entry)
> 61e87ab0f Dmitry Eremin 2015-05-21  1058  					    : -ENOMEM;
> 1d8cb70c7 Greg Donald   2014-08-25  1059  		CERROR("error %d setting up lprocfs for %s\n",
> 1d8cb70c7 Greg Donald   2014-08-25  1060  		       rc, obd->obd_name);
> 61e87ab0f Dmitry Eremin 2015-05-21  1061  		obd->obd_debugfs_entry = NULL;
> d7e09d039 Peng Tao      2013-05-02  1062  	}
> 9b8013023 Oleg Drokin   2015-05-21  1063
> d7e09d039 Peng Tao      2013-05-02  1064  	return rc;
> d7e09d039 Peng Tao      2013-05-02  1065  }
> da33f1dd9 Oleg Drokin   2016-02-06 @1066  EXPORT_SYMBOL_GPL(lprocfs_obd_setup);
> d7e09d039 Peng Tao      2013-05-02  1067
>
> :::::: The code at line 1033 was first introduced by commit
> :::::: 9b8013023cb62360b56c04313687e93a1c2bf3d6 staging/lustre/obdclass: Prepare for procfs to sysfs migration
>
> :::::: TO: Oleg Drokin <green at linuxhacker.ru>
> :::::: CC: Greg Kroah-Hartman <gregkh at linuxfoundation.org>
>
> ---
> 0-DAY kernel test infrastructure                Open Source Technology Center
> https://lists.01.org/pipermail/kbuild-all                   Intel Corporation

From arvind.yadav.cs at gmail.com  Mon Jul 17 07:41:33 2017
From: arvind.yadav.cs at gmail.com (Arvind Yadav)
Date: Mon, 17 Jul 2017 13:11:33 +0530
Subject: [lustre-devel] [PATCH 3/4] staging: lustre: obdclass: linux:
 constify attribute_group structures.
In-Reply-To: <1500270540-5928-4-git-send-email-arvind.yadav.cs@gmail.com>
References: <1500270540-5928-1-git-send-email-arvind.yadav.cs@gmail.com>
	<1500270540-5928-4-git-send-email-arvind.yadav.cs@gmail.com>
Message-ID: <e6ee7f53-62fd-4318-4450-cd8ecf433f25@gmail.com>

Sorry for noise. Please ignore this.

~arvind


On Monday 17 July 2017 11:18 AM, Arvind Yadav wrote:
> attribute_groups are not supposed to change at runtime. All functions
> working with attribute_groups provided by <linux/sysfs.h> work
> with const attribute_group. So mark the non-const structs as const.
>
> Signed-off-by: Arvind Yadav <arvind.yadav.cs at gmail.com>
> ---
>   drivers/staging/lustre/lustre/obdclass/linux/linux-module.c | 2 +-
>   1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/drivers/staging/lustre/lustre/obdclass/linux/linux-module.c b/drivers/staging/lustre/lustre/obdclass/linux/linux-module.c
> index 9f5e829..eb88bd9 100644
> --- a/drivers/staging/lustre/lustre/obdclass/linux/linux-module.c
> +++ b/drivers/staging/lustre/lustre/obdclass/linux/linux-module.c
> @@ -405,7 +405,7 @@ static int obd_device_list_open(struct inode *inode, struct file *file)
>   struct kobject *lustre_kobj;
>   EXPORT_SYMBOL_GPL(lustre_kobj);
>   
> -static struct attribute_group lustre_attr_group = {
> +static const struct attribute_group lustre_attr_group = {
>   	.attrs = lustre_attrs,
>   };
>   

From arvind.yadav.cs at gmail.com  Mon Jul 17 05:48:58 2017
From: arvind.yadav.cs at gmail.com (Arvind Yadav)
Date: Mon, 17 Jul 2017 11:18:58 +0530
Subject: [lustre-devel] [PATCH 2/4] staging: lustre: ldlm: constify
	attribute_group structures.
In-Reply-To: <1500270540-5928-1-git-send-email-arvind.yadav.cs@gmail.com>
References: <1500270540-5928-1-git-send-email-arvind.yadav.cs@gmail.com>
Message-ID: <1500270540-5928-3-git-send-email-arvind.yadav.cs@gmail.com>

attribute_groups are not supposed to change at runtime. All functions
working with attribute_groups provided by <linux/sysfs.h> work
with const attribute_group. So mark the non-const structs as const.

Signed-off-by: Arvind Yadav <arvind.yadav.cs at gmail.com>
---
 drivers/staging/lustre/lustre/ldlm/ldlm_lockd.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/staging/lustre/lustre/ldlm/ldlm_lockd.c b/drivers/staging/lustre/lustre/ldlm/ldlm_lockd.c
index fff930f..e0c3e5d 100644
--- a/drivers/staging/lustre/lustre/ldlm/ldlm_lockd.c
+++ b/drivers/staging/lustre/lustre/ldlm/ldlm_lockd.c
@@ -926,7 +926,7 @@ static ssize_t cancel_unused_locks_before_replay_store(struct kobject *kobj,
 	NULL,
 };
 
-static struct attribute_group ldlm_attr_group = {
+static const struct attribute_group ldlm_attr_group = {
 	.attrs = ldlm_attrs,
 };
 
-- 
1.9.1

From arvind.yadav.cs at gmail.com  Mon Jul 17 07:44:22 2017
From: arvind.yadav.cs at gmail.com (Arvind Yadav)
Date: Mon, 17 Jul 2017 13:14:22 +0530
Subject: [lustre-devel] [PATCH 0/4] constify lustre attribute_group
	structures
In-Reply-To: <1500270540-5928-1-git-send-email-arvind.yadav.cs@gmail.com>
References: <1500270540-5928-1-git-send-email-arvind.yadav.cs@gmail.com>
Message-ID: <34a2dd81-75e0-ab17-5817-6601640b0f18@gmail.com>

Sorry for noise. Please ignore this. It's having build error. Again, I 
will push all changes.

~arvind


On Monday 17 July 2017 11:18 AM, Arvind Yadav wrote:
> attribute_groups are not supposed to change at runtime. All functions
> working with attribute_groups provided by <linux/sysfs.h> work
> with const attribute_group. So mark the non-const structs as const.
>
> Arvind Yadav (4):
>    [PATCH 1/4] staging: lustre: constify attribute_group structures.
>    [PATCH 2/4] staging: lustre: ldlm: constify attribute_group structures.
>    [PATCH 3/4] staging: lustre: obdclass: linux: constify attribute_group structures.
>    [PATCH 4/4] staging: lustre: obdclass: constify attribute_group structures.
>
>   drivers/staging/lustre/lustre/include/lprocfs_status.h      | 4 ++--
>   drivers/staging/lustre/lustre/ldlm/ldlm_lockd.c             | 2 +-
>   drivers/staging/lustre/lustre/lmv/lproc_lmv.c               | 2 +-
>   drivers/staging/lustre/lustre/lov/lproc_lov.c               | 2 +-
>   drivers/staging/lustre/lustre/mdc/lproc_mdc.c               | 2 +-
>   drivers/staging/lustre/lustre/obdclass/linux/linux-module.c | 2 +-
>   drivers/staging/lustre/lustre/obdclass/lprocfs_status.c     | 2 +-
>   drivers/staging/lustre/lustre/osc/lproc_osc.c               | 2 +-
>   8 files changed, 9 insertions(+), 9 deletions(-)
>

From arvind.yadav.cs at gmail.com  Mon Jul 17 05:48:56 2017
From: arvind.yadav.cs at gmail.com (Arvind Yadav)
Date: Mon, 17 Jul 2017 11:18:56 +0530
Subject: [lustre-devel] [PATCH 0/4] constify lustre attribute_group
	structures
Message-ID: <1500270540-5928-1-git-send-email-arvind.yadav.cs@gmail.com>

attribute_groups are not supposed to change at runtime. All functions
working with attribute_groups provided by <linux/sysfs.h> work
with const attribute_group. So mark the non-const structs as const.

Arvind Yadav (4):
  [PATCH 1/4] staging: lustre: constify attribute_group structures.
  [PATCH 2/4] staging: lustre: ldlm: constify attribute_group structures.
  [PATCH 3/4] staging: lustre: obdclass: linux: constify attribute_group structures.
  [PATCH 4/4] staging: lustre: obdclass: constify attribute_group structures.

 drivers/staging/lustre/lustre/include/lprocfs_status.h      | 4 ++--
 drivers/staging/lustre/lustre/ldlm/ldlm_lockd.c             | 2 +-
 drivers/staging/lustre/lustre/lmv/lproc_lmv.c               | 2 +-
 drivers/staging/lustre/lustre/lov/lproc_lov.c               | 2 +-
 drivers/staging/lustre/lustre/mdc/lproc_mdc.c               | 2 +-
 drivers/staging/lustre/lustre/obdclass/linux/linux-module.c | 2 +-
 drivers/staging/lustre/lustre/obdclass/lprocfs_status.c     | 2 +-
 drivers/staging/lustre/lustre/osc/lproc_osc.c               | 2 +-
 8 files changed, 9 insertions(+), 9 deletions(-)

-- 
1.9.1

From arvind.yadav.cs at gmail.com  Mon Jul 17 07:41:53 2017
From: arvind.yadav.cs at gmail.com (Arvind Yadav)
Date: Mon, 17 Jul 2017 13:11:53 +0530
Subject: [lustre-devel] [PATCH 3/4] staging: lustre: obdclass: linux:
 constify attribute_group structures.
In-Reply-To: <1500270540-5928-4-git-send-email-arvind.yadav.cs@gmail.com>
References: <1500270540-5928-1-git-send-email-arvind.yadav.cs@gmail.com>
	<1500270540-5928-4-git-send-email-arvind.yadav.cs@gmail.com>
Message-ID: <97f530fa-1a6a-b129-4190-2e96f2f0b81f@gmail.com>

Sorry for noise. Please ignore this.

~arvind


On Monday 17 July 2017 11:18 AM, Arvind Yadav wrote:
> attribute_groups are not supposed to change at runtime. All functions
> working with attribute_groups provided by <linux/sysfs.h> work
> with const attribute_group. So mark the non-const structs as const.
>
> Signed-off-by: Arvind Yadav <arvind.yadav.cs at gmail.com>
> ---
>   drivers/staging/lustre/lustre/obdclass/linux/linux-module.c | 2 +-
>   1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/drivers/staging/lustre/lustre/obdclass/linux/linux-module.c b/drivers/staging/lustre/lustre/obdclass/linux/linux-module.c
> index 9f5e829..eb88bd9 100644
> --- a/drivers/staging/lustre/lustre/obdclass/linux/linux-module.c
> +++ b/drivers/staging/lustre/lustre/obdclass/linux/linux-module.c
> @@ -405,7 +405,7 @@ static int obd_device_list_open(struct inode *inode, struct file *file)
>   struct kobject *lustre_kobj;
>   EXPORT_SYMBOL_GPL(lustre_kobj);
>   
> -static struct attribute_group lustre_attr_group = {
> +static const struct attribute_group lustre_attr_group = {
>   	.attrs = lustre_attrs,
>   };
>   

From andreas.dilger at intel.com  Mon Jul 17 17:44:59 2017
From: andreas.dilger at intel.com (Dilger, Andreas)
Date: Mon, 17 Jul 2017 17:44:59 +0000
Subject: [lustre-devel] [PATCH v2 4/4] Staging: Lustre Fix block
	statement style issue
In-Reply-To: <774a2f25b839305f100e7fa99ebf835fb2ab7d44.1499849566.git.Craig@craiginches.com>
References: <cover.1499849566.git.Craig@craiginches.com>
	<774a2f25b839305f100e7fa99ebf835fb2ab7d44.1499849566.git.Craig@craiginches.com>
Message-ID: <F6F7D587-CD46-40B8-A5B5-678B6A16AD8F@intel.com>

On Jul 12, 2017, at 03:03, Craig Inches <craig at craiginches.com> wrote:
> 
> This fixes a block statement which didnt end with */
> 
> Signed-off-by: Craig Inches <Craig at craiginches.com>
> ---
> drivers/staging/lustre/include/linux/lnet/socklnd.h | 3 ++-
> 1 file changed, 2 insertions(+), 1 deletion(-)
> 
> diff --git a/drivers/staging/lustre/include/linux/lnet/socklnd.h b/drivers/staging/lustre/include/linux/lnet/socklnd.h
> index dd5bc0e46560..a1ae66ede7a8 100644
> --- a/drivers/staging/lustre/include/linux/lnet/socklnd.h
> +++ b/drivers/staging/lustre/include/linux/lnet/socklnd.h
> @@ -76,7 +76,8 @@ struct ksock_msg {
> 	__u64	ksm_zc_cookies[2];	/* Zero-Copy request/ACK cookie */
> 	union {
> 		struct ksock_lnet_msg lnetmsg; /* lnet message, it's empty if
> -					  * it's NOOP */
> +						* it's NOOP
> +						*/

Rather than making the multi-line comments even more space consuming (I'm not a big
fan of "*/ must be on a line by itself" since it wastes a lot of vertical space),
I'd prefer to just shorten the comment, if possible, like:

		struct ksock_lnet_msg lnetmsg; /* lnet message, empty if NOOP */

Cheers, Andreas
--
Andreas Dilger
Lustre Principal Architect
Intel Corporation







From craig at craiginches.com  Mon Jul 17 17:50:27 2017
From: craig at craiginches.com (Craig Inches)
Date: Mon, 17 Jul 2017 17:50:27 +0000 (UTC)
Subject: [lustre-devel] [PATCH v2 4/4] Staging: Lustre Fix block
	statement style issue
In-Reply-To: <F6F7D587-CD46-40B8-A5B5-678B6A16AD8F@intel.com>
References: <cover.1499849566.git.Craig@craiginches.com>
	<774a2f25b839305f100e7fa99ebf835fb2ab7d44.1499849566.git.Craig@craiginches.com>
	<F6F7D587-CD46-40B8-A5B5-678B6A16AD8F@intel.com>
Message-ID: <20170717175017.GA32268@battlestar>

On Mon, Jul 17, 2017 at 05:44:59PM +0000, Dilger, Andreas wrote:
> On Jul 12, 2017, at 03:03, Craig Inches <craig at craiginches.com> wrote:
> > 
> > This fixes a block statement which didnt end with */
> > 
> > Signed-off-by: Craig Inches <Craig at craiginches.com>
> > ---
> > drivers/staging/lustre/include/linux/lnet/socklnd.h | 3 ++-
> > 1 file changed, 2 insertions(+), 1 deletion(-)
> > 
> > diff --git a/drivers/staging/lustre/include/linux/lnet/socklnd.h b/drivers/staging/lustre/include/linux/lnet/socklnd.h
> > index dd5bc0e46560..a1ae66ede7a8 100644
> > --- a/drivers/staging/lustre/include/linux/lnet/socklnd.h
> > +++ b/drivers/staging/lustre/include/linux/lnet/socklnd.h
> > @@ -76,7 +76,8 @@ struct ksock_msg {
> > 	__u64	ksm_zc_cookies[2];	/* Zero-Copy request/ACK cookie */
> > 	union {
> > 		struct ksock_lnet_msg lnetmsg; /* lnet message, it's empty if
> > -					  * it's NOOP */
> > +						* it's NOOP
> > +						*/
> 
> Rather than making the multi-line comments even more space consuming (I'm not a big
> fan of "*/ must be on a line by itself" since it wastes a lot of vertical space),
> I'd prefer to just shorten the comment, if possible, like:
> 
> 		struct ksock_lnet_msg lnetmsg; /* lnet message, empty if NOOP */

That makes sense, I believe the series has already made it to
staging-next, so not sure what the process would be to fix it up.

I am doing another set of style fixes which I can include this change
aswell.

Cheers,

Craig

From dan.carpenter at oracle.com  Mon Jul 17 18:21:43 2017
From: dan.carpenter at oracle.com (Dan Carpenter)
Date: Mon, 17 Jul 2017 21:21:43 +0300
Subject: [lustre-devel] [PATCH v2 4/4] Staging: Lustre Fix block
	statement style issue
In-Reply-To: <20170717175017.GA32268@battlestar>
References: <cover.1499849566.git.Craig@craiginches.com>
	<774a2f25b839305f100e7fa99ebf835fb2ab7d44.1499849566.git.Craig@craiginches.com>
	<F6F7D587-CD46-40B8-A5B5-678B6A16AD8F@intel.com>
	<20170717175017.GA32268@battlestar>
Message-ID: <20170717182143.6xns7ncwdn2wqxyf@mwanda>

On Mon, Jul 17, 2017 at 05:50:27PM +0000, Craig Inches wrote:
> On Mon, Jul 17, 2017 at 05:44:59PM +0000, Dilger, Andreas wrote:
> > On Jul 12, 2017, at 03:03, Craig Inches <craig at craiginches.com> wrote:
> > > 
> > > This fixes a block statement which didnt end with */
> > > 
> > > Signed-off-by: Craig Inches <Craig at craiginches.com>
> > > ---
> > > drivers/staging/lustre/include/linux/lnet/socklnd.h | 3 ++-
> > > 1 file changed, 2 insertions(+), 1 deletion(-)
> > > 
> > > diff --git a/drivers/staging/lustre/include/linux/lnet/socklnd.h b/drivers/staging/lustre/include/linux/lnet/socklnd.h
> > > index dd5bc0e46560..a1ae66ede7a8 100644
> > > --- a/drivers/staging/lustre/include/linux/lnet/socklnd.h
> > > +++ b/drivers/staging/lustre/include/linux/lnet/socklnd.h
> > > @@ -76,7 +76,8 @@ struct ksock_msg {
> > > 	__u64	ksm_zc_cookies[2];	/* Zero-Copy request/ACK cookie */
> > > 	union {
> > > 		struct ksock_lnet_msg lnetmsg; /* lnet message, it's empty if
> > > -					  * it's NOOP */
> > > +						* it's NOOP
> > > +						*/
> > 
> > Rather than making the multi-line comments even more space consuming (I'm not a big
> > fan of "*/ must be on a line by itself" since it wastes a lot of vertical space),
> > I'd prefer to just shorten the comment, if possible, like:
> > 
> > 		struct ksock_lnet_msg lnetmsg; /* lnet message, empty if NOOP */
> 
> That makes sense, I believe the series has already made it to
> staging-next, so not sure what the process would be to fix it up.

staging-next can't be rebased (barring something very drastic).  Send a
new patch based on what's been committed.

regards,
dan carpenter

From neilb at suse.com  Tue Jul 18 23:26:46 2017
From: neilb at suse.com (NeilBrown)
Date: Wed, 19 Jul 2017 09:26:46 +1000
Subject: [lustre-devel] [PATCH 00/12] Assorted lustre fixes and improvements
Message-ID: <150041997277.20736.17112251996623587423.stgit@noble>

Hi,
 I've been looking through lustre, finding things, and fixing some of
 them.
 There are three distinct subsets of patches here.
 - 1 patch to fix a few typos in comments
 - 2 patches to fix dcache issues, particularly around
    DCACHE_DISCONNECTED
 - 9 patches to clean up code in ldlm_flock().  These started
   as some simple list_entry tidy-ups, and just grew....

 The most interesting from a review perspective is
      staging: lustre: llite: fix various issues with ll_splice_alias.
 there are real locking issues in there, but its hard to be 100%
 sure I've fixed them all and not introduced new issues.
 It would be *really* nice if we could get rid of the concept of
 "invalid" dentries, but I don't yet understand it well enough to
 propose and alternative.

Thanks,
NeilBrown

---

NeilBrown (12):
      staging: lustre: fix minor typos in comments
      staging: lustre: llite: fix incorrect DCACHE_DISCONNECTED test
      staging: lustre: llite: fix various issues with ll_splice_alias.
      staging: lustre: ldlm: remove 'first_enq' arg from ldlm_process_flock_lock()
      staging: lustre: ldlm: remove unused 'work_list' arg from ldlm_process_flock_lock()
      staging: lustre: ldlm: remove unneeded 'err' arg to ldlm_process_flock_lock()
      staging: lustre: ldlm: remove 'flags' arg from ldlm_process_flock_lock()
      staging: lustre: ldlm: remove unused 'overlaps' variable
      staging: lustre: ldlm: remove 'flags' arg from ldlm_flock_destroy()
      staging: lustre: ldlm: tidy list walking in ldlm_flock()
      staging: lustre: ldlm: remove unnecessary 'ownlocks' variable.
      staging: lustre: ldlm: remove unused field 'fwd_generation'


 drivers/staging/lustre/lustre/ldlm/ldlm_flock.c    |  208 ++++----------------
 .../staging/lustre/lustre/llite/llite_internal.h   |    2 
 drivers/staging/lustre/lustre/llite/namei.c        |   71 +++----
 drivers/staging/lustre/lustre/mdc/mdc_locks.c      |    4 
 4 files changed, 80 insertions(+), 205 deletions(-)

--
Signature

From neilb at suse.com  Tue Jul 18 23:26:47 2017
From: neilb at suse.com (NeilBrown)
Date: Wed, 19 Jul 2017 09:26:47 +1000
Subject: [lustre-devel] [PATCH 01/12] staging: lustre: fix minor typos in
	comments
In-Reply-To: <150041997277.20736.17112251996623587423.stgit@noble>
References: <150041997277.20736.17112251996623587423.stgit@noble>
Message-ID: <150042040735.20736.6997246449635538567.stgit@noble>

Signed-off-by: NeilBrown <neilb at suse.com>
---
 drivers/staging/lustre/lustre/llite/namei.c   |    2 +-
 drivers/staging/lustre/lustre/mdc/mdc_locks.c |    4 ++--
 2 files changed, 3 insertions(+), 3 deletions(-)

diff --git a/drivers/staging/lustre/lustre/llite/namei.c b/drivers/staging/lustre/lustre/llite/namei.c
index a208a8b02c2c..293a3180ec70 100644
--- a/drivers/staging/lustre/lustre/llite/namei.c
+++ b/drivers/staging/lustre/lustre/llite/namei.c
@@ -490,7 +490,7 @@ static int ll_lookup_it_finish(struct ptlrpc_request *request,
 	*de = alias;
 
 	if (!it_disposition(it, DISP_LOOKUP_NEG)) {
-		/* we have lookup look - unhide dentry */
+		/* We have the "lookup" lock, so unhide dentry */
 		if (bits & MDS_INODELOCK_LOOKUP)
 			d_lustre_revalidate(*de);
 	} else if (!it_disposition(it, DISP_OPEN_CREATE)) {
diff --git a/drivers/staging/lustre/lustre/mdc/mdc_locks.c b/drivers/staging/lustre/lustre/mdc/mdc_locks.c
index 3eb66cea65db..ae97c6f1aeb0 100644
--- a/drivers/staging/lustre/lustre/mdc/mdc_locks.c
+++ b/drivers/staging/lustre/lustre/mdc/mdc_locks.c
@@ -1030,7 +1030,7 @@ int mdc_revalidate_lock(struct obd_export *exp, struct lookup_intent *it,
  * If we're performing a creation, that means that unless the creation
  * failed with EEXIST, we should fake up a negative dentry.
  *
- * For everything else, we want to lookup to succeed.
+ * For everything else, we want the lookup to succeed.
  *
  * One additional note: if CREATE or OPEN succeeded, we add an extra
  * reference to the request because we need to keep it around until
@@ -1040,7 +1040,7 @@ int mdc_revalidate_lock(struct obd_export *exp, struct lookup_intent *it,
  * exactly what it_status refers to.
  *
  * If DISP_OPEN_OPEN is set, then it_status refers to the open() call,
- * otherwise if DISP_OPEN_CREATE is set, then it status is the
+ * otherwise if DISP_OPEN_CREATE is set, then it_status is the
  * creation failure mode.  In either case, one of DISP_LOOKUP_NEG or
  * DISP_LOOKUP_POS will be set, indicating whether the child lookup
  * was successful.


From neilb at suse.com  Tue Jul 18 23:26:47 2017
From: neilb at suse.com (NeilBrown)
Date: Wed, 19 Jul 2017 09:26:47 +1000
Subject: [lustre-devel] [PATCH 02/12] staging: lustre: llite: fix incorrect
 DCACHE_DISCONNECTED test
In-Reply-To: <150041997277.20736.17112251996623587423.stgit@noble>
References: <150041997277.20736.17112251996623587423.stgit@noble>
Message-ID: <150042040741.20736.10796456139862947849.stgit@noble>

It is almost always wrong to test DCACHE_DISCONNECTED, except in
"exportfs" code.
The flag tells us that this dentry *might* not be connected to the
root through a chain of d_parent links.  Following the d_parent
to an IS_ROOT() dentry *might* find one that is on the s_anon list
rather than s_root.

The code here need to know if it is safe to call __d_drop(), and
the correct test is "!IS_ROOT(dentry)".
If a dentry IS_ROOT(), then it might be the filesystem root, or it
might be the root of a DCACHE_DISCONNECTED tree, and so be on
the s_anon list.  In these two cases it should not be __d_drop()ed.
If !IS_ROOT(), then the dentry is attached to its parent through
d_subdir, and can safely be unhashed.

Signed-off-by: NeilBrown <neilb at suse.com>
---
 .../staging/lustre/lustre/llite/llite_internal.h   |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/staging/lustre/lustre/llite/llite_internal.h b/drivers/staging/lustre/lustre/llite/llite_internal.h
index cd3311abf999..4854985bf4d3 100644
--- a/drivers/staging/lustre/lustre/llite/llite_internal.h
+++ b/drivers/staging/lustre/lustre/llite/llite_internal.h
@@ -1299,7 +1299,7 @@ static inline void d_lustre_invalidate(struct dentry *dentry, int nested)
 	 * If we unhashed such a dentry, unmount would not be able to find
 	 * it and busy inodes would be reported.
 	 */
-	if (d_count(dentry) == 0 && !(dentry->d_flags & DCACHE_DISCONNECTED))
+	if (d_count(dentry) == 0 && !IS_ROOT(dentry))
 		__d_drop(dentry);
 	spin_unlock(&dentry->d_lock);
 }


From neilb at suse.com  Tue Jul 18 23:26:47 2017
From: neilb at suse.com (NeilBrown)
Date: Wed, 19 Jul 2017 09:26:47 +1000
Subject: [lustre-devel] [PATCH 03/12] staging: lustre: llite: fix various
 issues with ll_splice_alias.
In-Reply-To: <150041997277.20736.17112251996623587423.stgit@noble>
References: <150041997277.20736.17112251996623587423.stgit@noble>
Message-ID: <150042040744.20736.6110134708456645941.stgit@noble>

1/ The testing of DCACHE_DISCONNECTED is wrong.
  see upstream commit da093a9b76ef ("dcache: d_splice_alias should
  ignore DCACHE_DISCONNECTED")

  As this is a notoriously difficult piece of code to get right,
  it makes sense to use d_splice_alias() directly and no try to
  create a local version of it.

2/ ll_find_alias() currently:
     locks and alias
     checks that it is the one we want
     unlock it
     locks it again
     gets a reference
     unlocks it

   This isn't safe.  Anything could happen to the dentry while we
   don't hold a reference.  We need to dget the reference while
   still holding the lock.

3/ The d_move() in ll_splice_alias() is pointless.  We have
    already checked the hash, name, and parent are the same, and
    these are the only fields that d_move() will change.

4/ The call to d_add() is outside of any locking. This makes it
   possible for two identical dentries to be added to the same
   inode, which would cause confusion.

   Prior to 4.7, i_mutex would have provided exclusion, but since
   the VFS supports parallel lookups, only a shared lock is held
   on i_mutex.

   Because ll_d_init() creates a dentry in a state where
   ll_dcompare will no recognize it, the VFS provides no guarantee
   that we won't have two concurrent calls to ll_lookup_dn() for
   the same parent/name.


So: rename ll_find_alias() to ll_find_invalid_alias() and have it
just focus on finding an invalid alias.

For directories, we can just use d__splice_alias() directly.
There must only be one alias for a directory, and
ll_splice_alias() will find it where it is "invalid" or not.

For non-directories, we call ll_find_invalid_alias(), and either
use the result or call d_add().  We need a lock to protect from
races with other threads calling ll_find_invalid_alias() and
d_add() at the same time. lli_lock seems suitable for this
purpose.

Signed-off-by: NeilBrown <neilb at suse.com>
---
 drivers/staging/lustre/lustre/llite/namei.c |   69 +++++++++++++--------------
 1 file changed, 34 insertions(+), 35 deletions(-)

diff --git a/drivers/staging/lustre/lustre/llite/namei.c b/drivers/staging/lustre/lustre/llite/namei.c
index 293a3180ec70..6204c3e70d45 100644
--- a/drivers/staging/lustre/lustre/llite/namei.c
+++ b/drivers/staging/lustre/lustre/llite/namei.c
@@ -378,75 +378,74 @@ void ll_i2gids(__u32 *suppgids, struct inode *i1, struct inode *i2)
 }
 
 /*
- * try to reuse three types of dentry:
- * 1. unhashed alias, this one is unhashed by d_invalidate (but it may be valid
- *    by concurrent .revalidate).
- * 2. INVALID alias (common case for no valid ldlm lock held, but this flag may
- *    be cleared by others calling d_lustre_revalidate).
- * 3. DISCONNECTED alias.
+ * Try to find an "invalid" alias.  i.e. one that was unhashed by
+ * d_invalidate(), or that was instantiated with no valid ldlm lock.
+ * These can be rehased by d_lustre_revalidate(), which could race
+ * with this code.
  */
-static struct dentry *ll_find_alias(struct inode *inode, struct dentry *dentry)
+static struct dentry *ll_find_invalid_alias(struct inode *inode,
+					    struct dentry *dentry)
 {
-	struct dentry *alias, *discon_alias, *invalid_alias;
+	struct dentry *alias, *invalid_alias = NULL;
 
 	if (hlist_empty(&inode->i_dentry))
 		return NULL;
 
-	discon_alias = NULL;
-	invalid_alias = NULL;
-
 	spin_lock(&inode->i_lock);
 	hlist_for_each_entry(alias, &inode->i_dentry, d_u.d_alias) {
 		LASSERT(alias != dentry);
 
 		spin_lock(&alias->d_lock);
-		if ((alias->d_flags & DCACHE_DISCONNECTED) &&
-		    S_ISDIR(inode->i_mode))
-			/* LASSERT(last_discon == NULL); LU-405, bz 20055 */
-			discon_alias = alias;
-		else if (alias->d_parent == dentry->d_parent	     &&
-			 alias->d_name.hash == dentry->d_name.hash       &&
-			 alias->d_name.len == dentry->d_name.len	 &&
-			 memcmp(alias->d_name.name, dentry->d_name.name,
-				dentry->d_name.len) == 0)
+		if (alias->d_parent == dentry->d_parent       &&
+		    alias->d_name.hash == dentry->d_name.hash &&
+		    alias->d_name.len == dentry->d_name.len   &&
+		    memcmp(alias->d_name.name, dentry->d_name.name,
+			   dentry->d_name.len) == 0) {
+			dget_dlock(alias);
 			invalid_alias = alias;
+		}
 		spin_unlock(&alias->d_lock);
 
 		if (invalid_alias)
 			break;
 	}
-	alias = invalid_alias ?: discon_alias ?: NULL;
-	if (alias) {
-		spin_lock(&alias->d_lock);
-		dget_dlock(alias);
-		spin_unlock(&alias->d_lock);
-	}
 	spin_unlock(&inode->i_lock);
 
-	return alias;
+	return invalid_alias;
 }
 
 /*
- * Similar to d_splice_alias(), but lustre treats invalid alias
- * similar to DCACHE_DISCONNECTED, and tries to use it anyway.
+ * Similar to d_splice_alias(), but also look for an "invalid" alias,
+ * specific to lustre, and use that if found.
  */
 struct dentry *ll_splice_alias(struct inode *inode, struct dentry *de)
 {
-	if (inode) {
-		struct dentry *new = ll_find_alias(inode, de);
+	if (inode && !S_ISDIR(inode->i_mode)) {
+		struct ll_inode_info *lli = ll_i2info(inode);
+		struct dentry *new;
+
+		/* We need lli_lock here as another thread could
+		 * be running this code, and i_lock cannot protect us.
+		 */
+		spin_lock(&lli->lli_lock);
+		new = ll_find_invalid_alias(inode, de);
+		if (!new)
+			d_add(de, inode);
+		spin_lock(&lli->lli_lock);
 
 		if (new) {
-			d_move(new, de);
 			iput(inode);
 			CDEBUG(D_DENTRY,
 			       "Reuse dentry %p inode %p refc %d flags %#x\n",
 			      new, d_inode(new), d_count(new), new->d_flags);
 			return new;
 		}
+		return de;
 	}
-	d_add(de, inode);
-	CDEBUG(D_DENTRY, "Add dentry %p inode %p refc %d flags %#x\n",
-	       de, d_inode(de), d_count(de), de->d_flags);
+	de = d_splice_alias(inode, de);
+	if (!IS_ERR(de))
+		CDEBUG(D_DENTRY, "Add dentry %p inode %p refc %d flags %#x\n",
+		       de, d_inode(de), d_count(de), de->d_flags);
 	return de;
 }
 


From neilb at suse.com  Tue Jul 18 23:26:47 2017
From: neilb at suse.com (NeilBrown)
Date: Wed, 19 Jul 2017 09:26:47 +1000
Subject: [lustre-devel] [PATCH 04/12] staging: lustre: ldlm: remove
 'first_enq' arg from ldlm_process_flock_lock()
In-Reply-To: <150041997277.20736.17112251996623587423.stgit@noble>
References: <150041997277.20736.17112251996623587423.stgit@noble>
Message-ID: <150042040748.20736.16657316764497795112.stgit@noble>

it is only ever set to '1', so we can just assume that and remove the code.

Signed-off-by: NeilBrown <neilb at suse.com>
---
 drivers/staging/lustre/lustre/ldlm/ldlm_flock.c |   15 ++-------------
 1 file changed, 2 insertions(+), 13 deletions(-)

diff --git a/drivers/staging/lustre/lustre/ldlm/ldlm_flock.c b/drivers/staging/lustre/lustre/ldlm/ldlm_flock.c
index b7f28b39c7b3..8ba3eaf49c65 100644
--- a/drivers/staging/lustre/lustre/ldlm/ldlm_flock.c
+++ b/drivers/staging/lustre/lustre/ldlm/ldlm_flock.c
@@ -121,15 +121,9 @@ ldlm_flock_destroy(struct ldlm_lock *lock, enum ldlm_mode mode, __u64 flags)
  * It is also responsible for splitting a lock if a portion of the lock
  * is released.
  *
- * If \a first_enq is 0 (ie, called from ldlm_reprocess_queue):
- *   - blocking ASTs have already been sent
- *
- * If \a first_enq is 1 (ie, called from ldlm_lock_enqueue):
- *   - blocking ASTs have not been sent yet, so list of conflicting locks
- *     would be collected and ASTs sent.
  */
 static int ldlm_process_flock_lock(struct ldlm_lock *req, __u64 *flags,
-				   int first_enq, enum ldlm_error *err,
+				   enum ldlm_error *err,
 				   struct list_head *work_list)
 {
 	struct ldlm_resource *res = req->l_resource;
@@ -197,11 +191,6 @@ static int ldlm_process_flock_lock(struct ldlm_lock *req, __u64 *flags,
 			if (!ldlm_flocks_overlap(lock, req))
 				continue;
 
-			if (!first_enq) {
-				reprocess_failed = 1;
-				continue;
-			}
-
 			if (*flags & LDLM_FL_BLOCK_NOWAIT) {
 				ldlm_flock_destroy(req, mode, *flags);
 				*err = -EAGAIN;
@@ -605,7 +594,7 @@ ldlm_flock_completion_ast(struct ldlm_lock *lock, __u64 flags, void *data)
 		/* We need to reprocess the lock to do merges or splits
 		 * with existing locks owned by this process.
 		 */
-		ldlm_process_flock_lock(lock, &noreproc, 1, &err, NULL);
+		ldlm_process_flock_lock(lock, &noreproc, &err, NULL);
 	}
 	unlock_res_and_lock(lock);
 	return rc;


From neilb at suse.com  Tue Jul 18 23:26:47 2017
From: neilb at suse.com (NeilBrown)
Date: Wed, 19 Jul 2017 09:26:47 +1000
Subject: [lustre-devel] [PATCH 05/12] staging: lustre: ldlm: remove unused
 'work_list' arg from ldlm_process_flock_lock()
In-Reply-To: <150041997277.20736.17112251996623587423.stgit@noble>
References: <150041997277.20736.17112251996623587423.stgit@noble>
Message-ID: <150042040750.20736.7704797489296782229.stgit@noble>

'work_list' is only set to NULL, and is never used.
So discard it.

Signed-off-by: NeilBrown <neilb at suse.com>
---
 drivers/staging/lustre/lustre/ldlm/ldlm_flock.c |    5 ++---
 1 file changed, 2 insertions(+), 3 deletions(-)

diff --git a/drivers/staging/lustre/lustre/ldlm/ldlm_flock.c b/drivers/staging/lustre/lustre/ldlm/ldlm_flock.c
index 8ba3eaf49c65..dce81fbf3049 100644
--- a/drivers/staging/lustre/lustre/ldlm/ldlm_flock.c
+++ b/drivers/staging/lustre/lustre/ldlm/ldlm_flock.c
@@ -123,8 +123,7 @@ ldlm_flock_destroy(struct ldlm_lock *lock, enum ldlm_mode mode, __u64 flags)
  *
  */
 static int ldlm_process_flock_lock(struct ldlm_lock *req, __u64 *flags,
-				   enum ldlm_error *err,
-				   struct list_head *work_list)
+				   enum ldlm_error *err)
 {
 	struct ldlm_resource *res = req->l_resource;
 	struct ldlm_namespace *ns = ldlm_res_to_ns(res);
@@ -594,7 +593,7 @@ ldlm_flock_completion_ast(struct ldlm_lock *lock, __u64 flags, void *data)
 		/* We need to reprocess the lock to do merges or splits
 		 * with existing locks owned by this process.
 		 */
-		ldlm_process_flock_lock(lock, &noreproc, &err, NULL);
+		ldlm_process_flock_lock(lock, &noreproc, &err);
 	}
 	unlock_res_and_lock(lock);
 	return rc;


From neilb at suse.com  Tue Jul 18 23:26:47 2017
From: neilb at suse.com (NeilBrown)
Date: Wed, 19 Jul 2017 09:26:47 +1000
Subject: [lustre-devel] [PATCH 06/12] staging: lustre: ldlm: remove unneeded
 'err' arg to ldlm_process_flock_lock()
In-Reply-To: <150041997277.20736.17112251996623587423.stgit@noble>
References: <150041997277.20736.17112251996623587423.stgit@noble>
Message-ID: <150042040753.20736.10406373852456927974.stgit@noble>

This arg is used to return an error code, but the returned code is never
looked at.  So there is no point returning it.

Signed-off-by: NeilBrown <neilb at suse.com>
---
 drivers/staging/lustre/lustre/ldlm/ldlm_flock.c |   10 ++--------
 1 file changed, 2 insertions(+), 8 deletions(-)

diff --git a/drivers/staging/lustre/lustre/ldlm/ldlm_flock.c b/drivers/staging/lustre/lustre/ldlm/ldlm_flock.c
index dce81fbf3049..fbb627a2f6c1 100644
--- a/drivers/staging/lustre/lustre/ldlm/ldlm_flock.c
+++ b/drivers/staging/lustre/lustre/ldlm/ldlm_flock.c
@@ -122,8 +122,7 @@ ldlm_flock_destroy(struct ldlm_lock *lock, enum ldlm_mode mode, __u64 flags)
  * is released.
  *
  */
-static int ldlm_process_flock_lock(struct ldlm_lock *req, __u64 *flags,
-				   enum ldlm_error *err)
+static int ldlm_process_flock_lock(struct ldlm_lock *req, __u64 *flags)
 {
 	struct ldlm_resource *res = req->l_resource;
 	struct ldlm_namespace *ns = ldlm_res_to_ns(res);
@@ -145,8 +144,6 @@ static int ldlm_process_flock_lock(struct ldlm_lock *req, __u64 *flags,
 	       req->l_policy_data.l_flock.start,
 	       req->l_policy_data.l_flock.end);
 
-	*err = ELDLM_OK;
-
 	/* No blocking ASTs are sent to the clients for
 	 * Posix file & record locks
 	 */
@@ -192,7 +189,6 @@ static int ldlm_process_flock_lock(struct ldlm_lock *req, __u64 *flags,
 
 			if (*flags & LDLM_FL_BLOCK_NOWAIT) {
 				ldlm_flock_destroy(req, mode, *flags);
-				*err = -EAGAIN;
 				return LDLM_ITER_STOP;
 			}
 
@@ -330,7 +326,6 @@ static int ldlm_process_flock_lock(struct ldlm_lock *req, __u64 *flags,
 			if (IS_ERR(new2)) {
 				ldlm_flock_destroy(req, lock->l_granted_mode,
 						   *flags);
-				*err = PTR_ERR(new2);
 				return LDLM_ITER_STOP;
 			}
 			goto reprocess;
@@ -440,7 +435,6 @@ ldlm_flock_completion_ast(struct ldlm_lock *lock, __u64 flags, void *data)
 	struct obd_import	      *imp = NULL;
 	struct ldlm_flock_wait_data     fwd;
 	struct l_wait_info	      lwi;
-	enum ldlm_error		    err;
 	int			     rc = 0;
 
 	OBD_FAIL_TIMEOUT(OBD_FAIL_LDLM_CP_CB_WAIT2, 4);
@@ -593,7 +587,7 @@ ldlm_flock_completion_ast(struct ldlm_lock *lock, __u64 flags, void *data)
 		/* We need to reprocess the lock to do merges or splits
 		 * with existing locks owned by this process.
 		 */
-		ldlm_process_flock_lock(lock, &noreproc, &err);
+		ldlm_process_flock_lock(lock, &noreproc);
 	}
 	unlock_res_and_lock(lock);
 	return rc;


From neilb at suse.com  Tue Jul 18 23:26:47 2017
From: neilb at suse.com (NeilBrown)
Date: Wed, 19 Jul 2017 09:26:47 +1000
Subject: [lustre-devel] [PATCH 07/12] staging: lustre: ldlm: remove 'flags'
 arg from ldlm_process_flock_lock()
In-Reply-To: <150041997277.20736.17112251996623587423.stgit@noble>
References: <150041997277.20736.17112251996623587423.stgit@noble>
Message-ID: <150042040756.20736.13537770572336560890.stgit@noble>

This is only ever set to LDLM_FL_WAIT_NOREPROC, so we can remove the arg
and discard any code that is only run when it doesn't have that value.

Signed-off-by: NeilBrown <neilb at suse.com>
---
 drivers/staging/lustre/lustre/ldlm/ldlm_flock.c |  110 ++++-------------------
 1 file changed, 21 insertions(+), 89 deletions(-)

diff --git a/drivers/staging/lustre/lustre/ldlm/ldlm_flock.c b/drivers/staging/lustre/lustre/ldlm/ldlm_flock.c
index fbb627a2f6c1..b6d5a83e61cd 100644
--- a/drivers/staging/lustre/lustre/ldlm/ldlm_flock.c
+++ b/drivers/staging/lustre/lustre/ldlm/ldlm_flock.c
@@ -122,7 +122,7 @@ ldlm_flock_destroy(struct ldlm_lock *lock, enum ldlm_mode mode, __u64 flags)
  * is released.
  *
  */
-static int ldlm_process_flock_lock(struct ldlm_lock *req, __u64 *flags)
+static int ldlm_process_flock_lock(struct ldlm_lock *req)
 {
 	struct ldlm_resource *res = req->l_resource;
 	struct ldlm_namespace *ns = ldlm_res_to_ns(res);
@@ -138,8 +138,8 @@ static int ldlm_process_flock_lock(struct ldlm_lock *req, __u64 *flags)
 	const struct ldlm_callback_suite null_cbs = { };
 
 	CDEBUG(D_DLMTRACE,
-	       "flags %#llx owner %llu pid %u mode %u start %llu end %llu\n",
-	       *flags, new->l_policy_data.l_flock.owner,
+	       "owner %llu pid %u mode %u start %llu end %llu\n",
+	       new->l_policy_data.l_flock.owner,
 	       new->l_policy_data.l_flock.pid, mode,
 	       req->l_policy_data.l_flock.start,
 	       req->l_policy_data.l_flock.end);
@@ -150,74 +150,16 @@ static int ldlm_process_flock_lock(struct ldlm_lock *req, __u64 *flags)
 	req->l_blocking_ast = NULL;
 
 reprocess:
-	if ((*flags == LDLM_FL_WAIT_NOREPROC) || (mode == LCK_NL)) {
-		/* This loop determines where this processes locks start
-		 * in the resource lr_granted list.
-		 */
-		list_for_each(tmp, &res->lr_granted) {
-			lock = list_entry(tmp, struct ldlm_lock,
-					  l_res_link);
-			if (ldlm_same_flock_owner(lock, req)) {
-				ownlocks = tmp;
-				break;
-			}
-		}
-	} else {
-		int reprocess_failed = 0;
-
-		lockmode_verify(mode);
-
-		/* This loop determines if there are existing locks
-		 * that conflict with the new lock request.
-		 */
-		list_for_each(tmp, &res->lr_granted) {
-			lock = list_entry(tmp, struct ldlm_lock,
-					  l_res_link);
-
-			if (ldlm_same_flock_owner(lock, req)) {
-				if (!ownlocks)
-					ownlocks = tmp;
-				continue;
-			}
-
-			/* locks are compatible, overlap doesn't matter */
-			if (lockmode_compat(lock->l_granted_mode, mode))
-				continue;
-
-			if (!ldlm_flocks_overlap(lock, req))
-				continue;
-
-			if (*flags & LDLM_FL_BLOCK_NOWAIT) {
-				ldlm_flock_destroy(req, mode, *flags);
-				return LDLM_ITER_STOP;
-			}
-
-			if (*flags & LDLM_FL_TEST_LOCK) {
-				ldlm_flock_destroy(req, mode, *flags);
-				req->l_req_mode = lock->l_granted_mode;
-				req->l_policy_data.l_flock.pid =
-					lock->l_policy_data.l_flock.pid;
-				req->l_policy_data.l_flock.start =
-					lock->l_policy_data.l_flock.start;
-				req->l_policy_data.l_flock.end =
-					lock->l_policy_data.l_flock.end;
-				*flags |= LDLM_FL_LOCK_CHANGED;
-				return LDLM_ITER_STOP;
-			}
-
-			ldlm_resource_add_lock(res, &res->lr_waiting, req);
-			*flags |= LDLM_FL_BLOCK_GRANTED;
-			return LDLM_ITER_STOP;
+	/* This loop determines where this processes locks start
+	 * in the resource lr_granted list.
+	 */
+	list_for_each(tmp, &res->lr_granted) {
+		lock = list_entry(tmp, struct ldlm_lock,
+				  l_res_link);
+		if (ldlm_same_flock_owner(lock, req)) {
+			ownlocks = tmp;
+			break;
 		}
-		if (reprocess_failed)
-			return LDLM_ITER_CONTINUE;
-	}
-
-	if (*flags & LDLM_FL_TEST_LOCK) {
-		ldlm_flock_destroy(req, mode, *flags);
-		req->l_req_mode = LCK_NL;
-		*flags |= LDLM_FL_LOCK_CHANGED;
-		return LDLM_ITER_STOP;
 	}
 
 	/* Scan the locks owned by this process that overlap this request.
@@ -267,7 +209,8 @@ static int ldlm_process_flock_lock(struct ldlm_lock *req, __u64 *flags)
 			}
 
 			if (added) {
-				ldlm_flock_destroy(lock, mode, *flags);
+				ldlm_flock_destroy(lock, mode,
+						   LDLM_FL_WAIT_NOREPROC);
 			} else {
 				new = lock;
 				added = 1;
@@ -293,7 +236,8 @@ static int ldlm_process_flock_lock(struct ldlm_lock *req, __u64 *flags)
 					new->l_policy_data.l_flock.end + 1;
 				break;
 			}
-			ldlm_flock_destroy(lock, lock->l_req_mode, *flags);
+			ldlm_flock_destroy(lock, lock->l_req_mode,
+					   LDLM_FL_WAIT_NOREPROC);
 			continue;
 		}
 		if (new->l_policy_data.l_flock.end >=
@@ -325,7 +269,7 @@ static int ldlm_process_flock_lock(struct ldlm_lock *req, __u64 *flags)
 			lock_res_and_lock(req);
 			if (IS_ERR(new2)) {
 				ldlm_flock_destroy(req, lock->l_granted_mode,
-						   *flags);
+						   LDLM_FL_WAIT_NOREPROC);
 				return LDLM_ITER_STOP;
 			}
 			goto reprocess;
@@ -354,9 +298,8 @@ static int ldlm_process_flock_lock(struct ldlm_lock *req, __u64 *flags)
 					     &new2->l_remote_handle,
 					     &new2->l_exp_hash);
 		}
-		if (*flags == LDLM_FL_WAIT_NOREPROC)
-			ldlm_lock_addref_internal_nolock(new2,
-							 lock->l_granted_mode);
+		ldlm_lock_addref_internal_nolock(new2,
+						 lock->l_granted_mode);
 
 		/* insert new2 at lock */
 		ldlm_resource_add_lock(res, ownlocks, new2);
@@ -377,22 +320,13 @@ static int ldlm_process_flock_lock(struct ldlm_lock *req, __u64 *flags)
 		ldlm_resource_add_lock(res, ownlocks, req);
 	}
 
-	if (*flags != LDLM_FL_WAIT_NOREPROC) {
-		/* The only one possible case for client-side calls flock
-		 * policy function is ldlm_flock_completion_ast inside which
-		 * carries LDLM_FL_WAIT_NOREPROC flag.
-		 */
-		CERROR("Illegal parameter for client-side-only module.\n");
-		LBUG();
-	}
-
 	/* In case we're reprocessing the requested lock we can't destroy
 	 * it until after calling ldlm_add_ast_work_item() above so that laawi()
 	 * can bump the reference count on \a req. Otherwise \a req
 	 * could be freed before the completion AST can be sent.
 	 */
 	if (added)
-		ldlm_flock_destroy(req, mode, *flags);
+		ldlm_flock_destroy(req, mode, LDLM_FL_WAIT_NOREPROC);
 
 	ldlm_resource_dump(D_INFO, res);
 	return LDLM_ITER_CONTINUE;
@@ -582,12 +516,10 @@ ldlm_flock_completion_ast(struct ldlm_lock *lock, __u64 flags, void *data)
 		getlk->fl_start = (loff_t)lock->l_policy_data.l_flock.start;
 		getlk->fl_end = (loff_t)lock->l_policy_data.l_flock.end;
 	} else {
-		__u64 noreproc = LDLM_FL_WAIT_NOREPROC;
-
 		/* We need to reprocess the lock to do merges or splits
 		 * with existing locks owned by this process.
 		 */
-		ldlm_process_flock_lock(lock, &noreproc);
+		ldlm_process_flock_lock(lock);
 	}
 	unlock_res_and_lock(lock);
 	return rc;


From neilb at suse.com  Tue Jul 18 23:26:47 2017
From: neilb at suse.com (NeilBrown)
Date: Wed, 19 Jul 2017 09:26:47 +1000
Subject: [lustre-devel] [PATCH 08/12] staging: lustre: ldlm: remove unused
 'overlaps' variable
In-Reply-To: <150041997277.20736.17112251996623587423.stgit@noble>
References: <150041997277.20736.17112251996623587423.stgit@noble>
Message-ID: <150042040758.20736.9079474817471825999.stgit@noble>

'overlaps' is never used, only incremented.

Signed-off-by: NeilBrown <neilb at suse.com>
---
 drivers/staging/lustre/lustre/ldlm/ldlm_flock.c |    3 ---
 1 file changed, 3 deletions(-)

diff --git a/drivers/staging/lustre/lustre/ldlm/ldlm_flock.c b/drivers/staging/lustre/lustre/ldlm/ldlm_flock.c
index b6d5a83e61cd..0fb2c882ab4a 100644
--- a/drivers/staging/lustre/lustre/ldlm/ldlm_flock.c
+++ b/drivers/staging/lustre/lustre/ldlm/ldlm_flock.c
@@ -133,7 +133,6 @@ static int ldlm_process_flock_lock(struct ldlm_lock *req)
 	struct ldlm_lock *new2 = NULL;
 	enum ldlm_mode mode = req->l_req_mode;
 	int added = (mode == LCK_NL);
-	int overlaps = 0;
 	int splitted = 0;
 	const struct ldlm_callback_suite null_cbs = { };
 
@@ -226,8 +225,6 @@ static int ldlm_process_flock_lock(struct ldlm_lock *req)
 		    lock->l_policy_data.l_flock.start)
 			break;
 
-		++overlaps;
-
 		if (new->l_policy_data.l_flock.start <=
 		    lock->l_policy_data.l_flock.start) {
 			if (new->l_policy_data.l_flock.end <


From neilb at suse.com  Tue Jul 18 23:26:47 2017
From: neilb at suse.com (NeilBrown)
Date: Wed, 19 Jul 2017 09:26:47 +1000
Subject: [lustre-devel] [PATCH 09/12] staging: lustre: ldlm: remove 'flags'
 arg from ldlm_flock_destroy()
In-Reply-To: <150041997277.20736.17112251996623587423.stgit@noble>
References: <150041997277.20736.17112251996623587423.stgit@noble>
Message-ID: <150042040761.20736.2485410555432492945.stgit@noble>

The only value ever passed in LDLM_FL_WAIT_NOREPROC, so assume that
instead of passing it.

Signed-off-by: NeilBrown <neilb at suse.com>
---
 drivers/staging/lustre/lustre/ldlm/ldlm_flock.c |   36 ++++++++++-------------
 1 file changed, 16 insertions(+), 20 deletions(-)

diff --git a/drivers/staging/lustre/lustre/ldlm/ldlm_flock.c b/drivers/staging/lustre/lustre/ldlm/ldlm_flock.c
index 0fb2c882ab4a..9a888e1ce923 100644
--- a/drivers/staging/lustre/lustre/ldlm/ldlm_flock.c
+++ b/drivers/staging/lustre/lustre/ldlm/ldlm_flock.c
@@ -88,24 +88,23 @@ ldlm_flocks_overlap(struct ldlm_lock *lock, struct ldlm_lock *new)
 }
 
 static inline void
-ldlm_flock_destroy(struct ldlm_lock *lock, enum ldlm_mode mode, __u64 flags)
+ldlm_flock_destroy(struct ldlm_lock *lock, enum ldlm_mode mode)
 {
-	LDLM_DEBUG(lock, "ldlm_flock_destroy(mode: %d, flags: 0x%llx)",
-		   mode, flags);
+	LDLM_DEBUG(lock, "ldlm_flock_destroy(mode: %d)",
+		   mode);
 
 	/* Safe to not lock here, since it should be empty anyway */
 	LASSERT(hlist_unhashed(&lock->l_exp_flock_hash));
 
 	list_del_init(&lock->l_res_link);
-	if (flags == LDLM_FL_WAIT_NOREPROC) {
-		/* client side - set a flag to prevent sending a CANCEL */
-		lock->l_flags |= LDLM_FL_LOCAL_ONLY | LDLM_FL_CBPENDING;
 
-		/* when reaching here, it is under lock_res_and_lock(). Thus,
-		 * need call the nolock version of ldlm_lock_decref_internal
-		 */
-		ldlm_lock_decref_internal_nolock(lock, mode);
-	}
+	/* client side - set a flag to prevent sending a CANCEL */
+	lock->l_flags |= LDLM_FL_LOCAL_ONLY | LDLM_FL_CBPENDING;
+
+	/* when reaching here, it is under lock_res_and_lock(). Thus,
+	 * need call the nolock version of ldlm_lock_decref_internal
+	 */
+	ldlm_lock_decref_internal_nolock(lock, mode);
 
 	ldlm_lock_destroy_nolock(lock);
 }
@@ -208,8 +207,7 @@ static int ldlm_process_flock_lock(struct ldlm_lock *req)
 			}
 
 			if (added) {
-				ldlm_flock_destroy(lock, mode,
-						   LDLM_FL_WAIT_NOREPROC);
+				ldlm_flock_destroy(lock, mode);
 			} else {
 				new = lock;
 				added = 1;
@@ -233,8 +231,7 @@ static int ldlm_process_flock_lock(struct ldlm_lock *req)
 					new->l_policy_data.l_flock.end + 1;
 				break;
 			}
-			ldlm_flock_destroy(lock, lock->l_req_mode,
-					   LDLM_FL_WAIT_NOREPROC);
+			ldlm_flock_destroy(lock, lock->l_req_mode);
 			continue;
 		}
 		if (new->l_policy_data.l_flock.end >=
@@ -265,8 +262,7 @@ static int ldlm_process_flock_lock(struct ldlm_lock *req)
 						NULL, 0, LVB_T_NONE);
 			lock_res_and_lock(req);
 			if (IS_ERR(new2)) {
-				ldlm_flock_destroy(req, lock->l_granted_mode,
-						   LDLM_FL_WAIT_NOREPROC);
+				ldlm_flock_destroy(req, lock->l_granted_mode);
 				return LDLM_ITER_STOP;
 			}
 			goto reprocess;
@@ -323,7 +319,7 @@ static int ldlm_process_flock_lock(struct ldlm_lock *req)
 	 * could be freed before the completion AST can be sent.
 	 */
 	if (added)
-		ldlm_flock_destroy(req, mode, LDLM_FL_WAIT_NOREPROC);
+		ldlm_flock_destroy(req, mode);
 
 	ldlm_resource_dump(D_INFO, res);
 	return LDLM_ITER_CONTINUE;
@@ -477,7 +473,7 @@ ldlm_flock_completion_ast(struct ldlm_lock *lock, __u64 flags, void *data)
 			LDLM_DEBUG(lock, "client-side enqueue deadlock received");
 			rc = -EDEADLK;
 		}
-		ldlm_flock_destroy(lock, mode, LDLM_FL_WAIT_NOREPROC);
+		ldlm_flock_destroy(lock, mode);
 		unlock_res_and_lock(lock);
 
 		/* Need to wake up the waiter if we were evicted */
@@ -498,7 +494,7 @@ ldlm_flock_completion_ast(struct ldlm_lock *lock, __u64 flags, void *data)
 		 * in the lock changes we can decref the appropriate refcount.
 		 */
 		LASSERT(ldlm_is_test_lock(lock));
-		ldlm_flock_destroy(lock, getlk->fl_type, LDLM_FL_WAIT_NOREPROC);
+		ldlm_flock_destroy(lock, getlk->fl_type);
 		switch (lock->l_granted_mode) {
 		case LCK_PR:
 			getlk->fl_type = F_RDLCK;


From neilb at suse.com  Tue Jul 18 23:26:47 2017
From: neilb at suse.com (NeilBrown)
Date: Wed, 19 Jul 2017 09:26:47 +1000
Subject: [lustre-devel] [PATCH 10/12] staging: lustre: ldlm: tidy list
 walking in ldlm_flock()
In-Reply-To: <150041997277.20736.17112251996623587423.stgit@noble>
References: <150041997277.20736.17112251996623587423.stgit@noble>
Message-ID: <150042040763.20736.17969183285195201575.stgit@noble>

Use list_for_each_entry variants to
avoid the explicit list_entry() calls.
This allows us to use list_for_each_entry_safe_from()
instread of adding a local list-walking macro.

Also improve some comments so that it is more obvious
that the locks are sorted per-owner and that we need
to find the insertion point.

Signed-off-by: NeilBrown <neilb at suse.com>
---
 drivers/staging/lustre/lustre/ldlm/ldlm_flock.c |   45 ++++++++++-------------
 1 file changed, 19 insertions(+), 26 deletions(-)

diff --git a/drivers/staging/lustre/lustre/ldlm/ldlm_flock.c b/drivers/staging/lustre/lustre/ldlm/ldlm_flock.c
index 9a888e1ce923..58227728a002 100644
--- a/drivers/staging/lustre/lustre/ldlm/ldlm_flock.c
+++ b/drivers/staging/lustre/lustre/ldlm/ldlm_flock.c
@@ -59,17 +59,6 @@
 #include <linux/list.h>
 #include "ldlm_internal.h"
 
-/**
- * list_for_remaining_safe - iterate over the remaining entries in a list
- *	      and safeguard against removal of a list entry.
- * \param pos   the &struct list_head to use as a loop counter. pos MUST
- *	      have been initialized prior to using it in this macro.
- * \param n     another &struct list_head to use as temporary storage
- * \param head  the head for your list.
- */
-#define list_for_remaining_safe(pos, n, head) \
-	for (n = pos->next; pos != (head); pos = n, n = pos->next)
-
 static inline int
 ldlm_same_flock_owner(struct ldlm_lock *lock, struct ldlm_lock *new)
 {
@@ -125,8 +114,8 @@ static int ldlm_process_flock_lock(struct ldlm_lock *req)
 {
 	struct ldlm_resource *res = req->l_resource;
 	struct ldlm_namespace *ns = ldlm_res_to_ns(res);
-	struct list_head *tmp;
-	struct list_head *ownlocks = NULL;
+	struct ldlm_lock *tmp;
+	struct ldlm_lock *ownlocks = NULL;
 	struct ldlm_lock *lock = NULL;
 	struct ldlm_lock *new = req;
 	struct ldlm_lock *new2 = NULL;
@@ -151,23 +140,23 @@ static int ldlm_process_flock_lock(struct ldlm_lock *req)
 	/* This loop determines where this processes locks start
 	 * in the resource lr_granted list.
 	 */
-	list_for_each(tmp, &res->lr_granted) {
-		lock = list_entry(tmp, struct ldlm_lock,
-				  l_res_link);
+	list_for_each_entry(lock, &res->lr_granted, l_res_link) {
 		if (ldlm_same_flock_owner(lock, req)) {
-			ownlocks = tmp;
+			ownlocks = lock;
 			break;
 		}
 	}
 
-	/* Scan the locks owned by this process that overlap this request.
+	/* Scan the locks owned by this process to find the insertion point
+	 * (as locks are ordered), and to handle overlaps.
 	 * We may have to merge or split existing locks.
 	 */
-	if (!ownlocks)
-		ownlocks = &res->lr_granted;
-
-	list_for_remaining_safe(ownlocks, tmp, &res->lr_granted) {
-		lock = list_entry(ownlocks, struct ldlm_lock, l_res_link);
+	if (ownlocks)
+		lock = ownlocks;
+	else
+		lock = list_entry(&res->lr_granted,
+				  struct ldlm_lock, l_res_link);
+	list_for_each_entry_safe_from(lock, tmp, &res->lr_granted, l_res_link) {
 
 		if (!ldlm_same_flock_owner(lock, new))
 			break;
@@ -295,7 +284,7 @@ static int ldlm_process_flock_lock(struct ldlm_lock *req)
 						 lock->l_granted_mode);
 
 		/* insert new2 at lock */
-		ldlm_resource_add_lock(res, ownlocks, new2);
+		ldlm_resource_add_lock(res, &lock->l_res_link, new2);
 		LDLM_LOCK_RELEASE(new2);
 		break;
 	}
@@ -309,8 +298,12 @@ static int ldlm_process_flock_lock(struct ldlm_lock *req)
 
 	if (!added) {
 		list_del_init(&req->l_res_link);
-		/* insert new lock before ownlocks in list. */
-		ldlm_resource_add_lock(res, ownlocks, req);
+		/* insert new lock before "lock", which might be
+		 * the next lock for this owner, or might be the first
+		 * lock for the next owner, order might not be a lock
+		 * at all, but instead points at the head of the list
+		 */
+		ldlm_resource_add_lock(res, &lock->l_res_link, req);
 	}
 
 	/* In case we're reprocessing the requested lock we can't destroy


From neilb at suse.com  Tue Jul 18 23:26:47 2017
From: neilb at suse.com (NeilBrown)
Date: Wed, 19 Jul 2017 09:26:47 +1000
Subject: [lustre-devel] [PATCH 11/12] staging: lustre: ldlm: remove
 unnecessary 'ownlocks' variable.
In-Reply-To: <150041997277.20736.17112251996623587423.stgit@noble>
References: <150041997277.20736.17112251996623587423.stgit@noble>
Message-ID: <150042040766.20736.11654102832825755225.stgit@noble>

Now that the code has been simplified, 'ownlocks' is not
necessary.

The loop which sets it exits with 'lock' having the same value as
'ownlocks', or point to the head of the list if ownlocks is NULL.

The current code then tests ownlocks and sets 'lock' to exact the
value that it currently has.

So discard 'ownlocks'.

Also remove unnecessary initialization of 'lock'.

Signed-off-by: NeilBrown <neilb at suse.com>
---
 drivers/staging/lustre/lustre/ldlm/ldlm_flock.c |   15 +++------------
 1 file changed, 3 insertions(+), 12 deletions(-)

diff --git a/drivers/staging/lustre/lustre/ldlm/ldlm_flock.c b/drivers/staging/lustre/lustre/ldlm/ldlm_flock.c
index 58227728a002..4e8808103437 100644
--- a/drivers/staging/lustre/lustre/ldlm/ldlm_flock.c
+++ b/drivers/staging/lustre/lustre/ldlm/ldlm_flock.c
@@ -115,8 +115,7 @@ static int ldlm_process_flock_lock(struct ldlm_lock *req)
 	struct ldlm_resource *res = req->l_resource;
 	struct ldlm_namespace *ns = ldlm_res_to_ns(res);
 	struct ldlm_lock *tmp;
-	struct ldlm_lock *ownlocks = NULL;
-	struct ldlm_lock *lock = NULL;
+	struct ldlm_lock *lock;
 	struct ldlm_lock *new = req;
 	struct ldlm_lock *new2 = NULL;
 	enum ldlm_mode mode = req->l_req_mode;
@@ -140,22 +139,14 @@ static int ldlm_process_flock_lock(struct ldlm_lock *req)
 	/* This loop determines where this processes locks start
 	 * in the resource lr_granted list.
 	 */
-	list_for_each_entry(lock, &res->lr_granted, l_res_link) {
-		if (ldlm_same_flock_owner(lock, req)) {
-			ownlocks = lock;
+	list_for_each_entry(lock, &res->lr_granted, l_res_link)
+		if (ldlm_same_flock_owner(lock, req))
 			break;
-		}
-	}
 
 	/* Scan the locks owned by this process to find the insertion point
 	 * (as locks are ordered), and to handle overlaps.
 	 * We may have to merge or split existing locks.
 	 */
-	if (ownlocks)
-		lock = ownlocks;
-	else
-		lock = list_entry(&res->lr_granted,
-				  struct ldlm_lock, l_res_link);
 	list_for_each_entry_safe_from(lock, tmp, &res->lr_granted, l_res_link) {
 
 		if (!ldlm_same_flock_owner(lock, new))


From neilb at suse.com  Tue Jul 18 23:26:47 2017
From: neilb at suse.com (NeilBrown)
Date: Wed, 19 Jul 2017 09:26:47 +1000
Subject: [lustre-devel] [PATCH 12/12] staging: lustre: ldlm: remove unused
 field 'fwd_generation'
In-Reply-To: <150041997277.20736.17112251996623587423.stgit@noble>
References: <150041997277.20736.17112251996623587423.stgit@noble>
Message-ID: <150042040769.20736.8915983949201116682.stgit@noble>

With this field gone, we don't need local variables 'imp' or 'obd'
any more.

Signed-off-by: NeilBrown <neilb at suse.com>
---
 drivers/staging/lustre/lustre/ldlm/ldlm_flock.c |   21 +++------------------
 1 file changed, 3 insertions(+), 18 deletions(-)

diff --git a/drivers/staging/lustre/lustre/ldlm/ldlm_flock.c b/drivers/staging/lustre/lustre/ldlm/ldlm_flock.c
index 4e8808103437..b72a195d9635 100644
--- a/drivers/staging/lustre/lustre/ldlm/ldlm_flock.c
+++ b/drivers/staging/lustre/lustre/ldlm/ldlm_flock.c
@@ -311,7 +311,6 @@ static int ldlm_process_flock_lock(struct ldlm_lock *req)
 
 struct ldlm_flock_wait_data {
 	struct ldlm_lock *fwd_lock;
-	int	       fwd_generation;
 };
 
 static void
@@ -342,11 +341,9 @@ int
 ldlm_flock_completion_ast(struct ldlm_lock *lock, __u64 flags, void *data)
 {
 	struct file_lock		*getlk = lock->l_ast_data;
-	struct obd_device	      *obd;
-	struct obd_import	      *imp = NULL;
-	struct ldlm_flock_wait_data     fwd;
-	struct l_wait_info	      lwi;
-	int			     rc = 0;
+	struct ldlm_flock_wait_data	fwd;
+	struct l_wait_info		lwi;
+	int				rc = 0;
 
 	OBD_FAIL_TIMEOUT(OBD_FAIL_LDLM_CP_CB_WAIT2, 4);
 	if (OBD_FAIL_PRECHECK(OBD_FAIL_LDLM_CP_CB_WAIT3)) {
@@ -374,18 +371,6 @@ ldlm_flock_completion_ast(struct ldlm_lock *lock, __u64 flags, void *data)
 
 	LDLM_DEBUG(lock, "client-side enqueue returned a blocked lock, sleeping");
 	fwd.fwd_lock = lock;
-	obd = class_exp2obd(lock->l_conn_export);
-
-	/* if this is a local lock, there is no import */
-	if (obd)
-		imp = obd->u.cli.cl_import;
-
-	if (imp) {
-		spin_lock(&imp->imp_lock);
-		fwd.fwd_generation = imp->imp_generation;
-		spin_unlock(&imp->imp_lock);
-	}
-
 	lwi = LWI_TIMEOUT_INTR(0, NULL, ldlm_flock_interrupted_wait, &fwd);
 
 	/* Go to sleep until the lock is granted. */


From paf at cray.com  Wed Jul 19 01:54:27 2017
From: paf at cray.com (Patrick Farrell)
Date: Wed, 19 Jul 2017 01:54:27 +0000
Subject: [lustre-devel] [PATCH 10/12] staging: lustre: ldlm: tidy list
 walking in ldlm_flock()
In-Reply-To: <150042040763.20736.17969183285195201575.stgit@noble>
References: <150041997277.20736.17112251996623587423.stgit@noble>,
	<150042040763.20736.17969183285195201575.stgit@noble>
Message-ID: <BN6PR1101MB2132359E48C9904D3E305EE7CBA60@BN6PR1101MB2132.namprd11.prod.outlook.com>


Neil,

Minor...
"order might not be a lock" looks like it should say "or"?

- Patrick
________________________________
From: lustre-devel <lustre-devel-bounces at lists.lustre.org> on behalf of NeilBrown <neilb at suse.com>
Sent: Tuesday, July 18, 2017 6:26:47 PM
To: Oleg Drokin; Greg Kroah-Hartman; Andreas Dilger
Cc: Linux Kernel Mailing List; Lustre Development List
Subject: [lustre-devel] [PATCH 10/12] staging: lustre: ldlm: tidy list walking in ldlm_flock()

Use list_for_each_entry variants to
avoid the explicit list_entry() calls.
This allows us to use list_for_each_entry_safe_from()
instread of adding a local list-walking macro.

Also improve some comments so that it is more obvious
that the locks are sorted per-owner and that we need
to find the insertion point.

Signed-off-by: NeilBrown <neilb at suse.com>
---
 drivers/staging/lustre/lustre/ldlm/ldlm_flock.c |   45 ++++++++++-------------
 1 file changed, 19 insertions(+), 26 deletions(-)

diff --git a/drivers/staging/lustre/lustre/ldlm/ldlm_flock.c b/drivers/staging/lustre/lustre/ldlm/ldlm_flock.c
index 9a888e1ce923..58227728a002 100644
--- a/drivers/staging/lustre/lustre/ldlm/ldlm_flock.c
+++ b/drivers/staging/lustre/lustre/ldlm/ldlm_flock.c
@@ -59,17 +59,6 @@
 #include <linux/list.h>
 #include "ldlm_internal.h"

-/**
- * list_for_remaining_safe - iterate over the remaining entries in a list
- *           and safeguard against removal of a list entry.
- * \param pos   the &struct list_head to use as a loop counter. pos MUST
- *           have been initialized prior to using it in this macro.
- * \param n     another &struct list_head to use as temporary storage
- * \param head  the head for your list.
- */
-#define list_for_remaining_safe(pos, n, head) \
-       for (n = pos->next; pos != (head); pos = n, n = pos->next)
-
 static inline int
 ldlm_same_flock_owner(struct ldlm_lock *lock, struct ldlm_lock *new)
 {
@@ -125,8 +114,8 @@ static int ldlm_process_flock_lock(struct ldlm_lock *req)
 {
         struct ldlm_resource *res = req->l_resource;
         struct ldlm_namespace *ns = ldlm_res_to_ns(res);
-       struct list_head *tmp;
-       struct list_head *ownlocks = NULL;
+       struct ldlm_lock *tmp;
+       struct ldlm_lock *ownlocks = NULL;
         struct ldlm_lock *lock = NULL;
         struct ldlm_lock *new = req;
         struct ldlm_lock *new2 = NULL;
@@ -151,23 +140,23 @@ static int ldlm_process_flock_lock(struct ldlm_lock *req)
         /* This loop determines where this processes locks start
          * in the resource lr_granted list.
          */
-       list_for_each(tmp, &res->lr_granted) {
-               lock = list_entry(tmp, struct ldlm_lock,
-                                 l_res_link);
+       list_for_each_entry(lock, &res->lr_granted, l_res_link) {
                 if (ldlm_same_flock_owner(lock, req)) {
-                       ownlocks = tmp;
+                       ownlocks = lock;
                         break;
                 }
         }

-       /* Scan the locks owned by this process that overlap this request.
+       /* Scan the locks owned by this process to find the insertion point
+        * (as locks are ordered), and to handle overlaps.
          * We may have to merge or split existing locks.
          */
-       if (!ownlocks)
-               ownlocks = &res->lr_granted;
-
-       list_for_remaining_safe(ownlocks, tmp, &res->lr_granted) {
-               lock = list_entry(ownlocks, struct ldlm_lock, l_res_link);
+       if (ownlocks)
+               lock = ownlocks;
+       else
+               lock = list_entry(&res->lr_granted,
+                                 struct ldlm_lock, l_res_link);
+       list_for_each_entry_safe_from(lock, tmp, &res->lr_granted, l_res_link) {

                 if (!ldlm_same_flock_owner(lock, new))
                         break;
@@ -295,7 +284,7 @@ static int ldlm_process_flock_lock(struct ldlm_lock *req)
                                                  lock->l_granted_mode);

                 /* insert new2 at lock */
-               ldlm_resource_add_lock(res, ownlocks, new2);
+               ldlm_resource_add_lock(res, &lock->l_res_link, new2);
                 LDLM_LOCK_RELEASE(new2);
                 break;
         }
@@ -309,8 +298,12 @@ static int ldlm_process_flock_lock(struct ldlm_lock *req)

         if (!added) {
                 list_del_init(&req->l_res_link);
-               /* insert new lock before ownlocks in list. */
-               ldlm_resource_add_lock(res, ownlocks, req);
+               /* insert new lock before "lock", which might be
+                * the next lock for this owner, or might be the first
+                * lock for the next owner, order might not be a lock
+                * at all, but instead points at the head of the list
+                */
+               ldlm_resource_add_lock(res, &lock->l_res_link, req);
         }

         /* In case we're reprocessing the requested lock we can't destroy


_______________________________________________
lustre-devel mailing list
lustre-devel at lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-devel-lustre.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-devel_lists.lustre.org/attachments/20170719/087655cb/attachment.html>

From oleg.drokin at intel.com  Wed Jul 19 02:51:34 2017
From: oleg.drokin at intel.com (Oleg Drokin)
Date: Tue, 18 Jul 2017 22:51:34 -0400
Subject: [lustre-devel] [PATCH 03/12] staging: lustre: llite: fix
	various issues with ll_splice_alias.
In-Reply-To: <150042040744.20736.6110134708456645941.stgit@noble>
References: <150041997277.20736.17112251996623587423.stgit@noble>
	<150042040744.20736.6110134708456645941.stgit@noble>
Message-ID: <030281CD-CC74-429C-910F-86706CFC57DA@intel.com>

Unfortunately this patch causes insta-crash on first stat call after mount.
Sorry, I cannot dig into this deeper right this moment, but I will a bit later.
I am adding Al that we discussed this code at some length and he found no problems
here, so I am a bit surprised by your findings.
Also the reason we reinvent the d_splice_alias is because we need to
splice not just directories, but also regular files.

I also am less sure by your previous DCACHE_DISCONECTED patch that we in fact might
still need, I just need to dig up a test case for that.

Thanks for looking into it!

[  170.000858] Lustre: Mounted lustre-client
[  172.799813] Lustre: DEBUG MARKER: Using TIMEOUT=20
[  186.627954] BUG: unable to handle kernel NULL pointer dereference at 00000000000000a8
[  186.628742] IP: __lock_acquire+0x125/0x1370
[  186.629137] PGD 0 
[  186.629138] P4D 0 
[  186.629496] 
[  186.630216] Oops: 0000 [#1] SMP DEBUG_PAGEALLOC
[  186.630613] Modules linked in: osc(C) mgc(C) lustre(C) lmv(C) fld(C) mdc(C) fid(C) lov(C) ksocklnd(C) ptlrpc(C) obdclass(C) lnet(C) sha512_ssse3 sha512_generic crc32_generic libcfs(C) joydev pcspkr i2c_piix4 virtio_console rpcsec_gss_krb5 ttm drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm serio_raw virtio_blk floppy
[  186.633238] CPU: 2 PID: 10897 Comm: sh Tainted: G         C      4.13.0-rc1-vm-nfs+ #154
[  186.633990] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011
[  186.634402] task: ffff8800d6a1c180 task.stack: ffffc900066a0000
[  186.634807] RIP: 0010:__lock_acquire+0x125/0x1370
[  186.635206] RSP: 0018:ffffc900066a3750 EFLAGS: 00010002
[  186.635597] RAX: 0000000000000046 RBX: 0000000000000001 RCX: 0000000000000000
[  186.636013] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 0000000000000000
[  186.636432] RBP: ffffc900066a3810 R08: ffffffffa05221a9 R09: 0000000000000000
[  186.636844] R10: 0000000000000000 R11: ffff8800d6a1c180 R12: 0000000000000001
[  186.637264] R13: 0000000000000000 R14: 0000000000000001 R15: 00000000000000a8
[  186.637679] FS:  00007fd679eaa700(0000) GS:ffff88011a200000(0000) knlGS:0000000000000000
[  186.638402] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  186.638803] CR2: 00000000000000a8 CR3: 0000000109c0b000 CR4: 00000000000006e0
[  186.639228] Call Trace:
[  186.639589]  lock_acquire+0xe3/0x1d0
[  186.639962]  ? lock_acquire+0xe3/0x1d0
[  186.640352]  ? ll_lookup_it_finish+0x379/0xca0 [lustre]
[  186.640747]  _raw_spin_lock+0x34/0x70
[  186.641130]  ? ll_lookup_it_finish+0x379/0xca0 [lustre]
[  186.641529]  ll_lookup_it_finish+0x379/0xca0 [lustre]
[  186.641943]  ? req_capsule_server_get+0x15/0x20 [ptlrpc]
[  186.642368]  ? lmv_revalidate_slaves+0x790/0x790 [lmv]
[  186.642779]  ll_lookup_it+0x26d/0x820 [lustre]
[  186.643175]  ll_lookup_nd+0x162/0x1a0 [lustre]
[  186.643575]  lookup_slow+0x132/0x220
[  186.643947]  ? __wake_up+0x23/0x50
[  186.644322]  walk_component+0x1bf/0x350
[  186.644714]  link_path_walk+0x1b8/0x630
[  186.645097]  path_lookupat+0x99/0x220
[  186.645459]  ? __kernel_map_pages+0x131/0x140
[  186.645830]  ? __kernel_map_pages+0x131/0x140
[  186.646206]  filename_lookup+0xb8/0x1a0
[  186.646575]  ? __check_object_size+0xb1/0x1a0
[  186.646952]  ? strncpy_from_user+0x4d/0x160
[  186.647326]  user_path_at_empty+0x36/0x40
[  186.647694]  ? user_path_at_empty+0x36/0x40
[  186.648068]  vfs_statx+0x76/0xe0
[  186.648429]  SYSC_newstat+0x3d/0x70
[  186.648789]  ? trace_hardirqs_on_caller+0xf4/0x190
[  186.649171]  ? trace_hardirqs_on_thunk+0x1a/0x1c
[  186.649547]  SyS_newstat+0xe/0x10
[  186.649906]  entry_SYSCALL_64_fastpath+0x1f/0xbe
[  186.650289] RIP: 0033:0x7fd679599475
[  186.650651] RSP: 002b:00007ffc2e568fc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000004
[  186.651350] RAX: ffffffffffffffda RBX: 00007fd679862ae0 RCX: 00007fd679599475
[  186.651758] RDX: 00007ffc2e568fe0 RSI: 00007ffc2e568fe0 RDI: 000000eec2877730
[  186.652171] RBP: 00007fd679862ae0 R08: 000000eec2cafcb0 R09: 0000000000000008
[  186.652579] R10: 000000eec2cafcb0 R11: 0000000000000246 R12: 0000000000000020
[  186.652982] R13: 000000eec2cc7e10 R14: 000000eec2cb2190 R15: 00007ffc2e5690f8
[  186.653393] Code: c8 65 48 33 3c 25 28 00 00 00 44 89 e0 0f 85 91 0d 00 00 48 81 c4 90 00 00 00 5b 41 5a 41 5c 41 5d 41 5e 41 5f 5d 49 8d 62 f8 c3 <49> 81 3f 00 90 6f 82 41 bc 00 00 00 00 44 0f 45 e2 83 fe 01 0f 
[  186.654536] RIP: __lock_acquire+0x125/0x1370 RSP: ffffc900066a3750
[  186.654933] CR2: 00000000000000a8


On Jul 18, 2017, at 7:26 PM, NeilBrown wrote:

> 1/ The testing of DCACHE_DISCONNECTED is wrong.
>  see upstream commit da093a9b76ef ("dcache: d_splice_alias should
>  ignore DCACHE_DISCONNECTED")
> 
>  As this is a notoriously difficult piece of code to get right,
>  it makes sense to use d_splice_alias() directly and no try to
>  create a local version of it.
> 
> 2/ ll_find_alias() currently:
>     locks and alias
>     checks that it is the one we want
>     unlock it
>     locks it again
>     gets a reference
>     unlocks it
> 
>   This isn't safe.  Anything could happen to the dentry while we
>   don't hold a reference.  We need to dget the reference while
>   still holding the lock.
> 
> 3/ The d_move() in ll_splice_alias() is pointless.  We have
>    already checked the hash, name, and parent are the same, and
>    these are the only fields that d_move() will change.
> 
> 4/ The call to d_add() is outside of any locking. This makes it
>   possible for two identical dentries to be added to the same
>   inode, which would cause confusion.
> 
>   Prior to 4.7, i_mutex would have provided exclusion, but since
>   the VFS supports parallel lookups, only a shared lock is held
>   on i_mutex.
> 
>   Because ll_d_init() creates a dentry in a state where
>   ll_dcompare will no recognize it, the VFS provides no guarantee
>   that we won't have two concurrent calls to ll_lookup_dn() for
>   the same parent/name.
> 
> 
> So: rename ll_find_alias() to ll_find_invalid_alias() and have it
> just focus on finding an invalid alias.
> 
> For directories, we can just use d__splice_alias() directly.
> There must only be one alias for a directory, and
> ll_splice_alias() will find it where it is "invalid" or not.
> 
> For non-directories, we call ll_find_invalid_alias(), and either
> use the result or call d_add().  We need a lock to protect from
> races with other threads calling ll_find_invalid_alias() and
> d_add() at the same time. lli_lock seems suitable for this
> purpose.
> 
> Signed-off-by: NeilBrown <neilb at suse.com>
> ---
> drivers/staging/lustre/lustre/llite/namei.c |   69 +++++++++++++--------------
> 1 file changed, 34 insertions(+), 35 deletions(-)
> 
> diff --git a/drivers/staging/lustre/lustre/llite/namei.c b/drivers/staging/lustre/lustre/llite/namei.c
> index 293a3180ec70..6204c3e70d45 100644
> --- a/drivers/staging/lustre/lustre/llite/namei.c
> +++ b/drivers/staging/lustre/lustre/llite/namei.c
> @@ -378,75 +378,74 @@ void ll_i2gids(__u32 *suppgids, struct inode *i1, struct inode *i2)
> }
> 
> /*
> - * try to reuse three types of dentry:
> - * 1. unhashed alias, this one is unhashed by d_invalidate (but it may be valid
> - *    by concurrent .revalidate).
> - * 2. INVALID alias (common case for no valid ldlm lock held, but this flag may
> - *    be cleared by others calling d_lustre_revalidate).
> - * 3. DISCONNECTED alias.
> + * Try to find an "invalid" alias.  i.e. one that was unhashed by
> + * d_invalidate(), or that was instantiated with no valid ldlm lock.
> + * These can be rehased by d_lustre_revalidate(), which could race
> + * with this code.
>  */
> -static struct dentry *ll_find_alias(struct inode *inode, struct dentry *dentry)
> +static struct dentry *ll_find_invalid_alias(struct inode *inode,
> +					    struct dentry *dentry)
> {
> -	struct dentry *alias, *discon_alias, *invalid_alias;
> +	struct dentry *alias, *invalid_alias = NULL;
> 
> 	if (hlist_empty(&inode->i_dentry))
> 		return NULL;
> 
> -	discon_alias = NULL;
> -	invalid_alias = NULL;
> -
> 	spin_lock(&inode->i_lock);
> 	hlist_for_each_entry(alias, &inode->i_dentry, d_u.d_alias) {
> 		LASSERT(alias != dentry);
> 
> 		spin_lock(&alias->d_lock);
> -		if ((alias->d_flags & DCACHE_DISCONNECTED) &&
> -		    S_ISDIR(inode->i_mode))
> -			/* LASSERT(last_discon == NULL); LU-405, bz 20055 */
> -			discon_alias = alias;
> -		else if (alias->d_parent == dentry->d_parent	     &&
> -			 alias->d_name.hash == dentry->d_name.hash       &&
> -			 alias->d_name.len == dentry->d_name.len	 &&
> -			 memcmp(alias->d_name.name, dentry->d_name.name,
> -				dentry->d_name.len) == 0)
> +		if (alias->d_parent == dentry->d_parent       &&
> +		    alias->d_name.hash == dentry->d_name.hash &&
> +		    alias->d_name.len == dentry->d_name.len   &&
> +		    memcmp(alias->d_name.name, dentry->d_name.name,
> +			   dentry->d_name.len) == 0) {
> +			dget_dlock(alias);
> 			invalid_alias = alias;
> +		}
> 		spin_unlock(&alias->d_lock);
> 
> 		if (invalid_alias)
> 			break;
> 	}
> -	alias = invalid_alias ?: discon_alias ?: NULL;
> -	if (alias) {
> -		spin_lock(&alias->d_lock);
> -		dget_dlock(alias);
> -		spin_unlock(&alias->d_lock);
> -	}
> 	spin_unlock(&inode->i_lock);
> 
> -	return alias;
> +	return invalid_alias;
> }
> 
> /*
> - * Similar to d_splice_alias(), but lustre treats invalid alias
> - * similar to DCACHE_DISCONNECTED, and tries to use it anyway.
> + * Similar to d_splice_alias(), but also look for an "invalid" alias,
> + * specific to lustre, and use that if found.
>  */
> struct dentry *ll_splice_alias(struct inode *inode, struct dentry *de)
> {
> -	if (inode) {
> -		struct dentry *new = ll_find_alias(inode, de);
> +	if (inode && !S_ISDIR(inode->i_mode)) {
> +		struct ll_inode_info *lli = ll_i2info(inode);
> +		struct dentry *new;
> +
> +		/* We need lli_lock here as another thread could
> +		 * be running this code, and i_lock cannot protect us.
> +		 */
> +		spin_lock(&lli->lli_lock);
> +		new = ll_find_invalid_alias(inode, de);
> +		if (!new)
> +			d_add(de, inode);
> +		spin_lock(&lli->lli_lock);
> 
> 		if (new) {
> -			d_move(new, de);
> 			iput(inode);
> 			CDEBUG(D_DENTRY,
> 			       "Reuse dentry %p inode %p refc %d flags %#x\n",
> 			      new, d_inode(new), d_count(new), new->d_flags);
> 			return new;
> 		}
> +		return de;
> 	}
> -	d_add(de, inode);
> -	CDEBUG(D_DENTRY, "Add dentry %p inode %p refc %d flags %#x\n",
> -	       de, d_inode(de), d_count(de), de->d_flags);
> +	de = d_splice_alias(inode, de);
> +	if (!IS_ERR(de))
> +		CDEBUG(D_DENTRY, "Add dentry %p inode %p refc %d flags %#x\n",
> +		       de, d_inode(de), d_count(de), de->d_flags);
> 	return de;
> }
> 
> 

From neilb at suse.com  Wed Jul 19 04:33:33 2017
From: neilb at suse.com (NeilBrown)
Date: Wed, 19 Jul 2017 14:33:33 +1000
Subject: [lustre-devel] [PATCH 03/12] staging: lustre: llite: fix
	various issues with ll_splice_alias.
In-Reply-To: <030281CD-CC74-429C-910F-86706CFC57DA@intel.com>
References: <150041997277.20736.17112251996623587423.stgit@noble>
	<150042040744.20736.6110134708456645941.stgit@noble>
	<030281CD-CC74-429C-910F-86706CFC57DA@intel.com>
Message-ID: <87o9shxcyq.fsf@notabene.neil.brown.name>

On Tue, Jul 18 2017, Oleg Drokin wrote:

> Unfortunately this patch causes insta-crash on first stat call after mount.
> Sorry, I cannot dig into this deeper right this moment, but I will a bit later.

V.strange.  The crash suggests that the lock, and hence the inode, is
not initialized.  I cannot see how that might happen.
though...

>> +		spin_lock(&lli->lli_lock);
>> +		new = ll_find_invalid_alias(inode, de);
>> +		if (!new)
>> +			d_add(de, inode);
>> +		spin_lock(&lli->lli_lock);

Had it not crashed, it would have deadlocked.  That second spin_lock()
should be spin_unlock() :-( I don't *think* that would have caused this crash...

> I am adding Al that we discussed this code at some length and he found no problems
> here, so I am a bit surprised by your findings.

I'd be very happy to read Al's thoughts.

> Also the reason we reinvent the d_splice_alias is because we need to
> splice not just directories, but also regular files.

I see that.  A key simplification I bring is that directories and
non-directories can be handled separately.  d_splice_alias() does
all we need for directories, and nothing useful for non-dirs.

Thanks,
NeilBrown
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 832 bytes
Desc: not available
URL: <http://lists.lustre.org/pipermail/lustre-devel_lists.lustre.org/attachments/20170719/3c6e7ef8/attachment.sig>

From neilb at suse.com  Wed Jul 19 04:36:07 2017
From: neilb at suse.com (NeilBrown)
Date: Wed, 19 Jul 2017 14:36:07 +1000
Subject: [lustre-devel] [PATCH 10/12] staging: lustre: ldlm: tidy list
	walking in ldlm_flock()
In-Reply-To: <BN6PR1101MB2132359E48C9904D3E305EE7CBA60@BN6PR1101MB2132.namprd11.prod.outlook.com>
References: <150041997277.20736.17112251996623587423.stgit@noble>
	<150042040763.20736.17969183285195201575.stgit@noble>
	<BN6PR1101MB2132359E48C9904D3E305EE7CBA60@BN6PR1101MB2132.namprd11.prod.outlook.com>
Message-ID: <87lgnlxcug.fsf@notabene.neil.brown.name>

On Wed, Jul 19 2017, Patrick Farrell wrote:

> Neil,
>
> Minor...
> "order might not be a lock" looks like it should say "or"?

Yes: s/order/or/ as you say.

Thanks,
NeilBrown


>
> - Patrick
> ________________________________
> From: lustre-devel <lustre-devel-bounces at lists.lustre.org> on behalf of NeilBrown <neilb at suse.com>
> Sent: Tuesday, July 18, 2017 6:26:47 PM
> To: Oleg Drokin; Greg Kroah-Hartman; Andreas Dilger
> Cc: Linux Kernel Mailing List; Lustre Development List
> Subject: [lustre-devel] [PATCH 10/12] staging: lustre: ldlm: tidy list walking in ldlm_flock()
>
> Use list_for_each_entry variants to
> avoid the explicit list_entry() calls.
> This allows us to use list_for_each_entry_safe_from()
> instread of adding a local list-walking macro.
>
> Also improve some comments so that it is more obvious
> that the locks are sorted per-owner and that we need
> to find the insertion point.
>
> Signed-off-by: NeilBrown <neilb at suse.com>
> ---
>  drivers/staging/lustre/lustre/ldlm/ldlm_flock.c |   45 ++++++++++-------------
>  1 file changed, 19 insertions(+), 26 deletions(-)
>
> diff --git a/drivers/staging/lustre/lustre/ldlm/ldlm_flock.c b/drivers/staging/lustre/lustre/ldlm/ldlm_flock.c
> index 9a888e1ce923..58227728a002 100644
> --- a/drivers/staging/lustre/lustre/ldlm/ldlm_flock.c
> +++ b/drivers/staging/lustre/lustre/ldlm/ldlm_flock.c
> @@ -59,17 +59,6 @@
>  #include <linux/list.h>
>  #include "ldlm_internal.h"
>
> -/**
> - * list_for_remaining_safe - iterate over the remaining entries in a list
> - *           and safeguard against removal of a list entry.
> - * \param pos   the &struct list_head to use as a loop counter. pos MUST
> - *           have been initialized prior to using it in this macro.
> - * \param n     another &struct list_head to use as temporary storage
> - * \param head  the head for your list.
> - */
> -#define list_for_remaining_safe(pos, n, head) \
> -       for (n = pos->next; pos != (head); pos = n, n = pos->next)
> -
>  static inline int
>  ldlm_same_flock_owner(struct ldlm_lock *lock, struct ldlm_lock *new)
>  {
> @@ -125,8 +114,8 @@ static int ldlm_process_flock_lock(struct ldlm_lock *req)
>  {
>          struct ldlm_resource *res = req->l_resource;
>          struct ldlm_namespace *ns = ldlm_res_to_ns(res);
> -       struct list_head *tmp;
> -       struct list_head *ownlocks = NULL;
> +       struct ldlm_lock *tmp;
> +       struct ldlm_lock *ownlocks = NULL;
>          struct ldlm_lock *lock = NULL;
>          struct ldlm_lock *new = req;
>          struct ldlm_lock *new2 = NULL;
> @@ -151,23 +140,23 @@ static int ldlm_process_flock_lock(struct ldlm_lock *req)
>          /* This loop determines where this processes locks start
>           * in the resource lr_granted list.
>           */
> -       list_for_each(tmp, &res->lr_granted) {
> -               lock = list_entry(tmp, struct ldlm_lock,
> -                                 l_res_link);
> +       list_for_each_entry(lock, &res->lr_granted, l_res_link) {
>                  if (ldlm_same_flock_owner(lock, req)) {
> -                       ownlocks = tmp;
> +                       ownlocks = lock;
>                          break;
>                  }
>          }
>
> -       /* Scan the locks owned by this process that overlap this request.
> +       /* Scan the locks owned by this process to find the insertion point
> +        * (as locks are ordered), and to handle overlaps.
>           * We may have to merge or split existing locks.
>           */
> -       if (!ownlocks)
> -               ownlocks = &res->lr_granted;
> -
> -       list_for_remaining_safe(ownlocks, tmp, &res->lr_granted) {
> -               lock = list_entry(ownlocks, struct ldlm_lock, l_res_link);
> +       if (ownlocks)
> +               lock = ownlocks;
> +       else
> +               lock = list_entry(&res->lr_granted,
> +                                 struct ldlm_lock, l_res_link);
> +       list_for_each_entry_safe_from(lock, tmp, &res->lr_granted, l_res_link) {
>
>                  if (!ldlm_same_flock_owner(lock, new))
>                          break;
> @@ -295,7 +284,7 @@ static int ldlm_process_flock_lock(struct ldlm_lock *req)
>                                                   lock->l_granted_mode);
>
>                  /* insert new2 at lock */
> -               ldlm_resource_add_lock(res, ownlocks, new2);
> +               ldlm_resource_add_lock(res, &lock->l_res_link, new2);
>                  LDLM_LOCK_RELEASE(new2);
>                  break;
>          }
> @@ -309,8 +298,12 @@ static int ldlm_process_flock_lock(struct ldlm_lock *req)
>
>          if (!added) {
>                  list_del_init(&req->l_res_link);
> -               /* insert new lock before ownlocks in list. */
> -               ldlm_resource_add_lock(res, ownlocks, req);
> +               /* insert new lock before "lock", which might be
> +                * the next lock for this owner, or might be the first
> +                * lock for the next owner, order might not be a lock
> +                * at all, but instead points at the head of the list
> +                */
> +               ldlm_resource_add_lock(res, &lock->l_res_link, req);
>          }
>
>          /* In case we're reprocessing the requested lock we can't destroy
>
>
> _______________________________________________
> lustre-devel mailing list
> lustre-devel at lists.lustre.org
> http://lists.lustre.org/listinfo.cgi/lustre-devel-lustre.org
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 832 bytes
Desc: not available
URL: <http://lists.lustre.org/pipermail/lustre-devel_lists.lustre.org/attachments/20170719/7f9a98d2/attachment.sig>

From oleg.drokin at intel.com  Wed Jul 19 06:18:39 2017
From: oleg.drokin at intel.com (Oleg Drokin)
Date: Wed, 19 Jul 2017 02:18:39 -0400
Subject: [lustre-devel] [PATCH 03/12] staging: lustre: llite: fix
	various issues with ll_splice_alias.
In-Reply-To: <87o9shxcyq.fsf@notabene.neil.brown.name>
References: <150041997277.20736.17112251996623587423.stgit@noble>
	<150042040744.20736.6110134708456645941.stgit@noble>
	<030281CD-CC74-429C-910F-86706CFC57DA@intel.com>
	<87o9shxcyq.fsf@notabene.neil.brown.name>
Message-ID: <244467F8-2775-4014-90FF-9CBC74F76B38@intel.com>


On Jul 19, 2017, at 12:33 AM, NeilBrown wrote:

> On Tue, Jul 18 2017, Oleg Drokin wrote:
> 
>> Unfortunately this patch causes insta-crash on first stat call after mount.
>> Sorry, I cannot dig into this deeper right this moment, but I will a bit later.
> 
> V.strange.  The crash suggests that the lock, and hence the inode, is
> not initialized.  I cannot see how that might happen.
> though...
> 
>>> +		spin_lock(&lli->lli_lock);
>>> +		new = ll_find_invalid_alias(inode, de);
>>> +		if (!new)
>>> +			d_add(de, inode);
>>> +		spin_lock(&lli->lli_lock);
> 
> Had it not crashed, it would have deadlocked.  That second spin_lock()
> should be spin_unlock() :-( I don't *think* that would have caused this crash…

No, that's not it.
-	d_add(de, inode);
-	CDEBUG(D_DENTRY, "Add dentry %p inode %p refc %d flags %#x\n",
-	       de, d_inode(de), d_count(de), de->d_flags);
+	de = d_splice_alias(inode, de);
+	if (!IS_ERR(de))
+		CDEBUG(D_DENTRY, "Add dentry %p inode %p refc %d flags %#x\n",
+		       de, d_inode(de), d_count(de), de->d_flags);
	return de;

This is likely it.
d_splice_alias would return NULL if there's no alias, after d_add of the de.
But ll_splice_alias callers expect either an ERR_PTR on error or dentry to use
otherwise.
So in ll_lookup_it_finish() we have:
        alias = ll_splice_alias(inode, *de);
        if (IS_ERR(alias)) {
                rc = PTR_ERR(alias);
                goto out;
        }
        *de = alias;
…
                if (md_revalidate_lock(ll_i2mdexp(parent), &parent_it, &fid,
                                       NULL)) {
===>>> whoops!          d_lustre_revalidate(*de);
                        ll_intent_release(&parent_it);
                }

>> I am adding Al that we discussed this code at some length and he found no problems
>> here, so I am a bit surprised by your findings.
> I'd be very happy to read Al's thoughts.

Some of the discussions were in lkml under subject of "More parallel
atomic_open/d_splice_alias fun with NFS and possibly more FSes"
in July 2016, in between nfs stuff.
There was more but I cannot readily find it.

>> Also the reason we reinvent the d_splice_alias is because we need to
>> splice not just directories, but also regular files.
> 
> I see that.  A key simplification I bring is that directories and
> non-directories can be handled separately.  d_splice_alias() does
> all we need for directories, and nothing useful for non-dirs.

I see.
I still need to  think some more about this whole thing.

Please see commit 99f1c013194e64d4b67d5d318148303b0e1585e1 for double d_add
of the same dentry fix.6

From gregkh at linuxfoundation.org  Wed Jul 19 08:29:55 2017
From: gregkh at linuxfoundation.org (Greg KH)
Date: Wed, 19 Jul 2017 10:29:55 +0200
Subject: [lustre-devel] [PATCH 01/12] staging: lustre: fix minor typos
	in comments
In-Reply-To: <150042040735.20736.6997246449635538567.stgit@noble>
References: <150041997277.20736.17112251996623587423.stgit@noble>
	<150042040735.20736.6997246449635538567.stgit@noble>
Message-ID: <20170719082955.GA30088@kroah.com>

On Wed, Jul 19, 2017 at 09:26:47AM +1000, NeilBrown wrote:
> Signed-off-by: NeilBrown <neilb at suse.com>
> ---
>  drivers/staging/lustre/lustre/llite/namei.c   |    2 +-
>  drivers/staging/lustre/lustre/mdc/mdc_locks.c |    4 ++--
>  2 files changed, 3 insertions(+), 3 deletions(-)

As "obvious" as this is, I can't take patches without any changelog text
at all, sorry.

greg k-h

From colin.king at canonical.com  Wed Jul 19 16:20:36 2017
From: colin.king at canonical.com (Colin King)
Date: Wed, 19 Jul 2017 17:20:36 +0100
Subject: [lustre-devel] [PATCH] staging: lustre: lnet: fix incorrect
	arguments order calling lstcon_session_new
Message-ID: <20170719162036.21621-1-colin.king@canonical.com>

From: Colin Ian King <colin.king at canonical.com>

The arguments args->lstio_ses_force and args->lstio_ses_timeout are
in the incorrect order. Fix this by swapping them around.

Detected by CoverityScan, CID#1226833 ("Arguments in wrong order")

Signed-off-by: Colin Ian King <colin.king at canonical.com>
---
 drivers/staging/lustre/lnet/selftest/conctl.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/staging/lustre/lnet/selftest/conctl.c b/drivers/staging/lustre/lnet/selftest/conctl.c
index 6ca7192b03b7..043eafb5ea40 100644
--- a/drivers/staging/lustre/lnet/selftest/conctl.c
+++ b/drivers/staging/lustre/lnet/selftest/conctl.c
@@ -69,8 +69,8 @@ lst_session_new_ioctl(struct lstio_session_new_args *args)
 	rc = lstcon_session_new(name,
 				args->lstio_ses_key,
 				args->lstio_ses_feats,
-				args->lstio_ses_force,
 				args->lstio_ses_timeout,
+				args->lstio_ses_force,
 				args->lstio_ses_idp);
 
 	LIBCFS_FREE(name, args->lstio_ses_nmlen + 1);
-- 
2.11.0

From dan.carpenter at oracle.com  Fri Jul 21 09:35:39 2017
From: dan.carpenter at oracle.com (Dan Carpenter)
Date: Fri, 21 Jul 2017 12:35:39 +0300
Subject: [lustre-devel] [PATCH v3 1/3] staging: lustre: constify
 attribute_group structures.
In-Reply-To: <1500616793-21569-2-git-send-email-arvind.yadav.cs@gmail.com>
References: <1500616793-21569-1-git-send-email-arvind.yadav.cs@gmail.com>
	<1500616793-21569-2-git-send-email-arvind.yadav.cs@gmail.com>
Message-ID: <20170721093539.qmwobmbzd3zigv4n@mwanda>

On Fri, Jul 21, 2017 at 11:29:51AM +0530, Arvind Yadav wrote:
> Changes in v3:
>               typo error in cover-latter.

The cover letter gets discarded so no one cares.

regards,
dan carpenter

From anna.fuchs at informatik.uni-hamburg.de  Fri Jul 21 15:15:30 2017
From: anna.fuchs at informatik.uni-hamburg.de (Anna Fuchs)
Date: Fri, 21 Jul 2017 17:15:30 +0200
Subject: [lustre-devel] Design proposal for client-side compression
In-Reply-To: <1487254536.4777.51.camel@informatik.uni-hamburg.de>
References: <1483967258.2622.24.camel@informatik.uni-hamburg.de>
	<B5E18A82-79B3-484F-BEBE-998B8AE56397@intel.com>
	<1484223328.3937.60.camel@informatik.uni-hamburg.de>
	<B924BE6D-3C6A-4BA4-96F9-AC053F032B2D@intel.com>
	<1484749185.3965.114.camel@informatik.uni-hamburg.de>
	<1487254536.4777.51.camel@informatik.uni-hamburg.de>
Message-ID: <1500650130.3158.6.camel@informatik.uni-hamburg.de>

Dear all, 

for compression within the osc module we need a bunch of pages for the
compressed output (at most the same size like original data), and few
pages for working memory of the algorithms. Since allocating (and later
freeing) the pages every time we enter the compression loop might be
expensive and annoying, we thought about a pool of pages, which is
present exclusively for compression purposes.

We would create that pool at file system start (when loading the osc
module) and destroy at file system stop (when unloading the osc
module). The condition is, of course, the configure option --enable-
compression. The pool would be a queue of page bunches where a thread
can pop pages for compression and put them back after the compressed
portion was transferred. The page content will not be visible to anyone
outside and will also not be cached after the transmission.

We would like to make the pool static since we think, we do not need a
lot of memory. However it depends on the number of stripes or MBs, that
one client can handle at the same time. E.g. for 32 stripes of 1MB
processed at the same time, we need at most 32 MB + few MB for
overhead. Where can I find the exact number or how can I estimate how
many stripes there are at most at the same time? Another limitation is
the number of threads, which can work in parallel on compression at the
same time. We think to exclusively reserve not more than 50 MB for the
compression page pool per client. Do you think it might hurt the
performance?

Once there are not enough pages, for whatever reason, we wouldn't wait,
but just skip the compression for the respective chunk. 

Are there any problems you see in that approach? 

Regards,
Anna

--
Anna Fuchs
https://wr.informatik.uni-hamburg.de/people/anna_fuchs

From paf at cray.com  Fri Jul 21 16:43:51 2017
From: paf at cray.com (Patrick Farrell)
Date: Fri, 21 Jul 2017 16:43:51 +0000
Subject: [lustre-devel] Design proposal for client-side compression
In-Reply-To: <1500650130.3158.6.camel@informatik.uni-hamburg.de>
References: <1483967258.2622.24.camel@informatik.uni-hamburg.de>
	<B5E18A82-79B3-484F-BEBE-998B8AE56397@intel.com>
	<1484223328.3937.60.camel@informatik.uni-hamburg.de>
	<B924BE6D-3C6A-4BA4-96F9-AC053F032B2D@intel.com>
	<1484749185.3965.114.camel@informatik.uni-hamburg.de>
	<1487254536.4777.51.camel@informatik.uni-hamburg.de>,
	<1500650130.3158.6.camel@informatik.uni-hamburg.de>
Message-ID: <BN6PR1101MB2132311AB19B104AD044797ACBA40@BN6PR1101MB2132.namprd11.prod.outlook.com>

I think basing this on the maximum number of stripes it too simple, and maybe not necessary.

Apologies in advance if what I say below rests on a misunderstanding of the compression design, I should know it better than I do.


But, here goes.

About based on maximum stripe count, there are a number of 1000 OST systems in the world today.  Imagine one of them with 16 MiB stripes, that's ~16 GiB of memory for this.  I think that's clearly too large.  But a global (rather than per OSC) pool could be tricky too, leading to contention on getting and returning pages.

You mention later a 50 MiB pool per client.  As a per OST pre-allocated pool, that would likely be too large.  As a global pool, it seems small...


But why use a global pool?  It sounds like the compression would be handled by the thread putting the data on the wire (Sorry if I've got that wrong).  So - What about a per-thread block of pages, for each ptlrpcd thread?  If the idea is that this compressed data is not retained for replay (instead, you would re-compress), then we only need a block of max rpc size for each thread (You could just use the largest RPC size supported by the client), so it can send that compressed data.

The overhead of compression for replay is probably not something we need to worry about.


Or even per-CPU blocks of pages.  That would probably be better still (less total memory if there are more ptlrpcds than CPUs), if we can guarantee not sleeping during the time the pool is in use.  (I'm not sure.)

Also, you mention limiting the # of threads.  Why is limiting the number of threads doing compression of interest?  What are you specifically trying to avoid with that?

________________________________
From: lustre-devel <lustre-devel-bounces at lists.lustre.org> on behalf of Anna Fuchs <anna.fuchs at informatik.uni-hamburg.de>
Sent: Friday, July 21, 2017 10:15:30 AM
To: Xiong, Jinshan
Cc: Matthew Ahrens; Zhuravlev, Alexey; lustre-devel
Subject: Re: [lustre-devel] Design proposal for client-side compression

Dear all,

for compression within the osc module we need a bunch of pages for the
compressed output (at most the same size like original data), and few
pages for working memory of the algorithms. Since allocating (and later
freeing) the pages every time we enter the compression loop might be
expensive and annoying, we thought about a pool of pages, which is
present exclusively for compression purposes.

We would create that pool at file system start (when loading the osc
module) and destroy at file system stop (when unloading the osc
module). The condition is, of course, the configure option --enable-
compression. The pool would be a queue of page bunches where a thread
can pop pages for compression and put them back after the compressed
portion was transferred. The page content will not be visible to anyone
outside and will also not be cached after the transmission.

We would like to make the pool static since we think, we do not need a
lot of memory. However it depends on the number of stripes or MBs, that
one client can handle at the same time. E.g. for 32 stripes of 1MB
processed at the same time, we need at most 32 MB + few MB for
overhead. Where can I find the exact number or how can I estimate how
many stripes there are at most at the same time? Another limitation is
the number of threads, which can work in parallel on compression at the
same time. We think to exclusively reserve not more than 50 MB for the
compression page pool per client. Do you think it might hurt the
performance?

Once there are not enough pages, for whatever reason, we wouldn't wait,
but just skip the compression for the respective chunk.

Are there any problems you see in that approach?

Regards,
Anna

--
Anna Fuchs
https://wr.informatik.uni-hamburg.de/people/anna_fuchs
_______________________________________________
lustre-devel mailing list
lustre-devel at lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-devel-lustre.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-devel_lists.lustre.org/attachments/20170721/4099f4fc/attachment.html>

From jinshan.xiong at intel.com  Fri Jul 21 19:12:02 2017
From: jinshan.xiong at intel.com (Xiong, Jinshan)
Date: Fri, 21 Jul 2017 19:12:02 +0000
Subject: [lustre-devel] Design proposal for client-side compression
In-Reply-To: <1500650130.3158.6.camel@informatik.uni-hamburg.de>
References: <1483967258.2622.24.camel@informatik.uni-hamburg.de>
	<B5E18A82-79B3-484F-BEBE-998B8AE56397@intel.com>
	<1484223328.3937.60.camel@informatik.uni-hamburg.de>
	<B924BE6D-3C6A-4BA4-96F9-AC053F032B2D@intel.com>
	<1484749185.3965.114.camel@informatik.uni-hamburg.de>
	<1487254536.4777.51.camel@informatik.uni-hamburg.de>
	<1500650130.3158.6.camel@informatik.uni-hamburg.de>
Message-ID: <0973C94D-74E2-4D88-8E41-7512A1C6310A@intel.com>

Please see inserted lines.

-----Original Message-----
From: Anna Fuchs <anna.fuchs at informatik.uni-hamburg.de>
Date: Friday, July 21, 2017 at 8:15 AM
To: "Xiong, Jinshan" <jinshan.xiong at intel.com>
Cc: Matthew Ahrens <mahrens at delphix.com>, "Zhuravlev, Alexey" <alexey.zhuravlev at intel.com>, lustre-devel <lustre-devel at lists.lustre.org>
Subject: Re: [lustre-devel] Design proposal for client-side compression

    Dear all, 
    
    for compression within the osc module we need a bunch of pages for the
    compressed output (at most the same size like original data), and few
    pages for working memory of the algorithms. Since allocating (and later
    freeing) the pages every time we enter the compression loop might be
    expensive and annoying, we thought about a pool of pages, which is
  present exclusively for compression purposes.
    
    We would create that pool at file system start (when loading the osc
    module) and destroy at file system stop (when unloading the osc
    module). The condition is, of course, the configure option --enable-
  compression. The pool would be a queue of page bunches where a thread

Is it possible to enable this by writing to a sysfs or procfs entry? So that users can try this out without having to recompile Lustre.

    can pop pages for compression and put them back after the compressed
    portion was transferred. The page content will not be visible to anyone
    outside and will also not be cached after the transmission.
    
    We would like to make the pool static since we think, we do not need a
    lot of memory. However it depends on the number of stripes or MBs, that
    one client can handle at the same time. E.g. for 32 stripes of 1MB
  processed at the same time, we need at most 32 MB + few MB for

Actually, we have increased the default RPC size to be 4MB so this assumption is no longer true.

    overhead. Where can I find the exact number or how can I estimate how
  many stripes there are at most at the same time? Another limitation is
  
It’s not scalable to have a pool per OSC because Lustre can support up to 2000 stripes. However, we don’t need to worry about wide stripe problem because no one can write a full stripe with even 1MB stripe size, because that means application has to issue 2GB size of write.
  
    the number of threads, which can work in parallel on compression at the
    same time. We think to exclusively reserve not more than 50 MB for the
    compression page pool per client. Do you think it might hurt the
  performance?

Yes, it’s reasonable to have a global pool for each client node. Let’s start from this number but please make it adjustable via sysfs or procfs.

Jinshan
    
    Once there are not enough pages, for whatever reason, we wouldn't wait,
    but just skip the compression for the respective chunk. 
    
    Are there any problems you see in that approach? 
    
    Regards,
    Anna
    
    --
    Anna Fuchs
    https://wr.informatik.uni-hamburg.de/people/anna_fuchs
    


From jinshan.xiong at intel.com  Fri Jul 21 19:19:39 2017
From: jinshan.xiong at intel.com (Xiong, Jinshan)
Date: Fri, 21 Jul 2017 19:19:39 +0000
Subject: [lustre-devel] Design proposal for client-side compression
In-Reply-To: <BN6PR1101MB2132311AB19B104AD044797ACBA40@BN6PR1101MB2132.namprd11.prod.outlook.com>
References: <1483967258.2622.24.camel@informatik.uni-hamburg.de>
	<B5E18A82-79B3-484F-BEBE-998B8AE56397@intel.com>
	<1484223328.3937.60.camel@informatik.uni-hamburg.de>
	<B924BE6D-3C6A-4BA4-96F9-AC053F032B2D@intel.com>
	<1484749185.3965.114.camel@informatik.uni-hamburg.de>
	<1487254536.4777.51.camel@informatik.uni-hamburg.de>
	<1500650130.3158.6.camel@informatik.uni-hamburg.de>
	<BN6PR1101MB2132311AB19B104AD044797ACBA40@BN6PR1101MB2132.namprd11.prod.outlook.com>
Message-ID: <11E369DA-8B2C-4DB3-B4A6-C7288B4290F6@intel.com>



From: Patrick Farrell <paf at cray.com>
Date: Friday, July 21, 2017 at 9:44 AM
To: Anna Fuchs <anna.fuchs at informatik.uni-hamburg.de>, "Xiong, Jinshan" <jinshan.xiong at intel.com>
Cc: Matthew Ahrens <mahrens at delphix.com>, "Zhuravlev, Alexey" <alexey.zhuravlev at intel.com>, lustre-devel <lustre-devel at lists.lustre.org>
Subject: Re: [lustre-devel] Design proposal for client-side compression


I think basing this on the maximum number of stripes it too simple, and maybe not necessary.

Apologies in advance if what I say below rests on a misunderstanding of the compression design, I should know it better than I do.



But, here goes.

About based on maximum stripe count, there are a number of 1000 OST systems in the world today.  Imagine one of them with 16 MiB stripes, that's ~16 GiB of memory for this.  I think that's clearly too large.  But a global (rather than per OSC) pool could be tricky too, leading to contention on getting and returning pages.

You mention later a 50 MiB pool per client.  As a per OST pre-allocated pool, that would likely be too large.  As a global pool, it seems small...



But why use a global pool?  It sounds like the compression would be handled by the thread putting the data on the wire (Sorry if I've got that wrong).  So - What about a per-thread block of pages, for each ptlrpcd thread?  If the idea is that this compressed data is not retained for replay (instead, you would re-compress), then we only need a block of max rpc size for each thread (You could just use the largest RPC size supported by the client), so it can send that compressed data.



The writing thread would also need to issue RPC in its own process context, but it can be revised to use ptlrpcd thread. I tend to think using a global ptlrpc thread would be reasonable for now because compression should be slow so I don’t expect there would be a lot of lock contention for the global pool.



Jinshan

The overhead of compression for replay is probably not something we need to worry about.



Or even per-CPU blocks of pages.  That would probably be better still (less total memory if there are more ptlrpcds than CPUs), if we can guarantee not sleeping during the time the pool is in use.  (I'm not sure.)

Also, you mention limiting the # of threads.  Why is limiting the number of threads doing compression of interest?  What are you specifically trying to avoid with that?

________________________________
From: lustre-devel <lustre-devel-bounces at lists.lustre.org> on behalf of Anna Fuchs <anna.fuchs at informatik.uni-hamburg.de>
Sent: Friday, July 21, 2017 10:15:30 AM
To: Xiong, Jinshan
Cc: Matthew Ahrens; Zhuravlev, Alexey; lustre-devel
Subject: Re: [lustre-devel] Design proposal for client-side compression

Dear all,

for compression within the osc module we need a bunch of pages for the
compressed output (at most the same size like original data), and few
pages for working memory of the algorithms. Since allocating (and later
freeing) the pages every time we enter the compression loop might be
expensive and annoying, we thought about a pool of pages, which is
present exclusively for compression purposes.

We would create that pool at file system start (when loading the osc
module) and destroy at file system stop (when unloading the osc
module). The condition is, of course, the configure option --enable-
compression. The pool would be a queue of page bunches where a thread
can pop pages for compression and put them back after the compressed
portion was transferred. The page content will not be visible to anyone
outside and will also not be cached after the transmission.

We would like to make the pool static since we think, we do not need a
lot of memory. However it depends on the number of stripes or MBs, that
one client can handle at the same time. E.g. for 32 stripes of 1MB
processed at the same time, we need at most 32 MB + few MB for
overhead. Where can I find the exact number or how can I estimate how
many stripes there are at most at the same time? Another limitation is
the number of threads, which can work in parallel on compression at the
same time. We think to exclusively reserve not more than 50 MB for the
compression page pool per client. Do you think it might hurt the
performance?

Once there are not enough pages, for whatever reason, we wouldn't wait,
but just skip the compression for the respective chunk.

Are there any problems you see in that approach?

Regards,
Anna

--
Anna Fuchs
https://wr.informatik.uni-hamburg.de/people/anna_fuchs
_______________________________________________
lustre-devel mailing list
lustre-devel at lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-devel-lustre.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-devel_lists.lustre.org/attachments/20170721/204968dc/attachment.html>

From arvind.yadav.cs at gmail.com  Fri Jul 21 05:59:53 2017
From: arvind.yadav.cs at gmail.com (Arvind Yadav)
Date: Fri, 21 Jul 2017 11:29:53 +0530
Subject: [lustre-devel] [PATCH v3 3/3] staging: lustre: obdclass: linux:
	constify attribute_group structures.
In-Reply-To: <1500616793-21569-1-git-send-email-arvind.yadav.cs@gmail.com>
References: <1500616793-21569-1-git-send-email-arvind.yadav.cs@gmail.com>
Message-ID: <1500616793-21569-4-git-send-email-arvind.yadav.cs@gmail.com>

attribute_groups are not supposed to change at runtime. All functions
working with attribute_groups provided by <linux/sysfs.h> work
with const attribute_group. So mark the non-const structs as const.

Signed-off-by: Arvind Yadav <arvind.yadav.cs at gmail.com>
---
Changes in v2:
              patch was sent to wrong group.
Changes in v3:
              typo error in cover-latter.

 drivers/staging/lustre/lustre/obdclass/linux/linux-module.c | 2 +-
 drivers/staging/lustre/lustre/obdclass/linux/linux-sysctl.c | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/drivers/staging/lustre/lustre/obdclass/linux/linux-module.c b/drivers/staging/lustre/lustre/obdclass/linux/linux-module.c
index 9f5e829..eb88bd9 100644
--- a/drivers/staging/lustre/lustre/obdclass/linux/linux-module.c
+++ b/drivers/staging/lustre/lustre/obdclass/linux/linux-module.c
@@ -405,7 +405,7 @@ static int obd_device_list_open(struct inode *inode, struct file *file)
 struct kobject *lustre_kobj;
 EXPORT_SYMBOL_GPL(lustre_kobj);
 
-static struct attribute_group lustre_attr_group = {
+static const struct attribute_group lustre_attr_group = {
 	.attrs = lustre_attrs,
 };
 
diff --git a/drivers/staging/lustre/lustre/obdclass/linux/linux-sysctl.c b/drivers/staging/lustre/lustre/obdclass/linux/linux-sysctl.c
index e6c785a..814334b 100644
--- a/drivers/staging/lustre/lustre/obdclass/linux/linux-sysctl.c
+++ b/drivers/staging/lustre/lustre/obdclass/linux/linux-sysctl.c
@@ -151,7 +151,7 @@ static ssize_t max_dirty_mb_store(struct kobject *kobj, struct attribute *attr,
 	NULL,
 };
 
-static struct attribute_group lustre_attr_group = {
+static const struct attribute_group lustre_attr_group = {
 	.attrs = lustre_attrs,
 };
 
-- 
1.9.1

From arvind.yadav.cs at gmail.com  Fri Jul 21 05:41:35 2017
From: arvind.yadav.cs at gmail.com (Arvind Yadav)
Date: Fri, 21 Jul 2017 11:11:35 +0530
Subject: [lustre-devel] [PATCH v2 0/3] Constify lustre attribute_group
	structures.
Message-ID: <1500615698-21001-1-git-send-email-arvind.yadav.cs@gmail.com>

ttribute_groups are not supposed to change at runtime. All functions
working with attribute_groups provided by <linux/sysfs.h> work
with const attribute_group. So mark the non-const structs as const.

Arvind Yadav (3):
  [PATCH v2 1/3] staging: lustre: constify attribute_group structures.
  [PATCH v2 2/3] staging: lustre: ldlm: constify attribute_group structures.
  [PATCH v2 3/3] staging: lustre: obdclass: linux: constify attribute_group structures.

 drivers/staging/lustre/lustre/include/lprocfs_status.h      | 4 ++--
 drivers/staging/lustre/lustre/ldlm/ldlm_lockd.c             | 2 +-
 drivers/staging/lustre/lustre/lmv/lproc_lmv.c               | 2 +-
 drivers/staging/lustre/lustre/lov/lproc_lov.c               | 2 +-
 drivers/staging/lustre/lustre/mdc/lproc_mdc.c               | 2 +-
 drivers/staging/lustre/lustre/obdclass/linux/linux-module.c | 2 +-
 drivers/staging/lustre/lustre/obdclass/linux/linux-sysctl.c | 2 +-
 drivers/staging/lustre/lustre/obdclass/lprocfs_status.c     | 2 +-
 drivers/staging/lustre/lustre/osc/lproc_osc.c               | 2 +-
 9 files changed, 10 insertions(+), 10 deletions(-)

--
1.9.1

From arvind.yadav.cs at gmail.com  Fri Jul 21 05:41:36 2017
From: arvind.yadav.cs at gmail.com (Arvind Yadav)
Date: Fri, 21 Jul 2017 11:11:36 +0530
Subject: [lustre-devel] [PATCH v2 1/3] staging: lustre: constify
	attribute_group structures.
In-Reply-To: <1500615698-21001-1-git-send-email-arvind.yadav.cs@gmail.com>
References: <1500615698-21001-1-git-send-email-arvind.yadav.cs@gmail.com>
Message-ID: <1500615698-21001-2-git-send-email-arvind.yadav.cs@gmail.com>

attribute_groups are not supposed to change at runtime. All functions
working with attribute_groups provided by <linux/sysfs.h> work
with const attribute_group. So mark the non-const structs as const.

File size before:
   text	   data	    bss	    dec	    hex	filename
   9489	    992	     40	  10521	   2919	lustre/lustre/osc/lproc_osc.o
   1289	    288	      0	   1577	    629	lustre/lustre/lmv/lproc_lmv.o
   3794	    928	     40	   4762	   129a	lustre/lustre/lov/lproc_lov.o
   3802	    576	     40	   4418	   1142	lustre/lustre/mdc/lproc_mdc.o

File size After adding 'const':
   text	   data	    bss	    dec	    hex	filename
   9553	    928	     40	  10521	   2919	lustre/lustre/osc/lproc_osc.o
   1353	    224	      0	   1577	    629	lustre/lustre/lmv/lproc_lmv.o
   3858	    864	     40	   4762	   129a	lustre/lustre/lov/lproc_lov.o
   3866	    512	     40	   4418	   1142	lustre/lustre/mdc/lproc_mdc.o

Signed-off-by: Arvind Yadav <arvind.yadav.cs at gmail.com>
---
Changes in v2:
              Fix kbuild error: conflicting types for 'lprocfs_obd_setup'.
              By changing protype of 'lprocfs_obd_setup'.

 drivers/staging/lustre/lustre/include/lprocfs_status.h  | 4 ++--
 drivers/staging/lustre/lustre/lmv/lproc_lmv.c           | 2 +-
 drivers/staging/lustre/lustre/lov/lproc_lov.c           | 2 +-
 drivers/staging/lustre/lustre/mdc/lproc_mdc.c           | 2 +-
 drivers/staging/lustre/lustre/obdclass/lprocfs_status.c | 2 +-
 drivers/staging/lustre/lustre/osc/lproc_osc.c           | 2 +-
 6 files changed, 7 insertions(+), 7 deletions(-)

diff --git a/drivers/staging/lustre/lustre/include/lprocfs_status.h b/drivers/staging/lustre/lustre/include/lprocfs_status.h
index 915283c..9054d37 100644
--- a/drivers/staging/lustre/lustre/include/lprocfs_status.h
+++ b/drivers/staging/lustre/lustre/include/lprocfs_status.h
@@ -59,7 +59,7 @@ struct lprocfs_vars {
 
 struct lprocfs_static_vars {
 	struct lprocfs_vars *obd_vars;
-	struct attribute_group *sysfs_vars;
+	const struct attribute_group *sysfs_vars;
 };
 
 /* if we find more consumers this could be generalized */
@@ -468,7 +468,7 @@ struct dentry *ldebugfs_register(const char *name,
 void ldebugfs_remove(struct dentry **entryp);
 
 int lprocfs_obd_setup(struct obd_device *obd, struct lprocfs_vars *list,
-		      struct attribute_group *attrs);
+		      const struct attribute_group *attrs);
 int lprocfs_obd_cleanup(struct obd_device *obd);
 
 int ldebugfs_seq_create(struct dentry *parent,
diff --git a/drivers/staging/lustre/lustre/lmv/lproc_lmv.c b/drivers/staging/lustre/lustre/lmv/lproc_lmv.c
index bf25f88..4c13e39 100644
--- a/drivers/staging/lustre/lustre/lmv/lproc_lmv.c
+++ b/drivers/staging/lustre/lustre/lmv/lproc_lmv.c
@@ -161,7 +161,7 @@ static int lmv_target_seq_open(struct inode *inode, struct file *file)
 	NULL,
 };
 
-static struct attribute_group lmv_attr_group = {
+static const struct attribute_group lmv_attr_group = {
 	.attrs = lmv_attrs,
 };
 
diff --git a/drivers/staging/lustre/lustre/lov/lproc_lov.c b/drivers/staging/lustre/lustre/lov/lproc_lov.c
index eb6d30d..ce46821 100644
--- a/drivers/staging/lustre/lustre/lov/lproc_lov.c
+++ b/drivers/staging/lustre/lustre/lov/lproc_lov.c
@@ -279,7 +279,7 @@ static int lov_target_seq_open(struct inode *inode, struct file *file)
 	NULL,
 };
 
-static struct attribute_group lov_attr_group = {
+static const struct attribute_group lov_attr_group = {
 	.attrs = lov_attrs,
 };
 
diff --git a/drivers/staging/lustre/lustre/mdc/lproc_mdc.c b/drivers/staging/lustre/lustre/mdc/lproc_mdc.c
index 9021c46..9fea535 100644
--- a/drivers/staging/lustre/lustre/mdc/lproc_mdc.c
+++ b/drivers/staging/lustre/lustre/mdc/lproc_mdc.c
@@ -219,7 +219,7 @@ static ssize_t max_pages_per_rpc_show(struct kobject *kobj,
 	NULL,
 };
 
-static struct attribute_group mdc_attr_group = {
+static const struct attribute_group mdc_attr_group = {
 	.attrs = mdc_attrs,
 };
 
diff --git a/drivers/staging/lustre/lustre/obdclass/lprocfs_status.c b/drivers/staging/lustre/lustre/obdclass/lprocfs_status.c
index bc19f19..ba41983 100644
--- a/drivers/staging/lustre/lustre/obdclass/lprocfs_status.c
+++ b/drivers/staging/lustre/lustre/obdclass/lprocfs_status.c
@@ -1031,7 +1031,7 @@ static void obd_sysfs_release(struct kobject *kobj)
 };
 
 int lprocfs_obd_setup(struct obd_device *obd, struct lprocfs_vars *list,
-		      struct attribute_group *attrs)
+		      const struct attribute_group *attrs)
 {
 	int rc = 0;
 
diff --git a/drivers/staging/lustre/lustre/osc/lproc_osc.c b/drivers/staging/lustre/lustre/osc/lproc_osc.c
index 86f252d..6e0fd15 100644
--- a/drivers/staging/lustre/lustre/osc/lproc_osc.c
+++ b/drivers/staging/lustre/lustre/osc/lproc_osc.c
@@ -831,7 +831,7 @@ int lproc_osc_attach_seqstat(struct obd_device *dev)
 	NULL,
 };
 
-static struct attribute_group osc_attr_group = {
+static const struct attribute_group osc_attr_group = {
 	.attrs = osc_attrs,
 };
 
-- 
1.9.1

From arvind.yadav.cs at gmail.com  Fri Jul 21 05:59:50 2017
From: arvind.yadav.cs at gmail.com (Arvind Yadav)
Date: Fri, 21 Jul 2017 11:29:50 +0530
Subject: [lustre-devel] [PATCH v3 0/3] Constify lustre attribute_group
	structures.
Message-ID: <1500616793-21569-1-git-send-email-arvind.yadav.cs@gmail.com>

attribute_groups are not supposed to change at runtime. All functions
working with attribute_groups provided by <linux/sysfs.h> work
with const attribute_group. So mark the non-const structs as const.

Arvind Yadav (3):
  [PATCH v3 1/3] staging: lustre: constify attribute_group structures.
  [PATCH v3 2/3] staging: lustre: ldlm: constify attribute_group structures.
  [PATCH v3 3/3] staging: lustre: obdclass: linux: constify attribute_group structures.

 drivers/staging/lustre/lustre/include/lprocfs_status.h      | 4 ++--
 drivers/staging/lustre/lustre/ldlm/ldlm_lockd.c             | 2 +-
 drivers/staging/lustre/lustre/lmv/lproc_lmv.c               | 2 +-
 drivers/staging/lustre/lustre/lov/lproc_lov.c               | 2 +-
 drivers/staging/lustre/lustre/mdc/lproc_mdc.c               | 2 +-
 drivers/staging/lustre/lustre/obdclass/linux/linux-module.c | 2 +-
 drivers/staging/lustre/lustre/obdclass/linux/linux-sysctl.c | 2 +-
 drivers/staging/lustre/lustre/obdclass/lprocfs_status.c     | 2 +-
 drivers/staging/lustre/lustre/osc/lproc_osc.c               | 2 +-
 9 files changed, 10 insertions(+), 10 deletions(-)

--
1.9.1

From arvind.yadav.cs at gmail.com  Fri Jul 21 05:59:51 2017
From: arvind.yadav.cs at gmail.com (Arvind Yadav)
Date: Fri, 21 Jul 2017 11:29:51 +0530
Subject: [lustre-devel] [PATCH v3 1/3] staging: lustre: constify
	attribute_group structures.
In-Reply-To: <1500616793-21569-1-git-send-email-arvind.yadav.cs@gmail.com>
References: <1500616793-21569-1-git-send-email-arvind.yadav.cs@gmail.com>
Message-ID: <1500616793-21569-2-git-send-email-arvind.yadav.cs@gmail.com>

attribute_groups are not supposed to change at runtime. All functions
working with attribute_groups provided by <linux/sysfs.h> work
with const attribute_group. So mark the non-const structs as const.

File size before:
   text	   data	    bss	    dec	    hex	filename
   9489	    992	     40	  10521	   2919	lustre/lustre/osc/lproc_osc.o
   1289	    288	      0	   1577	    629	lustre/lustre/lmv/lproc_lmv.o
   3794	    928	     40	   4762	   129a	lustre/lustre/lov/lproc_lov.o
   3802	    576	     40	   4418	   1142	lustre/lustre/mdc/lproc_mdc.o

File size After adding 'const':
   text	   data	    bss	    dec	    hex	filename
   9553	    928	     40	  10521	   2919	lustre/lustre/osc/lproc_osc.o
   1353	    224	      0	   1577	    629	lustre/lustre/lmv/lproc_lmv.o
   3858	    864	     40	   4762	   129a	lustre/lustre/lov/lproc_lov.o
   3866	    512	     40	   4418	   1142	lustre/lustre/mdc/lproc_mdc.o

Signed-off-by: Arvind Yadav <arvind.yadav.cs at gmail.com>
---
Changes in v2:
              Fix kbuild error: conflicting types for 'lprocfs_obd_setup'.
              By changing protype of 'lprocfs_obd_setup'.
Changes in v3:
              typo error in cover-latter.

 drivers/staging/lustre/lustre/include/lprocfs_status.h  | 4 ++--
 drivers/staging/lustre/lustre/lmv/lproc_lmv.c           | 2 +-
 drivers/staging/lustre/lustre/lov/lproc_lov.c           | 2 +-
 drivers/staging/lustre/lustre/mdc/lproc_mdc.c           | 2 +-
 drivers/staging/lustre/lustre/obdclass/lprocfs_status.c | 2 +-
 drivers/staging/lustre/lustre/osc/lproc_osc.c           | 2 +-
 6 files changed, 7 insertions(+), 7 deletions(-)

diff --git a/drivers/staging/lustre/lustre/include/lprocfs_status.h b/drivers/staging/lustre/lustre/include/lprocfs_status.h
index 915283c..9054d37 100644
--- a/drivers/staging/lustre/lustre/include/lprocfs_status.h
+++ b/drivers/staging/lustre/lustre/include/lprocfs_status.h
@@ -59,7 +59,7 @@ struct lprocfs_vars {
 
 struct lprocfs_static_vars {
 	struct lprocfs_vars *obd_vars;
-	struct attribute_group *sysfs_vars;
+	const struct attribute_group *sysfs_vars;
 };
 
 /* if we find more consumers this could be generalized */
@@ -468,7 +468,7 @@ struct dentry *ldebugfs_register(const char *name,
 void ldebugfs_remove(struct dentry **entryp);
 
 int lprocfs_obd_setup(struct obd_device *obd, struct lprocfs_vars *list,
-		      struct attribute_group *attrs);
+		      const struct attribute_group *attrs);
 int lprocfs_obd_cleanup(struct obd_device *obd);
 
 int ldebugfs_seq_create(struct dentry *parent,
diff --git a/drivers/staging/lustre/lustre/lmv/lproc_lmv.c b/drivers/staging/lustre/lustre/lmv/lproc_lmv.c
index bf25f88..4c13e39 100644
--- a/drivers/staging/lustre/lustre/lmv/lproc_lmv.c
+++ b/drivers/staging/lustre/lustre/lmv/lproc_lmv.c
@@ -161,7 +161,7 @@ static int lmv_target_seq_open(struct inode *inode, struct file *file)
 	NULL,
 };
 
-static struct attribute_group lmv_attr_group = {
+static const struct attribute_group lmv_attr_group = {
 	.attrs = lmv_attrs,
 };
 
diff --git a/drivers/staging/lustre/lustre/lov/lproc_lov.c b/drivers/staging/lustre/lustre/lov/lproc_lov.c
index eb6d30d..ce46821 100644
--- a/drivers/staging/lustre/lustre/lov/lproc_lov.c
+++ b/drivers/staging/lustre/lustre/lov/lproc_lov.c
@@ -279,7 +279,7 @@ static int lov_target_seq_open(struct inode *inode, struct file *file)
 	NULL,
 };
 
-static struct attribute_group lov_attr_group = {
+static const struct attribute_group lov_attr_group = {
 	.attrs = lov_attrs,
 };
 
diff --git a/drivers/staging/lustre/lustre/mdc/lproc_mdc.c b/drivers/staging/lustre/lustre/mdc/lproc_mdc.c
index 9021c46..9fea535 100644
--- a/drivers/staging/lustre/lustre/mdc/lproc_mdc.c
+++ b/drivers/staging/lustre/lustre/mdc/lproc_mdc.c
@@ -219,7 +219,7 @@ static ssize_t max_pages_per_rpc_show(struct kobject *kobj,
 	NULL,
 };
 
-static struct attribute_group mdc_attr_group = {
+static const struct attribute_group mdc_attr_group = {
 	.attrs = mdc_attrs,
 };
 
diff --git a/drivers/staging/lustre/lustre/obdclass/lprocfs_status.c b/drivers/staging/lustre/lustre/obdclass/lprocfs_status.c
index bc19f19..ba41983 100644
--- a/drivers/staging/lustre/lustre/obdclass/lprocfs_status.c
+++ b/drivers/staging/lustre/lustre/obdclass/lprocfs_status.c
@@ -1031,7 +1031,7 @@ static void obd_sysfs_release(struct kobject *kobj)
 };
 
 int lprocfs_obd_setup(struct obd_device *obd, struct lprocfs_vars *list,
-		      struct attribute_group *attrs)
+		      const struct attribute_group *attrs)
 {
 	int rc = 0;
 
diff --git a/drivers/staging/lustre/lustre/osc/lproc_osc.c b/drivers/staging/lustre/lustre/osc/lproc_osc.c
index 86f252d..6e0fd15 100644
--- a/drivers/staging/lustre/lustre/osc/lproc_osc.c
+++ b/drivers/staging/lustre/lustre/osc/lproc_osc.c
@@ -831,7 +831,7 @@ int lproc_osc_attach_seqstat(struct obd_device *dev)
 	NULL,
 };
 
-static struct attribute_group osc_attr_group = {
+static const struct attribute_group osc_attr_group = {
 	.attrs = osc_attrs,
 };
 
-- 
1.9.1

From arvind.yadav.cs at gmail.com  Fri Jul 21 05:59:52 2017
From: arvind.yadav.cs at gmail.com (Arvind Yadav)
Date: Fri, 21 Jul 2017 11:29:52 +0530
Subject: [lustre-devel] [PATCH v3 2/3] staging: lustre: ldlm: constify
	attribute_group structures.
In-Reply-To: <1500616793-21569-1-git-send-email-arvind.yadav.cs@gmail.com>
References: <1500616793-21569-1-git-send-email-arvind.yadav.cs@gmail.com>
Message-ID: <1500616793-21569-3-git-send-email-arvind.yadav.cs@gmail.com>

attribute_groups are not supposed to change at runtime. All functions
working with attribute_groups provided by <linux/sysfs.h> work
with const attribute_group. So mark the non-const structs as const.

Signed-off-by: Arvind Yadav <arvind.yadav.cs at gmail.com>
---
Changes in v2:
              patch was sent to wrong group.
Changes in v3:
              typo error in cover-latter.

 drivers/staging/lustre/lustre/ldlm/ldlm_lockd.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/staging/lustre/lustre/ldlm/ldlm_lockd.c b/drivers/staging/lustre/lustre/ldlm/ldlm_lockd.c
index fff930f..e0c3e5d 100644
--- a/drivers/staging/lustre/lustre/ldlm/ldlm_lockd.c
+++ b/drivers/staging/lustre/lustre/ldlm/ldlm_lockd.c
@@ -926,7 +926,7 @@ static ssize_t cancel_unused_locks_before_replay_store(struct kobject *kobj,
 	NULL,
 };
 
-static struct attribute_group ldlm_attr_group = {
+static const struct attribute_group ldlm_attr_group = {
 	.attrs = ldlm_attrs,
 };
 
-- 
1.9.1

From arvind.yadav.cs at gmail.com  Fri Jul 21 05:41:38 2017
From: arvind.yadav.cs at gmail.com (Arvind Yadav)
Date: Fri, 21 Jul 2017 11:11:38 +0530
Subject: [lustre-devel] [PATCH v2 3/3] staging: lustre: obdclass: linux:
	constify attribute_group structures.
In-Reply-To: <1500615698-21001-1-git-send-email-arvind.yadav.cs@gmail.com>
References: <1500615698-21001-1-git-send-email-arvind.yadav.cs@gmail.com>
Message-ID: <1500615698-21001-4-git-send-email-arvind.yadav.cs@gmail.com>

attribute_groups are not supposed to change at runtime. All functions
working with attribute_groups provided by <linux/sysfs.h> work
with const attribute_group. So mark the non-const structs as const.

Signed-off-by: Arvind Yadav <arvind.yadav.cs at gmail.com>
---
Changes in v2:
              patch was sent to wrong group.

 drivers/staging/lustre/lustre/obdclass/linux/linux-module.c | 2 +-
 drivers/staging/lustre/lustre/obdclass/linux/linux-sysctl.c | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/drivers/staging/lustre/lustre/obdclass/linux/linux-module.c b/drivers/staging/lustre/lustre/obdclass/linux/linux-module.c
index 9f5e829..eb88bd9 100644
--- a/drivers/staging/lustre/lustre/obdclass/linux/linux-module.c
+++ b/drivers/staging/lustre/lustre/obdclass/linux/linux-module.c
@@ -405,7 +405,7 @@ static int obd_device_list_open(struct inode *inode, struct file *file)
 struct kobject *lustre_kobj;
 EXPORT_SYMBOL_GPL(lustre_kobj);
 
-static struct attribute_group lustre_attr_group = {
+static const struct attribute_group lustre_attr_group = {
 	.attrs = lustre_attrs,
 };
 
diff --git a/drivers/staging/lustre/lustre/obdclass/linux/linux-sysctl.c b/drivers/staging/lustre/lustre/obdclass/linux/linux-sysctl.c
index e6c785a..814334b 100644
--- a/drivers/staging/lustre/lustre/obdclass/linux/linux-sysctl.c
+++ b/drivers/staging/lustre/lustre/obdclass/linux/linux-sysctl.c
@@ -151,7 +151,7 @@ static ssize_t max_dirty_mb_store(struct kobject *kobj, struct attribute *attr,
 	NULL,
 };
 
-static struct attribute_group lustre_attr_group = {
+static const struct attribute_group lustre_attr_group = {
 	.attrs = lustre_attrs,
 };
 
-- 
1.9.1

From arvind.yadav.cs at gmail.com  Fri Jul 21 05:41:37 2017
From: arvind.yadav.cs at gmail.com (Arvind Yadav)
Date: Fri, 21 Jul 2017 11:11:37 +0530
Subject: [lustre-devel] [PATCH v2 2/3] staging: lustre: ldlm: constify
	attribute_group structures.
In-Reply-To: <1500615698-21001-1-git-send-email-arvind.yadav.cs@gmail.com>
References: <1500615698-21001-1-git-send-email-arvind.yadav.cs@gmail.com>
Message-ID: <1500615698-21001-3-git-send-email-arvind.yadav.cs@gmail.com>

attribute_groups are not supposed to change at runtime. All functions
working with attribute_groups provided by <linux/sysfs.h> work
with const attribute_group. So mark the non-const structs as const.

Signed-off-by: Arvind Yadav <arvind.yadav.cs at gmail.com>
---
Changes in v2:
              patch was sent to wrong group.

 drivers/staging/lustre/lustre/ldlm/ldlm_lockd.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/staging/lustre/lustre/ldlm/ldlm_lockd.c b/drivers/staging/lustre/lustre/ldlm/ldlm_lockd.c
index fff930f..e0c3e5d 100644
--- a/drivers/staging/lustre/lustre/ldlm/ldlm_lockd.c
+++ b/drivers/staging/lustre/lustre/ldlm/ldlm_lockd.c
@@ -926,7 +926,7 @@ static ssize_t cancel_unused_locks_before_replay_store(struct kobject *kobj,
 	NULL,
 };
 
-static struct attribute_group ldlm_attr_group = {
+static const struct attribute_group ldlm_attr_group = {
 	.attrs = ldlm_attrs,
 };
 
-- 
1.9.1

From amir.shehata at intel.com  Wed Jul 19 20:26:20 2017
From: amir.shehata at intel.com (Shehata, Amir)
Date: Wed, 19 Jul 2017 20:26:20 +0000
Subject: [lustre-devel] [PATCH] staging: lustre: ko2iblnd: check
 copy_from_iter/copy_to_iter return code
In-Reply-To: <1500132728-28511-1-git-send-email-jsimmons@infradead.org>
References: <1500132728-28511-1-git-send-email-jsimmons@infradead.org>
Message-ID: <859F5EEE7D708F47B769E2BA7556E11A424232DF@CRSMSX104.amr.corp.intel.com>

The changes look good to me.

-----Original Message-----
From: James Simmons [mailto:jsimmons at infradead.org] 
Sent: Saturday, July 15, 2017 8:32 AM
To: Greg Kroah-Hartman <gregkh at linuxfoundation.org>; devel at driverdev.osuosl.org; Dilger, Andreas <andreas.dilger at intel.com>; Drokin, Oleg <oleg.drokin at intel.com>; Arnd Bergmann <arnd at arndb.de>; Al Viro <viro at zeniv.linux.org.uk>
Cc: Shehata, Amir <amir.shehata at intel.com>; Olaf Weber <olaf.weber at hpe.com>; Linux Kernel Mailing List <linux-kernel at vger.kernel.org>; Lustre Development List <lustre-devel at lists.lustre.org>; stable at vger.kernel.org; James Simmons <jsimmons at infradead.org>
Subject: [PATCH] staging: lustre: ko2iblnd: check copy_from_iter/copy_to_iter return code

From: Arnd Bergmann <arnd at arndb.de>

We now get a helpful warning for code that calls copy_{from,to}_iter without checking the return value, introduced by commit aa28de275a24
("iov_iter/hardening: move object size checks to inlined part").

drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd_cb.c: In function 'kiblnd_send':
drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd_cb.c:1643:2: error: ignoring return value of 'copy_from_iter', declared with attribute warn_unused_result [-Werror=unused-result]
drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd_cb.c: In function 'kiblnd_recv':
drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd_cb.c:1744:3: error: ignoring return value of 'copy_to_iter', declared with attribute warn_unused_result [-Werror=unused-result]

In case we get short copies here, we may get incorrect behavior.
I've added failure handling for both rx and tx now, returning -EFAULT as expected.

Cc: stable at vger.kernel.org
Signed-off-by: Arnd Bergmann <arnd at arndb.de>
Signed-off-by: James Simmons <jsimmons at infradead.org>
Signed-off-by: Amir Shehata <amir.shehata at intel.com>
---
 .../staging/lustre/lnet/klnds/o2iblnd/o2iblnd_cb.c    | 19 +++++++++++++++----
 1 file changed, 15 insertions(+), 4 deletions(-)

diff --git a/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd_cb.c b/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd_cb.c
index 85b242e..8fc191d 100644
--- a/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd_cb.c
+++ b/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd_cb.c
@@ -1640,8 +1640,13 @@ static int kiblnd_resolve_addr(struct rdma_cm_id *cmid,
 	ibmsg = tx->tx_msg;
 	ibmsg->ibm_u.immediate.ibim_hdr = *hdr;
 
-	copy_from_iter(&ibmsg->ibm_u.immediate.ibim_payload, IBLND_MSG_SIZE,
-		       &from);
+	rc = copy_from_iter(&ibmsg->ibm_u.immediate.ibim_payload, payload_nob,
+			    &from);
+	if (rc != payload_nob) {
+		kiblnd_pool_free_node(&tx->tx_pool->tpo_pool, &tx->tx_list);
+		return -EFAULT;
+	}
+
 	nob = offsetof(struct kib_immediate_msg, ibim_payload[payload_nob]);
 	kiblnd_init_tx_msg(ni, tx, IBLND_MSG_IMMEDIATE, nob);
 
@@ -1741,8 +1746,14 @@ static int kiblnd_resolve_addr(struct rdma_cm_id *cmid,
 			break;
 		}
 
-		copy_to_iter(&rxmsg->ibm_u.immediate.ibim_payload,
-			     IBLND_MSG_SIZE, to);
+		rc = copy_to_iter(&rxmsg->ibm_u.immediate.ibim_payload, rlen,
+				  to);
+		if (rc != rlen) {
+			rc = -EFAULT;
+			break;
+		}
+
+		rc = 0;
 		lnet_finalize(ni, lntmsg, 0);
 		break;
 
--
1.8.3.1

From joe at perches.com  Sat Jul 22 17:00:24 2017
From: joe at perches.com (Joe Perches)
Date: Sat, 22 Jul 2017 10:00:24 -0700
Subject: [lustre-devel] [PATCH] Staging: lustre: Place the open brace on
	previous line.
In-Reply-To: <1500742601-10733-1-git-send-email-shreeya.patel23498@gmail.com>
References: <1500742601-10733-1-git-send-email-shreeya.patel23498@gmail.com>
Message-ID: <1500742824.14415.37.camel@perches.com>

On Sat, 2017-07-22 at 22:26 +0530, Shreeya Patel wrote:
> Remove errors by placing the open braces on previous line.
> This saves the vertical space of the code.
> 
> Problem found by checkpatch.

No.  These are checkpatch false positives.

> diff --git a/drivers/staging/lustre/lnet/libcfs/hash.c b/drivers/staging/lustre/lnet/libcfs/hash.c
[]
> @@ -1348,8 +1348,7 @@ cfs_hash_lookup(struct cfs_hash *hs, const void *key)
>  EXPORT_SYMBOL(cfs_hash_lookup);
>  
>  static void
> -cfs_hash_for_each_enter(struct cfs_hash *hs)
> -{
> +cfs_hash_for_each_enter(struct cfs_hash *hs) {
>  	LASSERT(!cfs_hash_is_exiting(hs));
>  
>  	if (!cfs_hash_with_rehash(hs))

etc...

These are correctly written function definitions.

From anna.fuchs at informatik.uni-hamburg.de  Tue Jul 25 14:25:40 2017
From: anna.fuchs at informatik.uni-hamburg.de (Anna Fuchs)
Date: Tue, 25 Jul 2017 16:25:40 +0200
Subject: [lustre-devel] Design proposal for client-side compression
In-Reply-To: <BN6PR1101MB2132311AB19B104AD044797ACBA40@BN6PR1101MB2132.namprd11.prod.outlook.com>
References: <1483967258.2622.24.camel@informatik.uni-hamburg.de>
	<B5E18A82-79B3-484F-BEBE-998B8AE56397@intel.com>
	<1484223328.3937.60.camel@informatik.uni-hamburg.de>
	<B924BE6D-3C6A-4BA4-96F9-AC053F032B2D@intel.com>
	<1484749185.3965.114.camel@informatik.uni-hamburg.de>
	<1487254536.4777.51.camel@informatik.uni-hamburg.de>
	,<1500650130.3158.6.camel@informatik.uni-hamburg.de>
	<BN6PR1101MB2132311AB19B104AD044797ACBA40@BN6PR1101MB2132.namprd11.prod.outlook.com>
Message-ID: <1500992740.4728.5.camel@informatik.uni-hamburg.de>

Thank you for your responses. 

Patrick, 

On Fri, 2017-07-21 at 16:43 +0000, Patrick Farrell wrote:
> I think basing this on the maximum number of stripes it too simple,
> and maybe not necessary.
> 
> Apologies in advance if what I say below rests on a misunderstanding
> of the compression design, I should know it better than I do.

Probably I still don't get all the relevant internals of Lustre to
clearly describe what we are planning and what we need.

> About based on maximum stripe count, there are a number of 1000 OST
> systems in the world today.  Imagine one of them with 16 MiB stripes,
> that's ~16 GiB of memory for this.  I think that's clearly too
> large.  But a global (rather than per OSC) pool could be tricky too,
> leading to contention on getting and returning pages.

Well, does that mean that one Lustre client handles all the 16GiB of
stripes at the same time or does it somehow iterate over the stripes?
How much memory do we need at most at the same time? If the client
first processes 100 stripes, we need enough memory to compress 100
stripes at the same time. So the question is not about the maximum
stripe count, but the maximum, let me call it queue portion of stripes,
which can be processed at the same time within one client.

> 
> You mention later a 50 MiB pool per client.  As a per OST pre-
> allocated pool, that would likely be too large.  As a global pool, it
> seems small...
> 
> But why use a global pool?  It sounds like the compression would be
> handled by the thread putting the data on the wire (Sorry if I've got
> that wrong).  So - What about a per-thread block of pages, for each
> ptlrpcd thread?  If the idea is that this compressed data is not
> retained for replay (instead, you would re-compress), then we only
> need a block of max rpc size for each thread (You could just use the
> largest RPC size supported by the client), so it can send that
> compressed data.

Yes, we don't really need a very global pool, but still we need to
know, how many threads can there be at the same time? Is there one
thread per stripe or per RPC? And how many in total?

> 
> The overhead of compression for replay is probably not something we
> need to worry about.
> 
> Or even per-CPU blocks of pages.  That would probably be better still
> (less total memory if there are more ptlrpcds than CPUs), if we can
> guarantee not sleeping during the time the pool is in use.  (I'm not
> sure.)
> 
> Also, you mention limiting the # of threads.  Why is limiting the
> number of threads doing compression of interest?  What are you
> specifically trying to avoid with that?

I mean that the number of threads available for compression is somehow
limited. If we have 100 stripes at the same time, we still can compress
with #cores threads, which might be less than 100. So if there are more
stripes in flight than we have resources for compression (since it is
slower), we need to decide whether to slow down everything or to skip
compression of some stripes for overall better performance. 


Jinshan, 

> 
> Is it possible to enable this by writing to a sysfs or procfs entry?
> So that users can try this out without having to recompile Lustre.

The size should be controllable dynamically, but for the feature Lustre
has to be recompiled anyway. 


>   
> It’s not scalable to have a pool per OSC because Lustre can support
> up to 2000 stripes. However, we don’t need to worry about wide stripe
> problem because no one can write a full stripe with even 1MB stripe
> size, because that means application has to issue 2GB size of write.

Does that mean, we have 2000 stripes and we have 2000 messages/RPCs at
the same time? And we need to be able to compress all 2000 stripes at
the same time to avoid blocking? Is there not any limit how many one
client can have at one point of time? 

> Yes, it’s reasonable to have a global pool for each client node.
> Let’s start from this number but please make it adjustable via sysfs
> or procfs.

I am still not sure how large we can do it. Once we really need 16 GiB
for that pool to quickly serve the compression threads, it is not
doable and we have to think different. Once we have a smaller pool than
we need, we have to block or skip compression, which is undesirable.
But I don't know how to determine the required size. 

Thanks! 

Best regards,
Anna


From shreeya.patel23498 at gmail.com  Sat Jul 22 16:56:41 2017
From: shreeya.patel23498 at gmail.com (Shreeya Patel)
Date: Sat, 22 Jul 2017 22:26:41 +0530
Subject: [lustre-devel] [PATCH] Staging: lustre: Place the open brace on
	previous line.
Message-ID: <1500742601-10733-1-git-send-email-shreeya.patel23498@gmail.com>

Remove errors by placing the open braces on previous line.
This saves the vertical space of the code.

Problem found by checkpatch.

Signed-off-by: Shreeya Patel <shreeya.patel23498 at gmail.com>
---
 drivers/staging/lustre/lnet/libcfs/hash.c | 24 ++++++++----------------
 1 file changed, 8 insertions(+), 16 deletions(-)

diff --git a/drivers/staging/lustre/lnet/libcfs/hash.c b/drivers/staging/lustre/lnet/libcfs/hash.c
index 5c2ce2e..a255e04 100644
--- a/drivers/staging/lustre/lnet/libcfs/hash.c
+++ b/drivers/staging/lustre/lnet/libcfs/hash.c
@@ -1348,8 +1348,7 @@ cfs_hash_lookup(struct cfs_hash *hs, const void *key)
 EXPORT_SYMBOL(cfs_hash_lookup);
 
 static void
-cfs_hash_for_each_enter(struct cfs_hash *hs)
-{
+cfs_hash_for_each_enter(struct cfs_hash *hs) {
 	LASSERT(!cfs_hash_is_exiting(hs));
 
 	if (!cfs_hash_with_rehash(hs))
@@ -1375,8 +1374,7 @@ cfs_hash_for_each_enter(struct cfs_hash *hs)
 }
 
 static void
-cfs_hash_for_each_exit(struct cfs_hash *hs)
-{
+cfs_hash_for_each_exit(struct cfs_hash *hs) {
 	int remained;
 	int bits;
 
@@ -1407,8 +1405,7 @@ cfs_hash_for_each_exit(struct cfs_hash *hs)
  */
 static u64
 cfs_hash_for_each_tight(struct cfs_hash *hs, cfs_hash_for_each_cb_t func,
-			void *data, int remove_safe)
-{
+			void *data, int remove_safe) {
 	struct hlist_node *hnode;
 	struct hlist_node *pos;
 	struct cfs_hash_bd bd;
@@ -1501,8 +1498,7 @@ EXPORT_SYMBOL(cfs_hash_for_each);
 
 void
 cfs_hash_for_each_safe(struct cfs_hash *hs, cfs_hash_for_each_cb_t func,
-		       void *data)
-{
+		       void *data) {
 	cfs_hash_for_each_tight(hs, func, data, 1);
 }
 EXPORT_SYMBOL(cfs_hash_for_each_safe);
@@ -1551,8 +1547,7 @@ EXPORT_SYMBOL(cfs_hash_size_get);
  */
 static int
 cfs_hash_for_each_relax(struct cfs_hash *hs, cfs_hash_for_each_cb_t func,
-			void *data, int start)
-{
+			void *data, int start) {
 	struct hlist_node *hnode;
 	struct hlist_node *tmp;
 	struct cfs_hash_bd bd;
@@ -1629,8 +1624,7 @@ cfs_hash_for_each_relax(struct cfs_hash *hs, cfs_hash_for_each_cb_t func,
 
 int
 cfs_hash_for_each_nolock(struct cfs_hash *hs, cfs_hash_for_each_cb_t func,
-			 void *data, int start)
-{
+			 void *data, int start) {
 	if (cfs_hash_with_no_lock(hs) ||
 	    cfs_hash_with_rehash_key(hs) ||
 	    !cfs_hash_with_no_itemref(hs))
@@ -1661,8 +1655,7 @@ EXPORT_SYMBOL(cfs_hash_for_each_nolock);
  */
 int
 cfs_hash_for_each_empty(struct cfs_hash *hs, cfs_hash_for_each_cb_t func,
-			void *data)
-{
+			void *data) {
 	unsigned int i = 0;
 
 	if (cfs_hash_with_no_lock(hs))
@@ -1718,8 +1711,7 @@ EXPORT_SYMBOL(cfs_hash_hlist_for_each);
  */
 void
 cfs_hash_for_each_key(struct cfs_hash *hs, const void *key,
-		      cfs_hash_for_each_cb_t func, void *data)
-{
+		      cfs_hash_for_each_cb_t func, void *data) {
 	struct hlist_node *hnode;
 	struct cfs_hash_bd bds[2];
 	unsigned int i;
-- 
2.7.4

From jsimmons at infradead.org  Wed Jul 26 15:22:16 2017
From: jsimmons at infradead.org (James Simmons)
Date: Wed, 26 Jul 2017 11:22:16 -0400
Subject: [lustre-devel] [PATCH 00/20] staging: lustre: batch of fixes to
	decrease test failures
Message-ID: <1501082556-17962-1-git-send-email-jsimmons@infradead.org>

New batch of patches to reduce the number of the bugs we are
seeing in our testing of the upstream client. This set of
patches is order independent. Removal of some dead code in
the lov layer as well.

Andriy Skulysh (1):
  staging: lustre: ldlm: crash on umount in cleanup_resource

Bob Glosman (1):
  staging: lustre: ptlrpc: print times in microseconds

Bobi Jam (1):
  staging: lustre: osc: soft lock - osc_makes_rpc()

Dmitry Eremin (2):
  staging: lustre: llite: allow cached acls
  staging: lustre: llite: add xattr.h header to xattr.c

Fan Yong (1):
  staging: lustre: linkea: linkEA size limitation

James Simmons (6):
  staging: lustre: ldlm: restore interval_iterate_reverse function
  staging: lustre: lustre: fix all less than 0 comparison for unsigned values
  staging: lustre: ptlrpc: restore 64-bit time for struct ptlrpc_cli_req
  staging: lustre: ptlrpc: don't use CFS_DURATION_T for time64_t
  staging: lustre: libcfs: fix test for libcfs_ioctl_hdr minimum size
  staging: lustre: llite: set security xattr using __vfs_setxattr

John L. Hammond (4):
  staging: lustre: ldlm: restore missing newlines in ldlm sysfs files
  staging: lustre: lov: remove unused code
  staging: lustre: lmv: assume a real connection in lmv_connect()
  staging: lustre: ptlrpc: correct use of list_add_tail()

Nathaniel Clark (1):
  staging: lustre: lov: Ensure correct operation for large object sizes

Niu Yawei (1):
  staging: lustre: ptlrpc: no need to reassign mbits for replay

Robin Humble (1):
  staging: lustre: llite: Remove filtering of seclabel xattr

Sebastien Buisson (1):
  staging: lustre: lov: fix 'control flow' error in lov_io_init_released

 drivers/staging/lustre/lnet/libcfs/hash.c          |  47 ++++--
 .../lustre/lnet/libcfs/linux/linux-module.c        |   2 +-
 drivers/staging/lustre/lustre/include/cl_object.h  |   2 +-
 .../staging/lustre/lustre/include/interval_tree.h  |   4 +
 .../lustre/lustre/include/lustre/lustre_idl.h      |   7 +-
 .../staging/lustre/lustre/include/lustre_linkea.h  |  15 +-
 drivers/staging/lustre/lustre/include/lustre_net.h |  10 +-
 drivers/staging/lustre/lustre/include/obd.h        |   2 -
 drivers/staging/lustre/lustre/include/obd_class.h  |   9 +-
 drivers/staging/lustre/lustre/ldlm/interval_tree.c |  40 +++++
 drivers/staging/lustre/lustre/ldlm/ldlm_internal.h |   3 -
 drivers/staging/lustre/lustre/ldlm/ldlm_resource.c |  47 +-----
 drivers/staging/lustre/lustre/llite/file.c         |  61 +++++++-
 .../staging/lustre/lustre/llite/llite_internal.h   |   4 +
 drivers/staging/lustre/lustre/llite/llite_lib.c    |   6 +-
 drivers/staging/lustre/lustre/llite/namei.c        |   6 +
 drivers/staging/lustre/lustre/llite/rw.c           |   2 +-
 drivers/staging/lustre/lustre/llite/xattr.c        |   1 +
 .../staging/lustre/lustre/llite/xattr_security.c   |  23 ++-
 drivers/staging/lustre/lustre/lmv/lmv_intent.c     |   5 -
 drivers/staging/lustre/lustre/lmv/lmv_internal.h   |   2 -
 drivers/staging/lustre/lustre/lmv/lmv_obd.c        | 161 +++------------------
 drivers/staging/lustre/lustre/lov/Makefile         |   2 +-
 .../staging/lustre/lustre/lov/lov_cl_internal.h    |  90 ------------
 drivers/staging/lustre/lustre/lov/lov_dev.c        |  85 -----------
 drivers/staging/lustre/lustre/lov/lov_ea.c         |  23 +--
 drivers/staging/lustre/lustre/lov/lov_internal.h   |  27 ----
 drivers/staging/lustre/lustre/lov/lov_io.c         | 146 ++++++-------------
 drivers/staging/lustre/lustre/lov/lov_lock.c       |  11 --
 drivers/staging/lustre/lustre/lov/lov_obd.c        |   3 +-
 drivers/staging/lustre/lustre/lov/lov_object.c     |   8 +-
 drivers/staging/lustre/lustre/lov/lov_page.c       |   1 -
 drivers/staging/lustre/lustre/lov/lov_request.c    |  40 +----
 drivers/staging/lustre/lustre/lov/lovsub_dev.c     |   1 -
 drivers/staging/lustre/lustre/lov/lovsub_io.c      |  51 -------
 drivers/staging/lustre/lustre/lov/lovsub_lock.c    |   2 -
 drivers/staging/lustre/lustre/mdc/lproc_mdc.c      |   2 +-
 drivers/staging/lustre/lustre/obdclass/linkea.c    |  69 +++++++--
 drivers/staging/lustre/lustre/osc/osc_cache.c      |  31 +++-
 drivers/staging/lustre/lustre/ptlrpc/client.c      |  35 +++--
 .../staging/lustre/lustre/ptlrpc/lproc_ptlrpc.c    |  15 +-
 drivers/staging/lustre/lustre/ptlrpc/sec_gc.c      |   2 +-
 drivers/staging/lustre/lustre/ptlrpc/service.c     |  15 +-
 drivers/staging/lustre/lustre/ptlrpc/wiretest.c    |  16 +-
 44 files changed, 416 insertions(+), 718 deletions(-)
 delete mode 100644 drivers/staging/lustre/lustre/lov/lovsub_io.c

-- 
1.8.3.1

From jsimmons at infradead.org  Wed Jul 26 15:22:26 2017
From: jsimmons at infradead.org (James Simmons)
Date: Wed, 26 Jul 2017 11:22:26 -0400
Subject: [lustre-devel] [PATCH 10/20] staging: lustre: ldlm: restore
	interval_iterate_reverse function
In-Reply-To: <1501082556-17962-1-git-send-email-jsimmons@infradead.org>
References: <1501082556-17962-1-git-send-email-jsimmons@infradead.org>
Message-ID: <1501082556-17962-11-git-send-email-jsimmons@infradead.org>

Earlier the function interval_iterate_reverse function was
removed since it wasn't used by anyone. Now it is being
restored since it will be used by a future patch.

Signed-off-by: James Simmons <jsimmons at infradead.org>
---
 .../staging/lustre/lustre/include/interval_tree.h  |  4 +++
 drivers/staging/lustre/lustre/ldlm/interval_tree.c | 40 ++++++++++++++++++++++
 2 files changed, 44 insertions(+)

diff --git a/drivers/staging/lustre/lustre/include/interval_tree.h b/drivers/staging/lustre/lustre/include/interval_tree.h
index 0d4f92e..a4d7280 100644
--- a/drivers/staging/lustre/lustre/include/interval_tree.h
+++ b/drivers/staging/lustre/lustre/include/interval_tree.h
@@ -111,4 +111,8 @@ enum interval_iter interval_search(struct interval_node *root,
 				   struct interval_node_extent *ex,
 				   interval_callback_t func, void *data);
 
+enum interval_iter interval_iterate_reverse(struct interval_node *root,
+					    interval_callback_t func,
+					    void *data);
+
 #endif
diff --git a/drivers/staging/lustre/lustre/ldlm/interval_tree.c b/drivers/staging/lustre/lustre/ldlm/interval_tree.c
index e106902..5bebd9a 100644
--- a/drivers/staging/lustre/lustre/ldlm/interval_tree.c
+++ b/drivers/staging/lustre/lustre/ldlm/interval_tree.c
@@ -110,6 +110,15 @@ static struct interval_node *interval_first(struct interval_node *node)
 	return node;
 }
 
+static struct interval_node *interval_last(struct interval_node *node)
+{
+	if (!node)
+		return NULL;
+	while (node->in_right)
+		node = node->in_right;
+	return node;
+}
+
 static struct interval_node *interval_next(struct interval_node *node)
 {
 	if (!node)
@@ -121,6 +130,37 @@ static struct interval_node *interval_next(struct interval_node *node)
 	return node->in_parent;
 }
 
+static struct interval_node *interval_prev(struct interval_node *node)
+{
+	if (!node)
+		return NULL;
+
+	if (node->in_left)
+		return interval_last(node->in_left);
+
+	while (node->in_parent && node_is_left_child(node))
+		node = node->in_parent;
+
+	return node->in_parent;
+}
+
+enum interval_iter interval_iterate_reverse(struct interval_node *root,
+					    interval_callback_t func,
+					    void *data)
+{
+	enum interval_iter rc = INTERVAL_ITER_CONT;
+	struct interval_node *node;
+
+	for (node = interval_last(root); node; node = interval_prev(node)) {
+		rc = func(node, data);
+		if (rc == INTERVAL_ITER_STOP)
+			break;
+	}
+
+	return rc;
+}
+EXPORT_SYMBOL(interval_iterate_reverse);
+
 static void __rotate_change_maxhigh(struct interval_node *node,
 				    struct interval_node *rotate)
 {
-- 
1.8.3.1

From jsimmons at infradead.org  Wed Jul 26 15:22:29 2017
From: jsimmons at infradead.org (James Simmons)
Date: Wed, 26 Jul 2017 11:22:29 -0400
Subject: [lustre-devel] [PATCH 13/20] staging: lustre: ptlrpc: restore
	64-bit time for struct ptlrpc_cli_req
In-Reply-To: <1501082556-17962-1-git-send-email-jsimmons@infradead.org>
References: <1501082556-17962-1-git-send-email-jsimmons@infradead.org>
Message-ID: <1501082556-17962-14-git-send-email-jsimmons@infradead.org>

During the reorganization of ptlrpc_request some of the
time64_t fields were incorrectly turned into time_t.
Restore those fields back to time_64_t.

Signed-off-by: James Simmons <uja.ornl at yahoo.com>
Reviewed-on: https://review.whamcloud.com/24977
Intel-bug-id: https://jira.hpdd.intel.com/browse/LU-4423
Fixes: 32c8728d87dc ("staging/lustre/ptlrpc: reorganize ptlrpc_request")
CC: Liang Zhen <liang.zhen at intel.com>
Reviewed-by: Bob Glossman <bob.glossman at intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger at intel.com>
Signed-off-by: James Simmons <jsimmons at infradead.org>
---
 drivers/staging/lustre/lustre/include/lustre_net.h | 10 +++++-----
 drivers/staging/lustre/lustre/ptlrpc/client.c      |  2 +-
 2 files changed, 6 insertions(+), 6 deletions(-)

diff --git a/drivers/staging/lustre/lustre/include/lustre_net.h b/drivers/staging/lustre/lustre/include/lustre_net.h
index d61b000..05f02b1 100644
--- a/drivers/staging/lustre/lustre/include/lustre_net.h
+++ b/drivers/staging/lustre/lustre/include/lustre_net.h
@@ -558,13 +558,13 @@ struct ptlrpc_cli_req {
 	/** request sent timeval */
 	struct timespec64		 cr_sent_tv;
 	/** time for request really sent out */
-	time_t				 cr_sent_out;
+	time64_t			 cr_sent_out;
 	/** when req reply unlink must finish. */
-	time_t				 cr_reply_deadline;
+	time64_t			 cr_reply_deadline;
 	/** when req bulk unlink must finish. */
-	time_t				 cr_bulk_deadline;
+	time64_t			 cr_bulk_deadline;
 	/** when req unlink must finish. */
-	time_t				 cr_req_deadline;
+	time64_t			 cr_req_deadline;
 	/** Portal to which this request would be sent */
 	short				 cr_req_ptl;
 	/** Portal where to wait for reply and where reply would be sent */
@@ -663,7 +663,7 @@ struct ptlrpc_srv_req {
 	/** history sequence # */
 	__u64				sr_hist_seq;
 	/** the index of service's srv_at_array into which request is linked */
-	time_t				sr_at_index;
+	time64_t			sr_at_index;
 	/** authed uid */
 	uid_t				sr_auth_uid;
 	/** authed uid mapped to */
diff --git a/drivers/staging/lustre/lustre/ptlrpc/client.c b/drivers/staging/lustre/lustre/ptlrpc/client.c
index 1247686..9f4f17a 100644
--- a/drivers/staging/lustre/lustre/ptlrpc/client.c
+++ b/drivers/staging/lustre/lustre/ptlrpc/client.c
@@ -742,7 +742,7 @@ int ptlrpc_request_bufs_pack(struct ptlrpc_request *request,
 
 	/* Let's setup deadline for req/reply/bulk unlink for opcode. */
 	if (cfs_fail_val == opcode) {
-		time_t *fail_t = NULL, *fail2_t = NULL;
+		time64_t *fail_t = NULL, *fail2_t = NULL;
 
 		if (CFS_FAIL_CHECK(OBD_FAIL_PTLRPC_LONG_BULK_UNLINK)) {
 			fail_t = &request->rq_bulk_deadline;
-- 
1.8.3.1

From jsimmons at infradead.org  Wed Jul 26 15:22:34 2017
From: jsimmons at infradead.org (James Simmons)
Date: Wed, 26 Jul 2017 11:22:34 -0400
Subject: [lustre-devel] [PATCH 18/20] staging: lustre: llite: Remove
	filtering of seclabel xattr
In-Reply-To: <1501082556-17962-1-git-send-email-jsimmons@infradead.org>
References: <1501082556-17962-1-git-send-email-jsimmons@infradead.org>
Message-ID: <1501082556-17962-19-git-send-email-jsimmons@infradead.org>

From: Robin Humble <plaguedbypenguins at gmail.com>

The security.capability xattr is used to implement File
Capabilities in recent Linux versions. Capabilities are a
fine grained approach to granting executables elevated
privileges. eg. /bin/ping can have capabilities
cap_net_admin, cap_net_raw+ep instead of being setuid root.

This xattr has long been filtered out by llite, initially for
stability reasons (b15587), and later over performance
concerns as this xattr is read for every file with eg.
'ls --color'. Since LU-2869 xattr's are cached on clients,
alleviating most performance concerns.

Removing llite's filtering of the security.capability xattr
enables using Lustre as a root filesystem, which is used on
some large clusters.

Signed-off-by: Robin Humble <plaguedbypenguins at gmail.com>
Intel-bug-id: https://jira.hpdd.intel.com/browse/LU-9562
Reviewed-on: https://review.whamcloud.com/27292
Reviewed-by: John L. Hammond <john.hammond at intel.com>
Reviewed-by: Sebastien Buisson <sbuisson at ddn.com>
Reviewed-by: Oleg Drokin <oleg.drokin at intel.com>
Signed-off-by: James Simmons <jsimmons at infradead.org>
---
 drivers/staging/lustre/lustre/llite/file.c         | 58 ++++++++++++++++++++++
 .../staging/lustre/lustre/llite/llite_internal.h   |  4 ++
 drivers/staging/lustre/lustre/llite/namei.c        |  6 +++
 3 files changed, 68 insertions(+)

diff --git a/drivers/staging/lustre/lustre/llite/file.c b/drivers/staging/lustre/lustre/llite/file.c
index 215479a..a324580 100644
--- a/drivers/staging/lustre/lustre/llite/file.c
+++ b/drivers/staging/lustre/lustre/llite/file.c
@@ -3040,6 +3040,61 @@ struct posix_acl *ll_get_acl(struct inode *inode, int type)
 	return acl;
 }
 
+#ifdef CONFIG_FS_POSIX_ACL
+int ll_set_acl(struct inode *inode, struct posix_acl *acl, int type)
+{
+	const char *name = NULL;
+	char *value = NULL;
+	size_t size = 0;
+	int rc = 0;
+
+	switch (type) {
+	case ACL_TYPE_ACCESS:
+		if (acl) {
+			rc = posix_acl_update_mode(inode, &inode->i_mode, &acl);
+			if (rc)
+				goto out;
+		}
+		name = XATTR_NAME_POSIX_ACL_ACCESS;
+		break;
+	case ACL_TYPE_DEFAULT:
+		if (!S_ISDIR(inode->i_mode)) {
+			rc = acl ? -EACCES : 0;
+			goto out;
+		}
+		name = XATTR_NAME_POSIX_ACL_DEFAULT;
+		break;
+	default:
+		rc = -EINVAL;
+		goto out;
+	}
+
+	if (acl) {
+		size = posix_acl_xattr_size(acl->a_count);
+		value = kmalloc(size, GFP_NOFS);
+		if (!value) {
+			rc = -ENOMEM;
+			goto out;
+		}
+
+		rc = posix_acl_to_xattr(&init_user_ns, acl, value, size);
+		if (rc < 0)
+			goto out_free;
+	}
+
+	/* dentry is only used for *.lov attributes so it's safe to be NULL */
+	rc = __vfs_setxattr(NULL, inode, name, value, size, XATTR_CREATE);
+out_free:
+	kfree(value);
+out:
+	if (!rc)
+		set_cached_acl(inode, type, acl);
+	else
+		forget_cached_acl(inode, type);
+	return rc;
+}
+#endif /* CONFIG_FS_POSIX_ACL */
+
 int ll_inode_permission(struct inode *inode, int mask)
 {
 	struct ll_sb_info *sbi;
@@ -3162,6 +3217,9 @@ int ll_inode_permission(struct inode *inode, int mask)
 	.listxattr	= ll_listxattr,
 	.fiemap		= ll_fiemap,
 	.get_acl	= ll_get_acl,
+#ifdef CONFIG_FS_POSIX_ACL
+	.set_acl	= ll_set_acl,
+#endif
 };
 
 /* dynamic ioctl number support routines */
diff --git a/drivers/staging/lustre/lustre/llite/llite_internal.h b/drivers/staging/lustre/lustre/llite/llite_internal.h
index cd3311a..b3374bc 100644
--- a/drivers/staging/lustre/lustre/llite/llite_internal.h
+++ b/drivers/staging/lustre/lustre/llite/llite_internal.h
@@ -752,6 +752,10 @@ enum ldlm_mode ll_take_md_lock(struct inode *inode, __u64 bits,
 int ll_getattr(const struct path *path, struct kstat *stat,
 	       u32 request_mask, unsigned int flags);
 struct posix_acl *ll_get_acl(struct inode *inode, int type);
+#ifdef CONFIG_FS_POSIX_ACL
+int ll_set_acl(struct inode *inode, struct posix_acl *acl, int type);
+#endif /* CONFIG_FS_POSIX_ACL */
+
 int ll_migrate(struct inode *parent, struct file *file, int mdtidx,
 	       const char *name, int namelen);
 int ll_get_fid_by_name(struct inode *parent, const char *name,
diff --git a/drivers/staging/lustre/lustre/llite/namei.c b/drivers/staging/lustre/lustre/llite/namei.c
index a208a8b..14dccbe 100644
--- a/drivers/staging/lustre/lustre/llite/namei.c
+++ b/drivers/staging/lustre/lustre/llite/namei.c
@@ -1187,6 +1187,9 @@ static int ll_rename(struct inode *src, struct dentry *src_dchild,
 	.permission	 = ll_inode_permission,
 	.listxattr	  = ll_listxattr,
 	.get_acl	    = ll_get_acl,
+#ifdef CONFIG_FS_POSIX_ACL
+	.set_acl	= ll_set_acl,
+#endif
 };
 
 const struct inode_operations ll_special_inode_operations = {
@@ -1195,4 +1198,7 @@ static int ll_rename(struct inode *src, struct dentry *src_dchild,
 	.permission     = ll_inode_permission,
 	.listxattr      = ll_listxattr,
 	.get_acl	    = ll_get_acl,
+#ifdef CONFIG_FS_POSIX_ACL
+	.set_acl	= ll_set_acl,
+#endif
 };
-- 
1.8.3.1

From jsimmons at infradead.org  Wed Jul 26 15:22:22 2017
From: jsimmons at infradead.org (James Simmons)
Date: Wed, 26 Jul 2017 11:22:22 -0400
Subject: [lustre-devel] [PATCH 06/20] staging: lustre: lmv: assume a real
	connection in lmv_connect()
In-Reply-To: <1501082556-17962-1-git-send-email-jsimmons@infradead.org>
References: <1501082556-17962-1-git-send-email-jsimmons@infradead.org>
Message-ID: <1501082556-17962-7-git-send-email-jsimmons@infradead.org>

From: "John L. Hammond" <john.hammond at intel.com>

Assume a real connection in lmv_connect(). Mark OBD_CONNECT_REAL
obsolete. Remove the then unnecessary refcount and exp members of
struct lmv_obd. Remove calls to lmv_check_connect(). Disconnect the
export in the appropriate error path of lmv_connect().

Signed-off-by: John L. Hammond <john.hammond at intel.com>
Intel-bug-id: https://jira.hpdd.intel.com/browse/LU-7669
Reviewed-on: http://review.whamcloud.com/18018
Reviewed-by: Andreas Dilger <andreas.dilger at intel.com>
Reviewed-by: wangdi <di.wang at intel.com>
Signed-off-by: James Simmons <jsimmons at infradead.org>
---
 .../lustre/lustre/include/lustre/lustre_idl.h      |   2 +-
 drivers/staging/lustre/lustre/include/obd.h        |   2 -
 drivers/staging/lustre/lustre/llite/llite_lib.c    |   3 -
 drivers/staging/lustre/lustre/lmv/lmv_intent.c     |   5 -
 drivers/staging/lustre/lustre/lmv/lmv_internal.h   |   2 -
 drivers/staging/lustre/lustre/lmv/lmv_obd.c        | 161 +++------------------
 6 files changed, 23 insertions(+), 152 deletions(-)

diff --git a/drivers/staging/lustre/lustre/include/lustre/lustre_idl.h b/drivers/staging/lustre/lustre/include/lustre/lustre_idl.h
index 77995fa..441a2e3 100644
--- a/drivers/staging/lustre/lustre/include/lustre/lustre_idl.h
+++ b/drivers/staging/lustre/lustre/include/lustre/lustre_idl.h
@@ -1122,7 +1122,7 @@ struct ptlrpc_body_v2 {
 #define OBD_CONNECT_AT		    0x1000000ULL /*client uses AT */
 #define OBD_CONNECT_LRU_RESIZE      0x2000000ULL /*LRU resize feature. */
 #define OBD_CONNECT_MDS_MDS	    0x4000000ULL /*MDS-MDS connection */
-#define OBD_CONNECT_REAL	    0x8000000ULL /*real connection */
+#define OBD_CONNECT_REAL	    0x8000000ULL /* obsolete since 2.8 */
 #define OBD_CONNECT_CHANGE_QS      0x10000000ULL /*Not used since 2.4 */
 #define OBD_CONNECT_CKSUM	   0x20000000ULL /*support several cksum algos*/
 #define OBD_CONNECT_FID		   0x40000000ULL /*FID is supported by server */
diff --git a/drivers/staging/lustre/lustre/include/obd.h b/drivers/staging/lustre/lustre/include/obd.h
index 4ce8506..4dfc31e 100644
--- a/drivers/staging/lustre/lustre/include/obd.h
+++ b/drivers/staging/lustre/lustre/include/obd.h
@@ -404,12 +404,10 @@ struct lmv_tgt_desc {
 };
 
 struct lmv_obd {
-	int			refcount;
 	struct lu_client_fld	lmv_fld;
 	spinlock_t		lmv_lock;
 	struct lmv_desc		desc;
 	struct obd_uuid		cluuid;
-	struct obd_export	*exp;
 
 	struct mutex		lmv_init_mutex;
 	int			connected;
diff --git a/drivers/staging/lustre/lustre/llite/llite_lib.c b/drivers/staging/lustre/lustre/llite/llite_lib.c
index 974a05d..69d895d 100644
--- a/drivers/staging/lustre/lustre/llite/llite_lib.c
+++ b/drivers/staging/lustre/lustre/llite/llite_lib.c
@@ -222,9 +222,6 @@ static int client_common_fill_super(struct super_block *sb, char *md, char *dt,
 	else
 		sbi->ll_fop = &ll_file_operations_noflock;
 
-	/* real client */
-	data->ocd_connect_flags |= OBD_CONNECT_REAL;
-
 	/* always ping even if server suppress_pings */
 	if (sbi->ll_flags & LL_SBI_ALWAYS_PING)
 		data->ocd_connect_flags &= ~OBD_CONNECT_PINGLESS;
diff --git a/drivers/staging/lustre/lustre/lmv/lmv_intent.c b/drivers/staging/lustre/lustre/lmv/lmv_intent.c
index f49db6c..5f01a0c 100644
--- a/drivers/staging/lustre/lustre/lmv/lmv_intent.c
+++ b/drivers/staging/lustre/lustre/lmv/lmv_intent.c
@@ -474,7 +474,6 @@ int lmv_intent_lock(struct obd_export *exp, struct md_op_data *op_data,
 		    ldlm_blocking_callback cb_blocking,
 		    __u64 extra_lock_flags)
 {
-	struct obd_device *obd = exp->exp_obd;
 	int		rc;
 
 	LASSERT(fid_is_sane(&op_data->op_fid1));
@@ -484,10 +483,6 @@ int lmv_intent_lock(struct obd_export *exp, struct md_op_data *op_data,
 	       (int)op_data->op_namelen, op_data->op_name,
 	       PFID(&op_data->op_fid1));
 
-	rc = lmv_check_connect(obd);
-	if (rc)
-		return rc;
-
 	if (it->it_op & (IT_LOOKUP | IT_GETATTR | IT_LAYOUT))
 		rc = lmv_intent_lookup(exp, op_data, it, reqp, cb_blocking,
 				       extra_lock_flags);
diff --git a/drivers/staging/lustre/lustre/lmv/lmv_internal.h b/drivers/staging/lustre/lustre/lmv/lmv_internal.h
index 12731a17..e839b83 100644
--- a/drivers/staging/lustre/lustre/lmv/lmv_internal.h
+++ b/drivers/staging/lustre/lustre/lmv/lmv_internal.h
@@ -42,8 +42,6 @@
 #define LL_IT2STR(it)					\
 	((it) ? ldlm_it2str((it)->it_op) : "0")
 
-int lmv_check_connect(struct obd_device *obd);
-
 int lmv_intent_lock(struct obd_export *exp, struct md_op_data *op_data,
 		    struct lookup_intent *it, struct ptlrpc_request **reqp,
 		    ldlm_blocking_callback cb_blocking,
diff --git a/drivers/staging/lustre/lustre/lmv/lmv_obd.c b/drivers/staging/lustre/lustre/lmv/lmv_obd.c
index 64fcaef..1c9ff24 100644
--- a/drivers/staging/lustre/lustre/lmv/lmv_obd.c
+++ b/drivers/staging/lustre/lustre/lmv/lmv_obd.c
@@ -53,6 +53,8 @@
 #include "../include/lustre_kernelcomm.h"
 #include "lmv_internal.h"
 
+static int lmv_check_connect(struct obd_device *obd);
+
 static void lmv_activate_target(struct lmv_obd *lmv,
 				struct lmv_tgt_desc *tgt,
 				int activate)
@@ -183,59 +185,44 @@ static int lmv_notify(struct obd_device *obd, struct obd_device *watched,
 	return rc;
 }
 
-/**
- * This is fake connect function. Its purpose is to initialize lmv and say
- * caller that everything is okay. Real connection will be performed later.
- */
 static int lmv_connect(const struct lu_env *env,
-		       struct obd_export **exp, struct obd_device *obd,
+		       struct obd_export **pexp, struct obd_device *obd,
 		       struct obd_uuid *cluuid, struct obd_connect_data *data,
 		       void *localdata)
 {
 	struct lmv_obd	*lmv = &obd->u.lmv;
 	struct lustre_handle  conn = { 0 };
+	struct obd_export *exp;
 	int		    rc = 0;
 
-	/*
-	 * We don't want to actually do the underlying connections more than
-	 * once, so keep track.
-	 */
-	lmv->refcount++;
-	if (lmv->refcount > 1) {
-		*exp = NULL;
-		return 0;
-	}
-
 	rc = class_connect(&conn, obd, cluuid);
 	if (rc) {
 		CERROR("class_connection() returned %d\n", rc);
 		return rc;
 	}
 
-	*exp = class_conn2export(&conn);
-	class_export_get(*exp);
+	exp = class_conn2export(&conn);
 
-	lmv->exp = *exp;
 	lmv->connected = 0;
 	lmv->cluuid = *cluuid;
-
-	if (data)
-		lmv->conn_data = *data;
+	lmv->conn_data = *data;
 
 	lmv->lmv_tgts_kobj = kobject_create_and_add("target_obds",
 						    &obd->obd_kobj);
-	/*
-	 * All real clients should perform actual connection right away, because
-	 * it is possible, that LMV will not have opportunity to connect targets
-	 * and MDC stuff will be called directly, for instance while reading
-	 * ../mdc/../kbytesfree procfs file, etc.
-	 */
-	if (data && data->ocd_connect_flags & OBD_CONNECT_REAL)
-		rc = lmv_check_connect(obd);
+	rc = lmv_check_connect(obd);
+	if (rc)
+		goto out_sysfs;
+
+	*pexp = exp;
 
-	if (rc && lmv->lmv_tgts_kobj)
+	return rc;
+
+out_sysfs:
+	if (lmv->lmv_tgts_kobj)
 		kobject_put(lmv->lmv_tgts_kobj);
 
+	class_disconnect(exp);
+
 	return rc;
 }
 
@@ -475,7 +462,7 @@ static int lmv_add_target(struct obd_device *obd, struct obd_uuid *uuidp,
 	return rc;
 }
 
-int lmv_check_connect(struct obd_device *obd)
+static int lmv_check_connect(struct obd_device *obd)
 {
 	struct lmv_obd       *lmv = &obd->u.lmv;
 	struct lmv_tgt_desc  *tgt;
@@ -519,7 +506,6 @@ int lmv_check_connect(struct obd_device *obd)
 			goto out_disc;
 	}
 
-	class_export_put(lmv->exp);
 	lmv->connected = 1;
 	easize = lmv_mds_md_size(lmv->desc.ld_tgt_count, LMV_MAGIC);
 	lmv_init_ea_size(obd->obd_self_export, easize, 0);
@@ -543,7 +529,7 @@ int lmv_check_connect(struct obd_device *obd)
 			}
 		}
 	}
-	class_disconnect(lmv->exp);
+
 	mutex_unlock(&lmv->lmv_init_mutex);
 	return rc;
 }
@@ -598,13 +584,6 @@ static int lmv_disconnect(struct obd_export *exp)
 	if (!lmv->tgts)
 		goto out_local;
 
-	/*
-	 * Only disconnect the underlying layers on the final disconnect.
-	 */
-	lmv->refcount--;
-	if (lmv->refcount != 0)
-		goto out_local;
-
 	for (i = 0; i < lmv->desc.ld_tgt_count; i++) {
 		if (!lmv->tgts[i] || !lmv->tgts[i]->ltd_exp)
 			continue;
@@ -623,8 +602,7 @@ static int lmv_disconnect(struct obd_export *exp)
 	if (!lmv->connected)
 		class_export_put(exp);
 	rc = class_disconnect(exp);
-	if (lmv->refcount == 0)
-		lmv->connected = 0;
+	lmv->connected = 0;
 	return rc;
 }
 
@@ -1122,7 +1100,8 @@ static int lmv_iocontrol(unsigned int cmd, struct obd_export *exp,
 			err = obd_iocontrol(cmd, tgt->ltd_exp, len, karg, uarg);
 			if (err) {
 				if (tgt->ltd_active) {
-					CERROR("error: iocontrol MDC %s on MDTidx %d cmd %x: err = %d\n",
+					CERROR("%s: error: iocontrol MDC %s on MDTidx %d cmd %x: err = %d\n",
+					       lmv2obd_dev(lmv)->obd_name,
 					       tgt->ltd_uuid.uuid, i, cmd, err);
 					if (!rc)
 						rc = err;
@@ -1368,10 +1347,6 @@ static int lmv_statfs(const struct lu_env *env, struct obd_export *exp,
 	int		    rc = 0;
 	u32 i;
 
-	rc = lmv_check_connect(obd);
-	if (rc)
-		return rc;
-
 	temp = kzalloc(sizeof(*temp), GFP_NOFS);
 	if (!temp)
 		return -ENOMEM;
@@ -1418,11 +1393,6 @@ static int lmv_getstatus(struct obd_export *exp,
 {
 	struct obd_device    *obd = exp->exp_obd;
 	struct lmv_obd       *lmv = &obd->u.lmv;
-	int		   rc;
-
-	rc = lmv_check_connect(obd);
-	if (rc)
-		return rc;
 
 	return md_getstatus(lmv->tgts[0]->ltd_exp, fid);
 }
@@ -1435,11 +1405,6 @@ static int lmv_getxattr(struct obd_export *exp, const struct lu_fid *fid,
 	struct obd_device      *obd = exp->exp_obd;
 	struct lmv_obd	 *lmv = &obd->u.lmv;
 	struct lmv_tgt_desc    *tgt;
-	int		     rc;
-
-	rc = lmv_check_connect(obd);
-	if (rc)
-		return rc;
 
 	tgt = lmv_find_target(lmv, fid);
 	if (IS_ERR(tgt))
@@ -1458,11 +1423,6 @@ static int lmv_setxattr(struct obd_export *exp, const struct lu_fid *fid,
 	struct obd_device      *obd = exp->exp_obd;
 	struct lmv_obd	 *lmv = &obd->u.lmv;
 	struct lmv_tgt_desc    *tgt;
-	int		     rc;
-
-	rc = lmv_check_connect(obd);
-	if (rc)
-		return rc;
 
 	tgt = lmv_find_target(lmv, fid);
 	if (IS_ERR(tgt))
@@ -1479,11 +1439,6 @@ static int lmv_getattr(struct obd_export *exp, struct md_op_data *op_data,
 	struct obd_device       *obd = exp->exp_obd;
 	struct lmv_obd	  *lmv = &obd->u.lmv;
 	struct lmv_tgt_desc     *tgt;
-	int		      rc;
-
-	rc = lmv_check_connect(obd);
-	if (rc)
-		return rc;
 
 	tgt = lmv_find_target(lmv, &op_data->op_fid1);
 	if (IS_ERR(tgt))
@@ -1502,11 +1457,6 @@ static int lmv_null_inode(struct obd_export *exp, const struct lu_fid *fid)
 	struct obd_device   *obd = exp->exp_obd;
 	struct lmv_obd      *lmv = &obd->u.lmv;
 	u32 i;
-	int		  rc;
-
-	rc = lmv_check_connect(obd);
-	if (rc)
-		return rc;
 
 	CDEBUG(D_INODE, "CBDATA for " DFID "\n", PFID(fid));
 
@@ -1530,11 +1480,6 @@ static int lmv_close(struct obd_export *exp, struct md_op_data *op_data,
 	struct obd_device     *obd = exp->exp_obd;
 	struct lmv_obd	*lmv = &obd->u.lmv;
 	struct lmv_tgt_desc   *tgt;
-	int		    rc;
-
-	rc = lmv_check_connect(obd);
-	if (rc)
-		return rc;
 
 	tgt = lmv_find_target(lmv, &op_data->op_fid1);
 	if (IS_ERR(tgt))
@@ -1661,10 +1606,6 @@ static int lmv_create(struct obd_export *exp, struct md_op_data *op_data,
 	struct lmv_tgt_desc     *tgt;
 	int		      rc;
 
-	rc = lmv_check_connect(obd);
-	if (rc)
-		return rc;
-
 	if (!lmv->desc.ld_active_tgt_count)
 		return -EIO;
 
@@ -1718,11 +1659,6 @@ static int lmv_create(struct obd_export *exp, struct md_op_data *op_data,
 	struct obd_device	*obd = exp->exp_obd;
 	struct lmv_obd	   *lmv = &obd->u.lmv;
 	struct lmv_tgt_desc      *tgt;
-	int		       rc;
-
-	rc = lmv_check_connect(obd);
-	if (rc)
-		return rc;
 
 	CDEBUG(D_INODE, "ENQUEUE '%s' on " DFID "\n",
 	       LL_IT2STR(it), PFID(&op_data->op_fid1));
@@ -1749,10 +1685,6 @@ static int lmv_create(struct obd_export *exp, struct md_op_data *op_data,
 	struct mdt_body	 *body;
 	int		      rc;
 
-	rc = lmv_check_connect(obd);
-	if (rc)
-		return rc;
-
 	tgt = lmv_locate_mds(lmv, op_data, &op_data->op_fid1);
 	if (IS_ERR(tgt))
 		return PTR_ERR(tgt);
@@ -1845,10 +1777,6 @@ static int lmv_link(struct obd_export *exp, struct md_op_data *op_data,
 	struct lmv_tgt_desc     *tgt;
 	int		      rc;
 
-	rc = lmv_check_connect(obd);
-	if (rc)
-		return rc;
-
 	LASSERT(op_data->op_namelen != 0);
 
 	CDEBUG(D_INODE, "LINK " DFID ":%*s to " DFID "\n",
@@ -1907,10 +1835,6 @@ static int lmv_rename(struct obd_export *exp, struct md_op_data *op_data,
 	       (int)newlen, new, PFID(&op_data->op_fid2),
 	       op_data->op_mea2 ? op_data->op_mea2->lsm_md_stripe_count : 0);
 
-	rc = lmv_check_connect(obd);
-	if (rc)
-		return rc;
-
 	op_data->op_fsuid = from_kuid(&init_user_ns, current_fsuid());
 	op_data->op_fsgid = from_kgid(&init_user_ns, current_fsgid());
 	op_data->op_cap = cfs_curproc_cap_pack();
@@ -2063,11 +1987,6 @@ static int lmv_setattr(struct obd_export *exp, struct md_op_data *op_data,
 	struct obd_device       *obd = exp->exp_obd;
 	struct lmv_obd	  *lmv = &obd->u.lmv;
 	struct lmv_tgt_desc     *tgt;
-	int		      rc;
-
-	rc = lmv_check_connect(obd);
-	if (rc)
-		return rc;
 
 	CDEBUG(D_INODE, "SETATTR for " DFID ", valid 0x%x\n",
 	       PFID(&op_data->op_fid1), op_data->op_attr.ia_valid);
@@ -2086,11 +2005,6 @@ static int lmv_sync(struct obd_export *exp, const struct lu_fid *fid,
 	struct obd_device	 *obd = exp->exp_obd;
 	struct lmv_obd	    *lmv = &obd->u.lmv;
 	struct lmv_tgt_desc       *tgt;
-	int			rc;
-
-	rc = lmv_check_connect(obd);
-	if (rc)
-		return rc;
 
 	tgt = lmv_find_target(lmv, fid);
 	if (IS_ERR(tgt))
@@ -2272,7 +2186,6 @@ static int lmv_read_striped_page(struct obd_export *exp,
 {
 	struct inode *master_inode = op_data->op_data;
 	struct lu_fid master_fid = op_data->op_fid1;
-	struct obd_device *obd = exp->exp_obd;
 	__u64 hash_offset = offset;
 	__u32 ldp_flags;
 	struct page *min_ent_page = NULL;
@@ -2286,10 +2199,6 @@ static int lmv_read_striped_page(struct obd_export *exp,
 	void *area;
 	int rc;
 
-	rc = lmv_check_connect(obd);
-	if (rc)
-		return rc;
-
 	/*
 	 * Allocate a page and read entries from all of stripes and fill
 	 * the page by hash order
@@ -2408,11 +2317,6 @@ static int lmv_read_page(struct obd_export *exp, struct md_op_data *op_data,
 	struct obd_device *obd = exp->exp_obd;
 	struct lmv_obd *lmv = &obd->u.lmv;
 	struct lmv_tgt_desc *tgt;
-	int rc;
-
-	rc = lmv_check_connect(obd);
-	if (rc)
-		return rc;
 
 	if (unlikely(lsm))
 		return lmv_read_striped_page(exp, op_data, cb_op, offset, ppage);
@@ -2460,9 +2364,6 @@ static int lmv_unlink(struct obd_export *exp, struct md_op_data *op_data,
 	int stripe_index = 0;
 	int		     rc;
 
-	rc = lmv_check_connect(obd);
-	if (rc)
-		return rc;
 retry_unlink:
 	/* For striped dir, we need to locate the parent as well */
 	if (lsm) {
@@ -2647,10 +2548,6 @@ static int lmv_get_info(const struct lu_env *env, struct obd_export *exp,
 	if (keylen >= strlen("remote_flag") && !strcmp(key, "remote_flag")) {
 		int i;
 
-		rc = lmv_check_connect(obd);
-		if (rc)
-			return rc;
-
 		LASSERT(*vallen == sizeof(__u32));
 		for (i = 0; i < lmv->desc.ld_tgt_count; i++) {
 			struct lmv_tgt_desc *tgt = lmv->tgts[i];
@@ -2669,10 +2566,6 @@ static int lmv_get_info(const struct lu_env *env, struct obd_export *exp,
 	} else if (KEY_IS(KEY_MAX_EASIZE) ||
 		   KEY_IS(KEY_DEFAULT_EASIZE) ||
 		   KEY_IS(KEY_CONN_DATA)) {
-		rc = lmv_check_connect(obd);
-		if (rc)
-			return rc;
-
 		/*
 		 * Forwarding this request to first MDS, it should know LOV
 		 * desc.
@@ -3021,15 +2914,10 @@ static int lmv_intent_getattr_async(struct obd_export *exp,
 	struct lmv_obd	  *lmv = &obd->u.lmv;
 	struct lmv_tgt_desc *ptgt = NULL;
 	struct lmv_tgt_desc *ctgt = NULL;
-	int		      rc;
 
 	if (!fid_is_sane(&op_data->op_fid2))
 		return -EINVAL;
 
-	rc = lmv_check_connect(obd);
-	if (rc)
-		return rc;
-
 	ptgt = lmv_locate_mds(lmv, op_data, &op_data->op_fid1);
 	if (IS_ERR(ptgt))
 		return PTR_ERR(ptgt);
@@ -3056,11 +2944,6 @@ static int lmv_revalidate_lock(struct obd_export *exp, struct lookup_intent *it,
 	struct obd_device       *obd = exp->exp_obd;
 	struct lmv_obd	  *lmv = &obd->u.lmv;
 	struct lmv_tgt_desc     *tgt;
-	int		      rc;
-
-	rc = lmv_check_connect(obd);
-	if (rc)
-		return rc;
 
 	tgt = lmv_find_target(lmv, fid);
 	if (IS_ERR(tgt))
-- 
1.8.3.1

From jsimmons at infradead.org  Wed Jul 26 15:22:24 2017
From: jsimmons at infradead.org (James Simmons)
Date: Wed, 26 Jul 2017 11:22:24 -0400
Subject: [lustre-devel] [PATCH 08/20] staging: lustre: ptlrpc: correct use
	of list_add_tail()
In-Reply-To: <1501082556-17962-1-git-send-email-jsimmons@infradead.org>
References: <1501082556-17962-1-git-send-email-jsimmons@infradead.org>
Message-ID: <1501082556-17962-9-git-send-email-jsimmons@infradead.org>

From: "John L. Hammond" <john.hammond at intel.com>

With people starting to test security enabled ptlrpc
a list_del corruption was reported. The reason for
this error was tracked down to wrong arguments to
list_add_tail(). In sptlrpc_gc_add_sec() swap the
arguments to list_add_tail() so that it does what
we meant it to do.

Signed-off-by: John L. Hammond <john.hammond at intel.com>
Intel-bug-id: https://jira.hpdd.intel.com/browse/LU-8270
Reviewed-on: http://review.whamcloud.com/20784
Reviewed-by: Andreas Dilger <andreas.dilger at intel.com>
Reviewed-by: Bob Glossman <bob.glossman at intel.com>
Reviewed-by: Dmitry Eremin <dmitry.eremin at intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin at intel.com>
Signed-off-by: James Simmons <jsimmons at infradead.org>
---
 drivers/staging/lustre/lustre/ptlrpc/sec_gc.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/staging/lustre/lustre/ptlrpc/sec_gc.c b/drivers/staging/lustre/lustre/ptlrpc/sec_gc.c
index 8ffd000..026bec7 100644
--- a/drivers/staging/lustre/lustre/ptlrpc/sec_gc.c
+++ b/drivers/staging/lustre/lustre/ptlrpc/sec_gc.c
@@ -66,7 +66,7 @@ void sptlrpc_gc_add_sec(struct ptlrpc_sec *sec)
 	sec->ps_gc_next = ktime_get_real_seconds() + sec->ps_gc_interval;
 
 	spin_lock(&sec_gc_list_lock);
-	list_add_tail(&sec_gc_list, &sec->ps_gc_list);
+	list_add_tail(&sec->ps_gc_list, &sec_gc_list);
 	spin_unlock(&sec_gc_list_lock);
 
 	CDEBUG(D_SEC, "added sec %p(%s)\n", sec, sec->ps_policy->sp_name);
-- 
1.8.3.1

From jsimmons at infradead.org  Wed Jul 26 15:22:25 2017
From: jsimmons at infradead.org (James Simmons)
Date: Wed, 26 Jul 2017 11:22:25 -0400
Subject: [lustre-devel] [PATCH 09/20] staging: lustre: ptlrpc: no need to
	reassign mbits for replay
In-Reply-To: <1501082556-17962-1-git-send-email-jsimmons@infradead.org>
References: <1501082556-17962-1-git-send-email-jsimmons@infradead.org>
Message-ID: <1501082556-17962-10-git-send-email-jsimmons@infradead.org>

From: Niu Yawei <yawei.niu at intel.com>

It's not necessary reassgin & re-adjust rq_mbits for replay
request in ptlrpc_set_bulk_mbits(), they all must have already
been correctly assigned before.

Such unecessary reassign could make the first matchbit not
PTLRPC_BULK_OPS_MASK aligned, that'll trigger LASSERT in
ptlrpc_register_bulk():

- ptlrpc_set_bulk_mbits() is called when first time sending
  request, rq_mbits is set as xid, which is BULK_OPS aligned;

- ptlrpc_set_bulk_mbits() continue to adjust the mbits for
  multi-bulk RPC, rq_mbits is not aligned anymore, then rq_xid
  is changed accordingly if client is connecting to an old
  server, so rq_xid became unaligned too;

- The request is replayed, ptlrpc_set_bulk_mbits() reassign
  the rq_mbits as rq_xid, which isn't aligned already, but
  ptlrpc_register_bulk() still assumes this value as the
  first matchbits and LASSERT it's BULK_OPS aligned.

Signed-off-by: Niu Yawei <yawei.niu at intel.com>
Intel-bug-id: https://jira.hpdd.intel.com/browse/LU-6808
Reviewed-on: http://review.whamcloud.com/23048
Reviewed-by: Fan Yong <fan.yong at intel.com>
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev at intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin at intel.com>
Signed-off-by: James Simmons <jsimmons at infradead.org>
---
 drivers/staging/lustre/lustre/ptlrpc/client.c | 28 +++++++++++++++++++++------
 1 file changed, 22 insertions(+), 6 deletions(-)

diff --git a/drivers/staging/lustre/lustre/ptlrpc/client.c b/drivers/staging/lustre/lustre/ptlrpc/client.c
index 1c77792..1247686 100644
--- a/drivers/staging/lustre/lustre/ptlrpc/client.c
+++ b/drivers/staging/lustre/lustre/ptlrpc/client.c
@@ -3116,13 +3116,20 @@ void ptlrpc_set_bulk_mbits(struct ptlrpc_request *req)
 
 	LASSERT(bd);
 
-	if (!req->rq_resend) {
-		/* this request has a new xid, just use it as bulk matchbits */
-		req->rq_mbits = req->rq_xid;
-
-	} else { /* needs to generate a new matchbits for resend */
+	/*
+	 * Generate new matchbits for all resend requests, including
+	 * resend replay.
+	 */
+	if (req->rq_resend) {
 		u64 old_mbits = req->rq_mbits;
 
+		/*
+		 * First time resend on -EINPROGRESS will generate new xid,
+		 * so we can actually use the rq_xid as rq_mbits in such case,
+		 * however, it's bit hard to distinguish such resend with a
+		 * 'resend for the -EINPROGRESS resend'. To make it simple,
+		 * we opt to generate mbits for all resend cases.
+		 */
 		if ((bd->bd_import->imp_connect_data.ocd_connect_flags &
 		     OBD_CONNECT_BULK_MBITS)) {
 			req->rq_mbits = ptlrpc_next_xid();
@@ -3131,12 +3138,21 @@ void ptlrpc_set_bulk_mbits(struct ptlrpc_request *req)
 			spin_lock(&req->rq_import->imp_lock);
 			list_del_init(&req->rq_unreplied_list);
 			ptlrpc_assign_next_xid_nolock(req);
-			req->rq_mbits = req->rq_xid;
 			spin_unlock(&req->rq_import->imp_lock);
+			req->rq_mbits = req->rq_xid;
 		}
 
 		CDEBUG(D_HA, "resend bulk old x%llu new x%llu\n",
 		       old_mbits, req->rq_mbits);
+	} else if (!(lustre_msg_get_flags(req->rq_reqmsg) & MSG_REPLAY)) {
+		/* Request being sent first time, use xid as matchbits. */
+		req->rq_mbits = req->rq_xid;
+	} else {
+		/*
+		 * Replay request, xid and matchbits have already been
+		 * correctly assigned.
+		 */
+		return;
 	}
 
 	/*
-- 
1.8.3.1

From jsimmons at infradead.org  Wed Jul 26 15:22:27 2017
From: jsimmons at infradead.org (James Simmons)
Date: Wed, 26 Jul 2017 11:22:27 -0400
Subject: [lustre-devel] [PATCH 11/20] staging: lustre: lustre: fix all less
	than 0 comparison for unsigned values
In-Reply-To: <1501082556-17962-1-git-send-email-jsimmons@infradead.org>
References: <1501082556-17962-1-git-send-email-jsimmons@infradead.org>
Message-ID: <1501082556-17962-12-git-send-email-jsimmons@infradead.org>

Remove all test of less than zero for unsigned values
found with -Wtype-limits.

Signed-off-by: James Simmons <uja.ornl at yahoo.com>
Intel-bug-id: https://jira.hpdd.intel.com/browse/LU-8843
Reviewed-on: https://review.whamcloud.com/23811
Reviewed-by: Andreas Dilger <andreas.dilger at intel.com>
Reviewed-by: Jian Yu <jian.yu at intel.com>
Reviewed-by: Dmitry Eremin <dmitry.eremin at intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin at intel.com>
Signed-off-by: James Simmons <jsimmons at infradead.org>
---
 drivers/staging/lustre/lustre/include/cl_object.h | 2 +-
 drivers/staging/lustre/lustre/llite/rw.c          | 2 +-
 drivers/staging/lustre/lustre/lov/lov_object.c    | 8 ++++----
 drivers/staging/lustre/lustre/mdc/lproc_mdc.c     | 2 +-
 drivers/staging/lustre/lustre/osc/osc_cache.c     | 7 ++++---
 5 files changed, 11 insertions(+), 10 deletions(-)

diff --git a/drivers/staging/lustre/lustre/include/cl_object.h b/drivers/staging/lustre/lustre/include/cl_object.h
index 90a0c50..6887b81 100644
--- a/drivers/staging/lustre/lustre/include/cl_object.h
+++ b/drivers/staging/lustre/lustre/include/cl_object.h
@@ -1358,7 +1358,7 @@ struct cl_2queue {
 /** IO types */
 enum cl_io_type {
 	/** read system call */
-	CIT_READ,
+	CIT_READ = 1,
 	/** write system call */
 	CIT_WRITE,
 	/** truncate, utime system calls */
diff --git a/drivers/staging/lustre/lustre/llite/rw.c b/drivers/staging/lustre/lustre/llite/rw.c
index 1bac51f..166455e 100644
--- a/drivers/staging/lustre/lustre/llite/rw.c
+++ b/drivers/staging/lustre/lustre/llite/rw.c
@@ -115,7 +115,7 @@ void ll_ra_count_put(struct ll_sb_info *sbi, unsigned long len)
 
 static void ll_ra_stats_inc_sbi(struct ll_sb_info *sbi, enum ra_stat which)
 {
-	LASSERTF(which >= 0 && which < _NR_RA_STAT, "which: %u\n", which);
+	LASSERTF(which < _NR_RA_STAT, "which: %u\n", which);
 	lprocfs_counter_incr(sbi->ll_ra_stats, which);
 }
 
diff --git a/drivers/staging/lustre/lustre/lov/lov_object.c b/drivers/staging/lustre/lustre/lov/lov_object.c
index 14f3826..4913229 100644
--- a/drivers/staging/lustre/lustre/lov/lov_object.c
+++ b/drivers/staging/lustre/lustre/lov/lov_object.c
@@ -638,7 +638,7 @@ static int lov_attr_get_raid0(const struct lu_env *env, struct cl_object *obj,
 	enum lov_layout_type		    __llt;		  \
 									\
 	__llt = __obj->lo_type;					 \
-	LASSERT(0 <= __llt && __llt < ARRAY_SIZE(lov_dispatch));	\
+	LASSERT(__llt < ARRAY_SIZE(lov_dispatch));		\
 	lov_dispatch[__llt].op(__VA_ARGS__);			    \
 })
 
@@ -697,7 +697,7 @@ static inline void lov_conf_thaw(struct lov_object *lov)
 									\
 	lov_conf_freeze(__obj);						\
 	__llt = __obj->lo_type;					 \
-	LASSERT(0 <= __llt && __llt < ARRAY_SIZE(lov_dispatch));	\
+	LASSERT(__llt < ARRAY_SIZE(lov_dispatch));	\
 	lov_dispatch[__llt].op(__VA_ARGS__);			    \
 	lov_conf_thaw(__obj);						\
 } while (0)
@@ -748,13 +748,13 @@ static int lov_layout_change(const struct lu_env *unused,
 	u16 refcheck;
 	int rc;
 
-	LASSERT(0 <= lov->lo_type && lov->lo_type < ARRAY_SIZE(lov_dispatch));
+	LASSERT(lov->lo_type < ARRAY_SIZE(lov_dispatch));
 
 	env = cl_env_get(&refcheck);
 	if (IS_ERR(env))
 		return PTR_ERR(env);
 
-	LASSERT(0 <= llt && llt < ARRAY_SIZE(lov_dispatch));
+	LASSERT(llt < ARRAY_SIZE(lov_dispatch));
 
 	CDEBUG(D_INODE, DFID " from %s to %s\n",
 	       PFID(lu_object_fid(lov2lu(lov))),
diff --git a/drivers/staging/lustre/lustre/mdc/lproc_mdc.c b/drivers/staging/lustre/lustre/mdc/lproc_mdc.c
index 9021c46..e40644b 100644
--- a/drivers/staging/lustre/lustre/mdc/lproc_mdc.c
+++ b/drivers/staging/lustre/lustre/mdc/lproc_mdc.c
@@ -57,7 +57,7 @@ static ssize_t active_store(struct kobject *kobj, struct attribute *attr,
 	if (rc)
 		return rc;
 
-	if (val < 0 || val > 1)
+	if (val > 1)
 		return -ERANGE;
 
 	/* opposite senses */
diff --git a/drivers/staging/lustre/lustre/osc/osc_cache.c b/drivers/staging/lustre/lustre/osc/osc_cache.c
index 0100d27..e1207c2 100644
--- a/drivers/staging/lustre/lustre/osc/osc_cache.c
+++ b/drivers/staging/lustre/lustre/osc/osc_cache.c
@@ -783,6 +783,7 @@ static struct osc_extent *osc_extent_find(const struct lu_env *env,
 			/* pull ext's start back to cover cur */
 			ext->oe_start = cur->oe_start;
 			ext->oe_grants += chunksize;
+			LASSERT(*grants >= chunksize);
 			*grants -= chunksize;
 
 			found = osc_extent_hold(ext);
@@ -790,6 +791,7 @@ static struct osc_extent *osc_extent_find(const struct lu_env *env,
 			/* rear merge */
 			ext->oe_end = cur->oe_end;
 			ext->oe_grants += chunksize;
+			LASSERT(*grants >= chunksize);
 			*grants -= chunksize;
 
 			/* try to merge with the next one because we just fill
@@ -819,8 +821,8 @@ static struct osc_extent *osc_extent_find(const struct lu_env *env,
 		/* create a new extent */
 		EASSERT(osc_extent_is_overlapped(obj, cur) == 0, cur);
 		cur->oe_grants = chunksize + cli->cl_extent_tax;
+		LASSERT(*grants >= cur->oe_grants);
 		*grants -= cur->oe_grants;
-		LASSERT(*grants >= 0);
 
 		cur->oe_state = OES_CACHE;
 		found = osc_extent_hold(cur);
@@ -849,7 +851,6 @@ static struct osc_extent *osc_extent_find(const struct lu_env *env,
 
 out:
 	osc_extent_put(env, cur);
-	LASSERT(*grants >= 0);
 	return found;
 }
 
@@ -1219,8 +1220,8 @@ static int osc_extent_expand(struct osc_extent *ext, pgoff_t index,
 
 	ext->oe_end = end_index;
 	ext->oe_grants += chunksize;
+	LASSERT(*grants >= chunksize);
 	*grants -= chunksize;
-	LASSERT(*grants >= 0);
 	EASSERTF(osc_extent_is_overlapped(obj, ext) == 0, ext,
 		 "overlapped after expanding for %lu.\n", index);
 
-- 
1.8.3.1

From jsimmons at infradead.org  Wed Jul 26 15:22:30 2017
From: jsimmons at infradead.org (James Simmons)
Date: Wed, 26 Jul 2017 11:22:30 -0400
Subject: [lustre-devel] [PATCH 14/20] staging: lustre: ptlrpc: don't use
	CFS_DURATION_T for time64_t
In-Reply-To: <1501082556-17962-1-git-send-email-jsimmons@infradead.org>
References: <1501082556-17962-1-git-send-email-jsimmons@infradead.org>
Message-ID: <1501082556-17962-15-git-send-email-jsimmons@infradead.org>

The libcfs CFS_DURATION_T define is really only for
jiffies and its being used with time64_t in some of
the ptlrpc code. Lets remove CFS_DURATION_T and
replaced it with normal %lld instead.

Signed-off-by: James Simmons <uja.ornl at yahoo.com>
Reviewed-on: https://review.whamcloud.com/24977
Intel-bug-id: https://jira.hpdd.intel.com/browse/LU-4423
Reviewed-by: Bob Glossman <bob.glossman at intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger at intel.com>
Signed-off-by: James Simmons <jsimmons at infradead.org>
---
 drivers/staging/lustre/lustre/ptlrpc/client.c  |  5 ++---
 drivers/staging/lustre/lustre/ptlrpc/service.c | 15 +++++++--------
 2 files changed, 9 insertions(+), 11 deletions(-)

diff --git a/drivers/staging/lustre/lustre/ptlrpc/client.c b/drivers/staging/lustre/lustre/ptlrpc/client.c
index 9f4f17a..977d7a6 100644
--- a/drivers/staging/lustre/lustre/ptlrpc/client.c
+++ b/drivers/staging/lustre/lustre/ptlrpc/client.c
@@ -367,9 +367,8 @@ void ptlrpc_at_adj_net_latency(struct ptlrpc_request *req,
 		 */
 		CDEBUG((lustre_msg_get_flags(req->rq_reqmsg) & MSG_RESENT) ?
 		       D_ADAPTTO : D_WARNING,
-		       "Reported service time %u > total measured time " CFS_DURATION_T "\n",
-		       service_time,
-		       (long)(now - req->rq_sent));
+		       "Reported service time %u > total measured time %lld\n",
+		       service_time, now - req->rq_sent);
 		return;
 	}
 
diff --git a/drivers/staging/lustre/lustre/ptlrpc/service.c b/drivers/staging/lustre/lustre/ptlrpc/service.c
index 759aa6c..bcf5faf 100644
--- a/drivers/staging/lustre/lustre/ptlrpc/service.c
+++ b/drivers/staging/lustre/lustre/ptlrpc/service.c
@@ -1565,9 +1565,9 @@ static bool ptlrpc_server_normal_pending(struct ptlrpc_service_part *svcpt,
 
 	/* req_in handling should/must be fast */
 	if (ktime_get_real_seconds() - req->rq_arrival_time.tv_sec > 5)
-		DEBUG_REQ(D_WARNING, req, "Slow req_in handling " CFS_DURATION_T "s",
-			  (long)(ktime_get_real_seconds() -
-				 req->rq_arrival_time.tv_sec));
+		DEBUG_REQ(D_WARNING, req, "Slow req_in handling %llds",
+			  (s64)(ktime_get_real_seconds() -
+				req->rq_arrival_time.tv_sec));
 
 	/* Set rpc server deadline and add it to the timed list */
 	deadline = (lustre_msghdr_get_flags(req->rq_reqmsg) &
@@ -1674,12 +1674,11 @@ static bool ptlrpc_server_normal_pending(struct ptlrpc_service_part *svcpt,
 	 * The deadline is increased if we send an early reply.
 	 */
 	if (ktime_get_real_seconds() > request->rq_deadline) {
-		DEBUG_REQ(D_ERROR, request, "Dropping timed-out request from %s: deadline " CFS_DURATION_T ":" CFS_DURATION_T "s ago\n",
+		DEBUG_REQ(D_ERROR, request, "Dropping timed-out request from %s: deadline %lld:%llds ago\n",
 			  libcfs_id2str(request->rq_peer),
-			  (long)(request->rq_deadline -
-				 request->rq_arrival_time.tv_sec),
-			  (long)(ktime_get_real_seconds() -
-				 request->rq_deadline));
+			  request->rq_deadline -
+			  request->rq_arrival_time.tv_sec,
+			  ktime_get_real_seconds() - request->rq_deadline);
 		goto put_conn;
 	}
 
-- 
1.8.3.1

From jsimmons at infradead.org  Wed Jul 26 15:22:28 2017
From: jsimmons at infradead.org (James Simmons)
Date: Wed, 26 Jul 2017 11:22:28 -0400
Subject: [lustre-devel] [PATCH 12/20] staging: lustre: linkea: linkEA size
	limitation
In-Reply-To: <1501082556-17962-1-git-send-email-jsimmons@infradead.org>
References: <1501082556-17962-1-git-send-email-jsimmons@infradead.org>
Message-ID: <1501082556-17962-13-git-send-email-jsimmons@infradead.org>

From: Fan Yong <fan.yong at intel.com>

Under DNE mode, if we do not restrict the linkEA size, and if there
are too many cross-MDTs hard links to the same object, then it will
cause the llog overflow. On the other hand, too many linkEA entries
in the linkEA will serious affect the linkEA performance because we
only support to locate linkEA entry consecutively.

So we need to restrict the linkEA size. Currently, it is 4096 bytes,
that is independent from the backend. If too many hard links caused
the linkEA overflowed, we will add overflow timestamp in the linkEA
header.

Signed-off-by: Fan Yong <fan.yong at intel.com>
Intel-bug-id: https://jira.hpdd.intel.com/browse/LU-8569
Reviewed-on: https://review.whamcloud.com/23500
Reviewed-by: Andreas Dilger <andreas.dilger at intel.com>
Reviewed-by: wangdi <di.wang at intel.com>
Reviewed-by: Lai Siyao <lai.siyao at intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin at intel.com>
Signed-off-by: James Simmons <jsimmons at infradead.org>
---
 .../lustre/lustre/include/lustre/lustre_idl.h      |  5 +-
 .../staging/lustre/lustre/include/lustre_linkea.h  | 15 ++++-
 drivers/staging/lustre/lustre/llite/llite_lib.c    |  2 +-
 drivers/staging/lustre/lustre/obdclass/linkea.c    | 69 +++++++++++++++++-----
 drivers/staging/lustre/lustre/ptlrpc/wiretest.c    | 16 ++---
 5 files changed, 80 insertions(+), 27 deletions(-)

diff --git a/drivers/staging/lustre/lustre/include/lustre/lustre_idl.h b/drivers/staging/lustre/lustre/include/lustre/lustre_idl.h
index 441a2e3..1db1ab8 100644
--- a/drivers/staging/lustre/lustre/include/lustre/lustre_idl.h
+++ b/drivers/staging/lustre/lustre/include/lustre/lustre_idl.h
@@ -3217,9 +3217,8 @@ struct link_ea_header {
 	__u32 leh_magic;
 	__u32 leh_reccount;
 	__u64 leh_len;      /* total size */
-	/* future use */
-	__u32 padding1;
-	__u32 padding2;
+	__u32 leh_overflow_time;
+	__u32 leh_padding;
 };
 
 /** Hardlink data is name and parent fid.
diff --git a/drivers/staging/lustre/lustre/include/lustre_linkea.h b/drivers/staging/lustre/lustre/include/lustre_linkea.h
index 249e8bf..3ff008f 100644
--- a/drivers/staging/lustre/lustre/include/lustre_linkea.h
+++ b/drivers/staging/lustre/lustre/include/lustre_linkea.h
@@ -26,7 +26,19 @@
  * Author: di wang <di.wang at intel.com>
  */
 
-#define DEFAULT_LINKEA_SIZE	4096
+/* There are several reasons to restrict the linkEA size:
+ *
+ * 1. Under DNE mode, if we do not restrict the linkEA size, and if there
+ *    are too many cross-MDTs hard links to the same object, then it will
+ *    casue the llog overflow.
+ *
+ * 2. Some backend has limited size for EA. For example, if without large
+ *    EA enabled, the ldiskfs will make all EAs to share one (4K) EA block.
+ *
+ * 3. Too many entries in linkEA will seriously affect linkEA performance
+ *    because we only support to locate linkEA entry consecutively.
+ */
+#define MAX_LINKEA_SIZE		4096
 
 struct linkea_data {
 	/**
@@ -43,6 +55,7 @@ struct linkea_data {
 
 int linkea_data_new(struct linkea_data *ldata, struct lu_buf *buf);
 int linkea_init(struct linkea_data *ldata);
+int linkea_init_with_rec(struct linkea_data *ldata);
 void linkea_entry_unpack(const struct link_ea_entry *lee, int *reclen,
 			 struct lu_name *lname, struct lu_fid *pfid);
 int linkea_entry_pack(struct link_ea_entry *lee, const struct lu_name *lname,
diff --git a/drivers/staging/lustre/lustre/llite/llite_lib.c b/drivers/staging/lustre/lustre/llite/llite_lib.c
index 69d895d..ba7d3b9 100644
--- a/drivers/staging/lustre/lustre/llite/llite_lib.c
+++ b/drivers/staging/lustre/lustre/llite/llite_lib.c
@@ -2540,7 +2540,7 @@ static int ll_linkea_decode(struct linkea_data *ldata, unsigned int linkno,
 	unsigned int idx;
 	int rc;
 
-	rc = linkea_init(ldata);
+	rc = linkea_init_with_rec(ldata);
 	if (rc < 0)
 		return rc;
 
diff --git a/drivers/staging/lustre/lustre/obdclass/linkea.c b/drivers/staging/lustre/lustre/obdclass/linkea.c
index 0b1d2f0..cf3ad04 100644
--- a/drivers/staging/lustre/lustre/obdclass/linkea.c
+++ b/drivers/staging/lustre/lustre/obdclass/linkea.c
@@ -39,6 +39,8 @@ int linkea_data_new(struct linkea_data *ldata, struct lu_buf *buf)
 	ldata->ld_leh->leh_magic = LINK_EA_MAGIC;
 	ldata->ld_leh->leh_len = sizeof(struct link_ea_header);
 	ldata->ld_leh->leh_reccount = 0;
+	ldata->ld_leh->leh_overflow_time = 0;
+	ldata->ld_leh->leh_padding = 0;
 	return 0;
 }
 EXPORT_SYMBOL(linkea_data_new);
@@ -53,11 +55,15 @@ int linkea_init(struct linkea_data *ldata)
 		leh->leh_magic = LINK_EA_MAGIC;
 		leh->leh_reccount = __swab32(leh->leh_reccount);
 		leh->leh_len = __swab64(leh->leh_len);
-		/* entries are swabbed by linkea_entry_unpack */
+		leh->leh_overflow_time = __swab32(leh->leh_overflow_time);
+		leh->leh_padding = __swab32(leh->leh_padding);
+		/* individual entries are swabbed by linkea_entry_unpack() */
 	}
+
 	if (leh->leh_magic != LINK_EA_MAGIC)
 		return -EINVAL;
-	if (leh->leh_reccount == 0)
+
+	if (leh->leh_reccount == 0 && leh->leh_overflow_time == 0)
 		return -ENODATA;
 
 	ldata->ld_leh = leh;
@@ -65,6 +71,18 @@ int linkea_init(struct linkea_data *ldata)
 }
 EXPORT_SYMBOL(linkea_init);
 
+int linkea_init_with_rec(struct linkea_data *ldata)
+{
+	int rc;
+
+	rc = linkea_init(ldata);
+	if (!rc && ldata->ld_leh->leh_reccount == 0)
+		rc = -ENODATA;
+
+	return rc;
+}
+EXPORT_SYMBOL(linkea_init_with_rec);
+
 /**
  * Pack a link_ea_entry.
  * All elements are stored as chars to avoid alignment issues.
@@ -94,6 +112,8 @@ int linkea_entry_pack(struct link_ea_entry *lee, const struct lu_name *lname,
 void linkea_entry_unpack(const struct link_ea_entry *lee, int *reclen,
 			 struct lu_name *lname, struct lu_fid *pfid)
 {
+	LASSERT(lee);
+
 	*reclen = (lee->lee_reclen[0] << 8) | lee->lee_reclen[1];
 	memcpy(pfid, &lee->lee_parent_fid, sizeof(*pfid));
 	fid_be_to_cpu(pfid, pfid);
@@ -110,25 +130,44 @@ void linkea_entry_unpack(const struct link_ea_entry *lee, int *reclen,
 int linkea_add_buf(struct linkea_data *ldata, const struct lu_name *lname,
 		   const struct lu_fid *pfid)
 {
-	LASSERT(ldata->ld_leh);
+	struct link_ea_header *leh = ldata->ld_leh;
+	int reclen;
+
+	LASSERT(leh);
 
 	if (!lname || !pfid)
 		return -EINVAL;
 
-	ldata->ld_reclen = lname->ln_namelen + sizeof(struct link_ea_entry);
-	if (ldata->ld_leh->leh_len + ldata->ld_reclen >
-	    ldata->ld_buf->lb_len) {
+	reclen = lname->ln_namelen + sizeof(struct link_ea_entry);
+	if (unlikely(leh->leh_len + reclen > MAX_LINKEA_SIZE)) {
+		/*
+		 * Use 32-bits to save the overflow time, although it will
+		 * shrink the ktime_get_real_seconds() returned 64-bits value
+		 * to 32-bits value, it is still quite large and can be used
+		 * for about 140 years. That is enough.
+		 */
+		leh->leh_overflow_time = ktime_get_real_seconds();
+		if (unlikely(leh->leh_overflow_time == 0))
+			leh->leh_overflow_time++;
+
+		CDEBUG(D_INODE, "No enough space to hold linkea entry '" DFID ": %.*s' at %u\n",
+		       PFID(pfid), lname->ln_namelen,
+		       lname->ln_name, leh->leh_overflow_time);
+		return 0;
+	}
+
+	if (leh->leh_len + reclen > ldata->ld_buf->lb_len) {
 		if (lu_buf_check_and_grow(ldata->ld_buf,
-					  ldata->ld_leh->leh_len +
-					  ldata->ld_reclen) < 0)
+					  leh->leh_len + reclen) < 0)
 			return -ENOMEM;
+
+		leh = ldata->ld_leh = ldata->ld_buf->lb_buf;
 	}
 
-	ldata->ld_leh = ldata->ld_buf->lb_buf;
-	ldata->ld_lee = ldata->ld_buf->lb_buf + ldata->ld_leh->leh_len;
+	ldata->ld_lee = ldata->ld_buf->lb_buf + leh->leh_len;
 	ldata->ld_reclen = linkea_entry_pack(ldata->ld_lee, lname, pfid);
-	ldata->ld_leh->leh_len += ldata->ld_reclen;
-	ldata->ld_leh->leh_reccount++;
+	leh->leh_len += ldata->ld_reclen;
+	leh->leh_reccount++;
 	CDEBUG(D_INODE, "New link_ea name '" DFID ":%.*s' is added\n",
 	       PFID(pfid), lname->ln_namelen, lname->ln_name);
 	return 0;
@@ -139,6 +178,7 @@ int linkea_add_buf(struct linkea_data *ldata, const struct lu_name *lname,
 void linkea_del_buf(struct linkea_data *ldata, const struct lu_name *lname)
 {
 	LASSERT(ldata->ld_leh && ldata->ld_lee);
+	LASSERT(ldata->ld_leh->leh_reccount > 0);
 
 	ldata->ld_leh->leh_reccount--;
 	ldata->ld_leh->leh_len -= ldata->ld_reclen;
@@ -174,8 +214,9 @@ int linkea_links_find(struct linkea_data *ldata, const struct lu_name *lname,
 
 	LASSERT(ldata->ld_leh);
 
-	/* link #0 */
-	ldata->ld_lee = (struct link_ea_entry *)(ldata->ld_leh + 1);
+	/* link #0, if leh_reccount == 0 we skip the loop and return -ENOENT */
+	if (likely(ldata->ld_leh->leh_reccount > 0))
+		ldata->ld_lee = (struct link_ea_entry *)(ldata->ld_leh + 1);
 
 	for (count = 0; count < ldata->ld_leh->leh_reccount; count++) {
 		linkea_entry_unpack(ldata->ld_lee, &ldata->ld_reclen,
diff --git a/drivers/staging/lustre/lustre/ptlrpc/wiretest.c b/drivers/staging/lustre/lustre/ptlrpc/wiretest.c
index 367f7e2..311c526 100644
--- a/drivers/staging/lustre/lustre/ptlrpc/wiretest.c
+++ b/drivers/staging/lustre/lustre/ptlrpc/wiretest.c
@@ -3820,14 +3820,14 @@ void lustre_assert_wire_constants(void)
 		 (long long)(int)offsetof(struct link_ea_header, leh_len));
 	LASSERTF((int)sizeof(((struct link_ea_header *)0)->leh_len) == 8, "found %lld\n",
 		 (long long)(int)sizeof(((struct link_ea_header *)0)->leh_len));
-	LASSERTF((int)offsetof(struct link_ea_header, padding1) == 16, "found %lld\n",
-		 (long long)(int)offsetof(struct link_ea_header, padding1));
-	LASSERTF((int)sizeof(((struct link_ea_header *)0)->padding1) == 4, "found %lld\n",
-		 (long long)(int)sizeof(((struct link_ea_header *)0)->padding1));
-	LASSERTF((int)offsetof(struct link_ea_header, padding2) == 20, "found %lld\n",
-		 (long long)(int)offsetof(struct link_ea_header, padding2));
-	LASSERTF((int)sizeof(((struct link_ea_header *)0)->padding2) == 4, "found %lld\n",
-		 (long long)(int)sizeof(((struct link_ea_header *)0)->padding2));
+	LASSERTF((int)offsetof(struct link_ea_header, leh_overflow_time) == 16, "found %lld\n",
+		 (long long)(int)offsetof(struct link_ea_header, leh_overflow_time));
+	LASSERTF((int)sizeof(((struct link_ea_header *)0)->leh_overflow_time) == 4, "found %lld\n",
+		 (long long)(int)sizeof(((struct link_ea_header *)0)->leh_overflow_time));
+	LASSERTF((int)offsetof(struct link_ea_header, leh_padding) == 20, "found %lld\n",
+		 (long long)(int)offsetof(struct link_ea_header, leh_padding));
+	LASSERTF((int)sizeof(((struct link_ea_header *)0)->leh_padding) == 4, "found %lld\n",
+		 (long long)(int)sizeof(((struct link_ea_header *)0)->leh_padding));
 	BUILD_BUG_ON(LINK_EA_MAGIC != 0x11EAF1DFUL);
 
 	/* Checks for struct link_ea_entry */
-- 
1.8.3.1

From jsimmons at infradead.org  Wed Jul 26 15:22:31 2017
From: jsimmons at infradead.org (James Simmons)
Date: Wed, 26 Jul 2017 11:22:31 -0400
Subject: [lustre-devel] [PATCH 15/20] staging: lustre: ptlrpc: print times
	in microseconds
In-Reply-To: <1501082556-17962-1-git-send-email-jsimmons@infradead.org>
References: <1501082556-17962-1-git-send-email-jsimmons@infradead.org>
Message-ID: <1501082556-17962-16-git-send-email-jsimmons@infradead.org>

From: Bob Glosman <bob.glossman at intel.com>

report times and time diffs in seconds.microseconds instead of seconds

Signed-off-by: Bob Glosman <bob.glossman at intel.com>
Intel-bug-id: https://jira.hpdd.intel.com/browse/LU-7733
Reviewed-on: http://review.whamcloud.com/18335
Reviewed-by: Andreas Dilger <andreas.dilger at intel.com>
Reviewed-by: Minh Diep <minh.diep at intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin at intel.com>
Signed-off-by: James Simmons <jsimmons at infradead.org>
---
 drivers/staging/lustre/lustre/ptlrpc/lproc_ptlrpc.c | 15 ++++++++++++---
 1 file changed, 12 insertions(+), 3 deletions(-)

diff --git a/drivers/staging/lustre/lustre/ptlrpc/lproc_ptlrpc.c b/drivers/staging/lustre/lustre/ptlrpc/lproc_ptlrpc.c
index f874781..c429613 100644
--- a/drivers/staging/lustre/lustre/ptlrpc/lproc_ptlrpc.c
+++ b/drivers/staging/lustre/lustre/ptlrpc/lproc_ptlrpc.c
@@ -905,11 +905,18 @@ static int ptlrpc_lprocfs_svc_req_history_show(struct seq_file *s, void *iter)
 	rc = ptlrpc_lprocfs_svc_req_history_seek(svcpt, srhi, srhi->srhi_seq);
 
 	if (rc == 0) {
+		struct timespec64 arrival, sent, arrivaldiff;
 		char nidstr[LNET_NIDSTR_SIZE];
 
 		req = srhi->srhi_req;
 
 		libcfs_nid2str_r(req->rq_self, nidstr, sizeof(nidstr));
+		arrival.tv_sec = req->rq_arrival_time.tv_sec;
+		arrival.tv_nsec = req->rq_arrival_time.tv_nsec;
+		sent.tv_sec = req->rq_sent;
+		sent.tv_nsec = 0;
+		arrivaldiff = timespec64_sub(sent, arrival);
+
 		/* Print common req fields.
 		 * CAVEAT EMPTOR: we're racing with the service handler
 		 * here.  The request could contain any old crap, so you
@@ -917,13 +924,15 @@ static int ptlrpc_lprocfs_svc_req_history_show(struct seq_file *s, void *iter)
 		 * parser. Currently I only print stuff here I know is OK
 		 * to look at coz it was set up in request_in_callback()!!!
 		 */
-		seq_printf(s, "%lld:%s:%s:x%llu:%d:%s:%lld:%lds(%+lds) ",
+		seq_printf(s, "%lld:%s:%s:x%llu:%d:%s:%lld.%06lld:%lld.%06llds(%+lld.0s) ",
 			   req->rq_history_seq, nidstr,
 			   libcfs_id2str(req->rq_peer), req->rq_xid,
 			   req->rq_reqlen, ptlrpc_rqphase2str(req),
 			   (s64)req->rq_arrival_time.tv_sec,
-			   (long)(req->rq_sent - req->rq_arrival_time.tv_sec),
-			   (long)(req->rq_sent - req->rq_deadline));
+			   (s64)req->rq_arrival_time.tv_nsec / NSEC_PER_USEC,
+			   (s64)arrivaldiff.tv_sec,
+			   (s64)(arrivaldiff.tv_nsec / NSEC_PER_USEC),
+			   (s64)(req->rq_sent - req->rq_deadline));
 		if (!svc->srv_ops.so_req_printer)
 			seq_putc(s, '\n');
 		else
-- 
1.8.3.1

From jsimmons at infradead.org  Wed Jul 26 15:22:35 2017
From: jsimmons at infradead.org (James Simmons)
Date: Wed, 26 Jul 2017 11:22:35 -0400
Subject: [lustre-devel] [PATCH 19/20] staging: lustre: llite: add xattr.h
	header to xattr.c
In-Reply-To: <1501082556-17962-1-git-send-email-jsimmons@infradead.org>
References: <1501082556-17962-1-git-send-email-jsimmons@infradead.org>
Message-ID: <1501082556-17962-20-git-send-email-jsimmons@infradead.org>

From: Dmitry Eremin <dmitry.eremin at intel.com>

The reason xattr.c can compile without xattr.h is due
to lustre_compact.h being included. That header will
eventually go away so lets directly include xattr.h.

Signed-off-by: Dmitry Eremin <dmitry.eremin at intel.com>
Intel-bug-id: https://jira.hpdd.intel.com/browse/LU-7244
Reviewed-on: http://review.whamcloud.com/16707
Reviewed-by: John L. Hammond <john.hammond at intel.com>
Reviewed-by: James Simmons <uja.ornl at yahoo.com>
Reviewed-by: Oleg Drokin <oleg.drokin at intel.com>
Signed-off-by: James Simmons <jsimmons at infradead.org>
---
 drivers/staging/lustre/lustre/llite/xattr.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/drivers/staging/lustre/lustre/llite/xattr.c b/drivers/staging/lustre/lustre/llite/xattr.c
index bd30abd..d0cad7e 100644
--- a/drivers/staging/lustre/lustre/llite/xattr.c
+++ b/drivers/staging/lustre/lustre/llite/xattr.c
@@ -33,6 +33,7 @@
 #include <linux/fs.h>
 #include <linux/sched.h>
 #include <linux/mm.h>
+#include <linux/xattr.h>
 #include <linux/selinux.h>
 
 #define DEBUG_SUBSYSTEM S_LLITE
-- 
1.8.3.1

From jsimmons at infradead.org  Wed Jul 26 15:22:17 2017
From: jsimmons at infradead.org (James Simmons)
Date: Wed, 26 Jul 2017 11:22:17 -0400
Subject: [lustre-devel] [PATCH 01/20] staging: lustre: osc: soft lock -
	osc_makes_rpc()
In-Reply-To: <1501082556-17962-1-git-send-email-jsimmons@infradead.org>
References: <1501082556-17962-1-git-send-email-jsimmons@infradead.org>
Message-ID: <1501082556-17962-2-git-send-email-jsimmons@infradead.org>

From: Bobi Jam <bobijam.xu at intel.com>

It is possible that an osc_extent contains more than 256 chunks, and
the IO engine won't add this extent in one RPC
(try_to_add_extent_for_io) so that osc_check_rpcs() run into a loop
upon this extent and never break.

This patch changes osc_max_write_chunks() to make sure the value
can cover all possible osc_extent, so that all osc_extent will be
added into one RPC.

This patch also add another field erd_max_extents in extent_rpc_data
to make sure not to add too many fragments in a single RPC.

Signed-off-by: Bobi Jam <bobijam.xu at intel.com>
Intel-bug-id: https://jira.hpdd.intel.com/browse/LU-8680
Reviewed-on: http://review.whamcloud.com/23326
Reviewed-by: Jinshan Xiong <jinshan.xiong at intel.com>
Reviewed-by: Niu Yawei <yawei.niu at intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin at intel.com>
Signed-off-by: James Simmons <jsimmons at infradead.org>
---
 drivers/staging/lustre/lustre/osc/osc_cache.c | 24 +++++++++++++++++++++---
 1 file changed, 21 insertions(+), 3 deletions(-)

diff --git a/drivers/staging/lustre/lustre/osc/osc_cache.c b/drivers/staging/lustre/lustre/osc/osc_cache.c
index d8a95f8..0100d27 100644
--- a/drivers/staging/lustre/lustre/osc/osc_cache.c
+++ b/drivers/staging/lustre/lustre/osc/osc_cache.c
@@ -1887,6 +1887,7 @@ struct extent_rpc_data {
 	unsigned int		erd_page_count;
 	unsigned int		erd_max_pages;
 	unsigned int		erd_max_chunks;
+	unsigned int		erd_max_extents;
 };
 
 static inline unsigned int osc_extent_chunks(const struct osc_extent *ext)
@@ -1915,11 +1916,23 @@ static int try_to_add_extent_for_io(struct client_obd *cli,
 	EASSERT((ext->oe_state == OES_CACHE || ext->oe_state == OES_LOCK_DONE),
 		ext);
 
+	if (!data->erd_max_extents)
+		return 0;
+
 	chunk_count = osc_extent_chunks(ext);
+	EASSERTF(data->erd_page_count != 0 ||
+		 chunk_count <= data->erd_max_chunks, ext,
+		 "The first extent to be fit in a RPC contains %u chunks, which is over the limit %u.\n",
+		 chunk_count, data->erd_max_chunks);
+
 	if (chunk_count > data->erd_max_chunks)
 		return 0;
 
 	data->erd_max_pages = max(ext->oe_mppr, data->erd_max_pages);
+	EASSERTF(data->erd_page_count != 0 ||
+		 ext->oe_nr_pages <= data->erd_max_pages, ext,
+		 "The first extent to be fit in a RPC contains %u pages, which is over the limit %u.\n",
+		 ext->oe_nr_pages, data->erd_max_pages);
 	if (data->erd_page_count + ext->oe_nr_pages > data->erd_max_pages)
 		return 0;
 
@@ -1943,6 +1956,7 @@ static int try_to_add_extent_for_io(struct client_obd *cli,
 		break;
 	}
 
+	data->erd_max_extents--;
 	data->erd_max_chunks -= chunk_count;
 	data->erd_page_count += ext->oe_nr_pages;
 	list_move_tail(&ext->oe_link, data->erd_rpc_list);
@@ -1972,10 +1986,12 @@ static inline unsigned int osc_max_write_chunks(const struct client_obd *cli)
 	 *
 	 * This limitation doesn't apply to ldiskfs, which allows as many
 	 * chunks in one RPC as we want. However, it won't have any benefits
-	 * to have too many discontiguous pages in one RPC. Therefore, it
-	 * can only have 256 chunks at most in one RPC.
+	 * to have too many discontiguous pages in one RPC.
+	 *
+	 * An osc_extent won't cover over a RPC size, so the chunks in an
+	 * osc_extent won't bigger than PTLRPC_MAX_BRW_SIZE >> chunkbits.
 	 */
-	return min(PTLRPC_MAX_BRW_SIZE >> cli->cl_chunkbits, 256);
+	return PTLRPC_MAX_BRW_SIZE >> cli->cl_chunkbits;
 }
 
 /**
@@ -2002,6 +2018,7 @@ static unsigned int get_write_extents(struct osc_object *obj,
 		.erd_page_count = 0,
 		.erd_max_pages = cli->cl_max_pages_per_rpc,
 		.erd_max_chunks = osc_max_write_chunks(cli),
+		.erd_max_extents = 256,
 	};
 
 	LASSERT(osc_object_is_locked(obj));
@@ -2140,6 +2157,7 @@ static unsigned int get_write_extents(struct osc_object *obj,
 		.erd_page_count = 0,
 		.erd_max_pages = cli->cl_max_pages_per_rpc,
 		.erd_max_chunks = UINT_MAX,
+		.erd_max_extents = UINT_MAX,
 	};
 	int rc = 0;
 
-- 
1.8.3.1

From jsimmons at infradead.org  Wed Jul 26 15:22:19 2017
From: jsimmons at infradead.org (James Simmons)
Date: Wed, 26 Jul 2017 11:22:19 -0400
Subject: [lustre-devel] [PATCH 03/20] staging: lustre: ldlm: crash on umount
	in cleanup_resource
In-Reply-To: <1501082556-17962-1-git-send-email-jsimmons@infradead.org>
References: <1501082556-17962-1-git-send-email-jsimmons@infradead.org>
Message-ID: <1501082556-17962-4-git-send-email-jsimmons@infradead.org>

From: Andriy Skulysh <andriy.skulysh at seagate.com>

cfs_hash_for_each_relax() assumes that cfs_hash_put_locked()
doesn't release bd lock, but it isn't true for
ldlm_res_hop_put_locked().

Add recfcount on next hnode in cfs_hash_for_each_relax() and
remove ldlm_res_hop_put_locked()

Signed-off-by: Andriy Skulysh <andriy.skulysh at seagate.com>
Intel-bug-id: https://jira.hpdd.intel.com/browse/LU-6304
Xyratex-bug-id: MRP-2352
Reviewed-by: Vitaly Fertman <vitaly.fertman at seagate.com>
Reviewed-by: Alexander Boyko <alexander.boyko at seagate.com>
Tested-by: Alexander Lezhoev <alexander.lezhoev at seagate.com>
Reviewed-on: http://review.whamcloud.com/13908
Reviewed-by: Oleg Drokin <oleg.drokin at intel.com>
Signed-off-by: James Simmons <jsimmons at infradead.org>
---
 drivers/staging/lustre/lnet/libcfs/hash.c          | 47 ++++++++++++++--------
 drivers/staging/lustre/lustre/ldlm/ldlm_internal.h |  3 --
 drivers/staging/lustre/lustre/ldlm/ldlm_resource.c | 43 --------------------
 3 files changed, 31 insertions(+), 62 deletions(-)

diff --git a/drivers/staging/lustre/lnet/libcfs/hash.c b/drivers/staging/lustre/lnet/libcfs/hash.c
index 5c2ce2e..ff54eaf 100644
--- a/drivers/staging/lustre/lnet/libcfs/hash.c
+++ b/drivers/staging/lustre/lnet/libcfs/hash.c
@@ -1008,7 +1008,7 @@ struct cfs_hash *
 	LASSERT(ops->hs_object);
 	LASSERT(ops->hs_keycmp);
 	LASSERT(ops->hs_get);
-	LASSERT(ops->hs_put_locked);
+	LASSERT(ops->hs_put || ops->hs_put_locked);
 
 	if (flags & CFS_HASH_REHASH)
 		flags |= CFS_HASH_COUNTER; /* must have counter */
@@ -1553,19 +1553,20 @@ struct cfs_hash_cond_arg {
 cfs_hash_for_each_relax(struct cfs_hash *hs, cfs_hash_for_each_cb_t func,
 			void *data, int start)
 {
+	struct hlist_node *next = NULL;
 	struct hlist_node *hnode;
-	struct hlist_node *tmp;
 	struct cfs_hash_bd bd;
 	u32 version;
 	int count = 0;
 	int stop_on_change;
+	int has_put_locked;
 	int end = -1;
 	int rc = 0;
 	int i;
 
 	stop_on_change = cfs_hash_with_rehash_key(hs) ||
-			 !cfs_hash_with_no_itemref(hs) ||
-			 !hs->hs_ops->hs_put_locked;
+			 !cfs_hash_with_no_itemref(hs);
+	has_put_locked = hs->hs_ops->hs_put_locked != NULL;
 	cfs_hash_lock(hs, 0);
 again:
 	LASSERT(!cfs_hash_is_rehashing(hs));
@@ -1582,38 +1583,52 @@ struct cfs_hash_cond_arg {
 		version = cfs_hash_bd_version_get(&bd);
 
 		cfs_hash_bd_for_each_hlist(hs, &bd, hhead) {
-			for (hnode = hhead->first; hnode;) {
+			hnode = hhead->first;
+			if (!hnode)
+				continue;
+			cfs_hash_get(hs, hnode);
+
+			for (; hnode; hnode = next) {
 				cfs_hash_bucket_validate(hs, &bd, hnode);
-				cfs_hash_get(hs, hnode);
+				next = hnode->next;
+				if (next)
+					cfs_hash_get(hs, next);
 				cfs_hash_bd_unlock(hs, &bd, 0);
 				cfs_hash_unlock(hs, 0);
 
 				rc = func(hs, &bd, hnode, data);
-				if (stop_on_change)
+				if (stop_on_change || !has_put_locked)
 					cfs_hash_put(hs, hnode);
 				cond_resched();
 				count++;
 
 				cfs_hash_lock(hs, 0);
 				cfs_hash_bd_lock(hs, &bd, 0);
-				if (!stop_on_change) {
-					tmp = hnode->next;
-					cfs_hash_put_locked(hs, hnode);
-					hnode = tmp;
-				} else { /* bucket changed? */
+				if (stop_on_change) {
 					if (version !=
 					    cfs_hash_bd_version_get(&bd))
-						break;
-					/* safe to continue because no change */
-					hnode = hnode->next;
+						rc = -EINTR;
+				} else if (has_put_locked) {
+					cfs_hash_put_locked(hs, hnode);
 				}
 				if (rc) /* callback wants to break iteration */
 					break;
 			}
-			if (rc) /* callback wants to break iteration */
+			if (next) {
+				if (has_put_locked) {
+					cfs_hash_put_locked(hs, next);
+					next = NULL;
+				}
 				break;
+			} else if (rc) {
+				break;
+			}
 		}
 		cfs_hash_bd_unlock(hs, &bd, 0);
+		if (next && !has_put_locked) {
+			cfs_hash_put(hs, next);
+			next = NULL;
+		}
 		if (rc) /* callback wants to break iteration */
 			break;
 	}
diff --git a/drivers/staging/lustre/lustre/ldlm/ldlm_internal.h b/drivers/staging/lustre/lustre/ldlm/ldlm_internal.h
index ec3b23c..2bf5c84 100644
--- a/drivers/staging/lustre/lustre/ldlm/ldlm_internal.h
+++ b/drivers/staging/lustre/lustre/ldlm/ldlm_internal.h
@@ -106,9 +106,6 @@ int ldlm_cancel_lru_local(struct ldlm_namespace *ns,
 extern unsigned int ldlm_enqueue_min;
 extern unsigned int ldlm_cancel_unused_locks_before_replay;
 
-/* ldlm_resource.c */
-int ldlm_resource_putref_locked(struct ldlm_resource *res);
-
 /* ldlm_lock.c */
 
 struct ldlm_cb_set_arg {
diff --git a/drivers/staging/lustre/lustre/ldlm/ldlm_resource.c b/drivers/staging/lustre/lustre/ldlm/ldlm_resource.c
index eab44dc..4e805e7 100644
--- a/drivers/staging/lustre/lustre/ldlm/ldlm_resource.c
+++ b/drivers/staging/lustre/lustre/ldlm/ldlm_resource.c
@@ -536,16 +536,6 @@ static void ldlm_res_hop_get_locked(struct cfs_hash *hs,
 	ldlm_resource_getref(res);
 }
 
-static void ldlm_res_hop_put_locked(struct cfs_hash *hs,
-				    struct hlist_node *hnode)
-{
-	struct ldlm_resource *res;
-
-	res = hlist_entry(hnode, struct ldlm_resource, lr_hash);
-	/* cfs_hash_for_each_nolock is the only chance we call it */
-	ldlm_resource_putref_locked(res);
-}
-
 static void ldlm_res_hop_put(struct cfs_hash *hs, struct hlist_node *hnode)
 {
 	struct ldlm_resource *res;
@@ -561,7 +551,6 @@ static void ldlm_res_hop_put(struct cfs_hash *hs, struct hlist_node *hnode)
 	.hs_keycpy      = NULL,
 	.hs_object      = ldlm_res_hop_object,
 	.hs_get		= ldlm_res_hop_get_locked,
-	.hs_put_locked  = ldlm_res_hop_put_locked,
 	.hs_put		= ldlm_res_hop_put
 };
 
@@ -572,7 +561,6 @@ static void ldlm_res_hop_put(struct cfs_hash *hs, struct hlist_node *hnode)
 	.hs_keycpy      = NULL,
 	.hs_object      = ldlm_res_hop_object,
 	.hs_get		= ldlm_res_hop_get_locked,
-	.hs_put_locked  = ldlm_res_hop_put_locked,
 	.hs_put		= ldlm_res_hop_put
 };
 
@@ -1249,37 +1237,6 @@ int ldlm_resource_putref(struct ldlm_resource *res)
 }
 EXPORT_SYMBOL(ldlm_resource_putref);
 
-/* Returns 1 if the resource was freed, 0 if it remains. */
-int ldlm_resource_putref_locked(struct ldlm_resource *res)
-{
-	struct ldlm_namespace *ns = ldlm_res_to_ns(res);
-
-	LASSERT_ATOMIC_GT_LT(&res->lr_refcount, 0, LI_POISON);
-	CDEBUG(D_INFO, "putref res: %p count: %d\n",
-	       res, atomic_read(&res->lr_refcount) - 1);
-
-	if (atomic_dec_and_test(&res->lr_refcount)) {
-		struct cfs_hash_bd bd;
-
-		cfs_hash_bd_get(ldlm_res_to_ns(res)->ns_rs_hash,
-				&res->lr_name, &bd);
-		__ldlm_resource_putref_final(&bd, res);
-		cfs_hash_bd_unlock(ns->ns_rs_hash, &bd, 1);
-		/* NB: ns_rs_hash is created with CFS_HASH_NO_ITEMREF,
-		 * so we should never be here while calling cfs_hash_del,
-		 * cfs_hash_for_each_nolock is the only case we can get
-		 * here, which is safe to release cfs_hash_bd_lock.
-		 */
-		if (ns->ns_lvbo && ns->ns_lvbo->lvbo_free)
-			ns->ns_lvbo->lvbo_free(res);
-		kmem_cache_free(ldlm_resource_slab, res);
-
-		cfs_hash_bd_lock(ns->ns_rs_hash, &bd, 1);
-		return 1;
-	}
-	return 0;
-}
-
 /**
  * Add a lock into a given resource into specified lock list.
  */
-- 
1.8.3.1

From jsimmons at infradead.org  Wed Jul 26 15:22:18 2017
From: jsimmons at infradead.org (James Simmons)
Date: Wed, 26 Jul 2017 11:22:18 -0400
Subject: [lustre-devel] [PATCH 02/20] staging: lustre: ldlm: restore missing
	newlines in ldlm sysfs files
In-Reply-To: <1501082556-17962-1-git-send-email-jsimmons@infradead.org>
References: <1501082556-17962-1-git-send-email-jsimmons@infradead.org>
Message-ID: <1501082556-17962-3-git-send-email-jsimmons@infradead.org>

From: "John L. Hammond" <john.hammond at intel.com>

Restore the missing trailing newlines in
/sys/fs/lustre/ldlm/namespaces/*/lru_{max_age,size}.

Signed-off-by: John L. Hammond <john.hammond at intel.com>
Intel-bug-id: https://jira.hpdd.intel.com/browse/LU-9109
Reviewed-on: https://review.whamcloud.com/25522
Reviewed-by: James Simmons <uja.ornl at yahoo.com>
Reviewed-by: Mike Pershin <mike.pershin at intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin at intel.com>
Signed-off-by: James Simmons <jsimmons at infradead.org>
---
 drivers/staging/lustre/lustre/ldlm/ldlm_resource.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/drivers/staging/lustre/lustre/ldlm/ldlm_resource.c b/drivers/staging/lustre/lustre/ldlm/ldlm_resource.c
index c9ef247..eab44dc 100644
--- a/drivers/staging/lustre/lustre/ldlm/ldlm_resource.c
+++ b/drivers/staging/lustre/lustre/ldlm/ldlm_resource.c
@@ -223,7 +223,7 @@ static ssize_t lru_size_show(struct kobject *kobj, struct attribute *attr,
 
 	if (ns_connect_lru_resize(ns))
 		nr = &ns->ns_nr_unused;
-	return sprintf(buf, "%u", *nr);
+	return sprintf(buf, "%u\n", *nr);
 }
 
 static ssize_t lru_size_store(struct kobject *kobj, struct attribute *attr,
@@ -318,7 +318,7 @@ static ssize_t lru_max_age_show(struct kobject *kobj, struct attribute *attr,
 	struct ldlm_namespace *ns = container_of(kobj, struct ldlm_namespace,
 						 ns_kobj);
 
-	return sprintf(buf, "%u", ns->ns_max_age);
+	return sprintf(buf, "%u\n", ns->ns_max_age);
 }
 
 static ssize_t lru_max_age_store(struct kobject *kobj, struct attribute *attr,
-- 
1.8.3.1

From jsimmons at infradead.org  Wed Jul 26 15:22:20 2017
From: jsimmons at infradead.org (James Simmons)
Date: Wed, 26 Jul 2017 11:22:20 -0400
Subject: [lustre-devel] [PATCH 04/20] staging: lustre: lov: fix 'control
	flow' error in lov_io_init_released
In-Reply-To: <1501082556-17962-1-git-send-email-jsimmons@infradead.org>
References: <1501082556-17962-1-git-send-email-jsimmons@infradead.org>
Message-ID: <1501082556-17962-5-git-send-email-jsimmons@infradead.org>

From: Sebastien Buisson <sbuisson at ddn.com>

Fix "control flow" issues found by Coverity version 6.6.1:
Logically dead code (DEADCODE)
Execution cannot reach this statement.

Signed-off-by: Sebastien Buisson <sbuisson at ddn.com>
Intel-bug-id: https://jira.hpdd.intel.com/browse/LU-4048
Reviewed-on: http://review.whamcloud.com/7824
Reviewed-by: Jinshan Xiong <jinshan.xiong at intel.com>
Reviewed-by: Emoly Liu <emoly.liu at intel.com>
Reviewed-by: James Simmons <uja.ornl at yahoo.com>
Reviewed-by: John L. Hammond <john.hammond at intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin at intel.com>
Signed-off-by: James Simmons <jsimmons at infradead.org>
---
 drivers/staging/lustre/lustre/lov/lov_io.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/drivers/staging/lustre/lustre/lov/lov_io.c b/drivers/staging/lustre/lustre/lov/lov_io.c
index babf39a..3e37353 100644
--- a/drivers/staging/lustre/lustre/lov/lov_io.c
+++ b/drivers/staging/lustre/lustre/lov/lov_io.c
@@ -1047,6 +1047,8 @@ int lov_io_init_released(const struct lu_env *env, struct cl_object *obj,
 	switch (io->ci_type) {
 	default:
 		LASSERTF(0, "invalid type %d\n", io->ci_type);
+		result = -EOPNOTSUPP;
+		break;
 	case CIT_MISC:
 	case CIT_FSYNC:
 	case CIT_DATA_VERSION:
-- 
1.8.3.1

From jsimmons at infradead.org  Wed Jul 26 15:22:36 2017
From: jsimmons at infradead.org (James Simmons)
Date: Wed, 26 Jul 2017 11:22:36 -0400
Subject: [lustre-devel] [PATCH 20/20] staging: lustre: llite: set security
	xattr using __vfs_setxattr
In-Reply-To: <1501082556-17962-1-git-send-email-jsimmons@infradead.org>
References: <1501082556-17962-1-git-send-email-jsimmons@infradead.org>
Message-ID: <1501082556-17962-21-git-send-email-jsimmons@infradead.org>

Currently ll_initxattrs() initializes the security xattrs in
a very non-standard using get_xattr_types() to get the struct
handler that lustre created to then call indirectly the function
to set the xattr. The available __vfs_setxattr() function does
the same thing and also handles the case of when size is zero
the xattr should be set to empty EA.

Signed-off-by: James Simmons <uja.ornl at yahoo.com>
Intel-bug-id: https://jira.hpdd.intel.com/browse/LU-9183
Reviewed-on: https://review.whamcloud.com/27240
Reviewed-by: Dmitry Eremin <dmitry.eremin at intel.com>
Reviewed-by: Bob Glossman <bob.glossman at intel.com>
Reviewed-by: Sebastien Buisson <sbuisson at ddn.com>
Reviewed-by: Oleg Drokin <oleg.drokin at intel.com>
Signed-off-by: James Simmons <jsimmons at infradead.org>
---
 .../staging/lustre/lustre/llite/xattr_security.c   | 23 ++++++++++++++--------
 1 file changed, 15 insertions(+), 8 deletions(-)

diff --git a/drivers/staging/lustre/lustre/llite/xattr_security.c b/drivers/staging/lustre/lustre/llite/xattr_security.c
index d61d801..391fb25 100644
--- a/drivers/staging/lustre/lustre/llite/xattr_security.c
+++ b/drivers/staging/lustre/lustre/llite/xattr_security.c
@@ -28,7 +28,10 @@
  * lustre/llite/xattr_security.c
  * Handler for storing security labels as extended attributes.
  */
+
+#include <linux/types.h>
 #include <linux/security.h>
+#include <linux/selinux.h>
 #include <linux/xattr.h>
 #include "llite_internal.h"
 
@@ -48,19 +51,23 @@
 ll_initxattrs(struct inode *inode, const struct xattr *xattr_array,
 	      void *fs_info)
 {
-	const struct xattr_handler *handler;
 	struct dentry *dentry = fs_info;
 	const struct xattr *xattr;
 	int err = 0;
 
-	handler = get_xattr_type(XATTR_SECURITY_PREFIX);
-	if (!handler)
-		return -ENXIO;
-
 	for (xattr = xattr_array; xattr->name; xattr++) {
-		err = handler->set(handler, dentry, inode, xattr->name,
-				   xattr->value, xattr->value_len,
-				   XATTR_CREATE);
+		char *full_name;
+
+		full_name = kasprintf(GFP_KERNEL, "%s%s",
+				      XATTR_SECURITY_PREFIX, xattr->name);
+		if (!full_name) {
+			err = -ENOMEM;
+			break;
+		}
+
+		err = __vfs_setxattr(dentry, inode, full_name, xattr->value,
+				     xattr->value_len, XATTR_CREATE);
+		kfree(full_name);
 		if (err < 0)
 			break;
 	}
-- 
1.8.3.1

From jsimmons at infradead.org  Wed Jul 26 15:22:21 2017
From: jsimmons at infradead.org (James Simmons)
Date: Wed, 26 Jul 2017 11:22:21 -0400
Subject: [lustre-devel] [PATCH 05/20] staging: lustre: lov: remove unused
	code
In-Reply-To: <1501082556-17962-1-git-send-email-jsimmons@infradead.org>
References: <1501082556-17962-1-git-send-email-jsimmons@infradead.org>
Message-ID: <1501082556-17962-6-git-send-email-jsimmons@infradead.org>

From: "John L. Hammond" <john.hammond at intel.com>

Remove:
  the tested but never set flag OBD_STATFS_PTLRPCD,
  the empty file lustre/lov/lovsub_io.c,
  the unused ld_emerg member of struct lov_device,
  the unused struct lov_device_emerg and supporting functions,
  the unused struct lov_lock_link and supporting functions, and
  the unused, get only, or set only members of struct
  lovsub_device, lovsub_lock, lov_sublock_env, lov_thread_info,
  lov_io_sub, lov_io, lov_request, and lov_request_set.
Reduce the scope of several functions from lov_request.c.

Signed-off-by: John L. Hammond <john.hammond at intel.com>
Intel-bug-id: https://jira.hpdd.intel.com/browse/LU-5814
Reviewed-on: http://review.whamcloud.com/14878
Reviewed-by: Frank Zago <fzago at cray.com>
Reviewed-by: Ben Evans <bevans at cray.com>
Reviewed-by: James Simmons <uja.ornl at yahoo.com>
Reviewed-by: Oleg Drokin <oleg.drokin at intel.com>
Signed-off-by: James Simmons <jsimmons at infradead.org>
---
 drivers/staging/lustre/lustre/include/obd_class.h  |   9 +-
 drivers/staging/lustre/lustre/lov/Makefile         |   2 +-
 .../staging/lustre/lustre/lov/lov_cl_internal.h    |  90 -------------
 drivers/staging/lustre/lustre/lov/lov_dev.c        |  85 ------------
 drivers/staging/lustre/lustre/lov/lov_internal.h   |  27 ----
 drivers/staging/lustre/lustre/lov/lov_io.c         | 144 ++++++---------------
 drivers/staging/lustre/lustre/lov/lov_lock.c       |  11 --
 drivers/staging/lustre/lustre/lov/lov_obd.c        |   3 +-
 drivers/staging/lustre/lustre/lov/lov_page.c       |   1 -
 drivers/staging/lustre/lustre/lov/lov_request.c    |  40 +-----
 drivers/staging/lustre/lustre/lov/lovsub_dev.c     |   1 -
 drivers/staging/lustre/lustre/lov/lovsub_io.c      |  51 --------
 drivers/staging/lustre/lustre/lov/lovsub_lock.c    |   2 -
 13 files changed, 53 insertions(+), 413 deletions(-)
 delete mode 100644 drivers/staging/lustre/lustre/lov/lovsub_io.c

diff --git a/drivers/staging/lustre/lustre/include/obd_class.h b/drivers/staging/lustre/lustre/include/obd_class.h
index 083a6ff..a6a4b2f 100644
--- a/drivers/staging/lustre/lustre/include/obd_class.h
+++ b/drivers/staging/lustre/lustre/include/obd_class.h
@@ -46,14 +46,7 @@
 #define OBD_STATFS_FROM_CACHE   0x0002  /* the statfs callback should not update
 					 * obd_osfs_age
 					 */
-#define OBD_STATFS_PTLRPCD      0x0004  /* requests will be sent via ptlrpcd
-					 * instead of a specific set. This
-					 * means that we cannot rely on the set
-					 * interpret routine to be called.
-					 * lov_statfs_fini() must thus be called
-					 * by the request interpret routine
-					 */
-#define OBD_STATFS_FOR_MDT0	0x0008	/* The statfs is only for retrieving
+#define OBD_STATFS_FOR_MDT0	0x0004	/* The statfs is only for retrieving
 					 * information from MDT0.
 					 */
 
diff --git a/drivers/staging/lustre/lustre/lov/Makefile b/drivers/staging/lustre/lustre/lov/Makefile
index e4cc0db..ea93add 100644
--- a/drivers/staging/lustre/lustre/lov/Makefile
+++ b/drivers/staging/lustre/lustre/lov/Makefile
@@ -2,4 +2,4 @@ obj-$(CONFIG_LUSTRE_FS) += lov.o
 lov-y := lov_obd.o lov_pack.o lov_offset.o lov_merge.o \
 	 lov_request.o lov_ea.o lov_dev.o lov_object.o lov_page.o  \
 	 lov_lock.o lov_io.o lovsub_dev.o lovsub_object.o lovsub_page.o      \
-	 lovsub_lock.o lovsub_io.o lov_pool.o lproc_lov.o
+	 lovsub_lock.o lov_pool.o lproc_lov.o
diff --git a/drivers/staging/lustre/lustre/lov/lov_cl_internal.h b/drivers/staging/lustre/lustre/lov/lov_cl_internal.h
index e889d3a..38281b2 100644
--- a/drivers/staging/lustre/lustre/lov/lov_cl_internal.h
+++ b/drivers/staging/lustre/lustre/lov/lov_cl_internal.h
@@ -92,35 +92,6 @@ enum lov_device_flags {
  * Upper half.
  */
 
-/**
- * Resources that are used in memory-cleaning path, and whose allocation
- * cannot fail even when memory is tight. They are preallocated in sufficient
- * quantities in lov_device::ld_emerg[], and access to them is serialized
- * lov_device::ld_mutex.
- */
-struct lov_device_emerg {
-	/**
-	 * Page list used to submit IO when memory is in pressure.
-	 */
-	struct cl_page_list emrg_page_list;
-	/**
-	 * sub-io's shared by all threads accessing this device when memory is
-	 * too low to allocate sub-io's dynamically.
-	 */
-	struct cl_io	emrg_subio;
-	/**
-	 * Environments used by sub-io's in
-	 * lov_device_emerg::emrg_subio.
-	 */
-	struct lu_env      *emrg_env;
-	/**
-	 * Refchecks for lov_device_emerg::emrg_env.
-	 *
-	 * \see cl_env_get()
-	 */
-	u16		 emrg_refcheck;
-};
-
 struct lov_device {
 	/*
 	 * XXX Locking of lov-private data is missing.
@@ -131,14 +102,6 @@ struct lov_device {
 	__u32		     ld_target_nr;
 	struct lovsub_device    **ld_target;
 	__u32		     ld_flags;
-
-	/** Emergency resources used in memory-cleansing paths. */
-	struct lov_device_emerg **ld_emrg;
-	/**
-	 * Serializes access to lov_device::ld_emrg in low-memory
-	 * conditions.
-	 */
-	struct mutex		  ld_mutex;
 };
 
 /**
@@ -299,8 +262,6 @@ struct lov_page {
 
 struct lovsub_device {
 	struct cl_device   acid_cl;
-	struct lov_device *acid_super;
-	int		acid_idx;
 	struct cl_device  *acid_next;
 };
 
@@ -312,42 +273,10 @@ struct lovsub_object {
 };
 
 /**
- * A link between a top-lock and a sub-lock. Separate data-structure is
- * necessary, because top-locks and sub-locks are in M:N relationship.
- *
- * \todo This can be optimized for a (by far) most frequent case of a single
- * top-lock per sub-lock.
- */
-struct lov_lock_link {
-	struct lov_lock *lll_super;
-	/** An index within parent lock. */
-	int	      lll_idx;
-	/**
-	 * A linkage into per sub-lock list of all corresponding top-locks,
-	 * hanging off lovsub_lock::lss_parents.
-	 */
-	struct list_head       lll_list;
-};
-
-/**
  * Lock state at lovsub layer.
  */
 struct lovsub_lock {
 	struct cl_lock_slice  lss_cl;
-	/**
-	 * List of top-locks that have given sub-lock as their part. Protected
-	 * by cl_lock::cll_guard mutex.
-	 */
-	struct list_head	    lss_parents;
-	/**
-	 * Top-lock that initiated current operation on this sub-lock. This is
-	 * only set during top-to-bottom lock operations like enqueue, and is
-	 * used to optimize state change notification. Protected by
-	 * cl_lock::cll_guard mutex.
-	 *
-	 * \see lovsub_lock_state_one().
-	 */
-	struct cl_lock       *lss_active;
 };
 
 /**
@@ -356,7 +285,6 @@ struct lovsub_lock {
 struct lov_sublock_env {
 	const struct lu_env *lse_env;
 	struct cl_io	*lse_io;
-	struct lov_io_sub   *lse_sub;
 };
 
 struct lovsub_page {
@@ -366,12 +294,10 @@ struct lovsub_page {
 struct lov_thread_info {
 	struct cl_object_conf   lti_stripe_conf;
 	struct lu_fid	   lti_fid;
-	struct cl_lock_descr    lti_ldescr;
 	struct ost_lvb	  lti_lvb;
 	struct cl_2queue	lti_cl2q;
 	struct cl_page_list     lti_plist;
 	wait_queue_entry_t	  lti_waiter;
-	struct cl_attr          lti_attr;
 };
 
 /**
@@ -385,7 +311,6 @@ struct lov_io_sub {
 	 * \see cl_env_get()
 	 */
 	u16			 sub_refcheck;
-	u16			 sub_reenter;
 	/**
 	 * true, iff cl_io_init() was successfully executed against
 	 * lov_io_sub::sub_io.
@@ -445,7 +370,6 @@ struct lov_io {
 	 */
 	u64	    lis_endpos;
 
-	int		lis_mem_frozen;
 	int		lis_stripe_count;
 	int		lis_active_subios;
 
@@ -485,8 +409,6 @@ struct lov_session {
 extern struct kmem_cache *lovsub_lock_kmem;
 extern struct kmem_cache *lovsub_object_kmem;
 
-extern struct kmem_cache *lov_lock_link_kmem;
-
 int lov_object_init(const struct lu_env *env, struct lu_object *obj,
 		    const struct lu_object_conf *conf);
 int lovsub_object_init(const struct lu_env *env, struct lu_object *obj,
@@ -508,15 +430,9 @@ int lov_io_init_empty(const struct lu_env *env, struct cl_object *obj,
 		      struct cl_io *io);
 int lov_io_init_released(const struct lu_env *env, struct cl_object *obj,
 			 struct cl_io *io);
-void lov_lock_unlink(const struct lu_env *env, struct lov_lock_link *link,
-		     struct lovsub_lock *sub);
 
 struct lov_io_sub *lov_sub_get(const struct lu_env *env, struct lov_io *lio,
 			       int stripe);
-void lov_sub_put(struct lov_io_sub *sub);
-int lov_sublock_modify(const struct lu_env *env, struct lov_lock *lov,
-		       struct lovsub_lock *sublock,
-		       const struct cl_lock_descr *d, int idx);
 
 int lov_page_init(const struct lu_env *env, struct cl_object *ob,
 		  struct cl_page *page, pgoff_t index);
@@ -533,12 +449,6 @@ struct lu_object *lovsub_object_alloc(const struct lu_env *env,
 				      const struct lu_object_header *hdr,
 				      struct lu_device *dev);
 
-struct lov_lock_link *lov_lock_link_find(const struct lu_env *env,
-					 struct lov_lock *lck,
-					 struct lovsub_lock *sub);
-struct lov_io_sub *lov_page_subio(const struct lu_env *env, struct lov_io *lio,
-				  const struct cl_page_slice *slice);
-
 struct lov_stripe_md *lov_lsm_addref(struct lov_object *lov);
 int lov_page_stripe(const struct cl_page *page);
 
diff --git a/drivers/staging/lustre/lustre/lov/lov_dev.c b/drivers/staging/lustre/lustre/lov/lov_dev.c
index 7301f6e5..531b4fe 100644
--- a/drivers/staging/lustre/lustre/lov/lov_dev.c
+++ b/drivers/staging/lustre/lustre/lov/lov_dev.c
@@ -50,11 +50,6 @@
 struct kmem_cache *lovsub_lock_kmem;
 struct kmem_cache *lovsub_object_kmem;
 
-struct kmem_cache *lov_lock_link_kmem;
-
-/** Lock class of lov_device::ld_mutex. */
-static struct lock_class_key cl_lov_device_mutex_class;
-
 struct lu_kmem_descr lov_caches[] = {
 	{
 		.ckd_cache = &lov_lock_kmem,
@@ -87,11 +82,6 @@ struct lu_kmem_descr lov_caches[] = {
 		.ckd_size  = sizeof(struct lovsub_object)
 	},
 	{
-		.ckd_cache = &lov_lock_link_kmem,
-		.ckd_name  = "lov_lock_link_kmem",
-		.ckd_size  = sizeof(struct lov_lock_link)
-	},
-	{
 		.ckd_cache = NULL
 	}
 };
@@ -204,8 +194,6 @@ static int lov_device_init(const struct lu_env *env, struct lu_device *d,
 			break;
 		}
 		lsd = cl2lovsub_dev(cl);
-		lsd->acid_idx = i;
-		lsd->acid_super = ld;
 		ld->ld_target[i] = lsd;
 	}
 
@@ -217,34 +205,13 @@ static int lov_device_init(const struct lu_env *env, struct lu_device *d,
 	return rc;
 }
 
-static void lov_emerg_free(struct lov_device_emerg **emrg, int nr)
-{
-	int i;
-
-	for (i = 0; i < nr; ++i) {
-		struct lov_device_emerg *em;
-
-		em = emrg[i];
-		if (em) {
-			LASSERT(em->emrg_page_list.pl_nr == 0);
-			if (em->emrg_env)
-				cl_env_put(em->emrg_env, &em->emrg_refcheck);
-			kfree(em);
-		}
-	}
-	kfree(emrg);
-}
-
 static struct lu_device *lov_device_free(const struct lu_env *env,
 					 struct lu_device *d)
 {
 	struct lov_device *ld = lu2lov_dev(d);
-	const int	  nr = ld->ld_target_nr;
 
 	cl_device_fini(lu2cl_dev(d));
 	kfree(ld->ld_target);
-	if (ld->ld_emrg)
-		lov_emerg_free(ld->ld_emrg, nr);
 	kfree(ld);
 	return NULL;
 }
@@ -260,41 +227,6 @@ static void lov_cl_del_target(const struct lu_env *env, struct lu_device *dev,
 	}
 }
 
-static struct lov_device_emerg **lov_emerg_alloc(int nr)
-{
-	struct lov_device_emerg **emerg;
-	int i;
-	int result;
-
-	emerg = kcalloc(nr, sizeof(emerg[0]), GFP_NOFS);
-	if (!emerg)
-		return ERR_PTR(-ENOMEM);
-	for (result = i = 0; i < nr && result == 0; i++) {
-		struct lov_device_emerg *em;
-
-		em = kzalloc(sizeof(*em), GFP_NOFS);
-		if (em) {
-			emerg[i] = em;
-			cl_page_list_init(&em->emrg_page_list);
-			em->emrg_env = cl_env_alloc(&em->emrg_refcheck,
-						    LCT_REMEMBER | LCT_NOREF);
-			if (!IS_ERR(em->emrg_env)) {
-				em->emrg_env->le_ctx.lc_cookie = 0x2;
-			} else {
-				result = PTR_ERR(em->emrg_env);
-				em->emrg_env = NULL;
-			}
-		} else {
-			result = -ENOMEM;
-		}
-	}
-	if (result != 0) {
-		lov_emerg_free(emerg, nr);
-		emerg = ERR_PTR(result);
-	}
-	return emerg;
-}
-
 static int lov_expand_targets(const struct lu_env *env, struct lov_device *dev)
 {
 	int   result;
@@ -306,29 +238,17 @@ static int lov_expand_targets(const struct lu_env *env, struct lov_device *dev)
 	sub_size = dev->ld_target_nr;
 	if (sub_size < tgt_size) {
 		struct lovsub_device    **newd;
-		struct lov_device_emerg **emerg;
 		const size_t	      sz   = sizeof(newd[0]);
 
-		emerg = lov_emerg_alloc(tgt_size);
-		if (IS_ERR(emerg))
-			return PTR_ERR(emerg);
-
 		newd = kcalloc(tgt_size, sz, GFP_NOFS);
 		if (newd) {
-			mutex_lock(&dev->ld_mutex);
 			if (sub_size > 0) {
 				memcpy(newd, dev->ld_target, sub_size * sz);
 				kfree(dev->ld_target);
 			}
 			dev->ld_target    = newd;
 			dev->ld_target_nr = tgt_size;
-
-			if (dev->ld_emrg)
-				lov_emerg_free(dev->ld_emrg, sub_size);
-			dev->ld_emrg = emerg;
-			mutex_unlock(&dev->ld_mutex);
 		} else {
-			lov_emerg_free(emerg, tgt_size);
 			result = -ENOMEM;
 		}
 	}
@@ -362,8 +282,6 @@ static int lov_cl_add_target(const struct lu_env *env, struct lu_device *dev,
 				   tgt->ltd_obd->obd_lu_dev);
 		if (!IS_ERR(cl)) {
 			lsd = cl2lovsub_dev(cl);
-			lsd->acid_idx = index;
-			lsd->acid_super = ld;
 			ld->ld_target[index] = lsd;
 		} else {
 			CERROR("add failed (%d), deleting %s\n", rc,
@@ -428,9 +346,6 @@ static struct lu_device *lov_device_alloc(const struct lu_env *env,
 	d = lov2lu_dev(ld);
 	d->ld_ops	= &lov_lu_ops;
 
-	mutex_init(&ld->ld_mutex);
-	lockdep_set_class(&ld->ld_mutex, &cl_lov_device_mutex_class);
-
 	/* setup the LOV OBD */
 	obd = class_name2obd(lustre_cfg_string(cfg, 0));
 	LASSERT(obd);
diff --git a/drivers/staging/lustre/lustre/lov/lov_internal.h b/drivers/staging/lustre/lustre/lov/lov_internal.h
index 774499c..9905df2 100644
--- a/drivers/staging/lustre/lustre/lov/lov_internal.h
+++ b/drivers/staging/lustre/lustre/lov/lov_internal.h
@@ -161,42 +161,21 @@ struct lov_request {
 	struct list_head	       rq_link;
 
 	int		      rq_idx;	/* index in lov->tgts array */
-	int		      rq_stripe;     /* stripe number */
-	int		      rq_complete;
-	int		      rq_rc;
-
-	u32		      rq_oabufs;
-	u32		      rq_pgaidx;
 };
 
 struct lov_request_set {
 	struct obd_info			*set_oi;
-	atomic_t			set_refcount;
-	struct obd_export		*set_exp;
-	/* XXX: There is @set_exp already, however obd_statfs gets obd_device
-	 * only.
-	 */
 	struct obd_device		*set_obd;
 	int				set_count;
 	atomic_t			set_completes;
 	atomic_t			set_success;
-	atomic_t			set_finish_checked;
 	struct list_head			set_list;
-	wait_queue_head_t			set_waitq;
 };
 
 extern struct kmem_cache *lov_oinfo_slab;
 
 extern struct lu_kmem_descr lov_caches[];
 
-void lov_finish_set(struct lov_request_set *set);
-
-static inline void lov_put_reqset(struct lov_request_set *set)
-{
-	if (atomic_dec_and_test(&set->set_refcount))
-		lov_finish_set(set);
-}
-
 #define lov_uuid2str(lv, index) \
 	(char *)((lv)->lov_tgts[index]->ltd_uuid.uuid)
 
@@ -217,15 +196,9 @@ pgoff_t lov_stripe_pgoff(struct lov_stripe_md *lsm, pgoff_t stripe_index,
 			 int stripe);
 
 /* lov_request.c */
-int lov_prep_getattr_set(struct obd_export *exp, struct obd_info *oinfo,
-			 struct lov_request_set **reqset);
-int lov_fini_getattr_set(struct lov_request_set *set);
 int lov_prep_statfs_set(struct obd_device *obd, struct obd_info *oinfo,
 			struct lov_request_set **reqset);
-int lov_fini_statfs(struct obd_device *obd, struct obd_statfs *osfs,
-		    int success);
 int lov_fini_statfs_set(struct lov_request_set *set);
-int lov_statfs_interpret(struct ptlrpc_request_set *rqset, void *data, int rc);
 
 /* lov_obd.c */
 void lov_stripe_lock(struct lov_stripe_md *md);
diff --git a/drivers/staging/lustre/lustre/lov/lov_io.c b/drivers/staging/lustre/lustre/lov/lov_io.c
index 3e37353..f6b5df7 100644
--- a/drivers/staging/lustre/lustre/lov/lov_io.c
+++ b/drivers/staging/lustre/lustre/lov/lov_io.c
@@ -43,24 +43,12 @@
  *  @{
  */
 
-static inline void lov_sub_enter(struct lov_io_sub *sub)
-{
-	sub->sub_reenter++;
-}
-
-static inline void lov_sub_exit(struct lov_io_sub *sub)
-{
-	sub->sub_reenter--;
-}
-
 static void lov_io_sub_fini(const struct lu_env *env, struct lov_io *lio,
 			    struct lov_io_sub *sub)
 {
 	if (sub->sub_io) {
 		if (sub->sub_io_initialized) {
-			lov_sub_enter(sub);
 			cl_io_fini(sub->sub_env, sub->sub_io);
-			lov_sub_exit(sub);
 			sub->sub_io_initialized = 0;
 			lio->lis_active_subios--;
 		}
@@ -142,13 +130,11 @@ static int lov_io_sub_init(const struct lu_env *env, struct lov_io *lio,
 			   struct lov_io_sub *sub)
 {
 	struct lov_object *lov = lio->lis_object;
-	struct lov_device *ld  = lu2lov_dev(lov2cl(lov)->co_lu.lo_dev);
 	struct cl_io      *sub_io;
 	struct cl_object  *sub_obj;
 	struct cl_io      *io  = lio->lis_cl.cis_io;
-
 	int stripe = sub->sub_stripe;
-	int result;
+	int rc;
 
 	LASSERT(!sub->sub_io);
 	LASSERT(!sub->sub_env);
@@ -157,63 +143,53 @@ static int lov_io_sub_init(const struct lu_env *env, struct lov_io *lio,
 	if (unlikely(!lov_r0(lov)->lo_sub[stripe]))
 		return -EIO;
 
-	result = 0;
 	sub->sub_io_initialized = 0;
 	sub->sub_borrowed = 0;
 
-	if (lio->lis_mem_frozen) {
-		LASSERT(mutex_is_locked(&ld->ld_mutex));
-		sub->sub_io  = &ld->ld_emrg[stripe]->emrg_subio;
-		sub->sub_env = ld->ld_emrg[stripe]->emrg_env;
-		sub->sub_borrowed = 1;
-	} else {
-		sub->sub_env = cl_env_get(&sub->sub_refcheck);
-		if (IS_ERR(sub->sub_env))
-			result = PTR_ERR(sub->sub_env);
+	/* obtain new environment */
+	sub->sub_env = cl_env_get(&sub->sub_refcheck);
+	if (IS_ERR(sub->sub_env)) {
+		rc = PTR_ERR(sub->sub_env);
+		goto fini_lov_io;
+	}
 
-		if (result == 0) {
-			/*
-			 * First sub-io. Use ->lis_single_subio to
-			 * avoid dynamic allocation.
-			 */
-			if (lio->lis_active_subios == 0) {
-				sub->sub_io = &lio->lis_single_subio;
-				lio->lis_single_subio_index = stripe;
-			} else {
-				sub->sub_io = kzalloc(sizeof(*sub->sub_io),
-						      GFP_NOFS);
-				if (!sub->sub_io)
-					result = -ENOMEM;
-			}
+	/*
+	 * First sub-io. Use ->lis_single_subio to
+	 * avoid dynamic allocation.
+	 */
+	if (lio->lis_active_subios == 0) {
+		sub->sub_io = &lio->lis_single_subio;
+		lio->lis_single_subio_index = stripe;
+	} else {
+		sub->sub_io = kzalloc(sizeof(*sub->sub_io),
+				      GFP_NOFS);
+		if (!sub->sub_io) {
+			rc = -ENOMEM;
+			goto fini_lov_io;
 		}
 	}
 
-	if (result == 0) {
-		sub_obj = lovsub2cl(lov_r0(lov)->lo_sub[stripe]);
-		sub_io  = sub->sub_io;
-
-		sub_io->ci_obj    = sub_obj;
-		sub_io->ci_result = 0;
-
-		sub_io->ci_parent  = io;
-		sub_io->ci_lockreq = io->ci_lockreq;
-		sub_io->ci_type    = io->ci_type;
-		sub_io->ci_no_srvlock = io->ci_no_srvlock;
-		sub_io->ci_noatime = io->ci_noatime;
-
-		lov_sub_enter(sub);
-		result = cl_io_sub_init(sub->sub_env, sub_io,
-					io->ci_type, sub_obj);
-		lov_sub_exit(sub);
-		if (result >= 0) {
-			lio->lis_active_subios++;
-			sub->sub_io_initialized = 1;
-			result = 0;
-		}
+	sub_obj = lovsub2cl(lov_r0(lov)->lo_sub[stripe]);
+	sub_io = sub->sub_io;
+
+	sub_io->ci_obj = sub_obj;
+	sub_io->ci_result = 0;
+	sub_io->ci_parent = io;
+	sub_io->ci_lockreq = io->ci_lockreq;
+	sub_io->ci_type = io->ci_type;
+	sub_io->ci_no_srvlock = io->ci_no_srvlock;
+	sub_io->ci_noatime = io->ci_noatime;
+
+	rc = cl_io_sub_init(sub->sub_env, sub_io, io->ci_type, sub_obj);
+	if (rc >= 0) {
+		lio->lis_active_subios++;
+		sub->sub_io_initialized = 1;
+		rc = 0;
 	}
-	if (result != 0)
+fini_lov_io:
+	if (rc)
 		lov_io_sub_fini(env, lio, sub);
-	return result;
+	return rc;
 }
 
 struct lov_io_sub *lov_sub_get(const struct lu_env *env,
@@ -230,16 +206,10 @@ struct lov_io_sub *lov_sub_get(const struct lu_env *env,
 	} else {
 		rc = 0;
 	}
-	if (rc == 0)
-		lov_sub_enter(sub);
-	else
+	if (rc < 0)
 		sub = ERR_PTR(rc);
-	return sub;
-}
 
-void lov_sub_put(struct lov_io_sub *sub)
-{
-	lov_sub_exit(sub);
+	return sub;
 }
 
 /*****************************************************************************
@@ -258,22 +228,6 @@ int lov_page_stripe(const struct cl_page *page)
 	return cl2lov_page(slice)->lps_stripe;
 }
 
-struct lov_io_sub *lov_page_subio(const struct lu_env *env, struct lov_io *lio,
-				  const struct cl_page_slice *slice)
-{
-	struct lov_stripe_md *lsm  = lio->lis_object->lo_lsm;
-	struct cl_page       *page = slice->cpl_page;
-	int stripe;
-
-	LASSERT(lio->lis_cl.cis_io);
-	LASSERT(cl2lov(slice->cpl_obj) == lio->lis_object);
-	LASSERT(lsm);
-	LASSERT(lio->lis_nr_subios > 0);
-
-	stripe = lov_page_stripe(page);
-	return lov_sub_get(env, lio, stripe);
-}
-
 static int lov_io_subio_init(const struct lu_env *env, struct lov_io *lio,
 			     struct cl_io *io)
 {
@@ -431,12 +385,10 @@ static int lov_io_iter_init(const struct lu_env *env,
 
 		lov_io_sub_inherit(sub->sub_io, lio, stripe, start, end);
 		rc = cl_io_iter_init(sub->sub_env, sub->sub_io);
-		if (rc)
+		if (rc) {
 			cl_io_iter_fini(sub->sub_env, sub->sub_io);
-		lov_sub_put(sub);
-		if (rc)
 			break;
-
+		}
 		CDEBUG(D_VFSTRACE, "shrink: %d [%llu, %llu)\n",
 		       stripe, start, end);
 
@@ -488,9 +440,7 @@ static int lov_io_call(const struct lu_env *env, struct lov_io *lio,
 	int rc = 0;
 
 	list_for_each_entry(sub, &lio->lis_active, sub_linkage) {
-		lov_sub_enter(sub);
 		rc = iofunc(sub->sub_env, sub->sub_io);
-		lov_sub_exit(sub);
 		if (rc)
 			break;
 
@@ -610,7 +560,6 @@ static int lov_io_read_ahead(const struct lu_env *env,
 	rc = cl_io_read_ahead(sub->sub_env, sub->sub_io,
 			      cl_index(lovsub2cl(r0->lo_sub[stripe]), suboff),
 			      ra);
-	lov_sub_put(sub);
 
 	CDEBUG(D_READA, DFID " cra_end = %lu, stripes = %d, rc = %d\n",
 	       PFID(lu_object_fid(lov2lu(loo))), ra->cra_end, r0->lo_nr, rc);
@@ -679,7 +628,6 @@ static int lov_io_submit(const struct lu_env *env,
 		LASSERT(sub->sub_io == &lio->lis_single_subio);
 		rc = cl_io_submit_rw(sub->sub_env, sub->sub_io,
 				     crt, queue);
-		lov_sub_put(sub);
 		return rc;
 	}
 
@@ -707,7 +655,6 @@ static int lov_io_submit(const struct lu_env *env,
 		if (!IS_ERR(sub)) {
 			rc = cl_io_submit_rw(sub->sub_env, sub->sub_io,
 					     crt, cl2q);
-			lov_sub_put(sub);
 		} else {
 			rc = PTR_ERR(sub);
 		}
@@ -746,7 +693,6 @@ static int lov_io_commit_async(const struct lu_env *env,
 		LASSERT(sub->sub_io == &lio->lis_single_subio);
 		rc = cl_io_commit_async(sub->sub_env, sub->sub_io, queue,
 					from, to, cb);
-		lov_sub_put(sub);
 		return rc;
 	}
 
@@ -777,7 +723,6 @@ static int lov_io_commit_async(const struct lu_env *env,
 		if (!IS_ERR(sub)) {
 			rc = cl_io_commit_async(sub->sub_env, sub->sub_io,
 						plist, from, stripe_to, cb);
-			lov_sub_put(sub);
 		} else {
 			rc = PTR_ERR(sub);
 			break;
@@ -813,7 +758,6 @@ static int lov_io_fault_start(const struct lu_env *env,
 	if (IS_ERR(sub))
 		return PTR_ERR(sub);
 	sub->sub_io->u.ci_fault.ft_nob = fio->ft_nob;
-	lov_sub_put(sub);
 	return lov_io_start(env, ios);
 }
 
@@ -828,9 +772,7 @@ static void lov_io_fsync_end(const struct lu_env *env,
 	list_for_each_entry(sub, &lio->lis_active, sub_linkage) {
 		struct cl_io *subio = sub->sub_io;
 
-		lov_sub_enter(sub);
 		lov_io_end_wrapper(sub->sub_env, subio);
-		lov_sub_exit(sub);
 
 		if (subio->ci_result == 0)
 			*written += subio->u.ci_fsync.fi_nr_written;
diff --git a/drivers/staging/lustre/lustre/lov/lov_lock.c b/drivers/staging/lustre/lustre/lov/lov_lock.c
index 8502128..7d0d3ea 100644
--- a/drivers/staging/lustre/lustre/lov/lov_lock.c
+++ b/drivers/staging/lustre/lustre/lov/lov_lock.c
@@ -71,13 +71,11 @@ static struct lov_sublock_env *lov_sublock_env_get(const struct lu_env *env,
 	if (!io || !cl_object_same(io->ci_obj, parent->cll_descr.cld_obj)) {
 		subenv->lse_env = env;
 		subenv->lse_io  = io;
-		subenv->lse_sub = NULL;
 	} else {
 		sub = lov_sub_get(env, lio, lls->sub_stripe);
 		if (!IS_ERR(sub)) {
 			subenv->lse_env = sub->sub_env;
 			subenv->lse_io  = sub->sub_io;
-			subenv->lse_sub = sub;
 		} else {
 			subenv = (void *)sub;
 		}
@@ -85,12 +83,6 @@ static struct lov_sublock_env *lov_sublock_env_get(const struct lu_env *env,
 	return subenv;
 }
 
-static void lov_sublock_env_put(struct lov_sublock_env *subenv)
-{
-	if (subenv && subenv->lse_sub)
-		lov_sub_put(subenv->lse_sub);
-}
-
 static int lov_sublock_init(const struct lu_env *env,
 			    const struct cl_lock *parent,
 			    struct lov_lock_sub *lls)
@@ -102,7 +94,6 @@ static int lov_sublock_init(const struct lu_env *env,
 	if (!IS_ERR(subenv)) {
 		result = cl_lock_init(subenv->lse_env, &lls->sub_lock,
 				      subenv->lse_io);
-		lov_sublock_env_put(subenv);
 	} else {
 		/* error occurs. */
 		result = PTR_ERR(subenv);
@@ -244,7 +235,6 @@ static int lov_lock_enqueue(const struct lu_env *env,
 		}
 		rc = cl_lock_enqueue(subenv->lse_env, subenv->lse_io,
 				     &lls->sub_lock, anchor);
-		lov_sublock_env_put(subenv);
 		if (rc != 0)
 			break;
 
@@ -272,7 +262,6 @@ static void lov_lock_cancel(const struct lu_env *env,
 		subenv = lov_sublock_env_get(env, lock, lls);
 		if (!IS_ERR(subenv)) {
 			cl_lock_cancel(subenv->lse_env, sublock);
-			lov_sublock_env_put(subenv);
 		} else {
 			CL_LOCK_DEBUG(D_ERROR, env, slice->cls_lock,
 				      "lov_lock_cancel fails with %ld.\n",
diff --git a/drivers/staging/lustre/lustre/lov/lov_obd.c b/drivers/staging/lustre/lustre/lov/lov_obd.c
index 25f15da..f27b11f 100644
--- a/drivers/staging/lustre/lustre/lov/lov_obd.c
+++ b/drivers/staging/lustre/lustre/lov/lov_obd.c
@@ -947,7 +947,8 @@ int lov_process_config_base(struct obd_device *obd, struct lustre_cfg *lcfg,
 	return rc;
 }
 
-int lov_statfs_interpret(struct ptlrpc_request_set *rqset, void *data, int rc)
+static int
+lov_statfs_interpret(struct ptlrpc_request_set *rqset, void *data, int rc)
 {
 	struct lov_request_set *lovset = (struct lov_request_set *)data;
 	int err;
diff --git a/drivers/staging/lustre/lustre/lov/lov_page.c b/drivers/staging/lustre/lustre/lov/lov_page.c
index 62ceb6d..de43c60 100644
--- a/drivers/staging/lustre/lustre/lov/lov_page.c
+++ b/drivers/staging/lustre/lustre/lov/lov_page.c
@@ -100,7 +100,6 @@ int lov_page_init_raid0(const struct lu_env *env, struct cl_object *obj,
 				break;
 		}
 	}
-	lov_sub_put(sub);
 
 	return rc;
 }
diff --git a/drivers/staging/lustre/lustre/lov/lov_request.c b/drivers/staging/lustre/lustre/lov/lov_request.c
index 3a74791..70636e5 100644
--- a/drivers/staging/lustre/lustre/lov/lov_request.c
+++ b/drivers/staging/lustre/lustre/lov/lov_request.c
@@ -43,13 +43,10 @@ static void lov_init_set(struct lov_request_set *set)
 	set->set_count = 0;
 	atomic_set(&set->set_completes, 0);
 	atomic_set(&set->set_success, 0);
-	atomic_set(&set->set_finish_checked, 0);
 	INIT_LIST_HEAD(&set->set_list);
-	atomic_set(&set->set_refcount, 1);
-	init_waitqueue_head(&set->set_waitq);
 }
 
-void lov_finish_set(struct lov_request_set *set)
+static void lov_finish_set(struct lov_request_set *set)
 {
 	struct list_head *pos, *n;
 
@@ -66,32 +63,12 @@ void lov_finish_set(struct lov_request_set *set)
 	kfree(set);
 }
 
-static int lov_set_finished(struct lov_request_set *set, int idempotent)
-{
-	int completes = atomic_read(&set->set_completes);
-
-	CDEBUG(D_INFO, "check set %d/%d\n", completes, set->set_count);
-
-	if (completes == set->set_count) {
-		if (idempotent)
-			return 1;
-		if (atomic_inc_return(&set->set_finish_checked) == 1)
-			return 1;
-	}
-	return 0;
-}
-
 static void lov_update_set(struct lov_request_set *set,
 			   struct lov_request *req, int rc)
 {
-	req->rq_complete = 1;
-	req->rq_rc = rc;
-
 	atomic_inc(&set->set_completes);
 	if (rc == 0)
 		atomic_inc(&set->set_success);
-
-	wake_up(&set->set_waitq);
 }
 
 static void lov_set_add_req(struct lov_request *req,
@@ -173,8 +150,8 @@ static int lov_check_and_wait_active(struct lov_obd *lov, int ost_idx)
 			(tot) += (add);				 \
 	} while (0)
 
-int lov_fini_statfs(struct obd_device *obd, struct obd_statfs *osfs,
-		    int success)
+static int lov_fini_statfs(struct obd_device *obd, struct obd_statfs *osfs,
+			   int success)
 {
 	if (success) {
 		__u32 expected_stripes = lov_get_stripecnt(&obd->u.lov,
@@ -205,7 +182,9 @@ int lov_fini_statfs_set(struct lov_request_set *set)
 		rc = lov_fini_statfs(set->set_obd, set->set_oi->oi_osfs,
 				     atomic_read(&set->set_success));
 	}
-	lov_put_reqset(set);
+
+	lov_finish_set(set);
+
 	return rc;
 }
 
@@ -307,14 +286,7 @@ static int cb_statfs_update(void *cookie, int rc)
 out_update:
 	lov_update_statfs(osfs, lov_sfs, success);
 	obd_putref(lovobd);
-
 out:
-	if (set->set_oi->oi_flags & OBD_STATFS_PTLRPCD &&
-	    lov_set_finished(set, 0)) {
-		lov_statfs_interpret(NULL, set, set->set_count !=
-				     atomic_read(&set->set_success));
-	}
-
 	return 0;
 }
 
diff --git a/drivers/staging/lustre/lustre/lov/lovsub_dev.c b/drivers/staging/lustre/lustre/lov/lovsub_dev.c
index 5d6536f..d4646a0 100644
--- a/drivers/staging/lustre/lustre/lov/lovsub_dev.c
+++ b/drivers/staging/lustre/lustre/lov/lovsub_dev.c
@@ -77,7 +77,6 @@ static struct lu_device *lovsub_device_fini(const struct lu_env *env,
 
 	lsd = lu2lovsub_dev(d);
 	next = cl2lu_dev(lsd->acid_next);
-	lsd->acid_super = NULL;
 	lsd->acid_next = NULL;
 	return next;
 }
diff --git a/drivers/staging/lustre/lustre/lov/lovsub_io.c b/drivers/staging/lustre/lustre/lov/lovsub_io.c
deleted file mode 100644
index 6a98202..0000000
--- a/drivers/staging/lustre/lustre/lov/lovsub_io.c
+++ /dev/null
@@ -1,51 +0,0 @@
-/*
- * GPL HEADER START
- *
- * DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 only,
- * as published by the Free Software Foundation.
- *
- * This program is distributed in the hope that it will be useful, but
- * WITHOUT ANY WARRANTY; without even the implied warranty of
- * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
- * General Public License version 2 for more details (a copy is included
- * in the LICENSE file that accompanied this code).
- *
- * You should have received a copy of the GNU General Public License
- * version 2 along with this program; If not, see
- * http://www.gnu.org/licenses/gpl-2.0.html
- *
- * GPL HEADER END
- */
-/*
- * Copyright (c) 2002, 2010, Oracle and/or its affiliates. All rights reserved.
- * Use is subject to license terms.
- */
-/*
- * This file is part of Lustre, http://www.lustre.org/
- * Lustre is a trademark of Sun Microsystems, Inc.
- *
- * Implementation of cl_io for LOVSUB layer.
- *
- *   Author: Nikita Danilov <nikita.danilov at sun.com>
- */
-
-#define DEBUG_SUBSYSTEM S_LOV
-
-#include "lov_cl_internal.h"
-
-/** \addtogroup lov
- *  @{
- */
-
-/*****************************************************************************
- *
- * Lovsub io operations.
- *
- */
-
-/* All trivial */
-
-/** @} lov */
diff --git a/drivers/staging/lustre/lustre/lov/lovsub_lock.c b/drivers/staging/lustre/lustre/lov/lovsub_lock.c
index 38f9b73..d29f0bb 100644
--- a/drivers/staging/lustre/lustre/lov/lovsub_lock.c
+++ b/drivers/staging/lustre/lustre/lov/lovsub_lock.c
@@ -54,7 +54,6 @@ static void lovsub_lock_fini(const struct lu_env *env,
 	struct lovsub_lock   *lsl;
 
 	lsl = cl2lovsub_lock(slice);
-	LASSERT(list_empty(&lsl->lss_parents));
 	kmem_cache_free(lovsub_lock_kmem, lsl);
 }
 
@@ -70,7 +69,6 @@ int lovsub_lock_init(const struct lu_env *env, struct cl_object *obj,
 
 	lsk = kmem_cache_zalloc(lovsub_lock_kmem, GFP_NOFS);
 	if (lsk) {
-		INIT_LIST_HEAD(&lsk->lss_parents);
 		cl_lock_slice_add(lock, &lsk->lss_cl, obj, &lovsub_lock_ops);
 		result = 0;
 	} else {
-- 
1.8.3.1

From jsimmons at infradead.org  Wed Jul 26 15:22:23 2017
From: jsimmons at infradead.org (James Simmons)
Date: Wed, 26 Jul 2017 11:22:23 -0400
Subject: [lustre-devel] [PATCH 07/20] staging: lustre: lov: Ensure correct
	operation for large object sizes
In-Reply-To: <1501082556-17962-1-git-send-email-jsimmons@infradead.org>
References: <1501082556-17962-1-git-send-email-jsimmons@infradead.org>
Message-ID: <1501082556-17962-8-git-send-email-jsimmons@infradead.org>

From: Nathaniel Clark <nathaniel.l.clark at intel.com>

If a backing filesystem (ZFS) returns that it supports very large
(LLONG_MAX) object sizes, that should be correctly supported.  This
fixes the check for unitialized stripe_maxbytes in
lsm_unpackmd_common(), so that ZFS can return LLONG_MAX and it will be
okay. This issue is excersized by writing to or past the 2TB boundry
of a singly stripped file.

Signed-off-by: Nathaniel Clark <nathaniel.l.clark at intel.com>
Intel-bug-id: https://jira.hpdd.intel.com/browse/LU-7890
Reviewed-on: http://review.whamcloud.com/19066
Reviewed-by: Andreas Dilger <andreas.dilger at intel.com>
Reviewed-by: Jinshan Xiong <jinshan.xiong at intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin at intel.com>
Signed-off-by: James Simmons <jsimmons at infradead.org>
---
 drivers/staging/lustre/lustre/lov/lov_ea.c | 23 +++++++++++++----------
 1 file changed, 13 insertions(+), 10 deletions(-)

diff --git a/drivers/staging/lustre/lustre/lov/lov_ea.c b/drivers/staging/lustre/lustre/lov/lov_ea.c
index ac0bf64..2a8fee8 100644
--- a/drivers/staging/lustre/lustre/lov/lov_ea.c
+++ b/drivers/staging/lustre/lustre/lov/lov_ea.c
@@ -150,9 +150,10 @@ static int lsm_unpackmd_common(struct lov_obd *lov,
 			       struct lov_mds_md *lmm,
 			       struct lov_ost_data_v1 *objects)
 {
-	loff_t stripe_maxbytes = LLONG_MAX;
+	loff_t min_stripe_maxbytes = 0;
 	unsigned int stripe_count;
 	struct lov_oinfo *loi;
+	loff_t lov_bytes;
 	unsigned int i;
 
 	/*
@@ -168,8 +169,6 @@ static int lsm_unpackmd_common(struct lov_obd *lov,
 	stripe_count = lsm_is_released(lsm) ? 0 : lsm->lsm_stripe_count;
 
 	for (i = 0; i < stripe_count; i++) {
-		loff_t tgt_bytes;
-
 		loi = lsm->lsm_oinfo[i];
 		ostid_le_to_cpu(&objects[i].l_ost_oi, &loi->loi_oi);
 		loi->loi_ost_idx = le32_to_cpu(objects[i].l_ost_idx);
@@ -194,17 +193,21 @@ static int lsm_unpackmd_common(struct lov_obd *lov,
 			continue;
 		}
 
-		tgt_bytes = lov_tgt_maxbytes(lov->lov_tgts[loi->loi_ost_idx]);
-		stripe_maxbytes = min_t(loff_t, stripe_maxbytes, tgt_bytes);
+		lov_bytes = lov_tgt_maxbytes(lov->lov_tgts[loi->loi_ost_idx]);
+		if (min_stripe_maxbytes == 0 || lov_bytes < min_stripe_maxbytes)
+			min_stripe_maxbytes = lov_bytes;
 	}
 
-	if (stripe_maxbytes == LLONG_MAX)
-		stripe_maxbytes = LUSTRE_EXT3_STRIPE_MAXBYTES;
+	if (min_stripe_maxbytes == 0)
+		min_stripe_maxbytes = LUSTRE_EXT3_STRIPE_MAXBYTES;
+
+	stripe_count = lsm->lsm_stripe_count ?: lov->desc.ld_tgt_count;
+	lov_bytes = min_stripe_maxbytes * stripe_count;
 
-	if (!lsm->lsm_stripe_count)
-		lsm->lsm_maxbytes = stripe_maxbytes * lov->desc.ld_tgt_count;
+	if (lov_bytes < min_stripe_maxbytes) /* handle overflow */
+		lsm->lsm_maxbytes = MAX_LFS_FILESIZE;
 	else
-		lsm->lsm_maxbytes = stripe_maxbytes * lsm->lsm_stripe_count;
+		lsm->lsm_maxbytes = lov_bytes;
 
 	return 0;
 }
-- 
1.8.3.1

From jsimmons at infradead.org  Wed Jul 26 15:22:32 2017
From: jsimmons at infradead.org (James Simmons)
Date: Wed, 26 Jul 2017 11:22:32 -0400
Subject: [lustre-devel] [PATCH 16/20] staging: lustre: libcfs: fix test for
	libcfs_ioctl_hdr minimum size
In-Reply-To: <1501082556-17962-1-git-send-email-jsimmons@infradead.org>
References: <1501082556-17962-1-git-send-email-jsimmons@infradead.org>
Message-ID: <1501082556-17962-17-git-send-email-jsimmons@infradead.org>

The function libcfs_ioctl_getdata() test to see if libcfs_ioctl_hdr
is smaller than struct libcfs_ioctl_data in size. This is wrong
and it breaks the ioctl that is used to collect LNet stats. The
correct size to compare against is struct libcfs_ioctl_hdr.

Signed-off-by: James Simmons <uja.ornl at gmail.com>
Intel-bug-id: https://jira.hpdd.intel.com/browse/LU-5935
Reviewed-on: http://review.whamcloud.com/12782
Fixes: ed2f549dc0f6 ("staging: lustre: libcfs: test if userland data is to small")
Reported-by: Doug Oucharek <doug.s.oucharek at intel.com>
Reviewed-by: Liang Zhen <liang.zhen at intel.com>
Reviewed-by: Doug Oucharek <doug.s.oucharek at intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin at intel.com>
Signed-off-by: James Simmons <jsimmons at infradead.org>
---
 drivers/staging/lustre/lnet/libcfs/linux/linux-module.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/staging/lustre/lnet/libcfs/linux/linux-module.c b/drivers/staging/lustre/lnet/libcfs/linux/linux-module.c
index 075826b..c6837bc 100644
--- a/drivers/staging/lustre/lnet/libcfs/linux/linux-module.c
+++ b/drivers/staging/lustre/lnet/libcfs/linux/linux-module.c
@@ -134,7 +134,7 @@ int libcfs_ioctl_getdata(struct libcfs_ioctl_hdr **hdr_pp,
 		return -EINVAL;
 	}
 
-	if (hdr.ioc_len < sizeof(struct libcfs_ioctl_data)) {
+	if (hdr.ioc_len < sizeof(hdr)) {
 		CERROR("libcfs ioctl: user buffer too small for ioctl\n");
 		return -EINVAL;
 	}
-- 
1.8.3.1

From jsimmons at infradead.org  Wed Jul 26 15:22:33 2017
From: jsimmons at infradead.org (James Simmons)
Date: Wed, 26 Jul 2017 11:22:33 -0400
Subject: [lustre-devel] [PATCH 17/20] staging: lustre: llite: allow cached
	acls
In-Reply-To: <1501082556-17962-1-git-send-email-jsimmons@infradead.org>
References: <1501082556-17962-1-git-send-email-jsimmons@infradead.org>
Message-ID: <1501082556-17962-18-git-send-email-jsimmons@infradead.org>

From: Dmitry Eremin <dmitry.eremin at intel.com>

Move the freeing of all cached acls from ll_get_acl() to the
function ll_clear_inode(). This way we free all cached acls
for the inode just before clearing it. This allow us to take
advantage of cached acls and correctly free them before free.

Signed-off-by: Dmitry Eremin <dmitry.eremin at intel.com>
Reviewed-on: https://review.whamcloud.com/25965
Intel-bug-id: https://jira.hpdd.intel.com/browse/LU-9183
Reviewed-by: Bob Glossman <bob.glossman at intel.com>
Reviewed-by: James Simmons <uja.ornl at yahoo.com>
Reviewed-by: Oleg Drokin <oleg.drokin at intel.com>
Signed-off-by: James Simmons <jsimmons at infradead.org>
---
 drivers/staging/lustre/lustre/llite/file.c      | 3 ---
 drivers/staging/lustre/lustre/llite/llite_lib.c | 1 +
 2 files changed, 1 insertion(+), 3 deletions(-)

diff --git a/drivers/staging/lustre/lustre/llite/file.c b/drivers/staging/lustre/lustre/llite/file.c
index ab1c85c..215479a 100644
--- a/drivers/staging/lustre/lustre/llite/file.c
+++ b/drivers/staging/lustre/lustre/llite/file.c
@@ -3035,9 +3035,6 @@ struct posix_acl *ll_get_acl(struct inode *inode, int type)
 	spin_lock(&lli->lli_lock);
 	/* VFS' acl_permission_check->check_acl will release the refcount */
 	acl = posix_acl_dup(lli->lli_posix_acl);
-#ifdef CONFIG_FS_POSIX_ACL
-	forget_cached_acl(inode, type);
-#endif
 	spin_unlock(&lli->lli_lock);
 
 	return acl;
diff --git a/drivers/staging/lustre/lustre/llite/llite_lib.c b/drivers/staging/lustre/lustre/llite/llite_lib.c
index ba7d3b9..5c8405c 100644
--- a/drivers/staging/lustre/lustre/llite/llite_lib.c
+++ b/drivers/staging/lustre/lustre/llite/llite_lib.c
@@ -1316,6 +1316,7 @@ void ll_clear_inode(struct inode *inode)
 	ll_xattr_cache_destroy(inode);
 
 #ifdef CONFIG_FS_POSIX_ACL
+	forget_all_cached_acls(inode);
 	if (lli->lli_posix_acl) {
 		posix_acl_release(lli->lli_posix_acl);
 		lli->lli_posix_acl = NULL;
-- 
1.8.3.1

From jsimmons at infradead.org  Wed Jul 26 15:37:59 2017
From: jsimmons at infradead.org (James Simmons)
Date: Wed, 26 Jul 2017 11:37:59 -0400
Subject: [lustre-devel] [PATCH 2/2] staging: lustre: lov: refactor
	lov_object_fiemap()
In-Reply-To: <1501083479-23069-1-git-send-email-jsimmons@infradead.org>
References: <1501083479-23069-1-git-send-email-jsimmons@infradead.org>
Message-ID: <1501083479-23069-3-git-send-email-jsimmons@infradead.org>

From: Bobi Jam <bobijam.xu at intel.com>

Add fiemap_for_stripe() to get file map extent from each stripe
device.

Signed-off-by: Bobi Jam <bobijam.xu at intel.com>
Reported-by: Dan Carpenter <dan.carpenter at oracle.com>
Intel-bug-id: https://jira.hpdd.intel.com/browse/LU-8773
Reviewed-on: https://review.whamcloud.com/23461
Reviewed-by: James Simmons <uja.ornl at yahoo.com>
Reviewed-by: Andreas Dilger <andreas.dilger at intel.com>
Reviewed-by: Jinshan Xiong <jinshan.xiong at intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin at intel.com>
Signed-off-by: James Simmons <jsimmons at infradead.org>
---
 drivers/staging/lustre/lustre/lov/lov_object.c | 441 +++++++++++++------------
 1 file changed, 225 insertions(+), 216 deletions(-)

diff --git a/drivers/staging/lustre/lustre/lov/lov_object.c b/drivers/staging/lustre/lustre/lov/lov_object.c
index b33ed2a..8fc0bcc 100644
--- a/drivers/staging/lustre/lustre/lov/lov_object.c
+++ b/drivers/staging/lustre/lustre/lov/lov_object.c
@@ -1126,6 +1126,190 @@ static u64 fiemap_calc_fm_end_offset(struct fiemap *fiemap,
 	return fm_end_offset;
 }
 
+struct fiemap_state {
+	struct fiemap	*fs_fm;
+	u64		fs_start;
+	u64		fs_length;
+	u64		fs_end;
+	u64		fs_end_offset;
+	int		fs_cur_extent;
+	int		fs_cnt_need;
+	int		fs_start_stripe;
+	int		fs_last_stripe;
+	bool		fs_device_done;
+	bool		fs_finish;
+	bool		fs_enough;
+};
+
+int fiemap_for_stripe(const struct lu_env *env, struct cl_object *obj,
+		      struct lov_stripe_md *lsm,
+		      struct fiemap *fiemap, size_t *buflen,
+		      struct ll_fiemap_info_key *fmkey, int stripeno,
+		      struct fiemap_state *fs)
+{
+	struct cl_object *subobj;
+	struct lov_obd *lov = lu2lov_dev(obj->co_lu.lo_dev)->ld_lov;
+	struct fiemap_extent *fm_ext = &fs->fs_fm->fm_extents[0];
+	u64 req_fm_len; /* Stores length of required mapping */
+	u64 len_mapped_single_call;
+	u64 lun_start;
+	u64 lun_end;
+	u64 obd_object_end;
+	unsigned int ext_count;
+	/* EOF for object */
+	bool ost_eof = false;
+	/* done with required mapping for this OST? */
+	bool ost_done = false;
+	int ost_index;
+	int rc = 0;
+
+	fs->fs_device_done = false;
+	/* Find out range of mapping on this stripe */
+	if ((lov_stripe_intersects(lsm, stripeno, fs->fs_start, fs->fs_end,
+				   &lun_start, &obd_object_end)) == 0)
+		return 0;
+
+	if (lov_oinfo_is_dummy(lsm->lsm_oinfo[stripeno]))
+		return -EIO;
+
+	/* If this is a continuation FIEMAP call and we are on
+	 * starting stripe then lun_start needs to be set to
+	 * end_offset */
+	if (fs->fs_end_offset != 0 && stripeno == fs->fs_start_stripe)
+		lun_start = fs->fs_end_offset;
+
+	lun_end = fs->fs_length;
+	if (lun_end != ~0ULL) {
+		/* Handle fs->fs_start + fs->fs_length overflow */
+		if (fs->fs_start + fs->fs_length < fs->fs_start)
+			fs->fs_length = ~0ULL - fs->fs_start;
+		lun_end = lov_size_to_stripe(lsm, fs->fs_start + fs->fs_length,
+					     stripeno);
+	}
+
+	if (lun_start == lun_end)
+		return 0;
+
+	req_fm_len = obd_object_end - lun_start;
+	fs->fs_fm->fm_length = 0;
+	len_mapped_single_call = 0;
+
+	/* find lobsub object */
+	subobj = lov_find_subobj(env, cl2lov(obj), lsm, stripeno);
+	if (IS_ERR(subobj))
+		return PTR_ERR(subobj);
+	/* If the output buffer is very large and the objects have many
+	 * extents we may need to loop on a single OST repeatedly */
+	do {
+		if (fiemap->fm_extent_count > 0) {
+			/* Don't get too many extents. */
+			if (fs->fs_cur_extent + fs->fs_cnt_need >
+			    fiemap->fm_extent_count)
+				fs->fs_cnt_need = fiemap->fm_extent_count -
+						  fs->fs_cur_extent;
+		}
+
+		lun_start += len_mapped_single_call;
+		fs->fs_fm->fm_length = req_fm_len - len_mapped_single_call;
+		req_fm_len = fs->fs_fm->fm_length;
+		fs->fs_fm->fm_extent_count = fs->fs_enough ?
+					     1 : fs->fs_cnt_need;
+		fs->fs_fm->fm_mapped_extents = 0;
+		fs->fs_fm->fm_flags = fiemap->fm_flags;
+
+		ost_index = lsm->lsm_oinfo[stripeno]->loi_ost_idx;
+
+		if (ost_index < 0 || ost_index >= lov->desc.ld_tgt_count) {
+			rc = -EINVAL;
+			goto obj_put;
+		}
+		/* If OST is inactive, return extent with UNKNOWN flag. */
+		if (!lov->lov_tgts[ost_index]->ltd_active) {
+			fs->fs_fm->fm_flags |= FIEMAP_EXTENT_LAST;
+			fs->fs_fm->fm_mapped_extents = 1;
+
+			fm_ext[0].fe_logical = lun_start;
+			fm_ext[0].fe_length = obd_object_end - lun_start;
+			fm_ext[0].fe_flags |= FIEMAP_EXTENT_UNKNOWN;
+
+			goto inactive_tgt;
+		}
+
+		fs->fs_fm->fm_start = lun_start;
+		fs->fs_fm->fm_flags &= ~FIEMAP_FLAG_DEVICE_ORDER;
+		memcpy(&fmkey->lfik_fiemap, fs->fs_fm, sizeof(*fs->fs_fm));
+		*buflen = fiemap_count_to_size(fs->fs_fm->fm_extent_count);
+
+		rc = cl_object_fiemap(env, subobj, fmkey, fs->fs_fm, buflen);
+		if (rc)
+			goto obj_put;
+inactive_tgt:
+		ext_count = fs->fs_fm->fm_mapped_extents;
+		if (ext_count == 0) {
+			ost_done = true;
+			fs->fs_device_done = true;
+			/* If last stripe has hold at the end,
+			 * we need to return */
+			if (stripeno == fs->fs_last_stripe) {
+				fiemap->fm_mapped_extents = 0;
+				fs->fs_finish = true;
+				goto obj_put;
+			}
+			break;
+		} else if (fs->fs_enough) {
+			/*
+			 * We've collected enough extents and there are
+			 * more extents after it.
+			 */
+			fs->fs_finish = true;
+			goto obj_put;
+		}
+
+		/* If we just need num of extents, got to next device */
+		if (fiemap->fm_extent_count == 0) {
+			fs->fs_cur_extent += ext_count;
+			break;
+		}
+
+		/* prepare to copy retrived map extents */
+		len_mapped_single_call = fm_ext[ext_count - 1].fe_logical +
+					 fm_ext[ext_count - 1].fe_length -
+					 lun_start;
+
+		/* Have we finished mapping on this device? */
+		if (req_fm_len <= len_mapped_single_call) {
+			ost_done = true;
+			fs->fs_device_done = true;
+		}
+
+		/* Clear the EXTENT_LAST flag which can be present on
+		 * the last extent */
+		if (fm_ext[ext_count - 1].fe_flags & FIEMAP_EXTENT_LAST)
+			fm_ext[ext_count - 1].fe_flags &= ~FIEMAP_EXTENT_LAST;
+		if (lov_stripe_size(lsm, fm_ext[ext_count - 1].fe_logical +
+					 fm_ext[ext_count - 1].fe_length,
+				    stripeno) >= fmkey->lfik_oa.o_size) {
+			ost_eof = true;
+			fs->fs_device_done = true;
+		}
+
+		fiemap_prepare_and_copy_exts(fiemap, fm_ext, ost_index,
+					     ext_count, fs->fs_cur_extent);
+		fs->fs_cur_extent += ext_count;
+
+		/* Ran out of available extents? */
+		if (fs->fs_cur_extent >= fiemap->fm_extent_count)
+			fs->fs_enough = true;
+	} while (!ost_done && !ost_eof);
+
+	if (stripeno == fs->fs_last_stripe)
+		fs->fs_finish = true;
+obj_put:
+	cl_object_put(env, subobj);
+
+	return rc;
+}
+
 /**
  * Break down the FIEMAP request and send appropriate calls to individual OSTs.
  * This also handles the restarting of FIEMAP calls in case mapping overflows
@@ -1144,31 +1328,13 @@ static int lov_object_fiemap(const struct lu_env *env, struct cl_object *obj,
 			     struct ll_fiemap_info_key *fmkey,
 			     struct fiemap *fiemap, size_t *buflen)
 {
-	struct lov_obd *lov = lu2lov_dev(obj->co_lu.lo_dev)->ld_lov;
 	unsigned int buffer_size = FIEMAP_BUFFER_SIZE;
-	struct fiemap_extent *lcl_fm_ext;
-	struct cl_object *subobj = NULL;
 	struct fiemap *fm_local = NULL;
 	struct lov_stripe_md *lsm;
-	u64 fm_start;
-	u64 fm_end;
-	u64 fm_length;
-	u64 fm_end_offset;
-	int count_local;
-	int ost_index = 0;
-	int start_stripe;
-	int current_extent = 0;
 	int rc = 0;
-	int last_stripe;
-	int cur_stripe = 0;
-	int cur_stripe_wrap = 0;
+	int cur_stripe;
 	int stripe_count;
-	/* Whether have we collected enough extents */
-	bool enough = false;
-	/* EOF for object */
-	bool ost_eof = false;
-	/* done with required mapping for this OST? */
-	bool ost_done = false;
+	struct fiemap_state fs = { 0 };
 
 	lsm = lov_lsm_addref(cl2lov(obj));
 	if (!lsm)
@@ -1215,28 +1381,37 @@ static int lov_object_fiemap(const struct lu_env *env, struct cl_object *obj,
 		rc = -ENOMEM;
 		goto out;
 	}
-	lcl_fm_ext = &fm_local->fm_extents[0];
-	count_local = fiemap_size_to_count(buffer_size);
+	fs.fs_fm = fm_local;
+	fs.fs_cnt_need = fiemap_size_to_count(buffer_size);
 
-	fm_start = fiemap->fm_start;
-	fm_length = fiemap->fm_length;
+	fs.fs_start = fiemap->fm_start;
+	/* fs_start is beyond the end of the file */
+	if (fs.fs_start > fmkey->lfik_oa.o_size) {
+		rc = -EINVAL;
+		goto out;
+	}
 	/* Calculate start stripe, last stripe and length of mapping */
-	start_stripe = lov_stripe_number(lsm, fm_start);
-	fm_end = (fm_length == ~0ULL) ? fmkey->lfik_oa.o_size :
-					fm_start + fm_length - 1;
-	/* If fm_length != ~0ULL but fm_start_fm_length-1 exceeds file size */
-	if (fm_end > fmkey->lfik_oa.o_size)
-		fm_end = fmkey->lfik_oa.o_size;
-
-	last_stripe = fiemap_calc_last_stripe(lsm, fm_start, fm_end,
-					      start_stripe, &stripe_count);
-	fm_end_offset = fiemap_calc_fm_end_offset(fiemap, lsm, fm_start, fm_end,
-						  &start_stripe);
-	if (fm_end_offset == -EINVAL) {
+	fs.fs_start_stripe = lov_stripe_number(lsm, fs.fs_start);
+	fs.fs_end = (fs.fs_length == ~0ULL) ? fmkey->lfik_oa.o_size :
+					      fs.fs_start + fs.fs_length - 1;
+	/* If fs_length != ~0ULL but fs_start+fs_length-1 exceeds file size */
+	if (fs.fs_end > fmkey->lfik_oa.o_size) {
+		fs.fs_end = fmkey->lfik_oa.o_size;
+		fs.fs_length = fs.fs_end - fs.fs_start;
+	}
+
+	fs.fs_last_stripe = fiemap_calc_last_stripe(lsm, fs.fs_start, fs.fs_end,
+						    fs.fs_start_stripe,
+						    &stripe_count);
+	fs.fs_end_offset = fiemap_calc_fm_end_offset(fiemap, lsm, fs.fs_start,
+						     fs.fs_end,
+						     &fs.fs_start_stripe);
+	if (fs.fs_end_offset == -EINVAL) {
 		rc = -EINVAL;
 		goto out;
 	}
 
+
 	/**
 	 * Requested extent count exceeds the fiemap buffer size, shrink our
 	 * ambition.
@@ -1244,186 +1419,23 @@ static int lov_object_fiemap(const struct lu_env *env, struct cl_object *obj,
 	if (fiemap_count_to_size(fiemap->fm_extent_count) > *buflen)
 		fiemap->fm_extent_count = fiemap_size_to_count(*buflen);
 	if (!fiemap->fm_extent_count)
-		count_local = 0;
+		fs.fs_cnt_need = 0;
+
+	fs.fs_finish = false;
+	fs.fs_enough = false;
+	fs.fs_cur_extent = 0;
 
 	/* Check each stripe */
-	for (cur_stripe = start_stripe; stripe_count > 0;
+	for (cur_stripe = fs.fs_start_stripe; stripe_count > 0;
 	     --stripe_count,
 	     cur_stripe = (cur_stripe + 1) % lsm->lsm_stripe_count) {
-		u64 req_fm_len; /* Stores length of required mapping */
-		u64 len_mapped_single_call;
-		u64 lun_start;
-		u64 lun_end;
-		u64 obd_object_end;
-		unsigned int ext_count;
-
-		cur_stripe_wrap = cur_stripe;
-
-		/* Find out range of mapping on this stripe */
-		if (!(lov_stripe_intersects(lsm, cur_stripe, fm_start, fm_end,
-					    &lun_start, &obd_object_end)))
-			continue;
-
-		if (lov_oinfo_is_dummy(lsm->lsm_oinfo[cur_stripe])) {
-			rc = -EIO;
-			goto out;
-		}
-
-		/*
-		 * If this is a continuation FIEMAP call and we are on
-		 * starting stripe then lun_start needs to be set to
-		 * fm_end_offset
-		 */
-		if (fm_end_offset && cur_stripe == start_stripe)
-			lun_start = fm_end_offset;
-
-		if (fm_length != ~0ULL) {
-			/* Handle fm_start + fm_length overflow */
-			if (fm_start + fm_length < fm_start)
-				fm_length = ~0ULL - fm_start;
-			lun_end = lov_size_to_stripe(lsm, fm_start + fm_length,
-						     cur_stripe);
-		} else {
-			lun_end = ~0ULL;
-		}
-
-		if (lun_start == lun_end)
-			continue;
-
-		req_fm_len = obd_object_end - lun_start;
-		fm_local->fm_length = 0;
-		len_mapped_single_call = 0;
-
-		/* find lobsub object */
-		subobj = lov_find_subobj(env, cl2lov(obj), lsm,
-					 cur_stripe);
-		if (IS_ERR(subobj)) {
-			rc = PTR_ERR(subobj);
+		rc = fiemap_for_stripe(env, obj, lsm, fiemap, buflen, fmkey,
+				       cur_stripe, &fs);
+		if (rc < 0)
 			goto out;
-		}
-		/*
-		 * If the output buffer is very large and the objects have many
-		 * extents we may need to loop on a single OST repeatedly
-		 */
-		ost_eof = false;
-		ost_done = false;
-		do {
-			if (fiemap->fm_extent_count > 0) {
-				/* Don't get too many extents. */
-				if (current_extent + count_local >
-				    fiemap->fm_extent_count)
-					count_local = fiemap->fm_extent_count -
-						      current_extent;
-			}
-
-			lun_start += len_mapped_single_call;
-			fm_local->fm_length = req_fm_len -
-					      len_mapped_single_call;
-			req_fm_len = fm_local->fm_length;
-			fm_local->fm_extent_count = enough ? 1 : count_local;
-			fm_local->fm_mapped_extents = 0;
-			fm_local->fm_flags = fiemap->fm_flags;
-
-			ost_index = lsm->lsm_oinfo[cur_stripe]->loi_ost_idx;
-
-			if (ost_index < 0 ||
-			    ost_index >= lov->desc.ld_tgt_count) {
-				rc = -EINVAL;
-				goto obj_put;
-			}
-			/*
-			 * If OST is inactive, return extent with UNKNOWN
-			 * flag.
-			 */
-			if (!lov->lov_tgts[ost_index]->ltd_active) {
-				fm_local->fm_flags |= FIEMAP_EXTENT_LAST;
-				fm_local->fm_mapped_extents = 1;
-
-				lcl_fm_ext[0].fe_logical = lun_start;
-				lcl_fm_ext[0].fe_length = obd_object_end -
-							  lun_start;
-				lcl_fm_ext[0].fe_flags |= FIEMAP_EXTENT_UNKNOWN;
-
-				goto inactive_tgt;
-			}
-
-			fm_local->fm_start = lun_start;
-			fm_local->fm_flags &= ~FIEMAP_FLAG_DEVICE_ORDER;
-			memcpy(&fmkey->lfik_fiemap, fm_local, sizeof(*fm_local));
-			*buflen = fiemap_count_to_size(fm_local->fm_extent_count);
-
-			rc = cl_object_fiemap(env, subobj, fmkey, fm_local,
-					      buflen);
-			if (rc)
-				goto obj_put;
-inactive_tgt:
-			ext_count = fm_local->fm_mapped_extents;
-			if (!ext_count) {
-				ost_done = true;
-				/*
-				 * If last stripe has hold at the end,
-				 * we need to return
-				 */
-				if (cur_stripe_wrap == last_stripe) {
-					fiemap->fm_mapped_extents = 0;
-					goto finish;
-				}
-				break;
-			} else if (enough) {
-				/*
-				 * We've collected enough extents and there are
-				 * more extents after it.
-				 */
-				goto finish;
-			}
-
-			/* If we just need num of extents, got to next device */
-			if (!fiemap->fm_extent_count) {
-				current_extent += ext_count;
-				break;
-			}
-
-			/* prepare to copy retrived map extents */
-			len_mapped_single_call =
-				lcl_fm_ext[ext_count - 1].fe_logical -
-				lun_start + lcl_fm_ext[ext_count - 1].fe_length;
-
-			/* Have we finished mapping on this device? */
-			if (req_fm_len <= len_mapped_single_call)
-				ost_done = true;
-
-			/*
-			 * Clear the EXTENT_LAST flag which can be present on
-			 * the last extent
-			 */
-			if (lcl_fm_ext[ext_count - 1].fe_flags &
-			    FIEMAP_EXTENT_LAST)
-				lcl_fm_ext[ext_count - 1].fe_flags &=
-					~FIEMAP_EXTENT_LAST;
-
-			if (lov_stripe_size(lsm,
-					    lcl_fm_ext[ext_count - 1].fe_logical +
-					    lcl_fm_ext[ext_count - 1].fe_length,
-					    cur_stripe) >= fmkey->lfik_oa.o_size)
-				ost_eof = true;
-
-			fiemap_prepare_and_copy_exts(fiemap, lcl_fm_ext,
-						     ost_index, ext_count,
-						     current_extent);
-			current_extent += ext_count;
-
-			/* Ran out of available extents? */
-			if (current_extent >= fiemap->fm_extent_count)
-				enough = true;
-		} while (!ost_done && !ost_eof);
-
-		cl_object_put(env, subobj);
-		subobj = NULL;
-
-		if (cur_stripe_wrap == last_stripe)
-			goto finish;
+		if (fs.fs_finish)
+			break;
 	} /* for each stripe */
-finish:
 	/*
 	 * Indicate that we are returning device offsets unless file just has
 	 * single stripe
@@ -1438,14 +1450,11 @@ static int lov_object_fiemap(const struct lu_env *env, struct cl_object *obj,
 	 * Check if we have reached the last stripe and whether mapping for that
 	 * stripe is done.
 	 */
-	if ((cur_stripe_wrap == last_stripe) && (ost_done || ost_eof))
-		fiemap->fm_extents[current_extent - 1].fe_flags |=
+	if ((cur_stripe == fs.fs_last_stripe) && fs.fs_device_done)
+		fiemap->fm_extents[fs.fs_cur_extent - 1].fe_flags |=
 							FIEMAP_EXTENT_LAST;
 skip_last_device_calc:
-	fiemap->fm_mapped_extents = current_extent;
-obj_put:
-	if (subobj)
-		cl_object_put(env, subobj);
+	fiemap->fm_mapped_extents = fs.fs_cur_extent;
 out:
 	kvfree(fm_local);
 	lov_lsm_put(lsm);
-- 
1.8.3.1

From jsimmons at infradead.org  Wed Jul 26 15:37:58 2017
From: jsimmons at infradead.org (James Simmons)
Date: Wed, 26 Jul 2017 11:37:58 -0400
Subject: [lustre-devel] [PATCH 1/2] staging: lustre: lov: use u64 instead of
	loff_t in lov_object_fiemap()
In-Reply-To: <1501083479-23069-1-git-send-email-jsimmons@infradead.org>
References: <1501083479-23069-1-git-send-email-jsimmons@infradead.org>
Message-ID: <1501083479-23069-2-git-send-email-jsimmons@infradead.org>

From: Bobi Jam <bobijam.xu at intel.com>

Change loff_t to u64 in lov_object_fiemap() since loff_t is a signed
value type.

Otherwise there could be an overflow in
drivers/staging/lustre/lustre/lov/lov_object.c:1241 lov_object_fiemap()
warn: signed overflow undefined. 'fm_start + fm_length < fm_start'

Signed-off-by: Bobi Jam <bobijam.xu at intel.com>
Reported-by: Dan Carpenter <dan.carpenter at oracle.com>
Intel-bug-id: https://jira.hpdd.intel.com/browse/LU-8773
Reviewed-on: https://review.whamcloud.com/23461
Reviewed-by: James Simmons <uja.ornl at yahoo.com>
Reviewed-by: Andreas Dilger <andreas.dilger at intel.com>
Reviewed-by: Jinshan Xiong <jinshan.xiong at intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin at intel.com>
Signed-off-by: James Simmons <jsimmons at infradead.org>
---
 drivers/staging/lustre/lustre/lov/lov_object.c | 38 +++++++++++++-------------
 1 file changed, 19 insertions(+), 19 deletions(-)

diff --git a/drivers/staging/lustre/lustre/lov/lov_object.c b/drivers/staging/lustre/lustre/lov/lov_object.c
index 4913229..b33ed2a 100644
--- a/drivers/staging/lustre/lustre/lov/lov_object.c
+++ b/drivers/staging/lustre/lustre/lov/lov_object.c
@@ -1003,12 +1003,12 @@ int lov_lock_init(const struct lu_env *env, struct cl_object *obj,
  * \retval last_stripe		return the last stripe of the mapping
  */
 static int fiemap_calc_last_stripe(struct lov_stripe_md *lsm,
-				   loff_t fm_start, loff_t fm_end,
+				   u64 fm_start, u64 fm_end,
 				   int start_stripe, int *stripe_count)
 {
 	int last_stripe;
-	loff_t obd_start;
-	loff_t obd_end;
+	u64 obd_start;
+	u64 obd_end;
 	int i, j;
 
 	if (fm_end - fm_start > lsm->lsm_stripe_size * lsm->lsm_stripe_count) {
@@ -1076,14 +1076,14 @@ static void fiemap_prepare_and_copy_exts(struct fiemap *fiemap,
  * \param fm_end [in]		logical end of mapping
  * \param start_stripe [out]	starting stripe will be returned in this
  */
-static loff_t fiemap_calc_fm_end_offset(struct fiemap *fiemap,
-					struct lov_stripe_md *lsm,
-					loff_t fm_start, loff_t fm_end,
-					int *start_stripe)
+static u64 fiemap_calc_fm_end_offset(struct fiemap *fiemap,
+				     struct lov_stripe_md *lsm,
+				     u64 fm_start, u64 fm_end,
+				     int *start_stripe)
 {
-	loff_t local_end = fiemap->fm_extents[0].fe_logical;
-	loff_t lun_start, lun_end;
-	loff_t fm_end_offset;
+	u64 local_end = fiemap->fm_extents[0].fe_logical;
+	u64 lun_start, lun_end;
+	u64 fm_end_offset;
 	int stripe_no = -1;
 	int i;
 
@@ -1150,10 +1150,10 @@ static int lov_object_fiemap(const struct lu_env *env, struct cl_object *obj,
 	struct cl_object *subobj = NULL;
 	struct fiemap *fm_local = NULL;
 	struct lov_stripe_md *lsm;
-	loff_t fm_start;
-	loff_t fm_end;
-	loff_t fm_length;
-	loff_t fm_end_offset;
+	u64 fm_start;
+	u64 fm_end;
+	u64 fm_length;
+	u64 fm_end_offset;
 	int count_local;
 	int ost_index = 0;
 	int start_stripe;
@@ -1250,11 +1250,11 @@ static int lov_object_fiemap(const struct lu_env *env, struct cl_object *obj,
 	for (cur_stripe = start_stripe; stripe_count > 0;
 	     --stripe_count,
 	     cur_stripe = (cur_stripe + 1) % lsm->lsm_stripe_count) {
-		loff_t req_fm_len; /* Stores length of required mapping */
-		loff_t len_mapped_single_call;
-		loff_t lun_start;
-		loff_t lun_end;
-		loff_t obd_object_end;
+		u64 req_fm_len; /* Stores length of required mapping */
+		u64 len_mapped_single_call;
+		u64 lun_start;
+		u64 lun_end;
+		u64 obd_object_end;
 		unsigned int ext_count;
 
 		cur_stripe_wrap = cur_stripe;
-- 
1.8.3.1

From jsimmons at infradead.org  Wed Jul 26 15:37:57 2017
From: jsimmons at infradead.org (James Simmons)
Date: Wed, 26 Jul 2017 11:37:57 -0400
Subject: [lustre-devel] [PATCH 0/2] staging: lustre: lov: fix sparse errors
	for fiemap
Message-ID: <1501083479-23069-1-git-send-email-jsimmons@infradead.org>

The following error was reported by Dan Carpenter <dan.carpenter at oracle.com>

lov_object_fiemap() has following static checker warning:
drivers/staging/lustre/lustre/lov/lov_object.c:1241 lov_object_fiemap()
warn: signed overflow undefined. 'fm_start + fm_length < fm_start'

This patch set resolves those issues.

Bobi Jam (2):
  staging: lustre: lov: use u64 instead of loff_t in lov_object_fiemap()
  staging: lustre: lov: refactor lov_object_fiemap()

 drivers/staging/lustre/lustre/lov/lov_object.c | 461 +++++++++++++------------
 1 file changed, 235 insertions(+), 226 deletions(-)

-- 
1.8.3.1

From paf at cray.com  Wed Jul 26 18:26:45 2017
From: paf at cray.com (Patrick Farrell)
Date: Wed, 26 Jul 2017 18:26:45 +0000
Subject: [lustre-devel] Design proposal for client-side compression
In-Reply-To: <1500992740.4728.5.camel@informatik.uni-hamburg.de>
References: <1483967258.2622.24.camel@informatik.uni-hamburg.de>
	<B5E18A82-79B3-484F-BEBE-998B8AE56397@intel.com>
	<1484223328.3937.60.camel@informatik.uni-hamburg.de>
	<B924BE6D-3C6A-4BA4-96F9-AC053F032B2D@intel.com>
	<1484749185.3965.114.camel@informatik.uni-hamburg.de>
	<1487254536.4777.51.camel@informatik.uni-hamburg.de>
	,<1500650130.3158.6.camel@informatik.uni-hamburg.de>
	<BN6PR1101MB2132311AB19B104AD044797ACBA40@BN6PR1101MB2132.namprd11.prod.outlook.com>,
	<1500992740.4728.5.camel@informatik.uni-hamburg.de>
Message-ID: <BN6PR1101MB2132DD0CCFF2F83BB3BE7F54CBB90@BN6PR1101MB2132.namprd11.prod.outlook.com>

Anna,

Having reread your LAD presentation (I was there, but it's been a while...), I think you've got a good architecture.

A few thoughts.

1. Jinshan was just suggesting including in the code a switch to enable/disable the feature at runtime, for an example, see his fast read patch:
https://review.whamcloud.com/#/c/20255/
Especially the proc section:
https://review.whamcloud.com/#/c/20255/7/lustre/llite/lproc_llite.c
The effect of that is a file in proc that one can use to disable/enable the feature by echoing 0 or 1.
(I think there is probably a place for tuning beyond that, but that's separate.)
This is great for features that may have complex impacts, and also for people who want to test a feature to see how it changes things.
2. Lustre clients iterate over the stripes, basically.

Here's an explanation of the write path on the client that should help.  This explanation is heavily simplified and incorrect in some of the details, but should be accurate enough for your question.
The I/O model on the client (for buffered I/O, direct I/O is different) is that the writing process (userspace process) starts an I/O, then identifies which parts of the I/O go to which stripes, gets the locks it needs, then copies the data through the page cache...  Once the data is copied to the page cache, Lustre then works on writing out that data.  In general, it does it asynchronously, where the userspace process returns and then data write-out is handled by the ptlrpcd (daemon) threads, but in various exceptional conditions it may do the write-out in the userspace process.

In general, the write out is going to happen in parallel (to different OSTs) with different ptlrpcd threads taking different chunks of data and putting them on the wire, and sometimes the userspace thread doing that work for some of the data as well.

So "How much memory do we need at most at the same time?" is not a question with an easy answer.  When doing a bulk RPC, generally, the sender sends an RPC announcing the bulk data is ready, then the recipient copies the data (RDMA) (or the sender sends it over to a buffer if no RDMA) and announces to the client it has done so.  I'm not 100% clear on the sequencing here, but the key thing is there's a time where we've sent the RPC but we aren't done with the buffer.  So we can send another RPC before that buffer is retired.  (If I've got this badly wrong, I hope someone will correct me.

So the total amount of memory required to do this is going to depend on how fast data is being sent, rather than on the # of OSTs or any other constant.

There *is* a per OST limit to how many RPCs a client can have in flight at once, but it's generally set so the client can get good performance to one OST.  Allocating data for max_rpcs_in_flight*num OSTs would be far too much, because in the 1000 OST case, a client can probably only have a few hundred RPCs in flight (if that...) at once on a normal network.

But if we are writing from one client to many OSTs, how many RPCs are in flight at once is going to depend more on how fast our network is (or, possibly, CPU on the client if the network is fast and/or CPU is slow) than any explicit limits.  The explicit limits are much higher than we will hit in practice.

Does that make sense?  It doesn't make your problem any easier...

It actually seems like maybe a global pool of pages *is* the right answer.  The question is how big to make it...
What about making it grow on demand up to a configurable upper limit?

The allocation code for encryption is here (it's pretty complicated and it works on the assumption that it must get pages or return ENOMEM - The compression code doesn't absolutely have to get pages, so it could be changed):
sptlrpc_enc_pool_get_pages

It seems like maybe that code could be adjusted to serve both the encryption case (must not fail, if it can't get memory, return -ENOMEM to cause retries), and the compression case (can fail, if it fails, should not do compression...  Maybe should consume less memory)

About thread counts:
Encryption is handled in the ptlrpc code, and your presentation noted the plan is to mimic that, which sounds good to me.  That means there's no reason for you to explicitly control the number of threads doing compression, the same number of threads doing sending will be doing compression, which seems fine.  (Unless there's some point of contention in the compression code, but that seems unlikely...)

Hope that helps a bit.

- Patrick



________________________________
From: Anna Fuchs <anna.fuchs at informatik.uni-hamburg.de>
Sent: Tuesday, July 25, 2017 9:25:40 AM
To: Patrick Farrell; Xiong, Jinshan
Cc: Matthew Ahrens; Zhuravlev, Alexey; lustre-devel
Subject: Re: [lustre-devel] Design proposal for client-side compression

Thank you for your responses.

Patrick,

On Fri, 2017-07-21 at 16:43 +0000, Patrick Farrell wrote:
> I think basing this on the maximum number of stripes it too simple,
> and maybe not necessary.
>
> Apologies in advance if what I say below rests on a misunderstanding
> of the compression design, I should know it better than I do.

Probably I still don't get all the relevant internals of Lustre to
clearly describe what we are planning and what we need.

> About based on maximum stripe count, there are a number of 1000 OST
> systems in the world today.  Imagine one of them with 16 MiB stripes,
> that's ~16 GiB of memory for this.  I think that's clearly too
> large.  But a global (rather than per OSC) pool could be tricky too,
> leading to contention on getting and returning pages.

Well, does that mean that one Lustre client handles all the 16GiB of
stripes at the same time or does it somehow iterate over the stripes?
How much memory do we need at most at the same time? If the client
first processes 100 stripes, we need enough memory to compress 100
stripes at the same time. So the question is not about the maximum
stripe count, but the maximum, let me call it queue portion of stripes,
which can be processed at the same time within one client.

>
> You mention later a 50 MiB pool per client.  As a per OST pre-
> allocated pool, that would likely be too large.  As a global pool, it
> seems small...
>
> But why use a global pool?  It sounds like the compression would be
> handled by the thread putting the data on the wire (Sorry if I've got
> that wrong).  So - What about a per-thread block of pages, for each
> ptlrpcd thread?  If the idea is that this compressed data is not
> retained for replay (instead, you would re-compress), then we only
> need a block of max rpc size for each thread (You could just use the
> largest RPC size supported by the client), so it can send that
> compressed data.

Yes, we don't really need a very global pool, but still we need to
know, how many threads can there be at the same time? Is there one
thread per stripe or per RPC? And how many in total?

>
> The overhead of compression for replay is probably not something we
> need to worry about.
>
> Or even per-CPU blocks of pages.  That would probably be better still
> (less total memory if there are more ptlrpcds than CPUs), if we can
> guarantee not sleeping during the time the pool is in use.  (I'm not
> sure.)
>
> Also, you mention limiting the # of threads.  Why is limiting the
> number of threads doing compression of interest?  What are you
> specifically trying to avoid with that?

I mean that the number of threads available for compression is somehow
limited. If we have 100 stripes at the same time, we still can compress
with #cores threads, which might be less than 100. So if there are more
stripes in flight than we have resources for compression (since it is
slower), we need to decide whether to slow down everything or to skip
compression of some stripes for overall better performance.


Jinshan,

>
> Is it possible to enable this by writing to a sysfs or procfs entry?
> So that users can try this out without having to recompile Lustre.

The size should be controllable dynamically, but for the feature Lustre
has to be recompiled anyway.


>
> It’s not scalable to have a pool per OSC because Lustre can support
> up to 2000 stripes. However, we don’t need to worry about wide stripe
> problem because no one can write a full stripe with even 1MB stripe
> size, because that means application has to issue 2GB size of write.

Does that mean, we have 2000 stripes and we have 2000 messages/RPCs at
the same time? And we need to be able to compress all 2000 stripes at
the same time to avoid blocking? Is there not any limit how many one
client can have at one point of time?

> Yes, it’s reasonable to have a global pool for each client node.
> Let’s start from this number but please make it adjustable via sysfs
> or procfs.

I am still not sure how large we can do it. Once we really need 16 GiB
for that pool to quickly serve the compression threads, it is not
doable and we have to think different. Once we have a smaller pool than
we need, we have to block or skip compression, which is undesirable.
But I don't know how to determine the required size.

Thanks!

Best regards,
Anna


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-devel_lists.lustre.org/attachments/20170726/80e79b5b/attachment.html>

From jinshan.xiong at intel.com  Wed Jul 26 20:17:49 2017
From: jinshan.xiong at intel.com (Xiong, Jinshan)
Date: Wed, 26 Jul 2017 20:17:49 +0000
Subject: [lustre-devel] Design proposal for client-side compression
In-Reply-To: <BN6PR1101MB2132DD0CCFF2F83BB3BE7F54CBB90@BN6PR1101MB2132.namprd11.prod.outlook.com>
References: <1483967258.2622.24.camel@informatik.uni-hamburg.de>
	<B5E18A82-79B3-484F-BEBE-998B8AE56397@intel.com>
	<1484223328.3937.60.camel@informatik.uni-hamburg.de>
	<B924BE6D-3C6A-4BA4-96F9-AC053F032B2D@intel.com>
	<1484749185.3965.114.camel@informatik.uni-hamburg.de>
	<1487254536.4777.51.camel@informatik.uni-hamburg.de>
	<1500650130.3158.6.camel@informatik.uni-hamburg.de>
	<BN6PR1101MB2132311AB19B104AD044797ACBA40@BN6PR1101MB2132.namprd11.prod.outlook.com>
	<1500992740.4728.5.camel@informatik.uni-hamburg.de>
	<BN6PR1101MB2132DD0CCFF2F83BB3BE7F54CBB90@BN6PR1101MB2132.namprd11.prod.outlook.com>
Message-ID: <80491E10-0C6C-4826-AD55-5B78A20166D8@intel.com>

Thanks for the detailed explanation from Patrick.

“Does that mean, … Is there not any limit how many one
client can have at one point of time?”

In theory, it’s possible that there exist that many active RPCs at one time, which is why I think it’s not feasible to have per-OSC page pool.

“… Once we have a smaller pool than
we need, we have to block or skip compression, which is undesirable.
But I don't know how to determine the required size.”

It’s probably not good to skip compression once it runs out of pages in pool, instead it should be blocked waiting for pages to be available. It will spend some time on waiting for the available pages, but at the end it will transfer less data over the network, and OST will also write less data to disk, so that it can still be performant.
Of course, we can make it smarter by checking if there are too many threads waiting for available pages, and in that case we decide to not compress some RPCs. But this work can be deferred to the time after we have the code running and tune it by actual workload.

In order to decide the size of the pool, we should consider the number of CPUs on the client node, and the default RPC size. Let’s start with MAX(32, number_of_cpus) * default_RPC_size, and the default RPC size is 4MB in 2.10+ releases.

Jinshan

From: lustre-devel <lustre-devel-bounces at lists.lustre.org> on behalf of Patrick Farrell <paf at cray.com>
Date: Wednesday, July 26, 2017 at 11:27 AM
To: Anna Fuchs <anna.fuchs at informatik.uni-hamburg.de>, "Xiong, Jinshan" <jinshan.xiong at intel.com>
Cc: Matthew Ahrens <mahrens at delphix.com>, "Zhuravlev, Alexey" <alexey.zhuravlev at intel.com>, lustre-devel <lustre-devel at lists.lustre.org>
Subject: Re: [lustre-devel] Design proposal for client-side compression

Anna,

Having reread your LAD presentation (I was there, but it's been a while...), I think you've got a good architecture.

A few thoughts.

1. Jinshan was just suggesting including in the code a switch to enable/disable the feature at runtime, for an example, see his fast read patch:
https://review.whamcloud.com/#/c/20255/
Especially the proc section:
https://review.whamcloud.com/#/c/20255/7/lustre/llite/lproc_llite.c
The effect of that is a file in proc that one can use to disable/enable the feature by echoing 0 or 1.
(I think there is probably a place for tuning beyond that, but that's separate.)
This is great for features that may have complex impacts, and also for people who want to test a feature to see how it changes things.
2. Lustre clients iterate over the stripes, basically.

Here's an explanation of the write path on the client that should help.  This explanation is heavily simplified and incorrect in some of the details, but should be accurate enough for your question.
The I/O model on the client (for buffered I/O, direct I/O is different) is that the writing process (userspace process) starts an I/O, then identifies which parts of the I/O go to which stripes, gets the locks it needs, then copies the data through the page cache...  Once the data is copied to the page cache, Lustre then works on writing out that data.  In general, it does it asynchronously, where the userspace process returns and then data write-out is handled by the ptlrpcd (daemon) threads, but in various exceptional conditions it may do the write-out in the userspace process.

In general, the write out is going to happen in parallel (to different OSTs) with different ptlrpcd threads taking different chunks of data and putting them on the wire, and sometimes the userspace thread doing that work for some of the data as well.

So "How much memory do we need at most at the same time?" is not a question with an easy answer.  When doing a bulk RPC, generally, the sender sends an RPC announcing the bulk data is ready, then the recipient copies the data (RDMA) (or the sender sends it over to a buffer if no RDMA) and announces to the client it has done so.  I'm not 100% clear on the sequencing here, but the key thing is there's a time where we've sent the RPC but we aren't done with the buffer.  So we can send another RPC before that buffer is retired.  (If I've got this badly wrong, I hope someone will correct me.

So the total amount of memory required to do this is going to depend on how fast data is being sent, rather than on the # of OSTs or any other constant.

There *is* a per OST limit to how many RPCs a client can have in flight at once, but it's generally set so the client can get good performance to one OST.  Allocating data for max_rpcs_in_flight*num OSTs would be far too much, because in the 1000 OST case, a client can probably only have a few hundred RPCs in flight (if that...) at once on a normal network.

But if we are writing from one client to many OSTs, how many RPCs are in flight at once is going to depend more on how fast our network is (or, possibly, CPU on the client if the network is fast and/or CPU is slow) than any explicit limits.  The explicit limits are much higher than we will hit in practice.

Does that make sense?  It doesn't make your problem any easier...

It actually seems like maybe a global pool of pages *is* the right answer.  The question is how big to make it...
What about making it grow on demand up to a configurable upper limit?

The allocation code for encryption is here (it's pretty complicated and it works on the assumption that it must get pages or return ENOMEM - The compression code doesn't absolutely have to get pages, so it could be changed):
sptlrpc_enc_pool_get_pages

It seems like maybe that code could be adjusted to serve both the encryption case (must not fail, if it can't get memory, return -ENOMEM to cause retries), and the compression case (can fail, if it fails, should not do compression...  Maybe should consume less memory)

About thread counts:
Encryption is handled in the ptlrpc code, and your presentation noted the plan is to mimic that, which sounds good to me.  That means there's no reason for you to explicitly control the number of threads doing compression, the same number of threads doing sending will be doing compression, which seems fine.  (Unless there's some point of contention in the compression code, but that seems unlikely...)

Hope that helps a bit.

- Patrick


________________________________
From: Anna Fuchs <anna.fuchs at informatik.uni-hamburg.de>
Sent: Tuesday, July 25, 2017 9:25:40 AM
To: Patrick Farrell; Xiong, Jinshan
Cc: Matthew Ahrens; Zhuravlev, Alexey; lustre-devel
Subject: Re: [lustre-devel] Design proposal for client-side compression

Thank you for your responses.

Patrick,

On Fri, 2017-07-21 at 16:43 +0000, Patrick Farrell wrote:
> I think basing this on the maximum number of stripes it too simple,
> and maybe not necessary.
>
> Apologies in advance if what I say below rests on a misunderstanding
> of the compression design, I should know it better than I do.

Probably I still don't get all the relevant internals of Lustre to
clearly describe what we are planning and what we need.

> About based on maximum stripe count, there are a number of 1000 OST
> systems in the world today.  Imagine one of them with 16 MiB stripes,
> that's ~16 GiB of memory for this.  I think that's clearly too
> large.  But a global (rather than per OSC) pool could be tricky too,
> leading to contention on getting and returning pages.

Well, does that mean that one Lustre client handles all the 16GiB of
stripes at the same time or does it somehow iterate over the stripes?
How much memory do we need at most at the same time? If the client
first processes 100 stripes, we need enough memory to compress 100
stripes at the same time. So the question is not about the maximum
stripe count, but the maximum, let me call it queue portion of stripes,
which can be processed at the same time within one client.

>
> You mention later a 50 MiB pool per client.  As a per OST pre-
> allocated pool, that would likely be too large.  As a global pool, it
> seems small...
>
> But why use a global pool?  It sounds like the compression would be
> handled by the thread putting the data on the wire (Sorry if I've got
> that wrong).  So - What about a per-thread block of pages, for each
> ptlrpcd thread?  If the idea is that this compressed data is not
> retained for replay (instead, you would re-compress), then we only
> need a block of max rpc size for each thread (You could just use the
> largest RPC size supported by the client), so it can send that
> compressed data.

Yes, we don't really need a very global pool, but still we need to
know, how many threads can there be at the same time? Is there one
thread per stripe or per RPC? And how many in total?

>
> The overhead of compression for replay is probably not something we
> need to worry about.
>
> Or even per-CPU blocks of pages.  That would probably be better still
> (less total memory if there are more ptlrpcds than CPUs), if we can
> guarantee not sleeping during the time the pool is in use.  (I'm not
> sure.)
>
> Also, you mention limiting the # of threads.  Why is limiting the
> number of threads doing compression of interest?  What are you
> specifically trying to avoid with that?

I mean that the number of threads available for compression is somehow
limited. If we have 100 stripes at the same time, we still can compress
with #cores threads, which might be less than 100. So if there are more
stripes in flight than we have resources for compression (since it is
slower), we need to decide whether to slow down everything or to skip
compression of some stripes for overall better performance.


Jinshan,

>
> Is it possible to enable this by writing to a sysfs or procfs entry?
> So that users can try this out without having to recompile Lustre.

The size should be controllable dynamically, but for the feature Lustre
has to be recompiled anyway.


>
> It’s not scalable to have a pool per OSC because Lustre can support
> up to 2000 stripes. However, we don’t need to worry about wide stripe
> problem because no one can write a full stripe with even 1MB stripe
> size, because that means application has to issue 2GB size of write.

Does that mean, we have 2000 stripes and we have 2000 messages/RPCs at
the same time? And we need to be able to compress all 2000 stripes at
the same time to avoid blocking? Is there not any limit how many one
client can have at one point of time?

> Yes, it’s reasonable to have a global pool for each client node.
> Let’s start from this number but please make it adjustable via sysfs
> or procfs.

I am still not sure how large we can do it. Once we really need 16 GiB
for that pool to quickly serve the compression threads, it is not
doable and we have to think different. Once we have a smaller pool than
we need, we have to block or skip compression, which is undesirable.
But I don't know how to determine the required size.

Thanks!

Best regards,
Anna

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-devel_lists.lustre.org/attachments/20170726/0b1021f4/attachment.html>

From anna.fuchs at informatik.uni-hamburg.de  Thu Jul 27 08:26:00 2017
From: anna.fuchs at informatik.uni-hamburg.de (Anna Fuchs)
Date: Thu, 27 Jul 2017 10:26:00 +0200
Subject: [lustre-devel] Design proposal for client-side compression
In-Reply-To: <BN6PR1101MB2132DD0CCFF2F83BB3BE7F54CBB90@BN6PR1101MB2132.namprd11.prod.outlook.com>
References: <1483967258.2622.24.camel@informatik.uni-hamburg.de>
	<B5E18A82-79B3-484F-BEBE-998B8AE56397@intel.com>
	<1484223328.3937.60.camel@informatik.uni-hamburg.de>
	<B924BE6D-3C6A-4BA4-96F9-AC053F032B2D@intel.com>
	<1484749185.3965.114.camel@informatik.uni-hamburg.de>
	<1487254536.4777.51.camel@informatik.uni-hamburg.de>
	,<1500650130.3158.6.camel@informatik.uni-hamburg.de>
	<BN6PR1101MB2132311AB19B104AD044797ACBA40@BN6PR1101MB2132.namprd11.prod.outlook.com>
	,<1500992740.4728.5.camel@informatik.uni-hamburg.de>
	<BN6PR1101MB2132DD0CCFF2F83BB3BE7F54CBB90@BN6PR1101MB2132.namprd11.prod.outlook.com>
Message-ID: <1501143960.3412.3.camel@informatik.uni-hamburg.de>

Patrick, 

> Having reread your LAD presentation (I was there, but it's been a
> while...), I think you've got a good architecture.

There have been some changes since that, but the general things should
be the same.

> A few thoughts.
> 
> 1. Jinshan was just suggesting including in the code a switch to
> enable/disable the feature at runtime, for an example, see his fast
> read patch:
> https://review.whamcloud.com/#/c/20255/
> Especially the proc section:
> https://review.whamcloud.com/#/c/20255/7/lustre/llite/lproc_llite.c
> The effect of that is a file in proc that one can use to
> disable/enable the feature by echoing 0 or 1.
> (I think there is probably a place for tuning beyond that, but that's
> separate.)
> This is great for features that may have complex impacts, and also
> for people who want to test a feature to see how it changes things.

Oh, I misunderstood Jinshan last time, sorry. Yes, it would be much
easier for users and should be possible. Thank you for references!

> 2. Lustre clients iterate over the stripes, basically.
> 
> Here's an explanation of the write path on the client that should
> help.  This explanation is heavily simplified and incorrect in some
> of the details, but should be accurate enough for your question.
> The I/O model on the client (for buffered I/O, direct I/O is
> different) is that the writing process (userspace process) starts an
> I/O, then identifies which parts of the I/O go to which stripes, gets
> the locks it needs, then copies the data through the page cache... 
> Once the data is copied to the page cache, Lustre then works on
> writing out that data.  In general, it does it asynchronously, where
> the userspace process returns and then data write-out is handled by
> the ptlrpcd (daemon) threads, but in various exceptional conditions
> it may do the write-out in the userspace process.
> 
> In general, the write out is going to happen in parallel (to
> different OSTs) with different ptlrpcd threads taking different
> chunks of data and putting them on the wire, and sometimes the
> userspace thread doing that work for some of the data as well.
> 
> So "How much memory do we need at most at the same time?" is not a
> question with an easy answer.  When doing a bulk RPC, generally, the
> sender sends an RPC announcing the bulk data is ready, then the
> recipient copies the data (RDMA) (or the sender sends it over to a
> buffer if no RDMA) and announces to the client it has done so.  I'm
> not 100% clear on the sequencing here, but the key thing is there's a
> time where we've sent the RPC but we aren't done with the buffer.  So
> we can send another RPC before that buffer is retired.  (If I've got
> this badly wrong, I hope someone will correct me.
> 
> So the total amount of memory required to do this is going to depend
> on how fast data is being sent, rather than on the # of OSTs or any
> other constant.
> 
> There *is* a per OST limit to how many RPCs a client can have in
> flight at once, but it's generally set so the client can get good
> performance to one OST.  Allocating data for max_rpcs_in_flight*num
> OSTs would be far too much, because in the 1000 OST case, a client
> can probably only have a few hundred RPCs in flight (if that...) at
> once on a normal network.
> 
> But if we are writing from one client to many OSTs, how many RPCs are
> in flight at once is going to depend more on how fast our network is
> (or, possibly, CPU on the client if the network is fast and/or CPU is
> slow) than any explicit limits.  The explicit limits are much higher
> than we will hit in practice.
> 
> Does that make sense?  It doesn't make your problem any easier...

Totally, and you are right, it is more complex than I hoped. 

> 
> It actually seems like maybe a global pool of pages *is* the right
> answer.  The question is how big to make it...
> What about making it grow on demand up to a configurable upper limit?
> 
> The allocation code for encryption is here (it's pretty complicated
> and it works on the assumption that it must get pages or return
> ENOMEM - The compression code doesn't absolutely have to get pages,
> so it could be changed):
> sptlrpc_enc_pool_get_pages
> 
> It seems like maybe that code could be adjusted to serve both the
> encryption case (must not fail, if it can't get memory, return
> -ENOMEM to cause retries), and the compression case (can fail, if it
> fails, should not do compression...  Maybe should consume less
> memory)

Currently we are not very close to the sptlrpc layer and do not use any
of the encryption structures (it was initially planned, but turned out
differently). But we have already looked at those pools.

> 
> About thread counts:
> Encryption is handled in the ptlrpc code, and your presentation noted
> the plan is to mimic that, which sounds good to me.  That means
> there's no reason for you to explicitly control the number of threads
> doing compression, the same number of threads doing sending will be
> doing compression, which seems fine.  (Unless there's some point of
> contention in the compression code, but that seems unlikely...)

We currently intervene before the request is created
(osc_brw_prep_request) but still we don't do anything explicitly with
threads, just put some more tasks to the existing ones. Limited
resources is more the later part where we will optimize, tune and
introduce the adaptive part. 

> 
> Hope that helps a bit.

It helps a lot! Thank you!

Anna

From anna.fuchs at informatik.uni-hamburg.de  Thu Jul 27 08:26:58 2017
From: anna.fuchs at informatik.uni-hamburg.de (Anna Fuchs)
Date: Thu, 27 Jul 2017 10:26:58 +0200
Subject: [lustre-devel] Design proposal for client-side compression
In-Reply-To: <80491E10-0C6C-4826-AD55-5B78A20166D8@intel.com>
References: <1483967258.2622.24.camel@informatik.uni-hamburg.de>
	<B5E18A82-79B3-484F-BEBE-998B8AE56397@intel.com>
	<1484223328.3937.60.camel@informatik.uni-hamburg.de>
	<B924BE6D-3C6A-4BA4-96F9-AC053F032B2D@intel.com>
	<1484749185.3965.114.camel@informatik.uni-hamburg.de>
	<1487254536.4777.51.camel@informatik.uni-hamburg.de>
	<1500650130.3158.6.camel@informatik.uni-hamburg.de>
	<BN6PR1101MB2132311AB19B104AD044797ACBA40@BN6PR1101MB2132.namprd11.prod.outlook.com>
	<1500992740.4728.5.camel@informatik.uni-hamburg.de>
	<BN6PR1101MB2132DD0CCFF2F83BB3BE7F54CBB90@BN6PR1101MB2132.namprd11.prod.outlook.com>
	<80491E10-0C6C-4826-AD55-5B78A20166D8@intel.com>
Message-ID: <1501144018.3412.4.camel@informatik.uni-hamburg.de>

Jinshan,

On Wed, 2017-07-26 at 20:17 +0000, Xiong, Jinshan wrote:
> Thanks for the detailed explanation from Patrick.
>  
> “Does that mean, … Is there not any limit how many one
> client can have at one point of time?”
>  
> In theory, it’s possible that there exist that many active RPCs at
> one time, which is why I think it’s not feasible to have per-OSC page
> pool.

I see and agree, that could become a problem.

>  
> “… Once we have a smaller pool than
> we need, we have to block or skip compression, which is undesirable.
> But I don't know how to determine the required size.”
>  
> It’s probably not good to skip compression once it runs out of pages
> in pool, instead it should be blocked waiting for pages to be
> available. It will spend some time on waiting for the available
> pages, but at the end it will transfer less data over the network,
> and OST will also write less data to disk, so that it can still be
> performant.
> Of course, we can make it smarter by checking if there are too many
> threads waiting for available pages, and in that case we decide to
> not compress some RPCs. But this work can be deferred to the time
> after we have the code running and tune it by actual workload.

You are totally right, the smart tuning will be the second part of our
work, when the infrastructure is done. To start with something, we can
currently just block. 

> In order to decide the size of the pool, we should consider the
> number of CPUs on the client node, and the default RPC size. Let’s
> start with MAX(32, number_of_cpus) * default_RPC_size, and the
> default RPC size is 4MB in 2.10+ releases.

Thank you very much for the suggestions and explanations!

Anna

From marcin.s.ciupak at gmail.com  Thu Jul 27 10:59:00 2017
From: marcin.s.ciupak at gmail.com (Marcin Ciupak)
Date: Thu, 27 Jul 2017 12:59:00 +0200
Subject: [lustre-devel] [PATCH v2] staging: lustre: replace
	simple_strtoul with kstrtoint
In-Reply-To: <20170414081631.GC16598@kroah.com>
References: <20170321124609.GA3896@gentoo> <20170414081631.GC16598@kroah.com>
Message-ID: <20170727105900.GA21407@gentoo.lan>

I did test it and not everything works as expected. I
need to reconsider that change. Please drop this patch.

Thanks,
Marcin

P.S. Sorry for late (sic!) response.

On Fri, Apr 14, 2017 at 10:16:31AM +0200, Greg Kroah-Hartman wrote:
> On Tue, Mar 21, 2017 at 01:46:09PM +0100, Marcin Ciupak wrote:
> > Replace simple_strtoul with kstrtoint.
> > simple_strtoul is marked for obsoletion as reported by checkpatch.pl
> > 
> > Signed-off-by: Marcin Ciupak <marcin.s.ciupak at gmail.com>
> > ---
> > v2:
> > 	-improving kstrtoint error handling
> > 	-updating commit message
> > 
> >  drivers/staging/lustre/lustre/obdclass/obd_mount.c | 16 ++++++++++++----
> >  1 file changed, 12 insertions(+), 4 deletions(-)
> > 
> > diff --git a/drivers/staging/lustre/lustre/obdclass/obd_mount.c b/drivers/staging/lustre/lustre/obdclass/obd_mount.c
> > index 8e0d4b1d86dc..42858ee5b444 100644
> > --- a/drivers/staging/lustre/lustre/obdclass/obd_mount.c
> > +++ b/drivers/staging/lustre/lustre/obdclass/obd_mount.c
> > @@ -924,12 +924,20 @@ static int lmd_parse(char *options, struct lustre_mount_data *lmd)
> >  			lmd->lmd_flags |= LMD_FLG_ABORT_RECOV;
> >  			clear++;
> >  		} else if (strncmp(s1, "recovery_time_soft=", 19) == 0) {
> > -			lmd->lmd_recovery_time_soft = max_t(int,
> > -				simple_strtoul(s1 + 19, NULL, 10), time_min);
> > +			int res;
> > +
> > +			rc = kstrtoint(s1 + 19, 10, &res);
> > +			if (rc)
> > +				goto invalid;
> > +			lmd->lmd_recovery_time_soft = max_t(int, res, time_min);
> 
> Are you sure max_t is still needed here?
> 
> And have you tested this change?
> 
> thanks,
> 
> greg k-h

From rjh+lustre at cita.utoronto.ca  Thu Jul 27 17:14:31 2017
From: rjh+lustre at cita.utoronto.ca (Robin Humble)
Date: Thu, 27 Jul 2017 13:14:31 -0400
Subject: [lustre-devel] [PATCH 18/20] staging: lustre: llite: Remove
 filtering of seclabel xattr
In-Reply-To: <1501082556-17962-19-git-send-email-jsimmons@infradead.org>
References: <1501082556-17962-1-git-send-email-jsimmons@infradead.org>
	<1501082556-17962-19-git-send-email-jsimmons@infradead.org>
Message-ID: <20170727171431.GA14966@trinity.cita.utoronto.ca>

Hi James,

sorry, but this isn't my patch.
I deleted a couple of lines from xattr.c, not the below.

I think you might have got the wrong commit comment somehow?

I took lkml etc. out of the CC 'cos I'm not subscribed there.

cheers,
robin

On Wed, Jul 26, 2017 at 11:22:34AM -0400, James Simmons wrote:
>From: Robin Humble <plaguedbypenguins at gmail.com>
>
>The security.capability xattr is used to implement File
>Capabilities in recent Linux versions. Capabilities are a
>fine grained approach to granting executables elevated
>privileges. eg. /bin/ping can have capabilities
>cap_net_admin, cap_net_raw+ep instead of being setuid root.
>
>This xattr has long been filtered out by llite, initially for
>stability reasons (b15587), and later over performance
>concerns as this xattr is read for every file with eg.
>'ls --color'. Since LU-2869 xattr's are cached on clients,
>alleviating most performance concerns.
>
>Removing llite's filtering of the security.capability xattr
>enables using Lustre as a root filesystem, which is used on
>some large clusters.
>
>Signed-off-by: Robin Humble <plaguedbypenguins at gmail.com>
>Intel-bug-id: https://jira.hpdd.intel.com/browse/LU-9562
>Reviewed-on: https://review.whamcloud.com/27292
>Reviewed-by: John L. Hammond <john.hammond at intel.com>
>Reviewed-by: Sebastien Buisson <sbuisson at ddn.com>
>Reviewed-by: Oleg Drokin <oleg.drokin at intel.com>
>Signed-off-by: James Simmons <jsimmons at infradead.org>
>---
> drivers/staging/lustre/lustre/llite/file.c         | 58 ++++++++++++++++++++++
> .../staging/lustre/lustre/llite/llite_internal.h   |  4 ++
> drivers/staging/lustre/lustre/llite/namei.c        |  6 +++
> 3 files changed, 68 insertions(+)
>
>diff --git a/drivers/staging/lustre/lustre/llite/file.c b/drivers/staging/lustre/lustre/llite/file.c
>index 215479a..a324580 100644
>--- a/drivers/staging/lustre/lustre/llite/file.c
>+++ b/drivers/staging/lustre/lustre/llite/file.c
>@@ -3040,6 +3040,61 @@ struct posix_acl *ll_get_acl(struct inode *inode, int type)
> 	return acl;
> }
> 
>+#ifdef CONFIG_FS_POSIX_ACL
>+int ll_set_acl(struct inode *inode, struct posix_acl *acl, int type)
>+{
>+	const char *name = NULL;
>+	char *value = NULL;
>+	size_t size = 0;
>+	int rc = 0;
>+
>+	switch (type) {
>+	case ACL_TYPE_ACCESS:
>+		if (acl) {
>+			rc = posix_acl_update_mode(inode, &inode->i_mode, &acl);
>+			if (rc)
>+				goto out;
>+		}
>+		name = XATTR_NAME_POSIX_ACL_ACCESS;
>+		break;
>+	case ACL_TYPE_DEFAULT:
>+		if (!S_ISDIR(inode->i_mode)) {
>+			rc = acl ? -EACCES : 0;
>+			goto out;
>+		}
>+		name = XATTR_NAME_POSIX_ACL_DEFAULT;
>+		break;
>+	default:
>+		rc = -EINVAL;
>+		goto out;
>+	}
>+
>+	if (acl) {
>+		size = posix_acl_xattr_size(acl->a_count);
>+		value = kmalloc(size, GFP_NOFS);
>+		if (!value) {
>+			rc = -ENOMEM;
>+			goto out;
>+		}
>+
>+		rc = posix_acl_to_xattr(&init_user_ns, acl, value, size);
>+		if (rc < 0)
>+			goto out_free;
>+	}
>+
>+	/* dentry is only used for *.lov attributes so it's safe to be NULL */
>+	rc = __vfs_setxattr(NULL, inode, name, value, size, XATTR_CREATE);
>+out_free:
>+	kfree(value);
>+out:
>+	if (!rc)
>+		set_cached_acl(inode, type, acl);
>+	else
>+		forget_cached_acl(inode, type);
>+	return rc;
>+}
>+#endif /* CONFIG_FS_POSIX_ACL */
>+
> int ll_inode_permission(struct inode *inode, int mask)
> {
> 	struct ll_sb_info *sbi;
>@@ -3162,6 +3217,9 @@ int ll_inode_permission(struct inode *inode, int mask)
> 	.listxattr	= ll_listxattr,
> 	.fiemap		= ll_fiemap,
> 	.get_acl	= ll_get_acl,
>+#ifdef CONFIG_FS_POSIX_ACL
>+	.set_acl	= ll_set_acl,
>+#endif
> };
> 
> /* dynamic ioctl number support routines */
>diff --git a/drivers/staging/lustre/lustre/llite/llite_internal.h b/drivers/staging/lustre/lustre/llite/llite_internal.h
>index cd3311a..b3374bc 100644
>--- a/drivers/staging/lustre/lustre/llite/llite_internal.h
>+++ b/drivers/staging/lustre/lustre/llite/llite_internal.h
>@@ -752,6 +752,10 @@ enum ldlm_mode ll_take_md_lock(struct inode *inode, __u64 bits,
> int ll_getattr(const struct path *path, struct kstat *stat,
> 	       u32 request_mask, unsigned int flags);
> struct posix_acl *ll_get_acl(struct inode *inode, int type);
>+#ifdef CONFIG_FS_POSIX_ACL
>+int ll_set_acl(struct inode *inode, struct posix_acl *acl, int type);
>+#endif /* CONFIG_FS_POSIX_ACL */
>+
> int ll_migrate(struct inode *parent, struct file *file, int mdtidx,
> 	       const char *name, int namelen);
> int ll_get_fid_by_name(struct inode *parent, const char *name,
>diff --git a/drivers/staging/lustre/lustre/llite/namei.c b/drivers/staging/lustre/lustre/llite/namei.c
>index a208a8b..14dccbe 100644
>--- a/drivers/staging/lustre/lustre/llite/namei.c
>+++ b/drivers/staging/lustre/lustre/llite/namei.c
>@@ -1187,6 +1187,9 @@ static int ll_rename(struct inode *src, struct dentry *src_dchild,
> 	.permission	 = ll_inode_permission,
> 	.listxattr	  = ll_listxattr,
> 	.get_acl	    = ll_get_acl,
>+#ifdef CONFIG_FS_POSIX_ACL
>+	.set_acl	= ll_set_acl,
>+#endif
> };
> 
> const struct inode_operations ll_special_inode_operations = {
>@@ -1195,4 +1198,7 @@ static int ll_rename(struct inode *src, struct dentry *src_dchild,
> 	.permission     = ll_inode_permission,
> 	.listxattr      = ll_listxattr,
> 	.get_acl	    = ll_get_acl,
>+#ifdef CONFIG_FS_POSIX_ACL
>+	.set_acl	= ll_set_acl,
>+#endif
> };
>-- 
>1.8.3.1
>
>_______________________________________________
>lustre-devel mailing list
>lustre-devel at lists.lustre.org
>http://lists.lustre.org/listinfo.cgi/lustre-devel-lustre.org

From paf at cray.com  Thu Jul 27 19:15:57 2017
From: paf at cray.com (Patrick Farrell)
Date: Thu, 27 Jul 2017 19:15:57 +0000
Subject: [lustre-devel] HSM partial archive/restore
Message-ID: <BN6PR1101MB2132BF4D44A3965F09A83523CBBE0@BN6PR1101MB2132.namprd11.prod.outlook.com>

Hello all,


There has been some prior conversation around HSM partial archive/restore (Seagate at LUG'17: <http://cdn.opensfs.org/wp-content/uploads/2017/06/LUG-2017-Lustre-HSM-for-HPC-High-Speed-Distributed-Data-Mover-r1.11.pdf> http://cdn.opensfs.org/wp-content/uploads/2017/06/LUG-2017-Lustre-HSM-for-HPC-High-Speed-Distributed-Data-Mover-r1.11.pdf



<http://cdn.opensfs.org/wp-content/uploads/2017/06/LUG-2017-Lustre-HSM-for-HPC-High-Speed-Distributed-Data-Mover-r1.11.pdf>With complex file layouts and fallocate support being called out as requirements for enabling this.  But as far as I can tell, this has not progressed to any concrete design work.

Is this correct?  Is there work happening somewhere I'm not aware of?

If not, is there interest in starting some?  Cray has interest in this, and could help drive such a project, though we would need outside assistance (particularly with design) since we have no internal expertise relating to complex layouts.


Regards,

- Patrick


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-devel_lists.lustre.org/attachments/20170727/9b025580/attachment.html>

From paf at cray.com  Thu Jul 27 19:22:16 2017
From: paf at cray.com (Patrick Farrell)
Date: Thu, 27 Jul 2017 19:22:16 +0000
Subject: [lustre-devel] Design proposal for client-side compression
In-Reply-To: <1501143960.3412.3.camel@informatik.uni-hamburg.de>
References: <1483967258.2622.24.camel@informatik.uni-hamburg.de>
	<B5E18A82-79B3-484F-BEBE-998B8AE56397@intel.com>
	<1484223328.3937.60.camel@informatik.uni-hamburg.de>
	<B924BE6D-3C6A-4BA4-96F9-AC053F032B2D@intel.com>
	<1484749185.3965.114.camel@informatik.uni-hamburg.de>
	<1487254536.4777.51.camel@informatik.uni-hamburg.de>
	,<1500650130.3158.6.camel@informatik.uni-hamburg.de>
	<BN6PR1101MB2132311AB19B104AD044797ACBA40@BN6PR1101MB2132.namprd11.prod.outlook.com>
	,<1500992740.4728.5.camel@informatik.uni-hamburg.de>
	<BN6PR1101MB2132DD0CCFF2F83BB3BE7F54CBB90@BN6PR1101MB2132.namprd11.prod.outlook.com>,
	<1501143960.3412.3.camel@informatik.uni-hamburg.de>
Message-ID: <BN6PR1101MB2132BBFB8979EA3FBFA4AE2DCBBE0@BN6PR1101MB2132.namprd11.prod.outlook.com>

Ann,


I would be happy to help with review, etc, on this once it's ready to be posted.


In the meantime, I am curious about how you handled the compression and the discontiguous set of pages problem.  Did you use scatter-gather lists like the encryption code does, or some other solution?


Are you willing/able to share the current code, perhaps even off list?  I certainly understand if not, but I am curious to see how it will work and explore the performance implications.


- Patrick

________________________________
From: Anna Fuchs <anna.fuchs at informatik.uni-hamburg.de>
Sent: Thursday, July 27, 2017 3:26:00 AM
To: Patrick Farrell; Xiong, Jinshan
Cc: Matthew Ahrens; Zhuravlev, Alexey; lustre-devel
Subject: Re: [lustre-devel] Design proposal for client-side compression

Patrick,

> Having reread your LAD presentation (I was there, but it's been a
> while...), I think you've got a good architecture.

There have been some changes since that, but the general things should
be the same.

> A few thoughts.
>
> 1. Jinshan was just suggesting including in the code a switch to
> enable/disable the feature at runtime, for an example, see his fast
> read patch:
> https://review.whamcloud.com/#/c/20255/
> Especially the proc section:
> https://review.whamcloud.com/#/c/20255/7/lustre/llite/lproc_llite.c
> The effect of that is a file in proc that one can use to
> disable/enable the feature by echoing 0 or 1.
> (I think there is probably a place for tuning beyond that, but that's
> separate.)
> This is great for features that may have complex impacts, and also
> for people who want to test a feature to see how it changes things.

Oh, I misunderstood Jinshan last time, sorry. Yes, it would be much
easier for users and should be possible. Thank you for references!

> 2. Lustre clients iterate over the stripes, basically.
>
> Here's an explanation of the write path on the client that should
> help.  This explanation is heavily simplified and incorrect in some
> of the details, but should be accurate enough for your question.
> The I/O model on the client (for buffered I/O, direct I/O is
> different) is that the writing process (userspace process) starts an
> I/O, then identifies which parts of the I/O go to which stripes, gets
> the locks it needs, then copies the data through the page cache...
> Once the data is copied to the page cache, Lustre then works on
> writing out that data.  In general, it does it asynchronously, where
> the userspace process returns and then data write-out is handled by
> the ptlrpcd (daemon) threads, but in various exceptional conditions
> it may do the write-out in the userspace process.
>
> In general, the write out is going to happen in parallel (to
> different OSTs) with different ptlrpcd threads taking different
> chunks of data and putting them on the wire, and sometimes the
> userspace thread doing that work for some of the data as well.
>
> So "How much memory do we need at most at the same time?" is not a
> question with an easy answer.  When doing a bulk RPC, generally, the
> sender sends an RPC announcing the bulk data is ready, then the
> recipient copies the data (RDMA) (or the sender sends it over to a
> buffer if no RDMA) and announces to the client it has done so.  I'm
> not 100% clear on the sequencing here, but the key thing is there's a
> time where we've sent the RPC but we aren't done with the buffer.  So
> we can send another RPC before that buffer is retired.  (If I've got
> this badly wrong, I hope someone will correct me.
>
> So the total amount of memory required to do this is going to depend
> on how fast data is being sent, rather than on the # of OSTs or any
> other constant.
>
> There *is* a per OST limit to how many RPCs a client can have in
> flight at once, but it's generally set so the client can get good
> performance to one OST.  Allocating data for max_rpcs_in_flight*num
> OSTs would be far too much, because in the 1000 OST case, a client
> can probably only have a few hundred RPCs in flight (if that...) at
> once on a normal network.
>
> But if we are writing from one client to many OSTs, how many RPCs are
> in flight at once is going to depend more on how fast our network is
> (or, possibly, CPU on the client if the network is fast and/or CPU is
> slow) than any explicit limits.  The explicit limits are much higher
> than we will hit in practice.
>
> Does that make sense?  It doesn't make your problem any easier...

Totally, and you are right, it is more complex than I hoped.

>
> It actually seems like maybe a global pool of pages *is* the right
> answer.  The question is how big to make it...
> What about making it grow on demand up to a configurable upper limit?
>
> The allocation code for encryption is here (it's pretty complicated
> and it works on the assumption that it must get pages or return
> ENOMEM - The compression code doesn't absolutely have to get pages,
> so it could be changed):
> sptlrpc_enc_pool_get_pages
>
> It seems like maybe that code could be adjusted to serve both the
> encryption case (must not fail, if it can't get memory, return
> -ENOMEM to cause retries), and the compression case (can fail, if it
> fails, should not do compression...  Maybe should consume less
> memory)

Currently we are not very close to the sptlrpc layer and do not use any
of the encryption structures (it was initially planned, but turned out
differently). But we have already looked at those pools.

>
> About thread counts:
> Encryption is handled in the ptlrpc code, and your presentation noted
> the plan is to mimic that, which sounds good to me.  That means
> there's no reason for you to explicitly control the number of threads
> doing compression, the same number of threads doing sending will be
> doing compression, which seems fine.  (Unless there's some point of
> contention in the compression code, but that seems unlikely...)

We currently intervene before the request is created
(osc_brw_prep_request) but still we don't do anything explicitly with
threads, just put some more tasks to the existing ones. Limited
resources is more the later part where we will optimize, tune and
introduce the adaptive part.

>
> Hope that helps a bit.

It helps a lot! Thank you!

Anna
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-devel_lists.lustre.org/attachments/20170727/4cbace68/attachment.html>

From jsimmons at infradead.org  Thu Jul 27 19:35:33 2017
From: jsimmons at infradead.org (James Simmons)
Date: Thu, 27 Jul 2017 20:35:33 +0100 (BST)
Subject: [lustre-devel] [PATCH 18/20] staging: lustre: llite: Remove
 filtering of seclabel xattr
In-Reply-To: <1501082556-17962-19-git-send-email-jsimmons@infradead.org>
References: <1501082556-17962-1-git-send-email-jsimmons@infradead.org>
	<1501082556-17962-19-git-send-email-jsimmons@infradead.org>
Message-ID: <alpine.LFD.2.20.1707272034380.6670@casper.infradead.org>


> From: Robin Humble <plaguedbypenguins at gmail.com>
> 
> The security.capability xattr is used to implement File
> Capabilities in recent Linux versions. Capabilities are a
> fine grained approach to granting executables elevated
> privileges. eg. /bin/ping can have capabilities
> cap_net_admin, cap_net_raw+ep instead of being setuid root.
> 
> This xattr has long been filtered out by llite, initially for
> stability reasons (b15587), and later over performance
> concerns as this xattr is read for every file with eg.
> 'ls --color'. Since LU-2869 xattr's are cached on clients,
> alleviating most performance concerns.
> 
> Removing llite's filtering of the security.capability xattr
> enables using Lustre as a root filesystem, which is used on
> some large clusters.

The commit message for this patch is incorrect. Some how it got
mixed up with another patch which I missed in this push. Please
drop this patch and I will resent the correct patches later.
 
> Signed-off-by: Robin Humble <plaguedbypenguins at gmail.com>
> Intel-bug-id: https://jira.hpdd.intel.com/browse/LU-9562
> Reviewed-on: https://review.whamcloud.com/27292
> Reviewed-by: John L. Hammond <john.hammond at intel.com>
> Reviewed-by: Sebastien Buisson <sbuisson at ddn.com>
> Reviewed-by: Oleg Drokin <oleg.drokin at intel.com>
> Signed-off-by: James Simmons <jsimmons at infradead.org>
> ---
>  drivers/staging/lustre/lustre/llite/file.c         | 58 ++++++++++++++++++++++
>  .../staging/lustre/lustre/llite/llite_internal.h   |  4 ++
>  drivers/staging/lustre/lustre/llite/namei.c        |  6 +++
>  3 files changed, 68 insertions(+)
> 
> diff --git a/drivers/staging/lustre/lustre/llite/file.c b/drivers/staging/lustre/lustre/llite/file.c
> index 215479a..a324580 100644
> --- a/drivers/staging/lustre/lustre/llite/file.c
> +++ b/drivers/staging/lustre/lustre/llite/file.c
> @@ -3040,6 +3040,61 @@ struct posix_acl *ll_get_acl(struct inode *inode, int type)
>  	return acl;
>  }
>  
> +#ifdef CONFIG_FS_POSIX_ACL
> +int ll_set_acl(struct inode *inode, struct posix_acl *acl, int type)
> +{
> +	const char *name = NULL;
> +	char *value = NULL;
> +	size_t size = 0;
> +	int rc = 0;
> +
> +	switch (type) {
> +	case ACL_TYPE_ACCESS:
> +		if (acl) {
> +			rc = posix_acl_update_mode(inode, &inode->i_mode, &acl);
> +			if (rc)
> +				goto out;
> +		}
> +		name = XATTR_NAME_POSIX_ACL_ACCESS;
> +		break;
> +	case ACL_TYPE_DEFAULT:
> +		if (!S_ISDIR(inode->i_mode)) {
> +			rc = acl ? -EACCES : 0;
> +			goto out;
> +		}
> +		name = XATTR_NAME_POSIX_ACL_DEFAULT;
> +		break;
> +	default:
> +		rc = -EINVAL;
> +		goto out;
> +	}
> +
> +	if (acl) {
> +		size = posix_acl_xattr_size(acl->a_count);
> +		value = kmalloc(size, GFP_NOFS);
> +		if (!value) {
> +			rc = -ENOMEM;
> +			goto out;
> +		}
> +
> +		rc = posix_acl_to_xattr(&init_user_ns, acl, value, size);
> +		if (rc < 0)
> +			goto out_free;
> +	}
> +
> +	/* dentry is only used for *.lov attributes so it's safe to be NULL */
> +	rc = __vfs_setxattr(NULL, inode, name, value, size, XATTR_CREATE);
> +out_free:
> +	kfree(value);
> +out:
> +	if (!rc)
> +		set_cached_acl(inode, type, acl);
> +	else
> +		forget_cached_acl(inode, type);
> +	return rc;
> +}
> +#endif /* CONFIG_FS_POSIX_ACL */
> +
>  int ll_inode_permission(struct inode *inode, int mask)
>  {
>  	struct ll_sb_info *sbi;
> @@ -3162,6 +3217,9 @@ int ll_inode_permission(struct inode *inode, int mask)
>  	.listxattr	= ll_listxattr,
>  	.fiemap		= ll_fiemap,
>  	.get_acl	= ll_get_acl,
> +#ifdef CONFIG_FS_POSIX_ACL
> +	.set_acl	= ll_set_acl,
> +#endif
>  };
>  
>  /* dynamic ioctl number support routines */
> diff --git a/drivers/staging/lustre/lustre/llite/llite_internal.h b/drivers/staging/lustre/lustre/llite/llite_internal.h
> index cd3311a..b3374bc 100644
> --- a/drivers/staging/lustre/lustre/llite/llite_internal.h
> +++ b/drivers/staging/lustre/lustre/llite/llite_internal.h
> @@ -752,6 +752,10 @@ enum ldlm_mode ll_take_md_lock(struct inode *inode, __u64 bits,
>  int ll_getattr(const struct path *path, struct kstat *stat,
>  	       u32 request_mask, unsigned int flags);
>  struct posix_acl *ll_get_acl(struct inode *inode, int type);
> +#ifdef CONFIG_FS_POSIX_ACL
> +int ll_set_acl(struct inode *inode, struct posix_acl *acl, int type);
> +#endif /* CONFIG_FS_POSIX_ACL */
> +
>  int ll_migrate(struct inode *parent, struct file *file, int mdtidx,
>  	       const char *name, int namelen);
>  int ll_get_fid_by_name(struct inode *parent, const char *name,
> diff --git a/drivers/staging/lustre/lustre/llite/namei.c b/drivers/staging/lustre/lustre/llite/namei.c
> index a208a8b..14dccbe 100644
> --- a/drivers/staging/lustre/lustre/llite/namei.c
> +++ b/drivers/staging/lustre/lustre/llite/namei.c
> @@ -1187,6 +1187,9 @@ static int ll_rename(struct inode *src, struct dentry *src_dchild,
>  	.permission	 = ll_inode_permission,
>  	.listxattr	  = ll_listxattr,
>  	.get_acl	    = ll_get_acl,
> +#ifdef CONFIG_FS_POSIX_ACL
> +	.set_acl	= ll_set_acl,
> +#endif
>  };
>  
>  const struct inode_operations ll_special_inode_operations = {
> @@ -1195,4 +1198,7 @@ static int ll_rename(struct inode *src, struct dentry *src_dchild,
>  	.permission     = ll_inode_permission,
>  	.listxattr      = ll_listxattr,
>  	.get_acl	    = ll_get_acl,
> +#ifdef CONFIG_FS_POSIX_ACL
> +	.set_acl	= ll_set_acl,
> +#endif
>  };
> -- 
> 1.8.3.1
> 
> 

From jsimmons at infradead.org  Thu Jul 27 19:42:46 2017
From: jsimmons at infradead.org (James Simmons)
Date: Thu, 27 Jul 2017 20:42:46 +0100 (BST)
Subject: [lustre-devel] [PATCH 18/20] staging: lustre: llite: Remove
 filtering of seclabel xattr
In-Reply-To: <20170727171431.GA14966@trinity.cita.utoronto.ca>
References: <1501082556-17962-1-git-send-email-jsimmons@infradead.org>
	<1501082556-17962-19-git-send-email-jsimmons@infradead.org>
	<20170727171431.GA14966@trinity.cita.utoronto.ca>
Message-ID: <alpine.LFD.2.20.1707272042280.6670@casper.infradead.org>


> Hi James,
> 
> sorry, but this isn't my patch.
> I deleted a couple of lines from xattr.c, not the below.
> 
> I think you might have got the wrong commit comment somehow?
> 
> I took lkml etc. out of the CC 'cos I'm not subscribed there.

I sent the email. Sorry about the mix up.
 
> cheers,
> robin
> 
> On Wed, Jul 26, 2017 at 11:22:34AM -0400, James Simmons wrote:
> >From: Robin Humble <plaguedbypenguins at gmail.com>
> >
> >The security.capability xattr is used to implement File
> >Capabilities in recent Linux versions. Capabilities are a
> >fine grained approach to granting executables elevated
> >privileges. eg. /bin/ping can have capabilities
> >cap_net_admin, cap_net_raw+ep instead of being setuid root.
> >
> >This xattr has long been filtered out by llite, initially for
> >stability reasons (b15587), and later over performance
> >concerns as this xattr is read for every file with eg.
> >'ls --color'. Since LU-2869 xattr's are cached on clients,
> >alleviating most performance concerns.
> >
> >Removing llite's filtering of the security.capability xattr
> >enables using Lustre as a root filesystem, which is used on
> >some large clusters.
> >
> >Signed-off-by: Robin Humble <plaguedbypenguins at gmail.com>
> >Intel-bug-id: https://jira.hpdd.intel.com/browse/LU-9562
> >Reviewed-on: https://review.whamcloud.com/27292
> >Reviewed-by: John L. Hammond <john.hammond at intel.com>
> >Reviewed-by: Sebastien Buisson <sbuisson at ddn.com>
> >Reviewed-by: Oleg Drokin <oleg.drokin at intel.com>
> >Signed-off-by: James Simmons <jsimmons at infradead.org>
> >---
> > drivers/staging/lustre/lustre/llite/file.c         | 58 ++++++++++++++++++++++
> > .../staging/lustre/lustre/llite/llite_internal.h   |  4 ++
> > drivers/staging/lustre/lustre/llite/namei.c        |  6 +++
> > 3 files changed, 68 insertions(+)
> >
> >diff --git a/drivers/staging/lustre/lustre/llite/file.c b/drivers/staging/lustre/lustre/llite/file.c
> >index 215479a..a324580 100644
> >--- a/drivers/staging/lustre/lustre/llite/file.c
> >+++ b/drivers/staging/lustre/lustre/llite/file.c
> >@@ -3040,6 +3040,61 @@ struct posix_acl *ll_get_acl(struct inode *inode, int type)
> > 	return acl;
> > }
> > 
> >+#ifdef CONFIG_FS_POSIX_ACL
> >+int ll_set_acl(struct inode *inode, struct posix_acl *acl, int type)
> >+{
> >+	const char *name = NULL;
> >+	char *value = NULL;
> >+	size_t size = 0;
> >+	int rc = 0;
> >+
> >+	switch (type) {
> >+	case ACL_TYPE_ACCESS:
> >+		if (acl) {
> >+			rc = posix_acl_update_mode(inode, &inode->i_mode, &acl);
> >+			if (rc)
> >+				goto out;
> >+		}
> >+		name = XATTR_NAME_POSIX_ACL_ACCESS;
> >+		break;
> >+	case ACL_TYPE_DEFAULT:
> >+		if (!S_ISDIR(inode->i_mode)) {
> >+			rc = acl ? -EACCES : 0;
> >+			goto out;
> >+		}
> >+		name = XATTR_NAME_POSIX_ACL_DEFAULT;
> >+		break;
> >+	default:
> >+		rc = -EINVAL;
> >+		goto out;
> >+	}
> >+
> >+	if (acl) {
> >+		size = posix_acl_xattr_size(acl->a_count);
> >+		value = kmalloc(size, GFP_NOFS);
> >+		if (!value) {
> >+			rc = -ENOMEM;
> >+			goto out;
> >+		}
> >+
> >+		rc = posix_acl_to_xattr(&init_user_ns, acl, value, size);
> >+		if (rc < 0)
> >+			goto out_free;
> >+	}
> >+
> >+	/* dentry is only used for *.lov attributes so it's safe to be NULL */
> >+	rc = __vfs_setxattr(NULL, inode, name, value, size, XATTR_CREATE);
> >+out_free:
> >+	kfree(value);
> >+out:
> >+	if (!rc)
> >+		set_cached_acl(inode, type, acl);
> >+	else
> >+		forget_cached_acl(inode, type);
> >+	return rc;
> >+}
> >+#endif /* CONFIG_FS_POSIX_ACL */
> >+
> > int ll_inode_permission(struct inode *inode, int mask)
> > {
> > 	struct ll_sb_info *sbi;
> >@@ -3162,6 +3217,9 @@ int ll_inode_permission(struct inode *inode, int mask)
> > 	.listxattr	= ll_listxattr,
> > 	.fiemap		= ll_fiemap,
> > 	.get_acl	= ll_get_acl,
> >+#ifdef CONFIG_FS_POSIX_ACL
> >+	.set_acl	= ll_set_acl,
> >+#endif
> > };
> > 
> > /* dynamic ioctl number support routines */
> >diff --git a/drivers/staging/lustre/lustre/llite/llite_internal.h b/drivers/staging/lustre/lustre/llite/llite_internal.h
> >index cd3311a..b3374bc 100644
> >--- a/drivers/staging/lustre/lustre/llite/llite_internal.h
> >+++ b/drivers/staging/lustre/lustre/llite/llite_internal.h
> >@@ -752,6 +752,10 @@ enum ldlm_mode ll_take_md_lock(struct inode *inode, __u64 bits,
> > int ll_getattr(const struct path *path, struct kstat *stat,
> > 	       u32 request_mask, unsigned int flags);
> > struct posix_acl *ll_get_acl(struct inode *inode, int type);
> >+#ifdef CONFIG_FS_POSIX_ACL
> >+int ll_set_acl(struct inode *inode, struct posix_acl *acl, int type);
> >+#endif /* CONFIG_FS_POSIX_ACL */
> >+
> > int ll_migrate(struct inode *parent, struct file *file, int mdtidx,
> > 	       const char *name, int namelen);
> > int ll_get_fid_by_name(struct inode *parent, const char *name,
> >diff --git a/drivers/staging/lustre/lustre/llite/namei.c b/drivers/staging/lustre/lustre/llite/namei.c
> >index a208a8b..14dccbe 100644
> >--- a/drivers/staging/lustre/lustre/llite/namei.c
> >+++ b/drivers/staging/lustre/lustre/llite/namei.c
> >@@ -1187,6 +1187,9 @@ static int ll_rename(struct inode *src, struct dentry *src_dchild,
> > 	.permission	 = ll_inode_permission,
> > 	.listxattr	  = ll_listxattr,
> > 	.get_acl	    = ll_get_acl,
> >+#ifdef CONFIG_FS_POSIX_ACL
> >+	.set_acl	= ll_set_acl,
> >+#endif
> > };
> > 
> > const struct inode_operations ll_special_inode_operations = {
> >@@ -1195,4 +1198,7 @@ static int ll_rename(struct inode *src, struct dentry *src_dchild,
> > 	.permission     = ll_inode_permission,
> > 	.listxattr      = ll_listxattr,
> > 	.get_acl	    = ll_get_acl,
> >+#ifdef CONFIG_FS_POSIX_ACL
> >+	.set_acl	= ll_set_acl,
> >+#endif
> > };
> >-- 
> >1.8.3.1
> >
> >_______________________________________________
> >lustre-devel mailing list
> >lustre-devel at lists.lustre.org
> >http://lists.lustre.org/listinfo.cgi/lustre-devel-lustre.org
> 

From anna.fuchs at informatik.uni-hamburg.de  Fri Jul 28 09:57:27 2017
From: anna.fuchs at informatik.uni-hamburg.de (Anna Fuchs)
Date: Fri, 28 Jul 2017 11:57:27 +0200
Subject: [lustre-devel] Design proposal for client-side compression
In-Reply-To: <BN6PR1101MB2132BBFB8979EA3FBFA4AE2DCBBE0@BN6PR1101MB2132.namprd11.prod.outlook.com>
References: <1483967258.2622.24.camel@informatik.uni-hamburg.de>
	<B5E18A82-79B3-484F-BEBE-998B8AE56397@intel.com>
	<1484223328.3937.60.camel@informatik.uni-hamburg.de>
	<B924BE6D-3C6A-4BA4-96F9-AC053F032B2D@intel.com>
	<1484749185.3965.114.camel@informatik.uni-hamburg.de>
	<1487254536.4777.51.camel@informatik.uni-hamburg.de>
	,<1500650130.3158.6.camel@informatik.uni-hamburg.de>
	<BN6PR1101MB2132311AB19B104AD044797ACBA40@BN6PR1101MB2132.namprd11.prod.outlook.com>
	,<1500992740.4728.5.camel@informatik.uni-hamburg.de>
	<BN6PR1101MB2132DD0CCFF2F83BB3BE7F54CBB90@BN6PR1101MB2132.namprd11.prod.outlook.com>
	,<1501143960.3412.3.camel@informatik.uni-hamburg.de>
	<BN6PR1101MB2132BBFB8979EA3FBFA4AE2DCBBE0@BN6PR1101MB2132.namprd11.prod.outlook.com>
Message-ID: <1501235847.3412.12.camel@informatik.uni-hamburg.de>

Patrick,

On Thu, 2017-07-27 at 19:22 +0000, Patrick Farrell wrote:
> Ann,
> 
> I would be happy to help with review, etc, on this once it's ready to
> be posted.

thanks for that! 

> In the meantime, I am curious about how you handled the compression
> and the discontiguous set of pages problem.  Did you use scatter-
> gather lists like the encryption code does, or some other solution?

I am mainly working on the infrastructure and Lustre/ZFS integration
regardless the concrete algorithm, but we faced this problem very
early. In my prototype I still have the very costly approach of
allocating three contiguous buffers (src, dst, wrkmem), allocating
additional destination pages, copying original pages to void* src
buffer, compressing to void* dst buffer and again copying to dst page
buffer. A lot of expensive copies and memory wasting. But with the
original Kernel-LZ4 there is no other way. I can send you the
corresponding code part, but it is totally boring - alloc, alloc,
alloc... copy, copy, ... copy.

In parallel to my work site, we assigned a student to adopt LZ4 to the
page structure. Our first idea has also been scatter-lists seen in the
encryption code. Since scatter-lists use linked lists it somehow turned
out to be very inefficient for traversing the data. The corresponding
Bachelor's thesis will be submitted soon (within a month?), so we have
to proofread it in detail. However, the student implemented another
version of LZ4, which works directly on pages (code party online, full
version will follow).
It is tested, but might be not in the productive stage now (will
hopefully be after submission and reviewing). This version shows a
little lower compression ratio but comparable or better speed. We will
see how we can use it to avoid the memory and copy overhead. It seemed 
there is no good way how to change only the data structure in a clean
way without changing the de-/compressor's logic. 

Another interesting thing is the newest LZ4m [0], which is similar to
the work of our student in many aspects, but still differs (waiting for
final thesis). 

However, for the LZ4 we see good chances to get rid of that overhead by
using which ever modification. But since we want and should support
more algorithms, we still do not have any universal solution. E.g. for
zstd, which also seems to be suitable for our needs (another thesis..),
we would need to make same efforts again or pay the overhead. 

[0] http://csl.skku.edu/papers/icce17.pdf

If anyone has other ideas, please, let me know!

Regards,
Anna

From paf at cray.com  Fri Jul 28 13:46:45 2017
From: paf at cray.com (Patrick Farrell)
Date: Fri, 28 Jul 2017 13:46:45 +0000
Subject: [lustre-devel] Design proposal for client-side compression
In-Reply-To: <1501235847.3412.12.camel@informatik.uni-hamburg.de>
References: <1483967258.2622.24.camel@informatik.uni-hamburg.de>
	<B5E18A82-79B3-484F-BEBE-998B8AE56397@intel.com>
	<1484223328.3937.60.camel@informatik.uni-hamburg.de>
	<B924BE6D-3C6A-4BA4-96F9-AC053F032B2D@intel.com>
	<1484749185.3965.114.camel@informatik.uni-hamburg.de>
	<1487254536.4777.51.camel@informatik.uni-hamburg.de>
	,<1500650130.3158.6.camel@informatik.uni-hamburg.de>
	<BN6PR1101MB2132311AB19B104AD044797ACBA40@BN6PR1101MB2132.namprd11.prod.outlook.com>
	,<1500992740.4728.5.camel@informatik.uni-hamburg.de>
	<BN6PR1101MB2132DD0CCFF2F83BB3BE7F54CBB90@BN6PR1101MB2132.namprd11.prod.outlook.com>
	,<1501143960.3412.3.camel@informatik.uni-hamburg.de>
	<BN6PR1101MB2132BBFB8979EA3FBFA4AE2DCBBE0@BN6PR1101MB2132.namprd11.prod.outlook.com>,
	<1501235847.3412.12.camel@informatik.uni-hamburg.de>
Message-ID: <BN6PR1101MB2132018DF6EED212C690693FCBBF0@BN6PR1101MB2132.namprd11.prod.outlook.com>


Ah.  As it turns out, much more complicated than I anticipated.  Thanks for explaining...


I have no expertise in compression algorithms, so that I will have to just watch from the sidelines.  Good luck.

When you are further along, I remain interested in helping out with the Lustre side of things.


One more question - Do you have a plan to make this work *without* the ZFS integration as well, for those using ldiskfs?  That seems straightforward enough - compress/decompress at send and recieve time - even if the benefits would be smaller, but not everyone (Cray, f.x.) is using ZFS, so I'm very interested in something that would help ldiskfs as well.  (Which is not to say don't do the deeper integration with ZFS.  Just that we'd like something available for ldiskfs too.)


Regards,

- Patrick

________________________________
From: Anna Fuchs <anna.fuchs at informatik.uni-hamburg.de>
Sent: Friday, July 28, 2017 4:57:27 AM
To: Patrick Farrell; Xiong, Jinshan
Cc: Matthew Ahrens; Zhuravlev, Alexey; lustre-devel
Subject: Re: [lustre-devel] Design proposal for client-side compression

Patrick,

On Thu, 2017-07-27 at 19:22 +0000, Patrick Farrell wrote:
> Ann,
>
> I would be happy to help with review, etc, on this once it's ready to
> be posted.

thanks for that!

> In the meantime, I am curious about how you handled the compression
> and the discontiguous set of pages problem.  Did you use scatter-
> gather lists like the encryption code does, or some other solution?

I am mainly working on the infrastructure and Lustre/ZFS integration
regardless the concrete algorithm, but we faced this problem very
early. In my prototype I still have the very costly approach of
allocating three contiguous buffers (src, dst, wrkmem), allocating
additional destination pages, copying original pages to void* src
buffer, compressing to void* dst buffer and again copying to dst page
buffer. A lot of expensive copies and memory wasting. But with the
original Kernel-LZ4 there is no other way. I can send you the
corresponding code part, but it is totally boring - alloc, alloc,
alloc... copy, copy, ... copy.

In parallel to my work site, we assigned a student to adopt LZ4 to the
page structure. Our first idea has also been scatter-lists seen in the
encryption code. Since scatter-lists use linked lists it somehow turned
out to be very inefficient for traversing the data. The corresponding
Bachelor's thesis will be submitted soon (within a month?), so we have
to proofread it in detail. However, the student implemented another
version of LZ4, which works directly on pages (code party online, full
version will follow).
It is tested, but might be not in the productive stage now (will
hopefully be after submission and reviewing). This version shows a
little lower compression ratio but comparable or better speed. We will
see how we can use it to avoid the memory and copy overhead. It seemed
there is no good way how to change only the data structure in a clean
way without changing the de-/compressor's logic.

Another interesting thing is the newest LZ4m [0], which is similar to
the work of our student in many aspects, but still differs (waiting for
final thesis).

However, for the LZ4 we see good chances to get rid of that overhead by
using which ever modification. But since we want and should support
more algorithms, we still do not have any universal solution. E.g. for
zstd, which also seems to be suitable for our needs (another thesis..),
we would need to make same efforts again or pay the overhead.

[0] http://csl.skku.edu/papers/icce17.pdf
LZ4m: A Fast Compression Algorithm for In-Memory Data<http://csl.skku.edu/papers/icce17.pdf>
csl.skku.edu
LZ4m: A Fast Compression Algorithm for In-Memory Data Se-Jun Kwon ∗, Sang-Hoon Kim†, Hyeong-Jun Kim∗, and Jin-Soo Kim ∗College of Information and ...




If anyone has other ideas, please, let me know!

Regards,
Anna
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-devel_lists.lustre.org/attachments/20170728/38ce7e74/attachment.html>

From dan.carpenter at oracle.com  Fri Jul 28 14:42:27 2017
From: dan.carpenter at oracle.com (Dan Carpenter)
Date: Fri, 28 Jul 2017 17:42:27 +0300
Subject: [lustre-devel] [PATCH] staging: lustre: lustre: Off by two in
	lmv_fid2path()
Message-ID: <20170728144227.uml3x27mqug3ciox@mwanda>

We want to concatonate join string one, a '/' character, string two and
then a NUL terminator.  The destination buffer holds ori_gf->gf_pathlen
characters.  The strlen() function returns the number of characters not
counting the NUL terminator.  So we should be adding two extra spaces,
one for the foward slash and one for the NUL.

Signed-off-by: Dan Carpenter <dan.carpenter at oracle.com>
---
Not tested.  Please review this one carefully.

diff --git a/drivers/staging/lustre/lustre/lmv/lmv_obd.c b/drivers/staging/lustre/lustre/lmv/lmv_obd.c
index 64fcaef0bacd..70c58291814d 100644
--- a/drivers/staging/lustre/lustre/lmv/lmv_obd.c
+++ b/drivers/staging/lustre/lustre/lmv/lmv_obd.c
@@ -657,8 +657,8 @@ static int lmv_fid2path(struct obd_export *exp, int len, void *karg,
 		char *ptr;
 
 		ori_gf = karg;
-		if (strlen(ori_gf->gf_path) +
-		    strlen(gf->gf_path) > ori_gf->gf_pathlen) {
+		if (strlen(ori_gf->gf_path) + 1 +
+		    strlen(gf->gf_path) + 1 > ori_gf->gf_pathlen) {
 			rc = -EOVERFLOW;
 			goto out_fid2path;
 		}

From anna.fuchs at informatik.uni-hamburg.de  Fri Jul 28 15:12:16 2017
From: anna.fuchs at informatik.uni-hamburg.de (Anna Fuchs)
Date: Fri, 28 Jul 2017 17:12:16 +0200
Subject: [lustre-devel] Design proposal for client-side compression
In-Reply-To: <BN6PR1101MB2132018DF6EED212C690693FCBBF0@BN6PR1101MB2132.namprd11.prod.outlook.com>
References: <1483967258.2622.24.camel@informatik.uni-hamburg.de>
	<B5E18A82-79B3-484F-BEBE-998B8AE56397@intel.com>
	<1484223328.3937.60.camel@informatik.uni-hamburg.de>
	<B924BE6D-3C6A-4BA4-96F9-AC053F032B2D@intel.com>
	<1484749185.3965.114.camel@informatik.uni-hamburg.de>
	<1487254536.4777.51.camel@informatik.uni-hamburg.de>
	,<1500650130.3158.6.camel@informatik.uni-hamburg.de>
	<BN6PR1101MB2132311AB19B104AD044797ACBA40@BN6PR1101MB2132.namprd11.prod.outlook.com>
	,<1500992740.4728.5.camel@informatik.uni-hamburg.de>
	<BN6PR1101MB2132DD0CCFF2F83BB3BE7F54CBB90@BN6PR1101MB2132.namprd11.prod.outlook.com>
	,<1501143960.3412.3.camel@informatik.uni-hamburg.de>
	<BN6PR1101MB2132BBFB8979EA3FBFA4AE2DCBBE0@BN6PR1101MB2132.namprd11.prod.outlook.com>
	,<1501235847.3412.12.camel@informatik.uni-hamburg.de>
	<BN6PR1101MB2132018DF6EED212C690693FCBBF0@BN6PR1101MB2132.namprd11.prod.outlook.com>
Message-ID: <1501254736.3412.16.camel@informatik.uni-hamburg.de>


> Ah.  As it turns out, much more complicated than I anticipated.
>  Thanks for explaining...
> 
> I have no expertise in compression algorithms, so that I will have to
> just watch from the sidelines.  Good luck.
> 
> When you are further along, I remain interested in helping out with
> the Lustre side of things.
> 
> One more question - Do you have a plan to make this work *without*
> the ZFS integration as well, for those using ldiskfs?  That seems
> straightforward enough - compress/decompress at send and recieve time
> - even if the benefits would be smaller, but not everyone (Cray,
> f.x.) is using ZFS, so I'm very interested in something that would
> help ldiskfs as well.  (Which is not to say don't do the deeper
> integration with ZFS.  Just that we'd like something available for
> ldiskfs too.)

I fear it is also much more complicated :)

At the very beginning of the project proposal we hoped we wouldn't need
to touch the server so much. It turned out wrong, moreover we have to
modify not only the Lustre server, but also pretty much the backend
itself. We chose ZFS since it already provides a lot of infrastructure
that we would need to implement completely new in ldiskfs. Since, at
least for me, it is a research project, ldiskfs is out of scope. Once
we proved the concept, one could re-implement the whole compression
stack for ldiskfs. So it is not impossible, but not our focus for this
project. 

Nevertheless we tried to keep our changes as far as possible not very
backend specific. For example we need some additional information to be
stored per compressed chunk. One possibility would be to change the
block pointer of ZFS and add those fields, but I don't think anyone
except of us would like the BP to be modified :) So we decided to store
them as a header for every chunk. For ldiskfs, since one would need to
implement everything from scratch anyway, one might not need that
header, but take the required fields into account from the beginning
and add them to ldiskfs' "block pointer". For that reason, we wanted to
leave the compressed data "headerless" on client-side, and add the
header only on the server side if the corresponding backend requires
it. 

Well, we did it, and it even works sometimes, but it looks horrible 
and is really counterintuitive. We send less data from client than
lands on the OST, recalculate offsets, since we add the header during
receiving on server side, recalculate the sent and received sizes,
shift buffers by offsets and so on. The only advantage of this approach
is client's independence from backend. We decided the price is too
high. So now, I will construct the chunk with the header just after
compressing the data on client-side, get rid of all those offset stuff
on the server. But ldiskfs will have to deal with that ZFS-motivated
details. 

However, a light version of compression could work with smaller changes
to ldiskfs, when we only allow a completely compressed or not
compressed files and allow potential performance drops for broken read-
ahead (due to gaps within the data). 

Hope it is somehow more clear now.

Regards,
Anna

From jlayton at kernel.org  Fri Jul 28 14:23:21 2017
From: jlayton at kernel.org (Jeff Layton)
Date: Fri, 28 Jul 2017 10:23:21 -0400
Subject: [lustre-devel] [PATCH] fs: convert a pile of fsync routines to
	errseq_t based reporting
Message-ID: <20170728142321.11492-1-jlayton@kernel.org>

From: Jeff Layton <jlayton at redhat.com>

This patch converts most of the in-kernel filesystems that do writeback
out of the pagecache to report errors using the errseq_t-based
infrastructure that was recently added. This allows them to report
errors once for each open file description.

Most filesystems have a fairly straightforward fsync operation. They
call filemap_write_and_wait_range to write back all of the data and
wait on it, and then (sometimes) sync out the metadata.

For those filesystems this is a straightforward conversion from calling
filemap_write_and_wait_range in their fsync operation to calling
file_write_and_wait_range.

Signed-off-by: Jeff Layton <jlayton at redhat.com>
---
 arch/powerpc/platforms/cell/spufs/file.c   | 2 +-
 drivers/staging/lustre/lustre/llite/file.c | 2 +-
 drivers/video/fbdev/core/fb_defio.c        | 2 +-
 fs/9p/vfs_file.c                           | 4 ++--
 fs/affs/file.c                             | 2 +-
 fs/afs/write.c                             | 2 +-
 fs/cifs/file.c                             | 4 ++--
 fs/exofs/file.c                            | 2 +-
 fs/f2fs/file.c                             | 2 +-
 fs/hfs/inode.c                             | 2 +-
 fs/hfsplus/inode.c                         | 2 +-
 fs/hostfs/hostfs_kern.c                    | 2 +-
 fs/hpfs/file.c                             | 2 +-
 fs/jffs2/file.c                            | 2 +-
 fs/jfs/file.c                              | 2 +-
 fs/ncpfs/file.c                            | 2 +-
 fs/ntfs/dir.c                              | 2 +-
 fs/ntfs/file.c                             | 2 +-
 fs/ocfs2/file.c                            | 2 +-
 fs/reiserfs/dir.c                          | 2 +-
 fs/reiserfs/file.c                         | 2 +-
 fs/ubifs/file.c                            | 2 +-
 22 files changed, 24 insertions(+), 24 deletions(-)

Rolling up all of these conversions into a single patch, as Christoph
Hellwig suggested. Most of these are not tested, but the conversion
here is fairly straightforward.

Any maintainers who object, please let me know and I'll yank that
part out of this patch.

diff --git a/arch/powerpc/platforms/cell/spufs/file.c b/arch/powerpc/platforms/cell/spufs/file.c
index ae2f740a82f1..5ffcdeb1eb17 100644
--- a/arch/powerpc/platforms/cell/spufs/file.c
+++ b/arch/powerpc/platforms/cell/spufs/file.c
@@ -1749,7 +1749,7 @@ static int spufs_mfc_flush(struct file *file, fl_owner_t id)
 static int spufs_mfc_fsync(struct file *file, loff_t start, loff_t end, int datasync)
 {
 	struct inode *inode = file_inode(file);
-	int err = filemap_write_and_wait_range(inode->i_mapping, start, end);
+	int err = file_write_and_wait_range(file, start, end);
 	if (!err) {
 		inode_lock(inode);
 		err = spufs_mfc_flush(file, NULL);
diff --git a/drivers/staging/lustre/lustre/llite/file.c b/drivers/staging/lustre/lustre/llite/file.c
index ab1c85c1ed38..f7d07735ac66 100644
--- a/drivers/staging/lustre/lustre/llite/file.c
+++ b/drivers/staging/lustre/lustre/llite/file.c
@@ -2364,7 +2364,7 @@ int ll_fsync(struct file *file, loff_t start, loff_t end, int datasync)
 	       PFID(ll_inode2fid(inode)), inode);
 	ll_stats_ops_tally(ll_i2sbi(inode), LPROC_LL_FSYNC, 1);
 
-	rc = filemap_write_and_wait_range(inode->i_mapping, start, end);
+	rc = file_write_and_wait_range(file, start, end);
 	inode_lock(inode);
 
 	/* catch async errors that were recorded back when async writeback
diff --git a/drivers/video/fbdev/core/fb_defio.c b/drivers/video/fbdev/core/fb_defio.c
index 37f69c061210..487d5e336e1b 100644
--- a/drivers/video/fbdev/core/fb_defio.c
+++ b/drivers/video/fbdev/core/fb_defio.c
@@ -69,7 +69,7 @@ int fb_deferred_io_fsync(struct file *file, loff_t start, loff_t end, int datasy
 {
 	struct fb_info *info = file->private_data;
 	struct inode *inode = file_inode(file);
-	int err = filemap_write_and_wait_range(inode->i_mapping, start, end);
+	int err = file_write_and_wait_range(file, start, end);
 	if (err)
 		return err;
 
diff --git a/fs/9p/vfs_file.c b/fs/9p/vfs_file.c
index 3de3b4a89d89..4802d75b3cf7 100644
--- a/fs/9p/vfs_file.c
+++ b/fs/9p/vfs_file.c
@@ -445,7 +445,7 @@ static int v9fs_file_fsync(struct file *filp, loff_t start, loff_t end,
 	struct p9_wstat wstat;
 	int retval;
 
-	retval = filemap_write_and_wait_range(inode->i_mapping, start, end);
+	retval = file_write_and_wait_range(filp, start, end);
 	if (retval)
 		return retval;
 
@@ -468,7 +468,7 @@ int v9fs_file_fsync_dotl(struct file *filp, loff_t start, loff_t end,
 	struct inode *inode = filp->f_mapping->host;
 	int retval;
 
-	retval = filemap_write_and_wait_range(inode->i_mapping, start, end);
+	retval = file_write_and_wait_range(filp, start, end);
 	if (retval)
 		return retval;
 
diff --git a/fs/affs/file.c b/fs/affs/file.c
index 196ee7f6fdc4..00331810f690 100644
--- a/fs/affs/file.c
+++ b/fs/affs/file.c
@@ -954,7 +954,7 @@ int affs_file_fsync(struct file *filp, loff_t start, loff_t end, int datasync)
 	struct inode *inode = filp->f_mapping->host;
 	int ret, err;
 
-	err = filemap_write_and_wait_range(inode->i_mapping, start, end);
+	err = file_write_and_wait_range(filp, start, end);
 	if (err)
 		return err;
 
diff --git a/fs/afs/write.c b/fs/afs/write.c
index 2d2fccd5044b..106e43db1115 100644
--- a/fs/afs/write.c
+++ b/fs/afs/write.c
@@ -714,7 +714,7 @@ int afs_fsync(struct file *file, loff_t start, loff_t end, int datasync)
 	       vnode->fid.vid, vnode->fid.vnode, file,
 	       datasync);
 
-	ret = filemap_write_and_wait_range(inode->i_mapping, start, end);
+	ret = file_write_and_wait_range(file, start, end);
 	if (ret)
 		return ret;
 	inode_lock(inode);
diff --git a/fs/cifs/file.c b/fs/cifs/file.c
index bc09df6b473a..0786f19d288f 100644
--- a/fs/cifs/file.c
+++ b/fs/cifs/file.c
@@ -2329,7 +2329,7 @@ int cifs_strict_fsync(struct file *file, loff_t start, loff_t end,
 	struct inode *inode = file_inode(file);
 	struct cifs_sb_info *cifs_sb = CIFS_SB(inode->i_sb);
 
-	rc = filemap_write_and_wait_range(inode->i_mapping, start, end);
+	rc = file_write_and_wait_range(file, start, end);
 	if (rc)
 		return rc;
 	inode_lock(inode);
@@ -2371,7 +2371,7 @@ int cifs_fsync(struct file *file, loff_t start, loff_t end, int datasync)
 	struct cifs_sb_info *cifs_sb = CIFS_FILE_SB(file);
 	struct inode *inode = file->f_mapping->host;
 
-	rc = filemap_write_and_wait_range(inode->i_mapping, start, end);
+	rc = file_write_and_wait_range(file, start, end);
 	if (rc)
 		return rc;
 	inode_lock(inode);
diff --git a/fs/exofs/file.c b/fs/exofs/file.c
index 28645f0640f7..a94594ea2aa3 100644
--- a/fs/exofs/file.c
+++ b/fs/exofs/file.c
@@ -48,7 +48,7 @@ static int exofs_file_fsync(struct file *filp, loff_t start, loff_t end,
 	struct inode *inode = filp->f_mapping->host;
 	int ret;
 
-	ret = filemap_write_and_wait_range(inode->i_mapping, start, end);
+	ret = file_write_and_wait_range(filp, start, end);
 	if (ret)
 		return ret;
 
diff --git a/fs/f2fs/file.c b/fs/f2fs/file.c
index a0e6d2c65a9e..40fb3d4bb9c2 100644
--- a/fs/f2fs/file.c
+++ b/fs/f2fs/file.c
@@ -206,7 +206,7 @@ static int f2fs_do_sync_file(struct file *file, loff_t start, loff_t end,
 	/* if fdatasync is triggered, let's do in-place-update */
 	if (datasync || get_dirty_pages(inode) <= SM_I(sbi)->min_fsync_blocks)
 		set_inode_flag(inode, FI_NEED_IPU);
-	ret = filemap_write_and_wait_range(inode->i_mapping, start, end);
+	ret = file_write_and_wait_range(file, start, end);
 	clear_inode_flag(inode, FI_NEED_IPU);
 
 	if (ret) {
diff --git a/fs/hfs/inode.c b/fs/hfs/inode.c
index bfbba799430f..2538b49cc349 100644
--- a/fs/hfs/inode.c
+++ b/fs/hfs/inode.c
@@ -656,7 +656,7 @@ static int hfs_file_fsync(struct file *filp, loff_t start, loff_t end,
 	struct super_block * sb;
 	int ret, err;
 
-	ret = filemap_write_and_wait_range(inode->i_mapping, start, end);
+	ret = file_write_and_wait_range(filp, start, end);
 	if (ret)
 		return ret;
 	inode_lock(inode);
diff --git a/fs/hfsplus/inode.c b/fs/hfsplus/inode.c
index e8638d528195..4f26b6877130 100644
--- a/fs/hfsplus/inode.c
+++ b/fs/hfsplus/inode.c
@@ -283,7 +283,7 @@ int hfsplus_file_fsync(struct file *file, loff_t start, loff_t end,
 	struct hfsplus_sb_info *sbi = HFSPLUS_SB(inode->i_sb);
 	int error = 0, error2;
 
-	error = filemap_write_and_wait_range(inode->i_mapping, start, end);
+	error = file_write_and_wait_range(file, start, end);
 	if (error)
 		return error;
 	inode_lock(inode);
diff --git a/fs/hostfs/hostfs_kern.c b/fs/hostfs/hostfs_kern.c
index e61261a7417e..c148e7f4f451 100644
--- a/fs/hostfs/hostfs_kern.c
+++ b/fs/hostfs/hostfs_kern.c
@@ -374,7 +374,7 @@ static int hostfs_fsync(struct file *file, loff_t start, loff_t end,
 	struct inode *inode = file->f_mapping->host;
 	int ret;
 
-	ret = filemap_write_and_wait_range(inode->i_mapping, start, end);
+	ret = file_write_and_wait_range(file, start, end);
 	if (ret)
 		return ret;
 
diff --git a/fs/hpfs/file.c b/fs/hpfs/file.c
index b3be1b5a62e2..f26138425b16 100644
--- a/fs/hpfs/file.c
+++ b/fs/hpfs/file.c
@@ -24,7 +24,7 @@ int hpfs_file_fsync(struct file *file, loff_t start, loff_t end, int datasync)
 	struct inode *inode = file->f_mapping->host;
 	int ret;
 
-	ret = filemap_write_and_wait_range(file->f_mapping, start, end);
+	ret = file_write_and_wait_range(file, start, end);
 	if (ret)
 		return ret;
 	return sync_blockdev(inode->i_sb->s_bdev);
diff --git a/fs/jffs2/file.c b/fs/jffs2/file.c
index c12476e309c6..bd0428bebe9b 100644
--- a/fs/jffs2/file.c
+++ b/fs/jffs2/file.c
@@ -35,7 +35,7 @@ int jffs2_fsync(struct file *filp, loff_t start, loff_t end, int datasync)
 	struct jffs2_sb_info *c = JFFS2_SB_INFO(inode->i_sb);
 	int ret;
 
-	ret = filemap_write_and_wait_range(inode->i_mapping, start, end);
+	ret = file_write_and_wait_range(filp, start, end);
 	if (ret)
 		return ret;
 
diff --git a/fs/jfs/file.c b/fs/jfs/file.c
index 739492c7a3fd..36665fd37095 100644
--- a/fs/jfs/file.c
+++ b/fs/jfs/file.c
@@ -34,7 +34,7 @@ int jfs_fsync(struct file *file, loff_t start, loff_t end, int datasync)
 	struct inode *inode = file->f_mapping->host;
 	int rc = 0;
 
-	rc = filemap_write_and_wait_range(inode->i_mapping, start, end);
+	rc = file_write_and_wait_range(file, start, end);
 	if (rc)
 		return rc;
 
diff --git a/fs/ncpfs/file.c b/fs/ncpfs/file.c
index 76965e772264..a06c07619ee6 100644
--- a/fs/ncpfs/file.c
+++ b/fs/ncpfs/file.c
@@ -23,7 +23,7 @@
 
 static int ncp_fsync(struct file *file, loff_t start, loff_t end, int datasync)
 {
-	return filemap_write_and_wait_range(file->f_mapping, start, end);
+	return file_write_and_wait_range(file, start, end);
 }
 
 /*
diff --git a/fs/ntfs/dir.c b/fs/ntfs/dir.c
index 0ee19ecc982d..1a24be9e8405 100644
--- a/fs/ntfs/dir.c
+++ b/fs/ntfs/dir.c
@@ -1506,7 +1506,7 @@ static int ntfs_dir_fsync(struct file *filp, loff_t start, loff_t end,
 
 	ntfs_debug("Entering for inode 0x%lx.", vi->i_ino);
 
-	err = filemap_write_and_wait_range(vi->i_mapping, start, end);
+	err = file_write_and_wait_range(filp, start, end);
 	if (err)
 		return err;
 	inode_lock(vi);
diff --git a/fs/ntfs/file.c b/fs/ntfs/file.c
index c4f68c338735..331910fa8442 100644
--- a/fs/ntfs/file.c
+++ b/fs/ntfs/file.c
@@ -1989,7 +1989,7 @@ static int ntfs_file_fsync(struct file *filp, loff_t start, loff_t end,
 
 	ntfs_debug("Entering for inode 0x%lx.", vi->i_ino);
 
-	err = filemap_write_and_wait_range(vi->i_mapping, start, end);
+	err = file_write_and_wait_range(filp, start, end);
 	if (err)
 		return err;
 	inode_lock(vi);
diff --git a/fs/ocfs2/file.c b/fs/ocfs2/file.c
index bfeb647459d9..66e59d3163ea 100644
--- a/fs/ocfs2/file.c
+++ b/fs/ocfs2/file.c
@@ -196,7 +196,7 @@ static int ocfs2_sync_file(struct file *file, loff_t start, loff_t end,
 	if (ocfs2_is_hard_readonly(osb) || ocfs2_is_soft_readonly(osb))
 		return -EROFS;
 
-	err = filemap_write_and_wait_range(inode->i_mapping, start, end);
+	err = file_write_and_wait_range(file, start, end);
 	if (err)
 		return err;
 
diff --git a/fs/reiserfs/dir.c b/fs/reiserfs/dir.c
index 45aa05e2232f..5b50689d8539 100644
--- a/fs/reiserfs/dir.c
+++ b/fs/reiserfs/dir.c
@@ -34,7 +34,7 @@ static int reiserfs_dir_fsync(struct file *filp, loff_t start, loff_t end,
 	struct inode *inode = filp->f_mapping->host;
 	int err;
 
-	err = filemap_write_and_wait_range(inode->i_mapping, start, end);
+	err = file_write_and_wait_range(filp, start, end);
 	if (err)
 		return err;
 
diff --git a/fs/reiserfs/file.c b/fs/reiserfs/file.c
index b396eb09f288..843aadcc123c 100644
--- a/fs/reiserfs/file.c
+++ b/fs/reiserfs/file.c
@@ -154,7 +154,7 @@ static int reiserfs_sync_file(struct file *filp, loff_t start, loff_t end,
 	int err;
 	int barrier_done;
 
-	err = filemap_write_and_wait_range(inode->i_mapping, start, end);
+	err = file_write_and_wait_range(filp, start, end);
 	if (err)
 		return err;
 
diff --git a/fs/ubifs/file.c b/fs/ubifs/file.c
index 8cad0b19b404..f90a466ea5db 100644
--- a/fs/ubifs/file.c
+++ b/fs/ubifs/file.c
@@ -1337,7 +1337,7 @@ int ubifs_fsync(struct file *file, loff_t start, loff_t end, int datasync)
 		 */
 		return 0;
 
-	err = filemap_write_and_wait_range(inode->i_mapping, start, end);
+	err = file_write_and_wait_range(file, start, end);
 	if (err)
 		return err;
 	inode_lock(inode);
-- 
2.13.3

From paf at cray.com  Fri Jul 28 16:53:20 2017
From: paf at cray.com (Patrick Farrell)
Date: Fri, 28 Jul 2017 16:53:20 +0000
Subject: [lustre-devel] Design proposal for client-side compression
In-Reply-To: <1501254736.3412.16.camel@informatik.uni-hamburg.de>
References: <1483967258.2622.24.camel@informatik.uni-hamburg.de>
	<B5E18A82-79B3-484F-BEBE-998B8AE56397@intel.com>
	<1484223328.3937.60.camel@informatik.uni-hamburg.de>
	<B924BE6D-3C6A-4BA4-96F9-AC053F032B2D@intel.com>
	<1484749185.3965.114.camel@informatik.uni-hamburg.de>
	<1487254536.4777.51.camel@informatik.uni-hamburg.de>
	,<1500650130.3158.6.camel@informatik.uni-hamburg.de>
	<BN6PR1101MB2132311AB19B104AD044797ACBA40@BN6PR1101MB2132.namprd11.prod.outlook.com>
	,<1500992740.4728.5.camel@informatik.uni-hamburg.de>
	<BN6PR1101MB2132DD0CCFF2F83BB3BE7F54CBB90@BN6PR1101MB2132.namprd11.prod.outlook.com>
	,<1501143960.3412.3.camel@informatik.uni-hamburg.de>
	<BN6PR1101MB2132BBFB8979EA3FBFA4AE2DCBBE0@BN6PR1101MB2132.namprd11.prod.outlook.com>
	,<1501235847.3412.12.camel@informatik.uni-hamburg.de>
	<BN6PR1101MB2132018DF6EED212C690693FCBBF0@BN6PR1101MB2132.namprd11.prod.outlook.com>,
	<1501254736.3412.16.camel@informatik.uni-hamburg.de>
Message-ID: <BN6PR1101MB2132360AE4E81FD3597240F7CBBF0@BN6PR1101MB2132.namprd11.prod.outlook.com>

Ah, OK.  Reading this, I understand now that your intention is to keep the data compressed on disk - I hadn't thought through the implications of that fully.  There's obviously a lot of benefit from that.

That said, it seems like it would be relatively straightforward to make a version of this that uncompressed the data on arrival at the server, simply unpacking that buffer before writing it to disk.  (Straightforward, that is, once the actual compression/decompression code is ready...)

That obviously takes more CPU on the server side and does not reduce the space required, but...


If you don't mind, when you consider the performant version of the compression code to be ready for at least testing, I'd like to see the code so I can try out the on-the-wire-only compression idea.  It might have significant benefits for a case of interest to me, and if it worked well, it could (long term) probably coexist with the larger on-disk compression idea.  (Since who knows if we'll ever implement the whole thing for ldiskfs.)


Thanks again for engaging with me on this.


- Patrick

________________________________
From: Anna Fuchs <anna.fuchs at informatik.uni-hamburg.de>
Sent: Friday, July 28, 2017 10:12:16 AM
To: Patrick Farrell; Xiong, Jinshan
Cc: Matthew Ahrens; Zhuravlev, Alexey; lustre-devel
Subject: Re: [lustre-devel] Design proposal for client-side compression


> Ah.  As it turns out, much more complicated than I anticipated.
>  Thanks for explaining...
>
> I have no expertise in compression algorithms, so that I will have to
> just watch from the sidelines.  Good luck.
>
> When you are further along, I remain interested in helping out with
> the Lustre side of things.
>
> One more question - Do you have a plan to make this work *without*
> the ZFS integration as well, for those using ldiskfs?  That seems
> straightforward enough - compress/decompress at send and recieve time
> - even if the benefits would be smaller, but not everyone (Cray,
> f.x.) is using ZFS, so I'm very interested in something that would
> help ldiskfs as well.  (Which is not to say don't do the deeper
> integration with ZFS.  Just that we'd like something available for
> ldiskfs too.)

I fear it is also much more complicated :)

At the very beginning of the project proposal we hoped we wouldn't need
to touch the server so much. It turned out wrong, moreover we have to
modify not only the Lustre server, but also pretty much the backend
itself. We chose ZFS since it already provides a lot of infrastructure
that we would need to implement completely new in ldiskfs. Since, at
least for me, it is a research project, ldiskfs is out of scope. Once
we proved the concept, one could re-implement the whole compression
stack for ldiskfs. So it is not impossible, but not our focus for this
project.

Nevertheless we tried to keep our changes as far as possible not very
backend specific. For example we need some additional information to be
stored per compressed chunk. One possibility would be to change the
block pointer of ZFS and add those fields, but I don't think anyone
except of us would like the BP to be modified :) So we decided to store
them as a header for every chunk. For ldiskfs, since one would need to
implement everything from scratch anyway, one might not need that
header, but take the required fields into account from the beginning
and add them to ldiskfs' "block pointer". For that reason, we wanted to
leave the compressed data "headerless" on client-side, and add the
header only on the server side if the corresponding backend requires
it.

Well, we did it, and it even works sometimes, but it looks horrible
and is really counterintuitive. We send less data from client than
lands on the OST, recalculate offsets, since we add the header during
receiving on server side, recalculate the sent and received sizes,
shift buffers by offsets and so on. The only advantage of this approach
is client's independence from backend. We decided the price is too
high. So now, I will construct the chunk with the header just after
compressing the data on client-side, get rid of all those offset stuff
on the server. But ldiskfs will have to deal with that ZFS-motivated
details.

However, a light version of compression could work with smaller changes
to ldiskfs, when we only allow a completely compressed or not
compressed files and allow potential performance drops for broken read-
ahead (due to gaps within the data).

Hope it is somehow more clear now.

Regards,
Anna

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-devel_lists.lustre.org/attachments/20170728/9c1db15e/attachment.html>

From dmitriy at oss-tech.org  Sat Jul 29 00:13:43 2017
From: dmitriy at oss-tech.org (Dmitriy Cherkasov)
Date: Fri, 28 Jul 2017 17:13:43 -0700
Subject: [lustre-devel] [PATCH] staging: lustre: lov: remove dead code
Message-ID: <1501287223-18570-1-git-send-email-dmitriy@oss-tech.org>

This #if 0 block has been commented out for years. Assume it is not
needed and remove it.

Signed-off-by: Dmitriy Cherkasov <dmitriy at oss-tech.org>
---
 drivers/staging/lustre/lustre/lov/lov_io.c | 6 ------
 1 file changed, 6 deletions(-)

diff --git a/drivers/staging/lustre/lustre/lov/lov_io.c b/drivers/staging/lustre/lustre/lov/lov_io.c
index babf39a..0fb5723 100644
--- a/drivers/staging/lustre/lustre/lov/lov_io.c
+++ b/drivers/staging/lustre/lustre/lov/lov_io.c
@@ -939,12 +939,6 @@ static void lov_empty_impossible(const struct lu_env *env,
 	.op = {
 		[CIT_READ] = {
 			.cio_fini       = lov_empty_io_fini,
-#if 0
-			.cio_iter_init  = LOV_EMPTY_IMPOSSIBLE,
-			.cio_lock       = LOV_EMPTY_IMPOSSIBLE,
-			.cio_start      = LOV_EMPTY_IMPOSSIBLE,
-			.cio_end	= LOV_EMPTY_IMPOSSIBLE
-#endif
 		},
 		[CIT_WRITE] = {
 			.cio_fini      = lov_empty_io_fini,
-- 
1.9.1

From gregkh at linuxfoundation.org  Sun Jul 30 15:04:25 2017
From: gregkh at linuxfoundation.org (Greg Kroah-Hartman)
Date: Sun, 30 Jul 2017 08:04:25 -0700
Subject: [lustre-devel] [PATCH 18/20] staging: lustre: llite: Remove
 filtering of seclabel xattr
In-Reply-To: <alpine.LFD.2.20.1707272034380.6670@casper.infradead.org>
References: <1501082556-17962-1-git-send-email-jsimmons@infradead.org>
	<1501082556-17962-19-git-send-email-jsimmons@infradead.org>
	<alpine.LFD.2.20.1707272034380.6670@casper.infradead.org>
Message-ID: <20170730150425.GA25625@kroah.com>

On Thu, Jul 27, 2017 at 08:35:33PM +0100, James Simmons wrote:
> 
> > From: Robin Humble <plaguedbypenguins at gmail.com>
> > 
> > The security.capability xattr is used to implement File
> > Capabilities in recent Linux versions. Capabilities are a
> > fine grained approach to granting executables elevated
> > privileges. eg. /bin/ping can have capabilities
> > cap_net_admin, cap_net_raw+ep instead of being setuid root.
> > 
> > This xattr has long been filtered out by llite, initially for
> > stability reasons (b15587), and later over performance
> > concerns as this xattr is read for every file with eg.
> > 'ls --color'. Since LU-2869 xattr's are cached on clients,
> > alleviating most performance concerns.
> > 
> > Removing llite's filtering of the security.capability xattr
> > enables using Lustre as a root filesystem, which is used on
> > some large clusters.
> 
> The commit message for this patch is incorrect. Some how it got
> mixed up with another patch which I missed in this push. Please
> drop this patch and I will resent the correct patches later.

Now dropped.

From jack at suse.cz  Mon Jul 31 07:25:17 2017
From: jack at suse.cz (Jan Kara)
Date: Mon, 31 Jul 2017 09:25:17 +0200
Subject: [lustre-devel] [PATCH] fs: convert a pile of fsync routines to
 errseq_t based reporting
In-Reply-To: <20170728142321.11492-1-jlayton@kernel.org>
References: <20170728142321.11492-1-jlayton@kernel.org>
Message-ID: <20170731072517.GB18621@quack2.suse.cz>

On Fri 28-07-17 10:23:21, Jeff Layton wrote:
> From: Jeff Layton <jlayton at redhat.com>
> 
> This patch converts most of the in-kernel filesystems that do writeback
> out of the pagecache to report errors using the errseq_t-based
> infrastructure that was recently added. This allows them to report
> errors once for each open file description.
> 
> Most filesystems have a fairly straightforward fsync operation. They
> call filemap_write_and_wait_range to write back all of the data and
> wait on it, and then (sometimes) sync out the metadata.
> 
> For those filesystems this is a straightforward conversion from calling
> filemap_write_and_wait_range in their fsync operation to calling
> file_write_and_wait_range.
> 
> Signed-off-by: Jeff Layton <jlayton at redhat.com>

This all looks rather obvious. Feel free to add:

Acked-by: Jan Kara <jack at suse.cz>

								Honza


> ---
>  arch/powerpc/platforms/cell/spufs/file.c   | 2 +-
>  drivers/staging/lustre/lustre/llite/file.c | 2 +-
>  drivers/video/fbdev/core/fb_defio.c        | 2 +-
>  fs/9p/vfs_file.c                           | 4 ++--
>  fs/affs/file.c                             | 2 +-
>  fs/afs/write.c                             | 2 +-
>  fs/cifs/file.c                             | 4 ++--
>  fs/exofs/file.c                            | 2 +-
>  fs/f2fs/file.c                             | 2 +-
>  fs/hfs/inode.c                             | 2 +-
>  fs/hfsplus/inode.c                         | 2 +-
>  fs/hostfs/hostfs_kern.c                    | 2 +-
>  fs/hpfs/file.c                             | 2 +-
>  fs/jffs2/file.c                            | 2 +-
>  fs/jfs/file.c                              | 2 +-
>  fs/ncpfs/file.c                            | 2 +-
>  fs/ntfs/dir.c                              | 2 +-
>  fs/ntfs/file.c                             | 2 +-
>  fs/ocfs2/file.c                            | 2 +-
>  fs/reiserfs/dir.c                          | 2 +-
>  fs/reiserfs/file.c                         | 2 +-
>  fs/ubifs/file.c                            | 2 +-
>  22 files changed, 24 insertions(+), 24 deletions(-)
> 
> Rolling up all of these conversions into a single patch, as Christoph
> Hellwig suggested. Most of these are not tested, but the conversion
> here is fairly straightforward.
> 
> Any maintainers who object, please let me know and I'll yank that
> part out of this patch.
> 
> diff --git a/arch/powerpc/platforms/cell/spufs/file.c b/arch/powerpc/platforms/cell/spufs/file.c
> index ae2f740a82f1..5ffcdeb1eb17 100644
> --- a/arch/powerpc/platforms/cell/spufs/file.c
> +++ b/arch/powerpc/platforms/cell/spufs/file.c
> @@ -1749,7 +1749,7 @@ static int spufs_mfc_flush(struct file *file, fl_owner_t id)
>  static int spufs_mfc_fsync(struct file *file, loff_t start, loff_t end, int datasync)
>  {
>  	struct inode *inode = file_inode(file);
> -	int err = filemap_write_and_wait_range(inode->i_mapping, start, end);
> +	int err = file_write_and_wait_range(file, start, end);
>  	if (!err) {
>  		inode_lock(inode);
>  		err = spufs_mfc_flush(file, NULL);
> diff --git a/drivers/staging/lustre/lustre/llite/file.c b/drivers/staging/lustre/lustre/llite/file.c
> index ab1c85c1ed38..f7d07735ac66 100644
> --- a/drivers/staging/lustre/lustre/llite/file.c
> +++ b/drivers/staging/lustre/lustre/llite/file.c
> @@ -2364,7 +2364,7 @@ int ll_fsync(struct file *file, loff_t start, loff_t end, int datasync)
>  	       PFID(ll_inode2fid(inode)), inode);
>  	ll_stats_ops_tally(ll_i2sbi(inode), LPROC_LL_FSYNC, 1);
>  
> -	rc = filemap_write_and_wait_range(inode->i_mapping, start, end);
> +	rc = file_write_and_wait_range(file, start, end);
>  	inode_lock(inode);
>  
>  	/* catch async errors that were recorded back when async writeback
> diff --git a/drivers/video/fbdev/core/fb_defio.c b/drivers/video/fbdev/core/fb_defio.c
> index 37f69c061210..487d5e336e1b 100644
> --- a/drivers/video/fbdev/core/fb_defio.c
> +++ b/drivers/video/fbdev/core/fb_defio.c
> @@ -69,7 +69,7 @@ int fb_deferred_io_fsync(struct file *file, loff_t start, loff_t end, int datasy
>  {
>  	struct fb_info *info = file->private_data;
>  	struct inode *inode = file_inode(file);
> -	int err = filemap_write_and_wait_range(inode->i_mapping, start, end);
> +	int err = file_write_and_wait_range(file, start, end);
>  	if (err)
>  		return err;
>  
> diff --git a/fs/9p/vfs_file.c b/fs/9p/vfs_file.c
> index 3de3b4a89d89..4802d75b3cf7 100644
> --- a/fs/9p/vfs_file.c
> +++ b/fs/9p/vfs_file.c
> @@ -445,7 +445,7 @@ static int v9fs_file_fsync(struct file *filp, loff_t start, loff_t end,
>  	struct p9_wstat wstat;
>  	int retval;
>  
> -	retval = filemap_write_and_wait_range(inode->i_mapping, start, end);
> +	retval = file_write_and_wait_range(filp, start, end);
>  	if (retval)
>  		return retval;
>  
> @@ -468,7 +468,7 @@ int v9fs_file_fsync_dotl(struct file *filp, loff_t start, loff_t end,
>  	struct inode *inode = filp->f_mapping->host;
>  	int retval;
>  
> -	retval = filemap_write_and_wait_range(inode->i_mapping, start, end);
> +	retval = file_write_and_wait_range(filp, start, end);
>  	if (retval)
>  		return retval;
>  
> diff --git a/fs/affs/file.c b/fs/affs/file.c
> index 196ee7f6fdc4..00331810f690 100644
> --- a/fs/affs/file.c
> +++ b/fs/affs/file.c
> @@ -954,7 +954,7 @@ int affs_file_fsync(struct file *filp, loff_t start, loff_t end, int datasync)
>  	struct inode *inode = filp->f_mapping->host;
>  	int ret, err;
>  
> -	err = filemap_write_and_wait_range(inode->i_mapping, start, end);
> +	err = file_write_and_wait_range(filp, start, end);
>  	if (err)
>  		return err;
>  
> diff --git a/fs/afs/write.c b/fs/afs/write.c
> index 2d2fccd5044b..106e43db1115 100644
> --- a/fs/afs/write.c
> +++ b/fs/afs/write.c
> @@ -714,7 +714,7 @@ int afs_fsync(struct file *file, loff_t start, loff_t end, int datasync)
>  	       vnode->fid.vid, vnode->fid.vnode, file,
>  	       datasync);
>  
> -	ret = filemap_write_and_wait_range(inode->i_mapping, start, end);
> +	ret = file_write_and_wait_range(file, start, end);
>  	if (ret)
>  		return ret;
>  	inode_lock(inode);
> diff --git a/fs/cifs/file.c b/fs/cifs/file.c
> index bc09df6b473a..0786f19d288f 100644
> --- a/fs/cifs/file.c
> +++ b/fs/cifs/file.c
> @@ -2329,7 +2329,7 @@ int cifs_strict_fsync(struct file *file, loff_t start, loff_t end,
>  	struct inode *inode = file_inode(file);
>  	struct cifs_sb_info *cifs_sb = CIFS_SB(inode->i_sb);
>  
> -	rc = filemap_write_and_wait_range(inode->i_mapping, start, end);
> +	rc = file_write_and_wait_range(file, start, end);
>  	if (rc)
>  		return rc;
>  	inode_lock(inode);
> @@ -2371,7 +2371,7 @@ int cifs_fsync(struct file *file, loff_t start, loff_t end, int datasync)
>  	struct cifs_sb_info *cifs_sb = CIFS_FILE_SB(file);
>  	struct inode *inode = file->f_mapping->host;
>  
> -	rc = filemap_write_and_wait_range(inode->i_mapping, start, end);
> +	rc = file_write_and_wait_range(file, start, end);
>  	if (rc)
>  		return rc;
>  	inode_lock(inode);
> diff --git a/fs/exofs/file.c b/fs/exofs/file.c
> index 28645f0640f7..a94594ea2aa3 100644
> --- a/fs/exofs/file.c
> +++ b/fs/exofs/file.c
> @@ -48,7 +48,7 @@ static int exofs_file_fsync(struct file *filp, loff_t start, loff_t end,
>  	struct inode *inode = filp->f_mapping->host;
>  	int ret;
>  
> -	ret = filemap_write_and_wait_range(inode->i_mapping, start, end);
> +	ret = file_write_and_wait_range(filp, start, end);
>  	if (ret)
>  		return ret;
>  
> diff --git a/fs/f2fs/file.c b/fs/f2fs/file.c
> index a0e6d2c65a9e..40fb3d4bb9c2 100644
> --- a/fs/f2fs/file.c
> +++ b/fs/f2fs/file.c
> @@ -206,7 +206,7 @@ static int f2fs_do_sync_file(struct file *file, loff_t start, loff_t end,
>  	/* if fdatasync is triggered, let's do in-place-update */
>  	if (datasync || get_dirty_pages(inode) <= SM_I(sbi)->min_fsync_blocks)
>  		set_inode_flag(inode, FI_NEED_IPU);
> -	ret = filemap_write_and_wait_range(inode->i_mapping, start, end);
> +	ret = file_write_and_wait_range(file, start, end);
>  	clear_inode_flag(inode, FI_NEED_IPU);
>  
>  	if (ret) {
> diff --git a/fs/hfs/inode.c b/fs/hfs/inode.c
> index bfbba799430f..2538b49cc349 100644
> --- a/fs/hfs/inode.c
> +++ b/fs/hfs/inode.c
> @@ -656,7 +656,7 @@ static int hfs_file_fsync(struct file *filp, loff_t start, loff_t end,
>  	struct super_block * sb;
>  	int ret, err;
>  
> -	ret = filemap_write_and_wait_range(inode->i_mapping, start, end);
> +	ret = file_write_and_wait_range(filp, start, end);
>  	if (ret)
>  		return ret;
>  	inode_lock(inode);
> diff --git a/fs/hfsplus/inode.c b/fs/hfsplus/inode.c
> index e8638d528195..4f26b6877130 100644
> --- a/fs/hfsplus/inode.c
> +++ b/fs/hfsplus/inode.c
> @@ -283,7 +283,7 @@ int hfsplus_file_fsync(struct file *file, loff_t start, loff_t end,
>  	struct hfsplus_sb_info *sbi = HFSPLUS_SB(inode->i_sb);
>  	int error = 0, error2;
>  
> -	error = filemap_write_and_wait_range(inode->i_mapping, start, end);
> +	error = file_write_and_wait_range(file, start, end);
>  	if (error)
>  		return error;
>  	inode_lock(inode);
> diff --git a/fs/hostfs/hostfs_kern.c b/fs/hostfs/hostfs_kern.c
> index e61261a7417e..c148e7f4f451 100644
> --- a/fs/hostfs/hostfs_kern.c
> +++ b/fs/hostfs/hostfs_kern.c
> @@ -374,7 +374,7 @@ static int hostfs_fsync(struct file *file, loff_t start, loff_t end,
>  	struct inode *inode = file->f_mapping->host;
>  	int ret;
>  
> -	ret = filemap_write_and_wait_range(inode->i_mapping, start, end);
> +	ret = file_write_and_wait_range(file, start, end);
>  	if (ret)
>  		return ret;
>  
> diff --git a/fs/hpfs/file.c b/fs/hpfs/file.c
> index b3be1b5a62e2..f26138425b16 100644
> --- a/fs/hpfs/file.c
> +++ b/fs/hpfs/file.c
> @@ -24,7 +24,7 @@ int hpfs_file_fsync(struct file *file, loff_t start, loff_t end, int datasync)
>  	struct inode *inode = file->f_mapping->host;
>  	int ret;
>  
> -	ret = filemap_write_and_wait_range(file->f_mapping, start, end);
> +	ret = file_write_and_wait_range(file, start, end);
>  	if (ret)
>  		return ret;
>  	return sync_blockdev(inode->i_sb->s_bdev);
> diff --git a/fs/jffs2/file.c b/fs/jffs2/file.c
> index c12476e309c6..bd0428bebe9b 100644
> --- a/fs/jffs2/file.c
> +++ b/fs/jffs2/file.c
> @@ -35,7 +35,7 @@ int jffs2_fsync(struct file *filp, loff_t start, loff_t end, int datasync)
>  	struct jffs2_sb_info *c = JFFS2_SB_INFO(inode->i_sb);
>  	int ret;
>  
> -	ret = filemap_write_and_wait_range(inode->i_mapping, start, end);
> +	ret = file_write_and_wait_range(filp, start, end);
>  	if (ret)
>  		return ret;
>  
> diff --git a/fs/jfs/file.c b/fs/jfs/file.c
> index 739492c7a3fd..36665fd37095 100644
> --- a/fs/jfs/file.c
> +++ b/fs/jfs/file.c
> @@ -34,7 +34,7 @@ int jfs_fsync(struct file *file, loff_t start, loff_t end, int datasync)
>  	struct inode *inode = file->f_mapping->host;
>  	int rc = 0;
>  
> -	rc = filemap_write_and_wait_range(inode->i_mapping, start, end);
> +	rc = file_write_and_wait_range(file, start, end);
>  	if (rc)
>  		return rc;
>  
> diff --git a/fs/ncpfs/file.c b/fs/ncpfs/file.c
> index 76965e772264..a06c07619ee6 100644
> --- a/fs/ncpfs/file.c
> +++ b/fs/ncpfs/file.c
> @@ -23,7 +23,7 @@
>  
>  static int ncp_fsync(struct file *file, loff_t start, loff_t end, int datasync)
>  {
> -	return filemap_write_and_wait_range(file->f_mapping, start, end);
> +	return file_write_and_wait_range(file, start, end);
>  }
>  
>  /*
> diff --git a/fs/ntfs/dir.c b/fs/ntfs/dir.c
> index 0ee19ecc982d..1a24be9e8405 100644
> --- a/fs/ntfs/dir.c
> +++ b/fs/ntfs/dir.c
> @@ -1506,7 +1506,7 @@ static int ntfs_dir_fsync(struct file *filp, loff_t start, loff_t end,
>  
>  	ntfs_debug("Entering for inode 0x%lx.", vi->i_ino);
>  
> -	err = filemap_write_and_wait_range(vi->i_mapping, start, end);
> +	err = file_write_and_wait_range(filp, start, end);
>  	if (err)
>  		return err;
>  	inode_lock(vi);
> diff --git a/fs/ntfs/file.c b/fs/ntfs/file.c
> index c4f68c338735..331910fa8442 100644
> --- a/fs/ntfs/file.c
> +++ b/fs/ntfs/file.c
> @@ -1989,7 +1989,7 @@ static int ntfs_file_fsync(struct file *filp, loff_t start, loff_t end,
>  
>  	ntfs_debug("Entering for inode 0x%lx.", vi->i_ino);
>  
> -	err = filemap_write_and_wait_range(vi->i_mapping, start, end);
> +	err = file_write_and_wait_range(filp, start, end);
>  	if (err)
>  		return err;
>  	inode_lock(vi);
> diff --git a/fs/ocfs2/file.c b/fs/ocfs2/file.c
> index bfeb647459d9..66e59d3163ea 100644
> --- a/fs/ocfs2/file.c
> +++ b/fs/ocfs2/file.c
> @@ -196,7 +196,7 @@ static int ocfs2_sync_file(struct file *file, loff_t start, loff_t end,
>  	if (ocfs2_is_hard_readonly(osb) || ocfs2_is_soft_readonly(osb))
>  		return -EROFS;
>  
> -	err = filemap_write_and_wait_range(inode->i_mapping, start, end);
> +	err = file_write_and_wait_range(file, start, end);
>  	if (err)
>  		return err;
>  
> diff --git a/fs/reiserfs/dir.c b/fs/reiserfs/dir.c
> index 45aa05e2232f..5b50689d8539 100644
> --- a/fs/reiserfs/dir.c
> +++ b/fs/reiserfs/dir.c
> @@ -34,7 +34,7 @@ static int reiserfs_dir_fsync(struct file *filp, loff_t start, loff_t end,
>  	struct inode *inode = filp->f_mapping->host;
>  	int err;
>  
> -	err = filemap_write_and_wait_range(inode->i_mapping, start, end);
> +	err = file_write_and_wait_range(filp, start, end);
>  	if (err)
>  		return err;
>  
> diff --git a/fs/reiserfs/file.c b/fs/reiserfs/file.c
> index b396eb09f288..843aadcc123c 100644
> --- a/fs/reiserfs/file.c
> +++ b/fs/reiserfs/file.c
> @@ -154,7 +154,7 @@ static int reiserfs_sync_file(struct file *filp, loff_t start, loff_t end,
>  	int err;
>  	int barrier_done;
>  
> -	err = filemap_write_and_wait_range(inode->i_mapping, start, end);
> +	err = file_write_and_wait_range(filp, start, end);
>  	if (err)
>  		return err;
>  
> diff --git a/fs/ubifs/file.c b/fs/ubifs/file.c
> index 8cad0b19b404..f90a466ea5db 100644
> --- a/fs/ubifs/file.c
> +++ b/fs/ubifs/file.c
> @@ -1337,7 +1337,7 @@ int ubifs_fsync(struct file *file, loff_t start, loff_t end, int datasync)
>  		 */
>  		return 0;
>  
> -	err = filemap_write_and_wait_range(inode->i_mapping, start, end);
> +	err = file_write_and_wait_range(file, start, end);
>  	if (err)
>  		return err;
>  	inode_lock(inode);
> -- 
> 2.13.3
> 
-- 
Jan Kara <jack at suse.com>
SUSE Labs, CR

From anna.fuchs at informatik.uni-hamburg.de  Mon Jul 31 10:20:32 2017
From: anna.fuchs at informatik.uni-hamburg.de (Anna Fuchs)
Date: Mon, 31 Jul 2017 12:20:32 +0200
Subject: [lustre-devel] Design proposal for client-side compression
In-Reply-To: <BN6PR1101MB2132360AE4E81FD3597240F7CBBF0@BN6PR1101MB2132.namprd11.prod.outlook.com>
References: <1483967258.2622.24.camel@informatik.uni-hamburg.de>
	<B5E18A82-79B3-484F-BEBE-998B8AE56397@intel.com>
	<1484223328.3937.60.camel@informatik.uni-hamburg.de>
	<B924BE6D-3C6A-4BA4-96F9-AC053F032B2D@intel.com>
	<1484749185.3965.114.camel@informatik.uni-hamburg.de>
	<1487254536.4777.51.camel@informatik.uni-hamburg.de>
	,<1500650130.3158.6.camel@informatik.uni-hamburg.de>
	<BN6PR1101MB2132311AB19B104AD044797ACBA40@BN6PR1101MB2132.namprd11.prod.outlook.com>
	,<1500992740.4728.5.camel@informatik.uni-hamburg.de>
	<BN6PR1101MB2132DD0CCFF2F83BB3BE7F54CBB90@BN6PR1101MB2132.namprd11.prod.outlook.com>
	,<1501143960.3412.3.camel@informatik.uni-hamburg.de>
	<BN6PR1101MB2132BBFB8979EA3FBFA4AE2DCBBE0@BN6PR1101MB2132.namprd11.prod.outlook.com>
	,<1501235847.3412.12.camel@informatik.uni-hamburg.de>
	<BN6PR1101MB2132018DF6EED212C690693FCBBF0@BN6PR1101MB2132.namprd11.prod.outlook.com>
	,<1501254736.3412.16.camel@informatik.uni-hamburg.de>
	<BN6PR1101MB2132360AE4E81FD3597240F7CBBF0@BN6PR1101MB2132.namprd11.prod.outlook.com>
Message-ID: <1501496432.3864.1.camel@informatik.uni-hamburg.de>

On Fri, 2017-07-28 at 16:53 +0000, Patrick Farrell wrote:
> Ah, OK.  Reading this, I understand now that your intention is to
> keep the data compressed on disk - I hadn't thought through the
> implications of that fully.  There's obviously a lot of benefit from
> that.
> 
> That said, it seems like it would be relatively straightforward to
> make a version of this that uncompressed the data on arrival at the
> server, simply unpacking that buffer before writing it to disk. 
> (Straightforward, that is, once the actual compression/decompression
> code is ready...)

Sure, this should be easily doable independent from backend, although I
don't see many use cases when the efforts would pay off. 

> 
> That obviously takes more CPU on the server side and does not reduce
> the space required, but...
> 
> If you don't mind, when you consider the performant version of the
> compression code to be ready for at least testing, I'd like to see
> the code so I can try out the on-the-wire-only compression idea.  It
> might have significant benefits for a case of interest to me, and if
> it worked well, it could (long term) probably coexist with the larger
> on-disk compression idea.  (Since who knows if we'll ever implement
> the whole thing for ldiskfs.)

Sure, the client part should be stable very soon and I will share it. 

> Thanks again for engaging with me on this.
> 
> - Patrick

Same to you.

Anna
> From: Anna Fuchs <anna.fuchs at informatik.uni-hamburg.de>
> Sent: Friday, July 28, 2017 10:12:16 AM
> To: Patrick Farrell; Xiong, Jinshan
> Cc: Matthew Ahrens; Zhuravlev, Alexey; lustre-devel
> Subject: Re: [lustre-devel] Design proposal for client-side
> compression
>  
> 
> > Ah.  As it turns out, much more complicated than I anticipated.
> >  Thanks for explaining...
> > 
> > I have no expertise in compression algorithms, so that I will have
> to
> > just watch from the sidelines.  Good luck.
> > 
> > When you are further along, I remain interested in helping out with
> > the Lustre side of things.
> > 
> > One more question - Do you have a plan to make this work *without*
> > the ZFS integration as well, for those using ldiskfs?  That seems
> > straightforward enough - compress/decompress at send and recieve
> time
> > - even if the benefits would be smaller, but not everyone (Cray,
> > f.x.) is using ZFS, so I'm very interested in something that would
> > help ldiskfs as well.  (Which is not to say don't do the deeper
> > integration with ZFS.  Just that we'd like something available for
> > ldiskfs too.)
> 
> I fear it is also much more complicated :)
> 
> At the very beginning of the project proposal we hoped we wouldn't
> need
> to touch the server so much. It turned out wrong, moreover we have to
> modify not only the Lustre server, but also pretty much the backend
> itself. We chose ZFS since it already provides a lot of
> infrastructure
> that we would need to implement completely new in ldiskfs. Since, at
> least for me, it is a research project, ldiskfs is out of scope. Once
> we proved the concept, one could re-implement the whole compression
> stack for ldiskfs. So it is not impossible, but not our focus for
> this
> project. 
> 
> Nevertheless we tried to keep our changes as far as possible not very
> backend specific. For example we need some additional information to
> be
> stored per compressed chunk. One possibility would be to change the
> block pointer of ZFS and add those fields, but I don't think anyone
> except of us would like the BP to be modified :) So we decided to
> store
> them as a header for every chunk. For ldiskfs, since one would need
> to
> implement everything from scratch anyway, one might not need that
> header, but take the required fields into account from the beginning
> and add them to ldiskfs' "block pointer". For that reason, we wanted
> to
> leave the compressed data "headerless" on client-side, and add the
> header only on the server side if the corresponding backend requires
> it. 
> 
> Well, we did it, and it even works sometimes, but it looks horrible 
> and is really counterintuitive. We send less data from client than
> lands on the OST, recalculate offsets, since we add the header during
> receiving on server side, recalculate the sent and received sizes,
> shift buffers by offsets and so on. The only advantage of this
> approach
> is client's independence from backend. We decided the price is too
> high. So now, I will construct the chunk with the header just after
> compressing the data on client-side, get rid of all those offset
> stuff
> on the server. But ldiskfs will have to deal with that ZFS-motivated
> details. 
> 
> However, a light version of compression could work with smaller
> changes
> to ldiskfs, when we only allow a completely compressed or not
> compressed files and allow potential performance drops for broken
> read-
> ahead (due to gaps within the data). 
> 
> Hope it is somehow more clear now.
> 
> Regards,
> Anna
>