From linux at horizon.com Sun May 1 23:43:12 2016 From: linux at horizon.com (George Spelvin) Date: 1 May 2016 19:43:12 -0400 Subject: [lustre-devel] Lustre use of hash_long() & cfs_hash_u32_hash() Message-ID: <20160501234312.20601.qmail@ns.horizon.com> I notice you have copies of the functions in libcfs_hash.h, and use a mixture of the generic and copied functions in the Lustre code. I'm revising the hash_64() function to fix some horrible collision problems that Thomas Gleixner found, and I wanted to give you a heads-up. If there are cases where you use the generic ones but need reproducible outputs, they'll break. If there are places where you're using the copied ones and *don't* need reproducible outputs, you're getting bad performance. I made a brief attempt to figure out what is used for what, but it's too confusing for me. It's hard to figure out code like: static unsigned lu_obj_hop_hash(struct cfs_hash *hs, const void *key, unsigned mask) { struct lu_fid *fid = (struct lu_fid *)key; __u32 hash; hash = fid_flatten32(fid); hash += (hash >> 4) + (hash << 12); /* mixing oid and seq */ hash = hash_long(hash, hs->hs_bkt_bits); /* give me another random factor */ hash -= hash_long((unsigned long)hs, fid_oid(fid) % 11 + 3); hash <<= hs->hs_cur_bits - hs->hs_bkt_bits; hash |= (fid_seq(fid) + fid_oid(fid)) & (CFS_HASH_NBKT(hs) - 1); return hash & mask; } The whole business of selecting the number of bits of the hash based on a mod-11 hash of something else seems quite peculiar. Second, hash_long is multiplicative, meaning hash_long(a) - hash_long(b) = hash_long(a-b). The different bit shifts complicate it, but subtracting two such values from each other is not a great way to mix. Third, you're using the functions strangely. The first hash_long() takes a 32-bit input and would be more efficient on 64-bit platforms if it were hash_32. The second could be hash_ptr(), From jsimmons at infradead.org Wed May 4 14:28:51 2016 From: jsimmons at infradead.org (James Simmons) Date: Wed, 4 May 2016 10:28:51 -0400 Subject: [lustre-devel] [PATCH 0/9] staging: lustre: bug fixes from the lustre 2.5.[54-55] release Message-ID: <1462372140-27880-1-git-send-email-jsimmons@infradead.org> Here is the next batch of fixes and cleanups that went into Lustre versions 2.5.54 and 2.5.55 except for the work done for LU-3531. The changes for LU-3531 are large so they belong in their own special patch set. Most of the patches here are minor fixes and some removal of dead or obsolete code. Alexander Boyko (1): staging: lustre: osc: Allow lock to be canceled at ENQ time Andreas Dilger (1): staging: lustre: debug: clean up console messages Bobi Jam (1): staging: lustre: nfs: don't panic NFS server if MDS fails to find FID Fan Yong (1): staging: lustre: fid: packing ost_idx in IDIF John L. Hammond (2): staging: lustre: lov: remove lov and lod stuff from obd.h staging: lustre: lov: remove unused lov obd functions Niu Yawei (1): staging: lustre: ptlrpc: fix nrs cleanup wang di (2): staging: lustre: mdt: extra checking for getattr RPC. staging: lustre: fid: init FID client for OSP on MDT. drivers/staging/lustre/lustre/fid/fid_request.c | 7 +- .../lustre/lustre/include/lustre/lustre_idl.h | 76 +++++++++++++++------- drivers/staging/lustre/lustre/include/lustre_fid.h | 22 ++----- drivers/staging/lustre/lustre/include/obd.h | 60 ----------------- .../staging/lustre/lustre/include/obd_support.h | 1 + drivers/staging/lustre/lustre/llite/llite_lib.c | 6 +- drivers/staging/lustre/lustre/llite/llite_nfs.c | 13 ++-- drivers/staging/lustre/lustre/lov/lov_ea.c | 5 -- drivers/staging/lustre/lustre/lov/lov_internal.h | 32 ++++----- drivers/staging/lustre/lustre/lov/lov_obd.c | 3 - drivers/staging/lustre/lustre/lov/lov_pool.c | 62 +----------------- drivers/staging/lustre/lustre/lov/lov_request.c | 3 - drivers/staging/lustre/lustre/mdc/mdc_request.c | 3 +- drivers/staging/lustre/lustre/obdclass/obd_mount.c | 2 +- drivers/staging/lustre/lustre/osc/osc_request.c | 3 + drivers/staging/lustre/lustre/ptlrpc/nrs.c | 6 +- 16 files changed, 101 insertions(+), 203 deletions(-) -- 2.7.4 From jsimmons at infradead.org Wed May 4 14:28:55 2016 From: jsimmons at infradead.org (James Simmons) Date: Wed, 4 May 2016 10:28:55 -0400 Subject: [lustre-devel] [PATCH 4/9] staging: lustre: mdt: extra checking for getattr RPC. In-Reply-To: <1462372140-27880-1-git-send-email-jsimmons@infradead.org> References: <1462372140-27880-1-git-send-email-jsimmons@infradead.org> Message-ID: <1462372140-27880-5-git-send-email-jsimmons@infradead.org> From: wang di Check whether getattr RPC can hold layout MD(RMF_MDT_MD), in case the client sends some invalid RPC, which can cause panic on MDT. Client will retrieve cl_max_md_size/cl_default_md_size from MDS during mount process, so it will initialize cl_max_md_size/cl_default_md_size before sending getattr to MDS. Signed-off-by: wang di Intel-bug-id: https://jira.hpdd.intel.com/browse/LU-4222 Reviewed-on: http://review.whamcloud.com/8599 Reviewed-by: Fan Yong Reviewed-by: Andreas Dilger Signed-off-by: James Simmons --- drivers/staging/lustre/lustre/llite/llite_lib.c | 2 +- drivers/staging/lustre/lustre/mdc/mdc_request.c | 3 +-- 2 files changed, 2 insertions(+), 3 deletions(-) diff --git a/drivers/staging/lustre/lustre/llite/llite_lib.c b/drivers/staging/lustre/lustre/llite/llite_lib.c index 64be2c1..44dd3f3 100644 --- a/drivers/staging/lustre/lustre/llite/llite_lib.c +++ b/drivers/staging/lustre/lustre/llite/llite_lib.c @@ -445,7 +445,7 @@ static int client_common_fill_super(struct super_block *sb, char *md, char *dt, /* make root inode * XXX: move this to after cbd setup? */ - valid = OBD_MD_FLGETATTR | OBD_MD_FLBLOCKS; + valid = OBD_MD_FLGETATTR | OBD_MD_FLBLOCKS | OBD_MD_FLMODEASIZE; if (sbi->ll_flags & LL_SBI_RMT_CLIENT) valid |= OBD_MD_FLRMTPERM; else if (sbi->ll_flags & LL_SBI_ACL) diff --git a/drivers/staging/lustre/lustre/mdc/mdc_request.c b/drivers/staging/lustre/lustre/mdc/mdc_request.c index 46e3a7506..86b7445 100644 --- a/drivers/staging/lustre/lustre/mdc/mdc_request.c +++ b/drivers/staging/lustre/lustre/mdc/mdc_request.c @@ -142,9 +142,8 @@ static int mdc_getattr_common(struct obd_export *exp, CDEBUG(D_NET, "mode: %o\n", body->mode); + mdc_update_max_ea_from_body(exp, body); if (body->eadatasize != 0) { - mdc_update_max_ea_from_body(exp, body); - eadata = req_capsule_server_sized_get(pill, &RMF_MDT_MD, body->eadatasize); if (!eadata) -- 2.7.4 From jsimmons at infradead.org Wed May 4 14:28:54 2016 From: jsimmons at infradead.org (James Simmons) Date: Wed, 4 May 2016 10:28:54 -0400 Subject: [lustre-devel] [PATCH 3/9] staging: lustre: lov: remove lov and lod stuff from obd.h In-Reply-To: <1462372140-27880-1-git-send-email-jsimmons@infradead.org> References: <1462372140-27880-1-git-send-email-jsimmons@infradead.org> Message-ID: <1462372140-27880-4-git-send-email-jsimmons@infradead.org> From: "John L. Hammond" Remove QOS related data structures from obd.h to the lov_internal.h. Remove the declarations of several functions that no longer exist. Signed-off-by: John L. Hammond Intel-bug-id: https://jira.hpdd.intel.com/browse/LU-2675 Reviewed-on: http://review.whamcloud.com/8687 Reviewed-by: Andreas Dilger Reviewed-by: Jinshan Xiong Signed-off-by: James Simmons --- drivers/staging/lustre/lustre/include/obd.h | 60 ------------------------ drivers/staging/lustre/lustre/lov/lov_internal.h | 21 +++++++-- drivers/staging/lustre/lustre/lov/lov_obd.c | 2 - drivers/staging/lustre/lustre/lov/lov_pool.c | 15 +----- 4 files changed, 18 insertions(+), 80 deletions(-) diff --git a/drivers/staging/lustre/lustre/include/obd.h b/drivers/staging/lustre/lustre/include/obd.h index d0c0c26..2d926e0 100644 --- a/drivers/staging/lustre/lustre/include/obd.h +++ b/drivers/staging/lustre/lustre/include/obd.h @@ -166,9 +166,6 @@ struct obd_info { obd_enqueue_update_f oi_cb_up; }; -void lov_stripe_lock(struct lov_stripe_md *md); -void lov_stripe_unlock(struct lov_stripe_md *md); - struct obd_type { struct list_head typ_chain; struct obd_ops *typ_dt_ops; @@ -390,45 +387,9 @@ struct ost_pool { struct rw_semaphore op_rw_sem; /* to protect ost_pool use */ }; -/* Round-robin allocator data */ -struct lov_qos_rr { - __u32 lqr_start_idx; /* start index of new inode */ - __u32 lqr_offset_idx; /* aliasing for start_idx */ - int lqr_start_count; /* reseed counter */ - struct ost_pool lqr_pool; /* round-robin optimized list */ - unsigned long lqr_dirty:1; /* recalc round-robin list */ -}; - /* allow statfs data caching for 1 second */ #define OBD_STATFS_CACHE_SECONDS 1 -struct lov_statfs_data { - struct obd_info lsd_oi; - struct obd_statfs lsd_statfs; -}; - -/* Stripe placement optimization */ -struct lov_qos { - struct list_head lq_oss_list; /* list of OSSs that targets use */ - struct rw_semaphore lq_rw_sem; - __u32 lq_active_oss_count; - unsigned int lq_prio_free; /* priority for free space */ - unsigned int lq_threshold_rr;/* priority for rr */ - struct lov_qos_rr lq_rr; /* round robin qos data */ - unsigned long lq_dirty:1, /* recalc qos data */ - lq_same_space:1,/* the ost's all have approx. - * the same space avail - */ - lq_reset:1, /* zero current penalties */ - lq_statfs_in_progress:1; /* statfs op in - progress */ - /* qos statfs data */ - struct lov_statfs_data *lq_statfs_data; - wait_queue_head_t lq_statfs_waitq; /* waitqueue to notify statfs - * requests completion - */ -}; - struct lov_tgt_desc { struct list_head ltd_kill; struct obd_uuid ltd_uuid; @@ -441,25 +402,6 @@ struct lov_tgt_desc { ltd_reap:1; /* should this target be deleted */ }; -/* Pool metadata */ -#define pool_tgt_size(_p) _p->pool_obds.op_size -#define pool_tgt_count(_p) _p->pool_obds.op_count -#define pool_tgt_array(_p) _p->pool_obds.op_array -#define pool_tgt_rw_sem(_p) _p->pool_obds.op_rw_sem - -struct pool_desc { - char pool_name[LOV_MAXPOOLNAME + 1]; /* name of pool */ - struct ost_pool pool_obds; /* pool members */ - atomic_t pool_refcount; /* pool ref. counter */ - struct lov_qos_rr pool_rr; /* round robin qos */ - struct hlist_node pool_hash; /* access by poolname */ - struct list_head pool_list; /* serial access */ - struct dentry *pool_debugfs_entry; /* file in debugfs */ - struct obd_device *pool_lobd; /* obd of the lov/lod to which - * this pool belongs - */ -}; - struct lov_obd { struct lov_desc desc; struct lov_tgt_desc **lov_tgts; /* sparse array */ @@ -467,8 +409,6 @@ struct lov_obd { struct mutex lov_lock; struct obd_connect_data lov_ocd; atomic_t lov_refcount; - __u32 lov_tgt_count; /* how many OBD's */ - __u32 lov_active_tgt_count; /* how many active */ __u32 lov_death_row;/* tgts scheduled to be deleted */ __u32 lov_tgt_size; /* size of tgts array */ int lov_connects; diff --git a/drivers/staging/lustre/lustre/lov/lov_internal.h b/drivers/staging/lustre/lustre/lov/lov_internal.h index 9985855..ff01fe9 100644 --- a/drivers/staging/lustre/lustre/lov/lov_internal.h +++ b/drivers/staging/lustre/lustre/lov/lov_internal.h @@ -72,6 +72,21 @@ }) #endif +#define pool_tgt_size(p) ((p)->pool_obds.op_size) +#define pool_tgt_count(p) ((p)->pool_obds.op_count) +#define pool_tgt_array(p) ((p)->pool_obds.op_array) +#define pool_tgt_rw_sem(p) ((p)->pool_obds.op_rw_sem) + +struct pool_desc { + char pool_name[LOV_MAXPOOLNAME + 1]; + struct ost_pool pool_obds; + atomic_t pool_refcount; + struct hlist_node pool_hash; /* access by poolname */ + struct list_head pool_list; /* serial access */ + struct dentry *pool_debugfs_entry; /* file in debugfs */ + struct obd_device *pool_lobd; /* owner */ +}; + struct lov_request { struct obd_info rq_oi; struct lov_request_set *rq_rqset; @@ -149,10 +164,6 @@ int lov_stripe_number(struct lov_stripe_md *lsm, u64 lov_off); pgoff_t lov_stripe_pgoff(struct lov_stripe_md *lsm, pgoff_t stripe_index, int stripe); -/* lov_qos.c */ -#define LOV_USES_ASSIGNED_STRIPE 0 -#define LOV_USES_DEFAULT_STRIPE 1 - /* lov_request.c */ int lov_update_common_set(struct lov_request_set *set, struct lov_request *req, int rc); @@ -178,6 +189,8 @@ int lov_fini_statfs_set(struct lov_request_set *set); int lov_statfs_interpret(struct ptlrpc_request_set *rqset, void *data, int rc); /* lov_obd.c */ +void lov_stripe_lock(struct lov_stripe_md *md); +void lov_stripe_unlock(struct lov_stripe_md *md); void lov_fix_desc(struct lov_desc *desc); void lov_fix_desc_stripe_size(__u64 *val); void lov_fix_desc_stripe_count(__u32 *val); diff --git a/drivers/staging/lustre/lustre/lov/lov_obd.c b/drivers/staging/lustre/lustre/lov/lov_obd.c index 2de9255..bc9135a 100644 --- a/drivers/staging/lustre/lustre/lov/lov_obd.c +++ b/drivers/staging/lustre/lustre/lov/lov_obd.c @@ -2192,7 +2192,6 @@ void lov_stripe_lock(struct lov_stripe_md *md) LASSERT(md->lsm_lock_owner == 0); md->lsm_lock_owner = current_pid(); } -EXPORT_SYMBOL(lov_stripe_lock); void lov_stripe_unlock(struct lov_stripe_md *md) __releases(&md->lsm_lock) @@ -2201,7 +2200,6 @@ void lov_stripe_unlock(struct lov_stripe_md *md) md->lsm_lock_owner = 0; spin_unlock(&md->lsm_lock); } -EXPORT_SYMBOL(lov_stripe_unlock); static int lov_quotactl(struct obd_device *obd, struct obd_export *exp, struct obd_quotactl *oqctl) diff --git a/drivers/staging/lustre/lustre/lov/lov_pool.c b/drivers/staging/lustre/lustre/lov/lov_pool.c index 9ae1d6f..4254243 100644 --- a/drivers/staging/lustre/lustre/lov/lov_pool.c +++ b/drivers/staging/lustre/lustre/lov/lov_pool.c @@ -65,7 +65,6 @@ void lov_pool_putref(struct pool_desc *pool) LASSERT(hlist_unhashed(&pool->pool_hash)); LASSERT(list_empty(&pool->pool_list)); LASSERT(!pool->pool_debugfs_entry); - lov_ost_pool_free(&(pool->pool_rr.lqr_pool)); lov_ost_pool_free(&(pool->pool_obds)); kfree(pool); } @@ -424,11 +423,6 @@ int lov_pool_new(struct obd_device *obd, char *poolname) if (rc) goto out_err; - memset(&(new_pool->pool_rr), 0, sizeof(struct lov_qos_rr)); - rc = lov_ost_pool_init(&new_pool->pool_rr.lqr_pool, 0); - if (rc) - goto out_free_pool_obds; - INIT_HLIST_NODE(&new_pool->pool_hash); /* get ref for debugfs file */ @@ -469,13 +463,10 @@ out_err: list_del_init(&new_pool->pool_list); lov->lov_pool_count--; spin_unlock(&obd->obd_dev_lock); - ldebugfs_remove(&new_pool->pool_debugfs_entry); - - lov_ost_pool_free(&new_pool->pool_rr.lqr_pool); -out_free_pool_obds: lov_ost_pool_free(&new_pool->pool_obds); kfree(new_pool); + return rc; } @@ -543,8 +534,6 @@ int lov_pool_add(struct obd_device *obd, char *poolname, char *ostname) if (rc) goto out; - pool->pool_rr.lqr_dirty = 1; - CDEBUG(D_CONFIG, "Added %s to "LOV_POOLNAMEF" as member %d\n", ostname, poolname, pool_tgt_count(pool)); @@ -589,8 +578,6 @@ int lov_pool_remove(struct obd_device *obd, char *poolname, char *ostname) lov_ost_pool_remove(&pool->pool_obds, lov_idx); - pool->pool_rr.lqr_dirty = 1; - CDEBUG(D_CONFIG, "%s removed from "LOV_POOLNAMEF"\n", ostname, poolname); -- 2.7.4 From jsimmons at infradead.org Wed May 4 14:28:53 2016 From: jsimmons at infradead.org (James Simmons) Date: Wed, 4 May 2016 10:28:53 -0400 Subject: [lustre-devel] [PATCH 2/9] staging: lustre: osc: Allow lock to be canceled at ENQ time In-Reply-To: <1462372140-27880-1-git-send-email-jsimmons@infradead.org> References: <1462372140-27880-1-git-send-email-jsimmons@infradead.org> Message-ID: <1462372140-27880-3-git-send-email-jsimmons@infradead.org> From: Alexander Boyko A cl_lock can be canceled when it's in CLS_ENQUEUED state. We can't unuse this kind of lock in lov_lock_unuse() because it will bring this lock into CLS_NEW state and then confuse osc_lock_upcall(). Add a regression test case by Alexander Boyko. Signed-off-by: Jinshan Xiong Signed-off-by: Alexander Boyko Intel-bug-id: https://jira.hpdd.intel.com/browse/LU-3889 Reviewed-on: http://review.whamcloud.com/8405 Reviewed-by: Bobi Jam Reviewed-by: Oleg Drokin Signed-off-by: James Simmons --- drivers/staging/lustre/lustre/include/obd_support.h | 1 + drivers/staging/lustre/lustre/osc/osc_request.c | 3 +++ 2 files changed, 4 insertions(+) diff --git a/drivers/staging/lustre/lustre/include/obd_support.h b/drivers/staging/lustre/lustre/include/obd_support.h index 1358af1..60034d3 100644 --- a/drivers/staging/lustre/lustre/include/obd_support.h +++ b/drivers/staging/lustre/lustre/include/obd_support.h @@ -321,6 +321,7 @@ extern char obd_jobid_var[]; #define OBD_FAIL_LDLM_AGL_DELAY 0x31a #define OBD_FAIL_LDLM_AGL_NOLOCK 0x31b #define OBD_FAIL_LDLM_OST_LVB 0x31c +#define OBD_FAIL_LDLM_ENQUEUE_HANG 0x31d /* LOCKLESS IO */ #define OBD_FAIL_LDLM_SET_CONTENTION 0x385 diff --git a/drivers/staging/lustre/lustre/osc/osc_request.c b/drivers/staging/lustre/lustre/osc/osc_request.c index df06f4f..47417f8 100644 --- a/drivers/staging/lustre/lustre/osc/osc_request.c +++ b/drivers/staging/lustre/lustre/osc/osc_request.c @@ -2201,6 +2201,9 @@ static int osc_enqueue_interpret(const struct lu_env *env, */ ldlm_lock_addref(lockh, mode); + /* Let cl_lock_state_wait fail with -ERESTARTSYS to unuse sublocks. */ + OBD_FAIL_TIMEOUT(OBD_FAIL_LDLM_ENQUEUE_HANG, 2); + /* Let CP AST to grant the lock first. */ OBD_FAIL_TIMEOUT(OBD_FAIL_OSC_CP_ENQ_RACE, 1); -- 2.7.4 From jsimmons at infradead.org Wed May 4 14:28:59 2016 From: jsimmons at infradead.org (James Simmons) Date: Wed, 4 May 2016 10:28:59 -0400 Subject: [lustre-devel] [PATCH 8/9] staging: lustre: fid: init FID client for OSP on MDT. In-Reply-To: <1462372140-27880-1-git-send-email-jsimmons@infradead.org> References: <1462372140-27880-1-git-send-email-jsimmons@infradead.org> Message-ID: <1462372140-27880-9-git-send-email-jsimmons@infradead.org> From: wang di Initialize FID client for OSP on MDT. Signed-off-by: wang di Intel-bug-id: https://jira.hpdd.intel.com/browse/LU-3529 Reviewed-on: http://review.whamcloud.com/7158 Reviewed-by: John L. Hammond Reviewed-by: Alex Zhuravlev Reviewed-by: Oleg Drokin Signed-off-by: James Simmons --- drivers/staging/lustre/lustre/fid/fid_request.c | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/drivers/staging/lustre/lustre/fid/fid_request.c b/drivers/staging/lustre/lustre/fid/fid_request.c index 32e4d95..3a4df62 100644 --- a/drivers/staging/lustre/lustre/fid/fid_request.c +++ b/drivers/staging/lustre/lustre/fid/fid_request.c @@ -102,10 +102,13 @@ static int seq_client_rpc(struct lu_client_seq *seq, req->rq_no_delay = req->rq_no_resend = 1; debug_mask = D_CONSOLE; } else { - if (seq->lcs_type == LUSTRE_SEQ_METADATA) + if (seq->lcs_type == LUSTRE_SEQ_METADATA) { + req->rq_reply_portal = MDC_REPLY_PORTAL; req->rq_request_portal = SEQ_METADATA_PORTAL; - else + } else { + req->rq_reply_portal = OSC_REPLY_PORTAL; req->rq_request_portal = SEQ_DATA_PORTAL; + } debug_mask = D_INFO; } -- 2.7.4 From jsimmons at infradead.org Wed May 4 14:28:57 2016 From: jsimmons at infradead.org (James Simmons) Date: Wed, 4 May 2016 10:28:57 -0400 Subject: [lustre-devel] [PATCH 6/9] staging: lustre: debug: clean up console messages In-Reply-To: <1462372140-27880-1-git-send-email-jsimmons@infradead.org> References: <1462372140-27880-1-git-send-email-jsimmons@infradead.org> Message-ID: <1462372140-27880-7-git-send-email-jsimmons@infradead.org> From: Andreas Dilger Clean up overly verbose console error messages, improve others. Signed-off-by: Andreas Dilger Intel-bug-id: https://jira.hpdd.intel.com/browse/LU-1095 Reviewed-on: http://review.whamcloud.com/8617 Reviewed-by: Faccini Bruno Reviewed-by: Bob Glossman Reviewed-by: Oleg Drokin Signed-off-by: James Simmons --- drivers/staging/lustre/lustre/llite/llite_lib.c | 4 +--- drivers/staging/lustre/lustre/obdclass/obd_mount.c | 2 +- 2 files changed, 2 insertions(+), 4 deletions(-) diff --git a/drivers/staging/lustre/lustre/llite/llite_lib.c b/drivers/staging/lustre/lustre/llite/llite_lib.c index 44dd3f3..96c7e9f 100644 --- a/drivers/staging/lustre/lustre/llite/llite_lib.c +++ b/drivers/staging/lustre/lustre/llite/llite_lib.c @@ -897,10 +897,8 @@ int ll_fill_super(struct super_block *sb, struct vfsmount *mnt) cfg->cfg_callback = class_config_llog_handler; /* set up client obds */ err = lustre_process_log(sb, profilenm, cfg); - if (err < 0) { - CERROR("Unable to process log: %d\n", err); + if (err < 0) goto out_free; - } /* Profile set with LCFG_MOUNTOPT so we can find our mdc and osc obds */ lprof = class_get_profile(profilenm); diff --git a/drivers/staging/lustre/lustre/obdclass/obd_mount.c b/drivers/staging/lustre/lustre/obdclass/obd_mount.c index 9474aa9..e0c90ad 100644 --- a/drivers/staging/lustre/lustre/obdclass/obd_mount.c +++ b/drivers/staging/lustre/lustre/obdclass/obd_mount.c @@ -102,7 +102,7 @@ int lustre_process_log(struct super_block *sb, char *logname, LCONSOLE_ERROR_MSG(0x15b, "%s: The configuration from log '%s' failed from the MGS (%d). Make sure this client and the MGS are running compatible versions of Lustre.\n", mgc->obd_name, logname, rc); - if (rc) + else if (rc) LCONSOLE_ERROR_MSG(0x15c, "%s: The configuration from log '%s' failed (%d). This may be the result of communication errors between this node and the MGS, a bad configuration, or other errors. See the syslog for more information.\n", mgc->obd_name, logname, rc); -- 2.7.4 From jsimmons at infradead.org Wed May 4 14:28:52 2016 From: jsimmons at infradead.org (James Simmons) Date: Wed, 4 May 2016 10:28:52 -0400 Subject: [lustre-devel] [PATCH 1/9] staging: lustre: nfs: don't panic NFS server if MDS fails to find FID In-Reply-To: <1462372140-27880-1-git-send-email-jsimmons@infradead.org> References: <1462372140-27880-1-git-send-email-jsimmons@infradead.org> Message-ID: <1462372140-27880-2-git-send-email-jsimmons@infradead.org> From: Bobi Jam When MDS fails to retrive the parent's fid, we'd handle it without crashing the NFS server. Signed-off-by: Bobi Jam Intel-bug-id: https://jira.hpdd.intel.com/browse/LU-3952 Reviewed-on: http://review.whamcloud.com/8459 Reviewed-by: Fan Yong Reviewed-by: wangdi Reviewed-by: Oleg Drokin Signed-off-by: James Simmons --- drivers/staging/lustre/lustre/llite/llite_nfs.c | 13 ++++++++----- 1 file changed, 8 insertions(+), 5 deletions(-) diff --git a/drivers/staging/lustre/lustre/llite/llite_nfs.c b/drivers/staging/lustre/lustre/llite/llite_nfs.c index 61cb5ff..c1eef61 100644 --- a/drivers/staging/lustre/lustre/llite/llite_nfs.c +++ b/drivers/staging/lustre/lustre/llite/llite_nfs.c @@ -322,11 +322,14 @@ static struct dentry *ll_get_parent(struct dentry *dchild) return ERR_PTR(rc); } body = req_capsule_server_get(&req->rq_pill, &RMF_MDT_BODY); - LASSERT(body->valid & OBD_MD_FLID); - - CDEBUG(D_INFO, "parent for " DFID " is " DFID "\n", - PFID(ll_inode2fid(dir)), PFID(&body->fid1)); - + /* + * LU-3952: MDT may lost the FID of its parent, we should not crash + * the NFS server, ll_iget_for_nfs() will handle the error. + */ + if (body->valid & OBD_MD_FLID) { + CDEBUG(D_INFO, "parent for " DFID " is " DFID "\n", + PFID(ll_inode2fid(dir)), PFID(&body->fid1)); + } result = ll_iget_for_nfs(dir->i_sb, &body->fid1, NULL); ptlrpc_req_finished(req); -- 2.7.4 From jsimmons at infradead.org Wed May 4 14:29:00 2016 From: jsimmons at infradead.org (James Simmons) Date: Wed, 4 May 2016 10:29:00 -0400 Subject: [lustre-devel] [PATCH 9/9] staging: lustre: lov: remove unused lov obd functions In-Reply-To: <1462372140-27880-1-git-send-email-jsimmons@infradead.org> References: <1462372140-27880-1-git-send-email-jsimmons@infradead.org> Message-ID: <1462372140-27880-10-git-send-email-jsimmons@infradead.org> From: "John L. Hammond" Remove the unused lov functions lov_get_reqset, lov_check_index_in_pool, and lov_find_pool functions. Remove unused data structures. Signed-off-by: John L. Hammond Intel-bug-id: https://jira.hpdd.intel.com/browse/LU-2675 Reviewed-on: http://review.whamcloud.com/5581 Reviewed-by: Alex Zhuravlev Reviewed-by: Andreas Dilger Reviewed-by: James Simmons Signed-off-by: James Simmons --- drivers/staging/lustre/lustre/lov/lov_ea.c | 5 --- drivers/staging/lustre/lustre/lov/lov_internal.h | 11 ------ drivers/staging/lustre/lustre/lov/lov_obd.c | 1 - drivers/staging/lustre/lustre/lov/lov_pool.c | 47 ------------------------ drivers/staging/lustre/lustre/lov/lov_request.c | 3 -- 5 files changed, 67 deletions(-) diff --git a/drivers/staging/lustre/lustre/lov/lov_ea.c b/drivers/staging/lustre/lustre/lov/lov_ea.c index b652940..460f0fa 100644 --- a/drivers/staging/lustre/lustre/lov/lov_ea.c +++ b/drivers/staging/lustre/lustre/lov/lov_ea.c @@ -48,11 +48,6 @@ #include "lov_internal.h" -struct lovea_unpack_args { - struct lov_stripe_md *lsm; - int cursor; -}; - static int lsm_lmm_verify_common(struct lov_mds_md *lmm, int lmm_bytes, __u16 stripe_count) { diff --git a/drivers/staging/lustre/lustre/lov/lov_internal.h b/drivers/staging/lustre/lustre/lov/lov_internal.h index ff01fe9..eef9afa 100644 --- a/drivers/staging/lustre/lustre/lov/lov_internal.h +++ b/drivers/staging/lustre/lustre/lov/lov_internal.h @@ -103,7 +103,6 @@ struct lov_request { }; struct lov_request_set { - struct ldlm_enqueue_info *set_ei; struct obd_info *set_oi; atomic_t set_refcount; struct obd_export *set_exp; @@ -117,10 +116,8 @@ struct lov_request_set { atomic_t set_finish_checked; struct llog_cookie *set_cookies; int set_cookie_sent; - struct obd_trans_info *set_oti; struct list_head set_list; wait_queue_head_t set_waitq; - spinlock_t set_lock; }; extern struct kmem_cache *lov_oinfo_slab; @@ -129,12 +126,6 @@ extern struct lu_kmem_descr lov_caches[]; void lov_finish_set(struct lov_request_set *set); -static inline void lov_get_reqset(struct lov_request_set *set) -{ - LASSERT(atomic_read(&set->set_refcount) > 0); - atomic_inc(&set->set_refcount); -} - static inline void lov_put_reqset(struct lov_request_set *set) { if (atomic_dec_and_test(&set->set_refcount)) @@ -246,8 +237,6 @@ int lov_pool_new(struct obd_device *obd, char *poolname); int lov_pool_del(struct obd_device *obd, char *poolname); int lov_pool_add(struct obd_device *obd, char *poolname, char *ostname); int lov_pool_remove(struct obd_device *obd, char *poolname, char *ostname); -struct pool_desc *lov_find_pool(struct lov_obd *lov, char *poolname); -int lov_check_index_in_pool(__u32 idx, struct pool_desc *pool); void lov_pool_putref(struct pool_desc *pool); static inline struct lov_stripe_md *lsm_addref(struct lov_stripe_md *lsm) diff --git a/drivers/staging/lustre/lustre/lov/lov_obd.c b/drivers/staging/lustre/lustre/lov/lov_obd.c index bc9135a..e15ef2e 100644 --- a/drivers/staging/lustre/lustre/lov/lov_obd.c +++ b/drivers/staging/lustre/lustre/lov/lov_obd.c @@ -123,7 +123,6 @@ static int lov_set_osc_active(struct obd_device *obd, struct obd_uuid *uuid, static int lov_notify(struct obd_device *obd, struct obd_device *watched, enum obd_notify_event ev, void *data); -#define MAX_STRING_SIZE 128 int lov_connect_obd(struct obd_device *obd, __u32 index, int activate, struct obd_connect_data *data) { diff --git a/drivers/staging/lustre/lustre/lov/lov_pool.c b/drivers/staging/lustre/lustre/lov/lov_pool.c index 4254243..690292e 100644 --- a/drivers/staging/lustre/lustre/lov/lov_pool.c +++ b/drivers/staging/lustre/lustre/lov/lov_pool.c @@ -586,50 +586,3 @@ out: lov_pool_putref(pool); return rc; } - -int lov_check_index_in_pool(__u32 idx, struct pool_desc *pool) -{ - int i, rc; - - /* caller may no have a ref on pool if it got the pool - * without calling lov_find_pool() (e.g. go through the lov pool - * list) - */ - lov_pool_getref(pool); - - down_read(&pool_tgt_rw_sem(pool)); - - for (i = 0; i < pool_tgt_count(pool); i++) { - if (pool_tgt_array(pool)[i] == idx) { - rc = 0; - goto out; - } - } - rc = -ENOENT; -out: - up_read(&pool_tgt_rw_sem(pool)); - - lov_pool_putref(pool); - return rc; -} - -struct pool_desc *lov_find_pool(struct lov_obd *lov, char *poolname) -{ - struct pool_desc *pool; - - pool = NULL; - if (poolname[0] != '\0') { - pool = cfs_hash_lookup(lov->lov_pools_hash_body, poolname); - if (!pool) - CWARN("Request for an unknown pool ("LOV_POOLNAMEF")\n", - poolname); - if (pool && (pool_tgt_count(pool) == 0)) { - CWARN("Request for an empty pool ("LOV_POOLNAMEF")\n", - poolname); - /* pool is ignored, so we remove ref on it */ - lov_pool_putref(pool); - pool = NULL; - } - } - return pool; -} diff --git a/drivers/staging/lustre/lustre/lov/lov_request.c b/drivers/staging/lustre/lustre/lov/lov_request.c index 4d7073e..1be4b92 100644 --- a/drivers/staging/lustre/lustre/lov/lov_request.c +++ b/drivers/staging/lustre/lustre/lov/lov_request.c @@ -52,7 +52,6 @@ static void lov_init_set(struct lov_request_set *set) INIT_LIST_HEAD(&set->set_list); atomic_set(&set->set_refcount, 1); init_waitqueue_head(&set->set_waitq); - spin_lock_init(&set->set_lock); } void lov_finish_set(struct lov_request_set *set) @@ -362,7 +361,6 @@ int lov_prep_destroy_set(struct obd_export *exp, struct obd_info *oinfo, set->set_oi = oinfo; set->set_oi->oi_md = lsm; set->set_oi->oi_oa = src_oa; - set->set_oti = oti; if (oti && src_oa->o_valid & OBD_MD_FLCOOKIE) set->set_cookies = oti->oti_logcookies; @@ -479,7 +477,6 @@ int lov_prep_setattr_set(struct obd_export *exp, struct obd_info *oinfo, lov_init_set(set); set->set_exp = exp; - set->set_oti = oti; set->set_oi = oinfo; if (oti && oinfo->oi_oa->o_valid & OBD_MD_FLCOOKIE) set->set_cookies = oti->oti_logcookies; -- 2.7.4 From jsimmons at infradead.org Wed May 4 14:28:56 2016 From: jsimmons at infradead.org (James Simmons) Date: Wed, 4 May 2016 10:28:56 -0400 Subject: [lustre-devel] [PATCH 5/9] staging: lustre: fid: packing ost_idx in IDIF In-Reply-To: <1462372140-27880-1-git-send-email-jsimmons@infradead.org> References: <1462372140-27880-1-git-send-email-jsimmons@infradead.org> Message-ID: <1462372140-27880-6-git-send-email-jsimmons@infradead.org> From: Fan Yong For a normal FID, we can know on which target the related object is allocated via querying FLDB; but it is not true for an IDIF. To locate the OST via the given IDIF, when the IDIF is generated, we pack the OST index in it. Then for any given FID, in spite of t is a normal FID or not, we has the method to know which target it belongs to. That is useful for LFSCK. For old IDIF, the OST index is not part of the IDIF, means that ifferent OSTs may have the same IDIFs, that may cause the IFID in LMA does not match the read FID. Under such case, we need to make some compatible check to avoid to trigger unexpected. tgt_validate_obdo() converts the ostid contained in the RPC body to fid and changes the "struct ost_id" union, then the users can access ost_id::oi_fid directly without call ostid_to_fid() again. It also contains some other fixing and cleanup. Signed-off-by: wang di Signed-off-by: Fan Yong Intel-bug-id: https://jira.hpdd.intel.com/browse/LU-3569 Reviewed-on: http://review.whamcloud.com/7053 Reviewed-by: Andreas Dilger Reviewed-by: Alex Zhuravlev Signed-off-by: James Simmons --- .../lustre/lustre/include/lustre/lustre_idl.h | 76 +++++++++++++++------- drivers/staging/lustre/lustre/include/lustre_fid.h | 22 ++----- 2 files changed, 57 insertions(+), 41 deletions(-) diff --git a/drivers/staging/lustre/lustre/include/lustre/lustre_idl.h b/drivers/staging/lustre/lustre/include/lustre/lustre_idl.h index a70545a..9c53c17 100644 --- a/drivers/staging/lustre/lustre/include/lustre/lustre_idl.h +++ b/drivers/staging/lustre/lustre/include/lustre/lustre_idl.h @@ -584,7 +584,7 @@ static inline __u64 ostid_seq(const struct ost_id *ostid) if (fid_seq_is_mdt0(ostid->oi.oi_seq)) return FID_SEQ_OST_MDT0; - if (fid_seq_is_default(ostid->oi.oi_seq)) + if (unlikely(fid_seq_is_default(ostid->oi.oi_seq))) return FID_SEQ_LOV_DEFAULT; if (fid_is_idif(&ostid->oi_fid)) @@ -596,9 +596,12 @@ static inline __u64 ostid_seq(const struct ost_id *ostid) /* extract OST objid from a wire ost_id (id/seq) pair */ static inline __u64 ostid_id(const struct ost_id *ostid) { - if (fid_seq_is_mdt0(ostid_seq(ostid))) + if (fid_seq_is_mdt0(ostid->oi.oi_seq)) return ostid->oi.oi_id & IDIF_OID_MASK; + if (unlikely(fid_seq_is_default(ostid->oi.oi_seq))) + return ostid->oi.oi_id; + if (fid_is_idif(&ostid->oi_fid)) return fid_idif_id(fid_seq(&ostid->oi_fid), fid_oid(&ostid->oi_fid), 0); @@ -642,12 +645,22 @@ static inline void ostid_set_seq_llog(struct ost_id *oi) */ static inline void ostid_set_id(struct ost_id *oi, __u64 oid) { - if (fid_seq_is_mdt0(ostid_seq(oi))) { + if (fid_seq_is_mdt0(oi->oi.oi_seq)) { if (oid >= IDIF_MAX_OID) { CERROR("Bad %llu to set " DOSTID "\n", oid, POSTID(oi)); return; } oi->oi.oi_id = oid; + } else if (fid_is_idif(&oi->oi_fid)) { + if (oid >= IDIF_MAX_OID) { + CERROR("Bad %llu to set "DOSTID"\n", + oid, POSTID(oi)); + return; + } + oi->oi_fid.f_seq = fid_idif_seq(oid, + fid_idif_ost_idx(&oi->oi_fid)); + oi->oi_fid.f_oid = oid; + oi->oi_fid.f_ver = oid >> 48; } else { if (oid > OBIF_MAX_OID) { CERROR("Bad %llu to set " DOSTID "\n", oid, POSTID(oi)); @@ -657,25 +670,31 @@ static inline void ostid_set_id(struct ost_id *oi, __u64 oid) } } -static inline void ostid_inc_id(struct ost_id *oi) +static inline int fid_set_id(struct lu_fid *fid, __u64 oid) { - if (fid_seq_is_mdt0(ostid_seq(oi))) { - if (unlikely(ostid_id(oi) + 1 > IDIF_MAX_OID)) { - CERROR("Bad inc "DOSTID"\n", POSTID(oi)); - return; + if (unlikely(fid_seq_is_igif(fid->f_seq))) { + CERROR("bad IGIF, "DFID"\n", PFID(fid)); + return -EBADF; + } + + if (fid_is_idif(fid)) { + if (oid >= IDIF_MAX_OID) { + CERROR("Too large OID %#llx to set IDIF "DFID"\n", + (unsigned long long)oid, PFID(fid)); + return -EBADF; } - oi->oi.oi_id++; + fid->f_seq = fid_idif_seq(oid, fid_idif_ost_idx(fid)); + fid->f_oid = oid; + fid->f_ver = oid >> 48; } else { - oi->oi_fid.f_oid++; + if (oid > OBIF_MAX_OID) { + CERROR("Too large OID %#llx to set REG "DFID"\n", + (unsigned long long)oid, PFID(fid)); + return -EBADF; + } + fid->f_oid = oid; } -} - -static inline void ostid_dec_id(struct ost_id *oi) -{ - if (fid_seq_is_mdt0(ostid_seq(oi))) - oi->oi.oi_id--; - else - oi->oi_fid.f_oid--; + return 0; } /** @@ -690,30 +709,34 @@ static inline void ostid_dec_id(struct ost_id *oi) static inline int ostid_to_fid(struct lu_fid *fid, struct ost_id *ostid, __u32 ost_idx) { + __u64 seq = ostid_seq(ostid); + if (ost_idx > 0xffff) { CERROR("bad ost_idx, "DOSTID" ost_idx:%u\n", POSTID(ostid), ost_idx); return -EBADF; } - if (fid_seq_is_mdt0(ostid_seq(ostid))) { + if (fid_seq_is_mdt0(seq)) { + __u64 oid = ostid_id(ostid); + /* This is a "legacy" (old 1.x/2.early) OST object in "group 0" * that we map into the IDIF namespace. It allows up to 2^48 * objects per OST, as this is the object namespace that has * been in production for years. This can handle create rates * of 1M objects/s/OST for 9 years, or combinations thereof. */ - if (ostid_id(ostid) >= IDIF_MAX_OID) { + if (oid >= IDIF_MAX_OID) { CERROR("bad MDT0 id, " DOSTID " ost_idx:%u\n", POSTID(ostid), ost_idx); return -EBADF; } - fid->f_seq = fid_idif_seq(ostid_id(ostid), ost_idx); + fid->f_seq = fid_idif_seq(oid, ost_idx); /* truncate to 32 bits by assignment */ - fid->f_oid = ostid_id(ostid); + fid->f_oid = oid; /* in theory, not currently used */ - fid->f_ver = ostid_id(ostid) >> 48; - } else /* if (fid_seq_is_idif(seq) || fid_seq_is_norm(seq)) */ { + fid->f_ver = oid >> 48; + } else if (likely(!fid_seq_is_default(seq))) { /* This is either an IDIF object, which identifies objects across * all OSTs, or a regular FID. The IDIF namespace maps legacy * OST objects into the FID namespace. In both cases, we just @@ -1537,6 +1560,11 @@ static inline void lmm_oi_set_seq(struct ost_id *oi, __u64 seq) oi->oi.oi_seq = seq; } +static inline void lmm_oi_set_id(struct ost_id *oi, __u64 oid) +{ + oi->oi.oi_id = oid; +} + static inline __u64 lmm_oi_id(struct ost_id *oi) { return oi->oi.oi_id; diff --git a/drivers/staging/lustre/lustre/include/lustre_fid.h b/drivers/staging/lustre/lustre/include/lustre_fid.h index ab4a923..12e8b58 100644 --- a/drivers/staging/lustre/lustre/include/lustre_fid.h +++ b/drivers/staging/lustre/lustre/include/lustre_fid.h @@ -308,10 +308,10 @@ static inline int fid_seq_in_fldb(__u64 seq) fid_seq_is_root(seq) || fid_seq_is_dot(seq); } -static inline void lu_last_id_fid(struct lu_fid *fid, __u64 seq) +static inline void lu_last_id_fid(struct lu_fid *fid, __u64 seq, __u32 ost_idx) { if (fid_seq_is_mdt0(seq)) { - fid->f_seq = fid_idif_seq(0, 0); + fid->f_seq = fid_idif_seq(0, ost_idx); } else { LASSERTF(fid_seq_is_norm(seq) || fid_seq_is_echo(seq) || fid_seq_is_idif(seq), "%#llx\n", seq); @@ -498,19 +498,6 @@ static inline void ostid_build_res_name(struct ost_id *oi, } } -static inline void ostid_res_name_to_id(struct ost_id *oi, - struct ldlm_res_id *name) -{ - if (fid_seq_is_mdt0(name->name[LUSTRE_RES_ID_SEQ_OFF])) { - /* old resid */ - ostid_set_seq(oi, name->name[LUSTRE_RES_ID_VER_OID_OFF]); - ostid_set_id(oi, name->name[LUSTRE_RES_ID_SEQ_OFF]); - } else { - /* new resid */ - fid_extract_from_res_name(&oi->oi_fid, name); - } -} - /** * Return true if the resource is for the object identified by this id & group. */ @@ -546,7 +533,8 @@ static inline void ost_fid_build_resid(const struct lu_fid *fid, } static inline void ost_fid_from_resid(struct lu_fid *fid, - const struct ldlm_res_id *name) + const struct ldlm_res_id *name, + int ost_idx) { if (fid_seq_is_mdt0(name->name[LUSTRE_RES_ID_VER_OID_OFF])) { /* old resid */ @@ -554,7 +542,7 @@ static inline void ost_fid_from_resid(struct lu_fid *fid, ostid_set_seq(&oi, name->name[LUSTRE_RES_ID_VER_OID_OFF]); ostid_set_id(&oi, name->name[LUSTRE_RES_ID_SEQ_OFF]); - ostid_to_fid(fid, &oi, 0); + ostid_to_fid(fid, &oi, ost_idx); } else { /* new resid */ fid_extract_from_res_name(fid, name); -- 2.7.4 From jsimmons at infradead.org Wed May 4 14:28:58 2016 From: jsimmons at infradead.org (James Simmons) Date: Wed, 4 May 2016 10:28:58 -0400 Subject: [lustre-devel] [PATCH 7/9] staging: lustre: ptlrpc: fix nrs cleanup In-Reply-To: <1462372140-27880-1-git-send-email-jsimmons@infradead.org> References: <1462372140-27880-1-git-send-email-jsimmons@infradead.org> Message-ID: <1462372140-27880-8-git-send-email-jsimmons@infradead.org> From: Niu Yawei When service start failed due to short of memory, the cleanup code could operate on uninitialized structure and cause crash at the end. This patch fix the nrs_svcpt_cleanup_locked() to perform cleanup only on the nrs which has been properly initialized. Signed-off-by: Niu Yawei Intel-bug-id: https://jira.hpdd.intel.com/browse/LU-3772 Reviewed-on: http://review.whamcloud.com/7410 Reviewed-by: Andreas Dilger Reviewed-by: Oleg Drokin Signed-off-by: James Simmons --- drivers/staging/lustre/lustre/ptlrpc/nrs.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/drivers/staging/lustre/lustre/ptlrpc/nrs.c b/drivers/staging/lustre/lustre/ptlrpc/nrs.c index 99ff6e8..c444f51 100644 --- a/drivers/staging/lustre/lustre/ptlrpc/nrs.c +++ b/drivers/staging/lustre/lustre/ptlrpc/nrs.c @@ -975,7 +975,11 @@ static void nrs_svcpt_cleanup_locked(struct ptlrpc_service_part *svcpt) LASSERT(mutex_is_locked(&nrs_core.nrs_mutex)); again: - nrs = nrs_svcpt2nrs(svcpt, hp); + /* scp_nrs_hp could be NULL due to short of memory. */ + nrs = hp ? svcpt->scp_nrs_hp : &svcpt->scp_nrs_reg; + /* check the nrs_svcpt to see if nrs is initialized. */ + if (!nrs || !nrs->nrs_svcpt) + return; nrs->nrs_stopping = 1; list_for_each_entry_safe(policy, tmp, &nrs->nrs_policy_list, pol_list) { -- 2.7.4 From simmonsja at ornl.gov Wed May 4 15:06:27 2016 From: simmonsja at ornl.gov (Simmons, James A.) Date: Wed, 4 May 2016 15:06:27 +0000 Subject: [lustre-devel] New wiki documentation Message-ID: Hello folks I have added new documentation to the wiki. The first doc is about the upstream client which can be found at: http://wiki.lustre.org/Development#Upstream_contributing People have also been approaching me about how to get mlx4 hardware working with mlx5 hardware so I wrote a wiki about what I did at: http://wiki.lustre.org/Infiniband_Configuration_Howto -------------- next part -------------- An HTML attachment was scrubbed... URL: From andreas.dilger at intel.com Wed May 4 18:10:44 2016 From: andreas.dilger at intel.com (Dilger, Andreas) Date: Wed, 4 May 2016 18:10:44 +0000 Subject: [lustre-devel] New wiki documentation Message-ID: James, would it make sense/is it possible to have a default module parameters for MLX4 and/or MLX5 in ko2iblnd.conf that "do the right thing" to avoid the need for users to deal with this, or is that not possible due to conflicting requirements between these two drivers? Cheers, Andreas -- Andreas Dilger Lustre Principal Architect Intel High Performance Data Division On 2016/05/04, 09:06, "lustre-devel on behalf of Simmons, James A." on behalf of simmonsja at ornl.gov> wrote: Hello folks I have added new documentation to the wiki. The first doc is about the upstream client which can be found at: http://wiki.lustre.org/Development#Upstream_contributing People have also been approaching me about how to get mlx4 hardware working with mlx5 hardware so I wrote a wiki about what I did at: http://wiki.lustre.org/Infiniband_Configuration_Howto From jsimmons at infradead.org Wed May 4 20:05:11 2016 From: jsimmons at infradead.org (James Simmons) Date: Wed, 4 May 2016 21:05:11 +0100 (BST) Subject: [lustre-devel] New wiki documentation In-Reply-To: References: Message-ID: > James, > would it make sense/is it possible to have a default module parameters for MLX4 and/or MLX5 in ko2iblnd.conf that > "do the right thing" to avoid the need for users to deal with this, or is that not possible due to conflicting > requirements between these two drivers? For the stand alone mlx5 case yes we could have ko2iblnd.conf set peer_credits=16. As for making mlx5 <-> mlx4 to work together that requires setting map_on_demand=256 which normally is turned off. Since we can never know if someone is creating a system with a mix environment and if we want it to be automatic for the non-power user then map_ond_demand=256 would have to be apart of the normal configuration. One limitation of using the modprobe config for this setup when compared to using the DLC yaml configuration file is that with the DLC approach is that you can set the LND tunebles per interface. If you use the modprobe config file it is global to all IB interfaces on a node. Especially once multi-rail becomes reality we need to really rethink how to create a default configuration for the non-power user using DLC. > Cheers, Andreas > -- > Andreas Dilger > Lustre Principal Architect > Intel High Performance Data Division > > On 2016/05/04, 09:06, "lustre-devel on behalf of Simmons, James A." on behalf of simmonsja at ornl.gov> wrote: > > Hello folks > > I have added new documentation to the wiki. The first doc is about the upstream > client which can be found at: > > http://wiki.lustre.org/Development#Upstream_contributing > > People have also been approaching me about how to get mlx4 hardware working with > mlx5 hardware so I wrote a wiki about what I did at: > > http://wiki.lustre.org/Infiniband_Configuration_Howto > > _______________________________________________ > lustre-devel mailing list > lustre-devel at lists.lustre.org > http://lists.lustre.org/listinfo.cgi/lustre-devel-lustre.org > From jsimmons at infradead.org Thu May 5 18:52:59 2016 From: jsimmons at infradead.org (James Simmons) Date: Thu, 5 May 2016 14:52:59 -0400 Subject: [lustre-devel] [PATCH 0/8] staging: lustre: o2iblnd: Add Fast Reg memory registration support Message-ID: <1462474387-11406-1-git-send-email-jsimmons@infradead.org> FMR is deprecated and it not supported by the mlx5 driver. This patch adds memory management extensions support as backup of FMR. This was combined with the work from Li Dongyang to make it work with the latest kernels. Dmitry Eremin (8): staging: lustre: o2iblnd: rename kib_fmr_pool_t staging: lustre: o2iblnd: Use list_for_each_entry_safe in kiblnd_destroy_fmr_pool_list staging: lustre: o2iblnd: create union to contain FMR staging: lustre: o2iblnd: break up kiblnd_create_fmr_pool staging: lustre: o2iblnd: cache FMR key in kib_fmr_t staging: lustre: o2iblnd: handle unmapping of FMR in kiblnd_fmr_pool_unmap staging: lustre: o2iblnd: add IBLND_WID_MR staging: lustre: o2iblnd: Add Fast Reg memory registration support .../staging/lustre/lnet/klnds/o2iblnd/o2iblnd.c | 312 +++++++++++++++++--- .../staging/lustre/lnet/klnds/o2iblnd/o2iblnd.h | 33 ++- .../staging/lustre/lnet/klnds/o2iblnd/o2iblnd_cb.c | 58 ++-- 3 files changed, 322 insertions(+), 81 deletions(-) From jsimmons at infradead.org Thu May 5 18:53:04 2016 From: jsimmons at infradead.org (James Simmons) Date: Thu, 5 May 2016 14:53:04 -0400 Subject: [lustre-devel] [PATCH 5/8] staging: lustre: o2iblnd: cache FMR key in kib_fmr_t In-Reply-To: <1462474387-11406-1-git-send-email-jsimmons@infradead.org> References: <1462474387-11406-1-git-send-email-jsimmons@infradead.org> Message-ID: <1462474387-11406-6-git-send-email-jsimmons@infradead.org> From: Dmitry Eremin Move the FMR key handling from kiblnd_fmr_map_tx() to the function kiblnd_fmr_pool_map(). This will allow the function kiblnd_fmr_map_tx() to handle keys for both FMR and Fast Registration. Signed-off-by: Dmitry Eremin Intel-bug-id: https://jira.hpdd.intel.com/browse/LU-5783 Reviewed-on: http://review.whamcloud.com/17606 Reviewed-by: James Simmons Reviewed-by: Doug Oucharek Reviewed-by: Oleg Drokin Signed-off-by: James Simmons --- .../staging/lustre/lnet/klnds/o2iblnd/o2iblnd.c | 11 +++++++---- .../staging/lustre/lnet/klnds/o2iblnd/o2iblnd.h | 5 +++-- .../staging/lustre/lnet/klnds/o2iblnd/o2iblnd_cb.c | 8 ++++---- 3 files changed, 14 insertions(+), 10 deletions(-) diff --git a/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.c b/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.c index 9c30915..491bd6c 100644 --- a/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.c +++ b/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.c @@ -1506,7 +1506,7 @@ void kiblnd_fmr_pool_unmap(kib_fmr_t *fmr, int status) } int kiblnd_fmr_pool_map(kib_fmr_poolset_t *fps, __u64 *pages, int npages, - __u64 iov, kib_fmr_t *fmr) + __u32 nob, __u64 iov, bool is_rx, kib_fmr_t *fmr) { struct ib_pool_fmr *pfmr; kib_fmr_pool_t *fpo; @@ -1524,16 +1524,19 @@ int kiblnd_fmr_pool_map(kib_fmr_poolset_t *fps, __u64 *pages, int npages, pfmr = ib_fmr_pool_map_phys(fpo->fmr.fpo_fmr_pool, pages, npages, iov); if (likely(!IS_ERR(pfmr))) { - fmr->fmr_pool = fpo; + fmr->fmr_key = is_rx ? pfmr->fmr->rkey : + pfmr->fmr->lkey; fmr->fmr_pfmr = pfmr; + fmr->fmr_pool = fpo; return 0; } + rc = PTR_ERR(pfmr); spin_lock(&fps->fps_lock); fpo->fpo_map_count--; - if (PTR_ERR(pfmr) != -EAGAIN) { + if (rc != -EAGAIN) { spin_unlock(&fps->fps_lock); - return PTR_ERR(pfmr); + return rc; } /* EAGAIN and ... */ diff --git a/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.h b/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.h index 0120170..90c8aa5 100644 --- a/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.h +++ b/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.h @@ -308,6 +308,7 @@ typedef struct { typedef struct { kib_fmr_pool_t *fmr_pool; /* pool of FMR */ struct ib_pool_fmr *fmr_pfmr; /* IB pool fmr */ + u32 fmr_key; } kib_fmr_t; typedef struct kib_net { @@ -959,8 +960,8 @@ void kiblnd_unmap_rx_descs(kib_conn_t *conn); void kiblnd_pool_free_node(kib_pool_t *pool, struct list_head *node); struct list_head *kiblnd_pool_alloc_node(kib_poolset_t *ps); -int kiblnd_fmr_pool_map(kib_fmr_poolset_t *fps, __u64 *pages, - int npages, __u64 iov, kib_fmr_t *fmr); +int kiblnd_fmr_pool_map(kib_fmr_poolset_t *fps, __u64 *pages, int npages, + __u32 nob, __u64 iov, bool is_rx, kib_fmr_t *fmr); void kiblnd_fmr_pool_unmap(kib_fmr_t *fmr, int status); int kiblnd_tunables_init(void); diff --git a/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd_cb.c b/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd_cb.c index cd3fde7..b243f1f 100644 --- a/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd_cb.c +++ b/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd_cb.c @@ -561,7 +561,7 @@ kiblnd_kvaddr_to_page(unsigned long vaddr) } static int -kiblnd_fmr_map_tx(kib_net_t *net, kib_tx_t *tx, kib_rdma_desc_t *rd, int nob) +kiblnd_fmr_map_tx(kib_net_t *net, kib_tx_t *tx, kib_rdma_desc_t *rd, __u32 nob) { kib_hca_dev_t *hdev; __u64 *pages = tx->tx_pages; @@ -588,7 +588,8 @@ kiblnd_fmr_map_tx(kib_net_t *net, kib_tx_t *tx, kib_rdma_desc_t *rd, int nob) cpt = tx->tx_pool->tpo_pool.po_owner->ps_cpt; fps = net->ibn_fmr_ps[cpt]; - rc = kiblnd_fmr_pool_map(fps, pages, npages, 0, &tx->fmr); + rc = kiblnd_fmr_pool_map(fps, pages, npages, nob, 0, (rd != tx->tx_rd), + &tx->fmr); if (rc) { CERROR("Can't map %d pages: %d\n", npages, rc); return rc; @@ -598,8 +599,7 @@ kiblnd_fmr_map_tx(kib_net_t *net, kib_tx_t *tx, kib_rdma_desc_t *rd, int nob) * If rd is not tx_rd, it's going to get sent to a peer, who will need * the rkey */ - rd->rd_key = (rd != tx->tx_rd) ? tx->fmr.fmr_pfmr->fmr->rkey : - tx->fmr.fmr_pfmr->fmr->lkey; + rd->rd_key = tx->fmr.fmr_key; rd->rd_frags[0].rf_addr &= ~hdev->ibh_page_mask; rd->rd_frags[0].rf_nob = nob; rd->rd_nfrags = 1; -- 1.7.1 From jsimmons at infradead.org Thu May 5 18:53:03 2016 From: jsimmons at infradead.org (James Simmons) Date: Thu, 5 May 2016 14:53:03 -0400 Subject: [lustre-devel] [PATCH 4/8] staging: lustre: o2iblnd: break up kiblnd_create_fmr_pool In-Reply-To: <1462474387-11406-1-git-send-email-jsimmons@infradead.org> References: <1462474387-11406-1-git-send-email-jsimmons@infradead.org> Message-ID: <1462474387-11406-5-git-send-email-jsimmons@infradead.org> From: Dmitry Eremin Break the function kiblnd_create_fmr_pool() into two functions, with the new function called kiblnd_alloc_fmr_pool(). The function kiblnd_create_fmr_pool() will be used as the front end to allocate any type of pool. The new function will used to create specifically FMR pools. Signed-off-by: Dmitry Eremin Intel-bug-id: https://jira.hpdd.intel.com/browse/LU-5783 Reviewed-on: http://review.whamcloud.com/17606 Reviewed-by: James Simmons Reviewed-by: Doug Oucharek Reviewed-by: Oleg Drokin Signed-off-by: James Simmons --- .../staging/lustre/lnet/klnds/o2iblnd/o2iblnd.c | 55 +++++++++++++++----- 1 files changed, 41 insertions(+), 14 deletions(-) diff --git a/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.c b/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.c index 26d8a11..9c30915 100644 --- a/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.c +++ b/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.c @@ -1335,12 +1335,8 @@ static int kiblnd_fmr_flush_trigger(int ncpts) return max(IBLND_FMR_POOL_FLUSH, size); } -static int kiblnd_create_fmr_pool(kib_fmr_poolset_t *fps, - kib_fmr_pool_t **pp_fpo) +static int kiblnd_alloc_fmr_pool(kib_fmr_poolset_t *fps, kib_fmr_pool_t *fpo) { - /* FMR pool for RDMA */ - kib_dev_t *dev = fps->fps_net->ibn_dev; - kib_fmr_pool_t *fpo; struct ib_fmr_pool_param param = { .max_pages_per_fmr = LNET_MAX_PAYLOAD / PAGE_SIZE, .page_shift = PAGE_SHIFT, @@ -1351,6 +1347,26 @@ static int kiblnd_create_fmr_pool(kib_fmr_poolset_t *fps, .flush_function = NULL, .flush_arg = NULL, .cache = !!*kiblnd_tunables.kib_fmr_cache}; + int rc = 0; + + fpo->fmr.fpo_fmr_pool = ib_create_fmr_pool(fpo->fpo_hdev->ibh_pd, + ¶m); + if (IS_ERR(fpo->fmr.fpo_fmr_pool)) { + rc = PTR_ERR(fpo->fmr.fpo_fmr_pool); + if (rc != -ENOSYS) + CERROR("Failed to create FMR pool: %d\n", rc); + else + CERROR("FMRs are not supported\n"); + } + + return rc; +} + +static int kiblnd_create_fmr_pool(kib_fmr_poolset_t *fps, + kib_fmr_pool_t **pp_fpo) +{ + kib_dev_t *dev = fps->fps_net->ibn_dev; + kib_fmr_pool_t *fpo; int rc; LIBCFS_CPT_ALLOC(fpo, lnet_cpt_table(), fps->fps_cpt, sizeof(*fpo)); @@ -1359,21 +1375,32 @@ static int kiblnd_create_fmr_pool(kib_fmr_poolset_t *fps, fpo->fpo_hdev = kiblnd_current_hdev(dev); - fpo->fmr.fpo_fmr_pool = ib_create_fmr_pool(fpo->fpo_hdev->ibh_pd, ¶m); - if (IS_ERR(fpo->fmr.fpo_fmr_pool)) { - rc = PTR_ERR(fpo->fmr.fpo_fmr_pool); - CERROR("Failed to create FMR pool: %d\n", rc); - - kiblnd_hdev_decref(fpo->fpo_hdev); - LIBCFS_FREE(fpo, sizeof(*fpo)); - return rc; + /* Check for FMR support */ + if (fpo->fpo_hdev->ibh_ibdev->alloc_fmr && + fpo->fpo_hdev->ibh_ibdev->dealloc_fmr && + fpo->fpo_hdev->ibh_ibdev->map_phys_fmr && + fpo->fpo_hdev->ibh_ibdev->unmap_fmr) { + LCONSOLE_INFO("Using FMR for registration\n"); + } else { + rc = -ENOSYS; + LCONSOLE_ERROR_MSG(rc, "IB device does not support FMRs, can't register memory\n"); + goto out_fpo; } + rc = kiblnd_alloc_fmr_pool(fps, fpo); + if (rc) + goto out_fpo; + fpo->fpo_deadline = cfs_time_shift(IBLND_POOL_DEADLINE); - fpo->fpo_owner = fps; + fpo->fpo_owner = fps; *pp_fpo = fpo; return 0; + +out_fpo: + kiblnd_hdev_decref(fpo->fpo_hdev); + LIBCFS_FREE(fpo, sizeof(*fpo)); + return rc; } static void kiblnd_fail_fmr_poolset(kib_fmr_poolset_t *fps, -- 1.7.1 From jsimmons at infradead.org Thu May 5 18:53:07 2016 From: jsimmons at infradead.org (James Simmons) Date: Thu, 5 May 2016 14:53:07 -0400 Subject: [lustre-devel] [PATCH 8/8] staging: lustre: o2iblnd: Add Fast Reg memory registration support In-Reply-To: <1462474387-11406-1-git-send-email-jsimmons@infradead.org> References: <1462474387-11406-1-git-send-email-jsimmons@infradead.org> Message-ID: <1462474387-11406-9-git-send-email-jsimmons@infradead.org> From: Dmitry Eremin FMR is deprecated and it not supported by the mlx5 driver. This patch adds memory management extensions support as backup of FMR. This was combined with the work from Li Dongyang to make it work with the latest kernels. Signed-off-by: Dmitry Eremin Signed-off-by: Li Dongyang Intel-bug-id: https://jira.hpdd.intel.com/browse/LU-5783 Reviewed-on: http://review.whamcloud.com/17606 Reviewed-by: James Simmons Reviewed-by: Doug Oucharek Reviewed-by: Oleg Drokin Signed-off-by: James Simmons --- .../staging/lustre/lnet/klnds/o2iblnd/o2iblnd.c | 239 +++++++++++++++++--- .../staging/lustre/lnet/klnds/o2iblnd/o2iblnd.h | 25 ++- .../staging/lustre/lnet/klnds/o2iblnd/o2iblnd_cb.c | 50 +++-- 3 files changed, 258 insertions(+), 56 deletions(-) diff --git a/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.c b/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.c index fc29d5c..2bb300c 100644 --- a/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.c +++ b/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.c @@ -1302,8 +1302,24 @@ static void kiblnd_destroy_fmr_pool(kib_fmr_pool_t *fpo) { LASSERT(!fpo->fpo_map_count); - if (fpo->fmr.fpo_fmr_pool) - ib_destroy_fmr_pool(fpo->fmr.fpo_fmr_pool); + if (fpo->fpo_is_fmr) { + if (fpo->fmr.fpo_fmr_pool) + ib_destroy_fmr_pool(fpo->fmr.fpo_fmr_pool); + } else { + struct kib_fast_reg_descriptor *frd, *tmp; + int i = 0; + + list_for_each_entry_safe(frd, tmp, &fpo->fast_reg.fpo_pool_list, + frd_list) { + list_del(&frd->frd_list); + ib_dereg_mr(frd->frd_mr); + LIBCFS_FREE(frd, sizeof(*frd)); + i++; + } + if (i < fpo->fast_reg.fpo_pool_size) + CERROR("FastReg pool still has %d regions registered\n", + fpo->fast_reg.fpo_pool_size - i); + } if (fpo->fpo_hdev) kiblnd_hdev_decref(fpo->fpo_hdev); @@ -1362,10 +1378,61 @@ static int kiblnd_alloc_fmr_pool(kib_fmr_poolset_t *fps, kib_fmr_pool_t *fpo) return rc; } +static int kiblnd_alloc_freg_pool(kib_fmr_poolset_t *fps, kib_fmr_pool_t *fpo) +{ + struct kib_fast_reg_descriptor *frd, *tmp; + int i, rc; + + INIT_LIST_HEAD(&fpo->fast_reg.fpo_pool_list); + fpo->fast_reg.fpo_pool_size = 0; + for (i = 0; i < fps->fps_pool_size; i++) { + LIBCFS_CPT_ALLOC(frd, lnet_cpt_table(), fps->fps_cpt, + sizeof(*frd)); + if (!frd) { + CERROR("Failed to allocate a new fast_reg descriptor\n"); + rc = -ENOMEM; + goto out; + } + + frd->frd_mr = ib_alloc_mr(fpo->fpo_hdev->ibh_pd, + IB_MR_TYPE_MEM_REG, + LNET_MAX_PAYLOAD / PAGE_SIZE); + if (IS_ERR(frd->frd_mr)) { + rc = PTR_ERR(frd->frd_mr); + CERROR("Failed to allocate ib_alloc_mr: %d\n", rc); + frd->frd_mr = NULL; + goto out_middle; + } + + frd->frd_valid = true; + + list_add_tail(&frd->frd_list, &fpo->fast_reg.fpo_pool_list); + fpo->fast_reg.fpo_pool_size++; + } + + return 0; + +out_middle: + if (frd->frd_mr) + ib_dereg_mr(frd->frd_mr); + LIBCFS_FREE(frd, sizeof(*frd)); + +out: + list_for_each_entry_safe(frd, tmp, &fpo->fast_reg.fpo_pool_list, + frd_list) { + list_del(&frd->frd_list); + ib_dereg_mr(frd->frd_mr); + LIBCFS_FREE(frd, sizeof(*frd)); + } + + return rc; +} + static int kiblnd_create_fmr_pool(kib_fmr_poolset_t *fps, kib_fmr_pool_t **pp_fpo) { kib_dev_t *dev = fps->fps_net->ibn_dev; + struct ib_device_attr *dev_attr; kib_fmr_pool_t *fpo; int rc; @@ -1374,20 +1441,28 @@ static int kiblnd_create_fmr_pool(kib_fmr_poolset_t *fps, return -ENOMEM; fpo->fpo_hdev = kiblnd_current_hdev(dev); + dev_attr = &fpo->fpo_hdev->ibh_ibdev->attrs; - /* Check for FMR support */ + /* Check for FMR or FastReg support */ + fpo->fpo_is_fmr = 0; if (fpo->fpo_hdev->ibh_ibdev->alloc_fmr && fpo->fpo_hdev->ibh_ibdev->dealloc_fmr && fpo->fpo_hdev->ibh_ibdev->map_phys_fmr && fpo->fpo_hdev->ibh_ibdev->unmap_fmr) { LCONSOLE_INFO("Using FMR for registration\n"); + fpo->fpo_is_fmr = 1; + } else if (dev_attr->device_cap_flags & IB_DEVICE_MEM_MGT_EXTENSIONS) { + LCONSOLE_INFO("Using FastReg for registration\n"); } else { rc = -ENOSYS; - LCONSOLE_ERROR_MSG(rc, "IB device does not support FMRs, can't register memory\n"); + LCONSOLE_ERROR_MSG(rc, "IB device does not support FMRs nor FastRegs, can't register memory\n"); goto out_fpo; } - rc = kiblnd_alloc_fmr_pool(fps, fpo); + if (fpo->fpo_is_fmr) + rc = kiblnd_alloc_fmr_pool(fps, fpo); + else + rc = kiblnd_alloc_freg_pool(fps, fpo); if (rc) goto out_fpo; @@ -1466,6 +1541,28 @@ static int kiblnd_fmr_pool_is_idle(kib_fmr_pool_t *fpo, unsigned long now) return cfs_time_aftereq(now, fpo->fpo_deadline); } +static int +kiblnd_map_tx_pages(kib_tx_t *tx, kib_rdma_desc_t *rd) +{ + __u64 *pages = tx->tx_pages; + kib_hca_dev_t *hdev; + int npages; + int size; + int i; + + hdev = tx->tx_pool->tpo_hdev; + + for (i = 0, npages = 0; i < rd->rd_nfrags; i++) { + for (size = 0; size < rd->rd_frags[i].rf_nob; + size += hdev->ibh_page_size) { + pages[npages++] = (rd->rd_frags[i].rf_addr & + hdev->ibh_page_mask) + size; + } + } + + return npages; +} + void kiblnd_fmr_pool_unmap(kib_fmr_t *fmr, int status) { LIST_HEAD(zombies); @@ -1479,17 +1576,28 @@ void kiblnd_fmr_pool_unmap(kib_fmr_t *fmr, int status) return; fps = fpo->fpo_owner; - if (fmr->fmr_pfmr) { - rc = ib_fmr_pool_unmap(fmr->fmr_pfmr); - LASSERT(!rc); - fmr->fmr_pfmr = NULL; - } + if (fpo->fpo_is_fmr) { + if (fmr->fmr_pfmr) { + rc = ib_fmr_pool_unmap(fmr->fmr_pfmr); + LASSERT(!rc); + fmr->fmr_pfmr = NULL; + } - if (status) { - rc = ib_flush_fmr_pool(fpo->fmr.fpo_fmr_pool); - LASSERT(!rc); - } + if (status) { + rc = ib_flush_fmr_pool(fpo->fmr.fpo_fmr_pool); + LASSERT(!rc); + } + } else { + struct kib_fast_reg_descriptor *frd = fmr->fmr_frd; + if (frd) { + frd->frd_valid = false; + spin_lock(&fps->fps_lock); + list_add_tail(&frd->frd_list, &fpo->fast_reg.fpo_pool_list); + spin_unlock(&fps->fps_lock); + fmr->fmr_frd = NULL; + } + } fmr->fmr_pool = NULL; spin_lock(&fps->fps_lock); @@ -1511,11 +1619,15 @@ void kiblnd_fmr_pool_unmap(kib_fmr_t *fmr, int status) kiblnd_destroy_fmr_pool_list(&zombies); } -int kiblnd_fmr_pool_map(kib_fmr_poolset_t *fps, __u64 *pages, int npages, - __u32 nob, __u64 iov, bool is_rx, kib_fmr_t *fmr) +int kiblnd_fmr_pool_map(kib_fmr_poolset_t *fps, kib_tx_t *tx, + kib_rdma_desc_t *rd, __u32 nob, __u64 iov, + kib_fmr_t *fmr) { - struct ib_pool_fmr *pfmr; + __u64 *pages = tx->tx_pages; + bool is_rx = (rd != tx->tx_rd); + bool tx_pages_mapped = 0; kib_fmr_pool_t *fpo; + int npages = 0; __u64 version; int rc; @@ -1525,18 +1637,89 @@ int kiblnd_fmr_pool_map(kib_fmr_poolset_t *fps, __u64 *pages, int npages, list_for_each_entry(fpo, &fps->fps_pool_list, fpo_list) { fpo->fpo_deadline = cfs_time_shift(IBLND_POOL_DEADLINE); fpo->fpo_map_count++; - spin_unlock(&fps->fps_lock); - pfmr = ib_fmr_pool_map_phys(fpo->fmr.fpo_fmr_pool, - pages, npages, iov); - if (likely(!IS_ERR(pfmr))) { - fmr->fmr_key = is_rx ? pfmr->fmr->rkey : - pfmr->fmr->lkey; - fmr->fmr_pfmr = pfmr; - fmr->fmr_pool = fpo; - return 0; + if (fpo->fpo_is_fmr) { + struct ib_pool_fmr *pfmr; + + spin_unlock(&fps->fps_lock); + + if (!tx_pages_mapped) { + npages = kiblnd_map_tx_pages(tx, rd); + tx_pages_mapped = 1; + } + + pfmr = ib_fmr_pool_map_phys(fpo->fmr.fpo_fmr_pool, + pages, npages, iov); + if (likely(!IS_ERR(pfmr))) { + fmr->fmr_key = is_rx ? pfmr->fmr->rkey : + pfmr->fmr->lkey; + fmr->fmr_frd = NULL; + fmr->fmr_pfmr = pfmr; + fmr->fmr_pool = fpo; + return 0; + } + rc = PTR_ERR(pfmr); + } else { + if (!list_empty(&fpo->fast_reg.fpo_pool_list)) { + struct kib_fast_reg_descriptor *frd; + struct ib_reg_wr *wr; + struct ib_mr *mr; + int n; + + frd = list_first_entry(&fpo->fast_reg.fpo_pool_list, + struct kib_fast_reg_descriptor, + frd_list); + list_del(&frd->frd_list); + spin_unlock(&fps->fps_lock); + + mr = frd->frd_mr; + + if (!frd->frd_valid) { + __u32 key = is_rx ? mr->rkey : mr->lkey; + struct ib_send_wr *inv_wr; + + inv_wr = &frd->frd_inv_wr; + memset(inv_wr, 0, sizeof(*inv_wr)); + inv_wr->opcode = IB_WR_LOCAL_INV; + inv_wr->wr_id = IBLND_WID_MR; + inv_wr->ex.invalidate_rkey = key; + + /* Bump the key */ + key = ib_inc_rkey(key); + ib_update_fast_reg_key(mr, key); + } + + n = ib_map_mr_sg(mr, tx->tx_frags, + tx->tx_nfrags, PAGE_SIZE); + if (unlikely(n != tx->tx_nfrags)) { + CERROR("Failed to map mr %d/%d elements\n", + n, tx->tx_nfrags); + return n < 0 ? n : -EINVAL; + } + + mr->iova = iov; + + /* Prepare FastReg WR */ + wr = &frd->frd_fastreg_wr; + memset(wr, 0, sizeof(*wr)); + wr->wr.opcode = IB_WR_REG_MR; + wr->wr.wr_id = IBLND_WID_MR; + wr->wr.num_sge = 0; + wr->wr.send_flags = 0; + wr->mr = mr; + wr->key = is_rx ? mr->rkey : mr->lkey; + wr->access = (IB_ACCESS_LOCAL_WRITE | + IB_ACCESS_REMOTE_WRITE); + + fmr->fmr_key = is_rx ? mr->rkey : mr->lkey; + fmr->fmr_frd = frd; + fmr->fmr_pfmr = NULL; + fmr->fmr_pool = fpo; + return 0; + } + spin_unlock(&fps->fps_lock); + rc = -EBUSY; } - rc = PTR_ERR(pfmr); spin_lock(&fps->fps_lock); fpo->fpo_map_count--; diff --git a/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.h b/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.h index 277e633..52245e0 100644 --- a/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.h +++ b/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.h @@ -291,6 +291,14 @@ typedef struct { /* failed to allocate */ } kib_fmr_poolset_t; +struct kib_fast_reg_descriptor { /* For fast registration */ + struct list_head frd_list; + struct ib_send_wr frd_inv_wr; + struct ib_reg_wr frd_fastreg_wr; + struct ib_mr *frd_mr; + bool frd_valid; +}; + typedef struct { struct list_head fpo_list; /* chain on pool list */ struct kib_hca_dev *fpo_hdev; /* device for this pool */ @@ -299,16 +307,22 @@ typedef struct { struct { struct ib_fmr_pool *fpo_fmr_pool; /* IB FMR pool */ } fmr; + struct { /* For fast registration */ + struct list_head fpo_pool_list; + int fpo_pool_size; + } fast_reg; }; unsigned long fpo_deadline; /* deadline of this pool */ int fpo_failed; /* fmr pool is failed */ int fpo_map_count; /* # of mapped FMR */ + int fpo_is_fmr; } kib_fmr_pool_t; typedef struct { - kib_fmr_pool_t *fmr_pool; /* pool of FMR */ - struct ib_pool_fmr *fmr_pfmr; /* IB pool fmr */ - u32 fmr_key; + kib_fmr_pool_t *fmr_pool; /* pool of FMR */ + struct ib_pool_fmr *fmr_pfmr; /* IB pool fmr */ + struct kib_fast_reg_descriptor *fmr_frd; + u32 fmr_key; } kib_fmr_t; typedef struct kib_net { @@ -961,8 +975,9 @@ void kiblnd_unmap_rx_descs(kib_conn_t *conn); void kiblnd_pool_free_node(kib_pool_t *pool, struct list_head *node); struct list_head *kiblnd_pool_alloc_node(kib_poolset_t *ps); -int kiblnd_fmr_pool_map(kib_fmr_poolset_t *fps, __u64 *pages, int npages, - __u32 nob, __u64 iov, bool is_rx, kib_fmr_t *fmr); +int kiblnd_fmr_pool_map(kib_fmr_poolset_t *fps, kib_tx_t *tx, + kib_rdma_desc_t *rd, __u32 nob, __u64 iov, + kib_fmr_t *fmr); void kiblnd_fmr_pool_unmap(kib_fmr_t *fmr, int status); int kiblnd_tunables_init(void); diff --git a/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd_cb.c b/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd_cb.c index 7d1c750..c10e615 100644 --- a/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd_cb.c +++ b/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd_cb.c @@ -564,34 +564,20 @@ static int kiblnd_fmr_map_tx(kib_net_t *net, kib_tx_t *tx, kib_rdma_desc_t *rd, __u32 nob) { kib_hca_dev_t *hdev; - __u64 *pages = tx->tx_pages; kib_fmr_poolset_t *fps; - int npages; - int size; int cpt; int rc; - int i; LASSERT(tx->tx_pool); LASSERT(tx->tx_pool->tpo_pool.po_owner); hdev = tx->tx_pool->tpo_hdev; - - for (i = 0, npages = 0; i < rd->rd_nfrags; i++) { - for (size = 0; size < rd->rd_frags[i].rf_nob; - size += hdev->ibh_page_size) { - pages[npages++] = (rd->rd_frags[i].rf_addr & - hdev->ibh_page_mask) + size; - } - } - cpt = tx->tx_pool->tpo_pool.po_owner->ps_cpt; fps = net->ibn_fmr_ps[cpt]; - rc = kiblnd_fmr_pool_map(fps, pages, npages, nob, 0, (rd != tx->tx_rd), - &tx->fmr); + rc = kiblnd_fmr_pool_map(fps, tx, rd, nob, 0, &tx->fmr); if (rc) { - CERROR("Can't map %d pages: %d\n", npages, rc); + CERROR("Can't map %u bytes: %d\n", nob, rc); return rc; } @@ -849,14 +835,26 @@ kiblnd_post_tx_locked(kib_conn_t *conn, kib_tx_t *tx, int credit) /* close_conn will launch failover */ rc = -ENETDOWN; } else { - struct ib_send_wr *wrq = &tx->tx_wrq[tx->tx_nwrq - 1].wr; + struct kib_fast_reg_descriptor *frd = tx->fmr.fmr_frd; + struct ib_send_wr *bad = &tx->tx_wrq[tx->tx_nwrq - 1].wr; + struct ib_send_wr *wrq = &tx->tx_wrq[0].wr; + + if (frd) { + if (!frd->frd_valid) { + wrq = &frd->frd_inv_wr; + wrq->next = &frd->frd_fastreg_wr.wr; + } else { + wrq = &frd->frd_fastreg_wr.wr; + } + frd->frd_fastreg_wr.wr.next = &tx->tx_wrq[0].wr; + } - LASSERTF(wrq->wr_id == kiblnd_ptr2wreqid(tx, IBLND_WID_TX), + LASSERTF(bad->wr_id == kiblnd_ptr2wreqid(tx, IBLND_WID_TX), "bad wr_id %llx, opc %d, flags %d, peer: %s\n", - wrq->wr_id, wrq->opcode, wrq->send_flags, - libcfs_nid2str(conn->ibc_peer->ibp_nid)); - wrq = NULL; - rc = ib_post_send(conn->ibc_cmid->qp, &tx->tx_wrq->wr, &wrq); + bad->wr_id, bad->opcode, bad->send_flags, + libcfs_nid2str(conn->ibc_peer->ibp_nid)); + bad = NULL; + rc = ib_post_send(conn->ibc_cmid->qp, wrq, &bad); } conn->ibc_last_send = jiffies; @@ -1064,7 +1062,7 @@ kiblnd_init_rdma(kib_conn_t *conn, kib_tx_t *tx, int type, kib_msg_t *ibmsg = tx->tx_msg; kib_rdma_desc_t *srcrd = tx->tx_rd; struct ib_sge *sge = &tx->tx_sge[0]; - struct ib_rdma_wr *wrq = &tx->tx_wrq[0], *next; + struct ib_rdma_wr *wrq, *next; int rc = resid; int srcidx = 0; int dstidx = 0; @@ -3428,6 +3426,12 @@ kiblnd_complete(struct ib_wc *wc) default: LBUG(); + case IBLND_WID_MR: + if (wc->status != IB_WC_SUCCESS && + wc->status != IB_WC_WR_FLUSH_ERR) + CNETERR("FastReg failed: %d\n", wc->status); + break; + case IBLND_WID_RDMA: /* * We only get RDMA completion notification if it fails. All -- 1.7.1 From jsimmons at infradead.org Thu May 5 18:53:00 2016 From: jsimmons at infradead.org (James Simmons) Date: Thu, 5 May 2016 14:53:00 -0400 Subject: [lustre-devel] [PATCH 1/8] staging: lustre: o2iblnd: rename kib_fmr_pool_t In-Reply-To: <1462474387-11406-1-git-send-email-jsimmons@infradead.org> References: <1462474387-11406-1-git-send-email-jsimmons@infradead.org> Message-ID: <1462474387-11406-2-git-send-email-jsimmons@infradead.org> From: Dmitry Eremin Rename kib_fmr_pool_t named pool to fpo. Signed-off-by: Dmitry Eremin Intel-bug-id: https://jira.hpdd.intel.com/browse/LU-5783 Reviewed-on: http://review.whamcloud.com/17606 Reviewed-by: James Simmons Reviewed-by: Doug Oucharek Reviewed-by: Oleg Drokin Signed-off-by: James Simmons --- .../staging/lustre/lnet/klnds/o2iblnd/o2iblnd.c | 22 ++++++++++---------- 1 files changed, 11 insertions(+), 11 deletions(-) diff --git a/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.c b/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.c index e89c2a1..60647bc 100644 --- a/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.c +++ b/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.c @@ -1298,27 +1298,27 @@ struct ib_mr *kiblnd_find_rd_dma_mr(kib_hca_dev_t *hdev, kib_rdma_desc_t *rd, return hdev->ibh_mrs; } -static void kiblnd_destroy_fmr_pool(kib_fmr_pool_t *pool) +static void kiblnd_destroy_fmr_pool(kib_fmr_pool_t *fpo) { - LASSERT(!pool->fpo_map_count); + LASSERT(!fpo->fpo_map_count); - if (pool->fpo_fmr_pool) - ib_destroy_fmr_pool(pool->fpo_fmr_pool); + if (fpo->fpo_fmr_pool) + ib_destroy_fmr_pool(fpo->fpo_fmr_pool); - if (pool->fpo_hdev) - kiblnd_hdev_decref(pool->fpo_hdev); + if (fpo->fpo_hdev) + kiblnd_hdev_decref(fpo->fpo_hdev); - LIBCFS_FREE(pool, sizeof(*pool)); + LIBCFS_FREE(fpo, sizeof(*fpo)); } static void kiblnd_destroy_fmr_pool_list(struct list_head *head) { - kib_fmr_pool_t *pool; + kib_fmr_pool_t *fpo; while (!list_empty(head)) { - pool = list_entry(head->next, kib_fmr_pool_t, fpo_list); - list_del(&pool->fpo_list); - kiblnd_destroy_fmr_pool(pool); + fpo = list_entry(head->next, kib_fmr_pool_t, fpo_list); + list_del(&fpo->fpo_list); + kiblnd_destroy_fmr_pool(fpo); } } -- 1.7.1 From jsimmons at infradead.org Thu May 5 18:53:06 2016 From: jsimmons at infradead.org (James Simmons) Date: Thu, 5 May 2016 14:53:06 -0400 Subject: [lustre-devel] [PATCH 7/8] staging: lustre: o2iblnd: add IBLND_WID_MR In-Reply-To: <1462474387-11406-1-git-send-email-jsimmons@infradead.org> References: <1462474387-11406-1-git-send-email-jsimmons@infradead.org> Message-ID: <1462474387-11406-8-git-send-email-jsimmons@infradead.org> From: Dmitry Eremin We need to add a new error field to be passed with wr_id to handle Fast Registration failures. Signed-off-by: Dmitry Eremin Intel-bug-id: https://jira.hpdd.intel.com/browse/LU-5783 Reviewed-on: http://review.whamcloud.com/17606 Reviewed-by: James Simmons Reviewed-by: Doug Oucharek Reviewed-by: Oleg Drokin Signed-off-by: James Simmons --- .../staging/lustre/lnet/klnds/o2iblnd/o2iblnd.h | 3 ++- 1 files changed, 2 insertions(+), 1 deletions(-) diff --git a/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.h b/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.h index 90c8aa5..277e633 100644 --- a/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.h +++ b/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.h @@ -804,7 +804,8 @@ kiblnd_queue2str(kib_conn_t *conn, struct list_head *q) #define IBLND_WID_TX 1 #define IBLND_WID_RX 2 #define IBLND_WID_RDMA 3 -#define IBLND_WID_MASK 3UL +#define IBLND_WID_MR 4 +#define IBLND_WID_MASK 7UL static inline __u64 kiblnd_ptr2wreqid(void *ptr, int type) -- 1.7.1 From jsimmons at infradead.org Thu May 5 18:53:02 2016 From: jsimmons at infradead.org (James Simmons) Date: Thu, 5 May 2016 14:53:02 -0400 Subject: [lustre-devel] [PATCH 3/8] staging: lustre: o2iblnd: create union to contain FMR In-Reply-To: <1462474387-11406-1-git-send-email-jsimmons@infradead.org> References: <1462474387-11406-1-git-send-email-jsimmons@infradead.org> Message-ID: <1462474387-11406-4-git-send-email-jsimmons@infradead.org> From: Dmitry Eremin Create an union to contain the FMR pool structure. This is for the preparation of adding handling Fast Registeration support. Signed-off-by: Dmitry Eremin Intel-bug-id: https://jira.hpdd.intel.com/browse/LU-5783 Reviewed-on: http://review.whamcloud.com/17606 Reviewed-by: James Simmons Reviewed-by: Doug Oucharek Reviewed-by: Oleg Drokin Signed-off-by: James Simmons --- .../staging/lustre/lnet/klnds/o2iblnd/o2iblnd.c | 14 +++++++------- .../staging/lustre/lnet/klnds/o2iblnd/o2iblnd.h | 8 ++++++-- 2 files changed, 13 insertions(+), 9 deletions(-) diff --git a/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.c b/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.c index 4a4b68c..26d8a11 100644 --- a/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.c +++ b/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.c @@ -1302,8 +1302,8 @@ static void kiblnd_destroy_fmr_pool(kib_fmr_pool_t *fpo) { LASSERT(!fpo->fpo_map_count); - if (fpo->fpo_fmr_pool) - ib_destroy_fmr_pool(fpo->fpo_fmr_pool); + if (fpo->fmr.fpo_fmr_pool) + ib_destroy_fmr_pool(fpo->fmr.fpo_fmr_pool); if (fpo->fpo_hdev) kiblnd_hdev_decref(fpo->fpo_hdev); @@ -1359,9 +1359,9 @@ static int kiblnd_create_fmr_pool(kib_fmr_poolset_t *fps, fpo->fpo_hdev = kiblnd_current_hdev(dev); - fpo->fpo_fmr_pool = ib_create_fmr_pool(fpo->fpo_hdev->ibh_pd, ¶m); - if (IS_ERR(fpo->fpo_fmr_pool)) { - rc = PTR_ERR(fpo->fpo_fmr_pool); + fpo->fmr.fpo_fmr_pool = ib_create_fmr_pool(fpo->fpo_hdev->ibh_pd, ¶m); + if (IS_ERR(fpo->fmr.fpo_fmr_pool)) { + rc = PTR_ERR(fpo->fmr.fpo_fmr_pool); CERROR("Failed to create FMR pool: %d\n", rc); kiblnd_hdev_decref(fpo->fpo_hdev); @@ -1452,7 +1452,7 @@ void kiblnd_fmr_pool_unmap(kib_fmr_t *fmr, int status) LASSERT(!rc); if (status) { - rc = ib_flush_fmr_pool(fpo->fpo_fmr_pool); + rc = ib_flush_fmr_pool(fpo->fmr.fpo_fmr_pool); LASSERT(!rc); } @@ -1494,7 +1494,7 @@ int kiblnd_fmr_pool_map(kib_fmr_poolset_t *fps, __u64 *pages, int npages, fpo->fpo_map_count++; spin_unlock(&fps->fps_lock); - pfmr = ib_fmr_pool_map_phys(fpo->fpo_fmr_pool, + pfmr = ib_fmr_pool_map_phys(fpo->fmr.fpo_fmr_pool, pages, npages, iov); if (likely(!IS_ERR(pfmr))) { fmr->fmr_pool = fpo; diff --git a/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.h b/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.h index bfcbdd1..0120170 100644 --- a/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.h +++ b/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.h @@ -295,15 +295,19 @@ typedef struct { struct list_head fpo_list; /* chain on pool list */ struct kib_hca_dev *fpo_hdev; /* device for this pool */ kib_fmr_poolset_t *fpo_owner; /* owner of this pool */ - struct ib_fmr_pool *fpo_fmr_pool; /* IB FMR pool */ + union { + struct { + struct ib_fmr_pool *fpo_fmr_pool; /* IB FMR pool */ + } fmr; + }; unsigned long fpo_deadline; /* deadline of this pool */ int fpo_failed; /* fmr pool is failed */ int fpo_map_count; /* # of mapped FMR */ } kib_fmr_pool_t; typedef struct { - struct ib_pool_fmr *fmr_pfmr; /* IB pool fmr */ kib_fmr_pool_t *fmr_pool; /* pool of FMR */ + struct ib_pool_fmr *fmr_pfmr; /* IB pool fmr */ } kib_fmr_t; typedef struct kib_net { -- 1.7.1 From jsimmons at infradead.org Thu May 5 18:53:05 2016 From: jsimmons at infradead.org (James Simmons) Date: Thu, 5 May 2016 14:53:05 -0400 Subject: [lustre-devel] [PATCH 6/8] staging: lustre: o2iblnd: handle unmapping of FMR in kiblnd_fmr_pool_unmap In-Reply-To: <1462474387-11406-1-git-send-email-jsimmons@infradead.org> References: <1462474387-11406-1-git-send-email-jsimmons@infradead.org> Message-ID: <1462474387-11406-7-git-send-email-jsimmons@infradead.org> From: Dmitry Eremin Move FMR unmapping from kiblnd_unmap_tx() to the function kiblnd_fmr_pool_unmap() so kiblnd_unmap_tx() can be used with the Fast Registration API as well. Signed-off-by: Dmitry Eremin Intel-bug-id: https://jira.hpdd.intel.com/browse/LU-5783 Reviewed-on: http://review.whamcloud.com/17606 Reviewed-by: James Simmons Reviewed-by: Doug Oucharek Reviewed-by: Oleg Drokin Signed-off-by: James Simmons --- .../staging/lustre/lnet/klnds/o2iblnd/o2iblnd.c | 14 ++++++++++---- .../staging/lustre/lnet/klnds/o2iblnd/o2iblnd_cb.c | 4 +--- 2 files changed, 11 insertions(+), 7 deletions(-) diff --git a/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.c b/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.c index 491bd6c..fc29d5c 100644 --- a/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.c +++ b/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.c @@ -1470,13 +1470,20 @@ void kiblnd_fmr_pool_unmap(kib_fmr_t *fmr, int status) { LIST_HEAD(zombies); kib_fmr_pool_t *fpo = fmr->fmr_pool; - kib_fmr_poolset_t *fps = fpo->fpo_owner; + kib_fmr_poolset_t *fps; unsigned long now = cfs_time_current(); kib_fmr_pool_t *tmp; int rc; - rc = ib_fmr_pool_unmap(fmr->fmr_pfmr); - LASSERT(!rc); + if (!fpo) + return; + + fps = fpo->fpo_owner; + if (fmr->fmr_pfmr) { + rc = ib_fmr_pool_unmap(fmr->fmr_pfmr); + LASSERT(!rc); + fmr->fmr_pfmr = NULL; + } if (status) { rc = ib_flush_fmr_pool(fpo->fmr.fpo_fmr_pool); @@ -1484,7 +1491,6 @@ void kiblnd_fmr_pool_unmap(kib_fmr_t *fmr, int status) } fmr->fmr_pool = NULL; - fmr->fmr_pfmr = NULL; spin_lock(&fps->fps_lock); fpo->fpo_map_count--; /* decref the pool */ diff --git a/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd_cb.c b/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd_cb.c index b243f1f..7d1c750 100644 --- a/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd_cb.c +++ b/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd_cb.c @@ -613,10 +613,8 @@ static void kiblnd_unmap_tx(lnet_ni_t *ni, kib_tx_t *tx) LASSERT(net); - if (net->ibn_fmr_ps && tx->fmr.fmr_pfmr) { + if (net->ibn_fmr_ps) kiblnd_fmr_pool_unmap(&tx->fmr, tx->tx_status); - tx->fmr.fmr_pfmr = NULL; - } if (tx->tx_nfrags) { kiblnd_dma_unmap_sg(tx->tx_pool->tpo_hdev->ibh_ibdev, -- 1.7.1 From jsimmons at infradead.org Thu May 5 18:53:01 2016 From: jsimmons at infradead.org (James Simmons) Date: Thu, 5 May 2016 14:53:01 -0400 Subject: [lustre-devel] [PATCH 2/8] staging: lustre: o2iblnd: Use list_for_each_entry_safe in kiblnd_destroy_fmr_pool_list In-Reply-To: <1462474387-11406-1-git-send-email-jsimmons@infradead.org> References: <1462474387-11406-1-git-send-email-jsimmons@infradead.org> Message-ID: <1462474387-11406-3-git-send-email-jsimmons@infradead.org> From: Dmitry Eremin Doubly linked lists which are iterated using list_empty and list_entry macros have been replaced with list_for_each_entry_safe macro. This makes the iteration simpler and more readable. This patch replaces the while loop containing list_empty and list_entry with list_for_each_entry_safe. Signed-off-by: Dmitry Eremin Intel-bug-id: https://jira.hpdd.intel.com/browse/LU-5783 Reviewed-on: http://review.whamcloud.com/17606 Reviewed-by: James Simmons Reviewed-by: Doug Oucharek Reviewed-by: Oleg Drokin Signed-off-by: James Simmons --- .../staging/lustre/lnet/klnds/o2iblnd/o2iblnd.c | 5 ++--- 1 files changed, 2 insertions(+), 3 deletions(-) diff --git a/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.c b/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.c index 60647bc..4a4b68c 100644 --- a/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.c +++ b/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.c @@ -1313,10 +1313,9 @@ static void kiblnd_destroy_fmr_pool(kib_fmr_pool_t *fpo) static void kiblnd_destroy_fmr_pool_list(struct list_head *head) { - kib_fmr_pool_t *fpo; + kib_fmr_pool_t *fpo, *tmp; - while (!list_empty(head)) { - fpo = list_entry(head->next, kib_fmr_pool_t, fpo_list); + list_for_each_entry_safe(fpo, tmp, head, fpo_list) { list_del(&fpo->fpo_list); kiblnd_destroy_fmr_pool(fpo); } -- 1.7.1 From jsimmons at infradead.org Sat May 7 01:30:23 2016 From: jsimmons at infradead.org (James Simmons) Date: Fri, 6 May 2016 21:30:23 -0400 Subject: [lustre-devel] [PATCH 02/10] staging: lustre: lnet: use struct lnet_ioctl_config_data for NI query In-Reply-To: <1462584631-5388-1-git-send-email-jsimmons@infradead.org> References: <1462584631-5388-1-git-send-email-jsimmons@infradead.org> Message-ID: <1462584631-5388-3-git-send-email-jsimmons@infradead.org> From: Amir Shehata The functions used to query the NI state passes around the individual fields of struct lnet_ioctl_config_data. To simplify the code lets just pass struct lnet_ioctl_config directly. Signed-off-by: Amir Shehata Signed-off-by: James Simmons Intel-bug-id: https://jira.hpdd.intel.com/browse/LU-7101 Reviewed-on: http://review.whamcloud.com/16367 Reviewed-by: Doug Oucharek Reviewed-by: Olaf Weber Reviewed-by: Oleg Drokin Signed-off-by: James Simmons --- .../staging/lustre/include/linux/lnet/lib-lnet.h | 4 - drivers/staging/lustre/lnet/lnet/api-ni.c | 60 ++++++-------------- 2 files changed, 18 insertions(+), 46 deletions(-) diff --git a/drivers/staging/lustre/include/linux/lnet/lib-lnet.h b/drivers/staging/lustre/include/linux/lnet/lib-lnet.h index 1865c85..513a822 100644 --- a/drivers/staging/lustre/include/linux/lnet/lib-lnet.h +++ b/drivers/staging/lustre/include/linux/lnet/lib-lnet.h @@ -463,10 +463,6 @@ int lnet_del_route(__u32 net, lnet_nid_t gw_nid); void lnet_destroy_routes(void); int lnet_get_route(int idx, __u32 *net, __u32 *hops, lnet_nid_t *gateway, __u32 *alive, __u32 *priority); -int lnet_get_net_config(int idx, __u32 *cpt_count, __u64 *nid, - int *peer_timeout, int *peer_tx_credits, - int *peer_rtr_cr, int *max_tx_credits, - struct lnet_ioctl_net_config *net_config); int lnet_get_rtr_pool_cfg(int idx, struct lnet_ioctl_pool_cfg *pool_cfg); void lnet_router_debugfs_init(void); diff --git a/drivers/staging/lustre/lnet/lnet/api-ni.c b/drivers/staging/lustre/lnet/lnet/api-ni.c index 6340ad1..01e1808 100644 --- a/drivers/staging/lustre/lnet/lnet/api-ni.c +++ b/drivers/staging/lustre/lnet/lnet/api-ni.c @@ -1648,25 +1648,18 @@ EXPORT_SYMBOL(LNetNIFini); * parameters * * \param[in] ni network interface structure - * \param[out] cpt_count the number of cpts the ni is on - * \param[out] nid Network Interface ID - * \param[out] peer_timeout NI peer timeout - * \param[out] peer_tx_crdits NI peer transmit credits - * \param[out] peer_rtr_credits NI peer router credits - * \param[out] max_tx_credits NI max transmit credit - * \param[out] net_config Network configuration + * \param[out] config NI configuration */ static void -lnet_fill_ni_info(struct lnet_ni *ni, __u32 *cpt_count, __u64 *nid, - int *peer_timeout, int *peer_tx_credits, - int *peer_rtr_credits, int *max_tx_credits, - struct lnet_ioctl_net_config *net_config) +lnet_fill_ni_info(struct lnet_ni *ni, struct lnet_ioctl_config_data *config) { + struct lnet_ioctl_net_config *net_config; int i; - if (!ni) + if (!ni || !config) return; + net_config = (struct lnet_ioctl_net_config *) config->cfg_bulk; if (!net_config) return; @@ -1682,11 +1675,11 @@ lnet_fill_ni_info(struct lnet_ni *ni, __u32 *cpt_count, __u64 *nid, sizeof(net_config->ni_interfaces[i])); } - *nid = ni->ni_nid; - *peer_timeout = ni->ni_peertimeout; - *peer_tx_credits = ni->ni_peertxcredits; - *peer_rtr_credits = ni->ni_peerrtrcredits; - *max_tx_credits = ni->ni_maxtxcredits; + config->cfg_nid = ni->ni_nid; + config->cfg_config_u.cfg_net.net_peer_timeout = ni->ni_peertimeout; + config->cfg_config_u.cfg_net.net_max_tx_credits = ni->ni_maxtxcredits; + config->cfg_config_u.cfg_net.net_peer_tx_credits = ni->ni_peertxcredits; + config->cfg_config_u.cfg_net.net_peer_rtr_credits = ni->ni_peerrtrcredits; net_config->ni_status = ni->ni_status->ns_status; @@ -1696,18 +1689,16 @@ lnet_fill_ni_info(struct lnet_ni *ni, __u32 *cpt_count, __u64 *nid, for (i = 0; i < num_cpts; i++) net_config->ni_cpts[i] = ni->ni_cpts[i]; - *cpt_count = num_cpts; + config->cfg_ncpts = num_cpts; } } -int -lnet_get_net_config(int idx, __u32 *cpt_count, __u64 *nid, int *peer_timeout, - int *peer_tx_credits, int *peer_rtr_credits, - int *max_tx_credits, - struct lnet_ioctl_net_config *net_config) +static int +lnet_get_net_config(struct lnet_ioctl_config_data *config) { struct lnet_ni *ni; struct list_head *tmp; + int idx = config->cfg_count; int cpt, i = 0; int rc = -ENOENT; @@ -1719,9 +1710,7 @@ lnet_get_net_config(int idx, __u32 *cpt_count, __u64 *nid, int *peer_timeout, ni = list_entry(tmp, lnet_ni_t, ni_list); lnet_ni_lock(ni); - lnet_fill_ni_info(ni, cpt_count, nid, peer_timeout, - peer_tx_credits, peer_rtr_credits, - max_tx_credits, net_config); + lnet_fill_ni_info(ni, config); lnet_ni_unlock(ni); rc = 0; break; @@ -1927,27 +1916,14 @@ LNetCtl(unsigned int cmd, void *arg) &config->cfg_config_u.cfg_route.rtr_priority); case IOC_LIBCFS_GET_NET: { - struct lnet_ioctl_net_config *net_config; - size_t total = sizeof(*config) + sizeof(*net_config); - + size_t total = sizeof(*config) + + sizeof(struct lnet_ioctl_net_config); config = arg; if (config->cfg_hdr.ioc_len < total) return -EINVAL; - net_config = (struct lnet_ioctl_net_config *) - config->cfg_bulk; - if (!net_config) - return -EINVAL; - - return lnet_get_net_config(config->cfg_count, - &config->cfg_ncpts, - &config->cfg_nid, - &config->cfg_config_u.cfg_net.net_peer_timeout, - &config->cfg_config_u.cfg_net.net_peer_tx_credits, - &config->cfg_config_u.cfg_net.net_peer_rtr_credits, - &config->cfg_config_u.cfg_net.net_max_tx_credits, - net_config); + return lnet_get_net_config(config); } case IOC_LIBCFS_GET_LNET_STATS: { -- 1.7.1 From jsimmons at infradead.org Sat May 7 01:30:28 2016 From: jsimmons at infradead.org (James Simmons) Date: Fri, 6 May 2016 21:30:28 -0400 Subject: [lustre-devel] [PATCH 07/10] staging: lustre: o2iblnd: per NI map-on-demand value In-Reply-To: <1462584631-5388-1-git-send-email-jsimmons@infradead.org> References: <1462584631-5388-1-git-send-email-jsimmons@infradead.org> Message-ID: <1462584631-5388-8-git-send-email-jsimmons@infradead.org> From: Amir Shehata Enables support of different map-on-demand values per NI. This is required to support OPA coexistence with MLX5 cards. MLX5 does not support FMR, which is enabled via map-on-demand. However OPA's performance is greatly enahanced when FMR is enabled. In order to enable coexistence of both of these two types of cards we need to be able to set different map-on-demand values for both NIs. This patch also lays the ground work for other per NI tunables to be added in future patches. Signed-off-by: Amir Shehata Signed-off-by: James Simmons Intel-bug-id: https://jira.hpdd.intel.com/browse/LU-7101 Reviewed-on: http://review.whamcloud.com/16367 Reviewed-by: Doug Oucharek Reviewed-by: Olaf Weber Reviewed-by: Oleg Drokin Signed-off-by: James Simmons --- .../staging/lustre/lnet/klnds/o2iblnd/o2iblnd.c | 66 ++++++++++------ .../staging/lustre/lnet/klnds/o2iblnd/o2iblnd.h | 35 +++++---- .../staging/lustre/lnet/klnds/o2iblnd/o2iblnd_cb.c | 13 ++- .../lustre/lnet/klnds/o2iblnd/o2iblnd_modparams.c | 84 +++++++++++++------ 4 files changed, 125 insertions(+), 73 deletions(-) diff --git a/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.c b/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.c index 84a15d4..7711d96 100644 --- a/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.c +++ b/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.c @@ -1283,16 +1283,22 @@ static void kiblnd_map_tx_pool(kib_tx_pool_t *tpo) } } -struct ib_mr *kiblnd_find_rd_dma_mr(kib_hca_dev_t *hdev, kib_rdma_desc_t *rd, +struct ib_mr *kiblnd_find_rd_dma_mr(struct lnet_ni *ni, kib_rdma_desc_t *rd, int negotiated_nfrags) { - __u16 nfrags = (negotiated_nfrags != -1) ? - negotiated_nfrags : *kiblnd_tunables.kib_map_on_demand; + kib_net_t *net = ni->ni_data; + kib_hca_dev_t *hdev = net->ibn_dev->ibd_hdev; + struct lnet_ioctl_config_o2iblnd_tunables *tunables; + __u16 nfrags; + int mod; + + tunables = &ni->ni_lnd_tunables->lt_tun_u.lt_o2ib; + mod = tunables->lnd_map_on_demand; + nfrags = (negotiated_nfrags != -1) ? negotiated_nfrags : mod; LASSERT(hdev->ibh_mrs); - if (*kiblnd_tunables.kib_map_on_demand > 0 && - nfrags <= rd->rd_nfrags) + if (mod > 0 && nfrags <= rd->rd_nfrags) return NULL; return hdev->ibh_mrs; @@ -1337,16 +1343,20 @@ static void kiblnd_destroy_fmr_pool_list(struct list_head *head) } } -static int kiblnd_fmr_pool_size(int ncpts) +static int +kiblnd_fmr_pool_size(struct lnet_ioctl_config_o2iblnd_tunables *tunables, + int ncpts) { - int size = *kiblnd_tunables.kib_fmr_pool_size / ncpts; + int size = tunables->lnd_fmr_pool_size / ncpts; return max(IBLND_FMR_POOL, size); } -static int kiblnd_fmr_flush_trigger(int ncpts) +static int +kiblnd_fmr_flush_trigger(struct lnet_ioctl_config_o2iblnd_tunables *tunables, + int ncpts) { - int size = *kiblnd_tunables.kib_fmr_flush_trigger / ncpts; + int size = tunables->lnd_fmr_flush_trigger / ncpts; return max(IBLND_FMR_POOL_FLUSH, size); } @@ -1362,7 +1372,7 @@ static int kiblnd_alloc_fmr_pool(kib_fmr_poolset_t *fps, kib_fmr_pool_t *fpo) .dirty_watermark = fps->fps_flush_trigger, .flush_function = NULL, .flush_arg = NULL, - .cache = !!*kiblnd_tunables.kib_fmr_cache}; + .cache = !!fps->fps_cache }; int rc = 0; fpo->fmr.fpo_fmr_pool = ib_create_fmr_pool(fpo->fpo_hdev->ibh_pd, @@ -1508,9 +1518,10 @@ static void kiblnd_fini_fmr_poolset(kib_fmr_poolset_t *fps) } } -static int kiblnd_init_fmr_poolset(kib_fmr_poolset_t *fps, int cpt, - kib_net_t *net, int pool_size, - int flush_trigger) +static int +kiblnd_init_fmr_poolset(kib_fmr_poolset_t *fps, int cpt, int ncpts, + kib_net_t *net, + struct lnet_ioctl_config_o2iblnd_tunables *tunables) { kib_fmr_pool_t *fpo; int rc; @@ -1519,8 +1530,11 @@ static int kiblnd_init_fmr_poolset(kib_fmr_poolset_t *fps, int cpt, fps->fps_net = net; fps->fps_cpt = cpt; - fps->fps_pool_size = pool_size; - fps->fps_flush_trigger = flush_trigger; + + fps->fps_pool_size = kiblnd_fmr_pool_size(tunables, ncpts); + fps->fps_flush_trigger = kiblnd_fmr_flush_trigger(tunables, ncpts); + fps->fps_cache = tunables->lnd_fmr_cache; + spin_lock_init(&fps->fps_lock); INIT_LIST_HEAD(&fps->fps_pool_list); INIT_LIST_HEAD(&fps->fps_failed_pool_list); @@ -2151,25 +2165,28 @@ static void kiblnd_net_fini_pools(kib_net_t *net) } } -static int kiblnd_net_init_pools(kib_net_t *net, __u32 *cpts, int ncpts) +static int kiblnd_net_init_pools(kib_net_t *net, lnet_ni_t *ni, __u32 *cpts, + int ncpts) { + struct lnet_ioctl_config_o2iblnd_tunables *tunables; unsigned long flags; int cpt; - int rc = 0; + int rc; int i; + tunables = &ni->ni_lnd_tunables->lt_tun_u.lt_o2ib; + read_lock_irqsave(&kiblnd_data.kib_global_lock, flags); - if (!*kiblnd_tunables.kib_map_on_demand) { + if (!tunables->lnd_map_on_demand) { read_unlock_irqrestore(&kiblnd_data.kib_global_lock, flags); goto create_tx_pool; } read_unlock_irqrestore(&kiblnd_data.kib_global_lock, flags); - if (*kiblnd_tunables.kib_fmr_pool_size < - *kiblnd_tunables.kib_ntx / 4) { + if (tunables->lnd_fmr_pool_size < *kiblnd_tunables.kib_ntx / 4) { CERROR("Can't set fmr pool size (%d) < ntx / 4(%d)\n", - *kiblnd_tunables.kib_fmr_pool_size, + tunables->lnd_fmr_pool_size, *kiblnd_tunables.kib_ntx / 4); rc = -EINVAL; goto failed; @@ -2199,9 +2216,8 @@ static int kiblnd_net_init_pools(kib_net_t *net, __u32 *cpts, int ncpts) for (i = 0; i < ncpts; i++) { cpt = !cpts ? i : cpts[i]; - rc = kiblnd_init_fmr_poolset(net->ibn_fmr_ps[cpt], cpt, net, - kiblnd_fmr_pool_size(ncpts), - kiblnd_fmr_flush_trigger(ncpts)); + rc = kiblnd_init_fmr_poolset(net->ibn_fmr_ps[cpt], cpt, ncpts, + net, tunables); if (rc) { CERROR("Can't initialize FMR pool for CPT %d: %d\n", cpt, rc); @@ -2962,7 +2978,7 @@ static int kiblnd_startup(lnet_ni_t *ni) if (rc) goto failed; - rc = kiblnd_net_init_pools(net, ni->ni_cpts, ni->ni_ncpts); + rc = kiblnd_net_init_pools(net, ni, ni->ni_cpts, ni->ni_ncpts); if (rc) { CERROR("Failed to initialize NI pools: %d\n", rc); goto failed; diff --git a/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.h b/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.h index fffae0c..d458773 100644 --- a/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.h +++ b/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.h @@ -87,18 +87,10 @@ typedef struct { int *kib_timeout; /* comms timeout (seconds) */ int *kib_keepalive; /* keepalive timeout (seconds) */ int *kib_ntx; /* # tx descs */ - int *kib_peercredits_hiw; /* # when eagerly to return credits */ char **kib_default_ipif; /* default IPoIB interface */ int *kib_retry_count; int *kib_rnr_retry_count; - int *kib_concurrent_sends; /* send work queue sizing */ int *kib_ib_mtu; /* IB MTU */ - int *kib_map_on_demand; /* map-on-demand if RD has more */ - /* fragments than this value, 0 */ - /* disable map-on-demand */ - int *kib_fmr_pool_size; /* # FMRs in pool */ - int *kib_fmr_flush_trigger; /* When to trigger FMR flush */ - int *kib_fmr_cache; /* enable FMR pool cache? */ int *kib_require_priv_port; /* accept only privileged ports */ int *kib_use_priv_port; /* use privileged port for active connect */ int *kib_nscheds; /* # threads on each CPT */ @@ -112,9 +104,10 @@ extern kib_tunables_t kiblnd_tunables; #define IBLND_CREDITS_DEFAULT 8 /* default # of peer credits */ #define IBLND_CREDITS_MAX ((typeof(((kib_msg_t *) 0)->ibm_credits)) - 1) /* Max # of peer credits */ -#define IBLND_CREDITS_HIGHWATER(v) ((v) == IBLND_MSG_VERSION_1 ? \ - IBLND_CREDIT_HIGHWATER_V1 : \ - *kiblnd_tunables.kib_peercredits_hiw) /* when eagerly to return credits */ +/* when eagerly to return credits */ +#define IBLND_CREDITS_HIGHWATER(t, v) ((v) == IBLND_MSG_VERSION_1 ? \ + IBLND_CREDIT_HIGHWATER_V1 : \ + t->lnd_peercredits_hiw) #define kiblnd_rdma_create_id(cb, dev, ps, qpt) rdma_create_id(&init_net, \ cb, dev, \ @@ -260,6 +253,7 @@ typedef struct { int fps_cpt; /* CPT id */ int fps_pool_size; int fps_flush_trigger; + int fps_cache; int fps_increasing; /* is allocating new pool */ unsigned long fps_next_retry; /* time stamp for retry if*/ /* failed to allocate */ @@ -614,7 +608,11 @@ int kiblnd_msg_queue_size(int version, struct lnet_ni *ni); static inline int kiblnd_cfg_rdma_frags(struct lnet_ni *ni) { - int mod = *kiblnd_tunables.kib_map_on_demand; + struct lnet_ioctl_config_o2iblnd_tunables *tunables; + int mod; + + tunables = &ni->ni_lnd_tunables->lt_tun_u.lt_o2ib; + mod = tunables->lnd_map_on_demand; return mod ? mod : IBLND_MAX_RDMA_FRAGS; } @@ -629,9 +627,11 @@ kiblnd_rdma_frags(int version, struct lnet_ni *ni) static inline int kiblnd_concurrent_sends(int version, struct lnet_ni *ni) { + struct lnet_ioctl_config_o2iblnd_tunables *tunables; int concurrent_sends; - concurrent_sends = *kiblnd_tunables.kib_concurrent_sends; + tunables = &ni->ni_lnd_tunables->lt_tun_u.lt_o2ib; + concurrent_sends = tunables->lnd_concurrent_sends; if (version == IBLND_MSG_VERSION_1) { if (concurrent_sends > IBLND_MSG_QUEUE_SIZE_V1 * 2) @@ -766,10 +766,14 @@ kiblnd_send_keepalive(kib_conn_t *conn) static inline int kiblnd_need_noop(kib_conn_t *conn) { + struct lnet_ioctl_config_o2iblnd_tunables *tunables; + lnet_ni_t *ni = conn->ibc_peer->ibp_ni; + LASSERT(conn->ibc_state >= IBLND_CONN_ESTABLISHED); + tunables = &ni->ni_lnd_tunables->lt_tun_u.lt_o2ib; if (conn->ibc_outstanding_credits < - IBLND_CREDITS_HIGHWATER(conn->ibc_version) && + IBLND_CREDITS_HIGHWATER(tunables, conn->ibc_version) && !kiblnd_send_keepalive(conn)) return 0; /* No need to send NOOP */ @@ -977,8 +981,7 @@ static inline unsigned int kiblnd_sg_dma_len(struct ib_device *dev, #define KIBLND_CONN_PARAM(e) ((e)->param.conn.private_data) #define KIBLND_CONN_PARAM_LEN(e) ((e)->param.conn.private_data_len) -struct ib_mr *kiblnd_find_rd_dma_mr(kib_hca_dev_t *hdev, - kib_rdma_desc_t *rd, +struct ib_mr *kiblnd_find_rd_dma_mr(struct lnet_ni *ni, kib_rdma_desc_t *rd, int negotiated_nfrags); void kiblnd_map_rx_descs(kib_conn_t *conn); void kiblnd_unmap_rx_descs(kib_conn_t *conn); diff --git a/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd_cb.c b/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd_cb.c index c3bd7fa..c4daf95 100644 --- a/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd_cb.c +++ b/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd_cb.c @@ -612,8 +612,8 @@ static void kiblnd_unmap_tx(lnet_ni_t *ni, kib_tx_t *tx) static int kiblnd_map_tx(lnet_ni_t *ni, kib_tx_t *tx, kib_rdma_desc_t *rd, int nfrags) { - kib_hca_dev_t *hdev = tx->tx_pool->tpo_hdev; kib_net_t *net = ni->ni_data; + kib_hca_dev_t *hdev = net->ibn_dev->ibd_hdev; struct ib_mr *mr = NULL; __u32 nob; int i; @@ -636,7 +636,7 @@ static int kiblnd_map_tx(lnet_ni_t *ni, kib_tx_t *tx, kib_rdma_desc_t *rd, nob += rd->rd_frags[i].rf_nob; } - mr = kiblnd_find_rd_dma_mr(hdev, rd, tx->tx_conn ? + mr = kiblnd_find_rd_dma_mr(ni, rd, tx->tx_conn ? tx->tx_conn->ibc_max_frags : -1); if (mr) { /* found pre-mapping MR */ @@ -2585,12 +2585,15 @@ kiblnd_check_reconnect(kib_conn_t *conn, int version, reason = "Unknown"; break; - case IBLND_REJECT_RDMA_FRAGS: + case IBLND_REJECT_RDMA_FRAGS: { + struct lnet_ioctl_config_lnd_tunables *tunables; + if (!cp) { reason = "can't negotiate max frags"; goto out; } - if (!*kiblnd_tunables.kib_map_on_demand) { + tunables = peer->ibp_ni->ni_lnd_tunables; + if (!tunables->lt_tun_u.lt_o2ib.lnd_map_on_demand) { reason = "map_on_demand must be enabled"; goto out; } @@ -2602,7 +2605,7 @@ kiblnd_check_reconnect(kib_conn_t *conn, int version, peer->ibp_max_frags = frag_num; reason = "rdma fragments"; break; - + } case IBLND_REJECT_MSG_QUEUE_SIZE: if (!cp) { reason = "can't negotiate queue depth"; diff --git a/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd_modparams.c b/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd_modparams.c index e50a9cf..f8fdd4a 100644 --- a/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd_modparams.c +++ b/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd_modparams.c @@ -152,16 +152,10 @@ kib_tunables_t kiblnd_tunables = { .kib_timeout = &timeout, .kib_keepalive = &keepalive, .kib_ntx = &ntx, - .kib_peercredits_hiw = &peer_credits_hiw, .kib_default_ipif = &ipif_name, .kib_retry_count = &retry_count, .kib_rnr_retry_count = &rnr_retry_count, - .kib_concurrent_sends = &concurrent_sends, .kib_ib_mtu = &ib_mtu, - .kib_map_on_demand = &map_on_demand, - .kib_fmr_pool_size = &fmr_pool_size, - .kib_fmr_flush_trigger = &fmr_flush_trigger, - .kib_fmr_cache = &fmr_cache, .kib_require_priv_port = &require_privileged_port, .kib_use_priv_port = &use_privileged_port, .kib_nscheds = &nscheds @@ -182,6 +176,26 @@ int kiblnd_msg_queue_size(int version, lnet_ni_t *ni) int kiblnd_tunables_setup(struct lnet_ni *ni) { + struct lnet_ioctl_config_o2iblnd_tunables *tunables; + + /* + * if there was no tunables specified, setup the tunables to be + * defaulted + */ + if (!ni->ni_lnd_tunables) { + LIBCFS_ALLOC(ni->ni_lnd_tunables, + sizeof(*ni->ni_lnd_tunables)); + if (!ni->ni_lnd_tunables) + return -ENOMEM; + + memcpy(&ni->ni_lnd_tunables->lt_tun_u.lt_o2ib, + &default_tunables, sizeof(*tunables)); + } + tunables = &ni->ni_lnd_tunables->lt_tun_u.lt_o2ib; + + /* Current API version */ + tunables->lnd_version = 0; + if (kiblnd_translate_mtu(*kiblnd_tunables.kib_ib_mtu) < 0) { CERROR("Invalid ib_mtu %d, expected 256/512/1024/2048/4096\n", *kiblnd_tunables.kib_ib_mtu); @@ -209,38 +223,54 @@ int kiblnd_tunables_setup(struct lnet_ni *ni) if (ni->ni_peertxcredits > credits) ni->ni_peertxcredits = credits; - if (*kiblnd_tunables.kib_peercredits_hiw < ni->ni_peertxcredits / 2) - *kiblnd_tunables.kib_peercredits_hiw = ni->ni_peertxcredits / 2; + if (!tunables->lnd_peercredits_hiw) + tunables->lnd_peercredits_hiw = peer_credits_hiw; - if (*kiblnd_tunables.kib_peercredits_hiw >= ni->ni_peertxcredits) - *kiblnd_tunables.kib_peercredits_hiw = ni->ni_peertxcredits - 1; + if (tunables->lnd_peercredits_hiw < ni->ni_peertxcredits / 2) + tunables->lnd_peercredits_hiw = ni->ni_peertxcredits / 2; - if (*kiblnd_tunables.kib_map_on_demand < 0 || - *kiblnd_tunables.kib_map_on_demand > IBLND_MAX_RDMA_FRAGS) - *kiblnd_tunables.kib_map_on_demand = 0; /* disable map-on-demand */ + if (tunables->lnd_peercredits_hiw >= ni->ni_peertxcredits) + tunables->lnd_peercredits_hiw = ni->ni_peertxcredits - 1; - if (*kiblnd_tunables.kib_map_on_demand == 1) - *kiblnd_tunables.kib_map_on_demand = 2; /* don't make sense to create map if only one fragment */ + if (tunables->lnd_map_on_demand < 0 || + tunables->lnd_map_on_demand > IBLND_MAX_RDMA_FRAGS) { + /* disable map-on-demand */ + tunables->lnd_map_on_demand = 0; + } - if (!*kiblnd_tunables.kib_concurrent_sends) { - if (*kiblnd_tunables.kib_map_on_demand > 0 && - *kiblnd_tunables.kib_map_on_demand <= IBLND_MAX_RDMA_FRAGS / 8) - *kiblnd_tunables.kib_concurrent_sends = ni->ni_peertxcredits * 2; - else - *kiblnd_tunables.kib_concurrent_sends = ni->ni_peertxcredits; + if (tunables->lnd_map_on_demand == 1) { + /* don't make sense to create map if only one fragment */ + tunables->lnd_map_on_demand = 2; } - if (*kiblnd_tunables.kib_concurrent_sends > ni->ni_peertxcredits * 2) - *kiblnd_tunables.kib_concurrent_sends = ni->ni_peertxcredits * 2; + if (!tunables->lnd_concurrent_sends) { + if (tunables->lnd_map_on_demand > 0 && + tunables->lnd_map_on_demand <= IBLND_MAX_RDMA_FRAGS / 8) { + tunables->lnd_concurrent_sends = + ni->ni_peertxcredits * 2; + } else { + tunables->lnd_concurrent_sends = ni->ni_peertxcredits; + } + } + + if (tunables->lnd_concurrent_sends > ni->ni_peertxcredits * 2) + tunables->lnd_concurrent_sends = ni->ni_peertxcredits * 2; - if (*kiblnd_tunables.kib_concurrent_sends < ni->ni_peertxcredits / 2) - *kiblnd_tunables.kib_concurrent_sends = ni->ni_peertxcredits / 2; + if (tunables->lnd_concurrent_sends < ni->ni_peertxcredits / 2) + tunables->lnd_concurrent_sends = ni->ni_peertxcredits / 2; - if (*kiblnd_tunables.kib_concurrent_sends < ni->ni_peertxcredits) { + if (tunables->lnd_concurrent_sends < ni->ni_peertxcredits) { CWARN("Concurrent sends %d is lower than message queue size: %d, performance may drop slightly.\n", - *kiblnd_tunables.kib_concurrent_sends, ni->ni_peertxcredits); + tunables->lnd_concurrent_sends, ni->ni_peertxcredits); } + if (!tunables->lnd_fmr_pool_size) + tunables->lnd_fmr_pool_size = fmr_pool_size; + if (!tunables->lnd_fmr_flush_trigger) + tunables->lnd_fmr_flush_trigger = fmr_flush_trigger; + if (!tunables->lnd_fmr_cache) + tunables->lnd_fmr_cache = fmr_cache; + return 0; } -- 1.7.1 From jsimmons at infradead.org Sat May 7 01:30:29 2016 From: jsimmons at infradead.org (James Simmons) Date: Fri, 6 May 2016 21:30:29 -0400 Subject: [lustre-devel] [PATCH 08/10] staging: lustre: o2iblnd: remove last of kiblnd_tunables_fini In-Reply-To: <1462584631-5388-1-git-send-email-jsimmons@infradead.org> References: <1462584631-5388-1-git-send-email-jsimmons@infradead.org> Message-ID: <1462584631-5388-9-git-send-email-jsimmons@infradead.org> From: Amir Shehata The function kibnd_tunables_fini() no longer exist. Remove it from o2iblnd.h Signed-off-by: Amir Shehata Signed-off-by: James Simmons Intel-bug-id: https://jira.hpdd.intel.com/browse/LU-7101 Reviewed-on: http://review.whamcloud.com/16367 Reviewed-by: Doug Oucharek Reviewed-by: Olaf Weber Reviewed-by: Oleg Drokin Signed-off-by: James Simmons --- .../staging/lustre/lnet/klnds/o2iblnd/o2iblnd.h | 1 - 1 files changed, 0 insertions(+), 1 deletions(-) diff --git a/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.h b/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.h index d458773..b22984f 100644 --- a/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.h +++ b/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.h @@ -995,7 +995,6 @@ void kiblnd_fmr_pool_unmap(kib_fmr_t *fmr, int status); int kiblnd_tunables_setup(struct lnet_ni *ni); void kiblnd_tunables_init(void); -void kiblnd_tunables_fini(void); int kiblnd_connd(void *arg); int kiblnd_scheduler(void *arg); -- 1.7.1 From jsimmons at infradead.org Sat May 7 01:30:30 2016 From: jsimmons at infradead.org (James Simmons) Date: Fri, 6 May 2016 21:30:30 -0400 Subject: [lustre-devel] [PATCH 09/10] staging: lustre: o2iblnd: properly set ibr_why In-Reply-To: <1462584631-5388-1-git-send-email-jsimmons@infradead.org> References: <1462584631-5388-1-git-send-email-jsimmons@infradead.org> Message-ID: <1462584631-5388-10-git-send-email-jsimmons@infradead.org> From: Amir Shehata For the case of when the connections supported fragment count is smaller than what is supported locally only set ibr_why to IBLND_REJECT_RDMA_FRAGS if the ko2iblnd protocol verison is IBLND_MSG_VERSION. Signed-off-by: Amir Shehata Signed-off-by: James Simmons Intel-bug-id: https://jira.hpdd.intel.com/browse/LU-7101 Reviewed-on: http://review.whamcloud.com/16367 Reviewed-by: Doug Oucharek Reviewed-by: Olaf Weber Reviewed-by: Oleg Drokin Signed-off-by: James Simmons --- .../staging/lustre/lnet/klnds/o2iblnd/o2iblnd_cb.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd_cb.c b/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd_cb.c index c4daf95..0f7e3a1 100644 --- a/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd_cb.c +++ b/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd_cb.c @@ -2368,7 +2368,7 @@ kiblnd_passive_connect(struct rdma_cm_id *cmid, void *priv, int priv_nob) reqmsg->ibm_u.connparams.ibcp_max_frags, kiblnd_rdma_frags(version, ni)); - if (version >= IBLND_MSG_VERSION) + if (version == IBLND_MSG_VERSION) rej.ibr_why = IBLND_REJECT_RDMA_FRAGS; goto failed; -- 1.7.1 From jsimmons at infradead.org Sat May 7 01:30:24 2016 From: jsimmons at infradead.org (James Simmons) Date: Fri, 6 May 2016 21:30:24 -0400 Subject: [lustre-devel] [PATCH 03/10] staging: lustre: lnet: add per NI configuration structures In-Reply-To: <1462584631-5388-1-git-send-email-jsimmons@infradead.org> References: <1462584631-5388-1-git-send-email-jsimmons@infradead.org> Message-ID: <1462584631-5388-4-git-send-email-jsimmons@infradead.org> From: Amir Shehata Currently for LNet all hardware configuration is done with the modprobe configuration file. These settings are applied to all hardware instances for a node. That makes it impossible to configure two different pieces of infiniband hardware that require very different settings to function. Here we introduce the infrastructre to allow applying per interface settings. Signed-off-by: Amir Shehata Signed-off-by: James Simmons Intel-bug-id: https://jira.hpdd.intel.com/browse/LU-7101 Reviewed-on: http://review.whamcloud.com/16367 Reviewed-by: Doug Oucharek Reviewed-by: Olaf Weber Reviewed-by: Oleg Drokin Signed-off-by: James Simmons --- .../staging/lustre/include/linux/lnet/lib-dlc.h | 29 +++++++++++++++++++- .../staging/lustre/include/linux/lnet/lib-types.h | 2 + drivers/staging/lustre/lnet/lnet/config.c | 3 ++ 3 files changed, 33 insertions(+), 1 deletions(-) diff --git a/drivers/staging/lustre/include/linux/lnet/lib-dlc.h b/drivers/staging/lustre/include/linux/lnet/lib-dlc.h index 84a19e9..6ce9acc 100644 --- a/drivers/staging/lustre/include/linux/lnet/lib-dlc.h +++ b/drivers/staging/lustre/include/linux/lnet/lib-dlc.h @@ -37,10 +37,37 @@ #define LNET_MAX_SHOW_NUM_CPT 128 #define LNET_UNDEFINED_HOPS ((__u32) -1) +struct lnet_ioctl_config_lnd_cmn_tunables { + __u32 lct_version; + __u32 lct_peer_timeout; + __u32 lct_peer_tx_credits; + __u32 lct_peer_rtr_credits; + __u32 lct_max_tx_credits; +}; + +struct lnet_ioctl_config_o2iblnd_tunables { + __u32 lnd_version; + __u32 lnd_peercredits_hiw; + __u32 lnd_map_on_demand; + __u32 lnd_concurrent_sends; + __u32 lnd_fmr_pool_size; + __u32 lnd_fmr_flush_trigger; + __u32 lnd_fmr_cache; + __u32 pad; +}; + +struct lnet_ioctl_config_lnd_tunables { + struct lnet_ioctl_config_lnd_cmn_tunables lt_cmn; + union { + struct lnet_ioctl_config_o2iblnd_tunables lt_o2ib; + } lt_tun_u; +}; + struct lnet_ioctl_net_config { char ni_interfaces[LNET_MAX_INTERFACES][LNET_MAX_STR_LEN]; __u32 ni_status; __u32 ni_cpts[LNET_MAX_SHOW_NUM_CPT]; + char cfg_bulk[0]; }; #define LNET_TINY_BUF_IDX 0 @@ -81,7 +108,7 @@ struct lnet_ioctl_config_data { __s32 net_peer_rtr_credits; __s32 net_max_tx_credits; __u32 net_cksum_algo; - __u32 net_pad; + __u32 net_interface_count; } cfg_net; struct { __u32 buf_enable; diff --git a/drivers/staging/lustre/include/linux/lnet/lib-types.h b/drivers/staging/lustre/include/linux/lnet/lib-types.h index 29c72f8..24c4a08 100644 --- a/drivers/staging/lustre/include/linux/lnet/lib-types.h +++ b/drivers/staging/lustre/include/linux/lnet/lib-types.h @@ -273,6 +273,8 @@ typedef struct lnet_ni { int **ni_refs; /* percpt reference count */ time64_t ni_last_alive;/* when I was last alive */ lnet_ni_status_t *ni_status; /* my health status */ + /* per NI LND tunables */ + struct lnet_ioctl_config_lnd_tunables *ni_lnd_tunables; /* equivalent interfaces to use */ char *ni_interfaces[LNET_MAX_INTERFACES]; } lnet_ni_t; diff --git a/drivers/staging/lustre/lnet/lnet/config.c b/drivers/staging/lustre/lnet/lnet/config.c index 449069c..480cc9c 100644 --- a/drivers/staging/lustre/lnet/lnet/config.c +++ b/drivers/staging/lustre/lnet/lnet/config.c @@ -107,6 +107,9 @@ lnet_ni_free(struct lnet_ni *ni) if (ni->ni_cpts) cfs_expr_list_values_free(ni->ni_cpts, ni->ni_ncpts); + if (ni->ni_lnd_tunables) + LIBCFS_FREE(ni->ni_lnd_tunables, sizeof(*ni->ni_lnd_tunables)); + for (i = 0; i < LNET_MAX_INTERFACES && ni->ni_interfaces[i]; i++) { LIBCFS_FREE(ni->ni_interfaces[i], strlen(ni->ni_interfaces[i]) + 1); -- 1.7.1 From jsimmons at infradead.org Sat May 7 01:30:25 2016 From: jsimmons at infradead.org (James Simmons) Date: Fri, 6 May 2016 21:30:25 -0400 Subject: [lustre-devel] [PATCH 04/10] staging: lustre: o2iblnd: convert macros to inline functions In-Reply-To: <1462584631-5388-1-git-send-email-jsimmons@infradead.org> References: <1462584631-5388-1-git-send-email-jsimmons@infradead.org> Message-ID: <1462584631-5388-5-git-send-email-jsimmons@infradead.org> From: Amir Shehata Convert a few macros in o2iblnd.h to inline functions. Signed-off-by: Amir Shehata Signed-off-by: James Simmons Intel-bug-id: https://jira.hpdd.intel.com/browse/LU-7101 Reviewed-on: http://review.whamcloud.com/16367 Reviewed-by: Doug Oucharek Reviewed-by: Olaf Weber Reviewed-by: Oleg Drokin Signed-off-by: James Simmons --- .../staging/lustre/lnet/klnds/o2iblnd/o2iblnd.c | 2 +- .../staging/lustre/lnet/klnds/o2iblnd/o2iblnd.h | 62 ++++++++++++-------- .../staging/lustre/lnet/klnds/o2iblnd/o2iblnd_cb.c | 21 ++++--- .../lustre/lnet/klnds/o2iblnd/o2iblnd_modparams.c | 11 ++++ 4 files changed, 61 insertions(+), 35 deletions(-) diff --git a/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.c b/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.c index ddc3313..a2e188e 100644 --- a/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.c +++ b/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.c @@ -335,7 +335,7 @@ int kiblnd_create_peer(lnet_ni_t *ni, kib_peer_t **peerp, lnet_nid_t nid) peer->ibp_nid = nid; peer->ibp_error = 0; peer->ibp_last_alive = 0; - peer->ibp_max_frags = IBLND_CFG_RDMA_FRAGS; + peer->ibp_max_frags = kiblnd_cfg_rdma_frags(peer->ibp_ni); peer->ibp_queue_depth = *kiblnd_tunables.kib_peertxcredits; atomic_set(&peer->ibp_refcount, 1); /* 1 ref for caller */ diff --git a/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.h b/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.h index 52245e0..fab0ef9 100644 --- a/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.h +++ b/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.h @@ -116,9 +116,6 @@ extern kib_tunables_t kiblnd_tunables; #define IBLND_CREDITS_DEFAULT 8 /* default # of peer credits */ #define IBLND_CREDITS_MAX ((typeof(((kib_msg_t *) 0)->ibm_credits)) - 1) /* Max # of peer credits */ -#define IBLND_MSG_QUEUE_SIZE(v) ((v) == IBLND_MSG_VERSION_1 ? \ - IBLND_MSG_QUEUE_SIZE_V1 : \ - *kiblnd_tunables.kib_peertxcredits) /* # messages/RDMAs in-flight */ #define IBLND_CREDITS_HIGHWATER(v) ((v) == IBLND_MSG_VERSION_1 ? \ IBLND_CREDIT_HIGHWATER_V1 : \ *kiblnd_tunables.kib_peercredits_hiw) /* when eagerly to return credits */ @@ -127,32 +124,12 @@ extern kib_tunables_t kiblnd_tunables; cb, dev, \ ps, qpt) -static inline int -kiblnd_concurrent_sends_v1(void) -{ - if (*kiblnd_tunables.kib_concurrent_sends > IBLND_MSG_QUEUE_SIZE_V1 * 2) - return IBLND_MSG_QUEUE_SIZE_V1 * 2; - - if (*kiblnd_tunables.kib_concurrent_sends < IBLND_MSG_QUEUE_SIZE_V1 / 2) - return IBLND_MSG_QUEUE_SIZE_V1 / 2; - - return *kiblnd_tunables.kib_concurrent_sends; -} - -#define IBLND_CONCURRENT_SENDS(v) ((v) == IBLND_MSG_VERSION_1 ? \ - kiblnd_concurrent_sends_v1() : \ - *kiblnd_tunables.kib_concurrent_sends) /* 2 OOB shall suffice for 1 keepalive and 1 returning credits */ #define IBLND_OOB_CAPABLE(v) ((v) != IBLND_MSG_VERSION_1) #define IBLND_OOB_MSGS(v) (IBLND_OOB_CAPABLE(v) ? 2 : 0) #define IBLND_MSG_SIZE (4 << 10) /* max size of queued messages (inc hdr) */ #define IBLND_MAX_RDMA_FRAGS LNET_MAX_IOV /* max # of fragments supported */ -#define IBLND_CFG_RDMA_FRAGS (*kiblnd_tunables.kib_map_on_demand ? \ - *kiblnd_tunables.kib_map_on_demand : \ - IBLND_MAX_RDMA_FRAGS) /* max # of fragments configured by user */ -#define IBLND_RDMA_FRAGS(v) ((v) == IBLND_MSG_VERSION_1 ? \ - IBLND_MAX_RDMA_FRAGS : IBLND_CFG_RDMA_FRAGS) /************************/ /* derived constants... */ @@ -171,7 +148,8 @@ kiblnd_concurrent_sends_v1(void) /* WRs and CQEs (per connection) */ #define IBLND_RECV_WRS(c) IBLND_RX_MSGS(c) #define IBLND_SEND_WRS(c) \ - ((c->ibc_max_frags + 1) * IBLND_CONCURRENT_SENDS(c->ibc_version)) + ((c->ibc_max_frags + 1) * kiblnd_concurrent_sends(c->ibc_version, \ + c->ibc_peer->ibp_ni)) #define IBLND_CQ_ENTRIES(c) (IBLND_RECV_WRS(c) + IBLND_SEND_WRS(c)) struct kib_hca_dev; @@ -634,6 +612,42 @@ extern kib_data_t kiblnd_data; void kiblnd_hdev_destroy(kib_hca_dev_t *hdev); +int kiblnd_msg_queue_size(int version, struct lnet_ni *ni); + +/* max # of fragments configured by user */ +static inline int +kiblnd_cfg_rdma_frags(struct lnet_ni *ni) +{ + int mod = *kiblnd_tunables.kib_map_on_demand; + return mod ? mod : IBLND_MAX_RDMA_FRAGS; +} + +static inline int +kiblnd_rdma_frags(int version, struct lnet_ni *ni) +{ + return version == IBLND_MSG_VERSION_1 ? + IBLND_MAX_RDMA_FRAGS : + kiblnd_cfg_rdma_frags(ni); +} + +static inline int +kiblnd_concurrent_sends(int version, struct lnet_ni *ni) +{ + int concurrent_sends; + + concurrent_sends = *kiblnd_tunables.kib_concurrent_sends; + + if (version == IBLND_MSG_VERSION_1) { + if (concurrent_sends > IBLND_MSG_QUEUE_SIZE_V1 * 2) + return IBLND_MSG_QUEUE_SIZE_V1 * 2; + + if (concurrent_sends < IBLND_MSG_QUEUE_SIZE_V1 / 2) + return IBLND_MSG_QUEUE_SIZE_V1 / 2; + } + + return concurrent_sends; +} + static inline void kiblnd_hdev_addref_locked(kib_hca_dev_t *hdev) { diff --git a/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd_cb.c b/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd_cb.c index 8bbd116..c3bd7fa 100644 --- a/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd_cb.c +++ b/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd_cb.c @@ -757,6 +757,7 @@ kiblnd_post_tx_locked(kib_conn_t *conn, kib_tx_t *tx, int credit) { kib_msg_t *msg = tx->tx_msg; kib_peer_t *peer = conn->ibc_peer; + struct lnet_ni *ni = peer->ibp_ni; int ver = conn->ibc_version; int rc; int done; @@ -772,7 +773,7 @@ kiblnd_post_tx_locked(kib_conn_t *conn, kib_tx_t *tx, int credit) LASSERT(conn->ibc_credits >= 0); LASSERT(conn->ibc_credits <= conn->ibc_queue_depth); - if (conn->ibc_nsends_posted == IBLND_CONCURRENT_SENDS(ver)) { + if (conn->ibc_nsends_posted == kiblnd_concurrent_sends(ver, ni)) { /* tx completions outstanding... */ CDEBUG(D_NET, "%s: posted enough\n", libcfs_nid2str(peer->ibp_nid)); @@ -923,7 +924,7 @@ kiblnd_check_sends(kib_conn_t *conn) spin_lock(&conn->ibc_lock); - LASSERT(conn->ibc_nsends_posted <= IBLND_CONCURRENT_SENDS(ver)); + LASSERT(conn->ibc_nsends_posted <= kiblnd_concurrent_sends(ver, ni)); LASSERT(!IBLND_OOB_CAPABLE(ver) || conn->ibc_noops_posted <= IBLND_OOB_MSGS(ver)); LASSERT(conn->ibc_reserved_credits >= 0); @@ -2337,11 +2338,11 @@ kiblnd_passive_connect(struct rdma_cm_id *cmid, void *priv, int priv_nob) } if (reqmsg->ibm_u.connparams.ibcp_queue_depth > - IBLND_MSG_QUEUE_SIZE(version)) { + kiblnd_msg_queue_size(version, ni)) { CERROR("Can't accept conn from %s, queue depth too large: %d (<=%d wanted)\n", libcfs_nid2str(nid), reqmsg->ibm_u.connparams.ibcp_queue_depth, - IBLND_MSG_QUEUE_SIZE(version)); + kiblnd_msg_queue_size(version, ni)); if (version == IBLND_MSG_VERSION) rej.ibr_why = IBLND_REJECT_MSG_QUEUE_SIZE; @@ -2350,22 +2351,22 @@ kiblnd_passive_connect(struct rdma_cm_id *cmid, void *priv, int priv_nob) } if (reqmsg->ibm_u.connparams.ibcp_max_frags > - IBLND_RDMA_FRAGS(version)) { + kiblnd_rdma_frags(version, ni)) { CWARN("Can't accept conn from %s (version %x): max_frags %d too large (%d wanted)\n", libcfs_nid2str(nid), version, reqmsg->ibm_u.connparams.ibcp_max_frags, - IBLND_RDMA_FRAGS(version)); + kiblnd_rdma_frags(version, ni)); if (version >= IBLND_MSG_VERSION) rej.ibr_why = IBLND_REJECT_RDMA_FRAGS; goto failed; } else if (reqmsg->ibm_u.connparams.ibcp_max_frags < - IBLND_RDMA_FRAGS(version) && !net->ibn_fmr_ps) { + kiblnd_rdma_frags(version, ni) && !net->ibn_fmr_ps) { CWARN("Can't accept conn from %s (version %x): max_frags %d incompatible without FMR pool (%d wanted)\n", libcfs_nid2str(nid), version, reqmsg->ibm_u.connparams.ibcp_max_frags, - IBLND_RDMA_FRAGS(version)); + kiblnd_rdma_frags(version, ni)); if (version >= IBLND_MSG_VERSION) rej.ibr_why = IBLND_REJECT_RDMA_FRAGS; @@ -2532,8 +2533,8 @@ kiblnd_passive_connect(struct rdma_cm_id *cmid, void *priv, int priv_nob) lnet_ni_decref(ni); rej.ibr_version = version; - rej.ibr_cp.ibcp_queue_depth = IBLND_MSG_QUEUE_SIZE(version); - rej.ibr_cp.ibcp_max_frags = IBLND_RDMA_FRAGS(version); + rej.ibr_cp.ibcp_queue_depth = kiblnd_msg_queue_size(version, ni); + rej.ibr_cp.ibcp_max_frags = kiblnd_rdma_frags(version, ni); kiblnd_reject(cmid, &rej); return -ECONNREFUSED; diff --git a/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd_modparams.c b/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd_modparams.c index b4607da..a6415fc 100644 --- a/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd_modparams.c +++ b/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd_modparams.c @@ -171,6 +171,17 @@ kib_tunables_t kiblnd_tunables = { .kib_nscheds = &nscheds }; +/* # messages/RDMAs in-flight */ +int kiblnd_msg_queue_size(int version, lnet_ni_t *ni) +{ + if (version == IBLND_MSG_VERSION_1) + return IBLND_MSG_QUEUE_SIZE_V1; + else if (ni) + return ni->ni_peertxcredits; + else + return peer_credits; +} + int kiblnd_tunables_init(void) { -- 1.7.1 From jsimmons at infradead.org Sat May 7 01:30:27 2016 From: jsimmons at infradead.org (James Simmons) Date: Fri, 6 May 2016 21:30:27 -0400 Subject: [lustre-devel] [PATCH 06/10] staging: lustre: o2iblnd: use data in lnet_ni_t instead of kiblnd_tunables In-Reply-To: <1462584631-5388-1-git-send-email-jsimmons@infradead.org> References: <1462584631-5388-1-git-send-email-jsimmons@infradead.org> Message-ID: <1462584631-5388-7-git-send-email-jsimmons@infradead.org> From: Amir Shehata Currently the ko2iblnd creates a kib_tunable_t stucture to allow the ko2iblnd driver to access the module parameters throught the code. Some of those data fields also exist in lnet_ni_t. Migrate to using the lnet_ni_t data fields instead of kib_tunable_t. Signed-off-by: Amir Shehata Signed-off-by: James Simmons Intel-bug-id: https://jira.hpdd.intel.com/browse/LU-7101 Reviewed-on: http://review.whamcloud.com/16367 Reviewed-by: Doug Oucharek Reviewed-by: Olaf Weber Reviewed-by: Oleg Drokin Signed-off-by: James Simmons --- .../staging/lustre/lnet/klnds/o2iblnd/o2iblnd.c | 8 +-- .../staging/lustre/lnet/klnds/o2iblnd/o2iblnd.h | 6 +-- .../lustre/lnet/klnds/o2iblnd/o2iblnd_modparams.c | 54 +++++++++++-------- 3 files changed, 34 insertions(+), 34 deletions(-) diff --git a/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.c b/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.c index a06689b..84a15d4 100644 --- a/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.c +++ b/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.c @@ -336,7 +336,7 @@ int kiblnd_create_peer(lnet_ni_t *ni, kib_peer_t **peerp, lnet_nid_t nid) peer->ibp_error = 0; peer->ibp_last_alive = 0; peer->ibp_max_frags = kiblnd_cfg_rdma_frags(peer->ibp_ni); - peer->ibp_queue_depth = *kiblnd_tunables.kib_peertxcredits; + peer->ibp_queue_depth = ni->ni_peertxcredits; atomic_set(&peer->ibp_refcount, 1); /* 1 ref for caller */ INIT_LIST_HEAD(&peer->ibp_list); /* not in the peer table yet */ @@ -2921,13 +2921,9 @@ static int kiblnd_startup(lnet_ni_t *ni) net->ibn_incarnation = tv.tv_sec * USEC_PER_SEC + tv.tv_nsec / NSEC_PER_USEC; - rc = kiblnd_tunables_setup(); + rc = kiblnd_tunables_setup(ni); if (rc) goto net_failed; - ni->ni_peertimeout = *kiblnd_tunables.kib_peertimeout; - ni->ni_maxtxcredits = *kiblnd_tunables.kib_credits; - ni->ni_peertxcredits = *kiblnd_tunables.kib_peertxcredits; - ni->ni_peerrtrcredits = *kiblnd_tunables.kib_peerrtrcredits; if (ni->ni_interfaces[0]) { /* Use the IPoIB interface specified in 'networks=' */ diff --git a/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.h b/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.h index 80e11bc..fffae0c 100644 --- a/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.h +++ b/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.h @@ -87,11 +87,7 @@ typedef struct { int *kib_timeout; /* comms timeout (seconds) */ int *kib_keepalive; /* keepalive timeout (seconds) */ int *kib_ntx; /* # tx descs */ - int *kib_credits; /* # concurrent sends */ - int *kib_peertxcredits; /* # concurrent sends to 1 peer */ - int *kib_peerrtrcredits; /* # per-peer router buffer credits */ int *kib_peercredits_hiw; /* # when eagerly to return credits */ - int *kib_peertimeout; /* seconds to consider peer dead */ char **kib_default_ipif; /* default IPoIB interface */ int *kib_retry_count; int *kib_rnr_retry_count; @@ -994,7 +990,7 @@ int kiblnd_fmr_pool_map(kib_fmr_poolset_t *fps, kib_tx_t *tx, kib_fmr_t *fmr); void kiblnd_fmr_pool_unmap(kib_fmr_t *fmr, int status); -int kiblnd_tunables_setup(void); +int kiblnd_tunables_setup(struct lnet_ni *ni); void kiblnd_tunables_init(void); void kiblnd_tunables_fini(void); diff --git a/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd_modparams.c b/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd_modparams.c index cc1469d..e50a9cf 100644 --- a/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd_modparams.c +++ b/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd_modparams.c @@ -152,11 +152,7 @@ kib_tunables_t kiblnd_tunables = { .kib_timeout = &timeout, .kib_keepalive = &keepalive, .kib_ntx = &ntx, - .kib_credits = &credits, - .kib_peertxcredits = &peer_credits, .kib_peercredits_hiw = &peer_credits_hiw, - .kib_peerrtrcredits = &peer_buffer_credits, - .kib_peertimeout = &peer_timeout, .kib_default_ipif = &ipif_name, .kib_retry_count = &retry_count, .kib_rnr_retry_count = &rnr_retry_count, @@ -184,7 +180,7 @@ int kiblnd_msg_queue_size(int version, lnet_ni_t *ni) return peer_credits; } -int kiblnd_tunables_setup(void) +int kiblnd_tunables_setup(struct lnet_ni *ni) { if (kiblnd_translate_mtu(*kiblnd_tunables.kib_ib_mtu) < 0) { CERROR("Invalid ib_mtu %d, expected 256/512/1024/2048/4096\n", @@ -192,20 +188,32 @@ int kiblnd_tunables_setup(void) return -EINVAL; } - if (*kiblnd_tunables.kib_peertxcredits < IBLND_CREDITS_DEFAULT) - *kiblnd_tunables.kib_peertxcredits = IBLND_CREDITS_DEFAULT; + if (!ni->ni_peertimeout) + ni->ni_peertimeout = peer_timeout; - if (*kiblnd_tunables.kib_peertxcredits > IBLND_CREDITS_MAX) - *kiblnd_tunables.kib_peertxcredits = IBLND_CREDITS_MAX; + if (!ni->ni_maxtxcredits) + ni->ni_maxtxcredits = credits; - if (*kiblnd_tunables.kib_peertxcredits > *kiblnd_tunables.kib_credits) - *kiblnd_tunables.kib_peertxcredits = *kiblnd_tunables.kib_credits; + if (!ni->ni_peertxcredits) + ni->ni_peertxcredits = peer_credits; - if (*kiblnd_tunables.kib_peercredits_hiw < *kiblnd_tunables.kib_peertxcredits / 2) - *kiblnd_tunables.kib_peercredits_hiw = *kiblnd_tunables.kib_peertxcredits / 2; + if (!ni->ni_peerrtrcredits) + ni->ni_peerrtrcredits = peer_buffer_credits; - if (*kiblnd_tunables.kib_peercredits_hiw >= *kiblnd_tunables.kib_peertxcredits) - *kiblnd_tunables.kib_peercredits_hiw = *kiblnd_tunables.kib_peertxcredits - 1; + if (ni->ni_peertxcredits < IBLND_CREDITS_DEFAULT) + ni->ni_peertxcredits = IBLND_CREDITS_DEFAULT; + + if (ni->ni_peertxcredits > IBLND_CREDITS_MAX) + ni->ni_peertxcredits = IBLND_CREDITS_MAX; + + if (ni->ni_peertxcredits > credits) + ni->ni_peertxcredits = credits; + + if (*kiblnd_tunables.kib_peercredits_hiw < ni->ni_peertxcredits / 2) + *kiblnd_tunables.kib_peercredits_hiw = ni->ni_peertxcredits / 2; + + if (*kiblnd_tunables.kib_peercredits_hiw >= ni->ni_peertxcredits) + *kiblnd_tunables.kib_peercredits_hiw = ni->ni_peertxcredits - 1; if (*kiblnd_tunables.kib_map_on_demand < 0 || *kiblnd_tunables.kib_map_on_demand > IBLND_MAX_RDMA_FRAGS) @@ -217,20 +225,20 @@ int kiblnd_tunables_setup(void) if (!*kiblnd_tunables.kib_concurrent_sends) { if (*kiblnd_tunables.kib_map_on_demand > 0 && *kiblnd_tunables.kib_map_on_demand <= IBLND_MAX_RDMA_FRAGS / 8) - *kiblnd_tunables.kib_concurrent_sends = (*kiblnd_tunables.kib_peertxcredits) * 2; + *kiblnd_tunables.kib_concurrent_sends = ni->ni_peertxcredits * 2; else - *kiblnd_tunables.kib_concurrent_sends = (*kiblnd_tunables.kib_peertxcredits); + *kiblnd_tunables.kib_concurrent_sends = ni->ni_peertxcredits; } - if (*kiblnd_tunables.kib_concurrent_sends > *kiblnd_tunables.kib_peertxcredits * 2) - *kiblnd_tunables.kib_concurrent_sends = *kiblnd_tunables.kib_peertxcredits * 2; + if (*kiblnd_tunables.kib_concurrent_sends > ni->ni_peertxcredits * 2) + *kiblnd_tunables.kib_concurrent_sends = ni->ni_peertxcredits * 2; - if (*kiblnd_tunables.kib_concurrent_sends < *kiblnd_tunables.kib_peertxcredits / 2) - *kiblnd_tunables.kib_concurrent_sends = *kiblnd_tunables.kib_peertxcredits / 2; + if (*kiblnd_tunables.kib_concurrent_sends < ni->ni_peertxcredits / 2) + *kiblnd_tunables.kib_concurrent_sends = ni->ni_peertxcredits / 2; - if (*kiblnd_tunables.kib_concurrent_sends < *kiblnd_tunables.kib_peertxcredits) { + if (*kiblnd_tunables.kib_concurrent_sends < ni->ni_peertxcredits) { CWARN("Concurrent sends %d is lower than message queue size: %d, performance may drop slightly.\n", - *kiblnd_tunables.kib_concurrent_sends, *kiblnd_tunables.kib_peertxcredits); + *kiblnd_tunables.kib_concurrent_sends, ni->ni_peertxcredits); } return 0; -- 1.7.1 From jsimmons at infradead.org Sat May 7 01:30:21 2016 From: jsimmons at infradead.org (James Simmons) Date: Fri, 6 May 2016 21:30:21 -0400 Subject: [lustre-devel] [PATCH 00/10] staging: lustre: lnet: per NI map-on-demand value Message-ID: <1462584631-5388-1-git-send-email-jsimmons@infradead.org> Enables support of different map-on-demand values per NI. This is required to support OPA coexistence with MLX5 cards. MLX5 does not support FMR, which is enabled via map-on-demand. However OPA's performance is greatly enahanced when FMR is enabled. In order to enable coexistence of both of these two types of cards we need to be able to set different map-on-demand values for both NIs. This patch also lays the ground work for other per NI tunables to be added in future patches. Amir Shehata (10): staging: lustre: lnet: use struct lnet_ioctl_config_data for NI bring up staging: lustre: lnet: use struct lnet_ioctl_config_data for NI query staging: lustre: lnet: add per NI configuration structures staging: lustre: o2iblnd: convert macros to inline functions staging: lustre: o2iblnd: create default o2iblnd tunable settings staging: lustre: o2iblnd: use data in lnet_ni_t instead of kiblnd_tunables staging: lustre: o2iblnd: per NI map-on-demand value staging: lustre: o2iblnd: remove last of kiblnd_tunables_fini staging: lustre: o2iblnd: properly set ibr_why staging: lustre: lnet: enable configuration per NI interface .../staging/lustre/include/linux/lnet/lib-dlc.h | 29 ++++- .../staging/lustre/include/linux/lnet/lib-lnet.h | 9 +- .../staging/lustre/include/linux/lnet/lib-types.h | 2 + .../staging/lustre/lnet/klnds/o2iblnd/o2iblnd.c | 83 +++++++----- .../staging/lustre/lnet/klnds/o2iblnd/o2iblnd.h | 101 ++++++++------ .../staging/lustre/lnet/klnds/o2iblnd/o2iblnd_cb.c | 36 +++--- .../lustre/lnet/klnds/o2iblnd/o2iblnd_modparams.c | 139 ++++++++++++++------ drivers/staging/lustre/lnet/lnet/api-ni.c | 139 ++++++++++++-------- drivers/staging/lustre/lnet/lnet/config.c | 3 + drivers/staging/lustre/lnet/lnet/module.c | 7 +- 10 files changed, 341 insertions(+), 207 deletions(-) From jsimmons at infradead.org Sat May 7 01:30:22 2016 From: jsimmons at infradead.org (James Simmons) Date: Fri, 6 May 2016 21:30:22 -0400 Subject: [lustre-devel] [PATCH 01/10] staging: lustre: lnet: use struct lnet_ioctl_config_data for NI bring up In-Reply-To: <1462584631-5388-1-git-send-email-jsimmons@infradead.org> References: <1462584631-5388-1-git-send-email-jsimmons@infradead.org> Message-ID: <1462584631-5388-2-git-send-email-jsimmons@infradead.org> From: Amir Shehata The LNet layer passes around the individual fields of struct lnet_ioctl_config_data for the case of NI bring up. To simplify the code lets just pass struct lnet_ioctl_config directly. Signed-off-by: Amir Shehata Signed-off-by: James Simmons Intel-bug-id: https://jira.hpdd.intel.com/browse/LU-7101 Reviewed-on: http://review.whamcloud.com/16367 Reviewed-by: Doug Oucharek Reviewed-by: Olaf Weber Reviewed-by: Oleg Drokin Signed-off-by: James Simmons --- .../staging/lustre/include/linux/lnet/lib-lnet.h | 5 +-- drivers/staging/lustre/lnet/lnet/api-ni.c | 37 +++++++++++-------- drivers/staging/lustre/lnet/lnet/module.c | 7 +--- 3 files changed, 24 insertions(+), 25 deletions(-) diff --git a/drivers/staging/lustre/include/linux/lnet/lib-lnet.h b/drivers/staging/lustre/include/linux/lnet/lib-lnet.h index dfc0208..1865c85 100644 --- a/drivers/staging/lustre/include/linux/lnet/lib-lnet.h +++ b/drivers/staging/lustre/include/linux/lnet/lib-lnet.h @@ -478,9 +478,8 @@ int lnet_rtrpools_enable(void); void lnet_rtrpools_disable(void); void lnet_rtrpools_free(int keep_pools); lnet_remotenet_t *lnet_find_net_locked(__u32 net); -int lnet_dyn_add_ni(lnet_pid_t requested_pid, char *nets, - __s32 peer_timeout, __s32 peer_cr, __s32 peer_buf_cr, - __s32 credits); +int lnet_dyn_add_ni(lnet_pid_t requested_pid, + struct lnet_ioctl_config_data *conf); int lnet_dyn_del_ni(__u32 net); int lnet_clear_lazy_portal(struct lnet_ni *ni, int portal, char *reason); diff --git a/drivers/staging/lustre/lnet/lnet/api-ni.c b/drivers/staging/lustre/lnet/lnet/api-ni.c index f825784..6340ad1 100644 --- a/drivers/staging/lustre/lnet/lnet/api-ni.c +++ b/drivers/staging/lustre/lnet/lnet/api-ni.c @@ -1215,8 +1215,7 @@ lnet_shutdown_lndni(struct lnet_ni *ni) } static int -lnet_startup_lndni(struct lnet_ni *ni, __s32 peer_timeout, - __s32 peer_cr, __s32 peer_buf_cr, __s32 credits) +lnet_startup_lndni(struct lnet_ni *ni, struct lnet_ioctl_config_data *conf) { int rc = -EINVAL; int lnd_type; @@ -1292,20 +1291,28 @@ lnet_startup_lndni(struct lnet_ni *ni, __s32 peer_timeout, * If given some LND tunable parameters, parse those now to * override the values in the NI structure. */ - if (peer_buf_cr >= 0) - ni->ni_peerrtrcredits = peer_buf_cr; - if (peer_timeout >= 0) - ni->ni_peertimeout = peer_timeout; + if (conf && conf->cfg_config_u.cfg_net.net_peer_rtr_credits >= 0) { + ni->ni_peerrtrcredits = + conf->cfg_config_u.cfg_net.net_peer_rtr_credits; + } + if (conf && conf->cfg_config_u.cfg_net.net_peer_timeout >= 0) { + ni->ni_peertimeout = + conf->cfg_config_u.cfg_net.net_peer_timeout; + } /* * TODO * Note: For now, don't allow the user to change * peertxcredits as this number is used in the * IB LND to control queue depth. - * if (peer_cr != -1) - * ni->ni_peertxcredits = peer_cr; + * + * if (conf && conf->cfg_config_u.cfg_net.net_peer_tx_credits != -1) + * ni->ni_peertxcredits = + * conf->cfg_config_u.cfg_net.net_peer_tx_credits; */ - if (credits >= 0) - ni->ni_maxtxcredits = credits; + if (conf && conf->cfg_config_u.cfg_net.net_max_tx_credits >= 0) { + ni->ni_maxtxcredits = + conf->cfg_config_u.cfg_net.net_max_tx_credits; + } LASSERT(ni->ni_peertimeout <= 0 || lnd->lnd_query); @@ -1367,7 +1374,7 @@ lnet_startup_lndnis(struct list_head *nilist) while (!list_empty(nilist)) { ni = list_entry(nilist->next, lnet_ni_t, ni_list); list_del(&ni->ni_list); - rc = lnet_startup_lndni(ni, -1, -1, -1, -1); + rc = lnet_startup_lndni(ni, NULL); if (rc < 0) goto failed; @@ -1725,10 +1732,9 @@ lnet_get_net_config(int idx, __u32 *cpt_count, __u64 *nid, int *peer_timeout, } int -lnet_dyn_add_ni(lnet_pid_t requested_pid, char *nets, - __s32 peer_timeout, __s32 peer_cr, __s32 peer_buf_cr, - __s32 credits) +lnet_dyn_add_ni(lnet_pid_t requested_pid, struct lnet_ioctl_config_data *conf) { + char *nets = conf->cfg_config_u.cfg_net.net_intf; lnet_ping_info_t *pinfo; lnet_handle_md_t md_handle; struct lnet_ni *ni; @@ -1773,8 +1779,7 @@ lnet_dyn_add_ni(lnet_pid_t requested_pid, char *nets, list_del_init(&ni->ni_list); - rc = lnet_startup_lndni(ni, peer_timeout, peer_cr, - peer_buf_cr, credits); + rc = lnet_startup_lndni(ni, conf); if (rc) goto failed1; diff --git a/drivers/staging/lustre/lnet/lnet/module.c b/drivers/staging/lustre/lnet/lnet/module.c index 93037c1..246b5c1 100644 --- a/drivers/staging/lustre/lnet/lnet/module.c +++ b/drivers/staging/lustre/lnet/lnet/module.c @@ -108,12 +108,7 @@ lnet_dyn_configure(struct libcfs_ioctl_hdr *hdr) rc = -EINVAL; goto out_unlock; } - rc = lnet_dyn_add_ni(LNET_PID_LUSTRE, - conf->cfg_config_u.cfg_net.net_intf, - conf->cfg_config_u.cfg_net.net_peer_timeout, - conf->cfg_config_u.cfg_net.net_peer_tx_credits, - conf->cfg_config_u.cfg_net.net_peer_rtr_credits, - conf->cfg_config_u.cfg_net.net_max_tx_credits); + rc = lnet_dyn_add_ni(LNET_PID_LUSTRE, conf); out_unlock: mutex_unlock(&lnet_config_mutex); -- 1.7.1 From jsimmons at infradead.org Sat May 7 01:30:26 2016 From: jsimmons at infradead.org (James Simmons) Date: Fri, 6 May 2016 21:30:26 -0400 Subject: [lustre-devel] [PATCH 05/10] staging: lustre: o2iblnd: create default o2iblnd tunable settings In-Reply-To: <1462584631-5388-1-git-send-email-jsimmons@infradead.org> References: <1462584631-5388-1-git-send-email-jsimmons@infradead.org> Message-ID: <1462584631-5388-6-git-send-email-jsimmons@infradead.org> From: Amir Shehata Create and set the default o2iblnd tunables when the ko2iblnd module loads. Move kiblnd_tunables_setup() to when the NI iterface is initialized. Signed-off-by: Amir Shehata Signed-off-by: James Simmons Intel-bug-id: https://jira.hpdd.intel.com/browse/LU-7101 Reviewed-on: http://review.whamcloud.com/16367 Reviewed-by: Doug Oucharek Reviewed-by: Olaf Weber Reviewed-by: Oleg Drokin Signed-off-by: James Simmons --- .../staging/lustre/lnet/klnds/o2iblnd/o2iblnd.c | 9 ++++----- .../staging/lustre/lnet/klnds/o2iblnd/o2iblnd.h | 3 ++- .../lustre/lnet/klnds/o2iblnd/o2iblnd_modparams.c | 16 ++++++++++++++-- 3 files changed, 20 insertions(+), 8 deletions(-) diff --git a/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.c b/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.c index a2e188e..a06689b 100644 --- a/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.c +++ b/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.c @@ -2921,6 +2921,9 @@ static int kiblnd_startup(lnet_ni_t *ni) net->ibn_incarnation = tv.tv_sec * USEC_PER_SEC + tv.tv_nsec / NSEC_PER_USEC; + rc = kiblnd_tunables_setup(); + if (rc) + goto net_failed; ni->ni_peertimeout = *kiblnd_tunables.kib_peertimeout; ni->ni_maxtxcredits = *kiblnd_tunables.kib_credits; ni->ni_peertxcredits = *kiblnd_tunables.kib_peertxcredits; @@ -3006,8 +3009,6 @@ static void __exit ko2iblnd_exit(void) static int __init ko2iblnd_init(void) { - int rc; - CLASSERT(sizeof(kib_msg_t) <= IBLND_MSG_SIZE); CLASSERT(offsetof(kib_msg_t, ibm_u.get.ibgm_rd.rd_frags[IBLND_MAX_RDMA_FRAGS]) @@ -3016,9 +3017,7 @@ static int __init ko2iblnd_init(void) ibm_u.putack.ibpam_rd.rd_frags[IBLND_MAX_RDMA_FRAGS]) <= IBLND_MSG_SIZE); - rc = kiblnd_tunables_init(); - if (rc) - return rc; + kiblnd_tunables_init(); lnet_register_lnd(&the_o2iblnd); diff --git a/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.h b/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.h index fab0ef9..80e11bc 100644 --- a/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.h +++ b/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.h @@ -994,7 +994,8 @@ int kiblnd_fmr_pool_map(kib_fmr_poolset_t *fps, kib_tx_t *tx, kib_fmr_t *fmr); void kiblnd_fmr_pool_unmap(kib_fmr_t *fmr, int status); -int kiblnd_tunables_init(void); +int kiblnd_tunables_setup(void); +void kiblnd_tunables_init(void); void kiblnd_tunables_fini(void); int kiblnd_connd(void *arg); diff --git a/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd_modparams.c b/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd_modparams.c index a6415fc..cc1469d 100644 --- a/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd_modparams.c +++ b/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd_modparams.c @@ -171,6 +171,8 @@ kib_tunables_t kiblnd_tunables = { .kib_nscheds = &nscheds }; +static struct lnet_ioctl_config_o2iblnd_tunables default_tunables; + /* # messages/RDMAs in-flight */ int kiblnd_msg_queue_size(int version, lnet_ni_t *ni) { @@ -182,8 +184,7 @@ int kiblnd_msg_queue_size(int version, lnet_ni_t *ni) return peer_credits; } -int -kiblnd_tunables_init(void) +int kiblnd_tunables_setup(void) { if (kiblnd_translate_mtu(*kiblnd_tunables.kib_ib_mtu) < 0) { CERROR("Invalid ib_mtu %d, expected 256/512/1024/2048/4096\n", @@ -234,3 +235,14 @@ kiblnd_tunables_init(void) return 0; } + +void kiblnd_tunables_init(void) +{ + default_tunables.lnd_version = 0; + default_tunables.lnd_peercredits_hiw = peer_credits_hiw, + default_tunables.lnd_map_on_demand = map_on_demand; + default_tunables.lnd_concurrent_sends = concurrent_sends; + default_tunables.lnd_fmr_pool_size = fmr_pool_size; + default_tunables.lnd_fmr_flush_trigger = fmr_flush_trigger; + default_tunables.lnd_fmr_cache = fmr_cache; +} -- 1.7.1 From jsimmons at infradead.org Sat May 7 01:30:31 2016 From: jsimmons at infradead.org (James Simmons) Date: Fri, 6 May 2016 21:30:31 -0400 Subject: [lustre-devel] [PATCH 10/10] staging: lustre: lnet: enable configuration per NI interface In-Reply-To: <1462584631-5388-1-git-send-email-jsimmons@infradead.org> References: <1462584631-5388-1-git-send-email-jsimmons@infradead.org> Message-ID: <1462584631-5388-11-git-send-email-jsimmons@infradead.org> From: Amir Shehata Add the ability to configure each NI interface at bring up. Also give the ability for user land utilities to query the configuration of each NI interface. Signed-off-by: Amir Shehata Signed-off-by: James Simmons Intel-bug-id: https://jira.hpdd.intel.com/browse/LU-7101 Reviewed-on: http://review.whamcloud.com/16367 Reviewed-by: Doug Oucharek Reviewed-by: Olaf Weber Reviewed-by: Oleg Drokin Signed-off-by: James Simmons --- drivers/staging/lustre/lnet/lnet/api-ni.c | 42 +++++++++++++++++++++++++++++ 1 files changed, 42 insertions(+), 0 deletions(-) diff --git a/drivers/staging/lustre/lnet/lnet/api-ni.c b/drivers/staging/lustre/lnet/lnet/api-ni.c index 01e1808..fe0dbe7 100644 --- a/drivers/staging/lustre/lnet/lnet/api-ni.c +++ b/drivers/staging/lustre/lnet/lnet/api-ni.c @@ -1217,6 +1217,7 @@ lnet_shutdown_lndni(struct lnet_ni *ni) static int lnet_startup_lndni(struct lnet_ni *ni, struct lnet_ioctl_config_data *conf) { + struct lnet_ioctl_config_lnd_tunables *lnd_tunables = NULL; int rc = -EINVAL; int lnd_type; lnd_t *lnd; @@ -1274,6 +1275,21 @@ lnet_startup_lndni(struct lnet_ni *ni, struct lnet_ioctl_config_data *conf) ni->ni_lnd = lnd; + if (conf && conf->cfg_hdr.ioc_len > sizeof(*conf)) + lnd_tunables = (struct lnet_ioctl_config_lnd_tunables *)conf->cfg_bulk; + + if (lnd_tunables) { + LIBCFS_ALLOC(ni->ni_lnd_tunables, + sizeof(*ni->ni_lnd_tunables)); + if (!ni->ni_lnd_tunables) { + mutex_unlock(&the_lnet.ln_lnd_mutex); + rc = -ENOMEM; + goto failed0; + } + memcpy(ni->ni_lnd_tunables, lnd_tunables, + sizeof(*ni->ni_lnd_tunables)); + } + rc = lnd->lnd_startup(ni); mutex_unlock(&the_lnet.ln_lnd_mutex); @@ -1653,7 +1669,9 @@ EXPORT_SYMBOL(LNetNIFini); static void lnet_fill_ni_info(struct lnet_ni *ni, struct lnet_ioctl_config_data *config) { + struct lnet_ioctl_config_lnd_tunables *lnd_cfg = NULL; struct lnet_ioctl_net_config *net_config; + size_t min_size, tunable_size = 0; int i; if (!ni || !config) @@ -1691,6 +1709,30 @@ lnet_fill_ni_info(struct lnet_ni *ni, struct lnet_ioctl_config_data *config) config->cfg_ncpts = num_cpts; } + + /* + * See if user land tools sent in a newer and larger version + * of struct lnet_tunables than what the kernel uses. + */ + min_size = sizeof(*config) + sizeof(*net_config); + + if (config->cfg_hdr.ioc_len > min_size) + tunable_size = config->cfg_hdr.ioc_len - min_size; + + /* Don't copy to much data to user space */ + min_size = min(tunable_size, sizeof(*ni->ni_lnd_tunables)); + lnd_cfg = (struct lnet_ioctl_config_lnd_tunables *)net_config->cfg_bulk; + + if (ni->ni_lnd_tunables && lnd_cfg && min_size) { + memcpy(lnd_cfg, ni->ni_lnd_tunables, min_size); + config->cfg_config_u.cfg_net.net_interface_count = 1; + + /* Tell user land that kernel side has less data */ + if (tunable_size > sizeof(*ni->ni_lnd_tunables)) { + min_size = tunable_size - sizeof(ni->ni_lnd_tunables); + config->cfg_hdr.ioc_len -= min_size; + } + } } static int -- 1.7.1 From jsimmons at infradead.org Mon May 9 14:53:45 2016 From: jsimmons at infradead.org (James Simmons) Date: Mon, 9 May 2016 10:53:45 -0400 Subject: [lustre-devel] [PATCH 1/4] staging: lustre: o2iblnd: limit cap.max_send_wr for MLX5 In-Reply-To: <1462805628-23403-1-git-send-email-jsimmons@infradead.org> References: <1462805628-23403-1-git-send-email-jsimmons@infradead.org> Message-ID: <1462805628-23403-2-git-send-email-jsimmons@infradead.org> From: Dmitry Eremin Decrease cap.max_send_wr until it is accepted by rdma_create_qp() Signed-off-by: Dmitry Eremin Intel-bug-id: https://jira.hpdd.intel.com/browse/LU-7124 Reviewed-on: http://review.whamcloud.com/18347 Reviewed-by: Olaf Weber Reviewed-by: Doug Oucharek Reviewed-by: Oleg Drokin Signed-off-by: James Simmons --- .../staging/lustre/lnet/klnds/o2iblnd/o2iblnd.c | 11 ++++++++++- 1 files changed, 10 insertions(+), 1 deletions(-) diff --git a/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.c b/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.c index d99b4fa..bc179a2 100644 --- a/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.c +++ b/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.c @@ -768,7 +768,12 @@ kib_conn_t *kiblnd_create_conn(kib_peer_t *peer, struct rdma_cm_id *cmid, conn->ibc_sched = sched; - rc = rdma_create_qp(cmid, conn->ibc_hdev->ibh_pd, init_qp_attr); + do { + rc = rdma_create_qp(cmid, conn->ibc_hdev->ibh_pd, init_qp_attr); + if (!rc || init_qp_attr->cap.max_send_wr < 16) + break; + } while (rc); + if (rc) { CERROR("Can't create QP: %d, send_wr: %d, recv_wr: %d\n", rc, init_qp_attr->cap.max_send_wr, @@ -776,6 +781,10 @@ kib_conn_t *kiblnd_create_conn(kib_peer_t *peer, struct rdma_cm_id *cmid, goto failed_2; } + if (init_qp_attr->cap.max_send_wr != IBLND_SEND_WRS(conn)) + CDEBUG(D_NET, "original send wr %d, created with %d\n", + IBLND_SEND_WRS(conn), init_qp_attr->cap.max_send_wr); + LIBCFS_FREE(init_qp_attr, sizeof(*init_qp_attr)); /* 1 ref for caller and each rxmsg */ -- 1.7.1 From jsimmons at infradead.org Mon May 9 14:53:44 2016 From: jsimmons at infradead.org (James Simmons) Date: Mon, 9 May 2016 10:53:44 -0400 Subject: [lustre-devel] [PATCH 0/4] staging: lustre: small recent fixes for LNet/libcfs Message-ID: <1462805628-23403-1-git-send-email-jsimmons@infradead.org> Here are various fixes and improvements for the libcfs and LNet layer. Two memory leaks fixed as well as adding container support for the ko2iblnd driver. Lastly handle the lower limit of the mlx5 driver for cap.max_send_wr. Dmitry Eremin (1): staging: lustre: o2iblnd: limit cap.max_send_wr for MLX5 James Simmons (2): staging: lustre: libcfs: fix memort leak in libcfs crypto layer staging: lustre: ko2iblnd: fix memory corruption with fragments Li Dongyang (1): staging: lustre: o2iblnd: make rdma_create_id() support containers .../staging/lustre/lnet/klnds/o2iblnd/o2iblnd.c | 20 +++++++++++++++----- .../staging/lustre/lnet/klnds/o2iblnd/o2iblnd.h | 2 +- .../staging/lustre/lnet/klnds/o2iblnd/o2iblnd_cb.c | 8 ++++++++ .../lustre/lnet/libcfs/linux/linux-crypto.c | 1 + 4 files changed, 25 insertions(+), 6 deletions(-) From jsimmons at infradead.org Mon May 9 14:53:46 2016 From: jsimmons at infradead.org (James Simmons) Date: Mon, 9 May 2016 10:53:46 -0400 Subject: [lustre-devel] [PATCH 2/4] staging: lustre: o2iblnd: make rdma_create_id() support containers In-Reply-To: <1462805628-23403-1-git-send-email-jsimmons@infradead.org> References: <1462805628-23403-1-git-send-email-jsimmons@infradead.org> Message-ID: <1462805628-23403-3-git-send-email-jsimmons@infradead.org> From: Li Dongyang Add support for lustre's ko2iblnd driver to work with containers which was requested by Sebastien Buisson. Signed-off-by: Li Dongyang Intel-bug-id: https://jira.hpdd.intel.com/browse/LU-6215 Reviewed-on: http://review.whamcloud.com/18759 Reviewed-by: James Simmons Reviewed-by: Sebastien Buisson Reviewed-by: Oleg Drokin Signed-off-by: James Simmons --- .../staging/lustre/lnet/klnds/o2iblnd/o2iblnd.h | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.h b/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.h index b22984f..45bbe93 100644 --- a/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.h +++ b/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.h @@ -109,7 +109,7 @@ extern kib_tunables_t kiblnd_tunables; IBLND_CREDIT_HIGHWATER_V1 : \ t->lnd_peercredits_hiw) -#define kiblnd_rdma_create_id(cb, dev, ps, qpt) rdma_create_id(&init_net, \ +#define kiblnd_rdma_create_id(cb, dev, ps, qpt) rdma_create_id(current->nsproxy->net_ns, \ cb, dev, \ ps, qpt) -- 1.7.1 From jsimmons at infradead.org Mon May 9 14:53:47 2016 From: jsimmons at infradead.org (James Simmons) Date: Mon, 9 May 2016 10:53:47 -0400 Subject: [lustre-devel] [PATCH 3/4] staging: lustre: libcfs: fix memort leak in libcfs crypto layer In-Reply-To: <1462805628-23403-1-git-send-email-jsimmons@infradead.org> References: <1462805628-23403-1-git-send-email-jsimmons@infradead.org> Message-ID: <1462805628-23403-4-git-send-email-jsimmons@infradead.org> During code review Boyko discovered a memory leak. This patch fixes that leak. Signed-off-by: James Simmons Intel-bug-id: https://jira.hpdd.intel.com/browse/LU-4423 Reviewed-on: http://review.whamcloud.com/19716 Reviewed-by: Alexander Boyko Reviewed-by: Frank Zago Reviewed-by: Oleg Drokin Signed-off-by: James Simmons --- .../lustre/lnet/libcfs/linux/linux-crypto.c | 1 + 1 files changed, 1 insertions(+), 0 deletions(-) diff --git a/drivers/staging/lustre/lnet/libcfs/linux/linux-crypto.c b/drivers/staging/lustre/lnet/libcfs/linux/linux-crypto.c index 84f9b7b..5c0116a 100644 --- a/drivers/staging/lustre/lnet/libcfs/linux/linux-crypto.c +++ b/drivers/staging/lustre/lnet/libcfs/linux/linux-crypto.c @@ -99,6 +99,7 @@ static int cfs_crypto_hash_alloc(enum cfs_crypto_hash_alg hash_alg, (*type)->cht_size); if (err != 0) { + ahash_request_free(*req); crypto_free_ahash(tfm); return err; } -- 1.7.1 From jsimmons at infradead.org Mon May 9 14:53:48 2016 From: jsimmons at infradead.org (James Simmons) Date: Mon, 9 May 2016 10:53:48 -0400 Subject: [lustre-devel] [PATCH 4/4] staging: lustre: ko2iblnd: fix memory corruption with fragments In-Reply-To: <1462805628-23403-1-git-send-email-jsimmons@infradead.org> References: <1462805628-23403-1-git-send-email-jsimmons@infradead.org> Message-ID: <1462805628-23403-5-git-send-email-jsimmons@infradead.org> In my test of the upstream client this change exposed a long standing issues where we have a offset that is not page algined would causes us to access memory beyond the scatter gather list which was causing memory corruption when all 256 fragments were in use. Signed-off-by: James Simmons --- .../staging/lustre/lnet/klnds/o2iblnd/o2iblnd.c | 9 +++++---- .../staging/lustre/lnet/klnds/o2iblnd/o2iblnd_cb.c | 8 ++++++++ 2 files changed, 13 insertions(+), 4 deletions(-) diff --git a/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.c b/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.c index bc179a2..8a18384 100644 --- a/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.c +++ b/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.c @@ -2020,8 +2020,8 @@ static void kiblnd_destroy_tx_pool(kib_pool_t *pool) sizeof(*tx->tx_pages)); if (tx->tx_frags) LIBCFS_FREE(tx->tx_frags, - IBLND_MAX_RDMA_FRAGS * - sizeof(*tx->tx_frags)); + (1 + IBLND_MAX_RDMA_FRAGS) * + sizeof(*tx->tx_frags)); if (tx->tx_wrq) LIBCFS_FREE(tx->tx_wrq, (1 + IBLND_MAX_RDMA_FRAGS) * @@ -2099,11 +2099,12 @@ static int kiblnd_create_tx_pool(kib_poolset_t *ps, int size, } LIBCFS_CPT_ALLOC(tx->tx_frags, lnet_cpt_table(), ps->ps_cpt, - IBLND_MAX_RDMA_FRAGS * sizeof(*tx->tx_frags)); + (1 + IBLND_MAX_RDMA_FRAGS) * + sizeof(*tx->tx_frags)); if (!tx->tx_frags) break; - sg_init_table(tx->tx_frags, IBLND_MAX_RDMA_FRAGS); + sg_init_table(tx->tx_frags, IBLND_MAX_RDMA_FRAGS + 1); LIBCFS_CPT_ALLOC(tx->tx_wrq, lnet_cpt_table(), ps->ps_cpt, (1 + IBLND_MAX_RDMA_FRAGS) * diff --git a/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd_cb.c b/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd_cb.c index bbfee53..0f7e3a1 100644 --- a/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd_cb.c +++ b/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd_cb.c @@ -689,6 +689,10 @@ kiblnd_setup_rd_iov(lnet_ni_t *ni, kib_tx_t *tx, kib_rdma_desc_t *rd, sg_set_page(sg, page, fragnob, page_offset); sg = sg_next(sg); + if (!sg) { + CERROR("lacking enough sg entries to map tx\n"); + return -EFAULT; + } if (offset + fragnob < iov->iov_len) { offset += fragnob; @@ -733,6 +737,10 @@ kiblnd_setup_rd_kiov(lnet_ni_t *ni, kib_tx_t *tx, kib_rdma_desc_t *rd, sg_set_page(sg, kiov->kiov_page, fragnob, kiov->kiov_offset + offset); sg = sg_next(sg); + if (!sg) { + CERROR("lacking enough sg entries to map tx\n"); + return -EFAULT; + } offset = 0; kiov++; -- 1.7.1 From dan.carpenter at oracle.com Tue May 10 13:20:16 2016 From: dan.carpenter at oracle.com (Dan Carpenter) Date: Tue, 10 May 2016 16:20:16 +0300 Subject: [lustre-devel] [PATCH 5/9] staging: lustre: fid: packing ost_idx in IDIF In-Reply-To: <1462372140-27880-6-git-send-email-jsimmons@infradead.org> References: <1462372140-27880-1-git-send-email-jsimmons@infradead.org> <1462372140-27880-6-git-send-email-jsimmons@infradead.org> Message-ID: <20160510132016.GC4298@mwanda> On Wed, May 04, 2016 at 10:28:56AM -0400, James Simmons wrote: > From: Fan Yong > > For a normal FID, we can know on which target the related object > is allocated via querying FLDB; but it is not true for an IDIF. > > To locate the OST via the given IDIF, when the IDIF is generated, > we pack the OST index in it. Then for any given FID, in spite of > t is a normal FID or not, we has the method to know which target Missing words. "in spite of *whether it* is a normal FID or not." > it belongs to. That is useful for LFSCK. > > For old IDIF, the OST index is not part of the IDIF, means that > ifferent OSTs may have the same IDIFs, that may cause the IFID different. > in LMA does not match the read FID. s/does/to/ > Under such case, we need to > make some compatible check to avoid to trigger unexpected. > > tgt_validate_obdo() converts the ostid contained in the RPC body > to fid and changes the "struct ost_id" union, then the users can > access ost_id::oi_fid directly without call ostid_to_fid() again. calling. > > It also contains some other fixing and cleanup. These are trigger words to avoid. > > Signed-off-by: wang di > Signed-off-by: Fan Yong > Intel-bug-id: https://jira.hpdd.intel.com/browse/LU-3569 > Reviewed-on: http://review.whamcloud.com/7053 > Reviewed-by: Andreas Dilger > Reviewed-by: Alex Zhuravlev > Signed-off-by: James Simmons > --- > .../lustre/lustre/include/lustre/lustre_idl.h | 76 +++++++++++++++------- > drivers/staging/lustre/lustre/include/lustre_fid.h | 22 ++----- > 2 files changed, 57 insertions(+), 41 deletions(-) > > diff --git a/drivers/staging/lustre/lustre/include/lustre/lustre_idl.h b/drivers/staging/lustre/lustre/include/lustre/lustre_idl.h > index a70545a..9c53c17 100644 > --- a/drivers/staging/lustre/lustre/include/lustre/lustre_idl.h > +++ b/drivers/staging/lustre/lustre/include/lustre/lustre_idl.h > @@ -584,7 +584,7 @@ static inline __u64 ostid_seq(const struct ost_id *ostid) > if (fid_seq_is_mdt0(ostid->oi.oi_seq)) > return FID_SEQ_OST_MDT0; > > - if (fid_seq_is_default(ostid->oi.oi_seq)) > + if (unlikely(fid_seq_is_default(ostid->oi.oi_seq))) Adding unlikely() is the opposite of a cleanup. It's now messier. That sort of thing needs to be justified by benchmarks. Please remove all the new likely/unlikelys and add them in a separate patch later. > return FID_SEQ_LOV_DEFAULT; > > if (fid_is_idif(&ostid->oi_fid)) > @@ -596,9 +596,12 @@ static inline __u64 ostid_seq(const struct ost_id *ostid) > /* extract OST objid from a wire ost_id (id/seq) pair */ > static inline __u64 ostid_id(const struct ost_id *ostid) > { > - if (fid_seq_is_mdt0(ostid_seq(ostid))) > + if (fid_seq_is_mdt0(ostid->oi.oi_seq)) > return ostid->oi.oi_id & IDIF_OID_MASK; > > + if (unlikely(fid_seq_is_default(ostid->oi.oi_seq))) > + return ostid->oi.oi_id; > + > if (fid_is_idif(&ostid->oi_fid)) > return fid_idif_id(fid_seq(&ostid->oi_fid), > fid_oid(&ostid->oi_fid), 0); > @@ -642,12 +645,22 @@ static inline void ostid_set_seq_llog(struct ost_id *oi) > */ > static inline void ostid_set_id(struct ost_id *oi, __u64 oid) > { > - if (fid_seq_is_mdt0(ostid_seq(oi))) { > + if (fid_seq_is_mdt0(oi->oi.oi_seq)) { > if (oid >= IDIF_MAX_OID) { > CERROR("Bad %llu to set " DOSTID "\n", oid, POSTID(oi)); > return; > } > oi->oi.oi_id = oid; > + } else if (fid_is_idif(&oi->oi_fid)) { > + if (oid >= IDIF_MAX_OID) { > + CERROR("Bad %llu to set "DOSTID"\n", > + oid, POSTID(oi)); > + return; > + } > + oi->oi_fid.f_seq = fid_idif_seq(oid, > + fid_idif_ost_idx(&oi->oi_fid)); > + oi->oi_fid.f_oid = oid; > + oi->oi_fid.f_ver = oid >> 48; > } else { > if (oid > OBIF_MAX_OID) { > CERROR("Bad %llu to set " DOSTID "\n", oid, POSTID(oi)); > @@ -657,25 +670,31 @@ static inline void ostid_set_id(struct ost_id *oi, __u64 oid) > } > } > > -static inline void ostid_inc_id(struct ost_id *oi) > +static inline int fid_set_id(struct lu_fid *fid, __u64 oid) > { > - if (fid_seq_is_mdt0(ostid_seq(oi))) { > - if (unlikely(ostid_id(oi) + 1 > IDIF_MAX_OID)) { > - CERROR("Bad inc "DOSTID"\n", POSTID(oi)); > - return; > + if (unlikely(fid_seq_is_igif(fid->f_seq))) { > + CERROR("bad IGIF, "DFID"\n", PFID(fid)); > + return -EBADF; > + } > + > + if (fid_is_idif(fid)) { > + if (oid >= IDIF_MAX_OID) { > + CERROR("Too large OID %#llx to set IDIF "DFID"\n", > + (unsigned long long)oid, PFID(fid)); > + return -EBADF; > } > - oi->oi.oi_id++; > + fid->f_seq = fid_idif_seq(oid, fid_idif_ost_idx(fid)); > + fid->f_oid = oid; > + fid->f_ver = oid >> 48; > } else { > - oi->oi_fid.f_oid++; > + if (oid > OBIF_MAX_OID) { Is this off by one? Hopely it is. Otherwise, it's really confusing. > + CERROR("Too large OID %#llx to set REG "DFID"\n", > + (unsigned long long)oid, PFID(fid)); > + return -EBADF; > + } > + fid->f_oid = oid; > } regards, dan carpenter From quentin.bouget.ocre at cea.fr Tue May 10 14:14:55 2016 From: quentin.bouget.ocre at cea.fr (Quentin BOUGET) Date: Tue, 10 May 2016 16:14:55 +0200 Subject: [lustre-devel] Fwd: llmount.sh does not use fs-specific paths to the ost(s) and mds it mounts In-Reply-To: <5731EAC3.5070207@cea.fr> References: <5731EAC3.5070207@cea.fr> Message-ID: <5731ECDF.10801@cea.fr> Hello! While trying to mount two different lustre filesystems on a local node with the "llmount.sh" script, I noticed that the path to mount mds and ost(s) was computed like this: eval mntpt=${!var:-${MOUNT%/*}/$facet} # var is not defined by default, so this removes # everything after the last '/' of $MOUNT The code can be found at test-framework.sh in the function "facet_mntpt" around the line 3226. By default MOUNT is initialized like this: MOUNT=${MOUNT:-/mnt/${FSNAME}} # default: "/mnt/lustre" Thus the default mountpoints for mds and ost(s) are "/mnt/mds1" and "/mnt/ost1". Because of that, when trying to do two successive calls to "llmount.sh" with two different values for FSNAME, the second fs to be mounted will unmount the first's mds and ost(s) (paths are identical). Is there any reason to truncate the MOUNT variable like this ? Shouldn't the mountpoint path look more like "/mnt/-mds1" and "/mnt/-ost1" ? Regards, Quentin Bouget -------------- next part -------------- An HTML attachment was scrubbed... URL: From john.hammond at intel.com Tue May 10 16:50:01 2016 From: john.hammond at intel.com (Hammond, John) Date: Tue, 10 May 2016 16:50:01 +0000 Subject: [lustre-devel] sanity-hsm failures on master Message-ID: All, A recent switch from CentOS 6.7 to CentOS 7 for the lustre-reviews test nodes has broken a subtest of sanity-hsm. Everything pushed against any recent revision of master is likely to fail until the fix http://review.whamcloud.com/20093 has landed. Please see LU-6937 for details. So if you have something to push you might wait until 20093 has landed or base your change on 20093. Sorry for the inconvenience. Best, John From andreas.dilger at intel.com Tue May 10 19:42:44 2016 From: andreas.dilger at intel.com (Dilger, Andreas) Date: Tue, 10 May 2016 19:42:44 +0000 Subject: [lustre-devel] Fwd: llmount.sh does not use fs-specific paths to the ost(s) and mds it mounts In-Reply-To: <5731ECDF.10801@cea.fr> References: <5731EAC3.5070207@cea.fr> <5731ECDF.10801@cea.fr> Message-ID: On 2016/05/10, 08:14, "Quentin BOUGET" wrote: > >Hello! > >While trying to mount two different lustre filesystems on a local node >with the "llmount.sh" script, I noticed that the path to mount mds and >ost(s) was computed like this: > >eval mntpt=${!var:-${MOUNT%/*}/$facet} # var is not defined by default, >so this removes > # >everything after the last '/' of $MOUNT > > >The code can be found at test-framework.sh in the function "facet_mntpt" >around the line 3226. > > >By default MOUNT is initialized like this: > >MOUNT=${MOUNT:-/mnt/${FSNAME}} # default: "/mnt/lustre" > > >Thus the default mountpoints for mds and ost(s) are "/mnt/mds1" and >"/mnt/ost1". >Because of that, when trying to do two successive calls to "llmount.sh" >with two different values for FSNAME, the second fs to be mounted will >unmount the first's mds and ost(s) (paths are identical). > > >Is there any reason to truncate the MOUNT variable like this ? Shouldn't >the mountpoint path look more like "/mnt/-mds1" and >"/mnt/-ost1"? I don't think there is any particular reason it was implemented this way, just that it has worked for many years without the need to include the fsname into the MDT/OST mount points... :-) If you want to expand it to allow exporting multiple filesystems from the same MDS/OSS nodes then your suggestion to use $FSNAME-$facet in facet_mntpt() is reasonable. Please submit a patch to verify this is working correctly. There may be some places in the tests that aren't using facet_mntpt() yet. Cheers, Andreas -- Andreas Dilger Lustre Principal Architect Intel High Performance Data Division From jodi.levi at intel.com Mon May 9 16:23:52 2016 From: jodi.levi at intel.com (Levi, Jodi) Date: Mon, 9 May 2016 16:23:52 +0000 Subject: [lustre-devel] Onyx Patching (Auto test outage) notification - May 20th 2016 Message-ID: All, We are planning a quarterly maintenance window for our cluster that hosts our Auto test instances, Jenkins, and Gerrit. This outage is needed to complete some updates that include security related patching. Please see the planned schedule and impacts to users below. May 20th 8am MT: Pause and drain all Onyx auto test queues (both instances) - Trevis auto test instances will continue testing May 24th 8am MT: Patching all of Onyx (expected duration 6 hours) - No new tests will be consumed during this time, but currently running tests on Trevis will continue NOTE: any tests that have not completed by 8am MT on Tuesday will be stopped (regardless of the remaining test time left) and we will restart them after the auto test instances are up Notification will be sent again prior to starting maintenance and again when the maintenance is complete and all instances are back on line and testing resumes. If you have any questions or concerns with this planned maintenance outage, please reply to Jodi Levi – jodi.levi at intel.com Thank you! Jodi Levi, PMP Program Manager Intel High Performance Data Division Mobile: 303-241-1428 Email: jodi.levi at intel.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From oleg.drokin at intel.com Wed May 11 01:17:10 2016 From: oleg.drokin at intel.com (Drokin, Oleg) Date: Wed, 11 May 2016 01:17:10 +0000 Subject: [lustre-devel] llmount.sh does not use fs-specific paths to the ost(s) and mds it mounts In-Reply-To: References: <5731EAC3.5070207@cea.fr> <5731ECDF.10801@cea.fr> Message-ID: <5A0B9E9B-D8D8-4645-B9F6-BBC7EF41F769@intel.com> On May 10, 2016, at 3:42 PM, Dilger, Andreas wrote: > On 2016/05/10, 08:14, "Quentin BOUGET" wrote: >> >> Hello! >> >> While trying to mount two different lustre filesystems on a local node >> with the "llmount.sh" script, I noticed that the path to mount mds and >> ost(s) was computed like this: >> >> eval mntpt=${!var:-${MOUNT%/*}/$facet} # var is not defined by default, >> so this removes >> # >> everything after the last '/' of $MOUNT >> >> >> The code can be found at test-framework.sh in the function "facet_mntpt" >> around the line 3226. >> >> >> By default MOUNT is initialized like this: >> >> MOUNT=${MOUNT:-/mnt/${FSNAME}} # default: "/mnt/lustre" >> >> >> Thus the default mountpoints for mds and ost(s) are "/mnt/mds1" and >> "/mnt/ost1". >> Because of that, when trying to do two successive calls to "llmount.sh" >> with two different values for FSNAME, the second fs to be mounted will >> unmount the first's mds and ost(s) (paths are identical). >> >> >> Is there any reason to truncate the MOUNT variable like this ? Shouldn't >> the mountpoint path look more like "/mnt/-mds1" and >> "/mnt/-ost1"? > > I don't think there is any particular reason it was implemented this way, > just that it has worked for many years without the need to include the > fsname into the MDT/OST mount points... :-) > > If you want to expand it to allow exporting multiple filesystems from the > same MDS/OSS nodes then your suggestion to use $FSNAME-$facet in > facet_mntpt() is reasonable. Please submit a patch to verify this is > working correctly. There may be some places in the tests that aren't > using facet_mntpt() yet. I guess to expand on this, since llmount.sh is primarily a testing infrastructure and typically only one filesystem at a time is tested, the more than one mountpoint was never bothering anyone till you discovered this bit, I guess. Bye, Oleg From oleg.drokin at intel.com Wed May 11 02:52:02 2016 From: oleg.drokin at intel.com (Drokin, Oleg) Date: Wed, 11 May 2016 02:52:02 +0000 Subject: [lustre-devel] New tag 2.8.53 Message-ID: <710856DE-D1A6-4C66-8303-C1B51C9465EF@intel.com> Hello! I just tagged 2.8.53 in Lustre master branch, this also includes saity-hsm rhel7 fix that John was talking about earlier today, so now is a good time to rebase if you did not yet. Here's the changelog: Alex Zhuravlev (3): LU-7902 osd: no need to set same [acm]time LU-7521 ldlm: LDLM_DEBUG() shouldn't be passed \n LU-7904 osd: honor LOC_F_NEW Alexander Boyko (1): LU-6722 ldiskfs: fix credits at ldiskfs_delete_inode Andreas Dilger (3): LU-7746 tests: some fixes for sanity.sh with upstream kernel LU-6215 llite: remove obsolete conditional code LU-6844 tests: disable DNE testing of dbench Andrew Perepechko (1): LU-8059 ldiskfs: wrong ldiskfs patch Andriy Skulysh (1): LU-7791 ldlm: signal vs CP callback race Artem Blagodarenko (1): LU-7592 osd-ldiskfs: increase supported ldiskfs fs size limit Ashley Pittman (1): LU-7835 checkpatch: Update gerrit_checkpatch for new features. Bob Glossman (3): LU-8024 kernel: kernel update [SLES12 SP1 3.12.57-60.35] LU-8052 test: stop using unknonwn sed option LU-7957 kernel: kernel update [SLES11 SP4 3.0.101-71] Bobi Jam (1): LU-7198 clio: remove mtime check in vvp_io_fault_start() Bruno Faccini (2): LU-7973 llite: lock i_lock/dcache_lock before __d_drop() LU-7980 ldiskfs: always pre-allocate max depth for path Christopher J. Morrone (5): LU-7699 build: Convert lustre_ver.h.in into a static .h file LU-5725 ofd: Expose OFD site_stats through proc LU-7699 build: Convert version underscores to dashes for dpkg LU-8072 build: Restore module debuginfo LU-8073 build: Eliminate lustre-source binary package Dmitry Eremin (4): LU-7998 utils: fix fd lost LU-7937 utils: Fix NULL pointer dereference LU-8023 lbuild: add find-requires parameter to rpmbuild LU-7999 utils: fix double free the pointer Elena Gryaznova (1): LU-7435 test: managing OST pools properly Emoly Liu (2): LU-7972 tests: use $REFORMAT in test-framework.sh correctly LU-8064 lctl: fix a typo in lustre_cfg.c Fan Yong (2): LU-5070 utils: mkfs.lustre code cleanup LU-5070 osd-zfs: get object's FID from its LMA EA Frank Zago (1): LU-8014 hsm: remove kuc_ispayload Herbert Xu (1): LU-4423 libcfs: use ahash for libcfs crypto layer Hongchao Zhang (1): LU-7851 mdt: skip SOM related configuration James Simmons (4): LU-6245 libcfs: replace CFS_MODULE_PARAM with linux kernel module api LU-6245 libcfs: remove sysctl module handling LU-6245 libcfs: Remove kp30.h LU-8017 obd: report correct health state of a node Jian Yu (1): LU-8015 utils: fix lr_link() to get correct src and dest Jinshan Xiong (3): LU-7990 ofd: add a parameter to adjust preferred brw size LU-8005 osc: set lock data for readahead lock LU-7990 clio: revise readahead to support 16MB IO John L. Hammond (6): LU-7992 hsm: close KUC pipe before unregistering LU-7986 hsm: update actions llog in place LU-7402 llite: correct request handling after ll_lookup_it() LU-7403 llite: flatten struct lookup_intent LU-7152 hsm: sync volatile file before setting times LU-6937 test: use ps in get_agent_uuid() Kirtankumar Krishna Shetty (1): LU-8051 tests: Skipping sanity tests 300[a-d] for old servers. Kit Westneat (1): LU-8001 nodemap: fix null deref when reclassifying Kyrylo Shatskyy (1): LU-7130 test: sanity/162 doesn't handle DIR values properly Li Dongyang (3): LU-6215 gss: key->payload.data is an array in 4.4+ kernels LU-6215 o2iblnd: ib_alloc_fast_reg_mr() removed in 4.3 LU-6215 o2iblnd: rdma_create_id() takes extra parameter Li Xi (1): LU-7982 libcfs: memory allocation without CPT for binheap Lokesh Nagappa Jaliminche (1): LU-7922 ldiskfs: correction in ext4_kzalloc Mel Gorman (1): LU-4423 mem: remove GFP_IOFS Mikhal Pershin (1): LU-7558 import: don't reconnect during connect interpret Nathaniel Clark (1): LU-7859 tests: Use getent instead of /etc/passwd Oleg Drokin (3): Revert "LU-7434 ptlrpc: lost bulk leads to a hang" Revert "LU-7791 ldlm: signal vs CP callback race" New tag 2.8.53 Parinay Kondekar (4): LU-7110 tests: sanity-hsm.sh can erase all files on filesystem LU-7121 tests: Extra items shown in the stack trace in t-f LU-7323 test: Unknown symbol mb_cache_create (err 0)" on RHEL7 LU-6512 llite:ll_dir_ioctl cleanup of redundant comparisons Patrick Farrell (1): LU-7981 llite: take trunc_sem only at vvp layer Tatsushi Takamura (1): LU-8009 utils: Lustre_rsync does not sync files on MDT1 Vitaly Fertman (2): LU-7434 ptlrpc: lost bulk leads to a hang LU-7434 ptlrpc: Early Reply vs Reply MDunlink Wang Shilong (1): LU-8037 fileset: reject dotdot as a path component Yang Sheng (3): LU-1118 llite: Invoke file_update_time in page_mkwrite LU-6601 tests: test case validate OSP status op LU-8038 tests: test case for lfs migrate --block From quentin.bouget.ocre at cea.fr Wed May 11 07:07:15 2016 From: quentin.bouget.ocre at cea.fr (Quentin BOUGET) Date: Wed, 11 May 2016 09:07:15 +0200 Subject: [lustre-devel] llmount.sh does not use fs-specific paths to the ost(s) and mds it mounts In-Reply-To: <5A0B9E9B-D8D8-4645-B9F6-BBC7EF41F769@intel.com> References: <5731EAC3.5070207@cea.fr> <5731ECDF.10801@cea.fr> <5A0B9E9B-D8D8-4645-B9F6-BBC7EF41F769@intel.com> Message-ID: <5732DA23.6070603@cea.fr> Le 11/05/2016 03:17, Drokin, Oleg a écrit : > On May 10, 2016, at 3:42 PM, Dilger, Andreas wrote: > >> On 2016/05/10, 08:14, "Quentin BOUGET" wrote: >>> Hello! >>> >>> While trying to mount two different lustre filesystems on a local node >>> with the "llmount.sh" script, I noticed that the path to mount mds and >>> ost(s) was computed like this: >>> >>> eval mntpt=${!var:-${MOUNT%/*}/$facet} # var is not defined by default, >>> so this removes >>> # >>> everything after the last '/' of $MOUNT >>> >>> >>> The code can be found at test-framework.sh in the function "facet_mntpt" >>> around the line 3226. >>> >>> >>> By default MOUNT is initialized like this: >>> >>> MOUNT=${MOUNT:-/mnt/${FSNAME}} # default: "/mnt/lustre" >>> >>> >>> Thus the default mountpoints for mds and ost(s) are "/mnt/mds1" and >>> "/mnt/ost1". >>> Because of that, when trying to do two successive calls to "llmount.sh" >>> with two different values for FSNAME, the second fs to be mounted will >>> unmount the first's mds and ost(s) (paths are identical). >>> >>> >>> Is there any reason to truncate the MOUNT variable like this ? Shouldn't >>> the mountpoint path look more like "/mnt/-mds1" and >>> "/mnt/-ost1"? >> I don't think there is any particular reason it was implemented this way, >> just that it has worked for many years without the need to include the >> fsname into the MDT/OST mount points... :-) >> >> If you want to expand it to allow exporting multiple filesystems from the >> same MDS/OSS nodes then your suggestion to use $FSNAME-$facet in >> facet_mntpt() is reasonable. Please submit a patch to verify this is >> working correctly. There may be some places in the tests that aren't >> using facet_mntpt() yet. > I guess to expand on this, since llmount.sh is primarily a testing > infrastructure and typically only one filesystem at a time is tested, > the more than one mountpoint was never bothering anyone till you > discovered this bit, I guess. > > Bye, > Oleg > > Ok, thank you. I will submit a patch about it. Bye, Quentin From jsimmons at infradead.org Wed May 11 15:54:01 2016 From: jsimmons at infradead.org (James Simmons) Date: Wed, 11 May 2016 16:54:01 +0100 (BST) Subject: [lustre-devel] [PATCH] xattr handlers: fixup generic_listxattr In-Reply-To: <551ED650-CBF8-40D6-85A0-BDB3C281C247@dilger.ca> References: <1462229118-13123-1-git-send-email-agruenba@redhat.com> <551ED650-CBF8-40D6-85A0-BDB3C281C247@dilger.ca> Message-ID: While looking at porting lustre to use xattr handlers I noticed issues in generic_listxattr() when handle->list() is used. The function generic_listxattr() generates it results from the handle->name field. If the current handle name field is NULL then generic_listxattr() will call handle->list() if it exist. Calling handle->list() has no affect on the output since their is no way to set the name field of the handler. This patch allows the passing in of the handler to *list() so the handle->name field can be set. Now with the ability to create the name field after its contents are transfer to the passed in buffer the name field has to be freed to prevent memory leaks. Also freeing the memory of the name field ensures that the *list() function will be called at a latter time which ensures the xattr list is not stale. This patch is against Viro's work.xattrs branch. Signed-off-by: James Simmons --- fs/ext2/xattr.c | 2 +- fs/ext4/xattr.c | 2 +- fs/f2fs/xattr.c | 2 +- fs/reiserfs/xattr.c | 2 +- fs/squashfs/xattr.c | 2 +- fs/xattr.c | 8 ++++++-- include/linux/xattr.h | 2 +- 7 files changed, 12 insertions(+), 8 deletions(-) diff --git a/fs/ext2/xattr.c b/fs/ext2/xattr.c index 1a5e3bf..0aab6a5 100644 --- a/fs/ext2/xattr.c +++ b/fs/ext2/xattr.c @@ -290,7 +290,7 @@ bad_block: ext2_error(inode->i_sb, "ext2_xattr_list", const struct xattr_handler *handler = ext2_xattr_handler(entry->e_name_index); - if (handler && (!handler->list || handler->list(dentry))) { + if (handler && (!handler->list || handler->list(handler, dentry))) { const char *prefix = handler->prefix ?: handler->name; size_t prefix_len = strlen(prefix); size_t size = prefix_len + entry->e_name_len + 1; diff --git a/fs/ext4/xattr.c b/fs/ext4/xattr.c index 0441e05..0941198 100644 --- a/fs/ext4/xattr.c +++ b/fs/ext4/xattr.c @@ -402,7 +402,7 @@ ext4_xattr_list_entries(struct dentry *dentry, struct ext4_xattr_entry *entry, const struct xattr_handler *handler = ext4_xattr_handler(entry->e_name_index); - if (handler && (!handler->list || handler->list(dentry))) { + if (handler && (!handler->list || handler->list(handler, dentry))) { const char *prefix = handler->prefix ?: handler->name; size_t prefix_len = strlen(prefix); size_t size = prefix_len + entry->e_name_len + 1; diff --git a/fs/f2fs/xattr.c b/fs/f2fs/xattr.c index 17fd2b1..0cb0fc1 100644 --- a/fs/f2fs/xattr.c +++ b/fs/f2fs/xattr.c @@ -412,7 +412,7 @@ ssize_t f2fs_listxattr(struct dentry *dentry, char *buffer, size_t buffer_size) size_t prefix_len; size_t size; - if (!handler || (handler->list && !handler->list(dentry))) + if (!handler || (handler->list && !handler->list(handler, dentry))) continue; prefix = handler->prefix ?: handler->name; diff --git a/fs/reiserfs/xattr.c b/fs/reiserfs/xattr.c index 02137bb..509d2a0 100644 --- a/fs/reiserfs/xattr.c +++ b/fs/reiserfs/xattr.c @@ -787,7 +787,7 @@ static int listxattr_filler(struct dir_context *ctx, const char *name, handler = find_xattr_handler_prefix(b->dentry->d_sb->s_xattr, name); if (!handler /* Unsupported xattr name */ || - (handler->list && !handler->list(b->dentry))) + (handler->list && !handler->list(handler, b->dentry))) return 0; size = namelen + 1; if (b->buf) { diff --git a/fs/squashfs/xattr.c b/fs/squashfs/xattr.c index 1548b37..64e8573 100644 --- a/fs/squashfs/xattr.c +++ b/fs/squashfs/xattr.c @@ -67,7 +67,7 @@ ssize_t squashfs_listxattr(struct dentry *d, char *buffer, name_size = le16_to_cpu(entry.size); handler = squashfs_xattr_handler(le16_to_cpu(entry.type)); - if (handler && (!handler->list || handler->list(d))) { + if (handler && (!handler->list || handler->list(handler, d))) { const char *prefix = handler->prefix ?: handler->name; size_t prefix_size = strlen(prefix); diff --git a/fs/xattr.c b/fs/xattr.c index b11945e..1ec51a7 100644 --- a/fs/xattr.c +++ b/fs/xattr.c @@ -716,9 +716,11 @@ generic_listxattr(struct dentry *dentry, char *buffer, size_t buffer_size) if (!buffer) { for_each_xattr_handler(handlers, handler) { if (!handler->name || - (handler->list && !handler->list(dentry))) + (handler->list && !handler->list(handler, dentry))) continue; size += strlen(handler->name) + 1; + if (handler->list) + kfree(handler->name); } } else { char *buf = buffer; @@ -726,7 +728,7 @@ generic_listxattr(struct dentry *dentry, char *buffer, size_t buffer_size) for_each_xattr_handler(handlers, handler) { if (!handler->name || - (handler->list && !handler->list(dentry))) + (handler->list && !handler->list(handler, dentry))) continue; len = strlen(handler->name); if (len + 1 > buffer_size) @@ -734,6 +736,8 @@ generic_listxattr(struct dentry *dentry, char *buffer, size_t buffer_size) memcpy(buf, handler->name, len + 1); buf += len + 1; buffer_size -= len + 1; + if (handler->list) + kfree(handler->name); } size = buf - buffer; } diff --git a/include/linux/xattr.h b/include/linux/xattr.h index 1cc4c57..e4fb463 100644 --- a/include/linux/xattr.h +++ b/include/linux/xattr.h @@ -28,7 +28,7 @@ struct xattr_handler { const char *name; const char *prefix; int flags; /* fs private flags */ - bool (*list)(struct dentry *dentry); + bool (*list)(const struct xattr_handler *, struct dentry *dentry); int (*get)(const struct xattr_handler *, struct dentry *dentry, struct inode *inode, const char *name, void *buffer, size_t size); -- 1.7.1 From jsimmons at infradead.org Wed May 11 15:58:29 2016 From: jsimmons at infradead.org (James Simmons) Date: Wed, 11 May 2016 16:58:29 +0100 (BST) Subject: [lustre-devel] llmount.sh does not use fs-specific paths to the ost(s) and mds it mounts In-Reply-To: <5732DA23.6070603@cea.fr> References: <5731EAC3.5070207@cea.fr> <5731ECDF.10801@cea.fr> <5A0B9E9B-D8D8-4645-B9F6-BBC7EF41F769@intel.com> <5732DA23.6070603@cea.fr> Message-ID: > Le 11/05/2016 03:17, Drokin, Oleg a écrit : > > On May 10, 2016, at 3:42 PM, Dilger, Andreas wrote: > > > > > On 2016/05/10, 08:14, "Quentin BOUGET" wrote: > > > > Hello! > > > > > > > > While trying to mount two different lustre filesystems on a local node > > > > with the "llmount.sh" script, I noticed that the path to mount mds and > > > > ost(s) was computed like this: > > > > > > > > eval mntpt=${!var:-${MOUNT%/*}/$facet} # var is not defined by default, > > > > so this removes > > > > # > > > > everything after the last '/' of $MOUNT > > > > > > > > > > > > The code can be found at test-framework.sh in the function "facet_mntpt" > > > > around the line 3226. > > > > > > > > > > > > By default MOUNT is initialized like this: > > > > > > > > MOUNT=${MOUNT:-/mnt/${FSNAME}} # default: "/mnt/lustre" > > > > > > > > > > > > Thus the default mountpoints for mds and ost(s) are "/mnt/mds1" and > > > > "/mnt/ost1". > > > > Because of that, when trying to do two successive calls to "llmount.sh" > > > > with two different values for FSNAME, the second fs to be mounted will > > > > unmount the first's mds and ost(s) (paths are identical). > > > > > > > > > > > > Is there any reason to truncate the MOUNT variable like this ? Shouldn't > > > > the mountpoint path look more like "/mnt/-mds1" and > > > > "/mnt/-ost1"? > > > I don't think there is any particular reason it was implemented this way, > > > just that it has worked for many years without the need to include the > > > fsname into the MDT/OST mount points... :-) > > > > > > If you want to expand it to allow exporting multiple filesystems from the > > > same MDS/OSS nodes then your suggestion to use $FSNAME-$facet in > > > facet_mntpt() is reasonable. Please submit a patch to verify this is > > > working correctly. There may be some places in the tests that aren't > > > using facet_mntpt() yet. > > I guess to expand on this, since llmount.sh is primarily a testing > > infrastructure and typically only one filesystem at a time is tested, > > the more than one mountpoint was never bothering anyone till you > > discovered this bit, I guess. > > > > Bye, > > Oleg > > > > > Ok, thank you. I will submit a patch about it. Please post a link to the JIRA ticket here. Thanks for the work. > Bye, > Quentin > _______________________________________________ > lustre-devel mailing list > lustre-devel at lists.lustre.org > http://lists.lustre.org/listinfo.cgi/lustre-devel-lustre.org > From jsimmons at infradead.org Wed May 11 16:22:09 2016 From: jsimmons at infradead.org (James Simmons) Date: Wed, 11 May 2016 17:22:09 +0100 (BST) Subject: [lustre-devel] [PATCH 1/4] staging: lustre: o2iblnd: limit cap.max_send_wr for MLX5 (fwd) Message-ID: > James, > I’m pretty sure this port to upstream is flawed. See below Yep. I missed that line. Thanks for review. Greg you can drop this one. I will submit a updated version soon. This patch series is not order dependent so the other patches are safe to land. > On 5/9/16, 7:53 AM, "lustre-devel on behalf of James Simmons" wrote: > >>From: Dmitry Eremin >> >>Decrease cap.max_send_wr until it is accepted by rdma_create_qp() >> >>Signed-off-by: Dmitry Eremin >>Intel-bug-id: https://jira.hpdd.intel.com/browse/LU-7124 >>Reviewed-on: http://review.whamcloud.com/18347 >>Reviewed-by: Olaf Weber >>Reviewed-by: Doug Oucharek >>Reviewed-by: Oleg Drokin >>Signed-off-by: James Simmons >>--- >> .../staging/lustre/lnet/klnds/o2iblnd/o2iblnd.c | 11 ++++++++++- >> 1 files changed, 10 insertions(+), 1 deletions(-) >> >>diff --git a/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.c b/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.c >>index d99b4fa..bc179a2 100644 >>--- a/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.c >>+++ b/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.c >>@@ -768,7 +768,12 @@ kib_conn_t *kiblnd_create_conn(kib_peer_t *peer, struct rdma_cm_id *cmid, >> >> conn->ibc_sched = sched; >> >> - rc = rdma_create_qp(cmid, conn->ibc_hdev->ibh_pd, init_qp_attr); >> + do { >> + rc = rdma_create_qp(cmid, conn->ibc_hdev->ibh_pd, init_qp_attr); >> + if (!rc || init_qp_attr->cap.max_send_wr < 16) >> + break; > > I think the loop here is missing code to decrease the max_send_wr value. In current master it’s > > init_qp_attr->cap.max_send_wr -= init_qp_attr->cap.max_send_wr / 4; > >>+ } while (rc); >>+ >> if (rc) { >> CERROR("Can't create QP: %d, send_wr: %d, recv_wr: %d\n", >> rc, init_qp_attr->cap.max_send_wr, >>@@ -776,6 +781,10 @@ kib_conn_t *kiblnd_create_conn(kib_peer_t *peer, struct rdma_cm_id *cmid, >> goto failed_2; >> } >> >>+ if (init_qp_attr->cap.max_send_wr != IBLND_SEND_WRS(conn)) >>+ CDEBUG(D_NET, "original send wr %d, created with %d\n", >>+ IBLND_SEND_WRS(conn), init_qp_attr->cap.max_send_wr); >>+ >> LIBCFS_FREE(init_qp_attr, sizeof(*init_qp_attr)); >> >> /* 1 ref for caller and each rxmsg */ >> -- >> 1.7.1 >> From alexey.lyashkov at seagate.com Wed May 11 16:32:26 2016 From: alexey.lyashkov at seagate.com (Alexey Lyashkov) Date: Wed, 11 May 2016 19:32:26 +0300 Subject: [lustre-devel] [PATCH 1/4] staging: lustre: o2iblnd: limit cap.max_send_wr for MLX5 In-Reply-To: <1462805628-23403-2-git-send-email-jsimmons@infradead.org> References: <1462805628-23403-1-git-send-email-jsimmons@infradead.org> <1462805628-23403-2-git-send-email-jsimmons@infradead.org> Message-ID: As Sergey say in original ticket, It patch change system tunable without any visible effects. So user will continue to think it uses a max sends XX when system have change it to some other value. Let's change CDEBUG(D_NET to warning, or rework a patch. It patch needs because someone may set max sends over HW card limitation, so good fix will take config from card and adjust max sends without a loop. On Mon, May 9, 2016 at 5:53 PM, James Simmons wrote: > From: Dmitry Eremin > > Decrease cap.max_send_wr until it is accepted by rdma_create_qp() > > Signed-off-by: Dmitry Eremin > Intel-bug-id: https://jira.hpdd.intel.com/browse/LU-7124 > Reviewed-on: http://review.whamcloud.com/18347 > Reviewed-by: Olaf Weber > Reviewed-by: Doug Oucharek > Reviewed-by: Oleg Drokin > Signed-off-by: James Simmons > --- > .../staging/lustre/lnet/klnds/o2iblnd/o2iblnd.c | 11 ++++++++++- > 1 files changed, 10 insertions(+), 1 deletions(-) > > diff --git a/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.c > b/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.c > index d99b4fa..bc179a2 100644 > --- a/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.c > +++ b/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.c > @@ -768,7 +768,12 @@ kib_conn_t *kiblnd_create_conn(kib_peer_t *peer, > struct rdma_cm_id *cmid, > > conn->ibc_sched = sched; > > - rc = rdma_create_qp(cmid, conn->ibc_hdev->ibh_pd, init_qp_attr); > + do { > + rc = rdma_create_qp(cmid, conn->ibc_hdev->ibh_pd, > init_qp_attr); > + if (!rc || init_qp_attr->cap.max_send_wr < 16) > + break; > + } while (rc); > + > if (rc) { > CERROR("Can't create QP: %d, send_wr: %d, recv_wr: %d\n", > rc, init_qp_attr->cap.max_send_wr, > @@ -776,6 +781,10 @@ kib_conn_t *kiblnd_create_conn(kib_peer_t *peer, > struct rdma_cm_id *cmid, > goto failed_2; > } > > + if (init_qp_attr->cap.max_send_wr != IBLND_SEND_WRS(conn)) > + CDEBUG(D_NET, "original send wr %d, created with %d\n", > + IBLND_SEND_WRS(conn), > init_qp_attr->cap.max_send_wr); > + > LIBCFS_FREE(init_qp_attr, sizeof(*init_qp_attr)); > > /* 1 ref for caller and each rxmsg */ > -- > 1.7.1 > > _______________________________________________ > lustre-devel mailing list > lustre-devel at lists.lustre.org > http://lists.lustre.org/listinfo.cgi/lustre-devel-lustre.org > -- Alexey Lyashkov *·* Technical lead for a Morpheus team Seagate Technology, LLC www.seagate.com www.lustre.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From alexey.lyashkov at seagate.com Wed May 11 16:44:55 2016 From: alexey.lyashkov at seagate.com (Alexey Lyashkov) Date: Wed, 11 May 2016 19:44:55 +0300 Subject: [lustre-devel] [PATCH 1/4] staging: lustre: o2iblnd: limit cap.max_send_wr for MLX5 In-Reply-To: <1462805628-23403-2-git-send-email-jsimmons@infradead.org> References: <1462805628-23403-1-git-send-email-jsimmons@infradead.org> <1462805628-23403-2-git-send-email-jsimmons@infradead.org> Message-ID: Sergey Cheremensev comment - [49672.067906] mlx5_ib:mlx5_0:calc_sq_size:485:(pid 8297): wqe_size 192 [49672.067908] mlx5_ib:mlx5_0:calc_sq_size:507:(pid 8297): wqe count(65536) exceeds limits(16384) [49672.067910] mlx5_ib:mlx5_0:create_kernel_qp:1051:(pid 8297): err -12 According to above data mlx5 has internal limit for wqe count 16384: wq_size = roundup_pow_of_two(attr->cap.max_send_wr * wqe_size); qp->sq.wqe_cnt = wq_size / MLX5_SEND_WQE_BB; if (qp->sq.wqe_cnt > (1 << MLX5_CAP_GEN(dev->mdev, log_max_qp_sz))) { mlx5_ib_dbg(dev, "wqe count(%d) exceeds limits(%d)\n", qp->sq.wqe_cnt, 1 << MLX5_CAP_GEN(dev->mdev, log_max_qp_sz)); return -ENOMEM; } So -12(ENOMEM) in message "Can't create QP" doesn't point to any problems with free memory in the system. Imo better error code here is -EINVAL. It seems peer_credits==16 is the maximum value that is supported by mlx5. On Mon, May 9, 2016 at 5:53 PM, James Simmons wrote: > From: Dmitry Eremin > > Decrease cap.max_send_wr until it is accepted by rdma_create_qp() > > Signed-off-by: Dmitry Eremin > Intel-bug-id: https://jira.hpdd.intel.com/browse/LU-7124 > Reviewed-on: http://review.whamcloud.com/18347 > Reviewed-by: Olaf Weber > Reviewed-by: Doug Oucharek > Reviewed-by: Oleg Drokin > Signed-off-by: James Simmons > --- > .../staging/lustre/lnet/klnds/o2iblnd/o2iblnd.c | 11 ++++++++++- > 1 files changed, 10 insertions(+), 1 deletions(-) > > diff --git a/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.c > b/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.c > index d99b4fa..bc179a2 100644 > --- a/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.c > +++ b/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.c > @@ -768,7 +768,12 @@ kib_conn_t *kiblnd_create_conn(kib_peer_t *peer, > struct rdma_cm_id *cmid, > > conn->ibc_sched = sched; > > - rc = rdma_create_qp(cmid, conn->ibc_hdev->ibh_pd, init_qp_attr); > + do { > + rc = rdma_create_qp(cmid, conn->ibc_hdev->ibh_pd, > init_qp_attr); > + if (!rc || init_qp_attr->cap.max_send_wr < 16) > + break; > + } while (rc); > + > if (rc) { > CERROR("Can't create QP: %d, send_wr: %d, recv_wr: %d\n", > rc, init_qp_attr->cap.max_send_wr, > @@ -776,6 +781,10 @@ kib_conn_t *kiblnd_create_conn(kib_peer_t *peer, > struct rdma_cm_id *cmid, > goto failed_2; > } > > + if (init_qp_attr->cap.max_send_wr != IBLND_SEND_WRS(conn)) > + CDEBUG(D_NET, "original send wr %d, created with %d\n", > + IBLND_SEND_WRS(conn), > init_qp_attr->cap.max_send_wr); > + > LIBCFS_FREE(init_qp_attr, sizeof(*init_qp_attr)); > > /* 1 ref for caller and each rxmsg */ > -- > 1.7.1 > > _______________________________________________ > lustre-devel mailing list > lustre-devel at lists.lustre.org > http://lists.lustre.org/listinfo.cgi/lustre-devel-lustre.org > -- Alexey Lyashkov *·* Technical lead for a Morpheus team Seagate Technology, LLC www.seagate.com www.lustre.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From jsimmons at infradead.org Wed May 11 23:53:25 2016 From: jsimmons at infradead.org (James Simmons) Date: Thu, 12 May 2016 00:53:25 +0100 (BST) Subject: [lustre-devel] staging: add Lustre file system client support In-Reply-To: <20160427122846.GA6441@mwanda> References: <20160427122846.GA6441@mwanda> Message-ID: > Hello Lustre devs, > > The patch d7e09d0397e8: "staging: add Lustre file system client > support" from May 2, 2013, leads to the following Parfait warning. > Parfait is an Oracle static analysis tool. If there is a patch from > this could you give credit to Lidza Louina Just to let you know we have a ticket to track this issue at: https://jira.hpdd.intel.com/browse/LU-8128 The reason we have been avoiding this is that in the past small fixes to the ldlm layer has caused havoc at very large scale. So we are very careful when approaching this class of fixes. > drivers/staging/lustre/lustre/ldlm/interval_tree.c > 399 void interval_erase(struct interval_node *node, > 400 struct interval_node **root) > 401 { > 402 struct interval_node *child, *parent; > 403 int color; > 404 > 405 LASSERT(interval_is_intree(node)); > 406 node->in_intree = 0; > 407 if (!node->in_left) { > 408 child = node->in_right; > 409 } else if (!node->in_right) { > 410 child = node->in_left; > 411 } else { /* Both left and right child are not NULL */ > 412 struct interval_node *old = node; > 413 > 414 node = interval_next(node); > ^^^^^^^^^^^^^^^^^^^^^^^^^^ > It looks like interval_next() can return NULL. > > 415 child = node->in_right; > 416 parent = node->in_parent; > 417 color = node->in_color; > 418 > > Here is the interval_next() function: > > drivers/staging/lustre/lustre/ldlm/interval_tree.c > 111 static struct interval_node *interval_next(struct interval_node *node) > 112 { > 113 if (!node) > 114 return NULL; > 115 if (node->in_right) > 116 return interval_first(node->in_right); > 117 while (node->in_parent && node_is_right_child(node)) > ^^^^^^^^^^^^^^^ > We assume that ->in_parent can be NULL here. Is that actually possible? > > 118 node = node->in_parent; > 119 return node->in_parent; > 120 } > > regards, > dan carpenter > _______________________________________________ > devel mailing list > devel at linuxdriverproject.org > http://driverdev.linuxdriverproject.org/mailman/listinfo/driverdev-devel > From jsimmons at infradead.org Thu May 12 00:28:54 2016 From: jsimmons at infradead.org (James Simmons) Date: Thu, 12 May 2016 01:28:54 +0100 (BST) Subject: [lustre-devel] Lustre use of hash_long() & cfs_hash_u32_hash() In-Reply-To: <20160501234312.20601.qmail@ns.horizon.com> References: <20160501234312.20601.qmail@ns.horizon.com> Message-ID: > I notice you have copies of the functions in libcfs_hash.h, > and use a mixture of the generic and copied functions in the Lustre code. > > I'm revising the hash_64() function to fix some horrible collision > problems that Thomas Gleixner found, and I wanted to give you a heads-up. > > If there are cases where you use the generic ones but need > reproducible outputs, they'll break. > > If there are places where you're using the copied ones and *don't* > need reproducible outputs, you're getting bad performance. > > I made a brief attempt to figure out what is used for what, but it's > too confusing for me. It's hard to figure out code like: Thanks for looking into this. Do you have a tree some where we can look at for the changes? We opened a ticket to help track the progress for this at https://jira.hpdd.intel.com/browse/LU-8130 Opening tickets is what gets people to look at these types of changes. Again thanks for letting us know about these changes. > static unsigned lu_obj_hop_hash(struct cfs_hash *hs, > const void *key, unsigned mask) > { > struct lu_fid *fid = (struct lu_fid *)key; > __u32 hash; > > hash = fid_flatten32(fid); > hash += (hash >> 4) + (hash << 12); /* mixing oid and seq */ > hash = hash_long(hash, hs->hs_bkt_bits); > > /* give me another random factor */ > hash -= hash_long((unsigned long)hs, fid_oid(fid) % 11 + 3); > > hash <<= hs->hs_cur_bits - hs->hs_bkt_bits; > hash |= (fid_seq(fid) + fid_oid(fid)) & (CFS_HASH_NBKT(hs) - 1); > > return hash & mask; > } > > The whole business of selecting the number of bits of the hash based > on a mod-11 hash of something else seems quite peculiar. > > Second, hash_long is multiplicative, meaning hash_long(a) - hash_long(b) > = hash_long(a-b). The different bit shifts complicate it, but subtracting > two such values from each other is not a great way to mix. > > Third, you're using the functions strangely. The first hash_long() takes > a 32-bit input and would be more efficient on 64-bit platforms if it > were hash_32. The second could be hash_ptr(), Details about this change are at https://jira.hpdd.intel.com/browse/LU-143 and the original patch with those changes are at http://review.whamcloud.com/#/c/374 Is this the only code that doesn't make sense or is their more? Looking at the source tree I see the hash code being used in lnet/lnet/api-ni.c lnet/lnet/lib-ptl.c lustre/include/lustre_fid.h lustre/ldlm/ldlm_resource.c lustre/obdclass/lu_object.c From linux at horizon.com Thu May 12 01:33:22 2016 From: linux at horizon.com (George Spelvin) Date: 11 May 2016 21:33:22 -0400 Subject: [lustre-devel] Lustre use of hash_long() & cfs_hash_u32_hash() In-Reply-To: Message-ID: <20160512013322.5730.qmail@ns.horizon.com> Thank you very much for the response! > Thanks for looking into this. Do you have a tree some where we can look > at for the changes? Linus committed the most important fix to 4.6-rc7 as 689de1d6ca95b3b5bd8ee446863bf81a4883ea25 I'm following up with additional, less urgent cleanups of kernel hashing. (Fundamentally, I'm trying to reduce the number of different hash functions, and find a good balance between execution-time performance and mixing performance of what remains.) > Details about this change are at > > https://jira.hpdd.intel.com/browse/LU-143 > > and the original patch with those changes are at > > http://review.whamcloud.com/#/c/374 Thanks a lot for the pointers! > Is this the only code that doesn't make sense or is their more? I expect there's more; that was just what first jumped out at me. My primary goal was elsewhere, so once it became clear that Lustre would be a time-consuming side quest, I gave up and sent that e-mail. But if Lustre would appreciate more attention, I can take another look. I did do a pass over the kernel and fixed improper use of the current functions. I posted it as Subject: [RFC PATCH 3/2] (Rant) Fix various hash abuses https://marc.info/?l=linux-kernel&m=146218484201133 Since Lustre was the single biggest culprit (about 25% of that patch), I was planning on sending a broken-out patch. From quentin.bouget.ocre at cea.fr Thu May 12 07:17:42 2016 From: quentin.bouget.ocre at cea.fr (Quentin BOUGET) Date: Thu, 12 May 2016 09:17:42 +0200 Subject: [lustre-devel] llmount.sh does not use fs-specific paths to the ost(s) and mds it mounts In-Reply-To: References: <5731EAC3.5070207@cea.fr> <5731ECDF.10801@cea.fr> <5A0B9E9B-D8D8-4645-B9F6-BBC7EF41F769@intel.com> <5732DA23.6070603@cea.fr> Message-ID: <57342E16.5070702@cea.fr> Le 11/05/2016 17:58, James Simmons a écrit : >> Le 11/05/2016 03:17, Drokin, Oleg a écrit : >>> On May 10, 2016, at 3:42 PM, Dilger, Andreas wrote: >>> >>>> On 2016/05/10, 08:14, "Quentin BOUGET" wrote: >>>>> Hello! >>>>> >>>>> While trying to mount two different lustre filesystems on a local node >>>>> with the "llmount.sh" script, I noticed that the path to mount mds and >>>>> ost(s) was computed like this: >>>>> >>>>> eval mntpt=${!var:-${MOUNT%/*}/$facet} # var is not defined by default, >>>>> so this removes >>>>> # >>>>> everything after the last '/' of $MOUNT >>>>> >>>>> >>>>> The code can be found at test-framework.sh in the function "facet_mntpt" >>>>> around the line 3226. >>>>> >>>>> >>>>> By default MOUNT is initialized like this: >>>>> >>>>> MOUNT=${MOUNT:-/mnt/${FSNAME}} # default: "/mnt/lustre" >>>>> >>>>> >>>>> Thus the default mountpoints for mds and ost(s) are "/mnt/mds1" and >>>>> "/mnt/ost1". >>>>> Because of that, when trying to do two successive calls to "llmount.sh" >>>>> with two different values for FSNAME, the second fs to be mounted will >>>>> unmount the first's mds and ost(s) (paths are identical). >>>>> >>>>> >>>>> Is there any reason to truncate the MOUNT variable like this ? Shouldn't >>>>> the mountpoint path look more like "/mnt/-mds1" and >>>>> "/mnt/-ost1"? >>>> I don't think there is any particular reason it was implemented this way, >>>> just that it has worked for many years without the need to include the >>>> fsname into the MDT/OST mount points... :-) >>>> >>>> If you want to expand it to allow exporting multiple filesystems from the >>>> same MDS/OSS nodes then your suggestion to use $FSNAME-$facet in >>>> facet_mntpt() is reasonable. Please submit a patch to verify this is >>>> working correctly. There may be some places in the tests that aren't >>>> using facet_mntpt() yet. >>> I guess to expand on this, since llmount.sh is primarily a testing >>> infrastructure and typically only one filesystem at a time is tested, >>> the more than one mountpoint was never bothering anyone till you >>> discovered this bit, I guess. >>> >>> Bye, >>> Oleg >>> >>> >> Ok, thank you. I will submit a patch about it. > Please post a link to the JIRA ticket here. Thanks for the work. > >> Bye, >> Quentin >> _______________________________________________ >> lustre-devel mailing list >> lustre-devel at lists.lustre.org >> http://lists.lustre.org/listinfo.cgi/lustre-devel-lustre.org Here is a link to the JIRA ticket: https://jira.hpdd.intel.com/browse/LU-8132 From oleg.drokin at intel.com Fri May 13 16:18:57 2016 From: oleg.drokin at intel.com (Drokin, Oleg) Date: Fri, 13 May 2016 16:18:57 +0000 Subject: [lustre-devel] staging: add Lustre file system client support In-Reply-To: <20160427122846.GA6441@mwanda> References: <20160427122846.GA6441@mwanda> Message-ID: <2312D059-0594-4A81-90F8-E439C1A533B0@intel.com> On Apr 27, 2016, at 8:28 AM, Dan Carpenter wrote: > Hello Lustre devs, > > The patch d7e09d0397e8: "staging: add Lustre file system client > support" from May 2, 2013, leads to the following Parfait warning. > Parfait is an Oracle static analysis tool. If there is a patch from > this could you give credit to Lidza Louina ? > > drivers/staging/lustre/lustre/ldlm/interval_tree.c > 399 void interval_erase(struct interval_node *node, > 400 struct interval_node **root) > 401 { > 402 struct interval_node *child, *parent; > 403 int color; > 404 > 405 LASSERT(interval_is_intree(node)); > 406 node->in_intree = 0; > 407 if (!node->in_left) { > 408 child = node->in_right; > 409 } else if (!node->in_right) { > 410 child = node->in_left; > 411 } else { /* Both left and right child are not NULL */ > 412 struct interval_node *old = node; > 413 > 414 node = interval_next(node); > ^^^^^^^^^^^^^^^^^^^^^^^^^^ > It looks like interval_next() can return NULL. In fact it cannot. When we are called here, node->in_right is not NULL. > > 415 child = node->in_right; > 416 parent = node->in_parent; > 417 color = node->in_color; > 418 > > Here is the interval_next() function: > > drivers/staging/lustre/lustre/ldlm/interval_tree.c > 111 static struct interval_node *interval_next(struct interval_node *node) > 112 { > 113 if (!node) > 114 return NULL; > 115 if (node->in_right) > 116 return interval_first(node->in_right); So we take this branch here. and interval_first does not return NULL unless node itself is NULL. Thanks. Bye, Oleg From oleg.drokin at intel.com Fri May 13 16:43:09 2016 From: oleg.drokin at intel.com (Drokin, Oleg) Date: Fri, 13 May 2016 16:43:09 +0000 Subject: [lustre-devel] [PATCH] staging/lustre/ptlrpc: Removes potential null dereference In-Reply-To: <1463092392.16344.4.camel@oracle.com> References: <1463092392.16344.4.camel@oracle.com> Message-ID: <7641B85C-A5E1-499C-9426-8A860900C35C@intel.com> On May 12, 2016, at 6:33 PM, Lidza Louina wrote: > The lustre_msg_buf method could return NULL. Subsequent code didn't > check if it's null before using it. This patch adds two checks. Thank you for the patch, it looks good code-wise, but there are some style issues. > > Signed-off-by: Lidza Louina > > diff --git a/drivers/staging/lustre/lustre/ptlrpc/sec.c b/drivers/staging/lustre/lustre/ptlrpc/sec.c > index 187fd1d..e6fedc3 100644 > --- a/drivers/staging/lustre/lustre/ptlrpc/sec.c > +++ b/drivers/staging/lustre/lustre/ptlrpc/sec.c > @@ -2195,6 +2195,8 @@ int sptlrpc_pack_user_desc(struct lustre_msg *msg, int offset) > struct ptlrpc_user_desc *pud; > > pud = lustre_msg_buf(msg, offset, 0); > + if (!pud) > + return -EINVAL; > > pud->pud_uid = from_kuid(&init_user_ns, current_uid()); > pud->pud_gid = from_kgid(&init_user_ns, current_gid()); > diff --git a/drivers/staging/lustre/lustre/ptlrpc/sec_plain.c b/drivers/staging/lustre/lustre/ptlrpc/sec_plain.c > index 37c9f4c..7736aa9 100644 > --- a/drivers/staging/lustre/lustre/ptlrpc/sec_plain.c > +++ b/drivers/staging/lustre/lustre/ptlrpc/sec_plain.c > @@ -542,6 +542,7 @@ int plain_alloc_reqbuf(struct ptlrpc_sec *sec, > { > __u32 buflens[PLAIN_PACK_SEGMENTS] = { 0, }; > int alloc_len; > + int desc; > > buflens[PLAIN_PACK_HDR_OFF] = sizeof(struct plain_header); > buflens[PLAIN_PACK_MSG_OFF] = msgsize; > @@ -575,7 +576,10 @@ int plain_alloc_reqbuf(struct ptlrpc_sec *sec, > req->rq_reqmsg = lustre_msg_buf(req->rq_reqbuf, PLAIN_PACK_MSG_OFF, 0); > > if (req->rq_pack_udesc) > - sptlrpc_pack_user_desc(req->rq_reqbuf, PLAIN_PACK_USER_OFF); > + desc = sptlrpc_pack_user_desc(req->rq_reqbuf, PLAIN_PACK_USER_OFF); This makes the line longer than 80 chars. Also why not declare desc right here as opposed to the start of the function like you did? > + if (!desc){ > + return desc; > + } We don't really need these curvy braces here. Thanks! From falakreyaz at gmail.com Wed May 11 15:23:31 2016 From: falakreyaz at gmail.com (Muhammad Falak R Wani) Date: Wed, 11 May 2016 20:53:31 +0530 Subject: [lustre-devel] [PATCH] staging: lustre: use setup_timer(). Message-ID: <1462980251-6608-1-git-send-email-falakreyaz@gmail.com> Use setup_timer() for initializing the timer, instead of structure assignments. This is the preferred/standard way. Signed-off-by: Muhammad Falak R Wani --- drivers/staging/lustre/lnet/lnet/net_fault.c | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/drivers/staging/lustre/lnet/lnet/net_fault.c b/drivers/staging/lustre/lnet/lnet/net_fault.c index 7d76f28..e4aceb7 100644 --- a/drivers/staging/lustre/lnet/lnet/net_fault.c +++ b/drivers/staging/lustre/lnet/lnet/net_fault.c @@ -760,9 +760,7 @@ lnet_delay_rule_add(struct lnet_fault_attr *attr) wait_event(delay_dd.dd_ctl_waitq, delay_dd.dd_running); } - init_timer(&rule->dl_timer); - rule->dl_timer.function = delay_timer_cb; - rule->dl_timer.data = (unsigned long)rule; + setup_timer(&rule->dl_timer, delay_timer_cb, (unsigned long)rule); spin_lock_init(&rule->dl_lock); INIT_LIST_HEAD(&rule->dl_msg_list); -- 1.9.1 From lidza.louina at oracle.com Thu May 12 22:33:12 2016 From: lidza.louina at oracle.com (Lidza Louina) Date: Thu, 12 May 2016 18:33:12 -0400 Subject: [lustre-devel] [PATCH] staging/lustre/ptlrpc: Removes potential null dereference Message-ID: <1463092392.16344.4.camel@oracle.com> The lustre_msg_buf method could return NULL. Subsequent code didn't check if it's null before using it. This patch adds two checks.      Signed-off-by: Lidza Louina diff --git a/drivers/staging/lustre/lustre/ptlrpc/sec.c b/drivers/staging/lustre/lustre/ptlrpc/sec.c index 187fd1d..e6fedc3 100644 --- a/drivers/staging/lustre/lustre/ptlrpc/sec.c +++ b/drivers/staging/lustre/lustre/ptlrpc/sec.c @@ -2195,6 +2195,8 @@ int sptlrpc_pack_user_desc(struct lustre_msg *msg, int offset)   struct ptlrpc_user_desc *pud;     pud = lustre_msg_buf(msg, offset, 0); + if (!pud) + return -EINVAL;     pud->pud_uid = from_kuid(&init_user_ns, current_uid());   pud->pud_gid = from_kgid(&init_user_ns, current_gid()); diff --git a/drivers/staging/lustre/lustre/ptlrpc/sec_plain.c b/drivers/staging/lustre/lustre/ptlrpc/sec_plain.c index 37c9f4c..7736aa9 100644 --- a/drivers/staging/lustre/lustre/ptlrpc/sec_plain.c +++ b/drivers/staging/lustre/lustre/ptlrpc/sec_plain.c @@ -542,6 +542,7 @@ int plain_alloc_reqbuf(struct ptlrpc_sec *sec,  {   __u32 buflens[PLAIN_PACK_SEGMENTS] = { 0, };   int alloc_len; + int desc;     buflens[PLAIN_PACK_HDR_OFF] = sizeof(struct plain_header);   buflens[PLAIN_PACK_MSG_OFF] = msgsize; @@ -575,7 +576,10 @@ int plain_alloc_reqbuf(struct ptlrpc_sec *sec,   req->rq_reqmsg = lustre_msg_buf(req->rq_reqbuf, PLAIN_PACK_MSG_OFF, 0);     if (req->rq_pack_udesc) - sptlrpc_pack_user_desc(req->rq_reqbuf, PLAIN_PACK_USER_OFF); + desc = sptlrpc_pack_user_desc(req->rq_reqbuf, PLAIN_PACK_USER_OFF); + if (!desc){ + return desc; + }     return 0;  } From agruenba at redhat.com Wed May 11 17:01:37 2016 From: agruenba at redhat.com (Andreas Gruenbacher) Date: Wed, 11 May 2016 19:01:37 +0200 Subject: [lustre-devel] [PATCH] xattr handlers: fixup generic_listxattr In-Reply-To: References: <1462229118-13123-1-git-send-email-agruenba@redhat.com> <551ED650-CBF8-40D6-85A0-BDB3C281C247@dilger.ca> Message-ID: On Wed, May 11, 2016 at 5:54 PM, James Simmons wrote: > While looking at porting lustre to use xattr handlers I > noticed issues in generic_listxattr() when handle->list() > is used. The function generic_listxattr() generates it > results from the handle->name field. If the current handle > name field is NULL then generic_listxattr() will call > handle->list() if it exist. generic_listxattr() is different from generic_getxattr() / generic_setxattr() / generic_removexattr. It makes sense only for filesystems that support a fixed set of xattrs, which means that all handlers will have handler->name set. If any of the handlers has handler->prefix set instead, that handler matches a whole set of attributes. Generic_listxattr() would have to fill in all of those names matching that handler, but it doesn't know which those are. It is common for filesystems to have their own listxattr inode operation and still use generic_{get,set,remove}xattr. > Calling handle->list() has no > affect on the output since their is no way to set the > name field of the handler. This patch allows the passing > in of the handler to *list() so the handle->name field > can be set. No, that's broken. The handlers pointed at by sb->s_xattr must not be modified. Doing so would mess up all other concurrent getxattr / setxattr / listxattr / removexattr operations on that superblock. Andreas From jsimmons at infradead.org Fri May 13 19:38:31 2016 From: jsimmons at infradead.org (James Simmons) Date: Fri, 13 May 2016 20:38:31 +0100 (BST) Subject: [lustre-devel] Lustre use of hash_long() & cfs_hash_u32_hash() In-Reply-To: <20160512013322.5730.qmail@ns.horizon.com> References: <20160512013322.5730.qmail@ns.horizon.com> Message-ID: > Thank you very much for the response! > > > Thanks for looking into this. Do you have a tree some where we can look > > at for the changes? > > Linus committed the most important fix to 4.6-rc7 as > 689de1d6ca95b3b5bd8ee446863bf81a4883ea25 Not too huge of a changed. Looks like I need to move to using these functions. > I'm following up with additional, less urgent cleanups of kernel hashing. > > (Fundamentally, I'm trying to reduce the number of different hash > functions, and find a good balance between execution-time performance > and mixing performance of what remains.) Looking at our code I see our duplication is cfs_hash_u32_hash and cfs_hash_u64_hash which could be replaced by the standard linux functions. Am I missing anything else? > > Details about this change are at > > > > https://jira.hpdd.intel.com/browse/LU-143 > > > > and the original patch with those changes are at > > > > http://review.whamcloud.com/#/c/374 > > Thanks a lot for the pointers! Any time. I also CC Liang Zhen who wrote the bulk of the hashing code. > > Is this the only code that doesn't make sense or is their more? > > I expect there's more; that was just what first jumped out at me. > My primary goal was elsewhere, so once it became clear that Lustre would > be a time-consuming side quest, I gave up and sent that e-mail. > > But if Lustre would appreciate more attention, I can take another look. Yep. We are here to help. > I did do a pass over the kernel and fixed improper use of the current > functions. I posted it as > > Subject: [RFC PATCH 3/2] (Rant) Fix various hash abuses > https://marc.info/?l=linux-kernel&m=146218484201133 > > Since Lustre was the single biggest culprit (about 25% of that patch), > I was planning on sending a broken-out patch. I expect this is not all the changes needed. Do you have a newer patch or should I run with this patch? Also I will look into replace the cfs_hash_u[32|64]_* code with standard linux hash code. From jsimmons at infradead.org Fri May 13 19:42:42 2016 From: jsimmons at infradead.org (James Simmons) Date: Fri, 13 May 2016 20:42:42 +0100 (BST) Subject: [lustre-devel] [PATCH] staging: lustre: use setup_timer(). In-Reply-To: <1462980251-6608-1-git-send-email-falakreyaz@gmail.com> References: <1462980251-6608-1-git-send-email-falakreyaz@gmail.com> Message-ID: > Use setup_timer() for initializing the timer, instead of structure > assignments. > This is the preferred/standard way. > > Signed-off-by: Muhammad Falak R Wani Acked-by: James Simmons > --- > drivers/staging/lustre/lnet/lnet/net_fault.c | 4 +--- > 1 file changed, 1 insertion(+), 3 deletions(-) > > diff --git a/drivers/staging/lustre/lnet/lnet/net_fault.c b/drivers/staging/lustre/lnet/lnet/net_fault.c > index 7d76f28..e4aceb7 100644 > --- a/drivers/staging/lustre/lnet/lnet/net_fault.c > +++ b/drivers/staging/lustre/lnet/lnet/net_fault.c > @@ -760,9 +760,7 @@ lnet_delay_rule_add(struct lnet_fault_attr *attr) > wait_event(delay_dd.dd_ctl_waitq, delay_dd.dd_running); > } > > - init_timer(&rule->dl_timer); > - rule->dl_timer.function = delay_timer_cb; > - rule->dl_timer.data = (unsigned long)rule; > + setup_timer(&rule->dl_timer, delay_timer_cb, (unsigned long)rule); > > spin_lock_init(&rule->dl_lock); > INIT_LIST_HEAD(&rule->dl_msg_list); > -- > 1.9.1 > > _______________________________________________ > lustre-devel mailing list > lustre-devel at lists.lustre.org > http://lists.lustre.org/listinfo.cgi/lustre-devel-lustre.org > From linux at horizon.com Fri May 13 20:25:07 2016 From: linux at horizon.com (George Spelvin) Date: 13 May 2016 16:25:07 -0400 Subject: [lustre-devel] Lustre use of hash_long() & cfs_hash_u32_hash() In-Reply-To: Message-ID: <20160513202507.26286.qmail@ns.horizon.com> > Looking at our code I see our duplication is cfs_hash_u32_hash and > cfs_hash_u64_hash which could be replaced by the standard linux functions. > Am I missing anything else? The question is, why were they copied in the first place? The functions are for in-memory hash tables and *not* guaranteed stable. between boots. They could depend on kernel version, architecture, kernel configuration, and boot-time feature detection. I thought the reason for the copy might have been to make a stable hash that could e.g. be sent between hosts. (em28xx_hash_mem() in drivers/media/usb/em28xx/em28xx-i2c.c is an example of this. It's a copy of hash_mem() from .)) If there's no reason for the duplication, then yes, merge them. >> Since Lustre was the single biggest culprit (about 25% of that patch), >> I was planning on sending a broken-out patch. > I expect this is not all the changes needed. Do you have a newer patch or > should I run with this patch? Also I will look into replace the > cfs_hash_u[32|64]_* code with standard linux hash code. I don't have anything newer ATM. I agree there's probably more; if nothing else I didn't check the copied hash functions at all. What I posted was just what I noticed during a search through the kernel for all users of the functions I was planning on changing. From oleg.drokin at intel.com Fri May 13 20:35:03 2016 From: oleg.drokin at intel.com (Drokin, Oleg) Date: Fri, 13 May 2016 20:35:03 +0000 Subject: [lustre-devel] Lustre use of hash_long() & cfs_hash_u32_hash() In-Reply-To: <20160513202507.26286.qmail@ns.horizon.com> References: <20160513202507.26286.qmail@ns.horizon.com> Message-ID: <73B34265-39B4-4273-B1F4-0F209232008F@intel.com> On May 13, 2016, at 4:25 PM, George Spelvin wrote: >> Looking at our code I see our duplication is cfs_hash_u32_hash and >> cfs_hash_u64_hash which could be replaced by the standard linux functions. >> Am I missing anything else? > > The question is, why were they copied in the first place? > > The functions are for in-memory hash tables and *not* > guaranteed stable. between boots. They could depend on kernel version, > architecture, kernel configuration, and boot-time feature detection. > > I thought the reason for the copy might have been to make a stable hash > that could e.g. be sent between hosts. they were copied because of other platforms. There used to be a userspace library, a mac port and so on. That's why we have this (now trimmed down) compatibility layer that we are chipping at. the hashes are purely node-local and are not sent over the wire. Thanks! Bye, Oleg From jsimmons at infradead.org Mon May 16 17:28:20 2016 From: jsimmons at infradead.org (James Simmons) Date: Mon, 16 May 2016 18:28:20 +0100 (BST) Subject: [lustre-devel] Lustre use of hash_long() & cfs_hash_u32_hash() In-Reply-To: <20160513202507.26286.qmail@ns.horizon.com> References: <20160513202507.26286.qmail@ns.horizon.com> Message-ID: > > Looking at our code I see our duplication is cfs_hash_u32_hash and > > cfs_hash_u64_hash which could be replaced by the standard linux functions. > > Am I missing anything else? > > The question is, why were they copied in the first place? ... > If there's no reason for the duplication, then yes, merge them. As Oleg pointed out their is no need for the duplication. > >> Since Lustre was the single biggest culprit (about 25% of that patch), > >> I was planning on sending a broken-out patch. > > > I expect this is not all the changes needed. Do you have a newer patch or > > should I run with this patch? Also I will look into replace the > > cfs_hash_u[32|64]_* code with standard linux hash code. > > I don't have anything newer ATM. I agree there's probably more; if > nothing else I didn't check the copied hash functions at all. What I > posted was just what I noticed during a search through the kernel for > all users of the functions I was planning on changing. I started a patch with your cleanups. So I started to look at replacing cfs_hash_u[64|32]_hash() with the standard hash_[64|32]() function. First I need to explain the code path to make it clear what is going on. >From the top level the code path in which the cfs_hash_uXX_hash might be called is: cfs_hash_bd_get() -> cfs_hash_bd_from_key() -> cfs_hash_id() cfs_hash_id() is defined as: static inline unsigned cfs_hash_id(struct cfs_hash *hs, const void *key, unsigned mask) { return hs->hs_ops->hs_hash(hs, key, mask); } and hs_hash is struct cfs_hash_ops { /** return hashed value from @key */ unsigned (*hs_hash)(struct cfs_hash *hs, const void *key, unsigned mask); ... } So ops->hs_hash() is filled in by various lustre layers to generate some hash value. It is these *hs_hash() that some times uses cfs_hash_u64_hash() and cfs_hash_u32_hash(). An example of this is in cl_object.c: struct cfs_hash_ops pool_hash_operations = { .hs_hash = cl_env_hops_hash, ... }; which is defined as: static unsigned cl_env_hops_hash(struct cfs_hash *lh, const void *key, unsigned mask) { #if BITS_PER_LONG == 64 return cfs_hash_u64_hash((__u64)key, mask); #else return cfs_hash_u32_hash((__u32)key, mask); #endif } If we change to using hash_64() or hash_32() instead this will change the behavior we currently have. Now cfs_hash_u64_hash() is defined as static inline unsigned cfs_hash_u64_hash(const __u64 key, unsigned mask) { return ((unsigned)(key * CFS_GOLDEN_RATIO_PRIME_64) & mask); } which uses a mask directly. That mask is generated in cfs_hash_id() with : unsigned int index = cfs_hash_id(hs, key, (1U << bits) - 1); So the mask uses the lower bits. We can't change this behavior since other hp_ops->hs_hash functions that don't use cfs_hash_u[64|32]_hash() at all treat this as a mask. Now for using the standand functions we have for example: static __always_inline u64 hash_64(u64 val, unsigned int bits) { u64 hash = val; #if BITS_PER_LONG == 64 hash = hash * GOLDEN_RATIO_64; #else .... #endif /* High bits are more random, so use them. */ return hash >> (64 - bits); } which generates the mask on the fly using the higher bits. I can work the code so the mask can be turned into a bit offset but the mask will still be different. Will this difference break things? From lidza.louina at oracle.com Mon May 16 14:11:52 2016 From: lidza.louina at oracle.com (Lidza Louina) Date: Mon, 16 May 2016 10:11:52 -0400 Subject: [lustre-devel] [PATCH] staging/lustre/ptlrpc: Removes potential null dereference In-Reply-To: <7641B85C-A5E1-499C-9426-8A860900C35C@intel.com> References: <1463092392.16344.4.camel@oracle.com> <7641B85C-A5E1-499C-9426-8A860900C35C@intel.com> Message-ID: <5739D528.4000606@oracle.com> On 05/13/2016 12:43 PM, Drokin, Oleg wrote: > On May 12, 2016, at 6:33 PM, Lidza Louina wrote: > >> The lustre_msg_buf method could return NULL. Subsequent code didn't >> check if it's null before using it. This patch adds two checks. > Thank you for the patch, it looks good code-wise, but there are some style issues. > >> >> Signed-off-by: Lidza Louina >> >> diff --git a/drivers/staging/lustre/lustre/ptlrpc/sec.c b/drivers/staging/lustre/lustre/ptlrpc/sec.c >> index 187fd1d..e6fedc3 100644 >> --- a/drivers/staging/lustre/lustre/ptlrpc/sec.c >> +++ b/drivers/staging/lustre/lustre/ptlrpc/sec.c >> @@ -2195,6 +2195,8 @@ int sptlrpc_pack_user_desc(struct lustre_msg *msg, int offset) >> struct ptlrpc_user_desc *pud; >> >> pud = lustre_msg_buf(msg, offset, 0); >> + if (!pud) >> + return -EINVAL; >> >> pud->pud_uid = from_kuid(&init_user_ns, current_uid()); >> pud->pud_gid = from_kgid(&init_user_ns, current_gid()); >> diff --git a/drivers/staging/lustre/lustre/ptlrpc/sec_plain.c b/drivers/staging/lustre/lustre/ptlrpc/sec_plain.c >> index 37c9f4c..7736aa9 100644 >> --- a/drivers/staging/lustre/lustre/ptlrpc/sec_plain.c >> +++ b/drivers/staging/lustre/lustre/ptlrpc/sec_plain.c >> @@ -542,6 +542,7 @@ int plain_alloc_reqbuf(struct ptlrpc_sec *sec, >> { >> __u32 buflens[PLAIN_PACK_SEGMENTS] = { 0, }; >> int alloc_len; >> + int desc; >> >> buflens[PLAIN_PACK_HDR_OFF] = sizeof(struct plain_header); >> buflens[PLAIN_PACK_MSG_OFF] = msgsize; >> @@ -575,7 +576,10 @@ int plain_alloc_reqbuf(struct ptlrpc_sec *sec, >> req->rq_reqmsg = lustre_msg_buf(req->rq_reqbuf, PLAIN_PACK_MSG_OFF, 0); >> >> if (req->rq_pack_udesc) >> - sptlrpc_pack_user_desc(req->rq_reqbuf, PLAIN_PACK_USER_OFF); >> + desc = sptlrpc_pack_user_desc(req->rq_reqbuf, PLAIN_PACK_USER_OFF); > This makes the line longer than 80 chars. > Also why not declare desc right here as opposed to the start of the function like you did? > >> + if (!desc){ >> + return desc; >> + } > We don't really need these curvy braces here. > > Thanks! > > Thanks for the feedback. I'm sending a new patch in now. Lidza From lidza.louina at oracle.com Mon May 16 14:17:51 2016 From: lidza.louina at oracle.com (Lidza Louina) Date: Mon, 16 May 2016 10:17:51 -0400 Subject: [lustre-devel] [PATCH v2] staging/lustre/ptlrpc: Removes potential null dereference Message-ID: <1463408271-18079-1-git-send-email-lidza.louina@oracle.com> The lustre_msg_buf method could return NULL. Subsequent code didn't check if it's null before using it. This patch adds two checks. Signed-off-by: Lidza Louina --- drivers/staging/lustre/lustre/ptlrpc/sec.c | 2 ++ drivers/staging/lustre/lustre/ptlrpc/sec_plain.c | 9 +++++++-- 2 files changed, 9 insertions(+), 2 deletions(-) diff --git a/drivers/staging/lustre/lustre/ptlrpc/sec.c b/drivers/staging/lustre/lustre/ptlrpc/sec.c index 187fd1d..e6fedc3 100644 --- a/drivers/staging/lustre/lustre/ptlrpc/sec.c +++ b/drivers/staging/lustre/lustre/ptlrpc/sec.c @@ -2195,6 +2195,8 @@ int sptlrpc_pack_user_desc(struct lustre_msg *msg, int offset) struct ptlrpc_user_desc *pud; pud = lustre_msg_buf(msg, offset, 0); + if (!pud) + return -EINVAL; pud->pud_uid = from_kuid(&init_user_ns, current_uid()); pud->pud_gid = from_kgid(&init_user_ns, current_gid()); diff --git a/drivers/staging/lustre/lustre/ptlrpc/sec_plain.c b/drivers/staging/lustre/lustre/ptlrpc/sec_plain.c index 37c9f4c..51741c8 100644 --- a/drivers/staging/lustre/lustre/ptlrpc/sec_plain.c +++ b/drivers/staging/lustre/lustre/ptlrpc/sec_plain.c @@ -542,6 +542,7 @@ int plain_alloc_reqbuf(struct ptlrpc_sec *sec, { __u32 buflens[PLAIN_PACK_SEGMENTS] = { 0, }; int alloc_len; + int desc; buflens[PLAIN_PACK_HDR_OFF] = sizeof(struct plain_header); buflens[PLAIN_PACK_MSG_OFF] = msgsize; @@ -574,8 +575,12 @@ int plain_alloc_reqbuf(struct ptlrpc_sec *sec, lustre_init_msg_v2(req->rq_reqbuf, PLAIN_PACK_SEGMENTS, buflens, NULL); req->rq_reqmsg = lustre_msg_buf(req->rq_reqbuf, PLAIN_PACK_MSG_OFF, 0); - if (req->rq_pack_udesc) - sptlrpc_pack_user_desc(req->rq_reqbuf, PLAIN_PACK_USER_OFF); + if (req->rq_pack_udesc) { + desc = sptlrpc_pack_user_desc(req->rq_reqbuf, + PLAIN_PACK_USER_OFF); + if (!desc) + return desc; + } return 0; } -- 1.9.1 From paf at cray.com Mon May 16 17:57:55 2016 From: paf at cray.com (Patrick Farrell) Date: Mon, 16 May 2016 12:57:55 -0500 Subject: [lustre-devel] [PATCH v2] staging/lustre/ptlrpc: Removes potential null dereference In-Reply-To: <1463408271-18079-1-git-send-email-lidza.louina@oracle.com> References: <1463408271-18079-1-git-send-email-lidza.louina@oracle.com> Message-ID: <573A0A23.2010705@cray.com> This looks wrong - You return -EINVAL from sptlrpc_pack_user_desc, but then the caller checks "!desc". Desc will not be null, since you've returned -EINVAL. - Patrick On 05/16/2016 09:17 AM, Lidza Louina wrote: > The lustre_msg_buf method could return NULL. Subsequent code didn't > check if it's null before using it. This patch adds two checks. > > Signed-off-by: Lidza Louina > --- > drivers/staging/lustre/lustre/ptlrpc/sec.c | 2 ++ > drivers/staging/lustre/lustre/ptlrpc/sec_plain.c | 9 +++++++-- > 2 files changed, 9 insertions(+), 2 deletions(-) > > diff --git a/drivers/staging/lustre/lustre/ptlrpc/sec.c b/drivers/staging/lustre/lustre/ptlrpc/sec.c > index 187fd1d..e6fedc3 100644 > --- a/drivers/staging/lustre/lustre/ptlrpc/sec.c > +++ b/drivers/staging/lustre/lustre/ptlrpc/sec.c > @@ -2195,6 +2195,8 @@ int sptlrpc_pack_user_desc(struct lustre_msg *msg, int offset) > struct ptlrpc_user_desc *pud; > > pud = lustre_msg_buf(msg, offset, 0); > + if (!pud) > + return -EINVAL; > > pud->pud_uid = from_kuid(&init_user_ns, current_uid()); > pud->pud_gid = from_kgid(&init_user_ns, current_gid()); > diff --git a/drivers/staging/lustre/lustre/ptlrpc/sec_plain.c b/drivers/staging/lustre/lustre/ptlrpc/sec_plain.c > index 37c9f4c..51741c8 100644 > --- a/drivers/staging/lustre/lustre/ptlrpc/sec_plain.c > +++ b/drivers/staging/lustre/lustre/ptlrpc/sec_plain.c > @@ -542,6 +542,7 @@ int plain_alloc_reqbuf(struct ptlrpc_sec *sec, > { > __u32 buflens[PLAIN_PACK_SEGMENTS] = { 0, }; > int alloc_len; > + int desc; > > buflens[PLAIN_PACK_HDR_OFF] = sizeof(struct plain_header); > buflens[PLAIN_PACK_MSG_OFF] = msgsize; > @@ -574,8 +575,12 @@ int plain_alloc_reqbuf(struct ptlrpc_sec *sec, > lustre_init_msg_v2(req->rq_reqbuf, PLAIN_PACK_SEGMENTS, buflens, NULL); > req->rq_reqmsg = lustre_msg_buf(req->rq_reqbuf, PLAIN_PACK_MSG_OFF, 0); > > - if (req->rq_pack_udesc) > - sptlrpc_pack_user_desc(req->rq_reqbuf, PLAIN_PACK_USER_OFF); > + if (req->rq_pack_udesc) { > + desc = sptlrpc_pack_user_desc(req->rq_reqbuf, > + PLAIN_PACK_USER_OFF); > + if (!desc) > + return desc; > + } > > return 0; > } From jsimmons at infradead.org Mon May 16 18:16:38 2016 From: jsimmons at infradead.org (James Simmons) Date: Mon, 16 May 2016 19:16:38 +0100 (BST) Subject: [lustre-devel] [PATCH v2] staging/lustre/ptlrpc: Removes potential null dereference In-Reply-To: <573A0A23.2010705@cray.com> References: <1463408271-18079-1-git-send-email-lidza.louina@oracle.com> <573A0A23.2010705@cray.com> Message-ID: > This looks wrong - You return -EINVAL from sptlrpc_pack_user_desc, but then > the caller checks "!desc". Desc will not be null, since you've returned > -EINVAL. Actually 'if (!desc)' is equal to 'if (desc != 0). Yes it can be confusing. I recommend 'if (desc < 0)' instead to make it clearer what is being tested for. > - Patrick > > On 05/16/2016 09:17 AM, Lidza Louina wrote: > > The lustre_msg_buf method could return NULL. Subsequent code didn't > > check if it's null before using it. This patch adds two checks. > > > > Signed-off-by: Lidza Louina > > --- > > drivers/staging/lustre/lustre/ptlrpc/sec.c | 2 ++ > > drivers/staging/lustre/lustre/ptlrpc/sec_plain.c | 9 +++++++-- > > 2 files changed, 9 insertions(+), 2 deletions(-) > > > > diff --git a/drivers/staging/lustre/lustre/ptlrpc/sec.c > > b/drivers/staging/lustre/lustre/ptlrpc/sec.c > > index 187fd1d..e6fedc3 100644 > > --- a/drivers/staging/lustre/lustre/ptlrpc/sec.c > > +++ b/drivers/staging/lustre/lustre/ptlrpc/sec.c > > @@ -2195,6 +2195,8 @@ int sptlrpc_pack_user_desc(struct lustre_msg *msg, int > > offset) > > struct ptlrpc_user_desc *pud; > > > > pud = lustre_msg_buf(msg, offset, 0); > > + if (!pud) > > + return -EINVAL; > > > > pud->pud_uid = from_kuid(&init_user_ns, current_uid()); > > pud->pud_gid = from_kgid(&init_user_ns, current_gid()); > > diff --git a/drivers/staging/lustre/lustre/ptlrpc/sec_plain.c > > b/drivers/staging/lustre/lustre/ptlrpc/sec_plain.c > > index 37c9f4c..51741c8 100644 > > --- a/drivers/staging/lustre/lustre/ptlrpc/sec_plain.c > > +++ b/drivers/staging/lustre/lustre/ptlrpc/sec_plain.c > > @@ -542,6 +542,7 @@ int plain_alloc_reqbuf(struct ptlrpc_sec *sec, > > { > > __u32 buflens[PLAIN_PACK_SEGMENTS] = { 0, }; > > int alloc_len; > > + int desc; > > > > buflens[PLAIN_PACK_HDR_OFF] = sizeof(struct plain_header); > > buflens[PLAIN_PACK_MSG_OFF] = msgsize; > > @@ -574,8 +575,12 @@ int plain_alloc_reqbuf(struct ptlrpc_sec *sec, > > lustre_init_msg_v2(req->rq_reqbuf, PLAIN_PACK_SEGMENTS, buflens, NULL); > > req->rq_reqmsg = lustre_msg_buf(req->rq_reqbuf, PLAIN_PACK_MSG_OFF, 0); > > - if (req->rq_pack_udesc) > > - sptlrpc_pack_user_desc(req->rq_reqbuf, PLAIN_PACK_USER_OFF); > > + if (req->rq_pack_udesc) { > > + desc = sptlrpc_pack_user_desc(req->rq_reqbuf, > > + PLAIN_PACK_USER_OFF); > > + if (!desc) > > + return desc; > > + } > > > > return 0; > > } > > _______________________________________________ > lustre-devel mailing list > lustre-devel at lists.lustre.org > http://lists.lustre.org/listinfo.cgi/lustre-devel-lustre.org > From paf at cray.com Mon May 16 18:22:41 2016 From: paf at cray.com (Patrick Farrell) Date: Mon, 16 May 2016 13:22:41 -0500 Subject: [lustre-devel] [PATCH v2] staging/lustre/ptlrpc: Removes potential null dereference In-Reply-To: References: <1463408271-18079-1-git-send-email-lidza.louina@oracle.com> <573A0A23.2010705@cray.com> Message-ID: <573A0FF1.9090101@cray.com> James, No. You've got it backwards. 0 is false, any non-zero value is true. if(desc) would be equal to if (desc != 0). - Patrick On 05/16/2016 01:16 PM, James Simmons wrote: >> This looks wrong - You return -EINVAL from sptlrpc_pack_user_desc, but then >> the caller checks "!desc". Desc will not be null, since you've returned >> -EINVAL. > Actually 'if (!desc)' is equal to 'if (desc != 0). Yes it can be confusing. > I recommend 'if (desc < 0)' instead to make it clearer what is being > tested for. > >> - Patrick >> >> On 05/16/2016 09:17 AM, Lidza Louina wrote: >>> The lustre_msg_buf method could return NULL. Subsequent code didn't >>> check if it's null before using it. This patch adds two checks. >>> >>> Signed-off-by: Lidza Louina >>> --- >>> drivers/staging/lustre/lustre/ptlrpc/sec.c | 2 ++ >>> drivers/staging/lustre/lustre/ptlrpc/sec_plain.c | 9 +++++++-- >>> 2 files changed, 9 insertions(+), 2 deletions(-) >>> >>> diff --git a/drivers/staging/lustre/lustre/ptlrpc/sec.c >>> b/drivers/staging/lustre/lustre/ptlrpc/sec.c >>> index 187fd1d..e6fedc3 100644 >>> --- a/drivers/staging/lustre/lustre/ptlrpc/sec.c >>> +++ b/drivers/staging/lustre/lustre/ptlrpc/sec.c >>> @@ -2195,6 +2195,8 @@ int sptlrpc_pack_user_desc(struct lustre_msg *msg, int >>> offset) >>> struct ptlrpc_user_desc *pud; >>> >>> pud = lustre_msg_buf(msg, offset, 0); >>> + if (!pud) >>> + return -EINVAL; >>> >>> pud->pud_uid = from_kuid(&init_user_ns, current_uid()); >>> pud->pud_gid = from_kgid(&init_user_ns, current_gid()); >>> diff --git a/drivers/staging/lustre/lustre/ptlrpc/sec_plain.c >>> b/drivers/staging/lustre/lustre/ptlrpc/sec_plain.c >>> index 37c9f4c..51741c8 100644 >>> --- a/drivers/staging/lustre/lustre/ptlrpc/sec_plain.c >>> +++ b/drivers/staging/lustre/lustre/ptlrpc/sec_plain.c >>> @@ -542,6 +542,7 @@ int plain_alloc_reqbuf(struct ptlrpc_sec *sec, >>> { >>> __u32 buflens[PLAIN_PACK_SEGMENTS] = { 0, }; >>> int alloc_len; >>> + int desc; >>> >>> buflens[PLAIN_PACK_HDR_OFF] = sizeof(struct plain_header); >>> buflens[PLAIN_PACK_MSG_OFF] = msgsize; >>> @@ -574,8 +575,12 @@ int plain_alloc_reqbuf(struct ptlrpc_sec *sec, >>> lustre_init_msg_v2(req->rq_reqbuf, PLAIN_PACK_SEGMENTS, buflens, NULL); >>> req->rq_reqmsg = lustre_msg_buf(req->rq_reqbuf, PLAIN_PACK_MSG_OFF, 0); >>> - if (req->rq_pack_udesc) >>> - sptlrpc_pack_user_desc(req->rq_reqbuf, PLAIN_PACK_USER_OFF); >>> + if (req->rq_pack_udesc) { >>> + desc = sptlrpc_pack_user_desc(req->rq_reqbuf, >>> + PLAIN_PACK_USER_OFF); >>> + if (!desc) >>> + return desc; >>> + } >>> >>> return 0; >>> } >> _______________________________________________ >> lustre-devel mailing list >> lustre-devel at lists.lustre.org >> http://lists.lustre.org/listinfo.cgi/lustre-devel-lustre.org >> From jsimmons at infradead.org Mon May 16 18:25:57 2016 From: jsimmons at infradead.org (James Simmons) Date: Mon, 16 May 2016 19:25:57 +0100 (BST) Subject: [lustre-devel] [PATCH v2] staging/lustre/ptlrpc: Removes potential null dereference In-Reply-To: <573A0FF1.9090101@cray.com> References: <1463408271-18079-1-git-send-email-lidza.louina@oracle.com> <573A0A23.2010705@cray.com> <573A0FF1.9090101@cray.com> Message-ID: > James, > > No. You've got it backwards. 0 is false, any non-zero value is true. > > if(desc) would be equal to if (desc != 0). Ah, you are right. Didn't get much sleep last night :-( > - Patrick > > On 05/16/2016 01:16 PM, James Simmons wrote: > > > This looks wrong - You return -EINVAL from sptlrpc_pack_user_desc, but > > > then > > > the caller checks "!desc". Desc will not be null, since you've returned > > > -EINVAL. > > Actually 'if (!desc)' is equal to 'if (desc != 0). Yes it can be confusing. > > I recommend 'if (desc < 0)' instead to make it clearer what is being > > tested for. > > > > > - Patrick > > > > > > On 05/16/2016 09:17 AM, Lidza Louina wrote: > > > > The lustre_msg_buf method could return NULL. Subsequent code didn't > > > > check if it's null before using it. This patch adds two checks. > > > > > > > > Signed-off-by: Lidza Louina > > > > --- > > > > drivers/staging/lustre/lustre/ptlrpc/sec.c | 2 ++ > > > > drivers/staging/lustre/lustre/ptlrpc/sec_plain.c | 9 +++++++-- > > > > 2 files changed, 9 insertions(+), 2 deletions(-) > > > > > > > > diff --git a/drivers/staging/lustre/lustre/ptlrpc/sec.c > > > > b/drivers/staging/lustre/lustre/ptlrpc/sec.c > > > > index 187fd1d..e6fedc3 100644 > > > > --- a/drivers/staging/lustre/lustre/ptlrpc/sec.c > > > > +++ b/drivers/staging/lustre/lustre/ptlrpc/sec.c > > > > @@ -2195,6 +2195,8 @@ int sptlrpc_pack_user_desc(struct lustre_msg *msg, > > > > int > > > > offset) > > > > struct ptlrpc_user_desc *pud; > > > > > > > > pud = lustre_msg_buf(msg, offset, 0); > > > > + if (!pud) > > > > + return -EINVAL; > > > > > > > > pud->pud_uid = from_kuid(&init_user_ns, current_uid()); > > > > pud->pud_gid = from_kgid(&init_user_ns, current_gid()); > > > > diff --git a/drivers/staging/lustre/lustre/ptlrpc/sec_plain.c > > > > b/drivers/staging/lustre/lustre/ptlrpc/sec_plain.c > > > > index 37c9f4c..51741c8 100644 > > > > --- a/drivers/staging/lustre/lustre/ptlrpc/sec_plain.c > > > > +++ b/drivers/staging/lustre/lustre/ptlrpc/sec_plain.c > > > > @@ -542,6 +542,7 @@ int plain_alloc_reqbuf(struct ptlrpc_sec *sec, > > > > { > > > > __u32 buflens[PLAIN_PACK_SEGMENTS] = { 0, }; > > > > int alloc_len; > > > > + int desc; > > > > > > > > buflens[PLAIN_PACK_HDR_OFF] = sizeof(struct plain_header); > > > > buflens[PLAIN_PACK_MSG_OFF] = msgsize; > > > > @@ -574,8 +575,12 @@ int plain_alloc_reqbuf(struct ptlrpc_sec *sec, > > > > lustre_init_msg_v2(req->rq_reqbuf, PLAIN_PACK_SEGMENTS, buflens, > > > > NULL); > > > > req->rq_reqmsg = lustre_msg_buf(req->rq_reqbuf, PLAIN_PACK_MSG_OFF, > > > > 0); > > > > - if (req->rq_pack_udesc) > > > > - sptlrpc_pack_user_desc(req->rq_reqbuf, PLAIN_PACK_USER_OFF); > > > > + if (req->rq_pack_udesc) { > > > > + desc = sptlrpc_pack_user_desc(req->rq_reqbuf, > > > > + PLAIN_PACK_USER_OFF); > > > > + if (!desc) > > > > + return desc; > > > > + } > > > > > > > > return 0; > > > > } > > > _______________________________________________ > > > lustre-devel mailing list > > > lustre-devel at lists.lustre.org > > > http://lists.lustre.org/listinfo.cgi/lustre-devel-lustre.org > > > > > > From andreas.dilger at intel.com Mon May 16 18:35:50 2016 From: andreas.dilger at intel.com (Dilger, Andreas) Date: Mon, 16 May 2016 18:35:50 +0000 Subject: [lustre-devel] [PATCH v2] staging/lustre/ptlrpc: Removes potential null dereference In-Reply-To: References: <1463408271-18079-1-git-send-email-lidza.louina@oracle.com> <573A0A23.2010705@cray.com> Message-ID: <8AEE9006-6CF8-4BB7-A236-808ACE3AB302@intel.com> On 2016/05/16, 12:16, "James Simmons" wrote: > >> This looks wrong - You return -EINVAL from sptlrpc_pack_user_desc, but then >> the caller checks "!desc". Desc will not be null, since you've returned >> -EINVAL. > >Actually 'if (!desc)' is equal to 'if (desc != 0). Yes it can be confusing. Very confusing indeed. :-) "if (!desc)" actually means "if (desc == 0)"... That is the main reason why I don't like those shortcuts since they require an extra mental step by the reader to determine the logic. Having the explicit comparisons like "if (desc == 0)" or "if (desc != 0)" makes the code more clear, and doesn't make the compiled binary any slower. >I recommend 'if (desc < 0)' instead to make it clearer what is being >tested for. That would actually fix the problem. Patrick is correct that the current patch is broken, since it is either returning zero "if (!desc)" is true, or zero at the end of the function. The error is never returned. Lidza, to improve this patch further, the function should really use "rc" to hold the error return instead of "desc", since "rc" is typically used for error returns, and "desc" is normally a pointer to a bulk descriptor in this code. Also, as Oleg previously mentioned, please declare "int rc;" inside the "if (req->rq_pack_udesc)" block instead of at the top of the function, since it isn't used anywhere else. Cheers, Andreas >> - Patrick >> >> On 05/16/2016 09:17 AM, Lidza Louina wrote: >> > The lustre_msg_buf method could return NULL. Subsequent code didn't >> > check if it's null before using it. This patch adds two checks. >> > >> > Signed-off-by: Lidza Louina >> > --- >> > drivers/staging/lustre/lustre/ptlrpc/sec.c | 2 ++ >> > drivers/staging/lustre/lustre/ptlrpc/sec_plain.c | 9 +++++++-- >> > 2 files changed, 9 insertions(+), 2 deletions(-) >> > >> > diff --git a/drivers/staging/lustre/lustre/ptlrpc/sec.c >> > b/drivers/staging/lustre/lustre/ptlrpc/sec.c >> > index 187fd1d..e6fedc3 100644 >> > --- a/drivers/staging/lustre/lustre/ptlrpc/sec.c >> > +++ b/drivers/staging/lustre/lustre/ptlrpc/sec.c >> > @@ -2195,6 +2195,8 @@ int sptlrpc_pack_user_desc(struct lustre_msg *msg, int >> > offset) >> > struct ptlrpc_user_desc *pud; >> > >> > pud = lustre_msg_buf(msg, offset, 0); >> > + if (!pud) >> > + return -EINVAL; >> > >> > pud->pud_uid = from_kuid(&init_user_ns, current_uid()); >> > pud->pud_gid = from_kgid(&init_user_ns, current_gid()); >> > diff --git a/drivers/staging/lustre/lustre/ptlrpc/sec_plain.c >> > b/drivers/staging/lustre/lustre/ptlrpc/sec_plain.c >> > index 37c9f4c..51741c8 100644 >> > --- a/drivers/staging/lustre/lustre/ptlrpc/sec_plain.c >> > +++ b/drivers/staging/lustre/lustre/ptlrpc/sec_plain.c >> > @@ -542,6 +542,7 @@ int plain_alloc_reqbuf(struct ptlrpc_sec *sec, >> > { >> > __u32 buflens[PLAIN_PACK_SEGMENTS] = { 0, }; >> > int alloc_len; >> > + int desc; >> > >> > buflens[PLAIN_PACK_HDR_OFF] = sizeof(struct plain_header); >> > buflens[PLAIN_PACK_MSG_OFF] = msgsize; >> > @@ -574,8 +575,12 @@ int plain_alloc_reqbuf(struct ptlrpc_sec *sec, >> > lustre_init_msg_v2(req->rq_reqbuf, PLAIN_PACK_SEGMENTS, buflens, NULL); >> > req->rq_reqmsg = lustre_msg_buf(req->rq_reqbuf, PLAIN_PACK_MSG_OFF, 0); >> > - if (req->rq_pack_udesc) >> > - sptlrpc_pack_user_desc(req->rq_reqbuf, PLAIN_PACK_USER_OFF); >> > + if (req->rq_pack_udesc) { >> > + desc = sptlrpc_pack_user_desc(req->rq_reqbuf, >> > + PLAIN_PACK_USER_OFF); >> > + if (!desc) >> > + return desc; >> > + } >> > >> > return 0; >> > } >> >> _______________________________________________ >> lustre-devel mailing list >> lustre-devel at lists.lustre.org >> http://lists.lustre.org/listinfo.cgi/lustre-devel-lustre.org >> > Cheers, Andreas -- Andreas Dilger Lustre Principal Architect Intel High Performance Data Division From lidza.louina at oracle.com Mon May 16 18:47:28 2016 From: lidza.louina at oracle.com (Lidza Louina) Date: Mon, 16 May 2016 14:47:28 -0400 Subject: [lustre-devel] [PATCH v2] staging/lustre/ptlrpc: Removes potential null dereference In-Reply-To: <8AEE9006-6CF8-4BB7-A236-808ACE3AB302@intel.com> References: <1463408271-18079-1-git-send-email-lidza.louina@oracle.com> <573A0A23.2010705@cray.com> <8AEE9006-6CF8-4BB7-A236-808ACE3AB302@intel.com> Message-ID: <573A15C0.9070505@oracle.com> On 05/16/2016 02:35 PM, Dilger, Andreas wrote: > On 2016/05/16, 12:16, "James Simmons" wrote: > >>> This looks wrong - You return -EINVAL from sptlrpc_pack_user_desc, but then >>> the caller checks "!desc". Desc will not be null, since you've returned >>> -EINVAL. >> Actually 'if (!desc)' is equal to 'if (desc != 0). Yes it can be confusing. > Very confusing indeed. :-) "if (!desc)" actually means "if (desc == 0)"... > > That is the main reason why I don't like those shortcuts since they require > an extra mental step by the reader to determine the logic. Having the > explicit comparisons like "if (desc == 0)" or "if (desc != 0)" makes the > code more clear, and doesn't make the compiled binary any slower. > >> I recommend 'if (desc < 0)' instead to make it clearer what is being >> tested for. > That would actually fix the problem. Patrick is correct that the current > patch is broken, since it is either returning zero "if (!desc)" is true, > or zero at the end of the function. The error is never returned. > > Lidza, > to improve this patch further, the function should really use "rc" to hold > the error return instead of "desc", since "rc" is typically used for error > returns, and "desc" is normally a pointer to a bulk descriptor in this code. > > Also, as Oleg previously mentioned, please declare "int rc;" inside the > "if (req->rq_pack_udesc)" block instead of at the top of the function, > since it isn't used anywhere else. > > Cheers, Andreas Definitely, will do. I'll change desc to rc and update the if statement and send another patch now. Lidza > >>> - Patrick >>> >>> On 05/16/2016 09:17 AM, Lidza Louina wrote: >>>> The lustre_msg_buf method could return NULL. Subsequent code didn't >>>> check if it's null before using it. This patch adds two checks. >>>> >>>> Signed-off-by: Lidza Louina >>>> --- >>>> drivers/staging/lustre/lustre/ptlrpc/sec.c | 2 ++ >>>> drivers/staging/lustre/lustre/ptlrpc/sec_plain.c | 9 +++++++-- >>>> 2 files changed, 9 insertions(+), 2 deletions(-) >>>> >>>> diff --git a/drivers/staging/lustre/lustre/ptlrpc/sec.c >>>> b/drivers/staging/lustre/lustre/ptlrpc/sec.c >>>> index 187fd1d..e6fedc3 100644 >>>> --- a/drivers/staging/lustre/lustre/ptlrpc/sec.c >>>> +++ b/drivers/staging/lustre/lustre/ptlrpc/sec.c >>>> @@ -2195,6 +2195,8 @@ int sptlrpc_pack_user_desc(struct lustre_msg *msg, int >>>> offset) >>>> struct ptlrpc_user_desc *pud; >>>> >>>> pud = lustre_msg_buf(msg, offset, 0); >>>> + if (!pud) >>>> + return -EINVAL; >>>> >>>> pud->pud_uid = from_kuid(&init_user_ns, current_uid()); >>>> pud->pud_gid = from_kgid(&init_user_ns, current_gid()); >>>> diff --git a/drivers/staging/lustre/lustre/ptlrpc/sec_plain.c >>>> b/drivers/staging/lustre/lustre/ptlrpc/sec_plain.c >>>> index 37c9f4c..51741c8 100644 >>>> --- a/drivers/staging/lustre/lustre/ptlrpc/sec_plain.c >>>> +++ b/drivers/staging/lustre/lustre/ptlrpc/sec_plain.c >>>> @@ -542,6 +542,7 @@ int plain_alloc_reqbuf(struct ptlrpc_sec *sec, >>>> { >>>> __u32 buflens[PLAIN_PACK_SEGMENTS] = { 0, }; >>>> int alloc_len; >>>> + int desc; >>>> >>>> buflens[PLAIN_PACK_HDR_OFF] = sizeof(struct plain_header); >>>> buflens[PLAIN_PACK_MSG_OFF] = msgsize; >>>> @@ -574,8 +575,12 @@ int plain_alloc_reqbuf(struct ptlrpc_sec *sec, >>>> lustre_init_msg_v2(req->rq_reqbuf, PLAIN_PACK_SEGMENTS, buflens, NULL); >>>> req->rq_reqmsg = lustre_msg_buf(req->rq_reqbuf, PLAIN_PACK_MSG_OFF, 0); >>>> - if (req->rq_pack_udesc) >>>> - sptlrpc_pack_user_desc(req->rq_reqbuf, PLAIN_PACK_USER_OFF); >>>> + if (req->rq_pack_udesc) { >>>> + desc = sptlrpc_pack_user_desc(req->rq_reqbuf, >>>> + PLAIN_PACK_USER_OFF); >>>> + if (!desc) >>>> + return desc; >>>> + } >>>> >>>> return 0; >>>> } >>> _______________________________________________ >>> lustre-devel mailing list >>> lustre-devel at lists.lustre.org >>> http://lists.lustre.org/listinfo.cgi/lustre-devel-lustre.org >>> > > Cheers, Andreas From lidza.louina at oracle.com Mon May 16 18:51:42 2016 From: lidza.louina at oracle.com (Lidza Louina) Date: Mon, 16 May 2016 14:51:42 -0400 Subject: [lustre-devel] [PATCH v3] staging/lustre/ptlrpc: Removes potential null dereference Message-ID: <1463424702-5467-1-git-send-email-lidza.louina@oracle.com> The lustre_msg_buf method could return NULL. Subsequent code didn't check if it's null before using it. This patch adds two checks. Signed-off-by: Lidza Louina --- drivers/staging/lustre/lustre/ptlrpc/sec.c | 3 +++ drivers/staging/lustre/lustre/ptlrpc/sec_plain.c | 8 ++++++-- 2 files changed, 9 insertions(+), 2 deletions(-) diff --git a/drivers/staging/lustre/lustre/ptlrpc/sec.c b/drivers/staging/lustre/lustre/ptlrpc/sec.c index 187fd1d..657b41f 100644 --- a/drivers/staging/lustre/lustre/ptlrpc/sec.c +++ b/drivers/staging/lustre/lustre/ptlrpc/sec.c @@ -2196,6 +2196,9 @@ int sptlrpc_pack_user_desc(struct lustre_msg *msg, int offset) pud = lustre_msg_buf(msg, offset, 0); + if (!pud) + return -EINVAL; + pud->pud_uid = from_kuid(&init_user_ns, current_uid()); pud->pud_gid = from_kgid(&init_user_ns, current_gid()); pud->pud_fsuid = from_kuid(&init_user_ns, current_fsuid()); diff --git a/drivers/staging/lustre/lustre/ptlrpc/sec_plain.c b/drivers/staging/lustre/lustre/ptlrpc/sec_plain.c index 37c9f4c..ec8edbf 100644 --- a/drivers/staging/lustre/lustre/ptlrpc/sec_plain.c +++ b/drivers/staging/lustre/lustre/ptlrpc/sec_plain.c @@ -574,8 +574,12 @@ int plain_alloc_reqbuf(struct ptlrpc_sec *sec, lustre_init_msg_v2(req->rq_reqbuf, PLAIN_PACK_SEGMENTS, buflens, NULL); req->rq_reqmsg = lustre_msg_buf(req->rq_reqbuf, PLAIN_PACK_MSG_OFF, 0); - if (req->rq_pack_udesc) - sptlrpc_pack_user_desc(req->rq_reqbuf, PLAIN_PACK_USER_OFF); + if (req->rq_pack_udesc) { + int rc = sptlrpc_pack_user_desc(req->rq_reqbuf, + PLAIN_PACK_USER_OFF); + if (rc < 0) + return rc; + } return 0; } -- 1.9.1 From jsimmons at infradead.org Tue May 17 01:12:15 2016 From: jsimmons at infradead.org (James Simmons) Date: Tue, 17 May 2016 02:12:15 +0100 (BST) Subject: [lustre-devel] [PATCH] xattr handlers: fixup generic_listxattr In-Reply-To: References: <1462229118-13123-1-git-send-email-agruenba@redhat.com> <551ED650-CBF8-40D6-85A0-BDB3C281C247@dilger.ca> Message-ID: > generic_listxattr() is different from generic_getxattr() / > generic_setxattr() / generic_removexattr. It makes sense only for > filesystems that support a fixed set of xattrs, which means that all > handlers will have handler->name set. > > If any of the handlers has handler->prefix set instead, that handler > matches a whole set of attributes. Generic_listxattr() would have to > fill in all of those names matching that handler, but it doesn't know > which those are. > > It is common for filesystems to have their own listxattr inode > operation and still use generic_{get,set,remove}xattr. That clears things up a bit. So that leaves a few questions. First question is looking at several of the file system's implementations I noticed it contains loops such as: list_for_each_xattr(entry, base_addr) { const struct xattr_handler *handler = blah_xattr_handler(entry->e_name_index); const char *prefix; size_t prefix_len; size_t size; if (!handler || (handler->list && !handler->list(dentry))) continue; ... } Is the handler->list() test needed for a private listxattr implementation? Also I don't see anyone using handler->list() which which brings up the next question. What is the purpose of list() function? From oleg.drokin at intel.com Tue May 17 01:21:28 2016 From: oleg.drokin at intel.com (Drokin, Oleg) Date: Tue, 17 May 2016 01:21:28 +0000 Subject: [lustre-devel] [PATCH v3] staging/lustre/ptlrpc: Removes potential null dereference In-Reply-To: <1463424702-5467-1-git-send-email-lidza.louina@oracle.com> References: <1463424702-5467-1-git-send-email-lidza.louina@oracle.com> Message-ID: <9674F53E-3095-48A3-834F-4FB7105DCFA5@intel.com> On May 16, 2016, at 2:51 PM, Lidza Louina wrote: > The lustre_msg_buf method could return NULL. Subsequent code didn't > check if it's null before using it. This patch adds two checks. > > Signed-off-by: Lidza Louina Acked-by: Oleg Drokin > --- > drivers/staging/lustre/lustre/ptlrpc/sec.c | 3 +++ > drivers/staging/lustre/lustre/ptlrpc/sec_plain.c | 8 ++++++-- > 2 files changed, 9 insertions(+), 2 deletions(-) > > diff --git a/drivers/staging/lustre/lustre/ptlrpc/sec.c b/drivers/staging/lustre/lustre/ptlrpc/sec.c > index 187fd1d..657b41f 100644 > --- a/drivers/staging/lustre/lustre/ptlrpc/sec.c > +++ b/drivers/staging/lustre/lustre/ptlrpc/sec.c > @@ -2196,6 +2196,9 @@ int sptlrpc_pack_user_desc(struct lustre_msg *msg, int offset) > > pud = lustre_msg_buf(msg, offset, 0); > > + if (!pud) > + return -EINVAL; > + > pud->pud_uid = from_kuid(&init_user_ns, current_uid()); > pud->pud_gid = from_kgid(&init_user_ns, current_gid()); > pud->pud_fsuid = from_kuid(&init_user_ns, current_fsuid()); > diff --git a/drivers/staging/lustre/lustre/ptlrpc/sec_plain.c b/drivers/staging/lustre/lustre/ptlrpc/sec_plain.c > index 37c9f4c..ec8edbf 100644 > --- a/drivers/staging/lustre/lustre/ptlrpc/sec_plain.c > +++ b/drivers/staging/lustre/lustre/ptlrpc/sec_plain.c > @@ -574,8 +574,12 @@ int plain_alloc_reqbuf(struct ptlrpc_sec *sec, > lustre_init_msg_v2(req->rq_reqbuf, PLAIN_PACK_SEGMENTS, buflens, NULL); > req->rq_reqmsg = lustre_msg_buf(req->rq_reqbuf, PLAIN_PACK_MSG_OFF, 0); > > - if (req->rq_pack_udesc) > - sptlrpc_pack_user_desc(req->rq_reqbuf, PLAIN_PACK_USER_OFF); > + if (req->rq_pack_udesc) { > + int rc = sptlrpc_pack_user_desc(req->rq_reqbuf, > + PLAIN_PACK_USER_OFF); > + if (rc < 0) > + return rc; > + } > > return 0; > } > -- > 1.9.1 > > _______________________________________________ > lustre-devel mailing list > lustre-devel at lists.lustre.org > http://lists.lustre.org/listinfo.cgi/lustre-devel-lustre.org From dan.carpenter at oracle.com Tue May 17 06:53:31 2016 From: dan.carpenter at oracle.com (Dan Carpenter) Date: Tue, 17 May 2016 09:53:31 +0300 Subject: [lustre-devel] [PATCH v2] staging/lustre/ptlrpc: Removes potential null dereference In-Reply-To: <8AEE9006-6CF8-4BB7-A236-808ACE3AB302@intel.com> References: <1463408271-18079-1-git-send-email-lidza.louina@oracle.com> <573A0A23.2010705@cray.com> <8AEE9006-6CF8-4BB7-A236-808ACE3AB302@intel.com> Message-ID: <20160517065330.GA10957@mwanda> When I read the code, I just assumed desc was a pointer and it should have been: if (!desc) return NULL; For me, "if (rc) " is way more readable than "if (rc != 0) ". So readability could go either way depending on what you're used to, I suppose. It should definitely == 0 and != 0 if you are talking about the actual number zero instead of success/fail like we are here. Also it helps to use == 0 with strcmp() and friends (although half of the kernel does not know that trick yet). The other thing which I have noticed recently is that a lot of subsystems use a mix of "if (rc) " and "if (rc < 0) ". It's annoying for Smatch because say a function only returns zero but the some of the callers check for < 0 and some check for != 0. We can't know for sure that they are equivalent. regards, dan carpenter From lidza.louina at oracle.com Tue May 17 14:22:20 2016 From: lidza.louina at oracle.com (Lidza Louina) Date: Tue, 17 May 2016 10:22:20 -0400 Subject: [lustre-devel] [PATCH v2] staging/lustre/ptlrpc: Removes potential null dereference In-Reply-To: <20160517065330.GA10957@mwanda> References: <1463408271-18079-1-git-send-email-lidza.louina@oracle.com> <573A0A23.2010705@cray.com> <8AEE9006-6CF8-4BB7-A236-808ACE3AB302@intel.com> <20160517065330.GA10957@mwanda> Message-ID: <573B291C.2020609@oracle.com> On 05/17/2016 02:53 AM, Dan Carpenter wrote: > When I read the code, I just assumed desc was a pointer and it should > have been: > > if (!desc) > return NULL; > > For me, "if (rc) " is way more readable than "if (rc != 0) ". So > readability could go either way depending on what you're used to, I > suppose. > > It should definitely == 0 and != 0 if you are talking about the actual > number zero instead of success/fail like we are here. Also it helps to > use == 0 with strcmp() and friends (although half of the kernel does not > know that trick yet). > > The other thing which I have noticed recently is that a lot of > subsystems use a mix of "if (rc) " and "if (rc < 0) ". It's annoying > for Smatch because say a function only returns zero but the some of the > callers check for < 0 and some check for != 0. We can't know for sure > that they are equivalent. > > regards, > dan carpenter Hey Dan, if (rc < 0) and if (rc) pretty much translates to the same thing. It'll only return a negative error value if there are problems and 0 if it succeeds. I feel like the first way is more explicit, since negative numbers are usually used for errors. I've sent a 3rd version of the patch with (rc < 0). And I'm not sure about the way other subsystems use return values. Here it should only either be less than or equal to 0 so it makes sense to me in this circumstance. I ran smatch on my patched file `../smatch/smatch_scripts/kchecker drivers/staging/lustre/lustre/ptlrpc/sec_plain.c` and it didn't find any issues with it. Lidza From dan.carpenter at oracle.com Tue May 17 16:39:22 2016 From: dan.carpenter at oracle.com (Dan Carpenter) Date: Tue, 17 May 2016 19:39:22 +0300 Subject: [lustre-devel] [PATCH v2] staging/lustre/ptlrpc: Removes potential null dereference In-Reply-To: <573B291C.2020609@oracle.com> References: <1463408271-18079-1-git-send-email-lidza.louina@oracle.com> <573A0A23.2010705@cray.com> <8AEE9006-6CF8-4BB7-A236-808ACE3AB302@intel.com> <20160517065330.GA10957@mwanda> <573B291C.2020609@oracle.com> Message-ID: <20160517163922.GB10957@mwanda> On Tue, May 17, 2016 at 10:22:20AM -0400, Lidza Louina wrote: > if (rc < 0) and if (rc) pretty much translates to the same thing. I wasn't talking about this patch in particular; it's just something I have thinking about recently. For example, there are a lot of functions that don't initialize parameters if they return a non-zero value, but the caller checks for negatives. It's often tricky or impossible to determine that these mean the same thing just from analysing the code without making assumptions. I think "if (rc) " is more common than "if (rc < 0)". I also think it's prettier code. But mostly I wish people would do it all consistently. regards, dan carpenter From agruenba at redhat.com Tue May 17 02:03:32 2016 From: agruenba at redhat.com (Andreas Gruenbacher) Date: Tue, 17 May 2016 04:03:32 +0200 Subject: [lustre-devel] [PATCH] xattr handlers: fixup generic_listxattr In-Reply-To: References: <1462229118-13123-1-git-send-email-agruenba@redhat.com> <551ED650-CBF8-40D6-85A0-BDB3C281C247@dilger.ca> Message-ID: On Tue, May 17, 2016 at 3:12 AM, James Simmons wrote: >> generic_listxattr() is different from generic_getxattr() / >> generic_setxattr() / generic_removexattr. It makes sense only for >> filesystems that support a fixed set of xattrs, which means that all >> handlers will have handler->name set. >> >> If any of the handlers has handler->prefix set instead, that handler >> matches a whole set of attributes. Generic_listxattr() would have to >> fill in all of those names matching that handler, but it doesn't know >> which those are. >> >> It is common for filesystems to have their own listxattr inode >> operation and still use generic_{get,set,remove}xattr. > > That clears things up a bit. So that leaves a few questions. First > question is looking at several of the file system's implementations > I noticed it contains loops such as: > > list_for_each_xattr(entry, base_addr) { > const struct xattr_handler *handler = > blah_xattr_handler(entry->e_name_index); > const char *prefix; > size_t prefix_len; > size_t size; > > if (!handler || (handler->list && !handler->list(dentry))) > continue; > > ... > } > > Is the handler->list() test needed for a private listxattr implementation? > Also I don't see anyone using handler->list() which which brings up the > next question. What is the purpose of list() function? Trusted xattrs are meant to only be visible to tasks with the CAP_SYS_ADMIN capability. For deciding which names to include in its result, listxattr implementations can use the list function, or any other suitable mechanism. On filesystems that store the complete xattr names as strings, a strncmp check for a "trusted." prefix together with capable(CAP_SYS_ADMIN) would do, for example. Andreas From linux at sciencehorizons.net Tue May 17 08:52:22 2016 From: linux at sciencehorizons.net (George Spelvin) Date: 17 May 2016 04:52:22 -0400 Subject: [lustre-devel] Lustre use of hash_long() & cfs_hash_u32_hash() In-Reply-To: Message-ID: <20160517085222.10484.qmail@ns.sciencehorizons.net> > If we change to using hash_64() or hash_32() instead this > will change the behavior we currently have. There is hash_long for exactly this purpose. The problem was that Lustre was using hash_long() on values that are 64 bits even on 32-bit platforms. I wasn't trying to get rid of all calls to hash_long, just the buggy ones. hash_32: Hash 32 bits hash_64: Hash 64 bits hash_long: Alias for one of the preceding, depending on BITS_PER_LONG. The thing is to use the correct one for the argument. > Now cfs_hash_u64_hash() is defined as > > static inline unsigned > cfs_hash_u64_hash(const __u64 key, unsigned mask) > { > return ((unsigned)(key * CFS_GOLDEN_RATIO_PRIME_64) & mask); > } > > which uses a mask directly. That mask is generated in cfs_hash_id() with : > > unsigned int index = cfs_hash_id(hs, key, (1U << bits) - 1); > > So the mask uses the lower bits. We can't change this behavior > since other hp_ops->hs_hash functions that don't use > cfs_hash_u[64|32]_hash() at all treat this as a mask. This is a bit of a problem. My hash changes won't make it any worse, but it's a *current* problem that you probably want to fix. The thing is, a multiply has lower bits of the input affect higher bits of the output, but there is no propagation at all in the other direction. (See http://burtleburtle.net/bob/hash/integer.html for a discussion of "half-avalanche".) On other words, for a mask which is a power of 2 minus 1, (hash_64(x) & mask) == (hash_64(x & mask) & mask). (This is NOT true if the mask has trailing zero bits.) Given that a typical mask has far less than 64 bits set, you can see how this will lead to huge numbers of collisions. There are a few possible solutions depending on the typical size of mask: - Since you're returning only 32 bits at most, shift down 32 bits before masking. This still has you ignoring some high-order bits, but it improves things a lot. (One of my pending patches to hash_64() does exactly this.) - bswap() the result. May still ignore input bits if the mask is less than 8 bits, but it's getting decent. - (Less efficient on some embedded platforms, but none likely to run Lustre in the first place): Use the low bits of the *high* word of the product. This has high bits of the hash value unaffected by low bits of the argument (except by carry propagation, which is unlikely to propagate many bits), but gets you what you need. - Use a totally different hash function like jhash_2words() that achieves more mixing (but is slower). - Change the Lustre code to use the high bits everywhere. One alternative to jhash_2words() is Thomas Wang's 64-to-32 bit hash from https://web.archive.org/web/20071223173210/http://www.concentric.net/~Ttwang/tech/inthash.htm Translated from Java to C, that's: u32 hash6432shift(u64 key) { key = ~key + (key << 18); // key = (key << 18) - key - 1; key ^= key >> 31; key *= 21; // key = key + (key << 2) + (key << 4); key ^= key >> 11; key += key << 6; key ^= key >> 22; return (u32) key; } > I can work the code so the mask can be turned into a bit offset > but the mask will still be different. Will this difference > break things? I'm not sure I understand the question. As long as the hashing is always done the same way, the details of *how* the hashing is done are opaque to the rest of the code. The problem arises if someone assumes a mathematical relationship between hash_64(x, bits) and hash_64(x, bits+1), such as when growing or shrinking a dynamic hash table. If the calling code "knows" that hash64(x, mask) can be ANDed with mask/2 to produce the same number as hash64(x, mask/2) then things will make a mess. From simmonsja at ornl.gov Thu May 19 19:54:47 2016 From: simmonsja at ornl.gov (Simmons, James A.) Date: Thu, 19 May 2016 19:54:47 +0000 Subject: [lustre-devel] LWN article on kernel stats reveal upstream client progress. Message-ID: <11629f7822674b1d872d078bcb422717@EXCHCS32.ornl.gov> With all the work going into the upstream lustre client Oleg and I showed up in the linux kernel developments stats reported by LWN. https://lwn.net/Articles/686393 As the article pointed out Oleg showed up 3rd for the number of patches submitted as well as 5th for line changes. I showed up 16th. We both also showed up in the most active reviewers list. Note the changes in kernel 4.6 where the changes mostly done during the 4.5 development cycle. The staging tree work is merged to the 4.X-rc1 branch so any work done after that remains in the staging tree until the next merge window. The 4.6 version of the kernel contains the same LNet stack as the Lustre 2.8.0 release and Oleg removed most of the code style violations in the 4.6 version. When linux 4.7 is released it will contain the CLIO simplification work as well as new LNet updates which brings it into line with the latest LNet stack. It will be interesting to see what the new stats will be. -------------- next part -------------- An HTML attachment was scrubbed... URL: From doug.s.oucharek at intel.com Thu May 19 21:31:33 2016 From: doug.s.oucharek at intel.com (Oucharek, Doug S) Date: Thu, 19 May 2016 21:31:33 +0000 Subject: [lustre-devel] LWN article on kernel stats reveal upstream client progress. In-Reply-To: <11629f7822674b1d872d078bcb422717@EXCHCS32.ornl.gov> References: <11629f7822674b1d872d078bcb422717@EXCHCS32.ornl.gov> Message-ID: <924F1865-65B5-4DBE-85C3-982D0BAF99D7@intel.com> James, How close are we to getting LNet out of staging and into mainline? If we are getting close, should we consider “separating” LNet from Lustre making it a standalone item? Considering how we have to propagate all LNet fixes to every active repo/version, I get the feeling that LNet should be separate anyway to make updating just LNet in the field easier. Doug On May 19, 2016, at 12:54 PM, Simmons, James A. > wrote: With all the work going into the upstream lustre client Oleg and I showed up in the linux kernel developments stats reported by LWN. https://lwn.net/Articles/686393 As the article pointed out Oleg showed up 3rd for the number of patches submitted as well as 5th for line changes. I showed up 16th. We both also showed up in the most active reviewers list. Note the changes in kernel 4.6 where the changes mostly done during the 4.5 development cycle. The staging tree work is merged to the 4.X-rc1 branch so any work done after that remains in the staging tree until the next merge window. The 4.6 version of the kernel contains the same LNet stack as the Lustre 2.8.0 release and Oleg removed most of the code style violations in the 4.6 version. When linux 4.7 is released it will contain the CLIO simplification work as well as new LNet updates which brings it into line with the latest LNet stack. It will be interesting to see what the new stats will be. _______________________________________________ lustre-devel mailing list lustre-devel at lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-devel-lustre.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From andreas.dilger at intel.com Sat May 21 05:51:09 2016 From: andreas.dilger at intel.com (Dilger, Andreas) Date: Sat, 21 May 2016 05:51:09 +0000 Subject: [lustre-devel] [lustre-discuss] more on lustre striping In-Reply-To: <1ef5a267-334c-be0d-13f4-c0fab917d1bf@iodoctors.com> References: <1ef5a267-334c-be0d-13f4-c0fab917d1bf@iodoctors.com> Message-ID: <38EA9F32-3869-43BA-B215-EB2E5BA62FD5@intel.com> This is probably getting to be more of a topic for lustre-devel. There currently isn't any way to do what you ask, since (IIRC) it will cause an error for apps that try to write to the files before the layout is set. What you could do is to create an LD_PRELOAD library to intercept the open() calls and set O_LOV_DELAY_CREATE and set the layout explicitly for each file. This might be a win if each file needs a different layout, but since it uses two RPCs per file it would be slower than using the default layout. Cheers, Andreas On May 18, 2016, at 16:46, John Bauer > wrote: Since today's topic seems to be Lustre striping, I will revisit a previous line of questions I had. Andreas had put me on to O_LOV_DELAY_CREATE which I have been experimenting with. My question is : Is there a way to flag a directory with O_LOV_DELAY_CREATE so that a file created in that directory will be created with O_LOV_DELAY_CREATE also. Much like a file can inherit a directory's stripe count and stripe size, it would be convenient if a file could also inherit O_LOV_DELAY_CREATE? That way, for open()s that I can not intercept ( and thus can not set O_LOV_DELAY_CREATE in oflags) , such as those issued by fopen(), I can then get the fd with fileno() and set the striping with ioctl(fd, LL_IOC_LOV_SETSTRIPE, lum). Thanks John -- I/O Doctors, LLC 507-766-0378 bauerj at iodoctors.com _______________________________________________ lustre-discuss mailing list lustre-discuss at lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From oleg.drokin at intel.com Sat May 21 20:08:41 2016 From: oleg.drokin at intel.com (Drokin, Oleg) Date: Sat, 21 May 2016 20:08:41 +0000 Subject: [lustre-devel] [lustre-discuss] more on lustre striping In-Reply-To: References: <1ef5a267-334c-be0d-13f4-c0fab917d1bf@iodoctors.com> <38EA9F32-3869-43BA-B215-EB2E5BA62FD5@intel.com> Message-ID: <01EE2B9D-2359-49B5-A50F-E7C2D97E6696@intel.com> The thing is, when you open a file with no layout (the one you cteate with P_LOB_DELAY_CREATE) for write the next time - the default layout is created just the same as it would have been created on the first open. So if you want custom layouts - you do need to insert setstripe call between the creation and actual open for write. On the other hand if you open with O_LOV_DELAY_CREATE and then try to write into that fd - you will get a failure. On May 21, 2016, at 4:01 PM, John Bauer wrote: > Andreas, > > Thanks for the reply. For what it's worth, extending a file that does not have layout set does work. > > % rm -f file.dat > % ./no_stripe.exe file.dat > fd=3 > % lfs getstripe file.dat > file.dat has no stripe info > % date >> file.dat > % lfs getstripe file.dat > file.dat > lmm_stripe_count: 1 > lmm_stripe_size: 1048576 > lmm_pattern: 1 > lmm_layout_gen: 0 > lmm_stripe_offset: 21 > obdidx objid objid group > 21 6143298 0x5dbd42 0 > > % > The LD_PRELOAD is exactly what I am doing in my I/O library. Unfortunately, one can not intercept the open() that results from a call to fopen(). That open is hard linked to the open in libc and not satisfied by the runtime linker. This is what is driving this topic for me. I can not conveniently set the striping for a file opened with fopen() and other functions where the open is called from inside libc. I used to believe that not too many application use stdio for heavy I/O, but I have been come across several recently. > > John > > On 5/21/2016 12:51 AM, Dilger, Andreas wrote: >> This is probably getting to be more of a topic for lustre-devel. >> >> There currently isn't any way to do what you ask, since (IIRC) it will cause an error for apps that try to write to the files before the layout is set. >> >> What you could do is to create an LD_PRELOAD library to intercept the open() calls and set O_LOV_DELAY_CREATE and set the layout explicitly for each file. This might be a win if each file needs a different layout, but since it uses two RPCs per file it would be slower than using the default layout. >> >> Cheers, Andreas >> >> On May 18, 2016, at 16:46, John Bauer wrote: >> >>> Since today's topic seems to be Lustre striping, I will revisit a previous line of questions I had. >>> >>> Andreas had put me on to O_LOV_DELAY_CREATE which I have been experimenting with. My question is : Is there a way to flag a directory with O_LOV_DELAY_CREATE so that a file created in that directory will be created with O_LOV_DELAY_CREATE also. Much like a file can inherit a directory's stripe count and stripe size, it would be convenient if a file could also inherit O_LOV_DELAY_CREATE? That way, for open()s that I can not intercept ( and thus can not set O_LOV_DELAY_CREATE in oflags) , such as those issued by fopen(), I can then get the fd with fileno() and set the striping with ioctl(fd, LL_IOC_LOV_SETSTRIPE, lum). >>> >>> Thanks >>> >>> John >>> -- >>> I/O Doctors, LLC >>> 507-766-0378 >>> >>> bauerj at iodoctors.com >>> _______________________________________________ >>> lustre-discuss mailing list >>> lustre-discuss at lists.lustre.org >>> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org > > -- > I/O Doctors, LLC > 507-766-0378 > > bauerj at iodoctors.com > _______________________________________________ > lustre-discuss mailing list > lustre-discuss at lists.lustre.org > http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org From oleg.drokin at intel.com Sun May 22 00:33:59 2016 From: oleg.drokin at intel.com (Drokin, Oleg) Date: Sun, 22 May 2016 00:33:59 +0000 Subject: [lustre-devel] [lustre-discuss] more on lustre striping In-Reply-To: <44e2353d-3bef-ddd7-f15e-ad4bfdda040f@iodoctors.com> References: <1ef5a267-334c-be0d-13f4-c0fab917d1bf@iodoctors.com> <38EA9F32-3869-43BA-B215-EB2E5BA62FD5@intel.com> <01EE2B9D-2359-49B5-A50F-E7C2D97E6696@intel.com> <44e2353d-3bef-ddd7-f15e-ad4bfdda040f@iodoctors.com> Message-ID: btw I find it strange that you cannot intercept fopen (and in fact intercepting every library call like that is counterproductive). We used to have this "liblustre" library, that you an LD_PRELOAD into your application and it would work with Lustre even if you are not root and if Lustre is not mounted on that node (and in fact even if the node is not Linux at all). That had no problems at all to intercept all sorts of opens by intercepting syscalls. I wonder if you can intercept something deeper like sys_open or something like that? Perhaps checkout lustre 1.8 sources (or even 2.1) and see how we did it back there? On May 21, 2016, at 4:25 PM, John Bauer wrote: > Oleg > > So in my simple test, the second open of the file caused the layout to be created. Indeed, a write to the original fd did fail. > That complicates things considerably. > > Disregard the entire topic. > > Thanks > > John > > > On 5/21/2016 3:08 PM, Drokin, Oleg wrote: >> The thing is, when you open a file with no layout (the one you cteate with P_LOB_DELAY_CREATE) for write the next time - >> the default layout is created just the same as it would have been created on the first open. >> So if you want custom layouts - you do need to insert setstripe call between the creation and actual open for write. >> >> On the other hand if you open with O_LOV_DELAY_CREATE and then try to write into that fd - you will get a failure. >> >> >> On May 21, 2016, at 4:01 PM, John Bauer wrote: >> >> >>> Andreas, >>> >>> Thanks for the reply. For what it's worth, extending a file that does not have layout set does work. >>> >>> % rm -f file.dat >>> % ./no_stripe.exe file.dat >>> fd=3 >>> % lfs getstripe file.dat >>> file.dat has no stripe info >>> % date >> file.dat >>> % lfs getstripe file.dat >>> file.dat >>> lmm_stripe_count: 1 >>> lmm_stripe_size: 1048576 >>> lmm_pattern: 1 >>> lmm_layout_gen: 0 >>> lmm_stripe_offset: 21 >>> obdidx objid objid group >>> 21 6143298 0x5dbd42 0 >>> >>> % >>> The LD_PRELOAD is exactly what I am doing in my I/O library. Unfortunately, one can not intercept the open() that results from a call to fopen(). That open is hard linked to the open in libc and not satisfied by the runtime linker. This is what is driving this topic for me. I can not conveniently set the striping for a file opened with fopen() and other functions where the open is called from inside libc. I used to believe that not too many application use stdio for heavy I/O, but I have been come across several recently. >>> >>> John >>> >>> On 5/21/2016 12:51 AM, Dilger, Andreas wrote: >>> >>>> This is probably getting to be more of a topic for lustre-devel. >>>> >>>> There currently isn't any way to do what you ask, since (IIRC) it will cause an error for apps that try to write to the files before the layout is set. >>>> >>>> What you could do is to create an LD_PRELOAD library to intercept the open() calls and set O_LOV_DELAY_CREATE and set the layout explicitly for each file. This might be a win if each file needs a different layout, but since it uses two RPCs per file it would be slower than using the default layout. >>>> >>>> Cheers, Andreas >>>> >>>> On May 18, 2016, at 16:46, John Bauer >>>> >>>> wrote: >>>> >>>> >>>>> Since today's topic seems to be Lustre striping, I will revisit a previous line of questions I had. >>>>> >>>>> Andreas had put me on to O_LOV_DELAY_CREATE which I have been experimenting with. My question is : Is there a way to flag a directory with O_LOV_DELAY_CREATE so that a file created in that directory will be created with O_LOV_DELAY_CREATE also. Much like a file can inherit a directory's stripe count and stripe size, it would be convenient if a file could also inherit O_LOV_DELAY_CREATE? That way, for open()s that I can not intercept ( and thus can not set O_LOV_DELAY_CREATE in oflags) , such as those issued by fopen(), I can then get the fd with fileno() and set the striping with ioctl(fd, LL_IOC_LOV_SETSTRIPE, lum). >>>>> >>>>> Thanks >>>>> >>>>> John >>>>> -- >>>>> I/O Doctors, LLC >>>>> 507-766-0378 >>>>> >>>>> >>>>> bauerj at iodoctors.com >>>>> >>>>> _______________________________________________ >>>>> lustre-discuss mailing list >>>>> >>>>> lustre-discuss at lists.lustre.org >>>>> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org >>> -- >>> I/O Doctors, LLC >>> 507-766-0378 >>> >>> >>> bauerj at iodoctors.com >>> >>> _______________________________________________ >>> lustre-discuss mailing list >>> >>> lustre-discuss at lists.lustre.org >>> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org > > -- > I/O Doctors, LLC > 507-766-0378 > > bauerj at iodoctors.com From oleg.drokin at intel.com Sun May 22 02:49:19 2016 From: oleg.drokin at intel.com (Drokin, Oleg) Date: Sun, 22 May 2016 02:49:19 +0000 Subject: [lustre-devel] [lustre-discuss] more on lustre striping In-Reply-To: <9ebe9f4c-d5a2-4c93-4f00-49029850cbbe@iodoctors.com> References: <1ef5a267-334c-be0d-13f4-c0fab917d1bf@iodoctors.com> <38EA9F32-3869-43BA-B215-EB2E5BA62FD5@intel.com> <01EE2B9D-2359-49B5-A50F-E7C2D97E6696@intel.com> <44e2353d-3bef-ddd7-f15e-ad4bfdda040f@iodoctors.com> <9ebe9f4c-d5a2-4c93-4f00-49029850cbbe@iodoctors.com> Message-ID: <2C5AD38E-7DB8-4D49-A6FA-46066E2416E7@intel.com> $ nm -g lustre/liblustre/liblustre.so | grep open 0000000000237820 W __open 0000000000237820 W __open64 000000000023ba30 W __opendir 00000000002376c0 T _sysio_open U fopen@@GLIBC_2.2.5 0000000000237820 T open 0000000000237820 W open64 000000000023ba30 T opendir These are the open symbols we have in the .so it most certainly intercepts the open syscall no matter if it comes via open or fopen. so I suspect you just need to catch __open* stuff and this will catch both open and fopen for you too. At least quick googling around seems to confirm this. All intercepting was done via libsysio (since it reimplemented VFS in userspace), so if you need more info, perhaps you can consult with Lee Ward who is the main author of it. I know he used to read this list too so he might decide to chime in. On May 21, 2016, at 9:56 PM, John Bauer wrote: > Oleg > > I can intercept the fopen(), but that does me no good as I can't set the O_LOV_DELAY_CREATE bit. What I can not intercept is the open() downstream of fopen(). If one examines the symbols in libc you will see there are no unsatisfied externals relating to open, which means there is nothing for the runtime linker to find concerning open's. I will have a look at the Lustre 1.8 source, but I seriously doubt that the open beneath fopen() was intercepted with LD_PRELOAD. I would love to find a way to do that. I could throw away a lot of code. Thanks, John > % nm -g /lib64/libc.so.6 | grep open > 0000000000033d70 T catopen > 00000000003bfb80 B _dl_open_hook > 00000000000b9a60 W fdopendir > 000000000006b140 T fdopen@@GLIBC_2.2.5 > 00000000000755c0 T fmemopen > 000000000006ba00 W fopen64 > 000000000006bb60 T fopencookie@@GLIBC_2.2.5 > 000000000006ba00 T fopen@@GLIBC_2.2.5 > 00000000000736f0 T freopen > 0000000000074b50 T freopen64 > 00000000000ead40 T fts_open > 0000000000022220 T iconv_open > 000000000006b140 T _IO_fdopen@@GLIBC_2.2.5 > 0000000000077220 T _IO_file_fopen@@GLIBC_2.2.5 > 0000000000077170 T _IO_file_open > 000000000006ba00 T _IO_fopen@@GLIBC_2.2.5 > 000000000006d1d0 T _IO_popen@@GLIBC_2.2.5 > 000000000006cee0 T _IO_proc_open@@GLIBC_2.2.5 > 0000000000130b20 T __libc_dlopen_mode > 00000000000e7840 W open > 00000000000e7840 W __open > 00000000000ec690 T __open_2 > 00000000000e7840 W open64 > 00000000000e7840 W __open64 > 00000000000ec6b0 T __open64_2 > 00000000000e78d0 W openat > 00000000000e79b0 T __openat_2 > 00000000000e78d0 W openat64 > 00000000000e79b0 W __openat64_2 > 00000000000f6e00 T open_by_handle_at > 00000000000340b0 T __open_catalog > 00000000000b9510 W opendir > 00000000000f0850 T openlog > 0000000000073e90 T open_memstream > 00000000000731b0 T open_wmemstream > 000000000006d1d0 T popen@@GLIBC_2.2.5 > 000000000012fbd0 W posix_openpt > 00000000000e6460 T posix_spawn_file_actions_addopen > % > John > > On 5/21/2016 7:33 PM, Drokin, Oleg wrote: >> btw I find it strange that you cannot intercept fopen (and in fact intercepting every library call like that is counterproductive). >> >> We used to have this "liblustre" library, that you an LD_PRELOAD into your application and it would work with Lustre even if you are not root and if Lustre is not mounted on that node >> (and in fact even if the node is not Linux at all). That had no problems at all to intercept all sorts of opens by intercepting syscalls. >> I wonder if you can intercept something deeper like sys_open or something like that? >> Perhaps checkout lustre 1.8 sources (or even 2.1) and see how we did it back there? >> >> On May 21, 2016, at 4:25 PM, John Bauer wrote: >> >> >>> Oleg >>> >>> So in my simple test, the second open of the file caused the layout to be created. Indeed, a write to the original fd did fail. >>> That complicates things considerably. >>> >>> Disregard the entire topic. >>> >>> Thanks >>> >>> John >>> >>> >>> On 5/21/2016 3:08 PM, Drokin, Oleg wrote: >>> >>>> The thing is, when you open a file with no layout (the one you cteate with P_LOB_DELAY_CREATE) for write the next time - >>>> the default layout is created just the same as it would have been created on the first open. >>>> So if you want custom layouts - you do need to insert setstripe call between the creation and actual open for write. >>>> >>>> On the other hand if you open with O_LOV_DELAY_CREATE and then try to write into that fd - you will get a failure. >>>> >>>> >>>> On May 21, 2016, at 4:01 PM, John Bauer wrote: >>>> >>>> >>>> >>>>> Andreas, >>>>> >>>>> Thanks for the reply. For what it's worth, extending a file that does not have layout set does work. >>>>> >>>>> % rm -f file.dat >>>>> % ./no_stripe.exe file.dat >>>>> fd=3 >>>>> % lfs getstripe file.dat >>>>> file.dat has no stripe info >>>>> % date >> file.dat >>>>> % lfs getstripe file.dat >>>>> file.dat >>>>> lmm_stripe_count: 1 >>>>> lmm_stripe_size: 1048576 >>>>> lmm_pattern: 1 >>>>> lmm_layout_gen: 0 >>>>> lmm_stripe_offset: 21 >>>>> obdidx objid objid group >>>>> 21 6143298 0x5dbd42 0 >>>>> >>>>> % >>>>> The LD_PRELOAD is exactly what I am doing in my I/O library. Unfortunately, one can not intercept the open() that results from a call to fopen(). That open is hard linked to the open in libc and not satisfied by the runtime linker. This is what is driving this topic for me. I can not conveniently set the striping for a file opened with fopen() and other functions where the open is called from inside libc. I used to believe that not too many application use stdio for heavy I/O, but I have been come across several recently. >>>>> >>>>> John >>>>> >>>>> On 5/21/2016 12:51 AM, Dilger, Andreas wrote: >>>>> >>>>> >>>>>> This is probably getting to be more of a topic for lustre-devel. >>>>>> >>>>>> There currently isn't any way to do what you ask, since (IIRC) it will cause an error for apps that try to write to the files before the layout is set. >>>>>> >>>>>> What you could do is to create an LD_PRELOAD library to intercept the open() calls and set O_LOV_DELAY_CREATE and set the layout explicitly for each file. This might be a win if each file needs a different layout, but since it uses two RPCs per file it would be slower than using the default layout. >>>>>> >>>>>> Cheers, Andreas >>>>>> >>>>>> On May 18, 2016, at 16:46, John Bauer >>>>>> >>>>>> >>>>>> >>>>>> wrote: >>>>>> >>>>>> >>>>>> >>>>>>> Since today's topic seems to be Lustre striping, I will revisit a previous line of questions I had. >>>>>>> >>>>>>> Andreas had put me on to O_LOV_DELAY_CREATE which I have been experimenting with. My question is : Is there a way to flag a directory with O_LOV_DELAY_CREATE so that a file created in that directory will be created with O_LOV_DELAY_CREATE also. Much like a file can inherit a directory's stripe count and stripe size, it would be convenient if a file could also inherit O_LOV_DELAY_CREATE? That way, for open()s that I can not intercept ( and thus can not set O_LOV_DELAY_CREATE in oflags) , such as those issued by fopen(), I can then get the fd with fileno() and set the striping with ioctl(fd, LL_IOC_LOV_SETSTRIPE, lum). >>>>>>> >>>>>>> Thanks >>>>>>> >>>>>>> John >>>>>>> -- >>>>>>> I/O Doctors, LLC >>>>>>> 507-766-0378 >>>>>>> >>>>>>> >>>>>>> >>>>>>> bauerj at iodoctors.com >>>>>>> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> lustre-discuss mailing list >>>>>>> >>>>>>> >>>>>>> lustre-discuss at lists.lustre.org >>>>>>> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org >>>>> -- >>>>> I/O Doctors, LLC >>>>> 507-766-0378 >>>>> >>>>> >>>>> >>>>> bauerj at iodoctors.com >>>>> >>>>> >>>>> _______________________________________________ >>>>> lustre-discuss mailing list >>>>> >>>>> >>>>> lustre-discuss at lists.lustre.org >>>>> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org >>> -- >>> I/O Doctors, LLC >>> 507-766-0378 >>> >>> >>> bauerj at iodoctors.com > > -- > I/O Doctors, LLC > 507-766-0378 > > bauerj at iodoctors.com From gregkh at linuxfoundation.org Sun May 22 22:25:38 2016 From: gregkh at linuxfoundation.org (Greg Kroah-Hartman) Date: Sun, 22 May 2016 15:25:38 -0700 Subject: [lustre-devel] [PATCH] staging: lustre: llite: kzalloc/copy_to_user to memdup_user In-Reply-To: <1463953771-16942-1-git-send-email-me@tobin.cc> References: <1463953771-16942-1-git-send-email-me@tobin.cc> Message-ID: <20160522222538.GA17573@kroah.com> On Mon, May 23, 2016 at 07:49:29AM +1000, Tobin C Harding wrote: > kzalloc call followed by copy_to_user can be replaced by call to memdup_user. > > Signed-off-by: Tobin C Harding Why did you send this twice? > --- > drivers/gpu/drm/gma500/gma_display.c | 2 +- > drivers/staging/lustre/lustre/llite/dir.c | 22 +++++++--------------- > 2 files changed, 8 insertions(+), 16 deletions(-) > > diff --git a/drivers/gpu/drm/gma500/gma_display.c b/drivers/gpu/drm/gma500/gma_display.c > index 5bf765d..8a1fb25 100644 > --- a/drivers/gpu/drm/gma500/gma_display.c > +++ b/drivers/gpu/drm/gma500/gma_display.c > @@ -285,7 +285,7 @@ void gma_crtc_dpms(struct drm_crtc *crtc, int mode) > > /* Wait for vblank for the disable to take effect */ > gma_wait_for_vblank(dev); > - > +l What? What does this have to do with lustre? It's kind of obvious you didn't build this :( Please be more careful in the future. greg k-h From agruenba at redhat.com Mon May 23 13:09:49 2016 From: agruenba at redhat.com (Andreas Gruenbacher) Date: Mon, 23 May 2016 15:09:49 +0200 Subject: [lustre-devel] [PATCH] posix acls: Move namespace conversion into filesystem / xattr handlers Message-ID: <1464008989-3812-1-git-send-email-agruenba@redhat.com> Currently, getxattr() and setxattr() check for the xattr names "system.posix_acl_{access,default}" and perform in-place UID / GID namespace mappings in the xattr values. Filesystems then again check for the same xattr names to handle those attributes, almost always using the standard posix_acl_{access,default}_xattr_handler handlers. This is unnecessary overhead; move the namespace conversion into the xattr handlers instead. For filesystems that use the POSIX ACL xattr handlers, no change is required. Filesystems that don't use those handlers (cifs and lustre) need to take care of the namespace mapping themselves now. The only user left of the posix_acl_fix_xattr_{from,to}_user helpers is lustre, so this patch moves them into lustre. Signed-off-by: Andreas Gruenbacher --- I'm reasonably confident about the core and cifs changes in this patch. The lustre code is pretty weird though, so could I please get a careful review on the changes there? Thanks, Andreas drivers/staging/lustre/lustre/llite/Makefile | 1 + .../staging/lustre/lustre/llite/llite_internal.h | 3 ++ drivers/staging/lustre/lustre/llite/posix_acl.c | 62 ++++++++++++++++++++++ drivers/staging/lustre/lustre/llite/xattr.c | 8 ++- fs/cifs/cifssmb.c | 41 ++++++++++---- fs/posix_acl.c | 62 +--------------------- fs/xattr.c | 7 --- include/linux/posix_acl_xattr.h | 12 ----- 8 files changed, 107 insertions(+), 89 deletions(-) create mode 100644 drivers/staging/lustre/lustre/llite/posix_acl.c diff --git a/drivers/staging/lustre/lustre/llite/Makefile b/drivers/staging/lustre/lustre/llite/Makefile index 2ce10ff..67125f8 100644 --- a/drivers/staging/lustre/lustre/llite/Makefile +++ b/drivers/staging/lustre/lustre/llite/Makefile @@ -7,5 +7,6 @@ lustre-y := dcache.o dir.o file.o llite_close.o llite_lib.o llite_nfs.o \ glimpse.o lcommon_cl.o lcommon_misc.o \ vvp_dev.o vvp_page.o vvp_lock.o vvp_io.o vvp_object.o vvp_req.o \ lproc_llite.o +lustre-$(CONFIG_FS_POSIX_ACL) += posix_acl.o llite_lloop-y := lloop.o diff --git a/drivers/staging/lustre/lustre/llite/llite_internal.h b/drivers/staging/lustre/lustre/llite/llite_internal.h index ce1f949..d454dfb 100644 --- a/drivers/staging/lustre/lustre/llite/llite_internal.h +++ b/drivers/staging/lustre/lustre/llite/llite_internal.h @@ -1402,4 +1402,7 @@ int cl_local_size(struct inode *inode); __u64 cl_fid_build_ino(const struct lu_fid *fid, int api32); __u32 cl_fid_build_gen(const struct lu_fid *fid); +void posix_acl_fix_xattr_from_user(void *value, size_t size); +void posix_acl_fix_xattr_to_user(void *value, size_t size); + #endif /* LLITE_INTERNAL_H */ diff --git a/drivers/staging/lustre/lustre/llite/posix_acl.c b/drivers/staging/lustre/lustre/llite/posix_acl.c new file mode 100644 index 0000000..4fabd0f --- /dev/null +++ b/drivers/staging/lustre/lustre/llite/posix_acl.c @@ -0,0 +1,62 @@ +#include +#include +#include +#include + +/* + * Fix up the uids and gids in posix acl extended attributes in place. + */ +static void posix_acl_fix_xattr_userns( + struct user_namespace *to, struct user_namespace *from, + void *value, size_t size) +{ + posix_acl_xattr_header *header = (posix_acl_xattr_header *)value; + posix_acl_xattr_entry *entry = (posix_acl_xattr_entry *)(header+1), *end; + int count; + kuid_t uid; + kgid_t gid; + + if (!value) + return; + if (size < sizeof(posix_acl_xattr_header)) + return; + if (header->a_version != cpu_to_le32(POSIX_ACL_XATTR_VERSION)) + return; + + count = posix_acl_xattr_count(size); + if (count < 0) + return; + if (count == 0) + return; + + for (end = entry + count; entry != end; entry++) { + switch(le16_to_cpu(entry->e_tag)) { + case ACL_USER: + uid = make_kuid(from, le32_to_cpu(entry->e_id)); + entry->e_id = cpu_to_le32(from_kuid(to, uid)); + break; + case ACL_GROUP: + gid = make_kgid(from, le32_to_cpu(entry->e_id)); + entry->e_id = cpu_to_le32(from_kgid(to, gid)); + break; + default: + break; + } + } +} + +void posix_acl_fix_xattr_from_user(void *value, size_t size) +{ + struct user_namespace *user_ns = current_user_ns(); + if (user_ns == &init_user_ns) + return; + posix_acl_fix_xattr_userns(&init_user_ns, user_ns, value, size); +} + +void posix_acl_fix_xattr_to_user(void *value, size_t size) +{ + struct user_namespace *user_ns = current_user_ns(); + if (user_ns == &init_user_ns) + return; + posix_acl_fix_xattr_userns(user_ns, &init_user_ns, value, size); +} diff --git a/drivers/staging/lustre/lustre/llite/xattr.c b/drivers/staging/lustre/lustre/llite/xattr.c index ed4de04..c721b44 100644 --- a/drivers/staging/lustre/lustre/llite/xattr.c +++ b/drivers/staging/lustre/lustre/llite/xattr.c @@ -144,6 +144,9 @@ int ll_setxattr_common(struct inode *inode, const char *name, return -EOPNOTSUPP; #ifdef CONFIG_FS_POSIX_ACL + if (xattr_type == XATTR_ACL_ACCESS_T || + xattr_type == XATTR_ACL_DEFAULT_T) + posix_acl_fix_xattr_from_user((void *)value, size); if (sbi->ll_flags & LL_SBI_RMT_CLIENT && (xattr_type == XATTR_ACL_ACCESS_T || xattr_type == XATTR_ACL_DEFAULT_T)) { @@ -348,7 +351,7 @@ int ll_getxattr_common(struct inode *inode, const char *name, if (!acl) return -ENODATA; - rc = posix_acl_to_xattr(&init_user_ns, acl, buffer, size); + rc = posix_acl_to_xattr(current_user_ns(), acl, buffer, size); posix_acl_release(acl); return rc; } @@ -436,6 +439,9 @@ getxattr_nocache: goto out; } } + if (rc >= 0 && (xattr_type == XATTR_ACL_ACCESS_T || + xattr_type == XATTR_ACL_DEFAULT_T)) + posix_acl_fix_xattr_to_user(buffer, rc); #endif out_xattr: diff --git a/fs/cifs/cifssmb.c b/fs/cifs/cifssmb.c index d47197e..9dc001f 100644 --- a/fs/cifs/cifssmb.c +++ b/fs/cifs/cifssmb.c @@ -3337,10 +3337,25 @@ CIFSSMB_set_compression(const unsigned int xid, struct cifs_tcon *tcon, static void cifs_convert_ace(posix_acl_xattr_entry *ace, struct cifs_posix_ace *cifs_ace) { + u32 cifs_id, id = -1; + /* u8 cifs fields do not need le conversion */ ace->e_perm = cpu_to_le16(cifs_ace->cifs_e_perm); ace->e_tag = cpu_to_le16(cifs_ace->cifs_e_tag); - ace->e_id = cpu_to_le32(le64_to_cpu(cifs_ace->cifs_uid)); + switch(cifs_ace->cifs_e_tag) { + case ACL_USER: + cifs_id = le64_to_cpu(cifs_ace->cifs_uid); + id = from_kuid(current_user_ns(), + make_kuid(&init_user_ns, cifs_id)); + break; + + case ACL_GROUP: + cifs_id = le64_to_cpu(cifs_ace->cifs_uid); + id = from_kgid(current_user_ns(), + make_kgid(&init_user_ns, cifs_id)); + break; + } + ace->e_id = cpu_to_le32(id); /* cifs_dbg(FYI, "perm %d tag %d id %d\n", ace->e_perm, ace->e_tag, ace->e_id); @@ -3408,21 +3423,29 @@ static int cifs_copy_posix_acl(char *trgt, char *src, const int buflen, static __u16 convert_ace_to_cifs_ace(struct cifs_posix_ace *cifs_ace, const posix_acl_xattr_entry *local_ace) { - __u16 rc = 0; /* 0 = ACL converted ok */ + u32 cifs_id = -1, id; cifs_ace->cifs_e_perm = le16_to_cpu(local_ace->e_perm); cifs_ace->cifs_e_tag = le16_to_cpu(local_ace->e_tag); - /* BB is there a better way to handle the large uid? */ - if (local_ace->e_id == cpu_to_le32(-1)) { - /* Probably no need to le convert -1 on any arch but can not hurt */ - cifs_ace->cifs_uid = cpu_to_le64(-1); - } else - cifs_ace->cifs_uid = cpu_to_le64(le32_to_cpu(local_ace->e_id)); + switch(cifs_ace->cifs_e_tag) { + case ACL_USER: + id = le32_to_cpu(local_ace->e_id); + cifs_id = from_kuid(&init_user_ns, + make_kuid(current_user_ns(), id)); + break; + + case ACL_GROUP: + id = le32_to_cpu(local_ace->e_id); + cifs_id = from_kgid(&init_user_ns, + make_kgid(current_user_ns(), id)); + break; + } + cifs_ace->cifs_uid = cpu_to_le64(cifs_id); /* cifs_dbg(FYI, "perm %d tag %d id %d\n", ace->e_perm, ace->e_tag, ace->e_id); */ - return rc; + return 0; } /* Convert ACL from local Linux POSIX xattr to CIFS POSIX ACL wire format */ diff --git a/fs/posix_acl.c b/fs/posix_acl.c index 2c60f17..fca281c 100644 --- a/fs/posix_acl.c +++ b/fs/posix_acl.c @@ -627,64 +627,6 @@ no_mem: EXPORT_SYMBOL_GPL(posix_acl_create); /* - * Fix up the uids and gids in posix acl extended attributes in place. - */ -static void posix_acl_fix_xattr_userns( - struct user_namespace *to, struct user_namespace *from, - void *value, size_t size) -{ - posix_acl_xattr_header *header = (posix_acl_xattr_header *)value; - posix_acl_xattr_entry *entry = (posix_acl_xattr_entry *)(header+1), *end; - int count; - kuid_t uid; - kgid_t gid; - - if (!value) - return; - if (size < sizeof(posix_acl_xattr_header)) - return; - if (header->a_version != cpu_to_le32(POSIX_ACL_XATTR_VERSION)) - return; - - count = posix_acl_xattr_count(size); - if (count < 0) - return; - if (count == 0) - return; - - for (end = entry + count; entry != end; entry++) { - switch(le16_to_cpu(entry->e_tag)) { - case ACL_USER: - uid = make_kuid(from, le32_to_cpu(entry->e_id)); - entry->e_id = cpu_to_le32(from_kuid(to, uid)); - break; - case ACL_GROUP: - gid = make_kgid(from, le32_to_cpu(entry->e_id)); - entry->e_id = cpu_to_le32(from_kgid(to, gid)); - break; - default: - break; - } - } -} - -void posix_acl_fix_xattr_from_user(void *value, size_t size) -{ - struct user_namespace *user_ns = current_user_ns(); - if (user_ns == &init_user_ns) - return; - posix_acl_fix_xattr_userns(&init_user_ns, user_ns, value, size); -} - -void posix_acl_fix_xattr_to_user(void *value, size_t size) -{ - struct user_namespace *user_ns = current_user_ns(); - if (user_ns == &init_user_ns) - return; - posix_acl_fix_xattr_userns(user_ns, &init_user_ns, value, size); -} - -/* * Convert from extended attribute to in-memory representation. */ struct posix_acl * @@ -814,7 +756,7 @@ posix_acl_xattr_get(const struct xattr_handler *handler, if (acl == NULL) return -ENODATA; - error = posix_acl_to_xattr(&init_user_ns, acl, value, size); + error = posix_acl_to_xattr(current_user_ns(), acl, value, size); posix_acl_release(acl); return error; @@ -840,7 +782,7 @@ posix_acl_xattr_set(const struct xattr_handler *handler, return -EPERM; if (value) { - acl = posix_acl_from_xattr(&init_user_ns, value, size); + acl = posix_acl_from_xattr(current_user_ns(), value, size); if (IS_ERR(acl)) return PTR_ERR(acl); diff --git a/fs/xattr.c b/fs/xattr.c index b11945e..b648b05 100644 --- a/fs/xattr.c +++ b/fs/xattr.c @@ -20,7 +20,6 @@ #include #include #include -#include #include @@ -329,9 +328,6 @@ setxattr(struct dentry *d, const char __user *name, const void __user *value, error = -EFAULT; goto out; } - if ((strcmp(kname, XATTR_NAME_POSIX_ACL_ACCESS) == 0) || - (strcmp(kname, XATTR_NAME_POSIX_ACL_DEFAULT) == 0)) - posix_acl_fix_xattr_from_user(kvalue, size); } error = vfs_setxattr(d, kname, kvalue, size, flags); @@ -426,9 +422,6 @@ getxattr(struct dentry *d, const char __user *name, void __user *value, error = vfs_getxattr(d, kname, kvalue, size); if (error > 0) { - if ((strcmp(kname, XATTR_NAME_POSIX_ACL_ACCESS) == 0) || - (strcmp(kname, XATTR_NAME_POSIX_ACL_DEFAULT) == 0)) - posix_acl_fix_xattr_to_user(kvalue, size); if (size && copy_to_user(value, kvalue, error)) error = -EFAULT; } else if (error == -ERANGE && size >= XATTR_SIZE_MAX) { diff --git a/include/linux/posix_acl_xattr.h b/include/linux/posix_acl_xattr.h index e5e8ec4..9789aba 100644 --- a/include/linux/posix_acl_xattr.h +++ b/include/linux/posix_acl_xattr.h @@ -48,18 +48,6 @@ posix_acl_xattr_count(size_t size) return size / sizeof(posix_acl_xattr_entry); } -#ifdef CONFIG_FS_POSIX_ACL -void posix_acl_fix_xattr_from_user(void *value, size_t size); -void posix_acl_fix_xattr_to_user(void *value, size_t size); -#else -static inline void posix_acl_fix_xattr_from_user(void *value, size_t size) -{ -} -static inline void posix_acl_fix_xattr_to_user(void *value, size_t size) -{ -} -#endif - struct posix_acl *posix_acl_from_xattr(struct user_namespace *user_ns, const void *value, size_t size); int posix_acl_to_xattr(struct user_namespace *user_ns, -- 2.5.5 From jsimmons at infradead.org Mon May 23 21:06:27 2016 From: jsimmons at infradead.org (James Simmons) Date: Mon, 23 May 2016 22:06:27 +0100 (BST) Subject: [lustre-devel] [PATCH] posix acls: Move namespace conversion into filesystem / xattr handlers In-Reply-To: <1464008989-3812-1-git-send-email-agruenba@redhat.com> References: <1464008989-3812-1-git-send-email-agruenba@redhat.com> Message-ID: > Currently, getxattr() and setxattr() check for the xattr names > "system.posix_acl_{access,default}" and perform in-place UID / GID > namespace mappings in the xattr values. Filesystems then again check for > the same xattr names to handle those attributes, almost always using the > standard posix_acl_{access,default}_xattr_handler handlers. This is > unnecessary overhead; move the namespace conversion into the xattr > handlers instead. > > For filesystems that use the POSIX ACL xattr handlers, no change is > required. Filesystems that don't use those handlers (cifs and lustre) > need to take care of the namespace mapping themselves now. > > The only user left of the posix_acl_fix_xattr_{from,to}_user helpers is > lustre, so this patch moves them into lustre. > > Signed-off-by: Andreas Gruenbacher > --- > > I'm reasonably confident about the core and cifs changes in this patch. > The lustre code is pretty weird though, so could I please get a careful > review on the changes there? Nak on the Lustre changes. POSIX ACLs are also handled in mdc_request.c. Besides POSIX ACLs lustre has created a extended ACL as well that is grouped in with posix ACL handling. This extended ACL is like POSIX acls except it also contains the state (deleted, modified, ...) of the ACL. Besides normal local access control list handling Lustre manages remote access control list handling. You can read about this handling is in llite_rmtacl.c. This code was written long before I became involved with lustre so the exact details are fuzzy to me. The guts of this are handled is located at: drivers/staging/lustre/lustre/obdclass/acl.c P.S A you probably have noticed Lustre has had an uptick in development which means the code is now changing all the time in staging. How should we handle the changes you are working in your own trees verses what is happing in staging. For example I'm looking at moving lustre to the xattr_handles. Should I push it to you and wait until it works it way into Greg's tree. What do the merge schedules look like. Lastly the a_refcount for the POSIX acl changed with your xattr updates which now causes lustre to crash :-( > Thanks, > Andreas > > drivers/staging/lustre/lustre/llite/Makefile | 1 + > .../staging/lustre/lustre/llite/llite_internal.h | 3 ++ > drivers/staging/lustre/lustre/llite/posix_acl.c | 62 ++++++++++++++++++++++ > drivers/staging/lustre/lustre/llite/xattr.c | 8 ++- > fs/cifs/cifssmb.c | 41 ++++++++++---- > fs/posix_acl.c | 62 +--------------------- > fs/xattr.c | 7 --- > include/linux/posix_acl_xattr.h | 12 ----- > 8 files changed, 107 insertions(+), 89 deletions(-) > create mode 100644 drivers/staging/lustre/lustre/llite/posix_acl.c > > diff --git a/drivers/staging/lustre/lustre/llite/Makefile b/drivers/staging/lustre/lustre/llite/Makefile > index 2ce10ff..67125f8 100644 > --- a/drivers/staging/lustre/lustre/llite/Makefile > +++ b/drivers/staging/lustre/lustre/llite/Makefile > @@ -7,5 +7,6 @@ lustre-y := dcache.o dir.o file.o llite_close.o llite_lib.o llite_nfs.o \ > glimpse.o lcommon_cl.o lcommon_misc.o \ > vvp_dev.o vvp_page.o vvp_lock.o vvp_io.o vvp_object.o vvp_req.o \ > lproc_llite.o > +lustre-$(CONFIG_FS_POSIX_ACL) += posix_acl.o > > llite_lloop-y := lloop.o > diff --git a/drivers/staging/lustre/lustre/llite/llite_internal.h b/drivers/staging/lustre/lustre/llite/llite_internal.h > index ce1f949..d454dfb 100644 > --- a/drivers/staging/lustre/lustre/llite/llite_internal.h > +++ b/drivers/staging/lustre/lustre/llite/llite_internal.h > @@ -1402,4 +1402,7 @@ int cl_local_size(struct inode *inode); > __u64 cl_fid_build_ino(const struct lu_fid *fid, int api32); > __u32 cl_fid_build_gen(const struct lu_fid *fid); > > +void posix_acl_fix_xattr_from_user(void *value, size_t size); > +void posix_acl_fix_xattr_to_user(void *value, size_t size); > + > #endif /* LLITE_INTERNAL_H */ > diff --git a/drivers/staging/lustre/lustre/llite/posix_acl.c b/drivers/staging/lustre/lustre/llite/posix_acl.c > new file mode 100644 > index 0000000..4fabd0f > --- /dev/null > +++ b/drivers/staging/lustre/lustre/llite/posix_acl.c > @@ -0,0 +1,62 @@ > +#include > +#include > +#include > +#include > + > +/* > + * Fix up the uids and gids in posix acl extended attributes in place. > + */ > +static void posix_acl_fix_xattr_userns( > + struct user_namespace *to, struct user_namespace *from, > + void *value, size_t size) > +{ > + posix_acl_xattr_header *header = (posix_acl_xattr_header *)value; > + posix_acl_xattr_entry *entry = (posix_acl_xattr_entry *)(header+1), *end; > + int count; > + kuid_t uid; > + kgid_t gid; > + > + if (!value) > + return; > + if (size < sizeof(posix_acl_xattr_header)) > + return; > + if (header->a_version != cpu_to_le32(POSIX_ACL_XATTR_VERSION)) > + return; > + > + count = posix_acl_xattr_count(size); > + if (count < 0) > + return; > + if (count == 0) > + return; > + > + for (end = entry + count; entry != end; entry++) { > + switch(le16_to_cpu(entry->e_tag)) { > + case ACL_USER: > + uid = make_kuid(from, le32_to_cpu(entry->e_id)); > + entry->e_id = cpu_to_le32(from_kuid(to, uid)); > + break; > + case ACL_GROUP: > + gid = make_kgid(from, le32_to_cpu(entry->e_id)); > + entry->e_id = cpu_to_le32(from_kgid(to, gid)); > + break; > + default: > + break; > + } > + } > +} > + > +void posix_acl_fix_xattr_from_user(void *value, size_t size) > +{ > + struct user_namespace *user_ns = current_user_ns(); > + if (user_ns == &init_user_ns) > + return; > + posix_acl_fix_xattr_userns(&init_user_ns, user_ns, value, size); > +} > + > +void posix_acl_fix_xattr_to_user(void *value, size_t size) > +{ > + struct user_namespace *user_ns = current_user_ns(); > + if (user_ns == &init_user_ns) > + return; > + posix_acl_fix_xattr_userns(user_ns, &init_user_ns, value, size); > +} > diff --git a/drivers/staging/lustre/lustre/llite/xattr.c b/drivers/staging/lustre/lustre/llite/xattr.c > index ed4de04..c721b44 100644 > --- a/drivers/staging/lustre/lustre/llite/xattr.c > +++ b/drivers/staging/lustre/lustre/llite/xattr.c > @@ -144,6 +144,9 @@ int ll_setxattr_common(struct inode *inode, const char *name, > return -EOPNOTSUPP; > > #ifdef CONFIG_FS_POSIX_ACL > + if (xattr_type == XATTR_ACL_ACCESS_T || > + xattr_type == XATTR_ACL_DEFAULT_T) > + posix_acl_fix_xattr_from_user((void *)value, size); > if (sbi->ll_flags & LL_SBI_RMT_CLIENT && > (xattr_type == XATTR_ACL_ACCESS_T || > xattr_type == XATTR_ACL_DEFAULT_T)) { > @@ -348,7 +351,7 @@ int ll_getxattr_common(struct inode *inode, const char *name, > if (!acl) > return -ENODATA; > > - rc = posix_acl_to_xattr(&init_user_ns, acl, buffer, size); > + rc = posix_acl_to_xattr(current_user_ns(), acl, buffer, size); > posix_acl_release(acl); > return rc; > } > @@ -436,6 +439,9 @@ getxattr_nocache: > goto out; > } > } > + if (rc >= 0 && (xattr_type == XATTR_ACL_ACCESS_T || > + xattr_type == XATTR_ACL_DEFAULT_T)) > + posix_acl_fix_xattr_to_user(buffer, rc); > #endif > > out_xattr: > diff --git a/fs/cifs/cifssmb.c b/fs/cifs/cifssmb.c > index d47197e..9dc001f 100644 > --- a/fs/cifs/cifssmb.c > +++ b/fs/cifs/cifssmb.c > @@ -3337,10 +3337,25 @@ CIFSSMB_set_compression(const unsigned int xid, struct cifs_tcon *tcon, > static void cifs_convert_ace(posix_acl_xattr_entry *ace, > struct cifs_posix_ace *cifs_ace) > { > + u32 cifs_id, id = -1; > + > /* u8 cifs fields do not need le conversion */ > ace->e_perm = cpu_to_le16(cifs_ace->cifs_e_perm); > ace->e_tag = cpu_to_le16(cifs_ace->cifs_e_tag); > - ace->e_id = cpu_to_le32(le64_to_cpu(cifs_ace->cifs_uid)); > + switch(cifs_ace->cifs_e_tag) { > + case ACL_USER: > + cifs_id = le64_to_cpu(cifs_ace->cifs_uid); > + id = from_kuid(current_user_ns(), > + make_kuid(&init_user_ns, cifs_id)); > + break; > + > + case ACL_GROUP: > + cifs_id = le64_to_cpu(cifs_ace->cifs_uid); > + id = from_kgid(current_user_ns(), > + make_kgid(&init_user_ns, cifs_id)); > + break; > + } > + ace->e_id = cpu_to_le32(id); > /* > cifs_dbg(FYI, "perm %d tag %d id %d\n", > ace->e_perm, ace->e_tag, ace->e_id); > @@ -3408,21 +3423,29 @@ static int cifs_copy_posix_acl(char *trgt, char *src, const int buflen, > static __u16 convert_ace_to_cifs_ace(struct cifs_posix_ace *cifs_ace, > const posix_acl_xattr_entry *local_ace) > { > - __u16 rc = 0; /* 0 = ACL converted ok */ > + u32 cifs_id = -1, id; > > cifs_ace->cifs_e_perm = le16_to_cpu(local_ace->e_perm); > cifs_ace->cifs_e_tag = le16_to_cpu(local_ace->e_tag); > - /* BB is there a better way to handle the large uid? */ > - if (local_ace->e_id == cpu_to_le32(-1)) { > - /* Probably no need to le convert -1 on any arch but can not hurt */ > - cifs_ace->cifs_uid = cpu_to_le64(-1); > - } else > - cifs_ace->cifs_uid = cpu_to_le64(le32_to_cpu(local_ace->e_id)); > + switch(cifs_ace->cifs_e_tag) { > + case ACL_USER: > + id = le32_to_cpu(local_ace->e_id); > + cifs_id = from_kuid(&init_user_ns, > + make_kuid(current_user_ns(), id)); > + break; > + > + case ACL_GROUP: > + id = le32_to_cpu(local_ace->e_id); > + cifs_id = from_kgid(&init_user_ns, > + make_kgid(current_user_ns(), id)); > + break; > + } > + cifs_ace->cifs_uid = cpu_to_le64(cifs_id); > /* > cifs_dbg(FYI, "perm %d tag %d id %d\n", > ace->e_perm, ace->e_tag, ace->e_id); > */ > - return rc; > + return 0; > } > > /* Convert ACL from local Linux POSIX xattr to CIFS POSIX ACL wire format */ > diff --git a/fs/posix_acl.c b/fs/posix_acl.c > index 2c60f17..fca281c 100644 > --- a/fs/posix_acl.c > +++ b/fs/posix_acl.c > @@ -627,64 +627,6 @@ no_mem: > EXPORT_SYMBOL_GPL(posix_acl_create); > > /* > - * Fix up the uids and gids in posix acl extended attributes in place. > - */ > -static void posix_acl_fix_xattr_userns( > - struct user_namespace *to, struct user_namespace *from, > - void *value, size_t size) > -{ > - posix_acl_xattr_header *header = (posix_acl_xattr_header *)value; > - posix_acl_xattr_entry *entry = (posix_acl_xattr_entry *)(header+1), *end; > - int count; > - kuid_t uid; > - kgid_t gid; > - > - if (!value) > - return; > - if (size < sizeof(posix_acl_xattr_header)) > - return; > - if (header->a_version != cpu_to_le32(POSIX_ACL_XATTR_VERSION)) > - return; > - > - count = posix_acl_xattr_count(size); > - if (count < 0) > - return; > - if (count == 0) > - return; > - > - for (end = entry + count; entry != end; entry++) { > - switch(le16_to_cpu(entry->e_tag)) { > - case ACL_USER: > - uid = make_kuid(from, le32_to_cpu(entry->e_id)); > - entry->e_id = cpu_to_le32(from_kuid(to, uid)); > - break; > - case ACL_GROUP: > - gid = make_kgid(from, le32_to_cpu(entry->e_id)); > - entry->e_id = cpu_to_le32(from_kgid(to, gid)); > - break; > - default: > - break; > - } > - } > -} > - > -void posix_acl_fix_xattr_from_user(void *value, size_t size) > -{ > - struct user_namespace *user_ns = current_user_ns(); > - if (user_ns == &init_user_ns) > - return; > - posix_acl_fix_xattr_userns(&init_user_ns, user_ns, value, size); > -} > - > -void posix_acl_fix_xattr_to_user(void *value, size_t size) > -{ > - struct user_namespace *user_ns = current_user_ns(); > - if (user_ns == &init_user_ns) > - return; > - posix_acl_fix_xattr_userns(user_ns, &init_user_ns, value, size); > -} > - > -/* > * Convert from extended attribute to in-memory representation. > */ > struct posix_acl * > @@ -814,7 +756,7 @@ posix_acl_xattr_get(const struct xattr_handler *handler, > if (acl == NULL) > return -ENODATA; > > - error = posix_acl_to_xattr(&init_user_ns, acl, value, size); > + error = posix_acl_to_xattr(current_user_ns(), acl, value, size); > posix_acl_release(acl); > > return error; > @@ -840,7 +782,7 @@ posix_acl_xattr_set(const struct xattr_handler *handler, > return -EPERM; > > if (value) { > - acl = posix_acl_from_xattr(&init_user_ns, value, size); > + acl = posix_acl_from_xattr(current_user_ns(), value, size); > if (IS_ERR(acl)) > return PTR_ERR(acl); > > diff --git a/fs/xattr.c b/fs/xattr.c > index b11945e..b648b05 100644 > --- a/fs/xattr.c > +++ b/fs/xattr.c > @@ -20,7 +20,6 @@ > #include > #include > #include > -#include > > #include > > @@ -329,9 +328,6 @@ setxattr(struct dentry *d, const char __user *name, const void __user *value, > error = -EFAULT; > goto out; > } > - if ((strcmp(kname, XATTR_NAME_POSIX_ACL_ACCESS) == 0) || > - (strcmp(kname, XATTR_NAME_POSIX_ACL_DEFAULT) == 0)) > - posix_acl_fix_xattr_from_user(kvalue, size); > } > > error = vfs_setxattr(d, kname, kvalue, size, flags); > @@ -426,9 +422,6 @@ getxattr(struct dentry *d, const char __user *name, void __user *value, > > error = vfs_getxattr(d, kname, kvalue, size); > if (error > 0) { > - if ((strcmp(kname, XATTR_NAME_POSIX_ACL_ACCESS) == 0) || > - (strcmp(kname, XATTR_NAME_POSIX_ACL_DEFAULT) == 0)) > - posix_acl_fix_xattr_to_user(kvalue, size); > if (size && copy_to_user(value, kvalue, error)) > error = -EFAULT; > } else if (error == -ERANGE && size >= XATTR_SIZE_MAX) { > diff --git a/include/linux/posix_acl_xattr.h b/include/linux/posix_acl_xattr.h > index e5e8ec4..9789aba 100644 > --- a/include/linux/posix_acl_xattr.h > +++ b/include/linux/posix_acl_xattr.h > @@ -48,18 +48,6 @@ posix_acl_xattr_count(size_t size) > return size / sizeof(posix_acl_xattr_entry); > } > > -#ifdef CONFIG_FS_POSIX_ACL > -void posix_acl_fix_xattr_from_user(void *value, size_t size); > -void posix_acl_fix_xattr_to_user(void *value, size_t size); > -#else > -static inline void posix_acl_fix_xattr_from_user(void *value, size_t size) > -{ > -} > -static inline void posix_acl_fix_xattr_to_user(void *value, size_t size) > -{ > -} > -#endif > - > struct posix_acl *posix_acl_from_xattr(struct user_namespace *user_ns, > const void *value, size_t size); > int posix_acl_to_xattr(struct user_namespace *user_ns, > -- > 2.5.5 > > _______________________________________________ > lustre-devel mailing list > lustre-devel at lists.lustre.org > http://lists.lustre.org/listinfo.cgi/lustre-devel-lustre.org > From jsimmons at infradead.org Tue May 24 00:35:08 2016 From: jsimmons at infradead.org (James Simmons) Date: Mon, 23 May 2016 20:35:08 -0400 Subject: [lustre-devel] [PATCH] staging: lustre: llite: drop acl from cache Message-ID: <1464050108-28015-1-git-send-email-jsimmons@infradead.org> Commit b8a7a3a6 change get_acl() for posix xattr to always cache the ACL which increases the reference count. That reference count can be reduced by have ll_get_acl() call forget_cached_acl() which it wasn't. When an inode gets deleted by Lustre the POSIX ACL reference count is tested to ensure its 1 and if not produces an error. Since forget_cached_acl() was not called Lustre started to complain. This patch changes ll_get_acl() to call forget_cached_acl(). Signed-off-by: James Simmons --- drivers/staging/lustre/lustre/llite/file.c | 1 + 1 files changed, 1 insertions(+), 0 deletions(-) diff --git a/drivers/staging/lustre/lustre/llite/file.c b/drivers/staging/lustre/lustre/llite/file.c index f47f2ac..0191945 100644 --- a/drivers/staging/lustre/lustre/llite/file.c +++ b/drivers/staging/lustre/lustre/llite/file.c @@ -3124,6 +3124,7 @@ struct posix_acl *ll_get_acl(struct inode *inode, int type) spin_lock(&lli->lli_lock); /* VFS' acl_permission_check->check_acl will release the refcount */ acl = posix_acl_dup(lli->lli_posix_acl); + forget_cached_acl(inode, type); spin_unlock(&lli->lli_lock); return acl; -- 1.7.1 From hch at infradead.org Tue May 24 08:47:24 2016 From: hch at infradead.org (Christoph Hellwig) Date: Tue, 24 May 2016 01:47:24 -0700 Subject: [lustre-devel] [PATCH] posix acls: Move namespace conversion into filesystem / xattr handlers In-Reply-To: <1464008989-3812-1-git-send-email-agruenba@redhat.com> References: <1464008989-3812-1-git-send-email-agruenba@redhat.com> Message-ID: <20160524084724.GA8121@infradead.org> On Mon, May 23, 2016 at 03:09:49PM +0200, Andreas Gruenbacher wrote: > I'm reasonably confident about the core and cifs changes in this patch. > The lustre code is pretty weird though, so could I please get a careful > review on the changes there? Lustre is in the staging tree because it's an unrescuable piece of junk. Please just ignore it and move the proper parts of the kernel forward. From jsimmons at infradead.org Tue May 24 14:44:32 2016 From: jsimmons at infradead.org (James Simmons) Date: Tue, 24 May 2016 15:44:32 +0100 (BST) Subject: [lustre-devel] LWN article on kernel stats reveal upstream client progress. In-Reply-To: <924F1865-65B5-4DBE-85C3-982D0BAF99D7@intel.com> References: <11629f7822674b1d872d078bcb422717@EXCHCS32.ornl.gov> <924F1865-65B5-4DBE-85C3-982D0BAF99D7@intel.com> Message-ID: > James, > How close are we to getting LNet out of staging and into mainline?  If we are getting close, should we consider > “separating” LNet from Lustre making it a standalone item?  Considering how we have to propagate all LNet fixes to > every active repo/version, I get the feeling that LNet should be separate anyway to make updating just LNet in the > field easier. Well that depends on Greg. Before Greg stated it was an all or nothing deal for destaging. Some people would like to see LNet leave destaging first but we still need to resolve some things. First we have some more checkpatch cleanups to do. Lots of typedefs are left. I cleaned up the lnetself test ones but more are left. Other odds and ends for checkpatch as well. Besides these cleanups Greg discovered lnetself uses linux kernel list to pass data back and forth with the ioctls. This gave Greg very bad heartburn. Other missing pieces are moving from the obsolete socket api's to the netlink api. Another piece of concern is the lack of IPv6 but that would require a massive rewrite :-(. We might be able to avoid the lack of IPv6 to get out of staging. From agruenba at redhat.com Tue May 24 16:31:51 2016 From: agruenba at redhat.com (Andreas Gruenbacher) Date: Tue, 24 May 2016 18:31:51 +0200 Subject: [lustre-devel] [PATCH] posix acls: Move namespace conversion into filesystem / xattr handlers In-Reply-To: <20160524153949.GA2830@dztty.fritz.box> References: <1464008989-3812-1-git-send-email-agruenba@redhat.com> <20160524153949.GA2830@dztty.fritz.box> Message-ID: On Tue, May 24, 2016 at 5:41 PM, Djalal Harouni wrote: > On Mon, May 23, 2016 at 03:09:49PM +0200, Andreas Gruenbacher wrote: >> Currently, getxattr() and setxattr() check for the xattr names >> "system.posix_acl_{access,default}" and perform in-place UID / GID >> namespace mappings in the xattr values. Filesystems then again check for >> the same xattr names to handle those attributes, almost always using the >> standard posix_acl_{access,default}_xattr_handler handlers. This is >> unnecessary overhead; move the namespace conversion into the xattr >> handlers instead. > > Please, are you sure that the changes in posix_acl_xattr_get() and > posix_acl_xattr_set() are safe ? you are reading into current user > namespace, from a first view this is not safe unless I'm missing > something... they should map into init_user_ns... Yes, moving the namespace conversion from the VFS into those functions so that we don't have to check for those attributes and parse them twice is exactly the point of this patch. > Please Cc the user namespace maintainers before. Thank you! Eric, Andy, anyone else? Thanks, Andreas From andreas.dilger at intel.com Tue May 24 18:28:23 2016 From: andreas.dilger at intel.com (Dilger, Andreas) Date: Tue, 24 May 2016 18:28:23 +0000 Subject: [lustre-devel] [PATCH] posix acls: Move namespace conversion into filesystem / xattr handlers In-Reply-To: References: <1464008989-3812-1-git-send-email-agruenba@redhat.com> Message-ID: On 2016/05/23, 15:06, "James Simmons" wrote: > >> Currently, getxattr() and setxattr() check for the xattr names >> "system.posix_acl_{access,default}" and perform in-place UID / GID >> namespace mappings in the xattr values. Filesystems then again check for >> the same xattr names to handle those attributes, almost always using the >> standard posix_acl_{access,default}_xattr_handler handlers. This is >> unnecessary overhead; move the namespace conversion into the xattr >> handlers instead. >> >> For filesystems that use the POSIX ACL xattr handlers, no change is >> required. Filesystems that don't use those handlers (cifs and lustre) >> need to take care of the namespace mapping themselves now. >> >> The only user left of the posix_acl_fix_xattr_{from,to}_user helpers is >> lustre, so this patch moves them into lustre. >> >> Signed-off-by: Andreas Gruenbacher >> --- >> >> I'm reasonably confident about the core and cifs changes in this patch. >> The lustre code is pretty weird though, so could I please get a careful >> review on the changes there? > >Nak on the Lustre changes. POSIX ACLs are also handled in mdc_request.c. >Besides POSIX ACLs lustre has created a extended ACL as well that is >grouped in with posix ACL handling. This extended ACL is like POSIX acls >except it also contains the state (deleted, modified, ...) of the ACL. >Besides normal local access control list handling Lustre manages remote >access control list handling. You can read about this handling is in >llite_rmtacl.c. This code was written long before I became involved with >lustre so the exact details are fuzzy to me. James, the remote ACL code never found any usage in the field and can be deleted. Please see http://review.whamcloud.com/19789 for details. Cheers, Andreas > The guts of this are handled is located at: > >drivers/staging/lustre/lustre/obdclass/acl.c > >P.S > > A you probably have noticed Lustre has had an uptick in development >which means the code is now changing all the time in staging. How should >we handle the changes you are working in your own trees verses what is >happing in staging. For example I'm looking at moving lustre to the >xattr_handles. Should I push it to you and wait until it works it way >into Greg's tree. What do the merge schedules look like. Lastly the >a_refcount for the POSIX acl changed with your xattr updates which now >causes lustre to crash :-( > >> Thanks, >> Andreas >> >> drivers/staging/lustre/lustre/llite/Makefile | 1 + >> .../staging/lustre/lustre/llite/llite_internal.h | 3 ++ >> drivers/staging/lustre/lustre/llite/posix_acl.c | 62 ++++++++++++++++++++++ >> drivers/staging/lustre/lustre/llite/xattr.c | 8 ++- >> fs/cifs/cifssmb.c | 41 ++++++++++---- >> fs/posix_acl.c | 62 +--------------------- >> fs/xattr.c | 7 --- >> include/linux/posix_acl_xattr.h | 12 ----- >> 8 files changed, 107 insertions(+), 89 deletions(-) >> create mode 100644 drivers/staging/lustre/lustre/llite/posix_acl.c >> >> diff --git a/drivers/staging/lustre/lustre/llite/Makefile b/drivers/staging/lustre/lustre/llite/Makefile >> index 2ce10ff..67125f8 100644 >> --- a/drivers/staging/lustre/lustre/llite/Makefile >> +++ b/drivers/staging/lustre/lustre/llite/Makefile >> @@ -7,5 +7,6 @@ lustre-y := dcache.o dir.o file.o llite_close.o llite_lib.o llite_nfs.o \ >> glimpse.o lcommon_cl.o lcommon_misc.o \ >> vvp_dev.o vvp_page.o vvp_lock.o vvp_io.o vvp_object.o vvp_req.o \ >> lproc_llite.o >> +lustre-$(CONFIG_FS_POSIX_ACL) += posix_acl.o >> >> llite_lloop-y := lloop.o >> diff --git a/drivers/staging/lustre/lustre/llite/llite_internal.h b/drivers/staging/lustre/lustre/llite/llite_internal.h >> index ce1f949..d454dfb 100644 >> --- a/drivers/staging/lustre/lustre/llite/llite_internal.h >> +++ b/drivers/staging/lustre/lustre/llite/llite_internal.h >> @@ -1402,4 +1402,7 @@ int cl_local_size(struct inode *inode); >> __u64 cl_fid_build_ino(const struct lu_fid *fid, int api32); >> __u32 cl_fid_build_gen(const struct lu_fid *fid); >> >> +void posix_acl_fix_xattr_from_user(void *value, size_t size); >> +void posix_acl_fix_xattr_to_user(void *value, size_t size); >> + >> #endif /* LLITE_INTERNAL_H */ >> diff --git a/drivers/staging/lustre/lustre/llite/posix_acl.c b/drivers/staging/lustre/lustre/llite/posix_acl.c >> new file mode 100644 >> index 0000000..4fabd0f >> --- /dev/null >> +++ b/drivers/staging/lustre/lustre/llite/posix_acl.c >> @@ -0,0 +1,62 @@ >> +#include >> +#include >> +#include >> +#include >> + >> +/* >> + * Fix up the uids and gids in posix acl extended attributes in place. >> + */ >> +static void posix_acl_fix_xattr_userns( >> + struct user_namespace *to, struct user_namespace *from, >> + void *value, size_t size) >> +{ >> + posix_acl_xattr_header *header = (posix_acl_xattr_header *)value; >> + posix_acl_xattr_entry *entry = (posix_acl_xattr_entry *)(header+1), *end; >> + int count; >> + kuid_t uid; >> + kgid_t gid; >> + >> + if (!value) >> + return; >> + if (size < sizeof(posix_acl_xattr_header)) >> + return; >> + if (header->a_version != cpu_to_le32(POSIX_ACL_XATTR_VERSION)) >> + return; >> + >> + count = posix_acl_xattr_count(size); >> + if (count < 0) >> + return; >> + if (count == 0) >> + return; >> + >> + for (end = entry + count; entry != end; entry++) { >> + switch(le16_to_cpu(entry->e_tag)) { >> + case ACL_USER: >> + uid = make_kuid(from, le32_to_cpu(entry->e_id)); >> + entry->e_id = cpu_to_le32(from_kuid(to, uid)); >> + break; >> + case ACL_GROUP: >> + gid = make_kgid(from, le32_to_cpu(entry->e_id)); >> + entry->e_id = cpu_to_le32(from_kgid(to, gid)); >> + break; >> + default: >> + break; >> + } >> + } >> +} >> + >> +void posix_acl_fix_xattr_from_user(void *value, size_t size) >> +{ >> + struct user_namespace *user_ns = current_user_ns(); >> + if (user_ns == &init_user_ns) >> + return; >> + posix_acl_fix_xattr_userns(&init_user_ns, user_ns, value, size); >> +} >> + >> +void posix_acl_fix_xattr_to_user(void *value, size_t size) >> +{ >> + struct user_namespace *user_ns = current_user_ns(); >> + if (user_ns == &init_user_ns) >> + return; >> + posix_acl_fix_xattr_userns(user_ns, &init_user_ns, value, size); >> +} >> diff --git a/drivers/staging/lustre/lustre/llite/xattr.c b/drivers/staging/lustre/lustre/llite/xattr.c >> index ed4de04..c721b44 100644 >> --- a/drivers/staging/lustre/lustre/llite/xattr.c >> +++ b/drivers/staging/lustre/lustre/llite/xattr.c >> @@ -144,6 +144,9 @@ int ll_setxattr_common(struct inode *inode, const char *name, >> return -EOPNOTSUPP; >> >> #ifdef CONFIG_FS_POSIX_ACL >> + if (xattr_type == XATTR_ACL_ACCESS_T || >> + xattr_type == XATTR_ACL_DEFAULT_T) >> + posix_acl_fix_xattr_from_user((void *)value, size); >> if (sbi->ll_flags & LL_SBI_RMT_CLIENT && >> (xattr_type == XATTR_ACL_ACCESS_T || >> xattr_type == XATTR_ACL_DEFAULT_T)) { >> @@ -348,7 +351,7 @@ int ll_getxattr_common(struct inode *inode, const char *name, >> if (!acl) >> return -ENODATA; >> >> - rc = posix_acl_to_xattr(&init_user_ns, acl, buffer, size); >> + rc = posix_acl_to_xattr(current_user_ns(), acl, buffer, size); >> posix_acl_release(acl); >> return rc; >> } >> @@ -436,6 +439,9 @@ getxattr_nocache: >> goto out; >> } >> } >> + if (rc >= 0 && (xattr_type == XATTR_ACL_ACCESS_T || >> + xattr_type == XATTR_ACL_DEFAULT_T)) >> + posix_acl_fix_xattr_to_user(buffer, rc); >> #endif >> >> out_xattr: >> diff --git a/fs/cifs/cifssmb.c b/fs/cifs/cifssmb.c >> index d47197e..9dc001f 100644 >> --- a/fs/cifs/cifssmb.c >> +++ b/fs/cifs/cifssmb.c >> @@ -3337,10 +3337,25 @@ CIFSSMB_set_compression(const unsigned int xid, struct cifs_tcon *tcon, >> static void cifs_convert_ace(posix_acl_xattr_entry *ace, >> struct cifs_posix_ace *cifs_ace) >> { >> + u32 cifs_id, id = -1; >> + >> /* u8 cifs fields do not need le conversion */ >> ace->e_perm = cpu_to_le16(cifs_ace->cifs_e_perm); >> ace->e_tag = cpu_to_le16(cifs_ace->cifs_e_tag); >> - ace->e_id = cpu_to_le32(le64_to_cpu(cifs_ace->cifs_uid)); >> + switch(cifs_ace->cifs_e_tag) { >> + case ACL_USER: >> + cifs_id = le64_to_cpu(cifs_ace->cifs_uid); >> + id = from_kuid(current_user_ns(), >> + make_kuid(&init_user_ns, cifs_id)); >> + break; >> + >> + case ACL_GROUP: >> + cifs_id = le64_to_cpu(cifs_ace->cifs_uid); >> + id = from_kgid(current_user_ns(), >> + make_kgid(&init_user_ns, cifs_id)); >> + break; >> + } >> + ace->e_id = cpu_to_le32(id); >> /* >> cifs_dbg(FYI, "perm %d tag %d id %d\n", >> ace->e_perm, ace->e_tag, ace->e_id); >> @@ -3408,21 +3423,29 @@ static int cifs_copy_posix_acl(char *trgt, char *src, const int buflen, >> static __u16 convert_ace_to_cifs_ace(struct cifs_posix_ace *cifs_ace, >> const posix_acl_xattr_entry *local_ace) >> { >> - __u16 rc = 0; /* 0 = ACL converted ok */ >> + u32 cifs_id = -1, id; >> >> cifs_ace->cifs_e_perm = le16_to_cpu(local_ace->e_perm); >> cifs_ace->cifs_e_tag = le16_to_cpu(local_ace->e_tag); >> - /* BB is there a better way to handle the large uid? */ >> - if (local_ace->e_id == cpu_to_le32(-1)) { >> - /* Probably no need to le convert -1 on any arch but can not hurt */ >> - cifs_ace->cifs_uid = cpu_to_le64(-1); >> - } else >> - cifs_ace->cifs_uid = cpu_to_le64(le32_to_cpu(local_ace->e_id)); >> + switch(cifs_ace->cifs_e_tag) { >> + case ACL_USER: >> + id = le32_to_cpu(local_ace->e_id); >> + cifs_id = from_kuid(&init_user_ns, >> + make_kuid(current_user_ns(), id)); >> + break; >> + >> + case ACL_GROUP: >> + id = le32_to_cpu(local_ace->e_id); >> + cifs_id = from_kgid(&init_user_ns, >> + make_kgid(current_user_ns(), id)); >> + break; >> + } >> + cifs_ace->cifs_uid = cpu_to_le64(cifs_id); >> /* >> cifs_dbg(FYI, "perm %d tag %d id %d\n", >> ace->e_perm, ace->e_tag, ace->e_id); >> */ >> - return rc; >> + return 0; >> } >> >> /* Convert ACL from local Linux POSIX xattr to CIFS POSIX ACL wire format */ >> diff --git a/fs/posix_acl.c b/fs/posix_acl.c >> index 2c60f17..fca281c 100644 >> --- a/fs/posix_acl.c >> +++ b/fs/posix_acl.c >> @@ -627,64 +627,6 @@ no_mem: >> EXPORT_SYMBOL_GPL(posix_acl_create); >> >> /* >> - * Fix up the uids and gids in posix acl extended attributes in place. >> - */ >> -static void posix_acl_fix_xattr_userns( >> - struct user_namespace *to, struct user_namespace *from, >> - void *value, size_t size) >> -{ >> - posix_acl_xattr_header *header = (posix_acl_xattr_header *)value; >> - posix_acl_xattr_entry *entry = (posix_acl_xattr_entry *)(header+1), *end; >> - int count; >> - kuid_t uid; >> - kgid_t gid; >> - >> - if (!value) >> - return; >> - if (size < sizeof(posix_acl_xattr_header)) >> - return; >> - if (header->a_version != cpu_to_le32(POSIX_ACL_XATTR_VERSION)) >> - return; >> - >> - count = posix_acl_xattr_count(size); >> - if (count < 0) >> - return; >> - if (count == 0) >> - return; >> - >> - for (end = entry + count; entry != end; entry++) { >> - switch(le16_to_cpu(entry->e_tag)) { >> - case ACL_USER: >> - uid = make_kuid(from, le32_to_cpu(entry->e_id)); >> - entry->e_id = cpu_to_le32(from_kuid(to, uid)); >> - break; >> - case ACL_GROUP: >> - gid = make_kgid(from, le32_to_cpu(entry->e_id)); >> - entry->e_id = cpu_to_le32(from_kgid(to, gid)); >> - break; >> - default: >> - break; >> - } >> - } >> -} >> - >> -void posix_acl_fix_xattr_from_user(void *value, size_t size) >> -{ >> - struct user_namespace *user_ns = current_user_ns(); >> - if (user_ns == &init_user_ns) >> - return; >> - posix_acl_fix_xattr_userns(&init_user_ns, user_ns, value, size); >> -} >> - >> -void posix_acl_fix_xattr_to_user(void *value, size_t size) >> -{ >> - struct user_namespace *user_ns = current_user_ns(); >> - if (user_ns == &init_user_ns) >> - return; >> - posix_acl_fix_xattr_userns(user_ns, &init_user_ns, value, size); >> -} >> - >> -/* >> * Convert from extended attribute to in-memory representation. >> */ >> struct posix_acl * >> @@ -814,7 +756,7 @@ posix_acl_xattr_get(const struct xattr_handler *handler, >> if (acl == NULL) >> return -ENODATA; >> >> - error = posix_acl_to_xattr(&init_user_ns, acl, value, size); >> + error = posix_acl_to_xattr(current_user_ns(), acl, value, size); >> posix_acl_release(acl); >> >> return error; >> @@ -840,7 +782,7 @@ posix_acl_xattr_set(const struct xattr_handler *handler, >> return -EPERM; >> >> if (value) { >> - acl = posix_acl_from_xattr(&init_user_ns, value, size); >> + acl = posix_acl_from_xattr(current_user_ns(), value, size); >> if (IS_ERR(acl)) >> return PTR_ERR(acl); >> >> diff --git a/fs/xattr.c b/fs/xattr.c >> index b11945e..b648b05 100644 >> --- a/fs/xattr.c >> +++ b/fs/xattr.c >> @@ -20,7 +20,6 @@ >> #include >> #include >> #include >> -#include >> >> #include >> >> @@ -329,9 +328,6 @@ setxattr(struct dentry *d, const char __user *name, const void __user *value, >> error = -EFAULT; >> goto out; >> } >> - if ((strcmp(kname, XATTR_NAME_POSIX_ACL_ACCESS) == 0) || >> - (strcmp(kname, XATTR_NAME_POSIX_ACL_DEFAULT) == 0)) >> - posix_acl_fix_xattr_from_user(kvalue, size); >> } >> >> error = vfs_setxattr(d, kname, kvalue, size, flags); >> @@ -426,9 +422,6 @@ getxattr(struct dentry *d, const char __user *name, void __user *value, >> >> error = vfs_getxattr(d, kname, kvalue, size); >> if (error > 0) { >> - if ((strcmp(kname, XATTR_NAME_POSIX_ACL_ACCESS) == 0) || >> - (strcmp(kname, XATTR_NAME_POSIX_ACL_DEFAULT) == 0)) >> - posix_acl_fix_xattr_to_user(kvalue, size); >> if (size && copy_to_user(value, kvalue, error)) >> error = -EFAULT; >> } else if (error == -ERANGE && size >= XATTR_SIZE_MAX) { >> diff --git a/include/linux/posix_acl_xattr.h b/include/linux/posix_acl_xattr.h >> index e5e8ec4..9789aba 100644 >> --- a/include/linux/posix_acl_xattr.h >> +++ b/include/linux/posix_acl_xattr.h >> @@ -48,18 +48,6 @@ posix_acl_xattr_count(size_t size) >> return size / sizeof(posix_acl_xattr_entry); >> } >> >> -#ifdef CONFIG_FS_POSIX_ACL >> -void posix_acl_fix_xattr_from_user(void *value, size_t size); >> -void posix_acl_fix_xattr_to_user(void *value, size_t size); >> -#else >> -static inline void posix_acl_fix_xattr_from_user(void *value, size_t size) >> -{ >> -} >> -static inline void posix_acl_fix_xattr_to_user(void *value, size_t size) >> -{ >> -} >> -#endif >> - >> struct posix_acl *posix_acl_from_xattr(struct user_namespace *user_ns, >> const void *value, size_t size); >> int posix_acl_to_xattr(struct user_namespace *user_ns, >> -- >> 2.5.5 >> >> _______________________________________________ >> lustre-devel mailing list >> lustre-devel at lists.lustre.org >> http://lists.lustre.org/listinfo.cgi/lustre-devel-lustre.org >> >_______________________________________________ >lustre-devel mailing list >lustre-devel at lists.lustre.org >http://lists.lustre.org/listinfo.cgi/lustre-devel-lustre.org > From jsimmons at infradead.org Tue May 24 20:35:27 2016 From: jsimmons at infradead.org (James Simmons) Date: Tue, 24 May 2016 21:35:27 +0100 (BST) Subject: [lustre-devel] [PATCH] posix acls: Move namespace conversion into filesystem / xattr handlers In-Reply-To: References: <1464008989-3812-1-git-send-email-agruenba@redhat.com> Message-ID: > On 2016/05/23, 15:06, "James Simmons" wrote: > > > >> Currently, getxattr() and setxattr() check for the xattr names > >> "system.posix_acl_{access,default}" and perform in-place UID / GID > >> namespace mappings in the xattr values. Filesystems then again check for > >> the same xattr names to handle those attributes, almost always using the > >> standard posix_acl_{access,default}_xattr_handler handlers. This is > >> unnecessary overhead; move the namespace conversion into the xattr > >> handlers instead. > >> > >> For filesystems that use the POSIX ACL xattr handlers, no change is > >> required. Filesystems that don't use those handlers (cifs and lustre) > >> need to take care of the namespace mapping themselves now. > >> > >> The only user left of the posix_acl_fix_xattr_{from,to}_user helpers is > >> lustre, so this patch moves them into lustre. > >> > >> Signed-off-by: Andreas Gruenbacher > >> --- > >> > >> I'm reasonably confident about the core and cifs changes in this patch. > >> The lustre code is pretty weird though, so could I please get a careful > >> review on the changes there? > > > >Nak on the Lustre changes. POSIX ACLs are also handled in mdc_request.c. > >Besides POSIX ACLs lustre has created a extended ACL as well that is > >grouped in with posix ACL handling. This extended ACL is like POSIX acls > >except it also contains the state (deleted, modified, ...) of the ACL. > >Besides normal local access control list handling Lustre manages remote > >access control list handling. You can read about this handling is in > >llite_rmtacl.c. This code was written long before I became involved with > >lustre so the exact details are fuzzy to me. > > James, > the remote ACL code never found any usage in the field and can be > deleted. Please see http://review.whamcloud.com/19789 for details. That is a huge cleanup which will make Greg very happy. The tools and test are going to be updated so the landing has to be just right so we don't have a flood of test failures. Gruenbacher with the 19789 patch Dilger pointed out Lustre's POSIX ACL code just becomes ordinary which means we can use the default POSIS xattr handler. You wouldn't have to keep posix_acl.c around with these changes. > Cheers, Andreas > > > The guts of this are handled is located at: > > > >drivers/staging/lustre/lustre/obdclass/acl.c > > > >P.S > > > > A you probably have noticed Lustre has had an uptick in development > >which means the code is now changing all the time in staging. How should > >we handle the changes you are working in your own trees verses what is > >happing in staging. For example I'm looking at moving lustre to the > >xattr_handles. Should I push it to you and wait until it works it way > >into Greg's tree. What do the merge schedules look like. Lastly the > >a_refcount for the POSIX acl changed with your xattr updates which now > >causes lustre to crash :-( > > > >> Thanks, > >> Andreas > >> > >> drivers/staging/lustre/lustre/llite/Makefile | 1 + > >> .../staging/lustre/lustre/llite/llite_internal.h | 3 ++ > >> drivers/staging/lustre/lustre/llite/posix_acl.c | 62 ++++++++++++++++++++++ > >> drivers/staging/lustre/lustre/llite/xattr.c | 8 ++- > >> fs/cifs/cifssmb.c | 41 ++++++++++---- > >> fs/posix_acl.c | 62 +--------------------- > >> fs/xattr.c | 7 --- > >> include/linux/posix_acl_xattr.h | 12 ----- > >> 8 files changed, 107 insertions(+), 89 deletions(-) > >> create mode 100644 drivers/staging/lustre/lustre/llite/posix_acl.c > >> > >> diff --git a/drivers/staging/lustre/lustre/llite/Makefile b/drivers/staging/lustre/lustre/llite/Makefile > >> index 2ce10ff..67125f8 100644 > >> --- a/drivers/staging/lustre/lustre/llite/Makefile > >> +++ b/drivers/staging/lustre/lustre/llite/Makefile > >> @@ -7,5 +7,6 @@ lustre-y := dcache.o dir.o file.o llite_close.o llite_lib.o llite_nfs.o \ > >> glimpse.o lcommon_cl.o lcommon_misc.o \ > >> vvp_dev.o vvp_page.o vvp_lock.o vvp_io.o vvp_object.o vvp_req.o \ > >> lproc_llite.o > >> +lustre-$(CONFIG_FS_POSIX_ACL) += posix_acl.o > >> > >> llite_lloop-y := lloop.o > >> diff --git a/drivers/staging/lustre/lustre/llite/llite_internal.h b/drivers/staging/lustre/lustre/llite/llite_internal.h > >> index ce1f949..d454dfb 100644 > >> --- a/drivers/staging/lustre/lustre/llite/llite_internal.h > >> +++ b/drivers/staging/lustre/lustre/llite/llite_internal.h > >> @@ -1402,4 +1402,7 @@ int cl_local_size(struct inode *inode); > >> __u64 cl_fid_build_ino(const struct lu_fid *fid, int api32); > >> __u32 cl_fid_build_gen(const struct lu_fid *fid); > >> > >> +void posix_acl_fix_xattr_from_user(void *value, size_t size); > >> +void posix_acl_fix_xattr_to_user(void *value, size_t size); > >> + > >> #endif /* LLITE_INTERNAL_H */ > >> diff --git a/drivers/staging/lustre/lustre/llite/posix_acl.c b/drivers/staging/lustre/lustre/llite/posix_acl.c > >> new file mode 100644 > >> index 0000000..4fabd0f > >> --- /dev/null > >> +++ b/drivers/staging/lustre/lustre/llite/posix_acl.c > >> @@ -0,0 +1,62 @@ > >> +#include > >> +#include > >> +#include > >> +#include > >> + > >> +/* > >> + * Fix up the uids and gids in posix acl extended attributes in place. > >> + */ > >> +static void posix_acl_fix_xattr_userns( > >> + struct user_namespace *to, struct user_namespace *from, > >> + void *value, size_t size) > >> +{ > >> + posix_acl_xattr_header *header = (posix_acl_xattr_header *)value; > >> + posix_acl_xattr_entry *entry = (posix_acl_xattr_entry *)(header+1), *end; > >> + int count; > >> + kuid_t uid; > >> + kgid_t gid; > >> + > >> + if (!value) > >> + return; > >> + if (size < sizeof(posix_acl_xattr_header)) > >> + return; > >> + if (header->a_version != cpu_to_le32(POSIX_ACL_XATTR_VERSION)) > >> + return; > >> + > >> + count = posix_acl_xattr_count(size); > >> + if (count < 0) > >> + return; > >> + if (count == 0) > >> + return; > >> + > >> + for (end = entry + count; entry != end; entry++) { > >> + switch(le16_to_cpu(entry->e_tag)) { > >> + case ACL_USER: > >> + uid = make_kuid(from, le32_to_cpu(entry->e_id)); > >> + entry->e_id = cpu_to_le32(from_kuid(to, uid)); > >> + break; > >> + case ACL_GROUP: > >> + gid = make_kgid(from, le32_to_cpu(entry->e_id)); > >> + entry->e_id = cpu_to_le32(from_kgid(to, gid)); > >> + break; > >> + default: > >> + break; > >> + } > >> + } > >> +} > >> + > >> +void posix_acl_fix_xattr_from_user(void *value, size_t size) > >> +{ > >> + struct user_namespace *user_ns = current_user_ns(); > >> + if (user_ns == &init_user_ns) > >> + return; > >> + posix_acl_fix_xattr_userns(&init_user_ns, user_ns, value, size); > >> +} > >> + > >> +void posix_acl_fix_xattr_to_user(void *value, size_t size) > >> +{ > >> + struct user_namespace *user_ns = current_user_ns(); > >> + if (user_ns == &init_user_ns) > >> + return; > >> + posix_acl_fix_xattr_userns(user_ns, &init_user_ns, value, size); > >> +} > >> diff --git a/drivers/staging/lustre/lustre/llite/xattr.c b/drivers/staging/lustre/lustre/llite/xattr.c > >> index ed4de04..c721b44 100644 > >> --- a/drivers/staging/lustre/lustre/llite/xattr.c > >> +++ b/drivers/staging/lustre/lustre/llite/xattr.c > >> @@ -144,6 +144,9 @@ int ll_setxattr_common(struct inode *inode, const char *name, > >> return -EOPNOTSUPP; > >> > >> #ifdef CONFIG_FS_POSIX_ACL > >> + if (xattr_type == XATTR_ACL_ACCESS_T || > >> + xattr_type == XATTR_ACL_DEFAULT_T) > >> + posix_acl_fix_xattr_from_user((void *)value, size); > >> if (sbi->ll_flags & LL_SBI_RMT_CLIENT && > >> (xattr_type == XATTR_ACL_ACCESS_T || > >> xattr_type == XATTR_ACL_DEFAULT_T)) { > >> @@ -348,7 +351,7 @@ int ll_getxattr_common(struct inode *inode, const char *name, > >> if (!acl) > >> return -ENODATA; > >> > >> - rc = posix_acl_to_xattr(&init_user_ns, acl, buffer, size); > >> + rc = posix_acl_to_xattr(current_user_ns(), acl, buffer, size); > >> posix_acl_release(acl); > >> return rc; > >> } > >> @@ -436,6 +439,9 @@ getxattr_nocache: > >> goto out; > >> } > >> } > >> + if (rc >= 0 && (xattr_type == XATTR_ACL_ACCESS_T || > >> + xattr_type == XATTR_ACL_DEFAULT_T)) > >> + posix_acl_fix_xattr_to_user(buffer, rc); > >> #endif > >> > >> out_xattr: > >> diff --git a/fs/cifs/cifssmb.c b/fs/cifs/cifssmb.c > >> index d47197e..9dc001f 100644 > >> --- a/fs/cifs/cifssmb.c > >> +++ b/fs/cifs/cifssmb.c > >> @@ -3337,10 +3337,25 @@ CIFSSMB_set_compression(const unsigned int xid, struct cifs_tcon *tcon, > >> static void cifs_convert_ace(posix_acl_xattr_entry *ace, > >> struct cifs_posix_ace *cifs_ace) > >> { > >> + u32 cifs_id, id = -1; > >> + > >> /* u8 cifs fields do not need le conversion */ > >> ace->e_perm = cpu_to_le16(cifs_ace->cifs_e_perm); > >> ace->e_tag = cpu_to_le16(cifs_ace->cifs_e_tag); > >> - ace->e_id = cpu_to_le32(le64_to_cpu(cifs_ace->cifs_uid)); > >> + switch(cifs_ace->cifs_e_tag) { > >> + case ACL_USER: > >> + cifs_id = le64_to_cpu(cifs_ace->cifs_uid); > >> + id = from_kuid(current_user_ns(), > >> + make_kuid(&init_user_ns, cifs_id)); > >> + break; > >> + > >> + case ACL_GROUP: > >> + cifs_id = le64_to_cpu(cifs_ace->cifs_uid); > >> + id = from_kgid(current_user_ns(), > >> + make_kgid(&init_user_ns, cifs_id)); > >> + break; > >> + } > >> + ace->e_id = cpu_to_le32(id); > >> /* > >> cifs_dbg(FYI, "perm %d tag %d id %d\n", > >> ace->e_perm, ace->e_tag, ace->e_id); > >> @@ -3408,21 +3423,29 @@ static int cifs_copy_posix_acl(char *trgt, char *src, const int buflen, > >> static __u16 convert_ace_to_cifs_ace(struct cifs_posix_ace *cifs_ace, > >> const posix_acl_xattr_entry *local_ace) > >> { > >> - __u16 rc = 0; /* 0 = ACL converted ok */ > >> + u32 cifs_id = -1, id; > >> > >> cifs_ace->cifs_e_perm = le16_to_cpu(local_ace->e_perm); > >> cifs_ace->cifs_e_tag = le16_to_cpu(local_ace->e_tag); > >> - /* BB is there a better way to handle the large uid? */ > >> - if (local_ace->e_id == cpu_to_le32(-1)) { > >> - /* Probably no need to le convert -1 on any arch but can not hurt */ > >> - cifs_ace->cifs_uid = cpu_to_le64(-1); > >> - } else > >> - cifs_ace->cifs_uid = cpu_to_le64(le32_to_cpu(local_ace->e_id)); > >> + switch(cifs_ace->cifs_e_tag) { > >> + case ACL_USER: > >> + id = le32_to_cpu(local_ace->e_id); > >> + cifs_id = from_kuid(&init_user_ns, > >> + make_kuid(current_user_ns(), id)); > >> + break; > >> + > >> + case ACL_GROUP: > >> + id = le32_to_cpu(local_ace->e_id); > >> + cifs_id = from_kgid(&init_user_ns, > >> + make_kgid(current_user_ns(), id)); > >> + break; > >> + } > >> + cifs_ace->cifs_uid = cpu_to_le64(cifs_id); > >> /* > >> cifs_dbg(FYI, "perm %d tag %d id %d\n", > >> ace->e_perm, ace->e_tag, ace->e_id); > >> */ > >> - return rc; > >> + return 0; > >> } > >> > >> /* Convert ACL from local Linux POSIX xattr to CIFS POSIX ACL wire format */ > >> diff --git a/fs/posix_acl.c b/fs/posix_acl.c > >> index 2c60f17..fca281c 100644 > >> --- a/fs/posix_acl.c > >> +++ b/fs/posix_acl.c > >> @@ -627,64 +627,6 @@ no_mem: > >> EXPORT_SYMBOL_GPL(posix_acl_create); > >> > >> /* > >> - * Fix up the uids and gids in posix acl extended attributes in place. > >> - */ > >> -static void posix_acl_fix_xattr_userns( > >> - struct user_namespace *to, struct user_namespace *from, > >> - void *value, size_t size) > >> -{ > >> - posix_acl_xattr_header *header = (posix_acl_xattr_header *)value; > >> - posix_acl_xattr_entry *entry = (posix_acl_xattr_entry *)(header+1), *end; > >> - int count; > >> - kuid_t uid; > >> - kgid_t gid; > >> - > >> - if (!value) > >> - return; > >> - if (size < sizeof(posix_acl_xattr_header)) > >> - return; > >> - if (header->a_version != cpu_to_le32(POSIX_ACL_XATTR_VERSION)) > >> - return; > >> - > >> - count = posix_acl_xattr_count(size); > >> - if (count < 0) > >> - return; > >> - if (count == 0) > >> - return; > >> - > >> - for (end = entry + count; entry != end; entry++) { > >> - switch(le16_to_cpu(entry->e_tag)) { > >> - case ACL_USER: > >> - uid = make_kuid(from, le32_to_cpu(entry->e_id)); > >> - entry->e_id = cpu_to_le32(from_kuid(to, uid)); > >> - break; > >> - case ACL_GROUP: > >> - gid = make_kgid(from, le32_to_cpu(entry->e_id)); > >> - entry->e_id = cpu_to_le32(from_kgid(to, gid)); > >> - break; > >> - default: > >> - break; > >> - } > >> - } > >> -} > >> - > >> -void posix_acl_fix_xattr_from_user(void *value, size_t size) > >> -{ > >> - struct user_namespace *user_ns = current_user_ns(); > >> - if (user_ns == &init_user_ns) > >> - return; > >> - posix_acl_fix_xattr_userns(&init_user_ns, user_ns, value, size); > >> -} > >> - > >> -void posix_acl_fix_xattr_to_user(void *value, size_t size) > >> -{ > >> - struct user_namespace *user_ns = current_user_ns(); > >> - if (user_ns == &init_user_ns) > >> - return; > >> - posix_acl_fix_xattr_userns(user_ns, &init_user_ns, value, size); > >> -} > >> - > >> -/* > >> * Convert from extended attribute to in-memory representation. > >> */ > >> struct posix_acl * > >> @@ -814,7 +756,7 @@ posix_acl_xattr_get(const struct xattr_handler *handler, > >> if (acl == NULL) > >> return -ENODATA; > >> > >> - error = posix_acl_to_xattr(&init_user_ns, acl, value, size); > >> + error = posix_acl_to_xattr(current_user_ns(), acl, value, size); > >> posix_acl_release(acl); > >> > >> return error; > >> @@ -840,7 +782,7 @@ posix_acl_xattr_set(const struct xattr_handler *handler, > >> return -EPERM; > >> > >> if (value) { > >> - acl = posix_acl_from_xattr(&init_user_ns, value, size); > >> + acl = posix_acl_from_xattr(current_user_ns(), value, size); > >> if (IS_ERR(acl)) > >> return PTR_ERR(acl); > >> > >> diff --git a/fs/xattr.c b/fs/xattr.c > >> index b11945e..b648b05 100644 > >> --- a/fs/xattr.c > >> +++ b/fs/xattr.c > >> @@ -20,7 +20,6 @@ > >> #include > >> #include > >> #include > >> -#include > >> > >> #include > >> > >> @@ -329,9 +328,6 @@ setxattr(struct dentry *d, const char __user *name, const void __user *value, > >> error = -EFAULT; > >> goto out; > >> } > >> - if ((strcmp(kname, XATTR_NAME_POSIX_ACL_ACCESS) == 0) || > >> - (strcmp(kname, XATTR_NAME_POSIX_ACL_DEFAULT) == 0)) > >> - posix_acl_fix_xattr_from_user(kvalue, size); > >> } > >> > >> error = vfs_setxattr(d, kname, kvalue, size, flags); > >> @@ -426,9 +422,6 @@ getxattr(struct dentry *d, const char __user *name, void __user *value, > >> > >> error = vfs_getxattr(d, kname, kvalue, size); > >> if (error > 0) { > >> - if ((strcmp(kname, XATTR_NAME_POSIX_ACL_ACCESS) == 0) || > >> - (strcmp(kname, XATTR_NAME_POSIX_ACL_DEFAULT) == 0)) > >> - posix_acl_fix_xattr_to_user(kvalue, size); > >> if (size && copy_to_user(value, kvalue, error)) > >> error = -EFAULT; > >> } else if (error == -ERANGE && size >= XATTR_SIZE_MAX) { > >> diff --git a/include/linux/posix_acl_xattr.h b/include/linux/posix_acl_xattr.h > >> index e5e8ec4..9789aba 100644 > >> --- a/include/linux/posix_acl_xattr.h > >> +++ b/include/linux/posix_acl_xattr.h > >> @@ -48,18 +48,6 @@ posix_acl_xattr_count(size_t size) > >> return size / sizeof(posix_acl_xattr_entry); > >> } > >> > >> -#ifdef CONFIG_FS_POSIX_ACL > >> -void posix_acl_fix_xattr_from_user(void *value, size_t size); > >> -void posix_acl_fix_xattr_to_user(void *value, size_t size); > >> -#else > >> -static inline void posix_acl_fix_xattr_from_user(void *value, size_t size) > >> -{ > >> -} > >> -static inline void posix_acl_fix_xattr_to_user(void *value, size_t size) > >> -{ > >> -} > >> -#endif > >> - > >> struct posix_acl *posix_acl_from_xattr(struct user_namespace *user_ns, > >> const void *value, size_t size); > >> int posix_acl_to_xattr(struct user_namespace *user_ns, > >> -- > >> 2.5.5 > >> > >> _______________________________________________ > >> lustre-devel mailing list > >> lustre-devel at lists.lustre.org > >> http://lists.lustre.org/listinfo.cgi/lustre-devel-lustre.org > >> > >_______________________________________________ > >lustre-devel mailing list > >lustre-devel at lists.lustre.org > >http://lists.lustre.org/listinfo.cgi/lustre-devel-lustre.org > > > > > > From jsimmons at infradead.org Tue May 24 20:38:19 2016 From: jsimmons at infradead.org (James Simmons) Date: Tue, 24 May 2016 21:38:19 +0100 (BST) Subject: [lustre-devel] [PATCH] staging: lustre: llite: drop acl from cache In-Reply-To: References: <1464050108-28015-1-git-send-email-jsimmons@infradead.org> Message-ID: > 2016-05-24 2:35 GMT+02:00 James Simmons : > > Commit b8a7a3a6 change get_acl() for posix xattr to always cache > > the ACL which increases the reference count. That reference count > > can be reduced by have ll_get_acl() call forget_cached_acl() which > > it wasn't. When an inode gets deleted by Lustre the POSIX ACL > > reference count is tested to ensure its 1 and if not produces an error. > > Lustre shouldn't assume that the VFS immediately drops the reference > it is passed. Please remove that check as well. The piece of code in question from ll_delete_inode() is #ifdef CONFIG_FS_POSIX_ACL else if (lli->lli_posix_acl) { LASSERT(atomic_read(&lli->lli_posix_acl->a_refcount) == 1); LASSERT(!lli->lli_remote_perms); posix_acl_release(lli->lli_posix_acl); lli->lli_posix_acl = NULL; } #endif So we want to prevent a leak should I do a while (atomic_read(&lli->lli_posix_acl->a_refcount)) posix_acl_release(lli->lli_posix_acl); lli->lli_posix_acl = NULL; Or does the VFS do this cleanup for us? > > Since forget_cached_acl() was not called Lustre started to complain. > > This patch changes ll_get_acl() to call forget_cached_acl(). > > > > Signed-off-by: James Simmons > > Reviewed-by: Andreas Gruenbacher > > > --- > > drivers/staging/lustre/lustre/llite/file.c | 1 + > > 1 files changed, 1 insertions(+), 0 deletions(-) > > > > diff --git a/drivers/staging/lustre/lustre/llite/file.c b/drivers/staging/lustre/lustre/llite/file.c > > index f47f2ac..0191945 100644 > > --- a/drivers/staging/lustre/lustre/llite/file.c > > +++ b/drivers/staging/lustre/lustre/llite/file.c > > @@ -3124,6 +3124,7 @@ struct posix_acl *ll_get_acl(struct inode *inode, int type) > > spin_lock(&lli->lli_lock); > > /* VFS' acl_permission_check->check_acl will release the refcount */ > > acl = posix_acl_dup(lli->lli_posix_acl); > > + forget_cached_acl(inode, type); > > spin_unlock(&lli->lli_lock); > > > > return acl; > > -- > > 1.7.1 > > > From adilger at dilger.ca Tue May 24 20:47:57 2016 From: adilger at dilger.ca (Andreas Dilger) Date: Tue, 24 May 2016 14:47:57 -0600 Subject: [lustre-devel] [PATCH] posix acls: Move namespace conversion into filesystem / xattr handlers In-Reply-To: <20160524084724.GA8121@infradead.org> References: <1464008989-3812-1-git-send-email-agruenba@redhat.com> <20160524084724.GA8121@infradead.org> Message-ID: On May 24, 2016, at 2:47 AM, Christoph Hellwig wrote: > > On Mon, May 23, 2016 at 03:09:49PM +0200, Andreas Gruenbacher wrote: >> I'm reasonably confident about the core and cifs changes in this patch. >> The lustre code is pretty weird though, so could I please get a careful >> review on the changes there? > > Lustre is in the staging tree because it's an unrescuable piece of junk. > Please just ignore it and move the proper parts of the kernel forward. Don't be shy Christoph, tell us how you really feel about Lustre... :-) Fortunately, that "piece of junk" is working fine and is running on most of the supercomputers in the world (9 of the top 10, 70%+ of the top 100, ...) with several sites having 50PB+ of storage, and 1TB/s+ to 20000+ clients, in a single coherent POSIX filesystem. It's true that the staging client isn't getting cleaned up as fast as anyone likes, but that is because it is complex software in use at many sites for many years and we can't just stop to completely rewrite the code and hope it still works. Even XFS took many years to get rid of the IRIX wrapping layers, and still does a lot of things outside of the VFS/VM (stats, memory allocation, IO submission, etc.) because the VFS/VM doesn't work the way XFS wants it to (or vice versa). This is not a slight against XFS, just a sign that non-trivial filesystems take a long time to change while maintaining existing functionality. This is mainly a difference in how code is added to the kernel today vs. how it was added back then. Cheers, Andreas -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: Message signed with OpenPGP using GPGMail URL: From agruenba at redhat.com Tue May 24 22:08:33 2016 From: agruenba at redhat.com (Andreas Gruenbacher) Date: Wed, 25 May 2016 00:08:33 +0200 Subject: [lustre-devel] [PATCH] staging: lustre: llite: drop acl from cache In-Reply-To: References: <1464050108-28015-1-git-send-email-jsimmons@infradead.org> Message-ID: On Tue, May 24, 2016 at 10:38 PM, James Simmons wrote: >> 2016-05-24 2:35 GMT+02:00 James Simmons : >> > Commit b8a7a3a6 change get_acl() for posix xattr to always cache >> > the ACL which increases the reference count. That reference count >> > can be reduced by have ll_get_acl() call forget_cached_acl() which >> > it wasn't. When an inode gets deleted by Lustre the POSIX ACL >> > reference count is tested to ensure its 1 and if not produces an error. >> >> Lustre shouldn't assume that the VFS immediately drops the reference >> it is passed. Please remove that check as well. > > The piece of code in question from ll_delete_inode() is > > #ifdef CONFIG_FS_POSIX_ACL > else if (lli->lli_posix_acl) { > LASSERT(atomic_read(&lli->lli_posix_acl->a_refcount) == > 1); > LASSERT(!lli->lli_remote_perms); > posix_acl_release(lli->lli_posix_acl); > lli->lli_posix_acl = NULL; > } > #endif > > So we want to prevent a leak should I do a > > while (atomic_read(&lli->lli_posix_acl->a_refcount)) > posix_acl_release(lli->lli_posix_acl); > lli->lli_posix_acl = NULL; > > Or does the VFS do this cleanup for us? This conversation is unreal. Just remove the misguided assert and you're good. After that, please have someone explain basic reference counting to you. Thanks, Andreas From agruenba at redhat.com Tue May 24 22:40:54 2016 From: agruenba at redhat.com (Andreas Gruenbacher) Date: Wed, 25 May 2016 00:40:54 +0200 Subject: [lustre-devel] [PATCH] posix acls: Move namespace conversion into filesystem / xattr handlers In-Reply-To: References: <1464008989-3812-1-git-send-email-agruenba@redhat.com> Message-ID: On Tue, May 24, 2016 at 10:35 PM, James Simmons wrote: > That is a huge cleanup which will make Greg very happy. The tools and test > are going to be updated so the landing has to be just right so we > don't have a flood of test failures. Gruenbacher with the 19789 patch > Dilger pointed out Lustre's POSIX ACL code just becomes ordinary which > means we can use the default POSIS xattr handler. You wouldn't have to > keep posix_acl.c around with these changes. Change http://review.whamcloud.com/19789 doesn't switch Lustre over to using xattr handlers. As long as Lustre doesn't use those (i.e., use posix_acl_access_xattr_handler and posix_acl_default_xattr_handler), it will still need the uid/gid namespace conversion code in ll_getxattr and ll_setxattr for this patch to be correct. The merge conflict between this patch and http://review.whamcloud.com/19789 is easily resolved though. Andreas From agruenba at redhat.com Thu May 26 11:36:23 2016 From: agruenba at redhat.com (Andreas Gruenbacher) Date: Thu, 26 May 2016 13:36:23 +0200 Subject: [lustre-devel] [PATCH] posix acls: Move namespace conversion into filesystem / xattr handlers In-Reply-To: <87h9dllnq2.fsf@x220.int.ebiederm.org> References: <1464008989-3812-1-git-send-email-agruenba@redhat.com> <20160524153949.GA2830@dztty.fritz.box> <87h9dllnq2.fsf@x220.int.ebiederm.org> Message-ID: On Thu, May 26, 2016 at 1:30 AM, Eric W. Biederman wrote: > Andreas Gruenbacher writes: > >> On Tue, May 24, 2016 at 5:41 PM, Djalal Harouni wrote: >>> On Mon, May 23, 2016 at 03:09:49PM +0200, Andreas Gruenbacher wrote: >>>> Currently, getxattr() and setxattr() check for the xattr names >>>> "system.posix_acl_{access,default}" and perform in-place UID / GID >>>> namespace mappings in the xattr values. Filesystems then again check for >>>> the same xattr names to handle those attributes, almost always using the >>>> standard posix_acl_{access,default}_xattr_handler handlers. This is >>>> unnecessary overhead; move the namespace conversion into the xattr >>>> handlers instead. >>> >>> Please, are you sure that the changes in posix_acl_xattr_get() and >>> posix_acl_xattr_set() are safe ? you are reading into current user >>> namespace, from a first view this is not safe unless I'm missing >>> something... they should map into init_user_ns... >> >> Yes, moving the namespace conversion from the VFS into those functions >> so that we don't have to check for those attributes and parse them >> twice is exactly the point of this patch. > > In general I am in favor of cleaning up the xattr and acl code in the > kernel. However I am not certain that your changes succeed in getting > us where we want to go. > > My feel is that what we want to do is to update the cached acl when we > write it from userspace, to update the disk with the new acl when the > inode is sync'd to disk, and let the vfs handle the translation from > the cached posix acl to the vfs getxattr and setxattr system calls. > In the long term anything else is madness. > > Currently posix acl reads and updates bypass the vfs acl cache for the > inode and that just looks wrong. Not all filesystems cache acls in the inode (i_acl and i_default_acl), so there has to be some way to "bypass the cache" if you want to put it that way. All normal filesystems that cache ACLs locally implement the get_acl and set_acl inode operations. They put posix_acl_access_xattr_handler and posix_acl_default_xattr_handler into s_xattr and use the generic_{get,set,remove}xattr inode operations to hook things up appropriately. The xattr handlers convert to/from xattrs and struct posix_acl and call the get_acl and set_acl inode operations. That's where the namespace conversion is supposed to happen as well; this avoids special-casing it in the VFS. The only filesystems that don't do things that way are CIFS and Lustre. Doing the appropriate namespace mapping in CIFS is easy (see this patch). So it's only Lustre that's left with the original in-place xattr conversion. Another pending cleanup (https://lwn.net/Articles/688390/) gets rid of the {get,set,remove}xattr inode operations in favor of using s_xattr directly. > The reason that fixup happens in a separate pass from everything else > today is that when I wrote posix_acl_fix_xattr_to_user and > posix_acl_xattr_from_user a number of filesystems had a very strange > structure that completely bypassed any code conversion routines and > made some strange assumptions. I don't remember which filesystems those > were but it was neither lustre, nor cifs, nor nfs that were the problem > cases when I was looking. I don't see your patch addressing that > problem so either someone has already fixed those issues or they have > been overlooked. There have been cleanups in this area before, so I assume things were fixed. In any case, I didn't blindly hack this together, I've actually checked all the filesystems. > There is a complication that is comming shortly (next merge window > unless major unfixable bugs show up between now and them). There is a > new field s_user_ns that is being introduced for filesystems to record > their owner and their default translation rules to kuids. That will > make the hard coded &init_user_ns values in your patch wrong. Okay, that conflict should be easy enough to resolve. Where's that code? >>> Please Cc the user namespace maintainers before. Thank you! >> >> Eric, Andy, anyone else? > > Serge Hallyn has a pending patch that adds a similar translation to > the security.capability xattr. Which is one more case of where caching > and translation at the VFS layer are makes sense. No. Just like with this patch, that mapping really needs to go into the appropriate xattr handler. Again, where's that code, please? Thanks, Andreas From jodi.levi at intel.com Fri May 20 17:03:36 2016 From: jodi.levi at intel.com (Levi, Jodi) Date: Fri, 20 May 2016 17:03:36 +0000 Subject: [lustre-devel] FW: Onyx Patching (Auto test outage) notification - May 20th 2016 In-Reply-To: References: Message-ID: All, Reminder that this outage has begun. We will respond again when the patching is complete and all instances are back up and running. (next Tuesday) Thank you! Jodi Levi, PMP Program Manager Intel High Performance Data Division Mobile: 303-241-1428 Email: jodi.levi at intel.com From: Jodi Levi > Date: Monday, May 9, 2016 at 11:23 AM To: HPDD-eng > Cc: "lustre-devel at lists.lustre.org" > Subject: Onyx Patching (Auto test outage) notification - May 20th 2016 All, We are planning a quarterly maintenance window for our cluster that hosts our Auto test instances, Jenkins, and Gerrit. This outage is needed to complete some updates that include security related patching. Please see the planned schedule and impacts to users below. May 20th 8am MT: Pause and drain all Onyx auto test queues (both instances) - Trevis auto test instances will continue testing May 24th 8am MT: Patching all of Onyx (expected duration 6 hours) - No new tests will be consumed during this time, but currently running tests on Trevis will continue NOTE: any tests that have not completed by 8am MT on Tuesday will be stopped (regardless of the remaining test time left) and we will restart them after the auto test instances are up Notification will be sent again prior to starting maintenance and again when the maintenance is complete and all instances are back on line and testing resumes. If you have any questions or concerns with this planned maintenance outage, please reply to Jodi Levi – jodi.levi at intel.com Thank you! Jodi Levi, PMP Program Manager Intel High Performance Data Division Mobile: 303-241-1428 Email: jodi.levi at intel.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From bauerj at iodoctors.com Sat May 21 20:25:02 2016 From: bauerj at iodoctors.com (John Bauer) Date: Sat, 21 May 2016 15:25:02 -0500 Subject: [lustre-devel] [lustre-discuss] more on lustre striping In-Reply-To: <01EE2B9D-2359-49B5-A50F-E7C2D97E6696@intel.com> References: <1ef5a267-334c-be0d-13f4-c0fab917d1bf@iodoctors.com> <38EA9F32-3869-43BA-B215-EB2E5BA62FD5@intel.com> <01EE2B9D-2359-49B5-A50F-E7C2D97E6696@intel.com> Message-ID: <44e2353d-3bef-ddd7-f15e-ad4bfdda040f@iodoctors.com> Oleg So in my simple test, the second open of the file caused the layout to be created. Indeed, a write to the original fd did fail. That complicates things considerably. Disregard the entire topic. Thanks John On 5/21/2016 3:08 PM, Drokin, Oleg wrote: > The thing is, when you open a file with no layout (the one you cteate with P_LOB_DELAY_CREATE) for write the next time - > the default layout is created just the same as it would have been created on the first open. > So if you want custom layouts - you do need to insert setstripe call between the creation and actual open for write. > > On the other hand if you open with O_LOV_DELAY_CREATE and then try to write into that fd - you will get a failure. > > > On May 21, 2016, at 4:01 PM, John Bauer wrote: > >> Andreas, >> >> Thanks for the reply. For what it's worth, extending a file that does not have layout set does work. >> >> % rm -f file.dat >> % ./no_stripe.exe file.dat >> fd=3 >> % lfs getstripe file.dat >> file.dat has no stripe info >> % date >> file.dat >> % lfs getstripe file.dat >> file.dat >> lmm_stripe_count: 1 >> lmm_stripe_size: 1048576 >> lmm_pattern: 1 >> lmm_layout_gen: 0 >> lmm_stripe_offset: 21 >> obdidx objid objid group >> 21 6143298 0x5dbd42 0 >> >> % >> The LD_PRELOAD is exactly what I am doing in my I/O library. Unfortunately, one can not intercept the open() that results from a call to fopen(). That open is hard linked to the open in libc and not satisfied by the runtime linker. This is what is driving this topic for me. I can not conveniently set the striping for a file opened with fopen() and other functions where the open is called from inside libc. I used to believe that not too many application use stdio for heavy I/O, but I have been come across several recently. >> >> John >> >> On 5/21/2016 12:51 AM, Dilger, Andreas wrote: >>> This is probably getting to be more of a topic for lustre-devel. >>> >>> There currently isn't any way to do what you ask, since (IIRC) it will cause an error for apps that try to write to the files before the layout is set. >>> >>> What you could do is to create an LD_PRELOAD library to intercept the open() calls and set O_LOV_DELAY_CREATE and set the layout explicitly for each file. This might be a win if each file needs a different layout, but since it uses two RPCs per file it would be slower than using the default layout. >>> >>> Cheers, Andreas >>> >>> On May 18, 2016, at 16:46, John Bauer wrote: >>> >>>> Since today's topic seems to be Lustre striping, I will revisit a previous line of questions I had. >>>> >>>> Andreas had put me on to O_LOV_DELAY_CREATE which I have been experimenting with. My question is : Is there a way to flag a directory with O_LOV_DELAY_CREATE so that a file created in that directory will be created with O_LOV_DELAY_CREATE also. Much like a file can inherit a directory's stripe count and stripe size, it would be convenient if a file could also inherit O_LOV_DELAY_CREATE? That way, for open()s that I can not intercept ( and thus can not set O_LOV_DELAY_CREATE in oflags) , such as those issued by fopen(), I can then get the fd with fileno() and set the striping with ioctl(fd, LL_IOC_LOV_SETSTRIPE, lum). >>>> >>>> Thanks >>>> >>>> John >>>> -- >>>> I/O Doctors, LLC >>>> 507-766-0378 >>>> >>>> bauerj at iodoctors.com >>>> _______________________________________________ >>>> lustre-discuss mailing list >>>> lustre-discuss at lists.lustre.org >>>> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org >> -- >> I/O Doctors, LLC >> 507-766-0378 >> >> bauerj at iodoctors.com >> _______________________________________________ >> lustre-discuss mailing list >> lustre-discuss at lists.lustre.org >> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org -- I/O Doctors, LLC 507-766-0378 bauerj at iodoctors.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From bauerj at iodoctors.com Sun May 22 01:56:07 2016 From: bauerj at iodoctors.com (John Bauer) Date: Sat, 21 May 2016 20:56:07 -0500 Subject: [lustre-devel] [lustre-discuss] more on lustre striping In-Reply-To: References: <1ef5a267-334c-be0d-13f4-c0fab917d1bf@iodoctors.com> <38EA9F32-3869-43BA-B215-EB2E5BA62FD5@intel.com> <01EE2B9D-2359-49B5-A50F-E7C2D97E6696@intel.com> <44e2353d-3bef-ddd7-f15e-ad4bfdda040f@iodoctors.com> Message-ID: <9ebe9f4c-d5a2-4c93-4f00-49029850cbbe@iodoctors.com> Oleg I can intercept the fopen(), but that does me no good as I can't set the O_LOV_DELAY_CREATE bit. What I can not intercept is the open() downstream of fopen(). If one examines the symbols in libc you will see there are no unsatisfied externals relating to open, which means there is nothing for the runtime linker to find concerning open's. I will have a look at the Lustre 1.8 source, but I seriously doubt that the open beneath fopen() was intercepted with LD_PRELOAD. I would love to find a way to do that. I could throw away a lot of code. Thanks, John % nm -g /lib64/libc.so.6 | grep open 0000000000033d70 T catopen 00000000003bfb80 B _dl_open_hook 00000000000b9a60 W fdopendir 000000000006b140 T fdopen@@GLIBC_2.2.5 00000000000755c0 T fmemopen 000000000006ba00 W fopen64 000000000006bb60 T fopencookie@@GLIBC_2.2.5 000000000006ba00 T fopen@@GLIBC_2.2.5 00000000000736f0 T freopen 0000000000074b50 T freopen64 00000000000ead40 T fts_open 0000000000022220 T iconv_open 000000000006b140 T _IO_fdopen@@GLIBC_2.2.5 0000000000077220 T _IO_file_fopen@@GLIBC_2.2.5 0000000000077170 T _IO_file_open 000000000006ba00 T _IO_fopen@@GLIBC_2.2.5 000000000006d1d0 T _IO_popen@@GLIBC_2.2.5 000000000006cee0 T _IO_proc_open@@GLIBC_2.2.5 0000000000130b20 T __libc_dlopen_mode 00000000000e7840 W open 00000000000e7840 W __open 00000000000ec690 T __open_2 00000000000e7840 W open64 00000000000e7840 W __open64 00000000000ec6b0 T __open64_2 00000000000e78d0 W openat 00000000000e79b0 T __openat_2 00000000000e78d0 W openat64 00000000000e79b0 W __openat64_2 00000000000f6e00 T open_by_handle_at 00000000000340b0 T __open_catalog 00000000000b9510 W opendir 00000000000f0850 T openlog 0000000000073e90 T open_memstream 00000000000731b0 T open_wmemstream 000000000006d1d0 T popen@@GLIBC_2.2.5 000000000012fbd0 W posix_openpt 00000000000e6460 T posix_spawn_file_actions_addopen % John On 5/21/2016 7:33 PM, Drokin, Oleg wrote: > btw I find it strange that you cannot intercept fopen (and in fact intercepting every library call like that is counterproductive). > > We used to have this "liblustre" library, that you an LD_PRELOAD into your application and it would work with Lustre even if you are not root and if Lustre is not mounted on that node > (and in fact even if the node is not Linux at all). That had no problems at all to intercept all sorts of opens by intercepting syscalls. > I wonder if you can intercept something deeper like sys_open or something like that? > Perhaps checkout lustre 1.8 sources (or even 2.1) and see how we did it back there? > > On May 21, 2016, at 4:25 PM, John Bauer wrote: > >> Oleg >> >> So in my simple test, the second open of the file caused the layout to be created. Indeed, a write to the original fd did fail. >> That complicates things considerably. >> >> Disregard the entire topic. >> >> Thanks >> >> John >> >> >> On 5/21/2016 3:08 PM, Drokin, Oleg wrote: >>> The thing is, when you open a file with no layout (the one you cteate with P_LOB_DELAY_CREATE) for write the next time - >>> the default layout is created just the same as it would have been created on the first open. >>> So if you want custom layouts - you do need to insert setstripe call between the creation and actual open for write. >>> >>> On the other hand if you open with O_LOV_DELAY_CREATE and then try to write into that fd - you will get a failure. >>> >>> >>> On May 21, 2016, at 4:01 PM, John Bauer wrote: >>> >>> >>>> Andreas, >>>> >>>> Thanks for the reply. For what it's worth, extending a file that does not have layout set does work. >>>> >>>> % rm -f file.dat >>>> % ./no_stripe.exe file.dat >>>> fd=3 >>>> % lfs getstripe file.dat >>>> file.dat has no stripe info >>>> % date >> file.dat >>>> % lfs getstripe file.dat >>>> file.dat >>>> lmm_stripe_count: 1 >>>> lmm_stripe_size: 1048576 >>>> lmm_pattern: 1 >>>> lmm_layout_gen: 0 >>>> lmm_stripe_offset: 21 >>>> obdidx objid objid group >>>> 21 6143298 0x5dbd42 0 >>>> >>>> % >>>> The LD_PRELOAD is exactly what I am doing in my I/O library. Unfortunately, one can not intercept the open() that results from a call to fopen(). That open is hard linked to the open in libc and not satisfied by the runtime linker. This is what is driving this topic for me. I can not conveniently set the striping for a file opened with fopen() and other functions where the open is called from inside libc. I used to believe that not too many application use stdio for heavy I/O, but I have been come across several recently. >>>> >>>> John >>>> >>>> On 5/21/2016 12:51 AM, Dilger, Andreas wrote: >>>> >>>>> This is probably getting to be more of a topic for lustre-devel. >>>>> >>>>> There currently isn't any way to do what you ask, since (IIRC) it will cause an error for apps that try to write to the files before the layout is set. >>>>> >>>>> What you could do is to create an LD_PRELOAD library to intercept the open() calls and set O_LOV_DELAY_CREATE and set the layout explicitly for each file. This might be a win if each file needs a different layout, but since it uses two RPCs per file it would be slower than using the default layout. >>>>> >>>>> Cheers, Andreas >>>>> >>>>> On May 18, 2016, at 16:46, John Bauer >>>>> >>>>> wrote: >>>>> >>>>> >>>>>> Since today's topic seems to be Lustre striping, I will revisit a previous line of questions I had. >>>>>> >>>>>> Andreas had put me on to O_LOV_DELAY_CREATE which I have been experimenting with. My question is : Is there a way to flag a directory with O_LOV_DELAY_CREATE so that a file created in that directory will be created with O_LOV_DELAY_CREATE also. Much like a file can inherit a directory's stripe count and stripe size, it would be convenient if a file could also inherit O_LOV_DELAY_CREATE? That way, for open()s that I can not intercept ( and thus can not set O_LOV_DELAY_CREATE in oflags) , such as those issued by fopen(), I can then get the fd with fileno() and set the striping with ioctl(fd, LL_IOC_LOV_SETSTRIPE, lum). >>>>>> >>>>>> Thanks >>>>>> >>>>>> John >>>>>> -- >>>>>> I/O Doctors, LLC >>>>>> 507-766-0378 >>>>>> >>>>>> >>>>>> bauerj at iodoctors.com >>>>>> >>>>>> _______________________________________________ >>>>>> lustre-discuss mailing list >>>>>> >>>>>> lustre-discuss at lists.lustre.org >>>>>> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org >>>> -- >>>> I/O Doctors, LLC >>>> 507-766-0378 >>>> >>>> >>>> bauerj at iodoctors.com >>>> >>>> _______________________________________________ >>>> lustre-discuss mailing list >>>> >>>> lustre-discuss at lists.lustre.org >>>> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org >> -- >> I/O Doctors, LLC >> 507-766-0378 >> >> bauerj at iodoctors.com -- I/O Doctors, LLC 507-766-0378 bauerj at iodoctors.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From andreas.gruenbacher at gmail.com Mon May 23 22:30:07 2016 From: andreas.gruenbacher at gmail.com (=?UTF-8?Q?Andreas_Gr=C3=BCnbacher?=) Date: Tue, 24 May 2016 00:30:07 +0200 Subject: [lustre-devel] [PATCH] posix acls: Move namespace conversion into filesystem / xattr handlers In-Reply-To: References: <1464008989-3812-1-git-send-email-agruenba@redhat.com> Message-ID: 2016-05-23 23:06 GMT+02:00 James Simmons : > Nak on the Lustre changes. POSIX ACLs are also handled in mdc_request.c. > Besides POSIX ACLs lustre has created a extended ACL as well that is > grouped in with posix ACL handling. This extended ACL is like POSIX acls > except it also contains the state (deleted, modified, ...) of the ACL. > Besides normal local access control list handling Lustre manages remote > access control list handling. You can read about this handling is in > llite_rmtacl.c. This code was written long before I became involved with > lustre so the exact details are fuzzy to me. The guts of this are handled > is located at: > > drivers/staging/lustre/lustre/obdclass/acl.c I'm not sure we're talking about the same thing here. As far as I can see, the only code path from getxattr(2) and settxattr(2) into lustre is through ll_getxattr and ll_setxattr. So far, the vfs performed the uid/gid mappings before calling into the filesystem. With this patch, the filesystem performs this mapping instead. In the lustre case, pretty much directly after entering ll_getxattr and immediately before leaving ll_setxattr. The rest of the code has and still does operate in the "init uid/gid namespace". Did I miss anything? > A you probably have noticed Lustre has had an uptick in development > which means the code is now changing all the time in staging. How should > we handle the changes you are working in your own trees verses what is > happing in staging. For example I'm looking at moving lustre to the > xattr_handles. Okay, great. > Should I push it to you and wait until it works it way > into Greg's tree. Changing lustre to use xattr handlers does not dependent on any of my pending xattr changes, only the other way around. Please copy me for review and merge through Greg's tree as usual. Once I have your changes on a public branch, I can merge that into my xattr inode operation removal branch. > What do the merge schedules look like. I hope the {get,set,remove}xattr inode operations will go away in the next merge window, so it would be good to have the lustre xattr handler convestion on a public branch relatively soon if possible. > Lastly the a_refcount for the POSIX acl changed with your xattr updates which now > causes lustre to crash :-( You mean commit b8a7a3a6 "posix_acl: Inode acl caching fixes"? It seems a "forget_cached_acl(inode, type);" before returning the acl in ll_get_acl is missing -- but lustre still shouldn't crash because of that. Thanks, Andreas From tixxdz at gmail.com Tue May 24 15:41:33 2016 From: tixxdz at gmail.com (Djalal Harouni) Date: Tue, 24 May 2016 16:41:33 +0100 Subject: [lustre-devel] [PATCH] posix acls: Move namespace conversion into filesystem / xattr handlers In-Reply-To: <1464008989-3812-1-git-send-email-agruenba@redhat.com> References: <1464008989-3812-1-git-send-email-agruenba@redhat.com> Message-ID: <20160524153949.GA2830@dztty.fritz.box> Hi, On Mon, May 23, 2016 at 03:09:49PM +0200, Andreas Gruenbacher wrote: > Currently, getxattr() and setxattr() check for the xattr names > "system.posix_acl_{access,default}" and perform in-place UID / GID > namespace mappings in the xattr values. Filesystems then again check for > the same xattr names to handle those attributes, almost always using the > standard posix_acl_{access,default}_xattr_handler handlers. This is > unnecessary overhead; move the namespace conversion into the xattr > handlers instead. Please, are you sure that the changes in posix_acl_xattr_get() and posix_acl_xattr_set() are safe ? you are reading into current user namespace, from a first view this is not safe unless I'm missing something... they should map into init_user_ns... Please Cc the user namespace maintainers before. Thank you! > For filesystems that use the POSIX ACL xattr handlers, no change is > required. Filesystems that don't use those handlers (cifs and lustre) > need to take care of the namespace mapping themselves now. > > The only user left of the posix_acl_fix_xattr_{from,to}_user helpers is > lustre, so this patch moves them into lustre. > > Signed-off-by: Andreas Gruenbacher > --- > > I'm reasonably confident about the core and cifs changes in this patch. > The lustre code is pretty weird though, so could I please get a careful > review on the changes there? > > Thanks, > Andreas > > drivers/staging/lustre/lustre/llite/Makefile | 1 + > .../staging/lustre/lustre/llite/llite_internal.h | 3 ++ > drivers/staging/lustre/lustre/llite/posix_acl.c | 62 ++++++++++++++++++++++ > drivers/staging/lustre/lustre/llite/xattr.c | 8 ++- > fs/cifs/cifssmb.c | 41 ++++++++++---- > fs/posix_acl.c | 62 +--------------------- > fs/xattr.c | 7 --- > include/linux/posix_acl_xattr.h | 12 ----- > 8 files changed, 107 insertions(+), 89 deletions(-) > create mode 100644 drivers/staging/lustre/lustre/llite/posix_acl.c > > diff --git a/drivers/staging/lustre/lustre/llite/Makefile b/drivers/staging/lustre/lustre/llite/Makefile > index 2ce10ff..67125f8 100644 > --- a/drivers/staging/lustre/lustre/llite/Makefile > +++ b/drivers/staging/lustre/lustre/llite/Makefile > @@ -7,5 +7,6 @@ lustre-y := dcache.o dir.o file.o llite_close.o llite_lib.o llite_nfs.o \ > glimpse.o lcommon_cl.o lcommon_misc.o \ > vvp_dev.o vvp_page.o vvp_lock.o vvp_io.o vvp_object.o vvp_req.o \ > lproc_llite.o > +lustre-$(CONFIG_FS_POSIX_ACL) += posix_acl.o > > llite_lloop-y := lloop.o > diff --git a/drivers/staging/lustre/lustre/llite/llite_internal.h b/drivers/staging/lustre/lustre/llite/llite_internal.h > index ce1f949..d454dfb 100644 > --- a/drivers/staging/lustre/lustre/llite/llite_internal.h > +++ b/drivers/staging/lustre/lustre/llite/llite_internal.h > @@ -1402,4 +1402,7 @@ int cl_local_size(struct inode *inode); > __u64 cl_fid_build_ino(const struct lu_fid *fid, int api32); > __u32 cl_fid_build_gen(const struct lu_fid *fid); > > +void posix_acl_fix_xattr_from_user(void *value, size_t size); > +void posix_acl_fix_xattr_to_user(void *value, size_t size); > + > #endif /* LLITE_INTERNAL_H */ > diff --git a/drivers/staging/lustre/lustre/llite/posix_acl.c b/drivers/staging/lustre/lustre/llite/posix_acl.c > new file mode 100644 > index 0000000..4fabd0f > --- /dev/null > +++ b/drivers/staging/lustre/lustre/llite/posix_acl.c > @@ -0,0 +1,62 @@ > +#include > +#include > +#include > +#include > + > +/* > + * Fix up the uids and gids in posix acl extended attributes in place. > + */ > +static void posix_acl_fix_xattr_userns( > + struct user_namespace *to, struct user_namespace *from, > + void *value, size_t size) > +{ > + posix_acl_xattr_header *header = (posix_acl_xattr_header *)value; > + posix_acl_xattr_entry *entry = (posix_acl_xattr_entry *)(header+1), *end; > + int count; > + kuid_t uid; > + kgid_t gid; > + > + if (!value) > + return; > + if (size < sizeof(posix_acl_xattr_header)) > + return; > + if (header->a_version != cpu_to_le32(POSIX_ACL_XATTR_VERSION)) > + return; > + > + count = posix_acl_xattr_count(size); > + if (count < 0) > + return; > + if (count == 0) > + return; > + > + for (end = entry + count; entry != end; entry++) { > + switch(le16_to_cpu(entry->e_tag)) { > + case ACL_USER: > + uid = make_kuid(from, le32_to_cpu(entry->e_id)); > + entry->e_id = cpu_to_le32(from_kuid(to, uid)); > + break; > + case ACL_GROUP: > + gid = make_kgid(from, le32_to_cpu(entry->e_id)); > + entry->e_id = cpu_to_le32(from_kgid(to, gid)); > + break; > + default: > + break; > + } > + } > +} > + > +void posix_acl_fix_xattr_from_user(void *value, size_t size) > +{ > + struct user_namespace *user_ns = current_user_ns(); > + if (user_ns == &init_user_ns) > + return; > + posix_acl_fix_xattr_userns(&init_user_ns, user_ns, value, size); > +} > + > +void posix_acl_fix_xattr_to_user(void *value, size_t size) > +{ > + struct user_namespace *user_ns = current_user_ns(); > + if (user_ns == &init_user_ns) > + return; > + posix_acl_fix_xattr_userns(user_ns, &init_user_ns, value, size); > +} > diff --git a/drivers/staging/lustre/lustre/llite/xattr.c b/drivers/staging/lustre/lustre/llite/xattr.c > index ed4de04..c721b44 100644 > --- a/drivers/staging/lustre/lustre/llite/xattr.c > +++ b/drivers/staging/lustre/lustre/llite/xattr.c > @@ -144,6 +144,9 @@ int ll_setxattr_common(struct inode *inode, const char *name, > return -EOPNOTSUPP; > > #ifdef CONFIG_FS_POSIX_ACL > + if (xattr_type == XATTR_ACL_ACCESS_T || > + xattr_type == XATTR_ACL_DEFAULT_T) > + posix_acl_fix_xattr_from_user((void *)value, size); > if (sbi->ll_flags & LL_SBI_RMT_CLIENT && > (xattr_type == XATTR_ACL_ACCESS_T || > xattr_type == XATTR_ACL_DEFAULT_T)) { > @@ -348,7 +351,7 @@ int ll_getxattr_common(struct inode *inode, const char *name, > if (!acl) > return -ENODATA; > > - rc = posix_acl_to_xattr(&init_user_ns, acl, buffer, size); > + rc = posix_acl_to_xattr(current_user_ns(), acl, buffer, size); > posix_acl_release(acl); > return rc; > } > @@ -436,6 +439,9 @@ getxattr_nocache: > goto out; > } > } > + if (rc >= 0 && (xattr_type == XATTR_ACL_ACCESS_T || > + xattr_type == XATTR_ACL_DEFAULT_T)) > + posix_acl_fix_xattr_to_user(buffer, rc); > #endif > > out_xattr: > diff --git a/fs/cifs/cifssmb.c b/fs/cifs/cifssmb.c > index d47197e..9dc001f 100644 > --- a/fs/cifs/cifssmb.c > +++ b/fs/cifs/cifssmb.c > @@ -3337,10 +3337,25 @@ CIFSSMB_set_compression(const unsigned int xid, struct cifs_tcon *tcon, > static void cifs_convert_ace(posix_acl_xattr_entry *ace, > struct cifs_posix_ace *cifs_ace) > { > + u32 cifs_id, id = -1; > + > /* u8 cifs fields do not need le conversion */ > ace->e_perm = cpu_to_le16(cifs_ace->cifs_e_perm); > ace->e_tag = cpu_to_le16(cifs_ace->cifs_e_tag); > - ace->e_id = cpu_to_le32(le64_to_cpu(cifs_ace->cifs_uid)); > + switch(cifs_ace->cifs_e_tag) { > + case ACL_USER: > + cifs_id = le64_to_cpu(cifs_ace->cifs_uid); > + id = from_kuid(current_user_ns(), > + make_kuid(&init_user_ns, cifs_id)); > + break; > + > + case ACL_GROUP: > + cifs_id = le64_to_cpu(cifs_ace->cifs_uid); > + id = from_kgid(current_user_ns(), > + make_kgid(&init_user_ns, cifs_id)); > + break; > + } > + ace->e_id = cpu_to_le32(id); > /* > cifs_dbg(FYI, "perm %d tag %d id %d\n", > ace->e_perm, ace->e_tag, ace->e_id); > @@ -3408,21 +3423,29 @@ static int cifs_copy_posix_acl(char *trgt, char *src, const int buflen, > static __u16 convert_ace_to_cifs_ace(struct cifs_posix_ace *cifs_ace, > const posix_acl_xattr_entry *local_ace) > { > - __u16 rc = 0; /* 0 = ACL converted ok */ > + u32 cifs_id = -1, id; > > cifs_ace->cifs_e_perm = le16_to_cpu(local_ace->e_perm); > cifs_ace->cifs_e_tag = le16_to_cpu(local_ace->e_tag); > - /* BB is there a better way to handle the large uid? */ > - if (local_ace->e_id == cpu_to_le32(-1)) { > - /* Probably no need to le convert -1 on any arch but can not hurt */ > - cifs_ace->cifs_uid = cpu_to_le64(-1); > - } else > - cifs_ace->cifs_uid = cpu_to_le64(le32_to_cpu(local_ace->e_id)); > + switch(cifs_ace->cifs_e_tag) { > + case ACL_USER: > + id = le32_to_cpu(local_ace->e_id); > + cifs_id = from_kuid(&init_user_ns, > + make_kuid(current_user_ns(), id)); > + break; > + > + case ACL_GROUP: > + id = le32_to_cpu(local_ace->e_id); > + cifs_id = from_kgid(&init_user_ns, > + make_kgid(current_user_ns(), id)); > + break; > + } > + cifs_ace->cifs_uid = cpu_to_le64(cifs_id); > /* > cifs_dbg(FYI, "perm %d tag %d id %d\n", > ace->e_perm, ace->e_tag, ace->e_id); > */ > - return rc; > + return 0; > } > > /* Convert ACL from local Linux POSIX xattr to CIFS POSIX ACL wire format */ > diff --git a/fs/posix_acl.c b/fs/posix_acl.c > index 2c60f17..fca281c 100644 > --- a/fs/posix_acl.c > +++ b/fs/posix_acl.c > @@ -627,64 +627,6 @@ no_mem: > EXPORT_SYMBOL_GPL(posix_acl_create); > > /* > - * Fix up the uids and gids in posix acl extended attributes in place. > - */ > -static void posix_acl_fix_xattr_userns( > - struct user_namespace *to, struct user_namespace *from, > - void *value, size_t size) > -{ > - posix_acl_xattr_header *header = (posix_acl_xattr_header *)value; > - posix_acl_xattr_entry *entry = (posix_acl_xattr_entry *)(header+1), *end; > - int count; > - kuid_t uid; > - kgid_t gid; > - > - if (!value) > - return; > - if (size < sizeof(posix_acl_xattr_header)) > - return; > - if (header->a_version != cpu_to_le32(POSIX_ACL_XATTR_VERSION)) > - return; > - > - count = posix_acl_xattr_count(size); > - if (count < 0) > - return; > - if (count == 0) > - return; > - > - for (end = entry + count; entry != end; entry++) { > - switch(le16_to_cpu(entry->e_tag)) { > - case ACL_USER: > - uid = make_kuid(from, le32_to_cpu(entry->e_id)); > - entry->e_id = cpu_to_le32(from_kuid(to, uid)); > - break; > - case ACL_GROUP: > - gid = make_kgid(from, le32_to_cpu(entry->e_id)); > - entry->e_id = cpu_to_le32(from_kgid(to, gid)); > - break; > - default: > - break; > - } > - } > -} > - > -void posix_acl_fix_xattr_from_user(void *value, size_t size) > -{ > - struct user_namespace *user_ns = current_user_ns(); > - if (user_ns == &init_user_ns) > - return; > - posix_acl_fix_xattr_userns(&init_user_ns, user_ns, value, size); > -} > - > -void posix_acl_fix_xattr_to_user(void *value, size_t size) > -{ > - struct user_namespace *user_ns = current_user_ns(); > - if (user_ns == &init_user_ns) > - return; > - posix_acl_fix_xattr_userns(user_ns, &init_user_ns, value, size); > -} > - > -/* > * Convert from extended attribute to in-memory representation. > */ > struct posix_acl * > @@ -814,7 +756,7 @@ posix_acl_xattr_get(const struct xattr_handler *handler, > if (acl == NULL) > return -ENODATA; > > - error = posix_acl_to_xattr(&init_user_ns, acl, value, size); > + error = posix_acl_to_xattr(current_user_ns(), acl, value, size); > posix_acl_release(acl); > > return error; > @@ -840,7 +782,7 @@ posix_acl_xattr_set(const struct xattr_handler *handler, > return -EPERM; > > if (value) { > - acl = posix_acl_from_xattr(&init_user_ns, value, size); > + acl = posix_acl_from_xattr(current_user_ns(), value, size); > if (IS_ERR(acl)) > return PTR_ERR(acl); > > diff --git a/fs/xattr.c b/fs/xattr.c > index b11945e..b648b05 100644 > --- a/fs/xattr.c > +++ b/fs/xattr.c > @@ -20,7 +20,6 @@ > #include > #include > #include > -#include > > #include > > @@ -329,9 +328,6 @@ setxattr(struct dentry *d, const char __user *name, const void __user *value, > error = -EFAULT; > goto out; > } > - if ((strcmp(kname, XATTR_NAME_POSIX_ACL_ACCESS) == 0) || > - (strcmp(kname, XATTR_NAME_POSIX_ACL_DEFAULT) == 0)) > - posix_acl_fix_xattr_from_user(kvalue, size); > } > > error = vfs_setxattr(d, kname, kvalue, size, flags); > @@ -426,9 +422,6 @@ getxattr(struct dentry *d, const char __user *name, void __user *value, > > error = vfs_getxattr(d, kname, kvalue, size); > if (error > 0) { > - if ((strcmp(kname, XATTR_NAME_POSIX_ACL_ACCESS) == 0) || > - (strcmp(kname, XATTR_NAME_POSIX_ACL_DEFAULT) == 0)) > - posix_acl_fix_xattr_to_user(kvalue, size); > if (size && copy_to_user(value, kvalue, error)) > error = -EFAULT; > } else if (error == -ERANGE && size >= XATTR_SIZE_MAX) { > diff --git a/include/linux/posix_acl_xattr.h b/include/linux/posix_acl_xattr.h > index e5e8ec4..9789aba 100644 > --- a/include/linux/posix_acl_xattr.h > +++ b/include/linux/posix_acl_xattr.h > @@ -48,18 +48,6 @@ posix_acl_xattr_count(size_t size) > return size / sizeof(posix_acl_xattr_entry); > } > > -#ifdef CONFIG_FS_POSIX_ACL > -void posix_acl_fix_xattr_from_user(void *value, size_t size); > -void posix_acl_fix_xattr_to_user(void *value, size_t size); > -#else > -static inline void posix_acl_fix_xattr_from_user(void *value, size_t size) > -{ > -} > -static inline void posix_acl_fix_xattr_to_user(void *value, size_t size) > -{ > -} > -#endif > - > struct posix_acl *posix_acl_from_xattr(struct user_namespace *user_ns, > const void *value, size_t size); > int posix_acl_to_xattr(struct user_namespace *user_ns, > -- > 2.5.5 > > -- > To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in > the body of a message to majordomo at vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html -- Djalal Harouni http://opendz.org From smfrench at gmail.com Tue May 24 01:47:34 2016 From: smfrench at gmail.com (Steve French) Date: Mon, 23 May 2016 20:47:34 -0500 Subject: [lustre-devel] [PATCH] posix acls: Move namespace conversion into filesystem / xattr handlers In-Reply-To: <1464008989-3812-1-git-send-email-agruenba@redhat.com> References: <1464008989-3812-1-git-send-email-agruenba@redhat.com> Message-ID: Reviewed by: Steve French (for the non-lustre pieces of the patch) On Mon, May 23, 2016 at 8:09 AM, Andreas Gruenbacher wrote: > Currently, getxattr() and setxattr() check for the xattr names > "system.posix_acl_{access,default}" and perform in-place UID / GID > namespace mappings in the xattr values. Filesystems then again check for > the same xattr names to handle those attributes, almost always using the > standard posix_acl_{access,default}_xattr_handler handlers. This is > unnecessary overhead; move the namespace conversion into the xattr > handlers instead. > > For filesystems that use the POSIX ACL xattr handlers, no change is > required. Filesystems that don't use those handlers (cifs and lustre) > need to take care of the namespace mapping themselves now. > > The only user left of the posix_acl_fix_xattr_{from,to}_user helpers is > lustre, so this patch moves them into lustre. > > Signed-off-by: Andreas Gruenbacher > --- > > I'm reasonably confident about the core and cifs changes in this patch. > The lustre code is pretty weird though, so could I please get a careful > review on the changes there? > > Thanks, > Andreas > > drivers/staging/lustre/lustre/llite/Makefile | 1 + > .../staging/lustre/lustre/llite/llite_internal.h | 3 ++ > drivers/staging/lustre/lustre/llite/posix_acl.c | 62 ++++++++++++++++++++++ > drivers/staging/lustre/lustre/llite/xattr.c | 8 ++- > fs/cifs/cifssmb.c | 41 ++++++++++---- > fs/posix_acl.c | 62 +--------------------- > fs/xattr.c | 7 --- > include/linux/posix_acl_xattr.h | 12 ----- > 8 files changed, 107 insertions(+), 89 deletions(-) > create mode 100644 drivers/staging/lustre/lustre/llite/posix_acl.c > > diff --git a/drivers/staging/lustre/lustre/llite/Makefile b/drivers/staging/lustre/lustre/llite/Makefile > index 2ce10ff..67125f8 100644 > --- a/drivers/staging/lustre/lustre/llite/Makefile > +++ b/drivers/staging/lustre/lustre/llite/Makefile > @@ -7,5 +7,6 @@ lustre-y := dcache.o dir.o file.o llite_close.o llite_lib.o llite_nfs.o \ > glimpse.o lcommon_cl.o lcommon_misc.o \ > vvp_dev.o vvp_page.o vvp_lock.o vvp_io.o vvp_object.o vvp_req.o \ > lproc_llite.o > +lustre-$(CONFIG_FS_POSIX_ACL) += posix_acl.o > > llite_lloop-y := lloop.o > diff --git a/drivers/staging/lustre/lustre/llite/llite_internal.h b/drivers/staging/lustre/lustre/llite/llite_internal.h > index ce1f949..d454dfb 100644 > --- a/drivers/staging/lustre/lustre/llite/llite_internal.h > +++ b/drivers/staging/lustre/lustre/llite/llite_internal.h > @@ -1402,4 +1402,7 @@ int cl_local_size(struct inode *inode); > __u64 cl_fid_build_ino(const struct lu_fid *fid, int api32); > __u32 cl_fid_build_gen(const struct lu_fid *fid); > > +void posix_acl_fix_xattr_from_user(void *value, size_t size); > +void posix_acl_fix_xattr_to_user(void *value, size_t size); > + > #endif /* LLITE_INTERNAL_H */ > diff --git a/drivers/staging/lustre/lustre/llite/posix_acl.c b/drivers/staging/lustre/lustre/llite/posix_acl.c > new file mode 100644 > index 0000000..4fabd0f > --- /dev/null > +++ b/drivers/staging/lustre/lustre/llite/posix_acl.c > @@ -0,0 +1,62 @@ > +#include > +#include > +#include > +#include > + > +/* > + * Fix up the uids and gids in posix acl extended attributes in place. > + */ > +static void posix_acl_fix_xattr_userns( > + struct user_namespace *to, struct user_namespace *from, > + void *value, size_t size) > +{ > + posix_acl_xattr_header *header = (posix_acl_xattr_header *)value; > + posix_acl_xattr_entry *entry = (posix_acl_xattr_entry *)(header+1), *end; > + int count; > + kuid_t uid; > + kgid_t gid; > + > + if (!value) > + return; > + if (size < sizeof(posix_acl_xattr_header)) > + return; > + if (header->a_version != cpu_to_le32(POSIX_ACL_XATTR_VERSION)) > + return; > + > + count = posix_acl_xattr_count(size); > + if (count < 0) > + return; > + if (count == 0) > + return; > + > + for (end = entry + count; entry != end; entry++) { > + switch(le16_to_cpu(entry->e_tag)) { > + case ACL_USER: > + uid = make_kuid(from, le32_to_cpu(entry->e_id)); > + entry->e_id = cpu_to_le32(from_kuid(to, uid)); > + break; > + case ACL_GROUP: > + gid = make_kgid(from, le32_to_cpu(entry->e_id)); > + entry->e_id = cpu_to_le32(from_kgid(to, gid)); > + break; > + default: > + break; > + } > + } > +} > + > +void posix_acl_fix_xattr_from_user(void *value, size_t size) > +{ > + struct user_namespace *user_ns = current_user_ns(); > + if (user_ns == &init_user_ns) > + return; > + posix_acl_fix_xattr_userns(&init_user_ns, user_ns, value, size); > +} > + > +void posix_acl_fix_xattr_to_user(void *value, size_t size) > +{ > + struct user_namespace *user_ns = current_user_ns(); > + if (user_ns == &init_user_ns) > + return; > + posix_acl_fix_xattr_userns(user_ns, &init_user_ns, value, size); > +} > diff --git a/drivers/staging/lustre/lustre/llite/xattr.c b/drivers/staging/lustre/lustre/llite/xattr.c > index ed4de04..c721b44 100644 > --- a/drivers/staging/lustre/lustre/llite/xattr.c > +++ b/drivers/staging/lustre/lustre/llite/xattr.c > @@ -144,6 +144,9 @@ int ll_setxattr_common(struct inode *inode, const char *name, > return -EOPNOTSUPP; > > #ifdef CONFIG_FS_POSIX_ACL > + if (xattr_type == XATTR_ACL_ACCESS_T || > + xattr_type == XATTR_ACL_DEFAULT_T) > + posix_acl_fix_xattr_from_user((void *)value, size); > if (sbi->ll_flags & LL_SBI_RMT_CLIENT && > (xattr_type == XATTR_ACL_ACCESS_T || > xattr_type == XATTR_ACL_DEFAULT_T)) { > @@ -348,7 +351,7 @@ int ll_getxattr_common(struct inode *inode, const char *name, > if (!acl) > return -ENODATA; > > - rc = posix_acl_to_xattr(&init_user_ns, acl, buffer, size); > + rc = posix_acl_to_xattr(current_user_ns(), acl, buffer, size); > posix_acl_release(acl); > return rc; > } > @@ -436,6 +439,9 @@ getxattr_nocache: > goto out; > } > } > + if (rc >= 0 && (xattr_type == XATTR_ACL_ACCESS_T || > + xattr_type == XATTR_ACL_DEFAULT_T)) > + posix_acl_fix_xattr_to_user(buffer, rc); > #endif > > out_xattr: > diff --git a/fs/cifs/cifssmb.c b/fs/cifs/cifssmb.c > index d47197e..9dc001f 100644 > --- a/fs/cifs/cifssmb.c > +++ b/fs/cifs/cifssmb.c > @@ -3337,10 +3337,25 @@ CIFSSMB_set_compression(const unsigned int xid, struct cifs_tcon *tcon, > static void cifs_convert_ace(posix_acl_xattr_entry *ace, > struct cifs_posix_ace *cifs_ace) > { > + u32 cifs_id, id = -1; > + > /* u8 cifs fields do not need le conversion */ > ace->e_perm = cpu_to_le16(cifs_ace->cifs_e_perm); > ace->e_tag = cpu_to_le16(cifs_ace->cifs_e_tag); > - ace->e_id = cpu_to_le32(le64_to_cpu(cifs_ace->cifs_uid)); > + switch(cifs_ace->cifs_e_tag) { > + case ACL_USER: > + cifs_id = le64_to_cpu(cifs_ace->cifs_uid); > + id = from_kuid(current_user_ns(), > + make_kuid(&init_user_ns, cifs_id)); > + break; > + > + case ACL_GROUP: > + cifs_id = le64_to_cpu(cifs_ace->cifs_uid); > + id = from_kgid(current_user_ns(), > + make_kgid(&init_user_ns, cifs_id)); > + break; > + } > + ace->e_id = cpu_to_le32(id); > /* > cifs_dbg(FYI, "perm %d tag %d id %d\n", > ace->e_perm, ace->e_tag, ace->e_id); > @@ -3408,21 +3423,29 @@ static int cifs_copy_posix_acl(char *trgt, char *src, const int buflen, > static __u16 convert_ace_to_cifs_ace(struct cifs_posix_ace *cifs_ace, > const posix_acl_xattr_entry *local_ace) > { > - __u16 rc = 0; /* 0 = ACL converted ok */ > + u32 cifs_id = -1, id; > > cifs_ace->cifs_e_perm = le16_to_cpu(local_ace->e_perm); > cifs_ace->cifs_e_tag = le16_to_cpu(local_ace->e_tag); > - /* BB is there a better way to handle the large uid? */ > - if (local_ace->e_id == cpu_to_le32(-1)) { > - /* Probably no need to le convert -1 on any arch but can not hurt */ > - cifs_ace->cifs_uid = cpu_to_le64(-1); > - } else > - cifs_ace->cifs_uid = cpu_to_le64(le32_to_cpu(local_ace->e_id)); > + switch(cifs_ace->cifs_e_tag) { > + case ACL_USER: > + id = le32_to_cpu(local_ace->e_id); > + cifs_id = from_kuid(&init_user_ns, > + make_kuid(current_user_ns(), id)); > + break; > + > + case ACL_GROUP: > + id = le32_to_cpu(local_ace->e_id); > + cifs_id = from_kgid(&init_user_ns, > + make_kgid(current_user_ns(), id)); > + break; > + } > + cifs_ace->cifs_uid = cpu_to_le64(cifs_id); > /* > cifs_dbg(FYI, "perm %d tag %d id %d\n", > ace->e_perm, ace->e_tag, ace->e_id); > */ > - return rc; > + return 0; > } > > /* Convert ACL from local Linux POSIX xattr to CIFS POSIX ACL wire format */ > diff --git a/fs/posix_acl.c b/fs/posix_acl.c > index 2c60f17..fca281c 100644 > --- a/fs/posix_acl.c > +++ b/fs/posix_acl.c > @@ -627,64 +627,6 @@ no_mem: > EXPORT_SYMBOL_GPL(posix_acl_create); > > /* > - * Fix up the uids and gids in posix acl extended attributes in place. > - */ > -static void posix_acl_fix_xattr_userns( > - struct user_namespace *to, struct user_namespace *from, > - void *value, size_t size) > -{ > - posix_acl_xattr_header *header = (posix_acl_xattr_header *)value; > - posix_acl_xattr_entry *entry = (posix_acl_xattr_entry *)(header+1), *end; > - int count; > - kuid_t uid; > - kgid_t gid; > - > - if (!value) > - return; > - if (size < sizeof(posix_acl_xattr_header)) > - return; > - if (header->a_version != cpu_to_le32(POSIX_ACL_XATTR_VERSION)) > - return; > - > - count = posix_acl_xattr_count(size); > - if (count < 0) > - return; > - if (count == 0) > - return; > - > - for (end = entry + count; entry != end; entry++) { > - switch(le16_to_cpu(entry->e_tag)) { > - case ACL_USER: > - uid = make_kuid(from, le32_to_cpu(entry->e_id)); > - entry->e_id = cpu_to_le32(from_kuid(to, uid)); > - break; > - case ACL_GROUP: > - gid = make_kgid(from, le32_to_cpu(entry->e_id)); > - entry->e_id = cpu_to_le32(from_kgid(to, gid)); > - break; > - default: > - break; > - } > - } > -} > - > -void posix_acl_fix_xattr_from_user(void *value, size_t size) > -{ > - struct user_namespace *user_ns = current_user_ns(); > - if (user_ns == &init_user_ns) > - return; > - posix_acl_fix_xattr_userns(&init_user_ns, user_ns, value, size); > -} > - > -void posix_acl_fix_xattr_to_user(void *value, size_t size) > -{ > - struct user_namespace *user_ns = current_user_ns(); > - if (user_ns == &init_user_ns) > - return; > - posix_acl_fix_xattr_userns(user_ns, &init_user_ns, value, size); > -} > - > -/* > * Convert from extended attribute to in-memory representation. > */ > struct posix_acl * > @@ -814,7 +756,7 @@ posix_acl_xattr_get(const struct xattr_handler *handler, > if (acl == NULL) > return -ENODATA; > > - error = posix_acl_to_xattr(&init_user_ns, acl, value, size); > + error = posix_acl_to_xattr(current_user_ns(), acl, value, size); > posix_acl_release(acl); > > return error; > @@ -840,7 +782,7 @@ posix_acl_xattr_set(const struct xattr_handler *handler, > return -EPERM; > > if (value) { > - acl = posix_acl_from_xattr(&init_user_ns, value, size); > + acl = posix_acl_from_xattr(current_user_ns(), value, size); > if (IS_ERR(acl)) > return PTR_ERR(acl); > > diff --git a/fs/xattr.c b/fs/xattr.c > index b11945e..b648b05 100644 > --- a/fs/xattr.c > +++ b/fs/xattr.c > @@ -20,7 +20,6 @@ > #include > #include > #include > -#include > > #include > > @@ -329,9 +328,6 @@ setxattr(struct dentry *d, const char __user *name, const void __user *value, > error = -EFAULT; > goto out; > } > - if ((strcmp(kname, XATTR_NAME_POSIX_ACL_ACCESS) == 0) || > - (strcmp(kname, XATTR_NAME_POSIX_ACL_DEFAULT) == 0)) > - posix_acl_fix_xattr_from_user(kvalue, size); > } > > error = vfs_setxattr(d, kname, kvalue, size, flags); > @@ -426,9 +422,6 @@ getxattr(struct dentry *d, const char __user *name, void __user *value, > > error = vfs_getxattr(d, kname, kvalue, size); > if (error > 0) { > - if ((strcmp(kname, XATTR_NAME_POSIX_ACL_ACCESS) == 0) || > - (strcmp(kname, XATTR_NAME_POSIX_ACL_DEFAULT) == 0)) > - posix_acl_fix_xattr_to_user(kvalue, size); > if (size && copy_to_user(value, kvalue, error)) > error = -EFAULT; > } else if (error == -ERANGE && size >= XATTR_SIZE_MAX) { > diff --git a/include/linux/posix_acl_xattr.h b/include/linux/posix_acl_xattr.h > index e5e8ec4..9789aba 100644 > --- a/include/linux/posix_acl_xattr.h > +++ b/include/linux/posix_acl_xattr.h > @@ -48,18 +48,6 @@ posix_acl_xattr_count(size_t size) > return size / sizeof(posix_acl_xattr_entry); > } > > -#ifdef CONFIG_FS_POSIX_ACL > -void posix_acl_fix_xattr_from_user(void *value, size_t size); > -void posix_acl_fix_xattr_to_user(void *value, size_t size); > -#else > -static inline void posix_acl_fix_xattr_from_user(void *value, size_t size) > -{ > -} > -static inline void posix_acl_fix_xattr_to_user(void *value, size_t size) > -{ > -} > -#endif > - > struct posix_acl *posix_acl_from_xattr(struct user_namespace *user_ns, > const void *value, size_t size); > int posix_acl_to_xattr(struct user_namespace *user_ns, > -- > 2.5.5 > > -- > To unsubscribe from this list: send the line "unsubscribe linux-cifs" in > the body of a message to majordomo at vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html -- Thanks, Steve From bauerj at iodoctors.com Sat May 21 20:01:22 2016 From: bauerj at iodoctors.com (John Bauer) Date: Sat, 21 May 2016 15:01:22 -0500 Subject: [lustre-devel] [lustre-discuss] more on lustre striping In-Reply-To: <38EA9F32-3869-43BA-B215-EB2E5BA62FD5@intel.com> References: <1ef5a267-334c-be0d-13f4-c0fab917d1bf@iodoctors.com> <38EA9F32-3869-43BA-B215-EB2E5BA62FD5@intel.com> Message-ID: Andreas, Thanks for the reply. For what it's worth, extending a file that does not have layout set does work. % *rm -f file.dat* % *./no_stripe.exe file.dat* fd=3 % *lfs getstripe file.dat* file.dat has no stripe info % *date >> file.dat* % *lfs getstripe file.dat* file.dat lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 21 obdidx objid objid group 21 6143298 0x5dbd42 0 % The LD_PRELOAD is exactly what I am doing in my I/O library. Unfortunately, one can not intercept the open() that results from a call to fopen(). That open is hard linked to the open in libc and not satisfied by the runtime linker. This is what is driving this topic for me. I can not conveniently set the striping for a file opened with fopen() and other functions where the open is called from inside libc. I used to believe that not too many application use stdio for heavy I/O, but I have been come across several recently. John On 5/21/2016 12:51 AM, Dilger, Andreas wrote: > This is probably getting to be more of a topic for lustre-devel. > > There currently isn't any way to do what you ask, since (IIRC) it will > cause an error for apps that try to write to the files before the > layout is set. > > What you could do is to create an LD_PRELOAD library to intercept the > open() calls and set O_LOV_DELAY_CREATE and set the layout explicitly > for each file. This might be a win if each file needs a different > layout, but since it uses two RPCs per file it would be slower than > using the default layout. > > Cheers, Andreas > > On May 18, 2016, at 16:46, John Bauer > wrote: > >> Since today's topic seems to be Lustre striping, I will revisit a >> previous line of questions I had. >> >> Andreas had put me on to O_LOV_DELAY_CREATE which I have been >> experimenting with. My question is : Is there a way to flag a >> directory with O_LOV_DELAY_CREATE so that a file created in that >> directory will be created with O_LOV_DELAY_CREATE also. Much like a >> file can inherit a directory's stripe count and stripe size, it would >> be convenient if a file could also inherit O_LOV_DELAY_CREATE? That >> way, for open()s that I can not intercept ( and thus can not set >> O_LOV_DELAY_CREATE in oflags) , such as those issued by fopen(), I >> can then get the fd with fileno() and set the striping withioctl(fd, >> LL_IOC_LOV_SETSTRIPE, lum). >> >> Thanks >> >> John >> >> -- >> I/O Doctors, LLC >> 507-766-0378 >> bauerj at iodoctors.com >> _______________________________________________ >> lustre-discuss mailing list >> lustre-discuss at lists.lustre.org >> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org -- I/O Doctors, LLC 507-766-0378 bauerj at iodoctors.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From me at tobin.cc Sun May 22 21:49:29 2016 From: me at tobin.cc (Tobin C Harding) Date: Mon, 23 May 2016 07:49:29 +1000 Subject: [lustre-devel] [PATCH] staging: lustre: llite: kzalloc/copy_to_user to memdup_user Message-ID: <1463953771-16942-1-git-send-email-me@tobin.cc> kzalloc call followed by copy_to_user can be replaced by call to memdup_user. Signed-off-by: Tobin C Harding --- drivers/gpu/drm/gma500/gma_display.c | 2 +- drivers/staging/lustre/lustre/llite/dir.c | 22 +++++++--------------- 2 files changed, 8 insertions(+), 16 deletions(-) diff --git a/drivers/gpu/drm/gma500/gma_display.c b/drivers/gpu/drm/gma500/gma_display.c index 5bf765d..8a1fb25 100644 --- a/drivers/gpu/drm/gma500/gma_display.c +++ b/drivers/gpu/drm/gma500/gma_display.c @@ -285,7 +285,7 @@ void gma_crtc_dpms(struct drm_crtc *crtc, int mode) /* Wait for vblank for the disable to take effect */ gma_wait_for_vblank(dev); - +l /* Disable plane */ temp = REG_READ(map->cntr); if ((temp & DISPLAY_PLANE_ENABLE) != 0) { diff --git a/drivers/staging/lustre/lustre/llite/dir.c b/drivers/staging/lustre/lustre/llite/dir.c index 4b00d1a..b0eb102 100644 --- a/drivers/staging/lustre/lustre/llite/dir.c +++ b/drivers/staging/lustre/lustre/llite/dir.c @@ -1076,19 +1076,14 @@ static int copy_and_ioctl(int cmd, struct obd_export *exp, void *copy; int rc; - copy = kzalloc(size, GFP_NOFS); - if (!copy) - return -ENOMEM; - - if (copy_from_user(copy, data, size)) { - rc = -EFAULT; + copy = memdup_user(data, size); + if (IS_ERR(copy)) { + rc = PTR_ERR(copy); goto out; } rc = obd_iocontrol(cmd, exp, size, copy, NULL); out: - kfree(copy); - return rc; } @@ -1689,12 +1684,9 @@ out_poll: case LL_IOC_QUOTACTL: { struct if_quotactl *qctl; - qctl = kzalloc(sizeof(*qctl), GFP_NOFS); - if (!qctl) - return -ENOMEM; - - if (copy_from_user(qctl, (void __user *)arg, sizeof(*qctl))) { - rc = -EFAULT; + qctl = memdup_user((void __user *)arg, sizeof(*qctl)); + if (IS_ERR(qctl)) { + rc = PTR_ERR(qctl); goto out_quotactl; } @@ -1704,8 +1696,8 @@ out_poll: sizeof(*qctl))) rc = -EFAULT; -out_quotactl: kfree(qctl); +out_quotactl: return rc; } case OBD_IOC_GETDTNAME: -- 2.8.2 From me at tobin.cc Mon May 23 00:14:22 2016 From: me at tobin.cc (Tobin C Harding) Date: Mon, 23 May 2016 10:14:22 +1000 Subject: [lustre-devel] [PATCH v2] staging: lustre: llite: kzalloc/copy_to_user to memdup_user Message-ID: <1463962462-14335-1-git-send-email-me@tobin.cc> kzalloc call followed by copy_to_user can be replaced by call to memdup_user. Signed-off-by: Tobin C Harding --- drivers/staging/lustre/lustre/llite/dir.c | 12 +++--------- 1 file changed, 3 insertions(+), 9 deletions(-) diff --git a/drivers/staging/lustre/lustre/llite/dir.c b/drivers/staging/lustre/lustre/llite/dir.c index 4b00d1a..85c50e0 100644 --- a/drivers/staging/lustre/lustre/llite/dir.c +++ b/drivers/staging/lustre/lustre/llite/dir.c @@ -1076,17 +1076,11 @@ static int copy_and_ioctl(int cmd, struct obd_export *exp, void *copy; int rc; - copy = kzalloc(size, GFP_NOFS); - if (!copy) - return -ENOMEM; - - if (copy_from_user(copy, data, size)) { - rc = -EFAULT; - goto out; - } + copy = memdup_user(data, size); + if (IS_ERR(copy)) + return PTR_ERR(copy); rc = obd_iocontrol(cmd, exp, size, copy, NULL); -out: kfree(copy); return rc; -- 2.8.2 From me at tobin.cc Mon May 23 00:20:07 2016 From: me at tobin.cc (Tobin Harding) Date: Mon, 23 May 2016 10:20:07 +1000 Subject: [lustre-devel] [PATCH] staging: lustre: llite: kzalloc/copy_to_user to memdup_user In-Reply-To: <20160522222538.GA17573@kroah.com> References: <1463953771-16942-1-git-send-email-me@tobin.cc> <20160522222538.GA17573@kroah.com> Message-ID: <20160523002007.GA14493@eros.local> On Sun, May 22, 2016 at 03:25:38PM -0700, Greg Kroah-Hartman wrote: > On Mon, May 23, 2016 at 07:49:29AM +1000, Tobin C Harding wrote: > > kzalloc call followed by copy_to_user can be replaced by call to memdup_user. > > > > Signed-off-by: Tobin C Harding > > Why did you send this twice? Because I'm a git send-email amateur. > > --- > > drivers/gpu/drm/gma500/gma_display.c | 2 +- > > drivers/staging/lustre/lustre/llite/dir.c | 22 +++++++--------------- > > 2 files changed, 8 insertions(+), 16 deletions(-) > > > > diff --git a/drivers/gpu/drm/gma500/gma_display.c b/drivers/gpu/drm/gma500/gma_display.c > > index 5bf765d..8a1fb25 100644 > > --- a/drivers/gpu/drm/gma500/gma_display.c > > +++ b/drivers/gpu/drm/gma500/gma_display.c > > @@ -285,7 +285,7 @@ void gma_crtc_dpms(struct drm_crtc *crtc, int mode) > > > > /* Wait for vblank for the disable to take effect */ > > gma_wait_for_vblank(dev); > > - > > +l > > What? What does this have to do with lustre? Turns out I'm a git amateur in general, the gma_display changes should not have been committed on this branch. > It's kind of obvious you didn't build this :( Epic fail. > Please be more careful in the future. > > greg k-h I am humbly attempting to learn the way of kernel dev, thank you for your patience, I will endeavor to be more careful. Regards, Tobin Harding. From linuxdev.baldrick at gmail.com Mon May 23 11:17:49 2016 From: linuxdev.baldrick at gmail.com (David Binderman) Date: Mon, 23 May 2016 12:17:49 +0100 Subject: [lustre-devel] drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd_modparams.c:235: pointless test ? Message-ID: Hello there, drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd_modparams.c:235:34: warning: comparison of unsigned expression < 0 is always false [-Wtype-limits] Source code is if (tunables->lnd_map_on_demand < 0 || Regards David Binderman From andreas.gruenbacher at gmail.com Tue May 24 02:08:05 2016 From: andreas.gruenbacher at gmail.com (=?UTF-8?Q?Andreas_Gr=C3=BCnbacher?=) Date: Tue, 24 May 2016 04:08:05 +0200 Subject: [lustre-devel] [PATCH] staging: lustre: llite: drop acl from cache In-Reply-To: <1464050108-28015-1-git-send-email-jsimmons@infradead.org> References: <1464050108-28015-1-git-send-email-jsimmons@infradead.org> Message-ID: 2016-05-24 2:35 GMT+02:00 James Simmons : > Commit b8a7a3a6 change get_acl() for posix xattr to always cache > the ACL which increases the reference count. That reference count > can be reduced by have ll_get_acl() call forget_cached_acl() which > it wasn't. When an inode gets deleted by Lustre the POSIX ACL > reference count is tested to ensure its 1 and if not produces an error. Lustre shouldn't assume that the VFS immediately drops the reference it is passed. Please remove that check as well. > Since forget_cached_acl() was not called Lustre started to complain. > This patch changes ll_get_acl() to call forget_cached_acl(). > > Signed-off-by: James Simmons Reviewed-by: Andreas Gruenbacher > --- > drivers/staging/lustre/lustre/llite/file.c | 1 + > 1 files changed, 1 insertions(+), 0 deletions(-) > > diff --git a/drivers/staging/lustre/lustre/llite/file.c b/drivers/staging/lustre/lustre/llite/file.c > index f47f2ac..0191945 100644 > --- a/drivers/staging/lustre/lustre/llite/file.c > +++ b/drivers/staging/lustre/lustre/llite/file.c > @@ -3124,6 +3124,7 @@ struct posix_acl *ll_get_acl(struct inode *inode, int type) > spin_lock(&lli->lli_lock); > /* VFS' acl_permission_check->check_acl will release the refcount */ > acl = posix_acl_dup(lli->lli_posix_acl); > + forget_cached_acl(inode, type); > spin_unlock(&lli->lli_lock); > > return acl; > -- > 1.7.1 > From me at tobin.cc Sun May 22 21:52:17 2016 From: me at tobin.cc (Tobin C Harding) Date: Mon, 23 May 2016 07:52:17 +1000 Subject: [lustre-devel] [PATCH] staging: lustre: llite: kzalloc/copy_to_user to memdup_user Message-ID: <1463953938-17192-1-git-send-email-me@tobin.cc> kzalloc call followed by copy_to_user can be replaced by call to memdup_user. Signed-off-by: Tobin C Harding --- drivers/gpu/drm/gma500/gma_display.c | 2 +- drivers/staging/lustre/lustre/llite/dir.c | 22 +++++++--------------- 2 files changed, 8 insertions(+), 16 deletions(-) diff --git a/drivers/gpu/drm/gma500/gma_display.c b/drivers/gpu/drm/gma500/gma_display.c index 5bf765d..8a1fb25 100644 --- a/drivers/gpu/drm/gma500/gma_display.c +++ b/drivers/gpu/drm/gma500/gma_display.c @@ -285,7 +285,7 @@ void gma_crtc_dpms(struct drm_crtc *crtc, int mode) /* Wait for vblank for the disable to take effect */ gma_wait_for_vblank(dev); - +l /* Disable plane */ temp = REG_READ(map->cntr); if ((temp & DISPLAY_PLANE_ENABLE) != 0) { diff --git a/drivers/staging/lustre/lustre/llite/dir.c b/drivers/staging/lustre/lustre/llite/dir.c index 4b00d1a..b0eb102 100644 --- a/drivers/staging/lustre/lustre/llite/dir.c +++ b/drivers/staging/lustre/lustre/llite/dir.c @@ -1076,19 +1076,14 @@ static int copy_and_ioctl(int cmd, struct obd_export *exp, void *copy; int rc; - copy = kzalloc(size, GFP_NOFS); - if (!copy) - return -ENOMEM; - - if (copy_from_user(copy, data, size)) { - rc = -EFAULT; + copy = memdup_user(data, size); + if (IS_ERR(copy)) { + rc = PTR_ERR(copy); goto out; } rc = obd_iocontrol(cmd, exp, size, copy, NULL); out: - kfree(copy); - return rc; } @@ -1689,12 +1684,9 @@ out_poll: case LL_IOC_QUOTACTL: { struct if_quotactl *qctl; - qctl = kzalloc(sizeof(*qctl), GFP_NOFS); - if (!qctl) - return -ENOMEM; - - if (copy_from_user(qctl, (void __user *)arg, sizeof(*qctl))) { - rc = -EFAULT; + qctl = memdup_user((void __user *)arg, sizeof(*qctl)); + if (IS_ERR(qctl)) { + rc = PTR_ERR(qctl); goto out_quotactl; } @@ -1704,8 +1696,8 @@ out_poll: sizeof(*qctl))) rc = -EFAULT; -out_quotactl: kfree(qctl); +out_quotactl: return rc; } case OBD_IOC_GETDTNAME: -- 2.8.2 From me at tobin.cc Mon May 23 10:17:46 2016 From: me at tobin.cc (Tobin C Harding) Date: Mon, 23 May 2016 20:17:46 +1000 Subject: [lustre-devel] [PATCH] staging: lustre: set function scope with static Message-ID: <1463998666-4636-1-git-send-email-me@tobin.cc> A number of function definitions were found to be candidates for static scoping. This patch adds static to these functions. Signed-off-by: Tobin C Harding --- drivers/staging/lustre/lustre/llite/vvp_dev.c | 8 ++++---- drivers/staging/lustre/lustre/llite/vvp_io.c | 6 +++--- drivers/staging/lustre/lustre/llite/vvp_req.c | 12 ++++++------ drivers/staging/lustre/lustre/obdclass/obd_mount.c | 2 +- 4 files changed, 14 insertions(+), 14 deletions(-) diff --git a/drivers/staging/lustre/lustre/llite/vvp_dev.c b/drivers/staging/lustre/lustre/llite/vvp_dev.c index 47101de..c63def9 100644 --- a/drivers/staging/lustre/lustre/llite/vvp_dev.c +++ b/drivers/staging/lustre/lustre/llite/vvp_dev.c @@ -150,8 +150,8 @@ struct lu_context_key vvp_session_key = { .lct_fini = vvp_session_key_fini }; -void *vvp_thread_key_init(const struct lu_context *ctx, - struct lu_context_key *key) +static void *vvp_thread_key_init(const struct lu_context *ctx, + struct lu_context_key *key) { struct vvp_thread_info *vti; @@ -161,8 +161,8 @@ void *vvp_thread_key_init(const struct lu_context *ctx, return vti; } -void vvp_thread_key_fini(const struct lu_context *ctx, - struct lu_context_key *key, void *data) +static void vvp_thread_key_fini(const struct lu_context *ctx, + struct lu_context_key *key, void *data) { struct vvp_thread_info *vti = data; diff --git a/drivers/staging/lustre/lustre/llite/vvp_io.c b/drivers/staging/lustre/lustre/llite/vvp_io.c index 5bf9592..e26e0f8 100644 --- a/drivers/staging/lustre/lustre/llite/vvp_io.c +++ b/drivers/staging/lustre/lustre/llite/vvp_io.c @@ -47,8 +47,8 @@ #include "llite_internal.h" #include "vvp_internal.h" -struct vvp_io *cl2vvp_io(const struct lu_env *env, - const struct cl_io_slice *slice) +static struct vvp_io *cl2vvp_io(const struct lu_env *env, + const struct cl_io_slice *slice) { struct vvp_io *vio; @@ -1259,7 +1259,7 @@ static int vvp_io_read_page(const struct lu_env *env, return 0; } -void vvp_io_end(const struct lu_env *env, const struct cl_io_slice *ios) +static void vvp_io_end(const struct lu_env *env, const struct cl_io_slice *ios) { CLOBINVRNT(env, ios->cis_io->ci_obj, vvp_object_invariant(ios->cis_io->ci_obj)); diff --git a/drivers/staging/lustre/lustre/llite/vvp_req.c b/drivers/staging/lustre/lustre/llite/vvp_req.c index fb88629..9fe9d6c 100644 --- a/drivers/staging/lustre/lustre/llite/vvp_req.c +++ b/drivers/staging/lustre/lustre/llite/vvp_req.c @@ -60,10 +60,10 @@ static inline struct vvp_req *cl2vvp_req(const struct cl_req_slice *slice) * - o_ioepoch, * */ -void vvp_req_attr_set(const struct lu_env *env, - const struct cl_req_slice *slice, - const struct cl_object *obj, - struct cl_req_attr *attr, u64 flags) +static void vvp_req_attr_set(const struct lu_env *env, + const struct cl_req_slice *slice, + const struct cl_object *obj, + struct cl_req_attr *attr, u64 flags) { struct inode *inode; struct obdo *oa; @@ -87,8 +87,8 @@ void vvp_req_attr_set(const struct lu_env *env, JOBSTATS_JOBID_SIZE); } -void vvp_req_completion(const struct lu_env *env, - const struct cl_req_slice *slice, int ioret) +static void vvp_req_completion(const struct lu_env *env, + const struct cl_req_slice *slice, int ioret) { struct vvp_req *vrq; diff --git a/drivers/staging/lustre/lustre/obdclass/obd_mount.c b/drivers/staging/lustre/lustre/obdclass/obd_mount.c index e0c90ad..bc4e1e4 100644 --- a/drivers/staging/lustre/lustre/obdclass/obd_mount.c +++ b/drivers/staging/lustre/lustre/obdclass/obd_mount.c @@ -192,7 +192,7 @@ static int lustre_start_simple(char *obdname, char *type, char *uuid, return rc; } -DEFINE_MUTEX(mgc_start_lock); +static DEFINE_MUTEX(mgc_start_lock); /** Set up a mgc obd to process startup logs * -- 2.8.2 From ebiederm at xmission.com Wed May 25 23:30:45 2016 From: ebiederm at xmission.com (Eric W. Biederman) Date: Wed, 25 May 2016 18:30:45 -0500 Subject: [lustre-devel] [PATCH] posix acls: Move namespace conversion into filesystem / xattr handlers In-Reply-To: (Andreas Gruenbacher's message of "Tue, 24 May 2016 18:31:51 +0200") References: <1464008989-3812-1-git-send-email-agruenba@redhat.com> <20160524153949.GA2830@dztty.fritz.box> Message-ID: <87h9dllnq2.fsf@x220.int.ebiederm.org> Andreas Gruenbacher writes: > On Tue, May 24, 2016 at 5:41 PM, Djalal Harouni wrote: >> On Mon, May 23, 2016 at 03:09:49PM +0200, Andreas Gruenbacher wrote: >>> Currently, getxattr() and setxattr() check for the xattr names >>> "system.posix_acl_{access,default}" and perform in-place UID / GID >>> namespace mappings in the xattr values. Filesystems then again check for >>> the same xattr names to handle those attributes, almost always using the >>> standard posix_acl_{access,default}_xattr_handler handlers. This is >>> unnecessary overhead; move the namespace conversion into the xattr >>> handlers instead. >> >> Please, are you sure that the changes in posix_acl_xattr_get() and >> posix_acl_xattr_set() are safe ? you are reading into current user >> namespace, from a first view this is not safe unless I'm missing >> something... they should map into init_user_ns... > > Yes, moving the namespace conversion from the VFS into those functions > so that we don't have to check for those attributes and parse them > twice is exactly the point of this patch. In general I am in favor of cleaning up the xattr and acl code in the kernel. However I am not certain that your changes succeed in getting us where we want to go. My feel is that what we want to do is to update the cached acl when we write it from userspace, to update the disk with the new acl when the inode is sync'd to disk, and let the vfs handle the translation from the cached posix acl to the vfs getxattr and setxattr system calls. In the long term anything else is madness. Currently posix acl reads and updates bypass the vfs acl cache for the inode and that just looks wrong. The reason that fixup happens in a separate pass from everything else today is that when I was wrote posix_acl_fix_xattr_to_user and posix_acl_xattr_from_user a number of filesystems had a very strange structure that completely bypassed any code conversion routines and made some strange assumptions. I don't remember which filesystems those were but it was neither lustre, nor cifs, nor nfs that were the problem cases when I was looking. I don't see your patch addressing that problem so either someone has already fixed those issues or they have been overlooked. There is a complication that is comming shortly (next merge window unless major unfixable bugs show up between now and them). There is a new field s_user_ns that is being introduced for filesystems to record their owner and their default translation rules to kuids. That will make the hard coded &init_user_ns values in your patch wrong. >> Please Cc the user namespace maintainers before. Thank you! > > Eric, Andy, anyone else? Serge Hallyn has a pending patch that adds a similar translation to the security.capability xattr. Which is one more case of where caching and translation at the VFS layer are makes sense. All of that said I am definitely in favor of cleaning up this area of code. Eric From ebiederm at xmission.com Fri May 27 18:07:09 2016 From: ebiederm at xmission.com (Eric W. Biederman) Date: Fri, 27 May 2016 13:07:09 -0500 Subject: [lustre-devel] [PATCH] posix acls: Move namespace conversion into filesystem / xattr handlers In-Reply-To: (Andreas Gruenbacher's message of "Thu, 26 May 2016 13:36:23 +0200") References: <1464008989-3812-1-git-send-email-agruenba@redhat.com> <20160524153949.GA2830@dztty.fritz.box> <87h9dllnq2.fsf@x220.int.ebiederm.org> Message-ID: <87bn3ridde.fsf@x220.int.ebiederm.org> Andreas Gruenbacher writes: > On Thu, May 26, 2016 at 1:30 AM, Eric W. Biederman > wrote: >> Andreas Gruenbacher writes: >> >>> On Tue, May 24, 2016 at 5:41 PM, Djalal Harouni wrote: >>>> On Mon, May 23, 2016 at 03:09:49PM +0200, Andreas Gruenbacher wrote: >>>>> Currently, getxattr() and setxattr() check for the xattr names >>>>> "system.posix_acl_{access,default}" and perform in-place UID / GID >>>>> namespace mappings in the xattr values. Filesystems then again check for >>>>> the same xattr names to handle those attributes, almost always using the >>>>> standard posix_acl_{access,default}_xattr_handler handlers. This is >>>>> unnecessary overhead; move the namespace conversion into the xattr >>>>> handlers instead. >>>> >>>> Please, are you sure that the changes in posix_acl_xattr_get() and >>>> posix_acl_xattr_set() are safe ? you are reading into current user >>>> namespace, from a first view this is not safe unless I'm missing >>>> something... they should map into init_user_ns... >>> >>> Yes, moving the namespace conversion from the VFS into those functions >>> so that we don't have to check for those attributes and parse them >>> twice is exactly the point of this patch. >> >> In general I am in favor of cleaning up the xattr and acl code in the >> kernel. However I am not certain that your changes succeed in getting >> us where we want to go. >> >> My feel is that what we want to do is to update the cached acl when we >> write it from userspace, to update the disk with the new acl when the >> inode is sync'd to disk, and let the vfs handle the translation from >> the cached posix acl to the vfs getxattr and setxattr system calls. >> In the long term anything else is madness. >> >> Currently posix acl reads and updates bypass the vfs acl cache for the >> inode and that just looks wrong. > > Not all filesystems cache acls in the inode (i_acl and i_default_acl), so there > has to be some way to "bypass the cache" if you want to put it that way. > > All normal filesystems that cache ACLs locally implement the get_acl and > set_acl inode operations. They put posix_acl_access_xattr_handler > and posix_acl_default_xattr_handler into s_xattr and use the > generic_{get,set,remove}xattr inode operations to hook things up > appropriately. The xattr handlers convert to/from xattrs and struct posix_acl > and call the get_acl and set_acl inode operations. That's where the namespace > conversion is supposed to happen as well; this avoids special-casing > it in the VFS. The acl support does look like what I would expect this code to look like. Unfortunately some fileystems still (as of 4.6) write xattrs directly to disk. > The only filesystems that don't do things that way are CIFS and Lustre. Doing > the appropriate namespace mapping in CIFS is easy (see this patch). So it's only > Lustre that's left with the original in-place xattr conversion. You know I really wish this was the case. And perhaps I need to look to see what has just been merged in the merge window. There are clearly some xattr changes post 4.6 that I am not seeing. As of 4.6.0 writing xattrs from a user namespace is broken by this change. > There have been cleanups in this area before, so I assume things were > fixed. In any case, I didn't blindly hack this together, I've actually checked > all the filesystems. You know that scares me. Because I just started looking and the first oddball case I checked was broken. > >> There is a complication that is comming shortly (next merge window >> unless major unfixable bugs show up between now and them). There is a >> new field s_user_ns that is being introduced for filesystems to record >> their owner and their default translation rules to kuids. That will >> make the hard coded &init_user_ns values in your patch wrong. > > Okay, that conflict should be easy enough to resolve. Where's that code? > >>>> Please Cc the user namespace maintainers before. Thank you! >>> >>> Eric, Andy, anyone else? >> >> Serge Hallyn has a pending patch that adds a similar translation to >> the security.capability xattr. Which is one more case of where caching >> and translation at the VFS layer are makes sense. > > No. Just like with this patch, that mapping really needs to go into the > appropriate xattr handler. Again, where's that code, please? I will start looking at what is in Linus's tree. But unless it happens to address my concerns, I am not even going to consider the notion that things should be in an ``xattr handler''. There are very good reasons why that conversion does not, nor should not happen inside of filesystem specific code. Eric From ebiederm at xmission.com Fri May 27 18:26:10 2016 From: ebiederm at xmission.com (Eric W. Biederman) Date: Fri, 27 May 2016 13:26:10 -0500 Subject: [lustre-devel] [PATCH] posix acls: Move namespace conversion into filesystem / xattr handlers In-Reply-To: <87bn3ridde.fsf@x220.int.ebiederm.org> (Eric W. Biederman's message of "Fri, 27 May 2016 13:07:09 -0500") References: <1464008989-3812-1-git-send-email-agruenba@redhat.com> <20160524153949.GA2830@dztty.fritz.box> <87h9dllnq2.fsf@x220.int.ebiederm.org> <87bn3ridde.fsf@x220.int.ebiederm.org> Message-ID: <87y46vfjct.fsf@x220.int.ebiederm.org> ebiederm at xmission.com (Eric W. Biederman) writes: >> >> No. Just like with this patch, that mapping really needs to go into the >> appropriate xattr handler. Again, where's that code, please? > > I will start looking at what is in Linus's tree. But unless it happens > to address my concerns, I am not even going to consider the notion that > things should be in an ``xattr handler''. > > There are very good reasons why that conversion does not, nor should not > happen inside of filesystem specific code. Ok. I have looked. The cleanups in Linus' tree seem worthwhile. The notion that mapping needs to go into a non-generic xattr handler is broken. If everything can always use a single set of acl handlers for posix acls then I will be happy to consider something like your patch. Fundamentally for vfs and security module supported attributes what a user writes in setxattr needs to be disconnected from what actually gets written to disk. Things evolve and userspace breaks if we don't handle the compatibility between old and new in the kernel. Eric