From jsimmons at infradead.org Thu Jul 2 00:04:40 2020 From: jsimmons at infradead.org (James Simmons) Date: Wed, 1 Jul 2020 20:04:40 -0400 Subject: [lustre-devel] [PATCH 00/18] Port of OpenSFS landing as of July 1, 2020 Message-ID: <1593648298-10571-1-git-send-email-jsimmons@infradead.org> Port of patches that landed to the OpenSFS branch. A few patches added the were missing that enables potential lustre utilities building against the Linux client. Please review to make sure everything is okay. Alexander Boyko (1): lustre: llite: don't hold inode_lock for security notify Alexey Lyashkov (2): lnet: restore an maximal fragments count lnet: o2ib: fix page mapping error Amir Shehata (1): lnet: handle undefined parameters Andriy Skulysh (1): lustre: llite: truncate deadlock with DoM files Chris Horn (1): lnet: Skip health and resends for single rail configs Emoly Liu (1): lustre: obd: add new LPROCFS_TYPE_* Gregoire Pichon (1): lnet: define new network driver ptl4lnd Hongchao Zhang (1): lustre: mdc: chlg device could be used after free James Simmons (2): lustre: llite: bind kthread thread to accepted node set lustre: lov: use lov_pattern_support() to verify lmm Lai Siyao (1): lustre: mdt: don't fetch LOOKUP lock for remote object Mikhail Pershin (1): lustre: ptlrpc: limit rate of lock replays Sebastien Buisson (5): lustre: sec: encryption for write path lustre: sec: decryption for read path lustre: sec: deal with encrypted object size lustre: sec: support truncate for encrypted files lustre: sec: ioctls to handle encryption policies fs/lustre/include/lprocfs_status.h | 9 +- fs/lustre/include/lustre_import.h | 2 + fs/lustre/include/lustre_osc.h | 1 + fs/lustre/include/obd.h | 19 ++- fs/lustre/include/obd_class.h | 3 +- fs/lustre/include/obd_support.h | 5 + fs/lustre/ldlm/ldlm_request.c | 69 ++++++++++- fs/lustre/llite/crypto.c | 15 ++- fs/lustre/llite/dir.c | 50 +++++++- fs/lustre/llite/file.c | 19 ++- fs/lustre/llite/llite_internal.h | 1 + fs/lustre/llite/llite_lib.c | 187 ++++++++++++++++++++++++++++- fs/lustre/llite/lproc_llite.c | 8 +- fs/lustre/llite/namei.c | 105 +++++++++++++--- fs/lustre/llite/rw.c | 13 +- fs/lustre/llite/rw26.c | 4 + fs/lustre/llite/statahead.c | 11 +- fs/lustre/llite/vvp_io.c | 17 ++- fs/lustre/lmv/lmv_intent.c | 19 ++- fs/lustre/lmv/lmv_internal.h | 1 + fs/lustre/lmv/lmv_obd.c | 3 +- fs/lustre/lov/lov_ea.c | 6 +- fs/lustre/mdc/mdc_changelog.c | 46 ++++--- fs/lustre/mdc/mdc_internal.h | 1 + fs/lustre/mdc/mdc_request.c | 8 +- fs/lustre/obdclass/genops.c | 1 + fs/lustre/obdecho/echo_client.c | 2 + fs/lustre/obdecho/echo_internal.h | 3 + fs/lustre/osc/osc_internal.h | 1 + fs/lustre/osc/osc_request.c | 121 ++++++++++++++++++- fs/lustre/ptlrpc/import.c | 8 +- include/linux/lnet/lib-lnet.h | 4 +- include/linux/lnet/lib-types.h | 2 +- include/uapi/linux/lnet/nidstr.h | 1 + include/uapi/linux/lustre/lustre_user.h | 8 ++ net/lnet/klnds/o2iblnd/o2iblnd.c | 7 +- net/lnet/klnds/o2iblnd/o2iblnd_cb.c | 3 +- net/lnet/klnds/o2iblnd/o2iblnd_modparams.c | 4 +- net/lnet/klnds/socklnd/socklnd_modparams.c | 4 +- net/lnet/lnet/api-ni.c | 26 +++- net/lnet/lnet/lib-msg.c | 65 +++++++--- net/lnet/lnet/nidstrings.c | 9 ++ 42 files changed, 759 insertions(+), 132 deletions(-) -- 1.8.3.1 From jsimmons at infradead.org Thu Jul 2 00:04:46 2020 From: jsimmons at infradead.org (James Simmons) Date: Wed, 1 Jul 2020 20:04:46 -0400 Subject: [lustre-devel] [PATCH 06/18] lustre: sec: support truncate for encrypted files In-Reply-To: <1593648298-10571-1-git-send-email-jsimmons@infradead.org> References: <1593648298-10571-1-git-send-email-jsimmons@infradead.org> Message-ID: <1593648298-10571-7-git-send-email-jsimmons@infradead.org> From: Sebastien Buisson Truncation of encrypted files is not a trivial operation. The page corresponding to the point where truncation occurs must be read, decrypted, zeroed after truncation point, re-encrypted and then written back. WC-bug-id: https://jira.whamcloud.com/browse/LU-12275 Lustre-commit: adf46db962f65 ("LU-12275 sec: support truncate for encrypted files") Signed-off-by: Sebastien Buisson Reviewed-on: https://review.whamcloud.com/37794 Reviewed-by: John L. Hammond Reviewed-by: Andreas Dilger Reviewed-by: Oleg Drokin Signed-off-by: James Simmons --- fs/lustre/llite/file.c | 7 ++ fs/lustre/llite/llite_lib.c | 182 +++++++++++++++++++++++++++++++++++++++++++- fs/lustre/llite/rw.c | 13 +++- fs/lustre/llite/vvp_io.c | 9 ++- fs/lustre/osc/osc_request.c | 7 +- 5 files changed, 211 insertions(+), 7 deletions(-) diff --git a/fs/lustre/llite/file.c b/fs/lustre/llite/file.c index 3b04952..55ae2b3 100644 --- a/fs/lustre/llite/file.c +++ b/fs/lustre/llite/file.c @@ -2086,6 +2086,13 @@ static int ll_lov_setstripe(struct inode *inode, struct file *file, goto out; rc = ll_file_getstripe(inode, arg, lum_size); + if (S_ISREG(inode->i_mode) && IS_ENCRYPTED(inode) && + ll_i2info(inode)->lli_clob) { + struct iattr attr = { 0 }; + + rc = cl_setattr_ost(ll_i2info(inode)->lli_clob, &attr, + OP_XVALID_FLAGS, LUSTRE_ENCRYPT_FL); + } } cl_lov_delay_create_clear(&file->f_flags); diff --git a/fs/lustre/llite/llite_lib.c b/fs/lustre/llite/llite_lib.c index aad19a2..0db9eae 100644 --- a/fs/lustre/llite/llite_lib.c +++ b/fs/lustre/llite/llite_lib.c @@ -1665,6 +1665,164 @@ static int ll_md_setattr(struct dentry *dentry, struct md_op_data *op_data) return rc; } +/** + * Zero portion of page that is part of @inode. + * This implies, if necessary: + * - taking cl_lock on range corresponding to concerned page + * - grabbing vm page + * - associating cl_page + * - proceeding to clio read + * - zeroing range in page + * - proceeding to cl_page flush + * - releasing cl_lock + * + * @inode inode + * @inde page index + * @offset offset in page to start zero from + * @len len to zero + * + * Return: 0 on success + * errno on failure + */ +int ll_io_zero_page(struct inode *inode, pgoff_t index, pgoff_t offset, + unsigned int len) +{ + struct ll_inode_info *lli = ll_i2info(inode); + struct cl_object *clob = lli->lli_clob; + u16 refcheck; + struct lu_env *env = NULL; + struct cl_io *io = NULL; + struct cl_page *clpage = NULL; + struct page *vmpage = NULL; + unsigned int from = index << PAGE_SHIFT; + struct cl_lock *lock = NULL; + struct cl_lock_descr *descr = NULL; + struct cl_2queue *queue = NULL; + struct cl_sync_io *anchor = NULL; + bool holdinglock = false; + bool lockedbymyself = true; + int rc; + + env = cl_env_get(&refcheck); + if (IS_ERR(env)) + return PTR_ERR(env); + + io = vvp_env_thread_io(env); + io->ci_obj = clob; + rc = cl_io_rw_init(env, io, CIT_WRITE, from, PAGE_SIZE); + if (rc) + goto putenv; + + lock = vvp_env_lock(env); + descr = &lock->cll_descr; + descr->cld_obj = io->ci_obj; + descr->cld_start = cl_index(io->ci_obj, from); + descr->cld_end = cl_index(io->ci_obj, from + PAGE_SIZE - 1); + descr->cld_mode = CLM_WRITE; + descr->cld_enq_flags = CEF_MUST | CEF_NONBLOCK; + + /* request lock for page */ + rc = cl_lock_request(env, io, lock); + /* -ECANCELED indicates a matching lock with a different extent + * was already present, and -EEXIST indicates a matching lock + * on exactly the same extent was already present. + * In both cases it means we are covered. + */ + if (rc == -ECANCELED || rc == -EEXIST) + rc = 0; + else if (rc < 0) + goto iofini; + else + holdinglock = true; + + /* grab page */ + vmpage = grab_cache_page_nowait(inode->i_mapping, index); + if (!vmpage) { + rc = -EOPNOTSUPP; + goto rellock; + } + + if (!PageDirty(vmpage)) { + /* associate cl_page */ + clpage = cl_page_find(env, clob, vmpage->index, + vmpage, CPT_CACHEABLE); + if (IS_ERR(clpage)) { + rc = PTR_ERR(clpage); + goto pagefini; + } + + cl_page_assume(env, io, clpage); + } + + if (!PageUptodate(vmpage) && !PageDirty(vmpage) && + !PageWriteback(vmpage)) { + /* read page */ + /* set PagePrivate2 to detect special case of empty page + * in osc_brw_fini_request() + */ + SetPagePrivate2(vmpage); + rc = ll_io_read_page(env, io, clpage, NULL); + if (!PagePrivate2(vmpage)) + /* PagePrivate2 was cleared in osc_brw_fini_request() + * meaning we read an empty page. In this case, in order + * to avoid allocating unnecessary block in truncated + * file, we must not zero and write as below. Subsequent + * server-side truncate will handle things correctly. + */ + goto clpfini; + ClearPagePrivate2(vmpage); + if (rc) + goto clpfini; + lockedbymyself = trylock_page(vmpage); + cl_page_assume(env, io, clpage); + } + + /* zero range in page */ + zero_user(vmpage, offset, len); + + if (holdinglock && clpage) { + /* explicitly write newly modified page */ + queue = &io->ci_queue; + cl_2queue_init(queue); + anchor = &vvp_env_info(env)->vti_anchor; + cl_sync_io_init(anchor, 1); + clpage->cp_sync_io = anchor; + cl_page_list_add(&queue->c2_qin, clpage); + rc = cl_io_submit_rw(env, io, CRT_WRITE, queue); + if (rc) + goto queuefini1; + rc = cl_sync_io_wait(env, anchor, 0); + if (rc) + goto queuefini2; + cl_page_assume(env, io, clpage); + +queuefini2: + cl_2queue_discard(env, io, queue); +queuefini1: + cl_2queue_disown(env, io, queue); + cl_2queue_fini(env, queue); + } + +clpfini: + if (clpage) + cl_page_put(env, clpage); +pagefini: + if (lockedbymyself) { + unlock_page(vmpage); + put_page(vmpage); + } +rellock: + if (holdinglock) + cl_lock_release(env, lock); +iofini: + cl_io_fini(env, io); +putenv: + if (env) + cl_env_put(env, &refcheck); + + return rc; +} + /* If this inode has objects allocated to it (lsm != NULL), then the OST * object(s) determine the file size and mtime. Otherwise, the MDS will * keep these values until such a time that objects are allocated for it. @@ -1798,6 +1956,8 @@ int ll_setattr_raw(struct dentry *dentry, struct iattr *attr, goto out; } } else { + unsigned int flags = 0; + /* For truncate and utimes sending attributes to OSTs, * setting mtime/atime to the past will be performed * under PW [0:EOF] extent lock (new_size:EOF for @@ -1806,8 +1966,23 @@ int ll_setattr_raw(struct dentry *dentry, struct iattr *attr, * it is necessary due to possible time * de-synchronization between MDT inode and OST objects */ + if (S_ISREG(inode->i_mode) && IS_ENCRYPTED(inode) && + attr->ia_valid & ATTR_SIZE) { + xvalid |= OP_XVALID_FLAGS; + flags = LUSTRE_ENCRYPT_FL; + if (attr->ia_size & ~PAGE_MASK) { + pgoff_t offset; + + offset = attr->ia_size & (PAGE_SIZE - 1); + rc = ll_io_zero_page(inode, + attr->ia_size >> PAGE_SHIFT, + offset, PAGE_SIZE - offset); + if (rc) + goto out; + } + } rc = cl_setattr_ost(ll_i2info(inode)->lli_clob, - attr, xvalid, 0); + attr, xvalid, flags); } } @@ -1875,6 +2050,11 @@ int ll_setattr(struct dentry *de, struct iattr *attr) { int mode = d_inode(de)->i_mode; enum op_xvalid xvalid = 0; + int rc; + + rc = llcrypt_prepare_setattr(de, attr); + if (rc) + return rc; if ((attr->ia_valid & (ATTR_CTIME | ATTR_SIZE | ATTR_MODE)) == (ATTR_CTIME | ATTR_SIZE | ATTR_MODE)) diff --git a/fs/lustre/llite/rw.c b/fs/lustre/llite/rw.c index ff8f3c6..54f0b9a 100644 --- a/fs/lustre/llite/rw.c +++ b/fs/lustre/llite/rw.c @@ -1453,8 +1453,8 @@ int ll_io_read_page(const struct lu_env *env, struct cl_io *io, struct cl_page *page, struct file *file) { struct inode *inode = vvp_object_inode(page->cp_obj); - struct ll_file_data *fd = file->private_data; - struct ll_readahead_state *ras = &fd->fd_ras; + struct ll_file_data *fd = NULL; + struct ll_readahead_state *ras = NULL; struct cl_2queue *queue = &io->ci_queue; struct ll_sb_info *sbi = ll_i2sbi(inode); struct cl_sync_io *anchor = NULL; @@ -1464,10 +1464,15 @@ int ll_io_read_page(const struct lu_env *env, struct cl_io *io, struct vvp_page *vpg; bool uptodate; + if (file) { + fd = file->private_data; + ras = &fd->fd_ras; + } + vpg = cl2vvp_page(cl_object_page_slice(page->cp_obj, page)); uptodate = vpg->vpg_defer_uptodate; - if (ll_readahead_enabled(sbi) && !vpg->vpg_ra_updated) { + if (ll_readahead_enabled(sbi) && !vpg->vpg_ra_updated && ras) { struct vvp_io *vio = vvp_env_io(env); enum ras_update_flags flags = 0; @@ -1494,7 +1499,7 @@ int ll_io_read_page(const struct lu_env *env, struct cl_io *io, io_start_index = cl_index(io->ci_obj, io->u.ci_rw.crw_pos); io_end_index = cl_index(io->ci_obj, io->u.ci_rw.crw_pos + io->u.ci_rw.crw_count - 1); - if (ll_readahead_enabled(sbi)) { + if (ll_readahead_enabled(sbi) && ras) { rc2 = ll_readahead(env, io, &queue->c2_qin, ras, uptodate, file); CDEBUG(D_READA, DFID " %d pages read ahead at %lu\n", diff --git a/fs/lustre/llite/vvp_io.c b/fs/lustre/llite/vvp_io.c index 371d988..8df5d39 100644 --- a/fs/lustre/llite/vvp_io.c +++ b/fs/lustre/llite/vvp_io.c @@ -620,7 +620,14 @@ static int vvp_io_setattr_lock(const struct lu_env *env, u32 enqflags = 0; if (cl_io_is_trunc(io)) { - if (io->u.ci_setattr.sa_attr.lvb_size == 0) + struct inode *inode = vvp_object_inode(io->ci_obj); + + /* set enqueue flags to CEF_MUST in case of encrypted file, + * to prevent lockless truncate + */ + if (S_ISREG(inode->i_mode) && IS_ENCRYPTED(inode)) + enqflags = CEF_MUST; + else if (io->u.ci_setattr.sa_attr.lvb_size == 0) enqflags = CEF_DISCARD_DATA; } else if (cl_io_is_fallocate(io)) { lock_start = io->u.ci_setattr.sa_falloc_offset; diff --git a/fs/lustre/osc/osc_request.c b/fs/lustre/osc/osc_request.c index b27a259..1968d62 100644 --- a/fs/lustre/osc/osc_request.c +++ b/fs/lustre/osc/osc_request.c @@ -2084,8 +2084,13 @@ static int osc_brw_fini_request(struct ptlrpc_request *req, int rc) break; p++; } - if (p - q == PAGE_SIZE / sizeof(*p)) + if (p - q == PAGE_SIZE / sizeof(*p)) { + /* if page is empty forward info to upper layers + * (ll_io_zero_page) by clearing PagePrivate2 + */ + ClearPagePrivate2(pg->pg); continue; + } rc = llcrypt_decrypt_pagecache_blocks(pg->pg, PAGE_SIZE, 0); -- 1.8.3.1 From jsimmons at infradead.org Thu Jul 2 00:04:43 2020 From: jsimmons at infradead.org (James Simmons) Date: Wed, 1 Jul 2020 20:04:43 -0400 Subject: [lustre-devel] [PATCH 03/18] lustre: sec: encryption for write path In-Reply-To: <1593648298-10571-1-git-send-email-jsimmons@infradead.org> References: <1593648298-10571-1-git-send-email-jsimmons@infradead.org> Message-ID: <1593648298-10571-4-git-send-email-jsimmons@infradead.org> From: Sebastien Buisson First aspect is to make sure encryption context is properly set on files/dirs that are created or opened/looked up. Then encryption itself is carried out in osc_brw_prep_request(), just before pages are added to the request to be sent. Because pages in the page cache must hold clear text data, we have to use bounce pages for encryption. The allocation is handled by fscrypt, and for deallocation we call fscrypt_pullback_bio_page() and/or fscrypt_pullback_bio_page(). WC-bug-id: https://jira.whamcloud.com/browse/LU-12275 Lustre-commit: a9ed5b149646f ("LU-12275 sec: encryption for write path") Signed-off-by: Sebastien Buisson Reviewed-on: https://review.whamcloud.com/36144 Reviewed-by: John L. Hammond Reviewed-by: Andreas Dilger Reviewed-by: Oleg Drokin Signed-off-by: James Simmons --- fs/lustre/include/lustre_osc.h | 1 + fs/lustre/include/obd.h | 17 ++++++++ fs/lustre/include/obd_support.h | 5 +++ fs/lustre/llite/crypto.c | 5 ++- fs/lustre/llite/dir.c | 16 +++++++ fs/lustre/llite/namei.c | 87 ++++++++++++++++++++++++++++++++++----- fs/lustre/llite/rw26.c | 4 ++ fs/lustre/obdecho/echo_client.c | 2 + fs/lustre/obdecho/echo_internal.h | 3 ++ fs/lustre/osc/osc_internal.h | 1 + fs/lustre/osc/osc_request.c | 77 +++++++++++++++++++++++++++++++++- 11 files changed, 206 insertions(+), 12 deletions(-) diff --git a/fs/lustre/include/lustre_osc.h b/fs/lustre/include/lustre_osc.h index 4b448b9..11b7e92 100644 --- a/fs/lustre/include/lustre_osc.h +++ b/fs/lustre/include/lustre_osc.h @@ -52,6 +52,7 @@ #include #include #include +#include struct osc_quota_info { /* linkage for quota hash table */ diff --git a/fs/lustre/include/obd.h b/fs/lustre/include/obd.h index 0ff19c8..f9e0920 100644 --- a/fs/lustre/include/obd.h +++ b/fs/lustre/include/obd.h @@ -116,6 +116,11 @@ struct brw_page { struct page *pg; unsigned int count; u32 flag; + /* used for encryption: difference with offset in clear text page */ + u16 bp_off_diff; + /* used for encryption: difference with count in clear text page */ + u16 bp_count_diff; + u32 bp_padding; }; struct timeout_item { @@ -1161,4 +1166,16 @@ static inline void client_adjust_max_dirty(struct client_obd *cli) 1 << (20 - PAGE_SHIFT)); } +static inline struct inode *page2inode(struct page *page) +{ + if (page->mapping) { + if (PageAnon(page)) + return NULL; + else + return page->mapping->host; + } else { + return NULL; + } +} + #endif /* __OBD_H */ diff --git a/fs/lustre/include/obd_support.h b/fs/lustre/include/obd_support.h index b5736f8..35c7ef3 100644 --- a/fs/lustre/include/obd_support.h +++ b/fs/lustre/include/obd_support.h @@ -583,4 +583,9 @@ struct obd_heat_instance { u64 ohi_count; }; +/* Define a fixed 4096-byte encryption unit size */ +#define LUSTRE_ENCRYPTION_BLOCKBITS 12 +#define LUSTRE_ENCRYPTION_UNIT_SIZE ((size_t)1 << LUSTRE_ENCRYPTION_BLOCKBITS) +#define LUSTRE_ENCRYPTION_MASK (~(LUSTRE_ENCRYPTION_UNIT_SIZE - 1)) + #endif diff --git a/fs/lustre/llite/crypto.c b/fs/lustre/llite/crypto.c index 94189c9..f411343 100644 --- a/fs/lustre/llite/crypto.c +++ b/fs/lustre/llite/crypto.c @@ -52,7 +52,7 @@ static int ll_set_context(struct inode *inode, const void *ctx, size_t len, struct ptlrpc_request *req = NULL; int rc; - if (inode == NULL) + if (!inode) return 0; ext_flags = ll_inode_to_ext_flags(inode->i_flags) | LUSTRE_ENCRYPT_FL; @@ -80,6 +80,9 @@ static int ll_set_context(struct inode *inode, const void *ctx, size_t len, if (rc) return rc; + /* used as encryption unit size */ + if (S_ISREG(inode->i_mode)) + inode->i_blkbits = LUSTRE_ENCRYPTION_BLOCKBITS; ll_update_inode_flags(inode, ext_flags); return 0; } diff --git a/fs/lustre/llite/dir.c b/fs/lustre/llite/dir.c index 0ffe134..2c93908 100644 --- a/fs/lustre/llite/dir.c +++ b/fs/lustre/llite/dir.c @@ -388,6 +388,7 @@ static int ll_dir_setdirstripe(struct dentry *dparent, struct lmv_user_md *lump, strlen(dirname)), }, }; + bool encrypt = false; int err; if (unlikely(!lmv_user_magic_supported(lump->lum_magic))) @@ -446,6 +447,18 @@ static int ll_dir_setdirstripe(struct dentry *dparent, struct lmv_user_md *lump, if (IS_ERR(op_data)) return PTR_ERR(op_data); + if (IS_ENCRYPTED(parent) || + unlikely(llcrypt_dummy_context_enabled(parent))) { + err = llcrypt_get_encryption_info(parent); + if (err) + goto out_op_data; + if (!llcrypt_has_encryption_key(parent)) { + err = -ENOKEY; + goto out_op_data; + } + encrypt = true; + } + if (sbi->ll_flags & LL_SBI_FILE_SECCTX) { /* * selinux_dentry_init_security() uses dentry->d_parent and name @@ -484,6 +497,9 @@ static int ll_dir_setdirstripe(struct dentry *dparent, struct lmv_user_md *lump, err = ll_inode_init_security(&dentry, inode, parent); } + if (encrypt) + err = llcrypt_inherit_context(parent, inode, NULL, false); + out_inode: iput(inode); out_request: diff --git a/fs/lustre/llite/namei.c b/fs/lustre/llite/namei.c index aa2dd13..2353a8f 100644 --- a/fs/lustre/llite/namei.c +++ b/fs/lustre/llite/namei.c @@ -47,7 +47,8 @@ #include "llite_internal.h" static int ll_create_it(struct inode *dir, struct dentry *dentry, - struct lookup_intent *it, void *secctx, u32 secctxlen); + struct lookup_intent *it, + void *secctx, u32 secctxlen, bool encrypt); /* called from iget5_locked->find_inode() under inode_hash_lock spinlock */ static int ll_test_inode(struct inode *inode, void *opaque) @@ -605,7 +606,7 @@ static int ll_lookup_it_finish(struct ptlrpc_request *request, struct lookup_intent *it, struct inode *parent, struct dentry **de, void *secctx, u32 secctxlen, - ktime_t kstart) + ktime_t kstart, bool encrypt) { struct inode *inode = NULL; u64 bits = 0; @@ -679,6 +680,16 @@ static int ll_lookup_it_finish(struct ptlrpc_request *request, /* We have the "lookup" lock, so unhide dentry */ if (bits & MDS_INODELOCK_LOOKUP) d_lustre_revalidate(*de); + + if (encrypt) { + rc = llcrypt_get_encryption_info(inode); + if (rc) + goto out; + if (!llcrypt_has_encryption_key(inode)) { + rc = -ENOKEY; + goto out; + } + } } else if (!it_disposition(it, DISP_OPEN_CREATE)) { /* * If file was created on the server, the dentry is revalidated @@ -725,7 +736,8 @@ static int ll_lookup_it_finish(struct ptlrpc_request *request, static struct dentry *ll_lookup_it(struct inode *parent, struct dentry *dentry, struct lookup_intent *it, void **secctx, u32 *secctxlen, - struct pcc_create_attach *pca) + struct pcc_create_attach *pca, + bool encrypt) { ktime_t kstart = ktime_get(); struct lookup_intent lookup_it = { .it_op = IT_LOOKUP }; @@ -894,7 +906,7 @@ static struct dentry *ll_lookup_it(struct inode *parent, struct dentry *dentry, rc = ll_lookup_it_finish(req, it, parent, &dentry, secctx ? *secctx : NULL, secctxlen ? *secctxlen : 0, - kstart); + kstart, encrypt); if (rc != 0) { ll_intent_release(it); retval = ERR_PTR(rc); @@ -952,7 +964,7 @@ static struct dentry *ll_lookup_nd(struct inode *parent, struct dentry *dentry, itp = NULL; else itp = ⁢ - de = ll_lookup_it(parent, dentry, itp, NULL, NULL, NULL); + de = ll_lookup_it(parent, dentry, itp, NULL, NULL, NULL, false); if (itp) ll_intent_release(itp); @@ -972,8 +984,9 @@ static int ll_atomic_open(struct inode *dir, struct dentry *dentry, void *secctx = NULL; u32 secctxlen = 0; struct dentry *de; - struct ll_sb_info *sbi; + struct ll_sb_info *sbi = NULL; struct pcc_create_attach pca = { NULL, NULL }; + bool encrypt = false; int rc = 0; CDEBUG(D_VFSTRACE, @@ -1025,8 +1038,23 @@ static int ll_atomic_open(struct inode *dir, struct dentry *dentry, it->it_flags = (open_flags & ~O_ACCMODE) | OPEN_FMODE(open_flags); it->it_flags &= ~MDS_OPEN_FL_INTERNAL; + if (IS_ENCRYPTED(dir)) { + /* we know that we are going to create a regular file because + * we set S_IFREG bit on it->it_create_mode above + */ + rc = llcrypt_get_encryption_info(dir); + if (rc) + goto out_release; + if (!llcrypt_has_encryption_key(dir)) { + rc = -ENOKEY; + goto out_release; + } + encrypt = true; + rc = 0; + } + /* Dentry added to dcache tree in ll_lookup_it */ - de = ll_lookup_it(dir, dentry, it, &secctx, &secctxlen, &pca); + de = ll_lookup_it(dir, dentry, it, &secctx, &secctxlen, &pca, encrypt); if (IS_ERR(de)) rc = PTR_ERR(de); else if (de) @@ -1035,7 +1063,8 @@ static int ll_atomic_open(struct inode *dir, struct dentry *dentry, if (!rc) { if (it_disposition(it, DISP_OPEN_CREATE)) { /* Dentry instantiated in ll_create_it. */ - rc = ll_create_it(dir, dentry, it, secctx, secctxlen); + rc = ll_create_it(dir, dentry, it, secctx, secctxlen, + encrypt); security_release_secctx(secctx, secctxlen); if (rc) { /* We dget in ll_splice_alias. */ @@ -1150,7 +1179,8 @@ static struct inode *ll_create_node(struct inode *dir, struct lookup_intent *it) * with d_instantiate(). */ static int ll_create_it(struct inode *dir, struct dentry *dentry, - struct lookup_intent *it, void *secctx, u32 secctxlen) + struct lookup_intent *it, + void *secctx, u32 secctxlen, bool encrypt) { struct inode *inode; u64 bits = 0; @@ -1185,6 +1215,12 @@ static int ll_create_it(struct inode *dir, struct dentry *dentry, d_instantiate(dentry, inode); + if (encrypt) { + rc = llcrypt_inherit_context(dir, inode, dentry, true); + if (rc) + return rc; + } + if (!(ll_i2sbi(inode)->ll_flags & LL_SBI_FILE_SECCTX)) rc = ll_inode_init_security(dentry, inode, dir); @@ -1214,10 +1250,11 @@ static int ll_new_node(struct inode *dir, struct dentry *dentry, u32 opc) { struct ptlrpc_request *request = NULL; - struct md_op_data *op_data; + struct md_op_data *op_data = NULL; struct inode *inode = NULL; struct ll_sb_info *sbi = ll_i2sbi(dir); int tgt_len = 0; + int encrypt = 0; int err; if (unlikely(tgt)) @@ -1241,6 +1278,19 @@ static int ll_new_node(struct inode *dir, struct dentry *dentry, goto err_exit; } + if ((IS_ENCRYPTED(dir) && + (S_ISREG(mode) || S_ISDIR(mode) || S_ISLNK(mode))) || + (unlikely(llcrypt_dummy_context_enabled(dir)) && S_ISDIR(mode))) { + err = llcrypt_get_encryption_info(dir); + if (err) + goto err_exit; + if (!llcrypt_has_encryption_key(dir)) { + err = -ENOKEY; + goto err_exit; + } + encrypt = 1; + } + err = md_create(sbi->ll_md_exp, op_data, tgt, tgt_len, mode, from_kuid(&init_user_ns, current_fsuid()), from_kgid(&init_user_ns, current_fsgid()), @@ -1335,6 +1385,12 @@ static int ll_new_node(struct inode *dir, struct dentry *dentry, d_instantiate(dentry, inode); + if (encrypt) { + err = llcrypt_inherit_context(dir, inode, NULL, true); + if (err) + goto err_exit; + } + if (!(sbi->ll_flags & LL_SBI_FILE_SECCTX)) err = ll_inode_init_security(dentry, inode, dir); err_exit: @@ -1547,6 +1603,10 @@ static int ll_link(struct dentry *old_dentry, struct inode *dir, PFID(ll_inode2fid(src)), src, PFID(ll_inode2fid(dir)), dir, new_dentry); + err = llcrypt_prepare_link(old_dentry, dir, new_dentry); + if (err) + return err; + op_data = ll_prep_md_op_data(NULL, src, dir, new_dentry->d_name.name, new_dentry->d_name.len, 0, LUSTRE_OPC_ANY, NULL); @@ -1584,6 +1644,13 @@ static int ll_rename(struct inode *src, struct dentry *src_dchild, src_dchild, PFID(ll_inode2fid(src)), src, tgt_dchild, PFID(ll_inode2fid(tgt)), tgt); + if (unlikely(d_mountpoint(src_dchild) || d_mountpoint(tgt_dchild))) + return -EBUSY; + + err = llcrypt_prepare_rename(src, src_dchild, tgt, tgt_dchild, flags); + if (err) + return err; + op_data = ll_prep_md_op_data(NULL, src, tgt, NULL, 0, 0, LUSTRE_OPC_ANY, NULL); if (IS_ERR(op_data)) diff --git a/fs/lustre/llite/rw26.c b/fs/lustre/llite/rw26.c index 5e7aa6e..0971185 100644 --- a/fs/lustre/llite/rw26.c +++ b/fs/lustre/llite/rw26.c @@ -291,6 +291,10 @@ static ssize_t ll_direct_IO(struct kiocb *iocb, struct iov_iter *iter) loff_t file_offset = iocb->ki_pos; int rw = iov_iter_rw(iter); + /* if file is encrypted, return 0 so that we fall back to buffered IO */ + if (IS_ENCRYPTED(inode)) + return 0; + /* Check EOF by ourselves */ if (rw == READ && file_offset >= i_size_read(inode)) return 0; diff --git a/fs/lustre/obdecho/echo_client.c b/fs/lustre/obdecho/echo_client.c index 2324e38..a52e0362 100644 --- a/fs/lustre/obdecho/echo_client.c +++ b/fs/lustre/obdecho/echo_client.c @@ -1317,6 +1317,8 @@ static int echo_client_kbrw(struct echo_device *ed, int rw, struct obdo *oa, if (!pgp->pg) goto out; + /* set mapping so page is not considered encrypted */ + pgp->pg->mapping = ECHO_MAPPING_UNENCRYPTED; pages[i] = pgp->pg; pgp->count = PAGE_SIZE; pgp->off = off; diff --git a/fs/lustre/obdecho/echo_internal.h b/fs/lustre/obdecho/echo_internal.h index f9bb0b91..95b0149 100644 --- a/fs/lustre/obdecho/echo_internal.h +++ b/fs/lustre/obdecho/echo_internal.h @@ -43,4 +43,7 @@ int block_debug_setup(void *addr, int len, u64 off, u64 id); int block_debug_check(char *who, void *addr, int len, u64 off, u64 id); +/* mapping value to tell page is not encrypted */ +#define ECHO_MAPPING_UNENCRYPTED ((void *)1) + #endif diff --git a/fs/lustre/osc/osc_internal.h b/fs/lustre/osc/osc_internal.h index d05595a..6bec6bf 100644 --- a/fs/lustre/osc/osc_internal.h +++ b/fs/lustre/osc/osc_internal.h @@ -216,4 +216,5 @@ static inline void osc_set_io_portal(struct ptlrpc_request *req) else req->rq_request_portal = OST_IO_PORTAL; } + #endif /* OSC_INTERNAL_H */ diff --git a/fs/lustre/osc/osc_request.c b/fs/lustre/osc/osc_request.c index b1bf8c6..db97d37 100644 --- a/fs/lustre/osc/osc_request.c +++ b/fs/lustre/osc/osc_request.c @@ -36,6 +36,7 @@ #include #include #include +#include #include #include @@ -1354,6 +1355,26 @@ static int osc_checksum_bulk_rw(const char *obd_name, return rc; } +static inline void osc_release_bounce_pages(struct brw_page **pga, + u32 page_count) +{ +#ifdef CONFIG_FS_ENCRYPTION + int i; + + for (i = 0; i < page_count; i++) { + if (pga[i]->pg->mapping) + /* bounce pages are unmapped */ + continue; + if (pga[i]->flag & OBD_BRW_SYNC) + /* sync transfer cannot have encrypted pages */ + continue; + llcrypt_finalize_bounce_page(&pga[i]->pg); + pga[i]->count -= pga[i]->bp_count_diff; + pga[i]->off += pga[i]->bp_off_diff; + } +#endif +} + static int osc_brw_prep_request(int cmd, struct client_obd *cli, struct obdo *oa, u32 page_count, struct brw_page **pga, @@ -1371,7 +1392,9 @@ static int osc_brw_prep_request(int cmd, struct client_obd *cli, struct brw_page *pg_prev; void *short_io_buf; const char *obd_name = cli->cl_import->imp_obd->obd_name; + struct inode *inode; + inode = page2inode(pga[0]->pg); if (OBD_FAIL_CHECK(OBD_FAIL_OSC_BRW_PREP_REQ)) return -ENOMEM; /* Recoverable */ if (OBD_FAIL_CHECK(OBD_FAIL_OSC_BRW_PREP_REQ2)) @@ -1389,6 +1412,51 @@ static int osc_brw_prep_request(int cmd, struct client_obd *cli, if (!req) return -ENOMEM; + if (opc == OST_WRITE && inode && IS_ENCRYPTED(inode)) { + for (i = 0; i < page_count; i++) { + struct brw_page *pg = pga[i]; + struct page *data_page = NULL; + bool retried = false; + bool lockedbymyself; + +retry_encrypt: + /* The page can already be locked when we arrive here. + * This is possible when cl_page_assume/vvp_page_assume + * is stuck on wait_on_page_writeback with page lock + * held. In this case there is no risk for the lock to + * be released while we are doing our encryption + * processing, because writeback against that page will + * end in vvp_page_completion_write/cl_page_completion, + * which means only once the page is fully processed. + */ + lockedbymyself = trylock_page(pg->pg); + data_page = + llcrypt_encrypt_pagecache_blocks(pg->pg, + PAGE_SIZE, 0, + GFP_NOFS); + if (lockedbymyself) + unlock_page(pg->pg); + if (IS_ERR(data_page)) { + rc = PTR_ERR(data_page); + if (rc == -ENOMEM && !retried) { + retried = true; + rc = 0; + goto retry_encrypt; + } + ptlrpc_request_free(req); + return rc; + } + /* len is forced to PAGE_SIZE, and poff to 0 + * so store the old, clear text info + */ + pg->pg = data_page; + pg->bp_count_diff = PAGE_SIZE - pg->count; + pg->count = PAGE_SIZE; + pg->bp_off_diff = pg->off & ~PAGE_MASK; + pg->off = pg->off & PAGE_MASK; + } + } + for (niocount = i = 1; i < page_count; i++) { if (!can_merge_pages(pga[i - 1], pga[i])) niocount++; @@ -2115,6 +2183,10 @@ static int brw_interpret(const struct lu_env *env, rc = osc_brw_fini_request(req, rc); CDEBUG(D_INODE, "request %p aa %p rc %d\n", req, aa, rc); + + /* restore clear text pages */ + osc_release_bounce_pages(aa->aa_ppga, aa->aa_page_count); + /* * When server returns -EINPROGRESS, client should always retry * regardless of the number of times the bulk was resent already. @@ -2430,7 +2502,10 @@ int osc_build_rpc(const struct lu_env *env, struct client_obd *cli, LASSERT(!req); kmem_cache_free(osc_obdo_kmem, oa); - kfree(pga); + if (pga) { + osc_release_bounce_pages(pga, page_count); + osc_release_ppga(pga, page_count); + } /* this should happen rarely and is pretty bad, it makes the * pending list not follow the dirty order */ -- 1.8.3.1 From jsimmons at infradead.org Thu Jul 2 00:04:42 2020 From: jsimmons at infradead.org (James Simmons) Date: Wed, 1 Jul 2020 20:04:42 -0400 Subject: [lustre-devel] [PATCH 02/18] lnet: o2ib: fix page mapping error In-Reply-To: <1593648298-10571-1-git-send-email-jsimmons@infradead.org> References: <1593648298-10571-1-git-send-email-jsimmons@infradead.org> Message-ID: <1593648298-10571-3-git-send-email-jsimmons@infradead.org> From: Alexey Lyashkov IB DMA mapping can merge a physically continues page region into single one. It's confused a kiblnd_fmr_pool_map function who expect to see all fragments mapped. It's generate a error (o2iblnd.c:1926:kiblnd_fmr_pool_map()) Failed to map mr 1/16 elements By study an IB code, it looks ib_map_mr_sg return code should checked against of result of ib_dma_map_sg instead of original fragments count, same data should be used as argument of ib_map_mr_sg function. Cray-bug-id: LUS-8139 WC-bug-id: https://jira.whamcloud.com/browse/LU-13181 Lustre-commit: 40385cda7afbd ("LU-13181 o2ib: fix page mapping error") Signed-off-by: Alexey Lyashkov Reviewed-on: https://review.whamcloud.com/37388 Reviewed-by: Shaun Tancheff Reviewed-by: Alexander Boyko Reviewed-by: James Simmons Reviewed-by: Oleg Drokin Signed-off-by: James Simmons --- net/lnet/klnds/o2iblnd/o2iblnd.c | 7 ++++--- net/lnet/klnds/o2iblnd/o2iblnd_cb.c | 3 ++- 2 files changed, 6 insertions(+), 4 deletions(-) diff --git a/net/lnet/klnds/o2iblnd/o2iblnd.c b/net/lnet/klnds/o2iblnd/o2iblnd.c index 3a76447..16edfba 100644 --- a/net/lnet/klnds/o2iblnd/o2iblnd.c +++ b/net/lnet/klnds/o2iblnd/o2iblnd.c @@ -1737,10 +1737,11 @@ int kiblnd_fmr_pool_map(struct kib_fmr_poolset *fps, struct kib_tx *tx, } n = ib_map_mr_sg(mr, tx->tx_frags, - tx->tx_nfrags, NULL, PAGE_SIZE); - if (unlikely(n != tx->tx_nfrags)) { + rd->rd_nfrags, NULL, + PAGE_SIZE); + if (unlikely(n != rd->rd_nfrags)) { CERROR("Failed to map mr %d/%d elements\n", - n, tx->tx_nfrags); + n, rd->rd_nfrags); return n < 0 ? n : -EINVAL; } diff --git a/net/lnet/klnds/o2iblnd/o2iblnd_cb.c b/net/lnet/klnds/o2iblnd/o2iblnd_cb.c index 40e196d..3b9d10d 100644 --- a/net/lnet/klnds/o2iblnd/o2iblnd_cb.c +++ b/net/lnet/klnds/o2iblnd/o2iblnd_cb.c @@ -595,7 +595,8 @@ static int kiblnd_init_rdma(struct kib_conn *conn, struct kib_tx *tx, int type, fps = net->ibn_fmr_ps[cpt]; rc = kiblnd_fmr_pool_map(fps, tx, rd, nob, 0, &tx->tx_fmr); if (rc) { - CERROR("Can't map %u bytes: %d\n", nob, rc); + CERROR("Can't map %u bytes (%u/%u)s: %d\n", nob, + tx->tx_nfrags, rd->rd_nfrags, rc); return rc; } -- 1.8.3.1 From jsimmons at infradead.org Thu Jul 2 00:04:49 2020 From: jsimmons at infradead.org (James Simmons) Date: Wed, 1 Jul 2020 20:04:49 -0400 Subject: [lustre-devel] [PATCH 09/18] lustre: llite: bind kthread thread to accepted node set In-Reply-To: <1593648298-10571-1-git-send-email-jsimmons@infradead.org> References: <1593648298-10571-1-git-send-email-jsimmons@infradead.org> Message-ID: <1593648298-10571-10-git-send-email-jsimmons@infradead.org> Bind both the agl and statahead kernel threads to a node that is apart of the cpt table that Lustre use. This limits the polluting of the cache of HPC applications. WC-bug-id: https://jira.whamcloud.com/browse/LU-9441 Lustre-commit: d6e103e6950d9 ("LU-9441 llite: bind kthread thread to accepted node set") Signed-off-by: James Simmons Reviewed-on: https://review.whamcloud.com/38730 Reviewed-by: Yang Sheng Reviewed-by: Andreas Dilger Reviewed-by: Oleg Drokin --- fs/lustre/llite/statahead.c | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-) diff --git a/fs/lustre/llite/statahead.c b/fs/lustre/llite/statahead.c index fb25520..895e496 100644 --- a/fs/lustre/llite/statahead.c +++ b/fs/lustre/llite/statahead.c @@ -39,6 +39,7 @@ #define DEBUG_SUBSYSTEM S_LLITE +#include #include #include #include "llite_internal.h" @@ -954,6 +955,7 @@ static void ll_stop_agl(struct ll_statahead_info *sai) /* start agl thread */ static void ll_start_agl(struct dentry *parent, struct ll_statahead_info *sai) { + int node = cfs_cpt_spread_node(cfs_cpt_tab, CFS_CPT_ANY); struct ll_inode_info *plli; struct task_struct *task; @@ -961,8 +963,8 @@ static void ll_start_agl(struct dentry *parent, struct ll_statahead_info *sai) sai, parent); plli = ll_i2info(d_inode(parent)); - task = kthread_create(ll_agl_thread, parent, "ll_agl_%u", - plli->lli_opendir_pid); + task = kthread_create_on_node(ll_agl_thread, parent, node, "ll_agl_%u", + plli->lli_opendir_pid); if (IS_ERR(task)) { CERROR("can't start ll_agl thread, rc: %ld\n", PTR_ERR(task)); sai->sai_agl_valid = 0; @@ -1535,6 +1537,7 @@ static int revalidate_statahead_dentry(struct inode *dir, static int start_statahead_thread(struct inode *dir, struct dentry *dentry, bool agl) { + int node = cfs_cpt_spread_node(cfs_cpt_tab, CFS_CPT_ANY); struct ll_inode_info *lli = ll_i2info(dir); struct ll_statahead_info *sai = NULL; struct task_struct *task; @@ -1586,8 +1589,8 @@ static int start_statahead_thread(struct inode *dir, struct dentry *dentry, CDEBUG(D_READA, "start statahead thread: [pid %d] [parent %pd]\n", current->pid, parent); - task = kthread_create(ll_statahead_thread, parent, "ll_sa_%u", - lli->lli_opendir_pid); + task = kthread_create_on_node(ll_statahead_thread, parent, node, + "ll_sa_%u", lli->lli_opendir_pid); if (IS_ERR(task)) { spin_lock(&lli->lli_sa_lock); lli->lli_sai = NULL; -- 1.8.3.1 From jsimmons at infradead.org Thu Jul 2 00:04:53 2020 From: jsimmons at infradead.org (James Simmons) Date: Wed, 1 Jul 2020 20:04:53 -0400 Subject: [lustre-devel] [PATCH 13/18] lustre: sec: ioctls to handle encryption policies In-Reply-To: <1593648298-10571-1-git-send-email-jsimmons@infradead.org> References: <1593648298-10571-1-git-send-email-jsimmons@infradead.org> Message-ID: <1593648298-10571-14-git-send-email-jsimmons@infradead.org> From: Sebastien Buisson Introduce support for fscrypt IOCTLs that handle encryption policies v2. It enables setting/getting encryption policies on individual directories, letting users decide how they want to encrypt specific directories. fscrypt encryption policies v2 are supported from Linux 5.4. WC-bug-id: https://jira.whamcloud.com/browse/LU-12275 Lustre-commit: 3973cf8dc955c ("LU-12275 sec: ioctls to handle encryption policies") Signed-off-by: Sebastien Buisson Reviewed-on: https://review.whamcloud.com/37673 Reviewed-by: Andreas Dilger Reviewed-by: John L. Hammond Reviewed-by: James Simmons Reviewed-by: Oleg Drokin Signed-off-by: James Simmons --- fs/lustre/llite/dir.c | 28 ++++++++++++++++++++++++++++ 1 file changed, 28 insertions(+) diff --git a/fs/lustre/llite/dir.c b/fs/lustre/llite/dir.c index 2c93908..463c5d7 100644 --- a/fs/lustre/llite/dir.c +++ b/fs/lustre/llite/dir.c @@ -48,6 +48,7 @@ #include #include +#include #include #include #include @@ -2103,6 +2104,33 @@ static long ll_dir_ioctl(struct file *file, unsigned int cmd, unsigned long arg) kfree(detach); return rc; } +#ifdef CONFIG_FS_ENCRYPTION + case FS_IOC_SET_ENCRYPTION_POLICY: + if (!ll_sbi_has_encrypt(ll_i2sbi(inode))) + return -EOPNOTSUPP; + return llcrypt_ioctl_set_policy(file, (const void __user *)arg); + case FS_IOC_GET_ENCRYPTION_POLICY_EX: + if (!ll_sbi_has_encrypt(ll_i2sbi(inode))) + return -EOPNOTSUPP; + return llcrypt_ioctl_get_policy_ex(file, (void __user *)arg); + case FS_IOC_ADD_ENCRYPTION_KEY: + if (!ll_sbi_has_encrypt(ll_i2sbi(inode))) + return -EOPNOTSUPP; + return llcrypt_ioctl_add_key(file, (void __user *)arg); + case FS_IOC_REMOVE_ENCRYPTION_KEY: + if (!ll_sbi_has_encrypt(ll_i2sbi(inode))) + return -EOPNOTSUPP; + return llcrypt_ioctl_remove_key(file, (void __user *)arg); + case FS_IOC_REMOVE_ENCRYPTION_KEY_ALL_USERS: + if (!ll_sbi_has_encrypt(ll_i2sbi(inode))) + return -EOPNOTSUPP; + return llcrypt_ioctl_remove_key_all_users(file, + (void __user *)arg); + case FS_IOC_GET_ENCRYPTION_KEY_STATUS: + if (!ll_sbi_has_encrypt(ll_i2sbi(inode))) + return -EOPNOTSUPP; + return llcrypt_ioctl_get_key_status(file, (void __user *)arg); +#endif default: return obd_iocontrol(cmd, sbi->ll_dt_exp, 0, NULL, (void __user *)arg); -- 1.8.3.1 From jsimmons at infradead.org Thu Jul 2 00:04:56 2020 From: jsimmons at infradead.org (James Simmons) Date: Wed, 1 Jul 2020 20:04:56 -0400 Subject: [lustre-devel] [PATCH 16/18] lustre: mdt: don't fetch LOOKUP lock for remote object In-Reply-To: <1593648298-10571-1-git-send-email-jsimmons@infradead.org> References: <1593648298-10571-1-git-send-email-jsimmons@infradead.org> Message-ID: <1593648298-10571-17-git-send-email-jsimmons@infradead.org> From: Lai Siyao Pack parent FID in getattr by FID, which will be used to check whether child is remote object on parent. The helper function is called mdt_is_remote_object(). NB, directory shard is not treated as remote object, because if so, client needs to revalidate shards when dir is accessed, which will hurt performance much. For getattr by FID, if object is remote file on parent, don't fetch LOOKUP lock, otherwise client may see stale dir entries. WC-bug-id: https://jira.whamcloud.com/browse/LU-13437 Lustre-commit: f9a2da63abab5 ("LU-13437 mdt: don't fetch LOOKUP lock for remote object") Signed-off-by: Lai Siyao Reviewed-on: https://review.whamcloud.com/38561 Reviewed-by: Andreas Dilger Reviewed-by: Yingjin Qian Reviewed-by: Oleg Drokin Signed-off-by: James Simmons --- fs/lustre/include/obd.h | 2 +- fs/lustre/include/obd_class.h | 3 ++- fs/lustre/llite/file.c | 6 +++--- fs/lustre/llite/llite_lib.c | 4 ++-- fs/lustre/lmv/lmv_intent.c | 19 +++++++++++++------ fs/lustre/lmv/lmv_internal.h | 1 + fs/lustre/lmv/lmv_obd.c | 3 ++- 7 files changed, 24 insertions(+), 14 deletions(-) diff --git a/fs/lustre/include/obd.h b/fs/lustre/include/obd.h index f9e0920..438f4ca 100644 --- a/fs/lustre/include/obd.h +++ b/fs/lustre/include/obd.h @@ -1004,7 +1004,7 @@ struct md_ops { int (*free_lustre_md)(struct obd_export *, struct lustre_md *); - int (*merge_attr)(struct obd_export *, + int (*merge_attr)(struct obd_export *, const struct lu_fid *fid, const struct lmv_stripe_md *lsm, struct cl_attr *attr, ldlm_blocking_callback); diff --git a/fs/lustre/include/obd_class.h b/fs/lustre/include/obd_class.h index 746782b..78f7b16 100644 --- a/fs/lustre/include/obd_class.h +++ b/fs/lustre/include/obd_class.h @@ -1458,6 +1458,7 @@ static inline int md_free_lustre_md(struct obd_export *exp, } static inline int md_merge_attr(struct obd_export *exp, + const struct lu_fid *fid, const struct lmv_stripe_md *lsm, struct cl_attr *attr, ldlm_blocking_callback cb) @@ -1468,7 +1469,7 @@ static inline int md_merge_attr(struct obd_export *exp, if (rc) return rc; - return MDP(exp->exp_obd, merge_attr)(exp, lsm, attr, cb); + return MDP(exp->exp_obd, merge_attr)(exp, fid, lsm, attr, cb); } static inline int md_setxattr(struct obd_export *exp, const struct lu_fid *fid, diff --git a/fs/lustre/llite/file.c b/fs/lustre/llite/file.c index 55ae2b3..1849229 100644 --- a/fs/lustre/llite/file.c +++ b/fs/lustre/llite/file.c @@ -4500,8 +4500,8 @@ static int ll_inode_revalidate(struct dentry *dentry, enum ldlm_intent_flags op) PFID(ll_inode2fid(inode)), inode, dentry); /* Call getattr by fid, so do not provide name at all. */ - op_data = ll_prep_md_op_data(NULL, inode, inode, NULL, 0, 0, - LUSTRE_OPC_ANY, NULL); + op_data = ll_prep_md_op_data(NULL, dentry->d_parent->d_inode, inode, + NULL, 0, 0, LUSTRE_OPC_ANY, NULL); if (IS_ERR(op_data)) return PTR_ERR(op_data); @@ -4548,7 +4548,7 @@ static int ll_merge_md_attr(struct inode *inode) return 0; down_read(&lli->lli_lsm_sem); - rc = md_merge_attr(ll_i2mdexp(inode), ll_i2info(inode)->lli_lsm_md, + rc = md_merge_attr(ll_i2mdexp(inode), &lli->lli_fid, lli->lli_lsm_md, &attr, ll_md_blocking_ast); up_read(&lli->lli_lsm_sem); if (rc) diff --git a/fs/lustre/llite/llite_lib.c b/fs/lustre/llite/llite_lib.c index b30feb0..1a7d805 100644 --- a/fs/lustre/llite/llite_lib.c +++ b/fs/lustre/llite/llite_lib.c @@ -1525,8 +1525,8 @@ static int ll_update_lsm_md(struct inode *inode, struct lustre_md *md) } /* validate the lsm */ - rc = md_merge_attr(ll_i2mdexp(inode), lli->lli_lsm_md, attr, - ll_md_blocking_ast); + rc = md_merge_attr(ll_i2mdexp(inode), &lli->lli_fid, lli->lli_lsm_md, + attr, ll_md_blocking_ast); if (!rc) { if (md->body->mbo_valid & OBD_MD_FLNLINK) md->body->mbo_nlink = attr->cat_nlink; diff --git a/fs/lustre/lmv/lmv_intent.c b/fs/lustre/lmv/lmv_intent.c index a847770..4af449e 100644 --- a/fs/lustre/lmv/lmv_intent.c +++ b/fs/lustre/lmv/lmv_intent.c @@ -153,6 +153,7 @@ static int lmv_intent_remote(struct obd_export *exp, struct lookup_intent *it, } int lmv_revalidate_slaves(struct obd_export *exp, + const struct lu_fid *pfid, const struct lmv_stripe_md *lsm, ldlm_blocking_callback cb_blocking, int extra_lock_flags) @@ -196,7 +197,7 @@ int lmv_revalidate_slaves(struct obd_export *exp, * which is not needed here. */ memset(op_data, 0, sizeof(*op_data)); - op_data->op_fid1 = fid; + op_data->op_fid1 = *pfid; op_data->op_fid2 = fid; tgt = lmv_tgt(lmv, lsm->lsm_md_oinfo[i].lmo_mds); @@ -444,13 +445,18 @@ static int lmv_intent_lookup(struct obd_export *exp, } retry: - tgt = lmv_locate_tgt(lmv, op_data); + if (op_data->op_name) { + tgt = lmv_locate_tgt(lmv, op_data); + if (!fid_is_sane(&op_data->op_fid2)) + fid_zero(&op_data->op_fid2); + } else if (fid_is_sane(&op_data->op_fid2)) { + tgt = lmv_fid2tgt(lmv, &op_data->op_fid2); + } else { + tgt = lmv_fid2tgt(lmv, &op_data->op_fid1); + } if (IS_ERR(tgt)) return PTR_ERR(tgt); - if (!fid_is_sane(&op_data->op_fid2)) - fid_zero(&op_data->op_fid2); - CDEBUG(D_INODE, "LOOKUP_INTENT with fid1=" DFID ", fid2=" DFID ", name='%s' -> mds #%u\n", PFID(&op_data->op_fid1), PFID(&op_data->op_fid2), @@ -470,7 +476,8 @@ static int lmv_intent_lookup(struct obd_export *exp, * during update_inode process (see ll_update_lsm_md) */ if (lmv_dir_striped(op_data->op_mea2)) { - rc = lmv_revalidate_slaves(exp, op_data->op_mea2, + rc = lmv_revalidate_slaves(exp, &op_data->op_fid2, + op_data->op_mea2, cb_blocking, extra_lock_flags); if (rc != 0) diff --git a/fs/lustre/lmv/lmv_internal.h b/fs/lustre/lmv/lmv_internal.h index e42b141..756fa27 100644 --- a/fs/lustre/lmv/lmv_internal.h +++ b/fs/lustre/lmv/lmv_internal.h @@ -53,6 +53,7 @@ int lmv_fid_alloc(const struct lu_env *env, struct obd_export *exp, struct lu_fid *fid, struct md_op_data *op_data); int lmv_revalidate_slaves(struct obd_export *exp, + const struct lu_fid *pfid, const struct lmv_stripe_md *lsm, ldlm_blocking_callback cb_blocking, int extra_lock_flags); diff --git a/fs/lustre/lmv/lmv_obd.c b/fs/lustre/lmv/lmv_obd.c index c5f21cd..4131b49 100644 --- a/fs/lustre/lmv/lmv_obd.c +++ b/fs/lustre/lmv/lmv_obd.c @@ -3477,6 +3477,7 @@ static int lmv_quotactl(struct obd_device *unused, struct obd_export *exp, } static int lmv_merge_attr(struct obd_export *exp, + const struct lu_fid *fid, const struct lmv_stripe_md *lsm, struct cl_attr *attr, ldlm_blocking_callback cb_blocking) @@ -3486,7 +3487,7 @@ static int lmv_merge_attr(struct obd_export *exp, if (!lmv_dir_striped(lsm)) return 0; - rc = lmv_revalidate_slaves(exp, lsm, cb_blocking, 0); + rc = lmv_revalidate_slaves(exp, fid, lsm, cb_blocking, 0); if (rc < 0) return rc; -- 1.8.3.1 From jsimmons at infradead.org Thu Jul 2 00:04:51 2020 From: jsimmons at infradead.org (James Simmons) Date: Wed, 1 Jul 2020 20:04:51 -0400 Subject: [lustre-devel] [PATCH 11/18] lustre: llite: truncate deadlock with DoM files In-Reply-To: <1593648298-10571-1-git-send-email-jsimmons@infradead.org> References: <1593648298-10571-1-git-send-email-jsimmons@infradead.org> Message-ID: <1593648298-10571-12-git-send-email-jsimmons@infradead.org> From: Andriy Skulysh All MDT intent RPCs are sent with inode mutex locked while read/write and setattr unlocks inode mutex on entry, takes LDLM lock and locks inode mutex again and sends the RPC. So a deadlock can occur since LDLM lock is the same in case of DoM. In fact read/write and setattr takes lli_trunc_sem, so inode mutex can be ommited in truncate case. Replace inode_lock with new lli_setattr_mutex to keep protection from concurrent setattr time updates. HPE-bug-id: LUS-8455 WC-bug-id: https://jira.whamcloud.com/browse/LU-13467 Lustre-commit: 8958ecee22010 ("LU-13467 llite: truncate deadlock with DoM files") Signed-off-by: Andriy Skulysh Reviewed-on: https://review.whamcloud.com/38288 Reviewed-by: Andrew Perepechko Reviewed-by: Vitaly Fertman Reviewed-by: Mike Pershin Reviewed-by: Andreas Dilger Reviewed-by: Oleg Drokin Signed-off-by: James Simmons --- fs/lustre/llite/llite_internal.h | 1 + fs/lustre/llite/llite_lib.c | 1 + fs/lustre/llite/vvp_io.c | 8 ++++---- 3 files changed, 6 insertions(+), 4 deletions(-) diff --git a/fs/lustre/llite/llite_internal.h b/fs/lustre/llite/llite_internal.h index a688bd8..2556dd8 100644 --- a/fs/lustre/llite/llite_internal.h +++ b/fs/lustre/llite/llite_internal.h @@ -191,6 +191,7 @@ struct ll_inode_info { char *lli_symlink_name; struct ll_trunc_sem lli_trunc_sem; struct range_lock_tree lli_write_tree; + struct mutex lli_setattr_mutex; struct rw_semaphore lli_glimpse_sem; ktime_t lli_glimpse_time; diff --git a/fs/lustre/llite/llite_lib.c b/fs/lustre/llite/llite_lib.c index 0db9eae..b30feb0 100644 --- a/fs/lustre/llite/llite_lib.c +++ b/fs/lustre/llite/llite_lib.c @@ -1014,6 +1014,7 @@ void ll_lli_init(struct ll_inode_info *lli) init_rwsem(&lli->lli_lsm_sem); } else { mutex_init(&lli->lli_size_mutex); + mutex_init(&lli->lli_setattr_mutex); lli->lli_symlink_name = NULL; ll_trunc_sem_init(&lli->lli_trunc_sem); range_lock_tree_init(&lli->lli_write_tree); diff --git a/fs/lustre/llite/vvp_io.c b/fs/lustre/llite/vvp_io.c index 8df5d39..8edd3c1 100644 --- a/fs/lustre/llite/vvp_io.c +++ b/fs/lustre/llite/vvp_io.c @@ -702,13 +702,13 @@ static int vvp_io_setattr_start(const struct lu_env *env, if (cl_io_is_trunc(io)) { trunc_sem_down_write(&lli->lli_trunc_sem); - inode_lock(inode); + mutex_lock(&lli->lli_setattr_mutex); inode_dio_wait(inode); } else if (cl_io_is_fallocate(io)) { inode_lock(inode); inode_dio_wait(inode); } else { - inode_lock(inode); + mutex_lock(&lli->lli_setattr_mutex); } if (io->u.ci_setattr.sa_avalid & TIMES_SET_FLAGS) @@ -729,12 +729,12 @@ static void vvp_io_setattr_end(const struct lu_env *env, * because osc has already notified to destroy osc_extents. */ vvp_do_vmtruncate(inode, io->u.ci_setattr.sa_attr.lvb_size); - inode_unlock(inode); + mutex_unlock(&lli->lli_setattr_mutex); trunc_sem_up_write(&lli->lli_trunc_sem); } else if (cl_io_is_fallocate(io)) { inode_unlock(inode); } else { - inode_unlock(inode); + mutex_unlock(&lli->lli_setattr_mutex); } } -- 1.8.3.1 From jsimmons at infradead.org Thu Jul 2 00:04:41 2020 From: jsimmons at infradead.org (James Simmons) Date: Wed, 1 Jul 2020 20:04:41 -0400 Subject: [lustre-devel] [PATCH 01/18] lnet: restore an maximal fragments count In-Reply-To: <1593648298-10571-1-git-send-email-jsimmons@infradead.org> References: <1593648298-10571-1-git-send-email-jsimmons@infradead.org> Message-ID: <1593648298-10571-2-git-send-email-jsimmons@infradead.org> From: Alexey Lyashkov Lowering a number of fragments blocks a connection from older clients who wants to use 256 fragments to transfer. Let's restore this number to the original value. Fixes: ab7e089b8eda ("lustre: lnet: make LNET_MAX_IOV dependent on page size") Cray-bug-id: LUS-8139 WC-bug-id: https://jira.whamcloud.com/browse/LU-10157 Lustre-commit: 4072d863c240f ("LU-10157 lnet: restore an maximal fragments count") Signed-off-by: Alexey Lyashkov Reviewed-on: https://review.whamcloud.com/37385 Reviewed-by: Serguei Smirnov Reviewed-by: Shaun Tancheff Reviewed-by: James Simmons Reviewed-by: Oleg Drokin Signed-off-by: James Simmons --- include/linux/lnet/lib-types.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/include/linux/lnet/lib-types.h b/include/linux/lnet/lib-types.h index 6aa691e..1c016fd 100644 --- a/include/linux/lnet/lib-types.h +++ b/include/linux/lnet/lib-types.h @@ -51,7 +51,7 @@ /* Max payload size */ #define LNET_MAX_PAYLOAD LNET_MTU -#define LNET_MAX_IOV (LNET_MAX_PAYLOAD >> PAGE_SHIFT) +#define LNET_MAX_IOV 256 /* * This is the maximum health value. -- 1.8.3.1 From jsimmons at infradead.org Thu Jul 2 00:04:44 2020 From: jsimmons at infradead.org (James Simmons) Date: Wed, 1 Jul 2020 20:04:44 -0400 Subject: [lustre-devel] [PATCH 04/18] lustre: sec: decryption for read path In-Reply-To: <1593648298-10571-1-git-send-email-jsimmons@infradead.org> References: <1593648298-10571-1-git-send-email-jsimmons@infradead.org> Message-ID: <1593648298-10571-5-git-send-email-jsimmons@infradead.org> From: Sebastien Buisson With the support for encryption, all files need to be opened with fscrypt_file_open(). fscrypt will retrieve encryption context if file is encrypted, or immediately return if not. Decryption itself is carried out in osc_brw_fini_request(), right after the reply has been received from the server. WC-bug-id: https://jira.whamcloud.com/browse/LU-12275 Lustre-commit: eecf86131d099 ("LU-12275 sec: decryption for read path") Signed-off-by: Sebastien Buisson Reviewed-on: https://review.whamcloud.com/36145 Reviewed-by: Patrick Farrell Reviewed-by: John L. Hammond Reviewed-by: Andreas Dilger Reviewed-by: Oleg Drokin Signed-off-by: James Simmons --- fs/lustre/llite/crypto.c | 10 ++++++++-- fs/lustre/llite/file.c | 6 ++++++ fs/lustre/osc/osc_request.c | 31 +++++++++++++++++++++++++++++++ 3 files changed, 45 insertions(+), 2 deletions(-) diff --git a/fs/lustre/llite/crypto.c b/fs/lustre/llite/crypto.c index f411343..157017f 100644 --- a/fs/lustre/llite/crypto.c +++ b/fs/lustre/llite/crypto.c @@ -32,6 +32,7 @@ static int ll_get_context(struct inode *inode, void *ctx, size_t len) { struct dentry *dentry; + int rc; if (hlist_empty(&inode->i_dentry)) return -ENODATA; @@ -39,8 +40,13 @@ static int ll_get_context(struct inode *inode, void *ctx, size_t len) hlist_for_each_entry(dentry, &inode->i_dentry, d_u.d_alias) break; - return __vfs_getxattr(dentry, inode, LL_XATTR_NAME_ENCRYPTION_CONTEXT, - ctx, len); + rc = __vfs_getxattr(dentry, inode, LL_XATTR_NAME_ENCRYPTION_CONTEXT, + ctx, len); + + /* used as encryption unit size */ + if (S_ISREG(inode->i_mode)) + inode->i_blkbits = LUSTRE_ENCRYPTION_BLOCKBITS; + return rc; } static int ll_set_context(struct inode *inode, const void *ctx, size_t len, diff --git a/fs/lustre/llite/file.c b/fs/lustre/llite/file.c index 8264b86..3b04952 100644 --- a/fs/lustre/llite/file.c +++ b/fs/lustre/llite/file.c @@ -714,6 +714,12 @@ int ll_file_open(struct inode *inode, struct file *file) it = file->private_data; /* XXX: compat macro */ file->private_data = NULL; /* prevent ll_local_open assertion */ + if (S_ISREG(inode->i_mode)) { + rc = llcrypt_file_open(inode, file); + if (rc) + goto out_nofiledata; + } + fd = ll_file_data_get(); if (!fd) { rc = -ENOMEM; diff --git a/fs/lustre/osc/osc_request.c b/fs/lustre/osc/osc_request.c index db97d37..65d17a8 100644 --- a/fs/lustre/osc/osc_request.c +++ b/fs/lustre/osc/osc_request.c @@ -1865,6 +1865,7 @@ static int osc_brw_fini_request(struct ptlrpc_request *req, int rc) const char *obd_name = cli->cl_import->imp_obd->obd_name; struct ost_body *body; u32 client_cksum = 0; + struct inode *inode; if (rc < 0 && rc != -EDQUOT) { DEBUG_REQ(D_INFO, req, "Failed request: rc = %d", rc); @@ -2055,6 +2056,36 @@ static int osc_brw_fini_request(struct ptlrpc_request *req, int rc) } else { rc = 0; } + + inode = page2inode(aa->aa_ppga[0]->pg); + if (inode && IS_ENCRYPTED(inode)) { + int idx; + + if (!llcrypt_has_encryption_key(inode)) { + CDEBUG(D_SEC, "no enc key for ino %lu\n", inode->i_ino); + goto out; + } + for (idx = 0; idx < aa->aa_page_count; idx++) { + struct brw_page *pg = aa->aa_ppga[idx]; + u64 *p, *q; + + /* do not decrypt if page is all 0s */ + p = q = page_address(pg->pg); + while (p - q < PAGE_SIZE / sizeof(*p)) { + if (*p != 0) + break; + p++; + } + if (p - q == PAGE_SIZE / sizeof(*p)) + continue; + + rc = llcrypt_decrypt_pagecache_blocks(pg->pg, + PAGE_SIZE, 0); + if (rc) + goto out; + } + } + out: if (rc >= 0) lustre_get_wire_obdo(&req->rq_import->imp_connect_data, -- 1.8.3.1 From jsimmons at infradead.org Thu Jul 2 00:04:45 2020 From: jsimmons at infradead.org (James Simmons) Date: Wed, 1 Jul 2020 20:04:45 -0400 Subject: [lustre-devel] [PATCH 05/18] lustre: sec: deal with encrypted object size In-Reply-To: <1593648298-10571-1-git-send-email-jsimmons@infradead.org> References: <1593648298-10571-1-git-send-email-jsimmons@infradead.org> Message-ID: <1593648298-10571-6-git-send-email-jsimmons@infradead.org> From: Sebastien Buisson Problem with size of encrypted file comes from the fact that an encrypted page will always contain PAGE_SIZE bytes of data, even if clear text page is only a few bytes. And server infers object size from content of encrypted page. The way to address this is the following. Upon writing, when the client encrypts the page representing the end of the file, it puts into o_size info of the request's body, the size of the clear text version of the file. On server side, this information is used to adjust isize of the object, but still storing the complete pages on disk. WC-bug-id: https://jira.whamcloud.com/browse/LU-12275 Lustre-commit: 83d660436a164 ("LU-12275 sec: deal with encrypted object size") Signed-off-by: Sebastien Buisson Reviewed-on: https://review.whamcloud.com/36146 Reviewed-by: John L. Hammond Reviewed-by: Andreas Dilger Reviewed-by: Oleg Drokin Signed-off-by: James Simmons --- fs/lustre/osc/osc_request.c | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/fs/lustre/osc/osc_request.c b/fs/lustre/osc/osc_request.c index 65d17a8..b27a259 100644 --- a/fs/lustre/osc/osc_request.c +++ b/fs/lustre/osc/osc_request.c @@ -1446,10 +1446,18 @@ static int osc_brw_prep_request(int cmd, struct client_obd *cli, ptlrpc_request_free(req); return rc; } + pg->pg = data_page; + /* there should be no gap in the middle of page array */ + if (i == page_count - 1) { + struct osc_async_page *oap = brw_page2oap(pg); + + oa->o_size = oap->oap_count + + oap->oap_obj_off + + oap->oap_page_off; + } /* len is forced to PAGE_SIZE, and poff to 0 * so store the old, clear text info */ - pg->pg = data_page; pg->bp_count_diff = PAGE_SIZE - pg->count; pg->count = PAGE_SIZE; pg->bp_off_diff = pg->off & ~PAGE_MASK; -- 1.8.3.1 From jsimmons at infradead.org Thu Jul 2 00:04:47 2020 From: jsimmons at infradead.org (James Simmons) Date: Wed, 1 Jul 2020 20:04:47 -0400 Subject: [lustre-devel] [PATCH 07/18] lustre: ptlrpc: limit rate of lock replays In-Reply-To: <1593648298-10571-1-git-send-email-jsimmons@infradead.org> References: <1593648298-10571-1-git-send-email-jsimmons@infradead.org> Message-ID: <1593648298-10571-8-git-send-email-jsimmons@infradead.org> From: Mikhail Pershin Clients send all lock replays at once and that may overwhelm server with huge amount of replays in recovery queue causing OOM effects. Patch adds rate control for lock replays on client WC-bug-id: https://jira.whamcloud.com/browse/LU-13600 Lustre-commit: 3b613a442b869 ("LU-13600 ptlrpc: limit rate of lock replays") Signed-off-by: Mikhail Pershin Reviewed-on: https://review.whamcloud.com/38920 Reviewed-by: Andreas Dilger Reviewed-by: Hongchao Zhang Reviewed-by: Oleg Drokin Signed-off-by: James Simmons --- fs/lustre/include/lustre_import.h | 2 ++ fs/lustre/ldlm/ldlm_request.c | 69 +++++++++++++++++++++++++++++++++++---- fs/lustre/obdclass/genops.c | 1 + fs/lustre/ptlrpc/import.c | 8 ++--- 4 files changed, 70 insertions(+), 10 deletions(-) diff --git a/fs/lustre/include/lustre_import.h b/fs/lustre/include/lustre_import.h index 4e9a228..72a303e 100644 --- a/fs/lustre/include/lustre_import.h +++ b/fs/lustre/include/lustre_import.h @@ -226,6 +226,8 @@ struct obd_import { atomic_t imp_unregistering; /** Number of replay requests inflight */ atomic_t imp_replay_inflight; + /** In-flight replays rate control */ + wait_queue_head_t imp_replay_waitq; /** Number of currently happening import invalidations */ atomic_t imp_inval_count; /** Numbner of request timeouts */ diff --git a/fs/lustre/ldlm/ldlm_request.c b/fs/lustre/ldlm/ldlm_request.c index 12ee241..e1ba596 100644 --- a/fs/lustre/ldlm/ldlm_request.c +++ b/fs/lustre/ldlm/ldlm_request.c @@ -2098,6 +2098,8 @@ static int replay_lock_interpret(const struct lu_env *env, struct obd_export *exp; atomic_dec(&req->rq_import->imp_replay_inflight); + wake_up(&req->rq_import->imp_replay_waitq); + if (rc != ELDLM_OK) goto out; @@ -2205,7 +2207,7 @@ static int replay_one_lock(struct obd_import *imp, struct ldlm_lock *lock) LDLM_DEBUG(lock, "replaying lock:"); - atomic_inc(&req->rq_import->imp_replay_inflight); + atomic_inc(&imp->imp_replay_inflight); aa = ptlrpc_req_async_args(aa, req); aa->lock_handle = body->lock_handle[0]; req->rq_interpret_reply = replay_lock_interpret; @@ -2245,22 +2247,32 @@ static void ldlm_cancel_unused_locks_for_replay(struct ldlm_namespace *ns) canceled, ldlm_ns_name(ns)); } -int ldlm_replay_locks(struct obd_import *imp) +static int lock_can_replay(struct obd_import *imp) +{ + struct client_obd *cli = &imp->imp_obd->u.cli; + + CDEBUG(D_HA, "check lock replay limit, inflights = %u(%u)\n", + atomic_read(&imp->imp_replay_inflight) - 1, + cli->cl_max_rpcs_in_flight); + + /* +1 due to ldlm_lock_replay() increment */ + return atomic_read(&imp->imp_replay_inflight) < + 1 + min_t(u32, cli->cl_max_rpcs_in_flight, 8); +} + +int __ldlm_replay_locks(struct obd_import *imp, bool rate_limit) { struct ldlm_namespace *ns = imp->imp_obd->obd_namespace; LIST_HEAD(list); struct ldlm_lock *lock; int rc = 0; - LASSERT(atomic_read(&imp->imp_replay_inflight) == 0); + LASSERT(atomic_read(&imp->imp_replay_inflight) == 1); /* don't replay locks if import failed recovery */ if (imp->imp_vbr_failed) return 0; - /* ensure this doesn't fall to 0 before all have been queued */ - atomic_inc(&imp->imp_replay_inflight); - if (ldlm_cancel_unused_locks_before_replay) ldlm_cancel_unused_locks_for_replay(ns); @@ -2276,9 +2288,54 @@ int ldlm_replay_locks(struct obd_import *imp) } rc = replay_one_lock(imp, lock); LDLM_LOCK_RELEASE(lock); + + if (rate_limit) + wait_event_idle_exclusive(imp->imp_replay_waitq, + lock_can_replay(imp)); } + return rc; +} + +/** + * Lock replay uses rate control and can sleep waiting so + * must be in separate thread from ptlrpcd itself + */ +static int ldlm_lock_replay_thread(void *data) +{ + struct obd_import *imp = data; + + CDEBUG(D_HA, "lock replay thread %s to %s@%s\n", + imp->imp_obd->obd_name, obd2cli_tgt(imp->imp_obd), + imp->imp_connection->c_remote_uuid.uuid); + + __ldlm_replay_locks(imp, true); atomic_dec(&imp->imp_replay_inflight); + ptlrpc_import_recovery_state_machine(imp); + class_import_put(imp); + + return 0; +} + +int ldlm_replay_locks(struct obd_import *imp) +{ + struct task_struct *task; + int rc = 0; + + class_import_get(imp); + /* ensure this doesn't fall to 0 before all have been queued */ + atomic_inc(&imp->imp_replay_inflight); + + task = kthread_run(ldlm_lock_replay_thread, imp, "ldlm_lock_replay"); + if (IS_ERR(task)) { + rc = PTR_ERR(task); + CDEBUG(D_HA, "can't start lock replay thread: rc = %d\n", rc); + + /* run lock replay without rate control */ + rc = __ldlm_replay_locks(imp, false); + atomic_dec(&imp->imp_replay_inflight); + class_import_put(imp); + } return rc; } diff --git a/fs/lustre/obdclass/genops.c b/fs/lustre/obdclass/genops.c index 1647fe9..759d97e 100644 --- a/fs/lustre/obdclass/genops.c +++ b/fs/lustre/obdclass/genops.c @@ -1001,6 +1001,7 @@ struct obd_import *class_new_import(struct obd_device *obd) atomic_set(&imp->imp_reqs, 0); atomic_set(&imp->imp_inflight, 0); atomic_set(&imp->imp_replay_inflight, 0); + init_waitqueue_head(&imp->imp_replay_waitq); atomic_set(&imp->imp_inval_count, 0); INIT_LIST_HEAD(&imp->imp_conn_list); init_imp_at(&imp->imp_at); diff --git a/fs/lustre/ptlrpc/import.c b/fs/lustre/ptlrpc/import.c index 6b0b115..7ec3638 100644 --- a/fs/lustre/ptlrpc/import.c +++ b/fs/lustre/ptlrpc/import.c @@ -1486,6 +1486,8 @@ int ptlrpc_import_recovery_state_machine(struct obd_import *imp) int target_len; if (imp->imp_state == LUSTRE_IMP_EVICTED) { + struct task_struct *task; + deuuidify(obd2cli_tgt(imp->imp_obd), NULL, &target_start, &target_len); /* Don't care about MGC eviction */ @@ -1505,8 +1507,6 @@ int ptlrpc_import_recovery_state_machine(struct obd_import *imp) imp->imp_vbr_failed = 0; spin_unlock(&imp->imp_lock); - { - struct task_struct *task; /* bug 17802: XXX client_disconnect_export vs connect request * race. if client is evicted at this time, we start * invalidate thread without reference to import and import can @@ -1517,13 +1517,13 @@ int ptlrpc_import_recovery_state_machine(struct obd_import *imp) "ll_imp_inval"); if (IS_ERR(task)) { class_import_put(imp); - CERROR("error starting invalidate thread: %d\n", rc); rc = PTR_ERR(task); + CERROR("%s: can't start invalidate thread: rc = %d\n", + imp->imp_obd->obd_name, rc); } else { rc = 0; } return rc; - } } if (imp->imp_state == LUSTRE_IMP_REPLAY) { -- 1.8.3.1 From jsimmons at infradead.org Thu Jul 2 00:04:55 2020 From: jsimmons at infradead.org (James Simmons) Date: Wed, 1 Jul 2020 20:04:55 -0400 Subject: [lustre-devel] [PATCH 15/18] lustre: llite: don't hold inode_lock for security notify In-Reply-To: <1593648298-10571-1-git-send-email-jsimmons@infradead.org> References: <1593648298-10571-1-git-send-email-jsimmons@infradead.org> Message-ID: <1593648298-10571-16-git-send-email-jsimmons@infradead.org> From: Alexander Boyko With selinux enabled client has a dead lock which leads to client eviction from MDS. 1 thread 2 thread do file open do stat inode_lock(parend dir) got LDLM_PR(parent dir) enqueue LDLM_CW(parent dir) waits on inode_lock to notify security waits timeout on enqueue and client eviction because client didn't cancel a LDLM_PR lock security_inode_notifysecctx()->selinux_inode_notifysecctx()-> selinux_inode_setsecurity() The call of selinux_inode_setsecurity doesn't need to hold inode_lock. Fixes: f4d3cf7642 ("lustre: llite: set sec ctx on client's inode at create time") Cray-bug-id: LUS-8924 WC-bug-id: https://jira.whamcloud.com/browse/LU-13617 Lustre-commit: f87359b51f61a ("LU-13617 llite: don't hold inode_lock for security notify") Signed-off-by: Alexander Boyko Reviewed-on: https://review.whamcloud.com/38792 Reviewed-by: Sebastien Buisson Reviewed-by: Andrew Perepechko Reviewed-by: Oleg Drokin Signed-off-by: James Simmons --- fs/lustre/llite/dir.c | 6 ++++-- fs/lustre/llite/namei.c | 18 ++++++++++++------ 2 files changed, 16 insertions(+), 8 deletions(-) diff --git a/fs/lustre/llite/dir.c b/fs/lustre/llite/dir.c index 463c5d7..e3305f7 100644 --- a/fs/lustre/llite/dir.c +++ b/fs/lustre/llite/dir.c @@ -489,11 +489,13 @@ static int ll_dir_setdirstripe(struct dentry *dparent, struct lmv_user_md *lump, dentry.d_inode = inode; if (sbi->ll_flags & LL_SBI_FILE_SECCTX) { - inode_lock(inode); + /* no need to protect selinux_inode_setsecurity() by + * inode_lock. Taking it would lead to a client deadlock + * LU-13617 + */ err = security_inode_notifysecctx(inode, op_data->op_file_secctx, op_data->op_file_secctx_size); - inode_unlock(inode); } else { err = ll_inode_init_security(&dentry, inode, parent); } diff --git a/fs/lustre/llite/namei.c b/fs/lustre/llite/namei.c index 2353a8f..251d6be 100644 --- a/fs/lustre/llite/namei.c +++ b/fs/lustre/llite/namei.c @@ -659,10 +659,12 @@ static int ll_lookup_it_finish(struct ptlrpc_request *request, } if (secctx && secctxlen != 0) { - inode_lock(inode); + /* no need to protect selinux_inode_setsecurity() by + * inode_lock. Taking it would lead to a client deadlock + * LU-13617 + */ rc = security_inode_notifysecctx(inode, secctx, secctxlen); - inode_unlock(inode); if (rc) CWARN("cannot set security context for " DFID ": rc = %d\n", PFID(ll_inode2fid(inode)), rc); @@ -1198,13 +1200,15 @@ static int ll_create_it(struct inode *dir, struct dentry *dentry, return PTR_ERR(inode); if ((ll_i2sbi(inode)->ll_flags & LL_SBI_FILE_SECCTX) && secctx) { - inode_lock(inode); /* must be done before d_instantiate, because it calls * security_d_instantiate, which means a getxattr if security * context is not set yet */ + /* no need to protect selinux_inode_setsecurity() by + * inode_lock. Taking it would lead to a client deadlock + * LU-13617 + */ rc = security_inode_notifysecctx(inode, secctx, secctxlen); - inode_unlock(inode); if (rc) return rc; } @@ -1370,15 +1374,17 @@ static int ll_new_node(struct inode *dir, struct dentry *dentry, goto err_exit; if (sbi->ll_flags & LL_SBI_FILE_SECCTX) { - inode_lock(inode); /* must be done before d_instantiate, because it calls * security_d_instantiate, which means a getxattr if security * context is not set yet */ + /* no need to protect selinux_inode_setsecurity() by + * inode_lock. Taking it would lead to a client deadlock + * LU-13617 + */ err = security_inode_notifysecctx(inode, op_data->op_file_secctx, op_data->op_file_secctx_size); - inode_unlock(inode); if (err) goto err_exit; } -- 1.8.3.1 From jsimmons at infradead.org Thu Jul 2 00:04:48 2020 From: jsimmons at infradead.org (James Simmons) Date: Wed, 1 Jul 2020 20:04:48 -0400 Subject: [lustre-devel] [PATCH 08/18] lustre: mdc: chlg device could be used after free In-Reply-To: <1593648298-10571-1-git-send-email-jsimmons@infradead.org> References: <1593648298-10571-1-git-send-email-jsimmons@infradead.org> Message-ID: <1593648298-10571-9-git-send-email-jsimmons@infradead.org> From: Hongchao Zhang There are some issue of the usage of dynamic devices used by the changelog in MDC, which could cause the device to be used after it is freed. WC-bug-id: https://jira.whamcloud.com/browse/LU-13508 Lustre-commit: 1e992e94eaf8a ("LU-13508 mdc: chlg device could be used after free") Signed-off-by: Hongchao Zhang Reviewed-on: https://review.whamcloud.com/38658 Reviewed-by: John L. Hammond Reviewed-by: James Simmons Reviewed-by: Oleg Drokin Signed-off-by: James Simmons --- fs/lustre/mdc/mdc_changelog.c | 46 ++++++++++++++++++++----------------------- fs/lustre/mdc/mdc_internal.h | 1 + fs/lustre/mdc/mdc_request.c | 8 +++++--- 3 files changed, 27 insertions(+), 28 deletions(-) diff --git a/fs/lustre/mdc/mdc_changelog.c b/fs/lustre/mdc/mdc_changelog.c index 3aace7e..8531edb 100644 --- a/fs/lustre/mdc/mdc_changelog.c +++ b/fs/lustre/mdc/mdc_changelog.c @@ -61,7 +61,7 @@ struct chlg_registered_dev { char ced_name[32]; /* changelog char device */ struct cdev ced_cdev; - struct device *ced_device; + struct device ced_device; /* OBDs referencing this device (multiple mount point) */ struct list_head ced_obds; /* Reference counter for proper deregistration */ @@ -112,7 +112,7 @@ enum { CDEV_CHLG_MAX_PREFETCH = 1024, }; -static DEFINE_IDR(chlg_minor_idr); +DEFINE_IDR(mdc_changelog_minor_idr); static DEFINE_SPINLOCK(chlg_minor_lock); static int chlg_minor_alloc(int *pminor) @@ -122,7 +122,7 @@ static int chlg_minor_alloc(int *pminor) idr_preload(GFP_KERNEL); spin_lock(&chlg_minor_lock); - minor = idr_alloc(&chlg_minor_idr, minor_allocated, 0, + minor = idr_alloc(&mdc_changelog_minor_idr, minor_allocated, 0, MDC_CHANGELOG_DEV_COUNT, GFP_NOWAIT); spin_unlock(&chlg_minor_lock); idr_preload_end(); @@ -137,7 +137,7 @@ static int chlg_minor_alloc(int *pminor) static void chlg_minor_free(int minor) { spin_lock(&chlg_minor_lock); - idr_remove(&chlg_minor_idr, minor); + idr_remove(&mdc_changelog_minor_idr, minor); spin_unlock(&chlg_minor_lock); } @@ -160,8 +160,8 @@ static void chlg_dev_clear(struct kref *kref) ced_refs); list_del(&entry->ced_link); - cdev_del(&entry->ced_cdev); - device_destroy(mdc_changelog_class, entry->ced_cdev.dev); + cdev_device_del(&entry->ced_cdev, &entry->ced_device); + put_device(&entry->ced_device); } static inline struct obd_device *chlg_obd_get(struct chlg_registered_dev *dev) @@ -790,8 +790,6 @@ int mdc_changelog_cdev_init(struct obd_device *obd) { struct chlg_registered_dev *exist; struct chlg_registered_dev *entry; - struct device *device; - dev_t dev; int minor, rc; entry = kzalloc(sizeof(*entry), GFP_KERNEL); @@ -816,35 +814,33 @@ int mdc_changelog_cdev_init(struct obd_device *obd) list_add_tail(&obd->u.cli.cl_chg_dev_linkage, &entry->ced_obds); list_add_tail(&entry->ced_link, &chlg_registered_devices); - /* Register new character device */ - cdev_init(&entry->ced_cdev, &chlg_fops); - entry->ced_cdev.owner = THIS_MODULE; - rc = chlg_minor_alloc(&minor); if (rc) goto out_unlock; - dev = MKDEV(MAJOR(mdc_changelog_dev), minor); - rc = cdev_add(&entry->ced_cdev, dev, 1); + device_initialize(&entry->ced_device); + entry->ced_device.devt = MKDEV(MAJOR(mdc_changelog_dev), minor); + entry->ced_device.class = mdc_changelog_class; + entry->ced_device.release = chlg_device_release; + dev_set_drvdata(&entry->ced_device, entry); + rc = dev_set_name(&entry->ced_device, "%s-%s", MDC_CHANGELOG_DEV_NAME, + entry->ced_name); if (rc) goto out_minor; - device = device_create(mdc_changelog_class, NULL, dev, entry, "%s-%s", - MDC_CHANGELOG_DEV_NAME, entry->ced_name); - if (IS_ERR(device)) { - rc = PTR_ERR(device); - goto out_cdev; - } - - device->release = chlg_device_release; - entry->ced_device = device; + /* Register new character device */ + cdev_init(&entry->ced_cdev, &chlg_fops); + entry->ced_cdev.owner = THIS_MODULE; + rc = cdev_device_add(&entry->ced_cdev, &entry->ced_device); + if (rc) + goto out_device_name; entry = NULL; /* prevent it from being freed below */ rc = 0; goto out_unlock; -out_cdev: - cdev_del(&entry->ced_cdev); +out_device_name: + kfree_const(entry->ced_device.kobj.name); out_minor: chlg_minor_free(minor); diff --git a/fs/lustre/mdc/mdc_internal.h b/fs/lustre/mdc/mdc_internal.h index 9656231..b7ccc58 100644 --- a/fs/lustre/mdc/mdc_internal.h +++ b/fs/lustre/mdc/mdc_internal.h @@ -142,6 +142,7 @@ enum ldlm_mode mdc_lock_match(struct obd_export *exp, u64 flags, #define MDC_CHANGELOG_DEV_NAME "changelog" extern struct class *mdc_changelog_class; extern dev_t mdc_changelog_dev; +extern struct idr mdc_changelog_minor_idr; int mdc_changelog_cdev_init(struct obd_device *obd); diff --git a/fs/lustre/mdc/mdc_request.c b/fs/lustre/mdc/mdc_request.c index 369114b..d6d9f43 100644 --- a/fs/lustre/mdc/mdc_request.c +++ b/fs/lustre/mdc/mdc_request.c @@ -3013,10 +3013,12 @@ static int __init mdc_init(void) rc = class_register_type(&mdc_obd_ops, &mdc_md_ops, LUSTRE_MDC_NAME, &mdc_device_type); if (rc) - goto out_dev; + goto out_class; return 0; +out_class: + class_destroy(mdc_changelog_class); out_dev: unregister_chrdev_region(mdc_changelog_dev, MDC_CHANGELOG_DEV_COUNT); return rc; @@ -3024,9 +3026,9 @@ static int __init mdc_init(void) static void __exit mdc_exit(void) { - class_destroy(mdc_changelog_class); - unregister_chrdev_region(mdc_changelog_dev, MDC_CHANGELOG_DEV_COUNT); class_unregister_type(LUSTRE_MDC_NAME); + class_destroy(mdc_changelog_class); + idr_destroy(&mdc_changelog_minor_idr); } MODULE_AUTHOR("OpenSFS, Inc. "); -- 1.8.3.1 From jsimmons at infradead.org Thu Jul 2 00:04:50 2020 From: jsimmons at infradead.org (James Simmons) Date: Wed, 1 Jul 2020 20:04:50 -0400 Subject: [lustre-devel] [PATCH 10/18] lustre: lov: use lov_pattern_support() to verify lmm In-Reply-To: <1593648298-10571-1-git-send-email-jsimmons@infradead.org> References: <1593648298-10571-1-git-send-email-jsimmons@infradead.org> Message-ID: <1593648298-10571-11-git-send-email-jsimmons@infradead.org> We can use lov_pattern_support(), which is used by the server and userland code, to ensure lmm is valid instead of open coding. WC-bug-id: https://jira.whamcloud.com/browse/LU-12511 Lustre-commit: 0f607f22696ff ("LU-12511 lov: use lov_pattern_support() to verify lmm") Signed-off-by: James Simmons Reviewed-on: https://review.whamcloud.com/38791 Reviewed-by: Andreas Dilger Reviewed-by: Mike Pershin --- fs/lustre/lov/lov_ea.c | 6 ++---- include/uapi/linux/lustre/lustre_user.h | 8 ++++++++ 2 files changed, 10 insertions(+), 4 deletions(-) diff --git a/fs/lustre/lov/lov_ea.c b/fs/lustre/lov/lov_ea.c index 39ff50b..e198536 100644 --- a/fs/lustre/lov/lov_ea.c +++ b/fs/lustre/lov/lov_ea.c @@ -84,6 +84,7 @@ static loff_t lov_tgt_maxbytes(struct lov_tgt_desc *tgt) static int lsm_lmm_verify_v1v3(struct lov_mds_md *lmm, size_t lmm_size, u16 stripe_count) { + u32 pattern = le32_to_cpu(lmm->lmm_pattern); int rc = 0; if (stripe_count > LOV_V1_INSANE_STRIPE_COUNT) { @@ -101,10 +102,7 @@ static int lsm_lmm_verify_v1v3(struct lov_mds_md *lmm, size_t lmm_size, goto out; } - if (lov_pattern(le32_to_cpu(lmm->lmm_pattern)) != LOV_PATTERN_MDT && - lov_pattern(le32_to_cpu(lmm->lmm_pattern)) != LOV_PATTERN_RAID0 && - lov_pattern(le32_to_cpu(lmm->lmm_pattern)) != - (LOV_PATTERN_RAID0 | LOV_PATTERN_OVERSTRIPING)) { + if (!lov_pattern_supported(lov_pattern(pattern))) { rc = -EINVAL; CERROR("lov: unrecognized striping pattern: rc = %d\n", rc); lov_dump_lmm_common(D_WARNING, lmm); diff --git a/include/uapi/linux/lustre/lustre_user.h b/include/uapi/linux/lustre/lustre_user.h index 121c064..6a2d5f9 100644 --- a/include/uapi/linux/lustre/lustre_user.h +++ b/include/uapi/linux/lustre/lustre_user.h @@ -443,6 +443,14 @@ static inline bool lov_pattern_supported_normal_comp(__u32 pattern) #define LOV_OFFSET_DEFAULT ((__u16)-1) #define LMV_OFFSET_DEFAULT ((__u32)-1) +static inline bool lov_pattern_supported(__u32 pattern) +{ + return (pattern & ~LOV_PATTERN_F_RELEASED) == LOV_PATTERN_RAID0 || + (pattern & ~LOV_PATTERN_F_RELEASED) == + (LOV_PATTERN_RAID0 | LOV_PATTERN_OVERSTRIPING) || + (pattern & ~LOV_PATTERN_F_RELEASED) == LOV_PATTERN_MDT; +} + #define LOV_MIN_STRIPE_BITS 16 /* maximum PAGE_SIZE (ia64), power of 2 */ #define LOV_MIN_STRIPE_SIZE (1 << LOV_MIN_STRIPE_BITS) #define LOV_MAX_STRIPE_COUNT_OLD 160 -- 1.8.3.1 From jsimmons at infradead.org Thu Jul 2 00:04:58 2020 From: jsimmons at infradead.org (James Simmons) Date: Wed, 1 Jul 2020 20:04:58 -0400 Subject: [lustre-devel] [PATCH 18/18] lnet: handle undefined parameters In-Reply-To: <1593648298-10571-1-git-send-email-jsimmons@infradead.org> References: <1593648298-10571-1-git-send-email-jsimmons@infradead.org> Message-ID: <1593648298-10571-19-git-send-email-jsimmons@infradead.org> From: Amir Shehata If peer_tx_credits or peer_credits are 0, they should be defaulted to the system defaults 8 and 256 respectively WC-bug-id: https://jira.whamcloud.com/browse/LU-13662 Lustre-commit: d934eb3c4f638 ("LU-13662 lnet: handle undefined parameters") Signed-off-by: Amir Shehata Reviewed-on: https://review.whamcloud.com/38894 Reviewed-by: Serguei Smirnov Reviewed-by: Andreas Dilger Reviewed-by: Oleg Drokin Signed-off-by: James Simmons --- include/linux/lnet/lib-lnet.h | 4 +++- net/lnet/klnds/o2iblnd/o2iblnd_modparams.c | 4 ++-- net/lnet/klnds/socklnd/socklnd_modparams.c | 4 ++-- net/lnet/lnet/api-ni.c | 26 +++++++++++++++++++++++--- 4 files changed, 30 insertions(+), 8 deletions(-) diff --git a/include/linux/lnet/lib-lnet.h b/include/linux/lnet/lib-lnet.h index def0923..75c0da7 100644 --- a/include/linux/lnet/lib-lnet.h +++ b/include/linux/lnet/lib-lnet.h @@ -81,8 +81,10 @@ #define LNET_ACCEPTOR_MIN_RESERVED_PORT 512 #define LNET_ACCEPTOR_MAX_RESERVED_PORT 1023 -/* default timeout */ +/* default timeout and credits */ #define DEFAULT_PEER_TIMEOUT 180 +#define DEFAULT_PEER_CREDITS 8 +#define DEFAULT_CREDITS 256 int choose_ipv4_src(u32 *ret, int interface, u32 dst_ipaddr, struct net *ns); diff --git a/net/lnet/klnds/o2iblnd/o2iblnd_modparams.c b/net/lnet/klnds/o2iblnd/o2iblnd_modparams.c index 7407ced..f341376 100644 --- a/net/lnet/klnds/o2iblnd/o2iblnd_modparams.c +++ b/net/lnet/klnds/o2iblnd/o2iblnd_modparams.c @@ -67,11 +67,11 @@ MODULE_PARM_DESC(ntx, "# of message descriptors allocated for each pool"); /* NB: this value is shared by all CPTs */ -static int credits = 256; +static int credits = DEFAULT_CREDITS; module_param(credits, int, 0444); MODULE_PARM_DESC(credits, "# concurrent sends"); -static int peer_credits = 8; +static int peer_credits = DEFAULT_PEER_CREDITS; module_param(peer_credits, int, 0444); MODULE_PARM_DESC(peer_credits, "# concurrent sends to 1 peer"); diff --git a/net/lnet/klnds/socklnd/socklnd_modparams.c b/net/lnet/klnds/socklnd/socklnd_modparams.c index b511e54..017627f 100644 --- a/net/lnet/klnds/socklnd/socklnd_modparams.c +++ b/net/lnet/klnds/socklnd/socklnd_modparams.c @@ -28,11 +28,11 @@ module_param(sock_timeout, int, 0644); MODULE_PARM_DESC(sock_timeout, "dead socket timeout (seconds)"); -static int credits = 256; +static int credits = DEFAULT_CREDITS; module_param(credits, int, 0444); MODULE_PARM_DESC(credits, "# concurrent sends"); -static int peer_credits = 8; +static int peer_credits = DEFAULT_PEER_CREDITS; module_param(peer_credits, int, 0444); MODULE_PARM_DESC(peer_credits, "# concurrent sends to 1 peer"); diff --git a/net/lnet/lnet/api-ni.c b/net/lnet/lnet/api-ni.c index d6694cb..3e69435 100644 --- a/net/lnet/lnet/api-ni.c +++ b/net/lnet/lnet/api-ni.c @@ -3093,6 +3093,19 @@ static int lnet_add_net_common(struct lnet_net *net, return rc; } +static void +lnet_set_tune_defaults(struct lnet_ioctl_config_lnd_tunables *tun) +{ + if (tun) { + if (!tun->lt_cmn.lct_peer_timeout) + tun->lt_cmn.lct_peer_timeout = DEFAULT_PEER_TIMEOUT; + if (!tun->lt_cmn.lct_peer_tx_credits) + tun->lt_cmn.lct_peer_tx_credits = DEFAULT_PEER_CREDITS; + if (!tun->lt_cmn.lct_max_tx_credits) + tun->lt_cmn.lct_max_tx_credits = DEFAULT_CREDITS; + } +} + static int lnet_handle_legacy_ip2nets(char *ip2nets, struct lnet_ioctl_config_lnd_tunables *tun) { @@ -3109,6 +3122,8 @@ static int lnet_handle_legacy_ip2nets(char *ip2nets, if (rc < 0) return rc; + lnet_set_tune_defaults(tun); + mutex_lock(&the_lnet.ln_api_mutex); while ((net = list_first_entry_or_null(&net_head, struct lnet_net, @@ -3172,6 +3187,8 @@ int lnet_dyn_add_ni(struct lnet_ioctl_config_ni *conf) if (!ni) return -ENOMEM; + lnet_set_tune_defaults(tun); + mutex_lock(&the_lnet.ln_api_mutex); rc = lnet_add_net_common(net, tun); @@ -3304,13 +3321,16 @@ int lnet_dyn_del_ni(struct lnet_ioctl_config_ni *conf) memset(&tun, 0, sizeof(tun)); tun.lt_cmn.lct_peer_timeout = - conf->cfg_config_u.cfg_net.net_peer_timeout; + (!conf->cfg_config_u.cfg_net.net_peer_timeout) ? DEFAULT_PEER_TIMEOUT : + conf->cfg_config_u.cfg_net.net_peer_timeout; tun.lt_cmn.lct_peer_tx_credits = - conf->cfg_config_u.cfg_net.net_peer_tx_credits; + (!conf->cfg_config_u.cfg_net.net_peer_tx_credits) ? DEFAULT_PEER_CREDITS : + conf->cfg_config_u.cfg_net.net_peer_tx_credits; tun.lt_cmn.lct_peer_rtr_credits = conf->cfg_config_u.cfg_net.net_peer_rtr_credits; tun.lt_cmn.lct_max_tx_credits = - conf->cfg_config_u.cfg_net.net_max_tx_credits; + (!conf->cfg_config_u.cfg_net.net_max_tx_credits) ? DEFAULT_CREDITS : + conf->cfg_config_u.cfg_net.net_max_tx_credits; rc = lnet_add_net_common(net, &tun); -- 1.8.3.1 From jsimmons at infradead.org Thu Jul 2 00:04:52 2020 From: jsimmons at infradead.org (James Simmons) Date: Wed, 1 Jul 2020 20:04:52 -0400 Subject: [lustre-devel] [PATCH 12/18] lnet: Skip health and resends for single rail configs In-Reply-To: <1593648298-10571-1-git-send-email-jsimmons@infradead.org> References: <1593648298-10571-1-git-send-email-jsimmons@infradead.org> Message-ID: <1593648298-10571-13-git-send-email-jsimmons@infradead.org> From: Chris Horn If the sender of a message only has a single interface it doesn't make sense to have LNet track the health of that interface, nor should it attempt to resend a message when it encounters a local error. There aren't any alternative interfaces to use for a resend. Similarly, we needn't track health values of a peer's NIs if the peer only has a single interface. Nor do we need to attempt to resend a message to a peer with a single interface. There's an exception for routers. We rely on NI health to determine route aliveness, so even if a router only has a single interface we still need to track its health. We can use the ln_ping_target to get the count of local NIs, and the lnet_peer struct already contains a count of the number of peer NIs. HPE-bug-id: LUS-8826 WC-bug-id: https://jira.whamcloud.com/browse/LU-13501 Lustre-commit: c5381d73b1d83 ("LU-13501 lnet: Skip health and resends for single rail configs") Signed-off-by: Chris Horn Reviewed-on: https://review.whamcloud.com/38448 Reviewed-by: Serguei Smirnov Reviewed-by: Alexander Boyko Reviewed-by: Oleg Drokin Signed-off-by: James Simmons --- net/lnet/lnet/lib-msg.c | 65 ++++++++++++++++++++++++++++++++++--------------- 1 file changed, 46 insertions(+), 19 deletions(-) diff --git a/net/lnet/lnet/lib-msg.c b/net/lnet/lnet/lib-msg.c index 7ce9c47..f759b2d 100644 --- a/net/lnet/lnet/lib-msg.c +++ b/net/lnet/lnet/lib-msg.c @@ -774,6 +774,10 @@ struct lnet_peer_ni *lpni; struct lnet_ni *ni; bool lo = false; + bool attempt_local_resend; + bool attempt_remote_resend; + bool handle_local_health; + bool handle_remote_health; /* if we're shutting down no point in handling health. */ if (the_lnet.ln_mt_state != LNET_MT_STATE_RUNNING) @@ -800,9 +804,45 @@ if (msg->msg_tx_committed) { ni = msg->msg_txni; lpni = msg->msg_txpeer; + attempt_local_resend = true; + attempt_remote_resend = true; } else { ni = msg->msg_rxni; lpni = msg->msg_rxpeer; + attempt_local_resend = false; + attempt_remote_resend = false; + } + + /* Don't further decrement the health value if a recovery message + * failed. + */ + if (msg->msg_recovery) { + handle_local_health = false; + handle_remote_health = false; + } else { + handle_local_health = false; + handle_remote_health = true; + } + + /* For local failures, health/recovery/resends are not needed if I only + * have a single (non-lolnd) interface. NB: pb_nnis includes the lolnd + * interface, so a single-rail node would have pb_nnis == 2. + */ + if (the_lnet.ln_ping_target->pb_nnis <= 2) { + handle_local_health = false; + attempt_local_resend = false; + } + + /* For remote failures, health/recovery/resends are not needed if the + * peer only has a single interface. Special case for routers where we + * rely on health feature to manage route aliveness. NB: unlike pb_nnis + * above, lp_nnis does _not_ include the lolnd, so a single-rail node + * would have lp_nnis == 1. + */ + if (lpni && lpni->lpni_peer_net->lpn_peer->lp_nnis <= 1) { + attempt_remote_resend = false; + if (!lnet_isrouter(lpni)) + handle_remote_health = false; } if (!lo) @@ -865,41 +905,28 @@ case LNET_MSG_STATUS_LOCAL_ABORTED: case LNET_MSG_STATUS_LOCAL_NO_ROUTE: case LNET_MSG_STATUS_LOCAL_TIMEOUT: - /* don't further decrement the health value if the - * recovery message failed. - */ - if (!msg->msg_recovery) + if (handle_local_health) lnet_handle_local_failure(ni); - if (msg->msg_tx_committed) - /* add to the re-send queue */ + if (attempt_local_resend) return lnet_attempt_msg_resend(msg); break; - /* These errors will not trigger a resend so simply - * finalize the message - */ case LNET_MSG_STATUS_LOCAL_ERROR: - /* don't further decrement the health value if the - * recovery message failed. - */ - if (!msg->msg_recovery) + if (handle_local_health) lnet_handle_local_failure(ni); return -1; - /* TODO: since the remote dropped the message we can - * attempt a resend safely. - */ case LNET_MSG_STATUS_REMOTE_DROPPED: - if (!msg->msg_recovery) + if (handle_remote_health) lnet_handle_remote_failure(lpni); - if (msg->msg_tx_committed) + if (attempt_remote_resend) return lnet_attempt_msg_resend(msg); break; case LNET_MSG_STATUS_REMOTE_ERROR: case LNET_MSG_STATUS_REMOTE_TIMEOUT: case LNET_MSG_STATUS_NETWORK_TIMEOUT: - if (!msg->msg_recovery) + if (handle_remote_health) lnet_handle_remote_failure(lpni); return -1; default: -- 1.8.3.1 From jsimmons at infradead.org Thu Jul 2 00:04:57 2020 From: jsimmons at infradead.org (James Simmons) Date: Wed, 1 Jul 2020 20:04:57 -0400 Subject: [lustre-devel] [PATCH 17/18] lustre: obd: add new LPROCFS_TYPE_* In-Reply-To: <1593648298-10571-1-git-send-email-jsimmons@infradead.org> References: <1593648298-10571-1-git-send-email-jsimmons@infradead.org> Message-ID: <1593648298-10571-18-git-send-email-jsimmons@infradead.org> From: Emoly Liu Move LPROCFS_TYPE_LATENCY from llite later to lprocfs_status.h. Create new LPROCFS_TYPE_BYTES_FULL settings. WC-bug-id: https://jira.whamcloud.com/browse/LU-13597 Lustre-commit: cd8fb1e8d300c ("LU-13597 ofd: add more information to job_stats") Signed-off-by: Emoly Liu Reviewed-on: https://review.whamcloud.com/38816 Reviewed-by: Andreas Dilger Reviewed-by: Wang Shilong Reviewed-by: Oleg Drokin Signed-off-by: James Simmons --- fs/lustre/include/lprocfs_status.h | 9 +++++++-- fs/lustre/llite/lproc_llite.c | 8 ++------ 2 files changed, 9 insertions(+), 8 deletions(-) diff --git a/fs/lustre/include/lprocfs_status.h b/fs/lustre/include/lprocfs_status.h index 0d90b31..759e66b 100644 --- a/fs/lustre/include/lprocfs_status.h +++ b/fs/lustre/include/lprocfs_status.h @@ -143,6 +143,13 @@ enum { LPROCFS_TYPE_BYTES = 0x0200, LPROCFS_TYPE_PAGES = 0x0400, LPROCFS_TYPE_USEC = 0x0800, + + LPROCFS_TYPE_LATENCY = LPROCFS_TYPE_USEC | + LPROCFS_CNTR_AVGMINMAX | + LPROCFS_CNTR_STDDEV, + LPROCFS_TYPE_BYTES_FULL = LPROCFS_TYPE_BYTES | + LPROCFS_CNTR_AVGMINMAX | + LPROCFS_CNTR_STDDEV, }; #define LC_MIN_INIT ((~(u64)0) >> 1) @@ -364,8 +371,6 @@ enum { #define JOBSTATS_SESSION "session" /* obd_config.c */ -void lustre_register_client_process_config(int (*cpc)(struct lustre_cfg *lcfg)); - int lprocfs_stats_alloc_one(struct lprocfs_stats *stats, unsigned int cpuid); int lprocfs_stats_lock(struct lprocfs_stats *stats, diff --git a/fs/lustre/llite/lproc_llite.c b/fs/lustre/llite/lproc_llite.c index 4bce3a6..f5a1940 100644 --- a/fs/lustre/llite/lproc_llite.c +++ b/fs/lustre/llite/lproc_llite.c @@ -1552,18 +1552,14 @@ static void sbi_kobj_release(struct kobject *kobj) .release = sbi_kobj_release, }; -#define LPROCFS_TYPE_LATENCY \ - (LPROCFS_TYPE_USEC | LPROCFS_CNTR_AVGMINMAX | LPROCFS_CNTR_STDDEV) static const struct llite_file_opcode { u32 opcode; u32 type; const char *opname; } llite_opcode_table[LPROC_LL_FILE_OPCODES] = { /* file operation */ - { LPROC_LL_READ_BYTES, LPROCFS_CNTR_AVGMINMAX | LPROCFS_TYPE_BYTES, - "read_bytes" }, - { LPROC_LL_WRITE_BYTES, LPROCFS_CNTR_AVGMINMAX | LPROCFS_TYPE_BYTES, - "write_bytes" }, + { LPROC_LL_READ_BYTES, LPROCFS_TYPE_BYTES_FULL, "read_bytes" }, + { LPROC_LL_WRITE_BYTES, LPROCFS_TYPE_BYTES_FULL, "write_bytes" }, { LPROC_LL_READ, LPROCFS_TYPE_LATENCY, "read" }, { LPROC_LL_WRITE, LPROCFS_TYPE_LATENCY, "write" }, { LPROC_LL_IOCTL, LPROCFS_TYPE_REQS, "ioctl" }, -- 1.8.3.1 From jsimmons at infradead.org Thu Jul 2 00:04:54 2020 From: jsimmons at infradead.org (James Simmons) Date: Wed, 1 Jul 2020 20:04:54 -0400 Subject: [lustre-devel] [PATCH 14/18] lnet: define new network driver ptl4lnd In-Reply-To: <1593648298-10571-1-git-send-email-jsimmons@infradead.org> References: <1593648298-10571-1-git-send-email-jsimmons@infradead.org> Message-ID: <1593648298-10571-15-git-send-email-jsimmons@infradead.org> From: Gregoire Pichon Assign an ID to the new network driver ptl4lnd developed by Bull that implements a LND based on Portals 4 API. It is intended to be used with BXI, the Bull interconnect hardware. WC-bug-id: https://jira.whamcloud.com/browse/LU-8932 Lustre-commit: c54bf3faa29f5 ("LU-8932 lnet: define new network driver ptl4lnd") Signed-off-by: Gregoire Pichon Reviewed-on: https://review.whamcloud.com/24768 Reviewed-by: Doug Oucharek Reviewed-by: Andreas Dilger Reviewed-by: James Simmons Signed-off-by: James Simmons --- include/uapi/linux/lnet/nidstr.h | 1 + net/lnet/lnet/nidstrings.c | 9 +++++++++ 2 files changed, 10 insertions(+) diff --git a/include/uapi/linux/lnet/nidstr.h b/include/uapi/linux/lnet/nidstr.h index 9133641..34ba497 100644 --- a/include/uapi/linux/lnet/nidstr.h +++ b/include/uapi/linux/lnet/nidstr.h @@ -53,6 +53,7 @@ enum { /*MXLND = 12, removed v2_7_50_0-34-g8be9e41 */ GNILND = 13, GNIIPLND = 14, + PTL4LND = 15, NUM_LNDS }; diff --git a/net/lnet/lnet/nidstrings.c b/net/lnet/lnet/nidstrings.c index eca5092..fb8d3e2 100644 --- a/net/lnet/lnet/nidstrings.c +++ b/net/lnet/lnet/nidstrings.c @@ -692,6 +692,15 @@ int cfs_print_nidlist(char *buffer, int count, struct list_head *nidlist) .nf_print_addrlist = libcfs_ip_addr_range_print, .nf_match_addr = cfs_ip_addr_match }, + { .nf_type = PTL4LND, + .nf_name = "ptlf", + .nf_modname = "kptl4lnd", + .nf_addr2str = libcfs_decnum_addr2str, + .nf_str2addr = libcfs_num_str2addr, + .nf_parse_addrlist = libcfs_num_parse, + .nf_print_addrlist = libcfs_num_addr_range_print, + .nf_match_addr = libcfs_num_match + }, }; static const size_t libcfs_nnetstrfns = ARRAY_SIZE(libcfs_netstrfns); -- 1.8.3.1 From neilb at suse.de Thu Jul 2 04:47:35 2020 From: neilb at suse.de (NeilBrown) Date: Thu, 02 Jul 2020 14:47:35 +1000 Subject: [lustre-devel] [PATCH 00/18] Port of OpenSFS landing as of July 1, 2020 In-Reply-To: <1593648298-10571-1-git-send-email-jsimmons@infradead.org> References: <1593648298-10571-1-git-send-email-jsimmons@infradead.org> Message-ID: <87zh8i1q1k.fsf@notabene.neil.brown.name> On Wed, Jul 01 2020, James Simmons wrote: > Port of patches that landed to the OpenSFS branch. A few patches added > the were missing that enables potential lustre utilities building > against the Linux client. Please review to make sure everything is > okay. Hi James, could you please always be explicit about which Linux client tree these are ported to? It avoids confusion. Thanks, NeilBrown > > Alexander Boyko (1): > lustre: llite: don't hold inode_lock for security notify > > Alexey Lyashkov (2): > lnet: restore an maximal fragments count > lnet: o2ib: fix page mapping error > > Amir Shehata (1): > lnet: handle undefined parameters > > Andriy Skulysh (1): > lustre: llite: truncate deadlock with DoM files > > Chris Horn (1): > lnet: Skip health and resends for single rail configs > > Emoly Liu (1): > lustre: obd: add new LPROCFS_TYPE_* > > Gregoire Pichon (1): > lnet: define new network driver ptl4lnd > > Hongchao Zhang (1): > lustre: mdc: chlg device could be used after free > > James Simmons (2): > lustre: llite: bind kthread thread to accepted node set > lustre: lov: use lov_pattern_support() to verify lmm > > Lai Siyao (1): > lustre: mdt: don't fetch LOOKUP lock for remote object > > Mikhail Pershin (1): > lustre: ptlrpc: limit rate of lock replays > > Sebastien Buisson (5): > lustre: sec: encryption for write path > lustre: sec: decryption for read path > lustre: sec: deal with encrypted object size > lustre: sec: support truncate for encrypted files > lustre: sec: ioctls to handle encryption policies > > fs/lustre/include/lprocfs_status.h | 9 +- > fs/lustre/include/lustre_import.h | 2 + > fs/lustre/include/lustre_osc.h | 1 + > fs/lustre/include/obd.h | 19 ++- > fs/lustre/include/obd_class.h | 3 +- > fs/lustre/include/obd_support.h | 5 + > fs/lustre/ldlm/ldlm_request.c | 69 ++++++++++- > fs/lustre/llite/crypto.c | 15 ++- > fs/lustre/llite/dir.c | 50 +++++++- > fs/lustre/llite/file.c | 19 ++- > fs/lustre/llite/llite_internal.h | 1 + > fs/lustre/llite/llite_lib.c | 187 ++++++++++++++++++++++++++++- > fs/lustre/llite/lproc_llite.c | 8 +- > fs/lustre/llite/namei.c | 105 +++++++++++++--- > fs/lustre/llite/rw.c | 13 +- > fs/lustre/llite/rw26.c | 4 + > fs/lustre/llite/statahead.c | 11 +- > fs/lustre/llite/vvp_io.c | 17 ++- > fs/lustre/lmv/lmv_intent.c | 19 ++- > fs/lustre/lmv/lmv_internal.h | 1 + > fs/lustre/lmv/lmv_obd.c | 3 +- > fs/lustre/lov/lov_ea.c | 6 +- > fs/lustre/mdc/mdc_changelog.c | 46 ++++--- > fs/lustre/mdc/mdc_internal.h | 1 + > fs/lustre/mdc/mdc_request.c | 8 +- > fs/lustre/obdclass/genops.c | 1 + > fs/lustre/obdecho/echo_client.c | 2 + > fs/lustre/obdecho/echo_internal.h | 3 + > fs/lustre/osc/osc_internal.h | 1 + > fs/lustre/osc/osc_request.c | 121 ++++++++++++++++++- > fs/lustre/ptlrpc/import.c | 8 +- > include/linux/lnet/lib-lnet.h | 4 +- > include/linux/lnet/lib-types.h | 2 +- > include/uapi/linux/lnet/nidstr.h | 1 + > include/uapi/linux/lustre/lustre_user.h | 8 ++ > net/lnet/klnds/o2iblnd/o2iblnd.c | 7 +- > net/lnet/klnds/o2iblnd/o2iblnd_cb.c | 3 +- > net/lnet/klnds/o2iblnd/o2iblnd_modparams.c | 4 +- > net/lnet/klnds/socklnd/socklnd_modparams.c | 4 +- > net/lnet/lnet/api-ni.c | 26 +++- > net/lnet/lnet/lib-msg.c | 65 +++++++--- > net/lnet/lnet/nidstrings.c | 9 ++ > 42 files changed, 759 insertions(+), 132 deletions(-) > > -- > 1.8.3.1 > > _______________________________________________ > lustre-devel mailing list > lustre-devel at lists.lustre.org > http://lists.lustre.org/listinfo.cgi/lustre-devel-lustre.org -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 832 bytes Desc: not available URL: From neilb at suse.de Fri Jul 3 06:37:17 2020 From: neilb at suse.de (NeilBrown) Date: Fri, 03 Jul 2020 16:37:17 +1000 Subject: [lustre-devel] [PATCH 00/45] lustre: merged OpenSFS client patches from April 30 to today In-Reply-To: <87k0zx1dzs.fsf@notabene.neil.brown.name> References: <1590444502-20533-1-git-send-email-jsimmons@infradead.org> <87sgfj6ypq.fsf@notabene.neil.brown.name> <87pn9q1kxf.fsf@notabene.neil.brown.name> <87k0zx1dzs.fsf@notabene.neil.brown.name> Message-ID: <87tuyp14v6.fsf@notabene.neil.brown.name> On Wed, Jun 24 2020, NeilBrown wrote: >> >> By the way, where's your tree available Neil? I'm checking your github account but not sure this is there. >> > > In the "linux" repo I have quite a few branches. > "lustre/lustre" is the latest lustre code (as of late april). > > https://github.com/neilbrown/linux/tree/lustre/lustre Just FYI, I've updated this to Linux 5.7 and *almost* current master. There is one patch which causes all my testing to fail, so I've postponed that one until I understand the problem. NeilBrown -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 832 bytes Desc: not available URL: From jsimmons at infradead.org Wed Jul 15 20:44:41 2020 From: jsimmons at infradead.org (James Simmons) Date: Wed, 15 Jul 2020 16:44:41 -0400 Subject: [lustre-devel] [PATCH 00/37] lustre: latest patches landed to OpenSFS 07/14/2020 Message-ID: <1594845918-29027-1-git-send-email-jsimmons@infradead.org> Latest patches to landed to the OpenSFS tree as of July 14, 2020. Please review to make sure they are correct. Amir Shehata (2): lnet: socklnd: fix local interface binding lnet: check rtr_nid is a gateway Andreas Dilger (1): lustre: misc: quiet compiler warning on armv7l Chris Horn (2): lnet: Allow router to forward to healthier NID lnet: Set remote NI status in lnet_notify Hongchao Zhang (1): lustre: ptlrpc: fix endless loop issue James Simmons (1): lustre: ptlrpc: handle conn_hash rhashtable resize Mikhail Pershin (1): lustre: ptlrpc: re-enterable signal_completed_replay() Mr NeilBrown (18): lustre: osc: fix osc_extent_find() lustre: obdclass: remove init to 0 from lustre_init_lsi() lustre: lu_object: convert lu_object cache to rhashtable lnet: o2iblnd: allocate init_qp_attr on stack. lnet: Fix some out-of-date comments. lnet: socklnd: don't fall-back to tcp_sendpage. lustre: remove some "#ifdef CONFIG*" from .c files. lnet: o2iblnd: Use ib_mtu_int_to_enum() lnet: o2iblnd: wait properly for fps->increasing. lnet: o2iblnd: use need_resched() lnet: o2iblnd: Use list_for_each_entry_safe lnet: socklnd: use need_resched() lnet: socklnd: use list_for_each_entry_safe() lnet: socklnd: convert various refcounts to refcount_t lnet: libcfs: don't call unshare_fs_struct() lustre: llite: annotate non-owner locking lnet: remove LNetMEUnlink and clean up related code lnet: socklnd: change ksnd_nthreads to atomic_t Sebastien Buisson (1): lustre: sec: better struct sepol_downcall_data Shaun Tancheff (1): lustre: llite: Fix lock ordering in pagevec_dirty Vladimir Saveliev (1): lustre: osc: consume grants for direct I/O Wang Shilong (7): lustre: ldlm: check slv and limit before updating lustre: osc: disable ext merging for rdma only pages and non-rdma lustre: obdclass: use offset instead of cp_linkage lustre: obdclass: re-declare cl_page variables to reduce its size lustre: osc: re-declare ops_from/to to shrink osc_page lustre: llite: fix to free cl_dio_aio properly lustre: llite: fix short io for AIO Yang Sheng (1): lustre: obdcalss: ensure LCT_QUIESCENT take sync fs/lustre/include/cl_object.h | 34 +- fs/lustre/include/lu_object.h | 28 +- fs/lustre/include/lustre_osc.h | 12 +- fs/lustre/include/obd.h | 21 ++ fs/lustre/ldlm/ldlm_request.c | 8 + fs/lustre/llite/file.c | 32 +- fs/lustre/llite/llite_internal.h | 29 ++ fs/lustre/llite/llite_lib.c | 54 +-- fs/lustre/llite/rw26.c | 43 ++- fs/lustre/llite/vvp_dev.c | 105 ++---- fs/lustre/llite/vvp_internal.h | 3 +- fs/lustre/llite/vvp_io.c | 111 +++--- fs/lustre/lov/lovsub_dev.c | 5 +- fs/lustre/mdc/mdc_request.c | 8 +- fs/lustre/obdclass/cl_io.c | 19 +- fs/lustre/obdclass/cl_page.c | 360 ++++++++++--------- fs/lustre/obdclass/llog.c | 2 - fs/lustre/obdclass/lu_object.c | 539 ++++++++++++++--------------- fs/lustre/obdclass/lu_tgt_descs.c | 2 +- fs/lustre/obdclass/obd_mount.c | 6 +- fs/lustre/osc/osc_cache.c | 64 ++-- fs/lustre/osc/osc_page.c | 21 +- fs/lustre/ptlrpc/connection.c | 12 +- fs/lustre/ptlrpc/import.c | 6 +- fs/lustre/ptlrpc/niobuf.c | 12 +- fs/lustre/ptlrpc/pinger.c | 11 +- fs/lustre/ptlrpc/ptlrpcd.c | 1 - fs/lustre/ptlrpc/sec_lproc.c | 134 ++++++- fs/lustre/ptlrpc/service.c | 3 - include/linux/lnet/api.h | 6 +- include/linux/lnet/lib-lnet.h | 4 +- include/uapi/linux/lustre/lustre_user.h | 16 +- net/lnet/klnds/o2iblnd/o2iblnd.c | 103 ++---- net/lnet/klnds/o2iblnd/o2iblnd.h | 2 - net/lnet/klnds/o2iblnd/o2iblnd_cb.c | 12 +- net/lnet/klnds/o2iblnd/o2iblnd_modparams.c | 4 +- net/lnet/klnds/socklnd/socklnd.c | 97 +++--- net/lnet/klnds/socklnd/socklnd.h | 56 ++- net/lnet/klnds/socklnd/socklnd_cb.c | 26 +- net/lnet/klnds/socklnd/socklnd_lib.c | 8 +- net/lnet/lnet/api-ni.c | 5 +- net/lnet/lnet/lib-md.c | 62 ++-- net/lnet/lnet/lib-me.c | 39 --- net/lnet/lnet/lib-move.c | 57 ++- net/lnet/lnet/router.c | 6 +- net/lnet/selftest/rpc.c | 1 - 46 files changed, 1176 insertions(+), 1013 deletions(-) -- 1.8.3.1 From jsimmons at infradead.org Wed Jul 15 20:44:45 2020 From: jsimmons at infradead.org (James Simmons) Date: Wed, 15 Jul 2020 16:44:45 -0400 Subject: [lustre-devel] [PATCH 04/37] lustre: obdclass: remove init to 0 from lustre_init_lsi() In-Reply-To: <1594845918-29027-1-git-send-email-jsimmons@infradead.org> References: <1594845918-29027-1-git-send-email-jsimmons@infradead.org> Message-ID: <1594845918-29027-5-git-send-email-jsimmons@infradead.org> From: Mr NeilBrown After allocating a struct with kzalloc, there is no value in setting a few of the fields to zero. And as all fields were zero, it must be safe to kfree lmd_exclude, whether lmd_exclude_count is zero or not. WC-bug-id: https://jira.whamcloud.com/browse/LU-9679 Lustre-commit: 513dde601d2e9 ("LU-9679 obdclass: remove init to 0 from lustre_init_lsi()") Signed-off-by: Mr NeilBrown Reviewed-on: https://review.whamcloud.com/39135 Reviewed-by: James Simmons Reviewed-by: Yang Sheng Reviewed-by: Oleg Drokin Signed-off-by: James Simmons --- fs/lustre/obdclass/obd_mount.c | 6 +----- 1 file changed, 1 insertion(+), 5 deletions(-) diff --git a/fs/lustre/obdclass/obd_mount.c b/fs/lustre/obdclass/obd_mount.c index 13e6521..ea5b469 100644 --- a/fs/lustre/obdclass/obd_mount.c +++ b/fs/lustre/obdclass/obd_mount.c @@ -515,9 +515,6 @@ struct lustre_sb_info *lustre_init_lsi(struct super_block *sb) return NULL; } - lsi->lsi_lmd->lmd_exclude_count = 0; - lsi->lsi_lmd->lmd_recovery_time_soft = 0; - lsi->lsi_lmd->lmd_recovery_time_hard = 0; s2lsi_nocast(sb) = lsi; /* we take 1 extra ref for our setup */ atomic_set(&lsi->lsi_mounts, 1); @@ -544,8 +541,7 @@ static int lustre_free_lsi(struct super_block *sb) kfree(lsi->lsi_lmd->lmd_fileset); kfree(lsi->lsi_lmd->lmd_mgssec); kfree(lsi->lsi_lmd->lmd_opts); - if (lsi->lsi_lmd->lmd_exclude_count) - kfree(lsi->lsi_lmd->lmd_exclude); + kfree(lsi->lsi_lmd->lmd_exclude); kfree(lsi->lsi_lmd->lmd_mgs); kfree(lsi->lsi_lmd->lmd_osd_type); kfree(lsi->lsi_lmd->lmd_params); -- 1.8.3.1 From jsimmons at infradead.org Wed Jul 15 20:44:49 2020 From: jsimmons at infradead.org (James Simmons) Date: Wed, 15 Jul 2020 16:44:49 -0400 Subject: [lustre-devel] [PATCH 08/37] lnet: socklnd: fix local interface binding In-Reply-To: <1594845918-29027-1-git-send-email-jsimmons@infradead.org> References: <1594845918-29027-1-git-send-email-jsimmons@infradead.org> Message-ID: <1594845918-29027-9-git-send-email-jsimmons@infradead.org> From: Amir Shehata When a node is configured with multiple interfaces in Multi-Rail config, socklnd was not utilizing the local interface requested by LNet. In essence LNet was using all the NIDs in round robin, however the socklnd module was not binding to the correct interface. Traffic was thus sent on a subset of the interfaces. The reason is that the route interface number was not being set. In most cases lnet_connect() is called to create a socket. The socket is bound to the interface provided and then ksocknal_create_conn() is called to create the socklnd connection. ksocknal_create_conn() calls ksocknal_associate_route_conn_locked() at which point the route's local interface is assigned. However, this is already too late as the socket has already been created and bound to a local interface. Therefore, it's important to assign the route's interface before calling lnet_connect() to ensure socket is bound to correct local interface. To address this issue, the route's interface index is initialized to the NI's interface index when it's added to the peer_ni. Another bug fixed: The interface index was not being initialized in the startup routine. Note: We're strictly assuming that there is one interface for each NI. This is because tcp bonding will be removed from the socklnd as it has been deprecated by LNet mutli-rail. WC-bug-id: https://jira.whamcloud.com/browse/LU-13566 Lustre-commit: a7c9aba5eb96d ("LU-13566 socklnd: fix local interface binding") Signed-off-by: Amir Shehata Reviewed-on: https://review.whamcloud.com/38743 Reviewed-by: Neil Brown Reviewed-by: Serguei Smirnov Reviewed-by: James Simmons Reviewed-by: Oleg Drokin Signed-off-by: James Simmons --- net/lnet/klnds/socklnd/socklnd.c | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/net/lnet/klnds/socklnd/socklnd.c b/net/lnet/klnds/socklnd/socklnd.c index 444b90b..2b8fd3d 100644 --- a/net/lnet/klnds/socklnd/socklnd.c +++ b/net/lnet/klnds/socklnd/socklnd.c @@ -409,12 +409,14 @@ struct ksock_peer_ni * { struct ksock_conn *conn; struct ksock_route *route2; + struct ksock_net *net = peer_ni->ksnp_ni->ni_data; LASSERT(!peer_ni->ksnp_closing); LASSERT(!route->ksnr_peer); LASSERT(!route->ksnr_scheduled); LASSERT(!route->ksnr_connecting); LASSERT(!route->ksnr_connected); + LASSERT(net->ksnn_ninterfaces > 0); /* LASSERT(unique) */ list_for_each_entry(route2, &peer_ni->ksnp_routes, ksnr_list) { @@ -428,6 +430,11 @@ struct ksock_peer_ni * route->ksnr_peer = peer_ni; ksocknal_peer_addref(peer_ni); + + /* set the route's interface to the current net's interface */ + route->ksnr_myiface = net->ksnn_interfaces[0].ksni_index; + net->ksnn_interfaces[0].ksni_nroutes++; + /* peer_ni's routelist takes over my ref on 'route' */ list_add_tail(&route->ksnr_list, &peer_ni->ksnp_routes); @@ -2667,6 +2674,7 @@ static int ksocknal_push(struct lnet_ni *ni, struct lnet_process_id id) net->ksnn_ninterfaces = 1; ni->ni_dev_cpt = ifaces[0].li_cpt; ksi->ksni_ipaddr = ifaces[0].li_ipaddr; + ksi->ksni_index = ksocknal_ip2index(ksi->ksni_ipaddr, ni); ksi->ksni_netmask = ifaces[0].li_netmask; strlcpy(ksi->ksni_name, ifaces[0].li_name, sizeof(ksi->ksni_name)); @@ -2706,6 +2714,8 @@ static int ksocknal_push(struct lnet_ni *ni, struct lnet_process_id id) ksi = &net->ksnn_interfaces[j]; ni->ni_dev_cpt = ifaces[j].li_cpt; ksi->ksni_ipaddr = ifaces[j].li_ipaddr; + ksi->ksni_index = + ksocknal_ip2index(ksi->ksni_ipaddr, ni); ksi->ksni_netmask = ifaces[j].li_netmask; strlcpy(ksi->ksni_name, ifaces[j].li_name, sizeof(ksi->ksni_name)); -- 1.8.3.1 From jsimmons at infradead.org Wed Jul 15 20:44:47 2020 From: jsimmons at infradead.org (James Simmons) Date: Wed, 15 Jul 2020 16:44:47 -0400 Subject: [lustre-devel] [PATCH 06/37] lustre: lu_object: convert lu_object cache to rhashtable In-Reply-To: <1594845918-29027-1-git-send-email-jsimmons@infradead.org> References: <1594845918-29027-1-git-send-email-jsimmons@infradead.org> Message-ID: <1594845918-29027-7-git-send-email-jsimmons@infradead.org> From: Mr NeilBrown The lu_object cache is a little more complex than the other lustre hash tables for two reasons. 1/ there is a debugfs file which displays the contents of the cache, so we need to use rhashtable_walk in a way that works for seq_file. 2/ There is a (sharded) lru list for objects which are no longer referenced, so finding an object needs to consider races with the lru as well as with the hash table. The debugfs file already manages walking the libcfs hash table keeping a current-position in the private data. We can fairly easily convert that to a struct rhashtable_iter. The debugfs file actually reports pages, and there are multiple pages per hashtable object. So as well as rhashtable_iter, we need the current page index. For the double-locking, the current code uses direct-access to the bucket locks that libcfs_hash provides. rhashtable doesn't provide that access - callers must provide their own locking or use rcu techniques. The lsb_waitq.lock is still used to manage the lru list, but with this patch it is no longer nested *inside* the hashtable locks, but instead is outside. It is used to protect an object with a refcount of zero. When purging old objects from an lru, we first set LU_OBJECT_HEARD_BANSHEE while holding the lsb_waitq.lock, then remove all the entries from the hashtable separately. When removing the last reference from an object, we first take the lsb_waitq.lock, then decrement the reference and add to the lru list or discard it setting LU_OBJECT_UNHASHED. When we find an object in the hashtable with a refcount of zero, we take the corresponding lsb_waitq.lock and check that neither LU_OBJECT_HEARD_BANSHEE or LU_OBJECT_UNHASHED is set. If neither is, we can safely increment the refcount. If either are, the object is gone. This way, we only ever manipulate an object with a refcount of zero while holding the lsb_waitq.lock. As there is nothing to stop us using the resizing capabilities of rhashtable, the code to try to guess the perfect hash size has been removed. Also: the "is_dying" variable in lu_object_put() is racey - the value could change the moment it is sampled. It is also not needed as it is only used to avoid a wakeup, which is not particularly expensive. In the same code as comment says that 'top' could not be accessed, but the code then immediately accesses 'top' to calculate 'bkt'. So move the initialization of 'bkt' to before 'top' becomes unsafe. Also: Change "wake_up_all()" to "wake_up()". wake_up_all() is only relevant when an exclusive wait is used. Moving from the libcfs hashtable to rhashtable also gives the benefit of a very large performance boost. Before patch: SUMMARY rate: (of 5 iterations) Operation Max Min Mean Std Dev --------- --- --- ---- ------- Directory creation: 12036.610 11091.880 11452.978 318.829 Directory stat: 25871.734 24232.310 24935.661 574.996 Directory removal: 12698.769 12239.685 12491.008 149.149 File creation: 11722.036 11673.961 11692.157 15.966 File stat: 62304.540 58237.124 60282.003 1479.103 File read: 24204.811 23889.091 24048.577 110.245 File removal: 9412.930 9111.828 9217.546 120.894 Tree creation: 3515.536 3195.627 3442.609 123.792 Tree removal: 433.917 418.935 428.038 5.545 After patch: SUMMARY rate: (of 5 iterations) Operation Max Min Mean Std Dev --------- --- --- ---- ------- Directory creation: 11873.308 303.626 9371.860 4539.539 Directory stat: 31116.512 30190.574 30568.091 335.545 Directory removal: 13082.121 12645.228 12943.239 157.695 File creation: 12607.135 12293.319 12466.647 138.307 File stat: 124419.347 105240.996 116919.977 7847.165 File read: 39707.270 36295.477 38266.011 1328.857 File removal: 9614.333 9273.931 9477.299 140.201 Tree creation: 3572.602 3017.580 3339.547 207.061 Tree removal: 487.687 0.004 282.188 230.659 WC-bug-id: https://jira.whamcloud.com/browse/LU-8130 Lustre-commit: aff14dbc522e1 ("LU-8130 lu_object: convert lu_object cache to rhashtable") Signed-off-by: Mr NeilBrown Reviewed-on: https://review.whamcloud.com/36707 Reviewed-by: James Simmons Reviewed-by: Shaun Tancheff Reviewed-by: Oleg Drokin Signed-off-by: James Simmons --- fs/lustre/include/lu_object.h | 20 +- fs/lustre/llite/vvp_dev.c | 105 +++------ fs/lustre/lov/lovsub_dev.c | 5 +- fs/lustre/obdclass/lu_object.c | 481 +++++++++++++++++++---------------------- 4 files changed, 272 insertions(+), 339 deletions(-) diff --git a/fs/lustre/include/lu_object.h b/fs/lustre/include/lu_object.h index 2a2f38e..1a6b6e1 100644 --- a/fs/lustre/include/lu_object.h +++ b/fs/lustre/include/lu_object.h @@ -36,6 +36,7 @@ #include #include +#include #include #include #include @@ -469,11 +470,6 @@ enum lu_object_header_flags { * initialized yet, the object allocator will initialize it. */ LU_OBJECT_INITED = 2, - /** - * Object is being purged, so mustn't be returned by - * htable_lookup() - */ - LU_OBJECT_PURGING = 3, }; enum lu_object_header_attr { @@ -496,6 +492,8 @@ enum lu_object_header_attr { * it is created for things like not-yet-existing child created by mkdir or * create calls. lu_object_operations::loo_exists() can be used to check * whether object is backed by persistent storage entity. + * Any object containing this structre which might be placed in an + * rhashtable via loh_hash MUST be freed using call_rcu() or rcu_kfree(). */ struct lu_object_header { /** @@ -517,9 +515,9 @@ struct lu_object_header { */ u32 loh_attr; /** - * Linkage into per-site hash table. Protected by lu_site::ls_guard. + * Linkage into per-site hash table. */ - struct hlist_node loh_hash; + struct rhash_head loh_hash; /** * Linkage into per-site LRU list. Protected by lu_site::ls_guard. */ @@ -566,7 +564,7 @@ struct lu_site { /** * objects hash table */ - struct cfs_hash *ls_obj_hash; + struct rhashtable ls_obj_hash; /* * buckets for summary data */ @@ -643,6 +641,8 @@ int lu_object_init(struct lu_object *o, void lu_object_fini(struct lu_object *o); void lu_object_add_top(struct lu_object_header *h, struct lu_object *o); void lu_object_add(struct lu_object *before, struct lu_object *o); +struct lu_object *lu_object_get_first(struct lu_object_header *h, + struct lu_device *dev); /** * Helpers to initialize and finalize device types. @@ -697,8 +697,8 @@ static inline int lu_site_purge(const struct lu_env *env, struct lu_site *s, return lu_site_purge_objects(env, s, nr, true); } -void lu_site_print(const struct lu_env *env, struct lu_site *s, void *cookie, - lu_printer_t printer); +void lu_site_print(const struct lu_env *env, struct lu_site *s, atomic_t *ref, + int msg_flags, lu_printer_t printer); struct lu_object *lu_object_find_at(const struct lu_env *env, struct lu_device *dev, const struct lu_fid *f, diff --git a/fs/lustre/llite/vvp_dev.c b/fs/lustre/llite/vvp_dev.c index e1d87f9..aa8b2c5 100644 --- a/fs/lustre/llite/vvp_dev.c +++ b/fs/lustre/llite/vvp_dev.c @@ -361,21 +361,13 @@ int cl_sb_fini(struct super_block *sb) * ****************************************************************************/ -struct vvp_pgcache_id { - unsigned int vpi_bucket; - unsigned int vpi_depth; - u32 vpi_index; - - unsigned int vpi_curdep; - struct lu_object_header *vpi_obj; -}; - struct vvp_seq_private { struct ll_sb_info *vsp_sbi; struct lu_env *vsp_env; u16 vsp_refcheck; struct cl_object *vsp_clob; - struct vvp_pgcache_id vsp_id; + struct rhashtable_iter vsp_iter; + u32 vsp_page_index; /* * prev_pos is the 'pos' of the last object returned * by ->start of ->next. @@ -383,81 +375,43 @@ struct vvp_seq_private { loff_t vsp_prev_pos; }; -static int vvp_pgcache_obj_get(struct cfs_hash *hs, struct cfs_hash_bd *bd, - struct hlist_node *hnode, void *data) -{ - struct vvp_pgcache_id *id = data; - struct lu_object_header *hdr = cfs_hash_object(hs, hnode); - - if (lu_object_is_dying(hdr)) - return 1; - - if (id->vpi_curdep-- > 0) - return 0; /* continue */ - - cfs_hash_get(hs, hnode); - id->vpi_obj = hdr; - return 1; -} - -static struct cl_object *vvp_pgcache_obj(const struct lu_env *env, - struct lu_device *dev, - struct vvp_pgcache_id *id) -{ - LASSERT(lu_device_is_cl(dev)); - - id->vpi_obj = NULL; - id->vpi_curdep = id->vpi_depth; - - cfs_hash_hlist_for_each(dev->ld_site->ls_obj_hash, id->vpi_bucket, - vvp_pgcache_obj_get, id); - if (id->vpi_obj) { - struct lu_object *lu_obj; - - lu_obj = lu_object_locate(id->vpi_obj, dev->ld_type); - if (lu_obj) { - lu_object_ref_add(lu_obj, "dump", current); - return lu2cl(lu_obj); - } - lu_object_put(env, lu_object_top(id->vpi_obj)); - } - return NULL; -} - static struct page *vvp_pgcache_current(struct vvp_seq_private *priv) { struct lu_device *dev = &priv->vsp_sbi->ll_cl->cd_lu_dev; + struct lu_object_header *h; + struct page *vmpage = NULL; - while (1) { + rhashtable_walk_start(&priv->vsp_iter); + while ((h = rhashtable_walk_next(&priv->vsp_iter)) != NULL) { struct inode *inode; - struct page *vmpage; int nr; if (!priv->vsp_clob) { - struct cl_object *clob; - - while ((clob = vvp_pgcache_obj(priv->vsp_env, dev, &priv->vsp_id)) == NULL && - ++(priv->vsp_id.vpi_bucket) < CFS_HASH_NHLIST(dev->ld_site->ls_obj_hash)) - priv->vsp_id.vpi_depth = 0; - if (!clob) - return NULL; - priv->vsp_clob = clob; - priv->vsp_id.vpi_index = 0; + struct lu_object *lu_obj; + + lu_obj = lu_object_get_first(h, dev); + if (!lu_obj) + continue; + + priv->vsp_clob = lu2cl(lu_obj); + lu_object_ref_add(lu_obj, "dump", current); + priv->vsp_page_index = 0; } inode = vvp_object_inode(priv->vsp_clob); nr = find_get_pages_contig(inode->i_mapping, - priv->vsp_id.vpi_index, 1, &vmpage); + priv->vsp_page_index, 1, &vmpage); if (nr > 0) { - priv->vsp_id.vpi_index = vmpage->index; - return vmpage; + priv->vsp_page_index = vmpage->index; + break; } lu_object_ref_del(&priv->vsp_clob->co_lu, "dump", current); cl_object_put(priv->vsp_env, priv->vsp_clob); priv->vsp_clob = NULL; - priv->vsp_id.vpi_index = 0; - priv->vsp_id.vpi_depth++; + priv->vsp_page_index = 0; } + rhashtable_walk_stop(&priv->vsp_iter); + return vmpage; } #define seq_page_flag(seq, page, flag, has_flags) do { \ @@ -521,7 +475,10 @@ static int vvp_pgcache_show(struct seq_file *f, void *v) static void vvp_pgcache_rewind(struct vvp_seq_private *priv) { if (priv->vsp_prev_pos) { - memset(&priv->vsp_id, 0, sizeof(priv->vsp_id)); + struct lu_site *s = priv->vsp_sbi->ll_cl->cd_lu_dev.ld_site; + + rhashtable_walk_exit(&priv->vsp_iter); + rhashtable_walk_enter(&s->ls_obj_hash, &priv->vsp_iter); priv->vsp_prev_pos = 0; if (priv->vsp_clob) { lu_object_ref_del(&priv->vsp_clob->co_lu, "dump", @@ -534,7 +491,7 @@ static void vvp_pgcache_rewind(struct vvp_seq_private *priv) static struct page *vvp_pgcache_next_page(struct vvp_seq_private *priv) { - priv->vsp_id.vpi_index += 1; + priv->vsp_page_index += 1; return vvp_pgcache_current(priv); } @@ -548,7 +505,7 @@ static void *vvp_pgcache_start(struct seq_file *f, loff_t *pos) /* Return the current item */; } else { WARN_ON(*pos != priv->vsp_prev_pos + 1); - priv->vsp_id.vpi_index += 1; + priv->vsp_page_index += 1; } priv->vsp_prev_pos = *pos; @@ -580,6 +537,7 @@ static void vvp_pgcache_stop(struct seq_file *f, void *v) static int vvp_dump_pgcache_seq_open(struct inode *inode, struct file *filp) { struct vvp_seq_private *priv; + struct lu_site *s; priv = __seq_open_private(filp, &vvp_pgcache_ops, sizeof(*priv)); if (!priv) @@ -588,13 +546,16 @@ static int vvp_dump_pgcache_seq_open(struct inode *inode, struct file *filp) priv->vsp_sbi = inode->i_private; priv->vsp_env = cl_env_get(&priv->vsp_refcheck); priv->vsp_clob = NULL; - memset(&priv->vsp_id, 0, sizeof(priv->vsp_id)); if (IS_ERR(priv->vsp_env)) { int err = PTR_ERR(priv->vsp_env); seq_release_private(inode, filp); return err; } + + s = priv->vsp_sbi->ll_cl->cd_lu_dev.ld_site; + rhashtable_walk_enter(&s->ls_obj_hash, &priv->vsp_iter); + return 0; } @@ -607,8 +568,8 @@ static int vvp_dump_pgcache_seq_release(struct inode *inode, struct file *file) lu_object_ref_del(&priv->vsp_clob->co_lu, "dump", current); cl_object_put(priv->vsp_env, priv->vsp_clob); } - cl_env_put(priv->vsp_env, &priv->vsp_refcheck); + rhashtable_walk_exit(&priv->vsp_iter); return seq_release_private(inode, file); } diff --git a/fs/lustre/lov/lovsub_dev.c b/fs/lustre/lov/lovsub_dev.c index 69380fc..0555737 100644 --- a/fs/lustre/lov/lovsub_dev.c +++ b/fs/lustre/lov/lovsub_dev.c @@ -88,10 +88,7 @@ static struct lu_device *lovsub_device_free(const struct lu_env *env, struct lovsub_device *lsd = lu2lovsub_dev(d); struct lu_device *next = cl2lu_dev(lsd->acid_next); - if (atomic_read(&d->ld_ref) && d->ld_site) { - LIBCFS_DEBUG_MSG_DATA_DECL(msgdata, D_ERROR, NULL); - lu_site_print(env, d->ld_site, &msgdata, lu_cdebug_printer); - } + lu_site_print(env, d->ld_site, &d->ld_ref, D_ERROR, lu_cdebug_printer); cl_device_fini(lu2cl_dev(d)); kfree(lsd); return next; diff --git a/fs/lustre/obdclass/lu_object.c b/fs/lustre/obdclass/lu_object.c index ec3f6a3..5cd8231 100644 --- a/fs/lustre/obdclass/lu_object.c +++ b/fs/lustre/obdclass/lu_object.c @@ -41,12 +41,11 @@ #define DEBUG_SUBSYSTEM S_CLASS +#include #include #include #include -/* hash_long() */ -#include #include #include #include @@ -85,12 +84,10 @@ enum { #define LU_CACHE_NR_MAX_ADJUST 512 #define LU_CACHE_NR_UNLIMITED -1 #define LU_CACHE_NR_DEFAULT LU_CACHE_NR_UNLIMITED -#define LU_CACHE_NR_LDISKFS_LIMIT LU_CACHE_NR_UNLIMITED -#define LU_CACHE_NR_ZFS_LIMIT 256 -#define LU_SITE_BITS_MIN 12 -#define LU_SITE_BITS_MAX 24 -#define LU_SITE_BITS_MAX_CL 19 +#define LU_CACHE_NR_MIN 4096 +#define LU_CACHE_NR_MAX 0x80000000UL + /** * Max 256 buckets, we don't want too many buckets because: * - consume too much memory (currently max 16K) @@ -111,7 +108,7 @@ enum { static void lu_object_free(const struct lu_env *env, struct lu_object *o); static u32 ls_stats_read(struct lprocfs_stats *stats, int idx); -static u32 lu_fid_hash(const void *data, u32 seed) +static u32 lu_fid_hash(const void *data, u32 len, u32 seed) { const struct lu_fid *fid = data; @@ -120,9 +117,17 @@ static u32 lu_fid_hash(const void *data, u32 seed) return seed; } +static const struct rhashtable_params obj_hash_params = { + .key_len = sizeof(struct lu_fid), + .key_offset = offsetof(struct lu_object_header, loh_fid), + .head_offset = offsetof(struct lu_object_header, loh_hash), + .hashfn = lu_fid_hash, + .automatic_shrinking = true, +}; + static inline int lu_bkt_hash(struct lu_site *s, const struct lu_fid *fid) { - return lu_fid_hash(fid, s->ls_bkt_seed) & + return lu_fid_hash(fid, sizeof(*fid), s->ls_bkt_seed) & (s->ls_bkt_cnt - 1); } @@ -147,9 +152,7 @@ void lu_object_put(const struct lu_env *env, struct lu_object *o) struct lu_object_header *top = o->lo_header; struct lu_site *site = o->lo_dev->ld_site; struct lu_object *orig = o; - struct cfs_hash_bd bd; const struct lu_fid *fid = lu_object_fid(o); - bool is_dying; /* * till we have full fids-on-OST implemented anonymous objects @@ -157,7 +160,6 @@ void lu_object_put(const struct lu_env *env, struct lu_object *o) * so we should not remove it from the site. */ if (fid_is_zero(fid)) { - LASSERT(!top->loh_hash.next && !top->loh_hash.pprev); LASSERT(list_empty(&top->loh_lru)); if (!atomic_dec_and_test(&top->loh_ref)) return; @@ -169,40 +171,45 @@ void lu_object_put(const struct lu_env *env, struct lu_object *o) return; } - cfs_hash_bd_get(site->ls_obj_hash, &top->loh_fid, &bd); - - is_dying = lu_object_is_dying(top); - if (!cfs_hash_bd_dec_and_lock(site->ls_obj_hash, &bd, &top->loh_ref)) { - /* at this point the object reference is dropped and lock is + bkt = &site->ls_bkts[lu_bkt_hash(site, &top->loh_fid)]; + if (atomic_add_unless(&top->loh_ref, -1, 1)) { +still_active: + /* + * At this point the object reference is dropped and lock is * not taken, so lu_object should not be touched because it - * can be freed by concurrent thread. Use local variable for - * check. + * can be freed by concurrent thread. + * + * Somebody may be waiting for this, currently only used for + * cl_object, see cl_object_put_last(). */ - if (is_dying) { - /* - * somebody may be waiting for this, currently only - * used for cl_object, see cl_object_put_last(). - */ - bkt = &site->ls_bkts[lu_bkt_hash(site, &top->loh_fid)]; - wake_up_all(&bkt->lsb_waitq); - } + wake_up(&bkt->lsb_waitq); + return; } + spin_lock(&bkt->lsb_waitq.lock); + if (!atomic_dec_and_test(&top->loh_ref)) { + spin_unlock(&bkt->lsb_waitq.lock); + goto still_active; + } + /* - * When last reference is released, iterate over object - * layers, and notify them that object is no longer busy. + * Refcount is zero, and cannot be incremented without taking the bkt + * lock, so object is stable. + */ + + /* + * When last reference is released, iterate over object layers, and + * notify them that object is no longer busy. */ list_for_each_entry_reverse(o, &top->loh_layers, lo_linkage) { if (o->lo_ops->loo_object_release) o->lo_ops->loo_object_release(env, o); } - bkt = &site->ls_bkts[lu_bkt_hash(site, &top->loh_fid)]; - spin_lock(&bkt->lsb_waitq.lock); - - /* don't use local 'is_dying' here because if was taken without lock - * but here we need the latest actual value of it so check lu_object + /* + * Don't use local 'is_dying' here because if was taken without lock but + * here we need the latest actual value of it so check lu_object * directly here. */ if (!lu_object_is_dying(top)) { @@ -210,26 +217,26 @@ void lu_object_put(const struct lu_env *env, struct lu_object *o) list_add_tail(&top->loh_lru, &bkt->lsb_lru); spin_unlock(&bkt->lsb_waitq.lock); percpu_counter_inc(&site->ls_lru_len_counter); - CDEBUG(D_INODE, "Add %p/%p to site lru. hash: %p, bkt: %p\n", - orig, top, site->ls_obj_hash, bkt); - cfs_hash_bd_unlock(site->ls_obj_hash, &bd, 1); + CDEBUG(D_INODE, "Add %p/%p to site lru. bkt: %p\n", + orig, top, bkt); return; } /* - * If object is dying (will not be cached), then removed it - * from hash table (it is already not on the LRU). + * If object is dying (will not be cached) then removed it from hash + * table (it is already not on the LRU). * - * This is done with hash table lists locked. As the only - * way to acquire first reference to previously unreferenced - * object is through hash-table lookup (lu_object_find()) - * which is done under hash-table, no race with concurrent - * object lookup is possible and we can safely destroy object below. + * This is done with bucket lock held. As the only way to acquire first + * reference to previously unreferenced object is through hash-table + * lookup (lu_object_find()) which takes the lock for first reference, + * no race with concurrent object lookup is possible and we can safely + * destroy object below. */ if (!test_and_set_bit(LU_OBJECT_UNHASHED, &top->loh_flags)) - cfs_hash_bd_del_locked(site->ls_obj_hash, &bd, &top->loh_hash); + rhashtable_remove_fast(&site->ls_obj_hash, &top->loh_hash, + obj_hash_params); + spin_unlock(&bkt->lsb_waitq.lock); - cfs_hash_bd_unlock(site->ls_obj_hash, &bd, 1); /* Object was already removed from hash above, can kill it. */ lu_object_free(env, orig); } @@ -247,21 +254,19 @@ void lu_object_unhash(const struct lu_env *env, struct lu_object *o) set_bit(LU_OBJECT_HEARD_BANSHEE, &top->loh_flags); if (!test_and_set_bit(LU_OBJECT_UNHASHED, &top->loh_flags)) { struct lu_site *site = o->lo_dev->ld_site; - struct cfs_hash *obj_hash = site->ls_obj_hash; - struct cfs_hash_bd bd; + struct rhashtable *obj_hash = &site->ls_obj_hash; + struct lu_site_bkt_data *bkt; - cfs_hash_bd_get_and_lock(obj_hash, &top->loh_fid, &bd, 1); + bkt = &site->ls_bkts[lu_bkt_hash(site, &top->loh_fid)]; + spin_lock(&bkt->lsb_waitq.lock); if (!list_empty(&top->loh_lru)) { - struct lu_site_bkt_data *bkt; - - bkt = &site->ls_bkts[lu_bkt_hash(site, &top->loh_fid)]; - spin_lock(&bkt->lsb_waitq.lock); list_del_init(&top->loh_lru); - spin_unlock(&bkt->lsb_waitq.lock); percpu_counter_dec(&site->ls_lru_len_counter); } - cfs_hash_bd_del_locked(obj_hash, &bd, &top->loh_hash); - cfs_hash_bd_unlock(obj_hash, &bd, 1); + spin_unlock(&bkt->lsb_waitq.lock); + + rhashtable_remove_fast(obj_hash, &top->loh_hash, + obj_hash_params); } } EXPORT_SYMBOL(lu_object_unhash); @@ -445,11 +450,9 @@ int lu_site_purge_objects(const struct lu_env *env, struct lu_site *s, LINVRNT(lu_bkt_hash(s, &h->loh_fid) == i); - /* Cannot remove from hash under current spinlock, - * so set flag to stop object from being found - * by htable_lookup(). - */ - set_bit(LU_OBJECT_PURGING, &h->loh_flags); + set_bit(LU_OBJECT_UNHASHED, &h->loh_flags); + rhashtable_remove_fast(&s->ls_obj_hash, &h->loh_hash, + obj_hash_params); list_move(&h->loh_lru, &dispose); percpu_counter_dec(&s->ls_lru_len_counter); if (did_sth == 0) @@ -470,7 +473,6 @@ int lu_site_purge_objects(const struct lu_env *env, struct lu_site *s, while ((h = list_first_entry_or_null(&dispose, struct lu_object_header, loh_lru)) != NULL) { - cfs_hash_del(s->ls_obj_hash, &h->loh_fid, &h->loh_hash); list_del_init(&h->loh_lru); lu_object_free(env, lu_object_top(h)); lprocfs_counter_incr(s->ls_stats, LU_SS_LRU_PURGED); @@ -582,9 +584,9 @@ void lu_object_header_print(const struct lu_env *env, void *cookie, (*printer)(env, cookie, "header@%p[%#lx, %d, " DFID "%s%s%s]", hdr, hdr->loh_flags, atomic_read(&hdr->loh_ref), PFID(&hdr->loh_fid), - hlist_unhashed(&hdr->loh_hash) ? "" : " hash", - list_empty((struct list_head *)&hdr->loh_lru) ? \ - "" : " lru", + test_bit(LU_OBJECT_UNHASHED, + &hdr->loh_flags) ? "" : " hash", + list_empty(&hdr->loh_lru) ? "" : " lru", hdr->loh_attr & LOHA_EXISTS ? " exist":""); } EXPORT_SYMBOL(lu_object_header_print); @@ -621,54 +623,94 @@ void lu_object_print(const struct lu_env *env, void *cookie, EXPORT_SYMBOL(lu_object_print); /* - * NOTE: htable_lookup() is called with the relevant - * hash bucket locked, but might drop and re-acquire the lock. + * Limit the lu_object cache to a maximum of lu_cache_nr objects. Because the + * calculation for the number of objects to reclaim is not covered by a lock the + * maximum number of objects is capped by LU_CACHE_MAX_ADJUST. This ensures + * that many concurrent threads will not accidentally purge the entire cache. */ -static struct lu_object *htable_lookup(struct lu_site *s, - struct cfs_hash_bd *bd, +static void lu_object_limit(const struct lu_env *env, + struct lu_device *dev) +{ + u64 size, nr; + + if (lu_cache_nr == LU_CACHE_NR_UNLIMITED) + return; + + size = atomic_read(&dev->ld_site->ls_obj_hash.nelems); + nr = (u64)lu_cache_nr; + if (size <= nr) + return; + + lu_site_purge_objects(env, dev->ld_site, + min_t(u64, size - nr, LU_CACHE_NR_MAX_ADJUST), + false); +} + +static struct lu_object *htable_lookup(const struct lu_env *env, + struct lu_device *dev, + struct lu_site_bkt_data *bkt, const struct lu_fid *f, - u64 *version) + struct lu_object_header *new) { + struct lu_site *s = dev->ld_site; struct lu_object_header *h; - struct hlist_node *hnode; - u64 ver = cfs_hash_bd_version_get(bd); - if (*version == ver) +try_again: + rcu_read_lock(); + if (new) + h = rhashtable_lookup_get_insert_fast(&s->ls_obj_hash, + &new->loh_hash, + obj_hash_params); + else + h = rhashtable_lookup(&s->ls_obj_hash, f, obj_hash_params); + if (IS_ERR_OR_NULL(h)) { + /* Not found */ + if (!new) + lprocfs_counter_incr(s->ls_stats, LU_SS_CACHE_MISS); + rcu_read_unlock(); + if (PTR_ERR(h) == -ENOMEM) { + msleep(20); + goto try_again; + } + lu_object_limit(env, dev); + if (PTR_ERR(h) == -E2BIG) + goto try_again; + return ERR_PTR(-ENOENT); + } - *version = ver; - /* cfs_hash_bd_peek_locked is a somehow "internal" function - * of cfs_hash, it doesn't add refcount on object. - */ - hnode = cfs_hash_bd_peek_locked(s->ls_obj_hash, bd, (void *)f); - if (!hnode) { + if (atomic_inc_not_zero(&h->loh_ref)) { + rcu_read_unlock(); + return lu_object_top(h); + } + + spin_lock(&bkt->lsb_waitq.lock); + if (lu_object_is_dying(h) || + test_bit(LU_OBJECT_UNHASHED, &h->loh_flags)) { + spin_unlock(&bkt->lsb_waitq.lock); + rcu_read_unlock(); + if (new) { + /* + * Old object might have already been removed, or will + * be soon. We need to insert our new object, so + * remove the old one just in case it is still there. + */ + rhashtable_remove_fast(&s->ls_obj_hash, &h->loh_hash, + obj_hash_params); + goto try_again; + } lprocfs_counter_incr(s->ls_stats, LU_SS_CACHE_MISS); return ERR_PTR(-ENOENT); } + /* Now protected by spinlock */ + rcu_read_unlock(); - h = container_of(hnode, struct lu_object_header, loh_hash); if (!list_empty(&h->loh_lru)) { - struct lu_site_bkt_data *bkt; - - bkt = &s->ls_bkts[lu_bkt_hash(s, &h->loh_fid)]; - spin_lock(&bkt->lsb_waitq.lock); - /* Might have just been moved to the dispose list, in which - * case LU_OBJECT_PURGING will be set. In that case, - * delete it from the hash table immediately. - * When lu_site_purge_objects() tried, it will find it - * isn't there, which is harmless. - */ - if (test_bit(LU_OBJECT_PURGING, &h->loh_flags)) { - spin_unlock(&bkt->lsb_waitq.lock); - cfs_hash_bd_del_locked(s->ls_obj_hash, bd, hnode); - lprocfs_counter_incr(s->ls_stats, LU_SS_CACHE_MISS); - return ERR_PTR(-ENOENT); - } list_del_init(&h->loh_lru); - spin_unlock(&bkt->lsb_waitq.lock); percpu_counter_dec(&s->ls_lru_len_counter); } - cfs_hash_get(s->ls_obj_hash, hnode); + atomic_inc(&h->loh_ref); + spin_unlock(&bkt->lsb_waitq.lock); lprocfs_counter_incr(s->ls_stats, LU_SS_CACHE_HIT); return lu_object_top(h); } @@ -687,28 +729,37 @@ static struct lu_object *lu_object_find(const struct lu_env *env, } /* - * Limit the lu_object cache to a maximum of lu_cache_nr objects. Because - * the calculation for the number of objects to reclaim is not covered by - * a lock the maximum number of objects is capped by LU_CACHE_MAX_ADJUST. - * This ensures that many concurrent threads will not accidentally purge - * the entire cache. + * Get a 'first' reference to an object that was found while looking through the + * hash table. */ -static void lu_object_limit(const struct lu_env *env, struct lu_device *dev) +struct lu_object *lu_object_get_first(struct lu_object_header *h, + struct lu_device *dev) { - u64 size, nr; + struct lu_site *s = dev->ld_site; + struct lu_object *ret; - if (lu_cache_nr == LU_CACHE_NR_UNLIMITED) - return; + if (IS_ERR_OR_NULL(h) || lu_object_is_dying(h)) + return NULL; - size = cfs_hash_size_get(dev->ld_site->ls_obj_hash); - nr = (u64)lu_cache_nr; - if (size <= nr) - return; + ret = lu_object_locate(h, dev->ld_type); + if (!ret) + return ret; - lu_site_purge_objects(env, dev->ld_site, - min_t(u64, size - nr, LU_CACHE_NR_MAX_ADJUST), - false); + if (!atomic_inc_not_zero(&h->loh_ref)) { + struct lu_site_bkt_data *bkt; + + bkt = &s->ls_bkts[lu_bkt_hash(s, &h->loh_fid)]; + spin_lock(&bkt->lsb_waitq.lock); + if (!lu_object_is_dying(h) && + !test_bit(LU_OBJECT_UNHASHED, &h->loh_flags)) + atomic_inc(&h->loh_ref); + else + ret = NULL; + spin_unlock(&bkt->lsb_waitq.lock); + } + return ret; } +EXPORT_SYMBOL(lu_object_get_first); /** * Core logic of lu_object_find*() functions. @@ -725,10 +776,8 @@ struct lu_object *lu_object_find_at(const struct lu_env *env, struct lu_object *o; struct lu_object *shadow; struct lu_site *s; - struct cfs_hash *hs; - struct cfs_hash_bd bd; struct lu_site_bkt_data *bkt; - u64 version = 0; + struct rhashtable *hs; int rc; /* @@ -750,16 +799,13 @@ struct lu_object *lu_object_find_at(const struct lu_env *env, * */ s = dev->ld_site; - hs = s->ls_obj_hash; + hs = &s->ls_obj_hash; if (unlikely(OBD_FAIL_PRECHECK(OBD_FAIL_OBD_ZERO_NLINK_RACE))) lu_site_purge(env, s, -1); bkt = &s->ls_bkts[lu_bkt_hash(s, f)]; - cfs_hash_bd_get(hs, f, &bd); if (!(conf && conf->loc_flags & LOC_F_NEW)) { - cfs_hash_bd_lock(hs, &bd, 1); - o = htable_lookup(s, &bd, f, &version); - cfs_hash_bd_unlock(hs, &bd, 1); + o = htable_lookup(env, dev, bkt, f, NULL); if (!IS_ERR(o)) { if (likely(lu_object_is_inited(o->lo_header))) @@ -795,29 +841,31 @@ struct lu_object *lu_object_find_at(const struct lu_env *env, CFS_RACE_WAIT(OBD_FAIL_OBD_ZERO_NLINK_RACE); - cfs_hash_bd_lock(hs, &bd, 1); - - if (conf && conf->loc_flags & LOC_F_NEW) - shadow = ERR_PTR(-ENOENT); - else - shadow = htable_lookup(s, &bd, f, &version); + if (conf && conf->loc_flags & LOC_F_NEW) { + int status = rhashtable_insert_fast(hs, &o->lo_header->loh_hash, + obj_hash_params); + if (status) + /* Strange error - go the slow way */ + shadow = htable_lookup(env, dev, bkt, f, o->lo_header); + else + shadow = ERR_PTR(-ENOENT); + } else { + shadow = htable_lookup(env, dev, bkt, f, o->lo_header); + } if (likely(PTR_ERR(shadow) == -ENOENT)) { - cfs_hash_bd_add_locked(hs, &bd, &o->lo_header->loh_hash); - cfs_hash_bd_unlock(hs, &bd, 1); - /* + * The new object has been successfully inserted. + * * This may result in rather complicated operations, including * fld queries, inode loading, etc. */ rc = lu_object_start(env, dev, o, conf); if (rc) { - set_bit(LU_OBJECT_HEARD_BANSHEE, - &o->lo_header->loh_flags); lu_object_put(env, o); return ERR_PTR(rc); } - wake_up_all(&bkt->lsb_waitq); + wake_up(&bkt->lsb_waitq); lu_object_limit(env, dev); @@ -825,10 +873,10 @@ struct lu_object *lu_object_find_at(const struct lu_env *env, } lprocfs_counter_incr(s->ls_stats, LU_SS_CACHE_RACE); - cfs_hash_bd_unlock(hs, &bd, 1); lu_object_free(env, o); if (!(conf && conf->loc_flags & LOC_F_NEW) && + !IS_ERR(shadow) && !lu_object_is_inited(shadow->lo_header)) { wait_event_idle(bkt->lsb_waitq, lu_object_is_inited(shadow->lo_header) || @@ -906,14 +954,9 @@ struct lu_site_print_arg { lu_printer_t lsp_printer; }; -static int -lu_site_obj_print(struct cfs_hash *hs, struct cfs_hash_bd *bd, - struct hlist_node *hnode, void *data) +static void +lu_site_obj_print(struct lu_object_header *h, struct lu_site_print_arg *arg) { - struct lu_site_print_arg *arg = (struct lu_site_print_arg *)data; - struct lu_object_header *h; - - h = hlist_entry(hnode, struct lu_object_header, loh_hash); if (!list_empty(&h->loh_layers)) { const struct lu_object *o; @@ -924,36 +967,45 @@ struct lu_site_print_arg { lu_object_header_print(arg->lsp_env, arg->lsp_cookie, arg->lsp_printer, h); } - return 0; } /** * Print all objects in @s. */ -void lu_site_print(const struct lu_env *env, struct lu_site *s, void *cookie, - lu_printer_t printer) +void lu_site_print(const struct lu_env *env, struct lu_site *s, atomic_t *ref, + int msg_flag, lu_printer_t printer) { struct lu_site_print_arg arg = { .lsp_env = (struct lu_env *)env, - .lsp_cookie = cookie, .lsp_printer = printer, }; + struct rhashtable_iter iter; + struct lu_object_header *h; + LIBCFS_DEBUG_MSG_DATA_DECL(msgdata, msg_flag, NULL); + + if (!s || !atomic_read(ref)) + return; - cfs_hash_for_each(s->ls_obj_hash, lu_site_obj_print, &arg); + arg.lsp_cookie = (void *)&msgdata; + + rhashtable_walk_enter(&s->ls_obj_hash, &iter); + rhashtable_walk_start(&iter); + while ((h = rhashtable_walk_next(&iter)) != NULL) { + if (IS_ERR(h)) + continue; + lu_site_obj_print(h, &arg); + } + rhashtable_walk_stop(&iter); + rhashtable_walk_exit(&iter); } EXPORT_SYMBOL(lu_site_print); /** * Return desired hash table order. */ -static unsigned long lu_htable_order(struct lu_device *top) +static void lu_htable_limits(struct lu_device *top) { - unsigned long bits_max = LU_SITE_BITS_MAX; unsigned long cache_size; - unsigned long bits; - - if (!strcmp(top->ld_type->ldt_name, LUSTRE_VVP_NAME)) - bits_max = LU_SITE_BITS_MAX_CL; /* * Calculate hash table size, assuming that we want reasonable @@ -979,75 +1031,12 @@ static unsigned long lu_htable_order(struct lu_device *top) lu_cache_percent = LU_CACHE_PERCENT_DEFAULT; } cache_size = cache_size / 100 * lu_cache_percent * - (PAGE_SIZE / 1024); - - for (bits = 1; (1 << bits) < cache_size; ++bits) - ; - return clamp_t(typeof(bits), bits, LU_SITE_BITS_MIN, bits_max); -} - -static unsigned int lu_obj_hop_hash(struct cfs_hash *hs, - const void *key, unsigned int mask) -{ - struct lu_fid *fid = (struct lu_fid *)key; - u32 hash; + (PAGE_SIZE / 1024); - hash = fid_flatten32(fid); - hash += (hash >> 4) + (hash << 12); /* mixing oid and seq */ - hash = hash_long(hash, hs->hs_bkt_bits); - - /* give me another random factor */ - hash -= hash_long((unsigned long)hs, fid_oid(fid) % 11 + 3); - - hash <<= hs->hs_cur_bits - hs->hs_bkt_bits; - hash |= (fid_seq(fid) + fid_oid(fid)) & (CFS_HASH_NBKT(hs) - 1); - - return hash & mask; -} - -static void *lu_obj_hop_object(struct hlist_node *hnode) -{ - return hlist_entry(hnode, struct lu_object_header, loh_hash); -} - -static void *lu_obj_hop_key(struct hlist_node *hnode) -{ - struct lu_object_header *h; - - h = hlist_entry(hnode, struct lu_object_header, loh_hash); - return &h->loh_fid; -} - -static int lu_obj_hop_keycmp(const void *key, struct hlist_node *hnode) -{ - struct lu_object_header *h; - - h = hlist_entry(hnode, struct lu_object_header, loh_hash); - return lu_fid_eq(&h->loh_fid, (struct lu_fid *)key); -} - -static void lu_obj_hop_get(struct cfs_hash *hs, struct hlist_node *hnode) -{ - struct lu_object_header *h; - - h = hlist_entry(hnode, struct lu_object_header, loh_hash); - atomic_inc(&h->loh_ref); + lu_cache_nr = clamp_t(typeof(cache_size), cache_size, + LU_CACHE_NR_MIN, LU_CACHE_NR_MAX); } -static void lu_obj_hop_put_locked(struct cfs_hash *hs, struct hlist_node *hnode) -{ - LBUG(); /* we should never called it */ -} - -static struct cfs_hash_ops lu_site_hash_ops = { - .hs_hash = lu_obj_hop_hash, - .hs_key = lu_obj_hop_key, - .hs_keycmp = lu_obj_hop_keycmp, - .hs_object = lu_obj_hop_object, - .hs_get = lu_obj_hop_get, - .hs_put_locked = lu_obj_hop_put_locked, -}; - static void lu_dev_add_linkage(struct lu_site *s, struct lu_device *d) { spin_lock(&s->ls_ld_lock); @@ -1062,35 +1051,19 @@ static void lu_dev_add_linkage(struct lu_site *s, struct lu_device *d) int lu_site_init(struct lu_site *s, struct lu_device *top) { struct lu_site_bkt_data *bkt; - unsigned long bits; - unsigned long i; - char name[16]; + unsigned int i; int rc; memset(s, 0, sizeof(*s)); mutex_init(&s->ls_purge_mutex); + lu_htable_limits(top); rc = percpu_counter_init(&s->ls_lru_len_counter, 0, GFP_NOFS); if (rc) return -ENOMEM; - snprintf(name, sizeof(name), "lu_site_%s", top->ld_type->ldt_name); - for (bits = lu_htable_order(top); bits >= LU_SITE_BITS_MIN; bits--) { - s->ls_obj_hash = cfs_hash_create(name, bits, bits, - bits - LU_SITE_BKT_BITS, - 0, 0, 0, - &lu_site_hash_ops, - CFS_HASH_SPIN_BKTLOCK | - CFS_HASH_NO_ITEMREF | - CFS_HASH_DEPTH | - CFS_HASH_ASSERT_EMPTY | - CFS_HASH_COUNTER); - if (s->ls_obj_hash) - break; - } - - if (!s->ls_obj_hash) { - CERROR("failed to create lu_site hash with bits: %lu\n", bits); + if (rhashtable_init(&s->ls_obj_hash, &obj_hash_params) != 0) { + CERROR("failed to create lu_site hash\n"); return -ENOMEM; } @@ -1101,8 +1074,7 @@ int lu_site_init(struct lu_site *s, struct lu_device *top) s->ls_bkts = kvmalloc_array(s->ls_bkt_cnt, sizeof(*bkt), GFP_KERNEL | __GFP_ZERO); if (!s->ls_bkts) { - cfs_hash_putref(s->ls_obj_hash); - s->ls_obj_hash = NULL; + rhashtable_destroy(&s->ls_obj_hash); s->ls_bkts = NULL; return -ENOMEM; } @@ -1116,9 +1088,8 @@ int lu_site_init(struct lu_site *s, struct lu_device *top) s->ls_stats = lprocfs_alloc_stats(LU_SS_LAST_STAT, 0); if (!s->ls_stats) { kvfree(s->ls_bkts); - cfs_hash_putref(s->ls_obj_hash); - s->ls_obj_hash = NULL; s->ls_bkts = NULL; + rhashtable_destroy(&s->ls_obj_hash); return -ENOMEM; } @@ -1161,13 +1132,12 @@ void lu_site_fini(struct lu_site *s) percpu_counter_destroy(&s->ls_lru_len_counter); - if (s->ls_obj_hash) { - cfs_hash_putref(s->ls_obj_hash); - s->ls_obj_hash = NULL; + if (s->ls_bkts) { + rhashtable_destroy(&s->ls_obj_hash); + kvfree(s->ls_bkts); + s->ls_bkts = NULL; } - kvfree(s->ls_bkts); - if (s->ls_top_dev) { s->ls_top_dev->ld_site = NULL; lu_ref_del(&s->ls_top_dev->ld_reference, "site-top", s); @@ -1323,7 +1293,6 @@ int lu_object_header_init(struct lu_object_header *h) { memset(h, 0, sizeof(*h)); atomic_set(&h->loh_ref, 1); - INIT_HLIST_NODE(&h->loh_hash); INIT_LIST_HEAD(&h->loh_lru); INIT_LIST_HEAD(&h->loh_layers); lu_ref_init(&h->loh_reference); @@ -1338,7 +1307,6 @@ void lu_object_header_fini(struct lu_object_header *h) { LASSERT(list_empty(&h->loh_layers)); LASSERT(list_empty(&h->loh_lru)); - LASSERT(hlist_unhashed(&h->loh_hash)); lu_ref_fini(&h->loh_reference); } EXPORT_SYMBOL(lu_object_header_fini); @@ -1933,7 +1901,7 @@ struct lu_site_stats { static void lu_site_stats_get(const struct lu_site *s, struct lu_site_stats *stats) { - int cnt = cfs_hash_size_get(s->ls_obj_hash); + int cnt = atomic_read(&s->ls_obj_hash.nelems); /* * percpu_counter_sum_positive() won't accept a const pointer * as it does modify the struct by taking a spinlock @@ -2235,16 +2203,23 @@ static u32 ls_stats_read(struct lprocfs_stats *stats, int idx) */ int lu_site_stats_print(const struct lu_site *s, struct seq_file *m) { + const struct bucket_table *tbl; struct lu_site_stats stats; + unsigned int chains; memset(&stats, 0, sizeof(stats)); lu_site_stats_get(s, &stats); - seq_printf(m, "%d/%d %d/%ld %d %d %d %d %d %d %d\n", + rcu_read_lock(); + tbl = rht_dereference_rcu(s->ls_obj_hash.tbl, + &((struct lu_site *)s)->ls_obj_hash); + chains = tbl->size; + rcu_read_unlock(); + seq_printf(m, "%d/%d %d/%u %d %d %d %d %d %d %d\n", stats.lss_busy, stats.lss_total, stats.lss_populated, - CFS_HASH_NHLIST(s->ls_obj_hash), + chains, stats.lss_max_search, ls_stats_read(s->ls_stats, LU_SS_CREATED), ls_stats_read(s->ls_stats, LU_SS_CACHE_HIT), -- 1.8.3.1 From jsimmons at infradead.org Wed Jul 15 20:44:56 2020 From: jsimmons at infradead.org (James Simmons) Date: Wed, 15 Jul 2020 16:44:56 -0400 Subject: [lustre-devel] [PATCH 15/37] lustre: obdclass: use offset instead of cp_linkage In-Reply-To: <1594845918-29027-1-git-send-email-jsimmons@infradead.org> References: <1594845918-29027-1-git-send-email-jsimmons@infradead.org> Message-ID: <1594845918-29027-16-git-send-email-jsimmons@infradead.org> From: Wang Shilong Since we have fixed-size cl_page allocations, we could use offset array to store every slices pointer for cl_page. With this patch, we will reduce cl_page size from 392 bytes to 336 bytes which means we could allocate from 10 to 12 objects. WC-bug-id: https://jira.whamcloud.com/browse/LU-13134 Lustre-commit: 55967f1e5c701 ("LU-13134 obdclass: use offset instead of cp_linkage") Signed-off-by: Wang Shilong Reviewed-on: https://review.whamcloud.com/37428 Reviewed-by: Andreas Dilger Reviewed-by: James Simmons Reviewed-by: Oleg Drokin Signed-off-by: James Simmons --- fs/lustre/include/cl_object.h | 8 +- fs/lustre/obdclass/cl_page.c | 284 ++++++++++++++++++++++++------------------ 2 files changed, 168 insertions(+), 124 deletions(-) diff --git a/fs/lustre/include/cl_object.h b/fs/lustre/include/cl_object.h index a0b9e87..47997f8 100644 --- a/fs/lustre/include/cl_object.h +++ b/fs/lustre/include/cl_object.h @@ -737,8 +737,10 @@ struct cl_page { struct page *cp_vmpage; /** Linkage of pages within group. Pages must be owned */ struct list_head cp_batch; - /** List of slices. Immutable after creation. */ - struct list_head cp_layers; + /** array of slices offset. Immutable after creation. */ + unsigned char cp_layer_offset[3]; + /** current slice index */ + unsigned char cp_layer_count:2; /** * Page state. This field is const to avoid accidental update, it is * modified only internally within cl_page.c. Protected by a VM lock. @@ -781,8 +783,6 @@ struct cl_page_slice { */ struct cl_object *cpl_obj; const struct cl_page_operations *cpl_ops; - /** Linkage into cl_page::cp_layers. Immutable after creation. */ - struct list_head cpl_linkage; }; /** diff --git a/fs/lustre/obdclass/cl_page.c b/fs/lustre/obdclass/cl_page.c index d5be0c5..cced026 100644 --- a/fs/lustre/obdclass/cl_page.c +++ b/fs/lustre/obdclass/cl_page.c @@ -72,22 +72,47 @@ static void cl_page_get_trust(struct cl_page *page) refcount_inc(&page->cp_ref); } +static struct cl_page_slice * +cl_page_slice_get(const struct cl_page *cl_page, int index) +{ + if (index < 0 || index >= cl_page->cp_layer_count) + return NULL; + + /* To get the cp_layer_offset values fit under 256 bytes, we + * use the offset beyond the end of struct cl_page. + */ + return (struct cl_page_slice *)((char *)cl_page + sizeof(*cl_page) + + cl_page->cp_layer_offset[index]); +} + +#define cl_page_slice_for_each(cl_page, slice, i) \ + for (i = 0, slice = cl_page_slice_get(cl_page, 0); \ + i < (cl_page)->cp_layer_count; \ + slice = cl_page_slice_get(cl_page, ++i)) + +#define cl_page_slice_for_each_reverse(cl_page, slice, i) \ + for (i = (cl_page)->cp_layer_count - 1, \ + slice = cl_page_slice_get(cl_page, i); i >= 0; \ + slice = cl_page_slice_get(cl_page, --i)) + /** - * Returns a slice within a page, corresponding to the given layer in the + * Returns a slice within a cl_page, corresponding to the given layer in the * device stack. * * \see cl_lock_at() */ static const struct cl_page_slice * -cl_page_at_trusted(const struct cl_page *page, +cl_page_at_trusted(const struct cl_page *cl_page, const struct lu_device_type *dtype) { const struct cl_page_slice *slice; + int i; - list_for_each_entry(slice, &page->cp_layers, cpl_linkage) { + cl_page_slice_for_each(cl_page, slice, i) { if (slice->cpl_obj->co_lu.lo_dev->ld_type == dtype) return slice; } + return NULL; } @@ -104,28 +129,28 @@ static void __cl_page_free(struct cl_page *cl_page, unsigned short bufsize) } } -static void cl_page_free(const struct lu_env *env, struct cl_page *page, +static void cl_page_free(const struct lu_env *env, struct cl_page *cl_page, struct pagevec *pvec) { - struct cl_object *obj = page->cp_obj; - struct cl_page_slice *slice; + struct cl_object *obj = cl_page->cp_obj; unsigned short bufsize = cl_object_header(obj)->coh_page_bufsize; + struct cl_page_slice *slice; + int i; - PASSERT(env, page, list_empty(&page->cp_batch)); - PASSERT(env, page, !page->cp_owner); - PASSERT(env, page, page->cp_state == CPS_FREEING); + PASSERT(env, cl_page, list_empty(&cl_page->cp_batch)); + PASSERT(env, cl_page, !cl_page->cp_owner); + PASSERT(env, cl_page, cl_page->cp_state == CPS_FREEING); - while ((slice = list_first_entry_or_null(&page->cp_layers, - struct cl_page_slice, - cpl_linkage)) != NULL) { - list_del_init(page->cp_layers.next); + cl_page_slice_for_each(cl_page, slice, i) { if (unlikely(slice->cpl_ops->cpo_fini)) slice->cpl_ops->cpo_fini(env, slice, pvec); } - lu_object_ref_del_at(&obj->co_lu, &page->cp_obj_ref, "cl_page", page); + cl_page->cp_layer_count = 0; + lu_object_ref_del_at(&obj->co_lu, &cl_page->cp_obj_ref, + "cl_page", cl_page); cl_object_put(env, obj); - lu_ref_fini(&page->cp_reference); - __cl_page_free(page, bufsize); + lu_ref_fini(&cl_page->cp_reference); + __cl_page_free(cl_page, bufsize); } /** @@ -212,7 +237,6 @@ struct cl_page *cl_page_alloc(const struct lu_env *env, page->cp_vmpage = vmpage; cl_page_state_set_trust(page, CPS_CACHED); page->cp_type = type; - INIT_LIST_HEAD(&page->cp_layers); INIT_LIST_HEAD(&page->cp_batch); lu_ref_init(&page->cp_reference); cl_object_for_each(o2, o) { @@ -455,22 +479,23 @@ static void cl_page_owner_set(struct cl_page *page) } void __cl_page_disown(const struct lu_env *env, - struct cl_io *io, struct cl_page *pg) + struct cl_io *io, struct cl_page *cl_page) { const struct cl_page_slice *slice; enum cl_page_state state; + int i; - state = pg->cp_state; - cl_page_owner_clear(pg); + state = cl_page->cp_state; + cl_page_owner_clear(cl_page); if (state == CPS_OWNED) - cl_page_state_set(env, pg, CPS_CACHED); + cl_page_state_set(env, cl_page, CPS_CACHED); /* * Completion call-backs are executed in the bottom-up order, so that * uppermost layer (llite), responsible for VFS/VM interaction runs * last and can release locks safely. */ - list_for_each_entry_reverse(slice, &pg->cp_layers, cpl_linkage) { + cl_page_slice_for_each_reverse(cl_page, slice, i) { if (slice->cpl_ops->cpo_disown) (*slice->cpl_ops->cpo_disown)(env, slice, io); } @@ -494,12 +519,12 @@ int cl_page_is_owned(const struct cl_page *pg, const struct cl_io *io) * Waits until page is in cl_page_state::CPS_CACHED state, and then switch it * into cl_page_state::CPS_OWNED state. * - * \pre !cl_page_is_owned(pg, io) - * \post result == 0 iff cl_page_is_owned(pg, io) + * \pre !cl_page_is_owned(cl_page, io) + * \post result == 0 iff cl_page_is_owned(cl_page, io) * * Return: 0 success * - * -ve failure, e.g., page was destroyed (and landed in + * -ve failure, e.g., cl_page was destroyed (and landed in * cl_page_state::CPS_FREEING instead of * cl_page_state::CPS_CACHED). or, page was owned by * another thread, or in IO. @@ -510,19 +535,20 @@ int cl_page_is_owned(const struct cl_page *pg, const struct cl_io *io) * \see cl_page_own */ static int __cl_page_own(const struct lu_env *env, struct cl_io *io, - struct cl_page *pg, int nonblock) + struct cl_page *cl_page, int nonblock) { const struct cl_page_slice *slice; int result = 0; + int i; io = cl_io_top(io); - if (pg->cp_state == CPS_FREEING) { + if (cl_page->cp_state == CPS_FREEING) { result = -ENOENT; goto out; } - list_for_each_entry(slice, &pg->cp_layers, cpl_linkage) { + cl_page_slice_for_each(cl_page, slice, i) { if (slice->cpl_ops->cpo_own) result = (*slice->cpl_ops->cpo_own)(env, slice, io, nonblock); @@ -533,13 +559,13 @@ static int __cl_page_own(const struct lu_env *env, struct cl_io *io, result = 0; if (result == 0) { - PASSERT(env, pg, !pg->cp_owner); - pg->cp_owner = cl_io_top(io); - cl_page_owner_set(pg); - if (pg->cp_state != CPS_FREEING) { - cl_page_state_set(env, pg, CPS_OWNED); + PASSERT(env, cl_page, !cl_page->cp_owner); + cl_page->cp_owner = cl_io_top(io); + cl_page_owner_set(cl_page); + if (cl_page->cp_state != CPS_FREEING) { + cl_page_state_set(env, cl_page, CPS_OWNED); } else { - __cl_page_disown(env, io, pg); + __cl_page_disown(env, io, cl_page); result = -ENOENT; } } @@ -575,51 +601,53 @@ int cl_page_own_try(const struct lu_env *env, struct cl_io *io, * * Called when page is already locked by the hosting VM. * - * \pre !cl_page_is_owned(pg, io) - * \post cl_page_is_owned(pg, io) + * \pre !cl_page_is_owned(cl_page, io) + * \post cl_page_is_owned(cl_page, io) * * \see cl_page_operations::cpo_assume() */ void cl_page_assume(const struct lu_env *env, - struct cl_io *io, struct cl_page *pg) + struct cl_io *io, struct cl_page *cl_page) { const struct cl_page_slice *slice; + int i; io = cl_io_top(io); - list_for_each_entry(slice, &pg->cp_layers, cpl_linkage) { + cl_page_slice_for_each(cl_page, slice, i) { if (slice->cpl_ops->cpo_assume) (*slice->cpl_ops->cpo_assume)(env, slice, io); } - PASSERT(env, pg, !pg->cp_owner); - pg->cp_owner = cl_io_top(io); - cl_page_owner_set(pg); - cl_page_state_set(env, pg, CPS_OWNED); + PASSERT(env, cl_page, !cl_page->cp_owner); + cl_page->cp_owner = cl_io_top(io); + cl_page_owner_set(cl_page); + cl_page_state_set(env, cl_page, CPS_OWNED); } EXPORT_SYMBOL(cl_page_assume); /** * Releases page ownership without unlocking the page. * - * Moves page into cl_page_state::CPS_CACHED without releasing a lock on the - * underlying VM page (as VM is supposed to do this itself). + * Moves cl_page into cl_page_state::CPS_CACHED without releasing a lock + * on the underlying VM page (as VM is supposed to do this itself). * - * \pre cl_page_is_owned(pg, io) - * \post !cl_page_is_owned(pg, io) + * \pre cl_page_is_owned(cl_page, io) + * \post !cl_page_is_owned(cl_page, io) * * \see cl_page_assume() */ void cl_page_unassume(const struct lu_env *env, - struct cl_io *io, struct cl_page *pg) + struct cl_io *io, struct cl_page *cl_page) { const struct cl_page_slice *slice; + int i; io = cl_io_top(io); - cl_page_owner_clear(pg); - cl_page_state_set(env, pg, CPS_CACHED); + cl_page_owner_clear(cl_page); + cl_page_state_set(env, cl_page, CPS_CACHED); - list_for_each_entry_reverse(slice, &pg->cp_layers, cpl_linkage) { + cl_page_slice_for_each_reverse(cl_page, slice, i) { if (slice->cpl_ops->cpo_unassume) (*slice->cpl_ops->cpo_unassume)(env, slice, io); } @@ -646,21 +674,22 @@ void cl_page_disown(const struct lu_env *env, EXPORT_SYMBOL(cl_page_disown); /** - * Called when page is to be removed from the object, e.g., as a result of - * truncate. + * Called when cl_page is to be removed from the object, e.g., + * as a result of truncate. * * Calls cl_page_operations::cpo_discard() top-to-bottom. * - * \pre cl_page_is_owned(pg, io) + * \pre cl_page_is_owned(cl_page, io) * * \see cl_page_operations::cpo_discard() */ void cl_page_discard(const struct lu_env *env, - struct cl_io *io, struct cl_page *pg) + struct cl_io *io, struct cl_page *cl_page) { const struct cl_page_slice *slice; + int i; - list_for_each_entry(slice, &pg->cp_layers, cpl_linkage) { + cl_page_slice_for_each(cl_page, slice, i) { if (slice->cpl_ops->cpo_discard) (*slice->cpl_ops->cpo_discard)(env, slice, io); } @@ -669,22 +698,24 @@ void cl_page_discard(const struct lu_env *env, /** * Version of cl_page_delete() that can be called for not fully constructed - * pages, e.g,. in a error handling cl_page_find()->__cl_page_delete() + * cl_pages, e.g,. in a error handling cl_page_find()->__cl_page_delete() * path. Doesn't check page invariant. */ -static void __cl_page_delete(const struct lu_env *env, struct cl_page *pg) +static void __cl_page_delete(const struct lu_env *env, + struct cl_page *cl_page) { const struct cl_page_slice *slice; + int i; - PASSERT(env, pg, pg->cp_state != CPS_FREEING); + PASSERT(env, cl_page, cl_page->cp_state != CPS_FREEING); /* - * Sever all ways to obtain new pointers to @pg. + * Sever all ways to obtain new pointers to @cl_page. */ - cl_page_owner_clear(pg); - __cl_page_state_set(env, pg, CPS_FREEING); + cl_page_owner_clear(cl_page); + __cl_page_state_set(env, cl_page, CPS_FREEING); - list_for_each_entry_reverse(slice, &pg->cp_layers, cpl_linkage) { + cl_page_slice_for_each_reverse(cl_page, slice, i) { if (slice->cpl_ops->cpo_delete) (*slice->cpl_ops->cpo_delete)(env, slice); } @@ -729,11 +760,13 @@ void cl_page_delete(const struct lu_env *env, struct cl_page *pg) * * \see cl_page_operations::cpo_export() */ -void cl_page_export(const struct lu_env *env, struct cl_page *pg, int uptodate) +void cl_page_export(const struct lu_env *env, struct cl_page *cl_page, + int uptodate) { const struct cl_page_slice *slice; + int i; - list_for_each_entry(slice, &pg->cp_layers, cpl_linkage) { + cl_page_slice_for_each(cl_page, slice, i) { if (slice->cpl_ops->cpo_export) (*slice->cpl_ops->cpo_export)(env, slice, uptodate); } @@ -741,34 +774,36 @@ void cl_page_export(const struct lu_env *env, struct cl_page *pg, int uptodate) EXPORT_SYMBOL(cl_page_export); /** - * Returns true, if @pg is VM locked in a suitable sense by the calling + * Returns true, if @cl_page is VM locked in a suitable sense by the calling * thread. */ -int cl_page_is_vmlocked(const struct lu_env *env, const struct cl_page *pg) +int cl_page_is_vmlocked(const struct lu_env *env, + const struct cl_page *cl_page) { const struct cl_page_slice *slice; int result; - slice = list_first_entry(&pg->cp_layers, - const struct cl_page_slice, cpl_linkage); - PASSERT(env, pg, slice->cpl_ops->cpo_is_vmlocked); + slice = cl_page_slice_get(cl_page, 0); + PASSERT(env, cl_page, slice->cpl_ops->cpo_is_vmlocked); /* * Call ->cpo_is_vmlocked() directly instead of going through * CL_PAGE_INVOKE(), because cl_page_is_vmlocked() is used by * cl_page_invariant(). */ result = slice->cpl_ops->cpo_is_vmlocked(env, slice); - PASSERT(env, pg, result == -EBUSY || result == -ENODATA); + PASSERT(env, cl_page, result == -EBUSY || result == -ENODATA); + return result == -EBUSY; } EXPORT_SYMBOL(cl_page_is_vmlocked); -void cl_page_touch(const struct lu_env *env, const struct cl_page *pg, - size_t to) +void cl_page_touch(const struct lu_env *env, + const struct cl_page *cl_page, size_t to) { const struct cl_page_slice *slice; + int i; - list_for_each_entry(slice, &pg->cp_layers, cpl_linkage) { + cl_page_slice_for_each(cl_page, slice, i) { if (slice->cpl_ops->cpo_page_touch) (*slice->cpl_ops->cpo_page_touch)(env, slice, to); } @@ -799,20 +834,21 @@ static void cl_page_io_start(const struct lu_env *env, * transfer now. */ int cl_page_prep(const struct lu_env *env, struct cl_io *io, - struct cl_page *pg, enum cl_req_type crt) + struct cl_page *cl_page, enum cl_req_type crt) { const struct cl_page_slice *slice; int result = 0; + int i; /* - * XXX this has to be called bottom-to-top, so that llite can set up + * this has to be called bottom-to-top, so that llite can set up * PG_writeback without risking other layers deciding to skip this * page. */ if (crt >= CRT_NR) return -EINVAL; - list_for_each_entry(slice, &pg->cp_layers, cpl_linkage) { + cl_page_slice_for_each(cl_page, slice, i) { if (slice->cpl_ops->cpo_own) result = (*slice->cpl_ops->io[crt].cpo_prep)(env, slice, io); @@ -822,10 +858,10 @@ int cl_page_prep(const struct lu_env *env, struct cl_io *io, if (result >= 0) { result = 0; - cl_page_io_start(env, pg, crt); + cl_page_io_start(env, cl_page, crt); } - CL_PAGE_HEADER(D_TRACE, env, pg, "%d %d\n", crt, result); + CL_PAGE_HEADER(D_TRACE, env, cl_page, "%d %d\n", crt, result); return result; } EXPORT_SYMBOL(cl_page_prep); @@ -840,35 +876,36 @@ int cl_page_prep(const struct lu_env *env, struct cl_io *io, * uppermost layer (llite), responsible for the VFS/VM interaction runs last * and can release locks safely. * - * \pre pg->cp_state == CPS_PAGEIN || pg->cp_state == CPS_PAGEOUT - * \post pg->cp_state == CPS_CACHED + * \pre cl_page->cp_state == CPS_PAGEIN || cl_page->cp_state == CPS_PAGEOUT + * \post cl_page->cp_state == CPS_CACHED * * \see cl_page_operations::cpo_completion() */ void cl_page_completion(const struct lu_env *env, - struct cl_page *pg, enum cl_req_type crt, int ioret) + struct cl_page *cl_page, enum cl_req_type crt, + int ioret) { - struct cl_sync_io *anchor = pg->cp_sync_io; + struct cl_sync_io *anchor = cl_page->cp_sync_io; const struct cl_page_slice *slice; + int i; - PASSERT(env, pg, crt < CRT_NR); - PASSERT(env, pg, pg->cp_state == cl_req_type_state(crt)); - - CL_PAGE_HEADER(D_TRACE, env, pg, "%d %d\n", crt, ioret); + PASSERT(env, cl_page, crt < CRT_NR); + PASSERT(env, cl_page, cl_page->cp_state == cl_req_type_state(crt)); - cl_page_state_set(env, pg, CPS_CACHED); + CL_PAGE_HEADER(D_TRACE, env, cl_page, "%d %d\n", crt, ioret); + cl_page_state_set(env, cl_page, CPS_CACHED); if (crt >= CRT_NR) return; - list_for_each_entry_reverse(slice, &pg->cp_layers, cpl_linkage) { + cl_page_slice_for_each_reverse(cl_page, slice, i) { if (slice->cpl_ops->io[crt].cpo_completion) (*slice->cpl_ops->io[crt].cpo_completion)(env, slice, ioret); } if (anchor) { - LASSERT(pg->cp_sync_io == anchor); - pg->cp_sync_io = NULL; + LASSERT(cl_page->cp_sync_io == anchor); + cl_page->cp_sync_io = NULL; cl_sync_io_note(env, anchor, ioret); } } @@ -878,53 +915,56 @@ void cl_page_completion(const struct lu_env *env, * Notify layers that transfer formation engine decided to yank this page from * the cache and to make it a part of a transfer. * - * \pre pg->cp_state == CPS_CACHED - * \post pg->cp_state == CPS_PAGEIN || pg->cp_state == CPS_PAGEOUT + * \pre cl_page->cp_state == CPS_CACHED + * \post cl_page->cp_state == CPS_PAGEIN || cl_page->cp_state == CPS_PAGEOUT * * \see cl_page_operations::cpo_make_ready() */ -int cl_page_make_ready(const struct lu_env *env, struct cl_page *pg, +int cl_page_make_ready(const struct lu_env *env, struct cl_page *cl_page, enum cl_req_type crt) { - const struct cl_page_slice *sli; + const struct cl_page_slice *slice; int result = 0; + int i; if (crt >= CRT_NR) return -EINVAL; - list_for_each_entry(sli, &pg->cp_layers, cpl_linkage) { - if (sli->cpl_ops->io[crt].cpo_make_ready) - result = (*sli->cpl_ops->io[crt].cpo_make_ready)(env, - sli); + cl_page_slice_for_each(cl_page, slice, i) { + if (slice->cpl_ops->io[crt].cpo_make_ready) + result = (*slice->cpl_ops->io[crt].cpo_make_ready)(env, + slice); if (result != 0) break; } if (result >= 0) { - PASSERT(env, pg, pg->cp_state == CPS_CACHED); - cl_page_io_start(env, pg, crt); + PASSERT(env, cl_page, cl_page->cp_state == CPS_CACHED); + cl_page_io_start(env, cl_page, crt); result = 0; } - CL_PAGE_HEADER(D_TRACE, env, pg, "%d %d\n", crt, result); + CL_PAGE_HEADER(D_TRACE, env, cl_page, "%d %d\n", crt, result); + return result; } EXPORT_SYMBOL(cl_page_make_ready); /** - * Called if a pge is being written back by kernel's intention. + * Called if a page is being written back by kernel's intention. * - * \pre cl_page_is_owned(pg, io) - * \post ergo(result == 0, pg->cp_state == CPS_PAGEOUT) + * \pre cl_page_is_owned(cl_page, io) + * \post ergo(result == 0, cl_page->cp_state == CPS_PAGEOUT) * * \see cl_page_operations::cpo_flush() */ int cl_page_flush(const struct lu_env *env, struct cl_io *io, - struct cl_page *pg) + struct cl_page *cl_page) { const struct cl_page_slice *slice; int result = 0; + int i; - list_for_each_entry(slice, &pg->cp_layers, cpl_linkage) { + cl_page_slice_for_each(cl_page, slice, i) { if (slice->cpl_ops->cpo_flush) result = (*slice->cpl_ops->cpo_flush)(env, slice, io); if (result != 0) @@ -933,7 +973,7 @@ int cl_page_flush(const struct lu_env *env, struct cl_io *io, if (result > 0) result = 0; - CL_PAGE_HEADER(D_TRACE, env, pg, "%d\n", result); + CL_PAGE_HEADER(D_TRACE, env, cl_page, "%d\n", result); return result; } EXPORT_SYMBOL(cl_page_flush); @@ -943,14 +983,14 @@ int cl_page_flush(const struct lu_env *env, struct cl_io *io, * * \see cl_page_operations::cpo_clip() */ -void cl_page_clip(const struct lu_env *env, struct cl_page *pg, +void cl_page_clip(const struct lu_env *env, struct cl_page *cl_page, int from, int to) { const struct cl_page_slice *slice; + int i; - CL_PAGE_HEADER(D_TRACE, env, pg, "%d %d\n", from, to); - - list_for_each_entry(slice, &pg->cp_layers, cpl_linkage) { + CL_PAGE_HEADER(D_TRACE, env, cl_page, "%d %d\n", from, to); + cl_page_slice_for_each(cl_page, slice, i) { if (slice->cpl_ops->cpo_clip) (*slice->cpl_ops->cpo_clip)(env, slice, from, to); } @@ -972,24 +1012,24 @@ void cl_page_header_print(const struct lu_env *env, void *cookie, EXPORT_SYMBOL(cl_page_header_print); /** - * Prints human readable representation of @pg to the @f. + * Prints human readable representation of @cl_page to the @f. */ void cl_page_print(const struct lu_env *env, void *cookie, - lu_printer_t printer, const struct cl_page *pg) + lu_printer_t printer, const struct cl_page *cl_page) { const struct cl_page_slice *slice; int result = 0; + int i; - cl_page_header_print(env, cookie, printer, pg); - - list_for_each_entry(slice, &pg->cp_layers, cpl_linkage) { + cl_page_header_print(env, cookie, printer, cl_page); + cl_page_slice_for_each(cl_page, slice, i) { if (slice->cpl_ops->cpo_print) result = (*slice->cpl_ops->cpo_print)(env, slice, cookie, printer); if (result != 0) break; } - (*printer)(env, cookie, "end page@%p\n", pg); + (*printer)(env, cookie, "end page@%p\n", cl_page); } EXPORT_SYMBOL(cl_page_print); @@ -1032,14 +1072,18 @@ size_t cl_page_size(const struct cl_object *obj) * * \see cl_lock_slice_add(), cl_req_slice_add(), cl_io_slice_add() */ -void cl_page_slice_add(struct cl_page *page, struct cl_page_slice *slice, +void cl_page_slice_add(struct cl_page *cl_page, struct cl_page_slice *slice, struct cl_object *obj, const struct cl_page_operations *ops) { - list_add_tail(&slice->cpl_linkage, &page->cp_layers); + unsigned int offset = (char *)slice - + ((char *)cl_page + sizeof(*cl_page)); + + LASSERT(offset < (1 << sizeof(cl_page->cp_layer_offset[0]) * 8)); + cl_page->cp_layer_offset[cl_page->cp_layer_count++] = offset; slice->cpl_obj = obj; slice->cpl_ops = ops; - slice->cpl_page = page; + slice->cpl_page = cl_page; } EXPORT_SYMBOL(cl_page_slice_add); -- 1.8.3.1 From jsimmons at infradead.org Wed Jul 15 20:44:55 2020 From: jsimmons at infradead.org (James Simmons) Date: Wed, 15 Jul 2020 16:44:55 -0400 Subject: [lustre-devel] [PATCH 14/37] lustre: remove some "#ifdef CONFIG*" from .c files. In-Reply-To: <1594845918-29027-1-git-send-email-jsimmons@infradead.org> References: <1594845918-29027-1-git-send-email-jsimmons@infradead.org> Message-ID: <1594845918-29027-15-git-send-email-jsimmons@infradead.org> From: Mr NeilBrown It is Linux policy to avoid #ifdef in C files where convenient - .h files are OK. This patch defines a few inline functions which differ depending on CONFIG_LUSTRE_FS_POSIX_ACL, and removes some #ifdefs from .c files. WC-bug-id: https://jira.whamcloud.com/browse/LU-9679 Lustre-commit: f37e26964a34f ("LU-9679 lustre: remove some "#ifdef CONFIG*" from .c files.") Signed-off-by: Mr NeilBrown Reviewed-on: https://review.whamcloud.com/39131 Reviewed-by: James Simmons Reviewed-by: Jian Yu Reviewed-by: Oleg Drokin Signed-off-by: James Simmons --- fs/lustre/include/obd.h | 21 ++++++++++++++++++++ fs/lustre/llite/llite_internal.h | 29 +++++++++++++++++++++++++++ fs/lustre/llite/llite_lib.c | 43 +++++++++------------------------------- fs/lustre/mdc/mdc_request.c | 8 +++----- 4 files changed, 62 insertions(+), 39 deletions(-) diff --git a/fs/lustre/include/obd.h b/fs/lustre/include/obd.h index 438f4ca..ad2b2f4 100644 --- a/fs/lustre/include/obd.h +++ b/fs/lustre/include/obd.h @@ -34,6 +34,8 @@ #ifndef __OBD_H #define __OBD_H +#include +#include #include #include #include @@ -930,6 +932,25 @@ struct lustre_md { struct mdt_remote_perm *remote_perm; }; +#ifdef CONFIG_LUSTRE_FS_POSIX_ACL +static inline void lmd_clear_acl(struct lustre_md *md) +{ + if (md->posix_acl) { + posix_acl_release(md->posix_acl); + md->posix_acl = NULL; + } +} + +#define OBD_CONNECT_ACL_FLAGS \ + (OBD_CONNECT_ACL | OBD_CONNECT_UMASK | OBD_CONNECT_LARGE_ACL) +#else +static inline void lmd_clear_acl(struct lustre_md *md) +{ +} + +#define OBD_CONNECT_ACL_FLAGS (0) +#endif + struct md_open_data { struct obd_client_handle *mod_och; struct ptlrpc_request *mod_open_req; diff --git a/fs/lustre/llite/llite_internal.h b/fs/lustre/llite/llite_internal.h index 2556dd8..31c528f 100644 --- a/fs/lustre/llite/llite_internal.h +++ b/fs/lustre/llite/llite_internal.h @@ -350,6 +350,35 @@ static inline void trunc_sem_up_write(struct ll_trunc_sem *sem) wake_up_var(&sem->ll_trunc_readers); } +#ifdef CONFIG_LUSTRE_FS_POSIX_ACL +static inline void lli_clear_acl(struct ll_inode_info *lli) +{ + if (lli->lli_posix_acl) { + posix_acl_release(lli->lli_posix_acl); + lli->lli_posix_acl = NULL; + } +} + +static inline void lli_replace_acl(struct ll_inode_info *lli, + struct lustre_md *md) +{ + spin_lock(&lli->lli_lock); + if (lli->lli_posix_acl) + posix_acl_release(lli->lli_posix_acl); + lli->lli_posix_acl = md->posix_acl; + spin_unlock(&lli->lli_lock); +} +#else +static inline void lli_clear_acl(struct ll_inode_info *lli) +{ +} + +static inline void lli_replace_acl(struct ll_inode_info *lli, + struct lustre_md *md) +{ +} +#endif + static inline u32 ll_layout_version_get(struct ll_inode_info *lli) { u32 gen; diff --git a/fs/lustre/llite/llite_lib.c b/fs/lustre/llite/llite_lib.c index 1a7d805..c62e182 100644 --- a/fs/lustre/llite/llite_lib.c +++ b/fs/lustre/llite/llite_lib.c @@ -265,10 +265,7 @@ static int client_common_fill_super(struct super_block *sb, char *md, char *dt) if (sbi->ll_flags & LL_SBI_LRU_RESIZE) data->ocd_connect_flags |= OBD_CONNECT_LRU_RESIZE; -#ifdef CONFIG_LUSTRE_FS_POSIX_ACL - data->ocd_connect_flags |= OBD_CONNECT_ACL | OBD_CONNECT_UMASK | - OBD_CONNECT_LARGE_ACL; -#endif + data->ocd_connect_flags |= OBD_CONNECT_ACL_FLAGS; data->ocd_cksum_types = obd_cksum_types_supported_client(); @@ -618,13 +615,8 @@ static int client_common_fill_super(struct super_block *sb, char *md, char *dt) ptlrpc_req_finished(request); if (IS_ERR(root)) { -#ifdef CONFIG_LUSTRE_FS_POSIX_ACL - if (lmd.posix_acl) { - posix_acl_release(lmd.posix_acl); - lmd.posix_acl = NULL; - } -#endif - err = -EBADF; + lmd_clear_acl(&lmd); + err = IS_ERR(root) ? PTR_ERR(root) : -EBADF; CERROR("lustre_lite: bad iget4 for root\n"); goto out_root; } @@ -1584,13 +1576,7 @@ void ll_clear_inode(struct inode *inode) ll_xattr_cache_destroy(inode); -#ifdef CONFIG_LUSTRE_FS_POSIX_ACL - forget_all_cached_acls(inode); - if (lli->lli_posix_acl) { - posix_acl_release(lli->lli_posix_acl); - lli->lli_posix_acl = NULL; - } -#endif + lli_clear_acl(lli); lli->lli_inode_magic = LLI_INODE_DEAD; if (S_ISDIR(inode->i_mode)) @@ -2233,15 +2219,9 @@ int ll_update_inode(struct inode *inode, struct lustre_md *md) return rc; } -#ifdef CONFIG_LUSTRE_FS_POSIX_ACL - if (body->mbo_valid & OBD_MD_FLACL) { - spin_lock(&lli->lli_lock); - if (lli->lli_posix_acl) - posix_acl_release(lli->lli_posix_acl); - lli->lli_posix_acl = md->posix_acl; - spin_unlock(&lli->lli_lock); - } -#endif + if (body->mbo_valid & OBD_MD_FLACL) + lli_replace_acl(lli, md); + inode->i_ino = cl_fid_build_ino(&body->mbo_fid1, sbi->ll_flags & LL_SBI_32BIT_API); inode->i_generation = cl_fid_build_gen(&body->mbo_fid1); @@ -2691,13 +2671,8 @@ int ll_prep_inode(struct inode **inode, struct ptlrpc_request *req, sbi->ll_flags & LL_SBI_32BIT_API), &md); if (IS_ERR(*inode)) { -#ifdef CONFIG_LUSTRE_FS_POSIX_ACL - if (md.posix_acl) { - posix_acl_release(md.posix_acl); - md.posix_acl = NULL; - } -#endif - rc = PTR_ERR(*inode); + lmd_clear_acl(&md); + rc = IS_ERR(*inode) ? PTR_ERR(*inode) : -ENOMEM; CERROR("new_inode -fatal: rc %d\n", rc); goto out; } diff --git a/fs/lustre/mdc/mdc_request.c b/fs/lustre/mdc/mdc_request.c index d6d9f43..cacc58b 100644 --- a/fs/lustre/mdc/mdc_request.c +++ b/fs/lustre/mdc/mdc_request.c @@ -675,11 +675,9 @@ static int mdc_get_lustre_md(struct obd_export *exp, } out: - if (rc) { -#ifdef CONFIG_LUSTRE_FS_POSIX_ACL - posix_acl_release(md->posix_acl); -#endif - } + if (rc) + lmd_clear_acl(md); + return rc; } -- 1.8.3.1 From jsimmons at infradead.org Wed Jul 15 20:44:53 2020 From: jsimmons at infradead.org (James Simmons) Date: Wed, 15 Jul 2020 16:44:53 -0400 Subject: [lustre-devel] [PATCH 12/37] lustre: ptlrpc: re-enterable signal_completed_replay() In-Reply-To: <1594845918-29027-1-git-send-email-jsimmons@infradead.org> References: <1594845918-29027-1-git-send-email-jsimmons@infradead.org> Message-ID: <1594845918-29027-13-git-send-email-jsimmons@infradead.org> From: Mikhail Pershin The signal_completed_replay() can meet race conditions while checking imp_replay_inflight counter, so remove assertion and check race conditions instead. Fixes: 8cc7f22847 ("lustre: ptlrpc: limit rate of lock replays") WC-bug-id: https://jira.whamcloud.com/browse/LU-13600 Lustre-commit: 24451f3790503 ("LU-13600 ptlrpc: re-enterable signal_completed_replay()") Signed-off-by: Mikhail Pershin Reviewed-on: https://review.whamcloud.com/39140 Reviewed-by: Andreas Dilger Reviewed-by: Hongchao Zhang Reviewed-by: Oleg Drokin Signed-off-by: James Simmons --- fs/lustre/ptlrpc/import.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/fs/lustre/ptlrpc/import.c b/fs/lustre/ptlrpc/import.c index 7ec3638..1b62b81 100644 --- a/fs/lustre/ptlrpc/import.c +++ b/fs/lustre/ptlrpc/import.c @@ -1407,8 +1407,8 @@ static int signal_completed_replay(struct obd_import *imp) if (unlikely(OBD_FAIL_CHECK(OBD_FAIL_PTLRPC_FINISH_REPLAY))) return 0; - LASSERT(atomic_read(&imp->imp_replay_inflight) == 0); - atomic_inc(&imp->imp_replay_inflight); + if (!atomic_add_unless(&imp->imp_replay_inflight, 1, 1)) + return 0; req = ptlrpc_request_alloc_pack(imp, &RQF_OBD_PING, LUSTRE_OBD_VERSION, OBD_PING); -- 1.8.3.1 From jsimmons at infradead.org Wed Jul 15 20:44:59 2020 From: jsimmons at infradead.org (James Simmons) Date: Wed, 15 Jul 2020 16:44:59 -0400 Subject: [lustre-devel] [PATCH 18/37] lustre: llite: Fix lock ordering in pagevec_dirty In-Reply-To: <1594845918-29027-1-git-send-email-jsimmons@infradead.org> References: <1594845918-29027-1-git-send-email-jsimmons@infradead.org> Message-ID: <1594845918-29027-19-git-send-email-jsimmons@infradead.org> From: Shaun Tancheff In vvp_set_pagevec_dirty lock order between i_pages and lock_page_memcg was inverted with the expectation that no other users would conflict. However in vvp_page_completion_write the call to test_clear_page_writeback does expect to be able to lock_page_memcg then lock i_pages which appears to conflict with the original analysis. The reported case shows as RCU stalls with vvp_set_pagevec_dirty blocked attempting to lock i_pages. Fixes: f8a5fb036ae ("lustre: vvp: dirty pages with pagevec") HPE-bug-id: LUS-8798 WC-bug-id: https://jira.whamcloud.com/browse/LU-13746 Lustre-commit: c4ed9b0fb1013 ("LU-13476 llite: Fix lock ordering in pagevec_dirty") Signed-off-by: Shaun Tancheff Reviewed-on: https://review.whamcloud.com/38317 Reviewed-by: Wang Shilong Reviewed-by: Patrick Farrell Reviewed-by: Oleg Drokin Signed-off-by: James Simmons --- fs/lustre/llite/vvp_io.c | 34 +++++++++++++++++++++------------- 1 file changed, 21 insertions(+), 13 deletions(-) diff --git a/fs/lustre/llite/vvp_io.c b/fs/lustre/llite/vvp_io.c index 8edd3c1..7627431 100644 --- a/fs/lustre/llite/vvp_io.c +++ b/fs/lustre/llite/vvp_io.c @@ -897,19 +897,31 @@ void vvp_set_pagevec_dirty(struct pagevec *pvec) struct page *page = pvec->pages[0]; struct address_space *mapping = page->mapping; unsigned long flags; + unsigned long skip_pages = 0; int count = pagevec_count(pvec); int dirtied = 0; - int i = 0; - - /* From set_page_dirty */ - for (i = 0; i < count; i++) - ClearPageReclaim(pvec->pages[i]); + int i; + BUILD_BUG_ON(PAGEVEC_SIZE > BITS_PER_LONG); LASSERTF(page->mapping, "mapping must be set. page %p, page->private (cl_page) %p\n", page, (void *) page->private); - /* Rest of code derived from __set_page_dirty_nobuffers */ + for (i = 0; i < count; i++) { + page = pvec->pages[i]; + + ClearPageReclaim(page); + + lock_page_memcg(page); + if (TestSetPageDirty(page)) { + /* page is already dirty .. no extra work needed + * set a flag for the i'th page to be skipped + */ + unlock_page_memcg(page); + skip_pages |= (1 << i); + } + } + xa_lock_irqsave(&mapping->i_pages, flags); /* Notes on differences with __set_page_dirty_nobuffers: @@ -920,17 +932,13 @@ void vvp_set_pagevec_dirty(struct pagevec *pvec) * 3. No mapping is impossible. (Race w/truncate mentioned in * dirty_nobuffers should be impossible because we hold the page lock.) * 4. All mappings are the same because i/o is only to one file. - * 5. We invert the lock order on lock_page_memcg(page) and the mapping - * xa_lock, but this is the only function that should use that pair of - * locks and it can't race because Lustre locks pages throughout i/o. */ for (i = 0; i < count; i++) { page = pvec->pages[i]; - lock_page_memcg(page); - if (TestSetPageDirty(page)) { - unlock_page_memcg(page); + /* if the i'th page was unlocked above, skip it here */ + if ((skip_pages >> i) & 1) continue; - } + LASSERTF(page->mapping == mapping, "all pages must have the same mapping. page %p, mapping %p, first mapping %p\n", page, page->mapping, mapping); -- 1.8.3.1 From jsimmons at infradead.org Wed Jul 15 20:44:51 2020 From: jsimmons at infradead.org (James Simmons) Date: Wed, 15 Jul 2020 16:44:51 -0400 Subject: [lustre-devel] [PATCH 10/37] lnet: Fix some out-of-date comments. In-Reply-To: <1594845918-29027-1-git-send-email-jsimmons@infradead.org> References: <1594845918-29027-1-git-send-email-jsimmons@infradead.org> Message-ID: <1594845918-29027-11-git-send-email-jsimmons@infradead.org> From: Mr NeilBrown The structures these comments describe have changed or been removed, but the comments weren't updated at the time. WC-bug-id: https://jira.whamcloud.com/browse/LU-12678 Lustre-commit: 617ad3af720a3 ("LU-12678 lnet: Fix some out-of-date comments.") Signed-off-by: Mr NeilBrown Reviewed-on: https://review.whamcloud.com/39127 Reviewed-by: James Simmons Reviewed-by: Serguei Smirnov Reviewed-by: Oleg Drokin Signed-off-by: James Simmons --- net/lnet/klnds/socklnd/socklnd.h | 12 ++++-------- 1 file changed, 4 insertions(+), 8 deletions(-) diff --git a/net/lnet/klnds/socklnd/socklnd.h b/net/lnet/klnds/socklnd/socklnd.h index 7d49fff..0ac3637 100644 --- a/net/lnet/klnds/socklnd/socklnd.h +++ b/net/lnet/klnds/socklnd/socklnd.h @@ -255,15 +255,13 @@ struct ksock_nal_data { #define SOCKNAL_INIT_DATA 1 #define SOCKNAL_INIT_ALL 2 -/* - * A packet just assembled for transmission is represented by 1 or more - * struct iovec fragments (the first frag contains the portals header), - * followed by 0 or more struct bio_vec fragments. +/* A packet just assembled for transmission is represented by 1 + * struct iovec fragment - the portals header - followed by 0 + * or more struct bio_vec fragments. * * On the receive side, initially 1 struct iovec fragment is posted for * receive (the header). Once the header has been received, the payload is - * received into either struct iovec or struct bio_vec fragments, depending on - * what the header matched or whether the message needs forwarding. + * received into struct bio_vec fragments. */ struct ksock_conn; /* forward ref */ struct ksock_route; /* forward ref */ @@ -296,8 +294,6 @@ struct ksock_tx { /* transmit packet */ #define KSOCK_NOOP_TX_SIZE (offsetof(struct ksock_tx, tx_payload[0])) -/* network zero copy callback descriptor embedded in struct ksock_tx */ - #define SOCKNAL_RX_KSM_HEADER 1 /* reading ksock message header */ #define SOCKNAL_RX_LNET_HEADER 2 /* reading lnet message header */ #define SOCKNAL_RX_PARSE 3 /* Calling lnet_parse() */ -- 1.8.3.1 From jsimmons at infradead.org Wed Jul 15 20:44:58 2020 From: jsimmons at infradead.org (James Simmons) Date: Wed, 15 Jul 2020 16:44:58 -0400 Subject: [lustre-devel] [PATCH 17/37] lustre: osc: re-declare ops_from/to to shrink osc_page In-Reply-To: <1594845918-29027-1-git-send-email-jsimmons@infradead.org> References: <1594845918-29027-1-git-send-email-jsimmons@infradead.org> Message-ID: <1594845918-29027-18-git-send-email-jsimmons@infradead.org> From: Wang Shilong @ops_from and @ops_to is within PAGE_SIZE, use PAGE_SHIFT bits to limit it is fine, on x86_64 platform, this patch will reduce another 8 bytes. Notice, previous @ops_to is exclusive which could be PAGE_SIZE, this patch change it to inclusive which means max value will be PAGE_SIZE - 1, and be careful to calculate its length. After this patch, cl_page size could reduce from 320 to 312 bytes, and we are able to allocate 13 objects using slab pool for 4K page. WC-bug-id: https://jira.whamcloud.com/browse/LU-13134 Lustre-commit: 9821754235e24 ("LU-13134 osc: re-declare ops_from/to to shrink osc_page") Signed-off-by: Wang Shilong Reviewed-on: https://review.whamcloud.com/37487 Reviewed-by: Andreas Dilger Reviewed-by: Neil Brown Reviewed-by: Oleg Drokin Signed-off-by: James Simmons --- fs/lustre/include/lustre_osc.h | 8 ++++---- fs/lustre/osc/osc_cache.c | 5 +++-- fs/lustre/osc/osc_page.c | 21 +++++++++++---------- 3 files changed, 18 insertions(+), 16 deletions(-) diff --git a/fs/lustre/include/lustre_osc.h b/fs/lustre/include/lustre_osc.h index cd08f27..3956ab4 100644 --- a/fs/lustre/include/lustre_osc.h +++ b/fs/lustre/include/lustre_osc.h @@ -507,17 +507,17 @@ struct osc_page { * An offset within page from which next transfer starts. This is used * by cl_page_clip() to submit partial page transfers. */ - int ops_from; + unsigned int ops_from:PAGE_SHIFT, /* - * An offset within page at which next transfer ends. + * An offset within page at which next transfer ends(inclusive). * * \see osc_page::ops_from. */ - int ops_to; + ops_to:PAGE_SHIFT, /* * Boolean, true iff page is under transfer. Used for sanity checking. */ - unsigned ops_transfer_pinned:1, + ops_transfer_pinned:1, /* * in LRU? */ diff --git a/fs/lustre/osc/osc_cache.c b/fs/lustre/osc/osc_cache.c index f811dadb..fe03c0d 100644 --- a/fs/lustre/osc/osc_cache.c +++ b/fs/lustre/osc/osc_cache.c @@ -2395,7 +2395,7 @@ int osc_queue_async_io(const struct lu_env *env, struct cl_io *io, oap->oap_cmd = cmd; oap->oap_page_off = ops->ops_from; - oap->oap_count = ops->ops_to - ops->ops_from; + oap->oap_count = ops->ops_to - ops->ops_from + 1; /* * No need to hold a lock here, * since this page is not in any list yet. @@ -2664,7 +2664,8 @@ int osc_queue_sync_pages(const struct lu_env *env, const struct cl_io *io, ++page_count; mppr <<= (page_count > mppr); - if (unlikely(opg->ops_from > 0 || opg->ops_to < PAGE_SIZE)) + if (unlikely(opg->ops_from > 0 || + opg->ops_to < PAGE_SIZE - 1)) can_merge = false; } diff --git a/fs/lustre/osc/osc_page.c b/fs/lustre/osc/osc_page.c index 2856f30..bb605af 100644 --- a/fs/lustre/osc/osc_page.c +++ b/fs/lustre/osc/osc_page.c @@ -211,7 +211,8 @@ static void osc_page_clip(const struct lu_env *env, struct osc_async_page *oap = &opg->ops_oap; opg->ops_from = from; - opg->ops_to = to; + /* argument @to is exclusive, but @ops_to is inclusive */ + opg->ops_to = to - 1; spin_lock(&oap->oap_lock); oap->oap_async_flags |= ASYNC_COUNT_STABLE; spin_unlock(&oap->oap_lock); @@ -246,28 +247,28 @@ static void osc_page_touch(const struct lu_env *env, }; int osc_page_init(const struct lu_env *env, struct cl_object *obj, - struct cl_page *page, pgoff_t index) + struct cl_page *cl_page, pgoff_t index) { struct osc_object *osc = cl2osc(obj); - struct osc_page *opg = cl_object_page_slice(obj, page); + struct osc_page *opg = cl_object_page_slice(obj, cl_page); struct osc_io *oio = osc_env_io(env); int result; opg->ops_from = 0; - opg->ops_to = PAGE_SIZE; + opg->ops_to = PAGE_SIZE - 1; INIT_LIST_HEAD(&opg->ops_lru); - result = osc_prep_async_page(osc, opg, page->cp_vmpage, + result = osc_prep_async_page(osc, opg, cl_page->cp_vmpage, cl_offset(obj, index)); if (result != 0) return result; opg->ops_srvlock = osc_io_srvlock(oio); - cl_page_slice_add(page, &opg->ops_cl, obj, &osc_page_ops); - page->cp_osc_index = index; + cl_page_slice_add(cl_page, &opg->ops_cl, obj, &osc_page_ops); + cl_page->cp_osc_index = index; - /* reserve an LRU space for this page */ - if (page->cp_type == CPT_CACHEABLE) { + /* reserve an LRU space for this cl_page */ + if (cl_page->cp_type == CPT_CACHEABLE) { result = osc_lru_alloc(env, osc_cli(osc), opg); if (result == 0) { result = radix_tree_preload(GFP_KERNEL); @@ -308,7 +309,7 @@ void osc_page_submit(const struct lu_env *env, struct osc_page *opg, oap->oap_cmd = crt == CRT_WRITE ? OBD_BRW_WRITE : OBD_BRW_READ; oap->oap_page_off = opg->ops_from; - oap->oap_count = opg->ops_to - opg->ops_from; + oap->oap_count = opg->ops_to - opg->ops_from + 1; oap->oap_brw_flags = OBD_BRW_SYNC | brw_flags; if (oio->oi_cap_sys_resource) { -- 1.8.3.1 From jsimmons at infradead.org Wed Jul 15 20:45:00 2020 From: jsimmons at infradead.org (James Simmons) Date: Wed, 15 Jul 2020 16:45:00 -0400 Subject: [lustre-devel] [PATCH 19/37] lustre: misc: quiet compiler warning on armv7l In-Reply-To: <1594845918-29027-1-git-send-email-jsimmons@infradead.org> References: <1594845918-29027-1-git-send-email-jsimmons@infradead.org> Message-ID: <1594845918-29027-20-git-send-email-jsimmons@infradead.org> From: Andreas Dilger Avoid overflow in lu_prandom_u64_max(). Quiet printk() warning for mismatched type of size_t variables by using %z modifier for those variables. Fixes: bc2e21c54ba2 ("lustre: obdclass: generate random u64 max correctly") WC-bug-id: https://jira.whamcloud.com/browse/LU-13673 Lustre-commit: 57bb302461383 ("LU-13673 misc: quiet compiler warning on armv7l") Signed-off-by: Andreas Dilger Reviewed-on: https://review.whamcloud.com/38927 Reviewed-by: James Simmons Reviewed-by: Lai Siyao Reviewed-by: Yang Sheng Reviewed-by: Oleg Drokin Signed-off-by: James Simmons --- fs/lustre/llite/vvp_io.c | 4 ++-- fs/lustre/obdclass/lu_tgt_descs.c | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/fs/lustre/llite/vvp_io.c b/fs/lustre/llite/vvp_io.c index 7627431..c3fb03a 100644 --- a/fs/lustre/llite/vvp_io.c +++ b/fs/lustre/llite/vvp_io.c @@ -791,7 +791,7 @@ static int vvp_io_read_start(const struct lu_env *env, goto out; LU_OBJECT_HEADER(D_INODE, env, &obj->co_lu, - "Read ino %lu, %lu bytes, offset %lld, size %llu\n", + "Read ino %lu, %zu bytes, offset %lld, size %llu\n", inode->i_ino, cnt, pos, i_size_read(inode)); /* turn off the kernel's read-ahead */ @@ -1197,7 +1197,7 @@ static int vvp_io_write_start(const struct lu_env *env, } if (vio->vui_iocb->ki_pos != (pos + io->ci_nob - nob)) { CDEBUG(D_VFSTRACE, - "%s: write position mismatch: ki_pos %lld vs. pos %lld, written %ld, commit %ld rc %ld\n", + "%s: write position mismatch: ki_pos %lld vs. pos %lld, written %zd, commit %zd rc %zd\n", file_dentry(file)->d_name.name, vio->vui_iocb->ki_pos, pos + io->ci_nob - nob, written, io->ci_nob - nob, result); diff --git a/fs/lustre/obdclass/lu_tgt_descs.c b/fs/lustre/obdclass/lu_tgt_descs.c index db5a93b..469c935 100644 --- a/fs/lustre/obdclass/lu_tgt_descs.c +++ b/fs/lustre/obdclass/lu_tgt_descs.c @@ -62,7 +62,7 @@ u64 lu_prandom_u64_max(u64 ep_ro) * 32 bits (truncated to the upper limit, if needed) */ if (ep_ro > 0xffffffffULL) - rand = prandom_u32_max((u32)(ep_ro >> 32)) << 32; + rand = (u64)prandom_u32_max((u32)(ep_ro >> 32)) << 32; if (rand == (ep_ro & 0xffffffff00000000ULL)) rand |= prandom_u32_max((u32)ep_ro); -- 1.8.3.1 From jsimmons at infradead.org Wed Jul 15 20:45:03 2020 From: jsimmons at infradead.org (James Simmons) Date: Wed, 15 Jul 2020 16:45:03 -0400 Subject: [lustre-devel] [PATCH 22/37] lnet: o2iblnd: wait properly for fps->increasing. In-Reply-To: <1594845918-29027-1-git-send-email-jsimmons@infradead.org> References: <1594845918-29027-1-git-send-email-jsimmons@infradead.org> Message-ID: <1594845918-29027-23-git-send-email-jsimmons@infradead.org> From: Mr NeilBrown If we need to allocate a new fmr_pool and another thread is currently allocating one, we call schedule() and then try again. This can spin, consuming a CPU and wasting power. Instead, use wait_var_event() and wake_up_var() to wait for fps_increasing to be cleared. WC-bug-id: https://jira.whamcloud.com/browse/LU-12678 Lustre-commit: 530eca31556f7 ("LU-12768 o2iblnd: wait properly for fps->increasing.") Signed-off-by: Mr NeilBrown Reviewed-on: https://review.whamcloud.com/39124 Reviewed-by: James Simmons Reviewed-by: Shaun Tancheff Reviewed-by: Chris Horn Reviewed-by: Oleg Drokin Signed-off-by: James Simmons --- net/lnet/klnds/o2iblnd/o2iblnd.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/net/lnet/klnds/o2iblnd/o2iblnd.c b/net/lnet/klnds/o2iblnd/o2iblnd.c index e2e94b7..6c7659c 100644 --- a/net/lnet/klnds/o2iblnd/o2iblnd.c +++ b/net/lnet/klnds/o2iblnd/o2iblnd.c @@ -1750,7 +1750,7 @@ int kiblnd_fmr_pool_map(struct kib_fmr_poolset *fps, struct kib_tx *tx, if (fps->fps_increasing) { spin_unlock(&fps->fps_lock); CDEBUG(D_NET, "Another thread is allocating new FMR pool, waiting for her to complete\n"); - schedule(); + wait_var_event(fps, !fps->fps_increasing); goto again; } @@ -1767,6 +1767,7 @@ int kiblnd_fmr_pool_map(struct kib_fmr_poolset *fps, struct kib_tx *tx, rc = kiblnd_create_fmr_pool(fps, &fpo); spin_lock(&fps->fps_lock); fps->fps_increasing = 0; + wake_up_var(fps); if (!rc) { fps->fps_version++; list_add_tail(&fpo->fpo_list, &fps->fps_pool_list); -- 1.8.3.1 From jsimmons at infradead.org Wed Jul 15 20:45:07 2020 From: jsimmons at infradead.org (James Simmons) Date: Wed, 15 Jul 2020 16:45:07 -0400 Subject: [lustre-devel] [PATCH 26/37] lnet: socklnd: use list_for_each_entry_safe() In-Reply-To: <1594845918-29027-1-git-send-email-jsimmons@infradead.org> References: <1594845918-29027-1-git-send-email-jsimmons@infradead.org> Message-ID: <1594845918-29027-27-git-send-email-jsimmons@infradead.org> From: Mr NeilBrown Several loops use list_for_each_safe(), then call list_entry() as first step. These can be merged using list_for_each_entry_safe(). In one case, the 'safe' version is clearly not needed, so just use list_for_each_entry(). WC-bug-id: https://jira.whamcloud.com/browse/LU-12678 Lustre-commit: 03f375e9f6390 ("LU-12678 socklnd: use list_for_each_entry_safe()") Signed-off-by: Mr NeilBrown Reviewed-on: https://review.whamcloud.com/39129 Reviewed-by: James Simmons Reviewed-by: Chris Horn Reviewed-by: Oleg Drokin Signed-off-by: James Simmons --- net/lnet/klnds/socklnd/socklnd.c | 55 ++++++++++++++-------------------------- 1 file changed, 19 insertions(+), 36 deletions(-) diff --git a/net/lnet/klnds/socklnd/socklnd.c b/net/lnet/klnds/socklnd/socklnd.c index 2b8fd3d..2e11737 100644 --- a/net/lnet/klnds/socklnd/socklnd.c +++ b/net/lnet/klnds/socklnd/socklnd.c @@ -453,15 +453,12 @@ struct ksock_peer_ni * struct ksock_peer_ni *peer_ni = route->ksnr_peer; struct ksock_interface *iface; struct ksock_conn *conn; - struct list_head *ctmp; - struct list_head *cnxt; + struct ksock_conn *cnxt; LASSERT(!route->ksnr_deleted); /* Close associated conns */ - list_for_each_safe(ctmp, cnxt, &peer_ni->ksnp_conns) { - conn = list_entry(ctmp, struct ksock_conn, ksnc_list); - + list_for_each_entry_safe(conn, cnxt, &peer_ni->ksnp_conns, ksnc_list) { if (conn->ksnc_route != route) continue; @@ -548,9 +545,9 @@ struct ksock_peer_ni * ksocknal_del_peer_locked(struct ksock_peer_ni *peer_ni, u32 ip) { struct ksock_conn *conn; + struct ksock_conn *cnxt; struct ksock_route *route; - struct list_head *tmp; - struct list_head *nxt; + struct ksock_route *rnxt; int nshared; LASSERT(!peer_ni->ksnp_closing); @@ -558,9 +555,8 @@ struct ksock_peer_ni * /* Extra ref prevents peer_ni disappearing until I'm done with it */ ksocknal_peer_addref(peer_ni); - list_for_each_safe(tmp, nxt, &peer_ni->ksnp_routes) { - route = list_entry(tmp, struct ksock_route, ksnr_list); - + list_for_each_entry_safe(route, rnxt, &peer_ni->ksnp_routes, + ksnr_list) { /* no match */ if (!(!ip || route->ksnr_ipaddr == ip)) continue; @@ -571,29 +567,23 @@ struct ksock_peer_ni * } nshared = 0; - list_for_each_safe(tmp, nxt, &peer_ni->ksnp_routes) { - route = list_entry(tmp, struct ksock_route, ksnr_list); + list_for_each_entry(route, &peer_ni->ksnp_routes, ksnr_list) nshared += route->ksnr_share_count; - } if (!nshared) { - /* - * remove everything else if there are no explicit entries + /* remove everything else if there are no explicit entries * left */ - list_for_each_safe(tmp, nxt, &peer_ni->ksnp_routes) { - route = list_entry(tmp, struct ksock_route, ksnr_list); - + list_for_each_entry_safe(route, rnxt, &peer_ni->ksnp_routes, + ksnr_list) { /* we should only be removing auto-entries */ LASSERT(!route->ksnr_share_count); ksocknal_del_route_locked(route); } - list_for_each_safe(tmp, nxt, &peer_ni->ksnp_conns) { - conn = list_entry(tmp, struct ksock_conn, ksnc_list); - + list_for_each_entry_safe(conn, cnxt, &peer_ni->ksnp_conns, + ksnc_list) ksocknal_close_conn_locked(conn, 0); - } } ksocknal_peer_decref(peer_ni); @@ -1752,13 +1742,10 @@ struct ksock_peer_ni * u32 ipaddr, int why) { struct ksock_conn *conn; - struct list_head *ctmp; - struct list_head *cnxt; + struct ksock_conn *cnxt; int count = 0; - list_for_each_safe(ctmp, cnxt, &peer_ni->ksnp_conns) { - conn = list_entry(ctmp, struct ksock_conn, ksnc_list); - + list_for_each_entry_safe(conn, cnxt, &peer_ni->ksnp_conns, ksnc_list) { if (!ipaddr || conn->ksnc_ipaddr == ipaddr) { count++; ksocknal_close_conn_locked(conn, why); @@ -1992,10 +1979,10 @@ static int ksocknal_push(struct lnet_ni *ni, struct lnet_process_id id) ksocknal_peer_del_interface_locked(struct ksock_peer_ni *peer_ni, u32 ipaddr, int index) { - struct list_head *tmp; - struct list_head *nxt; struct ksock_route *route; + struct ksock_route *rnxt; struct ksock_conn *conn; + struct ksock_conn *cnxt; int i; int j; @@ -2008,9 +1995,8 @@ static int ksocknal_push(struct lnet_ni *ni, struct lnet_process_id id) break; } - list_for_each_safe(tmp, nxt, &peer_ni->ksnp_routes) { - route = list_entry(tmp, struct ksock_route, ksnr_list); - + list_for_each_entry_safe(route, rnxt, &peer_ni->ksnp_routes, + ksnr_list) { if (route->ksnr_myiface != index) continue; @@ -2022,12 +2008,9 @@ static int ksocknal_push(struct lnet_ni *ni, struct lnet_process_id id) } } - list_for_each_safe(tmp, nxt, &peer_ni->ksnp_conns) { - conn = list_entry(tmp, struct ksock_conn, ksnc_list); - + list_for_each_entry_safe(conn, cnxt, &peer_ni->ksnp_conns, ksnc_list) if (conn->ksnc_myipaddr == ipaddr) ksocknal_close_conn_locked(conn, 0); - } } static int -- 1.8.3.1 From jsimmons at infradead.org Wed Jul 15 20:44:44 2020 From: jsimmons at infradead.org (James Simmons) Date: Wed, 15 Jul 2020 16:44:44 -0400 Subject: [lustre-devel] [PATCH 03/37] lustre: sec: better struct sepol_downcall_data In-Reply-To: <1594845918-29027-1-git-send-email-jsimmons@infradead.org> References: <1594845918-29027-1-git-send-email-jsimmons@infradead.org> Message-ID: <1594845918-29027-4-git-send-email-jsimmons@infradead.org> From: Sebastien Buisson struct sepol_downcall_data is badly formed for several reasons: - it uses a __kernel_time_t field, which can be variably sized, depending on the size of __kernel_long_t. Replace it with a fixed-size __s64 type; - it has __u32 sdd_magic that is immediately before a potentially 64-bit field, whereas the 64-bit fields in a structure should always be naturally aligned on 64-bit boundaries to avoid potential incompatibility in the structure definition; - it has __u16 sdd_sepol_len which may be followed by padding. So create a better struct sepol_downcall_data, while maintaining compatibility with 2.12 by keeping a struct sepol_downcall_data_old. WC-bug-id: https://jira.whamcloud.com/browse/LU-13525 Lustre-commit: 82b8cb5528f48 ("LU-13525 sec: better struct sepol_downcall_data") Signed-off-by: Sebastien Buisson Reviewed-on: https://review.whamcloud.com/38580 Reviewed-by: Olaf Faaland-LLNL Reviewed-by: Andreas Dilger Reviewed-by: Oleg Drokin Signed-off-by: James Simmons --- fs/lustre/ptlrpc/sec_lproc.c | 134 ++++++++++++++++++++++++++++---- include/uapi/linux/lustre/lustre_user.h | 16 +++- 2 files changed, 135 insertions(+), 15 deletions(-) diff --git a/fs/lustre/ptlrpc/sec_lproc.c b/fs/lustre/ptlrpc/sec_lproc.c index 7db7e81..b34ced4 100644 --- a/fs/lustre/ptlrpc/sec_lproc.c +++ b/fs/lustre/ptlrpc/sec_lproc.c @@ -131,6 +131,86 @@ static int sptlrpc_ctxs_lprocfs_seq_show(struct seq_file *seq, void *v) LPROC_SEQ_FOPS_RO(sptlrpc_ctxs_lprocfs); +#if LUSTRE_VERSION_CODE < OBD_OCD_VERSION(2, 16, 53, 0) +static ssize_t sepol_seq_write_old(struct obd_device *obd, + const char __user *buffer, + size_t count) +{ + struct client_obd *cli = &obd->u.cli; + struct obd_import *imp = cli->cl_import; + struct sepol_downcall_data_old *param; + int size = sizeof(*param); + u16 len; + int rc = 0; + + if (count < size) { + rc = -EINVAL; + CERROR("%s: invalid data count = %lu, size = %d: rc = %d\n", + obd->obd_name, (unsigned long) count, size, rc); + return rc; + } + + param = kmalloc(size, GFP_KERNEL); + if (!param) + return -ENOMEM; + + if (copy_from_user(param, buffer, size)) { + rc = -EFAULT; + CERROR("%s: bad sepol data: rc = %d\n", obd->obd_name, rc); + goto out; + } + + if (param->sdd_magic != SEPOL_DOWNCALL_MAGIC_OLD) { + rc = -EINVAL; + CERROR("%s: sepol downcall bad params: rc = %d\n", + obd->obd_name, rc); + goto out; + } + + if (param->sdd_sepol_len == 0 || + param->sdd_sepol_len >= sizeof(imp->imp_sec->ps_sepol)) { + rc = -EINVAL; + CERROR("%s: invalid sepol data returned: rc = %d\n", + obd->obd_name, rc); + goto out; + } + len = param->sdd_sepol_len; /* save sdd_sepol_len */ + kfree(param); + size = offsetof(struct sepol_downcall_data_old, + sdd_sepol[len]); + + if (count < size) { + rc = -EINVAL; + CERROR("%s: invalid sepol count = %lu, size = %d: rc = %d\n", + obd->obd_name, (unsigned long) count, size, rc); + return rc; + } + + /* alloc again with real size */ + param = kmalloc(size, GFP_KERNEL); + if (!param) + return -ENOMEM; + + if (copy_from_user(param, buffer, size)) { + rc = -EFAULT; + CERROR("%s: cannot copy sepol data: rc = %d\n", + obd->obd_name, rc); + goto out; + } + + spin_lock(&imp->imp_sec->ps_lock); + snprintf(imp->imp_sec->ps_sepol, param->sdd_sepol_len + 1, "%s", + param->sdd_sepol); + imp->imp_sec->ps_sepol_mtime = ktime_set(param->sdd_sepol_mtime, 0); + spin_unlock(&imp->imp_sec->ps_lock); + +out: + kfree(param); + + return rc ? rc : count; +} +#endif + static ssize_t lprocfs_wr_sptlrpc_sepol(struct file *file, const char __user *buffer, size_t count, void *data) @@ -140,13 +220,41 @@ static int sptlrpc_ctxs_lprocfs_seq_show(struct seq_file *seq, void *v) struct client_obd *cli = &obd->u.cli; struct obd_import *imp = cli->cl_import; struct sepol_downcall_data *param; - int size = sizeof(*param); + u32 magic; + int size = sizeof(magic); + u16 len; int rc = 0; if (count < size) { - CERROR("%s: invalid data count = %lu, size = %d\n", - obd->obd_name, (unsigned long) count, size); - return -EINVAL; + rc = -EINVAL; + CERROR("%s: invalid buffer count = %lu, size = %d: rc = %d\n", + obd->obd_name, (unsigned long) count, size, rc); + return rc; + } + + if (copy_from_user(&magic, buffer, size)) { + rc = -EFAULT; + CERROR("%s: bad sepol magic: rc = %d\n", obd->obd_name, rc); + return rc; + } + + if (magic != SEPOL_DOWNCALL_MAGIC) { +#if LUSTRE_VERSION_CODE < OBD_OCD_VERSION(2, 16, 53, 0) + if (magic == SEPOL_DOWNCALL_MAGIC_OLD) + return sepol_seq_write_old(obd, buffer, count); +#endif + rc = -EINVAL; + CERROR("%s: sepol downcall bad magic '%#08x': rc = %d\n", + obd->obd_name, magic, rc); + return rc; + } + + size = sizeof(*param); + if (count < size) { + rc = -EINVAL; + CERROR("%s: invalid data count = %lu, size = %d: rc = %d\n", + obd->obd_name, (unsigned long) count, size, rc); + return rc; } param = kzalloc(size, GFP_KERNEL); @@ -154,39 +262,39 @@ static int sptlrpc_ctxs_lprocfs_seq_show(struct seq_file *seq, void *v) return -ENOMEM; if (copy_from_user(param, buffer, size)) { - CERROR("%s: bad sepol data\n", obd->obd_name); rc = -EFAULT; + CERROR("%s: bad sepol data: rc = %d\n", obd->obd_name, rc); goto out; } if (param->sdd_magic != SEPOL_DOWNCALL_MAGIC) { - CERROR("%s: sepol downcall bad params\n", - obd->obd_name); rc = -EINVAL; + CERROR("%s: invalid sepol data returned: rc = %d\n", + obd->obd_name, rc); goto out; } if (param->sdd_sepol_len == 0 || param->sdd_sepol_len >= sizeof(imp->imp_sec->ps_sepol)) { - CERROR("%s: invalid sepol data returned\n", - obd->obd_name); rc = -EINVAL; + CERROR("%s: invalid sepol data returned: rc = %d\n", + obd->obd_name, rc); goto out; } - rc = param->sdd_sepol_len; /* save sdd_sepol_len */ + len = param->sdd_sepol_len; /* save sdd_sepol_len */ kfree(param); size = offsetof(struct sepol_downcall_data, - sdd_sepol[rc]); + sdd_sepol[len]); /* alloc again with real size */ - rc = 0; param = kzalloc(size, GFP_KERNEL); if (!param) return -ENOMEM; if (copy_from_user(param, buffer, size)) { - CERROR("%s: bad sepol data\n", obd->obd_name); rc = -EFAULT; + CERROR("%s: cannot copy sepol data: rc = %d\n", + obd->obd_name, rc); goto out; } diff --git a/include/uapi/linux/lustre/lustre_user.h b/include/uapi/linux/lustre/lustre_user.h index 6a2d5f9..b0301e1 100644 --- a/include/uapi/linux/lustre/lustre_user.h +++ b/include/uapi/linux/lustre/lustre_user.h @@ -51,6 +51,7 @@ #include #include #include +#include #ifndef __KERNEL__ # define __USE_ISOC99 1 @@ -980,7 +981,6 @@ static inline const char *qtype_name(int qtype) } #define IDENTITY_DOWNCALL_MAGIC 0x6d6dd629 -#define SEPOL_DOWNCALL_MAGIC 0x8b8bb842 /* permission */ #define N_PERMS_MAX 64 @@ -1002,13 +1002,25 @@ struct identity_downcall_data { __u32 idd_groups[0]; }; -struct sepol_downcall_data { +#if LUSTRE_VERSION_CODE < OBD_OCD_VERSION(2, 16, 53, 0) +/* old interface struct is deprecated in 2.14 */ +#define SEPOL_DOWNCALL_MAGIC_OLD 0x8b8bb842 +struct sepol_downcall_data_old { __u32 sdd_magic; __s64 sdd_sepol_mtime; __u16 sdd_sepol_len; char sdd_sepol[0]; }; +#endif +#define SEPOL_DOWNCALL_MAGIC 0x8b8bb843 +struct sepol_downcall_data { + __u32 sdd_magic; + __u16 sdd_sepol_len; + __u16 sdd_padding1; + __s64 sdd_sepol_mtime; + char sdd_sepol[0]; +}; /* lustre volatile file support * file name header: ".^L^S^T^R:volatile" -- 1.8.3.1 From jsimmons at infradead.org Wed Jul 15 20:44:42 2020 From: jsimmons at infradead.org (James Simmons) Date: Wed, 15 Jul 2020 16:44:42 -0400 Subject: [lustre-devel] [PATCH 01/37] lustre: osc: fix osc_extent_find() In-Reply-To: <1594845918-29027-1-git-send-email-jsimmons@infradead.org> References: <1594845918-29027-1-git-send-email-jsimmons@infradead.org> Message-ID: <1594845918-29027-2-git-send-email-jsimmons@infradead.org> From: Mr NeilBrown - fix a pre-existing bug - osc_extent_merge() should never try to merge two extends with different ->oe_mppr as later alignment checks can get confused. - Remove a redundant list_del_init() which is already included in __osc_extent_remove(). Fixes: 85ebb57ddc ("lustre: osc: simplify osc_extent_find()") WC-bug-id: https://jira.whamcloud.com/browse/LU-9679 Lustre-commit: 80e21cce3dd67 ("LU-9679 osc: simplify osc_extent_find()") Signed-off-by: Mr NeilBrown Reviewed-on: https://review.whamcloud.com/37607 Reviewed-by: Andreas Dilger Reviewed-by: James Simmons Reviewed-by: Oleg Drokin Signed-off-by: James Simmons --- fs/lustre/osc/osc_cache.c | 33 +++++++++++++++------------------ 1 file changed, 15 insertions(+), 18 deletions(-) diff --git a/fs/lustre/osc/osc_cache.c b/fs/lustre/osc/osc_cache.c index 5049aaa..474b711 100644 --- a/fs/lustre/osc/osc_cache.c +++ b/fs/lustre/osc/osc_cache.c @@ -574,6 +574,14 @@ static int osc_extent_merge(const struct lu_env *env, struct osc_extent *cur, if (cur->oe_max_end != victim->oe_max_end) return -ERANGE; + /* + * In the rare case max_pages_per_rpc (mppr) is changed, don't + * merge extents until after old ones have been sent, or the + * "extents are aligned to RPCs" checks are unhappy. + */ + if (cur->oe_mppr != victim->oe_mppr) + return -ERANGE; + LASSERT(cur->oe_dlmlock == victim->oe_dlmlock); ppc_bits = osc_cli(obj)->cl_chunkbits - PAGE_SHIFT; chunk_start = cur->oe_start >> ppc_bits; @@ -601,7 +609,6 @@ static int osc_extent_merge(const struct lu_env *env, struct osc_extent *cur, cur->oe_urgent |= victim->oe_urgent; cur->oe_memalloc |= victim->oe_memalloc; list_splice_init(&victim->oe_pages, &cur->oe_pages); - list_del_init(&victim->oe_link); victim->oe_nr_pages = 0; osc_extent_get(victim); @@ -727,8 +734,7 @@ static struct osc_extent *osc_extent_find(const struct lu_env *env, cur->oe_start = descr->cld_start; if (cur->oe_end > max_end) cur->oe_end = max_end; - LASSERT(*grants >= chunksize); - cur->oe_grants = chunksize; + cur->oe_grants = chunksize + cli->cl_grant_extent_tax; cur->oe_mppr = max_pages; if (olck->ols_dlmlock) { LASSERT(olck->ols_hold); @@ -800,17 +806,8 @@ static struct osc_extent *osc_extent_find(const struct lu_env *env, */ continue; - /* it's required that an extent must be contiguous at chunk - * level so that we know the whole extent is covered by grant - * (the pages in the extent are NOT required to be contiguous). - * Otherwise, it will be too much difficult to know which - * chunks have grants allocated. - */ - /* On success, osc_extent_merge() will put cur, - * so we take an extra reference - */ - osc_extent_get(cur); if (osc_extent_merge(env, ext, cur) == 0) { + LASSERT(*grants >= chunksize); *grants -= chunksize; found = osc_extent_hold(ext); @@ -824,19 +821,19 @@ static struct osc_extent *osc_extent_find(const struct lu_env *env, break; } - osc_extent_put(env, cur); } osc_extent_tree_dump(D_CACHE, obj); if (found) { LASSERT(!conflict); - LASSERT(found->oe_dlmlock == cur->oe_dlmlock); - OSC_EXTENT_DUMP(D_CACHE, found, - "found caching ext for %lu.\n", index); + if (!IS_ERR(found)) { + LASSERT(found->oe_dlmlock == cur->oe_dlmlock); + OSC_EXTENT_DUMP(D_CACHE, found, + "found caching ext for %lu.\n", index); + } } else if (!conflict) { /* create a new extent */ EASSERT(osc_extent_is_overlapped(obj, cur) == 0, cur); - cur->oe_grants = chunksize + cli->cl_grant_extent_tax; LASSERT(*grants >= cur->oe_grants); *grants -= cur->oe_grants; -- 1.8.3.1 From jsimmons at infradead.org Wed Jul 15 20:44:48 2020 From: jsimmons at infradead.org (James Simmons) Date: Wed, 15 Jul 2020 16:44:48 -0400 Subject: [lustre-devel] [PATCH 07/37] lustre: osc: disable ext merging for rdma only pages and non-rdma In-Reply-To: <1594845918-29027-1-git-send-email-jsimmons@infradead.org> References: <1594845918-29027-1-git-send-email-jsimmons@infradead.org> Message-ID: <1594845918-29027-8-git-send-email-jsimmons@infradead.org> From: Wang Shilong This patch try to add logic to prevent CPU memory pages and RDMA memory pages from merging into one RPC, codes which set OBD_BRW_RDMA_ONLY will be added whenever RDMA only codes added later. WC-bug-id: https://jira.whamcloud.com/browse/LU-13180 Lustre-commit: 9f6c9fa44d6e6 ("LU-13180 osc: disable ext merging for rdma only pages and non-rdma") Signed-off-by: Wang Shilong Reviewed-on: https://review.whamcloud.com/37567 Reviewed-by: Andreas Dilger Reviewed-by: Gu Zheng Reviewed-by: Yingjin Qian Signed-off-by: James Simmons --- fs/lustre/include/lustre_osc.h | 4 +++- fs/lustre/osc/osc_cache.c | 4 ++++ 2 files changed, 7 insertions(+), 1 deletion(-) diff --git a/fs/lustre/include/lustre_osc.h b/fs/lustre/include/lustre_osc.h index 11b7e92..cd08f27 100644 --- a/fs/lustre/include/lustre_osc.h +++ b/fs/lustre/include/lustre_osc.h @@ -939,7 +939,9 @@ struct osc_extent { /* Non-delay RPC should be used for this extent. */ oe_ndelay:1, /* direct IO pages */ - oe_dio:1; + oe_dio:1, + /* this extent consists of RDMA only pages */ + oe_is_rdma_only; /* how many grants allocated for this extent. * Grant allocated for this extent. There is no grant allocated * for reading extents and sync write extents. diff --git a/fs/lustre/osc/osc_cache.c b/fs/lustre/osc/osc_cache.c index 474b711..f811dadb 100644 --- a/fs/lustre/osc/osc_cache.c +++ b/fs/lustre/osc/osc_cache.c @@ -1927,6 +1927,9 @@ static inline unsigned int osc_extent_chunks(const struct osc_extent *ext) if (in_rpc->oe_dio && overlapped(ext, in_rpc)) return false; + if (ext->oe_is_rdma_only != in_rpc->oe_is_rdma_only) + return false; + return true; } @@ -2688,6 +2691,7 @@ int osc_queue_sync_pages(const struct lu_env *env, const struct cl_io *io, ext->oe_srvlock = !!(brw_flags & OBD_BRW_SRVLOCK); ext->oe_ndelay = !!(brw_flags & OBD_BRW_NDELAY); ext->oe_dio = !!(brw_flags & OBD_BRW_NOCACHE); + ext->oe_is_rdma_only = !!(brw_flags & OBD_BRW_RDMA_ONLY); ext->oe_nr_pages = page_count; ext->oe_mppr = mppr; list_splice_init(list, &ext->oe_pages); -- 1.8.3.1 From jsimmons at infradead.org Wed Jul 15 20:44:50 2020 From: jsimmons at infradead.org (James Simmons) Date: Wed, 15 Jul 2020 16:44:50 -0400 Subject: [lustre-devel] [PATCH 09/37] lnet: o2iblnd: allocate init_qp_attr on stack. In-Reply-To: <1594845918-29027-1-git-send-email-jsimmons@infradead.org> References: <1594845918-29027-1-git-send-email-jsimmons@infradead.org> Message-ID: <1594845918-29027-10-git-send-email-jsimmons@infradead.org> From: Mr NeilBrown 'struct ib_qp_init_attr' is not so large that it cannot be allocated on the stack. It is about 100 bytes, various other function in Linux allocate it on the stack, and the stack isn't as constrained as it once was. So allocate on stack instead of using kmalloc and handling errors. WC-bug-id: https://jira.whamcloud.com/browse/LU-12678 Lustre-commit: 524a5a733ba1c ("LU-12678 o2iblnd: allocate init_qp_attr on stack.") Signed-off-by: Mr NeilBrown Reviewed-on: https://review.whamcloud.com/39122 Reviewed-by: Serguei Smirnov Reviewed-by: James Simmons Reviewed-by: Oleg Drokin Signed-off-by: James Simmons --- net/lnet/klnds/o2iblnd/o2iblnd.c | 45 +++++++++++++++------------------------- 1 file changed, 17 insertions(+), 28 deletions(-) diff --git a/net/lnet/klnds/o2iblnd/o2iblnd.c b/net/lnet/klnds/o2iblnd/o2iblnd.c index 16edfba..d8fca2a 100644 --- a/net/lnet/klnds/o2iblnd/o2iblnd.c +++ b/net/lnet/klnds/o2iblnd/o2iblnd.c @@ -699,7 +699,7 @@ struct kib_conn *kiblnd_create_conn(struct kib_peer_ni *peer_ni, rwlock_t *glock = &kiblnd_data.kib_global_lock; struct kib_net *net = peer_ni->ibp_ni->ni_data; struct kib_dev *dev; - struct ib_qp_init_attr *init_qp_attr; + struct ib_qp_init_attr init_qp_attr = {}; struct kib_sched_info *sched; struct ib_cq_init_attr cq_attr = {}; struct kib_conn *conn; @@ -727,18 +727,11 @@ struct kib_conn *kiblnd_create_conn(struct kib_peer_ni *peer_ni, */ cpt = sched->ibs_cpt; - init_qp_attr = kzalloc_cpt(sizeof(*init_qp_attr), GFP_NOFS, cpt); - if (!init_qp_attr) { - CERROR("Can't allocate qp_attr for %s\n", - libcfs_nid2str(peer_ni->ibp_nid)); - goto failed_0; - } - conn = kzalloc_cpt(sizeof(*conn), GFP_NOFS, cpt); if (!conn) { CERROR("Can't allocate connection for %s\n", libcfs_nid2str(peer_ni->ibp_nid)); - goto failed_1; + goto failed_0; } conn->ibc_state = IBLND_CONN_INIT; @@ -819,27 +812,27 @@ struct kib_conn *kiblnd_create_conn(struct kib_peer_ni *peer_ni, goto failed_2; } - init_qp_attr->event_handler = kiblnd_qp_event; - init_qp_attr->qp_context = conn; - init_qp_attr->cap.max_send_sge = *kiblnd_tunables.kib_wrq_sge; - init_qp_attr->cap.max_recv_sge = 1; - init_qp_attr->sq_sig_type = IB_SIGNAL_REQ_WR; - init_qp_attr->qp_type = IB_QPT_RC; - init_qp_attr->send_cq = cq; - init_qp_attr->recv_cq = cq; + init_qp_attr.event_handler = kiblnd_qp_event; + init_qp_attr.qp_context = conn; + init_qp_attr.cap.max_send_sge = *kiblnd_tunables.kib_wrq_sge; + init_qp_attr.cap.max_recv_sge = 1; + init_qp_attr.sq_sig_type = IB_SIGNAL_REQ_WR; + init_qp_attr.qp_type = IB_QPT_RC; + init_qp_attr.send_cq = cq; + init_qp_attr.recv_cq = cq; /* kiblnd_send_wrs() can change the connection's queue depth if * the maximum work requests for the device is maxed out */ - init_qp_attr->cap.max_send_wr = kiblnd_send_wrs(conn); - init_qp_attr->cap.max_recv_wr = IBLND_RECV_WRS(conn); + init_qp_attr.cap.max_send_wr = kiblnd_send_wrs(conn); + init_qp_attr.cap.max_recv_wr = IBLND_RECV_WRS(conn); - rc = rdma_create_qp(cmid, conn->ibc_hdev->ibh_pd, init_qp_attr); + rc = rdma_create_qp(cmid, conn->ibc_hdev->ibh_pd, &init_qp_attr); if (rc) { CERROR("Can't create QP: %d, send_wr: %d, recv_wr: %d, send_sge: %d, recv_sge: %d\n", - rc, init_qp_attr->cap.max_send_wr, - init_qp_attr->cap.max_recv_wr, - init_qp_attr->cap.max_send_sge, - init_qp_attr->cap.max_recv_sge); + rc, init_qp_attr.cap.max_send_wr, + init_qp_attr.cap.max_recv_wr, + init_qp_attr.cap.max_send_sge, + init_qp_attr.cap.max_recv_sge); goto failed_2; } @@ -851,8 +844,6 @@ struct kib_conn *kiblnd_create_conn(struct kib_peer_ni *peer_ni, peer_ni->ibp_queue_depth, conn->ibc_queue_depth); - kfree(init_qp_attr); - conn->ibc_rxs = kzalloc_cpt(IBLND_RX_MSGS(conn) * sizeof(*conn->ibc_rxs), GFP_NOFS, cpt); @@ -918,8 +909,6 @@ struct kib_conn *kiblnd_create_conn(struct kib_peer_ni *peer_ni, failed_2: kiblnd_destroy_conn(conn); kfree(conn); -failed_1: - kfree(init_qp_attr); failed_0: return NULL; } -- 1.8.3.1 From jsimmons at infradead.org Wed Jul 15 20:45:10 2020 From: jsimmons at infradead.org (James Simmons) Date: Wed, 15 Jul 2020 16:45:10 -0400 Subject: [lustre-devel] [PATCH 29/37] lnet: Allow router to forward to healthier NID In-Reply-To: <1594845918-29027-1-git-send-email-jsimmons@infradead.org> References: <1594845918-29027-1-git-send-email-jsimmons@infradead.org> Message-ID: <1594845918-29027-30-git-send-email-jsimmons@infradead.org> From: Chris Horn When a final-hop router (aka edge router) is forwarding a message, if both the originator and destination of the message are mutli-rail capable, then allow the router to choose a new destination lpni if the one selected by the message originator is unhealthy or down. HPE-bug-id: LUS-8905 WC-bug-id: https://jira.whamcloud.com/browse/LU-13606 Lustre-commit: b0e8ab1a5f6f8 ("LU-13606 lnet: Allow router to forward to healthier NID") Signed-off-by: Chris Horn Reviewed-on: https://review.whamcloud.com/38798 Reviewed-by: Serguei Smirnov Reviewed-by: Amir Shehata Reviewed-by: Oleg Drokin Signed-off-by: James Simmons --- include/linux/lnet/lib-lnet.h | 4 ++-- net/lnet/lnet/lib-move.c | 37 +++++++++++++++++++++++++++++++++++-- 2 files changed, 37 insertions(+), 4 deletions(-) diff --git a/include/linux/lnet/lib-lnet.h b/include/linux/lnet/lib-lnet.h index 75c0da7..b069422 100644 --- a/include/linux/lnet/lib-lnet.h +++ b/include/linux/lnet/lib-lnet.h @@ -819,8 +819,8 @@ int lnet_get_peer_ni_info(u32 peer_index, u64 *nid, } /* - * A peer is alive if it satisfies the following two conditions: - * 1. peer health >= LNET_MAX_HEALTH_VALUE * router_sensitivity_percentage + * A peer NI is alive if it satisfies the following two conditions: + * 1. peer NI health >= LNET_MAX_HEALTH_VALUE * router_sensitivity_percentage * 2. the cached NI status received when we discover the peer is UP */ static inline bool diff --git a/net/lnet/lnet/lib-move.c b/net/lnet/lnet/lib-move.c index 2f3ef8c..234fbb5 100644 --- a/net/lnet/lnet/lib-move.c +++ b/net/lnet/lnet/lib-move.c @@ -2371,6 +2371,8 @@ struct lnet_ni * int cpt, rc; int md_cpt; u32 send_case = 0; + bool final_hop; + bool mr_forwarding_allowed; memset(&send_data, 0, sizeof(send_data)); @@ -2447,16 +2449,47 @@ struct lnet_ni * else send_case |= REMOTE_DST; + final_hop = false; + if (msg->msg_routing && (send_case & LOCAL_DST)) + final_hop = true; + + /* Determine whether to allow MR forwarding for this message. + * NB: MR forwarding is allowed if the message originator and the + * destination are both MR capable, and the destination lpni that was + * originally chosen by the originator is unhealthy or down. + * We check the MR capability of the destination further below + */ + mr_forwarding_allowed = false; + if (final_hop) { + struct lnet_peer *src_lp; + struct lnet_peer_ni *src_lpni; + + src_lpni = lnet_nid2peerni_locked(msg->msg_hdr.src_nid, + LNET_NID_ANY, cpt); + /* We don't fail the send if we hit any errors here. We'll just + * try to send it via non-multi-rail criteria + */ + if (!IS_ERR(src_lpni)) { + src_lp = lpni->lpni_peer_net->lpn_peer; + if (lnet_peer_is_multi_rail(src_lp) && + !lnet_is_peer_ni_alive(lpni)) + mr_forwarding_allowed = true; + } + CDEBUG(D_NET, "msg %p MR forwarding %s\n", msg, + mr_forwarding_allowed ? "allowed" : "not allowed"); + } + /* Deal with the peer as NMR in the following cases: * 1. the peer is NMR * 2. We're trying to recover a specific peer NI - * 3. I'm a router sending to the final destination + * 3. I'm a router sending to the final destination and MR forwarding is + * not allowed for this message (as determined above). * In this case the source of the message would've * already selected the final destination so my job * is to honor the selection. */ if (!lnet_peer_is_multi_rail(peer) || msg->msg_recovery || - (msg->msg_routing && (send_case & LOCAL_DST))) + (final_hop && !mr_forwarding_allowed)) send_case |= NMR_DST; else send_case |= MR_DST; -- 1.8.3.1 From jsimmons at infradead.org Wed Jul 15 20:45:17 2020 From: jsimmons at infradead.org (James Simmons) Date: Wed, 15 Jul 2020 16:45:17 -0400 Subject: [lustre-devel] [PATCH 36/37] lnet: socklnd: change ksnd_nthreads to atomic_t In-Reply-To: <1594845918-29027-1-git-send-email-jsimmons@infradead.org> References: <1594845918-29027-1-git-send-email-jsimmons@infradead.org> Message-ID: <1594845918-29027-37-git-send-email-jsimmons@infradead.org> From: Mr NeilBrown This variable is treated like an atomic_t, but a global spinlock is used to protect updates - and also unnecessarily to protect reads. Change to atomic_t and avoid using the spinlock. WC-bug-id: https://jira.whamcloud.com/browse/LU-12678 Lustre-commit: 4b0d3c0e41201 ("LU-12678 socklnd: change ksnd_nthreads to atomic_t") Signed-off-by: Mr NeilBrown Reviewed-on: https://review.whamcloud.com/39121 Reviewed-by: James Simmons Reviewed-by: Chris Horn Reviewed-by: Oleg Drokin Signed-off-by: James Simmons --- net/lnet/klnds/socklnd/socklnd.c | 4 ++-- net/lnet/klnds/socklnd/socklnd.h | 2 +- net/lnet/klnds/socklnd/socklnd_cb.c | 8 ++------ 3 files changed, 5 insertions(+), 9 deletions(-) diff --git a/net/lnet/klnds/socklnd/socklnd.c b/net/lnet/klnds/socklnd/socklnd.c index 22a73c3..91925475 100644 --- a/net/lnet/klnds/socklnd/socklnd.c +++ b/net/lnet/klnds/socklnd/socklnd.c @@ -2260,9 +2260,9 @@ static int ksocknal_push(struct lnet_ni *ni, struct lnet_process_id id) } wait_var_event_warning(&ksocknal_data.ksnd_nthreads, - ksocknal_data.ksnd_nthreads == 0, + atomic_read(&ksocknal_data.ksnd_nthreads) == 0, "waiting for %d threads to terminate\n", - ksocknal_data.ksnd_nthreads); + atomic_read(&ksocknal_data.ksnd_nthreads)); ksocknal_free_buffers(); diff --git a/net/lnet/klnds/socklnd/socklnd.h b/net/lnet/klnds/socklnd/socklnd.h index df863f2..350f2c8 100644 --- a/net/lnet/klnds/socklnd/socklnd.h +++ b/net/lnet/klnds/socklnd/socklnd.h @@ -196,7 +196,7 @@ struct ksock_nal_data { * known peers */ - int ksnd_nthreads; /* # live threads */ + atomic_t ksnd_nthreads; /* # live threads */ int ksnd_shuttingdown; /* tell threads to exit */ struct ksock_sched **ksnd_schedulers; /* schedulers info */ diff --git a/net/lnet/klnds/socklnd/socklnd_cb.c b/net/lnet/klnds/socklnd/socklnd_cb.c index 9b3b604..a1c0c3d 100644 --- a/net/lnet/klnds/socklnd/socklnd_cb.c +++ b/net/lnet/klnds/socklnd/socklnd_cb.c @@ -976,19 +976,15 @@ struct ksock_route * if (IS_ERR(task)) return PTR_ERR(task); - write_lock_bh(&ksocknal_data.ksnd_global_lock); - ksocknal_data.ksnd_nthreads++; - write_unlock_bh(&ksocknal_data.ksnd_global_lock); + atomic_inc(&ksocknal_data.ksnd_nthreads); return 0; } void ksocknal_thread_fini(void) { - write_lock_bh(&ksocknal_data.ksnd_global_lock); - if (--ksocknal_data.ksnd_nthreads == 0) + if (atomic_dec_and_test(&ksocknal_data.ksnd_nthreads)) wake_up_var(&ksocknal_data.ksnd_nthreads); - write_unlock_bh(&ksocknal_data.ksnd_global_lock); } int -- 1.8.3.1 From jsimmons at infradead.org Wed Jul 15 20:44:52 2020 From: jsimmons at infradead.org (James Simmons) Date: Wed, 15 Jul 2020 16:44:52 -0400 Subject: [lustre-devel] [PATCH 11/37] lnet: socklnd: don't fall-back to tcp_sendpage. In-Reply-To: <1594845918-29027-1-git-send-email-jsimmons@infradead.org> References: <1594845918-29027-1-git-send-email-jsimmons@infradead.org> Message-ID: <1594845918-29027-12-git-send-email-jsimmons@infradead.org> From: Mr NeilBrown sk_prot->sendpage is never NULL, so there is no need for a fallback to tcp_sendpage. WC-bug-id: https://jira.whamcloud.com/browse/LU-12678 Lustre-commit: 011d760069142 ("LU-12678 socklnd: don't fall-back to tcp_sendpage.") Signed-off-by: Mr NeilBrown Reviewed-on: https://review.whamcloud.com/39134 Reviewed-by: Shaun Tancheff Reviewed-by: James Simmons Reviewed-by: Oleg Drokin Signed-off-by: James Simmons --- net/lnet/klnds/socklnd/socklnd_lib.c | 8 ++------ 1 file changed, 2 insertions(+), 6 deletions(-) diff --git a/net/lnet/klnds/socklnd/socklnd_lib.c b/net/lnet/klnds/socklnd/socklnd_lib.c index 2adc99c..1d6cd0e 100644 --- a/net/lnet/klnds/socklnd/socklnd_lib.c +++ b/net/lnet/klnds/socklnd/socklnd_lib.c @@ -123,12 +123,8 @@ fragsize < tx->tx_resid) msgflg |= MSG_MORE; - if (sk->sk_prot->sendpage) { - rc = sk->sk_prot->sendpage(sk, page, - offset, fragsize, msgflg); - } else { - rc = tcp_sendpage(sk, page, offset, fragsize, msgflg); - } + rc = sk->sk_prot->sendpage(sk, page, + offset, fragsize, msgflg); } else { struct msghdr msg = { .msg_flags = MSG_DONTWAIT }; int i; -- 1.8.3.1 From jsimmons at infradead.org Wed Jul 15 20:44:54 2020 From: jsimmons at infradead.org (James Simmons) Date: Wed, 15 Jul 2020 16:44:54 -0400 Subject: [lustre-devel] [PATCH 13/37] lustre: obdcalss: ensure LCT_QUIESCENT take sync In-Reply-To: <1594845918-29027-1-git-send-email-jsimmons@infradead.org> References: <1594845918-29027-1-git-send-email-jsimmons@infradead.org> Message-ID: <1594845918-29027-14-git-send-email-jsimmons@infradead.org> From: Yang Sheng Add locking in lu_device_init ensure LCT_QUIESCENT operating can be seen on other thread in parallel mounting. Also add extra checking before unset the flag to make sure we don't do it after device has been started. (osd_handler.c:7730:osd_device_init0()) ASSERTION( info ) failed: (osd_handler.c:7730:osd_device_init0()) LBUG Pid: 28098, comm: mount.lustre 3.10.0-1062.9.1.el7_lustre.x86_64 Call Trace: libcfs_call_trace+0x8c/0xc0 [libcfs] lbug_with_loc+0x4c/0xa0 [libcfs] osd_device_alloc+0x778/0x8f0 [osd_ldiskfs] obd_setup+0x129/0x2f0 [obdclass] class_setup+0x48f/0x7f0 [obdclass] class_process_config+0x190f/0x2830 [obdclass] do_lcfg+0x258/0x500 [obdclass] lustre_start_simple+0x88/0x210 [obdclass] server_fill_super+0xf55/0x1890 [obdclass] lustre_fill_super+0x498/0x990 [obdclass] mount_nodev+0x4f/0xb0 lustre_mount+0x18/0x20 [obdclass] mount_fs+0x3e/0x1b0 vfs_kern_mount+0x67/0x110 do_mount+0x1ef/0xce0 SyS_mount+0x83/0xd0 system_call_fastpath+0x25/0x2a 0xffffffffffffffff Kernel panic - not syncing: LBUG WC-bug-id: https://jira.whamcloud.com/browse/LU-11814 Lustre-commit: 979f5e1db041d ("LU-11814 obdcalss: ensure LCT_QUIESCENT take sync") Signed-off-by: Yang Sheng Reviewed-on: https://review.whamcloud.com/38416 Reviewed-by: Andreas Dilger Reviewed-by: Wang Shilong Reviewed-by: Oleg Drokin Signed-off-by: James Simmons --- fs/lustre/include/lu_object.h | 8 +++--- fs/lustre/obdclass/lu_object.c | 58 ++++++++++++++++++++++++------------------ 2 files changed, 38 insertions(+), 28 deletions(-) diff --git a/fs/lustre/include/lu_object.h b/fs/lustre/include/lu_object.h index 1a6b6e1..6c47f43 100644 --- a/fs/lustre/include/lu_object.h +++ b/fs/lustre/include/lu_object.h @@ -1151,7 +1151,8 @@ struct lu_context_key { void lu_context_key_degister(struct lu_context_key *key); void *lu_context_key_get(const struct lu_context *ctx, const struct lu_context_key *key); -void lu_context_key_quiesce(struct lu_context_key *key); +void lu_context_key_quiesce(struct lu_device_type *t, + struct lu_context_key *key); void lu_context_key_revive(struct lu_context_key *key); /* @@ -1199,7 +1200,7 @@ void *lu_context_key_get(const struct lu_context *ctx, #define LU_TYPE_STOP(mod, ...) \ static void mod##_type_stop(struct lu_device_type *t) \ { \ - lu_context_key_quiesce_many(__VA_ARGS__, NULL); \ + lu_context_key_quiesce_many(t, __VA_ARGS__, NULL); \ } \ struct __##mod##_dummy_type_stop {; } @@ -1223,7 +1224,8 @@ void *lu_context_key_get(const struct lu_context *ctx, int lu_context_key_register_many(struct lu_context_key *k, ...); void lu_context_key_degister_many(struct lu_context_key *k, ...); void lu_context_key_revive_many(struct lu_context_key *k, ...); -void lu_context_key_quiesce_many(struct lu_context_key *k, ...); +void lu_context_key_quiesce_many(struct lu_device_type *t, + struct lu_context_key *k, ...); /* * update/clear ctx/ses tags. diff --git a/fs/lustre/obdclass/lu_object.c b/fs/lustre/obdclass/lu_object.c index 5cd8231..42bb7a6 100644 --- a/fs/lustre/obdclass/lu_object.c +++ b/fs/lustre/obdclass/lu_object.c @@ -1185,14 +1185,25 @@ void lu_device_put(struct lu_device *d) } EXPORT_SYMBOL(lu_device_put); +enum { /* Maximal number of tld slots. */ + LU_CONTEXT_KEY_NR = 40 +}; +static struct lu_context_key *lu_keys[LU_CONTEXT_KEY_NR] = { NULL, }; +static DECLARE_RWSEM(lu_key_initing); + /** * Initialize device @d of type @t. */ int lu_device_init(struct lu_device *d, struct lu_device_type *t) { - if (atomic_inc_return(&t->ldt_device_nr) == 1 && - t->ldt_ops->ldto_start) - t->ldt_ops->ldto_start(t); + if (atomic_add_unless(&t->ldt_device_nr, 1, 0) == 0) { + down_write(&lu_key_initing); + if (t->ldt_ops->ldto_start && + atomic_read(&t->ldt_device_nr) == 0) + t->ldt_ops->ldto_start(t); + atomic_inc(&t->ldt_device_nr); + up_write(&lu_key_initing); + } memset(d, 0, sizeof(*d)); atomic_set(&d->ld_ref, 0); @@ -1358,17 +1369,6 @@ void lu_stack_fini(const struct lu_env *env, struct lu_device *top) } } -enum { - /** - * Maximal number of tld slots. - */ - LU_CONTEXT_KEY_NR = 40 -}; - -static struct lu_context_key *lu_keys[LU_CONTEXT_KEY_NR] = { NULL, }; - -static DECLARE_RWSEM(lu_key_initing); - /** * Global counter incremented whenever key is registered, unregistered, * revived or quiesced. This is used to void unnecessary calls to @@ -1442,7 +1442,7 @@ void lu_context_key_degister(struct lu_context_key *key) LASSERT(atomic_read(&key->lct_used) >= 1); LINVRNT(0 <= key->lct_index && key->lct_index < ARRAY_SIZE(lu_keys)); - lu_context_key_quiesce(key); + lu_context_key_quiesce(NULL, key); key_fini(&lu_shrink_env.le_ctx, key->lct_index); @@ -1527,13 +1527,14 @@ void lu_context_key_revive_many(struct lu_context_key *k, ...) /** * Quiescent a number of keys. */ -void lu_context_key_quiesce_many(struct lu_context_key *k, ...) +void lu_context_key_quiesce_many(struct lu_device_type *t, + struct lu_context_key *k, ...) { va_list args; va_start(args, k); do { - lu_context_key_quiesce(k); + lu_context_key_quiesce(t, k); k = va_arg(args, struct lu_context_key*); } while (k); va_end(args); @@ -1564,18 +1565,22 @@ void *lu_context_key_get(const struct lu_context *ctx, * values in "shared" contexts (like service threads), when a module owning * the key is about to be unloaded. */ -void lu_context_key_quiesce(struct lu_context_key *key) +void lu_context_key_quiesce(struct lu_device_type *t, + struct lu_context_key *key) { struct lu_context *ctx; + if (key->lct_tags & LCT_QUIESCENT) + return; + /* + * The write-lock on lu_key_initing will ensure that any + * keys_fill() which didn't see LCT_QUIESCENT will have + * finished before we call key_fini(). + */ + down_write(&lu_key_initing); if (!(key->lct_tags & LCT_QUIESCENT)) { - /* - * The write-lock on lu_key_initing will ensure that any - * keys_fill() which didn't see LCT_QUIESCENT will have - * finished before we call key_fini(). - */ - down_write(&lu_key_initing); - key->lct_tags |= LCT_QUIESCENT; + if (!t || atomic_read(&t->ldt_device_nr) == 0) + key->lct_tags |= LCT_QUIESCENT; up_write(&lu_key_initing); spin_lock(&lu_context_remembered_guard); @@ -1584,7 +1589,10 @@ void lu_context_key_quiesce(struct lu_context_key *key) key_fini(ctx, key->lct_index); } spin_unlock(&lu_context_remembered_guard); + + return; } + up_write(&lu_key_initing); } void lu_context_key_revive(struct lu_context_key *key) -- 1.8.3.1 From jsimmons at infradead.org Wed Jul 15 20:45:11 2020 From: jsimmons at infradead.org (James Simmons) Date: Wed, 15 Jul 2020 16:45:11 -0400 Subject: [lustre-devel] [PATCH 30/37] lustre: llite: annotate non-owner locking In-Reply-To: <1594845918-29027-1-git-send-email-jsimmons@infradead.org> References: <1594845918-29027-1-git-send-email-jsimmons@infradead.org> Message-ID: <1594845918-29027-31-git-send-email-jsimmons@infradead.org> From: Mr NeilBrown The lli_lsm_sem locks taken by ll_prep_md_op_data() are sometimes released by a different thread. This confuses lockdep unless we explain the situation. So use down_read_non_owner() and up_read_non_owner(). WC-bug-id: https://jira.whamcloud.com/browse/LU-9679 Lustre-commit: f34392412fe22 ("LU-9679 llite: annotate non-owner locking") Signed-off-by: Mr NeilBrown Reviewed-on: https://review.whamcloud.com/39234 Reviewed-by: Andreas Dilger Reviewed-by: James Simmons Reviewed-by: Shaun Tancheff Reviewed-by: Oleg Drokin Signed-off-by: James Simmons --- fs/lustre/llite/llite_lib.c | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-) diff --git a/fs/lustre/llite/llite_lib.c b/fs/lustre/llite/llite_lib.c index c62e182..f52d2b5 100644 --- a/fs/lustre/llite/llite_lib.c +++ b/fs/lustre/llite/llite_lib.c @@ -2783,12 +2783,12 @@ int ll_obd_statfs(struct inode *inode, void __user *arg) void ll_unlock_md_op_lsm(struct md_op_data *op_data) { if (op_data->op_mea2_sem) { - up_read(op_data->op_mea2_sem); + up_read_non_owner(op_data->op_mea2_sem); op_data->op_mea2_sem = NULL; } if (op_data->op_mea1_sem) { - up_read(op_data->op_mea1_sem); + up_read_non_owner(op_data->op_mea1_sem); op_data->op_mea1_sem = NULL; } } @@ -2823,7 +2823,7 @@ struct md_op_data *ll_prep_md_op_data(struct md_op_data *op_data, op_data->op_code = opc; if (S_ISDIR(i1->i_mode)) { - down_read(&ll_i2info(i1)->lli_lsm_sem); + down_read_non_owner(&ll_i2info(i1)->lli_lsm_sem); op_data->op_mea1_sem = &ll_i2info(i1)->lli_lsm_sem; op_data->op_mea1 = ll_i2info(i1)->lli_lsm_md; op_data->op_default_mea1 = ll_i2info(i1)->lli_default_lsm_md; @@ -2833,7 +2833,10 @@ struct md_op_data *ll_prep_md_op_data(struct md_op_data *op_data, op_data->op_fid2 = *ll_inode2fid(i2); if (S_ISDIR(i2->i_mode)) { if (i2 != i1) { - down_read(&ll_i2info(i2)->lli_lsm_sem); + /* i2 is typically a child of i1, and MUST be + * further from the root to avoid deadlocks. + */ + down_read_non_owner(&ll_i2info(i2)->lli_lsm_sem); op_data->op_mea2_sem = &ll_i2info(i2)->lli_lsm_sem; } -- 1.8.3.1 From jsimmons at infradead.org Wed Jul 15 20:44:46 2020 From: jsimmons at infradead.org (James Simmons) Date: Wed, 15 Jul 2020 16:44:46 -0400 Subject: [lustre-devel] [PATCH 05/37] lustre: ptlrpc: handle conn_hash rhashtable resize In-Reply-To: <1594845918-29027-1-git-send-email-jsimmons@infradead.org> References: <1594845918-29027-1-git-send-email-jsimmons@infradead.org> Message-ID: <1594845918-29027-6-git-send-email-jsimmons@infradead.org> The errors returned by rhashtable_lookup_get_insert_fast() of the values -ENOMEM or -EBUSY is due to the hashtable being resized. This is not fatal so retry a lookup. Fixes: ac2370ac2b ("staging: lustre: ptlrpc: convert conn_hash to rhashtable") WC-bug-id: https://jira.whamcloud.com/browse/LU-8130 Lustre-commit: 37b29a8f709aa ("LU-8130 ptlrpc: convert conn_hash to rhashtable") Signed-off-by: James Simmons Reviewed-on: https://review.whamcloud.com/33616 Reviewed-by: Shaun Tancheff Reviewed-by: Oleg Drokin --- fs/lustre/ptlrpc/connection.c | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-) diff --git a/fs/lustre/ptlrpc/connection.c b/fs/lustre/ptlrpc/connection.c index 5466755..a548d99 100644 --- a/fs/lustre/ptlrpc/connection.c +++ b/fs/lustre/ptlrpc/connection.c @@ -32,6 +32,8 @@ */ #define DEBUG_SUBSYSTEM S_RPC + +#include #include #include #include @@ -103,13 +105,21 @@ struct ptlrpc_connection * * connection. The object which exists in the hash will be * returned, otherwise NULL is returned on success. */ +try_again: conn2 = rhashtable_lookup_get_insert_fast(&conn_hash, &conn->c_hash, conn_hash_params); if (conn2) { /* insertion failed */ kfree(conn); - if (IS_ERR(conn2)) + if (IS_ERR(conn2)) { + /* hash table could be resizing. */ + if (PTR_ERR(conn2) == -ENOMEM || + PTR_ERR(conn2) == -EBUSY) { + msleep(20); + goto try_again; + } return NULL; + } conn = conn2; ptlrpc_connection_addref(conn); } -- 1.8.3.1 From jsimmons at infradead.org Wed Jul 15 20:45:06 2020 From: jsimmons at infradead.org (James Simmons) Date: Wed, 15 Jul 2020 16:45:06 -0400 Subject: [lustre-devel] [PATCH 25/37] lnet: socklnd: use need_resched() In-Reply-To: <1594845918-29027-1-git-send-email-jsimmons@infradead.org> References: <1594845918-29027-1-git-send-email-jsimmons@infradead.org> Message-ID: <1594845918-29027-26-git-send-email-jsimmons@infradead.org> From: Mr NeilBrown Rather than using a counter to decide when to drop the lock and see if we need to reshedule we can use need_resched(), which is a precise test instead of a guess. WC-bug-id: https://jira.whamcloud.com/browse/LU-12678 Lustre-commit: 3f848f85ba3d3 ("LU-12678 socklnd: use need_resched()") Signed-off-by: Mr NeilBrown Reviewed-on: https://review.whamcloud.com/39128 Reviewed-by: James Simmons Reviewed-by: Shaun Tancheff Reviewed-by: Chris Horn Reviewed-by: Oleg Drokin Signed-off-by: James Simmons --- net/lnet/klnds/socklnd/socklnd.h | 1 - net/lnet/klnds/socklnd/socklnd_cb.c | 12 +++--------- 2 files changed, 3 insertions(+), 10 deletions(-) diff --git a/net/lnet/klnds/socklnd/socklnd.h b/net/lnet/klnds/socklnd/socklnd.h index 0ac3637..0a0f0a7 100644 --- a/net/lnet/klnds/socklnd/socklnd.h +++ b/net/lnet/klnds/socklnd/socklnd.h @@ -55,7 +55,6 @@ #define SOCKNAL_NSCHEDS_HIGH (SOCKNAL_NSCHEDS << 1) #define SOCKNAL_PEER_HASH_BITS 7 /* # log2 of # of peer_ni lists */ -#define SOCKNAL_RESCHED 100 /* # scheduler loops before reschedule */ #define SOCKNAL_INSANITY_RECONN 5000 /* connd is trying on reconn infinitely */ #define SOCKNAL_ENOMEM_RETRY 1 /* seconds between retries */ diff --git a/net/lnet/klnds/socklnd/socklnd_cb.c b/net/lnet/klnds/socklnd/socklnd_cb.c index 623478c..936054ee 100644 --- a/net/lnet/klnds/socklnd/socklnd_cb.c +++ b/net/lnet/klnds/socklnd/socklnd_cb.c @@ -1328,7 +1328,6 @@ int ksocknal_scheduler(void *arg) struct ksock_conn *conn; struct ksock_tx *tx; int rc; - int nloops = 0; long id = (long)arg; sched = ksocknal_data.ksnd_schedulers[KSOCK_THREAD_CPT(id)]; @@ -1470,12 +1469,10 @@ int ksocknal_scheduler(void *arg) did_something = 1; } - if (!did_something || /* nothing to do */ - ++nloops == SOCKNAL_RESCHED) { /* hogging CPU? */ + if (!did_something || /* nothing to do */ + need_resched()) { /* hogging CPU? */ spin_unlock_bh(&sched->kss_lock); - nloops = 0; - if (!did_something) { /* wait for something to do */ rc = wait_event_interruptible_exclusive( sched->kss_waitq, @@ -2080,7 +2077,6 @@ void ksocknal_write_callback(struct ksock_conn *conn) spinlock_t *connd_lock = &ksocknal_data.ksnd_connd_lock; struct ksock_connreq *cr; wait_queue_entry_t wait; - int nloops = 0; int cons_retry = 0; init_waitqueue_entry(&wait, current); @@ -2158,10 +2154,9 @@ void ksocknal_write_callback(struct ksock_conn *conn) } if (dropped_lock) { - if (++nloops < SOCKNAL_RESCHED) + if (!need_resched()) continue; spin_unlock_bh(connd_lock); - nloops = 0; cond_resched(); spin_lock_bh(connd_lock); continue; @@ -2173,7 +2168,6 @@ void ksocknal_write_callback(struct ksock_conn *conn) &wait); spin_unlock_bh(connd_lock); - nloops = 0; schedule_timeout(timeout); remove_wait_queue(&ksocknal_data.ksnd_connd_waitq, &wait); -- 1.8.3.1 From jsimmons at infradead.org Wed Jul 15 20:45:12 2020 From: jsimmons at infradead.org (James Simmons) Date: Wed, 15 Jul 2020 16:45:12 -0400 Subject: [lustre-devel] [PATCH 31/37] lustre: osc: consume grants for direct I/O In-Reply-To: <1594845918-29027-1-git-send-email-jsimmons@infradead.org> References: <1594845918-29027-1-git-send-email-jsimmons@infradead.org> Message-ID: <1594845918-29027-32-git-send-email-jsimmons@infradead.org> From: Vladimir Saveliev New IO engine implementation lost consuming grants by direct I/O writes. That led to early emergence of out of space condition during direct I/O. The below illustrates the problem: # OSTSIZE=100000 sh llmount.sh # dd if=/dev/zero of=/mnt/lustre/file bs=4k count=100 oflag=direct dd: error writing ‘/mnt/lustre/file’: No space left on device Consume grants for direct I/O. Try to consume grants in osc_queue_sync_pages() when it is called for pages which are being writted in direct i/o. Tests are added to verify grant consumption in buffered and direct i/o and to verify direct i/o overwrite when ost is full. The overwrite test is for ldiskfs only as zfs is unable to overwrite when it is full. Cray-bug-id: LUS-7036 WC-bug-id: https://jira.whamcloud.com/browse/LU-12687 Lustre-commit: 05f326a7988a7a ("LU-12687 osc: consume grants for direct I/O") Signed-off-by: Vladimir Saveliev Reviewed-on: https://review.whamcloud.com/35896 Reviewed-by: Wang Shilong Reviewed-by: Andreas Dilger Reviewed-by: Mike Pershin Signed-off-by: James Simmons --- fs/lustre/osc/osc_cache.c | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+) diff --git a/fs/lustre/osc/osc_cache.c b/fs/lustre/osc/osc_cache.c index fe03c0d..c7aaabb 100644 --- a/fs/lustre/osc/osc_cache.c +++ b/fs/lustre/osc/osc_cache.c @@ -2692,6 +2692,28 @@ int osc_queue_sync_pages(const struct lu_env *env, const struct cl_io *io, ext->oe_srvlock = !!(brw_flags & OBD_BRW_SRVLOCK); ext->oe_ndelay = !!(brw_flags & OBD_BRW_NDELAY); ext->oe_dio = !!(brw_flags & OBD_BRW_NOCACHE); + if (ext->oe_dio && !ext->oe_rw) { /* direct io write */ + int grants; + int ppc; + + ppc = 1 << (cli->cl_chunkbits - PAGE_SHIFT); + grants = cli->cl_grant_extent_tax; + grants += (1 << cli->cl_chunkbits) * + ((page_count + ppc - 1) / ppc); + + spin_lock(&cli->cl_loi_list_lock); + if (osc_reserve_grant(cli, grants) == 0) { + list_for_each_entry(oap, list, oap_pending_item) { + osc_consume_write_grant(cli, + &oap->oap_brw_page); + atomic_long_inc(&obd_dirty_pages); + } + osc_unreserve_grant_nolock(cli, grants, 0); + ext->oe_grants = grants; + } + spin_unlock(&cli->cl_loi_list_lock); + } + ext->oe_is_rdma_only = !!(brw_flags & OBD_BRW_RDMA_ONLY); ext->oe_nr_pages = page_count; ext->oe_mppr = mppr; -- 1.8.3.1 From jsimmons at infradead.org Wed Jul 15 20:45:13 2020 From: jsimmons at infradead.org (James Simmons) Date: Wed, 15 Jul 2020 16:45:13 -0400 Subject: [lustre-devel] [PATCH 32/37] lnet: remove LNetMEUnlink and clean up related code In-Reply-To: <1594845918-29027-1-git-send-email-jsimmons@infradead.org> References: <1594845918-29027-1-git-send-email-jsimmons@infradead.org> Message-ID: <1594845918-29027-33-git-send-email-jsimmons@infradead.org> From: Mr NeilBrown LNetMEUnlink is not particularly useful, and exposing it as an LNet interface only provides the opportunity for it to be misused. Every successful call to LNetMEAttach() is followed by a call to LNetMDAttach(). If that call succeeds, the ME is owned by the MD and the caller mustn't touch it again. If the call fails, the caller is currently required to call LNetMEUnlink(), which all callers do, and these are the only places that LNetMEUnlink() are called. As LNetMDAttach() knows when it will fail, it can unlink the ME itself and save the caller the effort. This allows LNetMEUnlink() to be removed which simplifies the LNet interface. LNetMEUnlink() is also used in in ptl_send_rpc() in a situation where ptl_send_buf() fails. In this case both the ME and the MD need to be unlinked, as as they are interconnected, LNetMEUnlink() or LNetMDUnlink() can equally do the job. So change it to use LNetMDUnlink(). LNetMEUnlink() is primarily a call the lnet_me_unlink(). It also - has some handling if ->me_md is not NULL, but that is never the case - takes the lnet_res_lock(). However LNetMDAttach() already takes that lock. So none of this functionality is useful to LNetMDAttach(). On failure, it can call lnet_me_unlink() directly while ensuring it still has the lock. This patch: - moves the calls to lnet_md_validate() into lnet_md_build() - changes LNetMDAttach() to always take the lnet_res_lock(), and to call lnet_me_unlink() on failure. - removes all calls to LNetMEUnlink() and sometimes simplifies surrounding code. - changes lnet_md_link() to 'void' as it only ever returns '0', and thus simplify error handling in LNetMDAttach() and LNetMDBind() WC-bug-id: https://jira.whamcloud.com/browse/LU-12678 Lustre-commit: e17ee2296c201 ("LU-12678 lnet: remove LNetMEUnlink and clean up related code") Signed-off-by: Mr NeilBrown Reviewed-on: https://review.whamcloud.com/38646 Reviewed-by: Yang Sheng Reviewed-by: James Simmons Reviewed-by: Oleg Drokin Signed-off-by: James Simmons --- fs/lustre/ptlrpc/niobuf.c | 12 +++------ include/linux/lnet/api.h | 6 ++--- net/lnet/lnet/api-ni.c | 5 +--- net/lnet/lnet/lib-md.c | 62 +++++++++++++++-------------------------------- net/lnet/lnet/lib-me.c | 39 ----------------------------- net/lnet/selftest/rpc.c | 1 - 6 files changed, 26 insertions(+), 99 deletions(-) diff --git a/fs/lustre/ptlrpc/niobuf.c b/fs/lustre/ptlrpc/niobuf.c index 6fb79a2..924b9c4 100644 --- a/fs/lustre/ptlrpc/niobuf.c +++ b/fs/lustre/ptlrpc/niobuf.c @@ -203,7 +203,6 @@ static int ptlrpc_register_bulk(struct ptlrpc_request *req) CERROR("%s: LNetMDAttach failed x%llu/%d: rc = %d\n", desc->bd_import->imp_obd->obd_name, mbits, posted_md, rc); - LNetMEUnlink(me); break; } } @@ -676,7 +675,7 @@ int ptl_send_rpc(struct ptlrpc_request *request, int noreply) request->rq_receiving_reply = 0; spin_unlock(&request->rq_lock); rc = -ENOMEM; - goto cleanup_me; + goto cleanup_bulk; } percpu_ref_get(&ptlrpc_pending); @@ -720,12 +719,8 @@ int ptl_send_rpc(struct ptlrpc_request *request, int noreply) if (noreply) goto out; -cleanup_me: - /* MEUnlink is safe; the PUT didn't even get off the ground, and - * nobody apart from the PUT's target has the right nid+XID to - * access the reply buffer. - */ - LNetMEUnlink(reply_me); + LNetMDUnlink(request->rq_reply_md_h); + /* UNLINKED callback called synchronously */ LASSERT(!request->rq_receiving_reply); @@ -802,7 +797,6 @@ int ptlrpc_register_rqbd(struct ptlrpc_request_buffer_desc *rqbd) CERROR("ptlrpc: LNetMDAttach failed: rc = %d\n", rc); LASSERT(rc == -ENOMEM); - LNetMEUnlink(me); rqbd->rqbd_refcount = 0; return -ENOMEM; diff --git a/include/linux/lnet/api.h b/include/linux/lnet/api.h index 24115eb..95805de 100644 --- a/include/linux/lnet/api.h +++ b/include/linux/lnet/api.h @@ -90,8 +90,8 @@ * list is a chain of MEs. Each ME includes a pointer to a memory descriptor * and a set of match criteria. The match criteria can be used to reject * incoming requests based on process ID or the match bits provided in the - * request. MEs can be dynamically inserted into a match list by LNetMEAttach() - * and removed from its list by LNetMEUnlink(). + * request. MEs can be dynamically inserted into a match list by LNetMEAttach(), + * and must then be attached to an MD with LNetMDAttach(). * @{ */ struct lnet_me * @@ -101,8 +101,6 @@ struct lnet_me * u64 ignore_bits_in, enum lnet_unlink unlink_in, enum lnet_ins_pos pos_in); - -void LNetMEUnlink(struct lnet_me *current_in); /** @} lnet_me */ /** \defgroup lnet_md Memory descriptors diff --git a/net/lnet/lnet/api-ni.c b/net/lnet/lnet/api-ni.c index 3e69435..5f35468 100644 --- a/net/lnet/lnet/api-ni.c +++ b/net/lnet/lnet/api-ni.c @@ -1645,14 +1645,12 @@ struct lnet_ping_buffer * rc = LNetMDAttach(me, &md, LNET_RETAIN, ping_mdh); if (rc) { CERROR("Can't attach ping target MD: %d\n", rc); - goto fail_unlink_ping_me; + goto fail_decref_ping_buffer; } lnet_ping_buffer_addref(*ppbuf); return 0; -fail_unlink_ping_me: - LNetMEUnlink(me); fail_decref_ping_buffer: LASSERT(atomic_read(&(*ppbuf)->pb_refcnt) == 1); lnet_ping_buffer_decref(*ppbuf); @@ -1855,7 +1853,6 @@ int lnet_push_target_post(struct lnet_ping_buffer *pbuf, rc = LNetMDAttach(me, &md, LNET_UNLINK, mdhp); if (rc) { CERROR("Can't attach push MD: %d\n", rc); - LNetMEUnlink(me); lnet_ping_buffer_decref(pbuf); pbuf->pb_needs_post = true; return rc; diff --git a/net/lnet/lnet/lib-md.c b/net/lnet/lnet/lib-md.c index e80dc6f..48249f3 100644 --- a/net/lnet/lnet/lib-md.c +++ b/net/lnet/lnet/lib-md.c @@ -123,6 +123,8 @@ int lnet_cpt_of_md(struct lnet_libmd *md, unsigned int offset) return cpt; } +static int lnet_md_validate(const struct lnet_md *umd); + static struct lnet_libmd * lnet_md_build(const struct lnet_md *umd, int unlink) { @@ -132,6 +134,9 @@ int lnet_cpt_of_md(struct lnet_libmd *md, unsigned int offset) struct lnet_libmd *lmd; unsigned int size; + if (lnet_md_validate(umd) != 0) + return ERR_PTR(-EINVAL); + if (umd->options & LNET_MD_KIOV) niov = umd->length; else @@ -228,15 +233,14 @@ int lnet_cpt_of_md(struct lnet_libmd *md, unsigned int offset) } /* must be called with resource lock held */ -static int +static void lnet_md_link(struct lnet_libmd *md, lnet_handler_t handler, int cpt) { struct lnet_res_container *container = the_lnet.ln_md_containers[cpt]; /* * NB we are passed an allocated, but inactive md. - * if we return success, caller may lnet_md_unlink() it. - * otherwise caller may only kfree() it. + * Caller may lnet_md_unlink() it, or may lnet_md_free() it. */ /* * This implementation doesn't know how to create START events or @@ -255,8 +259,6 @@ int lnet_cpt_of_md(struct lnet_libmd *md, unsigned int offset) LASSERT(list_empty(&md->md_list)); list_add(&md->md_list, &container->rec_active); - - return 0; } /* must be called with lnet_res_lock held */ @@ -304,14 +306,11 @@ int lnet_cpt_of_md(struct lnet_libmd *md, unsigned int offset) * @handle On successful returns, a handle to the newly created MD is * saved here. This handle can be used later in LNetMDUnlink(). * + * The ME will either be linked to the new MD, or it will be freed. + * * Return: 0 on success. * -EINVAL If @umd is not valid. * -ENOMEM If new MD cannot be allocated. - * -ENOENT Either @me or @umd.handle does not point to a - * valid object. Note that it's OK to supply a NULL @umd.handle - * by calling LNetInvalidateHandle() on it. - * -EBUSY if the ME pointed to by @me is already associated with - * a MD. */ int LNetMDAttach(struct lnet_me *me, const struct lnet_md *umd, @@ -321,33 +320,27 @@ int lnet_cpt_of_md(struct lnet_libmd *md, unsigned int offset) LIST_HEAD(drops); struct lnet_libmd *md; int cpt; - int rc; LASSERT(the_lnet.ln_refcount > 0); - if (lnet_md_validate(umd)) - return -EINVAL; + LASSERT(!me->me_md); if (!(umd->options & (LNET_MD_OP_GET | LNET_MD_OP_PUT))) { CERROR("Invalid option: no MD_OP set\n"); - return -EINVAL; - } - - md = lnet_md_build(umd, unlink); - if (IS_ERR(md)) - return PTR_ERR(md); + md = ERR_PTR(-EINVAL); + } else + md = lnet_md_build(umd, unlink); cpt = me->me_cpt; - lnet_res_lock(cpt); - if (me->me_md) - rc = -EBUSY; - else - rc = lnet_md_link(md, umd->handler, cpt); + if (IS_ERR(md)) { + lnet_me_unlink(me); + lnet_res_unlock(cpt); + return PTR_ERR(md); + } - if (rc) - goto out_unlock; + lnet_md_link(md, umd->handler, cpt); /* * attach this MD to portal of ME and check if it matches any @@ -363,11 +356,6 @@ int lnet_cpt_of_md(struct lnet_libmd *md, unsigned int offset) lnet_recv_delayed_msg_list(&matches); return 0; - -out_unlock: - lnet_res_unlock(cpt); - kfree(md); - return rc; } EXPORT_SYMBOL(LNetMDAttach); @@ -383,9 +371,6 @@ int lnet_cpt_of_md(struct lnet_libmd *md, unsigned int offset) * Return: 0 On success. * -EINVAL If @umd is not valid. * -ENOMEM If new MD cannot be allocated. - * -ENOENT @umd.handle does not point to a valid EQ. - * Note that it's OK to supply a NULL @umd.handle by - * calling LNetInvalidateHandle() on it. */ int LNetMDBind(const struct lnet_md *umd, enum lnet_unlink unlink, @@ -397,9 +382,6 @@ int lnet_cpt_of_md(struct lnet_libmd *md, unsigned int offset) LASSERT(the_lnet.ln_refcount > 0); - if (lnet_md_validate(umd)) - return -EINVAL; - if ((umd->options & (LNET_MD_OP_GET | LNET_MD_OP_PUT))) { CERROR("Invalid option: GET|PUT illegal on active MDs\n"); return -EINVAL; @@ -418,17 +400,13 @@ int lnet_cpt_of_md(struct lnet_libmd *md, unsigned int offset) cpt = lnet_res_lock_current(); - rc = lnet_md_link(md, umd->handler, cpt); - if (rc) - goto out_unlock; + lnet_md_link(md, umd->handler, cpt); lnet_md2handle(handle, md); lnet_res_unlock(cpt); return 0; -out_unlock: - lnet_res_unlock(cpt); out_free: kfree(md); diff --git a/net/lnet/lnet/lib-me.c b/net/lnet/lnet/lib-me.c index 14ab21f..f75f3cb 100644 --- a/net/lnet/lnet/lib-me.c +++ b/net/lnet/lnet/lib-me.c @@ -118,45 +118,6 @@ struct lnet_me * } EXPORT_SYMBOL(LNetMEAttach); -/** - * Unlink a match entry from its match list. - * - * This operation also releases any resources associated with the ME. If a - * memory descriptor is attached to the ME, then it will be unlinked as well - * and an unlink event will be generated. It is an error to use the ME handle - * after calling LNetMEUnlink(). - * - * @me The ME to be unlinked. - * - * \see LNetMDUnlink() for the discussion on delivering unlink event. - */ -void -LNetMEUnlink(struct lnet_me *me) -{ - struct lnet_libmd *md; - struct lnet_event ev; - int cpt; - - LASSERT(the_lnet.ln_refcount > 0); - - cpt = me->me_cpt; - lnet_res_lock(cpt); - - md = me->me_md; - if (md) { - md->md_flags |= LNET_MD_FLAG_ABORTED; - if (md->md_handler && !md->md_refcount) { - lnet_build_unlink_event(md, &ev); - md->md_handler(&ev); - } - } - - lnet_me_unlink(me); - - lnet_res_unlock(cpt); -} -EXPORT_SYMBOL(LNetMEUnlink); - /* call with lnet_res_lock please */ void lnet_me_unlink(struct lnet_me *me) diff --git a/net/lnet/selftest/rpc.c b/net/lnet/selftest/rpc.c index 799ad99..a72e485 100644 --- a/net/lnet/selftest/rpc.c +++ b/net/lnet/selftest/rpc.c @@ -383,7 +383,6 @@ struct srpc_bulk * CERROR("LNetMDAttach failed: %d\n", rc); LASSERT(rc == -ENOMEM); - LNetMEUnlink(me); return -ENOMEM; } -- 1.8.3.1 From jsimmons at infradead.org Wed Jul 15 20:44:57 2020 From: jsimmons at infradead.org (James Simmons) Date: Wed, 15 Jul 2020 16:44:57 -0400 Subject: [lustre-devel] [PATCH 16/37] lustre: obdclass: re-declare cl_page variables to reduce its size In-Reply-To: <1594845918-29027-1-git-send-email-jsimmons@infradead.org> References: <1594845918-29027-1-git-send-email-jsimmons@infradead.org> Message-ID: <1594845918-29027-17-git-send-email-jsimmons@infradead.org> From: Wang Shilong With following changes: 1) make CPS_CACHED declare start from 1 consistent with CPT_CACHED 2) add CPT_NR to indicate max allowed CPT state value. 3) Reserve 4 bits for @cp_state which allow 15 kind of states 4) Reserve 2 bits for @cp_type which allow 3 kinds of cl_page types 5) use short int for @cp_kmem_index and We still have another 16 bits reserved for future extension. 6) move @cp_lov_index after @cp_ref to fill 4 bytes hole. After this patch, cl_page size could reduce from 336 bytes to 320 bytes WC-bug-id: https://jira.whamcloud.com/browse/LU-13134 Lustre-commit: 5fb29cd1e77ca ("LU-13134 obdclass: re-declare cl_page variables to reduce its size") Signed-off-by: Wang Shilong Reviewed-on: https://review.whamcloud.com/37480 Reviewed-by: Andreas Dilger Reviewed-by: James Simmons Reviewed-by: Oleg Drokin Signed-off-by: James Simmons --- fs/lustre/include/cl_object.h | 26 +++++++++------ fs/lustre/obdclass/cl_page.c | 76 +++++++++++++++++++++---------------------- 2 files changed, 53 insertions(+), 49 deletions(-) diff --git a/fs/lustre/include/cl_object.h b/fs/lustre/include/cl_object.h index 47997f8..8611285 100644 --- a/fs/lustre/include/cl_object.h +++ b/fs/lustre/include/cl_object.h @@ -621,7 +621,7 @@ enum cl_page_state { * * \invariant cl_page::cp_owner == NULL && cl_page::cp_req == NULL */ - CPS_CACHED, + CPS_CACHED = 1, /** * Page is exclusively owned by some cl_io. Page may end up in this * state as a result of @@ -715,8 +715,13 @@ enum cl_page_type { * it is used in DirectIO and lockless IO. */ CPT_TRANSIENT, + CPT_NR }; +#define CP_STATE_BITS 4 +#define CP_TYPE_BITS 2 +#define CP_MAX_LAYER 3 + /** * Fields are protected by the lock on struct page, except for atomics and * immutables. @@ -729,8 +734,9 @@ enum cl_page_type { struct cl_page { /** Reference counter. */ refcount_t cp_ref; - /* which slab kmem index this memory allocated from */ - int cp_kmem_index; + /** layout_entry + stripe index, composed using lov_comp_index() */ + unsigned int cp_lov_index; + pgoff_t cp_osc_index; /** An object this page is a part of. Immutable after creation. */ struct cl_object *cp_obj; /** vmpage */ @@ -738,19 +744,22 @@ struct cl_page { /** Linkage of pages within group. Pages must be owned */ struct list_head cp_batch; /** array of slices offset. Immutable after creation. */ - unsigned char cp_layer_offset[3]; + unsigned char cp_layer_offset[CP_MAX_LAYER]; /* 24 bits */ /** current slice index */ - unsigned char cp_layer_count:2; + unsigned char cp_layer_count:2; /* 26 bits */ /** * Page state. This field is const to avoid accidental update, it is * modified only internally within cl_page.c. Protected by a VM lock. */ - const enum cl_page_state cp_state; + enum cl_page_state cp_state:CP_STATE_BITS; /* 30 bits */ /** * Page type. Only CPT_TRANSIENT is used so far. Immutable after * creation. */ - enum cl_page_type cp_type; + enum cl_page_type cp_type:CP_TYPE_BITS; /* 32 bits */ + /* which slab kmem index this memory allocated from */ + short int cp_kmem_index; /* 48 bits */ + unsigned int cp_unused1:16; /* 64 bits */ /** * Owning IO in cl_page_state::CPS_OWNED state. Sub-page can be owned @@ -765,9 +774,6 @@ struct cl_page { struct lu_ref_link cp_queue_ref; /** Assigned if doing a sync_io */ struct cl_sync_io *cp_sync_io; - /** layout_entry + stripe index, composed using lov_comp_index() */ - unsigned int cp_lov_index; - pgoff_t cp_osc_index; }; /** diff --git a/fs/lustre/obdclass/cl_page.c b/fs/lustre/obdclass/cl_page.c index cced026..53f88a7 100644 --- a/fs/lustre/obdclass/cl_page.c +++ b/fs/lustre/obdclass/cl_page.c @@ -153,17 +153,6 @@ static void cl_page_free(const struct lu_env *env, struct cl_page *cl_page, __cl_page_free(cl_page, bufsize); } -/** - * Helper function updating page state. This is the only place in the code - * where cl_page::cp_state field is mutated. - */ -static inline void cl_page_state_set_trust(struct cl_page *page, - enum cl_page_state state) -{ - /* bypass const. */ - *(enum cl_page_state *)&page->cp_state = state; -} - static struct cl_page *__cl_page_alloc(struct cl_object *o) { int i = 0; @@ -217,44 +206,50 @@ static struct cl_page *__cl_page_alloc(struct cl_object *o) return cl_page; } -struct cl_page *cl_page_alloc(const struct lu_env *env, - struct cl_object *o, pgoff_t ind, - struct page *vmpage, +struct cl_page *cl_page_alloc(const struct lu_env *env, struct cl_object *o, + pgoff_t ind, struct page *vmpage, enum cl_page_type type) { - struct cl_page *page; + struct cl_page *cl_page; struct cl_object *o2; - page = __cl_page_alloc(o); - if (page) { + cl_page = __cl_page_alloc(o); + if (cl_page) { int result = 0; - refcount_set(&page->cp_ref, 1); - page->cp_obj = o; + /* + * Please fix cl_page:cp_state/type declaration if + * these assertions fail in the future. + */ + BUILD_BUG_ON((1 << CP_STATE_BITS) < CPS_NR); /* cp_state */ + BUILD_BUG_ON((1 << CP_TYPE_BITS) < CPT_NR); /* cp_type */ + refcount_set(&cl_page->cp_ref, 1); + cl_page->cp_obj = o; cl_object_get(o); - lu_object_ref_add_at(&o->co_lu, &page->cp_obj_ref, "cl_page", - page); - page->cp_vmpage = vmpage; - cl_page_state_set_trust(page, CPS_CACHED); - page->cp_type = type; - INIT_LIST_HEAD(&page->cp_batch); - lu_ref_init(&page->cp_reference); + lu_object_ref_add_at(&o->co_lu, &cl_page->cp_obj_ref, + "cl_page", cl_page); + cl_page->cp_vmpage = vmpage; + cl_page->cp_state = CPS_CACHED; + cl_page->cp_type = type; + INIT_LIST_HEAD(&cl_page->cp_batch); + lu_ref_init(&cl_page->cp_reference); cl_object_for_each(o2, o) { if (o2->co_ops->coo_page_init) { result = o2->co_ops->coo_page_init(env, o2, - page, ind); + cl_page, + ind); if (result != 0) { - __cl_page_delete(env, page); - cl_page_free(env, page, NULL); - page = ERR_PTR(result); + __cl_page_delete(env, cl_page); + cl_page_free(env, cl_page, NULL); + cl_page = ERR_PTR(result); break; } } } } else { - page = ERR_PTR(-ENOMEM); + cl_page = ERR_PTR(-ENOMEM); } - return page; + return cl_page; } /** @@ -317,7 +312,8 @@ static inline int cl_page_invariant(const struct cl_page *pg) } static void __cl_page_state_set(const struct lu_env *env, - struct cl_page *page, enum cl_page_state state) + struct cl_page *cl_page, + enum cl_page_state state) { enum cl_page_state old; @@ -363,12 +359,13 @@ static void __cl_page_state_set(const struct lu_env *env, } }; - old = page->cp_state; - PASSERT(env, page, allowed_transitions[old][state]); - CL_PAGE_HEADER(D_TRACE, env, page, "%d -> %d\n", old, state); - PASSERT(env, page, page->cp_state == old); - PASSERT(env, page, equi(state == CPS_OWNED, page->cp_owner)); - cl_page_state_set_trust(page, state); + old = cl_page->cp_state; + PASSERT(env, cl_page, allowed_transitions[old][state]); + CL_PAGE_HEADER(D_TRACE, env, cl_page, "%d -> %d\n", old, state); + PASSERT(env, cl_page, cl_page->cp_state == old); + PASSERT(env, cl_page, equi(state == CPS_OWNED, + cl_page->cp_owner)); + cl_page->cp_state = state; } static void cl_page_state_set(const struct lu_env *env, @@ -1079,6 +1076,7 @@ void cl_page_slice_add(struct cl_page *cl_page, struct cl_page_slice *slice, unsigned int offset = (char *)slice - ((char *)cl_page + sizeof(*cl_page)); + LASSERT(cl_page->cp_layer_count < CP_MAX_LAYER); LASSERT(offset < (1 << sizeof(cl_page->cp_layer_offset[0]) * 8)); cl_page->cp_layer_offset[cl_page->cp_layer_count++] = offset; slice->cpl_obj = obj; -- 1.8.3.1 From jsimmons at infradead.org Wed Jul 15 20:44:43 2020 From: jsimmons at infradead.org (James Simmons) Date: Wed, 15 Jul 2020 16:44:43 -0400 Subject: [lustre-devel] [PATCH 02/37] lustre: ldlm: check slv and limit before updating In-Reply-To: <1594845918-29027-1-git-send-email-jsimmons@infradead.org> References: <1594845918-29027-1-git-send-email-jsimmons@infradead.org> Message-ID: <1594845918-29027-3-git-send-email-jsimmons@infradead.org> From: Wang Shilong slv and limit do not change for most of time, ldlm_cli_update_pool() could be called for each RPC reply, try hold read lock to check firstly could avoid heavy write lock in hot path. WC-bug-id: https://jira.whamcloud.com/browse/LU-13365 Lustre-commit: 3116b9e19dc09 ("LU-13365 ldlm: check slv and limit before updating") Signed-off-by: Wang Shilong Reviewed-on: https://review.whamcloud.com/37969 Reviewed-by: Andreas Dilger Reviewed-by: Mike Pershin Signed-off-by: James Simmons --- fs/lustre/ldlm/ldlm_request.c | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/fs/lustre/ldlm/ldlm_request.c b/fs/lustre/ldlm/ldlm_request.c index e1ba596..6318137 100644 --- a/fs/lustre/ldlm/ldlm_request.c +++ b/fs/lustre/ldlm/ldlm_request.c @@ -1163,6 +1163,14 @@ int ldlm_cli_update_pool(struct ptlrpc_request *req) new_slv = lustre_msg_get_slv(req->rq_repmsg); obd = req->rq_import->imp_obd; + read_lock(&obd->obd_pool_lock); + if (obd->obd_pool_slv == new_slv && + obd->obd_pool_limit == new_limit) { + read_unlock(&obd->obd_pool_lock); + return 0; + } + read_unlock(&obd->obd_pool_lock); + /* * Set new SLV and limit in OBD fields to make them accessible * to the pool thread. We do not access obd_namespace and pool -- 1.8.3.1 From jsimmons at infradead.org Wed Jul 15 20:45:14 2020 From: jsimmons at infradead.org (James Simmons) Date: Wed, 15 Jul 2020 16:45:14 -0400 Subject: [lustre-devel] [PATCH 33/37] lnet: Set remote NI status in lnet_notify In-Reply-To: <1594845918-29027-1-git-send-email-jsimmons@infradead.org> References: <1594845918-29027-1-git-send-email-jsimmons@infradead.org> Message-ID: <1594845918-29027-34-git-send-email-jsimmons@infradead.org> From: Chris Horn The gnilnd receives node health information asynchronous from any tx failure, so aliveness of lpni as reported by lnet_is_peer_ni_alive() may not match what LND is telling us. Use existing reset flag to set cached NI status down so we can be sure that remote NIs are correctly set down. HPE-bug-id: LUS-8897 WC-bug-id: https://jira.whamcloud.com/browse/LU-13648 Lustre-commit: 8010dbb660766 ("LU-13648 lnet: Set remote NI status in lnet_notify") Signed-off-by: Chris Horn Reviewed-on: https://review.whamcloud.com/38862 Reviewed-by: Amir Shehata Reviewed-by: Serguei Smirnov Reviewed-by: Oleg Drokin Signed-off-by: James Simmons --- net/lnet/lnet/router.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/net/lnet/lnet/router.c b/net/lnet/lnet/router.c index c0578d9..e3b3e71 100644 --- a/net/lnet/lnet/router.c +++ b/net/lnet/lnet/router.c @@ -1671,8 +1671,7 @@ bool lnet_router_checker_active(void) CDEBUG(D_NET, "%s notifying %s: %s\n", !ni ? "userspace" : libcfs_nid2str(ni->ni_nid), - libcfs_nid2str(nid), - alive ? "up" : "down"); + libcfs_nid2str(nid), alive ? "up" : "down"); if (ni && LNET_NIDNET(ni->ni_nid) != LNET_NIDNET(nid)) { @@ -1714,6 +1713,7 @@ bool lnet_router_checker_active(void) if (alive) { if (reset) { + lpni->lpni_ns_status = LNET_NI_STATUS_UP; lnet_set_lpni_healthv_locked(lpni, LNET_MAX_HEALTH_VALUE); } else { @@ -1726,6 +1726,8 @@ bool lnet_router_checker_active(void) (sensitivity) ? sensitivity : lnet_health_sensitivity); } + } else if (reset) { + lpni->lpni_ns_status = LNET_NI_STATUS_DOWN; } /* recalculate aliveness */ -- 1.8.3.1 From jsimmons at infradead.org Wed Jul 15 20:45:16 2020 From: jsimmons at infradead.org (James Simmons) Date: Wed, 15 Jul 2020 16:45:16 -0400 Subject: [lustre-devel] [PATCH 35/37] lustre: llite: fix short io for AIO In-Reply-To: <1594845918-29027-1-git-send-email-jsimmons@infradead.org> References: <1594845918-29027-1-git-send-email-jsimmons@infradead.org> Message-ID: <1594845918-29027-36-git-send-email-jsimmons@infradead.org> From: Wang Shilong The problem is currently AIO could not handle i/o size > stripe size: We need cl io loop to handle io across stripes, since -EIOCBQUEUED is returned for AIO, io loop will be stopped thus short io happen. The patch try to fix the problem by making IO engine aware of special error, and it will be proceed to finish all IO requests. Fixes: fde7ac1942f5 ("lustre: clio: AIO support for direct IO") WC-bug-id: https://jira.whamcloud.com/browse/LU-13697 Lustre-commit: 84c3e85ced2dd ("LU-13697 llite: fix short io for AIO") Signed-off-by: Wang Shilong Reviewed-on: https://review.whamcloud.com/39104 Reviewed-by: Andreas Dilger Reviewed-by: Bobi Jam Reviewed-by: Oleg Drokin Signed-off-by: James Simmons --- fs/lustre/include/cl_object.h | 2 ++ fs/lustre/llite/file.c | 32 +++++++++++++++++- fs/lustre/llite/rw26.c | 43 +++++++++++++++++-------- fs/lustre/llite/vvp_internal.h | 3 +- fs/lustre/llite/vvp_io.c | 73 ++++++++++++++++++++++++++++-------------- fs/lustre/obdclass/cl_io.c | 9 +++++- 6 files changed, 122 insertions(+), 40 deletions(-) diff --git a/fs/lustre/include/cl_object.h b/fs/lustre/include/cl_object.h index e656c68..e849f23 100644 --- a/fs/lustre/include/cl_object.h +++ b/fs/lustre/include/cl_object.h @@ -1814,6 +1814,8 @@ struct cl_io { enum cl_io_state ci_state; /** main object this io is against. Immutable after creation. */ struct cl_object *ci_obj; + /** one AIO request might be split in cl_io_loop */ + struct cl_dio_aio *ci_aio; /** * Upper layer io, of which this io is a part of. Immutable after * creation. diff --git a/fs/lustre/llite/file.c b/fs/lustre/llite/file.c index 1849229..757950f 100644 --- a/fs/lustre/llite/file.c +++ b/fs/lustre/llite/file.c @@ -1514,6 +1514,7 @@ static void ll_heat_add(struct inode *inode, enum cl_io_type iot, int rc = 0; unsigned int retried = 0; unsigned int ignore_lockless = 0; + bool is_aio = false; CDEBUG(D_VFSTRACE, "file: %pD, type: %d ppos: %llu, count: %zu\n", file, iot, *ppos, count); @@ -1536,6 +1537,15 @@ static void ll_heat_add(struct inode *inode, enum cl_io_type iot, vio->vui_fd = file->private_data; vio->vui_iter = args->u.normal.via_iter; vio->vui_iocb = args->u.normal.via_iocb; + if (file->f_flags & O_DIRECT) { + if (!is_sync_kiocb(vio->vui_iocb)) + is_aio = true; + io->ci_aio = cl_aio_alloc(vio->vui_iocb); + if (!io->ci_aio) { + rc = -ENOMEM; + goto out; + } + } /* * Direct IO reads must also take range lock, * or multiple reads will try to work on the same pages @@ -1567,7 +1577,14 @@ static void ll_heat_add(struct inode *inode, enum cl_io_type iot, rc = io->ci_result; } - if (io->ci_nob > 0) { + /* + * In order to move forward AIO, ci_nob was increased, + * but that doesn't mean io have been finished, it just + * means io have been submited, we will always return + * EIOCBQUEUED to the caller, So we could only return + * number of bytes in non-AIO case. + */ + if (io->ci_nob > 0 && !is_aio) { result += io->ci_nob; count -= io->ci_nob; *ppos = io->u.ci_wr.wr.crw_pos; @@ -1577,6 +1594,19 @@ static void ll_heat_add(struct inode *inode, enum cl_io_type iot, args->u.normal.via_iter = vio->vui_iter; } out: + if (io->ci_aio) { + /** + * Drop one extra reference so that end_io() could be + * called for this IO context, we could call it after + * we make sure all AIO requests have been proceed. + */ + cl_sync_io_note(env, &io->ci_aio->cda_sync, + rc == -EIOCBQUEUED ? 0 : rc); + if (!is_aio) { + cl_aio_free(io->ci_aio); + io->ci_aio = NULL; + } + } cl_io_fini(env, io); CDEBUG(D_VFSTRACE, diff --git a/fs/lustre/llite/rw26.c b/fs/lustre/llite/rw26.c index d0e3ff6..b3802cf 100644 --- a/fs/lustre/llite/rw26.c +++ b/fs/lustre/llite/rw26.c @@ -290,6 +290,7 @@ static ssize_t ll_direct_IO(struct kiocb *iocb, struct iov_iter *iter) ssize_t tot_bytes = 0, result = 0; loff_t file_offset = iocb->ki_pos; int rw = iov_iter_rw(iter); + struct vvp_io *vio; /* if file is encrypted, return 0 so that we fall back to buffered IO */ if (IS_ENCRYPTED(inode)) @@ -319,12 +320,13 @@ static ssize_t ll_direct_IO(struct kiocb *iocb, struct iov_iter *iter) env = lcc->lcc_env; LASSERT(!IS_ERR(env)); + vio = vvp_env_io(env); io = lcc->lcc_io; LASSERT(io); - aio = cl_aio_alloc(iocb); - if (!aio) - return -ENOMEM; + aio = io->ci_aio; + LASSERT(aio); + LASSERT(aio->cda_iocb == iocb); /* 0. Need locking between buffered and direct access. and race with * size changing by concurrent truncates and writes. @@ -368,24 +370,39 @@ static ssize_t ll_direct_IO(struct kiocb *iocb, struct iov_iter *iter) } out: - aio->cda_bytes = tot_bytes; - cl_sync_io_note(env, &aio->cda_sync, result); + aio->cda_bytes += tot_bytes; if (is_sync_kiocb(iocb)) { + struct cl_sync_io *anchor = &aio->cda_sync; ssize_t rc2; - rc2 = cl_sync_io_wait(env, &aio->cda_sync, 0); + /** + * @anchor was inited as 1 to prevent end_io to be + * called before we add all pages for IO, so drop + * one extra reference to make sure we could wait + * count to be zero. + */ + cl_sync_io_note(env, anchor, result); + + rc2 = cl_sync_io_wait(env, anchor, 0); if (result == 0 && rc2) result = rc2; + /** + * One extra reference again, as if @anchor is + * reused we assume it as 1 before using. + */ + atomic_add(1, &anchor->csi_sync_nr); if (result == 0) { - struct vvp_io *vio = vvp_env_io(env); /* no commit async for direct IO */ - vio->u.write.vui_written += tot_bytes; + vio->u.readwrite.vui_written += tot_bytes; result = tot_bytes; } - cl_aio_free(aio); } else { + if (rw == WRITE) + vio->u.readwrite.vui_written += tot_bytes; + else + vio->u.readwrite.vui_read += tot_bytes; result = -EIOCBQUEUED; } @@ -523,7 +540,7 @@ static int ll_write_begin(struct file *file, struct address_space *mapping, vmpage = grab_cache_page_nowait(mapping, index); if (unlikely(!vmpage || PageDirty(vmpage) || PageWriteback(vmpage))) { struct vvp_io *vio = vvp_env_io(env); - struct cl_page_list *plist = &vio->u.write.vui_queue; + struct cl_page_list *plist = &vio->u.readwrite.vui_queue; /* if the page is already in dirty cache, we have to commit * the pages right now; otherwise, it may cause deadlock @@ -685,17 +702,17 @@ static int ll_write_end(struct file *file, struct address_space *mapping, LASSERT(cl_page_is_owned(page, io)); if (copied > 0) { - struct cl_page_list *plist = &vio->u.write.vui_queue; + struct cl_page_list *plist = &vio->u.readwrite.vui_queue; lcc->lcc_page = NULL; /* page will be queued */ /* Add it into write queue */ cl_page_list_add(plist, page); if (plist->pl_nr == 1) /* first page */ - vio->u.write.vui_from = from; + vio->u.readwrite.vui_from = from; else LASSERT(from == 0); - vio->u.write.vui_to = from + copied; + vio->u.readwrite.vui_to = from + copied; /* * To address the deadlock in balance_dirty_pages() where diff --git a/fs/lustre/llite/vvp_internal.h b/fs/lustre/llite/vvp_internal.h index cff85ea..6956d6b 100644 --- a/fs/lustre/llite/vvp_internal.h +++ b/fs/lustre/llite/vvp_internal.h @@ -88,9 +88,10 @@ struct vvp_io { struct { struct cl_page_list vui_queue; unsigned long vui_written; + unsigned long vui_read; int vui_from; int vui_to; - } write; + } readwrite; /* normal io */ } u; /** diff --git a/fs/lustre/llite/vvp_io.c b/fs/lustre/llite/vvp_io.c index c3fb03a..59da56d 100644 --- a/fs/lustre/llite/vvp_io.c +++ b/fs/lustre/llite/vvp_io.c @@ -249,10 +249,20 @@ static int vvp_io_write_iter_init(const struct lu_env *env, { struct vvp_io *vio = cl2vvp_io(env, ios); - cl_page_list_init(&vio->u.write.vui_queue); - vio->u.write.vui_written = 0; - vio->u.write.vui_from = 0; - vio->u.write.vui_to = PAGE_SIZE; + cl_page_list_init(&vio->u.readwrite.vui_queue); + vio->u.readwrite.vui_written = 0; + vio->u.readwrite.vui_from = 0; + vio->u.readwrite.vui_to = PAGE_SIZE; + + return 0; +} + +static int vvp_io_read_iter_init(const struct lu_env *env, + const struct cl_io_slice *ios) +{ + struct vvp_io *vio = cl2vvp_io(env, ios); + + vio->u.readwrite.vui_read = 0; return 0; } @@ -262,7 +272,7 @@ static void vvp_io_write_iter_fini(const struct lu_env *env, { struct vvp_io *vio = cl2vvp_io(env, ios); - LASSERT(vio->u.write.vui_queue.pl_nr == 0); + LASSERT(vio->u.readwrite.vui_queue.pl_nr == 0); } static int vvp_io_fault_iter_init(const struct lu_env *env, @@ -824,7 +834,13 @@ static int vvp_io_read_start(const struct lu_env *env, io->ci_continue = 0; io->ci_nob += result; result = 0; + } else if (result == -EIOCBQUEUED) { + io->ci_nob += vio->u.readwrite.vui_read; + if (vio->vui_iocb) + vio->vui_iocb->ki_pos = pos + + vio->u.readwrite.vui_read; } + return result; } @@ -1017,23 +1033,24 @@ int vvp_io_write_commit(const struct lu_env *env, struct cl_io *io) struct cl_object *obj = io->ci_obj; struct inode *inode = vvp_object_inode(obj); struct vvp_io *vio = vvp_env_io(env); - struct cl_page_list *queue = &vio->u.write.vui_queue; + struct cl_page_list *queue = &vio->u.readwrite.vui_queue; struct cl_page *page; int rc = 0; int bytes = 0; - unsigned int npages = vio->u.write.vui_queue.pl_nr; + unsigned int npages = vio->u.readwrite.vui_queue.pl_nr; if (npages == 0) return 0; CDEBUG(D_VFSTRACE, "commit async pages: %d, from %d, to %d\n", - npages, vio->u.write.vui_from, vio->u.write.vui_to); + npages, vio->u.readwrite.vui_from, vio->u.readwrite.vui_to); LASSERT(page_list_sanity_check(obj, queue)); /* submit IO with async write */ rc = cl_io_commit_async(env, io, queue, - vio->u.write.vui_from, vio->u.write.vui_to, + vio->u.readwrite.vui_from, + vio->u.readwrite.vui_to, write_commit_callback); npages -= queue->pl_nr; /* already committed pages */ if (npages > 0) { @@ -1041,18 +1058,18 @@ int vvp_io_write_commit(const struct lu_env *env, struct cl_io *io) bytes = npages << PAGE_SHIFT; /* first page */ - bytes -= vio->u.write.vui_from; + bytes -= vio->u.readwrite.vui_from; if (queue->pl_nr == 0) /* last page */ - bytes -= PAGE_SIZE - vio->u.write.vui_to; + bytes -= PAGE_SIZE - vio->u.readwrite.vui_to; LASSERTF(bytes > 0, "bytes = %d, pages = %d\n", bytes, npages); - vio->u.write.vui_written += bytes; + vio->u.readwrite.vui_written += bytes; CDEBUG(D_VFSTRACE, "Committed %d pages %d bytes, tot: %ld\n", - npages, bytes, vio->u.write.vui_written); + npages, bytes, vio->u.readwrite.vui_written); /* the first page must have been written. */ - vio->u.write.vui_from = 0; + vio->u.readwrite.vui_from = 0; } LASSERT(page_list_sanity_check(obj, queue)); LASSERT(ergo(rc == 0, queue->pl_nr == 0)); @@ -1060,10 +1077,10 @@ int vvp_io_write_commit(const struct lu_env *env, struct cl_io *io) /* out of quota, try sync write */ if (rc == -EDQUOT && !cl_io_is_mkwrite(io)) { rc = vvp_io_commit_sync(env, io, queue, - vio->u.write.vui_from, - vio->u.write.vui_to); + vio->u.readwrite.vui_from, + vio->u.readwrite.vui_to); if (rc > 0) { - vio->u.write.vui_written += rc; + vio->u.readwrite.vui_written += rc; rc = 0; } } @@ -1181,15 +1198,15 @@ static int vvp_io_write_start(const struct lu_env *env, result = vvp_io_write_commit(env, io); /* Simulate short commit */ if (CFS_FAULT_CHECK(OBD_FAIL_LLITE_SHORT_COMMIT)) { - vio->u.write.vui_written >>= 1; - if (vio->u.write.vui_written > 0) + vio->u.readwrite.vui_written >>= 1; + if (vio->u.readwrite.vui_written > 0) io->ci_need_restart = 1; } - if (vio->u.write.vui_written > 0) { - result = vio->u.write.vui_written; + if (vio->u.readwrite.vui_written > 0) { + result = vio->u.readwrite.vui_written; io->ci_nob += result; - - CDEBUG(D_VFSTRACE, "write: nob %zd, result: %zd\n", + CDEBUG(D_VFSTRACE, "%s: write: nob %zd, result: %zd\n", + file_dentry(file)->d_name.name, io->ci_nob, result); } else { io->ci_continue = 0; @@ -1215,11 +1232,18 @@ static int vvp_io_write_start(const struct lu_env *env, if (result > 0 || result == -EIOCBQUEUED) { set_bit(LLIF_DATA_MODIFIED, &(ll_i2info(inode))->lli_flags); - if (result < cnt) + if (result != -EIOCBQUEUED && result < cnt) io->ci_continue = 0; if (result > 0) result = 0; + /* move forward */ + if (result == -EIOCBQUEUED) { + io->ci_nob += vio->u.readwrite.vui_written; + vio->vui_iocb->ki_pos = pos + + vio->u.readwrite.vui_written; + } } + return result; } @@ -1509,6 +1533,7 @@ static int vvp_io_read_ahead(const struct lu_env *env, .op = { [CIT_READ] = { .cio_fini = vvp_io_fini, + .cio_iter_init = vvp_io_read_iter_init, .cio_lock = vvp_io_read_lock, .cio_start = vvp_io_read_start, .cio_end = vvp_io_rw_end, diff --git a/fs/lustre/obdclass/cl_io.c b/fs/lustre/obdclass/cl_io.c index dcf940f..1564d9f 100644 --- a/fs/lustre/obdclass/cl_io.c +++ b/fs/lustre/obdclass/cl_io.c @@ -695,6 +695,7 @@ int cl_io_submit_sync(const struct lu_env *env, struct cl_io *io, int cl_io_loop(const struct lu_env *env, struct cl_io *io) { int result = 0; + int rc = 0; LINVRNT(cl_io_is_loopable(io)); @@ -727,7 +728,13 @@ int cl_io_loop(const struct lu_env *env, struct cl_io *io) } } cl_io_iter_fini(env, io); - } while (result == 0 && io->ci_continue); + if (result) + rc = result; + } while ((result == 0 || result == -EIOCBQUEUED) && + io->ci_continue); + + if (rc && !result) + result = rc; if (result == -EWOULDBLOCK && io->ci_ndelay) { io->ci_need_restart = 1; -- 1.8.3.1 From jsimmons at infradead.org Wed Jul 15 20:45:15 2020 From: jsimmons at infradead.org (James Simmons) Date: Wed, 15 Jul 2020 16:45:15 -0400 Subject: [lustre-devel] [PATCH 34/37] lustre: ptlrpc: fix endless loop issue In-Reply-To: <1594845918-29027-1-git-send-email-jsimmons@infradead.org> References: <1594845918-29027-1-git-send-email-jsimmons@infradead.org> Message-ID: <1594845918-29027-35-git-send-email-jsimmons@infradead.org> From: Hongchao Zhang In ptlrpc_pinger_main, if the process to ping the recoverable clients takes too long time, it could be stuck in endless loop because of the negative value returned by pinger_check_timeout. WC-bug-id: https://jira.whamcloud.com/browse/LU-13667 Lustre-commit: 6be2dbb259512 ("LU-13667 ptlrpc: fix endless loop issue") Signed-off-by: Hongchao Zhang Reviewed-on: https://review.whamcloud.com/38915 Reviewed-by: Andreas Dilger Reviewed-by: Olaf Faaland-LLNL Reviewed-by: Oleg Drokin Signed-off-by: James Simmons --- fs/lustre/ptlrpc/pinger.c | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-) diff --git a/fs/lustre/ptlrpc/pinger.c b/fs/lustre/ptlrpc/pinger.c index ec4c51a..9f57c61 100644 --- a/fs/lustre/ptlrpc/pinger.c +++ b/fs/lustre/ptlrpc/pinger.c @@ -258,12 +258,13 @@ static void ptlrpc_pinger_process_import(struct obd_import *imp, static void ptlrpc_pinger_main(struct work_struct *ws) { - time64_t this_ping = ktime_get_seconds(); - time64_t time_to_next_wake; + time64_t this_ping, time_after_ping, time_to_next_wake; struct timeout_item *item; struct obd_import *imp; do { + this_ping = ktime_get_seconds(); + mutex_lock(&pinger_mutex); list_for_each_entry(item, &timeout_list, ti_chain) { item->ti_cb(item, item->ti_cb_data); @@ -277,6 +278,12 @@ static void ptlrpc_pinger_main(struct work_struct *ws) } mutex_unlock(&pinger_mutex); + time_after_ping = ktime_get_seconds(); + + if ((ktime_get_seconds() - this_ping - 3) > PING_INTERVAL) + CDEBUG(D_HA, "long time to ping: %lld, %lld, %lld\n", + this_ping, time_after_ping, ktime_get_seconds()); + /* Wait until the next ping time, or until we're stopped. */ time_to_next_wake = pinger_check_timeout(this_ping); /* The ping sent by ptlrpc_send_rpc may get sent out -- 1.8.3.1 From jsimmons at infradead.org Wed Jul 15 20:45:04 2020 From: jsimmons at infradead.org (James Simmons) Date: Wed, 15 Jul 2020 16:45:04 -0400 Subject: [lustre-devel] [PATCH 23/37] lnet: o2iblnd: use need_resched() In-Reply-To: <1594845918-29027-1-git-send-email-jsimmons@infradead.org> References: <1594845918-29027-1-git-send-email-jsimmons@infradead.org> Message-ID: <1594845918-29027-24-git-send-email-jsimmons@infradead.org> From: Mr NeilBrown Rather than using a counter to decide when to drop the lock and see if we need to reshedule we can use need_resched(), which is a precise test instead of a guess. WC-bug-id: https://jira.whamcloud.com/browse/LU-12678 Lustre-commit: dcd799269f693 ("LU-12678 o2iblnd: use need_resched()") Signed-off-by: Mr NeilBrown Reviewed-on: https://review.whamcloud.com/39125 Reviewed-by: James Simmons Reviewed-by: Shaun Tancheff Reviewed-by: Chris Horn Reviewed-by: Oleg Drokin Signed-off-by: James Simmons --- net/lnet/klnds/o2iblnd/o2iblnd.h | 2 -- net/lnet/klnds/o2iblnd/o2iblnd_cb.c | 5 +---- 2 files changed, 1 insertion(+), 6 deletions(-) diff --git a/net/lnet/klnds/o2iblnd/o2iblnd.h b/net/lnet/klnds/o2iblnd/o2iblnd.h index f60a69d..9a2fb42 100644 --- a/net/lnet/klnds/o2iblnd/o2iblnd.h +++ b/net/lnet/klnds/o2iblnd/o2iblnd.h @@ -67,8 +67,6 @@ #include #define IBLND_PEER_HASH_SIZE 101 /* # peer_ni lists */ -/* # scheduler loops before reschedule */ -#define IBLND_RESCHED 100 #define IBLND_N_SCHED 2 #define IBLND_N_SCHED_HIGH 4 diff --git a/net/lnet/klnds/o2iblnd/o2iblnd_cb.c b/net/lnet/klnds/o2iblnd/o2iblnd_cb.c index 3b9d10d..2c670a33 100644 --- a/net/lnet/klnds/o2iblnd/o2iblnd_cb.c +++ b/net/lnet/klnds/o2iblnd/o2iblnd_cb.c @@ -3605,7 +3605,6 @@ static int kiblnd_resolve_addr(struct rdma_cm_id *cmid, unsigned long flags; struct ib_wc wc; int did_something; - int busy_loops = 0; int rc; init_waitqueue_entry(&wait, current); @@ -3621,11 +3620,10 @@ static int kiblnd_resolve_addr(struct rdma_cm_id *cmid, spin_lock_irqsave(&sched->ibs_lock, flags); while (!kiblnd_data.kib_shutdown) { - if (busy_loops++ >= IBLND_RESCHED) { + if (need_resched()) { spin_unlock_irqrestore(&sched->ibs_lock, flags); cond_resched(); - busy_loops = 0; spin_lock_irqsave(&sched->ibs_lock, flags); } @@ -3718,7 +3716,6 @@ static int kiblnd_resolve_addr(struct rdma_cm_id *cmid, spin_unlock_irqrestore(&sched->ibs_lock, flags); schedule(); - busy_loops = 0; remove_wait_queue(&sched->ibs_waitq, &wait); spin_lock_irqsave(&sched->ibs_lock, flags); -- 1.8.3.1 From jsimmons at infradead.org Wed Jul 15 20:45:08 2020 From: jsimmons at infradead.org (James Simmons) Date: Wed, 15 Jul 2020 16:45:08 -0400 Subject: [lustre-devel] [PATCH 27/37] lnet: socklnd: convert various refcounts to refcount_t In-Reply-To: <1594845918-29027-1-git-send-email-jsimmons@infradead.org> References: <1594845918-29027-1-git-send-email-jsimmons@infradead.org> Message-ID: <1594845918-29027-28-git-send-email-jsimmons@infradead.org> From: Mr NeilBrown Each of these refcounts exactly follows the expectations of refcount_t, so change the atomic_t to refcoun_t. We can remove the LASSERTs on incref/decref as they can now be enabled at build time with CONFIG_REFCOUNT_FULL WC-bug-id: https://jira.whamcloud.com/browse/LU-12678 Lustre-commit: db3e51f612069 ("LU-12678 socklnd: convert various refcounts to refcount_t") Signed-off-by: Mr NeilBrown Reviewed-on: https://review.whamcloud.com/39130 Reviewed-by: James Simmons Reviewed-by: Shaun Tancheff Reviewed-by: Chris Horn Reviewed-by: Oleg Drokin Signed-off-by: James Simmons --- net/lnet/klnds/socklnd/socklnd.c | 28 ++++++++++++------------- net/lnet/klnds/socklnd/socklnd.h | 41 +++++++++++++++---------------------- net/lnet/klnds/socklnd/socklnd_cb.c | 6 +++--- 3 files changed, 33 insertions(+), 42 deletions(-) diff --git a/net/lnet/klnds/socklnd/socklnd.c b/net/lnet/klnds/socklnd/socklnd.c index 2e11737..22a73c3 100644 --- a/net/lnet/klnds/socklnd/socklnd.c +++ b/net/lnet/klnds/socklnd/socklnd.c @@ -123,7 +123,7 @@ static int ksocknal_ip2index(__u32 ipaddress, struct lnet_ni *ni) if (!route) return NULL; - atomic_set(&route->ksnr_refcount, 1); + refcount_set(&route->ksnr_refcount, 1); route->ksnr_peer = NULL; route->ksnr_retry_interval = 0; /* OK to connect at any time */ route->ksnr_ipaddr = ipaddr; @@ -142,7 +142,7 @@ static int ksocknal_ip2index(__u32 ipaddress, struct lnet_ni *ni) void ksocknal_destroy_route(struct ksock_route *route) { - LASSERT(!atomic_read(&route->ksnr_refcount)); + LASSERT(!refcount_read(&route->ksnr_refcount)); if (route->ksnr_peer) ksocknal_peer_decref(route->ksnr_peer); @@ -174,7 +174,7 @@ static int ksocknal_ip2index(__u32 ipaddress, struct lnet_ni *ni) peer_ni->ksnp_ni = ni; peer_ni->ksnp_id = id; - atomic_set(&peer_ni->ksnp_refcount, 1); /* 1 ref for caller */ + refcount_set(&peer_ni->ksnp_refcount, 1); /* 1 ref for caller */ peer_ni->ksnp_closing = 0; peer_ni->ksnp_accepting = 0; peer_ni->ksnp_proto = NULL; @@ -198,7 +198,7 @@ static int ksocknal_ip2index(__u32 ipaddress, struct lnet_ni *ni) CDEBUG(D_NET, "peer_ni %s %p deleted\n", libcfs_id2str(peer_ni->ksnp_id), peer_ni); - LASSERT(!atomic_read(&peer_ni->ksnp_refcount)); + LASSERT(!refcount_read(&peer_ni->ksnp_refcount)); LASSERT(!peer_ni->ksnp_accepting); LASSERT(list_empty(&peer_ni->ksnp_conns)); LASSERT(list_empty(&peer_ni->ksnp_routes)); @@ -235,7 +235,7 @@ struct ksock_peer_ni * CDEBUG(D_NET, "got peer_ni [%p] -> %s (%d)\n", peer_ni, libcfs_id2str(id), - atomic_read(&peer_ni->ksnp_refcount)); + refcount_read(&peer_ni->ksnp_refcount)); return peer_ni; } return NULL; @@ -1069,10 +1069,10 @@ struct ksock_peer_ni * * 2 ref, 1 for conn, another extra ref prevents socket * being closed before establishment of connection */ - atomic_set(&conn->ksnc_sock_refcount, 2); + refcount_set(&conn->ksnc_sock_refcount, 2); conn->ksnc_type = type; ksocknal_lib_save_callback(sock, conn); - atomic_set(&conn->ksnc_conn_refcount, 1); /* 1 ref for me */ + refcount_set(&conn->ksnc_conn_refcount, 1); /* 1 ref for me */ conn->ksnc_rx_ready = 0; conn->ksnc_rx_scheduled = 0; @@ -1667,7 +1667,7 @@ struct ksock_peer_ni * { /* Queue the conn for the reaper to destroy */ - LASSERT(!atomic_read(&conn->ksnc_conn_refcount)); + LASSERT(!refcount_read(&conn->ksnc_conn_refcount)); spin_lock_bh(&ksocknal_data.ksnd_reaper_lock); list_add_tail(&conn->ksnc_list, &ksocknal_data.ksnd_zombie_conns); @@ -1684,8 +1684,8 @@ struct ksock_peer_ni * /* Final coup-de-grace of the reaper */ CDEBUG(D_NET, "connection %p\n", conn); - LASSERT(!atomic_read(&conn->ksnc_conn_refcount)); - LASSERT(!atomic_read(&conn->ksnc_sock_refcount)); + LASSERT(!refcount_read(&conn->ksnc_conn_refcount)); + LASSERT(!refcount_read(&conn->ksnc_sock_refcount)); LASSERT(!conn->ksnc_sock); LASSERT(!conn->ksnc_route); LASSERT(!conn->ksnc_tx_scheduled); @@ -2412,7 +2412,7 @@ static int ksocknal_push(struct lnet_ni *ni, struct lnet_process_id id) CWARN("Active peer_ni on shutdown: %s, ref %d, closing %d, accepting %d, err %d, zcookie %llu, txq %d, zc_req %d\n", libcfs_id2str(peer_ni->ksnp_id), - atomic_read(&peer_ni->ksnp_refcount), + refcount_read(&peer_ni->ksnp_refcount), peer_ni->ksnp_closing, peer_ni->ksnp_accepting, peer_ni->ksnp_error, peer_ni->ksnp_zc_next_cookie, @@ -2421,7 +2421,7 @@ static int ksocknal_push(struct lnet_ni *ni, struct lnet_process_id id) list_for_each_entry(route, &peer_ni->ksnp_routes, ksnr_list) { CWARN("Route: ref %d, schd %d, conn %d, cnted %d, del %d\n", - atomic_read(&route->ksnr_refcount), + refcount_read(&route->ksnr_refcount), route->ksnr_scheduled, route->ksnr_connecting, route->ksnr_connected, @@ -2430,8 +2430,8 @@ static int ksocknal_push(struct lnet_ni *ni, struct lnet_process_id id) list_for_each_entry(conn, &peer_ni->ksnp_conns, ksnc_list) { CWARN("Conn: ref %d, sref %d, t %d, c %d\n", - atomic_read(&conn->ksnc_conn_refcount), - atomic_read(&conn->ksnc_sock_refcount), + refcount_read(&conn->ksnc_conn_refcount), + refcount_read(&conn->ksnc_sock_refcount), conn->ksnc_type, conn->ksnc_closing); } goto done; diff --git a/net/lnet/klnds/socklnd/socklnd.h b/net/lnet/klnds/socklnd/socklnd.h index 0a0f0a7..df863f2 100644 --- a/net/lnet/klnds/socklnd/socklnd.h +++ b/net/lnet/klnds/socklnd/socklnd.h @@ -37,6 +37,7 @@ #include #include #include +#include #include #include #include @@ -270,7 +271,7 @@ struct ksock_tx { /* transmit packet */ struct list_head tx_list; /* queue on conn for transmission etc */ struct list_head tx_zc_list; /* queue on peer_ni for ZC request */ - atomic_t tx_refcount; /* tx reference count */ + refcount_t tx_refcount; /* tx reference count */ int tx_nob; /* # packet bytes */ int tx_resid; /* residual bytes */ int tx_niov; /* # packet iovec frags */ @@ -311,8 +312,8 @@ struct ksock_conn { void *ksnc_saved_write_space; /* socket's original * write_space() callback */ - atomic_t ksnc_conn_refcount; /* conn refcount */ - atomic_t ksnc_sock_refcount; /* sock refcount */ + refcount_t ksnc_conn_refcount; /* conn refcount */ + refcount_t ksnc_sock_refcount; /* sock refcount */ struct ksock_sched *ksnc_scheduler; /* who schedules this connection */ u32 ksnc_myipaddr; /* my IP */ @@ -374,7 +375,7 @@ struct ksock_route { struct list_head ksnr_list; /* chain on peer_ni route list */ struct list_head ksnr_connd_list; /* chain on ksnr_connd_routes */ struct ksock_peer_ni *ksnr_peer; /* owning peer_ni */ - atomic_t ksnr_refcount; /* # users */ + refcount_t ksnr_refcount; /* # users */ time64_t ksnr_timeout; /* when (in secs) reconnection * can happen next */ @@ -404,7 +405,7 @@ struct ksock_peer_ni { * alive */ struct lnet_process_id ksnp_id; /* who's on the other end(s) */ - atomic_t ksnp_refcount; /* # users */ + refcount_t ksnp_refcount; /* # users */ int ksnp_closing; /* being closed */ int ksnp_accepting; /* # passive connections pending */ @@ -510,8 +511,7 @@ struct ksock_proto { static inline void ksocknal_conn_addref(struct ksock_conn *conn) { - LASSERT(atomic_read(&conn->ksnc_conn_refcount) > 0); - atomic_inc(&conn->ksnc_conn_refcount); + refcount_inc(&conn->ksnc_conn_refcount); } void ksocknal_queue_zombie_conn(struct ksock_conn *conn); @@ -520,8 +520,7 @@ struct ksock_proto { static inline void ksocknal_conn_decref(struct ksock_conn *conn) { - LASSERT(atomic_read(&conn->ksnc_conn_refcount) > 0); - if (atomic_dec_and_test(&conn->ksnc_conn_refcount)) + if (refcount_dec_and_test(&conn->ksnc_conn_refcount)) ksocknal_queue_zombie_conn(conn); } @@ -532,8 +531,7 @@ struct ksock_proto { read_lock(&ksocknal_data.ksnd_global_lock); if (!conn->ksnc_closing) { - LASSERT(atomic_read(&conn->ksnc_sock_refcount) > 0); - atomic_inc(&conn->ksnc_sock_refcount); + refcount_inc(&conn->ksnc_sock_refcount); rc = 0; } read_unlock(&ksocknal_data.ksnd_global_lock); @@ -544,8 +542,7 @@ struct ksock_proto { static inline void ksocknal_connsock_decref(struct ksock_conn *conn) { - LASSERT(atomic_read(&conn->ksnc_sock_refcount) > 0); - if (atomic_dec_and_test(&conn->ksnc_sock_refcount)) { + if (refcount_dec_and_test(&conn->ksnc_sock_refcount)) { LASSERT(conn->ksnc_closing); sock_release(conn->ksnc_sock); conn->ksnc_sock = NULL; @@ -556,8 +553,7 @@ struct ksock_proto { static inline void ksocknal_tx_addref(struct ksock_tx *tx) { - LASSERT(atomic_read(&tx->tx_refcount) > 0); - atomic_inc(&tx->tx_refcount); + refcount_inc(&tx->tx_refcount); } void ksocknal_tx_prep(struct ksock_conn *, struct ksock_tx *tx); @@ -566,16 +562,14 @@ struct ksock_proto { static inline void ksocknal_tx_decref(struct ksock_tx *tx) { - LASSERT(atomic_read(&tx->tx_refcount) > 0); - if (atomic_dec_and_test(&tx->tx_refcount)) + if (refcount_dec_and_test(&tx->tx_refcount)) ksocknal_tx_done(NULL, tx, 0); } static inline void ksocknal_route_addref(struct ksock_route *route) { - LASSERT(atomic_read(&route->ksnr_refcount) > 0); - atomic_inc(&route->ksnr_refcount); + refcount_inc(&route->ksnr_refcount); } void ksocknal_destroy_route(struct ksock_route *route); @@ -583,16 +577,14 @@ struct ksock_proto { static inline void ksocknal_route_decref(struct ksock_route *route) { - LASSERT(atomic_read(&route->ksnr_refcount) > 0); - if (atomic_dec_and_test(&route->ksnr_refcount)) + if (refcount_dec_and_test(&route->ksnr_refcount)) ksocknal_destroy_route(route); } static inline void ksocknal_peer_addref(struct ksock_peer_ni *peer_ni) { - LASSERT(atomic_read(&peer_ni->ksnp_refcount) > 0); - atomic_inc(&peer_ni->ksnp_refcount); + refcount_inc(&peer_ni->ksnp_refcount); } void ksocknal_destroy_peer(struct ksock_peer_ni *peer_ni); @@ -600,8 +592,7 @@ struct ksock_proto { static inline void ksocknal_peer_decref(struct ksock_peer_ni *peer_ni) { - LASSERT(atomic_read(&peer_ni->ksnp_refcount) > 0); - if (atomic_dec_and_test(&peer_ni->ksnp_refcount)) + if (refcount_dec_and_test(&peer_ni->ksnp_refcount)) ksocknal_destroy_peer(peer_ni); } diff --git a/net/lnet/klnds/socklnd/socklnd_cb.c b/net/lnet/klnds/socklnd/socklnd_cb.c index 936054ee..9b3b604 100644 --- a/net/lnet/klnds/socklnd/socklnd_cb.c +++ b/net/lnet/klnds/socklnd/socklnd_cb.c @@ -52,7 +52,7 @@ struct ksock_tx * if (!tx) return NULL; - atomic_set(&tx->tx_refcount, 1); + refcount_set(&tx->tx_refcount, 1); tx->tx_zc_aborted = 0; tx->tx_zc_capable = 0; tx->tx_zc_checked = 0; @@ -381,7 +381,7 @@ struct ksock_tx * tx->tx_hstatus = LNET_MSG_STATUS_LOCAL_ERROR; } - LASSERT(atomic_read(&tx->tx_refcount) == 1); + LASSERT(refcount_read(&tx->tx_refcount) == 1); ksocknal_tx_done(ni, tx, error); } } @@ -1072,7 +1072,7 @@ struct ksock_route * struct lnet_process_id *id; int rc; - LASSERT(atomic_read(&conn->ksnc_conn_refcount) > 0); + LASSERT(refcount_read(&conn->ksnc_conn_refcount) > 0); /* NB: sched lock NOT held */ /* SOCKNAL_RX_LNET_HEADER is here for backward compatibility */ -- 1.8.3.1 From jsimmons at infradead.org Wed Jul 15 20:45:09 2020 From: jsimmons at infradead.org (James Simmons) Date: Wed, 15 Jul 2020 16:45:09 -0400 Subject: [lustre-devel] [PATCH 28/37] lnet: libcfs: don't call unshare_fs_struct() In-Reply-To: <1594845918-29027-1-git-send-email-jsimmons@infradead.org> References: <1594845918-29027-1-git-send-email-jsimmons@infradead.org> Message-ID: <1594845918-29027-29-git-send-email-jsimmons@infradead.org> From: Mr NeilBrown A kthread runs with the same fs_struct as init. It is only helpful to unshare this if the thread will change one of the fields in the fs_struct: root directory current working directory umask. No lustre kthread changes any of these, so there is no need to call unshare_fs_struct(). WC-bug-id: https://jira.whamcloud.com/browse/LU-9859 Lustre-commit: 9013eb2bb5492 ("LU-9859 libcfs: don't call unshare_fs_struct()") Signed-off-by: Mr NeilBrown Reviewed-on: https://review.whamcloud.com/39132 Reviewed-by: James Simmons Reviewed-by: Yang Sheng Reviewed-by: Oleg Drokin Signed-off-by: James Simmons --- fs/lustre/obdclass/llog.c | 2 -- fs/lustre/ptlrpc/import.c | 2 -- fs/lustre/ptlrpc/ptlrpcd.c | 1 - fs/lustre/ptlrpc/service.c | 3 --- 4 files changed, 8 deletions(-) diff --git a/fs/lustre/obdclass/llog.c b/fs/lustre/obdclass/llog.c index b2667d9..e172ebc 100644 --- a/fs/lustre/obdclass/llog.c +++ b/fs/lustre/obdclass/llog.c @@ -449,8 +449,6 @@ static int llog_process_thread_daemonize(void *arg) struct lu_env env; int rc; - unshare_fs_struct(); - /* client env has no keys, tags is just 0 */ rc = lu_env_init(&env, LCT_LOCAL | LCT_MG_THREAD); if (rc) diff --git a/fs/lustre/ptlrpc/import.c b/fs/lustre/ptlrpc/import.c index 1b62b81..1490dcf 100644 --- a/fs/lustre/ptlrpc/import.c +++ b/fs/lustre/ptlrpc/import.c @@ -1438,8 +1438,6 @@ static int ptlrpc_invalidate_import_thread(void *data) { struct obd_import *imp = data; - unshare_fs_struct(); - CDEBUG(D_HA, "thread invalidate import %s to %s@%s\n", imp->imp_obd->obd_name, obd2cli_tgt(imp->imp_obd), imp->imp_connection->c_remote_uuid.uuid); diff --git a/fs/lustre/ptlrpc/ptlrpcd.c b/fs/lustre/ptlrpc/ptlrpcd.c index 533f592..b0b81cc 100644 --- a/fs/lustre/ptlrpc/ptlrpcd.c +++ b/fs/lustre/ptlrpc/ptlrpcd.c @@ -393,7 +393,6 @@ static int ptlrpcd(void *arg) int rc = 0; int exit = 0; - unshare_fs_struct(); if (cfs_cpt_bind(cfs_cpt_tab, pc->pc_cpt) != 0) CWARN("Failed to bind %s on CPT %d\n", pc->pc_name, pc->pc_cpt); diff --git a/fs/lustre/ptlrpc/service.c b/fs/lustre/ptlrpc/service.c index 4d5e6b3..5881e0a 100644 --- a/fs/lustre/ptlrpc/service.c +++ b/fs/lustre/ptlrpc/service.c @@ -2175,7 +2175,6 @@ static int ptlrpc_main(void *arg) thread->t_task = current; thread->t_pid = current->pid; - unshare_fs_struct(); if (svc->srv_cpt_bind) { rc = cfs_cpt_bind(svc->srv_cptable, svcpt->scp_cpt); @@ -2391,8 +2390,6 @@ static int ptlrpc_hr_main(void *arg) if (!env) return -ENOMEM; - unshare_fs_struct(); - rc = cfs_cpt_bind(ptlrpc_hr.hr_cpt_table, hrp->hrp_cpt); if (rc != 0) { char threadname[20]; -- 1.8.3.1 From jsimmons at infradead.org Wed Jul 15 20:45:01 2020 From: jsimmons at infradead.org (James Simmons) Date: Wed, 15 Jul 2020 16:45:01 -0400 Subject: [lustre-devel] [PATCH 20/37] lustre: llite: fix to free cl_dio_aio properly In-Reply-To: <1594845918-29027-1-git-send-email-jsimmons@infradead.org> References: <1594845918-29027-1-git-send-email-jsimmons@infradead.org> Message-ID: <1594845918-29027-21-git-send-email-jsimmons@infradead.org> From: Wang Shilong @cl_dio_aio is allocated by slab, we should use slab free helper to free its memory. Fixes: ebdbecbaf50b ("lustre: obdclass: use slab allocation for cl_dio_aio") WC-bug-id: https://jira.whamcloud.com/browse/LU-13134 Lustre-commit: f71a539c3e41b ("LU-13134 llite: fix to free cl_dio_aio properly") Signed-off-by: Wang Shilong Reviewed-on: https://review.whamcloud.com/39103 Reviewed-by: Patrick Farrell Reviewed-by: Andreas Dilger Reviewed-by: James Simmons Signed-off-by: James Simmons --- fs/lustre/include/cl_object.h | 2 ++ fs/lustre/llite/rw26.c | 2 +- fs/lustre/obdclass/cl_io.c | 10 ++++++++-- 3 files changed, 11 insertions(+), 3 deletions(-) diff --git a/fs/lustre/include/cl_object.h b/fs/lustre/include/cl_object.h index 8611285..e656c68 100644 --- a/fs/lustre/include/cl_object.h +++ b/fs/lustre/include/cl_object.h @@ -2538,6 +2538,8 @@ int cl_sync_io_wait(const struct lu_env *env, struct cl_sync_io *anchor, void cl_sync_io_note(const struct lu_env *env, struct cl_sync_io *anchor, int ioret); struct cl_dio_aio *cl_aio_alloc(struct kiocb *iocb); +void cl_aio_free(struct cl_dio_aio *aio); + static inline void cl_sync_io_init(struct cl_sync_io *anchor, int nr) { cl_sync_io_init_notify(anchor, nr, NULL, NULL); diff --git a/fs/lustre/llite/rw26.c b/fs/lustre/llite/rw26.c index 0971185..d0e3ff6 100644 --- a/fs/lustre/llite/rw26.c +++ b/fs/lustre/llite/rw26.c @@ -384,7 +384,7 @@ static ssize_t ll_direct_IO(struct kiocb *iocb, struct iov_iter *iter) vio->u.write.vui_written += tot_bytes; result = tot_bytes; } - kfree(aio); + cl_aio_free(aio); } else { result = -EIOCBQUEUED; } diff --git a/fs/lustre/obdclass/cl_io.c b/fs/lustre/obdclass/cl_io.c index 2f597d1..dcf940f 100644 --- a/fs/lustre/obdclass/cl_io.c +++ b/fs/lustre/obdclass/cl_io.c @@ -1106,6 +1106,13 @@ struct cl_dio_aio *cl_aio_alloc(struct kiocb *iocb) } EXPORT_SYMBOL(cl_aio_alloc); +void cl_aio_free(struct cl_dio_aio *aio) +{ + if (aio) + kmem_cache_free(cl_dio_aio_kmem, aio); +} +EXPORT_SYMBOL(cl_aio_free); + /** * Indicate that transfer of a single page completed. */ @@ -1143,8 +1150,7 @@ void cl_sync_io_note(const struct lu_env *env, struct cl_sync_io *anchor, * If anchor->csi_aio is set, we are responsible for freeing * memory here rather than when cl_sync_io_wait() completes. */ - if (aio) - kmem_cache_free(cl_dio_aio_kmem, aio); + cl_aio_free(aio); } } EXPORT_SYMBOL(cl_sync_io_note); -- 1.8.3.1 From jsimmons at infradead.org Wed Jul 15 20:45:05 2020 From: jsimmons at infradead.org (James Simmons) Date: Wed, 15 Jul 2020 16:45:05 -0400 Subject: [lustre-devel] [PATCH 24/37] lnet: o2iblnd: Use list_for_each_entry_safe In-Reply-To: <1594845918-29027-1-git-send-email-jsimmons@infradead.org> References: <1594845918-29027-1-git-send-email-jsimmons@infradead.org> Message-ID: <1594845918-29027-25-git-send-email-jsimmons@infradead.org> From: Mr NeilBrown Several loops use list_for_each_safe(), then call list_entry() as first step. These can be merged using list_for_each_entry_safe(). WC-bug-id: https://jira.whamcloud.com/browse/LU-12678 Lustre-commit: e5574f72f2fd9 ("LU-12678 o2iblnd: Use list_for_each_entry_safe") Signed-off-by: Mr NeilBrown Reviewed-on: https://review.whamcloud.com/39126 Reviewed-by: James Simmons Reviewed-by: Shaun Tancheff Reviewed-by: Chris Horn Reviewed-by: Oleg Drokin Signed-off-by: James Simmons --- net/lnet/klnds/o2iblnd/o2iblnd.c | 26 ++++++++++---------------- net/lnet/klnds/o2iblnd/o2iblnd_cb.c | 7 ++----- 2 files changed, 12 insertions(+), 21 deletions(-) diff --git a/net/lnet/klnds/o2iblnd/o2iblnd.c b/net/lnet/klnds/o2iblnd/o2iblnd.c index 6c7659c..c6a077b 100644 --- a/net/lnet/klnds/o2iblnd/o2iblnd.c +++ b/net/lnet/klnds/o2iblnd/o2iblnd.c @@ -454,18 +454,16 @@ static int kiblnd_get_peer_info(struct lnet_ni *ni, int index, static void kiblnd_del_peer_locked(struct kib_peer_ni *peer_ni) { - struct list_head *ctmp; - struct list_head *cnxt; + struct kib_conn *cnxt; struct kib_conn *conn; if (list_empty(&peer_ni->ibp_conns)) { kiblnd_unlink_peer_locked(peer_ni); } else { - list_for_each_safe(ctmp, cnxt, &peer_ni->ibp_conns) { - conn = list_entry(ctmp, struct kib_conn, ibc_list); - + list_for_each_entry_safe(conn, cnxt, &peer_ni->ibp_conns, + ibc_list) kiblnd_close_conn_locked(conn, 0); - } + /* NB closing peer_ni's last conn unlinked it. */ } /* @@ -952,13 +950,11 @@ void kiblnd_destroy_conn(struct kib_conn *conn) int kiblnd_close_peer_conns_locked(struct kib_peer_ni *peer_ni, int why) { struct kib_conn *conn; - struct list_head *ctmp; - struct list_head *cnxt; + struct kib_conn *cnxt; int count = 0; - list_for_each_safe(ctmp, cnxt, &peer_ni->ibp_conns) { - conn = list_entry(ctmp, struct kib_conn, ibc_list); - + list_for_each_entry_safe(conn, cnxt, &peer_ni->ibp_conns, + ibc_list) { CDEBUG(D_NET, "Closing conn -> %s, version: %x, reason: %d\n", libcfs_nid2str(peer_ni->ibp_nid), conn->ibc_version, why); @@ -974,13 +970,11 @@ int kiblnd_close_stale_conns_locked(struct kib_peer_ni *peer_ni, int version, u64 incarnation) { struct kib_conn *conn; - struct list_head *ctmp; - struct list_head *cnxt; + struct kib_conn *cnxt; int count = 0; - list_for_each_safe(ctmp, cnxt, &peer_ni->ibp_conns) { - conn = list_entry(ctmp, struct kib_conn, ibc_list); - + list_for_each_entry_safe(conn, cnxt, &peer_ni->ibp_conns, + ibc_list) { if (conn->ibc_version == version && conn->ibc_incarnation == incarnation) continue; diff --git a/net/lnet/klnds/o2iblnd/o2iblnd_cb.c b/net/lnet/klnds/o2iblnd/o2iblnd_cb.c index 2c670a33..ba2f46f 100644 --- a/net/lnet/klnds/o2iblnd/o2iblnd_cb.c +++ b/net/lnet/klnds/o2iblnd/o2iblnd_cb.c @@ -1982,15 +1982,12 @@ static int kiblnd_resolve_addr(struct rdma_cm_id *cmid, kiblnd_abort_txs(struct kib_conn *conn, struct list_head *txs) { LIST_HEAD(zombies); - struct list_head *tmp; - struct list_head *nxt; + struct kib_tx *nxt; struct kib_tx *tx; spin_lock(&conn->ibc_lock); - list_for_each_safe(tmp, nxt, txs) { - tx = list_entry(tmp, struct kib_tx, tx_list); - + list_for_each_entry_safe(tx, nxt, txs, tx_list) { if (txs == &conn->ibc_active_txs) { LASSERT(!tx->tx_queued); LASSERT(tx->tx_waiting || tx->tx_sending); -- 1.8.3.1 From jsimmons at infradead.org Wed Jul 15 20:45:18 2020 From: jsimmons at infradead.org (James Simmons) Date: Wed, 15 Jul 2020 16:45:18 -0400 Subject: [lustre-devel] [PATCH 37/37] lnet: check rtr_nid is a gateway In-Reply-To: <1594845918-29027-1-git-send-email-jsimmons@infradead.org> References: <1594845918-29027-1-git-send-email-jsimmons@infradead.org> Message-ID: <1594845918-29027-38-git-send-email-jsimmons@infradead.org> From: Amir Shehata The rtr_nid is specified for all REPLY/ACK. However it is possible for the route through the gateway specified by rtr_nid to be removed. In this case we don't want to use it. We should lookup alternative paths. This patch checks if the peer looked up is indeed a gateway. If it's not a gateway then we attempt to find another path. There is no need to fail right away. It's not a hard requirement to fail if the default rtr_nid is not valid. WC-bug-id: https://jira.whamcloud.com/browse/LU-13713 Lustre-commit: 07397a2e7473c ("LU-13713 lnet: check rtr_nid is a gateway") Signed-off-by: Amir Shehata Reviewed-on: https://review.whamcloud.com/39175 Reviewed-by: Chris Horn Reviewed-by: Serguei Smirnov Reviewed-by: Oleg Drokin Signed-off-by: James Simmons --- net/lnet/lnet/lib-move.c | 20 +++++++++++++------- 1 file changed, 13 insertions(+), 7 deletions(-) diff --git a/net/lnet/lnet/lib-move.c b/net/lnet/lnet/lib-move.c index 234fbb5..c0dd30c 100644 --- a/net/lnet/lnet/lib-move.c +++ b/net/lnet/lnet/lib-move.c @@ -1777,6 +1777,7 @@ struct lnet_ni * struct lnet_route *last_route = NULL; struct lnet_peer_ni *lpni = NULL; struct lnet_peer_ni *gwni = NULL; + bool route_found = false; lnet_nid_t src_nid = (sd->sd_src_nid != LNET_NID_ANY) ? sd->sd_src_nid : sd->sd_best_ni ? sd->sd_best_ni->ni_nid : LNET_NID_ANY; @@ -1790,15 +1791,20 @@ struct lnet_ni * */ if (sd->sd_rtr_nid != LNET_NID_ANY) { gwni = lnet_find_peer_ni_locked(sd->sd_rtr_nid); - if (!gwni) { - CERROR("No peer NI for gateway %s\n", + if (gwni) { + gw = gwni->lpni_peer_net->lpn_peer; + lnet_peer_ni_decref_locked(gwni); + if (gw->lp_rtr_refcount) { + local_lnet = LNET_NIDNET(sd->sd_rtr_nid); + route_found = true; + } + } else { + CWARN("No peer NI for gateway %s. Attempting to find an alternative route.\n", libcfs_nid2str(sd->sd_rtr_nid)); - return -EHOSTUNREACH; } - gw = gwni->lpni_peer_net->lpn_peer; - lnet_peer_ni_decref_locked(gwni); - local_lnet = LNET_NIDNET(sd->sd_rtr_nid); - } else { + } + + if (!route_found) { /* we've already looked up the initial lpni using dst_nid */ lpni = sd->sd_best_lpni; /* the peer tree must be in existence */ -- 1.8.3.1 From jsimmons at infradead.org Wed Jul 15 20:45:02 2020 From: jsimmons at infradead.org (James Simmons) Date: Wed, 15 Jul 2020 16:45:02 -0400 Subject: [lustre-devel] [PATCH 21/37] lnet: o2iblnd: Use ib_mtu_int_to_enum() In-Reply-To: <1594845918-29027-1-git-send-email-jsimmons@infradead.org> References: <1594845918-29027-1-git-send-email-jsimmons@infradead.org> Message-ID: <1594845918-29027-22-git-send-email-jsimmons@infradead.org> From: Mr NeilBrown Rather than bespoke code for converting an MTU into the enum, use ib_mtu_int_to_enum(). This has slightly different behaviour for invalid values, but those are caught when the parameter is set. WC-bug-id: https://jira.whamcloud.com/browse/LU-12678 Lustre-commit: 1b622e2007483 ("LU-12678 o2iblnd: Use ib_mtu_int_to_enum()") Signed-off-by: Mr NeilBrown Reviewed-on: https://review.whamcloud.com/39123 Reviewed-by: James Simmons Reviewed-by: Shaun Tancheff Reviewed-by: Chris Horn Reviewed-by: Oleg Drokin Signed-off-by: James Simmons --- net/lnet/klnds/o2iblnd/o2iblnd.c | 29 +++-------------------------- net/lnet/klnds/o2iblnd/o2iblnd_modparams.c | 4 +++- 2 files changed, 6 insertions(+), 27 deletions(-) diff --git a/net/lnet/klnds/o2iblnd/o2iblnd.c b/net/lnet/klnds/o2iblnd/o2iblnd.c index d8fca2a..e2e94b7 100644 --- a/net/lnet/klnds/o2iblnd/o2iblnd.c +++ b/net/lnet/klnds/o2iblnd/o2iblnd.c @@ -560,38 +560,15 @@ static struct kib_conn *kiblnd_get_conn_by_idx(struct lnet_ni *ni, int index) return NULL; } -int kiblnd_translate_mtu(int value) -{ - switch (value) { - default: - return -1; - case 0: - return 0; - case 256: - return IB_MTU_256; - case 512: - return IB_MTU_512; - case 1024: - return IB_MTU_1024; - case 2048: - return IB_MTU_2048; - case 4096: - return IB_MTU_4096; - } -} - static void kiblnd_setup_mtu_locked(struct rdma_cm_id *cmid) { - int mtu; - /* XXX There is no path record for iWARP, set by netdev->change_mtu? */ if (!cmid->route.path_rec) return; - mtu = kiblnd_translate_mtu(*kiblnd_tunables.kib_ib_mtu); - LASSERT(mtu >= 0); - if (mtu) - cmid->route.path_rec->mtu = mtu; + if (*kiblnd_tunables.kib_ib_mtu) + cmid->route.path_rec->mtu = + ib_mtu_int_to_enum(*kiblnd_tunables.kib_ib_mtu); } static int kiblnd_get_completion_vector(struct kib_conn *conn, int cpt) diff --git a/net/lnet/klnds/o2iblnd/o2iblnd_modparams.c b/net/lnet/klnds/o2iblnd/o2iblnd_modparams.c index f341376..73ad22d 100644 --- a/net/lnet/klnds/o2iblnd/o2iblnd_modparams.c +++ b/net/lnet/klnds/o2iblnd/o2iblnd_modparams.c @@ -230,7 +230,9 @@ int kiblnd_tunables_setup(struct lnet_ni *ni) /* Current API version */ tunables->lnd_version = 0; - if (kiblnd_translate_mtu(*kiblnd_tunables.kib_ib_mtu) < 0) { + if (*kiblnd_tunables.kib_ib_mtu && + ib_mtu_enum_to_int(ib_mtu_int_to_enum(*kiblnd_tunables.kib_ib_mtu)) != + *kiblnd_tunables.kib_ib_mtu) { CERROR("Invalid ib_mtu %d, expected 256/512/1024/2048/4096\n", *kiblnd_tunables.kib_ib_mtu); return -EINVAL; -- 1.8.3.1 From green at whamcloud.com Wed Jul 22 16:58:38 2020 From: green at whamcloud.com (Oleg Drokin) Date: Wed, 22 Jul 2020 16:58:38 +0000 Subject: [lustre-devel] New tag 2.13.55 Message-ID: <81696147-A0C5-45ED-BE82-70B53F2BD2C2@whamcloud.com> Hello! I just added 2.13.55 tag in the main Kustre development branch repo. Changelog is below: Alex Zhuravlev (4): LU-2225 tests: sanity/27 tests to poll for state LU-13520 ldiskfs: fastpath in bitmap prefetching LU-13195 osp: invalidate object on write error LU-12905 tests: wrappers for createmany and unlinkmany Alexander Boyko (1): LU-13617 llite: don't hold inode_lock for security notify Alexey Lyashkov (5): LU-10157 ptlrpc: separate number MD and refrences for bulk LU-10157 ptlrpc: fill md correctly. LU-10157 lnet: restore an maximal fragments count LU-13181 o2ib: fix page mapping error LU-12214 build: fix build with gss enabled Amir Shehata (4): LU-10973 lnet: infrastructure to build the LUTF LU-13662 lnet: handle undefined parameters LU-13566 socklnd: fix local interface binding LU-13713 lnet: check rtr_nid is a gateway Andreas Dilger (8): LU-13503 mdc: allow setting max_mod_rpcs_in_flight larger LU-13225 utils: fix install path for bash-completion LU-13675 o2iblnd: revert 'Timed out tx' patch LU-930 misc: update URLs in README LU-13721 utils: fix 'lfs find --pool' for PFL files LU-13673 misc: quiet compiler warning on armv7l LU-13680 osd-ldiskfs: handle large allocations LU-5338 tests: sanity-lfsck 11b allow larger last_id Andriy Skulysh (1): LU-13467 llite: truncate deadlock with DoM files Arshad Hussain (16): LU-6142 utils: Fix style issues for mount_lustre.c LU-6142 utils: Fix style issues for llog_reader.c LU-6142 utils: Fix style issues for lctl.c LU-6142 utils: Fix style issues for liblustreapi.c LU-6142 mdt: Fix style issues for mdt_xattr.c LU-6142 ost: Fix style issues for ost_handler.c LU-6142 osd-ldiskfs: Fix style issues for osd_oi.c LU-6142 mdt: Fix style issues for mdt_reint.c LU-6142 llite: Fix style issues for vvp_page.c LU-6142 lov: Fix style issues for lov_merge.c LU-6142 mdt: Fix style issues for mdt_recovery.c LU-6142 utils: Fix style issues for lfs.c LU-6142 mdt: Fix style issues for mdt_identity.c LU-13723 lustre: Convert ERR_PTR(PTR_ERR()) to ERR_CAST() LU-6142 osd-ldiskfs: Fix style issues for osd_io.c LU-6142 fld: Fix style issues for fld_cache.c Bobi Jam (2): LU-13473 llite: don't check mirror info for page discard LU-12586 lov: one more fix to write_intent end for trunc Chris Horn (5): LU-13509 ptlrpc: Clear bd_registered in ptlrpc_unregister_bulk LU-13501 lnet: Skip health and resends for single rail configs LU-13501 tests: Add tests for LNet health and resends LU-13606 lnet: Allow router to forward to healthier NID LU-13648 lnet: Set remote NI status in lnet_notify Dominique Martinet (1): LU-5710 build: fix typo suggesting openssl-devel requirement Elena Gryaznova (2): LU-12205 tests: host_nids_address() fix for MR setup LU-13297 tests: parallel-scale enhancement Emoly Liu (5): LU-13635 lfs: add -D option back to lfs_migrate LU-13597 ofd: add more information to job_stats LU-13609 llog: list all the log files correctly on MGS/MDT LU-13460 lfs: make function print_failed_tgt() work correctly LU-13732 lfs: fid2path should match the root path correctly Etienne AUJAMES (2): LU-13595 scripts: Add a debug option to lustre_rmmod LU-12549 utils: Check range of quota ID for "lfs" arguments Hongchao Zhang (4): LU-13508 mdc: chlg device could be used after free LU-13653 mdt: ignore quota when creating slave stripe LU-13608 tgt: abort recovery while reading update llog LU-13667 ptlrpc: fix endless loop issue James Nunez (2): LU-13701 tests: stop running sanity-lnet tests LU-13514 tests: remove upgrade images for conf-sanity James Simmons (13): LU-9897 build: add binaries to .gitignore LU-12511 build: ignore kmod handling in spec file for utilities only build LU-12511 utils: Move utilies specific values out of Lustre UAPI headers LU-8130 obd: convert obd_nid_hash to rhashtable LU-9325 mdt: replace simple_strtol() with kstrtol() LU-9441 llite: bind kthread thread to accepted node set LU-12511 lov: use lov_pattern_support() to verify lmm LU-9859 libcfs: Fix using smp_processor_id() in preemptible context LU-9897 utils: have lfs.c use lstddef.h LU-930 doc: update James Simmons contact info LU-9859 libcfs: move tgt_descs to standard Linux bitmaps. LU-6142 libcfs: resolve debug.c checkpatch issues LU-12514 utils: try lustre_tgt filesystem for mount Jian Yu (8): LU-13488 kernel: new kernel [RHEL 8.2 4.18.0-193.1.2.el8] LU-13111 kernel: new kernel [SLES12 SP5 4.12.14-122.20.1] LU-13556 kernel: kernel update RHEL7.8 [3.10.0-1127.8.2.el7] LU-13488 kernel: RHEL 8.2 server support LU-13345 kernel: kernel update SLES12 SP4 [4.12.14-95.48.1] LU-13657 kernel: kernel update RHEL8.2 [4.18.0-193.6.3.el8_2] LU-13731 llite: include linux/mm_types.h for VM_FAULT_RETRY LU-13731 autoconf: check if VM_FAULT_RETRY is defined John L. Hammond (3): LU-13690 mdd: remove warnings in obd_lookup() LU-13693 lfs: avoid opening regular files for getstripe LU-13693 lfs: check early for MDS_OPEN_DIRECTORY Kévin Baillergeau (1): LU-13559 utils: fix lfs mirror delete error message Lai Siyao (7): LU-13437 lmv: check stripe FID sanity LU-11025 dne: directory restripe and auto split LU-11025 mdt: remove unused code LU-13408 target: update in-memory per client data LU-13437 mdt: don't fetch LOOKUP lock for remote object LU-13437 mdt: rename misses remote LOOKUP lock revoke LU-13437 uapi: add OBD_CONNECT2_GETATTR_PFID Li Dongyang (1): LU-13604 doc: update e2fsprogs to 1.45.6.wc1 Mikhail Pershin (3): LU-13600 ptlrpc: limit rate of lock replays LU-12785 dom: fix DoM component deletion code LU-13600 ptlrpc: re-enterable signal_completed_replay() Mr NeilBrown (37): LU-6142 lustre: convert some container_of to *_safe LU-9859 libcfs: merge linux-debug.c into debug.c LU-11310 ldiskfs: Repair support for SUSE 15 again LU-12780 quota: don't use ptlrpc_thead of qmt_pool_recalc LU-6142 obdclass: convert calls to container_of0() LU-10391 lnet: fix uninitialize var in choose_ipv4_src() LU-9859 libcfs: merge linux-tracefile.c into tracefile.c LU-9859 libcfs: remove cfs_trace_refill_stack() LU-9859 libcfs: move cfs_trace_data data to tracefile.c LU-10934 tests: fix compilation without SELINUX LU-12780 scrub: all update to bitfields must be protected. LU-9679 obdclass: remove init to 0 from lustre_init_lsi() LU-8130 ptlrpc: convert conn_hash to rhashtable LU-8130 lu_object: convert lu_object cache to rhashtable LU-9859 libcfs: fold cfs_tracefile_*_arch into their only callers. LU-12678 o2iblnd: allocate init_qp_attr on stack. LU-12678 lnet: Fix some out-of-date comments. LU-12678 socklnd: don't fall-back to tcp_sendpage. LU-6142 lustre: convert use of container_of0 in include/ LU-6142 lov: convert container_of0() calls. LU-9679 nrs: remove ts_opcodes bitmap. LU-9679 lustre: remove some "#ifdef CONFIG*" from .c files. LU-9859 libcfs: cleanup tracefile.h LU-12678 o2iblnd: Use ib_mtu_int_to_enum() LU-12768 o2iblnd: wait properly for fps->increasing. LU-12678 o2iblnd: use need_resched() LU-12678 o2iblnd: Use list_for_each_entry_safe LU-12678 socklnd: use need_resched() LU-12678 socklnd: use list_for_each_entry_safe() LU-12678 socklnd: convert various refcounts to refcount_t LU-9859 libcfs: don't call unshare_fs_struct() LU-9679 llite: annotate non-owner locking LU-12678 lnet: remove LNetMEUnlink and clean up related code LU-12678 socklnd: change ksnd_nthreads to atomic_t LU-9679 lov: annotate nested locking of obd_dev_mutex LU-9679 lustre: make ptlrpc_connection_put() static inline LU-9679 mdc: create mdc_acl.c Mr. NeilBrown (1): LU-9859 libcfs: discard libcfs_prim.h Nathaniel Clark (1): LU-9812 spec: Fail rpmbuild if with servers but unconfigured NeilBrown (1): LU-9679 osc: simplify osc_extent_find() Oleg Drokin (2): LU-13776 tests: make sure pjdfstest.sh writes to tmp New tag 2.13.55 Patrick Farrell (2): LU-11963 obd: Rename OS_STATE flags to OS_STATFS LU-13196 llite: Remove mutex on dio read Qian Yingjin (2): LU-13546 pcc: exclude mmap_sanity tst8/tst9 from test list LU-13628 tests: replace btime with crtime for statx test Sebastien Buisson (20): LU-12275 sec: documentation for client-side encryption LU-12275 sec: enable client side encryption LU-13628 tests: add sanityn test_106 to ALWAYS_EXCEPT LU-12275 sec: control client side encryption LU-12275 sec: encryption for write path LU-12275 sec: decryption for read path LU-12275 sec: deal with encrypted object size LU-12275 sec: support truncate for encrypted files LU-13580 tests: fix retrieval of SELinux context LU-12275 tests: exercise file content encryption/decryption LU-12275 sec: ioctls to handle encryption policies LU-13711 build: fix typo on SSL dependency for Ubuntu LU-13525 sec: better struct sepol_downcall_data LU-13754 gss: open sptlrpc init channel in R+W mode LU-12275 sec: introduce null algo for filename encryption LU-12275 sec: force file name encryption policy to null LU-12275 sec: atomicity of encryption context getting/setting LU-13593 ptlrpc: fix growing message buffer LU-12275 sec: encryption support for DoM files LU-12275 sec: check if page is empty with ZERO_PAGE Sergey Cheremencev (2): LU-13677 quota: qunit sorting doesn't work LU-13586 tests: Quota Pools with PFL and SEL Shaun Tancheff (7): LU-12477 ldiskfs: drop SUSE kernel 4.4 and earlier LU-13344 osd-ldiskfs: timespec64 is broken LU-13562 build: SUSE build support for azure, cray_ari_s LU-13476 llite: Fix lock ordering in pagevec_dirty LU-13581 build: xarray and lockdep_is_held const clash LU-13344 all: Separate debugfs and procfs handling LU-13485 build: Make parallel tests names unique Sonia Sharma (1): LU-1742 o2iblnd: 'Timed out tx' error message Vitaly Fertman (3): LU-13649 mdd: orhpan cleanup fix LU-13366 lod: check for extension size at instantiation time LU-13366 tests: add SEL support to racer Vladimir Saveliev (1): LU-12687 osc: consume grants for direct I/O Wang Shilong (9): LU-13365 ldlm: check slv and limit before updating LU-13180 osc: disable ext merging for rdma only pages and non-rdma LU-13134 obdclass: use offset instead of cp_linkage LU-13134 obdclass: re-declare cl_page variables to reduce its size LU-13134 osc: re-declare ops_from/to to shrink osc_page LU-13519 osd-ldiskfs: expand inode project quota for upgrading LU-13134 llite: fix to free cl_dio_aio properly LU-13697 llite: fix short io for AIO LU-13729 osd-ldiskfs: race access to iam_formats during setup Yang Sheng (2): LU-10401 procs: print new line based on distro LU-11814 obdcalss: ensure LCT_QUIESCENT take sync From ankit at tripock.com Fri Jul 24 09:45:26 2020 From: ankit at tripock.com (Ankit Ladhania) Date: Fri, 24 Jul 2020 15:15:26 +0530 Subject: [lustre-devel] /usr/bin/fakeroot: debian/rules: Permission denied - Building Lustre client (v2.13.53) Message-ID: Hi, We are installing the Lustre Client (v2.13.53) on Ubuntu 18.04 (linux header: 5.4.0-42-generic) While doing `make debs` we are getting below error: make[1]: Leaving directory '/home/zeblok/lustre-release' dpkg-genbuildinfo dpkg-genchanges >../lustre_2.13.53-1_amd64.changes dpkg-genchanges: warning: package lustre-server-utils in control file but not in files list dpkg-genchanges: warning: package lustre-resource-agents in control file but not in files list dpkg-genchanges: warning: package lustre-client-modules-dkms in control file but not in files list dpkg-genchanges: info: including full source code in upload dpkg-source -I.git -I*.out[0-9]* -I*.swp --after-build lustre-release dpkg-buildpackage: info: full upload; Debian-native package (full source is included) ~/lustre-release/debian/tmp/modules-deb ~/lustre-release Removing old /home/zeblok/lustre-release/debian/tmp/modules-deb/usr_src/modules/lustre/ tree Extracting the package tarball, /usr/src/lustre.tar.bz2, please wait... /usr/bin/fakeroot: 175: /usr/bin/fakeroot: debian/rules: Permission denied /usr/bin/fakeroot: 175: /usr/bin/fakeroot: debian/rules: Permission denied BUILD FAILED! See /home/zeblok/lustre-release/debian/tmp/modules-deb/var_cache_modass/lustre.buildlog.5.4.0-42-generic.1595583181 for details. mv: cannot stat '../lustre-dev__*.deb': No such file or directory mv: cannot stat '../lustre-source__all.deb': No such file or directory mv: cannot stat '../lustre-iokit__*.deb': No such file or directory mv: cannot stat '../lustre-tests__*.deb': No such file or directory mv: cannot stat '../lustre-*-utils__*.deb': No such file or directory mv: cannot stat '../lustre_.dsc': No such file or directory mv: cannot stat '../lustre__*.changes': No such file or directory mv: cannot stat '../lustre_.tar.gz': No such file or directory mv: cannot stat '../lustre-*-modules-5.4.0-42-generic__*.deb': No such file or directory autoMakefile:1190: recipe for target 'debs' failed make: *** [debs] Error 1 Also tried doing 'chmod a+x debian/rules' but with no progress. Not able to understand why this is breaking, any help is highly appreciated. *Regards,Ankit Ladhania* IN: +91 933 444 2209 US: +1 (256) 203-0077 Skype: ankit.ladhania -------------- next part -------------- An HTML attachment was scrubbed... URL: From degremoa at amazon.com Fri Jul 24 10:00:05 2020 From: degremoa at amazon.com (Degremont, Aurelien) Date: Fri, 24 Jul 2020 10:00:05 +0000 Subject: [lustre-devel] /usr/bin/fakeroot: debian/rules: Permission denied - Building Lustre client (v2.13.53) In-Reply-To: References: Message-ID: <72DF4D14-559E-42B7-B18D-1565E5DA699B@amazon.com> If I remember correctly, it looks like you already tried building Debian package in this directory. Try cleaning everything and restart: make clean git clean -dfx sh ./autogen.sh ./configure --disable-server make debs Aurélien De : lustre-devel au nom de Ankit Ladhania Date : vendredi 24 juillet 2020 à 11:46 À : "lustre-devel at lists.lustre.org" Objet : [EXTERNAL] [lustre-devel] /usr/bin/fakeroot: debian/rules: Permission denied - Building Lustre client (v2.13.53) CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe. Hi, We are installing the Lustre Client (v2.13.53) on Ubuntu 18.04 (linux header: 5.4.0-42-generic) While doing `make debs` we are getting below error: make[1]: Leaving directory '/home/zeblok/lustre-release' dpkg-genbuildinfo dpkg-genchanges >../lustre_2.13.53-1_amd64.changes dpkg-genchanges: warning: package lustre-server-utils in control file but not in files list dpkg-genchanges: warning: package lustre-resource-agents in control file but not in files list dpkg-genchanges: warning: package lustre-client-modules-dkms in control file but not in files list dpkg-genchanges: info: including full source code in upload dpkg-source -I.git -I*.out[0-9]* -I*.swp --after-build lustre-release dpkg-buildpackage: info: full upload; Debian-native package (full source is included) ~/lustre-release/debian/tmp/modules-deb ~/lustre-release Removing old /home/zeblok/lustre-release/debian/tmp/modules-deb/usr_src/modules/lustre/ tree Extracting the package tarball, /usr/src/lustre.tar.bz2, please wait... /usr/bin/fakeroot: 175: /usr/bin/fakeroot: debian/rules: Permission denied /usr/bin/fakeroot: 175: /usr/bin/fakeroot: debian/rules: Permission denied BUILD FAILED! See /home/zeblok/lustre-release/debian/tmp/modules-deb/var_cache_modass/lustre.buildlog.5.4.0-42-generic.1595583181 for details. mv: cannot stat '../lustre-dev__*.deb': No such file or directory mv: cannot stat '../lustre-source__all.deb': No such file or directory mv: cannot stat '../lustre-iokit__*.deb': No such file or directory mv: cannot stat '../lustre-tests__*.deb': No such file or directory mv: cannot stat '../lustre-*-utils__*.deb': No such file or directory mv: cannot stat '../lustre_.dsc': No such file or directory mv: cannot stat '../lustre__*.changes': No such file or directory mv: cannot stat '../lustre_.tar.gz': No such file or directory mv: cannot stat '../lustre-*-modules-5.4.0-42-generic__*.deb': No such file or directory autoMakefile:1190: recipe for target 'debs' failed make: *** [debs] Error 1 Also tried doing 'chmod a+x debian/rules' but with no progress. Not able to understand why this is breaking, any help is highly appreciated. Regards, Ankit Ladhania IN: +91 933 444 2209 US: +1 (256) 203-0077 Skype: ankit.ladhania [Image supprimée par l'expéditeur.] -------------- next part -------------- An HTML attachment was scrubbed... URL: From ankit at tripock.com Fri Jul 24 10:50:42 2020 From: ankit at tripock.com (Ankit Ladhania) Date: Fri, 24 Jul 2020 16:20:42 +0530 Subject: [lustre-devel] /usr/bin/fakeroot: debian/rules: Permission denied - Building Lustre client (v2.13.53) In-Reply-To: <72DF4D14-559E-42B7-B18D-1565E5DA699B@amazon.com> References: <72DF4D14-559E-42B7-B18D-1565E5DA699B@amazon.com> Message-ID: Hi Aurelien, Thanks for replying. Its a clean install. It was working fine in the previous linux header v5.3.0-62-generic but suddenly has stopped working on the newer version (v5.4.0-42-generic). the steps I followed: *Steps to Build custom kernel - but same as the server it will run the client on.* 1. uname -r (output: 5.4.0-42-generic) 2. git clone git://kernel.ubuntu.com/ubuntu/ubuntu-bionic.git ubuntu-kernel 3. cd ubuntu-kernel/ 4. git tag | grep 5.4.0-42 5. git checkout Ubuntu-hwe-5.4-5.4.0-42.46_18.04.1 6. cp /boot/config-5.4.0-42-generic .config 7. touch .scmversion 8. sudo apt-get install libncurses-dev flex bison openssl libssl-dev dkms libelf-dev libudev-dev libpci-dev libiberty-dev autoconf 9. sudo apt-get build-dep linux linux-image-$(uname -r) 10. make -j $(nproc) *Steps to Build Lustre client:* 1. git clone git://git.whamcloud.com/fs/lustre-release.git 2. cd lustre-release/ 3. git checkout v2_13_53 4. sh autogen.sh 5. pwd 6. ./configure --disable-server --with-linux=/home/zeblok/ubuntu-kernel 7. make debs -j $(nproc) its breaking in the last step, with below error. make[1]: Leaving directory '/home/zeblok/lustre-release' dpkg-genbuildinfo dpkg-genchanges >../lustre_2.13.53-1_amd64.changes dpkg-genchanges: warning: package lustre-server-utils in control file but not in files list dpkg-genchanges: warning: package lustre-resource-agents in control file but not in files list dpkg-genchanges: warning: package lustre-client-modules-dkms in control file but not in files list dpkg-genchanges: info: including full source code in upload dpkg-source -I.git -I*.out[0-9]* -I*.swp --after-build lustre-release dpkg-buildpackage: info: full upload; Debian-native package (full source is included) ~/lustre-release/debian/tmp/modules-deb ~/lustre-release Removing old /home/zeblok/lustre-release/debian/tmp/modules-deb/usr_src/modules/lustre/ tree Extracting the package tarball, /usr/src/lustre.tar.bz2, please wait... /usr/bin/fakeroot: 175: /usr/bin/fakeroot: debian/rules: Permission denied /usr/bin/fakeroot: 175: /usr/bin/fakeroot: debian/rules: Permission denied BUILD FAILED! See /home/zeblok/lustre-release/debian/tmp/modules-deb/var_cache_modass/lustre.buildlog.5.4.0-42-generic.1595583181 for details. mv: cannot stat '../lustre-dev__*.deb': No such file or directory mv: cannot stat '../lustre-source__all.deb': No such file or directory mv: cannot stat '../lustre-iokit__*.deb': No such file or directory mv: cannot stat '../lustre-tests__*.deb': No such file or directory mv: cannot stat '../lustre-*-utils__*.deb': No such file or directory mv: cannot stat '../lustre_.dsc': No such file or directory mv: cannot stat '../lustre__*.changes': No such file or directory mv: cannot stat '../lustre_.tar.gz': No such file or directory mv: cannot stat '../lustre-*-modules-5.4.0-42-generic__*.deb': No such file or directory autoMakefile:1190: recipe for target 'debs' failed make: *** [debs] Error 1 I tried the step you said but got the same result. *Regards,Ankit Ladhania* IN: +91 933 444 2209 US: +1 (256) 203-0077 Skype: ankit.ladhania On Fri, Jul 24, 2020 at 3:30 PM Degremont, Aurelien wrote: > If I remember correctly, it looks like you already tried building Debian > package in this directory. > > Try cleaning everything and restart: > > > > make clean > > git clean -dfx > > sh ./autogen.sh > > ./configure --disable-server > > make debs > > > > Aurélien > > > > *De : *lustre-devel au nom de > Ankit Ladhania > *Date : *vendredi 24 juillet 2020 à 11:46 > *À : *"lustre-devel at lists.lustre.org" > *Objet : *[EXTERNAL] [lustre-devel] /usr/bin/fakeroot: debian/rules: > Permission denied - Building Lustre client (v2.13.53) > > > > *CAUTION*: This email originated from outside of the organization. Do not > click links or open attachments unless you can confirm the sender and know > the content is safe. > > > > > > Hi, > > We are installing the Lustre Client (v2.13.53) on Ubuntu 18.04 (linux > header: 5.4.0-42-generic) > > While doing `make debs` we are getting below error: > > make[1]: Leaving directory '/home/zeblok/lustre-release' > > dpkg-genbuildinfo > > dpkg-genchanges >../lustre_2.13.53-1_amd64.changes > > dpkg-genchanges: warning: package lustre-server-utils in control file but not in files list > > dpkg-genchanges: warning: package lustre-resource-agents in control file but not in files list > > dpkg-genchanges: warning: package lustre-client-modules-dkms in control file but not in files list > > dpkg-genchanges: info: including full source code in upload > > dpkg-source -I.git -I*.out[0-9]* -I*.swp --after-build lustre-release > > dpkg-buildpackage: info: full upload; Debian-native package (full source is included) > > ~/lustre-release/debian/tmp/modules-deb ~/lustre-release > > Removing old /home/zeblok/lustre-release/debian/tmp/modules-deb/usr_src/modules/lustre/ tree > > Extracting the package tarball, /usr/src/lustre.tar.bz2, please wait... > > /usr/bin/fakeroot: 175: /usr/bin/fakeroot: debian/rules: Permission denied > > /usr/bin/fakeroot: 175: /usr/bin/fakeroot: debian/rules: Permission denied > > BUILD FAILED! > > See /home/zeblok/lustre-release/debian/tmp/modules-deb/var_cache_modass/lustre.buildlog.5.4.0-42-generic.1595583181 for details. > > mv: cannot stat '../lustre-dev__*.deb': No such file or directory > > mv: cannot stat '../lustre-source__all.deb': No such file or directory > > mv: cannot stat '../lustre-iokit__*.deb': No such file or directory > > mv: cannot stat '../lustre-tests__*.deb': No such file or directory > > mv: cannot stat '../lustre-*-utils__*.deb': No such file or directory > > mv: cannot stat '../lustre_.dsc': No such file or directory > > mv: cannot stat '../lustre__*.changes': No such file or directory > > mv: cannot stat '../lustre_.tar.gz': No such file or directory > > mv: cannot stat '../lustre-*-modules-5.4.0-42-generic__*.deb': No such file or directory > > autoMakefile:1190: recipe for target 'debs' failed > > make: *** [debs] Error 1 > > > > Also tried doing 'chmod a+x debian/rules' but with no progress. > > > > Not able to understand why this is breaking, any help is highly > appreciated. > > > > *Regards, Ankit Ladhania* > > IN: +91 933 444 2209 > US: +1 (256) 203-0077 > > Skype: ankit.ladhania > > [image: Image supprimée par l'expéditeur.] > -------------- next part -------------- An HTML attachment was scrubbed... URL: From degremoa at amazon.com Fri Jul 24 11:35:55 2020 From: degremoa at amazon.com (Degremont, Aurelien) Date: Fri, 24 Jul 2020 11:35:55 +0000 Subject: [lustre-devel] [UNVERIFIED SENDER] Re: /usr/bin/fakeroot: debian/rules: Permission denied - Building Lustre client (v2.13.53) In-Reply-To: References: <72DF4D14-559E-42B7-B18D-1565E5DA699B@amazon.com> Message-ID: <621C5D4F-D091-4D2F-A5F5-E038DAFE6339@amazon.com> I remember seeing this problem… Could you check your full log to see if there is no error above? I'm pretty sure this permission error if not the real problem. De : Ankit Ladhania Date : vendredi 24 juillet 2020 à 12:51 À : "Degremont, Aurelien" Cc : "lustre-devel at lists.lustre.org" Objet : [EXTERNAL] [UNVERIFIED SENDER] Re: [lustre-devel] /usr/bin/fakeroot: debian/rules: Permission denied - Building Lustre client (v2.13.53) CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe. Hi Aurelien, Thanks for replying. Its a clean install. It was working fine in the previous linux header v5.3.0-62-generic but suddenly has stopped working on the newer version (v5.4.0-42-generic). the steps I followed: Steps to Build custom kernel - but same as the server it will run the client on. 1. uname -r (output: 5.4.0-42-generic) 2. git clone git://kernel.ubuntu.com/ubuntu/ubuntu-bionic.git ubuntu-kernel 3. cd ubuntu-kernel/ 4. git tag | grep 5.4.0-42 5. git checkout Ubuntu-hwe-5.4-5.4.0-42.46_18.04.1 6. cp /boot/config-5.4.0-42-generic .config 7. touch .scmversion 8. sudo apt-get install libncurses-dev flex bison openssl libssl-dev dkms libelf-dev libudev-dev libpci-dev libiberty-dev autoconf 9. sudo apt-get build-dep linux linux-image-$(uname -r) 10. make -j $(nproc) Steps to Build Lustre client: 1. git clone git://git.whamcloud.com/fs/lustre-release.git 2. cd lustre-release/ 3. git checkout v2_13_53 4. sh autogen.sh 5. pwd 6. ./configure --disable-server --with-linux=/home/zeblok/ubuntu-kernel 7. make debs -j $(nproc) its breaking in the last step, with below error. make[1]: Leaving directory '/home/zeblok/lustre-release' dpkg-genbuildinfo dpkg-genchanges >../lustre_2.13.53-1_amd64.changes dpkg-genchanges: warning: package lustre-server-utils in control file but not in files list dpkg-genchanges: warning: package lustre-resource-agents in control file but not in files list dpkg-genchanges: warning: package lustre-client-modules-dkms in control file but not in files list dpkg-genchanges: info: including full source code in upload dpkg-source -I.git -I*.out[0-9]* -I*.swp --after-build lustre-release dpkg-buildpackage: info: full upload; Debian-native package (full source is included) ~/lustre-release/debian/tmp/modules-deb ~/lustre-release Removing old /home/zeblok/lustre-release/debian/tmp/modules-deb/usr_src/modules/lustre/ tree Extracting the package tarball, /usr/src/lustre.tar.bz2, please wait... /usr/bin/fakeroot: 175: /usr/bin/fakeroot: debian/rules: Permission denied /usr/bin/fakeroot: 175: /usr/bin/fakeroot: debian/rules: Permission denied BUILD FAILED! See /home/zeblok/lustre-release/debian/tmp/modules-deb/var_cache_modass/lustre.buildlog.5.4.0-42-generic.1595583181 for details. mv: cannot stat '../lustre-dev__*.deb': No such file or directory mv: cannot stat '../lustre-source__all.deb': No such file or directory mv: cannot stat '../lustre-iokit__*.deb': No such file or directory mv: cannot stat '../lustre-tests__*.deb': No such file or directory mv: cannot stat '../lustre-*-utils__*.deb': No such file or directory mv: cannot stat '../lustre_.dsc': No such file or directory mv: cannot stat '../lustre__*.changes': No such file or directory mv: cannot stat '../lustre_.tar.gz': No such file or directory mv: cannot stat '../lustre-*-modules-5.4.0-42-generic__*.deb': No such file or directory autoMakefile:1190: recipe for target 'debs' failed make: *** [debs] Error 1 I tried the step you said but got the same result. Regards, Ankit Ladhania IN: +91 933 444 2209 US: +1 (256) 203-0077 Skype: ankit.ladhania On Fri, Jul 24, 2020 at 3:30 PM Degremont, Aurelien > wrote: If I remember correctly, it looks like you already tried building Debian package in this directory. Try cleaning everything and restart: make clean git clean -dfx sh ./autogen.sh ./configure --disable-server make debs Aurélien De : lustre-devel > au nom de Ankit Ladhania > Date : vendredi 24 juillet 2020 à 11:46 À : "lustre-devel at lists.lustre.org" > Objet : [EXTERNAL] [lustre-devel] /usr/bin/fakeroot: debian/rules: Permission denied - Building Lustre client (v2.13.53) CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe. Hi, We are installing the Lustre Client (v2.13.53) on Ubuntu 18.04 (linux header: 5.4.0-42-generic) While doing `make debs` we are getting below error: make[1]: Leaving directory '/home/zeblok/lustre-release' dpkg-genbuildinfo dpkg-genchanges >../lustre_2.13.53-1_amd64.changes dpkg-genchanges: warning: package lustre-server-utils in control file but not in files list dpkg-genchanges: warning: package lustre-resource-agents in control file but not in files list dpkg-genchanges: warning: package lustre-client-modules-dkms in control file but not in files list dpkg-genchanges: info: including full source code in upload dpkg-source -I.git -I*.out[0-9]* -I*.swp --after-build lustre-release dpkg-buildpackage: info: full upload; Debian-native package (full source is included) ~/lustre-release/debian/tmp/modules-deb ~/lustre-release Removing old /home/zeblok/lustre-release/debian/tmp/modules-deb/usr_src/modules/lustre/ tree Extracting the package tarball, /usr/src/lustre.tar.bz2, please wait... /usr/bin/fakeroot: 175: /usr/bin/fakeroot: debian/rules: Permission denied /usr/bin/fakeroot: 175: /usr/bin/fakeroot: debian/rules: Permission denied BUILD FAILED! See /home/zeblok/lustre-release/debian/tmp/modules-deb/var_cache_modass/lustre.buildlog.5.4.0-42-generic.1595583181 for details. mv: cannot stat '../lustre-dev__*.deb': No such file or directory mv: cannot stat '../lustre-source__all.deb': No such file or directory mv: cannot stat '../lustre-iokit__*.deb': No such file or directory mv: cannot stat '../lustre-tests__*.deb': No such file or directory mv: cannot stat '../lustre-*-utils__*.deb': No such file or directory mv: cannot stat '../lustre_.dsc': No such file or directory mv: cannot stat '../lustre__*.changes': No such file or directory mv: cannot stat '../lustre_.tar.gz': No such file or directory mv: cannot stat '../lustre-*-modules-5.4.0-42-generic__*.deb': No such file or directory autoMakefile:1190: recipe for target 'debs' failed make: *** [debs] Error 1 Also tried doing 'chmod a+x debian/rules' but with no progress. Not able to understand why this is breaking, any help is highly appreciated. Regards, Ankit Ladhania IN: +91 933 444 2209 US: +1 (256) 203-0077 Skype: ankit.ladhania Erreur ! Nom du fichier non spécifié. [Image supprimée par l'expéditeur.] -------------- next part -------------- An HTML attachment was scrubbed... URL: From pjones at whamcloud.com Fri Jul 24 12:44:47 2020 From: pjones at whamcloud.com (Peter Jones) Date: Fri, 24 Jul 2020 12:44:47 +0000 Subject: [lustre-devel] /usr/bin/fakeroot: debian/rules: Permission denied - Building Lustre client (v2.13.53) In-Reply-To: References: Message-ID: Hopefully you understand this already but just in case.- 2.13.53 is a development build and not intended for production usage. You should be able to build Ubuntu 18.04 clients for 2.12.x releases. From: lustre-devel on behalf of Ankit Ladhania Date: Friday, July 24, 2020 at 2:45 AM To: "lustre-devel at lists.lustre.org" Subject: [lustre-devel] /usr/bin/fakeroot: debian/rules: Permission denied - Building Lustre client (v2.13.53) Hi, We are installing the Lustre Client (v2.13.53) on Ubuntu 18.04 (linux header: 5.4.0-42-generic) While doing `make debs` we are getting below error: make[1]: Leaving directory '/home/zeblok/lustre-release' dpkg-genbuildinfo dpkg-genchanges >../lustre_2.13.53-1_amd64.changes dpkg-genchanges: warning: package lustre-server-utils in control file but not in files list dpkg-genchanges: warning: package lustre-resource-agents in control file but not in files list dpkg-genchanges: warning: package lustre-client-modules-dkms in control file but not in files list dpkg-genchanges: info: including full source code in upload dpkg-source -I.git -I*.out[0-9]* -I*.swp --after-build lustre-release dpkg-buildpackage: info: full upload; Debian-native package (full source is included) ~/lustre-release/debian/tmp/modules-deb ~/lustre-release Removing old /home/zeblok/lustre-release/debian/tmp/modules-deb/usr_src/modules/lustre/ tree Extracting the package tarball, /usr/src/lustre.tar.bz2, please wait... /usr/bin/fakeroot: 175: /usr/bin/fakeroot: debian/rules: Permission denied /usr/bin/fakeroot: 175: /usr/bin/fakeroot: debian/rules: Permission denied BUILD FAILED! See /home/zeblok/lustre-release/debian/tmp/modules-deb/var_cache_modass/lustre.buildlog.5.4.0-42-generic.1595583181 for details. mv: cannot stat '../lustre-dev__*.deb': No such file or directory mv: cannot stat '../lustre-source__all.deb': No such file or directory mv: cannot stat '../lustre-iokit__*.deb': No such file or directory mv: cannot stat '../lustre-tests__*.deb': No such file or directory mv: cannot stat '../lustre-*-utils__*.deb': No such file or directory mv: cannot stat '../lustre_.dsc': No such file or directory mv: cannot stat '../lustre__*.changes': No such file or directory mv: cannot stat '../lustre_.tar.gz': No such file or directory mv: cannot stat '../lustre-*-modules-5.4.0-42-generic__*.deb': No such file or directory autoMakefile:1190: recipe for target 'debs' failed make: *** [debs] Error 1 Also tried doing 'chmod a+x debian/rules' but with no progress. Not able to understand why this is breaking, any help is highly appreciated. Regards, Ankit Ladhania IN: +91 933 444 2209 US: +1 (256) 203-0077 Skype: ankit.ladhania [Image removed by sender.] -------------- next part -------------- An HTML attachment was scrubbed... URL: From cory.spitz at hpe.com Fri Jul 24 21:47:42 2020 From: cory.spitz at hpe.com (Spitz, Cory James) Date: Fri, 24 Jul 2020 21:47:42 +0000 Subject: [lustre-devel] git packfile In-Reply-To: <7288E2CB-A8C3-4B37-AF4B-291EED278833@cray.com> References: <30534141-20CF-43C3-BA49-0149AF7CEFE6@cray.com> <61FFBC77-CF2E-4EDC-9060-BF9EBF7B27D0@whamcloud.com> <66BF972C-4BD3-4AD7-88A5-DDDF8D1CC988@amazon.com> <5D50C124-1325-447C-9CEB-B91646B49E5E@amazon.com> <7288E2CB-A8C3-4B37-AF4B-291EED278833@cray.com> Message-ID: Hi, Strangely this git packfile issue has come back recently. I was able to workaround it again by just checking out and fetching a few times. I found this stack overflow mention about it: https://stackoverflow.com/questions/11401434/git-packfile-claims-to-have-more-objects-inaccessable , but my local git was newer than the version mentioned there. HPE is also having this problem for an automated checkout of git.whamcloud.com with a much, much older git implementation, but the workaround for re-fetching git.whamcloud.com is not working for us there. Does anyone know more? I tried `git gc` in my local checkout and it seemed fine. I'd like the automated checkout to try `git gc` too. I'm not a git expert, do we just need to run this garbage collection on the 'remote' repo too? Thanks, -Cory On 3/25/19, 10:17 PM, "lustre-devel on behalf of Cory Spitz" wrote: I ended up checking out a few different branches and then going back to master. At that point, the pull seemed to be OK. We did also experience this once in our 'official' Cray pull from upstream, but the problem went away there too after some fiddling. I'm not sure what is making it come and go. Aurélien and Andreas, thank you both for your input. I'll keep an eye out for it and try the workaround again if it pops back up. -Cory -- On 3/25/19, 12:22 PM, "Degremont, Aurelien" wrote: It depends. It persisted for a while. Now, I do not see them anymore. I don't know what changed in-between. Le 25/03/2019 11:02, « Andreas Dilger » a écrit : On Mar 25, 2019, at 03:47, Degremont, Aurelien wrote: > > I'm facing similar problem than Cory, times to times. > I did not track the detailed error messages but, for example, here are the errors I got when running 'git fetch' on a directory where only git://git.whamcloud.com/fs/lustre-release.git is declared. I did not update the repo for few days/weeks. > > remote: warning: packfile ./objects/pack/pack-2b5d1356632dfdbfd86c2d36e529a0a8af47f9af.pack cannot be accessed > remote: error: refs/changes/47/33547/4 does not point to a valid object! > remote: warning: packfile ./objects/pack/pack-d2307b0367688a7b6599e36453a20e02d6a0302e.pack cannot be accessed > remote: error: refs/changes/47/33547/5 does not point to a valid object! > remote: warning: packfile ./objects/pack/pack-69f40cb9d10db9a97f6df59c80a6279f808b9d5d.pack cannot be accessed > remote: error: refs/changes/47/33547/6 does not point to a valid object! > remote: warning: packfile ./objects/pack/pack-090b03d5723083bb4a1e7b524bc83dc9de504f17.pack cannot be accessed > remote: error: refs/changes/47/33547/7 does not point to a valid object! > remote: warning: packfile ./objects/pack/pack-0397e40c8813c5601975c99c94afed5ddfbbf1dc.pack cannot be accessed > remote: error: refs/changes/47/33547/8 does not point to a valid object! > remote: warning: packfile ./objects/pack/pack-a20995b7160fa2708c42f05111b39126a96b9bd4.pack cannot be accessed > remote: warning: packfile ./objects/pack/pack-765a1f16499228e6910765c218733cfb8852e953.pack cannot be accessed > > This problem started few months ago. If you do a fetch on the same tree again right away does the error persist or go away? Cheers, Andreas > Le 25/03/2019 07:05, « lustre-devel au nom de Andreas Dilger » a écrit : > > On Mar 22, 2019, at 09:59, Cory Spitz wrote: >> >> Hello. >> >> I’ve occasionally seen errors like the following when pulling upstream master: >> >> remote: warning: packfile ./objects/pack/pack-8a0ac446942c02bf656a79a2d7fa79d33f1fa4e2.pack cannot be accessed >> remote: error: refs/changes/47/5347/9 does not point to a valid object! >> remote: warning: packfile ./objects/pack/pack-8a0ac446942c02bf656a79a2d7fa79d33f1fa4e2.pack cannot be accessed >> remote: error: refs/changes/47/7247/23 does not point to a valid object! >> remote: warning: packfile ./objects/pack/pack-8a0ac446942c02bf656a79a2d7fa79d33f1fa4e2.pack cannot be accessed >> remote: error: refs/changes/47/7747/14 does not point to a valid object! >> remote: warning: packfile ./objects/pack/pack-8a0ac446942c02bf656a79a2d7fa79d33f1fa4e2.pack cannot be accessed >> >> Other clones/checkouts seem to be OK. Has anyone else seen this? It seems that this has something to do with Gerrit changes. I just did a clone of master and I never set things up to my knowledge to pull all of these refs. Anyone have a workaround (besides a fresh clone) or any tips? > > Cory, does this happen repeatedly, or just one time? There are definitely errors that are generated because the master-next branch is "rewound" every time that Oleg lands those patches to master. > > It is a bit strange about the changes that are referenced above. https://review.whamcloud.com/7247 and https://review.whamcloud.com/5347 were landed 5 years ago, so it is unlikely that anything related to them was changed recently. Looking at the changes in Gerrit they appear to be OK. > > Cheers, Andreas > --- > Andreas Dilger > Principal Lustre Architect > Whamcloud > > > > > > > > _______________________________________________ > lustre-devel mailing list > lustre-devel at lists.lustre.org > http://lists.lustre.org/listinfo.cgi/lustre-devel-lustre.org > > Cheers, Andreas --- Andreas Dilger Principal Lustre Architect Whamcloud _______________________________________________ lustre-devel mailing list lustre-devel at lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-devel-lustre.org From c.kuntz at opendrives.com Fri Jul 24 22:29:13 2020 From: c.kuntz at opendrives.com (Christian Kuntz) Date: Fri, 24 Jul 2020 15:29:13 -0700 Subject: [lustre-devel] /usr/bin/fakeroot: debian/rules: Permission denied - Building Lustre client (v2.13.53) In-Reply-To: References: <72DF4D14-559E-42B7-B18D-1565E5DA699B@amazon.com> Message-ID: Oh hey, I think I might know what this is! In debian/rules, line 344 uses some sed fanciness to make a new rules file in a different directory. Since the send file is just a text stream into a file, it doesn't maintain the x bit of the parent. To fix, I placed this in line 345: chmod +x debian/$(SOURCE_PKG)/usr/src/modules/lustre/debian/rules Hope this helps! Christian On Fri, Jul 24, 2020, 3:50 AM Ankit Ladhania wrote: > Hi Aurelien, > Thanks for replying. > > Its a clean install. It was working fine in the previous linux header > v5.3.0-62-generic but suddenly has stopped working on the newer version > (v5.4.0-42-generic). > > the steps I followed: > > *Steps to Build custom kernel - but same as the server it will run the > client on.* > 1. uname -r (output: 5.4.0-42-generic) > 2. git clone git://kernel.ubuntu.com/ubuntu/ubuntu-bionic.git > ubuntu-kernel > 3. cd ubuntu-kernel/ > 4. git tag | grep 5.4.0-42 > 5. git checkout Ubuntu-hwe-5.4-5.4.0-42.46_18.04.1 > 6. cp /boot/config-5.4.0-42-generic .config > 7. touch .scmversion > 8. sudo apt-get install libncurses-dev flex bison openssl libssl-dev dkms > libelf-dev libudev-dev libpci-dev libiberty-dev autoconf > 9. sudo apt-get build-dep linux linux-image-$(uname -r) > 10. make -j $(nproc) > > *Steps to Build Lustre client:* > 1. git clone git://git.whamcloud.com/fs/lustre-release.git > 2. cd lustre-release/ > 3. git checkout v2_13_53 > 4. sh autogen.sh > 5. pwd > 6. ./configure --disable-server --with-linux=/home/zeblok/ubuntu-kernel > 7. make debs -j $(nproc) > > its breaking in the last step, with below error. > > make[1]: Leaving directory '/home/zeblok/lustre-release' > dpkg-genbuildinfo > dpkg-genchanges >../lustre_2.13.53-1_amd64.changes > dpkg-genchanges: warning: package lustre-server-utils in control file but not in files list > dpkg-genchanges: warning: package lustre-resource-agents in control file but not in files list > dpkg-genchanges: warning: package lustre-client-modules-dkms in control file but not in files list > dpkg-genchanges: info: including full source code in upload > dpkg-source -I.git -I*.out[0-9]* -I*.swp --after-build lustre-release > dpkg-buildpackage: info: full upload; Debian-native package (full source is included) > ~/lustre-release/debian/tmp/modules-deb ~/lustre-release > Removing old /home/zeblok/lustre-release/debian/tmp/modules-deb/usr_src/modules/lustre/ tree > Extracting the package tarball, /usr/src/lustre.tar.bz2, please wait... > /usr/bin/fakeroot: 175: /usr/bin/fakeroot: debian/rules: Permission denied > /usr/bin/fakeroot: 175: /usr/bin/fakeroot: debian/rules: Permission denied > BUILD FAILED! > See /home/zeblok/lustre-release/debian/tmp/modules-deb/var_cache_modass/lustre.buildlog.5.4.0-42-generic.1595583181 for details. > mv: cannot stat '../lustre-dev__*.deb': No such file or directory > mv: cannot stat '../lustre-source__all.deb': No such file or directory > mv: cannot stat '../lustre-iokit__*.deb': No such file or directory > mv: cannot stat '../lustre-tests__*.deb': No such file or directory > mv: cannot stat '../lustre-*-utils__*.deb': No such file or directory > mv: cannot stat '../lustre_.dsc': No such file or directory > mv: cannot stat '../lustre__*.changes': No such file or directory > mv: cannot stat '../lustre_.tar.gz': No such file or directory > mv: cannot stat '../lustre-*-modules-5.4.0-42-generic__*.deb': No such file or directory > autoMakefile:1190: recipe for target 'debs' failed > make: *** [debs] Error 1 > > > I tried the step you said but got the same result. > > > > *Regards,Ankit Ladhania* > IN: +91 933 444 2209 > US: +1 (256) 203-0077 > Skype: ankit.ladhania > > > On Fri, Jul 24, 2020 at 3:30 PM Degremont, Aurelien > wrote: > >> If I remember correctly, it looks like you already tried building Debian >> package in this directory. >> >> Try cleaning everything and restart: >> >> >> >> make clean >> >> git clean -dfx >> >> sh ./autogen.sh >> >> ./configure --disable-server >> >> make debs >> >> >> >> Aurélien >> >> >> >> *De : *lustre-devel au nom de >> Ankit Ladhania >> *Date : *vendredi 24 juillet 2020 à 11:46 >> *À : *"lustre-devel at lists.lustre.org" >> *Objet : *[EXTERNAL] [lustre-devel] /usr/bin/fakeroot: debian/rules: >> Permission denied - Building Lustre client (v2.13.53) >> >> >> >> *CAUTION*: This email originated from outside of the organization. Do >> not click links or open attachments unless you can confirm the sender and >> know the content is safe. >> >> >> >> >> >> Hi, >> >> We are installing the Lustre Client (v2.13.53) on Ubuntu 18.04 (linux >> header: 5.4.0-42-generic) >> >> While doing `make debs` we are getting below error: >> >> make[1]: Leaving directory '/home/zeblok/lustre-release' >> >> dpkg-genbuildinfo >> >> dpkg-genchanges >../lustre_2.13.53-1_amd64.changes >> >> dpkg-genchanges: warning: package lustre-server-utils in control file but not in files list >> >> dpkg-genchanges: warning: package lustre-resource-agents in control file but not in files list >> >> dpkg-genchanges: warning: package lustre-client-modules-dkms in control file but not in files list >> >> dpkg-genchanges: info: including full source code in upload >> >> dpkg-source -I.git -I*.out[0-9]* -I*.swp --after-build lustre-release >> >> dpkg-buildpackage: info: full upload; Debian-native package (full source is included) >> >> ~/lustre-release/debian/tmp/modules-deb ~/lustre-release >> >> Removing old /home/zeblok/lustre-release/debian/tmp/modules-deb/usr_src/modules/lustre/ tree >> >> Extracting the package tarball, /usr/src/lustre.tar.bz2, please wait... >> >> /usr/bin/fakeroot: 175: /usr/bin/fakeroot: debian/rules: Permission denied >> >> /usr/bin/fakeroot: 175: /usr/bin/fakeroot: debian/rules: Permission denied >> >> BUILD FAILED! >> >> See /home/zeblok/lustre-release/debian/tmp/modules-deb/var_cache_modass/lustre.buildlog.5.4.0-42-generic.1595583181 for details. >> >> mv: cannot stat '../lustre-dev__*.deb': No such file or directory >> >> mv: cannot stat '../lustre-source__all.deb': No such file or directory >> >> mv: cannot stat '../lustre-iokit__*.deb': No such file or directory >> >> mv: cannot stat '../lustre-tests__*.deb': No such file or directory >> >> mv: cannot stat '../lustre-*-utils__*.deb': No such file or directory >> >> mv: cannot stat '../lustre_.dsc': No such file or directory >> >> mv: cannot stat '../lustre__*.changes': No such file or directory >> >> mv: cannot stat '../lustre_.tar.gz': No such file or directory >> >> mv: cannot stat '../lustre-*-modules-5.4.0-42-generic__*.deb': No such file or directory >> >> autoMakefile:1190: recipe for target 'debs' failed >> >> make: *** [debs] Error 1 >> >> >> >> Also tried doing 'chmod a+x debian/rules' but with no progress. >> >> >> >> Not able to understand why this is breaking, any help is highly >> appreciated. >> >> >> >> *Regards, Ankit Ladhania* >> >> IN: +91 933 444 2209 >> US: +1 (256) 203-0077 >> >> Skype: ankit.ladhania >> >> [image: Image supprimée par l'expéditeur.] >> > _______________________________________________ > lustre-devel mailing list > lustre-devel at lists.lustre.org > http://lists.lustre.org/listinfo.cgi/lustre-devel-lustre.org > -- -------------- next part -------------- An HTML attachment was scrubbed... URL: From ankit at tripock.com Mon Jul 27 17:08:14 2020 From: ankit at tripock.com (Ankit Ladhania) Date: Mon, 27 Jul 2020 22:38:14 +0530 Subject: [lustre-devel] /usr/bin/fakeroot: debian/rules: Permission denied - Building Lustre client (v2.13.53) In-Reply-To: References: <72DF4D14-559E-42B7-B18D-1565E5DA699B@amazon.com> Message-ID: Hi, I have been trying to install Lustre client (v2.13.55) on Ubuntu 18.04 and 16.04 for the past 6 days and have not been successful. I have been getting the below errors: All I want is a direction to debug the issue (Have tried almost all the solutions present in the web). I dont understand what is the requirement for Lustre as on some nodes I'm able to install it and on nodes it just wouldn't install. (I checked the versions of the installed packages (dpkg -l) and found only minor version changes) Again Any Help is really appreciated. *Ubuntu 18.04:* *Steps to Build custom kernel - but same as the server it will run the client on.* 1. uname -r (output: 5.4.0-42-generic) 2. git clone git://kernel.ubuntu.com/ubuntu/ubuntu-bionic.git ubuntu-kernel 3. cd ubuntu-kernel/ 4. git tag | grep 5.4.0-42 5. git checkout Ubuntu-hwe-5.4-5.4.0-42.46_18.04.1 6. cp /boot/config-5.4.0-42-generic .config 7. touch .scmversion 8. sudo apt-get install libncurses-dev flex bison openssl libssl-dev dkms libelf-dev libudev-dev libpci-dev libiberty-dev autoconf 9. sudo apt-get build-dep linux linux-image-$(uname -r) 10. make -j $(nproc) *Steps to Build Lustre client:* 1. git clone git://git.whamcloud.com/fs/lustre-release.git 2. cd lustre-release/ 3. git checkout v2_13_53 4. sh autogen.sh 5. pwd 6. ./configure --disable-server --with-linux=/home/zeblok/ubuntu-kernel 7. make debs -j $(nproc) *Error:* make[1]: Leaving directory '/home/zeblok/lustre-release' dpkg-genbuildinfo dpkg-genchanges >../lustre_2.13.53-1_amd64.changes dpkg-genchanges: warning: package lustre-server-utils in control file but not in files list dpkg-genchanges: warning: package lustre-resource-agents in control file but not in files list dpkg-genchanges: warning: package lustre-client-modules-dkms in control file but not in files list dpkg-genchanges: info: including full source code in upload dpkg-source -I.git -I*.out[0-9]* -I*.swp --after-build lustre-release dpkg-buildpackage: info: full upload; Debian-native package (full source is included) ~/lustre-release/debian/tmp/modules-deb ~/lustre-release Removing old /home/zeblok/lustre-release/debian/tmp/modules-deb/usr_src/modules/lustre/ tree Extracting the package tarball, /usr/src/lustre.tar.bz2, please wait... /usr/bin/fakeroot: 175: /usr/bin/fakeroot: debian/rules: Permission denied /usr/bin/fakeroot: 175: /usr/bin/fakeroot: debian/rules: Permission denied BUILD FAILED! See /home/zeblok/lustre-release/debian/tmp/modules-deb/var_cache_modass/lustre.buildlog.5.4.0-42-generic.1595583181 for details. mv: cannot stat '../lustre-dev__*.deb': No such file or directory mv: cannot stat '../lustre-source__all.deb': No such file or directory mv: cannot stat '../lustre-iokit__*.deb': No such file or directory mv: cannot stat '../lustre-tests__*.deb': No such file or directory mv: cannot stat '../lustre-*-utils__*.deb': No such file or directory mv: cannot stat '../lustre_.dsc': No such file or directory mv: cannot stat '../lustre__*.changes': No such file or directory mv: cannot stat '../lustre_.tar.gz': No such file or directory mv: cannot stat '../lustre-*-modules-5.4.0-42-generic__*.deb': No such file or directory autoMakefile:1190: recipe for target 'debs' failed make: *** [debs] Error 1 *Ubuntu 16.04:* *Similar building process for Ubuntu 16.04:* LD [M] /home/zeblok/lustre-release/lustre/ptlrpc/ptlrpc.o Makefile:1471: recipe for target '_module_/home/zeblok/lustre-release' failed make[5]: *** [_module_/home/zeblok/lustre-release] Error 2 make[5]: Leaving directory '/usr/src/linux-headers-4.4.0-186-generic' autoMakefile:1051: recipe for target 'modules' failed make[4]: *** [modules] Error 2 make[4]: Leaving directory '/home/zeblok/lustre-release' autoMakefile:592: recipe for target 'all-recursive' failed make[3]: *** [all-recursive] Error 1 make[3]: Leaving directory '/home/zeblok/lustre-release' autoMakefile:490: recipe for target 'all' failed make[2]: *** [all] Error 2 make[2]: Leaving directory '/home/zeblok/lustre-release' debian/rules:218: recipe for target 'build-stamp' failed make[1]: *** [build-stamp] Error 2 make[1]: Leaving directory '/home/zeblok/lustre-release' dpkg-buildpackage: error: debian/rules build gave error exit status 2 cp: cannot stat '/control.bkp': No such file or directory cp: cannot stat '/control.main.bkp': No such file or directory cp: cannot stat '/control.modules.in.bkp': No such file or directory autoMakefile:1188: recipe for target 'debs' failed make: *** [debs] Error 2 On Sat, Jul 25, 2020 at 3:59 AM Christian Kuntz wrote: > Oh hey, I think I might know what this is! > > In debian/rules, line 344 uses some sed fanciness to make a new rules file > in a different directory. Since the send file is just a text stream into a > file, it doesn't maintain the x bit of the parent. To fix, I placed this in > line 345: > > chmod +x debian/$(SOURCE_PKG)/usr/src/modules/lustre/debian/rules > > Hope this helps! > > Christian > > On Fri, Jul 24, 2020, 3:50 AM Ankit Ladhania wrote: > >> Hi Aurelien, >> Thanks for replying. >> >> Its a clean install. It was working fine in the previous linux header >> v5.3.0-62-generic but suddenly has stopped working on the newer version >> (v5.4.0-42-generic). >> >> the steps I followed: >> >> *Steps to Build custom kernel - but same as the server it will run the >> client on.* >> 1. uname -r (output: 5.4.0-42-generic) >> 2. git clone git://kernel.ubuntu.com/ubuntu/ubuntu-bionic.git >> >> ubuntu-kernel >> 3. cd ubuntu-kernel/ >> 4. git tag | grep 5.4.0-42 >> 5. git checkout Ubuntu-hwe-5.4-5.4.0-42.46_18.04.1 >> 6. cp /boot/config-5.4.0-42-generic .config >> 7. touch .scmversion >> 8. sudo apt-get install libncurses-dev flex bison openssl libssl-dev dkms >> libelf-dev libudev-dev libpci-dev libiberty-dev autoconf >> 9. sudo apt-get build-dep linux linux-image-$(uname -r) >> 10. make -j $(nproc) >> >> *Steps to Build Lustre client:* >> 1. git clone git://git.whamcloud.com/fs/lustre-release.git >> >> 2. cd lustre-release/ >> 3. git checkout v2_13_53 >> 4. sh autogen.sh >> 5. pwd >> 6. ./configure --disable-server --with-linux=/home/zeblok/ubuntu-kernel >> 7. make debs -j $(nproc) >> >> its breaking in the last step, with below error. >> >> make[1]: Leaving directory '/home/zeblok/lustre-release' >> dpkg-genbuildinfo >> dpkg-genchanges >../lustre_2.13.53-1_amd64.changes >> dpkg-genchanges: warning: package lustre-server-utils in control file but not in files list >> dpkg-genchanges: warning: package lustre-resource-agents in control file but not in files list >> dpkg-genchanges: warning: package lustre-client-modules-dkms in control file but not in files list >> dpkg-genchanges: info: including full source code in upload >> dpkg-source -I.git -I*.out[0-9]* -I*.swp --after-build lustre-release >> dpkg-buildpackage: info: full upload; Debian-native package (full source is included) >> ~/lustre-release/debian/tmp/modules-deb ~/lustre-release >> Removing old /home/zeblok/lustre-release/debian/tmp/modules-deb/usr_src/modules/lustre/ tree >> Extracting the package tarball, /usr/src/lustre.tar.bz2, please wait... >> /usr/bin/fakeroot: 175: /usr/bin/fakeroot: debian/rules: Permission denied >> /usr/bin/fakeroot: 175: /usr/bin/fakeroot: debian/rules: Permission denied >> BUILD FAILED! >> See /home/zeblok/lustre-release/debian/tmp/modules-deb/var_cache_modass/lustre.buildlog.5.4.0-42-generic.1595583181 for details. >> mv: cannot stat '../lustre-dev__*.deb': No such file or directory >> mv: cannot stat '../lustre-source__all.deb': No such file or directory >> mv: cannot stat '../lustre-iokit__*.deb': No such file or directory >> mv: cannot stat '../lustre-tests__*.deb': No such file or directory >> mv: cannot stat '../lustre-*-utils__*.deb': No such file or directory >> mv: cannot stat '../lustre_.dsc': No such file or directory >> mv: cannot stat '../lustre__*.changes': No such file or directory >> mv: cannot stat '../lustre_.tar.gz': No such file or directory >> mv: cannot stat '../lustre-*-modules-5.4.0-42-generic__*.deb': No such file or directory >> autoMakefile:1190: recipe for target 'debs' failed >> make: *** [debs] Error 1 >> >> >> I tried the step you said but got the same result. >> >> >> >> *Regards,Ankit Ladhania* >> IN: +91 933 444 2209 >> US: +1 (256) 203-0077 >> Skype: ankit.ladhania >> >> >> On Fri, Jul 24, 2020 at 3:30 PM Degremont, Aurelien >> wrote: >> >>> If I remember correctly, it looks like you already tried building Debian >>> package in this directory. >>> >>> Try cleaning everything and restart: >>> >>> >>> >>> make clean >>> >>> git clean -dfx >>> >>> sh ./autogen.sh >>> >>> ./configure --disable-server >>> >>> make debs >>> >>> >>> >>> Aurélien >>> >>> >>> >>> *De : *lustre-devel au nom de >>> Ankit Ladhania >>> *Date : *vendredi 24 juillet 2020 à 11:46 >>> *À : *"lustre-devel at lists.lustre.org" >>> *Objet : *[EXTERNAL] [lustre-devel] /usr/bin/fakeroot: debian/rules: >>> Permission denied - Building Lustre client (v2.13.53) >>> >>> >>> >>> *CAUTION*: This email originated from outside of the organization. Do >>> not click links or open attachments unless you can confirm the sender and >>> know the content is safe. >>> >>> >>> >>> >>> >>> Hi, >>> >>> We are installing the Lustre Client (v2.13.53) on Ubuntu 18.04 (linux >>> header: 5.4.0-42-generic) >>> >>> While doing `make debs` we are getting below error: >>> >>> make[1]: Leaving directory '/home/zeblok/lustre-release' >>> >>> dpkg-genbuildinfo >>> >>> dpkg-genchanges >../lustre_2.13.53-1_amd64.changes >>> >>> dpkg-genchanges: warning: package lustre-server-utils in control file but not in files list >>> >>> dpkg-genchanges: warning: package lustre-resource-agents in control file but not in files list >>> >>> dpkg-genchanges: warning: package lustre-client-modules-dkms in control file but not in files list >>> >>> dpkg-genchanges: info: including full source code in upload >>> >>> dpkg-source -I.git -I*.out[0-9]* -I*.swp --after-build lustre-release >>> >>> dpkg-buildpackage: info: full upload; Debian-native package (full source is included) >>> >>> ~/lustre-release/debian/tmp/modules-deb ~/lustre-release >>> >>> Removing old /home/zeblok/lustre-release/debian/tmp/modules-deb/usr_src/modules/lustre/ tree >>> >>> Extracting the package tarball, /usr/src/lustre.tar.bz2, please wait... >>> >>> /usr/bin/fakeroot: 175: /usr/bin/fakeroot: debian/rules: Permission denied >>> >>> /usr/bin/fakeroot: 175: /usr/bin/fakeroot: debian/rules: Permission denied >>> >>> BUILD FAILED! >>> >>> See /home/zeblok/lustre-release/debian/tmp/modules-deb/var_cache_modass/lustre.buildlog.5.4.0-42-generic.1595583181 for details. >>> >>> mv: cannot stat '../lustre-dev__*.deb': No such file or directory >>> >>> mv: cannot stat '../lustre-source__all.deb': No such file or directory >>> >>> mv: cannot stat '../lustre-iokit__*.deb': No such file or directory >>> >>> mv: cannot stat '../lustre-tests__*.deb': No such file or directory >>> >>> mv: cannot stat '../lustre-*-utils__*.deb': No such file or directory >>> >>> mv: cannot stat '../lustre_.dsc': No such file or directory >>> >>> mv: cannot stat '../lustre__*.changes': No such file or directory >>> >>> mv: cannot stat '../lustre_.tar.gz': No such file or directory >>> >>> mv: cannot stat '../lustre-*-modules-5.4.0-42-generic__*.deb': No such file or directory >>> >>> autoMakefile:1190: recipe for target 'debs' failed >>> >>> make: *** [debs] Error 1 >>> >>> >>> >>> Also tried doing 'chmod a+x debian/rules' but with no progress. >>> >>> >>> >>> Not able to understand why this is breaking, any help is highly >>> appreciated. >>> >>> >>> >>> *Regards, Ankit Ladhania* >>> >>> IN: +91 933 444 2209 >>> US: +1 (256) 203-0077 >>> >>> Skype: ankit.ladhania >>> >>> [image: Image supprimée par l'expéditeur.] >>> >> _______________________________________________ >> lustre-devel mailing list >> lustre-devel at lists.lustre.org >> http://lists.lustre.org/listinfo.cgi/lustre-devel-lustre.org >> >> > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jsbevilacqua at linagora.com Wed Jul 22 07:59:55 2020 From: jsbevilacqua at linagora.com (=?UTF-8?Q?Jean-S=c3=a9bastien_BEVILACQUA?=) Date: Wed, 22 Jul 2020 07:59:55 -0000 Subject: [lustre-devel] Lustre optimization for billion of files Message-ID: Hello, I am working on adding Lustre support to Openstack Cinder and Openstack Swift (conception step). I created a blueprint for Cinder here: https://blueprints.launchpad.net/cinder/+spec/add-lustre-driver I am currently looking for ways to optimize the file access. For what I know, Lustre is better in managing few big files instead of a lot small files. An optimization would be to store lot of small files in one big file managed on Openstack side. * I would like to know what you thing about it ? * Do you have ideas about others optimizations ? * Do you know about optimizations on the Lustre side ? Sincerely, Jean-Sébastien BEVILACQUA -- Jean-Sébastien BEVILACQUA> Directeur Technique OSSA Tél : +33(0)7 85 55 35 11 Courriel : jsbevilacqua at linagora.com www.linagora.com -------------- next part -------------- An HTML attachment was scrubbed... URL: