From jsimmons at infradead.org Sat Jun 1 00:38:33 2019 From: jsimmons at infradead.org (James Simmons) Date: Sat, 1 Jun 2019 01:38:33 +0100 (BST) Subject: [lustre-devel] [PATCH v2 07/29] lustre: obd: collect all resource releasing for obj_type. In-Reply-To: <4617C98C-31F0-4773-97F3-54B2BCAF1E7A@whamcloud.com> References: <1558356671-29599-1-git-send-email-jsimmons@infradead.org> <1558356671-29599-8-git-send-email-jsimmons@infradead.org> <4617C98C-31F0-4773-97F3-54B2BCAF1E7A@whamcloud.com> Message-ID: > >>> int class_register_type(struct obd_ops *dt_ops, struct md_ops *md_ops, > >>> @@ -181,10 +163,17 @@ int class_register_type(struct obd_ops *dt_ops, > >> struct md_ops *md_ops, > >>> return -EEXIST; > >>> } > >>> > >>> - rc = -ENOMEM; > >>> type = kzalloc(sizeof(*type), GFP_NOFS); > >>> if (!type) > >>> - return rc; > >>> + return -ENOMEM; > >>> + > >>> + type->typ_kobj.kset = lustre_kset; > >>> + rc = kobject_init_and_add(&type->typ_kobj, &class_ktype, > >>> + &lustre_kset->kobj, "%s", name); > >> > >> I don't know that I would actually cause a problem, but I don't like > >> "add"ing and object (above) before fully initializing it (below). So > >> I've kept the split from my version where kobject_init() happens early > >> and kobject_add() happens later. > >> I've included the other changes that you made. > >> > >> Thanks, > >> NeilBrown > > > > The reason I did it that way was to handle the server case down the road. > > So for the case when both client and server are on the same node, and yes > > people do such setups for testing this is important. > > > > Consider the case we have both the lov and lod layer on a single node. > > Both layers attempt to create "lov" obd_type. When the lov module loads > > first then a complete obd_type is created. Once lod loads then it just > > uses real lov obd_type and creates the needed symlinks. > > We've had the "lov->lod" symlinks on servers since Lustre 2.4 or so (when > osd-zfs was first added). We could just remove this compatibility, so > long as the test scripts were updated to always use "lod" on the server > and "lov" on the client (previously they shared the same "lov" code module > so the path was the same). We don't need test script interop going back > further than that, so this should be OK. > > Cheers, Andreas I ripped off the band aid and boy did it bleed. I going to have to work out the test suite changes. Well will need to back port the test suite changes to 2.12 for interop testing. > > You end up with > > > > ls -al /sys/fs/lustre/lov/ > > total 0 > > drwxr-xr-x 2 root root 0 May 22 14:27 . > > drwxr-xr-x 14 root root 0 May 22 14:27 .. > > lrwxrwxrwx 1 root root 0 May 22 14:27 lustre-MDT0000-mdtlov -> > > ../lod/lustre-MDT0000-mdtlov > > lrwxrwxrwx 1 root root 0 May 22 14:27 lustre-MDT0002-mdtlov -> > > ../lod/lustre-MDT0002-mdtlov > > > > Plus the real lov obd devices. > > > > Now if lod loads first then the lod module creates a "lov" obd_type > > using class_create_symlink() but is not fully initialized nor does > > it need to be. If the client lov module is present and it loads then > > it takes the "lov" obd_type create by the lod and finishes initializing > > it. > > > > Looking at the final code and added in server case: > > > > type = class_search_type(name); > > if (type) { > > kobject_put(&type->typ_kobj); > > if (strcmp(name, LUSTRE_LOV_NAME) == 0 || > > strcmp(name, LUSTRE_OSC_NAME) == 0) > > goto dir_exist; > > CDEBUG(D_IOCTL, "Type %s already registered\n", name); > > return -EEXIST; > > } > > > > type = kzalloc(sizeof(*type), GFP_NOFS); > > if (!type) > > return rc; > > > > type->typ_kobj.kset = lustre_kset; > > kobject_init(&type->typ_kobj, &class_ktype); > > > > type->typ_dt_ops = dt_ops; /* lov obd_type is never set to the > > * correct values if lod created it. > > */ > > type->typ_md_ops = md_ops; > > > > rc = kobject_add(&type->typ_kobj, &lustre_kset->kobj, "%s", name); > > if (rc) > > goto failed; > > > > type->typ_debugfs_entry = debugfs_create_dir(type->typ_name, > > debugfs_lustre_root); > > dir_exit: > > > > Now if this needed for the later module handling patches I guess we could > > do: > > > > if (strcmp(name, LUSTRE_LOV_NAME) == 0 || > > strcmp(name, LUSTRE_OSC_NAME) == 0) { > > type->typ_dt_ops = dt_ops; > > type->typ_md_ops = md_ops; > > goto dir_exist; > > } > > > > Is that the case? > > > > > > Cheers, Andreas > -- > Andreas Dilger > Principal Lustre Architect > Whamcloud > > From green at whamcloud.com Wed Jun 5 06:41:26 2019 From: green at whamcloud.com (Oleg Drokin) Date: Wed, 5 Jun 2019 06:41:26 +0000 Subject: [lustre-devel] New tag 2.12.54 Message-ID: <45C1AB6D-E8F5-4F50-9921-CC00C237BC67@whamcloud.com> Hello! I just tagged 2.12.54 in lustre development branch. Here’s the changelog: Alex Zhuravlev (7): LU-11233 build: support for gcc8 LU-2233 tests: improve tests sanityn/40-47 LU-10754 tests: sanityn/47b to sleep for 1s LU-12013 lfsck: use correct buffer LU-11233 utils: fix build warnings for gcc8 LU-11233 tests: fix gcc8 build warnings LU-12034 obdclass: put all service's env on the list Alexander Boyko (1): LU-12140 lnet: adds checking msg len Amir Shehata (1): LU-12279 lnet: use number of wrs to calculate CQEs Andreas Dilger (3): LU-12159 utils: improve lfs getname functionality LU-11233 utils: fix double-free of params fields LU-10602 utils: fix file heat support Andrew Perepechko (2): LU-6951 tests: sanity test_27m failure LU-12352 libcfs: crashes with certain cpu part numbers Andriy Skulysh (3): LU-12218 ptlrpc: Bulk assertion fails on -ENOMEM LU-11090 quota: Oops in qsd_config LU-12095 ptlrpc: ocd_connect_flags are wrong during reconnect Ann Koehler (1): LU-12221 statahead: sa_handle_callback get lli_sa_lock earlier Arshad Hussain (12): LU-9010 ptlrpc: Change static defines to use macro for gss_krb5_mech.c LU-6142 ldlm: Fix style issues for interval_tree.c LU-6142 ptlrpc: Fix style issues for sec_null.c LU-6142 ptlrpc: Fix style issues for service.c LU-12314 tests: Add Missing Description to sanity test 258a LU-6142 ldlm: Fix style issues for ldlm_resource.c LU-6142 ptlrpc: Fix style issues for sec_plain.c LU-6142 ptlrpc: Fix style issues for sec_gc.c LU-6142 ptlrpc: Fix style issues for sec.c LU-6142 ptlrpc: Fix style issues for client.c LU-6142 ptlrpc: Fix style issues for llog_client.c LU-6142 ldlm: Fix style issues for ldlm_plain.c Artem Blagodarenko (2): LU-12335 ldiskfs: fixed size preallocation table LU-12345 ldiskfs: optimize nodelalloc mode Aurelien Degremont (2): LU-12227 scripts: check for mounted ZFS devices too LU-12236 gss: remove unused code in gss_svc_upcall.c Ben Menadue (3): LU-12269 build: fix hardened builds in rpm spec file LU-12269 build: remove %{fullrelease} from Provides LU-12269 build: add value to definition of with_gss in spec Bruno Faccini (2): LU-11376 lov: new foreign LOV format LU-11376 lmv: new foreign LMV format Chris Horn (3): LU-12302 lnet: Fix NI status in proc for loopback ni LU-12333 ptlrpc: Add more flags to DEBUG_REQ_FLAGS macro LU-12323 libcfs: check if save_stack_trace_tsk is exported Dmitry Eremin (1): LU-8066 ptlrpc: move sptlrpc procfs entry to debugfs Götz Waschk (1): LU-12342 spec: mark lsvcgss as a config file in the rpm Hongchao Zhang (1): LU-12232 test: commit before df James Nunez (1): LU-9581 tests: remove recovery-small test from ALWAYS_EXCEPT James Simmons (5): LU-11803 obd: replace class_uuid with linux kernel version. LU-11157 obd: round values to nearest MiB for *_mb syfs files LU-11771 ldlm: use hrtimer for recovery to fix timeout messages LU-11893 ksocklnd: add secondary IP address handling LU-11893 o2iblnd: add secondary IP address handling Jian Yu (5): LU-12276 lnet: check const parameters for ib_post_send and ib_post_recv LU-12139 kernel: kernel update [SLES12 SP3 4.4.176-94.88] LU-12242 kernel: kernel update RHEL7.6 [3.10.0-957.12.1.el7] LU-12269 kernel: new kernel [RHEL 8.0 4.18.0-80.el8] LU-12306 kernel: kernel update RHEL7.6 [3.10.0-957.12.2.el7] Lai Siyao (2): LU-11907 dne: allow access to striped dir with broken layout LU-11213 dne: add new dir hash type "space" Li Dongyang (7): LU-11838 lnet: getname dropping addrlen argument LU-12248 lov: fix ost objects calculation in lod_statfs LU-12093 osc: don't check capability for every page LU-11838 osp: atomic64_read() returns s64 LU-11838 ldlm: struct timespec64.tv_sec type change LU-11838 llite: address_space ->page_tree renamed ->i_pages LU-12270 o2iblnd: pci_unmap_addr() removed in 4.19 Mikhail Pershin (5): LU-12212 mdt: fix SECCTX reply buffer handling LU-11359 mdt: fix mdt_dom_discard_data() timeouts LU-10894 dom: per-resource ELC for WRITE lock enqueue LU-10894 dom: mdc_lock_flush() improvement LU-12120 grants: prevent negative ted_grant value Minh Diep (2): LU-12282 build: export IB_OPTIONS before build LU-11041 kernel: Enable tons of kernel debug options Nathaniel Clark (3): Revert "LU-8384 scripts: Add scripts to systemd for EL7" LU-12298 init: Add init info to lustre sysvinit script LU-12309 osd-zfs: Support disabled project quotas NeilBrown (6): LU-8066 obd: embed typ_kobj in obd_type LU-11089 obdclass: remove locking from lu_context_exit() LU-11838: lnet: remove lnet_ipif_enumerate() LU-11089 obd: use wait_event_var() in lu_context_key_degister() LU-11089 obd: remove lock from key register/degister LU-10467 ptlrpc: discard SVC_SIGNAL and related functions Olaf Faaland (2): LU-11946 build: no zlib check during configure --enable-dist LU-11946 build: no yaml check during configure --enable-dist Oleg Drokin (6): LU-12263 build: push depreciation of LMV_HASH_FLAG_DEAD to 2.12.55 LU-12324 mdd: Do not record xattr size get in changelogs LU-10948 llite: Revalidate dentries in ll_intent_file_open LU-11623 llite: hash just created files if lock allows LU-12341 tests: Add kmemleak awareness to test-framework New tag 2.12.54 Patrick Farrell (7): LU-9846 obd: Add overstriping CONNECT flag LU-11403 tests: Fix $tfile usage LU-11690 lod: fix LBUG with wide striping LU-10754 tests: Clear mdc locks before tests LU-9846 lod: Add overstriping support LU-12350 tests: Do not use background failover LU-11963 osd: Add nonrotational flag to statfs Qian Yingjin (1): LU-11526 rpc: support maximum 64MB I/O RPC Sebastien Buisson (5): LU-12131 tests: only create lgssc.conf file if necessary LU-12266 mdd: fix up non-dir creation in SGID dirs LU-12098 mdd: explicitly clear changelogs on deregister LU-12131 tests: fix SSK handling in tests LU-12267 tests: update filter in acl for SElinux case Sergey Cheremencev (1): LU-11760 ofd: formatted OST recognition change Thomas Stibor (1): LU-12019 build: Recognize Debian Kernel and set KMP dir Vitaly Fertman (1): LU-11251 mdt: ASSERTION (req_transno < next_transno) failed Vladimir Saveliev (1): LU-12206 mdt: mdt_init0 failure handling Wang Shilong (6): LU-12043 llite, readahead: fix to call ll_ras_enter() properly LU-12225 obdclass: fix race access vs removal of jobid_hash LU-12225 obdclass: improve jobid memory reclaim policy LU-12299 libcfs: fix panic for too large cpu partions LU-11872 utils: don't follow link files in default LU-12361 lov: fix wrong calculated length for fiemap Yang Sheng (1): LU-11851 ldiskfs: reschedule for htree thread. From shaun at tancheff.com Thu Jun 6 22:31:10 2019 From: shaun at tancheff.com (Shaun Tancheff) Date: Thu, 6 Jun 2019 17:31:10 -0500 Subject: [lustre-devel] [PATCH] Place a memory barrier around cp_state changes Message-ID: <20190606223110.15725-1-stancheff@cray.com> Ensure all uses of cl_page->cp_state are smp-safe. LBUG Output: In __cl_page_state_set() .. old = page->cp_state; PASSERT(env, page, allowed_transitions[old][state]); .. Asserted with the following: page at ffff80be1fcd6600[3 ffff80be1fca5cc0 0 1 ffff809e60e2 8cc0] However cp_state 0 (CPS_CACHED) to 1 (CPS_OWNED) is a valid transition leading to the conclusion that cp_state became 0 during the assertion. Signed-off-by: Shaun Tancheff --- fs/lustre/include/cl_object.h | 11 +++++++++++ fs/lustre/llite/rw26.c | 2 +- fs/lustre/llite/vvp_page.c | 6 +++--- fs/lustre/obdclass/cl_page.c | 34 +++++++++++++++++++-------------- fs/lustre/obdecho/echo_client.c | 2 +- fs/lustre/osc/osc_cache.c | 18 +++++++++-------- 6 files changed, 46 insertions(+), 27 deletions(-) diff --git a/fs/lustre/include/cl_object.h b/fs/lustre/include/cl_object.h index 691c2f5da53a..d6e1f6f05f50 100644 --- a/fs/lustre/include/cl_object.h +++ b/fs/lustre/include/cl_object.h @@ -752,6 +752,17 @@ struct cl_page { struct cl_sync_io *cp_sync_io; }; +static inline enum cl_page_state cl_page_state_get(const struct cl_page *pg) +{ + /* + * Paired with smp_store_release in cl_page_state_set_trust + * and ensures that we see the most recent value of cp_state + * even when the last modification was not performed on the + * current processor + */ + return smp_load_acquire(&pg->cp_state); +} + /** * Per-layer part of cl_page. * diff --git a/fs/lustre/llite/rw26.c b/fs/lustre/llite/rw26.c index e4ce3b6f5772..364dec208ccd 100644 --- a/fs/lustre/llite/rw26.c +++ b/fs/lustre/llite/rw26.c @@ -200,7 +200,7 @@ static ssize_t ll_direct_IO_seg(const struct lu_env *env, struct cl_io *io, rc = cl_page_own(env, io, clp); if (rc) { - LASSERT(clp->cp_state == CPS_FREEING); + LASSERT(cl_page_state_get(clp) == CPS_FREEING); cl_page_put(env, clp); break; } diff --git a/fs/lustre/llite/vvp_page.c b/fs/lustre/llite/vvp_page.c index 590e5f5e43c9..38b8c488d765 100644 --- a/fs/lustre/llite/vvp_page.c +++ b/fs/lustre/llite/vvp_page.c @@ -323,18 +323,18 @@ static int vvp_page_make_ready(const struct lu_env *env, lock_page(vmpage); if (clear_page_dirty_for_io(vmpage)) { - LASSERT(pg->cp_state == CPS_CACHED); + LASSERT(cl_page_state_get(pg) == CPS_CACHED); /* This actually clears the dirty bit in the radix tree. */ set_page_writeback(vmpage); CL_PAGE_HEADER(D_PAGE, env, pg, "readied\n"); - } else if (pg->cp_state == CPS_PAGEOUT) { + } else if (cl_page_state_get(pg) == CPS_PAGEOUT) { /* is it possible for osc_flush_async_page() to already * make it ready? */ result = -EALREADY; } else { CL_PAGE_DEBUG(D_ERROR, env, pg, "Unexpecting page state %d.\n", - pg->cp_state); + cl_page_state_get(pg)); LBUG(); } unlock_page(vmpage); diff --git a/fs/lustre/obdclass/cl_page.c b/fs/lustre/obdclass/cl_page.c index 349f19e014e0..da4429b82932 100644 --- a/fs/lustre/obdclass/cl_page.c +++ b/fs/lustre/obdclass/cl_page.c @@ -97,7 +97,7 @@ static void cl_page_free(const struct lu_env *env, struct cl_page *page) PASSERT(env, page, list_empty(&page->cp_batch)); PASSERT(env, page, !page->cp_owner); - PASSERT(env, page, page->cp_state == CPS_FREEING); + PASSERT(env, page, cl_page_state_get(page) == CPS_FREEING); while ((slice = list_first_entry_or_null(&page->cp_layers, struct cl_page_slice, @@ -119,8 +119,14 @@ static void cl_page_free(const struct lu_env *env, struct cl_page *page) static inline void cl_page_state_set_trust(struct cl_page *page, enum cl_page_state state) { - /* bypass const. */ - *(enum cl_page_state *)&page->cp_state = state; + /* + * Paired with smp_load_acquire in cl_page_state_get + * and ensures that we see the most recent value of cp_state + * is available even when the next access is not performed on the + * current processor. + * Note we also cast away const as the only modifier of cp_state. + */ + smp_store_release((enum cl_page_state *)&page->cp_state, state); } struct cl_page *cl_page_alloc(const struct lu_env *env, @@ -270,10 +276,10 @@ static void __cl_page_state_set(const struct lu_env *env, } }; - old = page->cp_state; + old = cl_page_state_get(page); PASSERT(env, page, allowed_transitions[old][state]); CL_PAGE_HEADER(D_TRACE, env, page, "%d -> %d\n", old, state); - PASSERT(env, page, page->cp_state == old); + PASSERT(env, page, cl_page_state_get(page) == old); PASSERT(env, page, equi(state == CPS_OWNED, page->cp_owner)); cl_page_state_set_trust(page, state); } @@ -313,7 +319,7 @@ void cl_page_put(const struct lu_env *env, struct cl_page *page) refcount_read(&page->cp_ref)); if (refcount_dec_and_test(&page->cp_ref)) { - LASSERT(page->cp_state == CPS_FREEING); + LASSERT(cl_page_state_get(page) == CPS_FREEING); LASSERT(refcount_read(&page->cp_ref) == 0); PASSERT(env, page, !page->cp_owner); @@ -378,7 +384,7 @@ void __cl_page_disown(const struct lu_env *env, const struct cl_page_slice *slice; enum cl_page_state state; - state = pg->cp_state; + state = cl_page_state_get(pg); cl_page_owner_clear(pg); if (state == CPS_OWNED) @@ -402,7 +408,7 @@ int cl_page_is_owned(const struct cl_page *pg, const struct cl_io *io) struct cl_io *top = cl_io_top((struct cl_io *)io); LINVRNT(cl_object_same(pg->cp_obj, io->ci_obj)); - return pg->cp_state == CPS_OWNED && pg->cp_owner == top; + return cl_page_state_get(pg) == CPS_OWNED && pg->cp_owner == top; } EXPORT_SYMBOL(cl_page_is_owned); @@ -434,7 +440,7 @@ static int __cl_page_own(const struct lu_env *env, struct cl_io *io, io = cl_io_top(io); - if (pg->cp_state == CPS_FREEING) { + if (cl_page_state_get(pg) == CPS_FREEING) { result = -ENOENT; goto out; } @@ -453,7 +459,7 @@ static int __cl_page_own(const struct lu_env *env, struct cl_io *io, PASSERT(env, pg, !pg->cp_owner); pg->cp_owner = cl_io_top(io); cl_page_owner_set(pg); - if (pg->cp_state != CPS_FREEING) { + if (cl_page_state_get(pg) != CPS_FREEING) { cl_page_state_set(env, pg, CPS_OWNED); } else { __cl_page_disown(env, io, pg); @@ -593,7 +599,7 @@ static void __cl_page_delete(const struct lu_env *env, struct cl_page *pg) { const struct cl_page_slice *slice; - PASSERT(env, pg, pg->cp_state != CPS_FREEING); + PASSERT(env, pg, cl_page_state_get(pg) != CPS_FREEING); /* * Sever all ways to obtain new pointers to @pg. @@ -756,7 +762,7 @@ void cl_page_completion(const struct lu_env *env, const struct cl_page_slice *slice; PASSERT(env, pg, crt < CRT_NR); - PASSERT(env, pg, pg->cp_state == cl_req_type_state(crt)); + PASSERT(env, pg, cl_page_state_get(pg) == cl_req_type_state(crt)); CL_PAGE_HEADER(D_TRACE, env, pg, "%d %d\n", crt, ioret); @@ -805,7 +811,7 @@ int cl_page_make_ready(const struct lu_env *env, struct cl_page *pg, } if (result >= 0) { - PASSERT(env, pg, pg->cp_state == CPS_CACHED); + PASSERT(env, pg, cl_page_state_get(pg) == CPS_CACHED); cl_page_io_start(env, pg, crt); result = 0; } @@ -870,7 +876,7 @@ void cl_page_header_print(const struct lu_env *env, void *cookie, (*printer)(env, cookie, "page@%p[%d %p %d %d %p]\n", pg, refcount_read(&pg->cp_ref), pg->cp_obj, - pg->cp_state, pg->cp_type, + cl_page_state_get(pg), pg->cp_type, pg->cp_owner); } EXPORT_SYMBOL(cl_page_header_print); diff --git a/fs/lustre/obdecho/echo_client.c b/fs/lustre/obdecho/echo_client.c index 317123fd27cb..d879f109e641 100644 --- a/fs/lustre/obdecho/echo_client.c +++ b/fs/lustre/obdecho/echo_client.c @@ -1046,7 +1046,7 @@ static int cl_echo_object_brw(struct echo_object *eco, int rw, u64 offset, rc = cl_page_own(env, io, clp); if (rc) { - LASSERT(clp->cp_state == CPS_FREEING); + LASSERT(cl_page_state_get(clp) == CPS_FREEING); cl_page_put(env, clp); break; } diff --git a/fs/lustre/osc/osc_cache.c b/fs/lustre/osc/osc_cache.c index f8fddbfe6a7e..75984b98b229 100644 --- a/fs/lustre/osc/osc_cache.c +++ b/fs/lustre/osc/osc_cache.c @@ -1045,7 +1045,7 @@ static int osc_extent_truncate(struct osc_extent *ext, pgoff_t trunc_index, cl_page_discard(env, io, page); cl_page_disown(env, io, page); } else { - LASSERT(page->cp_state == CPS_FREEING); + LASSERT(cl_page_state_get(page) == CPS_FREEING); LASSERT(0); } @@ -1356,10 +1356,12 @@ static int osc_completion(const struct lu_env *env, struct osc_async_page *oap, int srvlock; cmd &= ~OBD_BRW_NOQUOTA; - LASSERTF(equi(page->cp_state == CPS_PAGEIN, cmd == OBD_BRW_READ), - "cp_state:%u, cmd:%d\n", page->cp_state, cmd); - LASSERTF(equi(page->cp_state == CPS_PAGEOUT, cmd == OBD_BRW_WRITE), - "cp_state:%u, cmd:%d\n", page->cp_state, cmd); + LASSERTF(equi(cl_page_state_get(page) == CPS_PAGEIN, + cmd == OBD_BRW_READ), + "cp_state:%u, cmd:%d\n", cl_page_state_get(page), cmd); + LASSERTF(equi(cl_page_state_get(page) == CPS_PAGEOUT, + cmd == OBD_BRW_WRITE), + "cp_state:%u, cmd:%d\n", cl_page_state_get(page), cmd); LASSERT(opg->ops_transfer_pinned); crt = cmd == OBD_BRW_READ ? CRT_READ : CRT_WRITE; @@ -3061,7 +3063,7 @@ bool osc_page_gang_lookup(const struct lu_env *env, struct cl_io *io, page = ops->ops_cl.cpl_page; LASSERT(page->cp_type == CPT_CACHEABLE); - if (page->cp_state == CPS_FREEING) + if (cl_page_state_get(page) == CPS_FREEING) continue; cl_page_get(page); @@ -3142,7 +3144,7 @@ static bool check_and_discard_cb(const struct lu_env *env, struct cl_io *io, cl_page_discard(env, io, page); cl_page_disown(env, io, page); } else { - LASSERT(page->cp_state == CPS_FREEING); + LASSERT(cl_page_state_get(page) == CPS_FREEING); } } @@ -3169,7 +3171,7 @@ static bool discard_cb(const struct lu_env *env, struct cl_io *io, cl_page_discard(env, io, page); cl_page_disown(env, io, page); } else { - LASSERT(page->cp_state == CPS_FREEING); + LASSERT(cl_page_state_get(page) == CPS_FREEING); } return true; -- 2.17.1 From neilb at suse.com Fri Jun 7 00:56:55 2019 From: neilb at suse.com (NeilBrown) Date: Fri, 07 Jun 2019 10:56:55 +1000 Subject: [lustre-devel] [PATCH] Place a memory barrier around cp_state changes In-Reply-To: <20190606223110.15725-1-stancheff@cray.com> References: <20190606223110.15725-1-stancheff@cray.com> Message-ID: <87wohyw6ug.fsf@notabene.neil.brown.name> On Thu, Jun 06 2019, Shaun Tancheff wrote: > Ensure all uses of cl_page->cp_state are smp-safe. > > LBUG Output: > In __cl_page_state_set() > .. > old = page->cp_state; > PASSERT(env, page, allowed_transitions[old][state]); > .. > Asserted with the following: > page at ffff80be1fcd6600[3 ffff80be1fca5cc0 0 1 ffff809e60e2 8cc0] What do all these fields mean? Or where is this printed? The closest I can find is in cl_page_header_print() which uses the format string: "page@%p[%d %p %d %d %p]\n", But that only has 5 fields in the [], while the output you provided has 6. > > However cp_state 0 (CPS_CACHED) to 1 (CPS_OWNED) is a valid transition > leading to the conclusion that cp_state became 0 during the > assertion. Where is the evidence that this was the transition that was happening? If it was in some part of the LBUG output that wasn't quoted - then please fix that by quoting the entire LBUG output. We need to understand which change happened at this time to cause the race. Then we need to explain why the added barriers actually close the race window. If these barrier do actually fix a race, then there is something *very* wrong. The comments say that changes to cp_state are protected by the page lock on the corresponding VM page. Locking and unlocking a VM page entails sufficient barriers that changes made while one thread holds the lock will be visible to another thread once it also gets the lock. So the only possible explanation for a race as you suspect, is if the vmpage *isn't* locked when something changes cp_state, and that would be bad. Thanks, NeilBrown > > Signed-off-by: Shaun Tancheff > --- > fs/lustre/include/cl_object.h | 11 +++++++++++ > fs/lustre/llite/rw26.c | 2 +- > fs/lustre/llite/vvp_page.c | 6 +++--- > fs/lustre/obdclass/cl_page.c | 34 +++++++++++++++++++-------------- > fs/lustre/obdecho/echo_client.c | 2 +- > fs/lustre/osc/osc_cache.c | 18 +++++++++-------- > 6 files changed, 46 insertions(+), 27 deletions(-) > > diff --git a/fs/lustre/include/cl_object.h b/fs/lustre/include/cl_object.h > index 691c2f5da53a..d6e1f6f05f50 100644 > --- a/fs/lustre/include/cl_object.h > +++ b/fs/lustre/include/cl_object.h > @@ -752,6 +752,17 @@ struct cl_page { > struct cl_sync_io *cp_sync_io; > }; > > +static inline enum cl_page_state cl_page_state_get(const struct cl_page *pg) > +{ > + /* > + * Paired with smp_store_release in cl_page_state_set_trust > + * and ensures that we see the most recent value of cp_state > + * even when the last modification was not performed on the > + * current processor > + */ > + return smp_load_acquire(&pg->cp_state); > +} > + > /** > * Per-layer part of cl_page. > * > diff --git a/fs/lustre/llite/rw26.c b/fs/lustre/llite/rw26.c > index e4ce3b6f5772..364dec208ccd 100644 > --- a/fs/lustre/llite/rw26.c > +++ b/fs/lustre/llite/rw26.c > @@ -200,7 +200,7 @@ static ssize_t ll_direct_IO_seg(const struct lu_env *env, struct cl_io *io, > > rc = cl_page_own(env, io, clp); > if (rc) { > - LASSERT(clp->cp_state == CPS_FREEING); > + LASSERT(cl_page_state_get(clp) == CPS_FREEING); > cl_page_put(env, clp); > break; > } > diff --git a/fs/lustre/llite/vvp_page.c b/fs/lustre/llite/vvp_page.c > index 590e5f5e43c9..38b8c488d765 100644 > --- a/fs/lustre/llite/vvp_page.c > +++ b/fs/lustre/llite/vvp_page.c > @@ -323,18 +323,18 @@ static int vvp_page_make_ready(const struct lu_env *env, > > lock_page(vmpage); > if (clear_page_dirty_for_io(vmpage)) { > - LASSERT(pg->cp_state == CPS_CACHED); > + LASSERT(cl_page_state_get(pg) == CPS_CACHED); > /* This actually clears the dirty bit in the radix tree. */ > set_page_writeback(vmpage); > CL_PAGE_HEADER(D_PAGE, env, pg, "readied\n"); > - } else if (pg->cp_state == CPS_PAGEOUT) { > + } else if (cl_page_state_get(pg) == CPS_PAGEOUT) { > /* is it possible for osc_flush_async_page() to already > * make it ready? > */ > result = -EALREADY; > } else { > CL_PAGE_DEBUG(D_ERROR, env, pg, "Unexpecting page state %d.\n", > - pg->cp_state); > + cl_page_state_get(pg)); > LBUG(); > } > unlock_page(vmpage); > diff --git a/fs/lustre/obdclass/cl_page.c b/fs/lustre/obdclass/cl_page.c > index 349f19e014e0..da4429b82932 100644 > --- a/fs/lustre/obdclass/cl_page.c > +++ b/fs/lustre/obdclass/cl_page.c > @@ -97,7 +97,7 @@ static void cl_page_free(const struct lu_env *env, struct cl_page *page) > > PASSERT(env, page, list_empty(&page->cp_batch)); > PASSERT(env, page, !page->cp_owner); > - PASSERT(env, page, page->cp_state == CPS_FREEING); > + PASSERT(env, page, cl_page_state_get(page) == CPS_FREEING); > > while ((slice = list_first_entry_or_null(&page->cp_layers, > struct cl_page_slice, > @@ -119,8 +119,14 @@ static void cl_page_free(const struct lu_env *env, struct cl_page *page) > static inline void cl_page_state_set_trust(struct cl_page *page, > enum cl_page_state state) > { > - /* bypass const. */ > - *(enum cl_page_state *)&page->cp_state = state; > + /* > + * Paired with smp_load_acquire in cl_page_state_get > + * and ensures that we see the most recent value of cp_state > + * is available even when the next access is not performed on the > + * current processor. > + * Note we also cast away const as the only modifier of cp_state. > + */ > + smp_store_release((enum cl_page_state *)&page->cp_state, state); > } > > struct cl_page *cl_page_alloc(const struct lu_env *env, > @@ -270,10 +276,10 @@ static void __cl_page_state_set(const struct lu_env *env, > } > }; > > - old = page->cp_state; > + old = cl_page_state_get(page); > PASSERT(env, page, allowed_transitions[old][state]); > CL_PAGE_HEADER(D_TRACE, env, page, "%d -> %d\n", old, state); > - PASSERT(env, page, page->cp_state == old); > + PASSERT(env, page, cl_page_state_get(page) == old); > PASSERT(env, page, equi(state == CPS_OWNED, page->cp_owner)); > cl_page_state_set_trust(page, state); > } > @@ -313,7 +319,7 @@ void cl_page_put(const struct lu_env *env, struct cl_page *page) > refcount_read(&page->cp_ref)); > > if (refcount_dec_and_test(&page->cp_ref)) { > - LASSERT(page->cp_state == CPS_FREEING); > + LASSERT(cl_page_state_get(page) == CPS_FREEING); > > LASSERT(refcount_read(&page->cp_ref) == 0); > PASSERT(env, page, !page->cp_owner); > @@ -378,7 +384,7 @@ void __cl_page_disown(const struct lu_env *env, > const struct cl_page_slice *slice; > enum cl_page_state state; > > - state = pg->cp_state; > + state = cl_page_state_get(pg); > cl_page_owner_clear(pg); > > if (state == CPS_OWNED) > @@ -402,7 +408,7 @@ int cl_page_is_owned(const struct cl_page *pg, const struct cl_io *io) > struct cl_io *top = cl_io_top((struct cl_io *)io); > > LINVRNT(cl_object_same(pg->cp_obj, io->ci_obj)); > - return pg->cp_state == CPS_OWNED && pg->cp_owner == top; > + return cl_page_state_get(pg) == CPS_OWNED && pg->cp_owner == top; > } > EXPORT_SYMBOL(cl_page_is_owned); > > @@ -434,7 +440,7 @@ static int __cl_page_own(const struct lu_env *env, struct cl_io *io, > > io = cl_io_top(io); > > - if (pg->cp_state == CPS_FREEING) { > + if (cl_page_state_get(pg) == CPS_FREEING) { > result = -ENOENT; > goto out; > } > @@ -453,7 +459,7 @@ static int __cl_page_own(const struct lu_env *env, struct cl_io *io, > PASSERT(env, pg, !pg->cp_owner); > pg->cp_owner = cl_io_top(io); > cl_page_owner_set(pg); > - if (pg->cp_state != CPS_FREEING) { > + if (cl_page_state_get(pg) != CPS_FREEING) { > cl_page_state_set(env, pg, CPS_OWNED); > } else { > __cl_page_disown(env, io, pg); > @@ -593,7 +599,7 @@ static void __cl_page_delete(const struct lu_env *env, struct cl_page *pg) > { > const struct cl_page_slice *slice; > > - PASSERT(env, pg, pg->cp_state != CPS_FREEING); > + PASSERT(env, pg, cl_page_state_get(pg) != CPS_FREEING); > > /* > * Sever all ways to obtain new pointers to @pg. > @@ -756,7 +762,7 @@ void cl_page_completion(const struct lu_env *env, > const struct cl_page_slice *slice; > > PASSERT(env, pg, crt < CRT_NR); > - PASSERT(env, pg, pg->cp_state == cl_req_type_state(crt)); > + PASSERT(env, pg, cl_page_state_get(pg) == cl_req_type_state(crt)); > > CL_PAGE_HEADER(D_TRACE, env, pg, "%d %d\n", crt, ioret); > > @@ -805,7 +811,7 @@ int cl_page_make_ready(const struct lu_env *env, struct cl_page *pg, > } > > if (result >= 0) { > - PASSERT(env, pg, pg->cp_state == CPS_CACHED); > + PASSERT(env, pg, cl_page_state_get(pg) == CPS_CACHED); > cl_page_io_start(env, pg, crt); > result = 0; > } > @@ -870,7 +876,7 @@ void cl_page_header_print(const struct lu_env *env, void *cookie, > (*printer)(env, cookie, > "page@%p[%d %p %d %d %p]\n", > pg, refcount_read(&pg->cp_ref), pg->cp_obj, > - pg->cp_state, pg->cp_type, > + cl_page_state_get(pg), pg->cp_type, > pg->cp_owner); > } > EXPORT_SYMBOL(cl_page_header_print); > diff --git a/fs/lustre/obdecho/echo_client.c b/fs/lustre/obdecho/echo_client.c > index 317123fd27cb..d879f109e641 100644 > --- a/fs/lustre/obdecho/echo_client.c > +++ b/fs/lustre/obdecho/echo_client.c > @@ -1046,7 +1046,7 @@ static int cl_echo_object_brw(struct echo_object *eco, int rw, u64 offset, > > rc = cl_page_own(env, io, clp); > if (rc) { > - LASSERT(clp->cp_state == CPS_FREEING); > + LASSERT(cl_page_state_get(clp) == CPS_FREEING); > cl_page_put(env, clp); > break; > } > diff --git a/fs/lustre/osc/osc_cache.c b/fs/lustre/osc/osc_cache.c > index f8fddbfe6a7e..75984b98b229 100644 > --- a/fs/lustre/osc/osc_cache.c > +++ b/fs/lustre/osc/osc_cache.c > @@ -1045,7 +1045,7 @@ static int osc_extent_truncate(struct osc_extent *ext, pgoff_t trunc_index, > cl_page_discard(env, io, page); > cl_page_disown(env, io, page); > } else { > - LASSERT(page->cp_state == CPS_FREEING); > + LASSERT(cl_page_state_get(page) == CPS_FREEING); > LASSERT(0); > } > > @@ -1356,10 +1356,12 @@ static int osc_completion(const struct lu_env *env, struct osc_async_page *oap, > int srvlock; > > cmd &= ~OBD_BRW_NOQUOTA; > - LASSERTF(equi(page->cp_state == CPS_PAGEIN, cmd == OBD_BRW_READ), > - "cp_state:%u, cmd:%d\n", page->cp_state, cmd); > - LASSERTF(equi(page->cp_state == CPS_PAGEOUT, cmd == OBD_BRW_WRITE), > - "cp_state:%u, cmd:%d\n", page->cp_state, cmd); > + LASSERTF(equi(cl_page_state_get(page) == CPS_PAGEIN, > + cmd == OBD_BRW_READ), > + "cp_state:%u, cmd:%d\n", cl_page_state_get(page), cmd); > + LASSERTF(equi(cl_page_state_get(page) == CPS_PAGEOUT, > + cmd == OBD_BRW_WRITE), > + "cp_state:%u, cmd:%d\n", cl_page_state_get(page), cmd); > LASSERT(opg->ops_transfer_pinned); > > crt = cmd == OBD_BRW_READ ? CRT_READ : CRT_WRITE; > @@ -3061,7 +3063,7 @@ bool osc_page_gang_lookup(const struct lu_env *env, struct cl_io *io, > > page = ops->ops_cl.cpl_page; > LASSERT(page->cp_type == CPT_CACHEABLE); > - if (page->cp_state == CPS_FREEING) > + if (cl_page_state_get(page) == CPS_FREEING) > continue; > > cl_page_get(page); > @@ -3142,7 +3144,7 @@ static bool check_and_discard_cb(const struct lu_env *env, struct cl_io *io, > cl_page_discard(env, io, page); > cl_page_disown(env, io, page); > } else { > - LASSERT(page->cp_state == CPS_FREEING); > + LASSERT(cl_page_state_get(page) == CPS_FREEING); > } > } > > @@ -3169,7 +3171,7 @@ static bool discard_cb(const struct lu_env *env, struct cl_io *io, > cl_page_discard(env, io, page); > cl_page_disown(env, io, page); > } else { > - LASSERT(page->cp_state == CPS_FREEING); > + LASSERT(cl_page_state_get(page) == CPS_FREEING); > } > > return true; > -- > 2.17.1 -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 832 bytes Desc: not available URL: From shaun at tancheff.com Fri Jun 7 03:54:19 2019 From: shaun at tancheff.com (Shaun Tancheff) Date: Thu, 6 Jun 2019 22:54:19 -0500 Subject: [lustre-devel] [PATCH] Place a memory barrier around cp_state changes In-Reply-To: <87wohyw6ug.fsf@notabene.neil.brown.name> References: <20190606223110.15725-1-stancheff@cray.com> <87wohyw6ug.fsf@notabene.neil.brown.name> Message-ID: On Thu, Jun 6, 2019 at 7:57 PM NeilBrown wrote: > > On Thu, Jun 06 2019, Shaun Tancheff wrote: > > > Ensure all uses of cl_page->cp_state are smp-safe. > > > > LBUG Output: > > In __cl_page_state_set() > > .. > > old = page->cp_state; > > PASSERT(env, page, allowed_transitions[old][state]); > > .. > > Asserted with the following: > > page at ffff80be1fcd6600[3 ffff80be1fca5cc0 0 1 ffff809e60e2 8cc0] > > What do all these fields mean? Or where is this printed? > The closest I can find is in cl_page_header_print() which uses the > format string: > > "page@%p[%d %p %d %d %p]\n", > > But that only has 5 fields in the [], while the output you provided has > 6. > Sorry for the confusion, here is my (long) explanation with source snippets: The LBUG in full LustreError: 86407:0:(cl_page.c:342:cl_page_state_set0()) page at ffff80be1fcd6600[3 ffff80be1fca5cc0 0 1 ffff809e60e28cc0] LustreError: 86407:0:(cl_page.c:342:cl_page_state_set0()) vvp-page at ffff80be1fcd6650(0:0) vm at ffff7e02fa0e60c0 5000000000001029 4:0 ffff80be1fcd6600 1049548 lru LustreError: 86407:0:(cl_page.c:342:cl_page_state_set0()) lov-page at ffff80be1fcd6690, comp index: 0, gen: 0 LustreError: 86407:0:(cl_page.c:342:cl_page_state_set0()) osc-page at ffff80be1fcd66c8 1049548: 1< 0x845fed 2 0 - - > 2< 4298948608 0 4096 0x0 0x420 | (null) ffff809e473708e8 ffff80be199c1e40 > 3< 0 0 0 > 4< 5 13 64 0 + | - - - - > 5< - - - - | 0 - | 0 - -> LustreError: 86407:0:(cl_page.c:342:cl_page_state_set0()) end page at ffff80be1fcd6600 LustreError: 86407:0:(cl_page.c:342:cl_page_state_set0()) allowed_transitions[old][state] LustreError: 86407:0:(cl_page.c:342:cl_page_state_set0()) ASSERTION( 0 ) failed: LustreError: 86407:0:(cl_page.c:342:cl_page_state_set0()) LBUG The source snippet around L342 ... 339 340 ENTRY; 341 old = page->cp_state; 342 PASSERT(env, page, allowed_transitions[old][state]); 343 CL_PAGE_HEADER(D_TRACE, env, page, "%d -> %d\n", old, state); 344 PASSERT(env, page, page->cp_state == old); 345 PASSERT(env, page, equi(state == CPS_OWNED, page->cp_owner != NULL)); 346 347 cs_pagestate_dec(page->cp_obj, page->cp_state); 348 cs_pagestate_inc(page->cp_obj, state); The PASSERT macro ... ... # define PASSERT(env, page, expr) \ do { \ if (unlikely(!(expr))) { \ CL_PAGE_DEBUG(D_ERROR, (env), (page), #expr "\n"); \ LASSERT(0); \ } \ } while (0) #else /* !LIBCFS_DEBUG */ ... /** * Helper macro, dumping detailed information about \a page into a log. */ #define CL_PAGE_DEBUG(mask, env, page, format, ...) \ do { \ if (cfs_cdebug_show(mask, DEBUG_SUBSYSTEM)) { \ LIBCFS_DEBUG_MSG_DATA_DECL(msgdata, mask, NULL); \ cl_page_print(env, &msgdata, lu_cdebug_printer, page); \ CDEBUG(mask, format , ## __VA_ARGS__); \ } \ } while (0) Almost there ... 1062 /** 1063 * Prints human readable representation of \a pg to the \a f. 1064 */ 1065 void cl_page_print(const struct lu_env *env, void *cookie, 1066 lu_printer_t printer, const struct cl_page *pg) 1067 { 1068 const struct cl_page_slice *slice; 1069 int result = 0; 1070 1071 cl_page_header_print(env, cookie, printer, pg); 1072 list_for_each_entry(slice, &pg->cp_layers, cpl_linkage) { 1073 if (slice->cpl_ops->cpo_print != NULL) 1074 result = (*slice->cpl_ops->cpo_print)(env, slice, 1075 cookie, printer); Finally the 'good stuff' ... 1051 void cl_page_header_print(const struct lu_env *env, void *cookie, 1052 lu_printer_t printer, const struct cl_page *pg) 1053 { 1054 (*printer)(env, cookie, 1055 "page@%p[%d %p %d %d %p]\n", 1056 pg, atomic_read(&pg->cp_ref), pg->cp_obj, 1057 pg->cp_state, pg->cp_type, 1058 pg->cp_owner); 1059 } ... Given the above along with the first line from the LBUG .... LustreError: 86407:0:(cl_page.c:342:cl_page_state_set0()) page at ffff80be1fcd6600[3 ffff80be1fca5cc0 0 1 ffff809e60e28cc0] Implies the following: pg == ffff80be1fcd6600 pg->cp_ref == 3 pg->cp_obj == ffff80be1fca5cc0 pg->cp_state == 0 (state == CPS_CACHED) pg->cp_type == 1 pg->cp_owner == ffff809e60e28cc0 Also known caller was cl_page_assume+0xdc/0x3e0 [obdclass] (from the backtrace) So we know the on-stack / register value of state == CPS_OWNED The backtrace: ... dump_backtrace+0x0/0x248 show_stack+0x24/0x30 dump_stack+0xbc/0xf4 libcfs_call_trace+0xec/0x120 [libcfs] lbug_with_loc+0x4c/0xb8 [libcfs] cl_page_state_set0+0x2b4/0x6e0 [obdclass] cl_page_assume+0xdc/0x3e0 [obdclass] ll_io_read_page+0x144c/0x1de0 [lustre] ll_write_begin+0x3d4/0xee8 [lustre] and the contents of cl_page_assume 596 void cl_page_assume(const struct lu_env *env, 597 struct cl_io *io, struct cl_page *pg) 598 { 599 const struct cl_page_slice *slice; 600 601 PINVRNT(env, pg, cl_object_same(pg->cp_obj, io->ci_obj)); 602 603 ENTRY; 604 io = cl_io_top(io); 605 606 list_for_each_entry(slice, &pg->cp_layers, cpl_linkage) { 607 if (slice->cpl_ops->cpo_assume != NULL) 608 (*slice->cpl_ops->cpo_assume)(env, slice, io); 609 } 610 611 PASSERT(env, pg, pg->cp_owner == NULL); 612 pg->cp_owner = cl_io_top(io); 613 cl_page_owner_set(pg); 614 cl_page_state_set(env, pg, CPS_OWNED); 615 EXIT; 616 } While I believe my conclusion to be correct I am certainly open to being dead wrong. On Thu, Jun 6, 2019 at 7:57 PM NeilBrown wrote: > On Thu, Jun 06 2019, Shaun Tancheff wrote: > > > Ensure all uses of cl_page->cp_state are smp-safe. > > > > LBUG Output: > > In __cl_page_state_set() > > .. > > old = page->cp_state; > > PASSERT(env, page, allowed_transitions[old][state]); > > .. > > Asserted with the following: > > page at ffff80be1fcd6600[3 ffff80be1fca5cc0 0 1 ffff809e60e2 8cc0] > > What do all these fields mean? Or where is this printed? > The closest I can find is in cl_page_header_print() which uses the > format string: > > "page@%p[%d %p %d %d %p]\n", > > But that only has 5 fields in the [], while the output you provided has > 6. > > > > > However cp_state 0 (CPS_CACHED) to 1 (CPS_OWNED) is a valid transition > > leading to the conclusion that cp_state became 0 during the > > assertion. > > Where is the evidence that this was the transition that was happening? > If it was in some part of the LBUG output that wasn't quoted - then > please fix that by quoting the entire LBUG output. > > We need to understand which change happened at this time to cause the > race. > Then we need to explain why the added barriers actually close the race > window. > > If these barrier do actually fix a race, then there is something *very* > wrong. > The comments say that changes to cp_state are protected by the page lock > on the corresponding VM page. > Locking and unlocking a VM page entails sufficient barriers that changes > made while one thread holds the lock will be visible to another > thread once it also gets the lock. > > True, but a page lock is just a bit on the page flags. And we absolutely wait on I/O while the page is 'locked' so a ping-pong (our task *can* sleep) is entirely reasonable, as far as I understand anyway. Since there is multiple state-changes during a single page lock/unlock there is a race, however rare and unlikely, and it's arch specific based on what I understand of the memory barrier rules. So the only possible explanation for a race as you suspect, is if the > vmpage *isn't* locked when something changes cp_state, and that would be > bad. > > Thanks, > NeilBrown > > > > > > Signed-off-by: Shaun Tancheff > > --- > > fs/lustre/include/cl_object.h | 11 +++++++++++ > > fs/lustre/llite/rw26.c | 2 +- > > fs/lustre/llite/vvp_page.c | 6 +++--- > > fs/lustre/obdclass/cl_page.c | 34 +++++++++++++++++++-------------- > > fs/lustre/obdecho/echo_client.c | 2 +- > > fs/lustre/osc/osc_cache.c | 18 +++++++++-------- > > 6 files changed, 46 insertions(+), 27 deletions(-) > > > > diff --git a/fs/lustre/include/cl_object.h > b/fs/lustre/include/cl_object.h > > index 691c2f5da53a..d6e1f6f05f50 100644 > > --- a/fs/lustre/include/cl_object.h > > +++ b/fs/lustre/include/cl_object.h > > @@ -752,6 +752,17 @@ struct cl_page { > > struct cl_sync_io *cp_sync_io; > > }; > > > > +static inline enum cl_page_state cl_page_state_get(const struct cl_page > *pg) > > +{ > > + /* > > + * Paired with smp_store_release in cl_page_state_set_trust > > + * and ensures that we see the most recent value of cp_state > > + * even when the last modification was not performed on the > > + * current processor > > + */ > > + return smp_load_acquire(&pg->cp_state); > > +} > > + > > /** > > * Per-layer part of cl_page. > > * > > diff --git a/fs/lustre/llite/rw26.c b/fs/lustre/llite/rw26.c > > index e4ce3b6f5772..364dec208ccd 100644 > > --- a/fs/lustre/llite/rw26.c > > +++ b/fs/lustre/llite/rw26.c > > @@ -200,7 +200,7 @@ static ssize_t ll_direct_IO_seg(const struct lu_env > *env, struct cl_io *io, > > > > rc = cl_page_own(env, io, clp); > > if (rc) { > > - LASSERT(clp->cp_state == CPS_FREEING); > > + LASSERT(cl_page_state_get(clp) == CPS_FREEING); > > cl_page_put(env, clp); > > break; > > } > > diff --git a/fs/lustre/llite/vvp_page.c b/fs/lustre/llite/vvp_page.c > > index 590e5f5e43c9..38b8c488d765 100644 > > --- a/fs/lustre/llite/vvp_page.c > > +++ b/fs/lustre/llite/vvp_page.c > > @@ -323,18 +323,18 @@ static int vvp_page_make_ready(const struct lu_env > *env, > > > > lock_page(vmpage); > > if (clear_page_dirty_for_io(vmpage)) { > > - LASSERT(pg->cp_state == CPS_CACHED); > > + LASSERT(cl_page_state_get(pg) == CPS_CACHED); > > /* This actually clears the dirty bit in the radix tree. */ > > set_page_writeback(vmpage); > > CL_PAGE_HEADER(D_PAGE, env, pg, "readied\n"); > > - } else if (pg->cp_state == CPS_PAGEOUT) { > > + } else if (cl_page_state_get(pg) == CPS_PAGEOUT) { > > /* is it possible for osc_flush_async_page() to already > > * make it ready? > > */ > > result = -EALREADY; > > } else { > > CL_PAGE_DEBUG(D_ERROR, env, pg, "Unexpecting page state > %d.\n", > > - pg->cp_state); > > + cl_page_state_get(pg)); > > LBUG(); > > } > > unlock_page(vmpage); > > diff --git a/fs/lustre/obdclass/cl_page.c b/fs/lustre/obdclass/cl_page.c > > index 349f19e014e0..da4429b82932 100644 > > --- a/fs/lustre/obdclass/cl_page.c > > +++ b/fs/lustre/obdclass/cl_page.c > > @@ -97,7 +97,7 @@ static void cl_page_free(const struct lu_env *env, > struct cl_page *page) > > > > PASSERT(env, page, list_empty(&page->cp_batch)); > > PASSERT(env, page, !page->cp_owner); > > - PASSERT(env, page, page->cp_state == CPS_FREEING); > > + PASSERT(env, page, cl_page_state_get(page) == CPS_FREEING); > > > > while ((slice = list_first_entry_or_null(&page->cp_layers, > > struct cl_page_slice, > > @@ -119,8 +119,14 @@ static void cl_page_free(const struct lu_env *env, > struct cl_page *page) > > static inline void cl_page_state_set_trust(struct cl_page *page, > > enum cl_page_state state) > > { > > - /* bypass const. */ > > - *(enum cl_page_state *)&page->cp_state = state; > > + /* > > + * Paired with smp_load_acquire in cl_page_state_get > > + * and ensures that we see the most recent value of cp_state > > + * is available even when the next access is not performed on the > > + * current processor. > > + * Note we also cast away const as the only modifier of cp_state. > > + */ > > + smp_store_release((enum cl_page_state *)&page->cp_state, state); > > } > > > > struct cl_page *cl_page_alloc(const struct lu_env *env, > > @@ -270,10 +276,10 @@ static void __cl_page_state_set(const struct > lu_env *env, > > } > > }; > > > > - old = page->cp_state; > > + old = cl_page_state_get(page); > > PASSERT(env, page, allowed_transitions[old][state]); > > CL_PAGE_HEADER(D_TRACE, env, page, "%d -> %d\n", old, state); > > - PASSERT(env, page, page->cp_state == old); > > + PASSERT(env, page, cl_page_state_get(page) == old); > > PASSERT(env, page, equi(state == CPS_OWNED, page->cp_owner)); > > cl_page_state_set_trust(page, state); > > } > > @@ -313,7 +319,7 @@ void cl_page_put(const struct lu_env *env, struct > cl_page *page) > > refcount_read(&page->cp_ref)); > > > > if (refcount_dec_and_test(&page->cp_ref)) { > > - LASSERT(page->cp_state == CPS_FREEING); > > + LASSERT(cl_page_state_get(page) == CPS_FREEING); > > > > LASSERT(refcount_read(&page->cp_ref) == 0); > > PASSERT(env, page, !page->cp_owner); > > @@ -378,7 +384,7 @@ void __cl_page_disown(const struct lu_env *env, > > const struct cl_page_slice *slice; > > enum cl_page_state state; > > > > - state = pg->cp_state; > > + state = cl_page_state_get(pg); > > cl_page_owner_clear(pg); > > > > if (state == CPS_OWNED) > > @@ -402,7 +408,7 @@ int cl_page_is_owned(const struct cl_page *pg, const > struct cl_io *io) > > struct cl_io *top = cl_io_top((struct cl_io *)io); > > > > LINVRNT(cl_object_same(pg->cp_obj, io->ci_obj)); > > - return pg->cp_state == CPS_OWNED && pg->cp_owner == top; > > + return cl_page_state_get(pg) == CPS_OWNED && pg->cp_owner == top; > > } > > EXPORT_SYMBOL(cl_page_is_owned); > > > > @@ -434,7 +440,7 @@ static int __cl_page_own(const struct lu_env *env, > struct cl_io *io, > > > > io = cl_io_top(io); > > > > - if (pg->cp_state == CPS_FREEING) { > > + if (cl_page_state_get(pg) == CPS_FREEING) { > > result = -ENOENT; > > goto out; > > } > > @@ -453,7 +459,7 @@ static int __cl_page_own(const struct lu_env *env, > struct cl_io *io, > > PASSERT(env, pg, !pg->cp_owner); > > pg->cp_owner = cl_io_top(io); > > cl_page_owner_set(pg); > > - if (pg->cp_state != CPS_FREEING) { > > + if (cl_page_state_get(pg) != CPS_FREEING) { > > cl_page_state_set(env, pg, CPS_OWNED); > > } else { > > __cl_page_disown(env, io, pg); > > @@ -593,7 +599,7 @@ static void __cl_page_delete(const struct lu_env > *env, struct cl_page *pg) > > { > > const struct cl_page_slice *slice; > > > > - PASSERT(env, pg, pg->cp_state != CPS_FREEING); > > + PASSERT(env, pg, cl_page_state_get(pg) != CPS_FREEING); > > > > /* > > * Sever all ways to obtain new pointers to @pg. > > @@ -756,7 +762,7 @@ void cl_page_completion(const struct lu_env *env, > > const struct cl_page_slice *slice; > > > > PASSERT(env, pg, crt < CRT_NR); > > - PASSERT(env, pg, pg->cp_state == cl_req_type_state(crt)); > > + PASSERT(env, pg, cl_page_state_get(pg) == cl_req_type_state(crt)); > > > > CL_PAGE_HEADER(D_TRACE, env, pg, "%d %d\n", crt, ioret); > > > > @@ -805,7 +811,7 @@ int cl_page_make_ready(const struct lu_env *env, > struct cl_page *pg, > > } > > > > if (result >= 0) { > > - PASSERT(env, pg, pg->cp_state == CPS_CACHED); > > + PASSERT(env, pg, cl_page_state_get(pg) == CPS_CACHED); > > cl_page_io_start(env, pg, crt); > > result = 0; > > } > > @@ -870,7 +876,7 @@ void cl_page_header_print(const struct lu_env *env, > void *cookie, > > (*printer)(env, cookie, > > "page@%p[%d %p %d %d %p]\n", > > pg, refcount_read(&pg->cp_ref), pg->cp_obj, > > - pg->cp_state, pg->cp_type, > > + cl_page_state_get(pg), pg->cp_type, > > pg->cp_owner); > > } > > EXPORT_SYMBOL(cl_page_header_print); > > diff --git a/fs/lustre/obdecho/echo_client.c > b/fs/lustre/obdecho/echo_client.c > > index 317123fd27cb..d879f109e641 100644 > > --- a/fs/lustre/obdecho/echo_client.c > > +++ b/fs/lustre/obdecho/echo_client.c > > @@ -1046,7 +1046,7 @@ static int cl_echo_object_brw(struct echo_object > *eco, int rw, u64 offset, > > > > rc = cl_page_own(env, io, clp); > > if (rc) { > > - LASSERT(clp->cp_state == CPS_FREEING); > > + LASSERT(cl_page_state_get(clp) == CPS_FREEING); > > cl_page_put(env, clp); > > break; > > } > > diff --git a/fs/lustre/osc/osc_cache.c b/fs/lustre/osc/osc_cache.c > > index f8fddbfe6a7e..75984b98b229 100644 > > --- a/fs/lustre/osc/osc_cache.c > > +++ b/fs/lustre/osc/osc_cache.c > > @@ -1045,7 +1045,7 @@ static int osc_extent_truncate(struct osc_extent > *ext, pgoff_t trunc_index, > > cl_page_discard(env, io, page); > > cl_page_disown(env, io, page); > > } else { > > - LASSERT(page->cp_state == CPS_FREEING); > > + LASSERT(cl_page_state_get(page) == CPS_FREEING); > > LASSERT(0); > > } > > > > @@ -1356,10 +1356,12 @@ static int osc_completion(const struct lu_env > *env, struct osc_async_page *oap, > > int srvlock; > > > > cmd &= ~OBD_BRW_NOQUOTA; > > - LASSERTF(equi(page->cp_state == CPS_PAGEIN, cmd == OBD_BRW_READ), > > - "cp_state:%u, cmd:%d\n", page->cp_state, cmd); > > - LASSERTF(equi(page->cp_state == CPS_PAGEOUT, cmd == OBD_BRW_WRITE), > > - "cp_state:%u, cmd:%d\n", page->cp_state, cmd); > > + LASSERTF(equi(cl_page_state_get(page) == CPS_PAGEIN, > > + cmd == OBD_BRW_READ), > > + "cp_state:%u, cmd:%d\n", cl_page_state_get(page), cmd); > > + LASSERTF(equi(cl_page_state_get(page) == CPS_PAGEOUT, > > + cmd == OBD_BRW_WRITE), > > + "cp_state:%u, cmd:%d\n", cl_page_state_get(page), cmd); > > LASSERT(opg->ops_transfer_pinned); > > > > crt = cmd == OBD_BRW_READ ? CRT_READ : CRT_WRITE; > > @@ -3061,7 +3063,7 @@ bool osc_page_gang_lookup(const struct lu_env > *env, struct cl_io *io, > > > > page = ops->ops_cl.cpl_page; > > LASSERT(page->cp_type == CPT_CACHEABLE); > > - if (page->cp_state == CPS_FREEING) > > + if (cl_page_state_get(page) == CPS_FREEING) > > continue; > > > > cl_page_get(page); > > @@ -3142,7 +3144,7 @@ static bool check_and_discard_cb(const struct > lu_env *env, struct cl_io *io, > > cl_page_discard(env, io, page); > > cl_page_disown(env, io, page); > > } else { > > - LASSERT(page->cp_state == CPS_FREEING); > > + LASSERT(cl_page_state_get(page) == CPS_FREEING); > > } > > } > > > > @@ -3169,7 +3171,7 @@ static bool discard_cb(const struct lu_env *env, > struct cl_io *io, > > cl_page_discard(env, io, page); > > cl_page_disown(env, io, page); > > } else { > > - LASSERT(page->cp_state == CPS_FREEING); > > + LASSERT(cl_page_state_get(page) == CPS_FREEING); > > } > > > > return true; > > -- > > 2.17.1 > -------------- next part -------------- An HTML attachment was scrubbed... URL: From neilb at suse.com Tue Jun 11 00:49:17 2019 From: neilb at suse.com (NeilBrown) Date: Tue, 11 Jun 2019 10:49:17 +1000 Subject: [lustre-devel] [PATCH] Place a memory barrier around cp_state changes In-Reply-To: References: <20190606223110.15725-1-stancheff@cray.com> <87wohyw6ug.fsf@notabene.neil.brown.name> Message-ID: <87o935vtde.fsf@notabene.neil.brown.name> On Thu, Jun 06 2019, Shaun Tancheff wrote: > On Thu, Jun 6, 2019 at 7:57 PM NeilBrown wrote: >> >> On Thu, Jun 06 2019, Shaun Tancheff wrote: >> >> > Ensure all uses of cl_page->cp_state are smp-safe. >> > >> > LBUG Output: >> > In __cl_page_state_set() >> > .. >> > old = page->cp_state; >> > PASSERT(env, page, allowed_transitions[old][state]); >> > .. >> > Asserted with the following: >> > page at ffff80be1fcd6600[3 ffff80be1fca5cc0 0 1 ffff809e60e2 8cc0] I now see that this space..... ^^^ shouldn't be there.. ..... > Also known caller was cl_page_assume+0xdc/0x3e0 [obdclass] (from the > backtrace) > So we know the on-stack / register value of state == CPS_OWNED > > The backtrace: > ... > dump_backtrace+0x0/0x248 > show_stack+0x24/0x30 > dump_stack+0xbc/0xf4 > libcfs_call_trace+0xec/0x120 [libcfs] > lbug_with_loc+0x4c/0xb8 [libcfs] > cl_page_state_set0+0x2b4/0x6e0 [obdclass] > cl_page_assume+0xdc/0x3e0 [obdclass] > ll_io_read_page+0x144c/0x1de0 [lustre] > ll_write_begin+0x3d4/0xee8 [lustre] Thanks for this extra detail. I agree that the evidence suggests that page->cp_state changed (to CPS_CACHED) during the processing of the PASSERT(). It isn't clear to me though that this problem can be fixed just by adding barriers - particularly if this was on x86_64 as I suspect it is. Intel architecture processors have strongly ordered memory semantics. I'm not entirely sure what that means, but as smp_load_acquire() is very nearly a no-op on x86, I doubt that the memory barriers you added actually fix anything there. My understanding (based on a fairly shallow reading of the code) is that a partial-page write is being attempted, and the target page was not in memory, so it was read in synchronously. The fragment of ll_io_read_page happening is: if (anchor != NULL && !cl_page_is_owned(page, io)) { /* have sent */ rc = cl_sync_io_wait(env, anchor, 0); cl_page_assume(env, io, page); So cl_sync_io_wait() has just run. The page was (presumably) previously owned by something else. cl_sync_io_wait() waited for csi_sync_nr to drop to zero, which should say it isn't owned any more. cl_page_assume() then takes ownership. But it seems that the old ownership wasn't quite gone yet. The VM page remains locked the whole time, so the barriers implicit in locking and unlocking a page do not some into play here. There is an implicit barrier on the 'store' side when cl_sync_io_note() calls atomic_dec_and_test() to decrement csi_sync_nr, but I cannot see any implicit barrier on the 'load' side. If l_wait_event() in cl_sync_io_wait() took a spinlock, that would be enough. But it is possible for the wait to complete without any locking. So I can just about see the possibility of the race that you think you have hit. But it couldn't happen on x86, as it has strong memory ordering. What CPU architecture did this ASSERT trigger on? Thanks, NeilBrown -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 832 bytes Desc: not available URL: From shaun at tancheff.com Tue Jun 11 04:05:33 2019 From: shaun at tancheff.com (Shaun Tancheff) Date: Mon, 10 Jun 2019 23:05:33 -0500 Subject: [lustre-devel] [PATCH] Place a memory barrier around cp_state changes In-Reply-To: <87o935vtde.fsf@notabene.neil.brown.name> References: <20190606223110.15725-1-stancheff@cray.com> <87wohyw6ug.fsf@notabene.neil.brown.name> <87o935vtde.fsf@notabene.neil.brown.name> Message-ID: On Mon, Jun 10, 2019 at 7:49 PM NeilBrown wrote: > On Thu, Jun 06 2019, Shaun Tancheff wrote: > > > On Thu, Jun 6, 2019 at 7:57 PM NeilBrown wrote: > >> > >> On Thu, Jun 06 2019, Shaun Tancheff wrote: > >> > >> > Ensure all uses of cl_page->cp_state are smp-safe. > >> > > >> > LBUG Output: > >> > In __cl_page_state_set() > >> > .. > >> > old = page->cp_state; > >> > PASSERT(env, page, allowed_transitions[old][state]); > >> > .. > >> > Asserted with the following: > >> > page at ffff80be1fcd6600[3 ffff80be1fca5cc0 0 1 ffff809e60e2 8cc0] > > I now see that this space..... ^^^ > shouldn't be there.. > ..... > > > Also known caller was cl_page_assume+0xdc/0x3e0 [obdclass] (from the > > backtrace) > > So we know the on-stack / register value of state == CPS_OWNED > > > > The backtrace: > > ... > > dump_backtrace+0x0/0x248 > > show_stack+0x24/0x30 > > dump_stack+0xbc/0xf4 > > libcfs_call_trace+0xec/0x120 [libcfs] > > lbug_with_loc+0x4c/0xb8 [libcfs] > > cl_page_state_set0+0x2b4/0x6e0 [obdclass] > > cl_page_assume+0xdc/0x3e0 [obdclass] > > ll_io_read_page+0x144c/0x1de0 [lustre] > > ll_write_begin+0x3d4/0xee8 [lustre] > > Thanks for this extra detail. > I agree that the evidence suggests that page->cp_state changed (to > CPS_CACHED) during the processing of the PASSERT(). > > It isn't clear to me though that this problem can be fixed just by > adding barriers - particularly if this was on x86_64 as I suspect it > is. Intel architecture processors have strongly ordered memory > semantics. I'm not entirely sure what that means, but as > smp_load_acquire() is very nearly a no-op on x86, I doubt that the > memory barriers you added actually fix anything there. > Agree. This occurred on aarch64. I suspect one of the reasons that this has not been seen is that the majority of users are x86_64. It appears to be very rare on aarch64 but it could also, theoretically, happen on the other relaxed architectures. > My understanding (based on a fairly shallow reading of the code) is that > a partial-page write is being attempted, and the target page was not in > memory, so it was read in synchronously. > The fragment of ll_io_read_page happening is: > > if (anchor != NULL && !cl_page_is_owned(page, io)) { /* have sent > */ > rc = cl_sync_io_wait(env, anchor, 0); > > cl_page_assume(env, io, page); > > So cl_sync_io_wait() has just run. The page was (presumably) previously > owned by something else. cl_sync_io_wait() waited for csi_sync_nr > to drop to zero, which should say it isn't owned any more. > cl_page_assume() then takes ownership. > But it seems that the old ownership wasn't quite gone yet. > > The VM page remains locked the whole time, so the barriers implicit in > locking and unlocking a page do not some into play here. > There is an implicit barrier on the 'store' side when cl_sync_io_note() > calls atomic_dec_and_test() to decrement csi_sync_nr, but I cannot see > any implicit barrier on the 'load' side. If l_wait_event() in > cl_sync_io_wait() took a spinlock, that would be enough. But it is > possible for the wait to complete without any locking. > > So I can just about see the possibility of the race that you think you > have hit. But it couldn't happen on x86, as it has strong memory > ordering. > > What CPU architecture did this ASSERT trigger on? > Agree with everything above, unfortunately was aarch64. > Thanks, > NeilBrown > -------------- next part -------------- An HTML attachment was scrubbed... URL: From adilger at whamcloud.com Tue Jun 11 06:27:35 2019 From: adilger at whamcloud.com (Andreas Dilger) Date: Tue, 11 Jun 2019 06:27:35 +0000 Subject: [lustre-devel] lnet ip2nets option broken in master? In-Reply-To: <6FB8A669-6641-4579-B4F2-E687F7CCF116@amazon.com> References: <6FB8A669-6641-4579-B4F2-E687F7CCF116@amazon.com> Message-ID: Aurelien, I just saw your reply in my junk folder... :-( Could you please open an LU ticket, and possibly push a patch with the fix that is working for you, rather than me guessing what is supposed to be done. Cheers, Andreas On May 28, 2019, at 07:57, Degremont, Aurelien wrote: > > Oh, I was thinking of a smaller change. > This patch looks like (good) code refactoring. > > It seems the fix for ip2nets in master is indeed just a "ntohl(ifa->ifa_local);" for lnet_ipaddr_enumerate() > > > Aurélien > > Le 28/05/2019 15:13, « James Simmons » a écrit : > > > On Mon, 27 May 2019, Degremont, Aurelien wrote: > >> Hi Chris, >> >> Thanks for the info! >> >> @James, most of the patches you were waiting for are almost landed (last > ones are in master-next). >> Do you have a draft of the fix for ip2nets you can push in gerrit > fortestsonly? > > Yes I do but I need the the ko2iblnd fix to land as well. I embedded the > patch if you want to do an early review. This is a rough version but does > have light testing. I have some style changes to be sepatated out but > haven't got to it yet. I made this patch some time ago. So Neil be gentle > :-) > > diff --git a/lnet/include/lnet/lib-lnet.h b/lnet/include/lnet/lib-lnet.h > index 95c4df1..2eb161e 100644 > --- a/lnet/include/lnet/lib-lnet.h > +++ b/lnet/include/lnet/lib-lnet.h > @@ -40,9 +40,7 @@ > /* LNET has 0xeXXX */ > #define CFS_FAIL_PTLRPC_OST_BULK_CB2 0xe000 > > -#ifndef __KERNEL__ > -# error This include is only for kernel use. > -#endif > +#include > > #include > #include > @@ -841,6 +839,16 @@ int lnet_acceptor_port(void); > int lnet_acceptor_start(void); > void lnet_acceptor_stop(void); > > +struct lnet_inetdev { > + unsigned int li_interfaces; > + int li_flags; > + u32 li_ipaddr[LNET_INTERFACES_NUM]; > + u32 li_netmask[LNET_INTERFACES_NUM]; > + char li_name[LNET_INTERFACES_NUM][IFNAMSIZ]; > +}; > + > +int lnet_inet_enumerate(struct lnet_ni *ni, struct lnet_inetdev *ifaces, > + char *iname, int ifaces_done); > int lnet_sock_setbuf(struct socket *socket, int txbufsize, int rxbufsize); > int lnet_sock_getbuf(struct socket *socket, int *txbufsize, int *rxbufsize); > int lnet_sock_getaddr(struct socket *socket, bool remote, __u32 *ip, int *port); > diff --git a/lnet/klnds/gnilnd/gnilnd_aries.h b/lnet/klnds/gnilnd/gnilnd_aries.h > index d9698b6..431442f 100644 > --- a/lnet/klnds/gnilnd/gnilnd_aries.h > +++ b/lnet/klnds/gnilnd/gnilnd_aries.h > @@ -24,7 +24,6 @@ > #ifndef _GNILND_ARIES_H > #define _GNILND_ARIES_H > > -/* for lnet_ipif_query */ > #include > > #ifndef _GNILND_HSS_OPS_H > diff --git a/lnet/klnds/o2iblnd/o2iblnd.c b/lnet/klnds/o2iblnd/o2iblnd.c > index bf71ace..29335ed 100644 > --- a/lnet/klnds/o2iblnd/o2iblnd.c > +++ b/lnet/klnds/o2iblnd/o2iblnd.c > @@ -2887,80 +2887,6 @@ kiblnd_destroy_dev(struct kib_dev *dev) > LIBCFS_FREE(dev, sizeof(*dev)); > } > > -static struct kib_dev * > -kiblnd_create_dev(char *ifname) > -{ > - struct net_device *netdev; > - struct kib_dev *dev = NULL; > - int flags; > - int rc; > - > - rtnl_lock(); > - for_each_netdev(&init_net, netdev) { > - struct in_device *in_dev; > - > - if (strcmp(netdev->name, "lo") == 0) /* skip the loopback IF */ > - continue; > - > - flags = dev_get_flags(netdev); > - if (!(flags & IFF_UP)) { > - CERROR("Can't query IPoIB interface %s: it's down\n", > - netdev->name); > - goto unlock; > - } > - > - in_dev = __in_dev_get_rtnl(netdev); > - if (!in_dev) { > - CWARN("Interface %s has no IPv4 status.\n", > - netdev->name); > - goto unlock; > - } > - > - for_ifa(in_dev) > - if (strcmp(ifname, ifa->ifa_label) == 0) { > - LIBCFS_ALLOC(dev, sizeof(*dev)); > - if (!dev) > - goto unlock; > - > - dev->ibd_can_failover = !!(flags & IFF_MASTER); > - dev->ibd_ifip = ntohl(ifa->ifa_local); > - > - INIT_LIST_HEAD(&dev->ibd_nets); > - INIT_LIST_HEAD(&dev->ibd_list); /* not yet in kib_devs */ > - INIT_LIST_HEAD(&dev->ibd_fail_list); > - break; > - } > - endfor_ifa(in_dev); > - } > - rtnl_unlock(); > - > - if (!dev) { > - CERROR("Can't find any usable interfaces\n"); > - return NULL; > - } > - > - if (dev->ibd_ifip == 0) { > - CERROR("Can't initialize device: no IP address\n"); > - goto free_dev; > - } > - strcpy(&dev->ibd_ifname[0], ifname); > - > - /* initialize the device */ > - rc = kiblnd_dev_failover(dev); > - if (rc != 0) { > - CERROR("Can't initialize device: %d\n", rc); > - goto free_dev; > - } > - > - list_add_tail(&dev->ibd_list, &kiblnd_data.kib_devs); > - return dev; > -unlock: > - rtnl_unlock(); > -free_dev: > - LIBCFS_FREE(dev, sizeof(*dev)); > - return NULL; > -} > - > static void > kiblnd_base_shutdown(void) > { > @@ -3241,8 +3167,7 @@ kiblnd_start_schedulers(struct kib_sched_info *sched) > return rc; > } > > -static int > -kiblnd_dev_start_threads(struct kib_dev *dev, int newdev, u32 *cpts, int ncpts) > +static int kiblnd_dev_start_threads(struct kib_dev *dev, u32 *cpts, int ncpts) > { > int cpt; > int rc; > @@ -3254,7 +3179,7 @@ kiblnd_dev_start_threads(struct kib_dev *dev, int newdev, u32 *cpts, int ncpts) > cpt = (cpts == NULL) ? i : cpts[i]; > sched = kiblnd_data.kib_scheds[cpt]; > > - if (!newdev && sched->ibs_nthreads > 0) > + if (sched->ibs_nthreads > 0) > continue; > > rc = kiblnd_start_schedulers(kiblnd_data.kib_scheds[cpt]); > @@ -3267,49 +3192,15 @@ kiblnd_dev_start_threads(struct kib_dev *dev, int newdev, u32 *cpts, int ncpts) > return 0; > } > > -static struct kib_dev * > -kiblnd_dev_search(char *ifname) > -{ > - struct kib_dev *alias = NULL; > - struct kib_dev *dev; > - char *colon; > - char *colon2; > - > - colon = strchr(ifname, ':'); > - list_for_each_entry(dev, &kiblnd_data.kib_devs, ibd_list) { > - if (strcmp(&dev->ibd_ifname[0], ifname) == 0) > - return dev; > - > - if (alias != NULL) > - continue; > - > - colon2 = strchr(dev->ibd_ifname, ':'); > - if (colon != NULL) > - *colon = 0; > - if (colon2 != NULL) > - *colon2 = 0; > - > - if (strcmp(&dev->ibd_ifname[0], ifname) == 0) > - alias = dev; > - > - if (colon != NULL) > - *colon = ':'; > - if (colon2 != NULL) > - *colon2 = ':'; > - } > - return alias; > -} > - > static int > kiblnd_startup(struct lnet_ni *ni) > { > - char *ifname; > struct kib_dev *ibdev = NULL; > + struct lnet_inetdev ifaces; > struct kib_net *net; > unsigned long flags; > int rc; > - int newdev; > - int node_id; > + int i; > > LASSERT (ni->ni_net->net_lnd == &the_o2iblnd); > > @@ -3333,43 +3224,65 @@ kiblnd_startup(struct lnet_ni *ni) > * tcp bonding for ksocklnd. Multi-Rail wants each secondary > * IP to be treated as an unique 'struct ni' interfaces instead. > */ > - if (ni->ni_interfaces[0] != NULL) { > - /* Use the IPoIB interface specified in 'networks=' */ > - > - CLASSERT(LNET_INTERFACES_NUM > 1); > - if (ni->ni_interfaces[1] != NULL) { > - CERROR("Multiple interfaces not supported\n"); > + ifaces.li_interfaces = 0; > + if (!ni->ni_interfaces[0]) { > + rc = lnet_inet_enumerate(ni, &ifaces, NULL, 0); > + if (rc < 0) > goto failed; > - } > - > - ifname = ni->ni_interfaces[0]; > } else { > - ifname = *kiblnd_tunables.kib_default_ipif; > + for (i = 0; i < LNET_INTERFACES_NUM; i++) { > + int j; > + > + if (!ni->ni_interfaces[i]) > + break; > + > + for (j = 0; j < LNET_INTERFACES_NUM; j++) { > + if (strlen(ifaces.li_name[j]) && > + !strcmp(ifaces.li_name[j], > + ni->ni_interfaces[i])) { > + CERROR("found duplicate %s\n", > + ni->ni_interfaces[i]); > + rc = -EEXIST; > + goto failed; > + } > + } > + > + rc = lnet_inet_enumerate(ni, &ifaces, > + ni->ni_interfaces[i], 0); > + if (rc < 0) > + goto failed; > + } > } > > - if (strlen(ifname) >= sizeof(ibdev->ibd_ifname)) { > - CERROR("IPoIB interface name too long: %s\n", ifname); > - goto failed; > - } > + for (i = 0; i < ifaces.li_interfaces; i++) { > + LIBCFS_ALLOC(ibdev, sizeof(*ibdev)); > + if (!ibdev) > + goto failed; > > - ibdev = kiblnd_dev_search(ifname); > + ibdev->ibd_ifip = ifaces.li_ipaddr[i]; > + ibdev->ibd_can_failover = !!(ifaces.li_flags & IFF_MASTER); > + strlcpy(ibdev->ibd_ifname, ifaces.li_name[i], > + sizeof(ibdev->ibd_ifname)); > > - newdev = ibdev == NULL; > - /* hmm...create kib_dev even for alias */ > - if (ibdev == NULL || strcmp(&ibdev->ibd_ifname[0], ifname) != 0) > - ibdev = kiblnd_create_dev(ifname); > + INIT_LIST_HEAD(&ibdev->ibd_nets); > + INIT_LIST_HEAD(&ibdev->ibd_list); /* not yet in kib_devs */ > + INIT_LIST_HEAD(&ibdev->ibd_fail_list); > > - if (ibdev == NULL) > - goto failed; > + /* initialize the device */ > + rc = kiblnd_dev_failover(ibdev); > + if (rc) { > + CERROR("Can't initialize device: %d\n", rc); > + LIBCFS_FREE(ibdev, sizeof(*ibdev)); > + goto failed; > + } > > - node_id = dev_to_node(ibdev->ibd_hdev->ibh_ibdev->dma_device); > - ni->ni_dev_cpt = cfs_cpt_of_node(lnet_cpt_table(), node_id); > + list_add_tail(&ibdev->ibd_list, &kiblnd_data.kib_devs); > + } > > net->ibn_dev = ibdev; > ni->ni_nid = LNET_MKNID(LNET_NIDNET(ni->ni_nid), ibdev->ibd_ifip); > > - rc = kiblnd_dev_start_threads(ibdev, newdev, > - ni->ni_cpts, ni->ni_ncpts); > + rc = kiblnd_dev_start_threads(ibdev, ni->ni_cpts, ni->ni_ncpts); > if (rc != 0) > goto failed; > > diff --git a/lnet/klnds/socklnd/socklnd.c b/lnet/klnds/socklnd/socklnd.c > index 9cabeec..1657413 100644 > --- a/lnet/klnds/socklnd/socklnd.c > +++ b/lnet/klnds/socklnd/socklnd.c > @@ -2571,62 +2571,6 @@ ksocknal_shutdown(struct lnet_ni *ni) > } > > static int > -ksocknal_enumerate_interfaces(struct ksock_net *net, char *iname) > -{ > - struct net_device *dev; > - > - rtnl_lock(); > - for_each_netdev(&init_net, dev) { > - /* The iname specified by an user land configuration can > - * map to an ifa_label so always treat iname as an ifa_label. > - * If iname is NULL then fall back to the net device name. > - */ > - const char *name = iname ? iname : dev->name; > - struct in_device *in_dev; > - > - if (strcmp(dev->name, "lo") == 0) /* skip the loopback IF */ > - continue; > - > - if (!(dev_get_flags(dev) & IFF_UP)) { > - CWARN("Ignoring interface %s (down)\n", dev->name); > - continue; > - } > - > - in_dev = __in_dev_get_rtnl(dev); > - if (!in_dev) { > - CWARN("Interface %s has no IPv4 status.\n", dev->name); > - continue; > - } > - > - for_ifa(in_dev) > - if (strcmp(name, ifa->ifa_label) == 0) { > - int idx = net->ksnn_ninterfaces; > - struct ksock_interface *ksi; > - > - if (idx >= ARRAY_SIZE(net->ksnn_interfaces)) { > - rtnl_unlock(); > - return -E2BIG; > - } > - > - ksi = &net->ksnn_interfaces[idx]; > - ksi->ksni_ipaddr = ntohl(ifa->ifa_local); > - ksi->ksni_netmask = ifa->ifa_mask; > - strlcpy(ksi->ksni_name, > - name, sizeof(ksi->ksni_name)); > - net->ksnn_ninterfaces++; > - break; > - } > - endfor_ifa(in_dev); > - } > - rtnl_unlock(); > - > - if (net->ksnn_ninterfaces == 0) > - CERROR("Can't find any usable interfaces\n"); > - > - return net->ksnn_ninterfaces > 0 ? 0 : -ENOENT; > -} > - > -static int > ksocknal_search_new_ipif(struct ksock_net *net) > { > int new_ipif = 0; > @@ -2743,12 +2687,11 @@ ksocknal_net_start_threads(struct ksock_net *net, __u32 *cpts, int ncpts) > int > ksocknal_startup(struct lnet_ni *ni) > { > - struct ksock_net *net; > struct lnet_ioctl_config_lnd_cmn_tunables *net_tunables; > + struct lnet_inetdev ifaces; > + struct ksock_net *net; > int rc; > int i; > - struct net_device *net_dev; > - int node_id; > > LASSERT (ni->ni_net->net_lnd == &the_ksocklnd); > > @@ -2788,8 +2731,9 @@ ksocknal_startup(struct lnet_ni *ni) > net_tunables->lct_peer_rtr_credits = > *ksocknal_tunables.ksnd_peerrtrcredits; > > + ifaces.li_interfaces = 0; > if (!ni->ni_interfaces[0]) { > - rc = ksocknal_enumerate_interfaces(net, NULL); > + rc = lnet_inet_enumerate(ni, &ifaces, NULL, 0); > if (rc < 0) > goto fail_1; > } else { > @@ -2809,34 +2753,32 @@ ksocknal_startup(struct lnet_ni *ni) > if (!ni->ni_interfaces[i]) > break; > > - for (j = 0; j < net->ksnn_ninterfaces; j++) { > - struct ksock_interface *ksi; > - > - ksi = &net->ksnn_interfaces[j]; > - > - if (strcmp(ni->ni_interfaces[i], > - ksi->ksni_name) == 0) { > + for (j = 0; j < LNET_INTERFACES_NUM; j++) { > + if (strcmp(ifaces.li_name[j], > + ni->ni_interfaces[i]) == 0) { > CERROR("found duplicate %s\n", > - ksi->ksni_name); > + ni->ni_interfaces[i]); > rc = -EEXIST; > goto fail_1; > } > } > > - rc = ksocknal_enumerate_interfaces(net, ni->ni_interfaces[i]); > + rc = lnet_inet_enumerate(ni, &ifaces, > + ni->ni_interfaces[i], 0); > if (rc < 0) > goto fail_1; > } > } > > - net_dev = dev_get_by_name(&init_net, > - net->ksnn_interfaces[0].ksni_name); > - if (net_dev != NULL) { > - node_id = dev_to_node(&net_dev->dev); > - ni->ni_dev_cpt = cfs_cpt_of_node(lnet_cpt_table(), node_id); > - dev_put(net_dev); > - } else { > - ni->ni_dev_cpt = CFS_CPT_ANY; > + net->ksnn_ninterfaces = ifaces.li_interfaces; > + for (i = 0; i < net->ksnn_ninterfaces; i++) { > + struct ksock_interface *ksi; > + > + ksi = &net->ksnn_interfaces[i]; > + ksi->ksni_ipaddr = ifaces.li_ipaddr[i]; > + ksi->ksni_netmask = ifaces.li_netmask[i]; > + strlcpy(ksi->ksni_name, ifaces.li_name[i], > + sizeof(ksi->ksni_name)); > } > > /* call it before add it to ksocknal_data.ksnd_nets */ > diff --git a/lnet/lnet/config.c b/lnet/lnet/config.c > index 12b2442..666f0f0 100644 > --- a/lnet/lnet/config.c > +++ b/lnet/lnet/config.c > @@ -1611,34 +1611,28 @@ lnet_match_networks (char **networksp, char *ip2nets, __u32 *ipaddrs, int nip) > return count; > } > > -static void > -lnet_ipaddr_free_enumeration(__u32 *ipaddrs, int nip) > -{ > - LIBCFS_FREE(ipaddrs, nip * sizeof(*ipaddrs)); > -} > - > -static int > -lnet_ipaddr_enumerate(u32 **ipaddrsp) > +int lnet_inet_enumerate(struct lnet_ni *ni, struct lnet_inetdev *ifaces, > + char *iname, int ifa_done) > { > struct net_device *dev; > - u32 *ipaddrs; > - int nalloc = 64; > - int nip = 0; > - > - LIBCFS_ALLOC(ipaddrs, nalloc * sizeof(*ipaddrs)); > - if (!ipaddrs) { > - CERROR("Can't allocate ipaddrs[%d]\n", nalloc); > - return -ENOMEM; > - } > + int skip = 0; > + int rc = 0; > > rtnl_lock(); > for_each_netdev(&init_net, dev) { > + /* The iname specified by an user land configuration can > + * map to an ifa_label so always treat iname as an ifa_label. > + * If iname is NULL then fall back to the net device name. > + */ > + const char *name = iname ? iname : dev->name; > struct in_device *in_dev; > > - if (strcmp(dev->name, "lo") == 0) > + ifaces->li_flags = dev_get_flags(dev); > + > + if (ifaces->li_flags & IFF_LOOPBACK) /* skip the loopback IF */ > continue; > > - if (!(dev_get_flags(dev) & IFF_UP)) { > + if (!(ifaces->li_flags & IFF_UP)) { > CWARN("Ignoring interface %s: it's down\n", dev->name); > continue; > } > @@ -1649,65 +1643,109 @@ lnet_ipaddr_enumerate(u32 **ipaddrsp) > continue; > } > > - if (nip >= nalloc) { > - u32 *ipaddrs2; > + if (ni && ni->ni_dev_cpt != CFS_CPT_ANY) { > + int node_id = dev_to_node(&dev->dev); > + int cpt = cfs_cpt_of_node(lnet_cpt_table(), node_id); > > - nalloc += nalloc; > - ipaddrs2 = krealloc(ipaddrs, nalloc * sizeof(*ipaddrs2), > - GFP_KERNEL); > - if (!ipaddrs2) { > - kfree(ipaddrs); > - CERROR("Can't allocate ipaddrs[%d]\n", nip); > - return -ENOMEM; > + if (ni->ni_dev_cpt != cpt && > + ifaces->li_interfaces > 0) { > + ni->ni_dev_cpt = CFS_CPT_ANY; > + CWARN("Not all the network interfaces belong to the same CPT\n"); > + } else { > + ni->ni_dev_cpt = cpt; > } > - ipaddrs = ipaddrs2; > } > > - for_primary_ifa(in_dev) > - if (strcmp(ifa->ifa_label, dev->name) == 0) { > - ipaddrs[nip++] = ifa->ifa_local; > - break; > + for_ifa(in_dev) > + if (strcmp(name, ifa->ifa_label) == 0) { > + int idx = ifaces->li_interfaces; > + > + if (idx >= ARRAY_SIZE(ifaces->li_name)) > + return -E2BIG; > + > + if (skip++ < ifa_done) > + continue; > + > + ifaces->li_ipaddr[idx] = ntohl(ifa->ifa_local); > + ifaces->li_netmask[idx] = ifa->ifa_mask; > + strlcpy(ifaces->li_name[idx], name, > + sizeof(ifaces->li_name[idx])); > + ifaces->li_interfaces++; > + goto unlock_rtnl; > } > endfor_ifa(in_dev); > } > +unlock_rtnl: > rtnl_unlock(); > > - *ipaddrsp = ipaddrs; > - return nip; > + if (ifaces->li_interfaces == 0) { > + CERROR("Can't find any usable interfaces\n"); > + rc = -ENOENT; > + } > + > + return rc; > } > +EXPORT_SYMBOL(lnet_inet_enumerate); > > int > -lnet_parse_ip2nets (char **networksp, char *ip2nets) > +lnet_parse_ip2nets(char **networksp, char *ip2nets) > { > - __u32 *ipaddrs = NULL; > - int nip = lnet_ipaddr_enumerate(&ipaddrs); > - int rc; > - > - if (nip < 0) { > - LCONSOLE_ERROR_MSG(0x117, "Error %d enumerating local IP " > - "interfaces for ip2nets to match\n", nip); > - return nip; > - } > + struct lnet_inetdev ifaces; > + u32 *ipaddrs = NULL; > + int nalloc = 0; > + int nip = 0; > + int rc; > > - if (nip == 0) { > - LCONSOLE_ERROR_MSG(0x118, "No local IP interfaces " > - "for ip2nets to match\n"); > - return -ENOENT; > - } > +realloc: > + ifaces.li_interfaces = 0; > + rc = lnet_inet_enumerate(NULL, &ifaces, NULL, nip); > + if (rc < 0) { > + /* Handle edge case where we nip is muliple of > + * LNET_INTERFACES_NUM and no more devices are > + * detected. > + */ > + if (rc == -ENOENT && nip > 0) > + goto match_networks; > > - rc = lnet_match_networks(networksp, ip2nets, ipaddrs, nip); > - lnet_ipaddr_free_enumeration(ipaddrs, nip); > + LCONSOLE_ERROR_MSG(0x117, > + "Error %d enumerating local IP interfaces for ip2nets to match\n", > + rc); > + goto out_free_addrs; > + } else { > + u32 *ipaddrs2; > + int i; > + > + nalloc += ifaces.li_interfaces; > + ipaddrs2 = krealloc(ipaddrs, nalloc * sizeof(*ipaddrs), > + GFP_KERNEL); > + if (!ipaddrs2) { > + CERROR("Can't allocate ipaddrs[%d]\n", nalloc); > + goto out_free_addrs; > + } > + ipaddrs = ipaddrs2; > > + for (i = 0; i < ifaces.li_interfaces; i++) { > + memcpy(ipaddrs + nip, ifaces.li_ipaddr, > + sizeof(*ipaddrs) * LNET_INTERFACES_NUM); > + } > + > + /* Maybe more ? */ > + if (ifaces.li_interfaces == LNET_INTERFACES_NUM) { > + nip += LNET_INTERFACES_NUM; > + goto realloc; > + } > + } > +match_networks: > + rc = lnet_match_networks(networksp, ip2nets, ipaddrs, nip); > if (rc < 0) { > LCONSOLE_ERROR_MSG(0x119, "Error %d parsing ip2nets\n", rc); > return rc; > - } > - > - if (rc == 0) { > + } else if (rc == 0) { > LCONSOLE_ERROR_MSG(0x11a, "ip2nets does not match " > "any local IP interfaces\n"); > return -ENOENT; > } > - > - return 0; > +out_free_addrs: > + LIBCFS_FREE(ipaddrs, nip * sizeof(*ipaddrs)); > + return rc; > } > -- > 1.8.3.1 > > > > _______________________________________________ > lustre-devel mailing list > lustre-devel at lists.lustre.org > http://lists.lustre.org/listinfo.cgi/lustre-devel-lustre.org Cheers, Andreas -- Andreas Dilger Principal Lustre Architect Whamcloud From degremoa at amazon.com Tue Jun 11 09:02:37 2019 From: degremoa at amazon.com (Degremont, Aurelien) Date: Tue, 11 Jun 2019 09:02:37 +0000 Subject: [lustre-devel] lnet ip2nets option broken in master? In-Reply-To: References: <6FB8A669-6641-4579-B4F2-E687F7CCF116@amazon.com> Message-ID: <18F876C0-9EF5-4536-B227-7E03A84A5172@amazon.com> Since, dependencies have landed and James pushed its patch, https://review.whamcloud.com/#/c/34993/ I don't want to have a patch competing with this one if this one is likely to land. Le 11/06/2019 08:28, « Andreas Dilger » a écrit : Aurelien, I just saw your reply in my junk folder... :-( Could you please open an LU ticket, and possibly push a patch with the fix that is working for you, rather than me guessing what is supposed to be done. Cheers, Andreas On May 28, 2019, at 07:57, Degremont, Aurelien wrote: > > Oh, I was thinking of a smaller change. > This patch looks like (good) code refactoring. > > It seems the fix for ip2nets in master is indeed just a "ntohl(ifa->ifa_local);" for lnet_ipaddr_enumerate() > > > Aurélien > > Le 28/05/2019 15:13, « James Simmons » a écrit : > > > On Mon, 27 May 2019, Degremont, Aurelien wrote: > >> Hi Chris, >> >> Thanks for the info! >> >> @James, most of the patches you were waiting for are almost landed (last > ones are in master-next). >> Do you have a draft of the fix for ip2nets you can push in gerrit > fortestsonly? > > Yes I do but I need the the ko2iblnd fix to land as well. I embedded the > patch if you want to do an early review. This is a rough version but does > have light testing. I have some style changes to be sepatated out but > haven't got to it yet. I made this patch some time ago. So Neil be gentle > :-) > > diff --git a/lnet/include/lnet/lib-lnet.h b/lnet/include/lnet/lib-lnet.h > index 95c4df1..2eb161e 100644 > --- a/lnet/include/lnet/lib-lnet.h > +++ b/lnet/include/lnet/lib-lnet.h > @@ -40,9 +40,7 @@ > /* LNET has 0xeXXX */ > #define CFS_FAIL_PTLRPC_OST_BULK_CB2 0xe000 > > -#ifndef __KERNEL__ > -# error This include is only for kernel use. > -#endif > +#include > > #include > #include > @@ -841,6 +839,16 @@ int lnet_acceptor_port(void); > int lnet_acceptor_start(void); > void lnet_acceptor_stop(void); > > +struct lnet_inetdev { > + unsigned int li_interfaces; > + int li_flags; > + u32 li_ipaddr[LNET_INTERFACES_NUM]; > + u32 li_netmask[LNET_INTERFACES_NUM]; > + char li_name[LNET_INTERFACES_NUM][IFNAMSIZ]; > +}; > + > +int lnet_inet_enumerate(struct lnet_ni *ni, struct lnet_inetdev *ifaces, > + char *iname, int ifaces_done); > int lnet_sock_setbuf(struct socket *socket, int txbufsize, int rxbufsize); > int lnet_sock_getbuf(struct socket *socket, int *txbufsize, int *rxbufsize); > int lnet_sock_getaddr(struct socket *socket, bool remote, __u32 *ip, int *port); > diff --git a/lnet/klnds/gnilnd/gnilnd_aries.h b/lnet/klnds/gnilnd/gnilnd_aries.h > index d9698b6..431442f 100644 > --- a/lnet/klnds/gnilnd/gnilnd_aries.h > +++ b/lnet/klnds/gnilnd/gnilnd_aries.h > @@ -24,7 +24,6 @@ > #ifndef _GNILND_ARIES_H > #define _GNILND_ARIES_H > > -/* for lnet_ipif_query */ > #include > > #ifndef _GNILND_HSS_OPS_H > diff --git a/lnet/klnds/o2iblnd/o2iblnd.c b/lnet/klnds/o2iblnd/o2iblnd.c > index bf71ace..29335ed 100644 > --- a/lnet/klnds/o2iblnd/o2iblnd.c > +++ b/lnet/klnds/o2iblnd/o2iblnd.c > @@ -2887,80 +2887,6 @@ kiblnd_destroy_dev(struct kib_dev *dev) > LIBCFS_FREE(dev, sizeof(*dev)); > } > > -static struct kib_dev * > -kiblnd_create_dev(char *ifname) > -{ > - struct net_device *netdev; > - struct kib_dev *dev = NULL; > - int flags; > - int rc; > - > - rtnl_lock(); > - for_each_netdev(&init_net, netdev) { > - struct in_device *in_dev; > - > - if (strcmp(netdev->name, "lo") == 0) /* skip the loopback IF */ > - continue; > - > - flags = dev_get_flags(netdev); > - if (!(flags & IFF_UP)) { > - CERROR("Can't query IPoIB interface %s: it's down\n", > - netdev->name); > - goto unlock; > - } > - > - in_dev = __in_dev_get_rtnl(netdev); > - if (!in_dev) { > - CWARN("Interface %s has no IPv4 status.\n", > - netdev->name); > - goto unlock; > - } > - > - for_ifa(in_dev) > - if (strcmp(ifname, ifa->ifa_label) == 0) { > - LIBCFS_ALLOC(dev, sizeof(*dev)); > - if (!dev) > - goto unlock; > - > - dev->ibd_can_failover = !!(flags & IFF_MASTER); > - dev->ibd_ifip = ntohl(ifa->ifa_local); > - > - INIT_LIST_HEAD(&dev->ibd_nets); > - INIT_LIST_HEAD(&dev->ibd_list); /* not yet in kib_devs */ > - INIT_LIST_HEAD(&dev->ibd_fail_list); > - break; > - } > - endfor_ifa(in_dev); > - } > - rtnl_unlock(); > - > - if (!dev) { > - CERROR("Can't find any usable interfaces\n"); > - return NULL; > - } > - > - if (dev->ibd_ifip == 0) { > - CERROR("Can't initialize device: no IP address\n"); > - goto free_dev; > - } > - strcpy(&dev->ibd_ifname[0], ifname); > - > - /* initialize the device */ > - rc = kiblnd_dev_failover(dev); > - if (rc != 0) { > - CERROR("Can't initialize device: %d\n", rc); > - goto free_dev; > - } > - > - list_add_tail(&dev->ibd_list, &kiblnd_data.kib_devs); > - return dev; > -unlock: > - rtnl_unlock(); > -free_dev: > - LIBCFS_FREE(dev, sizeof(*dev)); > - return NULL; > -} > - > static void > kiblnd_base_shutdown(void) > { > @@ -3241,8 +3167,7 @@ kiblnd_start_schedulers(struct kib_sched_info *sched) > return rc; > } > > -static int > -kiblnd_dev_start_threads(struct kib_dev *dev, int newdev, u32 *cpts, int ncpts) > +static int kiblnd_dev_start_threads(struct kib_dev *dev, u32 *cpts, int ncpts) > { > int cpt; > int rc; > @@ -3254,7 +3179,7 @@ kiblnd_dev_start_threads(struct kib_dev *dev, int newdev, u32 *cpts, int ncpts) > cpt = (cpts == NULL) ? i : cpts[i]; > sched = kiblnd_data.kib_scheds[cpt]; > > - if (!newdev && sched->ibs_nthreads > 0) > + if (sched->ibs_nthreads > 0) > continue; > > rc = kiblnd_start_schedulers(kiblnd_data.kib_scheds[cpt]); > @@ -3267,49 +3192,15 @@ kiblnd_dev_start_threads(struct kib_dev *dev, int newdev, u32 *cpts, int ncpts) > return 0; > } > > -static struct kib_dev * > -kiblnd_dev_search(char *ifname) > -{ > - struct kib_dev *alias = NULL; > - struct kib_dev *dev; > - char *colon; > - char *colon2; > - > - colon = strchr(ifname, ':'); > - list_for_each_entry(dev, &kiblnd_data.kib_devs, ibd_list) { > - if (strcmp(&dev->ibd_ifname[0], ifname) == 0) > - return dev; > - > - if (alias != NULL) > - continue; > - > - colon2 = strchr(dev->ibd_ifname, ':'); > - if (colon != NULL) > - *colon = 0; > - if (colon2 != NULL) > - *colon2 = 0; > - > - if (strcmp(&dev->ibd_ifname[0], ifname) == 0) > - alias = dev; > - > - if (colon != NULL) > - *colon = ':'; > - if (colon2 != NULL) > - *colon2 = ':'; > - } > - return alias; > -} > - > static int > kiblnd_startup(struct lnet_ni *ni) > { > - char *ifname; > struct kib_dev *ibdev = NULL; > + struct lnet_inetdev ifaces; > struct kib_net *net; > unsigned long flags; > int rc; > - int newdev; > - int node_id; > + int i; > > LASSERT (ni->ni_net->net_lnd == &the_o2iblnd); > > @@ -3333,43 +3224,65 @@ kiblnd_startup(struct lnet_ni *ni) > * tcp bonding for ksocklnd. Multi-Rail wants each secondary > * IP to be treated as an unique 'struct ni' interfaces instead. > */ > - if (ni->ni_interfaces[0] != NULL) { > - /* Use the IPoIB interface specified in 'networks=' */ > - > - CLASSERT(LNET_INTERFACES_NUM > 1); > - if (ni->ni_interfaces[1] != NULL) { > - CERROR("Multiple interfaces not supported\n"); > + ifaces.li_interfaces = 0; > + if (!ni->ni_interfaces[0]) { > + rc = lnet_inet_enumerate(ni, &ifaces, NULL, 0); > + if (rc < 0) > goto failed; > - } > - > - ifname = ni->ni_interfaces[0]; > } else { > - ifname = *kiblnd_tunables.kib_default_ipif; > + for (i = 0; i < LNET_INTERFACES_NUM; i++) { > + int j; > + > + if (!ni->ni_interfaces[i]) > + break; > + > + for (j = 0; j < LNET_INTERFACES_NUM; j++) { > + if (strlen(ifaces.li_name[j]) && > + !strcmp(ifaces.li_name[j], > + ni->ni_interfaces[i])) { > + CERROR("found duplicate %s\n", > + ni->ni_interfaces[i]); > + rc = -EEXIST; > + goto failed; > + } > + } > + > + rc = lnet_inet_enumerate(ni, &ifaces, > + ni->ni_interfaces[i], 0); > + if (rc < 0) > + goto failed; > + } > } > > - if (strlen(ifname) >= sizeof(ibdev->ibd_ifname)) { > - CERROR("IPoIB interface name too long: %s\n", ifname); > - goto failed; > - } > + for (i = 0; i < ifaces.li_interfaces; i++) { > + LIBCFS_ALLOC(ibdev, sizeof(*ibdev)); > + if (!ibdev) > + goto failed; > > - ibdev = kiblnd_dev_search(ifname); > + ibdev->ibd_ifip = ifaces.li_ipaddr[i]; > + ibdev->ibd_can_failover = !!(ifaces.li_flags & IFF_MASTER); > + strlcpy(ibdev->ibd_ifname, ifaces.li_name[i], > + sizeof(ibdev->ibd_ifname)); > > - newdev = ibdev == NULL; > - /* hmm...create kib_dev even for alias */ > - if (ibdev == NULL || strcmp(&ibdev->ibd_ifname[0], ifname) != 0) > - ibdev = kiblnd_create_dev(ifname); > + INIT_LIST_HEAD(&ibdev->ibd_nets); > + INIT_LIST_HEAD(&ibdev->ibd_list); /* not yet in kib_devs */ > + INIT_LIST_HEAD(&ibdev->ibd_fail_list); > > - if (ibdev == NULL) > - goto failed; > + /* initialize the device */ > + rc = kiblnd_dev_failover(ibdev); > + if (rc) { > + CERROR("Can't initialize device: %d\n", rc); > + LIBCFS_FREE(ibdev, sizeof(*ibdev)); > + goto failed; > + } > > - node_id = dev_to_node(ibdev->ibd_hdev->ibh_ibdev->dma_device); > - ni->ni_dev_cpt = cfs_cpt_of_node(lnet_cpt_table(), node_id); > + list_add_tail(&ibdev->ibd_list, &kiblnd_data.kib_devs); > + } > > net->ibn_dev = ibdev; > ni->ni_nid = LNET_MKNID(LNET_NIDNET(ni->ni_nid), ibdev->ibd_ifip); > > - rc = kiblnd_dev_start_threads(ibdev, newdev, > - ni->ni_cpts, ni->ni_ncpts); > + rc = kiblnd_dev_start_threads(ibdev, ni->ni_cpts, ni->ni_ncpts); > if (rc != 0) > goto failed; > > diff --git a/lnet/klnds/socklnd/socklnd.c b/lnet/klnds/socklnd/socklnd.c > index 9cabeec..1657413 100644 > --- a/lnet/klnds/socklnd/socklnd.c > +++ b/lnet/klnds/socklnd/socklnd.c > @@ -2571,62 +2571,6 @@ ksocknal_shutdown(struct lnet_ni *ni) > } > > static int > -ksocknal_enumerate_interfaces(struct ksock_net *net, char *iname) > -{ > - struct net_device *dev; > - > - rtnl_lock(); > - for_each_netdev(&init_net, dev) { > - /* The iname specified by an user land configuration can > - * map to an ifa_label so always treat iname as an ifa_label. > - * If iname is NULL then fall back to the net device name. > - */ > - const char *name = iname ? iname : dev->name; > - struct in_device *in_dev; > - > - if (strcmp(dev->name, "lo") == 0) /* skip the loopback IF */ > - continue; > - > - if (!(dev_get_flags(dev) & IFF_UP)) { > - CWARN("Ignoring interface %s (down)\n", dev->name); > - continue; > - } > - > - in_dev = __in_dev_get_rtnl(dev); > - if (!in_dev) { > - CWARN("Interface %s has no IPv4 status.\n", dev->name); > - continue; > - } > - > - for_ifa(in_dev) > - if (strcmp(name, ifa->ifa_label) == 0) { > - int idx = net->ksnn_ninterfaces; > - struct ksock_interface *ksi; > - > - if (idx >= ARRAY_SIZE(net->ksnn_interfaces)) { > - rtnl_unlock(); > - return -E2BIG; > - } > - > - ksi = &net->ksnn_interfaces[idx]; > - ksi->ksni_ipaddr = ntohl(ifa->ifa_local); > - ksi->ksni_netmask = ifa->ifa_mask; > - strlcpy(ksi->ksni_name, > - name, sizeof(ksi->ksni_name)); > - net->ksnn_ninterfaces++; > - break; > - } > - endfor_ifa(in_dev); > - } > - rtnl_unlock(); > - > - if (net->ksnn_ninterfaces == 0) > - CERROR("Can't find any usable interfaces\n"); > - > - return net->ksnn_ninterfaces > 0 ? 0 : -ENOENT; > -} > - > -static int > ksocknal_search_new_ipif(struct ksock_net *net) > { > int new_ipif = 0; > @@ -2743,12 +2687,11 @@ ksocknal_net_start_threads(struct ksock_net *net, __u32 *cpts, int ncpts) > int > ksocknal_startup(struct lnet_ni *ni) > { > - struct ksock_net *net; > struct lnet_ioctl_config_lnd_cmn_tunables *net_tunables; > + struct lnet_inetdev ifaces; > + struct ksock_net *net; > int rc; > int i; > - struct net_device *net_dev; > - int node_id; > > LASSERT (ni->ni_net->net_lnd == &the_ksocklnd); > > @@ -2788,8 +2731,9 @@ ksocknal_startup(struct lnet_ni *ni) > net_tunables->lct_peer_rtr_credits = > *ksocknal_tunables.ksnd_peerrtrcredits; > > + ifaces.li_interfaces = 0; > if (!ni->ni_interfaces[0]) { > - rc = ksocknal_enumerate_interfaces(net, NULL); > + rc = lnet_inet_enumerate(ni, &ifaces, NULL, 0); > if (rc < 0) > goto fail_1; > } else { > @@ -2809,34 +2753,32 @@ ksocknal_startup(struct lnet_ni *ni) > if (!ni->ni_interfaces[i]) > break; > > - for (j = 0; j < net->ksnn_ninterfaces; j++) { > - struct ksock_interface *ksi; > - > - ksi = &net->ksnn_interfaces[j]; > - > - if (strcmp(ni->ni_interfaces[i], > - ksi->ksni_name) == 0) { > + for (j = 0; j < LNET_INTERFACES_NUM; j++) { > + if (strcmp(ifaces.li_name[j], > + ni->ni_interfaces[i]) == 0) { > CERROR("found duplicate %s\n", > - ksi->ksni_name); > + ni->ni_interfaces[i]); > rc = -EEXIST; > goto fail_1; > } > } > > - rc = ksocknal_enumerate_interfaces(net, ni->ni_interfaces[i]); > + rc = lnet_inet_enumerate(ni, &ifaces, > + ni->ni_interfaces[i], 0); > if (rc < 0) > goto fail_1; > } > } > > - net_dev = dev_get_by_name(&init_net, > - net->ksnn_interfaces[0].ksni_name); > - if (net_dev != NULL) { > - node_id = dev_to_node(&net_dev->dev); > - ni->ni_dev_cpt = cfs_cpt_of_node(lnet_cpt_table(), node_id); > - dev_put(net_dev); > - } else { > - ni->ni_dev_cpt = CFS_CPT_ANY; > + net->ksnn_ninterfaces = ifaces.li_interfaces; > + for (i = 0; i < net->ksnn_ninterfaces; i++) { > + struct ksock_interface *ksi; > + > + ksi = &net->ksnn_interfaces[i]; > + ksi->ksni_ipaddr = ifaces.li_ipaddr[i]; > + ksi->ksni_netmask = ifaces.li_netmask[i]; > + strlcpy(ksi->ksni_name, ifaces.li_name[i], > + sizeof(ksi->ksni_name)); > } > > /* call it before add it to ksocknal_data.ksnd_nets */ > diff --git a/lnet/lnet/config.c b/lnet/lnet/config.c > index 12b2442..666f0f0 100644 > --- a/lnet/lnet/config.c > +++ b/lnet/lnet/config.c > @@ -1611,34 +1611,28 @@ lnet_match_networks (char **networksp, char *ip2nets, __u32 *ipaddrs, int nip) > return count; > } > > -static void > -lnet_ipaddr_free_enumeration(__u32 *ipaddrs, int nip) > -{ > - LIBCFS_FREE(ipaddrs, nip * sizeof(*ipaddrs)); > -} > - > -static int > -lnet_ipaddr_enumerate(u32 **ipaddrsp) > +int lnet_inet_enumerate(struct lnet_ni *ni, struct lnet_inetdev *ifaces, > + char *iname, int ifa_done) > { > struct net_device *dev; > - u32 *ipaddrs; > - int nalloc = 64; > - int nip = 0; > - > - LIBCFS_ALLOC(ipaddrs, nalloc * sizeof(*ipaddrs)); > - if (!ipaddrs) { > - CERROR("Can't allocate ipaddrs[%d]\n", nalloc); > - return -ENOMEM; > - } > + int skip = 0; > + int rc = 0; > > rtnl_lock(); > for_each_netdev(&init_net, dev) { > + /* The iname specified by an user land configuration can > + * map to an ifa_label so always treat iname as an ifa_label. > + * If iname is NULL then fall back to the net device name. > + */ > + const char *name = iname ? iname : dev->name; > struct in_device *in_dev; > > - if (strcmp(dev->name, "lo") == 0) > + ifaces->li_flags = dev_get_flags(dev); > + > + if (ifaces->li_flags & IFF_LOOPBACK) /* skip the loopback IF */ > continue; > > - if (!(dev_get_flags(dev) & IFF_UP)) { > + if (!(ifaces->li_flags & IFF_UP)) { > CWARN("Ignoring interface %s: it's down\n", dev->name); > continue; > } > @@ -1649,65 +1643,109 @@ lnet_ipaddr_enumerate(u32 **ipaddrsp) > continue; > } > > - if (nip >= nalloc) { > - u32 *ipaddrs2; > + if (ni && ni->ni_dev_cpt != CFS_CPT_ANY) { > + int node_id = dev_to_node(&dev->dev); > + int cpt = cfs_cpt_of_node(lnet_cpt_table(), node_id); > > - nalloc += nalloc; > - ipaddrs2 = krealloc(ipaddrs, nalloc * sizeof(*ipaddrs2), > - GFP_KERNEL); > - if (!ipaddrs2) { > - kfree(ipaddrs); > - CERROR("Can't allocate ipaddrs[%d]\n", nip); > - return -ENOMEM; > + if (ni->ni_dev_cpt != cpt && > + ifaces->li_interfaces > 0) { > + ni->ni_dev_cpt = CFS_CPT_ANY; > + CWARN("Not all the network interfaces belong to the same CPT\n"); > + } else { > + ni->ni_dev_cpt = cpt; > } > - ipaddrs = ipaddrs2; > } > > - for_primary_ifa(in_dev) > - if (strcmp(ifa->ifa_label, dev->name) == 0) { > - ipaddrs[nip++] = ifa->ifa_local; > - break; > + for_ifa(in_dev) > + if (strcmp(name, ifa->ifa_label) == 0) { > + int idx = ifaces->li_interfaces; > + > + if (idx >= ARRAY_SIZE(ifaces->li_name)) > + return -E2BIG; > + > + if (skip++ < ifa_done) > + continue; > + > + ifaces->li_ipaddr[idx] = ntohl(ifa->ifa_local); > + ifaces->li_netmask[idx] = ifa->ifa_mask; > + strlcpy(ifaces->li_name[idx], name, > + sizeof(ifaces->li_name[idx])); > + ifaces->li_interfaces++; > + goto unlock_rtnl; > } > endfor_ifa(in_dev); > } > +unlock_rtnl: > rtnl_unlock(); > > - *ipaddrsp = ipaddrs; > - return nip; > + if (ifaces->li_interfaces == 0) { > + CERROR("Can't find any usable interfaces\n"); > + rc = -ENOENT; > + } > + > + return rc; > } > +EXPORT_SYMBOL(lnet_inet_enumerate); > > int > -lnet_parse_ip2nets (char **networksp, char *ip2nets) > +lnet_parse_ip2nets(char **networksp, char *ip2nets) > { > - __u32 *ipaddrs = NULL; > - int nip = lnet_ipaddr_enumerate(&ipaddrs); > - int rc; > - > - if (nip < 0) { > - LCONSOLE_ERROR_MSG(0x117, "Error %d enumerating local IP " > - "interfaces for ip2nets to match\n", nip); > - return nip; > - } > + struct lnet_inetdev ifaces; > + u32 *ipaddrs = NULL; > + int nalloc = 0; > + int nip = 0; > + int rc; > > - if (nip == 0) { > - LCONSOLE_ERROR_MSG(0x118, "No local IP interfaces " > - "for ip2nets to match\n"); > - return -ENOENT; > - } > +realloc: > + ifaces.li_interfaces = 0; > + rc = lnet_inet_enumerate(NULL, &ifaces, NULL, nip); > + if (rc < 0) { > + /* Handle edge case where we nip is muliple of > + * LNET_INTERFACES_NUM and no more devices are > + * detected. > + */ > + if (rc == -ENOENT && nip > 0) > + goto match_networks; > > - rc = lnet_match_networks(networksp, ip2nets, ipaddrs, nip); > - lnet_ipaddr_free_enumeration(ipaddrs, nip); > + LCONSOLE_ERROR_MSG(0x117, > + "Error %d enumerating local IP interfaces for ip2nets to match\n", > + rc); > + goto out_free_addrs; > + } else { > + u32 *ipaddrs2; > + int i; > + > + nalloc += ifaces.li_interfaces; > + ipaddrs2 = krealloc(ipaddrs, nalloc * sizeof(*ipaddrs), > + GFP_KERNEL); > + if (!ipaddrs2) { > + CERROR("Can't allocate ipaddrs[%d]\n", nalloc); > + goto out_free_addrs; > + } > + ipaddrs = ipaddrs2; > > + for (i = 0; i < ifaces.li_interfaces; i++) { > + memcpy(ipaddrs + nip, ifaces.li_ipaddr, > + sizeof(*ipaddrs) * LNET_INTERFACES_NUM); > + } > + > + /* Maybe more ? */ > + if (ifaces.li_interfaces == LNET_INTERFACES_NUM) { > + nip += LNET_INTERFACES_NUM; > + goto realloc; > + } > + } > +match_networks: > + rc = lnet_match_networks(networksp, ip2nets, ipaddrs, nip); > if (rc < 0) { > LCONSOLE_ERROR_MSG(0x119, "Error %d parsing ip2nets\n", rc); > return rc; > - } > - > - if (rc == 0) { > + } else if (rc == 0) { > LCONSOLE_ERROR_MSG(0x11a, "ip2nets does not match " > "any local IP interfaces\n"); > return -ENOENT; > } > - > - return 0; > +out_free_addrs: > + LIBCFS_FREE(ipaddrs, nip * sizeof(*ipaddrs)); > + return rc; > } > -- > 1.8.3.1 > > > > _______________________________________________ > lustre-devel mailing list > lustre-devel at lists.lustre.org > http://lists.lustre.org/listinfo.cgi/lustre-devel-lustre.org Cheers, Andreas -- Andreas Dilger Principal Lustre Architect Whamcloud From neilb at suse.com Wed Jun 12 04:08:49 2019 From: neilb at suse.com (NeilBrown) Date: Wed, 12 Jun 2019 14:08:49 +1000 Subject: [lustre-devel] [PATCH] Place a memory barrier around cp_state changes In-Reply-To: References: <20190606223110.15725-1-stancheff@cray.com> <87wohyw6ug.fsf@notabene.neil.brown.name> <87o935vtde.fsf@notabene.neil.brown.name> Message-ID: <87wohrv41a.fsf@notabene.neil.brown.name> On Mon, Jun 10 2019, Shaun Tancheff wrote: > On Mon, Jun 10, 2019 at 7:49 PM NeilBrown wrote: > >>> >> So I can just about see the possibility of the race that you think you >> have hit. But it couldn't happen on x86, as it has strong memory >> ordering. >> >> What CPU architecture did this ASSERT trigger on? >> > > Agree with everything above, unfortunately was aarch64. Ahh. OK, I'm now reasonably convinced that the patch probably fixes the bug. It is a shame that there is no standard barrier between the wake_up and the wait_event(), but there isn't and we need to work with that. But I still don't like the patch.... :-( I have a few problems with it. Firstly, I generally don't like accessor functions. They tend to hide useful information. Rather than cl_page_state_get() I would rather have smp_load_acquire(&pg->cp_state) where ever it is needed, with a comment explaining why it is needed in that particular context. Some people seem to like accessor functions - there certainly are a lot of them in the kernel - but I don't think they help. Having an accessor function that just adds a barrier is, I think, particularly ugly as the comment can only be general in nature, and so doesn't really help the reader to understand why the barrier is needed. But even that isn't completely without precedent. "key_read_state()" in include/linux/key.h simply wraps smp_load_acquire(), and the comment there doesn't really tell me anything useful. "inet_sk_state_load" is slightly better it says it is for use in places where the socket lock isn't held - and names some of those. My second problem is that has "get" in the accessor function name is potentially misleading and "get" normally has a matching "put", and usually increments a refcount. Having "read" or "load" in place of "get" in the accessor function name would remove my second objection. My third objection is that the bug doesn't exist in the upstream client code. I'm not sure whether this is luck or good management, but either way it is gone. cl_sync_io_wait() has been changed to always take csi_waitq.lock. This avoids the need for anchor->csi_barrier, and provides a more robust way to ensure that cl_sync_io_note() is completely finished before cl_sync_io_wait() completes. That isn't quite the outcome I was expecting... I'm sorry if I seem to have been rather harsh on your patch - my intention was only to make sure it was the best it could be. It really needed to have the stack trace and the arch were the LBUG happened to be at all complete. I would have been nice if it identified the particular code where the barrier was needed: between cl_sync_io_note() and cl_sync_io_wait() because that mediates and ownership transition that happens while the VM page is locked. The rest are probably just issues of taste. Thanks, NeilBrown -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 832 bytes Desc: not available URL: From shaun at tancheff.com Wed Jun 12 04:36:13 2019 From: shaun at tancheff.com (Shaun Tancheff) Date: Tue, 11 Jun 2019 23:36:13 -0500 Subject: [lustre-devel] [PATCH] Place a memory barrier around cp_state changes In-Reply-To: <87wohrv41a.fsf@notabene.neil.brown.name> References: <20190606223110.15725-1-stancheff@cray.com> <87wohyw6ug.fsf@notabene.neil.brown.name> <87o935vtde.fsf@notabene.neil.brown.name> <87wohrv41a.fsf@notabene.neil.brown.name> Message-ID: On Tue, Jun 11, 2019 at 11:09 PM NeilBrown wrote: > On Mon, Jun 10 2019, Shaun Tancheff wrote: > > > On Mon, Jun 10, 2019 at 7:49 PM NeilBrown wrote: > > > >>> > >> So I can just about see the possibility of the race that you think you > >> have hit. But it couldn't happen on x86, as it has strong memory > >> ordering. > >> > >> What CPU architecture did this ASSERT trigger on? > >> > > > > Agree with everything above, unfortunately was aarch64. > > Ahh. > OK, I'm now reasonably convinced that the patch probably fixes the bug. > It is a shame that there is no standard barrier between the wake_up and > the wait_event(), but there isn't and we need to work with that. > > But I still don't like the patch.... :-( > > I have a few problems with it. > > Firstly, I generally don't like accessor functions. They tend to hide > useful information. > Rather than cl_page_state_get() I would rather have > smp_load_acquire(&pg->cp_state) > > where ever it is needed, with a comment explaining why it is needed in > that particular context. > Some people seem to like accessor functions - there certainly are a lot > of them in the kernel - but I don't think they help. > Having an accessor function that just adds a barrier is, I think, > particularly ugly as the comment can only be general in nature, and so > doesn't really help the reader to understand why the barrier is needed. > But even that isn't completely without precedent. > "key_read_state()" in include/linux/key.h simply wraps > smp_load_acquire(), and the comment there doesn't really tell me > anything useful. > "inet_sk_state_load" is slightly better it says it is for use in places > where the socket lock isn't held - and names some of those. > Understood and a fair assessment. > My second problem is that has "get" in the accessor function name is > potentially misleading and "get" normally has a matching "put", and > usually increments a refcount. Having "read" or "load" in place of > "get" in the accessor function name would remove my second objection. > Yeah, I was trying to balance with the set .. but really I'm not enamored with that either :P > My third objection is that the bug doesn't exist in the upstream client > code. I'm not sure whether this is luck or good management, but either > way it is gone. > This is really good to know. Personally I was itching to refactor the whole lot but this gives a me a much better path. > cl_sync_io_wait() has been changed to always take csi_waitq.lock. This > avoids the need for anchor->csi_barrier, and provides a more robust way > to ensure that cl_sync_io_note() is completely finished before > cl_sync_io_wait() completes. > > That isn't quite the outcome I was expecting... > > I'm sorry if I seem to have been rather harsh on your patch - my > intention was only to make sure it was the best it could be. > > It really needed to have the stack trace and the arch were the LBUG > happened to be at all complete. > I would have been nice if it identified the particular code where the > barrier was needed: between cl_sync_io_note() and cl_sync_io_wait() > because that mediates and ownership transition that happens while the VM > page is locked. > > The rest are probably just issues of taste. > > Thanks, > NeilBrown > Thanks I really appreciate the time need to consider the issue. And I agree any patch needs to be as good and tasteful as we can make them. Regards, Shaun -------------- next part -------------- An HTML attachment was scrubbed... URL: From pfarrell at whamcloud.com Wed Jun 12 13:04:03 2019 From: pfarrell at whamcloud.com (Patrick Farrell) Date: Wed, 12 Jun 2019 13:04:03 +0000 Subject: [lustre-devel] [PATCH] Place a memory barrier around cp_state changes In-Reply-To: References: <20190606223110.15725-1-stancheff@cray.com> <87wohyw6ug.fsf@notabene.neil.brown.name> <87o935vtde.fsf@notabene.neil.brown.name> <87wohrv41a.fsf@notabene.neil.brown.name>, Message-ID: As an interested observer (thanks to both of you for an interesting exchange), Shaun, what’s your plan going forward for the OpenSFS/WC branch? It sounds like you’re thinking you’ll try to emulate what Neil did upstream? (which sounds good to me, I always prefer avoiding explicit memory barriers if reasonable) Will you be opening an LU for this one? Thanks, - Patrick ________________________________ From: lustre-devel on behalf of Shaun Tancheff Sent: Tuesday, June 11, 2019 11:36:13 PM To: NeilBrown Cc: Bobi Jam; lustre-devel at lists.lustre.org Subject: Re: [lustre-devel] [PATCH] Place a memory barrier around cp_state changes On Tue, Jun 11, 2019 at 11:09 PM NeilBrown > wrote: On Mon, Jun 10 2019, Shaun Tancheff wrote: > On Mon, Jun 10, 2019 at 7:49 PM NeilBrown > wrote: > >>> >> So I can just about see the possibility of the race that you think you >> have hit. But it couldn't happen on x86, as it has strong memory >> ordering. >> >> What CPU architecture did this ASSERT trigger on? >> > > Agree with everything above, unfortunately was aarch64. Ahh. OK, I'm now reasonably convinced that the patch probably fixes the bug. It is a shame that there is no standard barrier between the wake_up and the wait_event(), but there isn't and we need to work with that. But I still don't like the patch.... :-( I have a few problems with it. Firstly, I generally don't like accessor functions. They tend to hide useful information. Rather than cl_page_state_get() I would rather have smp_load_acquire(&pg->cp_state) where ever it is needed, with a comment explaining why it is needed in that particular context. Some people seem to like accessor functions - there certainly are a lot of them in the kernel - but I don't think they help. Having an accessor function that just adds a barrier is, I think, particularly ugly as the comment can only be general in nature, and so doesn't really help the reader to understand why the barrier is needed. But even that isn't completely without precedent. "key_read_state()" in include/linux/key.h simply wraps smp_load_acquire(), and the comment there doesn't really tell me anything useful. "inet_sk_state_load" is slightly better it says it is for use in places where the socket lock isn't held - and names some of those. Understood and a fair assessment. My second problem is that has "get" in the accessor function name is potentially misleading and "get" normally has a matching "put", and usually increments a refcount. Having "read" or "load" in place of "get" in the accessor function name would remove my second objection. Yeah, I was trying to balance with the set .. but really I'm not enamored with that either :P My third objection is that the bug doesn't exist in the upstream client code. I'm not sure whether this is luck or good management, but either way it is gone. This is really good to know. Personally I was itching to refactor the whole lot but this gives a me a much better path. cl_sync_io_wait() has been changed to always take csi_waitq.lock. This avoids the need for anchor->csi_barrier, and provides a more robust way to ensure that cl_sync_io_note() is completely finished before cl_sync_io_wait() completes. That isn't quite the outcome I was expecting... I'm sorry if I seem to have been rather harsh on your patch - my intention was only to make sure it was the best it could be. It really needed to have the stack trace and the arch were the LBUG happened to be at all complete. I would have been nice if it identified the particular code where the barrier was needed: between cl_sync_io_note() and cl_sync_io_wait() because that mediates and ownership transition that happens while the VM page is locked. The rest are probably just issues of taste. Thanks, NeilBrown Thanks I really appreciate the time need to consider the issue. And I agree any patch needs to be as good and tasteful as we can make them. Regards, Shaun -------------- next part -------------- An HTML attachment was scrubbed... URL: From shaun at tancheff.com Wed Jun 12 16:12:36 2019 From: shaun at tancheff.com (Shaun Tancheff) Date: Wed, 12 Jun 2019 11:12:36 -0500 Subject: [lustre-devel] [PATCH] Place a memory barrier around cp_state changes In-Reply-To: References: <20190606223110.15725-1-stancheff@cray.com> <87wohyw6ug.fsf@notabene.neil.brown.name> <87o935vtde.fsf@notabene.neil.brown.name> <87wohrv41a.fsf@notabene.neil.brown.name> Message-ID: On Wed, Jun 12, 2019 at 8:04 AM Patrick Farrell wrote: > As an interested observer (thanks to both of you for an interesting > exchange), Shaun, what’s your plan going forward for the OpenSFS/WC > branch? It sounds like you’re thinking you’ll try to emulate what Neil did > upstream? (which sounds good to me, I always prefer avoiding explicit > memory barriers if reasonable) Will you be opening an LU for this one? > Yes, I will be opening an LU for this. My plan is to emulate upstream. > Thanks, > - Patrick > ------------------------------ > *From:* lustre-devel on behalf of > Shaun Tancheff > *Sent:* Tuesday, June 11, 2019 11:36:13 PM > *To:* NeilBrown > *Cc:* Bobi Jam; lustre-devel at lists.lustre.org > *Subject:* Re: [lustre-devel] [PATCH] Place a memory barrier around > cp_state changes > > On Tue, Jun 11, 2019 at 11:09 PM NeilBrown wrote: > > On Mon, Jun 10 2019, Shaun Tancheff wrote: > > > On Mon, Jun 10, 2019 at 7:49 PM NeilBrown wrote: > > > >>> > >> So I can just about see the possibility of the race that you think you > >> have hit. But it couldn't happen on x86, as it has strong memory > >> ordering. > >> > >> What CPU architecture did this ASSERT trigger on? > >> > > > > Agree with everything above, unfortunately was aarch64. > > Ahh. > OK, I'm now reasonably convinced that the patch probably fixes the bug. > It is a shame that there is no standard barrier between the wake_up and > the wait_event(), but there isn't and we need to work with that. > > But I still don't like the patch.... :-( > > I have a few problems with it. > > Firstly, I generally don't like accessor functions. They tend to hide > useful information. > Rather than cl_page_state_get() I would rather have > smp_load_acquire(&pg->cp_state) > > where ever it is needed, with a comment explaining why it is needed in > that particular context. > Some people seem to like accessor functions - there certainly are a lot > of them in the kernel - but I don't think they help. > Having an accessor function that just adds a barrier is, I think, > particularly ugly as the comment can only be general in nature, and so > doesn't really help the reader to understand why the barrier is needed. > But even that isn't completely without precedent. > "key_read_state()" in include/linux/key.h simply wraps > smp_load_acquire(), and the comment there doesn't really tell me > anything useful. > "inet_sk_state_load" is slightly better it says it is for use in places > where the socket lock isn't held - and names some of those. > > > Understood and a fair assessment. > > > My second problem is that has "get" in the accessor function name is > potentially misleading and "get" normally has a matching "put", and > usually increments a refcount. Having "read" or "load" in place of > "get" in the accessor function name would remove my second objection. > > > Yeah, I was trying to balance with the set .. but really I'm not enamored > with that either :P > > > My third objection is that the bug doesn't exist in the upstream client > code. I'm not sure whether this is luck or good management, but either > way it is gone. > > > This is really good to know. Personally I was itching to refactor the whole > lot but this gives a me a much better path. > > > cl_sync_io_wait() has been changed to always take csi_waitq.lock. This > avoids the need for anchor->csi_barrier, and provides a more robust way > to ensure that cl_sync_io_note() is completely finished before > cl_sync_io_wait() completes. > > That isn't quite the outcome I was expecting... > > I'm sorry if I seem to have been rather harsh on your patch - my > intention was only to make sure it was the best it could be. > > It really needed to have the stack trace and the arch were the LBUG > happened to be at all complete. > I would have been nice if it identified the particular code where the > barrier was needed: between cl_sync_io_note() and cl_sync_io_wait() > because that mediates and ownership transition that happens while the VM > page is locked. > > The rest are probably just issues of taste. > > Thanks, > NeilBrown > > > Thanks I really appreciate the time need to consider the issue. > And I agree any patch needs to be as good and tasteful as we can make them. > > Regards, > Shaun > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From anna.fuchs at informatik.uni-hamburg.de Thu Jun 13 11:54:47 2019 From: anna.fuchs at informatik.uni-hamburg.de (Anna Fuchs) Date: Thu, 13 Jun 2019 13:54:47 +0200 Subject: [lustre-devel] Request arc buffer, zerocopy Message-ID: <1560426887.3392.0@informatik.uni-hamburg.de> Dear all, in osd-zfs/osd_io.c:osd_bufs_get_write you can find a comment regarding zerocopy: /* * currently only full blocks are subject to zerocopy approach: * so that we're sure nobody is trying to update the same block */ Whenever a block to be written is full, an arc buffer is requested, otherwise alloc_page. I do not really understand the conclusion. Why and how do full blocks prevent updates? And put it differently - why not to try zerocopy for not full blocks? What could happen if I tried to request an arc buffer for e.g. a block with missing last page? I would be greateful for details. Best regards Anna -------------- next part -------------- An HTML attachment was scrubbed... URL: From adilger at whamcloud.com Thu Jun 13 17:26:58 2019 From: adilger at whamcloud.com (Andreas Dilger) Date: Thu, 13 Jun 2019 17:26:58 +0000 Subject: [lustre-devel] Request arc buffer, zerocopy In-Reply-To: <1560426887.3392.0@informatik.uni-hamburg.de> References: <1560426887.3392.0@informatik.uni-hamburg.de> Message-ID: <4A887B54-96F2-439C-9207-1BBA01350869@whamcloud.com> Add relevant developers to CC list. Cheers, Andreas > On Jun 13, 2019, at 05:54, Anna Fuchs wrote: > > Dear all, > > in osd-zfs/osd_io.c:osd_bufs_get_write you can find a comment regarding zerocopy: > > /* > * currently only full blocks are subject to zerocopy approach: > * so that we're sure nobody is trying to update the same block > */ > > Whenever a block to be written is full, an arc buffer is requested, otherwise alloc_page. > > I do not really understand the conclusion. Why and how do full blocks prevent updates? > And put it differently - why not to try zerocopy for not full blocks? > What could happen if I tried to request an arc buffer for e.g. a block with missin g last page? > > I would be greateful for details. > Best regards > Anna > > > > > > _______________________________________________ > lustre-devel mailing list > lustre-devel at lists.lustre.org > http://lists.lustre.org/listinfo.cgi/lustre-devel-lustre.org From green at whamcloud.com Sun Jun 16 03:59:37 2019 From: green at whamcloud.com (Oleg Drokin) Date: Sun, 16 Jun 2019 03:59:37 +0000 Subject: [lustre-devel] New tag 2.12.55 Message-ID: <6E3FF7D2-78C5-4AC9-A91E-172E9CA92F34@whamcloud.com> Hello! I just tagged 2.12.55 tag in Lustre community master branch. This mostly brings remaining 2.13 cycle features like PCC and LNet multirail. Here’s the changelog: Alexander Boyko (1): LU-12219 obdfilter: changes PAGE_SIZE variable Amir Shehata (42): LU-12080 lnet: recovery event handling broken LU-12080 lnet: clean mt_eqh properly LU-12344 lnet: handle remote health error LU-11816 lnet: setup health timeout defaults LU-12163 lnet: fix cpt locking LU-12201 lnet: detach response tracker LU-11297 lnet: invalidate recovery ping mdh LU-12249 lnet: fix list corruption LU-12254 lnet: correct discovery LNetEQFree() LU-12199 lnet: verify msg is commited for send/recv LU-12339 lnet: select LO interface for sending LU-10153 lnet: remove route add restriction LU-11292 lnet: Discover routers on first use LU-11298 lnet: use peer for gateway LU-11299 lnet: lnet_add/del_route() LU-11551 lnet: Do not allow deleting of router nis LU-11300 lnet: router sensitivity LU-11300 lnet: configure lnet router senstivity LU-11300 lnet: cache ni status LU-11300 lnet: Cache the routing feature LU-11300 lnet: peer aliveness LU-11300 lnet: router aliveness LU-11300 lnet: simplify lnet_handle_local_failure() LU-11299 lnet: Cleanup rcd LU-11299 lnet: modify lnd notification mechanism LU-11299 lnet: use discovery for routing LU-11378 lnet: MR aware gateway selection LU-11300 lnet: consider alive_router_check_interval LU-11475 lnet: allow deleting router primary_nid LU-11475 lnet: transfer routers LU-11477 lnet: handle health for incoming messages LU-11478 lnet: misleading discovery seqno. LU-11470 lnet: drop all rule LU-11641 lnet: handle discovery off LU-11297 lnet: handle router health off LU-11297 lnet: set gw sensitivity from lnetctl LU-11664 lnet: push router interface updates LU-11299 lnet: net aliveness LU-11299 lnet: discover each gateway Net LU-12053 lnet: look up MR peers routes LU-12200 lnet: check peer timeout on a router LU-11297 lnet: MR Routing Feature Andreas Dilger (1): LU-1538 tests: standardize test script init - sanity Arshad Hussain (1): LU-6142 ptlrpc: Fix style issues for pinger.c Chris Horn (5): LU-12264 lnet: Protect lp_dc_pendq manipulation with lp_lock LU-12199 lnet: Ensure md is detached when msg is not committed LU-12387 utils: Read existing ldd data in l_tunedisk LU-12387 utils: Avoid passing symlink to tune_block_dev LU-12387 tests: Validate l_tunedisk max_sectors_kb tuning James Nunez (1): LU-12267 tests: filter trailing '.' for SELinux James Simmons (1): LU-12381 ko2iblnd: ignore down interfaces Lai Siyao (5): LU-11213 ptlrpc: intent_getattr fetches default LMV LU-11213 mdc: add async statfs LU-11213 lmv: mkdir with balanced space usage LU-9846 utils: hash may be overridden in 'lfs setdirstripe' LU-11213 lmv: reuse object alloc QoS code from LOD Li Dongyang (3): LU-11838 ldiskfs: add rhel8 server support LU-11838 osd-ldiskfs: inode times switched to timespec64 LU-12395 build: require python2 for lustre-iokit Li Xi (1): LU-10092 llite: Add persistent cache on client Mikhail Pershin (2): LU-11204 obdclass: remove unprotected access to lu_object LU-12014 llite: check correct size in ll_dom_finish_open() Nathaniel Clark (1): LU-12375 scripts: Start lnet after opa Oleg Drokin (4): LU-10092 First phase of persistent client cache project merging LU-11623 tests: Fix sanity 27E to ensure getattr RPC LU-10948 mdt: Remove openlock compat code with 2.1 New tag 2.12.55 Patrick Farrell (1): LU-12396 utils: lfs should not output 'nul' char Qian Yingjin (5): LU-10092 pcc: Non-blocking PCC caching LU-10092 pcc: security and permission for non-root user access LU-10918 llite: Rule based auto PCC caching when create files LU-10092 pcc: auto attach during open for valid cache LU-10092 pcc: change detach behavior and add keep option Sebastien Buisson (1): LU-12399 tests: avoid 'pdsh localhost' in sanity test_420 Sergey Cheremencev (2): LU-11758 osp: remove assertion from statfs LU-12412 recovery: wake all waiters of trd_finishing Shaun Tancheff (2): LU-12438 llite: vfs_read/write removed, use kernel_read/write LU-12355 lnet: Adjust checks for ib_device_ops Vitaly Fertman (1): LU-9846 test: a test number fix Wang Shilong (2): LU-12374 lustre: push rcu_barrier() before destroying slab LU-12383 utils: only check project inherit bit for dir From green at whamcloud.com Mon Jun 17 16:30:24 2019 From: green at whamcloud.com (Oleg Drokin) Date: Mon, 17 Jun 2019 16:30:24 +0000 Subject: [lustre-devel] Additional testing on reviews Message-ID: <9B27101C-F0E6-4E4D-AE7E-DDCD049D28F2@whamcloud.com> Hello! As we all know it’s the tests and configurations that are not executed that break the most. In order to prevent everything to be tailored to a ingle maloo “monoculture” in our testing, I’ve been running a separate test rig(s) for integration testing with full-debug kernel and such that was helping to find issues not found in maloo for one reason or another. The setup is also tailored to mimic how a developer would setup their Lustre testing with out of tree test execution (as opposed to the lengthly rpm building process). This was a pretty useful exercise, but somewhat expensive in terms of found bugs (in the sense that bugs should preferably be found earlier, not later in the development cycle, since the later it’s found the more steps need to be redone including various reviews and such). Now similarly with what I’ve done with static code analysis, I am doing for at least some bits of testing too: You will see comments from “Lustre Gerrit Janitor” on your (master-only for now) patches denoting build and testing progress. Importantly - this is not a replacement of maloo/autotest testing, this is just additional testing aimed at providing more coverage where it was currently lacking: - All builds are only for centos 7 (I plan to switch to rhel8.0 as soon as it’s stable enough in my setup) and only x86_64 (I might add ARM but it’s not a priority ATM). - Everything runs out of a build tree. You’ll get an early warning if your new test is not working in that configuration and I expect you to fix it or talk to me about possible needed changes to make it work on the infrastructure side. People love to hate on this configuration, but I strongly believe this is a useful setup so I will insist on this working going forward so people keep running tests at least somewhat during their development (which everybody already does anyway, right?) - test cluster configuration is a somewhat limited: one client node and one server node where on the server node we have MGS (combined into MDT1), one or two MDTs (2.5G) and two OSTs(4G) on tcp-only networking. * But at least I have a whole bunch of them so hopefully things would move fast. - Only a single test script is run per session - No hard failover testing (tests that do it unconditionally are frowned upon, ahem, recovery small-test 136) - It runs every sensible working test * notable exception is sanity-gss that crashes a lot and I was told is unsupported. * it tries to divine by what you changed for what tests don’t need to be run and what tests need to be run as a priority because you changed the test file itself. * all tests run in parallel (subject to cluster availability) - Tests runs are split into two sections: initial and comprehensive testing: * Normally initial is just basic runtests with ldiskfs+DNE, zfs and SSK configurations, but the list would also include every testscript you happened to touch and everything you requested in the Test-parameters string. * If initial testing did not fail for some reason, then the rest of tests would be run in the comprehensive group. Since I want this to be developer friendly (which also helps me) there’s additional information you get out of those test runs: - Idea is to get a quick turnaround. build ready in under 10 minutes, initial testing complete in under 15 minutes, full test session is done in a little more over 2 hours (not currently possible because some tests like conf-sanity take longer. Work is underway to split them into several) - You always get all syslogs and all console logs no matter if test failed or succeeded - For crashed tests you’ll see if this if a known triaged failure because an LU- ticket number would be listed, otherwise you’ll see something like "(Untriaged #824, seen 2 times before)” This tells you how often did we see this crash from multiple sources including autotest. You can see all the other hits if you paste the untriaged number into this url: https://knox.linuxhacker.ru/crashdb_ui_external.py.cgi?newid= after the ‘=‘ sign. while it looks like a full featured triaging ui, it would not let you triage anything so if you want to make some known - email me for now. * Additionally for crashed tests if you go into the test results link for it, you will see there: vmcore and debug vmlinux + lustre modules - in case you want to dig in yourself. lustre debug log extracted from the vmcore. backtrace processed by the crash tool with source line numbers attached (additionally if the backtrace comes from anywhere in your patch and the crash is new enough, it will post an early comment at that location in gerrit to save you a click). - For timeouts - a crashdump is taken from both server and client nodes and same information is extracted as above in case you wanted to check for something yourself (hopefully helps with debugging those pesky timeouts) - For a regular test failure you’ll get a status if the failure was seen on the baseline branch recently or if it’s believed to be a new one. It’s not 100% perfect, but it’s good enough, so obviously do pay attention to the “new” failures (conf-sanity 69 has a variable error message, consider not doing that in the future) - Additionally logs are checked for other conditions and warning (shown as new or old depending on if it was seen before on a baseline branch or not) would be printed. Currently checked for (new warnings should be always looked into ASAP obviously): * memory leaks (currently in conf-sanity zfs test 59, LU-12038 ; rarely in sanity zfs) * sleeping in atomic context (currently - in runtests in ssk mode LU-12338, sanityn thanks to CRR code and sanity-quota LU-12193) * busy inodes * linked list manipulation warnings from kernel * kobject warning messages * busy inodes after umount * If you have other ideas for message clearly showing up in kernel logs I can check for - let me know and I’ll add. * libc use after free messages in userspace from Lustre tools Currently all of this is not fed into maloo DB, but I am thinking if I perhaps should do it, so meanwhile there’s no search and you just get a link to your testrun pretty much. You can also see current testing queue, some stats and last 100 completed test runs at http://testing.linuxhacker.ru:3333/lustre-reports/status.html (useful to find last baseline run in particular if you wanted to compare to) For those curious how results currently looks like, here’s a sample you can take a look at: http://testing.linuxhacker.ru:3333/lustre-reports/366/results.html While I tested this a bit at smaller scale, I am sure a bunch of stuff will break as it’s unveiled at the full scale, so do let me know if you see anything strange and otherwise bear with me. Also feel free to ignore the test results, like I said, these are not binding and would not set +1 or -1 all by itself, I am just trying to propagate errors to you as early as possible so you can act faster based on this information. If you have any other cool ideas about how this whole thing could be improved - do let me know as well. Bye, Oleg From adilger at whamcloud.com Mon Jun 24 06:05:05 2019 From: adilger at whamcloud.com (Andreas Dilger) Date: Mon, 24 Jun 2019 06:05:05 +0000 Subject: [lustre-devel] Dev day slides In-Reply-To: <6B3ABFDD-4BB1-4D21-B63B-A8D208E3EBF6@cray.com> References: <6B3ABFDD-4BB1-4D21-B63B-A8D208E3EBF6@cray.com> Message-ID: <46386440-9E68-420F-913E-F79917BDAD64@whamcloud.com> On May 31, 2019, at 16:32, Cory Spitz wrote: > > Hello. > > It was nice to see many of you at this year’s LUG Developer Day. I’ve created a page at http://wiki.lustre.org/Developer_Day_2019-05-14 to hold the info. If you presented, please add your content. > > Thank you very much! Hi Cory, thanks for creating that page. I uploaded the deck from myself and Sebastien, and created "empty" links for all of the other presentations. That will just require each presenter to click on the respective link to upload their presentation. Cheers, Andreas -- Andreas Dilger Principal Lustre Architect Whamcloud From anna.fuchs at informatik.uni-hamburg.de Wed Jun 26 13:11:48 2019 From: anna.fuchs at informatik.uni-hamburg.de (Anna Fuchs) Date: Wed, 26 Jun 2019 15:11:48 +0200 Subject: [lustre-devel] Request arc buffer, zerocopy In-Reply-To: <1560426887.3392.0@informatik.uni-hamburg.de> References: <1560426887.3392.0@informatik.uni-hamburg.de> Message-ID: <1561554708.16396.0@informatik.uni-hamburg.de> Dear all, one more question related to ZFS-buffers in Lustre. There is a function osd_grow_blocksize(obj, oh, ...) called after the fist portion of data (first rnb?) has been committed to ZFS. There are some restrictions for block size changing: dmu_object_set_blocksize says: The object cannot have any blocks allcated beyond the first. If * the first block is allocated already, the new size must be greater * than the current block size. and later on /* * Try to change the block size for the indicated dnode. This can only * succeed if there are no blocks allocated or dirty beyond first block */ I am now interested on the frist block's size, which seems to be set when creating the dnode. This size comes from ZFS and is something like dnp->dn_datablkszsec << SPA_MINBLOCKSHIFT or SPA_MINBLOCKSIZE (not sure). I would like to specify this size on Lustre's side, not just take what ZFS offers. E.g. make the first block 128K instead of 4K. Is it possible? Could I just overwrite the block size before the corresponsing memory for the block is allocated? I am not able to call osd_grow_blocksize for the first block, since I do not have any thread context there, not yet. Do I need to grab into dnode_allocate and dnode_create? And for better understanding, does one dnode always represent one lustre object? I would be greatful for any suggestions. *** Some context for my questions: I have compressed data chunks coming from the Lustre client. I want to hand them over to ZFS like they were compressed by ZFS. ZFS offers some structures, e.g. compressed arc-buffers, which know how the data has been compressed (which algo, physical and logical sizes). I want and need my chunks to be aligned to the records (arc buffers). We have already extended the interfaces of the internal ZFS compression structures. But currently ZFS (or osd-zfs) first defines the sizes of buffers and the data is put in there. In my case, the data should "dictate" how many buffers there are and how large they can be. Best regards Anna -- Anna Fuchs Universität Hamburg On Thu, Jun 13, 2019 at 1:54 PM, Anna Fuchs wrote: > Dear all, > > in osd-zfs/osd_io.c:osd_bufs_get_write you can find a comment > regarding zerocopy: > > /* > * currently only full blocks are subject to zerocopy approach: > * so that we're sure nobody is trying to update the same block > */ > > Whenever a block to be written is full, an arc buffer is requested, > otherwise alloc_page. > > I do not really understand the conclusion. Why and how do full blocks > prevent updates? > And put it differently - why not to try zerocopy for not full blocks? > What could happen if I tried to request an arc buffer for e.g. a > block with missin g last page? > > I would be greateful for details. > Best regards > Anna > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mahrens at delphix.com Thu Jun 27 18:13:20 2019 From: mahrens at delphix.com (Matthew Ahrens) Date: Thu, 27 Jun 2019 11:13:20 -0700 Subject: [lustre-devel] Request arc buffer, zerocopy In-Reply-To: <1561554708.16396.0@informatik.uni-hamburg.de> References: <1560426887.3392.0@informatik.uni-hamburg.de> <1561554708.16396.0@informatik.uni-hamburg.de> Message-ID: On Wed, Jun 26, 2019 at 6:11 AM Anna Fuchs < anna.fuchs at informatik.uni-hamburg.de> wrote: > Dear all, > > one more question related to ZFS-buffers in Lustre. > > There is a function osd_grow_blocksize(obj, oh, ...) called after the fist > portion of data (first rnb?) > has been committed to ZFS. > There are some restrictions for block size changing: > dmu_object_set_blocksize says: > The object cannot have any blocks allcated beyond the first. If > * the first block is allocated already, the new size must be greater > * than the current block size. > and later on > /* > * Try to change the block size for the indicated dnode. This can only > * succeed if there are no blocks allocated or dirty beyond first block > */ > > I am now interested on the frist block's size, which seems to be set when > creating the dnode. > This size comes from ZFS and is something like > dnp->dn_datablkszsec << SPA_MINBLOCKSHIFT or SPA_MINBLOCKSIZE (not sure). > The block size in bytes is `dnp->dn_datablkszsec << SPA_MINBLOCKSHIFT`. FYI, SPA_MINBLOCKSIZE == 1< > I would like to specify this size on Lustre's side, not just take what ZFS > offers. > E.g. make the first block 128K instead of 4K. > You can set the block size (of the first and only block) using dmu_object_set_blocksize(). FYI, I think that this comment is incorrect: * If the first block is allocated already, the new size must be greater * than the current block size. You can increase or decrease the block size with this routine. Is it possible? Could I just overwrite the block size before the > corresponsing memory for the block is allocated? > > I am not able to call osd_grow_blocksize for the first block, since I do > not have any thread context there, not yet. > Do I need to grab into dnode_allocate and dnode_create? > > And for better understanding, does one dnode always represent one lustre > object? > > I would be greatful for any suggestions. > > *** > > Some context for my questions: > > I have compressed data chunks coming from the Lustre client. I want to > hand them over to ZFS like they > were compressed by ZFS. ZFS offers some structures, e.g. compressed > arc-buffers, which know how the data has been > compressed (which algo, physical and logical sizes). I want and need my > chunks to be aligned to the records (arc buffers). > We have already extended the interfaces of the internal ZFS compression > structures. But currently ZFS (or osd-zfs) first defines > the sizes of buffers and the data is put in there. In my case, the data > should "dictate" how many buffers there are and how large they can be. > > I'd recommend that you hand the compressed data to ZFS similarly to how "zfs receive" does (for compressed send streams). It sounds like the is the direction you're going, which is great. FYI, here are some of the routines you'd want to use (copied from dmu_recv.c): abuf = arc_loan_compressed_buf( dmu_objset_spa(drc->drc_os), drrw->drr_compressed_size, drrw->drr_logical_size, drrw->drr_compressiontype); dmu_assign_arcbuf(bonus, drrw->drr_offset, abuf, tx); (or dmu_assign_arcbuf_dnode()) dmu_return_arcbuf(rrd->write_buf); --matt Best regards > Anna > > -- > Anna Fuchs > Universität Hamburg > > On Thu, Jun 13, 2019 at 1:54 PM, Anna Fuchs < > anna.fuchs at informatik.uni-hamburg.de> wrote: > > Dear all, > > in osd-zfs/osd_io.c:osd_bufs_get_write you can find a comment regarding > zerocopy: > > /* > * currently only full blocks are subject to zerocopy approach: > * so that we're sure nobody is trying to update the same block > */ > > Whenever a block to be written is full, an arc buffer is requested, > otherwise alloc_page. > > I do not really understand the conclusion. Why and how do full blocks > prevent updates? > And put it differently - why not to try zerocopy for not full blocks? > What could happen if I tried to request an arc buffer for e.g. a block > with missin g last page? > > I would be greateful for details. > Best regards > Anna > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From anna.fuchs at informatik.uni-hamburg.de Fri Jun 28 09:50:12 2019 From: anna.fuchs at informatik.uni-hamburg.de (Anna Fuchs) Date: Fri, 28 Jun 2019 11:50:12 +0200 Subject: [lustre-devel] Request arc buffer, zerocopy In-Reply-To: References: <1560426887.3392.0@informatik.uni-hamburg.de> <1561554708.16396.0@informatik.uni-hamburg.de> Message-ID: <1561715412.12733.3@informatik.uni-hamburg.de> Hello Matt, thanks for your reply. > > You can set the block size (of the first and only block) using > dmu_object_set_blocksize(). FYI, I think that this comment is > incorrect: > * If the first block is allocated already, the new size must be > greater > * than the current block size. > > You can increase or decrease the block size with this routine. This is a deeper call of Lustre's osd_grow_blocksize I mentioned before. If I understand it correctly, they are called in the context of transactions, right? If so, I can not use it - I need the blocksize to be set in the buffer preparation stage, before comitting in a transaction. Lustre's original routine looks simplified as follows: osd_bufs_get_write bs = dn->dn_datablksz while (len > 0) if (sz_in_block == bs) /* full block, try zerocopy */ abuf = osd_request_arcbuf(dn, bs); else /* can't use zerocopy, allocate temp. buffers */ ... alloc_page ... here going later on the dmu_write path (pagewise!) Currently, in the very first iteration this blocksize (bs) is taken from the dnode and is e.g. 4K. When writing a chunk of 16K, I get 4 arcbufs 4K each. For the next chunk the block size might be grown up to x (recordsize?). Here I need the blocksize to be set to 16K (or 128K or later some generic value defined by the Lustre client) before the first arcbuf is requested, because the compressed chunk sent from client is logically this size. At this point I don't have any dmu_tx yet to grow the blocksize saved in dn->dn_datablksz before the while loop. So I am not sure how deep to go? This min size is set on dnode creation by ZFS, how can I "reset" it? > > I'd recommend that you hand the compressed data to ZFS similarly to > how "zfs receive" does (for compressed send streams). It sounds like > the is the direction you're going, which is great. FYI, here are > some of the routines you'd want to use (copied from dmu_recv.c): > > abuf = arc_loan_compressed_buf( > > dmu_objset_spa(drc->drc_os), > > drrw->drr_compressed_size, drrw->drr_logical_size, > > drrw->drr_compressiontype); > > > dmu_assign_arcbuf(bonus, drrw->drr_offset, abuf, tx); > > (or dmu_assign_arcbuf_dnode()) > > > dmu_return_arcbuf(rrd->write_buf); > Yes, thanks for that. We have two paths how Lustre interacts with ZFS - requesting arc buffers or dmu_write. The common dmu_request_arcbuf goes over arc_loan_buf, so we introduced dmu_request_compressed_arcbuf to go over arc_loan_compressed_buf to reuse the receive functionality. We try to make as few changes as possible on Lustre's interface since we want mix compressed and uncompressed data chunks (and be at the same time compatible with ZFS' on disk format..) The dmu_write path will be tricky, though. Any comments are welcome. Best regards Anna -- Anna Fuchs Universität Hamburg From charlie at whamcloud.com Fri Jun 28 16:43:45 2019 From: charlie at whamcloud.com (Charlie Olmstead) Date: Fri, 28 Jun 2019 16:43:45 +0000 Subject: [lustre-devel] sanity-pcc suite added Message-ID: Starting this weekend, the suite sanity-pcc will run in the following sessions: - review-dne-part-2 and review-dne-zfs-part-2 for all master review patches - full for master branch testing -------------- next part -------------- An HTML attachment was scrubbed... URL: