[lustre-discuss] GID only mapping in 2.8.60?
sthiell at stanford.edu
Mon Nov 7 14:59:43 PST 2016
> On Nov 4, 2016, at 11:05 PM, Dilger, Andreas <andreas.dilger at intel.com> wrote:
> Actually, the nodemap feature will work with any client, since it is only affecting lookups on the MDS and quota on the OSS.
> It probably would take less time for you to implement the flag feature than the time it is taking to create the thousands of UID entries. While I think it should scale very large, I don't think we have tested the 1M or so entries you are creating. The good news is that since this is using a hash table on the server it shouldn't hurt performance too much.
> Let us know how many you finally create, and how it is working with so many entries.
I was able to add ~820k idmaps in 48h [with all servers up and running, see my note below]… but I can’t go much beyond that point, as I am seeing less than 1 idmap being added per second. Even with such a low rate, I noticed a cpu-bound thread on all MDS and OSS:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
30258 root 20 0 0 0 0 R 100.0 0.0 1798:38 ll_cfg_requeue
I have two vmcore dumps that show backtraces for this running thread that can be found at the end of this email.
I don’t know why the nodemap_idx_insert is called so often, as the apparent idmap creation rate is very slow.
Last minute note: I was able to quickly add a more idmaps from the MGS much faster by disconnecting the lustre servers first. Once the idmaps were added to the MGS, I started all MDS and OSS servers again, and the idmap synchronization from the MGS was immediate. So for large uid ranges, I would recommend to add all the idmaps with only the MGS up and running. That can be very useful in case of writeconf until a uid range-based or identity mapping solution is available.
Also, to finish with a good news, I just completed a few small scale metadata benchmarks using files with mapped uig/gid, but I didn't notice any difference in terms of md performance like create or delete/sec rates when the nodemap feature is enable or disabled. Tested with 820k loaded idmaps.
All the best,
bt running ll_cfg_requeue #1:
--- <NMI exception stack> ---
#4 [ffff881f3e93fae8] memset at ffffffff813017d9
#5 [ffff881f3e93fae8] kmem_cache_alloc_trace at ffffffff811c117e
#6 [ffff881f3e93fb30] lu_context_init at ffffffffa0a72d06 [obdclass]
#7 [ffff881f3e93fb50] osd_trans_start at ffffffffa0de0151 [osd_ldiskfs]
#8 [ffff881f3e93fb88] nodemap_idx_insert at ffffffffa0f28af4 [ptlrpc]
#9 [ffff881f3e93fbd0] nodemap_save_config_cache at ffffffffa0f2c5e0 [ptlrpc]
#10 [ffff881f3e93fc78] nodemap_config_set_active_mgc at ffffffffa0f2c9ad [ptlrpc]
#11 [ffff881f3e93fce0] mgc_process_recover_nodemap_log at ffffffffa0a0ce6b [mgc]
#12 [ffff881f3e93fd70] mgc_process_log at ffffffffa0a0f894 [mgc]
#13 [ffff881f3e93fe30] mgc_requeue_thread at ffffffffa0a11908 [mgc]
#14 [ffff881f3e93fec8] kthread at ffffffff810a5aef
#15 [ffff881f3e93ff50] ret_from_fork at ffffffff81645a58
bt running ll_cfg_requeue #2:
--- <NMI exception stack> ---
#4 [ffff883f5f0239a8] iam_it_init at ffffffffa1168563 [osd_ldiskfs]
#5 [ffff883f5f0239b0] iam_insert at ffffffffa116a2f3 [osd_ldiskfs]
#6 [ffff883f5f023b20] osd_index_iam_insert at ffffffffa1157027 [osd_ldiskfs]
#7 [ffff883f5f023b88] nodemap_idx_insert at ffffffffa0d41c0c [ptlrpc]
#8 [ffff883f5f023bd0] nodemap_save_config_cache at ffffffffa0d455e0 [ptlrpc]
#9 [ffff883f5f023c78] nodemap_config_set_active_mgc at ffffffffa0d459ad [ptlrpc]
#10 [ffff883f5f023ce0] mgc_process_recover_nodemap_log at ffffffffa0a30e6b [mgc]
#11 [ffff883f5f023d70] mgc_process_log at ffffffffa0a33894 [mgc]
#12 [ffff883f5f023e30] mgc_requeue_thread at ffffffffa0a35908 [mgc]
#13 [ffff883f5f023ec8] kthread at ffffffff810a5aef
#14 [ffff883f5f023f50] ret_from_fork at ffffffff81645a58
More information about the lustre-discuss