<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">
Hi Anna,
<div class=""><br class="">
</div>
<div class="">Please see inserted lines.</div>
<div class=""><br class="">
<div>
<blockquote type="cite" class="">
<div class="">On Jan 12, 2017, at 4:15 AM, Anna Fuchs <<a href="mailto:anna.fuchs@informatik.uni-hamburg.de" class="">anna.fuchs@informatik.uni-hamburg.de</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div class="">Hello all,<br class="">
<br class="">
thank you for the responses. <br class="">
<br class="">
<br class="">
Jinshan,<br class="">
<br class="">
<blockquote type="cite" class=""><br class="">
I assume the purpose of this proposal is to fully utilize the CPU<br class="">
cycles on the client nodes to compress and decompress data, because<br class="">
there are much more client nodes than server nodes. After data is<br class="">
compressed, it will need less network bandwidth to transfer it to<br class="">
server and write them back to storage.<br class="">
</blockquote>
<br class="">
Yes, that is our goal for the moment.<br class="">
</div>
</div>
</blockquote>
<div><br class="">
</div>
cool. This should be a good approach.</div>
<div><br class="">
<blockquote type="cite" class="">
<div class="">
<div class=""><br class="">
<blockquote type="cite" class=""><br class="">
There would be more changes to implement this feature:<br class="">
1. I guess dmu_read() needs change as well to transfer compressed<br class="">
data back to client, otherwise how it would improve readahead<br class="">
performance. Please let me know if I overlooked something;<br class="">
</blockquote>
<br class="">
Sure, I might have shortened my explanation too much. The read path<br class="">
will be affected for providing compressed data and record-wise<br class="">
"metadata" back to the client. The client will then decompress it.<br class="">
<br class="">
<blockquote type="cite" class="">2. read-modify-write on client chunks - if only partial chunk is<br class="">
modified on the client side, the OSC will have to read the chunk<br class="">
back, uncompress it, and modify the data in chunk, and compress it<br class="">
again to get ready for write back. We may have to maintain a separate<br class="">
chunk cache on the OSC layer;<br class="">
</blockquote>
<br class="">
We keep the rmw problem in mind and will definitely need to work on<br class="">
optimization once the basic functionality is done. When compressing<br class="">
only sub-stripes (record size), we already hope to reduce the<br class="">
performance loss since we do not need to transfer and decompress the<br class="">
whole stripe anymore. <br class="">
We would want to keep the compressed data within bd_enc_vec and<br class="">
uncompressed in the normal vector. The space for that vector is<br class="">
allocated in sptlrpc_enc_pool_get_pages. Are those not cached? Could<br class="">
you give me some hints for the approach and what to look at? Is it a<br class="">
right place at all?<br class="">
</div>
</div>
</blockquote>
<div><br class="">
</div>
<div>I don’t think sptlrpc page pool will cache any data. However, it’s the right place to do compress in the sptlrpc layer, you just extend the sptlrpc for a new flavor.</div>
<div><br class="">
</div>
<div>With that being said, we’re going to have two options to support partial block write:</div>
<div><br class="">
</div>
<div>1. In the OSC I/O engine, it only submits ZFS block size aligned plain data to ptlrpc layer and it does compress in the new flavor of sptlrpc. When partial blocks are written, the OSC will have to issue read RPC if the corresponding data belonging to the
same block are not cached;</div>
<div><br class="">
</div>
<div>2. Or we can just disable this optimization that means plain data will be issued to the sever for partial block written. It only do compress for full blocks.</div>
<div><br class="">
</div>
<div>I feel the option 2 would be much simpler but needs some requirements to the workload to take full advantage, e.g. if applications are writing bulk and sequential data.</div>
<br class="">
<blockquote type="cite" class="">
<div class="">
<div class="">Though, the naive prototype I am currently working on is very memory<br class="">
intensive anyway (additional buffers, many copies). There is much work<br class="">
for me until I can dive into optimizations...<br class="">
<br class="">
<br class="">
<blockquote type="cite" class="">3. the OST should grant LDLM lock to align with ZFS block size<br class="">
otherwise it will be very complex if the OSC has to request locks to<br class="">
do RMW;<br class="">
</blockquote>
<br class="">
I am not very familiar with the locking in Lustre yet. <br class="">
You mean, once we want to modify part of the data on OST, we want to<br class="">
have a lock for the complete chunk (record), right? Currently, Lustre<br class="">
can do byte-range locks, instead we wanted record-ranged in this case?<br class="">
</div>
</div>
</blockquote>
<div><br class="">
</div>
<div>Right now Lustre aligns BRW lock by the page size on the client side. Please check the code and comments in function ldlm_extent_internal_policy_fixup(). Since client doesn’t provide the page size to server explicitly, the code just guess it by the req_end.</div>
<div><br class="">
</div>
<div>In the new code with this feature supported, the LDLM lock should be aligned to MAX(zfs_block_size, req_align).</div>
<br class="">
<blockquote type="cite" class="">
<div class="">
<div class=""><br class="">
<br class="">
<blockquote type="cite" class="">4. OSD-ZFS can dynamically extend the block size by the write<br class="">
pattern, so we need to disable it to accommodate this feature;<br class="">
</blockquote>
<br class="">
We thought to set the sizes from Lustre (client or later server) and<br class="">
force ZFS to use them. ZFS itself will not be able to change any<br class="">
layouts.<br class="">
</div>
</div>
</blockquote>
<div><br class="">
</div>
<div>Sounds good to me. There is a work in progress to support setting block size from client side in LU-8591.</div>
<div><br class="">
</div>
<blockquote type="cite" class="">
<div class="">
<div class=""><br class="">
Matt, <br class="">
<br class="">
<blockquote type="cite" class=""> <br class="">
<blockquote type="cite" class="">a possible feature is to enable ZFS to decompress the data<br class="">
</blockquote>
<br class="">
I would recommend that you plan to integrate this compression with<br class="">
ZFS from the beginning, by using compression formats that ZFS already<br class="">
supports (e.g. lz4), or by adding support in ZFS for the algorithm<br class="">
you will use for Lustre. This will provide better flexibility and<br class="">
compatibility.<br class="">
</blockquote>
<br class="">
We currently experiment with lz4 fast, which our students try to submit<br class="">
to the linux kernel. The ZFS patch for that will hopefully follow soon.<br class="">
We thought it would be nice to have the opportunity to use some brand<br class="">
new algorithms on the client within Lustre even if they are not yet<br class="">
supported by ZFS. Though it is great that the ZFS community is open to<br class="">
integrate new features so we probably could completely match our needs.<br class="">
</div>
</div>
</blockquote>
<div><br class="">
</div>
<div>That’ll be cool. I think ZFS community should be open to accept new algorithm. Please make a patch and submit it to <a href="https://github.com/zfsonlinux/zfs/issues" class="">https://github.com/zfsonlinux/zfs/issues</a></div>
<br class="">
<blockquote type="cite" class="">
<div class="">
<div class=""><br class="">
<blockquote type="cite" class=""> <br class="">
Also, I agree with what Jinshan said below. Assuming that you want<br class="">
to do compressed read as well, you will need to add a compressed read<br class="">
function to the DMU. For compressed send/receive we only added<br class="">
compressed write to the DMU, because zfs send reads directly from the<br class="">
ARC (which can do compressed read).<br class="">
</blockquote>
<br class="">
We are working on it right now, the functionality should be similar to<br class="">
the write case or do I miss some fundamental issues? <br class="">
</div>
</div>
</blockquote>
<div><br class="">
</div>
It should be similar to write case, i.e., to bypass the layer of dmu buffer.</div>
<div><br class="">
</div>
<div>Jinshan</div>
<div><br class="">
<blockquote type="cite" class="">
<div class="">
<div class=""><br class="">
<br class="">
Best regards,<br class="">
Anna<br class="">
<br class="">
_______________________________________________<br class="">
lustre-devel mailing list<br class="">
<a href="mailto:lustre-devel@lists.lustre.org" class="">lustre-devel@lists.lustre.org</a><br class="">
http://lists.lustre.org/listinfo.cgi/lustre-devel-lustre.org<br class="">
</div>
</div>
</blockquote>
</div>
<br class="">
</div>
</body>
</html>