<div dir="ltr"><div dir="ltr"><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Jun 26, 2019 at 6:11 AM Anna Fuchs <<a href="mailto:anna.fuchs@informatik.uni-hamburg.de">anna.fuchs@informatik.uni-hamburg.de</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div id="gmail-m_-4230956602455003064geary-body" dir="auto"><div><div>Dear all, </div><div><br></div><div>one more question related to ZFS-buffers in Lustre.</div><div><br></div><div>There is a function osd_grow_blocksize(obj, oh, ...) called after the fist portion of data (first rnb?) </div><div>has been committed to ZFS.</div><div>There are some restrictions for block size changing:</div><div>dmu_object_set_blocksize says:</div><div>The object cannot have any blocks allcated beyond the first. If</div><div>* the first block is allocated already, the new size must be greater</div><div>* than the current block size.</div><div>and later on</div><div>/*</div><div> * Try to change the block size for the indicated dnode. This can only</div><div> * succeed if there are no blocks allocated or dirty beyond first block</div><div> */ </div><div><br></div><div>I am now interested on the frist block's size, which seems to be set when creating the dnode.</div><div>This size comes from ZFS and is something like</div><div>dnp->dn_datablkszsec << SPA_MINBLOCKSHIFT or SPA_MINBLOCKSIZE (not sure).</div></div></div></blockquote><div><br></div><div> The block size in bytes is `dnp->dn_datablkszsec << SPA_MINBLOCKSHIFT`.</div><div>FYI, SPA_MINBLOCKSIZE == 1<<SPA_MINBLOCKSHIFT</div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div id="gmail-m_-4230956602455003064geary-body" dir="auto"><div><div><br></div><div>I would like to specify this size on Lustre's side, not just take what ZFS offers.</div><div>E.g. make the first block 128K instead of 4K.</div></div></div></blockquote><div><br></div><div>You can set the block size (of the first and only block) using dmu_object_set_blocksize(). FYI, I think that this comment is incorrect:</div><div><span style="color:rgb(78,144,114);font-family:Monaco;font-size:12px"> * If </span><span style="color:rgb(78,144,114);font-family:Monaco;font-size:12px">the first block is allocated already, the new size must be greater</span></div>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:Monaco;color:rgb(78,144,114)"><span class="gmail-Apple-converted-space"> </span>* than the current block size.</p><div>You can increase or decrease the block size with this routine.</div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div id="gmail-m_-4230956602455003064geary-body" dir="auto"><div><div>Is it possible? Could I just overwrite the block size before the corresponsing memory for the block is allocated?</div><div><br></div><div>I am not able to call osd_grow_blocksize for the first block, since I do not have any thread context there, not yet.</div><div>Do I need to grab into dnode_allocate and dnode_create?</div><div><br></div><div>And for better understanding, does one dnode always represent one lustre object?</div><div><br></div><div>I would be greatful for any suggestions.</div><div><br></div><div>***</div><div><br></div><div>Some context for my questions:</div><div><br></div><div>I have compressed data chunks coming from the Lustre client. I want to hand them over to ZFS like they</div><div>were compressed by ZFS. ZFS offers some structures, e.g. compressed arc-buffers, which know how the data has been </div><div>compressed (which algo, physical and logical sizes). I want and need my chunks to be aligned to the records (arc buffers).</div><div>We have already extended the interfaces of the internal ZFS compression structures. But currently ZFS (or osd-zfs) first defines</div><div>the sizes of buffers and the data is put in there. In my case, the data should "dictate" how many buffers there are and how large they can be.</div><div><br></div></div></div></blockquote><div><br></div><div>I'd recommend that you hand the compressed data to ZFS similarly to how "zfs receive" does (for compressed send streams). It sounds like the is the direction you're going, which is great. FYI, here are some of the routines you'd want to use (copied from dmu_recv.c):</div><div><br></div><div>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:Monaco"><span class="gmail-Apple-tab-span" style="white-space:pre"> </span><span class="gmail-Apple-tab-span" style="white-space:pre"> </span><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>abuf = arc_loan_compressed_buf(</p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:Monaco"><span class="gmail-Apple-tab-span" style="white-space:pre"> </span><span class="gmail-Apple-tab-span" style="white-space:pre"> </span><span class="gmail-Apple-tab-span" style="white-space:pre"> </span><span class="gmail-Apple-converted-space"> </span>dmu_objset_spa(drc-><span class="gmail-s1" style="color:rgb(3,38,204)">drc_os</span>),</p>
<p class="gmail-p2" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:Monaco;color:rgb(3,38,204)"><span class="gmail-s2" style="color:rgb(0,0,0)"><span class="gmail-Apple-tab-span" style="white-space:pre"> </span><span class="gmail-Apple-tab-span" style="white-space:pre"> </span><span class="gmail-Apple-tab-span" style="white-space:pre"> </span><span class="gmail-Apple-converted-space"> </span>drrw-></span>drr_compressed_size<span class="gmail-s2" style="color:rgb(0,0,0)">, drrw-></span>drr_logical_size<span class="gmail-s2" style="color:rgb(0,0,0)">,</span></p>
<p class="gmail-p2" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:Monaco;color:rgb(3,38,204)"><span class="gmail-s2" style="color:rgb(0,0,0)"><span class="gmail-Apple-tab-span" style="white-space:pre"> </span><span class="gmail-Apple-tab-span" style="white-space:pre"> </span><span class="gmail-Apple-tab-span" style="white-space:pre"> </span><span class="gmail-Apple-converted-space"> </span>drrw-></span>drr_compressiontype<span class="gmail-s2" style="color:rgb(0,0,0)">);</span></p><p class="gmail-p2" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:Monaco;color:rgb(3,38,204)"><span class="gmail-s2" style="color:rgb(0,0,0)"><br></span></p></div><div>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:Monaco"><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>dmu_assign_arcbuf(bonus, drrw-><span class="gmail-s1" style="color:rgb(3,38,204)">drr_offset</span>, abuf, tx);</p><p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:Monaco"> (or dmu_assign_arcbuf_dnode())</p><p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:Monaco"><br></p>
</div><div>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:Monaco"><span class="gmail-Apple-tab-span" style="white-space:pre"> </span><span class="gmail-Apple-tab-span" style="white-space:pre"> </span><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>dmu_return_arcbuf(rrd-><span class="gmail-s1" style="color:rgb(3,38,204)">write_buf</span>);</p></div><div> </div><div>--matt</div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div id="gmail-m_-4230956602455003064geary-body" dir="auto"><div><div></div><div>Best regards</div><div>Anna</div></div><div><br></div><div>--</div><div>Anna Fuchs</div><div>Universität Hamburg</div></div><div id="gmail-m_-4230956602455003064geary-quote" dir="auto"><br>On Thu, Jun 13, 2019 at 1:54 PM, Anna Fuchs <<a href="mailto:anna.fuchs@informatik.uni-hamburg.de" target="_blank">anna.fuchs@informatik.uni-hamburg.de</a>> wrote:<br><blockquote type="cite"><div id="gmail-m_-4230956602455003064geary-body" dir="auto"><div>Dear all,</div><div><br></div><div>in osd-zfs/osd_io.c:osd_bufs_get_write you can find a comment regarding zerocopy:</div><div><br></div><div><div><span class="gmail-m_-4230956602455003064Apple-tab-span" style="white-space:pre-wrap"> </span>/*</div><div><span class="gmail-m_-4230956602455003064Apple-tab-span" style="white-space:pre-wrap"> </span> * currently only full blocks are subject to zerocopy approach:</div><div><span class="gmail-m_-4230956602455003064Apple-tab-span" style="white-space:pre-wrap"> </span> * so that we're sure nobody is trying to update the same block</div><div><span class="gmail-m_-4230956602455003064Apple-tab-span" style="white-space:pre-wrap"> </span> */</div></div><div><br></div><div>Whenever a block to be written is full, an arc buffer is requested, otherwise alloc_page.</div><div><br></div><div>I do not really understand the conclusion. Why and how do full blocks prevent updates?</div><div>And put it differently - why not to try zerocopy for not full blocks?</div><div>What could happen if I tried to request an arc buffer for e.g. a block with missin
g last page?</div><div><br></div><div>I would be greateful for details.</div><div>Best regards</div><div>Anna</div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div></div></blockquote></div></blockquote></div></div>