<html><head></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><br><div><div>On Jun 8, 2011, at 9:08 AM, Nic Henke wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite"><div>On 06/07/2011 06:57 PM, Oleg Drokin wrote:<br><blockquote type="cite">Hello!<br></blockquote><blockquote type="cite"><br></blockquote><br><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">It used to be that only the first and last page in an IOV were allowed<br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">to be of a offset + length< PAGE_SIZE.<br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">Quite correct. LNDs have relied on this for years now.<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">A change like this should not have occurred without discussion<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">about the wider impact.<br></blockquote></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">Actually now that we found what's happening, I think the issue is a bit less clear-cut.<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">What happens here is the client is submitting two niobufs that are not contiguous.<br></blockquote><blockquote type="cite">As such I see no reason why they need to be contiguous in VM too. Sure the 1.8 way of handling<br></blockquote><blockquote type="cite">this situation was to send separate RPCs, but I think even if two RDMA descriptors need to be<br></blockquote><blockquote type="cite">made, we still save plenty of overhead to justify this.<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">(basically we send three niobufs in this case: file pages 0-1, 40-47 (the 47th one is partial) and 49 (full) ).<br></blockquote><br>Oleg - it isn't clear to me what fix you are suggesting here. Are you <br>saying LNet/LNDs should handle this situation (partial internal page) <br>under the covers by setting up multiple RDMA on their own? This sounds <br>like an LND API change, requiring a fix and validation for every LND. I <br>*think* we might end up violating LNet layering here by having to adjust <br>internal LNet structures from the LND to make sure the 2nd and <br>subsequent RDMA landed at the correct spot in the MD, etc.<br></div></blockquote><div><br></div><div>Please refer to LU-394 for detail description of this problem. For those who cannot access our jira system, I'm going to summarize it here.</div><div><br></div><div>The problem is as follows:</div><div>0. First, the app wrote a partial page A at the end of file, and it had enough grant on client side, so page A was cached;</div><div>1. app sought forward and wrote another page B which exceeded the quota limit, so it has to be written in sync mode(see vvp_io_commit_write);</div><div>2. in current implementation of CLIO, for performance consideration, writing page B will include as many cached pages as possible to compose an RPC, this includes page A;</div><div><br></div><div>So here comes the problem. The file size can only be extended until writing of page B succeed, otherwise, the file size is wrong in case writing of B fails. This causes ap_refresh_count() to page A returned oap_count less than CFS_PAGE_SIZE. This is why LND saw uncontiguous pages.</div><div><br></div><div>Fixing this issue is easy, we only write the sync page in a standalone RPC(not combine with cached pages). This is not a big issue, since it occurs only when quota runs out.</div><br><blockquote type="cite"><div><br>At least for our network, and I'd venture a guess for others, there is <br>no way to handle the partial page other than multiple RDMA at the LND <br>layer. When mapping these pages for RDMA, the internal hole can't be <br>handled as we just map a set of physical pages for the HW to read <br>from/write into with a single (address,length) vector. The internal hole <br>would be ignored and would end up corrupting data as we overwrite the hole.<br></div></blockquote><div><br></div><div>Can't aglee more. Having multiple RDMA descriptors will make LND more complex.</div><div><br></div><div>What if we transferred the whole page anyway? This is okay because page offset and length will tell server which part of data is really useful. It will waste some bandwidth, but it's far better than issuing one more RPC.</div><div><br></div><div>Thanks,</div><div>Jinshan</div><br><blockquote type="cite"><div><br>Cheers,<br>Nic<br>_______________________________________________<br>Lustre-devel mailing list<br><a href="mailto:Lustre-devel@lists.lustre.org">Lustre-devel@lists.lustre.org</a><br>http://lists.lustre.org/mailman/listinfo/lustre-devel<br></div></blockquote></div><br></body></html>