<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
Kevin,
<div class="">Qian has a patch <a href="https://review.whamcloud.com/40092" class="">https://review.whamcloud.com/40092</a> "<a href="https://jira.whamcloud.com/browse/LU-14003" class="">LU-14003</a> pcc: rework PCC mmap implementation" that is changing the
PCC MMAP code significantly, but is waiting for the 2.16.0 feature landing window to open. It needs to be refreshed, but it would be helpful if you could take a look through that patch to see if it would resolve the issue you are seeing.<br class="">
<div><br class="">
<blockquote type="cite" class="">
<div class="">On Mar 11, 2022, at 01:18, Kevin Zhao via lustre-devel <<a href="mailto:lustre-devel@lists.lustre.org" class="">lustre-devel@lists.lustre.org</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div dir="ltr" class="">Hi,
<div class=""><br class="">
</div>
<div class="">Recently we've worked on the bug <a href="https://jira.whamcloud.com/browse/LU-14346" class="">https://jira.whamcloud.com/browse/LU-14346</a>. This bug will make the
<b class="">mmap write</b> hang forever. This one is first occurring on Aarch64, but if we do a
<a href="https://github.com/kevinzs2048/devbox/blob/master/notes/lustre/pcc_sanity-pcc-7a-analysis.md#reproduced-on-x86_64-with-a-small-change" class="">
small change</a>, we <b class="">can easily reproduce it on X86_64</b>. For more details analysis of this bug, you can also<a href="https://github.com/kevinzs2048/devbox/blob/master/notes/lustre/pcc_sanity-pcc-7a-analysis.md" class=""> check the link</a>.</div>
<div class=""><br class="">
</div>
<div class="">The <a href="https://github.com/lustre/lustre-release/blob/master/lustre/tests/multiop.c#L725" class="">
hang location is here</a> as below:<br class="">
==============<br class="">
case 'W':<br class="">
for (i = 0; i < mmap_len && mmap_ptr; i += 4096)<br class="">
mmap_ptr[i] += junk++;<br class="">
break;<br class="">
===============</div>
<div class=""><br class="">
</div>
<div class=""><b class="">Bug Analysis - different behavior when run </b><b class="">mmap_ptr[i] += junk++ on different platform.</b></div>
<div class="">Traditionally, this process is:<br class="">
1. read from mmap_ptr[i]first(Execute the read page fault)<br class="">
2. Write a value to the same page(execute the page_mkwrite to change the page to writable).<br class="">
<br class="">
But on different platforms, it executes quite differently.<br class="">
On aarch64 platform: do_page_fault, no FAULT_FLAG_WRITE set, so handle_pte_fault will call do_read_fault
<ul class="">
<li class="">do_read_fault:<br class="">
__do_fault -> call ll_fault, get a page from pcc_fault<br class="">
finish_fault(map the returned page to page tables)<br class="">
unlock_page<br class="">
vmf->flags is VM_FAULT_LOCKED</li><li class="">call do_wp_page --> do_page_mkwrite --> ll_page_mkwrite</li></ul>
On X86_64 platform, the mechanism is different. On X86_64, do_page_fault, with <b class="">
FAULT_FLAG_WRITE set</b>, so handle_pte_fault will call <b class="">do_shared_fault</b>.
<ul class="">
<li class="">do_shared_fault</li><ul class="">
<li class=""> __do_fault -> call ll_fault, get a page from pcc_fault</li><li class="">do_page_mkwrite-> call ll_page_mkwrite</li><li class="">finish_fault(map the returned page to page tables)</li><li class="">fault_dirty_shared_page</li></ul>
</ul>
</div>
<div class=""><b class="">Bug Analysis: why hang forever:</b></div>
<div class="">Also can check: <a href="https://github.com/kevinzs2048/devbox/blob/master/notes/lustre/pcc_sanity-pcc-7a-analysis.md#kernel-do_page_fault-process-analysis" class="">https://github.com/kevinzs2048/devbox/blob/master/notes/lustre/pcc_sanity-pcc-7a-analysis.md#kernel-do_page_fault-process-analysis</a>
for more details.</div>
<div class=""><br class="">
</div>
<div class="">do_page_mkwrite--->ll_page_mkwrite.<br class="">
Insert the issue 0x1412 OBD_FAIL_LLITE_PCC_DETACH_MKWRITE.<br class="">
Return with VM_FAULT_RETRY | VM_FAULT_NOPAGE<br class="">
RETRY again, due to PTE is not NULL, vmf->flags FAULT_FLAG_WRITE, will call do_wp_page again.<br class="">
So that next time we will enter into do_page_mkwrite again. hanging forever.<br class="">
</div>
<div class=""><br class="">
</div>
<div class=""><b class="">Seek a good solution</b></div>
As the above code snippet shows, <b class="">we want to let the kernel retry the mmap write (->fault() and ->page_mkwrite).</b><br class="">
In handle_pte_fault, if there is no page or the page is not mapped(no PTE found), then<br class="">
__do_page_fault will try the memory fault handling. <br class="">
<br class="">
The easy fix here is to<b class=""> remove the page and page table entry when we do fail injection in pcc_page_mkwrite.</b> But I don't find a good method to execute this, so list the info here and ask for community help.<br class="">
<br class="">
Some tried fix is:<br class="">
<div class="">add function: <b class="">generic_error_remove_page</b>, but the mapped page still can not be unmapped successfully.
<a href="https://github.com/kevinzs2048/devbox/blob/master/notes/lustre/pcc_sanity-pcc-7a-analysis.md#solution" class="">
The error log is here</a>.<br class="">
</div>
<div class=""><br class="">
</div>
<div class="">Since I'm a newbie to Lustre and not quite familiar with the memory management process, so please give some advice on this bug fix. Thanks in advance.</div>
</div>
</div>
</blockquote>
</div>
<br class="">
<div class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div>Cheers, Andreas</div>
<div>--</div>
<div>Andreas Dilger</div>
<div>Lustre Principal Architect</div>
<div>Whamcloud</div>
<div><br class="">
</div>
<div><br class="">
</div>
<div><br class="">
</div>
</div>
</div>
</div>
</div>
</div>
<br class="Apple-interchange-newline">
</div>
<br class="Apple-interchange-newline">
<br class="Apple-interchange-newline">
</div>
<br class="">
</div>
</body>
</html>