<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
Hello!
<div class=""><br class="">
</div>
<div class=""> Thank you for your work on this.</div>
<div class=""><br class="">
</div>
<div class="">the test infra is not open source, but it is open - it’s just gerrit, so you can plug your own CI nodes</div>
<div class="">to report test results for configurations that you think are well working (to ensure they don’t break).</div>
<div class=""> This applies to everybody having ideas about gaps in test coverage for patches in review (which is</div>
<div class=""> the most important part of the pipeline IMO - stuff that was not caught on this step and got committed is much</div>
<div class=""> harder to fix - it then needs a triage for failures, a ticket and somebody assigned to do it and another patch) </div>
<div class=""><br class="">
</div>
<div class=""> It’ll need some work on your end to create the setup that meets the level of testing you feel is adequate,</div>
<div class=""> e.g. you could do something simple using this builder <a href="https://wiki.lustre.org/index.php?title=Simple_Gerrit_Builder_Howto" class="">https://wiki.lustre.org/index.php?title=Simple_Gerrit_Builder_Howto</a></div>
<div class=""><br class="">
</div>
<div class=""> or you could do something really advanced like what I have with the janitortester that is also on github <a href="https://github.com/verygreen/lustretester" class="">https://github.com/verygreen/lustretester</a></div>
<div class=""><br class="">
</div>
<div class=""> or I guess you can use some off the shelf CI solution that integrtes with gerrit.</div>
<div class=""><br class="">
</div>
<div class=""> You can also report results into Maloo DB if you don’t want to host your own logs infra, there’s API for that, though I see it’s not exactly public</div>
<div class=""> but that’s probably just an omission and it should be made public <a href="https://wiki.whamcloud.com/display/TEI/Test+results+format" class="">https://wiki.whamcloud.com/display/TEI/Test+results+format</a></div>
<div class=""> Let me know if that’s something you are interested in and I will try to provide you with this data.</div>
<div class=""><br class="">
</div>
<div class="">
<div><br class="">
<blockquote type="cite" class="">
<div class="">On Dec 19, 2021, at 9:30 PM, Xinliang Liu <<a href="mailto:xinliang.liu@linaro.org" class="">xinliang.liu@linaro.org</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div dir="ltr" class="">(Maybe converting to plain text email will be more readable.)<br class="">
<br class="">
Hi Peter and all,<br class="">
As Kevin(on cc) and I have been working on Lustre Arm stuff for some time.<br class="">
We want to give a status and progress report to the community and list our work plan for the next year.<br class="">
Please help to review our work plan and give some comments and suggestions. Thanks.<br class="">
<br class="">
<br class="">
Status and Progress<br class="">
================<br class="">
Release<br class="">
-------<br class="">
- No Arm packages built on official community release yet.<br class="">
<br class="">
Build<br class="">
------<br class="">
- Verified Lustre, openZFS build and multi-nodes setup on Arm64 CentOS 8, all are ok.<br class="">
- Lbuild script support for Arm is on review, LU-15293.<br class="">
<br class="">
CI<br class="">
---<br class="">
- No Arm server end CI support yet.<br class="">
- Arm client with x86_64 server test is already in the CI gate.<br class="">
- Only run a few ldiskfs test suites(sanity, sanity-sec, sanctity-lnet, etc.), not a full test.<br class="">
- A full test (with empty GRANT_CHECK_LIST) shows several Arm client related failed test cases, see test results page:
<a href="https://testing.whamcloud.com/test_sessions?jobs=lustre-reviews&builds=82774&start_date=2021-08-26#redirect" class="">
https://testing.whamcloud.com/test_sessions?jobs=lustre-reviews&builds=82774&start_date=2021-08-26#redirect</a><br class="">
- sanity test 317: LU-11667 (Workaround fix landed)<br class="">
- sanityn test 16a: LU-11597, test 71a: LU-11787<br class="">
- conf-sanity test 98: LU-11785, test 112: LU-13813<br class="">
- sanity-flr test 50a: LU-14970<br class="">
- sanity-pcc test 7a: LU-14346<br class="">
<br class="">
Arm server end test on local setup<br class="">
---------------------------------------------<br class="">
- Run a full ldiskfs test with all test suites.<br class="">
- Due to the multi MDTs crash issue, some multi MDTs tests are not run.<br class="">
- Many new failed tests come, see the test result google sheet for details: <a href="https://docs.google.com/spreadsheets/d/1EE5zU96_lqlkS0uk6NJeeNBrikYpd_ZEO7hdVt5spsw/edit#gid=969410610" class="">
https://docs.google.com/spreadsheets/d/1EE5zU96_lqlkS0uk6NJeeNBrikYpd_ZEO7hdVt5spsw/edit#gid=969410610</a><br class="">
- The openZFS full test is not run, but heard that it should be more stable than ldiskfs.<br class="">
<br class="">
Bugfix<br class="">
-------<br class="">
- Old Arm always_except bugs <a href="https://jira.whamcloud.com/issues/?filter=15555" class="">
https://jira.whamcloud.com/issues/?filter=15555</a> , the Arm related ones are almost addressed.<br class="">
- LU-11596, LU-11597, LU-14067, LU-11787: addressed, patch sent and waiting for Arm client CI recovery to land.<br class="">
- LU-10073, LU-11671: can't be reproduced on Arm or happen on x86_64 also.<br class="">
<br class="">
- Other old Arm bugs LU-11785, LU-13813, LU-14970, LU-14346 to be fixed.<br class="">
<br class="">
- New created server end bugs<br class="">
- LU-15122 : ASSERTION( iobuf->dr_rw == 0 ) crash issue, fixed patch is landed.<br class="">
- LU-15364: multi MDTs kernel oops issue, related to atomic unaligned memory access, work in progress.<br class="">
- LU-15223: 64K page size read/write improvement, long-term work, in progress.<br class="">
<br class="">
- Full Arm related bug list with label arm: <a href="https://jira.whamcloud.com/issues/?filter=16710" class="">
https://jira.whamcloud.com/issues/?filter=16710</a><br class="">
<br class="">
Reference to:<br class="">
James Simmons’ Lustre Arm update: <a href="https://connect.linaro.org/resources/san19/san19-224/" class="">
https://connect.linaro.org/resources/san19/san19-224/</a><br class="">
<br class="">
<br class="">
Work Plan<br class="">
========<br class="">
- Lustre Server End Critical Bug Fix target 2022-06<br class="">
- Lustre Multiple MDTs kernel OOPS when stripe issue: LU-15364<br class="">
- Lustre hangs at Sanity Test 807<br class="">
- Lustre Conf-sanity test 44 kernel crash<br class="">
- Lustre Conf sanity case 58 kernel crash<br class="">
- Lustre Conf sanity case 78 kernel crash<br class="">
- Lustre Conf sanity case 79 crash<br class="">
- Lustre sanity-pcc 7a case hang the cluster<br class="">
<br class="">
- Lustre Server End Non-critical Bug Fix target 2022-12<br class="">
- Lustre Sanity failure cases: 33 cases<br class="">
- Lustre server replay-single: 1 case<br class="">
- Lustre sanity-flr 200 cases fix: 1 case<br class="">
- Lustre sanity-hsm failure cases: 25 cases<br class="">
- Lustre lustre-rsync-test failure test: 3 cases<br class="">
- Lustre recovery-small/sanity-scrub: 2 cases<br class="">
- Lustre sanityn test cases fix: 12 cases<br class="">
- Lustre sanity-lfsck failure cases fix: 3 cases<br class="">
- Lustre sanity-sec failure cases fix ：7 cases<br class="">
- Lustre sanity-lnet failure cases test fix: 2 cases<br class="">
<br class="">
- Continuous add more test suites for Arm client CI ??<br class="">
- Once a test suite is all passed for Arm then add it into CI.<br class="">
<br class="">
- Server CI support for Arm on Centos8 ??<br class="">
- Ideally, Arm server CI can come with Arm server end fixes patches and ensure future patches merged don’t make any regressions on Arm.<br class="">
- As the test infra is not open source and maintained by whamcloud, it might need whamcloud to make it ??<br class="">
<br class="">
- Other works in future<br class="">
- Full test with openZFS backend.<br class="">
- Test x86 client with Arm64 Server<br class="">
- Test other distros like ubuntu, SUSE etc.<br class="">
- Basic Optimised: CRC/AES<br class="">
- All-flash optimization<br class="">
<br class="">
<br class="">
Best Regards,<br class="">
Xinliang<br class="">
</div>
</div>
</blockquote>
</div>
<br class="">
</div>
</body>
</html>