[lustre-devel] Lustre Arm stuff status and work plan

Sun Dec 26 23:56:00 PST 2021

Hello Oleg,
Thank you for your suggestions about Arm CI.

On Thu, 23 Dec 2021 at 15:43, Oleg Drokin <green at whamcloud.com> wrote:

> Hello!
>
>  Thank you for your work on this.
>
> the test infra is not open source, but it is open - it’s just gerrit, so
> you can plug your own CI nodes
> to report test results for configurations that you think are well working
> (to ensure they don’t break).
>  This applies to everybody having ideas about gaps in test coverage for
> patches in review (which is
>   the most important part of the pipeline IMO - stuff that was not caught
> on this step and got committed is much
>   harder to fix - it then needs a triage for failures, a ticket and
> somebody assigned to do it and another patch)
>
>  It’ll need some work on your end to create the setup that meets the level
> of testing you feel is adequate,
>  e.g. you could do something simple using this builder
> https://wiki.lustre.org/index.php?title=Simple_Gerrit_Builder_Howto
>
>  or you could do something really advanced like what I have with the
> janitortester that is also on github
> https://github.com/verygreen/lustretester
>
>   or I guess you can use some off the shelf CI solution that integrtes
> with gerrit.
>
>   You can also report results into Maloo DB if you don’t want to host your
> own logs infra, there’s API for that, though I see it’s not exactly public
>    but that’s probably just an omission and it should be made public
> https://wiki.whamcloud.com/display/TEI/Test+results+format
>    Let me know if that’s something you are interested in and I will try to
> provide you with this data.
>

Yeah, that sounds interesting. Please provide me with this data, the link
responses 404 now., thanks.

Ideally, we want enough Arm CI tests as x86_64. And these tests are running
and maintained in the Lustre official test infra like review-ldiskfs-arm we
have now.
But if it is impossible now, we will consider setting up such CI on our own.

And if we run some kind of tests externally in our end, we can see the test
result (pass or not, running log link) displayed on the gerrit patch page
as the community running tests.
We want to make our test results public to the Lustre community, so that
developers can see the test results and adjust the patch if necessary. Thus
avoiding breaking previous Arm work.
That can be made via reporting results into Maloo DB, right? Am I
understanding correctly?

We will look into the links and evaluate on setting up our own or
self-hosted test infra and see if we can make it.
Such as,
1. How many VMs resources for the test infra and test running;
2. Other requirements for the self-hosted test infra;
3. Work items to be done;
..

Thanks,
Xinliang

>
> On Dec 19, 2021, at 9:30 PM, Xinliang Liu <xinliang.liu at linaro.org> wrote:
>
> (Maybe converting to plain text email will be more readable.)
>
> Hi Peter and all,
> As Kevin(on cc) and I have been working on Lustre Arm stuff for some time.
> We want to give a status and progress report to the community and list our
> work plan for the next year.
> Please help to review our work plan and give some comments and
> suggestions. Thanks.
>
>
> Status and Progress
> ================
> Release
> -------
> - No Arm packages built on official community release yet.
>
> Build
> ------
> - Verified Lustre, openZFS build and multi-nodes setup on Arm64 CentOS 8,
> all are ok.
> - Lbuild script support for Arm is on review, LU-15293.
>
> CI
> ---
> - No Arm server end CI support yet.
> - Arm client with x86_64 server test is already in the CI gate.
>   - Only run a few ldiskfs test suites(sanity, sanity-sec, sanctity-lnet,
> etc.), not a full test.
>   - A full test (with empty GRANT_CHECK_LIST) shows several Arm client
> related failed test cases, see test results page:
> https://testing.whamcloud.com/test_sessions?jobs=lustre-reviews&builds=82774&start_date=2021-08-26#redirect
>     - sanity test 317: LU-11667 (Workaround fix landed)
>     - sanityn test 16a: LU-11597, test 71a: LU-11787
>     - conf-sanity test 98: LU-11785, test 112: LU-13813
>     - sanity-flr test 50a: LU-14970
>     - sanity-pcc test 7a: LU-14346
>
> Arm server end test on local setup
> ---------------------------------------------
> - Run a full ldiskfs test with all test suites.
>   - Due to the multi MDTs crash issue, some multi MDTs tests are not run.
>   - Many new failed tests come, see the test result google sheet for
> details:
> https://docs.google.com/spreadsheets/d/1EE5zU96_lqlkS0uk6NJeeNBrikYpd_ZEO7hdVt5spsw/edit#gid=969410610
>   - The openZFS full test is not run, but heard that it should be more
> stable than ldiskfs.
>
> Bugfix
> -------
> - Old Arm always_except bugs
> https://jira.whamcloud.com/issues/?filter=15555 , the Arm related ones
> are almost addressed.
>   - LU-11596, LU-11597, LU-14067, LU-11787: addressed, patch sent and
> waiting for Arm client CI recovery to land.
>   - LU-10073, LU-11671: can't be reproduced on Arm or happen on x86_64
> also.
>
> - Other old Arm bugs  LU-11785, LU-13813, LU-14970, LU-14346 to be fixed.
>
> - New created server end bugs
>   - LU-15122 : ASSERTION( iobuf->dr_rw == 0 ) crash issue, fixed patch is
> landed.
>   - LU-15364: multi MDTs kernel oops issue, related to atomic unaligned
> memory access, work in progress.
>   - LU-15223: 64K page size read/write improvement, long-term work, in
> progress.
>
> - Full Arm related bug list with label arm:
> https://jira.whamcloud.com/issues/?filter=16710
>
> Reference to:
> James Simmons’ Lustre Arm update:
> https://connect.linaro.org/resources/san19/san19-224/
>
>
> Work Plan
> ========
> - Lustre Server End Critical Bug Fix target 2022-06
>   - Lustre Multiple MDTs kernel OOPS when stripe issue: LU-15364
>   - Lustre hangs at Sanity Test 807
>   - Lustre Conf-sanity test 44 kernel crash
>   - Lustre Conf sanity case 58 kernel crash
>   - Lustre Conf sanity case 78 kernel crash
>   - Lustre Conf sanity case 79 crash
>   - Lustre sanity-pcc 7a case hang the cluster
>
> - Lustre Server End Non-critical Bug Fix target 2022-12
>   - Lustre Sanity failure cases: 33 cases
>   - Lustre server replay-single: 1 case
>   - Lustre sanity-flr 200 cases fix: 1 case
>   - Lustre sanity-hsm failure cases: 25 cases
>   - Lustre lustre-rsync-test failure test: 3 cases
>   - Lustre recovery-small/sanity-scrub: 2 cases
>   - Lustre sanityn test cases fix: 12 cases
>   - Lustre sanity-lfsck failure cases fix: 3 cases
>   - Lustre sanity-sec failure cases fix ：7 cases
>   - Lustre sanity-lnet failure cases test fix: 2 cases
>
> - Continuous add more test suites for Arm client CI ??
>   - Once a test suite is all passed for Arm then add it into CI.
>
> - Server CI support for Arm on Centos8 ??
>   - Ideally, Arm server CI can come with Arm server end fixes patches and
> ensure future patches merged don’t make any regressions on Arm.
>   - As the test infra is not open source and maintained by whamcloud, it
> might need whamcloud to make it ??
>
> - Other works in future
>   - Full test with openZFS backend.
>   - Test x86 client with Arm64 Server
>   - Test other distros like ubuntu, SUSE etc.
>   - Basic Optimised: CRC/AES
>   - All-flash optimization
>
>
> Best Regards,
> Xinliang
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-devel-lustre.org/attachments/20211227/10f49c4d/attachment.html>