[lustre-devel] [LSF/MM/BPF TOPIC] [DRAFT] Lustre client upstreaming

[Day, Timothy]

On 1/16/25, 7:12 PM, "Andreas Dilger" <adilger at ddn.com <mailto:adilger at ddn.com>> wrote:
> On Jan 16, 2025, at 14:25, Day, Timothy <timday at amazon.com <mailto:timday at amazon.com>> wrote:
> >
> > The following is a draft topic for the upcoming LSF/MM conference.
> > I wanted to solicit feedback from the wider Lustre development
> > community before submitting this to fsdevel. If I’ve omitted anything,
> > something doesn’t seem right, or you know of something that strengthens
> > the argument, please let me know!
> > ----------------------------------------------------
> > Lustre is a high-performance parallel filesystem used for HPC and AI/ML
> > compute clusters available under GPLv2. Lustre has achieved widespread
> > adoption in the HPC and AI/ML and is commercially supported by numerous
> > vendors and cloud service providers [1].
>
>
> I don't see Peter's graph that shows adoption as a fraction of the Top-100
> systems. I think this could be listed here explicitly, like:
>
>
> "Lustre is currently used by 65% of the Top-500 (9 of Top-10) systems in HPC
> and is used by the largest AI/ML clusters in the world, and is commercially ..."

That's a good stat. I also want to show that Lustre is gaining a lot of adoption
outside of HPC. I should probably just say that explicitly.

> > After 21 years and an ill-fated stint in staging, Lustre is still maintained as
> > an out-of-tree module [6]. The previous upstreaming effort suffered from a
> > lack of developer focus and user adoption, which eventually led to Lustre
> > being removed from staging altogether [2].
> > However, the work to improve Lustre has not stopped.
>
> "... has continued regardless."
>
> > In the intervening
> > years, the code improvements that would preempt a return to mainline
>
> s/would preempt/previously prevented/
>
> > have been steadily progressing. At least 25% of patches accepted for
> > Lustre 2.16 were related to the upstreaming effort [3].
>
> While [3] is showing the distribution of submissions between organizations,
> it isn't clear how that translates to "25% of patches relate to upstreaming",
> unless you count all of the ORNL, AWS, and AEON submissions toward this?

It's kind of heuristic - the majority of patches from ORNL/AEON/SuSe/AWS are
related to upstreaming work. And quite a lot from other contributors as well. I
should include that thinking in the appendix part.

> > And all of the
> > remaining work is in-flight [4][5].
>
>
> Looking at [5] it would appear that most of the items in LU-12511 are *not* finished, so it
> would make sense to go through those linked tickets and tasks listed in the Description
> to see if they can be closed and/or struck out so that it shows that we are nearly complete.
>
> Similarly, James' presentation in [4] is missing the commentary that would explain which
> of the listed items/tickets were actually finished and which ones are enumerating the
> "todo" list.

I'm going to update LU-12511 before sending the email more widely. I synced with James
to discuss some of the outstanding work.

> > Our eventual goal is to a get a minimal
> > TCP/IP-only Lustre client to an acceptable quality before submitting to
> > mainline.
> > I propose to discuss:
> > - Expectations for a new filesystem to be accepted to mainline
> > - Weaknesses in the previous upstreaming effort in staging
>
>
> Rather than discuss the "weaknesses" in the previous upstreaming, this should be
> focus in the positive direction:
>
>
> - Expectations for a new filesystem to be accepted to mainline
> - How to manage inclusion of a large code base without rewriting (2.5x XFS)

The second bullet probably sounds too negative. But a lot of folks upstream
were not happy with the state of the codebase at the time Lustre was
removed from staging. And quite a lot of Lustre has been reworked/rewritten
since then. Maybe something in middle:

- Expectations for a new filesystem to be accepted to mainline
- How to manage inclusion of a large code base (client is 200kLoC) without
increasing the burden on fs/net maintainers

I imagine (2) would requiring at least some rewriting to avoid deprecated
interfaces. Support for the new mount API might be one ask, for example.

> > Lustre has already received a plethora of feedback in the past. While much
> > of that has been addressed since - the kernel is a moving target. Several
> > filesystems have been merged (and removed) since Lustre left staging. We're
> > aiming to avoid the mistakes of the past and hope to address as many
> > concerns as possible before submitting for inclusion.
> > Thanks!
> > Timothy Day (Amazon Web Services - AWS)
> > James Simmons (Oak Ridge National Labs - ORNL)
> > [1] Lustre Community Update: https://youtu.be/BE--ySVQb2M?si=YMHitJfcE4ASWQcE&t=960 <https://youtu.be/BE--ySVQb2M?si=YMHitJfcE4ASWQcE&t=960>
> > [2] Kicked out of staging: https://lwn.net/Articles/756565/ <https://lwn.net/Articles/756565/>
> > [3] ORNL, Aeon, SuSe, AWS, and more: https://youtu.be/BE--ySVQb2M?si=YMHitJfcE4ASWQcE&t=960 <https://youtu.be/BE--ySVQb2M?si=YMHitJfcE4ASWQcE&t=960>
>
>
> Your [1] and [3] URLs are exactly the same. Did you mean to be showing something
> different for each (e.g. different "t=NNN" for one or the other)?

For [1], I should probably just enumerate some organizations. I want to
demonstrate that the Lustre community and adoption is growing. I could
also just link to https://en.wikipedia.org/wiki/Lustre_(file_system)#Commercial_technical_support.

> > [4] LUG24 Upstreaming Update: https://www.depts.ttu.edu/hpcc/events/LUG24/slides/Day1/LUG_2024_Talk_02-Native_Linux_client_status.pdf <https://www.depts.ttu.edu/hpcc/events/LUG24/slides/Day1/LUG_2024_Talk_02-Native_Linux_client_status.pdf>
> > [5] Lustre Jira Upstream Progress: https://jira.whamcloud.com/browse/LU-12511 <https://jira.whamcloud.com/browse/LU-12511>
> > [6] Out-of-tree codebase: https://git.whamcloud.com/?p=fs/lustre-release.git;a=tree <https://git.whamcloud.com/?p=fs/lustre-release.git;a=tree>
>
>
> Cheers, Andreas
> —
> Andreas Dilger
> Lustre Principal Architect
> Whamcloud/DDN

I tried to reply inline - hopefully Outlook doesn't mangle the email.

Tim Day