[lustre-devel] [LSF/MM/BPF TOPIC] [DRAFT] Lustre client upstreaming

[Day, Timothy]

The following is a draft topic for the upcoming LSF/MM conference.
I wanted to solicit feedback from the wider Lustre development
community before submitting this to fsdevel. If I’ve omitted anything,
something doesn’t seem right, or you know of something that strengthens
the argument, please let me know!

----------------------------------------------------

Lustre is a high-performance parallel filesystem used for HPC and AI/ML
compute clusters available under GPLv2. Lustre has achieved widespread
adoption in the HPC and AI/ML and is commercially supported by numerous
vendors and cloud service providers [1].

After 21 years and an ill-fated stint in staging, Lustre is still maintained as
an out-of-tree module [6]. The previous upstreaming effort suffered from a
lack of developer focus and user adoption, which eventually led to Lustre
being removed from staging altogether [2].

However, the work to improve Lustre has not stopped. In the intervening
years, the code improvements that would preempt a return to mainline
have been steadily progressing. At least 25% of patches accepted for
Lustre 2.16 were related to the upstreaming effort [3]. And all of the
remaining work is in-flight [4][5]. Our eventual goal is to a get a minimal
TCP/IP-only Lustre client to an acceptable quality before submitting to
mainline.

I propose to discuss:

- Expectations for a new filesystem to be accepted to mainline
- Weaknesses in the previous upstreaming effort in staging

Lustre has already received a plethora of feedback in the past. While much
of that has been addressed since - the kernel is a moving target. Several
filesystems have been merged (and removed) since Lustre left staging. We're
aiming to avoid the mistakes of the past and hope to address as many
concerns as possible before submitting for inclusion.

Thanks!

Timothy Day (Amazon Web Services - AWS)
James Simmons (Oak Ridge National Labs - ORNL)

[1] Lustre Community Update: https://youtu.be/BE--ySVQb2M?si=YMHitJfcE4ASWQcE&t=960
[2] Kicked out of staging: https://lwn.net/Articles/756565/
[3] ORNL, Aeon, SuSe, AWS, and more: https://youtu.be/BE--ySVQb2M?si=YMHitJfcE4ASWQcE&t=960
[4] LUG24 Upstreaming Update: https://www.depts.ttu.edu/hpcc/events/LUG24/slides/Day1/LUG_2024_Talk_02-Native_Linux_client_status.pdf
[5] Lustre Jira Upstream Progress: https://jira.whamcloud.com/browse/LU-12511
[6] Out-of-tree codebase: https://git.whamcloud.com/?p=fs/lustre-release.git;a=tree

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-devel-lustre.org/attachments/20250116/31f11017/attachment.htm>