<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;">
On Jan 21, 2025, at 23:35, Day, Timothy <timday@amazon.com> wrote:<br>
<div>
<blockquote type="cite"><br class="Apple-interchange-newline">
<div>
<div>I've created a second draft of the topic for LSF/MM. I tried<br>
to include everyone's feedback. It's at the end of the email.<br>
<br>
Before that, I wanted to elaborate on Neil's idea about updating<br>
our development model to an upstream-focused model. For upstreaming<br>
to work, the normal development flow has to generate patches to mainline<br>
Linux - while still supporting the distro kernels that most people use<br>
to run Lustre. I think we can get this point in stages. I've provided<br>
a high-level overview in the next section. This won't be without<br>
challenges - but the majority of the transition could happen without<br>
interrupting feature work or normal development.<br>
<br>
[I] Separate the kernel code, compatibility code, and userspace code<br>
<br>
We should reorganize the Lustre tree to have a clear separation<br>
of concerns:<br>
<br>
fs/lustre/<br>
net/lnet/<br>
net/libcfs/<br>
lustre_compat/<br>
tests/<br>
utils/<br>
<br>
The functional components of libcfs/ would stay in that directory<br>
and the compatibility components would live in lustre_compat/.<br>
Centralizing the compatibility code makes it easier to maintain and<br>
update and allows us to start removing the compatibility code from<br>
the modules themselves. lustre_compat/ could still be compiled into<br>
libcfs.ko, if we want to avoid creating even more modules.<br>
</div>
</div>
</blockquote>
<div><br>
</div>
I think this proposal is pretty reasonable. If the directory renaming</div>
<div>is done in the Lustre repo (say in the 2.17 timeframe) without also</div>
<div>restructuring all of the files and code at the same time, then it </div>
<div>_should_ be reasonable to backport patches to older maintenance</div>
<div>branches without having to rewrite them completely.</div>
<div><br>
</div>
<div>This would essentially form an "overlay" to the existing kernel tree,</div>
<div>and should allow it to be built as a standalone project until such a</div>
<div>time that it is accepted.</div>
<div><br>
</div>
<div>Conversely, if the consensus from LSF is that Lustre will never get</div>
<div>into the upstream kernel (which will probably be Christoph Hellwig's</div>
<div>preference regardless of what we change) then this reorg won't have</div>
<div>broken the whole tree or current development process, with the</div>
<div>exception of years of muscle-memory for typing the old pathnames.</div>
<div>Maybe symlinks could be used to ease the transition?</div>
<div><br>
</div>
<div>Cheers, Andreas</div>
<div><br>
<blockquote type="cite">
<div>
<div>[II] Get fs/ and net/ to compile on a mainline kernel<br>
<br>
Once the compatibility code is isolated, we must get fs/ and net/<br>
to compile on a mainline kernel - without any configuration or<br>
lustre_compat/ layer.<br>
<br>
We would validate this by adding build validation to each patch<br>
submitted to Gerrit. The kernel version would be pinned (similar<br>
to how we pin ZFS version) and we'd periodically update it and fix<br>
any new build failures.<br>
<br>
Once this is achieved, we'll have a native Linux client/server<br>
that can be run on older distros via a compatibility layer.<br>
<br>
[III] Move fs/ and net/ to a separate kernel tree<br>
<br>
Transition to maintaining fs/ and net/ as a series on patches<br>
on top of a mainline kernel release. At this point, we'll generating<br>
patches to mainline Linux while retaining the ability to support<br>
older distro kernels via lustre_compat/. Similar to the previous<br>
step, we periodically rebase our Lustre patch series - fixing<br>
lustre_compat/ as needed.<br>
<br>
This is the only step that requires a change the Lustre development<br>
workflow - patches would have to be split and sent to two<br>
different repos. We can delay this step until we have some<br>
confidence that Lustre has a path to be accepted to mainline.<br>
<br>
[IV] Submit the patch series for inclusion<br>
<br>
Once we are comfortable with the above process, we can submit the<br>
initial patches to add Lustre support to the kernel. Our normal<br>
development flow will generate a batch of patches to be submitted<br>
during each merge window. After the merge window, we can focus<br>
on testing and making sure that our backport to older distro<br>
kernels it still working.<br>
<br>
FAQ:<br>
<br>
Q: Who will actually run the Lustre code in mainline Linux?<br>
A: Everyone. Releases for older distros will be a combination<br>
of the upstream Lustre combined with lustre_compat/ and<br>
whatever stuff the kernel won't allow (like GPUDirect).<br>
<br>
Q: What does a Lustre release look like?<br>
A: We can generate a tarball by combining an upstream Lustre<br>
release from mainline along with lustre_compat/ and the<br>
userspace stuff. Vendors and third-parties can base<br>
their versions of Lustre on those tarballs. Every time a<br>
new kernel releases - a new Lustre release tarball will<br>
be created. LTS releases can center around the LTS kernel<br>
releases.<br>
<br>
Q: How will we validate that fs/ and net/ build on mainline?<br>
A: It would probably be easiest to create a minimalist mainline<br>
kernel build in Jenkins. This would allow us to reuse most<br>
of the existing lbuild scripting. The build would be<br>
non-enforced at first. Testing would remain on distro<br>
kernels, since most people use those.<br>
<br>
Q: Will you create a wiki project tracking page for upstreaming<br>
Lustre?<br>
A: Yes<br>
<br>
Q: Does anyone else have a similar model? Does this even work?<br>
A: AMD GPU seems to have a similar approach, at least [1]. I'm<br>
looking to get more feedback of LSF. We should talk to other<br>
developers working in a model similar to this.<br>
<br>
This is still a high level sketch, but I think this is a feasible<br>
path to upstreaming Lustre. We need to define a clear roadmap<br>
with tangible milestones to have a hope of upstreaming working.<br>
<br>
But it's important that we don't disrupt developers established<br>
workflows. We don't want to complicate contributing to Lustre<br>
and we don't want to discourage people from contributing their<br>
changes upstream.<br>
<br>
Please give me any feedback or criticisms on this proposal. If we<br>
think this is workable, I'm going to create a wiki project page for<br>
this and attach it to the LSF/MM email.<br>
<br>
[1] AMD GPU DKMS: https://github.com/geohot/amdgpu-dkms<br>
<br>
--------------------------------------------------------------------------------<br>
<br>
Lustre is a high-performance parallel filesystem used for HPC<br>
and AI/ML compute clusters available under GPLv2. Lustre is<br>
currently used by 65% of the Top-500 (9 of Top-10) systems in<br>
HPC [7]. Outside of HPC, Lustre is used by many of the largest<br>
AI/ML clusters in the world, and is commercially supported by<br>
numerous vendors and cloud service providers [1].<br>
<br>
After 21 years and an ill-fated stint in staging, Lustre is still<br>
maintained as an out-of-tree module [6]. The previous upstreaming<br>
effort suffered from a lack of developer focus and user adoption,<br>
which eventually led to Lustre being removed from staging<br>
altogether [2].<br>
<br>
However, the work to improve Lustre has continued regardless. In<br>
the intervening years, the code improvements that previously<br>
prevented a return to mainline have been steadily progressing. At<br>
least 25% of patches accepted for Lustre 2.16 were related to the<br>
upstreaming effort [3]. And all of the remaining work is<br>
in-flight [4][5]. Our eventual goal is to a get both the Lustre<br>
client and server (on ext4) along with at least TCP/IP networking to<br>
an acceptable quality before submitting to mainline. The remaining<br>
network support would follow soon afterwards.<br>
<br>
I propose to discuss:<br>
<br>
- As we alter our development model to support upstream development,<br>
what is a sufficient demonstration of commitment that our model works? [8]<br>
- Should the client and server be submitted together? Or split?<br>
- Expectations for a new filesystem to be accepted to mainline<br>
- How to manage inclusion of a large code base (the client alone is<br>
200kLoC) without increasing the burden on fs/net maintainers<br>
<br>
Lustre has already received a plethora of feedback in the past.<br>
While much of that has been addressed since - the kernel is a<br>
moving target. Several filesystems have been merged (or removed)<br>
since Lustre left staging. We're aiming to avoid the mistakes of<br>
the past and hope to address as many concerns as possible before<br>
submitting for inclusion.<br>
<br>
Thanks!<br>
<br>
Timothy Day (Amazon Web Services - AWS)<br>
James Simmons (Oak Ridge National Labs - ORNL)<br>
<br>
[1] Wikipedia: https://en.wikipedia.org/wiki/Lustre_(file_system)#Commercial_technical_support<br>
[2] Kicked out of staging: https://lwn.net/Articles/756565/<br>
[3] This is a heuristic, based on the combined commit counts of<br>
ORNL, Aeon, SuSe, and AWS - which have been primarily working<br>
on upstreaming issues: https://youtu.be/BE--ySVQb2M?si=YMHitJfcE4ASWQcE&t=960<br>
[4] LUG24 Upstreaming Update: https://www.depts.ttu.edu/hpcc/events/LUG24/slides/Day1/LUG_2024_Talk_02-Native_Linux_client_status.pdf<br>
[5] Lustre Jira Upstream Progress: TODO<br>
[6] Out-of-tree codebase: https://git.whamcloud.com/?p=fs/lustre-release.git;a=tree<br>
[7] I couldn't find a link to this? TODO<br>
[8] Include a link to a project wiki: TODO<br>
<br>
<br>
_______________________________________________<br>
lustre-devel mailing list<br>
lustre-devel@lists.lustre.org<br>
http://lists.lustre.org/listinfo.cgi/lustre-devel-lustre.org<br>
</div>
</div>
</blockquote>
</div>
<br>
<div>
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;">
<div>Cheers, Andreas</div>
<div>—</div>
<div>Andreas Dilger</div>
<div>Lustre Principal Architect</div>
<div>Whamcloud/DDN</div>
</div>
<br class="Apple-interchange-newline">
</div>
<br class="Apple-interchange-newline">
<br class="Apple-interchange-newline">
</div>
<br>
</body>
</html>