[lustre-devel] [LSF/MM/BPF TOPIC] [DRAFT] Lustre client upstreaming

[Alexey Lyahkov]

> On 24 Jan 2025, at 20:06, Day, Timothy <timday at amazon.com> wrote:
> 
>> 
>> 
>>> There is an issue if distros don't want to update their clients.
>> 
>> It is not “if don’t want update”, Ubuntu don’t update own lustre code in past.
>> I don’t expect it will be changed. Because distro owner will needs to hire more developers to have extra support.
>> But have no money from it.
>> 
>> 
>> 
>> 
>>> That's why we'll
>>> still support running latest Lustre on older distros. Specifically, it'll be the Lustre
>>> code from a mainline kernel combined with our lustre_compat/ compatibility
>>> code. So normal Lustre releases will be derived directly from the in-tree kernel
>>> code. This provides a path for vendors to deploy bug fixes, custom features, and
>>> allows users to optionally run the latest and greatest Lustre code.
>> 
>> And OOPS. Both codes (in-kernel and out-of-tree) have a same sort of defines in config.h which have conflicts with building for out-of-free Lustre.
>> Some examples for MOFED hacks to solve same problem you can see in the o2iblnd:
>>>>> 
>> #if defined(EXTERNAL_OFED_BUILD) && !defined(HAVE_OFED_IB_DMA_MAP_SG_SANE)
>> #undef CONFIG_INFINIBAND_VIRT_DMA
>> #endif
>>>>> 
>> As I remember this problem broke an ability to build a lustre as out-of-tree kernel on the ubuntu 18.06 with lustre in staging/.
> 
> I think we should be able to validate the Lustre still builds as an
> out-of-tree module by re-using a lot of the testing we already
> do today in Jenkins/Maloo.
Yes. Me do. But it needs many extra resources. Did Amazon ready to provide such HW resources for it?
Or who will be pay for it? It’s cost of the moving to the kernel.


> All we'd need to do it kick off test/build
> sessions once the merge window closes. Based on the MOFED
> example you gave, it seems like this is solvable.
> 
Sure, All can be solved. But what are cost for this and cost for support these changes?
And next question - who will pay for this cost? Who will provide an HW for extra testing?
So second face of “no cost for kernel API changes” - it will be a problems with back porting these changes and extra testing.



>>> 
>>> [1] Lustre changelog: https://git.whamcloud.com/?p=fs/lustre-release.git;a=blob_plain;f=lustre/ChangeLog;hb=HEAD <https://git.whamcloud.com/?p=fs/lustre-release.git;a=blob_plain;f=lustre/ChangeLog;hb=HEAD>
>>> 
>>>> It this is not enough - lets one more. Kernel API isn’t stable enough - so large number resources will be needs spent to solve each kernel change in lustre. Currently, it’s in the background and don’t interrupt primary work for supporting and development a new Lustre features.
>>>> 
>>>> So that is problems for Lustre world - what is benefits?
>>> 
>>> By upstreaming Lustre, we'll benefit from developers updating the kernel
>>> API "for free".
>> It’s not a “for free” - did you really think any of kernel developers have a cluster to run lustre client to test a changes?
>> I think not, so testing will be “just compile with proposed/default config”.
>> Once it will be lack of proper testing (don’t remember it’s full run for lustre test suite ~12-24h) - lustre developers needs review each change in the lustre code.
> 
> That's why a put "for free" in quotes. We need to make it easier for
> upstream developers to test their changes so they don't completely
> break Lustre.
Ah.. so Lustre will have a vote to stop any landing in kernel until Lustre testing will done? 
Did you understand how many tests will be needs to run?
Full testing needs a ~24h of run time for single node.
How many HW resources Amazon may share to run these tests?
Did you understand - if lustre code changed by someone in upstream that change can’t be backported to the main tree because compatibility code can’t be handle it.
Sometimes needs to stay with old behavior which re-implemented with new kernel code.


> If we upstream the client and server concurrently, we
> can implement xfstests support [1]. This would provide at least basic
> validation. NFS does something similar. We could even copy over a
> subset of Lustre specific tests from sanity.sh into xfstests.
NFS server don’t have a many Lustre features and it don’t expect to be build as out-of-tree module for different kernels.

> 
> It's not perfect - but it'd be a much better situation compared to the
> previous attempt in staging.
> 
> [1] https://github.com/kdave/xfstests
I’m sorry. This is very simple test cases. Lustre much complex FS.


> 
>> And it needs to back port all these changes in the out-of-free version. Once lustre part needs changes also.
>> Best example is ‘folio’ - this need changes for both sides.
> 
> If the out-of-tree version is derived from the in-tree version of
> Lustre - I don't think the backporting will be that burdensome.
> We're essentially do the same work now, but in reverse. Instead
> of porting an upstream driver to old kernels, we are porting an
> older driver to new kernels.
Except some notes.
1) lustre release cycles. Now it’s not a refined with kernel one. None situation when senior developer should stop own work to review kernel changes because it might affects a lustre stability. But with lustre-in-kernel - any change in kernel affects a lustre - need reviewed / tested ungent. 
So extra developers positions/HW needs.

2) no problem with have a custom patches in upstream.
Someone may think something needs cleaned in the lustre code and this patch will accepted.
So it generate a conflict in code changed in the same place for lustre main repository. 
Moving a whole lustre development in the kernel not possible because no server part, but servers have an “client” code on own side, sometimes.

Not so small cost for “updates for free” ?

> 
>>> We Lustre was in staging/, there wasn't as much obligation
>>> to keep Lustre in a working state. But if we get Lustre merged properly, 
>>> developer will not be able to merge changes that break Lustre. So we'll
>>> get support for the latest and greatest kernels with less effort. That's one
>>> of the main benefits of this effort.
>>> 
>>> 
>>> We also get benefit from more say over the future of the kernel. A lot
>>> of difficulty with updating Lustre for new kernels comes when upstream
>>> kernel developers lock down symbols or features to in-tree modules. This
>>> could get even worse in the future, with stuff like symbol namespaces get
>>> more use [2].
>>> 
>>> Even if most users use the out-of-tree backported-from-mainline-Linux
>>> Lustre release, I think we'll still be in a stronger position after
>>> upstreaming.
>>> 
>>> [2] https://lwn.net/Articles/760045/ <https://lwn.net/Articles/760045/>
>>> 
> 
>> 
>> PS. Lustre able to run a server with very very light modified ext4 code. Mostly some exports / callbacks from core.
>> 
> 
> That's good to hear - I think that'll make it easier to convince
> upstream to accept the ext4 patches needed to run the server.
> 
>> 
>> Alex
>> 
> 
> Tim Day
>