[lustre-devel] [LSF/MM/BPF TOPIC] [DRAFT] Lustre client upstreaming

[NeilBrown]

On Sat, 18 Jan 2025, Oleg Drokin wrote:
> On Sat, 2025-01-18 at 11:45 +1100, NeilBrown wrote:
> > We need to demonstrate a process for, and commitment to, moving away
> > from the dual-tree model.  We need patches to those parts of Lustre
> > that are upstream to land in upstream first (mostly).
> 
> I think this is not very realistic.
> Large chunk (100%?) of users do not run not only the latest kernel
> release, they don't run the latest LTS either.

Are you referring to lustre users or all Linux users?
If the latter, then xfs etc face the same problem and seem to manage.
If lustre users: they can't because the latest kernel doesn't include
lustre.  Maybe you are seeing a chicken-and-egg problem?


> 
> When we were in staging last this manifested in random patches being
> landed and breaking the client completely and nobody noticing for
> months.

The staging exercise was a mess in various ways and suffered a lot of
problems that we wouldn't expect if we did the upstreaming properly.
If net/lnet and fs/lustre were managed by lustre developers rather than
by GregKH, then the "random patches" would be avoided and we could, as
you say below and as all other fs teams do, run tests before committing
to patches.

> 
> Of course some automatic infrastructure could be built up to make it
> somewhat better, but it does not remove the problem of "nobody would
> run this mainline tree", I am afraid.

We've never had a credible lustre in a mainline tree, so we cannot know
how many people would use it.  Importantly developers would use it
because that is where development would happen.

> 
> It does not hep that there are what 3? 4? trees, not "dual-tree" by any
> stretch of imagination.
> 
> There's DDN/whamcloud (that's really two trees), there's HPE, LLNL
> keeps their fork still I think (thought it's mostly backports?). There
> are likely others I am less exposed to.

"dual-tree" maybe isn't the best way of describing what was wrong with
the previous approach.  "upstream-first" is one way of describing how it
should be run, though that needs to be in understood correctly.

Patches should always flow upstream first, then flow downstream into
distro.  So I write a patch in my own devel tree.  I post it or submit a
pull request and eventually it is accepted into the maintainers
"testing" tree (upsream from me).  There it gets more testing and moves
to the maintainers "next" tree from which it is pulled into linux-next
for integration testing.  Then it goes upstream to Linus (possibly
through an intermediary).  From Linus it goes to -stable and to various
distros etc.  Individual patches are selected for further backporting to
all sorts of different LTS tree.

Occasionally there are short-cuts.  I might submit a patch from my tree
to a SUSE kernel before it is accepted upstream, or maybe even before it
is sent if it is urgent.  But these are not the norm.

But you know all this I expect.  It isn't about the total number of
trees. It is about the flow of patches which must all flow through Linus.
And developers must develop against current linus, or something very
close to that.  Developing against an older kernel is simply making more
work for yourself.

> 
> Sure, only one of those trees is considered "community Lustre", but if
> it will detach too much from what majority of developers really runs
> and gets paid to do - the "community Lustre" contributions probably
> would diminish greatly, I am afraid.
> 
> The past situation of "oh, this new enterprise linux comes with a
> community lustre version, so the first step to get something usable is
> to rip it entirely off and then apply the new good version" is not
> exactly desirable either I am afraid.

Obviously that is not what we want, and clearly people aren't tempted to
do that with any of FS so why do you think it will happen with lustre?
The "new good version" will simply be a few patches on top of whatever
kernel you have.  Hopefully the distributor of that kernel will have
applied those already if any of their customers care about the filesystem.

> > 
> > We need to quickly reach a point where a lustre release is:
> > 
> >  - a verbatim copy of relevant files from a chosen upstream release,
> >    or just a dependency on that kernel source.
> >  - a bunch of extra files that might one day go upstream: server code
> >    and LNet protocol code
> >  - a *few* patches to integrate that code
> >  - some number of patches which have since gone upstream - bugfixes
> > etc.
> >  - libcfs which contains a compat layer for older kernels.
> >  - user-space code, documentation, test scripts, etc for which there
> >    is no expectation of upstreaming to linux kernel.
> 
> All these sound like an awful lot of dedicated developer-hours.
> 
> > Maybe the question for LSF is : what is a sufficient demonstration of
> > commitment?
> > 
> > The big question for us is : how are we going to transition our
> > infrastructure to this model?
> 
> and who would pay for it.

Obviously there will be a cost to transition.  It seems someone is
already willing to pay some of that because patches have been landing
which are only there to make the ultimate transition easier.  Why do you
think that will stop.

Once the transition completes there will still be process difficulties,
but there are plenty of of process difficulties now (gerrit: how do I
hate thee, let me count the ways...) but people seem to simply include
that in the cost of doing business.

> 
> This in the end was the downfall of the previous attempt. There never
> was any serious funding behind the effort so it became an afterthought
> for most.

I don't think funding is the big problem.  I think it is "buy-in".
Individual people in positions of power - such as yourself - need to see
the value and be willing to change they way they work.  If you,
personally, are not willing to change then there is no point even
talking about this any more.

NeilBrown