[Lustre-discuss] Lustre 2.x support on vanilla kernels

[Andrew Savchenko]

Hello Andreas,

On Tue, 5 Jun 2012 09:16:51 -0600 Andreas Dilger wrote:
> On 2012-06-05, at 7:11 AM, Andrew Savchenko wrote:
> > we are building an HPC IB-backed cluster and are considering to
> > use Lustre 2.x for our distributed and parallel cluster storage.
> > 
> > What bothers me that only RedHat-based systems are claimed to be
> > supported. We will likely use Gentoo on our computing nodes, this
> > effectively means vanilla kernels. On Lustre servers we may install
> > RedHat, this is an acceptable, but not a desirable solution.
> 
> On servers, the reason that we only support vendor kernels is because
> there are patches needed for ext4 to support extra features that we
> have implemented.  That means it takes some non-trivial effort to add
> support for a different kernel.  With Lustre 2.4 + ZFS it will be
> possible to run Lustre servers completely without patches.

That's a good news to hear. Is there any time schedule for future
Lustre releases? On lustre.org I can find only up to version 2.0.
 
> Since the Lustre servers are intended to be separate from the rest of
> the compute cluster, it is not usually a problem to run RHEL on those
> nodes, 

There may be some problems with running too old kernel on too new
hardware, but I hope we'll find a fay to handle this.

> and whatever distro you want on the client nodes (assuming you
> want to build your own packages for them).

It shouldn't be a problem if it will not require a heavy kernel code
patching.

> > On clients we will definitely use 3.x kernels, so I'd like to know
> > a current status of Lustre client support. They are patchless to my
> > knowledge, so should work on any recent kernel, but your mileage may
> > vary.
> 
> There are continuous changes to the kernel APIs that filesystems need
> to use to work on Linux.  In 2.6.38 there was a major change to how
> clients interact with the dcache, which will be included in Lustre
> clients for the 2.3 release.  There are some patches for making the
> Lustre client work with 3.1 and 3.3 kernels, but those patches are not
> yet landed.
> 
> Having interested users test newer kernels and report back their
> experience to the list helps everyone involved.  It allows users
> such as yourself to participate in the development of Lustre, and
> allows maintainers to have some assurance that the patches for those
> kernels actually allow Lustre to work on the newer kernels, and not
> just compile without errors.
> 
> As you can imagine, we cannot possibly build and test every combination
> of Linux kernels and distros available.  We do include tools to build
> packages for Debian/Ubuntu based systems, and do regular builds for
> Ubuntu 10, but as yet there has been little interest in Gentoo.  If
> this is something you feel strongly about, possibly you could contribute
> Makefile rules/scripts to build Gentoo packages to help others also
> use Lustre with Gentoo.
 
Okay, let's start from a Gentoo client. I have some experience in
package creation and support, so this shouldn't be that hard if it
comes to build subsystem and packaging. But I'm new to Lustre and if
it will come to serious code changes, I will need a help from
developers.

But at this very moment I have troubles with testing grounds. Our
cluster is in the early construction stage now, so I will not be able
to test a package on the actual hardware. Estimated completion time
to the point when I'll be able to test Lustre is 6 +5/-3 months. I
can setup some virtual test environment, but it will have a poor
interconnect (no more than 1 Gb/s ethernet) and a few hosts, so
emulation may be inadequate.

Anyway I need some time to study Lustre docs in order to understand
its internals and how to set up it properly. When I'll have a
prepared test environment I'll let you know (this may take some
considerable time as I have other duties).

As for now, should I stick to something like Lustre-2.3 branch,
RHEL-5.4 on the server and 2.6.38-3.0 kernels on the clients?

Best regards,
Andrew Savchenko
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: not available
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20120606/d0804006/attachment.pgp>