[lustre-discuss] Lustre client/server versions/compatibility

[Andreas Dilger]

Cory,
I think the protocol change for object deletion was done as part of the OSD API change (to add ZFS support) in 2.4, and LU-5814 doesn't seem to be related to that.  Or maybe the removal of the interoperability support for pre-2.4 clients was removed as part of that change (which is marked landed in 2.8.0)?

In any case, it is entirely possible that there is some change that can cause interop problems, so definitely upgrading is preferred.

Cheers, Andreas



On Oct 20, 2019, at 14:05, Cory Spitz <spitzcor at cray.com<mailto:spitzcor at cray.com>> wrote:

Hi, William.

You should plan to update your 2.4.x and 2.5.x clients.  There is at least one known compatibility problem with clients that old.  The code associated with LU-5814 changed the protocol for deleting objects.  You’ll find that files deleted from those old clients will leak objects on the OSTs.  Sure, you can live with it and lfsck can clean things up for you, but because of this issue those old clients don’t seem to be used anywhere in production along with modern servers and there may be other problems to worry about.

-Cory

--


On 10/17/19, 12:12 AM, "lustre-discuss on behalf of Andreas Dilger" <lustre-discuss-bounces at lists.lustre.org<mailto:lustre-discuss-bounces at lists.lustre.org> on behalf of adilger at whamcloud.com<mailto:adilger at whamcloud.com>> wrote:

On Oct 17, 2019, at 00:34, William D. Colburn <wcolburn at nrao.edu<mailto:wcolburn at nrao.edu>> wrote:

We are moving closer to migrating our lustre 2.5.5 servers to lustre
2.12.2.  But we still have a collection of clients.  We have a vague bit
of folklore, handed down word of mouth from the village elder, that if
we upgrade the server to 2.12.2 that the clients all have to be upgraded
to 2.7.something.  I've tried google, but all the searches I've tried
for compatibility all lead to information on what versions of lustre
run on what versions of linux, and not what versions of clients interact
with what versions of the server.

Is there a compatibility matrix for clients versus servers?

In theory, there is a very long window of compatibility between Lustre versions
between the client and server (much less leeway is allowed between servers).

Since each client is independent (they never communicate with each other, only
with the servers) this also provides a good deal of flexibility, unlike e.g. shared
storage filesystem like GPFS or GlusterFS or peer-peer filesystems.

The main reason there is not a large compatibility matrix is that this takes a lot
of effort to test various versions.  For releases we always test release N against
N-1 and the most recent LTS release (e.g. 2.12.3 is being tested against 2.11.0
and 2.10.8, and 2.13 is tested against 2.12) for both client and server versions.
This is already 4 sets of interop tests (new client/old server, old client/new server
for both), plus disk format upgrade tests, plus several different distros (RHEL7/8,
Ubuntu) and CPU architectures (x86, PPC64, Arm)...

So, for software we release for free we can't do exhaustive testing of all the older
release versions as well.  While client and server interop _should_ work between
all versions, there are occasional bugs, so we only report compatibility with versions
that we actually tested.  As shown with your list of client versions below (2.12.2 clients
with 2.5.5 servers), there is a pretty wide range of compatibility.  Each client and
server negotiate protocol compatibility at connection time, so if new features are
added they are only active if both systems support them. We are only just removing
some Lustre 1.8 and 2.1 and kernel 2.6.32 compatibility code from the 2.13 release,
which were first released about 8 years ago.

If you wanted, you could create a wiki.lustre.org<http://wiki.lustre.org/> page that included the various
versions that are actually tested, and allow users to fill in the blanks?  I've heard
several reports on the list of sites using 2.10.x or 2.12.x clients with 2.5.x servers,
but not so much the other way around.


Alternatively, does someone just know what version of the clients we
need to have everywhere for a 2.12.2 server?

Our clients include:
 2.4.3
 2.5.5
 2.10.1
 2.10.2
 2.10.4
 2.10.5
 2.10.6
 2.10.7
 2.10.8
 2.12.2

My recommendation would be to upgrade the 2.4 and 2.5 clients to 2.10.8 and then
you could run 2.12.2 servers.  If you really need to run older clients because of old
applications, you could try using 2.7.x on those clients and 2.10.8 servers.  You might
be able to use 2.7.x clients with 2.12 servers, but that isn't tested for those releases.
Alternately, run old apps in VMs/containers and use a newer kernel+Lustre underneath.

Cheers, Andreas
--
Andreas Dilger
Principal Lustre Architect
Whamcloud

Cheers, Andreas
--
Andreas Dilger
Principal Lustre Architect
Whamcloud






-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20191020/820d5af2/attachment-0001.html>