<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
Cory,
<div class="">I think the protocol change for object deletion was done as part of the OSD API change (to add ZFS support) in 2.4, and LU-5814 doesn't seem to be related to that. Or maybe the removal of the interoperability support for pre-2.4 clients was removed
as part of that change (which is marked landed in 2.8.0)?</div>
<div class=""><br class="">
</div>
<div class="">In any case, it is entirely possible that there is some change that can cause interop problems, so definitely upgrading is preferred.</div>
<div class=""><br class="">
</div>
<div class="">Cheers, Andreas</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
<div><br class="">
<blockquote type="cite" class="">
<div class="">On Oct 20, 2019, at 14:05, Cory Spitz <<a href="mailto:spitzcor@cray.com" class="">spitzcor@cray.com</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div class="WordSection1" style="page: WordSection1; caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;">
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
Hi, William.<o:p class=""></o:p></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""> </o:p></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
You should plan to update your 2.4.x and 2.5.x clients. There is at least one known compatibility problem with clients that old. The code associated with LU-5814 changed the protocol for deleting objects. You’ll find that files deleted from those old clients
will leak objects on the OSTs. Sure, you can live with it and lfsck can clean things up for you, but because of this issue those old clients don’t seem to be used anywhere in production along with modern servers and there may be other problems to worry about.<o:p class=""></o:p></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""> </o:p></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
-Cory<o:p class=""></o:p></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""> </o:p></div>
<div class="">
<div class="">
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
-- <o:p class=""></o:p></div>
</div>
</div>
</div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""> </o:p></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""> </o:p></div>
<div class="">
<div class="">
<div style="margin: 0in 0in 0.0001pt 0.5in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
On 10/17/19, 12:12 AM, "lustre-discuss on behalf of Andreas Dilger" <<a href="mailto:lustre-discuss-bounces@lists.lustre.org" style="color: purple; text-decoration: underline;" class="">lustre-discuss-bounces@lists.lustre.org</a><span class="Apple-converted-space"> </span>on
behalf of<span class="Apple-converted-space"> </span><a href="mailto:adilger@whamcloud.com" style="color: purple; text-decoration: underline;" class="">adilger@whamcloud.com</a>> wrote:<o:p class=""></o:p></div>
</div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt 0.5in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""> </o:p></div>
</div>
<div style="margin: 0in 0in 0.0001pt 0.5in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
On Oct 17, 2019, at 00:34, William D. Colburn <<a href="mailto:wcolburn@nrao.edu" style="color: purple; text-decoration: underline;" class="">wcolburn@nrao.edu</a>> wrote:<o:p class=""></o:p></div>
<div class="">
<blockquote style="margin-top: 5pt; margin-bottom: 5pt;" class="" type="cite">
<div style="margin: 0in 0in 0.0001pt 0.5in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""> </o:p></div>
<div class="">
<div class="">
<div style="margin: 0in 0in 0.0001pt 0.5in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
We are moving closer to migrating our lustre 2.5.5 servers to lustre<br class="">
2.12.2. But we still have a collection of clients. We have a vague bit<br class="">
of folklore, handed down word of mouth from the village elder, that if<br class="">
we upgrade the server to 2.12.2 that the clients all have to be upgraded<br class="">
to 2.7.something. I've tried google, but all the searches I've tried<br class="">
for compatibility all lead to information on what versions of lustre<br class="">
run on what versions of linux, and not what versions of clients interact<br class="">
with what versions of the server.<br class="">
<br class="">
Is there a compatibility matrix for clients versus servers?<o:p class=""></o:p></div>
</div>
</div>
</blockquote>
<div class="">
<div style="margin: 0in 0in 0.0001pt 0.5in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""> </o:p></div>
</div>
<div style="margin: 0in 0in 0.0001pt 0.5in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
In theory, there is a very long window of compatibility between Lustre versions<o:p class=""></o:p></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt 0.5in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
between the client and server (much less leeway is allowed between servers).<o:p class=""></o:p></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt 0.5in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""> </o:p></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt 0.5in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
Since each client is independent (they never communicate with each other, only<o:p class=""></o:p></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt 0.5in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
with the servers) this also provides a good deal of flexibility, unlike e.g. shared<o:p class=""></o:p></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt 0.5in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
storage filesystem like GPFS or GlusterFS or peer-peer filesystems.<o:p class=""></o:p></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt 0.5in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""> </o:p></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt 0.5in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
The main reason there is not a large compatibility matrix is that this takes a lot<o:p class=""></o:p></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt 0.5in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
of effort to test various versions. For releases we always test release N against<o:p class=""></o:p></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt 0.5in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
N-1 and the most recent LTS release (e.g. 2.12.3 is being tested against 2.11.0<o:p class=""></o:p></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt 0.5in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
and 2.10.8, and 2.13 is tested against 2.12) for both client and server versions.<o:p class=""></o:p></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt 0.5in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
This is already 4 sets of interop tests (new client/old server, old client/new server<o:p class=""></o:p></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt 0.5in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
for both), plus disk format upgrade tests, plus several different distros (RHEL7/8,<o:p class=""></o:p></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt 0.5in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
Ubuntu) and CPU architectures (x86, PPC64, Arm)...<o:p class=""></o:p></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt 0.5in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""> </o:p></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt 0.5in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
So, for software we release for free we can't do exhaustive testing of all the older<o:p class=""></o:p></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt 0.5in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
release versions as well. While client and server interop _should_ work between<o:p class=""></o:p></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt 0.5in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
all versions, there are occasional bugs, so we only report compatibility with versions<o:p class=""></o:p></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt 0.5in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
that we actually tested. As shown with your list of client versions below (2.12.2 clients<o:p class=""></o:p></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt 0.5in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
with 2.5.5 servers), there is a pretty wide range of compatibility. Each client and<o:p class=""></o:p></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt 0.5in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
server negotiate protocol compatibility at connection time, so if new features are<o:p class=""></o:p></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt 0.5in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
added they are only active if both systems support them. We are only just removing<o:p class=""></o:p></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt 0.5in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
some Lustre 1.8 and 2.1 and kernel 2.6.32 compatibility code from the 2.13 release,<o:p class=""></o:p></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt 0.5in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
which were first released about 8 years ago.<o:p class=""></o:p></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt 0.5in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""> </o:p></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt 0.5in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
If you wanted, you could create a<span class="Apple-converted-space"> </span><a href="http://wiki.lustre.org/" style="color: purple; text-decoration: underline;" class="">wiki.lustre.org</a><span class="Apple-converted-space"> </span>page that included the
various<o:p class=""></o:p></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt 0.5in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
versions that are actually tested, and allow users to fill in the blanks? I've heard<o:p class=""></o:p></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt 0.5in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
several reports on the list of sites using 2.10.x or 2.12.x clients with 2.5.x servers,<o:p class=""></o:p></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt 0.5in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
but not so much the other way around.<o:p class=""></o:p></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt 0.5in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<br class="">
<br class="">
<o:p class=""></o:p></div>
<blockquote style="margin-top: 5pt; margin-bottom: 5pt;" class="" type="cite">
<div class="">
<div class="">
<div style="margin: 0in 0in 0.0001pt 0.5in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
Alternatively, does someone just know what version of the clients we<br class="">
need to have everywhere for a 2.12.2 server?<br class="">
<br class="">
Our clients include:<br class="">
2.4.3<br class="">
2.5.5<br class="">
2.10.1<br class="">
2.10.2<br class="">
2.10.4<br class="">
2.10.5<br class="">
2.10.6<br class="">
2.10.7<br class="">
2.10.8<br class="">
2.12.2<o:p class=""></o:p></div>
</div>
</div>
</blockquote>
<div style="margin: 0in 0in 0.0001pt 0.5in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""> </o:p></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt 0.5in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
My recommendation would be to upgrade the 2.4 and 2.5 clients to 2.10.8 and then<o:p class=""></o:p></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt 0.5in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
you could run 2.12.2 servers. If you really need to run older clients because of old<o:p class=""></o:p></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt 0.5in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
applications, you could try using 2.7.x on those clients and 2.10.8 servers. You might<o:p class=""></o:p></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt 0.5in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
be able to use 2.7.x clients with 2.12 servers, but that isn't tested for those releases.<o:p class=""></o:p></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt 0.5in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
Alternately, run old apps in VMs/containers and use a newer kernel+Lustre underneath.<o:p class=""></o:p></div>
</div>
<div style="margin: 0in 0in 0.0001pt 0.5in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""> </o:p></div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div style="margin: 0in 0in 0.0001pt 0.5in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="" class="">Cheers, Andreas<o:p class=""></o:p></span></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt 0.5in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="" class="">--<o:p class=""></o:p></span></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt 0.5in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="" class="">Andreas Dilger<o:p class=""></o:p></span></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt 0.5in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="" class="">Principal Lustre Architect<o:p class=""></o:p></span></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt 0.5in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="" class="">Whamcloud</span></div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<br class="">
<div class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div>Cheers, Andreas</div>
<div>--</div>
<div>Andreas Dilger</div>
<div>Principal Lustre Architect</div>
<div>Whamcloud</div>
<div><br class="">
</div>
<div><br class="">
</div>
<div><br class="">
</div>
</div>
</div>
</div>
</div>
</div>
<br class="Apple-interchange-newline">
<br class="Apple-interchange-newline">
</div>
<br class="">
</div>
</body>
</html>