<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Aptos;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
font-size:11.0pt;
font-family:"Aptos",sans-serif;
mso-ligatures:standardcontextual;
mso-fareast-language:EN-US;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#467886;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Aptos",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:11.0pt;
mso-fareast-language:EN-US;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="en-NL" link="#467886" vlink="#96607D" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal"><span lang="EN-US">Dear Lustre community,<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="en-NL">We have a FSx for Lustre configuration at AWS (FSx Persistant SSD 1.2TB 250mb/s/TiB; FSx for Lustre server version 2.12). From an EC2 with Ubuntu that file system is mounted using the Lustre Client Modules. The version
of these lustre clients depends on the Linux kernel.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="en-NL"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="en-NL">On the file system we have compressed data tables (80000 row x 1440 col and transposed as well). These are stored using the fst library/package for R (<a href="https://www.fstpackage.org/" target="_blank">https://www.fstpackage.org/</a>). The
data are stored columnwise in a serialised manner. The benefit is, one can read a set of columns (or rows) without having to read the whole file (similar to Parquet files). It is (one of) the fastest way to read/write data from R.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="en-NL"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="en-NL">Recently we discovered that on a newly configured EC2 reading data from these files on the Lustre file system is a lot slower than on an older EC2.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="en-NL">After some debugging it was found that an EC2 with Ubuntu 18.04.6 LTS and kernel 5.4.0-1083-aws using the Lustre Client 2.10.8 has the same fast performance as expected (same as the older EC2). However, upgrading the
Lustre Client to 2.12.8 (nothing else is different... same machine) results in poor performance. The job to test the speed is reading 4 columns from 180 files containing a data table as described above. This takes about 5 seconds when it is fast, but slows
down to 20 seconds in the slow case.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="en-NL"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="en-NL">In addition, just reading one whole table (one file) using read_fst takes about: 1 - 2 seconds with Lustre Client 2.10.8 20 - 22 seconds with Lustre Client 2.12.8 1-2 seconds with Lustre Client 2.15.4 (on Ubuntu 22...
another EC2).<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="en-NL"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="en-NL">Reading the files immediately again using the Lustre Client 2.12.8 improves the performance back to 1 - 2 seconds. So, when they are cached (somewhere), the performance is OK, but a cold read is very slow. In contrast,
the other two (2.10.8 and 2.15.4) are already very fast reading the files the first time.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="en-NL"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="en-NL">I would use 2.15.4 which is the latest supported version using the highest supported Ubuntu version (22), but unfortunately the performance of 2.15.4 is similar to that of 2.12.8 in the first test (reading 4 columns from
180 files takes about 20 seconds instead of around 5). As a result, we're stuck with Ubuntu 18 and kernel 5.4.0 which is the latest combination that still supports Lustre Client 2.10.8 (which is fast in all cases).<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="en-NL"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="en-NL">The test has been repeated a lot of times at different times to rule out caching behaviour.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="en-NL"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="en-NL">What could be the reason for these large performance differences (4x to 10x slower)? Are there perhaps some parameter settings different between the Lustre Client versions? Can those be adjusted?<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">I also posted this question on the AWS re:Post “forum”:
<a href="https://repost.aws/questions/QUCiF-XpFaS0al162IYKXg1w/fsx-for-lustre-client-2-12-very-slow-compared-to-2-10">
https://repost.aws/questions/QUCiF-XpFaS0al162IYKXg1w/fsx-for-lustre-client-2-12-very-slow-compared-to-2-10</a><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<table class="MsoNormalTable" border="0" cellspacing="0" cellpadding="0" width="748" style="width:560.65pt;border-collapse:collapse">
<tbody>
<tr style="height:20.8pt">
<td width="748" colspan="5" valign="top" style="width:560.65pt;padding:0cm 0cm 0cm 0cm;height:20.8pt">
<p class="MsoNormal"><span style="color:#131D31;mso-fareast-language:NL">Kind regards,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#131D31;mso-fareast-language:NL"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="NL" style="color:#131D31;mso-fareast-language:NL">Anton Wijbenga</span><span lang="NL" style="font-size:12.0pt;color:#131D31;mso-fareast-language:NL"><o:p></o:p></span></p>
</td>
</tr>
<tr style="height:16.65pt">
<td width="170" rowspan="4" valign="top" style="width:127.6pt;padding:0cm 0cm 0cm 0cm;height:16.65pt">
<p class="MsoNormal"><span style="font-size:12.0pt;color:#131D31;mso-fareast-language:NL"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:12.0pt;color:#131D31;mso-fareast-language:NL"><img border="0" width="155" height="90" style="width:1.6145in;height:.9375in" id="Picture_x0020_7" src="cid:image001.png@01DAFAEF.4B4DD250" alt="A white and orange logo
Description automatically generated"><o:p></o:p></span></p>
</td>
<td width="577" colspan="4" style="width:433.05pt;padding:0cm 0cm 0cm 0cm;height:16.65pt">
</td>
</tr>
<tr style="height:25.5pt">
<td width="38" valign="top" style="width:1.0cm;padding:0cm 0cm 0cm 0cm;height:25.5pt">
<p class="MsoNormal"><span style="font-size:12.0pt;color:#131D31;mso-fareast-language:NL"><img border="0" width="22" height="22" style="width:.2291in;height:.2291in" id="Picture_x0020_8" src="cid:image002.png@01DAFAEF.4B4DD250"><o:p></o:p></span></p>
</td>
<td width="189" valign="top" style="width:5.0cm;padding:0cm 0cm 0cm 0cm;height:25.5pt">
<p class="MsoNormal"><span style="font-size:12.0pt;mso-fareast-language:#2000"><a href="+31614868667"><span style="font-size:11.0pt;color:#131D31;mso-fareast-language:NL;text-decoration:none">+31 6 14 86 86 67</span></a></span><span style="color:#131D31;mso-fareast-language:NL"><o:p></o:p></span></p>
</td>
<td width="38" valign="top" style="width:1.0cm;padding:0cm 0cm 0cm 0cm;height:25.5pt">
<p class="MsoNormal"><span style="color:#131D31;mso-fareast-language:NL"><img border="0" width="22" height="26" style="width:.2291in;height:.2708in" id="Picture_x0020_9" src="cid:image003.png@01DAFAEF.4B4DD250"><o:p></o:p></span></p>
</td>
<td width="313" valign="top" style="width:234.6pt;padding:0cm 0cm 0cm 0cm;height:25.5pt">
<p class="MsoNormal"><span style="font-size:12.0pt;mso-fareast-language:#2000"><a href="https://maps.app.goo.gl/uLNCBe5z6FQVMLfK6"><span style="font-size:11.0pt;color:#131D31;mso-fareast-language:NL;text-decoration:none">Van Deventerlaan 20</span></a></span><span lang="NL" style="mso-fareast-language:#2000">,
</span><span style="font-size:12.0pt;mso-fareast-language:#2000"><a href="https://maps.app.goo.gl/uLNCBe5z6FQVMLfK6"><span style="font-size:11.0pt;color:#131D31;mso-fareast-language:NL;text-decoration:none">3528 AE, Utrecht</span></a></span><span style="mso-fareast-language:NL"><o:p></o:p></span></p>
</td>
</tr>
<tr style="height:25.5pt">
<td width="38" valign="top" style="width:1.0cm;padding:0cm 0cm 0cm 0cm;height:25.5pt">
<p class="MsoNormal"><span style="font-size:12.0pt;color:#131D31;mso-fareast-language:NL"><img border="0" width="22" height="22" style="width:.2291in;height:.2291in" id="Picture_x0020_10" src="cid:image004.png@01DAFAEF.4B4DD250"><o:p></o:p></span></p>
</td>
<td width="189" valign="top" style="width:5.0cm;padding:0cm 0cm 0cm 0cm;height:25.5pt">
<p class="MsoNormal"><span style="font-size:12.0pt;mso-fareast-language:#2000"><a href="https://www.linkedin.com/in/anton-wijbenga-b9798623/"><span style="font-size:11.0pt;color:#131D31;mso-fareast-language:NL;text-decoration:none">/a</span><span style="font-size:11.0pt;color:#131D31;text-decoration:none">nton</span><span style="font-size:11.0pt;color:#131D31;mso-fareast-language:NL;text-decoration:none">-w</span><span style="font-size:11.0pt;color:#131D31;text-decoration:none">ijbenga</span></a></span><span style="color:#131D31;mso-fareast-language:NL"><o:p></o:p></span></p>
</td>
<td width="38" valign="top" style="width:1.0cm;padding:0cm 0cm 0cm 0cm;height:25.5pt">
<p class="MsoNormal"><span style="color:#131D31;mso-fareast-language:NL"><img border="0" width="22" height="22" style="width:.2291in;height:.2291in" id="Picture_x0020_11" src="cid:image005.png@01DAFAEF.4B4DD250"><o:p></o:p></span></p>
</td>
<td width="313" valign="top" style="width:234.6pt;padding:0cm 0cm 0cm 0cm;height:25.5pt">
<p class="MsoNormal"><span style="font-size:12.0pt;mso-fareast-language:#2000"><a href="https://www.maptm.nl/"><span style="font-size:11.0pt;color:#131D31;mso-fareast-language:NL;text-decoration:none">https://www.maptm.nl</span></a></span><span style="color:#131D31;mso-fareast-language:NL"><o:p></o:p></span></p>
</td>
</tr>
<tr style="height:19.75pt">
<td width="38" valign="top" style="width:1.0cm;padding:0cm 0cm 0cm 0cm;height:19.75pt">
<p class="MsoNormal"><span style="font-size:12.0pt;color:#131D31;mso-fareast-language:NL"><img border="0" width="22" height="22" style="width:.2291in;height:.2291in" id="Picture_x0020_6" src="cid:image006.png@01DAFAEF.4B4DD250"><o:p></o:p></span></p>
</td>
<td width="540" colspan="3" valign="top" style="width:404.7pt;padding:0cm 0cm 0cm 0cm;height:19.75pt">
<p class="MsoNormal"><span style="font-size:12.0pt;mso-fareast-language:#2000"><a href="mailto:anton.wijbenga@maptm.nl"><span style="font-size:11.0pt;color:#131D31;mso-fareast-language:NL;text-decoration:none">anton.wijbenga@maptm.nl</span></a></span><span lang="NL" style="color:#131D31;mso-fareast-language:NL"><o:p></o:p></span></p>
</td>
</tr>
</tbody>
</table>
<p class="MsoNormal"><b><span lang="NL" style="color:#002060;mso-fareast-language:#2000">Not available on wednesdays<o:p></o:p></span></b></p>
<p class="MsoNormal"><span lang="NL"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="NL"><o:p> </o:p></span></p>
</div>
</body>
</html>