<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta name=Title content=""><meta name=Keywords content=""><meta http-equiv=Content-Type content="text/html; charset=utf-8"><meta name=Generator content="Microsoft Word 15 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:Arial;
panose-1:2 11 6 4 2 2 2 2 2 4;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:DengXian;
panose-1:2 1 6 0 3 1 1 1 1 1;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-reply;
font-family:"Arial",sans-serif;
color:windowtext;
letter-spacing:0pt;
font-emphasize:none;
font-weight:normal;
font-style:normal;}
span.msoIns
{mso-style-type:export-only;
mso-style-name:"";
text-decoration:underline;
color:teal;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style></head><body bgcolor=white lang=EN-US link=blue vlink=purple><div class=WordSection1><p class=MsoNormal><span style='font-family:"Arial",sans-serif'>Hi Andreas,<o:p></o:p></span></p><p class=MsoNormal><span style='font-family:"Arial",sans-serif'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-family:"Arial",sans-serif'>I have seen very similar errors in our 2.10.1 environment. Same errors from different clients to<o:p></o:p></span></p><p class=MsoNormal><span style='font-family:"Arial",sans-serif'>different OSS servers and OSTs. Our network is OPA and we are using the latest driver and<o:p></o:p></span></p><p class=MsoNormal><span style='font-family:"Arial",sans-serif'>firmware for all HFIs and switches (10.6).<o:p></o:p></span></p><p class=MsoNormal><span style='font-family:"Arial",sans-serif'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-family:"Arial",sans-serif'>Thanks,<o:p></o:p></span></p><p class=MsoNormal><span style='font-family:"Arial",sans-serif'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-family:"Arial",sans-serif'>Lixin Liu<o:p></o:p></span></p><p class=MsoNormal><span style='font-family:"Arial",sans-serif'>Compute Canada<o:p></o:p></span></p><p class=MsoNormal><span style='font-family:"Arial",sans-serif'><o:p> </o:p></span></p><div style='border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm'><p class=MsoNormal style='margin-left:36.0pt'><b><span style='font-size:12.0pt;color:black'>From: </span></b><span style='font-size:12.0pt;color:black'>lustre-discuss <lustre-discuss-bounces@lists.lustre.org> on behalf of "Dilger, Andreas" <andreas.dilger@intel.com><br><b>Date: </b>Saturday, December 9, 2017 at 9:07 PM<br><b>To: </b>Hans Henrik Happe <happe@nbi.dk><br><b>Cc: </b>"lustre-discuss@lists.lustre.org" <lustre-discuss@lists.lustre.org><br><b>Subject: </b>Re: [lustre-discuss] BAD CHECKSUM<o:p></o:p></span></p></div><div><p class=MsoNormal style='margin-left:36.0pt'><o:p> </o:p></p></div><p class=MsoNormal style='margin-left:36.0pt'>Based on the messages on the client, this isn’t related to mmap() or writes done by the client, since the data has the same checksum from before it was sent and after it got the checksum error returned from the server. That means the pages did not change on the client. <o:p></o:p></p><div><p class=MsoNormal style='margin-left:36.0pt'><o:p> </o:p></p></div><div><p class=MsoNormal style='margin-left:36.0pt'>Possible causes include the client network card, server network card, memory, or possibly the OFED driver? It could of course be something in Lustre/LNet, though we haven’t had any reports of anything similar. <o:p></o:p></p></div><div><p class=MsoNormal style='margin-left:36.0pt'><o:p> </o:p></p></div><div><p class=MsoNormal style='margin-left:36.0pt'>When the checksum code was first written, it was motivated by a faulty Ethernet NIC that had TCP checksum offload, but bad onboard cache, and the data was corrupted when copied onto the NIC but the TCP checksum was computed on the bad data and the checksum was “correct” when received by the server, so it didn’t cause TCP resends. <o:p></o:p></p></div><div><p class=MsoNormal style='margin-left:36.0pt'><o:p> </o:p></p></div><div><p class=MsoNormal style='margin-left:36.0pt'>Are you seeing this on multiple servers? The client log only shows one server, while the server log shows multiple clients. If it is only happening on one server it might point to hardware. <o:p></o:p></p></div><div><p class=MsoNormal style='margin-left:36.0pt'><o:p> </o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:0cm;margin-right:0cm;margin-bottom:12.0pt;margin-left:36.0pt'>Did you also upgrade the kernel and OFED at the same time as Lustre? You could try building Lustre 2.10.1 on the old 2.9.0 kernel and OFED to see if that works properly. <o:p></o:p></p><div id=AppleMailSignature><p class=MsoNormal style='margin-left:36.0pt'>Cheers, Andreas<o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:0cm;margin-right:0cm;margin-bottom:12.0pt;margin-left:36.0pt'><br>On Dec 9, 2017, at 11:09, Hans Henrik Happe <<a href="mailto:happe@nbi.dk">happe@nbi.dk</a>> wrote:<o:p></o:p></p></div><blockquote style='margin-top:5.0pt;margin-bottom:5.0pt'><div><p class=MsoNormal style='margin-left:36.0pt'><br><br>On 09-12-2017 18:57, Hans Henrik Happe wrote:<br><br><o:p></o:p></p><blockquote style='margin-top:5.0pt;margin-bottom:5.0pt'><p class=MsoNormal style='margin-left:36.0pt'>On 07-12-2017 21:36, Dilger, Andreas wrote:<o:p></o:p></p></blockquote><blockquote style='margin-top:5.0pt;margin-bottom:5.0pt'><blockquote style='margin-top:5.0pt;margin-bottom:5.0pt'><p class=MsoNormal style='margin-left:36.0pt'>On Dec 7, 2017, at 10:37, Hans Henrik Happe <<a href="mailto:happe@nbi.dk">happe@nbi.dk</a>> wrote:<o:p></o:p></p></blockquote></blockquote><blockquote style='margin-top:5.0pt;margin-bottom:5.0pt'><blockquote style='margin-top:5.0pt;margin-bottom:5.0pt'><blockquote style='margin-top:5.0pt;margin-bottom:5.0pt'><p class=MsoNormal style='margin-left:36.0pt'>Hi,<o:p></o:p></p></blockquote></blockquote></blockquote><blockquote style='margin-top:5.0pt;margin-bottom:5.0pt'><blockquote style='margin-top:5.0pt;margin-bottom:5.0pt'><blockquote style='margin-top:5.0pt;margin-bottom:5.0pt'><p class=MsoNormal style='margin-left:36.0pt'><o:p> </o:p></p></blockquote></blockquote></blockquote><blockquote style='margin-top:5.0pt;margin-bottom:5.0pt'><blockquote style='margin-top:5.0pt;margin-bottom:5.0pt'><blockquote style='margin-top:5.0pt;margin-bottom:5.0pt'><p class=MsoNormal style='margin-left:36.0pt'>Can an application cause BAD CHECKSUM errors in Lustre logs by somehow<o:p></o:p></p></blockquote></blockquote></blockquote><blockquote style='margin-top:5.0pt;margin-bottom:5.0pt'><blockquote style='margin-top:5.0pt;margin-bottom:5.0pt'><blockquote style='margin-top:5.0pt;margin-bottom:5.0pt'><p class=MsoNormal style='margin-left:36.0pt'>overwriting memory while being DMA'ed to network?<o:p></o:p></p></blockquote></blockquote></blockquote><blockquote style='margin-top:5.0pt;margin-bottom:5.0pt'><blockquote style='margin-top:5.0pt;margin-bottom:5.0pt'><blockquote style='margin-top:5.0pt;margin-bottom:5.0pt'><p class=MsoNormal style='margin-left:36.0pt'><o:p> </o:p></p></blockquote></blockquote></blockquote><blockquote style='margin-top:5.0pt;margin-bottom:5.0pt'><blockquote style='margin-top:5.0pt;margin-bottom:5.0pt'><blockquote style='margin-top:5.0pt;margin-bottom:5.0pt'><p class=MsoNormal style='margin-left:36.0pt'>After upgrading to 2.10.1 on the server side we started seeing this from<o:p></o:p></p></blockquote></blockquote></blockquote><blockquote style='margin-top:5.0pt;margin-bottom:5.0pt'><blockquote style='margin-top:5.0pt;margin-bottom:5.0pt'><blockquote style='margin-top:5.0pt;margin-bottom:5.0pt'><p class=MsoNormal style='margin-left:36.0pt'>a user's application (MPI I/O). Both 2.9.0 and 2.10.1 clients emit these<o:p></o:p></p></blockquote></blockquote></blockquote><blockquote style='margin-top:5.0pt;margin-bottom:5.0pt'><blockquote style='margin-top:5.0pt;margin-bottom:5.0pt'><blockquote style='margin-top:5.0pt;margin-bottom:5.0pt'><p class=MsoNormal style='margin-left:36.0pt'>errors. We have not yet established weather the application is doing<o:p></o:p></p></blockquote></blockquote></blockquote><blockquote style='margin-top:5.0pt;margin-bottom:5.0pt'><blockquote style='margin-top:5.0pt;margin-bottom:5.0pt'><blockquote style='margin-top:5.0pt;margin-bottom:5.0pt'><p class=MsoNormal style='margin-left:36.0pt'>things correctly.<o:p></o:p></p></blockquote></blockquote></blockquote><blockquote style='margin-top:5.0pt;margin-bottom:5.0pt'><blockquote style='margin-top:5.0pt;margin-bottom:5.0pt'><p class=MsoNormal style='margin-left:36.0pt'>If applications are using mmap IO it is possible for the page to become inconsistent after the checksum has been computed. However, mmap IO is<o:p></o:p></p></blockquote></blockquote><blockquote style='margin-top:5.0pt;margin-bottom:5.0pt'><blockquote style='margin-top:5.0pt;margin-bottom:5.0pt'><p class=MsoNormal style='margin-left:36.0pt'>normally detected by the client and no message should be printed.<o:p></o:p></p></blockquote></blockquote><blockquote style='margin-top:5.0pt;margin-bottom:5.0pt'><blockquote style='margin-top:5.0pt;margin-bottom:5.0pt'><p class=MsoNormal style='margin-left:36.0pt'><o:p> </o:p></p></blockquote></blockquote><blockquote style='margin-top:5.0pt;margin-bottom:5.0pt'><blockquote style='margin-top:5.0pt;margin-bottom:5.0pt'><p class=MsoNormal style='margin-left:36.0pt'>There isn't anything that the application needs to do, since the client will resend the data if there is a checksum error, but the resends do slow down the IO. If the inconsistency is on the client, there is no cause for concern (though it would be good to figure out the root cause).<o:p></o:p></p></blockquote></blockquote><blockquote style='margin-top:5.0pt;margin-bottom:5.0pt'><blockquote style='margin-top:5.0pt;margin-bottom:5.0pt'><p class=MsoNormal style='margin-left:36.0pt'><o:p> </o:p></p></blockquote></blockquote><blockquote style='margin-top:5.0pt;margin-bottom:5.0pt'><blockquote style='margin-top:5.0pt;margin-bottom:5.0pt'><p class=MsoNormal style='margin-left:36.0pt'>It would be interesting to see what the exact error message is, since that will say whether the data became inconsistent on the client, or over the network. If the inconsistency is over the network or on the server, then that may point to hardware issues.<o:p></o:p></p></blockquote></blockquote><blockquote style='margin-top:5.0pt;margin-bottom:5.0pt'><p class=MsoNormal style='margin-left:36.0pt'>I've attached logs from a server and a client.<o:p></o:p></p></blockquote><p class=MsoNormal style='mso-margin-top-alt:0cm;margin-right:0cm;margin-bottom:12.0pt;margin-left:36.0pt'><br>There was a cut n' paste error in the first set of files. This should be<br>better.<br><br>Looks like a something goes wrong over the network.<br><br>Cheers,<br>Hans Henrik<o:p></o:p></p></div></blockquote><blockquote style='margin-top:5.0pt;margin-bottom:5.0pt'><div><p class=MsoNormal style='margin-left:36.0pt'><client.log><o:p></o:p></p></div></blockquote><blockquote style='margin-top:5.0pt;margin-bottom:5.0pt'><div><p class=MsoNormal style='margin-left:36.0pt'><server.log><o:p></o:p></p></div></blockquote><blockquote style='margin-top:5.0pt;margin-bottom:5.0pt'><div><p class=MsoNormal style='margin-left:36.0pt'>_______________________________________________<br>lustre-discuss mailing list<br><a href="mailto:lustre-discuss@lists.lustre.org">lustre-discuss@lists.lustre.org</a><br><a href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org">http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org</a><o:p></o:p></p></div></blockquote></div><p class=MsoNormal style='margin-left:36.0pt'>_______________________________________________ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org <o:p></o:p></p></div></body></html>