<div dir="ltr"><div dir="ltr">After some investigation it looks like a timeout issue in the smartpqi kernel module is causing the disks to be removed soon after they are initially added based on what is reported in "dmesg"</div><div dir="ltr"><br></div><div dir="ltr">This issue first occurred in RHEL/Centos 7.4 and should have been resolved by centos 7.7. I've emailed the maintainer of the module and he's come back to me with an offer to create a test driver to see if increasing the timeout fixes the issue. There is an existing patch but its version is less than the one in Centos 7.9.</div><div dir="ltr"><br></div><div dir="ltr">On the bright side, I've built and rebuilt the Lustre MDS and OSS config several times as I optimise the installation while running under Pacemaker and have been able to mount /lustre and /home on the Compute nodes so this new system is 50% of the way there :)<br clear="all"><div><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div><br></div><div><br></div><div>Sid Young</div></div></div></div></div></div></div></div></div></div></div></div></div><div class="gmail_quote"><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Today's Topics:<br>
<br>
1. Re: MDS using D3710 DAS (Sid Young)<br>
2. Re: MDS using D3710 DAS (Christopher Mountford)<br>
<br><br><br>---------- Forwarded message ----------<br>From: Sid Young <<a href="mailto:sid.young@gmail.com" target="_blank">sid.young@gmail.com</a>><br>To: Christopher Mountford <<a href="mailto:cjm14@leicester.ac.uk" target="_blank">cjm14@leicester.ac.uk</a>>, <a href="mailto:lustre-discuss@lists.lustre.org" target="_blank">lustre-discuss@lists.lustre.org</a><br>Cc: <br>Bcc: <br>Date: Mon, 15 Feb 2021 08:42:43 +1000<br>Subject: Re: [lustre-discuss] MDS using D3710 DAS<br><div dir="ltr">Hi Christopher,<div><br></div><div>Just some background, all servers are DL385's all servers are running the same image of Centos 7.9, The MDS HA pair have a SAS connected D3710 and the dual OSS HA pair have a D8000 each with 45 disks in each of them.</div><div><br></div><div>The D3710 (which has 24x 960G SSD's) seams a bit hit and miss at presenting two LV's, I had setup a /lustre and /home which I was going to use ldiskfs rather than zfs however I am finding that the disks MAY present to both servers after some reboots but usually the first server to reboot see's the LV presented and the other only see's its local internal disks only, so the array appears to only present the LV's to one host most of the time.</div><div><br></div><div>With the 4 OSS servers. i see the same issue, sometimes the LV's present and sometimes they don't. </div><div><br></div><div>I was planning on setting up the OST's as ldiskfs as well, but I could also go zfs, my test bed system and my current HPC uses ldsikfs.</div><div><br></div><div>Correct me if I am wrong, but disks should present to both servers all the time and using PCS I should be able to mount up a /lustre and /home one the first server while the disks present on the second server but no software is mounting them so there should be no issues?<br clear="all"><div><div dir="ltr"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div><br></div><div><br></div><div>Sid Young</div></div></div></div></div></div></div></div></div></div></div></div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Feb 12, 2021 at 7:27 PM Christopher Mountford <<a href="mailto:cjm14@leicester.ac.uk" target="_blank">cjm14@leicester.ac.uk</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hi Sid,<br>
<br>
We've a similar hardware configuration - 2 MDS pairs and 1 OSS pair which each consist of 2 DL360 connected to a single D3700. However we are using Lustre on ZFS with each array split into 2 or 4 zpools (depending on the usage) and haven't seen any problems of this sort. Are you using ldiskfs?<br>
<br>
- Chris<br>
<br>
<br>
On Fri, Feb 12, 2021 at 03:14:58PM +1000, Sid Young wrote:<br>
> G'day all,<br>
> Is anyone using a HPe D3710 with two HPeDL380/385 servers in a MDS HA<br>
> Configuration? If so, is your D3710 presenting LV's to both servers at<br>
> the same time AND are you using PCS with the Lustre PCS Resources?<br>
> I've just received new kit and cannot get disk to present to the MDS<br>
> servers at the same time..... :(<br>
> Sid Young<br>
<br>
> _______________________________________________<br>
> lustre-discuss mailing list<br>
> <a href="mailto:lustre-discuss@lists.lustre.org" target="_blank">lustre-discuss@lists.lustre.org</a><br>
> <a href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org" rel="noreferrer" target="_blank">http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org</a><br>
<br>
</blockquote></div>
<br><br><br>---------- Forwarded message ----------<br>From: Christopher Mountford <<a href="mailto:cjm14@leicester.ac.uk" target="_blank">cjm14@leicester.ac.uk</a>><br>To: Sid Young <<a href="mailto:sid.young@gmail.com" target="_blank">sid.young@gmail.com</a>><br>Cc: Christopher Mountford <<a href="mailto:cjm14@leicester.ac.uk" target="_blank">cjm14@leicester.ac.uk</a>>, <a href="mailto:lustre-discuss@lists.lustre.org" target="_blank">lustre-discuss@lists.lustre.org</a><br>Bcc: <br>Date: Mon, 15 Feb 2021 10:44:10 +0000<br>Subject: Re: [lustre-discuss] MDS using D3710 DAS<br><br>
Hi Sid.<br>
<br>
We use the D3700s (and our D8000s) as JBODS with zfs providing the redundancy - do you have some kind of hardware RAID? If so, are your raid controller the array corntrollers or on the HBAs? Off the top of my head, if the latter, there might be an issue with multiple HBAs trying to assemble the same RAID array? <br>
<br>
- Chris.<br>
<br>
On Mon, Feb 15, 2021 at 08:42:43AM +1000, Sid Young wrote:<br>
> Hi Christopher,<br>
> Just some background, all servers are DL385's all servers are running<br>
> the same image of Centos 7.9, The MDS HA pair have a SAS connected<br>
> D3710 and the dual OSS HA pair have a D8000 each with 45 disks in each<br>
> of them.<br>
> The D3710 (which has 24x 960G SSD's) seams a bit hit and miss at<br>
> presenting two LV's, I had setup a /lustre and /home which I was going<br>
> to use ldiskfs rather than zfs however I am finding that the disks MAY<br>
> present to both servers after some reboots but usually the first server<br>
> to reboot see's the LV presented and the other only see's its local<br>
> internal disks only, so the array appears to only present the LV's to<br>
> one host most of the time.<br>
> With the 4 OSS servers. i see the same issue, sometimes the LV's<br>
> present and sometimes they don't.<br>
> I was planning on setting up the OST's as ldiskfs as well, but I could<br>
> also go zfs, my test bed system and my current HPC uses ldsikfs.<br>
> Correct me if I am wrong, but disks should present to both servers all<br>
> the time and using PCS I should be able to mount up a /lustre and /home<br>
> one the first server while the disks present on the second server but<br>
> no software is mounting them so there should be no issues?<br>
> Sid Young<br>
> <br>
> On Fri, Feb 12, 2021 at 7:27 PM Christopher Mountford<br>
> <[1]<a href="mailto:cjm14@leicester.ac.uk" target="_blank">cjm14@leicester.ac.uk</a>> wrote:<br>
> <br>
> Hi Sid,<br>
> We've a similar hardware configuration - 2 MDS pairs and 1 OSS pair<br>
> which each consist of 2 DL360 connected to a single D3700. However<br>
> we are using Lustre on ZFS with each array split into 2 or 4 zpools<br>
> (depending on the usage) and haven't seen any problems of this sort.<br>
> Are you using ldiskfs?<br>
> - Chris<br>
> On Fri, Feb 12, 2021 at 03:14:58PM +1000, Sid Young wrote:<br>
> > G'day all,<br>
> > Is anyone using a HPe D3710 with two HPeDL380/385 servers in a<br>
> MDS HA<br>
> > Configuration? If so, is your D3710 presenting LV's to both<br>
> servers at<br>
> > the same time AND are you using PCS with the Lustre PCS<br>
> Resources?<br>
> > I've just received new kit and cannot get disk to present to<br>
> the MDS<br>
> > servers at the same time..... :(<br>
> > Sid Young<br>
> > _______________________________________________<br>
> > lustre-discuss mailing list<br>
> > [2]<a href="mailto:lustre-discuss@lists.lustre.org" target="_blank">lustre-discuss@lists.lustre.org</a><br>
> > [3]<a href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org" rel="noreferrer" target="_blank">http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org</a><br>
> <br>
> References<br>
> <br>
> 1. mailto:<a href="mailto:cjm14@leicester.ac.uk" target="_blank">cjm14@leicester.ac.uk</a><br>
> 2. mailto:<a href="mailto:lustre-discuss@lists.lustre.org" target="_blank">lustre-discuss@lists.lustre.org</a><br>
> 3. <a href="https://eur03.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.lustre.org%2Flistinfo.cgi%2Flustre-discuss-lustre.org&data=04%7C01%7Ccjm14%40leicester.ac.uk%7C4d86239b31b545d327db08d8d139f050%7Caebecd6a31d44b0195ce8274afe853d9%7C0%7C0%7C637489394067185599%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=x1PMOvlWp3bocS%2Bub1mpvE1Mn59Q0EU0M18NQbj1wOk%3D&reserved=0" rel="noreferrer" target="_blank">https://eur03.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.lustre.org%2Flistinfo.cgi%2Flustre-discuss-lustre.org&data=04%7C01%7Ccjm14%40leicester.ac.uk%7C4d86239b31b545d327db08d8d139f050%7Caebecd6a31d44b0195ce8274afe853d9%7C0%7C0%7C637489394067185599%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=x1PMOvlWp3bocS%2Bub1mpvE1Mn59Q0EU0M18NQbj1wOk%3D&reserved=0</a><br>
<br>
_______________________________________________<br>
lustre-discuss mailing list<br>
<a href="mailto:lustre-discuss@lists.lustre.org" target="_blank">lustre-discuss@lists.lustre.org</a><br>
<a href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org" rel="noreferrer" target="_blank">http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org</a><br>
</blockquote></div></div>