<div dir="ltr"><div><div><div><div><div><div><div><div>Thanks for the input everyone.<br><br></div>At one other site (check other mail, Subject: Speeding up Recovery), I have two gateway nodes with Samba+CTDB, working fine and is quite stable. I don't have NFS on it though.<br><br></div>Andreas, I am considering RAID60 to balance space and speed (currently they have RAID50).<br></div><div>I will consider adding more OSS (budget is a constraint though) as you suggested.<br><br></div>I was thinking of connecting all clients through the gateway to speed up recovery after a fail-over.<br></div>In my other setup, (with MGS and MDT on the same server, 300 clients, Lustre 2.4) it takes at least twenty minutes for a recovery.<br><br></div><div></div>If I limit the clients to just the 3 gateway nodes, I guess I will be lowering the failover time to a couple of minutes. I do plan to keep the MGS and MDT on different boxes this time, but I just want to be doubly sure that the failover wont take too much time.<br></div></div><div><br>Do correct me if I am wrong and tell if there is some other ways / methods to reduce the recovery time.<br><br></div>Regards,<br><br><br></div>Indivar Nair<br><div><div><div><div><div><div><div><div><div><div><br></div></div></div></div></div></div></div></div></div></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Jul 22, 2015 at 8:03 AM, Patrick Farrell <span dir="ltr"><<a href="mailto:paf@cray.com" target="_blank">paf@cray.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Note the other email also seemed to suggest that multiple NFS exports of Lustre wouldn't work. I don't think that's the case, as we have this sort of setup at a number of our customers without particular trouble. In the abstract, I could see the possibility of some caching errors between different clients, but that would be only namespace stuff, not data. And I think in practice that's ok.<br>
<br>
But regardless, as Andreas said, for the Linux clients, Lustre directly will give much better results.<br>
________________________________________<br>
From: lustre-discuss [<a href="mailto:lustre-discuss-bounces@lists.lustre.org">lustre-discuss-bounces@lists.lustre.org</a>] on behalf of Dilger, Andreas [<a href="mailto:andreas.dilger@intel.com">andreas.dilger@intel.com</a>]<br>
Sent: Tuesday, July 21, 2015 6:59 PM<br>
To: Indivar Nair<br>
Cc: hpdd-discuss; to: lustre-discuss<br>
Subject: Re: [lustre-discuss] Lustre Server Sizing<br>
<div><div class="h5"><br>
Having only 3 OSS will limit the performance you can get, and having so many OSTs on each OSS will give sub-optimal performance. 4-6 OSTs/OSS is more reasonable.<br>
<br>
It also isn't clear why you want RAID-60 instead of just RAID-10?<br>
<br>
Finally, for Linux clients it is much better to use direct Lustre access instead of NFS as mentioned in another email.<br>
<br>
Cheers, Andreas<br>
<br>
On Jul 21, 2015, at 08:58, Indivar Nair <<a href="mailto:indivar.nair@techterra.in">indivar.nair@techterra.in</a><mailto:<a href="mailto:indivar.nair@techterra.in">indivar.nair@techterra.in</a>>> wrote:<br>
<br>
Hi ...,<br>
<br>
One of our customers has a 3 x 240 Disk SAN Storage Array and would like to convert it to Lustre.<br>
<br>
They have around 150 Workstations and around 200 Compute (Render) nodes.<br>
The File Sizes they generally work with are -<br>
1 to 1.5 million files (images) of 10-20MB in size.<br>
And a few thousand files of 500-1000MB in size.<br>
<br>
Almost 50% of the infra is on MS Windows or Apple MACs<br>
<br>
I was thinking of the following configuration -<br>
1 MDS<br>
1 Failover MDS<br>
3 OSS (failover to each other)<br>
3 NFS+CIFS Gateway Servers<br>
FDR Infiniband backend network (to connect the Gateways to Lustre)<br>
Each Gateway Server will have 8 x 10GbE Frontend Network (connecting the clients)<br>
<br>
Option A<br>
10+10 Disk RAID60 Array with 64KB Chunk Size i.e. 1MB Stripe Width<br>
720 Disks / (10+10) = 36 Arrays.<br>
12 OSTs per OSS<br>
18 OSTs per OSS in case of Failover<br>
<br>
Option B<br>
10+10+10+10 Disk RAID60 Array with 128KB Chunk Size i.e. 4MB Stripe Width<br>
720 Disks / (10+10+10+10) = 18 Arrays<br>
6 OSTs per OSS<br>
9 OSTs per OSS in case of Failover<br>
4MB RPC and I/O<br>
<br>
Questions<br>
1. Would it be better to let Lustre do most of the striping / file distribution (as in Option A) OR would it be better to let the RAID Controllers do it (as in Option B)<br>
<br>
2. Will Option B allow us to have lesser CPU/RAM than Option A?<br>
<br>
Regards,<br>
<br>
<br>
Indivar Nair<br>
<br>
_______________________________________________<br>
lustre-discuss mailing list<br>
<a href="mailto:lustre-discuss@lists.lustre.org">lustre-discuss@lists.lustre.org</a><mailto:<a href="mailto:lustre-discuss@lists.lustre.org">lustre-discuss@lists.lustre.org</a>><br>
<a href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org" rel="noreferrer" target="_blank">http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org</a><br>
_______________________________________________<br>
lustre-discuss mailing list<br>
<a href="mailto:lustre-discuss@lists.lustre.org">lustre-discuss@lists.lustre.org</a><br>
</div></div><a href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org" rel="noreferrer" target="_blank">http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org</a><br>
</blockquote></div><br></div>