<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 14 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri","sans-serif";
mso-fareast-language:EN-US;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-GB" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif"">I’ve seen CTDB + Samba deployed on several sites running Lustre. It’s stable in my experience, and straightforward to get installed and set up, although the process is time-consuming.
The most significant hurdle is integrating with AD and maybe load balancing for the CTDB servers (RR DNS is the easiest and most common solution).<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif"">Performance is not nearly as good as for native Lustre client (apart from anything else, IIRC, SMB is a “chatty” protocol, esp with xattrs?). One downside of CTDB is that
Lustre client must be mounted with -oflock in order for the recovery lock manager to work. Each individual connection to Samba from a Windows client is limited to the bandwidth and single thread performance of the CTDB node. Clients remain connected to a single
CTDB node for the duration of their session, so there is a possibility of an imbalance in connections over time. Load balancing is strictly round-robin through DNS lookups, unless a more sophisticated load balancer is placed in front of the CTDB cluster.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif"">There are references to CTDB + NFS / Ganesha as well but I haven’t had an opportunity to try it out. Most of the demand for non-native client access to Lustre involves Windows
machines.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif"">Malcolm.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p> </o:p></span></p>
<p class="MsoNormal" style="margin-left:36.0pt"><b><span lang="EN-US" style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span lang="EN-US" style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> lustre-discuss [mailto:lustre-discuss-bounces@lists.lustre.org]
<b>On Behalf Of </b>Jeff Johnson<br>
<b>Sent:</b> Wednesday, July 22, 2015 5:54 AM<br>
<b>To:</b> Indivar Nair<br>
<b>Cc:</b> lustre-discuss<br>
<b>Subject:</b> Re: [lustre-discuss] [HPDD-discuss] Lustre Server Sizing<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:36.0pt"><o:p> </o:p></p>
<div>
<p class="MsoNormal" style="margin-left:36.0pt">Indivar,<o:p></o:p></p>
<div>
<p class="MsoNormal" style="margin-left:36.0pt"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:36.0pt">Since your CIFS or NFS gateways operate as Lustre clients there can be issues with running multiple NFS or CIFS gateway machines frontending the same Lustre filesystem. As Lustre clients there are no issues in
terms of file locking but the NFS and CIFS caching and multi-client file access mechanics don't interface with Lustre's file locking mechanics. Perhaps that may have changed recently and a developer on the list may comment on developments there. So while you
could provide client access through multiple NFS or CIFS gateway machines there would not be much in the way of file locking protection. There is a way to configure pCIFS with CTDB and get close to what you envision with Samba. I did that configuration once
as a proof of concept (no valuable data). It is a *very* complex configuration and based on the state of software when I did it I wouldn't say it was a production grade environment.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:36.0pt"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:36.0pt">As I said before, my understanding may be a year out of date and someone else could speak to the current state of things. Hopefully that would be a better story.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:36.0pt"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:36.0pt">--Jeff<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:36.0pt"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:36.0pt"><o:p> </o:p></p>
</div>
</div>
<div>
<p class="MsoNormal" style="margin-left:36.0pt"><o:p> </o:p></p>
<div>
<p class="MsoNormal" style="margin-left:36.0pt">On Tue, Jul 21, 2015 at 10:26 AM, Indivar Nair <<a href="mailto:indivar.nair@techterra.in" target="_blank">indivar.nair@techterra.in</a>> wrote:<o:p></o:p></p>
<div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal" style="margin-left:36.0pt">Hi Scott, <br>
<br>
The 3 - SAN Storages with 240 disks each has its own 3 NAS Headers (NAS Appliances).<o:p></o:p></p>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:0cm;margin-right:0cm;margin-bottom:12.0pt;margin-left:36.0pt">
However, even with 240 10K RPM disk and RAID50, it is only providing around 1.2 - 1.4GB/s per NAS Header.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:36.0pt">There is no clustered file system, and each NAS Header has its own file-system.<o:p></o:p></p>
</div>
<p class="MsoNormal" style="margin-left:36.0pt">It uses some custom mechanism to present the 3 file systems as single name space.<o:p></o:p></p>
</div>
<p class="MsoNormal" style="margin-left:36.0pt">But the directories have to be manually spread across for load-balancing.<o:p></o:p></p>
</div>
<p class="MsoNormal" style="margin-left:36.0pt">As you can guess, this doesn't work most of the time.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:36.0pt">Many a times, most of the compute nodes access a single NAS Header, overloading it.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:36.0pt"><o:p> </o:p></p>
</div>
<p class="MsoNormal" style="margin-left:36.0pt">The customer wants *at least* 9GB/s throughput from a single file-system.<o:p></o:p></p>
<div>
<div>
<p class="MsoNormal" style="margin-left:36.0pt"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal" style="margin-left:36.0pt">But I think, if we architect the Lustre Storage correctly, with these many disks, we should get at least 18GB/s throughput, if not more.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:36.0pt"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:0cm;margin-right:0cm;margin-bottom:12.0pt;margin-left:36.0pt">
Regards,<br>
<br>
<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:36.0pt">Indivar Nair<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:36.0pt"><o:p> </o:p></p>
</div>
</div>
</div>
</div>
</div>
<div>
<p class="MsoNormal" style="margin-left:36.0pt"><o:p> </o:p></p>
<div>
<div>
<div>
<p class="MsoNormal" style="margin-left:36.0pt">On Tue, Jul 21, 2015 at 10:15 PM, Scott Nolin <<a href="mailto:scott.nolin@ssec.wisc.edu" target="_blank">scott.nolin@ssec.wisc.edu</a>> wrote:<o:p></o:p></p>
</div>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm">
<div>
<div>
<p class="MsoNormal" style="margin-left:36.0pt">An important question is what performance do they have now, and what do they expect if converting it to Lustre. Our more basically, what are they looking for in general in changing?<br>
<br>
The performance requirements may help drive your OSS numbers for example, or interconnect, and all kinds of stuff.<br>
<br>
Also I don't have a lot of experience with NFS/CIFS gateways, but that is perhaps it's own topic and may need some close attention.<br>
<br>
Scott<br>
<br>
On 7/21/2015 10:57 AM, Indivar Nair wrote:<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:0cm;margin-right:0cm;margin-bottom:12.0pt;margin-left:36.0pt">
Hi ...,<br>
<br>
One of our customers has a 3 x 240 Disk SAN Storage Array and would like<br>
to convert it to Lustre.<br>
<br>
They have around 150 Workstations and around 200 Compute (Render) nodes.<br>
The File Sizes they generally work with are -<br>
1 to 1.5 million files (images) of 10-20MB in size.<br>
And a few thousand files of 500-1000MB in size.<br>
<br>
Almost 50% of the infra is on MS Windows or Apple MACs<br>
<br>
I was thinking of the following configuration -<br>
1 MDS<br>
1 Failover MDS<br>
3 OSS (failover to each other)<br>
3 NFS+CIFS Gateway Servers<br>
FDR Infiniband backend network (to connect the Gateways to Lustre)<br>
Each Gateway Server will have 8 x 10GbE Frontend Network (connecting the<br>
clients)<br>
<br>
*Option A*<br>
10+10 Disk RAID60 Array with 64KB Chunk Size i.e. 1MB Stripe Width<br>
720 Disks / (10+10) = 36 Arrays.<br>
12 OSTs per OSS<br>
18 OSTs per OSS in case of Failover<br>
<br>
*Option B*<br>
10+10+10+10 Disk RAID60 Array with 128KB Chunk Size i.e. 4MB Stripe<br>
Width<br>
720 Disks / (10+10+10+10) = 18 Arrays<br>
6 OSTs per OSS<br>
9 OSTs per OSS in case of Failover<br>
4MB RPC and I/O<br>
<br>
*Questions*<br>
1. Would it be better to let Lustre do most of the striping / file<br>
distribution (as in Option A) OR would it be better to let the RAID<br>
Controllers do it (as in Option B)<br>
<br>
2. Will Option B allow us to have lesser CPU/RAM than Option A?<br>
<br>
Regards,<br>
<br>
<br>
Indivar Nair<br>
<br>
<br>
<br>
_______________________________________________<br>
HPDD-discuss mailing list<br>
<a href="mailto:HPDD-discuss@lists.01.org" target="_blank">HPDD-discuss@lists.01.org</a><br>
<a href="https://lists.01.org/mailman/listinfo/hpdd-discuss" target="_blank">https://lists.01.org/mailman/listinfo/hpdd-discuss</a><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:0cm;margin-right:0cm;margin-bottom:12.0pt;margin-left:36.0pt">
<br>
<br>
<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:0cm;margin-right:0cm;margin-bottom:12.0pt;margin-left:36.0pt">
_______________________________________________<br>
lustre-discuss mailing list<br>
<a href="mailto:lustre-discuss@lists.lustre.org" target="_blank">lustre-discuss@lists.lustre.org</a><br>
<a href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org" target="_blank">http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org</a><o:p></o:p></p>
</blockquote>
</div>
<p class="MsoNormal" style="margin-left:36.0pt"><o:p> </o:p></p>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:0cm;margin-right:0cm;margin-bottom:12.0pt;margin-left:36.0pt">
<br>
_______________________________________________<br>
lustre-discuss mailing list<br>
<a href="mailto:lustre-discuss@lists.lustre.org">lustre-discuss@lists.lustre.org</a><br>
<a href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org" target="_blank">http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org</a><o:p></o:p></p>
</div>
<p class="MsoNormal" style="margin-left:36.0pt"><br>
<br clear="all">
<o:p></o:p></p>
<div>
<p class="MsoNormal" style="margin-left:36.0pt"><o:p> </o:p></p>
</div>
<p class="MsoNormal" style="margin-left:36.0pt">-- <o:p></o:p></p>
<div>
<div>
<p class="MsoNormal" style="margin-left:36.0pt">------------------------------<br>
Jeff Johnson<br>
Co-Founder<br>
Aeon Computing<br>
<br>
<a href="mailto:jeff.johnson@aeoncomputing.com" target="_blank">jeff.johnson@aeoncomputing.com</a><br>
<a href="http://www.aeoncomputing.com" target="_blank">www.aeoncomputing.com</a><br>
t: 858-412-3810 x1001 f: 858-412-3845<br>
m: 619-204-9061<br>
<br>
4170 Morena Boulevard, Suite D - San Diego, CA 92117<o:p></o:p></p>
<div>
<p class="MsoNormal" style="margin-left:36.0pt"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:36.0pt">High-Performance Computing / Lustre Filesystems / Scale-out Storage<o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
</body>
</html>