<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns:mv="http://macVmlSchemaUri" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Title" content="">
<meta name="Keywords" content="">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:DengXian;
panose-1:2 1 6 0 3 1 1 1 1 1;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p
{mso-style-priority:99;
margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
p.emailquote, li.emailquote, div.emailquote
{mso-style-name:emailquote;
margin-top:0in;
margin-right:0in;
margin-bottom:0in;
margin-left:1.0pt;
margin-bottom:.0001pt;
border:none;
padding:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.EmailStyle19
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
span.msoIns
{mso-style-type:export-only;
mso-style-name:"";
text-decoration:underline;
color:teal;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style>
</head>
<body bgcolor="white" lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:14.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:14.0pt"><o:p> </o:p></span></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal" style="margin-left:.5in"><b><span style="font-size:12.0pt;color:black">From:
</span></b><span style="font-size:12.0pt;color:black">Patrick Farrell <paf@cray.com><br>
<b>Date: </b>Friday, July 21, 2017 at 9:44 AM<br>
<b>To: </b>Anna Fuchs <anna.fuchs@informatik.uni-hamburg.de>, "Xiong, Jinshan" <jinshan.xiong@intel.com><br>
<b>Cc: </b>Matthew Ahrens <mahrens@delphix.com>, "Zhuravlev, Alexey" <alexey.zhuravlev@intel.com>, lustre-devel <lustre-devel@lists.lustre.org><br>
<b>Subject: </b>Re: [lustre-devel] Design proposal for client-side compression<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
</div>
<div>
<div id="x_divtagdefaultwrapper">
<p style="margin-left:.5in"><span style="font-size:12.0pt;color:black">I think basing this on the maximum number of stripes it too simple, and maybe not necessary.<br>
<br>
Apologies in advance if what I say below rests on a misunderstanding of the compression design, I should know it better than I do.<o:p></o:p></span></p>
<p style="margin-left:.5in"><span style="font-size:12.0pt;color:black"><o:p> </o:p></span></p>
<p style="margin-left:.5in"><span style="font-size:12.0pt;color:black">But, here goes.<br>
<br>
About based on maximum stripe count, there are a number of 1000 OST systems in the world today. Imagine one of them with 16 MiB stripes, that's ~16 GiB of memory for this. I think that's clearly too large. But a global (rather than per OSC) pool could be
tricky too, leading to contention on getting and returning pages.<o:p></o:p></span></p>
<p style="margin-left:.5in"><span style="font-size:12.0pt;color:black"><br>
You mention later a 50 MiB pool per client. As a per OST pre-allocated pool, that would likely be too large. As a global pool, it seems small...<o:p></o:p></span></p>
<p style="margin-left:.5in"><span style="font-size:12.0pt;color:black"><o:p> </o:p></span></p>
<p style="margin-left:.5in"><span style="font-size:12.0pt;color:black">But why use a global pool? It sounds like the compression would be handled by the thread putting the data on the wire (Sorry if I've got that wrong). So - What about a per-thread block
of pages, for each ptlrpcd thread? If the idea is that this compressed data is not retained for replay (instead, you would re-compress), then we only need a block of max rpc size for each thread (You could just use the largest RPC size supported by the client),
so it can send that compressed data.<o:p></o:p></span></p>
<p style="margin-left:.5in"><span style="font-size:12.0pt;color:black"><o:p> </o:p></span></p>
<p><span style="font-size:12.0pt;color:black">The writing thread would also need to issue RPC in its own process context, but it can be revised to use ptlrpcd thread. I tend to think using a global ptlrpc thread would be reasonable for now because compression
should be slow so I don’t expect there would be a lot of lock contention for the global pool.<o:p></o:p></span></p>
<p><span style="font-size:12.0pt;color:black"><o:p> </o:p></span></p>
<p><span style="font-size:12.0pt;color:black">Jinshan<o:p></o:p></span></p>
<p style="margin-left:.5in"><span style="font-size:12.0pt;color:black"><br>
The overhead of compression for replay is probably not something we need to worry about.<o:p></o:p></span></p>
<p style="margin-left:.5in"><span style="font-size:12.0pt;color:black"><o:p> </o:p></span></p>
<p style="margin-left:.5in"><span style="font-size:12.0pt;color:black">Or even per-CPU blocks of pages. That would probably be better still (less total memory if there are more ptlrpcds than CPUs), if we can guarantee not sleeping during the time the pool
is in use. (I'm not sure.)<br>
<br>
Also, you mention limiting the # of threads. Why is limiting the number of threads doing compression of interest? What are you specifically trying to avoid with that?<o:p></o:p></span></p>
</div>
<div class="MsoNormal" align="center" style="margin-left:.5in;text-align:center">
<hr size="2" width="98%" align="center">
</div>
<div id="x_divRplyFwdMsg">
<p class="MsoNormal" style="margin-left:.5in"><b><span style="color:black">From:</span></b><span style="color:black"> lustre-devel <lustre-devel-bounces@lists.lustre.org> on behalf of Anna Fuchs <anna.fuchs@informatik.uni-hamburg.de><br>
<b>Sent:</b> Friday, July 21, 2017 10:15:30 AM<br>
<b>To:</b> Xiong, Jinshan<br>
<b>Cc:</b> Matthew Ahrens; Zhuravlev, Alexey; lustre-devel<br>
<b>Subject:</b> Re: [lustre-devel] Design proposal for client-side compression</span>
<o:p></o:p></p>
<div>
<p class="MsoNormal" style="margin-left:.5in"> <o:p></o:p></p>
</div>
</div>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:10.0pt">Dear all,
<br>
<br>
for compression within the osc module we need a bunch of pages for the<br>
compressed output (at most the same size like original data), and few<br>
pages for working memory of the algorithms. Since allocating (and later<br>
freeing) the pages every time we enter the compression loop might be<br>
expensive and annoying, we thought about a pool of pages, which is<br>
present exclusively for compression purposes.<br>
<br>
We would create that pool at file system start (when loading the osc<br>
module) and destroy at file system stop (when unloading the osc<br>
module). The condition is, of course, the configure option --enable-<br>
compression. The pool would be a queue of page bunches where a thread<br>
can pop pages for compression and put them back after the compressed<br>
portion was transferred. The page content will not be visible to anyone<br>
outside and will also not be cached after the transmission.<br>
<br>
We would like to make the pool static since we think, we do not need a<br>
lot of memory. However it depends on the number of stripes or MBs, that<br>
one client can handle at the same time. E.g. for 32 stripes of 1MB<br>
processed at the same time, we need at most 32 MB + few MB for<br>
overhead. Where can I find the exact number or how can I estimate how<br>
many stripes there are at most at the same time? Another limitation is<br>
the number of threads, which can work in parallel on compression at the<br>
same time. We think to exclusively reserve not more than 50 MB for the<br>
compression page pool per client. Do you think it might hurt the<br>
performance?<br>
<br>
Once there are not enough pages, for whatever reason, we wouldn't wait,<br>
but just skip the compression for the respective chunk. <br>
<br>
Are there any problems you see in that approach? <br>
<br>
Regards,<br>
Anna<br>
<br>
--<br>
Anna Fuchs<br>
<a href="https://wr.informatik.uni-hamburg.de/people/anna_fuchs">https://wr.informatik.uni-hamburg.de/people/anna_fuchs</a><br>
_______________________________________________<br>
lustre-devel mailing list<br>
lustre-devel@lists.lustre.org<br>
<a href="http://lists.lustre.org/listinfo.cgi/lustre-devel-lustre.org">http://lists.lustre.org/listinfo.cgi/lustre-devel-lustre.org</a><o:p></o:p></span></p>
</div>
</div>
</body>
</html>