<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Exchange Server">
<!-- converted from text --><style><!-- .EmailQuote { margin-left: 1pt; padding-left: 4pt; border-left: #800000 2px solid; } --></style>
</head>
<body>
<meta content="text/html; charset=UTF-8">
<style type="text/css" style="">
<!--
p
{margin-top:0;
margin-bottom:0}
-->
</style>
<div dir="ltr">
<div id="x_divtagdefaultwrapper" dir="ltr" style="font-size:12pt; color:#000000; font-family:Calibri,Helvetica,sans-serif">
<p>I think basing this on the maximum number of stripes it too simple, and maybe not necessary.<br>
<br>
Apologies in advance if what I say below rests on a misunderstanding of the compression design, I should know it better than I do.</p>
<p><br>
</p>
<p>But, here goes.<br>
<br>
About based on maximum stripe count, there are a number of 1000 OST systems in the world today. Imagine one of them with 16 MiB stripes, that's ~16 GiB of memory for this. I think that's clearly too large. But a global (rather than per OSC) pool could be
tricky too, leading to contention on getting and returning pages.</p>
<p><br>
You mention later a 50 MiB pool per client. <span>As a per OST pre-allocated pool, that would likely be too large.</span> As a global pool, it seems small...<br>
</p>
<p><br>
</p>
<p>But why use a global pool? It sounds like the compression would be handled by the thread putting the data on the wire (Sorry if I've got that wrong). So - What about a per-thread block of pages, for each ptlrpcd thread? If the idea is that this compressed
data is not retained for replay (instead, you would re-compress), then we only need a block of max rpc size for each thread (You could just use the largest RPC size supported by the client), so it can send that compressed data.<br>
<br>
The overhead of compression for replay is probably not something we need to worry about.<br>
</p>
<p><br>
</p>
<p>Or even per-CPU blocks of pages. That would probably be better still (less total memory if there are more ptlrpcds than CPUs), if we can guarantee not sleeping during the time the pool is in use. (I'm not sure.)<br>
<br>
Also, you mention limiting the # of threads. Why is limiting the number of threads doing compression of interest? What are you specifically trying to avoid with that?<br>
</p>
</div>
<hr tabindex="-1" style="display:inline-block; width:98%">
<div id="x_divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" color="#000000" style="font-size:11pt"><b>From:</b> lustre-devel <lustre-devel-bounces@lists.lustre.org> on behalf of Anna Fuchs <anna.fuchs@informatik.uni-hamburg.de><br>
<b>Sent:</b> Friday, July 21, 2017 10:15:30 AM<br>
<b>To:</b> Xiong, Jinshan<br>
<b>Cc:</b> Matthew Ahrens; Zhuravlev, Alexey; lustre-devel<br>
<b>Subject:</b> Re: [lustre-devel] Design proposal for client-side compression</font>
<div> </div>
</div>
</div>
<font size="2"><span style="font-size:10pt;">
<div class="PlainText">Dear all, <br>
<br>
for compression within the osc module we need a bunch of pages for the<br>
compressed output (at most the same size like original data), and few<br>
pages for working memory of the algorithms. Since allocating (and later<br>
freeing) the pages every time we enter the compression loop might be<br>
expensive and annoying, we thought about a pool of pages, which is<br>
present exclusively for compression purposes.<br>
<br>
We would create that pool at file system start (when loading the osc<br>
module) and destroy at file system stop (when unloading the osc<br>
module). The condition is, of course, the configure option --enable-<br>
compression. The pool would be a queue of page bunches where a thread<br>
can pop pages for compression and put them back after the compressed<br>
portion was transferred. The page content will not be visible to anyone<br>
outside and will also not be cached after the transmission.<br>
<br>
We would like to make the pool static since we think, we do not need a<br>
lot of memory. However it depends on the number of stripes or MBs, that<br>
one client can handle at the same time. E.g. for 32 stripes of 1MB<br>
processed at the same time, we need at most 32 MB + few MB for<br>
overhead. Where can I find the exact number or how can I estimate how<br>
many stripes there are at most at the same time? Another limitation is<br>
the number of threads, which can work in parallel on compression at the<br>
same time. We think to exclusively reserve not more than 50 MB for the<br>
compression page pool per client. Do you think it might hurt the<br>
performance?<br>
<br>
Once there are not enough pages, for whatever reason, we wouldn't wait,<br>
but just skip the compression for the respective chunk. <br>
<br>
Are there any problems you see in that approach? <br>
<br>
Regards,<br>
Anna<br>
<br>
--<br>
Anna Fuchs<br>
<a href="https://wr.informatik.uni-hamburg.de/people/anna_fuchs">https://wr.informatik.uni-hamburg.de/people/anna_fuchs</a><br>
_______________________________________________<br>
lustre-devel mailing list<br>
lustre-devel@lists.lustre.org<br>
<a href="http://lists.lustre.org/listinfo.cgi/lustre-devel-lustre.org">http://lists.lustre.org/listinfo.cgi/lustre-devel-lustre.org</a><br>
</div>
</span></font>
</body>
</html>