<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
</head>
<body>
Rick,<br>
<br>
Your description gets (to my eyes) a little vague in the middle there.<br>
<br>
Can you be more specific about the concurrent reading and writing? This is to a single file, at the same time? Im going to assume so and write accordingly.<br>
<br>
In general, that will be slow in Lustre, depending on the details of the pattern and striping. Even if your I/O doesnt actually overlap, Lustre locks larger areas of the file (most commonly, whole stripes) during I/O, then it tries to hold on to the lock so
it can use it again.<br>
<br>
Normally this behavior is great - the lock is already there when you need it, and that helps performance a ton in various ways. But in certain specific cases where writes (or writes and reads) are being done to one file at the same time, you end up doing
lock exchange, where much of the I/O ends up having to cancel the lock held by another node in order to get its own lock (and this repeats over and over).<br>
<br>
Theres no simple, general solution to this problem. The lockahead feature can help but it requires either that your app use MPIIO already or a lot of coding on your part. The best bet for you today is probably to increase stripe count as high as you can
in your setup - since locking is per stripe, that gives you more locks to play with, which should help some. (Id be interested to know how much.)<br>
<br>
Best of luck.<br>
<br>
Regards,<br>
- Patrick<br>
<br>
<br>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> lustre-discuss <lustre-discuss-bounces@lists.lustre.org> on behalf of Riccardo Veraldi <Riccardo.Veraldi@cnaf.infn.it><br>
<b>Sent:</b> Thursday, May 10, 2018 11:55:28 PM<br>
<b>To:</b> lustre-discuss@lists.lustre.org<br>
<b>Subject:</b> [lustre-discuss] parallel write/reads problem</font>
<div> </div>
</div>
<div class="BodyFragment"><font size="2"><span style="font-size:11pt;">
<div class="PlainText">Hello,<br>
So far I am not able to solve this problem on my Lustre setup.<br>
I can reach very good performance with multi threaded writes or reads,<br>
that are sequential writes and sequential reads at different times.<br>
I can saturate Infiniband FDR capabilities reaching 6GB/s.<br>
The problem rises when while writing I start also reading the same file<br>
or even a different file.<br>
In our I/O model there are writers and readers which start reading files<br>
after a while they're being written. In this case read<br>
performances drops dramatically. Writes go up to 6GB/s but reads have a<br>
barrier and won't go more than 3GB/s.<br>
I Tried all kind of optimizations. ZFS is performing very well itself,<br>
but when Lustre is on top of it I have this problem.<br>
Infiniband is working at full speed and Lnet test also is at full speed.<br>
So I do not understand while I have concurrent writes/reads the reading<br>
performances go down.<br>
<br>
I also tweaked the ko2blnd parameters to gain more parallelism:<br>
<br>
options ko2iblnd timeout=100 peer_credits=63 credits=2560<br>
concurrent_sends=63 ntx=2048 fmr_pool_size=1280 fmr_flush_trigger=1024<br>
ntx=5120<br>
<br>
then on OSS side:<br>
<br>
lctl set_param timeout=600<br>
lctl set_param ldlm_timeout=200<br>
lctl set_param at_min=250<br>
lctl set_param at_max=600<br>
<br>
on client side:<br>
<br>
lctl set_param osc.*.checksums=0<br>
lctl set_param timeout=600<br>
lctl set_param at_min=250<br>
lctl set_param at_max=600<br>
lctl set_param ldlm.namespaces.*.lru_size=2000<br>
lctl set_param osc.*.max_rpcs_in_flight=64<br>
lctl set_param osc.*.max_dirty_mb=1024<br>
lctl set_param llite.*.max_read_ahead_mb=1024<br>
lctl set_param llite.*.max_cached_mb=81920<br>
lctl set_param llite.*.max_read_ahead_per_file_mb=1024<br>
lctl set_param subsystem_debug=0<br>
<br>
I tried to set<br>
<br>
lctl set_param osc.*.max_pages_per_rpc=1024<br>
<br>
but it is not allowed...<br>
<br>
lctl set_param osc.*.max_pages_per_rpc=1024<br>
error: set_param: setting<br>
/proc/fs/lustre/osc/drplu-OST0001-osc-ffff881ed6b05800/max_pages_per_rpc=1024:<br>
Numerical result out of range<br>
<br>
<br>
any other idea on what I may work on to get better performance on<br>
concurrent writes/reads ?<br>
<br>
thank you<br>
<br>
<br>
Rick<br>
<br>
<br>
<br>
_______________________________________________<br>
lustre-discuss mailing list<br>
lustre-discuss@lists.lustre.org<br>
<a href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org">http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org</a><br>
</div>
</span></font></div>
</body>
</html>