<!DOCTYPE html>
<html>
<head>
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
</head>
<body>
<p>All,</p>
<p>I realized this morning that the issue described here is exactly
why I think there should be a mechanism to use legacy buffered IO
( avoiding Hybrid behavior ), even for large user requests. I
have added Image 4 which depicts the forward/backward access of
the file over time when not using O_DIRECT on the client open.
The file easily fit in the 512GB client host. The forward reads
now are 8.0 GB/s. The backward reads are 7.8 GB/s. The amount of
time spent reading drops from 505 seconds to 237 seconds and
elapsed time of the entire job improves by 19%. But the key
benefit is that my job is no longer subject to the interference of
the checkpointing application. Hybrid I/O may test out great on
dedicated clients on a dedicated Lustre file system, but in a
production environment where one runs on dedicated nodes on a
shared Lustre file system ( frequently the case ), it would be a
shame that an application not be allowed the benefit of using the
memory on its client node and also be subject to the competition
for the shared resources ( OSSs ).</p>
<p>John</p>
<p><br>
</p>
<p>All,</p>
<p>My questions of recent are related to my trying to understand the
following issue. I have an application that is writing, reading
forwards, and reading backwards, a single file multiple times ( as
seen in bottom frame of Image 1). The file is striped 4x16M on 4
ssd OSTs on 2 OSS. Everything runs along just great with transfer
rates in the 5GB/s range. At some point, another application
triggers approximately 135 GB of writes to each of the 32 hdd
OSTs on the16 OSSs of the file system. When this happens my
applications performance drops to 4.8 MB/s, a 99.9% loss of
performance for the 33+ second duration of the other
application's writes. My application is doing 16MB preads and
pwrites in parallel using 4 pthreads, with O_DIRECT on the
client. The main question I have is: "Why do the writes from the
other application affect my application so dramatically?" I am
making demands of the 2 OSS of about the same order of magnitude,
2.5GB/s each from 2 OSS, as the other application is getting from
the same 2 OSS, about 4 GB/s each. There should be no competition
for the OSTs, as I am using ssd and the other application is using
hdd. If both applications are triggering Direct I/O on the OSSs,
I would think there would be minimal competition for compute
resources on the OSSs. But as seen below in Image 3, there is a
huge spike in cpu load during the other application's writes.
This is not a one-off event. I see this about 2 out of every 3
times I run this job. I suspect the other application is one that
checkpoints on a regular interval, but I am a non-root user and
have no way to determine. I am using PCP/pmapi to get the OSS
data during my run. If the images get removed from the email, I
have used alternate text with links to Dropbox for the images.</p>
<p>Thanks,</p>
<p>John</p>
<p><font size="6">Image 1:</font></p>
<p><img moz-do-not-send="false"
src="cid:part1.Hys0vabd.0UfYTovd@iodoctors.com"
alt="https://www.dropbox.com/scl/fi/kih8qf6byl3bi5gc9r296/floorpan_oss_pause.png?rlkey=0o00o7x3oaw24h3cl3dyxyb2p&st=wahbm0gg&dl=0"
width="1354" height="870" class=""></p>
<p><br>
</p>
<p><font size="6">Image2:</font></p>
<p><img moz-do-not-send="false"
src="cid:part2.60ETowE0.16PJa5y3@iodoctors.com"
alt="https://www.dropbox.com/scl/fi/e36jjoomqa3xkadcyhdw9/disk_V_RTC.png?rlkey=ujzx02n3us42ga9prsxm5dbkh&st=ato9s3gj&dl=0"
width="1378" height="872" class=""></p>
<p><font size="6"><br>
</font></p>
<p><font size="6">Image 3:</font></p>
<p><img moz-do-not-send="false"
src="cid:part3.ZRlY6gF1.fPehY0GA@iodoctors.com"
alt="https://www.dropbox.com/scl/fi/bzudgnwnecvkp3ra4kjvp/kernelAllLoad.png?rlkey=fni6lv4zwbt53aprg6twjmnsv&st=sy9expz6&dl=0"
width="1362" height="877" class=""></p>
<p><br>
</p>
<p><font size="6">Image 4:</font></p>
<p><font size="6"><img moz-do-not-send="false"
src="cid:part4.BmAK8Nhy.PmHEap0I@iodoctors.com"
alt="https://www.dropbox.com/scl/fi/cfjpl0jko3zk6tu0f8o71/non-direct.png?rlkey=55pwhiknqiid5fyu1ax2gf28g&st=mqs5y78w&dl=0"
width="1372" height="872"></font></p>
<p><br>
</p>
</body>
</html>