<!DOCTYPE html>
<html>
<head>
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
</head>
<body>
<p>Hello!</p>
<p>If someone could help clarify the behavior of the NRS TBF policy,
I would really appreciate it. I'm particularly interested in how
queue/bucket creation, classification, and rate limiting work.<br>
<br>
I am configuring TBF on OSTs with rules similar to:<br>
<i><font face="Times New Roman, Times, serif"><br>
lctl set_param ost.OSS.ost_io.nrs_policies="tbf"<br>
<br>
lctl set_param ost.OSS.ost_io.nrs_tbf_rule="start lt_$uid
nid={192.168.43.[250-251]@tcp}&uid={$uid} rate=10000
realtime=1"<br>
<br>
lctl set_param ost.OSS.ost_io.nrs_tbf_rule="start rt_$uid
nid={192.168.43.*@tcp}&uid={$uid} rate=10000"</font></i><br>
<br>
The goal of this configuration is to protect interactive I/O from
login nodes while controlling heavy I/O from compute nodes.<br>
<br>
According to the documentation, using:<br>
<br>
<i><font face="Times New Roman, Times, serif">lctl set_param
ost.OSS.ost_io.nrs_policies="tbf"</font></i><br>
<br>
enables all available classifiers. I would like to clarify how
these classifiers affect queue creation and rule matching.<br>
<br>
I have the following questions:<br>
<br>
<b>1. How are TBF queues/buckets created and how do rule fields
affect their identity?</b><br>
<br>
According to the documentation, using:<br>
<br>
<font face="Times New Roman, Times, serif"><i>lctl set_param
ost.OSS.ost_io.nrs_policies="tbf"</i></font><br>
<br>
enables all available classifiers (NID, JOBID, OPCODE, UID/GID).<br>
<br>
How is the TBF queue/resource identity determined when multiple
classifiers are enabled?<br>
<br>
For example, with:<br>
<br>
<i><font face="Times New Roman, Times, serif">start rt_1000
nid={192.168.43.*@tcp}&uid={1000} rate=10000</font></i><br>
<br>
and the same UID accessing from:<br>
<font face="Times New Roman, Times, serif"><i><br>
192.168.43.1@tcp<br>
192.168.43.2@tcp</i></font><br>
<br>
does Lustre create:<br>
<br>
a) one shared bucket:<br>
<i><font face="Times New Roman, Times, serif"><br>
UID=1000<br>
rate=10000</font></i><br>
<br>
or:<br>
<br>
b) separate buckets:<br>
<br>
<i><font face="Times New Roman, Times, serif">NID=192.168.43.1@tcp<br>
rate=10000</font></i><br>
<br>
<i><font face="Times New Roman, Times, serif">NID=192.168.43.2@tcp<br>
rate=10000</font></i><br>
<br>
or:<br>
<br>
c) another classifier-dependent/composite resource key?<br>
<br>
More generally, are fields in nrs_tbf_rule (nid, uid, gid, jobid,
etc.) only match conditions used to select the rule, or can they
also participate in determining the TBF queue/resource identity?<br>
<br>
For example, if gid or jobid are not specified in a rule, are they
ignored, or can they still cause additional queue separation
because those classifiers are enabled?<br>
<br>
<b>2. Does realtime=1 provide scheduling priority?</b><br>
<br>
For example, if login traffic and compute traffic from the same
user both have pending RPCs:<br>
<br>
<i><font face="Times New Roman, Times, serif">login:<br>
lt_$uid<br>
realtime=1</font></i><br>
<i><font face="Times New Roman, Times, serif"><br>
compute:<br>
rt_$uid</font></i><br>
<br>
will login traffic be serviced ahead of compute traffic?<br>
<br>
<b>3. Rate limiting versus fair sharing</b><br>
<br>
My understanding is that TBF rate= is a token bucket rate limit
rather than a fair-share weight.<br>
<br>
For example:<br>
<br>
<i><font face="Times New Roman, Times, serif">user A: rate=10000<br>
user B: rate=10000</font></i><br>
<br>
If only user A is active, will user A remain limited to the
configured rate, or can it consume unused OST bandwidth?<br>
<br>
The desired behavior for our environment is:<br>
<br>
<i>when multiple users are active:<br>
provide isolation and prevent starvation<br>
<br>
when there is no contention:<br>
allow active users to use available OST bandwidth<br>
<br>
always:<br>
prioritize interactive login-node traffic over compute-node
traffic</i><br>
<br>
If TBF is not intended for this type of dynamic sharing, is there
another NRS policy or recommended approach?<br>
<br>
Any clarification would be greatly appreciated.<br>
<br>
Rasool</p>
</body>
</html>