[lustre-discuss] NRS TBF queue classification, rule matching, and rate limiting behavior

[Rasool Almasikoupaei]

Hello!

If someone could help clarify the behavior of the NRS TBF policy, I 
would really appreciate it. I'm particularly interested in how 
queue/bucket creation, classification, and rate limiting work.

I am configuring TBF on OSTs with rules similar to:
/
lctl set_param ost.OSS.ost_io.nrs_policies="tbf"

lctl set_param ost.OSS.ost_io.nrs_tbf_rule="start lt_$uid 
nid={192.168.43.[250-251]@tcp}&uid={$uid} rate=10000 realtime=1"

lctl set_param ost.OSS.ost_io.nrs_tbf_rule="start rt_$uid 
nid={192.168.43.*@tcp}&uid={$uid} rate=10000"/

The goal of this configuration is to protect interactive I/O from login 
nodes while controlling heavy I/O from compute nodes.

According to the documentation, using:

/lctl set_param ost.OSS.ost_io.nrs_policies="tbf"/

enables all available classifiers. I would like to clarify how these 
classifiers affect queue creation and rule matching.

I have the following questions:

*1. How are TBF queues/buckets created and how do rule fields affect 
their identity?*

According to the documentation, using:

/lctl set_param ost.OSS.ost_io.nrs_policies="tbf"/

enables all available classifiers (NID, JOBID, OPCODE, UID/GID).

How is the TBF queue/resource identity determined when multiple 
classifiers are enabled?

For example, with:

/start rt_1000 nid={192.168.43.*@tcp}&uid={1000} rate=10000/

and the same UID accessing from:
/
192.168.43.1 at tcp
192.168.43.2 at tcp/

does Lustre create:

a) one shared bucket:
/
UID=1000
rate=10000/

or:

b) separate buckets:

/NID=192.168.43.1 at tcp
rate=10000/

/NID=192.168.43.2 at tcp
rate=10000/

or:

c) another classifier-dependent/composite resource key?

More generally, are fields in nrs_tbf_rule (nid, uid, gid, jobid, etc.) 
only match conditions used to select the rule, or can they also 
participate in determining the TBF queue/resource identity?

For example, if gid or jobid are not specified in a rule, are they 
ignored, or can they still cause additional queue separation because 
those classifiers are enabled?

*2. Does realtime=1 provide scheduling priority?*

For example, if login traffic and compute traffic from the same user 
both have pending RPCs:

/login:
     lt_$uid
     realtime=1/
/
compute:
     rt_$uid/

will login traffic be serviced ahead of compute traffic?

*3. Rate limiting versus fair sharing*

My understanding is that TBF rate= is a token bucket rate limit rather 
than a fair-share weight.

For example:

/user A: rate=10000
user B: rate=10000/

If only user A is active, will user A remain limited to the configured 
rate, or can it consume unused OST bandwidth?

The desired behavior for our environment is:

/when multiple users are active:
     provide isolation and prevent starvation

when there is no contention:
     allow active users to use available OST bandwidth

always:
     prioritize interactive login-node traffic over compute-node traffic/

If TBF is not intended for this type of dynamic sharing, is there 
another NRS policy or recommended approach?

Any clarification would be greatly appreciated.

Rasool

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-discuss_lists.lustre.org/attachments/20260722/8304ee73/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 5095 bytes
Desc: S/MIME Cryptographic Signature
URL: <http://lists.lustre.org/pipermail/lustre-discuss_lists.lustre.org/attachments/20260722/8304ee73/attachment.p7s>