refactor(streaming): unify and refine rate_limit related type

[MrCroxx]

Background

Currently, we are using two types to present "rate_limit".

In .proto files, all rate limit values are optional uint32:

• None: infinite
• 0: pause
• other value v: rate limit at v

However, when we are parsing the rate limit with SQL, we are using i32:

• -1 (and other negative value for compatibility): infinite
• 0: pause
• other positive value v: rate limit at v

Although it looks wired, it works fine for now.

But things are getting worse when we want to introduce some new rate limit policy. It is too tricky to use more conventional values, especially with u32 type. (e.g. To introduce the adaptive rate limiter for backfill, #19743 )

Design

Use a new message Throttle to unify and refine all rate limits.

e.g.

enum ThrottleKind {
	THROTTLE_KIND_ADAPTIVE = 1;
	THROTTLE_KIND_INFINITE = 2
	THROTTLE_KIND_FIXED = 3;
}

message Throttle {
	ThrottleKind kind = 1;
	// Only valid when `kind` is `FIXED`.
	uint32 value = 2;
}

Future Optimizations

No response

Discussions

No response

Q&A

No response

[MrCroxx]

Any comments? cc @hzxa21 @StrikeW @kwannoel @tabVersion

[tabVersion]

at least it should not be THROTTLE_KIND_INFINiTE (THROTTLE_KIND_INFINITE) 🤣

[MrCroxx]

at least it should not be THROTTLE_KIND_INFINiTE (THROTTLE_KIND_INFINITE) 🤣

Fixed. 🫠

[tabVersion]

proposing

enum ThrottleKind {
	THROTTLE_KIND_ADAPTIVE = 1;
	THROTTLE_KIND_INFINITE = 2
	THROTTLE_KIND_FIXED = 3;
++  THROTTLE_KIND_PAUSE = 4; // more distinguishable than contained by FIXED
}

message Throttle {
	ThrottleKind kind = 1;
	// Only valid when `kind` is `FIXED`.
	uint32 value = 2;
}

[kwannoel]

How about THROTTLE_KIND_DISABLED instead of infinite.

[MrCroxx]

Proposings on naming accepted.

Can I assume this refactor works for you?

We need to keep both fields for a few versions. Because the streaming plan is stored as proto in meta

[kwannoel]

Proposings on naming accepted.

Can I assume this refactor works for you?

We need to keep both fields for a few versions. Because the streaming plan is stored as proto in meta

LGTM, thanks for the initiative.