Throttle

[dturn]

We use vector to move logs between Kafka and our logging system. We need to enable rate limiting. But we our applications have very spikey log production. Instead of fixed rate limiting per topic we'd like to be able to set limits and then share unused capacity from other topics. Do you have any insight on how to have more dynamic control of the Throttle transform?

[jszwedko]

Hmm, yeah, that's interesting. Unfortunately, I don't think there is a way to do that in Vector currently. I think it would need to be an enhancement to the throttle transform.

[jszwedko]

But as long as every topic can use their specified limits

I think this would not be true if one topic can steal the capacity of other topics 🤔

[dturn]

I should have been more clear, I was thinking it could steal unused capacity. We have some wiggle room here for what that means, probably based on the usage in the last time period.

[jszwedko]

I should have been more clear, I was thinking it could steal unused capacity. We have some wiggle room here for what that means, probably based on the usage in the last time period.

Ah I see. I think we'd need to get specific about exactly how it decides what capacity is "unused" but I think I understand the overall intent here.

[dturn]

What is the process for moving a proposal like this forward?

[jszwedko]

I think a proposal like this where we still need to sort out the details of how it would work would be best served by a short RFC (see https://github.com/vectordotdev/vector/blob/master/rfcs/README.md) to describe the proposal in more detail. Is that something you'd be interested in taking forward?