How to implement backpressure in custom input

[fearfate]

I'd added a git input, which will watch the git repository and read the files. while watching a big repository, this will going to leader to a oom crash.

[mihaitodor]

Hey @fearfate 👋 Hard to say without looking at the code, but, in general, the output dictates how many messages (or message batches) end up being in flight at any given time. You can also add a rate_limit processor to artificially throttle the flow. If your input buffers data preemptively, then it's up to you to make sure that buffer doesn't blow up. You could even use the API to access a configured rate limit inside your input and throttle message consumption based on it, but that's probably overkill.