sentry ingest consumer transactions max batch size

[letnab]

Environment

self-hosted (https://develop.sentry.dev/self-hosted/)

What are you trying to accomplish?

increase the number of events processed
sentry ingest consumer transactions

How are you getting stuck?

Now the application reads about 100-150 messages from Kafka. This is very little. The lag in Kafka is growing. As soon as it reaches 1 million the application crashes with an error
arroyo.errors.OffsetOutOfRange: KafkaError{code=_AUTO_OFFSET_RESET,val=-140,str="fetch failed due to requested offset not available on the broker: Broker: Offset out of range (broker 1)"}

there is no max-batch-size parameter anymore

Where in the product are you?

Issues

Link

No response

DSN

No response

Version

24.4.1

[getsantry]

Assigning to @getsentry/support for routing ⏲️

[aldy505]

Usually this happens because some of your processor containers (on the docker-compose.yml it's something around sentry worker, sentry post-processor, snuba consumer or sentry consumer) don't have enough processing power. I'd suggest to increase your CPU cores for this one.

For the Kafka offset out of range problem, you should see #2951

[getsantry]

This issue has gone three weeks without activity. In another week, I will close it.

But! If you comment or otherwise update it, I will reset the clock, and if you remove the label Waiting for: Community, I will leave it alone ... forever!

---

"A weed is but an unloved flower." ― Ella Wheeler Wilcox 🥀