-
Notifications
You must be signed in to change notification settings - Fork 219
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Wrong last consumed offset when using transaction #911
Comments
Could it be related to the fact of doing manual commit ? i.e no "real" message are consumed, so the consumer loop doesn't trigger and it is not committing offsets ? Any idea how a consuming loop could trigger a commit on transactional messages ? |
We're also seeing this issue. It feels like the library should support iterating over the end of transaction record. Both of the |
Describe the bug
Here what I posted on the kafka user mailing list
and the response I got
Expected behaviour
So as mentioned by the response I got, when a consumer receive a control message, it should "step over" in term of offset consumed. To be honest, I am not 100% sure that the behavior of the java KafkaConsumer, as kafka internal tool (like kafka-consumer-groups.sh) that reports the lag, also show an offset of 1 or 2
Environment (please complete the following information):
python -c "import aiokafka; print(aiokafka.__version__)"
): 0.8.1python -c "import kafka; print(kafka.__version__)"
): 2.0.2kafka-topics.sh --version
): 2.8.1Reproducible example
Not a reproducable example, but here my recovery code
and instead of
while recovering
when I am trying to do something like offset < end_offset (or end_offset -1, I never remember the logic here), it doesn't work when the latest messages are control onesThe text was updated successfully, but these errors were encountered: