Time for another follow up. Looks like I was right about batch consuming.
What's actually happening is our consumer is committing offsets asynchronously, e.g.:
Receive message 0: process it
Receive message 1: process it
Except in the above case, we finish message 1 and commit, and then _after_ that we finishing process message 0 and commit. This means that although we've processed message 1, Kafka thinks our offset is still message 0!
Melbourne Social is a small Mastodon instance for users from/interested in/living in Melbourne, Australia 📍