Today, I'm attempting to kill Kafka on my local machine and make the consumer lag forever

And then trying to fix it, so it rejoins the group, or the Kafka broker never reaps the connection to the consumer

I've mostly managed to fix the issue...

It comes down to session timeout being too low, heartbeat being too low, and the consumer being set to consume the topic off the latest offset, rather than the earliest!

It also looks like our consumers are consuming topics fast enough, and should be applying back pressure to Kafka, rather than consuming too many messages, then not committing an offset..

This means we should be basically batch consuming, e.g. consume 10 messages, do work, consume 10 more messages, rather than consuming a stream of events!

Follow

Time for another follow up. Looks like I was right about batch consuming.

What's actually happening is our consumer is committing offsets asynchronously, e.g.:

Receive message 0: process it
Receive message 1: process it

Except in the above case, we finish message 1 and commit, and then _after_ that we finishing process message 0 and commit. This means that although we've processed message 1, Kafka thinks our offset is still message 0!

Sign in to participate in the conversation
Melbourne Mastodon

Melbourne Social is a small Mastodon instance for users from/interested in/living in Melbourne, Australia πŸ“