Curious what was the catalyst to switch from Redis to Kafka? Reliability that a given message was received and processed and replaying that to a different consumer in the event of failure?
mostly just scale. Keeping your queue in memory isnt necesarilly a requirement unless you have a lambda arch or a realtime requirement (which we started with). For our larger operations, kafka, which is disk backed, distributed with a little more ease and cheaper was a nice option. For reliability, we were using some of storms primitives as well as the mechanisms inside qless, the queuing library we were using on top of redis. https://github.com/ChannelIQ/qless-java
