cft
Become a CreatorSign inGet Started

Using Kafka as a Message Queue

Kafka is just there. Anyone doing big-data, streaming analytics, anything that requires reading in lots of data from different sources and pumping it into a processing system, is either using Kafka or evaluating it. It’s that successful.


user

Barak Merimovich

3 months ago | 1 min read
Follow

using-kafka-messagequeue-kn05m

I don’t usually just re-post stuff, but this article really hit the spot for me.

I like message queues – they are a wonderfully useful abstraction. Want to send a message from one component to another in a distributed system? Don’t let the components talk to each other directly – decoupling your components makes them more robust, and makes it far easier to swap in other service implementations at a later date.

But I am not here to discuss message queues in general. I’m here to discuss everyone’s (well, most people’s) favorite pub/sub system – Apache Kafka.

Kafka is just there. Anyone doing big-data, streaming analytics, anything that requires reading in lots of data from different sources and pumping it into a processing system, is either using Kafka or evaluating it. It’s that successful.

But it’s not a message queue. It was not designed for it, and does not make any pretense to compete with the likes of RabbitMQ and ActiveMQ in this type of scenario.

But it’s there. If you already have a Kafka cluster all set-up and running nicely, maybe even running across data-centers (Kafka was built for this!), and you don’t need spectacular performance from your Message Queue, can you somehow use Kafka and spare yourself the setup and maintenance of another clustered system?

Originally published here!

Upvote


user
Created by

Barak Merimovich

Follow

Head of Architecture at Logz.io


people
Post

Upvote

Downvote

Comment

Bookmark

Share


Related Articles