Kafka hands-on Guide to using publish-subscribe based messaging system (PART II)

The previous article covers the basic concepts of Kafka you should know before starting on coding.

In the coding session, we will learn the following things:-

1. Setup Kafka on your machine using docker
2. Use Kafka Console Producer to produce on a particular topic.
3. Use Kafka Console Consumer to consume messages on the same topic.

Tip:-

If you install `librdkafka` library you can directly use Kafka commands without providing an absolute path.

Use this command to install librdkafka

brew install librdkafka

1. Install Kafka using docker

In your project inside docker-compose.yml add following code snippet.

docker-compose.yml

Up your docker container using the following command.

docker-compose up -d

you can see whether Kafka server and zookeeper started on port 9092 and 2181 respectively. for checking this use docker ps command.

docker ps

Kafka server and zookeeper started.

Next step is to open server.properties. uncomment following line.

vim /usr/local/etc/kafka/server.properties
*********************************************
listeners=PLAINTEXT://localhost:9092

Create a Topic on which we will use to produce the message

create topic kafka-topics — create — zookeeper localhost:9092 — replication-factor 1 — partitions 0 — topic testing

If you want to see the list of topics use the following the command.

kafka-topics — zookeeper localhost:2181 — list

2. Kafka console producer — to produce messages using the Topic name.

kafka-console-producer — broker-list localhost:2181 — topic hello

one you start Kafka producer on 9092 port you will need to start Kafka consumer to listen to the data. you can start consumer using the following command.

3. Kafka console consumer to consume messages using the Topic name.

kafka-console-consumer --bootstrap-server localhost:2181 --topic hello

Kafka console producer and consumer streaming

Bonus:-

I have implemented kafka console producer and kafka console consumer using the following problem statement:-

Write a fronting REST service which put the message on Kafka. Use kafka console producer. — Link to kafka console producer sourcecode

Implement gRPC service which will eventually read the message from the kafka topic and stores it into the Postgres database. Link to kafka console consumer sourcecode

I faced a lot of issues while understanding the kafka producer and consumer process. If you are trying and facing issues feel free to discuss your issue and I will more than happy to solve an issue with you.!

If you have found this blog useful Click the 💚 below to show your support :)