Apache Kafka Tutorial on Local Windows Machine

This tutorial will show you how to do the following:

• How to install Apache Kafka on your local Windows 10 desktop or laptop
• How to configure the Kafka service
• How to create a Kafka Topic
• How to send (Kafka Producer example) a message to the Kafka Topic
• How to receive (Kafka Consumer example) a message from the Kafka Topic

You will need to download the following tools/software and install them on your local computer:

• JDK - https://www.oracle.com/java/technologies/javase-jdk14-downloads.html
• Kafka - https://kafka.apache.org/downloads
• 7-Zip - https://www.7-zip.org/download.html - Use this to unzip the downloaded Kafka installer

After installing all the above, the first step is to configure the following:

• Zookeeper – zookeeper.properties file
• Kafka – server.properties file

Zookeeper is a key-value based configuration service for distributed systems by Apache. Follow the steps outlined below to configure Zookeeper for Kafka.

• Navigate to the local path C:\software\kafka\config OR where ever you have placed your Kafka folder after unzipping
• Open zookeeper.properties file in a text editor of your choice
• Change “dataDir” path to point to the location where you have placed your kafka folder – make sure you use front slashes (/) in the path otherwise there will be an error during starting Zookeeper

• Save the file and close it
• Open windows Command Prompt
• Browse to the Kafka root directory and issue the following command to start the Zookeeper service

• After hitting enter, if everything is right, the Zookeeper service will start and you should see a lot of verbose logs on the screen. If you slowly scroll down, you will notice that the Zookeeper service has started by binding to port number 2181 on your local computer

• Now if you browse back to the Kafka root folder you will notice that a folder titled “zookeeper-data” has been created

• Minimize this Command Prompt window so that Zookeeper keeps running and proceed to the next step.

The next steps is to start the Kafka message broker service. Follow the steps outlined below.

• Navigate to the local path C:\software\kafka\config OR where ever you have placed your Kafka folder after unzipping
• Open server.properties file in a text editor of your choice
• Change “log.dirs” path to point to the location where you have placed your kafka folder – make sure you use front slashes (/) in the path otherwise there will be an error during starting Kafka

• Save the file and close it
• Open a new Command Prompt window
• Browse to the Kafka root directory and issue the following command to start the Kafka service

• After hitting enter, if everything is right, the Kafka service will start and you should see a lot of verbose logs on the screen. If you slowly scroll down to the end, you will notice that the Kafka service has started on your local computer

• Now if you browse back to the Kafka root folder you will notice that a folder titled “kafka-log” has been created

• Minimize this Command Prompt window so that Kafka keeps running and proceed to the next step

Now let a create a Kafka Topic, so that we can start sending and receiving message to and from it respectively. The instructions to create a Kafka Topic are outlined below:

• Open windows Command Prompt
• Browse to the Kafka root directory and issue the following command to create a new Topic

• After hitting enter, if everything is right, a new Topic will get created
• You may check whether the Topic really got created by issuing the following command

So now you are all set to send and receive messages to this Topic named “Test_Topic_1”. Let’s do it!

Execute the following steps to send (produce) messages to the Topic:

• Open windows Command Prompt
• Browse to the Kafka root directory and issue the following command to start sending messages to the newly created Topic – note that each line is treated as a separate message

Execute the following steps to receive/read (consume) messages from the Topic:

• Open windows Command Prompt
• Browse to the Kafka root directory and issue the following command to start receiving/reading messages from the newly created Topic – note the BAT file supplied by Apache has many options that can be explored further

Given below is a bird’s eye view of my computer screen depicting the running Kafka services and the producer and the consumer.

Glossary of useful commands

cd software\kafka

.\bin\windows\zookeeper-server-start.bat .\config\zookeeper.properties

cd software\kafka

.\bin\windows\kafka-server-start.bat .\config\server.properties

cd software\kafka

.\bin\windows\kafka-topics.bat --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic From_Streams_Processor_Topic

cd software\kafka

.\bin\windows\kafka-console-producer.bat --broker-list localhost:9092 --topic Test_Topic_1

cd software\kafka

.\bin\windows\kafka-console-consumer.bat --bootstrap-server localhost:9092 --topic Test_Topic_1 --from-beginning