Getting Started with Apache Kafka and Node.js: A Practical Guide

Apache Kafka has emerged as a cornerstone technology for building real-time streaming applications that efficiently handle massive amounts of data. Its ability to publish and subscribe to streams of records, store data durably, and provide fault tolerance makes it a powerful tool for various use cases. In this article, we will delve into using Apache Kafka with Node.js, showcasing a simple example of a Kafka producer and consumer using the node-rdkafka library.

Prerequisites

Before you dive into the code examples, please make sure that you have Apache Kafka installed and running in your environment. You can download Kafka from the official Apache Kafka website and follow their installation instructions.

Setting up Kafka with Node.js

1. Install Dependencies

First, we must install the necessary Node.js package, node-rdkafka, which serves as a Kafka client for Node.js. Open your terminal and run the following command:

npm install node-rdkafka        

2. Creating a Kafka Producer

The Kafka producer is responsible for sending messages to Kafka topics. Here's a simple example of a Kafka producer in Node.js:

const kafka = require('node-rdkafka')

const producer = new kafka.Producer({
  'metadata.broker.list': 'localhost:9092', // Replace with your Kafka broker address
  'dr_cb': true
});


producer.on('ready', () => {
  console.log('Producer ready');

  const topic = 'test';
  const message = 'Hello, Kafka!';

  producer.produce(topic, null, Buffer.from(message));
});


producer.on('delivery-report', (err, report) => {
  if (err) {
    console.error('Error delivering message:', err);
  } else {
    console.log('Message delivered:', report);
  }
});


producer.connect();        

3. Creating a Kafka Consumer

The Kafka consumer subscribes to topics and processes incoming messages. Here's a simple example of a Kafka consumer in Node.js:

const kafka = require('node-rdkafka')

const consumer = new kafka.KafkaConsumer({
  'group.id': 'test-group',
  'metadata.broker.list': 'localhost:9092', // Replace with your Kafka broker address
  'auto.offset.reset': 'earliest'
});

consumer.on('ready', () => {
  console.log('Consumer ready');

  consumer.subscribe(['test']);
  consumer.consume();
});

consumer.on('data', (message) => {
  console.log('Received message:', message.value.toString());
});

consumer.on('event.error', (error) => {
  console.error('Consumer error:', error);
});

consumer.connect();        

Running the Example

1. Open two terminal windows.

2. In one terminal, navigate to the directory containing the producer script and run it:

node producer.js        

3. In the second terminal, navigate to the directory containing the consumer script and run it:

node consumer.js        

Conclusion

Apache Kafka offers a robust solution for real-time data streaming and processing, and integrating it with Node.js using the node-rdkafka library makes it accessible and efficient. This article provided a hands-on introduction to setting up a Kafka producer and consumer using Node.js, allowing you to start building scalable and responsive data streaming applications. As you delve deeper into Kafka and Node.js, you'll uncover even more advanced features and possibilities for harnessing the power of real-time data streams.

Github Repo: kafka-node-app