thelinuxvault blog

What is Apache Kafka?

In the modern era of big data and real - time data processing, the ability to handle large volumes of data streams efficiently is crucial. Apache Kafka is a powerful open - source distributed event streaming platform that has become a cornerstone in many data - intensive applications. It was initially developed at LinkedIn and later open - sourced in 2011. Kafka provides a high - throughput, fault - tolerant, and scalable solution for handling real - time data feeds.

2026-06

Table of Contents#

  1. Key Concepts
  2. Architecture of Apache Kafka
  3. How Kafka Works
  4. Common Use Cases
  5. Best Practices
  6. Example Usage
  7. Conclusion
  8. References

Key Concepts#

Topics#

A topic is a category or feed name to which records are published. It is similar to a table in a database or a folder in a file system. Topics are partitioned, which means that a topic can be split across multiple brokers (servers) in a Kafka cluster. This partitioning allows Kafka to scale horizontally, handling large amounts of data.

Producers#

Producers are responsible for publishing records to topics in Kafka. They can send messages to specific partitions within a topic. Producers can be configured to send messages in a round - robin fashion across partitions or based on a custom partitioning strategy.

Consumers#

Consumers read records from topics. They belong to consumer groups. A consumer group is a set of consumers that work together to consume records from a topic. Each partition in a topic can be assigned to at most one consumer within a consumer group. This allows for parallel processing of messages.

Brokers#

Brokers are the servers in a Kafka cluster. Each broker stores a subset of the partitions of the topics in the cluster. Brokers communicate with each other to maintain the overall state of the cluster. They handle the storage and retrieval of messages.

Zookeeper#

Zookeeper is used by Kafka for cluster management. It stores metadata about the Kafka cluster, such as the list of brokers, the partition assignments, and the consumer group offsets. Kafka uses Zookeeper to ensure that the cluster remains stable and consistent.

Architecture of Apache Kafka#

Multi - Broker and Multi - Partition Architecture#

Kafka clusters typically consist of multiple brokers. Each topic can have multiple partitions, and these partitions are distributed across the brokers. This architecture provides high availability and scalability. If a broker fails, the partitions on that broker can be replicated on other brokers, ensuring that data is not lost.

Replication#

Kafka uses replication to ensure data durability and availability. Each partition can have multiple replicas. One of the replicas is designated as the leader, and the others are followers. Producers send messages to the leader partition, and the followers replicate the data from the leader. If the leader fails, one of the followers is elected as the new leader.

How Kafka Works#

Message Production#

When a producer wants to send a message to a topic, it first serializes the message into a byte array. It then determines the partition to which the message should be sent. If a key is provided, the producer can use a hashing function to map the key to a specific partition. Once the partition is determined, the producer sends the message to the leader of that partition.

Message Storage#

The leader partition stores the message on disk. Kafka uses a log - based storage system, where messages are appended to the end of the log. This makes writing messages very fast. The log is segmented into files, and old segments are deleted or archived based on a retention policy.

Message Consumption#

Consumers subscribe to topics and read messages from the partitions assigned to them. They maintain an offset, which indicates the position of the last message they have read. When a consumer reads a message, it can commit the offset, which ensures that in case of a failure, the consumer can resume reading from the correct position.

Common Use Cases#

Log Aggregation#

Kafka can be used to collect and aggregate logs from multiple sources, such as web servers, application servers, and database servers. The logs can be sent to Kafka topics, and then processed by other systems for analysis and monitoring.

Real - time Analytics#

Kafka is well - suited for real - time analytics applications. It can handle high - volume data streams from various sources, such as IoT devices, social media platforms, and financial transactions. Analytics tools can consume the data from Kafka topics in real - time to generate insights.

Event - Driven Architectures#

In event - driven architectures, Kafka can be used as a central event bus. Applications can publish events to Kafka topics, and other applications can subscribe to these topics to react to the events. This allows for loose coupling between components and enables scalable and flexible systems.

Best Practices#

Topic Design#

  • Partitioning: Choose the number of partitions based on the expected throughput and the number of consumers. More partitions allow for higher parallelism, but also increase the management overhead.
  • Replication Factor: Set an appropriate replication factor to ensure data durability. A replication factor of 3 is commonly used in production environments.

Producer Configuration#

  • Acks: Configure the acks parameter to control the level of durability. Setting acks = all ensures that the producer waits for all replicas to acknowledge the message, providing the highest level of durability.
  • Buffering: Use producer buffering to improve performance. The producer can buffer messages and send them in batches, reducing the number of network requests.

Consumer Configuration#

  • Offset Management: Use automatic offset commits carefully. Manual offset management gives more control over the consumption process, especially in cases where the consumer needs to handle messages in a specific order.
  • Consumer Group Management: Ensure that the number of consumers in a consumer group is appropriate for the number of partitions in the topic. Having more consumers than partitions can lead to some consumers being idle.

Example Usage#

Java Producer Example#

import org.apache.kafka.clients.producer.*;
import java.util.Properties;
 
public class KafkaProducerExample {
    public static void main(String[] args) {
        Properties props = new Properties();
        props.put("bootstrap.servers", "localhost:9092");
        props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
        props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");
 
        Producer<String, String> producer = new KafkaProducer<>(props);
        String topic = "test_topic";
        String key = "key1";
        String value = "Hello, Kafka!";
 
        ProducerRecord<String, String> record = new ProducerRecord<>(topic, key, value);
        producer.send(record, new Callback() {
            @Override
            public void onCompletion(RecordMetadata metadata, Exception exception) {
                if (exception != null) {
                    System.out.println("Error sending message: " + exception.getMessage());
                } else {
                    System.out.println("Message sent successfully. Offset: " + metadata.offset());
                }
            }
        });
 
        producer.close();
    }
}

Java Consumer Example#

import org.apache.kafka.clients.consumer.*;
import java.time.Duration;
import java.util.Collections;
import java.util.Properties;
 
public class KafkaConsumerExample {
    public static void main(String[] args) {
        Properties props = new Properties();
        props.put("bootstrap.servers", "localhost:9092");
        props.put("group.id", "test_group");
        props.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
        props.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
 
        KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props);
        String topic = "test_topic";
        consumer.subscribe(Collections.singletonList(topic));
 
        while (true) {
            ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(100));
            for (ConsumerRecord<String, String> record : records) {
                System.out.printf("offset = %d, key = %s, value = %s%n", record.offset(), record.key(), record.value());
            }
        }
    }
}

Conclusion#

Apache Kafka is a versatile and powerful platform for handling real - time data streams. Its distributed architecture, high throughput, and fault - tolerance make it suitable for a wide range of applications. By understanding the key concepts, architecture, and best practices, developers can effectively use Kafka to build scalable and reliable data - intensive systems.

References#