Kafka Internals For Troubleshooting
It is not necessary to know Kafka internals in order to run it but understanding these internals helps to provide context during troubleshooting.
In this article, I would cover how replication works, how consumer-producer requests are handled, and how replica partition allocation happens.
Replication: The heart of Kafka’s architecture
The first sentence of Kafka documentation describes it as A distributed, Partitioned, Replicated commit log service. As we know data in Kafka is organized by topics and each topic is partitioned and each partition can have multiple replicas. Kafka guarantees availability and durability by using these replicas as if the leader replica goes down user won’t suffer from data loss as a new replica would be assigned as a leader. Now let’s first discuss types of replicas:
Leader Replica: Each partition assigns a single replica as the leader and all producer and consumer requests come to it in order to guarantee consistency. Leader Replica also maintains which of the followers are in an up-to-date state with itself.
Follower Replicas: Their only job is to replicate the message from leader Replica and stay up-to-date with the most recent consumer message. They make fetch requests to leader replica and that too happens in order. For example, a follower would only request message 2 when it has received message 1 so that way leader would also know the replication state of the follower. They don’t serve any client requests. In an event when the leader crashes one of the followers would be promoted as a leader.
Handling Producer-Consumer Requests
Before understanding how Broker handles client(producer-consumer) requests, let’s first understand how the client discovers the broker from which it has to fetch data. Kafka client makes a metadata request to any broker which includes a list of topics in which the client is interested. The server response tells about the location of partitions and replicas of that topic. This metadata request can be made to any broker as all the broker contains this information. The client generally caches this information and contacts a specific broker containing leader replica.
Since the client has figured out the leader replica, now it will start making produce requests to that topic. On receiving that request broker does few validations like checking to write privileges of the user sending data, the number of acks specified(0,1, all) then it will write data to leader partition. Then based upon the value of ack it will respond.
Since we have the message stored at our partition some of the applications might want to fetch data for their use. So the client from these applications makes a fetch call to broker specifying a list of topics, partition, and offsets along with the limit of data that it can process at one go. On receiving the request broker again do necessary checks like the existence of the requested offset. If the checks are passed then the broker will read from that offset to the limit specified by the client and will return it back to the client. Kafka's client also provides the option of setting up lower limits of fetched data.
One important thing to take care of is that the consumer client can only make fetch calls for messages that are already replicated to in-sync replicas. Below is the illustration for the same:
Physical Storage
Partition replica is the basic storage limit. They cannot be split over multiple brokers nor over multiple disks in the same broker. So we can infer that size of a partition is limited by space available on a single mount point(single disk).
There are various ways in which partition allocation can happen I am not going to spend time here but the main intention is to put replicas to different brokers such that availability is maximized.
Note: Allocation of partitions to the broker does not take available space or existing load into account but it takes the number of partitions into account. Which means if some broker has more disk space than others, some partitions can be abnormally large, so you need to be careful with partition allocation
So this was all about internals of Kafka, hope I was able to explain it in simpler terms.
