beginnerintermediateadvanced

System Design · Section 28

Kafka and Log-Based Messaging

Level: advanced
Read: 24 min
Concepts: 4

A Kafka topic is split into partitions, each an independently ordered, append-only log — Kafka guarantees message order within a partition, never across partitions of the same topic, which makes the partition key (what determines which partition a message lands in) a genuine ordering decision. Consumer groups split a topic's partitions among consumers, capping parallelism at the partition count. Kafka retains messages instead of deleting them on consumption, making a topic a durable, replayable log that multiple independent consumer groups can each read from their own position — the foundation of event-driven architecture. Delivery is honestly at-least-once for most real pipelines; Kafka's "exactly-once" semantics are real but scoped to Kafka-to-Kafka processing, not to external side effects a consumer triggers.

This section

Apache Kafka: a Distributed Messaging System for Log ProcessingGaurav Sen

What is true here

Ordering is guaranteed within a partition only — the partition key decides what gets that guarantee.
Consumer groups cap parallelism at partition count — one partition, one consumer within a group, at a time.
Retention means consumption never deletes a message — a topic is a durable, replayable log.
Exactly-once semantics are real but scoped to Kafka-internal pipelines — external side effects need their own idempotency.
Partition count and partition key are hard-to-reverse decisions that affect both throughput and correctness.

What you will be able to do

Reason correctly about what ordering guarantee a given partitioning scheme actually provides
Choose the right delivery semantic and know exactly where Kafka's exactly-once guarantee stops applying
Use replay to backfill a new consumer from a topic's retained history
Anticipate the ordering risk of changing partition count on a live topic

The core model and its guarantees

Topics, partitions, offsets and consumer groups, and exactly what delivery guarantee each configuration actually provides.

Topics, partitions, offsets and consumer groups

coreadvanced

A Kafka topic is a named stream of messages, physically split into partitions for parallelism — each partition is an ordered, append-only log with its own offset counter. Kafka only guarantees message order within a single partition, never across partitions of the same topic. Consumer groups let multiple consumers split a topic's partitions between them, each partition read by exactly one consumer in the group at a time.

Think of it as

A topic is a multi-lane highway, and each partition is one lane — cars (messages) in the same lane keep their relative order, but a car in lane 2 has no guaranteed relationship to a car in lane 5's position. A consumer group is a team of toll workers, one assigned per lane at a time — more workers than lanes just means some workers stand idle with nothing to do.

text

topic "orders" → partition 0: [msg0, msg1, msg2, ...]
                → partition 1: [msg0, msg1, msg2, ...]
                → partition 2: [msg0, msg1, msg2, ...]

consumer group "billing":
  consumer A ← partition 0
  consumer B ← partition 1, partition 2

What we're doing: Show why ordering only holds within a partition, using an order-events topic keyed by order_id.

partition-ordering.txttext

Topic "order-events", 3 partitions, keyed by order_id
(Kafka routes all messages with the same key to the
same partition, deterministically).

order_id=42's events: "created", "paid", "shipped"
  → all hash to partition 1
  → guaranteed to be read in that exact order.

order_id=99's events: "created", "cancelled"
  → hash to partition 0
  → also guaranteed in order relative to EACH OTHER.

But order 42's "created" and order 99's "created" have
NO guaranteed relative order — they're in different
partitions, processed independently.

6: Keying by order_id is what makes per-order ordering possible — same key always routes to the same partition.
13: This is the actual scope of the ordering guarantee — it never extends across partitions, even within the same topic.

Why this works: Choosing the right partition key is what determines what ordering guarantee a consumer actually gets — a design that needs strict ordering for a given entity (like one order's lifecycle) has to key on that entity's ID, not leave partitioning to a round robin.

Assuming a Kafka topic guarantees global message order

Wrong

text

"We're using Kafka, so events are processed
in the exact order they were published."

Better

text

"Order is guaranteed only within a partition.
Events for the same entity (keyed to the same
partition) are ordered relative to each other;
events across different entities/partitions
have no guaranteed relative order — design
consumers accordingly, and choose the
partition key deliberately."

What you see: A consumer processes a "shipped" event for one order before the "created" event for a completely different order, and a team is surprised — because global ordering across a whole topic was never actually a guarantee Kafka made, only per-partition ordering was.

Why: Kafka scales partitions in parallel specifically by giving up a global ordering guarantee — the trade is deliberate: partitioning is what allows a topic's throughput to scale past a single log's write speed, at the cost of only guaranteeing order within each individual partition.

Topic "orders" — 3 partitions, 2 consumers

Topic: orders

Partition 0

Partition 1

Partition 2

Consumer group: billing

Consumer A

Consumer B (×2)

Kafka's core vocabulary

Kafka's core vocabulary
Term	What it is
Topic	A named stream of messages
Partition	One ordered, append-only log within a topic
Offset	A message's position within its partition
Consumer group	Consumers that split a topic's partitions between them
Retention	How long messages are kept, independent of whether they were read

Remember: A topic's partitions are independently ordered logs — Kafka guarantees order within a partition, never across partitions. Consumer groups split partitions among consumers, one partition per consumer at a time, so parallelism is capped by partition count.

At-least-once, at-most-once and practical exactly-once

coreadvanced

At-least-once means a message may be delivered more than once but never lost — the common default, requiring idempotent consumers. At-most-once means a message may be lost but never duplicated — rarely what anyone actually wants. "Exactly-once" is achievable within Kafka itself (via idempotent producers and transactions) for Kafka-to-Kafka pipelines, but the moment a side effect touches something outside Kafka (a database write, an external API call), true exactly-once delivery becomes effectively impossible to guarantee end-to-end.

Think of it as

At-most-once is dropping a letter in a mailbox with no tracking — if it's lost, you never find out or resend. At-least-once is a courier who keeps redelivering until they get a signature — reliable, but a lost signature (not a lost letter) means a second, duplicate delivery. Kafka's exactly-once is a courier service that can guarantee single delivery only within its own depot network — the moment the package leaves for an outside carrier (an external system), that guarantee no longer applies.

text

at-most-once:  ack BEFORE processing (or no retry)
at-least-once: ack AFTER processing succeeds, retry
               on failure/timeout
exactly-once:  idempotent producer + transactions,
               scoped to Kafka-to-Kafka pipelines

What we're doing: Show why a consumer that writes to an external database still needs idempotency even under Kafka's exactly-once semantics.

exactly-once-boundary.txttext

Kafka Streams pipeline: reads from topic "orders",
transforms, writes to topic "shipping-queue" — this
part genuinely gets exactly-once semantics, entirely
within Kafka's transactional guarantees.

A downstream consumer reads "shipping-queue" and
calls an external shipping API to create a shipment.

Sequence: consumer calls the shipping API, the API
call SUCCEEDS, but the consumer crashes before
committing its Kafka offset.

Result: on restart, the consumer re-reads the same
message (Kafka doesn't know the shipping API call
already succeeded) and calls the shipping API AGAIN
— a duplicate shipment, despite Kafka's own internal
exactly-once guarantee being fully intact.

4: This part really is exactly-once — Kafka's transactional guarantee genuinely holds here.
15: This is the boundary crossing: the external API call has no way to participate in Kafka's transaction, so its own success can't be atomically tied to the offset commit.

Why this works: This is the honest limit of "exactly-once" — it is a real, strong guarantee for pipelines that stay inside Kafka, and it stops being a guarantee the instant a side effect (an external API call, a non-Kafka database write) is involved, no matter how the pipeline is configured.

Building an external side effect assuming Kafka's exactly-once semantics cover it

Wrong

text

"We enabled exactly-once semantics in our
Kafka config, so our shipping API calls will
never be duplicated."

Better

text

"Exactly-once semantics cover Kafka-to-Kafka
processing. For the shipping API call, we need
our own idempotency — e.g. an idempotency key
per order, so a duplicate call from a redelivery
has no additional effect."

What you see: A duplicate external side effect (a duplicate shipment, a duplicate charge) occurs despite "exactly-once" being enabled — because the guarantee was scoped to Kafka's internal pipeline, not to the external system the consumer called.

Why: Kafka's exactly-once semantics work by making the offset commit and the Kafka write atomic within a transaction — an external system has no way to join that transaction, so its own success or failure can never be made atomic with the offset commit, regardless of Kafka configuration.

Three delivery semantics

At-most-once

can lose, never duplicates

At-least-once

never loses, can duplicate

"Exactly-once"

Kafka-to-Kafka only

The three delivery semantics

The three delivery semantics
Semantic	Can lose a message?	Can duplicate a message?	Typical use
At-most-once	Yes	No	Metrics/logs where an occasional loss is acceptable
At-least-once	No	Yes	The common default — pair with idempotent processing
"Exactly-once" (Kafka-to-Kafka)	No	No, within Kafka's transactional boundary	Kafka Streams pipelines that stay entirely within Kafka

Remember: At-most-once can lose messages, never duplicates. At-least-once never loses messages, can duplicate — pair it with idempotent consumers. Kafka's "exactly-once" is real but scoped to Kafka-to-Kafka pipelines — any external side effect needs its own idempotency.

Replay and partitioning as deliberate decisions

Why retention enables replay and event-driven architecture, and why partitioning is a scaling AND ordering choice, not a storage detail.

Replayable event logs and event-driven architecture

coreadvanced

Because Kafka retains messages instead of deleting them once read, a topic functions as a durable, replayable log rather than a transient queue — any consumer can re-read from an earlier offset, and a brand-new consumer can process the entire history from the beginning. This is the core capability that makes Kafka a fit for event-driven architectures, where multiple independent services each want their own full view of the same event stream.

Think of it as

A traditional queue is a single shared inbox where reading a letter and throwing it away happen together — once read, it's gone for everyone. A Kafka topic is a public bulletin board instead: pinning a notice up doesn't remove it, so anyone who walks by — today or next week — can still read it, and a brand-new person can catch up on everything posted since the board went up, not just what's posted from the moment they arrive.

text

topic "orders" (retained 7 days)
  ├─ consumer group "billing"    reads from offset 500
  ├─ consumer group "analytics"  reads from offset 0
  └─ consumer group "shipping"   reads from offset 480
-- each group tracks its own independent position

What we're doing: Show a new service using replay to backfill from a Kafka topic's history, rather than only seeing new events.

replay-backfill.txttext

Topic "order-events" has 30 days of retention and
has been running for 6 months (many millions of
events retained within that rolling 30-day window).

A new "fraud-detection" service is deployed. Instead
of only seeing events from its deployment moment
forward, it creates a new consumer group and resets
its offset to the earliest available message.

Result: fraud-detection processes the full 30 days
of retained history first, building up its internal
model from real historical data, then seamlessly
continues consuming new events as they arrive —
no separate backfill pipeline or data export needed.

7: This is the replay capability itself — a brand-new consumer group choosing to start from the earliest retained offset instead of "now."
10: The service gets real historical data for free, purely because Kafka retained it — a transient queue could never offer this.

Why this works: This is the specific capability a traditional message queue does not offer — once a queue message is consumed, it's gone, so a new consumer only ever sees the future, never the past; Kafka's retention model turns the topic itself into reusable history.

Treating a Kafka topic exactly like a traditional queue, assuming consumption removes a message

Wrong

text

"Once our billing consumer processes an order
event, it's gone — no other service can use it
anymore."

Better

text

"Consumption doesn't remove anything — the
event stays in the topic (until retention
expires) and any number of other consumer
groups can independently read it, including
services that don't exist yet today."

What you see: A team builds a separate data pipeline or database export just to give a new service access to historical events, not realizing the Kafka topic already retains that history and a new consumer group could simply read it directly.

Why: The mental model carried over from traditional queues — "a message is gone once consumed" — is exactly backwards for Kafka, and missing this leads to solving an already-solved problem (getting historical data to a new service) with unnecessary extra infrastructure.

One topic, three independent readers

Independent consumer groups

billing @500

analytics @0

shipping @480

Remember: Kafka retains messages instead of deleting them on consumption, so a topic is a durable, replayable log — multiple independent consumer groups can each read it from their own position, and a new service can replay full history rather than only seeing events from its own start time.

Partitioning is a scaling AND ordering decision

standardadvanced

How many partitions a topic has, and what key routes a message to a partition, are not implementation details to set once and forget — partition count caps consumer parallelism, and the partition key determines what ordering guarantee consumers actually get. Both choices are hard to change later without real operational cost, so they deserve the same deliberate attention as a database's shard key.

Think of it as

Partition count is like deciding how many checkout lanes a store opens — more lanes serve more customers at once, but idle lanes (more than needed) waste staff. The partition key is like which line a customer joins based on their last name — as long as the rule stays fixed, the same customer always lands in the same line, keeping their repeat visits in a predictable order; change the rule (the number of lines) and returning customers can suddenly be routed to a different line than before.

What we're doing: Show increasing partition count breaking a previously-working ordering assumption.

repartition-breaks-ordering.txttext

Topic "order-events", 4 partitions, keyed by
order_id. hash(order_id=42) % 4 = partition 2 —
every event for order 42 has always landed there,
so consumers relying on that consistency see order
42's full event history in order.

Partition count increased to 8 to add more consumer
parallelism. Now hash(order_id=42) % 8 = partition 6
— a DIFFERENT partition than before.

New events for order 42 go to partition 6; old ones
remain in partition 2. A consumer reading order 42's
history now has to read from TWO partitions to see
the full, correctly-ordered sequence — the "ordered
within one partition" guarantee silently broke for
every order whose partition assignment changed.

8: This is the exact moment the guarantee breaks — the same key now maps to a different partition purely because the modulus changed.
12: The consequence is subtle and easy to miss in testing — it only shows up for entities whose events span the repartitioning moment.

Why this works: This is why partition count is not "just a storage/throughput knob" — changing it can silently invalidate an ordering guarantee that downstream consumers were built assuming would always hold, without any error or warning at the moment it happens.

Treating partition count as freely adjustable without considering ordering consumers depend on

Wrong

text

"Throughput is a bit low — let's just bump
partition count from 4 to 8, it's a config
change."

Better

text

"Bumping partition count changes key-to-
partition mapping for a hash-based partitioner,
which can break per-key ordering for any
consumer relying on it. Plan this as a real
migration — e.g. a new topic with the new
partition count, or confirm no consumer
actually depends on ordering across the change."

What you see: After a partition-count change, a consumer occasionally processes events for the same entity out of order — specifically for entities whose events span the moment of the change, which makes the bug intermittent and hard to reproduce on demand.

Why: A hash-based partitioner's key-to-partition mapping is a function of the current partition count — changing that count changes the function, which silently reroutes future messages for existing keys to different partitions than their past messages used.

Repartitioning order_id=42, 4 → 8 partitions

Before (4 partitions)

+hash(42) % 4 = partition 2
+Every event for order 42 always landed there
+Full history readable from one partition, in order

After (8 partitions)

−hash(42) % 8 = partition 6 — a different partition
−New events go to partition 6; old ones stay in partition 2
−Full order history now spans two partitions

What partition count and partition key each control

What partition count and partition key each control
Decision	Controls	Consequence of getting it wrong
Partition count	Max consumer parallelism within a group	Too few: throughput ceiling. Too many: broker overhead
Partition key	Which messages are ordered relative to each other	Wrong key: either no useful ordering, or a hot partition

Remember: Partition count caps consumer parallelism and, if changed later, can silently break per-key ordering by changing which partition a key maps to — treat both partition count and partition key as deliberate, hard-to-reverse design decisions, not storage tuning.