beginnerintermediateadvanced

System Design · Kafka and Log-Based Messaging

Kafka and Log-Based Messaging

Ordered logs, delivery guarantees, replay, and why partitioning is a design decision

Topics + partitionsordered within a partition only
Delivery guaranteesat-least-once is the honest default
Replayable logconsumption never deletes a message
Partitioninga scaling AND ordering decision

1.Topics, partitions, offsets and consumer groups

A Kafka topic is a named stream of messages, physically split into partitions for parallelism — each partition is an ordered, append-only log with its own offset counter. Kafka only guarantees message order within a single partition, never across partitions of the same topic. Consumer groups let multiple consumers split a topic's partitions between them, each partition read by exactly one consumer in the group at a time.

Key features

Topic: a named category/stream of messages, e.g. "order-events"
Partition: a topic is split into one or more partitions, each an independently ordered, append-only log
Offset: a message's position within its partition — strictly increasing, unique per partition
Ordering: guaranteed within a partition only — two messages in different partitions have no guaranteed relative order
Consumer group: a set of consumers that split a topic's partitions among themselves — each partition is read by exactly one consumer in the group at any time
Retention: Kafka keeps messages for a configured period (or size limit) regardless of whether they've been consumed — unlike a traditional queue, reading a message does not delete it

Kafka's core vocabulary

Kafka's core vocabulary
Term	What it is
Topic	A named stream of messages
Partition	One ordered, append-only log within a topic
Offset	A message's position within its partition
Consumer group	Consumers that split a topic's partitions between them
Retention	How long messages are kept, independent of whether they were read

Topic "orders" — 3 partitions, 2 consumers

Topic: orders

Partition 0

Partition 1

Partition 2

Consumer group: billing

Consumer A

Consumer B (×2)

Command

text

topic "orders" → partition 0: [msg0, msg1, msg2, ...]
                → partition 1: [msg0, msg1, msg2, ...]
                → partition 2: [msg0, msg1, msg2, ...]

consumer group "billing":
  consumer A ← partition 0
  consumer B ← partition 1, partition 2

A topic's partitions are independently ordered logs — Kafka guarantees order within a partition, never across partitions. Consumer groups split partitions among consumers, one partition per consumer at a time, so parallelism is capped by partition count.

2.At-least-once, at-most-once and practical exactly-once

At-least-once means a message may be delivered more than once but never lost — the common default, requiring idempotent consumers. At-most-once means a message may be lost but never duplicated — rarely what anyone actually wants. "Exactly-once" is achievable within Kafka itself (via idempotent producers and transactions) for Kafka-to-Kafka pipelines, but the moment a side effect touches something outside Kafka (a database write, an external API call), true exactly-once delivery becomes effectively impossible to guarantee end-to-end.

Key features

At-least-once: a message is retried until acknowledged, so a crash between processing and acking can cause duplicate delivery — never loses a message
At-most-once: a message is acked before processing (or not retried on failure) — never duplicates, but can lose a message if the consumer crashes before finishing
Kafka's "exactly-once semantics" (idempotent producer + transactions) applies to Kafka-to-Kafka pipelines — producing, transforming and consuming entirely within Kafka
Once a consumer's side effect leaves Kafka's transactional boundary (writing to a database, calling an external API), exactly-once is no longer strictly guaranteed by Kafka alone
The practical, honest target for most systems is at-least-once delivery paired with idempotent consumer logic — the equivalent effect of "exactly-once processing" without needing the delivery layer alone to guarantee it

The three delivery semantics

The three delivery semantics
Semantic	Can lose a message?	Can duplicate a message?	Typical use
At-most-once	Yes	No	Metrics/logs where an occasional loss is acceptable
At-least-once	No	Yes	The common default — pair with idempotent processing
"Exactly-once" (Kafka-to-Kafka)	No	No, within Kafka's transactional boundary	Kafka Streams pipelines that stay entirely within Kafka

Three delivery semantics

At-most-once

can lose, never duplicates

At-least-once

never loses, can duplicate

"Exactly-once"

Kafka-to-Kafka only

Command

text

at-most-once:  ack BEFORE processing (or no retry)
at-least-once: ack AFTER processing succeeds, retry
               on failure/timeout
exactly-once:  idempotent producer + transactions,
               scoped to Kafka-to-Kafka pipelines

At-most-once can lose messages, never duplicates. At-least-once never loses messages, can duplicate — pair it with idempotent consumers. Kafka's "exactly-once" is real but scoped to Kafka-to-Kafka pipelines — any external side effect needs its own idempotency.

3.Replayable event logs and event-driven architecture

Because Kafka retains messages instead of deleting them once read, a topic functions as a durable, replayable log rather than a transient queue — any consumer can re-read from an earlier offset, and a brand-new consumer can process the entire history from the beginning. This is the core capability that makes Kafka a fit for event-driven architectures, where multiple independent services each want their own full view of the same event stream.

Key features

A traditional queue removes a message once consumed; a Kafka topic keeps it until retention expires, regardless of consumption
Multiple independent consumer groups can each read the same topic from their own position — one group's consumption doesn't affect another's
Replay: a consumer can reset its offset backward and reprocess history — useful for recovering from a bug, backfilling a new service, or rebuilding derived state
A new service added later can consume the topic's full retained history from the beginning, rather than only seeing events from the moment it started
This replayability is what lets Kafka act as a shared source of truth for an event-driven system, not just a transient pipe between two services

One topic, three independent readers

Independent consumer groups

billing @500

analytics @0

shipping @480

Command

text

topic "orders" (retained 7 days)
  ├─ consumer group "billing"    reads from offset 500
  ├─ consumer group "analytics"  reads from offset 0
  └─ consumer group "shipping"   reads from offset 480
-- each group tracks its own independent position

Kafka retains messages instead of deleting them on consumption, so a topic is a durable, replayable log — multiple independent consumer groups can each read it from their own position, and a new service can replay full history rather than only seeing events from its own start time.

4.Partitioning is a scaling AND ordering decision

How many partitions a topic has, and what key routes a message to a partition, are not implementation details to set once and forget — partition count caps consumer parallelism, and the partition key determines what ordering guarantee consumers actually get. Both choices are hard to change later without real operational cost, so they deserve the same deliberate attention as a database's shard key.

Key features

Partition count sets the ceiling on consumer parallelism within one consumer group — more partitions allow more concurrent consumers
The partitioning key (what determines which partition a message lands in) determines the ordering guarantee — same key, same partition, ordered; different keys, no ordering guarantee between them
Increasing partition count later is possible but does not retroactively reorder already-written data, and can change which partition a given key routes to (breaking a previously-relied-on ordering guarantee)
Too few partitions limits throughput and parallelism; too many partitions adds broker overhead (more file handles, more replication traffic, slower leader elections)
The right partition count is workload-dependent — sized for expected peak consumer parallelism, not an arbitrary round number

What partition count and partition key each control

What partition count and partition key each control
Decision	Controls	Consequence of getting it wrong
Partition count	Max consumer parallelism within a group	Too few: throughput ceiling. Too many: broker overhead
Partition key	Which messages are ordered relative to each other	Wrong key: either no useful ordering, or a hot partition

Repartitioning order_id=42, 4 → 8 partitions

Before (4 partitions)

hash(42) % 4 = partition 2
Every event for order 42 always landed there
Full history readable from one partition, in order

After (8 partitions)

hash(42) % 8 = partition 6 — a different partition
New events go to partition 6; old ones stay in partition 2
Full order history now spans two partitions

Signature

partition count = parallelism ceiling · partition key = ordering guarantee

Partition count caps consumer parallelism and, if changed later, can silently break per-key ordering by changing which partition a key maps to — treat both partition count and partition key as deliberate, hard-to-reverse design decisions, not storage tuning.

Spotted a mistake on this page? Tell us.

Read Sheet