beginnerintermediateadvanced

AWS · Section 29

Kinesis and Streaming Concepts

Level: intermediate
Read: 26 min
Concepts: 4

Kinesis Data Streams organizes records into shards, each a fixed unit of ordered capacity — a record's partition key hashes to a shard, and ordering holds only within that shard. Retention (24 hours by default, extendable to 365 days) is what makes replay possible for a consumer that fell behind or needs to reprocess history. Each shard is a hard throughput ceiling (1,000 records/sec or 1 MB/sec write, 2 MB/sec read); shared consumers split that read throughput while enhanced fan-out consumers each get their own dedicated 2 MB/sec, and exceeding the limit throttles immediately rather than queuing — resharding a hot shard is the fix, not retrying harder. Choosing Kinesis over SQS or EventBridge comes down to the coordination shape needed: ordered data many independent consumers must read and replay, versus a single-worker-per-item queue, versus content-based routing to multiple targets. In practice, one stream commonly feeds several independent, concurrent consumers at once — real-time aggregation, data lake delivery, and custom alerting logic all reading the same data without coordinating with each other.

AWS overview

AWS Full Course 2026 | AWS Cloud Computing Tutorial for Beginners | AWS Training | SimplilearnSimplilearn

What is true here

A partition key's hash decides its shard; ordering holds only within a shard, for records sharing that key.
Retention defaults to 24 hours and can be raised to 365 days — that window is what enables replay.
Each shard is a fixed ceiling (1,000 rec/s or 1 MB/s write, 2 MB/s read); exceeding it throttles immediately rather than queuing.
Enhanced fan-out gives each consumer its own dedicated 2 MB/sec per shard; shared consumers poll and split that throughput.
Choose Kinesis for ordered, multi-consumer, replayable data; SQS for a single-worker work queue; EventBridge for content-based event routing.

What you will be able to do

Explain how a partition key determines shard placement and what ordering guarantee that actually gives
Choose between shared and enhanced fan-out consumers based on contention and propagation-delay needs
Recognize backpressure (ProvisionedThroughputExceededException) and respond by resharding rather than retrying harder
Decide between Kinesis, SQS, and EventBridge for a given coordination problem
Describe how Kinesis, Managed Service for Apache Flink, Data Firehose, and Lambda combine into a real-time ingestion pipeline

From shards and ordering to ingestion architectures

Shards, partition keys, ordering, retention, replay

Consumers, throughput scaling, backpressure

Streaming vs SQS vs EventBridge

Real-time analytics & ingestion architectures

Kinesis and Streaming Concepts

The core vocabulary (shards, partition keys, ordering, retention, replay), consumer throughput and backpressure, choosing streaming over SQS/EventBridge, and real-time ingestion architectures.

Kinesis Core Vocabulary: Shards, Partition Keys, Ordering, Retention, Replay

coreintermediate

A Kinesis data stream is a set of shards; each shard holds an ordered sequence of records. Every record carries a partition key, and Kinesis hashes that key to decide which shard the record lands on. Records with the same partition key always land on the same shard, in the order they arrived. A stream keeps records for 24 hours by default (up to 365 days), so a consumer can replay history instead of only seeing new data.

Think of it as

Think of a stream as several parallel conveyor belts (shards). The partition key is the label on each box that decides which belt it goes on. Boxes on the same belt stay in the order they were placed — that per-belt order is the only ordering guarantee Kinesis makes. Nothing orders boxes across belts.

What we're doing: See why two records with the same partition key stay in order, but records with different keys have no ordering relationship.

shard-assignment.txttext

PutRecord(partitionKey="user-42", data=e1)  → shard-0, sequence 100
PutRecord(partitionKey="user-42", data=e2)  → shard-0, sequence 118
PutRecord(partitionKey="user-7",  data=e3)  → shard-1, sequence 54
# e1 always precedes e2 for a consumer reading shard-0.
# e3 has no defined order relative to e1 or e2 — different shard.

1: user-42 hashes into shard-0's hash key range, so this record lands there.
2: Same partition key, same shard — and its sequence number is higher, so a shard-0 consumer always sees e1 before e2.
3: user-7 hashes to a different shard. Kinesis makes no promise about how e3 interleaves with shard-0's records in wall-clock time.

Why this works: Kinesis Data Streams scales by spreading records across shards, so it can only afford to order records within a shard — ordering across the whole stream would require every shard to coordinate on every write, which would remove the parallelism that makes shards useful in the first place.

Assuming Kinesis orders records across the entire stream

Wrong

text

# Consumer logic: "records will arrive at every consumer in exactly the
# order they were written to the stream, regardless of shard"

Better

text

# Design the partition key so records that must stay ordered relative to
# each other (e.g. all events for one user) share a partition key, and
# only rely on ordering within that one shard

What you see: Events for two different entities appear interleaved in an order that does not match wall-clock arrival, and a consumer reading multiple shards in parallel sees them out of the sequence the application assumed.

Why: The sequence number that establishes order is assigned per shard, and different shards are read independently by consumers — there is no cross-shard sequence to fall back on.

A record's path from partition key to shard

Record + partition key

MD5 hash of partition key

Shard (ordered sequence)

Remember: A stream is a set of shards; a partition key's hash decides the shard. Ordering holds only within a shard, for one partition key. Retention defaults to 24 hours, extendable to 365 days — that window is what makes replay possible.

Consumers, Throughput Scaling, and Backpressure

coreintermediate

Each shard supports up to 1,000 records/sec or 1 MB/sec of writes, and up to 2 MB/sec of reads. A shared (classic) consumer polls with GetRecords and splits that 2 MB/sec with every other consumer reading the same shard. An enhanced fan-out consumer gets its own dedicated 2 MB/sec per shard, pushed to it over HTTP/2, independent of other consumers. When writes or reads exceed a shard's fixed limit, Kinesis throttles the call with a ProvisionedThroughputExceededException instead of queuing it — that is backpressure, and resharding (splitting a hot shard) is how you scale past it.

Think of it as

Each shard is a pipe with a fixed diameter — 1 MB/sec in, 2 MB/sec out, no matter how much you push. A shared consumer is a straw stuck into that outgoing pipe alongside every other shared consumer, so they all draw from the same 2 MB/sec. Enhanced fan-out gives each consumer its own separate pipe instead of a shared straw. If more water arrives than the pipe can carry, it does not back up and wait — it gets rejected on the spot, and you have to add another pipe (split the shard).

What we're doing: See what happens when a producer's write rate exceeds a single shard's throughput.

throttled-write.txttext

# Shard has 1 shard, limit 1,000 records/sec.
# Producer sends 1,400 records/sec to it.
PutRecords(...) → 400 records/sec rejected with ProvisionedThroughputExceededException
# Kinesis does not buffer the excess — the caller must retry those records.

1: A single shard is a fixed 1,000 records/sec (or 1 MB/sec) ceiling — it does not flex upward under load.
2: The write rate is 40% over the shard's limit.
3: The exception is thrown per over-limit call; the excess records are not silently queued for later — the producer's own retry logic is what determines whether they are eventually written.

Why this works: Kinesis enforces capacity per shard, not per stream — a stream's total capacity is the sum of its shards' capacities, so hitting a limit here is a signal to add shards (reshard) for that hot partition key range, not to keep retrying the same shard indefinitely.

Retrying a ProvisionedThroughputExceededException without backoff, or without ever resharding

Wrong

text

# On ProvisionedThroughputExceededException, immediately retry the same
# PutRecord call in a tight loop with no delay and no shard-count change

Better

text

# Retry with exponential backoff for transient bursts, and separately
# watch CloudWatch's WriteProvisionedThroughputExceeded metric — if it is
# sustained rather than a brief burst, split the hot shard(s) or move to
# on-demand capacity mode

What you see: A tight retry loop adds more load to an already-saturated shard, making the throttling worse instead of better, while the underlying hot shard never gets addressed.

Why: The limit is a hard per-shard ceiling, not a soft one that yields under retry pressure — retrying faster cannot create throughput that shard was never provisioned to carry; only more shards (or a partition key that spreads load more evenly) can.

Shared fan-out vs enhanced fan-out

Shared (classic) consumer

+Pulls with GetRecords (polling)
+Shares the shard's 2 MB/sec with every other consumer on it
+Propagation delay grows as consumers are added (~200ms at 1, ~1000ms at 5)

Enhanced fan-out consumer

−Kinesis pushes records over HTTP/2 via SubscribeToShard
−Own dedicated 2 MB/sec per shard, independent of other consumers
−Typically ~70ms propagation delay regardless of consumer count

Remember: Each shard: 1,000 records/sec or 1 MB/sec write, 2 MB/sec read. Shared consumers split that read throughput; enhanced fan-out gives each consumer its own 2 MB/sec via push. Exceeding a limit throws ProvisionedThroughputExceededException immediately — reshard the hot shard, don't just retry harder.

When Streaming Fits vs SQS or EventBridge

coreintermediate

Reach for Kinesis when several independent consumers must read the same data, in order, and possibly replay history. Reach for SQS when you need a work queue: each message processed exactly once by exactly one worker, with per-message retry and delay. Reach for EventBridge when you are routing typed events to different targets based on content, not processing a high-volume ordered sequence.

Think of it as

A queue (SQS) is a to-do list — each item gets picked up once and crossed off. A stream (Kinesis) is a recorded broadcast — many listeners can tune in independently, rewind, and each keeps their own place. An event router (EventBridge) is a mail-sorting office — it reads an event's content and routes a copy to whichever mailboxes (targets) match a rule, without caring about ordering across events at all.

What we're doing: Pick the right service for three different pieces of the same order-processing system.

order-system-choices.txttext

1. Fulfillment worker pool: pick one order, ship it, done      → SQS
2. "order.created" fans out to billing, analytics, notifications → EventBridge
3. Clickstream feeding a live fraud-scoring model + a data lake  → Kinesis

1: Exactly one worker should ship any given order — a queue with visibility-timeout retry fits, and Kinesis or EventBridge would need extra machinery to prevent double-processing.
2: One event type needs to reach several unrelated targets based on the event itself, not on volume or order — this is what EventBridge rules are built for.
3: High-volume, ordered, and read independently by two different real-time consumers (the fraud model and the data lake loader) — this is exactly the multi-consumer-replay shape Kinesis is built for.

Why this works: The three services solve different coordination problems — work distribution, content-based routing, and multi-consumer ordered replay — and picking by "which is faster" instead of by which coordination problem you actually have is how a system ends up fighting its own infrastructure (e.g. building manual fan-out logic on top of SQS instead of just using Kinesis or EventBridge).

Building multi-consumer fan-out on top of SQS instead of switching to Kinesis or EventBridge

Wrong

text

# Need 3 independent services to see every order event.
# "Solution": create 3 separate SQS queues and have the producer write
# the same message to all 3 queues.

Better

text

# Use EventBridge (rule-based fan-out to each service's target) if the
# events are discrete and routing is content-based, or Kinesis (each
# service registers its own consumer against the same stream) if the
# data is high-volume and ordering/replay matters.

What you see: The producer now owns fan-out logic and must change every time a new consumer is added, and the queues can drift out of sync if one write succeeds and another fails.

Why: SQS is designed around single-consumer delivery per message; multi-consumer fan-out is a problem EventBridge and Kinesis already solve natively — reimplementing it on top of SQS duplicates infrastructure the other two services provide for free.

Streaming vs queue vs event router, by shape of the problem

Ordered, high-volume sequenceContent-based routing

Single worker per itemMany independent readers of the same data

SQS: Work queue: one worker processes and acknowledges each message
EventBridge: Routes typed events to targets by rule; not an ordered log
Kinesis Data Streams: Many consumers read the same ordered, replayable stream independently

Remember: Same data, many independent readers, order/replay matters → Kinesis. One worker per item, retry/delay per message → SQS. Route typed events to different targets by content → EventBridge. Pick by coordination shape, not by which sounds more "real-time."

Real-Time Analytics and Event Ingestion Architectures

standardintermediate

A real-time ingestion pipeline usually has one entry point (a Kinesis data stream taking writes from many producers) and several parallel consumers doing different jobs on the same data at once: one computing live aggregates, one archiving to a data lake, one triggering alerts. Managed Service for Apache Flink runs SQL or code over the stream for aggregation; Data Firehose delivers records to S3, Redshift, or OpenSearch without you managing delivery infrastructure; Lambda runs custom per-batch logic.

Think of it as

Picture Kinesis as a single pipe splitting into several taps, each tap a different consumer doing its own job on the same water: one tap feeds a dashboard (Flink aggregating in real time), one tap fills a reservoir for later (Firehose to S3), one tap trips an alarm on specific readings (Lambda). Nothing about adding a new tap changes what is flowing through the pipe.

ingestion-pipeline-shape.txttext

Producers → Kinesis Data Stream ─┬→ Managed Service for Apache Flink → live dashboard
                                   ├→ Data Firehose → S3 (data lake)
                                   └→ Lambda → alerting / custom logic

Remember: One Kinesis stream, several independent concurrent consumers: Flink for real-time aggregation, Firehose for delivery to storage/analytics services, Lambda for custom per-batch logic — all reading the same ordered data without coordinating with each other.