Kinesis and Streaming Concepts

Shards and ordering, consumer throughput and backpressure, choosing streaming over SQS/EventBridge, and real-time ingestion architectures

Shard = ordered sequencepartition key hash decides shard, order holds per-shard only
24h retention default, up to 365dthe window that makes replay possible
1,000 rec/s or 1 MB/s write, 2 MB/s reada fixed per-shard ceiling — exceeding it throttles, not queues
Kinesis vs SQS vs EventBridgeordered multi-consumer replay vs work queue vs content-based routing

1.Kinesis Core Vocabulary: Shards, Partition Keys, Ordering, Retention, Replay

A Kinesis data stream is a set of shards; each shard holds an ordered sequence of records. Every record carries a partition key, and Kinesis hashes that key to decide which shard the record lands on. Records with the same partition key always land on the same shard, in the order they arrived. A stream keeps records for 24 hours by default (up to 365 days), so a consumer can replay history instead of only seeing new data.

Key features

A stream is a set of shards; each shard is a fixed unit of capacity with its own sequence of records
A partition key (a string, max 256 characters) is hashed with MD5 to pick which shard a record goes to
Ordering is guaranteed only within a shard, for records sharing the same partition key — not across the whole stream
Retention defaults to 24 hours after a stream is created; it can be raised up to 8,760 hours (365 days)
Because records stay in the stream for the retention window, a new or restarted consumer can replay from an earlier point instead of only reading new arrivals
A data blob inside a record can be up to 1 MB

A record's path from partition key to shard

hashedmaps to a hashkey range

Record + partition key

MD5 hash of partition key

Shard (ordered sequence)

Signature

Stream = set of shards · partition key → MD5 hash → shard · order guaranteed per-shard only

A stream is a set of shards; a partition key's hash decides the shard. Ordering holds only within a shard, for one partition key. Retention defaults to 24 hours, extendable to 365 days — that window is what makes replay possible.

2.Consumers, Throughput Scaling, and Backpressure

Each shard supports up to 1,000 records/sec or 1 MB/sec of writes, and up to 2 MB/sec of reads. A shared (classic) consumer polls with GetRecords and splits that 2 MB/sec with every other consumer reading the same shard. An enhanced fan-out consumer gets its own dedicated 2 MB/sec per shard, pushed to it over HTTP/2, independent of other consumers. When writes or reads exceed a shard's fixed limit, Kinesis throttles the call with a ProvisionedThroughputExceededException instead of queuing it — that is backpressure, and resharding (splitting a hot shard) is how you scale past it.

Key features

Per-shard write limit: 1,000 records/sec or 1 MB/sec, whichever is hit first, including partition keys in the size
Per-shard read limit: 2 MB/sec total, and up to 5 GetRecords calls/sec, whichever is hit first
Shared-throughput consumers pull via GetRecords and split the shard's 2 MB/sec across every consumer reading that shard
Enhanced fan-out consumers get their own dedicated 2 MB/sec per shard, pushed over HTTP/2 via SubscribeToShard — no contention with other consumers
Exceeding a shard's limit throws ProvisionedThroughputExceededException on that call — there is no server-side queue, so the producer or consumer must retry with backoff
Resharding (splitting a shard in two, or merging two into one) is how you add or remove per-shard capacity; a hot shard (one partition key sending disproportionate traffic) is fixed by splitting it, not by retrying harder

Shared fan-out vs enhanced fan-out

Shared (classic) consumer

Pulls with GetRecords (polling)
Shares the shard's 2 MB/sec with every other consumer on it
Propagation delay grows as consumers are added (~200ms at 1, ~1000ms at 5)

Enhanced fan-out consumer

Kinesis pushes records over HTTP/2 via SubscribeToShard
Own dedicated 2 MB/sec per shard, independent of other consumers
Typically ~70ms propagation delay regardless of consumer count

Signature

Shard: 1,000 rec/s or 1 MB/s write · 2 MB/s read (shared or, per enhanced fan-out consumer, dedicated)

Each shard: 1,000 records/sec or 1 MB/sec write, 2 MB/sec read. Shared consumers split that read throughput; enhanced fan-out gives each consumer its own 2 MB/sec via push. Exceeding a limit throws ProvisionedThroughputExceededException immediately — reshard the hot shard, don't just retry harder.

3.When Streaming Fits vs SQS or EventBridge

Reach for Kinesis when several independent consumers must read the same data, in order, and possibly replay history. Reach for SQS when you need a work queue: each message processed exactly once by exactly one worker, with per-message retry and delay. Reach for EventBridge when you are routing typed events to different targets based on content, not processing a high-volume ordered sequence.

Key features

Kinesis: multiple independent consumers read the same ordered data concurrently, and can replay records for up to the retention window (default 24h, up to 365 days)
SQS: a message is delivered for one worker to process and acknowledge; once acknowledged (deleted) it is gone — not built for multiple independent readers of the same message
SQS scales throughput transparently with no shard management; Kinesis throughput is a function of shard count, which you manage (or that on-demand mode manages for you)
EventBridge routes typed events to multiple targets by content-based rules (e.g. "route order.created events to this Lambda and that SNS topic") — it is a router, not an ordered log
Kinesis preserves per-shard ordering; SQS standard queues do not guarantee order at all, and FIFO queues guarantee order only within a message group
SQS supports per-message delivery delay (up to 15 minutes) and per-message visibility timeout for retry — concepts that do not exist in Kinesis, where retry/redelivery is the consumer's own responsibility

Streaming vs queue vs event router, by shape of the problem

Ordered, high-volume sequenceContent-based routing

Single worker per itemMany independent readers of the same data

SQS: Work queue: one worker processes and acknowledges each message
EventBridge: Routes typed events to targets by rule; not an ordered log
Kinesis Data Streams: Many consumers read the same ordered, replayable stream independently

Signature

Kinesis: ordered, multi-consumer, replayable · SQS: single-worker work queue · EventBridge: content-based event routing

Same data, many independent readers, order/replay matters → Kinesis. One worker per item, retry/delay per message → SQS. Route typed events to different targets by content → EventBridge. Pick by coordination shape, not by which sounds more "real-time."

4.Real-Time Analytics and Event Ingestion Architectures

A real-time ingestion pipeline usually has one entry point (a Kinesis data stream taking writes from many producers) and several parallel consumers doing different jobs on the same data at once: one computing live aggregates, one archiving to a data lake, one triggering alerts. Managed Service for Apache Flink runs SQL or code over the stream for aggregation; Data Firehose delivers records to S3, Redshift, or OpenSearch without you managing delivery infrastructure; Lambda runs custom per-batch logic.

Key features

Producers write events into one Kinesis stream, decoupling ingestion from however many downstream consumers exist
Managed Service for Apache Flink (formerly Kinesis Data Analytics) runs SQL or Flink application code directly against the stream, for windowed aggregation and real-time transforms
Data Firehose delivers stream records to S3, Redshift, OpenSearch, or an HTTP endpoint without you building or operating the delivery pipeline yourself
Lambda can consume a stream directly, running your function once per batch of records it polls from a shard
Multiple consumers (Flink, Firehose, Lambda, a custom app) can read the same stream independently and concurrently — a dashboard, a data lake load, and an alerting rule all draw from one ingestion point
This is the same decouple-ingestion-from-processing shape as a message queue, but ordered, replayable, and built for many concurrent readers rather than one worker per item

Command

ingestion-pipeline-shape.txttext

Producers → Kinesis Data Stream ─┬→ Managed Service for Apache Flink → live dashboard
                                   ├→ Data Firehose → S3 (data lake)
                                   └→ Lambda → alerting / custom logic

One Kinesis stream, several independent concurrent consumers: Flink for real-time aggregation, Firehose for delivery to storage/analytics services, Lambda for custom per-batch logic — all reading the same ordered data without coordinating with each other.

Spotted a mistake on this page? Tell us.

Read Sheet