Kinesis
Protocol: JSON 1.1 (X-Amz-Target: Kinesis_20131202.*)
Endpoint: POST http://localhost:4566/
Supported Actions
| Action | Description |
|---|---|
CreateStream |
Create a stream |
DeleteStream |
Delete a stream |
ListStreams |
List all streams |
DescribeStream |
Get stream details and shard info |
DescribeStreamSummary |
Lightweight stream description |
RegisterStreamConsumer |
Register an enhanced fan-out consumer |
DeregisterStreamConsumer |
Remove a consumer |
DescribeStreamConsumer |
Get consumer details |
ListStreamConsumers |
List consumers for a stream |
SubscribeToShard |
Subscribe to a shard for enhanced fan-out |
AddTagsToStream |
Tag a stream |
RemoveTagsFromStream |
Remove tags |
ListTagsForStream |
List tags |
TagResource |
Add or update tags on a stream or consumer |
UntagResource |
Remove tags from a stream or consumer |
ListTagsForResource |
List tags on a stream or consumer |
StartStreamEncryption |
Enable KMS encryption |
StopStreamEncryption |
Disable encryption |
SplitShard |
Split a shard into two |
MergeShards |
Merge two adjacent shards |
PutRecord |
Write a single record |
PutRecords |
Write up to 500 records |
GetShardIterator |
Get an iterator for reading |
GetRecords |
Read records from a shard |
ListShards |
- |
IncreaseStreamRetentionPeriod |
Increase retention up to 8760 hours (365 days) |
DecreaseStreamRetentionPeriod |
Decrease retention down to 24 hours |
EnableEnhancedMonitoring |
- |
DisableEnhancedMonitoring |
- |
UpdateStreamMode |
- |
UpdateMaxRecordSize |
- |
UpdateShardCount |
- |
Stream Listing
ListStreams returns sorted StreamNames, with at most 100 names per page by default.
Limit accepts integers from 1 to 10000, but values above 100 still return at most 100 names.
When HasMoreStreams is true, pass the last returned name as ExclusiveStartStreamName
to continue. Truncated pages also return NextToken, which CLI and SDK paginators can send
back without an ExclusiveStartStreamName. Tokens are scoped to this operation and region,
use the same configurable lifetime as ListShards tokens (see the TTL setting below),
and remain usable if the cursor stream is deleted.
Each page also returns StreamSummaries, one entry per name in StreamNames and in the same
order, with StreamName, StreamARN, StreamStatus, StreamModeDetails and
StreamCreationTimestamp.
Local Inspection Endpoints
These endpoints are Floci-local read-only helpers for the UI and tests. AWS SDK
traffic should continue to use the JSON 1.1 Kinesis API on /.
| Endpoint | Description |
|---|---|
GET /_aws/kinesis/streams |
List streams in the resolved region with shard, mode, tag, and record counts |
GET /_aws/kinesis/records?StreamName=<name> |
Peek up to 100 stream records with shard attribution without consuming them |
GET /_aws/kinesis/records?StreamName=<name>&Limit=<n> |
Peek up to n records, capped at 1000 |
GET /_aws/kinesis/records?StreamName=<name>&ShardId=<id> |
Peek records for one shard |
Stream Addressing
Most actions accept either StreamName or StreamARN to identify a stream. When both are provided, StreamName takes precedence. CreateStream only accepts StreamName.
# By name
aws kinesis describe-stream --stream-name events --endpoint-url $AWS_ENDPOINT_URL
# By ARN
aws kinesis describe-stream --stream-arn arn:aws:kinesis:us-east-1:000000000000:stream/events --endpoint-url $AWS_ENDPOINT_URL
Shard Iterators
GetShardIterator supports all five iterator types: TRIM_HORIZON, LATEST, AT_SEQUENCE_NUMBER, AFTER_SEQUENCE_NUMBER, AT_TIMESTAMP.
A LATEST iterator is positioned at the shard tip at the moment the iterator is created, matching AWS: records written after the iterator was obtained are returned, records written before are not. This supports the standard tailing pattern: obtain a LATEST iterator, trigger the action that produces the record, then poll GetRecords following NextShardIterator.
A closed shard returns a JSON null NextShardIterator on its final page, including
when that page contains records. Earlier pages keep their continuation iterator.
Only the terminal page includes ChildShards: each direct child's ID, all of its
parent IDs and its hash-key range. A caught-up open shard keeps a non-null iterator.
Consumers should process the final records before stopping at the null iterator.
The internal Pipes and Firehose readers retain a sequence cursor after the final record to avoid replaying a terminal batch. Firehose commits that cursor only after delivery succeeds; its unflushed buffer remains in memory. Pipes keeps its existing advance-before-delivery policy and first-shard polling limitation; child scheduling and full lineage orchestration are not implemented.
Record Retention
A stream's retention period (default 24 hours, IncreaseStreamRetentionPeriod/DecreaseStreamRetentionPeriod up to 8760 hours) is enforced: a record older than the current retention period is not returned by GetRecords or the GET /_aws/kinesis/records inspection endpoint, and is actually removed from memory rather than merely hidden. Expiry is checked lazily on each stream's next PutRecord/PutRecords or read after a record ages out; there is no scheduled sweep. After DecreaseStreamRetentionPeriod, records outside the new window are inaccessible from the next put or read, in line with AWS making them inaccessible almost immediately. Shard iterators (including NextShardIterator continuation tokens and LATEST) resolve by sequence number, so they keep working correctly across pruning even if the exact record they were positioned at has since expired.
Record Routing
PutRecord and PutRecords honor ExplicitHashKey when it is provided. The
value must be a decimal integer in the Kinesis hash-key space, and records are
written to the open shard whose HashKeyRange contains that value.
Without ExplicitHashKey, Kinesis takes the MD5 digest of the partition key's
UTF-8 bytes as an unsigned 128-bit integer. Floci writes to the open shard whose
HashKeyRange contains that value, including after a split or merge. Closed
historical shards are never selected. Untouched legacy streams whose open root
shards all persisted the full hash span are repartitioned before selection.
Other overlapping stored ranges are rejected rather than rewriting existing
child ranges or records.
Shard Scaling (UpdateShardCount)
Stream creation and control-plane updates normally report ACTIVE immediately. Set
FLOCI_SERVICES_KINESIS_CREATING_DELAY_MILLIS or
FLOCI_SERVICES_KINESIS_UPDATING_DELAY_MILLIS to test clients that wait for a stream
to become active or retry a conflicting update. During a CREATING window,
PutRecord and PutRecords reject writes until the stream becomes active. During an
UPDATING window, reads and record writes remain available, but another split,
merge, scaling, mode, size, encryption, retention, or stream-tag change returns
ResourceInUseException. Tag and retention changes also reject a CREATING stream.
The window is evaluated on
read from a stored deadline set when the change begins, so a slow synchronous write
can consume some or all of a very small window before the call returns. Resharding
does not defer the shard change.
UpdateShardCount enforces the documented default limits: at most double the current
open shard count per call, at most ten calls per rolling 24-hour period per stream, a
10000-shard ceiling, and the asymmetric rule that a stream already above 10000 shards can
only move to a target strictly below it.
The minimum bound (TargetShardCount cannot go below half the current open shard count)
rounds up for an odd current count, so current=5 accepts target=3 but rejects
target=2. AWS's documentation says only "below half" with no stated rounding, so which
direction an odd count rounds is Floci's own call rather than observed AWS behavior; this
is the more conservative reading (fewer shards allowed, not more).
AWS also documents a separate 10 TPS call-rate limit on this action ("Make over 10 TPS. TPS over 10 will trigger the LimitExceededException"), independent of the ten-calls-per-24h limit above. Floci does not simulate real-time request throttling for this or any other action, so that limit is intentionally unenforced here.
Configuration
| Variable | Default | Description |
|---|---|---|
FLOCI_SERVICES_KINESIS_ENABLED |
true |
Enable or disable the service |
FLOCI_SERVICES_KINESIS_CREATING_DELAY_MILLIS |
0 |
Optional CREATING status window after CreateStream |
FLOCI_SERVICES_KINESIS_UPDATING_DELAY_MILLIS |
0 |
Optional UPDATING status window after a stream change |
FLOCI_SERVICES_KINESIS_LIST_SHARDS_NEXT_TOKEN_TTL_MILLIS |
300000 |
Lifetime of ListShards and ListStreams NextToken values in milliseconds. Lower it to exercise ExpiredNextTokenException without waiting |
Enhanced Fan-Out (EFO)
SubscribeToShard uses a snapshot-and-close model: the server returns one batch
of records as a binary EventStream response and closes the connection. For a
nonterminal batch, use ContinuationSequenceNumber from the last delivered record
when resubscribing. At a closed shard's end, the event includes the same direct
ChildShards metadata and omits ContinuationSequenceNumber, even when it includes
final records. This remains a snapshot response rather than a long-lived
subscription. All five StartingPosition types are supported: TRIM_HORIZON,
LATEST, AT_SEQUENCE_NUMBER, AFTER_SEQUENCE_NUMBER, AT_TIMESTAMP.
export AWS_ENDPOINT_URL=http://localhost:4566
STREAM=my-stream
# Register a consumer
aws kinesis register-stream-consumer \
--stream-arn $(aws kinesis describe-stream --stream-name $STREAM \
--query StreamDescription.StreamARN --output text) \
--consumer-name my-consumer
# Subscribe (AWS CLI streams events to stdout)
aws kinesis subscribe-to-shard \
--consumer-arn <consumer-arn> \
--shard-id shardId-000000000000 \
--starting-position Type=TRIM_HORIZON
Examples
export AWS_ENDPOINT_URL=http://localhost:4566
# Create a stream
aws kinesis create-stream \
--stream-name events \
--shard-count 2 \
--endpoint-url $AWS_ENDPOINT_URL
# Put a record
aws kinesis put-record \
--stream-name events \
--partition-key "user-123" \
--data '{"event":"page_view","page":"/home"}' \
--endpoint-url $AWS_ENDPOINT_URL
# Get a shard iterator
SHARD_ID=$(aws kinesis describe-stream \
--stream-name events \
--query 'StreamDescription.Shards[0].ShardId' --output text \
--endpoint-url $AWS_ENDPOINT_URL)
ITERATOR=$(aws kinesis get-shard-iterator \
--stream-name events \
--shard-id $SHARD_ID \
--shard-iterator-type TRIM_HORIZON \
--query ShardIterator --output text \
--endpoint-url $AWS_ENDPOINT_URL)
# Read records
aws kinesis get-records \
--shard-iterator $ITERATOR \
--endpoint-url $AWS_ENDPOINT_URL