MSK (Managed Streaming for Kafka)

Protocol: REST-JSON Endpoint: http://localhost:4566/

Floci emulates Amazon MSK by orchestrating Redpanda containers. This provides high compatibility with the Kafka API while maintaining a low footprint.

Supported Actions

Action	Description
`CreateCluster`	Spawns a new Redpanda container for the cluster
`CreateClusterV2`	Create a provisioned or serverless cluster
`ListClusters`	List all emulated clusters
`ListClustersV2`	List all emulated clusters using V2 API
`DescribeCluster`	Get cluster metadata and state
`DescribeClusterV2`	Get cluster metadata and state using V2 API
`DeleteCluster`	Stops and removes the Redpanda container
`GetBootstrapBrokers`	Get the connection strings for the cluster
`CreateConfiguration`	Create a broker configuration (`server.properties`)
`ListConfigurations`	List all configurations
`DescribeConfiguration`	Get configuration metadata and latest revision
`DeleteConfiguration`	Delete a configuration
`UpdateConfiguration`	Create a new revision of a configuration
`ListConfigurationRevisions`	List all revisions of a configuration
`DescribeConfigurationRevision`	Get a specific revision, including its `server.properties`
`ListTagsForResource`	List the tags on a cluster or configuration
`TagResource`	Add tags to a cluster or configuration
`UntagResource`	Remove tags from a cluster or configuration

Configuration

Variable	Default	Description
`FLOCI_SERVICES_MSK_ENABLED`	`true`	Enable or disable the service
`FLOCI_SERVICES_MSK_MOCK`	`false`	`true` = metadata-only CRUD, no Docker containers
`FLOCI_SERVICES_MSK_DEFAULT_IMAGE`	`redpandadata/redpanda:latest`	Docker image for Redpanda (Kafka) containers

How it works

When mock is set to false (default), Floci uses the Docker API to start a Redpanda container for each created cluster. For Docker socket setup, private registry authentication, and other Docker settings see Docker Configuration.

Port Mapping: The Kafka API (9092) is mapped to a dynamic host port.
Persistence: Each cluster gets a named Docker volume (floci-msk-{volumeId}). In memory mode the volume is removed on cluster delete; in persistent modes it is retained unless FLOCI_STORAGE_PRUNE_VOLUMES_ON_DELETE=true.
Readiness: The cluster state transitions to ACTIVE once the Redpanda /ready endpoint is reachable.

Cluster metadata

CreateCluster and CreateClusterV2 persist the metadata you pass — broker node group (instance type, subnets, security groups, storage, connectivity), number of broker nodes, encryption, client authentication, enhanced monitoring, open monitoring, logging, configuration, storage mode, rebalancing and tags — and echo it back from DescribeCluster/DescribeClusterV2 and the matching List calls.

The two API versions return different shapes, matching AWS:

v1 (DescribeCluster) returns a flat ClusterInfo: brokerNodeGroupInfo, encryptionInfo, clientAuthentication, enhancedMonitoring, loggingInfo, openMonitoring, storageMode, rebalancing, numberOfBrokerNodes and zookeeperConnectString all sit directly on the cluster.
v2 (DescribeClusterV2) nests all of those under ClusterInfo.Provisioned, alongside a top-level ClusterType of PROVISIONED. ClusterArn, ClusterName, State, CreationTime, CurrentVersion and Tags stay top-level.

Two details worth knowing:

configurationInfo is a request-only member. The configuration a cluster was created with is reported back on currentBrokerSoftwareInfo as configurationArn and configurationRevision, which is where the Terraform AWS provider reads it from.
Members AWS defaults server-side are defaulted here too, so a terraform plan converges: enhancedMonitoring becomes DEFAULT, and encryptionInfo.encryptionInTransit becomes clientBroker: TLS_PLAINTEXT / inCluster: true when the request omits them.

Serverless clusters

CreateClusterV2 accepts a serverless member as well as provisioned — exactly one of the two, as on AWS. A serverless cluster stores its vpcConfigs and clientAuthentication, reports clusterType: SERVERLESS, and comes back from DescribeClusterV2 under a serverless envelope rather than a provisioned one. It is backed by the same emulated Kafka endpoint, so GetBootstrapBrokers works normally.

The v1 API predates serverless and its ClusterInfo cannot represent one, so — as on AWS — DescribeCluster on a serverless cluster returns BadRequestException pointing you at DescribeClusterV2, and ListClusters omits serverless clusters entirely.

Validation

CreateCluster/CreateClusterV2 reject the values AWS rejects: a missing or over-64-character clusterName, a numberOfBrokerNodes below 1, an EBS volumeSize outside 1–16384, a configurationInfo.revision below 1, and any unrecognised enhancedMonitoring, storageMode, rebalancing.status or encryptionInTransit.clientBroker value. Each 400 names the offending member in invalidParameter, as MSK's Error schema does.

There is deliberately no upper bound on numberOfBrokerNodes. The SDK model caps it at 15, but the REST API reference documents no maximum for it — on a page that gives ranges for its neighbours — and the quota page allows 30 brokers per ZooKeeper cluster and 60 per KRaft cluster, both adjustable. A 30-broker cluster is an ordinary thing to have.

Unlike AWS, the presence of kafkaVersion, numberOfBrokerNodes and brokerNodeGroupInfo is not required — the emulator defaults them so that a minimal create request still works.

Examples

export AWS_ENDPOINT_URL=http://localhost:4566

# Create a cluster
aws kafka create-cluster \
  --cluster-name my-cluster \
  --kafka-version "3.6.1" \
  --number-of-broker-nodes 1 \
  --broker-node-group-info '{"InstanceType":"kafka.m5.large","ClientSubnets":["subnet-1"]}' \
  --endpoint-url $AWS_ENDPOINT_URL

# List clusters
aws kafka list-clusters --endpoint-url $AWS_ENDPOINT_URL

# Get bootstrap brokers
CLUSTER_ARN=$(aws kafka list-clusters --query 'ClusterInfoList[0].ClusterArn' --output text --endpoint-url $AWS_ENDPOINT_URL)
aws kafka get-bootstrap-brokers --cluster-arn $CLUSTER_ARN --endpoint-url $AWS_ENDPOINT_URL

# Delete a cluster
aws kafka delete-cluster --cluster-arn $CLUSTER_ARN --endpoint-url $AWS_ENDPOINT_URL