Managed Kafka
floci-gcp emulates Google Cloud Managed Service for Apache Kafka (MSK) over REST JSON using the real GCP Managed Kafka API.
Configuration
| Variable | Default | Description |
|---|---|---|
FLOCI_GCP_SERVICES_KAFKA_ENABLED |
true |
Enable/disable Managed Kafka |
FLOCI_GCP_SERVICES_KAFKA_MOCK |
false |
Use mock mode (no Docker; cluster operations are done at once and the cluster is ACTIVE immediately) |
FLOCI_GCP_SERVICES_KAFKA_CONNECT_IMAGE |
apache/kafka:4.3.1 |
Kafka Connect worker image for Connect clusters; must keep the Apache Kafka layout under /opt/kafka |
Quick Start
# Create a cluster
curl -X POST \
"http://localhost:4588/v1/projects/floci-local/locations/us-central1/clusters" \
-H "Content-Type: application/json" \
-d '{
"clusterId": "my-cluster",
"cluster": {
"capacityConfig": { "vcpuCount": 3, "memoryBytes": 3221225472 },
"gcpConfig": { "accessConfig": { "networkConfigs": [{ "subnet": "projects/floci-local/regions/us-central1/subnetworks/default" }] } }
}
}'
# List clusters
curl "http://localhost:4588/v1/projects/floci-local/locations/us-central1/clusters"
# Create a topic
curl -X POST \
"http://localhost:4588/v1/projects/floci-local/locations/us-central1/clusters/my-cluster/topics" \
-H "Content-Type: application/json" \
-d '{"topicId":"my-topic","topic":{"partitionCount":3,"replicationFactor":1}}'
# List topics
curl "http://localhost:4588/v1/projects/floci-local/locations/us-central1/clusters/my-cluster/topics"
Mock Mode
Set FLOCI_GCP_SERVICES_KAFKA_MOCK=true to use mock mode. In mock mode, clusters are created in memory and return ACTIVE state immediately without requiring a backing Redpanda container. Useful for testing Terraform or SDK code that provisions Kafka resources but does not produce or consume messages.
Long-running operations
CreateCluster, UpdateCluster and DeleteCluster return a google.longrunning.Operation whose
metadata is google.cloud.managedkafka.v1.OperationMetadata (createTime, endTime, target,
verb, apiVersion). The response is the Cluster for create and update and google.protobuf.Empty
for delete, each with its @type. Operations are served at
/v1/projects/{project}/locations/{location}/operations (get, list, delete, cancel), so clients
that poll, such as createClusterAsync(...).get() in the Java SDK, work against floci.
With Docker (the default):
CreateClusterstores the cluster asCREATINGand returns the operation running. Redpanda starts in the background; when its admin API answers, the cluster turnsACTIVEand the operation finishes with the cluster. If the broker cannot start, or is not ready within 90 seconds, the operation finishes with an error (INTERNALorDEADLINE_EXCEEDED) and the cluster is removed. If Docker cannot remove the broker either, the cluster is kept asACTIVEinstead, so floci still stops its broker on exit and a delete can retry the removal.DeleteClusterturns the clusterDELETINGand returns the operation running; it finishes once Docker confirms the broker container and (under the volume retention policy) its volume are gone. If Docker cannot remove them, the cluster returns toACTIVEand the operation carries the error, so the delete can be retried.UpdateClusteronly changes stored capacity (the broker is not resized), so its operation is done at once.- While a cluster is not
ACTIVE,UpdateCluster,DeleteCluster, topic writes, ACL writes and every consumer group call return400 FAILED_PRECONDITION. Topic and ACL reads still work. - Work still running when floci stops cannot finish. On the next start every cluster left
CREATINGorDELETINGhas its broker container removed, including one the previous run left running, and is removed itself; its operation, if it still exists, finishes asABORTEDfor a create and done for a delete. If Docker cannot remove the broker, the cluster becomesACTIVEagain so it can be deleted, and a delete operation fails withINTERNAL.
In mock mode every operation is done when it is returned and the cluster is ACTIVE at once.
Deviations and unverified behaviour:
CancelOperationsucceeds and leaves the operation unchanged; provisioning and teardown always run to completion.- Which calls real Managed Kafka refuses on a
CREATINGorDELETINGcluster, and with which status, was not verified against the live service. - Operation ids are UUIDs, not the
operation-<timestamp>-<hash>form GCP uses. - Eventarc serves operations on the same path. A list never mixes the two APIs, since an SDK client
cannot unpack the other API's metadata type: a request whose host is
managedkafka.googleapis.comlists Managed Kafka operations, and any other host (such aslocalhost:4588) lists Eventarc's, as before Managed Kafka shared the path.get,deleteandcancelfind an operation by name whichever API owns it, so polling a Kafka operation works through any host. To list Kafka operations through floci's endpoint, a Managed Kafka client must send themanagedkafka.googleapis.comhost (for example aHostheader); without it the list is Eventarc's. - A name no API owns is
404 NOT_FOUND, unless Managed Kafka and Eventarc are both disabled, when it is503 UNAVAILABLEas on their own routes.
Consumer Groups
# List consumer groups
curl "http://localhost:4588/v1/projects/floci-local/locations/us-central1/clusters/my-cluster/consumerGroups"
# Get a specific consumer group
curl "http://localhost:4588/v1/projects/floci-local/locations/us-central1/clusters/my-cluster/consumerGroups/my-group"
# Delete a consumer group
curl -X DELETE \
"http://localhost:4588/v1/projects/floci-local/locations/us-central1/clusters/my-cluster/consumerGroups/my-group"
Supported Operations
Clusters:
CreateClusterGetClusterListClustersUpdateClusterDeleteCluster
Operations: GetOperation, ListOperations, DeleteOperation, CancelOperation
Topics:
CreateTopicGetTopicListTopicsUpdateTopicDeleteTopic
Consumer Groups:
GetConsumerGroupListConsumerGroupsUpdateConsumerGroupDeleteConsumerGroup
ACLs:
CreateAcl,GetAcl,ListAcls,UpdateAcl,DeleteAclAddAclEntry,RemoveAclEntry
Kafka Connect (control plane only, no Connect runtime):
CreateConnectClusterGetConnectClusterListConnectClustersUpdateConnectClusterDeleteConnectClusterCreateConnectorGetConnectorListConnectorsUpdateConnectorDeleteConnectorPauseConnectorResumeConnectorRestartConnectorStopConnector
Schema Registry (ManagedSchemaRegistry):
CreateSchemaRegistry,GetSchemaRegistry,ListSchemaRegistries,DeleteSchemaRegistryGetContext,ListContextsCreateVersion,LookupVersion,GetVersion,GetRawSchemaVersion,ListVersions,DeleteVersionListSubjects,DeleteSubject,ListReferencedSchemasGetSchema,GetRawSchema,ListSchemaVersions,ListSubjectsBySchemaId,ListSchemaTypesCheckCompatibilityGetSchemaConfig,UpdateSchemaConfig,DeleteSchemaConfigGetSchemaMode,UpdateSchemaMode,DeleteSchemaMode
This is the full ManagedKafka v1 RPC surface plus ManagedKafkaConnect and ManagedSchemaRegistry.
ACLs
An ACL is addressed by an acl_id that encodes the Kafka resource pattern, exactly as the real
API spells it: cluster; topic/{name}, consumerGroup/{name}, transactionalId/{name};
topicPrefixed/{name}, consumerGroupPrefixed/{name}, transactionalIdPrefixed/{name}; and
allTopics, allConsumerGroups, allTransactionalIds. Anything else is 400 INVALID_ARGUMENT.
The output-only resourceType, resourceName and patternType fields are derived from the id.
B=http://localhost:4588/v1/projects/p/locations/us-central1/clusters/c
curl -s -X POST "$B/acls?aclId=topic/orders" -H 'Content-Type: application/json' \
-d '{"aclEntries":[{"principal":"User:svc@p.iam.gserviceaccount.com","permissionType":"ALLOW","operation":"READ","host":"*"}]}'
curl -s -X POST "$B/acls/topic/orders:addAclEntry" -H 'Content-Type: application/json' \
-d '{"principal":"User:svc@p.iam.gserviceaccount.com","permissionType":"ALLOW","operation":"WRITE","host":"*"}'
curl -s "$B/acls/topic/orders"
- Entries follow the proto's field rules:
principalcarries theUser:prefix (or isUser:*),permissionTypeisALLOWorDENY,operationis one of the Kafka operations (ALL,READ,WRITE,CREATE,DELETE,ALTER,DESCRIBE,CLUSTER_ACTION,DESCRIBE_CONFIGS,ALTER_CONFIGS,IDEMPOTENT_WRITE), matched case-insensitively and stored upper-case, andhostmust be*. At most 100 entries per ACL. addAclEntrycreates the ACL if it does not exist (aclCreated: true);removeAclEntrydeletes it when the last entry goes (aclDeleted: true). Adding an identical entry twice is a no-op.etagchanges on every write.UpdateAclwith a staleetagis409 ABORTED; without one it is unconditional.updateMaskmay only nameaclEntries.- ACLs are control-plane metadata, like topics in this emulator: the Redpanda container runs without an authorizer, so entries are recorded and read back but do not gate produce or consume.
Kafka Connect
The ManagedKafkaConnect service (connectClusters and their connectors) attaches a Connect
cluster to a Kafka cluster in the same project and location.
Outside mock mode, each Connect cluster runs a real Kafka Connect worker (distributed mode, one member) in a container beside the Kafka cluster's Redpanda container:
CreateConnectClusterreturns once the worker serves its REST API, usually within ten seconds. The Kafka cluster must beACTIVE, otherwise the call is400 FAILED_PRECONDITION.- Connectors are created, updated, paused, resumed, restarted, stopped and deleted on the worker,
and
stateis read back from it, so a misconfigured connector reportsFAILEDand one the worker has not assigned yet reportsUNASSIGNED. A connector with any failed task also reportsFAILED, since that task is not processing records. A config the worker refuses (for example an unknownconnector.class) is400 INVALID_ARGUMENTand nothing is stored. - Deleting the Kafka cluster a Connect cluster is attached to stops that Connect cluster's worker.
The Connect cluster remains, its connectors report
FAILED, and connector writes are400 FAILED_PRECONDITIONuntil itsconfigis changed (which starts a new worker on the current Kafka cluster, without the old connectors) or it is deleted. The same applies after an emulator restart, since workers are not restarted. - The worker's REST API is published on
127.0.0.1only when the emulator runs on the host, and is not published at all when it runs in a container: only the emulator drives it. - The worker keeps connector configs, offsets and status in three internal topics on the Kafka
cluster (
_floci-connect-<id>-configs,-offsets,-status), which is what makes that cluster the Connect cluster's primary cluster. Deleting the Connect cluster removes the worker and those topics; topics the connectors wrote are kept. configon the Connect cluster is applied as Kafka Connect worker properties. Changing it restarts the worker, and connectors survive the restart.bootstrap.servers,group.id,listenersand the three storage topics are owned by the emulator and cannot be overridden.- The default image ships the MirrorMaker 2 connectors (
MirrorSourceConnector,MirrorCheckpointConnector,MirrorHeartbeatConnector). To run other connectors, build an image fromapache/kafkawith the plugins added, setFLOCI_GCP_SERVICES_KAFKA_CONNECT_IMAGE, and pointplugin.pathat them through the Connect cluster'sconfig. Inside the worker, the Kafka cluster is reachable asfloci-kafka-broker:29092.
In mock mode nothing is started: a Connect cluster is ACTIVE immediately, connectors are
metadata, any connector.class is accepted, and state follows the lifecycle calls (RUNNING,
PAUSED, STOPPED).
BASE=http://localhost:4588/v1/projects/floci-local/locations/us-central1
# Create a Connect cluster bound to an existing Kafka cluster (returns a done LRO)
curl -X POST "$BASE/connectClusters?connectClusterId=my-connect" -H 'Content-Type: application/json' -d '{
"kafkaCluster": "projects/floci-local/locations/us-central1/clusters/my-cluster",
"capacityConfig": {"vcpuCount": 12, "memoryBytes": "21474836480"},
"gcpConfig": {"accessConfig": {"networkConfigs": [
{"primarySubnet": "projects/floci-local/regions/us-central1/subnetworks/default"}]}}
}'
# Create a MirrorMaker 2 heartbeat connector, then pause and resume it
curl -X POST "$BASE/connectClusters/my-connect/connectors?connectorId=heartbeats" -H 'Content-Type: application/json' -d '{
"configs": {
"connector.class": "org.apache.kafka.connect.mirror.MirrorHeartbeatConnector",
"source.cluster.alias": "primary", "target.cluster.alias": "primary",
"source.cluster.bootstrap.servers": "floci-kafka-broker:29092",
"target.cluster.bootstrap.servers": "floci-kafka-broker:29092",
"tasks.max": "1"
},
"taskRestartPolicy": {"minimumBackoff": "60s", "maximumBackoff": "1800s"}
}'
curl -X POST "$BASE/connectClusters/my-connect/connectors/heartbeats:pause" -H 'Content-Type: application/json' -d '{}'
curl "$BASE/connectClusters/my-connect/connectors/heartbeats" # "state": "PAUSED"
curl -X POST "$BASE/connectClusters/my-connect/connectors/heartbeats:resume" -H 'Content-Type: application/json' -d '{}'
kafkaCluster is immutable after create; PATCH honours updateMask (labels, capacityConfig,
gcpConfig, config on a Connect cluster; configs, taskRestartPolicy on a connector) and, like
UpdateCluster, applies the mutable fields present in the body when no mask is sent. Deleting a
Connect cluster deletes its connectors. taskRestartPolicy has no Kafka Connect counterpart, so it
is stored and read back but does not change how the worker restarts tasks. Workers do not outlive
the emulator process: they stop on shutdown, like the Redpanda containers, and are not restarted
on the next start.
Schema Registry
A schema registry lives under a project and location, not under a Kafka cluster, and a project
can have several. The routes under a registry mirror the Confluent Schema Registry API, so a
Confluent serializer or deserializer configured with
schema.registry.url=http://localhost:4588/v1/projects/<project>/locations/<location>/schemaRegistries/<registry>
works unchanged (request bodies are accepted with application/vnd.schemaregistry.v1+json).
Outside mock mode, each registry runs in its own Redpanda container (the same
FLOCI_GCP_SERVICES_KAFKA_DEFAULT_IMAGE), whose built-in schema registry stores the schemas and
checks Avro, Protobuf and JSON Schema compatibility. CreateSchemaRegistry returns once the
registry answers, usually within a few seconds. The container stops with the emulator and starts
again on the registry's first use after a restart; its data is kept on the registry's volume.
Deleting the registry removes the container. Its data is removed too when storage mode is
memory; in the persistent modes the volume (or host directory) is kept unless
prune-volumes-on-delete is enabled, which it is not by default.
In mock mode the registry is kept in the emulator's storage with the Confluent rules for schema
ids, versions, soft and hard deletes, config and mode, but schemas are not checked for
compatibility (CheckCompatibility always answers is_compatible: true) and only Avro and JSON
Schema payloads are checked for being valid JSON.
R=http://localhost:4588/v1/projects/floci-local/locations/us-central1/schemaRegistries
curl -X POST "$R" -H 'Content-Type: application/json' -d '{"schemaRegistryId":"my_registry"}'
# Register a schema (CreateVersion returns {"id": ...}) and read it back by id
curl -X POST "$R/my_registry/subjects/orders-value/versions" \
-H 'Content-Type: application/vnd.schemaregistry.v1+json' \
-d '{"schema":"{\"type\":\"record\",\"name\":\"Order\",\"fields\":[{\"name\":\"id\",\"type\":\"string\"}]}"}'
curl "$R/my_registry/schemas/ids/1"
curl "$R/my_registry/subjects/orders-value/versions/latest"
# Compatibility level and mode
curl -X PUT "$R/my_registry/config/orders-value" -H 'Content-Type: application/json' -d '{"compatibility":"FULL"}'
curl -X PUT "$R/my_registry/mode" -H 'Content-Type: application/json' -d '{"mode":"READONLY"}'
Contexts
Every route under a registry also takes the proto's contexts/{context} binding, which scopes it
to that context: .../schemaRegistries/r/contexts/.team/subjects/orders/versions registers
orders in context .team. Contexts follow the Confluent API: each has its own schema ids, a
subject written :.team:orders belongs to .team wherever it is addressed, subjects outside the
default context are reported in that qualified form, and a context with no config or mode of its
own uses the built in default (BACKWARD, READWRITE), not the registry's. A context exists once a subject is registered in it; the default context,
., always exists. HTTP clients collapse a contexts/. path segment, so the default context is
addressed by leaving the contexts/{context} binding out.
ListContexts returns the Confluent context ids ([".", ".team"]), SchemaRegistry.contexts
their resource names (.../schemaRegistries/r/contexts/.team), and GetContext a context's name
and the resource names of its subjects (.../contexts/.team/subjects/orders). Listing registries reads each
registry's contexts, so after an emulator restart it starts every registry's container.
Deviations and unverified behaviour:
- The methods that return
google.api.HttpBodyin the proto (ListContexts,ListSubjects,DeleteSubject,ListVersions,DeleteVersion,ListSchemaVersions,ListSchemaTypes,ListSubjectsBySchemaId,ListReferencedSchemas) return the Confluent JSON as the body.GetRawSchemaandGetRawSchemaVersionreturn the schema text:application/jsonfor Avro and JSON Schema,text/plainfor a Protobuf schema. - The resource names in
SchemaRegistry.contextsandContext.subjects, and the name of the default context (.), follow the proto's patterns but were not checked against production. - Registry errors use the Google error envelope with the Confluent message: an unknown subject,
version or schema is
404 NOT_FOUND, an incompatible schema is400 FAILED_PRECONDITION, and an invalid schema, compatibility level or mode, or a write inREADONLYmode, is400 INVALID_ARGUMENT. Production's status codes and messages for these were not verified. - A context with no context level config answers
404with code40408and the messageContext '.team' does not have context-level compatibility configured, as Redpanda does; Confluent's message for the same case isSubject ':.team:' does not have subject-level compatibility configured. - Which config and mode production applies to a context with none of its own was not verified; floci follows Confluent 7.9 and Redpanda (the built in default, not the registry's).
UpdateSchemaConfigwithnormalize: trueis400 INVALID_ARGUMENT: Redpanda does not storenormalizein a config. PassnormalizeonCreateVersionorLookupVersioninstead.CheckCompatibilityagainstcompatibility/subjects/<subject>/versions(no version) checks every version for the*_TRANSITIVElevels and the latest version otherwise, and answersis_compatible: truefor a subject with no versions.- The registry id must contain only letters, digits and underscores, up to 63 characters, and
must not start with a digit, as the proto describes; other ids are
400 INVALID_ARGUMENT.