Pub/Sub
floci-gcp emulates Google Cloud Pub/Sub over gRPC using the real google.pubsub.v1 protocol.
It also exposes the Pub/Sub REST v1 JSON surface used by tools such as Terraform.
Configuration
| Variable | Default | Description |
|---|---|---|
FLOCI_GCP_SERVICES_PUBSUB_ENABLED |
true |
Enable/disable Pub/Sub |
Emulator Variable
GCP Pub/Sub SDK clients use this variable to route requests to floci-gcp instead of pubsub.googleapis.com.
IAM enforcement
With FLOCI_GCP_SERVICES_IAM_AUTHORIZATION_MODE=enforce, Floci-issued
service-account credentials are checked against topic, subscription, snapshot,
and inherited project policies. Anonymous and external credentials retain the
emulator's setup bypass. The default disabled mode leaves operations permissive.
REST JSON and gRPC topic and subscription data methods check the corresponding
pubsub.topics.* or pubsub.subscriptions.* permission. Policy reads and writes
check the resource's getIamPolicy or setIamPolicy permission; testIamPermissions
filters its response. Both transports check the snapshot methods and Seek; the
gRPC transport also checks ListTopicSubscriptions, DetachSubscription,
ModifyPushConfig, ModifyAckDeadline, and StreamingPull. Topic creation checks
pubsub.topics.create using the proposed topic name and its inherited project
policy, so a condition on resource.name can scope creation. Subscription creation checks
pubsub.subscriptions.create on the containing project and
pubsub.topics.attachSubscription on its topic. Snapshot creation checks
pubsub.snapshots.create on the containing project and
pubsub.subscriptions.consume on its subscription. Seek requires
pubsub.subscriptions.consume and, when seeking to a snapshot,
pubsub.snapshots.seek. ListTopicSubscriptions and ListTopicSnapshots require
pubsub.topics.get; DetachSubscription requires
pubsub.topics.detachSubscription on the subscription's topic. StreamingPull
requires pubsub.subscriptions.consume on the subscription named in its first
message. Later messages must omit the subscription name. Floci checks active
streams again before delivering newly available messages and closes a stream
if its grant has been revoked. This stream-closing behavior is an emulator
choice, not a verified behavior of Google Cloud Pub/Sub.
Supported predefined roles are roles/pubsub.admin, roles/pubsub.editor,
roles/pubsub.viewer, roles/pubsub.publisher, and roles/pubsub.subscriber.
Admin grants the implemented Pub/Sub permissions. Editor excludes IAM policy
read and write; viewer grants implemented metadata reads and lists; publisher
grants topic publish; subscriber grants subscription consume, topic attachment,
and snapshot seek. Project basic roles contribute corresponding supported
Pub/Sub permissions.
Version 3 policy conditions can use the shared restricted profile: resource.name,
resource.type, resource.service, and request.time, with supported string,
logical, and time comparisons. Pub/Sub resource names are
projects/{project}/topics/{topic}, projects/{project}/subscriptions/{subscription},
and projects/{project}/snapshots/{snapshot}. Types are
pubsub.googleapis.com/Topic, pubsub.googleapis.com/Subscription, and
pubsub.googleapis.com/Snapshot. A list call evaluates its project resource,
so a condition limited to one child name does not grant listing.
Permission names and multi-resource checks follow Google's Pub/Sub access-control reference. Role slices follow the Pub/Sub predefined-role catalog. Request fields and StreamingPull's first-message rule follow the Pub/Sub v1 proto.
REST / Terraform Endpoint
Terraform's Google provider can use the REST endpoint with:
The REST surface supports topic, subscription and snapshot create/read/list/update/delete, publish, pull, acknowledge, modifyAckDeadline, and seek:
PUT /v1/projects/{project}/topics/{topic}GET /v1/projects/{project}/topics/{topic}GET /v1/projects/{project}/topicsPATCH /v1/projects/{project}/topics/{topic}?updateMask=...DELETE /v1/projects/{project}/topics/{topic}POST /v1/projects/{project}/topics/{topic}:publishPUT /v1/projects/{project}/subscriptions/{subscription}GET /v1/projects/{project}/subscriptions/{subscription}GET /v1/projects/{project}/subscriptionsPATCH /v1/projects/{project}/subscriptions/{subscription}?updateMask=...DELETE /v1/projects/{project}/subscriptions/{subscription}POST /v1/projects/{project}/subscriptions/{subscription}:pullPOST /v1/projects/{project}/subscriptions/{subscription}:acknowledgePOST /v1/projects/{project}/subscriptions/{subscription}:modifyAckDeadlinePOST /v1/projects/{project}/subscriptions/{subscription}:seekPUT /v1/projects/{project}/snapshots/{snapshot}GET /v1/projects/{project}/snapshots/{snapshot}GET /v1/projects/{project}/snapshotsGET /v1/projects/{project}/topics/{topic}/snapshotsPATCH /v1/projects/{project}/snapshots/{snapshot}?updateMask=...DELETE /v1/projects/{project}/snapshots/{snapshot}
IAM policy methods are served for topics, subscriptions, and snapshots:
GET /v1/projects/{project}/topics/{topic}:getIamPolicyPOST /v1/projects/{project}/topics/{topic}:setIamPolicyPOST /v1/projects/{project}/topics/{topic}:testIamPermissionsGET /v1/projects/{project}/subscriptions/{subscription}:getIamPolicyPOST /v1/projects/{project}/subscriptions/{subscription}:setIamPolicyPOST /v1/projects/{project}/subscriptions/{subscription}:testIamPermissionsGET /v1/projects/{project}/snapshots/{snapshot}:getIamPolicyPOST /v1/projects/{project}/snapshots/{snapshot}:setIamPolicyPOST /v1/projects/{project}/snapshots/{snapshot}:testIamPermissions
Quick Start
export PUBSUB_EMULATOR_HOST=localhost:4588
gcloud config set project floci-local
# Create topic and subscription
gcloud pubsub topics create my-topic
gcloud pubsub subscriptions create my-sub --topic=my-topic
# Publish a message
gcloud pubsub topics publish my-topic --message="hello from floci-gcp"
# Pull messages
gcloud pubsub subscriptions pull my-sub --auto-ack --limit=10
ManagedChannel channel = ManagedChannelBuilder
.forTarget("localhost:4588")
.usePlaintext()
.build();
TransportChannelProvider channelProvider =
FixedTransportChannelProvider.create(GrpcTransportChannel.create(channel));
CredentialsProvider credentialsProvider = NoCredentialsProvider.create();
// Create topic
TopicAdminClient topicAdminClient = TopicAdminClient.create(
TopicAdminSettings.newBuilder()
.setTransportChannelProvider(channelProvider)
.setCredentialsProvider(credentialsProvider)
.build());
topicAdminClient.createTopic(TopicName.of("floci-local", "my-topic"));
// Create subscription
SubscriptionAdminClient subscriptionAdminClient = SubscriptionAdminClient.create(
SubscriptionAdminSettings.newBuilder()
.setTransportChannelProvider(channelProvider)
.setCredentialsProvider(credentialsProvider)
.build());
subscriptionAdminClient.createSubscription(
SubscriptionName.of("floci-local", "my-sub"),
TopicName.of("floci-local", "my-topic"),
PushConfig.getDefaultInstance(),
10);
// Publish
Publisher publisher = Publisher.newBuilder(TopicName.of("floci-local", "my-topic"))
.setChannelProvider(channelProvider)
.setCredentialsProvider(credentialsProvider)
.build();
PubsubMessage message = PubsubMessage.newBuilder()
.setData(ByteString.copyFromUtf8("hello from floci-gcp"))
.build();
publisher.publish(message).get();
// Pull
SubscriberStubSettings subscriberSettings = SubscriberStubSettings.newBuilder()
.setTransportChannelProvider(channelProvider)
.setCredentialsProvider(credentialsProvider)
.build();
try (SubscriberStub subscriber = GrpcSubscriberStub.create(subscriberSettings)) {
PullRequest pullRequest = PullRequest.newBuilder()
.setMaxMessages(10)
.setSubscription(SubscriptionName.of("floci-local", "my-sub").toString())
.build();
PullResponse response = subscriber.pullCallable().call(pullRequest);
response.getReceivedMessagesList().forEach(msg ->
System.out.println(msg.getMessage().getData().toStringUtf8()));
}
import os
os.environ["PUBSUB_EMULATOR_HOST"] = "localhost:4588"
from google.cloud import pubsub_v1
project_id = "floci-local"
# Create topic
publisher = pubsub_v1.PublisherClient()
topic_path = publisher.topic_path(project_id, "my-topic")
publisher.create_topic(request={"name": topic_path})
# Create subscription
subscriber = pubsub_v1.SubscriberClient()
sub_path = subscriber.subscription_path(project_id, "my-sub")
subscriber.create_subscription(request={
"name": sub_path,
"topic": topic_path,
})
# Publish
future = publisher.publish(topic_path, b"hello from floci-gcp")
future.result()
# Pull
response = subscriber.pull(request={"subscription": sub_path, "max_messages": 10})
for msg in response.received_messages:
print(msg.message.data.decode())
Subscription Filters
A subscription can declare a filter so it only receives messages whose attributes match it.
Non-matching messages are never delivered to that subscription. As in GCP, the service acknowledges
them automatically on your behalf. Subscriptions without a filter receive every message published
to the topic.
subscriptionAdminClient.createSubscription(Subscription.newBuilder()
.setName(SubscriptionName.of("floci-local", "invoices").toString())
.setTopic(TopicName.of("floci-local", "events").toString())
.setAckDeadlineSeconds(10)
.setFilter("attributes.event_type = \"ocr-invoice\"")
.build());
subscriber.create_subscription(request={
"name": sub_path,
"topic": topic_path,
"filter": 'attributes.event_type = "ocr-invoice"',
})
Supported syntax
| Form | Example | Matches |
|---|---|---|
| Attribute exists | attributes:name |
messages that carry a name attribute |
| Quoted key | attributes:"iana.org/language_tag" |
keys with characters other than hyphens, underscores or alphanumerics |
| Equality | attributes.name = "com" |
name is exactly com |
| Inequality | attributes.name != "com" |
name differs from com, including when the attribute is absent |
| Prefix | hasPrefix(attributes.name, "co") |
name starts with co |
| Conjunction | attributes:a AND attributes.b = "1" |
both operands match |
| Disjunction | attributes.a = "1" OR attributes.a = "2" |
either operand matches |
| Negation | NOT attributes:a / -attributes:a |
operand does not match |
Rules that mirror GCP:
- Keys and values are case-sensitive;
AND,ORandNOTmust be uppercase. NOThas the highest precedence;-is a unary alias for it.ANDandORcannot be combined without parentheses.a AND b OR cis a syntax error; writea AND (b OR c).hasPrefixis the only function: there is no regular-expression support.- String literals may contain unicode, hexadecimal and octal escape sequences, for example
attributes:"みんな". Escapes outside a string literal are invalid. - A filter must be at most 256 bytes.
An unparseable filter, or one over the byte limit, is rejected with INVALID_ARGUMENT at creation
time, rather than being accepted and silently ignored.
The filter is immutable
As in GCP, the filter is a property of the subscription that cannot change after creation. A
subscriptions.patch that names filter in its update mask is rejected with INVALID_ARGUMENT,
whatever value it carries. GCP rejects on the presence of the field in the mask, not on whether the
value differs, so restating the current filter fails too. A patch that does not name filter in its
mask succeeds and leaves the filter untouched, even when the request body carries one. The update
mask governs, so clients that echo a whole subscription back keep working.
GCP requires an update mask on subscriptions.patch; floci-gcp also accepts a patch without one and
treats it as replacing every field. On that path a body carrying the subscription's current filter is
accepted and the filter is left alone, a body carrying a different one is rejected, and a body
omitting it leaves the filter in place rather than clearing it.
To change a filter, follow the same path as in GCP: snapshot the subscription, create a new one with
the desired filter, Seek to the snapshot, move subscribers over, then delete the old subscription.
Push Subscriptions
floci-gcp supports push subscriptions: it delivers messages to an HTTP endpoint you configure:
subscriptionAdminClient.createSubscription(
SubscriptionName.of("floci-local", "my-sub"),
TopicName.of("floci-local", "my-topic"),
PushConfig.newBuilder()
.setPushEndpoint("http://my-app:8080/pubsub/push")
.build(),
0);
Messages are delivered as HTTP POST requests to the configured endpoint.
Snapshots
A snapshot keeps what its subscription had not acknowledged when it was created, plus every message published to the topic afterwards. Seeking a subscription on the same topic to it makes exactly those messages unacknowledged again, through the seeking subscription's filter; a push subscription gets them pushed again. A message still waiting in the subscription keeps its delivery attempt count.
A snapshot expires 7 days after the oldest unacknowledged message it captured (7 days after creation
when there was none), and CreateSnapshot fails with FAILED_PRECONDITION when that is less than an
hour away. Seeking to an expired snapshot returns NOT_FOUND, to a snapshot of another topic
INVALID_ARGUMENT, and seeking a detached subscription FAILED_PRECONDITION.
The retained messages are kept in memory, like the subscription queues: after a restart a snapshot still exists but seeking to it restores nothing.
Create and restore snapshots to replay messages:
// Create snapshot
snapshotAdminClient.createSnapshot(
SnapshotName.of("floci-local", "my-snapshot"),
SubscriptionName.of("floci-local", "my-sub"));
// Seek to snapshot (replay messages from snapshot point)
subscriptionAdminClient.seek(SeekRequest.newBuilder()
.setSubscription(SubscriptionName.of("floci-local", "my-sub").toString())
.setSnapshot(SnapshotName.of("floci-local", "my-snapshot").toString())
.build());
IAM Policies
IAM policies on topics, subscriptions, and snapshots are stored and returned.
In enforce mode, supported Pub/Sub methods evaluate them for Floci-issued
service-account credentials, including supported version 3 conditions.
Policy semantics:
getIamPolicyon an existing resource with no policy returns an empty policy with the well-known etagACAB, matching GCP.getIamPolicy/setIamPolicyagainst a resource that does not exist fail withNOT_FOUND.- The etag rotates on every write. A
setIamPolicycarrying a stale etag fails withABORTED(HTTP 409), so read-modify-write flows such as Terraform'sgoogle_pubsub_topic_iam_memberbehave as they do against real GCP. Omitting the etag performs a blind write. - Deleting a resource deletes its policy; recreating the same name starts empty.
testIamPermissionsreturns the caller's granted permissions inenforcemode. Indisabledmode and for bypass callers, it echoes the requested permissions. A missing resource returns an empty permission set.- Schemas are not implemented, so schema IAM paths are not served.
Over gRPC these methods are served by the standalone google.iam.v1.IAMPolicy
service (Pub/Sub declares IAM as a service-config mixin), which the Pub/Sub
SDKs call transparently via TopicAdminClient / SubscriptionAdminClient.
// Grant a worker publish access to a topic, then read it back
TopicName topic = TopicName.of("floci-local", "my-topic");
Policy policy = topicAdminClient.getIamPolicy(GetIamPolicyRequest.newBuilder()
.setResource(topic.toString()).build());
Policy updated = topicAdminClient.setIamPolicy(SetIamPolicyRequest.newBuilder()
.setResource(topic.toString())
.setPolicy(policy.toBuilder()
.addBindings(Binding.newBuilder()
.setRole("roles/pubsub.publisher")
.addMembers("serviceAccount:worker@floci-local.iam.gserviceaccount.com"))
.build())
.build());
Supported Operations
Publisher:
CreateTopicUpdateTopicDeleteTopicGetTopicListTopicsListTopicSubscriptionsListTopicSnapshotsDetachSubscriptionPublish
Subscriber:
CreateSubscriptionUpdateSubscriptionDeleteSubscriptionGetSubscriptionListSubscriptionsPullStreamingPullAcknowledgeModifyAckDeadlineModifyPushConfigCreateSnapshotGetSnapshotListSnapshotsUpdateSnapshotDeleteSnapshotSeek(a seek to a time only affects unacknowledged messages, because acknowledged messages are not retained here even withretain_acked_messages; see Snapshots for a seek to a snapshot)
IAM (google.iam.v1.IAMPolicy mixin):
GetIamPolicySetIamPolicyTestIamPermissions(filters granted permissions inenforcemode; empty set for missing resources)