Skip to content

Cloud Storage (GCS)

floci-gcp emulates Google Cloud Storage using the real GCP wire protocols:

  • gRPC v2, bucket and object management plus streaming reads and writes
  • REST XML, object operations (upload, download, delete, list objects)
  • REST JSON, bucket and object management, IAM, ACLs, notifications, HMAC keys, and uploads

IAM enforcement scope

With FLOCI_GCP_SERVICES_IAM_AUTHORIZATION_MODE=enforce, supported REST bucket metadata, bucket IAM-policy, retention-lock, storage-layout, and notification operations evaluate stored bucket and owning-project policies. Bucket testIamPermissions returns only granted permissions. Anonymous and unrecognized external credentials are evaluated as anonymous, so allUsers bindings work.

Supported REST JSON and XML object reads, writes, updates, deletes, listing, compose, copy, rewrite, move, restore, resumable uploads, and XML multipart uploads use the same evaluator. GCS applies Credential Access Boundary checks before IAM and retains the source principal for valid downscoped credentials. ACLs and GCS v2 data methods are not IAM-enforced. See IAM enforcement and limitations.

Configuration

Variable Default Description
FLOCI_GCP_SERVICES_GCS_ENABLED true Enable/disable Cloud Storage
FLOCI_GCP_SERVICES_GCS_UPLOAD_SESSION_IDLE_TIMEOUT_SECONDS 604800 Idle time before an unfinished resumable or streaming upload session is dropped, matching the real GCS seven-day session window. Lower it on long-lived instances to reclaim buffered bytes sooner
FLOCI_GCP_SERVICES_GCS_UPLOAD_SESSION_SWEEP_INTERVAL_SECONDS 3600 How often abandoned upload sessions are swept; 0 disables the sweeper
FLOCI_GCP_BASE_URL http://localhost:4588 Base URL embedded in object URLs and pre-signed URLs

Emulator Variable

export STORAGE_EMULATOR_HOST=http://localhost:4588
export STORAGE_EMULATOR_HOST_GRPC=localhost:4588

REST clients use STORAGE_EMULATOR_HOST. The Go gRPC client uses STORAGE_EMULATOR_HOST_GRPC without a URL scheme. Java gRPC clients configure the endpoint through StorageOptions.grpc().

Service Account Authentication

Clients can exercise the normal service-account OAuth flow by setting the credential's token_uri to the emulator:

{
  "type": "service_account",
  "project_id": "floci-local",
  "private_key_id": "test-key",
  "private_key": "-----BEGIN PRIVATE KEY-----\n...\n-----END PRIVATE KEY-----\n",
  "client_email": "test@floci-local.iam.gserviceaccount.com",
  "client_id": "123456789",
  "token_uri": "http://localhost:4588/token"
}

floci-gcp implements the standard OAuth JWT bearer exchange and returns a short-lived Bearer token. As with other emulator credentials, JWT signatures and Bearer tokens are not validated.

Quick Start

export STORAGE_EMULATOR_HOST=http://localhost:4588

# Create a bucket
gcloud storage buckets create gs://my-bucket

# Upload an object
echo "hello from floci-gcp" | gcloud storage cp - gs://my-bucket/hello.txt

# List objects
gcloud storage ls gs://my-bucket

# Download
gcloud storage cp gs://my-bucket/hello.txt -

# Delete
gcloud storage rm gs://my-bucket/hello.txt

Storage storage = StorageOptions.newBuilder()
    .setHost("http://localhost:4588")
    .setProjectId("floci-local")
    .setCredentials(NoCredentials.getInstance())
    .build()
    .getService();

// Create bucket
Bucket bucket = storage.create(BucketInfo.of("my-bucket"));

// Upload object
Blob blob = storage.create(
    BlobInfo.newBuilder("my-bucket", "hello.txt").build(),
    "hello from floci-gcp".getBytes());

// Download object
byte[] content = storage.readAllBytes("my-bucket", "hello.txt");

// List objects
Page<Blob> blobs = storage.list("my-bucket");
blobs.iterateAll().forEach(b -> System.out.println(b.getName()));

// Delete object
storage.delete("my-bucket", "hello.txt");

Storage storage = StorageOptions.grpc()
    .setHost("http://localhost:4588")
    .setProjectId("floci-local")
    .setCredentials(NoCredentials.getInstance())
    .build()
    .getService();

storage.create(BucketInfo.of("my-bucket"));
storage.create(
    BlobInfo.newBuilder("my-bucket", "hello.txt").build(),
    "hello from floci-gcp".getBytes());

os.Setenv("STORAGE_EMULATOR_HOST_GRPC", "localhost:4588")
client, err := storage.NewGRPCClient(ctx)
if err != nil {
    log.Fatal(err)
}
defer client.Close()

import os
os.environ["STORAGE_EMULATOR_HOST"] = "http://localhost:4588"

from google.cloud import storage

client = storage.Client(project="floci-local")

# Create bucket
bucket = client.bucket("my-bucket")
client.create_bucket(bucket)

# Upload object
blob = bucket.blob("hello.txt")
blob.upload_from_string("hello from floci-gcp")

# Download object
content = blob.download_as_text()

# List objects
for b in client.list_blobs("my-bucket"):
    print(b.name)

# Delete
blob.delete()

import { Storage } from "@google-cloud/storage";

const storage = new Storage({
  apiEndpoint: "http://localhost:4588",
  projectId: "floci-local",
});

// Create bucket
await storage.createBucket("my-bucket");

// Upload object
await storage.bucket("my-bucket").file("hello.txt")
    .save("hello from floci-gcp");

// Download
const [content] = await storage.bucket("my-bucket").file("hello.txt").download();

// List objects
const [files] = await storage.bucket("my-bucket").getFiles();
files.forEach(f => console.log(f.name));

Multipart Upload

floci-gcp supports multipart (resumable) upload, the standard GCS mechanism for large objects. The GCP SDK uses this automatically for objects above a threshold.

Resumable Upload Sessions

POST /upload/storage/v1/b/{bucket}/o?uploadType=resumable opens a session and returns the session URL in the Location header. Chunks go to that URL with a Content-Range header, using either PUT or POST, the Java, Node and Python SDKs send PUT, the Go SDK sends POST, and both are handled the same way.

The session URL is an authorization token. After the opening request is authorized, status queries and chunks need only the session URL and may omit the Authorization header. Keep session URLs secret and share them only with clients that may complete the upload.

Session behavior matches GCS:

Request to the session URL Response
Chunk that leaves bytes missing 308 with Range: bytes=0-<last received byte>
Chunk that completes the object 200 with the object metadata
Status query (Content-Range: bytes */<total> or bytes */*) 308 with the received range, or 200 with the object metadata once complete
Chunk already received in full 308 with the unchanged received range, without appending it again
Chunk starting past the received bytes 503 with a text/plain body, matching GCS
Any request after completion 200 with the stored object metadata
Unknown or expired upload_id 404

The offset gap is the one session error GCS does not serve as JSON. Its body is the bare message as text/plain, and the Java SDK sniffs that content type to tell a client-side data loss apart from a transient 503, failing fast rather than retrying. floci-gcp reproduces the response byte for byte, down to the double space after Invalid request.:

HTTP/1.1 503 Service Unavailable
content-type: text/plain; charset=utf-8
content-length: 138

Invalid request.  According to the Content-Range header, the upload offset is 4 byte(s), which exceeds already uploaded size of 0 byte(s).

The Go SDK sends X-GUploader-No-308: yes because 308 collides with the RFC 7238 "Permanent Redirect" semantics. floci-gcp answers those requests the way GCS does: 200 with X-HTTP-Status-Code-Override: 308 and the same Range header. Other status codes are unaffected by the header.

Documented deviations from real GCS:

  • GCS requires every non-final chunk to be a multiple of 256 KiB; floci-gcp accepts any chunk size.
  • GCS answers a chunk POST that carries no uploadType with 405; floci-gcp accepts it, since the upload_id alone identifies the session.
  • Completed sessions are remembered for the most recent 1024 uploads rather than for a week.

Object Versioning

Enable versioning on a bucket:

storage.update(BucketInfo.newBuilder("my-bucket")
    .setVersioningEnabled(true)
    .build());

Each overwrite creates a new object generation. List all versions:

Page<Blob> versions = storage.list("my-bucket",
    Storage.BlobListOption.versions(true));

Pre-signed URLs

Generate a pre-signed URL for temporary public access:

URL signedUrl = storage.signUrl(
    BlobInfo.newBuilder("my-bucket", "hello.txt").build(),
    15, TimeUnit.MINUTES,
    Storage.SignUrlOption.withV4Signature());

Pre-signed URLs are generated using the FLOCI_GCP_BASE_URL as the base. With IAM authorization in enforce mode, signed-URL identity is not evaluated: because signatures are not cryptographically verified, these requests are treated as anonymous and require an allUsers grant. The default disabled mode retains the usual emulator behavior.

Virtual-Hosted Style URLs

floci-gcp supports virtual-hosted style GCS URLs:

http://my-bucket.localhost.floci.io:4588/hello.txt

The embedded DNS server resolves *.localhost.floci.io to floci-gcp's container IP when running inside Docker, so virtual-hosted URLs work from sidecar containers without extra DNS configuration.

Supported Operations

Cloud Storage gRPC v2:

  • Buckets: CreateBucket, GetBucket, ListBuckets, UpdateBucket, DeleteBucket
  • Objects: ComposeObject, GetObject, ListObjects, UpdateObject, DeleteObject; object system metadata (cacheControl, contentDisposition, contentEncoding, contentLanguage, customTime, storageClass) round-trips with the REST path
  • Data path: ReadObject, WriteObject, BidiWriteObject
  • Resumable writes: StartResumableWrite, QueryWriteStatus

Bucket responses use projects/{projectNumber}, matching the REST bucket's projectNumber. The number is derived from the project ID and is the same one Resource Manager reports for GET /v1/projects/{project}, so each project gets its own stable number. ListBuckets accepts either projects/{projectId} or projects/{projectNumber} as the parent (REST ?project= accepts either too). The owning project ID is persisted with the bucket (it is not part of the JSON bucket resource), so listing by project survives a restart. Buckets persisted by older emulator versions, which did not record it, are assigned to floci-gcp.default-project-id and report that project's number.

The v2 MVP does not implement IAM policy RPCs, retention locking, rewrite, move, restore, resumable cancellation, bidi reads, appendable objects, write handles, or redirection. Unsupported RPCs return gRPC UNIMPLEMENTED.

Bucket management (REST JSON):

  • CreateBucket (names validated against the GCS naming rules; a duplicate reports reason: conflict) (with location, customPlacementConfig.dataLocations, storageClass, versioning, lifecycle, cors, retentionPolicy)
  • Every bucket resource (REST and gRPC) reports locationType: dual-region for the predefined dual-regions ASIA1, EUR4, EUR5, EUR7, EUR8 and NAM4 or when customPlacementConfig names two data locations, multi-region for US, EU and ASIA, otherwise region. GET /b/{bucket}/storageLayout reports the same value. customPlacementConfig is set at creation and echoed back; like location, it cannot be changed afterwards
  • GetBucket
  • ListBuckets (with pageToken pagination)
  • UpdateBucket / PatchBucket
  • DeleteBucket
  • GetBucketIamPolicy / SetBucketIamPolicy
  • TestBucketIamPermissions, GET /b/{bucket}/iam/testPermissions?permissions=... (the GCS spelling); the Cloud IAM style POST /b/{bucket}/iam:testPermissions is also accepted

Bucket ACLs (REST JSON):

  • ListBucketAcl / CreateBucketAcl
  • GetBucketAcl / UpdateBucketAcl / DeleteBucketAcl
  • ListDefaultObjectAcl / CreateDefaultObjectAcl
  • GetDefaultObjectAcl / UpdateDefaultObjectAcl / DeleteDefaultObjectAcl

Project resources (REST JSON):

  • GetServiceAccount (/storage/v1/projects/{project}/serviceAccount), the principal Cloud Storage publishes as; grant it publish rights before wiring notifications
  • hmacKeys create/list/get/update/delete (/storage/v1/projects/{project}/hmacKeys), the credentials S3-compatible clients present to GCS; the secret is returned once on create, and a key must be INACTIVE before it can be deleted

Object operations (REST XML + REST JSON):

  • PutObject (simple and multipart/resumable upload)
  • GetObject
  • DeleteObject
  • ListObjects (with pageToken, prefix, delimiter pagination, plus startOffset, endOffset, matchGlob and includeTrailingDelimiter filtering)
  • Decompressive transcoding (an object stored with contentEncoding: gzip is served decompressed to a client that did not send Accept-Encoding: gzip, and as stored to one that did; Range is ignored on a transcoded read)
  • CopyObject
  • MoveObject
  • HeadObject
  • PatchObject (update metadata: contentType, contentDisposition, contentEncoding, contentLanguage, cacheControl, customTime, custom metadata)
  • System metadata at upload time (contentEncoding, contentDisposition, contentLanguage, cacheControl, customTime, storageClass) in the JSON metadata part of a multipart/resumable upload; on the upload URL only contentEncoding is honoured, as on GCS
  • customTime follows the GCS rules: rendered in UTC, never removed once set (a null patch or an unset gRPC custom_time under the mask is a no-op), and a decrease is rejected with 400 / INVALID_ARGUMENT
  • ComposeObject (concatenate 1 to 32 source objects; 0 or more than 32 is a 400)
  • RewriteObject (multi-call when the source and destination span storage classes or bucket locations: a maxBytesRewrittenPerCall, which must be a multiple of 1 MiB, below the object size returns done: false with a rewriteToken and the client loops until it completes; a same-class, same-location copy finishes in one call whatever limit is sent, as in GCS)
  • hmacKeys are stored through the emulator's storage backend, so they survive restarts in persistent mode and are cleared by the reset endpoint
  • Soft delete (softDeletePolicy on the bucket, ?softDeleted=true listing, objects.restore)
  • Pre-signed GET/PUT URLs (V4 signature via IAM SignBlob)
  • Batch requests (/batch/storage/v1), including when the container's published port differs from its internal one (docker run -p 9000:4588)
  • Customer-supplied encryption keys (CSEK)

Object names containing /, spaces, +, or percent-encoded sequences round-trip correctly, the emulator preserves URI-encoded names exactly as real GCS does.

Pub/Sub notifications (REST JSON):

  • CreateNotification / ListNotifications / GetNotification / DeleteNotification (/storage/v1/b/{bucket}/notificationConfigs), object changes publish to the configured Pub/Sub topic in the local backend

Object ACLs (REST JSON):

  • ListObjectAcl / CreateObjectAcl

  • GetObjectAcl / UpdateObjectAcl / DeleteObjectAcl

IAM allow-policy enforcement

Set FLOCI_GCP_SERVICES_IAM_AUTHORIZATION_MODE=enforce to evaluate supported bucket IAM allow policies for REST JSON and XML bucket and object operations. GCS gRPC v2 remains outside IAM allow-policy enforcement; its existing Credential Access Boundary checks still apply. The default remains disabled, preserving the emulator's no-auth behavior. See the IAM service for supported principals, roles, conditions, bootstrap administration, and intentional exclusions.

Conditional requests (preconditions):

  • ifGenerationMatch / ifGenerationNotMatch
  • ifMetagenerationMatch / ifMetagenerationNotMatch
  • ifSourceGenerationMatch / ifSourceGenerationNotMatch for object moves
  • ifSourceMetagenerationMatch / ifSourceMetagenerationNotMatch for object moves
  • Returns HTTP 412 on precondition failure
  • Enforced atomically on object mutation paths under object locks, with a monotonic generation sequence, concurrent writers with ifGenerationMatch=0 race safely (exactly one wins)

XML multipart uploads

Native XML object routes support POST ?uploads, PUT ?uploadId=...&partNumber=..., GET parts, POST completion and DELETE abort. Bucket GET ?uploads lists pending uploads with prefix and key/upload-ID markers. Parts and sessions use StorageFactory and survive graceful restarts in persistent, hybrid and WAL modes. Incomplete uploads do not appear in ordinary object inventory.

Part numbers range from 1 to 10000. Completion validates increasing order, ETags and the 5 MiB minimum for every non-final part. Retrying a part replaces that part; a failed completion preserves the session. Completed objects have CRC32C and an opaque ETag, but no MD5 hash. Multipart upload delimiter grouping, customer-supplied encryption, upload preconditions and lifecycle expiry of incomplete sessions are unsupported. The emulator buffers parts and completed bytes in memory. Object publication and session removal are separate checkpoints; crash-atomic completion across those stores is not guaranteed.

Authentication retains the existing emulator credential acceptance and CAB checks. In IAM enforce mode, every XML multipart request also evaluates its documented storage.multipartUploads.* and object permissions; an upload ID is not an authenticator. Signed URL expiry checks do not prove cryptographic signature enforcement. The SDK suite uses only synthetic credentials and fixture signing keys.

Multipart object metadata is finalized before storage publication and Pub/Sub or Eventarc finalization events. The event and stored generation agree on the opaque ETag, CRC32C, content metadata and absence of MD5. Ordinary uploads continue to include their MD5. Object publication and multipart-session removal still use separate checkpoints; crash-atomic multipart completion is not guaranteed.