Cloud Storage (GCS)
floci-gcp emulates Google Cloud Storage using the real GCP wire protocols:
- gRPC v2, bucket and object management plus streaming reads and writes
- REST XML, object operations (upload, download, delete, list objects)
- REST JSON, bucket and object management, IAM, ACLs, notifications, HMAC keys, and uploads
IAM enforcement scope
With FLOCI_GCP_SERVICES_IAM_AUTHORIZATION_MODE=enforce, supported REST bucket
metadata, bucket IAM-policy, retention-lock, storage-layout, and notification
operations evaluate stored bucket and owning-project policies. Bucket
testIamPermissions returns only granted permissions. Anonymous and unrecognized
external credentials are evaluated as anonymous, so allUsers bindings work.
Supported REST JSON and XML object reads, writes, updates, deletes, listing, compose, copy, rewrite, move, restore, resumable uploads, and XML multipart uploads use the same evaluator. GCS applies Credential Access Boundary checks before IAM and retains the source principal for valid downscoped credentials. ACLs and GCS v2 data methods are not IAM-enforced. See IAM enforcement and limitations.
Configuration
| Variable | Default | Description |
|---|---|---|
FLOCI_GCP_SERVICES_GCS_ENABLED |
true |
Enable/disable Cloud Storage |
FLOCI_GCP_SERVICES_GCS_UPLOAD_SESSION_IDLE_TIMEOUT_SECONDS |
604800 |
Idle time before an unfinished resumable or streaming upload session is dropped, matching the real GCS seven-day session window. Lower it on long-lived instances to reclaim buffered bytes sooner |
FLOCI_GCP_SERVICES_GCS_UPLOAD_SESSION_SWEEP_INTERVAL_SECONDS |
3600 |
How often abandoned upload sessions are swept; 0 disables the sweeper |
FLOCI_GCP_SERVICES_GCS_SOFT_DELETE_SWEEP_INTERVAL_SECONDS |
3600 |
How often soft-deleted objects past their hardDeleteTime are deleted for good; 0 disables the sweeper |
FLOCI_GCP_BASE_URL |
http://localhost:4588 |
Base URL embedded in object URLs and pre-signed URLs |
Emulator Variable
REST clients use STORAGE_EMULATOR_HOST. The Go gRPC client uses
STORAGE_EMULATOR_HOST_GRPC without a URL scheme. Java gRPC clients configure
the endpoint through StorageOptions.grpc().
Service Account Authentication
Clients can exercise the normal service-account OAuth flow by setting the
credential's token_uri to the emulator:
{
"type": "service_account",
"project_id": "floci-local",
"private_key_id": "test-key",
"private_key": "-----BEGIN PRIVATE KEY-----\n...\n-----END PRIVATE KEY-----\n",
"client_email": "test@floci-local.iam.gserviceaccount.com",
"client_id": "123456789",
"token_uri": "http://localhost:4588/token"
}
floci-gcp implements the standard OAuth JWT bearer exchange and returns a
short-lived Bearer token. With the default IAM authorization mode, as with other
emulator credentials, JWT signatures and Bearer tokens are not validated. In
enforce mode the assertion is verified against the service account's keys and
the token acts as that service account; see
OAuth token exchange.
The exchange is served at /token (Google's oauth2.googleapis.com/token) and
at /oauth2/v4/token (the legacy www.googleapis.com/oauth2/v4/token), and
accepts the request as application/x-www-form-urlencoded or application/json.
Quick Start
export STORAGE_EMULATOR_HOST=http://localhost:4588
# Create a bucket
gcloud storage buckets create gs://my-bucket
# Upload an object
echo "hello from floci-gcp" | gcloud storage cp - gs://my-bucket/hello.txt
# List objects
gcloud storage ls gs://my-bucket
# Download
gcloud storage cp gs://my-bucket/hello.txt -
# Delete
gcloud storage rm gs://my-bucket/hello.txt
Storage storage = StorageOptions.newBuilder()
.setHost("http://localhost:4588")
.setProjectId("floci-local")
.setCredentials(NoCredentials.getInstance())
.build()
.getService();
// Create bucket
Bucket bucket = storage.create(BucketInfo.of("my-bucket"));
// Upload object
Blob blob = storage.create(
BlobInfo.newBuilder("my-bucket", "hello.txt").build(),
"hello from floci-gcp".getBytes());
// Download object
byte[] content = storage.readAllBytes("my-bucket", "hello.txt");
// List objects
Page<Blob> blobs = storage.list("my-bucket");
blobs.iterateAll().forEach(b -> System.out.println(b.getName()));
// Delete object
storage.delete("my-bucket", "hello.txt");
Storage storage = StorageOptions.grpc()
.setHost("http://localhost:4588")
.setProjectId("floci-local")
.setCredentials(NoCredentials.getInstance())
.build()
.getService();
storage.create(BucketInfo.of("my-bucket"));
storage.create(
BlobInfo.newBuilder("my-bucket", "hello.txt").build(),
"hello from floci-gcp".getBytes());
import os
os.environ["STORAGE_EMULATOR_HOST"] = "http://localhost:4588"
from google.cloud import storage
client = storage.Client(project="floci-local")
# Create bucket
bucket = client.bucket("my-bucket")
client.create_bucket(bucket)
# Upload object
blob = bucket.blob("hello.txt")
blob.upload_from_string("hello from floci-gcp")
# Download object
content = blob.download_as_text()
# List objects
for b in client.list_blobs("my-bucket"):
print(b.name)
# Delete
blob.delete()
import { Storage } from "@google-cloud/storage";
const storage = new Storage({
apiEndpoint: "http://localhost:4588",
projectId: "floci-local",
});
// Create bucket
await storage.createBucket("my-bucket");
// Upload object
await storage.bucket("my-bucket").file("hello.txt")
.save("hello from floci-gcp");
// Download
const [content] = await storage.bucket("my-bucket").file("hello.txt").download();
// List objects
const [files] = await storage.bucket("my-bucket").getFiles();
files.forEach(f => console.log(f.name));
Multipart Upload
floci-gcp supports multipart (resumable) upload, the standard GCS mechanism for large objects. The GCP SDK uses this automatically for objects above a threshold.
Resumable Upload Sessions
POST /upload/storage/v1/b/{bucket}/o?uploadType=resumable opens a session and returns
the session URL in the Location header. Chunks go to that URL with a Content-Range
header, using either PUT or POST, the Java, Node and Python SDKs send PUT, the Go
SDK sends POST, and both are handled the same way.
The session URL is an authorization token. After the opening request is authorized,
status queries and chunks need only the session URL and may omit the Authorization
header. Keep session URLs secret and share them only with clients that may complete
the upload.
Session behavior matches GCS:
| Request to the session URL | Response |
|---|---|
| Chunk that leaves bytes missing | 308 with Range: bytes=0-<last received byte> |
| Chunk that completes the object | 200 with the object metadata |
Status query (Content-Range: bytes */<total> or bytes */*) |
308 with the received range, or 200 with the object metadata once complete |
| Chunk already received in full | 308 with the unchanged received range, without appending it again |
| Chunk starting past the received bytes | 503 with a text/plain body, matching GCS |
| Any request after completion | 200 with the stored object metadata |
Unknown or expired upload_id |
404 |
The offset gap is the one session error GCS does not serve as JSON. Its body is the bare
message as text/plain, and the Java SDK sniffs that content type to tell a client-side
data loss apart from a transient 503, failing fast rather than retrying. floci-gcp
reproduces the response byte for byte, down to the double space after Invalid request.:
HTTP/1.1 503 Service Unavailable
content-type: text/plain; charset=utf-8
content-length: 138
Invalid request. According to the Content-Range header, the upload offset is 4 byte(s), which exceeds already uploaded size of 0 byte(s).
The Go SDK sends X-GUploader-No-308: yes because 308 collides with the RFC 7238
"Permanent Redirect" semantics. floci-gcp answers those requests the way GCS does: 200
with X-HTTP-Status-Code-Override: 308 and the same Range header. Other status codes
are unaffected by the header.
Documented deviations from real GCS:
- GCS requires every non-final chunk to be a multiple of 256 KiB; floci-gcp accepts any chunk size.
- GCS answers a chunk
POSTthat carries nouploadTypewith405; floci-gcp accepts it, since theupload_idalone identifies the session. - Completed sessions are remembered for the most recent 1024 uploads rather than for a week.
Object Versioning
Enable versioning on a bucket:
Each overwrite creates a new object generation. List all versions:
Deleting the live object by name makes it a noncurrent version with timeDeleted set, also when
the bucket has a soft delete policy. A delete by generation soft-deletes that generation under a
soft delete policy and removes it otherwise.
Pre-signed URLs
Generate a pre-signed URL for temporary public access:
URL signedUrl = storage.signUrl(
BlobInfo.newBuilder("my-bucket", "hello.txt").build(),
15, TimeUnit.MINUTES,
Storage.SignUrlOption.withV4Signature());
Pre-signed URLs are generated using the FLOCI_GCP_BASE_URL as the base.
With IAM authorization in enforce mode, signed-URL identity is not evaluated:
because signatures are not cryptographically verified, these requests are treated
as anonymous and require an allUsers grant. The default disabled mode retains
the usual emulator behavior.
Virtual-Hosted Style URLs
floci-gcp supports virtual-hosted style GCS URLs:
The embedded DNS server resolves *.localhost.floci.io to floci-gcp's container IP when running inside Docker, so virtual-hosted URLs work from sidecar containers without extra DNS configuration.
Supported Operations
Cloud Storage gRPC v2:
- Buckets:
CreateBucket,GetBucket,ListBuckets,UpdateBucket,DeleteBucket - Objects:
ComposeObject,GetObject,ListObjects,UpdateObject,DeleteObject; object system metadata (cacheControl,contentDisposition,contentEncoding,contentLanguage,customTime,storageClass) round-trips with the REST path - Data path:
ReadObject,WriteObject,BidiWriteObject - Resumable writes:
StartResumableWrite,QueryWriteStatus
Bucket responses use projects/{projectNumber}, matching the REST bucket's
projectNumber. The number is derived from the project ID and is the same one
Resource Manager reports for GET /v1/projects/{project}, so each project gets
its own stable number. ListBuckets accepts either projects/{projectId} or
projects/{projectNumber} as the parent (REST ?project= accepts either too).
The owning project ID is persisted with the bucket (it is not part of the JSON
bucket resource), so listing by project survives a restart. Buckets persisted by
older emulator versions, which did not record it, are assigned to
floci-gcp.default-project-id and report that project's number.
The v2 MVP does not implement IAM policy RPCs, retention locking, rewrite,
move, restore, resumable cancellation, bidi reads, appendable objects, write
handles, or redirection. Unsupported RPCs return gRPC UNIMPLEMENTED.
Bucket management (REST JSON):
CreateBucket(names validated against the GCS naming rules; a duplicate reportsreason: conflict) (withlocation,customPlacementConfig.dataLocations,storageClass,versioning,lifecycle,cors,retentionPolicy)- Every bucket resource (REST and gRPC) reports
locationType:dual-regionfor the predefined dual-regionsASIA1,EUR4,EUR5,EUR7,EUR8andNAM4or whencustomPlacementConfignames two data locations,multi-regionforUS,EUandASIA, otherwiseregion.GET /b/{bucket}/storageLayoutreports the same value.customPlacementConfigis set at creation and echoed back; likelocation, it cannot be changed afterwards GetBucketListBuckets(withpageTokenpagination)UpdateBucket/PatchBucket. A JSONPATCHmergeslabelsinto the existing labels, and a label set tonullis removedDeleteBucketGetBucketIamPolicy/SetBucketIamPolicyTestBucketIamPermissions,GET /b/{bucket}/iam/testPermissions?permissions=...(the GCS spelling); the Cloud IAM stylePOST /b/{bucket}/iam:testPermissionsis also accepted
Bucket ACLs (REST JSON):
ListBucketAcl/CreateBucketAclGetBucketAcl/UpdateBucketAcl/DeleteBucketAclListDefaultObjectAcl/CreateDefaultObjectAclGetDefaultObjectAcl/UpdateDefaultObjectAcl/DeleteDefaultObjectAcl
Project resources (REST JSON):
GetServiceAccount(/storage/v1/projects/{project}/serviceAccount), the principal Cloud Storage publishes as; grant it publish rights before wiring notifications. It isservice-{projectNumber}@gs-project-accounts.iam.gserviceaccount.com, the same address Service Usage'sgenerateServiceIdentityreturns forstorage.googleapis.com, and an unknown project number returns 400 withUnknown project id: {project}and reasoninvalidhmacKeyscreate/list/get/update/delete (/storage/v1/projects/{project}/hmacKeys), the credentials S3-compatible clients present to GCS; the secret is returned once on create, and a key must be INACTIVE before it can be deleted
Object operations (REST XML + REST JSON):
PutObject(simple and multipart/resumable upload)GetObjectDeleteObjectListObjects(withpageToken,prefix,delimiterpagination, plusstartOffset,endOffset,matchGlobandincludeTrailingDelimiterfiltering)- Decompressive transcoding (an object stored with
contentEncoding: gzipis served decompressed to a client that did not sendAccept-Encoding: gzip, and as stored to one that did;Rangeis ignored on a transcoded read) CopyObject(includingsourceGenerationto copy or restore a noncurrent version)MoveObjectHeadObject- Media reads (
alt=mediadownloads,/download/storage/v1/..., and XML API GET and HEAD) carry the object's system metadata:ETag,Last-Modifiedin IMF-fixdate form,x-goog-generation,x-goog-metageneration,x-goog-storage-class,x-goog-hashand thex-goog-stored-content-*pair, plus the custom metadata asx-goog-meta-*. A client that reads an object's update time from the response header rather than the resource body relies onLast-Modified.x-goog-hashis one field per algorithm on the XML API and a single comma-separated field on the JSON API, which also omits it from a partial read Content-Lengthon a media read is the length of the bytes that read returns: the stored size, the range length on a206, or the decompressed size when the read is transcoded. HEAD reports the same value the matching GET would- The XML API advertises
Accept-Ranges: byteson a media read, except on a transcoded one, which GCS does not advertise ranges on. JSON API media responses do not advertise ranges at all, as on GCS. ARangethe emulator cannot make sense of is ignored and the whole object is served with a200: an unknown unit, an empty or unparseable set, a range whose last byte precedes its first, a position beyond the signed 64-bit integer range, and a set of ranges that does not walk forward, wherever the unreadable element sits in the set. A set whose every range begins after the previous one ends returns400InvalidArgument. A first byte at or past the end, or a suffix length of zero, returns416InvalidRange PatchObject(update metadata:contentType,contentDisposition,contentEncoding,contentLanguage,cacheControl,customTime, custom metadata)- System metadata at upload time (
contentEncoding,contentDisposition,contentLanguage,cacheControl,customTime,storageClass) in the JSON metadata part of a multipart/resumable upload; on the upload URL onlycontentEncodingis honoured, as on GCS customTimefollows the GCS rules: rendered in UTC, never removed once set (anullpatch or an unset gRPCcustom_timeunder the mask is a no-op), and a decrease is rejected with400/INVALID_ARGUMENTComposeObject(concatenate 1 to 32 source objects; 0 or more than 32 is a 400)RewriteObject(multi-call when the source and destination span storage classes or bucket locations: amaxBytesRewrittenPerCall, which must be a multiple of 1 MiB, below the object size returnsdone: falsewith arewriteTokenand the client loops until it completes; a same-class, same-location copy finishes in one call whatever limit is sent, as in GCS)hmacKeysare stored through the emulator's storage backend, so they survive restarts in persistent mode and are cleared by the reset endpoint- Soft delete (
softDeletePolicyon the bucket,?softDeleted=truelisting,objects.restore). A bucket created without a policy gets the GCS default ofretentionDurationSeconds: "604800"(7 days), REST and gRPC alike.0turns soft delete off, and any other value must fall between 7 and 90 days, otherwise the request fails with400andreason: invalid. Without Object Versioning, an object replaced by an upload, compose, copy, rewrite or move is soft-deleted, as a deleted one is; with versioning it becomes noncurrent.effectiveTimeis set by the server and moves forward when the retention grows. Buckets saved by older emulator versions keep the policy they had. Once itshardDeleteTimepasses, a soft-deleted object leaves the listing and can no longer be restored, and a background sweep frees its bytes - Pre-signed GET/PUT URLs (V4 signature via IAM
SignBlob) - Batch requests (
/batch/storage/v1), including when the container's published port differs from its internal one (docker run -p 9000:4588) - Customer-supplied encryption keys (CSEK)
Object names containing /, spaces, +, or percent-encoded sequences round-trip correctly, the emulator preserves URI-encoded names exactly as real GCS does.
Pub/Sub notifications (REST JSON):
CreateNotification/ListNotifications/GetNotification/DeleteNotification(/storage/v1/b/{bucket}/notificationConfigs), object changes publish to the configured Pub/Sub topic in the local backend
Object ACLs (REST JSON):
-
ListObjectAcl/CreateObjectAcl -
GetObjectAcl/UpdateObjectAcl/DeleteObjectAcl
IAM allow-policy enforcement
Set FLOCI_GCP_SERVICES_IAM_AUTHORIZATION_MODE=enforce to evaluate supported
bucket IAM allow policies for REST JSON and XML bucket and object operations.
GCS gRPC v2 remains outside IAM allow-policy enforcement; its existing Credential
Access Boundary checks still apply. The default remains disabled, preserving the
emulator's no-auth behavior. See the IAM service for supported principals,
roles, conditions, bootstrap administration, and intentional exclusions.
Conditional requests (preconditions):
ifGenerationMatch/ifGenerationNotMatchifMetagenerationMatch/ifMetagenerationNotMatchifSourceGenerationMatch/ifSourceGenerationNotMatchfor object movesifSourceMetagenerationMatch/ifSourceMetagenerationNotMatchfor object moves- Returns HTTP 412 on precondition failure
- Enforced atomically on object mutation paths under object locks, with a monotonic generation sequence, concurrent writers with
ifGenerationMatch=0race safely (exactly one wins)
XML multipart uploads
Native XML object routes support POST ?uploads, PUT ?uploadId=...&partNumber=...,
GET parts, POST completion and DELETE abort. Parts and sessions use StorageFactory
and survive graceful restarts in persistent, hybrid and WAL modes. Incomplete
uploads do not appear in ordinary object inventory.
Bucket GET ?uploads answers with ListMultipartUploadsResult the way GCS does, as
text/html: Bucket, KeyMarker, UploadIdMarker, NextKeyMarker, then Prefix and
Delimiter when they were sent, NextUploadIdMarker, MaxUploads, IsTruncated, every
Upload entry (Key, UploadId, StorageClass, Initiated), every CommonPrefixes entry,
and EncodingType under encoding-type=url. A page holds at most max-uploads entries,
uploads and common prefixes together, in key order. The default and the ceiling are 1000.
max-uploads=0 returns an empty page that is not truncated. A negative, empty or
non-numeric max-uploads, an empty upload-id-marker and any encoding-type other than
the lower-case url return HTTP 400 InvalidArgument. key-marker skips every upload up to
and including that key. With upload-id-marker it skips only the uploads of that key up to
that ID. An upload-id-marker without a key-marker is ignored. encoding-type=url encodes
KeyMarker, NextKeyMarker, Prefix, Delimiter, every Key and every common prefix the
way object listing does. When a truncated page would end on a
common prefix, GCS answers HTTP 400 InvalidArgument without Details, and so does the
emulator. An empty key-marker, which GCS answers with HTTP 500, is treated as not sent.
Part numbers range from 1 to 10000. Completion validates increasing order, ETags
and the 5 MiB minimum for every non-final part. Retrying a part replaces that part;
a failed completion preserves the session. A part receipt in the completion is accepted
with or without its quotes. A receipt that names a part that was not uploaded, or a
wrong ETag, returns HTTP 404 NoSuchPart. The initiation result is sent as text/html,
as GCS sends it. The completion result is sent the way GCS
sends it: as text/html, with the ETag in the body only, a Location that is the XML
URL of the object, and the x-goog-stored-content-length, x-goog-hash,
x-goog-generation and x-goog-metageneration headers. Completed objects have CRC32C
and the multipart ETag described under XML API entity tags,
but no MD5 hash. Customer-supplied encryption, upload preconditions and lifecycle expiry
of incomplete sessions are unsupported. The
emulator buffers parts and completed bytes in memory. Object publication and session
removal are separate checkpoints; crash-atomic completion across those stores is not
guaranteed.
Authentication retains the existing emulator credential acceptance and CAB checks.
In IAM enforce mode, every XML multipart request also evaluates its documented
storage.multipartUploads.* and object permissions; an upload ID is not an
authenticator. Signed URL expiry checks do not prove cryptographic signature
enforcement. The SDK suite uses only synthetic credentials and fixture signing keys.
Multipart object metadata is finalized before storage publication and Pub/Sub or Eventarc finalization events. The event and stored generation agree on the ETag, CRC32C, content metadata and absence of MD5. Ordinary uploads continue to include their MD5. Object publication and multipart-session removal still use separate checkpoints; crash-atomic multipart completion is not guaranteed.
XML API entity tags
The XML API writes the ETag header of PUT, GET and HEAD, the ETag element of
ListObjects and the result of a multipart completion the way GCS does, in quotes.
An object with an MD5 carries that MD5 in hex. A multipart upload carries the MD5
of its part digests, a dash and the part count. A composite object, which has no
MD5, carries a dash followed by its resource etag. The JSON API, including
alt=media downloads, keeps the resource etag of the object.
XML API object listing
GET /{bucket} answers with ListBucketResult the way GCS does. Without list-type, or
with any value other than 2, the result has the version 1 shape: Name, Prefix, Marker,
NextMarker when the page is truncated, MaxKeys when max-keys was sent, Delimiter
when delimiter was sent, IsTruncated, then every Contents entry and then every
CommonPrefixes entry. list-type=2 replaces Marker and NextMarker with StartAfter,
ContinuationToken and NextContinuationToken, and adds KeyCount, the number of
Contents entries on the page. A continuation-token wins over start-after. An empty
start-after or continuation-token, with or without =, is echoed as an empty element and
does not move the page. A marker with a value sent with list-type=2 returns HTTP 400
NotImplemented, an empty one is ignored. A token the emulator did not issue returns HTTP 400
InvalidArgument.
A page holds at most max-keys entries, objects and common prefixes together, in name
order. max-keys=0 returns an empty page that is not truncated. A negative max-keys is
ignored, and a value that is not a number returns HTTP 400 InvalidArgument. Without
max-keys the emulator returns every entry on one page.
Each Contents entry carries Key, Generation, MetaGeneration, LastModified, ETag
and Size. encoding-type=url URL-encodes Prefix, Marker, NextMarker, StartAfter,
Delimiter, every Key and every common prefix the way GCS does: a space becomes +, the
characters ! $ ' ( ) * , ; : @ / ? stay as they are, and everything else outside letters,
digits, -, ., _ and ~ becomes a percent escape. The result then carries
Encoding-Type in version 1 and EncodingType in version 2. The response carries the bucket
metageneration in x-goog-metageneration. The versions listing is not supported over XML.
XML API error responses
Failures from supported XML object transfers, listings, and multipart operations use
application/xml with an Error root, a machine-readable Code, and a Message.
This does not depend on the request's Accept header. Missing buckets, objects
(including generations), and multipart uploads return distinct NoSuchBucket,
NoSuchKey, and NoSuchUpload codes with HTTP 404. A completion that names an unknown
part or a wrong part ETag returns NoSuchPart with HTTP 404. Other multipart receipt
and checksum errors retain their specific codes. Signed URL errors retain their optional
Details and ParameterName fields.
Anonymous permission denial returns HTTP 403 AccessDenied. A denied request
whose credentials resolve as external or unsupported returns HTTP 401
AuthenticationRequired. Unknown Floci tokens also return AuthenticationRequired;
expired Floci tokens return InvalidAuthentication on every retry. HTTP 401 XML
responses include a WWW-Authenticate: Bearer challenge. These failures use a
fixed message and a separate Details element for the failure explanation.
Valid credentials rejected by IAM or their Credential Access Boundary return
HTTP 403 AccessDenied.
The emulator still accepts external credentials when an operation is allowed by its configured authorization mode and policies. This remains an intentional emulator behavior, rather than cryptographic validation of external credentials. Expired Floci credential records are retained until credential state is cleared, so repeated lookups can distinguish expiry from an unknown token; they never authorize a request.
Unexpected storage and backend failures return a sanitized HTTP 500
InternalError. Their diagnostic exceptions are logged on the server. GCS JSON
and gRPC errors keep their existing contracts.
This mapping covers failures raised while executing the selected XML resource. Pre-resource routing, filters that abort a request, and response serialization failures remain separate framework boundaries. Service Usage retains its existing XML denial response. Exact messages for all upstream failures have not been validated against live GCS; the status and code mappings follow the Cloud Storage XML status reference, and credential distinctions follow the live observations recorded in issue #282.