Blob Storage
Compatible with the azure-storage-blob SDKs (Java, Python, Node.js), focused Java
azure-storage-file-datalake SDK flows, the Azure CLI (az storage blob), and Azurite-style
connection strings. Speaks the Azure Storage Blob REST protocol with Shared Key authentication,
Blob XML responses, and the Data Lake Storage Gen2 DFS host alias.
HTTP-only: no Docker. Data is held by the configured storage backend (
memoryby default;persistent,hybrid, orwalfor durability).
Features
- Containers: Create, Get properties, Delete, List (
?comp=list); duplicate create returns409 ContainerAlreadyExists - Blobs: Put (upload), Get (download), Delete, List within a container; overwrite semantics
- Block blobs: staged block upload (
?comp=block) followed by commit (?comp=blocklist) for large payloads, in addition to single-requestPut Blob - Data Lake Storage Gen2 endpoint alias: the
{account}.dfs.core.windows.nethost maps to the Blob backend so ADLS SDK path clients can create, read, write, and delete paths through the same local data store - User delegation key vending:
POST ?restype=service&comp=userdelegationkeyreturns Azure-shaped XML for SDK-generated user delegation SAS flows - User delegation SAS enforcement: validates SDK-generated user delegation SAS signatures, expiry, signed key validity, permissions, and container/blob/directory resource scope for Blob and ADLS path operations
- ADLS Gen2 / Hadoop ABFS 3.3.4 path operations - supports the DFS wire shapes used by
org.apache.hadoop:hadoop-azure:3.3.4: conditional file/directory create, path status and properties, append/flush, recursive delete, file/directory rename, POSIX owner/group/permission/ACL metadata,checkAccess, and the ADLS Path Lease POST API. Hadoop's default conditional-create overwrite flow (409-> status/ETag ->If-Match) is modeled explicitly. Access-control operations require an HNS-enabled account. Accounts listed inhierarchical-namespace-accountshave HNS enabled by default; ARM-created accounts can also enable it withproperties.isHnsEnabled: true. Other custom account names use a flat namespace by default. As in Azure, HNS is fixed when the account is created: a later PUT that sends a differentisHnsEnabledvalue returns400 AccountPropertyCannotBeUpdated, while a PUT that omits the property keeps the current setting. - ADLS filesystem operations - create/delete a filesystem and get/set filesystem properties via
?resource=filesystem. RootgetAccessControlis supported so Hadoop can auto-detect HNS and resolvegetFileStatus("/")without forcingfs.azure.account.hns.enabled=true. - ADLS Path - List - supports Java DataLake SDK
listPathsand HadooplistStatus, including recursive/non-recursive listing, directory filters, exact-file singleton results, server pagination, and Hadoop 3.3.4's Base64 HNSstartFromcontinuation token. - ADLS AppendBlob mode - recognizes Hadoop's
blobType=AppendBlobcreate mode and enforces sequential append positions, including append requests carryingflush=true/close=true. - Range download:
Range: bytes=…returns206 Partial Content - Conditional download:
If-Match/If-None-Matchhonored; a stale ETag is rejected - ETags: sent in the quoted
0xhex form the Blob service uses ("0x8DC1A2B3C4D5E6F"). List Blobs reports each blob's ETag without the quotes, as Azurite does. A container keeps the ETag andLast-Modifiedfrom Create Container, so Get Container Properties and List Containers repeat them - Metadata:
x-ms-meta-*set on upload and returned on Get, round-tripped exactly - Blob index tags: Get and Set Blob Tags (
?comp=tags, including on a snapshot),x-ms-tagson Put Blob and Put Block List, andx-ms-tag-counton Get Blob and Get Blob Properties. Tag sets are validated (at most 10 tags, keys of 1 to 128 characters, values of up to 256, letters, digits, space and+ - . / : = _). Setting tags leaves the blob's ETag and Last-Modified unchanged - Find Blobs by Tags:
GET ?comp=blobs&where=...across the account andGET /{container}?restype=container&comp=blobs&where=...within one container, with=,>,>=,<,<=,AND, parentheses and@container, andmaxresults/markerpaging. Each result lists the blob's values for the tags the expression names. Snapshots are not indexed - Set Blob Properties:
PUT ?comp=propertiesreplaces the blob's HTTP headers (x-ms-blob-content-type,x-ms-blob-cache-control, and the otherx-ms-blob-content-*headers) x-ms-if-tagsconditions: honored on Get Blob, Get Blob Properties, Put Blob, Append Block, Delete Blob, Get/Set Blob Metadata, Snapshot Blob and Get/Set Blob Tags (which also takex-ms-blob-if-matchandx-ms-blob-if-none-match); the expression also acceptsORand<>- Copy Blob and Copy Blob From URL:
PUTwithx-ms-copy-source(andx-ms-requires-sync: truefor Copy Blob From URL) copies a blob, or a snapshot of one with?snapshot=, from any container or account in the emulator. The source may be a path-style URL, a host-style URL ({account}.blob.{host}) or a bare/{account}/{container}/{blob}path. Metadata is the source's, or thex-ms-meta-*headers when the request sends any; destination conditions and leases are enforced, and an existing destination must be of the source's blob type (409 InvalidBlobType). A new destination needs create permission and an existing one write permission, as for Put Blob. Tags are the request'sx-ms-tags, never the source's, except that Copy Blob From URL withx-ms-copy-source-tag-option: COPYkeeps the source's. A SAS on the source URL is checked as a read of the source, as Azurite checks it, and a failed check isCannotVerifyCopySourcewith the status the read got. Thex-ms-source-if-match,-if-none-match,-if-modified-sinceand-if-unmodified-sinceconditions are checked against the source (412 SourceConditionNotMet). Both answer202withx-ms-copy-status: success - Put Blob From URL and Put Block From URL:
PUTwithx-ms-copy-sourceandx-ms-blob-type: BlockBlobwrites a block blob holding the source's bytes, and?comp=blockwithx-ms-copy-sourcestages the source, or itsx-ms-source-range, as a block. Sources are found, SAS-checked and conditioned as for Copy Blob; a missing source is404 CannotVerifyCopySource, a range starting past the source's end is416 CannotVerifyCopySource, a malformed range or a request body is400 InvalidHeaderValue, and anx-ms-source-content-md5that does not match is400 Md5Mismatch. A source written with a customer-provided key is read only with that key inx-ms-source-encryption-key(and its-sha256); a missing or different key is409 CannotVerifyCopySource. Put Blob From URL keeps Put Blob's destination rules (conditions, leases,If-None-Match: *); its content properties are the source's unlessx-ms-copy-source-blob-properties: false, anx-ms-blob-*header sets its own property either way, request metadata replaces the source's, and tags are the request's or, withx-ms-copy-source-tag-option: COPY, the source's (400when both are sent). Both answer201with the copied bytes'Content-MD5 - ARM containers and queues:
.../blobServices/default/containers/{name}and.../queueServices/default/queues/{name}answer from the data plane's own containers and queues, so a container or queue made on either plane is read on the other, and an ARM delete removes it with its blobs or messages. A missing one is404(ContainerNotFound/QueueNotFound, the Storage service's codes; Azure's ARM spec records no code for it). Statuses follow the spec: a container delete is200, or204when there was none; a queue create is200and a queue delete204 - ARM management policy and blob service properties:
.../managementPolicies/defaultkeeps the lifecyclepolicyas sent, with a generatedlastModifiedTime; itsPUTandGETanswer200, aGETbefore one is set answers404, and aDELETEanswers200, or204when there was none..../blobServices/defaultkeeps thepropertiesof its lastPUT(versioning, delete retention, CORS and the rest), which replace the ones set before; until one is set itsGETanswers emptyproperties. Both go with the account when it is deleted. The rules are stored, not run: no blob is tiered or deleted by a policy - Not-found semantics: missing blob/container returns the Azure
404 BlobNotFound/ContainerNotFoundXML error shape
Endpoint
http://localhost:4577/{account}/{container} # container operations
http://localhost:4577/{account}/{container}/{blob} # blob operations
http://localhost:4577/{account}/{filesystem}?resource=filesystem&recursive=true # ADLS listPaths
The account also answers at the host-style address {account}.blob.core.windows.net (and the Data
Lake Gen2 alias {account}.dfs.core.windows.net, which maps to the same blob backend) when the
Host header is set, matching how the SDKs address storage endpoints.
ARM storage account responses include both blob and dfs primary endpoints so Data Lake SDK
clients can discover the Gen2 endpoint shape.
ARM storage accounts are kept in the arm-storage store, which follows the global storage mode, so under
persistent or hybrid an account survives a restart with its data, its tags and its isHnsEnabled
setting. ARM ids are case-insensitive: a PUT that differs from the account's id only in case updates that
account, and responses keep the casing of its first PUT. An account name is owned by one account across every
subscription, resource group and cloud until that account is deleted; checkNameAvailability answers from that
ownership.
Quickstart
from azure.storage.blob import BlobServiceClient
conn = ("DefaultEndpointsProtocol=http;AccountName=devstoreaccount1;"
"AccountKey=Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==;"
"BlobEndpoint=http://localhost:4577/devstoreaccount1;")
svc = BlobServiceClient.from_connection_string(conn)
container = svc.create_container("my-container")
container.upload_blob("hello.txt", b"hello world")
print(container.download_blob("hello.txt").readall())
The well-known Azurite development key (Eby8vdM0…) is accepted; any account name works.
Configuration
floci-az:
services:
blob:
enabled: true
hierarchical-namespace-accounts: devstoreaccount1
storage:
services:
blob:
# mode: wal # override the global storage.mode for blob only
flush-interval-ms: 5000
| Property | Env var | Default | Description |
|---|---|---|---|
enabled |
FLOCI_AZ_SERVICES_BLOB_ENABLED |
true |
Enables the Blob Storage service |
hierarchical-namespace-accounts |
FLOCI_AZ_SERVICES_BLOB_HIERARCHICAL_NAMESPACE_ACCOUNTS |
devstoreaccount1 |
Comma-separated account names with hierarchical namespace enabled |
storage.services.blob.mode |
FLOCI_AZ_STORAGE_SERVICES_BLOB_MODE |
(inherits storage.mode) |
Per-service backend override (memory / persistent / hybrid / wal) |
storage.services.blob.flush-interval-ms |
FLOCI_AZ_STORAGE_SERVICES_BLOB_FLUSH_INTERVAL_MS |
5000 |
Background flush-to-disk interval for the hybrid mode only; ignored by memory / persistent / wal (wal compacts on storage.wal.compaction-interval-ms instead) |
Intentional deviations
- Shared Key signatures are accepted but not cryptographically verified: the emulator is a
local dev target; any well-formed
Authorizationheader (or the Azurite key) is honored. - ADLS large-operation behavior is simplified - directory rename and recursive delete complete atomically in the local backend rather than reproducing Azure's server-side multi-request batching. Hadoop observes a completed operation with no continuation token, which is protocol-compatible for the caller. The full Azure source/destination lease-transfer and time-condition matrix is not modeled.
- ADLS access control is metadata-compatible, not a full POSIX authorization engine - owner,
group, permissions and ACLs round-trip through the Hadoop/DFS wire protocol, and
checkAccessvalidates path existence and request shape. It does not deny requests based on emulated POSIX ACLs; authentication/SAS authorization remains the emulator's access-control boundary. - ADLS close/event semantics are storage-only -
close=trueis accepted and committed data is immediately visible, but Azure Event Grid/change-notification side effects are not emulated. - SAS enforcement supports shared-key service SAS and user delegation SAS. Service SAS
signatures use
floci-az.auth.storage-account-keys, a map from account names to base64 keys. Accounts without an explicit entry use the standard Azurite key, including ARM-created accounts. Explicit entries take precedence. ARMlistKeyson a storage account returns the same key (configured or default) askey1andkey2, so a SAS signed with a listed key is accepted. Invalid signatures are rejected, including in dev mode, as are expired tokens and operations outside the granted permissions. This is separate from the permissive Shared KeyAuthorizationheader behavior above. Shared-key service SAS supports directory scope (sr=d). Directory service SAS requiressddand version 2020-02-10 or later. SDK-generated user delegation SAS tokens for container (sr=c), blob (sr=b), Blob snapshot (sr=bs), and ADLS directory (sr=d) resources are validated. Account SAS, stored access policies, IP/protocol restrictions, and the full SAS feature matrix are not fully modeled. User delegation keys are protected by a process-local secret, so SAS tokens issued by a previous emulator process are invalid after restart. - Copies complete at once and stay inside the emulator. Copy Blob reports
successin its202, so there is no pending state to poll and Abort Copy Blob is not served. A source outside the emulator,x-ms-source-if-tags,x-ms-copy-source-authorization,x-ms-content-crc64, and Get Blob Properties'x-ms-copy-*fields are not modeled. A source without a SAS is read like any other request, since the emulator enforces only SAS tokens. Append Block From URL (?comp=appendblockwithx-ms-copy-source) answers501, as does a copy source with anyx-ms-blob-typeother thanBlockBlob. - Blob snapshots are modeled. They capture immutable blob state and support creation, reads,
and deletion. Listing snapshots with
include=snapshotson List Blobs is not modeled. Deleting a base blob with snapshots requiresx-ms-delete-snapshots: includeoronly; mutations other than deletion directed at a snapshot returnSnapshotsPresent. Blob versioning and tiering are not modeled. - Blob leases and ADLS Path leases share the emulator's in-memory lease state and support acquire/renew/change/release/break. Lease state is intentionally process-local and is lost when the emulator restarts.
x-ms-server-encrypted: trueis reported although no encryption is performed: blob data is stored as-is by the configured storage backend. The header mirrors thex-ms-request-server-encryptedalready returned on upload and exists for SDK compatibility.