Skip to content

Blob Storage

Compatible with the azure-storage-blob SDKs (Java, Python, Node.js), focused Java azure-storage-file-datalake SDK flows, the Azure CLI (az storage blob), and Azurite-style connection strings. Speaks the Azure Storage Blob REST protocol with Shared Key authentication, Blob XML responses, and the Data Lake Storage Gen2 DFS host alias.

HTTP-only: no Docker. Data is held by the configured storage backend (memory by default; persistent, hybrid, or wal for durability).


Features

  • Containers: Create, Get properties, Delete, List (?comp=list); duplicate create returns 409 ContainerAlreadyExists
  • Blobs: Put (upload), Get (download), Delete, List within a container; overwrite semantics
  • Block blobs: staged block upload (?comp=block) followed by commit (?comp=blocklist) for large payloads, in addition to single-request Put Blob
  • Data Lake Storage Gen2 endpoint alias: the {account}.dfs.core.windows.net host maps to the Blob backend so ADLS SDK path clients can create, read, write, and delete paths through the same local data store
  • User delegation key vending: POST ?restype=service&comp=userdelegationkey returns Azure-shaped XML for SDK-generated user delegation SAS flows
  • User delegation SAS enforcement: validates SDK-generated user delegation SAS signatures, expiry, signed key validity, permissions, and container/blob/directory resource scope for Blob and ADLS path operations
  • ADLS Gen2 / Hadoop ABFS 3.3.4 path operations - supports the DFS wire shapes used by org.apache.hadoop:hadoop-azure:3.3.4: conditional file/directory create, path status and properties, append/flush, recursive delete, file/directory rename, POSIX owner/group/permission/ACL metadata, checkAccess, and the ADLS Path Lease POST API. Hadoop's default conditional-create overwrite flow (409 -> status/ETag -> If-Match) is modeled explicitly. Access-control operations require an HNS-enabled account. Accounts listed in hierarchical-namespace-accounts have HNS enabled by default; ARM-created accounts can also enable it with properties.isHnsEnabled: true. Other custom account names use a flat namespace by default. As in Azure, HNS is fixed when the account is created: a later PUT that sends a different isHnsEnabled value returns 400 AccountPropertyCannotBeUpdated, while a PUT that omits the property keeps the current setting.
  • ADLS filesystem operations - create/delete a filesystem and get/set filesystem properties via ?resource=filesystem. Root getAccessControl is supported so Hadoop can auto-detect HNS and resolve getFileStatus("/") without forcing fs.azure.account.hns.enabled=true.
  • ADLS Path - List - supports Java DataLake SDK listPaths and Hadoop listStatus, including recursive/non-recursive listing, directory filters, exact-file singleton results, server pagination, and Hadoop 3.3.4's Base64 HNS startFrom continuation token.
  • ADLS AppendBlob mode - recognizes Hadoop's blobType=AppendBlob create mode and enforces sequential append positions, including append requests carrying flush=true / close=true.
  • Range download: Range: bytes=… returns 206 Partial Content
  • Conditional download: If-Match / If-None-Match honored; a stale ETag is rejected
  • ETags: sent in the quoted 0x hex form the Blob service uses ("0x8DC1A2B3C4D5E6F"). List Blobs reports each blob's ETag without the quotes, as Azurite does. A container keeps the ETag and Last-Modified from Create Container, so Get Container Properties and List Containers repeat them
  • Metadata: x-ms-meta-* set on upload and returned on Get, round-tripped exactly
  • Blob index tags: Get and Set Blob Tags (?comp=tags, including on a snapshot), x-ms-tags on Put Blob and Put Block List, and x-ms-tag-count on Get Blob and Get Blob Properties. Tag sets are validated (at most 10 tags, keys of 1 to 128 characters, values of up to 256, letters, digits, space and + - . / : = _). Setting tags leaves the blob's ETag and Last-Modified unchanged
  • Find Blobs by Tags: GET ?comp=blobs&where=... across the account and GET /{container}?restype=container&comp=blobs&where=... within one container, with =, >, >=, <, <=, AND, parentheses and @container, and maxresults/marker paging. Each result lists the blob's values for the tags the expression names. Snapshots are not indexed
  • Set Blob Properties: PUT ?comp=properties replaces the blob's HTTP headers (x-ms-blob-content-type, x-ms-blob-cache-control, and the other x-ms-blob-content-* headers)
  • x-ms-if-tags conditions: honored on Get Blob, Get Blob Properties, Put Blob, Append Block, Delete Blob, Get/Set Blob Metadata, Snapshot Blob and Get/Set Blob Tags (which also take x-ms-blob-if-match and x-ms-blob-if-none-match); the expression also accepts OR and <>
  • Copy Blob and Copy Blob From URL: PUT with x-ms-copy-source (and x-ms-requires-sync: true for Copy Blob From URL) copies a blob, or a snapshot of one with ?snapshot=, from any container or account in the emulator. The source may be a path-style URL, a host-style URL ({account}.blob.{host}) or a bare /{account}/{container}/{blob} path. Metadata is the source's, or the x-ms-meta-* headers when the request sends any; destination conditions and leases are enforced, and an existing destination must be of the source's blob type (409 InvalidBlobType). A new destination needs create permission and an existing one write permission, as for Put Blob. Tags are the request's x-ms-tags, never the source's, except that Copy Blob From URL with x-ms-copy-source-tag-option: COPY keeps the source's. A SAS on the source URL is checked as a read of the source, as Azurite checks it, and a failed check is CannotVerifyCopySource with the status the read got. The x-ms-source-if-match, -if-none-match, -if-modified-since and -if-unmodified-since conditions are checked against the source (412 SourceConditionNotMet). Both answer 202 with x-ms-copy-status: success
  • Put Blob From URL and Put Block From URL: PUT with x-ms-copy-source and x-ms-blob-type: BlockBlob writes a block blob holding the source's bytes, and ?comp=block with x-ms-copy-source stages the source, or its x-ms-source-range, as a block. Sources are found, SAS-checked and conditioned as for Copy Blob; a missing source is 404 CannotVerifyCopySource, a range starting past the source's end is 416 CannotVerifyCopySource, a malformed range or a request body is 400 InvalidHeaderValue, and an x-ms-source-content-md5 that does not match is 400 Md5Mismatch. A source written with a customer-provided key is read only with that key in x-ms-source-encryption-key (and its -sha256); a missing or different key is 409 CannotVerifyCopySource. Put Blob From URL keeps Put Blob's destination rules (conditions, leases, If-None-Match: *); its content properties are the source's unless x-ms-copy-source-blob-properties: false, an x-ms-blob-* header sets its own property either way, request metadata replaces the source's, and tags are the request's or, with x-ms-copy-source-tag-option: COPY, the source's (400 when both are sent). Both answer 201 with the copied bytes' Content-MD5
  • ARM containers and queues: .../blobServices/default/containers/{name} and .../queueServices/default/queues/{name} answer from the data plane's own containers and queues, so a container or queue made on either plane is read on the other, and an ARM delete removes it with its blobs or messages. A missing one is 404 (ContainerNotFound / QueueNotFound, the Storage service's codes; Azure's ARM spec records no code for it). Statuses follow the spec: a container delete is 200, or 204 when there was none; a queue create is 200 and a queue delete 204
  • ARM management policy and blob service properties: .../managementPolicies/default keeps the lifecycle policy as sent, with a generated lastModifiedTime; its PUT and GET answer 200, a GET before one is set answers 404, and a DELETE answers 200, or 204 when there was none. .../blobServices/default keeps the properties of its last PUT (versioning, delete retention, CORS and the rest), which replace the ones set before; until one is set its GET answers empty properties. Both go with the account when it is deleted. The rules are stored, not run: no blob is tiered or deleted by a policy
  • Not-found semantics: missing blob/container returns the Azure 404 BlobNotFound / ContainerNotFound XML error shape

Endpoint

http://localhost:4577/{account}/{container}                  # container operations
http://localhost:4577/{account}/{container}/{blob}           # blob operations
http://localhost:4577/{account}/{filesystem}?resource=filesystem&recursive=true  # ADLS listPaths

The account also answers at the host-style address {account}.blob.core.windows.net (and the Data Lake Gen2 alias {account}.dfs.core.windows.net, which maps to the same blob backend) when the Host header is set, matching how the SDKs address storage endpoints.

ARM storage account responses include both blob and dfs primary endpoints so Data Lake SDK clients can discover the Gen2 endpoint shape.

ARM storage accounts are kept in the arm-storage store, which follows the global storage mode, so under persistent or hybrid an account survives a restart with its data, its tags and its isHnsEnabled setting. ARM ids are case-insensitive: a PUT that differs from the account's id only in case updates that account, and responses keep the casing of its first PUT. An account name is owned by one account across every subscription, resource group and cloud until that account is deleted; checkNameAvailability answers from that ownership.

Quickstart

from azure.storage.blob import BlobServiceClient

conn = ("DefaultEndpointsProtocol=http;AccountName=devstoreaccount1;"
        "AccountKey=Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==;"
        "BlobEndpoint=http://localhost:4577/devstoreaccount1;")
svc = BlobServiceClient.from_connection_string(conn)
container = svc.create_container("my-container")
container.upload_blob("hello.txt", b"hello world")
print(container.download_blob("hello.txt").readall())
az storage container create --name my-container \
  --connection-string "DefaultEndpointsProtocol=http;AccountName=devstoreaccount1;AccountKey=...;BlobEndpoint=http://localhost:4577/devstoreaccount1;"

The well-known Azurite development key (Eby8vdM0…) is accepted; any account name works.

Configuration

floci-az:
  services:
    blob:
      enabled: true
      hierarchical-namespace-accounts: devstoreaccount1
  storage:
    services:
      blob:
        # mode: wal            # override the global storage.mode for blob only
        flush-interval-ms: 5000
Property Env var Default Description
enabled FLOCI_AZ_SERVICES_BLOB_ENABLED true Enables the Blob Storage service
hierarchical-namespace-accounts FLOCI_AZ_SERVICES_BLOB_HIERARCHICAL_NAMESPACE_ACCOUNTS devstoreaccount1 Comma-separated account names with hierarchical namespace enabled
storage.services.blob.mode FLOCI_AZ_STORAGE_SERVICES_BLOB_MODE (inherits storage.mode) Per-service backend override (memory / persistent / hybrid / wal)
storage.services.blob.flush-interval-ms FLOCI_AZ_STORAGE_SERVICES_BLOB_FLUSH_INTERVAL_MS 5000 Background flush-to-disk interval for the hybrid mode only; ignored by memory / persistent / wal (wal compacts on storage.wal.compaction-interval-ms instead)

Intentional deviations

  • Shared Key signatures are accepted but not cryptographically verified: the emulator is a local dev target; any well-formed Authorization header (or the Azurite key) is honored.
  • ADLS large-operation behavior is simplified - directory rename and recursive delete complete atomically in the local backend rather than reproducing Azure's server-side multi-request batching. Hadoop observes a completed operation with no continuation token, which is protocol-compatible for the caller. The full Azure source/destination lease-transfer and time-condition matrix is not modeled.
  • ADLS access control is metadata-compatible, not a full POSIX authorization engine - owner, group, permissions and ACLs round-trip through the Hadoop/DFS wire protocol, and checkAccess validates path existence and request shape. It does not deny requests based on emulated POSIX ACLs; authentication/SAS authorization remains the emulator's access-control boundary.
  • ADLS close/event semantics are storage-only - close=true is accepted and committed data is immediately visible, but Azure Event Grid/change-notification side effects are not emulated.
  • SAS enforcement supports shared-key service SAS and user delegation SAS. Service SAS signatures use floci-az.auth.storage-account-keys, a map from account names to base64 keys. Accounts without an explicit entry use the standard Azurite key, including ARM-created accounts. Explicit entries take precedence. ARM listKeys on a storage account returns the same key (configured or default) as key1 and key2, so a SAS signed with a listed key is accepted. Invalid signatures are rejected, including in dev mode, as are expired tokens and operations outside the granted permissions. This is separate from the permissive Shared Key Authorization header behavior above. Shared-key service SAS supports directory scope (sr=d). Directory service SAS requires sdd and version 2020-02-10 or later. SDK-generated user delegation SAS tokens for container (sr=c), blob (sr=b), Blob snapshot (sr=bs), and ADLS directory (sr=d) resources are validated. Account SAS, stored access policies, IP/protocol restrictions, and the full SAS feature matrix are not fully modeled. User delegation keys are protected by a process-local secret, so SAS tokens issued by a previous emulator process are invalid after restart.
  • Copies complete at once and stay inside the emulator. Copy Blob reports success in its 202, so there is no pending state to poll and Abort Copy Blob is not served. A source outside the emulator, x-ms-source-if-tags, x-ms-copy-source-authorization, x-ms-content-crc64, and Get Blob Properties' x-ms-copy-* fields are not modeled. A source without a SAS is read like any other request, since the emulator enforces only SAS tokens. Append Block From URL (?comp=appendblock with x-ms-copy-source) answers 501, as does a copy source with any x-ms-blob-type other than BlockBlob.
  • Blob snapshots are modeled. They capture immutable blob state and support creation, reads, and deletion. Listing snapshots with include=snapshots on List Blobs is not modeled. Deleting a base blob with snapshots requires x-ms-delete-snapshots: include or only; mutations other than deletion directed at a snapshot return SnapshotsPresent. Blob versioning and tiering are not modeled.
  • Blob leases and ADLS Path leases share the emulator's in-memory lease state and support acquire/renew/change/release/break. Lease state is intentionally process-local and is lost when the emulator restarts.
  • x-ms-server-encrypted: true is reported although no encryption is performed: blob data is stored as-is by the configured storage backend. The header mirrors the x-ms-request-server-encrypted already returned on upload and exists for SDK compatibility.