Skip to content

Datastore

floci-gcp emulates Google Cloud Datastore over gRPC and binary HTTP/protobuf using the real google.datastore.v1 protocol. Current Google Cloud Datastore SDK clients use the HTTP/protobuf path.

Configuration

Variable Default Description
FLOCI_GCP_SERVICES_DATASTORE_ENABLED true Enable/disable Datastore

Emulator Variable

export DATASTORE_EMULATOR_HOST=localhost:4588

The GCP Datastore SDK uses this variable to route requests to floci-gcp instead of datastore.googleapis.com.

Quick Start

DatastoreOptions options = DatastoreOptions.newBuilder()
    .setHost("http://localhost:4588")
    .setProjectId("floci-local")
    .setCredentials(NoCredentials.getInstance())
    .build();

Datastore datastore = options.getService();

// Create entity
KeyFactory keyFactory = datastore.newKeyFactory().setKind("Task");
Key taskKey = datastore.allocateId(keyFactory.newKey());

Entity task = Entity.newBuilder(taskKey)
    .set("description", "Buy groceries")
    .set("done", false)
    .set("priority", 4L)
    .build();

datastore.put(task);

// Query entities
Query<Entity> query = Query.newEntityQueryBuilder()
    .setKind("Task")
    .setFilter(PropertyFilter.eq("done", false))
    .build();

QueryResults<Entity> results = datastore.run(query);
while (results.hasNext()) {
    Entity e = results.next();
    System.out.println(e.getString("description"));
}
import os
os.environ["DATASTORE_EMULATOR_HOST"] = "localhost:4588"

from google.cloud import datastore

client = datastore.Client(project="floci-local")

# Create entity
key = client.key("Task")
task = datastore.Entity(key=key)
task.update({
    "description": "Buy groceries",
    "done": False,
    "priority": 4,
})
client.put(task)

# Query entities
query = client.query(kind="Task")
query.add_filter("done", "=", False)
results = list(query.fetch())
for task in results:
    print(task["description"])
process.env.DATASTORE_EMULATOR_HOST = "localhost:4588";

import { Datastore } from "@google-cloud/datastore";

const datastore = new Datastore({ projectId: "floci-local" });

// Create entity
const key = datastore.key("Task");
const data = [
    { name: "description", value: "Buy groceries" },
    { name: "done", value: false },
    { name: "priority", value: 4 },
];

await datastore.save({ key, data });

// Query entities
const query = datastore.createQuery("Task").filter("done", "=", false);
const [entities] = await datastore.runQuery(query);
entities.forEach(e => console.log(e.description));
export DATASTORE_EMULATOR_HOST=localhost:4588
gcloud config set project floci-local

# Use gcloud datastore operations
gcloud datastore operations list

Transactions

Transaction RPCs are available over both transports, but transaction IDs are currently advisory. Commits apply their mutations without isolation or conflict detection, and rollback is a no-op.

TransactionCallable<Void> callable = transaction -> {
    Key taskKey = datastore.newKeyFactory().setKind("Task").newKey(1);
    Entity task = transaction.get(taskKey);

    if (task != null) {
        Entity updated = Entity.newBuilder(task)
            .set("done", true)
            .build();
        transaction.put(updated);
    }
    return null;
};

datastore.runInTransaction(callable);

Indexes

floci-gcp does not load datastore.indexes.yaml or enforce Datastore index requirements. Structured and GQL queries scan the emulator's stored entities, so a query that requires a composite index in GCP may still run locally.

GQL Queries

Datastore supports GQL (Google Query Language) in RunQuery over both gRPC and HTTP/protobuf. GQL aggregation queries (AGGREGATE ... OVER (...)) are not supported and return UNIMPLEMENTED.

Query<Entity> query = Query.newGqlQueryBuilder(Query.ResultType.ENTITY,
    "SELECT * FROM Task WHERE done = @done LIMIT @limit")
    .setBinding("done", false)
    .setBinding("limit", 10)
    .build();

QueryResults<Entity> results = datastore.run(query);

Supported GQL syntax: - SELECT * FROM Kind - WHERE prop = value (and !=, <, <=, >, >=) - WHERE cond1 AND cond2 - LIMIT n / OFFSET n - Named bindings (@name) and positional bindings (@1)

Deviations from real Datastore

  • With persistence enabled, a key property written by floci-gcp 0.11.0 or earlier was stored only as a flattened path, which cannot be rebuilt into a key. It reads back as NULL until the entity is written again, although equality filters still match it. An embedded entity written then reads back without its own key, since that key was not stored.
  • RunAggregationQuery does not reject duplicate aliases, zero aggregations, or more than five.
  • Value.exclude_from_indexes is stored and read back, but it does not change queries: a filter or sort on an excluded property still matches the entity, where real Datastore leaves it out of the result.

Supported Operations

  • Lookup (gRPC and HTTP/protobuf)
  • RunQuery (structured and GQL queries over both transports)
  • RunAggregationQuery (COUNT with up_to, SUM and AVG over both transports)
  • BeginTransaction (gRPC and HTTP/protobuf; limited semantics described above)
  • Commit (gRPC and HTTP/protobuf; limited semantics described above)
  • Rollback (gRPC and HTTP/protobuf; no-op)
  • AllocateIds (gRPC and HTTP/protobuf)
  • ReserveIds (gRPC and HTTP/protobuf)