Google Cloud Datastore

Google Cloud Datastore
Developer	Google Inc.
Release	May 2013 (2013-05)
Operating system	Cross-platform
Platform	Google Cloud Platform
Type	Document-oriented database
License	Proprietary
Website	cloud.google.com/datastore/

Google Cloud Datastore was a NoSQL database service provided by Google Cloud Platform. It is now superseded by Firestore which offers a "Datastore compatibility" mode to ensure legacy applications built on Datastore continue operating seamlessly. Google Cloud Datastore API organizes data in entities and properties, where entities are grouped into kinds. This concept is similar to tables in relational databases, however since this is NoSQL database, it is without the schema constraints. Each entity in Datastore is uniquely identified by a key. This key can have a custom user-defined identifier or it can be auto generated key by the system.

History

Google Cloud Datastore was announced on April 11, 2013, as a fully managed NoSQL document database and designed to support large-scale web applications. It was based on the original Datastore used in Google App Engine since 2008. But it was designed to offer feature such as scalability, higher availability, and automatic data replication across multiple data centers.

Before the launch of Cloud Datastore, developers on Google App Engine used to work on a built-in Datastore that only worked with App Engine apps. When Google Cloud Platform started to grow in the market, developers wanted a database which they could use outside of App Engine to integrate with their apps. They needed more flexibility and wide availability. Cloud Datastore met this need by adding features like automatic sharding, indexing, and support for eventual consistency.

Google launched Cloud Firestore in 2018 as new NoSQL database with features such as real-time updates, offline support, and faster query execution. New users are encouraged to use Firestore instead. Firestore offers a "Datastore compatibility" mode such that existing users can continue using the familiar Datastore API and features with an option to upgrade to Firestore when needed to benefit from enterprise-capabilities, strong-consistency, ACID-compliant multi-document transactions and advanced query engine that supports complex filters, aggregations, joins, subqueries. Google Cloud continues to support these legacy systems so that they remain reliable and fully functional even in the future.

Overview

Data organization

Data is organized into entities in Google Cloud Datastore. These are like individual records. These entities are grouped into kinds. This is just like tables in a traditional database. However, unlike relational databases, entities in the same kind do not have to follow a fixed structure (like a pre-defined schema). They can have different sets of properties.

Entities and properties

Each entity represents a structured set of properties. Properties are key-value pairs. Examples of values can be strings, numbers, Booleans, timestamps, arrays, and geographic points. The flexible nature of properties allows developers to model complex data structures without a rigid schema.

Entity keys

Every entity in Datastore is uniquely identified by a key. A key includes:

• A project ID (the identifier of the Google Cloud project),
• An optional namespace (used for multi-tenancy),
• A kind (defining the entity type),
• And a name or numeric ID that distinguishes the entity within its kind.

Entities can optionally be grouped into entity groups to allow for transactional updates involving multiple entities.

Data types

Datastore API supports a range of property data types as shown below:

• String
• Integer
• Float
• Boolean
• Timestamp
• Array (List)
• Embedded Entity
• GeoPoint (geographical coordinates)
• Binary data (Blob)

This variety allows modeling of structured and semi-structured data into various datatypes.

GQL

GQL (Google Cloud Datastore Query Language) is a query language just like SQL and it is designed to interact with Google Cloud Datastore. GQL allows users to query the Datastore service using a statements just like SQL, however specifically designed to the NoSQL nature of this platform. GQL provides ways to filter, order, and perform operations on Datastore entities without needing to write complex queries in the underlying datastore APIs.

Unlike SQL, GQL is limited in terms of the types of joins and relationships it can handle. However, it supports querying by properties, including equality and inequality, as well as range queries. Users can use GQL to query entities based on multiple conditions. This makes GQL suitable for a wide range of use cases such as retrieving user data, product catalogs, and even updating the database.

GQL also has support for ancestor queries. This lets users to get related entities based on their place in a hierarchy. This is very much needed for applications where we need to manage hierarchical data like a content management systems or data models that have parent-child relationships. Even though GQL can help in simplifying querying, it operates within the constraints of Datastore's eventual consistency model.

Example GQL Query:

SELECT * FROM Task WHERE status = 'completed' AND priority = 'high' ORDER BY created DESC

This query will get all Task entities with a status of "completed" and a priority of "high". It will also order them by the created timestamp with descending order.

Even though GQL has an easy-to-use interface for querying Google Cloud Datastore, when dealing with more complex queries and joins, we need to use Datastore's native APIs such as the Google Cloud Datastore API. This API has greater flexibility and control for developers. For example, we cannot do below join operation:

SELECT * FROM Customer JOIN `Order` ON Customer.customer_id = `Order`.customer_id

We can implement a logic like below code. It is basically a two step process. First, use the Datastore API to fetch the customer by ID. Then, use that customer ID to create another query to retrieve related orders.

from google.cloud import datastore

client = datastore.Client()

# step 1
customerkey = client.key("Customer", "C123456")
customer = client.get(customerkey)

# step 2
query = client.query(kind="Order")
query.add_filter("customer_id", "=", "C123456")
orders = list(query.fetch())

Best practices for developers

Query limitations

Datastore does not support joins, subqueries, or aggregation operations like those found in relational databases, such as MS SQL and MySQL. Because of this, application design often requires denormalization. It is a process of storing related data together within a single entity or using entity groups to maintain hierarchical relationships. Query filters are required to match existing indexes, and certain combinations of inequality and sort operations may require custom indexes. Datastore's successor, Firestore supports an advanced query pipeline API which supports aggregations, joins and subqueries.

Language and API support

Google Cloud Datastore APIs consist of a RESTful interface and a gRPC API. This is quite useful for developers who need to design distributed applications. These APIs offer direct access to Datastore's features such as transactions, queries, and entity management. The client libraries are built on top of these APIs and hide complex details. They make it easier to use different languages such as Python, Java, Go, Node.js, and C# by handling things like connection handling, retries, and serialization. Datastore's API is also optimized for giving high-throughput and low-latency access. This API also supports batch operations, ancestor queries, and strongly consistent reads within entity groups. With the help of such features, developers can build scalable applications without the need of handling complex database infrastructure on their own.

Data modeling trade-offs

Google Cloud Datastore is designed to scale easily and simplify operations, but this comes with some unavoidable trade-offs. Especially when it comes to features like joins and transactions involving multiple rows. To achieve high scalability, Datastore requires developers to denormalize their data and design their schema around read patterns, rather than focusing on normalization. This approach helps reduce complexity and enhances performance, especially when developers are working on read-heavy applications. Hence, at the time of designing database, developers must plan their data models with the assumption that Cloud Datastore won't offer join operations. For even more complex operations such as aggregations or full-text search, developers often need to integrate Datastore with other services like GCP BigQuery for analytics or Elasticsearch for search capabilities. Which means, developers need to design a system that connects multiple tools to achieve their goals. However, new applications can use Firestore which offers built-in full-text search capabilities.

Comparison with other databases

Cloud Datastore vs. MongoDB

Cloud Datastore and MongoDB are both NoSQL document databases, but still they differ in terms of their structure and capabilities. Cloud Datastore enforces a more structured document model with entity groups and prioritizes scalability and simplicity over rich querying features. Whereas MongoDB gives a flexible document model (BSON), which is a powerful aggregation pipeline, and has limited join capabilities. Cloud Datastore is tightly combined with Google Cloud Platform (GCP), but MongoDB can be self-hosted or used as a managed service via MongoDB Atlas on different cloud platforms. Google Cloud Datastore's successor, Firestore offers MongoDB wire-compatible API to enable lift-shift migrations.

Cloud Datastore vs. relational databases

Relational databases like MySQL and PostgreSQL follow a normalized schema with support for complex joins and full ACID transactions. These are ideal for making applications requiring strong consistency and relational logic. In contrast, Cloud Datastore follows a denormalized and schema-less NoSQL approach, where data is structured around access patterns to optimize read performance at scale. Traditional RDBMS are better for transactional systems and complex queries and Cloud Datastore is more effective for large-scale applications that need high availability and horizontal scalability. For applications requiring complex query capabilities, Firestore APIs are strongly recommended.

Cloud Datastore vs. Redis

Redis is also a NoSQL database with in-memory key-value store designed for extremely quick access. It is an ideal choice for caching, pub/sub systems, and real-time analytics. It prioritizes speed over durability, but persistence options are available as well. Cloud Datastore is a persistent and disk-based document database that has structured queries. It is also optimized for long-term data storage and retrieval. Redis is great for fast and transient operations, but Cloud Datastore is better suited for structured and long-lived application data that need managed scaling and indexing. Redis is often used in combination with a database like Cloud Datastore not as a replacement.

See also

• Firestore
• Azure Cosmos DB
• Amazon DynamoDB
• Oracle Cloud NoSQL DB

External links

• Official website
• Firestore website
• Google Cloud Platform site