From 9cf3a222e91f5d1a55dffa90e9eeec7d11fa2467 Mon Sep 17 00:00:00 2001 From: "Matthew D. Groves" Date: Thu, 31 Jul 2025 10:33:07 -0400 Subject: [PATCH 1/4] JSON Data Modeling Guide: updates, improvements, tweaks, fixes, corrections --- ...aring-document-oriented-relational-data.md | 2 +- .../02-using-json-documents.md | 14 ++- .../03-schemaless-data-modeling.md | 16 +-- .../04-phases-of-data-modeling.md | 22 ++-- .../05-json-design-choices.md | 118 +++--------------- .../06-document-key-design.md | 38 +++--- ...ndardization-optimization-and-resources.md | 33 ++--- 7 files changed, 75 insertions(+), 168 deletions(-) diff --git a/tutorial/markdown/couchbase-server/best-practices/json-data-modeling-guide/01-comparing-document-oriented-relational-data.md b/tutorial/markdown/couchbase-server/best-practices/json-data-modeling-guide/01-comparing-document-oriented-relational-data.md index 680a835..cc29625 100644 --- a/tutorial/markdown/couchbase-server/best-practices/json-data-modeling-guide/01-comparing-document-oriented-relational-data.md +++ b/tutorial/markdown/couchbase-server/best-practices/json-data-modeling-guide/01-comparing-document-oriented-relational-data.md @@ -8,7 +8,7 @@ description: - Learn the different characteristics of each and explore the flexibility of Couchbase - See an example dataset get mapped from relational to document-based content_type: tutorial -filter: n1ql +filter: sql++ technology: - kv - capella diff --git a/tutorial/markdown/couchbase-server/best-practices/json-data-modeling-guide/02-using-json-documents.md b/tutorial/markdown/couchbase-server/best-practices/json-data-modeling-guide/02-using-json-documents.md index cc0fb1c..636d0ee 100644 --- a/tutorial/markdown/couchbase-server/best-practices/json-data-modeling-guide/02-using-json-documents.md +++ b/tutorial/markdown/couchbase-server/best-practices/json-data-modeling-guide/02-using-json-documents.md @@ -7,7 +7,7 @@ description: - A brief introduction to the JSON data format that illustrates the readability and ease of use - Learn about the basic data types supported in JSON and view a few example documents content_type: tutorial -filter: n1ql +filter: sql++ technology: - kv - capella @@ -31,7 +31,9 @@ The following are basic data types supported in JSON: For more information about creating valid JSON documents, please refer to [http://www.json.org](http://www.json.org). -When you use JSON documents to represent your application data, you should think about the document as a logical container for information. This involves thinking about how data from your application fits into natural groups. It also requires thinking about the information you want to manage in your application. Doing data modeling for Couchbase Server is a similar process that you would do for traditional relational databases; there is however much more flexibility and you can change your mind later on your data structures. As a best practice, during your data/document design phase, you want to evaluate: +When you use JSON documents to represent your application data, you should think about the document as a logical container for information. In Couchbase, JSON documents are organized into collections within scopes, which are similar in concept to tables and schemas in relational databases. Collections help isolate types of documents (e.g., `beers`, `breweries`, `users`) within a bucket, improving data organization, access control, and query performance. You should plan your document modeling with collections in mind. + + This involves thinking about how data from your application fits into natural groups. It also requires thinking about the information you want to manage in your application. Data modeling for Couchbase is a similar process that you would do for traditional relational databases; there is however much more flexibility and you can change your mind later on your data structures. During your data/document design phase, you should evaluate: - What are the _**things**_ you want to manage in your applications, for instance, _users_, _breweries_, _beers_ and so forth. - What do you want to store about the _**things**_. For example, this could be _alcohol percentage_, _aroma_, _location_, etc. @@ -48,7 +50,7 @@ For instance, if you are creating a beer application, you might want a particula } ``` -For each of the keys in this JSON document you would provide unique values to represent individual beers. If you want to provide more detailed information in your beer application about the actual breweries, you could create a JSON structure to represent a brewery: +For each of the keys in this JSON document, you would provide unique values to represent individual beers. If you want to provide more detailed information in your beer application about the breweries, you could create a JSON structure to represent a brewery: ```json { @@ -61,7 +63,9 @@ For each of the keys in this JSON document you would provide unique values to re } ``` -Performing data modeling for a document-based application is no different than the work you would need to do for a relational database. For the most part it can be much more flexible, it can provide a more realistic representation or your application data, and it also enables you to change your mind later about data structure. For more complex items in your application, one option is to use nested pairs to represent the information: +Performing data modeling for a document-based application is no different than the work you would need to do for a relational database. However, it can be much more flexible, it can provide a more realistic representation or your application data, and it also enables you to more easily change your mind later about data structure. + +For more complex items in your application, one option is to use nested objects to represent the information: ```json { @@ -85,6 +89,6 @@ Performing data modeling for a document-based application is no different than t } ``` -In this case we added a nested attribute for the geolocation of the brewery and for beers. Within the location, we provide an exact longitude and latitude, as well as level of accuracy for plotting it on a map. The level of nesting you provide is your decision; as long as a document is under the maximum storage size for Couchbase Server, you can provide any level of nesting that you can handle in your application. +In this case we added a nested attribute for the geolocation of the brewery and a nested attribute of its beers. The level of nesting you provide is your decision; as long as a document is under the maximum storage size (20MB per document), you can provide any level of nesting that you can handle in your application. In traditional relational database modeling, you would create tables that contain a subset of information for an item. For instance a _brewery_ may contain types of beers which are stored in a separate table and referenced by the _beer ID_. In the case of JSON documents, you use key-values pairs, or even nested key-value pairs. diff --git a/tutorial/markdown/couchbase-server/best-practices/json-data-modeling-guide/03-schemaless-data-modeling.md b/tutorial/markdown/couchbase-server/best-practices/json-data-modeling-guide/03-schemaless-data-modeling.md index 1750fd4..7489ccc 100644 --- a/tutorial/markdown/couchbase-server/best-practices/json-data-modeling-guide/03-schemaless-data-modeling.md +++ b/tutorial/markdown/couchbase-server/best-practices/json-data-modeling-guide/03-schemaless-data-modeling.md @@ -8,7 +8,7 @@ description: - Explore important design considerations to keep in mind when designing JSON documents - See how you can use Couchbase's built-in Compare and Swap (CAS) functionality to maintain data consistency content_type: tutorial -filter: n1ql +filter: sql++ technology: - kv - capella @@ -34,17 +34,15 @@ While you can choose any structure for your documents, the JSON model in particu There are several considerations to have in mind when you design your JSON document: -- Whether you want to use a type field at the highest level of your JSON document in order to group and filter object types. - What particular keys, ids, prefixes or conventions you want to use for items, for instance **beer\_My\_Brew**. - When you want a document to expire, if at all, and what expiration would be best. - If want to use a document to access other documents. In other words, you can store keys that refer other documents in a JSON document and get the keys through this document. In the NoSQL database jargon, this is often known as using _composite keys_. -You can use a _type_ field to group together sets of records. For example, the following JSON document contains a _type_ field with the value _beer_ to indicate that the document represents a beer. A document that represents another kind of beverage would use a different value in the type field, such as _ale_ or _cider_. +You can group together sets of records into collections. For example, the following JSON document could be in the "beers" collection. A document that represents another kind of entity would be in a different collection, such as _brewery_ or _user_. ```json { "beer_id": "beer_Hoptimus_Prime", - "type": "beer", "abv": 10, "category": "North American Ale", "name": "Hoptimus Prime", @@ -52,24 +50,22 @@ You can use a _type_ field to group together sets of records. For example, the f } ``` -Here is another _type_ of document in our application which we use to represent breweries. As in the case of beers, we have a type field we can use now or later to group and categorize our beer producers: +Here is another document in the _breweries_ collection: ```json { "brewery_id": "brewery_Legacy_Brewing_Co", - "type": "brewery", "name": "Legacy Brewing Co.", "address": "525 Canal Street, Reading, Pennsylvania, 19601 United States", "updated": "2010-07-22 20:00:20" } ``` -What happens if we want to change the fields we store for a brewery? In this case we just add the fields to brewery documents. In this case we decide later that we want to include GPS location of the brewery: +What happens if we want to change the fields we store for a brewery? Add the fields to brewery documents. In this case we decide later that we want to include GPS location of the brewery: ```json { "brewery_id": "brewery_Legacy_Brewing_Co", - "type": "brewery", "name": "Legacy Brewing Co.", "address": "525 Canal Street, Reading, Pennsylvania, 19601 United States", "updated": "2010-07-22 20:00:20", @@ -78,7 +74,7 @@ What happens if we want to change the fields we store for a brewery? In this cas } ``` -So in the case of document-based data, we extend the record by just adding the two new fields for _latitude_ and _longitude_. When we add other breweries after this one, we would include these two new fields. For older breweries we can update them with the new fields or provide programming logic that shows a default for older breweries. The best approach for adding new fields to a document is to perform a compare and swap operation on the document to change it; with this type of operation, Couchbase Server will send you a message that the data has already changed if someone has already changed the record. For more information about compare and swap methods with Couchbase, see [Compare and Swap (CAS)](https://developer.couchbase.com/documentation/server/3.x/developer/dev-guide-3.0/update-info.html#concept29631__cas). +In the case of document-based data, we extend the record by just adding the two new fields for _latitude_ and _longitude_. When we add other breweries after this one, we would include these two new fields. For older breweries we can update them with the new fields or provide programming logic that shows a default for older breweries. The best approach for adding new fields to a document is to perform a compare and swap operation on the document to change it; with this type of operation, Couchbase will send you a message that the data has already changed if someone has already changed the record. For more information about compare and swap methods with Couchbase, see [Compare and Swap (CAS)](https://docs.couchbase.com/java-sdk/current/howtos/concurrent-document-mutations.html). To create relationships between items, we again use fields. In this example we create a logical connection between beers and breweries using the _brewery_ field in our beer document which relates to the _ID_ field in the brewery document. This is analogous to the idea of using a foreign key in traditional relational database design. @@ -87,7 +83,6 @@ This first document represents a beer, Hoptimus Prime: ```json { "beer_id": "beer_Hoptimus_Prime", - "type": "beer", "abv": 10, "brewery": "brewery_Legacy_Brewing_Co", "category": "North American Ale", @@ -101,7 +96,6 @@ This second document represents the brewery which brews Hoptimus Prime: ```json { "brewery_id": "brewery_Legacy_Brewing_Co", - "type": "brewery", "name": "Legacy Brewing Co.", "address": "525 Canal Street Reading, Pennsylvania, 19601 United States", "updated": "2010-07-22 20:00:20", diff --git a/tutorial/markdown/couchbase-server/best-practices/json-data-modeling-guide/04-phases-of-data-modeling.md b/tutorial/markdown/couchbase-server/best-practices/json-data-modeling-guide/04-phases-of-data-modeling.md index 7902d2c..6621c75 100644 --- a/tutorial/markdown/couchbase-server/best-practices/json-data-modeling-guide/04-phases-of-data-modeling.md +++ b/tutorial/markdown/couchbase-server/best-practices/json-data-modeling-guide/04-phases-of-data-modeling.md @@ -8,7 +8,7 @@ description: - Compare embedded documents to referenced documents and learn when to use each method - See best practices for building your data model content_type: tutorial -filter: n1ql +filter: sql++ technology: - kv - capella @@ -189,7 +189,6 @@ Referring to canonical instances of documents is a good default when modeling wi - You want to ensure your cache is used efficiently. - The embedded version would be unwieldy. - That last point is particularly important where your documents have an unbound potential for growth. Imagine we were storing activity logs related to each user of our system. Embedding those logs in the user profile could lead to a rather large document. @@ -211,17 +210,18 @@ It's unlikely we'd breach Couchbase's 20 MB upper limit for an individual docume The physical data model takes the logical data model and maps the entities and relationships to physical containers. -In Couchbase Server, items are used to store associated values that can be accessed with a unique key. Couchbase Server also provides buckets to group items. Based on the access patterns, performance requirements, and atomicity and consistency requirements, you can choose the type of container(s) to use to represent your logical data model. +In Couchbase, items are used to store associated values that can be accessed with a unique key. Couchbase also provides buckets to group items. Based on the access patterns, performance requirements, and atomicity and consistency requirements, you can choose the type of container(s) to use to represent your logical data model. -The data representation and containment in Couchbase Server is drastically different from relational databases. The following table provides a high level comparison to help you get familiar with Couchbase Server containers. +The data representation and containment in Couchbase is drastically different from relational databases. The following table provides a high level comparison to help you get familiar with Couchbase containers. -**Data representation and containment in Couchbase Server versus relational databases:** +**Data representation and containment in Couchbase versus relational databases:** -| Couchbase Server | Relational databases | +| Couchbase | Relational databases | |:------------------------------------ |:--------------------- | -| Buckets | Databases | -| Buckets or Items (with type designator attribute) | Tables | -| Items (key-value or document) | Rows | +| Buckets | Databases | +| Scopes | Schema/namespace | +| Collections | Tables | +| Documents/Items | Rows | | Index | Index | ### Items @@ -230,8 +230,8 @@ Items consist of a key and a value. A key is a unique identifier within the buck - **Keys**: Each value (binary or JSON) is identified by a unique key. The key is typically a surrogate key generated using a counter or a UUID generator. Keys are immutable. Thus, if you use composite or compound keys, ensure that you use attributes that don't change over time. - **Values** - - **Binary values**: Binary values can be used for high performance access to compact data through keys. Encrypted secrets, IoT instrument measurements, session states, or other non-human-readable data are typical cases for binary data. _Binary_ data may not necessarily be binary, but could be non-JSON formatted text like XML, String, etc. However, using binary values limits the functionality your application can take advantage of, ruling out indexing and querying in Couchbase Server as binary values have a proprietary representation. - - **JSON values**: JSON provides rich representation for entities. Couchbase Server can parse, index and query JSON values. JSON provide a name and a value for each attribute. You can find the JSON definition at [RFC 7159](https://tools.ietf.org/html/rfc7159) or at [ECMA 404](http://www.ecma-international.org/publications/files/ECMA-ST/ECMA-404.pdf). + - **Binary values**: Binary values can be used for high performance access to compact data through keys. Encrypted secrets, IoT instrument measurements, session states, or other non-human-readable data are typical cases for binary data. _Binary_ data may not necessarily be binary, but could be non-JSON formatted text like XML, String, etc. However, using binary values limits the functionality your application can take advantage of, ruling out indexing and querying in Couchbase as binary values have a proprietary representation. + - **JSON values**: JSON provides rich representation for entities. Couchbase can parse, index and query JSON values. JSON provide a name and a value for each attribute. You can find the JSON definition at [RFC 7159](https://tools.ietf.org/html/rfc7159) or at [ECMA 404](http://www.ecma-international.org/publications/files/ECMA-ST/ECMA-404.pdf). These type of items are typically called "documents" The JSON document attributes can represent both basic types such as number, string, Boolean, and complex types including embedded documents and arrays. In the examples below, a1 and a2 represent attributes that have a numeric and string value respectively, a3 represents an embedded document, and a4 represents an array of embedded documents. diff --git a/tutorial/markdown/couchbase-server/best-practices/json-data-modeling-guide/05-json-design-choices.md b/tutorial/markdown/couchbase-server/best-practices/json-data-modeling-guide/05-json-design-choices.md index f9e3620..2653931 100644 --- a/tutorial/markdown/couchbase-server/best-practices/json-data-modeling-guide/05-json-design-choices.md +++ b/tutorial/markdown/couchbase-server/best-practices/json-data-modeling-guide/05-json-design-choices.md @@ -8,7 +8,7 @@ description: - Explore versioning and document structure in more depth - Learn the difference between objects and object arrays content_type: tutorial -filter: n1ql +filter: sql++ technology: - kv - capella @@ -20,7 +20,7 @@ sdk_language: length: 20 Mins --- -Couchbase Server neither enforces nor validates for any particular document structure. Below are the design choices that impact JSON document design. +Couchbase neither enforces nor validates for any particular document structure. Below are the design choices that impact JSON document design. - **Document typing and versioning** - Key Prefixing @@ -33,21 +33,23 @@ Couchbase Server neither enforces nor validates for any particular document stru - Timestamp format - Valued, Missing, Empty, and Null attribute values -## Document Key Prefixing +## Document Collections -The document ID is the primary identifier of a document in the database. Multiple data sets are expected to share a common bucket in Couchbase. To ensure each data set has an isolated keyspace, it is a best practice to include a type/class/use-case/sub-domain prefix in all document keys. As an example of a User Model, you might have a property called `"userId": 123`, the document key might look like `user:123`, `user_123`, or `user::123`. Every Document ID is a combination of two or more parts/values, that should be delimited by a character such as a colon or an underscore. Pick a delimiter, and be consistent throughout your enterprise. +In Couchbase, different types of documents should be stored in separate collections. For example, users, orders, and products should each have their own collection within a scope. -Just as each Document ID should contain a prefix of the type/model, it is also a best practice to include that same value in the body of the document. This allows for efficient filtering by document type at query time or filtered XDCR replications. This property can be named many different names: `type`, `docType`, `_type`, and `_class` are all common choices, choose one that fits your organization's standards. +Each document key only needs to be unique within its collection. You do not need to include prefixes or type indicators in the key. For instance, a document in the users collection might simply use the key "123": ```json +key: "123" { - "_type": "user", - "userId": 123 + "firstName": "Leslie", + "lastName": "Knope" +} ``` ## Document Management Fields -At a minimum, every JSON document should contain a type and version property. Depending on your application requirements, use case, the line of business, etc. other common properties to consider at: +It may be helpful for documents to contain a version property and other document management fields. Depending on your application requirements, use case, the line of business, etc. other common properties to consider at: - `_created` - A timestamp of when the document was created in epoch time (milliseconds or seconds if millisecond precision is not required) - `_createdBy` - A user ID/name of the person or application that created the document @@ -60,10 +62,8 @@ The use of a leading `_` creates a standardized approach to global attributes ac ```json { - "_type": "user", "_schema": "1.2", "_created": 1544734688923 - "userId": 123 } ``` @@ -72,19 +72,16 @@ The same can be applied through a top-level property i.e. `"meta": {}`. ```json { "meta": { - "type": "user", "schema": "1.2", "created": 1544734688923 }, - "userId": 123 + "shoeSize": 13 } ``` Choose an approach that works within your organization and be consistent throughout your applications. -{% hint style="info" %} -**Note**: There is not a right or wrong property name, however, if you're application will leverage Couchbase Mobile (in particular Sync-Gateway), the use of a leading underscore should be avoided, as any document that contains root level properties with a leading underscore will fail to replicate. This is not a bug, and it meant to facilitate backward compatibility with v1.0 of the replication protocol. -{% endhint %} +> **Note**: There is not a right or wrong property name. However, if your application will be using Couchbase Mobile (in particular Sync-Gateway / Capella App Services), the use of a leading underscore should be avoided, as any document that contains [root level properties of reserved property names](https://docs.couchbase.com/sync-gateway/current/data-modeling.html) will be rejected. This is not a bug, and it meant to support backward compatibility with v1.0 of the replication protocol. ## Field name length, style, consistency @@ -92,84 +89,8 @@ Choose an approach that works within your organization and be consistent through - Self-documenting names reduce doc effort/maintenance `userName vs usyslogintxt` - Consistent patterns reduce bugs (pick and stick to a standard) `firstName or first_name or firstname`, but pick one. - Use plural names for array fields, and singular for others `"phones": [ ... ], "address": { ... }, "genre": " ... ", "scale": 2.3`. -- Avoid words that are reserved in N1QL (else, escape in N1QL) `user, bucket, cluster, role, select, insert` etc., Please refer [N1QL Reserved Word](https://docs.couchbase.com/server/current/n1ql/n1ql-language-reference/reservedwords.html) for more details on how to escape reserved words in N1QL. -- Use letters, numbers, or underscore (else, escape in N1QL) `first_name vs first-name`. - -## Root Attributes vs. Embedded Attributes - -The query model changes based on the choice of having a single root attribute or the `type` attribute embedded. Lets take a look at the `track` document as an example. - -### Root Attributes - -_Root_ attribute is a single, top-level attribute with all other attributes encapsulated as an object value of the root attribute. In the below example, the root element of the JSON document is `track`. - -```json -{ - "track": { - "artist": "Paul Lekakis", - "created": "2015-08-18T19:57:07", - "genre": "Hi-NRG", - "id": "3305311F4A0FAAFEABD001D324906748B18FB24A", - "mp3": "https://goo.gl/KgKoR7", - "ver": "1.0", - "ratings": [ - { - "created": "2015-08-20T12:24:44", - "rating": 4, - "username": "sublimatingraga37014" - }, - { - "created": "2015-08-21T09:23:57", - "rating": 4, - "username": "untillableshowings34122" - } - ], - "title": "My House", - "modified": "2015-08-18T19:57:07" - } -} -``` - -### Embedded Attributes - -In this example, the JSON document is in a flat structure but there is an attribute called `type` embedded within the document. - -```json -{ - "artist": "Paul Lekakis", - "created": "2015-08-18T19:57:07", - "genre": "Hi-NRG", - "id": "3305311F4A0FAAFEABD001D324906748B18FB24A", - "mp3": "https://goo.gl/KgKoR7", - "ver": "1.0", - "ratings": [ - { - "created": "2015-08-20T12:24:44", - "rating": 4, - "username": "sublimatingraga37014" - }, - { - "created": "2015-08-21T09:23:57", - "rating": 4, - "username": "untillableshowings34122" - } - ], - "title": "My House", - "modified": "2015-08-18T19:57:07", - "_type": "track" -} -``` - -This is the recommended approach since we can use the `type` field to create index. - -```sql -CREATE INDEX cb2_type ON couchmusic2(_type); - -SELECT COUNT(*) AS count -FROM couchmusic2 -WHERE _type = "track" -GROUP BY genre; -``` +- Avoid words that are reserved in SQL++: `user, bucket, cluster, role, select, insert` etc., Please refer to [SQL++ Reserved Words](https://docs.couchbase.com/server/current/n1ql/n1ql-language-reference/reservedwords.html) for more details on how to escape reserved words in SQL++. +- Use letters, numbers, or underscore: `first_name vs first-name`. ## Objects vs. Object Arrays @@ -179,7 +100,6 @@ There are two different ways to represent objects. ```json { - "type": "userProfile", "created": "2015-01-28T13:50:56", "dateOfBirth": "1986-06-09", "email": "andy.bowman@games.com", @@ -202,7 +122,6 @@ There are two different ways to represent objects. ```json { - "type": "userProfile", "created": "2015-01-28T13:50:56", "dateOfBirth": "1986-06-09", "email": "andy.bowman@games.com", @@ -236,7 +155,6 @@ Array values may be _simple_ or _object_. "id": "003c6f65-641a-4c9a-8e5e-41c947086cae", "name": "Eclectic Summer Mix", "owner": "copilotmarks61569", - "type": "playlist", "tracks": [ "9FFAF88C1C3550245A19CE3BD91D3DC0BE616778", "3305311F4A0FAAFEABD001D324906748B18FB24A", @@ -246,8 +164,9 @@ Array values may be _simple_ or _object_. } ``` -- Or, nest a summary to avoid a lookup/join - - There are lot of advantages in this approach over the first one. In this choice, all we have to do is _one_ get to retrieve all the information that we need regarding the playlist. +- Another approach is to nest a summary (copy some of the attributes from other documents) to avoid a lookup/join + - There are lot of advantages in this approach over the first one. In this choice, only _one_ get is required to retrieve all the information that we need regarding the playlist. + - However, is this copied data is changed often, it might not be worth duplicating data ```json { @@ -285,7 +204,6 @@ The following are examples of commonly used date formats. ```json { "countryCode": "US", - "type": "country", "gdp": 53548, "name": "United States of America", "region": "Americas", @@ -300,7 +218,6 @@ The following are examples of commonly used date formats. ```json { "countryCode": "US", - "type": "country", "gdp": 53548, "name": "United States of America", "region": "Americas", @@ -315,7 +232,6 @@ The following are examples of commonly used date formats. ```json { "countryCode": "US", - "type": "country", "gdp": 53548, "name": "United States of America", "region": "Americas", diff --git a/tutorial/markdown/couchbase-server/best-practices/json-data-modeling-guide/06-document-key-design.md b/tutorial/markdown/couchbase-server/best-practices/json-data-modeling-guide/06-document-key-design.md index c7d5aa9..4fbef4e 100644 --- a/tutorial/markdown/couchbase-server/best-practices/json-data-modeling-guide/06-document-key-design.md +++ b/tutorial/markdown/couchbase-server/best-practices/json-data-modeling-guide/06-document-key-design.md @@ -8,7 +8,7 @@ description: - See examples of several patterns you can follow when designing a document key - Explore the benefits of combining key design patterns content_type: tutorial -filter: n1ql +filter: sql++ technology: - kv - capella @@ -20,7 +20,7 @@ sdk_language: length: 10 Mins --- -The most important part of NoSQL database modeling is how do we design our document keys. There are different patterns as mentioned below when it comes to designing a document key. +An important part of JSON database modeling is how to design document keys. There are different patterns as mentioned below when it comes to designing a document key. - Prefixing - Predictable @@ -30,23 +30,23 @@ The most important part of NoSQL database modeling is how do we design our docum ## Prefixing -Multiple data sets are expected to share a common bucket in Couchbase. To ensure each data set has an isolated keyspace, it is a best practice to include a type/class/use-case/sub-domain prefix in all document keys. As an example of a User Model, you might have a property called `"userId": 123`, the document key might look like `user:123`, `user_123`, or `user::123`. Every Document ID is a combination of two or more parts/values, that should be delimited by a character such as a colon or an underscore. Pick a delimiter, and be consistent throughout your enterprise. +Remember that in Couchbase, different types of documents should be stored in separate collections. For example, users, orders, and products should each have their own collection within a scope. You do not need prefixing to isolate document types. -- DocType:ID - `userprofile:fredsmith79` - `playlist:003c6f65-641a-4c9a-8e5e-41c947086cae` -- AppName:DocType:ID - `couchmusic:userprofile:fredsmith79` -- DocType:ParentID:ChildID - `playlist:fredsmith79:003c6f65-641a-4c9a-8e5e-41c947086cae` +However, if you want to use prefixing for other purposes, pick a delimiter, and be consistent throughout your enterprise. For instance, you might use any of these approaches: `myPrefix:123`, `myPrefix_123`, or `myPrefix::123`. + +You may also use a prefix as a combination of two other documents (similar to a composite key), in order to store information on the relationship between those documents. + +For instance: + +- `123::987` in the `studentEnrollment` collection may be a document with information about student 123's enrollment in class 987. ## Predictable Let's say we're storing a user profile. Assuming no cookies, what are we guaranteed to know about our user after they've logged in? Well, one thing would be their login name. -So, if we want to make life easy for ourselves in retrieving our user profile, then we can key it with that user's login name. Everything else we need to know about that person could be derived from their user profile, in one way or another. +To make life easy for ourselves in retrieving the user profile, key it with that user's login name. Everything else we need to know about that person could be derived from their user profile, in one way or another. -Pretty quickly we might encounter a problem: for a user to change their login name, we now have to either create a new user profile under a new key or create a look-up document. We could insist that our users can never change their login names but it's unreasonable to make our users suffer unnecessarily. +But since keys are immutable, for a user to change their login name, we now have to either create a new user profile under a new key or create a look-up document. We could insist that our users can never change their login names but it's unreasonable to make our users suffer unnecessarily. The main downside of a predictable key is that, usually, it'll be an element of the data that we're storing. @@ -54,14 +54,14 @@ The main downside of a predictable key is that, usually, it'll be an element of ## Counter ID -We can get Couchbase to generate the key for us using a counter. if you're using a counter ID pattern, every insert (not update) requires 2 mutations. One to increment the counter and the other to mutate the document. +We can get Couchbase to generate a numeric key by using a counter. If you're using a counter ID pattern, every insert (not update) requires 2 mutations. One to increment a counter document and the other to mutate the document. Here's how it works: 1. Someone fills out the new user account form and clicks "Submit". -2. We increment our counter document and it returns the next number up (e.g. 123). -3. We create a new user profile document keyed with 123. -4. We then create look-up documents for things such as their user id, enabling us to do a simple look-up on the data we hold at login time. +2. Increment our counter document and it returns the next number up (e.g. 123). +3. Create a new user profile document keyed with 123. +4. Create look-up documents for things such as their user id, enabling us to do a simple look-up on the data we hold at login time. ![Counter Key](./assets/counter_key.png) @@ -75,10 +75,10 @@ This pattern uses system generated unique IDs like UUID. ## Combinations -It's when we combine both these methods that we can start to do really interesting things with key names. +Combining these methods can lead to the optimal strategy for your application. We've looked before at when to embed data in one large document and when it's best to refer to other documents. When we choose to refer to data held in separate documents, we can build predictable key names from components that tell us something about what the document holds. -Let's look at our user profile again. The main document is stored under the key 1001. We're working on an ecommerce site so we also want to know all of the orders our customer has made. Simple: we store the list of orders under the key `1001:orders`. +Let's look at our user profile again. The main document is stored under the key 1001. We're working on an ecommerce site so we also want to know all of the orders our customer has made. Simple: we store the list of orders under the key `1001:orders` in a `customerOrders` collection. -Similarly, our system might judge what sort of marketing emails to send to customers based on their total spend with the site. Rather than have the system calculate that afresh each time, we instead do it the NoSQL way: we calculate it once and then store it for later retrieval under the key `1001:orders:value`. +Similarly, our system might judge what sort of marketing emails to send to customers based on their total spend with the site. Rather than have the system calculate that each time, it can be calculated once and then stored for later retrieval under the key `1001:orders:value` in a `customerOrdersTotalSpend` collection. diff --git a/tutorial/markdown/couchbase-server/best-practices/json-data-modeling-guide/07-standardization-optimization-and-resources.md b/tutorial/markdown/couchbase-server/best-practices/json-data-modeling-guide/07-standardization-optimization-and-resources.md index 9acdd87..6b596a5 100644 --- a/tutorial/markdown/couchbase-server/best-practices/json-data-modeling-guide/07-standardization-optimization-and-resources.md +++ b/tutorial/markdown/couchbase-server/best-practices/json-data-modeling-guide/07-standardization-optimization-and-resources.md @@ -8,7 +8,7 @@ description: - Explore some resources to dive deeper into document optimization - See examples of standardized fields and why they are recommended content_type: tutorial -filter: n1ql +filter: sql++ technology: - kv - capella @@ -24,15 +24,11 @@ Wrapping up our Data Modeling Guide, we talk about standardization and optimizat ## Standardized Fields -### docType - -We have discussed about `docType` in the earlier section above. Please refer _Document Key Prefixing_ section in this document for details on docType. - ### Delimiter -Every Document ID can be a combination of two or more parts/values, that should be delimited by a character such as a colon or an underscore. Pick a delimiter, and be consistent throughout your enterprise. +When a Document ID is a combination of two or more parts/values, that should be delimited by a character such as a colon or an underscore. Pick a delimiter, and be consistent throughout your enterprise. -It is a best practice to only use a single-byte delimiter since this can make a significant difference based on the volume of data. +Consider using a single-byte delimiter since this can make a significant difference for large volumes of data. ### Schema @@ -43,16 +39,15 @@ Applications are typically versioned using Semantic Versioning, i.e. 2.5.1. Wher - 1 is bugfix/maintenance version Versioning the application informs users of features, functionality, updates, etc. -The term "schemaless", is often associated with NoSQL, while this is technically correct, it is better stated as: +The term "schemaless", is often associated with JSON databases. And while this is technically correct, it is better stated as: -> **Note**: "There is no schema managed by the database, however, there is still a schema, and it is an "Application Enforced Schema." The application is now responsible for enforcing the schema as well as maintaining the integrity of the data and relationships". +> "There is no schema managed by the database, however, there is still a schema, and it is an "Application Enforced Schema." The application is now responsible for enforcing the schema as well as maintaining the integrity of the data and relationships". As schemas change and evolve, documenting the version of the schema provides a mechanism of notifying applications about the schema version of the document that they're working with. This also enables a migration path for updating models which is discussed further in the Schema Versioning section. ```json { - "_type": "user", "_schema": "1.2", "userId": 123 } @@ -66,10 +61,8 @@ The use of a leading `_` creates a standardized approach to global attributes ac ```json { - "_type": "user", "_schema": "1.2", "_created": 1544734688923 - "userId": 123 } ``` @@ -78,11 +71,10 @@ The same can be applied through a top-level property i.e. `"meta": {}`. ```json { "meta": { - "type": "user", "schema": "1.2", "created": 1544734688923 }, - "userId": 123 + "shoeSize": 13 } ``` @@ -90,25 +82,27 @@ Choose an approach that works within your organization and be consistent through ## Optimizations -JSON gives us a flexible schema, that allows our models to rapidly adapt to change, this is because the schema is explicitly stored alongside each value. Whereas, in an RDBMS the schema is defined by the table columns, which are defined once. In any database, every byte of stored data adds up, historically this has been abstracted from developers as the schema and the database are managed by a DBA. With an application enforced schema, the model size is now controlled by the application. As developers we tend to be overly verbose when describing variables throughout our applications, this practice tends to carry over to our JSON models. While it is generally preferred to maintain human-readable field names for developer productivity, there are often well-understood abbreviations for many fields that will not reduce document readability. +JSON gives us a flexible schema that allows our models to rapidly adapt to change. The schema is explicitly stored alongside each value. In an RDBMS, the schema is defined by the table columns, which are defined once. + +In any database, every byte of stored data adds up. Often, this has been abstracted from developers as the schema and the database are managed by a DBA. With an application enforced schema, the model size is now controlled by the application. Developers cab be verbose when describing variables throughout our applications, and this practice tends to carry over to our JSON models. While human-readable field names are good for developer productivity, there are often well-understood abbreviations for many fields that will not reduce document readability, and will reduce the footprint of the data. As a general approach, consider the following options to proactively reduce document sizes: - Don't store the document ID as a repeated value in the document. - Convert ISO-8601 timestamps to epoch time in milliseconds, saving at least 11 bytes. When millisecond precision is not required, convert to a smaller value \(i.e. divide by 1000 to convert to seconds, 60 for minutes, 60 for hours, 24 for days\), saving at least 4 bytes. - Store dates as an ISO format `YYYY-MM-DD` instead of `MMM DD, YYYY`. -- When using GUID's strip all dashes saving an additional 4 bytes per GUID. +- When using GUID's, strip all dashes saving an additional 4 bytes per GUID. - Use shorter property names. - Don't store properties whose value is `null` , empty `String/Array/Object`, or a known default. - Don't repeat values in arrays whose value is not unique, use a top-level property on the document. ### Storing Dates -It is very common in almost any application, there is a need to store a date. This could be when the document was created, modified, when an order was placed, etc. Generally, this date is stored in [ISO-8601](https://www.iso.org/iso-8601-date-and-time-format.html) format. +In almost any application, there is a need to store a date. This could be when the document was created, modified, when an order was placed, etc. Generally, this date is stored in [ISO-8601](https://www.iso.org/iso-8601-date-and-time-format.html) format. -Take the date `2018-12-14T03:45:24.478Z` as an example, this is very _readable_, but is it the most efficient way to store the date? Storing this same date as [Unix Epoch Time](https://en.wikipedia.org/wiki/Unix_time) we can represent this same date as `1544759124478`. ISO-8601 is 24 bytes, where epoch format is 13 bytes, this saves 11 bytes. This might not seem like a lot, but consider this scenario: 500,000,000 documents and each document has an average of 2 date properties. If we used epoch format, we'd save 11,000,000,000 bytes or 11Gb of space. +Take the date `2018-12-14T03:45:24.478Z` as an example, this is very _readable_, but is it the most efficient way to store the date? Storing this same date as [Unix Epoch Time](https://en.wikipedia.org/wiki/Unix_time) we can represent this same date as `1544759124478`. ISO-8601 is 24 bytes, where epoch format is 13 bytes, this saves 11 bytes. This might not seem like a lot, but consider this scenario: 500,000,000 documents and each document has an average of 2 date properties. If we used epoch format, we'd save 11,000,000,000 bytes or 11Gb of space. -Now, take this a step further and ask the question, "What level of precision does the application require?". Often times we do not need millisecond precision, we can divide the epoch date accordingly for seconds, minutes, hours, etc. This applies if dates are being stored in Epoch format. +Now, take this a step further and ask the question, "What level of precision does the application require?". Often times we do not need millisecond precision. We can divide the epoch date accordingly for seconds, minutes, hours, etc. This applies if dates are being stored in Epoch format. | **Epoch Date** | **Precision** | **Reduction** | **Output** | **Length / Bytes** | | :--- | :--- | :--- | :--- | :--- | @@ -123,6 +117,5 @@ Please refer _Document Management Strategies_ guide for a more in-depth discussi ## Resources - [JSON Data Modeling for RDBMS Users](https://blog.couchbase.com/json-data-modeling-rdbms-users/) -- [Data Modeling for Couchbase with erwin DM NoSQL](https://blog.couchbase.com/data-modeling-for-couchbase-with-erwin-dm-nosql/) - [SQL to JSON Data Modeling with Hackolade](https://blog.couchbase.com/sql-to-json-data-modeling-hackolade/) - [Moving from SQL Server to Couchbase - Data Modeling](https://blog.couchbase.com/moving-from-sql-server-to-couchbase-part-1-data-modeling/) From 1aef28c44f186b7174621daa6eeeae7c163ef222 Mon Sep 17 00:00:00 2001 From: "Matthew D. Groves" Date: Fri, 1 Aug 2025 10:12:52 -0400 Subject: [PATCH 2/4] Update tutorial/markdown/couchbase-server/best-practices/json-data-modeling-guide/02-using-json-documents.md Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> --- .../json-data-modeling-guide/02-using-json-documents.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tutorial/markdown/couchbase-server/best-practices/json-data-modeling-guide/02-using-json-documents.md b/tutorial/markdown/couchbase-server/best-practices/json-data-modeling-guide/02-using-json-documents.md index 636d0ee..fdf423d 100644 --- a/tutorial/markdown/couchbase-server/best-practices/json-data-modeling-guide/02-using-json-documents.md +++ b/tutorial/markdown/couchbase-server/best-practices/json-data-modeling-guide/02-using-json-documents.md @@ -33,7 +33,7 @@ For more information about creating valid JSON documents, please refer to [http: When you use JSON documents to represent your application data, you should think about the document as a logical container for information. In Couchbase, JSON documents are organized into collections within scopes, which are similar in concept to tables and schemas in relational databases. Collections help isolate types of documents (e.g., `beers`, `breweries`, `users`) within a bucket, improving data organization, access control, and query performance. You should plan your document modeling with collections in mind. - This involves thinking about how data from your application fits into natural groups. It also requires thinking about the information you want to manage in your application. Data modeling for Couchbase is a similar process that you would do for traditional relational databases; there is however much more flexibility and you can change your mind later on your data structures. During your data/document design phase, you should evaluate: +This involves thinking about how data from your application fits into natural groups. It also requires thinking about the information you want to manage in your application. Data modeling for Couchbase is a similar process that you would do for traditional relational databases; there is however much more flexibility and you can change your mind later on your data structures. During your data/document design phase, you should evaluate: - What are the _**things**_ you want to manage in your applications, for instance, _users_, _breweries_, _beers_ and so forth. - What do you want to store about the _**things**_. For example, this could be _alcohol percentage_, _aroma_, _location_, etc. From 393298a2085cba292896ee2b08923932dae78584 Mon Sep 17 00:00:00 2001 From: "Matthew D. Groves" Date: Fri, 1 Aug 2025 10:13:06 -0400 Subject: [PATCH 3/4] Update tutorial/markdown/couchbase-server/best-practices/json-data-modeling-guide/04-phases-of-data-modeling.md Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> --- .../json-data-modeling-guide/04-phases-of-data-modeling.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tutorial/markdown/couchbase-server/best-practices/json-data-modeling-guide/04-phases-of-data-modeling.md b/tutorial/markdown/couchbase-server/best-practices/json-data-modeling-guide/04-phases-of-data-modeling.md index 6621c75..ef89f9f 100644 --- a/tutorial/markdown/couchbase-server/best-practices/json-data-modeling-guide/04-phases-of-data-modeling.md +++ b/tutorial/markdown/couchbase-server/best-practices/json-data-modeling-guide/04-phases-of-data-modeling.md @@ -231,7 +231,7 @@ Items consist of a key and a value. A key is a unique identifier within the buck - **Keys**: Each value (binary or JSON) is identified by a unique key. The key is typically a surrogate key generated using a counter or a UUID generator. Keys are immutable. Thus, if you use composite or compound keys, ensure that you use attributes that don't change over time. - **Values** - **Binary values**: Binary values can be used for high performance access to compact data through keys. Encrypted secrets, IoT instrument measurements, session states, or other non-human-readable data are typical cases for binary data. _Binary_ data may not necessarily be binary, but could be non-JSON formatted text like XML, String, etc. However, using binary values limits the functionality your application can take advantage of, ruling out indexing and querying in Couchbase as binary values have a proprietary representation. - - **JSON values**: JSON provides rich representation for entities. Couchbase can parse, index and query JSON values. JSON provide a name and a value for each attribute. You can find the JSON definition at [RFC 7159](https://tools.ietf.org/html/rfc7159) or at [ECMA 404](http://www.ecma-international.org/publications/files/ECMA-ST/ECMA-404.pdf). These type of items are typically called "documents" + - **JSON values**: JSON provides rich representation for entities. Couchbase can parse, index and query JSON values. JSON provide a name and a value for each attribute. You can find the JSON definition at [RFC 7159](https://tools.ietf.org/html/rfc7159) or at [ECMA 404](http://www.ecma-international.org/publications/files/ECMA-ST/ECMA-404.pdf). These type of items are typically called "documents". The JSON document attributes can represent both basic types such as number, string, Boolean, and complex types including embedded documents and arrays. In the examples below, a1 and a2 represent attributes that have a numeric and string value respectively, a3 represents an embedded document, and a4 represents an array of embedded documents. From 78571fb1312f72fe25dc5c5a01d8da2db0ef857d Mon Sep 17 00:00:00 2001 From: "Matthew D. Groves" Date: Fri, 1 Aug 2025 10:13:17 -0400 Subject: [PATCH 4/4] Update tutorial/markdown/couchbase-server/best-practices/json-data-modeling-guide/05-json-design-choices.md Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> --- .../json-data-modeling-guide/05-json-design-choices.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tutorial/markdown/couchbase-server/best-practices/json-data-modeling-guide/05-json-design-choices.md b/tutorial/markdown/couchbase-server/best-practices/json-data-modeling-guide/05-json-design-choices.md index 2653931..4c75986 100644 --- a/tutorial/markdown/couchbase-server/best-practices/json-data-modeling-guide/05-json-design-choices.md +++ b/tutorial/markdown/couchbase-server/best-practices/json-data-modeling-guide/05-json-design-choices.md @@ -166,7 +166,7 @@ Array values may be _simple_ or _object_. - Another approach is to nest a summary (copy some of the attributes from other documents) to avoid a lookup/join - There are lot of advantages in this approach over the first one. In this choice, only _one_ get is required to retrieve all the information that we need regarding the playlist. - - However, is this copied data is changed often, it might not be worth duplicating data + - However, if this copied data is changed often, it might not be worth duplicating data ```json {