[docs] Update documentation for dynamic namespace configuration in Coordinator (#2874)

Author: nbroyles

site/content/docs/how_to/cluster_hard_way.md

@@ -163,6 +163,14 @@ Shortly after, you should see your node complete bootstrapping:
 20:10:14.764771[I] successfully updated topology to 3 hosts
 ```
 
+Once the node has completed bootstrapping, mark the namespace as ready so the coordinator knows it's ready to receive reads and writes:
+
+```shell
+curl -X POST http://localhost:7201/api/v1/services/m3db/namespace/ready -d '{
+  "name": "1week_namespace"
+}
+```
+
 If you need to setup multiple namespaces, you can run the above `/api/v1/database/create` command multiple times with different namespace configurations.
 
 ### Replication factor (RF)

site/content/docs/how_to/kubernetes.md

@@ -245,6 +245,16 @@ $ curl -sSf -X POST http://localhost:9003/query -d '{
 }
 ```
 
+To read and write metrics via the Coordinator (and not directly through the DB node API), you must mark the namespace as ready:
+
+```shell
+curl -X POST http://localhost:7201/api/v1/services/m3db/namespace/ready -d '{
+  "name": "default"
+}
+```
+
+You should now be able to use both the Coordinator and DB node API to perform reads and writes.
+
 ### Adding nodes
 
 You can easily scale your M3DB cluster by scaling the StatefulSet and informing the cluster topology of the change:

site/content/docs/how_to/query.md

@@ -34,26 +34,61 @@ The configuration file linked above uses an embedded etcd cluster, which is fine
 
 ## Aggregation
 
-You will notice that in the setup linked above, M3DB has just one unaggregated namespace configured. If you want aggregated metrics, you will need to set up an aggregated namespace in M3DB **and** in the m3query configuration. It is important to note that all writes go to all namespaces so as long as you include all namespaces in your query config, you will be querying all namespaces. Aggregation is done strictly by the query service. For example if you have an aggregated namespace setup in M3DB named `metrics_10s_48h`, you can add the following to the query config:
+You will notice that in the setup linked above, M3DB has just one unaggregated namespace configured. If you want aggregated metrics, you will need to set up an aggregated namespace. It is important to note that all writes go to all namespaces marked as ready. Aggregation is done strictly by the query service. As an example, to configure an aggregated namespace named `metrics_10s_48h`, you can execute the following API call:
 
-```yaml
-- namespace: metrics_10s_48h
-  type: aggregated
-  retention: 48h
-  resolution: 10s
+```shell
+curl -X POST <M3_COORDINATOR_IP_ADDRESS>:<CONFIGURED_PORT(default 7201)>/api/v1/services/m3db/namespace -d '{
+  "name": "metrics_10s_48h",
+  "options": {
+    "bootstrapEnabled": true,
+    "flushEnabled": true,
+    "writesToCommitLog": true,
+    "cleanupEnabled": true,
+    "snapshotEnabled": true,
+    "repairEnabled": false,
+    "retentionOptions": {
+      "retentionPeriodDuration": "48h",
+      "blockSizeDuration": "2h",
+      "bufferFutureDuration": "10m",
+      "bufferPastDuration": "10m",
+      "blockDataExpiry": true,
+      "blockDataExpiryAfterNotAccessedPeriodDuration": "5m"
+    },
+    "indexOptions": {
+      "enabled": true,
+      "blockSizeDuration": "2h"
+    },
+    "aggregationOptions": {
+      "aggregations": [
+         {
+          "aggregated": true,
+          "attributes": { "resolutionDuration": "10s" }
+        }
+      ]
+    }
+  }
+}'
 ```
 
+
 ### Disabling automatic aggregation
 
-If you run Statsite, m3agg, or some other aggregation tier, you will want to set the `all` flag under `downsample` to `false`. Otherwise, you will be aggregating metrics that have already been aggregated.
+If you run Statsite, m3agg, or some other aggregation tier, you will want to set the `all` flag under `downsample` to `false`. Otherwise, you will be aggregating metrics that have already been aggregated. Using the example above, `aggregationOptions` would be configured as follows
 
-```yaml
-- namespace: metrics_10s_48h
-  type: aggregated
-  retention: 48h
-  resolution: 10s
-  downsample:
-    all: false
+```shell
+    ...
+    "aggregationOptions": {
+      "aggregations": [
+        {
+          "aggregated": true,
+          "attributes": {
+            "resolutionDuration": "10s",
+            "downsampleOptions": { "all": false }
+          }
+        }
+      ]
+    }
+    ...
 ```
 
 ## ID generation

site/content/docs/how_to/use_as_tsdb.md

@@ -143,7 +143,15 @@ curl -X POST http://localhost:7201/api/v1/database/create -d '{
 
 Note that the `retentionTime` is set artificially low to conserve resources.
 
-After a few moments, the M3DB container should finish bootstrapping. At this point it should be ready to serve write and read queries.
+After a few moments, the M3DB container should finish bootstrapping. Once bootstrapping is finished, mark the namespace as ready to receive traffic:
+
+```shell
+curl -X POST http://localhost:7201/api/v1/services/m3db/namespace/ready -d '{
+  "name": "default"
+}
+```
+
+At this point it should be ready to serve write and read queries.
 
 #### Clients
 

site/content/docs/operational_guide/mapping_rollup.md

@@ -140,16 +140,22 @@ downsample:
           retention: 720h
 ```
 
-**Note:** In order to store rolled up metrics in an `unaggregated` namespace,
-a matching `aggregated` namespace must be added to the coordinator config. For 
-example, if in the above rule, the `720h` namespace under `storagePolicies` 
-is `unaggregated`, the following will need to be added to the coordinator config.
-
-```yaml
-- namespace: default
-  resolution: 30s
-  retention: 720h
-  type: aggregated
-  downsample:
-    all: false
+**Note:** In order to store rolled up metrics in an `unaggregated` namespace, the namespace's `aggregationOptions` must have a matching `aggregation`. For example, if in the above rule, the `720h` namespace under `storagePolicies` 
+is `unaggregated`, the `aggregationOptions` for that namespace should resemble the following:
+
+```json
+"aggregationOptions": {
+  "aggregations": [
+    {
+      "aggregated": false
+    },
+    {
+      "aggregated": true,
+      "attributes": {
+        "resolutionDuration": "30s",
+        "downsampleOptions": { "all": false }
+      }
+    }
+  ]
+}
 ```

site/content/docs/operational_guide/multiple_m3db_clusters.md

@@ -17,12 +17,7 @@ Example config file with `clusterManagement` (see end of the config):
 
 ```bash
 clusters:
-  # Should match the namespace(s) set up in the DB nodes
-  - namespaces:
-       - namespace: 21d
-         retention: 504h
-         type: unaggregated
-     client:
+   - client:
        config:
          service:
            env: default_env
@@ -45,16 +40,7 @@ clusters:
          backoffFactor: 2
          maxRetries: 3
          jitter: true
-   - namespaces:
-       - namespace: 90d
-         retention: 2160h
-         type: aggregated
-         resolution: 10m
-       - namespace: 500d
-         retention: 12000h
-         type: aggregated
-         resolution: 1h
-     client:
+   - client:
        config:
          service:
            env: lts_env

site/content/docs/operational_guide/namespace_configuration.md

@@ -32,14 +32,20 @@ curl -X POST <M3_COORDINATOR_IP_ADDRESS>:<CONFIGURED_PORT(default 7201)>/api/v1/
 
 will create a namespace called `default_unaggregated` with a retention of `24 hours`. All of the other namespace options will either use reasonable default values or be calculated based on the provided `retentionTime`.
 
-Adding a namespace does not require restarting M3DB, but will require modifying the M3Coordinator configuration to include the new namespace, and then restarting it.
+Adding a namespace requires you mark the namespace as ready once M3DB has finished bootstrapping it. This is done so that M3Coordinator knows the namespace is ready to receive reads and writes. Use the `api/v1/services/m3db/namespace/ready` endpoint to accomplish this:
+
+```shell
+curl -X POST <M3_COORDINATOR_IP_ADDRESS>:<CONFIGURED_PORT(default 7201)>/api/v1/services/m3db/namespace/ready -d '{
+  "name": "default_unaggregated"
+}
+```
 
 If you feel the need to configure the namespace options yourself (for performance or other reasons), read the `Advanced` section below.
 
 #### Advanced (Hard Way)
 
 The "advanced" API allows you to configure every aspect of the namespace that you're adding which can sometimes be helpful for development, debugging, and tuning clusters for maximum performance.
-Adding a namespace is a simple as using the `POST` `api/v1/namespace` API on an M3Coordinator instance.
+Adding a namespace is a simple as using the `POST` `api/v1/services/m3db/namespace` API on an M3Coordinator instance.
 
 ```shell
 curl -X POST <M3_COORDINATOR_IP_ADDRESS>:<CONFIGURED_PORT(default 7201)>/api/v1/services/m3db/namespace -d '{
@@ -52,31 +58,41 @@ curl -X POST <M3_COORDINATOR_IP_ADDRESS>:<CONFIGURED_PORT(default 7201)>/api/v1/
     "snapshotEnabled": true,
     "repairEnabled": false,
     "retentionOptions": {
-      "retentionPeriod": "2d",
-      "blockSize": "2h",
-      "bufferFuture": "10m",
-      "bufferPast": "10m",
+      "retentionPeriodDuration": "2d",
+      "blockSizeDuration": "2h",
+      "bufferFutureDuration": "10m",
+      "bufferPastDuration": "10m",
       "blockDataExpiry": true,
-      "blockDataExpiryAfterNotAccessedPeriod": "5m"
+      "blockDataExpiryAfterNotAccessedPeriodDuration": "5m"
     },
     "indexOptions": {
       "enabled": true,
-      "blockSize": "2h"
+      "blockSizeDuration": "2h"
+    },
+    "aggregationOptions": {
+      "aggregations": [
+        { "aggregated": false }
+      ]
     }
   }
 }'
 ```
 
-Adding a namespace does not require restarting M3DB, but will require modifying the M3Coordinator configuration to include the new namespace, and then restarting it.
+Adding a namespace requires you to mark the namespace as ready so M3Coordinator know it is ready to receive traffic:
+
+```shell
+curl -X POST <M3_COORDINATOR_IP_ADDRESS>:<CONFIGURED_PORT(default 7201)>/api/v1/services/m3db/namespace/ready -d '{
+  "name": "default_unaggregated"
+}
+```
 
 ### Deleting a Namespace
 
 Deleting a namespace is a simple as using the `DELETE` `/api/v1/services/m3db/namespace` API on an M3Coordinator instance.
 
 `curl -X DELETE <M3_COORDINATOR_IP_ADDRESS>:<CONFIGURED_PORT(default 7201)>/api/v1/services/m3db/namespace/<NAMESPACE_NAME>`
 
-Note that deleting a namespace will not have any effect on the M3DB nodes until they are all restarted. In addition, the namespace will need to be removed from the M3Coordinator configuration and then the M3Coordinator node will need to be restarted.
-
+Note that deleting a namespace will not have any effect on the M3DB nodes until they are all restarted.
 ### Modifying a Namespace
 
 There is currently no atomic namespace modification endpoint. Instead, you will need to delete a namespace and then add it back again with the same name, but modified settings. Review the individual namespace settings above to determine whether or not a given setting is safe to modify. For example, it is never safe to modify the blockSize of a namespace.
@@ -169,7 +185,7 @@ While it may be tempting to configure `bufferPast` and `bufferFuture` to very la
 
 Can be modified without creating a new namespace: `yes`
 
-### Index Options
+### indexOptions
 
 #### enabled
 
@@ -183,3 +199,26 @@ The size of blocks (in duration) that the index uses.
 Should match the databases [blocksize](#blocksize) for optimal memory usage.
 
 Can be modified without creating a new namespace: `no`
+
+### aggregationOptions
+Options for the Coordinator to use to make decisions around how to aggregate data points.
+
+Can be modified without creating a new namespace: `yes`
+
+#### aggregations
+One or more set of instructions on how data points should be aggregated within the namespace.
+
+##### aggregated
+Whether data points are aggregated.
+
+##### attributes
+If aggregated is true, specifies how to aggregate data.
+
+###### resolutionNanos
+The time range to aggregate data across.
+
+###### downsampleOptions
+Options related to downsampling data
+
+###### _all_
+Whether to send data points to this namespace. If false, the coordinator will not auto-aggregate incoming data points and data points must be sent the namespace via rules. Defaults to true.

site/content/docs/quickstart/_index.md

@@ -94,7 +94,6 @@ This quickstart uses the _{{% apiendpoint %}}database/create_ endpoint that crea
 
 You can create [placements](/docs/operational_guide/placement_configuration/) and [namespaces](/docs/operational_guide/namespace_configuration/#advanced-hard-way) separately if you need more control over their settings.
 
-The `namespaceName` argument must match the namespace in the `local` section of the `M3Coordinator` YAML configuration. If you [add any namespaces](/docs/operational_guide/namespace_configuration) you also need to add them to the `local` section of `M3Coordinator`'s YAML config.
 
 In another terminal, use the following command.
 
@@ -267,6 +266,31 @@ curl {{% apiendpoint %}}placement | jq .
 [Read more about the bootstrapping process](/docs/operational_guide/bootstrapping_crash_recovery/).
 {{% /notice %}}
 
+### Readying a Namespace
+
+Once a namespace has finished bootstrapping, it must be marked as ready before receiving traffic. This can be done by calling the _{{% apiendpoint %}}namespace/ready_.
+
+{{< tabs name="ready_namespaces" >}}
+{{% tab name="Command" %}}
+
+```shell
+curl -X POST http://localhost:7201/api/v1/services/m3db/namespace/ready -d '{
+  "name": "default"
+} | jq .
+```
+
+{{% /tab %}}
+{{% tab name="Output" %}}
+
+```json
+{
+  "ready": true
+}
+```
+
+{{% /tab %}}
+{{< /tabs >}}
+
 ### View Details of a Namespace
 
 You can also view the attributes of all namespaces by calling the _{{% apiendpoint %}}namespace_ endpoint