|
| 1 | +--- |
| 2 | +custom_edit_url: "https://github.com/netdata/netdata/edit/master/docs/extreme-cardinality-protection.md" |
| 3 | +sidebar_label: "Extreme Cardinality Protection" |
| 4 | +learn_status: "Published" |
| 5 | +learn_rel_path: "Netdata Agent/Resource Utilization" |
| 6 | +keywords: [cardinality, ephemerality, retention, metadata, netdata-meta.db] |
| 7 | +description: "How Netdata limits long-term resource use from short-lived chart instances, and how metadata cleanup affects disk space." |
| 8 | +sidebar_position: "50" |
| 9 | +learn_link: "https://learn.netdata.cloud/docs/netdata-agent/resource-utilization/extreme-cardinality-protection" |
| 10 | +slug: "/netdata-agent/resource-utilization/extreme-cardinality-protection" |
| 11 | +--- |
| 12 | + |
| 13 | +# Extreme cardinality protection |
| 14 | + |
| 15 | +Extreme cardinality occurs when a node continually creates new, short-lived chart instances. Common causes include ephemeral containers and collectors that repeatedly discover transient resources. |
| 16 | + |
| 17 | +Netdata's extreme cardinality protection limits the long-term resource cost of this churn. It removes access to selected historical metrics after they have no samples left in tier 0, allowing their obsolete metadata to be cleaned up later. |
| 18 | + |
| 19 | +:::warning |
| 20 | + |
| 21 | +The protection deliberately makes some long-term historical samples inaccessible. Review the thresholds before lowering them or enabling the protection in an environment where old, short-lived instances must remain queryable. |
| 22 | + |
| 23 | +::: |
| 24 | + |
| 25 | +## What the protection measures |
| 26 | + |
| 27 | +Netdata evaluates cardinality independently for each node and context: |
| 28 | + |
| 29 | +- A **context** groups charts that describe the same type of resource, such as disks, network interfaces, or containers. |
| 30 | +- An **instance** is one chart within that context, such as a specific disk or container. |
| 31 | +- A **metric** is one dimension within an instance. |
| 32 | + |
| 33 | +An instance is eligible for cleanup only after all its remaining metrics have no retention in tier 0. Netdata calculates the context's ephemerality as the percentage of its retained instances that meet this condition. |
| 34 | + |
| 35 | +Counts shown in dashboards can aggregate multiple nodes. They do not necessarily match the per-node, per-context counts used by this protection. |
| 36 | + |
| 37 | +## When cleanup runs |
| 38 | + |
| 39 | +The protection is enabled by default when the Agent uses `dbengine` with more than one storage tier. It is disabled by default for other database configurations. |
| 40 | + |
| 41 | +After the first real rotation in any `dbengine` tier, Netdata schedules a full retention scan approximately two minutes later. During that scan, a context is eligible only when: |
| 42 | + |
| 43 | +- the protection is enabled; |
| 44 | +- the number of instances without tier-0 retention is at least `extreme cardinality keep instances`; and |
| 45 | +- those instances represent at least `extreme cardinality min ephemerality` percent of all retained instances in the context. |
| 46 | + |
| 47 | +Netdata retains at least the larger of: |
| 48 | + |
| 49 | +- the configured `extreme cardinality keep instances` value; or |
| 50 | +- the configured ephemerality percentage of all instances in the context. |
| 51 | + |
| 52 | +Only eligible instances above that retained amount are cleaned. Meeting both thresholds does not guarantee that anything will be removed; there must also be eligible instances above the amount Netdata keeps. |
| 53 | + |
| 54 | +With the defaults, for example, a context with 3,000 instances and 2,000 instances without tier-0 retention has 66% ephemerality. Netdata keeps 1,500 of those instances and clears long-term retention from up to 500 eligible instances. |
| 55 | + |
| 56 | +## What cleanup keeps and removes |
| 57 | + |
| 58 | +Netdata never selects an instance or metric that is currently being collected, even if it has not yet established tier-0 retention. |
| 59 | + |
| 60 | +For selected archived instances, Netdata clears the retention records for eligible metrics in every configured storage tier. Those historical samples become unavailable to queries immediately. The operation does not rewrite or compact existing `dbengine` datafiles; their physical blocks continue through normal database rotation. |
| 61 | + |
| 62 | +Netdata emits a notice only when it actually clears retention from at least one metric. |
| 63 | + |
| 64 | +## Metadata cleanup and disk space |
| 65 | + |
| 66 | +Metric metadata, including chart definitions, dimensions, and chart labels, is stored in `netdata-meta.db` in the Agent's [cache directory](/docs/netdata-agent/maintenance/backup-and-restore-an-agent). The directory varies by operating system and installation method. |
| 67 | + |
| 68 | +Clearing metric retention does not synchronously delete its SQLite metadata. Metadata maintenance: |
| 69 | + |
| 70 | +- starts its first scan about 30 minutes after the Agent starts; |
| 71 | +- processes large tables in bounded batches; |
| 72 | +- rescans dimensions weekly; and |
| 73 | +- deletes a chart and its labels after the chart's final dimension is removed. |
| 74 | + |
| 75 | +Depending on when retention is cleared relative to the scan, obsolete rows can remain until a later weekly pass. |
| 76 | + |
| 77 | +Deleted rows first become free pages that SQLite can reuse inside `netdata-meta.db`. With the default `[sqlite] auto vacuum = INCREMENTAL` setting, Netdata checks for reclaimable pages at most once per minute. It requests an incremental vacuum only when free pages exceed 5% of the database, and each pass requests 10% of the current free pages. |
| 78 | + |
| 79 | +As a result: |
| 80 | + |
| 81 | +- the database file can shrink gradually after enough pages become free; |
| 82 | +- smaller amounts of free space can remain allocated and be reused by later writes; |
| 83 | +- cleanup and file shrinkage do not have a fixed completion time; |
| 84 | +- with `NONE` or `OFF` auto-vacuum, freed pages remain available for reuse but are not automatically returned by this mechanism; and |
| 85 | +- `FULL` auto-vacuum follows SQLite's separate full-auto-vacuum behavior. |
| 86 | + |
| 87 | +These outcomes are normal and do not mean the cleanup failed. See the [SQLite configuration reference](/docs/netdata-agent/configuration/daemon) for the `auto vacuum` setting. |
| 88 | + |
| 89 | +## Configure the protection |
| 90 | + |
| 91 | +The settings are in the `[db]` section of `netdata.conf`: |
| 92 | + |
| 93 | +| Setting | Default | Set to | Effect | |
| 94 | +|:---------------------------------------|:------------------------------------------------|:---------------------|:----------------------------------------------------------------------------------------------------------------------------------------------| |
| 95 | +| `extreme cardinality protection` | `yes` for multi-tier `dbengine`; otherwise `no` | `yes` or `no` | Enables or disables forced cardinality cleanup. Normal size- and time-based retention still expires data when this protection is disabled. | |
| 96 | +| `extreme cardinality keep instances` | `1000` | `1` to `1000000` | Sets the minimum eligible-instance count and a lower bound on how many eligible instances remain. Higher values make cleanup less aggressive. | |
| 97 | +| `extreme cardinality min ephemerality` | `50` | `0` to `100` percent | Sets the minimum ephemerality and contributes to the number of eligible instances retained. Higher values make cleanup less aggressive. | |
| 98 | + |
| 99 | +The defaults need no manual configuration for a multi-tier `dbengine` Agent. To retain more short-lived instances, use higher thresholds, for example: |
| 100 | + |
| 101 | +```ini |
| 102 | +[db] |
| 103 | + extreme cardinality keep instances = 2000 |
| 104 | + extreme cardinality min ephemerality = 75 |
| 105 | +``` |
| 106 | + |
| 107 | +Use the [Netdata Agent configuration guide](/docs/netdata-agent/configuration#edit-configuration-files) to locate and edit `netdata.conf`, then [restart the Agent](/docs/netdata-agent/maintenance/service-control) for the changes to take effect. |
| 108 | + |
| 109 | +## Verify cleanup |
| 110 | + |
| 111 | +When cleanup removes retention, the notice follows this pattern: |
| 112 | + |
| 113 | +```text |
| 114 | +EXTREME CARDINALITY PROTECTION: host '<HOST>', context '<CONTEXT>', total active instances <TOTAL>, not in tier0 <WITHOUT_TIER0>, ephemerality <PERCENT>%: forcefully cleared the retention of <METRICS> metrics and <INSTANCES> instances, having non-tier0 retention from <START_TIME> to <END_TIME>. |
| 115 | +``` |
| 116 | + |
| 117 | +The event uses this message ID: |
| 118 | + |
| 119 | +```text |
| 120 | +d1f59606dd4d41e3b217a0cfcae8e632 |
| 121 | +``` |
| 122 | + |
| 123 | +On Linux systems that use the Netdata journal namespace, query it with: |
| 124 | + |
| 125 | +```bash |
| 126 | +sudo journalctl --namespace=netdata MESSAGE_ID=d1f59606dd4d41e3b217a0cfcae8e632 |
| 127 | +``` |
| 128 | + |
| 129 | +You can also open the **Logs** tab and filter `MESSAGE_ID` by **Netdata extreme cardinality**. For platform-specific log access and permission requirements, see [Netdata Logging](/docs/netdata-agent/logging#platform-specific-log-access). |
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?v=4&size=40){kind=avatar}
0 commit comments