|
| 1 | +// Copyright 2026 The Cockroach Authors. |
| 2 | +// |
| 3 | +// Use of this software is governed by the CockroachDB Software License |
| 4 | +// included in the /LICENSE file. |
| 5 | + |
| 6 | +/* |
| 7 | +Package multiregion provides the core abstractions for CockroachDB's |
| 8 | +multi-region SQL layer. It defines the read-only RegionConfig struct that |
| 9 | +represents a database's user-configured multi-region state and the utilities |
| 10 | +that consume it. |
| 11 | +
|
| 12 | +# Multi-Region Architecture |
| 13 | +
|
| 14 | +CockroachDB's multi-region implementation spans several packages. This package |
| 15 | +sits at the center, providing the metadata layer that other packages depend on. |
| 16 | +The key packages and their roles: |
| 17 | +
|
| 18 | + - pkg/sql/catalog/multiregion (this package) — Defines RegionConfig, the |
| 19 | + central read-only abstraction that captures a database's multi-region state: |
| 20 | + regions, primary and secondary regions, survival goal, placement policy, |
| 21 | + super regions, and zone config extensions. All downstream operations |
| 22 | + consume a RegionConfig rather than reading descriptors directly. |
| 23 | +
|
| 24 | + - pkg/sql/regions — Zone config synthesis engine. Given a RegionConfig and |
| 25 | + a table's locality, it deterministically produces the zone configuration |
| 26 | + that enforces the desired replica placement. This includes voter and |
| 27 | + non-voter constraint generation, lease preference synthesis, and |
| 28 | + super-region-aware constraint scoping. |
| 29 | +
|
| 30 | + - pkg/sql/regionliveness — Region availability probing. Uses the |
| 31 | + system.region_liveness table to detect and quarantine unavailable regions. |
| 32 | + This is consulted before DDL operations to prevent changes that reference |
| 33 | + unreachable regions. |
| 34 | +
|
| 35 | + - pkg/ccl/multiregionccl — Contains the multi-region initialization. |
| 36 | + Constructs the initial RegionConfig when a database first becomes |
| 37 | + multi-region (CREATE DATABASE ... PRIMARY REGION or ALTER DATABASE ... |
| 38 | + SET PRIMARY REGION). This package also determines the sorted insertion |
| 39 | + position for new region enum values. |
| 40 | +
|
| 41 | +# Data Flow |
| 42 | +
|
| 43 | +The typical flow when multi-region state changes is: |
| 44 | +
|
| 45 | + 1. The database descriptor and the multi-region enum type descriptor store |
| 46 | + the persistent multi-region state. Regions are represented as enum values |
| 47 | + in a special enum type descriptor associated with the database. This |
| 48 | + enum-based representation enables transactional region addition and |
| 49 | + removal: a region being added has its enum member Direction set to ADD; |
| 50 | + once the transition completes, the Direction becomes NONE and the |
| 51 | + member becomes fully public. A region being removed transitions through |
| 52 | + the REMOVE direction before the member is dropped entirely. |
| 53 | +
|
| 54 | + 2. SynthesizeRegionConfig reads those descriptors and produces a RegionConfig. |
| 55 | + This is an immutable snapshot of the database's multi-region state at that |
| 56 | + point in time. For DDL operations, the config is synthesized fresh (not |
| 57 | + cached) so it reflects in-flight changes. For DML operations, a cached |
| 58 | + version may be used. |
| 59 | +
|
| 60 | + 3. The zone config synthesis layer (pkg/sql/regions) consumes the RegionConfig |
| 61 | + along with each table's locality to produce zone configurations. These zone |
| 62 | + configs specify voter constraints, non-voter constraints, replica counts, |
| 63 | + and lease preferences that enforce the multi-region semantics. |
| 64 | +
|
| 65 | + 4. Zone configs are applied to the database and all affected tables, either |
| 66 | + directly during DDL execution or via the databaseRegionChangeFinalizer |
| 67 | + that runs after region enum value transitions complete. |
| 68 | +
|
| 69 | +# RegionConfig |
| 70 | +
|
| 71 | +Key fields of RegionConfig: |
| 72 | +
|
| 73 | + - regions: The set of PUBLIC regions in the database. These are the regions |
| 74 | + currently available for replica placement. |
| 75 | + - transitioningRegions: Regions being added or removed (in-flight enum |
| 76 | + transitions). These are tracked separately so that DDL validation can |
| 77 | + account for them. |
| 78 | + - addingRegions: The subset of transitioningRegions that are specifically |
| 79 | + being added (not removed). Exposed via the AddingRegions() method. |
| 80 | + - regionEnumID: The ID of the multi-region enum type descriptor that |
| 81 | + this RegionConfig was synthesized from. Links the config back to its |
| 82 | + source descriptor and is used during validation. |
| 83 | + - primaryRegion: The database's primary region. Tables without an explicit |
| 84 | + home region default to this region for leaseholder placement. |
| 85 | + - secondaryRegion: An optional failover region. If the primary region becomes |
| 86 | + unavailable, leaseholders move to the secondary region rather than to an |
| 87 | + arbitrary region. |
| 88 | + - survivalGoal: Either SURVIVE ZONE FAILURE (default) or SURVIVE REGION |
| 89 | + FAILURE. This controls how many voters and replicas are created and how |
| 90 | + they are distributed. |
| 91 | + - placement: Either DEFAULT or RESTRICTED. RESTRICTED suppresses non-voting |
| 92 | + replicas outside of the home region or super region, reducing replication |
| 93 | + cost at the expense of cross-region read performance. |
| 94 | + - superRegions: Named groupings of regions that define domiciling boundaries. |
| 95 | + See the Super Regions section below. |
| 96 | + - zoneCfgExtensions: Per-locality zone config overrides that let users |
| 97 | + customize zone config properties at different granularities. |
| 98 | +
|
| 99 | +# Table Localities |
| 100 | +
|
| 101 | +Multi-region databases support three table locality types, each of which |
| 102 | +produces different zone configurations: |
| 103 | +
|
| 104 | +GLOBAL tables replicate data to all database regions with non-blocking |
| 105 | +transactions (GlobalReads = true). Reads are fast everywhere because every |
| 106 | +region has a local replica. Writes pay the cost of writing to all regions. |
| 107 | +GLOBAL tables are not constrained by super regions — they intentionally |
| 108 | +replicate everywhere. |
| 109 | +
|
| 110 | +REGIONAL BY TABLE tables have an affinity to a single region (either the |
| 111 | +database primary region or an explicitly specified region). Leaseholders are |
| 112 | +placed in the affinity region. Voters are placed according to the survival |
| 113 | +goal: for zone survival, all voters go in the home region; for region |
| 114 | +survival, voters are distributed across regions to tolerate a region loss. |
| 115 | +
|
| 116 | +REGIONAL BY ROW tables are partitioned by a crdb_region column. Each |
| 117 | +partition behaves like a REGIONAL BY TABLE with affinity to the partition's |
| 118 | +region. This enables data domiciling: different rows in the same table can |
| 119 | +reside in different regions, and when super regions are configured, each |
| 120 | +row's replicas are confined to the super region that contains its region. |
| 121 | +
|
| 122 | +# Super Regions |
| 123 | +
|
| 124 | +A super region is a named grouping of database regions that defines a |
| 125 | +domiciling boundary. When data is homed in a region that belongs to a super |
| 126 | +region, all replicas — both voters and non-voters — are confined to the |
| 127 | +member regions of that super region. This is the foundation for data |
| 128 | +domiciling use cases where regulatory requirements mandate that data stays |
| 129 | +within a geographic or legal boundary. |
| 130 | +
|
| 131 | +Super regions are orthogonal to the survival goal. A super region with zone |
| 132 | +survivability confines replicas to its member regions and tolerates zone |
| 133 | +loss within those regions. A super region with region survivability also |
| 134 | +confines replicas but tolerates the loss of an entire member region. In |
| 135 | +both cases, no replica is placed outside the super region. |
| 136 | +
|
| 137 | +There are two kinds of super regions: |
| 138 | +
|
| 139 | +Explicit super regions are created by the user via ALTER DATABASE ... ADD |
| 140 | +SUPER REGION. They are stored in the multi-region enum type descriptor and |
| 141 | +appear in RegionConfig.SuperRegions(). A region may belong to at most one |
| 142 | +explicit super region. |
| 143 | +
|
| 144 | +Implicit super regions arise when a database has exactly three regions and |
| 145 | +uses SURVIVE REGION FAILURE. In this case, the system treats all three |
| 146 | +regions as a single implicit super region so that tables receive explicit |
| 147 | +per-region replica constraints (e.g. a 2+2+1 voter distribution) rather |
| 148 | +than inheriting the looser database-level zone config. This produces more |
| 149 | +deterministic replica placement across the three regions. |
| 150 | +HasImplicitSuperRegion returns true in this case. |
| 151 | +
|
| 152 | +RegionConfig provides methods to query super region membership, retrieve |
| 153 | +the member regions of a containing super region, and determine the |
| 154 | +effective survival goal for a given region (the super region's own goal |
| 155 | +if set, otherwise the database-level goal). |
| 156 | +
|
| 157 | +Super regions interact with table localities as follows: |
| 158 | +
|
| 159 | + - REGIONAL BY ROW: Each row's replicas are confined to the super region |
| 160 | + containing the row's crdb_region value. Different rows may be in |
| 161 | + different super regions within the same table. |
| 162 | + - REGIONAL BY TABLE: Replicas are confined to the super region containing |
| 163 | + the table's home region. |
| 164 | + - GLOBAL: Unaffected by super regions. GLOBAL tables replicate everywhere. |
| 165 | +
|
| 166 | +# Zone Config Extensions |
| 167 | +
|
| 168 | +Zone config extensions allow users to customize zone config properties at |
| 169 | +different levels of the multi-region hierarchy. Extensions are applied in |
| 170 | +a fixed precedence order during zone config synthesis: |
| 171 | +
|
| 172 | + 1. Regional — Applies to all REGIONAL tables and partitions. |
| 173 | + 2. Super Region — Applies to tables/partitions homed in a specific super |
| 174 | + region. Overrides the regional extension. |
| 175 | + 3. Regional In — Applies to tables/partitions homed in a specific region. |
| 176 | + Overrides both the regional and super region extensions. |
| 177 | +
|
| 178 | +For GLOBAL tables, only the global extension applies. |
| 179 | +
|
| 180 | +Extension inheritance works by having the extension "fill in" any fields |
| 181 | +not explicitly set from the base zone config, then replacing the base |
| 182 | +with the result. This means an extension only overrides the fields it |
| 183 | +explicitly sets. |
| 184 | +
|
| 185 | +The ExtendZoneConfigWithRegionalIn method applies all three regional tiers |
| 186 | +in order. The ExtendZoneConfigWithGlobal method applies the global |
| 187 | +extension. Both methods validate that the resulting zone config does not |
| 188 | +violate survival goal requirements (e.g., num_voters cannot be reduced |
| 189 | +below the minimum required for the survival goal). |
| 190 | +
|
| 191 | +# Validation |
| 192 | +
|
| 193 | +RegionConfig provides several validation functions: |
| 194 | +
|
| 195 | + - ValidateRegionConfig: Validates overall consistency — region enum ID is |
| 196 | + set, at least one region exists, placement and survival goal are |
| 197 | + compatible, super regions are valid, and zone config extensions reference |
| 198 | + valid regions. |
| 199 | + - ValidateSuperRegions: Validates that super region names are unique and |
| 200 | + sorted, regions within each super region are unique, sorted, and belong |
| 201 | + to the database, no region appears in multiple super regions, and each |
| 202 | + super region has enough regions for its effective survival goal. |
| 203 | + - CanSatisfySurvivalGoal: Checks whether the given number of regions is |
| 204 | + sufficient for the survival goal (REGION_FAILURE requires at least 3). |
| 205 | + - CanDropRegion: Checks whether dropping a region would violate the |
| 206 | + survival goal or break a super region membership constraint. |
| 207 | +*/ |
| 208 | +package multiregion |
0 commit comments