Skip to content

Commit f9a41c7

Browse files
Kota-SHKota-SH
authored
HBASE-30167: Document the Read Replica feature (#8459)
Signed-off-by: Andor Molnár <andor@apache.org> Signed-off-by: Peng Lu <lupeng@apache.org> Signed-off-by: Tak Lon (Stephen) Wu <taklwu@apache.org>
1 parent e3da929 commit f9a41c7

2 files changed

Lines changed: 206 additions & 0 deletions

File tree

‎hbase-website/app/pages/_docs/docs/_mdx/(multi-page)/meta.json‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -36,6 +36,7 @@
3636
"case-studies",
3737
"---Operations & Development---",
3838
"operational-management",
39+
"read-replica-cluster",
3940
"building-and-developing",
4041
"unit-testing",
4142
"---Internals---",

‎hbase-website/app/pages/_docs/docs/_mdx/(multi-page)/read-replica-cluster.mdx‎

Lines changed: 205 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,205 @@
1+
---
2+
title: "Read Replica Cluster"
3+
description: "Running a secondary HBase cluster in read-only mode against shared cloud storage to scale read workloads."
4+
---
5+
6+
## Background [#read-replica-cluster-background]
7+
8+
A _Read Replica Cluster_ is an entire HBase cluster running in global read-only mode against the same shared
9+
storage (`hbase.rootdir`) as an active read-write cluster. Both clusters list the same HFiles in the same
10+
HDFS / cloud-object-store location; no data is copied. Reads can be served from either cluster, letting the
11+
read workload be fanned out across multiple clusters without doubling storage cost.
12+
13+
Typical use cases:
14+
15+
- Fan out heavy scan / analytical workloads off the primary cluster.
16+
- Add cross-availability-zone read capacity backed by a single shared bucket.
17+
- Stand up an isolated cluster for read-mostly experiments without copying data.
18+
19+
<Callout type="info">
20+
**Eventual consistency.** A replica only sees data once (a) the active cluster has flushed the data to
21+
HFiles in shared storage, and (b) the replica has been told to re-read shared storage via the
22+
`refresh_meta` and `refresh_hfiles` commands. MemStore data on the active cluster is invisible to the
23+
replica until flushed.
24+
</Callout>
25+
26+
The parent design lives on [HBASE-29081](https://issues.apache.org/jira/browse/HBASE-29081).
27+
28+
## Design
29+
30+
The feature has three parts.
31+
32+
### Custom `hbase:meta` per cluster
33+
34+
Every cluster sharing a `hbase.rootdir` needs its own `hbase:meta` and its own master local region
35+
directory, because region assignments and master-local state are node-scoped and cannot be shared. Other
36+
system tables (`hbase:acl`, `hbase:replication`) _are_ safe to share because their contents are storage-wide
37+
and the replica never writes to them.
38+
39+
The configuration key `hbase.meta.table.suffix` selects a per-cluster suffix; the meta table becomes
40+
`hbase:meta_<suffix>` and the master's local store directory becomes `MasterData_<suffix>`. Each cluster
41+
sharing the same `hbase.rootdir` must be configured with a distinct suffix so its `hbase:meta` and
42+
`MasterData` directory do not collide with any other cluster's. The suffix must match `[a-zA-Z0-9]+`.
43+
44+
### Global read-only mode
45+
46+
`hbase.global.readonly.enabled=true` puts a cluster into read-only mode. Five coprocessor controllers under
47+
`org.apache.hadoop.hbase.security.access` intercept every user-table mutation path and throw
48+
`WriteAttemptedOnReadOnlyClusterException` (a `DoNotRetryIOException`) with the message
49+
`Operation not allowed in Read-Only Mode`:
50+
51+
| Class | Coprocessor host | Blocks |
52+
| -------------------------------- | ---------------- | ----------------------------------------------------------------------------------------------------- |
53+
| `MasterReadOnlyController` | Master | DDL, snapshots, splits, merges, namespace ops, ACL/quota ops, replication-peer ops |
54+
| `RegionReadOnlyController` | Region | put, delete, batchMutate, checkAnd\*, append, increment, flush, compaction, WAL append, commit/replay |
55+
| `RegionServerReadOnlyController` | RegionServer | WAL roll, replication sink mutations, log replay |
56+
| `BulkLoadReadOnlyController` | Region | bulk-load prepare/cleanup |
57+
| `EndpointReadOnlyController` | Region | all coprocessor endpoint invocations |
58+
59+
Operators do not load these classes manually. `CoprocessorConfigurationUtil.syncReadOnlyConfigurations`
60+
adds them to `hbase.coprocessor.master.classes`, `hbase.coprocessor.regionserver.classes`, and
61+
`hbase.coprocessor.region.classes` at startup and on every dynamic
62+
`ConfigurationManager.notifyAllObservers` event — so the flag can be flipped at runtime with
63+
`update_all_config` (see Case 3).
64+
65+
### Preventing Multiple Active Clusters (active.cluster.suffix.id)
66+
67+
Two clusters writing to the same `hbase.rootdir` would corrupt shared storage. To enforce a single writer, an
68+
active master creates a protobuf-serialized sentinel at `<hbase.rootdir>/active.cluster.suffix.id` recording
69+
its cluster ID and meta suffix. `MasterFileSystem.negotiateActiveClusterSuffixFile` runs at master startup:
70+
71+
- An **active** cluster (`hbase.global.readonly.enabled=false`) creates the file if absent, or verifies its
72+
contents match its own identity. If the file belongs to another cluster, startup aborts with an
73+
`IOException`.
74+
- A **replica** cluster (`hbase.global.readonly.enabled=true`) does not read or write the file; it logs
75+
`[Read-replica feature] Replica cluster is being started in Read Only Mode` and continues.
76+
77+
`AbstractReadOnlyController.manageActiveClusterIdFile` handles the dynamic toggle: switching to read-only
78+
deletes the file if this cluster owns it, and switching back to read-write creates the file if absent.
79+
80+
## Configuration
81+
82+
On every node of the **read replica cluster**, add the following to `hbase-site.xml`:
83+
84+
```xml
85+
<property>
86+
<name>hbase.global.readonly.enabled</name>
87+
<value>true</value>
88+
<description>
89+
Put this cluster into global read-only mode. All user-table writes, flushes,
90+
compactions, splits, and merges are blocked. The five ReadOnly coprocessor
91+
controllers are loaded automatically.
92+
</description>
93+
</property>
94+
<property>
95+
<name>hbase.meta.table.suffix</name>
96+
<value>replica1</value>
97+
<description>
98+
Optional. If set, the meta table is named hbase:meta_<suffix> and the
99+
master's local store directory is MasterData_<suffix>. Value must match
100+
[a-zA-Z0-9]+. Each cluster sharing the same hbase.rootdir MUST be
101+
configured with a distinct suffix so its hbase:meta and MasterData
102+
directory do not collide with any other cluster's.
103+
</description>
104+
</property>
105+
```
106+
107+
The **active cluster** uses the same `hbase.rootdir` but its own `hbase.meta.table.suffix` (distinct from
108+
every replica's suffix), and leaves `hbase.global.readonly.enabled` unset or `false`.
109+
110+
`hbase.global.readonly.enabled` is a dynamic configuration — a config-change event reloads the read-only
111+
coprocessors without restarting the process. All nodes must agree on the value; operators are responsible for
112+
keeping every `hbase-site.xml` in sync before issuing `update_all_config`.
113+
114+
## Operation and maintenance
115+
116+
### Case 1. Bring up a new read replica cluster
117+
118+
1. Provision the replica cluster on hardware that can reach the active cluster's `hbase.rootdir` (typically
119+
the same HDFS or object store).
120+
2. Set `hbase.global.readonly.enabled=true` in the replica's `hbase-site.xml`. And set up
121+
`hbase.meta.table.suffix`, to distinguish the replica cluster's meta table on the shared storage.
122+
3. Start the cluster and verify if the Master log shows
123+
`[Read-replica feature] Replica cluster is being started in Read Only Mode`.
124+
4. From the replica shell, run `refresh_meta` and then `refresh_hfiles` to materialize the active cluster's
125+
current state on the replica.
126+
127+
### Case 2. Routine sync after writes on the active cluster
128+
129+
```ruby
130+
# On the active cluster
131+
hbase> flush 'my_namespace:my_table'
132+
```
133+
134+
```ruby
135+
# On the read replica cluster
136+
hbase> refresh_meta
137+
hbase> refresh_hfiles 'TABLE_NAME' => 'my_namespace:my_table'
138+
```
139+
140+
Always run `refresh_meta` first, then `refresh_hfiles`. `refresh_hfiles` only refreshes regions that are open
141+
on the replica, so newly discovered regions must be in meta (and assigned) before their HFiles can be picked
142+
up. `refresh_hfiles` supports three scopes:
143+
144+
```ruby
145+
hbase> refresh_hfiles # all user tables
146+
hbase> refresh_hfiles 'TABLE_NAME' => 'ns:table' # one table
147+
hbase> refresh_hfiles 'NAMESPACE' => 'ns' # one namespace
148+
```
149+
150+
Passing both `TABLE_NAME` and `NAMESPACE` to `refresh_hfiles` is rejected. Both commands return a procedure
151+
ID that can be tracked through the master UI or `Admin.getProcedures()`.
152+
153+
If the replica's block cache holds stale entries for a table that has just been refreshed, evict them with
154+
the pre-existing `clear_block_cache 'my_namespace:my_table'` shell command.
155+
156+
`Admin` and `AsyncAdmin` expose the same operations programmatically:
157+
158+
```java
159+
long pid;
160+
pid = admin.refreshMeta();
161+
pid = admin.refreshHFiles(); // all user tables
162+
pid = admin.refreshHFiles(TableName.valueOf("ns:table")); // one table
163+
pid = admin.refreshHFiles("ns"); // one namespace
164+
```
165+
166+
### Case 3. Dynamically toggle read-only mode
167+
168+
`hbase.global.readonly.enabled` can be changed without a restart. Edit `hbase-site.xml` then
169+
trigger a configuration refresh (for example, `update_all_config` from the shell). `ConfigurationManager`
170+
notifies its observers, which load or unload the read-only coprocessors and call
171+
`AbstractReadOnlyController.manageActiveClusterIdFile`:
172+
173+
- **false → true (becoming a replica):** the `active.cluster.suffix.id` file is deleted only if its contents
174+
match this cluster; if another cluster owns the file, it is left in place.
175+
- **true → false (becoming active):** the file is recreated with this cluster's identity, unless it already
176+
exists.
177+
178+
In-flight batch operations are not interrupted; write operations submitted _after_ the toggle throw
179+
`WriteAttemptedOnReadOnlyClusterException`. The caller is responsible for handling and (if desired) resubmitting the failed
180+
mutations.
181+
182+
### Case 4. Promote a replica when the active cluster is lost
183+
184+
1. Confirm the original active cluster is fully down.
185+
2. If a stale `active.cluster.suffix.id` from the previous active is still present, remove it manually
186+
(e.g. `hdfs dfs -rm <hbase.rootdir>/active.cluster.suffix.id`, or the equivalent CLI for your object
187+
store). The new active master will refuse to start while a foreign sentinel file is in place.
188+
3. Set `hbase.global.readonly.enabled=false` on the replica and apply the change (dynamic update or
189+
restart). The master writes a fresh sentinel file with this cluster's identity.
190+
191+
## Configurations and Commands
192+
193+
### New configs
194+
195+
| Config | Default | Description |
196+
| ------------------------------- | ------- | ---------------------------------------------------------------------------------------------- |
197+
| `hbase.meta.table.suffix` | `""` | Adds a suffix to the meta table name. `value='test'` produces the table name `hbase:meta_test`. |
198+
| `hbase.global.readonly.enabled` | `false` | Puts the entire cluster into read-only mode. |
199+
200+
### New commands
201+
202+
| Command | Usage | Description |
203+
| ---------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------ |
204+
| `refresh_hfiles` | `refresh_hfiles`<br />`refresh_hfiles 'TABLE_NAME' => 'tablename'`<br />`refresh_hfiles 'TABLE_NAME' => 'namespace:test_table'`<br />`refresh_hfiles 'NAMESPACE' => 'namespace'` | Refreshes HFiles from disk. Used to pick up new edits on the read replica. |
205+
| `refresh_meta` | `refresh_meta` | Syncs the meta table with the backing storage. Used to pick up new tables and regions. |

0 commit comments

Comments
 (0)