Skip to content

Commit 6ea8799

Browse files
g-despotg-despot
authored
Merge pull request #446 from weaviate/fix/publish-backup-restore-tooling
Publish WCD test-cluster restore tooling
2 parents 2d8c9a1 + 9c47c11 commit 6ea8799

3 files changed

Lines changed: 783 additions & 2 deletions

File tree

‎.gitignore‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -269,8 +269,9 @@ _includes/code/csharp/quickstart/bin
269269
_includes/code/csharp/quickstart/obj
270270
*.sln
271271

272-
# Exclude WCD backups
273-
tests/backups/
272+
# Exclude WCD snapshot data (hundreds of MB), but track the restore tooling
273+
# (restore.py, README.md). Snapshots live in backup_<timestamp>/ directories.
274+
tests/backups/backup_*/
274275

275276
# Ignore LLM/agent config files
276277
.claude/

‎tests/backups/README.md‎

Lines changed: 223 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,223 @@
1+
# WCD test-cluster restore
2+
3+
Tooling to rebuild the Weaviate Cloud (WCD) cluster the docs CI runs against,
4+
from a point-in-time snapshot of every collection's schema and objects (with
5+
their stored vectors and UUIDs).
6+
7+
## What's here
8+
9+
```
10+
restore.py # single entrypoint — runs stages 1–3 (see --stage)
11+
README.md # this file
12+
backup_<timestamp>/ # the snapshot — NOT committed (see "The snapshot")
13+
```
14+
15+
`restore.py` is the only script; the three restore stages are flags on it
16+
(`--stage`). The snapshot directory is **not committed** — it is hundreds of MB
17+
(one objects file alone is ~470 MB), so it is gitignored and obtained
18+
separately (see below).
19+
20+
## The snapshot
21+
22+
A snapshot is a directory named `backup_<timestamp>/` with this layout:
23+
24+
| File pattern | Content |
25+
|--------------------------------------|--------------------------------------------------|
26+
| `backup_metadata.json` | Index: every collection's name, MT flag, tenants |
27+
| `<Collection>_config.json` | Schema (properties, generative/MT config, …) |
28+
| `<Collection>_objects.json` | Objects, UUIDs, and stored vectors |
29+
| `<Collection>_<tenant>_objects.json` | MT objects, one file per tenant |
30+
31+
The current baseline, `backup_20251126_164527/`, was taken on 2025-11-26 and
32+
contains 23 collections (one multi-tenant). Because it is gitignored, it is
33+
**not in a clean checkout** — ask another maintainer for a copy, or restore from
34+
wherever your team stores it, and drop it next to `restore.py`.
35+
36+
`restore.py` locates the snapshot in this order:
37+
38+
1. `$WEAVIATE_BACKUP_DIR`, if set, or
39+
2. the newest `backup_*` directory next to `restore.py`.
40+
41+
Stages 1 and 2 read the snapshot; stage 3 does not (it uses the canonical
42+
dataset package), so a stage-3-only run needs no snapshot.
43+
44+
### Collections owned by the agents tests
45+
46+
`restore.py` **skips** `ECommerce`, `Weather`, and `FinancialContracts` (the
47+
`AGENTS_OWNED_COLLECTIONS` set). They are owned by the Query Agent tests
48+
(`docs/agents/_includes/query_agent.*`), which create them with
49+
`text2vec-weaviate` named vectors and load their data from HuggingFace — but
50+
only `if not collections.exists(...)`. The snapshot's lossy copies (no
51+
vectorizer config) would shadow that and break named-vector queries
52+
(`WEAVIATE_NAMED_VECTOR_ERROR` / `collection_vectors: []`). So a restore rebuilds
53+
**20** collections; the agents tests manage the other three. No non-agents test
54+
depends on the snapshot versions.
55+
56+
## Why three stages
57+
58+
The original backup tool serialized config via `str(...)`, which dropped
59+
structured details: vectorizer config, cross-references, inverted-index flags.
60+
Stage 1 alone gives a cluster with all the *data* back, but `near_text`/`hybrid`
61+
are silently broken because the vectorizer is unset — queries can't be embedded
62+
at runtime. Stage 2 plugs that hole for the 8 collections the docs tests
63+
actually search. Stage 3 reseeds Jeopardy from the canonical package because the
64+
snapshot lost `JeopardyQuestion.hasCategory` too.
65+
66+
```
67+
--stage 1 restore (bulk replay)
68+
├─ recreates schemas from *_config.json
69+
├─ batch-inserts objects with stored vectors + UUIDs
70+
├─ idempotent (skips collections with objects, recreates empty ones)
71+
└─ vectorizer left at "none" → near_vector works, near_text does not
72+
73+
--stage 2 repopulate (drops + recreates 8 specific collections)
74+
├─ adds text2vec-openai (ada-002) so near_text/hybrid work
75+
├─ adds cross-references + inverted-index flags the original tool lost
76+
└─ re-imports preserving stored vectors (no re-embedding cost)
77+
78+
--stage 3 jeopardy reseed (overwrites Jeopardy from canonical pkg)
79+
├─ ignores the snapshot for JeopardyQuestion + JeopardyCategory
80+
└─ uploads weaviate_datasets.JeopardyQuestions10k() with overwrite=True
81+
```
82+
83+
## Running the restore
84+
85+
```bash
86+
export WEAVIATE_URL="<cluster host without scheme>"
87+
export WEAVIATE_API_KEY="<admin api key>"
88+
export OPENAI_API_KEY="<openai key>" # stages 2 + 3 only
89+
90+
uv run python tests/backups/restore.py # all stages (default)
91+
uv run python tests/backups/restore.py --stage 1 # one stage
92+
uv run python tests/backups/restore.py --stage 2,3 # a subset
93+
```
94+
95+
A clean restore on an empty cluster takes a few minutes; stage 1 is the slowest
96+
because of the 10k-object `JeopardyQuestion` batch insert.
97+
98+
- **Stage 1 is idempotent** — re-running it skips collections that already have
99+
objects (and recreates empty/failed ones).
100+
- **Stages 2 and 3 are destructive** — they delete and recreate the collections
101+
they touch.
102+
103+
Stage 1 needs only `WEAVIATE_URL` + `WEAVIATE_API_KEY`. Stages 2 and 3 also need
104+
`OPENAI_API_KEY` because the recreated collections use `text2vec-openai` as the
105+
vectorizer (passed via the `X-OpenAI-Api-Key` header). `restore.py` validates
106+
that the variables required by the selected stages are set before connecting.
107+
108+
## Why ada-002 specifically (don't change this)
109+
110+
The stored vectors in the snapshot are 1536-d `text-embedding-ada-002`. Stage 2
111+
pins the vectorizer to `text-embedding-ada-002` so that **query-time** embedding
112+
lands in the same space as the **stored** vectors. Using v4's current default
113+
(`text-embedding-3-small`) would silently return semantically wrong results —
114+
the embedding spaces don't overlap.
115+
116+
## What stage 2 fixes per collection
117+
118+
Stage 2 only touches the 8 collections the docs tests semantically search
119+
(`COLLECTIONS_TO_FIX` in `restore.py`); everything else stays as stage 1
120+
restored it.
121+
122+
| Collection | Vectorizer | Other repairs |
123+
|----------------------|-------------------------------------------------------|-------------------------------------------------------------------|
124+
| `JeopardyQuestion` | `text2vec-openai` (ada-002), single | `hasCategory` cross-ref → `JeopardyCategory` |
125+
| `Article` | single | `inPublication`, `hasAuthors` cross-refs; `index_timestamps=true` |
126+
| `ArxivPapers` | single | — |
127+
| `Publication` | single | `hasArticles` cross-ref → `Article` |
128+
| `WineReview` | single | `index_null_state=true` (tests filter `IsNull`) |
129+
| `WineReviewMT` | single (MT) | `index_null_state=true` |
130+
| `GitBookChunk` | single | — |
131+
| `WineReviewNV` | named vectors `title`, `title_country`, `review_body` | `index_null_state=true` |
132+
133+
`WineReviewNV` is the only collection with multiple named vectors; everything
134+
else uses the legacy single `"default"` vectorizer.
135+
136+
### A wire-format quirk worth knowing
137+
138+
When a collection uses the legacy single-vectorizer form (not named vectors),
139+
inserts expect an **unnamed** vector. The snapshot stores it as
140+
`{"default": [...]}` (the named-vector shape). The stage-2 `coerce` closure in
141+
`restore.py` unwraps the `default` key before batch insert. Without that,
142+
inserts to those collections fail with a wire-format error.
143+
144+
## Stage 3 — why Jeopardy gets its own path
145+
146+
The snapshot's `JeopardyQuestion` lost the `hasCategory` cross-reference (the
147+
original backup tool didn't serialize references at all). Stage 2 declares the
148+
reference, but the per-object reference data isn't in the snapshot, so reads
149+
would still return empty for `hasCategory`. Stage 3 calls
150+
`weaviate_datasets.JeopardyQuestions10k().upload_dataset(...)`, which ships clean
151+
ada-002 vectors **and** the per-object `hasCategory` links wired up.
152+
153+
Stage 3 overwrites stage 2's Jeopardy work (`overwrite=True`). That's
154+
intentional — stage 2 keeps Jeopardy in its loop because it's the simplest
155+
re-import for the other 8 collections, and stage 3 then supersedes Jeopardy with
156+
the canonical source. The cluster ends at exactly 10,000 Jeopardy objects
157+
(stage 1 imports 10,004 from the snapshot; stage 3 finishes at 10,000).
158+
159+
## Verifying a restore worked
160+
161+
A few quick sanity checks against the restored cluster (run with the same env
162+
vars):
163+
164+
```python
165+
import os, weaviate
166+
from weaviate.classes.init import Auth
167+
168+
c = weaviate.connect_to_weaviate_cloud(
169+
cluster_url=os.environ["WEAVIATE_URL"],
170+
auth_credentials=Auth.api_key(os.environ["WEAVIATE_API_KEY"]),
171+
headers={"X-OpenAI-Api-Key": os.environ["OPENAI_API_KEY"]},
172+
)
173+
try:
174+
for name, expected in [("JeopardyQuestion", 10000), ("Article", 4403),
175+
("ArxivPapers", 2000), ("WineReview", 50)]:
176+
n = c.collections.get(name).aggregate.over_all(total_count=True).total_count
177+
print(f"{name}: {n}{' ✓' if n == expected else f' (expected {expected})'}")
178+
# Query-time vectorization works (proves stage 2 ran):
179+
r = c.collections.get("JeopardyQuestion").query.near_text("famous scientists", limit=1)
180+
print(f"near_text: {r.objects[0].properties.get('question')!r}")
181+
finally:
182+
c.close()
183+
```
184+
185+
Expected counts after a fresh restore (stages 1 → 2 → 3). `ECommerce`,
186+
`Weather`, and `FinancialContracts` are intentionally absent — the agents tests
187+
own them.
188+
189+
| Collection | Count |
190+
|---------------------------------|-------------------|
191+
| `JeopardyQuestion` | 10,000 |
192+
| `Article` | 4,403 |
193+
| `ArxivPapers` | 2,000 |
194+
| `Recipes` | 100 |
195+
| `WineReview` / `WineReviewNV` | 50 / 50 |
196+
| `WineReviewMT` | 50 per tenant × 2 |
197+
| `GitBookChunk` / `JeopardyTiny` | 10 / 10 |
198+
| `JeopardyCategory` | 40 |
199+
| `Movie` | 3 |
200+
201+
## Taking a new snapshot
202+
203+
There is **no snapshot script here** — `restore.py` only *reads* a snapshot. The
204+
current one was produced by a separate tool (iterating
205+
`client.collections.list_all()` and dumping properties + objects + vectors per
206+
collection). If you ever need a newer baseline:
207+
208+
1. Produce the same file layout (`backup_metadata.json` plus per-collection
209+
`_config.json` / `_objects.json`) in a new `backup_<timestamp>/` directory.
210+
2. Be aware of the lossy fields the original tool didn't capture — the schema
211+
features in the `EXTRA_SCHEMA` map in `restore.py` (references,
212+
`index_timestamps`, `index_null_state`). Either fix the backup tool to
213+
serialize them properly, or extend `EXTRA_SCHEMA`.
214+
3. `restore.py` auto-detects the newest `backup_*` directory, so no code change
215+
is needed; or point `$WEAVIATE_BACKUP_DIR` at the new directory explicitly.
216+
217+
## When to restore
218+
219+
- The CI test cluster gets wedged after a failed test run (collections in
220+
inconsistent states, half-deleted data, etc.).
221+
- You're spinning up a fresh WCD cluster for testing and want the same baseline
222+
the docs CI uses.
223+
- You suspect a flaky test is caused by cluster drift, not a real regression.

0 commit comments

Comments
 (0)