Skip to content

Commit e36ffde

Browse files
firat-kayafirat-kaya
authored
Merge pull request #31 from looker-open-source/on-prem-verify
Add onprem-data-verifier tool
2 parents 868203d + de67c14 commit e36ffde

22 files changed

Lines changed: 1787 additions & 0 deletions

‎.gitignore‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
.idea

‎onprem-data-verifier/.idea/vcs.xml‎

Lines changed: 6 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎onprem-data-verifier/README.md‎

Lines changed: 174 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,174 @@
1+
# On-Prem Data Verifier
2+
3+
A robust validation tool written in Go, designed to verify Looker On-Premise backup artifacts before migration to Looker
4+
Cloud (hosted).
5+
6+
## Overview
7+
8+
Migrations are sensitive operations. This tool ensures that the data provided by the customer is:
9+
10+
1. **Complete:** Verifies the presence of all required encrypted and decrypted artifacts.
11+
12+
2. **Integrity Verified:** Matches the MD5 checksums generated at the source.
13+
14+
3. **Secure:** Confirms files are encrypted for the correct **Looker Public Key** (dynamically resolved via the
15+
Customer's LUID = Looker Unique ID).
16+
17+
4. **Valid:** The SQL dump is structurally sound, performant (Extended Inserts), uses the correct Charset (utf8mb4), and
18+
is for supported MYSQL version.
19+
20+
5. **Recoverable:** Customer Master Key (CMK), used for decrypting DB contents, is correct.
21+
22+
## Prerequisites
23+
24+
The tool relies on the system's GPG installation to verify encryption keys.
25+
26+
* **Go** 1.20+ (to build)
27+
* **GnuPG (gpg)** installed and on the system PATH (ie. `apt install gpg`).
28+
* **Imported Key:** You must import the customer's specific GPG Public Key into your local keyring before running the
29+
tool.
30+
```bash
31+
# Example (This is usually done via the customer script instructions)
32+
echo "PUBLIC_KEY_BLOCK" | gpg --import
33+
```
34+
35+
## Build
36+
37+
```bash
38+
go build -o onprem-verifier main.go
39+
```
40+
41+
## Usage
42+
43+
The tool operates on a **Workspace Directory**. This is a dedicated local directory on your machine where you
44+
consolidate
45+
all the backup files related to a single customer migration. This directory must contain both the *encrypted bundle*
46+
sent by the customer and the *decrypted artifacts* you extracted from that bundle. Think of it as a staging area for all
47+
the files the verifier needs to check.
48+
49+
### Command
50+
51+
```bash
52+
./onprem-verifier \
53+
--backupDir /path/to/migration_workspace \
54+
--customerName "lookersre-instance-1" \
55+
--luid "7b973058-f7e0-49f7-9262-54e1987659bc"
56+
```
57+
58+
| Flag | Shorthand | Required | Description |
59+
|:-----------------|:----------|:---------|:--------------------------------------------------------------------------|
60+
| `--backupDir` | `-b` | **Yes** | Path to the workspace directory containing ALL exported backup files. |
61+
| `--customerName` | `-c` | **Yes** | Customer Name (e.g., `lookersre-instance-1`). Used to validate filenames. |
62+
| `--luid` | `-l` | **Yes** | Looker User ID (e.g., `7b97...`). Used to resolve the GPG Public Key. |
63+
64+
## Workspace Requirements
65+
66+
The tool enforces a strict naming convention. The directory provided via `--backupDir` must contain **exactly** the
67+
following 7 files (where `${customer}` matches the `--customerName` flag):
68+
69+
1. **Encrypted Artifacts (Source)**
70+
71+
* `${customer}_looker_db_backup.sql.gz.enc`
72+
* `${customer}_looker_fs_backup.tar.gz.enc`
73+
* `${customer}_looker_cmk_key.enc`
74+
* `${customer}_backup.md5`
75+
76+
2. **Decrypted Artifacts (Target)**
77+
78+
* `${customer}_looker_db_backup.sql.gz`
79+
* `${customer}_looker_fs_backup.tar.gz`
80+
* `${customer}_looker_cmk_key`
81+
82+
## Validation Pipeline
83+
84+
The tool executes checks in the following order. If any step fails, the process terminates.
85+
86+
### 1. Workspace Structure
87+
88+
* **Action:** Checks if all 7 required files exist in `--backupDir`.
89+
* **Goal:** Fail fast if data is missing or named incorrectly.
90+
91+
### 2. Integrity Check
92+
93+
* **Action:** Parses `${customer}_backup.md5` and verifies the hash of every file listed.
94+
* **Goal:** Ensure no file corruption occurred during transfer.
95+
96+
### 3. Security Check (Dynamic GPG)
97+
98+
* **Action:**
99+
1. Constructs the expected migration email: `looker-devops+migration-{LUID}@google.com`.
100+
2. Queries your local GPG keyring (`gpg --list-keys`) to find **ALL** Key IDs (Primary + Subkeys) associated with
101+
that email.
102+
3. Inspects the headers of `.enc` files (`gpg --list-packets`) to ensure they are encrypted for one of those valid
103+
Key IDs.
104+
* **Goal:** Prevent importing data encrypted with the wrong key.
105+
106+
### 4. Database Validation
107+
108+
* **Action:** Streams the `.sql.gz` file (single-pass scan) to analyze content without full decompression.
109+
* **Checks:**
110+
* **Looker Version:** Must match the supported version list.
111+
* **Charset:** Must be `utf8mb4` (Looker Requirement).
112+
* **Extended Inserts:** Ensures `INSERT` statements are batched (Critical for performance).
113+
* **Critical Tables:** Verifies existence of `user`, `dashboard`, `db_connection`.
114+
115+
### 5. CMK Validation
116+
117+
* **Action:** Reads the `${customer}_looker_cmk_key`.
118+
* **Check:** Validates the key is either 32 bytes (Raw) or 44 bytes (Base64).
119+
120+
### 6. FileSystem Analysis
121+
122+
* **Action:** Analyzes the `${customer}_looker_fs_backup.tar.gz` archive size and metadata.
123+
124+
## Output
125+
126+
### Console (STDOUT)
127+
128+
Clean, colored, step-by-step logs indicating progress and specific validation results.
129+
130+
```text
131+
=== Looker On-Prem Verification Pipeline ===
132+
133+
>> [1/6] Checking Workspace Structure
134+
Directory: /workspace
135+
[OK] Workspace structure verified
136+
137+
>> [2/6] Verifying MD5 Checksums
138+
[OK] All files match their checksums
139+
140+
>> [3/6] Resolving Security Keys
141+
[OK] Found Valid Key IDs: [000ECF...]
142+
[OK] sm-restore_looker_db_backup.sql.gz.enc is encrypted correctly
143+
...
144+
145+
>> [4/6] Analyzing Database: sm-restore_looker_db_backup.sql.gz
146+
[OK] Version 25.18.33 is supported
147+
[OK] Database Charset: utf8mb4
148+
[OK] Extended Inserts detected
149+
[OK] Critical tables verified: [user dashboard db_connection]
150+
151+
[SUCCESS] VERIFICATION COMPLETE
152+
Customer: sm-restore
153+
LUID: d5c8...
154+
Duration: 6.54s
155+
```
156+
157+
### Report File
158+
159+
A JSON report is generated at `metadata.json` in the current directory or specified path.
160+
161+
```json
162+
{
163+
"customer_name": "lookersre-instance-1",
164+
"instance_id": "7b973058-f7e0-49f7-9262-54e1987659bc",
165+
"generated_at": "2026-01-16T15:30:00Z",
166+
"fs_total_size_bytes": 5368709120,
167+
"db_total_size_bytes": 1073741824,
168+
"table_count": 142,
169+
"cmk_status": "Valid",
170+
"cmk_encoding": "Base64",
171+
"looker_version": "25.18.33",
172+
"duration_in_seconds": 6.54
173+
}
174+
```

‎onprem-data-verifier/cmd/root.go‎

Lines changed: 103 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,103 @@
1+
package cmd
2+
3+
import (
4+
"fmt"
5+
"os"
6+
7+
"onprem-data-verifier/logger"
8+
"onprem-data-verifier/validator"
9+
10+
"github.com/spf13/cobra"
11+
"github.com/spf13/viper"
12+
)
13+
14+
var (
15+
// Flags
16+
backupDir string
17+
customerName string
18+
luid string
19+
outputFile string
20+
21+
// RootCmd represents the base command
22+
RootCmd = &cobra.Command{
23+
Use: "onprem-data-verifier",
24+
Short: "Validates On-Premise Looker backups before migration",
25+
Long: `
26+
On-Prem Data Verifier
27+
=====================
28+
Validates integrity and schema of Looker On-Premise backups.
29+
30+
Requires a workspace directory (--backupDir) and explicit paths to critical files.
31+
32+
Example:
33+
onprem-data-verifier \
34+
--backupDir ./workspace \
35+
--customerName lookersre-scotty-1 \
36+
--luid "u-12345-6789" \
37+
--output metadata.json"`,
38+
39+
Run: func(cmd *cobra.Command, args []string) {
40+
// Initialize Orchestrator with the new flag variables
41+
orchestrator := validator.NewOrchestrator(backupDir, customerName, luid)
42+
43+
report, err := orchestrator.Run()
44+
if err != nil {
45+
// Use the logger to print the Red Error Block and Exit
46+
logger.Fatal(err)
47+
}
48+
49+
// Use the logger for the Green Success Block
50+
// Format the float as a string (e.g., "1.23s")
51+
logger.Completion(
52+
report.CustomerName,
53+
report.InstanceID,
54+
report.GeneratedAt,
55+
report.LookerVer,
56+
report.TableCount,
57+
report.FSTotalSizeBytes,
58+
report.DBTotalSizeBytes,
59+
report.CmkStatus,
60+
report.CmkFormat,
61+
report.DurationInSeconds,
62+
)
63+
64+
// Save the report to the output file
65+
if err := report.Save(outputFile); err != nil {
66+
logger.Warn("Could not save report: %v", err)
67+
}
68+
},
69+
}
70+
)
71+
72+
func Execute() {
73+
if err := RootCmd.Execute(); err != nil {
74+
// Changed fmt.Println to logger.Error for consistency
75+
logger.Error("%v", err)
76+
os.Exit(1)
77+
}
78+
}
79+
80+
func init() {
81+
// 1. Define Persistent Flags
82+
RootCmd.PersistentFlags().StringVarP(&backupDir, "backupDir", "b", "", "Backup Directory containing ALL backup files")
83+
RootCmd.PersistentFlags().StringVarP(&customerName, "customerName", "c", "", "Customer Name for the report")
84+
RootCmd.PersistentFlags().StringVarP(&luid, "luid", "l", "", "Looker User ID (LUID)")
85+
RootCmd.PersistentFlags().StringVarP(&outputFile, "output", "o", "metadata.json", "Output file for the report")
86+
87+
// 2. Mark ALL specific flags as REQUIRED
88+
requiredFlags := []string{"backupDir", "customerName", "luid"}
89+
for _, flag := range requiredFlags {
90+
if err := RootCmd.MarkPersistentFlagRequired(flag); err != nil {
91+
// Using fmt here is fine for initialization panics
92+
panic(fmt.Sprintf("Failed to mark flag required: %s - %v", flag, err))
93+
}
94+
}
95+
96+
// 3. Bind Viper
97+
bindFlags := []string{"backupDir", "customerName", "luid", "output"}
98+
for _, flag := range bindFlags {
99+
if err := viper.BindPFlag(flag, RootCmd.PersistentFlags().Lookup(flag)); err != nil {
100+
panic(fmt.Sprintf("Failed to bind flag to viper: %s - %v", flag, err))
101+
}
102+
}
103+
}

‎onprem-data-verifier/go.mod‎

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
module onprem-data-verifier
2+
3+
go 1.24
4+
5+
require (
6+
github.com/spf13/cobra v1.10.2
7+
github.com/spf13/viper v1.21.0
8+
github.com/stretchr/testify v1.11.1
9+
)
10+
11+
require (
12+
github.com/davecgh/go-spew v1.1.1 // indirect
13+
github.com/fsnotify/fsnotify v1.9.0 // indirect
14+
github.com/go-viper/mapstructure/v2 v2.4.0 // indirect
15+
github.com/inconshreveable/mousetrap v1.1.0 // indirect
16+
github.com/pelletier/go-toml/v2 v2.2.4 // indirect
17+
github.com/pmezard/go-difflib v1.0.0 // indirect
18+
github.com/sagikazarmark/locafero v0.11.0 // indirect
19+
github.com/sourcegraph/conc v0.3.1-0.20240121214520-5f936abd7ae8 // indirect
20+
github.com/spf13/afero v1.15.0 // indirect
21+
github.com/spf13/cast v1.10.0 // indirect
22+
github.com/spf13/pflag v1.0.10 // indirect
23+
github.com/subosito/gotenv v1.6.0 // indirect
24+
go.yaml.in/yaml/v3 v3.0.4 // indirect
25+
golang.org/x/sys v0.29.0 // indirect
26+
golang.org/x/text v0.28.0 // indirect
27+
gopkg.in/yaml.v3 v3.0.1 // indirect
28+
)

‎onprem-data-verifier/go.sum‎

Lines changed: 54 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,54 @@
1+
github.com/cpuguy83/go-md2man/v2 v2.0.6/go.mod h1:oOW0eioCTA6cOiMLiUPZOpcVxMig6NIQQ7OS05n1F4g=
2+
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
3+
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
4+
github.com/frankban/quicktest v1.14.6 h1:7Xjx+VpznH+oBnejlPUj8oUpdxnVs4f8XU8WnHkI4W8=
5+
github.com/frankban/quicktest v1.14.6/go.mod h1:4ptaffx2x8+WTWXmUCuVU6aPUX1/Mz7zb5vbUoiM6w0=
6+
github.com/fsnotify/fsnotify v1.9.0 h1:2Ml+OJNzbYCTzsxtv8vKSFD9PbJjmhYF14k/jKC7S9k=
7+
github.com/fsnotify/fsnotify v1.9.0/go.mod h1:8jBTzvmWwFyi3Pb8djgCCO5IBqzKJ/Jwo8TRcHyHii0=
8+
github.com/go-viper/mapstructure/v2 v2.4.0 h1:EBsztssimR/CONLSZZ04E8qAkxNYq4Qp9LvH92wZUgs=
9+
github.com/go-viper/mapstructure/v2 v2.4.0/go.mod h1:oJDH3BJKyqBA2TXFhDsKDGDTlndYOZ6rGS0BRZIxGhM=
10+
github.com/google/go-cmp v0.6.0 h1:ofyhxvXcZhMsU5ulbFiLKl/XBFqE1GSq7atu8tAmTRI=
11+
github.com/google/go-cmp v0.6.0/go.mod h1:17dUlkBOakJ0+DkrSSNjCkIjxS6bF9zb3elmeNGIjoY=
12+
github.com/inconshreveable/mousetrap v1.1.0 h1:wN+x4NVGpMsO7ErUn/mUI3vEoE6Jt13X2s0bqwp9tc8=
13+
github.com/inconshreveable/mousetrap v1.1.0/go.mod h1:vpF70FUmC8bwa3OWnCshd2FqLfsEA9PFc4w1p2J65bw=
14+
github.com/kr/pretty v0.3.1 h1:flRD4NNwYAUpkphVc1HcthR4KEIFJ65n8Mw5qdRn3LE=
15+
github.com/kr/pretty v0.3.1/go.mod h1:hoEshYVHaxMs3cyo3Yncou5ZscifuDolrwPKZanG3xk=
16+
github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY=
17+
github.com/kr/text v0.2.0/go.mod h1:eLer722TekiGuMkidMxC/pM04lWEeraHUUmBw8l2grE=
18+
github.com/pelletier/go-toml/v2 v2.2.4 h1:mye9XuhQ6gvn5h28+VilKrrPoQVanw5PMw/TB0t5Ec4=
19+
github.com/pelletier/go-toml/v2 v2.2.4/go.mod h1:2gIqNv+qfxSVS7cM2xJQKtLSTLUE9V8t9Stt+h56mCY=
20+
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
21+
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
22+
github.com/rogpeppe/go-internal v1.9.0 h1:73kH8U+JUqXU8lRuOHeVHaa/SZPifC7BkcraZVejAe8=
23+
github.com/rogpeppe/go-internal v1.9.0/go.mod h1:WtVeX8xhTBvf0smdhujwtBcq4Qrzq/fJaraNFVN+nFs=
24+
github.com/russross/blackfriday/v2 v2.1.0/go.mod h1:+Rmxgy9KzJVeS9/2gXHxylqXiyQDYRxCVz55jmeOWTM=
25+
github.com/sagikazarmark/locafero v0.11.0 h1:1iurJgmM9G3PA/I+wWYIOw/5SyBtxapeHDcg+AAIFXc=
26+
github.com/sagikazarmark/locafero v0.11.0/go.mod h1:nVIGvgyzw595SUSUE6tvCp3YYTeHs15MvlmU87WwIik=
27+
github.com/sourcegraph/conc v0.3.1-0.20240121214520-5f936abd7ae8 h1:+jumHNA0Wrelhe64i8F6HNlS8pkoyMv5sreGx2Ry5Rw=
28+
github.com/sourcegraph/conc v0.3.1-0.20240121214520-5f936abd7ae8/go.mod h1:3n1Cwaq1E1/1lhQhtRK2ts/ZwZEhjcQeJQ1RuC6Q/8U=
29+
github.com/spf13/afero v1.15.0 h1:b/YBCLWAJdFWJTN9cLhiXXcD7mzKn9Dm86dNnfyQw1I=
30+
github.com/spf13/afero v1.15.0/go.mod h1:NC2ByUVxtQs4b3sIUphxK0NioZnmxgyCrfzeuq8lxMg=
31+
github.com/spf13/cast v1.10.0 h1:h2x0u2shc1QuLHfxi+cTJvs30+ZAHOGRic8uyGTDWxY=
32+
github.com/spf13/cast v1.10.0/go.mod h1:jNfB8QC9IA6ZuY2ZjDp0KtFO2LZZlg4S/7bzP6qqeHo=
33+
github.com/spf13/cobra v1.10.2 h1:DMTTonx5m65Ic0GOoRY2c16WCbHxOOw6xxezuLaBpcU=
34+
github.com/spf13/cobra v1.10.2/go.mod h1:7C1pvHqHw5A4vrJfjNwvOdzYu0Gml16OCs2GRiTUUS4=
35+
github.com/spf13/pflag v1.0.9/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg=
36+
github.com/spf13/pflag v1.0.10 h1:4EBh2KAYBwaONj6b2Ye1GiHfwjqyROoF4RwYO+vPwFk=
37+
github.com/spf13/pflag v1.0.10/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg=
38+
github.com/spf13/viper v1.21.0 h1:x5S+0EU27Lbphp4UKm1C+1oQO+rKx36vfCoaVebLFSU=
39+
github.com/spf13/viper v1.21.0/go.mod h1:P0lhsswPGWD/1lZJ9ny3fYnVqxiegrlNrEmgLjbTCAY=
40+
github.com/stretchr/testify v1.11.1 h1:7s2iGBzp5EwR7/aIZr8ao5+dra3wiQyKjjFuvgVKu7U=
41+
github.com/stretchr/testify v1.11.1/go.mod h1:wZwfW3scLgRK+23gO65QZefKpKQRnfz6sD981Nm4B6U=
42+
github.com/subosito/gotenv v1.6.0 h1:9NlTDc1FTs4qu0DDq7AEtTPNw6SVm7uBMsUCUjABIf8=
43+
github.com/subosito/gotenv v1.6.0/go.mod h1:Dk4QP5c2W3ibzajGcXpNraDfq2IrhjMIvMSWPKKo0FU=
44+
go.yaml.in/yaml/v3 v3.0.4 h1:tfq32ie2Jv2UxXFdLJdh3jXuOzWiL1fo0bu/FbuKpbc=
45+
go.yaml.in/yaml/v3 v3.0.4/go.mod h1:DhzuOOF2ATzADvBadXxruRBLzYTpT36CKvDb3+aBEFg=
46+
golang.org/x/sys v0.29.0 h1:TPYlXGxvx1MGTn2GiZDhnjPA9wZzZeGKHHmKhHYvgaU=
47+
golang.org/x/sys v0.29.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
48+
golang.org/x/text v0.28.0 h1:rhazDwis8INMIwQ4tpjLDzUhx6RlXqZNPEM0huQojng=
49+
golang.org/x/text v0.28.0/go.mod h1:U8nCwOR8jO/marOQ0QbDiOngZVEBB7MAiitBuMjXiNU=
50+
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
51+
gopkg.in/check.v1 v1.0.0-20190902080502-41f04d3bba15 h1:YR8cESwS4TdDjEe65xsg0ogRM/Nc3DYOhEAlW+xobZo=
52+
gopkg.in/check.v1 v1.0.0-20190902080502-41f04d3bba15/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
53+
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
54+
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=

0 commit comments

Comments
 (0)