Skip to content

Commit 5a25a4c

Browse files
firat-kayafirat-kaya
committed
Add onprem-data-verifier tool This commit introduces a new Go application, , designed to validate Looker
 on-premise backup artifacts before migration to Looker Cloud. The tool performs several checks, including: Workspace structure verification MD5 checksum integrity checks GPG key validation against the customer's LUID SQL dump analysis (version, charset, extended inserts, critical tables) CMK validation (raw and base64) Includes unit tests for various components. A file is also added to exclude IDE files.
1 parent 868203d commit 5a25a4c

22 files changed

Lines changed: 1791 additions & 0 deletions

‎.gitignore‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
.idea

‎onprem-data-verifier/.idea/vcs.xml‎

Lines changed: 6 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎onprem-data-verifier/README.md‎

Lines changed: 174 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,174 @@
1+
# On-Prem Data Verifier
2+
3+
A robust validation tool written in Go, designed to verify Looker On-Premise backup artifacts before migration to Looker
4+
Cloud (hosted).
5+
6+
## Overview
7+
8+
Migrations are sensitive operations. This tool ensures that the data provided by the customer is:
9+
10+
1. **Complete:** Verifies the presence of all required encrypted and decrypted artifacts.
11+
12+
2. **Integrity Verified:** Matches the MD5 checksums generated at the source.
13+
14+
3. **Secure:** Confirms files are encrypted for the correct **Looker Public Key** (dynamically resolved via the
15+
Customer's LUID = Looker Unique ID).
16+
17+
4. **Valid:** The SQL dump is structurally sound, performant (Extended Inserts), uses the correct Charset (utf8mb4), and
18+
is for supported MYSQL version.
19+
20+
5. **Recoverable:** Customer Master Key (CMK), used for decrypting DB contents, is correct.
21+
22+
## Prerequisites
23+
24+
The tool relies on the system's GPG installation to verify encryption keys.
25+
26+
* **Go** 1.20+ (to build)
27+
* **GnuPG (gpg)** installed and on the system PATH (ie. `apt install gpg`).
28+
* **Imported Key:** You must import the customer's specific GPG Public Key into your local keyring before running the
29+
tool.
30+
```bash
31+
# Example (This is usually done via the customer script instructions)
32+
echo "PUBLIC_KEY_BLOCK" | gpg --import
33+
```
34+
35+
## Build
36+
37+
```bash
38+
go build -o onprem-verifier main.go
39+
```
40+
41+
## Usage
42+
43+
The tool operates on a **Workspace Directory**. This is a dedicated local directory on your machine where you
44+
consolidate
45+
all the backup files related to a single customer migration. This directory must contain both the *encrypted bundle*
46+
sent by the customer and the *decrypted artifacts* you extracted from that bundle. Think of it as a staging area for all
47+
the files the verifier needs to check.
48+
49+
### Command
50+
51+
```bash
52+
./onprem-verifier \
53+
--backupDir /path/to/migration_workspace \
54+
--customerName "lookersre-instance-1" \
55+
--luid "7b973058-f7e0-49f7-9262-54e1987659bc"
56+
```
57+
58+
| Flag | Shorthand | Required | Description |
59+
|:-----------------|:----------|:---------|:--------------------------------------------------------------------------|
60+
| `--backupDir` | `-b` | **Yes** | Path to the workspace directory containing ALL exported backup files. |
61+
| `--customerName` | `-c` | **Yes** | Customer Name (e.g., `lookersre-instance-1`). Used to validate filenames. |
62+
| `--luid` | `-l` | **Yes** | Looker User ID (e.g., `7b97...`). Used to resolve the GPG Public Key. |
63+
64+
## Workspace Requirements
65+
66+
The tool enforces a strict naming convention. The directory provided via `--backupDir` must contain **exactly** the
67+
following 7 files (where `${customer}` matches the `--customerName` flag):
68+
69+
1. **Encrypted Artifacts (Source)**
70+
71+
* `${customer}_looker_db_backup.sql.gz.enc`
72+
* `${customer}_looker_fs_backup.tar.gz.enc`
73+
* `${customer}_looker_cmk_key.enc`
74+
* `${customer}_backup.md5`
75+
76+
2. **Decrypted Artifacts (Target)**
77+
78+
* `${customer}_looker_db_backup.sql.gz`
79+
* `${customer}_looker_fs_backup.tar.gz`
80+
* `${customer}_looker_cmk_key`
81+
82+
## Validation Pipeline
83+
84+
The tool executes checks in the following order. If any step fails, the process terminates.
85+
86+
### 1. Workspace Structure
87+
88+
* **Action:** Checks if all 7 required files exist in `--backupDir`.
89+
* **Goal:** Fail fast if data is missing or named incorrectly.
90+
91+
### 2. Integrity Check
92+
93+
* **Action:** Parses `${customer}_backup.md5` and verifies the hash of every file listed.
94+
* **Goal:** Ensure no file corruption occurred during transfer.
95+
96+
### 3. Security Check (Dynamic GPG)
97+
98+
* **Action:**
99+
1. Constructs the expected migration email: `looker-devops+migration-{LUID}@google.com`.
100+
2. Queries your local GPG keyring (`gpg --list-keys`) to find **ALL** Key IDs (Primary + Subkeys) associated with
101+
that email.
102+
3. Inspects the headers of `.enc` files (`gpg --list-packets`) to ensure they are encrypted for one of those valid
103+
Key IDs.
104+
* **Goal:** Prevent importing data encrypted with the wrong key.
105+
106+
### 4. Database Validation
107+
108+
* **Action:** Streams the `.sql.gz` file (single-pass scan) to analyze content without full decompression.
109+
* **Checks:**
110+
* **Looker Version:** Must match the supported version list.
111+
* **Charset:** Must be `utf8mb4` (Looker Requirement).
112+
* **Extended Inserts:** Ensures `INSERT` statements are batched (Critical for performance).
113+
* **Critical Tables:** Verifies existence of `user`, `dashboard`, `db_connection`.
114+
115+
### 5. CMK Validation
116+
117+
* **Action:** Reads the `${customer}_looker_cmk_key`.
118+
* **Check:** Validates the key is either 32 bytes (Raw) or 44 bytes (Base64).
119+
120+
### 6. FileSystem Analysis
121+
122+
* **Action:** Analyzes the `${customer}_looker_fs_backup.tar.gz` archive size and metadata.
123+
124+
## Output
125+
126+
### Console (STDOUT)
127+
128+
Clean, colored, step-by-step logs indicating progress and specific validation results.
129+
130+
```text
131+
=== Looker On-Prem Verification Pipeline ===
132+
133+
>> [1/6] Checking Workspace Structure
134+
Directory: /workspace
135+
[OK] Workspace structure verified
136+
137+
>> [2/6] Verifying MD5 Checksums
138+
[OK] All files match their checksums
139+
140+
>> [3/6] Resolving Security Keys
141+
[OK] Found Valid Key IDs: [000ECF...]
142+
[OK] sm-restore_looker_db_backup.sql.gz.enc is encrypted correctly
143+
...
144+
145+
>> [4/6] Analyzing Database: sm-restore_looker_db_backup.sql.gz
146+
[OK] Version 25.18.33 is supported
147+
[OK] Database Charset: utf8mb4
148+
[OK] Extended Inserts detected
149+
[OK] Critical tables verified: [user dashboard db_connection]
150+
151+
[SUCCESS] VERIFICATION COMPLETE
152+
Customer: sm-restore
153+
LUID: d5c8...
154+
Duration: 6.54s
155+
```
156+
157+
### Report File
158+
159+
A JSON report is generated at `metadata.json` in the current directory or specified path.
160+
161+
```json
162+
{
163+
"customer_name": "lookersre-instance-1",
164+
"instance_id": "7b973058-f7e0-49f7-9262-54e1987659bc",
165+
"generated_at": "2026-01-16T15:30:00Z",
166+
"fs_total_size_bytes": 5368709120,
167+
"db_total_size_bytes": 1073741824,
168+
"table_count": 142,
169+
"cmk_status": "Valid",
170+
"cmk_encoding": "Base64",
171+
"looker_version": "25.18.33",
172+
"duration_in_seconds": 6.54
173+
}
174+
```

‎onprem-data-verifier/cmd/root.go‎

Lines changed: 103 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,103 @@
1+
package cmd
2+
3+
import (
4+
"fmt"
5+
"os"
6+
7+
"onprem-data-verifier/logger"
8+
"onprem-data-verifier/validator"
9+
10+
"github.com/spf13/cobra"
11+
"github.com/spf13/viper"
12+
)
13+
14+
var (
15+
// Flags
16+
backupDir string
17+
customerName string
18+
luid string
19+
outputFile string
20+
21+
// RootCmd represents the base command
22+
RootCmd = &cobra.Command{
23+
Use: "onprem-data-verifier",
24+
Short: "Validates On-Premise Looker backups before migration",
25+
Long: `
26+
On-Prem Data Verifier
27+
=====================
28+
Validates integrity and schema of Looker On-Premise backups.
29+
30+
Requires a workspace directory (--backupDir) and explicit paths to critical files.
31+
32+
Example:
33+
onprem-data-verifier \
34+
--backupDir ./workspace \
35+
--customerName lookersre-scotty-1 \
36+
--luid "u-12345-6789
37+
--output metadata.json"`,
38+
39+
Run: func(cmd *cobra.Command, args []string) {
40+
// Initialize Orchestrator with the new flag variables
41+
orchestrator := validator.NewOrchestrator(backupDir, customerName, luid)
42+
43+
report, err := orchestrator.Run()
44+
if err != nil {
45+
// Use the logger to print the Red Error Block and Exit
46+
logger.Fatal(err)
47+
}
48+
49+
// Use the logger for the Green Success Block
50+
// Format the float as a string (e.g., "1.23s")
51+
logger.Completion(
52+
report.CustomerName,
53+
report.InstanceID,
54+
report.GeneratedAt,
55+
report.LookerVer,
56+
report.TableCount,
57+
report.FSTotalSizeBytes,
58+
report.DBTotalSizeBytes,
59+
report.CmkStatus,
60+
report.CmkFormat,
61+
report.DurationInSeconds,
62+
)
63+
64+
// Save the report to the output file
65+
if err := report.Save(outputFile); err != nil {
66+
logger.Warn("Could not save report: %v", err)
67+
}
68+
},
69+
}
70+
)
71+
72+
func Execute() {
73+
if err := RootCmd.Execute(); err != nil {
74+
// Changed fmt.Println to logger.Error for consistency
75+
logger.Error("%v", err)
76+
os.Exit(1)
77+
}
78+
}
79+
80+
func init() {
81+
// 1. Define Persistent Flags
82+
RootCmd.PersistentFlags().StringVarP(&backupDir, "backupDir", "b", "", "Backup Directory containing ALL backup files")
83+
RootCmd.PersistentFlags().StringVarP(&customerName, "customerName", "c", "", "Customer Name for the report")
84+
RootCmd.PersistentFlags().StringVarP(&luid, "luid", "l", "", "Looker User ID (LUID)")
85+
RootCmd.PersistentFlags().StringVarP(&outputFile, "output", "o", "metadata.json", "Output file for the report")
86+
87+
// 2. Mark ALL specific flags as REQUIRED
88+
requiredFlags := []string{"backupDir", "customerName", "luid"}
89+
for _, flag := range requiredFlags {
90+
if err := RootCmd.MarkPersistentFlagRequired(flag); err != nil {
91+
// Using fmt here is fine for initialization panics
92+
panic(fmt.Sprintf("Failed to mark flag required: %s - %v", flag, err))
93+
}
94+
}
95+
96+
// 3. Bind Viper
97+
bindFlags := []string{"backupDir", "customerName", "luid", "output"}
98+
for _, flag := range bindFlags {
99+
if err := viper.BindPFlag(flag, RootCmd.PersistentFlags().Lookup(flag)); err != nil {
100+
panic(fmt.Sprintf("Failed to bind flag to viper: %s - %v", flag, err))
101+
}
102+
}
103+
}

‎onprem-data-verifier/go.mod‎

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
module onprem-data-verifier
2+
3+
go 1.24.8
4+
5+
require (
6+
github.com/spf13/cobra v1.10.2
7+
github.com/spf13/viper v1.21.0
8+
github.com/stretchr/testify v1.11.1
9+
)
10+
11+
require (
12+
github.com/davecgh/go-spew v1.1.1 // indirect
13+
github.com/fsnotify/fsnotify v1.9.0 // indirect
14+
github.com/go-viper/mapstructure/v2 v2.4.0 // indirect
15+
github.com/inconshreveable/mousetrap v1.1.0 // indirect
16+
github.com/pelletier/go-toml/v2 v2.2.4 // indirect
17+
github.com/pmezard/go-difflib v1.0.0 // indirect
18+
github.com/sagikazarmark/locafero v0.11.0 // indirect
19+
github.com/sourcegraph/conc v0.3.1-0.20240121214520-5f936abd7ae8 // indirect
20+
github.com/spf13/afero v1.15.0 // indirect
21+
github.com/spf13/cast v1.10.0 // indirect
22+
github.com/spf13/pflag v1.0.10 // indirect
23+
github.com/subosito/gotenv v1.6.0 // indirect
24+
go.yaml.in/yaml/v3 v3.0.4 // indirect
25+
golang.org/x/sys v0.29.0 // indirect
26+
golang.org/x/text v0.28.0 // indirect
27+
gopkg.in/yaml.v3 v3.0.1 // indirect
28+
)

‎onprem-data-verifier/go.sum‎

Lines changed: 54 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,54 @@
1+
github.com/cpuguy83/go-md2man/v2 v2.0.6/go.mod h1:oOW0eioCTA6cOiMLiUPZOpcVxMig6NIQQ7OS05n1F4g=
2+
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
3+
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
4+
github.com/frankban/quicktest v1.14.6 h1:7Xjx+VpznH+oBnejlPUj8oUpdxnVs4f8XU8WnHkI4W8=
5+
github.com/frankban/quicktest v1.14.6/go.mod h1:4ptaffx2x8+WTWXmUCuVU6aPUX1/Mz7zb5vbUoiM6w0=
6+
github.com/fsnotify/fsnotify v1.9.0 h1:2Ml+OJNzbYCTzsxtv8vKSFD9PbJjmhYF14k/jKC7S9k=
7+
github.com/fsnotify/fsnotify v1.9.0/go.mod h1:8jBTzvmWwFyi3Pb8djgCCO5IBqzKJ/Jwo8TRcHyHii0=
8+
github.com/go-viper/mapstructure/v2 v2.4.0 h1:EBsztssimR/CONLSZZ04E8qAkxNYq4Qp9LvH92wZUgs=
9+
github.com/go-viper/mapstructure/v2 v2.4.0/go.mod h1:oJDH3BJKyqBA2TXFhDsKDGDTlndYOZ6rGS0BRZIxGhM=
10+
github.com/google/go-cmp v0.6.0 h1:ofyhxvXcZhMsU5ulbFiLKl/XBFqE1GSq7atu8tAmTRI=
11+
github.com/google/go-cmp v0.6.0/go.mod h1:17dUlkBOakJ0+DkrSSNjCkIjxS6bF9zb3elmeNGIjoY=
12+
github.com/inconshreveable/mousetrap v1.1.0 h1:wN+x4NVGpMsO7ErUn/mUI3vEoE6Jt13X2s0bqwp9tc8=
13+
github.com/inconshreveable/mousetrap v1.1.0/go.mod h1:vpF70FUmC8bwa3OWnCshd2FqLfsEA9PFc4w1p2J65bw=
14+
github.com/kr/pretty v0.3.1 h1:flRD4NNwYAUpkphVc1HcthR4KEIFJ65n8Mw5qdRn3LE=
15+
github.com/kr/pretty v0.3.1/go.mod h1:hoEshYVHaxMs3cyo3Yncou5ZscifuDolrwPKZanG3xk=
16+
github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY=
17+
github.com/kr/text v0.2.0/go.mod h1:eLer722TekiGuMkidMxC/pM04lWEeraHUUmBw8l2grE=
18+
github.com/pelletier/go-toml/v2 v2.2.4 h1:mye9XuhQ6gvn5h28+VilKrrPoQVanw5PMw/TB0t5Ec4=
19+
github.com/pelletier/go-toml/v2 v2.2.4/go.mod h1:2gIqNv+qfxSVS7cM2xJQKtLSTLUE9V8t9Stt+h56mCY=
20+
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
21+
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
22+
github.com/rogpeppe/go-internal v1.9.0 h1:73kH8U+JUqXU8lRuOHeVHaa/SZPifC7BkcraZVejAe8=
23+
github.com/rogpeppe/go-internal v1.9.0/go.mod h1:WtVeX8xhTBvf0smdhujwtBcq4Qrzq/fJaraNFVN+nFs=
24+
github.com/russross/blackfriday/v2 v2.1.0/go.mod h1:+Rmxgy9KzJVeS9/2gXHxylqXiyQDYRxCVz55jmeOWTM=
25+
github.com/sagikazarmark/locafero v0.11.0 h1:1iurJgmM9G3PA/I+wWYIOw/5SyBtxapeHDcg+AAIFXc=
26+
github.com/sagikazarmark/locafero v0.11.0/go.mod h1:nVIGvgyzw595SUSUE6tvCp3YYTeHs15MvlmU87WwIik=
27+
github.com/sourcegraph/conc v0.3.1-0.20240121214520-5f936abd7ae8 h1:+jumHNA0Wrelhe64i8F6HNlS8pkoyMv5sreGx2Ry5Rw=
28+
github.com/sourcegraph/conc v0.3.1-0.20240121214520-5f936abd7ae8/go.mod h1:3n1Cwaq1E1/1lhQhtRK2ts/ZwZEhjcQeJQ1RuC6Q/8U=
29+
github.com/spf13/afero v1.15.0 h1:b/YBCLWAJdFWJTN9cLhiXXcD7mzKn9Dm86dNnfyQw1I=
30+
github.com/spf13/afero v1.15.0/go.mod h1:NC2ByUVxtQs4b3sIUphxK0NioZnmxgyCrfzeuq8lxMg=
31+
github.com/spf13/cast v1.10.0 h1:h2x0u2shc1QuLHfxi+cTJvs30+ZAHOGRic8uyGTDWxY=
32+
github.com/spf13/cast v1.10.0/go.mod h1:jNfB8QC9IA6ZuY2ZjDp0KtFO2LZZlg4S/7bzP6qqeHo=
33+
github.com/spf13/cobra v1.10.2 h1:DMTTonx5m65Ic0GOoRY2c16WCbHxOOw6xxezuLaBpcU=
34+
github.com/spf13/cobra v1.10.2/go.mod h1:7C1pvHqHw5A4vrJfjNwvOdzYu0Gml16OCs2GRiTUUS4=
35+
github.com/spf13/pflag v1.0.9/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg=
36+
github.com/spf13/pflag v1.0.10 h1:4EBh2KAYBwaONj6b2Ye1GiHfwjqyROoF4RwYO+vPwFk=
37+
github.com/spf13/pflag v1.0.10/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg=
38+
github.com/spf13/viper v1.21.0 h1:x5S+0EU27Lbphp4UKm1C+1oQO+rKx36vfCoaVebLFSU=
39+
github.com/spf13/viper v1.21.0/go.mod h1:P0lhsswPGWD/1lZJ9ny3fYnVqxiegrlNrEmgLjbTCAY=
40+
github.com/stretchr/testify v1.11.1 h1:7s2iGBzp5EwR7/aIZr8ao5+dra3wiQyKjjFuvgVKu7U=
41+
github.com/stretchr/testify v1.11.1/go.mod h1:wZwfW3scLgRK+23gO65QZefKpKQRnfz6sD981Nm4B6U=
42+
github.com/subosito/gotenv v1.6.0 h1:9NlTDc1FTs4qu0DDq7AEtTPNw6SVm7uBMsUCUjABIf8=
43+
github.com/subosito/gotenv v1.6.0/go.mod h1:Dk4QP5c2W3ibzajGcXpNraDfq2IrhjMIvMSWPKKo0FU=
44+
go.yaml.in/yaml/v3 v3.0.4 h1:tfq32ie2Jv2UxXFdLJdh3jXuOzWiL1fo0bu/FbuKpbc=
45+
go.yaml.in/yaml/v3 v3.0.4/go.mod h1:DhzuOOF2ATzADvBadXxruRBLzYTpT36CKvDb3+aBEFg=
46+
golang.org/x/sys v0.29.0 h1:TPYlXGxvx1MGTn2GiZDhnjPA9wZzZeGKHHmKhHYvgaU=
47+
golang.org/x/sys v0.29.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
48+
golang.org/x/text v0.28.0 h1:rhazDwis8INMIwQ4tpjLDzUhx6RlXqZNPEM0huQojng=
49+
golang.org/x/text v0.28.0/go.mod h1:U8nCwOR8jO/marOQ0QbDiOngZVEBB7MAiitBuMjXiNU=
50+
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
51+
gopkg.in/check.v1 v1.0.0-20190902080502-41f04d3bba15 h1:YR8cESwS4TdDjEe65xsg0ogRM/Nc3DYOhEAlW+xobZo=
52+
gopkg.in/check.v1 v1.0.0-20190902080502-41f04d3bba15/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
53+
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
54+
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=

0 commit comments

Comments
 (0)