Skip to content

Commit 38d2bd1

Browse files
jonstjohnjonstjohn
authored
Export table statistics (#5)
* feat: export table statistics
1 parent b8a152e commit 38d2bd1

5 files changed

Lines changed: 298 additions & 23 deletions

File tree

‎.claude/rules/development.md‎

Lines changed: 273 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,273 @@
1+
# Development Tasks & Workflows
2+
3+
## Common Development Tasks
4+
5+
### Adding a New Export Table
6+
7+
1. Add a `Table` entry to `exportTables` in `pkg/export/exporter.go`:
8+
```go
9+
Table{Database: "system", Name: "my_table", TimeColumn: "created_at"},
10+
```
11+
- Set `TimeColumn` to the timestamp column if time-range filtering is needed, or `""` for no filtering.
12+
- Set `Optional: true` if the table may not exist in all cluster configurations (e.g. Cloud virtual clusters).
13+
- Use `Database: ""` with a dotted `Name` (e.g. `"crdb_internal.table_indexes"`) to query across all databases.
14+
15+
2. Add or update a test in `pkg/export/exporter_test.go`.
16+
17+
3. Build and verify:
18+
```bash
19+
go build -o workload-exporter .
20+
./workload-exporter export --help
21+
```
22+
23+
### Adding a New CLI Command
24+
25+
1. Create a new file in `cmd/` (e.g. `cmd/mycommand.go`).
26+
2. Register the command on the root command in `cmd/root.go`.
27+
3. Implement any supporting logic in `pkg/`.
28+
4. Add tests.
29+
5. Update `--help` text and docs as needed.
30+
31+
### Adding a New Dependency
32+
33+
```bash
34+
go get github.com/some/package
35+
go mod tidy
36+
```
37+
38+
## Build & Verification
39+
40+
```bash
41+
# Standard build
42+
go build -o workload-exporter .
43+
44+
# Build with version info (matches CI/release process)
45+
go build -ldflags="-X main.Version=vX.Y.Z" -o workload-exporter .
46+
47+
# Verify binary works
48+
./workload-exporter --version
49+
./workload-exporter export --help
50+
./workload-exporter update --help
51+
```
52+
53+
## Testing
54+
55+
```bash
56+
# Run all unit tests
57+
go test ./...
58+
59+
# Run with race detector
60+
go test -race ./...
61+
62+
# Run specific package
63+
go test ./pkg/export/...
64+
go test ./pkg/update/...
65+
66+
# Verbose output
67+
go test -v ./pkg/export/...
68+
69+
# Short flag (skips long-running tests if tagged)
70+
go test ./... -short
71+
```
72+
73+
Integration tests require a live CockroachDB cluster. See `docs/TESTING.md`.
74+
75+
## Code Quality
76+
77+
```bash
78+
# Static analysis
79+
go vet ./...
80+
81+
# Format code
82+
go fmt ./...
83+
84+
# Linter (project config: .golangci.yml)
85+
golangci-lint run --timeout=5m
86+
```
87+
88+
## Making Changes
89+
90+
### Standard Workflow
91+
92+
1. **Create feature branch** (optional but recommended):
93+
```bash
94+
git checkout -b feat/my-feature
95+
```
96+
97+
2. **Make changes** in `pkg/` and/or `cmd/`.
98+
99+
3. **Check for errors**:
100+
```bash
101+
go vet ./...
102+
go test ./...
103+
```
104+
105+
4. **Build and verify**:
106+
```bash
107+
go build -o workload-exporter .
108+
./workload-exporter --version
109+
```
110+
111+
5. **Commit**:
112+
```bash
113+
git add .
114+
git commit -m "feat: description of changes"
115+
```
116+
117+
## Feature Completion Checklist
118+
119+
Before marking a feature as complete, verify all items below.
120+
121+
### 1. Code Quality ✅
122+
123+
```bash
124+
go vet ./...
125+
golangci-lint run --timeout=5m
126+
go fmt ./...
127+
```
128+
129+
### 2. Testing ✅
130+
131+
```bash
132+
# Unit tests
133+
go test ./...
134+
135+
# Race detector
136+
go test -race ./...
137+
138+
# Integration tests (downloads CockroachDB binaries on first run — takes several minutes)
139+
go test -tags=integration -v ./pkg/export/
140+
```
141+
142+
### 3. Permissions Update ✅
143+
144+
**IMPORTANT**: Update `.claude/settings.local.json` if you:
145+
- Added new bash commands Claude needs to run
146+
- Added new tool invocations
147+
- Added new file paths that need reading
148+
149+
**After updating, always prompt the user to test**:
150+
```
151+
I've updated .claude/settings.local.json with new permissions.
152+
Please test the following command to verify it works:
153+
[example command here]
154+
```
155+
156+
**Common permission patterns**:
157+
```json
158+
"Bash(go test ./pkg/mynewpkg/...)" // New package tests
159+
"Bash(./workload-exporter mycommand:*)" // New CLI command
160+
"WebFetch(domain:example.com)" // New web domain
161+
```
162+
163+
### 4. Documentation ✅
164+
165+
- `docs/DEVELOPMENT.md` — update if workflows changed
166+
- `docs/TESTING.md` — update if testing approach changed
167+
- `.claude/rules/development.md` — update if development patterns changed
168+
- Code comments — document complex or non-obvious logic
169+
- CLI `--help` text — update if commands or flags changed
170+
- `README.md` — update if user-facing behaviour changed
171+
172+
### 5. Build Verification ✅
173+
174+
```bash
175+
go build -o workload-exporter .
176+
./workload-exporter --version
177+
./workload-exporter export --help
178+
```
179+
180+
### 6. Git Hygiene ✅
181+
182+
```bash
183+
# Review what's changing
184+
git status
185+
git diff
186+
187+
# Check for debug leftovers
188+
grep -r "fmt.Println\|log.Print" pkg/ cmd/
189+
190+
# Check for sensitive data
191+
grep -r "PASSWORD\|SECRET\|API_KEY" .
192+
```
193+
194+
- No commented-out code blocks
195+
- Commit message follows `type: description` format (`feat:`, `fix:`, `docs:`, `refactor:`)
196+
197+
### 7. User Communication ✅
198+
199+
After completing a feature, provide:
200+
1. **Summary** of what was implemented
201+
2. **Example commands** showing how to use it
202+
3. **Testing results** from your verification
203+
4. **Documentation locations** that were updated
204+
5. **Any caveats or limitations** to be aware of
205+
206+
## Feature Completion Template
207+
208+
```markdown
209+
## [Feature Name] - Completion Report
210+
211+
### ✅ Code Quality
212+
- [ ] go vet passes
213+
- [ ] golangci-lint passes
214+
- [ ] go fmt applied
215+
216+
### ✅ Testing
217+
- [ ] Unit tests pass (`go test ./...`)
218+
- [ ] Race detector clean (`go test -race ./...`)
219+
- [ ] Integration tests pass (`go test -tags=integration -v ./pkg/export/`)
220+
221+
### ✅ Permissions
222+
- [ ] .claude/settings.local.json updated (if needed)
223+
- [ ] User prompted to test new permissions
224+
225+
### ✅ Documentation
226+
- [ ] docs/ updated (if workflows changed)
227+
- [ ] .claude/rules/ updated (if patterns changed)
228+
- [ ] Code comments added
229+
- [ ] --help text updated
230+
231+
### ✅ Build
232+
- [ ] Clean build successful
233+
- [ ] Binary verified working
234+
235+
### ✅ Git
236+
- [ ] No debug code
237+
- [ ] No sensitive data
238+
- [ ] Meaningful commit message
239+
240+
### 📊 Summary
241+
[Brief summary of what was implemented]
242+
243+
### 🧪 Testing Results
244+
[What you tested and results]
245+
246+
### 📚 Documentation
247+
[List of files updated]
248+
249+
### ⚠️ Limitations
250+
[Any known limitations or future improvements]
251+
```
252+
253+
## Debugging
254+
255+
```bash
256+
# Enable debug logging
257+
./workload-exporter export -c "connection-string" --debug
258+
259+
# Check CI pipeline config
260+
cat .github/workflows/ci.yaml
261+
262+
# Inspect linter config
263+
cat .golangci.yml
264+
```
265+
266+
## Getting Help
267+
268+
When stuck:
269+
1. Check `.claude/rules/` documentation
270+
2. Look at similar existing code in `pkg/export/exporter.go`
271+
3. Read test files for usage examples (`pkg/export/exporter_test.go`)
272+
4. Use `--help` flag on CLI commands
273+
5. Check recent git commits for similar changes

‎README.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -204,6 +204,7 @@ The export creates a **zip file** containing the following files:
204204
- **`crdb_internal.transaction_contention_events.csv`** - Lock contention events
205205
- **`crdb_internal.gossip_nodes.csv`** - Node information and topology
206206
- **`crdb_internal.table_indexes.csv`** - Table and index descriptor IDs across all databases
207+
- **`system.table_statistics.csv`** - Optimizer table statistics (column-level stats used by the query planner)
207208

208209
*Statistics files only include data within the specified time range*
209210

‎docs/TESTING.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -110,6 +110,7 @@ Each integration test validates:
110110
- `crdb_internal.transaction_contention_events.csv`
111111
- `crdb_internal.gossip_nodes.csv`
112112
- `crdb_internal.table_indexes.csv`
113+
- `system.table_statistics.csv`
113114
- `zone_configurations.txt`
114115
- `testdb.schema.txt` (test database schema)
115116
4. ✅ CSV files have valid headers

‎pkg/export/exporter.go‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -86,6 +86,7 @@ var exportTables = []Table{
8686
Table{Database: "crdb_internal", Name: "transaction_contention_events", TimeColumn: "collection_ts"},
8787
Table{Database: "crdb_internal", Name: "gossip_nodes", TimeColumn: "", Optional: true},
8888
Table{Database: "", Name: "crdb_internal.table_indexes", TimeColumn: ""}, // Use "" to query across all databases
89+
Table{Database: "system", Name: "table_statistics", TimeColumn: ""},
8990
}
9091

9192
// NewExporter creates a new Exporter instance with the given configuration.
@@ -158,7 +159,7 @@ func (exporter *Exporter) Close() error {
158159
// - Cluster metadata (version, ID, name, organization, settings)
159160
// - Database schemas (CREATE statements for all user databases)
160161
// - Zone configurations
161-
// - Statistics tables (statement_statistics, transaction_statistics, transaction_contention_events, gossip_nodes, table_indexes across all databases)
162+
// - Statistics tables (statement_statistics, transaction_statistics, transaction_contention_events, gossip_nodes, table_indexes across all databases, system.table_statistics)
162163
//
163164
// The statistics tables are filtered by the TimeRange specified in Config.
164165
// All exported data is written to the OutputFile specified in Config.

0 commit comments

Comments
 (0)