Skip to content

Commit 79444f4

Browse files
austbaustb
authored
Merge pull request #189 from OpenVoxProject/add-dev-helper-script
Add dev helper script
2 parents 7dcb38f + 23f5178 commit 79444f4

File tree

15 files changed

+977
-29
lines changed

15 files changed

+977
-29
lines changed

‎.gitignore‎

Lines changed: 12 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -69,8 +69,8 @@ checkouts
6969
/.clj-kondo/.cache/
7070
/.eastwood
7171
/.nrepl-port
72-
/pdb.bcpkix
73-
/pdb.bcprov
72+
/uberjar.bcpkix
73+
/uberjar.bcprov
7474

7575
# locust output files and python cache
7676
*.csv
@@ -79,3 +79,13 @@ locust/load-test/__pycache__/
7979

8080
# inellij configs
8181
/.idea
82+
83+
# ovdb local config
84+
/.ovdb.conf
85+
86+
# ovdb sandboxes (default location for pdbboxes)
87+
/dev-resources/sandboxes/
88+
89+
# ovdb test output
90+
/last-integration-run.log
91+
/last-test-run.log

‎README.md‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,10 @@ PuppetDB is the fast, scalable, and reliable data warehouse for Puppet. It cache
88

99
For documentation on this product, consult the [latest documentation][docs].
1010

11+
## Development
12+
13+
For local development setup, see [dev-docs/local-dev.md](dev-docs/local-dev.md).
14+
1115
## Contributing
1216

1317
If you would like to contribute to PuppetDB, please take a look at our [contributing doc][contributing].

‎dev-docs/local-dev.md‎

Lines changed: 178 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,178 @@
1+
# Local Development Setup
2+
3+
This guide will help you set up a local development environment for OpenVoxDB.
4+
5+
## Prerequisites
6+
7+
- PostgreSQL 16+ installed
8+
- [pgbox](https://gitlab.com/pgbox-org/pgbox) on your `$PATH`
9+
- Leiningen (Clojure build tool)
10+
- Java 17+
11+
12+
## Quick Start
13+
14+
1. **Configure the helper script**
15+
16+
Copy the example config and set your PostgreSQL path:
17+
18+
```bash
19+
cp dev-resources/ovdb.conf.example .ovdb.conf
20+
```
21+
22+
Edit `.ovdb.conf` and set `pg_bin` to your PostgreSQL bin directory:
23+
24+
```bash
25+
# Mac (Homebrew)
26+
pg_bin=/opt/homebrew/opt/postgresql@17/bin
27+
28+
# Mac (Postgres.app)
29+
pg_bin=/Applications/Postgres.app/Contents/Versions/17/bin
30+
31+
# Linux (Debian/Ubuntu)
32+
pg_bin=/usr/lib/postgresql/17/bin
33+
34+
# Linux (RHEL/CentOS)
35+
pg_bin=/usr/pgsql-17/bin
36+
37+
# FreeBSD
38+
pg_bin=/usr/local/bin
39+
```
40+
41+
2. **Validate your configuration**
42+
43+
```bash
44+
./ovdb doctor
45+
```
46+
47+
3. **Initialize and start the database**
48+
49+
```bash
50+
./ovdb init
51+
```
52+
53+
This creates a default postgres sandbox along with the necessary OpenVoxDB
54+
configuration inside `dev-resources/sandboxes/tmp_pg`.
55+
56+
If you wish to store your sandboxes in a different directory that can be
57+
set with the `OVDB_SANDBOX` environment variable or in the `.ovdb.conf`
58+
file.
59+
60+
4. **Run OpenVoxDB**
61+
62+
```bash
63+
./ovdb run
64+
```
65+
66+
OpenVoxDB will be available at `http://localhost:8080`.
67+
68+
## Common Commands
69+
70+
| Command | Description |
71+
|---------|-------------|
72+
| `./ovdb init` | Initialize a new PostgreSQL sandbox |
73+
| `./ovdb run` | Run OpenVoxDB |
74+
| `./ovdb start` | Start the PostgreSQL server |
75+
| `./ovdb stop` | Stop the PostgreSQL server |
76+
| `./ovdb test` | Run unit tests |
77+
| `./ovdb integration` | Run integration tests |
78+
| `./ovdb repl` | Start a Clojure REPL |
79+
| `./ovdb psql` | Connect to the database with psql |
80+
| `./ovdb doctor` | Validate configuration |
81+
82+
Run `./ovdb --help` for the full list of commands.
83+
84+
## Multiple Sandboxes
85+
86+
You can create multiple PostgreSQL sandboxes for different purposes:
87+
88+
```bash
89+
# Create a sandbox named "pg-18-test"
90+
./ovdb --name pg-18-test --pgver 18 init
91+
92+
# Run against that sandbox
93+
./ovdb --name pg-18-test run
94+
95+
# Run tests against it
96+
./ovdb --name pg-18-test test
97+
```
98+
99+
## Running Tests
100+
101+
### Unit Tests
102+
103+
Clojure unit tests that verify individual functions and components. These run
104+
against your PostgreSQL sandbox and test the core logic without requiring
105+
other openvox projects.
106+
107+
```bash
108+
./ovdb test
109+
```
110+
111+
To run a specific test:
112+
113+
```bash
114+
./ovdb test :only puppetlabs.puppetdb.scf.migrate-test/some-test
115+
```
116+
117+
### Integration Tests
118+
119+
Tests that verify OpenVoxDB works correctly with OpenVox and OpenVox-Server.
120+
These tests clone and configure the OpenVox and OpenVox-Server repositories,
121+
then run scenarios that exercise the full command submission and query
122+
pipeline.
123+
124+
```bash
125+
./ovdb integration
126+
```
127+
128+
### External Tests
129+
130+
Black-box tests that run against a built uberjar. These verify the packaged
131+
application behaves correctly, including CLI argument handling, database
132+
migrations, error handling (like OOM and schema mismatches), and upgrade paths.
133+
134+
```bash
135+
lein uberjar
136+
./ovdb ext
137+
```
138+
139+
## Populating Test Data
140+
141+
Use the benchmark command to populate the database with test data:
142+
143+
```bash
144+
# Add 10 nodes (default)
145+
./ovdb benchmark
146+
147+
# Add 100 nodes
148+
./ovdb benchmark -- 100
149+
```
150+
151+
## Configuration Reference
152+
153+
Configuration is read from (in order of precedence):
154+
155+
1. Command-line flags (`--pgver`, `--port`, etc.)
156+
2. Environment variables (`OVDB_PG_BIN`, `OVDB_SANDBOX`, etc.)
157+
3. Config file (`.ovdb.conf` in the repo root)
158+
4. Built-in defaults
159+
160+
See `dev-resources/ovdb.conf.example` for all options.
161+
162+
## Troubleshooting
163+
164+
### "pg_bin is not configured"
165+
166+
Set the `pg_bin` variable in `.ovdb.conf` to point to your PostgreSQL installation's bin directory.
167+
168+
### "pg_ctl not found"
169+
170+
Ensure `pg_bin` points to the directory containing `pg_ctl`, `psql`, and other PostgreSQL binaries.
171+
172+
### Database won't start after reboot
173+
174+
PostgreSQL sandboxes don't persist across reboots. Start the server with:
175+
176+
```bash
177+
./ovdb start
178+
```

‎dev-resources/ovdb.conf.example‎

Lines changed: 62 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,62 @@
1+
# OpenVoxDB Dev Helper Configuration
2+
# ===================================
3+
# Copy this file to .ovdb.conf (in the same directory as the ovdb script) and edit.
4+
#
5+
# Configuration precedence (highest wins):
6+
# 1. Command-line flags (e.g., --pgver, --port)
7+
# 2. Environment variables (e.g., OVDB_PG_BIN)
8+
# 3. Config file (.ovdb.conf)
9+
# 4. Built-in defaults
10+
#
11+
# All paths should be absolute. Use $HOME instead of ~ for shell expansion.
12+
# Lines starting with # are comments.
13+
14+
# PostgreSQL bin directory (REQUIRED)
15+
# -----------------------------------
16+
# Full path to the directory containing pg_ctl, psql, etc.
17+
#
18+
# Mac (Homebrew Intel): pg_bin=/usr/local/opt/postgresql@16/bin
19+
# Mac (Homebrew Apple): pg_bin=/opt/homebrew/opt/postgresql@16/bin
20+
# Mac (Postgres.app): pg_bin=/Applications/Postgres.app/Contents/Versions/16/bin
21+
# Linux (Debian/Ubuntu): pg_bin=/usr/lib/postgresql/16/bin
22+
# Linux (RHEL/CentOS): pg_bin=/usr/pgsql-16/bin
23+
# asdf: pg_bin=$HOME/.asdf/installs/postgres/16.2/bin
24+
# nix (shell): pg_bin=$(dirname $(which psql))
25+
#
26+
# Env var override: OVDB_PG_BIN
27+
pg_bin=
28+
29+
# Sandbox directory
30+
# -----------------
31+
# Where pdbboxes will be created. Each pdbbox gets a subdirectory here.
32+
# Default: dev-resources/sandboxes (relative to ovdb script location)
33+
#
34+
# Env var override: OVDB_SANDBOX
35+
# pg_sandbox=$HOME/sandbox/ovdb
36+
37+
# OpenVoxDB directory (optional)
38+
# ------------------------------
39+
# Full path to your openvoxdb checkout. If not set, the script will use
40+
# the current working directory when you're inside a checkout.
41+
#
42+
# Env var override: OVDB_DIR
43+
# ovdb_dir=$HOME/src/openvoxdb
44+
45+
# Parallel test state file (optional)
46+
# -----------------------------------
47+
# Path to the EDN file used by lein test-in-parallel for storing test timing.
48+
# Default: $pg_sandbox/ovdb-parallel-testing.edn
49+
#
50+
# Env var override: OVDB_PARALLEL_STATE
51+
# parallel_test_state=$HOME/ovdb-parallel-testing.edn
52+
53+
# Version-specific PostgreSQL paths (optional)
54+
# --------------------------------------------
55+
# If you work with multiple PostgreSQL versions, you can define version-specific
56+
# paths that will be used when you pass --pgver <version> to the script.
57+
#
58+
# Example:
59+
# pg_bin_14=/usr/lib/postgresql/14/bin
60+
# pg_bin_15=/usr/lib/postgresql/15/bin
61+
# pg_bin_16=/usr/lib/postgresql/16/bin
62+
# pg_bin_17=/usr/lib/postgresql/17/bin

‎documentation/CONTRIBUTING.md‎

Lines changed: 2 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -55,11 +55,8 @@ top of things.
5555

5656
### Testing
5757

58-
Before you do anything else, you may want to consider setting
59-
`PUPPET_SUPPRESS_INTERNAL_LEIN_REPOS=1` in your environment. We'll
60-
eventually make that the default, but for now that setting may help
61-
avoid delays incurred if lein tries to reach unreachable internal
62-
repositories.
58+
> **Quick setup:** For a streamlined development workflow using the `ovdb` helper
59+
> script, see the [Local Development Guide](../dev-docs/local-dev.md).
6360
6461
The easiest way to run the tests until you need to do it often is to
6562
use the built-in sandbox harness. If you just want to check some

‎ext/bin/check-command-perf‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -15,7 +15,7 @@ Usage: [PDB_JAR=JAR] PDBBOX=BOX_DIR $(basename "$0") STOCKPILE_ARCHIVE"
1515
endpoint. See info at:
1616
https://puppet.com/docs/puppetdb/latest/api/metrics/v1/mbeans.html#re-enable-metrics-v1-endpoint
1717
18-
The version of pdb will be the one run via ./pdb which defaults to
18+
The version of pdb will be the one run via ./uberjar which defaults to
1919
the uberjar in target/.
2020
2121
EOF
@@ -81,7 +81,7 @@ echo "Running PDB" 1>&2
8181
tmpdir="$(mktemp -d "test-command-performance-XXXXXX")"
8282
tmpdir="$(cd "$tmpdir" && pwd)"
8383

84-
./pdb services -c "$PDBBOX/conf.d" >"$tmpdir/log.txt" 2>&1 &
84+
./uberjar services -c "$PDBBOX/conf.d" >"$tmpdir/log.txt" 2>&1 &
8585
pdb_id=$!
8686

8787
printf "Waiting for PDB startup (log: %q)\n" "$tmpdir/log.txt" 1>&2

‎ext/bin/pdbbox-init‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
#!/bin/bash
1+
#!/usr/bin/env bash
22

33
# Creates a local PuppetDB sandbox directory
44
# Also creates a PostgreSQL sandbox inside of new PuppetDB sandbox in directory "pg"

‎ext/bin/render-pdb-schema‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -12,7 +12,7 @@ Usage: [PDB_JAR=JAR] $(basename "$0") [--pgbin PGBIN] [--pgport PGPORT] DEST"
1212
tree if they've been set via test-config, and the pgbin setting may
1313
be guessed. Otherwise, you'll need to specify them.
1414
15-
The version of pdb will be the one run via ./pdb which defaults to
15+
The version of pdb will be the one run via ./uberjar which defaults to
1616
the uberjar in target/.
1717
1818
EOF
@@ -85,7 +85,7 @@ tmpdir="$(mktemp -d "test-upgrade-and-exit-XXXXXX")"
8585
tmpdir="$(cd "$tmpdir" && pwd)"
8686
trap "$(printf 'rm -rf %q' "$tmpdir")" EXIT
8787

88-
./pdb upgrade -c "$PDBBOX/conf.d"
88+
./uberjar upgrade -c "$PDBBOX/conf.d"
8989

9090
postgresql_autodoc -t dot -u puppetdb -d puppetdb -f "$tmpdir/pdb"
9191
dot -Tsvg "$tmpdir/pdb.dot" -o "$dest"

‎ext/test/database-migration-config‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -87,7 +87,7 @@ mv "$tmpdir/pdb.ini" "$PDBBOX/conf.d/pdb.ini"
8787
for cmd in services upgrade; do
8888
rc=0
8989
# Start PuppetDB jar with config from PuppetDB sandbox
90-
./pdb "$cmd" -c "$PDBBOX/conf.d" 1>"$tmpdir/out" 2>"$tmpdir/err" || rc=$?
90+
./uberjar "$cmd" -c "$PDBBOX/conf.d" 1>"$tmpdir/out" 2>"$tmpdir/err" || rc=$?
9191
cat "$tmpdir/out" "$tmpdir/err"
9292
echo # in case the output doesn't end with a newline
9393
# This will become 109 once trapperkeeper supports custom exit statuses/

‎ext/test/oom-causes-shutdown‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -80,7 +80,7 @@ set -x
8080
rc=0
8181
# Show err and out, but save them separately
8282
PDB_TEST_ALLOCATE_AT_LEAST_MB_AT_STARTUP=4096 \
83-
./pdb services -c "$PDBBOX/conf.d" \
83+
./uberjar services -c "$PDBBOX/conf.d" \
8484
> >(tee -a "$tmpdir/pdb-out") \
8585
2> >(tee -a "$tmpdir/pdb-err") \
8686
|| rc=$?

0 commit comments

Comments
 (0)