Skip to content

Commit 0226200

Browse files
anandgupta42claude
anandgupta42
and
claude
authored
test: expand Verdaccio sanity suite with 50 new tests across 3 phases (#562)
* ci: add Verdaccio sanity suite to CI and release workflows Adds the Verdaccio-based sanity suite (real `npm install -g` flow) to both CI and release pipelines: **CI (`ci.yml`):** - New `sanity-verdaccio` job on push to main - Builds linux-x64 binary + dbt-tools, runs full Docker Compose suite - Independent of other jobs (doesn't block PRs) **Release (`release.yml`):** - New `sanity-verdaccio` job between build and npm publish - Downloads linux-x64 artifact from build matrix - **Blocks `publish-npm`** — broken install flow prevents release - Dependency chain: build → sanity-verdaccio → publish-npm → github-release Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * test: expand Verdaccio sanity suite with 51 new tests across 3 phases Adds comprehensive test coverage for install integrity, resilience edge cases, and security boundaries to the sanity test suite. **Phase 1b: Extended Installation Verification (20 tests)** - Binary aliases, permissions, symlinks, zero-size check - Postinstall artifacts (skills dir, version marker, permissions) - Complete skill suites (5 dbt, 6 SQL/data), all subcommands accessible - Clean version output, invalid subcommand/flag handling - No hardcoded CI paths, no credential files, NAPI platform match - Package dir resolved dynamically (scoped + unscoped) **Phase 3b: Extended Resilience Tests (15 tests)** - DB recreation after deletion, corrupted DB with trap-based restore - Config edge cases: empty, whitespace, wrong schema, JSONC comments - Environment: spaces in XDG, unset HOME, read-only config dir - Project context: non-git dir, empty git repo, deep nesting (30 levels) - Concurrency: 5 parallel + 10 rapid sequential invocations - Session list with empty DB **Phase 5: Security Tests (16 tests)** - Credential leakage: API key not in --help/--version/error output - DB password not in connection errors (dynamic fake credentials) - Side-effect shell injection test (marker file, not argument parsing) - Path traversal, null bytes, 100KB args, SQL injection fixture - Filesystem isolation, config dir permissions, core dump prevention - No stack traces or internal paths in errors - Unicode, RTL, large SQL (5000 lines) **CI fix:** Added `OPENCODE_RELEASE=1` to sanity build step. Incorporates fixes from 6-model code review (Claude, GPT 5.2 Codex, Gemini 3.1 Pro, Kimi K2.5, MiniMax M2.5, GLM-5). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
1 parent 73852bc commit 0226200

6 files changed

Lines changed: 936 additions & 0 deletions

File tree

‎.github/workflows/ci.yml‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -360,9 +360,11 @@ jobs:
360360
run: bun install
361361

362362
- name: Build CLI binary
363+
# target-index=1 = linux-x64 (see release.yml matrix)
363364
run: bun run packages/opencode/script/build.ts --target-index=1
364365
env:
365366
OPENCODE_VERSION: 0.0.0-sanity-${{ github.sha }}
367+
OPENCODE_RELEASE: "1"
366368
MODELS_DEV_API_JSON: test/tool/fixtures/models-api.json
367369

368370
- name: Build dbt-tools

‎test/sanity/fixtures/injection-test.sql‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
-- SQL injection test patterns for security phase
2+
-- These should be caught/handled gracefully, never executed literally
3+
4+
SELECT * FROM users WHERE id = 1; DROP TABLE users; --
5+
SELECT * FROM users WHERE name = '' OR '1'='1';
6+
SELECT * FROM users WHERE id = 1 UNION SELECT username, password FROM admin_users;
7+
SELECT * FROM users WHERE name = 'Robert''); DROP TABLE students;--';
8+
SELECT LOAD_FILE('/etc/passwd');
9+
SELECT * INTO OUTFILE '/tmp/pwned.txt' FROM users;

‎test/sanity/phases/resilience-extended.sh‎

Lines changed: 306 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,306 @@
1+
#!/bin/bash
2+
# Phase 3b: Extended resilience tests — edge cases, recovery, concurrency
3+
set -uo pipefail
4+
5+
SCRIPT_DIR="$(cd "$(dirname "$0")/.." && pwd)"
6+
source "$SCRIPT_DIR/lib/assert.sh"
7+
source "$SCRIPT_DIR/lib/altimate-run.sh"
8+
9+
echo "--- Phase 3b: Extended Resilience Tests ---"
10+
11+
# Set up git repo
12+
WORKDIR=$(mktemp -d /tmp/sanity-resilience-ext-XXXXXX)
13+
cd "$WORKDIR" || { echo "FAIL: cannot cd to $WORKDIR"; exit 1; }
14+
git init -q
15+
git config user.name "sanity-test"
16+
git config user.email "sanity@test.local"
17+
echo '{}' > package.json
18+
git add -A && git commit -q -m "init"
19+
20+
# ─────────────────────────────────────────────────────────────
21+
# Database Recovery
22+
# ─────────────────────────────────────────────────────────────
23+
24+
# 1. Deleted DB is recreated on next run
25+
echo " [1/15] DB recreation after deletion..."
26+
DB_DIR="${XDG_DATA_HOME:-$HOME/.local/share}/altimate-code"
27+
if [ -n "${ANTHROPIC_API_KEY:-}" ] && [ -d "$DB_DIR" ]; then
28+
# Find and delete the DB
29+
DB_PATH=$(find "$DB_DIR" -name "opencode*.db" -not -name "*-wal" -not -name "*-shm" 2>/dev/null | head -1)
30+
if [ -n "$DB_PATH" ]; then
31+
rm -f "$DB_PATH" "${DB_PATH}-wal" "${DB_PATH}-shm"
32+
altimate_run "db-recreate" "say hello" || true
33+
NEW_DB=$(find "$DB_DIR" -name "opencode*.db" -not -name "*-wal" -not -name "*-shm" 2>/dev/null | head -1)
34+
if [ -n "$NEW_DB" ]; then
35+
echo " PASS: DB recreated after deletion"
36+
PASS_COUNT=$((PASS_COUNT + 1))
37+
else
38+
echo " FAIL: DB not recreated after deletion"
39+
FAIL_COUNT=$((FAIL_COUNT + 1))
40+
fi
41+
else
42+
skip_test "DB recreation" "no DB found to delete"
43+
fi
44+
else
45+
skip_test "DB recreation" "no API key or DB dir"
46+
fi
47+
48+
# 2. Corrupted DB handled gracefully
49+
echo " [2/15] Corrupted DB handling..."
50+
if [ -d "$DB_DIR" ]; then
51+
DB_PATH=$(find "$DB_DIR" -name "opencode*.db" -not -name "*-wal" -not -name "*-shm" 2>/dev/null | head -1)
52+
if [ -n "$DB_PATH" ]; then
53+
# Corrupt the DB by writing garbage — use trap for guaranteed restore
54+
cp "$DB_PATH" "${DB_PATH}.bak"
55+
_restore_db() { mv "${DB_PATH}.bak" "$DB_PATH" 2>/dev/null || rm -f "$DB_PATH" "${DB_PATH}-wal" "${DB_PATH}-shm"; }
56+
trap '_restore_db' EXIT
57+
echo "THIS IS NOT A SQLITE DATABASE" > "$DB_PATH"
58+
CORRUPT_OUTPUT=$(timeout 15 altimate run --max-turns 1 --yolo "hello" 2>&1 || true)
59+
if echo "$CORRUPT_OUTPUT" | grep -qi "SIGSEGV\|segfault\|core dumped"; then
60+
echo " FAIL: corrupted DB caused crash"
61+
FAIL_COUNT=$((FAIL_COUNT + 1))
62+
else
63+
echo " PASS: corrupted DB handled gracefully"
64+
PASS_COUNT=$((PASS_COUNT + 1))
65+
fi
66+
# Restore and clear trap
67+
_restore_db
68+
trap - EXIT
69+
else
70+
skip_test "Corrupted DB" "no DB found"
71+
fi
72+
else
73+
skip_test "Corrupted DB" "DB dir not found"
74+
fi
75+
76+
# ─────────────────────────────────────────────────────────────
77+
# Config Edge Cases
78+
# ─────────────────────────────────────────────────────────────
79+
80+
CONFIG_DIR="${XDG_CONFIG_HOME:-$HOME/.config}/altimate-code"
81+
mkdir -p "$CONFIG_DIR"
82+
83+
# 3. Empty config file handled
84+
echo " [3/15] Empty config file..."
85+
echo "" > "$CONFIG_DIR/opencode.json"
86+
EMPTY_OUTPUT=$(timeout 15 altimate run --max-turns 1 --yolo "hello" 2>&1 || true)
87+
if echo "$EMPTY_OUTPUT" | grep -qi "SyntaxError\|parse error\|SIGSEGV"; then
88+
echo " FAIL: empty config caused parse error or crash"
89+
FAIL_COUNT=$((FAIL_COUNT + 1))
90+
else
91+
echo " PASS: empty config handled gracefully"
92+
PASS_COUNT=$((PASS_COUNT + 1))
93+
fi
94+
rm -f "$CONFIG_DIR/opencode.json"
95+
96+
# 4. Config with only whitespace
97+
echo " [4/15] Whitespace-only config..."
98+
echo " " > "$CONFIG_DIR/opencode.json"
99+
WS_OUTPUT=$(timeout 15 altimate run --max-turns 1 --yolo "hello" 2>&1 || true)
100+
if echo "$WS_OUTPUT" | grep -qi "SyntaxError\|SIGSEGV"; then
101+
echo " FAIL: whitespace config caused error"
102+
FAIL_COUNT=$((FAIL_COUNT + 1))
103+
else
104+
echo " PASS: whitespace config handled"
105+
PASS_COUNT=$((PASS_COUNT + 1))
106+
fi
107+
rm -f "$CONFIG_DIR/opencode.json"
108+
109+
# 5. Config with valid JSON but wrong schema
110+
echo " [5/15] Wrong schema config..."
111+
echo '{"this_is_not_a_real_field": true, "another_fake": [1,2,3]}' > "$CONFIG_DIR/opencode.json"
112+
SCHEMA_OUTPUT=$(timeout 15 altimate run --max-turns 1 --yolo "hello" 2>&1 || true)
113+
if echo "$SCHEMA_OUTPUT" | grep -qi "TypeError\|Cannot read\|SIGSEGV"; then
114+
echo " FAIL: wrong schema config caused crash"
115+
FAIL_COUNT=$((FAIL_COUNT + 1))
116+
else
117+
echo " PASS: wrong schema config handled gracefully"
118+
PASS_COUNT=$((PASS_COUNT + 1))
119+
fi
120+
rm -f "$CONFIG_DIR/opencode.json"
121+
122+
# 6. Config with JSONC comments
123+
echo " [6/15] JSONC config with comments..."
124+
cat > "$CONFIG_DIR/opencode.jsonc" <<'JSONCEOF'
125+
{
126+
// This is a comment
127+
/* Block comment */
128+
"provider": {
129+
"default": "anthropic"
130+
}
131+
}
132+
JSONCEOF
133+
JSONC_OUTPUT=$(timeout 15 altimate run --max-turns 1 --yolo "hello" 2>&1 || true)
134+
if echo "$JSONC_OUTPUT" | grep -qi "SyntaxError\|parse error\|SIGSEGV"; then
135+
echo " FAIL: JSONC config not supported"
136+
FAIL_COUNT=$((FAIL_COUNT + 1))
137+
else
138+
echo " PASS: JSONC config with comments handled"
139+
PASS_COUNT=$((PASS_COUNT + 1))
140+
fi
141+
rm -f "$CONFIG_DIR/opencode.jsonc"
142+
143+
# ─────────────────────────────────────────────────────────────
144+
# Environment Variable Edge Cases
145+
# ─────────────────────────────────────────────────────────────
146+
147+
# 7. XDG directories with spaces in path
148+
echo " [7/15] XDG dirs with spaces..."
149+
SPACE_DIR="/tmp/sanity dir with spaces"
150+
mkdir -p "$SPACE_DIR"
151+
SPACE_OUTPUT=$(XDG_CONFIG_HOME="$SPACE_DIR" timeout 10 altimate --version 2>&1 || true)
152+
if echo "$SPACE_OUTPUT" | grep -qi "ENOENT\|SIGSEGV\|segfault"; then
153+
echo " FAIL: spaces in XDG path caused crash"
154+
FAIL_COUNT=$((FAIL_COUNT + 1))
155+
else
156+
echo " PASS: spaces in XDG path handled"
157+
PASS_COUNT=$((PASS_COUNT + 1))
158+
fi
159+
rm -rf "$SPACE_DIR"
160+
161+
# 8. HOME directory unset
162+
echo " [8/15] Unset HOME handling..."
163+
NOHOME_OUTPUT=$(env -u HOME timeout 10 altimate --version 2>&1 || true)
164+
if echo "$NOHOME_OUTPUT" | grep -qi "TypeError\|Cannot read\|SIGSEGV"; then
165+
echo " FAIL: unset HOME caused crash"
166+
FAIL_COUNT=$((FAIL_COUNT + 1))
167+
else
168+
echo " PASS: unset HOME handled (got: $(echo "$NOHOME_OUTPUT" | head -1))"
169+
PASS_COUNT=$((PASS_COUNT + 1))
170+
fi
171+
172+
# 9. Read-only config directory
173+
echo " [9/15] Read-only config directory..."
174+
RO_DIR=$(mktemp -d /tmp/sanity-ro-config-XXXXXX)
175+
mkdir -p "$RO_DIR/altimate-code"
176+
chmod 555 "$RO_DIR/altimate-code"
177+
RO_OUTPUT=$(XDG_CONFIG_HOME="$RO_DIR" timeout 10 altimate --version 2>&1 || true)
178+
chmod 755 "$RO_DIR/altimate-code"
179+
rm -rf "$RO_DIR"
180+
if echo "$RO_OUTPUT" | grep -qi "EACCES\|SIGSEGV\|segfault"; then
181+
echo " WARN: read-only config dir may cause issues (non-fatal for --version)"
182+
PASS_COUNT=$((PASS_COUNT + 1))
183+
else
184+
echo " PASS: read-only config dir handled gracefully"
185+
PASS_COUNT=$((PASS_COUNT + 1))
186+
fi
187+
188+
# ─────────────────────────────────────────────────────────────
189+
# Project Context Edge Cases
190+
# ─────────────────────────────────────────────────────────────
191+
192+
# 10. Non-git directory handled
193+
echo " [10/15] Non-git directory..."
194+
NOGIT_DIR=$(mktemp -d /tmp/sanity-nogit-XXXXXX)
195+
cd "$NOGIT_DIR"
196+
NOGIT_OUTPUT=$(timeout 10 altimate --version 2>&1 || true)
197+
if echo "$NOGIT_OUTPUT" | grep -qi "SIGSEGV\|segfault\|fatal"; then
198+
echo " FAIL: non-git directory caused crash"
199+
FAIL_COUNT=$((FAIL_COUNT + 1))
200+
else
201+
echo " PASS: non-git directory handled"
202+
PASS_COUNT=$((PASS_COUNT + 1))
203+
fi
204+
rm -rf "$NOGIT_DIR"
205+
cd "$WORKDIR"
206+
207+
# 11. Empty git repo (no commits)
208+
echo " [11/15] Empty git repo..."
209+
EMPTY_GIT=$(mktemp -d /tmp/sanity-emptygit-XXXXXX)
210+
cd "$EMPTY_GIT"
211+
git init -q
212+
git config user.name "test"
213+
git config user.email "test@test.local"
214+
EMPTYGIT_OUTPUT=$(timeout 10 altimate --version 2>&1 || true)
215+
if echo "$EMPTYGIT_OUTPUT" | grep -qi "SIGSEGV\|segfault"; then
216+
echo " FAIL: empty git repo caused crash"
217+
FAIL_COUNT=$((FAIL_COUNT + 1))
218+
else
219+
echo " PASS: empty git repo handled"
220+
PASS_COUNT=$((PASS_COUNT + 1))
221+
fi
222+
rm -rf "$EMPTY_GIT"
223+
cd "$WORKDIR"
224+
225+
# 12. Deeply nested project directory
226+
echo " [12/15] Deeply nested directory..."
227+
DEEP_DIR="$WORKDIR"
228+
for i in $(seq 1 30); do
229+
DEEP_DIR="$DEEP_DIR/level$i"
230+
done
231+
mkdir -p "$DEEP_DIR"
232+
cd "$DEEP_DIR"
233+
DEEP_OUTPUT=$(timeout 10 altimate --version 2>&1 || true)
234+
if echo "$DEEP_OUTPUT" | grep -qi "SIGSEGV\|ENAMETOOLONG"; then
235+
echo " FAIL: deeply nested directory caused crash"
236+
FAIL_COUNT=$((FAIL_COUNT + 1))
237+
else
238+
echo " PASS: deeply nested directory handled"
239+
PASS_COUNT=$((PASS_COUNT + 1))
240+
fi
241+
cd "$WORKDIR"
242+
243+
# ─────────────────────────────────────────────────────────────
244+
# Concurrent Access
245+
# ─────────────────────────────────────────────────────────────
246+
247+
# 13. Multiple concurrent --version calls don't interfere
248+
echo " [13/15] Concurrent --version..."
249+
PIDS=()
250+
CONCURRENT_DIR=$(mktemp -d /tmp/sanity-concurrent-XXXXXX)
251+
for i in $(seq 1 5); do
252+
altimate --version > "$CONCURRENT_DIR/v$i.txt" 2>&1 &
253+
PIDS+=($!)
254+
done
255+
CONCURRENT_FAIL=0
256+
for pid in "${PIDS[@]}"; do
257+
wait "$pid" || CONCURRENT_FAIL=$((CONCURRENT_FAIL + 1))
258+
done
259+
# All outputs should be identical
260+
FIRST=$(cat "$CONCURRENT_DIR/v1.txt" 2>/dev/null | head -1)
261+
for i in $(seq 2 5); do
262+
OTHER=$(cat "$CONCURRENT_DIR/v$i.txt" 2>/dev/null | head -1)
263+
if [ "$FIRST" != "$OTHER" ]; then
264+
CONCURRENT_FAIL=$((CONCURRENT_FAIL + 1))
265+
fi
266+
done
267+
rm -rf "$CONCURRENT_DIR"
268+
if [ "$CONCURRENT_FAIL" -eq 0 ]; then
269+
echo " PASS: 5 concurrent --version calls all succeeded with same output"
270+
PASS_COUNT=$((PASS_COUNT + 1))
271+
else
272+
echo " FAIL: concurrent --version inconsistency ($CONCURRENT_FAIL mismatches)"
273+
FAIL_COUNT=$((FAIL_COUNT + 1))
274+
fi
275+
276+
# 14. Rapid sequential invocations don't corrupt state
277+
echo " [14/15] Rapid sequential invocations..."
278+
RAPID_FAIL=0
279+
for i in $(seq 1 10); do
280+
if ! altimate --version >/dev/null 2>&1; then
281+
RAPID_FAIL=$((RAPID_FAIL + 1))
282+
fi
283+
done
284+
if [ "$RAPID_FAIL" -eq 0 ]; then
285+
echo " PASS: 10 rapid sequential calls all succeeded"
286+
PASS_COUNT=$((PASS_COUNT + 1))
287+
else
288+
echo " FAIL: $RAPID_FAIL of 10 rapid calls failed"
289+
FAIL_COUNT=$((FAIL_COUNT + 1))
290+
fi
291+
292+
# 15. Session list works even with zero sessions (after fresh install)
293+
echo " [15/15] Session list with empty DB..."
294+
SESSION_OUTPUT=$(timeout 10 altimate session list 2>&1 || true)
295+
if echo "$SESSION_OUTPUT" | grep -qi "TypeError\|Cannot read\|SIGSEGV"; then
296+
echo " FAIL: session list with empty DB crashed"
297+
FAIL_COUNT=$((FAIL_COUNT + 1))
298+
else
299+
echo " PASS: session list with empty/missing DB handled"
300+
PASS_COUNT=$((PASS_COUNT + 1))
301+
fi
302+
303+
# Cleanup
304+
rm -rf "$WORKDIR"
305+
306+
report_results "Phase 3b: Extended Resilience Tests"

0 commit comments

Comments
 (0)