Skip to content

Commit 00c7b01

Browse files
senamakelclaude
andauthored
fix(skills): debug infrastructure + disconnect credential cleanup (tinyhumansai#154)
* feat(debug): add skills debug script and E2E tests - Introduced a new script `debug-skill.sh` for running end-to-end tests on skills, allowing users to easily test specific skills with customizable parameters. - Added comprehensive integration tests in `skills_debug_e2e.rs` to validate the full lifecycle of skills, including discovery, starting, tool listing, and execution. - Enhanced logging and error handling in the tests to improve observability and debugging capabilities. These additions facilitate better testing and debugging of skills, improving the overall development workflow. * feat(tests): add end-to-end tests for Skills RPC over HTTP JSON-RPC - Introduced a new test file `skills_rpc_e2e.rs` to validate the full stack of skill operations via HTTP JSON-RPC. - Implemented comprehensive tests covering skill discovery, starting, tool listing, and execution, ensuring robust functionality. - Enhanced logging for better observability during test execution, facilitating easier debugging and validation of skill interactions. These tests improve the reliability and maintainability of the skills framework by ensuring all critical operations are thoroughly validated. * refactor(tests): update RPC method names in end-to-end tests for skills - Changed RPC method names in `skills_rpc_e2e.rs` to use the new `openhuman` prefix, reflecting the updated API structure. - Updated corresponding test assertions to ensure consistency with the new method names. - Enhanced logging messages to align with the new method naming conventions, improving clarity during test execution. These changes ensure that the end-to-end tests accurately reflect the current API and improve maintainability. * feat(debug): add live debugging script and corresponding tests for Notion skill - Introduced `debug-notion-live.sh` script to facilitate debugging of the Notion skill with a live backend, including health checks and OAuth proxy testing. - Added `skills_notion_live.rs` test file to validate the Notion skill's functionality using real data and backend interactions. - Enhanced logging and error handling in both the script and tests to improve observability and debugging capabilities. These additions streamline the debugging process and ensure the Notion skill operates correctly with live data. * feat(env): enhance environment configuration for debugging scripts - Updated `.env.example` to include a new `JWT_TOKEN` variable for session management in debugging scripts. - Modified `debug-notion-live.sh` and `debug-skill.sh` scripts to load environment variables from `.env`, improving flexibility and usability. - Enhanced error handling in the scripts to ensure required variables are set, providing clearer feedback during execution. These changes streamline the debugging process for skills by ensuring necessary configurations are easily managed and accessible. * feat(tests): add disconnect flow test for skills - Introduced a new end-to-end test `skill_disconnect_flow` to validate the disconnect process for skills, mirroring the expected frontend behavior. - The test covers the stopping of a skill, handling OAuth credentials, and verifying cleanup after a disconnect. - Enhanced logging throughout the test to improve observability and debugging capabilities. These additions ensure that the disconnect flow is properly validated, improving the reliability of skill interactions. * fix(skills): revoke OAuth credentials on skill disconnect disconnectSkill() was only stopping the skill and resetting setup_complete, leaving oauth_credential.json on disk. On restart the stale credential would be restored, causing confusing auth state. Now sends oauth/revoked RPC before stopping so the event loop deletes the credential file and clears memory. Also adds revokeOAuth() and disableSkill() to the skills RPC API layer. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * style: apply cargo fmt to skill debug tests Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * refactor(tests): improve skills directory discovery and error handling - Renamed `find_skills_dir` to `try_find_skills_dir`, returning an `Option<PathBuf>` to handle cases where the skills directory is not found. - Introduced a macro `require_skills_dir!` to simplify the usage of skills directory discovery in tests, providing clearer error messages when the directory is unavailable. - Updated multiple test functions to utilize the new macro, enhancing readability and maintainability of the test code. These changes improve the robustness of the skills directory discovery process and streamline the test setup. * refactor(tests): enhance skills directory discovery with improved error handling - Renamed `find_skills_dir` to `try_find_skills_dir`, returning an `Option<PathBuf>` to better handle cases where the skills directory is not found. - Introduced a new macro `require_skills_dir!` to streamline the usage of skills directory discovery in tests, providing clearer error messages when the directory is unavailable. - Updated test functions to utilize the new macro, improving code readability and maintainability. These changes enhance the robustness of the skills directory discovery process and simplify test setup. * fix(tests): skip skill tests gracefully when skills dir unavailable Tests that require the openhuman-skills repo now return early with a SKIPPED message instead of panicking when the directory is not found. Fixes CI failures where the skills repo is not checked out. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(skills): harden disconnect flow, test assertions, and secret redaction - disconnectSkill: read stored credentialId from snapshot and pass it to oauth/revoked for correct memory bucket cleanup; add host-side fallback to delete oauth_credential.json when the runtime is already stopped. - revokeOAuth: make integrationId required (no more "default" fabrication); add removePersistedOAuthCredential helper for host-side cleanup. - skills_debug_e2e: hard-assert oauth_credential.json is deleted after oauth/revoked instead of soft logging. - skills_notion_live: gate behind RUN_LIVE_NOTION=1; require all env vars (BACKEND_URL, JWT_TOKEN, CREDENTIAL_ID, SKILLS_DATA_DIR); redact JWT and credential file contents from logs. - skills_rpc_e2e: check_result renamed to assert_rpc_ok and now panics on JSON-RPC errors so protocol regressions fail fast. - debug-notion-live.sh: capture cargo exit code separately from grep/head to avoid spurious failures under set -euo pipefail. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * style: apply cargo fmt to skills_notion_live.rs Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
1 parent a193240 commit 00c7b01

8 files changed

Lines changed: 1611 additions & 2 deletions

File tree

.env.example

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,14 @@ BACKEND_URL=https://staging-api.alphahuman.xyz
1212
# [required] Also read by Vite frontend (VITE_ prefix required for browser exposure)
1313
VITE_BACKEND_URL=https://staging-api.alphahuman.xyz
1414

15+
# ---------------------------------------------------------------------------
16+
# Authentication (for skills OAuth proxy and debug scripts)
17+
# ---------------------------------------------------------------------------
18+
# [optional] Session JWT — used by QuickJS skills sandbox for oauth.fetch proxy calls.
19+
# Also used by debug scripts (scripts/debug-skill.sh, scripts/debug-notion-live.sh).
20+
# Get from login flow or browser devtools.
21+
JWT_TOKEN=
22+
1523
# ---------------------------------------------------------------------------
1624
# Core process
1725
# ---------------------------------------------------------------------------

app/src/lib/skills/manager.ts

Lines changed: 50 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,12 @@
88
import { callCoreRpc } from "../../services/coreRpcClient";
99
import { SkillRuntime } from "./runtime";
1010
import { emitSkillStateChange } from "./skillEvents";
11-
import { setSetupComplete as rpcSetSetupComplete } from "./skillsApi";
11+
import {
12+
getSkillSnapshot,
13+
setSetupComplete as rpcSetSetupComplete,
14+
revokeOAuth as rpcRevokeOAuth,
15+
removePersistedOAuthCredential,
16+
} from "./skillsApi";
1217
import { syncToolsToBackend } from "./sync";
1318
import type {
1419
SkillManifest,
@@ -340,10 +345,53 @@ class SkillManager {
340345
}
341346

342347
/**
343-
* Disconnect a skill — stop it and reset setup state.
348+
* Disconnect a skill — revoke OAuth credentials, stop it, and reset setup state.
344349
*/
345350
async disconnectSkill(skillId: string): Promise<void> {
351+
// Read the stored credential ID so oauth/revoked clears the right memory bucket.
352+
let credentialId: string | undefined;
353+
try {
354+
const snap = await getSkillSnapshot(skillId);
355+
const cred = snap?.state?.__oauth_credential as
356+
| { credentialId?: string }
357+
| string
358+
| undefined;
359+
if (cred && typeof cred === "object") {
360+
credentialId = cred.credentialId;
361+
}
362+
} catch {
363+
// Snapshot may fail if skill isn't registered yet
364+
}
365+
366+
// Revoke OAuth credential before stopping so the running skill can clean up
367+
// its in-memory state and the event loop deletes oauth_credential.json.
368+
let revokeSucceeded = false;
369+
if (credentialId) {
370+
try {
371+
await rpcRevokeOAuth(skillId, credentialId);
372+
revokeSucceeded = true;
373+
} catch (err) {
374+
console.debug(
375+
"[SkillManager] oauth/revoked failed (runtime may be stopped):",
376+
err,
377+
);
378+
}
379+
}
380+
346381
await this.stopSkill(skillId);
382+
383+
// Host-side fallback: if the RPC couldn't reach the runtime (already stopped,
384+
// or non-OAuth skill), delete the persisted credential file so it isn't
385+
// restored on next start.
386+
if (!revokeSucceeded) {
387+
await removePersistedOAuthCredential(skillId).catch((err) => {
388+
console.debug(
389+
"[SkillManager] host-side credential cleanup failed:",
390+
err,
391+
);
392+
});
393+
}
394+
347395
await rpcSetSetupComplete(skillId, false).catch(() => {});
348396
emitSkillStateChange(skillId);
349397
syncToolsToBackend();

app/src/lib/skills/skillsApi.ts

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -118,6 +118,35 @@ export async function setSetupComplete(skillId: string, complete: boolean): Prom
118118
});
119119
}
120120

121+
export async function revokeOAuth(skillId: string, integrationId: string): Promise<void> {
122+
await callCoreRpc({
123+
method: 'openhuman.skills_rpc',
124+
params: {
125+
skill_id: skillId,
126+
method: 'oauth/revoked',
127+
params: { integrationId },
128+
},
129+
});
130+
}
131+
132+
/**
133+
* Host-side fallback: delete oauth_credential.json from the skill's data dir.
134+
* Used when the runtime is already stopped so oauth/revoked RPC can't reach it.
135+
*/
136+
export async function removePersistedOAuthCredential(skillId: string): Promise<void> {
137+
await callCoreRpc({
138+
method: 'openhuman.skills_data_write',
139+
params: { skill_id: skillId, filename: 'oauth_credential.json', content: '' },
140+
});
141+
}
142+
143+
export async function disableSkill(skillId: string): Promise<void> {
144+
await callCoreRpc({
145+
method: 'openhuman.skills_disable',
146+
params: { skill_id: skillId },
147+
});
148+
}
149+
121150
export async function fetchRegistryFresh(): Promise<void> {
122151
await callCoreRpc({
123152
method: 'openhuman.skills_registry_fetch',

scripts/debug-notion-live.sh

Lines changed: 137 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,137 @@
1+
#!/usr/bin/env bash
2+
#
3+
# debug-notion-live.sh — Debug Notion skill with a live backend + JWT.
4+
#
5+
# Loads environment from .env (BACKEND_URL, JWT_TOKEN, etc.)
6+
#
7+
# Tests the full OAuth proxy chain that the Notion skill uses:
8+
# 1. Raw HTTP call to backend proxy endpoint
9+
# 2. Skill startup with BACKEND_URL + session token
10+
# 3. Tool call that uses oauth.fetch (proxied through backend)
11+
#
12+
# Usage:
13+
# bash scripts/debug-notion-live.sh
14+
#
15+
# Environment variables (set in .env or override via export):
16+
# BACKEND_URL — staging or prod backend
17+
# JWT_TOKEN — session JWT
18+
# CREDENTIAL_ID — Notion OAuth credential ID
19+
#
20+
set -euo pipefail
21+
22+
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
23+
REPO_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)"
24+
25+
# Load .env
26+
if [ -f "$REPO_ROOT/.env" ]; then
27+
source "$SCRIPT_DIR/load-dotenv.sh" "$REPO_ROOT/.env"
28+
fi
29+
30+
BACKEND_URL="${BACKEND_URL:-}"
31+
JWT_TOKEN="${JWT_TOKEN:-}"
32+
CREDENTIAL_ID="${CREDENTIAL_ID:-}"
33+
34+
# Read credential ID from oauth_credential.json if not set
35+
if [ -z "$CREDENTIAL_ID" ]; then
36+
CRED_FILE="$HOME/.openhuman/skills_data/notion/oauth_credential.json"
37+
if [ -f "$CRED_FILE" ]; then
38+
CREDENTIAL_ID=$(python3 -c "import json; print(json.load(open('$CRED_FILE')).get('credentialId',''))" 2>/dev/null || echo "")
39+
fi
40+
fi
41+
42+
if [ -z "$BACKEND_URL" ]; then
43+
echo "ERROR: BACKEND_URL not set. Add it to .env or export it."
44+
exit 1
45+
fi
46+
47+
if [ -z "$JWT_TOKEN" ]; then
48+
echo "ERROR: JWT_TOKEN not set. Add it to .env or export it."
49+
exit 1
50+
fi
51+
52+
echo "╔════════════════════════════════════════════════════════╗"
53+
echo "║ Notion Skill Live Debug ║"
54+
echo "╠════════════════════════════════════════════════════════╣"
55+
echo "║ Backend: $BACKEND_URL"
56+
echo "║ Credential ID: ${CREDENTIAL_ID:-<not found>}"
57+
echo "║ JWT: ${JWT_TOKEN:0:20}..."
58+
echo "╚════════════════════════════════════════════════════════╝"
59+
echo ""
60+
61+
# ── Step 1: Check backend health ──
62+
echo "--- Step 1: Backend Health Check ---"
63+
HTTP_CODE=$(curl -s -o /dev/null -w "%{http_code}" "$BACKEND_URL/settings" -H "Authorization: Bearer $JWT_TOKEN" 2>/dev/null || echo "000")
64+
echo " GET /settings → HTTP $HTTP_CODE"
65+
66+
if [ "$HTTP_CODE" = "000" ] || [ "$HTTP_CODE" = "502" ] || [ "$HTTP_CODE" = "503" ]; then
67+
echo " ✗ Backend is DOWN (HTTP $HTTP_CODE)"
68+
echo ""
69+
echo " The backend at $BACKEND_URL is unreachable."
70+
echo " The Notion skill uses oauth.fetch() which proxies through:"
71+
echo " $BACKEND_URL/proxy/by-id/$CREDENTIAL_ID/{path}"
72+
echo ""
73+
echo " Fix: Bring the backend online, then re-run this script."
74+
exit 1
75+
fi
76+
77+
if [ "$HTTP_CODE" = "401" ]; then
78+
echo " ✗ JWT is invalid or expired (HTTP 401)"
79+
echo " Get a fresh JWT and set JWT_TOKEN in .env"
80+
exit 1
81+
fi
82+
83+
echo " ✓ Backend reachable (HTTP $HTTP_CODE)"
84+
85+
# ── Step 2: Raw proxy call ──
86+
if [ -n "$CREDENTIAL_ID" ]; then
87+
echo ""
88+
echo "--- Step 2: Raw OAuth Proxy Call ---"
89+
echo " Testing: GET $BACKEND_URL/proxy/by-id/$CREDENTIAL_ID/v1/users?page_size=1"
90+
PROXY_RESP=$(curl -s -w "\n__HTTP_CODE__:%{http_code}" \
91+
"$BACKEND_URL/proxy/by-id/$CREDENTIAL_ID/v1/users?page_size=1" \
92+
-H "Authorization: Bearer $JWT_TOKEN" \
93+
-H "Content-Type: application/json" 2>/dev/null || echo "__HTTP_CODE__:000")
94+
95+
PROXY_BODY=$(echo "$PROXY_RESP" | sed '/__HTTP_CODE__/d')
96+
PROXY_CODE=$(echo "$PROXY_RESP" | grep "__HTTP_CODE__" | cut -d: -f2)
97+
98+
echo " HTTP $PROXY_CODE"
99+
if [ "$PROXY_CODE" = "200" ]; then
100+
echo " ✓ Notion API accessible via proxy"
101+
echo " Response: ${PROXY_BODY:0:200}..."
102+
else
103+
echo " ✗ Proxy returned HTTP $PROXY_CODE"
104+
echo " Response: $PROXY_BODY"
105+
fi
106+
else
107+
echo ""
108+
echo "--- Step 2: SKIPPED (no CREDENTIAL_ID) ---"
109+
fi
110+
111+
# ── Step 3: Test via Rust runtime ──
112+
echo ""
113+
echo "--- Step 3: Skill Runtime Test (with live backend) ---"
114+
115+
export SKILL_DEBUG_ID=notion
116+
export SKILL_DEBUG_TOOL=sync-status
117+
export RUST_LOG="${RUST_LOG:-info}"
118+
119+
STEP3_OUT=$(cargo test --test skills_debug_e2e skill_full_lifecycle -- --nocapture 2>&1) || true
120+
STEP3_RC=${PIPESTATUS[0]:-$?}
121+
echo "$STEP3_OUT" | grep -E "(✓|✗|·|---|====|Text:|Result:)" | head -40 || true
122+
if [ "$STEP3_RC" -ne 0 ]; then
123+
echo " ✗ cargo test exited with code $STEP3_RC"
124+
fi
125+
126+
echo ""
127+
echo "--- Step 4: Notion Live Test (real data dir) ---"
128+
echo ""
129+
STEP4_OUT=$(RUN_LIVE_NOTION=1 cargo test --test skills_notion_live -- --nocapture 2>&1) || true
130+
STEP4_RC=${PIPESTATUS[0]:-$?}
131+
echo "$STEP4_OUT" | grep -E "(✓|✗|---|Step|Backend|OAuth|HTTP|status|connected|workspace|totals|Result:|is_error|Done|COMPLETE)" | head -30 || true
132+
if [ "$STEP4_RC" -ne 0 ]; then
133+
echo " ✗ cargo test exited with code $STEP4_RC"
134+
fi
135+
136+
echo ""
137+
echo "=== Done ==="

scripts/debug-skill.sh

Lines changed: 67 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,67 @@
1+
#!/usr/bin/env bash
2+
#
3+
# debug-skill.sh — Run the skills debug E2E test against a real skill.
4+
#
5+
# Loads environment from .env (BACKEND_URL, JWT_TOKEN, etc.)
6+
#
7+
# Usage:
8+
# bash scripts/debug-skill.sh # test example-skill (auto-find dir)
9+
# bash scripts/debug-skill.sh gmail # test a specific skill
10+
# bash scripts/debug-skill.sh gmail /path/to/skills # explicit skills dir
11+
# bash scripts/debug-skill.sh gmail "" get-emails '{"query":"test"}'
12+
#
13+
# Environment variables (set in .env or override via export):
14+
# BACKEND_URL — backend API URL
15+
# JWT_TOKEN — session JWT for OAuth proxy
16+
# SKILL_DEBUG_ID — skill ID (default: example-skill)
17+
# SKILL_DEBUG_DIR — path to skills dir containing skill folders
18+
# SKILL_DEBUG_TOOL — tool name to call (default: first tool)
19+
# SKILL_DEBUG_TOOL_ARGS — JSON args for the tool (default: "{}")
20+
# SKILL_DEBUG_VERBOSE — "1" for verbose logging
21+
# RUST_LOG — Rust log filter (default: info)
22+
#
23+
set -euo pipefail
24+
25+
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
26+
REPO_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)"
27+
28+
# Load .env (won't overwrite vars already set in the shell)
29+
if [ -f "$REPO_ROOT/.env" ]; then
30+
source "$SCRIPT_DIR/load-dotenv.sh" "$REPO_ROOT/.env"
31+
fi
32+
33+
# Parse positional args
34+
SKILL_ID="${1:-${SKILL_DEBUG_ID:-example-skill}}"
35+
SKILLS_DIR="${2:-${SKILL_DEBUG_DIR:-}}"
36+
TOOL_NAME="${3:-${SKILL_DEBUG_TOOL:-}}"
37+
TOOL_ARGS="${4:-${SKILL_DEBUG_TOOL_ARGS:-}}"
38+
39+
export SKILL_DEBUG_ID="$SKILL_ID"
40+
[ -n "$SKILLS_DIR" ] && export SKILL_DEBUG_DIR="$SKILLS_DIR"
41+
[ -n "$TOOL_NAME" ] && export SKILL_DEBUG_TOOL="$TOOL_NAME"
42+
[ -n "$TOOL_ARGS" ] && export SKILL_DEBUG_TOOL_ARGS="$TOOL_ARGS"
43+
44+
# Default log level
45+
export RUST_LOG="${RUST_LOG:-info}"
46+
47+
echo "╔══════════════════════════════════════════════════════╗"
48+
echo "║ Skills Debug Runner ║"
49+
echo "╠══════════════════════════════════════════════════════╣"
50+
echo "║ Skill: $SKILL_ID"
51+
echo "║ Skills dir: ${SKILL_DEBUG_DIR:-<auto-detect>}"
52+
echo "║ Tool: ${SKILL_DEBUG_TOOL:-<first available>}"
53+
echo "║ Tool args: ${SKILL_DEBUG_TOOL_ARGS:-{}}"
54+
echo "║ BACKEND_URL: ${BACKEND_URL:-<not set>}"
55+
echo "║ JWT_TOKEN: ${JWT_TOKEN:+${JWT_TOKEN:0:20}...}"
56+
echo "║ RUST_LOG: $RUST_LOG"
57+
echo "╚══════════════════════════════════════════════════════╝"
58+
echo ""
59+
60+
cd "$REPO_ROOT"
61+
62+
# Run just the full lifecycle test by default, with output
63+
cargo test --test skills_debug_e2e skill_full_lifecycle -- --nocapture 2>&1
64+
65+
echo ""
66+
echo "Done. To run all skill tests (including edge cases):"
67+
echo " cargo test --test skills_debug_e2e -- --nocapture"

0 commit comments

Comments
 (0)