Skip to content

Commit f91dd18

Browse files
committed
feat: complete cross-cli prompt improvement platform
1 parent 771b99f commit f91dd18

42 files changed

Lines changed: 2226 additions & 150 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

.github/workflows/ci.yml

Lines changed: 53 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@ name: CI
22

33
on:
44
push:
5-
branches: [ master ]
5+
branches: [ master, main ]
66
pull_request:
77

88
jobs:
@@ -35,3 +35,55 @@ jobs:
3535
run: |
3636
chmod +x node_modules/.bin/vitest 2>/dev/null || true
3737
node_modules/.bin/vitest run --exclude '**/correlation.test.ts'
38+
39+
acceptance:
40+
runs-on: ubuntu-latest
41+
defaults:
42+
run:
43+
working-directory: universal-refiner
44+
strategy:
45+
fail-fast: false
46+
matrix:
47+
model-order:
48+
- primary
49+
- reversed
50+
steps:
51+
- uses: actions/checkout@v4
52+
- uses: actions/setup-node@v4
53+
with:
54+
node-version: '22'
55+
cache: npm
56+
cache-dependency-path: universal-refiner/package-lock.json
57+
- run: npm ci --no-fund
58+
- run: npm rebuild better-sqlite3
59+
- run: npm run build
60+
- name: Run all-tool and provider acceptance
61+
run: npm run test:acceptance
62+
- name: Run fake-model semantic acceptance
63+
env:
64+
PROMPT_REFINER_PRIMARY_MODEL: ${{ matrix.model-order == 'primary' && 'gemma3:12b' || 'gemma3:1b' }}
65+
PROMPT_REFINER_FALLBACK_MODEL: ${{ matrix.model-order == 'primary' && 'gemma3:1b' || 'gemma3:12b' }}
66+
run: npm run acceptance:semantic
67+
68+
stress:
69+
runs-on: ubuntu-latest
70+
defaults:
71+
run:
72+
working-directory: universal-refiner
73+
steps:
74+
- uses: actions/checkout@v4
75+
- uses: actions/setup-node@v4
76+
with:
77+
node-version: '22'
78+
cache: npm
79+
cache-dependency-path: universal-refiner/package-lock.json
80+
- run: npm ci --no-fund
81+
- run: npm rebuild better-sqlite3
82+
- run: npm run build
83+
- name: Run restart and in-process concurrency tests
84+
run: npm run test:stress
85+
- name: Run multi-process EventStore stress
86+
env:
87+
PROMPT_REFINER_STRESS_WORKERS: '4'
88+
PROMPT_REFINER_STRESS_WRITES: '100'
89+
run: npm run stress:event-store

README.md

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -79,6 +79,8 @@ cd Promptimprover
7979

8080
Both installers perform a deterministic dependency install, run the full test suite, build the package, install it globally, and verify the `gemini-prompt-refiner` command. Add that command to your MCP client configuration. See the [Setup Guide](https://github.com/Coding-Autopilot-System/Promptimprover/wiki/Setup-Guide) for full configuration instructions.
8181

82+
For optional automatic pre-prompt linting and post-execution recording, see the [cross-CLI automation guide](./docs/cross-cli-automation.md). Claude Code and Gemini CLI expose the required lifecycle hooks. Codex currently requires MCP-first instructions or explicit helper invocation because its hook lifecycle does not transparently intercept each prompt.
83+
8284
## Local Semantic Model
8385

8486
PromptImprover uses a local OpenAI-compatible endpoint before optional MCP sampling. The safe defaults target `http://localhost:9000/v1`, use `gemma3:12b` first, and fall back to `gemma3:1b`. If neither local model nor MCP sampling is available, rule-based refinement continues without semantic output.

docs/cross-cli-automation.md

Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,32 @@
1+
# Cross-CLI Automation
2+
3+
PromptImprover ships fail-open pre-prompt and post-execution helpers:
4+
5+
- `promptimprover-hook-pre` makes one latency-safe rule-based `lint_prompt` call, creates a trackable prompt ID, and injects advisory context. Interactive MCP linting continues to use semantic providers by default.
6+
- `promptimprover-hook-post` records privacy-safe completion metadata with `record_agent_output`.
7+
8+
Both commands read hook JSON from stdin, write JSON only to stdout, report failures to stderr, and always allow the client to continue. They start the same built MCP server used by `gemini-prompt-refiner`. Set `PROMPTIMPROVER_SERVER_PATH` only when testing a nonstandard build.
9+
10+
The helpers store only prompt ID, client name, and creation time in the OS temporary directory. They do not persist prompt or response bodies. Completion records contain output length rather than response text.
11+
12+
## Claude Code
13+
14+
Claude Code supports `UserPromptSubmit` and `Stop`, so both phases can run transparently. Merge [`claude.settings.fragment.json`](../universal-refiner/hooks/config/claude.settings.fragment.json) into the desired user or project settings file after installing the package globally.
15+
16+
## Gemini CLI
17+
18+
Gemini CLI supports `BeforeAgent` and `AfterAgent`, so both phases can run transparently. Merge [`gemini.settings.fragment.json`](../universal-refiner/hooks/config/gemini.settings.fragment.json) into the desired user or project settings file after installing the package globally.
19+
20+
## Codex CLI
21+
22+
Codex CLI `0.138.0` has a stable hook system, but its exposed lifecycle currently does not provide transparent per-prompt pre/post hooks. Do not claim that `SessionStart` performs prompt interception.
23+
24+
Keep PromptImprover registered as an MCP server and use repo instructions that require `lint_prompt` and `record_agent_output`. External automation can pipe normalized JSON into the same helpers; see [`codex.config.fragment.toml`](../universal-refiner/hooks/config/codex.config.fragment.toml).
25+
26+
## Failure And Privacy Behavior
27+
28+
- MCP startup, timeout, parsing, and tool errors fail open.
29+
- Default timeout is 15 seconds; set `PROMPTIMPROVER_HOOK_TIMEOUT_MS` to a positive millisecond value to change it.
30+
- Hook stdout remains strict JSON.
31+
- No credentials or environment values are read or logged.
32+
- Prompt and response text are not written to hook state or completion summaries.
Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,26 @@
1+
{
2+
"hooks": {
3+
"UserPromptSubmit": [
4+
{
5+
"hooks": [
6+
{
7+
"type": "command",
8+
"command": "promptimprover-hook-pre",
9+
"timeout": 20
10+
}
11+
]
12+
}
13+
],
14+
"Stop": [
15+
{
16+
"hooks": [
17+
{
18+
"type": "command",
19+
"command": "promptimprover-hook-post",
20+
"timeout": 20
21+
}
22+
]
23+
}
24+
]
25+
}
26+
}
Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
# Codex CLI 0.138.0 does not expose per-prompt pre/post lifecycle hooks.
2+
# Keep PromptImprover registered as an MCP server and invoke lint_prompt and
3+
# record_agent_output through repo instructions or explicit tool calls.
4+
#
5+
# The generic helpers can be used by external Codex automation:
6+
# '{"client":"codex","session_id":"...","cwd":"...","prompt":"..."}' | promptimprover-hook-pre
7+
# '{"client":"codex","session_id":"...","cwd":"...","output":"..."}' | promptimprover-hook-post
Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
{
2+
"hooks": {
3+
"BeforeAgent": [
4+
{
5+
"hooks": [
6+
{
7+
"name": "promptimprover-pre-prompt",
8+
"type": "command",
9+
"command": "promptimprover-hook-pre",
10+
"timeout": 20
11+
}
12+
]
13+
}
14+
],
15+
"AfterAgent": [
16+
{
17+
"hooks": [
18+
{
19+
"name": "promptimprover-post-execution",
20+
"type": "command",
21+
"command": "promptimprover-hook-post",
22+
"timeout": 20
23+
}
24+
]
25+
}
26+
]
27+
}
28+
}

universal-refiner/hooks/hooks.json

Lines changed: 15 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -4,9 +4,22 @@
44
{
55
"hooks": [
66
{
7-
"name": "refine-prompt",
7+
"name": "promptimprover-pre-prompt",
88
"type": "command",
9-
"command": "node C:/repo/Promptimprover/gemini-extension/dist/hooks/refine_hook.js"
9+
"command": "promptimprover-hook-pre",
10+
"timeout": 20
11+
}
12+
]
13+
}
14+
],
15+
"AfterAgent": [
16+
{
17+
"hooks": [
18+
{
19+
"name": "promptimprover-post-execution",
20+
"type": "command",
21+
"command": "promptimprover-hook-post",
22+
"timeout": 20
1023
}
1124
]
1225
}
Lines changed: 180 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,180 @@
1+
import { createHash } from "crypto";
2+
import * as fs from "fs";
3+
import * as os from "os";
4+
import * as path from "path";
5+
6+
export type HookInput = Record<string, unknown>;
7+
8+
export interface HookState {
9+
promptId: string;
10+
client: string;
11+
createdAt: string;
12+
}
13+
14+
export interface McpToolCaller {
15+
(name: string, args: Record<string, unknown>): Promise<string>;
16+
}
17+
18+
const STATE_MAX_AGE_MS = 24 * 60 * 60 * 1000;
19+
20+
export function parseHookInput(raw: string): HookInput {
21+
const normalized = raw.replace(/^\uFEFF/, "").trim();
22+
return normalized ? JSON.parse(normalized) as HookInput : {};
23+
}
24+
25+
export function detectClient(input: HookInput): string {
26+
const explicit = stringField(input, "client");
27+
if (explicit) return explicit.toLowerCase();
28+
29+
const event = stringField(input, "hook_event_name");
30+
if (event === "UserPromptSubmit" || event === "Stop") return "claude";
31+
if (event === "BeforeAgent" || event === "AfterAgent") return "gemini";
32+
return "generic";
33+
}
34+
35+
export function extractPrompt(input: HookInput): string | undefined {
36+
return firstString(input, ["prompt", "user_prompt", "input"]);
37+
}
38+
39+
export function extractPromptId(text: string): string | undefined {
40+
const tagged = text.match(/\[PROMPT_ID:\s*([^\]\s]+)\]/)?.[1];
41+
if (tagged) return tagged;
42+
try {
43+
const parsed = JSON.parse(text) as { promptId?: unknown };
44+
return typeof parsed.promptId === "string" ? parsed.promptId : undefined;
45+
} catch {
46+
return undefined;
47+
}
48+
}
49+
50+
export function extractOutputLength(input: HookInput): number {
51+
return firstString(input, [
52+
"prompt_response",
53+
"last_assistant_message",
54+
"response",
55+
"output",
56+
])?.length ?? 0;
57+
}
58+
59+
export function buildLintContext(lintText: string, promptId?: string): string {
60+
let gaps: Array<{ message?: unknown; suggestedAction?: unknown }> = [];
61+
try {
62+
const parsed = JSON.parse(lintText) as { gaps?: Array<{ message?: unknown; suggestedAction?: unknown }> };
63+
gaps = Array.isArray(parsed.gaps) ? parsed.gaps : [];
64+
} catch {
65+
return promptId
66+
? `PromptImprover tracked this turn as ${promptId}. Continue normally.`
67+
: "PromptImprover linting completed. Continue normally.";
68+
}
69+
70+
const lines = gaps.slice(0, 5).map((gap) => {
71+
const message = typeof gap.message === "string" ? gap.message : "Prompt quality gap detected.";
72+
const action = typeof gap.suggestedAction === "string" ? ` ${gap.suggestedAction}` : "";
73+
return `- ${message}${action}`;
74+
});
75+
76+
const tracking = promptId ? ` Tracking ID: ${promptId}.` : "";
77+
if (lines.length === 0) return `PromptImprover found no actionable prompt gaps.${tracking}`;
78+
return `PromptImprover found advisory gaps.${tracking}\n${lines.join("\n")}`;
79+
}
80+
81+
export function allowOutput(input: HookInput, additionalContext?: string): Record<string, unknown> {
82+
const event = stringField(input, "hook_event_name");
83+
if (!additionalContext) return { decision: "allow" };
84+
85+
return {
86+
decision: "allow",
87+
hookSpecificOutput: {
88+
...(event ? { hookEventName: event } : {}),
89+
additionalContext,
90+
},
91+
};
92+
}
93+
94+
export function statePath(input: HookInput): string {
95+
const key = [
96+
detectClient(input),
97+
stringField(input, "session_id") ?? stringField(input, "sessionId") ?? "no-session",
98+
stringField(input, "cwd") ?? process.cwd(),
99+
].join("|");
100+
const hash = createHash("sha256").update(key).digest("hex");
101+
return path.join(os.tmpdir(), "promptimprover-hooks", `${hash}.json`);
102+
}
103+
104+
export function saveState(input: HookInput, state: HookState): void {
105+
const file = statePath(input);
106+
fs.mkdirSync(path.dirname(file), { recursive: true });
107+
fs.writeFileSync(file, JSON.stringify(state), { encoding: "utf8", mode: 0o600 });
108+
}
109+
110+
export function loadState(input: HookInput): HookState | undefined {
111+
const file = statePath(input);
112+
try {
113+
const state = JSON.parse(fs.readFileSync(file, "utf8")) as HookState;
114+
if (!state.promptId || Date.now() - Date.parse(state.createdAt) > STATE_MAX_AGE_MS) {
115+
fs.rmSync(file, { force: true });
116+
return undefined;
117+
}
118+
return state;
119+
} catch {
120+
return undefined;
121+
}
122+
}
123+
124+
export function clearState(input: HookInput): void {
125+
fs.rmSync(statePath(input), { force: true });
126+
}
127+
128+
export async function runPrePrompt(input: HookInput, callTool: McpToolCaller): Promise<Record<string, unknown>> {
129+
const prompt = extractPrompt(input);
130+
if (!prompt?.trim()) return allowOutput(input);
131+
132+
try {
133+
const lintText = await callTool("lint_prompt", { prompt, semantic: false });
134+
const promptId = extractPromptId(lintText);
135+
if (promptId) {
136+
saveState(input, {
137+
promptId,
138+
client: detectClient(input),
139+
createdAt: new Date().toISOString(),
140+
});
141+
}
142+
return allowOutput(input, buildLintContext(lintText, promptId));
143+
} catch {
144+
return allowOutput(input);
145+
}
146+
}
147+
148+
export async function runPostExecution(input: HookInput, callTool: McpToolCaller): Promise<Record<string, unknown>> {
149+
const state = loadState(input);
150+
const promptId = stringField(input, "prompt_id") ?? state?.promptId;
151+
if (!promptId) return allowOutput(input);
152+
153+
const client = state?.client ?? detectClient(input);
154+
const outputLength = extractOutputLength(input);
155+
await callTool("record_agent_output", {
156+
prompt_id: promptId,
157+
output_summary: `${client} completed the tracked turn; output_length=${outputLength}.`,
158+
artifacts_json: JSON.stringify({
159+
client,
160+
hook_event: stringField(input, "hook_event_name") ?? "manual",
161+
output_length: outputLength,
162+
}),
163+
status: stringField(input, "status") === "failed" ? "failed" : "completed",
164+
});
165+
clearState(input);
166+
return allowOutput(input);
167+
}
168+
169+
function firstString(input: HookInput, fields: string[]): string | undefined {
170+
for (const field of fields) {
171+
const value = stringField(input, field);
172+
if (value) return value;
173+
}
174+
return undefined;
175+
}
176+
177+
function stringField(input: HookInput, field: string): string | undefined {
178+
const value = input[field];
179+
return typeof value === "string" ? value : undefined;
180+
}

0 commit comments

Comments
 (0)