Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions DICTIONARY.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# SkillGym Dictionary
# Skillgym Dictionary

This is the domain dictionary for `skillgym`.

Expand All @@ -8,7 +8,7 @@ Before adding or changing a term, check this file and ask the user for approval.

## Terms

- `SkillGym`: The product, package, CLI, and repository.
- `Skillgym`: The product, package, CLI, and repository.
- `suite file`: Module file that exports one suite.
- `suite`: User-authored collection of cases and optional workspace config.
- `case`: One definition with a prompt, assertions, and optional metadata.
Expand All @@ -26,7 +26,7 @@ Before adding or changing a term, check this file and ask the user for approval.
- `failure classification`: Stable category assigned to a failure.
- `failure class`: The value representing a failure classification.
- `result`: Pass/fail outcome object; not a session report.
- `artifact`: Preserved file or directory written by SkillGym.
- `artifact`: Preserved file or directory written by Skillgym.
- `artifact directory`: Directory holding artifacts for one scope such as a suite run, execution, repetition, retry, or session.
- `workspace`: Directory where the agent runs.
- `no workspace`: Run directly in an existing working directory with no provisioning.
Expand Down
8 changes: 4 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ bun add --dev skillgym
Create `skillgym.config.ts` in your project root, or in a parent directory that the suite can discover upward:

```ts
import type { SkillGymConfig } from "skillgym";
import type { SkillgymConfig } from "skillgym";

const config = {
run: {
Expand Down Expand Up @@ -67,7 +67,7 @@ const config = {
},
},
},
} satisfies SkillGymConfig;
} satisfies SkillgymConfig;

export default config;
```
Expand Down Expand Up @@ -179,13 +179,13 @@ The execution unit is one case x runner pair. `skillgym` expands the suite into

`serial` is the default. `parallel` maximizes overlap across the full matrix up to the configured cap. `isolated-by-runner` is a middle ground when you want each runner to stay ordered internally but still allow different runners to overlap.

For concurrent schedules, `run.maxParallel` defaults to `os.availableParallelism()`. This limits how many SkillGym executions are active at once; it does not pin or limit CPU cores used by an individual agent process.
For concurrent schedules, `run.maxParallel` defaults to `os.availableParallelism()`. This limits how many Skillgym executions are active at once; it does not pin or limit CPU cores used by an individual agent process.

Concurrent schedules do not copy or isolate the workspace by themselves. Overlapping executions may still interact through the same filesystem state and live runner output unless you use isolated workspaces. OpenCode, Codex, and Claude Code runtime state are isolated per execution under each artifact directory.

`run.repeat` is useful when you want stability sampling instead of a single lucky pass. Each case x runner execution keeps running until it records the requested number of successful classified results, or stops early when one repetition still fails after exhausting `run.repeatFailure` retries.

`run.repeatFailure` retries only the current repetition when it still counts as failed after result classification. SkillGym keeps all repetition and retry artifacts, averages visible duration and token metrics across the final results of completed successful repetitions, and preserves the full nested detail in `results.json`.
`run.repeatFailure` retries only the current repetition when it still counts as failed after result classification. Skillgym keeps all repetition and retry artifacts, averages visible duration and token metrics across the final results of completed successful repetitions, and preserves the full nested detail in `results.json`.

Artifacts for repeated executions are grouped under the stable case x runner directory using `repeat-N` directories, with retry sessions nested as `session-N` inside the repetition that needed recovery.

Expand Down
2 changes: 1 addition & 1 deletion docs/assertions.md
Original file line number Diff line number Diff line change
Expand Up @@ -48,7 +48,7 @@ Rules:

## Soft assertions

`assert.soft` mirrors the sync assertion methods on the root `assert` export and the grouped SkillGym helpers.
`assert.soft` mirrors the sync assertion methods on the root `assert` export and the grouped Skillgym helpers.

- soft failures are collected in execution order
- the runner throws a single `AssertionError` after `case_.assert(report, ctx)` completes
Expand Down
4 changes: 2 additions & 2 deletions docs/workspaces.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@
- `shared`: create one workspace per suite run and reuse it across executions
- `isolated`: create a fresh workspace per case x runner execution

Use `none` when the agent should run in an existing directory. Use `shared` or `isolated` when the suite needs SkillGym to provision a workspace first.
Use `none` when the agent should run in an existing directory. Use `shared` or `isolated` when the suite needs Skillgym to provision a workspace first.

## Suite-level workspace config

Expand Down Expand Up @@ -61,7 +61,7 @@ Behavior:
- if omitted, none mode falls back to config `run.cwd`, then `process.cwd()`
- `templateDir` and `bootstrap` are not allowed

If a suite file does not export `workspace`, SkillGym falls back to config `run.workspace`, then to implicit `none` mode.
If a suite file does not export `workspace`, Skillgym falls back to config `run.workspace`, then to implicit `none` mode.

## Shared mode

Expand Down
4 changes: 3 additions & 1 deletion skillgym.config.js → skillgym.config.ts
Original file line number Diff line number Diff line change
@@ -1,3 +1,5 @@
import { SkillgymConfig } from "./src/index";

const config = {
run: {
cwd: ".",
Expand Down Expand Up @@ -34,6 +36,6 @@ const config = {
},
},
},
};
} satisfies SkillgymConfig;

export default config;
4 changes: 2 additions & 2 deletions src/assertions/assert.ts
Original file line number Diff line number Diff line change
Expand Up @@ -6,9 +6,9 @@ import { outputAssertions } from "./output.js";
import { skillAssertions } from "./skills.js";
import { softAssert } from "./soft.js";
import { toolCallAssertions } from "./tool-calls.js";
import type { SkillGymAssert } from "./types.js";
import type { SkillgymAssert } from "./types.js";

export const assert: SkillGymAssert = Object.assign(nodeAssert, {
export const assert: SkillgymAssert = Object.assign(nodeAssert, {
classify<T>(failureClass: FailureClassInput, callback: () => T): T {
try {
return callback();
Expand Down
4 changes: 2 additions & 2 deletions src/assertions/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -13,8 +13,8 @@ export type {
OutputAssertions,
SkillAssertionOptions,
SkillAssertions,
SkillGymAssert,
SkillGymSoftAssert,
SkillgymAssert,
SkillgymSoftAssert,
SkillConfidence,
StructuredCommandMatcher,
ToolCallAssertions,
Expand Down
4 changes: 2 additions & 2 deletions src/assertions/soft.ts
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ import type {
FileReadAssertions,
OutputAssertions,
SkillAssertions,
SkillGymSoftAssert,
SkillgymSoftAssert,
ToolCallAssertions,
} from "./types.js";

Expand Down Expand Up @@ -45,7 +45,7 @@ const softAssertionGroups = {
output: OutputAssertions;
};

export const softAssert: SkillGymSoftAssert = Object.assign(
export const softAssert: SkillgymSoftAssert = Object.assign(
createSoftNodeAssert(),
softAssertionGroups,
);
Expand Down
6 changes: 3 additions & 3 deletions src/assertions/types.ts
Original file line number Diff line number Diff line change
Expand Up @@ -162,16 +162,16 @@ export interface AssertionClassifier {
<T>(failureClass: string | FailureClass, callback: () => T): T;
}

export type SkillGymSoftAssert = typeof nodeAssert & {
export type SkillgymSoftAssert = typeof nodeAssert & {
skills: SkillAssertions;
commands: CommandAssertions;
fileReads: FileReadAssertions;
toolCalls: ToolCallAssertions;
output: OutputAssertions;
};

export type SkillGymAssert = typeof nodeAssert & {
soft: SkillGymSoftAssert;
export type SkillgymAssert = typeof nodeAssert & {
soft: SkillgymSoftAssert;
skills: SkillAssertions;
commands: CommandAssertions;
fileReads: FileReadAssertions;
Expand Down
26 changes: 13 additions & 13 deletions src/config.ts
Original file line number Diff line number Diff line change
Expand Up @@ -58,7 +58,7 @@ export interface SnapshotConfig {
tolerance: SnapshotToleranceConfig;
}

export interface SkillGymConfig {
export interface SkillgymConfig {
run?: {
cwd?: string;
outputDir?: string;
Expand All @@ -79,15 +79,15 @@ export interface SkillGymConfig {
snapshots?: SnapshotConfig;
}

export interface LoadedSkillGymConfig {
export interface LoadedSkillgymConfig {
filePath?: string;
config: SkillGymConfig;
config: SkillgymConfig;
}

export async function loadConfig(options: {
suitePath: string;
configPath?: string;
}): Promise<LoadedSkillGymConfig> {
}): Promise<LoadedSkillgymConfig> {
const filePath =
options.configPath !== undefined
? path.resolve(options.configPath)
Expand All @@ -109,7 +109,7 @@ export async function loadConfig(options: {
};
}

export function parseConfig(raw: unknown): SkillGymConfig {
export function parseConfig(raw: unknown): SkillgymConfig {
const record = parseObject(raw, undefined);
ensureKnownKeys(record, TOP_LEVEL_KEYS, undefined);

Expand All @@ -132,7 +132,7 @@ export function resolveRunOptions(
retryFailed?: string;
tags?: string[];
},
config: SkillGymConfig,
config: SkillgymConfig,
): {
cwd: string;
outputDir?: string;
Expand Down Expand Up @@ -184,7 +184,7 @@ export function resolveReporterOptions(
reporter?: string;
cwd?: string;
},
loadedConfig: LoadedSkillGymConfig,
loadedConfig: LoadedSkillgymConfig,
): {
reporter?: string;
cwd: string;
Expand All @@ -204,7 +204,7 @@ export function resolveReporterOptions(

export function getCaseExecutionOptions(
case_: { timeoutMs?: number },
config: SkillGymConfig,
config: SkillgymConfig,
): {
timeoutMs: number;
} {
Expand Down Expand Up @@ -257,7 +257,7 @@ function readConfigModule(imported: unknown): unknown {
return imported;
}

function resolveConfigPaths(config: SkillGymConfig, configDir: string): SkillGymConfig {
function resolveConfigPaths(config: SkillgymConfig, configDir: string): SkillgymConfig {
return {
run:
config.run === undefined
Expand Down Expand Up @@ -368,7 +368,7 @@ function looksPathLike(value: string): boolean {
);
}

function parseRunConfig(value: unknown, configPath: string): SkillGymConfig["run"] | undefined {
function parseRunConfig(value: unknown, configPath: string): SkillgymConfig["run"] | undefined {
if (value === undefined) {
return undefined;
}
Expand Down Expand Up @@ -519,7 +519,7 @@ function parseScheduleMode(value: unknown, configPath: string): ScheduleMode {
function parseDefaultsConfig(
value: unknown,
configPath: string,
): SkillGymConfig["defaults"] | undefined {
): SkillgymConfig["defaults"] | undefined {
if (value === undefined) {
return undefined;
}
Expand All @@ -537,7 +537,7 @@ function parseDefaultsConfig(
function parseSnapshotConfig(
value: unknown,
configPath: string,
): SkillGymConfig["snapshots"] | undefined {
): SkillgymConfig["snapshots"] | undefined {
if (value === undefined) {
return undefined;
}
Expand All @@ -554,7 +554,7 @@ function parseSnapshotConfig(
};
}

function parseRunnersConfig(value: unknown, configPath: string): SkillGymConfig["runners"] {
function parseRunnersConfig(value: unknown, configPath: string): SkillgymConfig["runners"] {
const record = parseObject(value, configPath);

if (Object.keys(record).length === 0) {
Expand Down
6 changes: 3 additions & 3 deletions src/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ export type {
RunnerSummary,
SuiteRunResult,
} from "./domain/result.js";
export type { SkillGymConfig } from "./config.js";
export type { SkillgymConfig } from "./config.js";
export { loadConfig, parseConfig } from "./config.js";
export type {
BenchmarkReporter,
Expand Down Expand Up @@ -61,8 +61,8 @@ export type {
OutputAssertions,
SkillAssertionOptions,
SkillAssertions,
SkillGymAssert,
SkillGymSoftAssert,
SkillgymAssert,
SkillgymSoftAssert,
SkillConfidence,
StructuredCommandMatcher,
ToolCallAssertions,
Expand Down
2 changes: 1 addition & 1 deletion src/reporters/github-actions.ts
Original file line number Diff line number Diff line change
Expand Up @@ -113,7 +113,7 @@ function formatJobSummary(result: SuiteRunResult): string {
const totalExecutions = countTotalExecutions(result.cases);
const failures = listFailures(result);
const lines = [
"## SkillGym Summary",
"## Skillgym Summary",
"",
`- Suite: \`${result.suitePath}\``,
`- Cases: ${passedCases} passed, ${result.cases.length - passedCases} failed`,
Expand Down
6 changes: 3 additions & 3 deletions test/public-api.test.ts
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
import { describe, expectTypeOf, it } from "vitest";
import type { Case, SkillGymConfig, Suite, TestCase, TestSuite } from "../src/index.js";
import type { Case, SkillgymConfig, Suite, TestCase, TestSuite } from "../src/index.js";

describe("public API compatibility", () => {
it("exports TestCase as an alias of Case", () => {
Expand All @@ -10,8 +10,8 @@ describe("public API compatibility", () => {
expectTypeOf<TestSuite>().toEqualTypeOf<Suite>();
});

it("exports SkillGymConfig from the root public API", () => {
expectTypeOf<SkillGymConfig>().toMatchTypeOf<{
it("exports SkillgymConfig from the root public API", () => {
expectTypeOf<SkillgymConfig>().toMatchTypeOf<{
runners: Record<string, { agent: { type: string; model: string } }>;
}>();
});
Expand Down
2 changes: 1 addition & 1 deletion test/reporters/github-actions.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -93,7 +93,7 @@ test("github-actions reporter writes a job summary when GITHUB_STEP_SUMMARY is s
});

const summary = await readFile(summaryPath, "utf8");
expect(summary).toContain("## SkillGym Summary");
expect(summary).toContain("## Skillgym Summary");
expect(summary).toContain("- Suite: `examples/basic-suite.ts`");
expect(summary).toContain("- Cases: 0 passed, 1 failed");
expect(summary).toContain("- Executions: 0 passed, 1 failed");
Expand Down
4 changes: 2 additions & 2 deletions website/src/pages/index.astro
Original file line number Diff line number Diff line change
Expand Up @@ -57,9 +57,9 @@ export const workspace = {

const configCode = `\
// skillgym.config.ts
import type { SkillGymConfig } from "skillgym";
import type { SkillgymConfig } from "skillgym";

const config: SkillGymConfig = {
const config: SkillgymConfig = {
runners: {
"my-agent": {
agent: {
Expand Down