Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 4 additions & 3 deletions DICTIONARY.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,10 +29,11 @@ Before adding or changing a term, check this file and ask the user for approval.
- `artifact`: Preserved file or directory written by SkillGym.
- `artifact directory`: Directory holding artifacts for one scope such as a suite run, execution, repetition, retry, or session.
- `workspace`: Directory where the agent runs.
- `shared workspace`: Run directly in one real directory.
- `no workspace`: Run directly in an existing working directory with no provisioning.
- `shared workspace`: One provisioned workspace created once per suite run and reused across executions.
- `isolated workspace`: Fresh workspace created per execution.
- `workspace template`: Directory copied into an isolated workspace before execution.
- `workspace bootstrap`: Command run in an isolated workspace before the agent starts.
- `workspace template`: Directory copied into an execution workspace before execution.
- `workspace bootstrap`: Command run in a provisioned workspace before the agent starts.
- `schedule`: Execution ordering and concurrency policy.
- `reporter`: Component rendering suite-run progress and results.
- `skill detection`: Evidence that a skill was used, with confidence and evidence.
Expand Down
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -200,7 +200,7 @@ A workspace is the directory where an execution runs.
- `shared`: run directly in one real directory
- `isolated`: create a fresh temporary workspace per case x runner execution

Use `shared` when you want the agent to work against your real project checkout. Use `isolated` when you want clean filesystem state per execution or need to prepare each execution from a template.
Use `shared` when you want the agent to work against your real project checkout. Shared mode can still seed that directory with `templateDir` and `bootstrap` before the agent starts. Use `isolated` when you want clean filesystem state per execution or need a fresh prepared workspace every time.

You can configure workspaces in `skillgym.config.*` with `run.workspace`, or per suite with a named `workspace` export. Suite-level workspace config overrides config-level `run.workspace`.

Expand Down
11 changes: 9 additions & 2 deletions docs/cases.md
Original file line number Diff line number Diff line change
Expand Up @@ -249,6 +249,13 @@ export type SuiteWorkspaceConfig =
| {
mode: "shared";
cwd?: string;
templateDir?: string;
bootstrap?: {
command: string;
args?: string[];
timeoutMs?: number;
env?: Record<string, string>;
};
}
| {
mode: "isolated";
Expand All @@ -264,8 +271,8 @@ export type SuiteWorkspaceConfig =

Rules:

- `shared` mode supports `cwd` only
- `isolated` mode supports `templateDir` and `bootstrap` only
- `shared` mode supports `cwd`, `templateDir`, and `bootstrap`
- `isolated` mode supports `templateDir` and `bootstrap` but not `cwd`
- relative suite workspace paths resolve from the suite file directory
- isolated workspaces start empty when `templateDir` is omitted
- `templateDir` copies the full directory contents, including dotfiles and `.git`
Expand Down
56 changes: 42 additions & 14 deletions docs/workspaces.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,12 +2,13 @@

## Overview

`skillgym` supports two workspace modes:
`skillgym` supports three workspace modes:

- `shared`: run directly in a real working directory
- `none`: run directly in an existing working directory
- `shared`: create one workspace per suite run and reuse it across executions
- `isolated`: create a fresh workspace per case x runner execution

Use isolated workspaces when suites need different filesystem setups or when executions should not mutate the original source tree.
Use `none` when the agent should run in an existing directory. Use `shared` or `isolated` when the suite needs SkillGym to provision a workspace first.

## Suite-level workspace config

Expand Down Expand Up @@ -44,22 +45,43 @@ Suite workspace config overrides config-level `run.workspace`.

See `../examples/workspace-isolation-suite.ts` for a complete example that copies a template directory and runs a bootstrap command before the agent starts.

## Shared mode
## None mode

```ts
export const workspace = {
mode: "shared",
mode: "none",
cwd: "./fixtures/repo-a",
};
```

Behavior:

- executions run directly in the shared directory
- executions run directly in `cwd`
- `cwd` is optional
- if omitted, shared mode falls back to config `run.cwd`, then `process.cwd()`
- if omitted, none mode falls back to config `run.cwd`, then `process.cwd()`
- `templateDir` and `bootstrap` are not allowed

If a suite file does not export `workspace`, SkillGym falls back to config `run.workspace`, then to implicit `none` mode.

## Shared mode

```ts
export const workspace = {
mode: "shared",
templateDir: "./fixtures/repo-template",
};
```

Behavior:

- one shared workspace is created under `outputDir/workspaces/shared`
- all executions reuse that same workspace for the suite run
- `cwd` is not allowed
- `templateDir` copies into the shared workspace before any execution starts
- `bootstrap` runs once inside the shared workspace before any execution starts
- successful suite runs delete the shared workspace
- failed suite runs preserve the shared workspace

## Isolated mode

```ts
Expand All @@ -79,11 +101,11 @@ Behavior:

## Bootstrap commands

Bootstrap commands run inside the isolated workspace before the agent starts.
Bootstrap commands run inside the execution workspace before the agent starts.

```ts
export const workspace = {
mode: "isolated",
mode: "shared",
bootstrap: {
command: "sh",
args: ["./scripts/bootstrap-workspace.sh", "--seed", "demo"],
Expand All @@ -93,22 +115,28 @@ export const workspace = {

Bootstrap command behavior:

- `cwd` is the isolated workspace
- `cwd` is the provisioned shared workspace or isolated workspace
- non-zero exit fails that execution before the agent runs
- stdout and stderr are written to the execution artifact directory
- shared-workspace bootstrap stdout and stderr are written to `outputDir/workspaces/shared-setup`
- isolated-workspace bootstrap stdout and stderr are written to the execution artifact directory

Runtime environment variables:

- `SKILLGYM_WORKSPACE`
- `SKILLGYM_CASE_ID`
- `SKILLGYM_RUNNER_ID`
- `SKILLGYM_OUTPUT_DIR`
- `SKILLGYM_ARTIFACT_DIR`

Isolated workspace bootstrap also receives:

- `SKILLGYM_CASE_ID`
- `SKILLGYM_RUNNER_ID`

## Cleanup

Cleanup behavior is fixed:

- successful shared suite runs delete their workspace
- failed shared suite runs preserve their workspace
- successful isolated executions delete their workspace
- failed isolated executions preserve their workspace

Expand All @@ -120,7 +148,7 @@ Path rules:

- config `run.workspace` paths resolve from the config file directory
- suite `workspace` paths resolve from the suite file directory
- `bootstrap.command` and path-like `bootstrap.args` are resolved from the config or suite directory before the bootstrap runs inside the isolated workspace
- `bootstrap.command` and path-like `bootstrap.args` are resolved from the config or suite directory before the bootstrap runs inside the execution workspace

## Limitations

Expand Down
7 changes: 7 additions & 0 deletions skills/workspaces.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,11 @@ Use isolated workspaces when executions should not mutate the original checkout
export const workspace = {
mode: "shared",
cwd: "./fixtures/repo-a",
templateDir: "./fixtures/base-project",
bootstrap: {
command: "sh",
args: ["./scripts/bootstrap-workspace.sh"],
},
};
```

Expand All @@ -28,6 +33,8 @@ Behavior:
- executions run directly in that directory
- `cwd` is optional
- if omitted, Skillgym falls back to config `run.cwd`, then `process.cwd()`
- `templateDir` copies into that directory before the agent starts
- `bootstrap` runs in that directory before the agent starts

## Isolated mode

Expand Down
4 changes: 2 additions & 2 deletions src/cli/run.ts
Original file line number Diff line number Diff line change
Expand Up @@ -63,9 +63,9 @@ export async function runCommand(options: {
suiteDir: suite.dirPath,
});

if (options.cwd !== undefined && effectiveWorkspace.mode === "isolated") {
if (options.cwd !== undefined && effectiveWorkspace.mode !== "none") {
throw new Error(
"CLI option --cwd is only supported when the effective workspace mode is shared.",
"CLI option --cwd is only supported when the effective workspace mode is none.",
);
}

Expand Down
39 changes: 33 additions & 6 deletions src/config.ts
Original file line number Diff line number Diff line change
Expand Up @@ -421,18 +421,18 @@ function parseWorkspaceConfig(value: unknown, configPath: string): SuiteWorkspac
const templateDir = parseOptionalNonEmptyString(record.templateDir, `${configPath}.templateDir`);
const bootstrap = parseOptionalBootstrapConfig(record.bootstrap, `${configPath}.bootstrap`);

if (mode === "shared") {
if (mode === "none") {
if (templateDir !== undefined) {
throw invalidConfig(
`${configPath}.templateDir`,
'expected this key to be omitted when workspace mode is "shared"',
'expected this key to be omitted when workspace mode is "none"',
);
}

if (bootstrap !== undefined) {
throw invalidConfig(
`${configPath}.bootstrap`,
'expected this key to be omitted when workspace mode is "shared"',
'expected this key to be omitted when workspace mode is "none"',
);
}

Expand All @@ -442,6 +442,21 @@ function parseWorkspaceConfig(value: unknown, configPath: string): SuiteWorkspac
};
}

if (mode === "shared") {
if (cwd !== undefined) {
throw invalidConfig(
`${configPath}.cwd`,
'expected this key to be omitted when workspace mode is "shared"',
);
}

return {
mode,
templateDir,
bootstrap,
};
}

if (cwd !== undefined) {
throw invalidConfig(
`${configPath}.cwd`,
Expand All @@ -457,8 +472,8 @@ function parseWorkspaceConfig(value: unknown, configPath: string): SuiteWorkspac
}

function parseWorkspaceMode(value: unknown, configPath: string): SuiteWorkspaceConfig["mode"] {
if (value !== "shared" && value !== "isolated") {
throw invalidConfig(configPath, 'expected "shared" or "isolated"');
if (value !== "none" && value !== "shared" && value !== "isolated") {
throw invalidConfig(configPath, 'expected "none", "shared", or "isolated"');
}

return value;
Expand Down Expand Up @@ -623,10 +638,22 @@ function resolveWorkspaceConfigPaths(
config: SuiteWorkspaceConfig,
configDir: string,
): SuiteWorkspaceConfig {
if (config.mode === "none") {
return {
mode: "none",
cwd: config.cwd === undefined ? undefined : path.resolve(configDir, config.cwd),
};
}

if (config.mode === "shared") {
return {
mode: "shared",
cwd: config.cwd === undefined ? undefined : path.resolve(configDir, config.cwd),
templateDir:
config.templateDir === undefined ? undefined : path.resolve(configDir, config.templateDir),
bootstrap:
config.bootstrap === undefined
? undefined
: resolveBootstrapConfigPaths(config.bootstrap, configDir),
};
}

Expand Down
8 changes: 7 additions & 1 deletion src/domain/case.ts
Original file line number Diff line number Diff line change
Expand Up @@ -10,11 +10,17 @@ export interface WorkspaceBootstrapConfig {

export type SuiteWorkspaceConfig =
| {
mode: "shared";
mode: "none";
cwd?: string;
templateDir?: never;
bootstrap?: never;
}
| {
mode: "shared";
cwd?: never;
templateDir?: string;
bootstrap?: WorkspaceBootstrapConfig;
}
| {
mode: "isolated";
cwd?: never;
Expand Down
2 changes: 1 addition & 1 deletion src/reporters/contract.ts
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ import type { Case } from "../domain/case.js";
export interface ReporterContext {
isInteractive: boolean;
cwd: string;
workspaceMode: "shared" | "isolated";
workspaceMode: "none" | "shared" | "isolated";
suitePath: string;
suiteRunArtifactDir: string;
selectedCaseCount: number;
Expand Down
11 changes: 7 additions & 4 deletions src/reporters/standard.ts
Original file line number Diff line number Diff line change
Expand Up @@ -105,10 +105,13 @@ export function createStandardReporter(options: StandardReporterOptions = {}): B
onSuiteStart(event) {
printBanner({ kind: "compact", stdout });
writeLine(`${colors.dim("Suite ")}${accent(event.context.suitePath)}`, stdout);
writeLine(
`${colors.dim("Workspace ")}${colors.bold(event.context.workspaceMode === "shared" ? event.context.cwd : `${event.context.workspaceMode} per execution`)}`,
stdout,
);
const workspaceLabel =
event.context.workspaceMode === "shared"
? event.context.cwd
: event.context.workspaceMode === "isolated"
? "isolated per execution"
: `${event.context.cwd} (none)`;
writeLine(`${colors.dim("Workspace ")}${colors.bold(workspaceLabel)}`, stdout);
writeLine(`${colors.dim("Cases ")}${String(event.context.selectedCaseCount)}`, stdout);
if (event.context.tagFilter !== undefined) {
writeLine(`${colors.dim("Tags ")}${event.context.tagFilter.join(", ")}`, stdout);
Expand Down
Loading