Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .github/workflows/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,8 @@ The target repository grants `contents: write`, `pull-requests: write`, and `iss

Docs Agent consumes the reusable WP Codebox workflow and helper implementation at accepted immutable revision `a6fe2d208e990a8d04104aa74aacbb8d1539fbc1` and passes `wp_codebox_release_ref: v0.12.29` only for released packaged runtime assets. The release tag resolves to immutable revision `bc982947ec33c78160125026e16d357b7ece3ea1`, distinct from both the workflow producer revision and the fixed Docs Agent package revision. WP Codebox materializes the declared native runtime closure, executes the selected package, applies bounded workspace changes, runs verification, and publishes approved host changes. The released assets are `01-wp-codebox.zip` and `02-wp-codebox-workspace-0.12.29.tgz`; the private package-recovery manifest is intentionally not a reviewer asset.

The hosted Agents API component is pinned to `21a8b9b7845e6b89f82bdf34e62f739220184ec7`. It includes the native `enabled_tools` and anchor-independent `require_tool_use` capabilities introduced at `89dd23903489882cfcf14a0d293daf939ad4e1a5`, and omits enabled tool names that have neither an installed ability nor a trusted runtime overlay. The runtime-capability fixture verifies this selection in both the hosted runtime closure and native importer CI so bootstrap cannot finish after a prose-only zero-tool turn or spend its run on unavailable host tools.

Provider artifacts retain only allowlisted review artifacts, controlled workflow envelopes, and canonical runtime-source and seed provenance. WP Codebox v0.12.24 uploads a reviewer-safe workflow-result projection: public control, verification, publication, output-projection, access, failure, artifact-declaration, and canonical transcript-provenance fields remain, while raw `runtime_result`, `outputs.engine_data`, model/provider/tool payloads, source content, private paths, and secrets are excluded. Before persistence or upload, it sanitizes private runtime paths from nested values, object keys, diagnostics, and command arguments. WP Codebox captures one trusted canonical `codebox-transcript` through a pre-sanitization reviewer-evidence descriptor, validates its artifact-root containment, schema, digest, and size, then emits the compact reviewer transcript as a `wp-codebox/reviewer-agent-transcript/v1` projection. Before apply-back, it compares the seed identity with the host workspace identity; rejected patches preserve the expected and actual identities plus patch and changed-file evidence for upload. Its pre-redaction trusted apply input preserves machine-applicable patch bytes in a private channel that is removed before durable artifact sanitization, as fixed in [WP Codebox #1842](https://github.com/Automattic/wp-codebox/pull/1842). The canonical transcript remains the bounded tool-observability surface. Lifecycle and downstream failures retain runtime evidence in their internal normalized failed result even when artifact preparation cannot complete. Diagnostic messages that name runtime classes remain reviewable, while PHP-shaped runtime source remains blocked. See [WP Codebox #1767](https://github.com/Automattic/wp-codebox/issues/1767), [WP Codebox #1814](https://github.com/Automattic/wp-codebox/issues/1814), [WP Codebox #1815](https://github.com/Automattic/wp-codebox/pull/1815), and [WP Codebox #1823](https://github.com/Automattic/wp-codebox/pull/1823). Successful canonical reviewer-evidence reference: [run `29350690551`](https://github.com/Automattic/build-with-wordpress/actions/runs/29350690551). The [hosted failure `29468956857`](https://github.com/Automattic/agents-api/actions/runs/29468956857) executed WP Codebox v0.12.20, published [Agents API #430](https://github.com/Automattic/agents-api/pull/430), then failed during upload preparation; it motivated #1814/#1815 and is not a successful v0.12.24 reviewer-safe workflow-result run.

WP Codebox v0.12.25 excludes Git-ignored verification artifacts such as pnpm's symlinked `node_modules` tree from publication integrity snapshots while continuing to reject tracked or otherwise publishable symlinks. This allows dependency installation during verification without weakening the bounded workspace publication contract; see [WP Codebox #1845](https://github.com/Automattic/wp-codebox/pull/1845).
Expand Down
7 changes: 6 additions & 1 deletion .github/workflows/maintain-docs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -69,6 +69,10 @@ env:
description: Validate the prepared Docs Agent task without starting a live agent run.
type: boolean
default: false
max_turns:
description: Maximum agent turns available for repository discovery, documentation writes, and final diff inspection.
type: number
default: 32
secrets:
OPENAI_API_KEY:
description: Optional OpenAI credential used for live Docs Agent runs.
Expand Down Expand Up @@ -293,7 +297,7 @@ jobs:
--argjson artifactDeclarations "$artifact_declarations" \
'{schema:"docs-agent/runner-recipe/v1",targetRepository:$targetRepository,prompt:$prompt,docsAgent:{externalPackageSource:{repository:$repository,revision:$revision,path:$packagePath,digest:$packageDigest},agentSlug:$agentSlug},runner:{contextRepositories:$contextRepositories,bootstrapContract:$bootstrapContract,sourceDelta:$sourceDelta,validationDependencies:$validationDependencies,verificationCommands:$verificationCommands,driftChecks:$driftChecks,writablePaths:$writablePaths,baseRef:$baseRef,docsBranch:$docsBranch,successRequiresPr:$successRequiresPr},artifacts:{expected:($artifactDeclarations|map(.name)),declarations:$artifactDeclarations}}')"
external_package_source="$(jq -cn --arg repository "Automattic/docs-agent" --arg revision "$DOCS_AGENT_PACKAGE_REVISION" --arg path "$package_path" --arg digest "$package_digest" '{repository:$repository,revision:$revision,path:$path,digest:$digest}')"
runtime_sources='[{"version":1,"role":"component","repository":"Automattic/agents-api","revision":"78e2dd409010f98fa4d26cdd72572117384ab18d","path":".","metadata":{"slug":"agents-api","loadAs":"mu-plugin","pluginFile":"agents-api.php"}},{"version":1,"role":"provider_plugin","source":{"type":"https_zip","url":"https://downloads.wordpress.org/plugin/ai-provider-for-openai.1.0.3.zip","sha256":"48f3c0c714b3164cda79d320829830d5a0ea1116e0b19653da8af898a22d3bb6","archive_root":"ai-provider-for-openai"},"metadata":{"slug":"ai-provider-for-openai","pluginFile":"plugin.php","activate":true,"providers":["openai"]}},{"version":1,"role":"bundled_library","repository":"WordPress/php-ai-client","revision":"631704201d15ffeff7091ad3bc7156db74054956","path":".","metadata":{"library":"php-ai-client","strategy":"wordpress-scoped-bundle"}}]'
runtime_sources='[{"version":1,"role":"component","repository":"Automattic/agents-api","revision":"21a8b9b7845e6b89f82bdf34e62f739220184ec7","path":".","metadata":{"slug":"agents-api","loadAs":"mu-plugin","pluginFile":"agents-api.php"}},{"version":1,"role":"provider_plugin","source":{"type":"https_zip","url":"https://downloads.wordpress.org/plugin/ai-provider-for-openai.1.0.3.zip","sha256":"48f3c0c714b3164cda79d320829830d5a0ea1116e0b19653da8af898a22d3bb6","archive_root":"ai-provider-for-openai"},"metadata":{"slug":"ai-provider-for-openai","pluginFile":"plugin.php","activate":true,"providers":["openai"]}},{"version":1,"role":"bundled_library","repository":"WordPress/php-ai-client","revision":"631704201d15ffeff7091ad3bc7156db74054956","path":".","metadata":{"library":"php-ai-client","strategy":"wordpress-scoped-bundle"}}]'

runner_workspace="$(jq -cn \
--arg repo "$GITHUB_REPOSITORY_NAME" \
Expand Down Expand Up @@ -364,6 +368,7 @@ jobs:
artifact_declarations: ${{ needs.prepare.outputs.artifact_declarations }}
output_projections: ${{ needs.prepare.outputs.output_projections }}
callback_data: ${{ needs.prepare.outputs.recipe_json }}
max_turns: ${{ inputs.max_turns }}
run_agent: ${{ inputs.run_agent }}
dry_run: ${{ inputs.dry_run }}
secrets:
Expand Down
4 changes: 3 additions & 1 deletion .github/workflows/tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -24,14 +24,16 @@ jobs:
uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
with:
repository: Automattic/agents-api
ref: 78e2dd409010f98fa4d26cdd72572117384ab18d
ref: 21a8b9b7845e6b89f82bdf34e62f739220184ec7
path: agents-api
fetch-depth: 0
token: ${{ github.token }}
persist-credentials: false

- name: Run package, link, and native importer tests
run: |
php tests/validate-docs-agent-packages.php
AGENTS_API_DIR="$GITHUB_WORKSPACE/agents-api" php tests/validate-agents-api-runtime-contract.php
php tests/validate-docs-agent-completion-contract.php
php tests/validate-external-native-package-sources.php
php tests/repair-docs-links-smoke.php
Expand Down
1 change: 1 addition & 0 deletions .github/workflows/validate.yml
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,7 @@ jobs:
WP_CODEBOX_DIR: ${{ github.workspace }}/.wp-codebox
run: |
php tests/validate-docs-agent-packages.php
php tests/validate-agents-api-runtime-contract.php
php tests/validate-docs-agent-completion-contract.php
php tests/validate-external-native-package-sources.php
php tests/validate-wp-codebox-run-agent-task-contract.php
Expand Down
3 changes: 2 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -262,6 +262,7 @@ For skills PRs, also confirm the live instructions match current upstream tool b

```bash
php tests/validate-docs-agent-packages.php
php tests/validate-agents-api-runtime-contract.php
php tests/validate-docs-agent-completion-contract.php
php tests/validate-external-native-package-sources.php
php tests/repair-docs-links-smoke.php
Expand All @@ -278,4 +279,4 @@ AGENTS_API_DIR=/path/to/agents-api php tests/native-agent-import.php

It imports every native package through `wp_agent_import_runtime_bundles()`, verifies registration and no-change-capable tool rules, and invokes the default native chat handler far enough to resolve each registered agent. It intentionally fails when `AGENTS_API_DIR` is unavailable rather than treating an unexecuted importer as a passing test. It does not execute a model turn because the packages intentionally leave provider/model selection to the caller.

The `Docs Agent Tests` GitHub Actions workflow runs on pull requests and pushes. It fetches Docs Agent history so it can run the immutable native package source validator, then runs the structural package validator, docs-link repair smoke test, and native importer integration test against `Automattic/agents-api` at `78e2dd409010f98fa4d26cdd72572117384ab18d`, the merged commit from [Agents API #428](https://github.com/Automattic/agents-api/pull/428).
The `Docs Agent Tests` GitHub Actions workflow runs on pull requests and pushes. It fetches Docs Agent history so it can run the immutable native package source validator, then runs the structural package validator, explicit Agents API runtime-capability contract, docs-link repair smoke test, and native importer integration test against `Automattic/agents-api` at `21a8b9b7845e6b89f82bdf34e62f739220184ec7`. That revision includes the native `enabled_tools` and anchor-independent `require_tool_use` capabilities introduced by [Agents API #384](https://github.com/Automattic/agents-api/pull/384), and omits enabled tool names that have neither an installed ability nor a trusted runtime overlay.
11 changes: 11 additions & 0 deletions tests/agents-api-runtime-capability.fixture.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
{
"schema": "docs-agent/agents-api-runtime-capability/v1",
"repository": "Automattic/agents-api",
"revision": "21a8b9b7845e6b89f82bdf34e62f739220184ec7",
"capability_commit": "89dd23903489882cfcf14a0d293daf939ad4e1a5",
"capabilities": [
"native agent_config.enabled_tools are resolved into tool declarations",
"require_tool_use completion rules block a zero-tool first turn without an anchor",
"enabled tool names without an installed ability or trusted runtime overlay are omitted"
]
}
51 changes: 51 additions & 0 deletions tests/validate-agents-api-runtime-contract.php
Original file line number Diff line number Diff line change
@@ -0,0 +1,51 @@
<?php
/**
* Validate the immutable Agents API capability baseline for native packages.
*/

declare( strict_types=1 );

$root = dirname( __DIR__ );

$assert = static function ( bool $condition, string $message ): void {
if ( ! $condition ) {
throw new RuntimeException( $message );
}
};

$fixture_path = $root . '/tests/agents-api-runtime-capability.fixture.json';
$fixture = json_decode( (string) file_get_contents( $fixture_path ), true );
$assert( is_array( $fixture ), 'Agents API runtime capability fixture must be valid JSON.' );
$assert( 'docs-agent/agents-api-runtime-capability/v1' === ( $fixture['schema'] ?? null ), 'Agents API runtime capability fixture schema mismatch.' );
$revision = $fixture['revision'] ?? null;
$assert( is_string( $revision ) && preg_match( '/^[0-9a-f]{40}$/', $revision ) === 1, 'Agents API runtime capability fixture must declare an immutable revision.' );
$capability_commit = $fixture['capability_commit'] ?? null;
$assert( is_string( $capability_commit ) && preg_match( '/^[0-9a-f]{40}$/', $capability_commit ) === 1, 'Agents API runtime capability fixture must retain the immutable capability introduction commit.' );
$assert( is_array( $fixture['capabilities'] ?? null ) && 3 === count( $fixture['capabilities'] ), 'Agents API runtime capability fixture must identify every required native tool capability.' );

$runtime_source = array(
'version' => 1,
'role' => 'component',
'repository' => $fixture['repository'],
'revision' => $revision,
'path' => '.',
'metadata' => array( 'slug' => 'agents-api', 'loadAs' => 'mu-plugin', 'pluginFile' => 'agents-api.php' ),
);

$workflow = (string) file_get_contents( $root . '/.github/workflows/maintain-docs.yml' );
preg_match( "/runtime_sources='(?<json>[^']+)'/", $workflow, $runtime_sources_match );
$runtime_sources = json_decode( $runtime_sources_match['json'] ?? '', true );
$assert( $runtime_source === ( $runtime_sources[0] ?? null ), 'Hosted runtime sources must pin the Agents API native tool capability revision.' );

$tests_workflow = (string) file_get_contents( $root . '/.github/workflows/tests.yml' );
$assert( preg_match( '/repository: Automattic\/agents-api\s+ref: ' . preg_quote( $revision, '/' ) . '/s', $tests_workflow ) === 1, 'Native importer CI must test the hosted Agents API capability revision.' );

$agents_api_dir = getenv( 'AGENTS_API_DIR' );
if ( is_string( $agents_api_dir ) && is_dir( $agents_api_dir ) ) {
$checked_out_revision = trim( (string) shell_exec( 'git -C ' . escapeshellarg( $agents_api_dir ) . ' rev-parse HEAD' ) );
$assert( $revision === $checked_out_revision, 'The checked-out Agents API runtime must match the immutable capability revision.' );
exec( 'git -C ' . escapeshellarg( $agents_api_dir ) . ' merge-base --is-ancestor ' . escapeshellarg( $capability_commit ) . ' HEAD', $output, $status );
$assert( 0 === $status, 'The checked-out Agents API runtime must retain the original native tool capabilities.' );
}

fwrite( STDOUT, "Agents API runtime capability contract passed.\n" );
2 changes: 2 additions & 0 deletions tests/validate-docs-agent-packages.php
Original file line number Diff line number Diff line change
Expand Up @@ -98,6 +98,8 @@
$assert( str_contains( $maintain_docs_workflow, 'ACCESS_TOKEN: ${{ github.token }}' ), 'maintain-docs.yml must forward the caller-scoped GitHub token to the native runner.' );
$assert( str_contains( $maintain_docs_workflow, 'EXTERNAL_PACKAGE_SOURCE_POLICY: ${{ secrets.EXTERNAL_PACKAGE_SOURCE_POLICY }}' ), 'maintain-docs.yml must explicitly forward the external package source policy.' );
$assert( str_contains( $maintain_docs_workflow, 'runtime_sources: ${{ needs.prepare.outputs.runtime_sources }}' ), 'maintain-docs.yml must pass its complete native runtime closure to WP Codebox.' );
$assert( str_contains( $maintain_docs_workflow, 'max_turns: ${{ inputs.max_turns }}' ), 'maintain-docs.yml must forward its documentation-sized turn budget to WP Codebox.' );
$assert( 1 === preg_match( '/max_turns:\s*\n\s+description:.*\n\s+type: number\s*\n\s+default: 32/', $maintain_docs_workflow ), 'maintain-docs.yml must default to enough turns for bootstrap discovery, writes, and diff inspection.' );

$workflow_readme = (string) file_get_contents( $root . '/.github/workflows/README.md' );
foreach ( array( 'Docs Agent Runner Recipe', 'portable recipe', 'Docs Agent owns the native package' ) as $migration_note_text ) {
Expand Down
7 changes: 4 additions & 3 deletions tests/validate-wp-codebox-run-agent-task-contract.php
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,7 @@
$assert( 'wp-codebox/reusable-workflow-interface/v1' === ( $contract['schema'] ?? null ), 'WP Codebox producer schema version mismatch.' );
$assert( '.github/workflows/run-agent-task.yml' === ( $contract['workflow'] ?? null ), 'WP Codebox producer contract targets an unexpected workflow.' );
$release = $read_json( $root . '/tests/wp-codebox-release.fixture.json' );
$agents_api_runtime = $read_json( $root . '/tests/agents-api-runtime-capability.fixture.json' );
$release_tag = $release['tag'] ?? null;
$assert( is_string( $release_tag ) && preg_match( '/^v\d+\.\d+\.\d+$/', $release_tag ) === 1, 'WP Codebox release fixture must declare an exact release tag.' );
$release_revision = $release['revision'] ?? null;
Expand Down Expand Up @@ -155,7 +156,7 @@
$input = $contract['inputs'][ $name ] ?? null;
$assert( is_array( $input ), "Docs Agent passes producer input {$name}, which is absent from the producer schema." );
$type = $input['type'] ?? null;
$actual_type = preg_match( '/^(true|false)$/', $value ) === 1 || in_array( $name, array( 'success_requires_pr', 'run_agent', 'dry_run' ), true ) ? 'boolean' : ( is_numeric( $value ) ? 'number' : 'string' );
$actual_type = preg_match( '/^(true|false)$/', $value ) === 1 || in_array( $name, array( 'success_requires_pr', 'run_agent', 'dry_run' ), true ) ? 'boolean' : ( is_numeric( $value ) || 'max_turns' === $name ? 'number' : 'string' );
$assert( $type === $actual_type, "Docs Agent input {$name} has {$actual_type} shape; producer requires {$type}." );
}

Expand Down Expand Up @@ -243,8 +244,8 @@
$assert( array(
'version' => 1,
'role' => 'component',
'repository' => 'Automattic/agents-api',
'revision' => '78e2dd409010f98fa4d26cdd72572117384ab18d',
'repository' => $agents_api_runtime['repository'],
'revision' => $agents_api_runtime['revision'],
'path' => '.',
'metadata' => array( 'slug' => 'agents-api', 'loadAs' => 'mu-plugin', 'pluginFile' => 'agents-api.php' ),
) === ( $runtime_sources[0] ?? null ), 'Docs Agent must declare the pinned Agents API MU-plugin component.' );
Expand Down
Loading