[tests] Wrap structured PHPUnit output in command JSON

Author: coisa

CHANGELOG.md

@@ -9,7 +9,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Fixed
 
-- Keep `tests` JSON output parseable while surfacing structured PHPUnit result data for agent-driven runs, including a raw-output fallback when PHPUnit cannot emit machine-readable JSON (#248)
+- Keep `tests` JSON output parseable by exposing the PHPUnit agent-reporter payload under `output` in structured runs, including a raw-output fallback when PHPUnit writes extra text before the final JSON (#248)
 - Register ``ergebnis/phpunit-agent-reporter`` in the packaged ``phpunit.xml`` so AI agents receive compact PHPUnit JSON summaries without changing consumer overrides manually (#327)
 - Let `wiki`, `docs`, `tests`, `metrics`, and `reports` skip gracefully for guide-only repositories while keeping wiki/report workflows and published preview links aligned with the artifacts that were actually generated (#325)
 

README.md

@@ -220,11 +220,16 @@ rendering, and `--progress` re-enables it for human-readable terminal runs.
 When `--json` or `--pretty-json` is active on commands that orchestrate other
 tools, DevTools keeps progress suppressed, forwards JSON flags where the
 underlying tool supports structured output, and otherwise falls back to
-quieter subprocess modes so the captured payload stays machine-readable. In
-GitHub Actions, queued subprocess output is grouped into collapsible sections,
-and logged failures emit native workflow error annotations, including file and
-line metadata when commands provide it. The packaged tests, reports, wiki, and
-changelog workflows also append concise Markdown outcomes to
+quieter subprocess modes so the captured payload stays machine-readable. The
+`tests` command now captures the bundled PHPUnit agent-reporter payload in
+structured runs and stores it in `context.output`, preserving `result`,
+`summary`, optional `details`, and a `raw_output` fallback when coverage or
+other PHPUnit text is emitted before the final reporter JSON. In GitHub
+Actions, queued subprocess output is
+grouped into collapsible sections, and logged failures emit native workflow
+error annotations, including file and line metadata when commands provide it.
+The packaged tests, reports, wiki, and changelog workflows also append concise
+Markdown outcomes to
 `GITHUB_STEP_SUMMARY` so maintainers can scan versions, URLs, preview refs,
 verification status, and release results without expanding full logs. This
 repository also keeps a bounded retry workflow that reruns failed jobs once

docs/commands/tests.rst

@@ -162,14 +162,19 @@ Behavior
 - ``--json`` and ``--pretty-json`` keep progress output disabled so the
   structured payload stays clean, even when ``--progress`` is provided.
 - in agent-driven runs outside the Composer test suite, the command also
-  switches to the same structured capture mode automatically so PHPUnit output
-  is buffered instead of interleaving with top-level DevTools log payloads.
+  switches to the same structured capture mode automatically.
 - when structured capture is active and PHPUnit emits agent-reporter JSON, the
-  command exposes a ``phpunit`` object in the log context with ``tool``,
-  ``label``, ``exit_code``, and the reported ``result`` / ``summary`` /
-  ``details`` fields.
+  command stores that payload inside ``output`` while keeping the standard
+  DevTools JSON envelope. ``--json`` and ``--pretty-json`` therefore expose
+  the same structured result, with formatting as the only difference.
 - when structured capture is active but PHPUnit does not emit parseable JSON,
-  the command preserves the raw subprocess text under
-  ``context.phpunit.raw_output`` instead of dropping it.
+  the command preserves the raw subprocess text inside ``output.raw_output``
+  instead of dropping it.
+- when coverage generation or other PHPUnit text appears before the final
+  reporter payload, the command preserves that prelude in
+  ``output.raw_output`` while keeping the main JSON result parseable.
+- when ``--min-coverage`` is used in structured mode, the command appends a
+  ``coverage`` object under ``output`` and flips ``output.result`` to
+  ``failure`` if the threshold is not met.
 - The command fails if minimum coverage is not met (when ``--min-coverage`` is set).
 - The packaged configuration registers the DevTools PHPUnit extension.

docs/running/specialized-commands.rst

@@ -65,8 +65,10 @@ Important details:
   automatically;
 - ``--pretty-json`` stays valid JSON and does not add ANSI color escapes;
 - in agent-driven runs, the command also captures PHPUnit output in structured
-  mode automatically and exposes a ``phpunit`` summary object when the bundled
-  agent reporter is active;
+  mode automatically and stores the bundled PHPUnit agent-reporter payload in
+  ``output``. ``--json`` and ``--pretty-json`` therefore keep the same
+  structured shape, with ``raw_output`` preserved under ``output`` when
+  PHPUnit writes extra text before the final JSON;
 - the packaged configuration registers the DevTools PHPUnit extension.
 
 ``dependencies``

src/Console/Command/TestsCommand.php

@@ -45,6 +45,7 @@
 use Symfony\Component\Console\Input\InputOption;
 use Symfony\Component\Console\Output\BufferedOutput;
 use Symfony\Component\Console\Output\OutputInterface;
+use Symfony\Component\Process\Process;
 
 use function is_numeric;
 use function Safe\json_decode;
@@ -61,6 +62,10 @@ final class TestsCommand extends Command
     use HasJsonOption;
     use LogsCommandResults;
 
+    private const string AGENT_ENVIRONMENT_VARIABLE = 'AI_AGENT';
+
+    private const string AGENT_ENVIRONMENT_VALUE = 'fast-forward/dev-tools';
+
     private const string PROCESS_LABEL = 'Running PHPUnit Tests';
 
     /**
@@ -168,9 +173,12 @@ protected function configure(): void
      */
     protected function execute(InputInterface $input, OutputInterface $output): int
     {
-        $jsonOutput = $this->isJsonOutput($input)
+        $explicitJsonOutput = (bool) $input->getOption('json');
+        $prettyJsonOutput = $this->isPrettyJsonOutput($input);
+        $structuredOutput = $prettyJsonOutput
+            || $explicitJsonOutput
             || ($this->runtimeEnvironment->isAgentPresent() && ! $this->runtimeEnvironment->isComposerTestRun());
-        $processOutput = $jsonOutput ? new BufferedOutput() : $output;
+        $processOutput = $structuredOutput ? new BufferedOutput() : $output;
         $cacheEnabled = $this->isCacheEnabled($input);
 
         $this->getLogger()
@@ -215,11 +223,11 @@ protected function execute(InputInterface $input, OutputInterface $output): int
             ->withArgument('--display-incomplete')
             ->withArgument('--display-skipped');
 
-        if (! $input->getOption('progress') || $jsonOutput) {
+        if (! $input->getOption('progress') || $structuredOutput) {
             $processBuilder = $processBuilder->withArgument('--no-progress');
         }
 
-        if (! $jsonOutput) {
+        if (! $structuredOutput) {
             $processBuilder = $processBuilder->withArgument('--colors=always');
         }
 
@@ -241,15 +249,18 @@ protected function execute(InputInterface $input, OutputInterface $output): int
             $processBuilder = $processBuilder->withArgument('--filter', $input->getOption('filter'));
         }
 
-        $this->processQueue->add(
-            process: $processBuilder
-                ->withArgument($input->getArgument('path'))
-                ->build([DevToolsPathResolver::getPreferredToolBinaryPath('phpunit')]),
-            label: self::PROCESS_LABEL,
-        );
+        $process = $processBuilder
+            ->withArgument($input->getArgument('path'))
+            ->build([DevToolsPathResolver::getPreferredToolBinaryPath('phpunit')]);
+
+        if ($structuredOutput) {
+            $this->forceAgentReporter($process);
+        }
+
+        $this->processQueue->add(process: $process, label: self::PROCESS_LABEL);
 
         $result = $this->processQueue->run($processOutput);
-        $processResultContext = $this->resolveProcessResultContext($processOutput, $result, $jsonOutput);
+        $processResultContext = $this->resolveProcessResultContext($processOutput, $result, $structuredOutput);
 
         if (self::SUCCESS !== $result || null === $minimumCoverage || null === $coverageReportPath) {
             if (self::SUCCESS === $result) {
@@ -264,6 +275,16 @@ protected function execute(InputInterface $input, OutputInterface $output): int
             $minimumCoverage,
         );
 
+        if ($structuredOutput) {
+            $processResultContext = $this->withStructuredCoverageValidationContext(
+                $processResultContext,
+                $coverageContext,
+                $minimumCoverage,
+                $validationResult,
+                $message,
+            );
+        }
+
         if (self::SUCCESS === $validationResult) {
             return $this->success($message, $input, [...$processResultContext, ...$coverageContext]);
         }
@@ -283,46 +304,110 @@ protected function execute(InputInterface $input, OutputInterface $output): int
     private function resolveProcessResultContext(
         OutputInterface $processOutput,
         int $exitCode,
-        bool $structuredOutput
+        bool $structuredOutput,
     ): array {
-        if (! $structuredOutput) {
+        if ($structuredOutput) {
             return [
-                'output' => $processOutput,
+                'output' => $this->resolveStructuredProcessResultPayload($processOutput, $exitCode),
             ];
         }
 
-        $context = [
-            'phpunit' => [
-                'tool' => 'phpunit',
-                'label' => self::PROCESS_LABEL,
-                'exit_code' => $exitCode,
-            ],
+        return [
+            'output' => $processOutput,
+        ];
+    }
+
+    /**
+     * Forces the PHPUnit subprocess to expose the agent reporter payload.
+     *
+     * @param Process $process the configured PHPUnit process
+     *
+     * @return void
+     */
+    private function forceAgentReporter(Process $process): void
+    {
+        $env = $process->getEnv();
+
+        if (\array_key_exists(self::AGENT_ENVIRONMENT_VARIABLE, $env)) {
+            return;
+        }
+
+        $env[self::AGENT_ENVIRONMENT_VARIABLE] = self::AGENT_ENVIRONMENT_VALUE;
+        $process->setEnv($env);
+    }
+
+    /**
+     * Builds the structured payload that will be emitted for agent-oriented runs.
+     *
+     * @param OutputInterface $processOutput the output sink used while the process ran
+     * @param int $exitCode the exit code returned by the process queue
+     *
+     * @return array<string, mixed> the structured process payload
+     */
+    private function resolveStructuredProcessResultPayload(OutputInterface $processOutput, int $exitCode): array
+    {
+        $payload = [
+            'result' => self::SUCCESS === $exitCode ? 'success' : 'failure',
         ];
 
         if (! $processOutput instanceof BufferedOutput) {
-            return $context;
+            return $payload;
         }
 
         $rawOutput = trim($processOutput->fetch());
 
         if ('' === $rawOutput) {
-            return $context;
+            return $payload;
         }
 
         [$decoded, $supplementalOutput] = $this->decodeStructuredProcessOutput($rawOutput);
 
-        if (! \is_array($decoded)) {
-            $context['phpunit']['raw_output'] = $supplementalOutput;
+        if (\is_array($decoded)) {
+            $payload = $decoded;
+        }
 
+        if (null !== $supplementalOutput) {
+            $payload['raw_output'] = $supplementalOutput;
+        }
+
+        return $payload;
+    }
+
+    /**
+     * Appends minimum-coverage validation data to the structured PHPUnit output payload.
+     *
+     * @param array<string, mixed> $context the command result context
+     * @param array<string, float|int|string|null> $coverageContext structured coverage metrics
+     * @param float $minimumCoverage the required coverage percentage
+     * @param int $validationResult the post-PHPUnit validation status
+     * @param string $message the validation message
+     *
+     * @return array<string, mixed> the enriched structured command context
+     */
+    private function withStructuredCoverageValidationContext(
+        array $context,
+        array $coverageContext,
+        float $minimumCoverage,
+        int $validationResult,
+        string $message,
+    ): array {
+        if (! isset($context['output']) || ! \is_array($context['output'])) {
             return $context;
         }
 
-        $context['phpunit'] = [...$context['phpunit'], ...$decoded];
+        $payload = $context['output'];
+        $payload['coverage'] = [
+            ...$coverageContext,
+            'minimum' => $minimumCoverage,
+        ];
 
-        if (null !== $supplementalOutput) {
-            $context['phpunit']['raw_output'] = $supplementalOutput;
+        if (self::SUCCESS !== $validationResult) {
+            $payload['message'] = $message;
+            $payload['result'] = 'failure';
         }
 
+        $context['output'] = $payload;
+
         return $context;
     }