feat: improve local run, deploy, and invoke. Add temp eval infra for testing

Author: anchenyi

.gitignore

@@ -359,3 +359,4 @@ dashboard/**/dist/
 
 # Local vally eval outputs
 results/
+tmp/

plugin/skills/microsoft-foundry/foundry-agent/create/references/local-run.md

@@ -8,6 +8,14 @@ Use this when iterating on a hosted agent before deploying.
 > AZURE_AI_MODEL_DEPLOYMENT_NAME=<model-deployment-name>
 > ```
 > If you already ran `azd provision`, extract these from `azd env get-values`.
+>
+> **Critical: keep `.env` and `azd env` in sync.** `azd ai agent run` injects the active `azd env` values into the agent process before Python loads `.env`. Many samples use `load_dotenv(override=False)`, so an existing process environment value wins over `.env`. If you change the project endpoint or model deployment, update both `.env` and `azd env`:
+> ```bash
+> azd env set FOUNDRY_PROJECT_ENDPOINT "https://<account>.services.ai.azure.com/api/projects/<project>"
+> azd env set AZURE_AI_MODEL_DEPLOYMENT_NAME "<model-deployment-name>"
+> azd env get-values
+> ```
+> A stale `AZURE_AI_MODEL_DEPLOYMENT_NAME` in `azd env` can make local run call the wrong deployment even when `.env` is correct, commonly surfacing as a Foundry responses API `404 Not Found`.
 
 ## Start the agent locally
 
@@ -19,7 +27,7 @@ What this does:
 
 1. Resolves the agent service from `azure.yaml` (auto-picks when only one exists).
 2. Detects the project type (Python, .NET, Node.js) from files in the service source dir.
-3. Installs dependencies if needed (`uv pip install -e .`, `npm install`, `dotnet restore`).
+3. Installs dependencies if needed (`pip install -r requirements.txt`, `npm install`, `dotnet restore`).
 4. Starts the agent in the foreground on `localhost:8088` (default).
 5. Opens **Agent Inspector** in your browser (unless `--no-inspector`).
 

plugin/skills/microsoft-foundry/foundry-agent/deploy/deploy.md

@@ -164,7 +164,7 @@ azd deploy --no-prompt
 
 Each env has its own `AGENT_<SVC>_*` vars.
 
-## Common failure modes -- Hosted
+## Common failure -- Hosted
 
 | Error | Fix |
 |-------|-----|
@@ -176,6 +176,7 @@ Each env has its own `AGENT_<SVC>_*` vars.
 | `container registry endpoint not found` | ACR not configured. Use `azd env set AZURE_CONTAINER_REGISTRY_ENDPOINT <url>`, or switch to direct code deployment. |
 | Agent version poll times out | Image still building; retry `azd ai agent show` after a minute. |
 | `session_not_ready` (424) | Cold start — wait 30-60 seconds and retry. For direct code deployments, first invocation installs dependencies. Use `1` CPU / `2Gi` memory minimum. **Also verify:** (1) a capability host exists on the Foundry account (see Step 2 above), (2) the agent's managed identity has `Cognitive Services User` role on the Foundry account — missing capability host or role are the most common causes of persistent `session_not_ready`. See [direct-code-deployment Task 8-9](references/direct-code-deployment.md) and [invoke](../invoke/invoke.md). |
+| `could not resolve agent service in azd project: no azure.ai.agent service named '<agentName>' found in azure.yaml` from `azd ai agent invoke` | Name mismatch. Update the agent name to the deployed agent name. |
 | `subscription quota exceeded` | Ask user to request quota; don't auto-retry. |
 | Bicep deploy errors | Forward `error.details[]` verbatim to the user. |
 | `RoleAssignmentUpdateNotPermitted` during provision | A role assignment already exists but conflicts. Check for existing role assignments with `az role assignment list --scope <resource-scope>`. The provision may have succeeded for all resources except RBAC — verify with `azd ai project show` and manually assign the `Cognitive Services User` role to the agent identity if needed. |

plugin/skills/microsoft-foundry/foundry-agent/deploy/references/direct-code-deployment.md

@@ -315,6 +315,8 @@ Do not send `x-ms-agent-name` on `POST /agents/<agent-name>/versions` or `POST /
 
 Update agent and create version are idempotent on zip SHA-256 plus agent definition. If both are unchanged from the latest version, the service can return the existing version instead of creating a duplicate. To force a new version, change the zip contents or definition.
 
+If the write response contains `versions.latest`, use `versions.latest.version`, `versions.latest.status`, and `versions.latest.instance_identity.principal_id`. Do not poll `/versions/None`; if no version can be extracted, list versions first and pick the newest returned version.
+
 Other useful REST operations:
 
 | Purpose | Method and endpoint | Notes |

plugin/skills/microsoft-foundry/foundry-agent/invoke/invoke.md

@@ -120,6 +120,7 @@ Use `session_delete` to release compute resources when done. Undeleted sessions
 | Hosted agent not active | Version still provisioning or failed | Check version status via `agent_get` |
 | Session not found | Invalid ID or expired | Create new session with `session_create` |
 | `424 FailedDependency` or `session_not_ready` | Hosted agent session is still warming up or readiness has not completed | Wait 15-30 seconds, check `session_logstream` if needed, then retry `agent_invoke` with the same `sessionId` if one was returned; if no `sessionId` was returned, retry `session_create`. If this persists across 3+ retries (with exponential backoff: 15s, 30s, 60s), the container likely cannot start within the readiness probe deadline — redeploy with higher CPU/memory (recommended minimum: `1` CPU / `2Gi` for direct-code deployments). Also verify the model deployment name is correct via `model_deployment_get`. |
+| `could not resolve agent service in azd project: no azure.ai.agent service named '<agentName>' found in azure.yaml` from `azd ai agent invoke` | Name mismatch. | Update the agent name to the deployed agent name. |
 | Invocation failed | Model error, timeout, or invalid input | Check agent logs, verify model deployment |
 | Invocations schema mismatch | Request body does not match what the agent expects | Inspect agent's route handler or API docs for the correct JSON schema; do not guess |
 | File operation failed | Session not active or invalid path | Verify session with `session_get` |

tests/utils/agent-runner.ts

@@ -147,12 +147,22 @@ export interface AgentRunConfig {
   takeScreenshot?: {
     predicate: (agentMetadata: AgentMetadata) => boolean
   };
+
+  /**
+   * Optional sink for streaming redacted agent output while a run is in progress.
+   * Intended for long-running Vally tests where waiting for final reports is too slow.
+   */
+  realtimeOutputSink?: AgentRealtimeOutputSink;
 }
 
 interface KeywordOptions {
   caseSensitive?: boolean;
 }
 
+export interface AgentRealtimeOutputSink {
+  write: (chunk: string) => void | Promise<void>;
+}
+
 /** Tracks resources that need cleanup after each test */
 interface RunnerCleanup {
   session?: CopilotSession;
@@ -163,6 +173,162 @@ interface RunnerCleanup {
   agentMetadata?: AgentMetadata;
 }
 
+function stringifyForLog(value: unknown): string {
+  if (value === undefined || value === null) {
+    return "";
+  }
+
+  if (typeof value === "string") {
+    return value;
+  }
+
+  try {
+    return JSON.stringify(value, null, 2);
+  } catch {
+    return String(value);
+  }
+}
+
+function truncateForLog(value: string, maxLength = 4000): string {
+  if (value.length <= maxLength) {
+    return value;
+  }
+
+  return `${value.slice(0, maxLength)}... (truncated)`;
+}
+
+function createRealtimeOutputFlusher(sink: AgentRealtimeOutputSink | undefined) {
+  const messageIdsWithDeltas = new Set<string>();
+  const toolStartedAtByCallId = new Map<string, string>();
+  let pendingWrite = Promise.resolve();
+
+  function enqueue(chunk: string): void {
+    if (!sink || !chunk) {
+      return;
+    }
+
+    const redacted = redactSecrets(chunk);
+    pendingWrite = pendingWrite
+      .then(async () => {
+        await sink.write(redacted);
+      })
+      .catch((error: unknown) => {
+        if (process.env.DEBUG) {
+          console.error("Failed to write realtime agent output:", error);
+        }
+      });
+  }
+
+  function flushEvent(event: SessionEvent): void {
+    const timestamp = event.timestamp ?? new Date().toISOString();
+
+    if (event.type === "assistant.message_delta") {
+      const messageId = event.data.messageId as string | undefined;
+      const deltaContent = event.data.deltaContent as string | undefined;
+      if (!deltaContent) {
+        return;
+      }
+
+      if (messageId && !messageIdsWithDeltas.has(messageId)) {
+        messageIdsWithDeltas.add(messageId);
+        enqueue(`\n[${timestamp}] assistant.message_delta\n`);
+      }
+      enqueue(deltaContent);
+      return;
+    }
+
+    if (event.type === "assistant.message") {
+      const messageId = event.data.messageId as string | undefined;
+      const content = event.data.content as string | undefined;
+      if (!content) {
+        return;
+      }
+
+      if (messageId && messageIdsWithDeltas.has(messageId)) {
+        enqueue("\n\n");
+        return;
+      }
+
+      enqueue(`\n[${timestamp}] assistant.message\n${content}\n\n`);
+      return;
+    }
+
+    if (event.type === "tool.execution_start") {
+      const toolName = event.data.toolName as string | undefined;
+      const toolCallId = event.data.toolCallId as string | undefined;
+      if (toolCallId) {
+        toolStartedAtByCallId.set(toolCallId, event.timestamp);
+      }
+
+      const args = truncateForLog(stringifyForLog(event.data.arguments));
+      enqueue([
+        `\n[${timestamp}] tool.execution_start ${toolName ?? "unknown"}`,
+        toolCallId ? `toolCallId: ${toolCallId}` : undefined,
+        args ? `arguments:\n${args}` : undefined,
+        "",
+      ].filter((line) => line !== undefined).join("\n"));
+      return;
+    }
+
+    if (event.type === "tool.execution_complete") {
+      const toolCallId = event.data.toolCallId as string | undefined;
+      const startedAt = toolCallId ? toolStartedAtByCallId.get(toolCallId) : undefined;
+      const durationMs = startedAt
+        ? new Date(timestamp).getTime() - new Date(startedAt).getTime()
+        : undefined;
+      const success = event.data.success as boolean | undefined;
+      const payload = success ? event.data.result : event.data.error;
+      const payloadText = truncateForLog(stringifyForLog(payload));
+
+      enqueue([
+        `\n[${timestamp}] tool.execution_complete`,
+        toolCallId ? `toolCallId: ${toolCallId}` : undefined,
+        `success: ${success === true}`,
+        durationMs !== undefined && Number.isFinite(durationMs) ? `durationMs: ${durationMs}` : undefined,
+        payloadText ? `result:\n${payloadText}` : undefined,
+        "",
+      ].filter((line) => line !== undefined).join("\n"));
+      return;
+    }
+
+    if (event.type === "skill.invoked") {
+      const skillName = event.data.name as string | undefined;
+      const skillPath = event.data.path as string | undefined;
+      enqueue([
+        `\n[${timestamp}] skill.invoked ${skillName ?? "unknown"}`,
+        skillPath ? `path: ${skillPath}` : undefined,
+        "",
+      ].filter((line) => line !== undefined).join("\n"));
+      return;
+    }
+
+    if (event.type === "session.skills_loaded") {
+      const skills = event.data.skills as Array<{
+        name?: string;
+        source?: string;
+        enabled?: boolean;
+        path?: string;
+      }>;
+      const skillLines = skills
+        .map((skill) => [
+          `- ${skill.name ?? "unknown"}`,
+          skill.source ? `source=${skill.source}` : undefined,
+          skill.enabled !== undefined ? `enabled=${skill.enabled}` : undefined,
+          skill.path ? `path=${skill.path}` : undefined,
+        ].filter((part) => part !== undefined).join(" "))
+        .join("\n");
+
+      enqueue(`\n[${timestamp}] session.skills_loaded\n${skillLines}\n\n`);
+    }
+  }
+
+  async function drain(): Promise<void> {
+    await pendingWrite;
+  }
+
+  return { flushEvent, drain };
+}
+
 /**
  * Extract file-system paths from the serialized arguments of a tool call that
  * reference the given `skillDirectory`. Checks common argument keys
@@ -643,6 +809,7 @@ export function useAgentRunner(agentRunnerConfig: AgentRunnerConfig) {
     const FOLLOW_UP_TIMEOUT = config.followUpTimeout ?? 1800000; // 30 minutes by default
 
     let isComplete = false;
+    const realtimeOutput = createRealtimeOutputFlusher(config.realtimeOutputSink);
 
     const entry: RunnerCleanup = { config };
     currentCleanups.push(entry);
@@ -695,6 +862,9 @@ export function useAgentRunner(agentRunnerConfig: AgentRunnerConfig) {
       const session = await client.createSession({
         model: model,
         onPermissionRequest: approveAll,
+        // Keep integration tests isolated from user-level skills/config such as
+        // ~/.agents/skills so test runs only exercise the generated output.
+        enableConfigDiscovery: false,
         skillDirectories: noSkills ? [] : [skillDirectory],
         disabledSkills: disabledSkills,
         mcpServers: {
@@ -724,6 +894,7 @@ export function useAgentRunner(agentRunnerConfig: AgentRunnerConfig) {
           }
 
           agentMetadata.events.push(event);
+          realtimeOutput.flushEvent(event);
 
           if (config.shouldEarlyTerminate?.(agentMetadata)) {
             isComplete = true;
@@ -847,6 +1018,7 @@ export function useAgentRunner(agentRunnerConfig: AgentRunnerConfig) {
       if (!isTest()) {
         await cleanup();
       }
+      await realtimeOutput.drain();
     }
   }
 
@@ -862,6 +1034,24 @@ function buildShareFilePath(testName: string): string {
   return path.join(testCaseArtifactsDir, `agent-metadata-${new Date().toISOString().replace(/[:.]/g, "-")}.md`);
 }
 
+export function createRealtimeLogSink(testName: string): AgentRealtimeOutputSink & { filePath: string } {
+  const testCaseArtifactsDir = buildTestCaseDirPath(testName);
+  fs.mkdirSync(testCaseArtifactsDir, { recursive: true });
+  const filePath = path.join(testCaseArtifactsDir, "agent-realtime.log");
+  fs.writeFileSync(
+    filePath,
+    `# Agent realtime log\n\nStarted: ${new Date().toISOString()}\n\n`,
+    "utf-8",
+  );
+
+  return {
+    filePath,
+    write(chunk: string): void {
+      fs.appendFileSync(filePath, chunk, "utf-8");
+    },
+  };
+}
+
 export async function createMarkdownReport(testName: string, config: AgentRunConfig, agentMetadata: AgentMetadata): Promise<void> {
   writeMarkdownReport(testName, config, agentMetadata);
 }
@@ -1074,4 +1264,4 @@ function sanitizeFileName(name: string): string {
     .replace(/-+/g, "-")            // Collapse multiple dashes
     .replace(/_+/g, "_")            // Collapse multiple underscores
     .substring(0, 200);             // Limit length
-}
+}

tests/vally/vally-executor.ts

@@ -1,7 +1,7 @@
 import type { Executor, ExecutorOptions, ExecutorRegistry, Stimulus, Trajectory, TrajectoryEvent } from "@microsoft/vally";
 import { computeMetrics } from "@microsoft/vally";
 import type { AgentMetadata, AgentRunConfig } from "../utils/agent-runner.ts";
-import { useAgentRunner, createMarkdownReport } from "../utils/agent-runner.ts";
+import { useAgentRunner, createMarkdownReport, createRealtimeLogSink } from "../utils/agent-runner.ts";
 import { listSkills } from "../utils/skill-loader.ts";
 import { getEarlyTerminateCondition, getFollowUp, getSkillName, getSystemPrompt, getTakeScreenshotCondition } from "./tag-helpers.ts";
 import { normalizeTestName } from "./utils.ts";
@@ -31,6 +31,8 @@ export class IntegrationTestAgentRunner implements Executor {
     const systemPrompt = getSystemPrompt(tags);
     const { takeScreenshot } = getTakeScreenshotCondition(tags);
     const timeout = options.timeout;
+    const realtimeLogSink = createRealtimeLogSink(normalizedTestName);
+    console.log(`Agent realtime log: ${realtimeLogSink.filePath}`);
 
     const runConfig: AgentRunConfig = {
       workspace: workDir,
@@ -42,6 +44,7 @@ export class IntegrationTestAgentRunner implements Executor {
       systemPrompt: systemPrompt,
       followUpTimeout: timeout,
       takeScreenshot: takeScreenshot,
+      realtimeOutputSink: realtimeLogSink,
       // Always make our agent runner preserve workspace.
       // vally will delete the test workspace by default.
       preserveWorkspace: true