feat: improve the hosted agent getting started with coding

.gitignore

@@ -359,4 +359,5 @@ dashboard/**/dist/
 
 # Local vally eval outputs
 results/
+tmp/
 .vally-workspaces/

plugin/skills/microsoft-foundry/foundry-agent/agent-optimizer/references/eval-yaml.md

@@ -1,6 +1,6 @@
 # eval.yaml Guidance
 
-Create `eval.yaml` directly when the conversation or `.foundry/agent-metadata*.yaml` already selected the dataset/evaluators. Otherwise ask whether to run `azd ai agent eval init` or let optimize use built-in defaults.
+Create `eval.yaml` directly when the conversation or `.foundry/agent-metadata*.yaml` already selected the dataset/evaluators. Otherwise ask whether to run `azd ai agent eval generate` or let optimize use built-in defaults.
 
 ## Include
 

plugin/skills/microsoft-foundry/foundry-agent/create/references/azd-ai-cli.md

@@ -0,0 +1,149 @@
+# azd ai CLI Reference
+
+Core mental model for the `azd ai agent` extension. Use this when you need to understand command surface, file layout, or where a given setting lives.
+
+## CLI surface
+
+```bash
+azd ai project show                  # which Foundry project endpoint is active
+azd ai agent show                    # is the agent deployed? what version?
+azd ai agent doctor                  # full health check, suggests fixes
+
+azd ai agent sample list             # curated catalog -- pick a manifestUrl
+azd ai agent init -m <manifestUrl>   # scaffold from a sample
+azd ai agent init --from-code        # scaffold from existing source
+
+azd ai agent run                     # start the agent on localhost:8088
+azd ai agent invoke "<msg>"          # remote invoke (billed; gated)
+azd ai agent invoke --local "<msg>"  # local invoke (no billing)
+
+azd provision                        # core azd; creates Foundry project + infra
+azd deploy                           # core azd; packages + registers new agent version
+azd ai agent endpoint update         # patch agentEndpoint / agentCard in place
+
+azd ai agent connection list / show / create / update / delete
+azd ai toolbox list / show / create / update / delete
+azd ai toolbox connection add / remove / list
+azd ai toolbox version list
+
+azd ai agent files list / show / upload / download / delete / stat / mkdir
+azd ai agent sessions list / show / create / update / delete
+azd ai agent monitor                 # per-session log stream (SSE)
+
+azd ai agent eval generate / run / show / update / list
+azd ai agent optimize / optimize status / optimize apply / optimize deploy / optimize cancel
+```
+
+Read-only commands accept `--output json` and never require `--force`. Write commands are gated by a confirmation envelope (see "Confirmation envelope" below).
+
+## Two files, two schemas
+
+After `azd ai agent init`, every hosted agent is defined by **two** files plus the active azd env. Putting a field in the wrong file is the most common scaffolding failure.
+
+| File | What it holds |
+|------|---------------|
+| `<service-dir>/agent.yaml` | The flat `ContainerAgent`: `kind`, `name`, `protocols`, `environment_variables`, `agentEndpoint`, `agentCard`, `code_configuration` / `image`, container `resources` (cpu, memory). |
+| `azure.yaml services.<name>.config` | Model deployments, project connections, toolboxes, tool resources, container settings, `startupCommand`. |
+| `.azure/<env>/.env` (`azd env set`) | Secrets and `PARAM_<CONN>_<KEY>` credential values referenced from `azure.yaml`. |
+
+`azd deploy` reads `agent.yaml` and creates a new immutable agent version. `azd provision` reads `config.deployments[]` and `config.connections[]` and applies them via Bicep.
+
+`agent.manifest.yaml` (the file passed to `-m`) is the seed format -- it is NOT on disk after init. Init splits its `parameters:` / `resources:` blocks across the three files above. Don't reintroduce the `template:` wrapper into `agent.yaml`.
+
+### Minimal `agent.yaml` (hosted)
+
+```yaml
+# yaml-language-server: $schema=https://raw.githubusercontent.com/microsoft/AgentSchema/refs/heads/main/schemas/v1.0/ContainerAgent.yaml
+kind: hosted
+name: my-agent
+protocols:
+  - protocol: responses
+    version: "1.0.0"
+resources:
+  cpu: "0.25"
+  memory: "0.5Gi"
+environment_variables:
+  - name: AZURE_AI_MODEL_DEPLOYMENT_NAME
+    value: ${AZURE_AI_MODEL_DEPLOYMENT_NAME}
+code_configuration:
+  runtime: python_3_13
+  entry_point: app.py
+  dependency_resolution: remote_build   # or "bundled"
+```
+
+- `protocols` -- `responses` (OpenAI), `invocations` (A2A). Editing requires `azd deploy`.
+- `resources` -- valid tiers: `0.25/0.5Gi`, `1/2Gi`, `2/4Gi`.
+- `environment_variables` -- `${VAR}` resolves from the active azd env. Not for secrets.
+- `code_configuration` present -> direct code deploy (ZIP, Foundry builds). Absent -> container/ACR deploy (Dockerfile + `docker:` in `azure.yaml`). `image:` skips the Dockerfile build.
+- In non-interactive mode, `azd ai agent init` defaults to container deploy. Pass `--deploy-mode code --runtime <runtime> --entry-point <file>` during init to get `code_configuration`.
+- `agentEndpoint` / `agentCard` -- patch in place with `azd ai agent endpoint update` (no new version).
+
+### Minimal `azure.yaml` service config
+
+```yaml
+services:
+  my-agent:
+    project: ./src/my-agent
+    host: azure.ai.agent
+    language: python
+    config:
+      startupCommand: "python -m main"
+      container:
+        resources:
+          cpu: "0.5"
+          memory: "1Gi"
+      deployments:
+        - name: AZURE_AI_MODEL_DEPLOYMENT_NAME
+          model:
+            name: gpt-4.1-mini
+            format: OpenAI
+            version: "2024-04-09"
+          sku:
+            name: GlobalStandard
+            capacity: 50
+      connections: [...]        # see tools.md
+      toolboxes: [...]          # see tools.md
+```
+
+- `startupCommand` -- what `azd ai agent run` executes locally. Auto-detected at init.
+- `config.container.resources` -- deployment-time CPU/memory. Keep this aligned with `agent.yaml resources`; this value can override the agent file.
+- `deployments[]` -- model deployments provisioned via Bicep. `name` is the env var the agent reads.
+- `connections[]` -- project connections provisioned via Bicep. Use `PARAM_<CONN>_<KEY>` env-var references for secrets.
+- `toolboxes[]` -- declarative record of intent; today you still drive the toolbox CLI to materialize them on Foundry. See [tools](tools.md).
+
+## State (azd env vars)
+
+| Variable | Read by | Where to set |
+|----------|---------|--------------|
+| `AZURE_AI_PROJECT_ENDPOINT` | Every `azd ai agent` command | `azd env set` or `azd ai project show` |
+| `AZURE_AI_PROJECT_ID` | `azd ai agent show` (playground URL) | `azd env set` |
+| `AZURE_SUBSCRIPTION_ID`, `AZURE_LOCATION` | `azd provision` | `azd config get defaults` |
+| `AGENT_<SVC>_NAME` / `_VERSION` / `_<PROTO>_ENDPOINT` | Auto-written by deploy | Auto |
+| `PARAM_<CONN>_<KEY>` | Connection credentials in `azure.yaml` | `azd env set` |
+
+Manage with `azd env get-values`, `azd env set`, `azd env list`, `azd env new`, `azd env select`.
+
+The platform also injects `FOUNDRY_*` and `AGENT_*` into the running container at runtime. **Never** put these in the agent.yaml environment_variables section.
+
+## Resolving subscription / location
+
+`azd ai project show` returns only the Foundry project endpoint. For subscription / location, try in order:
+
+1. `azd config get defaults`
+2. `azd env get-values`
+3. Ask the user.
+4. Last resort, with explicit consent: `az account list --output json`.
+
+For the Foundry project ARM ID (`--project-id`), ask the user: "New project, or use an existing one?" If existing, ask for the ID and hint where to find it (https://ai.azure.com -> Operate -> Admin). Do NOT shell out to `az cognitiveservices` -- it returns the wrong resource shape.
+
+## Common error codes
+
+- `not_logged_in` / `login_expired` -- ask the user to run `azd auth login`.
+- `missing_project_endpoint` -- run `azd provision`, or `azd env set AZURE_AI_PROJECT_ENDPOINT <url>`.
+- `project_not_found` -- cwd has no `azure.yaml`. Move to project root or run init.
+- `invalid_agent_manifest` -- `agent.yaml` is malformed. Run `azd ai agent doctor` and read the named field.
+- `invalid_connection` -- inspect with `azd ai agent connection show <name>`.
+- `eval_config_invalid` -- `eval.yaml` failed validation. Run `azd ai agent doctor`.
+- `agent_definition_not_found` -- deployed name doesn't match `azure.yaml`. Re-deploy from project root.
+
+Any unfamiliar `code` value is safe to surface verbatim to the user.

plugin/skills/microsoft-foundry/foundry-agent/create/references/local-run.md

@@ -0,0 +1,146 @@
+# Local Run Reference
+
+Use this when iterating on a hosted agent before deploying.
+
+> **Prerequisite:** Local run does NOT require `azd provision` or any deployed Azure infrastructure. The agent runs on your machine and calls the Foundry model endpoint directly using your local credentials (`DefaultAzureCredential` — falls back to `az login` / VS Code identity). You only need a `.env` file in the agent directory with:
+> ```env
+> FOUNDRY_PROJECT_ENDPOINT=https://<account>.services.ai.azure.com/api/projects/<project>
+> AZURE_AI_MODEL_DEPLOYMENT_NAME=<model-deployment-name>
+> ```
+> If you already ran `azd provision`, extract these from `azd env get-values`.
+>
+> **If no project endpoint is available yet**, follow [deploy.md Step 2](../../deploy/deploy.md#step-2----provision-azure-resources-one-time-per-env) to provision or resolve the project, then return here for local iteration before deploying the agent.
+>
+> **Critical: keep `.env` and `azd env` in sync.** `azd ai agent run` injects the active `azd env` values into the agent process before Python loads `.env`. Many samples use `load_dotenv(override=False)`, so an existing process environment value wins over `.env`. If you change the project endpoint or model deployment, update both `.env` and `azd env`:
+> ```bash
+> azd env set FOUNDRY_PROJECT_ENDPOINT "https://<account>.services.ai.azure.com/api/projects/<project>"
+> azd env set AZURE_AI_MODEL_DEPLOYMENT_NAME "<model-deployment-name>"
+> azd env get-values
+> ```
+> A stale `AZURE_AI_MODEL_DEPLOYMENT_NAME` in `azd env` can make local run call the wrong deployment even when `.env` is correct, commonly surfacing as a Foundry responses API `404 Not Found`.
+
+## Prepare the local environment
+
+For Python agents, prepare the environment from the **agent's service source directory** -- the folder that contains `requirements.txt` and `agent.yaml` (typically `<repo>/src/<service-name>/`, not the azd project root). `azd ai agent run` resolves the venv relative to this folder; a `.venv` created in the project root is ignored and azd silently creates a second one without `uv`.
+
+1. `cd` into the service source directory.
+2. Create a venv, for example `python -m venv .venv`.
+3. Activate the venv.
+4. Install `uv` inside the active venv: `python -m pip install uv`.
+5. In the same shell with the service-dir `.venv` activated, run `azd ai agent run` (from any cwd in the project); it installs `requirements.txt` itself and uses `uv` from the active venv for faster Python dependency installation.
+
+> **Important:** The venv must live next to `requirements.txt`, not in the azd project root. Install `uv` before running `azd ai agent run`, and keep that venv activated when running the command; otherwise the local run falls back to slower dependency installation. Do NOT manually run `pip install -r requirements.txt` / `uv pip install -r requirements.txt --prerelease=allow`; let `azd ai agent run` install dependencies.
+
+## Start the agent locally
+
+Activate the service-dir `.venv`, then in that venv run:
+
+```bash
+azd ai agent run
+```
+
+What this does:
+
+1. Resolves the agent service from `azure.yaml` (auto-picks when only one exists).
+2. Detects the project type (Python, .NET, Node.js) from files in the service source dir.
+3. Installs dependencies if needed. For Python, `azd ai agent run` installs `requirements.txt` itself and uses `uv` from the active local environment when available.
+4. Starts the agent in the foreground on `localhost:8088` (default).
+5. Opens **Agent Inspector** in your browser (unless `--no-inspector`).
+
+> First startup takes 30-60 seconds. Wait before sending the first invocation.
+
+`Ctrl+C` stops the agent and clears the saved local session id in an interactive terminal.
+
+For headless or CI runs, pass `--no-inspector` and start the local server in a managed background session that later steps can monitor and stop. Wait for the "Agent ready" message, invoke it from a second command, then stop the same background session before deploying or leaving a temporary workspace.
+
+Do **not** start `azd ai agent run` as a detached process that you cannot monitor or stop (for example, a bare `azd ai agent run ... &`, or a popped PowerShell window on Windows). Keep logs, readiness polling, and the PID/process handle for cleanup.
+
+## Useful flags
+
+| Flag | Purpose |
+|------|---------|
+| `--port <n>` / `-p <n>` | Override the listen port. Useful when 8088 is taken. |
+| `--start-command "<cmd>"` / `-c "<cmd>"` | Override `azure.yaml` and auto-detect. Example: `--start-command "python app.py"`. |
+| `--no-inspector` | Skip opening Agent Inspector. Use in CI / SSH. |
+
+Pass the service name when there are multiple `ai.agent` services:
+
+```bash
+azd ai agent run my-agent
+```
+
+## Where the start command comes from
+
+Resolution order (first non-empty wins):
+
+1. `--start-command` flag.
+2. `azure.yaml services.<name>.config.startupCommand`.
+3. Auto-detected from project type.
+
+Example:
+
+```yaml
+# azure.yaml
+services:
+  my-agent:
+    project: src/my-agent
+    language: python
+    host: azure.ai.agent
+    config:
+      startupCommand: "uvicorn app:app --host 0.0.0.0 --port 4001"
+```
+
+If detection fails and no override is set, `run` errors with the project dir and asks for `--start-command` or `startupCommand`.
+
+## Invoke the local agent
+
+```bash
+azd ai agent invoke --local "hello, are you up?"
+```
+
+Do not use `--output json` with invoke. The invoke command supports `default` and `raw` output only.
+
+If the user did not explicitly specify a prompt, use `"hello, are you up"` for the local smoke test; only verify that the agent can return a response.
+
+Run one representative local invocation before deploying. If the local invocation returns a model `404` or wrong deployment error, check `azd env get-values` before changing code; stale azd env values are the most common cause.
+
+`--local` differs from a remote invoke in:
+
+- Targets `http://localhost:<port>` instead of the Foundry endpoint.
+- Skips the confirmation envelope (no billing, no remote mutation).
+- `--version` is rejected (versions are a remote concept).
+- Named-agent invocation is rejected (only one agent runs locally at a time).
+
+Other useful flags:
+
+| Flag | Purpose |
+|------|---------|
+| `--protocol responses` (default) / `--protocol invocations` | Wire format your agent speaks. |
+| `--input-file request.json` / `-f request.json` | Send a file body instead of a string message. |
+| `--new-session` | Drop the saved local session and start fresh. |
+| `--port <n>` | Match the port you started `run` with. |
+
+After the local invocation completes, stop the `azd ai agent run` process you started before moving on.
+
+## When to graduate to remote
+
+Local dev validates code shape; remote validates infra + identity + Foundry binding. Move to deploy when:
+
+- You changed `agent.yaml` `model:`, `tools:`, `connections:`, or `protocols:`. Those only take effect on the deployed agent.
+- You need to test against real Foundry connections (search indexes, Bing, MCP, A2A) that have no local mock.
+- You are ready to publish a new immutable agent version.
+
+Before proceeding to deploy, clean up the local agent process.
+
+Next step -> [deploy/deploy.md](../../deploy/deploy.md).
+
+## Common failures
+
+| Symptom | Likely cause | Fix |
+|---------|--------------|-----|
+| `could not connect to localhost:<port>` | `run` not started, or wrong port | Start `azd ai agent run`; pass `--port` to `invoke --local` if non-default. |
+| `could not detect project type in <dir>` | Missing project marker file | Set `startupCommand` in `azure.yaml` or pass `--start-command`. |
+| `cannot use --local with a named agent` | Named-agent invoke against localhost | Drop the name; only one local agent at a time. |
+| `cannot use --version with --local` | `--version` is remote-only | Drop `--version`, or remove `--local` to hit the deployed agent. |
+| Inspector never opens | Headless env, or extension install failed | Pass `--no-inspector`, or run `azd extension install azure.ai.inspector`. |
+| Auth / connection errors against Azure services | Local credentials not wired | Expected -- `DefaultAzureCredential` falls back to your `az login` / VS Code identity. Use `azd auth login` if needed. |