[WIP] feat: improve the hosted agent getting started with coding agent by anchenyi · Pull Request #2566 · microsoft/GitHub-Copilot-for-Azure

anchenyi · 2026-06-05T08:46:02Z

Description

Continue development based on #2486 and #2435

Checklist

Tests pass locally (cd tests && npm test)
Title has one of the prefixes: fix:, feat:, feature:, chore:, misc:, test:, eval:
If modifying skill descriptions: verified routing correctness with integration tests (In tests/, npm run test:integration -- <skill> or npm run test:vally -- --skill <skill>)

Related Issues

…eploy workflows - Streamline create-hosted.md and deploy.md with clearer step-by-step flows - Add reference docs for azd-ai-cli, local-run, and tools - Add environment verification scripts (PowerShell and Bash) - Update direct-code-deployment reference with consistent CPU/memory defaults - Update SKILL.md description for better routing Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

The previous defaults of 0.5 CPU / 1Gi often cause session_not_ready timeouts during cold start. Update all prose and JSON examples to use 1 CPU / 2Gi as the recommended default. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot

Pull request overview

This PR continues improving the microsoft-foundry skill’s “getting started” experience for hosted agents (coding agent / azd ai), while adding better observability for long-running Vally integration tests and introducing Foundry E2E eval suites.

Changes:

Add realtime (redacted) streaming agent output to Vally runs, persisted to per-test artifacts.
Restructure Foundry hosted-agent docs to emphasize direct code deployment and streamline create/deploy/invoke guidance; add environment verification scripts and new reference docs.
Add two new Foundry E2E Vally eval configs and update .gitignore for local temp outputs.

Reviewed changes

Copilot reviewed 15 out of 16 changed files in this pull request and generated 9 comments.

Show a summary per file

File	Description
`tests/vally/vally-executor.ts`	Wires a realtime log sink into the Vally integration executor and prints the log path.
`tests/utils/agent-runner.ts`	Adds a realtime output sink + flusher, drains pending writes, and disables config discovery for test isolation.
`plugin/skills/microsoft-foundry/SKILL.md`	Updates routing description and refines sub-skill descriptions for deploy/create.
`plugin/skills/microsoft-foundry/project/create/create-foundry-project.md`	Updates guidance to auto-generate a project name when missing.
`plugin/skills/microsoft-foundry/foundry-agent/invoke/invoke.md`	Expands troubleshooting guidance for `session_not_ready` and adds an `azd ai agent invoke` name-mismatch row.
`plugin/skills/microsoft-foundry/foundry-agent/deploy/references/direct-code-deployment.md`	Repositions direct-code deployment as the default, increases CPU/memory defaults, and adds RBAC + capability host verification steps.
`plugin/skills/microsoft-foundry/foundry-agent/deploy/deploy.md`	Major rewrite to streamline hosted/prompt flows and prefer direct code deployment; adds failure-mode table and handoffs.
`plugin/skills/microsoft-foundry/foundry-agent/create/scripts/verify-environment.sh`	New bash environment verification script for `azd ai` hosted-agent setup.
`plugin/skills/microsoft-foundry/foundry-agent/create/scripts/verify-environment.ps1`	New PowerShell environment verification script for `azd ai` hosted-agent setup.
`plugin/skills/microsoft-foundry/foundry-agent/create/references/tools.md`	New reference for toolboxes/connections and wiring MCP endpoints into agent deployments.
`plugin/skills/microsoft-foundry/foundry-agent/create/references/local-run.md`	New reference for local inner-loop (`azd ai agent run` / `invoke --local`) and common pitfalls.
`plugin/skills/microsoft-foundry/foundry-agent/create/references/azd-ai-cli.md`	New reference explaining `azd ai` mental model, key files, and common errors.
`plugin/skills/microsoft-foundry/foundry-agent/create/create-hosted.md`	Streamlines hosted agent creation workflow around `azd ai` and local iteration.
`evals/microsoft-foundry/test-new-project.yaml`	Adds a new-project E2E eval stimulus for microsoft-foundry.
`evals/microsoft-foundry/test-eval.yaml`	Adds an existing-project E2E eval stimulus for microsoft-foundry.
`.gitignore`	Ignores `tmp/` for local scratch outputs.

…testing

github-actions · 2026-06-06T01:28:52Z

Details

# 🔍 Token Analysis Report

@github-copilot-for-azure/scripts@1.0.0 tokens
node --import tsx src/tokens/cli.ts compare --base origin/main --head HEAD --markdown

fatal: path 'plugin/skills/microsoft-foundry/foundry-agent/create/references/azd-ai-cli.md' exists on disk, but not in 'origin/main'
fatal: path 'plugin/skills/microsoft-foundry/foundry-agent/create/references/local-run.md' exists on disk, but not in 'origin/main'
fatal: path 'plugin/skills/microsoft-foundry/foundry-agent/create/references/tools.md' exists on disk, but not in 'origin/main'

📊 Token Change Report

Comparing origin/main → HEAD

Summary

Metric	Value
📉 Total Change	-3,611 tokens (-14%)
Before	26,457 tokens
After	22,846 tokens
Files Changed	9

Changed Files

File	Before	After	Change
`plugin/skills/microsoft-foundry/foundry-agent/deploy/deploy.md`	8,924	4,787	-4,137 (-46%)
`plugin/skills/microsoft-foundry/foundry-agent/deploy/references/direct-code-deployment.md`	3,690	-	-3690
`plugin/skills/microsoft-foundry/foundry-agent/create/create-hosted.md`	5,262	3,083	-2,179 (-41%)
`plugin/skills/microsoft-foundry/foundry-agent/create/references/tools.md`	-	2,087	+2087
`plugin/skills/microsoft-foundry/foundry-agent/create/references/local-run.md`	-	1,924	+1924
`plugin/skills/microsoft-foundry/foundry-agent/create/references/azd-ai-cli.md`	-	1,891	+1891
`plugin/skills/microsoft-foundry/foundry-agent/invoke/invoke.md`	2,165	2,533	+368 (+17%)
`plugin/skills/microsoft-foundry/project/create/create-foundry-project.md`	1,400	1,587	+187 (+13%)
`plugin/skills/microsoft-foundry/SKILL.md`	5,016	4,954	-62 (-1%)

@github-copilot-for-azure/scripts@1.0.0 tokens
node --import tsx src/tokens/cli.ts check --markdown

📊 Token Limit Check Report

Checked: 714 files
Exceeded: 101 files

⚠️ Files Exceeding Token Limits

File	Tokens	Limit	Over By
`.github/skills/analyze-skill-issues/SKILL.md`	2109	500	+1609
`.github/skills/analyze-test-run/SKILL.md`	2471	500	+1971
`.github/skills/file-test-bug/SKILL.md`	628	500	+128
`.github/skills/sensei/README.md`	3531	2000	+1531
`.github/skills/sensei/SKILL.md`	3026	500	+2526
`.github/skills/sensei/references/EXAMPLES.md`	3701	2000	+1701
`.github/skills/sensei/references/LOOP.md`	4169	2000	+2169
`.github/skills/sensei/references/SCORING.md`	4299	2000	+2299
`.github/skills/skill-authoring/SKILL.md`	839	500	+339
`.github/skills/vally-eval/SKILL.md`	1608	500	+1108
`plugin/skills/airunway-aks-setup/SKILL.md`	1025	500	+525
`plugin/skills/appinsights-instrumentation/SKILL.md`	937	500	+437
`plugin/skills/azure-ai/SKILL.md`	820	500	+320
`plugin/skills/azure-aigateway/SKILL.md`	1261	500	+761
`plugin/skills/azure-aigateway/references/policies.md`	2342	2000	+342
`plugin/skills/azure-cloud-migrate/SKILL.md`	1085	500	+585
`plugin/skills/azure-cloud-migrate/references/services/container-apps/cloudrun-deployment-guide.md`	2029	2000	+29
`plugin/skills/azure-cloud-migrate/references/services/container-apps/deployment-guide.md`	2458	2000	+458
`plugin/skills/azure-cloud-migrate/references/services/container-apps/fargate-deployment-guide.md`	2587	2000	+587
`plugin/skills/azure-cloud-migrate/references/services/container-apps/spring-deployment-guide.md`	3871	2000	+1871
`plugin/skills/azure-cloud-migrate/references/services/functions/lambda-to-functions.md`	2600	2000	+600
`plugin/skills/azure-cloud-migrate/references/services/functions/runtimes/javascript.md`	2181	2000	+181
`plugin/skills/azure-compliance/SKILL.md`	1188	500	+688
`plugin/skills/azure-compute/SKILL.md`	821	500	+321
`plugin/skills/azure-compute/workflows/essential-machine-management/references/emm-enable-flow.md`	2344	2000	+344
`plugin/skills/azure-compute/workflows/vm-troubleshooter/vm-troubleshooter.md`	2509	2000	+509
`plugin/skills/azure-deploy/SKILL.md`	1645	500	+1145
`plugin/skills/azure-deploy/references/pre-deploy-checklist.md`	4692	2000	+2692
`plugin/skills/azure-deploy/references/recipes/azd/errors.md`	4004	2000	+2004
`plugin/skills/azure-deploy/references/troubleshooting.md`	2038	2000	+38
`plugin/skills/azure-diagnostics/SKILL.md`	1423	500	+923
`plugin/skills/azure-enterprise-infra-planner/SKILL.md`	911	500	+411
`plugin/skills/azure-enterprise-infra-planner/references/constraints/compute-apps.md`	2022	2000	+22
`plugin/skills/azure-hosted-copilot-sdk/SKILL.md`	1332	500	+832
`plugin/skills/azure-kubernetes/SKILL.md`	2634	500	+2134
`plugin/skills/azure-kubernetes/azure-kubernetes-automatic-readiness/SKILL.md`	3690	500	+3190
`plugin/skills/azure-kusto/SKILL.md`	2152	500	+1652
`plugin/skills/azure-messaging/SKILL.md`	821	500	+321
`plugin/skills/azure-prepare/SKILL.md`	3375	500	+2875
`plugin/skills/azure-prepare/references/aspire.md`	4617	2000	+2617
`plugin/skills/azure-prepare/references/plan-template.md`	2560	2000	+560
`plugin/skills/azure-prepare/references/recipes/azd/aspire.md`	2275	2000	+275
`plugin/skills/azure-prepare/references/recipes/azd/terraform.md`	3555	2000	+1555
`plugin/skills/azure-prepare/references/research.md`	2274	2000	+274
`plugin/skills/azure-prepare/references/resources-limits-quotas.md`	3322	2000	+1322
`plugin/skills/azure-prepare/references/security.md`	2147	2000	+147
`plugin/skills/azure-prepare/references/services/functions/bicep.md`	3043	2000	+1043
`plugin/skills/azure-prepare/references/services/functions/templates/recipes/composition.md`	2813	2000	+813
`plugin/skills/azure-prepare/references/services/functions/terraform.md`	3404	2000	+1404
`plugin/skills/azure-quotas/SKILL.md`	3006	500	+2506
`plugin/skills/azure-quotas/references/commands.md`	2644	2000	+644
`plugin/skills/azure-reliability/SKILL.md`	5922	500	+5422
`plugin/skills/azure-reliability/references/configure-multi-region.md`	4729	2000	+2729
`plugin/skills/azure-reliability/references/services/app-service/reliability.md`	2591	2000	+591
`plugin/skills/azure-resource-lookup/SKILL.md`	1367	500	+867
`plugin/skills/azure-resource-visualizer/SKILL.md`	2122	500	+1622
`plugin/skills/azure-storage/SKILL.md`	1228	500	+728
`plugin/skills/azure-upgrade/SKILL.md`	1542	500	+1042
`plugin/skills/azure-upgrade/references/languages/java/INSTRUCTION.md`	2893	2000	+893
`plugin/skills/azure-upgrade/references/languages/java/package-specific/com.microsoft.azure.management.md`	2428	2000	+428
`plugin/skills/azure-upgrade/references/languages/java/templates/PLAN_TEMPLATE.md`	2411	2000	+411
`plugin/skills/azure-upgrade/references/languages/java/templates/PROGRESS_TEMPLATE.md`	2315	2000	+315
`plugin/skills/azure-upgrade/references/languages/java/templates/SUMMARY_TEMPLATE.md`	2190	2000	+190
`plugin/skills/azure-upgrade/references/services/functions/automation.md`	3463	2000	+1463
`plugin/skills/azure-upgrade/references/services/functions/consumption-to-flex.md`	2773	2000	+773
`plugin/skills/azure-validate/SKILL.md`	950	500	+450
`plugin/skills/entra-agent-id/SKILL.md`	4001	500	+3501
`plugin/skills/entra-app-registration/SKILL.md`	2070	500	+1570
`plugin/skills/entra-app-registration/references/api-permissions.md`	2545	2000	+545
`plugin/skills/entra-app-registration/references/cli-commands.md`	2211	2000	+211
`plugin/skills/entra-app-registration/references/console-app-example.md`	2752	2000	+752
`plugin/skills/entra-app-registration/references/oauth-flows.md`	2375	2000	+375
`plugin/skills/microsoft-foundry/SKILL.md`	4954	500	+4454
`plugin/skills/microsoft-foundry/finetuning/SKILL.md`	1375	500	+875
`plugin/skills/microsoft-foundry/foundry-agent/create/create-hosted.md`	3083	2000	+1083
`plugin/skills/microsoft-foundry/foundry-agent/create/references/foundry-tool-catalog.md`	10856	2000	+8856
`plugin/skills/microsoft-foundry/foundry-agent/create/references/toolbox-reference.md`	2913	2000	+913
`plugin/skills/microsoft-foundry/foundry-agent/create/references/tools.md`	2087	2000	+87
`plugin/skills/microsoft-foundry/foundry-agent/create/references/use-toolbox-in-hosted-agent.md`	4527	2000	+2527
`plugin/skills/microsoft-foundry/foundry-agent/deploy/deploy.md`	4787	2000	+2787
`plugin/skills/microsoft-foundry/foundry-agent/eval-datasets/eval-datasets.md`	2863	2000	+863
`plugin/skills/microsoft-foundry/foundry-agent/eval-datasets/references/generate-seed-dataset.md`	2196	2000	+196
`plugin/skills/microsoft-foundry/foundry-agent/eval-datasets/references/trace-to-dataset.md`	4325	2000	+2325
`plugin/skills/microsoft-foundry/foundry-agent/invocations-ws/invocations-ws.md`	2593	2000	+593
`plugin/skills/microsoft-foundry/foundry-agent/invocations-ws/references/invocations-ws-protocol.md`	2009	2000	+9
`plugin/skills/microsoft-foundry/foundry-agent/invoke/invoke.md`	2533	2000	+533
`plugin/skills/microsoft-foundry/foundry-agent/observe/observe.md`	3856	2000	+1856
`plugin/skills/microsoft-foundry/foundry-agent/observe/references/continuous-eval.md`	3855	2000	+1855
`plugin/skills/microsoft-foundry/foundry-agent/observe/references/evaluate-step.md`	2175	2000	+175
`plugin/skills/microsoft-foundry/foundry-agent/observe/references/evaluation-suite-generation.md`	3084	2000	+1084
`plugin/skills/microsoft-foundry/foundry-agent/trace/references/kql-templates.md`	2701	2000	+701
`plugin/skills/microsoft-foundry/models/deploy-model/SKILL.md`	1640	500	+1140
`plugin/skills/microsoft-foundry/models/deploy-model/capacity/SKILL.md`	1739	500	+1239
`plugin/skills/microsoft-foundry/models/deploy-model/customize/SKILL.md`	2235	500	+1735
`plugin/skills/microsoft-foundry/models/deploy-model/customize/references/customize-workflow.md`	3335	2000	+1335
`plugin/skills/microsoft-foundry/models/deploy-model/preset/SKILL.md`	1226	500	+726
`plugin/skills/microsoft-foundry/models/deploy-model/preset/references/preset-workflow.md`	5534	2000	+3534
`plugin/skills/microsoft-foundry/quota/quota.md`	2288	2000	+288
`plugin/skills/microsoft-foundry/quota/references/capacity-planning.md`	2080	2000	+80
`plugin/skills/microsoft-foundry/references/agent-metadata-contract.md`	2075	2000	+75
`plugin/skills/microsoft-foundry/references/sdk/foundry-sdk-py.md`	2162	2000	+162

Consider moving content to references/ subdirectories.

Automated token analysis. See skill authoring guidelines for best practices.

* fix: restore eval * fix: comment * fix: use azd eval

zhenjiao-ms and others added 2 commits June 2, 2026 16:23

Copilot AI review requested due to automatic review settings June 5, 2026 08:46

anchenyi requested review from RickWinter, XOEEst, XiaofuHuang, ankitbko, jugonzales, tendau and vebudumu as code owners June 5, 2026 08:46

Copilot started reviewing on behalf of anchenyi June 5, 2026 08:46 View session

Copilot AI reviewed Jun 5, 2026

View reviewed changes

feat: improve local run, deploy, and invoke. Add temp eval infra for …

cd5c2a9

…testing

anchenyi force-pushed the dev/improve-getting-started branch from 2befd3e to cd5c2a9 Compare June 5, 2026 12:18

github-actions Bot mentioned this pull request Jun 5, 2026

[repo-status] Weekly Repo Status — June 5, 2026 #2573

Closed

anchenyi added 2 commits June 6, 2026 08:00

refactor: deploy sub skill and minors

234d0f3

Merge branch 'main' into dev/improve-getting-started

ea25cbe

anchenyi and others added 6 commits June 8, 2026 07:49

feat: swtich to acr as default, improve wording

c33b248

feat: direct code as default

b63e4d1

chore: align env with agent.yaml

c1de7d6

fix: reduce golden time (#2593)

5a0fd6a

fix: venv and arm id

d64e2fa

fix: restore eval (#2606)

af0f2a8

* fix: restore eval * fix: comment * fix: use azd eval

anchenyi closed this Jun 9, 2026

This was referenced Jun 11, 2026

[repo-status] Weekly Repo Status — June 11, 2026 #2626

Closed

[repo-status] Weekly Repo Status — June 12, 2026 #2631

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[WIP] feat: improve the hosted agent getting started with coding agent#2566

[WIP] feat: improve the hosted agent getting started with coding agent#2566
anchenyi wants to merge 11 commits into
mainfrom
dev/improve-getting-started

anchenyi commented Jun 5, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Jun 6, 2026 •

edited

Loading

📊 Token Change Report

Summary

Changed Files

📊 Token Limit Check Report

⚠️ Files Exceeding Token Limits

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

anchenyi commented Jun 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Checklist

Related Issues

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Jun 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

📊 Token Change Report

Summary

Changed Files

📊 Token Limit Check Report

⚠️ Files Exceeding Token Limits

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

anchenyi commented Jun 5, 2026 •

edited

Loading

github-actions Bot commented Jun 6, 2026 •

edited

Loading