Skip to content

Commit 203fcc5

Browse files
howieleungdargilcoCopilotM-Hietalashrutiyer
authored
Merge feature/azure-ai-projects/2.2.0 to Main (#46535)
* Update version to 2.2.0 * Restore tsp-location.yaml * Re-emit Python SDK from TypeSpec commit 94f9262ac1a8 (#46531) Emit from latest commit in feature/foundry-release branch: - Rename job status to 'cancelled' (#42637) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix pylint complaining about too many if-else branch in get_openai_cl… (#46536) * Fix pylint complaining about too many if-else branch in get_openai_client implementation * Resolved comments * Remove user-agent construction tests for sync and async clients * Howie/hosted agent with rbac (#46528) * Add telemetry models and update API view properties for enhanced telemetry support * Enhance sample_session_log_stream.py to verify and assign Azure AI User role for hosted agent identity * rbac * Refactor hosted agent samples to streamline session management and RBAC setup * update rbac util * tsp loc * update hosted agent samples * update recording * refactor test hosted agents samples to handle RBAC complications * change log * samples for async * Update installation instructions to include aiohttp for async samples * update tsp-location.yaml * Updates to README.md and CHANGELOG.md (#46588) * Add Copilot instructions file * fixing is_recording tracing bug (#46607) * Fixed telemetry instrumentor to correctly call is_recording() as a method * adding tests * updating changelog * Update CoPilot instructions * More CoPilot instructions update and README.md * Update instructions README.md * Fix CoPilot instructions. Add pip install black for post-emitter-fixes.cmd * Update CoPilot instructions to handle new FoundryFeaturesOptInKeys value * Formt CoPilot instructions * Update CoPilot instructions * More CoPilot Instructions refinements * Convert CoPilot instructions to skills (#46664) * Update skills readme * update skill * suppress spelling * Agent target traces samples (#46670) * Enhance sample script for Azure AI evaluations Updated the sample script to support additional command-line arguments for trace evaluations, including agent ID and trace IDs. Modified the lookback hours default value and improved the overall structure for better clarity. * Add sample for Azure AI Evaluations with hosted agent This sample demonstrates how to run Azure AI Evaluations against a hosted agent using the azure_ai_target_completions data source, evaluating agents live with built-in quality and safety evaluators. * Revert TRACE_LOOKBACK_HOURS default to 1 hour to match docstring Agent-Logs-Url: https://github.com/Azure/azure-sdk-for-python/sessions/f15ae794-b773-4e8c-860b-5aea55873600 Co-authored-by: shrutiyer <9905402+shrutiyer@users.noreply.github.com> * Add agent eval samples: trace evals and agent-as-target evals Add two new evaluation samples: - sample_evaluations_builtin_with_traces.py: Trace-based evaluation with three modes (client-side App Insights query, server-side agent ID, and explicit trace IDs) - sample_evaluations_agent_as_target.py: Live agent evaluation using azure_ai_target_completions data source Both samples use the azure_ai_evaluator config pattern with builtin intent_resolution and task_adherence evaluators. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Refactor evaluator configuration functions * Delete sdk/evaluation/azure-ai-evaluation/samples/sample_evaluations_agent_as_target.py * Delete sdk/evaluation/azure-ai-evaluation/samples/sample_evaluations_builtin_with_traces.py * Add sample_evaluations_agent_as_target to skip list * Fix pyright error in trace evaluation samples Use list() to narrow args.trace_ids from Optional[List[str]] to List[str] inside the elif guard, resolving pyright reportArgumentType error. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Skip sample_evaluations_agent_as_target in test discovery The sample requires a deployed Azure AI Agent and has no recording, so it must be excluded from the parametrized test_evaluation_samples. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Refactor the sample * pylint fixes * more pylint fixes * rename * rename * one leftover rename * undo naming * fix other samples pylint --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: Shruti Iyer <you@example.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * [azure-ai-projects] Emit SDK from TypeSpec (Data Generation Jobs) (#46667) * Emit SDK from TypeSpec and apply post-emitter fixes (#46673) Rename AgentEndpoint to AgentEndpointConfig per TypeSpec commit f6fde1c. Updated samples to use the new model name. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * [azure-ai-projects] Emit SDK from TypeSpec (evaluator generation jobs, remove isolation_key) (#46704) * Hook up .beta.datasets * Anoter fix for BetaDatasetsOperations * Update changelog. Add docs\subclients.md * Update skill * Update skill * Update skill * Update skill * update skill * [azure-ai-projects] Emit SDK from TypeSpec (code-based agents operations) (#46776) * [azure-ai-projects] Emit SDK from TypeSpec (model and parameter renames) (#46785) * update subclient report * Rename skill readme to README.md (#46778) * [azure-ai-projects] Emit SDK from TypeSpec (class renames) (#46797) * [azure-ai-projects] Remove unused snippet tags from samples (#46802) * Add Toolbox tool-search sample and update changelog and README (#46866) * Add Toolbox tool-search sample and update changelog and README - Introduced `sample_toolboxes_with_search_preview.py` demonstrating Toolbox creation with `ToolboxSearchPreviewTool`. - Updated CHANGELOG.md to include the new sample. - Added Toolbox Search (Preview) to the README.md features list. - Updated assets.json with the correct Tag version. - Enhanced conftest.py to sanitize Entra-ID JWTs and Cognitive Services hostnames. * resolved comments * resolved comment * Update version to 2.3.0, since 2.2.0 will be releases soon (originating from another feature branch) * Add hosted agent creation samples and update utilities (#46932) * Add hosted agent creation samples and update utilities - Added `sample_hosted_agent_create_from_code.py` and `sample_hosted_agent_create_from_code_async.py` for uploading code zips as new versions of hosted agents. - Introduced `select_echo_agent_code_zip` function to choose between bundled and remote build dependencies. - Updated `wait_for_agent_version_active` to support both sync and async variants. - Enhanced `hosted_agents_util.py` with new functionality for loading agent code zips and checking agent status. - Added new echo agent assets including Dockerfile, README, and requirements for deployment. - Updated tests to include new hosted agent creation scenarios for both sync and async samples. Co-authored-by: Copilot <copilot@github.com> * Resolved comment and conflict Co-authored-by: Copilot <copilot@github.com> * Add async samples for hosted agent creation and update usage instructions - Introduced `sample_create_hosted_agent_async.py` for async CRUD operations on hosted agent versions. - Updated existing samples to reference new async creation methods. - Enhanced changelog with new sample additions and clarified prerequisites for hosted agent usage. Co-authored-by: Copilot <copilot@github.com> * Fixed recording test and sanitize log files Co-authored-by: Copilot <copilot@github.com> --------- Co-authored-by: Copilot <copilot@github.com> * Add hosted agent configuration to .env.template and create .gitignore for update-env-vars (#46965) * Update skill * [azure-ai-projects] Emit SDK from TypeSpec (models, routines sub-clients) (#47009) * Update subclients report * More updates to subclients.md report * Add sample for dataset generation with evaluation and update tests (#47039) - Introduced a new sample script `sample_dataset_generation_job_with_evaluation.py` demonstrating end-to-end data generation and evaluation. - Updated `assets.json` with the new tag for the sample. - Modified test configuration to include the new sample in the test suite. - Added header sanitization in `conftest.py` to handle Content-Encoding during tests. * [azure-ai-projects] Emit SDK from TypeSpec (commit 53fbe53) (#47065) * Update sub client report * Add samples for Fabric IQ agent with synchronous and asynchronous cli… (#47058) * Add samples for Fabric IQ agent with synchronous and asynchronous clients, and update test base with Fabric IQ connection ID * Refactor instructions in Fabric IQ agent samples for clarity and consistency * Add external agent CRUD samples (#47081) * Add data generation job samples to azure-ai-projects (#47067) * Add 4 data generation job samples to azure-ai-projects Add four new samples to sdk/ai/azure-ai-projects/samples/datasets/ that cover the four hero scenarios of the 2.2.0a data generation API (`pc.beta.datasets.create_generation_job`), plus a README index: * sample_dataset_generation_job_traces_for_evaluation.py Traces -> Evaluation dataset. The Traces source consumes existing telemetry, so no model_options are required. * sample_dataset_generation_job_traces_for_finetuning.py Traces -> Supervised fine-tuning JSONL files (train_split=0.8 -> two file outputs). Resolves each FileDataGenerationJobOutput's real filename via openai_client.files.retrieve, because the service returns output.filename=None on FT outputs today. * sample_dataset_generation_job_simpleqna_with_dataset_source.py Multi-source SimpleQnA: a seed Dataset combined with an inline Prompt -> Evaluation dataset. Shows that caller-supplied output description and tags propagate onto the generated dataset (the service also auto-adds a data_generation_job_id tag). * sample_dataset_generation_job_simpleqna_for_finetuning.py Azure OpenAI File source -> short-answer + long-answer fine-tuning files (train_split=0.8). Polls the uploaded file until it reaches status=processed before submitting the job, otherwise the service rejects the reference with `must point to a completed file import`. All four match the style of sample_dataset_generation_job_with_evaluation.py: MIT header, parenthesized multi-with, dot-style polling, TERMINAL_STATUSES set, deletes the data generation job at the end but leaves the generated output (dataset or files) for the user to inspect. The SimpleQnA samples also delete the temporary inputs they upload (seed dataset / Azure OpenAI file). Each sample preflights `len(output_name) <= 50` because the service rejects longer output names with a 400 invalidPayload, and uses a short unique run id suffix so repeated runs do not collide. All four were live-validated end-to-end against a real Foundry project. README.md indexes the datasets folder, mirrors evaluations/README.md structure, and calls out per-sample environment variable requirements (FOUNDRY_AGENT_NAME for traces samples, FOUNDRY_MODEL_NAME for simple_qna samples, openai for samples that interact with Azure OpenAI files). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address PR review feedback on data generation samples - Add sample_dataset_generation_job_simpleqna_with_agent_source.py: self-contained sample that creates a short-lived PromptAgentDefinition, uses it as an AgentDataGenerationJobSource for SimpleQnA evaluation, and cleans up the agent in a finally block. - Add sample_dataset_generation_job_simpleqna_with_file_source.py: replaces the removed dataset-source sample with a multi-source File + Prompt SimpleQnA evaluation job that uploads the seed via the Azure OpenAI Files API and verifies description/tag propagation. - Remove sample_dataset_generation_job_simpleqna_with_dataset_source.py: the service no longer supports reading dataset input files for QnA generation. - Rename sample_dataset_generation_job_with_evaluation.py to sample_dataset_generation_job_simpleqna_with_prompt_source.py for naming consistency with the new _with_<source>_source samples. - Add full cleanup of generated artifacts (datasets, fine-tuning files, prompt agents) across every data-generation sample so repeated runs do not accumulate artifacts. - README: clarify FOUNDRY_MODEL_NAME refers to a model deployment, add a link to the Responses-API supported-model list for evaluation jobs and a chat-completions hint for fine-tuning jobs, note that traces sources also work with third-party (OpenTelemetry-instrumented) agents, and document the self-cleaning behavior of the samples. - CHANGELOG: update reference to renamed sample. - tests/samples/test_samples.py: update references to renamed sample. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Tighten datasets README: consistency pass - Bump pip line to azure-ai-projects>=2.2.0 to match all data-gen samples. - Drop redundant 'matches the FOUNDRY_MODEL_NAME convention' parenthetical in the FOUNDRY_MODEL_NAME description. - Reword the quickstart export line to use a placeholder '<your-model-deployment-name>' and rephrase the inline comment. - Align FOUNDRY_PROJECT_ENDPOINT placeholder casing across the prerequisites bullet and the quickstart code block. - Standardize the agent-source sample reference as 'simple_qna' (snake_case, code-formatted) to match the surrounding usage. - Reorder and rewrite the Data Generation Jobs sample table so rows are grouped by scenario (Evaluation first, Supervised fine-tuning second), and unify the description voice across rows. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Make datasets README more concise - Replace the three-bullet env-var section with a compact table; drop the per-bullet sample-filename enumerations (the Sample Index already maps samples to their scenarios) and the agent-source FOUNDRY_AGENT_NAME exception note (covered in the sample's own docstring). - Trim the async/openai prerequisites sentence. - Collapse the two Data Generation Jobs intro paragraphs into one. - Trim the file-source row description and drop a redundant 'evaluation' qualifier from the traces-eval row (Scenario column already says it). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Drop redundant 'Running a Sample' section from datasets README Prerequisites already documents the required environment variables, and 'python <sample>.py' is self-evident; the duplicate snippet only added noise. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Drop Data Generation Jobs intro paragraph from datasets README The table immediately below already shows the source, scenario, and description for each sample; the prose summary just restated it. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Drop intro blurb from datasets README The title plus the Sample Index already make the scope obvious. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Smooth out two awkward Sample Index descriptions - file-source row: drop the 'verifies output metadata propagation' tail (it's a test detail, not a user-facing feature) and 'multi-source job combining' (the Source column already says File + Prompt). - agent-source row: replace 'short-lived \PromptAgentDefinition\' with 'prompt agent' so the description reads naturally without leaking a type name into the table. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Note third-party agent support in trace sample docstrings Aligns the trace sample docstrings with the FOUNDRY_AGENT_NAME row in the datasets README: both Foundry Agents and OpenTelemetry-instrumented third-party agents are valid sources. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * [azure-ai-projects] Add samples for evaluator authoring and adaptive eval generation (#47057) * [azure-ai-projects] Add samples for evaluator authoring and adaptive eval generation Adds five end-to-end samples under sdk/ai/azure-ai-projects/samples/evaluations/ covering the new `beta.evaluators` surface in azure-ai-projects 2.2.0: * sample_evaluator_rubric_manual.py Hand-author a rubric evaluator with create_version + use it in an OpenAI evaluation run. * sample_adaptive_eval_generation_basic.py Generate a rubric evaluator from a single Prompt source + use it in an OpenAI evaluation run. * sample_adaptive_eval_generation_all_sources.py Demonstrate all four generation source types (Prompt, Agent, Dataset, traces). Submits a combined Prompt+Agent+Dataset job and a separate traces+Agent job (traces requires a companion source). * sample_adaptive_eval_generation_iterate.py Human-in-the-loop workflow: generate v1, edit dimensions locally (boost/drop/add), preserve ALWAYS-ON dimensions verbatim, save as v2. * sample_adaptive_eval_generation_lifecycle.py Full lifecycle: create_generation_job with idempotent operation_id, poll, list, delete (with delete_version cascading to the job record). Style matches sdk/ai/azure-ai-projects/samples/datasets/sample_dataset_generation_job_with_evaluation.py (DESCRIPTION/USAGE docstring, dot-style polling, module-level constants, POLL_INTERVAL_SECONDS env var, plain happy-path cleanup at end of `with`). README.md is updated with a new "Adaptive Eval Generation" section listing the five samples. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * [azure-ai-projects] Rename adaptive eval samples to rubric evaluator Drop the "adaptive" terminology and rename the five evaluator samples under sdk/ai/azure-ai-projects/samples/evaluations/ to use a consistent "rubric_evaluator" prefix: sample_evaluator_rubric_manual.py -> sample_rubric_evaluator_manual.py sample_adaptive_eval_generation_basic.py -> sample_rubric_evaluator_generation_basic.py sample_adaptive_eval_generation_all_sources.py -> sample_rubric_evaluator_generation_all_sources.py sample_adaptive_eval_generation_iterate.py -> sample_rubric_evaluator_generation_iterate.py sample_adaptive_eval_generation_lifecycle.py -> sample_rubric_evaluator_generation_lifecycle.py Also updates DESCRIPTION/USAGE docstrings, cross-references, internal operation_id prefixes, and the README section header + table entries to match the new naming. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address review feedback and align with datasets-sample style slister1001 review: - Drop 'LLM' / 'deployment' from FOUNDRY_MODEL_NAME docstrings in the four generation samples (all_sources / basic / iterate / lifecycle): rubric evaluator generation runs inline server side, no project model deployment required. basic.py keeps the caveat that the eval-run grader still needs a real deployment because it uses the same env var for both the generation job and the OpenAI eval run. - sample_rubric_evaluator_manual.py left untouched: its model is only the eval run's judge (genuine deployment). README: - Drop the intro paragraph under 'Rubric Evaluators' so the table speaks for itself, matching the datasets samples README pattern. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Evaluation Samples: add multi-turn conversation evaluation samples S1-S4 (#47034) * feat(samples): add multi-turn conversation evaluation sample (S1) Add sample_multiturn_conversation_evaluation.py demonstrating: - Custom data source config with messages/tool_definitions schema - Conversation-level evaluators (customer_satisfaction, task_completion, coherence, groundedness) - Dataset upload and evaluation run with evaluation_level=conversation - Polling for results Includes sample JSONL with 3 conversations: basic multi-turn, tool-calling, and extended support dialog. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * feat(samples): add S2, S3, S4 multi-turn evaluation samples - S2: sample_multiturn_trace_evaluation_by_id.py Evaluate traces by conversation_id or trace_id - S3: sample_multiturn_trace_evaluation_agent_filter.py Evaluate traces by agent name/version/id with optional smart filtering - S4: sample_multiturn_conversation_simulation.py Simulate multi-turn conversations against an agent and evaluate - Data: sample_data_simulation_scenarios.jsonl (3 seed scenarios for S4) All samples use 4 conversation-level evaluators: customer_satisfaction, task_completion, coherence, groundedness. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix(samples): narrow dataset.id type for pyright The .id property on upload_file() returns Optional[str], which pyright flags when passed to SourceFileID(id=...) which expects str. Split the chain and add an assert to narrow the type. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * update to release version * update to model --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Refactor sample scripts and clean up code formatting across various files * SDK operations for Models, Unit tests and Sample notebook (#46842) * SDK operations for Models, Unit tests and Sample notebook * modifying sample to .py instead of .ipynb and update changelog * post emitter fixes and resolving review comments * fix cpell - azcopy * re emit from typespec for PendingUploadType changes * reverting pyproject.toml * pulling base branch of foundry sdk release for build * Revert aio _patch.py to base; minimize sync _patch.py diff for BetaModelsOperations * Address PR #46842 review comments Darren (dargilco): - D1: Merge feature/azure-ai-projects/2.2.0 into branch (separate merge commit). - D2: Remove duplicate '.beta.models' CHANGELOG bullet. - D3: Remove stray blank line in CHANGELOG. - D4: Revert apiview-properties.json to base. - D5: Wire async patched BetaModelsOperations into aio/operations/_patch.py; add azure/ai/projects/aio/operations/_patch_models_async.py with async create_version() using azure.storage.blob.aio.ContainerClient. Howie (howieleung): - H1: Rename helper models_create -> create_version (and validator). - H2: Rename sample_models{,_async}.py -> sample_models_basic{,_async}.py. - H3: Print friendly per-field summaries in samples instead of raw model repr. - H4: Guard against None return from create_version before accessing fields. Also follows TypeSpec rename: generated create_async() -> pending_create_version(). * Rename .beta.models patched helper create_version -> create - Renames patched BetaModelsOperations.create_version() to create() (sync + async). - Renames internal validator _validate_create_version_inputs -> _validate_create_inputs. - Updates samples, tests, README, and CHANGELOG to use the new name. - Leaves generated spec method pending_create_version unchanged. * Add sample recordings for .beta.models and fix generated arg names - Add parameterized sample tests test_models_samples (sync + async). - Add slim modelsServicePreparer (only foundry_project_endpoint). - Add sanitizers for random model name, Foundry storage account/container, /workspaces/<name>, azureai:// asset URIs, account/project from FOUNDRY_PROJECT_ENDPOINT, and SAS query parameters (sig, skoid, sktid). - Use per-recording random MODEL_NAME (recsmplmdl<hex>) and sanitize to a stable value on playback (Foundry asset-store reserves <name>/<version> permanently). - Update assets.json tag to point at the new recordings. Bug fixes surfaced by live runs (renamed generated kwargs): - pending_upload: body= -> pending_upload_request= - pending_create_version: body= -> model_version= - get_credentials: body= -> credential_request= Applied to patched create() helper (sync + async), all three model samples, and the test_patch_models mock. Notes: - sample_models_basic.py (which uses AzCopy) is excluded from the parameterized sample tests because AzCopy traffic isn't captured by the test proxy. - LLM print-call validation is not invoked for models tests (canary project has no Azure OpenAI connection). * Exclude .beta.models.create from foundry-features header test The multi-step orchestrator helper performs local input validation before any HTTP call, so the test framework's synthetic placeholder args (e.g. source={}) cause it to raise TypeError before _RequestCaptured is ever raised. Add a shared EXCLUDED_BETA_METHODS mapping and skip excluded methods in both sync and async discovery. The header invariant is still enforced for every underlying HTTP method create() calls (pending_upload, pending_create_version, get). * Add cspell entries: recsmplmdl, simpleqna, skoid * Fix pyright and pylint issues in BetaModelsOperations patches - Replace hasattr-based duck typing with isinstance(dict) check so pyright can narrow types correctly on _extract_pending_upload_targets. - Remove unused HttpResponseError import. - Suppress do-not-import-asyncio for asyncio.sleep used in the async polling loop (transport sleep is not applicable). - Add :param/:keyword/:return/:rtype docstring sections on private helpers to satisfy azure-pylint-guidelines-checker. * Add @overload variants to .beta.models.create for precise return types * Rename sample_models_without_patch.py to sample_models_create_and_poll.py * Remove redundant 'if model is None' check in sample_models_basic.py (covered by overload) * [azure-ai-projects] Fix mypy errors in rubric evaluator samples (#47146) * [azure-ai-projects] Fix mypy errors in rubric evaluator samples Address 65 mypy errors across 5 samples on feature/azure-ai-projects/2.2.0: - Add `assert ... is not None` after extracting `job.result` (`Optional[EvaluatorVersion]`). - Narrow `EvaluatorDefinition` to `RubricBasedEvaluatorDefinition` via `isinstance` before accessing `dimensions` / `pass_threshold` (which only exist on the rubric subclass). - Guard `EvaluatorCategory` enum `.value` access with `isinstance` so list comprehensions remain mypy-clean over `list[Union[str, EvaluatorCategory]]`. - Guard `entry.inputs.evaluator_name` for Optional inputs in the lifecycle listing loop. No runtime semantics changed; the asserts hold by construction after a successful job (the service guarantees the invariants, but they aren't expressible in the static type system). Files: - sample_rubric_evaluator_manual.py - sample_rubric_evaluator_generation_lifecycle.py - sample_rubric_evaluator_generation_basic.py - sample_rubric_evaluator_generation_iterate.py - sample_rubric_evaluator_generation_all_sources.py Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * [azure-ai-projects] Trim noise in rubric evaluator samples Reduce per-step prints, ASCII section separators, polling progress dots, per-dimension enumeration loops, and verbose `Created X`/`Doing Y` announcements across the 5 rubric evaluator samples. Keep all service-constraint comments (`allow_preview`, `operation_id` idempotency, `traces` companion, `delete_version` cascade) and all type-narrowing `assert isinstance` calls. Total: 1217 -> 992 lines (-18.5%). mypy still clean. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Bring back toolboxers opt-in header value, in prep for next TypeSpec emit * Add missing rename in post-emitter_fixes.cmd * Add sample for emitting human evaluation events (#46962) * [azure-ai-projects] Emit SDK from TypeSpec (feature/foundry-release 10482dd) (#47185) * work iq samples (#47184) * work iq samples * Add work IQ project connection and user input to service preparer * update tests * push assets.json --------- Co-authored-by: Jessie Li <jessli@microsoft.com> --------- Co-authored-by: Darren Cohen <39422044+dargilco@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: M-Hietala <78813398+M-Hietala@users.noreply.github.com> Co-authored-by: Shruti Iyer <9905402+shrutiyer@users.noreply.github.com> Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: Shruti Iyer <you@example.com> Co-authored-by: Copilot <copilot@github.com> Co-authored-by: Billy Hu <ninhu@microsoft.com> Co-authored-by: aprilk-ms <55356546+aprilk-ms@users.noreply.github.com> Co-authored-by: Kayla Seager <kayla.seager@microsoft.com> Co-authored-by: Kshitij Chawla <166698309+kshitij-microsoft@users.noreply.github.com> Co-authored-by: seangayler-msft <seangayler@microsoft.com> Co-authored-by: Jessie Li <jessli@microsoft.com>
1 parent 1e8a307 commit 203fcc5

186 files changed

Lines changed: 35161 additions & 8714 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

sdk/ai/azure-ai-agents/azure/ai/agents/_utils/model_base.py

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -816,16 +816,16 @@ def _get_deserialize_callable_from_annotation( # pylint: disable=too-many-retur
816816

817817
# is it optional?
818818
try:
819-
if any(a for a in annotation.__args__ if a == type(None)): # pyright: ignore
819+
if any(a for a in annotation.__args__ if a == type(None)): # pyright: ignore # pylint: disable=unidiomatic-typecheck
820820
if len(annotation.__args__) <= 2: # pyright: ignore
821821
if_obj_deserializer = _get_deserialize_callable_from_annotation(
822-
next(a for a in annotation.__args__ if a != type(None)), module, rf # pyright: ignore
822+
next(a for a in annotation.__args__ if a != type(None)), module, rf # pyright: ignore # pylint: disable=unidiomatic-typecheck
823823
)
824824

825825
return functools.partial(_deserialize_with_optional, if_obj_deserializer)
826826
# the type is Optional[Union[...]], we need to remove the None type from the Union
827827
annotation_copy = copy.copy(annotation)
828-
annotation_copy.__args__ = [a for a in annotation_copy.__args__ if a != type(None)] # pyright: ignore
828+
annotation_copy.__args__ = [a for a in annotation_copy.__args__ if a != type(None)] # pyright: ignore # pylint: disable=unidiomatic-typecheck
829829
return _get_deserialize_callable_from_annotation(annotation_copy, module, rf)
830830
except AttributeError:
831831
pass

sdk/ai/azure-ai-agents/azure/ai/agents/models/_patch.py

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -333,9 +333,9 @@ def _map_type(annotation) -> Dict[str, Any]: # pylint: disable=too-many-return-
333333
if origin is Union:
334334
args = get_args(annotation)
335335
# If Union contains None, it is an optional parameter
336-
if type(None) in args:
336+
if type(None) in args: # pylint: disable=unidiomatic-typecheck
337337
# If Union contains only one non-None type, it is a nullable parameter
338-
non_none_args = [arg for arg in args if arg is not type(None)]
338+
non_none_args = [arg for arg in args if arg is not type(None)] # pylint: disable=unidiomatic-typecheck
339339
if len(non_none_args) == 1:
340340
schema = _map_type(non_none_args[0])
341341
if "type" in schema:

sdk/ai/azure-ai-projects/.env.template

Lines changed: 6 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -57,6 +57,9 @@ A2A_USER_INPUT=
5757
AZURE_TEST_RUN_LIVE=false
5858
AZURE_SKIP_LIVE_RECORDING=true
5959

60+
#Used by hosted agent
61+
FOUNDRY_HOSTED_AGENT_NAME=
62+
6063
# Used in Fine-tuning tests
6164
COMPLETED_OAI_MODEL_SFT_FINE_TUNING_JOB_ID=
6265
COMPLETED_OAI_MODEL_RFT_FINE_TUNING_JOB_ID=
@@ -88,6 +91,6 @@ RUN_EXTENDED_FINE_TUNING_LIVE_TESTS=false
8891
# - SAMPLE_TEST_ERROR_LOG: Sample crashed with an exception during execution
8992
# - SAMPLE_TEST_FAILED_LOG: Sample ran successfully but LLM validation failed (incorrect output)
9093
# - SAMPLE_TEST_PASSED_LOG: Sample ran successfully and LLM validation passed (correct output)
91-
# SAMPLE_TEST_PASSED_LOG=<sample_filename>_success_<timestamp>.log
92-
# SAMPLE_TEST_FAILED_LOG=<sample_filename>_failed_<timestamp>.log
93-
# SAMPLE_TEST_ERROR_LOG=<sample_filename>_errors_<timestamp>.log
94+
SAMPLE_TEST_PASSED_LOG=<sample_filename>_success_<timestamp>.log
95+
SAMPLE_TEST_FAILED_LOG=<sample_filename>_failed_<timestamp>.log
96+
SAMPLE_TEST_ERROR_LOG=<sample_filename>_errors_<timestamp>.log
Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,33 @@
1+
# CoPilot skills for azure-ai-projects development
2+
3+
## Prerequisite
4+
5+
* Clone the `azure-sdk-for-python` repo to your local machine, if you don't already have it:
6+
```
7+
git clone https://github.com/Azure/azure-sdk-for-python.git
8+
```
9+
* Change to the directory `sdk\ai\azure-ai-projects`.
10+
* Switch to the current feature branch: `git switch feature/azure-ai-projects/2.2.0`.
11+
* Make sure you don't have any files edited or added in this branch (clean `git status` state).
12+
13+
## Emit from TypeSpec and create a PR
14+
15+
### Using GitHub CoPilot in VSCode
16+
17+
* Open VSCode in the current folder.
18+
* Open the CoPilot chat window ("Toggle Chat").
19+
* Make sure you are in "Agent" mode.
20+
* Start typing `/azure-ai-projects` and press tab to auto complete it to `/azure-ai-projects-emit-from-typespec`, then press Enter.
21+
* Answer some questions and approve execution to go through the workflow
22+
23+
### Using CoPilot CLI or Agency Copilot CLI
24+
25+
* Install [GitHub CoPilot CLI](https://docs.github.com/copilot/how-tos/copilot-cli/set-up-copilot-cli/install-copilot-cli) or [Agency CoPilot CLI](https://aka.ms/agency) (VPN required) if you don't already have it.
26+
* Run CoPilot CLI by typing `copilot`
27+
* Start typing `/azure-ai-projects` and press tab to auto complete it to `/azure-ai-projects-emit-from-typespec`, then press Enter.
28+
* Answer some questions and approve execution to go through the workflow
29+
30+
31+
32+
33+
Lines changed: 234 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,234 @@
1+
---
2+
name: azure-ai-projects-emit-from-typespec
3+
license: MIT
4+
metadata:
5+
version: "1.0.0"
6+
distribution: local
7+
description: "Emit the azure-ai-projects Python SDK from TypeSpec, apply post-emitter fixes, update changelog, and create a Pull Request. WHEN: \"emit SDK from TypeSpec\", \"generate azure-ai-projects SDK\", \"update azure-ai-projects from TypeSpec\", \"emit from TypeSpec\", \"regenerate azure-ai-projects\". DO NOT USE FOR: other Azure SDK packages, manual code edits without TypeSpec. INVOKES: azsdk-common-generate-sdk-locally skill, post-emitter-fixes.cmd script, git commands, gh CLI for PR creation."
8+
compatibility:
9+
requires: "azure-sdk-mcp server, local azure-sdk-for-python clone, git, gh CLI"
10+
---
11+
12+
# Emit azure-ai-projects Python SDK from TypeSpec
13+
14+
This skill guides Copilot through emitting the azure-ai-projects Python SDK from TypeSpec,
15+
applying post-emitter fixes, updating the changelog, installing package from sources and creating a Pull Request.
16+
17+
**Working directory:** `sdk/ai/azure-ai-projects`
18+
19+
**Skills:** This workflow relies on skills defined under `.github/skills/` at the root of the repository. Use those skills for SDK generation, building, changelog updates, and other SDK lifecycle operations instead of running commands directly. In particular:
20+
21+
- **`azsdk-common-generate-sdk-locally`** – For generating SDK from TypeSpec, building, running checks/tests, updating changelog, metadata, and version.
22+
23+
---
24+
25+
## Step 1: Gather information from the user
26+
27+
Ask the user the following questions **one at a time**, waiting for each answer before proceeding.
28+
29+
### 1a. Topic branch name
30+
31+
Ask the user to choose **one** of the following two options for the target topic branch:
32+
33+
1. **Create a new topic branch (with default branch name)** – Create a new topic branch for the emitted changes. If selected, this default branch name will be used "<github-userid>/<emit-from-typespec-DD-MM-HH-MM>", where `github-userid` is the user's GitHub ID and `DD-MM-HHMM` is the current date-time using date, month, hour and minute. For example, if the GitHub ID is "dargilco" and the current date and time is May 1st, 2026 at 8:13am, the default branch name would be `dargilco/emit-from-typespec-01-05-0813`. This should be the default option, and the default branch name should be displayed. If you press enter without typing anything, this option will be selected.
34+
35+
2. **Create a new topic branch (branch name given by user)** - Ask the user for the branch name. Mention that a common format is "<github-userid>/<work-title>". If the user enters a branch name `feature/azure-ai-projects/2.2.0` then stop and report that they cannot emit directly to the current feature branch.
36+
37+
3. **Emit to current branch** – Emit directly to the current branch without creating a new topic branch. This is not common, but may be necessary if the user is re-running this workflow because of a previous failure, where the topic branch was already created. If the current branch is named `feature/azure-ai-projects/2.2.0` then stop and report that they cannot emit directly to the current feature branch.
38+
39+
### 1b. TypeSpec source
40+
41+
Ask the user to choose **one** of the following three options for the TypeSpec source:
42+
43+
1. **Latest commit on `feature/foundry-release`** – Automatically find the latest commit to the `feature/foundry-release` branch in [Azure/azure-rest-api-specs](https://github.com/Azure/azure-rest-api-specs) that touched files under `specification/ai-foundry/data-plane/Foundry`, and use that commit hash. This should be the default option. If you press enter without typing anything, this option will be selected.
44+
45+
2. **Local TypeSpec folder** – Emit from a local clone of the [azure-rest-api-specs](https://github.com/Azure/azure-rest-api-specs) repository. If selected, ask for the **full folder path** to the TypeSpec project. This is the folder ending with `\specification\ai-foundry\data-plane\Foundry`. If it does not end with that string, stop and report the error to the user. Do not continue.
46+
47+
3. **TypeSpec commit hash** – Emit from a specific commit in the [azure-rest-api-specs](https://github.com/Azure/azure-rest-api-specs) repository. If selected, ask for the **full commit SHA** (40 characters).
48+
49+
50+
---
51+
52+
## Step 2: Record the current branch
53+
54+
Before creating the topic branch, record the name of the **current Git branch**. This is the branch that the topic branch will be created from, and the branch the PR will target.
55+
56+
```
57+
git branch --show-current
58+
```
59+
60+
Save this as `BASE_BRANCH`.
61+
62+
---
63+
64+
## Step 3: Create the topic branch
65+
66+
Create the topic branch off the current branch and switch to it:
67+
68+
```
69+
git fetch
70+
git switch -c <topic-branch> origin/<BASE_BRANCH>
71+
```
72+
73+
Replace `<topic-branch>` with the name provided by the user in Step 1a.
74+
75+
---
76+
77+
## Step 4: Emit SDK from TypeSpec
78+
79+
Use the **`azsdk-common-generate-sdk-locally`** skill to generate the SDK code. The skill knows how to invoke `azsdk_package_generate_code` and related MCP tools.
80+
81+
Provide the skill with the TypeSpec source selected by the user. With is either:
82+
83+
- **Local folder:** Pass the local spec repo path for local generation. Or,
84+
- **Commit hash:** Update `commit:` in `tsp-location.yaml` to the full SHA first, then invoke the skill for generation.
85+
86+
Note:
87+
- You are only allowed to use the `tsp-client update` command. Do not use any of the other `tsp-client` commands.
88+
- If you are generating from local TypeSpec folder, do not edit the file `tsp-location.yaml`. Leave it as is. It should not be used by the emitter.
89+
- If you are generating from local TypeSpec folder, make sure that the local folder path you provide `tsp-client update --local-spec-repo` ends with `specification\ai-foundry\data-plane\Foundry`.
90+
- **If the generation fails**, stop and report the error to the user. Do not continue.
91+
92+
---
93+
94+
## Step 5: Revert changes to file pyproject.toml
95+
96+
After the emit, there will be changes to `pyproject.toml` that are not needed. Revert any changes to `pyproject.toml` by running:
97+
98+
```
99+
git restore pyproject.toml
100+
```
101+
102+
---
103+
104+
## Step 6: Commit and push
105+
106+
Stage all changes (excluding file names that start with `.env`), commit, and push the topic branch:
107+
108+
```
109+
git add -A -- ':!.env*'
110+
git commit -m "Part 1: Emit SDK from TypeSpec"
111+
112+
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>"
113+
git push -u origin <topic-branch>
114+
```
115+
116+
---
117+
118+
## Step 7: Run post-emitter fixes
119+
120+
After a successful emit, run the post-emitter fix script located in the `sdk/ai/azure-ai-projects` folder:
121+
122+
```
123+
post-emitter-fixes.cmd
124+
```
125+
126+
This script applies azure-ai-projects-specific corrections to the emitted code (restores `pyproject.toml`, fixes enum names, patches Sphinx doc-string issues, and runs `black` formatting).
127+
128+
**If the script fails**, stop and report the error to the user. Do not continue. Do not attempt to analyze the script failures and fix them with Copilot. The script should be fixed by the engineering team if it is not working.
129+
130+
---
131+
132+
## Step 8: Commit and push
133+
134+
Stage all changes (excluding file names that start with `.env`), commit, and push the topic branch:
135+
136+
```
137+
git add -A -- ':!.env*'
138+
git commit -m "Part 2: Apply post-emitter-fixes.cmd"
139+
140+
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>"
141+
git push -u origin <topic-branch>
142+
```
143+
144+
---
145+
146+
## Step 9: Fix patched code related to preview feature headers
147+
148+
The emitted code may have introduced another beta sub-client (a new property on class `BetaOperations`). It may have also added another enum value to the existing internal class `_FoundryFeaturesOptInKeys`. This means that the client library needs to set a new HTTP request header when making REST API calls to the service, to opt-in to the new service features which are still in preview. If that's the case, do the following:
149+
150+
* Update the dictionary `_BETA_OPERATION_FEATURE_HEADERS` defined in `azure\ai\projects\models\_patch.py`, to include a new key-value pair to map the new beta sub-client name to the proper value from `_FoundryFeaturesOptInKeys`. If no new beta sub-client was introduced, but a new enum value was added to `_FoundryFeaturesOptInKeys`, you will need to update one of the existing key-value pairs in `_BETA_OPERATION_FEATURE_HEADERS` to a comma-separated join of multiple values from `_FoundryFeaturesOptInKeys`.
151+
152+
* Do a similar change to the dictionary `EXPECTED_FOUNDRY_FEATURES` defined in the test file `tests\foundry_features_header\foundry_features_header_test_base.py`: add a new key-value pair if a new beta sub-client was introduced, or update an existing key-value pair to include the new enum value if no new beta sub-client was introduced.
153+
154+
* Finally, look at the two files `azure\ai\projects\operations\_patch.py` and `azure\ai\projects\aio\operations\_patch.py`. They define the public `BetaOperations` classes for the sync and async clients. To support a new sub-client, you will need to add a new property to this class with the proper doc string. You will need to update the import statement at the top of the file to import the new sub-client class. And you will need to update `__all__` statement at the bottom of the file to include the new sub-client class name. Follow the examples you see there for `BetaDatasetsOperations` or `BetaSkillsOperations`.
155+
156+
If a new enum value was added to `_AgentDefinitionOptInKeys`, please print a note on screen that mentions which value was added, and tell the user that a review is needed to make sure this new value is properly used. But otherwise continue on.
157+
158+
Important: Under the `azure\ai\projects` folder, you are only allowed to edit Python source files that start with "_patch". If you see that changes are needed in other files, stop and report this to the user instead of making the changes yourself.
159+
160+
---
161+
162+
## Step 10: Update samples and tests
163+
164+
If there were any breaking changes in existing APIs, like class or method renames:
165+
* update the patched code accordingly in the client library to reflect those changes. Changes should be made to Python source file names that start with "_patch", under the `azure\ai\projects` folder.
166+
* update the samples accordingly to reflect those changes. Changes should be made under `sdk/ai/azure-ai-projects/samples` folder.
167+
* update the tests accordingly to reflect those changes. Changes should be made under `sdk/ai/azure-ai-projects/tests` folder.
168+
169+
---
170+
171+
## Step 11: Install package from sources
172+
173+
In the folder `sdk\ai\azure-ai-projects`, run `pip install -e .` to install the package from sources. If there are any errors, stop and report the error to the user. Do not continue.
174+
175+
---
176+
177+
## Step 12: Update CHANGELOG.md
178+
179+
Use the **`azsdk-common-generate-sdk-locally`** skill's changelog capability (`azsdk_package_update_changelog_content`) to update `CHANGELOG.md` in the `sdk/ai/azure-ai-projects` folder with a summary of changes from the TypeSpec emit. Some guidelines to follow:
180+
* Start by examining the public SDK API surface of the latest released version of the azure-ai-projects package. The source code for this version can be found in the Main branch of the `azure-sdk-for-python` repository, in the folder `sdk\ai\azure-ai-projects`.
181+
* Then compare it to the public SDK API surface of current version in this topic branch.
182+
* Look at the existing change log from the latest version (if exists) and edit or add to it to capture all the changes you see. If a change log does not exist for the current version at the top of `CHANGELOG.md`, create a new one.
183+
* If a new method was added, there is no need to add the list of all new classes that define the inputs and output of the method. It's enough to mention that the new method was added.
184+
* Show the user the proposed changelog entry and ask for confirmation or edits before saving.
185+
186+
---
187+
188+
## Step 13: Commit and push
189+
190+
Stage all changes (excluding file names that start with `.env`), commit, and push the topic branch:
191+
192+
```
193+
git add -A -- ':!.env*'
194+
git commit -m "Part 3: Additional edits"
195+
196+
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>"
197+
git push -u origin <topic-branch>
198+
```
199+
200+
---
201+
202+
## Step 14: Create a Pull Request
203+
204+
Create a draft PR from the **topic branch** to the **base branch** (recorded in Step 2):
205+
206+
```
207+
gh pr create --draft --base <BASE_BRANCH> --head <topic-branch> --assignee @me --title "<PR title>" --body "<PR body>"
208+
```
209+
210+
- **Title:** Use a descriptive title such as `[azure-ai-projects] Emit SDK from TypeSpec (<short description>)`.
211+
- **Body:** Include which TypeSpec source was used and a summary of the changelog entry.
212+
213+
You must show the user the resulting PR URL on screen when done, before you continue to the next step.
214+
215+
Open a new tab in the default browser and navigate to the PR URL.
216+
217+
---
218+
219+
## Step 15: Optionally run tests locally
220+
221+
Prompt the user with this message: "Tests will run as part of the Pull Request. However, you can optionally run tests locally in a Python virtual environment, right now. It will take a few minutes. Do you want to run tests locally? (yes/no)"
222+
223+
If the user answers "yes", run all tests from recordings. Follow these guidelines:
224+
* Run tests in a local Python virtual environment. Create this virtual environment if it does not already exists:
225+
```
226+
python -m venv .venv
227+
```
228+
and activate it:
229+
```
230+
.venv\Scripts\activate
231+
```
232+
* Show test progress on screen, as tests are run.
233+
234+
Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
*
2+
!.gitignore

0 commit comments

Comments
 (0)