AI Engineer World Fair - AI Agent Breakout session fixes

[cedricvidal]

Description

Upgrade AI Agent Evaluation Notebooks to 1RP.

Those changes were made to present the notebooks at AI Engineer World Fair for this talk

Note

Including this image because I'm going to share this PR with participants so they can try the code until it's merged hopefully

Details of all the changes:

• Using separate env vars for AI Foundry report upload REPORT_* to accomodate for the fact that 1RP Projects currently cannot be used to upload reports so you need a 1RP Project for evaluation and a Hub project to upload the report.
• Added pip installation commands
• Added environment variable loading using dotenv across notebooks.
• Updated Azure OpenAI model settings and environment variable names.
• Improved evaluation examples and output formatting for intent resolution, task adherence, tool call accuracy, and response completeness.
• Updated .gitignore to exclude all .env* files.
• Improved README with environment variable setup instructions.

Environment Variables

Important

Environment variables are loaded from .env file. Here are the expected variables

To run the evaluations:

• AZURE_OPENAI_ENDPOINT: The endpoint URL for Azure OpenAI.
• AZURE_OPENAI_DEPLOYMENT: The deployment name for the Azure OpenAI model (e.g., gpt-4o).
• MODEL_DEPLOYMENT_NAME: The deployment name for the model used in evaluations (e.g., gpt-4o).
• AGENT_MODEL_DEPLOYMENT_NAME: The deployment name for the agent model (e.g., gpt-4o).
• AZURE_OPENAI_API_VERSION: The API version for Azure OpenAI.
• AZURE_SUBSCRIPTION_ID: The Azure subscription ID.
• PROJECT_NAME: The name of the Azure project.
• RESOURCE_GROUP_NAME: The name of the Azure resource group.
• AZURE_AI_PROJECT: The Azure AI project identifier.
• AZURE_OPENAI_API_KEY: The API key for Azure OpenAI.

To upload reports to Azure AI Foundry:

• REPORT_AZURE_SUBSCRIPTION_ID: The Azure subscription ID for report uploads.
• REPORT_PROJECT_NAME: The project name for report uploads.
• REPORT_RESOURCE_GROUP_NAME: The resource group name for report uploads.

Ensure all these variables are properly configured in your .env file before running the notebooks.

Checklist

• I have read the contribution guidelines
• I have coordinated with the docs team (mldocs@microsoft.com) if this PR deletes files or changes any file names or file extensions.
• This notebook or file is added to the CODEOWNERS file, pointing to the author or the author's team.

Thanks @singankit for the support helping me get ready for the conference!!

[hayescode]

@cedricvidal how can we use this with the new Foundry projects (no hub)? This code gives an error.

    azure_ai_project={
        "subscription_id": os.environ["REPORT_AZURE_SUBSCRIPTION_ID"],
        "project_name": os.environ["REPORT_PROJECT_NAME"],
        "resource_group_name": os.environ["REPORT_RESOURCE_GROUP_NAME"],
    },

EvaluationException: (InternalError) The get 'my_project' workspace request failed with HTTP 404 - (ResourceNotFound) The Resource 'Microsoft.MachineLearningServices/workspaces/my_project' under resource group 'my_rg' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix