Infra deployment scripts for RedTeam

scenarios/evaluate/AI_RedTeaming/.env.sample

@@ -0,0 +1,28 @@
+# Azure AI Red Teaming Environment Variables
+# Copy this file to .env and populate with your deployment outputs
+
+# Azure AI Project endpoint (from AZURE_AI_PROJECT output)
+AZURE_AI_PROJECT="https://your-foundry-name.services.ai.azure.com/api/projects/your-project-name"
+
+# Azure OpenAI Configuration (from AZURE_OPENAI_CONFIG output)
+AZURE_OPENAI_CONFIG='{"endpoint": "https://your-foundry-name.cognitiveservices.azure.com/openai/deployments/gpt-4o/chat/completions?api-version=2024-10-21", "api_key": "your-api-key", "deployment": "gpt-4o", "api_version": "2024-10-21"}'
+
+# Storage Account (from AZURE_STORAGE_ACCOUNT output)
+AZURE_STORAGE_ACCOUNT="your-storage-account-name"
+
+# Additional deployment outputs (for reference)
+AZURE_SUBSCRIPTION_ID="12345678-1234-1234-1234-123456789012"
+AZURE_RESOURCE_GROUP="rg-your-resource-group"
+AI_PROJECT_NAME="your-project-name"
+AI_FOUNDRY_ENDPOINT="https://your-foundry-name.services.ai.azure.com"
+AZURE_OPENAI_ENDPOINT="https://your-foundry-name.cognitiveservices.azure.com"
+AZURE_OPENAI_API_KEY="your-api-key"
+AZURE_STORAGE_ACCOUNT_NAME="your-storage-account-name"
+
+# Note: The main variables needed for RedTeam are:
+# - azure_ai_project (for the project endpoint)
+# - azure_openai_config (for the model configuration)
+# 
+# These values are automatically generated by the deployment script.
+# Run: .\infra\deploy.ps1
+# Then copy the output values to this file.

scenarios/evaluate/AI_RedTeaming/AI_RedTeaming.ipynb

@@ -15,11 +15,21 @@
     "## Before you begin\n",
     "\n",
     "### Prerequisite\n",
-    "First, if you have an Azure subscription, create an [Azure AI hub](https://learn.microsoft.com/en-us/azure/ai-studio/concepts/ai-resources) then [create an Azure AI project](https://learn.microsoft.com/en-us/azure/ai-studio/concepts/ai-resources). AI projects and Hubs can be served within a private network and are compatible with private endpoints. You **do not** need to provide your own LLM deployment as the AI Red Teaming Agent hosts adversarial models for both simulation and evaluation of harmful content and connects to it via your Azure AI project.\n",
+    "First, deploy the Azure AI Red Teaming infrastructure using the deployment script in the `infra/` folder:\n",
     "\n",
-    "**Note**: In order to upload your results to Azure AI Foundry, you must have the `Storage Blob Data Contributor` role\n",
+    "```bash\n",
+    "# Navigate to the AI_RedTeaming directory\n",
+    "cd scenarios/evaluate/AI_RedTeaming\n",
+    "\n",
+    "# Run the deployment script\n",
+    ".\\infra\\deploy.ps1\n",
+    "```\n",
     "\n",
-    "**Important**: First, ensure that you've installed the [Azure CLI](https://learn.microsoft.com/en-us/cli/azure/install-azure-cli) and then make sure to authenticate to Azure using `az login` in your terminal before running this notebook.\n",
+    "This will create an Azure AI Foundry hub and project with all necessary resources including GPT-4o model deployment. The deployment script will output environment variables that you need to copy to a `.env` file.\n",
+    "\n",
+    "**Note**: In order to upload your results to Azure AI Foundry, you must have the `Storage Blob Data Contributor` role (automatically assigned by the deployment script).\n",
+    "\n",
+    "**Important**: Ensure that you've installed the [Azure CLI](https://learn.microsoft.com/en-us/cli/azure/install-azure-cli) and authenticated using `az login` before running the deployment.\n",
     "\n",
     "### Installation\n",
     "From a terminal window, navigate to your working directory which contains this sample notebook, and execute the following.\n",
@@ -30,17 +40,19 @@
     "Then, activate the virtual environment created:\n",
     "\n",
     "```bash\n",
-    "# %source .venv/bin/activate # If using Mac/Linux OS\n",
+    "# source .venv/bin/activate # If using Mac/Linux OS\n",
     ".venv/Scripts/activate # If using Windows OS\n",
     "```\n",
     "\n",
     "With your virtual environment activated, install the following packages required to execute this notebook:\n",
     "\n",
     "```bash\n",
     "pip install uv\n",
-    "uv pip install azure-ai-evaluation[redteam] azure-identity openai\n",
+    "uv pip install azure-ai-evaluation[redteam] azure-identity openai python-dotenv\n",
     "```\n",
     "\n",
+    "### Environment Setup\n",
+    "After deployment, create a `.env` file in this directory and copy the environment variables from the deployment output. A sample file is provided at `.env.sample`.\n",
     "\n",
     "Now open VSCode with the following command, and ensure your virtual environment is used as kernel to run the remainder of this notebook.\n",
     "```bash\n",
@@ -64,6 +76,11 @@
     "from typing import Optional, Dict, Any\n",
     "import os\n",
     "\n",
+    "# Load environment variables from .env file\n",
+    "from dotenv import load_dotenv\n",
+    "\n",
+    "load_dotenv()\n",
+    "\n",
     "# Azure imports\n",
     "from azure.identity import DefaultAzureCredential, get_bearer_token_provider\n",
     "from azure.ai.evaluation.red_team import RedTeam, RiskCategory, AttackStrategy\n",
@@ -93,17 +110,23 @@
     "For reference, here's an example of what your populated environment variables should look like:\n",
     "\n",
     "```\n",
-    "# Azure OpenAI\n",
-    "AZURE_OPENAI_API_KEY=\"your-api-key-here\"\n",
-    "AZURE_OPENAI_ENDPOINT=\"https://endpoint-name.openai.azure.com/openai/deployments/deployment-name/chat/completions\"\n",
-    "AZURE_OPENAI_DEPLOYMENT_NAME=\"gpt-4\"\n",
-    "AZURE_OPENAI_API_VERSION=\"2024-12-01-preview\"\n",
+    "# Azure AI Project (from deployment output)\n",
+    "AZURE_AI_PROJECT=\"https://your-foundry-name.services.ai.azure.com/api/projects/your-project-name\"\n",
+    "\n",
+    "# Azure OpenAI Configuration (from deployment output)\n",
+    "AZURE_OPENAI_CONFIG='{\"endpoint\": \"https://your-foundry-name.cognitiveservices.azure.com/openai/deployments/gpt-4o/chat/completions?api-version=2024-10-21\", \"api_key\": \"your-api-key\", \"deployment\": \"gpt-4o\", \"api_version\": \"2024-10-21\"}'\n",
+    "\n",
+    "# Storage Account (from deployment output)\n",
+    "AZURE_STORAGE_ACCOUNT=\"your-storage-account-name\"\n",
     "\n",
-    "# Azure AI Project\n",
+    "# Additional deployment outputs\n",
     "AZURE_SUBSCRIPTION_ID=\"12345678-1234-1234-1234-123456789012\"\n",
-    "AZURE_RESOURCE_GROUP_NAME=\"your-resource-group\"\n",
-    "AZURE_PROJECT_NAME=\"your-project-name\"\n",
-    "```"
+    "AZURE_RESOURCE_GROUP=\"rg-your-resource-group\"\n",
+    "AI_PROJECT_NAME=\"your-project-name\"\n",
+    "AI_FOUNDRY_ENDPOINT=\"https://your-foundry-name.services.ai.azure.com\"\n",
+    "```\n",
+    "\n",
+    "**Note**: These values are automatically generated by the deployment script and should be set in your `.env` file."
    ]
   },
   {
@@ -112,20 +135,37 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "# Azure AI Project information\n",
-    "azure_ai_project = {\n",
-    "    \"subscription_id\": os.environ.get(\"AZURE_SUBSCRIPTION_ID\"),\n",
-    "    \"resource_group_name\": os.environ.get(\"AZURE_RESOURCE_GROUP_NAME\"),\n",
-    "    \"project_name\": os.environ.get(\"AZURE_PROJECT_NAME\"),\n",
-    "}\n",
-    "\n",
-    "# Azure OpenAI deployment information\n",
-    "azure_openai_deployment = os.environ.get(\"AZURE_OPENAI_DEPLOYMENT\")  # e.g., \"gpt-4\"\n",
-    "azure_openai_endpoint = os.environ.get(\n",
-    "    \"AZURE_OPENAI_ENDPOINT\"\n",
-    ")  # e.g., \"https://endpoint-name.openai.azure.com/openai/deployments/deployment-name/chat/completions\"\n",
-    "azure_openai_api_key = os.environ.get(\"AZURE_OPENAI_API_KEY\")  # e.g., \"your-api-key\"\n",
-    "azure_openai_api_version = os.environ.get(\"AZURE_OPENAI_API_VERSION\")  # Use the latest API version"
+    "import json\n",
+    "\n",
+    "# Azure AI Project endpoint (from deployment output)\n",
+    "azure_ai_project = os.environ.get(\"AZURE_AI_PROJECT\")\n",
+    "\n",
+    "# Parse Azure OpenAI configuration from deployment output\n",
+    "azure_openai_config_str = os.environ.get(\"AZURE_OPENAI_CONFIG\")\n",
+    "if azure_openai_config_str:\n",
+    "    azure_openai_config = json.loads(azure_openai_config_str)\n",
+    "    azure_openai_endpoint = azure_openai_config[\"endpoint\"]\n",
+    "    azure_openai_deployment = azure_openai_config[\"deployment\"]\n",
+    "    azure_openai_api_key = azure_openai_config[\"api_key\"]\n",
+    "    azure_openai_api_version = azure_openai_config[\"api_version\"]\n",
+    "else:\n",
+    "    # Fallback to individual environment variables if needed\n",
+    "    azure_openai_endpoint = os.environ.get(\"AZURE_OPENAI_ENDPOINT\")\n",
+    "    azure_openai_deployment = \"gpt-4o\"  # Default deployment name\n",
+    "    azure_openai_api_key = os.environ.get(\"AZURE_OPENAI_API_KEY\")\n",
+    "    azure_openai_api_version = \"2024-10-21\"\n",
+    "\n",
+    "    # Create config dict for RedTeam\n",
+    "    azure_openai_config = {\n",
+    "        \"azure_endpoint\": azure_openai_endpoint,\n",
+    "        \"azure_deployment\": azure_openai_deployment,\n",
+    "        \"api_key\": azure_openai_api_key,\n",
+    "        \"api_version\": azure_openai_api_version,\n",
+    "    }\n",
+    "\n",
+    "print(f\"Azure AI Project: {azure_ai_project}\")\n",
+    "print(f\"Azure OpenAI Deployment: {azure_openai_deployment}\")\n",
+    "print(f\"Azure OpenAI Endpoint: {azure_openai_endpoint}\")"
    ]
   },
   {
@@ -203,7 +243,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "# Create the `RedTeam` instance with minimal configurations\n",
+    "# Create the `RedTeam` instance with the deployed Azure AI Foundry project\n",
     "red_team = RedTeam(\n",
     "    azure_ai_project=azure_ai_project,\n",
     "    credential=credential,\n",
@@ -257,12 +297,9 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "# Define a model configuration to test\n",
-    "azure_oai_model_config = {\n",
-    "    \"azure_endpoint\": azure_openai_endpoint,\n",
-    "    \"azure_deployment\": azure_openai_deployment,\n",
-    "    \"api_key\": azure_openai_api_key,\n",
-    "}"
+    "# Use the model configuration from deployment output\n",
+    "# This configuration is automatically loaded from the azure_openai_config environment variable\n",
+    "print(f\"Using Azure OpenAI configuration: {azure_openai_config}\")"
    ]
   },
   {
@@ -278,9 +315,9 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "# Run the red team scan called \"Intermediary-Model-Target-Scan\"\n",
+    "# Run the red team scan called \"Intermediary-Model-Target-Scan\" using the deployed model\n",
     "result = await red_team.scan(\n",
-    "    target=azure_oai_model_config, scan_name=\"Intermediary-Model-Target-Scan\", attack_strategies=[AttackStrategy.Flip]\n",
+    "    target=azure_openai_config, scan_name=\"Intermediary-Model-Target-Scan\", attack_strategies=[AttackStrategy.Flip]\n",
     ")"
    ]
   },
@@ -346,7 +383,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "# Create the RedTeam instance with all of the risk categories with 5 attack objectives generated for each category\n",
+    "# Create the RedTeam instance with all of the risk categories using the deployed Azure AI Foundry project\n",
     "model_red_team = RedTeam(\n",
     "    azure_ai_project=azure_ai_project,\n",
     "    credential=credential,\n",
@@ -420,9 +457,10 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "path_to_prompts = \".\\data\\prompts.json\"\n",
+    "path_to_prompts = \"./data/prompts.json\"\n",
     "\n",
     "# Create the RedTeam specifying the custom attack seed prompts to use as objectives\n",
+    "# Using the deployed Azure AI Foundry project\n",
     "custom_red_team = RedTeam(\n",
     "    azure_ai_project=azure_ai_project,\n",
     "    credential=credential,\n",

scenarios/evaluate/AI_RedTeaming/azure.yaml

@@ -0,0 +1,45 @@
+# yaml-language-server: $schema=https://raw.githubusercontent.com/Azure/azure-dev/main/schemas/v1.0/azure.yaml.json
+
+name: ai-red-teaming
+metadata:
+  template: ai-red-teaming@0.0.1-beta
+summary: Azure AI Red Teaming Infrastructure
+description: |
+  Deploy the required Azure infrastructure for AI Red Teaming scenarios including Azure AI Foundry, 
+  OpenAI services, storage, and proper RBAC configuration for authentication.
+author: Azure AI Evaluation Team
+repositoryPath: .
+
+workflows:
+  up:
+    steps:
+      - azd: provision
+      - azd: deploy
+
+infra:
+  provider: bicep
+  path: infra
+
+pipeline:
+  provider: azdo
+  path: .azdo
+
+services: {}
+
+hooks:
+  postprovision:
+    shell: pwsh
+    run: |
+      # Output the environment variables for easy copy-paste
+      Write-Host "=== Azure AI Red Teaming Environment Variables ===" -ForegroundColor Green
+      Write-Host "Copy these environment variables to your .env file or notebook:" -ForegroundColor Yellow
+      Write-Host ""
+      Write-Host "AZURE_SUBSCRIPTION_ID=$env:AZURE_SUBSCRIPTION_ID" -ForegroundColor Cyan
+      Write-Host "AZURE_RESOURCE_GROUP=$env:AZURE_RESOURCE_GROUP" -ForegroundColor Cyan
+      Write-Host "AZURE_AI_PROJECT_NAME=$env:AZURE_AI_PROJECT_NAME" -ForegroundColor Cyan
+      Write-Host "AZURE_AI_PROJECT_CONNECTION_STRING=$env:AZURE_AI_PROJECT_CONNECTION_STRING" -ForegroundColor Cyan
+      Write-Host "AZURE_OPENAI_ENDPOINT=$env:AZURE_OPENAI_ENDPOINT" -ForegroundColor Cyan
+      Write-Host "AZURE_STORAGE_ACCOUNT_NAME=$env:AZURE_STORAGE_ACCOUNT_NAME" -ForegroundColor Cyan
+      Write-Host ""
+      Write-Host "Authentication is configured to use DefaultAzureCredential." -ForegroundColor Green
+      Write-Host "Make sure you're logged in with 'az login' or 'azd auth login'." -ForegroundColor Yellow

scenarios/evaluate/AI_RedTeaming/infra/.env.sample

@@ -0,0 +1,36 @@
+# Azure AI Red Teaming Environment Variables
+# Copy this file to .env and update with your actual values after deployment
+
+# Azure Subscription and Resource Group
+AZURE_SUBSCRIPTION_ID=your-subscription-id
+AZURE_TENANT_ID=your-tenant-id
+AZURE_RESOURCE_GROUP=your-resource-group-name
+AZURE_LOCATION=eastus
+
+# Azure AI Foundry
+AZURE_AI_HUB_NAME=your-ai-hub-name
+AZURE_AI_PROJECT_NAME=your-ai-project-name
+AZURE_AI_PROJECT_CONNECTION_STRING=azureml://subscriptions/your-sub-id/resourcegroups/your-rg/providers/Microsoft.MachineLearningServices/workspaces/your-project-name
+
+# Azure OpenAI (with latest models)
+AZURE_OPENAI_ENDPOINT=https://your-openai-account.openai.azure.com/
+AZURE_OPENAI_API_KEY=your-openai-api-key
+# Available models: gpt-4o, gpt-4o-mini, o1-preview
+
+# Azure Storage
+AZURE_STORAGE_ACCOUNT_NAME=your-storage-account-name
+AZURE_STORAGE_CONNECTION_STRING=DefaultEndpointsProtocol=https;AccountName=...
+
+# Azure AI Search (optional)
+AZURE_SEARCH_SERVICE_NAME=your-search-service-name
+AZURE_SEARCH_ENDPOINT=https://your-search-service.search.windows.net/
+AZURE_SEARCH_API_KEY=your-search-api-key
+
+# Key Vault (optional)
+AZURE_KEY_VAULT_NAME=your-key-vault-name
+
+# Authentication - DefaultAzureCredential will be used
+# Make sure you're logged in with one of:
+# - az login (Azure CLI)
+# - azd auth login (Azure Developer CLI)
+# - Connect-AzAccount (Azure PowerShell)