|
| 1 | +{ |
| 2 | + "cells": [ |
| 3 | + { |
| 4 | + "cell_type": "markdown", |
| 5 | + "metadata": {}, |
| 6 | + "source": [ |
| 7 | + "**NOTE: This notebook has no github workflows as the motive of this notebook is to provide a solution how to use private packages in AML SDK v2.**\n", |
| 8 | + "\n", |
| 9 | + "# Create Azure Machine Learning Custom Environment\n", |
| 10 | + "\n", |
| 11 | + "**Requirements** - In order to benefit from this tutorial, you will need:\n", |
| 12 | + "- A basic understanding of Machine Learning\n", |
| 13 | + "- An Azure account with an active subscription. [Create an account for free](https://azure.microsoft.com/free/?WT.mc_id=A261C142F)\n", |
| 14 | + "- An Azure ML workspace - [Configure workspace](../../jobs/configuration.ipynb) \n", |
| 15 | + "- A python environment\n", |
| 16 | + "- Installed Azure Machine Learning Python SDK v2 - [install instructions](../../README.md) - check the getting started section\n", |
| 17 | + "\n", |
| 18 | + "**Learning Objectives** - By the end of this tutorial, you should be able to:\n", |
| 19 | + "- Create a custom environment using private packages from python SDK using conda spec\n", |
| 20 | + " \n", |
| 21 | + "**Motivations** - Azure Machine Learning environments are an encapsulation of the environment where your machine learning training happens. By default your workspace has several curated environments already available. This notebook explains how to create a custom environment to run your specific task if you need to." |
| 22 | + ] |
| 23 | + }, |
| 24 | + { |
| 25 | + "cell_type": "markdown", |
| 26 | + "metadata": {}, |
| 27 | + "source": [ |
| 28 | + "# 1. Connect to Azure Machine Learning Workspace\n", |
| 29 | + "\n", |
| 30 | + "The [workspace](https://docs.microsoft.com/en-us/azure/machine-learning/concept-workspace) is the top-level resource for Azure Machine Learning, providing a centralized place to work with all the artifacts you create when you use Azure Machine Learning. In this section we will connect to the workspace in which the job will be run.\n", |
| 31 | + "\n", |
| 32 | + "## 1.1. Import the required libraries" |
| 33 | + ] |
| 34 | + }, |
| 35 | + { |
| 36 | + "cell_type": "code", |
| 37 | + "execution_count": null, |
| 38 | + "metadata": { |
| 39 | + "name": "libraries" |
| 40 | + }, |
| 41 | + "outputs": [], |
| 42 | + "source": [ |
| 43 | + "# import required libraries\n", |
| 44 | + "from azure.ai.ml import MLClient\n", |
| 45 | + "from azure.ai.ml.entities import Environment, BuildContext\n", |
| 46 | + "from azure.identity import DefaultAzureCredential" |
| 47 | + ] |
| 48 | + }, |
| 49 | + { |
| 50 | + "cell_type": "markdown", |
| 51 | + "metadata": {}, |
| 52 | + "source": [ |
| 53 | + "## 1.2. Configure workspace details and get a handle to the workspace\n", |
| 54 | + "\n", |
| 55 | + "To connect to a workspace, we need identifier parameters - a subscription, resource group and workspace name. We will use these details in the `MLClient` from `azure.ai.ml` to get a handle to the required Azure Machine Learning workspace. We use the default [default azure authentication](https://docs.microsoft.com/en-us/python/api/azure-identity/azure.identity.defaultazurecredential?view=azure-python) for this tutorial. Check the [configuration notebook](../../jobs/configuration.ipynb) for more details on how to configure credentials and connect to a workspace." |
| 56 | + ] |
| 57 | + }, |
| 58 | + { |
| 59 | + "cell_type": "code", |
| 60 | + "execution_count": null, |
| 61 | + "metadata": { |
| 62 | + "name": "workspace_details" |
| 63 | + }, |
| 64 | + "outputs": [], |
| 65 | + "source": [ |
| 66 | + "# Enter details of your AML workspace\n", |
| 67 | + "subscription_id = \"<SUBSCRIPTION_ID>\"\n", |
| 68 | + "resource_group = \"<RESOURCE_GROUP>\"\n", |
| 69 | + "workspace = \"<AML_WORKSPACE_NAME>\"" |
| 70 | + ] |
| 71 | + }, |
| 72 | + { |
| 73 | + "cell_type": "code", |
| 74 | + "execution_count": null, |
| 75 | + "metadata": { |
| 76 | + "name": "get_workspace" |
| 77 | + }, |
| 78 | + "outputs": [], |
| 79 | + "source": [ |
| 80 | + "# get a handle to the workspace\n", |
| 81 | + "ml_client = MLClient(\n", |
| 82 | + " DefaultAzureCredential(), subscription_id, resource_group, workspace\n", |
| 83 | + ")" |
| 84 | + ] |
| 85 | + }, |
| 86 | + { |
| 87 | + "cell_type": "markdown", |
| 88 | + "metadata": {}, |
| 89 | + "source": [ |
| 90 | + "OR" |
| 91 | + ] |
| 92 | + }, |
| 93 | + { |
| 94 | + "cell_type": "code", |
| 95 | + "execution_count": null, |
| 96 | + "metadata": {}, |
| 97 | + "outputs": [], |
| 98 | + "source": [ |
| 99 | + "ml_client = MLClient.from_config(DefaultAzureCredential())" |
| 100 | + ] |
| 101 | + }, |
| 102 | + { |
| 103 | + "cell_type": "markdown", |
| 104 | + "metadata": {}, |
| 105 | + "source": [ |
| 106 | + "# 2. Create Environment\n", |
| 107 | + "Azure Machine Learning [environments](https://docs.microsoft.com/en-us/azure/machine-learning/concept-environments) are an encapsulation of the environment where your machine learning training happens. They specify the Python packages, environment variables, and software settings around your training and scoring scripts. They also specify run times (Python, Spark, or Docker). The environments are managed and versioned entities within your Machine Learning workspace that enable reproducible, auditable, and portable machine learning workflows across a variety of computes.\n", |
| 108 | + "\n", |
| 109 | + "The workspace contains several curated environments by default to use as-is. However, you can create your own custom environment to meet your specific needs.\n", |
| 110 | + "\n", |
| 111 | + "The `Environment` class will be used to create a custom environment. It accepts the following key parameters:\n", |
| 112 | + "- `name` - Name of the environment.\t\t\n", |
| 113 | + "- `version`\t- Version of the environment. If omitted, Azure ML will autogenerate a version.\t\t\n", |
| 114 | + "- `image` - The Docker image to use for the environment. Either `image` or `build` is required to create environment.\n", |
| 115 | + "- `conda_file` - The standard conda YAML [configuration file](https://conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html#creating-an-environment-file-manually) of the dependencies for a conda environment. It can be used with a `image`. If specified, Azure ML will build the conda environment on top of the Docker image provided.\n", |
| 116 | + "- `BuildContext`- The Docker build context configuration to use for the environment. Either `image` or `build` is required to create environment.\n", |
| 117 | + " - `path`- Local path to the directory to use as the build context.\t\t\n", |
| 118 | + " - `dockerfile_path` - Relative path to the Dockerfile within the build context.\n", |
| 119 | + "- `description`\t- Description of the environment.\t\t\n", |
| 120 | + "\n", |
| 121 | + "## 2.1 Create a custom environment using a conda file\n", |
| 122 | + "In this sample we will create an environment in Azure Machine Learning workspace using a docker image. The `Environment` is used to initialize the details. The `MLClient` is used to create the environment in the workspace.\n", |
| 123 | + "\n", |
| 124 | + "#### Define the private wheel file URLs (for example, from Azure Blob Storage)" |
| 125 | + ] |
| 126 | + }, |
| 127 | + { |
| 128 | + "cell_type": "code", |
| 129 | + "execution_count": null, |
| 130 | + "metadata": {}, |
| 131 | + "outputs": [], |
| 132 | + "source": [ |
| 133 | + "private_wheel_1 = \"https://<your_blob_storage_account>.blob.core.windows.net/<your_container>/your-private-wheel-1.whl\"\n", |
| 134 | + "private_wheel_2 = \"https://<your_blob_storage_account>.blob.core.windows.net/<your_container>/your-private-wheel-2.whl\"" |
| 135 | + ] |
| 136 | + }, |
| 137 | + { |
| 138 | + "cell_type": "code", |
| 139 | + "execution_count": null, |
| 140 | + "metadata": {}, |
| 141 | + "outputs": [], |
| 142 | + "source": [ |
| 143 | + "env = Environment(\n", |
| 144 | + " name=\"my-custom-env\",\n", |
| 145 | + " image=\"mcr.microsoft.com/azureml/base:latest\", # You can specify a base image, such as the Azure ML base image\n", |
| 146 | + " conda_file={\n", |
| 147 | + " \"dependencies\": [\n", |
| 148 | + " \"python=3.8\", # Specify the Python version\n", |
| 149 | + " \"pip\",\n", |
| 150 | + " {\"pip\": [private_wheel_1, private_wheel_2]},\n", |
| 151 | + " ]\n", |
| 152 | + " },\n", |
| 153 | + ")\n", |
| 154 | + "\n", |
| 155 | + "# Register the environment\n", |
| 156 | + "ml_client.environments.create_or_update(env)" |
| 157 | + ] |
| 158 | + }, |
| 159 | + { |
| 160 | + "cell_type": "markdown", |
| 161 | + "metadata": {}, |
| 162 | + "source": [ |
| 163 | + "# 3. Define a job within the custom environment" |
| 164 | + ] |
| 165 | + }, |
| 166 | + { |
| 167 | + "cell_type": "code", |
| 168 | + "execution_count": null, |
| 169 | + "metadata": {}, |
| 170 | + "outputs": [], |
| 171 | + "source": [ |
| 172 | + "from azure.ai.ml.entities import Job\n", |
| 173 | + "from azure.ai.ml import command\n", |
| 174 | + "\n", |
| 175 | + "job = command(\n", |
| 176 | + " command=\"echo 'hello'\", # Your script or command\n", |
| 177 | + " code=\"./my-script-folder\", # Directory containing the script\n", |
| 178 | + " environment=env, # The environment with private wheels\n", |
| 179 | + " compute=\"my-compute\", # Compute target to use\n", |
| 180 | + " environment_variables={\"ENV_VAR\": \"value\"}, # Optional: environment variables\n", |
| 181 | + ")\n", |
| 182 | + "\n", |
| 183 | + "# Submit the job\n", |
| 184 | + "ml_client.jobs.create_or_update(job)" |
| 185 | + ] |
| 186 | + } |
| 187 | + ], |
| 188 | + "metadata": { |
| 189 | + "description": { |
| 190 | + "description": "Create custom environments from docker and/or conda YAML" |
| 191 | + }, |
| 192 | + "interpreter": { |
| 193 | + "hash": "66962d4c952b5ba37638a017d6cc83bab37d76f69b13c17d86b9f71233a0aa71" |
| 194 | + }, |
| 195 | + "kernelspec": { |
| 196 | + "display_name": "Python 3.10 - SDK v2", |
| 197 | + "language": "python", |
| 198 | + "name": "python310-sdkv2" |
| 199 | + } |
| 200 | + }, |
| 201 | + "nbformat": 4, |
| 202 | + "nbformat_minor": 4 |
| 203 | +} |
0 commit comments