As discussed in Azure/azureml-assets#4675, it is currently impossible to deploy mlflow models via the SDKv2 without providing a custom scoring script. This means that deploying mlflow models via the UI is not possible as you can not provide a custom scoring script if the selected model is a mlflow model.
We tracked the source of the bug to some missing dependencies and with these resolved, there seems to be an issue with some environment variables that are now missing.
Problem summary from the other issue:
Pinning the old image mcr.microsoft.com/azureml/curated/mlflow-py312-inference@sha256:a0b9f5738c9fc1394f1490b19daca346ccfbf584111bfdc7e22e77df6aee0c11
and its conda dependencies does not resolve the issue.
The problem does not appear to be caused by the image update itself.
As soon as an explicit Environment is attached to the deployment (even when the environment is an exact clone of the old curated image + conda), Azure ML seems to change the underlying inference behavior. MLflow autoinference no longer injects the expected environment variables (AZUREML_MODEL_DIR, MLFLOW_MODEL_FOLDER), and the autogenerated MLflow scoring script then fails with:
File "/azureml-envs/azureml_921778accf93023eacca932a13d4e1c0/lib/python3.10/site-packages/azureml_inference_server_http/server/user_script.py", line 77, in load_script
main_module_spec.loader.exec_module(user_module)
File "", line 883, in exec_module
File "", line 241, in _call_with_frames_removed
File "/var/mlflow_resources/mlflow_score_script.py", line 165, in
model_path = os.path.join(os.getenv("AZUREML_MODEL_DIR"), os.getenv("MLFLOW_MODEL_FOLDER"))
File "/azureml-envs/azureml_921778accf93023eacca932a13d4e1c0/lib/python3.10/posixpath.py", line 90, in join
genericpath._check_arg_types('join', a, *p)
File "/azureml-envs/azureml_921778accf93023eacca932a13d4e1c0/lib/python3.10/genericpath.py", line 152, in _check_arg_types
raise TypeError(f'{funcname}() argument must be str, bytes, or '
TypeError: join() argument must be str, bytes, or os.PathLike object, not `'NoneType'``
As discussed in Azure/azureml-assets#4675, it is currently impossible to deploy mlflow models via the SDKv2 without providing a custom scoring script. This means that deploying mlflow models via the UI is not possible as you can not provide a custom scoring script if the selected model is a mlflow model.
We tracked the source of the bug to some missing dependencies and with these resolved, there seems to be an issue with some environment variables that are now missing.
Problem summary from the other issue:
Pinning the old image mcr.microsoft.com/azureml/curated/mlflow-py312-inference@sha256:a0b9f5738c9fc1394f1490b19daca346ccfbf584111bfdc7e22e77df6aee0c11
and its conda dependencies does not resolve the issue.
The problem does not appear to be caused by the image update itself.
As soon as an explicit Environment is attached to the deployment (even when the environment is an exact clone of the old curated image + conda), Azure ML seems to change the underlying inference behavior. MLflow autoinference no longer injects the expected environment variables (AZUREML_MODEL_DIR, MLFLOW_MODEL_FOLDER), and the autogenerated MLflow scoring script then fails with:
File "/azureml-envs/azureml_921778accf93023eacca932a13d4e1c0/lib/python3.10/site-packages/azureml_inference_server_http/server/user_script.py", line 77, in load_script
main_module_spec.loader.exec_module(user_module)
File "", line 883, in exec_module
File "", line 241, in _call_with_frames_removed
File "/var/mlflow_resources/mlflow_score_script.py", line 165, in
model_path = os.path.join(os.getenv("AZUREML_MODEL_DIR"), os.getenv("MLFLOW_MODEL_FOLDER"))
File "/azureml-envs/azureml_921778accf93023eacca932a13d4e1c0/lib/python3.10/posixpath.py", line 90, in join
genericpath._check_arg_types('join', a, *p)
File "/azureml-envs/azureml_921778accf93023eacca932a13d4e1c0/lib/python3.10/genericpath.py", line 152, in _check_arg_types
raise TypeError(f'{funcname}() argument must be str, bytes, or '
TypeError: join() argument must be str, bytes, or os.PathLike object, not `'NoneType'``