[machinelearningservices] Azure ML workspace stuck in soft-deleted state: create blocked, permanently-delete cannot find workspace

[azure-client-tools-agent]

Source: Azure/azure-cli#33756 (by @vairakkumaarsvs)
Affected extension: machinelearningservices (src/machinelearningservices/)

---

<<UNTRUSTED:issue-33756>>

Issue #33756 (by @vairakkumaarsvs)

Title

Azure ML workspace stuck in soft-deleted state: create blocked, permanently-delete cannot find workspace

Body

Describe the bug

Azure CLI Version:
2.88.0

ML Extension:
2.44.1

Subscription:

Resource Group:
stmlwus2ws01-rg

Workspace:
stmlwus2ws01

Problem:

I deleted an Azure ML workspace.

When I attempt to recreate the workspace, deployment fails with:

"Soft-deleted workspace exists. Please purge or recover it."

However, attempting to permanently delete the workspace fails because the workspace cannot be found:

az ml workspace delete
--name myws
--resource-group my-rg
--subscription
--permanently-delete
--yes

Error:

(UserError) Unable to find workspace:
/subscriptions/.../resourcegroups/my-rg/providers/Microsoft.MachineLearningServices/workspaces/myws

This creates a deadlock:

• Workspace creation fails because a soft-deleted workspace exists.
• Permanent delete fails because the workspace cannot be found.
• No CLI command appears available to enumerate or purge the soft-deleted workspace.

Expected behavior:

Either:

1. az ml workspace delete --permanently-delete should purge the soft-deleted workspace, or
2. The CLI should provide a supported command to list and purge soft-deleted workspaces.

Actual behavior:

Workspace name remains blocked but cannot be purged.

Related command

az ml workspace delete
--name myws
--resource-group my-rg
--subscription
--permanently-delete
--yes

Errors

(UserError) Unable to find workspace:
Code: UserError
Message: Unable to find workspace:

Issue script & Debug output

cli: Received HttpResponseError: Traceback (most recent call last):
File "/home/myubuntuuser/.azure/cliextensions/ml/azext_mlv2/manual/custom/workspace.py", line 278, in ml_workspace_delete
del_result = ml_client.workspaces.begin_delete(
name=name, delete_dependent_resources=all_resources, permanently_delete=permanently_delete
)
File "/home/myubuntuuser/.azure/cliextensions/ml/azure/ai/ml/_telemetry/activity.py", line 288, in wrapper
return f(*args, **kwargs)
File "/opt/az/lib/python3.14/site-packages/azure/core/tracing/decorator.py", line 119, in wrapper_use_tracer
return func(*args, **kwargs)
File "/home/myubuntuuser/.azure/cliextensions/ml/azure/ai/ml/operations/_workspace_operations.py", line 355, in begin_delete
return super().begin_delete(
~~~~~~~~~~~~~~~~~~~~^
name, delete_dependent_resources=delete_dependent_resources, permanently_delete=permanently_delete, **kwargs
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
)
^
File "/home/myubuntuuser/.azure/cliextensions/ml/azure/ai/ml/operations/_workspace_operations_base.py", line 494, in begin_delete
workspace: Any = self.get(name, **kwargs)
~~~~~~~~^^^^^^^^^^^^^^^^
File "/home/myubuntuuser/.azure/cliextensions/ml/azure/ai/ml/_telemetry/activity.py", line 288, in wrapper
return f(*args, **kwargs)
File "/opt/az/lib/python3.14/site-packages/azure/core/tracing/decorator.py", line 119, in wrapper_use_tracer
return func(*args, **kwargs)
File "/home/myubuntuuser/.azure/cliextensions/ml/azure/ai/ml/operations/_workspace_operations.py", line 143, in get
return super().get(workspace_name=name, **kwargs)
~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/myubuntuuser/.azure/cliextensions/ml/azure/ai/ml/operations/_workspace_operations_base.py", line 91, in get
obj = self._operation.get(resource_group, workspace_name)
File "/opt/az/lib/python3.14/site-packages/azure/core/tracing/decorator.py", line 119, in wrapper_use_tracer
return func(*args, **kwargs)
File "/home/myubuntuuser/.azure/cliextensions/ml/azure/ai/ml/_restclient/v2024_10_01_preview_tsp/operations/_operations.py", line 10422, in get
map_error(status_code=response.status_code, response=response, error_map=error_map)
~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/az/lib/python3.14/site-packages/azure/core/exceptions.py", line 163, in map_error
raise error
azure.core.exceptions.ResourceNotFoundError: (ResourceNotFound) The Resource 'Microsoft.MachineLearningServices/workspaces/mymlws' under resource group 'mymlws-rg' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix
Code: ResourceNotFound
Message: The Resource 'Microsoft.MachineLearningServices/workspaces/mymlws' under resource group 'mymlws-rg' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix

cli.azure.cli.core.azclierror: Traceback (most recent call last):
File "/opt/az/lib/python3.14/site-packages/knack/cli.py", line 233, in invoke
cmd_result = self.invocation.execute(args)
File "/opt/az/lib/python3.14/site-packages/azure/cli/core/commands/init.py", line 677, in execute
raise ex
File "/opt/az/lib/python3.14/site-packages/azure/cli/core/commands/init.py", line 820, in _run_jobs_serially
results.append(self._run_job(expanded_arg, cmd_copy))
~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/az/lib/python3.14/site-packages/azure/cli/core/commands/init.py", line 789, in _run_job
result = cmd_copy(params)
File "/opt/az/lib/python3.14/site-packages/azure/cli/core/commands/init.py", line 335, in call
return self.handler(*args, **kwargs)
~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
File "/opt/az/lib/python3.14/site-packages/azure/cli/core/commands/command_operation.py", line 120, in handler
return op(**command_args)
File "/home/myubuntuuser/.azure/cliextensions/ml/azext_mlv2/manual/custom/workspace.py", line 285, in ml_workspace_delete
log_and_raise_error(err, debug)
~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^
File "/home/myubuntuuser/.azure/cliextensions/ml/azext_mlv2/manual/custom/raise_error.py", line 193, in log_and_raise_error
raise cli_error
File "/home/myubuntuuser/.azure/cliextensions/ml/azext_mlv2/manual/custom/workspace.py", line 278, in ml_workspace_delete
del_result = ml_client.workspaces.begin_delete(
name=name, delete_dependent_resources=all_resources, permanently_delete=permanently_delete
)
File "/home/myubuntuuser/.azure/cliextensions/ml/azure/ai/ml/_telemetry/activity.py", line 288, in wrapper
return f(*args, **kwargs)
File "/opt/az/lib/python3.14/site-packages/azure/core/tracing/decorator.py", line 119, in wrapper_use_tracer
return func(*args, **kwargs)
File "/home/myubuntuuser/.azure/cliextensions/ml/azure/ai/ml/operations/_workspace_operations.py", line 355, in begin_delete
return super().begin_delete(
~~~~~~~~~~~~~~~~~~~~^
name, delete_dependent_resources=delete_dependent_resources, permanently_delete=permanently_delete, **kwargs
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
)
^
File "/home/myubuntuuser/.azure/cliextensions/ml/azure/ai/ml/operations/_workspace_operations_base.py", line 494, in begin_delete
workspace: Any = self.get(name, **kwargs)
~~~~~~~~^^^^^^^^^^^^^^^^
File "/home/myubuntuuser/.azure/cliextensions/ml/azure/ai/ml/_telemetry/activity.py", line 288, in wrapper
return f(*args, **kwargs)
File "/opt/az/lib/python3.14/site-packages/azure/core/tracing/decorator.py", line 119, in wrapper_use_tracer
return func(*args, **kwargs)
File "/home/myubuntuuser/.azure/cliextensions/ml/azure/ai/ml/operations/_workspace_operations.py", line 143, in get
return super().get(workspace_name=name, **kwargs)
~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/myubuntuuser/.azure/cliextensions/ml/azure/ai/ml/operations/_workspace_operations_base.py", line 91, in get
obj = self._operation.get(resource_group, workspace_name)
File "/opt/az/lib/python3.14/site-packages/azure/core/tracing/decorator.py", line 119, in wrapper_use_tracer
return func(*args, **kwargs)
File "/home/
... [truncated, original was 10958 chars]

Comments

Comment by @yonzhan

Thank you for opening this issue, we will look into it.
<<END:issue-33756>>

[yonzhan]

Thank you for opening this issue, we will look into it.

[azure-client-tools-agent]

Bug Analysis

Root cause: ml_workspace_delete in
src/machinelearningservices/azext_mlv2/manual/custom/workspace.py calls
ml_client.workspaces.begin_delete(name=name, delete_dependent_resources=all_resources, permanently_delete=permanently_delete) directly and only wraps the call with a
generic except Exception that re-raises via log_and_raise_error. When a
workspace is already soft-deleted, the underlying SDK's begin_delete first
calls get(name) to fetch the workspace, which raises
azure.core.exceptions.ResourceNotFoundError (404) because the active
resource no longer exists (only the soft-deleted one does). This bubbles up as
a confusing (UserError) Unable to find workspace even though the caller
explicitly asked to --permanently-delete the workspace, creating a deadlock:
create is blocked by the soft-deleted record, and delete can't find it to purge.

Suggested fix: in ml_workspace_delete, when permanently_delete=True and the
underlying call raises ResourceNotFoundError/HttpResponseError with a 404
indicating the active workspace resource is missing, treat this as an
expected condition for a workspace that only exists in soft-deleted state —
either invoke the workspace's purge-only code path (if the SDK exposes one) or
surface an actionable error message (instead of a generic traceback) telling
the user the workspace is soft-deleted and how to purge it, e.g. via the ARM
recover/purge control-plane operation for
Microsoft.MachineLearningServices/locations/deletedWorkspaces. Add/adjust
error handling so az ml workspace delete --permanently-delete succeeds (or
gives clear guidance) when the target only exists as a soft-deleted resource.

Use this EXACT PR title: [Machinelearningservices] Fix #10126: az ml workspace delete --permanently-delete: Handle ResourceNotFoundError when purging soft-deleted workspace

PR title & description format (required)

This repo enforces a PR format (guide). Please author the PR exactly as follows or CI's Check the Format of Pull Request Title and Content will fail.

Use this EXACT PR title (copy verbatim, do not reword):

[Machinelearningservices] Fix #10126: `az ml workspace delete --permanently-delete`: Handle ResourceNotFoundError when purging soft-deleted workspace

Keep the backticks around the command and the Fix #10126: prefix. You may only adjust the wording after the command (the final summary) if the fix changes; the [Machinelearningservices] prefix, issue link, and backticked command must stay.

Description — follow the PR template and fill in:

• Link the issue — start the Description with a closing keyword so the PR auto-links and closes it: Fixes #10126.
• Related command — the az ... command this affects.
• Description (mandatory) — why the bug happens, what you changed, and the resulting behavior.
• Testing Guide — example command(s) showing the fix works.
• History Notes — leave the title to drive the history note, or add extra lines in the same format (component in brackets + the command in backticks), e.g. [Machinelearningservices] `az <command>`: <note>.
• Keep the template checklist and tick the items you've satisfied.

---

Posted by agent-assist (autonomous bug-fix pipeline).