Azure Arc can't enabled on kubernetes cluster which created on Azure

[dlsyliu]

Describe the bug

Hi team,
When i executed the below command use local machine Azure CLI for enable Arc, it can't executed due to my local "Permission denied" issue, i also tried "sudo az connectedk8s connect --resource-group AzureArcTest1 --name poc" still same error, could you help to guide how to fix that? Thank you very much.

• az aks get-credentials --resource-group AzureArcTest1 --name poc
• az connectedk8s connect --resource-group AzureArcTest1 --name poc

Related command

az connectedk8s connect --resource-group AzureArcTest1 --name poc

Errors

The command failed with an unexpected error. Here is the traceback:
[Errno 13] Permission denied: '/Users/shiyangliu/.azure/kubectl-client/kubectl'
Traceback (most recent call last):
File "/opt/homebrew/Cellar/azure-cli/2.70.0/libexec/lib/python3.12/site-packages/knack/cli.py", line 233, in invoke
cmd_result = self.invocation.execute(args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Cellar/azure-cli/2.70.0/libexec/lib/python3.12/site-packages/azure/cli/core/commands/init.py", line 666, in execute
raise ex
File "/opt/homebrew/Cellar/azure-cli/2.70.0/libexec/lib/python3.12/site-packages/azure/cli/core/commands/init.py", line 734, in _run_jobs_serially
results.append(self._run_job(expanded_arg, cmd_copy))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Cellar/azure-cli/2.70.0/libexec/lib/python3.12/site-packages/azure/cli/core/commands/init.py", line 703, in _run_job
result = cmd_copy(params)
^^^^^^^^^^^^^^^^
File "/opt/homebrew/Cellar/azure-cli/2.70.0/libexec/lib/python3.12/site-packages/azure/cli/core/commands/init.py", line 336, in call
return self.handler(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Cellar/azure-cli/2.70.0/libexec/lib/python3.12/site-packages/azure/cli/core/commands/command_operation.py", line 120, in handler
return op(**command_args)
^^^^^^^^^^^^^^^^^^
File "/Users/shiyangliu/.azure/cliextensions/connectedk8s/azext_connectedk8s/custom.py", line 776, in create_connectedk8s
crd_cleanup_force_delete(kubectl_client_location, kube_config, kube_context)
File "/Users/shiyangliu/.azure/cliextensions/connectedk8s/azext_connectedk8s/custom.py", line 4414, in crd_cleanup_force_delete
response_helm_delete = Popen(cmd_helm_delete, stdout=PIPE, stderr=PIPE)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Cellar/python@3.12/3.12.9/Frameworks/Python.framework/Versions/3.12/lib/python3.12/subprocess.py", line 1028, in init
self._execute_child(args, executable, preexec_fn, close_fds,
File "/opt/homebrew/Cellar/python@3.12/3.12.9/Frameworks/Python.framework/Versions/3.12/lib/python3.12/subprocess.py", line 1963, in _execute_child
raise child_exception_type(errno_num, err_msg, err_filename)
PermissionError: [Errno 13] Permission denied: '/Users/shiyangliu/.azure/kubectl-client/kubectl'

Issue script & Debug output

Get namespace of release: azure-arc.log

Expected behavior

Deploys the Azure Arc agents to the cluster, I can connect an existing Kubernetes cluster

Environment Summary

azure-cli: 2.70.0
Extensions:

• connectedk8s, "version": "1.10.7"
• k8s-extension, "version": "1.6.4"
• connectedmachine , "version": "1.0.0"

Additional context

No response

[yonzhan]

Thank you for opening this issue, we will look into it.

[mbifeld]

Hi @dlsyliu, is the AzureArcTest1 resource an AKS resource? If not, I don't believe the az aks get-credentials command will work. I can't speak to the issue with the connectedk8s command extension.

[dlsyliu]

Hello Mitchell, Thanks for your update, yes, the resource group name is not correct, i just giving an example names as "AzureArcTest1", That problem has been solved, after i rerun the command as below:
az aks get-credentials --resource-group POC --name poc --overwrite-existing
az connectedk8s connect --resource-group POC --name poc

[dlsyliu]

Hello @mbifeld I created a new Azure cluster, but the same error occurs, so I still can't enable Arc on my existing cluster. It seems to be a permission issue on my local machine. Even when I switch to the root account using the command as below, it still doesn't work. When I run the az command, it uses Azure's kubectl. My macOS version is 15.3.2. Do you have any other ideas? Thanks!

1. I tried moving the Azure kubectl to another location (e.g: /usr/local/bin/kubectl). While I can execute kubectl from other directories, it still can't be executed from within the .azure folder.

2. I also tried using my local kubectl by setting export KUBECTL_COMMAND=/opt/homebrew/opt/kubernetes-cli@1.31/bin/kubectl, but that didn't work either.

3. I tried giving permission using sudo chmod -R 755 /Users/shiyangliu/.azure/kubectl-client/kubectl, but it still doesn't work.

4. sudo az connectedk8s connect --name "aks1-arc" --resource-group "aks_group" --location "eastus" --correlation-id "c18ab9d0-685e-48e7-ab55-12588447b0ed" --tags "Datacenter City StateOrDistrict CountryOrRegion

[mbifeld]

Hi @dlsyliu, since the error trace fails in the crd_cleanup_force_delete, it might be a permission issue on the cluster itself. Can you delete the crd using kubectl directly, and see if there's a permission issue? That might determine if it's permissions on the cluster. If local permissions are the issue, you could try running the commands in Azure Cloud Shell.

[mbifeld]

Hi @dlsyliu, any updates on the issue?

[dlsyliu]

Hello @mbifeld
The issue has been resolved. After I removed the kubectl binary in the ./azure folder and copied my local kubectl (from /usr/local/bin/kubectl) into that folder, I gained permission to execute the command. Thanks for your help!

[mbifeld]

Great, I'm glad it's resolved!

[dlsyliu]

Yes, i tried to run the command in Azure Cloud Shell, but there not install the kubectl as well, so can't verify my command, Thanks a lot for your tracking this issues, have a nice day!