Skip to content

Latest commit

 

History

History
333 lines (240 loc) · 21.1 KB

File metadata and controls

333 lines (240 loc) · 21.1 KB

Managed Identity Using Azure Entra ID Workload Identity (AKS)

This documentation is for instructions on using ambient credentials within Azure Kubernetes Services (AKS). Full documentation on Command Cert Manager Issuer can be found here.

Prerequisites

Background

There are two types of managed identities that your Azure AKS workload may use:

  • System-assigned managed identity (MSI)
    • Automatically created and managed by Azure at the cluster level. This identity can not be shared with other Azure resources. This is used by default.
  • User-assigned managed identity (UAMI)
    • Created and managed by you. Identity can be shared with other Azure resources and associated with Kubernetes ServiceAccounts via Azure AD Workload Identity. Requires explicit workload identity configuration (show below).

Since you are using ambient credentials generated by your Azure AKS workload and targeting these credentials for your Command instance, you will need to create an Azure App Registration. We will walk through App Registration configuration in this document.

MSI vs UAMI: Which to use

While system-assigned managed identity (MSI) is easy to use and enabled by default, user-assigned managed identity (UAMI) is the recommended identity type to use for your workload. UAMI identities can be shared with other AKS clusters and workloads, and offer more control over how the identity is used. If your app registration requires a role assignment, you must use a UAMI. An MSI cannot be assigned to an app registration role.

Quick Decision Guide

Scenario Recommended Identity Type
Simple setup, single cluster System-Assigned (MSI)
Multiple clusters need same identity User-Assigned (UAMI)
App registration requires role assignment User-Assigned (UAMI) - Required
Production environments User-Assigned (UAMI)
Development/testing Either (MSI for simplicity)

System-Assigned Managed Identity (MSI)

By default, your AKS cluster is configured to use system-assigned managed identity. Your workload should automatically use the identity assigned to the cluster. You will need to set up the scope of the issuer to reference an app registration. Lastly, you will need to make sure the object ID of the managed identity is associated to a security claim in Keyfactor Command.

  1. Install cert-manager to your AKS cluster. Installation steps

  2. Install command-cert-manager-issuer to your AKS cluster. Installation steps

  3. Create an Azure App Registration. Installation steps

  4. Deploy Issuer or ClusterIssuer Resource. Installation steps

    • To use ambient credentials, do not supply a commandSecretName to your issuer's specification.
    • IMPORTANT: Fill in the scopes in your issuer's specification with the Application ID URI of your App Registration, suffixed with ./default. Example:
      # Example issuer configuration
      spec:
          scopes: "api://your-app-registration-id/.default"
  5. Add the system-assigned managed identity object ID to a security claim in Keyfactor Command

    export AKS_CLUSTER_RESOURCE_GROUP="" # the resource group your AKS cluster is deployed to
    export AKS_CLUSTER_NAME="" # the name of your AKS cluster
    export CURRENT_TENANT=$(az account show --query tenantId --output tsv)
    
    echo "AKS Cluster Resource Group: $AKS_CLUSTER_RESOURCE_GROUP"
    echo "AKS Cluster Name: $AKS_CLUSTER_NAME"
    
    # Get the principal ID of your AKS cluster
    AKS_CLUSTER_OBJECT_ID=$(az aks show --resource-group $AKS_CLUSTER_RESOURCE_GROUP --name $AKS_CLUSTER_NAME --query "identityProfile.kubeletidentity.objectId" -o tsv)
    echo "AKS Cluster MSI Object ID: $AKS_CLUSTER_OBJECT_ID"
    
    echo "View then OIDC configuration for the Entra OIDC token issuer: https://login.microsoftonline.com/$CURRENT_TENANT/v2.0/.well-known/openid-configuration"
    
    echo "Authority: https://login.microsoftonline.com/$CURRENT_TENANT/v2.0"

    Note: AKS workloads inherit the kubelet's managed identity, not the cluster's control plane identity. This is why we use identityProfile.kubeletidentity.objectId rather than identity.principalId.

    You can map the object ID to an OAuth Subject or OAuth Object ID security claim in Keyfactor Command. Make sure the security claim is associated to a security role with the required permissions. Please refer to the Configuring Command Configure Command Security Roles and Claims section for security role requirements.

    Make sure an identity provider is configured in Keyfactor Command with the authority set to the authority output above.

User-Assigned Managed Identity (UAMI)

User-assigned managed identity configuration is more involved, but allows the identity to be shared across different AKS clusters. The AKS cluster will need to be configured to allow workload identity and the Command Issuer's ServiceAccount will need to reference the client ID of the user-assigned managed identity. You will need to make sure the principal ID of the user-assigned managed identity is associated to a security claim in Keyfactor Command.

  1. Install cert-manager to your AKS cluster. Installation steps

  2. Enable OIDC and Workload Identity on your AKS cluster. Learn more

    export AKS_CLUSTER_RESOURCE_GROUP="" # the resource group your AKS cluster is deployed to
    export AKS_CLUSTER_NAME="" # the name of your AKS cluster
    
    echo "AKS Cluster Resource Group: $AKS_CLUSTER_RESOURCE_GROUP"
    echo "AKS Cluster Name: $AKS_CLUSTER_NAME"
    
    echo "Enabling OIDC and workload identity on AKS cluster..."
    
    az aks update \
        --name ${AKS_CLUSTER_NAME} \
        --resource-group ${AKS_CLUSTER_RESOURCE_GROUP} \
        --enable-oidc-issuer \
        --enable-workload-identity
  3. Create a user-assigned managed identity

    export UAMI_IDENTITY_NAME="command-issuer-uami" # the name you want to give your UAMI
    
    echo "Creating user assigned managed identity $UAMI_IDENTITY_NAME..."
    
    az identity create --name "${UAMI_IDENTITY_NAME}" --resource-group "${AKS_CLUSTER_RESOURCE_GROUP}"
    
    export UAMI_CLIENT_ID=$(az identity show --name $UAMI_IDENTITY_NAME --resource-group $AKS_CLUSTER_RESOURCE_GROUP --query clientId --output tsv)
    
    echo "Client ID of user-assigned managed identity: $UAMI_CLIENT_ID"
  4. Deploy Command Cert Manager Issuer with ServiceAccount labeled to use workload identity and UAMI client ID

    export UAMI_CLIENT_ID=$(az identity show --name $UAMI_IDENTITY_NAME --resource-group $AKS_CLUSTER_RESOURCE_GROUP --query clientId --output tsv) # should be the same as the previous step
    
    export ISSUER_NAMESPACE="command-issuer-system"
    
    echo "Installing command-cert-manager issuer to namespace $ISSUER_NAMESPACE"
    echo "Labeling ServiceAccount to use workload identity with user-assigned-managed-identity client ID $UAMI_CLIENT_ID..."
    
    helm install command-cert-manager-issuer command-issuer/command-cert-manager-issuer \
        --namespace $ISSUER_NAMESPACE \
        --create-namespace \
        --set "fullnameOverride=command-cert-manager-issuer" \
        --set-string "podLabels.azure\.workload\.identity/use=true" \
        --set-string "serviceAccount.labels.azure\.workload\.identity/use=true" \
        --set-string "serviceAccount.annotations.azure\.workload\.identity/client-id=${UAMI_CLIENT_ID}"    

    If successful, the Command Issuer Pod will have new environment variables and the Azure WI ServiceAccount token as a projected volume:

    kubectl -n command-issuer-system describe pod
    Containers:
      command-cert-manager-issuer:
        ...
        Environment:
          AZURE_CLIENT_ID:             <UAMI_CLIENT_ID>
          AZURE_TENANT_ID:             <GUID>
          AZURE_FEDERATED_TOKEN_FILE:  /var/run/secrets/azure/tokens/azure-identity-token
          AZURE_AUTHORITY_HOST:        https://login.microsoftonline.com/
        Mounts:
          /var/run/secrets/azure/tokens from azure-identity-token (ro)
          /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-6rmzz (ro)
    ...
    Volumes:
      ...
      azure-identity-token:
        Type:                    Projected (a volume that contains injected data from multiple sources)
        TokenExpirationSeconds:  3600
  5. Associate a Federated Identity Credential (FIC) with the User Assigned Managed Identity. The FIC allows Command Issuer to act on behalf of the Managed Identity by telling Azure to expect:

    • The iss claim of the ServiceAccount token to match the cluster's OIDC Issuer. Azure will also use the Issuer URL to download the JWT signing certificate.
    • The sub claim of the ServiceAccount token to match the ServiceAccount's name and namespace.
    export SERVICE_ACCOUNT_NAME=command-cert-manager-issuer # This is the default Kubernetes ServiceAccount used by the Command Issuer controller.
    export SERVICE_ACCOUNT_NAMESPACE=command-issuer-system # This is the default namespace for Command Issuer used in this doc.
    
    export SERVICE_ACCOUNT_ISSUER=$(az aks show --resource-group $AKS_CLUSTER_RESOURCE_GROUP --name $AKS_CLUSTER_NAME --query "oidcIssuerProfile.issuerUrl" -o tsv)
    
    echo "Service account issuer: $SERVICE_ACCOUNT_ISSUER"
    echo "Creating federated credentials for user-assigned managed identity $UAMI_IDENTITY_NAME in resource group $AKS_CLUSTER_RESOURCE_GROUP..."
    
    az identity federated-credential create \
        --name "${UAMI_IDENTITY_NAME}-federated-credentials" \
        --identity-name "${UAMI_IDENTITY_NAME}" \
        --resource-group "${AKS_CLUSTER_RESOURCE_GROUP}" \
        --issuer "${SERVICE_ACCOUNT_ISSUER}" \
        --subject "system:serviceaccount:${SERVICE_ACCOUNT_NAMESPACE}:${SERVICE_ACCOUNT_NAME}" \
        --audiences "api://AzureADTokenExchange"

    Read more about Workload Identity federation in the Entra ID documentation.

    Read more about the az identity federated-credential command.

  6. Create an Azure App Registration. Installation steps

  7. Deploy Issuer or ClusterIssuer Resource. Installation steps

    • To use ambient credentials, do not supply a commandSecretName to your issuer's specification.
    • IMPORTANT: Fill in the scopes in your issuer's specification with the Application ID URI of your App Registration, suffixed with ./default. Example:
      # Example issuer configuration
      spec:
          scopes: "api://your-app-registration-id/.default"
  8. Add the user-assigned managed identity principal ID to a security claim in Keyfactor Command

    export UAMI_PRINCIPAL_ID=$(az identity show --name $UAMI_IDENTITY_NAME --resource-group $AKS_CLUSTER_RESOURCE_GROUP --query principalId --output tsv)
    export CURRENT_TENANT=$(az account show --query tenantId --output tsv)
    echo "UAMI Principal ID: ${UAMI_PRINCIPAL_ID}"
    
    echo "View then OIDC configuration for the Entra OIDC token issuer: https://login.microsoftonline.com/$CURRENT_TENANT/v2.0/.well-known/openid-configuration"
    
    echo "Authority: https://login.microsoftonline.com/$CURRENT_TENANT/v2.0"

    You can map the principal ID to an OAuth Subject or OAuth Object ID security claim in Keyfactor Command. Make sure the security claim is associated to a security role with the required permissions. Please refer to the Configuring Command Configure Command Security Roles and Claims section for security role requirements.

    Make sure an identity provider is configured in Keyfactor Command with the authority set to the authority output above.

Azure App Registration

The identity server that generates the access token from DefaultAzureCredentials requires a valid scope. The scope supplied to DefaultAzureCredentials sets the audience claim of the access token. The access token is being used for authorization on a resource outside of Azure (Keyfactor Command), so an app registration for Entra AD to represent an external application.

Here is official Azure documentation on how to create an app registration.

After the App Registration is created, expose an API. You can do this by going to Manage > Expose an API and editing the Application ID URI.

IMPORTANT: The Application ID URI will be used in your scopes claim. Make sure to copy this value down. For example, if your Application ID URI is api://abcd, your scope value should be api://abcd/.default.

Application ID URI

App Registration Assignment Requirement

By default, Azure App Registrations do not require an assignment in order for an identity to access to the application. However, there may be some compliance need to require an assignment for an identity to access your app registration. This option can be toggled via the Enterprise Application properties of your App Registration. If enabled, you must use a user-assigned managed identity for your workload (a system-assigned managed identity cannot be assigned a role). If this identity does not have a role assignment to the app registration, you may see the error:

AADSTS501051: Application '<identity-object-id>'(<identity-name>) is not assigned to a role for the application 'api://<application-id-uri>'(<application-name>)

App Registration Assignment Required

If the UAMI identity is tied to an app registration role, the name of the security role can be added as a security claim in Keyfactor Command. Then, the identity can assume any Keyfactor Command security role with that security claim assigned to it.

You can assign the identity to an app registration role from the Enterprise Application. Please refer to the Azure documentation for more information.

For more information about the assignment requirement for app registrations and how this can affect your identities, please see this blog post.

Troubleshooting

This troubleshooting section is intended for issues specific to the Azure AKS environment. If you do not see your issue in these troubleshooting steps, please see the troubleshooting steps in the directory root.

Common Pitfalls

  1. Forgetting the /.default suffix in scopes configuration
  2. Using wrong object ID - If using MSI, MSI uses kubelet identity, not the cluster identity
  3. ServiceAccount mismatch - If using UAMI, federated credentials must exactly match ServiceAccount name/namespace

Determining Which Managed Identity Your AKS Workload is Using

Azure has documentation around determining the managed identity a cluster is using, but this section will confirm if your AKS workload is using UAMI or MSI for its managed identity.

Determine if workload identity is enabled on the AKS cluster

az aks show –-resource-group [group] –-name [name] --query "[oidcIssuerProfile,securityProfile]"

If you see something like this, your AKS cluster has workload identity enabled:

[
  {
    "enabled": true,
    "issuerUrl": "https://<issuer-url>"
  },
  {
    "azureKeyVaultKms": null,
    "defender": null,
    "imageCleaner": null,
    "workloadIdentity": {
      "enabled": true
    }
  }
]

Check if the ServiceAccount is annotated with a client ID and workload is enabled.

Run this script to see if your ServiceAccount is annotated with the client ID of the UAMI and workload identity is enabled.

kubectl describe serviceaccount <serivce-account-name> --namespace <service-account-namespace>
...
Labels:              ...
                     azure.workload.identity/use=true
Annotations:         azure.workload.identity/client-id: <uami-client-id>
                     ...
Image pull secrets:  <none>
Mountable secrets:   <none>
Tokens:              <none>
Events:              <none>

Check if your Kubernetes pod is labeled to use workload identity

Run this script to see if your Kubernetes pod is running workload identity enabled.

kubectl get pods --namespace <namespace> --show-labels
NAME                              READY   STATUS    RESTARTS   AGE    LABELS
command-issuer-86c4fdfb67-h4vqb   1/1     Running   0          105s   app.kubernetes.io/instance=cert-manager-issuer,app.kubernetes.io/name=command-cert-manager-issuer,azure.workload.identity/use=true,pod-template-hash=86c4fdfb67

Conclusion

If all of the above steps indicate your cluster has workload identity enabled, the pod is labeled to use workload identity, and the ServiceAccount is annotated with the UAMI client ID, your workload is most likely using user-assigned managed identity.

Required Query Variable 'Resource' Is Missing

If you see the following error, this indicates your issuer / cluster issuer is missing a scopes field in its spec. DefaultAzureCredentials requires a valid scope, which should reference the app registration.

failed to authenticate a system assigned identity. The endpoint responded with {\"error\":\"invalid_request\",\"error_description\":\"Required query variable 'resource' is missing\"}

AADSTS500011: Resource principal named was not found in the tenant

If you see the following error, this indicates the scopes specification on your issuer / cluster issuer is present but pointing to an invalid resource (make sure it's pointing to the app registration application ID URI).

AADSTS500011: The resource principal named <scope> was not found in the tenant named <tenant-id>. This can happen if the application has not been installed by the administrator of the tenant or consented to by any user in the tenant. You might have sent your authentication request to the wrong tenant

If the scopes field is set to a valid application ID URI, make sure you are targeting the /.default suffix.

AADSTS501051: Application is not assigned to a role

If you see the following error, this indicates the identity you're using does not have a role assignment to an app registration that requires a role assignment. See this section for more details.

AADSTS501051: Application '<identity-object-id>'(<identity-name>) is not assigned to a role for the application 'api://<application-id-uri>'(<application-name>)

AADSTS700213: No matching federated identity record found for presented assertion subject

If you see the following error, the user-assigned managed identity (UAMI) is assigned to the command issuer's Kubernete ServiceAccount and is trying to use it. However, the UAMI is missing a federated credential that trusts the ServiceAccount. Please refer to the user-assigned managed identity section and check for the instructions on creating a federated identity credential. The federeated credential must match the Kubernetes service account's name and namespace.

AADSTS700213: No matching federated identity record found for presented assertion subject 'system:serviceaccount:<service-account-namespace>:<service-account-name>'. Check your federated identity credential Subject, Audience and Issuer against the presented assertion. https://learn.microsoft.com/entra/workload-id/workload-identity-federation