Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions modules/proc-mcp-gateway-register-mcp-server.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -93,6 +93,11 @@ spec:
* Replace the `spec.targetRef.namespace:` field value with the namespace where your `HTTPRoute` CR is applied. In this example, `_<mcp_test>_` is used.
* Replace the `credentialRef.name:` field value with the name of your `Secret` CR. In this example, `_<mcp_server_one_secret>_` is used. You can omit this parameter if your MCP server does not require authentication or authorization.
* For more information about these parameters, see "Understanding the `MCPServerRegistration` custom resource."
+
[IMPORTANT]
====
A `toolPrefix` value cannot include spaces or special characters.
====

. Apply the CR by running the following command:
+
Expand Down
165 changes: 165 additions & 0 deletions modules/proc-mcp-gateway-ts-authn-issues.adoc
Original file line number Diff line number Diff line change
@@ -0,0 +1,165 @@
// Module included in the following assemblies:
//
// *mcp_gateway_config/mcp-gateway-troubleshooting.adoc

:_mod-docs-content-type: PROCEDURE
[id="proc-mcp-gateway-ts-authn-issues_{context}"]
= Troubleshooting {mcpg} authentication issues

[role="_abstract"]
Authentication errors can happen in a variety of ways, including silent failures, broken sessions, and tool-access denials, depending on your backend MCP server setup. You can troubleshoot common problems by checking your connections and custom resource (CR) configurations.

.Prerequisites

* You installed {mcpg}.
* You installed {prodname}.
* You installed the {oc-first}.
* You configured a `Gateway` object.
* You configured an `HTTPRoute` object for the gateway.
* You registered an MCP server.
* You created a `Secret` CR for authentication.

.Procedure

. When your clients cannot discover OAuth configuration, discovery is not working. Use the following steps to troubleshoot this situation:

.. Retrieve the JSON object that lists the security requirements for your backend MCP server by running the following command:
+
[source,terminal,subs="+quotes"]
----
$ curl http://_<mcp_hostname>_/_<.well-known/oauth-protected-resource>_
----
+
* Replace `_<mcp_hostname>_` with the name of the host for your MCP server.
* Replace `_<.well-known/oauth-protected-resource>_` with the reserved path and JSON file that describes the OAuth 2.0 security requirements for the MCP server.
+
.Example output
[source,text]
----
{
"resource": "https://mcp.example.com",
"authorization_servers": [
"https://auth.provider.com/oauth2/default"
],
"scopes_supported": ["mcp:tools", "mcp:resources"],
"bearer_methods_supported": ["header"],
"resource_documentation": "https://docs.example.com/mcp-help"
}
----

.. Check that your associated `HTTPRoute` CR includes a path for your `/.well-known/oauth-protected-resource` by running the following command:
+
[source,terminal,subs="+quotes"]
----
$ oc get httproute _<route_name>_ -o jsonpath='{.spec.rules[*].matches[*].path.value}'
----
+
Replace `_<route_name>_` with the associated `HTTPRoute` CR.

.. Check the specific `AuthPolicy` CR configuration by running the following command:
+
[source,terminal,subs="+quotes"]
----
$ oc describe authpolicy _<policy_name>_ -n _<namespace>_
----

.. Check that you excluded all `/.well-known` paths from your `AuthPolicy` CR by trying to reach the endpoint without any credentials by using the following command:
+
[source,terminal,subs="+quotes"]
----
$ curl -o /dev/null -s -w "%{http_code}\n" http://_<mcp_hostname>_/_<.well-known/oauth-protected-resource>_
----
+
* Replace `_<mcp_hostname>_` with the name of the host for your MCP server.
* Replace `_<.well-known/oauth-protected-resource>_` with the reserved path and JSON file that describes the OAuth 2.0 security requirements for the MCP server.
+
[NOTE]
====
The following codes are examples of possible outputs:

* `200`: Means that the exclusion exists and matches.
* `401`: Means that your `AuthPolicy` CR is still demanding a token for this path. The exclusion is either not present or not working.
* `404`: The exclusion might be present and working, but the `HTTPRoute` CR does not point to that path to a valid backend.
====

.. Optional. Check all MCP broker component environment variables by running the following command:
+
[source,terminal,subs="+quotes"]
----
$ oc get deployment _<mcp_gateway>_ -n _<mcp_system>_ -o yaml | grep -A 10 env
----
* Replace `_<mcp_gateway>_` with the name of your {mcpg} deployment.
* Replace `_<mcp_system>_` with the namespace where your {mcpg} deployment is applied.
//Q: do we actually make Deployment or DeploymentConfig objects for MCP gateway? If yes, when?

.. Check that the `OAUTH_*` environment variables are set on your MCP broker component.
+
[source,terminal,subs="+quotes"]
----
$ oc set env deployment/_<mcp_gateway>_ --list
----
+
Replace `_<mcp_gateway>_` with the name of your {mcpg} deployment.

.. Verify that the MCP broker component pod restarted after any environment variable changes.

. If your valid tokens are being rejected with `401` errors, your JWT token validation is failing. Use the following steps to troubleshoot this situation:

.. List the `AuthPolicy` CRs by running the following command:
+
[source,terminal]
----
$ oc get authpolicy -A
----

.. Check the specific `AuthPolicy` CR configuration by running the following command:
+
[source,terminal,subs="+quotes"]
----
$ oc describe authpolicy _<policy_name>_ -n _<namespace>_
----

.. Verify that the `issuerUrl` in the `AuthPolicy` CR matches your identity provider's `realm`.

.. Check the Authorino Operator logs by running the following command:
+
[source,terminal,subs="+quotes"]
----
$ oc logs -n kuadrant-system -l authorino-resource=authorino
----

.. Decode JWT to verify claims by running the following command:
+
[source,terminal,subs="+quotes"]
----
$ echo "_<example_token>_" | cut -d. -f2 | base64 -d | jq
----
+
Replace `_<example_token>_` with your token.

.. Ensure that your issuer URL is reachable from the cluster by using the `cluster-local` service name.

.. Check the token expiration time by examining the `exp` claim.

.. Verify the audience, if required, by using an `aud` claim.

.. Ensure that the token includes all required claims such as `groups`, `email`, and so on.

. If your `401 Unauthorized` responses do not include OAuth discovery information, the `WWW-Authenticate` header is missing. This usually means that your `AuthPolicy` CR is not properly configured. Use the following steps to troubleshoot this situation:

.. Isolate the failure point by using verbose output which lists the TLS handshake, the HTTP headers, and the server response code by running the following command:
+
[source,terminal,subs="+quotes"]
----
$ curl -v http://_<mcp_hostname>_/mcp \
-H "Content-Type: application/json" \
-d '{"jsonrpc": "2.0", "id": 1, "method": "tools/list"}'
----
+
Replace `_<mcp_hostname>_` with the hostname for your backend MCP server.

.. Verify that your `AuthPolicy` CR `spec.response.unauthenticated.headers:` list includes `response.unauthenticated.headers.WWW-Authenticate`.

.. Check that your `AuthPolicy` CR `response.unauthenticated.headers.WWW-Authenticate.value` includes the correct `metadata`.

.. Ensure that your `AuthPolicy` CR is applied to the correct `Gateway` object and listener.
128 changes: 128 additions & 0 deletions modules/proc-mcp-gateway-ts-authz-issues.adoc
Original file line number Diff line number Diff line change
@@ -0,0 +1,128 @@
// Module included in the following assemblies:
//
// *mcp_gateway_config/mcp-gateway-troubleshooting.adoc

:_mod-docs-content-type: PROCEDURE
[id="proc-mcp-gateway-ts-authz-issues_{context}"]
= Troubleshooting {mcpg} authorization issues

[role="_abstract"]
Authorization errors can happen in a variety of ways, including an authenticated user getting `403` errors for all tool calls, authorization checks not enforced, or authorization failing with `CEL` evaluation errors. You can troubleshoot these problems by checking your configurations and troubleshooting `CEL`.

.Prerequisites

* You installed {mcpg}.
* You installed {prodname}.
* You installed the {oc-first}.
* You configured a `Gateway` object.
* You configured an `HTTPRoute` object for the gateway.
* You registered an MCP server.
* You created a `Secret` CR for authentication.

.Procedure

. Check your `AuthPolicy` custom resource (CR) authorization rules by running the following command:
+
[source,terminal,subs="+quotes"]
----
$ oc get authpolicy _<policy_name>_ -n _<namespace>_ -o yaml | grep -A 20 authorization
----
+
* Replace `_<policy_name>_` with the name of your `AuthPolicy` CR.
* Replace `_<namespace>_ `with the namespace where your `AuthPolicy` CR is applied.

. Check the Authorino Operator logs for CEL evaluation by running the following command:
+
[source,terminal,subs="+quotes"]
----
$ oc logs -n kuadrant-system -l authorino-resource=authorino | grep -i authz
----

. Ensure that the Authorino Operator can communicate with your identity server.

. Verify that your JWT token includes `resource_access[server-name].roles` claims.

. When your authorization checks are not enforced, first check your `AuthPolicy` CR status by running the following command:
+
[source,terminal,subs="+quotes"]
----
$ oc describe authpolicy _<policy_name>_ -n _<namespace>_
----
+
* Replace `_<policy_name>_` with the name of your `AuthPolicy` CR.
* Replace `_<namespace>_ `with the namespace where your `AuthPolicy` CR is applied.

. Next, verify your `AuthPolicy` CR targets the correct resource by running the following command:
+
[source,terminal,subs="+quotes"]
----
$ oc get authpolicy _<policy_name>_ -n _<namespace>_ -o yaml | grep -A 5 targetRef
----
+
* Replace `_<policy_name>_` with the name of your `AuthPolicy` CR.
* Replace `_<namespace>_ `with the namespace where your `AuthPolicy` CR is applied.

. Ensure that your `AuthPolicy` CR `targetRef` matches your `Gateway` object name and namespace by running the following command:
+
[source,terminal,subs="+quotes"]
----
$ echo "--- AuthPolicy Targets ---" && \
oc get authpolicy -n _<mcp_system>_ -o jsonpath='{range .items[*]}{.metadata.name}{"\t targets -> \t"}{.spec.targetRef.kind}{"/"}{.spec.targetRef.name}{"\n"}{end}' && \
echo "--- Actual Gateways ---" && \
oc get gateway -n _<mcp_system>_ -o custom-columns=NAME:.metadata.name
----
+
Replace `_<mcp_system>_` with the name of your {mcpg} deployment.

. If your `AuthPolicy` CR and your `Gateway` object are in different namespaces, you must either move the `AuthPolicy` CR into the same namespace as the Gateway object, or target the `HTTPRoute` CR instead.

. Check that your `AuthPolicy` CR `sectionName` matches your `Gateway` object's listener name.
+
[source,terminal,subs="+quotes"]
----
$ oc describe authpolicy _<auth_policy_name>_ -n _<mcp_system>_
----
+
Replace `_<auth_policy_name>_` with the name of your `AuthPolicy` CR.
Replace `_<mcp_system>_` with the name of your {mcpg} deployment.

. Examine the `Status` block for an entry about your listener. If the `sectionName` is wrong, the policy shows `"Accepted"`, but the policy does not affect the intended traffic path.

. Check that Kuadrant Operator is working by running the following command:
+
[source,terminal,subs="+quotes"]
----
$ oc get pods -n kuadrant-system
----
+
.Example output
[source,text]
----
NAME READY STATUS RESTARTS AGE
authorino-78c5679f94-abc12 1/1 Running 0 5d
dns-operator-controller-manager-5d4789f6-x1y2z 1/1 Running 0 5d
kuadrant-operator-controller-manager-8495bc4d-98765 1/1 Running 0 5d
limitador-67f89bc5d4-z9w8v 1/1 Running 0 5d
----
+
* If the `authorino-*` pod shows `CrashLoopBackOff`, it either cannot reach your OIDC issuer or has an invalid configuration.
* If the `kuadrant-operator-controller-manager-*` pod is down, any changes you make to your `AuthPolicy` CR cannot be applied to the Gateway object because the controller pod reconciles your `AuthPolicy` CR.

. Remedy pod issues as required.

. Check the Authorino Operator logs for `CEL` errors by running the following command:
+
[source,terminal,subs="+quotes"]
----
$ oc logs -n kuadrant-system -l authorino-resource=authorino | grep -i cel
----

. Verify the CEL syntax in your authorization `rules`.

. Check that referenced fields exist, such as `auth.identity.groups`.

. Ensure that the `metadata` `source` is accessible and returns the expected structure.

. Test `CEL` expression syntax using online validators.

. Add persistent logging to understand the `CEL` evaluation context.
69 changes: 69 additions & 0 deletions modules/proc-mcp-gateway-ts-ext-mcp-server-auth-issues.adoc
Original file line number Diff line number Diff line change
@@ -0,0 +1,69 @@
// Module included in the following assemblies:
//
// *mcp_gateway_config/mcp-gateway-troubleshooting.adoc

:_mod-docs-content-type: PROCEDURE
[id="proc-mcp-gateway-ts-ext-mcp-server-auth-issues_{context}"]
= Troubleshooting external MCP server authentication issues

[role="_abstract"]
If your registered external MCP server fails authentication and returns `401` or `403` errors, troubleshoot by checking the credentials and the logs.

.Prerequisites

* You installed {mcpg}.
* You installed {prodname}.
* You installed the {oc-first}.
* You configured a `Gateway` object.
* You configured an `HTTPRoute` object for the gateway.
* You registered an external MCP server.
* You created a `Secret` CR for authentication.

.Procedure

. Check that a Secret custom resource (CR) exists and that it has the correct `label` by running the following command:
+
[source,terminal,subs="+quotes"]
----
$ oc get secret _<secret_name>_ -n _<namespace>_ --show-labels
----
+
* Replace `_<secret_name>_` with the name of your `Secret` CR.
* Replace `_<namespace>_` with the namespace where your `Secret` CR is applied.

. Verify the `Secret` CR contents by running the following command:
+
[source,terminal,subs="+quotes"]
----
$ oc get secret _<secret_name>_ -n _<namespace>_ -o yaml
----
+
* Replace `_<secret_name>_` with the name of your `Secret` CR.
* Replace `_<namespace>_` with the namespace where your `Secret` CR is applied.

. Check the `MCPServerRegistration` CR `credentialRef` parameter value by running the following command:
+
[source,terminal,subs="+quotes"]
----
$ oc get mcpsr _<mcpsr_name>_ -n _<namespace>_ -o yaml | grep -A 3 credentialRef
----
+
* Replace `_<mcpsr_name>_` with the name of your `MCPServerRegistration` CR.
* Replace `_<namespace>_` with the namespace where your `MCPServerRegistration` CR is applied.

. Ensure that the `Secret` CR has the label `mcp.kuadrant.io/secret: "true"`.

. Verify that the `Secret` CR data key matches the `credentialRef.key` in the `MCPServerRegistration` CR.

. Check the credential format.

. Verify that the credential you are using has the necessary permissions for the external service.

. Check the MCP gateway broker component logs for credential errors by running the following command:
+
[source,terminal,subs="+quotes"]
----
$ oc logs -n _<mcp_system>_ deployment/mcp-gateway | grep -i auth`
----
+
Replace `_<mcp_system>_` with the name of your MCP gateway deployment.
Loading