Skip to content

Commit daa59d1

Browse files
ShaunaDiazShaunaDiaz
committed
OSDOCS-18945: adds troubleshooting MCP gateway
1 parent f19c445 commit daa59d1

8 files changed

Lines changed: 411 additions & 1 deletion

‎modules/con-mcp-gateway-ts-gateway-routing.adoc‎

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
1+
// Module included in the following assemblies:
2+
//
3+
// *mcp_gateway_config/mcp-gateway-troubleshooting.adoc
4+
5+
:_mod-docs-content-type: CONCEPT
6+
[id="con-mcp-gateway-ts-gateway-routing_{context}"]
7+
= Gateway and routing troubleshooting
8+
9+
[role="_abstract"]
10+
When traffic is not flowing after you installed {mcpg}, you can investigate each component of the gateway routing to check system health. Depending on the errors you are receiving, you can troubleshoot at several layers. Breaks can occur at the gateway, route, or policy levels.
11+
12+
If you have a `Connection Refused/Timeout` error and your client cannot reach the IP address, the cause might be the
13+
listener. In this case, one of the following situations likely applies:
14+
15+
* The port is not open.
16+
* The load balancer has not assigned an IP address.
17+
* The TLS handshake is failing.
18+
19+
When you have this type of error, check the listener first.
20+
21+
If can connect to the `Gateway `object, but you get an `HTTP` error, such as `404`, the cause can be a problem with the `HTTPRoute` custom resource (CR). The route exists, but the `Gateway` object has rejected it or the connection has failed. When you get these types of codes, check the `HTTPRoute` CR first.
22+
23+
If requests either fail with a `503` error or bypass the router, this means that the route is recognized, but the connection to the backend failed or was not authorized properly. In this case, start with API-level checks and narrow your investigation to Envoy filters as needed.
24+
25+
If the `EnvoyFilter` is not present, it usually means one of the following situations has occurred:
26+
27+
* The `Gateway` CR status is not `Programmed`.
28+
* There is a `labels` mismatch, and the MCP controller logic does not send any pods through the filter.
29+
* The MCP controller is crashing or stuck.

‎modules/proc-mcp-gateway-register-mcp-server.adoc‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -93,6 +93,11 @@ spec:
9393
* Replace the `spec.targetRef.namespace:` field value with the namespace where your `HTTPRoute` CR is applied. In this example, `_<mcp_test>_` is used.
9494
* Replace the `credentialRef.name:` field value with the name of your `Secret` CR. In this example, `_<mcp_server_one_secret>_` is used. You can omit this parameter if your MCP server does not require authentication or authorization.
9595
* For more information about these parameters, see "Understanding the `MCPServerRegistration` custom resource."
96+
+
97+
[IMPORTANT]
98+
====
99+
A `toolPrefix` value cannot include spaces or special characters.
100+
====
96101
97102
. Apply the CR by running the following command:
98103
+

‎modules/proc-mcp-gateway-ts-gateway-listener-not-working.adoc‎

Lines changed: 84 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,84 @@
1+
2+
// Module included in the following assemblies:
3+
//
4+
// *mcp_gateway_config/mcp-gateway-troubleshooting.adoc
5+
6+
:_mod-docs-content-type: PROCEDURE
7+
[id="proc-mcp-gateway-ts-gateway-listener-not-working_{context}"]
8+
= Troubleshooting the gateway listener not working
9+
10+
[role="_abstract"]
11+
If your {mcpg} cannot reach an MCP endpoint at configured hostname, the cause might be that the `Listener` custom resource (CR) you configured is not working. You can troubleshoot this situation by using a few commands and some insight.
12+
13+
Use the following concepts in conjunction with the commands that follow to solve a non-functioning `Listener` CR:
14+
15+
* Ensure that your `Gateway` object has `Accepted` and `Programmed` conditions set to `True`.
16+
* Verify that the `hostname` in the `Listener `CR matches your DNS or hosts configuration.
17+
18+
.Prerequisites
19+
20+
* You installed {mcpg}.
21+
* You installed {prodname}.
22+
* You configured a `Gateway` object.
23+
* You configured an `HTTPRoute` object for the gateway.
24+
* You installed the {oc-first}.
25+
* You registered an MCP server.
26+
27+
.Procedure
28+
29+
. Check the general `Gateway` object configuration by running the following command:
30+
+
31+
[source,terminal]
32+
----
33+
$ oc get gateway -A
34+
----
35+
+
36+
This command returns general information about all `Gateway` objects in the cluster. If the `Gateway` object you are troubleshooting does exist, the command returns the `gatewayClassName` is it using, whether or not it has an IP address or hostname assigned, and a `status`, such as `Ready`, `Programmed`, or `Pending`.
37+
38+
. Check the full metadata and status history for one specific `Gateway` object by running the following command:
39+
+
40+
[source,terminal,subs="+quotes"]
41+
----
42+
$ oc describe gateway _<gateway_name>_ -n _<namespace>_
43+
----
44+
+
45+
* Replace `_<gateway_name>_` with the name of the `Gateway` object.
46+
* Replace `_<namespace>_` with the namespace where the `Gateway` object is applied.
47+
* This command can help you figure out why a `Gateway` object is stuck in `Pending` by checking for port conflicts and verifying that `SSL/TLS` certificates are correctly attached to `Listener` CRs.
48+
49+
. Verify the `Listener` CR configuration by running the following command:
50+
+
51+
[source,terminal,subs="+quotes"]
52+
----
53+
$ oc get gateway _<gateway_name>_ -n _<namespace>_ -o yaml | grep -A 10 listeners
54+
----
55+
+
56+
* Replace `_<gateway_name>_` with the name of the `Gateway` object.
57+
* Replace `_<namespace>_` with the namespace where the `Gateway object` is applied.
58+
59+
. Check all of your `Listener` CR configurations at the same time by running the following command:
60+
+
61+
[source,terminal,subs="+quotes"]
62+
----
63+
$ oc get gateway _<gateway_name>_ -n _<namespace>_ -o jsonpath='{range .spec.listeners[*]}{.name}{"\t"}{.hostname}{"\t"}{.port}{"\n"}{end}' | jq
64+
----
65+
* Replace `_<gateway_name>_` with the name of the `Gateway` object.
66+
* Replace `_<namespace>_` with the namespace where your `Gateway` object is applied.
67+
* This command uses `jq` for formatting.
68+
69+
. Check that the Istio gateway pod is running by using the following command:
70+
+
71+
[source,terminal,subs="+quotes"]
72+
----
73+
$ oc get pods -n _<gateway_system>_ -l istio=ingressgateway
74+
----
75+
+
76+
* Replace `_<gateway_system>_` with the name of your `Gateway` object deployment.
77+
* This command checks the status of Envoy-proxy pods and returns pod, traffic flow, and policy errors.
78+
79+
. Verify that the port you are trying to use is not already in use by running the following command:
80+
+
81+
[source,terminal]
82+
----
83+
$ oc get gateway -A -o yaml | grep "port:"
84+
----

‎modules/proc-mcp-gateway-ts-pods-not-starting.adoc‎

Lines changed: 57 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,57 @@
1+
// Module included in the following assemblies:
2+
//
3+
// *mcp_gateway_config/mcp-gateway-troubleshooting.adoc
4+
5+
:_mod-docs-content-type: PROCEDURE
6+
[id="proc-mcp-gateway-ts-pods-not-starting_{context}"]
7+
= {mcpg} pods not starting
8+
9+
[role="_abstract"]
10+
After installation, if your {mcpg} pods are stuck in one of several states that indicate that they are not starting as expected, you can take several steps to diagnose the problem.
11+
12+
Common causes include the following states and indicate an associated action:
13+
14+
* `ImagePullBackOff`: Check image repository access and credentials.
15+
* `CrashLoopBackOff`: Check the logs for application errors.
16+
* `Pending:` Check resource availability and node capacity.
17+
* `Init Container Failure*:` Check RBAC permissions.
18+
19+
.Prerequisites
20+
21+
* You installed {mcpg}.
22+
* You installed {prodname}.
23+
* You configured a `Gateway` object.
24+
* You configured an `HTTPRoute` object for the gateway.
25+
* You installed the {oc-first}.
26+
* You registered an MCP server.
27+
28+
.Procedure
29+
30+
. Check the pod status by running the following command:
31+
+
32+
[source,terminal,subs="+quotes"]
33+
----
34+
$ oc get pods -n _<mcp_system>_
35+
----
36+
+
37+
Replace `_<mcp_system>_` with the name of the {mcpg} deployment that you are checking.
38+
39+
. Describe problem pods by running the following command:
40+
+
41+
[source,terminal,subs="+quotes"]
42+
----
43+
$ oc describe pod -n _<mcp_system>_ _<pod_name>_
44+
----
45+
+
46+
* Replace `_<mcp_system>_` with the name of the {mcpg} deployment that you are checking.
47+
* Replace `_<pod_name>_` with the name of the pod that you are checking.
48+
49+
. Check the namespace logs by running the following command:
50+
+
51+
[source,terminal,subs="+quotes"]
52+
----
53+
$ oc logs -n _<mcp_system>_ _<pod_name>_
54+
----
55+
+
56+
* Replace `_<mcp_system>_` with the name of the {mcpg} deployment that you are checking.
57+
* Replace `_<pod_name>_` with the name of the pod that you are checking.

‎modules/proc-mcp-gateway-ts-requests-fail-or-bypass-router.adoc‎

Lines changed: 152 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,152 @@
1+
// Module included in the following assemblies:
2+
//
3+
// *mcp_gateway_config/mcp-gateway-troubleshooting.adoc
4+
5+
:_mod-docs-content-type: PROCEDURE
6+
[id="proc-mcp-gateway-ts-requests-fail-or-bypass-router_{context}"]
7+
= Troubleshooting requests failing or bypassing the router
8+
9+
[role="_abstract"]
10+
When you are certain that an `MCPGatewayExtension` custom resource (CR) exists for your MCP server, but requests either fail or bypass the router, it might mean that the `EnvoyFilter` CR is not applied properly. You can take several steps to troubleshoot the problem.
11+
12+
The `EnvoyFilter` CR is automatically created in the `Gateway` CR's namespace by the MCP gateway controller component when an `MCPGatewayExtension` CR is `Ready`.
13+
14+
[IMPORTANT]
15+
====
16+
You can look closely at the `EnvoyFilter` during deep troubleshooting, but do not manually edit or delete the CR.
17+
====
18+
19+
.Prerequisites
20+
21+
* You installed {mcpg}.
22+
* You installed {prodname}.
23+
* You configured a `Gateway` object.
24+
* You configured an `HTTPRoute` object for the gateway.
25+
* You installed the {oc-first}.
26+
* You registered an MCP server.
27+
28+
.Procedure
29+
30+
. Ensure that the `Gateway` object exists and is in the expected namespace by checking the general `Gateway` object configuration by using the following command:
31+
+
32+
[source,terminal]
33+
----
34+
$ oc get gateway -A
35+
----
36+
+
37+
This command returns general information about all `Gateway` objects in the cluster. If the `Gateway` object you are troubleshooting does exist, the command returns the `gatewayClassName` is it using, whether or not it has an IP address or hostname assigned, and a `status`, such as `Ready`, `Programmed`, or `Pending`.
38+
39+
. Verify that the `MCPGatewayExtension` CR is `Ready` by running the following command:
40+
+
41+
[source,terminal]
42+
----
43+
$ oc get mcpgatewayextension -A`
44+
----
45+
46+
. Verify that a `ReferenceGrant` CR exists if the `MCPGatewayExtension` CR is in a different namespace than the `Gateway` object by running the following command:
47+
+
48+
[source,terminal,subs="+quotes"]
49+
----
50+
$ oc get referencegrant _<referencegrant_name>_ -n _<referencegrant_namespace>_ -o yaml
51+
----
52+
+
53+
* Replace `_<referencegrant_namespace>_` with the namespace where the `ReferenceGrant` CR is applied.
54+
* Replace `_<referencegrant_name>_` with the names of the `ReferenceGrant` CR.
55+
56+
. Check the `HTTPRoute` CR by running the following command:
57+
+
58+
[source,terminal,subs="+quotes"]
59+
----
60+
$ oc describe httproute _<httproute_name>_ -n _<httproute_namespace>_
61+
----
62+
+
63+
* Replace `_<httproute_namespace>_` with the namespace where the `HTTPRoute` CR is applied.
64+
* Replace `_<httproute_name>_` with the names of the `HTTPRoute` CR.
65+
* If the `HTTPRoute` CR is not `Accepted` by the `Gateway` object, the route is not programmed into Envoy, causing a `404`.
66+
* The conditions `Status.Parents.Conditions: Accepted: True` and `Programmed: True` show that the route is correct.
67+
68+
. If any of the CRs you just checked are not `Ready`, check the controller logs for `EnvoyFilter` creation errors by running the following command:
69+
+
70+
[source,terminal,subs="+quotes"]
71+
----
72+
$ oc logs -n _<mcp_system>_ deployment/mcp-gateway-controller
73+
----
74+
+
75+
* Replace `_<mcp_system>_` with the name of your MCP gateway deployment.
76+
* This step verifies that the MCP controller is successfully generating the underlying Istio configurations.
77+
78+
. Check that the `EnvoyFilter` exists in the `Gateway` namespace by running the following command:
79+
+
80+
[source,terminal,subs="+quotes"]
81+
----
82+
$ oc get envoyfilter -n _<gateway_namespace>_ -l app.kubernetes.io/managed-by=mcp-gateway-controller
83+
----
84+
+
85+
* Replace `_<gateway_namespace>_` with the namespace where the `Gateway` object is applied.
86+
87+
. Verify the `EnvoyFilter` configuration by running the following command:
88+
+
89+
[source,terminal,subs="+quotes"]
90+
----
91+
$ oc describe envoyfilter -n _<gateway_namespace>_ -l app.kubernetes.io/managed-by=mcp-gateway-controller
92+
----
93+
+
94+
* Replace `_<gateway_namespace>_` with the namespace where the `Gateway` object is applied.
95+
* The `workloadSelector` labels must match your `Gateway` pods, or your policies are bypassed.
96+
97+
. Compare the `EnvoyFilter` labels against your pod labels by running the following command:
98+
+
99+
[source,terminal,subs="+quotes"]
100+
----
101+
$ oc get pods -n _<gateway_namespace>_ --show-labels
102+
----
103+
+
104+
* Replace `_<gateway_namespace>_` with the namespace where the `Gateway` object is applied.
105+
106+
. Identify the port that the `Gateway` object is configured to use by running the following command:
107+
+
108+
[source,terminal,subs="+quotes"]
109+
----
110+
$ oc get gateway <gateway_name> -n _<gateway_namespace>_ -o jsonpath='{range .spec.listeners[*]}{.name}{": "}{.port}{"\n"}{end}'
111+
----
112+
+
113+
* Replace `_<gateway_name>_` with the name of the `Gateway` object.
114+
* Replace `_<gateway_namespace>_` with the namespace where the `Gateway` object is applied.
115+
116+
. Confirm that Envoy opened the `Gateway`-specified port by running the following command:
117+
+
118+
[source,terminal,subs="+quotes"]
119+
----
120+
$ oc exec -n <gateway_namespace> deploy/<gateway_name>-istio -- curl -s localhost:15000/listeners | grep -E ":[0-9]+"
121+
----
122+
123+
. Verify the `EnvoyFilter` chain binding by running the following command:
124+
+
125+
[source,terminal,subs="+quotes"]
126+
----
127+
$ oc exec -n _<gateway_namespace>_ deploy/_<gateway_name>_-istio -- curl -s localhost:15000/config_dump | jq '.configs[] | select(.["@type"] | contains("ListenersConfigDump")) | .dynamic_listeners[] | select(.name | contains(":<PORT_FROM_STEP_6>"))'
128+
----
129+
+
130+
* Replace `_<gateway_name>_` with the name of the `Gateway` object.
131+
* Replace `_<gateway_namespace>_` with the namespace where the `Gateway` object is applied.
132+
* Check for `envoy.filters.http.ext_authz` for Authorino or `envoy.filters.http.ratelimit` for Limitador within the specific port config.
133+
134+
. Check the Istio gateway pod configuration by running the following command:
135+
+
136+
[source,terminal,subs="+quotes"]
137+
----
138+
$ oc exec -n _<gateway_namespace>_ deploy/_<gateway_name>_-istio -- curl localhost:15000/config_dump | grep ext_proc
139+
----
140+
+
141+
* Replace `_<gateway_namespace>_` with the namespace where the `Gateway` object is applied.
142+
* Replace `_<gateway_name>_` with the names of the `Gateway` object.
143+
* If this command returns empty, the `EnvoyFilter` CR is not active on this pod. Traffic is bypassing your policies.
144+
145+
. Restart the Istio gateway to force a configuration reload by running the following command:
146+
+
147+
[source,terminal,subs="+quotes"]
148+
----
149+
$ oc rollout restart deployment/_<gateway_name>_-istio -n _<gateway_namespace>_`
150+
----
151+
* Replace `_<gateway_namespace>_` with the namespace where the `Gateway` object is applied.
152+
* Replace `_<gateway_name>_` with the names of the `Gateway` object.

0 commit comments

Comments
 (0)