traefik is sending https to am service due to naming of service port

[pearj]

I'm using 2026.2.1 and out of the box, am doesn't work. After investigations, I noticed that am had some strange errors which indicated it was receiving https request initiation for unencrypted http

java.lang.IllegalArgumentException: Invalid character found in method name [0x160x030x010x050xba0x010x000x050xb60x030x030xafF0xab0xa4P0x9b/^0x11>O0x7f0x9b`0xfcT"k0xa02T0xb80xcd.0xe40x860xb80x17)O0x0c0xe3 ]. HTTP method names must be tokens

Then in the traefik logs I noticed:

ERR 500 Internal Server Error error="tls: first record does not look like a TLS handshake"

I tried using this annotation on the am ingress:

nginx.ingress.kubernetes.io/backend-protocol: HTTP

As it is apparently supported: https://doc.traefik.io/traefik/reference/routing-configuration/kubernetes/ingress-nginx/#opt-nginx-ingress-kubernetes-iobackend-protocol

However that didn't work, the only thing that worked was changing the am service port name from https to something else, e.g. http, bob, etc.

forgeops/kustomize/base/am/secret-agent/am-service.yaml

Lines 19 to 22 in 1529b4e

	- port: 80
	targetPort: https
	protocol: TCP
	name: https

The target port can still be called https, but the port name can't.

I managed to find this doco on the traefik website: https://doc.traefik.io/traefik/reference/routing-configuration/kubernetes/ingress-nginx/#opt-nginx-ingress-kubernetes-iobackend-protocol

There are 3 ways to configure Traefik to use HTTPS to communicate with pods:

1. If the service port defined in the ingress spec is 443 (note that you can still use targetPort to use a different port on your pod).
2. If the service port defined in the ingress spec has a name that starts with https (such as https-api, https-web or just https).
3. If the service spec includes the annotation traefik.ingress.kubernetes.io/service.serversscheme: https.

Adding traefik.ingress.kubernetes.io/service.serversscheme: http annotation, to the am service also fixes it.

[lee-baines]

Hi @pearj. Which environment are you running this in? Cloud, minikube? I would like to try and reproduce this but we have tests running in Google Cloud and minikube and the platform works ok. Also what platform endpoint are you navigating to? Do you get this error for any call to AM?

[pearj]

Hi @lee-baines, I'm running in microk8s, which is a bit like minikube, but I figured it's quite similar to minikube. I'm using the OOTB Traefik ingress addon.

I checked the Traefik namespace and it is running docker.io/traefik:v3.6.2, which is a little old it looks like Nov 2025: https://github.com/traefik/traefik/releases/tag/v3.6.2

These are the parameters traefik starts up with:

--entryPoints.metrics.address=:9100/tcp                                       
--entryPoints.traefik.address=:8080/tcp                                       
--entryPoints.web.address=:8000/tcp                                           
--entryPoints.websecure.address=:8443/tcp                                     
--api.dashboard=true                                                          
--ping=true                                                                   
--metrics.prometheus=true                                                     
--metrics.prometheus.entrypoint=metrics                                       
--providers.kubernetescrd                                                     
--providers.kubernetescrd.allowEmptyServices=true                             
--providers.kubernetesingress                                                 
--providers.kubernetesingress.allowEmptyServices=true                         
--providers.kubernetesingress.ingressendpoint.publishedservice=ingress/traefik
--providers.kubernetesgateway                                                 
--providers.kubernetesgateway.statusaddress.service.name=traefik              
--providers.kubernetesgateway.statusaddress.service.namespace=ingress         
--providers.kubernetesingressnginx                                            
--providers.kubernetesingressnginx.controllerclass=k8s.io/ingress-nginx       
--providers.kubernetesingressnginx.ingressclass=nginx                         
--entryPoints.websecure.http.tls=true                                         
--log.level=INFO                                                              

Also, I realised that I didn't post the correct URL from Traefik Ingress that describes how it chooses a HTTPS backend: https://doc.traefik.io/traefik/reference/routing-configuration/kubernetes/ingress/#communication-between-traefik-and-pods

I wonder why it works for you that's weird, the doco seems fairly clear that it'll choose https on the backend. I presume you're not actually using https inside the AM tomcat container?

And yes, I get that error for any AM call. IDM is fine though, as are the UI containers.

[lee-baines]

AM isn't actually configured with a proper ssl listener. Because ssl traffic is terminated at the ingress controller, all traffic to the pods are unencrypted. So I think you're right, the name shouldn't be https. I'll create a ticket to look into it further

[pearj]

It’s terminating to the “secure” 8081 port on tomcat, which tells tomcat that the port is https but at a reverse proxy. I think that’s what it means. That’s probably why it has been named https.

[lee-baines]

Yes thats correct. AM exposes a fake SSL port currently to allow HTTPS connections.

[damonconway]

Hi @pearj,
I have implemented this fix that will be in 2026.3.0 coming this month.

Thanks,
Damon