feat: GAP authz w/ docker compose (#997)

* feat: setup-gap authz ext

* fix: dns resolution fixes

* docs: document goose chase

* fix: sanitize variables before logging

* chore: cleanup

* chore: add changesets

* fix: dns

* fix: recursive loop for localhost requests

* chore: log status code during envoy check

* fix: localhost websocket listener

* docs: include note about docker compose dns

* chore: bump alpine image version

* fix: unique name for docker network

Author: erikburt

.changeset/gold-wolves-move.md

@@ -0,0 +1,5 @@
+---
+"setup-gap": minor
+---
+
+fix DNS issue for dynamic proxy

.changeset/poor-planets-deliver.md

@@ -0,0 +1,5 @@
+---
+"setup-gap": major
+---
+
+replace docker run with docker compose

.changeset/slimy-islands-swim.md

@@ -0,0 +1,5 @@
+---
+"setup-gap": major
+---
+
+replace blocking LUA script with authz filter

actions/setup-gap/README.md

@@ -6,3 +6,87 @@
 
 The contents of ./aws-ca.crt are pulled from the PEM files on the
 [AWS Certificate Authority](https://www.amazontrust.com/repository/) page.
+
+## Configuration Gotcha
+
+This sets up a local envoy proxy which is solely for augmenting the request with
+the necessary authorization headers, before it send a request to the upstream.
+
+Here is a major issue with the dynamic proxy. Example request:
+
+1. `curl "https://service.our.domain.com"`
+2. DNS lookup on the host machine resolves this domain to `127.0.0.1`
+   1. This is because we have made a specific entry in our `/etc/hosts` to
+      ensure this is routed through the local proxy
+3. Request is routed to the local envoy proxy, for the sake of this example,
+   lets assume it uses the dynamic proxy listener
+4. The dynamic proxy checks what kind of connection it is and handles, let's
+   assume it's an http request
+5. Envoy does TLS termination, it's SAN ext allows it to use hostname
+6. Envoy augments the request before proxying it upstream, adding authentication
+   headers using whichever method (lua or ext authz)
+7. Envoy takes the 'host' header from the original request, and attempts to
+   resolve that host (or use a cached entry)
+   1. Resolving `service.our.domain.com` may resolve to `127.0.0.1` because of
+      DNS resolution caching, and because of `/etc/hosts` entry
+8. If resolved to `127.0.0.1` envoy attempts to intiate a connection with it's
+   own listener.
+   1. This fails because the TLS cert is not valid according to the envoy
+      container's CAs
+9. If resolved to the proper upstream, everything works fine.
+
+### How to diagnose this?
+
+1. Intermittent failures
+   1. If this is failing occasionally it's probably because of this. It mostly
+      all depends on how the upstream resolves, and if the good/bad entry has
+      been cached or not.
+2. TLS errors
+   1. Curls to the endpoint will result in a 503 with logs like
+   ```
+   curl: (22) The requested URL returned error: 503
+   upstream connect error or disconnect/reset before headers. retried and the latest reset reason: remote connection failure, transport failure reason: TLS_error:|268435581:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED:TLS_error_end
+   ```
+   2. Envoy logs like:
+   ```
+   [2025-04-15 21:45:11.169][15][debug][connection] [source/common/tls/cert_validator/default_validator.cc:339] verify cert failed: X509_verify_cert: certificate verification error at depth 0: unable to get local issuer certificate
+   [2025-04-15 21:45:11.169][15][debug][connection] [source/common/tls/ssl_socket.cc:248] [Tags: "ConnectionId":"2"] remote address:127.0.0.1:443,TLS_error:|268435581:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED:TLS_error_end
+   ```
+3. Infintely recursive connections
+   1. If you are debugging and set `trust_chain_verification: ACCEPT_UNTRUSTED`,
+      then the TLS errors will disappear and you will get a recursive loop of
+      proxies, until the original request times out.
+
+### Resolution
+
+#### Envoy Config
+
+Forcing DNS resolution for the upstream to use a hardcoded DNS server. **Note:**
+The cluster config and listener config need to have similar configs otherwise
+it's an invalid config.
+
+```
+dns_cache_config:
+  name: dynamic_forward_proxy_cache_config
+  dns_lookup_family: V4_ONLY
+  dns_resolution_config:
+    resolvers:
+      - socket_address:
+          address: 8.8.8.8
+          port_value: 53
+    dns_resolver_options:
+      use_tcp_for_dns_lookups: true
+  host_ttl: 60s
+```
+
+#### Docker Compose
+
+Add specific DNS servers to the compose configuration. These may not be
+necessary but we believe they ensure that the host system's DNS cache is not
+used.
+
+```
+dns:
+  - 8.8.8.8
+  - 8.8.4.4
+```

actions/setup-gap/action.yml

@@ -117,6 +117,11 @@ inputs:
     default: ""
     required: false
 
+  auth-service-port:
+    description: "The port the auth service will listen on."
+    required: false
+    default: "9001"
+
 outputs:
   local-proxy-port:
     description: "The port the local proxy will listen on."
@@ -264,8 +269,9 @@ runs:
         # Verify the installation by checking version
         gomplate --version
 
-    - name: Run local Envoy proxy
-      shell: sh
+    - name: Setup and run services
+      id: setup-services
+      shell: bash
       env:
         GAP_NAME: "gap-${{ inputs.gap-name }}"
         DYNAMIC_PROXY_PORT: ${{ inputs.dynamic-proxy-port }}
@@ -279,6 +285,15 @@ runs:
         PROXY_PORT: ${{ inputs.proxy-port }}
         WEBSOCKETS_PROXY_PORT: ${{ inputs.websockets-proxy-port }}
         WEBSOCKETS_SERVICES: ${{ inputs.websockets-services }}
+        AUTH_SERVICE_NAME: ${{ inputs.gap-name }}-authz
+        AUTH_SERVICE_PORT: ${{ inputs.auth-service-port }}
+        PATH_CERTS_DIR: ${{ env.PATH_CERTS_DIR }}
+        REQUIRED_ENV_VARS: >-
+          WEBSOCKETS_PROXY_PORT DYNAMIC_PROXY_PORT PROXY_PORT
+          K8S_API_ENDPOINT_PORT MAIN_DNS_ZONE ENVOY_PROXY_IMAGE
+          ENABLE_PROXY_DEBUG PROXY_LOG_LEVEL AUTH_SERVICE_NAME AUTH_SERVICE_PORT
+          ACTIONS_ID_TOKEN_REQUEST_TOKEN ACTIONS_ID_TOKEN_REQUEST_URL
+          GITHUB_REPOSITORY GITHUB_OIDC_TOKEN_HEADER_NAME GITHUB_OIDC_HOSTNAME
       run: |
         # Get the Github OIDC hostname
         export GITHUB_OIDC_HOSTNAME=$(echo $ACTIONS_ID_TOKEN_REQUEST_URL | awk -F[/:] '{print $4}')
@@ -294,11 +309,8 @@ runs:
 
         echo "Using log level: ${PROXY_LOG_LEVEL}"
 
-        # List of required ENVs to check
-        required_env_vars="DYNAMIC_PROXY_PORT ENABLE_PROXY_DEBUG GITHUB_OIDC_TOKEN_HEADER_NAME ENVOY_PROXY_IMAGE GITHUB_OIDC_HOSTNAME K8S_API_ENDPOINT_PORT MAIN_DNS_ZONE PROXY_LOG_LEVEL PROXY_PORT GITHUB_REPOSITORY WEBSOCKETS_PROXY_PORT"
-
         # Loop through each variable and check if it's empty
-        for var in $required_env_vars; do
+        for var in $REQUIRED_ENV_VARS; do
           eval value=\$$var
           if [ -z "$value" ]; then
             echo "::error::Required environment variable '$var' is not set."
@@ -309,45 +321,30 @@ runs:
         # Generate Envoy config from template
         gomplate -f "${GITHUB_ACTION_PATH}/envoy.yaml.gotmpl" -o "${GITHUB_ACTION_PATH}/envoy.yaml"
 
-        # Generate LUA script from template
-        gomplate -f "${GITHUB_ACTION_PATH}/github_oidc.lua.gotmpl" -o "${GITHUB_ACTION_PATH}/github_oidc.lua"
-
         # Copy AWS CA certs
         cp ${{ github.action_path }}/aws-ca.crt "${PATH_CERTS_DIR}"
         chmod 644 "${PATH_CERTS_DIR}/server.key"
 
         echo "Validating Envoy config..."
         if ! docker run --rm \
-            --name "${GAP_NAME}" \
-            -p "${DYNAMIC_PROXY_PORT}:${DYNAMIC_PROXY_PORT}" \
-            -p "${PROXY_PORT}:${PROXY_PORT}" \
-            -p "${WEBSOCKETS_PROXY_PORT}:${WEBSOCKETS_PROXY_PORT}" \
-            -v "${PATH_CERTS_DIR}":/tls \
-            -v "${GITHUB_ACTION_PATH}/envoy.yaml":/etc/envoy/envoy.yaml \
-            -v "${GITHUB_ACTION_PATH}/json.lua":/etc/envoy/json.lua \
-            -v "${GITHUB_ACTION_PATH}/github_oidc.lua":/etc/envoy/github_oidc.lua \
+            --dns 8.8.8.8
+            --dns 8.8.4.4
+            --volume "${PATH_CERTS_DIR}":/tls \
+            --volume "${GITHUB_ACTION_PATH}/envoy.yaml":/etc/envoy/envoy.yaml \
             "${ENVOY_PROXY_IMAGE}" \
             /usr/local/bin/envoy --mode validate -c /etc/envoy/envoy.yaml \
-            --log-level "${PROXY_LOG_LEVEL}" \
-            "${ENVOY_EXTRA_ARGS}"; then
+            --log-level "${PROXY_LOG_LEVEL}" "${ENVOY_EXTRA_ARGS}"; then
           echo "::error::Envoy configuration validation failed."
           exit 1
         fi
 
-        echo "Starting the local Envoy proxy..."
-        docker run --rm -d \
-          --name "${GAP_NAME}" \
-          -p "${DYNAMIC_PROXY_PORT}:${DYNAMIC_PROXY_PORT}" \
-          -p "${PROXY_PORT}:${PROXY_PORT}" \
-          -p "${WEBSOCKETS_PROXY_PORT}:${WEBSOCKETS_PROXY_PORT}" \
-          -v "${PATH_CERTS_DIR}":/tls \
-          -v "${GITHUB_ACTION_PATH}/envoy.yaml":/etc/envoy/envoy.yaml \
-          -v "${GITHUB_ACTION_PATH}/json.lua":/etc/envoy/json.lua \
-          -v "${GITHUB_ACTION_PATH}/github_oidc.lua":/etc/envoy/github_oidc.lua \
-          "${ENVOY_PROXY_IMAGE}" \
-          /usr/local/bin/envoy -c /etc/envoy/envoy.yaml \
-          --log-level "${PROXY_LOG_LEVEL}" \
-          "${ENVOY_EXTRA_ARGS}"
+        if [ "$ENABLE_PROXY_DEBUG" = "true" ] || [ "$PROXY_LOG_LEVEL" = "debug" ]; then
+          echo "Docker compose configuration:"
+          docker compose -f "${GITHUB_ACTION_PATH}/docker-compose.yml" config
+        fi
+
+        echo "Starting the auth service and Envoy proxy..."
+        docker compose -f "${GITHUB_ACTION_PATH}/docker-compose.yml" up -d
 
     - name: Verify Envoy Proxy
       shell: bash
@@ -361,11 +358,24 @@ runs:
           local port=$1
           local name=$2
 
-          echo "Checking if the ${name} proxy is up and running on [https://localhost:${port}] ..."
+          echo "Checking if the ${name} proxy is up and running on [https://localhost:${port}]"
           for attempt in {1..10}; do
-            if curl --silent --cacert "${PATH_CERTS_DIR}/ca.crt" --head https://localhost:${port} > /dev/null 2>&1; then
-              echo "${name} proxy is up, and the HTTPS connection is successful."
+            # Get the status code using curl
+            status_code=$(curl --silent --cacert "${PATH_CERTS_DIR}/ca.crt" \
+                          -o /dev/null -w "%{http_code}" https://localhost:${port})
+
+            echo "Attempt ${attempt}/10: Status code: ${status_code}"
+
+            if [[ $status_code -eq 200 ]]; then
+              echo "${name} proxy is up, and the HTTPS connection is successful with status code 200."
               return 0
+            elif [[ $status_code -ne 000 ]]; then
+              echo "WARNING: ${name} proxy returned non-200 status code: ${status_code}"
+              if [[ $attempt -eq 10 ]]; then
+                echo "::error:: ${name} proxy is responding but with unexpected status code: ${status_code}"
+                return 1
+              fi
+              sleep 3
             else
               echo "Waiting for the ${name} proxy to start... Attempt ${attempt}/10"
               sleep 3

actions/setup-gap/authz/Dockerfile

@@ -0,0 +1,17 @@
+FROM golang:1.24-alpine AS builder
+
+WORKDIR /app
+COPY main.go .
+RUN go mod init github.com/auth-service && \
+    go build -o auth-service .
+
+FROM alpine:3.21
+
+RUN apk --no-cache add ca-certificates
+WORKDIR /app
+COPY --from=builder /app/auth-service .
+
+ENV AUTH_SERVICE_PORT=9001
+EXPOSE ${AUTH_SERVICE_PORT}
+
+CMD ["/app/auth-service"]