Fix jwt unit tests and add metrics integration test

.github/copilot-instructions.md

@@ -0,0 +1,96 @@
+# Copilot Coding Agent Instructions
+
+This document provides instructions for the GitHub Copilot coding agent when working on this repository.
+
+## Project Structure
+
+- `gateway/` - Gateway components (controller, policy-engine, router)
+  - `gateway/gateway-controller/` - Gateway controller service
+  - `gateway/policy-engine/` - Policy engine service
+  - `gateway/router/` - Envoy-based router
+  - `gateway/it/` - Integration tests
+- `kubernetes/` - Kubernetes operator and Helm charts
+  - `kubernetes/gateway-operator/` - Gateway Kubernetes operator
+
+## Running Integration Tests
+
+### When modifying code in `./gateway/**` (excluding `./gateway/it/`)
+
+If you change any code in the gateway components (gateway-controller, policy-engine, router, etc.), you MUST:
+
+1. **Rebuild the Docker images** to include your changes:
+   ```bash
+   cd gateway && make build-local
+   ```
+
+2. **Modify the test configuration** to use the non-coverage images. Edit `gateway/it/docker-compose.test.yaml` line 25 and remove the `-coverage` suffix from the gateway-controller image name:
+   - Change: `ghcr.io/wso2/api-platform/gateway-controller-coverage:<version>`
+   - To: `ghcr.io/wso2/api-platform/gateway-controller:<version>`
+
+   **Important:** Keep the existing version tag (e.g., `0.3.0-SNAPSHOT`) as-is. The version in the file is the current version - do not hardcode a different version.
+
+3. **Run the integration tests**:
+   ```bash
+   cd gateway/it && make test
+   ```
+
+4. **Revert the gateway-controller image name before committing**. The `-coverage` suffix removal was only for local testing. Before creating a PR, restore the `-coverage` suffix of `gateway/it/docker-compose.test.yaml`:
+   - Change: `ghcr.io/wso2/api-platform/gateway-controller:<version>`
+   - Back to: `ghcr.io/wso2/api-platform/gateway-controller-coverage:<version>`
+
+   **Important:** Do NOT include this image name change in your PR. The CI pipeline requires the coverage image. Other legitimate changes to docker-compose.test.yaml should still be committed.
+
+### When modifying only `./gateway/it/**` (integration test files only)
+
+If you are only modifying integration test files (test cases, test utilities, etc.) and not the gateway component source code:
+
+1. **Modify the test configuration** to use the non-coverage images. Edit `gateway/it/docker-compose.test.yaml` line 25 and remove the `-coverage` suffix from the gateway-controller image name:
+   - Change: `ghcr.io/wso2/api-platform/gateway-controller-coverage:<version>`
+   - To: `ghcr.io/wso2/api-platform/gateway-controller:<version>`
+
+   **Important:** Keep the existing version tag as-is. Do not change the version number.
+
+2. **Run the integration tests**:
+   ```bash
+   cd gateway/it && make test
+   ```
+
+3. **Revert the gateway-controller image name before committing**. Restore the `-coverage` suffix  of `gateway/it/docker-compose.test.yaml`:
+   - Change: `ghcr.io/wso2/api-platform/gateway-controller:<version>`
+   - Back to: `ghcr.io/wso2/api-platform/gateway-controller-coverage:<version>`
+
+   **Important:** Do NOT include this image name change in your PR. Other legitimate changes to docker-compose.test.yaml should still be committed.
+
+Note: The images are pre-built during the setup phase, so you can run tests directly without rebuilding.
+
+## Important Notes
+
+- The `copilot-setup-steps.yml` workflow pre-builds gateway images using `make build-local`
+- The docker-compose file by default references coverage images (for CI coverage reporting)
+- You must switch to non-coverage images when running tests in the Copilot environment
+- **Version tags:** Always use whatever version tag is currently in the docker-compose.test.yaml file. The version may change over time (e.g., `0.3.0-SNAPSHOT`, `0.4.0-SNAPSHOT`, etc.) - never hardcode a specific version
+
+## Build Commands Reference
+
+| Command | Description | Working Directory |
+|---------|-------------|-------------------|
+| `make build-local` | Build all gateway Docker images locally | `gateway/` |
+| `make test` | Run integration tests | `gateway/it/` |
+
+## Unit Tests
+
+To run unit tests for individual components:
+
+```bash
+# Gateway Controller
+cd gateway/gateway-controller && go test ./...
+
+# Policy Engine
+cd gateway/policy-engine && go test ./...
+
+# Router
+cd gateway/router && go test ./...
+
+# Gateway Operator
+cd kubernetes/gateway-operator && go test ./...
+```

.github/workflows/copilot-setup-steps.yml

@@ -0,0 +1,68 @@
+name: "Copilot Setup Steps"
+
+# Automatically run the setup steps when they are changed to allow for easy validation, and
+# allow manual testing through the repository's "Actions" tab
+on:
+  workflow_dispatch:
+  push:
+    paths:
+      - .github/workflows/copilot-setup-steps.yml
+  pull_request:
+    paths:
+      - .github/workflows/copilot-setup-steps.yml
+
+jobs:
+  # The job MUST be called `copilot-setup-steps` or it will not be picked up by Copilot.
+  copilot-setup-steps:
+    runs-on: ubuntu-24.04
+
+    # Set the permissions to the lowest permissions possible needed for your steps.
+    # Copilot will be given its own token for its operations.
+    permissions:
+      contents: read
+
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v4
+
+      - name: Set up Go
+        uses: actions/setup-go@v5
+        with:
+          go-version: '1.25'
+          cache: true
+          cache-dependency-path: |
+            gateway/**/go.sum
+            kubernetes/**/go.sum
+
+      - name: Set up Docker Buildx
+        uses: docker/setup-buildx-action@v3
+
+      - name: Install Helm
+        uses: azure/setup-helm@v4
+        with:
+          version: 'latest'
+
+      - name: Install kubectl
+        uses: azure/setup-kubectl@v4
+        with:
+          version: 'latest'
+
+      - name: Download Go dependencies for gateway-controller
+        run: go mod download
+        working-directory: gateway/gateway-controller
+
+      - name: Download Go dependencies for policy-engine
+        run: go mod download
+        working-directory: gateway/policy-engine
+
+      - name: Download Go dependencies for router
+        run: go mod download
+        working-directory: gateway/router
+
+      - name: Download Go dependencies for gateway-operator
+        run: go mod download
+        working-directory: kubernetes/gateway-operator
+
+      - name: Build Gateway images
+        run: make build-local
+        working-directory: gateway

gateway/it/docker-compose.test.yaml

@@ -30,6 +30,7 @@ services:
       - "9010:9090"   # REST API
       - "18000:18000" # xDS gRPC
       - "18001:18001"
+      - "9091:9091"   # Metrics
     environment:
       - GATEWAY_STORAGE_TYPE=sqlite
       - GATEWAY_STORAGE_SQLITE_PATH=./data/gateway.db
@@ -59,6 +60,7 @@ services:
     command: ["-xds-server", "it-gateway-controller:18001"]
     ports:
       - "9002:9002"   # Admin API
+      - "9003:9003"   # Metrics
     volumes:
       - ./test-config.yaml:/app/configs/config.yaml:ro
     depends_on:

gateway/it/features/metrics.feature

@@ -0,0 +1,67 @@
+# --------------------------------------------------------------------
+# Copyright (c) 2025, WSO2 LLC. (https://www.wso2.com).
+#
+# WSO2 LLC. licenses this file to you under the Apache License,
+# Version 2.0 (the "License"); you may not use this file except
+# in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+# --------------------------------------------------------------------
+
+Feature: Gateway Metrics
+  As an operator
+  I want to verify that gateway components expose Prometheus metrics
+  So that I can monitor the health and performance of the gateway
+
+  Background:
+    Given the gateway services are running
+
+  Scenario: Gateway controller metrics endpoint is accessible
+    When I send a GET request to the gateway controller metrics endpoint
+    Then the response status code should be 200
+    And the response should contain Prometheus metrics
+
+  Scenario: Policy engine metrics endpoint is accessible
+    When I send a GET request to the policy engine metrics endpoint
+    Then the response status code should be 200
+    And the response should contain Prometheus metrics
+
+  Scenario: Gateway controller metrics reflect API operations
+    Given I am authenticated as "admin" with password "admin"
+    When I create a new API with name "metrics-test-api"
+    And I send a GET request to the gateway controller metrics endpoint
+    Then the response should contain metric "gateway_controller_api_operations_total"
+    And the response should contain metric "gateway_controller_apis_total"
+
+  Scenario: Policy engine metrics reflect request processing
+    Given I am authenticated as "admin" with password "admin"
+    And I create a new API with the following configuration:
+      """
+      {
+        "name": "metrics-api",
+        "version": "1.0",
+        "basePath": "/metrics-api",
+        "backend": {
+          "url": "http://sample-backend:9080"
+        },
+        "routes": [
+          {
+            "path": "/test",
+            "methods": ["GET"]
+          }
+        ]
+      }
+      """
+    And I wait for API deployment to complete
+    When I send a GET request to "/metrics-api/test" through the router
+    Then the response status code should be 200
+    When I send a GET request to the policy engine metrics endpoint
+    Then the response should contain metric "policy_engine_requests_total"

gateway/it/steps_metrics.go

@@ -0,0 +1,111 @@
+/*
+ * Copyright (c) 2025, WSO2 LLC. (https://www.wso2.com).
+ *
+ * WSO2 LLC. licenses this file to you under the Apache License,
+ * Version 2.0 (the "License"); you may not use this file except
+ * in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package it
+
+import (
+	"fmt"
+	"strings"
+
+	"github.com/cucumber/godog"
+	"github.com/wso2/api-platform/gateway/it/steps"
+)
+
+const (
+	// GatewayControllerMetricsPort is the port for gateway-controller metrics
+	GatewayControllerMetricsPort = "9091"
+
+	// PolicyEngineMetricsPort is the port for policy-engine metrics
+	PolicyEngineMetricsPort = "9003"
+)
+
+// MetricsSteps wraps TestState and HTTPSteps for metrics step definitions
+type MetricsSteps struct {
+	state     *TestState
+	httpSteps *steps.HTTPSteps
+}
+
+// RegisterMetricsSteps registers all metrics step definitions
+func RegisterMetricsSteps(ctx *godog.ScenarioContext, state *TestState, httpSteps *steps.HTTPSteps) {
+	m := &MetricsSteps{state: state, httpSteps: httpSteps}
+	ctx.Step(`^I send a GET request to the gateway controller metrics endpoint$`, m.iSendGETRequestToGatewayControllerMetrics)
+	ctx.Step(`^I send a GET request to the policy engine metrics endpoint$`, m.iSendGETRequestToPolicyEngineMetrics)
+	ctx.Step(`^the response should contain Prometheus metrics$`, m.theResponseShouldContainPrometheusMetrics)
+	ctx.Step(`^the response should contain metric "([^"]*)"$`, m.theResponseShouldContainMetric)
+}
+
+// iSendGETRequestToGatewayControllerMetrics sends a GET request to the gateway controller metrics endpoint
+func (m *MetricsSteps) iSendGETRequestToGatewayControllerMetrics() error {
+	url := fmt.Sprintf("http://localhost:%s/metrics", GatewayControllerMetricsPort)
+	return m.httpSteps.SendGETRequest(url)
+}
+
+// iSendGETRequestToPolicyEngineMetrics sends a GET request to the policy engine metrics endpoint
+func (m *MetricsSteps) iSendGETRequestToPolicyEngineMetrics() error {
+	url := fmt.Sprintf("http://localhost:%s/metrics", PolicyEngineMetricsPort)
+	return m.httpSteps.SendGETRequest(url)
+}
+
+// theResponseShouldContainPrometheusMetrics verifies the response contains valid Prometheus metrics
+func (m *MetricsSteps) theResponseShouldContainPrometheusMetrics() error {
+	resp := m.httpSteps.LastResponse()
+	if resp == nil {
+		return fmt.Errorf("no response received")
+	}
+
+	body := m.httpSteps.LastBody()
+	bodyStr := string(body)
+
+	// Check for Prometheus metric format indicators
+	// Valid metrics should have lines starting with # (comments) or metric names
+	if !strings.Contains(bodyStr, "# HELP") && !strings.Contains(bodyStr, "# TYPE") {
+		return fmt.Errorf("response does not contain Prometheus metric format headers")
+	}
+
+	// Ensure there's actual metric data (not just comments)
+	lines := strings.Split(bodyStr, "\n")
+	hasMetricData := false
+	for _, line := range lines {
+		line = strings.TrimSpace(line)
+		// Skip empty lines and comments
+		if line == "" || strings.HasPrefix(line, "#") {
+			continue
+		}
+		// If we find a non-empty, non-comment line, it's metric data
+		hasMetricData = true
+		break
+	}
+
+	if !hasMetricData {
+		return fmt.Errorf("response contains Prometheus headers but no actual metric data")
+	}
+
+	return nil
+}
+
+// theResponseShouldContainMetric verifies the response contains a specific metric
+func (m *MetricsSteps) theResponseShouldContainMetric(metricName string) error {
+	body := m.httpSteps.LastBody()
+	bodyStr := string(body)
+
+	if !strings.Contains(bodyStr, metricName) {
+		return fmt.Errorf("response does not contain metric '%s'", metricName)
+	}
+
+	return nil
+}

gateway/it/suite_test.go

@@ -70,6 +70,7 @@ func TestFeatures(t *testing.T) {
 			Format: "pretty",
 			Paths: []string{
 				"features/health.feature",
+				"features/metrics.feature",
 				"features/api_deploy.feature",
 				"features/mcp_deploy.feature",
 				"features/ratelimit.feature",
@@ -218,6 +219,7 @@ func InitializeScenario(ctx *godog.ScenarioContext) {
 	// Register step definitions
 	if testState != nil {
 		RegisterHealthSteps(ctx, testState, httpSteps)
+		RegisterMetricsSteps(ctx, testState, httpSteps)
 		RegisterAuthSteps(ctx, testState, httpSteps)
 		RegisterAPISteps(ctx, testState, httpSteps)
 		RegisterMCPSteps(ctx, testState, httpSteps)