From 4f45b06cb36c066fbe4c2b716d1fb138c655670f Mon Sep 17 00:00:00 2001 From: JR Boos Date: Fri, 27 Mar 2026 08:57:10 -0400 Subject: [PATCH] (lcore-1251) added tls e2e tests (lcore-1251) fixed tls tests & removed other e2e tests for quicker test running (lcore-1251) restored test_list.txt (lcore-1251) use `trustme` for certs (lcore-1251) quick tls server fix (lcore-1251) removed tags in place of steps (fix) removed unused code fix tls config verified correct llm response clean LCORE-1253: Add e2e proxy and TLS networking tests Add comprehensive end-to-end tests verifying that Llama Stack's NetworkConfig (proxy, TLS) works correctly through the Lightspeed Stack pipeline. Test infrastructure: - TunnelProxy: Async HTTP CONNECT tunnel proxy that creates TCP tunnels for HTTPS traffic. Tracks CONNECT count and target hosts. - InterceptionProxy: Async TLS-intercepting (MITM) proxy using trustme CA to generate per-target server certificates. Simulates corporate SSL inspection proxies. Behave scenarios (tests/e2e/features/proxy.feature): - Tunnel proxy: Configures run.yaml with NetworkConfig proxy pointing to a local tunnel proxy. Verifies CONNECT to api.openai.com:443 is observed and the LLM query succeeds through the proxy. - Interception proxy: Configures run.yaml with proxy and custom CA cert (trustme). Verifies TLS interception of api.openai.com traffic and successful LLM query through the MITM proxy. - TLS version: Configures run.yaml with min_version TLSv1.2 and verifies the LLM query succeeds with the TLS constraint. Each scenario dynamically generates a modified run-ci.yaml with the appropriate NetworkConfig, restarts Llama Stack with the new config, restarts the Lightspeed Stack, and sends a query to verify the full pipeline. Added trustme>=1.2.1 to dev dependencies. LCORE-1253: Add negative tests, TLS/cipher scenarios, and cleanup hooks Expand proxy e2e test coverage to fully address all acceptance criteria: AC1 (tunnel proxy): - Add negative test: LLM query fails gracefully when proxy is unreachable AC2 (interception proxy with CA): - Add negative test: LLM query fails when interception proxy CA is not provided (verifies "only successful when correct CA is provided") AC3 (TLS version and ciphers): - Add TLSv1.3 minimum version scenario - Add custom cipher suite configuration scenario (ECDHE+AESGCM:DHE+AESGCM) Test infrastructure: - Add after_scenario cleanup hook in environment.py that restores original Llama Stack and Lightspeed Stack configs after @Proxy scenarios. Prevents config leaks between scenarios. - Use different ports for each interception proxy instance to avoid address-already-in-use errors in sequential scenarios. Documentation: - Update docs/e2e_scenarios.md with all 7 proxy test scenarios. - Update docs/e2e_testing.md with proxy-related Behave tags (@Proxy, @TunnelProxy, @InterceptionProxy, @TLSVersion, @TLSCipher). LCORE-1253: Address review feedback Changes requested by reviewer (tisnik) and CodeRabbit: - Detect Docker mode once in before_all and store as context.is_docker_mode. All proxy step functions now use the context attribute instead of calling _is_docker_mode() repeatedly. - Log exception in _restore_original_services instead of silently swallowing it. - Only clear context.services_modified on successful restoration, not when restoration fails (prevents leaking modified state). - Add 10-second timeout to tunnel proxy open_connection to prevent stalls on unreachable targets. - Handle malformed CONNECT port with ValueError catch and 400 response. LCORE-1253: Replace tag-based cleanup with Background restore step Move config restoration from @Proxy after_scenario hook to an explicit Background Given step. This follows the team convention that tags are used only for test selection (filtering), not for triggering behavior. The Background step "The original Llama Stack config is restored if modified" runs before every scenario. If a previous scenario left a modified run.yaml (detected by backup file existence), it restores the original and restarts services. This handles cleanup even when the previous scenario failed mid-way. Removed: - @Proxy tag from feature file (was triggering after_scenario hook) - after_scenario hook for @Proxy in environment.py - _restore_original_services function (replaced by Background step) - context.services_modified tracking (no hook reads it) Updated docs/e2e_testing.md: tags documented as selection-only, not behavior-triggering. LCORE-1253: Address radofuchs review feedback Rewrite proxy e2e tests to follow project conventions: - Reuse existing step definitions: use "I use query to ask question" from llm_query_response.py and "The status code of the response is" from common_http.py instead of custom query/response steps. - Split service restart into two explicit Given steps: "Llama Stack is restarted" and "Lightspeed Stack is restarted" so restart ordering is visible in the feature file. - Remove local (non-Docker) mode code path. Proxy tests use restart_container() exclusively, consistent with the rest of the e2e test suite. - Check specific status code 500 for error scenarios instead of the broad >= 400 range. - Remove custom send_query, verify_llm_response, and verify_error_response steps that duplicated existing functionality. Net reduction: -183 lines from step definitions. LCORE-1253: Clean up proxy servers between scenarios Stop proxy servers and their event loops explicitly in the Background restore step. Previously, proxy daemon threads were left running after each scenario, causing asyncio "Task was destroyed but it is pending" warnings at process exit. The _stop_proxy helper schedules an async stop on the proxy's event loop, waits for it to complete, then stops the loop. Context references are cleared so the next scenario starts clean. LCORE-1253: Stop proxy servers after last scenario in after_feature Add proxy cleanup in after_feature to stop proxy servers left running from the last scenario. The Background restore step handles cleanup between scenarios, but the last scenario's proxies persist until process exit, causing asyncio "Task was destroyed" warnings. The cleanup checks for proxy objects on context (no tag check needed) and calls _stop_proxy to gracefully shut down the event loops. fixed dup steps addressed comments debug fix fix readded tests to test fix addressed comments update added invalid cert test readded test list added expired and untrusted certs added more tests fix added hostname mismatch tests fix format addressed comments addressed comments fixed style run e2e providers fix improved healthcheck removed multiple health checks re added tests fix fix fix --- docker-compose.yaml | 22 ++ .../rhoai/configs/lightspeed-stack-tls.yaml | 22 ++ .../server-mode/lightspeed-stack-tls.yaml | 22 ++ tests/e2e/features/environment.py | 11 + tests/e2e/features/steps/proxy.py | 7 +- tests/e2e/features/steps/tls.py | 224 ++++++++++++ tests/e2e/features/tls.feature | 182 ++++++++++ .../e2e/mock_tls_inference_server/Dockerfile | 14 + tests/e2e/mock_tls_inference_server/server.py | 318 ++++++++++++++++++ tests/e2e/test_list.txt | 1 + 10 files changed, 820 insertions(+), 3 deletions(-) create mode 100644 tests/e2e-prow/rhoai/configs/lightspeed-stack-tls.yaml create mode 100644 tests/e2e/configuration/server-mode/lightspeed-stack-tls.yaml create mode 100644 tests/e2e/features/steps/tls.py create mode 100644 tests/e2e/features/tls.feature create mode 100644 tests/e2e/mock_tls_inference_server/Dockerfile create mode 100644 tests/e2e/mock_tls_inference_server/server.py diff --git a/docker-compose.yaml b/docker-compose.yaml index f2d55d3fb..cfbba7c75 100644 --- a/docker-compose.yaml +++ b/docker-compose.yaml @@ -25,12 +25,16 @@ services: container_name: llama-stack ports: - "8321:8321" # Expose llama-stack on 8321 (adjust if needed) + depends_on: + mock-tls-inference: + condition: service_healthy volumes: - ./run.yaml:/opt/app-root/run.yaml:z - ${GCP_KEYS_PATH:-./tmp/.gcp-keys-dummy}:/opt/app-root/.gcp-keys:ro - ./lightspeed-stack.yaml:/opt/app-root/lightspeed-stack.yaml:ro - llama-storage:/opt/app-root/src/.llama/storage - ./tests/e2e/rag:/opt/app-root/src/.llama/storage/rag:z + - mock-tls-certs:/certs:ro environment: - BRAVE_SEARCH_API_KEY=${BRAVE_SEARCH_API_KEY:-} - TAVILY_SEARCH_API_KEY=${TAVILY_SEARCH_API_KEY:-} @@ -141,9 +145,27 @@ services: retries: 3 start_period: 2s + # Mock TLS inference server for TLS E2E tests + mock-tls-inference: + build: + context: ./tests/e2e/mock_tls_inference_server + dockerfile: Dockerfile + container_name: mock-tls-inference + networks: + - lightspeednet + volumes: + - mock-tls-certs:/certs + healthcheck: + test: ["CMD", "python", "-c", "import urllib.request,ssl;c=ssl.create_default_context();c.check_hostname=False;c.verify_mode=ssl.CERT_NONE;urllib.request.urlopen('https://localhost:8443/health',context=c)"] + interval: 5s + timeout: 3s + retries: 3 + start_period: 5s + volumes: llama-storage: + mock-tls-certs: networks: lightspeednet: diff --git a/tests/e2e-prow/rhoai/configs/lightspeed-stack-tls.yaml b/tests/e2e-prow/rhoai/configs/lightspeed-stack-tls.yaml new file mode 100644 index 000000000..fd45ea744 --- /dev/null +++ b/tests/e2e-prow/rhoai/configs/lightspeed-stack-tls.yaml @@ -0,0 +1,22 @@ +name: Lightspeed Core Service (LCS) +service: + host: 0.0.0.0 + port: 8080 + auth_enabled: false + workers: 1 + color_log: true + access_log: true +llama_stack: + use_as_library_client: false + url: http://${env.E2E_LLAMA_HOSTNAME}:8321 + api_key: xyzzy +user_data_collection: + feedback_enabled: true + feedback_storage: "/tmp/data/feedback" + transcripts_enabled: true + transcripts_storage: "/tmp/data/transcripts" +authentication: + module: "noop" +inference: + default_provider: tls-openai + default_model: mock-tls-model diff --git a/tests/e2e/configuration/server-mode/lightspeed-stack-tls.yaml b/tests/e2e/configuration/server-mode/lightspeed-stack-tls.yaml new file mode 100644 index 000000000..babdc2b99 --- /dev/null +++ b/tests/e2e/configuration/server-mode/lightspeed-stack-tls.yaml @@ -0,0 +1,22 @@ +name: Lightspeed Core Service (LCS) +service: + host: 0.0.0.0 + port: 8080 + auth_enabled: false + workers: 1 + color_log: true + access_log: true +llama_stack: + use_as_library_client: false + url: http://llama-stack:8321 + api_key: xyzzy +user_data_collection: + feedback_enabled: true + feedback_storage: "/tmp/data/feedback" + transcripts_enabled: true + transcripts_storage: "/tmp/data/transcripts" +authentication: + module: "noop" +inference: + default_provider: tls-openai + default_model: mock-tls-model diff --git a/tests/e2e/features/environment.py b/tests/e2e/features/environment.py index c42a2f3e4..fa89c45e6 100644 --- a/tests/e2e/features/environment.py +++ b/tests/e2e/features/environment.py @@ -552,6 +552,17 @@ def after_feature(context: Context, feature: Feature) -> None: restart_container("lightspeed-stack") remove_config_backup(context.default_config_backup) + # Restore Lightspeed Stack config if the generic configure_service step switched it. + # This cleanup intentionally runs for any feature (not tag-gated) - any feature that + # leaves a backup file will trigger config restoration and container restarts. + backup_path = "lightspeed-stack.yaml.backup" + if os.path.exists(backup_path): + switch_config(backup_path) + remove_config_backup(backup_path) + if not context.is_library_mode: + restart_container("llama-stack") + restart_container("lightspeed-stack") + # Clean up any proxy servers left from the last scenario if hasattr(context, "tunnel_proxy") or hasattr(context, "interception_proxy"): from tests.e2e.features.steps.proxy import _stop_proxy diff --git a/tests/e2e/features/steps/proxy.py b/tests/e2e/features/steps/proxy.py index 2455b94bb..d9521a6c3 100644 --- a/tests/e2e/features/steps/proxy.py +++ b/tests/e2e/features/steps/proxy.py @@ -142,9 +142,10 @@ def restore_if_modified(context: Context) -> None: _stop_proxy(context, "interception_proxy", "interception_proxy_loop") if os.path.exists(_LLAMA_STACK_CONFIG_BACKUP): - print("Restoring original Llama Stack config from backup...") - shutil.copy(_LLAMA_STACK_CONFIG_BACKUP, _LLAMA_STACK_CONFIG) - os.remove(_LLAMA_STACK_CONFIG_BACKUP) + print( + f"Restoring original Llama Stack config from {_LLAMA_STACK_CONFIG_BACKUP}..." + ) + shutil.move(_LLAMA_STACK_CONFIG_BACKUP, _LLAMA_STACK_CONFIG) restart_container("llama-stack") restart_container("lightspeed-stack") diff --git a/tests/e2e/features/steps/tls.py b/tests/e2e/features/steps/tls.py new file mode 100644 index 000000000..8a195bc0c --- /dev/null +++ b/tests/e2e/features/steps/tls.py @@ -0,0 +1,224 @@ +"""Step definitions for TLS configuration e2e tests. + +These tests configure Llama Stack's run.yaml with NetworkConfig TLS settings +and verify the full pipeline works through the Lightspeed Stack. + +Config switching uses the same pattern as other e2e tests: overwrite the +host-mounted run.yaml and restart Docker containers. Cleanup is handled +by a Background step that restores the backup before each scenario. +""" + +import copy +from typing import Any, Optional + +from behave import given # pyright: ignore[reportAttributeAccessIssue] +from behave.runner import Context + +from tests.e2e.features.steps.proxy import ( + _LLAMA_STACK_CONFIG, + _backup_llama_config, + _load_llama_config, + _write_config, +) + +_TLS_PROVIDER_BASE: dict[str, Any] = { + "provider_id": "tls-openai", + "provider_type": "remote::openai", + "config": { + "api_key": "test-key", + "base_url": "https://mock-tls-inference:8443/v1", + "allowed_models": ["mock-tls-model"], + }, +} + +_TLS_MODEL_RESOURCE: dict[str, str] = { + "model_id": "mock-tls-model", + "provider_id": "tls-openai", + "provider_model_id": "mock-tls-model", +} + + +def _ensure_tls_provider(config: dict[str, Any]) -> dict[str, Any]: + """Find or create the tls-openai inference provider in the config. + + If the provider does not exist, it is added along with the + mock-tls-model registered resource. + + Parameters: + config: The Llama Stack configuration dictionary. + + Returns: + The tls-openai provider configuration dictionary. + """ + providers = config.setdefault("providers", {}) + inference = providers.setdefault("inference", []) + + for provider in inference: + if provider.get("provider_id") == "tls-openai": + return provider + + # Provider not found — add it + provider = copy.deepcopy(_TLS_PROVIDER_BASE) + inference.append(provider) + + # Also register the model resource + resources = config.setdefault("registered_resources", {}) + models = resources.setdefault("models", []) + if not any(m.get("model_id") == "mock-tls-model" for m in models): + models.append(copy.deepcopy(_TLS_MODEL_RESOURCE)) + + return provider + + +def _configure_tls(tls_config: dict[str, Any], base_url: Optional[str] = None) -> None: + """Configure TLS settings for the tls-openai provider. + + Parameters: + tls_config: The TLS configuration dictionary. + base_url: Optional base URL override for the provider. + """ + _backup_llama_config() + config = _load_llama_config() + provider = _ensure_tls_provider(config) + provider.setdefault("config", {}).setdefault("network", {}) + if base_url is not None: + provider["config"]["base_url"] = base_url + provider["config"]["network"]["tls"] = tls_config + _write_config(config, _LLAMA_STACK_CONFIG) + + +# --- Background Steps --- +# Restart steps ("The original Llama Stack config is restored if modified", +# "Llama Stack is restarted", "Lightspeed Stack is restarted") are defined in +# proxy.py and shared across features by behave. + + +# --- TLS Configuration Steps --- + + +@given("Llama Stack is configured with TLS verification disabled") +def configure_tls_verify_false(context: Context) -> None: + """Configure run.yaml with TLS verify: false.""" + _configure_tls({"verify": False}) + + +@given("Llama Stack is configured with CA certificate verification") +def configure_tls_verify_ca(context: Context) -> None: + """Configure run.yaml with TLS verify: /certs/ca.crt.""" + _configure_tls({"verify": "/certs/ca.crt", "min_version": "TLSv1.2"}) + + +@given("Llama Stack is configured with TLS verification enabled") +def configure_tls_verify_true(context: Context) -> None: + """Configure run.yaml with TLS verify: true (fails with self-signed certs).""" + _configure_tls({"verify": True}) + + +@given("Llama Stack is configured with mutual TLS authentication") +def configure_tls_mtls(context: Context) -> None: + """Configure run.yaml with mutual TLS (client cert and key).""" + _configure_tls( + { + "verify": "/certs/ca.crt", + "client_cert": "/certs/client.crt", + "client_key": "/certs/client.key", + }, + base_url="https://mock-tls-inference:8444/v1", + ) + + +@given('Llama Stack is configured with CA certificate path "{path}"') +def configure_tls_verify_ca_path(context: Context, path: str) -> None: + """Configure run.yaml with TLS verify pointing to a specific CA cert path.""" + _configure_tls({"verify": path}) + + +@given("Llama Stack is configured for mTLS without client certificate") +def configure_mtls_no_client_cert(context: Context) -> None: + """Configure run.yaml for mTLS port without client cert (should fail).""" + _configure_tls( + {"verify": "/certs/ca.crt"}, + base_url="https://mock-tls-inference:8444/v1", + ) + + +@given("Llama Stack is configured for mTLS with wrong client certificate") +def configure_mtls_wrong_client_cert(context: Context) -> None: + """Configure run.yaml for mTLS with invalid client cert (CA cert as client cert).""" + _configure_tls( + { + "verify": "/certs/ca.crt", + "client_cert": "/certs/ca.crt", + "client_key": "/certs/client.key", + }, + base_url="https://mock-tls-inference:8444/v1", + ) + + +@given("Llama Stack is configured for mTLS with untrusted client certificate") +def configure_mtls_untrusted_client_cert(context: Context) -> None: + """Configure run.yaml for mTLS with client cert from untrusted CA.""" + _configure_tls( + { + "verify": "/certs/ca.crt", + "client_cert": "/certs/untrusted-client.crt", + "client_key": "/certs/untrusted-client.key", + }, + base_url="https://mock-tls-inference:8444/v1", + ) + + +@given("Llama Stack is configured for mTLS with expired client certificate") +def configure_mtls_expired_client_cert(context: Context) -> None: + """Configure run.yaml for mTLS with an expired client certificate.""" + _configure_tls( + { + "verify": "/certs/ca.crt", + "client_cert": "/certs/expired-client.crt", + "client_key": "/certs/client.key", + }, + base_url="https://mock-tls-inference:8444/v1", + ) + + +@given("Llama Stack is configured with CA certificate and hostname mismatch server") +def configure_tls_hostname_mismatch(context: Context) -> None: + """Configure run.yaml to connect to hostname-mismatch server (should fail).""" + _configure_tls( + {"verify": "/certs/ca.crt"}, + base_url="https://mock-tls-inference:8445/v1", + ) + + +@given("Llama Stack is configured with mutual TLS and hostname mismatch server") +def configure_mtls_hostname_mismatch(context: Context) -> None: + """Configure run.yaml for mTLS against hostname-mismatch server (should fail).""" + _configure_tls( + { + "verify": "/certs/ca.crt", + "client_cert": "/certs/client.crt", + "client_key": "/certs/client.key", + }, + base_url="https://mock-tls-inference:8445/v1", + ) + + +@given( + 'Llama Stack is configured with TLS minimum version "{version}" and hostname mismatch server' +) +def configure_tls_min_version_hostname_mismatch(context: Context, version: str) -> None: + """Configure run.yaml with TLS min version against hostname-mismatch server.""" + _configure_tls( + {"verify": "/certs/ca.crt", "min_version": version}, + base_url="https://mock-tls-inference:8445/v1", + ) + + +@given( + 'Llama Stack is configured with TLS minimum version "{version}" and CA certificate path "{path}"' +) +def configure_tls_min_version_with_ca_path( + context: Context, version: str, path: str +) -> None: + """Configure run.yaml with TLS minimum version and a specific CA cert path.""" + _configure_tls({"verify": path, "min_version": version}) diff --git a/tests/e2e/features/tls.feature b/tests/e2e/features/tls.feature new file mode 100644 index 000000000..a9c179aaa --- /dev/null +++ b/tests/e2e/features/tls.feature @@ -0,0 +1,182 @@ +@skip-in-library-mode +Feature: TLS configuration for remote inference providers + Validate that Llama Stack's NetworkConfig.tls settings are applied correctly + when connecting to a remote inference provider over HTTPS. + + Background: + Given The service is started locally + And REST API service prefix is /v1 + And The service uses the lightspeed-stack-tls.yaml configuration + And The original Llama Stack config is restored if modified + + Scenario: Inference succeeds with TLS verification disabled + Given Llama Stack is configured with TLS verification disabled + And Llama Stack is restarted + And Lightspeed Stack is restarted + When I use "query" to ask question + """ + {"query": "Say hello", "model": "mock-tls-model", "provider": "tls-openai"} + """ + Then The status code of the response is 200 + + Scenario: Inference succeeds with CA certificate verification + Given Llama Stack is configured with CA certificate verification + And Llama Stack is restarted + And Lightspeed Stack is restarted + When I use "query" to ask question + """ + {"query": "Say hello", "model": "mock-tls-model", "provider": "tls-openai"} + """ + Then The status code of the response is 200 + + Scenario: Inference fails with an untrusted CA certificate + Given Llama Stack is configured with CA certificate path "/certs/untrusted-ca.crt" + And Llama Stack is restarted + And Lightspeed Stack is restarted + When I use "query" to ask question + """ + {"query": "Say hello", "model": "mock-tls-model", "provider": "tls-openai"} + """ + Then The status code of the response is 500 + And The body of the response does not contain Hello from the TLS mock inference server + + Scenario: Inference fails with an expired CA certificate + Given Llama Stack is configured with CA certificate path "/certs/expired-ca.crt" + And Llama Stack is restarted + And Lightspeed Stack is restarted + When I use "query" to ask question + """ + {"query": "Say hello", "model": "mock-tls-model", "provider": "tls-openai"} + """ + Then The status code of the response is 500 + And The body of the response does not contain Hello from the TLS mock inference server + + Scenario: Inference fails when TLS verify is true against self-signed cert + Given Llama Stack is configured with TLS verification enabled + And Llama Stack is restarted + And Lightspeed Stack is restarted + When I use "query" to ask question + """ + {"query": "Say hello", "model": "mock-tls-model", "provider": "tls-openai"} + """ + Then The status code of the response is 500 + And The body of the response does not contain Hello from the TLS mock inference server + + Scenario: Inference succeeds with mutual TLS authentication + Given Llama Stack is configured with mutual TLS authentication + And Llama Stack is restarted + And Lightspeed Stack is restarted + When I use "query" to ask question + """ + {"query": "Say hello", "model": "mock-tls-model", "provider": "tls-openai"} + """ + Then The status code of the response is 200 + + Scenario: Inference fails when mTLS is required but no client certificate is provided + Given Llama Stack is configured for mTLS without client certificate + And Llama Stack is restarted + And Lightspeed Stack is restarted + When I use "query" to ask question + """ + {"query": "Say hello", "model": "mock-tls-model", "provider": "tls-openai"} + """ + Then The status code of the response is 500 + And The body of the response does not contain Hello from the TLS mock inference server + + Scenario: Inference fails when mTLS is required but wrong client certificate is provided + Given Llama Stack is configured for mTLS with wrong client certificate + And Llama Stack is restarted + And Lightspeed Stack is restarted + When I use "query" to ask question + """ + {"query": "Say hello", "model": "mock-tls-model", "provider": "tls-openai"} + """ + Then The status code of the response is 500 + And The body of the response does not contain Hello from the TLS mock inference server + + Scenario: Inference fails when mTLS is required but untrusted client certificate is provided + Given Llama Stack is configured for mTLS with untrusted client certificate + And Llama Stack is restarted + And Lightspeed Stack is restarted + When I use "query" to ask question + """ + {"query": "Say hello", "model": "mock-tls-model", "provider": "tls-openai"} + """ + Then The status code of the response is 500 + And The body of the response does not contain Hello from the TLS mock inference server + + Scenario: Inference fails when mTLS is required but expired client certificate is provided + Given Llama Stack is configured for mTLS with expired client certificate + And Llama Stack is restarted + And Lightspeed Stack is restarted + When I use "query" to ask question + """ + {"query": "Say hello", "model": "mock-tls-model", "provider": "tls-openai"} + """ + Then The status code of the response is 500 + And The body of the response does not contain Hello from the TLS mock inference server + + Scenario: Inference fails with CA certificate verification and hostname mismatch + Given Llama Stack is configured with CA certificate and hostname mismatch server + And Llama Stack is restarted + And Lightspeed Stack is restarted + When I use "query" to ask question + """ + {"query": "Say hello", "model": "mock-tls-model", "provider": "tls-openai"} + """ + Then The status code of the response is 500 + And The body of the response does not contain Hello from the TLS mock inference server + + Scenario: Inference fails with mutual TLS and hostname mismatch + Given Llama Stack is configured with mutual TLS and hostname mismatch server + And Llama Stack is restarted + And Lightspeed Stack is restarted + When I use "query" to ask question + """ + {"query": "Say hello", "model": "mock-tls-model", "provider": "tls-openai"} + """ + Then The status code of the response is 500 + And The body of the response does not contain Hello from the TLS mock inference server + + Scenario: Inference succeeds with TLS minimum version TLSv1.3 + Given Llama Stack is configured with TLS minimum version "TLSv1.3" and CA certificate path "/certs/ca.crt" + And Llama Stack is restarted + And Lightspeed Stack is restarted + When I use "query" to ask question + """ + {"query": "Say hello", "model": "mock-tls-model", "provider": "tls-openai"} + """ + Then The status code of the response is 200 + + Scenario: Inference fails with TLS minimum version TLSv1.3 and untrusted CA certificate + Given Llama Stack is configured with TLS minimum version "TLSv1.3" and CA certificate path "/certs/untrusted-ca.crt" + And Llama Stack is restarted + And Lightspeed Stack is restarted + When I use "query" to ask question + """ + {"query": "Say hello", "model": "mock-tls-model", "provider": "tls-openai"} + """ + Then The status code of the response is 500 + And The body of the response does not contain Hello from the TLS mock inference server + + Scenario: Inference fails with TLS minimum version TLSv1.3 and hostname mismatch + Given Llama Stack is configured with TLS minimum version "TLSv1.3" and hostname mismatch server + And Llama Stack is restarted + And Lightspeed Stack is restarted + When I use "query" to ask question + """ + {"query": "Say hello", "model": "mock-tls-model", "provider": "tls-openai"} + """ + Then The status code of the response is 500 + And The body of the response does not contain Hello from the TLS mock inference server + + Scenario: Inference fails with TLS minimum version TLSv1.3 and expired CA certificate + Given Llama Stack is configured with TLS minimum version "TLSv1.3" and CA certificate path "/certs/expired-ca.crt" + And Llama Stack is restarted + And Lightspeed Stack is restarted + When I use "query" to ask question + """ + {"query": "Say hello", "model": "mock-tls-model", "provider": "tls-openai"} + """ + Then The status code of the response is 500 + And The body of the response does not contain Hello from the TLS mock inference server diff --git a/tests/e2e/mock_tls_inference_server/Dockerfile b/tests/e2e/mock_tls_inference_server/Dockerfile new file mode 100644 index 000000000..ee9cbde16 --- /dev/null +++ b/tests/e2e/mock_tls_inference_server/Dockerfile @@ -0,0 +1,14 @@ +FROM python:3.12-slim +WORKDIR /app + +# Install trustme for dynamic certificate generation +RUN pip install --no-cache-dir trustme + +# Copy server script +COPY server.py . + +# Create /certs directory for generated certificates +RUN mkdir -p /certs + +EXPOSE 8443 8444 +CMD ["python", "server.py"] diff --git a/tests/e2e/mock_tls_inference_server/server.py b/tests/e2e/mock_tls_inference_server/server.py new file mode 100644 index 000000000..eead2b475 --- /dev/null +++ b/tests/e2e/mock_tls_inference_server/server.py @@ -0,0 +1,318 @@ +#!/usr/bin/env python3 +"""Mock OpenAI-compatible HTTPS inference server for TLS e2e testing. + +Serves two HTTPS listeners using trustme-generated test certificates: + - Port 8443: standard TLS (no client certificate required) + - Port 8444: mutual TLS (client certificate required, verified against CA) + +Implements the minimal OpenAI API surface needed by Llama Stack's +remote::openai provider: /v1/models and /v1/chat/completions. + +Certificates are generated on-the-fly using trustme at server startup. +""" + +import datetime +import json +import ssl +import threading +import time +from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer +from pathlib import Path +from typing import Any + +import trustme +from cryptography.hazmat.primitives import hashes, serialization +from cryptography.x509 import CertificateBuilder, random_serial_number + +MODEL_ID = "mock-tls-model" +TLS_PORT = 8443 +MTLS_PORT = 8444 +HOSTNAME_MISMATCH_PORT = 8445 + + +class OpenAIHandler(BaseHTTPRequestHandler): + """Handles OpenAI-compatible API requests over HTTPS.""" + + def log_message( + self, format: str, *args: Any + ) -> None: # pylint: disable=redefined-builtin + """Timestamp log output.""" + print(f"[{time.strftime('%Y-%m-%d %H:%M:%S')}] {format % args}") + + def do_GET(self) -> None: # pylint: disable=invalid-name + """Handle GET requests.""" + if self.path == "/health": + self._send_json({"status": "ok"}) + elif self.path == "/v1/models": + self._send_json( + { + "object": "list", + "data": [ + { + "id": MODEL_ID, + "object": "model", + "created": 1700000000, + "owned_by": "test", + } + ], + } + ) + else: + self.send_error(404) + + def do_POST(self) -> None: # pylint: disable=invalid-name + """Handle POST requests (chat completions).""" + if self.path != "/v1/chat/completions": + self.send_error(404) + return + + content_length = int(self.headers.get("Content-Length", 0)) + body = self.rfile.read(content_length) if content_length > 0 else b"{}" + + try: + request_data = json.loads(body.decode("utf-8")) + except (json.JSONDecodeError, UnicodeDecodeError): + request_data = {} + + model = request_data.get("model", MODEL_ID) + + self._send_json( + { + "id": "chatcmpl-tls-test-001", + "object": "chat.completion", + "created": 1700000000, + "model": model, + "choices": [ + { + "index": 0, + "message": { + "role": "assistant", + "content": "Hello from the TLS mock inference server.", + }, + "finish_reason": "stop", + } + ], + "usage": { + "prompt_tokens": 8, + "completion_tokens": 9, + "total_tokens": 17, + }, + } + ) + + def _send_json(self, data: dict | list) -> None: + """Write a JSON response.""" + payload = json.dumps(data).encode() + self.send_response(200) + self.send_header("Content-Type", "application/json") + self.send_header("Content-Length", str(len(payload))) + self.end_headers() + self.wfile.write(payload) + + +def _export_expired_client_cert( + ca: trustme.CA, client_cert: trustme.LeafCert, path: Path +) -> None: + """Re-sign a client certificate with expired validity dates. + + Parameters: + ca: The CA that issued the original client certificate. + client_cert: The original client leaf certificate. + path: File path to write the expired client certificate PEM. + + Note: + Accesses ca._private_key which is a private attribute of trustme.CA. + This is fragile and may break if trustme changes its internal implementation. + No public API exists in trustme for re-signing certs with custom validity. + """ + original = client_cert.cert_chain_pems[0].bytes() + from cryptography.x509 import load_pem_x509_certificate + + orig_cert = load_pem_x509_certificate(original) + now = datetime.datetime.now(datetime.UTC) + builder = CertificateBuilder() + builder = builder.subject_name(orig_cert.subject) + builder = builder.issuer_name(orig_cert.issuer) + builder = builder.public_key(orig_cert.public_key()) + builder = builder.serial_number(random_serial_number()) + builder = builder.not_valid_before(now - datetime.timedelta(days=365)) + builder = builder.not_valid_after(now - datetime.timedelta(seconds=1)) + for ext in orig_cert.extensions: + builder = builder.add_extension(ext.value, ext.critical) + expired_cert = builder.sign(ca._private_key, hashes.SHA256()) + path.write_bytes(expired_cert.public_bytes(serialization.Encoding.PEM)) + + +def _export_expired_ca_cert(ca: trustme.CA, path: Path) -> None: + """Re-sign a trustme CA certificate with expired validity dates. + + Creates a copy of the CA's self-signed certificate but with validity + dates set in the past, making it an expired certificate. + + Parameters: + ca: The trustme CA whose certificate and key to use. + path: File path to write the expired CA certificate PEM. + + Note: + Accesses ca._certificate and ca._private_key which are private attributes + of trustme.CA. This is fragile and may break if trustme changes its + internal implementation. No public API exists for re-signing with custom validity. + """ + original = ca._certificate + now = datetime.datetime.now(datetime.UTC) + builder = CertificateBuilder() + builder = builder.subject_name(original.subject) + builder = builder.issuer_name(original.issuer) + builder = builder.public_key(original.public_key()) + builder = builder.serial_number(random_serial_number()) + builder = builder.not_valid_before(now - datetime.timedelta(days=365)) + builder = builder.not_valid_after(now - datetime.timedelta(seconds=1)) + for ext in original.extensions: + builder = builder.add_extension(ext.value, ext.critical) + expired_cert = builder.sign(ca._private_key, hashes.SHA256()) + path.write_bytes(expired_cert.public_bytes(serialization.Encoding.PEM)) + + +def _make_tls_context( + ca: trustme.CA, + server_cert: trustme.LeafCert, + require_client_cert: bool = False, +) -> ssl.SSLContext: + """Build an SSL context using trustme-generated certificates. + + Parameters: + ca: The trustme CA instance. + server_cert: The server certificate issued by the CA. + require_client_cert: Whether to require client certificate (mTLS). + + Returns: + Configured SSL context for server-side TLS. + """ + ctx = ssl.SSLContext(ssl.PROTOCOL_TLS_SERVER) + server_cert.configure_cert(ctx) + if require_client_cert: + ctx.verify_mode = ssl.CERT_REQUIRED + ca.configure_trust(ctx) + return ctx + + +def _run_server(httpd: ThreadingHTTPServer, label: str) -> None: + """Serve requests forever in a daemon thread.""" + print(f"{label} listening") + try: + httpd.serve_forever() + except Exception as exc: # pylint: disable=broad-except + print(f"{label} error: {exc}") + + +def main() -> None: + """Start standard-TLS (8443) and mTLS (8444) listeners. + + Generates certificates on-the-fly using trustme and exports the CA cert + to /certs/ca.crt and client cert to /certs/client.* for use by tests. + """ + print("=" * 60) + print("Generating TLS certificates with trustme...") + print("=" * 60) + + # Generate CA and certificates + ca = trustme.CA() + # Server cert with SANs for Docker service name and localhost + server_cert = ca.issue_cert("mock-tls-inference", "localhost", "127.0.0.1") + # Client cert for mTLS testing (use a simple hostname without spaces) + client_cert = ca.issue_cert("tls-e2e-test-client") + + # Export certificates to /certs directory for access by tests + certs_dir = Path("/certs") + certs_dir.mkdir(exist_ok=True, parents=True) + + # Export CA certificate + ca.cert_pem.write_to_path(str(certs_dir / "ca.crt")) + print(f" CA cert: {certs_dir / 'ca.crt'}") + + # Export client certificate and key for mTLS tests + client_cert.private_key_pem.write_to_path(str(certs_dir / "client.key")) + # Write certificate chain (may include multiple certs) + with (certs_dir / "client.crt").open("wb") as f: + for blob in client_cert.cert_chain_pems: + f.write(blob.bytes()) + print(f" Client cert: {certs_dir / 'client.crt'}") + print(f" Client key: {certs_dir / 'client.key'}") + + # Export untrusted CA certificate (from a separate CA that did not sign the server cert) + untrusted_ca = trustme.CA() + untrusted_ca.cert_pem.write_to_path(str(certs_dir / "untrusted-ca.crt")) + print(f" Untrusted CA cert: {certs_dir / 'untrusted-ca.crt'}") + + # Export expired CA certificate (re-signed with past validity dates) + _export_expired_ca_cert(ca, certs_dir / "expired-ca.crt") + print(f" Expired CA cert: {certs_dir / 'expired-ca.crt'}") + + # Export untrusted client certificate (issued by a different CA) + untrusted_client = untrusted_ca.issue_cert("tls-e2e-untrusted-client") + untrusted_client.private_key_pem.write_to_path( + str(certs_dir / "untrusted-client.key") + ) + with (certs_dir / "untrusted-client.crt").open("wb") as f: + for blob in untrusted_client.cert_chain_pems: + f.write(blob.bytes()) + print(f" Untrusted client cert: {certs_dir / 'untrusted-client.crt'}") + + # Export expired client certificate (signed by main CA but with past dates) + _export_expired_client_cert(ca, client_cert, certs_dir / "expired-client.crt") + print(f" Expired client cert: {certs_dir / 'expired-client.crt'}") + + # Issue a server cert with a hostname that does NOT match the Docker service + # name ("mock-tls-inference") — used to test hostname-mismatch rejection. + hostname_mismatch_cert = ca.issue_cert("wrong-hostname.example.com") + print(" Hostname-mismatch server cert: wrong-hostname.example.com (port 8445)") + + print("=" * 60) + print("Starting servers...") + print("=" * 60) + + # Create TLS server (no client cert required) + tls_server = ThreadingHTTPServer(("", TLS_PORT), OpenAIHandler) + tls_ctx = _make_tls_context(ca, server_cert, require_client_cert=False) + tls_server.socket = tls_ctx.wrap_socket(tls_server.socket, server_side=True) + + # Create mTLS server (client cert required) + mtls_server = ThreadingHTTPServer(("", MTLS_PORT), OpenAIHandler) + mtls_ctx = _make_tls_context(ca, server_cert, require_client_cert=True) + mtls_server.socket = mtls_ctx.wrap_socket(mtls_server.socket, server_side=True) + + # Create hostname-mismatch TLS server (cert SAN ≠ connecting hostname) + mismatch_server = ThreadingHTTPServer(("", HOSTNAME_MISMATCH_PORT), OpenAIHandler) + mismatch_ctx = _make_tls_context( + ca, hostname_mismatch_cert, require_client_cert=False + ) + mismatch_server.socket = mismatch_ctx.wrap_socket( + mismatch_server.socket, server_side=True + ) + + print("=" * 60) + print("Mock TLS Inference Server") + print("=" * 60) + print(f" TLS : https://localhost:{TLS_PORT} (no client cert)") + print(f" mTLS : https://localhost:{MTLS_PORT} (client cert required)") + print( + f" Mismatch : https://localhost:{HOSTNAME_MISMATCH_PORT}" + " (hostname-mismatch cert)" + ) + print(f" Model: {MODEL_ID}") + print("=" * 60) + + for srv, label in [ + (tls_server, f"TLS :{TLS_PORT}"), + (mtls_server, f"mTLS :{MTLS_PORT}"), + (mismatch_server, f"Mismatch :{HOSTNAME_MISMATCH_PORT}"), + ]: + t = threading.Thread(target=_run_server, args=(srv, label), daemon=True) + t.start() + + # Keep main thread alive (daemon threads run until container stops) + threading.Event().wait() + + +if __name__ == "__main__": + main() diff --git a/tests/e2e/test_list.txt b/tests/e2e/test_list.txt index 83ed6a17f..4668a551b 100644 --- a/tests/e2e/test_list.txt +++ b/tests/e2e/test_list.txt @@ -20,3 +20,4 @@ features/rest_api.feature features/mcp.feature features/models.feature features/proxy.feature +features/tls.feature \ No newline at end of file