(lcore-1251) added tls e2e tests

(lcore-1251) fixed tls tests & removed other e2e tests for quicker test running

(lcore-1251) restored test_list.txt

(lcore-1251) use `trustme` for certs

(lcore-1251) quick tls server fix

(lcore-1251) removed tags in place of steps

(fix) removed unused code

fix tls config

verified correct llm response

clean

LCORE-1253: Add e2e proxy and TLS networking tests

Add comprehensive end-to-end tests verifying that Llama Stack's
NetworkConfig (proxy, TLS) works correctly through the Lightspeed
Stack pipeline.

Test infrastructure:
- TunnelProxy: Async HTTP CONNECT tunnel proxy that creates TCP
  tunnels for HTTPS traffic. Tracks CONNECT count and target hosts.
- InterceptionProxy: Async TLS-intercepting (MITM) proxy using
  trustme CA to generate per-target server certificates. Simulates
  corporate SSL inspection proxies.

Behave scenarios (tests/e2e/features/proxy.feature):
- Tunnel proxy: Configures run.yaml with NetworkConfig proxy pointing
  to a local tunnel proxy. Verifies CONNECT to api.openai.com:443
  is observed and the LLM query succeeds through the proxy.
- Interception proxy: Configures run.yaml with proxy and custom CA
  cert (trustme). Verifies TLS interception of api.openai.com traffic
  and successful LLM query through the MITM proxy.
- TLS version: Configures run.yaml with min_version TLSv1.2 and
  verifies the LLM query succeeds with the TLS constraint.

Each scenario dynamically generates a modified run-ci.yaml with the
appropriate NetworkConfig, restarts Llama Stack with the new config,
restarts the Lightspeed Stack, and sends a query to verify the full
pipeline.

Added trustme>=1.2.1 to dev dependencies.

LCORE-1253: Add negative tests, TLS/cipher scenarios, and cleanup hooks

Expand proxy e2e test coverage to fully address all acceptance criteria:

AC1 (tunnel proxy):
- Add negative test: LLM query fails gracefully when proxy is unreachable

AC2 (interception proxy with CA):
- Add negative test: LLM query fails when interception proxy CA is not
  provided (verifies "only successful when correct CA is provided")

AC3 (TLS version and ciphers):
- Add TLSv1.3 minimum version scenario
- Add custom cipher suite configuration scenario (ECDHE+AESGCM:DHE+AESGCM)

Test infrastructure:
- Add after_scenario cleanup hook in environment.py that restores
  original Llama Stack and Lightspeed Stack configs after @Proxy
  scenarios. Prevents config leaks between scenarios.
- Use different ports for each interception proxy instance to avoid
  address-already-in-use errors in sequential scenarios.

Documentation:
- Update docs/e2e_scenarios.md with all 7 proxy test scenarios.
- Update docs/e2e_testing.md with proxy-related Behave tags
  (@Proxy, @tunnelproxy, @InterceptionProxy, @TLSVersion, @tlscipher).

LCORE-1253: Address review feedback

Changes requested by reviewer (tisnik) and CodeRabbit:

- Detect Docker mode once in before_all and store as
  context.is_docker_mode. All proxy step functions now use the
  context attribute instead of calling _is_docker_mode() repeatedly.
- Log exception in _restore_original_services instead of silently
  swallowing it.
- Only clear context.services_modified on successful restoration,
  not when restoration fails (prevents leaking modified state).
- Add 10-second timeout to tunnel proxy open_connection to prevent
  stalls on unreachable targets.
- Handle malformed CONNECT port with ValueError catch and 400
  response.

LCORE-1253: Replace tag-based cleanup with Background restore step

Move config restoration from @Proxy after_scenario hook to an
explicit Background Given step. This follows the team convention
that tags are used only for test selection (filtering), not for
triggering behavior.

The Background step "The original Llama Stack config is restored
if modified" runs before every scenario. If a previous scenario
left a modified run.yaml (detected by backup file existence), it
restores the original and restarts services. This handles cleanup
even when the previous scenario failed mid-way.

Removed:
- @Proxy tag from feature file (was triggering after_scenario hook)
- after_scenario hook for @Proxy in environment.py
- _restore_original_services function (replaced by Background step)
- context.services_modified tracking (no hook reads it)

Updated docs/e2e_testing.md: tags documented as selection-only,
not behavior-triggering.

LCORE-1253: Address radofuchs review feedback

Rewrite proxy e2e tests to follow project conventions:

- Reuse existing step definitions: use "I use query to ask question"
  from llm_query_response.py and "The status code of the response is"
  from common_http.py instead of custom query/response steps.
- Split service restart into two explicit Given steps: "Llama Stack
  is restarted" and "Lightspeed Stack is restarted" so restart
  ordering is visible in the feature file.
- Remove local (non-Docker) mode code path. Proxy tests use
  restart_container() exclusively, consistent with the rest of the
  e2e test suite.
- Check specific status code 500 for error scenarios instead of the
  broad >= 400 range.
- Remove custom send_query, verify_llm_response, and
  verify_error_response steps that duplicated existing functionality.

Net reduction: -183 lines from step definitions.

LCORE-1253: Clean up proxy servers between scenarios

Stop proxy servers and their event loops explicitly in the Background
restore step. Previously, proxy daemon threads were left running after
each scenario, causing asyncio "Task was destroyed but it is pending"
warnings at process exit.

The _stop_proxy helper schedules an async stop on the proxy's event
loop, waits for it to complete, then stops the loop. Context
references are cleared so the next scenario starts clean.

LCORE-1253: Stop proxy servers after last scenario in after_feature

Add proxy cleanup in after_feature to stop proxy servers left running
from the last scenario. The Background restore step handles cleanup
between scenarios, but the last scenario's proxies persist until
process exit, causing asyncio "Task was destroyed" warnings.

The cleanup checks for proxy objects on context (no tag check needed)
and calls _stop_proxy to gracefully shut down the event loops.

Author: jrobertboos

docker-compose-library.yaml

@@ -30,6 +30,8 @@ services:
         condition: service_healthy
       mock-mcp:
         condition: service_healthy
+      mock-tls-inference:
+        condition: service_healthy
     networks:
       - lightspeednet
     volumes:
@@ -40,6 +42,7 @@ services:
       - ./tests/e2e/rag:/opt/app-root/src/.llama/storage/rag:Z
       - ./tests/e2e/secrets/mcp-token:/tmp/mcp-token:ro
       - ./tests/e2e/secrets/invalid-mcp-token:/tmp/invalid-mcp-token:ro
+      - mock-tls-certs:/certs:ro
     environment:
       # LLM Provider API Keys
       - BRAVE_SEARCH_API_KEY=${BRAVE_SEARCH_API_KEY:-}
@@ -113,7 +116,30 @@ services:
       retries: 3
       start_period: 2s
 
+  # Mock TLS inference server for TLS E2E tests
+  mock-tls-inference:
+    build:
+      context: ./tests/e2e/mock_tls_inference_server
+      dockerfile: Dockerfile
+    container_name: mock-tls-inference
+    ports:
+      - "8443:8443"
+      - "8444:8444"
+    networks:
+      - lightspeednet
+    volumes:
+      - mock-tls-certs:/certs
+    healthcheck:
+      test: ["CMD", "python", "-c", "import urllib.request,ssl;c=ssl.create_default_context();c.check_hostname=False;c.verify_mode=ssl.CERT_NONE;urllib.request.urlopen('https://localhost:8443/health',context=c)"]
+      interval: 5s
+      timeout: 3s
+      retries: 3
+      start_period: 5s
+
 
 networks:
   lightspeednet:
-    driver: bridge
+    driver: bridge
+
+volumes:
+  mock-tls-certs:

docker-compose.yaml

@@ -25,12 +25,16 @@ services:
     container_name: llama-stack
     ports:
       - "8321:8321"  # Expose llama-stack on 8321 (adjust if needed)
+    depends_on:
+      mock-tls-inference:
+        condition: service_healthy
     volumes:
       - ./run.yaml:/opt/app-root/run.yaml:z
       - ${GCP_KEYS_PATH:-./tmp/.gcp-keys-dummy}:/opt/app-root/.gcp-keys:ro
       - ./lightspeed-stack.yaml:/opt/app-root/lightspeed-stack.yaml:ro
       - llama-storage:/opt/app-root/src/.llama/storage
       - ./tests/e2e/rag:/opt/app-root/src/.llama/storage/rag:z
+      - mock-tls-certs:/certs:ro
     environment:
       - BRAVE_SEARCH_API_KEY=${BRAVE_SEARCH_API_KEY:-}
       - TAVILY_SEARCH_API_KEY=${TAVILY_SEARCH_API_KEY:-}
@@ -140,9 +144,30 @@ services:
       retries: 3
       start_period: 2s
 
+  # Mock TLS inference server for TLS E2E tests
+  mock-tls-inference:
+    build:
+      context: ./tests/e2e/mock_tls_inference_server
+      dockerfile: Dockerfile
+    container_name: mock-tls-inference
+    ports:
+      - "8443:8443"
+      - "8444:8444"
+    networks:
+      - lightspeednet
+    volumes:
+      - mock-tls-certs:/certs
+    healthcheck:
+      test: ["CMD", "python", "-c", "import urllib.request,ssl;c=ssl.create_default_context();c.check_hostname=False;c.verify_mode=ssl.CERT_NONE;urllib.request.urlopen('https://localhost:8443/health',context=c)"]
+      interval: 5s
+      timeout: 3s
+      retries: 3
+      start_period: 5s
+
 
 volumes:
   llama-storage:
+  mock-tls-certs:
 
 networks:
   lightspeednet:

docs/e2e_scenarios.md

@@ -116,6 +116,16 @@
 * Check if models can be filtered
 * Check if filtering can return empty list of models
 
+## [`proxy.feature`](https://github.com/lightspeed-core/lightspeed-stack/blob/main/tests/e2e/features/proxy.feature)
+
+* LLM traffic is routed through a configured tunnel proxy
+* LLM query fails gracefully when proxy is unreachable
+* LLM traffic works through interception proxy with correct CA
+* LLM query fails when interception proxy CA is not provided
+* TLS minimum version TLSv1.2 is respected
+* TLS minimum version TLSv1.3 is respected
+* Custom cipher suite configuration is respected
+
 ## [`query.feature`](https://github.com/lightspeed-core/lightspeed-stack/blob/main/tests/e2e/features/query.feature)
 
 * Check if LLM responds properly to restrictive system prompt to sent question with different system prompt

docs/e2e_testing.md

@@ -192,6 +192,10 @@ All tag behaviour is implemented in **`features/environment.py`**: the hooks (`b
 | `@RHIdentity`                   | Feature-level: use RH identity config; restore in after_feature.                                                                                        |
 | `@Feedback`                     | Feature-level: set feedback conversation list; after_feature deletes those conversations.                                                               |
 | `@MCP`                          | Feature-level: use MCP config; restore in after_feature.                                                                                                |
+| `@TunnelProxy`                  | Selection: tunnel proxy (HTTP CONNECT) scenarios.                                                                                                       |
+| `@InterceptionProxy`            | Selection: TLS-intercepting proxy with trustme CA scenarios.                                                                                            |
+| `@TLSVersion`                   | Selection: TLS version configuration scenarios.                                                                                                          |
+| `@TLSCipher`                    | Selection: cipher suite configuration scenarios.                                                                                                         |
 
 
 ### Multiple Tags and Skip Comment

pyproject.toml

@@ -126,6 +126,7 @@ dev = [
     "ruff>=0.11.13",
     "aiosqlite",
     "behave>=1.3.0",
+    "trustme>=1.2.1",
     "types-cachetools>=6.1.0.20250717",
     "build>=1.2.2.post1",
     "twine>=6.1.0",

tests/e2e/configuration/library-mode/lightspeed-stack-tls.yaml

@@ -0,0 +1,21 @@
+name: Lightspeed Core Service (LCS)
+service:
+  host: 0.0.0.0
+  port: 8080
+  auth_enabled: false
+  workers: 1
+  color_log: true
+  access_log: true
+llama_stack:
+  use_as_library_client: true
+  library_client_config_path: run.yaml
+user_data_collection:
+  feedback_enabled: true
+  feedback_storage: "/tmp/data/feedback"
+  transcripts_enabled: true
+  transcripts_storage: "/tmp/data/transcripts"
+authentication:
+  module: "noop"
+inference:
+  default_provider: tls-openai
+  default_model: mock-tls-model

tests/e2e/configuration/server-mode/lightspeed-stack-tls.yaml

@@ -0,0 +1,22 @@
+name: Lightspeed Core Service (LCS)
+service:
+  host: 0.0.0.0
+  port: 8080
+  auth_enabled: false
+  workers: 1
+  color_log: true
+  access_log: true
+llama_stack:
+  use_as_library_client: false
+  url: http://llama-stack:8321
+  api_key: xyzzy
+user_data_collection:
+  feedback_enabled: true
+  feedback_storage: "/tmp/data/feedback"
+  transcripts_enabled: true
+  transcripts_storage: "/tmp/data/transcripts"
+authentication:
+  module: "noop"
+inference:
+  default_provider: tls-openai
+  default_model: mock-tls-model

tests/e2e/features/environment.py

@@ -157,6 +157,11 @@ def before_all(context: Context) -> None:
     context.deployment_mode = os.getenv("E2E_DEPLOYMENT_MODE", "server").lower()
     context.is_library_mode = context.deployment_mode == "library"
 
+    # Detect Docker mode once for proxy tests
+    from tests.e2e.features.steps.proxy import _is_docker_mode
+
+    context.is_docker_mode = _is_docker_mode()
+
     # Get first LLM model from running service
     print(f"Running tests in {context.deployment_mode} mode")
 
@@ -499,6 +504,14 @@ def before_feature(context: Context, feature: Feature) -> None:
         switch_config(context.feature_config)
         restart_container("lightspeed-stack")
 
+    if "TLS" in feature.tags:
+        mode_dir = "library-mode" if context.is_library_mode else "server-mode"
+        context.feature_config = (
+            f"tests/e2e/configuration/{mode_dir}/lightspeed-stack-tls.yaml"
+        )
+        context.default_config_backup = create_config_backup("lightspeed-stack.yaml")
+        switch_config(context.feature_config)
+
 
 def after_feature(context: Context, feature: Feature) -> None:
     """Run after each feature file is exercised.
@@ -546,3 +559,17 @@ def after_feature(context: Context, feature: Feature) -> None:
         switch_config(context.default_config_backup)
         restart_container("lightspeed-stack")
         remove_config_backup(context.default_config_backup)
+
+    if "TLS" in feature.tags:
+        switch_config(context.default_config_backup)
+        remove_config_backup(context.default_config_backup)
+        if not context.is_library_mode:
+            restart_container("llama-stack")
+        restart_container("lightspeed-stack")
+
+    # Clean up any proxy servers left from the last scenario
+    if hasattr(context, "tunnel_proxy") or hasattr(context, "interception_proxy"):
+        from tests.e2e.features.steps.proxy import _stop_proxy
+
+        _stop_proxy(context, "tunnel_proxy", "proxy_loop")
+        _stop_proxy(context, "interception_proxy", "interception_proxy_loop")

tests/e2e/features/proxy.feature

@@ -0,0 +1,106 @@
+@skip-in-library-mode
+Feature: Proxy and TLS networking tests for Llama Stack providers
+
+  Verify that the Lightspeed Stack works correctly when Llama Stack's
+  remote inference providers are configured with proxy and TLS settings
+  via the run.yaml NetworkConfig.
+
+  Background:
+    Given The service is started locally
+      And REST API service prefix is /v1
+      And The original Llama Stack config is restored if modified
+
+
+  # --- AC1: Tunnel proxy routing ---
+
+  @TunnelProxy
+  Scenario: LLM traffic is routed through a configured tunnel proxy
+    Given A tunnel proxy is running on port 8888
+      And Llama Stack is configured to route inference through the tunnel proxy
+      And Llama Stack is restarted
+      And Lightspeed Stack is restarted
+     When I use "query" to ask question
+    """
+    {"query": "What is 2+2?", "model": "{MODEL}", "provider": "{PROVIDER}"}
+    """
+     Then The status code of the response is 200
+      And The tunnel proxy handled at least 1 CONNECT request to the LLM provider
+
+  # NOTE: no_proxy is defined on Llama Stack's ProxyConfig model but not
+  # implemented in _build_proxy_mounts (http_client.py). The field is ignored.
+  # When Llama Stack implements no_proxy support, add a test here.
+
+  @TunnelProxy
+  Scenario: LLM query fails gracefully when proxy is unreachable
+    Given Llama Stack is configured to route inference through proxy "http://127.0.0.1:19999"
+      And Llama Stack is restarted
+      And Lightspeed Stack is restarted
+     When I use "query" to ask question
+    """
+    {"query": "What is 2+2?", "model": "{MODEL}", "provider": "{PROVIDER}"}
+    """
+     Then The status code of the response is 500
+
+
+  # --- AC2: Interception proxy with CA certificate ---
+
+  @InterceptionProxy
+  Scenario: LLM traffic works through interception proxy with correct CA
+    Given An interception proxy with trustme CA is running on port 8889
+      And Llama Stack is configured to route inference through the interception proxy with CA cert
+      And Llama Stack is restarted
+      And Lightspeed Stack is restarted
+     When I use "query" to ask question
+    """
+    {"query": "What is 2+2?", "model": "{MODEL}", "provider": "{PROVIDER}"}
+    """
+     Then The status code of the response is 200
+      And The interception proxy intercepted at least 1 connection
+
+  @InterceptionProxy
+  Scenario: LLM query fails when interception proxy CA is not provided
+    Given An interception proxy with trustme CA is running on port 8890
+      And Llama Stack is configured to route inference through the interception proxy without CA cert
+      And Llama Stack is restarted
+      And Lightspeed Stack is restarted
+     When I use "query" to ask question
+    """
+    {"query": "What is 2+2?", "model": "{MODEL}", "provider": "{PROVIDER}"}
+    """
+     Then The status code of the response is 500
+
+
+  # --- AC3: TLS version and cipher configuration ---
+
+  @TLSVersion
+  Scenario: TLS minimum version TLSv1.2 is respected
+    Given Llama Stack is configured with minimum TLS version "TLSv1.2"
+      And Llama Stack is restarted
+      And Lightspeed Stack is restarted
+     When I use "query" to ask question
+    """
+    {"query": "What is 2+2?", "model": "{MODEL}", "provider": "{PROVIDER}"}
+    """
+     Then The status code of the response is 200
+
+  @TLSVersion
+  Scenario: TLS minimum version TLSv1.3 is respected
+    Given Llama Stack is configured with minimum TLS version "TLSv1.3"
+      And Llama Stack is restarted
+      And Lightspeed Stack is restarted
+     When I use "query" to ask question
+    """
+    {"query": "What is 2+2?", "model": "{MODEL}", "provider": "{PROVIDER}"}
+    """
+     Then The status code of the response is 200
+
+  @TLSCipher
+  Scenario: Custom cipher suite configuration is respected
+    Given Llama Stack is configured with ciphers "ECDHE+AESGCM:DHE+AESGCM"
+      And Llama Stack is restarted
+      And Lightspeed Stack is restarted
+     When I use "query" to ask question
+    """
+    {"query": "What is 2+2?", "model": "{MODEL}", "provider": "{PROVIDER}"}
+    """
+     Then The status code of the response is 200