Skip to content

running-acceptance-tests.md

Latest commit

 

History

History
275 lines (205 loc) · 11.2 KB

running-acceptance-tests.md

File metadata and controls

275 lines (205 loc) · 11.2 KB

Running Routing Acceptance Tests (RATS)

This guide explains how to run the routing acceptance tests locally against a CF-on-Kubernetes deployment using kind.

Overview

The script start-local-rats.sh automates the full test lifecycle:

  1. Resolves the TCP router IP from the running cfk8s-worker Docker container
  2. Ensures required CF buildpacks and shared domains are present
  3. Runs HTTP and TCP routing acceptance tests via Ginkgo
  4. Cleans up shared domains (and optionally test orgs) after the run

Test Results Summary

With the correct configuration, you should see:

HTTP Routing Tests
- 1 spec skipped (IncludeHttpRoutes=false by default)

TCP Routing Tests  
- 6 specs passed in ~8-9 minutes
  ✓ single external port to single app port
  ✓ single external port to two different apps
  ✓ multiple external ports to single app port
  ✓ multiple external ports to single app port with two apps
  ✓ unmap and remap routes
  ✓ multiple apps sharing same external port

Acceptance Tests Complete; exit status: 0

Prerequisites

System Tools

Tool Minimum Version Purpose
docker 20+ Inspects the cfk8s-worker kind node to resolve its IP
kind 0.31+ Provides the local Kubernetes cluster running CF
kubectl 1.35.1+ Interacts with the kind cluster
jq 1.6+ Parses and patches the test config JSON
cf CLI v7+ Manages CF resources (orgs, spaces, routes, domains)
ginkgo v2+ Runs the Go-based acceptance test suites
make any Bootstraps CF buildpacks if not yet uploaded
go 1.21+ Required to build the rtr binary and compile test assets
kind-deployment any Required to run the CF on Kind setup. Please checkout the following documentation

Install ginkgo:

go install github.com/onsi/ginkgo/v2/ginkgo@latest

Linux Kernel Parameters

Running CF app containers inside Docker requires sufficient inotify limits on the host. Without this, the Envoy sidecar proxy will crash with exit code 134 (SIGABRT).

Check current values:

sudo sysctl fs.inotify.max_user_instances fs.inotify.max_user_watches

Apply immediately (no reboot needed):

sudo sysctl -w fs.inotify.max_user_instances=512
sudo sysctl -w fs.inotify.max_user_watches=524288

Make permanent:

echo -e "fs.inotify.max_user_instances=512\nfs.inotify.max_user_watches=524288" \
  | sudo tee /etc/sysctl.d/99-inotify.conf

Running CF-on-kind Cluster

A running CF-on-kind deployment is required. Use the kind-deployment project to set it up. The script expects a Docker container named cfk8s-worker to be present and running.

Verify:

docker inspect cfk8s-worker --format '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}'

rtr Binary

The rtr (Routing API CLI) binary must be built before running the tests:

cd src/code.cloudfoundry.org/routing-api-cli
go build -o rtr .

The script automatically prepends the rtr binary to $PATH.

Configuration

The tests are configured via a JSON file. A template is provided at config.json.

Config Fields

Field Description Example
addresses IPs of the TCP router (auto-set by the script from Docker inspect) ["172.21.0.3"]
api CF API endpoint "api.127-0-0-1.nip.io"
admin_user CF admin username "ccadmin"
admin_password CF admin password "secret"
apps_domain Default HTTP apps domain "apps.127-0-0-1.nip.io"
tcp_apps_domain Shared TCP routing domain "tcp.127-0-0-1.nip.io"
tcp_router_group Router group name for TCP routes "default-tcp"
skip_ssl_validation Skip TLS certificate verification true
include_http_routes Enable HTTP route tests false
default_timeout Default test timeout in seconds 120
cf_push_timeout Timeout for cf push in seconds 120
oauth.token_endpoint UAA token endpoint URL "https://uaa.127-0-0-1.nip.io"
oauth.client_name UAA client for routing API auth "tcp_emitter"
oauth.client_secret UAA client secret "secret"
oauth.port UAA port 443

Note

The addresses field is automatically updated by the script on each run — you do not need to set it manually. You can find the admin_password and the oauth.client_secret value in the secrets file of the kind-deployment/temp folder.

Verified Working Configuration

A tested and working config.json could look like this. Make sure to replace <secret> with the actual values (admin_password and oauth.client_secret) from the kind-deployment/temp folder:

{
  "addresses": ["172.21.0.3"],
  "api": "api.127-0-0-1.nip.io",
  "admin_user": "ccadmin",
  "admin_password": "<secret>",
  "skip_ssl_validation": true,
  "use_http": false,
  "apps_domain": "apps.127-0-0-1.nip.io",
  "include_http_routes": false,
  "default_timeout": 120,
  "cf_push_timeout": 120,
  "tcp_apps_domain": "tcp.127-0-0-1.nip.io",
  "tcp_router_group": "default-tcp",
  "oauth": {
    "token_endpoint": "https://uaa.127-0-0-1.nip.io",
    "client_name": "tcp_emitter",
    "client_secret": "<secret>",
    "port": 443,
    "skip_ssl_validation": true
  }
}

Key points:

  • include_http_routes: false - HTTP tests are skipped by default
  • TCP tests require 6 specs × ~8-9 seconds per spec = ~8 minutes total
  • The addresses array contains only one IP (the Docker bridge IP of cfk8s-worker)

Required Environment Variables

Variable Description Example
ACCEPTANCE_DIR Absolute path to the routing-acceptance-tests directory /home/user/routing-release/src/code.cloudfoundry.org/routing-acceptance-tests
KIND_DEPLOYMENT_DIR Absolute path to the kind-deployment repository /home/user/kind-deployment
CONFIG Absolute path to the test config JSON file /home/user/routing-acceptance-tests/config.json
RTR_BIN Absolute path to the directory containing the rtr binary /home/user/routing-release/src/code.cloudfoundry.org/rtr

Optional Environment Variables

Variable Description Example
VERBOSE When set to any value, enables verbose output for ginkgo tests VERBOSE=1 or VERBOSE=true
CLEANUP_CATS_ORGS When true, deletes all CF orgs matching CATS-* after the run CLEANUP_CATS_ORGS=true

Basic Usage

export ACCEPTANCE_DIR=/home/user/routing-release/src/code.cloudfoundry.org/routing-acceptance-tests
export KIND_DEPLOYMENT_DIR=/home/user/kind-deployment
export CONFIG=$ACCEPTANCE_DIR/config.json
export RTR_BIN=/home/user/routing-release/src/code.cloudfoundry.org/rtr

bash ./start-local-rats.sh

TCP Router Port Configuration

The CF TCP routing domain requires available ports in the default-tcp router group. The kind cluster worker node exposes ports 32000–32019 for TCP routing.

Verify the current port range:

cf curl /routing/v1/router_groups

If the range is too small and tests fail with "There are no more ports available for this domain", expand it:

ROUTER_GROUP_GUID=$(cf curl /routing/v1/router_groups | jq -r '.[0].guid')
cf curl /routing/v1/router_groups/$ROUTER_GROUP_GUID \
  -X PUT -d '{"reservable_ports":"32000-32019"}'

Troubleshooting

HTTP Tests Skipped: IncludeHttpRoutes is set to false

This is expected behavior when include_http_routes: false in config.json:

[SKIPPED] Skipping this test because Config.IncludeHttpRoutes is set to `false`.

The HTTP routing layer is stable; most development focuses on TCP routing. To enable:

"include_http_routes": true

App Logs Show Error on connection read: EOF

This is normal and happens during TCP route cleanup:

[APP/PROC/WEB/0] OUT Error on connection read: EOF

The test closes connections after verifying routing works. The app logs these as connection resets, which is expected behavior for connection-based protocol testing.

BuildPack Staging Fails: Error staging application: StagingError

The go_buildpack may be missing or not ready. Verify and bootstrap:

cf buildpacks | grep go_buildpack || make -C $KIND_DEPLOYMENT_DIR bootstrap

Then retry the tests.

Domain not found on cf create-route

The TCP shared domain (tcp_apps_domain) does not exist in CF yet. The script creates it automatically, but you can also do it manually:

cf create-shared-domain tcp.127-0-0-1.nip.io --router-group default-tcp

There are no more ports available for this domain

The TCP router group has exhausted its port pool. See TCP Router Port Configuration above.

dial tcp <ip>:<port>: i/o timeout

The test could not reach the TCP router. Verify:

  1. The cfk8s-worker container is running: docker ps | grep cfk8s-worker
  2. The Istio Gateway and NodePort service include the port in use: 
    kubectl get svc istio-gateway-istio -n default \
  -o jsonpath='{range .spec.ports[*]}{.port}{"\n"}{end}' | grep "^320"
  3. The addresses field in config.json matches the actual Docker network IP of cfk8s-worker.

Envoy crash / Exit status 134 on app start

The host's inotify limits are too low. See Linux Kernel Parameters above.