Skip to content

KCP e2e test #333

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
mjudeikis merged 14 commits into from
Oct 6, 2025
Merged

KCP e2e test #333

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
c4a91be
Bump kcp-dev/multicluster-provider to latest main
ntnn Oct 6, 2025
e9d6e52
Return generated id and secret in CreateDexClient
ntnn Oct 6, 2025
3ce60b8
Add cookie encryption key for backend
ntnn Oct 6, 2025
c1581e0
Add framework.ApplyFiles and .ApplyManifests
ntnn Oct 5, 2025
3b94015
Update framework.StartBackend and split CRD installation
ntnn Oct 6, 2025
e194c60
Add logging config for controller-runtime
ntnn Oct 5, 2025
a1c91ca
Add dex as dependency for kcp contrib tests
ntnn Oct 2, 2025
31b6270
Add WriteKubeconfig
ntnn Oct 1, 2025
84c95c0
Bump kcp to v0.28.3
ntnn Oct 1, 2025
c2fbf5e
Pin kcp to a functional sdk/testing commit
ntnn Sep 22, 2025
10e1c46
Update kcp readme
ntnn Oct 1, 2025
4e46da2
Add kcp e2e test
ntnn Oct 1, 2025
b7ecb5e
Just ignore dex on non-linux
ntnn Oct 6, 2025
72122c6
Drop dependency on build step for contrib kcp e2e tests
ntnn Oct 6, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -77,7 +77,7 @@ CODE_GENERATOR_BIN := code-generator
CODE_GENERATOR := $(TOOLS_GOBIN_DIR)/$(CODE_GENERATOR_BIN)-$(CODE_GENERATOR_VER)
export CODE_GENERATOR # so hack scripts can use it

KCP_VER := v0.28.0
KCP_VER := v0.28.3
KCP_BIN := kcp
KCP := $(TOOLS_GOBIN_DIR)/$(KCP_BIN)-$(KCP_VER)
KCP_CMD ?= $(KCP)
Expand Down Expand Up @@ -286,7 +286,7 @@ CONTRIBS_E2E := $(patsubst %,test-e2e-contrib-%,$(CONTRIBS))

.PHONY: test-e2e-contribs $(CONTRIBS_E2E)
test-e2e-contribs: $(CONTRIBS_E2E) ## Run e2e tests for external integrations
test-e2e-contrib-kcp: build $(KCP)
test-e2e-contrib-kcp: $(DEX) $(KCP)
$(CONTRIBS_E2E):
cd contrib/$(patsubst test-e2e-contrib-%,%,$@) && $(GO_TEST) -race -count $(COUNT) -p $(E2E_PARALLELISM) -parallel $(E2E_PARALLELISM) ./test/e2e/...

Expand Down
86 changes: 70 additions & 16 deletions contrib/kcp/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,9 +23,23 @@ It will do the following:

# How to run

## Preparation

1. Start dex

```bash
make run-dex
```

2. Start kcp
3. Bootstrap kcp:

```bash
make run-kcp
```

## Backend

2. Bootstrap kcp:
```bash
cp .kcp/admin.kubeconfig .kcp/backend.kubeconfig
export KUBECONFIG=.kcp/backend.kubeconfig
Expand All @@ -49,18 +63,22 @@ k ws use :root:kube-bind
--schema-source apiresourceschemas
```

This process will keep running, so open a new terminal.

## Provider

5. Copy the kubeconfig to the provider and create provider workspace:
```bash
cp .kcp/admin.kubeconfig .kcp/provider.kubeconfig
export KUBECONFIG=.kcp/provider.kubeconfig
k ws use :root
kubectl ws create provider --enter
kubectl create-workspace provider --enter
```

6. Bind the APIExport to the provider workspace
```bash
kubectl kcp bind apiexport root:kube-bind:kube-bind.io --accept-permission-claim clusterrolebindings.rbac.authorization.k8s.io \
kubectl kcp bind apiexport root:kube-bind:kube-bind.io \
--accept-permission-claim clusterrolebindings.rbac.authorization.k8s.io \
--accept-permission-claim clusterroles.rbac.authorization.k8s.io \
--accept-permission-claim customresourcedefinitions.apiextensions.k8s.io \
--accept-permission-claim serviceaccounts.core \
Expand All @@ -77,7 +95,6 @@ kubectl kcp bind apiexport root:kube-bind:kube-bind.io --accept-permission-claim
kubectl create -f contrib/kcp/deploy/examples/apiexport.yaml
kubectl create -f contrib/kcp/deploy/examples/apiresourceschema-cowboys.yaml
kubectl create -f contrib/kcp/deploy/examples/apiresourceschema-sheriffs.yaml
# recursive bind
kubectl kcp bind apiexport root:provider:cowboys-stable
```

Expand All @@ -86,9 +103,11 @@ kubectl kcp bind apiexport root:provider:cowboys-stable
```bash
kubectl get logicalcluster
# NAME PHASE URL AGE
# cluster Ready https://192.168.2.166:6443/clusters/1d5vpxvdpy0opbj1
# cluster Ready https://192.168.2.166:6443/clusters/1d5vpxvdpy0opbj1
```

## Consumer

9. Now we gonna initiate consumer:
```bash
cp .kcp/admin.kubeconfig .kcp/consumer.kubeconfig
Expand All @@ -104,15 +123,24 @@ kubectl ws create consumer --enter

# Extract secret for binding process. Note that secret name is not the same as output from command above. Check secret
# name by running `kubectl get secret -n kube-bind`
kubectl get secret kubeconfig-hxwlc -n kube-bind -o jsonpath='{.data.kubeconfig}' | base64 -d > remote.kubeconfig
kubectl get secrets -n kube-bind -o jsonpath='{.items[0].data.kubeconfig}' | base64 -d > remote.kubeconfig

Comment thread
ntnn marked this conversation as resolved.
./bin/kubectl-bind apiservice --remote-kubeconfig remote.kubeconfig -f contrib/kcp/deploy/examples/apiserviceexport-namespaced.yaml --skip-konnector --remote-namespace kube-bind-697cb
namespace=$(yq '.contexts[0].context.namespace' remote.kubeconfig)

export KUBECONFIG=.kcp/consumer.kubeconfig
go run ./cmd/konnector/ --lease-namespace default
./bin/kubectl-bind apiservice -v 6 --remote-kubeconfig remote.kubeconfig -f apiserviceexport.yaml --skip-konnector --remote-namespace "$namespace"
```

This will keep running, so switch to a new terminal.

### Consumer Konnector

11. (Optional) Add second consumer to test
Start konnector:

```bash
./bin/konnector --lease-namespace default --kubeconfig .kcp/consumer.kubeconfig
```

Optionally add second consumer to test

```bash
cp .kcp/admin.kubeconfig .kcp/consumer2.kubeconfig
Expand All @@ -121,27 +149,53 @@ kubectl ws use :root
kubectl ws create consumer2 --enter

./bin/kubectl-bind http://127.0.0.1:8080/clusters/2vgrh380y0cq38du/exports --dry-run -o yaml > apiserviceexport2.yaml
kubectl get secret kubeconfig-wvvsb -n kube-bind -o jsonpath='{.data.kubeconfig}' | base64 -d > remote2.kubeconfig

./bin/kubectl-bind apiservice --remote-kubeconfig remote2.kubeconfig -f apiserviceexport.yaml --skip-konnector --remote-namespace kube-bind-m5zx4
kubectl get secrets -n kube-bind -o jsonpath='{.items[0].data.kubeconfig}' | base64 -d > remote2.kubeconfig

./bin/kubectl-bind apiservice -v 6 --remote-kubeconfig remote2.kubeconfig -f apiserviceexport2.yaml --skip-konnector --remote-namespace "$(yq '.contexts[0].context.namespace' remote2.kubeconfig)"

export KUBECONFIG=.kcp/consumer2.kubeconfig
go run ./cmd/konnector/ --lease-namespace default --server-address :8091
./bin/konnector --lease-namespace default --kubeconfig .kcp/consumer2.kubeconfig --server-address :8091
```

This will keep running, so switch to a new terminal.

## Testing

Create objects:
```
Comment thread
ntnn marked this conversation as resolved.
export KUBECONFIG=.kcp/consumer.kubeconfig
kubectl apply -f contrib/kcp/deploy/examples/cowboy.yaml
kubectl apply -f contrib/kcp/deploy/examples/sheriff.yaml
```
Comment thread
ntnn marked this conversation as resolved.


## Debug

```bash

cp .kcp/admin.kubeconfig .kcp/debug.kubeconfig
export KUBECONFIG=.kcp/debug.kubeconfig
k ws use :root:kube-bind

k -s "$(kubectl get apiexportendpointslice kube-bind.io -o jsonpath="{.status.endpoints[0].url}")/clusters/*" api-resources
k -s "$(kubectl get apiexportendpointslice kube-bind.io -o jsonpath="{.status.endpoints[0].url}")/clusters/*" get crd
k -s "$(kubectl get apiexportendpointslice kube-bind.io -o jsonpath='{.status.endpoints[0].url}')/clusters/*" api-resources

k -s "$(kubectl get apiexportendpointslice kube-bind.io -o jsonpath='{.status.endpoints[0].url}')/clusters/*" get crd

namespace=$(yq '.contexts[0].context.namespace' remote.kubeconfig)

# some claimed objects

kubectl create cm provider -n "$namespace-default"
kubectl label cm provider app=wildwest -n "$namespace-default"

Comment thread
ntnn marked this conversation as resolved.
kubectl create cm consumer -n default
kubectl label cm consumer app=wildwest -n default

kubectl create secret generic provider-secret
kubectl label secret provider-secret app=wildwest

kubectl create namespace bob
kubectl create secret generic wildwest-secrets1 -n bob
kubectl label secret wildwest-secrets1 app=wildwest -n bob

```
48 changes: 46 additions & 2 deletions contrib/kcp/go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,37 +8,58 @@ replace (
github.com/kube-bind/kube-bind/sdk => ../../sdk
)

// kcp pinned to a commit on main as sdk/testing requires
// features not in the latest release
// Can use versioned when v0.28.2 releases
replace github.com/kcp-dev/kcp/sdk => github.com/kcp-dev/kcp/sdk v0.28.1-0.20251003164010-742ce0ea6b8c

require (
github.com/headzoo/surf v1.0.1
github.com/kcp-dev/client-go v0.0.0-20250728134101-0355faa9361b
github.com/kcp-dev/kcp v0.28.1
github.com/kcp-dev/kcp v0.28.3
github.com/kcp-dev/kcp/sdk v0.28.1
github.com/kcp-dev/logicalcluster/v3 v3.0.5
github.com/kube-bind/kube-bind v0.0.0-00010101000000-000000000000
github.com/kube-bind/kube-bind/sdk v0.4.1
github.com/spf13/pflag v1.0.7
github.com/stretchr/testify v1.10.0
k8s.io/apiextensions-apiserver v0.33.3
Comment thread
ntnn marked this conversation as resolved.
k8s.io/apimachinery v0.33.3
k8s.io/apiserver v0.33.3
k8s.io/cli-runtime v0.32.0
k8s.io/client-go v0.33.3
k8s.io/component-base v0.33.3
k8s.io/klog/v2 v2.130.1
sigs.k8s.io/yaml v1.4.0
)

require (
cel.dev/expr v0.19.1 // indirect
github.com/Azure/go-ansiterm v0.0.0-20230124172434-306776ec8161 // indirect
github.com/NYTimes/gziphandler v1.1.1 // indirect
github.com/PuerkitoBio/goquery v1.8.0 // indirect
github.com/andybalholm/cascadia v1.3.1 // indirect
github.com/antlr4-go/antlr/v4 v4.13.1 // indirect
github.com/beorn7/perks v1.0.1 // indirect
github.com/blang/semver/v4 v4.0.0 // indirect
github.com/cenkalti/backoff/v4 v4.3.0 // indirect
github.com/cespare/xxhash/v2 v2.3.0 // indirect
github.com/coreos/go-oidc/v3 v3.15.0 // indirect
github.com/coreos/go-semver v0.3.1 // indirect
github.com/coreos/go-systemd/v22 v22.5.0 // indirect
github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc // indirect
github.com/dexidp/dex/api/v2 v2.3.0 // indirect
github.com/egymgmbh/go-prefix-writer v0.0.0-20180609083313-7326ea162eca // indirect
github.com/emicklei/go-restful/v3 v3.12.1 // indirect
github.com/evanphx/json-patch/v5 v5.9.11 // indirect
github.com/felixge/httpsnoop v1.0.4 // indirect
github.com/fsnotify/fsnotify v1.7.0 // indirect
github.com/fxamacker/cbor/v2 v2.7.0 // indirect
github.com/go-errors/errors v1.4.2 // indirect
github.com/go-jose/go-jose/v4 v4.0.5 // indirect
github.com/go-logr/logr v1.4.2 // indirect
github.com/go-logr/stdr v1.2.2 // indirect
github.com/go-logr/zapr v1.3.0 // indirect
github.com/go-openapi/jsonpointer v0.21.0 // indirect
github.com/go-openapi/jsonreference v0.21.0 // indirect
github.com/go-openapi/swag v0.23.0 // indirect
Expand All @@ -48,27 +69,44 @@ require (
github.com/google/cel-go v0.23.2 // indirect
github.com/google/gnostic-models v0.6.9 // indirect
github.com/google/go-cmp v0.7.0 // indirect
github.com/google/shlex v0.0.0-20191202100458-e7afc7fbc510 // indirect
github.com/google/uuid v1.6.0 // indirect
github.com/gorilla/mux v1.8.0 // indirect
github.com/gorilla/securecookie v1.1.1 // indirect
github.com/gregjones/httpcache v0.0.0-20190611155906-901d90724c79 // indirect
github.com/grpc-ecosystem/go-grpc-prometheus v1.2.0 // indirect
github.com/grpc-ecosystem/grpc-gateway/v2 v2.25.1 // indirect
github.com/inconshreveable/mousetrap v1.1.0 // indirect
github.com/josharian/intern v1.0.0 // indirect
github.com/json-iterator/go v1.1.12 // indirect
github.com/kcp-dev/apimachinery/v2 v2.0.1-0.20250728122101-adbf20db3e51 // indirect
github.com/kcp-dev/kcp/pkg/apis v0.11.0 // indirect
github.com/kcp-dev/multicluster-provider v0.2.1-0.20251002133408-9a8d21dc2872 // indirect
github.com/kube-bind/kube-bind/cli v0.0.0-20250515145715-d9f20e7c840d // indirect
github.com/kylelemons/godebug v1.1.0 // indirect
github.com/liggitt/tabwriter v0.0.0-20181228230101-89fcab3d43de // indirect
github.com/mailru/easyjson v0.9.0 // indirect
github.com/martinlindhe/base36 v1.1.1 // indirect
github.com/mdp/qrterminal/v3 v3.2.0 // indirect
github.com/moby/term v0.5.0 // indirect
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect
github.com/modern-go/reflect2 v1.0.2 // indirect
github.com/monochromegane/go-gitignore v0.0.0-20200626010858-205db1a8cc00 // indirect
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 // indirect
github.com/onsi/gomega v1.36.2 // indirect
github.com/peterbourgon/diskv v2.0.1+incompatible // indirect
github.com/pkg/errors v0.9.1 // indirect
github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 // indirect
github.com/prometheus/client_golang v1.22.0 // indirect
github.com/prometheus/client_model v0.6.1 // indirect
github.com/prometheus/common v0.62.0 // indirect
github.com/prometheus/procfs v0.15.1 // indirect
github.com/spf13/cobra v1.9.1 // indirect
github.com/stoewer/go-strcase v1.3.0 // indirect
github.com/vmihailenco/msgpack/v4 v4.3.12 // indirect
github.com/vmihailenco/tagparser v0.1.1 // indirect
github.com/x448/float16 v0.8.4 // indirect
github.com/xlab/treeprint v1.2.0 // indirect
go.etcd.io/etcd/api/v3 v3.5.21 // indirect
go.etcd.io/etcd/client/pkg/v3 v3.5.21 // indirect
go.etcd.io/etcd/client/v3 v3.5.21 // indirect
Expand All @@ -93,6 +131,8 @@ require (
golang.org/x/term v0.32.0 // indirect
golang.org/x/text v0.25.0 // indirect
golang.org/x/time v0.11.0 // indirect
gomodules.xyz/jsonpatch/v2 v2.4.0 // indirect
google.golang.org/appengine v1.6.7 // indirect
google.golang.org/genproto/googleapis/api v0.0.0-20250106144421-5f5ef82da422 // indirect
google.golang.org/genproto/googleapis/rpc v0.0.0-20250106144421-5f5ef82da422 // indirect
google.golang.org/grpc v1.70.0 // indirect
Expand All @@ -103,11 +143,15 @@ require (
k8s.io/api v0.33.3 // indirect
k8s.io/kube-openapi v0.0.0-20250318190949-c8a335a9a2ff // indirect
k8s.io/utils v0.0.0-20250604170112-4c0f3b243397 // indirect
rsc.io/qr v0.2.0 // indirect
sigs.k8s.io/apiserver-network-proxy/konnectivity-client v0.31.2 // indirect
sigs.k8s.io/controller-runtime v0.21.0 // indirect
sigs.k8s.io/json v0.0.0-20241014173422-cfa47c3a1cc8 // indirect
sigs.k8s.io/kustomize/api v0.19.0 // indirect
sigs.k8s.io/kustomize/kyaml v0.19.0 // indirect
sigs.k8s.io/multicluster-runtime v0.21.0-alpha.9.0.20251002124257-36facc7fbe82 // indirect
sigs.k8s.io/randfill v1.0.0 // indirect
sigs.k8s.io/structured-merge-diff/v4 v4.6.0 // indirect
sigs.k8s.io/yaml v1.4.0 // indirect
)

replace (
Expand Down
Loading