test(infra): port the billing usage collector into the env

CI now runs test/e2e (incl. test/e2e/billing) on the prod-fidelity env,
but #239 wired the Vector collector into the retired bridge's
prepare-infra-cluster. Install it from the env instead, and point its
kubernetes_logs source at the env's Envoy namespace
(datum-downstream-gateway, not the bridge's envoy-gateway-system).

Also add workflow_dispatch to the e2e workflow so the run can be
triggered directly instead of only via the gated pull_request event.

Author: ecv

.github/workflows/test-e2e.yml

@@ -7,6 +7,7 @@ on:
   pull_request:
     branches:
       - main
+  workflow_dispatch:
 
 jobs:
   test-e2e:

Taskfile.test-infra.yml

@@ -74,6 +74,7 @@ tasks:
       - task: prepare-upstream
       - task: eg-downstream
       - task: extension-server
+      - task: billing-usage-collector
       - task: external-dns
       - task: link-clusters
       - task: wait-ready
@@ -168,6 +169,12 @@ tasks:
       # be in the manifest, not added afterward.
       - kubectl --context {{.DOWNSTREAM_CTX}} apply -f config/e2e-downstream/direct-namespace.yaml
 
+  billing-usage-collector:
+    desc: "Install the billing usage collector (Vector agent) that tails the downstream Envoy access logs and emits usage CloudEvents."
+    cmds:
+      - echo "🔧 installing billing-usage-collector (Vector agent, tailing datum-downstream-gateway)"
+      - '{{.KUSTOMIZE}} build --enable-helm config/tools/billing-usage-collector | kubectl --context {{.DOWNSTREAM_CTX}} apply --server-side=true --force-conflicts -f -'
+
   eg-downstream:
     desc: "Install the dedicated downstream Envoy Gateway (v1.7.4) with the ext-server extensionManager wiring on the downstream cluster."
     cmds:

config/tools/billing-usage-collector/kustomization.yaml

@@ -50,7 +50,7 @@ helmCharts:
             # Only tail the Envoy proxy pods; their access logs live on the
             # `envoy` container's stdout. Other namespaces/containers are
             # ignored so the agent does not parse unrelated cluster logs.
-            extra_field_selector: "metadata.namespace=envoy-gateway-system"
+            extra_field_selector: "metadata.namespace=datum-downstream-gateway"
             # Re-scan for new container log files every 2s instead of the 60s
             # default so the agent picks up freshly-created Envoy proxy pods
             # quickly -- the e2e test provisions a gateway and verifies within a