You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
chore: analytics in every install path + drop test-deploy-run + doc cleanup (#396)
## Why
User direction: "I always want clickhouse etc to work in all targets".
Post-#391/#395 audit found three install paths missing analytics + one
path that can't ever support it:
| Path | Status before | Status after |
|---|---|---|
| `make run` / Compose Quick start | ✅ analytics | ✅ |
| `make deploy` / `make deploy-release` (k3d) | ✅ analytics (after #395)
| ✅ |
| `make test-deploy-{dev,compose,oobe}` | ✅ (base compose + override) |
✅ |
| `make run-ghcr` | ❌ go-server only | ✅ |
| `make test-deploy-ghcr` | ❌ go-server only | ✅ |
| `make test-deploy-registry` | ❌ go-server only | ✅ |
| **`make test-deploy-run`** | ❌ bare docker run, can't have analytics |
**dropped** |
Plus `docs/DEPLOYMENT.md` referenced the now-deleted
`k8s-infinite-streaming{,-dev}.yaml` and still said "k3s".
## Changes
**Compose files — add analytics services:**
- `docker-compose.ghcr.yml` — adds `clickhouse`, `forwarder`, `grafana`.
Forwarder pulls from
`ghcr.io/jonathaneoliver/infinite-streaming-forwarder:latest`.
- `tests/deploy/docker-compose.registry.yml` — same set. Forwarder pulls
from `${K3S_REGISTRY}/infinite-streaming-forwarder:dev`.
**GHCR workflow — publish the forwarder too:**
- `.github/workflows/docker-publish.yml` — extends the existing
`infinite-streaming` build to also build/push
`infinite-streaming-forwarder` from `./analytics/go-forwarder`. Same tag
pattern (`:latest`, `:main`, `:sha-…`, semver). Docker Hub mirror gated
on `vars.DOCKERHUB_NAMESPACE` the same way.
**Drops:**
- `make test-deploy-run` (bare `docker run`) — incompatible with the
analytics-everywhere invariant. Removed from the recipe, from
`test-deploy-all`, and from the `test-clean` cleanup list.
- README "Docker run (single container, no compose)" section — gone.
Macos/Docker-Desktop TC note kept (now under the section preface,
applies to all compose-based installs).
- Makefile stale defaults orphaned by #395: `K3S_KUBECONFIG`,
`K8S_MANIFESTS`, `K8S_DEPLOYMENT`.
**Doc updates (post-#395 reality):**
- `README.md` — "Other ways to run it" rewritten. GHCR snippet now also
fetches `analytics/` via tarball so the bind mounts in
`docker-compose.ghcr.yml` resolve. k3s pointer → k3d.
- `docs/DEPLOYMENT.md` — rewritten for two k3d clusters (api ports
6543/6544, kubeconfigs at `~/.config/k3d/`, `make k3d-bootstrap`, `make
teardown-{dev,release}`). GHCR-publishing section mentions both images.
- `docs/API.md`, `docs/ARCHITECTURE.md`, `docs/TROUBLESHOOTING.md`,
`analytics/README.md` — `s/k3s/k3d/` where the change is real (port
mapping, deployment modes, troubleshooting playbook, htpasswd runbook).
## Test plan
- [x] `docker compose -f docker-compose.ghcr.yml config` parses.
- [x] `docker compose -f tests/deploy/docker-compose.registry.yml
config` parses.
- [x] `make -n test-deploy-all` parses with no `test-deploy-run`
reference.
- [x] No remaining stale references: `grep k3s` across docs returns zero
hits in active prose (only the migration-warning sentence in
TROUBLESHOOTING).
- [ ] Live verification: `make run-ghcr` / `make
test-deploy-{ghcr,registry}` end-to-end against the test-dev host.
Deferred until the GHCR forwarder image is first published — that
requires this PR to merge so the workflow runs once on `main`. Note in
the PR body that the first `make run-ghcr` after merge will hit a 404 on
the forwarder image until the workflow completes (~3 min).
## Out of scope
- k3d-equivalent maintenance targets for `analytics-update` /
`analytics-rebuild-forwarder` / `analytics-migrate` (currently target
the test-dev compose path only). Worth a separate issue if needed.
- Real ClickHouse password hardening (unrelated, came up in #391's
CodeQL discussion).
Closes#394.
Copy file name to clipboardExpand all lines: README.md
+20-25Lines changed: 20 additions & 25 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -131,7 +131,7 @@ From there, **Playback** plays the same clip standalone, **Testing Session** ope
131
131
132
132
Skipping the seed and uploading your own files via **Open Upload** or by dropping MP4s into `$CONTENT_DIR/originals/` works identically — the seed clip is just a zero-friction "Hello, World" that exercises the full pipeline against content the server generates itself.
133
133
134
-
Other deployment options (pre-built images, single container, k3s) are in [Advanced deployment](#advanced-deployment) at the bottom.
134
+
Other deployment options (pre-built images from GHCR, k3d clusters) are in [Other ways to run it](#other-ways-to-run-it) at the bottom.
135
135
136
136
---
137
137
@@ -342,7 +342,7 @@ A sidecar stack (ClickHouse + Grafana + a small Go forwarder) auto-archives sess
342
342
343
343
**Operating it**: `make analytics-rebuild-forwarder` recreates the forwarder container in-place (live UI untouched); `make analytics-update` reloads Grafana provisioning; `make analytics-migrate SQL='ALTER TABLE …'` runs a schema change. The data is exposed read-only to the dashboard via parameterized ClickHouse queries — no string interpolation, no auth-token leakage.
344
344
345
-
**Securing for WAN deployment**: opt-in HTTP Basic auth via `INFINITE_STREAM_AUTH_HTPASSWD` gates the dashboard, `/analytics/api/`, and `/grafana/`; player-app endpoints stay public so unattended Apple/Roku/AndroidTV clients keep working. ClickHouse binds to `127.0.0.1` only by default. See [`analytics/README.md`](analytics/README.md) for the docker-compose and k3s runbooks.
345
+
**Securing for WAN deployment**: opt-in HTTP Basic auth via `INFINITE_STREAM_AUTH_HTPASSWD` gates the dashboard, `/analytics/api/`, and `/grafana/`; player-app endpoints stay public so unattended Apple/Roku/AndroidTV clients keep working. ClickHouse binds to `127.0.0.1` only by default. See [`analytics/README.md`](analytics/README.md) for the docker-compose and k3d runbooks.
346
346
347
347
The two pages downstream of this stack — **Sessions view** (the picker) and **Session Viewer** (replay one) — are described in their own sections below.
348
348
@@ -694,7 +694,7 @@ That's it — no third-party services beyond a Cloudflare account.
694
694
|`INFINITE_STREAM_RENDEZVOUS_URL`| Rendezvous Worker base URL. Required to enable any pairing. |
695
695
|`INFINITE_STREAM_ANNOUNCE_URL`| URL that clients should use to reach this server (e.g. `http://lenovo.local:30000`). When set, this server appears in same-WAN auto-discovery. |
696
696
|`INFINITE_STREAM_ANNOUNCE_LABEL`| Optional friendly label. Defaults to `host:port` from the announce URL. |
697
-
|`INFINITE_STREAM_SERVER_ID`| Optional explicit announce ID (4–64 chars `[A-Za-z0-9_-]`). Defaults to a stable random ID persisted at `<data_dir>/server_id`. Set this when multiple deployments share the same data directory (e.g. dev + release pods on the same k3s node), otherwise their announces overwrite each other on the rendezvous. |
697
+
|`INFINITE_STREAM_SERVER_ID`| Optional explicit announce ID (4–64 chars `[A-Za-z0-9_-]`). Defaults to a stable random ID persisted at `<data_dir>/server_id`. Set this when multiple deployments share the same data directory (e.g. dev + release pods on the same k3d host), otherwise their announces overwrite each other on the rendezvous. |
698
698
699
699
### HTTP, HTTPS, and iOS App Transport Security
700
700
@@ -709,47 +709,42 @@ The server defaults to plain HTTP on its dashboard / API / playback ports. That'
709
709
710
710
The iOS/tvOS Info.plist files in this repo include an explicit `NSExceptionDomains` entry for `infinitestreaming.jeoliver.com` (the upstream maintainer's public domain). **If you fork and ship apps that talk to a different public-HTTP hostname** (your own server, a Tailscale MagicDNS name like `*.ts.net`, a Tailscale CGNAT IP in `100.64.0.0/10`, etc.) you must add it to both Info.plists or those clients will silently fail to load anything.
711
711
712
-
The cleaner long-term answer is to terminate TLS at the server so all clients use HTTPS and no per-domain ATS / cleartext exceptions are needed. The k3s manifests already mount a `certs-vol` for this; flipping the nginx template to `listen … ssl` and pointing it at a Let's Encrypt cert (or whichever cert lives in `K3S_CERTS_DIR`) gets you there.
712
+
The cleaner long-term answer is to terminate TLS at the server so all clients use HTTPS and no per-domain ATS / cleartext exceptions are needed. The k3d manifests already mount a `certs-vol` for this; flipping the nginx template to `listen … ssl` and pointing it at a Let's Encrypt cert (or whichever cert lives in `K3S_CERTS_DIR`) gets you there.
713
713
714
714
---
715
715
716
716
## Other ways to run it
717
717
718
718
Most users should stick with Docker Compose from the [Quick start](#quick-start). These variants are for specific scenarios.
Ports 30181–30881 are the per-session proxy ports that testing sessions get redirected to. Without mapping them, `testing-session.html` works but segments never load because the allocated session port is unreachable from the host.
737
-
738
720
> **macOS / Docker Desktop note:** Network shaping (TC/nftables) works on Docker Desktop for Mac with `--cap-add NET_ADMIN`, but the TC stats polling (every 100ms per session) spawns processes through the Linux VM layer, which causes significantly higher CPU usage and fan noise compared to native Linux. This is a Docker Desktop VM overhead issue, not a code issue. For sustained testing with shaping, use a native Linux host.
739
721
740
722
### Pre-built images from GHCR (no source checkout)
741
723
724
+
The compose file pulls `infinite-streaming` and `infinite-streaming-forwarder` from GHCR. ClickHouse and Grafana provisioning files are tiny (~30 KB) so the install grabs them via a tarball alongside the compose file:
Open `http://localhost:30000/`. The dashboard's Sessions / Session Viewer / Grafana features all work in this mode (analytics tier comes up alongside the main image — see [Analytics tier](#analytics-tier)).
744
+
745
+
### k3d, release tagging, GHCR publishing
751
746
752
-
See [`docs/DEPLOYMENT.md`](docs/DEPLOYMENT.md) for running in a k3s cluster (release + dev side by side), pinning immutable release tags, and configuring GHCR publishing from a fork.
747
+
See [`docs/DEPLOYMENT.md`](docs/DEPLOYMENT.md) for running in two side-by-side k3d clusters (release + dev), pinning immutable release tags, and configuring GHCR publishing from a fork.
753
748
754
749
---
755
750
@@ -773,7 +768,7 @@ Captured from the live dashboard; files live in [`docs/screenshots/`](docs/scree
773
768
-[`docs/ARCHITECTURE.md`](docs/ARCHITECTURE.md) — services, routing, port map, request flow
774
769
-[`docs/API.md`](docs/API.md) — HTTP endpoints across go-live, go-upload, go-proxy
775
770
-[`docs/FAULT_INJECTION.md`](docs/FAULT_INJECTION.md) — full fault and shaping reference
Copy file name to clipboardExpand all lines: docs/API.md
+1-1Lines changed: 1 addition & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -2,7 +2,7 @@
2
2
3
3
The dashboard is a thin client over this API. Every fault-injection action, every shaping change, every session control the UI exposes is available here — so anything you can do in the browser, a test script or CI job can do too. There are no UI-only controls.
4
4
5
-
All endpoints are exposed through nginx on port `30000` (Docker Compose and k3s release) or `40000` (k3s dev). nginx routes them to the backing service based on path.
5
+
All endpoints are exposed through nginx on port `30000` (Docker Compose and k3d release) or `40000` (k3d dev). nginx routes them to the backing service based on path.
6
6
7
7
For the **fault-injection** surface (`/api/session/*` patch payloads, `/api/nftables/*` shaping), see [`docs/FAULT_INJECTION.md`](FAULT_INJECTION.md) — this page only summarises those endpoints and points to the full reference.
0 commit comments