|
| 1 | +# AGENTS.md |
| 2 | + |
| 3 | +Guidance for AI coding assistants working in `fluxcd/source-controller`. Read this file before making changes. |
| 4 | + |
| 5 | +## Contribution workflow for AI agents |
| 6 | + |
| 7 | +These rules come from [`fluxcd/flux2/CONTRIBUTING.md`](https://github.com/fluxcd/flux2/blob/main/CONTRIBUTING.md) and apply to every Flux repository. |
| 8 | + |
| 9 | +- **Do not add `Signed-off-by` or `Co-authored-by` trailers with your agent name.** Only a human can legally certify the DCO. |
| 10 | +- **Disclose AI assistance** with an `Assisted-by` trailer naming your agent and model: |
| 11 | + ```sh |
| 12 | + git commit -s -m "Add support for X" --trailer "Assisted-by: <agent-name>/<model-id>" |
| 13 | + ``` |
| 14 | + The `-s` flag adds the human's `Signed-off-by` from their git config — do not remove it. |
| 15 | +- **Commit message format:** Subject in imperative mood ("Add feature X" instead of "Adding feature X"), capitalized, no trailing period, ≤50 characters. Body wrapped at 72 columns, explaining what and why. No `@mentions` or `#123` issue references in the commit — put those in the PR description. |
| 16 | +- **Trim verbiage:** in PR descriptions, commit messages, and code comments. No marketing prose, no restating the diff, no emojis. |
| 17 | +- **Rebase, don't merge:** Never merge `main` into the feature branch; rebase onto the latest `main` and push with `--force-with-lease`. Squash before merge when asked. |
| 18 | +- **Pre-PR gate:** `make tidy fmt vet && make test` must pass and the working tree must be clean after codegen. Commit regenerated files in the same PR. |
| 19 | +- **Flux is GA:** Backward compatibility is mandatory. Breaking changes to CRD fields, status, CLI flags, metrics, or observable behavior will be rejected. Design additive changes and keep older API versions round-tripping. |
| 20 | +- **Copyright:** All new `.go` files must begin with the boilerplate from `hack/boilerplate.go.txt` (Apache 2.0). Update the year to the current year when copying. |
| 21 | +- **Spec docs:** New features and API changes must be documented in `docs/spec/v1/` — one file per CRD: `gitrepositories.md`, `ocirepositories.md`, `helmrepositories.md`, `helmcharts.md`, `buckets.md`, `externalartifacts.md`. Update the relevant file in the same PR that introduces the change. |
| 22 | +- **Tests:** New features, improvements and fixes must have test coverage. Add unit tests in `internal/controller/*_test.go` and other `internal/*` packages as appropriate. Follow the existing patterns for test organization, fixtures, and assertions. Run tests locally before pushing. |
| 23 | + |
| 24 | +## Code quality |
| 25 | + |
| 26 | +Before submitting code, review your changes for the following: |
| 27 | + |
| 28 | +- **No secrets in logs or events.** Never surface auth tokens, passwords, or credential URLs in error messages, conditions, events, or log lines. Use `fluxcd/pkg/masktoken` and the `internal/error` sanitizers. |
| 29 | +- **No unchecked I/O.** Close HTTP response bodies, file handles, and archive readers in `defer` statements. Check and propagate errors from `io.Copy`, `os.Remove`, `os.Rename`. |
| 30 | +- **No path traversal.** Validate and sanitize file paths extracted from archives or user input. Use `securejoin` to ensure paths stay within the expected root. Never `filepath.Join` with untrusted components without validation. |
| 31 | +- **No unbounded reads.** Use `io.LimitReader` when reading from network or archive sources. Respect existing size limits; do not introduce new reads without bounds. |
| 32 | +- **No command injection.** Do not shell out via `os/exec`. Use Go libraries for git, helm, OCI, and cloud operations. |
| 33 | +- **No hardcoded defaults for security settings.** TLS verification must remain enabled by default; proxy and auth settings come from user-provided secrets, not environment variables. |
| 34 | +- **Error handling.** Wrap errors with `%w` for chain inspection. Do not swallow errors silently. Return actionable error messages that help users diagnose the issue without leaking internal state. |
| 35 | +- **Resource cleanup.** Ensure temporary files, directories, and cloned repos are cleaned up on all code paths (success and error). Use `defer` and `t.TempDir()` in tests. |
| 36 | +- **Concurrency safety.** Do not introduce shared mutable state without synchronization. Reconcilers run concurrently; per-object work must be isolated. Respect the existing `Storage.LockFor` pattern. |
| 37 | +- **No panics.** Never use `panic` in runtime code paths. Return errors and let the reconciler handle them gracefully. |
| 38 | +- **Minimal surface.** Keep new exported APIs, flags, and environment variables to the minimum needed. Every export is a backward-compatibility commitment. |
| 39 | + |
| 40 | +## Project overview |
| 41 | + |
| 42 | +source-controller is a core component of the [Flux GitOps Toolkit](https://fluxcd.io/flux/components/). It reconciles five custom resources in the `source.toolkit.fluxcd.io` API group — `GitRepository`, `OCIRepository`, `HelmRepository`, `HelmChart`, and `Bucket` — by fetching upstream content, verifying it (PGP, Cosign, Notation), and packaging it into an immutable `Artifact` (tarball or chart). Artifacts are written to a local filesystem rooted at `--storage-path` and served over HTTP from `--storage-addr` so downstream controllers (kustomize-controller, helm-controller, source-watcher) can consume them. Status conditions, events, and the artifact URL/revision are what other Flux controllers key off. |
| 43 | + |
| 44 | +## Repository layout |
| 45 | + |
| 46 | +- `main.go` — manager wiring: flags, scheme registration, `Storage` init, feature gates, setup of the five reconcilers, and the artifact file server. |
| 47 | +- `api/` — separate Go module (`github.com/fluxcd/source-controller/api`) holding the CRD types. The root module pulls it via a `replace` directive. |
| 48 | + - `api/v1/` — current storage version. Per-kind `*_types.go`, shared `artifact_types.go`, `condition_types.go`, `source.go`, `sts_types.go`, `ociverification_types.go`, `groupversion_info.go`, and generated `zz_generated.deepcopy.go`. |
| 49 | + - `api/v1beta1/`, `api/v1beta2/` — older versions kept for conversion/compat. |
| 50 | +- `config/` — Kustomize overlays. `config/crd/bases/` holds generated CRDs (one YAML per kind); do not hand-edit. `config/default/`, `config/manager/`, `config/rbac/`, `config/samples/`, `config/testdata/` cover install, manager Deployment, RBAC, samples, and fixtures. |
| 51 | +- `internal/` — controller implementation (not importable by other modules). |
| 52 | + - `controller/` — the five reconcilers, `Storage` (artifact lock, archive, GC, serving), `artifact.go`, `source_predicate.go`, envtest suite (`suite_test.go`), and per-kind `*_test.go` integration tests. |
| 53 | + - `reconcile/` — shared reconcile loop primitives; `reconcile/summarize/` collapses sub-results into terminal status and patches. |
| 54 | + - `helm/` — Helm logic split into `chart/` (local/remote chart builders, dependency manager, `secureloader`), `repository/` (HTTP and OCI `ChartRepository`), `getter/`, `registry/` (OCI client + auth), `common/`. |
| 55 | + - `oci/` — OCI auth plus `verifier.go`; `cosign/` and `notation/` implement the two `OCIRepository` verification providers. |
| 56 | + - `index/` — digest indexing for chart repositories. |
| 57 | + - `cache/` — in-memory TTL cache for Helm index files with Prometheus metrics. |
| 58 | + - `digest/` — canonical digest algorithm selection (sha256/384/512, blake3) and a hashing writer. |
| 59 | + - `predicates/`, `features/`, `fs/`, `util/`, `tls/`, `transport/`, `object/`, `error/`, `mock/` — small helpers; names match their responsibilities. |
| 60 | +- `pkg/` — importable provider clients consumed by `BucketReconciler`: `azure/` (Azure Blob), `gcp/` (GCS), `minio/` (S3). These are semi-public API. |
| 61 | +- `hack/` — `boilerplate.go.txt` license header, `api-docs/` templates, `ci/e2e.sh`. |
| 62 | +- `tests/` — `listener/`, `proxy/`, `registry/` harnesses used by integration tests. |
| 63 | +- `docs/` — `spec/` (user-facing API docs per version), `api/` (generated reference), `internal/release.md`, `diagrams/`. |
| 64 | + |
| 65 | +## APIs and CRDs |
| 66 | + |
| 67 | +- Group: `source.toolkit.fluxcd.io`. Storage version: `v1`. `v1beta1` and `v1beta2` remain for compatibility. |
| 68 | +- Kinds: `GitRepository`, `OCIRepository`, `HelmRepository`, `HelmChart`, `Bucket`. Shared `Artifact` type in `api/v1/artifact_types.go`; shared conditions in `api/v1/condition_types.go`; `Source` interface in `api/v1/source.go`. |
| 69 | +- CRD manifests under `config/crd/bases/source.toolkit.fluxcd.io_*.yaml` are generated from kubebuilder markers. Never edit them by hand — update the types and run `make manifests`. |
| 70 | +- `api/v1*/zz_generated.deepcopy.go` is generated — update types and run `make generate`. |
| 71 | +- `api/` is a distinct Go module so external projects can depend on the types without pulling controller deps. The root module uses `replace github.com/fluxcd/source-controller/api => ./api`. |
| 72 | + |
| 73 | +## Build, test, lint |
| 74 | + |
| 75 | +All targets live in the top-level `Makefile`. Extra `go test` flags go via `GO_TEST_ARGS` (default `-race`). Tool binaries install into `build/gobin/` on first use. |
| 76 | + |
| 77 | +- `make tidy` — tidy both the root and `api/` modules. |
| 78 | +- `make fmt` / `make vet` — run in both modules. |
| 79 | +- `make generate` — `controller-gen object` against `api/` (deepcopy). |
| 80 | +- `make manifests` — regenerate CRDs and RBAC from `+kubebuilder` markers. |
| 81 | +- `make api-docs` — regenerate `docs/api/v1/source.md`. |
| 82 | +- `make manager` — static build of `build/bin/manager`. |
| 83 | +- `make test` — chains `install-envtest` + `test-api` + `go test ./...` with coverage and `KUBEBUILDER_ASSETS` set to envtest binaries. |
| 84 | +- `make test-api` — unit tests inside `api/`. |
| 85 | +- `make test-ctrl GO_TEST_PREFIX=<name>` — run one reconciler suite under `internal/controller`. |
| 86 | +- `make install` / `make uninstall` / `make run` / `make deploy` / `make dev-deploy` / `make docker-build` / `make docker-push` — cluster workflows. |
| 87 | +- `make verify` — runs `fmt vet manifests api-docs tidy` and fails on a dirty tree. CI uses this. |
| 88 | +- `make e2e` — shells out to `hack/ci/e2e.sh`. |
| 89 | + |
| 90 | +## Codegen and generated files |
| 91 | + |
| 92 | +Check `go.mod` and the `Makefile` for current dependency and tool versions. After changing API types or kubebuilder markers, regenerate and commit the results: |
| 93 | + |
| 94 | +```sh |
| 95 | +make generate manifests api-docs |
| 96 | +``` |
| 97 | + |
| 98 | +Generated files (never hand-edit): |
| 99 | + |
| 100 | +- `api/v1*/zz_generated.deepcopy.go` |
| 101 | +- `config/crd/bases/*.yaml` |
| 102 | +- `docs/api/v1/source.md` |
| 103 | + |
| 104 | +Load-bearing `replace` directives in `go.mod` — do not remove: |
| 105 | + |
| 106 | +- `Masterminds/semver/v3` pinned (see issue #1738). |
| 107 | +- `opencontainers/go-digest` pinned to a master snapshot for BLAKE3 support. |
| 108 | + |
| 109 | +Bump `fluxcd/pkg/*` modules as a set — version skew breaks `go.sum`. Run `make tidy` after any bump. |
| 110 | + |
| 111 | +## Conventions |
| 112 | + |
| 113 | +- Standard `gofmt`. Every exported name needs a doc comment; non-trivial unexported declarations should have one too. |
| 114 | +- Reconcilers follow the Flux sub-reconciler pattern: the top-level `Reconcile` invokes an ordered slice of step functions, then `summarize.Processor` (in `internal/reconcile/summarize/`) collapses their `reconcile.Result` + errors into terminal conditions and patches status. Don't set conditions directly from step code. |
| 115 | +- Status writes go through the patch helper with `metav1.Condition` values from `api/v1/condition_types.go` and `fluxcd/pkg/apis/meta` (`Ready`, `Reconciling`, `Stalled`, kind-specific `ArtifactInStorage`, `SourceVerified`, `FetchFailed`, etc.). Use `fluxcd/pkg/runtime/conditions`. |
| 116 | +- Events: `EventRecorder` wired to `fluxcd/pkg/runtime/events.Recorder`. Event reasons match condition reasons. |
| 117 | +- Artifacts: create via `Storage.NewArtifactFor`, persist with `Storage.Archive`/`Storage.Copy` inside the per-object lock (`Storage.LockFor`), publish via `Storage.SetArtifactURL`, and run `Storage.GarbageCollect` after a successful write honoring `--artifact-retention-ttl` and `--artifact-retention-records`. |
| 118 | +- TLS/transport: build `http.Transport` via `internal/transport` and TLS configs via `internal/tls`. Proxy, HTTP/2, and keepalive settings must stay consistent. |
| 119 | +- Feature gates go through `internal/features` plus `fluxcd/pkg/runtime/features`. Define the constant in `features.go` with its default; check with `features.Enabled(...)`. |
| 120 | +- Digests default to `internal/digest.Canonical` and can be overridden by `--artifact-digest-algo`. Hash through the writer in `internal/digest` — never import `crypto/sha256` directly. |
| 121 | + |
| 122 | +## Testing |
| 123 | + |
| 124 | +- Integration tests live next to the reconcilers in `internal/controller/*_test.go`. `suite_test.go` spins up `testenv.Environment` (`fluxcd/pkg/runtime/testenv`), a local `testserver.ArtifactServer`, an in-memory distribution registry, and `foxcpp/go-mockdns`. |
| 125 | +- `make install-envtest` downloads kube-apiserver/etcd binaries into `build/testbin/`. `make test` sets `KUBEBUILDER_ASSETS` to that path. On macOS the Makefile forces `ENVTEST_ARCH=amd64`. |
| 126 | +- Plain Go + Gomega (`gomega.NewWithT(t)`); no Ginkgo. Reuse the package-level `k8sClient`, `testEnv`, `testStorage`, and `testServer` from `suite_test.go`. |
| 127 | +- Tests unset `HTTP_PROXY`/`HTTPS_PROXY` and set `GIT_CONFIG_GLOBAL=/dev/null` and `GIT_CONFIG_NOSYSTEM=true` to isolate from the developer's environment. Do the same for new git-shelling tests. |
| 128 | +- Run a single test: `make test GO_TEST_ARGS="-v -run TestGitRepositoryReconciler_reconcileSource"`. |
| 129 | +- Run one reconciler suite: `make test-ctrl GO_TEST_PREFIX=TestHelmChart`. |
| 130 | +- Fixtures: `internal/controller/testdata`, `internal/helm/testdata`, `internal/fs/testdata`. Reuse; don't add new large binaries. |
| 131 | + |
| 132 | +## Gotchas and non-obvious rules |
| 133 | + |
| 134 | +- Two Go modules: root and `api/`. `make tidy`, `fmt`, `vet`, `test` iterate both. A change to `api/` types requires running `make generate` **and** `make manifests` and committing the regenerated files in the same PR. |
| 135 | +- `make verify` is the CI gate — a dirty diff means you forgot to run codegen or tidy. |
| 136 | +- The `replace` directives in `go.mod` (semver and go-digest) exist for correctness. Leave them alone. |
| 137 | +- `Storage` serializes writes per object via `fluxcd/pkg/lockedfile` and expects `--storage-path` to be a real local directory. Never write artifacts outside `Storage.BasePath` — the file server exposes that path verbatim at `--storage-addr`. |
| 138 | +- Workload Identity is feature-gated by `auth.FeatureGateObjectLevelWorkloadIdentity` (from `fluxcd/pkg/auth`). Token caching is opt-in via `--token-cache-max-size`. |
| 139 | +- `CacheSecretsAndConfigMaps` (in `internal/features`) is off by default; `Secret`/`ConfigMap` lookups bypass the cache and hit the API server directly. Mind that before adding new secret reads to a hot path. |
| 140 | +- The controller watches a label-selected subset of its CRs — see `Cache.ByObject` in `mustSetupManager`. Adding a new kind requires updating both `main.go` and the scheme. |
| 141 | +- `pkg/azure`, `pkg/gcp`, `pkg/minio` are importable by external consumers. Treat their exported surface as semi-public API. |
0 commit comments