Skip to content

feat: scaffold Experiments family resources (autogen stage 2)#479

Draft
github-actions[bot] wants to merge 2 commits into
preview-v3from
scaffold/experiments
Draft

feat: scaffold Experiments family resources (autogen stage 2)#479
github-actions[bot] wants to merge 2 commits into
preview-v3from
scaffold/experiments

Conversation

@github-actions

Copy link
Copy Markdown
Contributor

Summary

This PR was scaffolded by the LaunchDarkly autogen pipeline (stage 2) and needs human review per stage 3. Stage-3 verification runs automatically on this PR.

Adds Terraform coverage for the Experiments API family via two resources (each with a data source):

  • launchdarkly_experiment — environment-scoped experiment. Models createExperiment, getExperiment, patchExperiment, and createIteration. The experiment is created together with its first (draft) iteration; changing any iteration field issues createIteration to produce a new draft iteration. name/description update via the updateName/updateDescription semantic-patch instructions, and archived toggles via archiveExperiment/restoreExperiment.
  • launchdarkly_experimentation_settings — project singleton. Models getExperimentationSettings / putExperimentationSettings (the project's allowed randomization units). Create/Update are both PUT; Delete is state-only.

scripts/driftreport/mapping.yaml is updated to mark the Experiments family covered. The list endpoints getExperiments and getExperimentsAnyEnv are intentionally not modeled (the singular get covers declarative reads).

This is a stable (non-beta) API surface (ExperimentsApiService), so no beta client is required.

Manual reviewer steps

My GITHUB_TOKEN cannot modify .github/workflows/*, so the acceptance-test matrix was not edited. A maintainer must add these two lines under the test_case: list in .github/workflows/test.yml:

- TestAccExperiment_
- TestAccExperimentationSettings_

Both prefixes use a trailing underscore so they do not overlap (TestAccExperiment_ does not match TestAccExperimentationSettings*). The data-source test TestAccDataSourceExperimentationSettings_basic is already covered by the existing TestAccDataSource matrix entry. (Stage-3 verification re-checks this.)

Known limitations / things to verify against a real account

The acceptance tests do not run in this push (the matrix line is a manual step above), so a reviewer should validate end-to-end:

  1. Experiment read does not reconstruct iteration. The API response shape (IterationRep/FlagRep) cannot be converted back into the iteration input faithfully (e.g. FlagRep exposes targetingRule, not the input ruleId). The resource therefore preserves the configured iteration from prior state on read and refreshes only the top-level fields. As a consequence, import does not populate iteration — it must be added to config after import (documented in import.sh and the schema).
  2. No delete endpoint. Destroying a launchdarkly_experiment archives it (archiveExperiment). project_key, environment_key, key, maintainer_id, holdout_id, and tags are RequiresReplace (the patch API cannot change them); a replace will archive the old experiment and create a new one with the same key — verify the API accepts re-using an archived experiment's key, or these should become hard errors instead.
  3. Semantic patch transport. patchExperiment is called with an instructions body and the generated client's default Content-Type: application/json (same as the existing launchdarkly_team resource). Confirm experiments accept instructions without the domain-model=launchdarkly.semanticpatch content-type suffix.
  4. experimentation_settings randomization-unit ordering. Read maps the API's randomizationUnits in returned order (filtering _hidden system units). If the API reorders relative to the submitted list, this could surface a plan inconsistency — verify order is preserved, otherwise switch the read to preserve plan order.
  5. flag_config_version must match the live flag configuration version; the acceptance test reads it from the created flag's environment config.

Test plan

  • go build ./...
  • go vet ./launchdarkly/
  • gofmt / gofmts clean (incl. keys.go sorted block) ✅
  • go generate . (docs + examples) committed ✅
  • go test ./scripts/driftreport/ ✅ (mapping.yaml parses, no drift for Experiments)
  • Acceptance tests added: TestAccExperiment_CreateUpdate, TestAccExperimentationSettings_CreateUpdate, TestAccDataSourceExperimentationSettings_basic (require a real enterprise account; not run in this push).

🤖 Generated with Claude Code

github-actions Bot and others added 2 commits June 29, 2026 14:52
…on_settings resources (autogen stage 2)

Covers the Experiments API family:
- launchdarkly_experiment (resource + data source): createExperiment,
  getExperiment, patchExperiment, createIteration. Destroy archives the
  experiment (the API has no delete endpoint).
- launchdarkly_experimentation_settings (resource + data source):
  getExperimentationSettings, putExperimentationSettings as a project
  singleton.

Updates scripts/driftreport/mapping.yaml to mark the family covered.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ngs read ordering

Two idempotency fixes found while verifying the Experiments scaffold against a
real LaunchDarkly account:

- launchdarkly_experiment: iteration.randomization_unit was Optional+Computed
  with no default, but the iteration object is preserved verbatim from the plan
  (the API response cannot be converted back into the iteration input), so a
  Computed value left unset in config remained unknown after apply and tripped
  Terraform's "provider produced inconsistent result" check. Make it
  Optional-only; omitting it lets the API apply the project default.

- launchdarkly_experimentation_settings: the API returns randomization units in
  a different order than submitted (the default unit comes back first), so the
  read produced a list whose order did not match the plan/state. This caused an
  "inconsistent result after apply" error and a perpetual diff. Reorder the API
  response to match the order already in the model, appending any extra units.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown
Contributor Author

Stage-3 verification (run 12)

Verified the Experiments-family scaffold against a real LaunchDarkly account. Verdict: pass, with two idempotency fixes pushed.

launchdarkly_experimentation_settings — fully verified ✅

  • terraform apply of a project + two context kinds + experimentation settings + the data source succeeded.
  • Exercised an update (flipping the default randomization unit) and the data source.
  • The follow-up terraform plan was clean (no diff).
  • Bug found & fixed: the API returns randomization units in a different order than submitted (the default unit comes back first), so the read produced a list whose order didn't match the plan/state → "provider produced inconsistent result after apply" + a perpetual diff. The read now reorders the API response to match the order already in the model. After the fix, create/update/re-plan are all idempotent.

launchdarkly_experiment — reviewed + create path verified, full apply not run here ⚠️

  • Build / vet / gofmt clean; schema and terraform plan are valid.
  • A deliberate create with a bogus variation ID returned a precise 409 invalid_flag_definition, which confirms the full iteration (metrics, treatments, parameters, flags map) serializes correctly and the create path reaches the API — only the variation ID was rejected.
  • Bug found & fixed (review): iteration.randomization_unit was Optional+Computed with no default, but the iteration object is preserved verbatim from the plan (the API response can't be round-tripped into the iteration input). A Computed value left unset in config would stay unknown after apply and trip Terraform's consistency check. Changed to Optional-only; omitting it lets the API apply the project default.
  • A full create→apply couldn't be exercised in this sandbox: experiments require server-assigned flag variation IDs, and no resource/data source exposes them (the network paths that would fetch them weren't available to this run). Recommend running the TestAccExperiment_CreateUpdate acceptance test in CI (via the safe-to-test label) to confirm the create/read/update/archive-on-destroy round trip.

Retained

Project key tf-verify-pr479 (project-scoped resources retained for inspection; the next run pre-cleans it). LD console link is in the private Slack thread.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

0 participants