docs(release): org-owned private App procedure with key handling (#353) (#366)

* docs(release): org-owned private App procedure with key handling (#353)

Mirror the issue-353 tracker into the runbook: the policy-read App
requires GoogleCloudPlatform organization-owner authority to install
(repository Administration permission), must be registered as
installable only on this account (private App), webhooks disabled and
no organization/account permissions, with explicit private-key
generation, installation scoped to this repository only, and PEM
handling guidance (delete the workstation copy after storing the
secret; record key-rotation ownership).

* docs(release): first-release setup parity + corrected recovery semantics (#353 review)

- Immutable-release refusal recovery: the GitHub release stays a
  draft, but PyPI/TestPyPI already contain the exact files when
  finalize reaches the policy precondition — enable the setting and
  re-run finalize from the ORIGINAL attempt, never a full rerun.
- Pre-flight: the first release requires PENDING Trusted Publishers
  on both indexes (no project exists yet; the pending publisher
  creates it on first trusted upload).
- One-time GitHub setup enumerates the tracing-v* tag restriction on
  all three environments and scopes required reviewers to pypi and
  release-promote only.
- Version-burn rule distinguishes true burns (defective bytes after
  acceptance, index deviations, reconciler burn states) from
  retryable failures where the index holds the exact anchor bytes
  (original-attempt rerun; the pre-checks pass without re-upload).

* docs(release): reject-vs-burn distinction, missing-release state, Owner requirement, publisher failure paths (#353 review)

- A full rerun is REJECTED (return to the recoverable original run),
  not itself a burn: a burn is asserted only for deviation from the
  original accepted anchor.
- The reconciler burn-state list includes missing-release.
- Post-first-publish: a second Google-controlled Owner (not
  Maintainer) on each index, timed to each index's first trusted
  upload, with 2FA/recovery for both owners.
- Publisher misconfiguration is documented per DAG path: missing
  TestPyPI publisher stops promote/publish-pypi; missing production
  publisher fails only publish-pypi after earlier stages pass.

* docs(release): 'original workflow build anchor' wording; first-attempt conflicts burn (#353 review round 2)

Author: caohy1988

producers/RELEASING.md

@@ -20,9 +20,12 @@ The tag namespace (`tracing-vX.Y.Z`) is distinct from the root SDK's
    the curated customer-first template via
    `scripts/render_release_notes.py`; generic generated notes are
    disabled — the repo-wide tag stream would pull unrelated SDK PRs).
-4. PyPI side: project + Trusted Publisher are already configured.
-   Setup is one-time per project — see "PyPI Trusted Publishing
-   setup" below.
+4. PyPI side: a **pending Trusted Publisher** is registered on BOTH
+   indexes (for the first release neither project exists yet — the
+   pending publisher creates it on the first trusted upload and does
+   not reserve the name before then; for later releases the existing
+   project publishers suffice). Setup is one-time per project — see
+   "PyPI Trusted Publishing setup" below.
 
 ## Cut the release
 
@@ -164,10 +167,30 @@ Post evidence on the release issue.
 ## Version-burn rule
 
 TestPyPI and PyPI must carry **byte-identical artifacts at the same
-version**. If a candidate fails the gate, that version is burned
-everywhere: bump the version, re-tag, rebuild from scratch. Never
-re-upload, never re-tag an image (staging and public tags are
-immutable — enforced at the repository level).
+version**. Not every failure burns the version — distinguish:
+
+- **Burned**: a lifecycle-gate failure caused by defective candidate
+  bytes after TestPyPI accepted them, an index deviation from the
+  **original workflow build anchor** (subset/extra/yanked/digest
+  mismatch — including pre-existing files found on the FIRST attempt,
+  where the current distributions are that anchor and the conflicting
+  filenames can never be replaced), or any reconciler burn state
+  (`empty-release`, `testpypi-partial`, `partial`,
+  `missing-release`). Bump the version, rebuild, re-tag. Never
+  re-upload, never re-tag an image (staging and public tags are
+  immutable — enforced at the repository level).
+- **NOT burned**: a transient job failure, a lost upload response, or
+  a `finalize` failure while the index carries the EXACT anchor bytes
+  — re-run the failed jobs from the **original** workflow attempt
+  (the rerun-safe pre-checks recognize the byte-identical publication
+  and pass without re-uploading).
+- **Rejected, but not a burn**: an accidental **full rerun**. Its
+  rebuilt bytes cannot match what an index accepted, so the guard
+  refuses the rebuilt attempt — abandon it and return to the
+  **original** workflow run, which remains recoverable (its draft and
+  artifact are preserved). A burn is asserted only against the
+  ORIGINAL workflow build anchor, never merely against a newly
+  rebuilt one.
 
 ## Verifying the release
 
@@ -217,9 +240,11 @@ tar -tzf /tmp/plugin.tar.gz | head
   original attempt: the pre-check verifies the index already carries
   the exact byte-identical files and skips the re-upload.
 - **`finalize` refuses to publish because immutable releases are
-  disabled** — nothing was published (the release is still a draft):
-  enable the setting (Settings → General → Releases → immutable
-  releases) and re-run `finalize` from the original attempt.
+  disabled** — the GitHub release remains a draft, but PyPI and
+  TestPyPI already contain the exact files (the index stages run
+  before `finalize`). Enable immutable releases (Settings → General →
+  Releases) and re-run `finalize` from the **original** workflow
+  attempt; never trigger a full rerun.
 - **a release was somehow published while the setting was off** —
   immutability is NOT retroactive: enabling the setting cannot protect
   it, so treat the version as burned (delete the release, yank any
@@ -248,10 +273,20 @@ These names must match exactly — the `environment:` blocks in the
 workflow are the binding contract.
 
 GitHub-side one-time setup: create environments `testpypi`, `pypi`,
-and `release-promote` in the repo settings, with required reviewers
-on `pypi` and `release-promote` (approving `release-promote` asserts
-the TestPyPI full-lifecycle gate passed), and **enable immutable
-releases BEFORE the first release** (Settings → General → Releases):
+and `release-promote` in the repo settings —
+
+- restrict **all three** environments to tag `tracing-v*`
+  (Settings → Environments → Deployment branches and tags → Selected →
+  Tag rule `tracing-v*`); the `testpypi` rule matters even without a
+  reviewer, because it stops a branch-modified copy of the workflow
+  from using the TestPyPI OIDC identity and burning a version;
+- required reviewers on `pypi` and `release-promote` ONLY (approving
+  `release-promote` asserts the TestPyPI full-lifecycle gate passed);
+  `testpypi` gets no reviewer — the manual lifecycle verdict is the
+  later `release-promote` approval —
+
+and **enable immutable releases BEFORE the first release**
+(Settings → General → Releases):
 `finalize` verifies the setting and refuses to publish while it is
 off, because GitHub applies immutability only at publish time — a
 release published while the setting is off keeps mutable assets and
@@ -261,19 +296,49 @@ release published while the setting is off keeps mutable assets and
 Policy-read credential (one-time): the immutable-releases check calls
 `GET /repos/{repo}/immutable-releases`, which requires repository
 **Administration: read** — a permission the workflow `GITHUB_TOKEN`
-can never be granted. Create a GitHub App whose ONLY repository
-permission is Administration: read, install it on this repository
-only, and set the repository variable `BQAA_RELEASE_POLICY_APP_ID`
-plus the secret `BQAA_RELEASE_POLICY_APP_PRIVATE_KEY`. The workflow
-mints a short-lived installation token from these only when a DRAFT
-publication is about to happen — idempotent reruns of an
+can never be granted. Because the App requests repository
+Administration permission, installation needs a **GoogleCloudPlatform
+organization owner** — a repository admin alone cannot complete this
+([installation requirements](https://docs.github.com/en/apps/using-github-apps/installing-a-github-app-from-a-third-party)):
+
+1. An organization owner creates an **organization-owned** GitHub
+   App.
+2. Registration setting: **"Where can this GitHub App be
+   installed?" → Only on this account** — keeps the App private so no
+   other account can install it
+   ([docs](https://docs.github.com/en/enterprise-cloud@latest/apps/creating-github-apps/registering-a-github-app/making-a-github-app-public-or-private)).
+3. Grant only repository **Administration: read**; disable webhooks;
+   request no organization or account permissions.
+4. Generate and download a private key for the App.
+5. The organization owner installs it on **only**
+   `BigQuery-Agent-Analytics-SDK`.
+6. Store the App ID as the repository variable
+   `BQAA_RELEASE_POLICY_APP_ID` and the PEM as the repository secret
+   `BQAA_RELEASE_POLICY_APP_PRIVATE_KEY`. Then delete the downloaded
+   workstation copy of the PEM unless it is retained in an approved
+   secret manager, and record who owns future key rotation — an App
+   private key can authenticate against every installation of that
+   App ([best practices](https://docs.github.com/en/apps/creating-github-apps/about-creating-github-apps/best-practices-for-creating-a-github-app)).
+
+The workflow mints a short-lived installation token from these only
+when a DRAFT publication is about to happen — idempotent reruns of an
 already-published release perform no policy read and never touch the
 App credentials, so a rotated key cannot break them or mask the
 mutable-release burn guidance; all other API calls keep the standard
 job token. Until they are configured, `finalize` fails at the token
 mint with a clear error and the release stays a draft.
 
-Until both publishers are configured, the `publish-testpypi` and
-`publish-pypi` jobs will fail with a clear error. The `build` and
-`github-release` jobs are independent and will still complete, so
-the tag stays valid.
+Publisher misconfiguration fails at two different points in the DAG:
+a missing **TestPyPI** publisher fails `publish-testpypi`, so
+`promote` and `publish-pypi` never run; a missing **production**
+publisher lets every earlier stage pass and then fails
+`publish-pypi`. In both cases the error is a clear OIDC/trusted-
+publisher failure, the `build` and `github-release` jobs still
+complete, and the tag stays valid.
+
+Immediately after each index's FIRST trusted upload creates that
+index project, add a second Google-controlled **Owner** (not
+Maintainer — a Maintainer can upload but cannot manage the project or
+collaborators, so it provides no administrative recovery;
+https://docs.pypi.org/organization-accounts/roles-entities/) and
+confirm 2FA/account recovery for both owners.