diff --git a/docs/development/changelog-bundle-registry.md b/docs/development/changelog-bundle-registry.md new file mode 100644 index 0000000000..790e644f87 --- /dev/null +++ b/docs/development/changelog-bundle-registry.md @@ -0,0 +1,278 @@ +--- +navigation_title: Changelog bundle registry +--- + +# Changelog bundle registry and CDN delivery + +This page describes how changelog **bundles** are published to a public, CDN-fronted +S3 bucket, how the per-product `registry.json` manifest is produced, and the +`cdn:` mode for the [`{changelog}` directive](/syntax/changelog.md) that consumes +bundles directly from the CDN instead of from a local folder. + +:::{note} +Both sides are implemented: the **producer** (manifest generation + scrubber pass-through) +and the **consumer** (`{changelog}` directive `cdn:` mode). Remaining follow-ups are listed +under [Implementation notes](#implementation-notes). +::: + +## Motivation + +Today the `{changelog}` directive only renders bundles that live in a folder inside the +docset (default `changelog/bundles/`). That requires every consuming repository to vendor +a copy of the bundle YAML it wants to render. + +The link service ([building block](/building-blocks/link-service.md)) already demonstrates +the pattern we want: an S3 bucket fronted by CloudFront, publicly readable, with a small +JSON index at a well-known key. We apply the same approach to changelog bundles so a docset +can render another product's release notes by pointing the directive at the CDN — no vendored +copies, no cross-repo file syncing. + +## Architecture + +``` +┌──────────────┐ changelog upload ┌────────────────────┐ s3:ObjectCreated ┌───────────────────┐ +│ Client CI │ --artifact-type │ Private bundles │ ───────────────────▶ │ Changelog scrubber │ +│ (docs-actions)│ bundle ───────────▶ │ S3 bucket │ │ Lambda │ +└──────────────┘ │ │ └─────────┬─────────┘ + │ │ {product}/bundle/*.yaml │ scrub + copy + │ also refreshes │ {product}/registry.json │ (pass-through for + └──────────────────────────────▶ │ │ registry.json) + └────────────────────┘ ▼ + ┌───────────────────┐ + {changelog} directive (cdn:) │ Public bundles │ + reads via CDN ◀─────────────── │ S3 bucket + CDN │ + └───────────────────┘ +``` + +1. **Producer** — `changelog upload --artifact-type bundle --target s3` (invoked by the + docs-actions changelog upload workflow) uploads each bundle to + `{product}/bundle/{file}` in the **private** bucket, then refreshes + `{product}/registry.json` for every product the run touched. +2. **Scrubber Lambda** — triggered by `s3:ObjectCreated` on the private bucket, it scrubs + private repository references out of bundle YAML and writes the sanitized copy to the + **public** bucket. The `registry.json` object is copied through **verbatim**. +3. **Consumer** — for each product declared under `release_notes` in `docset.yml`, docs-builder + reads `{product}/registry.json` from the CDN at build startup and fetches each listed bundle; + the `{changelog}` directive in `cdn:` mode then renders from the prefetched result. + +### Why a registry instead of an S3 listing + +The public surface is a CDN (CloudFront) in front of S3. CloudFront does not expose bucket +listing, so the consumer cannot enumerate `{product}/bundle/`. The registry is a stable, +cacheable manifest at a predictable key that lists exactly which bundles exist for a product. + +## `registry.json` format + +Stored at `{product}/registry.json`. Serialized with `snake_case` keys. + +```json +{ + "schema_version": 1, + "product": "elasticsearch", + "generated_at": "2026-05-06T12:00:00+00:00", + "bundles": [ + { "file": "9.4.0.yaml", "target": "9.4.0", "etag": "…" }, + { "file": "9.3.0.yaml", "target": "9.3.0", "etag": "…" } + ] +} +``` + +| Field | Meaning | +|---|---| +| `schema_version` | Bumped when consumers must change their parser. | +| `product` | Product identifier; matches the first S3 key segment. | +| `generated_at` | UTC timestamp of the last regeneration. | +| `bundles[].file` | Bundle file name, resolved at `{product}/bundle/{file}`. | +| `bundles[].target` | Target version/date from the bundle's declaration of **this** product (may be null). | +| `bundles[].etag` | See the ETag caveat below. | + +Bundles are sorted by `target` descending (newest first) with a deterministic tiebreak on +`file`, so the JSON is stable across reruns. + +### ETag caveat + +`bundles[].etag` is the ETag of the bundle object **as uploaded to the private bucket** +(pre-scrub). The scrubber rewrites any bundle that contains private references, so for +scrubbed bundles this value **will not match** the public (CDN) object's ETag. + +Consumers **must not** use it for integrity checks or HTTP cache validation against the +public bucket — use the CDN response's own `ETag`/`Last-Modified` for caching. The field is +only a best-effort change hint (e.g. detecting whether a bundle changed between two manifest +reads of the same bucket). + +## Producer details (implemented) + +The refresh runs inside `ChangelogUploadService` after a successful **bundle** upload (it is +skipped for `--artifact-type changelog`). `RegistryBuilder`: + +- Groups the run's upload targets by product (from the `{product}/bundle/{file}` key). +- For each product, derives one `registry` entry per bundle (file name, that product's + target, locally-computed S3 ETag). +- Reads the existing manifest from S3, merges by file name (re-uploads replace their entry; + others are preserved), and writes the merged manifest back. + +### Concurrency: optimistic, conditional writes + +Two uploads that touch the same product (for example two repositories that both map to one +product, or parallel CI) could otherwise clobber each other's index via a naive +read-modify-write. The writer instead uses **S3 conditional PUT**: + +- On **update**: `If-Match: ` — only succeeds if the object hasn't changed. +- On **create**: `If-None-Match: *` — only succeeds if the object still doesn't exist. + +A `412 Precondition Failed` means another writer won the race; the builder re-reads, +re-merges, and retries (bounded retries). This mirrors the link-index writer +(`AwsS3LinkIndexReaderWriter.SaveRegistry`). If the merge result already equals what's +published, the write is skipped so re-uploads stay idempotent. + +The refresh is **best-effort**: any failure is logged and surfaced as a warning but never +fails the upload, because the bundle objects themselves are already in S3. + +### Buckets and infrastructure + +The registry is written to the **private** bucket +(`elastic-docs-v3-changelog-bundles-private`) — the same bucket and key space as the bundles +themselves — and reaches the **public** bucket (`elastic-docs-v3-changelog-bundles`, served +only via CloudFront + OAC) through the scrubber's verbatim pass-through. The uploader never +writes to the public bucket; the scrubber Lambda is the sole writer there, which preserves the +invariant that everything on the public surface has been vetted. + +The required infrastructure already exists in `docs-infra` +(`aws/elastic-web/us-east-1/elastic-docs-v3-changelog-bundles/`) — **no infra change is needed +for the producer**: + +- The private bucket's S3 → SQS notification fires on `s3:ObjectCreated:*` / `s3:ObjectRemoved:*` + with **no suffix filter**, so registry `.json` events already reach the scrubber. +- The uploader (GitHub Actions OIDC) role already has `s3:GetObject`/`s3:PutObject`/`s3:ListBucket` + on the private bucket, so the producer's conditional GET + PUT work. Conditional + (`If-Match`/`If-None-Match`) writes need no extra permission. +- The scrubber role has `s3:GetObject` on private and `s3:PutObject`/`s3:DeleteObject` on public, + covering the registry `CopyObject` pass-through and the `ObjectRemoved` delete. +- A CloudFront cache policy tuned for the manifest already exists (default TTL 1h, min 60s). + +The scrubber only passes through keys accepted by `RegistryKey.IsRegistry` (a single +`{product}/registry.json` segment), so arbitrary JSON cannot reach the public surface. + +**No new docs-actions workflow logic is required** for the producer either: the refresh is a +side-effect of the existing `changelog upload` step; docs-actions only needs a docs-builder +build that includes this feature. + +### Consistency notes the consumer must tolerate + +- The manifest pass-through and the per-bundle scrub are independent S3 events, so the index + may briefly reference a bundle that is not yet on the public bucket. +- A bundle that fails scrubbing (private references that cannot be allowlisted) is never + written to the public bucket, even though the index may list it. + +Consumers must therefore treat a missing bundle as non-fatal (skip + warn), not an error. + +## Consumer: `{changelog}` directive `cdn:` mode (implemented) + +### Syntax + +```markdown +:::{changelog} +:cdn: elasticsearch +::: +``` + +The directive accepts a `:cdn:` option naming the **product** to render (validated against +`[a-zA-Z0-9_-]+`). It is a *selector* over bundles that were prefetched at build startup, so the +product must be declared under `release_notes` in `docset.yml` (see +[Declaration and prefetch](#declaration-and-prefetch)). The product is optional: a valueless +`:cdn:` infers the product from the current repository (`BuildContext.Git.RepositoryName`), +mapped to its canonical product id via `products.yml` (for example the `elastic-otel-java` repo → +`edot-java` product). Multi-product repositories (for example `cloud`, which publishes +`cloud-hosted`, `cloud-serverless`, and `cloud-enterprise`) must name the product explicitly. When +the product cannot be inferred (git information unavailable) or is not declared under +`release_notes`, the directive emits an error. + +The CDN base URL is environment configuration, not authored per page: it +defaults to the public changelog bundles distribution and is overridable via the +`DOCS_BUILDER_CHANGELOG_CDN` environment variable (absolute `http`/`https` URL) for +staging, local development, and testing. + +When `:cdn:` is set, the local-folder argument is ignored (a warning is emitted if one is +also given) and the directive renders the prefetched CDN bundles instead. + +### Declaration and prefetch [declaration-and-prefetch] + +CDN-sourced products are declared once per docset under `release_notes` in `docset.yml`, mirroring +the `cross_links` mechanism: + +```yaml +# docset.yml +release_notes: + - product: elasticsearch + - product: edot-java +``` + +Each entry is validated against `products.yml` (the id must exist and carry the `release-notes` +feature). At build startup — before any markdown is parsed — `ReleaseNotesFetcher` fetches the +registry and bundles for every declared product **concurrently**, stores the result in an immutable +`FetchedReleaseNotes`, and exposes it through `IReleaseNotesResolver`. The resolver is threaded into +the parser via `DocumentationSet`/`ParserContext`, so the `{changelog}` directive's `:cdn:` mode is +a pure in-memory lookup with no network I/O at parse time. Build paths that do not source release +notes use `NoopReleaseNotesResolver`. + +### Fetch flow + +1. `GET {cdnBase}/{product}/registry.json`. +2. Parse it; for each `bundles[].file`, `GET {cdnBase}/{product}/bundle/{file}`. +3. Feed the downloaded YAML into the existing `BundleLoader` → `MergeBundlesByTarget` → + render pipeline. **Rendering is unchanged**; only the source of the bundle bytes differs. + +Implemented by `CdnChangelogFetcher` (a stateless async fetch engine in +`Elastic.Documentation.Configuration`) and `BundleLoader.LoadBundlesFromContent`. Because public +bundles are already scrubbed and **resolved** (entries are inline/self-contained), the fetcher never +needs to download separate entry files; the existing private-repo link and description visibility +logic still applies via `assembler.yml`, exactly as for local bundles. + +### Behavior and decisions + +- **Async prefetch at startup.** Bundles are fetched once per declared product before parsing, via + `HttpClient.GetAsync`, rather than synchronously inside the Markdig block parser. The directive + then selects from the prefetched, immutable `FetchedReleaseNotes`. +- **Fail-fast registry, tolerant bundles.** A declared product whose registry cannot be fetched or + parsed fails the build; an individual bundle that 404s or fails to parse is a warning and is + skipped, per the [consistency notes](#consistency-notes-the-consumer-must-tolerate). +- **Undeclared product.** A `:cdn:` directive naming a product not declared under `release_notes` + is an error — its bundles were never prefetched — which keeps network sources auditable in one + place. +- **Schema evolution.** A `schema_version` newer than the consumer understands produces a + clear error rather than a silent mis-parse. +- **Filtering.** `:type:`, `:link-visibility:`, `:description-visibility:`, `:dropdowns:` and + `hide-features` apply identically to CDN-sourced bundles. +- **Version selection.** `:version:` renders a single target and works in both modes (shared match + on registry `target` or bundle file name, see `ChangelogVersionMatch`). In CDN mode it filters the + prefetched bundles at render time. +- **Security.** The base URL is trusted configuration; the product and registry-supplied bundle + file names are validated to single path segments so neither can traverse outside + `{product}/bundle/`. + +### Follow-ups (not yet implemented) [implementation-notes] + +- **Persistent / offline cache.** Each build prefetches declared products once into memory but does + not persist a disk cache, so a cold build always reaches the CDN and an unreachable declared + product fails the build. A follow-up should add an ETag-keyed on-disk cache under the + docs-builder app-data directory (mirroring `CrossLinkFetcher`) with offline fallback. +- **`serve` mode staleness.** The prefetch runs per reload, but within a single `serve` process a + product's CDN content is pinned until the next reload. Acceptable for now (serve targets local + markdown authoring, not changelog bundles); revisit alongside the disk cache. +- **CDN staleness.** The distribution caches the manifest with a 1h default TTL (60s min), so a + freshly uploaded bundle may not appear in the CDN-served `registry.json` for up to an + hour. If faster propagation is needed the producer (or a docs-actions step) would issue a + CloudFront invalidation on registry write. +- **Caching key.** When the disk cache lands, use the CDN response ETag (not the registry + `etag` field) for revalidation. + +### Out of scope + +- Cross-product aggregation in a single directive block (one product per block). +- Authenticated/private CDN access (the public bucket is anonymous-read by design). + +## Related + +- [Changelog directive](/syntax/changelog.md) — current (local-folder) behavior. +- [Publish changelogs](/contribute/publish-changelogs.md) — the upload workflow. +- [Link service](/building-blocks/link-service.md) — the S3 + CloudFront pattern this reuses. diff --git a/docs/development/toc.yml b/docs/development/toc.yml index 58a8dc394c..55c500fa67 100644 --- a/docs/development/toc.yml +++ b/docs/development/toc.yml @@ -1,5 +1,6 @@ toc: - file: index.md + - file: changelog-bundle-registry.md - folder: ingest children: - file: index.md diff --git a/docs/syntax/changelog.md b/docs/syntax/changelog.md index 1c28a02844..3b3f717a28 100644 --- a/docs/syntax/changelog.md +++ b/docs/syntax/changelog.md @@ -28,6 +28,8 @@ The directive supports the following options: | `:description-visibility: value` | Visibility of changelog **record** descriptions (YAML `description` on each entry) | `auto` | | `:dropdowns:` | Render breaking changes, deprecations, known issues, and highlights as expandable dropdowns instead of flattened bulleted lists | false | | `:config: path` | Path to `changelog.yml` configuration | auto-discover | +| `:cdn: [product]` | Render bundles for a product that is declared under `release_notes` in `docset.yml` and prefetched from the public changelog CDN. The product is optional and inferred from the current repository when omitted | (local folder) | +| `:version: target` | Render only the single bundle matching this target/version | (all versions) | ### Example with options @@ -161,6 +163,70 @@ Explicit path to a `changelog.yml` or `changelog.yaml` configuration file, relat Both explicit and auto-discovered paths must resolve within the repository checkout directory and must not traverse symlinks. +#### `:cdn:` [cdn] + +Renders bundles for a single **product** that the docset sources from the public changelog CDN, so a docset can show release notes without vendoring bundle YAML. The directive is a *selector*: it renders bundles that docs-builder prefetched at startup, so the product must first be declared under [`release_notes`](#declaring-cdn-backed-products) in `docset.yml`. + +```yaml +# docset.yml +release_notes: + - product: elasticsearch +``` + +```markdown +:::{changelog} +:cdn: elasticsearch +::: +``` + +The value names a product defined in [`products.yml`](https://github.com/elastic/docs-builder/blob/main/config/products.yml) (syntactically it must match `[a-zA-Z0-9_-]+`). The value is **optional**: leave it blank to infer the product from the repository that holds the doc. The repository name is mapped to its canonical product id via `products.yml` (for example the `elastic-otel-java` repo renders the `edot-java` product). + +```markdown +:::{changelog} +:cdn: +::: +``` + +If the product cannot be inferred, or is not declared under `release_notes`, the block emits an error rather than rendering empty. When `:cdn:` is set, the local-folder argument is ignored. All other options (`:type:`, `:link-visibility:`, `:description-visibility:`, `:dropdowns:`, `:subsections:`) and `hide-features` apply identically to CDN-sourced bundles. + +The CDN base URL is build configuration, not authored per page: it defaults to the public changelog bundles distribution and can be overridden with the `DOCS_BUILDER_CHANGELOG_CDN` environment variable (an absolute `http`/`https` URL) for staging or local testing. + +Bundles are fetched **once at build startup** for every declared product, not per directive. If a declared product's registry cannot be fetched the build fails; an individual bundle that is missing from the CDN is skipped with a warning. For the full design — including the manifest format and infrastructure — see [Changelog bundle registry and CDN delivery](/development/changelog-bundle-registry.md). + +##### Declaring CDN-backed products [declaring-cdn-backed-products] + +List each CDN-sourced product under `release_notes` in `docset.yml`. Every entry must reference a product id from `products.yml` that participates in the release-notes system: + +```yaml +# docset.yml +release_notes: + - product: elasticsearch + - product: edot-java +``` + +docs-builder prefetches the registry and bundles for each declared product at startup. A `:cdn:` directive that names an undeclared product is an error, which keeps the set of network sources auditable in one place rather than discovered dynamically across pages. + +#### `:version:` [version] + +Renders only the **single** bundle whose target matches the given value, instead of every bundle for the source. A bundle matches when the value equals its declared `target` (for example `9.4.0`, or a date like `2026-04-09`) or its file name (with or without extension). Matching is case-insensitive. + +```markdown +:::{changelog} +:version: 9.4.0 +::: +``` + +This works for both local-folder and `:cdn:` sources. In `:cdn:` mode it filters the prefetched bundles down to the matching target at render time. + +```markdown +:::{changelog} +:cdn: elasticsearch +:version: 9.4.0 +::: +``` + +If no bundle matches, the directive renders nothing and emits a warning (it does not fall back to showing all versions). + ## Filtering entries with bundle rules You can filter changelog entries at bundle time using the `rules.bundle` configuration in your `changelog.yml` file. This is evaluated during `changelog bundle` and `changelog gh-release`, before the bundle is written. Entries that don't match are excluded from the bundle entirely. diff --git a/src/Elastic.Codex/Building/CodexBuildService.cs b/src/Elastic.Codex/Building/CodexBuildService.cs index 8a5ba281d0..c1596b181d 100644 --- a/src/Elastic.Codex/Building/CodexBuildService.cs +++ b/src/Elastic.Codex/Building/CodexBuildService.cs @@ -12,6 +12,7 @@ using Elastic.Documentation.Configuration; using Elastic.Documentation.Configuration.Builder; using Elastic.Documentation.Configuration.Codex; +using Elastic.Documentation.Configuration.ReleaseNotes; using Elastic.Documentation.Diagnostics; using Elastic.Documentation.Isolated; using Elastic.Documentation.LinkIndex; @@ -221,8 +222,11 @@ public async Task BuildAll( crossLinkResolver = new CrossLinkResolver(FetchedCrossLinks.Empty, uriResolver); } + // Prefetch CDN-hosted release notes for products declared under `release_notes` in docset.yml. + var releaseNotesResolver = await ReleaseNotesFetcher.PrefetchAsync(buildContext, logFactory, ctx); + // Create documentation set - var documentationSet = new DocumentationSet(buildContext, logFactory, crossLinkResolver); + var documentationSet = new DocumentationSet(buildContext, logFactory, crossLinkResolver, releaseNotesResolver); await documentationSet.ResolveDirectoryTree(ctx); return new CodexDocumentationSetBuildContext(checkout, buildContext, documentationSet); diff --git a/src/Elastic.Documentation.Configuration/Builder/ConfigurationFile.cs b/src/Elastic.Documentation.Configuration/Builder/ConfigurationFile.cs index 8a94b087b7..95a40fb6dd 100644 --- a/src/Elastic.Documentation.Configuration/Builder/ConfigurationFile.cs +++ b/src/Elastic.Documentation.Configuration/Builder/ConfigurationFile.cs @@ -38,6 +38,13 @@ public record ConfigurationFile /// public CrossLinkEntry[] CrossLinkEntries { get; } = []; + /// + /// Canonical product ids declared under release_notes whose changelog content is sourced from + /// the public CDN. Validated against products.yml; drives startup prefetch for the {changelog} + /// directive and CDN sourcing for changelog bundle. + /// + public string[] ReleaseNotesProducts { get; } = []; + /// The maximum depth `toc.yml` files may appear public int MaxTocDepth { get; } = 1; @@ -138,6 +145,9 @@ public ConfigurationFile(DocumentationSetFile docSetFile, IDocumentationSetConte CrossLinkRepositories = CrossLinkEntries.Select(e => e.Repository).ToArray(); + // Parse and validate CDN-backed release-notes product declarations + ReleaseNotesProducts = ParseReleaseNotesProducts(docSetFile.ReleaseNotes, productsConfig, context); + // Extensions - assuming they're not in DocumentationSetFile yet Extensions = new EnabledExtensions(docSetFile.Extensions); @@ -371,6 +381,57 @@ private static BrandingConfiguration ValidateBranding(BrandingConfiguration bran return imagePath; } + private static string[] ParseReleaseNotesProducts( + IReadOnlyList references, + ProductsConfiguration productsConfig, + IDocumentationSetContext context) + { + if (references.Count == 0) + return []; + + var products = new List(references.Count); + foreach (var reference in references) + { + var product = reference.Product?.Trim(); + if (string.IsNullOrEmpty(product)) + { + context.EmitError(context.ConfigurationPath, "A 'release_notes' entry is missing a 'product' value."); + continue; + } + + if (!IsValidProductId(product)) + { + context.EmitError(context.ConfigurationPath, + $"Invalid 'release_notes' product '{product}'. Product ids must match [a-zA-Z0-9_-]+."); + continue; + } + + // products.yml keys are hyphenated; accept the underscore variant for parity with `products`. + var normalized = product.Replace('_', '-'); + if (!productsConfig.Products.TryGetValue(normalized, out var resolved)) + { + context.EmitError(context.ConfigurationPath, + $"Unknown 'release_notes' product '{product}'. It must be a product id defined in products.yml."); + continue; + } + + if (!resolved.Features.ReleaseNotes) + { + context.EmitError(context.ConfigurationPath, + $"Product '{product}' declared in 'release_notes' does not participate in the release-notes system (it lacks the 'release-notes' feature in products.yml)."); + continue; + } + + if (!products.Contains(resolved.Id, StringComparer.Ordinal)) + products.Add(resolved.Id); + } + + return [.. products]; + } + + private static bool IsValidProductId(string product) => + product.Length > 0 && product.All(c => char.IsAsciiLetterOrDigit(c) || c is '_' or '-'); + private static CrossLinkEntry? ParseCrossLinkEntry(string raw, DocSetRegistry docsetRegistry, IFileInfo configPath, IDocumentationContext context) { DocSetRegistry entryRegistry; diff --git a/src/Elastic.Documentation.Configuration/Elastic.Documentation.Configuration.csproj b/src/Elastic.Documentation.Configuration/Elastic.Documentation.Configuration.csproj index 0e2c924272..ff43a62399 100644 --- a/src/Elastic.Documentation.Configuration/Elastic.Documentation.Configuration.csproj +++ b/src/Elastic.Documentation.Configuration/Elastic.Documentation.Configuration.csproj @@ -16,6 +16,12 @@ + + + <_Parameter1>Elastic.Markdown.Tests + + + diff --git a/src/Elastic.Documentation.Configuration/ReleaseNotes/BundleLoader.cs b/src/Elastic.Documentation.Configuration/ReleaseNotes/BundleLoader.cs index 9dabf2d9ed..3137ed6f10 100644 --- a/src/Elastic.Documentation.Configuration/ReleaseNotes/BundleLoader.cs +++ b/src/Elastic.Documentation.Configuration/ReleaseNotes/BundleLoader.cs @@ -58,6 +58,64 @@ public IReadOnlyList LoadBundles( return loadedBundles; } + /// + /// Loads bundles from in-memory YAML content rather than a folder. Used by the changelog + /// directive in cdn: mode, where bundle files are fetched over HTTP. CDN bundles are + /// self-contained (entries are inline), so no entry-file resolution against the filesystem occurs; + /// any file-only entry reference is skipped with a warning. Amend files are still merged by name. + /// + /// Bundle file name and raw YAML content pairs. + /// Callback to emit warnings during loading. + /// A list of successfully loaded bundles. + public IReadOnlyList LoadBundlesFromContent( + IReadOnlyList<(string FileName, string Content)> bundles, + Action emitWarning) + { + var loadedBundles = new List(bundles.Count); + + foreach (var (fileName, content) in bundles) + { + Bundle bundleData; + try + { + bundleData = ReleaseNotesSerialization.DeserializeBundle(content); + } + catch (YamlException e) + { + emitWarning($"Failed to parse changelog bundle '{fileName}': {e.Message}"); + continue; + } + + var version = GetVersionFromBundle(bundleData) ?? fileSystem.Path.GetFileNameWithoutExtension(fileName); + var repo = GetRepoFromBundle(bundleData); + var owner = GetOwnerFromBundle(bundleData); + var entries = ResolveInlineEntries(bundleData, fileName, emitWarning); + + loadedBundles.Add(new LoadedBundle(version, repo, owner, bundleData, fileName, entries)); + } + + return MergeAmendFiles(loadedBundles); + } + + /// Resolves only inline entries; file-only references (unresolvable without the changelog dir) are skipped with a warning. + private static List ResolveInlineEntries(Bundle bundleData, string fileName, Action emitWarning) + { + var entries = new List(bundleData.Entries.Count); + foreach (var entry in bundleData.Entries) + { + if (!string.IsNullOrWhiteSpace(entry.Title) && entry.Type != null) + { + entries.Add(ReleaseNotesSerialization.ConvertBundledEntry(entry)); + continue; + } + + emitWarning( + $"Bundle '{fileName}' has a non-self-contained entry (file '{entry.File?.Name}'); CDN bundles must inline their entries. Skipping."); + } + + return entries; + } + /// /// Resolves entries from a bundle, loading from file references if needed. /// @@ -132,7 +190,7 @@ public IReadOnlyList FilterEntries( /// /// The sorted list of bundles to merge. /// A list of bundles where same-target bundles are merged. - public IReadOnlyList MergeBundlesByTarget(IReadOnlyList bundles) + public static IReadOnlyList MergeBundlesByTarget(IReadOnlyList bundles) { if (bundles.Count <= 1) return bundles; diff --git a/src/Elastic.Documentation.Configuration/ReleaseNotes/CdnChangelogFetcher.cs b/src/Elastic.Documentation.Configuration/ReleaseNotes/CdnChangelogFetcher.cs new file mode 100644 index 0000000000..21d33b3e1f --- /dev/null +++ b/src/Elastic.Documentation.Configuration/ReleaseNotes/CdnChangelogFetcher.cs @@ -0,0 +1,212 @@ +// Licensed to Elasticsearch B.V under one or more agreements. +// Elasticsearch B.V licenses this file to you under the Apache 2.0 License. +// See the LICENSE file in the project root for more information + +using System.IO.Abstractions; +using System.Net; +using System.Text.Json; +using Elastic.Documentation.ReleaseNotes; +using Microsoft.Extensions.Logging; + +namespace Elastic.Documentation.Configuration.ReleaseNotes; + +/// +/// Fetches changelog bundles for a single product from the public CDN. It reads +/// {base}/{product}/registry.json to enumerate bundles, downloads each +/// {base}/{product}/bundle/{file}, and parses them via +/// . +/// +/// +/// +/// This is a pure async fetch engine: it owns no caching. Memoization is the caller's concern (see +/// , which fetches all declared products once at startup and stores +/// the result in an immutable ). +/// +/// +/// Resilience follows the manifest's consistency model: a registry that cannot be fetched or parsed +/// is a hard error (an empty list is returned and the caller's emit-error callback is invoked), while +/// an individual bundle that 404s or fails to parse is a warning and is skipped — the index can +/// legitimately list a bundle whose scrubbed copy is not yet on the public bucket. +/// +/// +public sealed class CdnChangelogFetcher : IDisposable +{ + private const int SupportedSchemaVersion = 1; + + /// + /// Bounds an individual registry/bundle HTTP request so a stalled CDN connection cannot hang a build. + /// + private static readonly TimeSpan FetchTimeout = TimeSpan.FromSeconds(30); + + /// + /// Process-wide client shared by every fetcher built for the production (no injected handler) path. + /// is thread-safe and intended to be long-lived; a single static instance avoids + /// leaking a socket handle per fetch, and + /// bounds DNS staleness in long-lived serve/watch runs. It is intentionally never disposed — it + /// lives for the lifetime of the process. + /// + private static readonly HttpClient SharedHttpClient = new( + new SocketsHttpHandler + { + AutomaticDecompression = DecompressionMethods.All, + PooledConnectionLifetime = TimeSpan.FromMinutes(5) + }) + { Timeout = FetchTimeout }; + + private readonly ILogger _logger; + private readonly HttpClient _httpClient; + private readonly BundleLoader _bundleLoader; + + /// + /// Non-null only when a caller injects its own (tests): in that case we + /// own a per-instance client and must dispose it. On the production path points + /// at , which is never disposed. + /// + private readonly HttpClient? _ownedHttpClient; + + public CdnChangelogFetcher(ILoggerFactory logFactory, IFileSystem fileSystem, HttpMessageHandler? handler = null) + { + _logger = logFactory.CreateLogger(); + _bundleLoader = new BundleLoader(fileSystem); + + if (handler is null) + _httpClient = SharedHttpClient; + else + { + // disposeHandler: false — the injected handler is owned by the caller (tests), not by us. + _ownedHttpClient = new HttpClient(handler, disposeHandler: false) { Timeout = FetchTimeout }; + _httpClient = _ownedHttpClient; + } + } + + /// + /// Returns the loaded bundles for from the CDN at . + /// Bundles are merged-by-amend but not yet merged-by-target or sorted (the caller owns presentation). + /// When is set, only the matching registry entry is downloaded. + /// Returns an empty list on a registry-level failure. + /// + public async Task> FetchAsync( + Uri baseUri, + string product, + string? version, + Action emitError, + Action emitWarning, + Cancel ctx) + { + var registryUri = Combine(baseUri, product, "registry.json"); + + ChangelogRegistry? registry; + try + { + registry = await FetchRegistryAsync(registryUri, ctx).ConfigureAwait(false); + } + catch (Exception ex) when (ex is not OperationCanceledException) + { + emitError($"Could not fetch changelog registry for product '{product}' from {registryUri}: {ex.Message}"); + return []; + } + + if (registry is null) + { + emitError($"Changelog registry for product '{product}' at {registryUri} was empty or unparseable."); + return []; + } + + if (registry.SchemaVersion > SupportedSchemaVersion) + { + emitError( + $"Changelog registry for product '{product}' uses schema version {registry.SchemaVersion}, but this build only understands version {SupportedSchemaVersion}. Update docs-builder."); + return []; + } + + var contents = await DownloadBundlesAsync(baseUri, product, version, registry, emitWarning, ctx).ConfigureAwait(false); + if (contents.Count == 0) + { + _logger.LogInformation("No usable changelog bundles fetched for {Product} from {BaseUri}", product, baseUri); + return []; + } + + return _bundleLoader.LoadBundlesFromContent(contents, emitWarning); + } + + private async Task FetchRegistryAsync(Uri registryUri, Cancel ctx) + { + _logger.LogInformation("Fetching changelog registry {RegistryUri}", registryUri); + using var request = new HttpRequestMessage(HttpMethod.Get, registryUri); + using var response = await _httpClient.SendAsync(request, ctx).ConfigureAwait(false); + _ = response.EnsureSuccessStatusCode(); + await using var stream = await response.Content.ReadAsStreamAsync(ctx).ConfigureAwait(false); + return await JsonSerializer.DeserializeAsync(stream, ChangelogRegistryJsonContext.Default.ChangelogRegistry, ctx).ConfigureAwait(false); + } + + private async Task> DownloadBundlesAsync( + Uri baseUri, + string product, + string? version, + ChangelogRegistry registry, + Action emitWarning, + Cancel ctx) + { + var contents = new List<(string FileName, string Content)>(registry.Bundles.Count); + foreach (var bundle in registry.Bundles) + { + ctx.ThrowIfCancellationRequested(); + + // When a single version is requested, only download the matching entry; the directive + // re-applies the same match after load, so this is purely a fetch optimization. + if (!string.IsNullOrWhiteSpace(version) && !ChangelogVersionMatch.Matches(version, bundle.Target, bundle.File)) + continue; + + var fileName = bundle.File; + if (string.IsNullOrWhiteSpace(fileName) || !IsSafeBundleFileName(fileName)) + { + emitWarning($"Changelog registry for '{product}' lists an invalid bundle file name '{fileName}'; skipping."); + continue; + } + + var bundleUri = Combine(baseUri, product, "bundle", fileName); + try + { + contents.Add((fileName, await FetchTextAsync(bundleUri, ctx).ConfigureAwait(false))); + } + catch (Exception ex) when (ex is not OperationCanceledException) + { + // The registry can reference a bundle whose scrubbed copy is not yet public; skip + warn. + emitWarning($"Could not fetch changelog bundle '{fileName}' for '{product}' from {bundleUri}: {ex.Message}"); + } + } + + return contents; + } + + private async Task FetchTextAsync(Uri uri, Cancel ctx) + { + using var request = new HttpRequestMessage(HttpMethod.Get, uri); + using var response = await _httpClient.SendAsync(request, ctx).ConfigureAwait(false); + _ = response.EnsureSuccessStatusCode(); + return await response.Content.ReadAsStringAsync(ctx).ConfigureAwait(false); + } + + private static Uri Combine(Uri baseUri, params string[] segments) + { + var basePath = baseUri.AbsoluteUri.TrimEnd('/'); + var suffix = string.Join('/', segments.Select(Uri.EscapeDataString)); + return new Uri($"{basePath}/{suffix}"); + } + + /// Guards against registry-supplied path traversal: a bundle file name must be a single path segment. + private static bool IsSafeBundleFileName(string fileName) => + !fileName.Contains('/', StringComparison.Ordinal) + && !fileName.Contains('\\', StringComparison.Ordinal) + && fileName is not ("." or ".."); + + /// + /// Disposes the per-instance created for an injected handler. The shared + /// production client () is process-lived and intentionally not disposed. + /// + public void Dispose() + { + _ownedHttpClient?.Dispose(); + GC.SuppressFinalize(this); + } +} diff --git a/src/Elastic.Documentation.Configuration/ReleaseNotes/ChangelogCdn.cs b/src/Elastic.Documentation.Configuration/ReleaseNotes/ChangelogCdn.cs new file mode 100644 index 0000000000..0b654cfbc4 --- /dev/null +++ b/src/Elastic.Documentation.Configuration/ReleaseNotes/ChangelogCdn.cs @@ -0,0 +1,37 @@ +// Licensed to Elasticsearch B.V under one or more agreements. +// Elasticsearch B.V licenses this file to you under the Apache 2.0 License. +// See the LICENSE file in the project root for more information + +namespace Elastic.Documentation.Configuration.ReleaseNotes; + +/// +/// Single source of truth for the public changelog CDN base URL used by both the changelog +/// directive (:cdn: mode) and the changelog bundle command (CDN entry sourcing). +/// +public static class ChangelogCdn +{ + /// + /// Environment variable that overrides the changelog CDN base URL (staging/local/testing). + /// + public const string BaseUrlEnvironmentVariable = "DOCS_BUILDER_CHANGELOG_CDN"; + + /// + /// Default public CDN base for changelog content (CloudFront in front of the public S3 bucket). + /// Overridable via . + /// + public const string DefaultBaseUrl = "https://d10xozp44eyz7q.cloudfront.net"; + + /// + /// Resolves the configured CDN base URI, honoring and + /// falling back to . Returns null when the configured value is not a + /// valid absolute http(s) URL. + /// + public static Uri? ResolveBaseUri() + { + var configured = Environment.GetEnvironmentVariable(BaseUrlEnvironmentVariable); + var raw = string.IsNullOrWhiteSpace(configured) ? DefaultBaseUrl : configured; + return Uri.TryCreate(raw, UriKind.Absolute, out var uri) && uri.Scheme is "http" or "https" + ? uri + : null; + } +} diff --git a/src/Elastic.Documentation.Configuration/ReleaseNotes/ChangelogRegistry.cs b/src/Elastic.Documentation.Configuration/ReleaseNotes/ChangelogRegistry.cs new file mode 100644 index 0000000000..cccbf489b3 --- /dev/null +++ b/src/Elastic.Documentation.Configuration/ReleaseNotes/ChangelogRegistry.cs @@ -0,0 +1,48 @@ +// Licensed to Elasticsearch B.V under one or more agreements. +// Elasticsearch B.V licenses this file to you under the Apache 2.0 License. +// See the LICENSE file in the project root for more information + +using System.Text.Json.Serialization; + +namespace Elastic.Documentation.Configuration.ReleaseNotes; + +/// +/// Consumer-side view of the per-product {product}/registry.json manifest published +/// alongside scrubbed changelog bundles. Mirrors the producer's shape (see the changelog upload +/// service) but is intentionally lenient: only the fields the changelog directive needs to +/// enumerate bundles are declared, and nothing is required so a partially-written or +/// future-versioned manifest still deserializes. +/// +public sealed record ChangelogRegistry +{ + /// Manifest schema version. A higher major than the consumer understands is rejected upstream. + public int SchemaVersion { get; init; } = 1; + + /// Product identifier the manifest belongs to (matches the first S3 key segment). + public string? Product { get; init; } + + /// Bundles known for this product, newest first as written by the producer. + public IReadOnlyList Bundles { get; init; } = []; +} + +/// One entry in . +public sealed record ChangelogRegistryBundle +{ + /// Bundle file name, resolved at {product}/bundle/{file} on the CDN. + public string? File { get; init; } + + /// Target version or release date declared by the bundle (e.g. 9.3.0). + public string? Target { get; init; } + + /// + /// Best-effort change hint from the private bucket (pre-scrub). Not valid for public-object + /// integrity or HTTP cache validation; see the producer's documentation. Unused by the directive + /// today but retained for fidelity and future cache-keying. + /// + public string? ETag { get; init; } +} + +[JsonSourceGenerationOptions(PropertyNamingPolicy = JsonKnownNamingPolicy.SnakeCaseLower)] +[JsonSerializable(typeof(ChangelogRegistry))] +[JsonSerializable(typeof(ChangelogRegistryBundle))] +internal sealed partial class ChangelogRegistryJsonContext : JsonSerializerContext; diff --git a/src/Elastic.Documentation.Configuration/ReleaseNotes/FetchedReleaseNotes.cs b/src/Elastic.Documentation.Configuration/ReleaseNotes/FetchedReleaseNotes.cs new file mode 100644 index 0000000000..1a49ccda50 --- /dev/null +++ b/src/Elastic.Documentation.Configuration/ReleaseNotes/FetchedReleaseNotes.cs @@ -0,0 +1,28 @@ +// Licensed to Elasticsearch B.V under one or more agreements. +// Elasticsearch B.V licenses this file to you under the Apache 2.0 License. +// See the LICENSE file in the project root for more information + +using System.Collections.Frozen; +using Elastic.Documentation.ReleaseNotes; + +namespace Elastic.Documentation.Configuration.ReleaseNotes; + +/// +/// The immutable result of prefetching CDN-hosted changelog bundles for every product declared under +/// release_notes in docset.yml. Built once at startup by and +/// consumed by the {changelog} directive via . +/// +public sealed record FetchedReleaseNotes +{ + /// Loaded bundles per product id. A declared product with no usable bundles maps to an empty list. + public required FrozenDictionary> BundlesByProduct { get; init; } + + /// Product ids declared under release_notes, used to distinguish "undeclared" from "declared but empty". + public required FrozenSet DeclaredProducts { get; init; } + + public static FetchedReleaseNotes Empty { get; } = new() + { + BundlesByProduct = FrozenDictionary>.Empty, + DeclaredProducts = [] + }; +} diff --git a/src/Elastic.Documentation.Configuration/ReleaseNotes/IReleaseNotesResolver.cs b/src/Elastic.Documentation.Configuration/ReleaseNotes/IReleaseNotesResolver.cs new file mode 100644 index 0000000000..ad4738b693 --- /dev/null +++ b/src/Elastic.Documentation.Configuration/ReleaseNotes/IReleaseNotesResolver.cs @@ -0,0 +1,74 @@ +// Licensed to Elasticsearch B.V under one or more agreements. +// Elasticsearch B.V licenses this file to you under the Apache 2.0 License. +// See the LICENSE file in the project root for more information + +using Elastic.Documentation.ReleaseNotes; + +namespace Elastic.Documentation.Configuration.ReleaseNotes; + +/// +/// Resolves prefetched CDN changelog bundles for the {changelog} directive's :cdn: mode. +/// Mirrors the cross-link resolver pattern: the build fetches all declared products up front and the +/// directive only reads the already-loaded in-memory set — no per-directive HTTP. +/// +public interface IReleaseNotesResolver +{ + /// Whether was declared under release_notes in docset.yml. + bool IsDeclared(string product); + + /// + /// Gets the prefetched bundles for . Returns false when the product was not + /// declared (or not fetched); a declared product with no usable bundles returns true with an empty list. + /// + bool TryGetBundles(string product, out IReadOnlyList bundles); +} + +/// +/// Default resolver for build paths that do not source release notes from the CDN (tests, refactor +/// tooling). Every product is treated as undeclared so a stray :cdn: directive emits a clear error. +/// +public sealed class NoopReleaseNotesResolver : IReleaseNotesResolver +{ + public static NoopReleaseNotesResolver Instance { get; } = new(); + + private NoopReleaseNotesResolver() { } + + /// + public bool IsDeclared(string product) => false; + + /// + public bool TryGetBundles(string product, out IReadOnlyList bundles) + { + bundles = []; + return false; + } +} + +/// +/// Resolver backed by an immutable . The backing set can be populated +/// after construction () so the assembler can share a single resolver across all +/// documentation sets and fill it once every docset.yml has been parsed. +/// +public sealed class ReleaseNotesResolver(FetchedReleaseNotes? fetched = null) : IReleaseNotesResolver +{ + private FetchedReleaseNotes _fetched = fetched ?? FetchedReleaseNotes.Empty; + + /// Replaces the backing set. Used by the assembler's two-phase startup fetch. + public void Populate(FetchedReleaseNotes fetched) => _fetched = fetched; + + /// + public bool IsDeclared(string product) => _fetched.DeclaredProducts.Contains(product); + + /// + public bool TryGetBundles(string product, out IReadOnlyList bundles) + { + if (_fetched.BundlesByProduct.TryGetValue(product, out var found)) + { + bundles = found; + return true; + } + + bundles = []; + return false; + } +} diff --git a/src/Elastic.Documentation.Configuration/ReleaseNotes/ReleaseNotesFetcher.cs b/src/Elastic.Documentation.Configuration/ReleaseNotes/ReleaseNotesFetcher.cs new file mode 100644 index 0000000000..42a7be088f --- /dev/null +++ b/src/Elastic.Documentation.Configuration/ReleaseNotes/ReleaseNotesFetcher.cs @@ -0,0 +1,90 @@ +// Licensed to Elasticsearch B.V under one or more agreements. +// Elasticsearch B.V licenses this file to you under the Apache 2.0 License. +// See the LICENSE file in the project root for more information + +using System.Collections.Frozen; +using System.IO.Abstractions; +using Elastic.Documentation.Diagnostics; +using Elastic.Documentation.ReleaseNotes; +using Microsoft.Extensions.Logging; + +namespace Elastic.Documentation.Configuration.ReleaseNotes; + +/// +/// Prefetches CDN changelog bundles for every product declared under release_notes at build +/// startup, concurrently, mirroring how cross-links are fetched. Any failure to read a product's +/// registry is emitted as an error on the collector (strict fail-fast): a declared product whose +/// content cannot be sourced fails the build rather than silently rendering an empty changelog. +/// +public sealed class ReleaseNotesFetcher(ILoggerFactory logFactory, IFileSystem fileSystem, HttpMessageHandler? handler = null) +{ + private readonly ILoggerFactory _logFactory = logFactory; + private readonly IFileSystem _fileSystem = fileSystem; + private readonly HttpMessageHandler? _handler = handler; + private readonly ILogger _logger = logFactory.CreateLogger(); + + /// + /// Prefetches release notes for the products declared in 's docset.yml and + /// returns a ready resolver. Returns the no-op resolver when nothing is declared (no network is hit). + /// + public static async Task PrefetchAsync(BuildContext context, ILoggerFactory logFactory, Cancel ctx) + { + var products = context.Configuration.ReleaseNotesProducts; + if (products.Length == 0) + return NoopReleaseNotesResolver.Instance; + + var fetcher = new ReleaseNotesFetcher(logFactory, context.ReadFileSystem); + var fetched = await fetcher.FetchAsync(context.Collector, products, ctx).ConfigureAwait(false); + return new ReleaseNotesResolver(fetched); + } + + public async Task FetchAsync(IDiagnosticsCollector collector, IReadOnlyCollection products, Cancel ctx) + { + var declared = products + .Where(p => !string.IsNullOrWhiteSpace(p)) + .Select(p => p.Trim()) + .Distinct(StringComparer.Ordinal) + .ToArray(); + + if (declared.Length == 0) + return FetchedReleaseNotes.Empty; + + var declaredSet = declared.ToFrozenSet(StringComparer.Ordinal); + + var baseUri = ChangelogCdn.ResolveBaseUri(); + if (baseUri is null) + { + collector.EmitError(string.Empty, + $"No valid changelog CDN base URL is configured. Set the {ChangelogCdn.BaseUrlEnvironmentVariable} environment variable to an absolute http(s) URL."); + return new FetchedReleaseNotes + { + BundlesByProduct = FrozenDictionary>.Empty, + DeclaredProducts = declaredSet + }; + } + + _logger.LogInformation("Fetching release notes for {Count} product(s) from {BaseUri}", declared.Length, baseUri); + + using var fetcher = new CdnChangelogFetcher(_logFactory, _fileSystem, _handler); + var tasks = declared.Select(async product => + { + // version: null — prefetch the full set; each directive applies its own :version: filter later. + var bundles = await fetcher.FetchAsync( + baseUri, + product, + version: null, + msg => collector.EmitError(string.Empty, msg), + msg => collector.EmitWarning(string.Empty, msg), + ctx).ConfigureAwait(false); + return (product, bundles); + }); + + var results = await Task.WhenAll(tasks).ConfigureAwait(false); + + return new FetchedReleaseNotes + { + BundlesByProduct = results.ToFrozenDictionary(r => r.product, r => r.bundles, StringComparer.Ordinal), + DeclaredProducts = declaredSet + }; + } +} diff --git a/src/Elastic.Documentation.Configuration/Serialization/YamlStaticContext.cs b/src/Elastic.Documentation.Configuration/Serialization/YamlStaticContext.cs index 7fc6e8f614..2ac2880c01 100644 --- a/src/Elastic.Documentation.Configuration/Serialization/YamlStaticContext.cs +++ b/src/Elastic.Documentation.Configuration/Serialization/YamlStaticContext.cs @@ -41,6 +41,7 @@ namespace Elastic.Documentation.Configuration.Serialization; [YamlSerializable(typeof(DocumentationSetFeatures))] [YamlSerializable(typeof(DocumentationSetStorybook))] [YamlSerializable(typeof(CodexDocSetMetadata))] +[YamlSerializable(typeof(ReleaseNotesProductReference))] [YamlSerializable(typeof(TableOfContentsFile))] [YamlSerializable(typeof(SiteNavigationFile))] [YamlSerializable(typeof(PhantomRegistration))] diff --git a/src/Elastic.Documentation.Configuration/Toc/DocumentationSetFile.cs b/src/Elastic.Documentation.Configuration/Toc/DocumentationSetFile.cs index f82f871504..c4008495ed 100644 --- a/src/Elastic.Documentation.Configuration/Toc/DocumentationSetFile.cs +++ b/src/Elastic.Documentation.Configuration/Toc/DocumentationSetFile.cs @@ -26,6 +26,14 @@ public class DocumentationSetFile : TableOfContentsFile [YamlMember(Alias = "cross_links")] public List CrossLinks { get; set; } = []; + /// + /// Products whose changelog content is sourced from the public CDN. Declaring a product here lets + /// docs-builder prefetch its bundles at startup (consumed by the {changelog} :cdn: mode) + /// and lets changelog bundle source that product's entries from the CDN. + /// + [YamlMember(Alias = "release_notes")] + public List ReleaseNotes { get; set; } = []; + [YamlMember(Alias = "exclude")] public List Exclude { get; set; } = []; @@ -750,6 +758,16 @@ public class DocumentationSetStorybook public string? Registry { get; set; } } +/// +/// A single release_notes entry declaring a product whose changelog content is CDN-backed. +/// +[YamlSerializable] +public record ReleaseNotesProductReference +{ + [YamlMember(Alias = "product")] + public string Product { get; set; } = string.Empty; +} + /// /// Codex-specific metadata. Only contains group for navigation grouping in a codex environment. /// diff --git a/src/Elastic.Documentation/GitCheckoutInformation.cs b/src/Elastic.Documentation/GitCheckoutInformation.cs index d1714428cd..60dcff703c 100644 --- a/src/Elastic.Documentation/GitCheckoutInformation.cs +++ b/src/Elastic.Documentation/GitCheckoutInformation.cs @@ -28,6 +28,10 @@ public record GitCheckoutInformation [JsonPropertyName("name")] public string RepositoryName { get; init; } = "unavailable"; + /// Whether git checkout information was resolved (false for the sentinel). + [JsonIgnore] + public bool IsAvailable => RepositoryName != Unavailable.RepositoryName; + /// /// The full git ref from GitHub Actions (e.g. refs/pull/123/merge). Set from GITHUB_REF when running in CI. /// diff --git a/src/Elastic.Documentation/ReleaseNotes/ChangelogVersionMatch.cs b/src/Elastic.Documentation/ReleaseNotes/ChangelogVersionMatch.cs new file mode 100644 index 0000000000..f05fa73531 --- /dev/null +++ b/src/Elastic.Documentation/ReleaseNotes/ChangelogVersionMatch.cs @@ -0,0 +1,36 @@ +// Licensed to Elasticsearch B.V under one or more agreements. +// Elasticsearch B.V licenses this file to you under the Apache 2.0 License. +// See the LICENSE file in the project root for more information + +namespace Elastic.Documentation.ReleaseNotes; + +/// +/// Shared matching rule for the changelog directive's :version: filter. A bundle +/// matches a requested version when the requested value equals either the bundle's declared target +/// or its file name (with or without extension), compared case-insensitively. Kept in one place so +/// the directive's post-load filter and the CDN fetcher's download-time filter agree. +/// +public static class ChangelogVersionMatch +{ + /// The user-supplied :version: value (already trimmed). + /// The bundle's declared target (may be null/empty). + /// The bundle file name or path (may be null/empty). + public static bool Matches(string requested, string? target, string? file) + { + // empty/whitespace → match all versions + if (string.IsNullOrWhiteSpace(requested)) + return true; + + var value = requested.Trim(); + + if (!string.IsNullOrWhiteSpace(target) && string.Equals(target.Trim(), value, StringComparison.OrdinalIgnoreCase)) + return true; + + if (string.IsNullOrWhiteSpace(file)) + return false; + + var name = Path.GetFileName(file); + return string.Equals(name, value, StringComparison.OrdinalIgnoreCase) + || string.Equals(Path.GetFileNameWithoutExtension(file), value, StringComparison.OrdinalIgnoreCase); + } +} diff --git a/src/Elastic.Markdown/IO/DocumentationSet.cs b/src/Elastic.Markdown/IO/DocumentationSet.cs index 9068220898..b9f1a80210 100644 --- a/src/Elastic.Markdown/IO/DocumentationSet.cs +++ b/src/Elastic.Markdown/IO/DocumentationSet.cs @@ -10,6 +10,7 @@ using Elastic.Documentation; using Elastic.Documentation.Configuration; using Elastic.Documentation.Configuration.Builder; +using Elastic.Documentation.Configuration.ReleaseNotes; using Elastic.Documentation.Configuration.Toc; using Elastic.Documentation.Configuration.Toc.CliReference; using Elastic.Documentation.Links; @@ -50,6 +51,8 @@ public class DocumentationSet : INavigationTraversable public ICrossLinkResolver CrossLinkResolver { get; } + public IReleaseNotesResolver ReleaseNotesResolver { get; } + public FrozenDictionary Files { get; } public ConditionalWeakTable NavigationDocumentationFileLookup { get; } @@ -59,7 +62,8 @@ public class DocumentationSet : INavigationTraversable public DocumentationSet( BuildContext context, ILoggerFactory logFactory, - ICrossLinkResolver linkResolver + ICrossLinkResolver linkResolver, + IReleaseNotesResolver? releaseNotesResolver = null ) { _logger = logFactory.CreateLogger(); @@ -67,12 +71,14 @@ ICrossLinkResolver linkResolver SourceDirectory = context.DocumentationSourceDirectory; OutputDirectory = context.OutputDirectory; CrossLinkResolver = linkResolver; + ReleaseNotesResolver = releaseNotesResolver ?? NoopReleaseNotesResolver.Instance; Configuration = context.Configuration; EnabledExtensions = InstantiateExtensions(); var resolver = new ParserResolvers { CrossLinkResolver = CrossLinkResolver, + ReleaseNotesResolver = ReleaseNotesResolver, TryFindDocument = TryFindDocument, TryFindDocumentByRelativePath = TryFindDocumentByRelativePath, NavigationTraversable = this diff --git a/src/Elastic.Markdown/Myst/Directives/Changelog/ChangelogBlock.cs b/src/Elastic.Markdown/Myst/Directives/Changelog/ChangelogBlock.cs index f92048a499..56c15e630a 100644 --- a/src/Elastic.Markdown/Myst/Directives/Changelog/ChangelogBlock.cs +++ b/src/Elastic.Markdown/Myst/Directives/Changelog/ChangelogBlock.cs @@ -98,6 +98,20 @@ public class ChangelogBlock(DirectiveBlockParser parser, ParserContext context) /// public bool Found { get; private set; } + /// + /// Product to source bundles for from the public CDN (the :cdn: option). When set, the + /// directive fetches {cdnBase}/{product}/registry.json and the bundles it lists + /// instead of reading a local folder; any folder argument is ignored. + /// + public string? CdnProduct { get; private set; } + + /// + /// Optional single target to render (the :version: option, e.g. 9.4.0 or a release + /// date). When set, only the bundle whose target/file matches is rendered; applies to both local + /// folder and CDN sources. In CDN mode it also limits which bundles are downloaded. + /// + public string? VersionFilter { get; private set; } + /// /// Loaded and parsed bundles, sorted by version (semver descending). /// @@ -184,7 +198,6 @@ public class ChangelogBlock(DirectiveBlockParser parser, ParserContext context) public override void FinalizeAndValidate(ParserContext context) { - ExtractBundlesFolderPath(); Subsections = PropBool("subsections"); ConfigPath = Prop("config"); var productOpt = Prop("product"); @@ -197,6 +210,36 @@ public override void FinalizeAndValidate(ParserContext context) LinkVisibility = ParseLinkVisibility(); DescriptionVisibility = ParseDescriptionVisibility(); DropdownsEnabled = PropBool("dropdowns"); + VersionFilter = Prop("version") is { Length: > 0 } v ? v.Trim() : null; + + if (Properties?.ContainsKey("cdn") == true) + { + // :cdn: takes an explicit product, or may be valueless to infer the product from the + // repository that holds the doc (the common case where the repo name is the product id). + var product = Prop("cdn") is { Length: > 0 } explicitProduct + ? explicitProduct.Trim() + : InferCdnProductFromRepository(); + + if (string.IsNullOrWhiteSpace(product)) + { + this.EmitError( + "The :cdn: product could not be inferred from the repository; specify it explicitly, e.g. ':cdn: elasticsearch'."); + return; + } + + // Validate before assigning so an invalid product name is never stored on the block. + if (!IsValidCdnProduct(product)) + { + this.EmitError($"Invalid :cdn: product '{product}'. Product names must match [a-zA-Z0-9_-]+."); + return; + } + + CdnProduct = product; + LoadCdnBundles(product); + return; + } + + ExtractBundlesFolderPath(); if (Found) LoadAndCacheBundles(); } @@ -445,15 +488,42 @@ private void LoadAndCacheBundles() BundlesFolderPath, msg => this.EmitError(msg)); + ApplyLoadedBundles(loadedBundles); + } + + private void LoadCdnBundles(string product) + { + // Product validity is checked by the caller before CdnProduct is assigned. + if (!string.IsNullOrWhiteSpace(Arguments)) + this.EmitWarning("The bundles folder argument is ignored when :cdn: is set; bundles are sourced from the CDN."); + + // :cdn: is a selector over release notes prefetched at build startup. A product must be declared + // under `release_notes` in docset.yml; otherwise its bundles were never fetched. + if (!Context.ReleaseNotesResolver.IsDeclared(product)) + { + this.EmitError( + $"The :cdn: product '{product}' is not declared in docset.yml. Add it under 'release_notes:', for example:\n release_notes:\n - product: {product}"); + return; + } + + _ = Context.ReleaseNotesResolver.TryGetBundles(product, out var loadedBundles); + ApplyLoadedBundles(loadedBundles); + Found = LoadedBundles.Count > 0; + } + + private void ApplyLoadedBundles(IReadOnlyList loadedBundles) + { + var filteredBundles = FilterByVersion(loadedBundles); + // Sort by version (descending - newest first) // Supports both semver (e.g., "9.3.0") and date-based (e.g., "2025-08-05") versions - var sortedBundles = loadedBundles + var sortedBundles = filteredBundles .OrderByDescending(b => VersionOrDate.Parse(b.Version)) .ToList(); // Always merge bundles with the same target version // (e.g., Cloud Serverless with multiple repos contributing to a single dated release) - LoadedBundles = loader.MergeBundlesByTarget(sortedBundles); + LoadedBundles = BundleLoader.MergeBundlesByTarget(sortedBundles); // Collect hide-features from all loaded bundles foreach (var bundle in LoadedBundles) @@ -463,6 +533,35 @@ private void LoadAndCacheBundles() } } + /// Filters bundles by the optional :version: value; warns and renders empty when nothing matches. + private IReadOnlyList FilterByVersion(IReadOnlyList bundles) + { + if (VersionFilter is not { Length: > 0 } version) + return bundles; + + var matched = bundles + .Where(b => ChangelogVersionMatch.Matches(version, b.Version, b.FilePath)) + .ToList(); + + if (matched.Count == 0 && bundles.Count > 0) + this.EmitWarning($"No changelog bundle matches :version: '{version}'."); + + return matched; + } + + private static bool IsValidCdnProduct(string product) => + product.Length > 0 && product.All(c => char.IsAsciiLetterOrDigit(c) || c is '_' or '-'); + + /// Infers the CDN product for a valueless :cdn: from the repo, mapped to its canonical id via products.yml. + private string? InferCdnProductFromRepository() + { + if (!Build.Git.IsAvailable) + return null; + + var repository = Build.Git.RepositoryName; + return Build.ProductsConfiguration.GetProductByRepositoryName(repository)?.Id ?? repository; + } + private IEnumerable ComputeGeneratedAnchors() { var dedicatedPage = ChangelogInlineRenderer.IsDedicatedSeparatedTypePage(TypeFilter); diff --git a/src/Elastic.Markdown/Myst/Directives/DirectiveHtmlRenderer.cs b/src/Elastic.Markdown/Myst/Directives/DirectiveHtmlRenderer.cs index efbcb96333..50e0ed1e88 100644 --- a/src/Elastic.Markdown/Myst/Directives/DirectiveHtmlRenderer.cs +++ b/src/Elastic.Markdown/Myst/Directives/DirectiveHtmlRenderer.cs @@ -671,7 +671,12 @@ private static HtmlString RenderInlineMarkdown(ParagraphBlock paragraph) private static void WriteChangelogBlock(HtmlRenderer renderer, ChangelogBlock block) { - if (!block.Found || block.BundlesFolderPath is null) + if (!block.Found) + return; + + // Local-folder mode must also have resolved a bundles folder; CDN-sourced bundles never set one. + var isCdnSourced = !string.IsNullOrWhiteSpace(block.CdnProduct); + if (!isCdnSourced && block.BundlesFolderPath is null) return; var markdown = ChangelogInlineRenderer.RenderChangelogMarkdown(block); diff --git a/src/Elastic.Markdown/Myst/MarkdownParser.cs b/src/Elastic.Markdown/Myst/MarkdownParser.cs index 309ae7a578..4059a0ab05 100644 --- a/src/Elastic.Markdown/Myst/MarkdownParser.cs +++ b/src/Elastic.Markdown/Myst/MarkdownParser.cs @@ -48,6 +48,7 @@ private Task ParseFromFile(IFileInfo path, YamlFrontMatter? ma TryFindDocument = Resolvers.TryFindDocument, TryFindDocumentByRelativePath = Resolvers.TryFindDocumentByRelativePath, CrossLinkResolver = Resolvers.CrossLinkResolver, + ReleaseNotesResolver = Resolvers.ReleaseNotesResolver, NavigationTraversable = Resolvers.NavigationTraversable, SkipValidation = skip }; @@ -88,7 +89,8 @@ public static MarkdownDocument ParseMarkdownStringAsync(BuildContext build, IPar TryFindDocument = resolvers.TryFindDocument, TryFindDocumentByRelativePath = resolvers.TryFindDocumentByRelativePath, NavigationTraversable = resolvers.NavigationTraversable, - CrossLinkResolver = resolvers.CrossLinkResolver + CrossLinkResolver = resolvers.CrossLinkResolver, + ReleaseNotesResolver = resolvers.ReleaseNotesResolver }; var context = new ParserContext(state); @@ -108,6 +110,7 @@ public static Task ParseSnippetAsync(BuildContext build, IPars TryFindDocument = resolvers.TryFindDocument, TryFindDocumentByRelativePath = resolvers.TryFindDocumentByRelativePath, CrossLinkResolver = resolvers.CrossLinkResolver, + ReleaseNotesResolver = resolvers.ReleaseNotesResolver, NavigationTraversable = resolvers.NavigationTraversable, ParentMarkdownPath = parentPath, IncludeLine = includeLine diff --git a/src/Elastic.Markdown/Myst/ParserContext.cs b/src/Elastic.Markdown/Myst/ParserContext.cs index a4a5fce375..d3a0a694ad 100644 --- a/src/Elastic.Markdown/Myst/ParserContext.cs +++ b/src/Elastic.Markdown/Myst/ParserContext.cs @@ -6,6 +6,7 @@ using Elastic.Documentation; using Elastic.Documentation.Configuration; using Elastic.Documentation.Configuration.Builder; +using Elastic.Documentation.Configuration.ReleaseNotes; using Elastic.Documentation.Links.CrossLinks; using Elastic.Documentation.Navigation; using Elastic.Documentation.Site; @@ -31,6 +32,7 @@ processor.Context as ParserContext public interface IParserResolvers { ICrossLinkResolver CrossLinkResolver { get; } + IReleaseNotesResolver ReleaseNotesResolver { get; } Func TryFindDocument { get; } Func TryFindDocumentByRelativePath { get; } INavigationTraversable NavigationTraversable { get; } @@ -40,6 +42,12 @@ public record ParserResolvers : IParserResolvers { public required ICrossLinkResolver CrossLinkResolver { get; init; } + /// + /// Resolver for prefetched CDN changelog bundles. Defaults to a no-op so build paths that do not + /// source release notes from the CDN (tests, refactor tooling) don't have to set it. + /// + public IReleaseNotesResolver ReleaseNotesResolver { get; init; } = NoopReleaseNotesResolver.Instance; + public required Func TryFindDocument { get; init; } public required Func TryFindDocumentByRelativePath { get; init; } @@ -74,6 +82,7 @@ public class ParserContext : MarkdownParserContext, IParserResolvers { public ConfigurationFile Configuration { get; } public ICrossLinkResolver CrossLinkResolver { get; } + public IReleaseNotesResolver ReleaseNotesResolver { get; } public IFileInfo MarkdownSourcePath { get; } public IFileInfo? MarkdownParentPath { get; } public string CurrentUrlPath { get; } @@ -111,6 +120,7 @@ public ParserContext(ParserState state) OriginalSourcePath = state.OriginalSourcePath; CrossLinkResolver = state.CrossLinkResolver; + ReleaseNotesResolver = state.ReleaseNotesResolver; MarkdownSourcePath = state.MarkdownSourcePath; TryFindDocument = state.TryFindDocument; TryFindDocumentByRelativePath = state.TryFindDocumentByRelativePath; diff --git a/src/infra/docs-lambda-changelog-scrubber/Program.cs b/src/infra/docs-lambda-changelog-scrubber/Program.cs index c49d84c498..9ce65c7316 100644 --- a/src/infra/docs-lambda-changelog-scrubber/Program.cs +++ b/src/infra/docs-lambda-changelog-scrubber/Program.cs @@ -13,6 +13,7 @@ using Amazon.S3.Model; using Amazon.S3.Util; using Elastic.Changelog.Bundling; +using Elastic.Changelog.Uploading; using Elastic.Documentation.Configuration.Assembler; using Elastic.Documentation.Configuration.ReleaseNotes; using Elastic.Documentation.Diagnostics; @@ -121,8 +122,7 @@ async Task ScrubAndCopyToPublicBucket(IAmazonS3 s3Client, string sourceBucket, s { context.Logger.LogDebug("Scrubbing {Key} to public bucket", key); - var fileName = Path.GetFileName(key); - if (string.Equals(fileName, "registry-index.json", StringComparison.OrdinalIgnoreCase)) + if (RegistryKey.IsRegistry(key)) { await CopyPassThrough(s3Client, sourceBucket, key, context); return; diff --git a/src/infra/docs-lambda-changelog-scrubber/README.md b/src/infra/docs-lambda-changelog-scrubber/README.md index 6ade40a790..42551ee0da 100644 --- a/src/infra/docs-lambda-changelog-scrubber/README.md +++ b/src/infra/docs-lambda-changelog-scrubber/README.md @@ -31,7 +31,7 @@ The `bootstrap` binary should be available under: ## Event handling - **`s3:ObjectCreated:*`** on `.yaml`/`.yml` files: read from private bucket, scrub private references, write to public bucket -- **`s3:ObjectCreated:*`** on `.json` files: copy as-is (pass-through for `registry-index.json`) +- **`s3:ObjectCreated:*`** on `.json` files: only per-product registry manifests (keys matching `RegistryKey.IsRegistry`, i.e. `{product}/registry.json`) are passed through as-is; any other `.json` key is skipped - **`s3:ObjectRemoved:*`**: delete the same key from the public bucket - Other keys are silently skipped diff --git a/src/services/Elastic.Changelog/Uploading/ChangelogUploadService.cs b/src/services/Elastic.Changelog/Uploading/ChangelogUploadService.cs index 0852dc9c87..69761a1560 100644 --- a/src/services/Elastic.Changelog/Uploading/ChangelogUploadService.cs +++ b/src/services/Elastic.Changelog/Uploading/ChangelogUploadService.cs @@ -94,9 +94,38 @@ public async Task Upload(IDiagnosticsCollector collector, ChangelogUploadA if (result.Failed > 0) collector.EmitError(string.Empty, $"{result.Failed} file(s) failed to upload"); + // On a successful bundle upload, refresh the per-product registry.json so consumers + // (e.g. the changelog directive in cdn: mode) can enumerate bundles without an S3 listing. + // Failures here are logged but don't fail the upload — the bundles themselves are already in S3. + if (result.Failed == 0 && args.ArtifactType == ArtifactType.Bundle && targets.Count > 0) + await RefreshRegistries(collector, client, etagCalculator, args, targets, ctx); + return result.Failed == 0; } + private async Task RefreshRegistries( + IDiagnosticsCollector collector, + IAmazonS3 client, + IS3EtagCalculator etagCalculator, + ChangelogUploadArguments args, + IReadOnlyList bundleTargets, + Cancel ctx) + { + try + { + var builder = new RegistryBuilder(logFactory, _fileSystem, client, etagCalculator, args.S3BucketName); + var result = await builder.RefreshAsync(collector, bundleTargets, ctx); + _logger.LogInformation("Registry refresh: {Updated} updated, {Unchanged} unchanged, {Failed} failed", + result.Updated, result.Unchanged, result.Failed); + } + catch (Exception ex) when (ex is not OperationCanceledException) + { + // Leaving the manifest stale is non-fatal — bundle objects are unaffected. + _logger.LogWarning(ex, "Registry refresh failed; bundles uploaded successfully but manifests may be stale"); + collector.EmitWarning(string.Empty, $"Failed to refresh registry manifest(s): {ex.Message}"); + } + } + internal IReadOnlyList DiscoverUploadTargets(IDiagnosticsCollector collector, string changelogDir) { var rootDir = _fileSystem.DirectoryInfo.New(changelogDir); diff --git a/src/services/Elastic.Changelog/Uploading/Registry.cs b/src/services/Elastic.Changelog/Uploading/Registry.cs new file mode 100644 index 0000000000..e597ed908e --- /dev/null +++ b/src/services/Elastic.Changelog/Uploading/Registry.cs @@ -0,0 +1,81 @@ +// Licensed to Elasticsearch B.V under one or more agreements. +// Elasticsearch B.V licenses this file to you under the Apache 2.0 License. +// See the LICENSE file in the project root for more information + +using System.Text.Json.Serialization; + +namespace Elastic.Changelog.Uploading; + +/// +/// Per-product manifest published alongside scrubbed changelog bundles. +/// Lets consumers (e.g. the changelog directive in cdn: mode) enumerate +/// bundle files without an S3 listing call. +/// +/// +/// Stored at {product}/registry.json in the changelog bundles bucket. +/// The scrubber Lambda mirrors it verbatim to the public bucket (pass-through). +/// +public sealed record Registry +{ + /// + /// Manifest schema version. Incremented when consumers must change their parser. + /// + public int SchemaVersion { get; init; } = 1; + + /// + /// Product identifier (matches the first segment of the S3 key). + /// + public required string Product { get; init; } + + /// + /// Time the manifest was last regenerated, in UTC. + /// + public required DateTimeOffset GeneratedAt { get; init; } + + /// + /// Bundles currently known for this product, sorted by + /// descending (newest first), with a deterministic tiebreak on . + /// + public required IReadOnlyList Bundles { get; init; } +} + +/// +/// One entry in . +/// +public sealed record RegistryBundle +{ + /// + /// Bundle file name (e.g. 9.3.0.yaml or 2025-11.yaml), + /// resolved at {product}/bundle/{file}. + /// + public required string File { get; init; } + + /// + /// Target version or release date as declared in the bundle's first product + /// (e.g. 9.3.0 or 2025-11-01). May be null if the bundle declares no products. + /// + public string? Target { get; init; } + + /// + /// S3 ETag of the bundle object as uploaded to the private bundles bucket (pre-scrub). + /// For single-part uploads smaller than + /// this is the MD5 of the body. + /// + /// + /// Best-effort identity / change hint only. The public (CDN) object is produced by the changelog + /// scrubber Lambda, which rewrites any bundle that contains private references — so for scrubbed + /// bundles this value will not match the public object's ETag. Consumers MUST NOT use it + /// for integrity checks or HTTP cache validation against the public bucket; use the CDN response's + /// own ETag for that. It is safe to use to detect whether a bundle changed between manifest reads. + /// + public required string ETag { get; init; } +} + +[JsonSourceGenerationOptions( + WriteIndented = true, + PropertyNamingPolicy = JsonKnownNamingPolicy.SnakeCaseLower, + DefaultIgnoreCondition = JsonIgnoreCondition.WhenWritingNull +)] +[JsonSerializable(typeof(Registry))] +[JsonSerializable(typeof(RegistryBundle))] +public sealed partial class RegistryJsonContext : JsonSerializerContext; diff --git a/src/services/Elastic.Changelog/Uploading/RegistryBuilder.cs b/src/services/Elastic.Changelog/Uploading/RegistryBuilder.cs new file mode 100644 index 0000000000..ccaf0f3e19 --- /dev/null +++ b/src/services/Elastic.Changelog/Uploading/RegistryBuilder.cs @@ -0,0 +1,301 @@ +// Licensed to Elasticsearch B.V under one or more agreements. +// Elasticsearch B.V licenses this file to you under the Apache 2.0 License. +// See the LICENSE file in the project root for more information + +using System.IO.Abstractions; +using System.Net; +using System.Text.Json; +using Amazon.S3; +using Amazon.S3.Model; +using Elastic.Documentation.Configuration.ReleaseNotes; +using Elastic.Documentation.Diagnostics; +using Elastic.Documentation.Integrations.S3; +using Elastic.Documentation.Versions; +using Microsoft.Extensions.Logging; + +namespace Elastic.Changelog.Uploading; + +/// +/// Refreshes the per-product {product}/registry.json manifest in the private bundles +/// bucket after a bundle upload run. Each product touched in the run gets its manifest merged with +/// the bundles already known on S3 (read back, merged by file name, written with an optimistic +/// concurrency guard so parallel uploads for the same product cannot clobber each other). +/// +internal sealed class RegistryBuilder( + ILoggerFactory logFactory, + IFileSystem fileSystem, + IAmazonS3 s3Client, + IS3EtagCalculator etagCalculator, + string bucketName, + TimeProvider? timeProvider = null +) +{ + private readonly ILogger _logger = logFactory.CreateLogger(); + private readonly TimeProvider _time = timeProvider ?? TimeProvider.System; + + // Bounds the optimistic-concurrency retry loop. Concurrent uploads for the same product are + // expected to be rare (releases are largely serialized), so a small ceiling is plenty. + private const int MaxWriteAttempts = 5; + + /// Outcome counts for a manifest refresh run, used for logging only. + internal sealed record RefreshResult(int Updated, int Unchanged, int Failed); + + /// + /// Builds and writes per-product manifests for every product touched by . + /// Each manifest is merged with the copy already on S3 and written back with a conditional PUT + /// (If-Match on update, If-None-Match: * on create); a precondition failure means a + /// concurrent writer won the race, so we re-read, re-merge, and retry. + /// + /// Diagnostics sink for non-fatal warnings. + /// Upload targets produced by DiscoverBundleUploadTargets. + /// Cancellation token. + public async Task RefreshAsync( + IDiagnosticsCollector collector, + IReadOnlyList bundleTargets, + Cancel ctx) + { + // Each upload target carries a "{product}/bundle/{file}" S3 key. Group by product + // so we can produce one manifest per affected product. + var byProduct = bundleTargets + .Select(t => (Target: t, Product: ExtractProduct(t.S3Key))) + .Where(x => x.Product is not null) + .GroupBy(x => x.Product!, StringComparer.Ordinal); + + var updated = 0; + var unchanged = 0; + var failed = 0; + + foreach (var group in byProduct) + { + ctx.ThrowIfCancellationRequested(); + + var product = group.Key; + var localEntries = await BuildLocalEntries(collector, product, group.Select(x => x.Target).ToList(), ctx); + if (localEntries.Count == 0) + { + _logger.LogDebug("No usable manifest entries derived for product {Product}; skipping", product); + continue; + } + + switch (await WriteManifest(collector, product, localEntries, ctx)) + { + case WriteOutcome.Updated: + updated++; + break; + case WriteOutcome.Unchanged: + unchanged++; + break; + default: + failed++; + break; + } + } + + return new RefreshResult(updated, unchanged, failed); + } + + /// Extracts the leading product segment from a {product}/bundle/{file} S3 key, or null. + private static string? ExtractProduct(string s3Key) + { + var firstSlash = s3Key.IndexOf('/'); + if (firstSlash <= 0) + return null; + return s3Key.AsSpan(0, firstSlash).ToString(); + } + + /// Builds manifest entries for this run's bundles, recording each bundle's target for (not the first product). + private async Task> BuildLocalEntries( + IDiagnosticsCollector collector, + string product, + IReadOnlyList targets, + Cancel ctx) + { + var entries = new List(targets.Count); + foreach (var target in targets) + { + ctx.ThrowIfCancellationRequested(); + + var targetVersion = ReadTargetForProduct(collector, target.LocalPath, product); + + string etag; + try + { + etag = await etagCalculator.CalculateS3ETag(target.LocalPath, ctx); + } + catch (Exception ex) + { + collector.EmitWarning(target.LocalPath, + $"Could not compute ETag for manifest entry: {ex.Message}"); + continue; + } + + var fileName = fileSystem.Path.GetFileName(target.LocalPath); + entries.Add(new RegistryBundle + { + File = fileName, + Target = targetVersion, + ETag = etag + }); + } + + return entries; + } + + private string? ReadTargetForProduct(IDiagnosticsCollector collector, string localPath, string product) + { + try + { + var content = fileSystem.File.ReadAllText(localPath); + var bundle = ReleaseNotesSerialization.DeserializeBundle(content); + if (bundle.Products.Count == 0) + return null; + + var match = bundle.Products.FirstOrDefault(p => string.Equals(p.ProductId, product, StringComparison.Ordinal)); + return (match ?? bundle.Products[0]).Target; + } + catch (Exception ex) + { + collector.EmitWarning(localPath, $"Could not read bundle target for manifest: {ex.Message}"); + return null; + } + } + + private enum WriteOutcome { Updated, Unchanged, Failed } + + private async Task WriteManifest( + IDiagnosticsCollector collector, + string product, + IReadOnlyList localEntries, + Cancel ctx) + { + var key = $"{product}/registry.json"; + + for (var attempt = 1; attempt <= MaxWriteAttempts; attempt++) + { + ctx.ThrowIfCancellationRequested(); + + var (existing, etag) = await TryFetchExistingManifest(product, ctx); + var merged = Merge(existing, localEntries); + + // Re-uploading the same bundles must not churn the manifest (keeps reruns idempotent). + if (etag is not null && BundlesEqual(existing, merged)) + { + _logger.LogDebug("registry for {Product} already up to date; skipping write", product); + return WriteOutcome.Unchanged; + } + + var manifest = new Registry + { + Product = product, + GeneratedAt = _time.GetUtcNow(), + Bundles = merged + }; + var json = JsonSerializer.Serialize(manifest, RegistryJsonContext.Default.Registry); + + try + { + await PutManifest(key, json, etag, ctx); + _logger.LogInformation("Wrote registry.json for {Product} with {Count} bundle(s)", product, merged.Count); + return WriteOutcome.Updated; + } + catch (AmazonS3Exception ex) when (ex.StatusCode == HttpStatusCode.PreconditionFailed) + { + _logger.LogInformation( + "registry for {Product} changed concurrently (attempt {Attempt}/{Max}); re-reading and retrying", + product, attempt, MaxWriteAttempts); + } + } + + collector.EmitWarning(string.Empty, + $"registry for {product} could not be updated after {MaxWriteAttempts} attempts due to concurrent writes; the index may be stale."); + return WriteOutcome.Failed; + } + + /// Reads the existing manifest and its ETag (null ETag when absent; live ETag when corrupt, so the conditional write can overwrite). + private async Task<(IReadOnlyList Bundles, string? ETag)> TryFetchExistingManifest(string product, Cancel ctx) + { + var key = $"{product}/registry.json"; + string? etag = null; + try + { + using var response = await s3Client.GetObjectAsync(new GetObjectRequest + { + BucketName = bucketName, + Key = key + }, ctx); + + etag = response.ETag; + await using var stream = response.ResponseStream; + var existing = await JsonSerializer.DeserializeAsync( + stream, + RegistryJsonContext.Default.Registry, + ctx); + + return (existing?.Bundles ?? [], etag); + } + catch (AmazonS3Exception ex) when (ex.StatusCode == HttpStatusCode.NotFound) + { + return ([], null); + } + catch (JsonException ex) + { + // Only a genuinely corrupt (unparseable) manifest is rebuilt from this run; the captured ETag + // then lets the conditional write overwrite it safely. Transient S3/IO errors must NOT be + // treated as corruption — otherwise the If-Match PUT would replace a valid manifest with only + // this run's bundles and drop previously published entries. Let those bubble up to the + // best-effort handler in ChangelogUploadService instead. + _logger.LogWarning(ex, "Existing manifest for {Product} could not be parsed; recreating", product); + return ([], etag); + } + } + + private async Task PutManifest(string key, string json, string? etag, Cancel ctx) + { + var request = new PutObjectRequest + { + BucketName = bucketName, + Key = key, + ContentBody = json, + ContentType = "application/json" + }; + + // Optimistic concurrency: update only if unchanged, create only if still absent. + if (etag is null) + request.IfNoneMatch = "*"; + else + request.IfMatch = etag; + + _ = await s3Client.PutObjectAsync(request, ctx); + } + + /// Replaces existing entries by file name and sorts newest-target-first (with a file-name tiebreak) for a stable manifest. + private static List Merge( + IReadOnlyList existing, + IReadOnlyList incoming) + { + var byFile = existing.ToDictionary(b => b.File, b => b, StringComparer.Ordinal); + foreach (var entry in incoming) + byFile[entry.File] = entry; + + return byFile.Values + .OrderByDescending(b => VersionOrDate.Parse(b.Target ?? string.Empty)) + .ThenBy(b => b.File, StringComparer.Ordinal) + .ToList(); + } + + private static bool BundlesEqual(IReadOnlyList a, IReadOnlyList b) + { + if (a.Count != b.Count) + return false; + + for (var i = 0; i < a.Count; i++) + { + if (!string.Equals(a[i].File, b[i].File, StringComparison.Ordinal) || + !string.Equals(a[i].Target, b[i].Target, StringComparison.Ordinal) || + !string.Equals(a[i].ETag, b[i].ETag, StringComparison.Ordinal)) + return false; + } + + return true; + } +} diff --git a/src/services/Elastic.Changelog/Uploading/RegistryKey.cs b/src/services/Elastic.Changelog/Uploading/RegistryKey.cs new file mode 100644 index 0000000000..aa2d235294 --- /dev/null +++ b/src/services/Elastic.Changelog/Uploading/RegistryKey.cs @@ -0,0 +1,46 @@ +// Licensed to Elasticsearch B.V under one or more agreements. +// Elasticsearch B.V licenses this file to you under the Apache 2.0 License. +// See the LICENSE file in the project root for more information + +namespace Elastic.Changelog.Uploading; + +/// +/// Helpers for validating S3 object keys related to the per-product +/// manifest. +/// +public static class RegistryKey +{ + private const string Suffix = "/registry.json"; + + /// + /// Returns true when is a top-level per-product manifest of the + /// form {product}/registry.json, where {product} matches the same + /// character class enforced by ChangelogUploadService.ProductNameRegex + /// ([a-zA-Z0-9_-]+). + /// + /// + /// Used by the changelog scrubber Lambda to decide whether to pass an incoming + /// *.json object through to the public bucket. Anything else (e.g. nested + /// under a bundle/ prefix, or a multi-segment product) is rejected, which keeps + /// arbitrary JSON out of the public surface. + /// + public static bool IsRegistry(string key) + { + if (string.IsNullOrEmpty(key)) + return false; + + if (!key.EndsWith(Suffix, StringComparison.Ordinal)) + return false; + + var product = key.AsSpan(0, key.Length - Suffix.Length); + if (product.IsEmpty || product.Contains('/')) + return false; + + foreach (var c in product) + { + if (!(char.IsAsciiLetterOrDigit(c) || c == '_' || c == '-')) + return false; + } + return true; + } +} diff --git a/src/services/Elastic.Documentation.Assembler/AssembleSources.cs b/src/services/Elastic.Documentation.Assembler/AssembleSources.cs index e9693cb2e7..0b3d4cdd63 100644 --- a/src/services/Elastic.Documentation.Assembler/AssembleSources.cs +++ b/src/services/Elastic.Documentation.Assembler/AssembleSources.cs @@ -11,6 +11,7 @@ using Elastic.Documentation.Configuration.Assembler; using Elastic.Documentation.Configuration.Builder; using Elastic.Documentation.Configuration.LegacyUrlMappings; +using Elastic.Documentation.Configuration.ReleaseNotes; using Elastic.Documentation.Configuration.Toc; using Elastic.Documentation.LinkIndex; using Elastic.Documentation.Links.CrossLinks; @@ -52,6 +53,10 @@ Cancel ctx var crossLinkResolver = new CrossLinkResolver(crossLinks, uriResolver); logger.LogInformation(" AssembleAsync: FetchCrossLinks in {Elapsed:mm\\:ss\\.fff}", sw.Elapsed); + // Shared, mutable resolver: every documentation set gets the same instance now and the prefetched + // bundles are populated once below, after each set's docset.yml (and its `release_notes`) is parsed. + var releaseNotesResolver = new ReleaseNotesResolver(); + var sources = new AssembleSources( logFactory, context, @@ -61,9 +66,22 @@ Cancel ctx configurationContext.LegacyUrlMappings, uriResolver, crossLinkResolver, + releaseNotesResolver, availableExporters ); + var declaredProducts = sources.AssembleSets.Values + .SelectMany(s => s.BuildContext.Configuration.ReleaseNotesProducts) + .Distinct(StringComparer.Ordinal) + .ToArray(); + if (declaredProducts.Length > 0) + { + var releaseNotesFetcher = new ReleaseNotesFetcher(logFactory, context.ReadFileSystem); + var fetched = await releaseNotesFetcher.FetchAsync(context.Collector, declaredProducts, ctx).ConfigureAwait(false); + releaseNotesResolver.Populate(fetched); + logger.LogInformation(" AssembleAsync: Fetched release notes for {Count} product(s)", declaredProducts.Length); + } + foreach (var (_, set) in sources.AssembleSets) { logger.LogInformation("Resolving directory tree for {RepositoryName}", set.Checkout.Repository.Name); @@ -82,6 +100,7 @@ private AssembleSources( LegacyUrlMappingConfiguration legacyUrlMappings, PublishEnvironmentUriResolver uriResolver, ICrossLinkResolver crossLinkResolver, + IReleaseNotesResolver releaseNotesResolver, IReadOnlySet availableExporters ) { @@ -91,7 +110,7 @@ IReadOnlySet availableExporters AssembleContext = assembleContext; AssembleSets = checkouts .Where(c => c.Repository is { Skip: false }) - .Select(c => new AssemblerDocumentationSet(logFactory, assembleContext, c, crossLinkResolver, configurationContext, availableExporters)) + .Select(c => new AssemblerDocumentationSet(logFactory, assembleContext, c, crossLinkResolver, releaseNotesResolver, configurationContext, availableExporters)) .ToDictionary(s => s.Checkout.Repository.Name, s => s) .ToFrozenDictionary(); } diff --git a/src/services/Elastic.Documentation.Assembler/Navigation/AssemblerDocumentationSet.cs b/src/services/Elastic.Documentation.Assembler/Navigation/AssemblerDocumentationSet.cs index a837d7ae3d..4ddc90eeba 100644 --- a/src/services/Elastic.Documentation.Assembler/Navigation/AssemblerDocumentationSet.cs +++ b/src/services/Elastic.Documentation.Assembler/Navigation/AssemblerDocumentationSet.cs @@ -6,6 +6,7 @@ using Elastic.Documentation.Assembler.Sourcing; using Elastic.Documentation.Configuration; using Elastic.Documentation.Configuration.Assembler; +using Elastic.Documentation.Configuration.ReleaseNotes; using Elastic.Documentation.Links.CrossLinks; using Elastic.Markdown.IO; using Microsoft.Extensions.Logging; @@ -27,6 +28,7 @@ public AssemblerDocumentationSet( AssembleContext context, Checkout checkout, ICrossLinkResolver crossLinkResolver, + IReleaseNotesResolver releaseNotesResolver, IConfigurationContext configurationContext, IReadOnlySet availableExporters ) @@ -81,6 +83,6 @@ IReadOnlySet availableExporters }; BuildContext = buildContext; - DocumentationSet = new DocumentationSet(buildContext, logFactory, crossLinkResolver); + DocumentationSet = new DocumentationSet(buildContext, logFactory, crossLinkResolver, releaseNotesResolver); } } diff --git a/src/services/Elastic.Documentation.Isolated/IsolatedBuildService.cs b/src/services/Elastic.Documentation.Isolated/IsolatedBuildService.cs index 204292009f..afc306a2c4 100644 --- a/src/services/Elastic.Documentation.Isolated/IsolatedBuildService.cs +++ b/src/services/Elastic.Documentation.Isolated/IsolatedBuildService.cs @@ -9,6 +9,7 @@ using Elastic.Documentation.Configuration; using Elastic.Documentation.Configuration.Builder; using Elastic.Documentation.Configuration.Inference; +using Elastic.Documentation.Configuration.ReleaseNotes; using Elastic.Documentation.Diagnostics; using Elastic.Documentation.LinkIndex; using Elastic.Documentation.Links; @@ -141,8 +142,11 @@ public async Task Build( crossLinkResolver = new CrossLinkResolver(crossLinks, uriResolver); } + // Prefetch CDN-hosted release notes for products declared under `release_notes` in docset.yml. + var releaseNotesResolver = await ReleaseNotesFetcher.PrefetchAsync(context, logFactory, ctx); + // always delete output folder on CI - var set = new DocumentationSet(context, logFactory, crossLinkResolver); + var set = new DocumentationSet(context, logFactory, crossLinkResolver, releaseNotesResolver); if (runningOnCi) set.ClearOutputDirectory(); diff --git a/src/tooling/docs-builder/Http/ReloadableGeneratorState.cs b/src/tooling/docs-builder/Http/ReloadableGeneratorState.cs index 415411a110..344c1a062c 100644 --- a/src/tooling/docs-builder/Http/ReloadableGeneratorState.cs +++ b/src/tooling/docs-builder/Http/ReloadableGeneratorState.cs @@ -6,6 +6,7 @@ using Elastic.Documentation; using Elastic.Documentation.Configuration; using Elastic.Documentation.Configuration.Builder; +using Elastic.Documentation.Configuration.ReleaseNotes; using Elastic.Documentation.LinkIndex; using Elastic.Documentation.Links.CrossLinks; using Elastic.Markdown; @@ -94,7 +95,8 @@ public async Task ReloadAsync(Cancel ctx, bool reloadConfiguration = true) ? new CodexAwareUriResolver(crossLinks.CodexRepositories) : null; var crossLinkResolver = new CrossLinkResolver(crossLinks, uriResolver); - var docSet = new DocumentationSet(_context, _logFactory, crossLinkResolver); + var releaseNotesResolver = await ReleaseNotesFetcher.PrefetchAsync(_context, _logFactory, ctx); + var docSet = new DocumentationSet(_context, _logFactory, crossLinkResolver, releaseNotesResolver); // Add LLM markdown export for dev server var markdownExporters = new List(); diff --git a/tests/Elastic.Changelog.Tests/Changelogs/BundleLoading/BundleLoaderTests.cs b/tests/Elastic.Changelog.Tests/Changelogs/BundleLoading/BundleLoaderTests.cs index d272a31520..1b144b349a 100644 --- a/tests/Elastic.Changelog.Tests/Changelogs/BundleLoading/BundleLoaderTests.cs +++ b/tests/Elastic.Changelog.Tests/Changelogs/BundleLoading/BundleLoaderTests.cs @@ -403,7 +403,7 @@ public void MergeBundlesByTarget_WithSingleBundle_ReturnsSameBundle() }; // Act - var merged = service.MergeBundlesByTarget(bundles); + var merged = BundleLoader.MergeBundlesByTarget(bundles); // Assert merged.Should().HaveCount(1); @@ -424,7 +424,7 @@ public void MergeBundlesByTarget_WithDifferentVersions_KeepsSeparate() }; // Act - var merged = service.MergeBundlesByTarget(bundles); + var merged = BundleLoader.MergeBundlesByTarget(bundles); // Assert merged.Should().HaveCount(2); @@ -444,7 +444,7 @@ public void MergeBundlesByTarget_WithSameVersion_MergesEntries() }; // Act - var merged = service.MergeBundlesByTarget(bundles); + var merged = BundleLoader.MergeBundlesByTarget(bundles); // Assert merged.Should().HaveCount(1); @@ -467,7 +467,7 @@ public void MergeBundlesByTarget_PreservesSortOrder() }; // Act - var merged = service.MergeBundlesByTarget(bundles); + var merged = BundleLoader.MergeBundlesByTarget(bundles); // Assert merged.Should().HaveCount(3); @@ -489,7 +489,7 @@ public void MergeBundlesByTarget_WithDateVersions_SortsCorrectly() }; // Act - var merged = service.MergeBundlesByTarget(bundles); + var merged = BundleLoader.MergeBundlesByTarget(bundles); // Assert merged.Should().HaveCount(3); @@ -1366,7 +1366,7 @@ public void MergeBundlesByTarget_ReleaseDatePreserved() var loaded = service.LoadBundles(bundlesFolder, EmitWarning); // Act - var merged = service.MergeBundlesByTarget(loaded); + var merged = BundleLoader.MergeBundlesByTarget(loaded); // Assert merged.Should().HaveCount(1); diff --git a/tests/Elastic.Changelog.Tests/Uploading/ChangelogUploadServiceTests.cs b/tests/Elastic.Changelog.Tests/Uploading/ChangelogUploadServiceTests.cs index 6a6fe8f7c5..8887ef24e3 100644 --- a/tests/Elastic.Changelog.Tests/Uploading/ChangelogUploadServiceTests.cs +++ b/tests/Elastic.Changelog.Tests/Uploading/ChangelogUploadServiceTests.cs @@ -450,4 +450,92 @@ public void DiscoverBundleUploadTargets_EmptyDirectory_ReturnsEmpty() targets.Should().BeEmpty(); } + + [Fact] + public async Task Upload_BundleArtifactType_UploadsRegistryAlongsideBundle() + { + var bundleDir = _mockFileSystem.Path.Join(Paths.WorkingDirectoryRoot.FullName, Guid.NewGuid().ToString(), "releases"); + _mockFileSystem.Directory.CreateDirectory(bundleDir); + // language=yaml + _mockFileSystem.AddFile(_mockFileSystem.Path.Join(bundleDir, "9.3.0.yaml"), new MockFileData(""" + products: + - product: elasticsearch + target: 9.3.0 + repo: elasticsearch + owner: elastic + entries: + - file: + name: 1.yaml + checksum: c0ffee + type: enhancement + title: Sample + """)); + + A.CallTo(() => _s3Client.GetObjectMetadataAsync(A._, A._)) + .Throws(new AmazonS3Exception("Not Found") { StatusCode = HttpStatusCode.NotFound }); + A.CallTo(() => _s3Client.GetObjectAsync(A._, A._)) + .Throws(new AmazonS3Exception("Not Found") { StatusCode = HttpStatusCode.NotFound }); + A.CallTo(() => _s3Client.PutObjectAsync(A._, A._)) + .Returns(new PutObjectResponse()); + + var args = new ChangelogUploadArguments + { + ArtifactType = ArtifactType.Bundle, + Target = UploadTargetKind.S3, + S3BucketName = "test-bucket", + Directory = bundleDir + }; + var ct = TestContext.Current.CancellationToken; + var result = await _service.Upload(_collector, args, ct); + + result.Should().BeTrue(); + _collector.Errors.Should().Be(0); + + A.CallTo(() => _s3Client.PutObjectAsync( + A.That.Matches(r => r.Key == "elasticsearch/bundle/9.3.0.yaml"), + A._ + )).MustHaveHappenedOnceExactly(); + + A.CallTo(() => _s3Client.PutObjectAsync( + A.That.Matches(r => r.Key == "elasticsearch/registry.json"), + A._ + )).MustHaveHappenedOnceExactly(); + } + + [Fact] + public async Task Upload_ChangelogArtifactType_DoesNotUploadRegistry() + { + // language=yaml + AddChangelog("entry.yaml", """ + title: Plain entry + type: feature + products: + - product: elasticsearch + target: 9.2.0 + prs: + - "100" + """); + + A.CallTo(() => _s3Client.GetObjectMetadataAsync(A._, A._)) + .Throws(new AmazonS3Exception("Not Found") { StatusCode = HttpStatusCode.NotFound }); + A.CallTo(() => _s3Client.PutObjectAsync(A._, A._)) + .Returns(new PutObjectResponse()); + + var args = new ChangelogUploadArguments + { + ArtifactType = ArtifactType.Changelog, + Target = UploadTargetKind.S3, + S3BucketName = "test-bucket", + Directory = _changelogDir + }; + var ct = TestContext.Current.CancellationToken; + var result = await _service.Upload(_collector, args, ct); + + result.Should().BeTrue(); + + A.CallTo(() => _s3Client.PutObjectAsync( + A.That.Matches(r => r.Key.EndsWith("/registry.json", StringComparison.Ordinal)), + A._ + )).MustNotHaveHappened(); + } } diff --git a/tests/Elastic.Changelog.Tests/Uploading/RegistryBuilderTests.cs b/tests/Elastic.Changelog.Tests/Uploading/RegistryBuilderTests.cs new file mode 100644 index 0000000000..8ce29c5b39 --- /dev/null +++ b/tests/Elastic.Changelog.Tests/Uploading/RegistryBuilderTests.cs @@ -0,0 +1,403 @@ +// Licensed to Elasticsearch B.V under one or more agreements. +// Elasticsearch B.V licenses this file to you under the Apache 2.0 License. +// See the LICENSE file in the project root for more information + +using System.Diagnostics.CodeAnalysis; +using System.IO.Abstractions.TestingHelpers; +using System.Net; +using System.Text; +using System.Text.Json; +using Amazon.S3; +using Amazon.S3.Model; +using AwesomeAssertions; +using Elastic.Changelog.Tests.Changelogs; +using Elastic.Changelog.Uploading; +using Elastic.Documentation.Configuration; +using Elastic.Documentation.Integrations.S3; +using FakeItEasy; +using Microsoft.Extensions.Logging.Abstractions; +using Nullean.ScopedFileSystem; + +namespace Elastic.Changelog.Tests.Uploading; + +[SuppressMessage("Usage", "CA1001:Types that own disposable fields should be disposable")] +public class RegistryBuilderTests +{ + private readonly MockFileSystem _mockFileSystem; + private readonly ScopedFileSystem _fileSystem; + private readonly IAmazonS3 _s3Client = A.Fake(); + private readonly TestDiagnosticsCollector _collector; + private readonly string _bundleDir; + private readonly RegistryBuilder _builder; + private readonly List _puts = []; + + public RegistryBuilderTests(ITestOutputHelper output) + { + _mockFileSystem = new MockFileSystem(new MockFileSystemOptions + { + CurrentDirectory = Paths.WorkingDirectoryRoot.FullName + }); + _fileSystem = FileSystemFactory.ScopeCurrentWorkingDirectory(_mockFileSystem); + _collector = new TestDiagnosticsCollector(output); + _bundleDir = _mockFileSystem.Path.Join(Paths.WorkingDirectoryRoot.FullName, Guid.NewGuid().ToString(), "releases"); + _mockFileSystem.Directory.CreateDirectory(_bundleDir); + var etagCalculator = new S3EtagCalculator(NullLoggerFactory.Instance, _fileSystem); + // Pin time so generated_at is deterministic in tests. + var fixedTime = new FakeTimeProvider(new DateTimeOffset(2026, 5, 6, 12, 0, 0, TimeSpan.Zero)); + _builder = new RegistryBuilder( + NullLoggerFactory.Instance, + _fileSystem, + _s3Client, + etagCalculator, + "test-bucket", + fixedTime); + + // Capture every manifest PUT so tests can inspect the body and conditional headers. + A.CallTo(() => _s3Client.PutObjectAsync(A._, A._)) + .Invokes((PutObjectRequest r, CancellationToken _) => _puts.Add(r)) + .Returns(new PutObjectResponse()); + } + + private string AddBundle(string fileName, string product, string target) + { + var path = _mockFileSystem.Path.Join(_bundleDir, fileName); + // language=yaml + _mockFileSystem.AddFile(path, new MockFileData($$""" + products: + - product: {{product}} + target: {{target}} + repo: {{product}} + owner: elastic + entries: + - file: + name: 1-feature.yaml + checksum: deadbeef + type: enhancement + title: Sample + """)); + return path; + } + + private void StubExistingManifestNotFound() => + A.CallTo(() => _s3Client.GetObjectAsync(A._, A._)) + .Throws(new AmazonS3Exception("Not Found") { StatusCode = HttpStatusCode.NotFound }); + + private void StubExistingManifest(string product, Registry manifest, string etag = "\"existing-etag\"") => + A.CallTo(() => _s3Client.GetObjectAsync( + A.That.Matches(r => r.Key == $"{product}/registry.json"), + A._)) + .ReturnsLazily(() => MakeManifestResponse(manifest, etag)); + + private static GetObjectResponse MakeManifestResponse(Registry manifest, string etag) + { + var json = JsonSerializer.Serialize(manifest, RegistryJsonContext.Default.Registry); + return new GetObjectResponse + { + ETag = etag, + ResponseStream = new MemoryStream(Encoding.UTF8.GetBytes(json)) + }; + } + + private static Registry Deserialize(string? json) => + JsonSerializer.Deserialize(json!, RegistryJsonContext.Default.Registry)!; + + [Fact] + public async Task Refresh_NoExistingManifest_CreatesManifestWithIfNoneMatch() + { + var path = AddBundle("9.3.0.yaml", "elasticsearch", "9.3.0"); + var targets = new List { new(path, "elasticsearch/bundle/9.3.0.yaml") }; + StubExistingManifestNotFound(); + + var result = await _builder.RefreshAsync(_collector, targets, TestContext.Current.CancellationToken); + + result.Updated.Should().Be(1); + _puts.Should().ContainSingle(); + var put = _puts[0]; + put.Key.Should().Be("elasticsearch/registry.json"); + put.IfNoneMatch.Should().Be("*"); + put.IfMatch.Should().BeNull(); + + var manifest = Deserialize(put.ContentBody); + manifest.Product.Should().Be("elasticsearch"); + manifest.Bundles.Should().ContainSingle(); + manifest.Bundles[0].File.Should().Be("9.3.0.yaml"); + manifest.Bundles[0].Target.Should().Be("9.3.0"); + manifest.Bundles[0].ETag.Should().NotBeNullOrEmpty(); + } + + [Fact] + public async Task Refresh_ExistingManifest_MergesByFileNameAndUsesIfMatch() + { + var existing = new Registry + { + Product = "elasticsearch", + GeneratedAt = new DateTimeOffset(2026, 1, 1, 0, 0, 0, TimeSpan.Zero), + Bundles = + [ + new RegistryBundle { File = "9.2.0.yaml", Target = "9.2.0", ETag = "old-etag-1" }, + new RegistryBundle { File = "9.3.0.yaml", Target = "9.3.0", ETag = "old-etag-2" } + ] + }; + StubExistingManifest("elasticsearch", existing, "\"manifest-v1\""); + + // Re-upload of 9.3.0 (new content → new ETag) plus a brand-new 9.4.0. + var newer = AddBundle("9.3.0.yaml", "elasticsearch", "9.3.0"); + var ten = AddBundle("9.4.0.yaml", "elasticsearch", "9.4.0"); + var targets = new List + { + new(newer, "elasticsearch/bundle/9.3.0.yaml"), + new(ten, "elasticsearch/bundle/9.4.0.yaml") + }; + + var result = await _builder.RefreshAsync(_collector, targets, TestContext.Current.CancellationToken); + + result.Updated.Should().Be(1); + _puts.Should().ContainSingle(); + _puts[0].IfMatch.Should().Be("\"manifest-v1\""); + _puts[0].IfNoneMatch.Should().BeNull(); + + var manifest = Deserialize(_puts[0].ContentBody); + manifest.Bundles.Should().HaveCount(3); + manifest.Bundles.Should().Contain(b => b.File == "9.2.0.yaml" && b.ETag == "old-etag-1"); + + var nineThree = manifest.Bundles.Single(b => b.File == "9.3.0.yaml"); + nineThree.ETag.Should().NotBe("old-etag-2"); // replaced by the freshly-uploaded ETag + manifest.Bundles.Should().Contain(b => b.File == "9.4.0.yaml"); + + manifest.Bundles[0].File.Should().Be("9.4.0.yaml"); // sorted target-desc + } + + [Fact] + public async Task Refresh_SortsManifestByVersionNotLexicographically() + { + StubExistingManifestNotFound(); + + // Version order (not byte order): 9.10.0 must come before 9.9.0 in the written manifest. + var v910 = AddBundle("9.10.0.yaml", "elasticsearch", "9.10.0"); + var v99 = AddBundle("9.9.0.yaml", "elasticsearch", "9.9.0"); + var targets = new List + { + new(v99, "elasticsearch/bundle/9.9.0.yaml"), + new(v910, "elasticsearch/bundle/9.10.0.yaml") + }; + + _ = await _builder.RefreshAsync(_collector, targets, TestContext.Current.CancellationToken); + + var manifest = Deserialize(_puts[0].ContentBody); + manifest.Bundles.Select(b => b.Target).Should().Equal("9.10.0", "9.9.0"); + } + + [Fact] + public async Task Refresh_MultipleProducts_WritesOneManifestPerProduct() + { + var es = AddBundle("9.3.0.yaml", "elasticsearch", "9.3.0"); + var kb = AddBundle("kb-9.3.0.yaml", "kibana", "9.3.0"); + var targets = new List + { + new(es, "elasticsearch/bundle/9.3.0.yaml"), + new(kb, "kibana/bundle/kb-9.3.0.yaml") + }; + StubExistingManifestNotFound(); + + var result = await _builder.RefreshAsync(_collector, targets, TestContext.Current.CancellationToken); + + result.Updated.Should().Be(2); + _puts.Should().HaveCount(2); + _puts.Should().Contain(p => p.Key == "elasticsearch/registry.json"); + _puts.Should().Contain(p => p.Key == "kibana/registry.json"); + } + + [Fact] + public async Task Refresh_MultiProductBundle_RecordsTargetPerProduct() + { + // One bundle file declaring two products with *different* targets. + var path = _mockFileSystem.Path.Join(_bundleDir, "multi.yaml"); + // language=yaml + _mockFileSystem.AddFile(path, new MockFileData(""" + products: + - product: elasticsearch + target: 9.3.0 + repo: elasticsearch + owner: elastic + - product: kibana + target: 9.4.0 + repo: kibana + owner: elastic + entries: + - file: + name: 1-feature.yaml + checksum: deadbeef + type: enhancement + title: Sample + """)); + var targets = new List + { + new(path, "elasticsearch/bundle/multi.yaml"), + new(path, "kibana/bundle/multi.yaml") + }; + StubExistingManifestNotFound(); + + _ = await _builder.RefreshAsync(_collector, targets, TestContext.Current.CancellationToken); + + var es = Deserialize(_puts.Single(p => p.Key == "elasticsearch/registry.json").ContentBody); + es.Bundles[0].Target.Should().Be("9.3.0"); + + var kb = Deserialize(_puts.Single(p => p.Key == "kibana/registry.json").ContentBody); + kb.Bundles[0].Target.Should().Be("9.4.0"); + } + + [Fact] + public async Task Refresh_ExistingManifestUnreadable_OverwritesUsingLiveETag() + { + A.CallTo(() => _s3Client.GetObjectAsync(A._, A._)) + .ReturnsLazily(() => new GetObjectResponse + { + ETag = "\"corrupt-etag\"", + ResponseStream = new MemoryStream(Encoding.UTF8.GetBytes("not json {{{")) + }); + + var path = AddBundle("9.3.0.yaml", "elasticsearch", "9.3.0"); + var targets = new List { new(path, "elasticsearch/bundle/9.3.0.yaml") }; + + var result = await _builder.RefreshAsync(_collector, targets, TestContext.Current.CancellationToken); + + result.Updated.Should().Be(1); + _puts.Should().ContainSingle(); + _puts[0].IfMatch.Should().Be("\"corrupt-etag\""); // conditional overwrite of the corrupt object + var manifest = Deserialize(_puts[0].ContentBody); + manifest.Bundles.Should().ContainSingle(); + manifest.Bundles[0].File.Should().Be("9.3.0.yaml"); + } + + [Fact] + public async Task Refresh_BundleWithoutTarget_RecordsEntryWithoutTarget() + { + var path = _mockFileSystem.Path.Join(_bundleDir, "no-target.yaml"); + // language=yaml + _mockFileSystem.AddFile(path, new MockFileData(""" + entries: [] + """)); + var targets = new List { new(path, "elasticsearch/bundle/no-target.yaml") }; + StubExistingManifestNotFound(); + + var result = await _builder.RefreshAsync(_collector, targets, TestContext.Current.CancellationToken); + + result.Updated.Should().Be(1); + var manifest = Deserialize(_puts[0].ContentBody); + manifest.Bundles.Should().ContainSingle(); + manifest.Bundles[0].Target.Should().BeNull(); + manifest.Bundles[0].ETag.Should().NotBeNullOrEmpty(); + } + + [Fact] + public async Task Refresh_UnchangedManifest_SkipsWrite() + { + var path = AddBundle("9.3.0.yaml", "elasticsearch", "9.3.0"); + var etagCalculator = new S3EtagCalculator(NullLoggerFactory.Instance, _fileSystem); + var bundleEtag = await etagCalculator.CalculateS3ETag(path, TestContext.Current.CancellationToken); + + // Existing manifest already contains exactly what this run would produce. + var existing = new Registry + { + Product = "elasticsearch", + GeneratedAt = new DateTimeOffset(2026, 1, 1, 0, 0, 0, TimeSpan.Zero), + Bundles = [new RegistryBundle { File = "9.3.0.yaml", Target = "9.3.0", ETag = bundleEtag }] + }; + StubExistingManifest("elasticsearch", existing, "\"manifest-v1\""); + + var targets = new List { new(path, "elasticsearch/bundle/9.3.0.yaml") }; + var result = await _builder.RefreshAsync(_collector, targets, TestContext.Current.CancellationToken); + + result.Unchanged.Should().Be(1); + result.Updated.Should().Be(0); + _puts.Should().BeEmpty(); + } + + [Fact] + public async Task Refresh_ConcurrentWrite_RetriesAfterPreconditionFailed() + { + var v1 = new Registry + { + Product = "elasticsearch", + GeneratedAt = new DateTimeOffset(2026, 1, 1, 0, 0, 0, TimeSpan.Zero), + Bundles = [new RegistryBundle { File = "9.2.0.yaml", Target = "9.2.0", ETag = "etag-92" }] + }; + // Second read reflects a concurrent writer that added 9.3.0 and bumped the object ETag. + var v2 = new Registry + { + Product = "elasticsearch", + GeneratedAt = new DateTimeOffset(2026, 1, 2, 0, 0, 0, TimeSpan.Zero), + Bundles = + [ + new RegistryBundle { File = "9.3.0.yaml", Target = "9.3.0", ETag = "etag-93" }, + new RegistryBundle { File = "9.2.0.yaml", Target = "9.2.0", ETag = "etag-92" } + ] + }; + A.CallTo(() => _s3Client.GetObjectAsync(A._, A._)) + .ReturnsNextFromSequence( + MakeManifestResponse(v1, "\"manifest-v1\""), + MakeManifestResponse(v2, "\"manifest-v2\"")); + + // First PUT loses the optimistic-concurrency race; the second succeeds. + A.CallTo(() => _s3Client.PutObjectAsync(A._, A._)) + .Throws(new AmazonS3Exception("Precondition Failed") { StatusCode = HttpStatusCode.PreconditionFailed }) + .Once(); + + var path = AddBundle("9.4.0.yaml", "elasticsearch", "9.4.0"); + var targets = new List { new(path, "elasticsearch/bundle/9.4.0.yaml") }; + + var result = await _builder.RefreshAsync(_collector, targets, TestContext.Current.CancellationToken); + + result.Updated.Should().Be(1); + A.CallTo(() => _s3Client.GetObjectAsync(A._, A._)) + .MustHaveHappenedTwiceExactly(); + A.CallTo(() => _s3Client.PutObjectAsync(A._, A._)) + .MustHaveHappenedTwiceExactly(); + + // Two PUTs were attempted (asserted above); the first threw on the precondition failure before + // _puts.Add ran, so only the successful retry is captured here. It used the re-read ETag and + // merged both the concurrent and local entries. + _puts.Should().ContainSingle(); + _puts[0].IfMatch.Should().Be("\"manifest-v2\""); + var manifest = Deserialize(_puts[0].ContentBody); + manifest.Bundles.Select(b => b.File).Should().BeEquivalentTo(["9.4.0.yaml", "9.3.0.yaml", "9.2.0.yaml"]); + } + + [Fact] + public async Task Refresh_PersistentConcurrentWrite_EmitsWarningAndReportsFailure() + { + var existing = new Registry + { + Product = "elasticsearch", + GeneratedAt = new DateTimeOffset(2026, 1, 1, 0, 0, 0, TimeSpan.Zero), + Bundles = [new RegistryBundle { File = "9.2.0.yaml", Target = "9.2.0", ETag = "etag-92" }] + }; + StubExistingManifest("elasticsearch", existing, "\"manifest-v1\""); + + // Every PUT loses the race. + A.CallTo(() => _s3Client.PutObjectAsync(A._, A._)) + .Throws(new AmazonS3Exception("Precondition Failed") { StatusCode = HttpStatusCode.PreconditionFailed }); + + var path = AddBundle("9.4.0.yaml", "elasticsearch", "9.4.0"); + var targets = new List { new(path, "elasticsearch/bundle/9.4.0.yaml") }; + + var result = await _builder.RefreshAsync(_collector, targets, TestContext.Current.CancellationToken); + + result.Failed.Should().Be(1); + result.Updated.Should().Be(0); + _collector.Warnings.Should().BeGreaterThan(0); + } + + [Fact] + public async Task Refresh_NoTargets_WritesNothing() + { + var result = await _builder.RefreshAsync(_collector, [], TestContext.Current.CancellationToken); + result.Should().Be(new RegistryBuilder.RefreshResult(0, 0, 0)); + _puts.Should().BeEmpty(); + } + + private sealed class FakeTimeProvider(DateTimeOffset now) : TimeProvider + { + public override DateTimeOffset GetUtcNow() => now; + } +} diff --git a/tests/Elastic.Changelog.Tests/Uploading/RegistryKeyTests.cs b/tests/Elastic.Changelog.Tests/Uploading/RegistryKeyTests.cs new file mode 100644 index 0000000000..221765a619 --- /dev/null +++ b/tests/Elastic.Changelog.Tests/Uploading/RegistryKeyTests.cs @@ -0,0 +1,34 @@ +// Licensed to Elasticsearch B.V under one or more agreements. +// Elasticsearch B.V licenses this file to you under the Apache 2.0 License. +// See the LICENSE file in the project root for more information + +using AwesomeAssertions; +using Elastic.Changelog.Uploading; + +namespace Elastic.Changelog.Tests.Uploading; + +public class RegistryKeyTests +{ + [Theory] + [InlineData("elasticsearch/registry.json")] + [InlineData("kibana/registry.json")] + [InlineData("elastic-agent/registry.json")] + [InlineData("cloud_hosted/registry.json")] + [InlineData("a/registry.json")] + public void IsRegistry_ValidProductKeys_ReturnsTrue(string key) => + RegistryKey.IsRegistry(key).Should().BeTrue(); + + [Theory] + [InlineData("")] + [InlineData("registry.json")] + [InlineData("/registry.json")] + [InlineData("elasticsearch/bundle/registry.json")] + [InlineData("elasticsearch/registry.yaml")] + [InlineData("elasticsearch/changelog/registry.json")] + [InlineData("../registry.json")] + [InlineData("elastic search/registry.json")] + [InlineData("elastic.search/registry.json")] + [InlineData("elastic/search/registry.json")] + public void IsRegistry_InvalidKeys_ReturnsFalse(string key) => + RegistryKey.IsRegistry(key).Should().BeFalse(); +} diff --git a/tests/Elastic.Documentation.Configuration.Tests/ConfigurationFileReleaseNotesTests.cs b/tests/Elastic.Documentation.Configuration.Tests/ConfigurationFileReleaseNotesTests.cs new file mode 100644 index 0000000000..29516de06f --- /dev/null +++ b/tests/Elastic.Documentation.Configuration.Tests/ConfigurationFileReleaseNotesTests.cs @@ -0,0 +1,167 @@ +// Licensed to Elasticsearch B.V under one or more agreements. +// Elasticsearch B.V licenses this file to you under the Apache 2.0 License. +// See the LICENSE file in the project root for more information + +using System.Collections.Frozen; +using System.IO.Abstractions; +using System.IO.Abstractions.TestingHelpers; +using AwesomeAssertions; +using Elastic.Documentation.Configuration.Builder; +using Elastic.Documentation.Configuration.Products; +using Elastic.Documentation.Configuration.Toc; +using Elastic.Documentation.Configuration.Versions; +using Elastic.Documentation.Diagnostics; +using Nullean.ScopedFileSystem; + +namespace Elastic.Documentation.Configuration.Tests; + +public class ConfigurationFileReleaseNotesTests +{ + [Fact] + public async Task ReleaseNotes_DeclaredProduct_IsExposedWithoutErrors() + { + var (config, diagnostics) = await CreateConfiguration(DocSetWithReleaseNotes("elasticsearch")); + + config.ReleaseNotesProducts.Should().ContainSingle().Which.Should().Be("elasticsearch"); + diagnostics.Should().BeEmpty(); + } + + [Fact] + public async Task ReleaseNotes_UnderscoreVariant_NormalizesToCanonicalId() + { + var (config, diagnostics) = await CreateConfiguration(DocSetWithReleaseNotes("edot_java")); + + config.ReleaseNotesProducts.Should().ContainSingle().Which.Should().Be("edot-java"); + diagnostics.Should().BeEmpty(); + } + + [Fact] + public async Task ReleaseNotes_DuplicateDeclarations_AreDeduplicated() + { + var (config, diagnostics) = await CreateConfiguration(DocSetWithReleaseNotes("elasticsearch", "elasticsearch")); + + config.ReleaseNotesProducts.Should().ContainSingle().Which.Should().Be("elasticsearch"); + diagnostics.Should().BeEmpty(); + } + + [Fact] + public async Task ReleaseNotes_UnknownProduct_EmitsError() + { + var (config, diagnostics) = await CreateConfiguration(DocSetWithReleaseNotes("not-a-product")); + + config.ReleaseNotesProducts.Should().BeEmpty(); + diagnostics.Should().Contain(d => + d.Severity == Severity.Error && d.Message.Contains("Unknown 'release_notes' product")); + } + + [Fact] + public async Task ReleaseNotes_ProductWithoutReleaseNotesFeature_EmitsError() + { + var (config, diagnostics) = await CreateConfiguration(DocSetWithReleaseNotes("reference-only")); + + config.ReleaseNotesProducts.Should().BeEmpty(); + diagnostics.Should().Contain(d => + d.Severity == Severity.Error && d.Message.Contains("does not participate")); + } + + [Fact] + public async Task ReleaseNotes_InvalidProductId_EmitsError() + { + var (config, diagnostics) = await CreateConfiguration(DocSetWithReleaseNotes("bad/slug")); + + config.ReleaseNotesProducts.Should().BeEmpty(); + diagnostics.Should().Contain(d => + d.Severity == Severity.Error && d.Message.Contains("must match")); + } + + [Fact] + public async Task ReleaseNotes_EmptyProductValue_EmitsError() + { + var (config, diagnostics) = await CreateConfiguration(DocSetWithReleaseNotes(" ")); + + config.ReleaseNotesProducts.Should().BeEmpty(); + diagnostics.Should().Contain(d => + d.Severity == Severity.Error && d.Message.Contains("missing a 'product' value")); + } + + private static DocumentationSetFile DocSetWithReleaseNotes(params string[] products) => + new() + { + Project = "test", + TableOfContents = [], + ReleaseNotes = [.. products.Select(p => new ReleaseNotesProductReference { Product = p })] + }; + + private static async Task<(ConfigurationFile Config, IReadOnlyList Diagnostics)> CreateConfiguration(DocumentationSetFile docSet) + { + var recorder = new RecordingDiagnosticsOutput(); + var collector = new DiagnosticsCollector([recorder]); + _ = collector.StartAsync(TestContext.Current.CancellationToken); + + var root = Paths.WorkingDirectoryRoot.FullName; + var configFilePath = Path.Join(root, "docs", "_docset.yml"); + var fileSystem = new MockFileSystem(new Dictionary + { + { configFilePath, new MockFileData("") } + }, root); + + var configPath = fileSystem.FileInfo.New(configFilePath); + var docsDir = fileSystem.DirectoryInfo.New(Path.Join(root, "docs")); + + var context = new MockDocumentationSetContext(collector, fileSystem, configPath, docsDir); + var versionsConfig = new VersionsConfiguration + { + VersioningSystems = new Dictionary() + }; + var productsConfig = CreateProductsConfiguration(); + + var config = new ConfigurationFile(docSet, context, versionsConfig, productsConfig); + await collector.StopAsync(TestContext.Current.CancellationToken); + return (config, recorder.Diagnostics); + } + + private sealed class RecordingDiagnosticsOutput : IDiagnosticsOutput + { + public List Diagnostics { get; } = []; + public void Write(Diagnostic diagnostic) => Diagnostics.Add(diagnostic); + } + + private static ProductsConfiguration CreateProductsConfiguration() + { + var products = new Dictionary + { + ["elasticsearch"] = new() { Id = "elasticsearch", DisplayName = "Elasticsearch", Features = ProductFeatures.All }, + ["edot-java"] = new() { Id = "edot-java", DisplayName = "EDOT Java", Features = ProductFeatures.All }, + ["reference-only"] = new() + { + Id = "reference-only", + DisplayName = "Reference Only", + Features = new ProductFeatures { PublicReference = true, ReleaseNotes = false } + } + }; + + return new ProductsConfiguration + { + Products = products.ToFrozenDictionary(), + PublicReferenceProducts = products.ToFrozenDictionary(), + ProductDisplayNames = products.ToFrozenDictionary(kv => kv.Key, kv => kv.Value.DisplayName) + }; + } + + private sealed class MockDocumentationSetContext( + IDiagnosticsCollector collector, + IFileSystem fileSystem, + IFileInfo configurationPath, + IDirectoryInfo documentationSourceDirectory) + : IDocumentationSetContext + { + public IDiagnosticsCollector Collector => collector; + public ScopedFileSystem ReadFileSystem => WriteFileSystem; + public ScopedFileSystem WriteFileSystem { get; } = FileSystemFactory.ScopeCurrentWorkingDirectoryForWrite(fileSystem); + public IDirectoryInfo OutputDirectory => fileSystem.DirectoryInfo.New(Path.Join(Paths.WorkingDirectoryRoot.FullName, ".artifacts")); + public IFileInfo ConfigurationPath => configurationPath; + public BuildType BuildType => BuildType.Isolated; + public IDirectoryInfo DocumentationSourceDirectory => documentationSourceDirectory; + public GitCheckoutInformation Git => GitCheckoutInformationFactory.Create(documentationSourceDirectory, fileSystem); + } +} diff --git a/tests/Elastic.Documentation.Configuration.Tests/ReleaseNotes/BundleLoaderFromContentTests.cs b/tests/Elastic.Documentation.Configuration.Tests/ReleaseNotes/BundleLoaderFromContentTests.cs new file mode 100644 index 0000000000..541bea9bb3 --- /dev/null +++ b/tests/Elastic.Documentation.Configuration.Tests/ReleaseNotes/BundleLoaderFromContentTests.cs @@ -0,0 +1,112 @@ +// Licensed to Elasticsearch B.V under one or more agreements. +// Elasticsearch B.V licenses this file to you under the Apache 2.0 License. +// See the LICENSE file in the project root for more information + +using System.IO.Abstractions; +using AwesomeAssertions; +using Elastic.Documentation.Configuration.ReleaseNotes; + +namespace Elastic.Documentation.Configuration.Tests.ReleaseNotes; + +public class BundleLoaderFromContentTests +{ + private readonly BundleLoader _loader = new(new FileSystem()); + + private static (string FileName, string Content) Bundle(string fileName, string content) => (fileName, content); + + [Fact] + public void LoadBundlesFromContent_InlineEntries_AreLoaded() + { + var warnings = new List(); + // language=yaml + var bundle = Bundle("9.3.0.yaml", """ + products: + - product: elasticsearch + target: 9.3.0 + repo: elasticsearch + owner: elastic + entries: + - file: + name: 1.yaml + checksum: c0ffee + type: enhancement + title: Sample enhancement + """); + + var bundles = _loader.LoadBundlesFromContent([bundle], warnings.Add); + + warnings.Should().BeEmpty(); + bundles.Should().ContainSingle(); + var loaded = bundles[0]; + loaded.Version.Should().Be("9.3.0"); + loaded.Repo.Should().Be("elasticsearch"); + loaded.Entries.Should().ContainSingle(); + loaded.Entries[0].Title.Should().Be("Sample enhancement"); + } + + [Fact] + public void LoadBundlesFromContent_FileOnlyEntry_IsSkippedWithWarning() + { + var warnings = new List(); + // language=yaml + var bundle = Bundle("9.3.0.yaml", """ + products: + - product: elasticsearch + target: 9.3.0 + entries: + - file: + name: orphan.yaml + checksum: deadbeef + """); + + var bundles = _loader.LoadBundlesFromContent([bundle], warnings.Add); + + bundles.Should().ContainSingle(); + bundles[0].Entries.Should().BeEmpty(); + warnings.Should().ContainSingle() + .Which.Should().Contain("self-contained"); + } + + [Fact] + public void LoadBundlesFromContent_InvalidYaml_IsSkippedWithWarning() + { + var warnings = new List(); + var bundle = Bundle("broken.yaml", "products: [unterminated"); + + var bundles = _loader.LoadBundlesFromContent([bundle], warnings.Add); + + bundles.Should().BeEmpty(); + warnings.Should().ContainSingle() + .Which.Should().Contain("broken.yaml"); + } + + [Fact] + public void LoadBundlesFromContent_AmendFile_IsMergedIntoParent() + { + var warnings = new List(); + // language=yaml + var parent = Bundle("9.3.0.yaml", """ + products: + - product: elasticsearch + target: 9.3.0 + entries: + - type: enhancement + title: Base entry + """); + // language=yaml + var amend = Bundle("9.3.0.amend-1.yaml", """ + products: + - product: elasticsearch + target: 9.3.0 + entries: + - type: bug-fix + title: Amended fix + """); + + var bundles = _loader.LoadBundlesFromContent([parent, amend], warnings.Add); + + bundles.Should().ContainSingle("the amend file merges into its parent"); + bundles[0].Entries.Select(e => e.Title) + .Should().BeEquivalentTo("Base entry", "Amended fix"); + } +} diff --git a/tests/Elastic.Documentation.Configuration.Tests/ReleaseNotes/CdnChangelogFetcherTests.cs b/tests/Elastic.Documentation.Configuration.Tests/ReleaseNotes/CdnChangelogFetcherTests.cs new file mode 100644 index 0000000000..8de403a055 --- /dev/null +++ b/tests/Elastic.Documentation.Configuration.Tests/ReleaseNotes/CdnChangelogFetcherTests.cs @@ -0,0 +1,141 @@ +// Licensed to Elasticsearch B.V under one or more agreements. +// Elasticsearch B.V licenses this file to you under the Apache 2.0 License. +// See the LICENSE file in the project root for more information + +using System.IO.Abstractions; +using System.Net; +using AwesomeAssertions; +using Elastic.Documentation.Configuration.ReleaseNotes; +using Microsoft.Extensions.Logging.Abstractions; + +namespace Elastic.Documentation.Configuration.Tests.ReleaseNotes; + +public class CdnChangelogFetcherTests +{ + // language=yaml + private const string SampleBundle = """ + products: + - product: elasticsearch + target: 9.3.0 + repo: elasticsearch + owner: elastic + entries: + - type: enhancement + title: Sample enhancement + """; + + private static readonly Uri BaseUri = new("https://cdn.example"); + + private static CdnChangelogFetcher CreateFetcher(StubHandler handler) => + new(NullLoggerFactory.Instance, new FileSystem(), handler); + + private static (List Errors, List Warnings, Action EmitError, Action EmitWarning) Diagnostics() + { + var errors = new List(); + var warnings = new List(); + return (errors, warnings, errors.Add, warnings.Add); + } + + [Fact] + public async Task FetchAsync_HappyPath_ReturnsBundlesFromRegistry() + { + var handler = new StubHandler(req => + req.RequestUri!.AbsolutePath.EndsWith("/registry.json", StringComparison.Ordinal) + ? Json(/*lang=json,strict*/ """{ "schema_version": 1, "product": "elasticsearch", "bundles": [ { "file": "9.3.0.yaml", "target": "9.3.0" } ] }""") + : Yaml(SampleBundle)); + var (errors, warnings, emitError, emitWarning) = Diagnostics(); + + using var fetcher = CreateFetcher(handler); + var bundles = await fetcher.FetchAsync(BaseUri, "elasticsearch", version: null, emitError, emitWarning, TestContext.Current.CancellationToken); + + errors.Should().BeEmpty(); + warnings.Should().BeEmpty(); + bundles.Should().ContainSingle(); + bundles[0].Version.Should().Be("9.3.0"); + bundles[0].Entries.Should().ContainSingle().Which.Title.Should().Be("Sample enhancement"); + } + + [Fact] + public async Task FetchAsync_WithVersion_OnlyDownloadsMatchingBundle() + { + var handler = new StubHandler(req => + req.RequestUri!.AbsolutePath.EndsWith("/registry.json", StringComparison.Ordinal) + ? Json(/*lang=json,strict*/ """{ "schema_version": 1, "product": "elasticsearch", "bundles": [ { "file": "9.4.0.yaml", "target": "9.4.0" }, { "file": "9.3.0.yaml", "target": "9.3.0" } ] }""") + : Yaml(SampleBundle)); + var (errors, warnings, emitError, emitWarning) = Diagnostics(); + + using var fetcher = CreateFetcher(handler); + var bundles = await fetcher.FetchAsync(BaseUri, "elasticsearch", version: "9.3.0", emitError, emitWarning, TestContext.Current.CancellationToken); + + errors.Should().BeEmpty(); + warnings.Should().BeEmpty(); + bundles.Should().ContainSingle(); + handler.RequestedPaths.Should().NotContain(p => p.EndsWith("/9.4.0.yaml", StringComparison.Ordinal), + "only the requested version should be downloaded"); + handler.RequestedPaths.Should().Contain(p => p.EndsWith("/9.3.0.yaml", StringComparison.Ordinal)); + } + + [Fact] + public async Task FetchAsync_RegistryNotFound_EmitsErrorAndReturnsEmpty() + { + var handler = new StubHandler(_ => new HttpResponseMessage(HttpStatusCode.NotFound)); + var (errors, _, emitError, emitWarning) = Diagnostics(); + + using var fetcher = CreateFetcher(handler); + var bundles = await fetcher.FetchAsync(BaseUri, "elasticsearch", version: null, emitError, emitWarning, TestContext.Current.CancellationToken); + + bundles.Should().BeEmpty(); + errors.Should().ContainSingle().Which.Should().Contain("registry"); + } + + [Fact] + public async Task FetchAsync_BundleNotFound_EmitsWarningAndSkipsBundle() + { + var handler = new StubHandler(req => + req.RequestUri!.AbsolutePath.EndsWith("/registry.json", StringComparison.Ordinal) + ? Json(/*lang=json,strict*/ """{ "schema_version": 1, "product": "elasticsearch", "bundles": [ { "file": "9.3.0.yaml", "target": "9.3.0" } ] }""") + : new HttpResponseMessage(HttpStatusCode.NotFound)); + var (errors, warnings, emitError, emitWarning) = Diagnostics(); + + using var fetcher = CreateFetcher(handler); + var bundles = await fetcher.FetchAsync(BaseUri, "elasticsearch", version: null, emitError, emitWarning, TestContext.Current.CancellationToken); + + bundles.Should().BeEmpty(); + errors.Should().BeEmpty(); + warnings.Should().ContainSingle().Which.Should().Contain("9.3.0.yaml"); + } + + [Fact] + public async Task FetchAsync_SchemaVersionTooNew_EmitsError() + { + var handler = new StubHandler(_ => + Json(/*lang=json,strict*/ """{ "schema_version": 999, "product": "elasticsearch", "bundles": [] }""")); + var (errors, _, emitError, emitWarning) = Diagnostics(); + + using var fetcher = CreateFetcher(handler); + var bundles = await fetcher.FetchAsync(BaseUri, "elasticsearch", version: null, emitError, emitWarning, TestContext.Current.CancellationToken); + + bundles.Should().BeEmpty(); + errors.Should().ContainSingle().Which.Should().Contain("schema version"); + } + + private static HttpResponseMessage Json(string body) => + new(HttpStatusCode.OK) { Content = new StringContent(body, System.Text.Encoding.UTF8, "application/json") }; + + private static HttpResponseMessage Yaml(string body) => + new(HttpStatusCode.OK) { Content = new StringContent(body, System.Text.Encoding.UTF8, "text/yaml") }; + + private sealed class StubHandler(Func responder) : HttpMessageHandler + { + public int CallCount { get; private set; } + + public List RequestedPaths { get; } = []; + + protected override Task SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) + { + CallCount++; + RequestedPaths.Add(request.RequestUri!.AbsolutePath); + return Task.FromResult(responder(request)); + } + } +} diff --git a/tests/Elastic.Markdown.Tests/Directives/ChangelogBasicTests.cs b/tests/Elastic.Markdown.Tests/Directives/ChangelogBasicTests.cs index 4307d815f6..82aad278b2 100644 --- a/tests/Elastic.Markdown.Tests/Directives/ChangelogBasicTests.cs +++ b/tests/Elastic.Markdown.Tests/Directives/ChangelogBasicTests.cs @@ -2,8 +2,12 @@ // Elasticsearch B.V licenses this file to you under the Apache 2.0 License. // See the LICENSE file in the project root for more information +using System.Collections.Frozen; using System.IO.Abstractions.TestingHelpers; using AwesomeAssertions; +using Elastic.Documentation.Configuration; +using Elastic.Documentation.Configuration.ReleaseNotes; +using Elastic.Documentation.ReleaseNotes; using Elastic.Markdown.Myst.Directives.Changelog; namespace Elastic.Markdown.Tests.Directives; @@ -155,6 +159,103 @@ public void RendersAllVersions() } } +/// +/// Verifies the :version: option filters local-folder bundles down to the single matching +/// target, leaving the others out of both the loaded set and the rendered output. +/// +public class ChangelogVersionFilterTests : DirectiveTest +{ + public ChangelogVersionFilterTests(ITestOutputHelper output) : base(output, + // language=markdown + """ + :::{changelog} + :version: 9.3.0 + ::: + """) + { + FileSystem.AddFile("docs/changelog/bundles/9.2.0.yaml", new MockFileData( + // language=yaml + """ + products: + - product: elasticsearch + target: 9.2.0 + entries: + - title: Feature in 9.2.0 + type: feature + products: + - product: elasticsearch + target: 9.2.0 + prs: + - "111111" + """)); + + FileSystem.AddFile("docs/changelog/bundles/9.3.0.yaml", new MockFileData( + // language=yaml + """ + products: + - product: elasticsearch + target: 9.3.0 + entries: + - title: Feature in 9.3.0 + type: feature + products: + - product: elasticsearch + target: 9.3.0 + prs: + - "222222" + """)); + } + + [Fact] + public void CapturesVersionOption() => Block!.VersionFilter.Should().Be("9.3.0"); + + [Fact] + public void LoadsOnlyMatchingBundle() => Block!.LoadedBundles.Should().ContainSingle().Which.Version.Should().Be("9.3.0"); + + [Fact] + public void RendersOnlyMatchingVersion() + { + Html.Should().Contain("Feature in 9.3.0"); + Html.Should().NotContain("Feature in 9.2.0"); + } +} + +/// +/// Verifies a :version: value that matches no bundle renders nothing and warns instead of +/// silently falling back to all versions. +/// +public class ChangelogVersionFilterNoMatchTests : DirectiveTest +{ + public ChangelogVersionFilterNoMatchTests(ITestOutputHelper output) : base(output, + // language=markdown + """ + :::{changelog} + :version: 1.2.3 + ::: + """) => FileSystem.AddFile("docs/changelog/bundles/9.3.0.yaml", new MockFileData( + // language=yaml + """ + products: + - product: elasticsearch + target: 9.3.0 + entries: + - title: Feature in 9.3.0 + type: feature + products: + - product: elasticsearch + target: 9.3.0 + prs: + - "222222" + """)); + + [Fact] + public void LoadsNoBundles() => Block!.LoadedBundles.Should().BeEmpty(); + + [Fact] + public void EmitsWarningForUnmatchedVersion() => + Collector.Diagnostics.Should().Contain(d => d.Message.Contains("No changelog bundle matches :version:")); +} + public class ChangelogCustomPathTests : DirectiveTest { public ChangelogCustomPathTests(ITestOutputHelper output) : base(output, @@ -192,6 +293,257 @@ public void RendersContent() } } +/// +/// Verifies :cdn: product validation. An invalid product name is rejected before it is +/// assigned to the block and before any network access, so this test exercises the wiring without +/// touching the CDN. +/// +public class ChangelogCdnInvalidProductTests(ITestOutputHelper output) : DirectiveTest(output, + // language=markdown + """ + :::{changelog} + :cdn: invalid$product + ::: + """) +{ + [Fact] + public void DoesNotCaptureInvalidCdnProduct() => Block!.CdnProduct.Should().BeNull(); + + [Fact] + public void DoesNotSourceFromLocalFolder() => Block!.BundlesFolderPath.Should().BeNull(); + + [Fact] + public void EmitsErrorForInvalidProduct() + { + Collector.Diagnostics.Should().NotBeNullOrEmpty(); + Collector.Diagnostics.Should().Contain(d => d.Message.Contains("Invalid :cdn: product")); + } +} + +/// +/// Verifies CDN-sourced bundles render into the page body (not just the page TOC). The directive is a +/// selector over release notes prefetched at startup, so the test injects a resolver holding the bundle +/// instead of hitting the network. Regression guard: the HTML renderer previously gated on the +/// (CDN-null) local bundles folder path and silently emitted an empty body. +/// +public class ChangelogCdnRenderTests(ITestOutputHelper output) : DirectiveTest(output, + // language=markdown + """ + :::{changelog} + :cdn: cdn-render-test + ::: + """) +{ + private const string Product = "cdn-render-test"; + + protected override IReleaseNotesResolver GetReleaseNotesResolver() => + ChangelogCdnTestResolver.For(Product, + ("9.4.0.yaml", + // language=yaml + """ + products: + - product: cdn-render-test + target: 9.4.0 + repo: elasticsearch + owner: elastic + entries: + - title: Faster vector search on the CDN + type: enhancement + products: + - product: cdn-render-test + target: 9.4.0 + prs: + - "999" + """)); + + [Fact] + public void FoundFromCdn() => Block!.Found.Should().BeTrue(); + + [Fact] + public void RendersCdnBundleBody() + { + Html.Should().Contain("9.4.0"); + Html.Should().Contain("Features and enhancements"); + Html.Should().Contain("Faster vector search on the CDN"); + } +} + +/// +/// Verifies :cdn: combined with :version: renders only the matching prefetched bundle. +/// Version filtering is applied to the injected resolver's bundles, so no network access occurs. +/// +public class ChangelogCdnVersionFilterTests(ITestOutputHelper output) : DirectiveTest(output, + // language=markdown + """ + :::{changelog} + :cdn: cdn-version-test + :version: 9.4.0 + ::: + """) +{ + private const string Product = "cdn-version-test"; + + protected override IReleaseNotesResolver GetReleaseNotesResolver() => + ChangelogCdnTestResolver.For(Product, + ("9.4.0.yaml", + // language=yaml + """ + products: + - product: cdn-version-test + target: 9.4.0 + repo: elasticsearch + owner: elastic + entries: + - title: Selected version entry + type: enhancement + products: + - product: cdn-version-test + target: 9.4.0 + prs: + - "999" + """), + ("9.3.0.yaml", + // language=yaml + """ + products: + - product: cdn-version-test + target: 9.3.0 + repo: elasticsearch + owner: elastic + entries: + - title: Filtered out entry + type: enhancement + products: + - product: cdn-version-test + target: 9.3.0 + prs: + - "998" + """)); + + [Fact] + public void CapturesVersionFilter() => Block!.VersionFilter.Should().Be("9.4.0"); + + [Fact] + public void RendersOnlyMatchingVersion() + { + Block!.Found.Should().BeTrue(); + Html.Should().Contain("Selected version entry"); + Html.Should().NotContain("Filtered out entry"); + } +} + +/// +/// Verifies a valueless :cdn: infers the product from the current repository name. With a +/// .git marker present the mock git checkout reports the repository as docs-builder, so +/// the directive selects that product from the injected resolver. +/// +public class ChangelogCdnInferredProductTests(ITestOutputHelper output) : DirectiveTest(output, + // language=markdown + """ + :::{changelog} + :cdn: + ::: + """) +{ + private const string InferredProduct = "docs-builder"; + + // A .git marker makes FindGitRoot resolve a checkout directory, which lets the mock-aware + // GitCheckoutInformationFactory report the repository name (docs-builder) for inference. + protected override void AddToFileSystem(MockFileSystem fileSystem) => + fileSystem.AddDirectory(Path.Combine(Paths.WorkingDirectoryRoot.FullName, ".git")); + + protected override IReleaseNotesResolver GetReleaseNotesResolver() => + ChangelogCdnTestResolver.For(InferredProduct, + ("9.4.0.yaml", + // language=yaml + """ + products: + - product: docs-builder + target: 9.4.0 + repo: docs-builder + owner: elastic + entries: + - title: Inferred product from the repository + type: enhancement + products: + - product: docs-builder + target: 9.4.0 + prs: + - "999" + """)); + + [Fact] + public void InfersProductFromRepository() => Block!.CdnProduct.Should().Be(InferredProduct); + + [Fact] + public void RendersInferredCdnBundleBody() + { + Block!.Found.Should().BeTrue(); + Html.Should().Contain("Inferred product from the repository"); + } +} + +/// +/// A valueless :cdn: must fail with a clear error when the product cannot be inferred (no git +/// information available), rather than silently rendering empty. +/// +public class ChangelogCdnInferredProductUnavailableTests(ITestOutputHelper output) : DirectiveTest(output, + // language=markdown + """ + :::{changelog} + :cdn: + ::: + """) +{ + [Fact] + public void EmitsErrorWhenProductCannotBeInferred() + { + Block!.Found.Should().BeFalse(); + Collector.Diagnostics.Should().Contain(d => d.Message.Contains("could not be inferred")); + } +} + +/// +/// A :cdn: product that is not declared under release_notes in docset.yml must fail with a +/// clear error (the bundles were never prefetched), pointing the author at the declaration to add. +/// +public class ChangelogCdnUndeclaredProductTests(ITestOutputHelper output) : DirectiveTest(output, + // language=markdown + """ + :::{changelog} + :cdn: not-declared + ::: + """) +{ + [Fact] + public void EmitsErrorWhenProductIsNotDeclared() + { + Block!.Found.Should().BeFalse(); + Collector.Diagnostics.Should().Contain(d => + d.Message.Contains("not declared in docset.yml") && d.Message.Contains("release_notes")); + } +} + +/// +/// Test helper that builds an backed by in-memory bundle content, +/// standing in for the startup CDN prefetch. +/// +internal static class ChangelogCdnTestResolver +{ + public static IReleaseNotesResolver For(string product, params (string FileName, string Content)[] bundleContents) + { + var bundles = new BundleLoader(new MockFileSystem()).LoadBundlesFromContent(bundleContents, _ => { }); + return new ReleaseNotesResolver(new FetchedReleaseNotes + { + BundlesByProduct = new Dictionary>(StringComparer.Ordinal) + { + [product] = bundles + }.ToFrozenDictionary(StringComparer.Ordinal), + DeclaredProducts = new[] { product }.ToFrozenSet(StringComparer.Ordinal) + }); + } +} + public class ChangelogNotFoundTests(ITestOutputHelper output) : DirectiveTest(output, // language=markdown """ diff --git a/tests/Elastic.Markdown.Tests/Directives/DirectiveBaseTests.cs b/tests/Elastic.Markdown.Tests/Directives/DirectiveBaseTests.cs index 48edbc964b..a6710cf717 100644 --- a/tests/Elastic.Markdown.Tests/Directives/DirectiveBaseTests.cs +++ b/tests/Elastic.Markdown.Tests/Directives/DirectiveBaseTests.cs @@ -4,6 +4,7 @@ using System.IO.Abstractions.TestingHelpers; using AwesomeAssertions; using Elastic.Documentation.Configuration; +using Elastic.Documentation.Configuration.ReleaseNotes; using Elastic.Markdown.IO; using Elastic.Markdown.Myst.Directives; using JetBrains.Annotations; @@ -73,7 +74,8 @@ protected DirectiveTest(ITestOutputHelper output, [LanguageInjection("markdown") var configurationContext = TestHelpers.CreateConfigurationContext(FileSystem); var context = new BuildContext(Collector, FileSystemFactory.ScopeCurrentWorkingDirectory(FileSystem), configurationContext); var linkResolver = new TestCrossLinkResolver(); - Set = new DocumentationSet(context, logger, linkResolver); + // ReSharper disable once VirtualMemberCallInConstructor + Set = new DocumentationSet(context, logger, linkResolver, GetReleaseNotesResolver()); File = Set.TryFindDocument(FileSystem.FileInfo.New("docs/index.md")) as MarkdownFile ?? throw new NullReferenceException(); Html = default!; //assigned later Document = default!; @@ -89,6 +91,12 @@ protected virtual void AddToFileSystem(MockFileSystem fileSystem) { } protected virtual string? GetDocsetExtraYaml() => null; + /// + /// Override to inject a resolver of CDN-prefetched changelog bundles for the {changelog} + /// :cdn: directive. Returns null by default (no-op resolver), so non-CDN tests are unaffected. + /// + protected virtual IReleaseNotesResolver? GetReleaseNotesResolver() => null; + public virtual async ValueTask InitializeAsync() { _ = Collector.StartAsync(TestContext.Current.CancellationToken); diff --git a/tests/Navigation.Tests/Isolation/PhysicalDocsetTests.cs b/tests/Navigation.Tests/Isolation/PhysicalDocsetTests.cs index 4f16c0fcdf..09fe59cdf3 100644 --- a/tests/Navigation.Tests/Isolation/PhysicalDocsetTests.cs +++ b/tests/Navigation.Tests/Isolation/PhysicalDocsetTests.cs @@ -124,9 +124,9 @@ public async Task PhysicalDocsetNavigationIncludesNestedTocs() var developmentToc = tocNavs.FirstOrDefault(t => t.Url == "/development"); developmentToc.Should().NotBeNull(); - developmentToc.NavigationItems.Should().HaveCount(3); + developmentToc.NavigationItems.Should().HaveCount(4); developmentToc.Index.Should().NotBeNull(); - developmentToc.NavigationItems.OfType>().Should().HaveCount(0); + developmentToc.NavigationItems.OfType>().Should().HaveCount(1); developmentToc.NavigationItems.OfType>().Should().HaveCount(2); developmentToc.NavigationItems.OfType>().Should().HaveCount(1);