feat(snippets): entrypoint-based render + sdk.yaml reduced to id (#405)

* feat(snippets): entrypoint-based render + sdk.yaml reduced to id

Two related schema changes that simplify the contract between this
repo and its consumers:

1. Drop the `ld-application.get-started-file(s)` field entirely.
   The renderer no longer asks each sdk.yaml which file to rewrite.
   Instead the consumer (gonfalon, ld-docs-private) declares its own
   list of entrypoint directories, and the renderer walks them
   recursively for files containing the SDK_SNIPPET:RENDER sentinel.

   - `Render(sdksFS, entrypoints []string)` and the matching `Verify`
     replace the old `(sdksFS, appDir)` signatures.
   - The walk skips `node_modules`, `.git`, `.next`, `dist`, `build`,
     `.cache`, `coverage`, `out`, `.turbo`. Files are filtered by
     extension (.tsx/.jsx/.ts/.js/.mdx) and by a fast bytes.Contains
     pre-check for the marker sentinel before invoking the regex
     scanner.
   - Symlinks are skipped, so a malicious symlink farm in a vendored
     directory can't pull the renderer outside the entrypoint.
   - Overlapping entrypoints dedupe.
   - Empty entrypoint list, missing entrypoint, or entrypoint that
     points at a file all fail loudly.

2. Reduce sdk.yaml to a single field: `id:`. The id matches the
   SDK's identifier in launchdarkly/sdk-meta's API packages, where the
   authoritative metadata lives (display name, type, languages,
   regions, hello-world repo, docs URL). Every previously-modeled
   field on the descriptor was unread by this codebase; downstream
   consumers that need that metadata read sdk-meta directly.

   - The `descriptor` struct + `loadDescriptor` function are deleted.
     Nothing in the engine reads sdk.yaml today.
   - Each of the 23 sdk.yaml files reduces to a header comment plus
     `id: <sdk>`.

CLI:
  Old: snippets render --target=ld-application --out=<consumer>
  New: snippets render --target=ld-application --entrypoint=<dir> ...

  --entrypoint is a custom flag.Value that accumulates across repeats
  (mirrors the standard Go-stdlib stringSliceFlag idiom).

End-to-end against gonfalon's static/ld/components/getStarted:
  - Reset every marker hash to 0 (122 markers in 24 files).
  - `snippets render --entrypoint=...getStarted` rewrites all 24
    files via the embedded sdks/ tree.
  - Diff against pre-reset stash: byte-identical.
  - `snippets verify --entrypoint=...getStarted`: ok.

Companion changes follow:
  - launchdarkly/gh-actions PR #82 needs to add an `entrypoints` input
    that the consumer-side workflow declares.
  - This PR stacks on #404 (the embedding +
    release pipeline PR).

* feat(snippets): drop ld-application.slot

Per the schema audit, slot was a write-only field — declared on the
LDApplicationHints struct, decoded from frontmatter, never read by any
production code. The marker comment carries the snippet id directly,
so the per-page lookup slot was meant to power doesn't happen.

- Remove the LDApplicationHints struct and the LDApplication field on
  Frontmatter.
- Strip the `ld-application:\n  slot: <name>\n` block from all 122
  snippet files (KnownFields(true) would otherwise fail to parse them
  after the struct change).
- Render output unaffected: end-to-end against gonfalon shows
  `no changes` + `ok` from verify.

* fix(snippets): always descend into the entrypoint root

skipDirNames prunes well-known noise (node_modules, dist, build, …) under
each entrypoint, but filepath.WalkDir invokes the callback for the root
itself first. If a consumer pointed `--entrypoint=./build` at a project
that emits TSX there, the root's basename matched the skip-list and the
walk silently produced zero files. Guard the skip check so it only fires
beneath the root, and add a regression test.

Author: kinyoklion

snippets/cmd/snippets/main.go

@@ -5,6 +5,7 @@ import (
 	"fmt"
 	"io/fs"
 	"os"
+	"strings"
 
 	"github.com/launchdarkly/sdk-meta/snippets"
 	"github.com/launchdarkly/sdk-meta/snippets/internal/adapters/ldapplication"
@@ -23,19 +24,30 @@ func resolveSDKsFS(sdksFlag string) fs.FS {
 	return os.DirFS(sdksFlag)
 }
 
+// repeatableString implements flag.Value so a flag like --entrypoint can
+// be passed multiple times and accumulate into a slice. Mirrors the
+// stringSliceFlag idiom widely used in Go CLIs that stick to the stdlib
+// `flag` package.
+type repeatableString []string
+
+func (r *repeatableString) String() string     { return strings.Join(*r, ",") }
+func (r *repeatableString) Set(s string) error { *r = append(*r, s); return nil }
+
 const usage = `snippets — LaunchDarkly SDK snippet generator
 
 usage:
-  snippets render   --target=ld-application --out=<app-checkout> [--sdks=./sdks]
-      Rewrites the consumer's marked regions in place from the snippet sources.
-      This is the command authors run after editing a snippet, and the command
-      that produces the diff in the consumer repo's PR.
+  snippets render --target=ld-application --entrypoint=<dir> [--entrypoint=<dir2> ...] [--sdks=./sdks]
+      Walks each --entrypoint directory in the consumer checkout, finds files
+      that contain SDK_SNIPPET:RENDER markers, and rewrites each marked
+      region from the snippet sources. --entrypoint may be passed multiple
+      times. Authors run this after editing a snippet; the consumer's
+      sync action runs it on every release.
 
-  snippets verify   --target=ld-application --out=<app-checkout> [--sdks=./sdks]
-      Read-only check used by CI in the consumer repo. Re-renders every marked
-      region in memory and fails if the rendered bytes drift from what's on
-      disk, or if a marker's hash does not match its current region's content.
-      Never writes; never executes any snippet code.
+  snippets verify --target=ld-application --entrypoint=<dir> [--entrypoint=<dir2> ...] [--sdks=./sdks]
+      Read-only counterpart to render, used by CI in the consumer repo.
+      Fails if the rendered bytes would drift from what's on disk, or if
+      a marker's hash does not match its current region's content. Never
+      writes; never executes any snippet code.
 
   snippets validate --sdk=<sdk-id> [--sdks=./sdks] [--validators=./validators]
       Builds the SDK's per-language validator (Docker image or native harness),
@@ -73,15 +85,16 @@ func main() {
 func runRender(args []string) {
 	fset := flag.NewFlagSet("render", flag.ExitOnError)
 	target := fset.String("target", "", "adapter target: `ld-application`")
-	out := fset.String("out", "", "path to the consumer checkout")
+	var entrypoints repeatableString
+	fset.Var(&entrypoints, "entrypoint", "directory in the consumer checkout to walk for marker files (repeatable)")
 	sdks := fset.String("sdks", "", "path to a sdks/ directory (default: embedded)")
 	_ = fset.Parse(args)
 
-	if *target != "ld-application" || *out == "" {
-		fmt.Fprintf(os.Stderr, "render: --target=ld-application and --out are required\n")
+	if *target != "ld-application" || len(entrypoints) == 0 {
+		fmt.Fprintf(os.Stderr, "render: --target=ld-application and at least one --entrypoint are required\n")
 		os.Exit(2)
 	}
-	changed, err := ldapplication.Render(resolveSDKsFS(*sdks), *out)
+	changed, err := ldapplication.Render(resolveSDKsFS(*sdks), entrypoints)
 	if err != nil {
 		fmt.Fprintf(os.Stderr, "render failed: %v\n", err)
 		os.Exit(1)
@@ -98,15 +111,16 @@ func runRender(args []string) {
 func runVerify(args []string) {
 	fset := flag.NewFlagSet("verify", flag.ExitOnError)
 	target := fset.String("target", "", "adapter target: `ld-application`")
-	out := fset.String("out", "", "path to the consumer checkout")
+	var entrypoints repeatableString
+	fset.Var(&entrypoints, "entrypoint", "directory in the consumer checkout to walk for marker files (repeatable)")
 	sdks := fset.String("sdks", "", "path to a sdks/ directory (default: embedded)")
 	_ = fset.Parse(args)
 
-	if *target != "ld-application" || *out == "" {
-		fmt.Fprintf(os.Stderr, "verify: --target=ld-application and --out are required\n")
+	if *target != "ld-application" || len(entrypoints) == 0 {
+		fmt.Fprintf(os.Stderr, "verify: --target=ld-application and at least one --entrypoint are required\n")
 		os.Exit(2)
 	}
-	if err := ldapplication.Verify(resolveSDKsFS(*sdks), *out); err != nil {
+	if err := ldapplication.Verify(resolveSDKsFS(*sdks), entrypoints); err != nil {
 		fmt.Fprintf(os.Stderr, "verify failed: %v\n", err)
 		os.Exit(1)
 	}

snippets/docs/AUTHORING.md

@@ -75,11 +75,17 @@ export LAUNCHDARKLY_SDK_KEY=...     # server-side key
 export LAUNCHDARKLY_FLAG_KEY=...    # flag the snippet evaluates
 snippets validate --sdk=python-server-sdk
 
-# Rewrite all marked regions in an ld-application checkout
-snippets render --target=ld-application --out=/path/to/ld-application
-
-# Confirm consumer file matches what we'd render (no edits)
-snippets verify --target=ld-application --out=/path/to/ld-application
+# Rewrite every marked region under one or more entrypoint dirs in the
+# consumer checkout. --entrypoint is repeatable; the renderer walks each
+# directory recursively, only opens files whose extension it understands
+# (.tsx/.jsx/.ts/.js/.mdx) AND that contain the SDK_SNIPPET:RENDER
+# sentinel, and skips junk dirs (node_modules, .git, dist, build, ...).
+snippets render --target=ld-application \
+  --entrypoint=/path/to/ld-application/static/ld/components/getStarted
+
+# Confirm consumer files match what we'd render (no edits)
+snippets verify --target=ld-application \
+  --entrypoint=/path/to/ld-application/static/ld/components/getStarted
 ```
 
 ## Validator inputs

snippets/internal/adapters/ldapplication/descriptor.go

@@ -1,12 +1,12 @@
 package ldapplication
 
 import (
+	"bytes"
 	"crypto/rand"
 	"encoding/hex"
 	"fmt"
 	"io/fs"
 	"os"
-	"path"
 	"path/filepath"
 	"strings"
 
@@ -16,104 +16,136 @@ import (
 	"github.com/launchdarkly/sdk-meta/snippets/internal/version"
 )
 
-// Render walks every SDK's get-started TSX file under appDir, finds render
-// markers, and rewrites each marked region with the rendered snippet content.
-// sdksFS is the fs.FS rooted at the sdks/ directory (either embedded or
-// os.DirFS(path)). Returns one entry per file it touched.
-func Render(sdksFS fs.FS, appDir string) ([]string, error) {
-	snippets, err := model.LoadAll(sdksFS)
-	if err != nil {
-		return nil, err
-	}
+// markerSentinel is the substring every render marker contains. We use it as
+// a fast pre-filter so walking large consumer trees only invokes the marker
+// scanner on files that could possibly contain a marker. Cheaper than a
+// regex match per file.
+var markerSentinel = []byte("SDK_SNIPPET:RENDER:")
 
-	files, err := discoverTargetFiles(sdksFS, appDir)
-	if err != nil {
-		return nil, err
-	}
+// candidateExtensions are the file extensions whose comment syntax the
+// marker scanner understands today: JS, TS, JSX, TSX, plus MDX (whose JSX
+// expression comments share the `{/* ... */}` form). Everything else is
+// skipped on sight so a `node_modules` walk doesn't eat a million PNGs.
+var candidateExtensions = map[string]struct{}{
+	".tsx": {}, ".jsx": {}, ".ts": {}, ".js": {}, ".mdx": {},
+}
 
-	var changed []string
-	for _, path := range files {
-		ok, err := rewriteFile(path, snippets, false)
-		if err != nil {
-			return nil, fmt.Errorf("%s: %w", path, err)
-		}
-		if ok {
-			changed = append(changed, path)
-		}
-	}
-	return changed, nil
+// skipDirNames are directories we never descend into. They never carry
+// snippet markers and routinely have hundreds of thousands of files
+// (node_modules), generated build output, or version-control bookkeeping.
+var skipDirNames = map[string]struct{}{
+	"node_modules": {}, ".git": {}, ".next": {}, "dist": {}, "build": {},
+	".cache": {}, "coverage": {}, "out": {}, ".turbo": {},
+}
+
+// Render walks every entrypoint, finds files that contain render markers,
+// and rewrites each marked region with the rendered snippet content. sdksFS
+// is the fs.FS rooted at the sdks/ directory (either embedded or
+// os.DirFS(path)). entrypoints are absolute or repo-relative directory
+// paths the consumer (gonfalon, ld-docs) declares as roots to scan.
+// Returns one entry per file it touched.
+func Render(sdksFS fs.FS, entrypoints []string) ([]string, error) {
+	return run(sdksFS, entrypoints, false)
 }
 
 // Verify re-renders every marked region in memory and fails if any hash in a
 // marker does not match the hash of the bytes currently in the file, or if a
 // re-render would change content. Never modifies files.
-func Verify(sdksFS fs.FS, appDir string) error {
+func Verify(sdksFS fs.FS, entrypoints []string) error {
+	_, err := run(sdksFS, entrypoints, true)
+	return err
+}
+
+func run(sdksFS fs.FS, entrypoints []string, dryRun bool) ([]string, error) {
 	snippets, err := model.LoadAll(sdksFS)
 	if err != nil {
-		return err
+		return nil, err
 	}
 
-	files, err := discoverTargetFiles(sdksFS, appDir)
+	files, err := discoverFilesUnder(entrypoints)
 	if err != nil {
-		return err
+		return nil, err
 	}
 
-	for _, path := range files {
-		if _, err := rewriteFile(path, snippets, true); err != nil {
-			return fmt.Errorf("%s: %w", path, err)
+	var changed []string
+	for _, p := range files {
+		ok, err := rewriteFile(p, snippets, dryRun)
+		if err != nil {
+			return nil, fmt.Errorf("%s: %w", p, err)
+		}
+		if ok {
+			changed = append(changed, p)
 		}
 	}
-	return nil
+	return changed, nil
 }
 
-// discoverTargetFiles returns every file referenced by an sdk.yaml's
-// `ld-application.get-started-file` field, resolved relative to appDir.
-//
-// Each referenced path must be a clean relative path that stays inside
-// appDir. This guards against a malicious sdk.yaml committing
-// `get-started-file: ../../../foo` and the renderer overwriting arbitrary
-// files outside the consumer checkout.
-func discoverTargetFiles(sdksFS fs.FS, appDir string) ([]string, error) {
-	entries, err := fs.ReadDir(sdksFS, ".")
-	if err != nil {
-		return nil, err
-	}
-	absAppDir, err := filepath.Abs(appDir)
-	if err != nil {
-		return nil, err
+// discoverFilesUnder walks every entrypoint directory and returns the set
+// of files whose extension the marker scanner understands AND whose contents
+// contain the SDK_SNIPPET:RENDER: sentinel. Symlinks are not followed so
+// a malicious symlink farm in node_modules can't pull the renderer outside
+// its intended scope. Duplicates (when two entrypoints overlap) are
+// collapsed.
+func discoverFilesUnder(entrypoints []string) ([]string, error) {
+	if len(entrypoints) == 0 {
+		return nil, fmt.Errorf("at least one --entrypoint is required")
 	}
+	seen := map[string]struct{}{}
 	var out []string
-	for _, e := range entries {
-		if !e.IsDir() {
-			continue
+	for _, ep := range entrypoints {
+		abs, err := filepath.Abs(ep)
+		if err != nil {
+			return nil, fmt.Errorf("entrypoint %q: %w", ep, err)
 		}
-		descPath := path.Join(e.Name(), "sdk.yaml")
-		desc, err := loadDescriptor(sdksFS, descPath)
+		info, err := os.Stat(abs)
 		if err != nil {
-			if os.IsNotExist(err) {
-				continue
-			}
-			return nil, err
+			return nil, fmt.Errorf("entrypoint %q: %w", ep, err)
 		}
-		rels := desc.LDApplication.GetStartedFiles
-		if rel := desc.LDApplication.GetStartedFile; rel != "" {
-			rels = append([]string{rel}, rels...)
+		if !info.IsDir() {
+			return nil, fmt.Errorf("entrypoint %q: not a directory", ep)
 		}
-		for _, rel := range rels {
-			if filepath.IsAbs(rel) {
-				return nil, fmt.Errorf("descriptor %s: get-started-file %q must be relative", descPath, rel)
+		err = filepath.WalkDir(abs, func(p string, d os.DirEntry, walkErr error) error {
+			if walkErr != nil {
+				return walkErr
+			}
+			if d.IsDir() {
+				// Always descend into the entrypoint root, even if its
+				// basename happens to be in skipDirNames (e.g.
+				// `--entrypoint=./build` for a project that lays its
+				// generated TSX out there). The skip-list is meant to prune
+				// well-known noise *under* the root, not to silently turn
+				// the entire walk into a no-op.
+				if p != abs {
+					if _, skip := skipDirNames[d.Name()]; skip {
+						return filepath.SkipDir
+					}
+				}
+				return nil
+			}
+			// Skip symlinks — we don't follow links into other parts of the
+			// repo (or out of it).
+			if d.Type()&os.ModeSymlink != 0 {
+				return nil
 			}
-			full := filepath.Join(absAppDir, rel)
-			// Reject any path that escapes appDir. filepath.Rel followed by a
-			// `..` prefix check is the canonical way to do this.
-			relCheck, err := filepath.Rel(absAppDir, full)
-			if err != nil || relCheck == ".." || strings.HasPrefix(relCheck, ".."+string(filepath.Separator)) {
-				return nil, fmt.Errorf("descriptor %s: get-started-file %q escapes appDir", descPath, rel)
+			if _, ok := candidateExtensions[filepath.Ext(p)]; !ok {
+				return nil
 			}
-			if _, err := os.Stat(full); err != nil {
-				return nil, fmt.Errorf("descriptor %s: target not found: %w", descPath, err)
+			if _, dup := seen[p]; dup {
+				return nil
 			}
-			out = append(out, full)
+			data, err := os.ReadFile(p)
+			if err != nil {
+				return err
+			}
+			if !bytes.Contains(data, markerSentinel) {
+				return nil
+			}
+			seen[p] = struct{}{}
+			out = append(out, p)
+			return nil
+		})
+		if err != nil {
+			return nil, err
 		}
 	}
 	return out, nil