Merge branch 'main' into andreatgretel/fix/agentic-ci-auth-comment

Author: johnnygreco

.agents/agents/docs-searcher.md

@@ -63,7 +63,7 @@ Brief summary of what was found and any recommendations for the user.
 - Only include results that are actually relevant to the search topic
 - If no relevant documentation is found, clearly state that
 - Keep excerpts concise but include enough context to be useful
-- Prioritize user guides and examples over API reference when both exist
+- Prioritize user guides, concepts, tutorials, and recipes according to the user's task
 - If the docs/ folder doesn't exist or is empty, report that clearly
 
 ## Search Strategy

.agents/recipes/code-quality/recipe.md

@@ -152,7 +152,7 @@ Examples of things to test (pick 2-3 per run, and invent new ones):
 - Column names with special characters or very long strings
 - Recently changed validators (check `git log --oneline -10 -- packages/*/src/data_designer/config/`)
 
-**API reference:**
+**Useful imports:**
 
 ```python
 from data_designer.config.config_builder import DataDesignerConfigBuilder

.agents/recipes/docs-and-references/recipe.md

@@ -101,9 +101,6 @@ Review for accuracy against the current code:
   the most recent 3-5 posts for references to functions, classes, or
   architecture that have since been modified.
 
-**Code reference** (`docs/code_reference/`):
-- Check that autodoc module paths point to modules that still exist.
-
 **Prioritize by risk of drift**: pages with the most code symbols referenced
 are most likely to be stale. Don't read every page - sample 5-10 high-value
 pages and flag patterns.

.agents/recipes/test-health/recipe.md

@@ -208,7 +208,7 @@ without at least one provider configured. Stick to config-layer checks
 (`DataDesignerConfigBuilder.build()`, column type resolution) which do
 not require providers.
 
-**API reference** for writing checks:
+**Useful imports** for writing checks:
 
 ```python
 from data_designer.config.config_builder import DataDesignerConfigBuilder

.agents/skills/datadesigner-docs/SKILL.md

@@ -4,8 +4,8 @@ description: >
   Maintain the NeMo Data Designer Fern docs site under fern/. Use for any
   documentation change. Triggered by: "edit docs", "add doc page", "update
   docs", "rename page", "fix broken link", "add redirect", "preview docs",
-  "publish docs", "regenerate notebooks", "update dev note", "add API
-  reference", any request that touches `fern/`.
+  "publish docs", "regenerate notebooks", "update dev note", any request
+  that touches `fern/`.
 ---
 
 # Data Designer Docs Maintenance
@@ -16,7 +16,7 @@ Current URL: **`datadesigner.docs.buildwithfern.com/nemo/datadesigner`** (see `i
 
 ## Scope Rule
 
-**ALL doc edits happen under `fern/`.** The legacy `docs/` directory is the original MkDocs source. `docs/notebook_source/*.py` remains canonical for notebook code, but **do not add new top-level prose pages under `docs/`**. Concept pages, recipes, plugins, code reference, and Dev Notes prose live under `fern/versions/latest/pages/`.
+**ALL doc edits happen under `fern/`.** The legacy `docs/` directory is the original MkDocs source. `docs/notebook_source/*.py` remains canonical for notebook code, but **do not add new top-level prose pages under `docs/`**. Concept pages, recipes, plugins, and Dev Notes prose live under `fern/versions/latest/pages/`.
 
 ## Versioning Model
 
@@ -39,7 +39,7 @@ For future Fern-native releases, do not copy page trees by hand on `main`. The r
 ```
 fern/
 ├── README.md                  ← maintainer cheat sheet
-├── docs.yml                   ← title, theme, versions:, libraries:, redirects, custom-domain
+├── docs.yml                   ← title, theme, versions:, redirects, custom-domain
 ├── fern.config.json           ← organization + fern-api version pin
 ├── main.css                   ← bundled NVIDIA theme CSS
 ├── assets/                    ← logos, favicon, recipe assets, devnote post images (shared)
@@ -56,7 +56,6 @@ fern/
 │   └── devnotes/              ← .authors.yml, authors-data.ts, per-post trajectory data
 ├── scripts/
 │   └── ipynb-to-fern-json.py  ← .ipynb → fern/components/notebooks/*.{json,ts}
-├── code-reference/            ← gitignored; populated by `fern docs md generate`
 └── versions/
     ├── latest.yml             ← authoring navigation tree
     └── latest/pages/          ← authoring MDX content
@@ -401,47 +400,6 @@ import notebook from "@/components/notebooks/1-the-basics";
 
 The converter (`fern/scripts/ipynb-to-fern-json.py`) **auto-strips the leading Colab badge cell** — `<NotebookViewer>` renders its own banner from the `colabUrl` prop. Don't manually re-add it.
 
-## Python API Reference (`libraries:`)
-
-`docs.yml` keeps a Fern-native `libraries:` block for the config package. Local generation uses `py2fern` through `make generate-fern-api-reference` and writes multiple gitignored trees under `fern/code-reference/`:
-
-- `data-designer/` for `data_designer.config`
-- `interface/` for `data_designer.interface`
-- `engine/seed-readers/`
-- `engine/processors/`
-- `engine/mcp/`
-- `engine/column-generators/`
-
-To populate locally:
-
-```bash
-make generate-fern-api-reference
-```
-
-This does not require Fern auth. Re-run when the upstream Python source changes. If you need to compare with Fern's native generator, use `make generate-fern-api-reference-native` with Fern auth.
-
-The generated trees are wired into `versions/latest.yml` under `Code Reference`:
-
-- `Config` contains prose pages plus `Config API` from `../code-reference/data-designer/data_designer/config`
-- `Interface` contains prose pages plus `Interface API` from `../code-reference/interface/data_designer/interface`
-- `Engine Extension API` contains prose pages plus the seed reader, processor, MCP runtime, and column generator API folders
-
-There is no `Topic Overviews` section. Prose reference pages live beside the generated folders under `fern/versions/latest/pages/code_reference/`.
-
-To add another generated package, update the `generate-fern-api-reference` target and add the matching `folder:` entry under the right `Code Reference` section. Only add a `libraries:` entry when Fern's native generator should know about that source:
-
-```yaml
-libraries:
-  data-designer:
-    input:
-      git: https://github.com/NVIDIA-NeMo/DataDesigner
-      subpath: packages/data-designer-config/src/data_designer/config
-    output: { path: ./code-reference/data-designer }
-    lang: python
-```
-
-Pyright needs a regular Python package (with `__init__.py`). The `data_designer` namespace itself is PEP 420 (no `__init__.py`), so always point at a sub-package one level deeper.
-
 ## MDX Gotchas (the ones that bit during migration)
 
 | Pattern | Problem | Fix |
@@ -467,14 +425,6 @@ fern docs dev       # localhost:3000 hot-reload preview
 
 `fern check` must pass before commit. The local broken-link checker has known false positives — it computes URLs from file paths instead of from slugified nav titles, so cross-section absolute links sometimes flag incorrectly. Spot-check by clicking through the dev server.
 
-To generate the API reference for local preview:
-
-```bash
-make generate-fern-api-reference    # py2fern; populates fern/code-reference/ (gitignored)
-```
-
-If the "Python API" sidebar folder is empty, you forgot this step.
-
 ## Commit & Preview
 
 ```bash

.github/workflows/docs-preview.yml

@@ -88,8 +88,7 @@ jobs:
           cd "$FERN_PREVIEW_ROOT"
           make check-fern-docs \
             DOCS_PYTHON="$GITHUB_WORKSPACE/.venv/bin/python" \
-            DOCS_JUPYTEXT="$GITHUB_WORKSPACE/.venv/bin/jupytext" \
-            DOCS_PY2FERN="$GITHUB_WORKSPACE/.venv/bin/py2fern"
+            DOCS_JUPYTEXT="$GITHUB_WORKSPACE/.venv/bin/jupytext"
 
       - name: Skip hosted previews for fork PRs
         if: github.event.pull_request.head.repo.full_name != github.repository

.gitignore

@@ -85,6 +85,14 @@ venv.bak/
 # Local scratch space
 .scratch/
 
+# Generated benchmark/report output
+/artifacts/
+/reports/
+/scripts/benchmarks/benchmark_async_scheduling.py
+/scripts/benchmarks/export_async_scheduling_perfetto.py
+/scripts/benchmarks/generate_async_scheduling_idle_report.py
+/scripts/benchmarks/run_async_scheduling_idle_regression.py
+
 docs/notebooks/
 docs/notebook_source/*.ipynb
 docs/notebook_source/*.csv
@@ -107,10 +115,6 @@ packages/data-designer/README.md
 # Claude worktrees
 .claude/worktrees/
 
-# Fern libraries output — generated by `fern docs md generate` from `libraries:` in fern/docs.yml.
-# Regenerate locally (no token needed) or in CI before publishing.
-fern/code-reference/
-
 # Fern notebook output - generated by `make generate-fern-notebooks`.
 fern/components/notebooks/*.json
 fern/components/notebooks/*.ts

CONTRIBUTING.md

@@ -79,7 +79,7 @@ Data Designer is migrating from MkDocs to Fern over several releases. Until the
 - Use `make serve-docs-locally` to preview the legacy MkDocs site.
 - Use `make check-fern-docs` to regenerate local Fern artifacts and validate the Fern site.
 - Fern release publishing snapshots versioned docs into the CI-managed `docs-website` branch automatically.
-- Do not commit generated Fern API reference or notebook artifacts.
+- Do not commit generated notebook artifacts.
 
 ---
 

Makefile

@@ -80,8 +80,6 @@ help:
 	@echo "  generate-colab-notebooks  - Generate Colab-compatible notebooks"
 	@echo "  generate-fern-notebooks   - Convert docs/notebook_source/*.py → fern/components/notebooks/{json,ts}"
 	@echo "  generate-fern-notebooks-with-outputs - Full pipeline: execute notebooks (needs API key), colabify, convert to Fern"
-	@echo "  generate-fern-api-reference - Generate local Fern API reference with py2fern"
-	@echo "  generate-fern-api-reference-native - Generate Fern API reference with Fern CLI (requires auth)"
 	@echo "  install-docs-deps        - Install docs and notebook dependencies"
 	@echo "  prepare-fern-release VERSION=X.Y.Z - Add or refresh Fern version files for release preview"
 	@echo "  check-fern-release-version VERSION=X.Y.Z - Verify Fern has a version entry for release publishing"
@@ -474,15 +472,6 @@ DOCS_PYTHON_VERSION ?= 3.13
 DOCS_PYTHON ?= .venv/bin/python
 DOCS_JUPYTEXT ?= .venv/bin/jupytext
 DOCS_MKDOCS ?= .venv/bin/mkdocs
-DOCS_PY2FERN ?= .venv/bin/py2fern
-FERN_API_REFERENCE_OUTPUT ?= fern/code-reference
-FERN_API_REFERENCE_CONFIG_OUTPUT ?= $(FERN_API_REFERENCE_OUTPUT)/data-designer
-FERN_API_REFERENCE_CONFIG_SOURCE ?= packages/data-designer-config/src/data_designer/config
-FERN_API_REFERENCE_INTERFACE_SOURCE ?= packages/data-designer/src/data_designer/interface
-FERN_API_REFERENCE_ENGINE_COLUMN_GENERATORS_SOURCE ?= packages/data-designer-engine/src/data_designer/engine/column_generators/generators/base.py
-FERN_API_REFERENCE_ENGINE_MCP_SOURCE ?= packages/data-designer-engine/src/data_designer/engine/mcp
-FERN_API_REFERENCE_ENGINE_PROCESSORS_SOURCE ?= packages/data-designer-engine/src/data_designer/engine/processing/processors
-FERN_API_REFERENCE_ENGINE_SEED_READERS_SOURCE ?= packages/data-designer-engine/src/data_designer/engine/resources/seed_reader.py
 FERN_VERSION ?= $(shell jq -r .version fern/fern.config.json)
 FERN ?= npx -y fern-api@$(FERN_VERSION)
 
@@ -501,21 +490,6 @@ serve-docs-locally:
 	@echo "📝 Building and serving docs (Python $(DOCS_PYTHON_VERSION))..."
 	$(DOCS_MKDOCS) serve --livereload
 
-generate-fern-api-reference:
-	@echo "📚 Generating Fern API reference with py2fern ($(DOCS_PY2FERN))..."
-	@rm -rf $(FERN_API_REFERENCE_OUTPUT)
-	$(DOCS_PY2FERN) write $(FERN_API_REFERENCE_CONFIG_SOURCE) --module data_designer.config --output $(FERN_API_REFERENCE_CONFIG_OUTPUT) --clean
-	$(DOCS_PY2FERN) write $(FERN_API_REFERENCE_INTERFACE_SOURCE) --module data_designer.interface --output $(FERN_API_REFERENCE_OUTPUT)/interface --clean
-	$(DOCS_PY2FERN) write $(FERN_API_REFERENCE_ENGINE_COLUMN_GENERATORS_SOURCE) --module data_designer.engine.column_generators.generators.base --output $(FERN_API_REFERENCE_OUTPUT)/engine/column-generators --clean
-	$(DOCS_PY2FERN) write $(FERN_API_REFERENCE_ENGINE_MCP_SOURCE) --module data_designer.engine.mcp --output $(FERN_API_REFERENCE_OUTPUT)/engine/mcp --clean
-	$(DOCS_PY2FERN) write $(FERN_API_REFERENCE_ENGINE_PROCESSORS_SOURCE) --module data_designer.engine.processing.processors --output $(FERN_API_REFERENCE_OUTPUT)/engine/processors --clean
-	$(DOCS_PY2FERN) write $(FERN_API_REFERENCE_ENGINE_SEED_READERS_SOURCE) --module data_designer.engine.resources.seed_reader --output $(FERN_API_REFERENCE_OUTPUT)/engine/seed-readers --clean
-	$(DOCS_PYTHON) fern/scripts/normalize-py2fern-indexes.py $(FERN_API_REFERENCE_OUTPUT)
-
-generate-fern-api-reference-native:
-	@echo "📚 Generating Fern API reference with Fern CLI..."
-	cd fern && $(FERN) docs md generate
-
 prepare-fern-release:
 ifndef VERSION
 	$(error VERSION is required, e.g. make prepare-fern-release VERSION=0.5.10)
@@ -528,7 +502,7 @@ ifndef VERSION
 endif
 	$(DOCS_PYTHON) fern/scripts/fern-release-version.py check --version $(VERSION) $(if $(REQUIRE_LATEST),--require-latest-matches-release,)
 
-prepare-fern-docs: generate-fern-api-reference generate-fern-notebooks
+prepare-fern-docs: generate-fern-notebooks
 	@echo "✅ Fern local artifacts ready"
 
 check-fern-docs: prepare-fern-docs
@@ -759,7 +733,7 @@ clean-test-coverage:
         coverage coverage-config coverage-engine coverage-interface \
         format format-check format-check-config format-check-engine format-check-interface \
         format-config format-engine format-interface \
-        generate-colab-notebooks generate-fern-api-reference generate-fern-api-reference-native generate-fern-notebooks generate-fern-notebooks-with-outputs help \
+        generate-colab-notebooks generate-fern-notebooks generate-fern-notebooks-with-outputs help \
         install install-dev install-dev-notebooks install-dev-recipes install-docs-deps \
         lint lint-config lint-engine lint-fix lint-fix-config lint-fix-engine lint-fix-interface lint-interface \
         perf-import perf-import-runtime prepare-fern-docs prepare-fern-release publish serve-docs-locally serve-fern-docs-locally show-versions \

architecture/dataset-builders.md

@@ -35,7 +35,7 @@ Preparation (`_prepare_async_run`):
 4. Constructs `CompletionTracker`, `RowGroupBufferManager`, `AsyncTaskScheduler`
 5. Hooks `ProcessorRunner` for pre-batch and post-batch stages
 
-`AsyncTaskScheduler` runs on a dedicated async loop with frontier-driven dispatch, semaphore-based capacity limits, salvage rounds for failed tasks, and order-dependent locks for columns that must execute sequentially. Ready frontier tasks are admitted through a virtual-time fair queue so one hot column or model-backed generator cannot consume the whole submission window before peer work gets a turn.
+`AsyncTaskScheduler` runs on a dedicated async loop with frontier-driven dispatch, task-admission leases, salvage rounds for failed tasks, and order-dependent locks for columns that must execute sequentially. Ready frontier tasks enter `FairTaskQueue`, are selected through virtual-time ordering, and are committed only after `TaskAdmissionController` acquires the required scheduler resources.
 
 ### Execution Graph
 
@@ -121,19 +121,24 @@ DatasetBuilder.build()
   → _prepare_async_run()
       → ExecutionGraph.create()
       → CompletionTracker.with_graph()
-      → AsyncTaskScheduler(semaphores, salvage_rounds)
+      → AsyncTaskScheduler(task admission, fair queue, salvage_rounds)
   → scheduler.run()
-      → for each row group, fairly admit ready tasks from frontier
+      → admit row groups under the configured row-group cap
+      → fairly admit ready tasks from the frontier through task admission
       → tasks execute generators, update CompletionTracker
       → checkpoints via RowGroupBufferManager
   → collect TaskTraces, emit telemetry
 ```
 
+Row-group admission is fixed by default in the dataset-builder path: the configured row-group concurrency is the hard in-flight cap. The scheduler also has an internal adaptive row-group mode for direct use that only raises a soft target up to that cap; it is additive ramp-up, not AIMD shrink/recovery behavior.
+
+When request admission is available, async scheduling may use request-pressure snapshots as a read-only advisory during fair-queue selection. A request-pressured task can be skipped for an eligible peer without mutating request-admission state; provider/model/domain request limits remain owned by request admission.
+
 ## Design Decisions
 
 - **Dual execution engines behind one API.** The sequential engine is simpler and easier to debug; the async engine adds row-group parallelism for throughput. Users switch via an environment variable without changing their code.
 - **DAG-driven ordering** ensures columns with dependencies (e.g., a judge column that depends on a text column) are generated in the correct order, regardless of the order they appear in the config.
-- **Fair async admission** keeps the scheduler flowing across ready columns and model groups. Global semaphores still bound memory/coroutine growth, while per-group virtual-time queues prevent a large ready frontier from degenerating into a column-by-column wave. LLM admission caps are peer-sensitive: a solo model group can fill available global capacity, but once another scheduling group has queued work the saturated group yields until peers get admission slots or admitted tasks complete.
+- **Fair async admission with bounded borrow by default** keeps the scheduler flowing across ready columns and model groups. `FairTaskQueue.select_next(...)` chooses eligible ready work, `TaskAdmissionController` leases scheduler resources before spawn, and `FairTaskQueue.commit(...)` removes the selected task only after admission succeeds. The default `BoundedBorrowTaskAdmissionPolicyConfig` computes a strict per-group share, lets solo groups borrow only up to a capacity-derived reserve, and makes borrowed groups yield when eligible peer pressure appears. Passing `bounded_borrow=None` selects strict-fair admission for tests and benchmark comparisons. Per-group virtual-time ordering prevents a large ready frontier from degenerating into a column-by-column wave, and scheduler-resource accounting remains separate from provider/model request admission.
 - **Salvage rounds in async mode** retry failed tasks after all other tasks in a round complete, improving resilience against transient LLM failures without blocking the entire generation.
 - **Unified DAG construction.** `topologically_sort_column_configs` (in `execution_graph.py`) determines column ordering using Kahn's algorithm; the runtime `ExecutionGraph` adds strategy-aware dependency tracking for the async scheduler.