docs: Add API doc links to Geneva docs (#203)

dantasse · web-flow · commit 8ed60a365d1b · 2026-04-09T15:24:21.000-04:00
diff --git a/docs/geneva/index.mdx b/docs/geneva/index.mdx
@@ -56,3 +56,10 @@ Visit the following pages to learn more about featuring engineering in LanceDB E
 - **UDFs**: [Using UDFs](/geneva/udfs/udfs) · [Blob helpers](/geneva/udfs/blobs/) · [Error handling](/geneva/udfs/error_handling) · [Advanced configuration](/geneva/udfs/advanced-configuration) 
 - **Jobs**: [Backfilling](/geneva/jobs/backfilling/) · [Startup optimizations](/geneva/jobs/startup/) · [Materialized views](/geneva/jobs/materialized-views/) · [Execution contexts](/geneva/jobs/contexts/) · [Geneva console](/geneva/jobs/console) · [Performance](/geneva/jobs/performance/)
 - **Deployment**: [Deployment overview](/geneva/deployment/) · [Helm deployment](/geneva/deployment/helm/) · [Troubleshooting](/geneva/deployment/troubleshooting/)
+
+## API Reference
+
+- [`geneva.connect()`](https://lancedb.github.io/geneva/api/) — connect to a Geneva database
+- [Connection](https://lancedb.github.io/geneva/api/connection/) — manage tables, views, jobs, clusters, and manifests
+- [Table](https://lancedb.github.io/geneva/api/table/) — add columns, backfill, search, and manage table data
+- [UDF](https://lancedb.github.io/geneva/api/udf/) — define user-defined functions for feature computation
diff --git a/docs/geneva/jobs/backfilling.mdx b/docs/geneva/jobs/backfilling.mdx
@@ -125,3 +125,5 @@ tbl.backfill("embedding", where="content is not null and embeddding is not null"
 Reference:
 * [`backfill` API](https://lancedb.github.io/geneva/api/table/#geneva.table.Table.backfill)
 * [`backfill_async` API](https://lancedb.github.io/geneva/api/table/#geneva.table.Table.backfill_async)
+* [`plan_backfill` API](https://lancedb.github.io/geneva/api/table/#geneva.table.Table.plan_backfill)
+* [UDF](https://lancedb.github.io/geneva/api/udf/) — `@udf` decorator and UDF configuration options including `checkpoint_size`
diff --git a/docs/geneva/jobs/conflicts.mdx b/docs/geneva/jobs/conflicts.mdx
@@ -117,3 +117,7 @@ If using LanceDB Enterprise which has auto-compaction enabled, consider disablin
 
 - [Backfilling](/geneva/jobs/backfilling) - Triggering and configuring backfill operations
 - [Advanced Configuration](/geneva/udfs/advanced-configuration) - Environment variables for retry behavior
+
+## API Reference
+
+- [Table](https://lancedb.github.io/geneva/api/table/) — `backfill()`, `add()`, `merge_insert()`, and other table mutation methods
diff --git a/docs/geneva/jobs/console.mdx b/docs/geneva/jobs/console.mdx
@@ -44,3 +44,9 @@ See the Geneva clusters that you have defined to run jobs. Because clusters can
 
 ### Manifests
 See the Manifests you've defined and what packages/dependencies they contain. As with clusters, manifests are reusable, so it's easy to start a new job with the same dependencies as an old one by just specifying the manifest name.
+
+## API Reference
+
+- [Connection](https://lancedb.github.io/geneva/api/connection/) — `get_job()`, `list_jobs()`, `list_clusters()`, `list_manifests()`
+- [Cluster](https://lancedb.github.io/geneva/api/cluster/) — `GenevaCluster` and cluster configuration classes
+- [Manifest](https://lancedb.github.io/geneva/api/manifest/) — `GenevaManifest` and manifest builder classes
diff --git a/docs/geneva/jobs/contexts.mdx b/docs/geneva/jobs/contexts.mdx
@@ -296,3 +296,9 @@ ctx.__enter__()
 ctx.__exit__(None,None,None)
 ```
 </CodeGroup>
+
+## API Reference
+
+- [Connection](https://lancedb.github.io/geneva/api/connection/) — `context()`, `local_ray_context()`, `list_clusters()`, `list_manifests()`
+- [Cluster](https://lancedb.github.io/geneva/api/cluster/) — `KubeRayClusterBuilder`, `LocalRayClusterBuilder`, `ExternalRayClusterBuilder`, and worker configuration classes
+- [Manifest](https://lancedb.github.io/geneva/api/manifest/) — `PipManifestBuilder`, `CondaManifestBuilder`, `SiteManifestBuilder`
diff --git a/docs/geneva/jobs/index.mdx b/docs/geneva/jobs/index.mdx
@@ -41,3 +41,10 @@ Set up and access the Geneva Console for monitoring and managing Geneva jobs, cl
 4. **Monitor performance** and optimize based on usage patterns
 
 For detailed information about each job type and execution context, explore the documentation in this section.
+
+## API Reference
+
+- [Table](https://lancedb.github.io/geneva/api/table/) — `backfill()`, `backfill_async()`, `refresh()`, `plan_backfill()`, `plan_refresh()`, and other job-triggering methods
+- [Connection](https://lancedb.github.io/geneva/api/connection/) — `get_job()`, `list_jobs()`, `context()`, and `local_ray_context()`
+- [Cluster](https://lancedb.github.io/geneva/api/cluster/) — configure KubeRay and local Ray execution backends
+- [Manifest](https://lancedb.github.io/geneva/api/manifest/) — package Python environments for remote workers
diff --git a/docs/geneva/jobs/lifecycle.mdx b/docs/geneva/jobs/lifecycle.mdx
@@ -119,3 +119,8 @@ By default, checkpoints are stored in a `_ckp/` subdirectory inside the table's
 ### Resuming Failed Jobs
 
 To resume a failed job, simply re-run the same backfill or refresh command. The job will automatically detect existing checkpoints, skip already-processed fragments, and continue from where it left off.
+
+## API Reference
+
+- [Table](https://lancedb.github.io/geneva/api/table/) — `backfill()`, `backfill_async()`, `refresh()`, `JobFuture`
+- [Connection](https://lancedb.github.io/geneva/api/connection/) — `get_job()`, `list_jobs()`
diff --git a/docs/geneva/jobs/materialized-views.mdx b/docs/geneva/jobs/materialized-views.mdx
@@ -106,3 +106,9 @@ No. The UDF does not but any UDF calculated values in the original table come to
 ### On MV refresh, do we force materialization of UDFs cols on the source table?
 
 No. They are managed at the source table only. If it is null the null values are propagated. Future options may force materialization/backfill "recursively".
+
+## API Reference
+
+- [Connection](https://lancedb.github.io/geneva/api/connection/) — `create_materialized_view()`, `create_view()`
+- [Table](https://lancedb.github.io/geneva/api/table/) — `refresh()`, `plan_refresh()`
+- [Query](https://lancedb.github.io/geneva/api/query/) — `create_materialized_view()` on query builder
diff --git a/docs/geneva/jobs/performance.mdx b/docs/geneva/jobs/performance.mdx
@@ -158,3 +158,9 @@ Certain jobs that take a small dataset and expand it may appear as if the writer
 An example is a table that contains a list of URLs pointing to large media files. This list is relatively small (&lt; 100MB) and can fit into a single fragment. A UDF that downloads will fetch all the data and then attempt to write all of it out through the single writer. This single writer can then be responsible for serially writing out 500+GB of data to a single file!
 
 To mitigate this, you can load your initial table so that there will be multiple fragments. Each fragment with new outputs can be written in parallel with higher write throughput.
+
+## API Reference
+
+- [Cluster](https://lancedb.github.io/geneva/api/cluster/) — `KubeRayClusterBuilder`, `CpuWorkerBuilder`, `GpuWorkerBuilder` — configure CPU/GPU/memory resources
+- [Table](https://lancedb.github.io/geneva/api/table/) — `backfill()`, `compact_files()`, `optimize()`
+- [UDF](https://lancedb.github.io/geneva/api/udf/) — `@udf` decorator options: `num_cpus`, `num_gpus`, `memory`, `batch_size`
diff --git a/docs/geneva/jobs/troubleshooting.mdx b/docs/geneva/jobs/troubleshooting.mdx
@@ -180,3 +180,10 @@ Refreshing a materialized view runs admission control for each UDF in the view.
 - **Versions** – Same Ray version on client and cluster; same Python minor (e.g. 3.10.x) on both. See [Troubleshooting Geneva Deployments](/geneva/deployment/troubleshooting#confirming-dependency-versions).
 - **Remote execution** – Use `ray.available_resources()` and a simple `@ray.remote` task to confirm the cluster is reachable and has the expected CPUs/GPUs/memory.
 - **Permissions** – Run a minimal remote task that `import geneva` and touches the same bucket/path as your job to surface permission issues early.
+
+## API Reference
+
+- [UDF](https://lancedb.github.io/geneva/api/udf/) — `@udf` decorator options: `num_gpus`, `num_cpus`, `memory`
+- [Cluster](https://lancedb.github.io/geneva/api/cluster/) — `KubeRayClusterBuilder`, `CpuWorkerBuilder`, `GpuWorkerBuilder`
+- [Table](https://lancedb.github.io/geneva/api/table/) — `backfill()`, `refresh()`
+- [Error Handling](https://lancedb.github.io/geneva/api/error_handling/) — `FatalWorkerOOMError`, `FatalWorkerCrashError`, and other worker error types
diff --git a/docs/geneva/overview/index.mdx b/docs/geneva/overview/index.mdx
@@ -112,3 +112,10 @@ Managing this at scale means dealing with distributed systems, scheduling, and m
 
 In short, the biggest challenge in modern feature engineering is not just coming up with good features. It is also about building infrastructure that can handle complex, multimodal operations. The goal is to deliver fresh, high-quality features quickly and reliably at massive scale. 
 
+## API Reference
+
+- [`geneva.connect()`](https://lancedb.github.io/geneva/api/) — connect to a Geneva database
+- [Connection](https://lancedb.github.io/geneva/api/connection/) — manage tables, views, jobs, clusters, and manifests
+- [Table](https://lancedb.github.io/geneva/api/table/) — add columns, backfill, search, and manage table data
+- [UDF](https://lancedb.github.io/geneva/api/udf/) — define transforms as user-defined functions
+
diff --git a/docs/geneva/udfs/batch-udtfs.mdx b/docs/geneva/udfs/batch-udtfs.mdx
@@ -303,3 +303,4 @@ class GroupStats:
 
 Reference:
 * [`create_udtf_view` API](https://lancedb.github.io/geneva/api/connection/#geneva.db.Connection.create_udtf_view)
+* [UDTF](https://lancedb.github.io/geneva/api/udtf/) — full `@udtf` / `@batch_udtf` decorator reference including `output_schema`, `partition_by`, `num_gpus`, and `on_error`
diff --git a/docs/geneva/udfs/blobs.mdx b/docs/geneva/udfs/blobs.mdx
@@ -66,4 +66,9 @@ def batch_to_blob(batch: pa.RecordBatch) -> pa.Array:
         blobs.append(blob_data)
     return pa.array(blobs, type=pa.large_binary())
 ```
-</CodeGroup>
+</CodeGroup>
+
+## API Reference
+
+- [UDF](https://lancedb.github.io/geneva/api/udf/) — `@udf` decorator for defining blob-processing functions
+- [Table](https://lancedb.github.io/geneva/api/table/) — `add_columns()`, `backfill()`
diff --git a/docs/geneva/udfs/error_handling.mdx b/docs/geneva/udfs/error_handling.mdx
@@ -212,3 +212,8 @@ Retry(ValueError, match=r"[invalid")
 # But this will work:
 Retry(ValueError, match=r"rate.?limit")
 ```
+
+## API Reference
+
+- [Error Handling](https://lancedb.github.io/geneva/api/error_handling/) — `Retry`, `Skip`, `Fail`, `FatalWorkerOOMError`, `FatalWorkerCrashError`, and all exception matcher classes
+- [UDF](https://lancedb.github.io/geneva/api/udf/) — `@udf` decorator `on_error` parameter
diff --git a/docs/geneva/udfs/index.mdx b/docs/geneva/udfs/index.mdx
@@ -95,3 +95,10 @@ Batch UDTFs read from a source table (or partition) and **produce output with an
 **Use cases**: Deduplication, clustering, aggregation, cross-row joins.
 
 See [Batch UDTFs](/geneva/udfs/batch-udtfs) for the full guide.
+
+## API Reference
+
+- [UDF](https://lancedb.github.io/geneva/api/udf/) — `@udf` decorator and `UDF` class
+- [UDTF](https://lancedb.github.io/geneva/api/udtf/) — `@udtf`, `@scalar_udtf`, `@batch_udtf` decorators and `UDTF`/`ScalarUDTF` classes
+- [Table](https://lancedb.github.io/geneva/api/table/) — `add_columns()`, `backfill()`
+- [Connection](https://lancedb.github.io/geneva/api/connection/) — `create_udtf_view()`, `create_scalar_udtf_view()`
diff --git a/docs/geneva/udfs/providers/gemini.mdx b/docs/geneva/udfs/providers/gemini.mdx
@@ -84,3 +84,8 @@ table.add_columns({
     ),
 })
 ```
+
+## API Reference
+
+- [Gemini](https://lancedb.github.io/geneva/api/gemini/) — `gemini_udf()` parameters: `column`, `prompt`, `model`, `mime_type`, and more
+- [Embeddings](https://lancedb.github.io/geneva/api/embeddings/) — `gemini_embedding_udf()` parameters: `column`, `model`, `task_type`, `dimensionality`, `normalize`
diff --git a/docs/geneva/udfs/providers/index.mdx b/docs/geneva/udfs/providers/index.mdx
@@ -74,3 +74,9 @@ All built-in UDFs share these capabilities:
 - [Working with UDFs](/geneva/udfs/index) — Write custom scalar, batched, and stateful UDFs
 - [Error handling](/geneva/udfs/error_handling) — Fine-grained retry and skip policies
 - [Working with blobs](/geneva/udfs/blobs) — Process binary data (images, audio, video)
+
+## API Reference
+
+- [Embeddings](https://lancedb.github.io/geneva/api/embeddings/) — `sentence_transformer_udf()`, `gemini_embedding_udf()`, `openai_embedding_udf()`
+- [Gemini](https://lancedb.github.io/geneva/api/gemini/) — `gemini_udf()`
+- [OpenAI](https://lancedb.github.io/geneva/api/openai/) — `openai_udf()`, `openai_embedding_udf()`
diff --git a/docs/geneva/udfs/providers/openai.mdx b/docs/geneva/udfs/providers/openai.mdx
@@ -77,3 +77,7 @@ table.add_columns({
     ),
 })
 ```
+
+## API Reference
+
+- [OpenAI](https://lancedb.github.io/geneva/api/openai/) — `openai_udf()` and `openai_embedding_udf()` — all parameters including `column`, `prompt`, `model`, `mime_type`, `dimensions`, and `normalize`
diff --git a/docs/geneva/udfs/providers/sentence-transformers.mdx b/docs/geneva/udfs/providers/sentence-transformers.mdx
@@ -58,3 +58,7 @@ Setting `num_gpus` to a fractional value (e.g., `0.5`) tells the
 to co-locate multiple workers on the same physical GPU. For example, two UDFs with
 `num_gpus=0.5` will be scheduled on a single GPU. Note that Ray does not enforce GPU memory
 limits — it is your responsibility to ensure the combined models fit in GPU memory.
+
+## API Reference
+
+- [Embeddings](https://lancedb.github.io/geneva/api/embeddings/) — `sentence_transformer_udf()` — all parameters including `column`, `model`, `num_gpus`, `normalize`, and `batch_size`
diff --git a/docs/geneva/udfs/udfs.mdx b/docs/geneva/udfs/udfs.mdx
@@ -285,3 +285,5 @@ table.backfill("area", where="area is null")
 
 Reference:
 * [`alter_columns` API](https://lancedb.github.io/geneva/api/table/#geneva.table.Table.alter_columns)
+* [`add_columns` API](https://lancedb.github.io/geneva/api/table/#geneva.table.Table.add_columns)
+* [UDF](https://lancedb.github.io/geneva/api/udf/) — full `@udf` decorator reference including `data_type`, `num_gpus`, `batch_size`, and other options
