From 0ebd7c5f5c8ae0afe3266cd453bf24e472d812db Mon Sep 17 00:00:00 2001 From: Kurt Heiss Date: Wed, 1 Jul 2026 11:39:21 -0700 Subject: [PATCH] =?UTF-8?q?docs:=20Phase=201=20content=20IA=20=E2=80=94=20?= =?UTF-8?q?frontmatter,=20Start=20here,=20CLI=20quickstart?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add YAML frontmatter to four anchor pages for agent-readable metadata (content_type, audience, journey_stage, prerequisites). Promote getting started to Start here with a typed onboarding table, add an external documentation map for GitHub-only nav targets, and replace the snippet-only CLI quickstart with a self-contained tutorial. --- docs/docs/extraction/deployment-options.md | 26 ++++++++- .../extraction/external-documentation-map.md | 33 +++++++++++ docs/docs/extraction/getting-started-about.md | 37 ++++++++---- docs/docs/extraction/multimodal-extraction.md | 18 ++++++ docs/docs/extraction/overview.md | 17 ++++++ .../extraction/workflow-document-ingestion.md | 26 +++++++++ .../reference/retriever-cli-quickstart.md | 57 ++++++++++++++++++- docs/mkdocs.yml | 3 +- 8 files changed, 203 insertions(+), 14 deletions(-) create mode 100644 docs/docs/extraction/external-documentation-map.md diff --git a/docs/docs/extraction/deployment-options.md b/docs/docs/extraction/deployment-options.md index b57404685..8ecce95d9 100644 --- a/docs/docs/extraction/deployment-options.md +++ b/docs/docs/extraction/deployment-options.md @@ -1,3 +1,25 @@ +--- +title: "Deployment options" +content_type: how-to +audience: intermediate +journey_stage: try +product: nemo-retriever-library +features: + - deployment +technologies: + - python + - helm-kubernetes + - docker + - nim-hosted + - nim-self-hosted +prerequisites: + - extraction/overview.md + - extraction/prerequisites-support-matrix.md +duration_minutes: 15 +surface: published-docs +status: published +--- + # Deployment options Use this page to compare how you run NeMo Retriever — including when to use [NVIDIA-hosted NIMs](https://build.nvidia.com/) versus self-hosting on your own infrastructure. @@ -19,7 +41,7 @@ Build and run the NeMo Retriever service image with the [Docker service image gu 1. [Pre-Requisites & Support Matrix](prerequisites-support-matrix.md) 2. **NeMo Retriever Helm chart (supported):** [Deploy (Helm chart)](https://github.com/NVIDIA/NeMo-Retriever/blob/main/nemo_retriever/helm/README.md) — sources in [`nemo_retriever/helm`](https://github.com/NVIDIA/NeMo-Retriever/tree/main/nemo_retriever/helm) on GitHub -3. **Published Library Helm charts (supported):** cluster install and upgrade procedures are covered in [About getting started](getting-started-about.md) — use alongside the NeMo Retriever chart README for your release +3. **Published Library Helm charts (supported):** cluster install and upgrade procedures are covered in [Start here](getting-started-about.md) — use alongside the NeMo Retriever chart README for your release 4. [Environment variables](environment-config.md) and [Troubleshoot](troubleshoot.md) as needed **Core NIMs for the default extraction pipeline:** `page_elements`, `table_structure`, `ocr`, and `vlm_embed` (`llama-nemotron-embed-vl-1b-v2:1.12.0`). These four are auto-wired into the retriever service. **Nemotron Parse**, **Nemotron 3 Nano Omni**, the **VL reranker**, and **Parakeet ASR** are optional and not auto-wired. For a minimal GPU footprint, disable optional keys you do not need (refer to [Recommended minimal install](https://github.com/NVIDIA/NeMo-Retriever/blob/main/nemo_retriever/helm/README.md#recommended-minimal-install-2605)). Refer to [Pre-Requisites & Support Matrix — Default Helm NIMs](prerequisites-support-matrix.md#default-helm-nims). @@ -80,6 +102,6 @@ For offline image captioning, deploy the in-cluster [Nemotron 3 Nano Omni](prere **Related** - [Deploy (Helm chart)](https://github.com/NVIDIA/NeMo-Retriever/blob/main/nemo_retriever/helm/README.md) ([`nemo_retriever/helm`](https://github.com/NVIDIA/NeMo-Retriever/tree/main/nemo_retriever/helm) on GitHub) — [air-gapped deployment](https://github.com/NVIDIA/NeMo-Retriever/blob/main/nemo_retriever/helm/README.md#air-gapped-deployment) -- [About getting started](getting-started-about.md) (prerequisites through first deployment) +- [Start here](getting-started-about.md) (prerequisites through first deployment) - [Pre-Requisites & Support Matrix](prerequisites-support-matrix.md) - [Audio and video](audio-video.md) diff --git a/docs/docs/extraction/external-documentation-map.md b/docs/docs/extraction/external-documentation-map.md new file mode 100644 index 000000000..26ea5d272 --- /dev/null +++ b/docs/docs/extraction/external-documentation-map.md @@ -0,0 +1,33 @@ +--- +title: "External documentation map" +content_type: reference +audience: intermediate +journey_stage: try +product: nemo-retriever-library +features: + - deployment + - api +technologies: + - helm-kubernetes + - docker + - cli +prerequisites: + - extraction/getting-started-about.md +surface: published-docs +status: published +--- + +# External documentation map + +Several procedures in the NeMo Retriever documentation navigation point to the [NeMo-Retriever](https://github.com/NVIDIA/NeMo-Retriever) GitHub repository. Use this page to find the canonical location for each topic. + +| Topic | Canonical location | Surface | Owns | +|-------|-------------------|---------|------| +| Library quickstart and package source | [nemo_retriever/](https://github.com/NVIDIA/NeMo-Retriever/tree/main/nemo_retriever) | `github-readme` | Install, package layout, first code examples | +| Helm chart (Kubernetes) | [nemo_retriever/helm/README.md](https://github.com/NVIDIA/NeMo-Retriever/blob/main/nemo_retriever/helm/README.md) | `github-readme` | Chart values, NIM enablement, deploy procedures | +| OpenShift deployment | [nemo_retriever/helm/openshift.md](https://github.com/NVIDIA/NeMo-Retriever/blob/main/nemo_retriever/helm/openshift.md) | `github-readme` | OpenShift-specific Helm configuration | +| Docker service image | [nemo_retriever/docker.md](https://github.com/NVIDIA/NeMo-Retriever/blob/main/nemo_retriever/docker.md) | `github-readme` | Service container build and run | +| CLI reference | [nemo_retriever/docs/cli](https://github.com/NVIDIA/NeMo-Retriever/tree/main/nemo_retriever/docs/cli) | `github-cli-docs` | All `retriever` subcommands and flags | +| Graph and custom stages | [nemo_retriever graph README](https://github.com/NVIDIA/NeMo-Retriever/tree/main/nemo_retriever/src/nemo_retriever/graph#nemo-retriever-graph) | `github-readme` | Custom pipeline stages and UDF-style operations | + +For published extraction guides that stay on docs.nvidia.com, start from [Start here](getting-started-about.md). diff --git a/docs/docs/extraction/getting-started-about.md b/docs/docs/extraction/getting-started-about.md index 8f3a8ee69..b82f86025 100644 --- a/docs/docs/extraction/getting-started-about.md +++ b/docs/docs/extraction/getting-started-about.md @@ -1,14 +1,31 @@ -# About getting started +--- +title: "Start here: get access, deploy, and run examples" +content_type: tutorial +audience: beginner +journey_stage: try +product: nemo-retriever-library +features: [] +technologies: [] +prerequisites: + - extraction/overview.md +duration_minutes: 45 +surface: published-docs +status: published +--- -This section walks you from **access and prerequisites** through **first deployment** and **hands-on notebooks**. +# Start here: get access, deploy, and run examples -Typical order: +This page is the recommended entry path for NeMo Retriever Library (NRL). +Complete the steps in order, or jump to a step that matches your goal. -1. [Get your API key](api-keys.md) (NGC / API access as required by your workflow). -2. Confirm the [Pre-Requisites & Support Matrix](prerequisites-support-matrix.md) for your OS, GPU, and software stack. -3. Deploy using one of: - - [Deployment options](deployment-options.md) for library, hosted NIMs, and Kubernetes paths - - **Supported:** [Helm chart](https://github.com/NVIDIA/NeMo-Retriever/blob/main/nemo_retriever/helm/README.md) for Kubernetes cluster install and upgrade -4. Explore [Jupyter Notebooks](starter-kits.md) for end-to-end examples. +| Step | Type | Time | Action | +|------|------|------|--------| +| 1 | How-to | ~10 min | [Get API keys](api-keys.md) | +| 2 | Reference | ~15 min | [Prerequisites and support matrix](prerequisites-support-matrix.md) | +| 3 | How-to | ~15 min | [Choose a deployment option](deployment-options.md) | +| 4 | How-to | ~25 min | [Ingest documents into a searchable collection](workflow-document-ingestion.md) | +| 5 | Tutorial | ~20 min | [Run the retriever CLI quickstart](../reference/retriever-cli-quickstart.md) | -If you are new to the product, read [What is NeMo Retriever Library?](overview.md) and [Concepts](concepts.md) under **Introduction** first. +New to the product? Read [What is NeMo Retriever Library?](overview.md) and [Key concepts](concepts.md) under **Introduction** first. + +Helm, Docker, the full CLI reference, and graph customization live in the [NeMo-Retriever GitHub repository](external-documentation-map.md). For hands-on notebooks after your first ingest, refer to [Starter kits](starter-kits.md). diff --git a/docs/docs/extraction/multimodal-extraction.md b/docs/docs/extraction/multimodal-extraction.md index f44aee36c..99300a755 100644 --- a/docs/docs/extraction/multimodal-extraction.md +++ b/docs/docs/extraction/multimodal-extraction.md @@ -1,3 +1,21 @@ +--- +title: "Multimodal extraction" +content_type: explanation +audience: intermediate +journey_stage: build +product: nemo-retriever-library +features: + - extraction +technologies: + - python +prerequisites: + - extraction/overview.md + - extraction/concepts.md +duration_minutes: 20 +surface: published-docs +status: published +--- + # Multimodal extraction NeMo Retriever Library classifies and extracts text, tables, charts, infographics, and related layout from documents and media. This page groups formats, extraction modes, structured outputs, and throughput guidance in one place. Use the table of contents to jump to a topic. diff --git a/docs/docs/extraction/overview.md b/docs/docs/extraction/overview.md index f4837ae63..98a4c7975 100644 --- a/docs/docs/extraction/overview.md +++ b/docs/docs/extraction/overview.md @@ -1,3 +1,20 @@ +--- +title: "What is NeMo Retriever Library?" +content_type: explanation +audience: beginner +journey_stage: discover +product: nemo-retriever-library +features: + - ingestion + - extraction +technologies: + - python +prerequisites: [] +duration_minutes: 8 +surface: published-docs +status: published +--- + # What is NeMo Retriever Library? NVIDIA NeMo Retriever Library (NRL) is a high retrieval accuracy, performant, and scalable framework for content and metadata extraction from various media types (PDFs, HTML, Word docs, Powerpoint, audio, video, and image files). It supports both NVIDIA NIM microservices and a range of models to find, contextualize, and extract text, tables, charts, infographics, and transcripts for use in downstream generative and retrieval-augmented applications. diff --git a/docs/docs/extraction/workflow-document-ingestion.md b/docs/docs/extraction/workflow-document-ingestion.md index cba0164d6..44c7092b0 100644 --- a/docs/docs/extraction/workflow-document-ingestion.md +++ b/docs/docs/extraction/workflow-document-ingestion.md @@ -1,3 +1,29 @@ +--- +title: "Ingest documents into a searchable collection" +content_type: how-to +audience: intermediate +journey_stage: build +product: nemo-retriever-library +features: + - ingestion + - extraction + - embedding + - vector-storage +technologies: + - python + - cli + - lancedb + - ray +prerequisites: + - extraction/overview.md + - extraction/concepts.md + - extraction/api-keys.md + - extraction/deployment-options.md +duration_minutes: 25 +surface: published-docs +status: published +--- + # Workflow: Ingest documents into a searchable VDB collection This page covers extracting content from documents and turning that content into a searchable vector collection in one place so you can scroll and search (for example with Ctrl+F) instead of jumping across multiple short workflow stubs. diff --git a/docs/docs/reference/retriever-cli-quickstart.md b/docs/docs/reference/retriever-cli-quickstart.md index 7d730f980..d747e2725 100644 --- a/docs/docs/reference/retriever-cli-quickstart.md +++ b/docs/docs/reference/retriever-cli-quickstart.md @@ -1 +1,56 @@ ---8<-- "../nemo_retriever/docs/cli/README.md:quickstart" +--- +title: "Quickstart: retriever CLI" +content_type: tutorial +audience: beginner +journey_stage: try +product: nemo-retriever-library +features: + - ingestion +technologies: + - cli +prerequisites: + - extraction/api-keys.md + - extraction/getting-started-about.md +duration_minutes: 20 +surface: published-docs +status: published +--- + +# Quickstart: retriever CLI + +Use the `retriever` CLI to ingest documents locally and query the resulting LanceDB index. This quickstart covers a minimal local PDF ingest and query. For all flags and subcommands, refer to the [CLI reference on GitHub](https://github.com/NVIDIA/NeMo-Retriever/tree/main/nemo_retriever/docs/cli). + +## Before you begin + +1. Install the NeMo Retriever package (for example `pip install nemo-retriever` from [PyPI](https://pypi.org/project/nemo-retriever/) or an editable install from the [nemo_retriever](https://github.com/NVIDIA/NeMo-Retriever/tree/main/nemo_retriever) tree). +2. Confirm your environment against the [Prerequisites and support matrix](../extraction/prerequisites-support-matrix.md). +3. If you call [NVIDIA-hosted NIMs](https://build.nvidia.com/), set `NVIDIA_API_KEY` as described in [Get API keys](../extraction/api-keys.md). + +## Ingest a PDF locally + +From a checkout of NeMo-Retriever (or any directory that contains your PDF), run: + +```bash +retriever ingest ./data/multimodal_test.pdf \ + --method pdfium \ + --extract-text --extract-tables --extract-charts \ + --use-table-structure \ + --embed-model-name nvidia/llama-nemotron-embed-1b-v2 +``` + +By default, local ingest writes to `lancedb/nemo-retriever`. The [test PDF](https://github.com/NVIDIA/NeMo-Retriever/blob/main/data/multimodal_test.pdf) in the repository contains text, tables, charts, and images. + +## Query the index + +```bash +retriever query "What is in this document?" \ + --embed-model-name nvidia/llama-nemotron-embed-1b-v2 +``` + +`retriever query` reads from the same LanceDB URI and table name used during ingest unless you override `--lancedb-uri` or `--table-name`. + +## Next steps + +- For a Python-based ingest workflow, refer to [Ingest documents into a searchable collection](../extraction/workflow-document-ingestion.md). +- To route ingest to hosted NIM endpoints, export `NVIDIA_API_KEY` and pass `--page-elements-invoke-url`, `--ocr-invoke-url`, and related flags — refer to the [CLI reference — hosted endpoints](https://github.com/NVIDIA/NeMo-Retriever/tree/main/nemo_retriever/docs/cli). +- For batch Ray ingest or service ingest, use `retriever ingest batch` or `retriever ingest service` — refer to the full [CLI reference](https://github.com/NVIDIA/NeMo-Retriever/tree/main/nemo_retriever/docs/cli). diff --git a/docs/mkdocs.yml b/docs/mkdocs.yml index 4e23c7d0a..7240753e3 100644 --- a/docs/mkdocs.yml +++ b/docs/mkdocs.yml @@ -78,7 +78,8 @@ nav: - Key concepts: extraction/concepts.md - Release notes: extraction/releasenotes.md - "2. Get started": - - About this section: extraction/getting-started-about.md + - "Start here": extraction/getting-started-about.md + - "Documentation on GitHub": extraction/external-documentation-map.md - "Pre-Requisites & Support Matrix": extraction/prerequisites-support-matrix.md - "Quickstart: NeMo Retriever Library": https://github.com/NVIDIA/NeMo-Retriever/tree/main/nemo_retriever - "Authentication and API keys": extraction/api-keys.md