Skip to content

Commit e04bf77

Browse files
authored
add plugin discovery docs (#642)
Signed-off-by: Johnny Greco <jogreco@nvidia.com>
1 parent 893c70a commit e04bf77

11 files changed

Lines changed: 220 additions & 50 deletions

File tree

docs/plugins/available.md

Lines changed: 0 additions & 20 deletions
This file was deleted.

docs/plugins/discover.md

Lines changed: 101 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,101 @@
1+
# Discover Plugins
2+
3+
The Data Designer CLI is the recommended way to discover and install published plugins. It uses plugin catalogs to show install details and compatibility before installing the selected plugin package into your current environment or active `uv` project.
4+
5+
Plugins are distributed as Python packages. A single package can expose one or more runtime plugins, so the CLI installs and uninstalls packages rather than individual runtime plugin names.
6+
7+
## NVIDIA catalog
8+
9+
The default `nvidia` catalog is maintained in the [DataDesignerPlugins repository](https://github.com/NVIDIA-NeMo/DataDesignerPlugins). You do not need to configure it before using the CLI.
10+
11+
You can also browse the first-party [plugin documentation](https://nvidia-nemo.github.io/DataDesignerPlugins/plugins/) and [plugin package source](https://github.com/NVIDIA-NeMo/DataDesignerPlugins/tree/main/plugins) directly.
12+
13+
## Find a plugin package
14+
15+
When a CLI command requires a plugin package argument, you can pass either the full package name or the package alias. The package alias is the package name without the `data-designer-` prefix. For example, `data-designer-github` can be addressed as `github`.
16+
17+
Start by listing or searching the compatible packages in the default catalog. Search can match package names, package aliases, descriptions, runtime plugin names, and runtime plugin types.
18+
19+
```bash
20+
# List compatible plugin packages from the default NVIDIA catalog
21+
data-designer plugin list
22+
23+
# Search for a package
24+
data-designer plugin search github
25+
26+
# Inspect one package before installing it
27+
data-designer plugin info github
28+
```
29+
30+
## Install a plugin package
31+
32+
Install the package by full package name or package alias:
33+
34+
```bash
35+
data-designer plugin install github
36+
```
37+
38+
After installation, Data Designer discovers the package's `data_designer.plugins` entry points. Use `installed` to see the plugin packages available in the current Python environment and the runtime plugins they expose:
39+
40+
```bash
41+
data-designer plugin installed
42+
```
43+
44+
Uninstall with the same package name or alias:
45+
46+
```bash
47+
data-designer plugin uninstall github
48+
```
49+
50+
!!! note
51+
Plugins are ordinary Python packages. You can still publish a plugin to PyPI or another package index and install it directly with `pip` or `uv`. This is the path we recommend for individual plugin developers from the community. See [Community plugins](#community-plugins) below.
52+
53+
## How catalogs work
54+
55+
A plugin catalog is a JSON file that tells Data Designer which plugin packages are available and how to install them. The catalog can be hosted anywhere that serves raw JSON. Each entry points to an installable Python package and includes its docs URL, Python and Data Designer compatibility requirements, the runtime plugins it exposes after installation, and the installer metadata needed to fetch the package.
56+
57+
The package itself can live in any Python package index, or be referenced with any valid [PEP 508 direct reference](https://packaging.python.org/en/latest/specifications/dependency-specifiers/#direct-references). The package does not have to live in the same repository as the catalog.
58+
59+
The NVIDIA catalog is published at:
60+
61+
```text
62+
https://nvidia-nemo.github.io/DataDesignerPlugins/catalog/plugins.json
63+
```
64+
65+
The NVIDIA plugin packages are served from a PyPI-compatible Python Simple API index published beside that catalog:
66+
67+
```text
68+
https://nvidia-nemo.github.io/DataDesignerPlugins/simple/
69+
```
70+
71+
Catalog discovery and runtime plugin discovery are separate. Reading a catalog lets the CLI show available packages and install plans without importing plugin code. Runtime plugins become available only after their package is installed and Data Designer discovers the package's `data_designer.plugins` entry points.
72+
73+
Other catalogs can follow the same pattern as the NVIDIA plugin repository: publish a raw `catalog/plugins.json` file and, for index-backed packages, a PyPI-compatible hosted package index. Catalog entries can also point to packages on the installer's default index or to direct package references.
74+
75+
## Use another catalog
76+
77+
Add a catalog when a team or community publishes a compatible catalog JSON file. For example, an internal platform team might publish a catalog that lists approved Data Designer plugin packages and points each package at an internal Python package index. Teammates can then add that catalog once and install approved plugins by package name or alias.
78+
79+
Choose a short catalog name and use it with `--catalog`:
80+
81+
```bash
82+
data-designer plugin catalog add <name> <catalog-url-or-path>
83+
data-designer plugin --catalog <name> list
84+
data-designer plugin --catalog <name> install <package-or-alias>
85+
```
86+
87+
For published catalogs, prefer sharing the raw catalog JSON URL. Local catalog files and directories are useful while authoring or testing a catalog before publishing it.
88+
89+
```bash
90+
# See configured catalog names
91+
data-designer plugin catalog list
92+
93+
# Remove a catalog
94+
data-designer plugin catalog remove <name>
95+
```
96+
97+
## Community plugins
98+
99+
We do not have any community plugins to list here yet, but yours could be the first! If you build a plugin that could be useful to other Data Designer users, we would love to hear about it.
100+
101+
To get started, follow the patterns in the [plugin overview](overview.md) and [Build Your Own](build_your_own.md) guides, then publish your plugin package to PyPI. When your plugin is ready, open an issue on the [Data Designer GitHub repository](https://github.com/NVIDIA-NeMo/DataDesigner/issues) with the package name, source repository, documentation link, supported Data Designer versions, and the plugin types it provides. The Data Designer team will review the plugin and add it here if it seems generally useful for the community.

docs/plugins/overview.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -12,7 +12,7 @@ Data Designer supports three plugin types:
1212

1313
## Use an Installed Plugin
1414

15-
Plugin packages register their `Plugin` objects through Python package [entry points](https://packaging.python.org/en/latest/guides/creating-and-discovering-plugins/#using-package-metadata). Data Designer discovers installed plugin entry points automatically, so no extra registration code is required. Simply install the plugin package and use its new object types in your Data Designer workflow.
15+
Plugin packages register their `Plugin` objects through Python package [entry points](https://packaging.python.org/en/latest/guides/creating-and-discovering-plugins/#using-package-metadata). Data Designer discovers installed plugin entry points automatically, so no extra registration code is required. Once a plugin package is installed, use its new object types in your Data Designer workflow.
1616

1717
If you install a plugin after `data_designer` has already been imported, restart the Python process so plugin discovery can rebuild from the new entry points.
1818

@@ -22,7 +22,7 @@ For implementation instructions across all plugin types, see the [Build Your Own
2222

2323
## Find Plugins
2424

25-
NVIDIA-maintained plugin packages live in the [DataDesignerPlugins](https://github.com/NVIDIA-NeMo/DataDesignerPlugins) repository. See [Available Plugins](available.md) for lists of first-party and community-contributed plugins.
25+
Use the Data Designer CLI to discover and install published plugin packages from catalogs. See [Discover Plugins](discover.md) for the catalog workflow, first-party plugin documentation, and source links.
2626

2727
## Discovery troubleshooting
2828

fern/docs.yml

Lines changed: 0 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -340,9 +340,6 @@ redirects:
340340
destination: "/nemo/datadesigner/plugins/file-system-seed-reader-plugins"
341341
- source: "/nemo/datadesigner/plugins/example"
342342
destination: "/nemo/datadesigner/plugins/example-plugin"
343-
- source: "/nemo/datadesigner/plugins/available"
344-
destination: "/nemo/datadesigner/plugins/available-plugin-list"
345-
346343
# Code Reference: mkdocstrings tree -> Fern Topic Overviews subsection.
347344
# Underscored page names get kebab'd at the page-slug level too (Fern's title
348345
# slugifier drops underscores), so the snake_case modules need per-page rules.

fern/versions/latest.yml

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -137,8 +137,8 @@ navigation:
137137
path: ./v0.5.8/pages/plugins/example.mdx
138138
- page: FileSystemSeedReader Plugins
139139
path: ./v0.5.8/pages/plugins/filesystem_seed_reader.mdx
140-
- page: Available Plugin List
141-
path: ./v0.5.8/pages/plugins/available.mdx
140+
- page: Discover
141+
path: ./v0.5.8/pages/plugins/discover.mdx
142142
- section: Code Reference
143143
contents:
144144
- section: Topic Overviews

fern/versions/v0.5.8.yml

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -115,8 +115,8 @@ navigation:
115115
path: ./v0.5.8/pages/plugins/example.mdx
116116
- page: FileSystemSeedReader Plugins
117117
path: ./v0.5.8/pages/plugins/filesystem_seed_reader.mdx
118-
- page: Available Plugin List
119-
path: ./v0.5.8/pages/plugins/available.mdx
118+
- page: Discover
119+
path: ./v0.5.8/pages/plugins/discover.mdx
120120
- section: Code Reference
121121
contents:
122122
- section: Topic Overviews

fern/versions/v0.5.8/pages/plugins/available.mdx

Lines changed: 0 additions & 6 deletions
This file was deleted.
Lines changed: 106 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,106 @@
1+
---
2+
title: "Discover Plugins"
3+
description: "Discover and install Data Designer plugin packages from catalogs"
4+
position: 4
5+
---
6+
7+
The Data Designer CLI is the recommended way to discover and install published plugins. It uses plugin catalogs to show install details and compatibility before installing the selected plugin package into your current environment or active `uv` project.
8+
9+
Plugins are distributed as Python packages. A single package can expose one or more runtime plugins, so the CLI installs and uninstalls packages rather than individual runtime plugin names.
10+
11+
## NVIDIA catalog
12+
13+
The default `nvidia` catalog is maintained in the [DataDesignerPlugins repository](https://github.com/NVIDIA-NeMo/DataDesignerPlugins). You do not need to configure it before using the CLI.
14+
15+
You can also browse the first-party [plugin documentation](https://nvidia-nemo.github.io/DataDesignerPlugins/plugins/) and [plugin package source](https://github.com/NVIDIA-NeMo/DataDesignerPlugins/tree/main/plugins) directly.
16+
17+
## Find a plugin package
18+
19+
When a CLI command requires a plugin package argument, you can pass either the full package name or the package alias. The package alias is the package name without the `data-designer-` prefix. For example, `data-designer-github` can be addressed as `github`.
20+
21+
Start by listing or searching the compatible packages in the default catalog. Search can match package names, package aliases, descriptions, runtime plugin names, and runtime plugin types.
22+
23+
```bash
24+
# List compatible plugin packages from the default NVIDIA catalog
25+
data-designer plugin list
26+
27+
# Search for a package
28+
data-designer plugin search github
29+
30+
# Inspect one package before installing it
31+
data-designer plugin info github
32+
```
33+
34+
## Install a plugin package
35+
36+
Install the package by full package name or package alias:
37+
38+
```bash
39+
data-designer plugin install github
40+
```
41+
42+
After installation, Data Designer discovers the package's `data_designer.plugins` entry points. Use `installed` to see the plugin packages available in the current Python environment and the runtime plugins they expose:
43+
44+
```bash
45+
data-designer plugin installed
46+
```
47+
48+
Uninstall with the same package name or alias:
49+
50+
```bash
51+
data-designer plugin uninstall github
52+
```
53+
54+
<Note>
55+
Plugins are ordinary Python packages. You can still publish a plugin to PyPI or another package index and install it directly with `pip` or `uv`. This is the path we recommend for individual plugin developers from the community. See [Community plugins](#community-plugins) below.
56+
</Note>
57+
58+
## How catalogs work
59+
60+
A plugin catalog is a JSON file that tells Data Designer which plugin packages are available and how to install them. The catalog can be hosted anywhere that serves raw JSON. Each entry points to an installable Python package and includes its docs URL, Python and Data Designer compatibility requirements, the runtime plugins it exposes after installation, and the installer metadata needed to fetch the package.
61+
62+
The package itself can live in any Python package index, or be referenced with any valid [PEP 508 direct reference](https://packaging.python.org/en/latest/specifications/dependency-specifiers/#direct-references). The package does not have to live in the same repository as the catalog.
63+
64+
The NVIDIA catalog is published at:
65+
66+
```text
67+
https://nvidia-nemo.github.io/DataDesignerPlugins/catalog/plugins.json
68+
```
69+
70+
The NVIDIA plugin packages are served from a PyPI-compatible Python Simple API index published beside that catalog:
71+
72+
```text
73+
https://nvidia-nemo.github.io/DataDesignerPlugins/simple/
74+
```
75+
76+
Catalog discovery and runtime plugin discovery are separate. Reading a catalog lets the CLI show available packages and install plans without importing plugin code. Runtime plugins become available only after their package is installed and Data Designer discovers the package's `data_designer.plugins` entry points.
77+
78+
Other catalogs can follow the same pattern as the NVIDIA plugin repository: publish a raw `catalog/plugins.json` file and, for index-backed packages, a PyPI-compatible hosted package index. Catalog entries can also point to packages on the installer's default index or to direct package references.
79+
80+
## Use another catalog
81+
82+
Add a catalog when a team or community publishes a compatible catalog JSON file. For example, an internal platform team might publish a catalog that lists approved Data Designer plugin packages and points each package at an internal Python package index. Teammates can then add that catalog once and install approved plugins by package name or alias.
83+
84+
Choose a short catalog name and use it with `--catalog`:
85+
86+
```bash
87+
data-designer plugin catalog add <name> <catalog-url-or-path>
88+
data-designer plugin --catalog <name> list
89+
data-designer plugin --catalog <name> install <package-or-alias>
90+
```
91+
92+
For published catalogs, prefer sharing the raw catalog JSON URL. Local catalog files and directories are useful while authoring or testing a catalog before publishing it.
93+
94+
```bash
95+
# See configured catalog names
96+
data-designer plugin catalog list
97+
98+
# Remove a catalog
99+
data-designer plugin catalog remove <name>
100+
```
101+
102+
## Community plugins
103+
104+
We do not have any community plugins to list here yet, but yours could be the first! If you build a plugin that could be useful to other Data Designer users, we would love to hear about it.
105+
106+
To get started, follow the patterns in the [plugin overview](/plugins/overview) and [Example Plugin](/plugins/example-plugin) guides, then publish your plugin package to PyPI. When your plugin is ready, open an issue on the [Data Designer GitHub repository](https://github.com/NVIDIA-NeMo/DataDesigner/issues) with the package name, source repository, documentation link, supported Data Designer versions, and the plugin types it provides. The Data Designer team will review the plugin and add it here if it seems generally useful for the community.

fern/versions/v0.5.8/pages/plugins/overview.mdx

Lines changed: 4 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -20,17 +20,9 @@ Plugins are Python packages that extend Data Designer's capabilities without mod
2020

2121
## How do you use plugins?
2222

23-
A Data Designer plugin is just a Python package configured with an [entry point](https://packaging.python.org/en/latest/guides/creating-and-discovering-plugins/#using-package-metadata) that points to a Data Designer `Plugin` object. Using a plugin is as simple as installing the package:
23+
A Data Designer plugin is just a Python package configured with an [entry point](https://packaging.python.org/en/latest/guides/creating-and-discovering-plugins/#using-package-metadata) that points to a Data Designer `Plugin` object. Once installed, plugins are automatically discovered and ready to use — no additional registration or configuration needed.
2424

25-
```bash
26-
# Install a local plugin (for development and testing)
27-
uv pip install -e /path/to/your/plugin
28-
29-
# Or install a published plugin from PyPI
30-
pip install data-designer-{plugin-name}
31-
```
32-
33-
Once installed, plugins are automatically discovered and ready to use — no additional registration or configuration needed. See the [example plugin](/plugins/example-plugin) for a complete walkthrough, or jump to [FileSystemSeedReader Plugins](/plugins/filesystemseedreader-plugins) for filesystem-backed seed reader authoring.
25+
Use the Data Designer CLI to discover and install published plugin packages from catalogs. See [Discover Plugins](/plugins/discover) for the catalog workflow. See the [example plugin](/plugins/example-plugin) for a complete authoring walkthrough, or jump to [FileSystemSeedReader Plugins](/plugins/filesystemseedreader-plugins) for filesystem-backed seed reader authoring.
3426

3527
## How do you create plugins?
3628

@@ -64,8 +56,8 @@ Each plugin has three components, and we recommend organizing them into separate
6456

6557
### 4. Share Your Plugin (Optional)
6658

67-
- Publish to PyPI or another package index to make it installable by anyone via `pip install`
68-
- This step is only needed if you want others outside your environment to use the plugin
59+
- Publish to PyPI, another package index, or a direct package reference when you want others outside your environment to install the plugin
60+
- Add the package to a plugin catalog so users can discover and install it through `data-designer plugin`
6961

7062
**Example entry point for a processor plugin:**
7163

fern/versions/v0.5.9.yml

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -136,8 +136,8 @@ navigation:
136136
path: ./v0.5.8/pages/plugins/example.mdx
137137
- page: FileSystemSeedReader Plugins
138138
path: ./v0.5.8/pages/plugins/filesystem_seed_reader.mdx
139-
- page: Available Plugin List
140-
path: ./v0.5.8/pages/plugins/available.mdx
139+
- page: Discover
140+
path: ./v0.5.8/pages/plugins/discover.mdx
141141
- section: Code Reference
142142
contents:
143143
- section: Topic Overviews

0 commit comments

Comments
 (0)