|
| 1 | +# Usage |
| 2 | + |
| 3 | +This tutorial walks through the common patterns for turning repositories into |
| 4 | +Data Designer seed rows. The examples use the Python builder API, but the same |
| 5 | +configuration fields apply when a workflow is built from serialized config. |
| 6 | + |
| 7 | +## Read a GitHub repository |
| 8 | + |
| 9 | +Start with a small repository and a narrow file pattern. This keeps previews |
| 10 | +fast and makes it clear which rows are entering the workflow. |
| 11 | + |
| 12 | +```python |
| 13 | +from data_designer.config.config_builder import DataDesignerConfigBuilder |
| 14 | +from data_designer.interface.data_designer import DataDesigner |
| 15 | +from data_designer_github.config import GitHubSeedSource |
| 16 | + |
| 17 | +builder = DataDesignerConfigBuilder() |
| 18 | +builder.with_seed_dataset( |
| 19 | + GitHubSeedSource( |
| 20 | + repositories=["pallets/markupsafe"], |
| 21 | + file_pattern="*.py", |
| 22 | + recursive=True, |
| 23 | + ) |
| 24 | +) |
| 25 | + |
| 26 | +builder.add_column( |
| 27 | + name="_row_id", |
| 28 | + column_type="sampler", |
| 29 | + sampler_type="uuid", |
| 30 | + params={}, |
| 31 | +) |
| 32 | + |
| 33 | +preview = DataDesigner().preview(builder, num_records=5) |
| 34 | +print(preview.dataset[["repo_id", "relative_path", "code_lang", "content"]]) |
| 35 | +``` |
| 36 | + |
| 37 | +The seed rows contain repository provenance and file text. Downstream columns can |
| 38 | +then ask questions such as "summarize this file", "identify risky APIs", "write |
| 39 | +a short module description", or "extract candidate test scenarios" using the |
| 40 | +`content`, `relative_path`, `code_lang`, and `commit_sha` columns. |
| 41 | + |
| 42 | +## Pin a branch, tag, or commit |
| 43 | + |
| 44 | +Use `ref` when the dataset must be reproducible against a specific branch, tag, |
| 45 | +or commit. Branches and tags are passed to `git clone --branch`; commit SHAs are |
| 46 | +checked out after cloning. |
| 47 | + |
| 48 | +```python |
| 49 | +source = GitHubSeedSource( |
| 50 | + repositories=["NVIDIA-NeMo/DataDesigner"], |
| 51 | + ref="v0.5.7", |
| 52 | + clone_depth=1, |
| 53 | + file_pattern="*.py", |
| 54 | + recursive=True, |
| 55 | +) |
| 56 | +``` |
| 57 | + |
| 58 | +For arbitrary commit SHAs, set `clone_depth=None` if the commit may not be |
| 59 | +reachable from the shallow default clone. |
| 60 | + |
| 61 | +```python |
| 62 | +source = GitHubSeedSource( |
| 63 | + repositories=["NVIDIA-NeMo/DataDesigner"], |
| 64 | + ref="0123456789abcdef0123456789abcdef01234567", |
| 65 | + clone_depth=None, |
| 66 | + file_pattern="*.py", |
| 67 | + recursive=True, |
| 68 | +) |
| 69 | +``` |
| 70 | + |
| 71 | +## Read local repositories |
| 72 | + |
| 73 | +Local repositories are useful for private code, local experiments, or a checked |
| 74 | +out monorepo that already exists on disk. |
| 75 | + |
| 76 | +```python |
| 77 | +source = GitHubSeedSource( |
| 78 | + repository_paths=[ |
| 79 | + "/workspace/services/api", |
| 80 | + "/workspace/libraries/shared", |
| 81 | + ], |
| 82 | + file_pattern="*.py", |
| 83 | + recursive=True, |
| 84 | +) |
| 85 | +``` |
| 86 | + |
| 87 | +If `path` points at a git repository, that repository is read. If `path` points |
| 88 | +at a directory whose immediate children are git repositories, each child |
| 89 | +repository is discovered and read. |
| 90 | + |
| 91 | +```python |
| 92 | +source = GitHubSeedSource( |
| 93 | + path="/workspace/repos", |
| 94 | + file_pattern="*.ts", |
| 95 | + recursive=True, |
| 96 | +) |
| 97 | +``` |
| 98 | + |
| 99 | +## Control which files become rows |
| 100 | + |
| 101 | +The reader first applies `file_pattern` and `recursive`, then filters by |
| 102 | +extension, file name, exclude pattern, and file size. |
| 103 | + |
| 104 | +```python |
| 105 | +source = GitHubSeedSource( |
| 106 | + repositories=["NVIDIA-NeMo/DataDesigner"], |
| 107 | + file_pattern="*", |
| 108 | + recursive=True, |
| 109 | + include_extensions=["py", "toml", "md"], |
| 110 | + include_file_names=["Dockerfile", "Makefile"], |
| 111 | + exclude_patterns=[ |
| 112 | + ".git/**", |
| 113 | + "**/__pycache__/**", |
| 114 | + "**/build/**", |
| 115 | + "**/dist/**", |
| 116 | + "docs/generated/**", |
| 117 | + ], |
| 118 | + max_file_size_bytes=250_000, |
| 119 | +) |
| 120 | +``` |
| 121 | + |
| 122 | +Use `include_extensions=None` for broad repository inventory tasks where the |
| 123 | +glob and exclude patterns should decide the candidate set. |
| 124 | + |
| 125 | +```python |
| 126 | +source = GitHubSeedSource( |
| 127 | + repositories=["owner/repo"], |
| 128 | + file_pattern="LICENSE*", |
| 129 | + recursive=False, |
| 130 | + include_extensions=None, |
| 131 | +) |
| 132 | +``` |
| 133 | + |
| 134 | +## Typical workflows |
| 135 | + |
| 136 | +`data-designer-github` works best as the seed layer for file-level code |
| 137 | +workflows: |
| 138 | + |
| 139 | +- Repository QA: score files for risky dependencies, missing license headers, or |
| 140 | + stale implementation notes. |
| 141 | +- Documentation generation: turn source files into module summaries, migration |
| 142 | + notes, or API reference drafts. |
| 143 | +- Test ideation: derive test scenarios from implementation files and route them |
| 144 | + to a code-generation column. |
| 145 | +- Code search preparation: create embeddings or labels from stable file content |
| 146 | + and repository metadata. |
| 147 | +- Dataset construction: sample representative code files from several projects |
| 148 | + while preserving `repo_id`, `relative_path`, and `commit_sha` provenance. |
| 149 | + |
| 150 | +Because the reader emits full file content, prompts should account for file |
| 151 | +length and language. A common pattern is to filter or sample seed rows first, |
| 152 | +then generate focused columns that reference only the metadata and content each |
| 153 | +task needs. |
| 154 | + |
| 155 | +## Operational notes |
| 156 | + |
| 157 | +The plugin requires `git` on `PATH`. GitHub repositories are cloned into a |
| 158 | +temporary runtime directory for the reader attachment and local repositories are |
| 159 | +read in place. Files that exceed `max_file_size_bytes` are skipped before |
| 160 | +hydration. Files that cannot be decoded with `encoding` are skipped with a |
| 161 | +warning rather than producing partial text. |
| 162 | + |
| 163 | +The reader does not call the GitHub API, manage credentials, or expand GitHub |
| 164 | +issues and pull requests. It is scoped to repository file content so workflows |
| 165 | +can compose repository-aware seed data with the rest of Data Designer. |
0 commit comments