docs: add developer guide and related documentation for generators and validation

Co-authored-by: Copilot <copilot@github.com>

Author: gitmobkab

docs/dev/docstrings.md

@@ -0,0 +1,94 @@
+# Docstring Format
+
+Copia parses generator docstrings to build the in-app help and the documentation site. The format is a subset of Google-style docstrings with one Copia-specific addition.
+
+---
+
+## Structure
+
+```python
+def my_generator(param: str = "default") -> str:
+    """Short description of what the generator produces.
+
+    Locale dependent:
+        yes
+
+    Args:
+        param: Description of the parameter. Defaults to 'default'.
+
+    Returns:
+        Description of the returned value.
+
+    Note:
+        Optional warning or constraint the user should know about.
+    """
+```
+
+---
+
+## Sections
+
+### Description
+
+The first lines before any section header. Keep it short — one or two sentences. This is the most important part of the docstring, as it gives users a quick understanding of what the generator does.
+
+```python
+"""Generate a random job title."""
+```
+
+### `Locale dependent`
+
+Indicates whether the output varies based on the active generation locale. Must be `yes` or `no`.
+
+```
+Locale dependent:
+    yes
+```
+
+If omitted, the generator is assumed to be locale-independent.
+
+!!! note title="Not entirely strict"
+    To be honest, if the line below Locale dependent is not exactly `yes` it's treated as `no`. But it's best to follow the format strictly to avoid confusion.
+
+### `Args`
+
+Documents each parameter. One line per parameter in the format `name: description.`
+if the parameter has a default value, include it in the description for clarity. The type is inferred from the function signature — do not repeat it here.
+
+```
+Args:
+    length: Number of characters. Defaults to 12.
+    special_chars: Include special characters. Defaults to True.
+```
+
+### `Returns`
+
+A single line describing the returned value. The type is inferred from the return annotation — do not repeat it here.
+
+```
+Returns:
+    A randomly generated username string.
+```
+
+### `Note`
+
+An optional warning or constraint. Use sparingly — only for things that could cause runtime errors or surprising behavior.
+
+```
+Note:
+    Only uses values already present in the database.
+    An empty column will raise an error at runtime.
+```
+
+---
+
+## What gets rendered
+
+The parser extracts these sections and renders them as:
+
+- A signature blockquote
+- A description paragraph
+- A locale dependent checkbox
+- An arguments table
+- A returns line
+- A warning blockquote for notes

docs/dev/generators.md

@@ -0,0 +1,56 @@
+# Adding a Generator
+
+Generators are plain Python functions registered in the `GENERATORS_REGISTRY`. Adding one requires writing the function, documenting it, and placing it in the right module.
+
+---
+
+## 1. Write the function
+
+A generator is a regular Python function. It must follow these rules:
+
+- It must have a return type annotation
+- It must only use types compatible with SQL — `str`, `int`, `float`, `bool`, `date`, `datetime`, `UUID`
+- It must not use `**kwargs`
+- It may use `*args` (var positional), but not simultaneously with other positional-or-keyword parameters
+
+```python
+from copia.generators._core import get_faker
+
+def job() -> str:
+    return get_faker().job()
+```
+
+Use `get_faker()` to access the global Faker instance. This ensures the generator respects the active locale and generation settings.
+
+---
+
+## 2. Document it
+
+Follow the [docstring format](docstrings.md) to document the generator. Copia uses the docstring to generate the in-app help and the documentation site.
+
+```python
+def job() -> str:
+    """Generate a random job title.
+
+    Locale dependent:
+        yes
+
+    Returns:
+        A random job title string.
+    """
+    return get_faker().job()
+```
+
+---
+
+## 3. Register it
+
+Place the function in the appropriate module under `src/copia/generators/` and the rest will be automatically handled by `src/copia/generators/__init__.py`
+
+---
+
+## 4. Verify
+
+1. Run the test with `pytest tests/test_generators_discover.py` to ensure there are no import warnings or errors related to your new generator.
+
+2. Run `copia` and press `?`. to check how your generator docstring looks in the in-app help. If you see any formatting issues, adjust your docstring accordingly.

docs/dev/generators_registry.md

@@ -0,0 +1,83 @@
+# The `generators_registry` System
+
+in the `generators/__init__.py` file 
+
+there's a function called `_build_generators_registry`.
+this function is responsible for dynamically building the registry of generators based on a few rules and conventions.
+
+---
+
+## How it works
+
+- when the module is imported, `_build_generators_registry` is called to populate the `GENERATORS_REGISTRY` dictionary.
+
+- it scans the module's global namespace (copia.generators) for all non private modules (those that don't start with `_`).
+
+- for each module, it looks for all non private functions defined in that module. (again, those that don't start with `_`).
+
+- each function is registered in the `GENERATORS_REGISTRY` with its name as the key and the function object as the value.
+
+!!! note "Automatic discovery"
+    all this happens automatically when you import the generators module. You don't need to do anything special to register your generator — just define it in a non-private module and it will be picked up.
+
+---
+
+## Constraints
+
+### No overriding
+
+A generator name must be globally unique. If a function with the same name is already registered, `discover()` raises a `ImportWarning`.
+
+```python
+Traceback (most recent call last):
+    ImportWarning: found overrinding generator func at copia.generators.address with identifier city
+```
+
+### No `**kwargs`
+
+Generators must not use `**kwargs`. The DSL only supports positional and named arguments — variadic keyword arguments are not representable in the grammar.
+
+```python
+# Invalid
+def my_gen(**kwargs) -> str: ...
+```
+
+### `*args` and named parameters are mutually exclusive
+
+A generator may use either `*args` (var positional) or POSITIONAL_OR_KEYWORD parameters — but not both simultaneously.
+
+```python
+# Valid — only *args
+def enum(*args) -> Any: ...
+
+# Valid — only POSITIONAL_OR_KEYWORD params
+def email(safe: bool = True, domain: str = "") -> str: ...
+
+# All Valids — *args with only named params
+def enum(*args, extra: str) -> str: ...
+def yet_another_gen(*args, extra: str = "") -> str: ...
+
+# Invalid — POSITIONAL_OR_KEYWORD params with *args (var positional)
+def invalid_gen(extra: str, *args) -> str: ...
+
+# WHY ? Because the Semantic Validator for the DSL cannot currently handle this combination. It would require a more complex parsing logic to determine which arguments are meant for `*args` and which are actual positionals parameters, especially since the user can pass any number of values for `*args`. To keep things simple and unambiguous, we enforce this constraint.
+```
+
+!!! note "About union types"
+    The current implementation of the docstring parser and the semantic validator does not support union types (e.g., `str | int`) in generator signatures. This will be updated in future updates. and no, even Any is not supported too.
+
+---
+
+## Adding new generators
+
+To add a new generator, simply define a function in any non-private module under `copia.generators` and ensure it follows the constraints outlined above. The function will be automatically registered and available for use in the DSL.
+
+first check if the module you want to add the generator to already exists. If it does, add your function there. If not, create a new module and add your function there.
+
+since the registry is built dynamically, you don't need to do anything else to make your generator available — just define it and it will be picked up when the module is imported.
+
+to ensure your generator is properly documented, follow the docstring format outlined in the [docstrings guide](./docstrings.md). This will help users understand how to use your generator and what it does.
+
+and to finish, simply run pytest tests/test_generators_discover.py to avoid any import warnings or errors related to your new generator. If you see an ImportWarning about overriding a generator, it means you have a naming conflict — either rename your generator or remove the conflicting one. If you see an error about `**kwargs` or invalid parameter combinations, adjust your function signature accordingly.
+
+> just keep in mind, the function signature is a direct interface to users, so keep it simple and intuitive. Avoid complex parameter combinations that could confuse users or make the generator difficult to use in the DSL. The goal is to provide a clear and straightforward way for users to generate data without having to worry about the underlying implementation details.

docs/dev/index.md

@@ -0,0 +1,13 @@
+# Developer Guide
+
+This section documents the internals of Copia for contributors and developers who want to extend it.
+
+---
+
+## Contents
+
+- [Adding a generator](generators.md) — how to write and register a new generator
+- [Docstring format](docstrings.md) — the docstring convention Copia uses to generate documentation
+- [The dynamic generators system](generators_registry.md) — how generators are registered and what constraints apply
+- [The semantic validator](validator.md) — how Copia validates DSL calls against generator signatures
+- [How `ref` works](ref.md) — the internals of the `ref()` generator

docs/dev/ref.md

@@ -0,0 +1,67 @@
+# How `ref` Works
+
+`ref` is the only generator that reads from the database rather than generating data. It samples a random value from an existing column.
+
+---
+
+## Usage
+
+```
+user_id: ref('users.id')
+```
+
+The argument is a string in `table.column` format. `ref` fetches all existing values from that column and returns one at random on each call.
+
+---
+
+## Internals
+
+### Parsing the target
+
+When the DSL is parsed, `ref('users.id')` produces a `GeneratorCall` with a single positional argument — the string `"users.id"`. Before generation, the runner walks all parsed columns, identifies `ref` calls, and extracts the `table.column` string via `_parse_ref_input()`.
+
+The string is split on `.` and must produce exactly two non-empty parts — `table` and `column`. Any other format raises a `GeneratorValueError` immediately.
+
+### Phase 1 — Initialize
+
+Before any generation, the runner scans all parsed columns for `ref` calls and builds an empty `REF_COLLECTION` structure:
+
+```python
+REF_COLLECTION = {
+    "users": {
+        "id": []
+    },
+    "posts": {
+        "id": []
+    }
+}
+```
+
+All tables and columns are identified upfront. Columns belonging to the same table are grouped together — this is what makes the next phase efficient.
+
+### Phase 2 — Populate
+
+With the structure in hand, the runner fetches all columns of a given table in a single query per table. If two `ref` calls target `users.id` and `users.email`, only one query is issued:
+
+```sql
+SELECT id, email FROM users
+```
+
+This is more efficient than one query per `ref` call. The results are distributed into the corresponding lists in `REF_COLLECTION`.
+
+### Sampling
+
+On each row generation, `ref('users.id')` samples a random value from `REF_COLLECTION["users"]["id"]` using `random.choice()`.
+
+### Empty column behavior
+
+If the target column is empty, `REF_COLLECTION["table"]["column"]` will be an empty list. Sampling from an empty list raises a `GeneratorValueError` with a descriptive message pointing to the empty column.
+
+This is why the `ref` documentation warns: **always populate parent tables before referencing them.**
+
+---
+
+## Limitations
+
+- `ref` can only reference values already in the database at the time of the run. Values generated in the same run are not available to `ref`.
+- `ref` requires an active database connection.

docs/dev/validator.md

@@ -0,0 +1,56 @@
+# The Semantic Validator
+
+The semantic validator checks that a parsed DSL call is valid against the registered generator signatures. It runs after parsing and before generation.
+
+---
+
+## What it validates
+
+Given a `GeneratorCall` from the parser, the validator checks:
+
+- The generator name exists in `GENERATORS_REGISTRY`
+- The number of positional arguments does not exceed what the signature accepts
+- All named arguments exist in the signature
+- The types of all arguments match the parameter annotations
+
+If any check fails, the validator raises a typed exception that the TUI catches and displays in the results viewer.
+
+---
+
+## Errors
+
+| Exception | Cause | Can be skipped? |
+|-----------|-------|-----------------|
+| `UnknownGeneratorError` | The generator name is not in the registry | Never |
+| `TooManyPositionalsError` | More positional args than the signature accepts | Only for *args generators |
+| `UnknownNamedParamError` | A named argument does not exist in the signature | Never |
+| `MissingRequiredParamError` | A required parameter was not provided | Never |
+| `TypeMismatchError` | An argument type does not match the annotation | Never |
+| `PositionalNamedCollisionError` | A parameter is passed both positionally and by name | Only for *args generators |
+
+---
+
+## Type checking
+
+The validator maps DSL argument types to Python types:
+
+| DSL value | Python type |
+|-----------|-------------|
+| `42` | `int` |
+| `3.14` | `float` |
+| `'hello'` | `str` |
+| `True` / `False` | `bool` |
+
+`Literal` annotations are checked by value — the argument must be one of the allowed values.
+
+---
+
+## Var positional generators
+
+Generators that use `*args` (like `enum`) bypass the arity check — any number of positional arguments is valid. The type of each argument is still checked against the parameter annotation if present.
+
+---
+
+## When validation runs
+
+Validation runs once when the user clicks **Run**, before any generation starts. If validation fails, no rows are generated and the error is displayed in the logs tab.

mkdocs.yml

@@ -48,6 +48,13 @@ nav:
   - Blog:
     - blog/index.md
   - Changelog: changelog.md
+  - Developer Guide:
+    - dev/index.md
+    - dev/generators.md
+    - dev/docstrings.md
+    - dev/generators_registry.md
+    - dev/validator.md
+    - dev/ref.md
 
 
 markdown_extensions: