update and reorganise docs

Author: charliecloudberry

CONTRIBUTING.md

@@ -0,0 +1,123 @@
+# contributing
+
+## prerequisites
+
+- **Rust** (stable toolchain)
+- **Python 3.10+** (for running transpiled output)
+- **uv** (recommended for Python dependency management)
+
+## building
+
+basedpython is a Rust project managed with Cargo:
+
+```sh
+cargo build
+```
+
+the binary is built to `target/debug/by`. for a release build:
+
+```sh
+cargo build --release
+```
+
+## running tests
+
+```sh
+cargo test
+```
+
+- **inline tests** — most transform files in `src/transforms/` have `#[cfg(test)]` modules at the bottom with unit tests that call `transpile()` on a snippet and assert the output
+- **end-to-end tests** — `tests/e2e.rs` runs larger integration scenarios
+
+## project structure
+
+```
+src/
+├── main.rs              # CLI entry point (clap argument parsing)
+├── lib.rs               # transpile() — orchestrates the full pipeline
+├── args.rs              # CLI argument definitions
+├── config.rs            # Config struct and PythonVersion enum
+├── source_map.rs        # source map generation
+├── symbol_table.rs      # module-level symbol collection
+├── transforms/          # forward transforms (basedpython → Python)
+│   ├── mod.rs
+│   ├── subscript.rs
+│   ├── mutable_defaults.rs
+│   ├── generics.rs
+│   ├── modifiers.rs
+│   ├── coalesce.rs
+│   ├── none_chain.rs
+│   └── ...
+└── reverse_transforms/  # reverse transforms (Python → basedpython)
+    ├── mod.rs
+    ├── empty_class.rs
+    ├── literal_types.rs
+    └── subscript.rs
+crates/                  # forked ruff/ty crates
+docs/                    # documentation pages
+tests/                   # end-to-end tests
+```
+
+## adding a new transform
+
+1. **create the transform file** in `src/transforms/`, e.g. `my_feature.rs`
+
+2. **define a struct** that holds the source text and a `Vec<(TextRange, String)>` for edits:
+
+   ```rust
+   pub struct MyFeature<'src> {
+       source: &'src str,
+       pub edits: Vec<(ruff_text_size::TextRange, String)>,
+   }
+   ```
+
+3. **implement `Visitor`** from `ruff_python_ast::visitor` — walk the AST nodes you care about and push edits:
+
+   ```rust
+   impl Visitor<'_> for MyFeature<'_> {
+       fn visit_stmt(&mut self, stmt: &Stmt) {
+           // detect your pattern, push to self.edits
+           walk_stmt(self, stmt);
+       }
+   }
+   ```
+
+4. **register the transform** in `src/transforms/mod.rs` and wire it into the pipeline in `src/lib.rs`:
+   - instantiate it alongside the other transforms
+   - call `visitor.visit_stmt(stmt)` in the main loop
+   - extend the edits vec with your transform's edits
+   - if your transform needs generated imports (e.g. `from typing import TypeVar`), integrate with the **preamble system** — after all edits are applied, `lib.rs` collects import requests from transforms and prepends them to the output file. see how `generics.needed_imports` or `literal_types.needs_literal_import` for examples
+
+5. **add tests** in a `#[cfg(test)]` module at the bottom of your file:
+
+   ```rust
+   #[cfg(test)]
+   mod tests {
+       use crate::{Config, transpile};
+
+       fn t(input: &str) -> String {
+           transpile(input, &Config::default()).unwrap()
+       }
+
+       #[test]
+       fn basic() {
+           assert_eq!(t("input code"), "expected output");
+       }
+   }
+   ```
+
+6. **add a doc page** in `docs/` describing the syntax and showing examples
+
+## running the docs locally
+
+the documentation site is built with [zensical](https://github.com/kotlinisland/zensical). to view it locally, use:
+
+```sh
+uv run zensical serve
+```
+
+this starts a local dev server (usually at `http://localhost:8000`) with live reload. pages are in `docs/` and the site configuration is in `zensical.toml`
+
+## adding a reverse transform
+
+reverse transforms live in `src/reverse_transforms/` and follow the same pattern. they detect standard Python patterns that correspond to basedpython idioms and rewrite them back. register new reverse transforms in `src/reverse_transforms/mod.rs`

docs/generics.md docs/development/generics.mddocs/generics.md renamed to docs/development/generics.md

@@ -54,10 +54,10 @@ TODO: more top types like `Callable[Ts, None]`?
 `Concatenate` will be replaced with a new unpack notation:
 `***` means unpack positional and keyword items, a combination of `*` and `**`
 ```bython
-def f[P: Parameters](fn: Callable[P, None] -> Callable[[int, ***P], None]:
+def f[P: Parameters](fn: Callable[P, None]) -> Callable[[int, ***P], None]:
 ```
 
-`Callable` will expose it's type via "attributes as types" by forwarding to it's `Parameters` type parameter
+`Callable` will expose its type via "attributes as types" by forwarding to it's `Parameters` type parameter
 ```bython
 class Callable[P: Parameters, R]:
     @type_check_only

docs/development/how-transpilation-works.md

@@ -0,0 +1,71 @@
+# how transpilation works
+
+basedpython transpiles `.by` source files into standard Python
+
+## pipeline
+
+```
+source (.by)
+  │
+  ├─ 1. parse (ruff_python_parser)
+  │     └─ produces a Python AST
+  │
+  ├─ 2. build symbol table
+  │     └─ collects module-level names for use by transforms
+  │
+  ├─ 3. run transforms
+  │     └─ each transform walks the AST and emits text edits (range → replacement)
+  │
+  ├─ 4. collect & deduplicate edits
+  │     └─ overlapping edits are resolved
+  │
+  ├─ 5. apply edits to source text
+  │     └─ produces the output Python string
+  │
+  ├─ 6. append preamble
+  │     └─ generated imports (typing, typing_extensions, etc.) are prepended
+  │
+  └─ 7. build source map
+        └─ maps output byte offsets back to input byte offsets
+```
+
+## transforms
+
+each transform is a struct that implements ruff's `Visitor` trait. it walks the AST looking for patterns it handles and records text edits as `(TextRange, String)` pairs. transforms do not modify the AST — they only produce edits against the original source text
+
+all transforms run independently over the same AST in a single pass:
+
+| transform | file | what it does |
+|---|---|---|
+| subscription normalization | `subscript.rs` | normalizes tuple subscript keys |
+| mutable defaults | `mutable_defaults.rs` | rewrites mutable default arguments to sentinel pattern |
+| typing redirect | `typing_redirect.rs` | redirects `typing` imports to `typing_extensions` |
+| generics polyfill | `generics.rs` | desugars PEP 695 generics to TypeVar/Generic |
+| compat rewrites | `compat.rs` | rewrites expressions for older Python versions |
+| tuple literal types | `annotation.rs` | `(int, str)` → `tuple[int, str]` in annotations |
+| literal types | `literal_types.rs` | string/number literals in type positions → `Literal[...]` |
+| auto-quoting | `auto_quote.rs` | quotes forward self-references in class definitions |
+| intersection types | `intersection.rs` | `A & B` → `Intersection[A, B]` |
+| callable syntax | `callable.rs` | `(int) -> int` → `Callable[[int], int]` |
+| unpack syntax | `unpack.rs` | `*tuple[int, ...]` → `Unpack[tuple[int, ...]]` |
+| empty declarations | `empty_declarations.rs` | `class A` → `class A: ...` |
+| modifiers | `modifiers.rs` | keyword modifiers → decorators/annotations |
+| overload | `overload.rs` | stacked signatures → `@overload` |
+| none-coalescing | `coalesce.rs` | `a ?? b` → conditional expression |
+| none-chaining | `none_chain.rs` | `a?.b` → guarded access |
+| multiline dedent | `dedent_string.rs` | strips common indentation from triple-quoted strings |
+| typed lambda | `typed_lambda.rs` | `lambda (a: int): ...` → `lambda a: ...` |
+
+after all transforms run, their edits are merged, deduplicated, and applied to produce the final output
+
+## source maps
+
+`transpile_with_map` returns a `SourceMap` alongside the output string. the source map records how byte offsets in the generated Python correspond to byte offsets in the original `.by` source. this enables error messages and debuggers to point back to the correct line in the original file
+
+## reverse transforms
+
+basedpython also supports reverse transpilation — converting standard Python back into basedpython syntax. see the [reverse transforms](reverse-transforms.md) page for details
+
+## preamble generation
+
+transforms that require new imports (e.g. `from typing import TypeVar`) register them during the walk. after all edits are applied, the transpiler collects these imports and prepends them as a preamble to the output file

docs/development/reverse-transforms.md

@@ -0,0 +1,57 @@
+# reverse transforms
+
+basedpython can convert standard Python source back into basedpython syntax. this is the inverse of the normal transpilation pipeline
+
+## usage
+
+```sh
+by transpile --reverse file.py
+```
+
+## what it does
+
+reverse transforms detect patterns in standard Python that correspond to basedpython idioms and rewrite them back. this enables round-tripping: a Python file run through `--reverse` and then transpiled forward should produce code with the same AST as the original
+
+
+## implemented reverse transforms
+
+### empty class bodies
+
+detects `class A: ...` (with a `pass` or `...` body) and rewrites it to the basedpython shorthand:
+
+```python
+# standard Python
+class A: ...
+
+# basedpython
+class A
+```
+
+### literal types
+
+detects `Literal["foo", 5]` in annotations and rewrites to basedpython's inline literal syntax:
+
+```python
+# standard Python
+from typing import Literal
+a: Literal["foo", 5]
+
+# basedpython
+a: "foo" | 5
+```
+
+### subscription normalization
+
+detects the trailing-comma 1-tuple form `x[(a, b),]` and rewrites it back to the natural parenthesized form:
+
+```python
+# standard Python (transpiled)
+x[(a, b),]
+
+# basedpython
+x[(a, b)]
+```
+
+## design
+
+each reverse transform lives in `src/reverse_transforms/` and mirrors the structure of the forward transforms in `src/transforms/`. they use the same visitor-based approach: walk the AST, detect the polyfill pattern, and emit text edits to rewrite it back to basedpython syntax

docs/ruff-upstream-sync.md docs/development/ruff-upstream-sync.mddocs/ruff-upstream-sync.md renamed to docs/development/ruff-upstream-sync.md

@@ -0,0 +1,31 @@
+# auto-quoting
+
+basedpython automatically quotes forward self-references in class definitions, so you don't need `from __future__ import annotations` or manual string literals
+
+## transformation
+
+```python
+# basedpython
+class A(list[A]):
+    children: list[A]
+```
+```python
+# generated Python
+class A(list["A"]):
+    children: list["A"]
+```
+
+## when it applies
+
+the transform applies when the class's own name appears as a subscript slice argument in:
+
+- base class expressions (e.g. `list[A]` in `class A(list[A])`)
+- the class body (e.g. `list[A]` in an annotation)
+
+direct base class references (`class A(A):`) are left alone — that is a runtime error regardless of quoting
+
+## motivation
+
+in Python, a class name is not yet bound when its body is being evaluated. using the class name in a generic base class like `class Tree(list[Tree])` raises a `NameError` at runtime. the standard workaround is either `from __future__ import annotations` (which defers all annotation evaluation) or manually writing `list["Tree"]`
+
+basedpython handles this automatically — you write the natural syntax, and the transpiler inserts the quotes where needed

docs/features/auto-quoting.md

@@ -0,0 +1,84 @@
+# callable syntax
+
+basedpython lets you write callable types using arrow syntax instead of `typing.Callable`
+
+## transformation rules
+
+| basedpython | Python output |
+|---|---|
+| `(int) -> int` | `Callable[[int], int]` |
+| `(int, str) -> bool` | `Callable[[int, str], bool]` |
+| `() -> None` | `Callable[[], None]` |
+| `(int) -> (str) -> bool` | `Callable[[int], Callable[[str], bool]]` |
+
+the `Callable` import from `typing` is added automatically when needed
+
+## examples
+
+basic callable annotation:
+
+```python
+# basedpython
+a: (int) -> int
+```
+```python
+# generated Python
+from typing import Callable
+a: Callable[[int], int]
+```
+
+multiple parameters:
+
+```python
+# basedpython
+a: (int, str) -> bool
+```
+```python
+# generated Python
+from typing import Callable
+a: Callable[[int, str], bool]
+```
+
+no parameters:
+
+```python
+# basedpython
+a: () -> None
+```
+```python
+# generated Python
+from typing import Callable
+a: Callable[[], None]
+```
+
+nested callables:
+
+```python
+# basedpython
+a: (int) -> (str) -> bool
+```
+```python
+# generated Python
+from typing import Callable
+a: Callable[[int], Callable[[str], bool]]
+```
+
+works inside generic types:
+
+```python
+# basedpython
+a: list[(int) -> int]
+```
+```python
+# generated Python
+from typing import Callable
+a: list[Callable[[int], int]]
+```
+
+## annotation context only
+
+the arrow syntax is only rewritten in type annotation positions. in value expressions, parenthesized tuples are left unchanged:
+
+```python
+x = (int)  # not rewritten — this is a value expression
+```