fix: optional tree-sitter, untracked files in diff, binary read optimization

Author: nikolay-e

CLAUDE.md

@@ -1,11 +1,186 @@
 # TreeMapper
 
-> Extends [../CLAUDE.md](../CLAUDE.md)
+<!-- Extends ../CLAUDE.md -->
+
+[![PyPI](https://img.shields.io/pypi/v/treemapper)](https://pypi.org/project/treemapper/)
+[![Downloads](https://img.shields.io/pypi/dm/treemapper)](https://pypi.org/project/treemapper/)
+[![License](https://img.shields.io/github/license/nikolay-e/treemapper)](https://github.com/nikolay-e/treemapper/blob/main/LICENSE)
+
+**Export your codebase for AI/LLM context in one command.**
+
+```bash
+pip install treemapper                    # core (no native extensions)
+pip install 'treemapper[tree-sitter]'     # + AST parsing for 10 languages
+treemapper . -o context.yaml              # paste into ChatGPT/Claude
+```
+
+## Why TreeMapper?
+
+Unlike `tree` or `find`, TreeMapper exports **structure + file
+contents** in a format optimized for LLM context windows:
+
+```yaml
+name: myproject
+type: directory
+children:
+  - name: main.py
+    type: file
+    content: |
+      def hello():
+          print("Hello, World!")
+  - name: utils/
+    type: directory
+    children:
+      - name: helpers.py
+        type: file
+        content: |
+          def add(a, b):
+              return a + b
+```
+
+## Usage
+
+```bash
+treemapper                               # current dir, YAML to stdout
+treemapper .                          # YAML to stdout + token count
+treemapper . -o tree.yaml             # save to file
+treemapper . -o                       # save to tree.yaml (default)
+treemapper . -o -                     # explicit stdout output
+treemapper . -f json                  # JSON format
+treemapper . -f txt                   # plain text with indentation
+treemapper . -f md                    # Markdown with fenced code
+treemapper . -f yml                   # YAML (alias)
+treemapper . --no-content             # structure only
+treemapper . --max-depth 3            # limit depth (0=root, 1=children)
+treemapper . --max-file-bytes 10000   # skip files > 10KB (default: 10 MB)
+treemapper . --max-file-bytes 0       # no limit
+treemapper . -i custom.ignore         # custom ignore patterns
+treemapper . --no-default-ignores     # disable .gitignore + defaults
+treemapper . --log-level info         # log level (default: error)
+treemapper . -c                       # copy to clipboard
+treemapper . -c -o tree.yaml          # clipboard + save to file
+treemapper -v                         # show version
+```
+
+## Diff Context Mode
+
+Smart context selection for git diffs — automatically finds the
+minimal set of code fragments needed to understand a change:
+
+```bash
+treemapper . --diff HEAD~1..HEAD      # recent changes
+treemapper . --diff main..feature     # feature branch
+treemapper . --diff HEAD~1 --budget 30000  # limit tokens
+treemapper . --diff HEAD~1 --full     # all changed code
+```
+
+Uses graph-based relevance propagation (Personalized PageRank)
+to select the most important context. Output size is controlled
+by algorithm convergence (τ-stopping) by default, or an explicit
+`--budget` token limit. Understands imports, type references,
+config dependencies, and co-change patterns across 15+
+programming languages.
+
+Output format:
+
+```yaml
+name: myproject
+type: diff_context
+fragment_count: 5
+fragments:
+  - path: src/main.py
+    lines: "10-25"
+    kind: function
+    symbol: process_data
+    content: |
+      def process_data(items):
+          ...
+```
+
+Options:
+
+| Flag       | Default       | Description                                    |
+|------------|---------------|------------------------------------------------|
+| `--budget` | none          | Token limit (convergence-based by default)     |
+| `--alpha`  | 0.60          | PPR damping factor                             |
+| `--tau`    | 0.08          | Stopping threshold                             |
+| `--full`   | false         | Include all changed code                       |
+
+## Token Counting
+
+Token count and size are always displayed on stderr:
+
+```text
+12,847 tokens (o200k_base), 52.3 KB
+```
+
+For large outputs (>1MB), approximate counts with `~` prefix:
+
+```text
+~125,000 tokens (o200k_base), 5.2 MB
+```
+
+Uses tiktoken with `o200k_base` encoding (GPT-4o tokenizer).
+
+## Clipboard Support
+
+Copy output directly to clipboard with `-c` or `--copy`:
+
+```bash
+treemapper . -c                       # copy (no stdout)
+treemapper . -c -o tree.yaml          # copy + save to file
+```
+
+**System Requirements:**
+
+- **macOS:** `pbcopy` (pre-installed)
+- **Windows:** `clip` (pre-installed)
+- **Linux (Wayland):** `wl-copy`
+- **Linux (X11):** `xclip` or `xsel`
+
+## Python API
+
+```python
+from treemapper import map_directory
+from treemapper import to_yaml, to_json, to_text, to_markdown
+
+tree = map_directory(
+    path,                    # directory path
+    max_depth=None,          # limit traversal depth
+    no_content=False,        # exclude file contents
+    max_file_bytes=None,     # skip large files
+    ignore_file=None,        # custom ignore file
+    no_default_ignores=False,# disable default ignores
+)
+
+yaml_str = to_yaml(tree)
+json_str = to_json(tree)
+text_str = to_text(tree)
+md_str = to_markdown(tree)
+```
+
+## Ignore Patterns
+
+Respects `.gitignore` and `.treemapperignore` automatically.
+Use `--no-default-ignores` to disable all ignore processing
+(`.gitignore`, `.treemapperignore`, and built-in defaults).
+
+- Hierarchical: nested ignore files at each directory level
+- Negation patterns: `!important.log` un-ignores a file
+- Anchored patterns: `/root_only.txt` matches only in root
+- Output file is always auto-ignored
+
+## Content Placeholders
+
+- `<file too large: N bytes>` — exceeds `--max-file-bytes`
+- `<binary file: N bytes>` — binary file detected
+- `<unreadable content: not utf-8>` — not valid UTF-8
+- `<unreadable content>` — permission denied or I/O error
 
 ## Development
 
 ```bash
-pip install -e ".[dev]"
+pip install -e ".[dev,tree-sitter]"
 pytest
 pre-commit run --all-files
 ```
@@ -28,6 +203,8 @@ noise, catching regressions in relevance filtering. Each garbage
 file uses unique prefixed identifiers (e.g. `GARBAGE_*`) so leaks
 are unambiguously detectable.
 
+---
+
 ## Two Modes of Operation
 
 TreeMapper operates in two fundamentally different modes that
@@ -119,7 +296,10 @@ The engine operates as a 7-stage pipeline:
    fragments maximizing density (marginal utility per token). Core
    fragments are selected first, then expansion candidates ordered
    by a max-heap. A τ-based stopping threshold (relative to
-   baseline density median) prevents noise accumulation.
+   baseline density median) prevents noise accumulation. When no
+   explicit `--budget` is set, τ-stopping alone controls output
+   size — the algorithm converges naturally without a hard token
+   cap.
 
 ### Edge Taxonomy: Six Perspectives on Code Relationships
 
@@ -200,7 +380,7 @@ broader inclusion.
 
 Files are decomposed into semantic fragments using a
 priority-ordered parser pipeline. Language-specific parsers
-(tree-sitter for 13+ languages, Python AST, Mistune for Markdown)
+(tree-sitter for 10 languages, Python AST, Mistune for Markdown)
 produce function/class/section-level fragments. Fallback parsers
 handle config files (key-value boundaries), text (sentence-aware
 splitting), and generic content (line-count limits). The
@@ -238,12 +418,12 @@ without letting them dominate.
 
 ### Tunable Parameters
 
-| Parameter  | Default | Controls                          |
-|------------|---------|-----------------------------------|
-| `--budget` | 50000   | Maximum output tokens             |
-| `--alpha`  | 0.60    | PPR damping — broader propagation |
-| `--tau`    | 0.08    | Stopping — stricter = less noise  |
-| `--full`   | false   | Bypass smart selection            |
+| Parameter  | Default       | Controls                                       |
+|------------|---------------|------------------------------------------------|
+| `--budget` | none          | Token limit (convergence-based by default)     |
+| `--alpha`  | 0.60          | PPR damping — broader propagation              |
+| `--tau`    | 0.08          | Stopping — stricter = less noise               |
+| `--full`   | false         | Bypass smart selection                         |
 
 ---
 
@@ -254,7 +434,7 @@ without letting them dominate.
 | Output      | YAML              | LLM-readable, literal blocks |
 | Tokens      | tiktoken o200k    | GPT-4o standard, exact BPE   |
 | Ignores     | pathspec          | gitignore-compatible         |
-| Parsing     | tree-sitter       | 13+ languages, AST-level     |
+| Parsing     | tree-sitter       | 10 languages, AST-level      |
 | Ranking     | PPR               | Relevance with natural decay |
 | Selection   | Lazy greedy       | Near-optimal, linear time    |
 | Git         | subprocess UTF-8  | Platform-safe, non-ASCII     |