[skills] Add heavy file ingestion skill packages (#147)

* [skills] Add heavy file ingestion skill packages

* [skills] Harden heavy file ingestion failure handling

* [skills] Fix heavy file ingestion markdown lint

Author: justfinethanku

resources/README.md

@@ -5,3 +5,6 @@ Official companion files and packaged exports for Open Brain. Community-contribu
 | Resource | What It Is |
 | -------- | ---------- |
 | [Open Brain Companion](open-brain-companion.skill) | Claude Skill file for AI-assisted Open Brain help |
+| [Heavy File Ingestion for Claude Code](heavy-file-ingestion-claude-code.zip) | Downloadable Claude Code skill bundle with conversion script and references |
+| [Heavy File Ingestion for Codex](heavy-file-ingestion-codex.zip) | Downloadable Codex skill bundle with conversion script and references |
+| [Heavy File Ingestion for Claude Desktop](heavy-file-ingestion-claude-desktop.skill) | Claude Desktop skill package for conversion-first handling of bulky files |

resources/heavy-file-ingestion-claude-code.zip

@@ -10,6 +10,7 @@ Reusable AI client skills and prompt packs for Open Brain workflows. These are t
 | [Deal Memo Drafting Skill Pack](deal-memo-drafting/) | Turns existing diligence materials into structured deal, IC, or partnership memos | [@NateBJones](https://github.com/NateBJones) |
 | [Research Synthesis Skill Pack](research-synthesis/) | Synthesizes source sets into findings, contradictions, confidence markers, and next questions | [@NateBJones](https://github.com/NateBJones) |
 | [Meeting Synthesis Skill Pack](meeting-synthesis/) | Converts meeting notes or transcripts into decisions, action items, risks, and follow-up artifacts | [@NateBJones](https://github.com/NateBJones) |
+| [Heavy File Ingestion Skill Pack](heavy-file-ingestion/) | Converts PDFs, decks, spreadsheets, and other bulky files into markdown, CSV, and a cheap structural index before analysis | [@NateBJones](https://github.com/NateBJones) |
 | [Panning for Gold Skill Pack](panning-for-gold/) | Turns brain dumps and transcripts into evaluated idea inventories | [@jaredirish](https://github.com/jaredirish) |
 | [Claudeception Skill Pack](claudeception/) | Extracts reusable lessons from work sessions into new skills | [@jaredirish](https://github.com/jaredirish) |
 

resources/heavy-file-ingestion-claude-desktop.skill

@@ -0,0 +1,93 @@
+# Heavy File Ingestion
+
+> Convert heavyweight files into agent-friendly markdown, CSV, and a lightweight index before analysis.
+
+## What It Does
+
+Heavy File Ingestion stops agents from wasting expensive context on raw PDFs, slide decks, spreadsheets, and other bulky files. It routes each file through a deterministic conversion step first, writes a reusable artifact to disk, and creates a small index so the main agent can decide whether it even needs deeper analysis.
+
+## Supported Clients
+
+- Claude Code
+- Codex
+- Claude Desktop
+- Cursor
+- Any AI client that supports reusable skills, rules, or custom instructions and can run local scripts
+
+## Prerequisites
+
+- Python 3.10+
+- `uv` or `pip` for optional converter dependencies
+- AI client that can load a reusable skill file and run local commands
+- Working Open Brain setup if you want to pair this with Open Brain capture or retrieval flows ([guide](../../docs/01-getting-started.md))
+
+## Installation
+
+1. Copy the entire [`heavy-file-ingestion`](./) folder into a place your AI client can access, not just `SKILL.md`. The skill expects the bundled `scripts/` and `references/` folders to stay next to it.
+1. For Claude Code, place the folder at `~/.claude/skills/heavy-file-ingestion/`.
+1. For Codex or Cursor, keep the folder in your workspace or copy the contents into that client's skills or rules location.
+1. Restart or reload the client so it picks up [`SKILL.md`](./SKILL.md).
+1. When you want the deterministic converters available, run the skill script with either:
+
+```bash
+uv run \
+  --with pdfplumber \
+  --with python-docx \
+  --with python-pptx \
+  --with openpyxl \
+  python skills/heavy-file-ingestion/scripts/convert_heavy_file.py /absolute/path/to/file.pdf
+```
+
+1. If you already have `markitdown` installed and want to prefer it for rich document conversion, add `--prefer markitdown`.
+
+## Downloadable Variants
+
+If you want packaged client-specific downloads instead of the raw source folder, use:
+
+- Claude Code: [../../resources/heavy-file-ingestion-claude-code.zip](../../resources/heavy-file-ingestion-claude-code.zip)
+- Codex: [../../resources/heavy-file-ingestion-codex.zip](../../resources/heavy-file-ingestion-codex.zip)
+- Claude Desktop: [../../resources/heavy-file-ingestion-claude-desktop.skill](../../resources/heavy-file-ingestion-claude-desktop.skill)
+
+The Claude Code and Codex downloads include the bundled `scripts/` and `references/` directories. The Claude Desktop `.skill` is intentionally lighter because Claude Desktop is better treated as a policy layer than a local conversion runtime.
+
+## Trigger Conditions
+
+- The user asks the agent to "read," "analyze," "summarize," or "extract from" a PDF, DOCX, PPTX, XLSX, CSV, TSV, or other large file
+- The file is big enough or structured enough that raw ingestion would burn unnecessary tokens
+- The user wants a reusable markdown version, a CSV normalization step, or a quick structural index before analysis
+- The agent needs to know whether the file can be handled deterministically or should escalate to a cheap model fallback
+
+## Expected Outcome
+
+When the skill is working correctly, it should:
+
+- Detect that the file should be converted before the main model reads it
+- Write an extracted artifact to disk instead of pushing raw content into context
+- Create an `index.md` and `index.json` summary with counts, structure hints, preview lines, and quality flags
+- Recommend the cheapest safe next step: use the deterministic artifact, escalate to a small model, or retry with a stronger converter
+
+## Open Source Stack
+
+This skill was shaped around a small set of open-source projects with permissive licensing:
+
+- [Microsoft MarkItDown](https://github.com/microsoft/markitdown) for broad document-to-markdown conversion
+- [Docling](https://github.com/docling-project/docling) as the heavy-duty fallback for ugly or scanned documents
+- [xlsx2csv](https://github.com/dilshod/xlsx2csv) as the reference pattern for spreadsheet normalization
+- [pdfplumber](https://github.com/jsvine/pdfplumber) as the reference pattern for cheap PDF indexing and page-level extraction
+
+More detail lives in [`references/open-source-stack.md`](./references/open-source-stack.md).
+
+## Troubleshooting
+
+**Issue:** The extracted PDF markdown is sparse or missing whole pages.  
+Solution: Check the generated `index.md`. If it flags `scanned_pdf_suspected` or `low_text_density`, rerun with a stronger converter or use a cheap model only on the extracted artifact, not on the original PDF.
+
+**Issue:** The script says a dependency is missing.  
+Solution: Use the `uv run --with ...` command from this README or install the named package with `pip`.
+
+**Issue:** The client can load the skill text but cannot run scripts.  
+Solution: Use the skill as policy guidance only and run the bundled script manually from Terminal. The skill is still useful because it tells the main agent when not to read a raw heavyweight file.
+
+## Notes for Other Clients
+
+If your client only supports a single prompt file, paste the contents of [`SKILL.md`](./SKILL.md) into that client and keep the script path nearby. The reusable behavior is the policy: convert first, inspect the index second, and only spend model tokens on the compressed artifact.

resources/heavy-file-ingestion-codex.zip

@@ -0,0 +1,75 @@
+---
+name: heavy-file-ingestion
+description: Use when a user asks to read, analyze, summarize, or extract from a heavyweight file such as PDF, DOCX, PPTX, XLSX, CSV, or TSV. Convert the file into markdown or CSV first, generate a lightweight index, and only spend model tokens on the compressed artifact. Trigger on requests like "read this PDF", "look through this spreadsheet", "summarize this deck", or any time raw file ingestion would waste tokens.
+author: Nate B. Jones
+version: 1.0.0
+---
+
+# Heavy File Ingestion
+
+## Problem
+
+Agents waste money and context when they read heavyweight files raw. This skill turns bulky documents into cheaper working artifacts first, then tells the main agent how much reasoning power the file actually deserves.
+
+## Trigger Conditions
+
+- The user asks to read or analyze a PDF, slide deck, spreadsheet, or word-processing file
+- The file is large, structured, or expensive enough that raw ingestion is a bad trade
+- The user wants a markdown working copy, CSV extraction, or a quick map of the file before analysis
+- The agent needs a deterministic first pass before choosing whether a model fallback is worth the cost
+
+## Core Policy
+
+1. **Convert before reading.** Do not dump raw heavyweight files into model context if a deterministic converter can create a cheaper artifact.
+1. **Index before reasoning.** Read the generated `index.md` or `index.json` first. It should tell you what is in the file, how clean the extraction was, and whether escalation is justified.
+1. **Match the converter to the file type.**
+   - PDFs and documents: markdown artifact
+   - Presentations: markdown slide outline
+   - Spreadsheets: CSV per sheet plus a markdown manifest
+1. **Escalate by cost tier, not instinct.**
+   - Tier 1: deterministic converter plus index
+   - Tier 2: cheap model on the extracted artifact only if quality flags say the deterministic pass lost structure
+   - Tier 3: expensive model only after the file has already been compressed into markdown, CSV, or a sampled subset
+
+## Process
+
+1. Identify the file path, extension, and rough size.
+1. Run the converter script instead of reading the original file directly:
+
+```bash
+uv run \
+  --with pdfplumber \
+  --with python-docx \
+  --with python-pptx \
+  --with openpyxl \
+  python skills/heavy-file-ingestion/scripts/convert_heavy_file.py /absolute/path/to/file.ext
+```
+
+1. If you already have `markitdown` installed and want to prefer it for PDF or DOCX conversion, rerun with:
+
+```bash
+python skills/heavy-file-ingestion/scripts/convert_heavy_file.py /absolute/path/to/file.ext --prefer markitdown
+```
+
+1. Read the generated `index.md` first.
+2. Only read the extracted markdown or CSV outputs that the index says are worth reading.
+3. If the index flags weak extraction, use a cheap fallback:
+   - Try an alternate deterministic converter
+   - Use a small model to rebuild only the structure or outline from the extracted artifact
+   - Escalate to a stronger model only when the cheaper passes still leave critical ambiguity
+
+## Output
+
+The skill should leave behind:
+
+- A deterministic artifact the agent can work from
+- `index.md` with file counts, structure hints, preview lines, and a recommended next step
+- `index.json` with the same information in machine-friendly form
+- Warnings when the deterministic pass is not trustworthy enough for direct reasoning
+
+## Notes
+
+- Prefer the bundled script over rewriting ad hoc conversion code each time.
+- Do not treat "sub-agent" as the default answer to messy files. A cheap deterministic pass beats a cheap model when the task is conversion, counting, routing, or indexing.
+- For scanned PDFs, image-heavy decks, or bizarre layouts, the deterministic pass is still useful because it tells you that a fallback is needed before you waste a stronger model on the original file.
+- Use [`references/open-source-stack.md`](./references/open-source-stack.md) when you need to choose a better extractor or explain why one was picked.

skills/README.md

@@ -0,0 +1,20 @@
+{
+  "name": "Heavy File Ingestion",
+  "description": "Converts heavyweight files such as PDFs, slide decks, spreadsheets, and documents into markdown, CSV, and lightweight indexes before an agent spends model tokens on them.",
+  "category": "skills",
+  "author": {
+    "name": "Nate B. Jones",
+    "github": "NateBJones"
+  },
+  "version": "1.0.0",
+  "requires": {
+    "open_brain": true,
+    "services": [],
+    "tools": ["Python 3.10+", "uv or pip", "AI client with reusable skills/prompts"]
+  },
+  "tags": ["skill", "file-conversion", "markdown", "pdf", "spreadsheet", "token-efficiency"],
+  "difficulty": "intermediate",
+  "estimated_time": "10 minutes",
+  "created": "2026-03-31",
+  "updated": "2026-03-31"
+}

skills/heavy-file-ingestion/README.md

@@ -0,0 +1,53 @@
+# Open Source Stack Notes
+
+This skill uses a deterministic-first policy and keeps the tool stack small on purpose. The goal is not perfect document fidelity. The goal is to create an agent-friendly artifact cheaply enough that the main model only sees the compressed version.
+
+## Recommended Roles
+
+### 1. MarkItDown
+
+- Repo: <https://github.com/microsoft/markitdown>
+- License: MIT
+- Role in this skill: Best general-purpose document-to-markdown converter for PDFs, DOCX, PPTX, and mixed office-style documents when you want broad coverage fast.
+- Why it fits: It is explicitly designed to make documents easier for LLM workflows rather than chasing layout-perfect export.
+- Why it is not the only tool here: It can pull in bigger dependency trees than we want for every single file, especially when a sheet or deck can be normalized more cheaply with a tiny native extractor.
+
+### 2. Docling
+
+- Repo: <https://github.com/docling-project/docling>
+- License: MIT
+- Role in this skill: Heavy-duty fallback for ugly PDFs, OCR-heavy files, layout-sensitive extraction, and advanced document recovery.
+- Why it fits: Strong PDF understanding, OCR support, and multi-format export, including markdown.
+- Why it is not the default: It is overkill for cheap first-pass routing and raises the operational footprint.
+
+### 3. xlsx2csv
+
+- Repo: <https://github.com/dilshod/xlsx2csv>
+- License: MIT
+- Role in this skill: Reference pattern for spreadsheet normalization.
+- Why it fits: The right mental model for spreadsheets is usually "convert each sheet into a plain tabular artifact" rather than forcing the main model to inspect workbook internals.
+- How this skill uses the idea: Native spreadsheet handling creates one CSV per sheet plus a markdown manifest with sheet counts, headers, and row estimates.
+
+### 4. pdfplumber
+
+- Repo: <https://github.com/jsvine/pdfplumber>
+- License: MIT
+- Role in this skill: Cheap PDF indexing and page-level extraction.
+- Why it fits: Good for counts, per-page text, page density checks, and detecting when a PDF is likely scanned or image-heavy.
+- Why it matters: Even when the markdown extraction is weak, a cheap page-level index still tells the main agent whether escalation is worth the money.
+
+## Architecture Decision
+
+Use the smallest tool that preserves the structure the agent actually needs:
+
+1. Tabular files: native CSV normalization first
+2. Slide decks: native slide-outline extraction first
+3. PDFs and rich documents: native extraction or MarkItDown
+4. Scanned or degraded files: Docling or a cheap model only after the deterministic pass proves it is necessary
+
+## What We Are Avoiding
+
+- Reading raw binary-heavy files directly in the main model
+- Defaulting to expensive models for pure conversion work
+- Forcing a single converter to own every file type
+- Building a pipeline so "smart" that a solo operator cannot debug it six months later

skills/heavy-file-ingestion/SKILL.md

@@ -0,0 +1,80 @@
+#!/usr/bin/env python3
+from __future__ import annotations
+
+import shutil
+import zipfile
+from pathlib import Path
+
+
+ROOT = Path(__file__).resolve().parents[1]
+REPO_ROOT = ROOT.parents[1]
+RESOURCES_DIR = REPO_ROOT / "resources"
+VARIANTS_DIR = ROOT / "variants"
+
+
+def reset_dir(path: Path) -> None:
+    if path.exists():
+        shutil.rmtree(path)
+    path.mkdir(parents=True, exist_ok=True)
+
+
+def copy_tree(src: Path, dst: Path) -> None:
+    dst.mkdir(parents=True, exist_ok=True)
+    for item in src.iterdir():
+        target = dst / item.name
+        if item.is_dir():
+            shutil.copytree(item, target, dirs_exist_ok=True)
+        else:
+            shutil.copy2(item, target)
+
+
+def build_zip_from_dir(source_dir: Path, archive_path: Path) -> None:
+    with zipfile.ZipFile(archive_path, "w", compression=zipfile.ZIP_DEFLATED) as zf:
+        for path in sorted(source_dir.rglob("*")):
+            if path.is_file():
+                zf.write(path, path.relative_to(source_dir.parent))
+
+
+def build_exports() -> list[Path]:
+    build_root = ROOT / ".build-exports"
+    reset_dir(build_root)
+    RESOURCES_DIR.mkdir(parents=True, exist_ok=True)
+
+    created: list[Path] = []
+
+    code_bundle = build_root / "heavy-file-ingestion-claude-code"
+    copy_tree(VARIANTS_DIR / "claude-code", code_bundle)
+    (code_bundle / "scripts").mkdir(parents=True, exist_ok=True)
+    shutil.copy2(ROOT / "scripts" / "convert_heavy_file.py", code_bundle / "scripts" / "convert_heavy_file.py")
+    copy_tree(ROOT / "references", code_bundle / "references")
+    claude_code_zip = RESOURCES_DIR / "heavy-file-ingestion-claude-code.zip"
+    build_zip_from_dir(code_bundle, claude_code_zip)
+    created.append(claude_code_zip)
+
+    codex_bundle = build_root / "heavy-file-ingestion-codex"
+    copy_tree(VARIANTS_DIR / "codex", codex_bundle)
+    (codex_bundle / "scripts").mkdir(parents=True, exist_ok=True)
+    shutil.copy2(ROOT / "scripts" / "convert_heavy_file.py", codex_bundle / "scripts" / "convert_heavy_file.py")
+    copy_tree(ROOT / "references", codex_bundle / "references")
+    codex_zip = RESOURCES_DIR / "heavy-file-ingestion-codex.zip"
+    build_zip_from_dir(codex_bundle, codex_zip)
+    created.append(codex_zip)
+
+    desktop_bundle = build_root / "heavy-file-ingestion-claude-desktop"
+    copy_tree(VARIANTS_DIR / "claude-desktop", desktop_bundle)
+    desktop_skill = RESOURCES_DIR / "heavy-file-ingestion-claude-desktop.skill"
+    build_zip_from_dir(desktop_bundle, desktop_skill)
+    created.append(desktop_skill)
+
+    return created
+
+
+def main() -> int:
+    created = build_exports()
+    for path in created:
+        print(path)
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())