diff --git a/resources/README.md b/resources/README.md index bb27ecd8..85eb35b1 100644 --- a/resources/README.md +++ b/resources/README.md @@ -5,3 +5,6 @@ Official companion files and packaged exports for Open Brain. Community-contribu | Resource | What It Is | | -------- | ---------- | | [Open Brain Companion](open-brain-companion.skill) | Claude Skill file for AI-assisted Open Brain help | +| [Heavy File Ingestion for Claude Code](heavy-file-ingestion-claude-code.zip) | Downloadable Claude Code skill bundle with conversion script and references | +| [Heavy File Ingestion for Codex](heavy-file-ingestion-codex.zip) | Downloadable Codex skill bundle with conversion script and references | +| [Heavy File Ingestion for Claude Desktop](heavy-file-ingestion-claude-desktop.skill) | Claude Desktop skill package for conversion-first handling of bulky files | diff --git a/resources/heavy-file-ingestion-claude-code.zip b/resources/heavy-file-ingestion-claude-code.zip new file mode 100644 index 00000000..ce18a82c Binary files /dev/null and b/resources/heavy-file-ingestion-claude-code.zip differ diff --git a/resources/heavy-file-ingestion-claude-desktop.skill b/resources/heavy-file-ingestion-claude-desktop.skill new file mode 100644 index 00000000..a7f5d95e Binary files /dev/null and b/resources/heavy-file-ingestion-claude-desktop.skill differ diff --git a/resources/heavy-file-ingestion-codex.zip b/resources/heavy-file-ingestion-codex.zip new file mode 100644 index 00000000..ea47b699 Binary files /dev/null and b/resources/heavy-file-ingestion-codex.zip differ diff --git a/skills/README.md b/skills/README.md index 909da2d2..079d5bea 100644 --- a/skills/README.md +++ b/skills/README.md @@ -10,6 +10,7 @@ Reusable AI client skills and prompt packs for Open Brain workflows. These are t | [Deal Memo Drafting Skill Pack](deal-memo-drafting/) | Turns existing diligence materials into structured deal, IC, or partnership memos | [@NateBJones](https://github.com/NateBJones) | | [Research Synthesis Skill Pack](research-synthesis/) | Synthesizes source sets into findings, contradictions, confidence markers, and next questions | [@NateBJones](https://github.com/NateBJones) | | [Meeting Synthesis Skill Pack](meeting-synthesis/) | Converts meeting notes or transcripts into decisions, action items, risks, and follow-up artifacts | [@NateBJones](https://github.com/NateBJones) | +| [Heavy File Ingestion Skill Pack](heavy-file-ingestion/) | Converts PDFs, decks, spreadsheets, and other bulky files into markdown, CSV, and a cheap structural index before analysis | [@NateBJones](https://github.com/NateBJones) | | [Panning for Gold Skill Pack](panning-for-gold/) | Turns brain dumps and transcripts into evaluated idea inventories | [@jaredirish](https://github.com/jaredirish) | | [Claudeception Skill Pack](claudeception/) | Extracts reusable lessons from work sessions into new skills | [@jaredirish](https://github.com/jaredirish) | diff --git a/skills/heavy-file-ingestion/README.md b/skills/heavy-file-ingestion/README.md new file mode 100644 index 00000000..a35294e1 --- /dev/null +++ b/skills/heavy-file-ingestion/README.md @@ -0,0 +1,93 @@ +# Heavy File Ingestion + +> Convert heavyweight files into agent-friendly markdown, CSV, and a lightweight index before analysis. + +## What It Does + +Heavy File Ingestion stops agents from wasting expensive context on raw PDFs, slide decks, spreadsheets, and other bulky files. It routes each file through a deterministic conversion step first, writes a reusable artifact to disk, and creates a small index so the main agent can decide whether it even needs deeper analysis. + +## Supported Clients + +- Claude Code +- Codex +- Claude Desktop +- Cursor +- Any AI client that supports reusable skills, rules, or custom instructions and can run local scripts + +## Prerequisites + +- Python 3.10+ +- `uv` or `pip` for optional converter dependencies +- AI client that can load a reusable skill file and run local commands +- Working Open Brain setup if you want to pair this with Open Brain capture or retrieval flows ([guide](../../docs/01-getting-started.md)) + +## Installation + +1. Copy the entire [`heavy-file-ingestion`](./) folder into a place your AI client can access, not just `SKILL.md`. The skill expects the bundled `scripts/` and `references/` folders to stay next to it. +1. For Claude Code, place the folder at `~/.claude/skills/heavy-file-ingestion/`. +1. For Codex or Cursor, keep the folder in your workspace or copy the contents into that client's skills or rules location. +1. Restart or reload the client so it picks up [`SKILL.md`](./SKILL.md). +1. When you want the deterministic converters available, run the skill script with either: + +```bash +uv run \ + --with pdfplumber \ + --with python-docx \ + --with python-pptx \ + --with openpyxl \ + python skills/heavy-file-ingestion/scripts/convert_heavy_file.py /absolute/path/to/file.pdf +``` + +1. If you already have `markitdown` installed and want to prefer it for rich document conversion, add `--prefer markitdown`. + +## Downloadable Variants + +If you want packaged client-specific downloads instead of the raw source folder, use: + +- Claude Code: [../../resources/heavy-file-ingestion-claude-code.zip](../../resources/heavy-file-ingestion-claude-code.zip) +- Codex: [../../resources/heavy-file-ingestion-codex.zip](../../resources/heavy-file-ingestion-codex.zip) +- Claude Desktop: [../../resources/heavy-file-ingestion-claude-desktop.skill](../../resources/heavy-file-ingestion-claude-desktop.skill) + +The Claude Code and Codex downloads include the bundled `scripts/` and `references/` directories. The Claude Desktop `.skill` is intentionally lighter because Claude Desktop is better treated as a policy layer than a local conversion runtime. + +## Trigger Conditions + +- The user asks the agent to "read," "analyze," "summarize," or "extract from" a PDF, DOCX, PPTX, XLSX, CSV, TSV, or other large file +- The file is big enough or structured enough that raw ingestion would burn unnecessary tokens +- The user wants a reusable markdown version, a CSV normalization step, or a quick structural index before analysis +- The agent needs to know whether the file can be handled deterministically or should escalate to a cheap model fallback + +## Expected Outcome + +When the skill is working correctly, it should: + +- Detect that the file should be converted before the main model reads it +- Write an extracted artifact to disk instead of pushing raw content into context +- Create an `index.md` and `index.json` summary with counts, structure hints, preview lines, and quality flags +- Recommend the cheapest safe next step: use the deterministic artifact, escalate to a small model, or retry with a stronger converter + +## Open Source Stack + +This skill was shaped around a small set of open-source projects with permissive licensing: + +- [Microsoft MarkItDown](https://github.com/microsoft/markitdown) for broad document-to-markdown conversion +- [Docling](https://github.com/docling-project/docling) as the heavy-duty fallback for ugly or scanned documents +- [xlsx2csv](https://github.com/dilshod/xlsx2csv) as the reference pattern for spreadsheet normalization +- [pdfplumber](https://github.com/jsvine/pdfplumber) as the reference pattern for cheap PDF indexing and page-level extraction + +More detail lives in [`references/open-source-stack.md`](./references/open-source-stack.md). + +## Troubleshooting + +**Issue:** The extracted PDF markdown is sparse or missing whole pages. +Solution: Check the generated `index.md`. If it flags `scanned_pdf_suspected` or `low_text_density`, rerun with a stronger converter or use a cheap model only on the extracted artifact, not on the original PDF. + +**Issue:** The script says a dependency is missing. +Solution: Use the `uv run --with ...` command from this README or install the named package with `pip`. + +**Issue:** The client can load the skill text but cannot run scripts. +Solution: Use the skill as policy guidance only and run the bundled script manually from Terminal. The skill is still useful because it tells the main agent when not to read a raw heavyweight file. + +## Notes for Other Clients + +If your client only supports a single prompt file, paste the contents of [`SKILL.md`](./SKILL.md) into that client and keep the script path nearby. The reusable behavior is the policy: convert first, inspect the index second, and only spend model tokens on the compressed artifact. diff --git a/skills/heavy-file-ingestion/SKILL.md b/skills/heavy-file-ingestion/SKILL.md new file mode 100644 index 00000000..0dc201f9 --- /dev/null +++ b/skills/heavy-file-ingestion/SKILL.md @@ -0,0 +1,75 @@ +--- +name: heavy-file-ingestion +description: Use when a user asks to read, analyze, summarize, or extract from a heavyweight file such as PDF, DOCX, PPTX, XLSX, CSV, or TSV. Convert the file into markdown or CSV first, generate a lightweight index, and only spend model tokens on the compressed artifact. Trigger on requests like "read this PDF", "look through this spreadsheet", "summarize this deck", or any time raw file ingestion would waste tokens. +author: Nate B. Jones +version: 1.0.0 +--- + +# Heavy File Ingestion + +## Problem + +Agents waste money and context when they read heavyweight files raw. This skill turns bulky documents into cheaper working artifacts first, then tells the main agent how much reasoning power the file actually deserves. + +## Trigger Conditions + +- The user asks to read or analyze a PDF, slide deck, spreadsheet, or word-processing file +- The file is large, structured, or expensive enough that raw ingestion is a bad trade +- The user wants a markdown working copy, CSV extraction, or a quick map of the file before analysis +- The agent needs a deterministic first pass before choosing whether a model fallback is worth the cost + +## Core Policy + +1. **Convert before reading.** Do not dump raw heavyweight files into model context if a deterministic converter can create a cheaper artifact. +1. **Index before reasoning.** Read the generated `index.md` or `index.json` first. It should tell you what is in the file, how clean the extraction was, and whether escalation is justified. +1. **Match the converter to the file type.** + - PDFs and documents: markdown artifact + - Presentations: markdown slide outline + - Spreadsheets: CSV per sheet plus a markdown manifest +1. **Escalate by cost tier, not instinct.** + - Tier 1: deterministic converter plus index + - Tier 2: cheap model on the extracted artifact only if quality flags say the deterministic pass lost structure + - Tier 3: expensive model only after the file has already been compressed into markdown, CSV, or a sampled subset + +## Process + +1. Identify the file path, extension, and rough size. +1. Run the converter script instead of reading the original file directly: + +```bash +uv run \ + --with pdfplumber \ + --with python-docx \ + --with python-pptx \ + --with openpyxl \ + python skills/heavy-file-ingestion/scripts/convert_heavy_file.py /absolute/path/to/file.ext +``` + +1. If you already have `markitdown` installed and want to prefer it for PDF or DOCX conversion, rerun with: + +```bash +python skills/heavy-file-ingestion/scripts/convert_heavy_file.py /absolute/path/to/file.ext --prefer markitdown +``` + +1. Read the generated `index.md` first. +2. Only read the extracted markdown or CSV outputs that the index says are worth reading. +3. If the index flags weak extraction, use a cheap fallback: + - Try an alternate deterministic converter + - Use a small model to rebuild only the structure or outline from the extracted artifact + - Escalate to a stronger model only when the cheaper passes still leave critical ambiguity + +## Output + +The skill should leave behind: + +- A deterministic artifact the agent can work from +- `index.md` with file counts, structure hints, preview lines, and a recommended next step +- `index.json` with the same information in machine-friendly form +- Warnings when the deterministic pass is not trustworthy enough for direct reasoning + +## Notes + +- Prefer the bundled script over rewriting ad hoc conversion code each time. +- Do not treat "sub-agent" as the default answer to messy files. A cheap deterministic pass beats a cheap model when the task is conversion, counting, routing, or indexing. +- For scanned PDFs, image-heavy decks, or bizarre layouts, the deterministic pass is still useful because it tells you that a fallback is needed before you waste a stronger model on the original file. +- Use [`references/open-source-stack.md`](./references/open-source-stack.md) when you need to choose a better extractor or explain why one was picked. diff --git a/skills/heavy-file-ingestion/metadata.json b/skills/heavy-file-ingestion/metadata.json new file mode 100644 index 00000000..e62102af --- /dev/null +++ b/skills/heavy-file-ingestion/metadata.json @@ -0,0 +1,20 @@ +{ + "name": "Heavy File Ingestion", + "description": "Converts heavyweight files such as PDFs, slide decks, spreadsheets, and documents into markdown, CSV, and lightweight indexes before an agent spends model tokens on them.", + "category": "skills", + "author": { + "name": "Nate B. Jones", + "github": "NateBJones" + }, + "version": "1.0.0", + "requires": { + "open_brain": true, + "services": [], + "tools": ["Python 3.10+", "uv or pip", "AI client with reusable skills/prompts"] + }, + "tags": ["skill", "file-conversion", "markdown", "pdf", "spreadsheet", "token-efficiency"], + "difficulty": "intermediate", + "estimated_time": "10 minutes", + "created": "2026-03-31", + "updated": "2026-03-31" +} diff --git a/skills/heavy-file-ingestion/references/open-source-stack.md b/skills/heavy-file-ingestion/references/open-source-stack.md new file mode 100644 index 00000000..91c1e7c2 --- /dev/null +++ b/skills/heavy-file-ingestion/references/open-source-stack.md @@ -0,0 +1,53 @@ +# Open Source Stack Notes + +This skill uses a deterministic-first policy and keeps the tool stack small on purpose. The goal is not perfect document fidelity. The goal is to create an agent-friendly artifact cheaply enough that the main model only sees the compressed version. + +## Recommended Roles + +### 1. MarkItDown + +- Repo: +- License: MIT +- Role in this skill: Best general-purpose document-to-markdown converter for PDFs, DOCX, PPTX, and mixed office-style documents when you want broad coverage fast. +- Why it fits: It is explicitly designed to make documents easier for LLM workflows rather than chasing layout-perfect export. +- Why it is not the only tool here: It can pull in bigger dependency trees than we want for every single file, especially when a sheet or deck can be normalized more cheaply with a tiny native extractor. + +### 2. Docling + +- Repo: +- License: MIT +- Role in this skill: Heavy-duty fallback for ugly PDFs, OCR-heavy files, layout-sensitive extraction, and advanced document recovery. +- Why it fits: Strong PDF understanding, OCR support, and multi-format export, including markdown. +- Why it is not the default: It is overkill for cheap first-pass routing and raises the operational footprint. + +### 3. xlsx2csv + +- Repo: +- License: MIT +- Role in this skill: Reference pattern for spreadsheet normalization. +- Why it fits: The right mental model for spreadsheets is usually "convert each sheet into a plain tabular artifact" rather than forcing the main model to inspect workbook internals. +- How this skill uses the idea: Native spreadsheet handling creates one CSV per sheet plus a markdown manifest with sheet counts, headers, and row estimates. + +### 4. pdfplumber + +- Repo: +- License: MIT +- Role in this skill: Cheap PDF indexing and page-level extraction. +- Why it fits: Good for counts, per-page text, page density checks, and detecting when a PDF is likely scanned or image-heavy. +- Why it matters: Even when the markdown extraction is weak, a cheap page-level index still tells the main agent whether escalation is worth the money. + +## Architecture Decision + +Use the smallest tool that preserves the structure the agent actually needs: + +1. Tabular files: native CSV normalization first +2. Slide decks: native slide-outline extraction first +3. PDFs and rich documents: native extraction or MarkItDown +4. Scanned or degraded files: Docling or a cheap model only after the deterministic pass proves it is necessary + +## What We Are Avoiding + +- Reading raw binary-heavy files directly in the main model +- Defaulting to expensive models for pure conversion work +- Forcing a single converter to own every file type +- Building a pipeline so "smart" that a solo operator cannot debug it six months later diff --git a/skills/heavy-file-ingestion/scripts/build_client_exports.py b/skills/heavy-file-ingestion/scripts/build_client_exports.py new file mode 100644 index 00000000..a1f4ba12 --- /dev/null +++ b/skills/heavy-file-ingestion/scripts/build_client_exports.py @@ -0,0 +1,80 @@ +#!/usr/bin/env python3 +from __future__ import annotations + +import shutil +import zipfile +from pathlib import Path + + +ROOT = Path(__file__).resolve().parents[1] +REPO_ROOT = ROOT.parents[1] +RESOURCES_DIR = REPO_ROOT / "resources" +VARIANTS_DIR = ROOT / "variants" + + +def reset_dir(path: Path) -> None: + if path.exists(): + shutil.rmtree(path) + path.mkdir(parents=True, exist_ok=True) + + +def copy_tree(src: Path, dst: Path) -> None: + dst.mkdir(parents=True, exist_ok=True) + for item in src.iterdir(): + target = dst / item.name + if item.is_dir(): + shutil.copytree(item, target, dirs_exist_ok=True) + else: + shutil.copy2(item, target) + + +def build_zip_from_dir(source_dir: Path, archive_path: Path) -> None: + with zipfile.ZipFile(archive_path, "w", compression=zipfile.ZIP_DEFLATED) as zf: + for path in sorted(source_dir.rglob("*")): + if path.is_file(): + zf.write(path, path.relative_to(source_dir.parent)) + + +def build_exports() -> list[Path]: + build_root = ROOT / ".build-exports" + reset_dir(build_root) + RESOURCES_DIR.mkdir(parents=True, exist_ok=True) + + created: list[Path] = [] + + code_bundle = build_root / "heavy-file-ingestion-claude-code" + copy_tree(VARIANTS_DIR / "claude-code", code_bundle) + (code_bundle / "scripts").mkdir(parents=True, exist_ok=True) + shutil.copy2(ROOT / "scripts" / "convert_heavy_file.py", code_bundle / "scripts" / "convert_heavy_file.py") + copy_tree(ROOT / "references", code_bundle / "references") + claude_code_zip = RESOURCES_DIR / "heavy-file-ingestion-claude-code.zip" + build_zip_from_dir(code_bundle, claude_code_zip) + created.append(claude_code_zip) + + codex_bundle = build_root / "heavy-file-ingestion-codex" + copy_tree(VARIANTS_DIR / "codex", codex_bundle) + (codex_bundle / "scripts").mkdir(parents=True, exist_ok=True) + shutil.copy2(ROOT / "scripts" / "convert_heavy_file.py", codex_bundle / "scripts" / "convert_heavy_file.py") + copy_tree(ROOT / "references", codex_bundle / "references") + codex_zip = RESOURCES_DIR / "heavy-file-ingestion-codex.zip" + build_zip_from_dir(codex_bundle, codex_zip) + created.append(codex_zip) + + desktop_bundle = build_root / "heavy-file-ingestion-claude-desktop" + copy_tree(VARIANTS_DIR / "claude-desktop", desktop_bundle) + desktop_skill = RESOURCES_DIR / "heavy-file-ingestion-claude-desktop.skill" + build_zip_from_dir(desktop_bundle, desktop_skill) + created.append(desktop_skill) + + return created + + +def main() -> int: + created = build_exports() + for path in created: + print(path) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/skills/heavy-file-ingestion/scripts/convert_heavy_file.py b/skills/heavy-file-ingestion/scripts/convert_heavy_file.py new file mode 100644 index 00000000..d76614b5 --- /dev/null +++ b/skills/heavy-file-ingestion/scripts/convert_heavy_file.py @@ -0,0 +1,609 @@ +#!/usr/bin/env python3 +from __future__ import annotations + +import argparse +import csv +import importlib +import json +import mimetypes +import os +import re +import shutil +import subprocess +import sys +from dataclasses import dataclass, field +from pathlib import Path + + +MARKITDOWN_SPEC = "markitdown[pdf,docx,pptx,xlsx]" +PREVIEW_LINE_LIMIT = 12 +PREVIEW_CHAR_LIMIT = 160 + + +@dataclass +class Artifact: + path: str + kind: str + description: str + + +@dataclass +class ConversionResult: + source: Path + output_dir: Path + converter: str = "" + artifacts: list[Artifact] = field(default_factory=list) + warnings: list[str] = field(default_factory=list) + quality_flags: list[str] = field(default_factory=list) + stats: dict[str, object] = field(default_factory=dict) + + +def parse_args() -> argparse.Namespace: + parser = argparse.ArgumentParser( + description="Convert heavyweight files into markdown, CSV, and an index." + ) + parser.add_argument("source", type=Path, help="Path to the source file") + parser.add_argument( + "--output-dir", + type=Path, + help="Directory for extracted outputs. Defaults to .ob1/", + ) + parser.add_argument( + "--prefer", + choices=["auto", "native", "markitdown"], + default="auto", + help="Preferred converter strategy for supported formats", + ) + return parser.parse_args() + + +def slugify(value: str) -> str: + cleaned = re.sub(r"[^A-Za-z0-9]+", "-", value.strip().lower()).strip("-") + return cleaned or "sheet" + + +def require_module(module_name: str, package_name: str): + try: + return importlib.import_module(module_name) + except ImportError as exc: + raise RuntimeError( + f"Missing dependency '{package_name}'. " + f"Install it with `uv run --with {package_name} ...` or `pip install {package_name}`." + ) from exc + + +def write_text(path: Path, content: str) -> None: + path.parent.mkdir(parents=True, exist_ok=True) + path.write_text(content, encoding="utf-8") + + +def relpath(path: Path, base: Path) -> str: + try: + return str(path.relative_to(base)) + except ValueError: + return str(path) + + +def human_size(num_bytes: int) -> str: + size = float(num_bytes) + for unit in ["B", "KB", "MB", "GB"]: + if size < 1024 or unit == "GB": + if unit == "B": + return f"{int(size)} {unit}" + return f"{size:.1f} {unit}" + size /= 1024 + return f"{num_bytes} B" + + +def clean_preview_line(line: str) -> str: + line = re.sub(r"\s+", " ", line).strip() + if len(line) > PREVIEW_CHAR_LIMIT: + return f"{line[: PREVIEW_CHAR_LIMIT - 1]}…" + return line + + +def gather_preview_lines(path: Path) -> list[str]: + if not path.exists() or not path.is_file(): + return [] + if path.suffix.lower() not in {".md", ".txt", ".csv", ".tsv"}: + return [] + + previews: list[str] = [] + for raw_line in path.read_text(encoding="utf-8", errors="replace").splitlines(): + line = clean_preview_line(raw_line) + if not line: + continue + previews.append(line) + if len(previews) >= PREVIEW_LINE_LIMIT: + break + return previews + + +def infer_next_step(result: ConversionResult) -> str: + if "dependency_missing" in result.quality_flags: + return "install_dependency_and_retry" + if "conversion_failed" in result.quality_flags: + return "manual_review" + if {"scanned_pdf_suspected", "low_text_density"} & set(result.quality_flags): + return "cheap_model_or_stronger_converter" + if "low_text_output" in result.quality_flags and result.source.suffix.lower() in { + ".pdf", + ".docx", + ".pptx", + ".xlsx", + }: + return "cheap_model_or_stronger_converter" + return "read_extracted_artifact" + + +def build_index_markdown(result: ConversionResult) -> str: + source_type = result.source.suffix.lower().lstrip(".") or "unknown" + lines = [ + f"# Heavy File Index: {result.source.name}", + "", + f"- Source path: `{result.source}`", + f"- Source type: `{source_type}`", + f"- Source size: `{human_size(result.source.stat().st_size)}`", + f"- Converter: `{result.converter or 'none'}`", + f"- Recommended next step: `{infer_next_step(result)}`", + ] + + if result.quality_flags: + lines.append(f"- Quality flags: `{', '.join(result.quality_flags)}`") + if result.warnings: + lines.append(f"- Warnings: `{'; '.join(result.warnings)}`") + + lines.extend(["", "## Artifacts", ""]) + for artifact in result.artifacts: + lines.append(f"- `{artifact.kind}`: `{relpath(Path(artifact.path), result.output_dir)}` — {artifact.description}") + + if result.stats: + lines.extend(["", "## Stats", ""]) + for key, value in sorted(result.stats.items()): + lines.append(f"- `{key}`: `{value}`") + + previews: list[str] = [] + for artifact in result.artifacts: + previews.extend(gather_preview_lines(Path(artifact.path))) + if len(previews) >= PREVIEW_LINE_LIMIT: + break + + if previews: + lines.extend(["", "## Preview", ""]) + for index, preview in enumerate(previews[:PREVIEW_LINE_LIMIT], start=1): + lines.append(f"{index}. {preview}") + + return "\n".join(lines) + "\n" + + +def write_index_files(result: ConversionResult) -> None: + index_payload = { + "source": str(result.source), + "source_name": result.source.name, + "mime_type": mimetypes.guess_type(result.source.name)[0] or "application/octet-stream", + "source_size_bytes": result.source.stat().st_size, + "converter": result.converter, + "artifacts": [artifact.__dict__ for artifact in result.artifacts], + "warnings": result.warnings, + "quality_flags": result.quality_flags, + "recommended_next_step": infer_next_step(result), + "stats": result.stats, + } + + index_json = result.output_dir / "index.json" + index_md = result.output_dir / "index.md" + write_text(index_json, json.dumps(index_payload, indent=2, sort_keys=True) + "\n") + write_text(index_md, build_index_markdown(result)) + + +def maybe_markitdown(source: Path, output_path: Path, prefer: str) -> str | None: + if prefer == "native": + return None + + command_sets: list[list[str]] = [] + if shutil.which("markitdown"): + command_sets.append(["markitdown", str(source)]) + if prefer == "markitdown" and shutil.which("uvx"): + command_sets.append(["uvx", "--from", MARKITDOWN_SPEC, "markitdown", str(source)]) + + for command in command_sets: + try: + completed = subprocess.run( + command, + check=True, + capture_output=True, + text=True, + ) + except (OSError, subprocess.CalledProcessError): + continue + + if completed.stdout.strip(): + write_text(output_path, completed.stdout) + return "markitdown" + + return None + + +def convert_csv_like(source: Path, output_dir: Path, delimiter: str) -> ConversionResult: + result = ConversionResult(source=source, output_dir=output_dir, converter="native-csv") + normalized_path = output_dir / "table.csv" + summary_path = output_dir / "table.md" + + rows: list[list[str]] = [] + with source.open("r", encoding="utf-8", errors="replace", newline="") as handle: + reader = csv.reader(handle, delimiter=delimiter) + for row in reader: + rows.append([str(cell) for cell in row]) + + with normalized_path.open("w", encoding="utf-8", newline="") as handle: + writer = csv.writer(handle) + writer.writerows(rows) + + header = rows[0] if rows else [] + preview_rows = rows[1:6] if len(rows) > 1 else [] + summary_lines = [ + f"# Table Summary: {source.name}", + "", + f"- Rows: `{max(len(rows) - 1, 0)}`", + f"- Columns: `{len(header)}`", + f"- Header: `{', '.join(header) if header else 'none'}`", + "", + "## Preview", + "", + ] + + if preview_rows: + for index, row in enumerate(preview_rows, start=1): + summary_lines.append(f"{index}. `{row}`") + else: + summary_lines.append("No data rows found.") + + write_text(summary_path, "\n".join(summary_lines) + "\n") + + result.artifacts.extend( + [ + Artifact(path=str(normalized_path), kind="csv", description="Normalized CSV artifact"), + Artifact(path=str(summary_path), kind="markdown", description="Cheap table summary"), + ] + ) + result.stats.update( + { + "row_count": max(len(rows) - 1, 0), + "column_count": len(header), + } + ) + if not rows: + result.quality_flags.append("low_text_output") + return result + + +def convert_xlsx(source: Path, output_dir: Path) -> ConversionResult: + openpyxl = require_module("openpyxl", "openpyxl") + result = ConversionResult(source=source, output_dir=output_dir, converter="native-openpyxl") + workbook = openpyxl.load_workbook(source, read_only=True, data_only=True) + manifest_lines = [f"# Workbook Summary: {source.name}", ""] + sheet_count = 0 + + for sheet in workbook.worksheets: + sheet_count += 1 + slug = slugify(sheet.title) + csv_path = output_dir / f"{sheet_count:02d}-{slug}.csv" + + row_count = 0 + max_columns = 0 + first_rows: list[list[str]] = [] + with csv_path.open("w", encoding="utf-8", newline="") as handle: + writer = csv.writer(handle) + for values in sheet.iter_rows(values_only=True): + normalized = ["" if cell is None else str(cell) for cell in values] + writer.writerow(normalized) + if any(cell != "" for cell in normalized): + row_count += 1 + max_columns = max(max_columns, len(normalized)) + if len(first_rows) < 5: + first_rows.append(normalized) + + header = first_rows[0] if first_rows else [] + manifest_lines.extend( + [ + f"## Sheet {sheet_count}: {sheet.title}", + "", + f"- Rows with content: `{row_count}`", + f"- Columns observed: `{max_columns}`", + f"- Header preview: `{header}`", + f"- CSV artifact: `{csv_path.name}`", + "", + ] + ) + + result.artifacts.append( + Artifact(path=str(csv_path), kind="csv", description=f"Sheet {sheet_count}: {sheet.title}") + ) + + manifest_path = output_dir / "workbook.md" + write_text(manifest_path, "\n".join(manifest_lines)) + result.artifacts.append( + Artifact(path=str(manifest_path), kind="markdown", description="Workbook manifest and sheet preview") + ) + result.stats["sheet_count"] = sheet_count + if sheet_count == 0: + result.quality_flags.append("low_text_output") + return result + + +def extract_shape_text(shape, collector: list[str]) -> None: + if hasattr(shape, "has_text_frame") and shape.has_text_frame: + text = shape.text.strip() + if text: + collector.append(text) + if hasattr(shape, "shapes"): + for child in shape.shapes: + extract_shape_text(child, collector) + + +def convert_pptx(source: Path, output_dir: Path) -> ConversionResult: + pptx = require_module("pptx", "python-pptx") + presentation = pptx.Presentation(str(source)) + result = ConversionResult(source=source, output_dir=output_dir, converter="native-python-pptx") + markdown_path = output_dir / "presentation.md" + + lines = [f"# Presentation: {source.name}", ""] + titled_slides = 0 + for slide_number, slide in enumerate(presentation.slides, start=1): + title_shape = slide.shapes.title + title = title_shape.text.strip() if title_shape and title_shape.text else f"Slide {slide_number}" + if title_shape and title_shape.text.strip(): + titled_slides += 1 + + lines.append(f"## Slide {slide_number}: {title}") + lines.append("") + + slide_text: list[str] = [] + for shape in slide.shapes: + extract_shape_text(shape, slide_text) + + deduped: list[str] = [] + seen: set[str] = set() + for block in slide_text: + normalized = block.strip() + if not normalized or normalized in seen or normalized == title: + continue + seen.add(normalized) + deduped.append(normalized) + + if deduped: + for block in deduped: + lines.append(f"- {block}") + else: + lines.append("- No extractable text found on this slide.") + + notes_slide = getattr(slide, "notes_slide", None) + notes_frame = getattr(notes_slide, "notes_text_frame", None) if notes_slide else None + notes_text = notes_frame.text.strip() if notes_frame and notes_frame.text else "" + if notes_text: + lines.extend(["", "### Speaker Notes", "", notes_text]) + + lines.append("") + + write_text(markdown_path, "\n".join(lines)) + result.artifacts.append( + Artifact(path=str(markdown_path), kind="markdown", description="Slide outline and notes") + ) + result.stats.update( + { + "slide_count": len(presentation.slides), + "titled_slides": titled_slides, + } + ) + if len(presentation.slides) == 0: + result.quality_flags.append("low_text_output") + return result + + +def convert_docx(source: Path, output_dir: Path, prefer: str) -> ConversionResult: + markdown_path = output_dir / "document.md" + markitdown_used = maybe_markitdown(source, markdown_path, prefer) + if markitdown_used: + result = ConversionResult(source=source, output_dir=output_dir, converter=markitdown_used) + result.artifacts.append( + Artifact(path=str(markdown_path), kind="markdown", description="Document converted to markdown") + ) + text_chars = len(markdown_path.read_text(encoding="utf-8", errors="replace")) + result.stats["text_chars"] = text_chars + if text_chars < 200: + result.quality_flags.append("low_text_output") + return result + + docx = require_module("docx", "python-docx") + document = docx.Document(str(source)) + result = ConversionResult(source=source, output_dir=output_dir, converter="native-python-docx") + + lines = [f"# Document: {source.name}", ""] + heading_count = 0 + paragraph_count = 0 + for paragraph in document.paragraphs: + text = paragraph.text.strip() + if not text: + continue + style_name = paragraph.style.name if paragraph.style else "" + match = re.match(r"Heading (\d+)", style_name) + if match: + level = min(int(match.group(1)), 6) + lines.append(f"{'#' * level} {text}") + lines.append("") + heading_count += 1 + else: + lines.append(text) + lines.append("") + paragraph_count += 1 + + for table_index, table in enumerate(document.tables, start=1): + rows = [] + for row in table.rows[:11]: + rows.append([cell.text.strip() for cell in row.cells]) + if not rows: + continue + lines.append(f"## Table {table_index}") + lines.append("") + header = rows[0] + lines.append("| " + " | ".join(header) + " |") + lines.append("| " + " | ".join(["---"] * len(header)) + " |") + for row in rows[1:]: + lines.append("| " + " | ".join(row) + " |") + if len(table.rows) > len(rows): + lines.append("") + lines.append(f"_Table truncated after {len(rows)} rows for token efficiency._") + lines.append("") + + write_text(markdown_path, "\n".join(lines)) + result.artifacts.append( + Artifact(path=str(markdown_path), kind="markdown", description="Document converted to markdown") + ) + result.stats.update( + { + "heading_count": heading_count, + "paragraph_count": paragraph_count, + "table_count": len(document.tables), + } + ) + if heading_count == 0 and paragraph_count < 5: + result.quality_flags.append("low_text_output") + return result + + +def convert_pdf(source: Path, output_dir: Path, prefer: str) -> ConversionResult: + markdown_path = output_dir / "document.md" + markitdown_used = maybe_markitdown(source, markdown_path, prefer) + if markitdown_used: + result = ConversionResult(source=source, output_dir=output_dir, converter=markitdown_used) + result.artifacts.append( + Artifact(path=str(markdown_path), kind="markdown", description="PDF converted to markdown") + ) + text_chars = len(markdown_path.read_text(encoding="utf-8", errors="replace")) + result.stats["text_chars"] = text_chars + if text_chars < 400: + result.quality_flags.append("low_text_output") + return result + + pdfplumber = require_module("pdfplumber", "pdfplumber") + result = ConversionResult(source=source, output_dir=output_dir, converter="native-pdfplumber") + + lines = [f"# PDF: {source.name}", ""] + non_empty_pages = 0 + char_count = 0 + with pdfplumber.open(str(source)) as pdf: + page_count = len(pdf.pages) + for page_number, page in enumerate(pdf.pages, start=1): + text = (page.extract_text() or "").strip() + lines.append(f"## Page {page_number}") + lines.append("") + if text: + lines.append(text) + non_empty_pages += 1 + char_count += len(text) + else: + lines.append("_No extractable text found on this page._") + lines.append("") + + write_text(markdown_path, "\n".join(lines)) + result.artifacts.append( + Artifact(path=str(markdown_path), kind="markdown", description="PDF text extracted page by page") + ) + result.stats.update( + { + "page_count": page_count, + "non_empty_pages": non_empty_pages, + "text_chars": char_count, + "avg_chars_per_page": round(char_count / page_count, 1) if page_count else 0, + } + ) + if page_count and non_empty_pages / page_count < 0.7: + result.quality_flags.append("scanned_pdf_suspected") + if page_count >= 3 and char_count / page_count < 120: + result.quality_flags.append("low_text_density") + return result + + +def convert_text_file(source: Path, output_dir: Path) -> ConversionResult: + result = ConversionResult(source=source, output_dir=output_dir, converter="native-text") + text = source.read_text(encoding="utf-8", errors="replace") + copied_path = output_dir / ("document.md" if source.suffix.lower() != ".md" else source.name) + + if source.suffix.lower() == ".md": + shutil.copyfile(source, copied_path) + else: + write_text(copied_path, f"# Text File: {source.name}\n\n{text}") + + result.artifacts.append( + Artifact(path=str(copied_path), kind="markdown", description="Text copied into markdown") + ) + result.stats["text_chars"] = len(text) + return result + + +def handle_conversion(source: Path, output_dir: Path, prefer: str) -> ConversionResult: + suffix = source.suffix.lower() + if suffix == ".csv": + return convert_csv_like(source, output_dir, delimiter=",") + if suffix == ".tsv": + return convert_csv_like(source, output_dir, delimiter="\t") + if suffix == ".xlsx": + return convert_xlsx(source, output_dir) + if suffix == ".pptx": + return convert_pptx(source, output_dir) + if suffix == ".docx": + return convert_docx(source, output_dir, prefer) + if suffix == ".pdf": + return convert_pdf(source, output_dir, prefer) + if suffix in {".txt", ".md"}: + return convert_text_file(source, output_dir) + + result = ConversionResult(source=source, output_dir=output_dir, converter="unsupported") + result.warnings.append(f"Unsupported file type: {suffix or 'unknown'}") + result.quality_flags.extend(["conversion_failed", "unsupported_type"]) + return result + + +def main() -> int: + args = parse_args() + source = args.source.expanduser().resolve() + if not source.exists() or not source.is_file(): + print(f"Source file not found: {source}", file=sys.stderr) + return 1 + + output_dir = ( + args.output_dir.expanduser().resolve() + if args.output_dir + else source.parent / f"{source.name}.ob1" + ) + output_dir.mkdir(parents=True, exist_ok=True) + + try: + result = handle_conversion(source, output_dir, args.prefer) + except RuntimeError as exc: + result = ConversionResult(source=source, output_dir=output_dir, converter="failed") + result.warnings.append(str(exc)) + result.quality_flags.extend(["conversion_failed", "dependency_missing"]) + except Exception as exc: + result = ConversionResult(source=source, output_dir=output_dir, converter="failed") + result.warnings.append(f"{exc.__class__.__name__}: {exc}") + result.quality_flags.extend(["conversion_failed", "unexpected_error"]) + + result.stats.setdefault("source_extension", source.suffix.lower() or "none") + result.stats.setdefault("source_size_bytes", source.stat().st_size) + write_index_files(result) + + print(json.dumps( + { + "output_dir": str(output_dir), + "recommended_next_step": infer_next_step(result), + "quality_flags": result.quality_flags, + "artifacts": [artifact.__dict__ for artifact in result.artifacts], + }, + indent=2, + )) + return 0 if "conversion_failed" not in result.quality_flags else 2 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/skills/heavy-file-ingestion/variants/claude-code/SKILL.md b/skills/heavy-file-ingestion/variants/claude-code/SKILL.md new file mode 100644 index 00000000..87deb572 --- /dev/null +++ b/skills/heavy-file-ingestion/variants/claude-code/SKILL.md @@ -0,0 +1,57 @@ +--- +name: heavy-file-ingestion-claude-code +description: Use in Claude Code when a user asks to read, analyze, summarize, or extract from a heavyweight file such as PDF, DOCX, PPTX, XLSX, CSV, or TSV. Convert the file into markdown or CSV first with the bundled script, generate a lightweight index, and only spend model tokens on the compressed artifact. +author: Nate B. Jones +version: 1.0.0 +--- + +# Heavy File Ingestion For Claude Code + +## Problem + +Claude Code has the tools to convert files locally, so it should not waste context by reading heavyweight files raw. + +## Trigger Conditions + +- The user asks to read or analyze a PDF, DOCX, PPTX, XLSX, CSV, or TSV +- The file is large enough or structured enough that direct ingestion is a bad trade +- The user wants a markdown working copy, CSV normalization, or a fast map of the file before deeper analysis + +## Process + +1. Do not read the original heavyweight file directly into context if conversion is possible. +1. Resolve the bundled converter relative to this skill directory: `scripts/convert_heavy_file.py` +1. Run the converter first. Default command: + +```bash +python scripts/convert_heavy_file.py /absolute/path/to/file.ext +``` + +1. If dependencies are missing, prefer: + +```bash +uv run \ + --with pdfplumber \ + --with python-docx \ + --with python-pptx \ + --with openpyxl \ + python scripts/convert_heavy_file.py /absolute/path/to/file.ext +``` + +1. Read the generated `index.md` before reading any converted artifact. +2. Use the index to decide the cheapest next step: + - `read_extracted_artifact`: read the markdown or CSV and continue + - `install_dependency_and_retry`: install the missing deterministic dependency and rerun + - `cheap_model_or_stronger_converter`: retry with a better converter or use a cheaper model only on the extracted artifact +3. Only escalate to a stronger model after the file has already been compressed into markdown, CSV, or a short sampled subset. + +## Client Rules + +- Prefer deterministic scripts over model-based conversion. +- Save the converted artifacts next to the source file and work from those files. +- For spreadsheets, use the generated per-sheet CSV files instead of trying to reason over workbook internals directly. +- For PDFs, treat scan detection and low-density warnings as a routing signal, not as a reason to read the original PDF raw. + +## Bundled References + +- `references/open-source-stack.md` explains the tool choices and fallback strategy. diff --git a/skills/heavy-file-ingestion/variants/claude-desktop/SKILL.md b/skills/heavy-file-ingestion/variants/claude-desktop/SKILL.md new file mode 100644 index 00000000..d7f6f060 --- /dev/null +++ b/skills/heavy-file-ingestion/variants/claude-desktop/SKILL.md @@ -0,0 +1,55 @@ +--- +name: heavy-file-ingestion-claude-desktop +description: Use in Claude Desktop when a user asks to read, analyze, summarize, or extract from a heavyweight file such as PDF, DOCX, PPTX, XLSX, CSV, or TSV. Avoid raw ingestion of bulky files. Ask for a converted markdown or CSV artifact first, or give the user exact conversion commands to run outside Claude Desktop. +author: Nate B. Jones +version: 1.0.0 +--- + +# Heavy File Ingestion For Claude Desktop + +## Problem + +Claude Desktop does not have the same local shell workflow as coding agents, so it should avoid pretending it can efficiently process bulky files raw. + +## Trigger Conditions + +- The user asks Claude Desktop to read a PDF, PPTX, DOCX, XLSX, or another bulky attachment +- The file would cost too much context for too little value +- The user would be better served by a converted markdown or CSV artifact + +## Process + +1. Do not ingest the raw heavyweight file by default. +1. First ask for the cheapest workable artifact: + - PDF or DOCX: markdown + - PPTX: markdown slide outline + - XLSX: CSV per sheet or a small sample plus sheet names +1. If the user has not converted it yet, offer exact commands they can run outside Claude Desktop. + +### Suggested Conversion Commands + +```bash +python convert_heavy_file.py /absolute/path/to/file.pdf +python convert_heavy_file.py /absolute/path/to/file.docx +python convert_heavy_file.py /absolute/path/to/file.pptx +python convert_heavy_file.py /absolute/path/to/file.xlsx +``` + +If the script is not available, say so and ask the user for: + +- a markdown export +- a CSV export +- or a small representative excerpt + +1. Once the user provides the converted artifact, create a quick index: + - file type + - sections, slides, or sheet names + - row counts or page counts if available + - any obvious extraction-quality problems +2. Only then analyze the content. + +## Client Rules + +- Be explicit about the tradeoff: converting first is cheaper and usually better. +- If the user insists on staying inside Claude Desktop, ask for a smaller excerpt rather than taking the whole file raw. +- Use raw ingestion only for genuinely small files where conversion would cost more effort than it saves. diff --git a/skills/heavy-file-ingestion/variants/codex/SKILL.md b/skills/heavy-file-ingestion/variants/codex/SKILL.md new file mode 100644 index 00000000..70d824bc --- /dev/null +++ b/skills/heavy-file-ingestion/variants/codex/SKILL.md @@ -0,0 +1,56 @@ +--- +name: heavy-file-ingestion-codex +description: Use in Codex when a user asks to read, analyze, summarize, or extract from a heavyweight file such as PDF, DOCX, PPTX, XLSX, CSV, or TSV. Convert the file into markdown or CSV first with the bundled script, generate a lightweight index, and only spend model tokens on the compressed artifact. +author: Nate B. Jones +version: 1.0.0 +--- + +# Heavy File Ingestion For Codex + +## Problem + +Codex can run local commands and inspect files, so direct ingestion of bulky documents is usually the wrong move. Convert first, index second, reason last. + +## Trigger Conditions + +- The user asks to read or summarize a heavyweight document or spreadsheet +- The file is large, structured, or expensive enough that raw ingestion is wasteful +- The task would be better served by markdown, CSV, or a quick file map + +## Process + +1. Do not open the raw heavyweight file as your first move if a deterministic conversion path exists. +1. Run the bundled converter from this skill directory: + +```bash +python scripts/convert_heavy_file.py /absolute/path/to/file.ext +``` + +1. If the environment is clean and needs packages, prefer: + +```bash +uv run \ + --with pdfplumber \ + --with python-docx \ + --with python-pptx \ + --with openpyxl \ + python scripts/convert_heavy_file.py /absolute/path/to/file.ext +``` + +1. Read `index.md` first, not the original file. +2. Follow the index recommendation: + - `read_extracted_artifact`: inspect the generated markdown or CSV + - `cheap_model_or_stronger_converter`: retry with a better deterministic tool or use a cheaper model on the extracted artifact only + - `manual_review`: tell the user the deterministic route failed and propose the next cheapest fallback +3. Use expensive model context only after the file has already been compressed into a smaller artifact. + +## Client Rules + +- Keep the main model out of raw PDFs, decks, and spreadsheets whenever possible. +- Use the generated `.ob1/` folder as the working directory for follow-up analysis. +- For spreadsheets, reason from the CSV per sheet plus the workbook manifest. +- For presentations, reason from the slide outline before asking for a deeper pass. + +## Bundled References + +- `references/open-source-stack.md` explains the tool choices and fallback tiers.