Merge branch 'master' into 3420-support-flip-endianness

Author: ESultanik

.github/workflows/add_to_project.yml

@@ -14,7 +14,7 @@ jobs:
     name: Add to project
     runs-on: ubuntu-latest
     steps:
-      - uses: actions/add-to-project@v0.5.0
+      - uses: actions/add-to-project@v1.0.2
         with:
           project-url: https://github.com/orgs/trailofbits/projects/12
           github-token: ${{ secrets.ADD_TO_PROJECT_PAT }}

.github/workflows/pythonpublish.yml

@@ -13,11 +13,11 @@ jobs:
     runs-on: ubuntu-latest
 
     steps:
-    - uses: actions/checkout@v4.1.1
+    - uses: actions/checkout@v6.0.2
       with:
         submodules: recursive
     - name: Set up Python
-      uses: actions/setup-python@v4
+      uses: actions/setup-python@v6
       with:
         python-version: '3.x'
     - name: Install dependencies

.github/workflows/tests.yml

@@ -15,16 +15,16 @@ jobs:
     strategy:
       matrix:
         os: [ubuntu-latest] # windows-latest, macos-latest,
-        python-version: ["3.8", "3.9", "3.10", "3.11", "3.12"]
+        python-version: ["3.10", "3.11", "3.12", "3.13", "3.14"]
 
     runs-on: ${{ matrix.os }}
 
     steps:
-    - uses: actions/checkout@v4.1.1
+    - uses: actions/checkout@v6.0.2
       with:
         submodules: recursive
     - name: Set up Python ${{ matrix.python-version }}
-      uses: actions/setup-python@v4
+      uses: actions/setup-python@v6
       with:
         python-version: ${{ matrix.python-version }}
     - name: Install Python Dependencies
@@ -33,7 +33,7 @@ jobs:
         pip install setuptools
         pip install .[dev]
     - name: Scan with pip-audit
-      uses: trailofbits/gh-action-pip-audit@v1.0.8
+      uses: trailofbits/gh-action-pip-audit@v1.1.0
     - name: Lint with flake8
       run: |
         # stop the build if there are Python syntax errors or undefined names

CLAUDE.md

@@ -0,0 +1,209 @@
+# PolyFile Development Guide
+
+## Project Overview
+
+PolyFile is a file analysis utility that identifies and maps the semantic and syntactic structure of files—including polyglots, chimeras, and "schizophrenic" files that are validly multiple types simultaneously.
+
+**Key capabilities:**
+- Pure-Python libmagic implementation (263+ MIME types)
+- Recursive embedded file detection (like binwalk)
+- Parsers for PDF, ZIP, JPEG, iNES, and 188 Kaitai Struct formats
+- Interactive HTML hex viewer with structure mapping
+- Drop-in replacement for Unix `file` command
+
+Part of the [ALAN Parsers Project](https://github.com/trailofbits/polyfile#the-alan-parsers-project) alongside PolyTracker.
+
+## Architecture
+
+### Core Pattern: Matchers + Parsers
+
+**Matchers** classify file types. Two types:
+- libmagic DSL matchers (defined in `polyfile/magic_defs/`)
+- Python matchers (classes extending `Matcher`)
+
+**Parsers** create AST representations of file structure. Registered via decorator:
+```python
+from polyfile import register_parser
+
+@register_parser("application/pdf")
+def parse_pdf(file_stream, match):
+    # Return parsed structure
+    ...
+```
+
+### Key Modules
+
+| Module | Purpose | Size |
+|--------|---------|------|
+| `polyfile.py` | Core engine—Match, Parser, Submatch classes | 15 KB |
+| `magic.py` | Pure-Python libmagic DSL implementation | 117 KB |
+| `pdf.py` | PDF parser with embedded file detection | 48 KB |
+| `debugger.py` | Interactive GDB-style debugger | 42 KB |
+| `kaitaimatcher.py` | Kaitai Struct format bridge | — |
+
+### Directory Structure
+
+```
+polyfile/
+├── polyfile/              # Main package
+│   ├── magic_defs/        # 354 libmagic definition files
+│   ├── kaitai/parsers/    # 188 auto-generated Kaitai parsers (excluded from lint)
+│   └── templates/         # HTML output templates
+├── polymerge/             # Companion merge tool
+├── tests/                 # Test suite
+├── docs/                  # Extension guide, JSON format spec
+└── kaitai_struct_formats/ # Git submodule with KSY definitions
+```
+
+## Development Commands
+
+### Setup
+```bash
+# Install from source (requires Java for Kaitai compiler)
+pip install -e .[dev]
+
+# Install from PyPI
+pip install polyfile
+```
+
+### Linting
+```bash
+# Run flake8 (excludes auto-generated kaitai parsers)
+flake8 polyfile polymerge --max-complexity=10 --max-line-length=127 \
+    --exclude=polyfile/kaitai/parsers
+```
+
+### Testing
+```bash
+# Run all tests
+pytest tests
+
+# Run specific test file
+pytest tests/test_magic.py
+pytest tests/test_pdf.py
+pytest tests/test_corkami.py  # Polyglot corpus
+```
+
+### Security Audit
+```bash
+pip-audit
+```
+
+### Pre-Commit Checklist
+Run all checks before committing changes:
+```bash
+# Lint
+flake8 polyfile polymerge --max-complexity=10 --max-line-length=127 \
+    --exclude=polyfile/kaitai/parsers
+
+# Security audit (checks for vulnerable dependencies)
+pip-audit
+
+# Tests
+pytest tests
+```
+
+## Code Navigation
+
+### Finding Matchers
+```bash
+# Find libmagic definitions by MIME type
+rg "application/pdf" polyfile/magic_defs/
+
+# Find Python matchers
+ast-grep --pattern 'class $NAME(Matcher): $$$' --lang py polyfile/
+```
+
+### Finding Parsers
+```bash
+# Find registered parsers
+rg "@register_parser" polyfile/
+
+# Find parser for specific MIME type
+rg 'register_parser.*application/zip' polyfile/
+```
+
+### Key Entry Points
+- CLI: `polyfile/__main__.py`
+- Core analysis: `polyfile/polyfile.py:PolyFile.struc()`
+- Magic matching: `polyfile/magic.py:MagicMatcher.match()`
+
+## Testing
+
+### Test Structure
+```
+tests/
+├── test_magic.py      # libmagic implementation vs corpus
+├── test_pdf.py        # PDF parsing
+├── test_corkami.py    # Polyglot/chimera edge cases
+├── test_kaitai.py     # Kaitai format tests
+└── unit/
+    ├── test_ast.py    # AST utilities
+    └── test_http.py   # HTTP protocol parsing
+```
+
+### Test Conventions
+- Uses real file corpus including libmagic's official test suite
+- Tests polyglot files to verify multi-type detection
+- Parser tests validate structure extraction
+
+## Extending PolyFile
+
+### Adding a Custom Matcher (Python)
+```python
+from polyfile import Matcher, Match
+
+class MyMatcher(Matcher):
+    def match(self, data: bytes) -> Match | None:
+        if data.startswith(b'MAGIC'):
+            return Match(
+                mime_type="application/x-myformat",
+                name="My Format",
+                offset=0,
+                length=len(data)
+            )
+        return None
+```
+
+### Adding a Custom Parser
+```python
+from polyfile import register_parser, Parser, Submatch
+
+@register_parser("application/x-myformat")
+class MyParser(Parser):
+    def parse(self, file_stream, match) -> Iterator[Submatch]:
+        # Yield Submatch objects representing structure
+        yield Submatch(
+            name="header",
+            start=0,
+            length=8,
+            value=file_stream.read(8)
+        )
+```
+
+### Adding Kaitai Struct Format
+1. Add `.ksy` file to `kaitai_struct_formats/`
+2. Map MIME type in `polyfile/kaitai/parsers/__init__.py`
+3. Rebuild: `python compile_kaitai_parsers.py`
+
+See `docs/extending_polyfile.md` for detailed guide.
+
+## Internal API Patterns
+
+### File I/O
+- Use `FileStream` abstraction for seeking/reading
+- `PathOrStdin`/`PathOrStdout` for CLI flexibility
+
+### Match Hierarchy
+- `Match` → top-level file type match
+- `Submatch` → nested structure within a match
+- Build trees for embedded files (ZIP contents, PDF streams)
+
+### Error Handling
+- Raise `InvalidMatch` when parser cannot process data
+- Matchers return `None` for non-matching data
+
+### Gotchas
+- `polyfile/kaitai/parsers/` is auto-generated—never edit manually
+- Java required at install time for Kaitai compilation
+- libmagic DSL has quirks—see [blog post](https://blog.trailofbits.com/2022/07/01/libmagic-the-blathering/)

polyfile/__init__.py

@@ -1,6 +1,5 @@
 from . import (
     nes,
-    pdf,
     jpeg,
     zipmatcher,
     nitf,
@@ -13,3 +12,20 @@
 
 from .__main__ import main
 from .polyfile import __version__, InvalidMatch, Match, Matcher, Parser, PARSERS, register_parser, Submatch
+
+
+# Lazy PDF parser registration
+# This registers immediately but defers importing pdf.py (and pdfminer) until first use
+class _LazyPDFParser(Parser):
+    """Lazy wrapper that imports the actual PDF parser on first use."""
+
+    _actual_parser = None
+
+    def parse(self, stream, match):
+        if _LazyPDFParser._actual_parser is None:
+            from . import pdf
+            _LazyPDFParser._actual_parser = pdf.pdf_parser
+        yield from _LazyPDFParser._actual_parser(stream, match)
+
+
+PARSERS["application/pdf"].add(_LazyPDFParser())

polyfile/debugger.py

@@ -147,7 +147,7 @@ def should_break(
             parent_match: Optional[TestResult],
             result: Optional[TestResult]
     ) -> bool:
-        return self.pattern.is_contained_in(test.mimetypes())
+        return self.pattern.is_contained_in(test.mimetypes)
 
     @classmethod
     def parse(cls: Type[B], command: str) -> Optional[B]:
@@ -183,7 +183,7 @@ def should_break(
             parent_match: Optional[TestResult],
             result: Optional[TestResult]
     ) -> bool:
-        return self.ext in test.all_extensions()
+        return self.ext in test.all_extensions
 
     @classmethod
     def parse(cls: Type[B], command: str) -> Optional[B]:

polyfile/jpeg.py

@@ -4,11 +4,16 @@
 from .fileutils import FileStream, Tempfile
 from .polyfile import Match, register_parser, Submatch
 
-from PIL import Image
+
+def _get_pil_image():
+    """Lazy import PIL.Image only when needed (for JPEG2000 parsing)."""
+    from PIL import Image
+    return Image
 
 
 @register_parser("image/jp2")
 def parse_jpeg2000(file_stream: FileStream, parent: Match):
+    Image = _get_pil_image()
     with Tempfile(file_stream.read(parent.length)) as input_bytes:
         img = Image.open(input_bytes)
         with BytesIO() as img_data:

polyfile/kaitaimatcher.py

@@ -5,9 +5,6 @@
 from kaitaistruct import KaitaiStruct, KaitaiStructError
 
 from .kaitai.parser import ASTNode, KaitaiParser, RootNode
-from .kaitai.parsers.gif import Gif
-from .kaitai.parsers.jpeg import Jpeg
-from .kaitai.parsers.png import Png
 from .logger import getStatusLogger
 from .polyfile import register_parser, InvalidMatch, Match, Parser, Submatch
 
@@ -83,31 +80,41 @@ def ast_to_matches(ast: RootNode, parent: Match) -> Iterator[Submatch]:
         stack.extend(reversed([(new_node, c) for c in node.children]))
 
 
-for mimetype, kaitai_path in KAITAI_MIME_MAPPING.items():
-    class parse_:
-        kaitai_parser = KaitaiParser.load(kaitai_path)
-
-        def __call__(self, stream, match):
-            try:
-                ast = self.kaitai_parser.parse(stream).ast
-            except KaitaiStructError as e:
-                log.warning(f"Error parsing {stream.name} using {self.kaitai_parser}: {e!s}")
-                raise InvalidMatch()
-            except Exception as e:
-                log.error(f"Unexpected exception parsing {stream.name} using {self.kaitai_parser}: {e!s}")
-                raise InvalidMatch()
-            yield from ast_to_matches(ast, parent=match)
+class LazyKaitaiParser:
+    """Parser that lazily loads the Kaitai struct parser on first use."""
 
-    func_name = mimetype.replace("/", "_").replace("-", "_")
+    def __init__(self, kaitai_path: str, mimetype: str):
+        self.kaitai_path = kaitai_path
+        self.mimetype = mimetype
+        self._kaitai_parser = None
+
+    @property
+    def kaitai_parser(self):
+        if self._kaitai_parser is None:
+            self._kaitai_parser = KaitaiParser.load(self.kaitai_path)
+            MIME_BY_PARSER[self._kaitai_parser.struct_type] = self.mimetype
+        return self._kaitai_parser
 
-    parse_.__name__ = f"{parse_.__name__}{func_name}"
-    parse_.__qualname__ = f"{parse_.__qualname__}{func_name}"
+    def __call__(self, stream, match):
+        try:
+            ast = self.kaitai_parser.parse(stream).ast
+        except KaitaiStructError as e:
+            log.warning(f"Error parsing {stream.name} using {self.kaitai_parser}: {e!s}")
+            raise InvalidMatch()
+        except Exception as e:
+            log.error(f"Unexpected exception parsing {stream.name} using {self.kaitai_parser}: {e!s}")
+            raise InvalidMatch()
+        yield from ast_to_matches(ast, parent=match)
 
-    register_parser(mimetype)(parse_())
 
-    MIME_BY_PARSER[parse_.kaitai_parser.struct_type] = mimetype
+for mimetype, kaitai_path in KAITAI_MIME_MAPPING.items():
+    func_name = mimetype.replace("/", "_").replace("-", "_")
+    parser = LazyKaitaiParser(kaitai_path, mimetype)
+    parser.__name__ = f"parse_{func_name}"
+    parser.__qualname__ = f"parse_{func_name}"
+    register_parser(mimetype)(parser)
 
 del func_name
 del kaitai_path
 del mimetype
-del parse_
+del parser