Skip to content

Lazy parsing modes: metadata-only sweeps and deferred attachment decoding (mode="metadata"|"lazy") #97

Description

@kurok

Problem

parse_email always materializes everything: all text bodies decoded, every attachment's full content copied into Python bytes. But two of the highest-volume real workloads don't want the payloads:

  • Scanning/triage: classify by headers + subject + sender across millions of messages — attachment bytes are pure wasted allocation (often 95%+ of the message).
  • Selective extraction: find the one PDF in a mailbox — decode only that attachment, not every inline image in every message.

For a library whose selling point is throughput, "decode everything always" leaves the biggest single optimization on the table.

Proposal

1. Headers-only mode

mail = parse_email(payload, mode="metadata")
# subject/date/headers fully populated;
# text_plain/text_html: [] ; attachments: metadata only (mimetype, filename, size) with content=None
  • MIME structure is still walked (part inventory + sizes — cheap); transfer-decoding and content copying are skipped entirely.
  • PyAttachment gains size: int (decoded-size estimate or encoded size, documented which) in both modes — useful generally.

2. Deferred attachment content (full mode stays default)

  • parse_email(payload, mode="lazy"): bodies decoded as today; PyAttachment.content decodes on first access (the raw part bytes are retained internally; decode happens at property access, cached after). The mailbox-triage sweet spot: full text search + selective attachment pull.
  • Default mode="full" remains byte-identical to today — zero behavior change without opt-in.

Design notes

Acceptance criteria

  • mode="metadata": headers/subject/date identical to full mode on the whole test corpus (parametrized equality test); bodies empty; attachment metadata (mimetype, filename, size) present with content is None.
  • mode="lazy": content access returns bytes identical to full mode for every attachment in the corpus (equality test); repeated access returns the same object (cache test); un-accessed attachments never decode (instrumented via a decode counter in a test build or a timing proxy — mechanism chosen in review).
  • Default mode regression: parse_email(payload) output byte-identical to today across the corpus and RFC corpus (test_rfc_corpus.py).
  • Benchmark: metadata mode ≥ 5× faster than full mode on the attachment-heavy fixture; numbers in PR; benchmark file added alongside the existing ones.
  • size attribute documented (decoded vs encoded semantics stated) and tested against known fixtures.
  • Thread-safety of lazy decode under free-threaded/multi-thread access (Mutex/OnceCell semantics; race test with threads hammering one attachment).
  • Stubs + README table updated (mode parameter, size attribute); mypy --strict, ruff, clippy, fmt clean; CHANGELOG entry.

Out of scope

  • Zero-copy views into the original buffer (PyO3 lifetime complexity; measure the copy cost first).
  • Partial body decoding (first-N-bytes previews).

Part of the roadmap (see tracking issue). Composes with parse_many (metadata batch sweeps) — the combination is the pipeline-throughput story.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions