[32] Refactor from "replacement processors" to "engines" & add AsyncLookaheadTransformEngine

[TomStrepsil]

Issue

resolves #32

Details

Added

• Protocol-agnostic engine layer, all exported from replace-content-transformer:
  • SyncReplacementTransformEngine — absorbs all former sync processors; replacement function returns string | Iterable<string>
  • AsyncSerialReplacementTransformEngine — absorbs all former async processors; each replacement is fully consumed before scanning continues
  • AsyncLookaheadTransformEngine — eagerly initiates replacement work as matches are discovered rather than serially awaiting each; output order is preserved while concurrent initiation of async iterable replacements unlocks pipelined I/O (e.g. parallel fragment fetches with in-order rendering)
    • nested(source) sentinel for opt-in recursive re-scanning in AsyncLookaheadTransformEngine: returning nested(body) from a replacement function spawns a child engine (sharing the same search strategy, concurrency strategy, and replacement function) to re-process the body; a plain AsyncIterable<string> return emits verbatim. Nested work competes for the same concurrency budget and is ordered via tree-aware comparators
    • highWaterMark option on AsyncLookaheadTransformEngine (default 32) — caps buffered-ahead slots, providing upstream backpressure when downstream stalls
    • Pluggable ConcurrencyStrategy interface with two built-in implementations:
      • SemaphoreStrategy — FIFO arrival-order dispatch bounded by a concurrency limit
      • PriorityQueueStrategy — heap-backed, slot-tree-aware; pairs with a NodeComparator to order queued work across nesting levels
    • Two built-in comparators for PriorityQueueStrategy: streamOrder (earlier-in-output-stream first, via LCA) and breadthFirst (shallower first, sibling-index tie-break)
• New exported types: EngineSink, ReplacementContext, LookaheadReplacementContext, SyncTransformEngine, AsyncTransformEngine, SlotTreeNode, IterableSlotNode, TextSlotNode, SlotNode, NodeComparator

Breaking changes

• Adapters now accept an engine as their sole constructor argument, replacing the former (processor, stopReplacingSignal?) signature — applies to ReplaceContentTransformer, AsyncReplaceContentTransformer (web) and ReplaceContentTransform, AsyncReplaceContentTransform (Node)
  • stopReplacingSignal has moved into engine options (SyncReplacementTransformEngineOptions, AsyncSerialReplacementTransformEngineOptions, AsyncLookaheadTransformEngineOptions)
• SyncReplacementTransformEngineOptions no longer accepts Promise<string> replacements — the replacement function must now return string; the previous promise-enqueuing pattern is superseded by AsyncLookaheadTransformEngine. ReplaceContentTransformer drops its T extends string | Promise<string> type parameter (now void)
  • Migration: replace new ReplaceContentTransformer<Promise<string>>(new FunctionReplacementProcessor<Promise<string>>({...})) with new AsyncReplaceContentTransformer(new AsyncLookaheadTransformEngine({...}))
• All processor classes removed (StaticReplacementProcessor, FunctionReplacementProcessor, IterableFunctionReplacementProcessor, AsyncFunctionReplacementProcessor, AsyncIterableFunctionReplacementProcessor) along with the SyncProcessor / AsyncProcessor types; the engine layer below absorbs their responsibilities

Scout rule

• Selected harness/benchmark imports switched from .js to .ts so strict Deno module resolution works without --sloppy-imports
• Moved prepublishOnly script to prepack (canonically the right hook for pre-publish builds)

Semantic Version Impact

• PATCH - Bug fixes and minor changes (backwards compatible)
• MINOR - New features (backwards compatible)
• MAJOR - Breaking changes (not backwards compatible)

Checklist

• I have added an entry to the [Unreleased] section in CHANGELOG.md
• My code follows the style guidelines of this project
• I have performed a self-review of my own code
• I have added tests that prove my fix is effective or that my feature works
• New and existing unit tests pass locally with my changes

[Copilot]

Pull request overview

This PR introduces a new protocol-agnostic “engine” layer that replaces the previous processor-based architecture and updates web/Node adapters, harnesses, and benchmarks to use the new engines (including a new async lookahead engine with pluggable concurrency strategies and optional recursive re-scanning via nested()).

Changes:

• Added SyncReplacementTransformEngine, AsyncSerialReplacementTransformEngine, and AsyncLookaheadTransformEngine (plus concurrency strategies / slot-tree types) and re-plumbed adapters to accept engines.
• Updated search-strategy interfaces to support matchToString() and adjusted tests/harnesses accordingly.
• Expanded benchmark tooling (including lookahead-engine strategy benchmarks + reporting/timeline visualization) and added a v1→v2 codemod step for processor→engine migration.

Reviewed changes

Copilot reviewed 128 out of 128 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
package.json	Switch publish hook to prepack; add benchmark script delegation (currently with a script-name mismatch).
docs/CHANGELOG.md	Document breaking engine migration and new lookahead features.
src/index.ts	Re-export engines from the package root.
src/engines/index.ts	Public exports for new engine APIs and lookahead engine sub-exports.
src/engines/types.ts	Define EngineSink, ReplacementContext, and engine interfaces used by adapters.
src/engines/transform-engine-base.ts	Shared base class for engine state/sink management + flush helpers.
src/engines/sync-transform-engine.ts	New sync engine (static/string/iterable replacement support).
src/engines/async-serial-transform-engine.ts	New async serial engine (string/async-iterable replacements, serial semantics).
src/engines/async-lookahead-transform-engine/index.ts	Public exports for lookahead engine, strategies, comparators, slot-tree types, nested().
src/engines/async-lookahead-transform-engine/nested.ts	Implement Nested marker + nested() helper for recursive re-scanning.
src/engines/async-lookahead-transform-engine/slot-tree/constants.ts	Slot-kind constants for slot-tree nodes.
src/engines/async-lookahead-transform-engine/slot-tree/types.ts	Slot-tree node types for ordering/scheduling and drain representation.
src/engines/async-lookahead-transform-engine/slot-tree/async-child-queue.ts	Bounded async queue used for lookahead buffering/backpressure.
src/engines/async-lookahead-transform-engine/slot-tree/async-child-queue.test.ts	Unit tests for AsyncChildQueue.
src/engines/async-lookahead-transform-engine/concurrency-strategy/types.ts	Define ConcurrencyStrategy interface.
src/engines/async-lookahead-transform-engine/concurrency-strategy/semaphore.ts	Internal FIFO semaphore primitive.
src/engines/async-lookahead-transform-engine/concurrency-strategy/semaphore.test.ts	Unit tests for semaphore primitive.
src/engines/async-lookahead-transform-engine/concurrency-strategy/semaphore-strategy.ts	Default FIFO concurrency strategy implementation.
src/engines/async-lookahead-transform-engine/concurrency-strategy/semaphore-strategy.test.ts	Unit tests for semaphore-based strategy.
src/engines/async-lookahead-transform-engine/concurrency-strategy/min-heap.ts	Internal min-heap used by priority queue strategy.
src/engines/async-lookahead-transform-engine/concurrency-strategy/min-heap.test.ts	Unit tests for MinHeap.
src/engines/async-lookahead-transform-engine/concurrency-strategy/node-comparators.ts	streamOrder and breadthFirst comparators for tree-aware scheduling.
src/engines/async-lookahead-transform-engine/concurrency-strategy/node-comparators.test.ts	Unit tests for node comparators.
src/engines/async-lookahead-transform-engine/concurrency-strategy/priority-queue-strategy.ts	Priority-queue concurrency strategy implementation.
src/engines/async-lookahead-transform-engine/concurrency-strategy/priority-queue-strategy.test.ts	Unit tests for priority queue strategy.
src/engines/async-lookahead-transform-engine/engine.integration.test.ts	End-to-end integration tests for lookahead engine behavior (ordering, eager initiation, nested).
src/adapters/web/transformer-base.ts	Refactor web adapter base to wire engines via EngineSink.
src/adapters/web/sync-transformer.ts	Web sync adapter now wraps SyncTransformEngine.
src/adapters/web/sync-transformer.test.ts	Update sync web adapter tests for engine-based wiring.
src/adapters/web/sync-transformer.integration.test.ts	Integration tests updated for engine-based construction and abort handling.
src/adapters/web/async-transformer.ts	Web async adapter now wraps AsyncTransformEngine + forwards optional cancel().
src/adapters/web/async-transformer.test.ts	Update async web adapter tests for engine-based wiring/cancel forwarding.
src/adapters/web/benchmarking/sync-transformer-callback.ts	Remove old benchmarking callback adapter (processor-based).
src/adapters/web/benchmarking/async-transformer-callback.ts	Remove old benchmarking callback adapter (processor-based).
src/adapters/node/transform-base.ts	Refactor Node adapter base to start engines + map sink errors to destroy().
src/adapters/node/sync-transform.ts	Node sync adapter now wraps SyncTransformEngine.
src/adapters/node/sync-transform.test.ts	Update Node sync adapter tests for engine-based wiring and options.
src/adapters/node/async-transform.ts	Node async adapter now wraps AsyncTransformEngine and awaits write/end.
src/adapters/node/async-transform.test.ts	Update Node async adapter tests for engine-based wiring and error surfacing.
src/adapters/node/async-transform.integration.test.ts	New integration tests covering Node adapter specifics with lookahead engine.
src/adapters/node/benchmarking/sync-transform-callback.ts	Inline callback-processor interface for benchmarking transform (decoupled from removed processor types).
src/adapters/node/benchmarking/async-transform-callback.ts	Inline async callback-processor interface for benchmarking transform.
src/search-strategies/types.ts	Introduce StreamIndices alias + add matchToString() to SearchStrategy.
src/search-strategies/string-buffer-strategy-base.ts	Genericize base class and add default matchToString().
src/search-strategies/regex/search-strategy.ts	Make regex strategy’s matchToString() return match[0].
src/search-strategies/regex/search-strategy.test.ts	Add coverage for regex matchToString().
src/search-strategies/looped-indexOf-anchored/search-strategy.test.ts	Add coverage for string match matchToString().
src/search-strategies/benchmarking/index.ts	Switch selected benchmark exports/imports to .ts paths for strict Deno resolution.
src/search-strategies/benchmarking/anchor-sequence/search-strategy.ts	Update benchmarking anchor-sequence strategy imports/types for .ts + shared base.
src/search-strategies/benchmarking/regex-canonical/search-strategy.ts	Adjust canonical benchmark transformer to be adapter-agnostic + update types import.
src/search-strategies/benchmarking/regex-callback/search-strategy.ts	Adjust callback benchmark strategy to be adapter-agnostic + update types import.
src/search-strategies/benchmarking/looped-indexOf-cancellable/search-strategy.ts	Switch benchmarking imports to .ts.
src/search-strategies/benchmarking/looped-indexOf-callback/search-strategy.ts	Remove old processor interface implementation; minor cleanup.
src/search-strategies/benchmarking/indexOf-knuth-morris-pratt/search-strategy.ts	Switch benchmarking imports to .ts.
src/search-strategies/benchmarking/buffered-indexOf-canonical/search-strategy.ts	Remove WHATWG Transformer typing to stay harness-agnostic; minor cleanup.
src/search-strategies/benchmarking/buffered-indexOf-canonical-generator/search-strategy.ts	Switch imports to .ts + remove old processor interface implementation.
src/search-strategies/benchmarking/buffered-indexOf-cancellable/search-strategy.ts	Switch benchmarking imports to .ts.
src/search-strategies/benchmarking/buffered-indexOf-callback/search-strategy.ts	Remove old processor interface implementation; minor cleanup.
src/search-strategies/benchmarking/buffered-indexOf-anchored/search-strategy.ts	Switch benchmarking imports to .ts.
src/search-strategies/benchmarking/buffered-indexOf-anchored-callback/search-strategy.ts	Remove old processor interface implementation.
src/replacement-processors/README.md	Remove processor documentation (superseded by engine layer).
src/replacement-processors/index.ts	Remove processor exports.
src/replacement-processors/types.ts	Remove processor type definitions (SyncProcessor / AsyncProcessor).
src/replacement-processors/replacement-processor.base.ts	Remove processor base + ReplacementContext (moved to engines).
src/replacement-processors/static-replacement-processor.ts	Remove processor implementation (absorbed by sync engine).
src/replacement-processors/static-replacement-processor.test.ts	Remove tests for deleted processor.
src/replacement-processors/function-replacement-processor.ts	Remove processor implementation (absorbed by sync engine).
src/replacement-processors/function-replacement-processor.test.ts	Remove tests for deleted processor.
src/replacement-processors/function-replacement-processor.integration.test.ts	Remove integration tests for deleted processor.
src/replacement-processors/iterable-function-replacement-processor.ts	Remove processor implementation (absorbed by sync engine).
src/replacement-processors/iterable-function-replacement-processor.test.ts	Remove tests for deleted processor.
src/replacement-processors/iterable-function-replacement-processor.integration.test.ts	Remove integration tests for deleted processor.
src/replacement-processors/async-function-replacement-processor.ts	Remove processor implementation (absorbed by async-serial engine).
src/replacement-processors/async-function-replacement-processor.test.ts	Remove tests for deleted processor.
src/replacement-processors/async-iterable-function-replacement-processor.ts	Remove processor implementation (absorbed by async-serial engine).
src/replacement-processors/async-iterable-function-replacement-processor.test.ts	Remove tests for deleted processor.
src/replacement-processors/benchmarking/types.ts	Remove benchmarking processor types (replaced with inline interfaces where needed).
test/utilities.ts	Add engine/slot-tree/concurrency-test utilities and update mock search strategy to include matchToString.
test/harnesses/engine-harness.ts	New harness adapters to run engines/legacy transformers in benchmarks.
test/harnesses/types.ts	Update harness types to engine-era ReplacementContext and remove isStateful.
test/harnesses/regex.ts	Migrate regex harness to SyncReplacementTransformEngine + engine harness adapter.
test/harnesses/regex-canonical.ts	Wrap legacy canonical transformer via legacyHarnessTransformer.
test/harnesses/regex-callback.ts	Update callback harness wiring via new harness adapter.
test/harnesses/regex-anchor-sequence.ts	Migrate harness to engines and engine harness adapter.
test/harnesses/looped-indexOf-cancellable.ts	Migrate harness to engines and engine harness adapter.
test/harnesses/looped-indexOf-callback.ts	Update callback harness wiring via new harness adapter.
test/harnesses/looped-indexOf-anchored.ts	Migrate harness to engines and engine harness adapter.
test/harnesses/indexOf-knuth-morris-pratt.ts	Migrate harness to engines and engine harness adapter.
test/harnesses/buffered-indexOf-generator.ts	Wrap generator-based benchmark strategy via generatorHarnessTransformer.
test/harnesses/buffered-indexOf-canonical.ts	Wrap legacy transformer via legacyHarnessTransformer.
test/harnesses/buffered-indexOf-callback.ts	Update callback harness wiring via new harness adapter.
test/harnesses/buffered-indexOf-anchored.ts	Migrate harness to engines and engine harness adapter.
test/harnesses/buffered-indexOf-anchored-callback.ts	Update callback harness wiring via new harness adapter.
test/harnesses/buffered-indexOf-anchored-async.ts	Migrate async harness to AsyncSerialReplacementTransformEngine.
test/harnesses/buffered-indexOf-anchor-sequence.ts	Migrate harness to engines and engine harness adapter.
test/benchmarks/vitest.config.js	Add Vitest config for the benchmarks workspace.
test/benchmarks/runtime/benchmarks.ts	Update benchmark grouping comments (clarify sync baselines vs async engine runs).
test/benchmarks/README.md	Document new lookahead-engine benchmarks suite.
test/benchmarks/package.json	Add scripts to run lookahead-engine strategy benchmarks.
test/benchmarks/lookahead-engine/utils.ts	Add stats/PRNG utilities for benchmark reporting.
test/benchmarks/lookahead-engine/metrics.ts	Add measurement computation for inter-chunk gaps, timeline, in-flight counts.
test/benchmarks/lookahead-engine/block-design.ts	Add randomized complete block design harness to reduce drift bias.
test/benchmarks/lookahead-engine/report.ts	Add terminal report output + bootstrap CI for ratio comparisons.
test/benchmarks/lookahead-engine/timeline.ts	Add terminal Gantt-chart timeline visualization for concurrency behavior.
test/benchmarks/lookahead-engine/run.bench.ts	Add Vitest-driven virtual-clock benchmark runner with env-based configuration.
test/benchmarks/lookahead-engine/run.sh	Add CLI script wrapper for lookahead-engine benchmarks.
test/benchmarks/lookahead-engine/results/.gitignore	Ignore generated JSON results.
test/benchmarks/algorithm/validate-functionality.test.ts	Update imports to engine-era ReplacementContext.
test/benchmarks/algorithm/comparison.bench.ts	Remove isStateful branching; simplify strategy construction path.
test/benchmarks/algorithm/run.sh	Switch benchmark runner from --experimental-strip-types to --import tsx.
test/benchmarks/algorithm/run-succinct.sh	Switch exporter runner from --experimental-strip-types to --import tsx.
codemods/README.md	Update docs to describe the new two-step v1→v2 migration path.
codemods/package.json	Add script to run the new processor-to-engine codemod.
codemods/transforms/v1-v2/README.md	Expand v1→v2 docs with ordered steps and the processor→engine migration.
codemods/transforms/v1-v2/processor-to-engine.js	New codemod for migrating processor usage to engine usage.
codemods/transforms/v1-v2/replacement-callback-positional-to-context.test.js	Rename variables in tests to match engine terminology.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 128 out of 128 changed files in this pull request and generated 4 comments.

[Copilot]

Pull request overview

Copilot reviewed 128 out of 128 changed files in this pull request and generated 5 comments.