refactor: replace WASM DSL engines with subprocess calls#16
Merged
Conversation
Add compile-time embedding of the harmont Python DSL package and its vendored dependencies (croniter, python-dateutil, six) using include_dir. The WASI Python engine can extract these to a scratch directory without needing filesystem access to the source tree.
Two-stage pipeline: first rewrites harmont imports to globalThis destructuring at the text level, then strips TypeScript types via oxc_transformer (parse → semantic → transform → codegen). Uses only_remove_type_imports to preserve non-harmont value imports. Adds oxc_codegen and oxc_semantic as optional deps under the embedded-typescript feature.
QuickJS 2024-01-13 compiled from source to wasm32-wasip1 using Homebrew LLVM clang + wasi-libc, with POSIX stubs for features unavailable in WASI (termios, dlopen, popen, etc.) and an 8MB WASM stack to support QuickJS's recursive parser. The WasmJsEngine: - Bundles quickjs.wasm via include_bytes! (~940KB) - Evaluates JS scripts via `qjs --std -e <script>` - Preprocesses .harmont/*.ts files (oxc type-strip + import rewrite) - Wraps each file in an IIFE with export-default capture - Concatenates harmont-ts bundle + user code + evaluation footer - Calls harmont.renderEnvelope() and outputs pipeline JSON Wired into engine_for(DslLanguage::TypeScript) in lib.rs.
…feature flags The hm-dsl-engine workspace dependency was not setting default-features = false, causing wasmtime + all embedded engines to be pulled in unconditionally regardless of harmont-cli's feature flags. With this fix, --no-default-features produces a 15 MB binary (same as before embedded engines were added) instead of 33 MB.
…etched at build time
…e network requirement
Python and TypeScript interpreters are embedded in the hm binary via wasmtime, so CI no longer needs to install harmont-py into system Python. Added npm ci for esbuild (needed by build.rs to bundle harmont-ts). The python-lint and ts-dsl jobs that test the DSL libraries themselves are unchanged.
Build scripts legitimately use panic/unwrap/expect for error handling since there is no runtime to propagate errors to.
…harmont/ All examples ship both pipeline.py and pipeline.ts for backwards compatibility. Rather than erroring on mixed languages, prefer TypeScript when both are present.
SystemPythonEngine and SystemNodeEngine were never wired up. SystemNodeEngine was a complete stub. SystemPythonEngine referenced a nonexistent cidsl/py path. The embedded WASI engines are the real implementations.
…ecture Remove wasmtime/QuickJS/oxc WASM engine infrastructure and replace with a system-runtime approach using esbuild-bundled TypeScript and embedded Python sources. This eliminates ~3400 lines of WASM plumbing and 15+ heavy dependencies (wasmtime, wasmtime-wasi, oxc_*, reqwest, sha2, etc). Tasks 1-4 of the tearout plan: - Delete 10 WASM engine files (python_engine, js_engine, wasm_runtime, runtime_cache, ts_preprocess, build_fetch, sha256-polyfill, tests) - Simplify Cargo.toml: remove [features], WASM deps; add which, include_dir - Rewrite build.rs: esbuild-only bundling (two ESM outputs) - Replace embedded_sources.rs with bundled_sources.rs (Python tree + two TS bundles via include_str) - Stub lib.rs with DslEngine trait (engines wired in Tasks 5-7) - Make engine_for synchronous; remove .await in caller - Remove embedded-python/embedded-typescript features from hm crate
Introduces SubprocessPythonEngine which evaluates Python pipeline definitions by spawning python3 as a subprocess with the bundled harmont-py source on PYTHONPATH, replacing the old WASM-based approach.
…uation Detects bun/node on PATH, sets up a temp dir with bundled ESM modules, and runs a runner.mjs script to list or render pipeline definitions from .harmont/*.ts files via subprocess.
…process engines Exercise the full subprocess engine roundtrip (detect language, list pipelines, render pipeline JSON) for both Python and TypeScript DSLs. Tests skip gracefully when the required runtime is not on PATH.
…ss engines All four workflow files (ci, examples, release) no longer reference wasm32-wasip1 or build embedded WASM plugins -- those were removed in PR #12. Integration job now installs croniter + python-dateutil for the Python DSL subprocess engine. CLAUDE.md files updated to document the new DSL engine architecture (subprocess-based, no WASM).
- Remove stale `default-features = false` from workspace dep - Delete dead `render.rs` module (replaced by hm-dsl-engine) - Add `stdin(Stdio::null())` to both subprocess engines - Remove duplicate `serde_json` from dev-dependencies
markovejnovic
changed the title
The TS subprocess engine needs a JS runtime. Dogfood also had a stale wasm32-wasip1 target and installed harmont-py (no longer needed since detect_language prefers TypeScript when both .py and .ts are present).
Node 22's --experimental-strip-types may not handle dynamic .ts imports.
Bump to Node 23 for dogfood and examples jobs.
Also switch anyhow error formatting from `{e}` (outermost only) to
`{e:#}` (full chain) so JS runtime stderr is visible in error output.
Node ESM resolves bare specifiers relative to the importing file, ignoring NODE_PATH. User .ts files under .harmont/ need node_modules/harmont on their resolution path. Create a temporary symlink before subprocess runs, cleaned up via Drop guard.
Matches the Python pipeline which already had laravel=False. Without this, the TS-rendered pipeline runs `php artisan test` which fails because the example has no artisan script.
croniter and python-dateutil are no longer needed at the CLI level — the Python DSL package bundles its own dependencies and is embedded in the binary.
markovejnovic
force-pushed
the
feat/embedded-ts-python
branch
from
316e869 to
4ff489b
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.