Skip to content

Add experimental ZenBabel portable JSON steps#4836

Draft
strickvl wants to merge 9 commits into
developfrom
feature/zenbabel-portable-json
Draft

Add experimental ZenBabel portable JSON steps#4836
strickvl wants to merge 9 commits into
developfrom
feature/zenbabel-portable-json

Conversation

@strickvl
Copy link
Copy Markdown
Contributor

Summary

This PR adds an experimental ZenBabel Level 1 path for portable JSON step execution.

The goal is intentionally narrow: keep ZenML's existing Python control plane and orchestrator model, but allow a static pipeline step body to be executed by a non-Python command, demonstrated with TypeScript.

Concretely, this adds:

  • an optional StepExecutionSpec on StepSpec for language/protocol/command/source identity metadata
  • an experimental zenml-portable-json-v1 protocol
  • a PortableStepRunner sibling to StepRunner that stages JSON inputs, launches a subprocess, reads JSON outputs, and stores results through the existing materializer path
  • StepLauncher routing for portable JSON steps while keeping normal Python steps unchanged
  • an experimental zenml.zenbabel namespace with adapter/importer/authoring helpers
  • a mixed Python + TypeScript example under examples/zenbabel_mixed_static/
  • a docs page explaining the experimental TypeScript step pattern
  • unit coverage for spec identity, cache identity, routing, importer behavior, authoring helpers, and portable runner success/failure cases

Why

This is a proof of the execution seam for future multi-language ZenML support.

The important architectural claim is not that ZenML has a production TypeScript SDK yet. It does not. The claim is that an existing ZenML orchestrator can still schedule the normal Python entrypoint, and the final step-body execution can route to a portable subprocess when a step declares zenml-portable-json-v1.

That keeps the stack/orchestrator model intact while giving us a concrete substrate for later TypeScript-authored pipelines.

Current limitations

This is experimental and deliberately limited:

  • only static pipelines are supported for portable steps
  • only JSON-compatible artifact values are supported
  • the ZenML control plane and step entrypoint are still Python
  • the TypeScript SDK does not exist yet
  • dashboard/source display still mostly sees the Python adapter/placeholder surfaces
  • subprocess logs are not yet presented as first-class TypeScript logs
  • remote validation has been done on Kubernetes/EKS, not every orchestrator

Validation

Local checks run during the branch work included:

  • targeted unit tests for the new config, runner, launcher, cache, and ZenBabel helper/importer paths
  • uv run ruff check examples/zenbabel_mixed_static/run.py
  • uv run ruff format --check examples/zenbabel_mixed_static/run.py
  • uv run python examples/zenbabel_mixed_static/run.py --compile-only
  • local Docker end-to-end execution of the mixed Python/TypeScript example
  • Kubernetes/EKS end-to-end execution of the same example against a branch-compatible staging server and AWS stack

The Kubernetes validation ran the full pipeline shape:

load_data completed
score_or_transform completed
summarize completed
Orchestrator pod finished

Reviewer notes

The highest-value review areas are:

  1. Whether StepExecutionSpec is the right place/shape for portable execution identity.
  2. Whether routing in StepLauncher keeps enough of the existing bookkeeping path intact.
  3. Whether PortableStepRunner is strict enough about JSON boundaries and subprocess failures.
  4. Whether the experimental zenml.zenbabel namespace makes the example readable without implying stable SDK support.
  5. Whether the docs/example language is honest enough about the current limitations.

@strickvl strickvl added enhancement New feature or request internal To filter out internal PRs and issues no-release-notes Release notes will NOT be attached and used publicly for this PR. labels May 15, 2026
@strickvl strickvl requested a review from safoinme May 15, 2026 10:04
@github-actions
Copy link
Copy Markdown
Contributor

⚠️ This PR has been inactive for 2 weeks and has been marked as stale.
Timeline:

  • Week 2 (now): First reminder - PR marked as stale
  • Week 4: PR will be automatically closed if no activity
    Please update this PR or leave a comment to keep it active. Any activity will reset the timer and remove the stale label.

@github-actions github-actions Bot added the stale label May 30, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request internal To filter out internal PRs and issues no-release-notes Release notes will NOT be attached and used publicly for this PR. stale

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant