Skip to content

runtime-proof-v1.md

Latest commit

 

History

History
232 lines (182 loc) · 9.14 KB

runtime-proof-v1.md

File metadata and controls

232 lines (182 loc) · 9.14 KB
Raw

Workflow Runtime Proof V1 — Implementation Report

Generated: 2026-05-24

1. Summary

Added a complete, deterministic, local proof loop for the Open Workflow Library intelligence pipeline:

prompt
  -> Universal Workflow IR
  -> n8n workflow.json
  -> static n8n compatibility validation
  -> repair proposal (if needed)
  -> learning event (if useful)
  -> human-reviewable queue
  -> proof README

The loop is static-validation only. No workflow was imported into n8n, no node was executed, no external service was called, no LLM was used. Nothing is committed, pushed, or branched.

2. Tools created

Tool Role
tools/export_ir_to_n8n.py IR JSON → n8n workflow JSON (conservative whitelist, placeholders only)
tools/validate_n8n_workflow.py Static n8n compatibility validator
tools/prompt_to_n8n.py Orchestrator (prompt → IR → n8n → validation → proof README)
tools/propose_runtime_repair.py Emits repair proposals from validation output
tools/create_learning_event.py Captures learning events from validation/repair
tools/build_review_queue.py Aggregates everything needing human review

3. Demo cases run

# Prompt (paraphrased) Validation Errors Warnings Nodes Repair Learn
1 website leads → score → CRM → Slack alert PASS 0 0 6 0 0
2 check support tickets every morning, summarize, digest PASS 0 1 4 1 1
3 homecare admin intake → check docs → task → escalate PASS 0 0 4 0 0
4 monitor failed payments → finance tracker → notify CS PASS 0 0 3 0 0
5 content brief → draft task → assign → status update PASS 0 0 4 0 0

All five proof outputs live under reports/runtime-proof/<slug>/ with the full artefact set (prompt.txt, workflow.ir.json, workflow.n8n.json, validation.json, validation.md, repair-proposals.json, repair-proposals.md, learning-events.json, learning-events.md, README.md).

4. Prompt-to-IR outputs

Each demo case wrote a draft workflow.ir.json with generationStatus: planned and validationStatus: needs-review. The IR files satisfy schemas/workflow-ir.schema.json required-field checks.

5. n8n export outputs

Each demo case produced a workflow.n8n.json carrying:

  • top-level name, nodes, connections, settings, active: false, tags
  • a deterministic id and versionId (uuid5)
  • a sticky-note safety disclaimer ("Static validation only — not behaviourally tested. All URLs and credentials are placeholders.")
  • only nodes from the conservative whitelist (manualTrigger, webhook, scheduleTrigger, set, if, httpRequest, code, noOp, respondToWebhook, stickyNote)
  • placeholder URLs on api.example.invalid and placeholder body parameters only

6. Static validation results

All 5 demo workflows: PASS (0 errors).

Warnings observed: 1 total, on the "checks support tickets every morning, summarises urgent tickets" case — because that workflow contains a code (summarise) node, which the validator deliberately flags for review.

7. Repair proposal counts

  • Total runtime repair proposals across the 5 demos: 1.
  • It was produced from the one warning case, classified as n8n-code-node-review, severity medium, with requiresHumanReview: true and status: "proposed".

The pre-existing catalog-wide repair proposals (reports/repair-proposals.json) remain at the V1 cap of 500 (233 high, 267 medium).

8. Learning event counts

  • Total runtime learning events: 1.
  • failureType: review-required, proposed rule n8n-code-node-review-required (wiki-failure-case), confidence 0.60, humanReviewStatus: pending, appliedToWiki: false.

9. Review queue summary

reports/review-queue.json aggregates:

Type Count
high-risk-workflow 100
repair-proposal 50
prompt-to-n8n-proof 5
runtime-repair-proposal 1
learning-event 1
generated-pack-needs-behavioural-test 1
Total 158

Severity: high=150, medium=3, low=5. Every item is status: pending, humanReviewRequired: true.

10. Files created / modified

Created

  • tools/export_ir_to_n8n.py
  • tools/validate_n8n_workflow.py
  • tools/prompt_to_n8n.py
  • tools/propose_runtime_repair.py
  • tools/create_learning_event.py
  • tools/build_review_queue.py
  • docs/runtime-proof.md
  • wiki/generated/runtime-proof-findings.md
  • reports/runtime-proof-v1.{json,md}
  • reports/review-queue.{json,md}
  • reports/runtime-proof/<slug>/... (5 slugs × 10 files = 50 artefacts)

Modified (additive)

  • .github/workflows/audit.yml — added prompt-to-n8n smoke, per-runtime-proof validation, and review-queue build steps.
  • README.md — added a "Workflow Runtime Proof V1" section with the command set and honest limitations.

Refreshed by existing tools (existing behaviour)

  • catalog/workflows.index.json
  • catalog/unified-workflows.index.json
  • reports/workflow-audit.{json,md}
  • reports/unified-catalog-report.{json,md}
  • reports/schema-validation.{json,md}
  • reports/duplicates-report.{json,md}
  • reports/repair-proposals.{json,md}
  • wiki/generated/{README,category-summary,integration-summary,prompt-patterns,risk-review-queue,trigger-patterns}.md

No existing workflow file was deleted or overwritten.

11. Commands run

python tools/audit_workflows.py
python tools/validate_generated_pack.py
python tools/build_unified_catalog.py
python tools/validate_schemas.py
python tools/analyze_duplicates.py
python tools/propose_repairs.py
python tools/build_wiki_seed.py
python tools/prompt_to_n8n.py "Create a workflow that receives website leads, scores them, saves qualified leads to CRM, and alerts Slack."
python tools/prompt_to_n8n.py "Create a workflow that checks support tickets every morning, summarizes urgent tickets, and sends a manager digest."
python tools/prompt_to_n8n.py "Create a workflow for homecare admin intake that receives a referral, checks required documents, sends an internal task, and escalates missing information to a human."
python tools/prompt_to_n8n.py "Create a workflow that monitors failed payments, updates a finance tracker, and notifies the customer success team."
python tools/prompt_to_n8n.py "Create a workflow that receives a content brief, creates a draft task, assigns it to a reviewer, and sends a status update."
python tools/propose_runtime_repair.py --all
python tools/create_learning_event.py --all
python tools/build_review_queue.py
# plus: python tools/validate_n8n_workflow.py reports/runtime-proof/<slug>/workflow.n8n.json   (for each slug)

All commands exited 0.

12. What is actually proven

  • Prompt → IR → n8n export produces structurally valid n8n workflow JSON on five distinct prompts spanning sales, support, regulated-admin, finance, and content domains.
  • The static n8n validator catches the property classes it claims to cover (missing trigger, duplicate names/IDs, broken connection targets, embedded credentials, secret patterns, unsafe HTTP hosts, missing respond/setup notes).
  • Validator findings flow into repair proposals matching schemas/repair-proposal.schema.json with requiresHumanReview: true and status: "proposed".
  • Validator findings flow into learning events matching schemas/learning-event.schema.json with humanReviewStatus: "pending" and appliedToWiki: false.
  • All items needing human review aggregate into reports/review-queue.json and .md, grouped by type and severity.

13. What is not proven

  • The workflows have not been imported into n8n.
  • No node has been executed.
  • No external API has been called.
  • The prompt-to-IR layer remains keyword-rule. It misses plurals, synonyms, and context shifts. Three of the five demo prompts (those whose verbs are plural — "scores", "saves", "monitors", "summarizes", "creates") generated default action steps rather than richer IR shapes.
  • Multi-framework export is not implemented in V1. Only n8n.
  • Self-improvement is not autonomous. Learning events are evidence only.

14. Remaining risks / blockers

  • Plural / synonym handling in tools/prompt_to_ir.py is a known limitation. Out of scope for V1.
  • The static validator does not parse n8n's internal parameter schemas. A workflow can pass and still fail to import into n8n.
  • The review queue caps high-risk surfacing at 100 and duplicate clusters at 50; the full backlog is larger.
  • jsonschema is not installed in the local environment; validate_schemas.py ran in lightweight mode (existing limitation; documented).

15. Recommended next step

Behavioural validation pass: import one runtime-proof workflow into a sandbox n8n instance, replace the placeholders with realistic values, run the trigger, and record success/failure as a new learning event with source: "generation" or source: "ci". Use that result to decide which proposed rule(s) in wiki/generated/runtime-proof-findings.md to promote into wiki/repair-rules/ via a curated, human-reviewed PR.

Final judgment

READY FOR HUMAN DIFF REVIEW.

Not committed. Not pushed. No branch created. No git init run.