extension-roadmap.md

Extension runtime roadmap

This roadmap turns the programmable-runtime architecture into concrete engineering phases.

The guiding principle is simple:

Keep the default product polished in Go. Expose sharp extension-host primitives for optional customization.

Lua is the first runtime adapter, not the extension architecture itself. The host should eventually support multiple adapters such as Lua, shell hooks, toolbox executables, MCP, and experimental Go-like runtimes.

Current checkpoint

The runtime currently has:

• a runtime-adapter seam with Lua as the built-in adapter
• trusted Lua extension loading
• commands and extension tools
• event handlers/autocmds with priority, consume, and stop
• generic keymap targets by buffer/window/role/global
• runtime buffers
• runtime windows
• layout get/set
• low-level UI draw/cursor operations
• render and resize events
• per-window renderer ownership
• canonical composer buffer
• generic structured buffer blocks with bounded transcript block exposure

This is a strong foundation. Go intentionally owns stock chat rendering and assistant orchestration by default.

Important boundary: extensions are optional control/customization layers; Go remains the product core and fast terminal rendering backend. Complex hot renderers should not migrate to an interpreted extension runtime by default.

Immediate cleanup

0. Keep the host/runtime split

Extension APIs should be designed against the runtime-neutral host: events, commands, tools, keymaps, buffers, windows, layout, UI operations, and diagnostics. Lua-specific details belong in the Lua adapter.

Do not design new lifecycle or tool middleware APIs only as Lua callbacks. Define the host contract first, then expose it through runtime adapters.

Future runtime candidates:

• Lua: lightweight scripting, keymaps, commands, hooks, overlays
• shell hooks: deterministic team automation
• toolbox executables: custom tools without embedding a language runtime
• MCP: external tool servers
• experimental Go-like runtime adapters: possible future path for typed extensions

1. Do not add transcript-specific host writes

Do not merge or reintroduce host APIs like:

• librecode.transcript.append
• librecode.transcript.clear

Use generic buffer operations instead. If convenience is needed, implement it as a Lua helper module.

2. Keep transcript snapshots bounded

Any transcript/message exposure during render/stream events must be bounded by count and text length.

The previous render-loop slowdown came from rebuilding too much transcript text too often. Avoid repeating that mistake.

3. Treat composer, transcript, status, thinking, and tools as conventional default buffers

They are useful names and roles, but they should not become special host APIs.

Phase 0: explicit extension manager

Status: planned; see docs/extension-manager.md.

Goal: make extension dependencies explicit, installable, lockable, and reproducible without embedding official extensions in the binary.

Planned interface:

extensions:
  enabled: true
  use:
    - official:vim-mode
    - github:user/ext
    - source: github:user/ext
      version: v1.2.3
    - path:.librecode/extensions/local-dev

Planned commands:

• librecode extension list
• librecode extension add <source> [--version vX.Y.Z]
• librecode extension remove <source-or-name>
• librecode extension install
• librecode extension update
• librecode extension tidy
• librecode extension doctor

Design constraints:

• startup loads only entries declared in extensions.use
• extra directories on disk are ignored unless explicitly declared
• add installs immediately and updates config + lock
• remove tidies immediately and updates config + lock
• lockfiles pin human-readable versions/tags first
• official extensions are installed through the manager, not embedded into the binary

Phase 1: generic structured buffers

Status: implemented for the current runtime surface.

Goal: make transcript/message-like data possible without transcript-specific Go APIs.

Completed generic operations:

• buf.clear(name)
• buf.append(name, value) for text or structured blocks
• buf.get_blocks(name, start, stop)
• buf.set_blocks(name, start, stop, blocks)
• buf.delete_blocks(name, start, stop)
• buf.get_var(name, key) / buf.set_var(name, key, value)

Design constraints:

• operations must be bounded for render/stream events
• mutation must happen through an active event or scheduled transaction
• block schema should be generic: kind, text, metadata, plus optional application fields like role
• transcript should be one consumer of structured buffers, not its own host API family

Phase 2: optional Lua helper modules

Goal: give users ergonomic APIs without bloating the Go kernel.

Helper modules may live under a configured extension root and wrap primitives for a project or workflow. librecode should not depend on auto-loaded bundled Lua helpers for the stock UX.

Example direction:

local chat = require("my_workflow.chat")
chat.append_note("hello")

Internally, this should use buf, win, layout, ui, event, and action primitives.

Phase 3: optional UI overrides where parity is ready

Goal: let users customize the terminal UI without degrading the stock Go experience.

Keep default UI surfaces in Go:

• status/footer
• composer behavior/rendering
• prompt history UX
• autocomplete presentation
• default layout construction
• transcript/thinking/tool presentation

Allow opt-in Lua extensions to overlay or own windows when they can preserve parity and performance.

The failed transcript migration showed the boundary: Lua should not manually reimplement complex hot renderers before Go provides the right generic primitives.

Phase 4: render model improvements

Goal: make extension rendering powerful enough for full reskins while keeping hot terminal work in Go.

First-pass implemented:

• ui.measure
• ui.truncate
• ui.pad_right
• ui.wrap
• ui.draw_lines
• ui.draw_spans
• ui.draw_box
• ui.clear_region
• ui.viewport
• ui.virtual_list
• ui.draw_batch
• ui.theme_tokens

Still add:

• namespace-scoped highlights
• extmarks/virtual text or equivalent annotations
• richer window viewport/scroll APIs
• per-window renderer registration helpers

Keep raw draw operations available. Higher-level rendering should be Lua-composable, but measuring/wrapping/caching should use Go-backed primitives.

Phase 5: lifecycle events

Goal: add structured agent lifecycle events without making the default UI extension-owned.

Implementation should happen as a stack of small PRs. Each PR must define runtime-neutral Go contracts first, expose them through Lua second, and keep default behavior unchanged when no extension handles the event. The executable kanban lives under .librecode/work/plans/ so stable docs do not become scratch state.

Event stream vs middleware dispatcher

librecode uses two different mechanisms on purpose:

1. Reactive event stream — ro-backed, observational, async/fanout. Use this for lifecycle telemetry, logs, UI notifications, headless JSON streams, retry/timer streams, future file watchers, and other data that flows over time without needing an ordered return value.
2. Middleware dispatcher — ordered, synchronous, return-value driven. Use this for tool allow/reject/modify decisions, context contributions, provider request mutation, and any hook that must deterministically change runtime behavior.

Rule of thumb: if it is async, streaming, cancellable, or fanout, build it on the ro event spine. If it returns a decision or mutation, keep it in the explicit middleware dispatcher and emit observational ro events before/after the decision.

Phase 5.1: lifecycle event contracts

Add typed event names, payloads, results, and dispatch diagnostics for the agent loop. This phase should not change assistant behavior. It gives later phases a stable host contract.

Phase 5.2: session, input, and turn events

Status: implemented for prompt-time input/session/turn events.

Implemented events:

• input
• prompt_prepare
• session_start
• session_load
• before_agent_start
• agent_start
• turn_start
• message_append
• turn_end
• agent_end

The events are bounded and observational. They do not include full transcript history and do not currently mutate prompt text or session state.

Phase 5.3: context build seams

Add a typed context_build event for bounded context contributions and context accounting. Extensions may add labeled context blocks within a budget. The runtime should expose system/history/skills/extensions token estimates for debugging.

Phase 5.4: provider lifecycle hooks

Emit before_provider_request, after_provider_response, and provider_error with redacted, bounded payloads. Request mutation must be conservative, typed, and logged. Auth headers and secrets must never be exposed.

Initial lifecycle events:

• session_start
• session_load
• session_save
• session_shutdown
• resources_discover
• input
• prompt_prepare
• before_agent_start
• agent_start
• turn_start
• context_build
• before_provider_request
• after_provider_response
• provider_error
• tool_call
• tool_result
• tool_error
• message_append
• turn_end
• agent_end
• shutdown

Events should have structured payloads, bounded data, clear mutation contracts, extension timing diagnostics, and tests for both default behavior and extension-modified behavior.

Phase 6: tool middleware and extension tools

Goal: let extensions and hooks influence tool execution through explicit middleware contracts.

Phase 6.1: tool runtime hooks

Run tool lifecycle hooks around tool execution. librecode stays YOLO by default: hooks are not a built-in permission gate, but they can normalize arguments, annotate calls, redact or rewrite results, and add diagnostics. Users who want policy gates can implement them as extensions that return tool errors or synthetic results.

Implemented contract:

• tool_call can return tool_call.arguments to replace/merge tool arguments before execution
• tool_result can return tool_result.result, tool_result.details_json, and/or tool_result.error before the model sees the result
• hook failures are surfaced as tool errors instead of silently mutating state

Phase 6.2: unified tool registry

Replace ad-hoc built-in registry creation in the assistant loop with one model-visible registry that can hold built-ins, extension tools, future MCP tools, toolboxes, skill-bundled tools, subagents, and LSP tools.

Extension-registered tools should be wired into the same model-visible tool registry as built-ins. Skill-bundled tools should activate only when the skill activates.

Phase 7: hooks, skills, and subagents

Goal: add pragmatic workflow building blocks for teams and project-specific automation.

Add:

• deterministic shell hooks for teams (PreToolUse, PostToolUse, UserPromptSubmit, Stop, SessionStart, SessionEnd)
• skill-bundled MCP configuration through mcp.json
• skill-bundled toolboxes/scripts for simpler custom tools
• AGENTS.md hierarchy and subtree instruction loading
• markdown-defined subagents exposed through a Task tool
• /spec planning mode with optional separate planning model

Phase 8: assistant/model/tool/session primitives

Goal: allow extensions to replace or deeply reshape the assistant loop.

Expose primitive capabilities, not chat policies:

• model stream request primitive
• tool run primitive
• session read/write primitive
• config read/write primitive
• store/key-value primitive for extension state
• job/process spawning (job.spawn, job.stop) and schedule(fn)

Keep default assistant orchestration in Go; allow extensions to override or wrap it explicitly.

Phase 9: bare runtime mode

Goal: make librecode usable as a programmable terminal runtime without the stock chat app.

Possible shape:

librecode --bare
librecode --extension ./my-app.lua

Bare mode should load the kernel and selected extensions, but not the default chat distribution.

Anti-roadmap

Avoid these directions:

• a growing family of composer.*, transcript.*, thinking.*, or chat.* host APIs
• special extension hooks for one bundled feature when generic events/keymaps/buffers can handle it
• unbounded snapshots in hot render/stream paths
• hidden state mutation outside event transactions
• rewriting complex mature Go renderers in Lua before primitive parity exists
• making product policy live in the Go stock renderer instead of moving policy to Lua

It is acceptable to improve Go's generic rendering primitives; it is not acceptable to add transcript-specific rendering APIs just to paper over missing primitives.

Definition of done for the architecture

The architecture is in the target state when:

1. The default chat UI is fast and polished with extensions disabled.
2. Extensions can customize buffers/windows/events through public primitives.
3. Optional composer modes or render overrides need no private host support.
4. Transcript, statusline, thinking, and tool presentation remain Go-owned by default but expose bounded data and opt-in override points.
5. Extensions can wrap or replace assistant flow using model/tool/session primitives.
6. Render/stream performance stays bounded regardless of session history size.
7. Complex Lua-owned renderers use Go-backed generic rendering primitives rather than ad hoc Lua string math.