Skip to content

README.md

Latest commit

 

History

History
306 lines (235 loc) · 16.9 KB

README.md

File metadata and controls

306 lines (235 loc) · 16.9 KB
Raw

Project Overview: WilmerAI

WilmerAI is a middleware system that acts as an orchestration engine between front-end applications and various Large Language Model (LLM) backends. It allows you to build complex, multi-step AI behaviors using a node-based workflow system, all while presenting a simple, industry-standard API to your existing tools.

Instead of being a single model, WilmerAI is a highly configurable engine that lets you chain together different models, tools, and memory systems to process a single user request.

The Core Concept: The Workflow Engine

At the heart of WilmerAI is a node-based workflow engine. A workflow is a JSON file that defines a sequence of steps, or "nodes," to be executed in order. This allows you to create multi-step logic that goes far beyond a simple single-LLM response.

  • Nodes as Building Blocks: Each node in a workflow performs a specific task, such as calling an LLM, searching a database, running a Python script, or retrieving conversational memory.
  • Sequential Data Flow: The output of one node is automatically made available to all subsequent nodes. For example, the first node could extract keywords from a user's prompt, and the second node could use those keywords to perform a vector memory search. The third node could then take the search results and the original prompt to generate a final, context-aware answer.
  • Flexibility and Control: This architecture gives you precise control over the entire request-response lifecycle, allowing for the creation of specialized agents, Retrieval-Augmented Generation (RAG) pipelines, and dynamic, multi-tool automations.

Key Capabilities

WilmerAI integrates several features into its workflow engine, enabling you to build highly customized and intelligent systems.

Adaptable API Gateway

WilmerAI emulates the APIs of popular services like OpenAI and Ollama. This allows you to connect your existing front-end applications directly to WilmerAI without any code changes. The client application believes it is communicating with a standard LLM service, while WilmerAI orchestrates complex workflows in the background.

Endpoint Behavior

How a client application should format its requests depends on the type of endpoint it is connecting to.

  • Chat Completion Endpoints (e.g., /v1/chat/completions): When using these endpoints, the client sends a structured array of messages (e.g., with "role" and "content" pairs). WilmerAI processes this format directly. Generation parameters like temperature, top_k, etc., are controlled by the WilmerAI workflow, overriding most settings configured in the client application.

  • Text Completion Endpoints (e.g., /v1/completions or /api/generate): When connecting to these legacy-style endpoints, WilmerAI requires the client to format the prompt using a specific structure. This allows the middleware to correctly parse the conversation history. The format uses tags to separate the system prompt, user messages, and assistant messages, with no spaces or newlines between the tags.

    • Tag Structure:
      • [Beg_Sys]: Marks the beginning of the system prompt.
      • [Beg_User]: Marks the beginning of a user's turn.
      • [Beg_Assistant]: Marks the beginning of the assistant's turn.
    • Example: 
      [Beg_Sys]You are an intelligent AI Assistant.[Beg_User]SomeOddCodeGuy: Hey there![Beg_Assistant]Wilmer: Hello![Beg_User]SomeOddCodeGuy: This is a test[Beg_Assistant]

Flexible Backend Connections

You are not locked into a single LLM provider. WilmerAI's Adaptable LLM Connector can interface with any LLM backend, including local models via Ollama or KoboldCpp and cloud services from OpenAI. Furthermore, different nodes within a single workflow can be configured to use different models, allowing you to optimize for cost, speed, and capability at each step of a process.

Advanced Routing and Logic

WilmerAI provides two layers of routing for creating dynamic and adaptable agents:

  1. Prompt Routing: An initial routing engine can analyze the user's intent and direct the incoming request to the most appropriate workflow. For example, it can automatically send coding questions to a coding-specific workflow and factual queries to a research-focused one.
  2. In-Workflow Routing: Within a workflow, conditional "if/then" logic can be implemented. A node can make a decision based on the output of a previous step and execute a specific sub-workflow, enabling non-linear execution paths.

Modular and Reusable Workflows

Workflows can be nested, allowing you to build a task once and reuse it anywhere. A common process, such as summarizing text or searching a database, can be encapsulated in its own "child" workflow. This child workflow can then be called as a single node from any "parent" workflow, simplifying design and eliminating redundant logic.

Stateful Conversation Memory

WilmerAI includes a four-part memory system that provides long-term context for conversations. By including a [DiscussionId] in your request, you can enable:

  • Long-Term Memory File: Chronological, summarized chunks of the conversation.
  • Rolling Chat Summary: A continuously updated high-level summary of the entire discussion.
  • Searchable Vector Memory: A dedicated full-text search database for the discussion, allowing for efficient keyword-based search to retrieve relevant information, with optional embedding-based semantic or hybrid search when an embeddings endpoint is configured.
  • State Document: A continuously updated markdown snapshot of what is currently true in the conversation (a user profile, world state, etc.), maintained by the vector memory pipeline and readable by workflows at any time.

Per-User Encryption and Data Isolation

When a client sends an Authorization: Bearer <key> header, WilmerAI stores all discussion files in an isolated directory derived from the API key. This allows multiple users or applications to share a single WilmerAI instance without risk of data leakage. Directory isolation activates automatically based on the presence of the header. To also encrypt files at rest, set "encryptUsingApiKey": true in your user configuration file. Without an API key, behavior is unchanged. See Per-User Encryption and Data Isolation for full details.

External Tool Integration

The workflow engine can be extended with custom tools. This includes nodes for running local Python scripts or connecting to external services. A built-in example is the Offline Wikipedia Integration, which allows a workflow to query a local Wikipedia database to provide factual context for a response.

Tool Calling and Structured Output

Front-ends that use OpenAI-style tool calling work through WilmerAI end to end: tool definitions pass through to the backend, tool call responses relay back, and multi-round tool loops work reliably through authored-prompt workflows via native delivery of the live tool exchange. On backends with constrained decoding, demanded tool calls (tool_choice forced or "required") are grammar-enforced, and any workflow node can pin its output to a JSON Schema so downstream nodes consume guaranteed-parseable JSON. See Tool Calling and Structured Output.

How It All Works: A Typical Request Flow

  1. Standard API Request: A client application sends a request to one of WilmerAI's API endpoints (e.g., /v1/chat/completions) in a standard format.
  2. Workflow Selection: The engine receives the request. If configured, the Prompt Routing engine analyzes the prompt's intent to select the most appropriate workflow file.
  3. Node Execution: The Workflow Engine begins executing the nodes defined in the selected workflow, one by one.
  4. Orchestration: As the workflow runs, nodes might call different LLMs, query the vector memory database for relevant context, execute a nested sub-workflow for a specialized task, or run a Python script. The output from each step is passed along for the next step to use.
  5. Response Generation: One node in the workflow is designated as the "responder." Its final output is captured by the engine.
  6. Formatted API Response: WilmerAI formats the responder's output into the API schema the original client application expects (e.g., an OpenAI-compliant JSON object) and sends it back, completing the cycle.

Directory Breakdown

WilmerAI
│
├─ Middleware
│
├─ Public
│  └─ Configs
│     ├─ ApiTypes
│     ├─ Endpoints
│     │  ├─ folders named after usernames
│     │  └─ ...
│     ├─ Presets
│     │  ├─ folders named for specific types
│     │  │  ├─ folders named after usernames
│     │  │  └─ ...
│     │  └─ ...
│     ├─ PromptTemplates
│     ├─ Routing
│     ├─ Users
│     └─ Workflows
│        ├─ folders named after usernames
│        └─ ...
│
├─ run_macos.sh
├─ run_windows.bat
└─ server.py

Description of Directories and Key Files

Middleware/

This is the application's core logic. The code for the application can be found within here. This is essentially the src folder

Public/

Contains all user-facing JSON configuration files.

  • ApiTypes/: Contains json files that define the schemas for different LLM APIs, specifying property names for things like streaming or max_tokens. These are utilized in the Endpoint configs
  • Endpoints/: Contains json files that specify connection details for an LLM API (e.g., URL, API Type) that Wilmer will make calls to. Every LLM that the user intends to use in their workflows must be specified here. Every workflow node can specify a different endpoint.
  • Presets/: Contains json files with LLM generation parameters (temperature, top_k, etc.). These are applied per workflow node.
  • PromptTemplates/: Contains the json files that specify various prompt templates. Used in Endpoint configs
  • Routing/: Contains json files that specify the central semantic router instructions for users/workflows that do routing. You specify the domains you are routing to here, and what workflows they correspond with.
  • Users/: Contains json files with all of the specific settings for a user, including things like what port the app runs on, where to connect to things like the offline wikipedia api, and where certain files are saved, go here.
  • Workflows/: Contains json files that define the sequence of nodes for each workflow. Workflows are organized into subfolders, typically named after each user (e.g., Workflows/<username>/). The subfolder used can be customized via the workflowConfigsSubDirectoryOverride setting in the User config.
    • _shared/: A special folder for shared workflows. Workflows placed directly in _shared/ are listed by the /v1/models and /api/tags endpoints, allowing front-end applications to select them via the model dropdown. See "Workflow Selection via Model Field" below for details.

run_macos/run_windows

Scripts to automatically generate a venv, install the requirements.txt for the app, and run the application by calling server.py. Takes the following optional parameters:

  • --ConfigDirectory: String input that specifies where the Public/Configs folder is at.
  • --User: String input that specifies the name of the user you'd like to start the app as. Can be repeated for multi-user mode.
  • --port: Integer input that specifies the port to listen on. In single-user mode, falls back to the user's config. In multi-user mode, defaults to 5050.
  • --listen: Listen on the network. With no value, binds to 0.0.0.0 (all interfaces). Optionally accepts a specific address.
  • --concurrency: Integer input that sets the max concurrent requests (or LLM calls in endpoint mode). 0 = no limit. Default: 1.
  • --concurrency-timeout: Integer input that sets the seconds to wait for a concurrency slot before returning 503. Default: 900.
  • --concurrency-level: wilmer or endpoint. Selects where the gate is enforced. wilmer (default) gates at the request boundary; endpoint lifts the request gate and serializes only outbound LLM API calls, allowing reentrant requests (workflows that call back into the same Wilmer instance) to make progress. Default: wilmer.
  • --file-logging: Enable file logging. In single-user mode, falls back to the user's useFileLogging config setting. In multi-user mode, defaults to off.
  • --LoggingDirectory: Directory for log files. When unset, defaults to {PublicDirectory}/logs/ if --PublicDirectory is provided, otherwise {install_dir}/Public/logs/. The default is install-pinned (derived from the location of server.py on disk) and does not depend on the current working directory.

server.py

Main script of the app.

Workflow Selection via Model Field

When loading a user that has allowSharedWorkflows set to true, WilmerAI allows front-end applications to select specific workflows by using the model field in API requests. This enables users to switch between different workflows directly from their front-end's model dropdown without changing configuration files.

How It Works

  1. Models List Endpoints: The /v1/models (OpenAI) and /api/tags (Ollama) endpoints return a list of available models. In single-user mode, these are workflows from _shared/ in username:workflow format. In multi-user mode, models are aggregated from all configured users.

  2. Workflow Selection: When your front-end sends a request with a model field like "model": "chat-ui:general", WilmerAI extracts the workflow name and executes that workflow directly from the _shared folder. In multi-user mode, if chat-ui matches a configured user, that user's config is used for the request.

  3. Response Format: Responses include the model in username:workflow format when a workflow override is active, allowing your front-end to maintain the selected workflow across requests.

Multi-User Mode

A single Wilmer instance can serve multiple users. Instead of running separate instances for each user, start one instance with multiple --User flags:

bash run_macos.sh --User user-one --User user-two --User user-three

Each user must have a configuration file in Public/Configs/Users/ (e.g., user-one.json, user-two.json). In multi-user mode, the per-user port setting is ignored; use --port on the command line (defaults to 5050). All users share the same concurrency gate, which serializes requests to shared LLM hardware regardless of which user they belong to.

The models endpoint will list models from all configured users. A front-end can target a specific user by sending "model": "user-two" or "model": "user-two:general" in the request.

Folder Structure

Public/Configs/Workflows/
├── _shared/
│   ├── general/                    # Listed by models endpoint as folder name
│   │   └── _DefaultWorkflow.json   # Workflow loaded when folder is selected
│   ├── fast/
│   │   └── _DefaultWorkflow.json
│   ├── general-reasoning/
│   │   └── _DefaultWorkflow.json
│   ├── fast-reasoning/
│   │   └── _DefaultWorkflow.json
│   └── task/
│       └── _DefaultWorkflow.json
├── example-user/                   # A user's own workflow folder (optional)
│   └── ...

Usage

  1. Create a folder in Public/Configs/Workflows/_shared/ with your workflow name (e.g., general/)
  2. Place a _DefaultWorkflow.json file inside the folder
  3. Folder names become the selectable model names (e.g., general/ → chat-ui:general)
  4. Query /v1/models or /api/tags from your front-end to see available workflows
  5. Select a workflow from the model dropdown in your front-end application

Model Field Formats

The API accepts these model field formats:

Format Example Behavior
username:workflow chat-ui:general Uses the specified workflow (and user in multi-user mode)
username chat-ui In multi-user mode, routes to that user's default workflow
workflow general Uses workflow if it exists in _shared/
Anything else gpt-4 Falls back to normal routing

workflowConfigsSubDirectoryOverride

The workflowConfigsSubDirectoryOverride user config setting loads workflows from a named subfolder under Public/Configs/Workflows/ instead of the username folder. For example, setting "workflowConfigsSubDirectoryOverride": "coding-workflows" loads workflows from Public/Configs/Workflows/coding-workflows/.

Upgrade note: this setting previously resolved under Public/Configs/Workflows/_overrides/<value>/. It now resolves directly under Public/Configs/Workflows/<value>/; if you used the old location, move your override folder up out of _overrides/.