Skip to content

feat: add Exo distributed AI cluster as LLM provider#1002

Merged
stephanj merged 1 commit intomasterfrom
feature/exo-provider
Apr 10, 2026
Merged

feat: add Exo distributed AI cluster as LLM provider#1002
stephanj merged 1 commit intomasterfrom
feature/exo-provider

Conversation

@stephanj
Copy link
Copy Markdown
Collaborator

@stephanj stephanj commented Apr 10, 2026

Summary

  • Add Exo as a new local LLM provider for running large AI models across multiple Apple Silicon devices connected via Thunderbolt
  • Auto-discovers downloaded models from the Exo /state API (only shows models on disk)
  • Automatic instance creation with placement preview, background progress bar, and auto-recovery on disconnect
  • Collapsible cluster status panel above the chat window showing nodes, memory usage, GPU%, temperature, and active instance status
  • Dedicated Docusaurus documentation page with screenshots and troubleshooting guide
  • Fix: MarkdownConversationRenderer missing createConversationJEditorPane method

New files

File Purpose
ExoChatModelFactory.java Factory creating OpenAI-compatible chat models with async instance preparation
ExoModelService.java Manages Exo API calls: model discovery, instance lifecycle, cluster state, progress reporting
ExoClusterPanel.java Custom-painted collapsible panel showing cluster topology and node stats
ExoModelDTO.java / ExoModelEntryDTO.java DTOs for Exo /models API response
docusaurus/docs/llm-providers/exo.md Full documentation page with setup, configuration, troubleshooting

Screenshots

Cluster panel in IDE:
exo-view

Progress bar during instance loading:
exo-warmingup

Test plan

  • Enable Exo in Settings > LLM Providers, verify URL defaults to http://localhost:52415/v1/
  • Select Exo provider — only downloaded models appear in dropdown
  • Select a model — background progress bar shows instance creation
  • Send chat message — response received from Exo cluster
  • Verify cluster panel shows nodes with memory/GPU stats
  • Click cluster panel header — toggles collapsed/expanded
  • Switch to another provider — cluster panel disappears
  • Disconnect Exo instance — verify auto-recovery triggers on next chat
  • Run ExoChatModelFactoryTest, ExoModelServiceTest, ExoModelDTOTest

🤖 Generated with Claude Code

Add Exo as a new local LLM provider that enables running large AI models
across multiple Apple Silicon devices connected via Thunderbolt.

Features:
- Exo provider with auto-discovery of downloaded models from /state API
- Automatic instance creation with placement preview across cluster
- Background progress bar during model loading with cancellation support
- Auto-recovery when instances disconnect or get recycled
- Collapsible cluster status panel above chat showing nodes, memory,
  GPU usage, temperature, and active instance status
- OpenAI-compatible API integration via Langchain4j
- Dedicated Docusaurus documentation page with screenshots
- Fix: MarkdownConversationRenderer missing createConversationJEditorPane

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@stephanj stephanj merged commit a34c4d3 into master Apr 10, 2026
8 of 10 checks passed
@stephanj stephanj deleted the feature/exo-provider branch April 10, 2026 16:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant