Skip to content

Commit c003140

Browse files
odinshen-gitodinshen-git
committed
Add persistent local AI runtime learning path
1 parent 7cc6914 commit c003140

13 files changed

Lines changed: 3581 additions & 0 deletions

‎content/learning-paths/laptops-and-desktops/dgx_persistent_agent/1_introduce.md‎

Lines changed: 259 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,259 @@
1+
---
2+
title: Understand Persistent AI Runtime Architecture
3+
weight: 2
4+
5+
### FIXED, DO NOT MODIFY
6+
layout: learningpathall
7+
---
8+
9+
## Understand Persistent AI Runtime Architecture
10+
11+
In this Learning Path, you will build a ***persistent local AI runtime*** on NVIDIA [DGX Spark](https://www.nvidia.com/en-gb/products/workstations/dgx-spark/). The implementation is validated on DGX Spark, but the architecture also applies to other ***Arm Cortex-A platforms*** that can run containerized services and local AI runtimes.
12+
13+
The final system is not a single chatbot process. It is a set of local services that run continuously, share a workspace, react to file events, generate summaries, create embeddings, store vector memory, retrieve context, and periodically reason about the state of the workspace.
14+
15+
The core idea is: ***AI systems are orchestration systems, not just inference systems.***
16+
17+
DGX Spark is well suited to this type of workload because it combines ***Arm CPU orchestration*** with local GPU acceleration. In the [Grace Blackwell architecture](https://learn.arm.com/learning-paths/laptops-and-desktops/dgx_spark_llamacpp/1_gb10_introduction/), the Arm Grace CPU coordinates background services, filesystem events, scheduling, document processing, metadata handling, and service-to-service communication. The Blackwell GPU accelerates ***local LLM inference***, token generation, summarization, and embedding generation.
18+
19+
By the end of this Learning Path, you will have a local runtime with these capabilities:
20+
21+
| Capability | Runtime component |
22+
|---|---|
23+
| Local LLM inference | [Ollama](https://ollama.com/) |
24+
| Persistent vector memory | [Qdrant](https://qdrant.tech/) |
25+
| Workspace orchestration | Hermes Agent |
26+
| Browser-based interaction | [Open WebUI](https://github.com/open-webui/open-webui) |
27+
| Semantic retrieval | Hermes Agent + Qdrant + Ollama |
28+
| Autonomous workspace cognition | Hermes Agent + Ollama |
29+
30+
## Runtime Architecture Overview
31+
32+
The runtime uses four containerized services:
33+
34+
- Hermes Agent
35+
- Ollama
36+
- Qdrant
37+
- Open WebUI
38+
39+
These services communicate over a local Docker network and share a persistent workspace on the host.
40+
41+
```text
42+
+------------------+ HTTP API +------------------------------+
43+
| Open WebUI | -----------------> | Ollama Container |
44+
| User interface | | Local inference runtime |
45+
+------------------+ +--------------^---------------+
46+
|
47+
| inference
48+
| embeddings
49+
|
50+
+----------------+---------------+
51+
| Hermes Container |
52+
| CPU-side orchestration |
53+
+----------+-------------+-------+
54+
| |
55+
| files | vectors
56+
| events | metadata
57+
v v
58+
+----------+----+ +----+----------+
59+
| Shared | | Qdrant |
60+
| workspace | | vector memory |
61+
+---------------+ +---------------+
62+
```
63+
64+
The important architectural pattern is ***separation of responsibilities***. Each service has a narrow role, and Hermes coordinates the overall workflow.
65+
66+
| Layer | Service | Purpose |
67+
|---|---|---|
68+
| Interaction layer | Open WebUI | Provides browser-based access to local models |
69+
| Inference layer | Ollama | Runs local language and embedding models |
70+
| Memory layer | Qdrant | Stores and searches vector memory |
71+
| Orchestration layer | Hermes Agent | Watches files, schedules work, coordinates services |
72+
73+
## Runtime Components
74+
75+
### Hermes Runtime
76+
77+
Hermes is the ***orchestration runtime*** you will build in this Learning Path.
78+
79+
It runs as a persistent Python service inside a container. It watches the shared workspace, detects new files, reads documents, sends requests to Ollama, stores memory in Qdrant, performs semantic retrieval, and later generates autonomous workspace summaries.
80+
81+
Hermes is responsible for:
82+
83+
- Filesystem monitoring
84+
- Workflow orchestration
85+
- Runtime scheduling
86+
- Document parsing
87+
- Prompt preparation
88+
- Inference coordination
89+
- Memory coordination
90+
- Autonomous cognition
91+
92+
Hermes does not run the language model itself. Instead, it coordinates AI workflows across local services.
93+
94+
This is the main CPU-side workload in the system. The Arm CPU keeps the runtime alive, schedules background loops, tracks file events, moves data between services, and manages runtime state.
95+
96+
### Ollama Runtime
97+
98+
Ollama provides the local inference runtime in this Learning Path. It is used because it is a convenient way to run local models and expose a simple API, but the architecture is not limited to Ollama.
99+
100+
Conceptually, Ollama is one possible ***inference backend***. Hermes can orchestrate any local or remote inference service that exposes a compatible API, such as llama.cpp server, vLLM, a custom PyTorch service, or another model runtime.
101+
102+
In this Learning Path, Hermes uses Ollama for two types of model calls:
103+
104+
- Chat completion, using [`qwen2.5:7b`](https://huggingface.co/Qwen/Qwen2.5-7B)
105+
- Embedding generation, using [`nomic-embed-text`](https://ollama.com/library/nomic-embed-text)
106+
107+
108+
The chat model is used to summarize files, answer questions over retrieved memory, and generate workspace-level insights. The embedding model converts text into vectors so Qdrant can store and search semantic memory.
109+
110+
Ollama is responsible for:
111+
112+
- Local LLM inference
113+
- Token generation
114+
- AI summarization
115+
- Embedding generation
116+
117+
Ollama does not watch files, manage memory, or decide when work should happen. It provides model execution, and Hermes calls it when the workflow requires inference.
118+
119+
### Qdrant Memory Service
120+
121+
Qdrant provides ***persistent vector memory***.
122+
123+
Hermes stores document embeddings in a Qdrant collection named `workspace_memory`. Each stored point includes a vector and payload metadata, such as the document path, generated summary, and source content excerpt.
124+
125+
Qdrant is responsible for:
126+
127+
- Vector storage
128+
- Semantic indexing
129+
- Similarity search
130+
- Long-term memory persistence
131+
- Contextual retrieval
132+
133+
Qdrant does not perform LLM inference. It stores vectors and returns semantically similar memories when Hermes performs a retrieval query.
134+
135+
### Open WebUI
136+
137+
Open WebUI provides a local browser interface for interacting with the Ollama runtime.
138+
139+
It is useful for validating that local models are available, testing prompts, and giving users a simple interface to local inference. In this Learning Path, Open WebUI is not the orchestration layer and it is not the memory system.
140+
141+
Open WebUI is responsible for:
142+
143+
- Browser-based access
144+
- Local chat interaction
145+
- Model testing and exploration
146+
147+
The persistent AI runtime is still coordinated by Hermes.
148+
149+
## Shared Workspace
150+
151+
The services use a ***shared workspace*** mounted into the containers.
152+
153+
The workspace structure is:
154+
155+
```text
156+
workspace/
157+
|-- inbox/
158+
|-- memory/
159+
|-- logs/
160+
|-- processed/
161+
`-- config/
162+
```
163+
164+
Each directory has a specific purpose:
165+
166+
| Directory | Purpose |
167+
|---|---|
168+
| `workspace/inbox/` | Input files monitored by Hermes |
169+
| `workspace/memory/` | Generated memory artifacts and workspace summaries |
170+
| `workspace/logs/` | Runtime logs and diagnostics |
171+
| `workspace/processed/` | Optional location for processed files |
172+
| `workspace/config/` | Runtime policy configuration |
173+
174+
The shared workspace is what turns isolated containers into a coordinated local AI runtime. Hermes can observe files created on the host, use Ollama to process them, store memory in Qdrant, and write results back to persistent storage.
175+
176+
## Event-driven AI Workflows
177+
178+
Persistent AI systems are long-running systems. They do not wait for a single prompt and then exit. They monitor runtime state and react when something changes.
179+
180+
In this Learning Path, Hermes starts with a filesystem watcher:
181+
182+
```text
183+
[New document] -> [Filesystem event] -> [Hermes orchestration] -> [Document processing]
184+
```
185+
186+
As you add capabilities, the workflow grows:
187+
188+
```text
189+
[New document]
190+
-> [CPU watcher]
191+
-> [Document parsing]
192+
-> [GPU summarization]
193+
-> [GPU embedding]
194+
-> [Qdrant memory]
195+
```
196+
197+
This event-driven design is important because it shows how AI systems become continuous local runtimes. The model is only one part of the system. The surrounding runtime decides when to call the model, what context to provide, where to store results, and how later workflows can reuse those results.
198+
199+
## Semantic Memory and Retrieval
200+
201+
***Semantic memory*** gives the runtime a way to retain information over time.
202+
203+
| Flow | Runtime path |
204+
|---|---|
205+
| Store memory | `[Document] -> [Summary] -> [Embedding] -> [Qdrant vector storage]` |
206+
| Retrieve memory | `[Question] -> [Query embedding] -> [Similarity search] -> [Contextual response]` |
207+
208+
This is different from storing plain text files and searching for keywords. Vector search allows the runtime to retrieve content based on semantic similarity. For example, a question about "CPU scheduling" can retrieve a document that discusses "runtime orchestration" even if the exact words are different.
209+
210+
## Autonomous Workspace Cognition
211+
212+
The final stage of this Learning Path adds autonomous workspace cognition.
213+
214+
Instead of responding only when a new file appears or when a query is submitted, Hermes periodically reviews the accumulated semantic memory and generates a workspace-level summary.
215+
216+
The cognition workflow is:
217+
218+
```text
219+
[Semantic memory] -> [Scheduled analysis] -> [Workspace summary] -> [Runtime insights]
220+
```
221+
222+
Runtime behavior is controlled by a configuration file:
223+
224+
```text
225+
/workspace/config/runtime.json
226+
```
227+
228+
This allows the runtime to adjust settings such as supported file extensions, retrieval depth, summary interval, and summary output path without hardcoding every behavior into the agent.
229+
230+
## CPU and GPU Responsibilities
231+
232+
This Learning Path highlights heterogeneous AI computing. The CPU and GPU both matter, but they perform different roles.
233+
234+
The Arm Grace CPU coordinates persistent runtime work:
235+
236+
- Filesystem monitoring
237+
- Event scheduling
238+
- Runtime orchestration
239+
- Background service coordination
240+
- Document parsing
241+
- Metadata management
242+
- Vector database coordination
243+
- Runtime policy loading
244+
- Long-running process lifecycle management
245+
246+
The Blackwell GPU accelerates model execution:
247+
248+
- Local LLM inference
249+
- Token generation
250+
- AI summarization
251+
- Embedding generation
252+
- Contextual reasoning
253+
- Workspace summary generation
254+
255+
This separation is central to the architecture. The GPU accelerates model-heavy operations, while the CPU keeps the distributed AI runtime organized and continuously operating.
256+
257+
## Next Step
258+
259+
Next, you will build the DGX Spark runtime foundation: Docker, GPU-enabled containers, the shared workspace, and the initial Ollama, Qdrant, and Open WebUI services.

0 commit comments

Comments
 (0)