You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: content/learning-paths/laptops-and-desktops/dgx_persistent_agent/1_introduce.md
+22-19Lines changed: 22 additions & 19 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,22 +1,25 @@
1
1
---
2
-
title: Understand persistent AI runtime architecture
2
+
title: Explore persistent AI runtime architecture on NVIDIA DGX Spark
3
+
description: Explore the persistent AI runtime architecture for NVIDIA DGX Spark, including Hermes Agent orchestration, Ollama inference, Qdrant memory, and CPU/GPU responsibilities.
3
4
weight: 2
4
5
5
6
### FIXED, DO NOT MODIFY
6
7
layout: learningpathall
7
8
---
8
9
9
-
## Understand persistent AI runtime architecture
10
+
## Persistent AI runtime capabilities
10
11
11
-
You will build a persistent local AI runtime on NVIDIA [DGX Spark](https://www.nvidia.com/en-gb/products/workstations/dgx-spark/). The implementation is validated on DGX Spark, but the architecture also applies to other Arm Cortex-A platforms that can run containerized services and local AI runtimes.
12
+
In this Learning Path, you'll build a persistent local AI runtime on NVIDIA [DGX Spark](https://www.nvidia.com/en-gb/products/workstations/dgx-spark/). The implementation is validated on DGX Spark, but the architecture can also apply to other Arm Cortex-A platforms that can run containerized services and local AI runtimes.
12
13
13
-
The final system is not a single chatbot process. It is a set of local services that run continuously, share a workspace, react to file events, generate summaries, create embeddings, store vector memory, retrieve context, and periodically reason about the state of the workspace.
14
+
The final system isn't a single chatbot process. It's a set of local services that run continuously, share a workspace, react to file events, generate summaries, create embeddings, store vector memory, retrieve context, and periodically reason about the state of the workspace.
14
15
15
-
The core idea is: **AI systems are orchestration systems, not just inference systems.**
16
+
The core idea is that AI systems are orchestration systems, not just inference systems.
16
17
17
-
DGX Spark is well suited to this type of workload because it combines Arm CPU orchestration with local GPU acceleration. In the [Grace Blackwell architecture](https://learn.arm.com/learning-paths/laptops-and-desktops/dgx_spark_llamacpp/1_gb10_introduction/), the Arm Grace CPU coordinates background services, filesystem events, scheduling, document processing, metadata handling, and service-to-service communication. The Blackwell GPU accelerates local LLM inference, token generation, summarization, and embedding generation.
18
+
DGX Spark is well suited to this type of workload because it combines Arm CPU orchestration with local GPU acceleration. In the [Grace Blackwell architecture](/learning-paths/laptops-and-desktops/dgx_spark_llamacpp/1_gb10_introduction/), the Arm Grace CPU coordinates background services, filesystem events, scheduling, document processing, metadata handling, and service-to-service communication.
18
19
19
-
By the end of this Learning Path, you will have a local runtime with these capabilities:
20
+
The Blackwell GPU accelerates local LLM inference, token generation, summarization, and embedding generation.
21
+
22
+
By the end of this Learning Path, you'll have a local runtime with these capabilities:
20
23
21
24
| Capability | Runtime component |
22
25
|---|---|
@@ -27,7 +30,7 @@ By the end of this Learning Path, you will have a local runtime with these capab
Hermes is the orchestration runtime you will build.
78
+
Hermes is the orchestration runtime you'll build.
78
79
79
-
It runs as a persistent Python service inside a container. It watches the shared workspace, detects new files, reads documents, sends requests to Ollama, stores memory in Qdrant, performs semantic retrieval, and later generates autonomous workspace summaries. Hermes doesn't run the language model itself. It coordinates AI workflows across local services.
80
+
Hermes runs as a persistent Python service inside a container. It watches the shared workspace, detects new files, and reads documents. It sends requests to Ollama, stores memory in Qdrant, performs semantic retrieval, and later generates autonomous workspace summaries. Hermes doesn't run the language model itself. Instead, it coordinates AI workflows across local services.
80
81
81
-
This is the main CPU-side workload in the system. The Arm CPU keeps the runtime alive, schedules background loops, tracks file events, moves data between services, and manages runtime state.
82
+
The Hermes runtime is the main CPU-side workload in the system. The Arm CPU keeps the runtime alive, schedules background loops, tracks file events, moves data between services, and manages runtime state.
82
83
83
84
### Ollama runtime
84
85
85
86
Ollama provides the local inference runtime. It's a convenient way to run local models and expose a simple API, but the architecture isn't limited to Ollama.
86
87
87
-
Conceptually, Ollama is one possible inference backend. Hermes can orchestrate any local or remote inference service that exposes a compatible API, such as llama.cpp server, vLLM, a custom PyTorch service, or another model runtime.
88
+
Conceptually, Ollama is one possible inference backend. Hermes can orchestrate any local or remote inference service that exposes a compatible API, such as `llama.cpp` server, vLLM, a custom PyTorch service, or another model runtime.
88
89
89
90
Hermes uses Ollama for two types of model calls:
90
91
@@ -106,7 +107,7 @@ Open WebUI provides a local browser interface for interacting with the Ollama ru
106
107
107
108
Use it to confirm that local models are available, test prompts, or explore the inference runtime in a browser. The persistent AI runtime is still coordinated by Hermes.
108
109
109
-
## Shared workspace
110
+
## Shared workspace structure
110
111
111
112
The services use a shared workspace mounted into the containers.
112
113
@@ -169,7 +170,7 @@ This is different from storing plain text files and searching for keywords. Vect
169
170
170
171
## Autonomous workspace cognition
171
172
172
-
The final stage of this Learning Path adds autonomous workspace cognition.
173
+
In the final stage of this Learning Path, you'll add autonomous workspace cognition.
173
174
174
175
Instead of responding only when a new file appears or when a query is submitted, Hermes periodically reviews the accumulated semantic memory and generates a workspace-level summary.
175
176
@@ -185,7 +186,7 @@ Runtime behavior is controlled by a configuration file:
185
186
/workspace/config/runtime.json
186
187
```
187
188
188
-
This allows the runtime to adjust settings such as supported file extensions, retrieval depth, summary interval, and summary output path without hardcoding every behavior into the agent.
189
+
This file allows the runtime to adjust settings such as supported file extensions, retrieval depth, summary interval, and summary output path without hardcoding every behavior into the agent.
189
190
190
191
## CPU and GPU responsibilities
191
192
@@ -195,6 +196,8 @@ The Arm Grace CPU handles the persistent runtime side: watching the filesystem,
195
196
196
197
This separation is central to the architecture. The GPU accelerates model-heavy operations, while the CPU keeps the runtime organized and continuously operating.
197
198
198
-
## Next step
199
+
## What you've learned and what's next
200
+
201
+
You've now learned about the persistent AI runtime architecture that you'll be building on NVIDIA DGX Spark. You've explored the different components and workflows of this architecture, and how the CPU and GPU perform different roles.
199
202
200
-
Next, you will build the DGX Spark runtime foundation: Docker, GPU-enabled containers, the shared workspace, and the initial Ollama, Qdrant, and Open WebUI services.
203
+
Next, you'll build the DGX Spark runtime foundation: Docker, GPU-enabled containers, the shared workspace, and the initial Ollama, Qdrant, and Open WebUI services.
0 commit comments