Merge pull request #3455 from jaidev17/ai-agent-cpu-orchestration

Add Learning Path for CPU-orchestrated local AI agent

Author: pareenaverma

assets/contributors.csv

@@ -2,6 +2,7 @@ author,company,github,linkedin,twitter,website
 Jason Andrews,Arm,jasonrandrews,jason-andrews-7b05a8,,
 Doug Anson,Arm,DougAnsonAustinTx,douganson,,
 Pareena Verma,Arm,pareenaverma,pareena-verma-7853607,,
+Jaidev Singh Chadha,Arm,jaidev17,jaidevsinghchadha,,
 Ronan Synnott,Arm,,ronansynnott,,
 Florent Lebeau,Arm,,,,
 Brenda Strech,Remote.It,bstrech,bstrech,@remote_it,www.remote.it

content/learning-paths/cross-platform/ai-agent-cpu-orchestration/1-overview.md

@@ -0,0 +1,61 @@
+---
+title: Understand the agent and the CPU/GPU split
+weight: 2
+
+### FIXED, DO NOT MODIFY
+layout: learningpathall
+---
+
+## What you'll build
+
+In this Learning Path, you'll build and run a *local concierge agent*: a terminal application that answers research-style questions, such as finding restaurants, comparing products, or summarizing a topic, by searching the web, reading multiple pages, and writing a fact-checked summary.
+
+Everything runs on your own Arm machine. The agent uses an LLM served locally by [Ollama](https://ollama.com/), so your prompts and the pages you browse never leave the device. The same code runs on an Apple silicon MacBook, an Arm Linux laptop, or an NVIDIA DGX Spark.
+
+
+The agent is interesting not just because it runs locally, but because of *how the work is divided*. A common assumption is that an AI agent is "just the model." In practice, the model is only one stage in a longer pipeline, and most of the surrounding work runs on the CPU.
+
+## The agentic loop
+
+Each time you ask the agent a question, it runs through a chain of steps rather than a single model call:
+
+```text
+Your question
+    -> Decide what to search for          (GPU: model reasoning)
+    -> Expand into several search queries  (CPU: orchestration)
+    -> Run web searches in parallel        (CPU: network I/O)
+    -> Choose which pages to open          (GPU: model reasoning)
+    -> Scrape those pages in parallel       (CPU: network I/O)
+    -> Rank, deduplicate, extract data      (CPU: text processing)
+    -> Write a fact-checked summary         (GPU: model reasoning)
+```
+
+The model is called several times, but between every model call the CPU does a large amount of orchestration: generating query variants, dispatching parallel network requests, merging and deduplicating results, scoring pages for relevance, and extracting structured data.
+
+## Why the CPU matters in an agentic workflow
+
+It's easy to focus only on token generation, because that's the visible "thinking" part. But in an agent, the CPU is responsible for turning a single user request into useful, structured context for the model:
+
+| Stage | Runs on | Example work |
+|---|---|---|
+| Reasoning and summarization | GPU | Choosing search terms, selecting URLs, writing the final answer |
+| Orchestration | CPU | Expanding queries, scheduling parallel tasks, merging results |
+| Web I/O | CPU | Calling the search API, downloading and parsing web pages |
+| Text processing | CPU | TF-IDF ranking, deduplication, entity extraction, indexing |
+
+This division is a natural fit for Arm platforms. On an Apple silicon MacBook or an Arm Linux machine, the CPU handles all orchestration and I/O while the GPU accelerates the model. On an NVIDIA DGX Spark, the Arm CPU coordinates the workflow while the GPU runs inference. In every case, the model is only as good as the context the CPU prepares for it.
+
+## How the agent shows you the split
+
+The program is instrumented so you can see this division directly. As it runs, it prints:
+
+- <code style="color:#00aaaa"><strong>[CPU]</strong></code> log lines for every orchestration and processing step
+- <code style="color:#aaaa00"><strong>[GPU]</strong></code> log lines for every model call
+- A timing breakdown of CPU time versus GPU input-token processing and token generation
+- A text-based timeline that visualizes when the CPU and GPU are each active
+
+By the end of this Learning Path, you'll be able to look at a single query and see exactly how much of the work happened on the CPU before and after the model produced its answer.
+
+## What's next
+
+In the next section, you'll set up your environment: create a Python virtual environment, install the required packages, and get a Serper API key for web search.

content/learning-paths/cross-platform/ai-agent-cpu-orchestration/2-setup.md

@@ -0,0 +1,82 @@
+---
+title: Set up your environment
+weight: 3
+
+### FIXED, DO NOT MODIFY
+layout: learningpathall
+---
+
+## Set up your environment
+
+In this section, you'll prepare everything the agent needs before running it: a Serper API key for web search, a Python virtual environment, the required packages, and the agent script itself.
+
+These steps are identical on an Apple silicon MacBook, an Arm Linux machine, and an NVIDIA DGX Spark.
+
+## Get a Serper API key
+
+The agent searches the web through [Serper](https://serper.dev/), a Google Search API. The free tier is enough to complete this Learning Path.
+
+1. Go to [serper.dev](https://serper.dev/) and create a free account.
+2. Open your dashboard and copy your API key.
+
+You'll set this key as an environment variable later in this section, so the agent can read it without the key being written into the code.
+
+## Create a project directory and virtual environment
+
+A virtual environment keeps the agent's dependencies isolated from your system Python, so you always run against the right package versions.
+
+Create and enter a project directory:
+
+```bash
+mkdir concierge-agent
+cd concierge-agent
+```
+
+Create and activate a virtual environment:
+
+```bash
+python3 -m venv .venv
+source .venv/bin/activate
+```
+
+Your shell prompt now shows `(.venv)`, which confirms the environment is active.
+
+## Install the required packages
+
+The agent uses `requests` for HTTP calls and `beautifulsoup4` to parse web pages.
+
+Install the two packages:
+
+```bash
+pip install requests beautifulsoup4
+```
+
+## Set your Serper API key
+
+Export your API key as an environment variable in the same terminal session:
+
+```bash
+export SERPER_API_KEY="your-serper-api-key"
+```
+
+{{% notice Tip %}}
+This variable only lasts for the current terminal session. To keep it across sessions, add the same line to your shell profile, for example `~/.zshrc` on macOS or `~/.bashrc` on Linux.
+{{% /notice %}}
+
+## Download the agent script
+
+Download the complete agent script, <a href="/learning-paths/cross-platform/ai-agent-cpu-orchestration/concierge_agent.py" download>concierge_agent.py</a>, and move it into your `concierge-agent` directory. You'll run it at the end of this Learning Path, and the next sections walk through how the important parts work.
+
+Alternatively, download it directly from the command line:
+
+```bash
+curl -O https://raw.githubusercontent.com/ArmDeveloperEcosystem/arm-learning-paths/main/content/learning-paths/cross-platform/ai-agent-cpu-orchestration/concierge_agent.py
+```
+
+Confirm the file is in your project directory:
+
+```bash
+ls concierge_agent.py
+```
+
+With the environment ready and the script in place, the next step is to serve a model locally with Ollama.

content/learning-paths/cross-platform/ai-agent-cpu-orchestration/3-ollama.md

@@ -0,0 +1,98 @@
+---
+title: Serve a model locally with Ollama
+weight: 4
+
+### FIXED, DO NOT MODIFY
+layout: learningpathall
+---
+
+## Run AI models locally with Ollama
+
+The agent's reasoning steps, such as choosing search terms, selecting URLs, and writing the final summary, are handled by an LLM. Instead of calling a cloud API, this Learning Path serves the model locally with [Ollama](https://ollama.com/).
+
+Ollama is a lightweight runtime that downloads open models, loads them into memory, and exposes a local HTTP API at `http://localhost:11434`. When the agent calls that endpoint, the model runs directly on your machine: the CPU and GPU on your MacBook, Arm Linux laptop, or NVIDIA DGX Spark. Nothing is sent to an external service or cloud.
+
+Running locally has three benefits that matter for an agent:
+
+- **Privacy**: your prompts and the web content the agent reads stay on the device.
+- **Cost**: there are no per-token API charges, so you can run many queries freely.
+- **Control**: you choose the exact model and keep it resident in memory for fast repeated calls.
+
+## Install and start Ollama
+
+Install [Ollama](https://ollama.com/) and start its server using one of the following options. The server exposes a local API at `http://localhost:11434` that the agent connects to.
+
+{{< tabpane code=true >}}
+{{< tab header="Homebrew (Recommended)" language="bash">}}
+# Install Ollama
+brew install ollama
+
+# Start the server as a background service (no terminal to keep open)
+brew services start ollama
+{{< /tab >}}
+{{< tab header="Install script" language="bash">}}
+# Install Ollama
+curl -fsSL https://ollama.com/install.sh | sh
+
+# Start the server (leave this running, and open a second terminal)
+ollama serve
+{{< /tab >}}
+{{< /tabpane >}}
+
+With [Homebrew](https://brew.sh/), `brew services` runs Ollama in the background, so you can use the same terminal throughout. With the install script, `ollama serve` runs in the foreground, so keep that terminal open and use a second terminal for the remaining commands.
+
+## Pull the Gemma model
+
+This Learning Path uses [Gemma 3](https://ai.google.dev/gemma), Google's family of open models, in its 4-billion-parameter size (`gemma3:4b`). This size is a good default: it's capable enough for the agent's reasoning steps and small enough to run on a laptop.
+
+Download the model:
+
+```bash
+ollama pull gemma3:4b
+```
+
+Confirm the model is available:
+
+```bash
+ollama list
+```
+
+You'll see `gemma3:4b` in the list of installed models.
+
+## Choose a different model (optional)
+
+The agent reads the model name from the `OLLAMA_MODEL` environment variable, so you can switch models without editing the code. The default in the script is `gemma3:4b`.
+
+| Model | Size | Best for |
+|---|---|---|
+| `gemma3:4b` | 4B | Laptops and modest hardware; the default |
+| `gemma3:27b` | 27B | High-memory systems such as DGX Spark, for stronger reasoning |
+
+To use a larger model on a DGX Spark, pull it and set the environment variable before running the agent:
+
+```bash
+ollama pull gemma3:27b
+export OLLAMA_MODEL="gemma3:27b"
+```
+
+{{% notice Note %}}
+Larger models produce stronger summaries but use more memory and generate tokens more slowly. Start with `gemma3:4b` to confirm everything works, then experiment with larger models if your hardware allows.
+{{% /notice %}}
+
+## Test the model
+
+Before wiring it into the agent, confirm the model responds:
+
+```bash
+ollama run gemma3:4b
+```
+
+Enter a short prompt:
+
+```text
+Summarize what a local AI agent does in one sentence.
+```
+
+Type `/bye` to exit the model session.
+
+With Ollama serving `gemma3:4b`, you're ready to look at how the agent code uses it.

content/learning-paths/cross-platform/ai-agent-cpu-orchestration/4-code-walkthrough.md

@@ -0,0 +1,87 @@
+---
+title: Walk through the agent code
+weight: 5
+
+### FIXED, DO NOT MODIFY
+layout: learningpathall
+---
+
+## How the code is organized
+
+The `concierge_agent.py` script you downloaded earlier is organized into four parts. This section walks through the important pieces so you understand what runs on the CPU, what runs on the GPU, and how the agent chains them together.
+
+| Part | Responsibility | Runs on |
+|---|---|---|
+| Part 1: Tools | Web search, page scraping, optional email | CPU |
+| Part 2: The brain | Calls the local Gemma model through Ollama | GPU |
+| Part 3: Orchestration pipeline | Query expansion, parallelism, ranking, deduplication, extraction | CPU |
+| Part 4: The agentic chain | Ties everything together into one query workflow | CPU + GPU |
+
+## Part 1: The tools
+
+The tools are the agent's connection to the outside world. `search_web()` calls the Serper API and returns the top five results; `browse_website()` downloads a page and strips it down to clean text. Both run entirely on the CPU and both log their activity with `log_cpu()`:
+
+```python
+def search_web(query: str) -> str:
+    """Use the Serper.dev API to perform a web search."""
+    log_cpu(f"Searching web: '{query}'")
+    ...
+
+def browse_website(url: str) -> str:
+    """Scrape and clean the text content of a given URL."""
+    log_cpu(f"Browsing website: '{url}'")
+    ...
+    return text[:8000]
+```
+
+`browse_website()` uses BeautifulSoup to remove `<script>` and `<style>` tags, collapse whitespace, and cap the result at 8,000 characters so a single large page can't dominate the pipeline.
+
+## Part 2: The brain
+
+A single function, `call_gemma_ollama()`, is the only place the model is used. It sends a prompt to the local Ollama API and streams the response back token by token:
+
+```python
+def call_gemma_ollama(prompt, output_format="json", timing=None,
+                      label="ollama", timeline_events=None):
+    log_gpu("Thinking with local Gemma model...")
+    payload = {
+        "model": OLLAMA_MODEL,
+        "prompt": prompt,
+        "stream": True,
+        "keep_alive": -1,  # Keep model weights in GPU memory
+    }
+    ...
+```
+
+Two details are worth highlighting:
+
+- `keep_alive: -1` tells Ollama to keep the model resident in memory between calls. The agent calls the model several times per query, so this avoids reloading the weights each time.
+- The function records timing as it streams. The time until the *first* token arrives is the model's input-token processing (prefill); the time spent streaming the rest is token generation. Separating these two values is what lets the agent show you the GPU breakdown later.
+
+Because every model call goes through this one function, the agent only ever uses the GPU in clearly marked places.
+
+## Part 3: The CPU orchestration pipeline
+
+Between the model calls, the CPU prepares the context. Each stage below runs entirely on the CPU:
+
+| Stage | Function | What it does |
+|---|---|---|
+| Expand queries | `generate_search_queries()` | Turns one query into several variants |
+| Search in parallel | `parallel_search()` | Runs all searches concurrently |
+| Browse in parallel | `parallel_browse()` | Fetches up to ten websites at once |
+| Rank by relevance | `rank_and_filter_content()` | Scores pages with TF-IDF |
+| Deduplicate | `deduplicate_sentences()`, `find_near_duplicates()` | Removes repeated sentences and near-duplicate pages |
+| Extract data | `extract_entities()` | Pulls out phone numbers, hours, and prices |
+| Build index | `build_keyword_index()` | Indexes the most frequent terms |
+
+## Part 4: The agentic chain
+
+`run_concierge_agent()` ties the pipeline together. Reading it top to bottom shows how CPU and GPU steps alternate:
+
+1. **GPU** – `call_gemma_ollama()` turns your question into a search query.
+2. **CPU** – `generate_search_queries()` and `parallel_search()` expand and run the searches.
+3. **GPU** – the model selects which URLs to open.
+4. **CPU** – `parallel_browse()`, `rank_and_filter_content()`, `deduplicate_sentences()`, `extract_entities()`, and the indexing stages prepare the context.
+5. **GPU** – the model writes the final, fact-checked summary.
+
+Now that you understand the structure, you'll run the agent and watch the CPU and GPU work in real time.