docs: update

Author: BinglanLi

Multi_Agent_Future_Roadmap.md

@@ -392,353 +392,6 @@ log, result = agent.run_sync("Build an Alzheimer's disease knowledge graph")
 
 ---
 
-### Feature 10: Skills System Overhaul (Spec-Driven Loading + Progressive Disclosure)
-
-**Priority:** HIGH -- the current implementation defeats its own purpose and will not scale for multi-agent workflows.
-
-**Depends on:** None (can be built in parallel with other features).
-
-**Current state and problems:**
-
-The skills system exists (Feature "Agent Skills" in Existing Features) but has several design flaws that contradict the intended purpose of skills and will become blockers as multi-agent workflows add more domain-specific skills:
-
-1. **Full skill body always injected into context** (`base_agent.py:774-784`). Every selected skill has its entire `instructions` field dumped into the system prompt via `_SKILL_ENTRY_TEMPLATE`. There is no progressive disclosure -- if 5 skills are selected, all 5 bodies go in at once. The YAML frontmatter (name, description) was designed for lightweight selection, but selection immediately triggers full injection. This directly contradicts the reference pattern from [anthropics/skills](https://github.com/anthropics/skills/blob/main/skills/skill-creator/SKILL.md), which specifies three loading levels:
-   - Level 1: **Metadata** (name + description) -- always in context (~100 words)
-   - Level 2: **SKILL.md body** -- loaded when skill triggers (<500 lines ideal)
-   - Level 3: **Bundled resources** -- loaded as needed (unlimited, scripts execute without loading)
-
-2. **No bundled resources support**. `_parse_skill_file` (`base_agent.py:264-308`) only reads the single SKILL.md file. The reference pattern describes `scripts/`, `references/`, `assets/` directories alongside SKILL.md. A skill like `ontology-mapping` may need reference docs (OWL specs, mapping examples) and template scripts (Cypher export patterns) that should load on demand -- not bloat the system prompt or be absent entirely.
-
-3. **No context budget awareness**. Skills are injected unconditionally into the system prompt. In multi-agent workflows where each specialist agent may need 2-3 domain skills, unbounded injection will collide with Feature 5 (Context Window Management).
-
-4. **Retriever makes all-or-nothing decisions**. At `base_agent.py:1034-1038`, the retriever selects skills based on `{name, description}` -- appropriate for lightweight triage. But selection immediately injects the full body. There is no intermediate step where the agent could inspect a skill summary before committing to loading the full instructions.
-
-5. **`tools` field is purely decorative** (`resources.py:347`). `Skill.tools` is shown in the prompt but never used functionally -- it doesn't filter available tools, validate that referenced tools exist, or influence retriever decisions.
-
-6. **No skill directory tracking**. `source_path` stores the SKILL.md file path, but nothing records the parent directory. Even if bundled resources existed, the system wouldn't know where to find `references/` or `scripts/` relative to the skill.
-
-7. **Loading is wasteful for multi-agent**. `configure()` loads ALL skills from `skills_directory` via glob, then `spec.skill_names` filters via `select_skills_by_names()`. In BaseAgent multi-agent workflows, each agent needs 1-3 skills from a shared pool. Loading all skills just to discard most of them is unnecessary overhead and risks cross-agent state confusion.
-
-**Skill directory convention:**
-
-Skills reside in a conventional directory structure under the project root:
-
-```
-skills/                              # skills_directory points here
-  ontology-design/
-    SKILL.md                         # required: YAML frontmatter + markdown body
-    references/                      # optional: bundled reference docs
-      owl_spec.md
-    scripts/                         # optional: bundled template scripts
-      template.py
-  database-evaluation/
-    SKILL.md
-  parser-development/
-    SKILL.md
-    scripts/
-      csv_template.py
-  ontology-mapping/
-    SKILL.md
-    references/
-      owl_mapping_guide.md
-  memgraph-export/
-    SKILL.md
-    scripts/
-      cypher_template.py
-```
-
-`AgentSpec.skill_names=["ontology-design"]` resolves to `{skills_directory}/ontology-design/SKILL.md`. Each agent loads **only** the skills specified in its spec -- no glob, no load-all-then-filter.
-
-**Phase 1 -- Model cleanup: remove `trigger`, add `source_dir`**
-
-Remove `trigger` from the `Skill` model and all code that branches on it:
-
-- `resources.py:344` -- remove `trigger` field from `Skill`
-- `resource_manager.py:526-532` -- remove `trigger="manual"` guard; all skills follow the same selection logic
-- `base_agent.py:1033-1037` -- retriever passes all skills as candidates (no trigger filter)
-- `agent_spec.py:42` -- update docstring
-- `tests/test_skills.py` -- remove `test_manual_trigger`, update `test_select_skills_by_names_manual_not_deselected`
-
-Add `source_dir: str | None` to the `Skill` model. When loading from a file, set `source_dir` to the SKILL.md's parent directory. Add a `has_bundled_resources` computed property that checks for `references/`, `scripts/`, or `assets/` subdirectories.
-
-```python
-# In resources.py, add to Skill:
-source_dir: Optional[str] = Field(None, description="Directory containing the SKILL.md and optional bundled resources")
-
-@property
-def has_bundled_resources(self) -> bool:
-    if not self.source_dir:
-        return False
-    d = Path(self.source_dir)
-    return any((d / sub).is_dir() for sub in ("references", "scripts", "assets"))
-
-@property
-def bundled_resource_manifest(self) -> dict[str, list[str]]:
-    """List files in each bundled resource subdirectory."""
-    if not self.source_dir:
-        return {}
-    manifest = {}
-    for sub in ("references", "scripts", "assets"):
-        sub_dir = Path(self.source_dir) / sub
-        if sub_dir.is_dir():
-            manifest[sub] = [f.name for f in sorted(sub_dir.iterdir()) if f.is_file()]
-    return manifest
-```
-
-In `_parse_skill_file`, set `source_dir=str(path.parent)`.
-
-**Files to modify (Phase 1):**
-- `BaseAgent/resources.py` -- remove `trigger`, add `source_dir` + computed properties
-- `BaseAgent/resource_manager.py` -- remove trigger guard in `select_skills_by_names`
-- `BaseAgent/base_agent.py` -- `_parse_skill_file` (set `source_dir`), `_select_resources_for_prompt` (remove trigger filter)
-- `BaseAgent/agent_spec.py` -- update docstring
-- `BaseAgent/tests/test_skills.py` -- remove trigger tests, add `source_dir` / bundled resource tests
-
-**Phase 2 -- Spec-driven targeted skill loading**
-
-Replace the current "load all → filter" loading model with targeted loading. Each agent loads only the skills specified in its `AgentSpec.skill_names` directly from their conventional directory paths.
-
-Add `_resolve_skill_path(skill_name) -> Path | None` to `BaseAgent`:
-
-```python
-def _resolve_skill_path(self, skill_name: str) -> Path | None:
-    """Resolve a skill name to its SKILL.md path using directory conventions.
-
-    Looks up {skills_directory}/{skill_name}/SKILL.md.
-    Returns None if the file does not exist or skills_directory is not set.
-    """
-    if not self.skills_directory:
-        return None
-    candidate = Path(self.skills_directory) / skill_name / "SKILL.md"
-    return candidate if candidate.is_file() else None
-```
-
-Rewrite the skill loading block in `configure()` (`base_agent.py:867-876`):
-
-```python
-# Skill loading: targeted when spec provides skill_names, legacy glob otherwise
-if self.spec is not None and self.spec.skill_names is not None:
-    # Targeted: load only the skills named in the agent spec
-    for skill_name in self.spec.skill_names:
-        path = self._resolve_skill_path(skill_name)
-        if path:
-            skill = self._parse_skill_file(path)
-            self.resource_manager.add_skill(skill)
-        else:
-            print(f"Warning: skill '{skill_name}' not found in '{self.skills_directory}'")
-elif self.skills_directory:
-    # Legacy fallback: load all skills from directory via glob
-    self.load_skills(self.skills_directory)
-
-# Apply AgentSpec tool filters (skill filtering no longer needed -- we only loaded what we need)
-if self.spec is not None:
-    if self.spec.tool_names is not None:
-        self.resource_manager.select_tools_by_names(self.spec.tool_names)
-```
-
-The `select_skills_by_names()` call is removed from `configure()` -- since we only loaded the skills we need, there is nothing to filter. `load_skills()` and `add_skill()` remain as public API for ad-hoc single-agent usage.
-
-**Files to modify (Phase 2):**
-- `BaseAgent/base_agent.py` -- `_resolve_skill_path()` (new), `configure()` (rewrite skill loading block)
-
-**Phase 3 -- Progressive disclosure: metadata-only prompt + retriever-driven loading**
-
-This is the core behavioral change. The system prompt shows only skill metadata at configuration time; the `retrieve` node loads full bodies on demand when the task arrives.
-
-**New config field:**
-- `BaseAgentConfig.skill_retrieval: bool = True` (default `True` -- always retrieve skills on demand)
-- Env var override: `BASE_AGENT_SKILL_RETRIEVAL`
-
-When `skill_retrieval=True` (default):
-- `configure()` / `_generate_system_prompt(is_retrieval=False)`: injects only the **skill catalog** -- name, description, and bundled resource manifest per skill (~2-3 lines each)
-- `retrieve` node: always runs skill selection (even if `use_tool_retriever=False`), then regenerates system prompt with full instructions for selected skills
-- The agent sees full instructions only for task-relevant skills
-
-When `skill_retrieval=False`:
-- Falls back to current behavior: all loaded skills' full instructions injected into the system prompt at configure time
-- Provided as an escape hatch for simple single-skill setups
-
-**New prompt templates** in `prompts.py`:
-
-```python
-_SKILL_CATALOG_SECTION = \
-"""
-
-AVAILABLE SKILLS
-===============================
-The following skills provide specialized domain knowledge and workflows.
-Relevant skill instructions will be loaded based on the current task.
-{skill_catalog}
-"""
-
-_SKILL_CATALOG_ENTRY = \
-"""- {skill_name}: {skill_description}{bundled_note}"""
-
-_SKILL_SELECTION_PROMPT = \
-"""Select which skills are relevant for this task.
-
-TASK: {query}
-
-AVAILABLE SKILLS:
-{skills}
-
-Respond with: SKILLS: [list of indices]
-If none are relevant: SKILLS: []
-"""
-```
-
-**System prompt changes** in `_generate_system_prompt()` (`base_agent.py:774-784`):
-
-```python
-selected_skills = self.resource_manager.get_selected_skills()
-if selected_skills:
-    if self.skill_retrieval and not is_retrieval:
-        # Catalog mode: metadata only (initial prompt)
-        catalog_entries = []
-        for skill in selected_skills:
-            bundled = ""
-            if skill.has_bundled_resources:
-                bundled = " [has bundled resources]"
-            catalog_entries.append(_SKILL_CATALOG_ENTRY.format(
-                skill_name=skill.name,
-                skill_description=skill.description,
-                bundled_note=bundled,
-            ))
-        prompt_modifier += _SKILL_CATALOG_SECTION.format(
-            skill_catalog="\n".join(catalog_entries)
-        )
-    else:
-        # Full injection: either retrieval pass or opt-out
-        skill_parts = []
-        for skill in selected_skills:
-            skill_parts.append(_SKILL_ENTRY_TEMPLATE.format(
-                skill_name=skill.name,
-                skill_description=skill.description,
-                skill_instructions=skill.instructions,
-            ))
-        prompt_modifier += _SKILLS_SECTION.format(skills_content="\n".join(skill_parts))
-```
-
-**New method** `_select_skills_for_prompt(prompt)` in `BaseAgent`:
-
-```python
-def _select_skills_for_prompt(self, prompt: str) -> None:
-    """Select relevant skills based on task prompt via lightweight LLM call."""
-    all_skills = [
-        {"name": s.name, "description": s.description}
-        for s in self.resource_manager.get_all_skills()
-    ]
-    if not all_skills:
-        return
-
-    skills_text = "\n".join(
-        f"{i}. {s['name']}: {s['description']}" for i, s in enumerate(all_skills)
-    )
-    selection_prompt = _SKILL_SELECTION_PROMPT.format(query=prompt, skills=skills_text)
-    response = self.llm.invoke([HumanMessage(content=selection_prompt)])
-
-    # Parse SKILLS: [indices] response
-    match = re.search(r"SKILLS:\s*\[(.*?)\]", response.content, re.IGNORECASE)
-    if match and match.group(1).strip():
-        indices = [int(idx.strip()) for idx in match.group(1).split(",") if idx.strip()]
-        selected_names = [all_skills[i]["name"] for i in indices if i < len(all_skills)]
-    else:
-        selected_names = []
-
-    self.resource_manager.select_skills_by_names(selected_names)
-```
-
-**Retrieve node changes** (`nodes.py:310-332`):
-
-```python
-def retrieve(self, state: "AgentState") -> "AgentState":
-    agent = self.agent
-    prompt = state["input"][-1].content
-
-    # Skill retrieval: always runs when skill_retrieval=True and skills exist
-    if agent.skill_retrieval and agent.resource_manager.get_all_skills():
-        agent._select_skills_for_prompt(prompt)
-        agent.system_prompt = agent._generate_system_prompt(
-            self_critic=agent.self_critic,
-            is_retrieval=True,
-        )
-
-    # Tool/data/library retrieval: only when use_tool_retriever=True
-    if agent.use_tool_retriever:
-        agent._select_resources_for_prompt(prompt)
-        agent.system_prompt = agent._generate_system_prompt(
-            self_critic=agent.self_critic,
-            is_retrieval=True,
-        )
-
-    return state
-```
-
-The retrieve node is no longer a no-op when `use_tool_retriever=False` -- it always processes skill retrieval when `skill_retrieval=True` and there are registered skills.
-
-**Files to modify (Phase 3):**
-- `BaseAgent/config.py` -- `skill_retrieval: bool = True` field + env var override
-- `BaseAgent/prompts.py` -- `_SKILL_CATALOG_SECTION`, `_SKILL_CATALOG_ENTRY`, `_SKILL_SELECTION_PROMPT` (new templates)
-- `BaseAgent/base_agent.py` -- `_generate_system_prompt()` (catalog vs. full injection), `_select_skills_for_prompt()` (new method), `configure()` (store `self.skill_retrieval`)
-- `BaseAgent/nodes.py` -- `retrieve` node: skill retrieval always runs independently of `use_tool_retriever`
-- `BaseAgent/tests/test_skills.py` -- progressive disclosure tests (catalog in initial prompt, full body after retrieval)
-
-**Phase 4 -- Bundled resource access via REPL**
-
-Register a built-in helper function `read_skill_resource(skill_name, path)` that reads files from a skill's bundled resource directory. This lets the agent load reference docs and templates on demand from within `<execute>` blocks:
-
-```python
-# Injected into REPL namespace when skills with bundled resources are present
-def read_skill_resource(skill_name: str, path: str, _agent=None) -> str:
-    """Read a bundled resource file from a skill directory.
-
-    Args:
-        skill_name: Name of the skill (e.g. "ontology-mapping")
-        path: Relative path within the skill directory (e.g. "references/owl_spec.md")
-    """
-    if _agent is None:
-        raise RuntimeError("read_skill_resource requires an agent context")
-    skill = _agent.resource_manager.get_skill_by_name(skill_name)
-    if not skill or not skill.source_dir:
-        raise FileNotFoundError(f"Skill '{skill_name}' not found or has no source directory")
-    full_path = Path(skill.source_dir) / path
-    if not full_path.is_file():
-        raise FileNotFoundError(f"Resource '{path}' not found in skill '{skill_name}'")
-    return full_path.read_text(encoding="utf-8")
-```
-
-Injected into the per-instance REPL namespace via `functools.partial(_agent=self)` in `_inject_custom_functions_to_repl()` when any loaded skill has bundled resources.
-
-The skill instructions would then say things like:
-```markdown
-When you need the OWL mapping reference, load it:
-read_skill_resource("ontology-mapping", "references/owl_mapping_guide.md")
-```
-
-**Files to modify (Phase 4):**
-- `BaseAgent/tools/support_tools.py` -- `read_skill_resource` helper function
-- `BaseAgent/tools/tool_description/support_tools.py` -- tool description for `read_skill_resource`
-- `BaseAgent/base_agent.py` -- inject into REPL namespace in `_inject_custom_functions_to_repl()`
-- `BaseAgent/tests/test_skills.py` -- bundled resource access tests
-
-**Phase 5 -- Functional `tools` field**
-
-Make the `Skill.tools` field actionable:
-- At `configure()` time, validate that all tools listed in `skill.tools` actually exist in `ResourceManager`. Emit a warning for missing tools.
-- When a skill is selected by the retriever in `_select_skills_for_prompt()`, auto-select its referenced tools (even if the retriever didn't explicitly pick them). This ensures skills always have their required tools available.
-
-**Files to modify (Phase 5):**
-- `BaseAgent/base_agent.py` -- `configure()` (tool validation), `_select_skills_for_prompt()` (auto-select skill tools)
-- `BaseAgent/tests/test_skills.py` -- tool validation and auto-selection tests
-
-**Backwards compatibility:**
-- `spec=None` + `skills_directory` set: legacy `load_skills()` glob behavior preserved
-- `skill_retrieval=False`: restores current all-in body injection behavior
-- `add_skill()` and `load_skills()` remain as public API for ad-hoc usage
-- Default (`skill_retrieval=True`) changes the prompt behavior but is strictly better -- agents see the same skill instructions, just loaded on demand rather than always present
-
----
-
 ## Post-Prototype Feature Specifications
 
 Deferred until the prototype is validated. Listed here to inform design decisions -- do not build hooks or abstractions for these unless they are zero-cost.
@@ -817,14 +470,6 @@ New files to be created by future features:
 
 ```
                                                     Depends On       Effort
-== GROUP A (parallel, no dependencies) =============================================
-Feature 10  Skills system overhaul                  --               ~3 days
-  Phase 1   Model cleanup (remove trigger, add source_dir)            ~0.5 day
-  Phase 2   Spec-driven targeted skill loading                       ~0.5 day
-  Phase 3   Progressive disclosure in prompt injection               ~1 day
-  Phase 4   Bundled resource access via REPL                         ~0.5 day
-  Phase 5   Functional tools field                                   ~0.5 day
-
 == GROUP B (after Group A completes) ===============================================
 Feature 5   Context window management               --               ~2 days
   Phase 1   Sliding window for single-agent
@@ -854,7 +499,7 @@ Feature 9   Workflow orchestration                  8                ~3 days
   Phase 3   BaseAgent multi-agent demo script
 ```
 
-**Critical path:** `[1, 2, 3, 4, 10 parallel] -> [5, 6 parallel] -> 7 -> 8 -> 9`
+**Critical path:** `[1, 2, 3, 4, 10 ✅] -> [5, 6 parallel] -> 7 -> 8 -> 9`
 
 **Minimum viable prototype:** Features 1-6 + 8 + 10 Phase 1-3 (supervisor orchestrator, spec-driven skills with progressive disclosure)