Skip to content

Commit 6208b62

Browse files
jrobertboosjrobertboos
committed
(lcore-1594) agent skills feature breakdown
1 parent 0921018 commit 6208b62

2 files changed

Lines changed: 1005 additions & 0 deletions

File tree

‎docs/design/agent-skills/agent-skills-spike.md‎

Lines changed: 361 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,361 @@
1+
# Spike for Agent Skills Support
2+
3+
## Overview
4+
5+
**The problem**: Lightspeed Core has no mechanism for extending agent capabilities with specialized instructions or domain knowledge. Users cannot package reusable workflows, troubleshooting guides, or domain expertise into portable, discoverable units that the LLM can use on demand.
6+
7+
**The recommendation**: Implement the [Agent Skills open standard](https://agentskills.io) with a config-based discovery model. Skills are defined in `lightspeed-stack.yaml` pointing to directories containing `SKILL.md` files. The LLM sees a skill catalog (name + description) in the system prompt and can request full instructions on demand via a dedicated tool.
8+
9+
**PoC validation**: Not applicable for this spike. The feature is well-defined by the agentskills.io specification and has been implemented by 30+ agent products including Claude Code, GitHub Copilot, Cursor, and OpenAI Codex.
10+
11+
## Decisions for @sbunciak and @ptisnovs
12+
13+
These are the high-level decisions that determine scope, approach, and cost. Each has a recommendation confirmed during spike research.
14+
15+
### Decision 1: Which skill types to support?
16+
17+
| Option | Description |
18+
|--------|-------------|
19+
| A | Built-in skills only (Lightspeed developers ship pre-defined skills) |
20+
| B | Customer-defined only (end users import their own skill definitions) |
21+
| C | Both built-in and customer-defined |
22+
23+
**Recommendation**: **B** (Customer-defined only). This allows end users to extend Lightspeed with their own domain expertise without requiring changes to the core product. Built-in skills can be added later if needed.
24+
25+
### Decision 2: Discovery mechanism?
26+
27+
| Option | Description |
28+
|--------|-------------|
29+
| A | Filesystem-based (scan configured directories for `SKILL.md` files) |
30+
| B | Config-based (define skills in `lightspeed-stack.yaml`) |
31+
| C | API-based (skills registered/managed via REST API) |
32+
| D | Hybrid (built-in via config, customer-defined via filesystem or API) |
33+
34+
**Recommendation**: **B** (Config-based). Skills are defined in `lightspeed-stack.yaml` similar to `mcp_servers`. This provides explicit control over which skills are available, integrates with existing configuration patterns, and avoids filesystem scanning complexity.
35+
36+
### Decision 3: Script execution support?
37+
38+
| Option | Description |
39+
|--------|-------------|
40+
| A | No scripts (only `SKILL.md` instructions) |
41+
| B | Scripts allowed (full spec compliance) |
42+
| C | Deferred (start with no scripts, add later) |
43+
44+
**Recommendation**: **A** (No scripts). As noted in LCORE-1339, there are security concerns with executing arbitrary scripts. The core value of skills is in the instructions — scripts can be added in a future phase after security review if needed.
45+
46+
## Technical decisions for @ptisnovs
47+
48+
Architecture-level and implementation-level decisions.
49+
50+
### Decision 4: Skill content source
51+
52+
| Option | Description |
53+
|--------|-------------|
54+
| A | Path-based (config points to directory containing `SKILL.md`) |
55+
| B | Inline (instructions embedded directly in YAML) |
56+
| C | Both (support either path or inline content) |
57+
58+
**Recommendation**: **A** (Path-based). This follows the agentskills.io spec directory structure, keeps the YAML config clean, and allows skills to include `references/` subdirectories for additional documentation that can be loaded on demand.
59+
60+
### Decision 5: Activation mechanism
61+
62+
| Option | Description |
63+
|--------|-------------|
64+
| A | System prompt injection (skill catalog in system prompt, LLM decides) |
65+
| B | Dedicated tool (register `activate_skill` tool, LLM calls to load) |
66+
| C | Per-request parameter (client specifies which skills to activate) |
67+
68+
**Recommendation**: **A** (System prompt injection) for the catalog, combined with **B** (dedicated tool) for loading full instructions. The system prompt contains the skill catalog (name + description, ~50-100 tokens per skill). When the LLM decides a skill is relevant, it calls the `activate_skill` tool to load the full instructions.
69+
70+
### Decision 6: Skill context management
71+
72+
| Option | Description |
73+
|--------|-------------|
74+
| A | Always loaded (all skills' full instructions in every request) |
75+
| B | Catalog + on-demand (only name/description upfront, full content loaded when LLM requests) |
76+
77+
**Recommendation**: **B** (Catalog + on-demand). This follows the agentskills.io progressive disclosure pattern:
78+
1. **Catalog** (~50-100 tokens/skill) - name + description always in system prompt
79+
2. **Instructions** (<5000 tokens) - full `SKILL.md` body loaded via tool when needed
80+
3. **Resources** (on-demand) - `references/` files loaded via file-read tool when referenced
81+
82+
This keeps the base context small while giving the LLM access to specialized knowledge on demand.
83+
84+
### Decision 7: Configuration structure
85+
86+
Skills are configured as a list in `lightspeed-stack.yaml`, following the `mcp_servers` pattern:
87+
88+
```yaml
89+
skills:
90+
- name: "code-review"
91+
description: "Review code for best practices and security issues."
92+
path: "/var/skills/code-review"
93+
- name: "troubleshooting"
94+
description: "Diagnose and fix OpenShift deployment issues."
95+
path: "/var/skills/troubleshooting"
96+
```
97+
98+
Each skill entry specifies:
99+
- `name`: Unique identifier (validated against `SKILL.md` frontmatter)
100+
- `description`: What the skill does and when to use it
101+
- `path`: Absolute path to directory containing `SKILL.md`
102+
103+
**Recommendation**: Approved. This structure is consistent with existing config patterns and provides explicit control.
104+
105+
## Proposed JIRAs
106+
107+
Each JIRA includes an agentic tool instruction pointing to the spec doc.
108+
109+
<!-- type: Task -->
110+
<!-- key: LCORE-???? -->
111+
### LCORE-???? Add skill configuration model
112+
113+
**Description**: Add the `SkillConfiguration` Pydantic model and `skills` list to the main configuration. This enables defining skills in `lightspeed-stack.yaml`.
114+
115+
**Scope**:
116+
117+
- Create `SkillConfiguration` class in `src/models/config.py`
118+
- Add `skills: list[SkillConfiguration]` field to `Configuration` class
119+
- Add validation: path exists, contains `SKILL.md`, name matches frontmatter
120+
- Parse `SKILL.md` frontmatter on startup (extract name, description)
121+
122+
**Acceptance criteria**:
123+
124+
- Skills can be defined in `lightspeed-stack.yaml` using the documented format
125+
- Startup fails with clear error if skill path doesn't exist or lacks `SKILL.md`
126+
- Startup fails if configured `name` doesn't match `SKILL.md` frontmatter `name`
127+
- Unit tests cover validation scenarios
128+
129+
**Agentic tool instruction**:
130+
131+
```text
132+
Read the "Configuration" section in docs/design/agent-skills/agent-skills.md.
133+
Key files: src/models/config.py (around line 1817, Configuration class).
134+
Follow the MCP server config pattern (ModelContextProtocolServer class, line 468).
135+
```
136+
137+
<!-- type: Task -->
138+
<!-- key: LCORE-???? -->
139+
### LCORE-???? Implement skill catalog injection
140+
141+
**Description**: Inject the skill catalog (name + description for each skill) into the system prompt so the LLM knows what skills are available.
142+
143+
**Scope**:
144+
145+
- Create `src/utils/skills.py` module
146+
- Implement `build_skill_catalog()` function that formats skills as XML/structured text
147+
- Modify `get_system_prompt()` in `src/utils/prompts.py` to append skill catalog
148+
- Add behavioral instructions telling the LLM how to use skills
149+
150+
**Acceptance criteria**:
151+
152+
- System prompt includes skill catalog when skills are configured
153+
- Catalog format includes name, description, and path for each skill
154+
- Catalog is omitted when no skills are configured
155+
- Unit tests verify catalog formatting and system prompt injection
156+
157+
**Agentic tool instruction**:
158+
159+
```text
160+
Read the "System prompt injection" section in docs/design/agent-skills/agent-skills.md.
161+
Key files: src/utils/prompts.py (get_system_prompt function),
162+
src/utils/skills.py (new module).
163+
```
164+
165+
<!-- type: Task -->
166+
<!-- key: LCORE-???? -->
167+
### LCORE-???? Implement activate_skill tool
168+
169+
**Description**: Register a `activate_skill` tool that the LLM can call to load full skill instructions when it decides a skill is relevant.
170+
171+
**Scope**:
172+
173+
- Add `activate_skill` tool registration in `prepare_tools()` in `src/utils/responses.py`
174+
- Implement tool handler that reads `SKILL.md` body content
175+
- Return structured response with skill content and base directory path
176+
- Optionally list `references/` files if present
177+
178+
**Acceptance criteria**:
179+
180+
- LLM can call `activate_skill(name="skill-name")` to load skill instructions
181+
- Tool returns full `SKILL.md` body content (after frontmatter)
182+
- Tool returns skill directory path so LLM can resolve relative references
183+
- Tool returns error if skill name is invalid
184+
- Unit tests cover tool registration and invocation
185+
186+
**Agentic tool instruction**:
187+
188+
```text
189+
Read the "Skill activation tool" section in docs/design/agent-skills/agent-skills.md.
190+
Key files: src/utils/responses.py (prepare_tools function, line 204),
191+
src/utils/skills.py.
192+
```
193+
194+
<!-- type: Task -->
195+
<!-- key: LCORE-???? -->
196+
### LCORE-???? Add skill reference file access
197+
198+
**Description**: Enable the LLM to read files from the skill's `references/` subdirectory when skill instructions reference them.
199+
200+
**Scope**:
201+
202+
- Ensure existing file-read tool can access skill reference directories
203+
- Add path validation to restrict access to configured skill directories only
204+
- Document the pattern for skill authors to reference files
205+
206+
**Acceptance criteria**:
207+
208+
- LLM can read files from `<skill-path>/references/` using existing file-read capability
209+
- Access is restricted to configured skill directories (no arbitrary filesystem access)
210+
- Skill instructions can use relative paths like `references/troubleshooting-guide.md`
211+
- Integration test verifies reference file access works end-to-end
212+
213+
**Agentic tool instruction**:
214+
215+
```text
216+
Read the "Reference file access" section in docs/design/agent-skills/agent-skills.md.
217+
Key files: src/utils/responses.py, existing file-read tool implementation.
218+
```
219+
220+
<!-- type: Story -->
221+
<!-- key: LCORE-???? -->
222+
### LCORE-???? Document Agent Skills feature
223+
224+
**Description**: Create user-facing documentation for the Agent Skills feature including configuration guide, skill authoring guide, and examples.
225+
226+
**Scope**:
227+
228+
- Add configuration documentation to existing config docs
229+
- Create skill authoring guide (SKILL.md format, directory structure)
230+
- Add example skills to `examples/skills/`
231+
- Update README with feature overview
232+
233+
**Acceptance criteria**:
234+
235+
- Users can configure skills by following the documentation
236+
- Skill authors can create compliant skills using the authoring guide
237+
- Example skills demonstrate common use cases (troubleshooting, code review, etc.)
238+
239+
**Agentic tool instruction**:
240+
241+
```text
242+
Read the full spec doc at docs/design/agent-skills/agent-skills.md.
243+
Reference the agentskills.io specification for SKILL.md format details.
244+
```
245+
246+
<!-- type: Task -->
247+
<!-- key: LCORE-???? -->
248+
### LCORE-???? Add integration and E2E tests for skills
249+
250+
**Description**: Add integration tests and E2E tests to verify the skills feature works correctly end-to-end.
251+
252+
**Scope**:
253+
254+
- Integration tests: skill loading, catalog injection, tool invocation with mocked LLM
255+
- E2E tests: full flow with real LLM, verify skill activation and usage
256+
257+
**Acceptance criteria**:
258+
259+
- Integration tests cover skill configuration, catalog generation, and tool handling
260+
- E2E tests verify a configured skill can be discovered and used by the LLM
261+
- Tests use example skills from `examples/skills/`
262+
263+
**Agentic tool instruction**:
264+
265+
```text
266+
Read the "Testing" section in docs/design/agent-skills/agent-skills.md.
267+
Key test files: tests/integration/endpoints/, tests/e2e/features/.
268+
Follow existing test patterns in the codebase.
269+
```
270+
271+
## Background sections
272+
273+
### Agent Skills specification
274+
275+
The [Agent Skills open standard](https://agentskills.io) defines a portable format for giving AI agents specialized capabilities. Key elements:
276+
277+
**Directory structure**:
278+
```
279+
skill-name/
280+
├── SKILL.md # Required: metadata + instructions
281+
├── references/ # Optional: additional documentation
282+
└── assets/ # Optional: templates, resources
283+
```
284+
285+
**SKILL.md format**:
286+
```markdown
287+
---
288+
name: skill-name
289+
description: What this skill does and when to use it.
290+
---
291+
292+
# Instructions
293+
294+
Step-by-step instructions for the agent...
295+
```
296+
297+
**Frontmatter fields**:
298+
| Field | Required | Description |
299+
|-------|----------|-------------|
300+
| `name` | Yes | 1-64 chars, lowercase, hyphens only |
301+
| `description` | Yes | Max 1024 chars, describes what and when |
302+
| `license` | No | License name or file reference |
303+
| `compatibility` | No | Environment requirements |
304+
| `metadata` | No | Arbitrary key-value pairs |
305+
306+
### Progressive disclosure
307+
308+
The agentskills.io spec recommends a three-tier loading strategy:
309+
310+
| Tier | What's loaded | When | Token cost |
311+
|------|---------------|------|------------|
312+
| 1. Catalog | Name + description | Session start | ~50-100 tokens/skill |
313+
| 2. Instructions | Full SKILL.md body | When skill is activated | <5000 tokens (recommended) |
314+
| 3. Resources | References, assets | When instructions reference them | Varies |
315+
316+
This keeps base context small while giving the LLM access to specialized knowledge on demand.
317+
318+
### Adoption
319+
320+
Agent Skills are supported by 30+ agent products including:
321+
- Claude Code, Claude.ai
322+
- GitHub Copilot, VS Code
323+
- Cursor, OpenAI Codex
324+
- Gemini CLI, JetBrains Junie
325+
- Goose, Letta, Spring AI
326+
327+
OpenAI's SDK already includes `LocalSkill` and `Skill` types in its responses module.
328+
329+
### Security considerations
330+
331+
**Scripts excluded**: The `scripts/` subdirectory is not supported in this implementation. As noted in LCORE-1339, executing arbitrary scripts poses security risks. Skills provide value through instructions; script support can be evaluated in a future phase.
332+
333+
**Path restrictions**: The `activate_skill` tool and reference file access are restricted to configured skill directories. The LLM cannot access arbitrary filesystem paths through skills.
334+
335+
**Trust model**: Since skills are configured by administrators in `lightspeed-stack.yaml`, there's an implicit trust that configured skill paths contain appropriate content.
336+
337+
## Appendix A: Existing approaches research
338+
339+
### How other tools handle skills
340+
341+
| Tool | Discovery | Activation | Script support |
342+
|------|-----------|------------|----------------|
343+
| Claude Code | Filesystem scan | File-read or /command | Yes |
344+
| GitHub Copilot | Filesystem scan | System prompt + tool | Yes |
345+
| Cursor | Filesystem scan | System prompt + tool | Yes |
346+
| OpenAI Codex | API-based | API-based | Yes |
347+
348+
### Alternative designs considered
349+
350+
**Filesystem scanning**: Rejected in favor of config-based discovery. Filesystem scanning adds complexity, requires directory configuration anyway, and could inadvertently load untrusted skills from cloned repositories.
351+
352+
**Inline content**: Rejected in favor of path-based. Inline content would clutter the YAML config for multi-skill deployments and doesn't support reference files.
353+
354+
**Always-loaded instructions**: Rejected in favor of catalog + on-demand. Loading all skill instructions upfront wastes context tokens and doesn't scale to many skills.
355+
356+
## Appendix B: Reference sources
357+
358+
- Agent Skills Specification: https://agentskills.io/specification
359+
- Agent Skills Implementation Guide: https://agentskills.io/client-implementation/adding-skills-support
360+
- Agent Skills GitHub: https://github.com/agentskills/agentskills
361+
- Example Skills: https://github.com/anthropics/skills

0 commit comments

Comments
 (0)