Skip to content

Commit 746ba55

Browse files
fizznixfizznix
authored
add mini-context-graph skill (#1580)
* add mini-context-graph skill * remove pycache files * filename case update to SKILL.md * update readme
1 parent 1f96bce commit 746ba55

16 files changed

Lines changed: 2343 additions & 0 deletions

‎docs/README.skills.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -230,6 +230,7 @@ See [CONTRIBUTING.md](../CONTRIBUTING.md#adding-skills) for guidelines on how to
230230
| [microsoft-skill-creator](../skills/microsoft-skill-creator/SKILL.md)<br />`gh skills install github/awesome-copilot microsoft-skill-creator` | Create agent skills for Microsoft technologies using Learn MCP tools. Use when users want to create a skill that teaches agents about any Microsoft technology, library, framework, or service (Azure, .NET, M365, VS Code, Bicep, etc.). Investigates topics deeply, then generates a hybrid skill storing essential knowledge locally while enabling dynamic deeper investigation. | `references/skill-templates.md` |
231231
| [migrating-oracle-to-postgres-stored-procedures](../skills/migrating-oracle-to-postgres-stored-procedures/SKILL.md)<br />`gh skills install github/awesome-copilot migrating-oracle-to-postgres-stored-procedures` | Migrates Oracle PL/SQL stored procedures to PostgreSQL PL/pgSQL. Translates Oracle-specific syntax, preserves method signatures and type-anchored parameters, leverages orafce where appropriate, and applies COLLATE "C" for Oracle-compatible text sorting. Use when converting Oracle stored procedures or functions to PostgreSQL equivalents during a database migration. | None |
232232
| [minecraft-plugin-development](../skills/minecraft-plugin-development/SKILL.md)<br />`gh skills install github/awesome-copilot minecraft-plugin-development` | Use this skill when building or modifying Minecraft server plugins for Paper, Spigot, or Bukkit, including plugin.yml setup, commands, listeners, schedulers, player state, team or arena systems, persistent progression, economy or profile data, configuration files, Adventure text, and version-safe API usage. Trigger for requests like "build a Minecraft plugin", "add a Paper command", "fix a Bukkit listener", "create plugin.yml", "implement a minigame mechanic", "add a perk or quest system", or "debug server plugin behavior". | `references/bootstrap-registration.md`<br />`references/build-test-and-runtime-validation.md`<br />`references/config-data-and-async.md`<br />`references/maps-heroes-and-feature-modules.md`<br />`references/minigame-instance-flow.md`<br />`references/persistent-progression-and-events.md`<br />`references/project-patterns.md`<br />`references/state-sessions-and-phases.md` |
233+
| [mini-context-graph](../skills/mini-context-graph/SKILL.md)<br />`gh skills install github/awesome-copilot mini-context-graph` | A persistent, compounding knowledge base combining Karpathy's LLM Wiki pattern<br />with a structured knowledge graph. Ingest documents once — the LLM writes wiki<br />pages, extracts entities/relations into the graph, and stores raw content for<br />evidence retrieval. Knowledge accumulates and cross-references; it is never<br />re-derived from scratch. | `references/ingestion.md`<br />`references/lint.md`<br />`references/ontology.md`<br />`references/retrieval.md`<br />`scripts/config.py`<br />`scripts/contextgraph.py`<br />`scripts/template_agent_workflow.py`<br />`scripts/tools` |
233234
| [mkdocs-translations](../skills/mkdocs-translations/SKILL.md)<br />`gh skills install github/awesome-copilot mkdocs-translations` | Generate a language translation for a mkdocs documentation stack. | None |
234235
| [model-recommendation](../skills/model-recommendation/SKILL.md)<br />`gh skills install github/awesome-copilot model-recommendation` | Analyze chatmode or prompt files and recommend optimal AI models based on task complexity, required capabilities, and cost-efficiency | None |
235236
| [msstore-cli](../skills/msstore-cli/SKILL.md)<br />`gh skills install github/awesome-copilot msstore-cli` | Microsoft Store Developer CLI (msstore) for publishing Windows applications to the Microsoft Store. Use when asked to configure Store credentials, list Store apps, check submission status, publish submissions, manage package flights, set up CI/CD for Store publishing, or integrate with Partner Center. Supports Windows App SDK/WinUI, UWP, .NET MAUI, Flutter, Electron, React Native, and PWA applications. | None |

‎skills/mini-context-graph/SKILL.md‎

Lines changed: 194 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,194 @@
1+
---
2+
name: mini-context-graph
3+
description: |
4+
A persistent, compounding knowledge base combining Karpathy's LLM Wiki pattern
5+
with a structured knowledge graph. Ingest documents once — the LLM writes wiki
6+
pages, extracts entities/relations into the graph, and stores raw content for
7+
evidence retrieval. Knowledge accumulates and cross-references; it is never
8+
re-derived from scratch.
9+
---
10+
11+
# Mini Context Graph Skill
12+
13+
## The Core Idea
14+
15+
Standard RAG re-discovers knowledge from scratch on every query. This skill is different:
16+
17+
1. **Wiki layer** — The LLM writes and maintains persistent markdown pages (summaries, entity pages, topic syntheses). Cross-references are already there. The wiki gets richer with every ingest.
18+
2. **Graph layer** — Entities and relations are extracted once and stored as a navigable knowledge graph. BFS traversal answers structural queries without re-reading sources.
19+
3. **Raw source layer** — Original documents are stored immutably with chunks. Provenance links tie every graph node and edge back to the exact text that supports it.
20+
21+
> The LLM writes; the Python tools handle all bookkeeping.
22+
23+
---
24+
25+
## Three Layers
26+
27+
| Layer | Where | What the LLM does | What Python does |
28+
|-------|-------|-------------------|-----------------|
29+
| **Raw Sources** | `data/documents.json` | Reads (never modifies) | Stores chunks + metadata |
30+
| **Wiki** | `wiki/` (markdown) | Writes/updates pages | Manages index.md + log.md |
31+
| **Graph** | `data/graph.json` | Extracts entities + relations | Persists, deduplicates, traverses |
32+
33+
---
34+
35+
## ⚡ Quick Start for Agents
36+
37+
```python
38+
from scripts.contextgraph import ContextGraphSkill
39+
from scripts.tools import wiki_store
40+
41+
skill = ContextGraphSkill()
42+
43+
# ===== INGEST WITH FULL RAG + WIKI =====
44+
# 1. Read references/ingestion.md and references/ontology.md first
45+
# 2. Extract entities and relations (LLM reasoning step)
46+
entities = [
47+
{"name": "memory leak", "type": "issue", "supporting_text": "memory leaks cause crashes"},
48+
{"name": "system crash", "type": "issue", "supporting_text": "system crashes due to memory leaks"},
49+
]
50+
relations = [
51+
{"source": "memory leak", "target": "system crash", "type": "causes",
52+
"confidence": 1.0, "supporting_text": "System crashes due to memory leaks."},
53+
]
54+
55+
result = skill.ingest_with_content(
56+
doc_id="doc_001",
57+
title="System Crash Analysis",
58+
source="/docs/incident_report.pdf",
59+
raw_content="System crashes due to memory leaks. Memory leaks occur when objects are not released.",
60+
entities=entities,
61+
relations=relations,
62+
)
63+
# result = {"doc_id": "doc_001", "chunk_count": 1, "nodes_added": 2, "edges_added": 1}
64+
65+
# 3. Write a wiki summary page for this document
66+
wiki_store.write_page(
67+
category="summary",
68+
title="System Crash Analysis Summary",
69+
content="""---
70+
title: System Crash Analysis
71+
source_document: doc_001
72+
tags: [summary, incident]
73+
---
74+
75+
# System Crash Analysis
76+
77+
**Source:** incident_report.pdf
78+
79+
## Key Claims
80+
81+
- [[memory-leak]] causes [[system-crash]] (confidence: 1.0)
82+
83+
## Entities
84+
85+
- [[memory-leak]] (issue)
86+
- [[system-crash]] (issue)
87+
""",
88+
summary="Incident report: memory leaks cause system crashes.",
89+
)
90+
91+
# ===== QUERY WITH EVIDENCE =====
92+
result = skill.query_with_evidence("Why does the system crash?")
93+
# Returns: {"query": ..., "subgraph": ..., "supporting_documents": [...], "evidence_chain": ...}
94+
95+
# ===== WIKI SEARCH (read wiki before answering) =====
96+
pages = wiki_store.search_wiki("memory leak")
97+
# Returns: [{slug, category, path, snippet}, ...]
98+
```
99+
100+
---
101+
102+
## Operations
103+
104+
### Ingest
105+
106+
When a user provides a new document:
107+
108+
1. Read `references/ingestion.md` — entity/relation extraction rules.
109+
2. Read `references/ontology.md` — type normalization rules.
110+
3. Extract entities and relations using your LLM reasoning.
111+
4. Call `skill.ingest_with_content(...)` — stores raw content + chunks + graph nodes + provenance.
112+
5. **Write a wiki summary page** using `wiki_store.write_page(category="summary", ...)`.
113+
6. **Update entity pages** — for each new/updated entity, write or update `wiki_store.write_page(category="entity", ...)`.
114+
7. **Update topic pages** if the document touches an existing synthesis topic.
115+
8. A single document ingest will typically touch 3–10 wiki pages.
116+
117+
### Query
118+
119+
When a user asks a question:
120+
121+
1. **Check the wiki first** — `wiki_store.search_wiki(query)` to find relevant pages. Read them.
122+
2. If the wiki has a good answer, synthesize from wiki pages (fast path).
123+
3. If deeper graph traversal is needed, call `skill.query_with_evidence(query)`.
124+
4. Return the answer with evidence citations from `supporting_documents`.
125+
5. If the answer is valuable, file it back as a new wiki topic page.
126+
127+
### Lint
128+
129+
Periodically health-check the wiki:
130+
131+
```python
132+
from scripts.tools import wiki_store
133+
issues = wiki_store.lint_wiki()
134+
# Returns: {orphan_pages, missing_pages, broken_wikilinks, isolated_pages}
135+
```
136+
137+
Ask the LLM to review and fix: broken links, orphan pages, stale claims, missing cross-references. See `references/lint.md` for full lint workflow.
138+
139+
---
140+
141+
## Ingestion Constraints
142+
143+
- ❌ Do NOT hallucinate entities not present in the text
144+
- ❌ Do NOT add relations without explicit textual evidence
145+
- ❌ Do NOT add edges with confidence < 0.6
146+
- ✅ Provide `supporting_text` for every entity and relation — this enables provenance
147+
- ✅ Write a wiki summary page for every ingested document
148+
- ✅ Update existing entity pages when new information arrives
149+
- ✅ Flag contradictions in wiki pages when new data conflicts with old claims
150+
151+
---
152+
153+
## Retrieval Constraints
154+
155+
- 🔒 Traversal depth MUST NOT exceed 2 (config: MAX_GRAPH_DEPTH)
156+
- 🔒 Only edges with confidence ≥ 0.6 (config: MIN_CONFIDENCE)
157+
- 🔒 Maximum 50 nodes returned (config: MAX_NODES)
158+
- ❌ Do NOT fabricate nodes or edges not in the graph
159+
160+
---
161+
162+
## Full Python API Reference
163+
164+
| Method | Purpose | When to Use |
165+
|--------|---------|-------------|
166+
| `skill.ingest_with_content(doc_id, title, source, raw_content, entities, relations)` | Full RAG ingest: raw docs + graph + provenance | Every new document |
167+
| `skill.add_node(name, node_type)` | Add single entity (no provenance) | Quick additions without a source doc |
168+
| `skill.add_edge(source_name, target_name, relation, confidence)` | Add single relation | Quick additions without a source doc |
169+
| `skill.query(query)` | Graph-only retrieval → subgraph | Structural queries |
170+
| `skill.query_with_evidence(query)` | Graph + provenance → subgraph + source chunks | Queries requiring citations |
171+
| `wiki_store.write_page(category, title, content, summary)` | Write/update a wiki page | After every ingest; after answering queries |
172+
| `wiki_store.read_page(category, title)` | Read a wiki page | Before answering; for cross-referencing |
173+
| `wiki_store.search_wiki(query)` | Keyword search across wiki | Fast path before graph traversal |
174+
| `wiki_store.list_pages(category)` | List all wiki pages | Getting an overview |
175+
| `wiki_store.get_log(last_n)` | Read recent operations | Understanding wiki history |
176+
| `wiki_store.lint_wiki()` | Health check | Periodic maintenance |
177+
| `documents_store.list_documents()` | List all ingested raw sources | Audit / provenance checking |
178+
| `documents_store.search_chunks(query)` | Chunk-level search | Finding specific evidence |
179+
180+
---
181+
182+
## Design Philosophy
183+
184+
> "The wiki is a persistent, compounding artifact. The cross-references are already there. The synthesis already reflects everything you've read." — Karpathy
185+
186+
| Layer | What Happens | Who Owns It |
187+
|-------|-----------|-------------|
188+
| **LLM Reasoning** | Extraction, synthesis, writing wiki pages | Agent (.md guidance files) |
189+
| **Wiki Persistence** | Index, log, file I/O | `wiki_store.py` |
190+
| **Graph Persistence** | Dedup, index, BFS traverse | `graph_store.py`, `retrieval_engine.py` |
191+
| **Raw Source Storage** | Immutable docs + chunks + provenance | `documents_store.py` |
192+
193+
The human curates sources and asks questions. The LLM writes the wiki, extracts the graph, and answers with citations. Python handles all bookkeeping.
194+

‎skills/mini-context-graph/references/ingestion.md‎

Lines changed: 196 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,196 @@
1+
# Ingestion Instructions
2+
3+
This file defines how the agent extracts entities and relations from a raw document.
4+
5+
---
6+
7+
## Step 1: Read the Document
8+
9+
Read the provided text carefully. Identify:
10+
- **Entities**: noun phrases that refer to real-world objects, systems, components, actors, concepts, or events.
11+
- **Relations**: verb phrases that describe how one entity affects, contains, causes, uses, or is related to another.
12+
13+
---
14+
15+
## Step 2: Extract Entities
16+
17+
For each entity:
18+
- Record its **name** (normalized: lowercase, strip leading/trailing whitespace)
19+
- Assign a **type**: a short label (1–3 words) that categorizes the entity
20+
21+
### Entity Type Examples
22+
23+
| Entity Name | Suggested Type |
24+
|-------------|---------------|
25+
| Python interpreter | software |
26+
| memory leak | issue |
27+
| operating system | system |
28+
| database | infrastructure |
29+
| user | actor |
30+
| API endpoint | interface |
31+
| server | infrastructure |
32+
33+
**Rules:**
34+
- Types must be general enough to reuse across documents
35+
- Do NOT create unique types per entity (e.g., avoid `python-interpreter-type`)
36+
- Use `ontology.md` normalization rules to canonicalize types
37+
38+
---
39+
40+
## Step 3: Extract Relations
41+
42+
For each pair of entities with an explicit connection in the text:
43+
- Record the **source** entity name
44+
- Record the **target** entity name
45+
- Record the **relation type**: a verb or verb phrase (normalized: lowercase)
46+
- Assign a **confidence** score between 0 and 1:
47+
- 1.0 = stated explicitly ("A causes B")
48+
- 0.8 = strongly implied ("A is linked to B")
49+
- 0.6 = weakly implied ("A may affect B")
50+
- < 0.6 = do NOT include
51+
52+
---
53+
54+
## Step 4: Output Format
55+
56+
Produce a JSON object in this exact format:
57+
58+
```json
59+
{
60+
"entities": [
61+
{ "name": "entity name", "type": "entity type", "supporting_text": "exact quote mentioning this entity" }
62+
],
63+
"relations": [
64+
{
65+
"source": "source entity name",
66+
"target": "target entity name",
67+
"type": "relation type",
68+
"confidence": 0.9,
69+
"supporting_text": "exact quote that justifies this relation"
70+
}
71+
]
72+
}
73+
```
74+
75+
The `supporting_text` field is **required for provenance**. It must be a verbatim or near-verbatim quote from the document that mentions or supports the entity/relation. This is what links graph nodes and edges back to their source.
76+
77+
---
78+
79+
## Rules
80+
81+
- All names and types must be **lowercase**
82+
- Only include relations where **both entities** are present in the entities list
83+
- Do NOT invent entities or relations not supported by the text
84+
- Prefer **reusing existing entity and relation types** from the ontology over creating new ones
85+
- One entity can appear in multiple relations (as source or target)
86+
- Always include `supporting_text` — this enables evidence retrieval and audit trails
87+
88+
---
89+
90+
## Step 5: Write Wiki Pages (Required)
91+
92+
After calling `skill.ingest_with_content(...)`, you MUST write wiki pages:
93+
94+
### 5a. Write a summary page for the document
95+
96+
```python
97+
from scripts.tools import wiki_store
98+
99+
wiki_store.write_page(
100+
category="summary",
101+
title=f"{title} Summary",
102+
content=f"""---
103+
title: {title}
104+
source_document: {doc_id}
105+
tags: [summary]
106+
---
107+
108+
# {title}
109+
110+
**Source:** {source}
111+
112+
## Key Claims
113+
114+
{chr(10).join(f'- [[{r["source"].replace(" ", "-")}]] {r["type"]} [[{r["target"].replace(" ", "-")}]] (confidence: {r["confidence"]})' for r in relations)}
115+
116+
## Entities
117+
118+
{chr(10).join(f'- [[{e["name"].replace(" ", "-")}]] ({e["type"]})' for e in entities)}
119+
120+
## Open Questions
121+
122+
- (Add questions from reading the document here)
123+
""",
124+
summary=f"Summary of {title}",
125+
)
126+
```
127+
128+
### 5b. Write or update entity pages
129+
130+
For each **new** entity not already in the wiki, write an entity page:
131+
132+
```python
133+
wiki_store.write_page(
134+
category="entity",
135+
title=entity_name,
136+
content=f"""---
137+
title: {entity_name}
138+
type: {entity_type}
139+
source_document: {doc_id}
140+
tags: [{entity_type}]
141+
---
142+
143+
# {entity_name}
144+
145+
(Description from the document or prior knowledge.)
146+
147+
## Relations
148+
149+
(List any wikilinks to related entities extracted from relations.)
150+
151+
## Mentioned in
152+
153+
- [[{doc_id}-summary]]
154+
""",
155+
summary=f"{entity_name}: {entity_type}",
156+
)
157+
```
158+
159+
For **existing** entity pages, read the current page and append new information, updated relations, or flag contradictions.
160+
161+
---
162+
163+
## Example
164+
165+
**Input document:**
166+
```
167+
System crashes due to memory leaks.
168+
Memory leaks occur when objects are not released.
169+
```
170+
171+
**Expected extraction output:**
172+
```json
173+
{
174+
"entities": [
175+
{ "name": "system crash", "type": "issue", "supporting_text": "system crashes due to memory leaks" },
176+
{ "name": "memory leak", "type": "issue", "supporting_text": "memory leaks occur when objects are not released" },
177+
{ "name": "object", "type": "component", "supporting_text": "objects are not released" }
178+
],
179+
"relations": [
180+
{
181+
"source": "memory leak",
182+
"target": "system crash",
183+
"type": "causes",
184+
"confidence": 1.0,
185+
"supporting_text": "System crashes due to memory leaks."
186+
},
187+
{
188+
"source": "object",
189+
"target": "memory leak",
190+
"type": "contributes to",
191+
"confidence": 0.9,
192+
"supporting_text": "Memory leaks occur when objects are not released."
193+
}
194+
]
195+
}
196+
```

0 commit comments

Comments
 (0)