Replies: 3 comments
-
|
This is a great start. It will already unlock a lot of value. E.g., determine the highest RoI campaign from previous runs. We should include the commit IDs of nous & the target repo (e.g., BLIS) when available in the knowledge wiki. In terms of future work, I think the campaigns produce a bunch of other artifacts like code patches, detailed json results from the run (e.g., run of the simulations). We need a "raw" data store for the campaign artifacts as well. Perhaps a pointer to the raw data (in the wiki entry) would also be useful. During campaigns, we should be able to attach metadata, which should carry over to the wiki entries. |
Beta Was this translation helpful? Give feedback.
-
Metadata Capture in NousAll three suggestions are feasible. Here is how Nous can capture and maintain this metadata so it carries through to LLM Wiki automatically. Where it lives
{
"run_id": "epp-saturation-detector",
"phase": "DONE",
"owner": "sriumcp",
"target_repo": "AI-native-Systems-Research/inference-sim",
"target_commit": "4e88b200...",
"nous_commit": "abc123def...",
"metadata": {
"goal": "30% capacity improvement at saturation onset",
"tags": ["flow-control", "saturation", "aimd"]
}
}When it gets capturedIn import subprocess
from pathlib import Path
def _capture_git_meta(repo_path: str | None) -> dict:
meta = {}
# Owner from git username
meta["owner"] = subprocess.check_output(
["git", "config", "user.name"], text=True
).strip()
# Target repo commit + remote
if repo_path:
meta["target_commit"] = subprocess.check_output(
["git", "-C", repo_path, "rev-parse", "HEAD"], text=True
).strip()
remote = subprocess.check_output(
["git", "-C", repo_path, "remote", "get-url", "origin"], text=True
).strip()
# extract org/repo from remote URL
if ":" in remote and "github.com" in remote:
meta["target_repo"] = remote.split(":")[-1].removesuffix(".git")
elif "github.com/" in remote:
meta["target_repo"] = remote.split("github.com/")[-1].removesuffix(".git")
else:
meta["target_repo"] = remote
# Nous commit (the framework itself)
nous_dir = Path(__file__).parent
meta["nous_commit"] = subprocess.check_output(
["git", "-C", str(nous_dir), "rev-parse", "HEAD"], text=True
).strip()
return metaThis runs once at campaign init — before any experiments execute — so it captures the exact state the experiments ran against. Custom metadata from campaign.yamlAllow a research_question: >
Can AIMD improve saturation onset by >30%?
metadata:
tags: [flow-control, saturation]
goal: "30% capacity improvement"
jira: PERF-1234This gets merged into How it flows to the wiki
---
campaign: epp-saturation-detector
owner: sriumcp
target_repo: AI-native-Systems-Research/inference-sim
target_commit: 4e88b200
nous_commit: abc123def
tags: [flow-control, saturation, aimd]
---
# EPP Saturation Detector Report
...LLM Wiki preserves frontmatter as queryable page metadata — searchable by owner, target repo, or tags across all campaigns. Raw data pointersFor the raw artifacts (patches, simulation result JSONs), the exported markdown includes a pointer section: ## Raw Data
- Campaign dir: `.nous/epp-saturation-detector-archive-20260519-115428/`
- Patches: `runs/iter-2/patches/h-main.patch`, `runs/iter-3/patches/h-main.patch`
- Result count: 87 JSON files across 12 iterationsLLM Wiki keeps the narrative as a wiki page; the user follows the path to actual files. We avoid dumping hundreds of raw JSON simulation outputs into the wiki (they would not ingest well anyway). |
Beta Was this translation helpful? Give feedback.
-
Design Tradeoffs — Findings from Hands-On InvestigationWe spent a session probing LLM Wiki's API, testing its ingestion pipeline, and evaluating whether it can serve as a reasoning engine (not just documentation). Here's what we learned. What's Programmatically Achievable via the APILLM Wiki exposes a local HTTP API at
Auth: Bearer token configured in What works well:
What doesn't work:
When the App Must Be Open vs. Not
Bottom line: Any operation that involves the LLM (page generation, re-indexing) requires the app. Read/write of files on disk does not. What Value the Knowledge Graph BringsThe graph is computed FROM wiki pages (wikilinks between them). It provides: For humans (browsing):
For automated pipelines:
What the graph does NOT provide:
Assessment: The graph is a nice visualization layer for humans. For automated "suggest next experiment" pipelines, it adds marginal value over just reading the wiki pages directly — the LLM can reason about connections and gaps without needing quantitative topology. LLM Wiki as a Reasoning Engine — What We TestedWe tried customizing
Conclusion: LLM Wiki is a documentation tool. It converts source documents into structured reference pages. It is NOT a reasoning engine that can assess experimental outcomes and generate recommendations. The Claude Code Skill (
|
| Criterion | LLM Wiki | Custom Claude Skill |
|---|---|---|
| Ingestion (docs → structured pages) | Excellent — automatic, handles PDF/DOCX/MD | We'd build this ourselves |
| Knowledge graph visualization | Built-in, interactive | Would need separate tooling |
| Semantic search | Built-in (LanceDB vectors) | Would need to implement |
| Reasoning about next experiments | Cannot do this | Full control over prompts |
| Custom page types / output format | Limited to hardcoded types | Full flexibility |
| Headless operation | Not supported — app must be open | Runs anywhere Python runs |
| Team sharing | Wiki is just a folder (git/Box) | Same — markdown files on disk |
| Maintenance | Third-party app, updates may break API | We own and maintain it |
| Dependencies | macOS/Linux desktop app + Tauri | Python + LLM API endpoint |
When to Use LLM Wiki
- You want a browsable knowledge base for humans to explore visually
- You're ingesting diverse document types (PDFs, papers, DOCX) and want automatic structuring
- The graph visualization helps your team spot patterns
- You're OK keeping the app open during campaign runs
When to Build a Custom Skill
- The primary goal is automated reasoning (suggest next experiments, identify dead ends)
- You want full control over the prompt, output format, and page types
- You need headless/CI operation — no GUI dependency
- The output must be machine-readable by the orchestrator (structured YAML, not prose wiki pages)
Recommendation
Use both, with clear separation of concerns:
-
LLM Wiki — the knowledge base layer. Accumulates campaign outputs as searchable, interlinked documentation. Good for human exploration and as context for future campaigns. Keep the export hook (
nous2wiki.py) as proposed. -
Custom Claude skill — the reasoning layer. Reads raw campaign data (ledger, principles, findings) directly. Reasons about what worked, what failed, and what's unexplored. Outputs structured
campaign.yamlcandidates. No app dependency.
The two don't compete — LLM Wiki answers "what do we know?" and the custom skill answers "what should we do next?"
Alternatively, if the team decides the graph/visualization isn't worth the app-open constraint, a custom skill can handle both: accumulate knowledge as markdown files (same format LLM Wiki uses) and reason over them to suggest next experiments. This is simpler but loses the interactive graph UI.
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
-
Proposal
Connect Nous campaign outputs to LLM Wiki so that results from multiple campaigns accumulate into a searchable, interlinked knowledge graph — rather than sitting as isolated JSON files under
.nous/.The Problem
Each Nous campaign produces rich structured artifacts (
principles.json,ledger.json,report.md), but:.nous/<run_id>/directories with no cross-campaign visibilityWhat LLM Wiki Provides
LLM Wiki is a local-first desktop app that ingests documents and builds an interlinked wiki with:
[[wikilinks]]It stores everything as plain Markdown (Obsidian-compatible) and runs entirely locally.
Proposed Integration
Architecture
Multiple campaigns feed one wiki project. LLM Wiki cross-links them.
What Gets Exported
report.mdprinciples.jsonprinciples.mdledger.jsonledger.mdFilter: only export campaigns where
experiment_valid: trueon at least one iteration.Where to Hook It
In
run_campaign.py, right after_generate_report():Opt-in via
defaults.yaml:The Export Script (
nous2wiki.py)~100 lines of Python. Renders
principles.jsonandledger.jsonto readable markdown, copiesreport.mdverbatim, writes to the wiki raw/sources directory. No dependencies beyond stdlib + PyYAML.Supports single campaign or batch export:
Setup (Local, quick)
Install LLM Wiki — download the pre-built
.dmg(macOS) or.AppImage(Linux) from releases. No build step needed.Configure LLM provider — in LLM Wiki Settings, set it as OpenAI-compatible pointing at your LiteLLM proxy:
claude-sonnet-4-6(whatever your proxy exposes)LLM Wiki uses the standard
/v1/chat/completionsformat, so any LiteLLM proxy works directly.Create a project in LLM Wiki (e.g., "Nous Research"), then set that path in
defaults.yaml.Run campaigns as normal — results auto-export to the wiki on completion.
Value
defaults.yaml, automatic after thatFuture Extensions
localhost:19828)Questions
run_campaign.py) or a separate post-processing script?Beta Was this translation helpful? Give feedback.
All reactions