Skip to content

Commit d4eb1e4

Browse files
docs: Content Signals in robots.txt for AI content-usage preferences (#179)
* docs: declare AI content usage preferences via Content Signals in robots.txt Add docs/robots.txt with a Content-Signal directive (search=yes, ai-input=yes, ai-train=yes) per contentsignals.org / draft-romm-aipref-contentsignals, declaring that this public protocol documentation may be indexed, used to ground AI answers, and used for training. Values mirror the site's existing stance of explicitly allowing all AI crawlers (GPTBot, ClaudeBot, Google-Extended, etc.). Also includes the basic allow rules, AI-crawler allowlist, and sitemap reference so robots.txt is complete on its own (main had none). * docs: add Content-Signal directive to robots.txt Re-apply the Content Signals declaration (search=yes, ai-input=yes, ai-train=yes) per contentsignals.org / draft-romm-aipref-contentsignals. It was dropped when main (which already added robots.txt via #176) was merged into this branch. Values mirror the existing AI-crawler allowlist. * docs: publish .well-known agent-discovery files (MCP card + Agent Skills) Add hooks/emit_well_known.py to copy well-known/ into the built site's .well-known/ dir, publishing an MCP Server Card (SEP-1649) and an Agent Skills discovery index (v0.2.0) whose digests are computed at build time. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
1 parent 0ce3e33 commit d4eb1e4

5 files changed

Lines changed: 191 additions & 0 deletions

File tree

docs/robots.txt

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,9 @@
11
User-agent: *
22
Allow: /
3+
# Content Signals (contentsignals.org / draft-romm-aipref-contentsignals)
4+
# Public protocol documentation: we want it indexed, used to ground AI
5+
# answers, and available for training. Flip any value to "no" to opt out.
6+
Content-Signal: search=yes, ai-input=yes, ai-train=yes
37

48
# AI crawlers / assistants — explicitly allowed
59
User-agent: GPTBot

hooks/emit_well_known.py

Lines changed: 107 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,107 @@
1+
"""MkDocs hook: publish ``.well-known`` agent-discovery files.
2+
3+
MkDocs ignores files and directories whose names start with a dot, so a
4+
``.well-known`` directory under ``docs/`` is never copied into the built site.
5+
Instead we keep the source files under ``well-known/`` (no leading dot) at the
6+
repo root and copy them into ``<site_dir>/.well-known/`` here, mirroring the way
7+
``hooks/emit_markdown.py`` writes ``llms.txt`` straight into the site dir.
8+
9+
Two things are published:
10+
11+
1. ``/.well-known/mcp/server-card.json`` — MCP Server Card (SEP-1649) advertising
12+
the DoubleZero Data MCP server at https://data.malbeclabs.com/mcp.
13+
14+
2. ``/.well-known/agent-skills/index.json`` — Agent Skills discovery index
15+
(Agent Skills Discovery RFC v0.2.0). Each ``SKILL.md`` found under
16+
``well-known/agent-skills/<name>/`` becomes an entry whose ``digest`` is the
17+
SHA-256 of the copied artifact, computed at build time so it can never drift.
18+
19+
GitHub Pages serves the ``.well-known`` directory (it is exempt from the usual
20+
dotfile exclusion), and the deploy workflow tars the site dir with ``-cvf .``,
21+
so the hidden directory is preserved in the uploaded artifact.
22+
"""
23+
24+
from __future__ import annotations
25+
26+
import hashlib
27+
import json
28+
import os
29+
import shutil
30+
31+
SKILLS_SCHEMA = "https://schemas.agentskills.io/discovery/0.2.0/schema.json"
32+
SITE_BASE = "https://docs.malbeclabs.com"
33+
34+
35+
def _repo_root(config) -> str:
36+
"""Directory containing mkdocs.yml."""
37+
return os.path.dirname(os.path.abspath(config["config_file_path"]))
38+
39+
40+
def _frontmatter_description(skill_md_path: str) -> str:
41+
"""Best-effort read of the ``description:`` field from a SKILL.md YAML
42+
front matter block. Supports a single-line value or a folded/literal block
43+
(``>-`` / ``|``) whose continuation lines are indented."""
44+
try:
45+
with open(skill_md_path, encoding="utf-8") as fh:
46+
lines = fh.read().splitlines()
47+
except OSError:
48+
return ""
49+
50+
if not lines or lines[0].strip() != "---":
51+
return ""
52+
53+
desc_parts: list[str] = []
54+
capturing = False
55+
for line in lines[1:]:
56+
if line.strip() == "---":
57+
break
58+
if capturing:
59+
if line and not line[0].isspace():
60+
break # next top-level key
61+
stripped = line.strip()
62+
if stripped:
63+
desc_parts.append(stripped)
64+
continue
65+
if line.startswith("description:"):
66+
value = line[len("description:"):].strip()
67+
if value and value not in (">", "|", ">-", "|-", ">+", "|+"):
68+
return value
69+
capturing = True # block scalar; gather indented lines below
70+
71+
return " ".join(desc_parts)
72+
73+
74+
def on_post_build(config, **kwargs) -> None:
75+
src_root = os.path.join(_repo_root(config), "well-known")
76+
if not os.path.isdir(src_root):
77+
return
78+
79+
dest_root = os.path.join(config["site_dir"], ".well-known")
80+
shutil.copytree(src_root, dest_root, dirs_exist_ok=True)
81+
82+
# Build the Agent Skills discovery index from the copied SKILL.md files.
83+
skills_dir = os.path.join(dest_root, "agent-skills")
84+
if not os.path.isdir(skills_dir):
85+
return
86+
87+
skills = []
88+
for name in sorted(os.listdir(skills_dir)):
89+
skill_md = os.path.join(skills_dir, name, "SKILL.md")
90+
if not os.path.isfile(skill_md):
91+
continue
92+
with open(skill_md, "rb") as fh:
93+
digest = hashlib.sha256(fh.read()).hexdigest()
94+
skills.append(
95+
{
96+
"name": name,
97+
"type": "skill-md",
98+
"description": _frontmatter_description(skill_md),
99+
"url": f"{SITE_BASE}/.well-known/agent-skills/{name}/SKILL.md",
100+
"digest": f"sha256:{digest}",
101+
}
102+
)
103+
104+
index = {"$schema": SKILLS_SCHEMA, "skills": skills}
105+
with open(os.path.join(skills_dir, "index.json"), "w", encoding="utf-8") as fh:
106+
json.dump(index, fh, indent=2, ensure_ascii=False)
107+
fh.write("\n")

mkdocs.yml

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -114,3 +114,4 @@ extra_javascript:
114114
- path: https://cdn.jsdelivr.net/npm/mathjax@3/es5/tex-mml-chtml.js
115115
hooks:
116116
- hooks/emit_markdown.py
117+
- hooks/emit_well_known.py
Lines changed: 64 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,64 @@
1+
---
2+
name: doublezero-docs
3+
description: >-
4+
Help users connect to and operate on the DoubleZero / Malbec Labs network —
5+
validator and tenant setup, multicast, rewards, geolocation, and contributor
6+
operations — by grounding answers in the official documentation and live
7+
network data.
8+
license: See https://docs.malbeclabs.com/
9+
---
10+
11+
# DoubleZero documentation assistant
12+
13+
Use this skill when a user asks how to connect to, operate on, or troubleshoot
14+
the DoubleZero network (also referred to as Malbec Labs): validator and tenant
15+
setup, multicast publishing/subscribing, validator rewards, geolocation, or
16+
contributor (network operator) tasks.
17+
18+
## Ground answers in the documentation
19+
20+
The full documentation is published as clean Markdown on the docs domain. Always
21+
prefer these sources over general knowledge, and cite the page you used.
22+
23+
- **Index of pages:** `https://docs.malbeclabs.com/llms.txt` — a curated list of
24+
every page with its Markdown URL and a one-line description. Read this first to
25+
find the right page.
26+
- **Full corpus:** `https://docs.malbeclabs.com/llms-full.txt` — every page
27+
concatenated into a single Markdown file, when you need broad context.
28+
- **Per-page Markdown:** append `index.md` to any page URL, e.g.
29+
`https://docs.malbeclabs.com/setup/index.md` or
30+
`https://docs.malbeclabs.com/DZ%20Testnet%20Connection/index.md`. Translations
31+
live under a locale prefix, e.g. `https://docs.malbeclabs.com/es/setup/index.md`.
32+
33+
## Query live network data (MCP)
34+
35+
For questions about the current state of the network — which users/devices exist,
36+
publisher/subscriber status, link health, telemetry — use the DoubleZero Data MCP
37+
server instead of guessing.
38+
39+
- **Endpoint:** `https://data.malbeclabs.com/mcp` (MCP Streamable HTTP)
40+
- **Discovery:** `https://data.malbeclabs.com/.well-known/mcp/server-card.json`
41+
- **Tools:** `get_schema` (always call first to learn the available tables,
42+
columns, and types — never assume names), `execute_sql` and `execute_cypher`
43+
(read-only queries over the data), and `read_docs` (conceptual/procedural docs
44+
pages). All tools are read-only.
45+
46+
## Workflow
47+
48+
1. Identify whether the question is about **procedure/concepts** (use the docs)
49+
or **live state** (use the MCP), or both.
50+
2. For docs: fetch `llms.txt`, pick the relevant page(s), then fetch the
51+
`index.md` Markdown for those pages.
52+
3. For live data: call `get_schema` first, then issue a read-only `execute_sql`
53+
or `execute_cypher` query.
54+
4. Answer concisely and link the user to the canonical page on
55+
`https://docs.malbeclabs.com/` so they can follow the full steps.
56+
57+
## Notes
58+
59+
- The network is public and read-only from an agent's perspective; there is no
60+
agent login or write API. Do not attempt to register credentials.
61+
- Dashboards referenced in the docs (e.g.
62+
`https://data.malbeclabs.com/dz/publisher-check`,
63+
`https://data.malbeclabs.com/dz/users`) are human-facing; for programmatic
64+
access use the MCP tools above.

well-known/mcp/server-card.json

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
{
2+
"serverInfo": {
3+
"name": "DoubleZero Data",
4+
"version": "1.0.0"
5+
},
6+
"transport": {
7+
"type": "streamable-http",
8+
"endpoint": "https://data.malbeclabs.com/mcp"
9+
},
10+
"capabilities": {
11+
"tools": true,
12+
"resources": false,
13+
"prompts": false
14+
}
15+
}

0 commit comments

Comments
 (0)