LCORE-2664: add Confluence import PoC code and validation evidence

Moderate-ambition PoC validating the spike's core mechanism against real
Confluence Cloud (redhat.atlassian.net, space RHAT, 20 pages, read-only
API token):

- confluence_fetch.py: full + incremental crawl via v1 REST
  body.export_view; writes per-page HTML, manifest.json (id, title,
  canonical URL, version) and state.json (watermark + versions);
  incremental mode combines a CQL lastmodified delta with page-version
  comparison and detects deletions by full ID enumeration diff.
- confluence_processor.py: custom_processor-pattern build driver with
  ConfluenceMetadataProcessor resolving citation URL/title from the
  manifest (hermetic_build=True for auth-gated URLs); wires the docling
  HTMLReader via file_extractor and scopes the corpus with
  required_exts.
- confluence_test_page.py: helper for an editable test page; unused in
  the end (space creation is 403-forbidden for regular users), kept as
  evidence of the read-only-token constraint.
- poc-results/: sanitized evidence only (crawl stats, retrieval output
  with titles/URLs/scores, conversion-structure sample); raw crawled
  content is Red Hat internal and deliberately excluded.

Validated: end-to-end build (3.8 MB llamastack-faiss store), correct
top-3 retrieval with per-page URL citations, incremental re-run with
zero body fetches and zero re-embeds (2 API calls instead of 21).

Both directories are review-time artifacts and will be removed from the
branch before merge per docs/contributing/howto-organize-poc-output.md.

Author: max-svistunov

docs/design/byok-confluence-import/poc-results/README.md

@@ -0,0 +1,66 @@
+# PoC results: BYOK auto-import from Confluence (LCORE-2664)
+
+Evidence for the spike doc's PoC findings. Raw crawled content is **not**
+included: the PoC ran against the RHAT space on redhat.atlassian.net
+(Red Hat-internal content); this directory carries sanitized stats and
+excerpts only.
+
+## Design
+
+- **Target**: Confluence Cloud (redhat.atlassian.net), space `RHAT`
+  (20 pages), read-only API token of a regular user.
+- **Fetch**: v1 REST `body.export_view` (rendered HTML), one HTML file per
+  page + `manifest.json` (id → title, canonical URL, version) +
+  `state.json` (sync watermark, per-page versions).
+- **Build**: existing rag-content pipeline — docling `HTMLReader`,
+  `MarkdownNodeParser`, chunk 380/0, `all-mpnet-base-v2` (768-dim),
+  `llamastack-faiss` output.
+- **Verify**: `vector_io.query` via llama-stack library client, top-3.
+- **Incremental**: run 2 with `--incremental` — CQL
+  `lastmodified >= watermark − 10min` + full ID enumeration (deletion diff)
+  + version comparison to skip unchanged pages.
+
+## Results
+
+| Step | Outcome |
+|---|---|
+| Full crawl (run 1) | 20/20 pages fetched, ~1 API call/page + 1 listing call |
+| DB build | `faiss_store.db` 3.8 MB, no page failed conversion |
+| Retrieval | Top-3 chunks on a policy question: correct pages, scores 2.25/1.65/1.60 |
+| Citations | `title` + canonical page URL present on every chunk (see `retrieval-evidence.txt`) |
+| Incremental (run 2) | `fetched=0 unchanged=20 deleted=0` — 2 API calls total instead of 21 |
+| Conversion quality | Headings/bold/nested lists/expand-macros clean (see `conversion-sample.md` excerpt) |
+
+## Findings (carried into the spike doc)
+
+1. **The mechanism works end-to-end** with zero changes to rag-content
+   library code — only a fetch stage and a `MetadataProcessor` subclass.
+2. **Citations flow**: per-page Confluence URLs land in chunk metadata via
+   `url_function`; `hermetic_build=True` is required (page URLs are
+   auth-gated, reachability pings would mark all unreachable).
+3. **Read-only is all you get**: space creation 403-forbidden for regular
+   users on redhat.atlassian.net; the importer must not assume any write
+   permission. Update/delete sync paths are therefore implemented but
+   validated logically, not live (no writable page available).
+4. **Governance**: Red Hat's own Atlassian policy (retrieved by the PoC
+   itself) requires team integrations to use registered bot/service
+   accounts, not personal tokens — auth docs must say so.
+5. **Incidental bug**: the generated `llama-stack.yaml` +
+   `scripts/query_rag.py` fail out-of-the-box — re-registering the
+   vector store conflicts with the registration persisted inside
+   `faiss_store.db` (`provider_resource_id`/`vector_store_name`: None vs
+   set). Worked around by dropping `registered_resources.vector_stores`
+   and querying the persisted registration.
+6. **Portability gotcha confirmed live**: the built DB bakes the absolute
+   embedding-model path into `llama-stack.yaml` and the kv registry.
+7. **Incomplete-LFS gotcha**: an embeddings-model directory without
+   `model.safetensors` fails only at build time with an obscure error.
+
+## Implications
+
+- Recommended design (fetch stage + existing pipeline) is validated;
+  no design change needed.
+- Incremental sync as "skip unchanged re-embeds, full rebuild output"
+  is cheap and works; live update/delete validation moves to the e2e
+  test ticket (needs a writable test instance/mock).
+- Finding 5 becomes a proposed incidental JIRA.

docs/design/byok-confluence-import/poc-results/conversion-sample.md

@@ -0,0 +1,31 @@
+# Conversion quality evidence (sanitized)
+
+Largest crawled page ("Atlassian Cloud FAQ", export_view HTML → docling →
+Markdown, 0.14 s). Body text redacted (internal); structure shown verbatim:
+
+```
+# Atlassian Cloud FAQ
+
+## Review the answers to common questions you may have about our Cloud environment.
+
+##### **What is Atlassian Cloud?**
+
+[paragraph — clean prose, no markup artifacts]
+
+##### **What is the timeline and when is the migration?**
+
+**[bold lead sentence]**
+
+- [nested list, 2 levels, rendered correctly]
+    - [timezone conversion sub-items]
+```
+
+Observations:
+
+- Headings, bold, links, and 2-level nested lists all convert cleanly.
+- Confluence expand-macros arrive already expanded in `export_view`
+  (their content is present as regular headings/paragraphs).
+- No letter-spaced-font artifacts (the known docling weakness with
+  Confluence "Export to PDF" does not apply to the HTML path).
+- Minor: one stray list item rendered as an empty `-` bullet
+  (trailing empty list node in the source HTML) — harmless for chunking.

docs/design/byok-confluence-import/poc-results/crawl-runs.txt

@@ -0,0 +1,5 @@
+# Run 1 (full crawl) summary line:
+[full] spaces=['RHAT'] fetched=20 unchanged=0 deleted=0 total_on_disk=20
+
+# Run 2 (incremental, zero changes):
+[incremental] spaces=['RHAT'] fetched=0 unchanged=20 deleted=0 total_on_disk=20

docs/design/byok-confluence-import/poc-results/retrieval-evidence.txt

@@ -0,0 +1,20 @@
+Query: "What is the policy for using API scripts and bots in the Atlassian Cloud environment?"
+Store: llamastack-faiss, top_k=3, mode=vector
+
+chunk 1 | score=2.2482
+  title: APIs, Scripts, and Bots: Policy for Cloud Environment
+  docs_url: https://redhat.atlassian.net/wiki/spaces/RHAT/pages/303600805/APIs+Scripts+and+Bots+Policy+for+Cloud+Environment
+  text: "## Policy Statement\n\n- Personal experimentation using Atlassian REST APIs may use API tokens in a user's own account.\n- Team-supporting integrations must use a registered bot user or approved service account, not a personal token. [...]"
+
+chunk 2 | score=1.6468
+  title: APIs, Scripts, and Bots: Policy for Cloud Environment
+  docs_url: (same page as chunk 1)
+  text: [Purpose section — redacted, internal]
+
+chunk 3 | score=1.5955
+  title: API, Script, and Bot Policy - public
+  docs_url: https://redhat.atlassian.net/wiki/spaces/RHAT/pages/297468192/API+Script+and+Bot+Policy+-+public
+  text: "## Policy Statement\n\n- Users using Atlassian API, bots, or scripts against Jira or Confluence must have a bot user. All bots need to be registered with the PME team. [...]"
+
+All three chunks: correct pages for the query; every chunk carries title +
+canonical per-page Confluence URL in metadata (citation path proven).

docs/design/byok-confluence-import/poc/README.md

@@ -0,0 +1,51 @@
+# PoC: BYOK auto-import from Confluence (LCORE-2664 spike)
+
+Validates the core mechanism recommended by the spike doc: fetch Confluence
+pages as rendered HTML via the REST API, feed them to the existing
+`lightspeed-rag-content` pipeline (docling `HTMLReader` → Markdown →
+`MarkdownNodeParser` → embeddings), produce a `llamastack-faiss` vector DB,
+and query it with correct per-page citations. Also validates incremental
+change detection (CQL `lastmodified` + page-version comparison).
+
+Not production code: credentials come from the developer's Jira CLI
+credentials file, there is no retry/backoff, no attachments, no config
+surface.
+
+## Files
+
+| File | Purpose |
+|---|---|
+| `confluence_fetch.py` | Fetch stage: full + incremental crawl, manifest, state |
+| `confluence_processor.py` | `custom_processor.py`-pattern build driver with `ConfluenceMetadataProcessor` |
+| `confluence_test_page.py` | Test-space/page helper (unused in the end: space creation is 403-forbidden on redhat.atlassian.net; kept as evidence of the read-only constraint) |
+
+## Repro
+
+Prereqs: `~/.config/jira/credentials.json` (email + API token for
+redhat.atlassian.net), a checkout of `lightspeed-rag-content` with its uv
+venv, and a *complete* local copy of the embedding model
+(`model.safetensors` present — an LFS-less clone will fail).
+
+```bash
+RUN_DIR=/tmp/poc-run   # fetched content is Red Hat internal; keep out of the repo
+POC_DIR=docs/design/byok-confluence-import/poc
+
+# 1. Full crawl (run 1)
+python3 $POC_DIR/confluence_fetch.py --spaces RHAT --out $RUN_DIR/pages
+
+# 2. Build the vector DB (from the rag-content repo root)
+cd ../rag-content
+uv run python $POC_DIR/confluence_processor.py \
+  -f $RUN_DIR/pages -o $RUN_DIR/vector_db -i byok-confluence-poc \
+  -md embeddings_model -mn sentence-transformers/all-mpnet-base-v2 \
+  --vector-store-type llamastack-faiss
+
+# 3. Query (workaround script; scripts/query_rag.py hits a
+#    registered_resources conflict — see incidental findings in the spike doc)
+uv run python query_workaround.py "your question here" 3
+
+# 4. Incremental crawl (run 2) — re-uses state.json, fetches only changes
+python3 $POC_DIR/confluence_fetch.py --spaces RHAT --out $RUN_DIR/pages --incremental
+```
+
+Results and evidence: see `../poc-results/`.

docs/design/byok-confluence-import/poc/confluence_fetch.py

@@ -0,0 +1,189 @@
+#!/usr/bin/env python3
+"""PoC: fetch Confluence pages as rendered HTML for the BYOK RAG pipeline.
+
+Fetches all pages from the given Confluence spaces via the v1 REST API
+(`body.export_view` = rendered HTML), writes one HTML file per page plus a
+`manifest.json` (page id -> title/url/version) consumed by the metadata
+processor, and a `state.json` sync watermark.
+
+Modes:
+  - Full crawl (default, or no state file): enumerate every page in each space.
+  - Incremental (`--incremental`, requires state file): CQL `lastmodified >=`
+    watermark to find changed pages, full ID enumeration to detect deletions.
+
+PoC shortcuts (not production shape): credentials read from the Jira CLI
+credentials file; no retry/backoff beyond honoring nothing; no attachments.
+"""
+
+import argparse
+import base64
+import datetime
+import json
+import pathlib
+import sys
+import urllib.parse
+import urllib.request
+
+CRED_FILE = pathlib.Path.home() / ".config/jira/credentials.json"
+BASE = "https://redhat.atlassian.net"
+EXPAND = "body.export_view,version,space"
+# Overlap buffer so CQL minute granularity / timezone skew can't miss edits;
+# version comparison deduplicates re-fetches.
+WATERMARK_OVERLAP_MIN = 10
+
+
+def _auth_header() -> str:
+    creds = json.loads(CRED_FILE.read_text())
+    token = base64.b64encode(f"{creds['email']}:{creds['token']}".encode()).decode()
+    return f"Basic {token}"
+
+
+def api_get(path: str, params: dict | None = None) -> dict:
+    url = f"{BASE}{path}"
+    if params:
+        url += "?" + urllib.parse.urlencode(params)
+    req = urllib.request.Request(
+        url, headers={"Authorization": _auth_header(), "Accept": "application/json"}
+    )
+    with urllib.request.urlopen(req, timeout=60) as resp:
+        return json.load(resp)
+
+
+def list_space_page_ids(space: str) -> dict[str, int]:
+    """Enumerate all page ids (id -> version) in a space, no bodies."""
+    ids: dict[str, int] = {}
+    start = 0
+    while True:
+        res = api_get(
+            "/wiki/rest/api/content",
+            {"spaceKey": space, "type": "page", "limit": 100, "start": start,
+             "expand": "version"},
+        )
+        for page in res.get("results", []):
+            ids[page["id"]] = page["version"]["number"]
+        if len(res.get("results", [])) < 100:
+            return ids
+        start += 100
+
+
+def fetch_page(page_id: str) -> dict:
+    return api_get(f"/wiki/rest/api/content/{page_id}", {"expand": EXPAND})
+
+
+def changed_page_ids(spaces: list[str], since_utc: datetime.datetime) -> set[str]:
+    """CQL delta: page ids modified since the watermark (minus overlap)."""
+    since = since_utc - datetime.timedelta(minutes=WATERMARK_OVERLAP_MIN)
+    space_list = ",".join(f'"{s}"' for s in spaces)
+    cql = (
+        f"space in ({space_list}) and type=page"
+        f' and lastmodified >= "{since.strftime("%Y/%m/%d %H:%M")}"'
+    )
+    ids: set[str] = set()
+    start = 0
+    while True:
+        res = api_get(
+            "/wiki/rest/api/content/search",
+            {"cql": cql, "limit": 100, "start": start},
+        )
+        ids.update(r["id"] for r in res.get("results", []))
+        if len(res.get("results", [])) < 100:
+            return ids
+        start += 100
+
+
+def write_page(page: dict, out_dir: pathlib.Path) -> dict:
+    """Write one page as HTML; return its manifest entry."""
+    title = page["title"]
+    html_body = page["body"]["export_view"]["value"]
+    doc = (
+        "<!DOCTYPE html>\n<html>\n<head>\n<meta charset=\"utf-8\">\n"
+        f"<title>{title}</title>\n</head>\n<body>\n"
+        f"<h1>{title}</h1>\n{html_body}\n</body>\n</html>\n"
+    )
+    filename = f"{page['id']}.html"
+    (out_dir / filename).write_text(doc, encoding="utf-8")
+    return {
+        "id": page["id"],
+        "title": title,
+        "url": BASE + "/wiki" + page["_links"]["webui"],
+        "version": page["version"]["number"],
+        "when": page["version"]["when"],
+        "space": page["space"]["key"],
+        "file": filename,
+    }
+
+
+def main() -> int:
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument("--spaces", nargs="+", required=True, help="Space keys to crawl")
+    parser.add_argument("--out", required=True, help="Output directory for HTML files")
+    parser.add_argument("--incremental", action="store_true",
+                        help="Use CQL delta + deletion diff against state file")
+    args = parser.parse_args()
+
+    out_dir = pathlib.Path(args.out)
+    out_dir.mkdir(parents=True, exist_ok=True)
+    manifest_path = out_dir / "manifest.json"
+    state_path = out_dir / "state.json"
+
+    manifest: dict[str, dict] = (
+        json.loads(manifest_path.read_text()) if manifest_path.exists() else {}
+    )
+    state: dict = json.loads(state_path.read_text()) if state_path.exists() else {}
+    run_started = datetime.datetime.now(datetime.timezone.utc)
+
+    known_versions: dict[str, int] = state.get("pages", {})
+    stats = {"fetched": 0, "unchanged": 0, "deleted": 0, "api_calls_saved": 0}
+
+    # Current inventory (id -> version) across all spaces: needed for deletion
+    # detection in both modes, and as the fetch list in full mode.
+    inventory: dict[str, int] = {}
+    for space in args.spaces:
+        inventory.update(list_space_page_ids(space))
+
+    if args.incremental and state.get("last_sync"):
+        watermark = datetime.datetime.fromisoformat(state["last_sync"])
+        candidates = changed_page_ids(args.spaces, watermark)
+        # Also anything present in inventory but unknown to state (new pages
+        # CQL may have missed due to timezone skew).
+        candidates.update(pid for pid in inventory if pid not in known_versions)
+        to_fetch = {
+            pid for pid in candidates
+            if pid in inventory and inventory[pid] != known_versions.get(pid)
+        }
+        stats["unchanged"] = len(inventory) - len(to_fetch)
+        stats["api_calls_saved"] = stats["unchanged"]
+    else:
+        to_fetch = set(inventory)
+
+    for pid in sorted(to_fetch):
+        entry = write_page(fetch_page(pid), out_dir)
+        manifest[entry["file"]] = entry
+        stats["fetched"] += 1
+        print(f"  fetched [{entry['space']}] v{entry['version']} {entry['title']}")
+
+    # Deletion diff: anything on disk whose page id vanished from inventory.
+    for filename in list(manifest):
+        pid = manifest[filename]["id"]
+        if pid not in inventory:
+            (out_dir / filename).unlink(missing_ok=True)
+            del manifest[filename]
+            stats["deleted"] += 1
+            print(f"  deleted {filename}")
+
+    manifest_path.write_text(json.dumps(manifest, indent=2))
+    state_path.write_text(json.dumps({
+        "last_sync": run_started.isoformat(),
+        "spaces": args.spaces,
+        "pages": inventory,
+    }, indent=2))
+
+    mode = "incremental" if args.incremental and state.get("last_sync") else "full"
+    print(f"[{mode}] spaces={args.spaces} fetched={stats['fetched']} "
+          f"unchanged={stats['unchanged']} deleted={stats['deleted']} "
+          f"total_on_disk={len(manifest)}")
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())