diff --git a/skills/paper-glossary/README.md b/skills/paper-glossary/README.md index 352bd02..c0fb6e6 100644 --- a/skills/paper-glossary/README.md +++ b/skills/paper-glossary/README.md @@ -1,11 +1,44 @@ # Paper Glossary -Companion skill for building reusable Obsidian glossary notes from paper source artifacts. +`paper-glossary` builds reusable Obsidian glossary notes from `*_source_manifest.json` and `*_raw_sections.jsonl`. It remains self-contained and does not call a paper-reading workflow. -This directory is self-contained: scripts, tests, and file-contract reference material live under `skills/paper-glossary/`. It collaborates with paper-reading workflows only through `*_source_manifest.json` and `*_raw_sections.jsonl`. +## One-Time Device Setup -Run the focused tests with: +Each person configures one term directory once per device. The directory must be inside an Obsidian vault: its nearest vault ancestor must contain `.obsidian`. The local configuration is `~/.paper-glossary/config.json`; it stores an absolute resolved vault root and the terms subdirectory, not a repository setting. Relative and Windows drive-relative vault roots are rejected. Later runs reuse it. + +Setup may create `~/.paper-glossary/config.json`; before selection, no glossary notes or article Markdown are written. Run commands from the repository root and replace each quoted `` placeholder with the real vault path. + +## Windows PowerShell + +```powershell +py -3.12 skills\paper-glossary\scripts\configure_glossary.py --terms-dir '\Glossary' +py -3.12 skills\paper-glossary\scripts\configure_glossary.py --show +py -3.12 skills\paper-glossary\scripts\configure_glossary.py --reset +py -3.12 skills\paper-glossary\scripts\configure_glossary.py --validate-article '\Papers\example.md' +py -3.12 -m pytest -q skills\paper-glossary\tests --basetemp .pytest-tmp-paper-glossary +``` + +## macOS / Linux Bash ```bash -py -3.12 -m pytest -q skills/paper-glossary/tests --basetemp .pytest-tmp-paper-glossary +python3 skills/paper-glossary/scripts/configure_glossary.py --terms-dir '/Glossary' +python3 skills/paper-glossary/scripts/configure_glossary.py --show +python3 skills/paper-glossary/scripts/configure_glossary.py --reset +python3 skills/paper-glossary/scripts/configure_glossary.py --validate-article '/Papers/example.md' +python3 -m pytest -q skills/paper-glossary/tests --basetemp .pytest-tmp-paper-glossary ``` + +When article links are requested, provide the article Markdown path explicitly. It must be a `.md` file in the same Obsidian vault as the configured term directory; the skill never guesses an article from the paper source artifacts. + +`--source-manifest` is required for proposal, review recording, and triage. `--raw-sections` is an optional override for the manifest's `raw_sections_path`, not a standalone replacement. The artifact workflow is: + +1. Run `plan_glossary.py --propose ... --output proposal.json`. +2. Perform one semantic review, then record its exact ordered subset with `--review-proposal proposal.json --reviewed-terms ... --output reviewed.json`. + If this produces an **empty reviewed shortlist**, report `no_candidates` and stop without displaying an empty selector. +3. Display the complete reviewed shortlist as numbers and wait. Resolve number responses or `全部写入` to exact names outside the CLI. +4. Run triage with `--reviewed-shortlist reviewed.json --terms ...`; `--terms` alone is rejected. +5. Pass triage to inventory, inventory to the writer at the **configured glossary directory** (use `--config-path` only to select that device-local configuration), and successful writer results to the linker when an article was requested. + +For inventory, writer, and linker, always pass the current `--source-manifest`, optional `--raw-sections`, and saved `--reviewed-shortlist`. Inventory reads the saved `--triage` as its input; writer and linker also require that same saved `--triage` and enforce exact ordered `term` and `surface_forms` equality with it. These post-triage CLIs recompute the current proposal/review chain before writing. When the writer receives `--article`, `paper_link` is **derived from the resolved article Markdown stem**. A glossary-only writer invocation omits `--article` and derives `paper_link` from the **validated manifest `paper_id`**; do not run the linker for that branch because it rejects a writer artifact with **no bound article path**. + +Proposal candidates carry ordered paper-grounded `surface_forms` plus SHA-256 provenance and the bounded candidate/summary material needed to recompute it. Inventory and writer both fail closed unless proposal, review, and selection digests recompute from one matching paper/source chain and the exact ordered forms. Review, triage, inventory, and writer results preserve those forms; model-authored note aliases are metadata and are not article-match forms. Writer output also preserves `triage_sha256`, validated provenance, and context and binds the triage identity plus exact ordered mappings with `mappings_sha256`; the linker recomputes that digest and validates every note path, stem, and grounded form. See `SKILL.md` and `references/file-contract.md` for exact article and glossary-only invocations and JSON contracts. diff --git a/skills/paper-glossary/SKILL.md b/skills/paper-glossary/SKILL.md index fe575ea..22161db 100644 --- a/skills/paper-glossary/SKILL.md +++ b/skills/paper-glossary/SKILL.md @@ -1,72 +1,43 @@ --- name: paper-glossary -description: Use when building reusable glossary or term-library notes for research papers from an existing paper source manifest, raw_sections_path, *_source_manifest.json, or *_raw_sections.jsonl, especially when a reader wants terms triaged, grounded in paper occurrences, and written as shared Obsidian notes. +description: Use when building reusable Obsidian glossary notes from an existing paper source manifest, optionally with a raw-sections override, especially when a reader needs a reviewed shortlist before glossary notes and article links are changed. --- # Paper Glossary -## Overview +Build shared glossary notes from `*_source_manifest.json`. `--source-manifest` is always required; `--raw-sections` only overrides its `raw_sections_path` with an explicit `*_raw_sections.jsonl`. This skill never runs or modifies a paper-reading workflow; the manifest and raw-sections file are its only paper-content boundary. See `references/file-contract.md` for JSON and CLI contracts. -Build a companion glossary for a paper without changing the paper-reading workflow. Use the shared source files already produced by another paper pipeline: `*_source_manifest.json` with `raw_sections_path`, or an explicit `*_raw_sections.jsonl`. +## Preview And Wait (Default) -The scripts are deterministic support tools. They find candidate terms and paper anchors, but the model writes the actual concept explanations. +1. **Show saved configuration** on first use per person/device. If absent, ask for a term directory inside an Obsidian vault and configure it; reuse valid configuration later. **Setup may create `~/.paper-glossary/config.json`**. +2. **Require an explicit article Markdown path** when article links are requested. Validate it and the configured term directory are in the **same Obsidian vault**; never infer an article path. +3. **Run deterministic proposal** from the effective body. +4. **Perform exactly one grounded host semantic review**, bounded by `shortlist_limit`. It may **only drop or reorder candidates** and must preserve each retained proposal candidate's **exact `term` string**. +5. **Record the reviewed shortlist** with `plan_glossary.py --review-proposal PROPOSAL --reviewed-terms NAMES`; this validates the saved proposal against the current paper source and preserves each full candidate, ordered `surface_forms`, and provenance. + If review produces an **empty reviewed shortlist**, report `no_candidates` and stop without presenting a selector. +6. **Present every retained term** from that reviewed artifact as a numbered Markdown list. This is the terminal, Codex, and Claude Code interaction. Show the resolved term directory, the article Markdown path (or that none was requested), and that selection authorizes glossary writes/enrichment plus first-safe-occurrence article links when an article was supplied. +7. State: **No glossary notes or article Markdown have been written before selection.** **End the response and wait.** Do not triage, inventory, generate, link, or lint during preview. -## Boundary +A broad request, manifest, raw sections path, or article path is not selection approval. A host-native selector is allowed only when it displays the complete same list in one interaction. -- Do not run or modify a paper-reading pipeline from this skill. -- Do not call any main paper workflow script from this skill. -- Treat `*_source_manifest.json` and `*_raw_sections.jsonl` as the only collaboration contract. -- Keep glossary outputs separate from the paper note unless the user explicitly asks to add links. -- Keep world-knowledge explanations thin, labeled, and confidence-rated; keep paper-specific facts in the occurrence zone. +## After Selection -## Workflow +**Accept only numbers**, exact term names, or `全部写入` from the immediately preceding numbered list. Resolve them to the exact displayed `term` strings before invoking a script; `全部写入` applies only to that list. Invalid selections receive the valid range and another wait. Triage requires both `--reviewed-shortlist REVIEW` and the resolved exact names in `--terms`; never use `--terms` alone or add alias syntax. -1. Locate the source manifest for the paper. Prefer a file named like `paper_source_manifest.json`; it must contain `raw_sections_path`, or pass `--raw-sections` explicitly. -2. Propose candidate terms: +Pass the current `--source-manifest`, optional `--raw-sections`, and saved `--reviewed-shortlist` to inventory, writer, and linker. Inventory consumes the saved `--triage` artifact as its selected-term input; writer and linker must also receive that same saved `--triage`. They require exact ordered `term` and `surface_forms` equality with this independent authorization before any glossary or article write. - ```bash - python skills/paper-glossary/scripts/plan_glossary.py --propose --source-manifest paper_source_manifest.json --output glossary_candidates.json - ``` +1. **Triage** the selection; its selection identity binds each exact selected name and ordered paper-grounded forms. +2. Run library **inventory** from that triage artifact; it recomputes the complete proposal/review/selection provenance chain and fails closed on mismatched paper, source, shortlist, or forms. +3. Then **generate one action-aware batch**. +4. **Run one writer invocation** against the **configured glossary directory**; it revalidates the same provenance chain and requires its ordered inventory results to match the authorized triage exactly, then performs **whole-batch preflight** followed by the **create/enrich/reuse commit**. The writer resolves device-local configuration (or an explicit `--config-path` for that device) and does not accept a standalone write destination. With `--article`, its backlink is **derived from the resolved article Markdown stem**. Without an article, the glossary-only backlink comes from the **validated manifest `paper_id`**. The successful artifact preserves `triage_sha256`, provenance, article context, ordered mappings, and their deterministic `mappings_sha256`; the mapping digest binds the triage identity. +5. **Link each successful writer result** at its first safe occurrence **only if an article Markdown was supplied/requested**. The linker authenticates the writer provenance, context, digest, note paths, stems, and forms against the same current source/review, and rejects an artifact with **no bound article path**. **For a glossary-only request, skip `link_glossary_terms.py`**. +6. **Lint writer-returned changed glossary note files** whose **`action` is `created`, `enriched`, or `updated`** in **one `lint_glossary.py` invocation with repeated `--input PATH`** arguments. **Do not pass article Markdown to `lint_glossary.py`**. +7. **Report observable wall-clock timing** and statuses. **Time host-only phases separately**. **Each CLI emits top-level `elapsed_ms`** for its own complete invocation. The **single writer invocation includes whole-batch preflight and commit**. **Do not report separate preflight and commit timings**. -3. Ask the reader which terms to keep, or accept a supplied list. -4. Triage selected terms against the paper: +`new` entries are created; `existing_thin` entries receive only missing structured fields and a missing occurrence; `existing_complete` entries may receive only a missing occurrence. **Do not overwrite** existing note content. See `references/file-contract.md` for `existing_thin`, entry operations, writer `forms`, and link statuses. - ```bash - python skills/paper-glossary/scripts/plan_glossary.py --terms '["MoE", "knowledge distillation|KD"]' --source-manifest paper_source_manifest.json --output glossary_plan.json - ``` +## Grounding -5. Have the model write a glossary JSON with one entry per term. Follow `references/file-contract.md` for the schema. -6. Write central term notes: - - ```bash - python skills/paper-glossary/scripts/write_glossary_terms.py --glossary glossary_entries.json --terms-dir ./术语 --paper-link PaperNoteStem --output glossary_write.json - ``` - -7. Lint the glossary note folder before reporting completion: - - ```bash - python skills/paper-glossary/scripts/lint_glossary.py --terms-dir ./术语 - ``` - -## Model Responsibilities - -When writing `glossary_entries.json`, use the triage output this way: - -- `routing: "anchor_only"`: explain how this paper uses the term and keep the general definition short. -- `routing: "needs_explanation"`: write a textbook-level explanation with a visible warning that it is background knowledge, not the paper's own claim. -- Always include `name`, `aliases`, `definition`, `confidence`, and `occurrence`. -- Use `confidence` as one of `高`, `中`, or `低`. - -## Resources - -- `scripts/plan_glossary.py`: propose terms and triage selected terms from raw sections. -- `scripts/write_glossary_terms.py`: create or update one shared Markdown note per term. -- `scripts/lint_glossary.py`: validate term-note structure. -- `references/file-contract.md`: input and output schemas. - -## Common Mistakes - -- Do not count references-section-only occurrences as evidence that the paper explains a term. -- Do not let an absent term look grounded; mark it as `needs_explanation`. -- Do not duplicate existing term notes when an alias already exists. -- Do not remove the warning, definition, confidence tier, or paper occurrence section from term notes. +- `anchor_only`: concise paper use plus a thin general explanation. +- `needs_explanation`: labeled background explanation with confidence and paper occurrence. +- Exclude reference-only occurrences as evidence. Keep paper facts in `occurrence` and outside knowledge labeled. diff --git a/skills/paper-glossary/agents/openai.yaml b/skills/paper-glossary/agents/openai.yaml index 75bb23e..bc46ba2 100644 --- a/skills/paper-glossary/agents/openai.yaml +++ b/skills/paper-glossary/agents/openai.yaml @@ -1,4 +1,4 @@ interface: display_name: "Paper Glossary" short_description: "Build reusable glossary notes from paper raw sections." - default_prompt: "Use $paper-glossary to create glossary term notes from this paper manifest and raw sections." + default_prompt: "Use $paper-glossary in preview-and-wait mode: propose candidates from this paper manifest and raw sections, perform semantic review, record the reviewed artifact, present its complete numbered shortlist, then stop and wait for my explicit selection before writing glossary notes or article links; triage starts only after selection." diff --git a/skills/paper-glossary/references/file-contract.md b/skills/paper-glossary/references/file-contract.md index 12184b9..c13305f 100644 --- a/skills/paper-glossary/references/file-contract.md +++ b/skills/paper-glossary/references/file-contract.md @@ -1,64 +1,178 @@ # Paper Glossary File Contract -## Inputs +## Boundary And Configuration -`plan_glossary.py` accepts: +`--source-manifest` is always required as a JSON object or path. `--raw-sections` only overrides the manifest's `raw_sections_path`; it cannot replace the manifest. Together, `*_source_manifest.json` and the resolved `*_raw_sections.jsonl` are the complete shared-file collaboration boundary. `plan_glossary.py` reads records whose `record_type` is absent or `section`; `kind: references` is excluded from evidence. No DeepPaperNote import or workflow call is part of this contract. -- `--source-manifest`: path to a JSON object, or an inline JSON object. -- `--raw-sections`: optional explicit JSONL path. If omitted, the manifest must contain `raw_sections_path`. -- `--terms`: selected terms as a JSON array, file path, comma-separated text, newline-separated text, or aliases separated by `|`. +`configure_glossary.py --terms-dir PATH` writes device-local `~/.paper-glossary/config.json`: -Raw section records are JSONL objects. Records with `record_type` absent or equal to `section` are read. `kind: references` is excluded from evidence matching. +```json +{"vault_root":"/vault","terms_subdir":"Glossary"} +``` + +`vault_root` must be absolute; relative and Windows drive-relative values are invalid. Loading returns the resolved absolute vault root so write authority does not depend on the process working directory. + +`--show` returns that configuration plus resolved `terms_dir`, or `{"workflow_state":"needs_configuration"}`. `--reset` returns `{"workflow_state":"reset"}`. `--validate-article PATH` requires an existing `.md` file in the configured vault and returns: + +```json +{"terms_dir":"/vault/Glossary","article_path":"/vault/Papers/example.md","article_sha256":"sha256"} +``` + +## Proposal, Review, And Triage + +`plan_glossary.py --propose` emits a deterministic candidate pool for one host semantic review. Each candidate contains ordered paper-grounded `surface_forms`. `provenance.source_sha256` binds the current `paper_id`, manifest, and source records; `provenance.proposal_sha256` also binds the exact candidate payload and summary. The host may **only drop or reorder candidates**, must preserve each retained candidate's **exact `term` string** and full object, and must display no more than `summary.shortlist_limit` terms before waiting for selection. + +```json +{ + "status": "ok", + "script": "plan_glossary.py", + "mode": "propose", + "workflow_state": "awaiting_semantic_review", + "next_action": "record_reviewed_shortlist_then_present_and_wait", + "paper_id": "paper-id", + "candidates": [{"term": "MoE", "surface_forms": ["MoE", "ＭｏＥ"], "category": "acronym-or-model", "occurrences": 2, "section_id": "sec:method", "page_start": 3, "snippet": "..."}], + "summary": {"effective_body_characters": 9000, "shortlist_limit": 10, "pool_candidates": 1}, + "provenance": { + "paper_id": "paper-id", + "source_sha256": "sha256", + "candidates": [{"term": "MoE", "surface_forms": ["MoE", "ＭｏＥ"], "category": "acronym-or-model", "occurrences": 2, "section_id": "sec:method", "page_start": 3, "snippet": "..."}], + "summary": {"effective_body_characters": 9000, "shortlist_limit": 10, "pool_candidates": 1}, + "proposal_sha256": "sha256" + } +} +``` + +The dynamic `shortlist_limit` is 10 below 10,000 effective body characters, 18 below 30,000, 25 below 60,000, and 35 otherwise. The pool can contain up to ten review alternatives beyond that limit. With no proposal candidates, or when semantic review records an **empty reviewed shortlist**, the state is `no_candidates` and `next_action` is `"report_no_candidates"`; the host reports that terminal state without displaying an empty selector. + +Record the semantic review before display: + +```text +plan_glossary.py --review-proposal proposal.json --reviewed-terms '["MoE"]' --source-manifest paper_source_manifest.json --output reviewed.json +``` + +The reviewed artifact contains `mode: "review"`, `reviewed_shortlist` with the selected full candidate objects unchanged, `proposal_provenance`, and provenance that repeats the exact proposal paper/source/candidate/summary fields, adds the full ordered `reviewed_shortlist`, and binds them with `"review_sha256"`. Review recording rejects unknown or duplicate names, alias syntax, more than `shortlist_limit`, a changed source, and any proposal payload or identity that does not match the current source. + +After the complete reviewed list is displayed, resolve shortlist numbers and `全部写入` to exact names outside the CLI. Triage requires the reviewed artifact; the old `--terms`-only path fails: + +```text +plan_glossary.py --reviewed-shortlist reviewed.json --terms '["MoE"]' --source-manifest paper_source_manifest.json --output triage.json +``` + +Every selected name must exactly match `reviewed_shortlist`. Triage revalidates the review against the current source and proposal, copies candidate `surface_forms` unchanged, uses only those forms for occurrences and anchors, and returns proposal, review, and selection provenance. `selection_sha256` binds the review identity plus each selected exact `term` and ordered `surface_forms` list. + +## Inventory And Entries + +Pass triage to `inspect_glossary_library.py` with the current `--source-manifest`, optional `--raw-sections`, and saved `--reviewed-shortlist`: + +```text +inspect_glossary_library.py --terms triage.json --terms-dir /vault/Glossary --source-manifest paper_source_manifest.json --raw-sections paper_raw_sections.jsonl --reviewed-shortlist reviewed.json --output inventory.json +``` + +The CLI recomputes the current deterministic proposal and validates the saved review before comparing the proposal, review, and selection digests from the triage artifact. This saved `--triage` is the independent authorization artifact passed unchanged to writer and linker. Both require exact ordered `term` and `surface_forms` equality with it. Inventory requires matching top-level/proposal/review paper and source fields and preserves that provenance plus each exact ordered `surface_forms` list. Its output includes the validated top-level `paper_id`. Missing, empty, non-string, whitespace-altered, duplicate, term-omitting, shortlist-mismatched, or identity-mismatched form data fails instead of being repaired. Every result contains `term`, `surface_forms`, `state`, `file`, `link_stem`, and `missing_fields`. State is `"new"`, `"existing_thin"`, or `"existing_complete"`. + +The one generated glossary batch has an `entries` array. Every entry has `name`, `aliases`, `operation`, and `occurrence`; `operation` is one of `create`, `enrich`, or `reuse`. + +```json +{ + "entries": [{ + "name": "MoE", + "aliases": ["Mixture of Experts"], + "operation": "enrich", + "elaboration": "Only supplied when inventory marks it missing.", + "occurrence": "Paper-specific evidence." + }] +} +``` + +`create` requires a definition and valid confidence. `enrich` may add only inventory `missing_fields`. `reuse` cannot replace concept fields. Every operation requires an occurrence. The writer resolves the **configured glossary directory** from device-local configuration (or `--config-path`), requires it to remain a valid Obsidian-vault directory, and has no standalone destination override. It requires the inventory's full provenance, reruns the same chain validation against the ordered inventory result terms/forms, then rechecks current library state and preflights the complete batch before changing a note. + +## Writer And Linker Results + +The article-bound writer invocation is: + +```text +write_glossary_terms.py --glossary entries.json --inventory inventory.json --config-path config.json --source-manifest paper_source_manifest.json --raw-sections paper_raw_sections.jsonl --reviewed-shortlist reviewed.json --triage triage.json --article /vault/Papers/example.md --output writer.json +``` + +For a glossary-only invocation, omit `--article`: -## Triage Output +```text +write_glossary_terms.py --glossary entries.json --inventory inventory.json --config-path config.json --source-manifest paper_source_manifest.json --raw-sections paper_raw_sections.jsonl --reviewed-shortlist reviewed.json --triage triage.json --output writer.json +``` -`plan_glossary.py --terms ...` writes: +Both modes revalidate the current source and saved review before loading generated entries. With an article, `paper_link` is **derived from the resolved article Markdown stem** and `article_path` is its resolved absolute path. In glossary-only mode, `paper_link` is the wiki-link-safe **validated manifest `paper_id`** and `article_path` is empty. `write_glossary_terms.py` returns successful results with the actual note target and only the exact reviewed/inventory `forms`: ```json { "status": "ok", - "mode": "triage", + "script": "write_glossary_terms.py", "paper_id": "paper-id", - "terms": [ - { - "term": "MoE", - "surface_forms": ["MoE"], - "found_in_paper": true, - "occurrences": 2, - "routing": "anchor_only", - "paper_anchors": [ - { - "section_id": "sec:method", - "title": "Method", - "page_start": 3, - "page_end": 5, - "snippet": "..." - } - ] - } - ] + "triage_sha256": "sha256", + "provenance": { + "proposal": { + "paper_id": "paper-id", + "source_sha256": "sha256", + "candidates": [{"term": "MoE", "surface_forms": ["MoE", "ＭｏＥ"]}], + "summary": {"shortlist_limit": 10}, + "proposal_sha256": "sha256" + }, + "review": { + "paper_id": "paper-id", + "source_sha256": "sha256", + "candidates": [{"term": "MoE", "surface_forms": ["MoE", "ＭｏＥ"]}], + "summary": {"shortlist_limit": 10}, + "proposal_sha256": "sha256", + "reviewed_shortlist": [{"term": "MoE", "surface_forms": ["MoE", "ＭｏＥ"]}], + "review_sha256": "sha256" + }, + "selection_sha256": "sha256" + }, + "context": {"paper_id": "paper-id", "paper_link": "example", "article_path": "/vault/Papers/example.md"}, + "mappings_sha256": "sha256", + "results": [{ + "name": "MoE", + "forms": ["MoE", "ＭｏＥ"], + "file": "/vault/Glossary/MoE.md", + "action": "enriched", + "link_stem": "MoE", + "fields_added": ["elaboration"], + "occurrence_added": true + }] +} +``` + +`action` is `created`, `enriched`, `updated`, or `unchanged`. Entry `aliases` may be written to note metadata and participate in logical collision checks, but they are never added to writer `forms`. Generated names and aliases are checked against the full library's NFKC/casefold alias index before any write; an existing form is allowed only for the same validated target. Grounded inventory forms need not be repeated in entry `aliases`. Existing notes are never replaced, only missing fields and an absent occurrence may be added. `triage_sha256` deterministically identifies the authorized triage's paper, provenance, and terms. `mappings_sha256` binds that triage identity, validated provenance, derived context, and exact ordered `name`, `forms`, `file`, and `link_stem` mappings. + +Run the linker **only if an article Markdown was supplied/requested**: + +```text +link_glossary_terms.py --input /vault/Papers/example.md --write-result writer.json --expected-sha256 sha256 --config-path config.json --source-manifest paper_source_manifest.json --raw-sections paper_raw_sections.jsonl --reviewed-shortlist reviewed.json --triage triage.json --output linker.json +``` + +The linker revalidates the successful writer provenance against the current source, saved review, and same saved triage; requires its mappings to equal the triage's exact authorized selection; verifies `triage_sha256`; recomputes `mappings_sha256`; and checks context, order, files, stems, and full-library form resolution before touching the article. It rejects a writer artifact with **no bound article path**. Supply the preview's `article_sha256` as `--expected-sha256`. **For a glossary-only request, skip `link_glossary_terms.py`**. When run, it returns the validated `article_path`, per-note `results`, and counts: + +```json +{ + "status": "ok", + "script": "link_glossary_terms.py", + "article_path": "/vault/Papers/example.md", + "results": [{"link_stem": "MoE", "status": "already_linked"}], + "summary": {"linked": 0, "already_linked": 1, "not_found": 0} } ``` -## Glossary Entry Input +Link statuses are `linked`, `already_linked`, or `not_found`. A stale article hash stops linking before article modification. + +## Lint And Timing Boundaries + +Lint only **writer-returned changed glossary note files** whose **`action` is `created`, `enriched`, or `updated`**, using **one `lint_glossary.py` invocation with repeated `--input PATH`** arguments. Explicit inputs are resolved and deduplicated, and may be combined with `--terms-dir` for an intentional full-library audit. **Do not pass article Markdown to `lint_glossary.py`**; it validates the term-note structure, not ordinary articles. -`write_glossary_terms.py` expects: +**Time host-only phases separately**. **Each CLI emits top-level `elapsed_ms`** as a non-negative integer for its own complete invocation: ```json { - "entries": [ - { - "name": "KL 散度", - "aliases": ["KL divergence", "相对熵"], - "definition": "衡量两个概率分布差异的非对称度量。", - "elaboration": "常用于把学生分布拉近教师分布。", - "intuition": "把 Q 当作近似 P 时的信息损失。", - "distinction": "不同于对称距离。", - "confidence": "高", - "occurrence": "方法 式(4)，第 3-6 页" - } - ] + "elapsed_ms": 12 } ``` -Required fields for lint: `name`, `definition`, `confidence`, and one occurrence line linked to a paper note. +The **single writer invocation includes whole-batch preflight and commit**. **Do not report separate preflight and commit timings**. Host semantic review and model generation remain separately timed host-only phases and are not fabricated by Python. diff --git a/skills/paper-glossary/references/optimization-design.md b/skills/paper-glossary/references/optimization-design.md new file mode 100644 index 0000000..b43c809 --- /dev/null +++ b/skills/paper-glossary/references/optimization-design.md @@ -0,0 +1,191 @@ +# Paper Glossary Optimization Design + +Date: 2026-07-14 + +Status: approved for implementation + +## Goals + +- Keep `paper-glossary` independent from the DeepPaperNote workflow. +- Use only `*_source_manifest.json` and `*_raw_sections.jsonl` as the paper-content contract. +- Let each user configure one Obsidian term-library directory on first use per device. +- Present a high-quality, length-aware candidate list before any write. +- After selection, write or enrich term notes and add safe outbound links to the explicit article Markdown. +- Reduce model latency without making term notes too thin. + +## Non-goals + +- Do not modify `skills/deeppapernote/` or call its workflow. +- Do not change plugin manifests. +- Do not add a vector database or third-party dependency. +- Do not guess which Markdown article to modify. +- Do not replace or rewrite user-authored term-note content. + +## Inputs and Boundaries + +Term evidence comes only from a source manifest and its `raw_sections_path`, or from an explicit raw-sections JSONL file. The Markdown article path is a separate, explicit write target. It is never inferred from a PDF title, `full_text_md_path`, or a same-named file search. + +If the user provides only a PDF, manifest, or raw-sections path, preview may run, but article linking must wait until the user supplies the Markdown note path. + +## First-use Configuration + +On the first write-capable run, ask the user to choose a term-library directory inside an Obsidian vault. Find the nearest ancestor containing `.obsidian` and reject a directory without one. + +Store device-local configuration in `~/.paper-glossary/config.json`: + +```json +{ + "vault_root": "F:\\我的笔记\\123", + "terms_subdir": "book\\术语" +} +``` + +The config is not written into the plugin repository or Obsidian vault. Later runs reuse it without asking, and the preview still displays the resolved term-library path. Writer destination authority comes only from validated device config. `--config-path` selects a config file, not a direct destination. To use a different location, reset and configure the device again. + +Before any write: + +- verify that `vault_root/.obsidian` exists; +- resolve paths before checking containment; +- verify that the term directory remains inside the configured vault; +- verify that the target article Markdown is inside the same vault. + +If any check fails, stop before changing files and request configuration again. + +## Candidate Workflow + +### Deterministic prefilter + +Build a grounded candidate pool from prose, headings, emphasis, acronym expansions, named methods, models, datasets, tools, and important components. Exclude: + +- fenced and inline code; +- URLs and existing links; +- references-only records or reference sections; +- environment-variable fragments and code-identifier fragments; +- example-only brands or entities without reusable conceptual value; +- case-only duplicates and known aliases. + +Prefer longer and more specific forms before short forms. Keep evidence anchors and surface forms for every candidate. + +### Length-aware limit + +Measure effective body characters after excluded material is removed. The final shortlist limit is: + +| Effective body length | Maximum candidates | +| --- | ---: | +| `< 10,000` | 10 | +| `10,000-29,999` | 18 | +| `30,000-59,999` | 25 | +| `>= 60,000` | 35 | + +These are upper limits, not quotas. Never add noise to fill a tier. The deterministic pool may contain up to ten more items than the final limit so the semantic review has alternatives. + +### Semantic review + +Use one compact host-model review to rank the grounded pool, remove residual noise, and retain core concepts that the deterministic ranking would otherwise underrank. The review may only drop or reorder exact proposal candidates: every retained candidate preserves its exact `term` and ordered `surface_forms`. It may not merge aliases, author forms, or introduce a term without paper evidence in the pool. + +Present the complete final shortlist as a numbered Markdown list. Also display: + +- the resolved term-library directory; +- the article Markdown path, when available; +- a clear statement that selection will write term notes and add one safe article outlink per selected term. + +Then stop and wait. A broad request or paper path alone is not write approval. + +## Selection and Generation + +Accept numbers, names, or `全部写入` only from the immediately preceding shortlist. Resolve control input to canonical names before invoking triage. + +Classify selected terms as: + +- `new`: no matching name or alias in the central library; +- `existing_complete`: an existing note has a definition, confidence, and at least two substantive explanatory fields; +- `existing_thin`: an existing note is missing required fields or has fewer than two substantive explanatory fields. + +Use one batch generation step for all `new` entries and all missing fields in `existing_thin` entries: + +- `anchor_only`: concise definition, paper usage/occurrence, confidence, and only necessary supplementary context; +- `needs_explanation`: definition, elaboration, intuition, distinction, occurrence, and confidence. + +For every existing note, append the new paper occurrence when absent. For `existing_thin`, add only missing fields. Never remove, replace, reorder, or rephrase existing user content. `existing_complete` receives the occurrence only. + +Validate the entire generated batch before writing any entry. One structured-output repair attempt is allowed. If the repaired payload is still invalid, stop without writing generated content. + +## Article Outlinks + +Selection authorizes both glossary writes and article linking because the preview states both effects in advance. + +Use the actual `link_stem` returned by the glossary writer. In the explicit article Markdown, link the first safe prose occurrence: + +```md +[[ReAct]] +[[ReAct|REACT]] +``` + +Skip matches inside: + +- YAML frontmatter; +- headings; +- fenced or inline code; +- URLs, Markdown links, HTML, and existing wiki links; +- reference sections; +- a larger identifier or word. + +Match longer terms before shorter ones. Preserve the original surface text through an alias link. Add at most one outlink per term. A repeated run must not add another link. + +Record the article hash at preview. Recheck it immediately before link application. If it changed, stop article writing and report the conflict. Build the edited text in memory, write a temporary sibling file, and atomically replace the article only after validation. If no safe occurrence exists, keep the valid glossary result and report the missing outlink without forcing a fallback insertion. + +## Performance + +- Use at most one semantic-review phase before selection. +- Use one batch generation phase after selection; never call the model once per term. +- Generate content only for new entries and missing fields in thin entries. +- Lint only files changed by the current run; keep a separate full-library audit command. +- Report elapsed time for configuration, extraction, semantic review, generation, glossary write, article linking, and lint. +- Treat under-two-second local script work on the chapter-four regression fixture as an observed, environment-dependent target. Repeated benchmark runs report a median and range, or exact overruns. Overall model time is a visible target of roughly one to two minutes, not a platform-independent guarantee. + +## Result and Failure Reporting + +The final result reports separate counts and paths for: + +- created; +- enriched; +- reused; +- occurrences appended; +- article outlinks added; +- selected terms without a safe outlink; +- failures. + +Configuration errors, cross-vault paths, invalid selections, stale article hashes, and invalid generated payloads must fail before the corresponding write. A failed enrichment preserves the original note. A failed article link does not corrupt or delete a valid term note. + +## Implementation Surface + +Keep changes inside `skills/paper-glossary/`: + +- update `SKILL.md` for first-use setup, hybrid preview, selection effects, and adaptive generation; +- add device-local config resolution under `scripts/`; +- improve `plan_glossary.py` prefiltering, alias normalization, evidence, and dynamic limits; +- extend `write_glossary_terms.py` with preflight validation and non-destructive thin-note enrichment; +- add a focused article-linking script with hash and atomic-write protection; +- update `references/file-contract.md` and `README.md`; +- add focused unit and integration tests. + +## Verification + +Tests must cover: + +- first-use configuration, reuse, reset, deleted paths, non-vault paths, and cross-vault rejection; +- all four candidate-size tiers without quota filling; +- code, URL, references, environment-variable, example-entity, case, and alias filtering; +- preservation of core chapter concepts such as `Reflection`; +- concise and full entry schemas; +- new, complete, and thin existing-note behavior; +- preservation of frontmatter, custom sections, comments, and user text; +- idempotent occurrence enrichment; +- direct and alias outlinks, protected Markdown regions, longer-term priority, stale hashes, and idempotency; +- the complete preview, selection, write, link, and lint flow using repository fixtures. + +Run the focused paper-glossary tests and the repository's normal full test suite. No commit or push occurs until implementation and verification are complete and the user explicitly approves it. + +## Current Demo Cleanup + +The earlier demo wrote notes to `F:\我的笔记\123\学习\hello_agent\术语`. Do not migrate those notes blindly. After implementation passes tests, rerun preview for the chapter-four article with the improved filter, let the user select from the corrected shortlist, merge approved content into the configured central library, add article outlinks, and only then remove assistant-created demo files after confirming they contain no later user edits. diff --git a/skills/paper-glossary/references/optimization-implementation-plan.md b/skills/paper-glossary/references/optimization-implementation-plan.md new file mode 100644 index 0000000..ec207f8 --- /dev/null +++ b/skills/paper-glossary/references/optimization-implementation-plan.md @@ -0,0 +1,1019 @@ +# Paper Glossary Optimization Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Build a fast, first-use-configured paper-glossary workflow that proposes length-aware terms, non-destructively enriches a shared Obsidian term library, and adds safe first-occurrence links to the explicit article Markdown. + +**Architecture:** Keep paper evidence intake limited to `*_source_manifest.json` and `*_raw_sections.jsonl`. Add a device-local configuration layer, deterministic candidate prefilter followed by one host semantic review, deterministic library inventory, one batched model payload for new/thin entries, and a separate atomic Markdown linker. DeepPaperNote remains untouched. + +**Tech Stack:** Python 3.10+, standard library only, Markdown/JSON/JSONL, pytest 8+, existing repository scripts and contracts. + +## Global Constraints + +- Modify only `skills/paper-glossary/`; do not modify `skills/deeppapernote/` or root plugin manifests. +- Keep `*_source_manifest.json` and `*_raw_sections.jsonl` as the only paper-content collaboration contract. +- Do not add dependencies. +- Do not guess an article Markdown path from a title, PDF, or `full_text_md_path`. +- Require the configured term directory and article Markdown to belong to the same nearest Obsidian vault. +- Preserve all user-authored term-note content; enrichment may only add missing structured fields and a missing occurrence. +- Use TDD: run every named failing test before implementation and rerun it after the minimal change. +- Preserve the current uncommitted selection-gate work and build on it rather than reverting it. +- Do not commit or push during implementation. Each task ends with a diff/test checkpoint; commit only after the user explicitly approves the completed implementation. +- Before editing skill behavior, invoke `superpowers:writing-skills`; before each behavior change, invoke `superpowers:test-driven-development`. + +## File Map + +| File | Responsibility | +| --- | --- | +| `scripts/glossary_config.py` | Device-local config, nearest-vault validation, same-vault article validation, article hash | +| `scripts/configure_glossary.py` | Configure/show/reset/validate CLI for first-use interaction | +| `scripts/plan_glossary.py` | Effective-prose cleanup, dynamic limits, grounded candidate pool, selected-term triage | +| `scripts/glossary_library.py` | Alias index, existing-note inspection, completeness classification, additive field insertion | +| `scripts/inspect_glossary_library.py` | CLI inventory of selected terms before model generation | +| `scripts/write_glossary_terms.py` | Preflight operation validation and create/enrich/reuse writes | +| `scripts/link_glossary_terms.py` | Protected-region-aware, hash-guarded, atomic article linking | +| `scripts/lint_glossary.py` | Lint one or more changed notes without scanning the full library | +| `scripts/glossary_common.py` | Shared JSON emission and elapsed-time helper | +| `SKILL.md` and `agents/openai.yaml` | Cross-host first-use, preview, selection, write, and link instructions | +| `README.md` and `references/file-contract.md` | User setup and artifact contracts | +| `tests/` | Focused unit, contract, and end-to-end coverage | + +## Spec Coverage + +| Design section | Implemented by | +| --- | --- | +| Goals, non-goals, inputs, and boundaries | Global constraints; Tasks 1 and 5 | +| First-use configuration | Task 1 | +| Candidate workflow and dynamic limits | Task 2 | +| Selection, adaptive generation, and old-note enrichment | Tasks 3 and 5 | +| Article outlinks and write safety | Task 4 | +| Performance and stage timing | Task 6 | +| Result and failure reporting | Tasks 1, 3, 4, and 6 | +| Implementation surface and verification | Tasks 1-6 | +| Current demo cleanup and real acceptance | Task 7 | + +--- + +### Task 1: Device-local Obsidian configuration + +**Files:** +- Create: `skills/paper-glossary/scripts/glossary_config.py` +- Create: `skills/paper-glossary/scripts/configure_glossary.py` +- Create: `skills/paper-glossary/tests/test_glossary_config.py` + +**Interfaces:** +- Produces: `find_vault_root(path: Path) -> Path | None` +- Produces: `configure_terms_dir(terms_dir: Path, config_path: Path | None = None) -> dict[str, str]` +- Produces: `load_config(config_path: Path | None = None) -> dict[str, str] | None` +- Produces: `resolve_terms_dir(config: dict[str, str]) -> Path` +- Produces: `validate_article(article: Path, config: dict[str, str]) -> dict[str, str]` +- Produces CLI modes: `--terms-dir`, `--show`, `--reset`, `--validate-article`, plus `--config-path` and `--output` + +- [ ] **Step 1: Write failing configuration and vault-boundary tests** + +Add tests that exercise an existing term folder, a not-yet-created child folder, nearest nested vault selection, a non-vault path, same-vault article validation, cross-vault rejection, config reuse, reset, and missing configured paths. Core tests must include: + +```python +from __future__ import annotations + +import json +from pathlib import Path + +import pytest + +from glossary_config import ( + configure_terms_dir, + load_config, + resolve_terms_dir, + validate_article, +) + + +def test_configure_terms_dir_persists_nearest_vault_and_relative_subdir( + tmp_path: Path, +) -> None: + outer = tmp_path / "outer" + inner = outer / "research" + (outer / ".obsidian").mkdir(parents=True) + (inner / ".obsidian").mkdir() + terms_dir = inner / "book" / "术语" + config_path = tmp_path / "device" / "config.json" + + payload = configure_terms_dir(terms_dir, config_path) + + assert payload == { + "vault_root": str(inner.resolve()), + "terms_subdir": str(Path("book") / "术语"), + } + assert terms_dir.is_dir() + assert load_config(config_path) == payload + assert resolve_terms_dir(payload) == terms_dir.resolve() + + +def test_validate_article_rejects_other_vault(tmp_path: Path) -> None: + first = tmp_path / "first" + second = tmp_path / "second" + (first / ".obsidian").mkdir(parents=True) + (second / ".obsidian").mkdir(parents=True) + article = second / "Paper.md" + article.write_text("# Paper\n", encoding="utf-8") + config = configure_terms_dir(first / "术语", tmp_path / "config.json") + + with pytest.raises(SystemExit, match="same Obsidian vault"): + validate_article(article, config) + + +def test_load_config_returns_none_before_first_use(tmp_path: Path) -> None: + assert load_config(tmp_path / "missing.json") is None +``` + +- [ ] **Step 2: Run the focused test and verify RED** + +Run: + +```powershell +py -3.12 -m pytest -q skills/paper-glossary/tests/test_glossary_config.py --basetemp .pytest-tmp-paper-glossary-config +``` + +Expected: collection fails because `glossary_config` does not exist. + +- [ ] **Step 3: Implement config resolution and validation** + +Use `Path.resolve(strict=False)` before containment checks and walk from the selected path or its nearest existing parent. The core implementation must follow this shape: + +```python +from __future__ import annotations + +import hashlib +import json +from pathlib import Path + +CONFIG_DIRNAME = ".paper-glossary" +CONFIG_FILENAME = "config.json" + + +def default_config_path() -> Path: + return Path.home() / CONFIG_DIRNAME / CONFIG_FILENAME + + +def find_vault_root(path: Path) -> Path | None: + current = path.expanduser().resolve(strict=False) + while not current.exists() and current != current.parent: + current = current.parent + if current.is_file(): + current = current.parent + for candidate in (current, *current.parents): + if (candidate / ".obsidian").is_dir(): + return candidate.resolve() + return None + + +def _is_within(path: Path, parent: Path) -> bool: + try: + path.resolve(strict=False).relative_to(parent.resolve()) + except ValueError: + return False + return True + + +def configure_terms_dir( + terms_dir: Path, config_path: Path | None = None +) -> dict[str, str]: + resolved = terms_dir.expanduser().resolve(strict=False) + vault = find_vault_root(resolved) + if vault is None or not _is_within(resolved, vault): + raise SystemExit("Term directory must be inside an Obsidian vault containing .obsidian.") + resolved.mkdir(parents=True, exist_ok=True) + payload = { + "vault_root": str(vault), + "terms_subdir": str(resolved.relative_to(vault)), + } + target = (config_path or default_config_path()).expanduser() + target.parent.mkdir(parents=True, exist_ok=True) + target.write_text(json.dumps(payload, ensure_ascii=False, indent=2) + "\n", encoding="utf-8") + return payload + + +def load_config(config_path: Path | None = None) -> dict[str, str] | None: + target = (config_path or default_config_path()).expanduser() + if not target.is_file(): + return None + payload = json.loads(target.read_text(encoding="utf-8-sig")) + if not isinstance(payload, dict): + raise SystemExit(f"Invalid paper-glossary config object: {target}") + return {"vault_root": str(payload.get("vault_root", "")), "terms_subdir": str(payload.get("terms_subdir", ""))} + + +def resolve_terms_dir(config: dict[str, str]) -> Path: + vault = Path(config.get("vault_root", "")).expanduser().resolve(strict=False) + terms_dir = (vault / config.get("terms_subdir", "")).resolve(strict=False) + if not (vault / ".obsidian").is_dir() or not terms_dir.is_dir() or not _is_within(terms_dir, vault): + raise SystemExit("Configured Obsidian term directory is missing or invalid; configure it again.") + return terms_dir + + +def validate_article(article: Path, config: dict[str, str]) -> dict[str, str]: + resolved = article.expanduser().resolve() + if resolved.suffix.lower() != ".md" or not resolved.is_file(): + raise SystemExit(f"Article Markdown not found: {resolved}") + article_vault = find_vault_root(resolved) + configured_vault = Path(config["vault_root"]).expanduser().resolve() + if article_vault != configured_vault: + raise SystemExit("Article and term directory must be inside the same Obsidian vault.") + digest = hashlib.sha256(resolved.read_bytes()).hexdigest() + return {"article_path": str(resolved), "article_sha256": digest} +``` + +Keep JSON formatting readable and use `SystemExit` messages that the host can relay directly. + +- [ ] **Step 4: Add the small configuration CLI** + +Implement mutually exclusive configure/show/reset/validate modes. `--show` must return `workflow_state: "needs_configuration"` when no config exists instead of failing. `--validate-article` must include the resolved `terms_dir`, `article_path`, and `article_sha256` in its JSON output. `--reset` removes only the config file and never removes a vault or term directory. + +- [ ] **Step 5: Run Task 1 tests and verify GREEN** + +Run the command from Step 2. Expected: all tests in `test_glossary_config.py` pass. + +- [ ] **Step 6: Task 1 checkpoint without commit** + +Run: + +```powershell +git diff --check -- skills/paper-glossary/scripts/glossary_config.py skills/paper-glossary/scripts/configure_glossary.py skills/paper-glossary/tests/test_glossary_config.py +``` + +Expected: exit code 0. Record `git status --short`; do not stage or commit. + +--- + +### Task 2: Length-aware deterministic candidate pool + +**Files:** +- Modify: `skills/paper-glossary/scripts/plan_glossary.py:14-390` +- Modify: `skills/paper-glossary/tests/test_plan_glossary.py` + +**Interfaces:** +- Produces: `effective_prose_text(records: list[dict[str, Any]]) -> str` +- Produces: `shortlist_limit(effective_characters: int) -> int` +- Changes: `propose_candidates(records)` returns a grounded pool capped at `shortlist_limit + 10` +- Changes proposal JSON: `workflow_state`, `next_action`, `summary.effective_body_characters`, `summary.shortlist_limit`, `summary.pool_candidates` +- Preserves: selected-term triage and unresolved-selection rejection + +- [ ] **Step 1: Replace the fixed-limit test with failing tier and noise tests** + +Add parameterized limit tests and a chapter-style regression: + +```python +@pytest.mark.parametrize( + ("characters", "expected"), + [(9_999, 10), (10_000, 18), (29_999, 18), (30_000, 25), (59_999, 25), (60_000, 35)], +) +def test_shortlist_limit_uses_effective_body_length(characters: int, expected: int) -> None: + assert shortlist_limit(characters) == expected + + +def test_propose_filters_code_noise_and_keeps_emphasized_core_terms() -> None: + records = [ + { + "record_type": "section", + "section_id": "sec:chapter", + "kind": "chapter", + "title": "智能体经典范式", + "text": ( + "**ReAct**、**Plan-and-Solve** 与 **Reflection** 是三种经典范式。\n" + "SerpApi 为智能体提供搜索能力。\n" + "```python\nLLM_API_KEY = 'KEY'\nREACT_PROMPT_TEMPLATE = 'MODEL'\n" + "print('HUAWEI phone result')\n```\n" + ), + }, + { + "record_type": "section", + "section_id": "sec:refs", + "kind": "references", + "title": "References", + "text": "SERPAPI TEMPLATE PLANNER", + }, + ] + + terms = {item["term"] for item in propose_candidates(records)} + + assert {"ReAct", "Plan-and-Solve", "Reflection", "SerpApi"} <= terms + assert terms.isdisjoint({"KEY", "MODEL", "TEMPLATE", "HUAWEI"}) + assert not ({"ReAct", "REACT"} <= terms) + + +def test_proposal_reports_semantic_review_and_dynamic_limit( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + payload = _run_proposal(tmp_path, monkeypatch, _records()) + assert payload["workflow_state"] == "awaiting_semantic_review" + assert payload["next_action"] == "review_candidate_pool_then_present_and_wait" + assert payload["summary"]["shortlist_limit"] == 10 + assert payload["summary"]["pool_candidates"] == len(payload["candidates"]) +``` + +Update imports to include `shortlist_limit`. Keep the existing no-candidate terminal-state test. + +- [ ] **Step 2: Run candidate tests and verify RED** + +Run: + +```powershell +py -3.12 -m pytest -q skills/paper-glossary/tests/test_plan_glossary.py --basetemp .pytest-tmp-paper-glossary-plan +``` + +Expected: failures for the missing `shortlist_limit`, code-noise candidates, missing `Reflection`, and old workflow-state fields. + +- [ ] **Step 3: Implement effective-prose cleanup and limits** + +Add standard-library regexes and helpers. Preserve link labels while removing destinations, remove reference tails inside monolithic Markdown records, and count non-whitespace characters: + +```python +FENCED_CODE_RE = re.compile(r"(?ms)^(?:`{3,}|~{3,})[^\n]*\n.*?^(?:`{3,}|~{3,})\s*$") +INLINE_CODE_RE = re.compile(r"`+[^`\n]+`+") +MARKDOWN_LINK_RE = re.compile(r"\[([^\]]+)\]\([^\)]+\)") +URL_RE = re.compile(r"https?://\S+") +REFERENCE_TAIL_RE = re.compile(r"(?ims)^#{1,6}\s*(?:references|参考文献|引用)\s*$.*\Z") +EMPHASIS_RE = re.compile(r"(?:\*\*|__)([^\n*_]{2,80})(?:\*\*|__)") +STRONG_RE = re.compile(r"(?is)([^<]{2,80})") + + +def _candidate_prose(text: str) -> str: + cleaned = FENCED_CODE_RE.sub("\n", text) + cleaned = INLINE_CODE_RE.sub(" ", cleaned) + cleaned = MARKDOWN_LINK_RE.sub(r"\1", cleaned) + cleaned = URL_RE.sub(" ", cleaned) + return REFERENCE_TAIL_RE.sub("", cleaned) + + +def effective_prose_text(records: list[dict[str, Any]]) -> str: + return "\n".join( + _candidate_prose(str(record.get("text", ""))) + for record in _evidence_records(records) + ) + + +def shortlist_limit(effective_characters: int) -> int: + if effective_characters < 10_000: + return 10 + if effective_characters < 30_000: + return 18 + if effective_characters < 60_000: + return 25 + return 35 +``` + +Use cleaned record copies only for candidate extraction. Continue using original evidence records for occurrences and snippets. + +- [ ] **Step 4: Add grounded emphasis extraction and casefold deduplication** + +Change `add()` to use `unicodedata.normalize("NFKC", term).casefold()` as the candidate key. Prefer the first prose form and never emit a second case-only form. Feed bold and `` terms through conservative length/shape checks so `Reflection` and `SerpApi` survive without admitting sentences. Cap the deterministic pool at `shortlist_limit + 10`; the host semantic pass may present fewer than the final limit and may not invent terms outside this pool. + +- [ ] **Step 5: Emit the semantic-review contract** + +For a nonempty pool emit: + +```python +{ + "workflow_state": "awaiting_semantic_review", + "next_action": "review_candidate_pool_then_present_and_wait", + "summary": { + "effective_body_characters": effective_characters, + "shortlist_limit": display_limit, + "pool_candidates": len(candidates), + }, +} +``` + +For an empty pool preserve `workflow_state: "no_candidates"` and `next_action: "report_no_candidates"`. + +- [ ] **Step 6: Run Task 2 tests and verify GREEN** + +Run the command from Step 2. Expected: all `test_plan_glossary.py` tests pass. + +- [ ] **Step 7: Task 2 checkpoint without commit** + +Run `git diff --check` on the two Task 2 files and record the diff. Do not stage or commit. + +--- + +### Task 3: Existing-library inventory and non-destructive enrichment + +**Files:** +- Create: `skills/paper-glossary/scripts/glossary_library.py` +- Create: `skills/paper-glossary/scripts/inspect_glossary_library.py` +- Create: `skills/paper-glossary/tests/test_glossary_library.py` +- Modify: `skills/paper-glossary/scripts/write_glossary_terms.py:64-231` +- Modify: `skills/paper-glossary/tests/test_write_and_lint_glossary.py` + +**Interfaces:** +- Produces: `inspect_selected_terms(selected: list[dict[str, Any]], terms_dir: Path) -> list[dict[str, Any]]` +- Inventory CLI requires `--source-manifest`, optional `--raw-sections`, and saved `--reviewed-shortlist`; it consumes the saved triage artifact as `--terms triage.json`. +- Produces inventory states: `new`, `existing_thin`, `existing_complete` +- Produces: `missing_concept_fields(text: str) -> list[str]` +- Produces: `add_missing_concept_fields(text: str, patch: dict[str, Any]) -> tuple[str, list[str]]` +- Produces: `write_glossary_entries(glossary: dict[str, Any], inventory: dict[str, Any], terms_dir: Path, paper_link: str) -> dict[str, Any]` +- Changes glossary-entry contract: each entry has `operation` equal to `create`, `enrich`, or `reuse` +- Changes writer CLI: require `--inventory` for the optimized workflow and validate the entire batch before writes +- Changes writer result: include `forms`, `fields_added`, `occurrence_added`, and exact `link_stem` + +- [ ] **Step 1: Write failing inventory and preservation tests** + +Tests must prove classification by aliases and additive behavior: + +```python +def test_inventory_classifies_new_thin_and_complete_notes(tmp_path: Path) -> None: + terms_dir = tmp_path / "术语" + terms_dir.mkdir() + (terms_dir / "LLM.md").write_text( + "---\naliases:\n - \"大语言模型\"\n---\n\n# LLM\n\n" + "## 概念解释\n- 定义：语言模型。\n- 置信度：中\n\n" + "## 在论文中的出现\n- [[OldPaper]]：旧出处\n", + encoding="utf-8", + ) + (terms_dir / "ReAct.md").write_text( + "# ReAct\n\n## 概念解释\n" + "- 定义：交替进行推理与行动的智能体范式。\n" + "- 详解：模型在工具观察后继续推理。\n" + "- 直觉：边想边做。\n" + "- 置信度：高\n\n" + "## 在论文中的出现\n- [[OldPaper]]：旧出处\n", + encoding="utf-8", + ) + selected = [ + {"term": "大语言模型", "surface_forms": ["LLM", "大语言模型"]}, + {"term": "ReAct", "surface_forms": ["ReAct"]}, + {"term": "Reflection", "surface_forms": ["Reflection"]}, + ] + + items = inspect_selected_terms(selected, terms_dir) + states = {item["term"]: item["state"] for item in items} + + assert states == { + "大语言模型": "existing_thin", + "ReAct": "existing_complete", + "Reflection": "new", + } + + +def test_enrichment_adds_only_missing_fields_and_preserves_custom_text() -> None: + original = ( + "# LLM\n\n用户自己的说明。\n\n## 概念解释\n" + "- 定义：语言模型。\n- 置信度：中\n\n## 自定义\n不要改动。\n" + ) + updated, fields = add_missing_concept_fields( + original, + {"elaboration": "用于智能体推理。", "intuition": "智能体的大脑。"}, + ) + + assert fields == ["elaboration", "intuition"] + assert "用户自己的说明。" in updated + assert "## 自定义\n不要改动。" in updated + assert "- 定义：语言模型。" in updated + assert updated.count("- 详解：用于智能体推理。") == 1 + assert updated.count("- 直觉：智能体的大脑。") == 1 +``` + +Add writer tests that reject an invalid operation before creating any file, enrich a thin note without replacing existing fields, append only a missing occurrence, and remain unchanged on a second identical run. + +- [ ] **Step 2: Run Task 3 tests and verify RED** + +Run: + +```powershell +py -3.12 -m pytest -q skills/paper-glossary/tests/test_glossary_library.py skills/paper-glossary/tests/test_write_and_lint_glossary.py --basetemp .pytest-tmp-paper-glossary-library +``` + +Expected: missing module/functions and old writer behavior fail. + +- [ ] **Step 3: Implement deterministic library inspection** + +Move frontmatter alias, heading-name, and alias-index parsing into `glossary_library.py`; re-import those names from `write_glossary_terms.py` so existing imports remain compatible. Define completeness as: + +```python +CONCEPT_FIELDS = { + "definition": "定义：", + "elaboration": "详解：", + "intuition": "直觉：", + "distinction": "与相邻概念的区别：", + "confidence": "置信度：", +} + + +def note_state(text: str) -> tuple[str, list[str]]: + missing = missing_concept_fields(text) + optional_present = sum( + field not in missing for field in ("elaboration", "intuition", "distinction") + ) + required_missing = any(field in missing for field in ("definition", "confidence")) + state = "existing_thin" if required_missing or optional_present < 2 else "existing_complete" + return state, missing +``` + +Inventory output must include `term`, `surface_forms`, `state`, `file`, `link_stem`, and `missing_fields`. A missing alias match is `new` with an empty file and link stem based on the canonical term. + +- [ ] **Step 4: Implement additive field insertion** + +Insert only labels named in `missing_fields`. Place definition first, optional explanatory fields before confidence, and confidence last. If the concept section is absent, append a complete concept section without changing existing custom sections. Never replace a line whose label already exists. Return the exact list of fields added so repeat runs can prove idempotency. + +- [ ] **Step 5: Add inventory CLI** + +`inspect_glossary_library.py` accepts `--terms` pointing to triage JSON, `--terms-dir`, `--source-manifest`, optional `--raw-sections`, `--reviewed-shortlist`, and `--output`. It emits: + +```json +{ + "status": "ok", + "script": "inspect_glossary_library.py", + "results": [ + { + "term": "LLM", + "surface_forms": ["LLM", "大语言模型"], + "state": "existing_thin", + "file": "C:\\Notes\\Vault\\book\\术语\\LLM.md", + "link_stem": "LLM", + "missing_fields": ["elaboration", "intuition", "distinction"] + } + ] +} +``` + +- [ ] **Step 6: Extend writer preflight and operations** + +Validate every entry against inventory before writing any file: + +- `create`: inventory state must be `new`; require name, definition, confidence, occurrence. +- `enrich`: state must be `existing_thin`; allow only fields listed in `missing_fields`; require occurrence and a valid confidence when confidence is missing. +- `reuse`: state must be `existing_complete`; require occurrence; reject concept-field replacements. + +Apply writes only after the full batch validates. Enrich missing fields, append the paper occurrence once, and report `created`, `enriched`, `updated`, or `unchanged`. Include all canonical and surface forms in each result for article matching. + +- [ ] **Step 7: Run Task 3 tests and verify GREEN** + +Run the command from Step 2. Expected: all inventory, writer, and existing lint tests pass. + +- [ ] **Step 8: Task 3 checkpoint without commit** + +Run `git diff --check` on Task 3 files and record status. Do not stage or commit. + +--- + +### Task 4: Safe first-occurrence article linker + +**Files:** +- Create: `skills/paper-glossary/scripts/link_glossary_terms.py` +- Create: `skills/paper-glossary/tests/test_link_glossary_terms.py` + +**Interfaces:** +- Consumes: writer JSON `results[].link_stem` and `results[].forms` +- Consumes: configured vault and preview-time article SHA-256 +- Produces: `link_article_terms(article: Path, mappings: list[dict[str, Any]], expected_sha256: str) -> dict[str, Any]` +- Produces result records: `linked`, `already_linked`, `not_found` + +- [ ] **Step 1: Write failing protected-region, alias, hash, and idempotency tests** + +Use a single fixture containing frontmatter, heading, ordinary prose, inline code, fenced code, URL, Markdown link, wiki link, and a references heading: + +```python +def test_linker_links_first_safe_alias_and_skips_protected_regions(tmp_path: Path) -> None: + article = tmp_path / "Paper.md" + article.write_text( + "---\ntitle: ReAct\n---\n" + "# ReAct heading\n\n" + "`ReAct` and [ReAct](https://example.test) are protected.\n\n" + "REACT combines reasoning and acting. ReAct appears again.\n\n" + "```python\nReAct = 'code'\n```\n\n" + "## References\nReAct paper\n", + encoding="utf-8", + ) + before = article.read_bytes() + digest = hashlib.sha256(before).hexdigest() + result = link_article_terms( + article, + [{"link_stem": "ReAct", "forms": ["ReAct", "REACT"]}], + digest, + ) + + text = article.read_text(encoding="utf-8") + assert "[[ReAct|REACT]] combines reasoning" in text + assert "# ReAct heading" in text + assert "`ReAct`" in text + assert "```python\nReAct = 'code'\n```" in text + assert text.count("[[ReAct|REACT]]") == 1 + assert result["summary"] == {"linked": 1, "already_linked": 0, "not_found": 0} + + +def test_linker_rejects_stale_article_hash(tmp_path: Path) -> None: + article = tmp_path / "Paper.md" + article.write_text("ReAct text\n", encoding="utf-8") + + with pytest.raises(SystemExit, match="changed since preview"): + link_article_terms(article, [{"link_stem": "ReAct", "forms": ["ReAct"]}], "0" * 64) + + +def test_linker_is_idempotent(tmp_path: Path) -> None: + article = tmp_path / "Paper.md" + article.write_text("ReAct text\n", encoding="utf-8") + first_hash = hashlib.sha256(article.read_bytes()).hexdigest() + link_article_terms(article, [{"link_stem": "ReAct", "forms": ["ReAct"]}], first_hash) + linked_bytes = article.read_bytes() + second_hash = hashlib.sha256(linked_bytes).hexdigest() + result = link_article_terms(article, [{"link_stem": "ReAct", "forms": ["ReAct"]}], second_hash) + + assert article.read_bytes() == linked_bytes + assert result["summary"]["already_linked"] == 1 +``` + +Add tests for longer-term priority (`SerpApi` before `API`), no substring match, existing wiki links, UTF-8 BOM preservation, CRLF preservation, no safe occurrence, and same-vault validation through configured paths. + +- [ ] **Step 2: Run linker tests and verify RED** + +Run: + +```powershell +py -3.12 -m pytest -q skills/paper-glossary/tests/test_link_glossary_terms.py --basetemp .pytest-tmp-paper-glossary-link +``` + +Expected: collection fails because `link_glossary_terms` does not exist. + +- [ ] **Step 3: Implement protected-range discovery** + +Create spans for frontmatter at file start, heading lines, fenced blocks, inline code, Markdown links, wiki links, HTML tags, URLs, and the references tail. Merge overlapping spans before matching. The linker must test candidate match ranges against these spans and against previously planned edits. + +Use ASCII-aware boundaries for ASCII forms: + +```python +def term_pattern(form: str) -> re.Pattern[str]: + escaped = re.escape(form) + if form.isascii(): + return re.compile(rf"(? None: + text = (SKILL_DIR / "SKILL.md").read_text(encoding="utf-8") + for requirement in ( + "first use", + ".paper-glossary", + "same Obsidian vault", + "semantic review", + "effective body", + "全部写入", + "End the response and wait", + "article Markdown", + "first safe occurrence", + "existing_thin", + "Do not overwrite", + ): + assert requirement in text + + +def test_readme_explains_per_device_one_time_configuration() -> None: + text = (SKILL_DIR / "README.md").read_text(encoding="utf-8") + assert "~/.paper-glossary/config.json" in text + assert "Windows" in text + assert "macOS" in text + assert "Linux" in text + assert ".obsidian" in text + + +def test_contract_documents_dynamic_proposal_inventory_and_link_results() -> None: + text = (SKILL_DIR / "references" / "file-contract.md").read_text(encoding="utf-8") + for field in ( + '"shortlist_limit"', + '"awaiting_semantic_review"', + '"existing_thin"', + '"operation": "enrich"', + '"article_sha256"', + '"already_linked"', + ): + assert field in text +``` + +Keep the existing self-contained boundary test that rejects imports from DeepPaperNote. + +- [ ] **Step 2: Run skill-boundary tests and verify RED** + +Run: + +```powershell +py -3.12 -m pytest -q skills/paper-glossary/tests/test_skill_boundary.py --basetemp .pytest-tmp-paper-glossary-skill +``` + +Expected: failures for missing first-use, dynamic, inventory, and linker documentation. + +- [ ] **Step 3: Rewrite `SKILL.md` around the approved interaction** + +Keep it concise and route details to references. The operational order must be explicit: + +1. Show existing config or ask for an Obsidian term directory on first use. +2. Validate an explicit article Markdown when links are requested. +3. Run deterministic proposal. +4. Perform one grounded semantic review bounded by `shortlist_limit`. +5. Display every retained term as a numbered Markdown list, the term directory, article path, and both write effects. +6. State that no files have been written, end the response, and wait. +7. Resolve numbers/names/`全部写入` from the immediately preceding list. +8. Triage, inventory, generate one action-aware batch, preflight, write/enrich/reuse, link article, lint changed files, and report timing/status. + +The default prompt in `agents/openai.yaml` must trigger preview and waiting, not immediate writing. + +- [ ] **Step 4: Document first-use setup and contracts** + +`README.md` must explain that each person configures a directory once per device, that the directory must be inside an Obsidian vault, where local config lives, how to show/reset it, and that the article must be in the same vault. Include Windows, macOS, and Linux command examples without assuming a fixed drive. + +`file-contract.md` must document proposal metrics/state, inventory state, operation-aware entry schema, writer forms, article hash, and linker results. Preserve the manifest/raw-sections collaboration boundary. + +- [ ] **Step 5: Run Task 5 tests and verify GREEN** + +Run the command from Step 2. Expected: all boundary tests pass. + +- [ ] **Step 6: Validate the skill metadata** + +Run: + +```powershell +python -X utf8 C:\Users\lenovo\.codex\skills\.system\skill-creator\scripts\quick_validate.py skills\paper-glossary +``` + +Expected: `Skill is valid!` + +- [ ] **Step 7: Task 5 checkpoint without commit** + +Run `git diff --check` on Task 5 files and record status. Do not stage or commit. + +--- + +### Task 6: Changed-file linting, timing, and end-to-end verification + +**Files:** +- Modify: `skills/paper-glossary/scripts/glossary_common.py` +- Modify: `skills/paper-glossary/scripts/lint_glossary.py` +- Modify: all paper-glossary CLIs created or changed in Tasks 1-4 to emit elapsed time +- Create: `skills/paper-glossary/tests/test_workflow_integration.py` +- Modify: `skills/paper-glossary/tests/test_write_and_lint_glossary.py` + +**Interfaces:** +- Produces: `elapsed_ms(started: float) -> int` +- Changes lint CLI: repeatable `--input` values while preserving `--terms-dir` +- Produces end-to-end status across config, proposal, review recording, triage, inventory, write, link, and lint + +- [ ] **Step 1: Write failing timing, multi-input lint, and integration tests** + +The integration fixture must exercise the authenticated artifact chain in this exact order: proposal -> recorded review -> explicit selection -> triage -> authenticated inventory -> config-backed writer -> authenticated linker when an article is supplied -> changed-note lint. Its inventory invocation passes `--source-manifest`, optional `--raw-sections`, and `--reviewed-shortlist`; its writer and linker also pass the same saved `--triage`. Use CLI helpers that construct those authenticated commands rather than direct production helpers. + +Add a helper test and a complete fixture workflow. The integration test must use temporary files and deterministic model-authored entry JSON: + +```python +def test_complete_selected_flow_creates_enriches_links_and_lints( + tmp_path: Path, +) -> None: + vault = tmp_path / "vault" + (vault / ".obsidian").mkdir(parents=True) + article = vault / "Chapter.md" + article.write_text( + "# 智能体范式\n\nReAct 与 Reflection 是两种智能体范式。\n", + encoding="utf-8", + ) + terms_dir = vault / "book" / "术语" + config = configure_terms_dir(terms_dir, tmp_path / "config.json") + article_info = validate_article(article, config) + + entries = { + "entries": [ + { + "name": "ReAct", + "aliases": ["Reasoning and Acting"], + "operation": "create", + "definition": "交替进行推理与行动的智能体范式。", + "confidence": "高", + "occurrence": "章节正文首次介绍 ReAct。", + }, + { + "name": "Reflection", + "aliases": ["反思"], + "operation": "create", + "definition": "利用反馈检查并改进先前结果的智能体范式。", + "confidence": "高", + "occurrence": "章节正文将 Reflection 列为经典范式。", + }, + ] + } + + inventory = { + "results": _run_authenticated_inventory_cli( + [ + {"term": "ReAct", "surface_forms": ["ReAct"]}, + {"term": "Reflection", "surface_forms": ["Reflection", "反思"]}, + ], + terms_dir, + ) + } + write_result = _run_authenticated_writer_cli(entries, inventory, terms_dir, "Chapter") + link_result = _run_authenticated_linker_cli( + article, + write_result["results"], + article_info["article_sha256"], + ) + + assert link_result["summary"]["linked"] == 2 + assert "[[ReAct]]" in article.read_text(encoding="utf-8") + assert "[[Reflection]]" in article.read_text(encoding="utf-8") + for path in terms_dir.glob("*.md"): + assert lint_term_file_text(path.read_text(encoding="utf-8"))["passes"] is True +``` + +Keep the CLI helper assertions focused on emitted artifacts and article/note effects; do not duplicate writer or linker logic in the test. + +Add a CLI test proving `lint_glossary.py --input A.md --input B.md` checks exactly two files. + +- [ ] **Step 2: Run integration tests and verify RED** + +Run: + +```powershell +py -3.12 -m pytest -q skills/paper-glossary/tests/test_workflow_integration.py skills/paper-glossary/tests/test_write_and_lint_glossary.py --basetemp .pytest-tmp-paper-glossary-integration +``` + +Expected: failures for missing helper integration points, repeatable lint input, and elapsed-time fields. + +- [ ] **Step 3: Add shared elapsed-time reporting** + +In `glossary_common.py` add: + +```python +from time import perf_counter + + +def elapsed_ms(started: float) -> int: + return max(0, round((perf_counter() - started) * 1000)) +``` + +Each CLI captures `started = perf_counter()` immediately inside `main()` and adds `elapsed_ms` to its emitted top-level JSON. The host separately reports semantic-review and model-generation time because those phases do not run inside Python. + +- [ ] **Step 4: Make lint input repeatable** + +Change `--input` to `action="append", default=[]`, collect each resolved path once, and retain `--terms-dir` for explicit full-library audits. Reject an empty final file list as before. + +- [ ] **Step 5: Complete the integration path and run GREEN** + +Expose a writer function beneath the CLI so integration tests call production logic. Run the command from Step 2. Expected: all integration and writer/lint tests pass. + +- [ ] **Step 6: Run the full focused suite** + +Run: + +```powershell +py -3.12 -m pytest -q skills/paper-glossary/tests --basetemp .pytest-tmp-paper-glossary +``` + +Expected: zero failures. + +- [ ] **Step 7: Run syntax and style checks available in the environment** + +Run: + +```powershell +python -m compileall -q skills\paper-glossary\scripts +python -m ruff check skills\paper-glossary +git diff --check -- skills/paper-glossary +``` + +Expected: all installed checks exit 0. If `ruff` is unavailable, report the exact `No module named ruff` result and do not install a dependency solely for this task. + +- [ ] **Step 8: Run the repository test suite** + +Run: + +```powershell +py -3.12 -m pytest -q --basetemp .pytest-tmp-full +``` + +Expected: zero failures; existing third-party deprecation warnings may remain and must be reported separately. + +- [ ] **Step 9: Measure local chapter-fixture stages** + +Use PowerShell `Measure-Command` for proposal, review recording, triage, inventory, writer, linker on a temporary copy, and changed-file lint. The review command must consume the saved proposal and exact proposal names; the triage command must consume that saved reviewed-shortlist artifact and the resolved exact user selection. Record each stage and total. The under-two-second local-script number for the chapter-four fixture is an observed, environment-dependent target: repeated runs must report a median and range, or exact overruns. It is not a flaky pytest threshold. + +- [ ] **Step 10: Final implementation checkpoint without commit** + +Review `git diff --stat`, `git diff --check`, and `git status --short`. Confirm that no file outside `skills/paper-glossary/` changed and that the six pre-existing modified files remain incorporated. Do not stage, commit, push, or open a PR. + +--- + +### Task 7: Real chapter-four acceptance preview + +**Files:** +- Runtime config: `~/.paper-glossary/config.json` +- Input article: `F:\我的笔记\123\学习\hello_agent\docs\chapter4\第四章 智能体经典范式构建.md` +- Central library: `F:\我的笔记\123\book\术语` +- Existing demo directory retained for now: `F:\我的笔记\123\学习\hello_agent\术语` + +**Interfaces:** +- Consumes the completed Tasks 1-6 implementation +- Produces the normal numbered selection interaction and stops before write + +- [ ] **Step 1: Configure this device once** + +Run: + +```powershell +python -X utf8 skills\paper-glossary\scripts\configure_glossary.py --terms-dir 'F:\我的笔记\123\book\术语' --output tmp\paper-glossary-chapter4-demo\device_config_result.json +``` + +Expected: config points to nearest vault `F:\我的笔记\123` and relative directory `book\术语`. + +- [ ] **Step 2: Validate the explicit article and capture its preview hash** + +Run: + +```powershell +python -X utf8 skills\paper-glossary\scripts\configure_glossary.py --validate-article 'F:\我的笔记\123\学习\hello_agent\docs\chapter4\第四章 智能体经典范式构建.md' --output tmp\paper-glossary-chapter4-demo\article_validation.json +``` + +Expected: article and term library resolve to the same vault and output contains `article_sha256`. + +- [ ] **Step 3: Run the improved proposal** + +Run: + +```powershell +python -X utf8 skills\paper-glossary\scripts\plan_glossary.py --propose --source-manifest tmp\paper-glossary-chapter4-demo\chapter4_source_manifest.json --output tmp\paper-glossary-chapter4-demo\glossary_candidate_pool.json +``` + +Expected: a dynamic limit derived from effective prose, `Reflection` is present in the grounded pool, case-only duplicates are absent, and code/example noise such as `KEY`, `MODEL`, `TEMPLATE`, and `HUAWEI` is absent. + +- [ ] **Step 4: Perform one semantic review, record it, and display the actual selection UI** + +Retain only grounded, reusable concepts up to the returned dynamic limit and save their exact ordered proposal `term` strings to `tmp\paper-glossary-chapter4-demo\reviewed_terms.json`. Record that host-reviewed subset before displaying it: + +```powershell +python -X utf8 skills\paper-glossary\scripts\plan_glossary.py --review-proposal tmp\paper-glossary-chapter4-demo\glossary_candidate_pool.json --reviewed-terms tmp\paper-glossary-chapter4-demo\reviewed_terms.json --source-manifest tmp\paper-glossary-chapter4-demo\chapter4_source_manifest.json --output tmp\paper-glossary-chapter4-demo\glossary_reviewed_shortlist.json +``` + +Expected: `glossary_reviewed_shortlist.json` contains the exact retained full candidate objects, ordered forms, and proposal/review provenance. Display every retained item from this reviewed artifact as a numbered Markdown list together with: + +- `F:\我的笔记\123\book\术语` as the write destination; +- the exact chapter Markdown path as the link target; +- the statement that selecting terms will create/enrich notes and add first safe article links; +- the statement that no files have yet been written. + +End the response and wait for the user's numbers, names, or `全部写入`. This task intentionally stops at the approved selection gate. + +- [ ] **Step 5: Preserve old demo output until the corrected flow is approved** + +Do not move or delete `F:\我的笔记\123\学习\hello_agent\术语` during preview. After a later explicit selection completes the new write/link/lint flow, compare those assistant-created demo files with their recorded content and ask before removing any file that may contain later user edits. + +--- + +## Execution Completion Criteria + +- Tasks 1-6 pass focused and full tests with no unreported failures. +- The skill validator passes. +- Local script timings are reported and the chapter fixture is benchmarked. +- No DeepPaperNote workflow, plugin manifest, dependency file, or unrelated path changes. +- No commit or push occurs. +- Task 7 presents the corrected real shortlist and stops for explicit user selection. diff --git a/skills/paper-glossary/scripts/configure_glossary.py b/skills/paper-glossary/scripts/configure_glossary.py new file mode 100644 index 0000000..e92cb2e --- /dev/null +++ b/skills/paper-glossary/scripts/configure_glossary.py @@ -0,0 +1,79 @@ +#!/usr/bin/env python3 +"""Configure and inspect the device-local paper-glossary vault.""" + +from __future__ import annotations + +import argparse +import json +from pathlib import Path +from time import perf_counter +from typing import Any + +from glossary_common import elapsed_ms +from glossary_config import ( + configure_terms_dir, + default_config_path, + load_config, + resolve_terms_dir, + validate_article, +) + + +def parser() -> argparse.ArgumentParser: + command = argparse.ArgumentParser( + description=__doc__ or "configure paper-glossary" + ) + modes = command.add_mutually_exclusive_group(required=True) + modes.add_argument("--terms-dir", metavar="PATH", help="Term-library directory.") + modes.add_argument("--show", action="store_true", help="Show saved configuration.") + modes.add_argument("--reset", action="store_true", help="Remove saved configuration.") + modes.add_argument( + "--validate-article", metavar="PATH", help="Validate an article Markdown path." + ) + command.add_argument("--config-path", metavar="PATH", help="Config JSON path.") + command.add_argument("--output", default="", metavar="PATH", help="Output JSON path.") + return command + + +def _emit(payload: dict[str, Any], output: str) -> None: + text = json.dumps(payload, ensure_ascii=False, indent=2) + "\n" + if output: + target = Path(output).expanduser() + target.parent.mkdir(parents=True, exist_ok=True) + target.write_text(text, encoding="utf-8") + else: + print(text, end="") + + +def main() -> None: + started = perf_counter() + args = parser().parse_args() + config_path = Path(args.config_path).expanduser() if args.config_path else None + + if args.terms_dir: + payload = configure_terms_dir(Path(args.terms_dir), config_path) + elif args.show: + config = load_config(config_path) + if config is None: + payload = {"workflow_state": "needs_configuration"} + else: + payload = {**config, "terms_dir": str(resolve_terms_dir(config))} + elif args.reset: + target = config_path or default_config_path() + if target.expanduser().is_file(): + target.expanduser().unlink() + payload = {"workflow_state": "reset"} + else: + config = load_config(config_path) + if config is None: + raise SystemExit( + "Paper-glossary is not configured; run with --terms-dir first." + ) + terms_dir = resolve_terms_dir(config) + payload = {"terms_dir": str(terms_dir), **validate_article(Path(args.validate_article), config)} + + _emit({**payload, "elapsed_ms": elapsed_ms(started)}, args.output) + + +if __name__ == "__main__": + main() diff --git a/skills/paper-glossary/scripts/glossary_common.py b/skills/paper-glossary/scripts/glossary_common.py index 46d50e0..d39a94c 100644 --- a/skills/paper-glossary/scripts/glossary_common.py +++ b/skills/paper-glossary/scripts/glossary_common.py @@ -1,15 +1,324 @@ from __future__ import annotations +import hashlib import json import re from pathlib import Path +from time import perf_counter from typing import Any +def elapsed_ms(started: float) -> int: + return max(0, round((perf_counter() - started) * 1000)) + + def normalize_whitespace(value: str) -> str: return re.sub(r"\s+", " ", value).strip() +def canonical_sha256(value: Any) -> str: + encoded = json.dumps( + value, ensure_ascii=False, sort_keys=True, separators=(",", ":") + ).encode("utf-8") + return hashlib.sha256(encoded).hexdigest() + + +def proposal_sha256( + paper_id: str, + source_sha256: str, + candidates: list[dict[str, Any]], + summary: dict[str, Any], +) -> str: + return canonical_sha256( + { + "paper_id": paper_id, + "source_sha256": source_sha256, + "candidates": candidates, + "summary": summary, + } + ) + + +def review_sha256( + proposal_provenance: dict[str, Any], reviewed_shortlist: list[dict[str, Any]] +) -> str: + return canonical_sha256( + { + "paper_id": proposal_provenance.get("paper_id"), + "proposal_provenance": proposal_provenance, + "reviewed_shortlist": reviewed_shortlist, + } + ) + + +def selection_sha256(review_sha256: str, items: list[dict[str, Any]]) -> str: + selected_terms = [ + {"term": item.get("term"), "surface_forms": item.get("surface_forms")} + for item in items + ] + return canonical_sha256( + {"review_sha256": review_sha256, "selected_terms": selected_terms} + ) + + +def writer_mappings_sha256( + provenance: dict[str, Any], + results: list[dict[str, Any]], + context: dict[str, Any], + triage_sha256: str, +) -> str: + mappings: list[dict[str, Any]] = [] + for item in results: + if not isinstance(item, dict): + raise SystemExit("Writer results must contain mapping objects.") + try: + mappings.append( + { + field: item[field] + for field in ("name", "forms", "file", "link_stem") + } + ) + except KeyError as exc: + raise SystemExit( + f"Writer mapping is missing required field: {exc.args[0]}" + ) from None + if not _valid_digest(triage_sha256): + raise SystemExit("Writer mappings require a valid triage artifact digest.") + return canonical_sha256( + { + "provenance": provenance, + "mappings": mappings, + "context": context, + "triage_sha256": triage_sha256, + } + ) + + +def _valid_digest(value: Any) -> bool: + return isinstance(value, str) and re.fullmatch(r"[0-9a-f]{64}", value) is not None + + +def _validated_term_sequence(value: Any, label: str) -> list[dict[str, Any]]: + if not isinstance(value, list) or any(not isinstance(item, dict) for item in value): + raise SystemExit(f"Invalid {label} provenance structure.") + terms: set[str] = set() + for item in value: + term = item.get("term") + forms = item.get("surface_forms") + if ( + not isinstance(term, str) + or not term + or normalize_whitespace(term) != term + or term in terms + or not isinstance(forms, list) + or not forms + ): + raise SystemExit(f"Invalid {label} term or surface_forms provenance.") + exact_forms: set[str] = set() + for form in forms: + if ( + not isinstance(form, str) + or not form + or normalize_whitespace(form) != form + or form in exact_forms + ): + raise SystemExit(f"Invalid or duplicate {label} surface_forms provenance.") + exact_forms.add(form) + if term not in exact_forms: + raise SystemExit(f"Canonical {label} term must be present in surface_forms.") + terms.add(term) + return value + + +def validate_provenance_chain( + payload: dict[str, Any], selected_items: Any +) -> dict[str, Any]: + paper_id = payload.get("paper_id") + provenance = payload.get("provenance") + proposal = provenance.get("proposal") if isinstance(provenance, dict) else None + review = provenance.get("review") if isinstance(provenance, dict) else None + selection_digest = ( + provenance.get("selection_sha256") if isinstance(provenance, dict) else None + ) + if ( + not isinstance(paper_id, str) + or not paper_id + or not isinstance(proposal, dict) + or not isinstance(review, dict) + or not _valid_digest(selection_digest) + ): + raise SystemExit("Artifact must contain valid paper and selection provenance.") + + source_digest = proposal.get("source_sha256") + proposal_digest = proposal.get("proposal_sha256") + candidates = _validated_term_sequence( + proposal.get("candidates"), "proposal candidates" + ) + summary = proposal.get("summary") + if ( + proposal.get("paper_id") != paper_id + or not _valid_digest(source_digest) + or not _valid_digest(proposal_digest) + or not isinstance(summary, dict) + ): + raise SystemExit("Invalid proposal provenance or top-level paper binding.") + expected_proposal_digest = proposal_sha256( + paper_id, source_digest, candidates, summary + ) + if proposal_digest != expected_proposal_digest: + raise SystemExit("Proposal digest does not match embedded provenance material.") + + proposal_fields = ( + "paper_id", + "source_sha256", + "candidates", + "summary", + "proposal_sha256", + ) + if any(review.get(field) != proposal[field] for field in proposal_fields): + raise SystemExit("Proposal and review provenance fields do not match.") + reviewed_shortlist = _validated_term_sequence( + review.get("reviewed_shortlist"), "reviewed shortlist" + ) + reviewed_terms: set[str] = set() + candidates_by_term = {candidate["term"]: candidate for candidate in candidates} + for candidate in reviewed_shortlist: + term = candidate["term"] + if candidate != candidates_by_term.get(term) or term in reviewed_terms: + raise SystemExit("Reviewed shortlist is not an exact proposal subset.") + reviewed_terms.add(term) + limit = summary.get("shortlist_limit") + if not isinstance(limit, int) or isinstance(limit, bool) or limit < 0: + raise SystemExit("Invalid proposal shortlist limit provenance.") + if len(reviewed_shortlist) > limit: + raise SystemExit("Reviewed shortlist exceeds proposal shortlist limit.") + + review_digest = review.get("review_sha256") + if not _valid_digest(review_digest): + raise SystemExit("Invalid review digest provenance.") + proposal_material = {field: proposal[field] for field in proposal_fields} + if review_digest != review_sha256(proposal_material, reviewed_shortlist): + raise SystemExit("Review digest does not match embedded provenance material.") + + selected = _validated_term_sequence(selected_items, "selected") + reviewed_by_term = {candidate["term"]: candidate for candidate in reviewed_shortlist} + for item in selected: + reviewed = reviewed_by_term.get(item["term"]) + if reviewed is None or item["surface_forms"] != reviewed["surface_forms"]: + raise SystemExit( + "Selected term and surface_forms must exactly match the reviewed shortlist." + ) + if selection_digest != selection_sha256(review_digest, selected): + raise SystemExit("Selection digest does not match selected term provenance.") + return provenance + + +def validate_current_provenance_chain( + payload: dict[str, Any], + selected_items: Any, + source_manifest: str, + raw_sections: str, + reviewed_shortlist: str, +) -> dict[str, Any]: + # Imported lazily because plan_glossary itself uses the digest helpers above. + from plan_glossary import _validated_review, load_manifest_and_sections + + manifest, records = load_manifest_and_sections(source_manifest, raw_sections) + review_payload = load_json_file( + Path(reviewed_shortlist).expanduser().resolve() + ) + current_review, current_proposal = _validated_review( + review_payload, manifest, records + ) + provenance = validate_provenance_chain(payload, selected_items) + if ( + payload.get("paper_id") != current_proposal["paper_id"] + or provenance.get("proposal") != current_proposal["provenance"] + or provenance.get("review") != current_review["provenance"] + ): + raise SystemExit( + "Artifact provenance does not match the current source and reviewed shortlist." + ) + return { + "paper_id": current_proposal["paper_id"], + "manifest": manifest, + "provenance": provenance, + } + + +def _selected_term_forms( + value: Any, + label: str, + term_field: str = "term", + forms_field: str = "surface_forms", +) -> list[dict[str, Any]]: + if not isinstance(value, list) or any(not isinstance(item, dict) for item in value): + raise SystemExit(f"Invalid {label} authorization structure.") + selected = [ + {"term": item.get(term_field), "surface_forms": item.get(forms_field)} + for item in value + ] + return _validated_term_sequence(selected, label) + + +def triage_artifact_sha256(triage: dict[str, Any]) -> str: + return canonical_sha256( + { + "paper_id": triage["paper_id"], + "provenance": triage["provenance"], + "terms": triage["terms"], + } + ) + + +def validate_authorized_selection( + payload: dict[str, Any], + selected_items: Any, + triage_artifact: str, + source_manifest: str, + raw_sections: str, + reviewed_shortlist: str, + *, + term_field: str = "term", + forms_field: str = "surface_forms", +) -> dict[str, Any]: + triage = load_json_file(Path(triage_artifact).expanduser().resolve()) + if ( + triage.get("status") != "ok" + or triage.get("script") != "plan_glossary.py" + or triage.get("mode") != "triage" + ): + raise SystemExit("Authorized selection must be a saved triage artifact.") + authorized = _selected_term_forms(triage.get("terms"), "authorized triage") + current = validate_current_provenance_chain( + triage, + authorized, + source_manifest, + raw_sections, + reviewed_shortlist, + ) + selected = _selected_term_forms( + selected_items, + "downstream selected", + term_field, + forms_field, + ) + provenance = validate_provenance_chain(payload, selected) + if ( + selected != authorized + or payload.get("paper_id") != current["paper_id"] + or provenance != current["provenance"] + ): + raise SystemExit( + "Downstream selection does not exactly match the authorized triage artifact." + ) + return { + **current, + "authorized_selection": authorized, + "triage_sha256": triage_artifact_sha256(triage), + } + + def load_json_file(path: Path) -> dict[str, Any]: data = json.loads(path.read_text(encoding="utf-8-sig")) if not isinstance(data, dict): diff --git a/skills/paper-glossary/scripts/glossary_config.py b/skills/paper-glossary/scripts/glossary_config.py new file mode 100644 index 0000000..020843e --- /dev/null +++ b/skills/paper-glossary/scripts/glossary_config.py @@ -0,0 +1,139 @@ +#!/usr/bin/env python3 +"""Resolve and validate device-local paper-glossary configuration.""" + +from __future__ import annotations + +import hashlib +import json +from pathlib import Path + +CONFIG_DIRNAME = ".paper-glossary" +CONFIG_FILENAME = "config.json" + + +def default_config_path() -> Path: + return Path.home() / CONFIG_DIRNAME / CONFIG_FILENAME + + +def find_vault_root(path: Path) -> Path | None: + current = path.expanduser().resolve(strict=False) + while not current.exists() and current != current.parent: + current = current.parent + if current.is_file(): + current = current.parent + for candidate in (current, *current.parents): + if (candidate / ".obsidian").is_dir(): + return candidate.resolve() + return None + + +def _is_within(path: Path, parent: Path) -> bool: + try: + path.resolve(strict=False).relative_to(parent.resolve()) + except ValueError: + return False + return True + + +def _validated_config_payload( + payload: object, + error: str, + source_terms_dir: Path | None = None, +) -> dict[str, str]: + if not isinstance(payload, dict): + raise SystemExit(error) + for field in ("vault_root", "terms_subdir"): + value = payload.get(field) + if not isinstance(value, str) or not value.strip(): + raise SystemExit(error) + try: + subdir = Path(payload["terms_subdir"]) + vault_path = Path(payload["vault_root"]) + if not vault_path.is_absolute(): + raise SystemExit(error) + paths = (subdir,) if source_terms_dir is None else (subdir, source_terms_dir) + if any( + not part.strip() + for path in paths + for part in path.parts + if part != path.anchor + ): + raise SystemExit(error) + vault = vault_path.resolve(strict=False) + terms_dir = (vault / subdir).resolve(strict=False) + except (OSError, RuntimeError, ValueError) as exc: + raise SystemExit(error) from exc + if subdir.anchor or terms_dir == vault: + raise SystemExit(error) + return { + "vault_root": str(vault), + "terms_subdir": payload["terms_subdir"], + } + + +def configure_terms_dir( + terms_dir: Path, config_path: Path | None = None +) -> dict[str, str]: + resolved = terms_dir.expanduser().resolve(strict=False) + vault = find_vault_root(resolved) + if vault is None or not _is_within(resolved, vault): + raise SystemExit( + "Term directory must be inside an Obsidian vault containing .obsidian." + ) + payload = _validated_config_payload( + { + "vault_root": str(vault), + "terms_subdir": str(resolved.relative_to(vault)), + }, + "Term directory must be a real child inside an Obsidian vault.", + terms_dir.expanduser(), + ) + resolved.mkdir(parents=True, exist_ok=True) + target = (config_path or default_config_path()).expanduser() + target.parent.mkdir(parents=True, exist_ok=True) + target.write_text( + json.dumps(payload, ensure_ascii=False, indent=2) + "\n", encoding="utf-8" + ) + return payload + + +def load_config(config_path: Path | None = None) -> dict[str, str] | None: + target = (config_path or default_config_path()).expanduser() + if not target.is_file(): + return None + try: + payload = json.loads(target.read_text(encoding="utf-8-sig")) + except (OSError, UnicodeDecodeError, json.JSONDecodeError) as exc: + raise SystemExit(f"Invalid paper-glossary config: {target}") from exc + return _validated_config_payload( + payload, f"Invalid paper-glossary config object: {target}" + ) + + +def resolve_terms_dir(config: dict[str, str]) -> Path: + vault = Path(config.get("vault_root", "")).expanduser().resolve(strict=False) + terms_dir = (vault / config.get("terms_subdir", "")).resolve(strict=False) + if ( + not (vault / ".obsidian").is_dir() + or not terms_dir.is_dir() + or not _is_within(terms_dir, vault) + or find_vault_root(terms_dir) != vault + ): + raise SystemExit( + "Configured Obsidian term directory is missing or invalid; configure it again." + ) + return terms_dir + + +def validate_article(article: Path, config: dict[str, str]) -> dict[str, str]: + resolved = article.expanduser().resolve() + if resolved.suffix.lower() != ".md" or not resolved.is_file(): + raise SystemExit(f"Article Markdown not found: {resolved}") + article_vault = find_vault_root(resolved) + configured_vault = Path(config["vault_root"]).expanduser().resolve() + if article_vault != configured_vault: + raise SystemExit( + "Article and term directory must be inside the same Obsidian vault." + ) + digest = hashlib.sha256(resolved.read_bytes()).hexdigest() + return {"article_path": str(resolved), "article_sha256": digest} diff --git a/skills/paper-glossary/scripts/glossary_library.py b/skills/paper-glossary/scripts/glossary_library.py new file mode 100644 index 0000000..ceafae5 --- /dev/null +++ b/skills/paper-glossary/scripts/glossary_library.py @@ -0,0 +1,345 @@ +from __future__ import annotations + +import json +import re +import unicodedata +from pathlib import Path +from typing import Any + +from glossary_common import normalize_whitespace +from glossary_contracts import ( + GLOSSARY_CONCEPT_HEADING, + GLOSSARY_LABEL_CONFIDENCE, + GLOSSARY_LABEL_DEFINITION, +) + +CONCEPT_FIELDS = { + "definition": GLOSSARY_LABEL_DEFINITION, + "elaboration": "\u8be6\u89e3\uff1a", + "intuition": "\u76f4\u89c9\uff1a", + "distinction": "\u4e0e\u76f8\u90bb\u6982\u5ff5\u7684\u533a\u522b\uff1a", + "confidence": GLOSSARY_LABEL_CONFIDENCE, +} +OPTIONAL_CONCEPT_FIELDS = ("elaboration", "intuition", "distinction") + + +def _decode_frontmatter_scalar(value: str) -> str: + raw = value.strip() + if not raw: + raise SystemExit("Malformed frontmatter scalar: empty value") + if raw.startswith('"'): + if len(raw) < 2 or not raw.endswith('"'): + raise SystemExit(f"Malformed frontmatter scalar: {raw}") + try: + decoded = json.loads(raw) + except json.JSONDecodeError: + raise SystemExit(f"Malformed frontmatter scalar: {raw}") from None + if not isinstance(decoded, str): + raise SystemExit(f"Malformed frontmatter scalar: {raw}") + result = normalize_whitespace(decoded) + elif raw.startswith("'"): + if len(raw) < 2 or not raw.endswith("'"): + raise SystemExit(f"Malformed frontmatter scalar: {raw}") + inner = raw[1:-1] + decoded_chars: list[str] = [] + index = 0 + while index < len(inner): + if inner[index] == "'": + if index + 1 >= len(inner) or inner[index + 1] != "'": + raise SystemExit(f"Malformed frontmatter scalar: {raw}") + decoded_chars.append("'") + index += 2 + continue + decoded_chars.append(inner[index]) + index += 1 + result = normalize_whitespace("".join(decoded_chars)) + else: + if any(character in raw for character in "[]{}\"'#"): + raise SystemExit(f"Malformed frontmatter scalar: {raw}") + result = normalize_whitespace(raw) + if not result: + raise SystemExit(f"Malformed frontmatter scalar: {raw}") + return result + + +def _split_inline_aliases(content: str) -> list[str]: + if not content.strip(): + return [] + items: list[str] = [] + start = 0 + quote = "" + index = 0 + while index < len(content): + character = content[index] + if quote == '"': + if character == "\\": + index += 2 + continue + if character == quote: + quote = "" + elif quote == "'": + if character == quote: + if index + 1 < len(content) and content[index + 1] == quote: + index += 2 + continue + quote = "" + elif character in ("'", '"'): + quote = character + elif character == ",": + item = content[start:index].strip() + if not item: + raise SystemExit("Malformed aliases metadata: empty inline item.") + items.append(item) + start = index + 1 + index += 1 + if quote: + raise SystemExit("Malformed aliases metadata: unclosed quote.") + item = content[start:].strip() + if not item: + raise SystemExit("Malformed aliases metadata: empty inline item.") + items.append(item) + return items + + +def _frontmatter_block(text: str, source: Path | None = None) -> str | None: + opening = re.match(r"\A---[ \t]*(?:\r?\n|$)", text) + if opening is None: + return None + closing = re.search(r"(?m)^---[ \t]*\r?$", text[opening.end() :]) + if closing is None: + location = f" in {source}" if source is not None else "" + raise SystemExit(f"Unclosed frontmatter{location}.") + return text[opening.end() : opening.end() + closing.start()] + + +def read_frontmatter_aliases(text: str, source: Path | None = None) -> list[str]: + block = _frontmatter_block(text, source) + if block is None: + return [] + lines = block.splitlines() + declarations = [ + (index, match.group(1).strip()) + for index, line in enumerate(lines) + if (match := re.match(r"^aliases\s*:(.*)$", line)) + ] + if not declarations: + return [] + location = f" in {source}" if source is not None else "" + if len(declarations) != 1: + raise SystemExit(f"Unsupported aliases metadata{location}: duplicate aliases key.") + declaration_index, value = declarations[0] + if value: + if not value.startswith("[") or not value.endswith("]"): + raise SystemExit(f"Unsupported aliases metadata{location}: expected a list.") + return [ + _decode_frontmatter_scalar(item) + for item in _split_inline_aliases(value[1:-1]) + ] + + aliases: list[str] = [] + for line in lines[declaration_index + 1 :]: + if not line.strip(): + continue + if not line.startswith((" ", "\t")): + break + item = re.match(r"^[ \t]+-[ \t]+(.+?)[ \t]*$", line) + if item is None: + raise SystemExit(f"Unsupported aliases metadata{location}: expected list items.") + aliases.append(_decode_frontmatter_scalar(item.group(1))) + return aliases + + +def read_heading_name(text: str) -> str: + match = re.search(r"(?m)^#\s+(.+?)\s*$", text) + return normalize_whitespace(match.group(1)) if match else "" + + +def _normalized_key(value: str) -> str: + return unicodedata.normalize("NFKC", normalize_whitespace(value)).casefold() + + +def read_glossary_note(path: Path) -> str: + try: + return path.read_bytes().decode("utf-8-sig") + except (OSError, UnicodeDecodeError) as exc: + raise SystemExit(f"Unable to read glossary note {path}: {exc}") from None + + +def build_alias_index(terms_dir: Path) -> dict[str, Path]: + index: dict[str, Path] = {} + root = terms_dir.expanduser().resolve() + if not root.is_dir(): + return index + for unresolved in sorted(root.glob("*.md"), key=lambda item: item.name.casefold()): + try: + path = unresolved.resolve(strict=True) + except OSError as exc: + raise SystemExit(f"Unable to resolve glossary note {unresolved}: {exc}") from None + if not path.is_relative_to(root) or not path.is_file(): + raise SystemExit(f"Glossary note resolves outside terms directory: {unresolved}") + text = read_glossary_note(path) + keys = {_normalized_key(unresolved.stem)} + keys.update(_normalized_key(alias) for alias in read_frontmatter_aliases(text, path)) + heading = read_heading_name(text) + if heading: + keys.add(_normalized_key(heading)) + for key in keys: + existing = index.get(key) + if existing is not None and existing != path: + raise SystemExit( + f"Glossary key collision for '{key}' between {existing} and {path}." + ) + index[key] = path + return index + + +def _section_bounds(text: str) -> tuple[int, int] | None: + heading = re.search( + rf"(?m)^##\s+{re.escape(GLOSSARY_CONCEPT_HEADING)}\s*$(?:\r?\n)?", text + ) + if heading is None: + return None + following = re.search(r"(?m)^##\s+", text[heading.end() :]) + end = heading.end() + following.start() if following else len(text) + return heading.end(), end + + +def _field_present(section: str, label: str) -> bool: + return bool( + re.search(rf"(?m)^[ \t]*(?:-[ \t]*)?{re.escape(label)}[ \t]*\S", section) + ) + + +def missing_concept_fields(text: str) -> list[str]: + bounds = _section_bounds(text) + if bounds is None: + return list(CONCEPT_FIELDS) + section = text[bounds[0] : bounds[1]] + return [field for field, label in CONCEPT_FIELDS.items() if not _field_present(section, label)] + + +def note_state(text: str) -> tuple[str, list[str]]: + missing = missing_concept_fields(text) + optional_present = sum(field not in missing for field in OPTIONAL_CONCEPT_FIELDS) + required_missing = any(field in missing for field in ("definition", "confidence")) + state = "existing_thin" if required_missing or optional_present < 2 else "existing_complete" + return state, missing + + +def _line_ending(text: str) -> str: + return "\r\n" if "\r\n" in text else "\n" + + +def _field_line(field: str, value: Any, line_ending: str) -> str: + return f"- {CONCEPT_FIELDS[field]}{normalize_whitespace(str(value))}{line_ending}" + + +def _field_lines(fields: list[str], patch: dict[str, Any], line_ending: str) -> str: + return "".join(_field_line(field, patch[field], line_ending) for field in fields) + + +def add_missing_concept_fields(text: str, patch: dict[str, Any]) -> tuple[str, list[str]]: + missing = set(missing_concept_fields(text)) + fields_added = [ + field + for field in CONCEPT_FIELDS + if field in missing and normalize_whitespace(str(patch.get(field, ""))) + ] + if not fields_added: + return text, [] + + line_ending = _line_ending(text) + bounds = _section_bounds(text) + if bounds is None: + prefix = "" if not text else (line_ending if text.endswith(("\n", "\r")) else line_ending * 2) + if text and text.endswith(("\n", "\r")): + prefix = line_ending + section = f"## {GLOSSARY_CONCEPT_HEADING}{line_ending}" + return text + prefix + section + _field_lines(fields_added, patch, line_ending), fields_added + + start, end = bounds + section = text[start:end] + existing_confidence = re.search( + rf"(?m)^\s*(?:-\s*)?{re.escape(CONCEPT_FIELDS['confidence'])}", section + ) + optional_fields = [field for field in OPTIONAL_CONCEPT_FIELDS if field in fields_added] + confidence_fields = [field for field in ("confidence",) if field in fields_added] + definition_fields = [field for field in ("definition",) if field in fields_added] + insertions: list[tuple[int, str]] = [] + if definition_fields: + insertions.append((start, _field_lines(definition_fields, patch, line_ending))) + if optional_fields: + position = start + existing_confidence.start() if existing_confidence else end + insertions.append((position, _field_lines(optional_fields, patch, line_ending))) + if confidence_fields: + insertions.append((end, _field_lines(confidence_fields, patch, line_ending))) + updated = text + for position, insertion in sorted(insertions, key=lambda item: item[0], reverse=True): + updated = updated[:position] + insertion + updated[position:] + return updated, fields_added + + +def _surface_forms(item: dict[str, Any]) -> list[str]: + term = item.get("term") + surface_forms = item.get("surface_forms") + if not isinstance(term, str) or not term or normalize_whitespace(term) != term: + raise SystemExit("Selected term must be a non-empty normalized string.") + if not isinstance(surface_forms, list) or not surface_forms: + raise SystemExit(f"surface_forms must be a non-empty list for term: {term}") + forms: list[str] = [] + for value in surface_forms: + if not isinstance(value, str) or not value or normalize_whitespace(value) != value: + raise SystemExit(f"surface_forms must contain exact non-empty strings: {term}") + if value in forms: + raise SystemExit(f"surface_forms must not contain duplicates: {term}") + forms.append(value) + if term not in forms: + raise SystemExit(f"surface_forms must contain the exact selected term: {term}") + return forms + + +def inspect_selected_terms(selected: list[dict[str, Any]], terms_dir: Path) -> list[dict[str, Any]]: + index = build_alias_index(terms_dir) + results: list[dict[str, Any]] = [] + for item in selected: + if not isinstance(item, dict): + raise SystemExit("Selected terms must contain objects with surface_forms.") + term = item.get("term") + if not isinstance(term, str): + raise SystemExit("Selected term must be a string with surface_forms.") + forms = _surface_forms(item) + matches = { + index[_normalized_key(form)] + for form in forms + if _normalized_key(form) in index + } + if len(matches) > 1: + raise SystemExit( + f"Selected forms resolve to multiple glossary notes for term: {term}" + ) + path = next(iter(matches), None) + if path is None: + results.append( + { + "term": term, + "surface_forms": forms, + "state": "new", + "file": "", + "link_stem": term, + "missing_fields": [], + } + ) + continue + text = read_glossary_note(path) + state, missing = note_state(text) + results.append( + { + "term": term, + "surface_forms": forms, + "state": state, + "file": str(path), + "link_stem": path.stem, + "missing_fields": missing, + } + ) + return results diff --git a/skills/paper-glossary/scripts/inspect_glossary_library.py b/skills/paper-glossary/scripts/inspect_glossary_library.py new file mode 100644 index 0000000..ff71d13 --- /dev/null +++ b/skills/paper-glossary/scripts/inspect_glossary_library.py @@ -0,0 +1,87 @@ +#!/usr/bin/env python3 +"""Inspect selected terms against an existing glossary library.""" + +from __future__ import annotations + +import argparse +from pathlib import Path +from time import perf_counter +from typing import Any + +from glossary_common import ( + elapsed_ms, + emit, + load_json_file, + validate_current_provenance_chain, +) +from glossary_library import inspect_selected_terms + + +def parser() -> argparse.ArgumentParser: + result = argparse.ArgumentParser(description=__doc__) + result.add_argument("--terms", required=True, help="Selected-term JSON path.") + result.add_argument("--terms-dir", required=True, help="Existing glossary directory.") + result.add_argument("--source-manifest", required=True, help="Current source manifest.") + result.add_argument("--raw-sections", default="", help="Optional current raw-sections JSONL.") + result.add_argument( + "--reviewed-shortlist", required=True, help="Saved reviewed-shortlist artifact." + ) + result.add_argument("--output", default="", help="Output JSON status path.") + return result + + +def _selected_terms(payload: dict[str, Any]) -> list[dict[str, Any]]: + selected = payload.get("selected", payload.get("terms", [])) + if not isinstance(selected, list): + raise SystemExit("Selected-term JSON must contain a 'selected' list.") + return selected + + +def _selection_provenance( + payload: dict[str, Any], + selected: list[dict[str, Any]], + source_manifest: str, + raw_sections: str, + reviewed_shortlist: str, +) -> dict[str, Any]: + if payload.get("mode") != "triage": + raise SystemExit("Selected terms must be a triage artifact with valid provenance.") + return validate_current_provenance_chain( + payload, + selected, + source_manifest, + raw_sections, + reviewed_shortlist, + )["provenance"] + + +def main() -> None: + started = perf_counter() + args = parser().parse_args() + payload = load_json_file(Path(args.terms).expanduser().resolve()) + selected = _selected_terms(payload) + provenance = _selection_provenance( + payload, + selected, + args.source_manifest, + args.raw_sections, + args.reviewed_shortlist, + ) + results = inspect_selected_terms( + selected, Path(args.terms_dir).expanduser().resolve() + ) + emit( + { + "status": "ok", + "script": "inspect_glossary_library.py", + "paper_id": payload["paper_id"], + "provenance": provenance, + "results": results, + "elapsed_ms": elapsed_ms(started), + }, + args.output, + ) + + +if __name__ == "__main__": + main() diff --git a/skills/paper-glossary/scripts/link_glossary_terms.py b/skills/paper-glossary/scripts/link_glossary_terms.py new file mode 100644 index 0000000..8f80ac9 --- /dev/null +++ b/skills/paper-glossary/scripts/link_glossary_terms.py @@ -0,0 +1,943 @@ +#!/usr/bin/env python3 +"""Add one safe glossary wiki link per selected term to an explicit article.""" + +from __future__ import annotations + +import argparse +import hashlib +import os +import re +import tempfile +from pathlib import Path +from time import perf_counter +from typing import Any + +from glossary_common import ( + elapsed_ms, + emit, + load_json_file, + validate_authorized_selection, + writer_mappings_sha256, +) +from glossary_config import load_config, resolve_terms_dir, validate_article +from glossary_library import _normalized_key, build_alias_index + + +REFERENCE_HEADING_RE = re.compile( + r"^[ \t]{0,3}#{1,6}[ \t]+(?:references|bibliography|参考文献)[ \t]*\r?$", + re.IGNORECASE | re.MULTILINE, +) + + +def parser() -> argparse.ArgumentParser: + command = argparse.ArgumentParser(description=__doc__ or "link glossary terms") + command.add_argument("--input", required=True, help="Explicit article Markdown path.") + command.add_argument("--write-result", required=True, help="Successful writer result JSON path.") + command.add_argument("--expected-sha256", required=True, help="Preview-time article SHA-256.") + command.add_argument("--config-path", default="", help="Device-local configuration path.") + command.add_argument("--source-manifest", required=True, help="Current source manifest.") + command.add_argument("--raw-sections", default="", help="Optional current raw-sections JSONL.") + command.add_argument( + "--reviewed-shortlist", required=True, help="Saved reviewed-shortlist artifact." + ) + command.add_argument("--triage", required=True, help="Saved authorized triage artifact.") + command.add_argument("--output", default="", help="Output JSON status path.") + return command + + +def term_pattern(form: str) -> re.Pattern[str]: + escaped = re.escape(form) + if form.isascii(): + return re.compile(rf"(? int: + newline = text.find("\n", start) + return len(text) if newline == -1 else newline + 1 + + +def _offset_in_spans(offset: int, spans: list[tuple[int, int]]) -> bool: + for start, end in spans: + if offset < start: + return False + if offset < end: + return True + return False + + +def _list_content_start(content: str) -> int: + position = 0 + while True: + marker = re.match( + r"(?:[*+-]|\d{1,9}[.)])(?:[ \t]+|$)", content[position:] + ) + if marker is None: + return position + position += marker.end() + + +def _fenced_code_spans(text: str) -> list[tuple[int, int]]: + spans: list[tuple[int, int]] = [] + opener: tuple[str, int, int, int] | None = None + offset = 0 + for line in text.splitlines(keepends=True): + content_end = offset + len(line.rstrip("\r\n")) + line_content = _reference_line_content_start(text, offset) + if line_content is None: + content = "" + quote_depth = -1 + else: + content_start, quote_depth = line_content + content = text[content_start:content_end] + if opener is None: + list_content = content[_list_content_start(content) :] + match = re.match(r"(`{3,}|~{3,})", list_content) + if match: + marker = match.group(1) + opener = (marker[0], len(marker), offset, quote_depth) + else: + closer = re.fullmatch(r"(`+|~+)[ \t]*", content) + if ( + closer + and closer.group(1)[0] == opener[0] + and len(closer.group(1)) >= opener[1] + and quote_depth == opener[3] + ): + spans.append((opener[2], offset + len(line))) + opener = None + offset += len(line) + if opener is not None: + spans.append((opener[2], len(text))) + return spans + + +def _frontmatter_span(text: str) -> tuple[int, int] | None: + lines = text.splitlines(keepends=True) + if not lines or not re.fullmatch(r"---[ \t]*", lines[0].rstrip("\r\n")): + return None + offset = len(lines[0]) + for line in lines[1:]: + if re.fullmatch(r"(?:---|\.\.\.)[ \t]*", line.rstrip("\r\n")): + return 0, offset + len(line) + offset += len(line) + return 0, len(text) + + +def _inline_code_spans( + text: str, protected: list[tuple[int, int]] +) -> list[tuple[int, int]]: + spans: list[tuple[int, int]] = [] + cursor = 0 + while cursor < len(text): + start = text.find("`", cursor) + if start == -1: + break + opener_end = start + while opener_end < len(text) and text[opener_end] == "`": + opener_end += 1 + if _offset_in_spans(start, protected): + cursor = opener_end + continue + delimiter_length = opener_end - start + search = opener_end + matched = False + while search < len(text): + closer = text.find("`", search) + if closer == -1: + break + closer_end = closer + while closer_end < len(text) and text[closer_end] == "`": + closer_end += 1 + if ( + closer_end - closer == delimiter_length + and not _offset_in_spans(closer, protected) + ): + spans.append((start, closer_end)) + cursor = closer_end + matched = True + break + search = closer_end + if not matched: + cursor = opener_end + return spans + + +def _markdown_link_spans( + text: str, protected: list[tuple[int, int]] +) -> list[tuple[int, int]]: + spans: list[tuple[int, int]] = [] + cursor = 0 + while cursor < len(text): + label_start = text.find("[", cursor) + if label_start == -1: + break + if _offset_in_spans(label_start, protected): + cursor = label_start + 1 + continue + depth = 1 + position = label_start + 1 + while position < len(text) and depth: + if text[position] == "\\": + position += 2 + continue + if text[position] == "[": + depth += 1 + elif text[position] == "]": + depth -= 1 + position += 1 + if depth or position >= len(text) or text[position] != "(": + cursor = label_start + 1 + continue + destination_depth = 1 + position += 1 + while position < len(text) and destination_depth: + if text[position] == "\\": + position += 2 + continue + if text[position] == "(": + destination_depth += 1 + elif text[position] == ")": + destination_depth -= 1 + position += 1 + if destination_depth: + cursor = label_start + 1 + continue + start = label_start - 1 if label_start and text[label_start - 1] == "!" else label_start + spans.append((start, position)) + cursor = position + return spans + + +def _scan_markdown_label(text: str, start: int) -> tuple[str, int] | None: + if start >= len(text) or text[start] != "[": + return None + depth = 1 + position = start + 1 + while position < len(text) and depth: + if text[position] == "\\": + position += 2 + continue + if text[position] in "\r\n": + return None + if text[position] == "[": + depth += 1 + elif text[position] == "]": + depth -= 1 + position += 1 + if depth: + return None + return text[start + 1 : position - 1], position + + +def _reference_label_key(label: str) -> str: + unescaped = re.sub(r"\\(.)", r"\1", label) + return " ".join(unescaped.split()).casefold() + + +def _skip_spaces_tabs(text: str, position: int) -> int: + while position < len(text) and text[position] in " \t": + position += 1 + return position + + +def _after_line_ending(text: str, position: int) -> int | None: + if text.startswith("\r\n", position): + return position + 2 + if position < len(text) and text[position] in "\r\n": + return position + 1 + return None + + +def _scan_reference_destination(text: str, position: int) -> int | None: + if position >= len(text): + return None + if text[position] == "<": + position += 1 + while position < len(text): + if text[position] == "\\" and position + 1 < len(text): + position += 2 + continue + if text[position] == ">": + return position + 1 + if text[position] in "<>\r\n": + return None + position += 1 + return None + + start = position + depth = 0 + while position < len(text) and text[position] not in " \t\r\n": + if text[position] == "\\" and position + 1 < len(text): + position += 2 + continue + if text[position] == "(": + depth += 1 + elif text[position] == ")": + if depth == 0: + return None + depth -= 1 + position += 1 + return position if position > start and depth == 0 else None + + +def _scan_reference_title(text: str, position: int) -> int | None: + closer = {'"': '"', "'": "'", "(": ")"}.get( + text[position] if position < len(text) else "" + ) + if closer is None: + return None + position += 1 + while position < len(text): + if text[position] == "\\" and position + 1 < len(text): + position += 2 + continue + if text[position] == closer: + trailing = _skip_spaces_tabs(text, position + 1) + if trailing == len(text) or _after_line_ending(text, trailing) is not None: + return _line_end(text, trailing) + return None + next_line = _after_line_ending(text, position) + if next_line is not None: + content = _skip_spaces_tabs(text, next_line) + if content == len(text) or _after_line_ending(text, content) is not None: + return None + position = next_line + continue + position += 1 + return None + + +def _reference_line_content_start( + text: str, line_start: int, required_quote_depth: int | None = None +) -> tuple[int, int] | None: + content_end = line_start + len( + text[line_start : _line_end(text, line_start)].rstrip("\r\n") + ) + position = line_start + quote_depth = 0 + while True: + indent_start = position + while position < content_end and text[position] in " \t": + position += 1 + if position - indent_start > 3: + return None + if position >= content_end or text[position] != ">": + break + quote_depth += 1 + position += 1 + if position < content_end and text[position] in " \t": + position += 1 + if required_quote_depth is not None and quote_depth != required_quote_depth: + return None + return position, quote_depth + + +def _indented_code_spans( + text: str, protected: list[tuple[int, int]] +) -> list[tuple[int, int]]: + spans: list[tuple[int, int]] = [] + offset = 0 + while offset < len(text): + line_end = _line_end(text, offset) + content_end = offset + len(text[offset:line_end].rstrip("\r\n")) + position = offset + quote_depth = 0 + while True: + marker = position + spaces = 0 + while marker < content_end and text[marker] == " " and spaces < 3: + marker += 1 + spaces += 1 + if marker >= content_end or text[marker] != ">": + break + quote_depth += 1 + position = marker + 1 + if position < content_end and text[position] in " \t": + position += 1 + if quote_depth == 0: + position = offset + if ( + not _offset_in_spans(offset, protected) + and ( + text.startswith(" ", position) + or (position < content_end and text[position] == "\t") + ) + ): + spans.append((offset, line_end)) + offset = line_end + return spans + + +def _heading_spans( + text: str, protected: list[tuple[int, int]] +) -> list[tuple[int, int]]: + spans: list[tuple[int, int]] = [] + offset = 0 + while offset < len(text): + line_end = _line_end(text, offset) + line_content = _reference_line_content_start(text, offset) + if line_content is not None and not _offset_in_spans(offset, protected): + content_start, quote_depth = line_content + content = text[content_start:line_end].rstrip("\r\n") + heading = content[_list_content_start(content) :] + if re.match(r"#{1,6}(?:[ \t]+|$)", heading): + spans.append((offset, line_end)) + elif content.strip() and line_end < len(text): + underline_end = _line_end(text, line_end) + underline_content = _reference_line_content_start( + text, line_end, required_quote_depth=quote_depth + ) + if ( + underline_content is not None + and not _offset_in_spans(line_end, protected) + ): + underline_start, _ = underline_content + underline = text[underline_start:underline_end].rstrip("\r\n") + if re.fullmatch(r"(?:=+|-+)[ \t]*", underline): + spans.append((offset, underline_end)) + offset = line_end + return spans + + +def _scan_reference_definition( + text: str, line_start: int, protected: list[tuple[int, int]] +) -> tuple[str, int] | None: + content_end = line_start + len( + text[line_start : _line_end(text, line_start)].rstrip("\r\n") + ) + line_content = _reference_line_content_start(text, line_start) + if line_content is None: + return None + start, quote_depth = line_content + if _offset_in_spans(start, protected): + return None + label = _scan_markdown_label(text, start) + if label is None: + return None + value, position = label + if not value or position >= content_end or text[position] != ":": + return None + + position = _skip_spaces_tabs(text, position + 1) + next_line = _after_line_ending(text, position) + if next_line is not None: + continuation = _reference_line_content_start( + text, next_line, required_quote_depth=quote_depth + ) + if continuation is None: + return None + position, _ = continuation + destination_end = _scan_reference_destination(text, position) + if destination_end is None: + return None + block_end = _line_end(text, destination_end) + + title_start = _skip_spaces_tabs(text, destination_end) + if title_start < len(text) and text[title_start] not in "\r\n": + if title_start == destination_end: + return None + title_end = _scan_reference_title(text, title_start) + if title_end is None: + return None + block_end = title_end + else: + title_line = _after_line_ending(text, title_start) + if title_line is not None: + continuation = _reference_line_content_start( + text, title_line, required_quote_depth=quote_depth + ) + if continuation is not None: + possible_title, _ = continuation + title_end = _scan_reference_title(text, possible_title) + if title_end is not None: + block_end = title_end + return _reference_label_key(value), block_end + + +def _markdown_reference_spans( + text: str, protected: list[tuple[int, int]] +) -> list[tuple[int, int]]: + definitions: set[str] = set() + spans: list[tuple[int, int]] = [] + offset = 0 + while offset < len(text): + definition = _scan_reference_definition(text, offset, protected) + if definition is not None: + label, end = definition + definitions.add(label) + spans.append((offset, end)) + offset = _line_end(text, offset) + + cursor = 0 + while cursor < len(text): + start = text.find("[", cursor) + if start == -1: + break + if _offset_in_spans(start, protected): + cursor = start + 1 + continue + label = _scan_markdown_label(text, start) + if label is None: + cursor = start + 1 + continue + value, position = label + reference_key = _reference_label_key(value) + end = position + if position < len(text) and text[position] == "[": + reference = _scan_markdown_label(text, position) + if reference is None: + cursor = position + 1 + continue + reference_value, end = reference + reference_key = _reference_label_key(reference_value or value) + if value and reference_key in definitions: + link_start = start - 1 if start and text[start - 1] == "!" else start + spans.append((link_start, end)) + cursor = end + return spans + + +_MARKDOWN_AUTOLINK = re.compile( + r"<(?:" + r"[A-Za-z][A-Za-z0-9+.-]{1,31}:[^<>\x00-\x20]*" + r"|[A-Za-z0-9.!#$%&'*+/=?^_`{|}~-]+@" + r"[A-Za-z0-9](?:[A-Za-z0-9.-]*[A-Za-z0-9])?" + r")>" +) + + +def _markdown_autolink_spans( + text: str, protected: list[tuple[int, int]] +) -> list[tuple[int, int]]: + return [ + match.span() + for match in _MARKDOWN_AUTOLINK.finditer(text) + if not _offset_in_spans(match.start(), protected) + ] + + +_HTML_VOID_ELEMENTS = { + "area", + "base", + "br", + "col", + "embed", + "hr", + "img", + "input", + "link", + "meta", + "param", + "source", + "track", + "wbr", +} + +_HTML_LINKABLE_TEXT_ELEMENTS = {"b", "em", "i", "strong"} + + +def _scan_html_tag( + text: str, start: int +) -> tuple[str, int, bool, bool, bool] | None: + position = start + 1 + closing = position < len(text) and text[position] == "/" + if closing: + position += 1 + name_match = re.match(r"[A-Za-z][A-Za-z0-9:-]*", text[position:]) + if name_match is None: + return None + name = name_match.group(0).casefold() + position += name_match.end() + if position < len(text) and not ( + text[position].isspace() or text[position] in "/>" + ): + return None + + quote = "" + while position < len(text): + character = text[position] + if quote: + if character == quote: + quote = "" + elif character in ("'", '"'): + quote = character + elif character == ">": + before_close = text[start + 1 : position].rstrip() + self_closing = not closing and before_close.endswith("/") + return name, position + 1, closing, self_closing, True + position += 1 + return name, len(text), closing, False, False + + +def _html_spans( + text: str, protected: list[tuple[int, int]] +) -> list[tuple[int, int]]: + spans: list[tuple[int, int]] = [] + stack: list[tuple[str, int]] = [] + malformed_start: int | None = None + cursor = 0 + while cursor < len(text): + start = text.find("<", cursor) + if start == -1: + break + if _offset_in_spans(start, protected): + cursor = start + 1 + continue + if text.startswith("", start + 4) + if comment_end == -1: + spans.append((start, len(text))) + break + spans.append((start, comment_end + 3)) + cursor = comment_end + 3 + continue + tag = _scan_html_tag(text, start) + if tag is None: + cursor = start + 1 + continue + name, end, closing, self_closing, terminated = tag + spans.append((start, end)) + if not terminated: + break + if closing: + if stack and stack[-1][0] == name: + element_name, element_start = stack.pop() + if element_name not in _HTML_LINKABLE_TEXT_ELEMENTS: + spans.append((element_start, end)) + if malformed_start is not None and not stack: + spans.append((malformed_start, end)) + malformed_start = None + elif stack and malformed_start is None: + malformed_start = min(element_start for _, element_start in stack) + elif not self_closing and name not in _HTML_VOID_ELEMENTS: + stack.append((name, start)) + cursor = end + if stack or malformed_start is not None: + open_starts = [start for _, start in stack] + if malformed_start is not None: + open_starts.append(malformed_start) + spans.append((min(open_starts), len(text))) + return spans + + +def _merge_spans(spans: list[tuple[int, int]]) -> list[tuple[int, int]]: + merged: list[tuple[int, int]] = [] + for start, end in sorted((start, end) for start, end in spans if start < end): + if merged and start <= merged[-1][1]: + merged[-1] = (merged[-1][0], max(merged[-1][1], end)) + else: + merged.append((start, end)) + return merged + + +def _wiki_link_context( + text: str, protected: list[tuple[int, int]] +) -> tuple[list[tuple[int, int]], set[str]]: + spans: list[tuple[int, int]] = [] + targets: set[str] = set() + for match in re.finditer(r"\[\[([^\]\r\n]*)\]\]", text): + if _offset_in_spans(match.start(), protected): + continue + spans.append(match.span()) + target = match.group(1).split("|", 1)[0].split("#", 1)[0].strip() + if target: + targets.add(target.casefold()) + return spans, targets + + +def _protected_context(text: str) -> tuple[list[tuple[int, int]], set[str]]: + spans = _fenced_code_spans(text) + frontmatter = _frontmatter_span(text) + if frontmatter: + spans.append(frontmatter) + spans = _merge_spans(spans) + + spans = _merge_spans([*spans, *_indented_code_spans(text, spans)]) + spans = _merge_spans([*spans, *_inline_code_spans(text, spans)]) + spans = _merge_spans([*spans, *_markdown_autolink_spans(text, spans)]) + spans = _merge_spans([*spans, *_html_spans(text, spans)]) + spans = _merge_spans([*spans, *_markdown_link_spans(text, spans)]) + spans = _merge_spans([*spans, *_markdown_reference_spans(text, spans)]) + wiki_spans, targets = _wiki_link_context(text, spans) + spans = _merge_spans([*spans, *wiki_spans]) + high_priority_spans = spans + + headings = _heading_spans(text, spans) + spans = _merge_spans([*spans, *headings]) + html_tags = [ + match.span() + for match in re.finditer(r"<[^>\r\n]+>", text) + if not _offset_in_spans(match.start(), spans) + ] + spans = _merge_spans([*spans, *html_tags]) + urls = [ + match.span() + for match in re.finditer(r"https?://[^\s<>\]\)]+", text) + if not _offset_in_spans(match.start(), spans) + ] + spans = _merge_spans([*spans, *urls]) + references = next( + ( + match + for match in REFERENCE_HEADING_RE.finditer(text) + if not _offset_in_spans(match.start(), high_priority_spans) + ), + None, + ) + if references: + spans.append((references.start(), len(text))) + return _merge_spans(spans), targets + + +def protected_spans(text: str) -> list[tuple[int, int]]: + spans, _ = _protected_context(text) + return spans + + +def _overlaps(start: int, end: int, spans: list[tuple[int, int]]) -> bool: + return any(start < protected_end and end > protected_start for protected_start, protected_end in spans) + + +def _normalized_mappings(mappings: list[dict[str, Any]]) -> list[dict[str, Any]]: + normalized: list[dict[str, Any]] = [] + by_target: dict[str, dict[str, Any]] = {} + seen_forms: dict[str, set[str]] = {} + for mapping in mappings: + if not isinstance(mapping, dict): + raise SystemExit("Writer results must contain objects.") + link_stem = str(mapping.get("link_stem", "")).strip() + forms = mapping.get("forms", []) + if not link_stem or not isinstance(forms, list): + raise SystemExit("Writer result requires link_stem and forms.") + target_key = link_stem.casefold() + item = by_target.get(target_key) + if item is None: + item = {"link_stem": link_stem, "forms": []} + by_target[target_key] = item + seen_forms[target_key] = set() + normalized.append(item) + for value in forms: + if not isinstance(value, str) or not value.strip(): + raise SystemExit(f"Writer result has an invalid form for {link_stem}.") + form = value.strip() + key = form.casefold() + if key not in seen_forms[target_key]: + seen_forms[target_key].add(key) + item["forms"].append(form) + if not item["forms"]: + raise SystemExit(f"Writer result has no forms for {link_stem}.") + return normalized + + +def _write_atomically(path: Path, data: bytes, original: bytes) -> None: + descriptor, temp_name = tempfile.mkstemp(prefix=f".{path.name}.", suffix=".tmp", dir=path.parent) + temp_path = Path(temp_name) + replaced = False + try: + with os.fdopen(descriptor, "wb") as handle: + handle.write(data) + handle.flush() + os.fsync(handle.fileno()) + if path.read_bytes() != original: + raise SystemExit(f"Article changed while linking: {path}") + os.replace(temp_path, path) + replaced = True + finally: + if not replaced: + temp_path.unlink(missing_ok=True) + + +def link_article_terms( + article: Path, mappings: list[dict[str, Any]], expected_sha256: str +) -> dict[str, Any]: + path = article.expanduser().resolve() + original = path.read_bytes() + actual_sha256 = hashlib.sha256(original).hexdigest() + if actual_sha256 != expected_sha256: + raise SystemExit(f"Article changed since preview: {path}") + + had_bom = original.startswith(b"\xef\xbb\xbf") + text = original[3 if had_bom else 0 :].decode("utf-8") + protected, targets = _protected_context(text) + normalized = _normalized_mappings(mappings) + planned: list[tuple[int, int, str]] = [] + planned_targets: set[str] = set() + statuses: dict[int, str] = {} + + order = sorted( + range(len(normalized)), + key=lambda index: max(len(form) for form in normalized[index]["forms"]), + reverse=True, + ) + for index in order: + mapping = normalized[index] + link_stem = mapping["link_stem"] + target_key = link_stem.casefold() + if target_key in targets or target_key in planned_targets: + statuses[index] = "already_linked" + continue + matched = None + for form in sorted(mapping["forms"], key=len, reverse=True): + for candidate in term_pattern(form).finditer(text): + if _overlaps(candidate.start(), candidate.end(), protected): + continue + if _overlaps(candidate.start(), candidate.end(), [(start, end) for start, end, _ in planned]): + continue + matched = candidate + break + if matched is not None: + break + if matched is None: + statuses[index] = "not_found" + continue + surface = matched.group(0) + replacement = f"[[{link_stem}]]" if surface == link_stem else f"[[{link_stem}|{surface}]]" + planned.append((matched.start(), matched.end(), replacement)) + planned_targets.add(target_key) + statuses[index] = "linked" + + if planned: + updated = text + for start, end, replacement in sorted(planned, reverse=True): + updated = updated[:start] + replacement + updated[end:] + data = (b"\xef\xbb\xbf" if had_bom else b"") + updated.encode("utf-8") + _write_atomically(path, data, original) + + results = [ + {"link_stem": mapping["link_stem"], "status": statuses[index]} + for index, mapping in enumerate(normalized) + ] + summary = {status: sum(item["status"] == status for item in results) for status in ("linked", "already_linked", "not_found")} + return {"status": "ok", "article_path": str(path), "results": results, "summary": summary} + + +def _validated_writer_results( + path: Path, + terms_dir: Path, + article_path: str, + source_manifest: str, + raw_sections: str, + reviewed_shortlist: str, + triage_artifact: str, +) -> list[dict[str, Any]]: + payload = load_json_file(path) + results = payload.get("results") + if ( + payload.get("status") != "ok" + or payload.get("script") != "write_glossary_terms.py" + or not isinstance(results, list) + ): + raise SystemExit("Linker requires a successful writer result with results.") + provenance = payload.get("provenance") + context = payload.get("context") + if not isinstance(provenance, dict) or not isinstance(context, dict): + raise SystemExit("Writer result requires provenance and article context.") + current = validate_authorized_selection( + payload, + results, + triage_artifact, + source_manifest, + raw_sections, + reviewed_shortlist, + term_field="name", + forms_field="forms", + ) + triage_digest = payload.get("triage_sha256") + if triage_digest != current["triage_sha256"]: + raise SystemExit("Writer triage identity does not match the authorized artifact.") + if payload.get("mappings_sha256") != writer_mappings_sha256( + provenance, results, context, triage_digest + ): + raise SystemExit("Writer mapping digest does not match the ordered mappings.") + expected_context = { + "paper_id": current["paper_id"], + "paper_link": Path(article_path).stem, + "article_path": article_path, + } + if not context.get("article_path"): + raise SystemExit("Linker requires a writer artifact with bound article context.") + if context != expected_context: + raise SystemExit("Writer article context does not match the explicit article.") + + root = terms_dir.resolve(strict=True) + index = build_alias_index(root) + seen_paths: set[Path] = set() + seen_stems: set[str] = set() + for item in results: + file_value = item.get("file") + link_stem = item.get("link_stem") + forms = item.get("forms") + if ( + not isinstance(file_value, str) + or not file_value + or not isinstance(link_stem, str) + or not link_stem + or not isinstance(forms, list) + or not forms + ): + raise SystemExit("Writer mapping requires file, link_stem, and forms.") + unresolved = Path(file_value).expanduser() + try: + resolved = unresolved.resolve(strict=True) + except OSError as exc: + raise SystemExit(f"Writer mapping glossary file is invalid: {file_value}: {exc}") from None + if ( + not unresolved.is_absolute() + or file_value != str(resolved) + or not resolved.is_file() + or resolved.suffix.lower() != ".md" + or not resolved.is_relative_to(root) + ): + raise SystemExit(f"Writer mapping file is outside the configured glossary: {file_value}") + if link_stem != resolved.stem: + raise SystemExit("Writer mapping link_stem does not match its glossary file stem.") + if resolved in seen_paths or link_stem.casefold() in seen_stems: + raise SystemExit("Writer mappings must have unique glossary files and stems.") + seen_paths.add(resolved) + seen_stems.add(link_stem.casefold()) + for form in forms: + if not isinstance(form, str) or not form: + raise SystemExit("Writer mapping forms must be non-empty strings.") + owner = index.get(_normalized_key(form)) + if owner is None or not owner.samefile(resolved): + raise SystemExit( + f"Writer mapping form does not resolve to its glossary file: {form}" + ) + return results + + +def main() -> None: + started = perf_counter() + args = parser().parse_args() + config = load_config(Path(args.config_path)) if args.config_path else load_config() + if config is None: + raise SystemExit("Paper-glossary is not configured; configure a term directory first.") + terms_dir = resolve_terms_dir(config) + article_info = validate_article(Path(args.input), config) + article = Path(article_info["article_path"]) + result = link_article_terms( + article, + _validated_writer_results( + Path(args.write_result), + terms_dir, + article_info["article_path"], + args.source_manifest, + args.raw_sections, + args.reviewed_shortlist, + args.triage, + ), + args.expected_sha256, + ) + emit( + { + **result, + "script": "link_glossary_terms.py", + "elapsed_ms": elapsed_ms(started), + }, + args.output, + ) + + +if __name__ == "__main__": + main() diff --git a/skills/paper-glossary/scripts/lint_glossary.py b/skills/paper-glossary/scripts/lint_glossary.py index e9ff719..6ed781c 100644 --- a/skills/paper-glossary/scripts/lint_glossary.py +++ b/skills/paper-glossary/scripts/lint_glossary.py @@ -6,6 +6,7 @@ import argparse import re from pathlib import Path +from time import perf_counter from typing import Any from glossary_contracts import ( @@ -16,7 +17,7 @@ GLOSSARY_LABEL_DEFINITION, GLOSSARY_OCCURRENCE_HEADING, ) -from glossary_common import emit +from glossary_common import elapsed_ms, emit DISCLAIMER_KEYPHRASE = "非某篇论文" OCCURRENCE_LINK_RE = re.compile(r"(?m)^\s*-\s*\[\[[^\]]+\]\]") @@ -24,7 +25,12 @@ def parser() -> argparse.ArgumentParser: p = argparse.ArgumentParser(description=__doc__ or "lint glossary notes") - p.add_argument("--input", default="", help="Single term note Markdown path.") + p.add_argument( + "--input", + action="append", + default=[], + help="Term note Markdown path. Repeat for multiple changed notes.", + ) p.add_argument("--terms-dir", default="", help="Folder of term notes to lint.") p.add_argument("--output", default="", help="Output JSON path.") return p @@ -88,15 +94,27 @@ def lint_term_file_text(text: str) -> dict[str, Any]: def main() -> None: + started = perf_counter() args = parser().parse_args() if not args.input and not args.terms_dir: raise SystemExit("lint_glossary.py requires --input or --terms-dir.") files: list[Path] = [] - if args.input: - files.append(Path(args.input).expanduser().resolve()) + seen: set[Path] = set() + for value in args.input: + path = Path(value).expanduser().resolve() + if path not in seen: + seen.add(path) + files.append(path) if args.terms_dir: - files.extend(sorted(Path(args.terms_dir).expanduser().resolve().glob("*.md"))) + terms_dir = Path(args.terms_dir).expanduser().resolve() + if not terms_dir.is_dir(): + raise SystemExit(f"--terms-dir must be an existing directory: {terms_dir}") + for path in sorted(terms_dir.glob("*.md")): + path = path.resolve() + if path not in seen: + seen.add(path) + files.append(path) if not files: raise SystemExit("No glossary markdown files found.") @@ -113,6 +131,7 @@ def main() -> None: "passes_glossary": all(result["passes"] for result in results), "files": results, "summary": {"total": len(results), "passed": passed, "failed": len(results) - passed}, + "elapsed_ms": elapsed_ms(started), }, args.output, ) diff --git a/skills/paper-glossary/scripts/plan_glossary.py b/skills/paper-glossary/scripts/plan_glossary.py index 7a0a8bf..956f670 100644 --- a/skills/paper-glossary/scripts/plan_glossary.py +++ b/skills/paper-glossary/scripts/plan_glossary.py @@ -6,15 +6,51 @@ import argparse import json import re +import unicodedata +from functools import lru_cache from pathlib import Path +from time import perf_counter from typing import Any -from glossary_common import emit, maybe_load_json_record, normalize_whitespace +from glossary_common import ( + canonical_sha256, + elapsed_ms, + emit, + load_json_file, + maybe_load_json_record, + normalize_whitespace, + proposal_sha256, + review_sha256, + selection_sha256, +) +from link_glossary_terms import ( + _fenced_code_spans, + _inline_code_spans, + _markdown_link_spans, + _scan_markdown_label, +) NON_EVIDENCE_SECTION_KINDS = frozenset({"references"}) SNIPPET_RADIUS = 70 MAX_ANCHORS_PER_TERM = 3 -MAX_CANDIDATES = 60 +MAX_EMPHASIS_WORDS = 6 +POOL_BUFFER = 10 + +EMPHASIS_CONNECTORS = frozenset( + {"and", "by", "for", "from", "in", "of", "on", "or", "the", "to", "via", "with"} +) +CHINESE_SENTENCE_PREFIXES = ( + "我们", + "本文", + "本研究", + "本方法", + "本模型", + "该方法", + "该模型", + "这种方法", + "这种模型", +) +CHINESE_SENTENCE_PREDICATES = ("提出", "表明", "证明", "显示", "可以", "能够") TOKEN_RE = re.compile(r"[A-Za-z][A-Za-z0-9]*(?:[+\-][A-Za-z0-9]+)*") CANDIDATE_STOPWORDS = frozenset({"ID", "OK", "OOD", "DOI", "URL", "PDF"}) @@ -24,6 +60,22 @@ TITLECASE_RE = re.compile(r"[A-Z][a-z]{2,}(?:[ -][A-Z][a-z]+){1,3}") KEYWORDS_LINE_RE = re.compile(r"(?im)^[ \t]*(?:keywords|关键词)[ \t]*[:：·]?[ \t]*(.+)$") GREEK_RE = re.compile(r"[α-ωΑ-Ω]") +CJK_RE = re.compile(r"[\u3400-\u9fff]") +CAPITALIZED_TERM_TOKEN_RE = re.compile(r"[A-Z0-9][A-Za-z0-9+./-]*") +URL_RE = re.compile(r"https?://\S+") +REFERENCE_TAIL_RE = re.compile( + r"(?ims)^#{1,6}\s*(?:references|参考文献|引用)\s*$.*\Z" +) +EMPHASIS_RE = re.compile( + r"(?:(?:\*\*|__)([^\n*_]{2,80})(?:\*\*|__)|" + r"(?:\*|_)([^\n*_]{2,80})(?:\*|_))" +) +STRONG_RE = re.compile(r"(?is)([^<]{2,80})") +UNRESOLVED_SELECTION_RE = re.compile( + r"^\s*[#（(\[]?\s*\d+\s*[）)\]]?[.。]?" + r"(?:\s*[-,，、;；/\s]\s*[#（(\[]?\s*\d+\s*[）)\]]?[.。]?)*\s*$" +) +WRITE_ALL_SELECTION_RE = re.compile(r"^全部写入\s*[。.!！]?\s*$") TITLECASE_STOP_HEADS = frozenset( { "The", @@ -65,20 +117,37 @@ ) CATEGORY_ORDER = { "keyword": 0, - "full-name": 1, - "acronym-or-model": 2, - "term-phrase": 3, - "symbol": 4, + "emphasis": 1, + "full-name": 2, + "acronym-or-model": 3, + "term-phrase": 4, + "symbol": 5, } def parser() -> argparse.ArgumentParser: p = argparse.ArgumentParser(description=__doc__ or "plan glossary") - p.add_argument("--propose", action="store_true", help="List candidate glossary terms.") + mode = p.add_mutually_exclusive_group() + mode.add_argument("--propose", action="store_true", help="List candidate glossary terms.") + mode.add_argument( + "--review-proposal", + default="", + help="Saved proposal JSON path to bind a host-reviewed shortlist.", + ) + mode.add_argument( + "--reviewed-shortlist", + default="", + help="Saved reviewed-shortlist JSON path required for triage.", + ) p.add_argument( "--terms", default="", - help="Selected terms as JSON list, file path, comma/newline string, or pipe aliases.", + help="Exact selected names from a reviewed shortlist.", + ) + p.add_argument( + "--reviewed-terms", + default="", + help="Exact host-reviewed proposal names in display order.", ) p.add_argument("--source-manifest", required=True, help="Manifest JSON path or JSON object.") p.add_argument( @@ -90,19 +159,59 @@ def parser() -> argparse.ArgumentParser: return p +def _load_exact_names( + value: str, label: str, *, allow_empty: bool = False +) -> list[str]: + raw = value.strip() + if not raw: + raise SystemExit(f"{label} requires at least one exact candidate name.") + path = Path(raw).expanduser() + text = path.read_text(encoding="utf-8-sig").strip() if path.is_file() else raw + parsed = _maybe_json_list(text) if text.startswith("[") else None + items: list[Any] = parsed if parsed is not None else re.split(r"[,\r\n]+", text) + names: list[str] = [] + for item in items: + if not isinstance(item, str): + raise SystemExit(f"{label} must contain exact candidate name strings.") + if not item or normalize_whitespace(item) != item: + raise SystemExit(f"{label} contains a malformed exact candidate name.") + names.append(item) + if not names and not allow_empty: + raise SystemExit(f"{label} requires at least one exact candidate name.") + _reject_unresolved_selection(names) + return names + + def load_terms(value: str) -> list[str]: raw = value.strip() + _reject_unresolved_selection([raw]) parsed = _maybe_json_list(raw) if parsed is not None: - return _clean_terms(parsed) + terms = _clean_terms(parsed) + _reject_unresolved_selection(terms) + return terms path = Path(raw).expanduser() if path.exists() and path.is_file(): text = path.read_text(encoding="utf-8-sig") parsed = _maybe_json_list(text.strip()) if parsed is not None: - return _clean_terms(parsed) - return _clean_terms(re.split(r"[\r\n]+", text)) - return _clean_terms(re.split(r"[,\r\n]+", raw)) + terms = _clean_terms(parsed) + else: + terms = _clean_terms(re.split(r"[\r\n]+", text)) + else: + terms = _clean_terms(re.split(r"[,\r\n]+", raw)) + _reject_unresolved_selection(terms) + return terms + + +def _reject_unresolved_selection(terms: list[str]) -> None: + if any( + WRITE_ALL_SELECTION_RE.fullmatch(term) or UNRESOLVED_SELECTION_RE.fullmatch(term) + for term in terms + ): + raise SystemExit( + "Resolve shortlist numbers or 全部写入 to candidate names before passing --terms." + ) def _maybe_json_list(text: str) -> list[Any] | None: @@ -195,6 +304,57 @@ def _evidence_records(records: list[dict[str, Any]]) -> list[dict[str, Any]]: ] +def _mask_span(characters: list[str], start: int, end: int) -> None: + for index in range(start, end): + if characters[index] not in "\r\n": + characters[index] = " " + + +@lru_cache(maxsize=256) +def _candidate_prose_with_offsets(text: str) -> str: + characters = list(text) + fenced = _fenced_code_spans(text) + inline = _inline_code_spans(text, fenced) + protected = sorted([*fenced, *inline]) + links = _markdown_link_spans(text, protected) + for start, end in links: + label_start = start + 1 if text[start] == "!" else start + label = _scan_markdown_label(text, label_start) + if label is None: + _mask_span(characters, start, end) + continue + _, label_end = label + _mask_span(characters, start, label_start + 1) + _mask_span(characters, label_end - 1, end) + for start, end in [*fenced, *inline]: + _mask_span(characters, start, end) + for pattern in (URL_RE, REFERENCE_TAIL_RE): + for match in pattern.finditer(text): + _mask_span(characters, match.start(), match.end()) + return "".join(characters) + + +def _candidate_prose(text: str) -> str: + return _candidate_prose_with_offsets(text) + + +def effective_prose_text(records: list[dict[str, Any]]) -> str: + return "\n".join( + _candidate_prose(str(record.get("text", ""))) + for record in _evidence_records(records) + ) + + +def shortlist_limit(effective_characters: int) -> int: + if effective_characters < 10_000: + return 10 + if effective_characters < 30_000: + return 18 + if effective_characters < 60_000: + return 25 + return 35 + + def _term_pattern(term: str) -> re.Pattern[str]: if re.fullmatch(r"[\x00-\x7f]+", term): return re.compile(rf"(? str: return prefix + normalize_whitespace(text[left:right]) + suffix -def find_occurrences(term: str, records: list[dict[str, Any]]) -> tuple[int, list[dict[str, Any]]]: - pattern = _term_pattern(term) +def _non_overlapping_matches(forms: list[str], text: str) -> list[tuple[int, int]]: + spans = { + (match.start(), match.end()) + for form in forms + for match in _term_pattern(form).finditer(text) + } + selected: list[tuple[int, int]] = [] + for start, end in sorted( + spans, key=lambda span: (span[0], -(span[1] - span[0]), span[1]) + ): + if any( + start < selected_end and selected_start < end + for selected_start, selected_end in selected + ): + continue + selected.append((start, end)) + return selected + + +def _find_occurrences_for_forms( + forms: list[str], records: list[dict[str, Any]] +) -> tuple[int, list[dict[str, Any]]]: total = 0 anchors: list[dict[str, Any]] = [] for record in _evidence_records(records): text = str(record.get("text", "")) - matches = list(pattern.finditer(text)) + matches = _non_overlapping_matches(forms, _candidate_prose_with_offsets(text)) if not matches: continue total += len(matches) if len(anchors) < MAX_ANCHORS_PER_TERM: - first = matches[0] + first_start, first_end = matches[0] anchors.append( { "section_id": record.get("section_id", ""), "title": record.get("title", ""), "page_start": record.get("page_start"), "page_end": record.get("page_end"), - "snippet": _snippet(text, first.start(), first.end()), + "snippet": _snippet(text, first_start, first_end), } ) return total, anchors +def find_occurrences(term: str, records: list[dict[str, Any]]) -> tuple[int, list[dict[str, Any]]]: + return _find_occurrences_for_forms([term], records) + + def triage_terms(terms: list[str], records: list[dict[str, Any]]) -> list[dict[str, Any]]: - results: list[dict[str, Any]] = [] - for raw in terms: - forms = [form for form in (normalize_whitespace(form) for form in raw.split("|")) if form] - if not forms: - continue - occurrences = 0 - anchors: list[dict[str, Any]] = [] - seen_sections: set[str] = set() - for form in forms: - form_occurrences, form_anchors = find_occurrences(form, records) - occurrences += form_occurrences - for anchor in form_anchors: - section_id = str(anchor.get("section_id", "")) - if section_id not in seen_sections and len(anchors) < MAX_ANCHORS_PER_TERM: - seen_sections.add(section_id) - anchors.append(anchor) - found = occurrences > 0 - results.append( - { - "term": forms[0], - "surface_forms": forms, - "found_in_paper": found, - "occurrences": occurrences, - "routing": "anchor_only" if found else "needs_explanation", - "paper_anchors": anchors, - } - ) - return results + candidates: list[dict[str, Any]] = [] + for term in terms: + if ( + not isinstance(term, str) + or not term + or normalize_whitespace(term) != term + or "|" in term + ): + raise SystemExit("Triage terms must contain exact names without alias syntax.") + candidates.append({"term": term, "surface_forms": [term]}) + return _triage_candidates(candidates, records) def _is_acronym(token: str) -> bool: @@ -284,38 +451,99 @@ def _extract_keywords(records: list[dict[str, Any]]) -> list[str]: return [] +def _english_emphasis_term_shape(candidate: str) -> bool: + words = candidate.split() + if len(words) == 1: + return True + if words[0].casefold() in EMPHASIS_CONNECTORS or words[-1].casefold() in EMPHASIS_CONNECTORS: + return False + return all( + word.casefold() in EMPHASIS_CONNECTORS + or CAPITALIZED_TERM_TOKEN_RE.fullmatch(word) + or CJK_RE.search(word) + for word in words + ) + + +def _chinese_emphasis_term_shape(candidate: str) -> bool: + compact = "".join(candidate.split()) + for prefix in CHINESE_SENTENCE_PREFIXES: + if compact.startswith(prefix) and any( + predicate in compact[len(prefix) :] + for predicate in CHINESE_SENTENCE_PREDICATES + ): + return False + return True + + +def _emphasis_candidate(value: str) -> str: + candidate = normalize_whitespace(value) + if ( + not candidate + or len(candidate) > 80 + or len(candidate.split()) > MAX_EMPHASIS_WORDS + or re.search(r"[.!?。！？,:;；]", candidate) + or not re.search(r"[A-Za-z\u0080-\uffff]", candidate) + ): + return "" + if re.search(r"[A-Za-z]", candidate) and not _english_emphasis_term_shape(candidate): + return "" + if CJK_RE.search(candidate) and not _chinese_emphasis_term_shape(candidate): + return "" + return candidate + + def propose_candidates(records: list[dict[str, Any]]) -> list[dict[str, Any]]: evidence = _evidence_records(records) - categories: dict[str, str] = {} + cleaned_evidence = [ + {**record, "text": _candidate_prose(str(record.get("text", "")))} + for record in evidence + ] + categories: dict[str, tuple[str, str, list[str]]] = {} def add(term: str, category: str) -> None: cleaned = normalize_whitespace(term) - if cleaned and cleaned not in categories: - categories[cleaned] = category - - for keyword in _extract_keywords(records): + if not cleaned: + return + key = unicodedata.normalize("NFKC", cleaned).casefold() + existing = categories.get(key) + if existing is None: + categories[key] = (cleaned, category, [cleaned]) + elif cleaned not in existing[2]: + existing[2].append(cleaned) + + for keyword in _extract_keywords(cleaned_evidence): add(keyword, "keyword") - for record in evidence: + for record in cleaned_evidence: + for match in EMPHASIS_RE.finditer(str(record.get("text", ""))): + candidate = _emphasis_candidate(match.group(1) or match.group(2) or "") + if candidate: + add(candidate, "emphasis") + for match in STRONG_RE.finditer(str(record.get("text", ""))): + candidate = _emphasis_candidate(match.group(1)) + if candidate: + add(candidate, "emphasis") + for record in cleaned_evidence: for match in PAREN_EXPANSION_RE.finditer(str(record.get("text", ""))): if _titlecase_ok(match.group(1)): add(match.group(1), "full-name") if _is_acronym(match.group(2)): add(match.group(2), "acronym-or-model") - for record in evidence: + for record in cleaned_evidence: for match in TOKEN_RE.finditer(str(record.get("text", ""))): if _is_acronym(match.group(0)): add(match.group(0), "acronym-or-model") - for record in evidence: + for record in cleaned_evidence: for match in TITLECASE_RE.finditer(str(record.get("text", ""))): if _titlecase_ok(match.group(0)): add(match.group(0), "term-phrase") - for record in evidence: + for record in cleaned_evidence: for match in GREEK_RE.finditer(str(record.get("text", ""))): add(match.group(0), "symbol") results: list[dict[str, Any]] = [] - for term, category in categories.items(): - occurrences, anchors = find_occurrences(term, records) + for term, category, surface_forms in categories.values(): + occurrences, anchors = _find_occurrences_for_forms(surface_forms, records) if category == "term-phrase" and occurrences < 2: continue if occurrences == 0 and category not in ("keyword", "symbol"): @@ -324,6 +552,7 @@ def add(term: str, category: str) -> None: results.append( { "term": term, + "surface_forms": surface_forms, "category": category, "occurrences": occurrences, "section_id": anchor.get("section_id", ""), @@ -338,45 +567,249 @@ def add(term: str, category: str) -> None: str(item["term"]).lower(), ) ) - return results[:MAX_CANDIDATES] + effective_characters = sum( + 1 for character in effective_prose_text(records) if not character.isspace() + ) + return results[: shortlist_limit(effective_characters) + POOL_BUFFER] + + +def _proposal_summary( + records: list[dict[str, Any]], candidates: list[dict[str, Any]] +) -> dict[str, int]: + effective_characters = sum( + 1 for character in effective_prose_text(records) if not character.isspace() + ) + return { + "effective_body_characters": effective_characters, + "shortlist_limit": shortlist_limit(effective_characters), + "pool_candidates": len(candidates), + } + + +def _proposal_artifact( + manifest: dict[str, Any], records: list[dict[str, Any]] +) -> dict[str, Any]: + paper_id = str(manifest.get("paper_id", "")) + candidates = propose_candidates(records) + summary = _proposal_summary(records, candidates) + source_sha256 = canonical_sha256( + {"paper_id": paper_id, "manifest": manifest, "records": records} + ) + proposal_digest = proposal_sha256(paper_id, source_sha256, candidates, summary) + proposal_provenance = { + "paper_id": paper_id, + "source_sha256": source_sha256, + "candidates": candidates, + "summary": summary, + "proposal_sha256": proposal_digest, + } + has_candidates = bool(candidates) + return { + "status": "ok", + "script": "plan_glossary.py", + "mode": "propose", + "workflow_state": "awaiting_semantic_review" if has_candidates else "no_candidates", + "next_action": ( + "record_reviewed_shortlist_then_present_and_wait" + if has_candidates + else "report_no_candidates" + ), + "paper_id": paper_id, + "candidates": candidates, + "summary": summary, + "provenance": proposal_provenance, + } + + +def _validated_proposal( + proposal: dict[str, Any], manifest: dict[str, Any], records: list[dict[str, Any]] +) -> dict[str, Any]: + expected = _proposal_artifact(manifest, records) + fields = ( + "status", + "script", + "mode", + "workflow_state", + "next_action", + "paper_id", + "candidates", + "summary", + "provenance", + ) + if any(proposal.get(field) != expected[field] for field in fields): + raise SystemExit("Invalid proposal artifact for the current paper and source data.") + return expected + + +def _review_artifact( + proposal: dict[str, Any], reviewed_names: list[str] +) -> dict[str, Any]: + if len(reviewed_names) != len(set(reviewed_names)): + raise SystemExit("Host-reviewed candidate names must not contain duplicates.") + limit = int(proposal["summary"]["shortlist_limit"]) + if len(reviewed_names) > limit: + raise SystemExit(f"Host-reviewed candidate names exceed shortlist limit {limit}.") + by_name = {candidate["term"]: candidate for candidate in proposal["candidates"]} + unknown = [name for name in reviewed_names if name not in by_name] + if unknown: + raise SystemExit( + f"Host-reviewed name must exactly match a proposal term: {unknown[0]}" + ) + reviewed = [by_name[name] for name in reviewed_names] + proposal_provenance = proposal["provenance"] + review_digest = review_sha256(proposal_provenance, reviewed) + has_reviewed_candidates = bool(reviewed) + return { + "status": "ok", + "script": "plan_glossary.py", + "mode": "review", + "workflow_state": ( + "awaiting_user_selection" if has_reviewed_candidates else "no_candidates" + ), + "next_action": ( + "present_numbered_shortlist_and_wait" + if has_reviewed_candidates + else "report_no_candidates" + ), + "paper_id": proposal["paper_id"], + "proposal_provenance": proposal_provenance, + "reviewed_shortlist": reviewed, + "summary": {"shortlist_limit": limit, "reviewed_candidates": len(reviewed)}, + "provenance": { + **proposal_provenance, + "reviewed_shortlist": reviewed, + "review_sha256": review_digest, + }, + } + + +def _validated_review( + review: dict[str, Any], manifest: dict[str, Any], records: list[dict[str, Any]] +) -> tuple[dict[str, Any], dict[str, Any]]: + proposal = _proposal_artifact(manifest, records) + reviewed = review.get("reviewed_shortlist") + if not isinstance(reviewed, list) or any(not isinstance(item, dict) for item in reviewed): + raise SystemExit("Invalid reviewed-shortlist artifact.") + names = [item.get("term") for item in reviewed] + if any(not isinstance(name, str) for name in names): + raise SystemExit("Invalid reviewed candidate name.") + expected = _review_artifact(proposal, names) + fields = ( + "status", + "script", + "mode", + "workflow_state", + "next_action", + "paper_id", + "proposal_provenance", + "reviewed_shortlist", + "summary", + "provenance", + ) + if any(review.get(field) != expected[field] for field in fields): + raise SystemExit("Invalid reviewed-shortlist artifact identity or proposal binding.") + return expected, proposal + + +def _triage_candidates( + candidates: list[dict[str, Any]], records: list[dict[str, Any]] +) -> list[dict[str, Any]]: + results: list[dict[str, Any]] = [] + for candidate in candidates: + forms = candidate["surface_forms"] + occurrences, anchors = _find_occurrences_for_forms(forms, records) + found = occurrences > 0 + results.append( + { + "term": candidate["term"], + "surface_forms": forms, + "found_in_paper": found, + "occurrences": occurrences, + "routing": "anchor_only" if found else "needs_explanation", + "paper_anchors": anchors, + } + ) + return results def main() -> None: + started = perf_counter() args = parser().parse_args() - if not args.propose and not args.terms.strip(): - raise SystemExit("plan_glossary.py requires --propose or --terms.") + if not (args.propose or args.review_proposal or args.reviewed_shortlist): + if args.terms.strip(): + raise SystemExit("Triage requires --reviewed-shortlist with exact selected names.") + raise SystemExit( + "plan_glossary.py requires --propose, --review-proposal, or --reviewed-shortlist." + ) manifest, records = load_manifest_and_sections(args.source_manifest, args.raw_sections) - paper_id = manifest.get("paper_id", "") if args.propose: - candidates = propose_candidates(records) + if args.terms.strip() or args.reviewed_terms.strip(): + raise SystemExit("Proposal mode does not accept selected terms.") + emit({**_proposal_artifact(manifest, records), "elapsed_ms": elapsed_ms(started)}, args.output) + return + + if args.review_proposal: + if args.terms.strip() or not args.reviewed_terms.strip(): + raise SystemExit("Review mode requires --reviewed-terms and does not accept --terms.") + proposal = _validated_proposal( + load_json_file(Path(args.review_proposal).expanduser().resolve()), + manifest, + records, + ) + reviewed_names = _load_exact_names( + args.reviewed_terms, "--reviewed-terms", allow_empty=True + ) emit( { - "status": "ok", - "script": "plan_glossary.py", - "mode": "propose", - "paper_id": paper_id, - "candidates": candidates, - "summary": {"total_candidates": len(candidates)}, + **_review_artifact(proposal, reviewed_names), + "elapsed_ms": elapsed_ms(started), }, args.output, ) return - triaged = triage_terms(load_terms(args.terms), records) + if not args.terms.strip() or args.reviewed_terms.strip(): + raise SystemExit("Triage requires --terms and does not accept --reviewed-terms.") + review, proposal = _validated_review( + load_json_file(Path(args.reviewed_shortlist).expanduser().resolve()), + manifest, + records, + ) + selected_names = _load_exact_names(args.terms, "--terms") + if len(selected_names) != len(set(selected_names)): + raise SystemExit("Selected shortlist names must not contain duplicates.") + reviewed_by_name = { + candidate["term"]: candidate for candidate in review["reviewed_shortlist"] + } + unknown = [name for name in selected_names if name not in reviewed_by_name] + if unknown: + raise SystemExit( + f"Selected name must exactly match the reviewed shortlist: {unknown[0]}" + ) + selected_candidates = [reviewed_by_name[name] for name in selected_names] + triaged = _triage_candidates(selected_candidates, records) anchor_only = sum(1 for item in triaged if item["routing"] == "anchor_only") emit( { "status": "ok", "script": "plan_glossary.py", "mode": "triage", - "paper_id": paper_id, + "paper_id": proposal["paper_id"], "terms": triaged, + "provenance": { + "proposal": proposal["provenance"], + "review": review["provenance"], + "selection_sha256": selection_sha256( + review["provenance"]["review_sha256"], triaged + ), + }, "summary": { "total": len(triaged), "anchor_only": anchor_only, "needs_explanation": len(triaged) - anchor_only, }, + "elapsed_ms": elapsed_ms(started), }, args.output, ) diff --git a/skills/paper-glossary/scripts/write_glossary_terms.py b/skills/paper-glossary/scripts/write_glossary_terms.py index 9f03838..0cd8526 100644 --- a/skills/paper-glossary/scripts/write_glossary_terms.py +++ b/skills/paper-glossary/scripts/write_glossary_terms.py @@ -5,34 +5,91 @@ import argparse import json +import os import re +import tempfile from pathlib import Path +from time import perf_counter from typing import Any -from glossary_common import emit, maybe_load_json_record, normalize_whitespace +from glossary_common import ( + elapsed_ms, + emit, + maybe_load_json_record, + normalize_whitespace, + validate_authorized_selection, + validate_provenance_chain, + writer_mappings_sha256, +) +from glossary_config import load_config, resolve_terms_dir, validate_article from glossary_contracts import ( + GLOSSARY_CONFIDENCE_VALUES, GLOSSARY_CONCEPT_HEADING, GLOSSARY_DISCLAIMER, GLOSSARY_OCCURRENCE_HEADING, GLOSSARY_TERM_TAG, ) +from glossary_library import ( + CONCEPT_FIELDS, + _decode_frontmatter_scalar, + _normalized_key, + add_missing_concept_fields, + build_alias_index as _library_build_alias_index, + inspect_selected_terms, + note_state, + read_frontmatter_aliases as _read_frontmatter_aliases, + read_heading_name as _read_heading_name, +) ILLEGAL_FILENAME_CHARS = re.compile(r'[<>:"/\\|?*\x00-\x1f]') +MAX_FILENAME_COMPONENT_LENGTH = 240 +WINDOWS_RESERVED_NAMES = { + "CON", + "PRN", + "AUX", + "NUL", + *(f"COM{number}" for number in range(1, 10)), + *(f"LPT{number}" for number in range(1, 10)), +} def parser() -> argparse.ArgumentParser: p = argparse.ArgumentParser(description=__doc__ or "write glossary terms") p.add_argument("--glossary", required=True, help="Glossary JSON path or JSON object.") - p.add_argument("--terms-dir", required=True, help="Destination central term folder.") - p.add_argument("--paper-link", required=True, help="Paper note stem for backlink lines.") + p.add_argument("--inventory", required=True, help="Glossary inventory JSON path or JSON object.") + p.add_argument("--config-path", default="", help="Optional device-local config path override.") + p.add_argument("--source-manifest", required=True, help="Current source manifest.") + p.add_argument("--raw-sections", default="", help="Optional current raw-sections JSONL.") + p.add_argument( + "--reviewed-shortlist", required=True, help="Saved reviewed-shortlist artifact." + ) + p.add_argument("--triage", required=True, help="Saved authorized triage artifact.") + p.add_argument("--article", default="", help="Optional explicit article Markdown path.") p.add_argument("--output", default="", help="Output JSON status path.") return p def safe_term_filename(name: str) -> str: - cleaned = ILLEGAL_FILENAME_CHARS.sub(" ", name) + raw = str(name) + if not raw or raw in (".", "..") or raw.endswith((" ", ".")): + raise SystemExit(f"Unsafe glossary filename: {raw!r}") + cleaned = ILLEGAL_FILENAME_CHARS.sub(" ", raw) cleaned = normalize_whitespace(cleaned).strip(" .") - return cleaned or "term" + if not cleaned or cleaned in (".", ".."): + raise SystemExit(f"Unsafe glossary filename: {raw!r}") + if cleaned.split(".", 1)[0].upper() in WINDOWS_RESERVED_NAMES: + raise SystemExit(f"Unsafe glossary filename: reserved Windows name {raw!r}") + component = f"{cleaned}.md" + try: + component_lengths = ( + len(component.encode("utf-16-le")) // 2, + len(component.encode("utf-8")), + ) + except UnicodeEncodeError: + raise SystemExit(f"Unsafe glossary filename: invalid Unicode in {raw!r}") from None + if any(length > MAX_FILENAME_COMPONENT_LENGTH for length in component_lengths): + raise SystemExit(f"Unsafe glossary filename: component too long for {raw!r}") + return cleaned def _unique_term_path(terms_dir: Path, stem: str) -> Path: @@ -51,89 +108,22 @@ def _frontmatter_string(value: str) -> str: return json.dumps(value, ensure_ascii=False) -def _decode_frontmatter_scalar(value: str) -> str: - raw = normalize_whitespace(value) - if raw.startswith('"'): - try: - return normalize_whitespace(str(json.loads(raw))) - except json.JSONDecodeError: - pass - return normalize_whitespace(raw.strip("\"'")) - - def _alias_forms(entry: dict[str, Any]) -> list[str]: forms = [normalize_whitespace(str(entry.get("name", "")))] aliases = entry.get("aliases", []) - if isinstance(aliases, list): - forms.extend(normalize_whitespace(str(alias)) for alias in aliases) + if not isinstance(aliases, list): + raise SystemExit("Glossary entry aliases must be a list.") + forms.extend(normalize_whitespace(str(alias)) for alias in aliases) seen: set[str] = set() ordered: list[str] = [] for form in forms: - key = form.lower() + key = _normalized_key(form) if form and key not in seen: seen.add(key) ordered.append(form) return ordered -def _read_frontmatter_aliases(text: str) -> list[str]: - if not text.startswith("---"): - return [] - end = text.find("\n---", 3) - if end == -1: - return [] - block = text[3:end] - inline = re.search(r"(?m)^aliases:\s*\[(.*?)\]\s*$", block) - if inline: - content = inline.group(1).strip() - try: - data = json.loads(f"[{content}]") - except json.JSONDecodeError: - return [ - _decode_frontmatter_scalar(item) - for item in content.split(",") - if item.strip() - ] - return [normalize_whitespace(str(item)) for item in data if normalize_whitespace(str(item))] - aliases: list[str] = [] - in_block = False - for line in block.splitlines(): - if re.match(r"^aliases:\s*$", line): - in_block = True - continue - if in_block: - item = re.match(r"^\s*-\s*(.+?)\s*$", line) - if item: - aliases.append(_decode_frontmatter_scalar(item.group(1))) - elif line.strip() and not line.startswith(" "): - break - return aliases - - -def _read_heading_name(text: str) -> str: - match = re.search(r"(?m)^#\s+(.+?)\s*$", text) - return normalize_whitespace(match.group(1)) if match else "" - - -def build_alias_index(terms_dir: Path) -> dict[str, Path]: - index: dict[str, Path] = {} - if not terms_dir.is_dir(): - return index - for path in sorted(terms_dir.glob("*.md")): - keys = {path.stem.lower()} - try: - text = path.read_text(encoding="utf-8-sig") - keys.update(alias.lower() for alias in _read_frontmatter_aliases(text)) - heading = _read_heading_name(text) - if heading: - keys.add(heading.lower()) - except OSError: - pass - for key in keys: - index.setdefault(key, path) - return index - - def _concept_zone(entry: dict[str, Any]) -> list[str]: lines = [f"## {GLOSSARY_CONCEPT_HEADING}", f"- 定义：{entry.get('definition', '')}"] for label, key in (("详解", "elaboration"), ("直觉", "intuition"), ("与相邻概念的区别", "distinction")): @@ -162,26 +152,6 @@ def render_term_file(entry: dict[str, Any], paper_link: str) -> str: return "\n".join(front + body) -def append_occurrence(text: str, entry: dict[str, Any], paper_link: str) -> str: - if f"[[{paper_link}]]" in text: - return text - line = occurrence_line(entry, paper_link) - heading = f"## {GLOSSARY_OCCURRENCE_HEADING}" - if heading in text: - lines = text.rstrip("\n").splitlines() - start = next(index for index, value in enumerate(lines) if value.strip() == heading) - end = len(lines) - for index in range(start + 1, len(lines)): - if re.match(r"^##\s+", lines[index].strip()): - end = index - break - while end > start + 1 and not lines[end - 1].strip(): - end -= 1 - lines.insert(end, line) - return "\n".join(lines) + "\n" - return text.rstrip("\n") + f"\n\n{heading}\n{line}\n" - - def upsert_term_file( entry: dict[str, Any], paper_link: str, terms_dir: Path, index: dict[str, Path] ) -> dict[str, Any]: @@ -212,33 +182,579 @@ def upsert_term_file( return {"name": name, "file": str(path), "action": "created", "link_stem": path.stem} +def _file_identity(path: Path) -> tuple[int, int, int, int]: + stat = path.stat() + return (stat.st_dev, stat.st_ino, stat.st_size, stat.st_mtime_ns) + + +def _read_note_snapshot(path: Path) -> tuple[str, bool, bytes, tuple[int, int, int, int]]: + try: + identity_before = _file_identity(path) + raw = path.read_bytes() + identity_after = _file_identity(path) + if identity_before != identity_after: + raise SystemExit(f"Glossary note changed while planning: {path}") + return ( + raw.decode("utf-8-sig"), + raw.startswith(b"\xef\xbb\xbf"), + raw, + identity_after, + ) + except (OSError, UnicodeDecodeError) as exc: + raise SystemExit(f"Unable to read glossary note {path}: {exc}") from None + + +def append_occurrence(text: str, entry: dict[str, Any], paper_link: str) -> str: + line_ending = "\r\n" if "\r\n" in text else "\n" + line = occurrence_line(entry, paper_link) + heading = re.search( + rf"(?m)^##\s+{re.escape(GLOSSARY_OCCURRENCE_HEADING)}\s*$(?:\r?\n)?", text + ) + if heading is not None: + following = re.search(r"(?m)^##\s+", text[heading.end() :]) + position = heading.end() + following.start() if following else len(text) + section = text[heading.end() : position] + occurrence = re.search( + rf"(?m)^[ \t]*-[ \t]+\[\[{re.escape(paper_link)}\]\]", section + ) + if occurrence is not None: + return text + prefix = "" if text[:position].endswith(("\n", "\r")) else line_ending + suffix = "" if text[position:].startswith(("\n", "\r")) else line_ending + return text[:position] + prefix + line + line_ending + suffix + text[position:] + separator = "" if not text else (line_ending if text.endswith(("\n", "\r")) else line_ending * 2) + if text and text.endswith(("\n", "\r")): + separator = line_ending + return text + separator + f"## {GLOSSARY_OCCURRENCE_HEADING}{line_ending}{line}{line_ending}" + + +# Existing consumers import this name from this module. The implementation lives +# in glossary_library so inventory and write paths use identical alias matching. +build_alias_index = _library_build_alias_index + + +def _entry_forms(entry: dict[str, Any], inventory_item: dict[str, Any]) -> list[str]: + values: list[Any] = [entry.get("name", "")] + values.extend(inventory_item.get("surface_forms", [])) + aliases = entry.get("aliases", []) + if isinstance(aliases, list): + values.extend(aliases) + forms: list[str] = [] + seen: set[str] = set() + for value in values: + form = normalize_whitespace(str(value)) + key = _normalized_key(form) + if form and key not in seen: + seen.add(key) + forms.append(form) + return forms + + +def _entries_by_name(glossary: dict[str, Any]) -> dict[str, dict[str, Any]]: + entries = glossary.get("entries", []) + if not isinstance(entries, list): + raise SystemExit("Glossary 'entries' must be a list.") + indexed: dict[str, dict[str, Any]] = {} + for entry in entries: + if not isinstance(entry, dict): + raise SystemExit("Glossary entries must contain objects.") + name = normalize_whitespace(str(entry.get("name", ""))) + key = name.casefold() + if not name or key in indexed: + raise SystemExit("Glossary entry names must be unique and non-empty.") + _alias_forms(entry) + indexed[key] = entry + return indexed + + +def _inventory_by_term(inventory: dict[str, Any]) -> dict[str, dict[str, Any]]: + results = inventory.get("results", []) + if not isinstance(results, list): + raise SystemExit("Inventory 'results' must be a list.") + indexed: dict[str, dict[str, Any]] = {} + for item in results: + if not isinstance(item, dict): + raise SystemExit("Inventory results must contain objects.") + term = normalize_whitespace(str(item.get("term", ""))) + if not term or term.casefold() in indexed: + raise SystemExit("Inventory terms must be unique and non-empty.") + if not isinstance(item.get("surface_forms"), list): + raise SystemExit(f"Inventory surface_forms must be a list for term: {term}") + indexed[term.casefold()] = item + return indexed + + +def _inventory_files_match(external_item: dict[str, Any], fresh_item: dict[str, Any]) -> bool: + external_value = str(external_item.get("file", "")) + fresh_value = str(fresh_item.get("file", "")) + if not external_value or not fresh_value: + return external_value == fresh_value + try: + return Path(external_value).expanduser().samefile(Path(fresh_value).expanduser()) + except OSError: + return False + + +def _bound_forms(entry: dict[str, Any], inventory_item: dict[str, Any]) -> list[str]: + entry_name = normalize_whitespace(str(entry.get("name", ""))) + inventory_term = normalize_whitespace(str(inventory_item.get("term", ""))) + if entry_name.casefold() != inventory_term.casefold(): + raise SystemExit( + f"Inventory term does not match entry name: {inventory_term!r} != {entry_name!r}" + ) + inventory_forms = inventory_item.get("surface_forms", []) + if not isinstance(inventory_forms, list) or not inventory_forms: + raise SystemExit(f"Inventory surface_forms must be a non-empty list for term: {inventory_term}") + exact_inventory: list[str] = [] + for value in inventory_forms: + if not isinstance(value, str): + raise SystemExit(f"Inventory surface_forms must contain strings: {inventory_term}") + if not value or normalize_whitespace(value) != value: + raise SystemExit(f"Inventory surface_forms must contain exact values: {inventory_term}") + if value in exact_inventory: + raise SystemExit(f"Inventory surface_forms must not contain duplicates: {inventory_term}") + exact_inventory.append(value) + if inventory_term not in exact_inventory: + raise SystemExit(f"Inventory canonical name must be present in surface_forms: {entry_name}") + return exact_inventory + + +def _validated_fresh_inventory( + glossary: dict[str, Any], inventory: dict[str, Any], terms_dir: Path +) -> dict[str, dict[str, Any]]: + external = _inventory_by_term(inventory) + entries = _entries_by_name(glossary) + if set(entries) != set(external): + raise SystemExit("Inventory terms must exactly match glossary entry names.") + selected = [] + for key, item in external.items(): + entry = entries[key] + selected.append( + { + "term": normalize_whitespace(str(entry["name"])), + "surface_forms": _bound_forms(entry, item), + } + ) + fresh_items = inspect_selected_terms(selected, terms_dir) + fresh = _inventory_by_term({"results": fresh_items}) + checked_fields = ("state", "link_stem", "missing_fields") + for key, external_item in external.items(): + fresh_item = fresh.get(key) + if fresh_item is None: + raise SystemExit(f"Inventory mismatch for term: {external_item['term']}") + mismatch = any(external_item.get(field) != fresh_item.get(field) for field in checked_fields) + mismatch = mismatch or not _inventory_files_match(external_item, fresh_item) + if mismatch: + raise SystemExit(f"Inventory mismatch for term: {external_item['term']}") + return fresh + + +def _entry_patch(entry: dict[str, Any]) -> dict[str, Any]: + return { + field: entry[field] + for field in CONCEPT_FIELDS + if normalize_whitespace(str(entry.get(field, ""))) + } + + +def _valid_confidence(value: Any) -> bool: + normalized = normalize_whitespace(str(value)) + return any(level in normalized for level in GLOSSARY_CONFIDENCE_VALUES) + + +def _preflight_entries( + glossary: dict[str, Any], inventory_by_term: dict[str, dict[str, Any]] +) -> list[tuple[dict[str, Any], dict[str, Any]]]: + entries = glossary.get("entries", []) + if not isinstance(entries, list): + raise SystemExit("Glossary 'entries' must be a list.") + planned: list[tuple[dict[str, Any], dict[str, Any]]] = [] + seen: set[str] = set() + for entry in entries: + if not isinstance(entry, dict): + raise SystemExit("Glossary entries must contain objects.") + name = normalize_whitespace(str(entry.get("name", ""))) + if not name or name.casefold() in seen: + raise SystemExit("Glossary entry names must be unique and non-empty.") + seen.add(name.casefold()) + inventory_item = inventory_by_term.get(name.casefold()) + if inventory_item is None: + raise SystemExit(f"No inventory item found for term: {name}") + operation = entry.get("operation") + state = inventory_item.get("state") + occurrence = normalize_whitespace(str(entry.get("occurrence", ""))) + if not occurrence: + raise SystemExit(f"Operation for {name} requires occurrence data.") + patch = _entry_patch(entry) + if operation == "create": + if state != "new": + raise SystemExit(f"Invalid operation create for inventory state {state}: {name}") + if not normalize_whitespace(str(entry.get("definition", ""))) or not _valid_confidence( + entry.get("confidence", "") + ): + raise SystemExit(f"Create for {name} requires definition and valid confidence.") + elif operation == "enrich": + if state != "existing_thin": + raise SystemExit(f"Invalid operation enrich for inventory state {state}: {name}") + missing = set(inventory_item.get("missing_fields", [])) + if not set(patch).issubset(missing): + raise SystemExit(f"Enrich for {name} may only add inventory missing fields.") + if "confidence" in missing and not _valid_confidence(entry.get("confidence", "")): + raise SystemExit(f"Enrich for {name} requires valid confidence when confidence is missing.") + elif operation == "reuse": + if state != "existing_complete": + raise SystemExit(f"Invalid operation reuse for inventory state {state}: {name}") + if patch: + raise SystemExit(f"reuse for {name} cannot replace concept fields.") + else: + raise SystemExit(f"Invalid operation for {name}: {operation}") + if state != "new" and not Path(str(inventory_item.get("file", ""))).is_file(): + raise SystemExit(f"Inventory file is missing for term: {name}") + planned.append((entry, inventory_item)) + return planned + + +def _contained_existing_path(path_value: Any, terms_dir: Path) -> Path: + try: + path = Path(str(path_value)).expanduser().resolve(strict=True) + except OSError as exc: + raise SystemExit(f"Unable to resolve glossary target {path_value}: {exc}") from None + if not path.is_relative_to(terms_dir) or not path.is_file(): + raise SystemExit(f"Glossary target is outside terms directory: {path}") + return path + + +def _allocate_create_path(terms_dir: Path, stem: str, reserved: set[Path]) -> Path: + counter = 1 + while True: + suffix = "" if counter == 1 else f"-{counter}" + candidate = (terms_dir / f"{stem}{suffix}.md").resolve() + if not candidate.is_relative_to(terms_dir): + raise SystemExit(f"Glossary target is outside terms directory: {candidate}") + if candidate not in reserved and not candidate.exists(): + return candidate + counter += 1 + + +def _prepare_write_plans( + glossary: dict[str, Any], + inventory: dict[str, Any], + terms_dir: Path, + paper_link: str, +) -> list[dict[str, Any]]: + root = terms_dir.expanduser().resolve() + if root.exists() and not root.is_dir(): + raise SystemExit(f"Terms directory is not a directory: {root}") + fresh_inventory = _validated_fresh_inventory(glossary, inventory, root) + entries = _preflight_entries(glossary, fresh_inventory) + library_index = _library_build_alias_index(root) + target_owners: dict[Path, str] = {} + logical_owners: dict[str, str] = {} + reserved: set[Path] = set() + plans: list[dict[str, Any]] = [] + + for entry, inventory_item in entries: + operation = str(entry["operation"]) + name = normalize_whitespace(str(entry["name"])) + existing_path = ( + None + if operation == "create" + else _contained_existing_path(inventory_item["file"], root) + ) + collision_forms = _entry_forms(entry, inventory_item) + for form in collision_forms: + key = _normalized_key(form) + library_owner = library_index.get(key) + if library_owner is not None and ( + existing_path is None or not library_owner.samefile(existing_path) + ): + raise SystemExit( + f"Glossary alias collision for '{form}' with existing note {library_owner}." + ) + owner = logical_owners.get(key) + if owner is not None and owner != name: + raise SystemExit(f"Duplicate logical term '{form}' for {owner} and {name}.") + logical_owners[key] = name + + if operation == "create": + path = _allocate_create_path( + root, safe_term_filename(str(entry.get("name", ""))), reserved + ) + text = render_term_file(entry, paper_link) + had_bom = False + original_bytes = None + identity = None + fields_added = [field for field in CONCEPT_FIELDS if field in _entry_patch(entry)] + action = "created" + occurrence_added = True + changed = True + else: + path = existing_path + assert path is not None + text, had_bom, original_bytes, identity = _read_note_snapshot(path) + current_state, current_missing = note_state(text) + if ( + current_state != inventory_item["state"] + or current_missing != inventory_item["missing_fields"] + ): + raise SystemExit(f"Inventory mismatch for term: {name}") + enriched, fields_added = add_missing_concept_fields(text, _entry_patch(entry)) + updated = append_occurrence(enriched, entry, paper_link) + occurrence_added = updated != enriched + changed = updated != text + text = updated + if fields_added: + action = "enriched" + elif occurrence_added: + action = "updated" + else: + action = "unchanged" + + owner = target_owners.get(path) + if owner is not None: + raise SystemExit(f"Duplicate target {path} for {owner} and {name}.") + target_owners[path] = name + reserved.add(path) + plans.append( + { + "path": path, + "text": text, + "had_bom": had_bom, + "operation": operation, + "original_bytes": original_bytes, + "identity": identity, + "changed": changed, + "result": { + "name": name, + "forms": list(inventory_item["surface_forms"]), + "file": str(path), + "action": action, + "link_stem": path.stem, + "fields_added": fields_added, + "occurrence_added": occurrence_added, + }, + } + ) + return plans + + +def _encoded_plan_text(plan: dict[str, Any]) -> bytes: + prefix = b"\xef\xbb\xbf" if plan["had_bom"] else b"" + return prefix + str(plan["text"]).encode("utf-8") + + +def _stage_existing_temp(plan: dict[str, Any]) -> Path: + path = Path(plan["path"]) + descriptor, temp_name = tempfile.mkstemp( + dir=path.parent, + prefix=f".{path.name}.", + suffix=".tmp", + ) + temp_path = Path(temp_name) + try: + with os.fdopen(descriptor, "wb") as handle: + handle.write(_encoded_plan_text(plan)) + handle.flush() + os.fsync(handle.fileno()) + os.chmod(temp_path, path.stat().st_mode) + return temp_path + except BaseException: + temp_path.unlink(missing_ok=True) + raise + + +def _verify_plans_unchanged(plans: list[dict[str, Any]], terms_dir: Path) -> None: + for plan in plans: + path = Path(plan["path"]) + if plan["operation"] == "create": + if path.exists() or path.is_symlink(): + raise SystemExit(f"Create target appeared after planning: {path}") + try: + parent = path.parent.resolve(strict=True) + except OSError as exc: + raise SystemExit(f"Unable to resolve create target parent {path.parent}: {exc}") from None + if parent != terms_dir or not path.resolve().is_relative_to(terms_dir): + raise SystemExit(f"Glossary target is outside terms directory: {path}") + continue + + try: + resolved = path.resolve(strict=True) + identity_before = _file_identity(path) + raw = path.read_bytes() + identity_after = _file_identity(path) + except OSError as exc: + raise SystemExit(f"Glossary note changed after planning: {path}: {exc}") from None + if ( + resolved != path + or not resolved.is_relative_to(terms_dir) + or identity_before != identity_after + or identity_after != plan["identity"] + or raw != plan["original_bytes"] + ): + raise SystemExit(f"Glossary note changed after planning: {path}") + + +def _close_reservations(reservations: list[tuple[dict[str, Any], int]]) -> None: + for _, descriptor in reservations: + try: + os.close(descriptor) + except OSError: + pass + + +def _remove_created_paths(plans: list[dict[str, Any]]) -> None: + for plan in plans: + Path(plan["path"]).unlink(missing_ok=True) + + +def _commit_write_plans(plans: list[dict[str, Any]], terms_dir: Path) -> None: + terms_dir.mkdir(parents=True, exist_ok=True) + try: + resolved_root = terms_dir.resolve(strict=True) + except OSError as exc: + raise SystemExit(f"Unable to resolve terms directory {terms_dir}: {exc}") from None + if resolved_root != terms_dir or not resolved_root.is_dir(): + raise SystemExit(f"Terms directory changed after planning: {terms_dir}") + + staged: list[dict[str, Any]] = [] + reservations: list[tuple[dict[str, Any], int]] = [] + created_plans: list[dict[str, Any]] = [] + try: + for plan in plans: + if plan["operation"] != "create" and plan["changed"]: + plan["temp_path"] = _stage_existing_temp(plan) + staged.append(plan) + + _verify_plans_unchanged(plans, terms_dir) + + flags = os.O_WRONLY | os.O_CREAT | os.O_EXCL + if hasattr(os, "O_BINARY"): + flags |= os.O_BINARY + for plan in plans: + if plan["operation"] == "create" and plan["changed"]: + try: + descriptor = os.open(plan["path"], flags, 0o666) + except FileExistsError: + raise SystemExit(f"Create target appeared during commit: {plan['path']}") from None + reservations.append((plan, descriptor)) + + for plan, descriptor in reservations: + with os.fdopen(descriptor, "wb") as handle: + handle.write(_encoded_plan_text(plan)) + handle.flush() + os.fsync(handle.fileno()) + created_plans.append(plan) + reservations.clear() + + for plan in staged: + os.replace(plan["temp_path"], plan["path"]) + plan["temp_path"] = None + except BaseException: + reserved_plans = [plan for plan, _ in reservations] + _close_reservations(reservations) + _remove_created_paths(created_plans + reserved_plans) + raise + finally: + for plan in staged: + temp_path = plan.get("temp_path") + if temp_path is not None: + Path(temp_path).unlink(missing_ok=True) + + +def write_glossary_entries( + glossary: dict[str, Any], inventory: dict[str, Any], terms_dir: Path, paper_link: str +) -> dict[str, Any]: + validate_provenance_chain(inventory, inventory.get("results")) + plans = _prepare_write_plans(glossary, inventory, terms_dir, paper_link) + planned_mappings = [ + (plan["result"]["name"], plan["result"]["forms"]) for plan in plans + ] + inventory_mappings = [ + (item["term"], item["surface_forms"]) for item in inventory["results"] + ] + if planned_mappings != inventory_mappings: + raise SystemExit( + "Planned glossary mappings must exactly match the ordered inventory " + "term and surface_forms projection." + ) + _commit_write_plans(plans, terms_dir.expanduser().resolve()) + return {"status": "ok", "results": [plan["result"] for plan in plans]} + + +def _validated_wiki_link(value: str, label: str) -> str: + if ( + not isinstance(value, str) + or not value.strip() + or value != value.strip() + or any(ord(character) < 32 for character in value) + or any(character in value for character in "[]|#^") + ): + raise SystemExit(f"{label} must be a non-empty wiki-link-safe value.") + return value + + +def _writer_context( + paper_id: str, article_value: str, config: dict[str, str] +) -> dict[str, str]: + if article_value: + article = validate_article(Path(article_value), config)["article_path"] + paper_link = Path(article).stem + else: + article = "" + paper_link = paper_id + return { + "paper_id": paper_id, + "paper_link": _validated_wiki_link(paper_link, "Derived backlink"), + "article_path": article, + } + + def main() -> None: + started = perf_counter() args = parser().parse_args() + config = load_config(Path(args.config_path) if args.config_path else None) + if config is None: + raise SystemExit("Paper-glossary configuration is missing; configure it first.") + terms_dir = resolve_terms_dir(config) + inventory = maybe_load_json_record(args.inventory) + if inventory is None: + raise SystemExit(f"Expected JSON object or JSON file path for --inventory: {args.inventory}") + current = validate_authorized_selection( + inventory, + inventory.get("results"), + args.triage, + args.source_manifest, + args.raw_sections, + args.reviewed_shortlist, + ) + context = _writer_context(current["paper_id"], args.article, config) glossary = maybe_load_json_record(args.glossary) if glossary is None: raise SystemExit(f"Expected JSON object or JSON file path for --glossary: {args.glossary}") - entries = glossary.get("entries", []) - if not isinstance(entries, list): - raise SystemExit("Glossary 'entries' must be a list.") - terms_dir = Path(args.terms_dir).expanduser().resolve() - index = build_alias_index(terms_dir) - results = [ - upsert_term_file(entry, args.paper_link, terms_dir, index) - for entry in entries - if isinstance(entry, dict) and normalize_whitespace(str(entry.get("name", ""))) - ] - emit( - { - "status": "ok", - "script": "write_glossary_terms.py", - "terms_dir": str(terms_dir), - "paper_link": args.paper_link, - "results": results, - "links": [f"[[{result['link_stem']}]]" for result in results], - }, - args.output, + result = write_glossary_entries( + glossary, inventory, terms_dir, context["paper_link"] + ) + mappings_digest = writer_mappings_sha256( + current["provenance"], + result["results"], + context, + current["triage_sha256"], ) + emit({ + **result, + "script": "write_glossary_terms.py", + "paper_id": current["paper_id"], + "provenance": current["provenance"], + "context": context, + "triage_sha256": current["triage_sha256"], + "mappings_sha256": mappings_digest, + "terms_dir": str(terms_dir), + "paper_link": context["paper_link"], + "article_path": context["article_path"], + "links": [f"[[{item['link_stem']}]]" for item in result["results"]], + "elapsed_ms": elapsed_ms(started), + }, args.output) if __name__ == "__main__": diff --git a/skills/paper-glossary/tests/conftest.py b/skills/paper-glossary/tests/conftest.py index c9489ca..e871ce3 100644 --- a/skills/paper-glossary/tests/conftest.py +++ b/skills/paper-glossary/tests/conftest.py @@ -1,7 +1,169 @@ from __future__ import annotations +from copy import deepcopy +import json import sys from pathlib import Path +from typing import Any SCRIPTS_DIR = Path(__file__).resolve().parents[1] / "scripts" sys.path.insert(0, str(SCRIPTS_DIR)) + +from glossary_common import canonical_sha256, selection_sha256 +from glossary_library import inspect_selected_terms +from plan_glossary import _proposal_artifact, _review_artifact, _triage_candidates + + +def build_triage_payload( + terms: list[dict[str, Any]], *, paper_id: str = "paper-fixture" +) -> dict[str, Any]: + candidates = deepcopy(terms) + summary = { + "effective_body_characters": 100, + "shortlist_limit": max(1, len(candidates)), + "pool_candidates": len(candidates), + } + source_sha256 = canonical_sha256( + {"paper_id": paper_id, "fixture_source": "paper-glossary tests"} + ) + proposal = { + "paper_id": paper_id, + "source_sha256": source_sha256, + "candidates": candidates, + "summary": summary, + } + proposal["proposal_sha256"] = canonical_sha256(proposal) + review = { + **deepcopy(proposal), + "reviewed_shortlist": deepcopy(candidates), + } + review["review_sha256"] = canonical_sha256( + { + "paper_id": paper_id, + "proposal_provenance": proposal, + "reviewed_shortlist": review["reviewed_shortlist"], + } + ) + return { + "mode": "triage", + "paper_id": paper_id, + "provenance": { + "proposal": proposal, + "review": review, + "selection_sha256": selection_sha256(review["review_sha256"], terms), + }, + "terms": deepcopy(terms), + } + + +def build_inventory_payload( + terms: list[dict[str, Any]], terms_dir: Path, *, paper_id: str = "paper-fixture" +) -> dict[str, Any]: + return bind_inventory_results( + inspect_selected_terms(terms, terms_dir), paper_id=paper_id + ) + + +def bind_inventory_results( + results: list[dict[str, Any]], *, paper_id: str = "paper-fixture" +) -> dict[str, Any]: + exact_results = deepcopy(results) + terms = [ + {"term": item.get("term"), "surface_forms": item.get("surface_forms")} + for item in exact_results + ] + triage = build_triage_payload(terms, paper_id=paper_id) + return { + "status": "ok", + "script": "inspect_glossary_library.py", + "paper_id": paper_id, + "provenance": triage["provenance"], + "results": exact_results, + } + + +def write_current_workflow( + root: Path, + reviewed_names: list[str], + *, + selected_names: list[str] | None = None, + paper_id: str = "paper-fixture", + source_text: str | None = None, +) -> dict[str, Any]: + root.mkdir(parents=True, exist_ok=True) + selected_names = reviewed_names if selected_names is None else selected_names + text = source_text or " ".join( + f"**{name}** is a paper concept." for name in reviewed_names + ) + records = [ + { + "record_type": "section", + "section_id": "sec:fixture", + "kind": "body", + "title": "Fixture", + "text": text, + } + ] + raw_path = root / "paper_raw_sections.jsonl" + raw_path.write_text( + "\n".join(json.dumps(record, ensure_ascii=False) for record in records) + "\n", + encoding="utf-8", + ) + manifest = {"paper_id": paper_id, "raw_sections_path": str(raw_path.resolve())} + manifest_path = root / "paper_source_manifest.json" + manifest_path.write_text( + json.dumps(manifest, ensure_ascii=False), encoding="utf-8" + ) + + proposal = _proposal_artifact(manifest, records) + review = _review_artifact(proposal, reviewed_names) + reviewed_by_name = { + candidate["term"]: candidate for candidate in review["reviewed_shortlist"] + } + selected_candidates = [reviewed_by_name[name] for name in selected_names] + triaged = _triage_candidates(selected_candidates, records) + triage = { + "status": "ok", + "script": "plan_glossary.py", + "mode": "triage", + "paper_id": paper_id, + "terms": triaged, + "provenance": { + "proposal": proposal["provenance"], + "review": review["provenance"], + "selection_sha256": selection_sha256( + review["provenance"]["review_sha256"], triaged + ), + }, + } + proposal_path = root / "proposal.json" + review_path = root / "reviewed.json" + triage_path = root / "triage.json" + for path, payload in ( + (proposal_path, proposal), + (review_path, review), + (triage_path, triage), + ): + path.write_text(json.dumps(payload, ensure_ascii=False), encoding="utf-8") + return { + "manifest": manifest, + "manifest_path": manifest_path, + "raw_path": raw_path, + "proposal": proposal, + "proposal_path": proposal_path, + "review": review, + "review_path": review_path, + "triage": triage, + "triage_path": triage_path, + } + + +def build_current_inventory(workflow: dict[str, Any], terms_dir: Path) -> dict[str, Any]: + triage = workflow["triage"] + return { + "status": "ok", + "script": "inspect_glossary_library.py", + "paper_id": triage["paper_id"], + "provenance": deepcopy(triage["provenance"]), + "results": inspect_selected_terms(triage["terms"], terms_dir), + } diff --git a/skills/paper-glossary/tests/test_glossary_config.py b/skills/paper-glossary/tests/test_glossary_config.py new file mode 100644 index 0000000..2da1687 --- /dev/null +++ b/skills/paper-glossary/tests/test_glossary_config.py @@ -0,0 +1,392 @@ +from __future__ import annotations + +import hashlib +import json +import sys +from pathlib import Path + +import pytest + +from glossary_config import ( + configure_terms_dir, + find_vault_root, + load_config, + resolve_terms_dir, + validate_article, +) + + +INVALID_CONFIGURE_DESTINATIONS = ( + "vault-root", + "normalized-root", + "blank-child", + "blank-component", +) + + +def _invalid_terms_dir(vault: Path, case: str) -> Path: + if case == "vault-root": + return vault + if case == "normalized-root": + return vault / "not-created" / ".." + if case == "blank-child": + return vault / " " + return vault / " " / "Glossary" + + +def _tree_snapshot(root: Path) -> list[str]: + return sorted(str(path.relative_to(root)) for path in root.rglob("*")) + + +def test_configure_terms_dir_persists_nearest_vault_and_relative_subdir( + tmp_path: Path, +) -> None: + outer = tmp_path / "outer" + inner = outer / "research" + (outer / ".obsidian").mkdir(parents=True) + (inner / ".obsidian").mkdir(parents=True) + terms_dir = inner / "book" / "术语" + config_path = tmp_path / "device" / "config.json" + + payload = configure_terms_dir(terms_dir, config_path) + + assert payload == { + "vault_root": str(inner.resolve()), + "terms_subdir": str(Path("book") / "术语"), + } + assert terms_dir.is_dir() + assert load_config(config_path) == payload + assert resolve_terms_dir(payload) == terms_dir.resolve() + + +def test_configure_terms_dir_creates_a_missing_child_of_existing_vault( + tmp_path: Path, +) -> None: + vault = tmp_path / "vault" + (vault / ".obsidian").mkdir(parents=True) + terms_dir = vault / "new" / "nested" / "术语" + + payload = configure_terms_dir(terms_dir, tmp_path / "config.json") + + assert terms_dir.is_dir() + assert payload["vault_root"] == str(vault.resolve()) + assert payload["terms_subdir"] == str(Path("new") / "nested" / "术语") + + +def test_configure_terms_dir_reuses_an_existing_directory(tmp_path: Path) -> None: + vault = tmp_path / "vault" + (vault / ".obsidian").mkdir(parents=True) + terms_dir = vault / "existing-terms" + terms_dir.mkdir() + config_path = tmp_path / "config.json" + + payload = configure_terms_dir(terms_dir, config_path) + + assert load_config(config_path) == payload + assert resolve_terms_dir(payload) == terms_dir.resolve() + + +def test_find_vault_root_returns_none_for_non_vault_path(tmp_path: Path) -> None: + assert find_vault_root(tmp_path / "outside") is None + + +def test_configure_terms_dir_rejects_non_vault_path(tmp_path: Path) -> None: + with pytest.raises(SystemExit, match="inside an Obsidian vault"): + configure_terms_dir(tmp_path / "outside" / "术语", tmp_path / "config.json") + + +@pytest.mark.parametrize("case", INVALID_CONFIGURE_DESTINATIONS) +@pytest.mark.parametrize("existing_config", [False, True], ids=["absent", "existing"]) +def test_configure_terms_dir_rejects_invalid_child_without_config_mutation( + tmp_path: Path, case: str, existing_config: bool +) -> None: + vault = tmp_path / "vault" + (vault / ".obsidian").mkdir(parents=True) + config_path = tmp_path / "device" / "config.json" + original = b'{"sentinel":"keep"}\r\n' + if existing_config: + config_path.parent.mkdir() + config_path.write_bytes(original) + before = _tree_snapshot(vault) + + with pytest.raises(SystemExit, match="real child"): + configure_terms_dir(_invalid_terms_dir(vault, case), config_path) + + assert _tree_snapshot(vault) == before + if existing_config: + assert config_path.read_bytes() == original + else: + assert not config_path.exists() + assert not config_path.parent.exists() + + +@pytest.mark.parametrize("case", INVALID_CONFIGURE_DESTINATIONS) +@pytest.mark.parametrize("existing_config", [False, True], ids=["absent", "existing"]) +def test_configure_cli_rejects_invalid_child_without_config_mutation( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, + case: str, + existing_config: bool, +) -> None: + from configure_glossary import main + + vault = tmp_path / "vault" + (vault / ".obsidian").mkdir(parents=True) + config_path = tmp_path / "device" / "config.json" + original = b'{"sentinel":"keep"}\r\n' + if existing_config: + config_path.parent.mkdir() + config_path.write_bytes(original) + before = _tree_snapshot(vault) + monkeypatch.setattr( + sys, + "argv", + [ + "configure_glossary.py", + "--terms-dir", + str(_invalid_terms_dir(vault, case)), + "--config-path", + str(config_path), + ], + ) + + with pytest.raises(SystemExit, match="real child"): + main() + + assert _tree_snapshot(vault) == before + if existing_config: + assert config_path.read_bytes() == original + else: + assert not config_path.exists() + assert not config_path.parent.exists() + + +def test_load_config_rejects_blank_subdirectory_component(tmp_path: Path) -> None: + vault = tmp_path / "vault" + (vault / ".obsidian").mkdir(parents=True) + config_path = tmp_path / "config.json" + config_path.write_text( + json.dumps( + { + "vault_root": str(vault), + "terms_subdir": str(Path(" ") / "Glossary"), + } + ), + encoding="utf-8", + ) + + with pytest.raises(SystemExit, match="Invalid paper-glossary config object"): + load_config(config_path) + + +@pytest.mark.parametrize("vault_root", [".", "C:drive-relative"]) +def test_load_config_rejects_relative_and_drive_relative_vault_roots( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch, vault_root: str +) -> None: + vault = tmp_path / "vault" + (vault / ".obsidian").mkdir(parents=True) + (vault / "Glossary").mkdir() + config_path = tmp_path / "config.json" + config_path.write_text( + json.dumps({"vault_root": vault_root, "terms_subdir": "Glossary"}), + encoding="utf-8", + ) + monkeypatch.chdir(vault) + + with pytest.raises(SystemExit, match="Invalid paper-glossary config object"): + load_config(config_path) + + +def test_load_config_returns_canonical_absolute_vault_root(tmp_path: Path) -> None: + vault = tmp_path / "vault" + config_path = tmp_path / "config.json" + config_path.write_text( + json.dumps( + { + "vault_root": str(tmp_path / "nested" / ".." / "vault"), + "terms_subdir": "Glossary", + } + ), + encoding="utf-8", + ) + + loaded = load_config(config_path) + + assert loaded is not None + assert loaded["vault_root"] == str(vault.resolve()) + + +def test_validate_article_returns_path_and_hash_for_same_vault(tmp_path: Path) -> None: + vault = tmp_path / "vault" + (vault / ".obsidian").mkdir(parents=True) + terms_dir = vault / "术语" + article = vault / "papers" / "Paper.md" + article.parent.mkdir() + article.write_text("# Paper\n", encoding="utf-8") + config = configure_terms_dir(terms_dir, tmp_path / "config.json") + + result = validate_article(article, config) + + assert result == { + "article_path": str(article.resolve()), + "article_sha256": hashlib.sha256(article.read_bytes()).hexdigest(), + } + + +def test_validate_article_rejects_other_vault(tmp_path: Path) -> None: + first = tmp_path / "first" + second = tmp_path / "second" + (first / ".obsidian").mkdir(parents=True) + (second / ".obsidian").mkdir(parents=True) + article = second / "Paper.md" + article.write_text("# Paper\n", encoding="utf-8") + config = configure_terms_dir(first / "术语", tmp_path / "config.json") + + with pytest.raises(SystemExit, match="same Obsidian vault"): + validate_article(article, config) + + +def test_load_config_returns_none_before_first_use(tmp_path: Path) -> None: + assert load_config(tmp_path / "missing.json") is None + + +def test_resolve_terms_dir_rejects_missing_configured_paths(tmp_path: Path) -> None: + vault = tmp_path / "vault" + (vault / ".obsidian").mkdir(parents=True) + + with pytest.raises(SystemExit, match="missing or invalid"): + resolve_terms_dir( + {"vault_root": str(vault), "terms_subdir": "missing"} + ) + + +def test_resolve_terms_dir_rejects_new_nearest_nested_vault(tmp_path: Path) -> None: + vault = tmp_path / "vault" + (vault / ".obsidian").mkdir(parents=True) + terms_dir = vault / "research" / "terms" + config = configure_terms_dir(terms_dir, tmp_path / "config.json") + (terms_dir.parent / ".obsidian").mkdir() + + with pytest.raises(SystemExit, match="missing or invalid"): + resolve_terms_dir(config) + + +def test_configure_cli_show_reset_and_validate(tmp_path: Path, monkeypatch, capsys) -> None: + from configure_glossary import main + + config_path = tmp_path / "device" / "config.json" + vault = tmp_path / "vault" + (vault / ".obsidian").mkdir(parents=True) + terms_dir = vault / "术语" + article = vault / "Paper.md" + article.write_text("# Paper\n", encoding="utf-8") + + monkeypatch.setattr( + sys, + "argv", + ["configure_glossary.py", "--show", "--config-path", str(config_path)], + ) + main() + show_result = json.loads(capsys.readouterr().out) + assert isinstance(show_result.pop("elapsed_ms"), int) + assert show_result == { + "workflow_state": "needs_configuration" + } + + monkeypatch.setattr( + sys, + "argv", + [ + "configure_glossary.py", + "--terms-dir", + str(terms_dir), + "--config-path", + str(config_path), + ], + ) + main() + configure_result = json.loads(capsys.readouterr().out) + assert configure_result.pop("elapsed_ms") >= 0 + assert configure_result == { + "vault_root": str(vault.resolve()), + "terms_subdir": str(Path("术语")), + } + + monkeypatch.setattr( + sys, + "argv", + [ + "configure_glossary.py", + "--validate-article", + str(article), + "--config-path", + str(config_path), + ], + ) + main() + validation = json.loads(capsys.readouterr().out) + assert validation.pop("elapsed_ms") >= 0 + assert validation == { + "terms_dir": str(terms_dir.resolve()), + "article_path": str(article.resolve()), + "article_sha256": hashlib.sha256(article.read_bytes()).hexdigest(), + } + + monkeypatch.setattr( + sys, + "argv", + ["configure_glossary.py", "--reset", "--config-path", str(config_path)], + ) + main() + reset_result = json.loads(capsys.readouterr().out) + assert reset_result.pop("elapsed_ms") >= 0 + assert reset_result == {"workflow_state": "reset"} + assert not config_path.exists() + assert terms_dir.is_dir() + assert (vault / ".obsidian").is_dir() + + +def test_configure_cli_output_file_modes_emit_elapsed_ms( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + from configure_glossary import main + + config_path = tmp_path / "device" / "config.json" + vault = tmp_path / "vault" + (vault / ".obsidian").mkdir(parents=True) + terms_dir = vault / "terms" + article = vault / "Paper.md" + article.write_text("# Paper\n", encoding="utf-8") + + def run_mode(name: str, *arguments: str) -> dict: + output = tmp_path / f"{name}.json" + monkeypatch.setattr( + sys, + "argv", + [ + "configure_glossary.py", + *arguments, + "--config-path", + str(config_path), + "--output", + str(output), + ], + ) + main() + payload = json.loads(output.read_text(encoding="utf-8")) + assert isinstance(payload["elapsed_ms"], int) + assert payload["elapsed_ms"] >= 0 + return payload + + show_result = run_mode("show", "--show") + assert show_result["workflow_state"] == "needs_configuration" + + configure_result = run_mode("configure", "--terms-dir", str(terms_dir)) + assert configure_result["vault_root"] == str(vault.resolve()) + + validation = run_mode("validate", "--validate-article", str(article)) + assert validation["article_sha256"] == hashlib.sha256(article.read_bytes()).hexdigest() + + reset_result = run_mode("reset", "--reset") + assert reset_result["workflow_state"] == "reset" + assert not config_path.exists() diff --git a/skills/paper-glossary/tests/test_glossary_library.py b/skills/paper-glossary/tests/test_glossary_library.py new file mode 100644 index 0000000..f54f9a6 --- /dev/null +++ b/skills/paper-glossary/tests/test_glossary_library.py @@ -0,0 +1,616 @@ +from __future__ import annotations + +from copy import deepcopy +import json +import sys +from pathlib import Path + +import pytest + +from glossary_contracts import ( + GLOSSARY_CONCEPT_HEADING, + GLOSSARY_CONFIDENCE_VALUES, + GLOSSARY_LABEL_CONFIDENCE, + GLOSSARY_LABEL_DEFINITION, + GLOSSARY_OCCURRENCE_HEADING, +) +from glossary_library import ( + add_missing_concept_fields, + build_alias_index, + inspect_selected_terms, + missing_concept_fields, + read_frontmatter_aliases, +) +from inspect_glossary_library import main as inspect_main +from conftest import build_triage_payload, write_current_workflow + + +def _triage_payload(terms: list[dict]) -> dict: + return build_triage_payload(terms) + + +def _current_flags(workflow: dict) -> list[str]: + return [ + "--source-manifest", + str(workflow["manifest_path"]), + "--reviewed-shortlist", + str(workflow["review_path"]), + ] + + +def _note(*, name: str, fields: dict[str, str], aliases: list[str] | None = None) -> str: + aliases = aliases or [] + lines = ["---", "aliases:"] + lines.extend(f' - "{alias}"' for alias in aliases) + lines.extend(["---", "", f"# {name}", "", f"## {GLOSSARY_CONCEPT_HEADING}"]) + labels = { + "definition": GLOSSARY_LABEL_DEFINITION, + "elaboration": "\u8be6\u89e3\uff1a", + "intuition": "\u76f4\u89c9\uff1a", + "distinction": "\u4e0e\u76f8\u90bb\u6982\u5ff5\u7684\u533a\u522b\uff1a", + "confidence": GLOSSARY_LABEL_CONFIDENCE, + } + lines.extend(f"- {labels[field]}{value}" for field, value in fields.items()) + lines.extend(["", f"## {GLOSSARY_OCCURRENCE_HEADING}", "- [[OldPaper]]: old evidence", ""]) + return "\n".join(lines) + + +def test_inventory_classifies_alias_heading_and_new_notes(tmp_path: Path) -> None: + terms_dir = tmp_path / "terms" + terms_dir.mkdir() + (terms_dir / "LLM.md").write_text( + _note( + name="LLM", + aliases=["large language model"], + fields={ + "definition": "definition", + "confidence": GLOSSARY_CONFIDENCE_VALUES[1], + }, + ), + encoding="utf-8", + ) + (terms_dir / "ReAct.md").write_text( + _note( + name="ReAct", + fields={ + "definition": "definition", + "elaboration": "elaboration", + "intuition": "intuition", + "confidence": GLOSSARY_CONFIDENCE_VALUES[0], + }, + ), + encoding="utf-8", + ) + + results = inspect_selected_terms( + [ + {"term": "large language model", "surface_forms": ["LLM", "large language model"]}, + {"term": "ReAct", "surface_forms": ["ReAct"]}, + {"term": "Reflection", "surface_forms": ["Reflection"]}, + ], + terms_dir, + ) + + by_term = {item["term"]: item for item in results} + assert by_term["large language model"]["state"] == "existing_thin" + assert by_term["large language model"]["file"] == str(terms_dir / "LLM.md") + assert by_term["large language model"]["link_stem"] == "LLM" + assert by_term["large language model"]["missing_fields"] == [ + "elaboration", + "intuition", + "distinction", + ] + assert by_term["ReAct"]["state"] == "existing_complete" + assert by_term["Reflection"] == { + "term": "Reflection", + "surface_forms": ["Reflection"], + "state": "new", + "file": "", + "link_stem": "Reflection", + "missing_fields": [], + } + + +def test_add_missing_concept_fields_preserves_existing_content_and_is_idempotent() -> None: + original = _note( + name="LLM", + fields={ + "definition": "user definition", + "confidence": GLOSSARY_CONFIDENCE_VALUES[1], + }, + ).replace("- [[OldPaper]]: old evidence\n", "## Custom\nkeep this text\n") + + updated, fields_added = add_missing_concept_fields( + original, + {"elaboration": "new explanation", "intuition": "new intuition"}, + ) + + assert fields_added == ["elaboration", "intuition"] + assert "user definition" in updated + assert "## Custom\nkeep this text\n" in updated + assert updated.count("- \u8be6\u89e3\uff1anew explanation") == 1 + assert updated.count("- \u76f4\u89c9\uff1anew intuition") == 1 + assert updated.index("- \u8be6\u89e3\uff1anew explanation") < updated.index( + f"- {GLOSSARY_LABEL_CONFIDENCE}{GLOSSARY_CONFIDENCE_VALUES[1]}" + ) + repeated, repeated_fields = add_missing_concept_fields( + updated, + {"elaboration": "ignored", "intuition": "ignored"}, + ) + assert repeated == updated + assert repeated_fields == [] + + +def test_add_missing_concept_fields_appends_missing_concept_section() -> None: + original = "# Existing\n\n## Custom\nkeep this text\n" + + updated, fields_added = add_missing_concept_fields( + original, + { + "definition": "definition", + "confidence": GLOSSARY_CONFIDENCE_VALUES[0], + }, + ) + + assert updated.startswith(original) + assert fields_added == ["definition", "confidence"] + assert f"## {GLOSSARY_CONCEPT_HEADING}" in updated + assert missing_concept_fields(updated) == ["elaboration", "intuition", "distinction"] + + +def test_inspect_cli_emits_inventory_contract(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: + terms_dir = tmp_path / "terms" + terms_dir.mkdir() + workflow = write_current_workflow(tmp_path / "workflow", ["Reflection"]) + selected = workflow["triage_path"] + output = tmp_path / "inventory.json" + triage = workflow["triage"] + monkeypatch.setattr( + "sys.argv", + [ + "inspect_glossary_library.py", + "--terms", + str(selected), + "--terms-dir", + str(terms_dir), + *_current_flags(workflow), + "--output", + str(output), + ], + ) + + inspect_main() + + payload = json.loads(output.read_text(encoding="utf-8")) + assert isinstance(payload.pop("elapsed_ms"), int) + assert payload == { + "status": "ok", + "script": "inspect_glossary_library.py", + "paper_id": triage["paper_id"], + "provenance": triage["provenance"], + "results": [ + { + "term": "Reflection", + "surface_forms": ["Reflection"], + "state": "new", + "file": "", + "link_stem": "Reflection", + "missing_fields": [], + } + ], + } + + +def test_inspect_cli_preserves_selection_provenance_and_exact_surface_forms( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + terms_dir = tmp_path / "terms" + terms_dir.mkdir() + workflow = write_current_workflow( + tmp_path / "workflow", + ["ReAct"], + source_text="**ReAct** and **ＲｅＡｃｔ** are equivalent forms.", + ) + triage = workflow["triage_path"] + triage_payload = workflow["triage"] + provenance = triage_payload["provenance"] + output = tmp_path / "inventory.json" + monkeypatch.setattr( + sys, + "argv", + [ + "inspect_glossary_library.py", + "--terms", + str(triage), + "--terms-dir", + str(terms_dir), + *_current_flags(workflow), + "--output", + str(output), + ], + ) + + inspect_main() + + payload = json.loads(output.read_text(encoding="utf-8")) + assert payload["provenance"] == provenance + assert payload["results"][0]["surface_forms"] == ["ReAct", "ＲｅＡｃｔ"] + + +def test_inspect_cli_rejects_missing_selection_provenance( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + terms_dir = tmp_path / "terms" + terms_dir.mkdir() + workflow = write_current_workflow(tmp_path / "workflow", ["ReAct"]) + selected = tmp_path / "selected.json" + selected.write_text( + json.dumps({"selected": [{"term": "ReAct", "surface_forms": ["ReAct"]}]}), + encoding="utf-8", + ) + monkeypatch.setattr( + sys, + "argv", + [ + "inspect_glossary_library.py", + "--terms", + str(selected), + "--terms-dir", + str(terms_dir), + *_current_flags(workflow), + ], + ) + + with pytest.raises(SystemExit, match="triage|provenance"): + inspect_main() + + +def test_inspect_cli_rejects_surface_forms_changed_after_triage( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + terms_dir = tmp_path / "terms" + terms_dir.mkdir() + workflow = write_current_workflow( + tmp_path / "workflow", + ["ReAct"], + source_text="**ReAct** and **ＲｅＡｃｔ** are equivalent forms.", + ) + selected = tmp_path / "triage.json" + triage = deepcopy(workflow["triage"]) + triage["terms"][0]["surface_forms"].append("invented alias") + selected.write_text(json.dumps(triage, ensure_ascii=False), encoding="utf-8") + monkeypatch.setattr( + sys, + "argv", + [ + "inspect_glossary_library.py", + "--terms", + str(selected), + "--terms-dir", + str(terms_dir), + *_current_flags(workflow), + ], + ) + + with pytest.raises(SystemExit, match="selection|provenance|surface_forms"): + inspect_main() + + +def test_inventory_cli_rejects_source_changed_after_triage_before_inspection( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + workflow = write_current_workflow(tmp_path / "workflow", ["ReAct"]) + workflow["raw_path"].write_text( + workflow["raw_path"].read_text(encoding="utf-8") + + json.dumps( + { + "record_type": "section", + "section_id": "sec:changed", + "kind": "body", + "title": "Changed", + "text": "Reflection was added after triage.", + } + ) + + "\n", + encoding="utf-8", + ) + terms_dir = tmp_path / "terms" + terms_dir.mkdir() + sentinel = terms_dir / "Sentinel.md" + sentinel.write_bytes(b"# Sentinel\n") + monkeypatch.setattr( + sys, + "argv", + [ + "inspect_glossary_library.py", + "--terms", + str(workflow["triage_path"]), + "--terms-dir", + str(terms_dir), + "--source-manifest", + str(workflow["manifest_path"]), + "--reviewed-shortlist", + str(workflow["review_path"]), + ], + ) + + with pytest.raises(SystemExit, match="current|source|review|proposal"): + inspect_main() + + assert sentinel.read_bytes() == b"# Sentinel\n" + + +def test_inventory_cli_rejects_fully_recomputed_forged_chain( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + workflow = write_current_workflow(tmp_path / "workflow", ["ReAct"]) + forged = tmp_path / "forged-triage.json" + forged.write_text( + json.dumps( + build_triage_payload( + [{"term": "Reflection", "surface_forms": ["Reflection"]}], + paper_id="other-paper", + ) + ), + encoding="utf-8", + ) + terms_dir = tmp_path / "terms" + terms_dir.mkdir() + monkeypatch.setattr( + sys, + "argv", + [ + "inspect_glossary_library.py", + "--terms", + str(forged), + "--terms-dir", + str(terms_dir), + "--source-manifest", + str(workflow["manifest_path"]), + "--reviewed-shortlist", + str(workflow["review_path"]), + ], + ) + + with pytest.raises(SystemExit, match="current|source|review|proposal|paper"): + inspect_main() + + assert list(terms_dir.iterdir()) == [] + + +@pytest.mark.parametrize("field", ["paper_id", "source_sha256", "proposal_sha256"]) +def test_inventory_rejects_mismatched_proposal_review_common_fields( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch, field: str +) -> None: + terms_dir = tmp_path / "terms" + terms_dir.mkdir() + workflow = write_current_workflow(tmp_path / "workflow", ["ReAct"]) + triage = deepcopy(workflow["triage"]) + triage["provenance"]["review"][field] = ( + "other-paper" if field == "paper_id" else "0" * 64 + ) + selected = tmp_path / "triage.json" + selected.write_text(json.dumps(triage), encoding="utf-8") + monkeypatch.setattr( + sys, + "argv", + [ + "inspect_glossary_library.py", + "--terms", + str(selected), + "--terms-dir", + str(terms_dir), + *_current_flags(workflow), + ], + ) + + with pytest.raises(SystemExit, match="proposal|review|provenance|paper|source"): + inspect_main() + + +@pytest.mark.parametrize("stage", ["proposal", "review"]) +def test_inventory_rejects_digest_not_bound_to_embedded_material( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch, stage: str +) -> None: + terms_dir = tmp_path / "terms" + terms_dir.mkdir() + workflow = write_current_workflow(tmp_path / "workflow", ["ReAct"]) + triage = deepcopy(workflow["triage"]) + if stage == "proposal": + triage["provenance"]["proposal"]["summary"]["pool_candidates"] = 2 + else: + triage["provenance"]["review"]["reviewed_shortlist"].append( + {"term": "Reflection", "surface_forms": ["Reflection"]} + ) + selected = tmp_path / "triage.json" + selected.write_text(json.dumps(triage), encoding="utf-8") + monkeypatch.setattr( + sys, + "argv", + [ + "inspect_glossary_library.py", + "--terms", + str(selected), + "--terms-dir", + str(terms_dir), + *_current_flags(workflow), + ], + ) + + with pytest.raises(SystemExit, match="proposal|review|digest|provenance"): + inspect_main() + + +@pytest.mark.parametrize("change", ["term", "append_form", "reorder_forms"]) +def test_inventory_rejects_selected_term_or_forms_absent_from_reviewed_shortlist( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch, change: str +) -> None: + terms_dir = tmp_path / "terms" + terms_dir.mkdir() + workflow = write_current_workflow( + tmp_path / "workflow", + ["ReAct"], + source_text="**ReAct** and **\uff32\uff45\uff21\uff43\uff54** are equivalent forms.", + ) + triage = deepcopy(workflow["triage"]) + if change == "term": + triage["terms"][0]["term"] = "Reflection" + triage["terms"][0]["surface_forms"] = ["Reflection"] + elif change == "append_form": + triage["terms"][0]["surface_forms"].append("invented") + else: + triage["terms"][0]["surface_forms"].reverse() + review_sha = triage["provenance"]["review"]["review_sha256"] + from glossary_common import selection_sha256 + + triage["provenance"]["selection_sha256"] = selection_sha256( + review_sha, triage["terms"] + ) + selected = tmp_path / "triage.json" + selected.write_text(json.dumps(triage, ensure_ascii=False), encoding="utf-8") + monkeypatch.setattr( + sys, + "argv", + [ + "inspect_glossary_library.py", + "--terms", + str(selected), + "--terms-dir", + str(terms_dir), + *_current_flags(workflow), + ], + ) + + with pytest.raises(SystemExit, match="reviewed|shortlist|surface_forms|provenance"): + inspect_main() + + +@pytest.mark.parametrize( + "selected", + [ + [{"term": "ReAct"}], + [{"term": "ReAct", "surface_forms": "ReAct"}], + [{"term": "ReAct", "surface_forms": ["ReAct", 7]}], + [{"term": "ReAct", "surface_forms": [" ReAct"]}], + [{"term": "ReAct", "surface_forms": ["ＲｅＡｃｔ"]}], + ], +) +def test_inventory_rejects_malformed_or_altered_surface_forms( + tmp_path: Path, selected: list[dict] +) -> None: + with pytest.raises(SystemExit, match="surface_forms"): + inspect_selected_terms(selected, tmp_path / "terms") + + +def test_inventory_rejects_unreadable_existing_note(tmp_path: Path) -> None: + terms_dir = tmp_path / "terms" + terms_dir.mkdir() + (terms_dir / "ReAct.md").write_bytes(b"\xff\xfe\x00") + + with pytest.raises(SystemExit, match="Unable to read glossary note.*ReAct.md"): + inspect_selected_terms( + [{"term": "ReAct", "surface_forms": ["ReAct"]}], + terms_dir, + ) + + +def test_inventory_rejects_unclosed_frontmatter(tmp_path: Path) -> None: + terms_dir = tmp_path / "terms" + terms_dir.mkdir() + (terms_dir / "ReAct.md").write_text( + "---\naliases:\n - ReAct\n# ReAct\n", + encoding="utf-8", + ) + + with pytest.raises(SystemExit, match="Unclosed frontmatter.*ReAct.md"): + build_alias_index(terms_dir) + + +def test_inventory_rejects_malformed_quoted_alias(tmp_path: Path) -> None: + terms_dir = tmp_path / "terms" + terms_dir.mkdir() + (terms_dir / "ReAct.md").write_text( + "---\naliases:\n - \"unterminated\n---\n# ReAct\n", + encoding="utf-8", + ) + + with pytest.raises(SystemExit, match="Malformed frontmatter scalar"): + build_alias_index(terms_dir) + + +def test_inventory_rejects_normalized_alias_collision(tmp_path: Path) -> None: + terms_dir = tmp_path / "terms" + terms_dir.mkdir() + (terms_dir / "Alpha.md").write_text( + _note(name="Alpha", aliases=["Shared Alias"], fields={}), + encoding="utf-8", + ) + (terms_dir / "Beta.md").write_text( + _note(name="Beta", aliases=["shared alias"], fields={}), + encoding="utf-8", + ) + + with pytest.raises(SystemExit, match="Glossary key collision.*shared alias"): + build_alias_index(terms_dir) + + +def test_blank_field_is_missing_when_followed_by_another_label() -> None: + text = ( + f"# Empty\n\n## {GLOSSARY_CONCEPT_HEADING}\n" + f"- {GLOSSARY_LABEL_DEFINITION}\n" + f"- {GLOSSARY_LABEL_CONFIDENCE}{GLOSSARY_CONFIDENCE_VALUES[0]}\n" + ) + + assert "definition" in missing_concept_fields(text) + + +def test_frontmatter_alias_subset_supports_quoted_and_unquoted_scalars() -> None: + text = ( + "---\n" + "aliases: [Plain, \"Double, alias\", 'Single ''quoted'' alias']\n" + "---\n" + "# Term\n" + ) + + assert read_frontmatter_aliases(text) == [ + "Plain", + "Double, alias", + "Single 'quoted' alias", + ] + + +def test_inventory_rejects_unclosed_single_quoted_alias(tmp_path: Path) -> None: + terms_dir = tmp_path / "terms" + terms_dir.mkdir() + (terms_dir / "ReAct.md").write_text( + "---\naliases:\n - 'unterminated\n---\n# ReAct\n", + encoding="utf-8", + ) + + with pytest.raises(SystemExit, match="Malformed frontmatter scalar"): + build_alias_index(terms_dir) + + +def test_inventory_rejects_scalar_alias_structure(tmp_path: Path) -> None: + terms_dir = tmp_path / "terms" + terms_dir.mkdir() + (terms_dir / "ReAct.md").write_text( + "---\naliases: ScalarAlias\n---\n# ReAct\n", + encoding="utf-8", + ) + + with pytest.raises(SystemExit, match="Unsupported aliases metadata"): + build_alias_index(terms_dir) + + +def test_inventory_rejects_one_item_matching_multiple_files(tmp_path: Path) -> None: + terms_dir = tmp_path / "terms" + terms_dir.mkdir() + (terms_dir / "Alpha.md").write_text("# Alpha\n", encoding="utf-8") + (terms_dir / "Beta.md").write_text("# Beta\n", encoding="utf-8") + + with pytest.raises(SystemExit, match="Selected forms resolve to multiple glossary notes"): + inspect_selected_terms( + [{"term": "Alpha", "surface_forms": ["Alpha", "Beta"]}], + terms_dir, + ) diff --git a/skills/paper-glossary/tests/test_link_glossary_terms.py b/skills/paper-glossary/tests/test_link_glossary_terms.py new file mode 100644 index 0000000..dc59813 --- /dev/null +++ b/skills/paper-glossary/tests/test_link_glossary_terms.py @@ -0,0 +1,1435 @@ +from __future__ import annotations + +from copy import deepcopy +import hashlib +import json +import sys +from pathlib import Path + +import pytest + +import link_glossary_terms as linker +from conftest import write_current_workflow +from glossary_common import canonical_sha256, selection_sha256 +from glossary_config import configure_terms_dir +from link_glossary_terms import link_article_terms, main +from write_glossary_terms import render_term_file + + +def digest(path: Path) -> str: + return hashlib.sha256(path.read_bytes()).hexdigest() + + +def _writer_mappings_sha256(payload: dict) -> str: + mappings = [ + { + field: item[field] + for field in ("name", "forms", "file", "link_stem") + } + for item in payload["results"] + ] + return canonical_sha256( + { + "provenance": payload["provenance"], + "mappings": mappings, + "context": payload["context"], + "triage_sha256": payload["triage_sha256"], + } + ) + + +def _triage_sha256(payload: dict) -> str: + return canonical_sha256( + { + "paper_id": payload["paper_id"], + "provenance": payload["provenance"], + "terms": payload["terms"], + } + ) + + +def test_linker_links_first_safe_alias_and_skips_all_protected_regions( + tmp_path: Path, +) -> None: + article = tmp_path / "Paper.md" + article.write_text( + "---\ntitle: ReAct\n---\n" + "# ReAct heading\n\n" + "`ReAct` and [ReAct](https://example.test/ReAct) are protected.\n" + "ReAct https://example.test/ReAct [[Other|ReAct]]\n\n" + "REACT combines reasoning and acting. ReAct appears again.\n\n" + "```python\nReAct = 'code'\n```\n\n" + "## References\nReAct paper\n", + encoding="utf-8", + ) + + result = link_article_terms( + article, + [{"link_stem": "ReAct", "forms": ["ReAct", "REACT"]}], + digest(article), + ) + + text = article.read_text(encoding="utf-8") + assert "[[ReAct|REACT]] combines reasoning" in text + assert "# ReAct heading" in text + assert "`ReAct`" in text + assert "[ReAct](https://example.test/ReAct)" in text + assert "ReAct" in text + assert "https://example.test/ReAct" in text + assert "[[Other|ReAct]]" in text + assert "```python\nReAct = 'code'\n```" in text + assert "## References\nReAct paper" in text + assert text.count("[[ReAct|REACT]]") == 1 + assert result["summary"] == {"linked": 1, "already_linked": 0, "not_found": 0} + + +def test_linker_rejects_stale_article_hash_before_writing(tmp_path: Path) -> None: + article = tmp_path / "Paper.md" + original = b"ReAct text\n" + article.write_bytes(original) + + with pytest.raises(SystemExit, match="changed since preview"): + link_article_terms(article, [{"link_stem": "ReAct", "forms": ["ReAct"]}], "0" * 64) + + assert article.read_bytes() == original + + +def test_linker_prefers_longer_forms_and_rejects_substrings(tmp_path: Path) -> None: + article = tmp_path / "Paper.md" + article.write_text("SerpApi is an API. XAPI and APIClient are not API terms.\n", encoding="utf-8") + + result = link_article_terms( + article, + [ + {"link_stem": "API", "forms": ["API"]}, + {"link_stem": "SerpApi", "forms": ["SerpApi"]}, + ], + digest(article), + ) + + assert article.read_text(encoding="utf-8") == ( + "[[SerpApi]] is an [[API]]. XAPI and APIClient are not API terms.\n" + ) + assert result["summary"] == {"linked": 2, "already_linked": 0, "not_found": 0} + + +def test_linker_reports_existing_target_as_already_linked_without_rewriting( + tmp_path: Path, +) -> None: + article = tmp_path / "Paper.md" + original = b"[[ReAct|REACT]] is already linked. ReAct remains plain.\n" + article.write_bytes(original) + + result = link_article_terms( + article, + [{"link_stem": "ReAct", "forms": ["ReAct", "REACT"]}], + digest(article), + ) + + assert article.read_bytes() == original + assert result["results"] == [{"link_stem": "ReAct", "status": "already_linked"}] + assert result["summary"] == {"linked": 0, "already_linked": 1, "not_found": 0} + + +def test_linker_preserves_utf8_bom_and_crlf_newlines(tmp_path: Path) -> None: + article = tmp_path / "Paper.md" + article.write_bytes(b"\xef\xbb\xbfReAct text\r\nSecond line\r\n") + + link_article_terms( + article, + [{"link_stem": "ReAct", "forms": ["ReAct"]}], + digest(article), + ) + + assert article.read_bytes() == b"\xef\xbb\xbf[[ReAct]] text\r\nSecond line\r\n" + + +def test_linker_reports_not_found_and_leaves_bytes_unchanged(tmp_path: Path) -> None: + article = tmp_path / "Paper.md" + original = b"Only plain prose here.\n" + article.write_bytes(original) + + result = link_article_terms( + article, + [{"link_stem": "ReAct", "forms": ["ReAct"]}], + digest(article), + ) + + assert article.read_bytes() == original + assert result["results"] == [{"link_stem": "ReAct", "status": "not_found"}] + assert result["summary"] == {"linked": 0, "already_linked": 0, "not_found": 1} + + +def test_linker_is_idempotent(tmp_path: Path) -> None: + article = tmp_path / "Paper.md" + article.write_text("ReAct text\n", encoding="utf-8") + mappings = [{"link_stem": "ReAct", "forms": ["ReAct"]}] + + link_article_terms(article, mappings, digest(article)) + linked_bytes = article.read_bytes() + result = link_article_terms(article, mappings, digest(article)) + + assert article.read_bytes() == linked_bytes + assert result["summary"] == {"linked": 0, "already_linked": 1, "not_found": 0} + + +def test_cli_rejects_article_outside_configured_vault_before_linking( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + configured_vault = tmp_path / "configured" + other_vault = tmp_path / "other" + (configured_vault / ".obsidian").mkdir(parents=True) + (other_vault / ".obsidian").mkdir(parents=True) + config_path = tmp_path / "device" / "config.json" + configure_terms_dir(configured_vault / "Terms", config_path) + article = other_vault / "Paper.md" + original = b"ReAct text\n" + article.write_bytes(original) + writer_result = tmp_path / "writer-result.json" + writer_result.write_text( + json.dumps( + { + "status": "ok", + "results": [{"link_stem": "ReAct", "forms": ["ReAct"]}], + } + ), + encoding="utf-8", + ) + monkeypatch.setattr( + sys, + "argv", + [ + "link_glossary_terms.py", + "--input", + str(article), + "--write-result", + str(writer_result), + "--expected-sha256", + digest(article), + "--config-path", + str(config_path), + "--source-manifest", + str(tmp_path / "unused-manifest.json"), + "--reviewed-shortlist", + str(tmp_path / "unused-review.json"), + "--triage", + str(tmp_path / "unused-triage.json"), + ], + ) + + with pytest.raises(SystemExit, match="same Obsidian vault"): + main() + + assert article.read_bytes() == original + + +def test_cli_consumes_only_successful_writer_results( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + vault = tmp_path / "vault" + (vault / ".obsidian").mkdir(parents=True) + config_path = tmp_path / "device" / "config.json" + configure_terms_dir(vault / "Terms", config_path) + article = vault / "Paper.md" + original = b"ReAct text\n" + article.write_bytes(original) + writer_result = tmp_path / "writer-result.json" + writer_result.write_text( + json.dumps({"status": "failed", "results": []}), encoding="utf-8" + ) + monkeypatch.setattr( + sys, + "argv", + [ + "link_glossary_terms.py", + "--input", + str(article), + "--write-result", + str(writer_result), + "--expected-sha256", + digest(article), + "--config-path", + str(config_path), + "--source-manifest", + str(tmp_path / "unused-manifest.json"), + "--reviewed-shortlist", + str(tmp_path / "unused-review.json"), + "--triage", + str(tmp_path / "unused-triage.json"), + ], + ) + + with pytest.raises(SystemExit, match="successful writer result"): + main() + + assert article.read_bytes() == original + + +@pytest.mark.parametrize( + "original", + [ + b"---\ntitle: ReAct\nbody: ReAct\n", + b"```python\nReAct = 'protected'\n", + b"````python\nReAct = 'protected'\n```\nstill ReAct\n", + ], + ids=["unclosed-frontmatter", "unclosed-fence", "short-fence-closer"], +) +def test_linker_protects_unclosed_frontmatter_and_fences_to_eof( + tmp_path: Path, original: bytes +) -> None: + article = tmp_path / "Paper.md" + article.write_bytes(original) + + result = link_article_terms( + article, + [{"link_stem": "ReAct", "forms": ["ReAct"]}], + digest(article), + ) + + assert article.read_bytes() == original + assert result["summary"] == {"linked": 0, "already_linked": 0, "not_found": 1} + + +def test_linker_protects_nested_same_name_html_elements(tmp_path: Path) -> None: + article = tmp_path / "Paper.md" + article.write_text( + "nested ReAct\nReAct prose\n", encoding="utf-8" + ) + + link_article_terms( + article, + [{"link_stem": "ReAct", "forms": ["ReAct"]}], + digest(article), + ) + + assert article.read_text(encoding="utf-8") == ( + "nested ReAct\n[[ReAct]] prose\n" + ) + + +@pytest.mark.parametrize("tag", ["strong", "b", "em", "i"]) +def test_linker_links_visible_text_inside_simple_html_emphasis( + tmp_path: Path, tag: str +) -> None: + article = tmp_path / "Paper.md" + article.write_text(f"<{tag}>ReAct prose\n", encoding="utf-8") + + result = link_article_terms( + article, + [{"link_stem": "ReAct", "forms": ["ReAct"]}], + digest(article), + ) + + assert article.read_text(encoding="utf-8") == ( + f"<{tag}>[[ReAct]] prose\n" + ) + assert result["results"] == [{"link_stem": "ReAct", "status": "linked"}] + + +def test_linker_keeps_emphasis_text_protected_inside_html_container( + tmp_path: Path, +) -> None: + article = tmp_path / "Paper.md" + article.write_text( + "
ReAct
\nReAct prose\n", encoding="utf-8" + ) + + result = link_article_terms( + article, + [{"link_stem": "ReAct", "forms": ["ReAct"]}], + digest(article), + ) + + assert article.read_text(encoding="utf-8") == ( + "
ReAct
\n[[ReAct]] prose\n" + ) + assert result["results"] == [{"link_stem": "ReAct", "status": "linked"}] + + +def test_linker_handles_gt_and_term_text_inside_emphasis_attributes( + tmp_path: Path, +) -> None: + article = tmp_path / "Paper.md" + article.write_text( + 'Other\n' + 'ReAct prose\n', + encoding="utf-8", + ) + + result = link_article_terms( + article, + [{"link_stem": "ReAct", "forms": ["ReAct"]}], + digest(article), + ) + + assert article.read_text(encoding="utf-8") == ( + 'Other\n' + '[[ReAct]] prose\n' + ) + assert result["results"] == [{"link_stem": "ReAct", "status": "linked"}] + + +def test_linker_does_not_treat_attribute_markup_as_an_html_container( + tmp_path: Path, +) -> None: + article = tmp_path / "Paper.md" + article.write_text( + '
ReAct
\nReAct prose\n', + encoding="utf-8", + ) + + result = link_article_terms( + article, + [{"link_stem": "ReAct", "forms": ["ReAct"]}], + digest(article), + ) + + assert article.read_text(encoding="utf-8") == ( + '
ReAct
\n[[ReAct]] prose\n' + ) + assert result["results"] == [{"link_stem": "ReAct", "status": "linked"}] + + +def test_linker_protects_mismatched_html_then_links_following_prose( + tmp_path: Path, +) -> None: + article = tmp_path / "Paper.md" + article.write_text("ReAct\nReAct prose\n", encoding="utf-8") + + result = link_article_terms( + article, + [{"link_stem": "ReAct", "forms": ["ReAct"]}], + digest(article), + ) + + assert article.read_text(encoding="utf-8") == ( + "ReAct\n[[ReAct]] prose\n" + ) + assert result["results"] == [{"link_stem": "ReAct", "status": "linked"}] + + +@pytest.mark.parametrize( + "autolink", + ["", ""], + ids=["uri", "email"], +) +def test_linker_preserves_markdown_autolinks_and_links_following_prose( + tmp_path: Path, autolink: str +) -> None: + article = tmp_path / "Paper.md" + article.write_text(f"{autolink}\nReAct prose\n", encoding="utf-8") + + result = link_article_terms( + article, + [{"link_stem": "ReAct", "forms": ["ReAct"]}], + digest(article), + ) + + assert article.read_text(encoding="utf-8") == f"{autolink}\n[[ReAct]] prose\n" + assert result["results"] == [{"link_stem": "ReAct", "status": "linked"}] + + +@pytest.mark.parametrize( + "original", + [b"
\nReAct remains protected\n", b"\nReAct prose\n", + "\n[[ReAct]] prose\n", + ), + ], + ids=["inline", "fence", "frontmatter", "html-comment"], +) +def test_second_review_wiki_literals_in_protected_context_are_not_targets( + tmp_path: Path, source: str, expected: str +) -> None: + article = tmp_path / "Paper.md" + article.write_text(source, encoding="utf-8") + + result = link_article_terms( + article, + [{"link_stem": "ReAct", "forms": ["ReAct"]}], + digest(article), + ) + + assert article.read_text(encoding="utf-8") == expected + assert result["results"] == [{"link_stem": "ReAct", "status": "linked"}] + + +@pytest.mark.parametrize( + "source", + ["
ReAct prose\n", 'ReAct ReAct prose\n'], + ids=["br", "img"], +) +def test_second_review_void_html_tags_do_not_protect_following_prose( + tmp_path: Path, source: str +) -> None: + article = tmp_path / "Paper.md" + article.write_text(source, encoding="utf-8") + + link_article_terms( + article, + [{"link_stem": "ReAct", "forms": ["ReAct"]}], + digest(article), + ) + + assert article.read_text(encoding="utf-8") == source.replace( + " ReAct prose", " [[ReAct]] prose" + ) + + +def test_second_review_coalesced_target_exact_canonical_uses_direct_link( + tmp_path: Path, +) -> None: + article = tmp_path / "Paper.md" + article.write_text("ReAct prose\n", encoding="utf-8") + + result = link_article_terms( + article, + [ + {"link_stem": "ReAct", "forms": ["REACT"]}, + {"link_stem": "react", "forms": ["ReAct"]}, + ], + digest(article), + ) + + assert article.read_text(encoding="utf-8") == "[[ReAct]] prose\n" + assert result["results"] == [{"link_stem": "ReAct", "status": "linked"}] + + +def test_final_review_reference_heading_skips_fenced_literal(tmp_path: Path) -> None: + article = tmp_path / "Paper.md" + original = ( + "```text\n" + "## References\n" + "```\n\n" + "## References\n" + "ReAct citation\n" + ).encode("utf-8") + article.write_bytes(original) + + result = link_article_terms( + article, + [{"link_stem": "ReAct", "forms": ["ReAct"]}], + digest(article), + ) + + assert article.read_bytes() == original + assert result["results"] == [{"link_stem": "ReAct", "status": "not_found"}] + assert result["summary"] == {"linked": 0, "already_linked": 0, "not_found": 1} + + +def test_crlf_review_reference_heading_protects_tail(tmp_path: Path) -> None: + article = tmp_path / "Paper.md" + original = b"## References\r\nReAct citation\r\n" + article.write_bytes(original) + + result = link_article_terms( + article, + [{"link_stem": "ReAct", "forms": ["ReAct"]}], + digest(article), + ) + + assert article.read_bytes() == original + assert result["results"] == [{"link_stem": "ReAct", "status": "not_found"}] + + +def test_crlf_review_reference_heading_skips_fenced_literal(tmp_path: Path) -> None: + article = tmp_path / "Paper.md" + original = ( + b"```text\r\n" + b"## References\r\n" + b"```\r\n\r\n" + b"## References\r\n" + b"ReAct citation\r\n" + ) + article.write_bytes(original) + + result = link_article_terms( + article, + [{"link_stem": "ReAct", "forms": ["ReAct"]}], + digest(article), + ) + + assert article.read_bytes() == original + assert result["results"] == [{"link_stem": "ReAct", "status": "not_found"}] + + +@pytest.mark.parametrize( + ("reference_link", "definition"), + [ + ("[ReAct][method]", "[method]: https://example.test/react"), + ("[ReAct][]", "[ReAct]: https://example.test/react"), + ("[ReAct]", "[ReAct]: https://example.test/react"), + ], + ids=["full", "collapsed", "shortcut"], +) +def test_final_review_protects_valid_reference_links_and_definitions( + tmp_path: Path, reference_link: str, definition: str +) -> None: + article = tmp_path / "Paper.md" + original = f"{reference_link}\n\n{definition}\nReAct prose\n" + article.write_text(original, encoding="utf-8") + + result = link_article_terms( + article, + [{"link_stem": "ReAct", "forms": ["ReAct"]}], + digest(article), + ) + + assert article.read_text(encoding="utf-8") == ( + f"{reference_link}\n\n{definition}\n[[ReAct]] prose\n" + ) + assert result["results"] == [{"link_stem": "ReAct", "status": "linked"}] + + +@pytest.mark.parametrize( + ("reference_link", "definition"), + [ + ("[ReAct][method]", "[method]:\n https://example.test/react"), + ("[ReAct][]", "[ReAct]:\n https://example.test/react"), + ("[ReAct]", "[ReAct]:\n https://example.test/react"), + ], + ids=["full", "collapsed", "shortcut"], +) +def test_second_final_review_protects_next_line_reference_destinations( + tmp_path: Path, reference_link: str, definition: str +) -> None: + article = tmp_path / "Paper.md" + original = f"{reference_link}\n\n{definition}\nReAct prose\n" + article.write_text(original, encoding="utf-8") + + result = link_article_terms( + article, + [{"link_stem": "ReAct", "forms": ["ReAct"]}], + digest(article), + ) + + assert article.read_text(encoding="utf-8") == ( + f"{reference_link}\n\n{definition}\n[[ReAct]] prose\n" + ) + assert result["results"] == [{"link_stem": "ReAct", "status": "linked"}] + + +@pytest.mark.parametrize( + "definition", + [ + '[method]: /react\n "ReAct title"', + '[method]: /react\n "A title\n containing ReAct"', + ], + ids=["continuation-title", "multiline-title"], +) +def test_second_final_review_protects_complete_reference_definition_titles( + tmp_path: Path, definition: str +) -> None: + article = tmp_path / "Paper.md" + original = f"[ReAct][method]\n\n{definition}\nReAct prose\n" + article.write_text(original, encoding="utf-8") + + link_article_terms( + article, + [{"link_stem": "ReAct", "forms": ["ReAct"]}], + digest(article), + ) + + assert article.read_text(encoding="utf-8") == ( + f"[ReAct][method]\n\n{definition}\n[[ReAct]] prose\n" + ) + + +@pytest.mark.parametrize( + "definition", + [ + b"[method]:\r\n https://example.test/react", + b'[method]: /react\r\n "ReAct title"', + ], + ids=["destination-next-line", "title-next-line"], +) +def test_second_final_review_preserves_crlf_reference_definition_blocks( + tmp_path: Path, definition: bytes +) -> None: + article = tmp_path / "Paper.md" + original = b"[ReAct][method]\r\n\r\n" + definition + b"\r\nReAct prose\r\n" + article.write_bytes(original) + + link_article_terms( + article, + [{"link_stem": "ReAct", "forms": ["ReAct"]}], + digest(article), + ) + + assert article.read_bytes() == ( + b"[ReAct][method]\r\n\r\n" + definition + b"\r\n[[ReAct]] prose\r\n" + ) + + +@pytest.mark.parametrize( + ("reference_link", "definition"), + [ + ("[ReAct][method]", "[method]: /react"), + ("[ReAct][]", "[ReAct]: /react"), + ("[ReAct]", "[ReAct]: /react"), + ], + ids=["full", "collapsed", "shortcut"], +) +def test_final_review_protects_reference_links_inside_block_quotes( + tmp_path: Path, reference_link: str, definition: str +) -> None: + article = tmp_path / "Paper.md" + original = f"> {reference_link}\n>\n> {definition}\n\nReAct prose\n" + article.write_text(original, encoding="utf-8") + + result = link_article_terms( + article, + [{"link_stem": "ReAct", "forms": ["ReAct"]}], + digest(article), + ) + + assert article.read_text(encoding="utf-8") == original.replace( + "ReAct prose", "[[ReAct]] prose" + ) + assert result["results"] == [{"link_stem": "ReAct", "status": "linked"}] + + +def test_final_review_protects_multiline_reference_definitions_in_crlf_block_quotes( + tmp_path: Path, +) -> None: + article = tmp_path / "Paper.md" + original = ( + b"> [ReAct][method]\r\n>\r\n> [method]:\r\n" + b"> /react\r\n> \"ReAct title\"\r\n\r\nReAct prose\r\n" + ) + article.write_bytes(original) + + result = link_article_terms( + article, + [{"link_stem": "ReAct", "forms": ["ReAct"]}], + digest(article), + ) + + assert article.read_bytes() == original.replace( + b"ReAct prose", b"[[ReAct]] prose" + ) + assert result["results"] == [{"link_stem": "ReAct", "status": "linked"}] + + +def test_final_review_skips_setext_and_quoted_atx_headings(tmp_path: Path) -> None: + article = tmp_path / "Paper.md" + original = ( + b"ReAct setext heading\r\n====================\r\n\r\n" + b"> ReAct quoted setext\r\n> --------------------\r\n\r\n" + b"> ## ReAct quoted heading\r\n\r\nReAct prose\r\n" + ) + article.write_bytes(original) + + result = link_article_terms( + article, + [{"link_stem": "ReAct", "forms": ["ReAct"]}], + digest(article), + ) + + assert article.read_bytes() == original.replace( + b"ReAct prose", b"[[ReAct]] prose" + ) + assert result["results"] == [{"link_stem": "ReAct", "status": "linked"}] + + +@pytest.mark.parametrize( + "code_line", + [" ReAct code", "> ReAct quoted code"], + ids=["top-level", "blockquote"], +) +def test_final_review_skips_indented_code_lines( + tmp_path: Path, code_line: str +) -> None: + article = tmp_path / "Paper.md" + original = f"{code_line}\n\nReAct prose\n" + article.write_text(original, encoding="utf-8") + + result = link_article_terms( + article, + [{"link_stem": "ReAct", "forms": ["ReAct"]}], + digest(article), + ) + + assert article.read_text(encoding="utf-8") == original.replace( + "ReAct prose", "[[ReAct]] prose" + ) + assert result["results"] == [{"link_stem": "ReAct", "status": "linked"}] + + +@pytest.mark.parametrize( + "line", + [" ReAct prose", "> ReAct prose"], + ids=["top-level", "blockquote"], +) +def test_final_review_keeps_three_space_indented_prose_linkable( + tmp_path: Path, line: str +) -> None: + article = tmp_path / "Paper.md" + article.write_text(f"{line}\n", encoding="utf-8") + + result = link_article_terms( + article, + [{"link_stem": "ReAct", "forms": ["ReAct"]}], + digest(article), + ) + + expected = f"{line.replace('ReAct', '[[ReAct]]')}\n" + assert article.read_text(encoding="utf-8") == expected + assert result["results"] == [{"link_stem": "ReAct", "status": "linked"}] + + +def test_final_review_skips_nested_blockquote_fenced_code(tmp_path: Path) -> None: + article = tmp_path / "Paper.md" + original = "> > ```text\n> > ReAct code\n> > ```\n\nReAct prose\n" + article.write_text(original, encoding="utf-8") + + result = link_article_terms( + article, + [{"link_stem": "ReAct", "forms": ["ReAct"]}], + digest(article), + ) + + assert article.read_text(encoding="utf-8") == original.replace( + "ReAct prose", "[[ReAct]] prose" + ) + assert result["results"] == [{"link_stem": "ReAct", "status": "linked"}] + + +@pytest.mark.parametrize("marker", ["-", "1."], ids=["bullet", "ordered"]) +def test_final_review_skips_atx_headings_inside_list_items( + tmp_path: Path, marker: str +) -> None: + article = tmp_path / "Paper.md" + original = f"{marker} ## ReAct list heading\n\nReAct prose\n" + article.write_text(original, encoding="utf-8") + + result = link_article_terms( + article, + [{"link_stem": "ReAct", "forms": ["ReAct"]}], + digest(article), + ) + + assert article.read_text(encoding="utf-8") == original.replace( + "ReAct prose", "[[ReAct]] prose" + ) + assert result["results"] == [{"link_stem": "ReAct", "status": "linked"}] + + +@pytest.mark.parametrize("marker", ["-", "1."], ids=["bullet", "ordered"]) +def test_final_review_skips_fenced_code_inside_list_items( + tmp_path: Path, marker: str +) -> None: + article = tmp_path / "Paper.md" + indent = " " * (len(marker) + 1) + original = ( + f"{marker} ```text\n{indent}ReAct code\n{indent}```\n\nReAct prose\n" + ) + article.write_text(original, encoding="utf-8") + + result = link_article_terms( + article, + [{"link_stem": "ReAct", "forms": ["ReAct"]}], + digest(article), + ) + + assert article.read_text(encoding="utf-8") == original.replace( + "ReAct prose", "[[ReAct]] prose" + ) + assert result["results"] == [{"link_stem": "ReAct", "status": "linked"}] + + +def test_final_review_protects_long_delimiter_inline_code(tmp_path: Path) -> None: + article = tmp_path / "Paper.md" + original = "``prefix ` ReAct code``\n\nReAct prose\n" + article.write_text(original, encoding="utf-8") + + result = link_article_terms( + article, + [{"link_stem": "ReAct", "forms": ["ReAct"]}], + digest(article), + ) + + assert article.read_text(encoding="utf-8") == original.replace( + "ReAct prose", "[[ReAct]] prose" + ) + assert result["results"] == [{"link_stem": "ReAct", "status": "linked"}] diff --git a/skills/paper-glossary/tests/test_plan_glossary.py b/skills/paper-glossary/tests/test_plan_glossary.py index fb08c6e..c222b3b 100644 --- a/skills/paper-glossary/tests/test_plan_glossary.py +++ b/skills/paper-glossary/tests/test_plan_glossary.py @@ -1,14 +1,18 @@ from __future__ import annotations import json +import sys +import unicodedata from pathlib import Path import pytest +import plan_glossary from plan_glossary import ( find_occurrences, load_manifest_and_sections, load_terms, + main, propose_candidates, read_raw_sections, triage_terms, @@ -41,6 +45,94 @@ def _records() -> list[dict]: ] +def _run_proposal( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch, records: list[dict] +) -> dict: + raw = tmp_path / "paper_raw_sections.jsonl" + raw.write_text( + "\n".join(json.dumps(record, ensure_ascii=False) for record in records), + encoding="utf-8", + ) + manifest = tmp_path / "paper_source_manifest.json" + manifest.write_text( + json.dumps({"paper_id": "paper-1", "raw_sections_path": str(raw)}), + encoding="utf-8", + ) + output = tmp_path / "glossary_candidates.json" + monkeypatch.setattr( + sys, + "argv", + [ + "plan_glossary.py", + "--propose", + "--source-manifest", + str(manifest), + "--output", + str(output), + ], + ) + main() + payload = json.loads(output.read_text(encoding="utf-8")) + assert isinstance(payload["elapsed_ms"], int) + assert payload["elapsed_ms"] >= 0 + return payload + + +def _run_review( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, + reviewed_terms: list[str], +) -> dict: + output = tmp_path / "glossary_reviewed.json" + monkeypatch.setattr( + sys, + "argv", + [ + "plan_glossary.py", + "--review-proposal", + str(tmp_path / "glossary_candidates.json"), + "--reviewed-terms", + json.dumps(reviewed_terms, ensure_ascii=False), + "--source-manifest", + str(tmp_path / "paper_source_manifest.json"), + "--output", + str(output), + ], + ) + main() + return json.loads(output.read_text(encoding="utf-8")) + + +def _run_triage( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, + selected_terms: list[str] | str, +) -> dict: + output = tmp_path / "glossary_plan.json" + terms = ( + json.dumps(selected_terms, ensure_ascii=False) + if isinstance(selected_terms, list) + else selected_terms + ) + monkeypatch.setattr( + sys, + "argv", + [ + "plan_glossary.py", + "--reviewed-shortlist", + str(tmp_path / "glossary_reviewed.json"), + "--terms", + terms, + "--source-manifest", + str(tmp_path / "paper_source_manifest.json"), + "--output", + str(output), + ], + ) + main() + return json.loads(output.read_text(encoding="utf-8")) + + def test_loads_deeppapernote_manifest_raw_sections_contract(tmp_path: Path) -> None: raw = tmp_path / "paper_raw_sections.jsonl" raw.write_text("\n".join(json.dumps(record, ensure_ascii=False) for record in _records())) @@ -95,17 +187,27 @@ def test_term_found_in_paper_routes_to_anchor_only() -> None: assert result["paper_anchors"][0]["page_start"] == 3 +def test_triage_cli_emits_elapsed_ms_to_output_file( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + _run_proposal(tmp_path, monkeypatch, _records()) + _run_review(tmp_path, monkeypatch, ["MoE"]) + payload = _run_triage(tmp_path, monkeypatch, ["MoE"]) + assert isinstance(payload["elapsed_ms"], int) + assert payload["elapsed_ms"] >= 0 + assert payload["mode"] == "triage" + assert [item["term"] for item in payload["terms"]] == ["MoE"] + + def test_references_only_occurrence_does_not_count_as_found() -> None: occurrences, anchors = find_occurrences("Diffusion", _records()) assert occurrences == 0 assert anchors == [] -def test_alias_bridges_language_gap() -> None: - result = triage_terms(["知识蒸馏|knowledge distillation|KD"], _records())[0] - assert result["routing"] == "anchor_only" - assert result["term"] == "知识蒸馏" - assert result["surface_forms"] == ["知识蒸馏", "knowledge distillation", "KD"] +def test_triage_helper_rejects_caller_supplied_alias_syntax() -> None: + with pytest.raises(SystemExit, match="exact name|alias"): + triage_terms(["知识蒸馏|knowledge distillation|KD"], _records()) def test_ascii_term_matches_whole_word_only() -> None: @@ -117,6 +219,7 @@ def test_load_terms_accepts_json_list_and_delimited_string() -> None: assert load_terms('["MoE", "SSDG"]') == ["MoE", "SSDG"] assert load_terms("MoE, SSDG\nKD") == ["MoE", "SSDG", "KD"] assert load_terms('["MoE", "moe"]') == ["MoE"] + assert load_terms('["F1", "3D Gaussian Splatting"]') == ["F1", "3D Gaussian Splatting"] def test_load_terms_rejects_malformed_json_list(tmp_path: Path) -> None: @@ -132,6 +235,27 @@ def test_load_terms_rejects_malformed_json_list(tmp_path: Path) -> None: assert "Invalid terms JSON list" in str(file_exc.value) +@pytest.mark.parametrize( + "selection", + [ + "全部写入", + "全部写入。", + "1,3", + "1、3", + "1;3", + "1；3", + "1.", + "1)", + "#1", + "(1)", + '["1", "3"]', + ], +) +def test_load_terms_rejects_unresolved_shortlist_selection(selection: str) -> None: + with pytest.raises(SystemExit, match="Resolve shortlist"): + load_terms(selection) + + def test_propose_ranks_acronyms_model_names_and_keywords() -> None: records = [ { @@ -154,6 +278,7 @@ def test_propose_ranks_acronyms_model_names_and_keywords() -> None: candidate["term"]: candidate["category"] for candidate in propose_candidates(records) } + ordered_terms = [candidate["term"] for candidate in propose_candidates(records)] assert candidates["Domain Generalization"] == "keyword" assert candidates["Mixture-of-Experts"] == "keyword" @@ -161,3 +286,624 @@ def test_propose_ranks_acronyms_model_names_and_keywords() -> None: assert candidates["IED"] == "acronym-or-model" assert candidates["Knowledge Distillation"] == "keyword" assert "The Method" not in candidates + assert max( + ordered_terms.index(term) + for term in ("Domain Generalization", "Mixture-of-Experts", "Knowledge Distillation") + ) < ordered_terms.index("Interictal Epileptiform Discharges") + assert ordered_terms.index("Interictal Epileptiform Discharges") < ordered_terms.index("IED") + + +def test_propose_cli_emits_selection_gate_state( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + payload = _run_proposal(tmp_path, monkeypatch, _records()) + assert payload["status"] == "ok" + assert payload["mode"] == "propose" + assert payload["workflow_state"] == "awaiting_semantic_review" + assert payload["next_action"] == "record_reviewed_shortlist_then_present_and_wait" + assert payload["summary"]["effective_body_characters"] == len( + "".join(plan_glossary.effective_prose_text(_records()).split()) + ) + assert payload["summary"]["shortlist_limit"] == 10 + assert payload["summary"]["pool_candidates"] == len(payload["candidates"]) + + +def test_propose_cli_reports_no_candidates_without_waiting( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + records = [ + { + "record_type": "section", + "section_id": "sec:intro", + "kind": "introduction", + "title": "Introduction", + "page_start": 1, + "page_end": 1, + "text": "ordinary lowercase prose without candidate terminology.", + } + ] + + payload = _run_proposal(tmp_path, monkeypatch, records) + + assert payload["candidates"] == [] + assert payload["workflow_state"] == "no_candidates" + assert payload["next_action"] == "report_no_candidates" + assert payload["summary"] == { + "effective_body_characters": len( + "".join(plan_glossary.effective_prose_text(records).split()) + ), + "shortlist_limit": 10, + "pool_candidates": 0, + } + + +def test_review_can_drop_all_candidates_and_reports_no_candidates( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + proposal = _run_proposal(tmp_path, monkeypatch, _records()) + assert proposal["candidates"] + + review = _run_review(tmp_path, monkeypatch, []) + + assert review["reviewed_shortlist"] == [] + assert review["workflow_state"] == "no_candidates" + assert review["next_action"] == "report_no_candidates" + assert review["summary"]["reviewed_candidates"] == 0 + + +@pytest.mark.parametrize( + ("characters", "expected"), + [ + (9_999, 10), + (10_000, 18), + (29_999, 18), + (30_000, 25), + (59_999, 25), + (60_000, 35), + ], +) +def test_shortlist_limit_uses_effective_body_length( + characters: int, expected: int +) -> None: + assert plan_glossary.shortlist_limit(characters) == expected + + +def test_propose_filters_code_noise_and_keeps_emphasized_core_terms() -> None: + highlighted_sentence = ( + "This is one entire highlighted sentence with several ordinary prose words" + ) + records = [ + { + "record_type": "section", + "section_id": "sec:chapter", + "kind": "chapter", + "title": "Chapter", + "text": ( + "**ReAct**. *Plan-and-Solve* and **Reflection** are core methods.\n" + "SerpApi provides search.\n" + "**[LinkedTerm](https://example.invalid/SECRET_MODEL)** is linked prose.\n" + f"**{highlighted_sentence}**\n" + "```python\nLLM_API_KEY = 'KEY'\n" + "REACT_PROMPT_TEMPLATE = 'MODEL'\n" + "print('HUAWEI phone result')\n```\n" + "# References\n**CitationNoise** appears only in the reference tail.\n" + ), + }, + { + "record_type": "section", + "section_id": "sec:refs", + "kind": "references", + "title": "References", + "text": "SERPAPI TEMPLATE PLANNER", + }, + ] + + terms = {item["term"] for item in propose_candidates(records)} + + assert {"ReAct", "Plan-and-Solve", "Reflection", "SerpApi", "LinkedTerm"} <= terms + assert terms.isdisjoint({"KEY", "MODEL", "TEMPLATE", "HUAWEI", "SECRET", "CitationNoise"}) + assert highlighted_sentence not in terms + + +def test_occurrence_grounding_masks_non_prose_and_keeps_original_snippet_offsets() -> None: + records = [ + { + "record_type": "section", + "section_id": "sec:chapter", + "kind": "chapter", + "title": "Chapter", + "text": ( + "```text\nReAct fenced code\n```\n" + "`ReAct inline code`\n" + "https://example.invalid/ReAct\n" + + "x" * 100 + + "\n**ReAct** body evidence.\n" + + "y" * 100 + + "\n# References\n**ReAct** citation only.\n" + ), + } + ] + + candidate = next( + item for item in propose_candidates(records) if item["term"] == "ReAct" + ) + occurrences, anchors = plan_glossary._find_occurrences_for_forms( + candidate["surface_forms"], records + ) + triaged = plan_glossary._triage_candidates([candidate], records)[0] + + assert occurrences == candidate["occurrences"] == triaged["occurrences"] == 1 + assert anchors == triaged["paper_anchors"] + assert "body evidence" in anchors[0]["snippet"] + assert "fenced code" not in anchors[0]["snippet"] + assert "citation only" not in anchors[0]["snippet"] + + +def test_candidate_mask_handles_exact_fences_code_spans_and_nested_link_destinations() -> None: + records = [ + { + "record_type": "section", + "section_id": "sec:chapter", + "kind": "chapter", + "title": "Chapter", + "text": ( + "````python\nignored\n```\n**FenceLeak**\n````\n" + "``prefix ` **InlineLeak**``\n" + "[LinkedTerm](docs/topic_(SECRET)_MODEL)\n" + "**CoreTerm** is body evidence.\n" + ), + } + ] + + candidates = {item["term"]: item for item in propose_candidates(records)} + + assert {"LinkedTerm", "CoreTerm"} <= candidates.keys() + assert candidates["LinkedTerm"]["occurrences"] == 1 + assert {"FenceLeak", "InlineLeak", "SECRET", "MODEL"}.isdisjoint(candidates) + for term in ("FenceLeak", "InlineLeak", "SECRET", "MODEL"): + assert find_occurrences(term, records) == (0, []) + + +@pytest.mark.parametrize( + "sentence", + [ + "We propose a new method", + "This model improves benchmark accuracy", + "我们提出一种新的方法", + ], +) +def test_propose_rejects_emphasized_sentences(sentence: str) -> None: + records = [ + { + "record_type": "section", + "section_id": "sec:method", + "kind": "method", + "title": "Method", + "text": f"**{sentence}**", + } + ] + + terms = {item["term"] for item in propose_candidates(records)} + + assert sentence not in terms + + +@pytest.mark.parametrize( + "term", + [ + "ReAct", + "Plan-and-Solve", + "Reflection", + "SerpApi", + "Graph Memory Augmented Routing", + "智能体路由", + "归纳式与传导式推理", + ], +) +def test_propose_keeps_term_shaped_emphasis(term: str) -> None: + records = [ + { + "record_type": "section", + "section_id": "sec:method", + "kind": "method", + "title": "Method", + "text": f"**{term}** appears once.", + } + ] + + candidate = next(item for item in propose_candidates(records) if item["term"] == term) + + assert candidate["category"] == "emphasis" + assert candidate["occurrences"] == 1 + + +@pytest.mark.parametrize("heading", ["参考文献", "引用"]) +def test_propose_removes_chinese_reference_tail_from_single_record( + heading: str, +) -> None: + records = [ + { + "record_type": "section", + "section_id": "sec:chapter", + "kind": "chapter", + "title": "Chapter", + "text": ( + "**CoreTerm** is discussed in the chapter.\n" + f"# {heading}\n" + "**CitationNoise** appears only in the reference tail.\n" + ), + } + ] + + terms = {item["term"] for item in propose_candidates(records)} + + assert "CoreTerm" in terms + assert "CitationNoise" not in terms + assert f"# {heading}" not in plan_glossary.effective_prose_text(records) + + +def test_propose_merges_nfkc_equivalent_surface_form_occurrences() -> None: + records = [ + { + "record_type": "section", + "section_id": "sec:method", + "kind": "method", + "title": "Method", + "text": "**ReAct** and **ＲｅＡｃｔ** are equivalent spellings.", + } + ] + + matches = [ + item + for item in propose_candidates(records) + if unicodedata.normalize("NFKC", item["term"]).casefold() == "react" + ] + + assert len(matches) == 1 + assert matches[0]["term"] == "ReAct" + assert matches[0]["occurrences"] == 2 + assert "ＲｅＡｃｔ" in matches[0]["snippet"] + + +def test_propose_deduplicates_overlapping_casefold_surface_matches() -> None: + records = [ + { + "record_type": "section", + "section_id": "sec:method", + "kind": "method", + "title": "Method", + "text": "**ReAct** and **react** are two original occurrences.", + } + ] + + candidate = next( + item + for item in propose_candidates(records) + if unicodedata.normalize("NFKC", item["term"]).casefold() == "react" + ) + + assert candidate["occurrences"] == 2 + + +def test_equivalent_surface_forms_preserve_record_order_and_anchor_limit() -> None: + forms = ["ReAct", "ＲｅＡｃｔ"] + records = [ + { + "record_type": "section", + "section_id": f"sec:{index}", + "kind": "method", + "title": f"Method {index}", + "page_start": index + 1, + "page_end": index + 1, + "text": f"**{forms[index % 2]}** appears in record {index}.", + } + for index in range(5) + ] + + candidate = next( + item + for item in propose_candidates(records) + if unicodedata.normalize("NFKC", item["term"]).casefold() == "react" + ) + occurrences, anchors = plan_glossary._find_occurrences_for_forms(forms, records) + + assert candidate["occurrences"] == 5 + assert occurrences == 5 + assert [anchor["section_id"] for anchor in anchors] == ["sec:0", "sec:1", "sec:2"] + assert [anchor["page_start"] for anchor in anchors] == [1, 2, 3] + + +def test_propose_caps_pool_at_dynamic_shortlist_limit_plus_buffer() -> None: + terms = [f"Term{index:02d}" for index in range(21)] + records = [ + { + "record_type": "section", + "section_id": "sec:intro", + "kind": "introduction", + "title": "Introduction", + "page_start": 1, + "page_end": 1, + "text": "Keywords: " + ", ".join(reversed(terms)), + } + ] + + candidates = propose_candidates(records) + + assert len(candidates) == 20 + + +@pytest.mark.parametrize( + ("effective_characters", "expected_pool"), + [(10_000, 28), (30_000, 35), (60_000, 45)], +) +def test_propose_caps_pool_at_higher_dynamic_tiers( + effective_characters: int, expected_pool: int +) -> None: + terms = [f"Term{index:02d}" for index in range(50)] + keyword_line = "Keywords: " + ", ".join(terms) + keyword_characters = sum(1 for character in keyword_line if not character.isspace()) + records = [ + { + "record_type": "section", + "section_id": "sec:intro", + "kind": "introduction", + "title": "Introduction", + "page_start": 1, + "page_end": 1, + "text": keyword_line + "\n" + "x" * (effective_characters - keyword_characters), + } + ] + + assert sum( + 1 + for character in plan_glossary.effective_prose_text(records) + if not character.isspace() + ) == effective_characters + assert len(propose_candidates(records)) == expected_pool + + +def test_proposal_emits_ordered_surface_forms_and_stable_provenance( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + records = [ + { + "record_type": "section", + "section_id": "sec:method", + "kind": "method", + "title": "Method", + "page_start": 1, + "text": "**ReAct** and **ＲｅＡｃｔ** are equivalent source spellings.", + } + ] + + first = _run_proposal(tmp_path, monkeypatch, records) + second = _run_proposal(tmp_path, monkeypatch, records) + candidate = next(item for item in first["candidates"] if item["term"] == "ReAct") + + assert candidate["surface_forms"] == ["ReAct", "ＲｅＡｃｔ"] + assert first["provenance"] == second["provenance"] + assert first["provenance"]["paper_id"] == "paper-1" + assert len(first["provenance"]["source_sha256"]) == 64 + assert len(first["provenance"]["proposal_sha256"]) == 64 + + +def test_review_recording_preserves_exact_ordered_candidate_objects( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + proposal = _run_proposal(tmp_path, monkeypatch, _records()) + names = [item["term"] for item in proposal["candidates"][:2]][::-1] + + reviewed = _run_review(tmp_path, monkeypatch, names) + + expected = [ + next(item for item in proposal["candidates"] if item["term"] == name) + for name in names + ] + assert reviewed["mode"] == "review" + assert reviewed["reviewed_shortlist"] == expected + assert reviewed["proposal_provenance"] == proposal["provenance"] + assert reviewed["provenance"]["proposal_sha256"] == proposal["provenance"][ + "proposal_sha256" + ] + + +@pytest.mark.parametrize( + "reviewed_terms", + [ + ["renamed candidate"], + ["MoE", "MoE"], + ["MoE|invented-alias"], + ], +) +def test_review_recording_rejects_non_exact_or_duplicate_names( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, + reviewed_terms: list[str], +) -> None: + _run_proposal(tmp_path, monkeypatch, _records()) + + with pytest.raises(SystemExit, match="reviewed|proposal|duplicate|exact"): + _run_review(tmp_path, monkeypatch, reviewed_terms) + + +def test_review_recording_rejects_more_than_dynamic_shortlist_limit( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + records = [ + { + "record_type": "section", + "section_id": "sec:intro", + "kind": "introduction", + "title": "Introduction", + "text": "Keywords: " + ", ".join(f"Term{index:02d}" for index in range(12)), + } + ] + proposal = _run_proposal(tmp_path, monkeypatch, records) + names = [item["term"] for item in proposal["candidates"][:11]] + + with pytest.raises(SystemExit, match="shortlist limit"): + _run_review(tmp_path, monkeypatch, names) + + +def test_review_recording_rejects_tampered_proposal( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + proposal = _run_proposal(tmp_path, monkeypatch, _records()) + proposal["candidates"][0]["term"] = "Tampered" + (tmp_path / "glossary_candidates.json").write_text( + json.dumps(proposal, ensure_ascii=False), encoding="utf-8" + ) + + with pytest.raises(SystemExit, match="proposal"): + _run_review(tmp_path, monkeypatch, ["Tampered"]) + + +def test_review_recording_rejects_proposal_from_another_paper( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + proposal = _run_proposal(tmp_path, monkeypatch, _records()) + manifest = tmp_path / "paper_source_manifest.json" + payload = json.loads(manifest.read_text(encoding="utf-8")) + payload["paper_id"] = "paper-2" + manifest.write_text(json.dumps(payload), encoding="utf-8") + + with pytest.raises(SystemExit, match="paper|source|proposal"): + _run_review(tmp_path, monkeypatch, [proposal["candidates"][0]["term"]]) + + +def test_review_recording_rejects_changed_source_data( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + proposal = _run_proposal(tmp_path, monkeypatch, _records()) + raw = tmp_path / "paper_raw_sections.jsonl" + raw.write_text(raw.read_text(encoding="utf-8") + "\n" + json.dumps({ + "record_type": "section", + "section_id": "sec:new", + "kind": "method", + "text": "**NewTerm** changed the source.", + }), encoding="utf-8") + + with pytest.raises(SystemExit, match="source|proposal"): + _run_review(tmp_path, monkeypatch, [proposal["candidates"][0]["term"]]) + + +def test_triage_requires_reviewed_shortlist_artifact( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + _run_proposal(tmp_path, monkeypatch, _records()) + monkeypatch.setattr( + sys, + "argv", + [ + "plan_glossary.py", + "--terms", + '["MoE"]', + "--source-manifest", + str(tmp_path / "paper_source_manifest.json"), + ], + ) + + with pytest.raises(SystemExit, match="reviewed shortlist|--reviewed-shortlist"): + main() + + +def test_triage_uses_reviewed_surface_forms_and_preserves_provenance( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + records = [ + { + "record_type": "section", + "section_id": "sec:method", + "kind": "method", + "title": "Method", + "page_start": 1, + "text": "**ReAct** and **ＲｅＡｃｔ** are equivalent source spellings.", + } + ] + proposal = _run_proposal(tmp_path, monkeypatch, records) + reviewed = _run_review(tmp_path, monkeypatch, ["ReAct"]) + + triaged = _run_triage(tmp_path, monkeypatch, ["ReAct"]) + + assert triaged["terms"][0]["surface_forms"] == ["ReAct", "ＲｅＡｃｔ"] + assert triaged["terms"][0]["occurrences"] == 2 + assert triaged["provenance"]["proposal"] == proposal["provenance"] + assert triaged["provenance"]["review"] == reviewed["provenance"] + assert triaged["provenance"]["selection_sha256"] + + +@pytest.mark.parametrize( + "selection", + [["unknown"], ["ReAct|invented-alias"], "全部写入", "1", '["1"]'], +) +def test_triage_rejects_names_outside_review_or_unresolved_controls( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, + selection: list[str] | str, +) -> None: + proposal = _run_proposal(tmp_path, monkeypatch, _records()) + _run_review(tmp_path, monkeypatch, [proposal["candidates"][0]["term"]]) + + with pytest.raises(SystemExit, match="shortlist|Resolve|exact"): + _run_triage(tmp_path, monkeypatch, selection) + + +@pytest.mark.parametrize("field", ["reviewed_shortlist", "provenance"]) +def test_triage_rejects_tampered_reviewed_candidate_or_identity( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, + field: str, +) -> None: + proposal = _run_proposal(tmp_path, monkeypatch, _records()) + name = proposal["candidates"][0]["term"] + reviewed = _run_review(tmp_path, monkeypatch, [name]) + if field == "reviewed_shortlist": + reviewed[field][0]["surface_forms"] = ["invented"] + else: + reviewed[field]["review_sha256"] = "0" * 64 + (tmp_path / "glossary_reviewed.json").write_text( + json.dumps(reviewed, ensure_ascii=False), encoding="utf-8" + ) + + with pytest.raises(SystemExit, match="review|proposal|identity"): + _run_triage(tmp_path, monkeypatch, [name]) + + +def test_triage_rejects_reviewed_artifact_after_source_proposal_changes( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + proposal = _run_proposal(tmp_path, monkeypatch, _records()) + name = proposal["candidates"][0]["term"] + _run_review(tmp_path, monkeypatch, [name]) + raw = tmp_path / "paper_raw_sections.jsonl" + raw.write_text( + raw.read_text(encoding="utf-8") + + "\n" + + json.dumps( + { + "record_type": "section", + "section_id": "sec:changed", + "kind": "method", + "text": "**ChangedTerm** changes the proposal source.", + } + ), + encoding="utf-8", + ) + + with pytest.raises(SystemExit, match="source|proposal|review"): + _run_triage(tmp_path, monkeypatch, [name]) + + +def test_triage_rejects_reviewed_artifact_from_another_paper( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + proposal = _run_proposal(tmp_path, monkeypatch, _records()) + name = proposal["candidates"][0]["term"] + _run_review(tmp_path, monkeypatch, [name]) + manifest = tmp_path / "paper_source_manifest.json" + payload = json.loads(manifest.read_text(encoding="utf-8")) + payload["paper_id"] = "paper-2" + manifest.write_text(json.dumps(payload), encoding="utf-8") + + with pytest.raises(SystemExit, match="proposal|review"): + _run_triage(tmp_path, monkeypatch, [name]) diff --git a/skills/paper-glossary/tests/test_skill_boundary.py b/skills/paper-glossary/tests/test_skill_boundary.py index 34ab9e3..5db454e 100644 --- a/skills/paper-glossary/tests/test_skill_boundary.py +++ b/skills/paper-glossary/tests/test_skill_boundary.py @@ -1,17 +1,33 @@ from __future__ import annotations +import json +import re +import subprocess +import sys from pathlib import Path SKILL_DIR = Path(__file__).resolve().parents[1] -def test_skill_documents_shared_file_contract_not_direct_workflow_call() -> None: +def _section(text: str, start: str, end: str | None = None) -> str: + body = text.split(start, 1)[1] + return body.split(end, 1)[0] if end else body + + +def _assert_in_order(text: str, phrases: tuple[str, ...]) -> None: + positions = [text.index(phrase) for phrase in phrases] + assert positions == sorted(positions) + + +def test_skill_documents_required_manifest_override_file_boundary() -> None: text = (SKILL_DIR / "SKILL.md").read_text(encoding="utf-8") - assert "raw_sections_path" in text assert "*_source_manifest.json" in text + assert "raw_sections_path" in text assert "*_raw_sections.jsonl" in text + assert "`--source-manifest` is always required" in text + assert "`--raw-sections` only overrides" in text assert "scripts/run_pipeline.py" not in text @@ -22,3 +38,382 @@ def test_scripts_are_self_contained_within_paper_glossary() -> None: text = path.read_text(encoding="utf-8") assert "skills.deeppapernote" not in text assert "deeppapernote" not in text.lower() + + +def test_skill_orders_preview_gate_before_post_selection_writes() -> None: + text = (SKILL_DIR / "SKILL.md").read_text(encoding="utf-8") + preview = _section(text, "## Preview And Wait (Default)", "## After Selection") + after = _section(text, "## After Selection", "## Grounding") + + _assert_in_order( + preview, + ( + "Show saved configuration", + "Require an explicit article Markdown path", + "Run deterministic proposal", + "Perform exactly one grounded host semantic review", + "Record the reviewed shortlist", + "Present every retained term", + "No glossary notes or article Markdown have been written", + "End the response and wait", + ), + ) + _assert_in_order( + after, + ( + "Accept only numbers", + "Triage", + "inventory", + "generate one action-aware batch", + "Run one writer invocation", + "whole-batch preflight", + "create/enrich/reuse commit", + "Link each successful writer result", + "Lint writer-returned changed glossary note files", + "Report observable wall-clock timing", + ), + ) + + +def test_task7_acceptance_preview_records_review_before_display_and_wait() -> None: + text = (SKILL_DIR / "references" / "optimization-implementation-plan.md").read_text( + encoding="utf-8" + ) + task7 = _section(text, "### Task 7:", "## Execution Completion Criteria") + + _assert_in_order( + task7, + ( + "--review-proposal", + "Display every retained item from this reviewed artifact", + "End the response and wait", + ), + ) + + +def test_preview_discloses_setup_write_and_limits_no_write_claim() -> None: + text = (SKILL_DIR / "SKILL.md").read_text(encoding="utf-8") + preview = _section(text, "## Preview And Wait (Default)", "## After Selection") + + assert "Setup may create `~/.paper-glossary/config.json`" in preview + assert "No glossary notes or article Markdown have been written" in preview + assert "no files have been written" not in preview.lower() + assert "before selection" in preview + + +def test_semantic_review_is_an_exact_term_subset_permutation() -> None: + skill = (SKILL_DIR / "SKILL.md").read_text(encoding="utf-8") + contract = (SKILL_DIR / "references" / "file-contract.md").read_text( + encoding="utf-8" + ) + + for text in (skill, contract): + assert "only drop or reorder candidates" in text + assert "exact `term` string" in text + assert not re.search( + r"\b(?:merge|rename|canonical|canonicalize|invent)\b", text, re.IGNORECASE + ) + + +def test_linker_is_conditional_and_glossary_only_skips_it() -> None: + skill = (SKILL_DIR / "SKILL.md").read_text(encoding="utf-8") + contract = (SKILL_DIR / "references" / "file-contract.md").read_text( + encoding="utf-8" + ) + after = _section(skill, "## After Selection", "## Grounding") + + for text in (after, contract): + assert "only if an article Markdown was supplied/requested" in text + assert "For a glossary-only request, skip `link_glossary_terms.py`" in text + + +def test_lint_scope_is_writer_returned_changed_glossary_notes_only() -> None: + skill = (SKILL_DIR / "SKILL.md").read_text(encoding="utf-8") + contract = (SKILL_DIR / "references" / "file-contract.md").read_text( + encoding="utf-8" + ) + after = _section(skill, "## After Selection", "## Grounding") + + for text in (after, contract): + assert "writer-returned changed glossary note files" in text + assert "`action` is `created`, `enriched`, or `updated`" in text + assert "one `lint_glossary.py` invocation with repeated `--input PATH`" in text + assert "Do not pass article Markdown to `lint_glossary.py`" in text + + +def test_term_note_linter_rejects_an_ordinary_article() -> None: + scripts_dir = SKILL_DIR / "scripts" + sys.path.insert(0, str(scripts_dir)) + from lint_glossary import lint_term_file_text + + result = lint_term_file_text("# Paper\n\nOrdinary article prose.\n") + codes = {issue["code"] for issue in result["issues"]} + + assert result["passes"] is False + assert { + "term_disclaimer_missing", + "term_concept_zone_missing", + "term_occurrence_zone_missing", + }.issubset(codes) + + +def test_timing_matches_host_and_cli_observation_boundaries() -> None: + skill = (SKILL_DIR / "SKILL.md").read_text(encoding="utf-8") + contract = (SKILL_DIR / "references" / "file-contract.md").read_text( + encoding="utf-8" + ) + after = _section(skill, "## After Selection", "## Grounding") + + for text in (after, contract): + assert "Time host-only phases separately" in text + assert "Each CLI emits top-level `elapsed_ms`" in text + assert "single writer invocation includes whole-batch preflight and commit" in text + assert "Do not report separate preflight and commit timings" in text + assert "around each phase" not in text + assert "current CLIs do not emit `elapsed_ms`" not in after + assert '"elapsed_ms"' in contract + + +def test_default_prompt_enters_preview_and_wait_before_content_writes() -> None: + text = (SKILL_DIR / "agents" / "openai.yaml").read_text(encoding="utf-8") + default_prompt = next( + line.strip() for line in text.splitlines() if line.strip().startswith("default_prompt:") + ).lower() + + assert "preview-and-wait" in default_prompt + assert "numbered shortlist" in default_prompt + assert "stop and wait" in default_prompt + assert "before writing glossary notes or article links" in default_prompt + + +def test_readme_has_copy_paste_commands_for_each_launcher() -> None: + text = (SKILL_DIR / "README.md").read_text(encoding="utf-8") + windows = _section(text, "## Windows PowerShell", "## macOS / Linux Bash") + unix = _section(text, "## macOS / Linux Bash") + + windows_commands = [ + line.strip() for line in windows.splitlines() if line.startswith("py -3.12 ") + ] + unix_commands = [ + line.strip() for line in unix.splitlines() if line.startswith("python3 ") + ] + + assert len(windows_commands) == 5 + assert len(unix_commands) == 5 + assert "python3 " not in windows + assert "py -3.12 " not in unix + assert any("--terms-dir '\\Glossary'" in line for line in windows_commands) + assert any( + "--validate-article '\\Papers\\example.md'" in line + for line in windows_commands + ) + assert any("--terms-dir '/Glossary'" in line for line in unix_commands) + assert any( + "--validate-article '/Papers/example.md'" in line + for line in unix_commands + ) + for flag in ("--show", "--reset", "-m pytest"): + assert any(flag in line for line in windows_commands) + assert any(flag in line for line in unix_commands) + + +def test_writer_docs_use_configured_destination_only() -> None: + documents = [ + SKILL_DIR / "SKILL.md", + SKILL_DIR / "README.md", + SKILL_DIR / "references" / "file-contract.md", + ] + + for path in documents: + text = path.read_text(encoding="utf-8") + assert "configured glossary directory" in text + + +def test_test_suite_is_self_contained_within_skill() -> None: + forbidden_fragments = ( + "." + "superpowers", + "SKILL_DIR.parents" + "[1]", + ) + + for path in (SKILL_DIR / "tests").glob("test_*.py"): + text = path.read_text(encoding="utf-8") + for fragment in forbidden_fragments: + assert fragment not in text, f"{path.name} depends on repo-local scratch: {fragment}" + + +def test_workflow_integrations_use_config_backed_writer_cli() -> None: + text = (SKILL_DIR / "tests" / "test_workflow_integration.py").read_text( + encoding="utf-8" + ) + + assert "write_glossary_entries" not in text + assert 'SCRIPTS_DIR / "write_glossary_terms.py"' in text + assert text.count("_run_writer_cli(") == 4 + assert '"--config-path"' in text + assert text.count('"--triage"') == 2 + + +def test_contract_documents_current_json_and_required_manifest_override() -> None: + text = (SKILL_DIR / "references" / "file-contract.md").read_text(encoding="utf-8") + + assert "`--source-manifest` is always required" in text + assert "`--raw-sections` only overrides" in text + assert "cannot replace the manifest" in text + for field in ( + '"shortlist_limit"', + '"awaiting_semantic_review"', + '"existing_thin"', + '"operation": "enrich"', + '"article_sha256"', + '"already_linked"', + '"surface_forms"', + '"proposal_sha256"', + '"review_sha256"', + ): + assert field in text + + for flag in ("--review-proposal", "--reviewed-terms", "--reviewed-shortlist"): + assert flag in text + + +def test_docs_stop_when_semantic_review_drops_every_candidate() -> None: + documents = [ + SKILL_DIR / "SKILL.md", + SKILL_DIR / "README.md", + SKILL_DIR / "references" / "file-contract.md", + ] + + for path in documents: + text = path.read_text(encoding="utf-8") + assert "empty reviewed shortlist" in text + assert "`no_candidates`" in text + + +def test_post_triage_docs_define_current_chain_digest_and_writer_contexts() -> None: + documents = [ + SKILL_DIR / "SKILL.md", + SKILL_DIR / "README.md", + SKILL_DIR / "references" / "file-contract.md", + ] + + for path in documents: + text = path.read_text(encoding="utf-8") + assert "current `--source-manifest`" in text + assert "saved `--reviewed-shortlist`" in text + assert "saved `--triage`" in text + assert "`mappings_sha256`" in text + assert "`triage_sha256`" in text + assert "exact ordered `term` and `surface_forms`" in text + assert "derived from the resolved article Markdown stem" in text + assert "validated manifest `paper_id`" in text + assert "no bound article path" in text + assert "--paper-link" not in text + + +def test_plan_parser_rejects_raw_sections_without_required_manifest() -> None: + script = SKILL_DIR / "scripts" / "plan_glossary.py" + result = subprocess.run( + [sys.executable, str(script), "--propose", "--raw-sections", "paper.jsonl"], + capture_output=True, + text=True, + encoding="utf-8", + check=False, + ) + + assert result.returncode == 2 + assert "the following arguments are required: --source-manifest" in result.stderr + + +def test_optimization_design_preserves_reviewed_candidates_and_forms() -> None: + text = (SKILL_DIR / "references" / "optimization-design.md").read_text( + encoding="utf-8" + ) + review = _section(text, "### Semantic review", "## Selection and Generation") + + assert "only drop or reorder exact proposal candidates" in review + assert "exact `term`" in review + assert "ordered `surface_forms`" in review + assert "may not merge aliases" in review + assert "remove residual noise, merge aliases" not in review + + +def test_optimization_design_uses_config_only_writer_authority() -> None: + text = (SKILL_DIR / "references" / "optimization-design.md").read_text( + encoding="utf-8" + ) + configuration = _section(text, "## First-use Configuration", "## Candidate Workflow") + + assert "Writer destination authority comes only from validated device config" in configuration + assert "`--config-path` selects a config file" in configuration + assert "reset and configure" in configuration + assert "explicit path supplied for the current run overrides" not in configuration + assert "transient destination override" not in configuration + + +def test_implementation_plan_step5_inventory_cli_lists_every_required_flag() -> None: + text = (SKILL_DIR / "references" / "optimization-implementation-plan.md").read_text( + encoding="utf-8" + ) + task3 = _section(text, "### Task 3:", "### Task 4:") + inventory_step = _section( + task3, + "- [ ] **Step 5: Add inventory CLI**", + "- [ ] **Step 6:", + ) + + for flag in ( + "--terms", + "--terms-dir", + "--source-manifest", + "--raw-sections", + "--reviewed-shortlist", + "--output", + ): + assert flag in inventory_step + assert "triage JSON" in inventory_step + assert "optional `--raw-sections`" in inventory_step + + +def test_implementation_plan_documents_authenticated_integration_flow() -> None: + text = (SKILL_DIR / "references" / "optimization-implementation-plan.md").read_text( + encoding="utf-8" + ) + integration = _section(text, "### Task 6:", "### Task 7:") + + assert "proposal -> recorded review -> explicit selection -> triage -> authenticated inventory" in integration + assert "config-backed writer -> authenticated linker" in integration + for helper in ( + "inspect_selected_terms(", + "write_glossary_entries(", + "link_article_terms(", + ): + assert helper not in integration + + +def test_file_contract_writer_example_is_parseable_and_consistently_bound() -> None: + text = (SKILL_DIR / "references" / "file-contract.md").read_text(encoding="utf-8") + writer_section = text.split( + "`write_glossary_terms.py` returns successful results", 1 + )[1] + match = re.search( + r"```json\r?\n(?P.*?)\r?\n```", + writer_section, + re.DOTALL, + ) + assert match is not None + payload = json.loads(match.group("payload")) + + proposal = payload["provenance"]["proposal"] + review = payload["provenance"]["review"] + result = payload["results"][0] + candidate = proposal["candidates"][0] + + assert payload["paper_id"] == proposal["paper_id"] == review["paper_id"] + assert payload["context"]["paper_id"] == payload["paper_id"] + assert proposal["candidates"] == review["candidates"] + assert review["reviewed_shortlist"] == [candidate] + assert candidate["term"] == result["name"] == result["link_stem"] == "MoE" + assert candidate["surface_forms"] == result["forms"] == ["MoE", "ＭｏＥ"] + assert payload["provenance"]["selection_sha256"] == "sha256" + assert payload["triage_sha256"] == "sha256" + assert payload["mappings_sha256"] == "sha256" diff --git a/skills/paper-glossary/tests/test_workflow_integration.py b/skills/paper-glossary/tests/test_workflow_integration.py new file mode 100644 index 0000000..7343a9a --- /dev/null +++ b/skills/paper-glossary/tests/test_workflow_integration.py @@ -0,0 +1,399 @@ +from __future__ import annotations + +import json +import subprocess +import sys +from pathlib import Path + +import glossary_common +from conftest import build_current_inventory, write_current_workflow +from glossary_common import elapsed_ms +from glossary_config import configure_terms_dir, validate_article +from lint_glossary import lint_term_file_text +from write_glossary_terms import render_term_file + + +SCRIPTS_DIR = Path(__file__).resolve().parents[1] / "scripts" + + +def _run_writer_cli( + tmp_path: Path, + entries: dict, + inventory: dict, + config_path: Path, + source_manifest: Path, + reviewed_shortlist: Path, + triage: Path, + article: Path | None = None, +) -> dict: + glossary_path = tmp_path / "writer_glossary.json" + inventory_path = tmp_path / "writer_inventory.json" + output_path = tmp_path / "writer_output.json" + glossary_path.write_text( + json.dumps(entries, ensure_ascii=False), encoding="utf-8" + ) + inventory_path.write_text( + json.dumps(inventory, ensure_ascii=False), encoding="utf-8" + ) + arguments = [ + sys.executable, + str(SCRIPTS_DIR / "write_glossary_terms.py"), + "--glossary", + str(glossary_path), + "--inventory", + str(inventory_path), + "--config-path", + str(config_path), + "--source-manifest", + str(source_manifest), + "--reviewed-shortlist", + str(reviewed_shortlist), + "--triage", + str(triage), + "--output", + str(output_path), + ] + if article is not None: + arguments.extend(["--article", str(article)]) + subprocess.run( + arguments, + check=True, + capture_output=True, + text=True, + encoding="utf-8", + ) + return json.loads(output_path.read_text(encoding="utf-8")) + + +def _run_linker_cli( + tmp_path: Path, + article: Path, + expected_sha256: str, + config_path: Path, + source_manifest: Path, + reviewed_shortlist: Path, + triage: Path, +) -> dict: + output_path = tmp_path / "linker_output.json" + subprocess.run( + [ + sys.executable, + str(SCRIPTS_DIR / "link_glossary_terms.py"), + "--input", + str(article), + "--write-result", + str(tmp_path / "writer_output.json"), + "--expected-sha256", + expected_sha256, + "--config-path", + str(config_path), + "--source-manifest", + str(source_manifest), + "--reviewed-shortlist", + str(reviewed_shortlist), + "--triage", + str(triage), + "--output", + str(output_path), + ], + check=True, + capture_output=True, + text=True, + encoding="utf-8", + ) + return json.loads(output_path.read_text(encoding="utf-8")) + + +def test_elapsed_ms_returns_non_negative_integer(monkeypatch) -> None: + monkeypatch.setattr(glossary_common, "perf_counter", lambda: 2.125) + + assert elapsed_ms(2.0) == 125 + assert elapsed_ms(3.0) == 0 + assert isinstance(elapsed_ms(2.0), int) + + +def test_complete_selected_flow_creates_enriches_links_and_lints( + tmp_path: Path, +) -> None: + vault = tmp_path / "vault" + (vault / ".obsidian").mkdir(parents=True) + article = vault / "Chapter.md" + article.write_text( + "# Agent Patterns\n\nReAct and Reflection are two agent patterns.\n", + encoding="utf-8", + ) + terms_dir = vault / "book" / "Glossary" + config_path = tmp_path / "config.json" + config = configure_terms_dir(terms_dir, config_path) + article_info = validate_article(article, config) + + reflection = terms_dir / "Reflection.md" + reflection.write_text( + render_term_file( + { + "name": "Reflection", + "aliases": ["self-reflection"], + "definition": "An agent pattern that reviews prior results.", + "confidence": "\u9ad8", + "occurrence": "Earlier source evidence.", + }, + "Earlier Chapter", + ), + encoding="utf-8", + ) + selected = [ + {"term": "ReAct", "surface_forms": ["ReAct"]}, + {"term": "Reflection", "surface_forms": ["Reflection"]}, + ] + workflow = write_current_workflow( + tmp_path / "workflow", [item["term"] for item in selected] + ) + inventory = build_current_inventory(workflow, terms_dir) + entries = { + "entries": [ + { + "name": "ReAct", + "aliases": ["Reasoning and Acting"], + "operation": "create", + "definition": "An agent pattern that interleaves reasoning and action.", + "confidence": "\u9ad8", + "occurrence": "The chapter introduces ReAct.", + }, + { + "name": "Reflection", + "aliases": ["self-reflection"], + "operation": "enrich", + "elaboration": "The agent uses feedback to improve a prior result.", + "intuition": "Review the last attempt before trying again.", + "distinction": "Unlike ReAct, it focuses on reviewing prior output.", + "occurrence": "The chapter presents Reflection as an agent pattern.", + }, + ] + } + + write_result = _run_writer_cli( + tmp_path, + entries, + inventory, + config_path, + workflow["manifest_path"], + workflow["review_path"], + workflow["triage_path"], + article, + ) + assert write_result["script"] == "write_glossary_terms.py" + assert write_result["terms_dir"] == str(terms_dir.resolve()) + link_result = _run_linker_cli( + tmp_path, + article, + article_info["article_sha256"], + config_path, + workflow["manifest_path"], + workflow["review_path"], + workflow["triage_path"], + ) + changed_notes = [ + Path(item["file"]) + for item in write_result["results"] + if item["action"] in {"created", "enriched", "updated"} + ] + lint_results = [ + lint_term_file_text(path.read_text(encoding="utf-8-sig")) + for path in changed_notes + ] + + assert [item["action"] for item in write_result["results"]] == [ + "created", + "enriched", + ] + assert link_result["summary"] == { + "linked": 2, + "already_linked": 0, + "not_found": 0, + } + article_text = article.read_text(encoding="utf-8") + assert "[[ReAct]]" in article_text + assert "[[Reflection]]" in article_text + assert len(changed_notes) == 2 + assert all(result["passes"] for result in lint_results) + + +def test_model_only_alias_is_note_metadata_but_never_an_article_match_form( + tmp_path: Path, +) -> None: + vault = tmp_path / "vault" + (vault / ".obsidian").mkdir(parents=True) + article = vault / "Chapter.md" + article.write_text("# Paper\n\nReasoning and Acting appears here.\n", encoding="utf-8") + terms_dir = vault / "Glossary" + config_path = tmp_path / "config.json" + config = configure_terms_dir(terms_dir, config_path) + article_info = validate_article(article, config) + workflow = write_current_workflow( + tmp_path / "workflow", + ["ReAct"], + source_text="**ReAct** and **\uff32\uff45\uff21\uff43\uff54** are equivalent forms.", + ) + inventory = build_current_inventory(workflow, terms_dir) + + write_result = _run_writer_cli( + tmp_path, + { + "entries": [ + { + "name": "ReAct", + "aliases": ["Reasoning and Acting"], + "operation": "create", + "definition": "A reasoning and acting pattern.", + "confidence": "高", + "occurrence": "The paper uses ReAct.", + } + ] + }, + inventory, + config_path, + workflow["manifest_path"], + workflow["review_path"], + workflow["triage_path"], + article, + ) + + link_result = _run_linker_cli( + tmp_path, + article, + article_info["article_sha256"], + config_path, + workflow["manifest_path"], + workflow["review_path"], + workflow["triage_path"], + ) + + assert write_result["results"][0]["forms"] == ["ReAct", "ＲｅＡｃｔ"] + assert ' - "Reasoning and Acting"' in (terms_dir / "ReAct.md").read_text( + encoding="utf-8" + ) + assert link_result["summary"] == { + "linked": 0, + "already_linked": 0, + "not_found": 1, + } + assert "[[" not in article.read_text(encoding="utf-8") + + +def test_grounded_equivalent_form_survives_the_complete_artifact_chain( + tmp_path: Path, +) -> None: + raw = tmp_path / "paper_raw_sections.jsonl" + raw.write_text( + json.dumps( + { + "record_type": "section", + "section_id": "sec:method", + "kind": "method", + "title": "Method", + "text": "**ReAct** and **ＲｅＡｃｔ** are equivalent source spellings.", + }, + ensure_ascii=False, + ), + encoding="utf-8", + ) + manifest = tmp_path / "paper_source_manifest.json" + manifest.write_text( + json.dumps({"paper_id": "paper-1", "raw_sections_path": str(raw)}), + encoding="utf-8", + ) + proposal = tmp_path / "proposal.json" + reviewed = tmp_path / "reviewed.json" + triage = tmp_path / "triage.json" + inventory_path = tmp_path / "inventory.json" + vault = tmp_path / "vault" + (vault / ".obsidian").mkdir(parents=True) + terms_dir = vault / "terms" + config_path = tmp_path / "config.json" + configure_terms_dir(terms_dir, config_path) + + commands = [ + [ + sys.executable, + str(SCRIPTS_DIR / "plan_glossary.py"), + "--propose", + "--source-manifest", + str(manifest), + "--output", + str(proposal), + ], + [ + sys.executable, + str(SCRIPTS_DIR / "plan_glossary.py"), + "--review-proposal", + str(proposal), + "--reviewed-terms", + '["ReAct"]', + "--source-manifest", + str(manifest), + "--output", + str(reviewed), + ], + [ + sys.executable, + str(SCRIPTS_DIR / "plan_glossary.py"), + "--reviewed-shortlist", + str(reviewed), + "--terms", + '["ReAct"]', + "--source-manifest", + str(manifest), + "--output", + str(triage), + ], + [ + sys.executable, + str(SCRIPTS_DIR / "inspect_glossary_library.py"), + "--terms", + str(triage), + "--terms-dir", + str(terms_dir), + "--source-manifest", + str(manifest), + "--reviewed-shortlist", + str(reviewed), + "--output", + str(inventory_path), + ], + ] + for command in commands: + subprocess.run(command, check=True, capture_output=True, text=True, encoding="utf-8") + + proposal_payload = json.loads(proposal.read_text(encoding="utf-8")) + reviewed_payload = json.loads(reviewed.read_text(encoding="utf-8")) + triage_payload = json.loads(triage.read_text(encoding="utf-8")) + inventory = json.loads(inventory_path.read_text(encoding="utf-8")) + write_result = _run_writer_cli( + tmp_path, + { + "entries": [ + { + "name": "ReAct", + "aliases": [], + "operation": "create", + "definition": "A reasoning and acting pattern.", + "confidence": "高", + "occurrence": "The paper uses both forms.", + } + ] + }, + inventory, + config_path, + manifest, + reviewed, + triage, + ) + + assert write_result["script"] == "write_glossary_terms.py" + assert write_result["terms_dir"] == str(terms_dir.resolve()) + forms = ["ReAct", "ＲｅＡｃｔ"] + assert proposal_payload["candidates"][0]["surface_forms"] == forms + assert reviewed_payload["reviewed_shortlist"][0]["surface_forms"] == forms + assert triage_payload["terms"][0]["surface_forms"] == forms + assert inventory["results"][0]["surface_forms"] == forms + assert write_result["results"][0]["forms"] == forms diff --git a/skills/paper-glossary/tests/test_write_and_lint_glossary.py b/skills/paper-glossary/tests/test_write_and_lint_glossary.py index 041095a..7493c09 100644 --- a/skills/paper-glossary/tests/test_write_and_lint_glossary.py +++ b/skills/paper-glossary/tests/test_write_and_lint_glossary.py @@ -1,17 +1,36 @@ from __future__ import annotations +from copy import deepcopy +import json +import os +import sys from pathlib import Path import pytest +import write_glossary_terms as writer_module +from conftest import ( + bind_inventory_results, + build_current_inventory, + build_inventory_payload, + write_current_workflow, +) -from glossary_contracts import GLOSSARY_OCCURRENCE_HEADING +from glossary_config import configure_terms_dir +from glossary_common import selection_sha256 +from glossary_contracts import ( + GLOSSARY_CONCEPT_HEADING, + GLOSSARY_OCCURRENCE_HEADING, +) +from glossary_library import inspect_selected_terms from lint_glossary import GLOSSARY_DISCLAIMER, lint_term_file_text, main as lint_main from write_glossary_terms import ( append_occurrence, build_alias_index, + main as write_main, render_term_file, safe_term_filename, upsert_term_file, + write_glossary_entries, ) ENTRY = { @@ -30,6 +49,10 @@ def _codes(result: dict) -> set[str]: return {issue["code"] for issue in result["issues"]} +def _inventory(results: list[dict]) -> dict: + return bind_inventory_results(results) + + def test_render_term_file_is_lintable() -> None: text = render_term_file(ENTRY, "CAD-MoE") result = lint_term_file_text(text) @@ -143,6 +166,1823 @@ def test_lint_main_rejects_empty_terms_dir( assert not output.exists() +def test_lint_main_accepts_repeated_inputs_and_lints_exact_files( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + first = tmp_path / "First.md" + second = tmp_path / "Second.md" + ignored = tmp_path / "Ignored.md" + for path, name in ((first, "First"), (second, "Second"), (ignored, "Ignored")): + path.write_text(render_term_file({**ENTRY, "name": name}, "Paper"), encoding="utf-8") + output = tmp_path / "lint.json" + monkeypatch.setattr( + sys, + "argv", + [ + "lint_glossary.py", + "--input", + str(first), + "--input", + str(second), + "--output", + str(output), + ], + ) + + lint_main() + + payload = json.loads(output.read_text(encoding="utf-8")) + assert isinstance(payload["elapsed_ms"], int) + assert payload["elapsed_ms"] >= 0 + assert [item["path"] for item in payload["files"]] == [ + str(first.resolve()), + str(second.resolve()), + ] + assert payload["summary"] == {"total": 2, "passed": 2, "failed": 0} + + +def test_lint_main_rejects_missing_terms_dir_even_with_valid_input( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + note = tmp_path / "Term.md" + note.write_text(render_term_file({**ENTRY, "name": "Term"}, "Paper"), encoding="utf-8") + missing_terms = tmp_path / "missing" + monkeypatch.setattr( + sys, + "argv", + [ + "lint_glossary.py", + "--input", + str(note), + "--terms-dir", + str(missing_terms), + ], + ) + + with pytest.raises(SystemExit, match="--terms-dir must be an existing directory"): + lint_main() + + +def test_lint_main_rejects_file_terms_dir_even_with_valid_input( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + note = tmp_path / "Term.md" + note.write_text(render_term_file({**ENTRY, "name": "Term"}, "Paper"), encoding="utf-8") + terms_file = tmp_path / "not-a-directory" + terms_file.write_text("not a directory", encoding="utf-8") + monkeypatch.setattr( + sys, + "argv", + [ + "lint_glossary.py", + "--input", + str(note), + "--terms-dir", + str(terms_file), + ], + ) + + with pytest.raises(SystemExit, match="--terms-dir must be an existing directory"): + lint_main() + + +def test_lint_main_terms_dir_only_emits_elapsed_ms( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + terms_dir = tmp_path / "terms" + terms_dir.mkdir() + note = terms_dir / "Term.md" + note.write_text(render_term_file({**ENTRY, "name": "Term"}, "Paper"), encoding="utf-8") + output = tmp_path / "lint.json" + monkeypatch.setattr( + sys, + "argv", + [ + "lint_glossary.py", + "--terms-dir", + str(terms_dir), + "--output", + str(output), + ], + ) + + lint_main() + + payload = json.loads(output.read_text(encoding="utf-8")) + assert isinstance(payload["elapsed_ms"], int) + assert payload["elapsed_ms"] >= 0 + assert [item["path"] for item in payload["files"]] == [str(note.resolve())] + + +def test_lint_main_merges_terms_dir_and_deduplicates_resolved_inputs( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + terms_dir = tmp_path / "terms" + terms_dir.mkdir() + first = terms_dir / "First.md" + second = terms_dir / "Second.md" + for path, name in ((first, "First"), (second, "Second")): + path.write_text(render_term_file({**ENTRY, "name": name}, "Paper"), encoding="utf-8") + output = tmp_path / "lint.json" + monkeypatch.setattr( + sys, + "argv", + [ + "lint_glossary.py", + "--input", + str(first), + "--input", + str(first.parent / "." / first.name), + "--terms-dir", + str(terms_dir), + "--output", + str(output), + ], + ) + + lint_main() + + payload = json.loads(output.read_text(encoding="utf-8")) + assert isinstance(payload["elapsed_ms"], int) + assert payload["elapsed_ms"] >= 0 + assert [item["path"] for item in payload["files"]] == [ + str(first.resolve()), + str(second.resolve()), + ] + assert payload["summary"]["total"] == 2 + + +def test_write_cli_emits_elapsed_ms(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: + workflow = write_current_workflow(tmp_path / "workflow", ["ReAct"]) + vault = tmp_path / "vault" + (vault / ".obsidian").mkdir(parents=True) + terms_dir = vault / "Glossary" + config_path = tmp_path / "config.json" + configure_terms_dir(terms_dir, config_path) + glossary = tmp_path / "glossary.json" + inventory = tmp_path / "inventory.json" + output = tmp_path / "write.json" + glossary.write_text( + json.dumps( + { + "entries": [ + { + "name": "ReAct", + "operation": "create", + "definition": "Reasoning and acting pattern.", + "confidence": "\u9ad8", + "occurrence": "Chapter evidence.", + } + ] + } + ), + encoding="utf-8", + ) + inventory.write_text( + json.dumps(build_current_inventory(workflow, terms_dir)), + encoding="utf-8", + ) + monkeypatch.setattr( + sys, + "argv", + [ + "write_glossary_terms.py", + "--glossary", + str(glossary), + "--inventory", + str(inventory), + "--config-path", + str(config_path), + "--source-manifest", + str(workflow["manifest_path"]), + "--reviewed-shortlist", + str(workflow["review_path"]), + "--triage", + str(workflow["triage_path"]), + "--output", + str(output), + ], + ) + + write_main() + + payload = json.loads(output.read_text(encoding="utf-8")) + assert isinstance(payload["elapsed_ms"], int) + assert payload["elapsed_ms"] >= 0 + assert payload["results"][0]["action"] == "created" + + +@pytest.mark.parametrize( + ("authorized_names", "generated_names", "source_text"), + [ + (["ReAct", "Reflection"], ["Reflection", "ReAct"], None), + ( + ["ReAct"], + ["react"], + "**ReAct** and **react** are the same paper concept.", + ), + ], + ids=["reversed-entries", "case-only-name"], +) +def test_writer_cli_rejects_generated_mapping_drift_before_writes( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, + authorized_names: list[str], + generated_names: list[str], + source_text: str | None, +) -> None: + workflow = write_current_workflow( + tmp_path / "workflow", authorized_names, source_text=source_text + ) + vault = tmp_path / "vault" + (vault / ".obsidian").mkdir(parents=True) + terms_dir = vault / "Glossary" + terms_dir.mkdir() + (terms_dir / ".sentinel").write_bytes(b"unchanged glossary directory\n") + if source_text is not None: + (terms_dir / "ReAct.md").write_text( + render_term_file( + { + "name": "ReAct", + "aliases": ["react"], + "definition": "A reasoning and acting pattern.", + "elaboration": "It interleaves reasoning with actions.", + "intuition": "Think, act, and observe repeatedly.", + "distinction": "It is not a single reasoning pass.", + "confidence": "\u9ad8", + "occurrence": "Existing occurrence.", + }, + "ExistingPaper", + ), + encoding="utf-8", + ) + config_path = tmp_path / "config.json" + configure_terms_dir(terms_dir, config_path) + inventory_path = tmp_path / "inventory.json" + inventory_path.write_text( + json.dumps(build_current_inventory(workflow, terms_dir)), encoding="utf-8" + ) + glossary_path = tmp_path / "generated-glossary.json" + generated_entries = [] + for name in generated_names: + entry = { + "name": name, + "operation": "reuse" if source_text is not None else "create", + "occurrence": f"Occurrence for {name}.", + } + if entry["operation"] == "create": + entry.update( + { + "definition": f"Definition for {name}.", + "confidence": "\u9ad8", + } + ) + generated_entries.append(entry) + glossary_path.write_text( + json.dumps({"entries": generated_entries}), + encoding="utf-8", + ) + before = { + path.relative_to(terms_dir): path.read_bytes() + for path in terms_dir.rglob("*") + if path.is_file() + } + monkeypatch.setattr( + sys, + "argv", + [ + "write_glossary_terms.py", + "--glossary", + str(glossary_path), + "--inventory", + str(inventory_path), + "--config-path", + str(config_path), + "--source-manifest", + str(workflow["manifest_path"]), + "--reviewed-shortlist", + str(workflow["review_path"]), + "--triage", + str(workflow["triage_path"]), + ], + ) + + with pytest.raises(SystemExit, match="ordered inventory"): + try: + write_main() + finally: + after = { + path.relative_to(terms_dir): path.read_bytes() + for path in terms_dir.rglob("*") + if path.is_file() + } + assert after == before + + +@pytest.mark.parametrize( + ("with_article", "expected_paper_link"), + [(True, "Paper"), (False, "paper-fixture")], + ids=["article", "glossary-only"], +) +def test_writer_cli_derives_and_binds_backlink_context( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, + with_article: bool, + expected_paper_link: str, +) -> None: + workflow = write_current_workflow(tmp_path / "workflow", ["ReAct"]) + vault = tmp_path / "vault" + (vault / ".obsidian").mkdir(parents=True) + terms_dir = vault / "Glossary" + config_path = tmp_path / "config.json" + configure_terms_dir(terms_dir, config_path) + article = vault / "Paper.md" + article.write_text("# Paper\n\nReAct text.\n", encoding="utf-8") + glossary_path = tmp_path / "glossary.json" + inventory_path = tmp_path / "inventory.json" + output_path = tmp_path / "writer.json" + glossary_path.write_text( + json.dumps( + { + "entries": [ + { + "name": "ReAct", + "operation": "create", + "definition": "A reasoning and acting pattern.", + "confidence": "\u9ad8", + "occurrence": "Paper evidence.", + } + ] + } + ), + encoding="utf-8", + ) + inventory_path.write_text( + json.dumps(build_current_inventory(workflow, terms_dir)), encoding="utf-8" + ) + arguments = [ + "write_glossary_terms.py", + "--glossary", + str(glossary_path), + "--inventory", + str(inventory_path), + "--config-path", + str(config_path), + "--source-manifest", + str(workflow["manifest_path"]), + "--reviewed-shortlist", + str(workflow["review_path"]), + "--triage", + str(workflow["triage_path"]), + "--output", + str(output_path), + ] + if with_article: + arguments.extend(["--article", str(article)]) + monkeypatch.setattr(sys, "argv", arguments) + + write_main() + + payload = json.loads(output_path.read_text(encoding="utf-8")) + assert payload["context"] == { + "paper_id": "paper-fixture", + "paper_link": expected_paper_link, + "article_path": str(article.resolve()) if with_article else "", + } + assert len(payload["triage_sha256"]) == 64 + assert len(payload["mappings_sha256"]) == 64 + assert f"[[{expected_paper_link}]]" in (terms_dir / "ReAct.md").read_text( + encoding="utf-8" + ) + + +def test_writer_cli_rejects_unsafe_glossary_only_paper_id_without_writes( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + workflow = write_current_workflow( + tmp_path / "workflow", ["ReAct"], paper_id="unsafe|paper" + ) + vault = tmp_path / "vault" + (vault / ".obsidian").mkdir(parents=True) + terms_dir = vault / "Glossary" + config_path = tmp_path / "config.json" + configure_terms_dir(terms_dir, config_path) + glossary_path = tmp_path / "glossary.json" + inventory_path = tmp_path / "inventory.json" + glossary_path.write_text( + json.dumps( + { + "entries": [ + { + "name": "ReAct", + "operation": "create", + "definition": "definition", + "confidence": "\u9ad8", + "occurrence": "evidence", + } + ] + } + ), + encoding="utf-8", + ) + inventory_path.write_text( + json.dumps(build_current_inventory(workflow, terms_dir)), encoding="utf-8" + ) + monkeypatch.setattr( + sys, + "argv", + [ + "write_glossary_terms.py", + "--glossary", + str(glossary_path), + "--inventory", + str(inventory_path), + "--config-path", + str(config_path), + "--source-manifest", + str(workflow["manifest_path"]), + "--reviewed-shortlist", + str(workflow["review_path"]), + "--triage", + str(workflow["triage_path"]), + ], + ) + + with pytest.raises(SystemExit, match="paper_id|wiki|backlink"): + write_main() + + assert not (terms_dir / "ReAct.md").exists() + + +def test_writer_cli_rejects_changed_source_before_loading_generated_glossary( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + workflow = write_current_workflow(tmp_path / "workflow", ["ReAct"]) + vault = tmp_path / "vault" + (vault / ".obsidian").mkdir(parents=True) + terms_dir = vault / "Glossary" + config_path = tmp_path / "config.json" + configure_terms_dir(terms_dir, config_path) + inventory_path = tmp_path / "inventory.json" + inventory_path.write_text( + json.dumps(build_current_inventory(workflow, terms_dir)), encoding="utf-8" + ) + workflow["raw_path"].write_text( + workflow["raw_path"].read_text(encoding="utf-8") + + json.dumps( + { + "record_type": "section", + "section_id": "sec:changed", + "kind": "body", + "title": "Changed", + "text": "Source changed after inventory.", + } + ) + + "\n", + encoding="utf-8", + ) + malformed_glossary = tmp_path / "malformed-glossary.json" + malformed_glossary.write_text("{", encoding="utf-8") + monkeypatch.setattr( + sys, + "argv", + [ + "write_glossary_terms.py", + "--glossary", + str(malformed_glossary), + "--inventory", + str(inventory_path), + "--config-path", + str(config_path), + "--source-manifest", + str(workflow["manifest_path"]), + "--reviewed-shortlist", + str(workflow["review_path"]), + "--triage", + str(workflow["triage_path"]), + ], + ) + + with pytest.raises(SystemExit, match="current|source|review|proposal"): + write_main() + + assert not (terms_dir / "ReAct.md").exists() + + +def test_writer_cli_rejects_reviewed_but_unselected_forgery_before_loading_glossary( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + workflow = write_current_workflow( + tmp_path / "workflow", + ["ReAct", "Reflection"], + selected_names=["ReAct"], + ) + vault = tmp_path / "vault" + (vault / ".obsidian").mkdir(parents=True) + terms_dir = vault / "Glossary" + config_path = tmp_path / "config.json" + configure_terms_dir(terms_dir, config_path) + + reflection = next( + item + for item in workflow["review"]["reviewed_shortlist"] + if item["term"] == "Reflection" + ) + forged_results = inspect_selected_terms([reflection], terms_dir) + forged_provenance = deepcopy(workflow["triage"]["provenance"]) + forged_provenance["selection_sha256"] = selection_sha256( + forged_provenance["review"]["review_sha256"], forged_results + ) + inventory_path = tmp_path / "forged-inventory.json" + inventory_path.write_text( + json.dumps( + { + "status": "ok", + "script": "inspect_glossary_library.py", + "paper_id": workflow["triage"]["paper_id"], + "provenance": forged_provenance, + "results": forged_results, + } + ), + encoding="utf-8", + ) + glossary_path = tmp_path / "generated-glossary.json" + glossary_path.write_text( + json.dumps( + { + "entries": [ + { + "name": "Reflection", + "operation": "create", + "definition": "A reflection step.", + "confidence": "\u9ad8", + "occurrence": "Forged occurrence.", + } + ] + } + ), + encoding="utf-8", + ) + generated_loaded = False + real_loader = writer_module.maybe_load_json_record + + def tracked_loader(value: str) -> dict | None: + nonlocal generated_loaded + if value == str(glossary_path): + generated_loaded = True + return real_loader(value) + + monkeypatch.setattr(writer_module, "maybe_load_json_record", tracked_loader) + monkeypatch.setattr( + sys, + "argv", + [ + "write_glossary_terms.py", + "--glossary", + str(glossary_path), + "--inventory", + str(inventory_path), + "--config-path", + str(config_path), + "--source-manifest", + str(workflow["manifest_path"]), + "--reviewed-shortlist", + str(workflow["review_path"]), + "--triage", + str(workflow["triage_path"]), + ], + ) + + with pytest.raises(SystemExit, match="authorized|triage|selection"): + write_main() + + assert generated_loaded is False + assert not (terms_dir / "Reflection.md").exists() + + +@pytest.mark.parametrize("vault_root", [".", "C:drive-relative"]) +def test_writer_cli_rejects_relative_vault_root_without_writes( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, + vault_root: str, +) -> None: + workflow = write_current_workflow(tmp_path / "workflow", ["ReAct"]) + vault = tmp_path / "vault" + (vault / ".obsidian").mkdir(parents=True) + terms_dir = vault / "Glossary" + terms_dir.mkdir() + config_path = tmp_path / "config.json" + config_path.write_text( + json.dumps({"vault_root": vault_root, "terms_subdir": "Glossary"}), + encoding="utf-8", + ) + glossary_path = tmp_path / "glossary.json" + inventory_path = tmp_path / "inventory.json" + glossary_path.write_text( + json.dumps( + { + "entries": [ + { + "name": "ReAct", + "operation": "create", + "definition": "definition", + "confidence": "\u9ad8", + "occurrence": "evidence", + } + ] + } + ), + encoding="utf-8", + ) + inventory_path.write_text( + json.dumps(build_current_inventory(workflow, terms_dir)), encoding="utf-8" + ) + monkeypatch.chdir(vault) + monkeypatch.setattr( + sys, + "argv", + [ + "write_glossary_terms.py", + "--glossary", + str(glossary_path), + "--inventory", + str(inventory_path), + "--config-path", + str(config_path), + "--source-manifest", + str(workflow["manifest_path"]), + "--reviewed-shortlist", + str(workflow["review_path"]), + "--triage", + str(workflow["triage_path"]), + ], + ) + + with pytest.raises(SystemExit, match="Invalid paper-glossary config object"): + write_main() + + assert not (terms_dir / "ReAct.md").exists() + + +def test_write_cli_rejects_missing_config_before_creating_terms_dir( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + terms_dir = tmp_path / "arbitrary-destination" + glossary = tmp_path / "glossary.json" + inventory = tmp_path / "inventory.json" + glossary.write_text( + json.dumps( + { + "entries": [ + { + "name": "ReAct", + "operation": "create", + "definition": "Reasoning and acting pattern.", + "confidence": "\u9ad8", + "occurrence": "Chapter evidence.", + } + ] + } + ), + encoding="utf-8", + ) + inventory.write_text( + json.dumps(build_inventory_payload([{"term": "ReAct", "surface_forms": ["ReAct"]}], terms_dir)), + encoding="utf-8", + ) + monkeypatch.setattr( + sys, + "argv", + [ + "write_glossary_terms.py", + "--glossary", + str(glossary), + "--inventory", + str(inventory), + "--config-path", + str(tmp_path / "missing-config.json"), + "--source-manifest", + str(tmp_path / "unused-manifest.json"), + "--reviewed-shortlist", + str(tmp_path / "unused-review.json"), + "--triage", + str(tmp_path / "unused-triage.json"), + ], + ) + + with pytest.raises(SystemExit, match="configuration"): + write_main() + + assert not terms_dir.exists() + + +@pytest.mark.parametrize( + "config_payload, terms_dir, error", + [ + ("{", "Glossary", "Invalid paper-glossary config"), + ( + {"vault_root": "{vault}", "terms_subdir": "Missing"}, + "Missing", + "Configured Obsidian term directory", + ), + ( + {"vault_root": "{non_vault}", "terms_subdir": "Glossary"}, + "Glossary", + "Configured Obsidian term directory", + ), + ( + {"vault_root": "{vault}", "terms_subdir": "../outside"}, + "../outside", + "Configured Obsidian term directory", + ), + ], + ids=["malformed", "stale", "non-obsidian", "outside-vault"], +) +def test_write_cli_rejects_invalid_config_before_mutating_glossary( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, + config_payload: str | dict[str, str], + terms_dir: str, + error: str, +) -> None: + vault = tmp_path / "vault" + (vault / ".obsidian").mkdir(parents=True) + non_vault = tmp_path / "not-a-vault" + (non_vault / "Glossary").mkdir(parents=True) + outside = tmp_path / "outside" + outside.mkdir() + glossary = tmp_path / "glossary.json" + inventory = tmp_path / "inventory.json" + config_path = tmp_path / "config.json" + glossary.write_text( + json.dumps( + { + "entries": [ + { + "name": "ReAct", + "operation": "create", + "definition": "Reasoning and acting pattern.", + "confidence": "\u9ad8", + "occurrence": "Chapter evidence.", + } + ] + } + ), + encoding="utf-8", + ) + inventory.write_text( + json.dumps( + build_inventory_payload( + [{"term": "ReAct", "surface_forms": ["ReAct"]}], vault / "Glossary" + ) + ), + encoding="utf-8", + ) + if isinstance(config_payload, str): + config_path.write_text(config_payload, encoding="utf-8") + else: + config_path.write_text( + json.dumps( + { + key: value.format(vault=vault, non_vault=non_vault) + for key, value in config_payload.items() + } + ), + encoding="utf-8", + ) + monkeypatch.setattr( + sys, + "argv", + [ + "write_glossary_terms.py", + "--glossary", + str(glossary), + "--inventory", + str(inventory), + "--config-path", + str(config_path), + "--source-manifest", + str(tmp_path / "unused-manifest.json"), + "--reviewed-shortlist", + str(tmp_path / "unused-review.json"), + "--triage", + str(tmp_path / "unused-triage.json"), + ], + ) + + with pytest.raises(SystemExit, match=error): + write_main() + + assert not (vault / "Glossary" / "ReAct.md").exists() + assert not (non_vault / "Glossary" / "ReAct.md").exists() + assert not (outside / "ReAct.md").exists() + assert not (vault / terms_dir / "ReAct.md").exists() + + +@pytest.mark.parametrize( + "case", + [ + "missing-vault-root", + "missing-terms-subdir", + "empty-vault-root", + "whitespace-vault-root", + "empty-terms-subdir", + "whitespace-terms-subdir", + "non-string-vault-root", + "non-string-terms-subdir", + "absolute-terms-subdir", + "root-dot-terms-subdir", + "root-normalized-terms-subdir", + ], +) +def test_write_cli_rejects_invalid_config_schema_without_writes( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch, case: str +) -> None: + vault = tmp_path / "vault" + (vault / ".obsidian").mkdir(parents=True) + terms_dir = vault / "Glossary" + terms_dir.mkdir() + outside = tmp_path / "outside" + outside.mkdir() + config_path = tmp_path / "config.json" + glossary = tmp_path / "glossary.json" + inventory = tmp_path / "inventory.json" + glossary.write_text( + json.dumps( + { + "entries": [ + { + "name": "ReAct", + "operation": "create", + "definition": "Reasoning and acting pattern.", + "confidence": "\u9ad8", + "occurrence": "Chapter evidence.", + } + ] + } + ), + encoding="utf-8", + ) + inventory.write_text( + json.dumps( + build_inventory_payload( + [{"term": "ReAct", "surface_forms": ["ReAct"]}], terms_dir + ) + ), + encoding="utf-8", + ) + payload: dict[str, object] = { + "vault_root": str(vault), + "terms_subdir": "Glossary", + } + if case == "missing-vault-root": + del payload["vault_root"] + elif case == "missing-terms-subdir": + del payload["terms_subdir"] + elif case == "empty-vault-root": + payload["vault_root"] = "" + elif case == "whitespace-vault-root": + payload["vault_root"] = " \t" + elif case == "empty-terms-subdir": + payload["terms_subdir"] = "" + elif case == "whitespace-terms-subdir": + payload["terms_subdir"] = " \t" + elif case == "non-string-vault-root": + payload["vault_root"] = 7 + elif case == "non-string-terms-subdir": + payload["terms_subdir"] = ["Glossary"] + elif case == "absolute-terms-subdir": + payload["terms_subdir"] = str(outside) + elif case == "root-dot-terms-subdir": + payload["terms_subdir"] = "." + elif case == "root-normalized-terms-subdir": + payload["terms_subdir"] = str(Path("Glossary") / "..") + config_path.write_text(json.dumps(payload), encoding="utf-8") + monkeypatch.setattr( + sys, + "argv", + [ + "write_glossary_terms.py", + "--glossary", + str(glossary), + "--inventory", + str(inventory), + "--config-path", + str(config_path), + "--source-manifest", + str(tmp_path / "unused-manifest.json"), + "--reviewed-shortlist", + str(tmp_path / "unused-review.json"), + "--triage", + str(tmp_path / "unused-triage.json"), + ], + ) + + error = None + try: + write_main() + except SystemExit as exc: + error = str(exc) + + assert not (vault / "ReAct.md").exists() + assert not (terms_dir / "ReAct.md").exists() + assert not (outside / "ReAct.md").exists() + assert error is not None + assert "Invalid paper-glossary config object" in error + + +def test_writer_preflight_rejects_invalid_batch_without_creating_files(tmp_path: Path) -> None: + terms_dir = tmp_path / "terms" + entries = { + "entries": [ + { + "name": "Valid", + "aliases": [], + "operation": "create", + "definition": "definition", + "confidence": "\u9ad8", + "occurrence": "evidence", + }, + { + "name": "Invalid", + "aliases": [], + "operation": "enrich", + "occurrence": "evidence", + }, + ] + } + inventory = _inventory( + [ + { + "term": "Valid", + "surface_forms": ["Valid"], + "state": "new", + "file": "", + "link_stem": "Valid", + "missing_fields": [], + }, + { + "term": "Invalid", + "surface_forms": ["Invalid"], + "state": "new", + "file": "", + "link_stem": "Invalid", + "missing_fields": [], + }, + ] + ) + + with pytest.raises(SystemExit, match="Invalid operation"): + write_glossary_entries(entries, inventory, terms_dir, "Paper") + + assert not terms_dir.exists() + + +def test_writer_enriches_only_missing_fields_and_appends_occurrence_once(tmp_path: Path) -> None: + terms_dir = tmp_path / "terms" + terms_dir.mkdir() + note = terms_dir / "LLM.md" + original = ( + "# LLM\n\n## \u6982\u5ff5\u89e3\u91ca\n- \u5b9a\u4e49\uff1auser definition\n- \u7f6e\u4fe1\u5ea6\uff1a\u4e2d\n" + "\n## Custom\nkeep this text\n" + ) + note.write_text(original, encoding="utf-8") + inventory = _inventory( + [ + { + "term": "LLM", + "surface_forms": ["LLM", "large language model"], + "state": "existing_thin", + "file": str(note), + "link_stem": "LLM", + "missing_fields": ["elaboration", "intuition", "distinction"], + } + ] + ) + entries = { + "entries": [ + { + "name": "LLM", + "aliases": ["large language model"], + "operation": "enrich", + "elaboration": "new explanation", + "intuition": "new intuition", + "occurrence": "paper evidence", + } + ] + } + + first = write_glossary_entries(entries, inventory, terms_dir, "Paper") + first_bytes = note.read_bytes() + refreshed_inventory = _inventory( + inspect_selected_terms( + [{"term": "LLM", "surface_forms": ["LLM", "large language model"]}], + terms_dir, + ) + ) + second = write_glossary_entries( + { + "entries": [ + { + "name": "LLM", + "aliases": ["large language model"], + "operation": "reuse", + "occurrence": "paper evidence", + } + ] + }, + refreshed_inventory, + terms_dir, + "Paper", + ) + + text = note.read_text(encoding="utf-8") + assert "- \u5b9a\u4e49\uff1auser definition" in text + assert "## Custom\nkeep this text" in text + assert first["results"] == [ + { + "name": "LLM", + "forms": ["LLM", "large language model"], + "file": str(note), + "action": "enriched", + "link_stem": "LLM", + "fields_added": ["elaboration", "intuition"], + "occurrence_added": True, + } + ] + assert second["results"][0]["action"] == "unchanged" + assert second["results"][0]["fields_added"] == [] + assert second["results"][0]["occurrence_added"] is False + assert note.read_bytes() == first_bytes + assert text.count("[[Paper]]") == 1 + + +def test_writer_preserves_bom_crlf_custom_content_and_second_run_bytes( + tmp_path: Path, +) -> None: + terms_dir = tmp_path / "terms" + terms_dir.mkdir() + note = terms_dir / "BOM Term.md" + original_text = ( + f"# BOM Term\r\n\r\n## {GLOSSARY_CONCEPT_HEADING}\r\n" + "- \u5b9a\u4e49\uff1auser definition\r\n" + "- \u8be6\u89e3\uff1auser elaboration\r\n" + "- \u76f4\u89c9\uff1auser intuition\r\n" + "- \u7f6e\u4fe1\u5ea6\uff1a\u4e2d\r\n\r\n" + "## Custom\r\nkeep this text\r\n\r\n" + f"## {GLOSSARY_OCCURRENCE_HEADING}\r\n" + "- [[OldPaper]]: old evidence\r\n" + ) + original_bytes = b"\xef\xbb\xbf" + original_text.encode("utf-8") + note.write_bytes(original_bytes) + inventory = _inventory( + inspect_selected_terms( + [{"term": "BOM Term", "surface_forms": ["BOM Term"]}], terms_dir + ) + ) + glossary = { + "entries": [ + { + "name": "BOM Term", + "aliases": [], + "operation": "reuse", + "occurrence": "new evidence", + } + ] + } + + first = write_glossary_entries(glossary, inventory, terms_dir, "Paper") + first_bytes = note.read_bytes() + second = write_glossary_entries(glossary, inventory, terms_dir, "Paper") + + assert first["results"][0]["action"] == "updated" + assert second["results"][0]["action"] == "unchanged" + assert first_bytes != original_bytes + assert first_bytes.startswith(b"\xef\xbb\xbf") + assert b"\n" not in first_bytes[3:].replace(b"\r\n", b"") + assert b"## Custom\r\nkeep this text\r\n" in first_bytes + assert first_bytes.count(b"[[Paper]]") == 1 + assert note.read_bytes() == first_bytes + + +def test_writer_reuse_rejects_concept_replacement_and_can_add_occurrence(tmp_path: Path) -> None: + terms_dir = tmp_path / "terms" + terms_dir.mkdir() + note = terms_dir / "ReAct.md" + note.write_text(render_term_file({**ENTRY, "name": "ReAct"}, "OldPaper"), encoding="utf-8") + inventory = _inventory( + [ + { + "term": "ReAct", + "surface_forms": ["ReAct"], + "state": "existing_complete", + "file": str(note), + "link_stem": "ReAct", + "missing_fields": ["distinction"], + } + ] + ) + + with pytest.raises(SystemExit, match="reuse"): + write_glossary_entries( + {"entries": [{"name": "ReAct", "operation": "reuse", "definition": "replace", "occurrence": "evidence"}]}, + inventory, + terms_dir, + "Paper", + ) + + result = write_glossary_entries( + {"entries": [{"name": "ReAct", "operation": "reuse", "occurrence": "evidence"}]}, + inventory, + terms_dir, + "Paper", + ) + + assert result["results"][0]["action"] == "updated" + assert result["results"][0]["fields_added"] == [] + assert result["results"][0]["occurrence_added"] is True + + +def test_writer_rejects_forged_outside_target_and_link_stem(tmp_path: Path) -> None: + terms_dir = tmp_path / "terms" + terms_dir.mkdir() + inside = terms_dir / "LLM.md" + outside = tmp_path / "outside.md" + text = ( + f"# LLM\n\n## {GLOSSARY_CONCEPT_HEADING}\n" + "- \u5b9a\u4e49\uff1adefinition\n- \u7f6e\u4fe1\u5ea6\uff1a\u4e2d\n" + ) + inside.write_text(text, encoding="utf-8") + outside.write_text(text, encoding="utf-8") + outside_before = outside.read_bytes() + inventory = _inventory( + [ + { + "term": "LLM", + "surface_forms": ["LLM"], + "state": "existing_thin", + "file": str(outside), + "link_stem": "Forged", + "missing_fields": ["elaboration", "intuition", "distinction"], + } + ] + ) + + with pytest.raises(SystemExit, match="Inventory mismatch"): + write_glossary_entries( + { + "entries": [ + { + "name": "LLM", + "operation": "enrich", + "elaboration": "new", + "occurrence": "evidence", + } + ] + }, + inventory, + terms_dir, + "Paper", + ) + + assert outside.read_bytes() == outside_before + assert "new" not in inside.read_text(encoding="utf-8") + + +def test_writer_rejects_stale_inventory_state_before_writing(tmp_path: Path) -> None: + terms_dir = tmp_path / "terms" + terms_dir.mkdir() + note = terms_dir / "ReAct.md" + note.write_text(render_term_file({**ENTRY, "name": "ReAct"}, "OldPaper"), encoding="utf-8") + before = note.read_bytes() + stale = _inventory( + [ + { + "term": "ReAct", + "surface_forms": ["ReAct"], + "state": "existing_thin", + "file": str(note), + "link_stem": "ReAct", + "missing_fields": ["distinction"], + } + ] + ) + + with pytest.raises(SystemExit, match="Inventory mismatch"): + write_glossary_entries( + { + "entries": [ + { + "name": "ReAct", + "operation": "enrich", + "distinction": "replacement", + "occurrence": "evidence", + } + ] + }, + stale, + terms_dir, + "Paper", + ) + + assert note.read_bytes() == before + + +def test_writer_rejects_two_entries_resolving_to_same_target(tmp_path: Path) -> None: + terms_dir = tmp_path / "terms" + terms_dir.mkdir() + note = terms_dir / "Canonical.md" + note.write_text( + render_term_file({**ENTRY, "name": "Canonical", "aliases": ["Alias"]}, "OldPaper"), + encoding="utf-8", + ) + selected = [ + {"term": "Canonical", "surface_forms": ["Canonical"]}, + {"term": "Alias", "surface_forms": ["Alias"]}, + ] + inventory = _inventory(inspect_selected_terms(selected, terms_dir)) + before = note.read_bytes() + + with pytest.raises(SystemExit, match="Duplicate target"): + write_glossary_entries( + { + "entries": [ + {"name": "Canonical", "operation": "reuse", "occurrence": "one"}, + {"name": "Alias", "operation": "reuse", "occurrence": "two"}, + ] + }, + inventory, + terms_dir, + "Paper", + ) + + assert note.read_bytes() == before + + +def test_writer_prepares_all_targets_before_any_create(tmp_path: Path) -> None: + terms_dir = tmp_path / "terms" + terms_dir.mkdir() + (terms_dir / "Broken.md").write_bytes(b"\xff\xfe\x00") + inventory = _inventory( + [ + { + "term": "Created", + "surface_forms": ["Created"], + "state": "new", + "file": "", + "link_stem": "Created", + "missing_fields": [], + }, + { + "term": "Broken", + "surface_forms": ["Broken"], + "state": "existing_thin", + "file": str(terms_dir / "Broken.md"), + "link_stem": "Broken", + "missing_fields": ["definition", "elaboration", "intuition", "distinction", "confidence"], + }, + ] + ) + + with pytest.raises(SystemExit, match="Unable to read glossary note.*Broken.md"): + write_glossary_entries( + { + "entries": [ + { + "name": "Created", + "operation": "create", + "definition": "definition", + "confidence": "\u9ad8", + "occurrence": "evidence", + }, + { + "name": "Broken", + "operation": "enrich", + "definition": "definition", + "confidence": "\u9ad8", + "occurrence": "evidence", + }, + ] + }, + inventory, + terms_dir, + "Paper", + ) + + assert not (terms_dir / "Created.md").exists() + + +def test_writer_allocates_sanitized_create_collisions_and_reports_actual_stems( + tmp_path: Path, +) -> None: + terms_dir = tmp_path / "terms" + selected = [ + {"term": "A/B", "surface_forms": ["A/B"]}, + {"term": "A:B", "surface_forms": ["A:B"]}, + ] + inventory = _inventory(inspect_selected_terms(selected, terms_dir)) + entries = { + "entries": [ + { + "name": item["term"], + "operation": "create", + "definition": "definition", + "confidence": "\u9ad8", + "occurrence": "evidence", + } + for item in selected + ] + } + + result = write_glossary_entries(entries, inventory, terms_dir, "Paper") + + assert [item["link_stem"] for item in result["results"]] == ["A B", "A B-2"] + assert sorted(path.name for path in terms_dir.glob("*.md")) == ["A B-2.md", "A B.md"] + + +def test_append_occurrence_ignores_same_paper_link_outside_occurrence_section() -> None: + text = ( + "# Existing\n\nCustom mention [[Paper]] outside the occurrence section.\n\n" + f"## {GLOSSARY_OCCURRENCE_HEADING}\n- [[OldPaper]]: old evidence\n" + ) + + updated = append_occurrence(text, {"occurrence": "new evidence"}, "Paper") + repeated = append_occurrence(updated, {"occurrence": "new evidence"}, "Paper") + + occurrence = updated.split(f"## {GLOSSARY_OCCURRENCE_HEADING}", 1)[1] + assert occurrence.count("[[Paper]]") == 1 + assert repeated == updated + + +def test_writer_preserves_grounded_forms_when_entry_omits_or_adds_aliases( + tmp_path: Path, +) -> None: + terms_dir = tmp_path / "terms" + selected = [ + {"term": "ReAct", "surface_forms": ["ReAct", "ＲｅＡｃｔ"]} + ] + inventory = _inventory(inspect_selected_terms(selected, terms_dir)) + + result = write_glossary_entries( + { + "entries": [ + { + "name": "ReAct", + "aliases": ["Reasoning and Acting"], + "operation": "create", + "definition": "A reasoning and acting pattern.", + "confidence": "高", + "occurrence": "Paper evidence.", + } + ] + }, + inventory, + terms_dir, + "Paper", + ) + + assert result["results"][0]["forms"] == ["ReAct", "ＲｅＡｃｔ"] + assert ' - "Reasoning and Acting"' in (terms_dir / "ReAct.md").read_text( + encoding="utf-8" + ) + + +def test_writer_rejects_missing_inventory_provenance_without_writes(tmp_path: Path) -> None: + terms_dir = tmp_path / "terms" + inventory = { + "results": inspect_selected_terms( + [{"term": "ReAct", "surface_forms": ["ReAct"]}], terms_dir + ) + } + + with pytest.raises(SystemExit, match="provenance"): + write_glossary_entries( + { + "entries": [ + { + "name": "ReAct", + "operation": "create", + "definition": "A reasoning and acting pattern.", + "confidence": "高", + "occurrence": "Paper evidence.", + } + ] + }, + inventory, + terms_dir, + "Paper", + ) + + assert not terms_dir.exists() + + +@pytest.mark.parametrize("change", ["append", "reorder", "replace"]) +def test_writer_rejects_changed_inventory_surface_forms_without_writes( + tmp_path: Path, change: str +) -> None: + terms_dir = tmp_path / "terms" + source_forms = ["ReAct", "ＲｅＡｃｔ"] + inventory = build_inventory_payload( + [{"term": "ReAct", "surface_forms": source_forms}], terms_dir + ) + if change == "append": + inventory["results"][0]["surface_forms"].append("invented") + elif change == "reorder": + inventory["results"][0]["surface_forms"].reverse() + else: + inventory["results"][0]["surface_forms"][1] = "invented" + + with pytest.raises(SystemExit, match="selection|provenance|surface_forms"): + write_glossary_entries( + { + "entries": [ + { + "name": "ReAct", + "operation": "create", + "definition": "A reasoning and acting pattern.", + "confidence": "高", + "occurrence": "Paper evidence.", + } + ] + }, + inventory, + terms_dir, + "Paper", + ) + + assert not terms_dir.exists() + + +def test_writer_rejects_entry_only_alias_redirect_to_existing_note(tmp_path: Path) -> None: + terms_dir = tmp_path / "terms" + terms_dir.mkdir() + victim = terms_dir / "Victim.md" + victim.write_text( + render_term_file({**ENTRY, "name": "Victim", "aliases": []}, "OldPaper"), + encoding="utf-8", + ) + victim_item = inspect_selected_terms( + [{"term": "Victim", "surface_forms": ["Victim"]}], terms_dir + )[0] + inventory = _inventory( + [ + { + **victim_item, + "term": "Innocent", + "surface_forms": ["Innocent"], + } + ] + ) + before = victim.read_bytes() + + with pytest.raises(SystemExit, match="Inventory mismatch"): + write_glossary_entries( + { + "entries": [ + { + "name": "Innocent", + "aliases": ["Victim"], + "operation": "reuse", + "occurrence": "redirect", + } + ] + }, + inventory, + terms_dir, + "Paper", + ) + + assert victim.read_bytes() == before + + +def test_writer_rejects_create_alias_owned_by_unselected_existing_note_without_writes( + tmp_path: Path, +) -> None: + terms_dir = tmp_path / "terms" + terms_dir.mkdir() + owner = terms_dir / "Existing.md" + owner.write_text( + render_term_file( + {**ENTRY, "name": "Existing", "aliases": ["Generated Alias"]}, + "OldPaper", + ), + encoding="utf-8", + ) + inventory = build_inventory_payload( + [{"term": "ReAct", "surface_forms": ["ReAct"]}], terms_dir + ) + before = owner.read_bytes() + + with pytest.raises(SystemExit, match="alias|logical|collision|existing glossary"): + write_glossary_entries( + { + "entries": [ + { + "name": "ReAct", + "aliases": ["Generated Alias"], + "operation": "create", + "definition": "A reasoning and acting pattern.", + "confidence": "\u9ad8", + "occurrence": "Paper evidence.", + } + ] + }, + inventory, + terms_dir, + "Paper", + ) + + assert owner.read_bytes() == before + assert not (terms_dir / "ReAct.md").exists() + + +def test_writer_allows_generated_alias_owned_by_the_same_existing_target( + tmp_path: Path, +) -> None: + terms_dir = tmp_path / "terms" + terms_dir.mkdir() + note = terms_dir / "ReAct.md" + note.write_text( + render_term_file( + {**ENTRY, "name": "ReAct", "aliases": ["Reasoning and Acting"]}, + "OldPaper", + ), + encoding="utf-8", + ) + inventory = build_inventory_payload( + [{"term": "ReAct", "surface_forms": ["ReAct"]}], terms_dir + ) + + result = write_glossary_entries( + { + "entries": [ + { + "name": "ReAct", + "aliases": ["Reasoning and Acting"], + "operation": "reuse", + "occurrence": "New paper evidence.", + } + ] + }, + inventory, + terms_dir, + "Paper", + ) + + assert result["results"][0]["file"] == str(note.resolve()) + assert "[[Paper]]" in note.read_text(encoding="utf-8") + + +def test_writer_requires_canonical_name_in_inventory_surface_forms(tmp_path: Path) -> None: + terms_dir = tmp_path / "terms" + inventory = _inventory( + [ + { + "term": "Canonical", + "surface_forms": ["Alias"], + "state": "new", + "file": "", + "link_stem": "Canonical", + "missing_fields": [], + } + ] + ) + + with pytest.raises(SystemExit, match="Canonical|canonical.*surface_forms"): + write_glossary_entries( + { + "entries": [ + { + "name": "Canonical", + "aliases": ["Alias"], + "operation": "create", + "definition": "definition", + "confidence": "\u9ad8", + "occurrence": "evidence", + } + ] + }, + inventory, + terms_dir, + "Paper", + ) + + assert not terms_dir.exists() + + +@pytest.mark.parametrize( + "name", + ["", ".", "..", "CON", "nul.txt", "trailing.", "trailing "], +) +def test_safe_term_filename_rejects_unsafe_components(name: str) -> None: + with pytest.raises(SystemExit, match="Unsafe glossary filename"): + safe_term_filename(name) + + +def test_writer_rejects_overlong_create_before_any_write(tmp_path: Path) -> None: + terms_dir = tmp_path / "terms" + long_name = "X" * 300 + selected = [ + {"term": "First", "surface_forms": ["First"]}, + {"term": long_name, "surface_forms": [long_name]}, + ] + inventory = _inventory(inspect_selected_terms(selected, terms_dir)) + + with pytest.raises(SystemExit, match="Unsafe glossary filename.*too long"): + write_glossary_entries( + { + "entries": [ + { + "name": item["term"], + "operation": "create", + "definition": "definition", + "confidence": "\u9ad8", + "occurrence": "evidence", + } + for item in selected + ] + }, + inventory, + terms_dir, + "Paper", + ) + + assert not (terms_dir / "First.md").exists() + + +def test_writer_rejects_non_bmp_overlong_create_before_any_write(tmp_path: Path) -> None: + terms_dir = tmp_path / "terms" + long_name = "\U0001f600" * 130 + selected = [ + {"term": "First", "surface_forms": ["First"]}, + {"term": long_name, "surface_forms": [long_name]}, + ] + inventory = _inventory(inspect_selected_terms(selected, terms_dir)) + + with pytest.raises(SystemExit, match="Unsafe glossary filename.*too long"): + write_glossary_entries( + { + "entries": [ + { + "name": item["term"], + "operation": "create", + "definition": "definition", + "confidence": "\u9ad8", + "occurrence": "evidence", + } + for item in selected + ] + }, + inventory, + terms_dir, + "Paper", + ) + + assert not (terms_dir / "First.md").exists() + + +def test_writer_rechecks_existing_snapshot_before_first_write( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + terms_dir = tmp_path / "terms" + terms_dir.mkdir() + note = terms_dir / "ReAct.md" + note.write_text( + render_term_file({**ENTRY, "name": "ReAct", "aliases": []}, "OldPaper"), + encoding="utf-8", + ) + inventory = _inventory( + inspect_selected_terms( + [{"term": "ReAct", "surface_forms": ["ReAct"]}], terms_dir + ) + ) + original_prepare = writer_module._prepare_write_plans + concurrent = b"# Concurrent user edit\n" + + def prepare_then_modify(*args: object, **kwargs: object) -> list[dict]: + plans = original_prepare(*args, **kwargs) + note.write_bytes(concurrent) + return plans + + monkeypatch.setattr(writer_module, "_prepare_write_plans", prepare_then_modify) + + with pytest.raises(SystemExit, match="changed after planning"): + writer_module.write_glossary_entries( + {"entries": [{"name": "ReAct", "operation": "reuse", "occurrence": "new"}]}, + inventory, + terms_dir, + "Paper", + ) + + assert note.read_bytes() == concurrent + assert not list(terms_dir.glob("*.tmp")) + + +def test_writer_rechecks_create_absence_before_first_write( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + terms_dir = tmp_path / "terms" + selected = [{"term": "Created", "surface_forms": ["Created"]}] + inventory = _inventory(inspect_selected_terms(selected, terms_dir)) + original_prepare = writer_module._prepare_write_plans + concurrent = b"# Concurrent create\n" + + def prepare_then_create(*args: object, **kwargs: object) -> list[dict]: + plans = original_prepare(*args, **kwargs) + target = Path(plans[0]["path"]) + target.parent.mkdir(parents=True, exist_ok=True) + target.write_bytes(concurrent) + return plans + + monkeypatch.setattr(writer_module, "_prepare_write_plans", prepare_then_create) + + with pytest.raises(SystemExit, match="appeared after planning"): + writer_module.write_glossary_entries( + { + "entries": [ + { + "name": "Created", + "operation": "create", + "definition": "definition", + "confidence": "\u9ad8", + "occurrence": "evidence", + } + ] + }, + inventory, + terms_dir, + "Paper", + ) + + assert (terms_dir / "Created.md").read_bytes() == concurrent + + +def test_writer_rolls_back_prior_create_reservations_on_exclusive_race( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + terms_dir = tmp_path / "terms" + selected = [ + {"term": "First", "surface_forms": ["First"]}, + {"term": "Second", "surface_forms": ["Second"]}, + ] + inventory = _inventory(inspect_selected_terms(selected, terms_dir)) + entries = { + "entries": [ + { + "name": item["term"], + "operation": "create", + "definition": "definition", + "confidence": "\u9ad8", + "occurrence": "evidence", + } + for item in selected + ] + } + original_open = writer_module.os.open + concurrent = b"# Concurrent second create\n" + + def race_on_second(path: str | bytes, flags: int, mode: int = 0o777) -> int: + target = Path(path) + if target.name == "Second.md" and not target.exists(): + target.write_bytes(concurrent) + return original_open(path, flags, mode) + + monkeypatch.setattr(writer_module.os, "open", race_on_second) + + with pytest.raises(SystemExit, match="appeared during commit"): + writer_module.write_glossary_entries(entries, inventory, terms_dir, "Paper") + + assert not (terms_dir / "First.md").exists() + assert (terms_dir / "Second.md").read_bytes() == concurrent + + +@pytest.mark.skipif(os.name != "nt", reason="Windows path comparison semantics") +def test_writer_accepts_windows_case_variant_for_same_contained_file(tmp_path: Path) -> None: + terms_dir = tmp_path / "Terms" + terms_dir.mkdir() + note = terms_dir / "MixedCase.md" + note.write_text( + render_term_file({**ENTRY, "name": "MixedCase", "aliases": []}, "OldPaper"), + encoding="utf-8", + ) + inventory = _inventory( + inspect_selected_terms( + [{"term": "MixedCase", "surface_forms": ["MixedCase"]}], terms_dir + ) + ) + inventory["results"][0]["file"] = str(note.with_name("mixedcase.MD")) + + result = write_glossary_entries( + {"entries": [{"name": "MixedCase", "operation": "reuse", "occurrence": "case"}]}, + inventory, + terms_dir.with_name("TERMS"), + "Paper", + ) + + assert result["results"][0]["file"].casefold() == str(note).casefold() + assert "[[Paper]]" in note.read_text(encoding="utf-8") + + def test_lint_rejects_missing_required_term_note_fields() -> None: good = render_term_file(ENTRY, "CAD-MoE")