Wetland #30 backfill, metals extractor bug fix, lint cleanup, cross-repo ID validator (#80)

* Backfill #30 wetlands, fix metals extractor bug, lint cleanup, cross-repo validator

Combines four follow-ups against #30 (cross-repo environmental linking)
plus an unrelated lint cleanup, all of which build on each other and
share the same test surface.

1. Wetland backfill (#30 Phase 5)
   Apply the SPRUCE related_ingredients pattern to 6 more peatland and
   wetland communities (Stordalen Mire, Prairie Pothole, MUCC Freshwater
   Wetland, Asgard Wetland Soil, Coastal Forested Wetland, Wetland
   Oxygen-Sulfate GHG). Each entry uses CHEBI terms and evidence
   anchored to already-cached PubMed abstracts; no MediaIngredientMech
   IDs are minted yet.

2. Metals extractor bug fix + 65-file cleanup
   metal_extraction.py used plain substring matching against 2-letter
   element symbols ('ti' for TITANIUM, 'au' for GOLD), which matched
   inside unrelated words ('characteristic', 'australia') and salted
   metals_present with TITANIUM in 56/67 metal-annotated YAMLs and
   GOLD in several more. Switched to non-alphanumeric-boundary regex
   matching (case-insensitive), with tests pinning the behavior.
   scripts/clean_metals_inplace.py re-runs extraction and rewrites only
   the metals_present / rare_earth_elements_present / metal_relevance /
   metal_notes blocks via line-based replacement, preserving comments
   and unrelated formatting (unlike backfill_metals.py's yaml.dump
   path). Applied once across the corpus: 65 community YAMLs corrected.

3. Lint cleanup (just lint ruff/black)
   178 pre-existing ruff errors -> 0. Removed T20 (print) from the
   ruff selection with rationale: src/communitymech/ ships CLI entry
   points that legitimately use print. The remaining 44 non-print
   errors were fixed inline (unused imports, raise-from chains,
   collapsible ifs, redundant list() calls, zip strict, line splits,
   import order in batch_reporter.py) or suppressed with a per-file
   E501 ignore for llm/prompts.py (long prompt strings) and targeted
   `# noqa` lines with comments for S301/S701/S704/S112 cases that
   are intentional within their internal-only contexts. mypy still
   reports 256 pre-existing errors and is out of scope here.

4. Cross-repo ID validator (#30 Phase 3, local half)
   New module communitymech.validators.cross_repo_ids with a
   pattern + existence checker, plus a CLI
   (scripts/validate_cross_repo_ids.py) and justfile entries
   (validate-cross-repo-ids, validate-cross-repo-ids-all). Sibling
   repo paths are opt-in via env or flags; when omitted, the
   validator emits info-level skip notices rather than silently
   passing. 10 new tests cover pattern, existence, and edge cases.

Test plan: just test (136 passed, 9 skipped), just validate-all (all
265 communities clean), ruff/black green.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* Address Copilot review: surgical removal-only metals cleanup

Copilot flagged two serious bugs in scripts/clean_metals_inplace.py
from the prior commit:

1. _replace_scalar rewrote metal_notes by substituting only the first
   physical line of the YAML key. When the existing value spanned
   multiple lines (as PyYAML's folded scalars often do), the indented
   continuation lines were left orphaned and silently re-folded by
   the parser into the new value — producing strings like
   "...(context-validated) measurements; ...(context-validated)" and,
   on Ngawha, merging curator prose about mercury cycling into the
   auto-generated note.
2. The script unconditionally overwrote metal_notes and metal_relevance
   and removed any metals_present entries the (newly fixed) extractor
   wouldn't infer. That clobbered curator-authored values (Ngawha's
   MERCURY + curator note, Oak Ridge's NICKEL/COBALT/ZINC, Bayan Obo
   notes, etc.) — entries the extractor cannot derive but that are
   curator decisions to keep.

Reverted all 65 YAMLs the prior commit touched, then rewrote the script
to be surgical:

- Touches only metals_present. Never reads or writes metal_relevance
  or metal_notes, which sidesteps the multi-line scalar bug entirely
  and preserves curator metadata.
- Removes only entries whose extractor keyword list contains a known
  ambiguous short symbol (`ti`/`au`/`pd`) AND whose unambiguous tokens
  (full element name, charged ionic forms) do not appear anywhere
  else in the file as word-bounded tokens. Anything else is kept,
  including curator-added entries the extractor couldn't have inferred.
- Never adds metals. Surprising additions (e.g., Trichodesmium IRON
  via newly-correct CHEBI tier-1 matching) are out of scope; running
  `scripts/backfill_metals.py --dry-run` surfaces them for separate
  curator review.

Result: 56 files (down from 65), each diff is a 1-2 line removal of
TITANIUM and/or GOLD. Ngawha MERCURY, Oak Ridge metals, all curator
metal_notes preserved verbatim. 136 tests pass, all 265 communities
validate.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Author: realmarcin

docs/cross_repo_linking.md

@@ -299,6 +299,8 @@ All new fields are optional:
 
 ## Validation
 
+### Schema-level tests
+
 Run the cross-repo linking tests:
 
 ```bash
@@ -311,6 +313,31 @@ Test data files are in `tests/data/test_cross_repo_linking/`:
 - `community_no_links.yaml` -- Backward compatibility
 - `community_all_relationship_types.yaml` -- All 5 enum values
 
+### Cross-repo ID validator
+
+`just validate-cross-repo-ids FILE` checks that `culturemech_id` /
+`mediaingredientmech_id` values match their CURIE patterns and, when
+sibling-repo paths are configured, that the referenced IDs actually
+exist in those repos.
+
+```bash
+# Pattern check only (no sibling-repo paths)
+just validate-cross-repo-ids kb/communities/SPRUCE_Peatland_Methane_Cycling_Community.yaml
+
+# Pattern + existence check
+COMMUNITYMECH_SIBLING_REPOS="CultureMech=../CultureMech/kb/media,MediaIngredientMech=../MediaIngredientMech/kb/ingredients" \
+    just validate-cross-repo-ids-all
+```
+
+The validator returns:
+- `error` for malformed CURIEs or IDs missing from a configured sibling repo
+- `info` for IDs whose existence check was skipped because the relevant
+  sibling-repo path wasn't configured
+- nothing if a community has no cross-repo IDs at all
+
+Sibling-repo paths can also be passed via `--culturemech` /
+`--mediaingredientmech` flags to `scripts/validate_cross_repo_ids.py`.
+
 ## See Also
 
 - [Growth Media Linking](media_linking.md) -- Existing cultivation-based linking

justfile

@@ -35,6 +35,16 @@ validate-references-all:
         uv run linkml-reference-validator validate data "$file" -s src/communitymech/schema/communitymech.yaml --config conf/reference_validator.yaml
     done
 
+# Validate cross-repo IDs (CultureMech, MediaIngredientMech) in one community file.
+# Pattern checks always run; existence checks run when sibling-repo paths are
+# configured via COMMUNITYMECH_SIBLING_REPOS env (Name=path,Name=path).
+validate-cross-repo-ids FILE:
+    PYTHONPATH=src uv run python scripts/validate_cross_repo_ids.py {{FILE}}
+
+# Validate cross-repo IDs across all community files.
+validate-cross-repo-ids-all:
+    PYTHONPATH=src uv run python scripts/validate_cross_repo_ids.py kb/communities/*.yaml
+
 # Validate ontology terms in a community file
 validate-terms FILE:
     uv run linkml-term-validator validate-data {{FILE}} -s src/communitymech/schema/communitymech.yaml --labels

kb/communities/AMD_Acidophile_Heterotroph_Network.yaml

@@ -765,7 +765,6 @@ metals_present:
 - COPPER
 - GOLD
 - IRON
-- TITANIUM
 metal_relevance: PRIMARY
 metal_notes: Metal/REE detected via CHEBI terms in metabolites; Metal/REE detected via environmental factor
   measurements; Metal/REE detected via keyword matching in description (context-validated)

kb/communities/AMD_Nitrososphaerota_Archaeal.yaml

@@ -699,9 +699,7 @@ environmental_factors:
     explanation: Demonstrates value of genomic data for understanding archaeal adaptations
 metals_present:
 - COPPER
-- GOLD
 - IRON
-- TITANIUM
 metal_relevance: PRIMARY
 metal_notes: Metal/REE detected via CHEBI terms in metabolites; Metal/REE detected via environmental factor
   measurements; Metal/REE detected via keyword matching in description (context-validated)

kb/communities/Asgard_Wetland_Soil_Methanogenesis_Substrate_Community.yaml

@@ -179,6 +179,50 @@ environmental_factors:
     snippet: contributions in soil ecosystems remain unknown
     explanation: Supports terrestrial soil as the focus environment.
 growth_media: []
+related_ingredients:
+- preferred_term: acetate
+  chebi_term:
+    id: CHEBI:30089
+    label: acetate
+  relevance: Asgard archaeal acetogens in this wetland soil generate acetate
+    from carbohydrate breakdown via the Wood-Ljungdahl pathway; acetate is
+    therefore both the headline output of Asgard metabolism in this community
+    and the substrate that feeds the co-resident acetoclastic methanogens.
+  evidence:
+  - reference: PMID:39085194
+    supports: SUPPORT
+    evidence_source: COMPUTATIONAL
+    snippet: carbohydrate breakdown to acetate and formate
+    explanation: Anchors acetate as the central Asgard metabolic output that
+      modulates downstream methanogenesis substrates in wetland soil.
+- preferred_term: formate
+  chebi_term:
+    id: CHEBI:15740
+    label: formate
+  relevance: Formate co-produced with acetate is a major C1 substrate for
+    hydrogenotrophic and formate-utilizing methanogens, defining a second
+    Asgard-mediated methanogenesis-substrate channel in this community.
+  evidence:
+  - reference: PMID:39085194
+    supports: SUPPORT
+    evidence_source: COMPUTATIONAL
+    snippet: carbohydrate breakdown to acetate and formate
+    explanation: Anchors formate as the second Asgard-derived methanogenesis
+      substrate alongside acetate.
+- preferred_term: dihydrogen
+  chebi_term:
+    id: CHEBI:18276
+    label: dihydrogen
+  relevance: Expression of [NiFe]-hydrogenases by both Atabeyarchaeia and
+    Freyarchaeia genomes implicates H2 cycling as a core Asgard activity in
+    this wetland soil; any cultivation medium designed around the community
+    would need an H2 headspace.
+  evidence:
+  - reference: PMID:39085194
+    supports: SUPPORT
+    evidence_source: COMPUTATIONAL
+    snippet: expression of genes for [NiFe]-hydrogenases
+    explanation: Anchors H2 cycling as an in situ expressed Asgard activity.
 external_resources:
 - name: Primary publication for the Asgard wetland soil methanogenesis-substrate
     community

kb/communities/At_RSPHERE_SynCom.yaml

@@ -427,8 +427,7 @@ environmental_factors:
     plant-microbe interactions, and synthetic community design
 
     '
-metals_present:
-- TITANIUM
+metals_present: []
 metal_relevance: INCIDENTAL
 metal_notes: Metal/REE detected via environmental factor measurements; Metal/REE detected
   via keyword matching in description (context-validated)

kb/communities/Australian_Lead_Zinc_Polymetallic.yaml

@@ -966,10 +966,8 @@ environmental_factors:
     explanation: Documents long-term weathering profile development
 metals_present:
 - COPPER
-- GOLD
 - IRON
 - LEAD
-- TITANIUM
 - ZINC
 metal_relevance: SIGNIFICANT
 metal_notes: Metal/REE detected via CHEBI terms in metabolites; Metal/REE detected via environmental factor

kb/communities/Bayan_Obo_REE_Tailings_Consortium.yaml

@@ -544,9 +544,7 @@ environmental_factors:
       was narrated for plausible real-world use
     explanation: Describes REE mineralogy at Bayan Obo
 metals_present:
-- GOLD
 - IRON
-- TITANIUM
 rare_earth_elements_present:
 - CERIUM
 - LANTHANUM

kb/communities/Chlamydomonas_Bacterial_H2_Consortium.yaml

@@ -351,7 +351,6 @@ growth_media:
     explanation: Establishes mannitol and yeast extract as key medium components for sustained H2 production
   culturemech_id: CultureMech:000139
   culturemech_url: https://github.com/CultureBotAI/CultureMech/tree/main/kb/media/CultureMech:000139
-metals_present:
-- TITANIUM
+metals_present: []
 metal_relevance: SIGNIFICANT
 metal_notes: Metal/REE detected via environmental factor measurements

kb/communities/Chlamydomonas_Methylobacterium_Mutualism.yaml

@@ -327,6 +327,5 @@ growth_media:
   culturemech_url: https://github.com/CultureBotAI/CultureMech/tree/main/kb/media/CultureMech:000139
 metals_present:
 - IRON
-- TITANIUM
 metal_relevance: SIGNIFICANT
 metal_notes: Metal/REE detected via environmental factor measurements