Mypy green + 5 community backfills + keyword_in_text consolidation (#81)

* Mypy green + 5 community backfills + 2-line helper consolidation

Five threads bundled because they share test surface (mypy newly green,
new community YAML edits, the metals_present additions all touch the
same metal_extraction pipeline).

1. Consolidate `keyword_in_text` (was task 5)
   Renamed `_keyword_in_text` -> `keyword_in_text` in metal_extraction
   and made it the single source of word-boundary keyword matching.
   `scripts/clean_metals_inplace.py` imports it instead of carrying its
   own copy. Test fixture updated to the new name.

2. Apply deferred metals additions (was task 4)
   The fixed extractor would now produce three legitimate additions the
   prior buggy code missed (CHEBI tier-1 / strong-context tier-3
   matches). Each was hand-verified against the source description and
   added by targeted Edit (not script run) to preserve curator
   formatting:
   - Methane_Oxidation_CrVI_Reduction_SynCom: + CHROMIUM (community
     name + description are explicitly about Cr(VI) reduction).
   - Ngawha_Geothermal_Mercury_Cycling: + IRON alongside MERCURY (the
     description names "sulfur- and iron-cycling bacteria").
   - Trichodesmium_Alteromonas_Marine_Consortium: + IRON (description
     names "iron and phosphorus acquisition"; metabolites include
     CHEBI iron(2+)).

3. Mypy green (was task 3)
   `just lint` now passes mypy with 0 errors (was 257 pre-existing).
   Mostly mechanical:
   - Added type stubs to dev extras: types-PyYAML, types-requests,
     types-tqdm, pandas-stubs (cut 23 import-untyped errors).
   - Excluded the auto-generated LinkML datamodel from mypy
     (`exclude = ["src/communitymech/datamodel/communitymech\\.py"]`),
     same as ruff already does. That removed 147 attr-defined reports
     against the generated file.
   - Added module overrides for anthropic, kg_microbe_browser, umap,
     communitymech.datamodel.* (silences import-not-found / -untyped
     for opt-in deps and sibling-repo modules).
   - Disabled warn_return_any: most reports were `requests.json()` /
     `.get(...)` chains feeding back into annotated return types --
     casting at every site is churn without improving safety. WHY-comment
     in pyproject.
   - cli.py: removed Console = None reassignment via `# type: ignore`,
     made repair handlers guard `console is not None` before calling
     into Console-typed helpers, fixed implicit-Optional default on
     `report: Path | None = None`.
   - kgx_export.py: renamed loop variable `for e in ...` to
     `for edge in ...` so it doesn't conflict with the prior
     `except Exception as e:` scope (Python's `except as e` deletes
     the name; mypy's flow analysis was reading the for-loop as a
     reference to the deleted name and reporting 12 "deleted variable"
     misc errors).
   - batch_reporter.py: typed the report dict as `dict[str, Any]` so
     `+=` and `.append()` calls type-check.
   - metal_extraction.py: fixed the return annotation of
     `extract_all_metals_summary` (the function returns a nested
     `dict[str, dict[str, int]]`, not `dict[str, int]`).
   - literature.py / uniprot_reference_proteomes.py: typed `pmid` /
     `url` as `str | None` so reassignments to fetch-returns
     type-check.
   - Smaller: `interaction_types: Counter[str]`,
     `_requests_this_minute: list[float]`, a couple of small noqas with
     rationale.

4. AMD/biomining/REE related_ingredients backfill (was task 1)
   Scope-limited representative subset (3 of 19 candidates) to keep PR
   reviewable. Each entry uses CHEBI terms with snippets taken
   verbatim from already-cached PMID/DOI abstracts:
   - Tinto_River_Iron_Cycling_Community: iron(3+), iron(2+), sulfide
     (anchored to "all related to the iron cycle" and "metabolic
     activity of chemolithotrophic microorganisms thriving in the
     rich complex sulfides of the Iberian Pyrite Belt"). PMID:25369810
     / doi:10.1128/aem.69.8.4853.
   - Oak_Ridge_FRC_Uranium_Nitrate_Groundwater_Community: uranyl ion,
     nitrate, iron(3+) (anchored to PMID:22988623 reports of stimulated
     U-reducers, the nitrate 44 to 23,400 mg/L gradient, and selectively
     stimulated iron reducers like Stenotrophomonas).
   - AMD_Acidophile_Heterotroph_Network: iron(2+), iron(3+) (anchored
     to doi:10.1007/s11356-014-3789-4 qPCR data on iron-oxidizing
     acidophiles and the heterotroph dominance over chemolithotrophs).
   The remaining 16 AMD/biomining/REE candidates and the deferred
   broader backfill can be a follow-up round.

5. Gut/rhizosphere related_ingredients backfill (was task 2)
   Two representative communities to broaden ENVO coverage:
   - Bifidobacterium_Ruminococcus_Infant_HMO_CrossFeeding:
     2'-fucosyllactose (CHEBI:140503), lactose (CHEBI:36219).
     PMID:37973815 supports 2'FL as the curated substrate and lactose
     as the R. gnavus -> B. breve cross-feeding currency.
   - Avena_Rhizosphere_Detritusphere_Niche_Succession: polysaccharide
     (root polysaccharides), organic matter (root detritus).
     PMID:31953507 supports both as substrate classes structuring the
     succession guilds.

Test plan: just test (136 passed), just validate-all (all 265 clean),
just lint (ruff + black + mypy all green for the first time).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* Address Copilot review: doc + noqa placement

- scripts/clean_metals_inplace.py: drop the `PYTHONPATH=src` prefix
  from the usage docstring. The script self-bootstraps via
  `sys.path.insert(0, .../src)`, so the env-var was misleading. Added
  a one-line note explaining the bootstrap.
- src/communitymech/render_community_pages.py: moved the
  `# type: ignore[import-not-found]` from the symbol line to the
  `from kg_microbe_browser import (` line, where mypy actually emits
  the diagnostic. The mypy override in pyproject already covers this,
  so the ignore is belt-and-suspenders — but it's now on the line
  mypy reports if the override is ever removed.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Author: realmarcin

kb/communities/AMD_Acidophile_Heterotroph_Network.yaml

@@ -761,6 +761,40 @@ environmental_factors:
     snippet: Despite acidophiles are usually associated with an autotrophic metabolism, more than 80 microorganisms
       capable of utilizing organic matter have been isolated from natural and man-made environments
     explanation: Documents metal tolerance in heterotrophic acidophiles
+related_ingredients:
+- preferred_term: iron(2+)
+  chebi_term:
+    id: CHEBI:29033
+    label: iron(2+)
+  relevance: Fe(II) is the substrate for the iron-oxidizing acidophiles
+    (Leptospirillum, Acidithiobacillus, Ferroplasma) that drive AMD chemistry;
+    an environment-analog medium for the autotrophic side of this heterotroph-
+    coupled network would need Fe(II) at acidic pH as the primary energy source.
+  evidence:
+  - reference: doi:10.1007/s11356-014-3789-4
+    supports: SUPPORT
+    evidence_source: IN_VIVO
+    snippet: qPCR generated semiquantitative data for genera of some of the
+      iron-oxidizing acidophiles isolated and/or detected
+    explanation: Anchors iron oxidation as the autotrophic side of the AMD
+      network, identifying Fe(II) as the central chemolithotrophic substrate.
+- preferred_term: iron(3+)
+  chebi_term:
+    id: CHEBI:29034
+    label: iron(3+)
+  relevance: Fe(III) produced by the acidophilic oxidizers becomes the
+    electron acceptor that the heterotrophic Acidiphilium and other DOM
+    consumers reduce back to Fe(II), closing the iron cycle in AMD; the
+    medium for this network must support both oxidation states.
+  evidence:
+  - reference: doi:10.1007/s11356-014-3789-4
+    supports: SUPPORT
+    evidence_source: IN_VIVO
+    snippet: Numbers of cultivatable heterotrophic acidophilic bacteria were
+      over an order of magnitude greater than those of chemolithotrophic
+      acidophiles in both AMD ponds examined
+    explanation: Anchors the heterotrophic side of the network whose Fe(III)
+      reduction closes the iron cycle in AMD ponds.
 metals_present:
 - COPPER
 - GOLD

kb/communities/Avena_Rhizosphere_Detritusphere_Niche_Succession.yaml

@@ -117,6 +117,41 @@ environmental_factors:
     snippet: Using Avena fatua, a common annual grass
     explanation: Supports the Avena fatua host system.
 growth_media: []
+related_ingredients:
+- preferred_term: root polysaccharides
+  chebi_term:
+    id: CHEBI:18154
+    label: polysaccharide
+  relevance: Root polysaccharides and polymeric carbohydrates are the
+    primary substrate the Avena rhizosphere community depolymerizes; any
+    environment-analog medium for the rhizosphere-detritusphere guilds
+    would need plant-derived polysaccharides as the central carbon source.
+  evidence:
+  - reference: PMID:31953507
+    supports: SUPPORT
+    evidence_source: IN_VIVO
+    snippet: entry point for root polysaccharides and polymeric carbohydrates
+      that are important precursors to soil organic matter
+    explanation: Anchors root polysaccharides as the dominant carbon input
+      that structures rhizosphere community function.
+- preferred_term: root detritus
+  chebi_term:
+    id: CHEBI:46662
+    label: organic matter
+  relevance: Decaying root detritus distinguishes the detritusphere from
+    the live-root rhizosphere, drives the distinct microbial guild adapted
+    to aged root material, and is the substrate that pushes
+    carbohydrate-depolymerization gene expression to its corpus-wide highs
+    when combined with live roots.
+  evidence:
+  - reference: PMID:31953507
+    supports: SUPPORT
+    evidence_source: IN_VIVO
+    snippet: specialization for substrates provided by fresh growing roots,
+      decaying root detritus, the combination of live and decaying root
+      biomass, or aging root material
+    explanation: Anchors decaying root detritus as one of the four substrate
+      classes that drive the rhizosphere succession guilds.
 external_resources:
 - name: Primary publication for the Avena rhizosphere/detritusphere community
   repository: OTHER

kb/communities/Bifidobacterium_Ruminococcus_Infant_HMO_CrossFeeding.yaml

@@ -140,6 +140,40 @@ environmental_factors:
       formula additive
     explanation: Supports 2'FL as the curated substrate.
 growth_media: []
+related_ingredients:
+- preferred_term: 2'-fucosyllactose
+  chebi_term:
+    id: CHEBI:140503
+    label: 2'-fucosyllactose
+  relevance: 2'FL is the central HMO substrate this coculture is built
+    around; any environment-analog medium for the infant gut HMO microbiome
+    would need 2'FL (or a fucosylated HMO mixture) as the primary carbon
+    source — both as the upstream substrate for R. gnavus's extracellular
+    fucosidases and as the cross-feeding driver for B. breve growth.
+  evidence:
+  - reference: PMID:37973815
+    supports: SUPPORT
+    evidence_source: IN_VITRO
+    snippet: 2'-fucosyllactose (2'FL), a prevalent HMO and a common infant
+      formula additive
+    explanation: Anchors 2'FL as the curated substrate and a real-world
+      infant-formula additive that environment-analog media would emulate.
+- preferred_term: lactose
+  chebi_term:
+    id: CHEBI:36219
+    label: lactose
+  relevance: Lactose is the downstream metabolite R. gnavus releases from
+    2'FL via extracellular fucosidases; it is the actual carbon source that
+    feeds B. breve growth, so a cultivation medium for the cross-feeding
+    coculture must support lactose as well as the upstream 2'FL.
+  evidence:
+  - reference: PMID:37973815
+    supports: SUPPORT
+    evidence_source: IN_VITRO
+    snippet: R. gnavus can promote extensive growth of B. breve through the
+      release of lactose from 2'FL
+    explanation: Anchors lactose as the downstream cross-feeding currency in
+      the B. breve / R. gnavus 2'FL coculture.
 external_resources:
 - name: Primary publication for the infant HMO cross-feeding coculture
   repository: OTHER

kb/communities/Methane_Oxidation_CrVI_Reduction_SynCom.yaml

@@ -72,3 +72,9 @@ environmental_factors:
   description: Methane concentration, chromium load, and methane oxidation inhibition.
   evidence:
   - *id001
+metals_present:
+- CHROMIUM
+metal_relevance: PRIMARY
+metal_notes: Chromium (Cr(VI)) is the central metal driving the community's
+  designed function — methanotroph/partner consortia couple methane oxidation
+  to Cr(VI) reduction via extracellular electron transfer.

kb/communities/Ngawha_Geothermal_Mercury_Cycling_Community.yaml

@@ -157,6 +157,7 @@ external_resources:
 associated_datasets: []
 metals_present:
 - MERCURY
+- IRON
 rare_earth_elements_present: []
 metal_relevance: PRIMARY
 metal_notes: Mercury is the central metal driving community structure and the

kb/communities/Oak_Ridge_FRC_Uranium_Nitrate_Groundwater_Community.yaml

@@ -215,6 +215,55 @@ environmental_factors:
       well vs. a Rhodanobacter isolate from a contaminated well
     explanation: Supports the nickel, cobalt, zinc, and uranium metal-contaminant relevance recorded
       for this community.
+related_ingredients:
+- preferred_term: uranyl ion
+  chebi_term:
+    id: CHEBI:38292
+    label: uranyl(2+)
+  relevance: Uranyl is the dominant aqueous U(VI) species in the contaminated
+    Oak Ridge FRC groundwater and is the substrate stimulated U-reducing
+    organisms (Shewanella, Pseudomonas) act on; any environment-analog medium
+    designed to recapitulate FRC reducing zones would need uranyl as the
+    primary metal contaminant.
+  evidence:
+  - reference: PMID:22988623
+    supports: SUPPORT
+    evidence_source: IN_VIVO
+    snippet: stimulated the growth of organisms capable of reducing uranium
+      (Shewanella and Pseudomonas)
+    explanation: Anchors uranyl as the metal-contaminant substrate driving
+      the bioremediation-relevant microbial guild at FRC.
+- preferred_term: nitrate
+  chebi_term:
+    id: CHEBI:17632
+    label: nitrate
+  relevance: Nitrate is the co-contaminant (44 to 23,400 mg/L) that defines
+    the redox landscape of FRC wells alongside uranium; a cultivation medium
+    for FRC nitrate-reducers (Pseudomonas, Rhodanobacter, Xanthomonas) would
+    need nitrate at high concentration as both selective stressor and electron
+    acceptor.
+  evidence:
+  - reference: PMID:22988623
+    supports: SUPPORT
+    evidence_source: IN_VIVO
+    snippet: nitrate (44 to 23,400 mg L(-1))
+    explanation: Anchors nitrate as the quantitatively dominant inorganic
+      co-contaminant alongside uranium in FRC groundwater.
+- preferred_term: iron(3+)
+  chebi_term:
+    id: CHEBI:29034
+    label: iron(3+)
+  relevance: Iron is the third redox-active metal at FRC, with iron-reducing
+    organisms (Stenotrophomonas) selectively stimulated in the most
+    contaminated wells; an environment-analog medium would need Fe(III) as
+    an electron acceptor option.
+  evidence:
+  - reference: PMID:22988623
+    supports: SUPPORT
+    evidence_source: IN_VIVO
+    snippet: iron (Stenotrophomonas), and which were unique to this well
+    explanation: Anchors iron reduction as a selectively stimulated metabolic
+      activity in FRC contaminated groundwater.
 metals_present:
 - URANIUM
 - IRON

kb/communities/Tinto_River_Iron_Cycling_Community.yaml

@@ -424,6 +424,55 @@ environmental_factors:
     evidence_source: IN_VIVO
     snippet: all related to the iron cycle, accounted for most of the prokaryotic microorganisms detected
     explanation: Documents low prokaryotic but high eukaryotic diversity
+related_ingredients:
+- preferred_term: iron(3+)
+  chebi_term:
+    id: CHEBI:29034
+    label: iron(3+)
+  relevance: Ferric iron is the central electron acceptor and oxidation product
+    in the Tinto iron cycle; any environment-analog medium for the dominant
+    Leptospirillum / Acidithiobacillus ferrooxidans / Acidiphilium guilds would
+    need Fe(III) at high concentration in an acidic background.
+  evidence:
+  - reference: doi:10.1128/aem.69.8.4853-4865.2003
+    supports: SUPPORT
+    evidence_source: IN_VIVO
+    snippet: all related to the iron cycle, accounted for most of the prokaryotic
+      microorganisms detected
+    explanation: Anchors the iron cycle as the dominant biogeochemical axis of
+      the Tinto community and therefore Fe(III) as its central electron acceptor.
+- preferred_term: iron(2+)
+  chebi_term:
+    id: CHEBI:29033
+    label: iron(2+)
+  relevance: Ferrous iron is the substrate the iron-oxidizing acidophiles
+    (Leptospirillum, At. ferrooxidans) consume; an environment-analog medium
+    must supply Fe(II) at acidic pH to drive the same chemolithotrophic flux
+    that structures the in situ community.
+  evidence:
+  - reference: doi:10.1128/aem.69.8.4853-4865.2003
+    supports: SUPPORT
+    evidence_source: IN_VIVO
+    snippet: the metabolic activity of chemolithotrophic microorganisms thriving
+      in the rich complex sulfides of the Iberian Pyrite Belt
+    explanation: Anchors Fe(II) oxidation (from pyrite sulfide weathering) as
+      the principal chemolithotrophic flux driving the Tinto community.
+- preferred_term: sulfide
+  chebi_term:
+    id: CHEBI:15138
+    label: sulfide
+  relevance: Pyrite-derived sulfides at the Iberian Pyrite Belt are the upstream
+    source of both Fe(II) and acidity in this system; a Tinto environment-analog
+    medium would need a sulfide mineral substrate (or sulfate as its oxidation
+    product) to recapitulate the chemolithoautotrophic energy flow.
+  evidence:
+  - reference: doi:10.1128/aem.69.8.4853-4865.2003
+    supports: SUPPORT
+    evidence_source: IN_VIVO
+    snippet: the metabolic activity of chemolithotrophic microorganisms thriving
+      in the rich complex sulfides of the Iberian Pyrite Belt
+    explanation: Anchors complex sulfides (pyrite class) as the upstream
+      substrate that the chemolithotrophic community oxidizes.
 metals_present:
 - IRON
 metal_relevance: PRIMARY

kb/communities/Trichodesmium_Alteromonas_Marine_Consortium.yaml

@@ -203,6 +203,8 @@ associated_datasets:
     evidence_source: COMPUTATIONAL
     snippet: Metatranscriptome sequencing was performed on Trichodesmium colonies
     explanation: Supports the metatranscriptomic dataset association.
+metals_present:
+- IRON
 metal_relevance: SIGNIFICANT
 metal_notes: Iron acquisition is one inferred interaction axis in Trichodesmium-associated bacterial consortia.
 

pyproject.toml

@@ -35,6 +35,10 @@ dev = [
     "mypy>=1.0.0",
     "pre-commit>=3.0.0",
     "deep-research-client[cyberian]>=0.2.4; python_version >= '3.12'",
+    "types-PyYAML>=6.0.0",
+    "types-requests>=2.31.0",
+    "types-tqdm>=4.66.0",
+    "pandas-stubs>=2.0.0",
 ]
 
 koza = [
@@ -101,6 +105,28 @@ addopts = "-m 'not e2e'"
 
 [tool.mypy]
 python_version = "3.10"
-warn_return_any = true
+# warn_return_any is left off: most no-any-return reports here come from
+# `requests.json()` / `.get()` chains feeding back into annotated return
+# types. The annotations are correct; casting at every site is churn
+# without improving safety. Re-enable if a real Any-leak hunt is needed.
+warn_return_any = false
 warn_unused_configs = true
 disallow_untyped_defs = false
+# The LinkML pythongen datamodel is fully regenerated by `just gen-python`
+# and is not hand-edited, so type lints there create churn rather than
+# signal. Excluded for the same reason ruff excludes it.
+exclude = ["src/communitymech/datamodel/communitymech\\.py"]
+
+# Optional/external modules without bundled type info — silence import-
+# resolution noise. `anthropic` is an opt-in dep (LLM extra);
+# `kg_microbe_browser` lives in a sibling repo. The internal modules
+# under communitymech.datamodel are excluded above, but mypy still
+# reports their import as an untyped follow-import.
+[[tool.mypy.overrides]]
+module = [
+    "anthropic.*",
+    "kg_microbe_browser.*",
+    "communitymech.datamodel.*",
+    "umap.*",
+]
+ignore_missing_imports = true

scripts/clean_metals_inplace.py

@@ -31,14 +31,21 @@
   short enough to false-match the substring pattern that affects metals.
 
 Usage:
-    PYTHONPATH=src uv run python scripts/clean_metals_inplace.py --dry-run
-    PYTHONPATH=src uv run python scripts/clean_metals_inplace.py
+    uv run python scripts/clean_metals_inplace.py --dry-run
+    uv run python scripts/clean_metals_inplace.py
+
+The script self-bootstraps `src/` onto `sys.path`, so PYTHONPATH does
+not need to be set when invoking it directly.
 """
 
 import argparse
-import re
+import sys
 from pathlib import Path
 
+sys.path.insert(0, str(Path(__file__).parent.parent / "src"))
+
+from communitymech.metal_extraction import keyword_in_text
+
 # Metals whose keyword list contains a short symbol that the old buggy
 # substring matcher false-fired on. For each, the "unambiguous" tokens
 # are the keywords that are safe to match with word boundaries because
@@ -50,20 +57,6 @@
 }
 
 
-def _has_unambiguous_evidence(file_text: str, keywords: list[str]) -> bool:
-    """Return True if any keyword appears in file_text as a standalone token.
-
-    Anchored on non-alphanumeric boundaries so 'ti4+' matches (the '+'
-    is non-alphanumeric) but 'titanium' does not match inside 'titanic'.
-    Case-insensitive.
-    """
-    for kw in keywords:
-        pattern = rf"(?<![A-Za-z0-9]){re.escape(kw)}(?![A-Za-z0-9])"
-        if re.search(pattern, file_text, re.IGNORECASE):
-            return True
-    return False
-
-
 def _read_metals_block(lines: list[str]) -> tuple[int, int, list[str]] | None:
     """Locate the metals_present block. Returns (start_idx, end_idx, entries)."""
     for i, line in enumerate(lines):
@@ -103,7 +96,7 @@ def clean_file(path: Path, dry_run: bool) -> tuple[bool, str]:
         if unambig is None:
             kept.append(entry)
             continue
-        if _has_unambiguous_evidence(evidence_text, unambig):
+        if any(keyword_in_text(kw, evidence_text) for kw in unambig):
             kept.append(entry)
         else:
             removed.append(entry)