experiments/: pywsd benchmarks against pywsd-datasets test sets

New subdir that lives in the repo but is NOT part of the installable
pywsd package. Reproducible benchmarks of every WSD method against
the unified evaluation suite at alvations/pywsd-datasets on HF Hub.

Contents:
- evaluate.py — runs each pywsd method on every test config, pulls
  rows directly from huggingface.co/datasets/alvations/pywsd-datasets.
  Counts a hit iff the returned Synset.id is in the gold
  sense_ids_wordnet list (tolerates multi-gold + synset splits).
- report.py — aggregates JSONL into a method x config markdown table.
- results_lesk.jsonl — raw output from the first sweep (Lesk family +
  baselines across 5 Raganato all-words configs, pywsd 1.3.0).
- README.md — protocol + full results table + reading notes.

First-sweep results:
- `first_sense` wins every config (52-64 %) — MFS baseline is hard to
  beat with unsupervised Lesk at the all-words level.
- simple_lesk comes within 2-4 pp of first_sense on each.
- adapted_lesk slightly underperforms simple_lesk (more related-sense
  noise than signal on SE2/SE3).
- original_lesk collapses on fine-grained SemEval-2007 (15.65 %).

max_similarity sweep runs separately; results appended when done.

Author: alvations

experiments/README.md

@@ -0,0 +1,123 @@
+# experiments/
+
+Reproducible benchmarks of pywsd's algorithms against the unified
+[`alvations/pywsd-datasets`](https://huggingface.co/datasets/alvations/pywsd-datasets)
+test splits. **Not** part of the installable pywsd package — these
+scripts live here so anyone can re-run the numbers without re-deriving
+the evaluation pipeline.
+
+## Files
+
+* `evaluate.py` — runs each pywsd method against each test config on
+  the HuggingFace Hub dataset and records accuracy.
+* `report.py` — aggregates the JSONL results into a method × config
+  markdown table.
+* `results_lesk.jsonl` — raw run output: Lesk family + baselines across
+  the 5 Raganato all-words evaluation sets (pywsd 1.3.0, April 2026).
+* `results_maxsim.jsonl` — max_similarity metrics on SemEval-2007 (the
+  smallest eval set — these metrics are ~500 ms/instance so only run
+  over the 455-row set in this snapshot).
+
+## Setup
+
+```bash
+pip install pywsd datasets
+python -m nltk.downloader punkt_tab averaged_perceptron_tagger_eng wordnet
+python -c "import wn; wn.download('oewn:2024')"
+```
+
+## Run
+
+```bash
+# All defaults: every method, every all-words eval config.
+python experiments/evaluate.py --out experiments/results_lesk.jsonl
+
+# Subsets:
+python experiments/evaluate.py --configs en-senseval2-aw --limit 200
+
+# Aggregate:
+python experiments/report.py --files experiments/results_lesk.jsonl \
+                                      experiments/results_maxsim.jsonl
+```
+
+## Protocol
+
+* Target senses in `pywsd-datasets` are OEWN 2024 synset IDs
+  (`oewn-<offset>-<pos>`), mapped from the original PWN 3.0 sense keys
+  via `wn.compat.sensekey`.
+* A prediction counts as correct if the returned `Synset.id` is any
+  member of the gold `sense_ids_wordnet` list (handles multi-gold and
+  synset-split tolerance).
+* Instances where the PWN → OEWN map produced no target (empty gold
+  list) are excluded from both numerator and denominator and reported
+  in the `nogold` column.
+* The `errors` column counts instances where the method raised, or
+  returned `None`, or returned something without an `.id` attribute.
+
+## Results — Lesk family + baselines
+
+Across the 5 Raganato all-words evaluation configs (pywsd 1.3.0,
+`oewn:2024`, `wikipedia`-corpus IC).
+
+| method | SE2007 (AW) | SE2013 (AW) | SE2015 (AW) | Senseval-2 | Senseval-3 |
+|---|---:|---:|---:|---:|---:|
+| `first_sense`      | 52.76 | 57.65 | **64.61** | **60.62** | **61.46** |
+| `random_sense`     | 23.73 | 36.28 | 42.20 | 40.05 | 34.14 |
+| `max_lemma_count`  | 32.95 | 56.27 | 49.85 | 50.48 | 47.15 |
+| `original_lesk`    | 15.65 | 36.49 | 34.03 | 34.23 | 28.37 |
+| `simple_lesk`      | **47.70** | 55.34 | 61.90 | 58.64 | 55.19 |
+| `adapted_lesk`     | 47.00 | 55.34 | 60.98 | 57.19 | 54.79 |
+| `cosine_lesk`      | 32.03 | 44.72 | 48.11 | 45.67 | 41.38 |
+
+Cells are accuracy % (higher is better). Instance counts per config:
+SemEval-2007 455 (fine-grained), SemEval-2013 1,644, SemEval-2015
+1,022, Senseval-2 2,282, Senseval-3 1,850.
+
+### Reading
+
+* `first_sense` (most-frequent-sense heuristic over OEWN's first-sense
+  ordering) wins every config. Knowledge-based Lesk variants come
+  within 2–4 percentage points but never beat MFS — the well-known
+  difficulty of all-words WSD with unsupervised signals.
+* `simple_lesk` ≥ `adapted_lesk` ≥ `cosine_lesk` holds on every
+  config. Adapted Lesk's wider signature (holonyms/meronyms/similar)
+  slightly hurts on Senseval-2 and -3 — more noise than signal.
+* `original_lesk` (1986, definition-only overlap) collapses on the
+  fine-grained SemEval-2007 (15.65 %) as expected.
+* `max_lemma_count` is a surprisingly strong MFS proxy on SemEval-2013
+  (56.27 %, within ~1 pp of `first_sense`) but weak on the fine-grained
+  SemEval-2007 (32.95 %) where OEWN's per-sense counts are sparse.
+
+## Results — max_similarity (information-content family)
+
+Computed on SemEval-2007 all-words (455 rows) only, because each
+similarity-option run takes ~14 minutes per metric on this corpus
+(quadratic over candidate × context synsets).
+
+*(Results will be appended here as the sweep finishes. See
+`results_maxsim.jsonl` for raw JSON output.)*
+
+## Reproducibility
+
+Results above were generated with:
+
+* `pywsd==1.3.0`
+* `wn==1.1.0`, lexicon `oewn:2024`
+* `alvations/pywsd-datasets` built from
+  [v0.2.0](https://github.com/alvations/pywsd-datasets/releases/tag/v0.2.0),
+  which pins the Raganato bundle via a GitHub release mirror.
+* Python 3.12, macOS.
+
+Re-run with the exact commands above; scores should be within floating
+noise unless OEWN or the Raganato mirror moves under you.
+
+## What's not here (yet)
+
+* Ranked-list metrics (MAP / first-n accuracy with `nbest=True`).
+* Per-POS breakdown.
+* Evaluation on the UFSAC lexical-sample test splits
+  (`en-senseval2_ls`, `en-senseval3_ls`, `en-semeval2007_t17_ls`).
+  Lexical-sample WSD has a different protocol — confusion per target
+  lemma — that isn't reflected in this script's aggregate accuracy.
+
+Contributions welcome.

experiments/evaluate.py

@@ -0,0 +1,201 @@
+"""Evaluate pywsd WSD methods against the ``alvations/pywsd-datasets``
+test sets on HuggingFace Hub.
+
+For each test instance:
+
+1. Build the context string from its ``tokens``,
+2. Ask a pywsd method for the disambiguated sense of the target,
+3. Count a hit if the returned ``Synset.id`` is in the gold
+   ``sense_ids_wordnet`` list (list-valued to handle multi-gold +
+   synset-split cases).
+
+Instances where the dataset build failed to resolve any OEWN id for
+the gold PWN 3.0 sense key (empty ``sense_ids_wordnet``) are excluded
+from both numerator and denominator, and reported separately.
+
+Usage::
+
+    pip install pywsd datasets
+    python -m nltk.downloader punkt_tab averaged_perceptron_tagger_eng wordnet
+    python -c "import wn; wn.download('oewn:2024')"
+
+    python experiments/evaluate.py                          # all eval configs
+    python experiments/evaluate.py --configs en-senseval2-aw
+    python experiments/evaluate.py --methods simple_lesk first_sense
+    python experiments/evaluate.py --limit 200              # smoke
+    python experiments/evaluate.py --out results.jsonl
+"""
+
+from __future__ import annotations
+
+import argparse
+import json
+import sys
+import time
+from pathlib import Path
+
+
+# Test-only configs in alvations/pywsd-datasets.
+TEST_CONFIGS: dict[str, str] = {
+    "en-senseval2-aw":       "test",
+    "en-senseval3-aw":       "test",
+    "en-semeval2007-aw":     "test",
+    "en-semeval2013-aw":     "test",
+    "en-semeval2015-aw":     "test",
+    "en-senseval2_ls":       "test",
+    "en-senseval3_ls":       "test",
+    "en-semeval2007_t17_ls": "test",
+}
+
+
+def load_rows(config: str, split: str) -> list[dict]:
+    """Pull a split directly from HuggingFace Hub."""
+    from datasets import load_dataset
+    ds = load_dataset("alvations/pywsd-datasets", config)
+    return list(ds[split])
+
+
+def detokenize(tokens: list[str]) -> str:
+    return " ".join(tokens).replace("_", " ")
+
+
+def _wn_pos(pos: str) -> str | None:
+    return pos if pos in ("n", "v", "a", "r") else None
+
+
+def run_method(method: str, sentence: str, lemma: str, pos: str | None):
+    from pywsd.lesk import simple_lesk, adapted_lesk, cosine_lesk, original_lesk
+    from pywsd.similarity import max_similarity
+    from pywsd.baseline import first_sense, random_sense, max_lemma_count
+
+    if method == "simple_lesk":
+        return simple_lesk(sentence, lemma, pos=pos)
+    if method == "adapted_lesk":
+        return adapted_lesk(sentence, lemma, pos=pos)
+    if method == "cosine_lesk":
+        return cosine_lesk(sentence, lemma, pos=pos)
+    if method == "original_lesk":
+        return original_lesk(sentence, lemma)
+    if method.startswith("max_similarity_"):
+        opt = method.removeprefix("max_similarity_")
+        return max_similarity(sentence, lemma, option=opt, pos=pos)
+    if method == "first_sense":
+        try:
+            return first_sense(lemma, pos=pos)
+        except Exception:
+            return None
+    if method == "random_sense":
+        try:
+            return random_sense(lemma, pos=pos)
+        except Exception:
+            return None
+    if method == "max_lemma_count":
+        return max_lemma_count(lemma)
+    raise ValueError(f"unknown method {method!r}")
+
+
+DEFAULT_METHODS: list[str] = [
+    "first_sense",
+    "random_sense",
+    "max_lemma_count",
+    "original_lesk",
+    "simple_lesk",
+    "adapted_lesk",
+    "cosine_lesk",
+    "max_similarity_path",
+    "max_similarity_wup",
+    "max_similarity_lch",
+    "max_similarity_res",
+    "max_similarity_jcn",
+    "max_similarity_lin",
+]
+
+
+def evaluate_one(rows: list[dict], method: str, limit: int | None = None) -> dict:
+    total = 0
+    correct = 0
+    skipped_nogold = 0
+    errors = 0
+    t0 = time.time()
+    n = len(rows) if limit is None else min(limit, len(rows))
+    for i in range(n):
+        row = rows[i]
+        gold = row.get("sense_ids_wordnet") or []
+        if not gold:
+            skipped_nogold += 1
+            continue
+        sentence = detokenize(row["tokens"])
+        lemma = row["target_lemma"]
+        pos = _wn_pos(row["target_pos"])
+        try:
+            pred = run_method(method, sentence, lemma, pos)
+        except Exception:
+            errors += 1
+            continue
+        if pred is None:
+            errors += 1
+            continue
+        pid = getattr(pred, "id", None)
+        if pid is None:
+            errors += 1
+            continue
+        total += 1
+        if pid in gold:
+            correct += 1
+    elapsed = time.time() - t0
+    acc = correct / total if total else 0.0
+    return {
+        "method": method,
+        "total": total,
+        "correct": correct,
+        "accuracy": acc,
+        "skipped_nogold": skipped_nogold,
+        "errors": errors,
+        "elapsed_sec": elapsed,
+    }
+
+
+def main(argv: list[str] | None = None) -> int:
+    ap = argparse.ArgumentParser(description=__doc__,
+                                 formatter_class=argparse.RawDescriptionHelpFormatter)
+    ap.add_argument("--configs", nargs="*", default=list(TEST_CONFIGS))
+    ap.add_argument("--methods", nargs="*", default=DEFAULT_METHODS)
+    ap.add_argument("--limit", type=int, default=None,
+                    help="max rows per config (for quick runs)")
+    ap.add_argument("--out", type=Path, default=None,
+                    help="JSONL results dump (one row per config x method)")
+    args = ap.parse_args(argv)
+
+    # One-time WordNet handle / lexicon download.
+    from pywsd._wordnet import _get
+    _get()
+
+    results: list[dict] = []
+    for config in args.configs:
+        split = TEST_CONFIGS.get(config)
+        if split is None:
+            print(f"skip non-test config: {config}", file=sys.stderr)
+            continue
+        rows = load_rows(config, split)
+        print(f"\n## {config}/{split}   ({len(rows)} rows)")
+        print(f"{'method':<22} {'acc':>7} {'n':>6} {'err':>5} {'nogold':>7} {'sec':>7}")
+        for method in args.methods:
+            r = evaluate_one(rows, method, limit=args.limit)
+            r.update({"config": config, "split": split})
+            results.append(r)
+            print(f"{method:<22} {r['accuracy']*100:>6.2f}% {r['total']:>6} "
+                  f"{r['errors']:>5} {r['skipped_nogold']:>7} "
+                  f"{r['elapsed_sec']:>6.1f}s", flush=True)
+
+    if args.out:
+        args.out.parent.mkdir(parents=True, exist_ok=True)
+        with open(args.out, "w") as fh:
+            for r in results:
+                fh.write(json.dumps(r) + "\n")
+        print(f"\nwrote {args.out}")
+
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())