chore: execute P1-GET-ORGANIZED repo hygiene and MCP refactoring

- Enforce document frontmatter, folder promotion, and uppercase naming via project.sh

- Scrub client names and build repo metadata catalog

- Refactor MCP server to split env-probes and server-registry modules

- Add experimental live environment probe scripts

Author: noelsaw1

CHANGELOG.md

@@ -13,6 +13,30 @@ This project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.htm
 - Do not edit a version block that has already been committed and pushed
 -->
 
+## [2.1.1] - 2026-04-12
+
+### Changed
+- **`tools/mcp-server/src/index.ts` + `src/registrations/*`** — MCP server registration is now split by domain so the entrypoint is composition-only and schemas live with the tools/resources they describe, reducing the maintenance cost of adding or changing MCP surface area.
+- **External JSON contract hardening** — env probes, pw-auth JSON flows, Query Monitor profile retrieval, LocalWP JSON list parsing, and WPCC scan artifact loading now validate parsed JSON with Zod-backed schemas before returning data to MCP callers, turning silent shape drift into explicit contract errors.
+- **Server registry source of truth** — moved the port registry backend from markdown parsing to canonical JSON at `tools/servers.registry.json`, with `tools/servers.md` generated from that data block so MCP reads/writes stop depending on markdown table formatting.
+- **MCP test workflow** — normal unit tests and socket-binding tests are now separated into `test:unit`, `test:socket`, and `test:all`, so sandboxed environments can run the stable contract suite without failing on localhost/TCP or Unix-socket bind restrictions.
+
+## [2.1.0] - 2026-04-12
+
+### Added
+- **`PROJECT/1-INBOX/P1-GET-ORGANIZED.md`** — added a new inbox plan for repo organization work. The plan sequences inventory, lifecycle rules, metadata cataloging, cleanup, and hybrid lexical plus semantic retrieval so embeddings are treated as a discovery layer after structure and retention rules are in place.
+- **MCP tools: `servers_monitor_check`, `dev_context_status`** — two new live environment probe tools in MCP server v0.11.0. `servers_monitor_check` wraps `experimental/servers-monitor.sh --json` to detect port conflicts, wildcard binds on 80/443, stale `/etc/hosts` entries, and missing services; returns severity-grouped counts and structured issue list. `dev_context_status` wraps `experimental/dev-context.sh status --json` to report current development mode (`valet` / `localwp` / `conflict` / `none`), service health, port 80 listeners, and Valet proxy registrations. Both fail soft when their backing scripts are missing or unconfigured (returning `not_installed` / `not_configured`) so they're safe to ship across machines without custom setup. Intended use: agents call these at session start to surface port 80 issues *before* the user asks "why can't I reach my site?".
+- **`handlers/env-probes.ts`** — new MCP handler module for live machine state probes. Split from `handlers/servers.ts` (which stays pure and only reads/writes the `tools/servers.md` registry) to keep file-parsing logic separate from shell-execution logic. Accepts dependency injection for script paths and exec implementation, enabling per-deployment configuration and test isolation. Gracefully degrades when scripts are missing — probes return structured "not installed" states rather than throwing.
+- **`experimental/servers-monitor.sh`** — 30-minute conflict monitor for local dev environments with Resend.com email alerts. Checks 5 categories: multiple-process port conflicts, `0.0.0.0:80/443` wildcard binds (breaks coexistence), stale `.local` `/etc/hosts` entries (cross-referenced against Local WP `sites.json`), expected service health (dnsmasq, MySQL), and Valet IP binding correctness. Dedupes by issue-set hash so the same steady-state conflict never re-alerts. New `--json` flag emits structured output to stdout (skips email, skips baseline update) for MCP consumption. Severity tiers: CRITICAL / HIGH / MEDIUM / LOW / FYI with project/service context per port. Config at `~/secrets/servers-monitor.conf` (outside repo, never committed); example template at `experimental/servers-monitor.conf.example`.
+- **`experimental/dev-context.sh`** — context switcher for the Local WP ↔ Valet port 80 mutex. `dev-context localwp` stops Valet nginx (frees `127.0.0.1:80` for Local WP's wildcard bind) while leaving dnsmasq running so `*.test` DNS still resolves. `dev-context valet` restores Valet (refuses if Local WP router is still up). `dev-context status` reports current mode, service health, and port 80 listeners. New `status --json` flag for MCP consumption. Docker Dify is never touched — it runs on dedicated high ports (8741/8742) and is orthogonal to the port 80 mutex.
+- **`experimental/local-nginx-shim`** + **`experimental/local-nginx-shim-install.sh`** — fallback workaround for Local WP's hard-coded `listen 80` (wildcard) router binding. The shim is a shell wrapper installed in place of Local WP's bundled nginx binary; at runtime it rewrites router-only config files to `listen 127.0.0.1:80` before exec'ing the real nginx (backed up as `nginx.real`). Only touches files under `run/router/nginx/conf/` — per-site configs on high ports are untouched. Installer is idempotent, refuses to run while Local WP is active, and supports clean uninstall. This enables true coexistence (Local WP + Valet + Docker Dify all on port 80 simultaneously on separate loopback IPs) as an alternative to the `dev-context` mutex approach. Re-run `install` after Local WP app updates.
+- **Preflight step 4: local server environment check** — `experimental/preflight.md` now instructs agents to call `servers_monitor_check` and `dev_context_status` at session start (when the task involves a live site), so critical/high severity conflicts are surfaced before diagnostic questions arise. Numbered the subsequent steps accordingly.
+- **`~/bin/servers.md` (machine-specific) + `tools/servers.md` (generic template)** — forked the registry doc into a machine-specific snapshot (real ports, 21 Local WP site IDs, resolved-conflicts history, changelog) and a generic template used by `servers-audit.sh`. Both files include a **Troubleshooting Matrix** (11 symptom → fix rows) and an 8-branch **Decision Tree for LLM Agents** covering: `.test` reachability, `.local` reachability, `0.0.0.0:80/443` wildcard binds, false-positive port conflicts (nginx worker pattern, IPv4/IPv6 dual-stack, macOS system services), database shadowing, Valet config regeneration from stubs, `www.<site>` stale-entry false positives, and DNS cache staleness after `/etc/hosts` edits.
+
+### Changed
+- **MCP server version bump to v0.11.0** — new Environment Probes area added; tool count updated from 26 to 28 across the same 7 user-facing areas (probes are grouped under "Server Registry" thematically).
+- **`experimental/servers-monitor.sh` false-positive fixes** — three classes of false positives fixed: (1) dedupe port-conflict detection by unique process **command**, not PID, so nginx's 10 workers sharing one listening socket no longer register as 10 conflicts; (2) handle IPv4+IPv6 dual-stack binds (e.g., Postgres on `127.0.0.1:5432` and `[::1]:5432`) as a single listener; (3) strip `www.` prefix when comparing `/etc/hosts` entries against Local WP `sites.json` (Local WP writes `www.<domain>` aliases but only stores the bare domain). macOS system services (ControlCenter AirPlay on 5000/7000, rapportd on 64343) classified as FYI severity to prevent noise.
+
 ## [2.0.0] - 2026-04-11
 
 ### Added

fix-iterate-loop.md FIX-ITERATE-LOOP.mdfix-iterate-loop.md renamed to FIX-ITERATE-LOOP.md

@@ -0,0 +1,226 @@
+---
+title: "P1: Get Organized"
+status: inbox
+priority: P1
+created: 2026-04-11
+updated: 2026-04-11
+author: "GitHub Copilot"
+goal: "Create a practical repo-organization plan for AI-DDTK that reduces sprawl first and adds semantic retrieval only where it materially improves discovery."
+---
+
+<!-- TOC -->
+
+- [Phased Checklist (High-Level Progress)](#phased-checklist-high-level-progress)
+- [Overview](#overview)
+- [Goals](#goals)
+- [Non-Goals](#non-goals)
+- [Guiding Principle](#guiding-principle)
+- [Phase 0 — Inventory and Classification](#phase-0--inventory-and-classification)
+- [Phase 1 — Canonical Structure and Retention Rules](#phase-1--canonical-structure-and-retention-rules)
+- [Phase 2 — Build a Repo Metadata Catalog](#phase-2--build-a-repo-metadata-catalog)
+- [Phase 3 — Cleanup, Promotion, and Archival Pass](#phase-3--cleanup-promotion-and-archival-pass)
+- [Phase 4 — Add Hybrid Retrieval Where It Helps](#phase-4--add-hybrid-retrieval-where-it-helps)
+- [Phase 5 — Operating Rhythm and Ownership](#phase-5--operating-rhythm-and-ownership)
+- [Success Criteria](#success-criteria)
+- [Open Questions](#open-questions)
+
+<!-- /TOC -->
+
+---
+
+## Phased Checklist (High-Level Progress)
+
+> This document should be updated as work is completed. Mark off items immediately rather than batching status updates later.
+
+- [ ] **Phase 0 — Inventory and Classification**
+- [ ] **Phase 1 — Canonical Structure and Retention Rules**
+- [ ] **Phase 2 — Build a Repo Metadata Catalog**
+- [ ] **Phase 3 — Cleanup, Promotion, and Archival Pass**
+- [ ] **Phase 4 — Add Hybrid Retrieval Where It Helps**
+- [ ] **Phase 5 — Operating Rhythm and Ownership**
+
+## Overview
+
+AI-DDTK has grown into a toolkit repo with code, docs, recipes, project tracking, experiments, generated reports, and operational artifacts. The current problem is not only search. It is lifecycle clarity: what is canonical, what is in progress, what is generated, what is temporary, and what should be archived.
+
+Embeddings can help with discovery, clustering, and semantic lookup across notes, reports, and docs. They do not replace naming rules, retention rules, or folder discipline. This plan treats semantic retrieval as a later layer added on top of a cleaner repo model.
+
+## Goals
+
+- [ ] Reduce ambiguity about where new files belong.
+- [ ] Separate source-of-truth files from generated artifacts and temporary outputs.
+- [ ] Create a machine-readable catalog of important files and directories.
+- [ ] Make cleanup repeatable instead of one-off.
+- [ ] Add semantic retrieval only after the repo has usable metadata and folder hygiene.
+
+## Non-Goals
+
+- [ ] Do not redesign the entire repo structure in one pass.
+- [ ] Do not build a full knowledge platform before cleanup basics exist.
+- [ ] Do not index every file blindly into a vector store.
+- [ ] Do not treat generated artifacts as equal to canonical docs or source code.
+
+## Guiding Principle
+
+Organize first, index second.
+
+If the repo lacks clear file lifecycle rules, embeddings will help search the mess without reducing the mess. The right sequence is:
+
+1. Inventory the repo.
+2. Classify files by lifecycle and purpose.
+3. Clean up and normalize the highest-noise areas.
+4. Add metadata-backed discovery.
+5. Add hybrid lexical + semantic retrieval where it clearly improves workflow.
+
+## Phase 0 — Inventory and Classification
+
+Purpose: build a factual snapshot of what exists before making structural changes.
+
+### Checklist
+
+- [ ] Generate a repo-wide file inventory using tracked files as the baseline.
+- [ ] Break the inventory down by top-level area: `tools/`, `experimental/`, `PROJECT/`, `docs/`, `recipes/`, `templates/`, `test/`, `bin/`, `temp/`.
+- [ ] For each area, label files into one of these classes: `canonical-source`, `documentation`, `project-tracking`, `generated-artifact`, `temporary`, `experimental`, `archive-candidate`.
+- [ ] Identify the noisiest zones by count and churn, not by intuition.
+- [ ] Flag directories that mix multiple lifecycles in one place.
+- [ ] Produce a first-pass list of files that are probably duplicated, stale, or misfiled.
+- [ ] Record which generated outputs are intentionally checked in versus accidentally lingering.
+
+### Deliverable
+
+- [ ] A first-pass inventory snapshot stored in a machine-readable format such as JSON or CSV.
+
+### Exit Criteria
+
+- [ ] We can answer, with evidence, which parts of the repo are source, working notes, generated output, and probable cleanup targets.
+
+## Phase 1 — Canonical Structure and Retention Rules
+
+Purpose: define what belongs where and how long it should live.
+
+### Checklist
+
+- [ ] Define the canonical purpose of each top-level directory in one sentence.
+- [ ] Confirm `PROJECT/` is only for planning, tracking, inbox, working, and done states.
+- [ ] Confirm `temp/` is for sensitive and disposable runtime artifacts, not long-term reference docs.
+- [ ] Define what qualifies for `experimental/` and what conditions trigger promotion out of it.
+- [ ] Define which report outputs belong in-repo versus gitignored runtime storage.
+- [ ] Review `.gitignore` coverage for reports, screenshots, scans, auth state, and logs.
+- [ ] Write simple retention rules for generated content: keep, archive, or purge.
+- [ ] Decide what must always have an owning doc or README in dense directories.
+
+### Deliverable
+
+- [ ] A short policy section or reference doc update that names the lifecycle rules for canonical, experimental, generated, and temporary files.
+
+### Exit Criteria
+
+- [ ] A contributor can decide where a new file belongs without guessing.
+
+## Phase 2 — Build a Repo Metadata Catalog
+
+Purpose: create a lightweight system of record for discovery and cleanup.
+
+### Checklist
+
+- [ ] Define the metadata schema for the catalog.
+- [ ] Include at minimum: `path`, `area`, `file_type`, `lifecycle_class`, `owner_tool`, `canonical`, `generated`, `last_modified`, `status`, `notes`.
+- [ ] Add optional tags for themes such as `mcp`, `wpcc`, `playwright`, `local-wp`, `query-monitor`, `servers`, `project-doc`.
+- [ ] Generate the initial catalog automatically from the repo rather than maintaining it by hand.
+- [ ] Add a rule for how manual overrides are stored when auto-detection is wrong.
+- [ ] Mark high-value files explicitly as canonical references.
+- [ ] Mark low-value files explicitly as cleanup or archive candidates.
+- [ ] Decide where the catalog lives and whether it is checked in or regenerated.
+
+### Deliverable
+
+- [ ] A machine-readable catalog that can drive cleanup reports, folder summaries, and later search indexing.
+
+### Exit Criteria
+
+- [ ] We can query the repo by lifecycle and ownership instead of relying only on folder names.
+
+## Phase 3 — Cleanup, Promotion, and Archival Pass
+
+Purpose: reduce noise using the inventory and catalog rather than ad hoc decisions.
+
+### Checklist
+
+- [ ] Triage `PROJECT/1-INBOX` items into active, done, or misc states using existing doc rules.
+- [ ] Move finished project docs out of inbox.
+- [ ] Review `experimental/` for tools or docs that have effectively graduated.
+- [ ] Move obsolete or superseded planning docs to the appropriate archive location instead of leaving duplicates in place.
+- [ ] Consolidate duplicate instructions where one doc clearly supersedes another.
+- [ ] Remove or archive tracked generated artifacts that do not belong in the main repo surface.
+- [ ] Add missing README or index guidance in dense directories only where it reduces ambiguity.
+- [ ] Re-run the inventory after cleanup and measure count reduction and clearer classification.
+
+### Deliverable
+
+- [ ] A visibly smaller and more legible repo surface, especially in project-tracking and experimental areas.
+
+### Exit Criteria
+
+- [ ] The highest-noise folders have fewer ambiguous files and clearer ownership.
+
+## Phase 4 — Add Hybrid Retrieval Where It Helps
+
+Purpose: improve discovery after structure exists.
+
+### Checklist
+
+- [ ] Start with hybrid retrieval, not embeddings alone.
+- [ ] Use lexical search for exact names, paths, commands, headings, and schema keys.
+- [ ] Use embeddings for semantic discovery across docs, reports, changelog entries, project notes, and scan outputs.
+- [ ] Index docs and operational artifacts first.
+- [ ] Add code chunks only when documentation is insufficient for the target workflow.
+- [ ] Exclude binaries, screenshots, lockfiles, auth state, and highly repetitive output unless there is a clear use case.
+- [ ] Test real queries against the index before expanding scope.
+- [ ] Define success queries such as: “show me all files related to LocalWP auth failures” or “find similar past WPCC performance investigations.”
+
+### Deliverable
+
+- [ ] A narrow, high-signal retrieval layer aimed at discovery and clustering, not as a substitute for repo organization.
+
+### Exit Criteria
+
+- [ ] Semantic lookup answers real questions faster than plain grep without pulling in obvious noise.
+
+## Phase 5 — Operating Rhythm and Ownership
+
+Purpose: keep the repo organized after the first cleanup pass.
+
+### Checklist
+
+- [ ] Assign an owner or review rule for repo hygiene changes.
+- [ ] Add a recurring cleanup cadence for inbox, experimental, and generated-output areas.
+- [ ] Add a lightweight checklist for “new file acceptance” so artifacts do not accumulate silently.
+- [ ] Require new generated-output directories to declare whether they are checked in or gitignored.
+- [ ] Review the metadata catalog on a schedule rather than only during cleanup crises.
+- [ ] Add a simple report that shows new files by lifecycle class since the last review.
+- [ ] Revisit the hybrid index scope after one or two cleanup cycles.
+
+### Deliverable
+
+- [ ] A repeatable maintenance loop that prevents the repo from drifting back into ambiguity.
+
+### Exit Criteria
+
+- [ ] Repo organization becomes an operating habit, not a one-time project.
+
+## Success Criteria
+
+- [ ] The top-level repo areas each have a clear and enforced purpose.
+- [ ] New files can be classified quickly as canonical, generated, temporary, experimental, or project-tracking.
+- [ ] The noisiest folders have been reduced and normalized.
+- [ ] A metadata catalog exists and can be regenerated.
+- [ ] Semantic retrieval is scoped to the parts of the repo where it genuinely improves discovery.
+- [ ] Contributors can find the right file faster without memorizing tribal knowledge.
+
+## Open Questions
+
+- [ ] Should the metadata catalog live under `PROJECT/`, `tools/`, or a new repo-maintenance location?
+- [ ] Which generated artifacts are intentionally committed because they provide durable value?
+- [ ] Which `experimental/` items are actually production-grade and just waiting for promotion?
+- [ ] Should repo hygiene checks become part of `preflight.sh`, `post-flight.sh`, or a separate maintenance command?
+- [ ] Is the first retrieval target this repo alone, or this repo plus neighboring WordPress project artifacts and reports?

PROJECT/1-INBOX/P1-GET-ORGANIZED.md

@@ -1,3 +1,13 @@
+---
+title: "Post-Flight Session Cleanup Script"
+status: inbox
+priority: P1
+created: 2026-04-11
+updated: 2026-04-11
+author: noelsaw
+goal: 
+---
+
 # Post-Flight Session Cleanup Script
 
 ## Overview