Merge pull request #113 from DenisValeev/codex/add-project-documentation-in-md-format

Author: DenisValeev

AGENTS.md

@@ -29,3 +29,8 @@ This project is a static toolbox site. The landing page (`index.html`) lists sma
 - If you skip scripted coverage, at least open the affected HTML files in a browser (or start `python -m http.server`) to confirm layout, theme switching, and keyboard interactions still work.
 - If you adjust the asset observatory, regenerate its dataset snapshot with `node tools/generate-asset-report.js` so `apps/asset-observatory/asset-data.js` stays in sync.
 - A new regression spec enforces Home buttons across apps—after tweaking navigation chrome, run `npx playwright test tests/home-navigation.spec.js`.
+
+## Documentation
+- The `ai_docs/` directory hosts the Markdown knowledge base. Update the relevant guides whenever you change workflows, scripts, or datasets so the docs remain authoritative.
+- When you add, remove, or rename a guide, update the `guides` array in `ai_docs/index.html` so the navigation stays in sync.
+- After editing documentation, open `ai_docs/index.html` in a browser (or via `npx serve .`) to confirm the Markdown renders through Marked and the hash navigation works.

ai_docs/AGENTS.md

@@ -0,0 +1,7 @@
+# AI documentation workspace
+
+- Keep documentation in Markdown under `ai_docs/docs/`. Use descriptive headings and prefer tables for directory overviews.
+- Whenever you add a new guide, update `ai_docs/index.html` so the navigation lists the page. Each entry in the `guides` array should include an `id`, `title`, `file`, and concise `summary`.
+- Render tweaks should preserve the two-space indentation used throughout the HTML and CSS. Stick with the existing palette and typography to keep the docs visually aligned with the landing page.
+- After editing docs, load `ai_docs/index.html` in a browser (or via `npx serve .`) to confirm the Markdown renders and navigation links update the hash correctly.
+- Keep cross-references current: when workflows, scripts, or tests change, update the relevant Markdown sections and summaries so the knowledge base stays authoritative.

ai_docs/docs/apps.md

@@ -0,0 +1,79 @@
+# App catalog
+
+Each mini app lives under `apps/<name>/` with inline styles, a focused script, and any supporting datasets it needs. The landing page links directly to every `index.html`, and many collections ship an additional `all-*.html` archive that renders the full dataset in a searchable table. This catalog summarises the responsibilities, data dependencies, and regression coverage for each tool.
+
+## Dad Jokes (`apps/jokes`)
+
+- **Purpose.** Shuffle a curated deck of developer-friendly jokes, reveal punchlines on demand, and explore the full archive when you need deeper context.
+- **Key files.**
+  - `index.html` – Deck UI with shuffle/prev/next controls, keyboard shortcuts, and a punchline toggle.
+  - `app.js` – Guards missing DOM nodes, shuffles the dataset, renders the current joke, and wires keyboard shortcuts (`ArrowLeft`, `ArrowRight`, `Space`).
+  - `jokes.js` – Dataset that assigns an array of `{ joke, punchline }` objects to `window.jokes`.
+  - `all-jokes.html` & `render-all.js` – Table view with search/filter controls and a badge that reflects the dataset count via `data-count`.
+- **Data maintenance.** Update jokes with `tools/update-content-metadata.js`, regenerate embeddings with `tools/review-content-similarity.js`, and use the commands in `apps/jokes/AGENTS.md` to validate syntax and spot duplicates.
+- **Regression coverage.** `tests/jokes.spec.js` covers deck navigation; `tests/jokes-dataset.spec.js` verifies dataset invariants; `tests/home-navigation.spec.js` confirms the Home shortcut.
+
+## Quotes (`apps/quotes`)
+
+- **Purpose.** Surface themed quote decks with shuffle controls, reveal the full backlog, and maintain deterministic metadata for dedupe analysis.
+- **Key files.**
+  - `index.html` – Shuffle UI mirroring the jokes experience.
+  - `app.js` – Maintains the active deck, handles keyboard shortcuts, and toggles the quote attribution display.
+  - `quotes-data.js` – Dataset assigned to `window.quotesData` with `{ text, author, tags }` entries.
+  - `all-quotes.html` & `render-all.js` – Archive table with search and filter affordances plus the dataset count badge.
+- **Data maintenance.** Refresh the dataset metadata with `tools/update-content-metadata.js --dataset=quotes`, rerun embeddings for duplicate detection, and keep `data/quotes-embeddings*.json` in sync. Similarity overrides live in `data/similarity-overrides.json`.
+- **Regression coverage.** `tests/quotes.spec.js` exercises the deck UI, `tests/quotes-all.spec.js` covers the archive table, and `tests/quotes-data.spec.js` checks dataset integrity.
+
+## Slang (`apps/slang`)
+
+- **Purpose.** Educate visitors on trending internet slang with reveal-on-demand definitions and a searchable archive.
+- **Key files.**
+  - `index.html` and `app.js` – Shuffle UI, reveal toggle, and keyboard shortcuts identical to the jokes/quotes decks but targeting `window.slangEntries`.
+  - `slang.js` – Dataset assigned to `window.slangEntries`.
+  - `all-slang.html` & `render-all.js` – Archive listing with search, filters, and the dataset count badge.
+- **Data maintenance.** Follow `apps/slang/AGENTS.md`: run the syntax checks, regenerate metadata (`tools/update-content-metadata.js --dataset=slang`), and refresh embeddings via `tools/review-content-similarity.js` for the synthetic, Hugging Face, and 128-dimension stores.
+- **Regression coverage.** `tests/slang-app.spec.js`, `tests/slang-all.spec.js`, and `tests/slang-dataset.spec.js` validate the UI, archive view, and dataset respectively.
+
+## Cosine Similarity Lab (`apps/similarity-report`)
+
+- **Purpose.** Visualise cosine-similar joke/quote/slang pairs, toggle datasets and providers, and inspect per-entry vector statistics.
+- **Key files.**
+  - `index.html` – Standalone HTML+JS document that fetches the similarity reports under `data/`, renders dataset toggles, and presents side-by-side comparisons.
+  - Consumes `data/similarity-report-*.json` plus overrides declared in `data/similarity-overrides.json`.
+- **Data maintenance.** Regenerate similarity reports with `tools/review-content-similarity.js` whenever embeddings change. The tool reads cross-deck reports (`data/similarity-report-cross-deck*.json`) alongside per-dataset outputs.
+- **Regression coverage.** `tests/similarity-report.spec.js` exercises dataset filtering, score thresholds, and keyboard shortcuts.
+
+## Asset Observatory (`apps/asset-observatory`)
+
+- **Purpose.** Provide a high-level dashboard for dataset counts, embedding stores, similarity reports, and upstream sources powering the toolbox.
+- **Key files.**
+  - `index.html` – Layout and visualisations for the dataset cards, provider summaries, cross-deck rollups, and ledger table.
+  - `app.js` – Loads `asset-data.js`, renders the dashboard, and wires filtering controls.
+  - `asset-data.js` – Machine-generated snapshot produced by `tools/generate-asset-report.js`.
+- **Data maintenance.** Run `node tools/generate-asset-report.js` whenever datasets, embeddings, or similarity reports change. The `apps/asset-observatory/AGENTS.md` file captures the workflow and regression command.
+- **Regression coverage.** `tests/asset-observatory.spec.js` ensures cards, charts, and counts stay aligned with the snapshot.
+
+## Embedding Explorer (`apps/embedding-explorer`)
+
+- **Purpose.** Inspect embedding vectors, compare dataset records, and visualise high-impact dimensions.
+- **Key files.**
+  - `index.html` – Dashboard layout with dataset selector, record filter, neighbor list, and dual canvas plots.
+  - `app.js` – Parses sample embeddings, decodes base64 vectors, computes statistics, renders histograms, and highlights the strongest positive/negative dimensions.
+  - `sample-embeddings.js` – Bundled dataset and metadata sources exposed as `window.embeddingExplorerSamples` and `window.embeddingSources`.
+- **Data maintenance.** Update `sample-embeddings.js` when adding new demo vectors. The explorer reads live datasets by decoding the base64 payloads from the embeddings stores in `data/` when available.
+- **Regression coverage.** `tests/embedding-explorer.spec.js` verifies dataset switching, record filtering, neighbor insights, and chart rendering.
+
+## Value Formatter (`apps/value-formatter`)
+
+- **Purpose.** Transform newline-separated values into templated output for quick copy/paste workflows.
+- **Key files.**
+  - `index.html` – Single-page interface with preset transforms, custom transform inputs, and textareas for input/output. Stores the latest input in `localStorage` so the session persists.
+  - All logic lives inline inside `<script>` tags; the app does not depend on external datasets.
+- **Data maintenance.** None required beyond keeping the presets up to date. The tool stores user input locally and has no background data dependencies.
+- **Regression coverage.** `tests/value-formatter.spec.js` validates presets, persistence, and manual transformations.
+
+## Landing page integration
+
+- Every app exposes a “Home” navigation link that returns to `../../`, satisfying `tests/home-navigation.spec.js` and `tests/home.spec.js`.
+- The landing page’s `tests/index.spec.js` ensures new tools register their cards and developer shortcuts.
+- When adding an app, update `index.html` with the new card and extend the offline manifest so the service worker caches the new assets.

ai_docs/docs/architecture.md

@@ -0,0 +1,42 @@
+# Site architecture
+
+The toolbox intentionally avoids a build pipeline. Each page is authored as standalone HTML with inline CSS and a small script bundle. The repository’s shared behaviours live in the landing page, the offline bootstrap flow, and the service worker. This section documents how those pieces interact.
+
+## Landing page (`index.html`)
+
+- **Layout and theming.** The page renders the Toolbox banner plus grouped rows for deck-based apps (jokes, quotes, slang) and utilities (formatters, explorers, reports). CSS custom properties define the light and dark palettes, and the document honours `prefers-color-scheme` so browsers automatically switch palettes.
+- **App rows.** Each `<li class="app-row">` contains labelled buttons that deep-link into the relevant app or its archival pages (such as `apps/jokes/all-jokes.html`). Buttons rely on inline SVG chevrons copied across the project to keep the iconography consistent.
+- **Offline call-to-action.** The card with `data-offline-card` summarises the cached bundle, renders progress while assets stream in, and exposes a “Refresh offline bundle” button that triggers a manual cache reset. A progress bar pinned to the top edge communicates status for screen reader users via `aria-live` updates.
+- **Client bootstrap.** A script near the bottom of the document orchestrates service worker registration, fetches `offline-manifest.json`, displays bundle metadata, and pipes progress updates into the visual meter. It also exposes a `data-offline-reset` button that calls into the worker to rebuild caches on demand.
+
+## Offline manifest (`offline-manifest.json`)
+
+- Generated by `node tools/generate-offline-manifest.js` (surfaced via `npm run build:offline`).
+- Lists every HTML, JavaScript, and JSON asset shipped with the site, together with byte sizes and a combined SHA-256 digest.
+- Includes the Git commit metadata embedded by the generator so the landing page can show which revision produced the bundle.
+- Stored under versioned keys such as `"version": "2024-04-02T20-19-04"`, which doubles as the cache suffix used by the service worker.
+
+## Service worker (`service-worker.js`)
+
+- **Cache naming.** Uses a `toolbox-offline-v*` prefix, where `*` matches the manifest’s version. The worker persists the active version in `toolbox-offline-metadata` so it can recover after reloads.
+- **Lifecycle.** On install it calls `skipWaiting()` so the newest worker activates immediately. Activation reads the stored manifest, deletes stale caches, and claims open clients.
+- **Fetch handling.**
+  - Navigation requests fall back to the cached landing page when offline.
+  - Requests for `offline-manifest.json` are intercepted so the worker can serve a cached copy while streaming updates in the background.
+  - Static asset requests are served from the active cache first, with a network fallback that backfills the cache.
+- **Messaging.** Listens for two message types:
+  - `apply-manifest` enqueues a cache update using the provided manifest payload.
+  - `reset-offline-cache` clears caches and re-runs the manifest application from scratch.
+- **Progress reporting.** Tracks how many assets and bytes have been cached and posts structured messages back to the landing page. The page renders this information inside the progress meter and the offline summary card.
+
+## Data loading
+
+- Deck-style apps (jokes, quotes, slang) expose datasets by assigning arrays to globals (`window.jokes`, `window.proverbsData`, etc.). Each app imports the dataset script via `<script src="..." defer>` and reads it synchronously during initialisation.
+- Heavy reports (asset observatory, similarity report) load JSON snapshots that live under `data/`. Generators in `tools/` keep those snapshots fresh.
+- The embedding explorer includes bundled samples (`apps/embedding-explorer/sample-embeddings.js`) so the interface can demonstrate statistics without requiring external APIs.
+
+## Packaging and distribution
+
+- `package.json` declares a `pretest` hook that ensures Playwright’s Chromium build is available before tests run. No bundler dependencies are required.
+- Opening any HTML file from disk is sufficient during development, but `npx serve .` (or any HTTP server) enables service worker registration and cross-document navigation.
+- The offline bundle allows end users to “install” the toolbox on mobile devices by caching the entire asset list and pinning the landing page to the home screen.

ai_docs/docs/data-maintenance.md

@@ -0,0 +1,69 @@
+# Data maintenance
+
+Content-heavy apps rely on shared datasets stored under `data/` plus machine-generated snapshots that live alongside each app. This guide documents the files that need to stay in sync and the scripts that regenerate them.
+
+## Directory overview
+
+| File | Description |
+| --- | --- |
+| `jokes-manifest.json`, `quotes-manifest.json`, `slang-manifest.json` | Deterministic manifests produced by `tools/update-content-metadata.js`. They record record hashes, canonical IDs, total counts, and metadata used by downstream scripts. |
+| `*-embeddings.json`, `*-embeddings-openai.json`, `*-embeddings-cohere.json`, `slang-embeddings-hfspace.json`, `slang-embeddings-synthetic128.json` | Embedding stores keyed by record IDs. These power the similarity reports, the embedding explorer, and duplicate detection workflows. |
+| `similarity-report-*.json` | Cosine similarity sweeps grouped by dataset and provider (synthetic, OpenAI, Cohere, HF space, synthetic-128). Cross-deck variants highlight overlaps between jokes, quotes, and slang. |
+| `similarity-overrides.json` | Flags intentional near-duplicates so the cosine similarity lab can highlight protected pairs instead of suggesting removals. |
+| `test-runs/` | Archives Playwright trace data and logs captured by `tools/record-playwright-log.js` to help diagnose failures. |
+
+## Updating deck datasets
+
+1. Edit the dataset files under `apps/jokes/jokes.js`, `apps/quotes/quotes-data.js`, or `apps/slang/slang.js`.
+2. Run the syntax and shape checks documented in each app’s `AGENTS.md` file (`node --check ...` and the `node -e` globals check).
+3. Regenerate metadata and manifests:
+   ```bash
+   node tools/update-content-metadata.js --dataset=jokes
+   node tools/update-content-metadata.js --dataset=quotes
+   node tools/update-content-metadata.js --dataset=slang
+   ```
+   Pass `--dataset=<name>` to target a single collection when needed.
+4. Refresh embeddings for duplicate detection. Typical commands:
+   ```bash
+   node tools/review-content-similarity.js --dataset=jokes --provider=synthetic --write --update-manifest
+   node tools/review-content-similarity.js --dataset=jokes --provider=hfspace --model=bienkieu/sentence-embedding --batch-size=8 --report=data/similarity-report-jokes.json
+   node tools/review-content-similarity.js --dataset=quotes --provider=openai --write --update-manifest
+   node tools/review-content-similarity.js --dataset=slang --provider=synthetic --threshold-slang=0.8 --write --update-manifest
+   ```
+   Adjust providers and options according to your API access and the thresholds documented in `apps/slang/AGENTS.md` and the repository README.
+5. Review high-similarity pairs and move intentional overlaps into `data/similarity-overrides.json` so the similarity lab highlights them as protected.
+6. Commit refreshed datasets, manifests, embeddings, and similarity reports together to keep the bundle consistent.
+
+## Regenerating similarity reports
+
+- `tools/review-content-similarity.js` emits JSON files such as `data/similarity-report-jokes.json`, `data/similarity-report-quotes-openai.json`, and `data/similarity-report-cross-deck.json`.
+- Pass `--provider=<source>` plus `--write` and `--report=...` to persist new embeddings and the matching report. Providers include:
+  - `synthetic` for deterministic vectors embedded locally.
+  - `hfspace` for the BienKieu Hugging Face Space (`sentence-transformers/all-MiniLM-L6-v2`).
+  - `openai` and `cohere` for API-based embeddings when credentials are available.
+- Use `--threshold-<dataset>` flags to enforce dataset-specific duplication caps (for example, `--threshold-slang=0.8`).
+- After generating reports, rerun `node tools/generate-asset-report.js` so the asset observatory reflects the new counts and byte sizes.
+
+## Maintaining the asset observatory snapshot
+
+- The observatory reads `apps/asset-observatory/asset-data.js`, a machine-generated file containing dataset counts, provider rollups, cross-deck statistics, and ledger rows.
+- Regenerate it via:
+  ```bash
+  node tools/generate-asset-report.js
+  ```
+- The script inspects dataset files, manifests, embedding stores, similarity reports, and upstream sources. It records totals for entries, bytes, and asset counts per provider, plus cross-deck similarity pairs.
+- Commit the updated snapshot along with the underlying dataset changes so the dashboard stays consistent.
+
+## Offline manifest refreshes
+
+- Any change to HTML, JavaScript, or JSON assets requires regenerating `offline-manifest.json`:
+  ```bash
+  npm run build:offline
+  ```
+- The generator computes SHA-256 digests, total bytes, asset counts, and Git metadata. The manifest’s version feeds directly into the service worker’s cache key.
+- After updating the manifest, reload the landing page to confirm the offline card reflects the new bundle size.
+
+## Diagnostics and audit logs
+
+- Run `node tools/record-playwright-log.js` to capture Playwright traces for failing specs. Output lives under `data/test-runs/` with timestamps and spec names.
+- Use the JSON artifacts inside `test-runs/` to reproduce test paths or share failure context in pull requests.