v7.27.0 — Hardening & Polish (audit backlog)

Closes 17 audit findings spanning security, reliability, and schema integrity.

Critical / High:
- H1 sensor_readings table conflict (renamed Phase 18 -> iot_sensor_readings)
- H3 mutating rate limit actually enforced (per-IP sliding window, localhost exempt)
- H5 path traversal — commonpath replaces fragile normcase prefix on Win
- #10 disk-space pre-check before yt-dlp downloads

Medium:
- M1 LIMIT/OFFSET pagination via shared get_pagination() across 7 blueprints
  (financial, daily_living, training_knowledge, hunting_foraging,
   disaster_modules, movement_ops, evac_drills — 17 list endpoints)
- M2 log_activity audit trail added to contacts + vehicles CRUD
- M3 SSE client limit race condition collapsed into single locked path
- M4 access_logs renamed to platform_access_log + idempotent migration
- M7 _env_int() helper — bad NOMAD_PORT no longer crashes at import
- H2 input validation (financial — 4 schemas, 8 mutating endpoints)
- #11 Ollama streaming resilience — malformed chunks skipped not forwarded
- #12 Streaming CSV import for contacts (no full-buffer load)
- #13 Duty roster cleanup on pod member removal

Low:
- L2 hidden input name attrs in _tab_daily_living
- L3 sys.exit replaces os._exit so WAL frames flush
- L4 double preparedness import consolidated
- L6 PID recycling — psutil exe basename match in is_running()

XSS hardening (#8 partial):
- esc() helper seeded in all 9 Phase 17-20 partials
- _tab_medical_phase2, _tab_agriculture, _tab_hunting_foraging fully escaped
  (5 render loops + shared badge helpers)
- _tab_group_ops statusBadge + _tab_security_opsec classification/category
  badges wrapped (high call-site fan-out)

Stats: 32 files changed. Remaining backlog (H4 auth, M5 FTS5, M6 connection
pool, full M1/M2/H2 rollout) staged for v7.28.0+ per ROADMAP-v8.md.

Author: FOSSFOREVER

CHANGELOG.md

@@ -2,6 +2,27 @@
 
 All notable changes to project-nomad-desktop will be documented in this file.
 
+## [v7.27.0] — Hardening & Polish (Audit Backlog)
+- Fixed: Disk-space pre-check before yt-dlp downloads (media.py) — rejects when approx size + 500 MB margin exceeds free space on the video dir volume
+- Fixed: Streaming CSV import for contacts (interoperability.py) — new `_iter_upload_lines()` decoder + batched 500-row commits avoid loading multi-hundred-MB uploads fully into memory
+- Fixed: Duty roster cleanup on pod member removal (group_ops.py) — cancels scheduled/active shifts for the removed person in the same pod instead of leaving orphaned roster entries
+- Fixed: XSS — user-sourced strings rendered via innerHTML in `_tab_medical_phase2.html` and `_tab_agriculture.html` are now escaped through a local `esc()` helper that prefers the global `window.escapeHtml`
+- Fixed: Ollama streaming resilience (ai.py) — corrupt/partial JSON chunks from a crashing Ollama backend are now skipped with a debug log instead of forwarded to the client reader
+- Fixed: Config crashes on invalid env vars (config.py M7) — new `_env_int()` helper falls back to defaults with a warning instead of raising ValueError at import time
+- Fixed: Double preparedness import (app.py L4) — consolidated to a single import at the `start_alert_engine` site; blueprint is reused at registration
+- Fixed: `os._exit(0)` → `sys.exit(0)` on shutdown (nomad.py L3) — allows interpreter cleanup so in-flight DB commits actually land
+- Fixed: Missing `name` attrs on 5 hidden inputs in `_tab_daily_living.html` (L2) — satisfies the `test_partial_controls_have_names` contract
+- Added: `@validate_json` schemas applied to all 8 mutating financial endpoints (cash/metals/barter/documents × create/update) per audit H2. Schemas enforce types, max lengths (200-2000 chars), and numeric bounds (≤1B for monetary fields). Financial is the most sensitive blueprint per the audit and gets first coverage.
+- Fixed: `access_logs` table renamed to `platform_access_log` (audit M4) — disambiguates from `access_log` used by physical-security blueprint. New `_migrate_access_logs()` runs on every startup: idempotent, copies any existing rows into the new table via `INSERT OR IGNORE`, then drops the old. Index names also updated. SQL references in `platform_security.py` rewritten.
+- Fixed: Mutating rate limit actually enforced (audit H3) — replaced empty `pass` body with a per-remote-IP sliding-window counter (60s / N from `Config.RATELIMIT_MUTATING`). Localhost exempt. Returns 429 + `retry_after` on overflow.
+- Fixed: Path traversal on Windows in NukeMap/VIPTrack static-file routes (audit H5) — replaced `normcase` + prefix matching with `os.path.commonpath([full, base]) == base`, which is normalization-safe across mixed-case/mixed-separator paths.
+- Added: Shared `get_pagination()` helper in `web/blueprints/__init__.py` (default 100, max 1000) and applied `LIMIT ? OFFSET ?` to primary list endpoints in 7 blueprints — `financial` (cash/metals/barter/documents), `daily_living` (schedules/clothing/sanitation×2/morale/sleep/performance), `training_knowledge` (skill_trees/courses/drill_templates/knowledge_packages), `hunting_foraging` (trade_skills/preservation_methods/preservation_batches/hunting_zones), `disaster_modules` (energy_systems/building_materials), `movement_ops` (alt_vehicles/route_hazards/route_recon), `evac_drills` (drill_runs). Addresses audit M1 — blueprints were returning unbounded result sets that caused memory spikes and UI freezes on constrained hardware.
+- Added: `log_activity()` audit trail to `contacts` (create/update/delete) and `vehicles` (create/update/delete) — was blind spot per audit M2. Weather module deferred (most mutating endpoints are internal alert-rule triggers, not user data).
+- Fixed: PID recycling in service manager (services/manager.py L6) — `is_running()` now verifies the stored PID's process executable basename matches the service's recorded `exe_path` via psutil; `_pid_alive` alone could match a recycled PID that the OS had reassigned to an unrelated process after a crash
+- Added: `esc()` helper (XSS guard) in 7 remaining Phase 17-20 partials — `_tab_hunting_foraging`, `_tab_daily_living`, `_tab_disaster_modules`, `_tab_specialized_modules`, `_tab_group_ops`, `_tab_training_knowledge`, `_tab_security_opsec`. Foundation is in place; `_tab_group_ops` statusBadge and `_tab_security_opsec` classificationBadge/categoryBadge are already wrapped. Remaining per-row field escaping will land incrementally.
+- Fixed: XSS in `_tab_hunting_foraging.html` — 5 primary render functions (game, zones, fishing, foraging, edibles, traps) plus shared `gameTypeBadge`/`statusBadge`/`confClass` helpers now route all user-sourced strings (species, plant names, locations, scientific names, toxicity warnings, bait, notes) through `esc()`. This is the worst-offender Phase 17-20 partial per the audit (56 endpoints, 0 tests).
+- Stats: Addresses 9 backlog items (#8 partial, #10, #11, #12, #13, L2, L3, L4, M7) from the v7.27.0 hardening punch list in ROADMAP-v8.md
+
 ## [v7.26.0] — Phase 20: Specialized Modules & Community
 - Added: Supply caches with GPS and concealment tracking
 - Added: Pets & companion animals with food supply projections

README.md

@@ -1,7 +1,7 @@
 <div align="center">
 <img src="nomad-mark.png" width="140" height="140"/>
 
-# NOMAD Field Desk v7.26.0
+# NOMAD Field Desk v7.27.0
 
 ### Your Personal Intelligence & Preparedness Command Center
 

ROADMAP-v8.md

@@ -797,45 +797,38 @@ After Tier 1, shift to user-demand-driven priorities within Tier 2-3.
 - **Fix:** Rename the Phase 18 table to `iot_sensor_readings` and update hardware_sensors.py references. The legacy `sensor_readings` table serves power.py and should remain unchanged.
 - **Affected:** hardware_sensors.py (lines 153, 194, 216, 245, 269), db.py line 4641
 
-**H2. Input Validation Coverage — 9 of 1,628 Routes**
-- A `@validate_json` decorator exists in web/validation.py but is only applied to 9 routes. The remaining 1,600+ routes accept unvalidated JSON payloads. While parameterized SQL prevents injection, missing validation means:
-  - Oversized payloads can bloat the database (no max_length enforcement)
-  - Type mismatches cause 500 errors instead of 400s
-  - Required fields silently default to empty strings
-- **Fix:** Prioritize validation on mutating (POST/PUT) routes in inventory, medical, contacts, and financial blueprints first — these handle the most sensitive data. A phased rollout across all blueprints is realistic at ~5 blueprints per session.
+**H2. Input Validation Coverage — 9 of 1,628 Routes** — **Partial (v7.27.0)**
+- Financial blueprint fully covered: 8 mutating routes (cash/metals/barter/documents × create/update) now wrapped with reusable `_CASH_SCHEMA`/`_METALS_SCHEMA`/`_BARTER_SCHEMA`/`_DOCUMENTS_SCHEMA`.
+- Remaining: medical, medical_phase2, contacts (partial — `name` validated on create), inventory, vehicles, group_ops, security_opsec, agriculture, all Phase 17-20 blueprints. ~30 more routes high-priority, ~1500 low-priority.
 
-**H3. Mutating Rate Limit Not Enforced (web/app.py:144-149)**
-- `Config.RATELIMIT_MUTATING` (60/minute) is configured but the enforcement function body is `pass`. POST/PUT/DELETE requests are not actually rate-limited despite flask-limiter being active for GET requests.
-- **Fix:** Apply `limiter.limit(Config.RATELIMIT_MUTATING)` to the before_request hook for non-safe HTTP methods, or decorate mutating routes explicitly.
+**H3. Mutating Rate Limit Not Enforced (web/app.py:144-149)** — **Fixed in v7.27.0**
+- Replaced the empty `pass` body with a per-IP sliding-window counter. Localhost exempt. 429 returned on overflow.
 
 **H4. No Auth Middleware Wired to Routes**
 - Phase 19 built `app_users`, `app_sessions`, and role-based access (admin/user/viewer/guest) in platform_security.py, but zero routes across the 58 other blueprints use `@login_required` or `@auth_required` decorators. The auth system exists but is not enforced anywhere.
 - **Fix:** Create a `require_auth` decorator in web/utils.py. Apply it globally via a before_request hook that checks session tokens, with an allowlist for public routes (health check, login, SSE).
 
-**H5. Path Traversal Checks Fragile on Windows (web/app.py:1114, 1153)**
-- NukeMap and VIPTrack static file routes use `os.path.normcase()` + string prefix matching for path containment. On Windows, mixed-case and mixed-separator paths can bypass this check.
-- **Fix:** Replace with `os.path.commonpath([full_path, base_dir]) == os.path.normcase(base_dir)` which is normalization-safe on all platforms.
+**H5. Path Traversal Checks Fragile on Windows (web/app.py:1114, 1153)** — **Fixed in v7.27.0**
+- Both NukeMap and VIPTrack routes now use `os.path.commonpath([full, base]) == base` with a `ValueError` catch for cross-drive edge cases.
 
 ---
 
 ### Medium Priority
 
-**M1. 19 Blueprints Return Unbounded Query Results**
-- These blueprints have no LIMIT clause on their primary list endpoints: agriculture, consumption, daily_living, disaster_modules, emergency, evac_drills, family, financial, group_ops, hunting_foraging, land_assessment, meal_planning, movement_ops, readiness_goals, training_knowledge, and 4 others.
-- A table with thousands of rows will return the entire result set in a single JSON response. On constrained hardware (Raspberry Pi), this causes memory spikes and UI freezes.
-- **Fix:** Add server-side pagination (LIMIT/OFFSET with sensible defaults of 50-100 rows) to all list endpoints. The pattern already exists in inventory.py, contacts.py, and notes.py — replicate it.
+**M1. 19 Blueprints Return Unbounded Query Results** — **Partial (v7.27.0)**
+- 7 of 19 blueprints paginated via a shared `get_pagination()` helper in `web/blueprints/__init__.py`: `financial`, `daily_living`, `training_knowledge`, `hunting_foraging`, `disaster_modules`, `movement_ops`, `evac_drills`.
+- Remaining: `agriculture`, `consumption`, `emergency`, `family`, `group_ops`, `land_assessment`, `meal_planning`, `readiness_goals`, `specialized_modules`, `hardware_sensors`, `callshield`, `alerts`. Pattern is trivial to replicate once the helper is imported.
 
-**M2. 11 Blueprints Have No Activity Logging**
-- These blueprints never call `log_activity()`: brief, checklists, contacts, kit_builder, kiwix, print_routes, supplies, timeline, vehicles, weather, and __init__. This creates blind spots in the activity log — a user modifying contacts, vehicles, or weather data leaves no audit trail.
-- **Fix:** Add `log_activity()` calls to create/update/delete operations in each affected blueprint.
+**M2. 11 Blueprints Have No Activity Logging** — **Partial (v7.27.0)**
+- `contacts` and `vehicles` now call `log_activity()` on create/update/delete.
+- Remaining: `brief`, `checklists`, `kit_builder`, `kiwix`, `print_routes`, `supplies`, `timeline`, `weather`.
 
 **M3. SSE Client Limit Race Condition (web/app.py:1419-1423)**
 - The check `len(_sse_clients) >= MAX_SSE_CLIENTS` and the subsequent queue creation + registration happen outside a single lock acquisition. A concurrent thread can exceed the limit between the check and the registration.
 - **Fix:** Hold `_sse_lock` from the limit check through queue creation and registration in one atomic block.
 
-**M4. `access_log` vs `access_logs` Table Name Collision (db.py:862, 4827)**
-- Two tables with near-identical names serve different purposes: `access_log` (physical entry/exit, used by security.py) and `access_logs` (API access audit, used by platform_security.py). This is confusing and error-prone.
-- **Fix:** Rename `access_logs` to `api_access_log` or `platform_access_log` to disambiguate. Update platform_security.py SQL references.
+**M4. `access_log` vs `access_logs` Table Name Collision (db.py:862, 4827)** — **Fixed in v7.27.0**
+- Renamed `access_logs` → `platform_access_log` with idempotent migration (`_migrate_access_logs`). Indexes renamed. `platform_security.py` SQL updated. `access_log` (physical-security) untouched.
 
 **M5. No FTS5 Full-Text Search**
 - Phase 19 roadmap deliverables included "FTS5 full-text search" but no FTS5 virtual tables exist in db.py. Keyword search across notes, inventory, contacts, and knowledge base still uses LIKE queries, which are O(n) table scans.
@@ -845,9 +838,9 @@ After Tier 1, shift to user-demand-driven priorities within Tier 2-3.
 - Phase 19 deliverables included "connection pooling" but each request creates a new SQLite connection via `db_session()`. SQLite's file-based locking makes this acceptable at low concurrency, but under LAN multi-user access (Phase 19's multi-user auth), contention will cause "database is locked" errors.
 - **Fix:** Implement a thread-local connection pool with a configurable max size (default 5). Reuse connections within the same thread/request lifecycle.
 
-**M7. Config Crashes on Invalid Environment Variables (config.py:42-78)**
-- All `int(os.environ.get(...))` calls will raise `ValueError` if the env var is set to a non-numeric string. The app crashes at import time before any error handling can catch it.
-- **Fix:** Wrap each conversion in a try-except with a fallback to the default value, or use a validated config loader.
+**M7. Config Crashes on Invalid Environment Variables (config.py:42-78)** — **Fixed in v7.27.0**
+- All `int(os.environ.get(...))` calls would raise `ValueError` if the env var was set to a non-numeric string.
+- **Fix applied:** `_env_int()` helper now wraps all integer env reads, falling back to the default value and logging a warning instead of crashing at import time.
 
 **M8. DB Migration System Underpowered — 3 Files for 264 Tables**
 - The migration system (`db_migrations/`) has only 3 migration files despite 264 tables and 27 table-creation functions. Most schema changes are handled by `_apply_column_migrations()` which uses ALTER TABLE ADD COLUMN with try-except (silently ignoring if column exists). This works but:
@@ -884,9 +877,9 @@ After Tier 1, shift to user-demand-driven priorities within Tier 2-3.
 - `web/translations.py` (444 lines) exists but contains no `translate()` or `_t()` functions. The i18n system is scaffolded but not wired to any UI strings.
 - **Note:** This is architectural debt, not a bug. Wire it only when localization becomes a user requirement.
 
-**L6. PID Recycling in Service Manager (services/manager.py:277-295)**
-- `is_running()` checks if a PID is alive but does not verify the process name. If a service crashes and the OS recycles the PID to a different process, the health monitor will incorrectly report the service as running.
-- **Fix:** Store the process name or creation time alongside the PID and verify both in `is_running()`.
+**L6. PID Recycling in Service Manager (services/manager.py:277-295)** — **Fixed in v7.27.0**
+- `is_running()` checked if a PID was alive but did not verify the process name. After a crash, a recycled PID could cause false-positive "running" status.
+- **Fix applied:** New `_pid_matches_exe()` helper uses psutil to compare the live PID's executable basename to the service's recorded `exe_path`. Conservative fallback: if psutil isn't available or the PID can't be introspected, trust `_pid_alive`'s positive result rather than second-guessing it.
 
 ---
 
@@ -1014,12 +1007,12 @@ The following bugs were identified and fixed during this audit cycle:
 | 5 | WAL checkpoint before backup | Medium | **Done** |
 | 6 | Kolibri install state | Medium | **Done** |
 | 7 | Download queue lock | Medium | **Done** |
-| 8 | XSS: escapeHtml in Phase 17-20 partials | High | Backlog |
+| 8 | XSS: escapeHtml in Phase 17-20 partials | High | **Partial** — `esc()` helper added to all 9 partials; medical_phase2 + agriculture fully escaped; per-row field escaping in the other 7 still incremental (v7.27.0) |
 | 9 | Replace inline api() with safeFetch() | Medium | Backlog |
-| 10 | Disk space pre-check for downloads | High | Backlog |
-| 11 | Ollama streaming error resilience | Medium | Backlog |
-| 12 | CSV import chunked processing | Medium | Backlog |
-| 13 | Duty roster cleanup on member remove | Medium | Backlog |
+| 10 | Disk space pre-check for downloads | High | **Done** (v7.27.0) |
+| 11 | Ollama streaming error resilience | Medium | **Done** (v7.27.0) |
+| 12 | CSV import chunked processing | Medium | **Done** (contacts CSV — v7.27.0) |
+| 13 | Duty roster cleanup on member remove | Medium | **Done** (v7.27.0) |
 | 14 | Accessibility: aria-labels | Low | Backlog |
 | 15 | Add tests for Phase 10-20 blueprints | Low | Backlog |
 

config.py

@@ -32,34 +32,47 @@
 # Application Settings (class-based, env-overridable)
 # ---------------------------------------------------------------------------
 
+def _env_int(name, default):
+    """Read an int from the environment, falling back to `default` on missing
+    or non-numeric values rather than crashing at import time."""
+    raw = os.environ.get(name)
+    if raw is None or raw == '':
+        return default
+    try:
+        return int(raw)
+    except (ValueError, TypeError):
+        log.warning('Invalid int for %s=%r — using default %d', name, raw, default)
+        return default
+
+
 class Config:
     """Central configuration with environment variable overrides."""
 
     # --- App Identity ---
-    VERSION = os.environ.get('NOMAD_VERSION', '7.26.0')
+    VERSION = os.environ.get('NOMAD_VERSION', '7.27.0')
 
     # --- Upload / Content Limits ---
-    MAX_CONTENT_LENGTH = int(os.environ.get('NOMAD_MAX_CONTENT_LENGTH', 100 * 1024 * 1024))  # 100 MB
+    MAX_CONTENT_LENGTH = _env_int('NOMAD_MAX_CONTENT_LENGTH', 100 * 1024 * 1024)  # 100 MB
 
     # --- Knowledge Base / RAG ---
     EMBED_MODEL = os.environ.get('NOMAD_EMBED_MODEL', 'nomic-embed-text:v1.5')
-    CHUNK_SIZE = int(os.environ.get('NOMAD_CHUNK_SIZE', 500))
-    CHUNK_OVERLAP = int(os.environ.get('NOMAD_CHUNK_OVERLAP', 50))
+    CHUNK_SIZE = _env_int('NOMAD_CHUNK_SIZE', 500)
+    CHUNK_OVERLAP = _env_int('NOMAD_CHUNK_OVERLAP', 50)
 
     # --- SSE ---
-    MAX_SSE_CLIENTS = int(os.environ.get('NOMAD_MAX_SSE_CLIENTS', 20))
+    MAX_SSE_CLIENTS = _env_int('NOMAD_MAX_SSE_CLIENTS', 20)
 
     # --- Service Ports ---
-    APP_PORT = int(os.environ.get('NOMAD_PORT', 8080))
+    APP_PORT = _env_int('NOMAD_PORT', 8080)
     APP_HOST = os.environ.get('NOMAD_HOST', '127.0.0.1')
-    OLLAMA_PORT = int(os.environ.get('NOMAD_OLLAMA_PORT', 11434))
-    KIWIX_PORT = int(os.environ.get('NOMAD_KIWIX_PORT', 8888))
-    CYBERCHEF_PORT = int(os.environ.get('NOMAD_CYBERCHEF_PORT', 8889))
-    FLATNOTES_PORT = int(os.environ.get('NOMAD_FLATNOTES_PORT', 8890))
-    KOLIBRI_PORT = int(os.environ.get('NOMAD_KOLIBRI_PORT', 8300))
-    QDRANT_PORT = int(os.environ.get('NOMAD_QDRANT_PORT', 6333))
-    STIRLING_PORT = int(os.environ.get('NOMAD_STIRLING_PORT', 8443))
-    DISCOVERY_PORT = int(os.environ.get('NOMAD_DISCOVERY_PORT', 18080))
+    OLLAMA_PORT = _env_int('NOMAD_OLLAMA_PORT', 11434)
+    KIWIX_PORT = _env_int('NOMAD_KIWIX_PORT', 8888)
+    CYBERCHEF_PORT = _env_int('NOMAD_CYBERCHEF_PORT', 8889)
+    FLATNOTES_PORT = _env_int('NOMAD_FLATNOTES_PORT', 8890)
+    KOLIBRI_PORT = _env_int('NOMAD_KOLIBRI_PORT', 8300)
+    QDRANT_PORT = _env_int('NOMAD_QDRANT_PORT', 6333)
+    STIRLING_PORT = _env_int('NOMAD_STIRLING_PORT', 8443)
+    DISCOVERY_PORT = _env_int('NOMAD_DISCOVERY_PORT', 18080)
 
     # --- Map Extensions ---
     ALLOWED_MAP_EXTENSIONS = set(
@@ -74,8 +87,8 @@ class Config:
     RATELIMIT_MUTATING = os.environ.get('NOMAD_RATELIMIT_MUTATING', '60/minute')
 
     # --- Misc Magic Numbers ---
-    CPU_MONITOR_INTERVAL = int(os.environ.get('NOMAD_CPU_MONITOR_INTERVAL', 2))
-    OCR_PIPELINE_INTERVAL = int(os.environ.get('NOMAD_OCR_PIPELINE_INTERVAL', 60))
+    CPU_MONITOR_INTERVAL = _env_int('NOMAD_CPU_MONITOR_INTERVAL', 2)
+    OCR_PIPELINE_INTERVAL = _env_int('NOMAD_OCR_PIPELINE_INTERVAL', 60)
 
     @classmethod
     def secret_key(cls):