@@ -422,21 +422,40 @@ sudo systemctl restart secure-ai-inference
422422sudo nft list ruleset
423423```
424424
425- ### Panic Switch (Emergency Lockdown )
425+ ### Emergency Panic (securectl )
426426
427- If you suspect a compromise or data leak in progress:
427+ If you suspect a compromise or data leak in progress, ` securectl ` provides three severity levels :
428428
429429``` bash
430- sudo systemctl start secure-ai-panic
430+ # Level 1 — Lock (reversible)
431+ # Stops all AI services, kills workers, locks vault, invalidates sessions
432+ sudo securectl panic 1
433+
434+ # Level 2 — Wipe Keys (requires passphrase)
435+ # Level 1 + shreds LUKS header backup, cosign keys, TPM2 keys, MOK key
436+ sudo securectl panic 2 --confirm " your-passphrase"
437+
438+ # Level 3 — Full Wipe (requires passphrase, DATA UNRECOVERABLE)
439+ # Level 2 + re-encrypts vault with random key, clears memory, deletes all logs/registry/auth
440+ sudo securectl panic 3 --confirm " your-passphrase"
431441```
432442
433- This immediately:
434- 1 . Replaces all firewall rules with a total-deny policy (loopback only)
435- 2 . Flushes all network routes
436- 3 . Stops the airlock and UI services
437- 4 . Writes an audit record
443+ You can also trigger Level 1 from the Web UI via ** Settings > Emergency Lock** , or Level 2/3 via the API:
444+
445+ ``` bash
446+ curl -X POST http://127.0.0.1:8480/api/emergency/panic \
447+ -H ' Content-Type: application/json' \
448+ -d ' {"level": 1}'
449+ ```
450+
451+ A 5-second countdown with Ctrl+C cancel runs before any action (skip with ` --no-countdown ` ). All panic events are audit-logged before execution.
452+
453+ Check panic state:
454+ ``` bash
455+ sudo securectl status
456+ ```
438457
439- To recover from panic mode, reboot the system .
458+ To recover from Level 1, unlock the vault and restart services. Levels 2 and 3 require re-setup .
440459
441460### Airlock (Optional Online Access)
442461
@@ -462,6 +481,50 @@ To disable the airlock again:
462481sudo systemctl stop secure-ai-airlock
463482```
464483
484+ ### System Updates
485+
486+ Updates are cosign-verified and use a staged workflow — the system never applies an update without your confirmation.
487+
488+ ``` bash
489+ # Check for available updates
490+ sudo /usr/libexec/secure-ai/update-verify.sh check
491+
492+ # Stage (download without applying)
493+ sudo /usr/libexec/secure-ai/update-verify.sh stage
494+
495+ # Apply and reboot
496+ sudo /usr/libexec/secure-ai/update-verify.sh apply
497+
498+ # Roll back to previous deployment
499+ sudo /usr/libexec/secure-ai/update-verify.sh rollback
500+ ```
501+
502+ You can also manage updates from the Web UI via ** Settings > Updates** .
503+
504+ ** Auto-rollback:** If the system fails to boot after an update, the greenboot health check detects the failure and automatically rolls back via ` rpm-ostree rollback ` . After 2 failed rollback attempts, the system halts for manual intervention.
505+
506+ The update check timer runs every 6 hours and notifies the UI when updates are available.
507+
508+ ### Vault Management
509+
510+ The vault auto-locks after 30 minutes of inactivity. You can manage it from the Web UI or API:
511+
512+ ``` bash
513+ # Check vault status
514+ curl http://127.0.0.1:8480/api/vault/status
515+
516+ # Lock vault manually
517+ curl -X POST http://127.0.0.1:8480/api/vault/lock
518+
519+ # Unlock vault
520+ curl -X POST http://127.0.0.1:8480/api/vault/unlock \
521+ -H ' Content-Type: application/json' \
522+ -d ' {"passphrase": "your-passphrase"}'
523+
524+ # Keep vault alive during long tasks
525+ curl -X POST http://127.0.0.1:8480/api/vault/keepalive
526+ ```
527+
465528### Web Search (Tor-Routed, Optional)
466529
467530Web search is ** disabled by default** . When enabled, the LLM can augment its answers with web search results — all routed through Tor for anonymity.
@@ -526,18 +589,25 @@ Every model — whether downloaded from the catalog or imported by the user —
526589
527590| Layer | Mechanism |
528591| -------| -----------|
529- | ** Boot** | Immutable OS image (rpm-ostree), signed updates (cosign) |
592+ | ** Boot** | Immutable OS image (rpm-ostree), cosign-verified updates, greenboot auto-rollback |
593+ | ** Secure Boot** | UEFI Secure Boot with MOK signing, TPM2 vault key sealing (PCR 0,2,4,7) |
530594| ** Kernel** | IOMMU forced, ASLR, slab_nomerge, init_on_alloc/free, lockdown=confidentiality |
531- | ** Network ** | nftables default-deny egress, services use PrivateNetwork=yes |
532- | ** Swap ** | Disabled (kernel arg + runtime check) — prevents secrets hitting disk |
595+ | ** Memory ** | vm.swappiness=0, zswap disabled, core dumps discarded, mlock for sensitive buffers, TEE detection (AMD SEV/Intel TDX/TME) |
596+ | ** Network ** | nftables default-deny egress, DNS rate-limited, traffic analysis countermeasures (query padding, timing randomization) |
533597| ** Filesystem** | Encrypted vault (LUKS2/AES-256/Argon2id), restrictive permissions |
534598| ** Models** | 7-stage quarantine: source, format, integrity, provenance, static scan, behavioral test, diffusion scan |
535599| ** Tools** | Default-deny policy, path allowlisting, traversal protection, rate limiting |
536600| ** Egress** | Airlock disabled by default, PII/credential scanning, destination allowlist |
537- | ** Search** | Tor-routed, PII stripped from queries, injection detection on results, audit logged |
538- | ** Services** | Systemd sandboxing: ProtectSystem=strict, PrivateNetwork, syscall filters |
539- | ** GPU Isolation** | Vendor-specific DeviceAllow (NVIDIA ` /dev/nvidia* ` , AMD ` /dev/kfd ` , Intel ` /dev/dri/* ` ), PrivateNetwork on all |
540- | ** Emergency** | Panic switch: instant network kill + route flush + service stop |
601+ | ** Search** | Tor-routed with differential privacy (decoy queries, k-anonymity, batch timing), PII stripped, injection detection |
602+ | ** Audit** | Hash-chained tamper-evident logs with periodic verification |
603+ | ** Auth** | Local passphrase with scrypt hashing, rate-limited login, session management |
604+ | ** Vault** | Auto-lock after 30 min idle, TPM2-sealed keys, manual lock/unlock via UI |
605+ | ** Services** | Systemd sandboxing: ProtectSystem=strict, PrivateNetwork, seccomp-bpf, Landlock, PrivateUsers |
606+ | ** GPU Isolation** | Vendor-specific DeviceAllow, PrivateNetwork on all workers |
607+ | ** Clipboard** | VM clipboard agents disabled, auto-clear every 60s, PrivateUsers on non-UI services |
608+ | ** Tripwire** | Canary files in sensitive dirs, 5-min timer checks, inotify real-time monitoring |
609+ | ** Emergency** | 3-level panic (lock → wipe keys → full wipe) with passphrase gates and audit trail |
610+ | ** Updates** | Cosign-verified rpm-ostree upgrades, staged workflow, greenboot health checks, auto-rollback (max 2 attempts) |
541611
542612### Systemd Sandboxing
543613
@@ -668,8 +738,9 @@ files/
668738 etc/secure-ai/ Policy and config files baked into image
669739 etc/nftables/ Firewall rules (default-deny egress)
670740 etc/sysctl.d/ Kernel hardening parameters
741+ etc/greenboot/ Health check scripts for auto-rollback
671742 usr/lib/systemd/ Systemd service units (sandboxed)
672- usr/libexec/ Helper scripts (firstboot, vault, model select, panic )
743+ usr/libexec/ Helper scripts (firstboot, vault, securectl, canary, update-verify )
673744services/
674745 registry/ Go -- Trusted Registry
675746 tool-firewall/ Go -- Policy engine + tool gateway
@@ -682,7 +753,15 @@ services/
682753tests/
683754 test_pipeline.py Quarantine pipeline tests (48 tests)
684755 test_search.py Search mediator tests (27 tests)
685- test_ui.py Web UI tests (7 tests)
756+ test_ui.py Web UI tests (11 tests)
757+ test_vault_watchdog.py Vault auto-lock tests (18 tests)
758+ test_memory_protection.py Memory hardening tests (37 tests)
759+ test_traffic_analysis.py Traffic analysis protection tests (41 tests)
760+ test_differential_privacy.py Differential privacy tests (37 tests)
761+ test_clipboard_isolation.py Clipboard isolation tests (30 tests)
762+ test_canary_tripwire.py Canary/tripwire tests (49 tests)
763+ test_emergency_wipe.py Emergency wipe tests (65 tests)
764+ test_update_rollback.py Update verification tests (74 tests)
686765scripts/
687766 vm/
688767 build-qcow2.sh QCOW2 image builder (KVM/QEMU/Proxmox)
@@ -699,7 +778,7 @@ cd services/registry && go test -v -race ./...
699778cd services/tool-firewall && go test -v -race ./...
700779cd services/airlock && go test -v -race ./...
701780
702- # Python tests (82 total)
781+ # Python tests (547 total)
703782pip install pytest flask requests pyyaml
704783python -m pytest tests/ -v
705784
@@ -721,7 +800,20 @@ shellcheck files/system/usr/libexec/secure-ai/*.sh files/scripts/*.sh
721800- [x] ** M9 Multi-GPU Support** -- NVIDIA/AMD/Intel/Apple auto-detection, Vulkan fallback
722801- [x] ** M10 Tor-Routed Search** -- SearXNG + Tor, PII stripping, injection detection, audit
723802- [x] ** M11 VM Support** -- OVA/QCOW2 builds, VM detection, GPU passthrough toggle, security warnings
724- - [ ] ** M12 Polish** -- OPA/Rego policy engine, appliance setup wizard, documentation site
803+ - [x] ** M12 Model Integrity Monitoring** -- Periodic hash verification, auto-quarantine on mismatch
804+ - [x] ** M13 Tamper-Evident Audit Logs** -- Hash-chained JSONL logs with periodic chain verification
805+ - [x] ** M14 Local Passphrase Auth** -- Scrypt hashing, rate-limited login, session management
806+ - [x] ** M15 Vault Auto-Lock** -- Idle-based auto-lock watchdog, UI lock/unlock controls
807+ - [x] ** M16 Process Isolation** -- Seccomp-BPF profiles, Landlock filesystem restrictions, systemd hardening
808+ - [x] ** M17 Secure Boot Chain** -- MOK signing, TPM2 vault key sealing, measured boot (PCR 0,2,4,7)
809+ - [x] ** M18 Memory Protection** -- Swap/zswap disabled, core dumps discarded, mlock, TEE detection (AMD SEV/Intel TDX/TME)
810+ - [x] ** M19 Traffic Analysis Protection** -- Query timing randomization, fixed-size padding, Tor circuit rotation, DNS leak detection
811+ - [x] ** M20 Differential Privacy** -- Decoy search queries, query generalization, k-anonymity, batch timing
812+ - [x] ** M21 Clipboard Isolation** -- VM clipboard agent detection/disabling, auto-clear timer, PrivateUsers on services
813+ - [x] ** M22 Canary/Tripwire** -- Canary files with hashed tokens, 5-min timer checks, inotify real-time monitoring, auto-lockdown
814+ - [x] ** M23 Emergency Wipe** -- 3-level securectl panic (lock/wipe keys/full wipe), passphrase gates, audit trail
815+ - [x] ** M24 Update Verification** -- Cosign-verified rpm-ostree upgrades, greenboot health checks, auto-rollback
816+ - [ ] ** M25 Polish** -- OPA/Rego policy engine, appliance setup wizard, documentation site
725817
726818## Troubleshooting
727819
@@ -823,15 +915,28 @@ journalctl -k | grep secure-ai
823915sudo nft -f /etc/nftables/secure-ai.nft
824916```
825917
826- ### Recovering from Panic Mode
918+ ### Recovering from Emergency Panic
919+
920+ ** Level 1 (Lock):** Vault is locked, services stopped. To recover:
921+ ``` bash
922+ # Check panic state
923+ sudo securectl status
827924
828- Panic mode blocks all network and stops services. To recover:
925+ # Unlock vault (prompts for passphrase)
926+ # Then restart services
927+ sudo systemctl reboot
928+ ```
829929
930+ ** Level 2 (Keys Wiped):** Signing keys destroyed. Data still recoverable with vault passphrase:
830931``` bash
831- # Simply reboot — the normal firewall rules and services are restored
932+ # Reboot and enter vault passphrase at boot
832933sudo systemctl reboot
934+ # Re-generate signing keys after boot
935+ sudo /usr/libexec/secure-ai/firstboot.sh
833936```
834937
938+ ** Level 3 (Full Wipe):** Data is unrecoverable. System boots to factory-reset state and runs first-boot setup again.
939+
835940## License
836941
837942See [ LICENSE] ( LICENSE ) .
0 commit comments