Update version to 0.3.0, enhance PII handling instructions, and add PII scan script

Author: thebentern

.github/copilot-instructions.md

@@ -102,3 +102,63 @@ Kotlin `TakPacketV2Data` uses camelCase field names. Wire-generated proto types
 - Don't change the `Team.Unspecifed_Color` typo — it's the canonical Wire-generated name
 - Don't retrain dictionaries without coordinating a version bump — new dicts break wire compatibility with old receivers
 - Don't use `toInt()` for `(longitude * 1e7)` comparisons — IEEE 754 rounding requires `roundToInt()`
+
+## PII and test-fixture sanitization (CRITICAL)
+
+**Never commit unredacted real-world ATAK captures.** CoT XML carries position
+data (lat/lon to sub-meter precision), Android device IDs that act as
+permanent device fingerprints, private LAN IPs that reveal home network
+topology, and personal callsigns. Anything captured from an operator's actual
+deployment must be sanitized BEFORE landing in `testdata/`, the SDK README,
+or the proto submodule — once it's in a git commit pushed to a public mirror,
+no amount of force-pushing scrubs it from caches, forks, or pre-existing
+clones reliably.
+
+### Required replacements when sanitizing a real capture
+
+| What | Pattern | Replace with |
+|---|---|---|
+| GPS coordinates | any real lat/lon | Public landmark coords (e.g. `38.8895, -77.0353` Washington Monument; `38.8814, -77.0502` Lincoln Memorial). For routes/polygons, pick distinct landmark pairs so the path topology survives. |
+| Stationary / not-a-place markers | — | `lat="0.0" lon="0.0"` (matches ATAK's "null island" convention for non-positional events like `m-t-t` and `y-`) |
+| Android device IDs (`ANDROID-[16 hex chars]`) | real device fingerprint | Sequential placeholders: `ANDROID-0000000000000001`, `…02`, etc. (Test fakes are already in this format — extend the numbering.) |
+| Private LAN IPs (`192.168.x.x`, `10.x.x.x`, `172.16-31.x.x`) | real network IP | RFC 5737 documentation range: `192.0.2.1`, `198.51.100.1`, `203.0.113.1` (these ranges are reserved for examples and never route on the public internet). |
+| MAC addresses | real radio MAC | Random or `00:00:00:00:00:0X` placeholder |
+| Real callsigns | personally-identifying handles | Generic operator handles: `ALPHA-1`, `BRAVO-2`, `ASPEN`, `ETHEL`, `CHARLIE` are fine — they're well-worn placeholders. Avoid the actual operator's first name, military rank+last-name, or amateur radio call. |
+| UUIDs | (no action) | Random UUIDs are high-entropy and not by themselves identifying. Keep them. |
+| Voice profile IDs, room IDs, chatgrp UIDs | (no action unless they look like a real account ID) | Usually safe — they're UUIDs. |
+
+### Workflow when adding a new fixture from a real capture
+
+1. **Diff-redact, don't just rename.** Pull the source XML into `/tmp/` and edit there. Apply every replacement above. Save back into `testdata/cot_xml/`.
+2. **Scan before staging.** Run this against the new file:
+   ```bash
+   grep -nE '\b\d{1,3}\.\d{5,}\b|ANDROID-[0-9a-f]{12,}|\b(192\.168|10\.|172\.(1[6-9]|2[0-9]|3[01]))\.[0-9]+\.[0-9]+\b' testdata/cot_xml/<your-new-fixture>.xml
+   ```
+   Hits on the coord regex with 5+ decimal places, on a non-sequential ANDROID hex ID, or on an RFC 1918 IP, mean it isn't sanitized yet. The sequential `ANDROID-0+\d+` pattern is fine — that's the test-fake convention.
+3. **Add a redaction note** in the commit message: "redacted: coords → DC landmarks; android ids → 000…0NN; LAN IP → 192.0.2.1".
+4. **Regenerate goldens twice.** `gradle jvmTest --tests CompressionTest.generate compression report --rerun-tasks` — the first pass mints `.pb`/`.bin`; rerun to confirm steady state.
+5. **Never assume binary blobs auto-sanitize.** Filter-repo's `--replace-text` skips files containing null bytes, so the `.pb` protobuf intermediates and `.bin` golden files retain whatever bytes were generated. After scrubbing the source XML, ALWAYS regenerate the derived binaries.
+
+### Pre-push self-check
+
+Before `git push`, especially when pushing fixture changes, run from repo root:
+
+```bash
+# Quick PII sweep
+grep -rEon '\b34\.[0-9]{1,2}\.[0-9]{5,}|\b\d{1,3}\.\d{5,}\b|ANDROID-[0-9a-f]{12,}|\b(192\.168|10\.|172\.(1[6-9]|2[0-9]|3[01]))\.[0-9]+\.[0-9]+\b' \
+  --include='*.xml' --include='*.md' --include='*.swift' --include='*.kt' \
+  --include='*.cs' --include='*.py' --include='*.ts' \
+  | grep -v 'ANDROID-0\{12\}'
+```
+
+Empty output = clean. Any hit gets eyeballed before push.
+
+### If real PII slips through
+
+If you discover real PII in a commit that has already been pushed:
+1. **Stop pushing** anything that builds on the leak.
+2. Use `git filter-repo --replace-text` for text-format leaks (XML, README, code).
+3. Follow up with `git filter-repo --blob-callback` for binary leaks (`.pb`, `.bin`) — filter-repo's `--replace-text` SKIPS files with null bytes, so binaries need an explicit callback that swaps the bad blob SHA for a clean re-baselined version.
+4. Force-push every branch and tag (`git push --force origin master --tags`).
+5. Open a GitHub Support ticket to purge the orphaned commits from cache — they remain accessible by SHA URL for 30+ days otherwise.
+6. Reach out to fork owners — your force-push doesn't propagate.

CLAUDE.md

@@ -158,3 +158,38 @@ cd kotlin && gradle jvmTest
 
 - **Meshtastic-Android** (`core/takserver`): depends on `takpacket-sdk-jvm` via JitPack, proto submodule at `core/proto/src/main/proto`
 - **Meshtastic-Apple**: depends on `MeshtasticTAK` Swift package via remote SPM URL, proto submodule at `protobufs/`, regenerated `atak.pb.swift` at `MeshtasticProtobufs/Sources/meshtastic/`
+
+## PII / sensitive-data handling — read before adding any fixture
+
+The repo has had real-world ATAK captures land in test fixtures by accident
+before. The compressed `.bin` and `.pb` intermediates retain the leaked
+identifiers even after the source XML is fixed, because filter-repo's text
+replacement skips files containing null bytes. Treat this as a recurring
+hazard.
+
+**Patterns that must never appear in committed fixtures or docs:**
+
+| Pattern | What it leaks | Required substitute |
+|---|---|---|
+| High-precision lat/lon (5+ decimals) that's not a public landmark | Operator's actual location, often a home | DC-area public landmarks: `38.8895,-77.0353` (Washington Monument), `38.8814,-77.0502` (Lincoln Memorial), or `0.0,0.0` for non-positional events like `m-t-t` / `y-` |
+| `ANDROID-[16 hex chars]` where the hex isn't all-zeros | A specific Android device's ANDROID_ID — never changes, can fingerprint the device for years | Sequential placeholder: `ANDROID-0000000000000001`, `…02`, etc. |
+| RFC 1918 private IPs (`192.168.x.x`, `10.x.x.x`, `172.16-31.x.x`) | Home/office network topology | RFC 5737 docs range: `192.0.2.1`, `198.51.100.1`, `203.0.113.1` |
+| MAC addresses | Hardware fingerprint | `00:00:00:00:00:0X` placeholder |
+| Real operator callsigns (first names, military rank+name, ham calls) | Identity | Generic operator handles: `ALPHA-1`, `BRAVO-2`, `ASPEN`, `ETHEL` |
+
+**When you're handed a real CoT Explorer capture / pg_dump excerpt to add as a fixture:**
+
+1. Don't drop it into `testdata/` raw. Edit a redacted copy in `/tmp/` first.
+2. Apply every substitution above. Random UUIDs (message IDs, room IDs, chatgrp UIDs) can stay — they're high-entropy and not identifying on their own.
+3. Sanity-grep the redacted file before staging:
+   ```bash
+   grep -nE '\b\d{1,3}\.\d{5,}\b|ANDROID-[0-9a-f]{12,}|\b(192\.168|10\.|172\.(1[6-9]|2[0-9]|3[01]))\.[0-9]+\.[0-9]+\b' /tmp/<file>.xml
+   ```
+   Hits on coords with 5+ decimal places, non-sequential ANDROID hex, or RFC 1918 IPs mean it's still dirty. (Sequential `ANDROID-0+\d+` is allowed — that's the test-fake convention.)
+4. Regenerate goldens via `gradle jvmTest --tests "CompressionTest.generate compression report" --rerun-tasks` so the derived `.pb` / `.bin` artifacts pick up the clean strings. The Kotlin test writes both into `testdata/golden/` and `testdata/protobuf/`.
+
+**If a leak ships to master:**
+
+The recovery is `git filter-repo` in two passes — text substitution for the XML/code/docs, then a `--blob-callback` for the binary `.pb` and `.bin` files (filter-repo skips text replacement on files with null bytes, so binaries need an explicit callback that swaps the bad blob SHAs for clean re-baselined ones). Force-push every branch and tag. Open a GitHub Support ticket to purge orphaned commits — they remain reachable by SHA URL until cache expires (~30 days). Forks aren't touched by the force-push.
+
+The detailed playbook is in `.github/copilot-instructions.md` under "PII and test-fixture sanitization."

VERSION

@@ -1 +1 @@
-0.2.6
+0.3.0

kotlin/gradle.properties

@@ -5,4 +5,4 @@ org.gradle.parallel=true
 # Publishing coordinates (picked up by com.vanniktech.maven.publish)
 GROUP=org.meshtastic
 POM_ARTIFACT_ID=takpacket-sdk
-VERSION_NAME=0.2.6
+VERSION_NAME=0.3.0

scripts/pii-scan.sh

@@ -0,0 +1,94 @@
+#!/usr/bin/env bash
+# pii-scan.sh — flag candidate PII in staged or working-tree files.
+#
+# Two modes:
+#   ./scripts/pii-scan.sh                       # scan entire working tree
+#   ./scripts/pii-scan.sh --staged              # scan only files staged for commit
+#
+# Install as a pre-commit hook:
+#   ln -s ../../scripts/pii-scan.sh .git/hooks/pre-commit
+#
+# Exit code: 0 = clean, 1 = PII candidate found (review before commit).
+#
+# What it flags:
+#   - GPS coords with 5+ decimal places that aren't on the known-synthetic whitelist
+#   - ANDROID-[16+ hex chars] device IDs (allows ANDROID-0+\d+ test placeholders)
+#   - RFC 1918 private network IPs (allows 192.0.2.x / 198.51.100.x / 203.0.113.x docs ranges)
+#
+# What it does NOT flag (intentional false negatives):
+#   - High-entropy UUIDs (high-entropy, not identifying alone)
+#   - Real names embedded in callsign strings (callsigns vary; impossible to regex reliably)
+#   - Coordinates with <5 decimals (too imprecise to be identifying)
+
+set -euo pipefail
+
+STAGED_ONLY=0
+if [[ "${1:-}" == "--staged" ]]; then
+    STAGED_ONLY=1
+fi
+
+# Pattern: matches coords with 5+ decimal places, ANDROID device IDs,
+# and RFC 1918 IPs in one pass.
+PII_PATTERN='\b[0-9]{1,3}\.[0-9]{5,}\b|ANDROID-[0-9a-f]{12,}|\b(192\.168|10\.|172\.(1[6-9]|2[0-9]|3[01]))\.[0-9]+\.[0-9]+\b'
+
+# Whitelist (allowlist): things that match PII_PATTERN but are KNOWN safe.
+#   ANDROID-0+\d+              — sequential test placeholders (ANDROID-0000000000000001, ...02, ...)
+#   ANDROID-589520ccfcd20f01   — specific ATAK-CIV upstream sample device ID (public)
+#   192\.0\.2\.|198\.51\.100\.|203\.0\.113\.  — RFC 5737 docs IPs
+#   \b38\.[0-9]+               — DC-area lat (Washington Monument neighborhood, public landmarks); covers high-precision ATAK-CIV upstream samples
+#   \b77\.0[0-9]+              — DC-area lon (matches both -77.0x and 77.0x); public landmarks
+#   \b33\.12[0-9]+             — established synthetic NM coords used in Meshtastic-Android fixtures
+#   \b107\.25[0-9]+            — same NM range
+#   \b12\.0{4,5}\b             — established synthetic Bay-of-Bengal placeholders (low-precision)
+#   \b9[01]\.0{4,5}\b          — same Bay-of-Bengal range
+#
+# Numbers > 200 with 5+ decimals are very unlikely to be lat/lon (lat is
+# -90..90, lon is -180..180); they're almost always ellipse radii, bearings,
+# or other non-coord measurements. Filter those out too.
+ALLOW_PATTERN='ANDROID-0+[0-9]+|ANDROID-589520ccfcd20f01|\b192\.0\.2\.|\b198\.51\.100\.|\b203\.0\.113\.|\b38\.[0-9]+|\b77\.0[0-9]+|\b33\.1[0-9]+|\b107\.2[0-9]+|\b12\.0{4,5}\b|\b9[01]\.0{4,5}\b|\b16\.1[0-9]+|\b10\.00[0-9]+|\b95\.00[0-9]+|\b15\.181230\b|\b33\.30720360300985\b|\b45\.5965567[0-9]+\b|\b[2-9][0-9]{2,}\.[0-9]+\b|\b1[0-9]{3,}\.[0-9]+\b'
+
+# Files to scan: source + docs + fixtures + manifests.
+INCLUDE_GLOBS=(
+    "--include=*.xml" "--include=*.cot" "--include=*.md"
+    "--include=*.kt" "--include=*.kts" "--include=*.swift"
+    "--include=*.cs" "--include=*.py" "--include=*.ts" "--include=*.js"
+    "--include=*.json"
+)
+
+# Always-excluded paths (build artifacts, vendor dirs, the corpus dir in ZTSD).
+EXCLUDE_DIRS=(
+    "--exclude-dir=.git" "--exclude-dir=node_modules" "--exclude-dir=build"
+    "--exclude-dir=.build" "--exclude-dir=bin" "--exclude-dir=obj"
+    "--exclude-dir=venv" "--exclude-dir=.venv" "--exclude-dir=DerivedData"
+    "--exclude-dir=corpus" "--exclude-dir=Pods" "--exclude-dir=site-packages"
+)
+
+if [[ "$STAGED_ONLY" -eq 1 ]]; then
+    # Pre-commit mode: only scan files actually staged for commit.
+    FILES=$(git diff --cached --name-only --diff-filter=AM \
+            | grep -E '\.(xml|cot|md|kt|kts|swift|cs|py|ts|js|json)$' || true)
+    if [[ -z "$FILES" ]]; then
+        echo "pii-scan: no relevant staged files."
+        exit 0
+    fi
+    HITS=$(echo "$FILES" | xargs -I{} grep -EHno "$PII_PATTERN" "{}" 2>/dev/null \
+           | grep -Ev "$ALLOW_PATTERN" || true)
+else
+    HITS=$(grep -rEHno "$PII_PATTERN" "${EXCLUDE_DIRS[@]}" "${INCLUDE_GLOBS[@]}" . 2>/dev/null \
+           | grep -Ev "$ALLOW_PATTERN" || true)
+fi
+
+if [[ -n "$HITS" ]]; then
+    echo "❌ pii-scan found candidate PII (review before commit):" >&2
+    echo "" >&2
+    echo "$HITS" >&2
+    echo "" >&2
+    echo "Sanitization guide: .github/copilot-instructions.md → \"PII and test-fixture sanitization\"" >&2
+    echo "" >&2
+    echo "If a flagged match is actually safe (e.g. a new public landmark coord)," >&2
+    echo "extend ALLOW_PATTERN in scripts/pii-scan.sh." >&2
+    exit 1
+fi
+
+echo "✅ pii-scan clean."
+exit 0