Crawler hardening pass (#20)

* chore: commit wip changes on feature/wip-20260705

* Crawler hardening pass

- Update AGENTS.md to the shared hawk-eco extension-authoring format

Author: Patel230

.github/workflows/ci.yml

@@ -1,5 +1,5 @@
 # Canonical CI workflow for hawk-eco Go repos.
-# Source of truth: .shared-templates/workflows/go-ci.yml.tmpl
+# Source of truth: https://github.com/GrayCodeAI/hawk/blob/main/.shared-templates/workflows/go-ci.yml.tmpl
 #
 # Two deployment models:
 #

AGENTS.md

@@ -1,109 +1,168 @@
-# AGENTS.md — Inspect
+---
+description: Extending hawk-eco — how to write AGENTS.md files, custom specialists, skills, hooks, MCP servers, and plugins.
+globs: "*.go, *.js, *.md, *.json, *.toml, *.yaml, *.yml"
+alwaysApply: false
+---
 
-Website security auditing and crawling library for Go. Crawls sites concurrently, runs checks and declarative rules, generates findings with severity and CWE references.
+# Extending hawk-eco
 
-## Design Principles
+hawk-eco is an open-source code intelligence platform. This document describes how to extend it with custom tools, skills, hooks, and integrations.
 
-- **Library** — importable Go library + embeddable MCP server (no CLI binary)
-- **No LLM dependency** — pure static analysis on crawled pages
-- **Extensible** — custom checks (Go code) + declarative rules (no code required)
+## 1. Drop a project `AGENTS.md`
 
-## Build & Test
+When hawk-eco starts in a directory, it looks for project-level instructions and injects them into the system prompt. The lookup walks from your current working directory **up to the nearest git root** and reads the first matching file at each level — general rules at the repo root, more specific rules in sub-trees. Files are labeled with their directory in the prompt (e.g. `## Project guidelines (services/api/AGENTS.md)`).
+
+Accepted file names, in priority order at each level:
+
+| Path | Notes |
+| --- | --- |
+| `./AGENTS.md` | The classic spot — committed to your repo, shared with the team. |
+| `./ZERO.md` | Brand-specific alias. Same format, lower priority. |
+| `./.zero/AGENTS.md` | Project-local, hidden, gitignored. Personal notes that stay out of git. |
+
+Matching is **case-insensitive** on the basename, so `AGENTS.md`, `Agents.md`, and `agents.md` resolve to the same file on Windows and macOS. The git-tracked filename in this repo is `AGENTS.md` — keep that on case-sensitive filesystems (Linux, the WSL filesystem, or a CI runner) to match what the loader looks for.
+
+Both files use the same format. YAML frontmatter is optional; the markdown body is loaded as instructions for the agent. hawk-eco reads the file once at session start, so changes take effect on the next launch — not mid-session.
+
+```markdown
+# Project conventions for <your project>
+
+- Build with `make`, not `go build` directly.
+- Tests live next to the source file (`foo_test.go` next to `foo.go`).
+- Run `make lint` before opening a PR.
+- Never edit files under `third_party/` — those are vendored.
+```
+
+Tips:
+
+- Keep each file under ~8 KiB. hawk-eco caps the **total** across all matched files at 32 KiB; everything past the cap is dropped.
+- Re-state rules in the imperative voice: "Run `make lint`", not "you should consider running the linter".
+- Don't put secrets, model IDs, or environment-specific paths in `AGENTS.md`. Use config files for those.
+- In a monorepo, drop a narrower `AGENTS.md` in each sub-tree (e.g. `services/api/AGENTS.md`). hawk-eco picks those up automatically when you launch from inside the sub-tree.
+- A YAML frontmatter block (`---\n...\n---`) at the top is preserved verbatim in the injected prompt but is not parsed for `globs:` or `alwaysApply:` scoping today — keep the body self-contained.
+
+### Personal guidelines, across every project
+
+For preferences that follow *you*, not a specific repo (tone, tooling habits, workflow), drop a `ZERO.md` in your user config directory: `~/.config/hawk-eco/ZERO.md` on Linux/macOS, `%AppData%\Roaming\hawk-eco\ZERO.md` on Windows — the same directory as config files and your personal specialists. Same format and 8 KiB cap as the project files above, and the same case-insensitive basename match.
+
+This file is injected as its own `## User guidelines` section, before the project's `AGENTS.md`/`ZERO.md`, and is labeled as personal preference in the prompt: project guidelines are the later, more specific instruction and take precedence over it when the two conflict.
+
+## 2. Custom specialists
+
+Specialists are hawk-eco's sub-agents. Three scopes, in priority order:
+
+| Scope | Path | Shared? |
+| --- | --- | --- |
+| Built-in | compiled into hawk-eco | yes |
+| User | `~/.config/hawk-eco/specialists/*.md` | no — your machine only |
+| Project | `./.zero/specialists/*.md` | yes — the repo team |
+
+Project overrides user overrides built-in when names collide.
+
+A specialist is a markdown manifest with frontmatter and a system prompt:
+
+```markdown
+---
+description: Reviews API changes for breaking-change risk and missing tests.
+tools: read-only,plan
+---
+
+You review API changes. For every changed hunk in `internal/api/` or any file
+that ends in `_api.go`:
+
+1. Confirm the public signature is backward-compatible, or note the breaking
+   change explicitly with the migration path.
+2. Confirm a corresponding test exists in `internal/api/*_test.go` and that
+   the new behaviour is exercised.
+3. Flag any new exported symbol without a doc comment.
+
+Reply with one JSON object per finding: `{"file", "line", "severity", "message", "fix"}`.
+```
+
+CLI management:
 
 ```bash
-go test ./...                    # Run all tests
-go test -race ./...              # Race detector
-go test -coverprofile=c.out ./... # Coverage
-go vet ./...                     # Static analysis
-gofumpt -w .                     # Format
+hawk-eco specialist list
+hawk-eco specialist show api-reviewer
+hawk-eco specialist create api-reviewer \
+    --project \
+    --description "Reviews API changes" \
+    --tools read-only,plan \
+    --prompt "$(cat api-reviewer.md)"
+hawk-eco specialist edit api-reviewer --project
+hawk-eco specialist delete api-reviewer --project
+hawk-eco specialist path                       # prints the resolved specialists directory
 ```
 
-## Architecture
-
-- `crawler.go` — Concurrent website crawler with depth control
-- `check.go` — Check interface and built-in security checks
-- `rule.go` — Declarative rule engine (YAML-based)
-- `finding.go` — Findings with severity, CWE, and evidence
-- `report.go` — Report generation (JSON, SARIF, HTML)
-
-## Conventions
-
-- Go 1.26+, pure Go, no CGO
-- Table-driven tests
-- Conventional Commits: `feat:`, `fix:`, `docs:`, `refactor:`, `test:`
-- No `Co-authored-by:` trailers (auto-stripped by githook)
-- `gofumpt` formatting enforced in CI
-- CWE references required for all security findings
-
-## Common Pitfalls
-
-- Crawler tests need HTTP test servers — use `httptest.NewServer`
-- Rule YAML must be validated before execution
-- Session cookie matching uses substring, not exact match
-
-## Naming Conventions
-
-- **Types are domain nouns**: `Finding`, `Report`, `Stats`, `Page`, `PageLink`, `Checker`, `RuleCheck`
-- **Option functions use `With` prefix**: `WithChecks()`, `WithDepth()`, `WithConcurrency()`, `WithAllowPrivateIPs()`
-- **Preset options are bare vars**: `Quick`, `Standard`, `Deep`, `SecurityOnly`, `CI` — exported `var Option` values
-- **Severity is a type alias**: `type Severity = types.Severity` from `hawk-core-contracts/types` — shared across hawk-eco
-- **Internal adapters use `Adapter` suffix**: `ruleCheckAdapter`, `customCheckAdapter` — bridge public to internal interfaces
-- **Check names are lowercase strings**: `"security"`, `"links"`, `"forms"`, `"a11y"`, `"performance"` — used in `WithChecks()`
-- **Error handling**: `Scan()` returns `(*Report, error)` — validation errors for empty URL, nil errors for success
-
-## API Patterns
-
-- **Functional options pattern**: same as sight — `Option` interface with `optFunc` adapter, `buildConfig()` merge
-- **One-shot + reusable**: `Scan(ctx, target, opts...)` creates a `Scanner` internally; `NewScanner(opts...)` for reuse
-- **Checker interface for extensibility**: `Name() string` + `Run(ctx, pages) []Finding` — register via `RegisterCheck()`
-- **RuleCheck for declarative rules**: `HeaderMatch`, `HeaderMissing`, `BodyMatch`, `BodyMissing`, `URLMatch` patterns
-- **Global + per-scanner custom checks**: `RegisterCheck()`/`RegisterRule()` for global; pass slices to `Scanner` for scoped
-- **Report.Failed()**: checks if any finding meets `FailOn` severity threshold — same pattern as sight
-- **ReDoS protection**: all user-supplied regex patterns go through `compileWithTimeout()` and `matchWithTimeout()` with 1s/100ms limits
-- **Regex complexity check**: `checkRegexComplexity()` rejects nested quantifiers and deep group nesting before compilation
-
-## Testing Patterns
-
-- **External test package**: `package inspect_test` — tests import `inspect` as a consumer would
-- **httptest.NewServer for all tests**: each test spins up a mock HTTP server with specific HTML/headers/responses
-- **Test patterns by concern**: `TestScan_BasicSite` (links), `TestScan_SecurityHeaders`, `TestScan_FormCSRF`, `TestScan_Accessibility`
-- **Always pass `WithAllowPrivateIPs()`**: tests run against `127.0.0.1` — without this flag, localhost is blocked
-- **Always pass `WithDepth(1)`**: keeps tests fast by limiting crawl depth
-- **Finding assertions**: iterate `report.Findings` and check specific `Check`, `Severity`, `Message` fields
-- **Preset smoke test**: `TestScan_Presets` runs all presets against a simple server — catches config panics
-- **ClearCustomChecks() in tests**: call before registering test-specific checks to avoid global state leaks
-- **Report method tests**: `TestReport_Failed`, `TestReport_MaxSeverity` — test on struct literals, no HTTP needed
-
-## Refactoring Guidelines
-
-- **Safe to refactor**: `checkRegexComplexity()`, `compileWithTimeout()`, `matchWithTimeout()` — internal helpers
-- **Safe to refactor**: `truncateEvidence()`, `intIn()` — pure utility functions
-- **Safe to refactor**: `parseInspectTOML()`, `parseInspectKeyValue()`, `applyFileConfig()` — config parsing internals
-- **Do not touch**: `Checker` interface (`Name()`, `Run()`) — breaking change for all custom check implementations
-- **Do not touch**: `RuleCheck` struct field names — used by consumers to define declarative rules
-- **Do not touch**: `Finding`, `Report`, `Stats` struct field names/tags — JSON serialization contract
-- **Safe to extend**: add new `Option` functions, new presets, new built-in checks in `checks/` package
-- **When adding checks**: create a new file in `checks/`, implement `Checker` interface, register in `init()`
-
-## Key File Locations
-
-| What | Where |
-|---|---|
-| Public API entry point | `inspect.go` (types, `Scan()`, `Finding`, `Report`, `Stats`) |
-| Check interface & adapters | `check.go` (`Checker`, `RuleCheck`, `RegisterCheck()`, `RegisterRule()`, ReDoS protection) |
-| Scanner implementation | `scanner.go` (crawler orchestration, check execution) |
-| Configuration & presets | `options.go` (`config` struct, `With*` functions, presets) |
-| Config file loading | `config.go` (`.inspect.toml` parsing, `LoadConfig()`) |
-| Severity type alias | `severity.go` (re-exports from `hawk-core-contracts/types`) |
-| SARIF output | `sarif.go` |
-| CI output formatting | `ci_output.go` |
-| Built-in checks | `checks/` directory |
-| Internal crawler | `internal/crawler/` |
-| Internal check runner | `internal/check/` |
-| Browser-based crawling | `browser.go`, `browser/` |
-| LLM scanner integration | `llm_scanner.go` |
-| API security checks | `api_security.go` |
-| Dependency checking | `dependency_check.go` |
-| SBOM generation | `sbom.go` |
-| Main test file | `inspect_test.go` (httptest servers, per-concern scenarios) |
-| Linter config | `.golangci.yml` (errcheck, govet, staticcheck, gocritic, bodyclose, noctx) |
+## 3. Skills
+
+Skills are markdown instruction files that extend agent capabilities. They can be:
+- Project-scoped: dropped in `./.zero/skills/` or `./skills/`
+- User-scoped: dropped in `~/.config/hawk-eco/skills/`
+
+A skill manifest:
+
+```markdown
+---
+description: How to review Go code for security issues
+globs: "*.go"
+alwaysApply: true
+---
+
+When reviewing Go code for security:
+
+1. Check for SQL injection patterns
+2. Verify error handling doesn't expose sensitive data
+3. Confirm secrets are not hardcoded
+4. Validate input sanitization
+```
+
+## 4. Hooks
+
+Hooks allow custom commands to run at specific lifecycle points:
+- `beforeReview` — runs before code review starts
+- `afterReview` — runs after code review completes
+- `sessionStart` — runs at session initialization
+- `sessionEnd` — runs at session teardown
+
+```bash
+hawk-eco hook add beforeReview --command "lint-check"
+hawk-eco hook remove beforeReview
+hawk-eco hook list
+```
+
+## 5. MCP integration
+
+MCP (Model Context Protocol) servers can expose tools to hawk-eco:
+
+```bash
+hawk-eco mcp add --name server --url http://localhost:8080
+hawk-eco mcp remove server
+hawk-eco mcp list
+```
+
+## 6. Plugins
+
+Plugins extend hawk-eco with custom tools and capabilities:
+
+```bash
+hawk-eco plugin add --name my-plugin --path ./my-plugin
+hawk-eco plugin remove my-plugin
+hawk-eco plugin list
+```
+
+## 7. Verification
+
+hawk-eco includes a self-verification system to validate local changes before contributing:
+
+```bash
+hawk-eco verify
+hawk-eco verify --fix
+```
+
+## Development
+
+```bash
+make lint
+hawk-eco verify
+```

Makefile

@@ -1,5 +1,5 @@
 # Canonical hawk-eco Makefile for Go library repos.
-# Source of truth: .shared-templates/Makefile.library.tmpl at the eco root.
+# Source of truth: https://github.com/GrayCodeAI/hawk/blob/main/.shared-templates/Makefile.library.tmpl
 # Placeholders rendered per repo: inspect.
 
 # ---------------------------------------------------------------------------
@@ -115,5 +115,7 @@ help: ## Show this help.
 	@grep -E '^[a-zA-Z_-]+:.*?## .*$$' $(MAKEFILE_LIST) | sort | awk 'BEGIN {FS = ":.*?## "}; {printf "\033[36m%-15s\033[0m %s\n", $$1, $$2}'
 
 .PHONY: hooks
-hooks:
-	git config core.hooksPath .githooks
+hooks: ## Install git hooks via lefthook (format, lint, conventional commits, co-author strip).
+	@command -v lefthook >/dev/null 2>&1 || (echo "install: go install github.com/evilmartians/lefthook@latest" && exit 1)
+	git config --unset core.hooksPath 2>/dev/null || true
+	lefthook install

internal/crawler/crawler.go

@@ -565,8 +565,11 @@ func isPrivateIP(ip net.IP) bool {
 		{mustParseCIDR("172.16.0.0/12")},
 		{mustParseCIDR("192.168.0.0/16")},
 		{mustParseCIDR("127.0.0.0/8")},
+		{mustParseCIDR("169.254.0.0/16")}, // link-local, incl. cloud metadata (169.254.169.254)
+		{mustParseCIDR("100.64.0.0/10")},  // CGNAT shared address space
 		{mustParseCIDR("::1/128")},
 		{mustParseCIDR("fc00::/7")},
+		{mustParseCIDR("fe80::/10")}, // IPv6 link-local
 	}
 	for _, r := range privateRanges {
 		if r.network.Contains(ip) {

lefthook.yml

@@ -1,5 +1,5 @@
 # Canonical lefthook config for hawk-eco Go repos.
-# Source of truth: .shared-templates/lefthook.yml.tmpl
+# Source of truth: https://github.com/GrayCodeAI/hawk/blob/main/.shared-templates/lefthook.yml.tmpl
 #
 # Install lefthook:
 #   brew install lefthook        (macOS)
@@ -75,6 +75,9 @@ pre-commit:
 pre-push:
   commands:
 
+    boundaries:
+      run: bash ./scripts/check-ecosystem-boundaries.sh
+
     test:
       run: go test ./... -count=1 -timeout=60s