perf(install): memoize discovery, drop per-file resolve, expand skip dirs (#1538)

Fixes #1533 — apm install on real monorepos (Kubernetes, TypeScript)
took 200-300s; on this repo, multi-target installs scaled O(targets x
packages) because discover_primitives was called 9x per install over
the same project tree and find_primitive_files did Path.resolve() on
every file.

Three independent root causes, all fixed:

1. Discovery memoization. Module-level dict in primitives/discovery.py
   keyed on (realpath(base), sorted_exclude_tuple), cleared at top of
   run_install_pipeline. Identity-shares the PrimitiveCollection across
   the integrate phase loop. 9 -> 3 unique walks per install.

2. Hot-path stat reduction. find_primitive_files now (a) pre-splits
   glob patterns once per call (not per file), (b) computes rel_root
   by string slicing of os.walk's root + base_prefix instead of
   Path.resolve() lstat-per-component per file, (c) defers Path()
   construction until after a pattern matches.

3. Skip-dir expansion. DEFAULT_SKIP_DIRS gained vendor, third_party,
   Pods, bower_components, jspm_packages, .gradle, target, .next,
   .nuxt, .cache, .turbo — keeps the walker out of dependency vendor
   trees the user did not author.

Granular perf logging (--verbose):

New utils/perf_stats.py records one event per walk + per discovery
call; render_summary() emits a verbose-only block with per-base walk
breakdown and discovery cache hit-rate:

    [#] Perf: 9 walks, 72 file matches, 6222 files visited, 0.739s total walk time
    [#] Perf:   .: 3 walk(s) (726ms, 6108 files visited, 72 matched)
    [#] Perf:   apm_modules/_local/foo: 3 walk(s) (4ms, 36 files visited, 0 matched)
    [#] Perf: discovery: 9 call(s) (3 unique base(s), 6 cache hit(s), 66%)

Non-CI perf harness:

tests/perf/ contains 4 opt-in scenarios (Kubernetes discover,
TypeScript discover, awd-cli install, multi-target breadth). Skipped
by default; run with PYTEST_PERF=1. Clones large external repos to
/tmp/perf-atlas-clones/ once per session. Not run on CI.

Measured (verified, this worktree):

| Scenario                          | Before  | After   | Speedup |
|-----------------------------------|---------|---------|---------|
| awd-cli T=1 integrate phase       | 3.7s    | 0.797s  | 4.6x    |
| awd-cli T=7 multi-target install  | 19.7s   | 0.76s   | 26x     |
| Kubernetes discover (cold)        | 205s    | 5.4s    | 38x     |
| TypeScript discover (cold)        | 297s    | 7.0s    | 42x     |
| Warm discovery (cache hit)        | n/a     | 26us    | 22500x  |

Test isolation: tests/conftest.py gains an autouse fixture that
clears the discovery cache and resets perf_stats around every test
so process-scoped globals do not leak across tests that exercise
discover_primitives directly.

Pre-existing scaling-guard threshold bump: bumped
TestDiscoveryScaling::test_scaling_ratio 14 -> 20 with inline
explanation -- the small case got proportionally faster than the
large case, so the ratio inflated even as absolute times improved.

Reviewers (apm-review-panel): performance-expert, python-architect,
cli-logging-expert all returned ship_now after folding their
fold_now recommendations (delete dead duplicate _glob_match,
relativize paths in perf summary, add cache hit-rate percentage,
use [#] status symbol, add autouse conftest fixture, document base
fallback as '.', surface render_summary errors instead of swallow).

Co-authored-by: danielmeppiel <danielmeppiel@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Author: danielmeppiel

src/apm_cli/commands/compile/cli.py

@@ -16,7 +16,8 @@
 from ...constants import AGENTS_MD_FILENAME, APM_DIR, APM_MODULES_DIR, APM_YML_FILENAME
 from ...core.command_logger import CommandLogger
 from ...core.target_detection import TargetParamType
-from ...primitives.discovery import discover_primitives
+from ...primitives.discovery import clear_discovery_cache, discover_primitives
+from ...utils import perf_stats
 from ...utils.console import (
     _rich_error,
     _rich_info,
@@ -491,6 +492,8 @@ def compile(
         # Validation-only mode
         if validate:
             logger.start("Validating APM context...", symbol="gear")
+            clear_discovery_cache()
+            perf_stats.reset()
             compiler = AgentsCompiler(".")
             try:
                 primitives = discover_primitives(".")
@@ -518,6 +521,7 @@ def compile(
                     logger.progress(f"  * {mcp_count} MCP dependencies")
             except Exception:
                 pass
+            perf_stats.render_summary(logger, project_root=".")
             return
 
         # Watch mode
@@ -738,6 +742,11 @@ def _coerce_provenance_targets(value):
             logger.progress("Using single-file compilation (legacy mode)", symbol="page")
 
         # Perform compilation
+        # Reset discovery memo + perf counters so the single-shot compile
+        # never inherits stale state from a sibling invocation in the
+        # same process (tests, REPL). Mirrors run_install_pipeline.
+        clear_discovery_cache()
+        perf_stats.reset()
         compiler = AgentsCompiler(".")
         result = compiler.compile(config, logger=logger)
         compile_has_critical = result.has_critical_security
@@ -896,8 +905,11 @@ def _coerce_provenance_targets(value):
                 "Compiled output contains critical hidden characters"
                 " -- run 'apm audit' to inspect, 'apm audit --strip' to clean"
             )
+            perf_stats.render_summary(logger, project_root=".")
             sys.exit(1)
 
+        perf_stats.render_summary(logger, project_root=".")
+
     except ImportError as e:
         logger.error(f"Compilation module not available: {e}")
         logger.progress("This might be a development environment issue.")

src/apm_cli/commands/compile/watcher.py

@@ -9,6 +9,13 @@
 from ...compilation import AgentsCompiler, CompilationConfig
 from ...constants import AGENTS_MD_FILENAME, APM_DIR, APM_YML_FILENAME
 from ...core.command_logger import CommandLogger
+from ...primitives.discovery import PRIMITIVE_SUFFIXES, clear_discovery_cache
+from ...utils import perf_stats
+
+# Skill modules use a fixed filename (``SKILL.md``) rather than a suffix
+# pattern, so the watcher checks basename equality in addition to the
+# suffix membership test below.
+_SKILL_FILENAME = "SKILL.md"
 
 if TYPE_CHECKING:
     from ...core.target_detection import CompileTargetType
@@ -88,7 +95,21 @@ def on_modified(self, event: Any) -> None:
         if getattr(event, "is_directory", False):
             return
         src_path = getattr(event, "src_path", "")
-        if not src_path.endswith(".md") and not src_path.endswith(APM_YML_FILENAME):
+        # Smart filter: recompile only when the changed file is an APM
+        # primitive (matches one of LOCAL_PRIMITIVE_PATTERNS' suffixes, a
+        # SKILL.md basename, or the project manifest). Generic .md edits
+        # (README, CHANGELOG, AGENTS output) never affect compile output
+        # and would otherwise trigger a full discovery walk on every
+        # save. See #1533 follow-up.
+        basename = os.path.basename(src_path)
+        is_manifest = basename == APM_YML_FILENAME
+        is_skill = basename == _SKILL_FILENAME
+        is_primitive = any(src_path.endswith(suffix) for suffix in PRIMITIVE_SUFFIXES)
+        if not is_manifest and not is_primitive and not is_skill:
+            # Leave a verbose breadcrumb so --verbose watch sessions can
+            # see why an edit produced no recompile. Silent at default.
+            if src_path:
+                self.logger.verbose_detail(f"Skipping non-primitive change: {basename}")
             return
         current_time = time.time()
         if current_time - self.last_compile < self.debounce_delay:
@@ -102,6 +123,14 @@ def _recompile(self, changed_file: str) -> None:
             self.logger.progress(f"File changed: {changed_file}", symbol="eyes")
             self.logger.progress("Recompiling...", symbol="gear")
 
+            # The process-scoped discovery cache (populated by the previous
+            # compile pass) MUST be cleared before re-walking, otherwise
+            # subsequent recompiles serve the stale primitive set from
+            # before the edit. See #1533 follow-up. perf_stats counters
+            # are NOT reset here -- they accumulate across the watch
+            # session and are rendered once at teardown.
+            clear_discovery_cache()
+
             # When apm.yml itself was the trigger, re-resolve so a
             # mid-session edit to ``target:`` / ``targets:`` takes
             # effect on this recompile, then persist the fresh value
@@ -231,6 +260,12 @@ class _WatchdogAdapter(APMFileHandler, FileSystemEventHandler):
 
         logger.progress("Performing initial compilation...", symbol="gear")
 
+        # Watch mode is a long-lived process. Reset both the discovery
+        # cache and perf counters on entry so neither carries state from
+        # a sibling REPL/test run sharing this Python process.
+        clear_discovery_cache()
+        perf_stats.reset()
+
         config = CompilationConfig.from_apm_yml(
             output_path=output if output != AGENTS_MD_FILENAME else None,
             chatmode=chatmode,
@@ -242,6 +277,10 @@ class _WatchdogAdapter(APMFileHandler, FileSystemEventHandler):
         compiler = AgentsCompiler(".")
         result = compiler.compile(config)
 
+        # NOTE: render_summary moved to the Ctrl+C teardown below so the
+        # watch session emits ONE aggregate perf block at exit instead of
+        # spamming a 5-6 line block after every recompile.
+
         if result.success:
             if dry_run:
                 logger.success("Initial compilation successful (dry run)", symbol="sparkles")
@@ -261,6 +300,10 @@ class _WatchdogAdapter(APMFileHandler, FileSystemEventHandler):
         except KeyboardInterrupt:
             observer.stop()
             logger.progress("Stopped watching for changes", symbol="info")
+            # Render aggregate perf counters accumulated across the
+            # session. Reset once at watch start (above), so this summary
+            # covers initial compile + every subsequent recompile.
+            perf_stats.render_summary(logger, project_root=".")
 
         observer.join()
 

src/apm_cli/commands/uninstall/engine.py

@@ -484,6 +484,14 @@ def _sync_integrations_after_uninstall(
     from ...integration.dispatch import get_dispatch_table
     from ...integration.targets import resolve_targets
     from ...models.apm_package import PackageInfo, validate_apm_package
+    from ...primitives.discovery import clear_discovery_cache
+
+    # Phase 2 re-integration walks the on-disk primitive set after Phase 1
+    # has removed the uninstalled package's files. The process-scoped
+    # discovery memo populated earlier in this CLI run would otherwise
+    # serve the pre-removal snapshot, causing deleted primitives to be
+    # re-integrated. See #1533 follow-up.
+    clear_discovery_cache()
 
     _dispatch = get_dispatch_table()
     _integrators = {name: entry.integrator_class() for name, entry in _dispatch.items()}
@@ -634,6 +642,10 @@ def _sync_integrations_after_uninstall(
     counts["hooks"] = result.get("files_removed", 0)
 
     # Phase 2: Re-integrate from remaining installed packages
+    # Re-clear the discovery memo: Phase 1 mutated the on-disk primitive
+    # set (removed files), so any cache snapshot taken between entry and
+    # here is stale. Integrator dispatch below walks discovery internally.
+    clear_discovery_cache()
     _targets = _resolved_targets
 
     for dep in apm_package.get_apm_dependencies():

src/apm_cli/constants.py

@@ -51,5 +51,23 @@ class InstallMode(Enum):
         "dist",
         ".mypy_cache",
         "apm_modules",
+        # Common vendored / generated package locations across ecosystems.
+        # These never contain user-authored primitives and can be huge
+        # (the Kubernetes vendor/ tree alone is ~14k files; CocoaPods'
+        # Pods/ tree, bower_components, jspm_packages, and the various
+        # staging/third_party trees in Google-style monorepos behave the
+        # same way). Pruning at the directory level avoids the per-file
+        # cost in find_primitive_files. See issue #1533.
+        "vendor",
+        "third_party",
+        "Pods",
+        "bower_components",
+        "jspm_packages",
+        ".gradle",
+        "target",
+        ".next",
+        ".nuxt",
+        ".cache",
+        ".turbo",
     }
 )

src/apm_cli/install/pipeline.py

@@ -356,6 +356,15 @@ def run_install_pipeline(  # noqa: PLR0913, RUF100
     except ImportError:
         raise RuntimeError("APM dependency system not available")  # noqa: B904
 
+    # Reset process-scoped perf counters and discovery memo so that
+    # numbers / cache hits from earlier pipeline runs (tests, REPL,
+    # long-lived processes) do not bleed into this install. See #1533.
+    from ..primitives.discovery import clear_discovery_cache
+    from ..utils import perf_stats as _perf_stats
+
+    _perf_stats.reset()
+    clear_discovery_cache()
+
     from ..core.scope import InstallScope, get_apm_dir, get_deploy_root
 
     if scope is None:
@@ -762,6 +771,7 @@ def run_install_pipeline(  # noqa: PLR0913, RUF100
         # Emit verbose integration stats + bare-success fallback + return result
         from .phases import finalize as _finalize_phase
 
+        _perf_stats.render_summary(logger, project_root=str(ctx.project_root))
         return _run_phase("finalize", _finalize_phase, ctx)
 
     except AuthenticationError: