Add MAT CLI Heapdump Command Playbook and OQL Recipes documentation

Demogorgon314 · Demogorgon314 · commit c854b32e44da · 2026-03-15T23:37:36.000+08:00
diff --git a/skills/mat-cli-heapdump-investigator/SKILL.md b/skills/mat-cli-heapdump-investigator/SKILL.md
@@ -0,0 +1,95 @@
+---
+name: mat-cli-heapdump-investigator
+description: Investigate Java heap dumps with Eclipse MAT CLI. Use when Codex needs to analyze `.hprof`, `.phd`, or companion javacore-backed heap dumps to diagnose memory leaks, find the biggest retained objects or dominators, inspect object field values for a class instance, trace paths to GC roots, review thread state or stack snippets available in the dump, or run MAT reports and OQL queries for deeper heap forensics.
+---
+
+# MAT CLI Heapdump Investigator
+
+## Overview
+
+Use `mat-cli` as the first-line tool for Java heap-dump triage. Prefer the dedicated CLI commands with stable JSON contracts, then escalate to `query` or `oql` only when a direct command cannot answer the question cleanly.
+
+## Start Here
+
+- Confirm `mat-cli` is reachable with `mat-cli --help`.
+- Prefer `--format json` while reasoning, then rerun one or two focused commands with `--format text` when you need readable excerpts for the user.
+- Start broad and narrow gradually: `summary` -> `top-consumers` or `histogram` -> `instances` -> `inspect-object` -> `path2gc`.
+- Use absolute heap paths and keep every suspect object's `0x...` address in your notes.
+- Keep companion files beside the heap when possible. For OpenJ9 `.phd` dumps, a nearby `javacore` can improve thread analysis.
+- Use `--query-file` or `--command-file` for long expressions, embedded quotes, regexes, baseline paths, or inner class names containing `$`.
+- If `mat-cli` is not installed, install it first with `brew install demogorgon314/mat-cli/mat-cli`. If Homebrew is unavailable but the current workspace is the MAT source tree, build it from the `parent/` module with `mvn clean package -DskipTests -Dmat-product=mat-cli`.
+
+## Core Workflow
+
+1. Establish the snapshot baseline.
+
+```bash
+mat-cli summary <heap> --format json
+mat-cli threads <heap> --format json --limit 20
+```
+
+Use `summary` for heap format, object count, class count, and used heap. Use `threads` early when the leak might be thread-local, blocked, or finalizer-related. Remember that thread output is best-effort from the heap dump, not a full `jstack`.
+
+2. Identify dominant retainers and the biggest objects.
+
+```bash
+mat-cli top-consumers <heap> --format json --limit 20 --depth 3
+mat-cli histogram <heap> --format json --limit 50
+```
+
+Use `top-consumers` to find the biggest retained objects, classes, class loaders, and packages. Use `histogram` to spot explosive class counts and approximate retained sizes. Keep the `objectAddress` values from `biggestObjects`, `classes`, and `classLoaders` for follow-up drills.
+
+3. Narrow to a suspect class or instance set.
+
+```bash
+mat-cli instances <heap> --class com.example.CacheEntry --format json --limit 50
+mat-cli instances <heap> --class-regex 'com\\.example\\..*Cache.*' --format json --limit 50
+mat-cli instances <heap> --class-contains ThreadLocal --include-subclasses --format json --limit 50
+```
+
+Use `instances` when the user names a class or when `histogram` points at one. Prefer `--class` for exact matches. Use regex or contains matching only when the package or suffix is uncertain. Add `--include-subclasses` only when inheritance matters.
+
+4. Inspect concrete object state and field values.
+
+```bash
+mat-cli inspect-object <heap> --object 0x1234abcd --format json --depth 4 --limit 20
+mat-cli inspect-object <heap> --object 0x1234abcd --select-field value --format json --limit 20
+mat-cli inspect-object <heap> --object 0x1234abcd --field-path cleaner.offsetMap --format json
+```
+
+Use `inspect-object` to answer questions like "what value is stored in this object's fields?" Prefer `--field-path` when you need one nested value quickly. Prefer `--select-field` when one direct field is the real payload. Increase `--depth` carefully; the default is intentionally conservative. Use `--show-nulls` only when missing references are relevant to the diagnosis.
+
+5. Explain why a suspect survives GC.
+
+```bash
+mat-cli path2gc <heap> --object 0x1234abcd --format json --depth 8 --limit 20
+mat-cli query <heap> --command "show_dominator_tree 0x1234abcd" --format json --limit 20 --depth 4
+```
+
+Use `path2gc` for the shortest retaining path to GC roots. Use `show_dominator_tree` when `path2gc` is too narrow and you need local dominator context. If `path2gc` says the object is already a GC root, say that explicitly and pivot back to `inspect-object`, `threads`, or a dominator query.
+
+6. Run advanced MAT queries only after the direct commands.
+
+```bash
+mat-cli list-queries --format json
+mat-cli describe-query thread_overview --format json
+mat-cli query <heap> --command "thread_overview" --format json
+mat-cli query <heap> --command "finalizer_thread" --format json
+mat-cli query <heap> --command "default_report org.eclipse.mat.api:suspects" --format json
+```
+
+Use `query` for registered MAT reports and analyses that do not have dedicated CLI wrappers. Inspect the query id first if you are unsure about syntax or arguments.
+
+## Reporting Rules
+
+- Separate shallow heap from retained heap.
+- Quote exact class names and object addresses.
+- Treat histogram retained sizes as approximate unless the command guarantees otherwise.
+- Tie each leak suspect to at least two pieces of evidence, such as a top-consumers row, a retained path, a field value, a thread, or a MAT report section.
+- Call out uncertainty clearly when stack frames are missing or when the dump only offers best-effort thread metadata.
+- Prefer a short evidence chain over dumping raw JSON or huge tables back to the user.
+
+## References
+
+- Read [references/command-playbook.md](references/command-playbook.md) for ready-to-run commands and advanced MAT query patterns.
+- Read [references/oql-recipes.md](references/oql-recipes.md) when built-in commands are not enough and you need validated OQL snippets.
diff --git a/skills/mat-cli-heapdump-investigator/agents/openai.yaml b/skills/mat-cli-heapdump-investigator/agents/openai.yaml
@@ -0,0 +1,4 @@
+interface:
+  display_name: "MAT CLI Heapdump Investigator"
+  short_description: "Investigate Java heap dumps with MAT CLI"
+  default_prompt: "Use $mat-cli-heapdump-investigator to analyze this heap dump with mat-cli, find leak suspects, inspect retained memory, and explain the results."
diff --git a/skills/mat-cli-heapdump-investigator/references/command-playbook.md b/skills/mat-cli-heapdump-investigator/references/command-playbook.md
@@ -0,0 +1,151 @@
+# MAT CLI Heapdump Command Playbook
+
+## Contents
+
+- Baseline and schemas
+- Biggest objects and dominators
+- Class-focused drilldown
+- Object inspection and value extraction
+- Retention paths and merged paths
+- Threads, finalizers, and reports
+- Comparing two dumps
+- Failure handling
+
+## Baseline and Schemas
+
+If `mat-cli` is missing, install it with:
+
+```bash
+brew install demogorgon314/mat-cli/mat-cli
+```
+
+If Homebrew is unavailable and you are already inside the MAT source tree, fall back to building the standalone CLI from `parent/`.
+
+Use these first:
+
+```bash
+mat-cli summary <heap> --format json
+mat-cli describe top-consumers --format json
+mat-cli schema threads --format json
+```
+
+Use `describe` or `schema` when you need to know the stable JSON payload before scripting against a command. The dedicated commands already return stable `mat-cli/v2` envelopes, so prefer them over free-form MAT queries whenever possible.
+
+## Biggest Objects and Dominators
+
+Start with retained heap, not just object count:
+
+```bash
+mat-cli top-consumers <heap> --format json --limit 20 --depth 3
+mat-cli histogram <heap> --format json --limit 50
+```
+
+Interpret the output like this:
+
+- `top-consumers` answers "what dominates retained memory right now?"
+- `histogram` answers "which classes are numerous or large in aggregate?"
+- `top-consumers.biggestObjects[].objectAddress` is the best bridge into `inspect-object`, `path2gc`, and `show_dominator_tree`.
+- `histogram` is a table result. Look for `class_name`, instance count, shallow heap, and retained heap columns.
+
+Use a smaller `--limit` when you only need the top suspects. Increase `--depth` on `top-consumers` when package aggregation matters.
+
+## Class-Focused Drilldown
+
+Use `instances` once a class looks suspicious:
+
+```bash
+mat-cli instances <heap> --class com.example.CacheEntry --format json --limit 20
+mat-cli instances <heap> --class-regex 'com\\.example\\..*Cache.*' --format json --limit 20
+mat-cli instances <heap> --class-contains ThreadLocal --include-subclasses --format json --limit 50
+```
+
+Choose the selector deliberately:
+
+- Use `--class` for exact FQCN matches.
+- Use `--class-regex` for uncertain package names or multiple related classes.
+- Use `--class-contains` for fast discovery when the class name fragment is all you know.
+- Use `--include-subclasses` only when the leak might live in subtype instances.
+
+## Object Inspection and Value Extraction
+
+Use `inspect-object` for single-object truth:
+
+```bash
+mat-cli inspect-object <heap> --object 0x1234abcd --format json --depth 4 --limit 20
+mat-cli inspect-object <heap> --object 0x1234abcd --select-field value --format json --limit 20
+mat-cli inspect-object <heap> --object 0x1234abcd --field-path cleaner.offsetMap --format json
+mat-cli inspect-object <heap> --object 0x1234abcd --format text --field-path count
+```
+
+Use it this way:
+
+- Reach for `--field-path` when the user asks for one concrete nested value.
+- Reach for `--select-field` when the root object is just a wrapper and one direct field is the real payload.
+- Watch for `_meta.value.kind` previews in JSON. MAT can surface text previews or byte-array previews without walking the whole subtree.
+- Use `--show-nulls` only when null references are part of the bug story.
+
+If a field path fails, fix the path instead of assuming the value is null. MAT reports missing fields explicitly.
+
+## Retention Paths and Merged Paths
+
+Use the simplest retention question first:
+
+```bash
+mat-cli path2gc <heap> --object 0x1234abcd --format json --depth 8 --limit 20
+mat-cli query <heap> --command "show_dominator_tree 0x1234abcd" --format json --limit 20 --depth 4
+mat-cli query <heap> --command "merge_shortest_paths -groupby FROM_GC_ROOTS com.example.CacheEntry" --format json --limit 20 --depth 4
+```
+
+Pick the command based on the question:
+
+- `path2gc` finds the shortest retention chain for one object.
+- `show_dominator_tree` shows the local dominator neighborhood for one suspect.
+- `merge_shortest_paths` is useful when many instances of the same class appear to leak through the same retaining structure.
+
+If `path2gc` reports that the object is already a GC root, stop searching for an upstream holder and inspect the root object's own fields, thread ownership, or role.
+
+## Threads, Finalizers, and Reports
+
+Use dedicated thread support first:
+
+```bash
+mat-cli threads <heap> --format json --limit 50
+mat-cli query <heap> --command "thread_overview" --format json
+mat-cli query <heap> --command "finalizer_thread" --format json
+mat-cli query <heap> --command "finalizer_thread_locals" --format json
+```
+
+Use MAT reports when you need a broader narrative:
+
+```bash
+mat-cli query <heap> --command "default_report org.eclipse.mat.api:overview" --format json
+mat-cli query <heap> --command "default_report org.eclipse.mat.api:suspects" --format json
+mat-cli query <heap> --command "default_report org.eclipse.mat.api:top_components" --format json
+```
+
+Remember:
+
+- `threads` and `thread_overview` are best-effort views of heap data, not a runtime thread dump.
+- `default_report` commands often return section-style results. Summarize the important sections instead of replaying the full structure.
+- Use `list-queries` and `describe-query` before calling obscure MAT query ids.
+
+## Comparing Two Dumps
+
+When you have a baseline and a current dump from comparable workloads, use MAT's compare reports:
+
+```bash
+mat-cli query <new-heap> --command "default_report org.eclipse.mat.api:overview2 -params baseline=/abs/path/baseline.hprof" --format json
+mat-cli query <new-heap> --command "default_report org.eclipse.mat.api:suspects2 -params baseline=/abs/path/baseline.hprof" --format json
+```
+
+Prefer `--command-file` if the baseline path contains spaces or the command grows longer.
+
+## Failure Handling
+
+Use these recovery moves:
+
+- `Unknown MAT query`: run `mat-cli list-queries --format json` and `mat-cli describe-query <id> --format json`.
+- OQL syntax problems: move the expression into a UTF-8 file and use `--query-file`.
+- MAT query syntax problems: move the command into a UTF-8 file and use `--command-file`.
+- Invalid `path2gc` address: verify the address with `mat-cli oql <heap> --query "SELECT * FROM OBJECTS 0x1234abcd" --format json`.
+- Too much data: lower `--limit`, lower `--depth`, or pivot from `query` to a dedicated command.
diff --git a/skills/mat-cli-heapdump-investigator/references/oql-recipes.md b/skills/mat-cli-heapdump-investigator/references/oql-recipes.md
@@ -0,0 +1,102 @@
+# MAT CLI Heapdump OQL Recipes
+
+Use these only after the dedicated commands fail to answer the question cleanly. For one object and one field, `inspect-object` is usually simpler and safer than OQL.
+
+## Useful Pseudo Fields
+
+MAT OQL exposes metadata fields that are useful in leak analysis:
+
+- `@objectAddress`
+- `@usedHeapSize`
+- `@retainedHeapSize`
+- `@name`
+
+Use them to connect OQL results back to `inspect-object`, `path2gc`, or `top-consumers`.
+
+## List Objects with Address and Readable Value
+
+Validated in the CLI tests:
+
+```sql
+select s.@objectAddress as ADDRESS, toString(s) as VALUE
+from java.lang.String s
+```
+
+Run it with:
+
+```bash
+mat-cli oql <heap> --query-file strings.oql --format json --limit 20
+```
+
+Use this pattern when you need both an address and a human-readable rendering.
+
+## Jump to One Object by Address
+
+```sql
+SELECT * FROM OBJECTS 0x1234abcd
+```
+
+Use this to verify that an address from another command is valid before running `path2gc` or a more specific OQL expression.
+
+## Find Large or Suspicious String Objects
+
+Validated in the OQL tests:
+
+```sql
+SELECT * FROM java.lang.String s
+WHERE s.count > 100 AND s.@retainedHeapSize > s.@usedHeapSize
+```
+
+Use this pattern to find payload-heavy strings whose retention looks disproportionate.
+
+## Filter by Rendered Value
+
+Validated in the OQL tests:
+
+```sql
+SELECT * FROM java.lang.String s
+WHERE toString(s) LIKE "java.*"
+```
+
+This is useful when class identity is not enough and the content matters.
+
+## Ask for Dominators from OQL
+
+Validated in the OQL tests:
+
+```sql
+select objects dominators(s)
+from objects 0x1234abcd s
+```
+
+Use this when you need a quick dominator set in OQL form. For user-facing explanations, `path2gc` or `show_dominator_tree` is often easier to interpret.
+
+## Collapse to Distinct Classes
+
+Validated in the OQL tests:
+
+```sql
+select distinct objects classof(s)
+from java.lang.String s
+```
+
+Use this pattern when a broader instance query returns mixed runtime types and you want the concrete classes involved.
+
+## Inspect Map-Like Structures
+
+Validated in the OQL tests:
+
+```sql
+SELECT m AS map,
+       classof(m).@name AS type,
+       m[0:-1].size() AS size
+FROM INSTANCEOF java.util.AbstractMap m
+```
+
+Use this to find oversized maps before drilling into their entries with `inspect-object` or a more specific OQL query.
+
+## Quoting Guidance
+
+- Prefer `--query-file` for multi-line OQL.
+- Prefer `--query-file` when the expression contains `$`, regexes, nested quotes, or long baseline-dependent fragments.
+- If MAT parses the query but execution fails, narrow the scope instead of making the expression more complex immediately.
