azure-diagnostics: Add Inspektor Gadget Reference

Signed-off-by: Qasim Sarfraz <qasimsarfraz@microsoft.com>

Author: mqasimsarfraz

plugin/skills/azure-diagnostics/SKILL.md

@@ -4,7 +4,7 @@ description: "Debug Azure production issues on Azure using AppLens, Azure Monito
 license: MIT
 metadata:
   author: Microsoft
-  version: "1.0.4"
+  version: "1.0.5"
 ---
 
 # Azure Diagnostics

plugin/skills/azure-diagnostics/aks-troubleshooting/aks-troubleshooting.md

@@ -20,6 +20,8 @@ Primary AKS troubleshooting guide for incidents routed from [../SKILL.md](../SKI
 
 When gathering AKS diagnostic evidence, prefer `mcp_azure_mcp_aks`, then the smallest discovered AKS-MCP tool that fits the read, then supporting Azure tools such as `mcp_azure_mcp_applens`, `mcp_azure_mcp_monitor`, or `mcp_azure_mcp_resourcehealth`. Use raw `az aks` and `kubectl` only when the AKS-MCP surface cannot perform the needed check.
 
+When standard diagnostics do not reveal root cause, use **Inspektor Gadget** for real-time, low-level node and pod observability (DNS traces, TCP traces, process snapshots, file access traces). See [references/inspektor-gadget.md](references/inspektor-gadget.md) for the gadget catalog, command patterns, and symptom-to-gadget mapping.
+
 See [references/aks-mcp.md](references/aks-mcp.md), [references/structured-input-modes.md](references/structured-input-modes.md), [references/command-flows.md](references/command-flows.md)
 
 ## Required Inputs
@@ -47,6 +49,7 @@ If cluster identity is missing, stop and ask for it.
 1. Azure-side state first: cluster state, resource health, recent operations, node pool state, detector or monitoring output.
 2. Kubernetes-side state second: cluster reachability, nodes, `kube-system`, events, affected namespace, pod detail, logs.
 3. Use detector, warning-event, or metrics modes when the incoming data already matches them.
+4. Deep diagnostics; when steps 1–3 do not reveal root cause, use Inspektor Gadget for real-time tracing and snapshots on the affected node.
 
 ## Workflow
 

plugin/skills/azure-diagnostics/aks-troubleshooting/networking.md

@@ -30,6 +30,16 @@ kubectl run netdebug --image=curlimages/curl -it --rm -n <ns> -- \
 
 Pods that are running but not Ready are removed from Endpoints. Check `kubectl get pod <pod> -n <ns>`.
 
+**Deep diagnostics with Inspektor Gadget** (when the above checks are inconclusive):
+
+Use the [IG base command pattern](references/inspektor-gadget.md) with `--k8s-namespace <ns> --k8s-podname <pod-name>` and these gadgets:
+
+- `snapshot_socket` (timeout 5) — check what ports the pod is listening on
+- `trace_tcp` (timeout 30) — trace connect/accept/close events
+- `trace_tcpdrop` + `trace_tcpretrans` (timeout 30) — packet drops and retransmissions
+
+See [references/inspektor-gadget.md](references/inspektor-gadget.md).
+
 ---
 
 ## DNS Resolution Failures
@@ -69,6 +79,12 @@ kubectl get svc kube-dns -n kube-system -o jsonpath='{.spec.clusterIP}'
 
 Custom VNet DNS must forward `.cluster.local` to the CoreDNS ClusterIP and other lookups to `168.63.129.16`.
 
+**Deep diagnostics with Inspektor Gadget** (when the above checks are inconclusive):
+
+Use the [IG base command pattern](references/inspektor-gadget.md) with `--k8s-namespace <ns> --k8s-podname <pod-name>` and `trace_dns` (timeout 30). Key signals: `rcode=3` (NXDOMAIN), `rcode=2` (SERVFAIL), high `latency` values, queries going to unexpected destinations.
+
+See [references/inspektor-gadget.md](references/inspektor-gadget.md).
+
 ---
 
 ## Load Balancer Stuck in Pending

plugin/skills/azure-diagnostics/aks-troubleshooting/node-issues.md

@@ -17,7 +17,7 @@ kubectl describe node <node-name>
 | `Ready`              | `False` | kubelet stopped reporting         | SSH to node; if unrecoverable, consider cordon/drain/delete\* |
 | `MemoryPressure`     | `True`  | Node running out of memory        | Evict pods; scale out pool; reduce pod density                |
 | `DiskPressure`       | `True`  | OS disk or ephemeral storage full | Check logs and images; clean up or increase disk              |
-| `PIDPressure`        | `True`  | Too many processes                | App spawning excessive threads/processes                      |
+| `PIDPressure`        | `True`  | Too many processes                | App spawning excessive threads/processes; use IG `snapshot_process` |
 | `NetworkUnavailable` | `True`  | CNI plugin issue                  | Check CNI pods in kube-system; node network config            |
 
 \*Only after explicit user request for remediation and confirmation of workload impact.
@@ -103,6 +103,10 @@ kubectl debug node/<node> -it --image=mcr.microsoft.com/cbl-mariner/base/core:2.
 
 Common culprit: high-volume container logs accumulating in `/var/log/containers`.
 
+**Deep diagnostics with Inspektor Gadget** (PID pressure or unknown process load):
+
+Use `snapshot_process` (timeout 5) to list all processes on the node. For node-wide scope, omit pod filters. See [references/inspektor-gadget.md](references/inspektor-gadget.md).
+
 ---
 
 ## Node Image / OS Upgrade Issues

plugin/skills/azure-diagnostics/aks-troubleshooting/pod-failures.md

@@ -52,6 +52,18 @@ kubectl describe pod <pod-name> -n <namespace> | grep -A2 "Last State"
 
 Fix: increase `resources.limits.memory` or optimize application memory usage. Check `kubectl top pod <pod-name> -n <namespace>` for actual usage.
 
+**OOM kill tracing with Inspektor Gadget:** Use `trace_oomkill` (timeout 30) with `--k8s-namespace <namespace> --k8s-podname <pod-name>` to see which process was killed and memory at kill time. See [references/inspektor-gadget.md](references/inspektor-gadget.md).
+
+**Deep diagnostics with Inspektor Gadget** (when logs and describe are inconclusive):
+
+Use the [IG base command pattern](references/inspektor-gadget.md) with `--k8s-namespace <namespace> --k8s-podname <pod-name>` and these gadgets:
+
+- `trace_exec` (timeout 30) — see what the container executes at startup
+- `trace_open` (timeout 30) — find missing configs/secrets (retval -2 = ENOENT, -13 = EACCES)
+- `snapshot_process` (timeout 5) — list running processes in the pod
+
+See [references/inspektor-gadget.md](references/inspektor-gadget.md).
+
 ---
 
 ## ImagePullBackOff

plugin/skills/azure-diagnostics/aks-troubleshooting/references/command-flows.md

@@ -76,6 +76,14 @@ kubectl get pvc -n <namespace>
 kubectl describe quota -n <namespace>
 ```
 
+## Deep Diagnostics Flow (Inspektor Gadget)
+
+```text
+Standard diagnostics inconclusive -> resolve target node -> select gadget from symptom-to-gadget map -> run IG command with namespace/pod filters -> interpret output -> correlate with prior evidence
+```
+
+Use when steps 1–3 of the evidence order ((Azure-side, Kubernetes-side, and detector evidence)) do not reveal root cause. See [inspektor-gadget.md](inspektor-gadget.md) for the full gadget catalog and command patterns.
+
 ## Safety Boundary
 
 Treat the following as change operations and avoid them unless the user explicitly asks for remediation:

plugin/skills/azure-diagnostics/aks-troubleshooting/references/inspektor-gadget.md

@@ -0,0 +1,134 @@
+# Inspektor Gadget (IG) Reference
+
+Use Inspektor Gadget for real-time, low-level node/pod diagnostics when `kubectl` is insufficient.
+
+## IG Version
+
+`<ig-version>` = `v0.51.0` — substitute this exact tag (with `v` prefix) wherever `<ig-version>` appears. Bump this line only.
+
+## Base Command Pattern
+
+```bash
+kubectl debug --profile=sysadmin node/<node-name> --attach --quiet \
+  --image=mcr.microsoft.com/oss/v2/inspektor-gadget/ig:<ig-version> \
+  -- ig run <gadget>:<ig-version> -o json --timeout <seconds> [filters...]
+```
+
+Always set `--timeout` after `--` to cap runtime. Use `--timeout 5` for snapshot/top, `--timeout 30` for trace/profile.
+
+> **Note:** IG uses `kubectl debug --profile=sysadmin` (privileged debug pod). Only run with explicit user approval and appropriate RBAC.
+
+**Required:** Resolve the node name first:
+
+```bash
+kubectl get pod <pod-name> -n <namespace> -o jsonpath='{.spec.nodeName}'
+```
+
+## Common Filters
+
+| Filter | Description |
+|---|---|
+| `--k8s-namespace <ns>` | Scope to a Kubernetes namespace |
+| `--k8s-podname <pod>` | Scope to a specific pod |
+| `--k8s-containername <ctr>` | Scope to a specific container |
+| `--timeout <seconds>` | Cap streaming duration for trace/profile gadgets |
+| `--max-entries <n>` | Max entries per batch for top/profile gadgets |
+| `--map-fetch-interval <dur>` | Map fetch interval for top (except `top_process`) and profile gadgets (default `1000ms`) |
+| `--interval <dur>` | Reporting interval for `top_process` only (e.g. `5s`) |
+| `--syscall-filters <list>` | Comma-separated syscalls for `traceloop` (e.g. `open,connect,accept`). **Always specify** to limit data volume |
+
+> **Tip:** For top/profile, set `--map-fetch-interval` ≤ half of `--timeout` to collect at least one batch. E.g. `--timeout 2 --map-fetch-interval 1000ms --max-entries 20`.
+>
+> **Note:** `top_process` uses `--interval` instead of `--map-fetch-interval`. E.g. `--timeout 10 --interval 5s --max-entries 20`.
+
+## Gadget Catalog
+
+### Networking
+
+| Gadget | Type | What It Does | When To Use |
+|---|---|---|---|
+| `trace_dns` | trace | Trace DNS queries and responses with latency | DNS failures, NXDOMAIN, SERVFAIL, slow resolution, intermittent DNS |
+| `trace_tcp` | trace | Trace TCP connect/accept/close events | Connection refused, timeouts, unexpected drops, mapping pod connectivity |
+| `trace_tcpretrans` | trace | Trace TCP retransmissions | Network congestion, lossy links, high latency between pods/services |
+| `trace_bind` | trace | Trace socket bind calls | Port conflicts, address-already-in-use errors |
+| `trace_sni` | trace | Trace TLS SNI (Server Name Indication) values | HTTPS routing issues, ingress TLS debugging, mTLS problems |
+| `snapshot_socket` | snapshot | List open sockets (TCP/UDP/Unix) | Port conflicts, listening ports, connection leaks, ECONNREFUSED |
+| `tcpdump` | special | Capture raw packets in pcap-ng format | Deep packet inspection, protocol-level debugging, reproducing network issues |
+
+#### tcpdump gadget
+
+Outputs raw pcap-ng data. Pipe to `tcpdump` for readable output:
+
+```bash
+kubectl debug --profile=sysadmin node/<node-name> --attach --quiet \
+  --image=mcr.microsoft.com/oss/v2/inspektor-gadget/ig:<ig-version> \
+  -- ig run tcpdump:<ig-version> -o pcap-ng --k8s-namespace <ns> --k8s-podname <pod> \
+     --timeout 30 --filter "port 80" \
+  | tcpdump -nvr -
+```
+
+Use `--filter "<expr>"` for tcpdump filters (e.g., `port 80`, `host 10.0.0.1`). Output must be `-o pcap-ng` (not `-o json`).
+
+### Process & Workload
+
+| Gadget | Type | What It Does | When To Use |
+|---|---|---|---|
+| `snapshot_process` | snapshot | List running processes in pod/node | PID pressure, unknown processes, verifying entrypoint, CrashLoopBackOff |
+| `trace_exec` | trace | Trace process execution (execve calls) | CrashLoopBackOff (what actually runs), unexpected child processes, security audit |
+| `trace_oomkill` | trace | Trace OOM kill events with victim details | OOMKilled pods — see which process was killed, memory usage at kill time |
+| `trace_signal` | trace | Trace signals delivered to processes | Unexpected SIGKILL/SIGTERM, liveness probe kills, graceful shutdown issues |
+| `top_process` | top | Rank processes by CPU/memory usage | Identifying resource-hungry processes inside a pod or across a node |
+| `profile_cpu` | profile | CPU profiling via stack sampling | High CPU usage investigation, finding hot code paths |
+| `traceloop` | trace | Record syscalls as a flight recorder | Catch-all for intermittent issues. **Always use `--syscall-filters`** (e.g., `open,connect,accept`) to limit data volume |
+
+### File & Storage
+
+| Gadget | Type | What It Does | When To Use |
+|---|---|---|---|
+| `trace_open` | trace | Trace openat syscall | Missing config/secret files (ENOENT), permission denied (EACCES), startup failures |
+| `trace_fsslower` | trace | Trace slow filesystem operations | Slow disk I/O, PVC performance issues, NFS/Azure Disk latency |
+| `top_file` | top | Rank files by read/write activity | Identifying I/O-heavy files, noisy log writers, disk pressure diagnosis |
+
+### Security & Audit
+
+| Gadget | Type | What It Does | When To Use |
+|---|---|---|---|
+| `trace_capabilities` | trace | Trace Linux capability checks | Permission denied from dropped capabilities, SecurityContext debugging |
+
+## Symptom-to-Gadget Map
+
+| Symptom | Gadget(s) |
+|---|---|
+| DNS resolution failures | `trace_dns` |
+| Connection refused / timeout | `trace_tcp` + `snapshot_socket` |
+| Silent connection drops | `trace_tcpretrans` |
+| High network latency | `trace_tcpretrans` |
+| TLS / HTTPS routing issues | `trace_sni` |
+| Port already in use | `trace_bind` + `snapshot_socket` |
+| CrashLoopBackOff (unknown cause) | `trace_exec` + `trace_open` |
+| OOMKilled pods | `trace_oomkill` + `top_process` |
+| Pod killed unexpectedly | `trace_signal` |
+| PID pressure on node | `snapshot_process` + `top_process` |
+| "Too many open files" | `top_file` |
+| Missing config / secret mount | `trace_open` |
+| Slow disk / PVC performance | `trace_fsslower` + `top_file` |
+| Permission denied (capabilities) | `trace_capabilities` |
+| High CPU (unknown cause) | `profile_cpu` + `top_process` |
+| Deep packet inspection | `tcpdump` |
+| Catch-all / intermittent issues | `traceloop` (use `--syscall-filters`) |
+
+## Gadget Type Reference
+
+| Type | Behavior | IG --timeout |
+|---|---|---|
+| `snapshot` | Point-in-time data, returns immediately | `--timeout 5` |
+| `top` | Aggregated view, returns quickly | `--timeout 5` |
+| `trace` | Streams events in real-time | `--timeout 30` |
+| `profile` | Samples over a duration | `--timeout 30` |
+| `tcpdump` | Streams pcap-ng data, pipe to `tcpdump -nvr -` | `--timeout 30` |
+
+## Guardrails
+
+- IG gadgets are **read-only** — they do not modify cluster or application state.
+- Resolve the correct node name before running any IG command.
+- Always set `--timeout` to cap runtime. Prefer snapshot/top for quick checks; trace/profile for behavior over time.