Description
[Docs] Refresh + redesign the LLM Performance Matrix for Elastic Security
Live page: Large language model performance matrix for Elastic Security
Summary
Refresh the LLM performance matrix with our latest evaluation results and redesign the page so Security users can confidently choose a model for the workflow they actually run — alert triage, entity analytics, threat hunting, detection-rule authoring, workflow automation, multi-step response, Attack Discovery, or automatic migration — instead of reading a flat leaderboard with no context. The new evaluation breaks Agent Builder performance into seven sub-capabilities and adds an overall Agent Builder score, which lets us turn the page from "here are some numbers" into "here is the right model for your job, and here is how we know."
Why this matters for Security users
The current page lists six flat columns (Alerts, Security Knowledge, ES|QL Query Generation, Knowledge Base Retrieval, Attack Discovery, Automatic Migration), evaluated on AI Assistant (legacy) and a single average score. It is accurate but hard to act on. Each problem below maps to a concrete user benefit once fixed.
Updated results showing Agent Builder.
How the scores are calculated
The matrix uses three top-line capability scores — Agent Builder, Attack Discovery, and Automatic Migration — that roll up into a single Overall Score, so a user can read top-down from "how does this model perform across our AI features?" to "how good is it at the specific job I care about?":
- Overall Agent Builder Score is the average of the seven Agent Builder sub-capabilities (Alert Analysis, Entity Analytics, Threat Hunting, Detection Rules, Workflow Authoring, Triggering Workflows, Multi-Step Executions). It summarizes how well a model handles agentic Security work end to end.
- Overall Score is the average of the Agent Builder, Attack Discovery, and Automatic Migration scores. It reflects how a model performs across the breadth of our AI features rather than any single workflow, and is the default sort for the table.
Recommendation threshold
Any model that scores 5 or below for a capability is not recommended for that task. - we need to make sure this is very clear.
Rather than collapsing those results into a "Not Recommended" label that hides the underlying number, the refreshed matrix now shows each model's real score (see attached) so users can see exactly how far below the bar a model falls and make their own trade-off — a 4.9 and a 1.0 are very different signals, and the old label erased that distinction. The 1–10 scale stays the same; we simply surface the number and treat ≤ 5 as the "not recommended" line for that capability.
What each Agent Builder sub-capability measures
Plain-language definitions for the page (sourced from our evaluation rubric; no internal harness details exposed):
- Alert Analysis — triage an alert, reach the correct disposition, pull related alerts, and enrich with threat intel.
- Entity Analytics — investigate hosts/users using purpose-built entity lookups and risk context.
- Threat Hunting — generate and run queries against process/file/network telemetry to find specific hunt artifacts.
- Detection Rules — author a working detection rule, grounded in research where requested.
- Workflow Authoring — produce a valid, executable automation workflow (verified by actually creating, enabling, and running it).
- Triggering Workflows — call the correct backed action for the task (e.g. hash lookup, on-call schedule, case creation).
- Multi-Step Executions — chain several steps in the right order, carrying findings forward, without skipping or fabricating steps.
Resources
n/a
Which deployment methods does this change impact?
Unknown
Feature differences
n/a
What Elastic Stack release is this request related to?
N/A
Serverless release
No response
Collaboration model
Unknown
Point of contact.
Main contact: @dhru42
Stakeholders: @jamesspi
TriageBot — Complete
2026-06-30
Type: documentation (enhancement — redesign + refresh of existing docs page)
Section check:
- ✅ Problem statement: current page is a flat leaderboard that is hard to act on; Security users need workflow-specific model guidance
- ✅ Proposed outcome: new page surfaces per-capability real scores, Agent Builder sub-scores, and a ≤5 recommendation threshold with plain-language definitions
Cross-references:
- Live page URL (
/solutions/security/ai/large-language-model-performance-matrix) — valid and accessible
Next step: none — issue is complete
Generated by Gh Aw Issue Triage for issue #7146 · 22.9 AIC · ⌖ 7.43 AIC · ⊞ 9.1K · ◷
Description
[Docs] Refresh + redesign the LLM Performance Matrix for Elastic Security
Live page: Large language model performance matrix for Elastic Security
Summary
Refresh the LLM performance matrix with our latest evaluation results and redesign the page so Security users can confidently choose a model for the workflow they actually run — alert triage, entity analytics, threat hunting, detection-rule authoring, workflow automation, multi-step response, Attack Discovery, or automatic migration — instead of reading a flat leaderboard with no context. The new evaluation breaks Agent Builder performance into seven sub-capabilities and adds an overall Agent Builder score, which lets us turn the page from "here are some numbers" into "here is the right model for your job, and here is how we know."
Why this matters for Security users
The current page lists six flat columns (Alerts, Security Knowledge, ES|QL Query Generation, Knowledge Base Retrieval, Attack Discovery, Automatic Migration), evaluated on AI Assistant (legacy) and a single average score. It is accurate but hard to act on. Each problem below maps to a concrete user benefit once fixed.
Updated results showing Agent Builder.
How the scores are calculated
The matrix uses three top-line capability scores — Agent Builder, Attack Discovery, and Automatic Migration — that roll up into a single Overall Score, so a user can read top-down from "how does this model perform across our AI features?" to "how good is it at the specific job I care about?":
Recommendation threshold
Any model that scores 5 or below for a capability is not recommended for that task. - we need to make sure this is very clear.
Rather than collapsing those results into a "Not Recommended" label that hides the underlying number, the refreshed matrix now shows each model's real score (see attached) so users can see exactly how far below the bar a model falls and make their own trade-off — a 4.9 and a 1.0 are very different signals, and the old label erased that distinction. The 1–10 scale stays the same; we simply surface the number and treat ≤ 5 as the "not recommended" line for that capability.
What each Agent Builder sub-capability measures
Plain-language definitions for the page (sourced from our evaluation rubric; no internal harness details exposed):
Resources
n/a
Which deployment methods does this change impact?
Unknown
Feature differences
n/a
What Elastic Stack release is this request related to?
N/A
Serverless release
No response
Collaboration model
Unknown
Point of contact.
Main contact:
@dhru42Stakeholders:
@jamesspiTriageBot — Complete
2026-06-30
Type: documentation (enhancement — redesign + refresh of existing docs page)
Section check:
Cross-references:
/solutions/security/ai/large-language-model-performance-matrix) — valid and accessibleNext step: none — issue is complete