diff --git a/site/content/blog/yara-x-just-got-faster/graph.png b/site/content/blog/yara-x-just-got-faster/graph.png new file mode 100644 index 000000000..4fd4f5f1a Binary files /dev/null and b/site/content/blog/yara-x-just-got-faster/graph.png differ diff --git a/site/content/blog/yara-x-just-got-faster/index.md b/site/content/blog/yara-x-just-got-faster/index.md new file mode 100644 index 000000000..fe7b7d750 --- /dev/null +++ b/site/content/blog/yara-x-just-got-faster/index.md @@ -0,0 +1,143 @@ +--- +title: "YARA-X just got faster" +description: "Explore recent performance improvements in YARA-X, including daachorse Aho-Corasick integration and RegexSet evaluation optimizations." +summary: "" +date: 2026-05-13T13:20:00+02:00 +lastmod: 2026-05-13T13:20:00+02:00 +draft: false +weight: 50 +categories: [] +tags: [] +contributors: [ "Victor M. Alvarez" ] +pinned: false +homepage: false +seo: + title: "YARA-X just got faster" + description: "Explore recent performance improvements in YARA-X, including daachorse Aho-Corasick integration and RegexSet evaluation optimizations." + canonical: "" + noindex: false +--- +Almost a year ago, we published a blog post titled [YARA-X just got +smarter]({{< ref "blog/yara-x-just-got-smarter/index.md" >}}). It described +some improvements aimed at making life easier for our users. Today, we are +publishing a similar entry, but this time it focuses on performance +improvements rather than usability. + +While YARA-X has consistently offered robust performance, two major recent +changes—[PR #654](https://github.com/VirusTotal/yara-x/pull/654) and +[PR #657](https://github.com/VirusTotal/yara-x/pull/657)—introduce +groundbreaking optimizations to core pattern matching and rules that make heavy +use of the `matches` operator. + +Here is a deep dive into how these performance gains were achieved. + + +## 1. Replacing aho_corasick with daachorse + +When searching for hundreds of thousands of byte strings simultaneously, +YARA-X relies on the Aho-Corasick algorithm. This is the primary workhorse +of the YARA-X scanning engine. + +Previously, YARA-X utilized the widely used +[`aho_corasick`](https://github.com/BurntSushi/aho-corasick) crate, a robust +and performant implementation of the Aho-Corasick algorithm in Rust. However, +[PR #654](https://github.com/VirusTotal/yara-x/pull/654) replaces `aho_corasick` +with [`daachorse`](https://github.com/daac-tools/daachorse), a much faster +implementation built around a compact double-array trie data structure. + +Conceptually, the implementation in `daachorse` mirrors the core design +principles of the original C-based YARA engine. Its performance gains over the +standard `aho_corasick` crate stem from two primary architectural advantages: + +1. **Tighter inner loop**: Byte-by-byte traversal through NFA transitions + requires fewer memory indirections and branching instructions. + +2. **Cache-efficient storage**: The double-array automaton layout optimizes + memory locality, drastically reducing CPU cache misses during intensive + scanning over large files. + +Migrating to `daachorse` had its own challenges, though. To facilitate a +seamless transition and ensure YARA-X's matching semantics were fully +preserved, upstream contributions were made to `daachorse` to align its +API closer to the familiar `aho_corasick` crate. Most notably, +[PR #128](https://github.com/daac-tools/daachorse/pull/128) added +support for Aho-Corasick automatons with duplicate patterns or no patterns +at all. + +#### Real-world benchmarks + +Using the full rule set from [YARA Forge](https://yarahq.github.io/) across +a corpus of 500 random VirusTotal files (totaling ~24GB), `daachorse` achieved +remarkable benchmark results: + +| Scanner | Scan Time | Speedup | +|---|---|---| +| YARA-X (daachorse) | 26.152s | — | +| YARA-X (aho_corasick) | 57.584s | YARA-X with daachorse is 2.20x faster | +| YARA (legacy) | 48.275s | YARA-X with daachorse is 1.85x faster | + +This brings YARA-X up to speed with legacy YARA in terms of performance. Now, +we can guarantee that **YARA-X is faster than YARA** in almost all cases, with +very few exceptions. + +Starting with YARA-X v1.16.0, you can enjoy this improvement. + +## 2. Simultaneous regex evaluation + +When authoring complex YARA rules, analysts frequently evaluate a single +variable, structure field access, or module output against dozens of distinct +regular expressions inside an `or` condition. A common example we had +observed in VirusTotal is inspecting domain names or URLs: + +```yara +condition: + vt.net.domain.raw matches /evil-c2\.com/ or + vt.net.domain.raw matches /phishing-login\.net/ or + vt.net.domain.raw matches /malicious-dga[0-9]{4}\.org/ +``` + +The example above is only for illustrating the point; in real life, we have seen +rules with hundreds of `matches` operations like these. + +Previously, YARA-X evaluated each `matches` expression sequentially. If a rule +contained fifty regex checks against the same field, the scanning engine would +execute fifty separate passes over the target string. + +[PR #657](https://github.com/VirusTotal/yara-x/pull/657) introduces an advanced +compiler optimization that detects identical match targets within `or` +conditions and automatically groups their regular expressions into a unified +[`RegexSet`](https://docs.rs/regex/latest/regex/struct.RegexSet.html). + +A `RegexSet` is a specialized feature of the standard Rust +[`regex`](https://github.com/rust-lang/regex) crate designed precisely to speed +up the matching of a single string against multiple regular expressions at the +same time. + +When regular expressions are evaluated sequentially, the scanning engine incurs +repeated overhead: it must re-initialize its search state and scan the entire +target string from beginning to end for every single pattern. As rule +complexity grows, this redundant scanning becomes a major performance +bottleneck. + +By grouping these expressions into a `RegexSet`, the compiler combines all +individual regular expressions into a single, unified state machine +(automaton). When a target string is inspected, the scanning engine executes +exactly **one scan pass** over the data. As it steps through each character in +the target string, the unified state machine simultaneously tracks and evaluates +potential matches for all grouped regex patterns at once. + +This optimization shifts the scanning complexity from scaling linearly with the +number of regex patterns to a single, highly efficient pass over the input +string, delivering a huge speed improvement when executing complex rule sets. + +This improvement will be released in the upcoming YARA-X v1.17.0. However, we +have already deployed these changes to VirusTotal Livehunt and observed a +dramatic drop in scanning time for domain-related rules. The 99th percentile +execution time was cut in half, dropping from an average of over 6 seconds to +less than 3 seconds. In other words, 99% of rule evaluations now complete in +under 3 seconds, whereas they previously exceeded 6 seconds. Additionally, the CPU +costs associated to running this workload was also cut in half. + +{{< figure src="graph.png" caption="Scanning time for domain-related rules in VirusTotal Livehunt. Notice the abrupt drop on May 11." >}} + + diff --git a/site/content/blog/yara-x-is-getting-smarter/index.md b/site/content/blog/yara-x-just-got-smarter/index.md similarity index 99% rename from site/content/blog/yara-x-is-getting-smarter/index.md rename to site/content/blog/yara-x-just-got-smarter/index.md index e0b998547..eeb3b72ce 100644 --- a/site/content/blog/yara-x-is-getting-smarter/index.md +++ b/site/content/blog/yara-x-just-got-smarter/index.md @@ -12,7 +12,7 @@ contributors: [ "Victor M. Alvarez" ] pinned: false homepage: false seo: - title: "YARA-X is getting smarter!" + title: "YARA-X just got smarter" description: "Unpacking new features for detecting unsatisfiable expressions" canonical: "" # custom canonical URL (optional) noindex: false # false (default) or true diff --git a/site/hugo_stats.json b/site/hugo_stats.json index 63dd1a8bf..b627090fd 100644 --- a/site/hugo_stats.json +++ b/site/hugo_stats.json @@ -75,6 +75,7 @@ "bg-dots", "blog", "blog-header", + "blur-up", "btn", "btn-close", "btn-cta", @@ -171,6 +172,7 @@ "justify-content-center", "justify-content-end", "justify-content-start", + "lazyload", "lead", "list", "list-inline", @@ -322,6 +324,7 @@ "-t---tab-size-num_spaces", "-what-about-the-original-yara", "-x---module-data-modulefile", + "1-replacing-aho_corasick-with-daachorse", "1-smart-warnings-for-lowercase-string-comparisons", "100", "102", @@ -329,6 +332,7 @@ "104", "105", "2-enhanced-integer-range-validation", + "2-simultaneous-regex-evaluation", "92", "93", "94", @@ -359,11 +363,13 @@ "api-overview", "api-reference", "applying-the-same-condition-to-many-patterns", + "array-variables", "assembly", "assemblyref", "base64-modifier", "benefits-for-teams-and-organizations", "better-error-reporting", + "better-performance", "binary-similarity", "block-scanning-mode", "build", @@ -371,6 +377,7 @@ "building-the-go-library", "buildtool", "buildversion", + "built-in-len-method", "buttonColorMode", "calculate_checksum", "certificate", @@ -408,6 +415,7 @@ "crc32string", "data-consistency-in-overlapping-blocks", "define_globalidentifier-value", + "defining-external-variables", "deps", "deserialize_fromfile", "deviationoffset-size-mean", @@ -509,10 +517,8 @@ "has_importimport", "has_rpathrpath", "header", - "heading", "hexinteger", "hexmessage-integer", - "higher-overall-performance", "how-it-works", "how-yara-x-improves-on-yara", "identifier", @@ -621,6 +627,7 @@ "protoitem", "query", "reading-data-at-a-given-offset", + "real-world-benchmarks", "referencing-other-rules", "remote-paths", "resource", @@ -679,6 +686,7 @@ "stream", "stricter-escaped-characters-in-regular-expressions", "string-literals", + "struct-variables", "subsystem", "sym", "symbind",