|
| 1 | +--- |
| 2 | +title: "YARA-X just got faster" |
| 3 | +description: "Explore recent performance improvements in YARA-X, including daachorse Aho-Corasick integration and RegexSet evaluation optimizations." |
| 4 | +summary: "" |
| 5 | +date: 2026-05-13T13:20:00+02:00 |
| 6 | +lastmod: 2026-05-13T13:20:00+02:00 |
| 7 | +draft: false |
| 8 | +weight: 50 |
| 9 | +categories: [] |
| 10 | +tags: [] |
| 11 | +contributors: [ "Victor M. Alvarez" ] |
| 12 | +pinned: false |
| 13 | +homepage: false |
| 14 | +seo: |
| 15 | + title: "YARA-X just got faster" |
| 16 | + description: "Explore recent performance improvements in YARA-X, including daachorse Aho-Corasick integration and RegexSet evaluation optimizations." |
| 17 | + canonical: "" |
| 18 | + noindex: false |
| 19 | +--- |
| 20 | +Almost a year ago, we published a blog post titled [YARA-X just got |
| 21 | +smarter]({{< ref "blog/yara-x-just-got-smarter/index.md" >}}). It described |
| 22 | +some improvements aimed at making life easier for our users. Today, we are |
| 23 | +publishing a similar entry, but this time it focuses on performance |
| 24 | +improvements rather than usability. |
| 25 | + |
| 26 | +While YARA-X has consistently offered robust performance, two major recent |
| 27 | +changes—[PR #654](https://github.com/VirusTotal/yara-x/pull/654) and |
| 28 | +[PR #657](https://github.com/VirusTotal/yara-x/pull/657)—introduce |
| 29 | +groundbreaking optimizations to core pattern matching and rules that make heavy |
| 30 | +use of the `matches` operator. |
| 31 | + |
| 32 | +Here is a deep dive into how these performance gains were achieved. |
| 33 | + |
| 34 | + |
| 35 | +## 1. Replacing aho_corasick with daachorse |
| 36 | + |
| 37 | +When searching for hundreds of thousands of byte strings simultaneously, |
| 38 | +YARA-X relies on the Aho-Corasick algorithm. This is the primary workhorse |
| 39 | +of the YARA-X scanning engine. |
| 40 | + |
| 41 | +Previously, YARA-X utilized the widely used |
| 42 | +[`aho_corasick`](https://github.com/BurntSushi/aho-corasick) crate, a robust |
| 43 | +and performant implementation of the Aho-Corasick algorithm in Rust. However, |
| 44 | +[PR #654](https://github.com/VirusTotal/yara-x/pull/654) replaces `aho_corasick` |
| 45 | +with [`daachorse`](https://github.com/daac-tools/daachorse), a much faster |
| 46 | +implementation built around a compact double-array trie data structure. |
| 47 | + |
| 48 | +Conceptually, the implementation in `daachorse` mirrors the core design |
| 49 | +principles of the original C-based YARA engine. Its performance gains over the |
| 50 | +standard `aho_corasick` crate stem from two primary architectural advantages: |
| 51 | + |
| 52 | +1. **Tighter inner loop**: Byte-by-byte traversal through NFA transitions |
| 53 | + requires fewer memory indirections and branching instructions. |
| 54 | + |
| 55 | +2. **Cache-efficient storage**: The double-array automaton layout optimizes |
| 56 | + memory locality, drastically reducing CPU cache misses during intensive |
| 57 | + scanning over large files. |
| 58 | + |
| 59 | +Migrating to `daachorse` had its own challenges, though. To facilitate a |
| 60 | +seamless transition and ensure YARA-X's matching semantics were fully |
| 61 | +preserved, upstream contributions were made to `daachorse` to align its |
| 62 | +API closer to the familiar `aho_corasick` crate. Most notably, |
| 63 | +[PR #128](https://github.com/daac-tools/daachorse/pull/128) added |
| 64 | +support for Aho-Corasick automatons with duplicate patterns or no patterns |
| 65 | +at all. |
| 66 | + |
| 67 | +#### Real-world benchmarks |
| 68 | + |
| 69 | +Using the full rule set from [YARA Forge](https://yarahq.github.io/) across |
| 70 | +a corpus of 500 random VirusTotal files (totaling ~24GB), `daachorse` achieved |
| 71 | +remarkable benchmark results: |
| 72 | + |
| 73 | +| Scanner | Scan Time | Speedup | |
| 74 | +|---|---|---| |
| 75 | +| YARA-X (daachorse) | 26.152s | — | |
| 76 | +| YARA-X (aho_corasick) | 57.584s | YARA-X with daachorse is 2.20x faster | |
| 77 | +| YARA (legacy) | 48.275s | YARA-X with daachorse is 1.85x faster | |
| 78 | + |
| 79 | +This brings YARA-X up to speed with legacy YARA in terms of performance. Now, |
| 80 | +we can guarantee that **YARA-X is faster than YARA** in almost all cases, with |
| 81 | +very few exceptions. |
| 82 | + |
| 83 | +Starting with YARA-X v1.16.0, you can enjoy this improvement. |
| 84 | + |
| 85 | +## 2. Simultaneous regex evaluation |
| 86 | + |
| 87 | +When authoring complex YARA rules, analysts frequently evaluate a single |
| 88 | +variable, structure field access, or module output against dozens of distinct |
| 89 | +regular expressions inside an `or` condition. A common example we had |
| 90 | +observed in VirusTotal is inspecting domain names or URLs: |
| 91 | + |
| 92 | +```yara |
| 93 | +condition: |
| 94 | + vt.net.domain.raw matches /evil-c2\.com/ or |
| 95 | + vt.net.domain.raw matches /phishing-login\.net/ or |
| 96 | + vt.net.domain.raw matches /malicious-dga[0-9]{4}\.org/ |
| 97 | +``` |
| 98 | + |
| 99 | +The example above is only for illustrating the point; in real life, we have seen |
| 100 | +rules with hundreds of `matches` operations like these. |
| 101 | + |
| 102 | +Previously, YARA-X evaluated each `matches` expression sequentially. If a rule |
| 103 | +contained fifty regex checks against the same field, the scanning engine would |
| 104 | +execute fifty separate passes over the target string. |
| 105 | + |
| 106 | +[PR #657](https://github.com/VirusTotal/yara-x/pull/657) introduces an advanced |
| 107 | +compiler optimization that detects identical match targets within `or` |
| 108 | +conditions and automatically groups their regular expressions into a unified |
| 109 | +[`RegexSet`](https://docs.rs/regex/latest/regex/struct.RegexSet.html). |
| 110 | + |
| 111 | +A `RegexSet` is a specialized feature of the standard Rust |
| 112 | +[`regex`](https://github.com/rust-lang/regex) crate designed precisely to speed |
| 113 | +up the matching of a single string against multiple regular expressions at the |
| 114 | +same time. |
| 115 | + |
| 116 | +When regular expressions are evaluated sequentially, the scanning engine incurs |
| 117 | +repeated overhead: it must re-initialize its search state and scan the entire |
| 118 | +target string from beginning to end for every single pattern. As rule |
| 119 | +complexity grows, this redundant scanning becomes a major performance |
| 120 | +bottleneck. |
| 121 | + |
| 122 | +By grouping these expressions into a `RegexSet`, the compiler combines all |
| 123 | +individual regular expressions into a single, unified state machine |
| 124 | +(automaton). When a target string is inspected, the scanning engine executes |
| 125 | +exactly **one scan pass** over the data. As it steps through each character in |
| 126 | +the target string, the unified state machine simultaneously tracks and evaluates |
| 127 | +potential matches for all grouped regex patterns at once. |
| 128 | + |
| 129 | +This optimization shifts the scanning complexity from scaling linearly with the |
| 130 | +number of regex patterns to a single, highly efficient pass over the input |
| 131 | +string, delivering a huge speed improvement when executing complex rule sets. |
| 132 | + |
| 133 | +This improvement will be released in the upcoming YARA-X v1.17.0. However, we |
| 134 | +have already deployed these changes to VirusTotal Livehunt and observed a |
| 135 | +dramatic drop in scanning time for domain-related rules. The 99th percentile |
| 136 | +execution time was cut in half, dropping from an average of over 6 seconds to |
| 137 | +less than 3 seconds. In other words, 99% of rule evaluations now complete in |
| 138 | +under 3 seconds, whereas they previously exceeded 6 seconds. Additionally, the CPU |
| 139 | +costs associated to running this workload was also cut in half. |
| 140 | + |
| 141 | +{{< figure src="graph.png" caption="Scanning time for domain-related rules in VirusTotal Livehunt. Notice the abrupt drop on May 11." >}} |
| 142 | + |
| 143 | + |
0 commit comments