Merge pull request #202 from mitre-attack/detection_strategies_fix

Detection strategies fixes

Author: jondricek

LICENSE.txt

@@ -186,7 +186,7 @@
       same "printed page" as the copyright notice for easier
       identification within third-party archives.
 
-   Copyright [yyyy] [name of copyright owner]
+   Copyright [2025] [The MITRE Corporation]
 
    Licensed under the Apache License, Version 2.0 (the "License");
    you may not use this file except in compliance with the License.

README.md

@@ -1,7 +1,7 @@
 # mitreattack-python
 
-This repository contains a library of Python tools and utilities for working with ATT&CK data. For more information,
-see the [full documentation](https://mitreattack-python.readthedocs.io/) on ReadTheDocs.
+This repository contains a library of Python tools and utilities for working with ATT&CK data.
+For more information, see the [full documentation](https://mitreattack-python.readthedocs.io/) on ReadTheDocs.
 
 ## Install
 
@@ -11,26 +11,12 @@ To use this package, install the mitreattack-python library with [pip](https://p
 pip install mitreattack-python
 ```
 
-Note: the library requires [python3](https://www.python.org/).
-
 ## MitreAttackData Library
 
-The ``MitreAttackData`` library is used to read in and work with MITRE ATT&CK STIX 2.0 content. This library provides 
+The ``MitreAttackData`` library is used to read in and work with MITRE ATT&CK STIX 2.0 content. This library provides
 the ability to query the dataset for objects and their related objects. This is the main content of mitreattack-python;
 you can read more about other modules in this library under "Additional Modules".
 
-## Additional Modules
-
-More detailed information and examples about the specific usage of the additional modules in this package can be found in the individual README files for each module linked below.
-
-| module | description | documentation |
-|:------------|:------------|:--------------|
-| [navlayers](https://github.com/mitre-attack/mitreattack-python/tree/main/mitreattack/navlayers) | A collection of utilities for working with [ATT&CK Navigator](https://github.com/mitre-attack/attack-navigator) layers. Provides the ability to import, export, and manipulate layers. Layers can be read in from the filesystem or python dictionaries, combined and edited, and then exported to excel or SVG images. | Further documentation can be found [here](https://github.com/mitre-attack/mitreattack-python/blob/main/mitreattack/navlayers/README.md).|
-| [attackToExcel](https://github.com/mitre-attack/mitreattack-python/tree/main/mitreattack/attackToExcel) | A collection of utilities for converting [ATT&CK STIX data](https://github.com/mitre/cti) to Excel spreadsheets. It also provides access to [Pandas](https://pandas.pydata.org/) DataFrames representing the dataset for use in data analysis. | Further documentation can be found [here](https://github.com/mitre-attack/mitreattack-python/blob/main/mitreattack/attackToExcel/README.md).|
-| [collections](https://github.com/mitre-attack/mitreattack-python/tree/main/mitreattack/collections) | A set of utilities for working with [ATT&CK Collections and Collection Indexes](https://github.com/center-for-threat-informed-defense/attack-workbench-frontend/blob/main/docs/collections.md). Provides functionalities for converting and summarizing data in collections and collection indexes, as well as generating a collection from a raw stix bundle input. | Further documentation can be found [here](https://github.com/mitre-attack/mitreattack-python/blob/main/mitreattack/collections/README.md).|
-| [diffStix](https://github.com/mitre-attack/mitreattack-python/tree/main/mitreattack/diffStix) | Create markdown, HTML, JSON and/or ATT&CK Navigator layers reporting on the changes between two versions of the STIX2 bundles representing the ATT&CK content. Run `diff_stix -h` for full usage instructions. | Further documentation can be found [here](https://github.com/mitre-attack/mitreattack-python/blob/main/mitreattack/diffStix/README.md).|
-
-
 ## Related MITRE Work
 
 ### CTI
@@ -60,36 +46,7 @@ STIX is designed to improve many capabilities, such as collaborative threat anal
 
 <https://oasis-open.github.io/cti-documentation/>
 
-### ATT&CK scripts
-
-One-off scripts and code examples you can use as inspiration for how to work with ATT&CK programmatically. Many of the functionalities found in the mitreattack-python package were originally posted on attack-scripts.
-
-<https://github.com/mitre-attack/attack-scripts>
-
-
 ## Contributing
 
 To contribute to this project, either through a bug report, feature request, or merge request,
 please see the [Contributors Guide](https://github.com/mitre-attack/mitreattack-python/blob/main/docs/CONTRIBUTING.md).
-
-## Notice
-
-Copyright 2025 The MITRE Corporation
-
-Approved for Public Release; Distribution Unlimited. Case Number 19-0486.
-
-Licensed under the Apache License, Version 2.0 (the "License");
-you may not use this file except in compliance with the License.
-You may obtain a copy of the License at
-
-   <http://www.apache.org/licenses/LICENSE-2.0>
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-
-This project makes use of ATT&CK®
-
-[ATT&CK Terms of Use](https://attack.mitre.org/resources/terms-of-use/)

examples/.env.example

@@ -0,0 +1,16 @@
+# These environment variables can be helpful when running scripts in the examples directory
+#
+# In order to use them, copy this file to .env and modify the values as needed
+# Two optional tools worth considering to automatically use the .env file are:
+# 1. `python-dotenv` python library: https://github.com/theskumar/python-dotenv
+# 2. `direnv` tool: https://direnv.net
+# Setting up these tools is out of scope for this example.
+#
+# if you have mitreattack-python installed, you can use the following command to download all STIX bundles:
+#
+#     download_attack_stix --all
+#
+# the default download directory from the above command is "attack-releases"
+
+STIX_BASE_DIR=attack-releases/stix-2.0/v17.1
+STIX_BUNDLE=attack-releases/stix-2.0/v17.1/enterprise-attack.json

examples/README.md

@@ -0,0 +1,79 @@
+# Examples Directory
+
+This directory contains example scripts demonstrating how to use the [`mitreattack-python`](https://github.com/mitre-attack/mitreattack-python)
+library to extract, analyze, and report on MITRE ATT&CK data.
+These scripts cover a variety of use cases, including querying STIX bundles, generating reports, and automating ATT&CK data analysis.
+
+## Full Example Listing & Documentation
+
+A complete, categorized list of example scripts, usage details, and direct links is maintained in the built documentation:
+
+- [mitreattack-python Examples Documentation](https://mitreattack-python.readthedocs.io/en/latest/mitre_attack_data/examples.html)
+
+## Setup
+
+Many example scripts allow optional configuration via environment variables for paths to STIX bundles.
+If you want to set this up you can follow these instructions.
+
+- Copy the provided [`examples/.env.example`](examples/.env.example:1) file to `.env`:
+
+  ```sh
+  cp .env.example .env
+  ```
+
+- Edit `.env` to set the correct paths and variables for your environment.
+
+Creating a .env file is not enough however. You will need to use a tool such as the following to help manage the environment variables:
+
+- [`python-dotenv`](https://pypi.org/project/python-dotenv/) (automatically loads `.env` in Python scripts)
+- [`direnv`](https://direnv.net/) (manages environment variables per directory)
+
+Setting up these tools is out of scope for this README.
+
+### Dependencies
+
+- [`mitreattack-python`](https://github.com/mitre-attack/mitreattack-python)
+- Python 3.x
+- ATT&CK STIX bundles
+
+### Downloading ATT&CK STIX Bundles
+
+Many example scripts require ATT&CK STIX bundles, which must be downloaded and placed in the directory specified in your `.env` file (e.g., `attack-releases/stix-2.0/v17.1`).
+You can download these bundles using the provided CLI command if you have mitreattack-python installed:
+
+```sh
+download_attack_stix --all
+```
+
+This will download all available ATT&CK releases in STIX format to the default directory (`attack-releases`).
+You can customize the download location and versions using additional options. For example:
+
+- Download the latest release (default):
+
+  ```sh
+  download_attack_stix
+  ```
+
+- Download specific versions:
+
+  ```sh
+  download_attack_stix -v 16.1 -v 17.1
+  ```
+
+- Download all releases in both STIX formats:
+
+  ```sh
+  download_attack_stix --all --stix21
+  ```
+
+## How to Run Scripts
+
+- Run individual scripts with Python:
+
+  ```sh
+  python get_all_techniques.py
+  ```
+
+## Contribution & Customization
+
+Feel free to adapt these scripts for your own use cases. Contributions and improvements are welcome!

examples/analytic_extractor.py

@@ -11,7 +11,7 @@
 
 # Prune an input string to remove any non-analytic text
 # This assumes that all analytics start the <h4> html block
-STRING_RE = "Analytic(\s|.)*"
+STRING_RE = r"Analytic(\s|.)*"
 
 
 def pruneString(in_string):

examples/attack-object-counter.py

@@ -0,0 +1,186 @@
+"""Print comprehensive ATT&CK statistics across all domains."""
+
+import os
+from dataclasses import dataclass
+
+from mitreattack.stix20 import MitreAttackData
+
+# Get STIX base directory from environment or use default
+STIX_BASE_DIR = os.environ.get("STIX_BASE_DIR", "attack-releases/stix-2.0/v17.1")
+
+
+@dataclass
+class DomainStatistics:
+    """Statistics for a single ATT&CK domain."""
+
+    name: str
+    tactics: int
+    techniques: int
+    subtechniques: int
+    groups: int
+    software: int
+    campaigns: int
+    mitigations: int
+    datasources: int
+    assets: int = 0
+
+    def format_output(self) -> str:
+        """
+        Format domain statistics as a string.
+
+        Returns
+        -------
+        str
+            Formatted statistics string for display.
+        """
+        # Define all possible statistics with their labels
+        stats = [
+            (self.tactics, "Tactics"),
+            (self.techniques, "Techniques"),
+            (self.subtechniques, "Sub-Techniques"),
+            (self.groups, "Groups"),
+            (self.software, "Pieces of Software"),
+            (self.campaigns, "Campaigns"),
+            (self.mitigations, "Mitigations"),
+            (self.assets, "Assets"),
+            (self.datasources, "Data Sources"),
+        ]
+
+        # Build parts list, only including items with count > 0
+        parts = [f"{count} {label}" for count, label in stats if count > 0]
+
+        # Join all parts with proper formatting
+        return f"- {self.name}: {', '.join(parts[:-1])}, and {parts[-1]}"
+
+
+def load_domain_data() -> dict[str, MitreAttackData]:
+    """
+    Load STIX data for all ATT&CK domains.
+
+    Returns
+    -------
+    dict of str to MitreAttackData
+        Mapping of domain names to loaded MitreAttackData objects.
+    """
+    domains = {
+        "enterprise": "enterprise-attack.json",
+        "mobile": "mobile-attack.json",
+        "ics": "ics-attack.json",
+    }
+
+    return {
+        domain: MitreAttackData(stix_filepath=os.path.join(STIX_BASE_DIR, filename))
+        for domain, filename in domains.items()
+    }
+
+
+def collect_domain_statistics(data: MitreAttackData, domain_name: str) -> DomainStatistics:
+    """
+    Collect statistics for a single domain.
+
+    Parameters
+    ----------
+    data : MitreAttackData
+        The MitreAttackData object for the domain.
+    domain_name : str
+        Display name of the domain.
+
+    Returns
+    -------
+    DomainStatistics
+        Statistics for the domain.
+    """
+    # Get all object types, removing revoked and deprecated
+    tactics = data.get_tactics(remove_revoked_deprecated=True)
+    techniques = data.get_techniques(include_subtechniques=False, remove_revoked_deprecated=True)
+    subtechniques = data.get_subtechniques(remove_revoked_deprecated=True)
+    groups = data.get_groups(remove_revoked_deprecated=True)
+    software = data.get_software(remove_revoked_deprecated=True)
+    campaigns = data.get_campaigns(remove_revoked_deprecated=True)
+    mitigations = data.get_mitigations(remove_revoked_deprecated=True)
+    datasources = data.get_datasources(remove_revoked_deprecated=True)
+
+    # ICS domain has assets
+    assets = 0
+    if domain_name == "ICS":
+        assets = len(data.get_assets(remove_revoked_deprecated=True))
+
+    return DomainStatistics(
+        name=domain_name,
+        tactics=len(tactics),
+        techniques=len(techniques),
+        subtechniques=len(subtechniques),
+        groups=len(groups),
+        software=len(software),
+        campaigns=len(campaigns),
+        mitigations=len(mitigations),
+        datasources=len(datasources),
+        assets=assets,
+    )
+
+
+def collect_unique_object_counts(domain_data: dict[str, MitreAttackData]) -> dict[str, int]:
+    """
+    Collect counts of unique objects across all domains.
+
+    Some objects (Software, Groups, Campaigns) may appear in multiple domains.
+    This function counts unique objects to avoid double-counting.
+
+    Parameters
+    ----------
+    domain_data : dict of str to MitreAttackData
+        Mapping of domain names to MitreAttackData objects.
+
+    Returns
+    -------
+    dict of str to int
+        Counts of unique software, groups, and campaigns.
+    """
+    all_software_ids = set()
+    all_groups_ids = set()
+    all_campaigns_ids = set()
+
+    for data in domain_data.values():
+        software = data.get_software(remove_revoked_deprecated=True)
+        groups = data.get_groups(remove_revoked_deprecated=True)
+        campaigns = data.get_campaigns(remove_revoked_deprecated=True)
+
+        all_software_ids.update(obj["id"] for obj in software)
+        all_groups_ids.update(obj["id"] for obj in groups)
+        all_campaigns_ids.update(obj["id"] for obj in campaigns)
+
+    return {
+        "software": len(all_software_ids),
+        "groups": len(all_groups_ids),
+        "campaigns": len(all_campaigns_ids),
+    }
+
+
+def main():
+    """Print ATT&CK statistics for all domains."""
+    # Load data for all domains
+    domain_data = load_domain_data()
+
+    # Collect unique object counts across all domains
+    unique_counts = collect_unique_object_counts(domain_data)
+
+    # Collect statistics for each domain
+    enterprise_stats = collect_domain_statistics(domain_data["enterprise"], "Enterprise")
+    mobile_stats = collect_domain_statistics(domain_data["mobile"], "Mobile")
+    ics_stats = collect_domain_statistics(domain_data["ics"], "ICS")
+
+    # Print summary output
+    print(
+        f"This version of ATT&CK contains {unique_counts['software']} Pieces of Software, "
+        f"{unique_counts['groups']} Groups, and {unique_counts['campaigns']} Campaigns"
+    )
+    print("Broken out by domain:\n")
+
+    # Print domain statistics
+    print(enterprise_stats.format_output())
+    print(mobile_stats.format_output())
+    print(ics_stats.format_output())
+
+
+if __name__ == "__main__":
+    main()