Merge remote-tracking branch 'origin/main' into codex-prepare-xdist-setup

# Conflicts:
#	.github/workflows/lint-and-test.yml
#	docs/CONTRIBUTING.md
#	justfile
#	tests/changelog/conftest.py

Author: scotluns

.github/workflows/lint-and-test.yml

@@ -47,4 +47,4 @@ jobs:
         run: uv sync --all-extras
 
       - name: Run pytest
-        run: uv run --extra dev pytest -n 2 --cov=mitreattack
+        run: uv run --extra dev pytest -n 2 --cov=mitreattack --durations=20

.github/workflows/release-and-publish.yml

@@ -56,7 +56,7 @@ jobs:
         id: app-token
         uses: actions/create-github-app-token@v3
         with:
-          app-id: ${{ vars.ATTACK_AUTOBOT_APP_ID }}
+          client-id: ${{ vars.ATTACK_AUTOBOT_CLIENT_ID }}
           private-key: ${{ secrets.ATTACK_AUTOBOT_PRIVATE_KEY }}
 
       # Note: We checkout the repository at the branch that triggered the workflow.

docs/CONTRIBUTING.md

@@ -51,13 +51,16 @@ Run `just` with no arguments to see all available commands. Here are the most co
 
 ```bash
 just lint          # Run pre-commit hooks (ruff format) on all files
-just test          # Run tests
+just test          # Run the full test suite, matching CI expectations
+just test-fast     # Run the fast local subset, excluding integration and slow tests
 just test-xdist    # Run tests in parallel
 just test-cov      # Run tests with coverage report
 just test-cov-xdist  # Run tests with coverage in parallel
 just build         # Build the package
 ```
 
+Use `just test-fast` while iterating locally on changes that do not need full STIX-backed export or other slow integration coverage. Tests or setup steps that normally take longer than 10 seconds should be marked `slow`, so they are skipped by `just test-fast`. Before opening a PR, run `just test`; GitHub Actions also runs the full suite with coverage.
+
 Tests that need real ATT&CK STIX data should use the shared STIX fixtures instead of downloading or
 preparing bundles directly. Parallel test runs warm the shared STIX cache before workers start; if a
 new xdist-backed test needs an additional ATT&CK release, update the cache warmup list in

examples/generate_excel_files.py

@@ -1,19 +1,99 @@
-from mitreattack.attackToExcel import attackToExcel
+"""Generate ATT&CK Excel exports from local STIX bundles."""
+
+import argparse
+from os import environ
+from pathlib import Path
+
 from stix2 import MemoryStore
-import os
 
-def main():
+from mitreattack.attackToExcel import attackToExcel
+
+# Pass attack version via the command line or update the variable below
+DEFAULT_ATTACK_VERSION = "v19.0"
+# Parent directory where ATT&CK version export folders are written.
+OUTPUT_DIR = Path("output")
+# Set to true if you want the parent subfolder of the excel files to have a version.
+# Example - If you want the folder to be named enterprise-attack-v19.0 instead of enterprise-attack, set to True
+VERSIONED_OUTPUT_DIR = False
+
+
+def move_versioned_exports_to_domain_dir(output_dir, domain, version):
+    """Move versioned Excel exports into the unversioned domain folder."""
+    output_dir = Path(output_dir)
+    versioned_dir = output_dir / f"{domain}-{version}"
+    domain_dir = output_dir / domain
+
+    if not versioned_dir.is_dir():
+        return
+
+    domain_dir.mkdir(parents=True, exist_ok=True)
+
+    for source_path in versioned_dir.iterdir():
+        if not source_path.is_file():
+            continue
+
+        target_path = domain_dir / source_path.name
+        if target_path.exists():
+            target_path.unlink()
+
+        source_path.replace(target_path)
+
+    versioned_dir.rmdir()
+
+
+def format_missing_stix_bundle_error(stix_file, attack_version):
+    """Format a concise missing STIX bundle error."""
+    message = (
+        f"STIX bundle not found: {stix_file}\n"
+        "Download the STIX bundles before running this script, or set STIX_BASE_DIR to the directory containing "
+        "enterprise-attack.json, mobile-attack.json, and ics-attack.json."
+    )
+
+    if attack_version and not attack_version.startswith("v"):
+        message = f"{message}\nDid you mean -a v{attack_version}?"
+
+    return message
+
+
+def validate_stix_files(stix_files, attack_version):
+    """Exit with a clean error if any expected STIX bundle is missing."""
+    for stix_file in stix_files.values():
+        if not stix_file.is_file():
+            raise SystemExit(format_missing_stix_bundle_error(stix_file, attack_version))
+
+
+def parse_args(argv=None):
+    """Parse command line arguments."""
+    parser = argparse.ArgumentParser(
+        prog="generate_excel_files.py",
+        description="Generate ATT&CK Excel exports from local STIX bundles.",
+    )
+    parser.add_argument(
+        "-a",
+        "--attack-version",
+        default=DEFAULT_ATTACK_VERSION,
+        help=(f"ATT&CK version to export, such as v19.0. Defaults to {DEFAULT_ATTACK_VERSION}."),
+    )
+    return parser.parse_args(args=argv)
+
+
+def main(argv=None):
+    """Generate excel files for specific versions of ATT&CK."""
+    args = parse_args(argv)
+    attack_version = args.attack_version
+
     # List of domains and version to process
     domains = ["enterprise-attack", "mobile-attack", "ics-attack"]
-    output_dir = "output/"
+    output_dir = OUTPUT_DIR / attack_version
 
     # Path to the STIX bundles for each domain (assumes STIX files are downloaded)
-    stix_base_dir = os.environ.get("STIX_BASE_DIR", "attack-releases/stix-2.0/v18.0")
+    stix_base_dir = Path(environ.get("STIX_BASE_DIR", Path("attack-releases") / "stix-2.0" / attack_version))
     stix_files = {
-        "enterprise-attack": os.path.join(stix_base_dir, "enterprise-attack.json"),
-        "mobile-attack": os.path.join(stix_base_dir, "mobile-attack.json"),
-        "ics-attack": os.path.join(stix_base_dir, "ics-attack.json"),
+        "enterprise-attack": stix_base_dir / "enterprise-attack.json",
+        "mobile-attack": stix_base_dir / "mobile-attack.json",
+        "ics-attack": stix_base_dir / "ics-attack.json",
     }
+    validate_stix_files(stix_files, attack_version)
 
     for domain in domains:
         stix_file = stix_files[domain]
@@ -26,9 +106,14 @@ def main():
         # Export to Excel
         attackToExcel.export(
             domain=domain,
+            version=attack_version,
             output_dir=output_dir,
             mem_store=mem_store,
         )
 
+        if attack_version and not VERSIONED_OUTPUT_DIR:
+            move_versioned_exports_to_domain_dir(output_dir=output_dir, domain=domain, version=attack_version)
+
+
 if __name__ == "__main__":
     main()

examples/generate_multiple_attack_diffs.py

@@ -2,7 +2,7 @@
 
 import argparse
 
-from mitreattack.diffStix.changelog_helper import get_new_changelog_md
+from mitreattack.diffStix.attack_changelog import generate_attack_changelog
 
 DOMAINS = ["enterprise-attack", "mobile-attack", "ics-attack"]
 VERSION_PAIRS = [
@@ -11,17 +11,6 @@
 ]
 
 
-def get_release_output_folder(old_version: str, new_version: str) -> str:
-    """Return the output folder for a release comparison."""
-    return f"output/v{old_version}-v{new_version}"
-
-
-def get_artifact_link_prefix(old_version: str, new_version: str, *, attack_website_links: bool = False) -> str:
-    """Return the link prefix for generated layers and changelog JSON."""
-    if not attack_website_links:
-        return ""
-    return f"/docs/changelogs/v{old_version}-v{new_version}"
-
 
 def get_parsed_args():
     """Parse command line arguments for the example script."""
@@ -37,31 +26,18 @@ def get_parsed_args():
 
 def generate_diff(old_version: str, new_version: str, *, attack_website_links: bool = False):
     """Generate changelog outputs for a single ATT&CK release pair."""
-    output_folder = get_release_output_folder(old_version, new_version)
+    output_folder = f"output/v{old_version}-v{new_version}"
     print(f"Generating ATT&CK Diffs between {old_version}-{new_version}: {output_folder}")
 
-    get_new_changelog_md(
+    generate_attack_changelog(
+        old_version=old_version,
+        new_version=new_version,
         domains=DOMAINS,
-        layers=[
-            f"{output_folder}/layer-enterprise.json",
-            f"{output_folder}/layer-mobile.json",
-            f"{output_folder}/layer-ics.json",
-        ],
-        old=f"attack-releases/stix-2.0/v{old_version}",
-        new=f"attack-releases/stix-2.0/v{new_version}",
-        show_key=True,
-        # site_prefix: str = "",
+        output_dir=output_folder,
         verbose=True,
-        include_contributors=True,
-        markdown_file=f"{output_folder}/changelog.md",
-        html_file=f"{output_folder}/index.html",
-        html_file_detailed=f"{output_folder}/changelog-detailed.html",
-        additional_formats_prefix=get_artifact_link_prefix(
-            old_version,
-            new_version,
-            attack_website_links=attack_website_links,
-        ),
-        json_file=f"{output_folder}/changelog.json",
+        markdown_file=True,
+        html_file=True,
+        attack_website_links=attack_website_links,
     )
 
 

justfile

@@ -35,6 +35,10 @@ ruff-format:
 test:
     uv run pytest
 
+# Run the fast local test subset, excluding integration and slow tests
+test-fast:
+    uv run pytest -m "not integration and not slow"
+
 # Run tests in parallel
 test-xdist workers="auto":
     uv run --extra dev pytest -n {{ workers }}

mitreattack/attackToExcel/README.md

@@ -10,19 +10,40 @@ It also provides a means to access ATT&CK data as [Pandas](https://pandas.pydata
 Print full usage instructions:
 
 ```shell
-python3 attackToExcel.py -h
+attack-to-excel --help
 ```
 
 Example execution:
 
 ```shell
-python3 attackToExcel.py
+attack-to-excel from-stix
 ```
 
 Build a excel files corresponding to a specific domain and version of ATT&CK:
 
 ```shell
-python3 attackToExcel -domain mobile-attack -version v5.0
+attack-to-excel from-stix --domain mobile-attack --version v5.0
+```
+
+Build Excel files for all ATT&CK domains from a release. If local STIX files
+are missing under `attack-releases/stix-2.0/v19.0`, they are downloaded
+temporarily for the export:
+
+```shell
+attack-to-excel from-release --version v19.0
+```
+
+To persist release STIX files before exporting, use `download_attack_stix`:
+
+```shell
+download_attack_stix -v 19.0
+attack-to-excel from-release --version v19.0
+```
+
+Build Excel files for selected ATT&CK domains from a release:
+
+```shell
+attack-to-excel from-release --version v19.0 --domains mobile-attack --domains ics-attack
 ```
 
 ### Module
@@ -35,6 +56,14 @@ import mitreattack.attackToExcel.attackToExcel as attackToExcel
 attackToExcel.export("mobile-attack", "v5.0", "/path/to/export/folder")
 ```
 
+Example execution targeting all release domains:
+
+```python
+import mitreattack.attackToExcel.attackToExcel as attackToExcel
+
+attackToExcel.export_release(version="v19.0", output_dir="output")
+```
+
 ## Interfaces
 
 ### attackToExcel
@@ -48,6 +77,7 @@ overview of the available methods follows.
 |build_dataframes| `src`: MemoryStore or other stix2 DataSource object holding domain data<br> `domain`: domain of ATT&CK that `src` corresponds to| Builds a Pandas DataFrame collection as a dictionary, with keys for each type, based on the ATT&CK data provided|
 |write_excel| `dataframes`: pandas DataFrame dictionary (generated by build_dataframes) <br>  `domain`: domain of ATT&CK that `dataframes` corresponds to <br> `version`: optional parameter indicating which version of ATT&CK is in use <br> `output_dir`: optional parameter specifying output directory| Writes out DataFrame based ATT&CK data to excel files|
 |export| `domain`: the domain of ATT&CK to download <br> `version`: optional parameter specifying which version of ATT&CK to download <br> `output_dir`: optional parameter specifying output directory| Downloads ATT&CK data from MITRE/CTI and exports it to Excel spreadsheets |
+|export_release| `version`: optional ATT&CK release version <br> `stix_version`: STIX release tree, such as "2.0" or "2.1" <br> `output_dir`: parent output directory <br> `stix_base_dir`: optional directory containing release STIX files <br> `domains`: optional list of domains <br> `versioned_output_dir`: preserve domain-version output folders| Exports a full ATT&CK release to Excel spreadsheets, downloading missing STIX files temporarily when needed |
 
 ### stixToDf