feat: Project TideKit — Transform CoreTide into a modern Python package

[behemothsecurity]

Project TideKit

Transform CoreTide from a git submodule into opentide — a versioned Python package on PyPI with a CLI, MCP server, and clean API surface.

🧭 Epic / tracking issue. This is the source of truth for the TideKit transformation. The work is split into 10 phases (#61–#71), each a self-contained, agent-ready brief. Read this epic first, then pick up the lowest-numbered unblocked phase.

---

🤖 Agent Fleet Execution Model

This programme is designed to be executed by an autonomous agentic fleet. Every phase issue carries an Agent Execution Contract, a Context Primer (file paths verified against the codebase), explicit Guardrails, and copy-pasteable Verification commands. The rules below govern the whole fleet.

Ground rules

• Base branch: development (the repo's default/integration branch). All phase branches fork from it and merge back via PR.
• Branch naming: refactor/tidekit-phase<N>-<slug> or feat/tidekit-phase<N>-<slug> (matches house <type>/<short-description> convention).
• Commits: Conventional Commits. Scope by module (tide, models, deployment, indexing, validation, documentation, framework, mutation, export) or platform (sentinel, splunk, …).
• PRs target development, titled in conventional form, body ends with Closes #<phase-issue>, and follow the repo PR template (Summary / Changes / Backwards Compatibility / Testing / Related Issues).
• One phase = one agent = one PR. Do not bundle phases. Phase 6 and Phase 7 are the only pair that may run concurrently.
• Start gate: an agent may only begin a phase once every "Depends on" phase is merged to development. Re-base on development before starting.

Dependency DAG

#61 Phase 0 ─▶ #62 Phase 1 ─▶ #63 Phase 2 ─▶ #64 Phase 3 ─▶ #65 Phase 4 ─▶ #66 Phase 5 ─┬─▶ #67 Phase 6 ─┐
 (removal)     (decompose)    (rename)        (vocab)         (pydantic)      (package)    │   (CLI)        ├─▶ #69 Phase 8 ─▶ #71 Phase 9
                                                                                           └─▶ #68 Phase 7 ─┘   (PyPI)           (tests/docs)
                                                                                               (MCP)

Phase	Issue	Parallel?	Start gate
0	#61	no	immediately
1	#62	no	#61 merged
2	#63	no	#62 merged
3	#64	no	#63 merged
4	#65	no	#64 merged
5	#66	no	#65 merged
6	#67	with #68	#66 merged
7	#68	with #67	#66 merged
8	#69	no	#67 and #68 merged
9	#71	no	#69 merged

---

Vision

CoreTide today is consumed as a git submodule with sys.path.append() hacks, a 1,546-line monolith entry point (tide.py), no type safety on vocabularies, import-time side effects that load hundreds of objects, and a naming scheme that confuses internal plumbing (TideLoader, TideModels, TideDefinitionsModels) with domain concepts (detection rules, threat vectors, objectives).

Project TideKit transforms this into:

# Clean API surface — unified under OpenTide
from opentide import OpenTide

OpenTide.initialise()
rule = OpenTide.Rules[uuid]
rule.deploy("sentinel")                          # Delegation method — calls platform deployer
rule.validate()                                   # Delegation method — calls validation engine

# Platform-centric — operations on a platform
OpenTide.Platforms.Sentinel.deployer.deploy(uuids)
OpenTide.Platforms.Sentinel.validator.validate(uuids)

# CLI
opentide validate --platform sentinel --all
opentide deploy --platform splunk --dry-run
opentide generate
opentide init ./my-detections

// MCP server for AI-assisted detection engineering
{
  "mcpServers": {
    "opentide": { "command": "opentide-mcp" }
  }
}

# Installation
pip install opentide[sentinel,cli]

---

Architecture — Before & After

Before (Current — verified line counts)

Client Repo
├── CoreTide/              ← git submodule, sys.path hacked
│   ├── Engines/modules/
│   │   ├── tide.py        ← 1,546 lines, 6 classes, import-time initialisation
│   │   ├── models.py      ← 670 lines, mixed enums/configs/models
│   │   ├── deployment.py  ← 1,007 lines, CI + planning + resolvers
│   │   └── plugins.py     ← 134 lines, dynamic imports, bare except, dict registries
│   ├── Orchestration/     ← 7 standalone scripts, no CLI
│   └── Framework/         ← YAML metaschemas (triple-maintenance burden)

# Current: confusing, brittle
import sys, git
sys.path.append(str(git.Repo(".", search_parent_directories=True).working_dir))
from Engines.modules.tide import DataTide, IndexTide, HelperTide
from Engines.modules.models import TideModels, TideConfigs, TideDefinitionsModels
from Engines.modules.plugins import DeployTide

rule = DataTide.Models.mdr[uuid]          # "mdr"? "Models"? raw dict?
typed = DataTide.Models.MDR[uuid]         # capitals = typed? confusing
DeployTide.mdr["sentinel"].deploy(...)    # capability-centric dict lookup
DeployTide.query_validation["sentinel"].validate(...)

After (TideKit)

opentide/                    ← pip install opentide
├── core/                    ← Heart of the engine
│   ├── registry.py          ← OpenTide (singleton, lazy-loaded)
│   ├── index.py             ← IndexManager (lifecycle, staging)
│   ├── environment.py       ← Debug detection, env-var resolution
│   ├── errors.py, logs.py   ← Structured logging, error hierarchy
│   ├── files.py, utilities.py
│   └── loaders/             ← ObjectLoader, PlatformConfigLoader
├── models/                  ← Pydantic models per domain
│   ├── rule.py              ← DetectionRule (was TideModels.MDR)
│   ├── threat.py            ← ThreatVector (was TVM)
│   ├── objective.py         ← DetectionObjective (was DOM)
│   ├── enums.py, metadata.py
│   └── platforms/           ← Per-platform config models
├── platforms/               ← Per-platform packages
│   ├── base.py              ← Platform, PlatformCollection abstractions
│   ├── sentinel/            ← deployer.py, validator.py, connection.py
│   ├── splunk/
│   ├── crowdstrike/
│   ├── defender_for_endpoint/
│   ├── sentinel_one/
│   ├── carbon_black_cloud/
│   └── harfanglab/
├── generation/              ← Schemas, templates, snippets, index building
├── documentation/           ← Wiki generation
├── validation/              ← Schema + query validators
├── mutation/                ← Object mutations
├── export/                  ← ATT&CK Navigator, tables
├── cli/                     ← Typer-based CLI
├── mcp_server/              ← FastMCP server
└── data/                    ← Bundled configs, schemas, vocabularies

# After: clear, typed, semantic
from opentide import OpenTide

rule = OpenTide.Rules[uuid]                       # What engineers create
rule.deploy("sentinel")                            # Deploy to a platform (delegation)
rule.validate()                                    # Validate against schema (delegation)

platform = OpenTide.Platforms.Sentinel             # Where it goes
platform.deployer.deploy(uuids)                    # Batch deployment
platform.config                                    # How it's configured

---

The OpenTide Surface

A single, domain-native API. OpenTide is a registry — not an orchestrator. It provides data access and platform references. Operations are either delegation methods on models, or direct calls to platform engines. The CLI and pipelines handle orchestration (CI context, exit codes, sequencing).

OpenTide
├── .initialise(repo_root?, data_root?)
├── .reload()
├── .lookup(uuid) → object | None         # Cross-type UUID search
│
│  OBJECTS — Pydantic models with delegation methods
│
├── .Rules → dict[uuid, DetectionRule]
├── .Threats → dict[uuid, ThreatVector]
│   └── .chaining → dict                  # Computed threat chain relationships
├── .Objectives → dict[uuid, DetectionObjective]
│
│   DetectionRule (Pydantic — pure data + delegation)
│   ├── .uuid, .name, .status, .severity, .created, .updated
│   ├── .techniques → list[str]
│   ├── .platforms → dict[str, PlatformConfig]
│   ├── .references, .metadata
│   ├── .file → Path                      # Source YAML file
│   ├── .deploy(platform, dry_run?) → DeploymentResult   # Delegates to platform deployer
│   ├── .validate() → ValidationResult                    # Delegates to validation engine
│   ├── .validate_query(platform) → ValidationResult      # Delegates to platform validator
│   ├── .document() → str                                 # Delegates to doc engine
│   ├── .promote(target_status) → None                    # Delegates to mutation engine
│   └── .model_dump() → dict              # Pydantic built-in
│
│  PLATFORMS — Where deployment & validation happen
│
├── .Platforms → PlatformCollection
│   ├── ["sentinel"] → Platform           # Dict-style access
│   ├── .Sentinel → Platform              # Attribute-style access
│   ├── .Splunk, .Crowdstrike, .DefenderForEndpoint, ...
│   ├── .enabled() → list[Platform]
│   └── .__iter__()                       # Iterate enabled platforms
│
│   Platform                               # Composes config + engines
│   ├── .name → str
│   ├── .enabled → bool
│   ├── .config
│   │   ├── .platform → PlatformConfig
│   │   ├── .tenants → list[TenantConfig] | None
│   │   └── .modifiers → list[Modifier] | None
│   ├── .deployer → RuleDeployer | None
│   ├── .validator → QueryValidator | None
│   ├── .can_deploy, .can_validate → bool
│   └── .connect() → connection
│
│  CONFIGURATION — Read-only settings from TOML
│  (Per-platform configs live on Platform.config, not here)
│
├── .Configuration
│   ├── .Index → dict                     # Raw merged config (for template resolution)
│   ├── .Global (.objects, .data_fields, .Paths.Core, .Paths.Content, .Exports)
│   ├── .Deployment (.statuses, .promotion, .proxy)
│   ├── .Documentation (.scope, .wiki, .object_names, .titles, .icons)
│   ├── .Visibility (.assets, .detectors, .logsources)
│   ├── .Resources (.attack, .d3fend, .nist, .engage, .misp)
│   ├── .Schema (.extensions, .vocabulary_overrides)
│   └── .Sharing (.organisation)
│
│  SUPPORTING DATA
│
├── .Vocabularies → dict[name, VocabularyDefinition]
│   └── .resolve(field_path) → list[str]
├── .Schemas (.rule, .threat, .objective → dict)
├── .Templates (.rule, .threat, .objective → str)
│
│  INDEX — Lifecycle & derived data
│
├── .Index
│   ├── .raw → dict
│   ├── .revisions → dict
│   ├── .compiled → dict
│
│  RUNTIME — Split concerns (no monolith "Environment")
│
├── .debug → bool
├── .ci → CIContext | None
└── .root → Path

Design Principles

1. Models are data + delegation: DetectionRule is a Pydantic model (pure data fields). Its .deploy(), .validate(), .document() methods delegate to platform engines and services — the model carries no infrastructure state. This gives interactive/MCP/CLI convenience (rule.deploy("sentinel")) while keeping deployers batch-oriented internally.

2. Single config path: Per-platform configuration lives on Platform.config only. There is no Configuration.Platforms — that would duplicate the source of truth.

3. OpenTide is a registry, not an orchestrator: No top-level .deploy(), .generate(), .validate() methods. The CLI and pipeline scripts handle orchestration (CI context, deployment strategy, sequencing, exit codes). OpenTide provides everything needed for those operations.

4. Platforms are first-class, not "plugins": Engineers think "deploy to Sentinel" — the API reflects that directly. OpenTide.Platforms.Sentinel, not registry["sentinel"].

How Deployment Works

# CLI: opentide deploy --platform sentinel
# ↓ orchestration layer builds plan, handles CI context
plan = make_deploy_plan(strategy)
# ↓ calls platform deployer directly
platform = OpenTide.Platforms.Sentinel
platform.deployer.deploy(plan.uuids)
# ↓ deployer reads from OpenTide.Rules[uuid], connects via platform.connect()
# ↓ deploys to each tenant

# Interactive / MCP: single-rule convenience
rule = OpenTide.Rules[uuid]
rule.deploy("sentinel")  # delegates to OpenTide.Platforms.Sentinel.deployer.deploy([uuid])

---

Platform Capability Matrix (verified)

The 7 detection platforms are not uniformly capable. This matrix is ground truth from Engines/deployment/ and Engines/validation/ and must be respected by Phases 2, 4, 6, 7.

Platform	Deployer (Engines/deployment/)	Query validator (Engines/validation/)	Notes
Sentinel	✅ sentinel.py	✅ sentinel_query.py	KQL
Defender for Endpoint	✅ defender_for_endpoint.py	✅ defender_for_endpoint_query.py	KQL
Splunk	✅ splunk.py	✅ splunk_query.py	SPL
SentinelOne	✅ sentinel_one.py	✅ sentinel_one_query.py	S1QL
Carbon Black Cloud	✅ carbon_black_cloud.py	✅ carbon_black_cloud_query.py	Lucene
CrowdStrike	✅ crowdstrike.py	❌ none	Deploy-only — no query validator exists
HarfangLab	✅ harfanglab.py	❌ none	Deploy-only (Sigma/YARA)

⚠️ 7 deployers, 5 query validators. Any acceptance criterion mentioning "validate query for all platforms" means the 5 validators above. CrowdStrike and HarfangLab are deploy-only until validators are written (out of scope for TideKit).

---

Deprecated & Removed

The following features are confirmed deprecated and will be fully removed:

Feature	Status	Replaced By
CDM (Cyber Detection Models)	Deprecated	DOM (Detection Objectives)
BDR (Business Detection Requests)	Deprecated	—
Lookups (Splunk lookups pipeline)	Deprecated	—
MDRv2 (legacy MDR format)	Dead	MDRv4 typed framework
splunk_lookups.py, splunk_metadata.py	Remove	—
deploy_lookups.py	Remove	—
Tide2Patching (Engines/modules/patching.py)	Remove	Pydantic migration handles versioning

---

Key Design Decisions

1. Package Name: opentide

• PyPI: pip install opentide[sentinel,cli]
• Import: from opentide import OpenTide
• CLI: opentide validate, opentide deploy
• MCP: opentide-mcp

2. Single Package with Optional Extras

[project.optional-dependencies]
sentinel = ["azure-identity", "azure-mgmt-securityinsight"]
splunk = ["splunk-sdk"]
crowdstrike = ["falconpy"]
carbon-black = ["carbon-black-cloud-sdk"]
cli = ["typer[all]"]
mcp = ["mcp[cli]"]
all = ["opentide[sentinel,splunk,crowdstrike,carbon-black,cli,mcp]"]

3. Object Naming — Full English, No Abbreviations

Abbreviation	Full Name	Access
MDR	DetectionRule	OpenTide.Rules[uuid]
TVM	ThreatVector	OpenTide.Threats[uuid]
DOM	DetectionObjective	OpenTide.Objectives[uuid]
CDM	REMOVED	Fully decommissioned
BDR	REMOVED	Fully decommissioned

4. Platform-Centric Access (NOT Plugin Registry)

# Before: capability-centric dict lookup with generic "registry" name
DeployTide.mdr["sentinel"].deploy(...)
DeployTide.query_validation["sentinel"].validate(...)

# After: platforms are first-class on OpenTide
OpenTide.Platforms.Sentinel.deployer.deploy(uuids)
OpenTide.Platforms.Sentinel.validator.validate(uuids)

# Convenience via delegation on models
rule = OpenTide.Rules[uuid]
rule.deploy("sentinel")  # delegates to platform deployer

5. Delegation Methods on Models

Models carry no infrastructure state — but they delegate to engines for convenience:

class DetectionRule(BaseModel):
    # Pure data fields
    uuid: str
    name: str
    status: str
    # ...

    def deploy(self, platform: str, dry_run: bool = False) -> DeploymentResult:
        """Delegate to the platform's deployer."""
        p = self._registry.Platforms[platform]
        return p.deployer.deploy([self.uuid], dry_run=dry_run)

    def validate(self) -> ValidationResult:
        """Delegate to validation engine."""
        return self._registry.validate_rule(self)

This gives interactive/MCP/CLI convenience while keeping batch-oriented deployers unchanged internally.

6. Lazy Initialisation (No Import-Time Side Effects)

# Before: importing DataTide loads ALL objects
from Engines.modules.tide import DataTide  # 💥 hundreds of operations

# After: explicit lifecycle
from opentide import OpenTide
OpenTide.initialise()  # Explicit, controllable, testable

7. CLI Framework: Typer

opentide validate --platform sentinel --all
opentide deploy --platform splunk --dry-run --tenant prod-1
opentide generate
opentide document
opentide init ./my-detections

8. MCP Server: FastMCP

Tools: search, get_chaining, coverage, validate_rule, validate_query, run_query, deploy_rule, deployment_status
Resources: opentide://index, opentide://schemas/{type}, opentide://vocabularies/{name}

---

Research Outcomes

These design decisions are backed by codebase research:

.files mapping — Split per collection

The current DataTide.Models.files maps UUID → YAML filename across all object types. Research found only 4 call sites, ALL MDR-only (ExternalIdHelper in deployment.py, defender_for_endpoint.py rule ID tracking). Decision: Each model gets a .file → Path property directly. No cross-type flat file mapping.

Orchestration — Registry vs Orchestrator

Analysis of all 7 Orchestration/*.py scripts found: generate.py, document.py, mutate.py are pure sequencing; deploy.py, validate.py, validate_query.py have real business logic. Decision: OpenTide returns structured data; CLI/pipelines translate to CI signals (exit codes, env vars, annotations). OpenTide does NOT orchestrate.

Signals — Nested in Objectives

Signals are always authored inside Detection Objectives. The current top-level DataTide.Models.signal index is a derived artefact. Decision: Signals are nested in DetectionObjective.signals, not a top-level collection.

Adversarial Review — Key Fixes Applied

An adversarial review challenged the design and identified critical issues:

1. Operations on Pydantic models → Fixed: delegation methods, not business logic
2. Triple deployment path → Fixed: single canonical path via Platform.deployer
3. God object → Fixed: OpenTide is registry only, no orchestration methods
4. Config/Platforms split → Fixed: config lives on Platform.config only
5. Speculative collection filters → Fixed: removed .by_status(), .search(), etc.
6. Missing configs → Fixed: added Configuration.Schema, Configuration.Sharing

---

Phased Execution Plan

Each phase is independently committable and testable. Phases are ordered by dependency (see DAG above).

Phase	Issue	Name	Scope	Depends On
0	#61	CDM, BDR & Lookups Removal	Remove all deprecated code (CDM, BDR, lookups)	—
1	#62	Code Decomposition	Break tide.py, models.py, deployment.py monoliths	Phase 0
2	#63	Naming Architecture	Rename identifiers, Platform-centric access, full English	Phase 1
3	#64	Vocabulary System Rebuild	Typed vocabularies, fix EnumResolver bugs	Phase 2
4	#65	Pydantic Migration	Code-first models, schema generation, template rebuild	Phase 3
5	#66	Package Structure	src/opentide layout, pyproject.toml, data bundling	Phase 4
6	#67	CLI	Typer commands for all orchestration scripts	Phase 5
7	#68	MCP Server	FastMCP with detection engineering tools	Phase 5
8	#69	PyPI Distribution	Version management, CI/CD publishing, migration guide	Phase 6, 7
9	#71	Testing & Documentation	Test suite, docs site, API reference	Phase 8

---

Success Criteria

• pip install opentide[sentinel] works from PyPI
• from opentide import OpenTide — clean single-surface API
• OpenTide.Platforms.Sentinel.deployer.deploy(uuids) — platform-centric, no "plugin" language
• rule.deploy("sentinel") — delegation methods on models
• opentide validate --platform sentinel --all — full CLI
• MCP server exposes detection engineering tools
• Zero import-time side effects
• Pydantic models as single source of truth (no metaschemas)
• All 7 detection platforms deployable via OpenTide.Platforms.{Platform}.deployer
• All 5 query-validating platforms reachable via OpenTide.Platforms.{Platform}.validator
• CDM, BDR, and lookups fully removed, no dead code
• Vocabulary system typed and validated
• Existing OpenTide client repos can migrate from submodule to pip install

---

Related

• feat: Migrate entire data framework to Pydantic code-first architecture #59 — Original Pydantic-migration spec (closed; superseded by feat: Phase 4 — Pydantic code-first migration (single source of truth) #70, which was a duplicate of feat: Phase 4 — Pydantic code-first migration (single source of truth) #65). The detailed behaviour inventories it once promised do not exist as issue text — Phase 4 (feat: Phase 4 — Pydantic code-first migration (single source of truth) #65) re-derives them from code. See the broken-reference notice in feat: Phase 4 — Pydantic code-first migration (single source of truth) #65.
• feat: Phase 4 — Pydantic code-first migration (single source of truth) #70 — Duplicate of feat: Phase 4 — Pydantic code-first migration (single source of truth) #65 (closed in favour of feat: Phase 4 — Pydantic code-first migration (single source of truth) #65, the canonical Phase 4 tracker).
• feat(carbon-black): Migrate CBC deployer to MDRv4 typed framework #58 / feat: Migrate Carbon Black Cloud deployer to MDRv4 typed framework #57 — Carbon Black Cloud MDRv4 typed migration (reference implementation that validates the typed approach).