Skip to content

Add Fivetran community solution (DCR-based connector, analytics, workbook) - #14762

Open
woodsonsecurity wants to merge 4 commits into
Azure:masterfrom
woodsonsecurity:fivetran
Open

Add Fivetran community solution (DCR-based connector, analytics, workbook)#14762
woodsonsecurity wants to merge 4 commits into
Azure:masterfrom
woodsonsecurity:fivetran

Conversation

@woodsonsecurity

Copy link
Copy Markdown

Change(s)

Adds a new community solution: Fivetran.

Fivetran pushes its platform/connector External Logs into Azure Monitor via the DCR-based Logs Ingestion API. This solution ships the Sentinel content for that feed.

Included:

  • Data connector (gallery tile, status/docs) + DCR + custom table (Fivetran_CL)
  • Parser (Fivetran)
  • 2 Analytic rules: Ingestion Gap, Auth Failures
  • 1 Hunting query: Severe log spike
  • Workbook
  • SolutionMetadata, Data manifest, ReleaseNotes, README

How to test

Deploy the solution, connect a Fivetran account via External Logs (DCR mode), then run Fivetran_CL | take 10.

  • Analytic rules validated (KQL)
  • ARM-TTK validated (48/49; the 1 fail is the ignored IDs Should Be Derived From ResourceIDs false positive)

@woodsonsecurity
woodsonsecurity requested review from a team as code owners July 26, 2026 06:13
@woodsonsecurity

Copy link
Copy Markdown
Author

@microsoft-github-policy-service agree company="Microsoft"

@woodsonsecurity

woodsonsecurity commented Jul 26, 2026 via email

Copy link
Copy Markdown
Author

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Note

Copilot was unable to run its full agentic suite in this review.

Adds a new Fivetran community solution for Microsoft Sentinel, including a DCR-based ingestion path and associated Sentinel content (connector tile, parser, analytics, hunting, workbook), plus packaging metadata.

Changes:

  • Added Fivetran workbook, parser, hunting query, and two analytic rules referencing Fivetran_CL.
  • Added solution metadata, release notes, README/packaging docs, and MITRE coverage notes.
  • Added generated V3 solution package artifacts (mainTemplate.json, createUiDefinition.json, testParameters.json) and KQL validation custom table schema.

Reviewed changes

Copilot reviewed 18 out of 20 changed files in this pull request and generated 9 comments.

Show a summary per file
File Description
Solutions/Fivetran/Workbooks/Fivetran.json Adds an overview workbook with basic volume/severity visualizations.
Solutions/Fivetran/SolutionMetadata.json Adds Content Hub publisher/support metadata for the solution.
Solutions/Fivetran/ReleaseNotes.md Adds initial release notes entry.
Solutions/Fivetran/README.md Documents solution purpose, contents, and deployment guidance.
Solutions/Fivetran/Parsers/Fivetran.yaml Adds the Fivetran parser function over Fivetran_CL.
Solutions/Fivetran/Package/testParameters.json Adds ARM test parameters for validating the generated package.
Solutions/Fivetran/Package/mainTemplate.json Adds the generated solution deployment template (content + metadata).
Solutions/Fivetran/Package/createUiDefinition.json Adds the generated CreateUiDefinition for solution installation UX.
Solutions/Fivetran/PACKAGING.md Documents how the V3 package was generated and validated.
Solutions/Fivetran/MITRE-COVERAGE.md Documents MITRE mappings rationale for the included content.
Solutions/Fivetran/Hunting Queries/FivetranSevereSpike.yaml Adds a hunting query for SEVERE spikes.
Solutions/Fivetran/Data/Solution_Fivetran.json Adds the solution manifest used by the packaging tool.
Solutions/Fivetran/Data Connectors/Fivetran_CCF/Fivetran_Table.json Adds an ARM template to create the custom table schema.
Solutions/Fivetran/Data Connectors/Fivetran_CCF/Fivetran_DataConnector.json Adds a “status/docs” GenericUI connector tile definition.
Solutions/Fivetran/Data Connectors/Fivetran_CCF/Fivetran_DCR.json Adds an ARM template to create the DCR for Logs Ingestion API.
Solutions/Fivetran/Analytic Rules/FivetranIngestionGap.yaml Adds an analytic rule for ingestion gaps (telemetry loss).
Solutions/Fivetran/Analytic Rules/FivetranAuthFailures.yaml Adds an analytic rule for repeated auth/credential failures.
.script/tests/KqlvalidationsTests/CustomTables/Fivetran_CL.json Adds a test schema stub so KQL validation recognizes Fivetran_CL.

Comment thread Solutions/Fivetran/ReleaseNotes.md
Comment thread Solutions/Fivetran/Data/Solution_Fivetran.json Outdated
Comment thread Solutions/Fivetran/SolutionMetadata.json Outdated
Comment thread Solutions/Fivetran/README.md Outdated
Comment thread Solutions/Fivetran/Package/mainTemplate.json
Comment thread Solutions/Fivetran/Package/mainTemplate.json
Comment thread Solutions/Fivetran/Package/testParameters.json
Comment thread Solutions/Fivetran/MITRE-COVERAGE.md Outdated
…book)

Fivetran pushes its platform/connector logs to Azure Monitor via the Logs
Ingestion API (DCR-based custom table). Ships a data-connector tile, parser,
two analytics rules, a hunting query and a workbook, plus the generated V3
Content Hub package.

- Register Fivetran in detection-template ValidConnectorIds allow-list
- Connector tile uses canonical workspaces + workspaces/sharedKeys permissions
  (DCR Monitoring Metrics Publisher guidance remains in the instruction steps)
- Community support link uses valid GitHub handle (woodsonsecurity)
- Connector tile isPreview aligned with generated package
- README references only shipped artifacts
- MITRE-COVERAGE cites the authoritative T1562 -> T1685 redirect

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 7b63e6ab-b863-4d14-ba12-e1e7837eca8d
@woodsonsecurity

Copy link
Copy Markdown
Author

All items raised on this PR have now been addressed:

Copilot review threads (9/9 replied and resolved)

  • Support link now uses a valid GitHub handle (woodsonsecurity) in SolutionMetadata.json and mainTemplate.json.
  • Connector tile isPreview aligned with the generated package (false).
  • Connector tile permissions use the canonical Microsoft.OperationalInsights/workspaces + workspaces/sharedKeys block.
  • README references only shipped artifacts.
  • MITRE-COVERAGE notes the authoritative T1562 -> T1685 renumbering (T1562 retained because Sentinel tooling still requires it).
  • Remaining flags (ReleaseNotes table marker, generation-time BasePath, savedSearch scope, workspace-location boilerplate) were verified as false positives that match current merged V3 solutions; each has an in-thread explanation.

CI gate fixes

  • Registered Fivetran in .script/tests/detectionTemplateSchemaValidation/ValidConnectorIds.json.
  • Corrected the data-connector permissions block for Data Connector Validation.

Packaging completeness

  • Confirmed Logos/Fivetran.svg and .script/tests/KqlvalidationsTests/CustomTables/Fivetran_CL.json are included so logo and KQL table-schema validation pass.

The branch is rebased onto current master (single clean commit) and the PR is mergeable. Ready for maintainer review. Thanks!

woodsonsecurity and others added 3 commits July 29, 2026 17:30
…v3.0.1)

Bundle four source-specific ASIM parsers over Fivetran_CL:
- ASimAuthenticationFivetran / vimAuthenticationFivetran (Authentication 0.1.4)
- ASimAuditEventFivetran / vimAuditEventFivetran (AuditEvent 0.1.2)

The ASim* parameterless parsers delegate to the vim* filtering parsers.
Regenerated Package/ (mainTemplate, createUiDefinition, 3.0.1.zip) with the
V3 tool; ARM-TTK 48/49 (single fail is the documented IDs-Should-Be-Derived
false positive). Updated Data manifest, ReleaseNotes and README.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 7b63e6ab-b863-4d14-ba12-e1e7837eca8d
Adds Solutions/Fivetran/Platform-Connector-Ingest/, a self-contained, fully
sanitized reference for ingesting the Fivetran Platform Connector tables
(ADLS Gen2 parquet) into Sentinel via an Event Grid triggered Azure Function
and the Logs Ingestion API, normalising audit_trail to ASIM AuditEvent.

Includes bicep (tables + kind:Direct DCR), Python function, DCR body, ASIM
parsers + hunting query, and an offline parser test. All environment values
are placeholders (no customer names/resource ids/subscription ids).

Improvements over a first-generation build: Flex Consumption hosting,
user-assigned managed identity with pre-granted RBAC, Event Grid
dead-lettering + azurefunction endpoint + tight subject filtering, and
function security hardening. Reference table kept on Analytics (not
Basic/Auxiliary) so enrichment joins keep working, with shorter retention
for cost.

Registers Fivetran_AuditTrail_CL and Fivetran_Platform_CL custom-table
schemas for KqlValidations, and links the folder from the solution README.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 7b63e6ab-b863-4d14-ba12-e1e7837eca8d
…mismatch)

The vimAuditEventFivetran and vimAuthenticationFivetran parsers used
coalesce() with a dynamic first argument (_evt.* / _data[...]) followed by
string arguments (column_ifexists(...), ''). Kusto infers the coalesce common
type from the first argument (dynamic) and then rejects the string arguments
(KS141 'The expression must have the type dynamic'), which also broke the
ASim* wrappers that call these functions (KS211). Wrapped each dynamic accessor
in tostring() so every coalesce is uniformly string-typed. Updated the embedded
copies in Package/mainTemplate.json and Package/3.0.1.zip to match.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 2b220e19-0e51-4154-a464-81b7cf329450
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants