Skip to content

[Bitsight Connector] - Fixing duplicate records issue #14800

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
dhanunjaya1054 wants to merge 1 commit into
base: master
Choose a base branch
from
Open

[Bitsight Connector] - Fixing duplicate records issue #14800

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 6 additions & 18 deletions Solutions/BitSight/Data Connectors/BitSight_CCF/ConnectorDefinition.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -148,7 +148,7 @@
{
"type": "Markdown",
"parameters": {
"content": "## BitSight Statistics Connections\n\nManage multiple BitSight statistics connections. Each connection selects one or more **data streams** to ingest and assigns a **Connection Name** stored in the `connectionName` column of every ingested record.\n\n> **Authentication**: BitSight uses HTTP Basic Authentication where the API token is used as **both** the username and password."
"content": "## BitSight Statistics Connections\n\nManage multiple BitSight statistics connections. Each connection selects one or more **data streams** to ingest and assigns a **Connection Name** stored in the `connectionName` column of every ingested record.\n\n> **Authentication**: BitSight uses HTTP Basic Authentication with your API Token."
}
},
{
Expand All @@ -160,8 +160,8 @@
"columnValue": "properties.addOnAttributes.connectionName"
},
{
"columnName": "Active Streams",
"columnValue": "properties.addOnAttributes.streams"
"columnName": "Data Type",
"columnValue": "properties.dataType"
},
{
"columnName": "API URL",
Expand Down Expand Up @@ -255,28 +255,16 @@
{
"type": "Markdown",
"parameters": {
"content": "## 3. Authentication\n\nBitSight uses your API token as **both** the username and password for HTTP Basic Authentication."
"content": "## 3. Authentication\n\nProvide your BitSight API Token. It authenticates requests to the BitSight REST API using HTTP Basic Authentication."
}
},
{
"type": "Textbox",
"parameters": {
"label": "BitSight API Token (Username)",
"label": "BitSight API Token",
"placeholder": "Paste your BitSight API Token",
"type": "text",
"name": "username",
"validations": {
"required": true
}
}
},
{
"type": "Textbox",
"parameters": {
"label": "BitSight API Token (Password)",
"placeholder": "Paste your BitSight API Token again",
"type": "password",
"name": "password",
"name": "apiToken",
"validations": {
"required": true
}
Expand Down
58 changes: 33 additions & 25 deletions Solutions/BitSight/Data Connectors/BitSight_CCF/PollingConfig.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
[
[
{
"type": "Microsoft.SecurityInsights/dataConnectors",
"apiVersion": "2025-03-01",
Expand All @@ -7,9 +7,10 @@
"kind": "RestApiPoller",
"properties": {
"auth": {
"type": "Basic",
"UserName": "[[parameters('username')]",
"Password": "[[parameters('password')]"
"type": "APIKey",
"ApiKey": "[[base64(concat(parameters('apiToken'), ':'))]",
"ApiKeyName": "Authorization",
"ApiKeyIdentifier": "Basic"
},
"request": {
"apiEndpoint": "[[concat(parameters('bitSightApiUrl'), '/ratings/v2/portfolio')]",
Expand Down Expand Up @@ -97,9 +98,10 @@
"kind": "RestApiPoller",
"properties": {
"auth": {
"type": "Basic",
"UserName": "[[parameters('username')]",
"Password": "[[parameters('password')]"
"type": "APIKey",
"ApiKey": "[[base64(concat(parameters('apiToken'), ':'))]",
"ApiKeyName": "Authorization",
"ApiKeyIdentifier": "Basic"
},
"request": {
"apiEndpoint": "[[concat(parameters('bitSightApiUrl'), '/ratings/v2/portfolio')]",
Expand Down Expand Up @@ -185,9 +187,10 @@
"kind": "RestApiPoller",
"properties": {
"auth": {
"type": "Basic",
"UserName": "[[parameters('username')]",
"Password": "[[parameters('password')]"
"type": "APIKey",
"ApiKey": "[[base64(concat(parameters('apiToken'), ':'))]",
"ApiKeyName": "Authorization",
"ApiKeyIdentifier": "Basic"
},
"request": {
"apiEndpoint": "[[concat(parameters('bitSightApiUrl'), '/ratings/v2/portfolio')]",
Expand Down Expand Up @@ -275,9 +278,10 @@
"kind": "RestApiPoller",
"properties": {
"auth": {
"type": "Basic",
"UserName": "[[parameters('username')]",
"Password": "[[parameters('password')]"
"type": "APIKey",
"ApiKey": "[[base64(concat(parameters('apiToken'), ':'))]",
"ApiKeyName": "Authorization",
"ApiKeyIdentifier": "Basic"
},
"request": {
"apiEndpoint": "[[concat(parameters('bitSightApiUrl'), '/ratings/v2/portfolio')]",
Expand Down Expand Up @@ -365,9 +369,10 @@
"kind": "RestApiPoller",
"properties": {
"auth": {
"type": "Basic",
"UserName": "[[parameters('username')]",
"Password": "[[parameters('password')]"
"type": "APIKey",
"ApiKey": "[[base64(concat(parameters('apiToken'), ':'))]",
"ApiKeyName": "Authorization",
"ApiKeyIdentifier": "Basic"
},
"request": {
"apiEndpoint": "[[concat(parameters('bitSightApiUrl'), '/ratings/v2/portfolio')]",
Expand Down Expand Up @@ -455,9 +460,10 @@
"kind": "RestApiPoller",
"properties": {
"auth": {
"type": "Basic",
"UserName": "[[parameters('username')]",
"Password": "[[parameters('password')]"
"type": "APIKey",
"ApiKey": "[[base64(concat(parameters('apiToken'), ':'))]",
"ApiKeyName": "Authorization",
"ApiKeyIdentifier": "Basic"
},
"request": {
"apiEndpoint": "[[concat(parameters('bitSightApiUrl'), '/ratings/v2/portfolio')]",
Expand Down Expand Up @@ -545,9 +551,10 @@
"kind": "RestApiPoller",
"properties": {
"auth": {
"type": "Basic",
"UserName": "[[parameters('username')]",
"Password": "[[parameters('password')]"
"type": "APIKey",
"ApiKey": "[[base64(concat(parameters('apiToken'), ':'))]",
"ApiKeyName": "Authorization",
"ApiKeyIdentifier": "Basic"
},
"request": {
"apiEndpoint": "[[concat(parameters('bitSightApiUrl'), '/ratings/v2/portfolio')]",
Expand Down Expand Up @@ -635,9 +642,10 @@
"kind": "RestApiPoller",
"properties": {
"auth": {
"type": "Basic",
"UserName": "[[parameters('username')]",
"Password": "[[parameters('password')]"
"type": "APIKey",
"ApiKey": "[[base64(concat(parameters('apiToken'), ':'))]",
"ApiKeyName": "Authorization",
"ApiKeyIdentifier": "Basic"
},
"request": {
"apiEndpoint": "https://service.bitsighttech.com/customer-api/v1/defaults/vulnerabilities",
Expand Down
20 changes: 4 additions & 16 deletions Solutions/BitSight/Data Connectors/BitSight_Events_CCF/ConnectorDefinition.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -103,7 +103,7 @@
{
"type": "Markdown",
"parameters": {
"content": "## BitSight Connections\n\nManage multiple BitSight data stream connections. Each connection selects a specific data type - **Alerts**, **Breaches**, or **Findings** - and assigns a **Connection Name** that is stored in the `ConnectorName` column of every ingested record.\n\n> **Authentication**: BitSight uses HTTP Basic Authentication where the API token is used as **both** the username and password."
"content": "## BitSight Connections\n\nManage multiple BitSight data stream connections. Each connection selects a specific data type - **Alerts**, **Breaches**, or **Findings** - and assigns a **Connection Name** that is stored in the `ConnectorName` column of every ingested record.\n\n> **Authentication**: BitSight uses HTTP Basic Authentication with your API Token."
}
},
{
Expand Down Expand Up @@ -196,28 +196,16 @@
{
"type": "Markdown",
"parameters": {
"content": "## 3. Authentication\n\nBitSight uses your API token as **both** the username and password for HTTP Basic Authentication."
"content": "## 3. Authentication\n\nProvide your BitSight API Token. It authenticates requests to the BitSight REST API using HTTP Basic Authentication."
}
},
{
"type": "Textbox",
"parameters": {
"label": "BitSight API Token (Username)",
"label": "BitSight API Token",
"placeholder": "Paste your BitSight API Token",
"type": "text",
"name": "username",
"validations": {
"required": true
}
}
},
{
"type": "Textbox",
"parameters": {
"label": "BitSight API Token (Password)",
"placeholder": "Paste your BitSight API Token again",
"type": "password",
"name": "password",
"name": "apiToken",
"validations": {
"required": true
}
Expand Down
37 changes: 21 additions & 16 deletions Solutions/BitSight/Data Connectors/BitSight_Events_CCF/PollingConfig.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
[
[
{
"type": "Microsoft.SecurityInsights/dataConnectors",
"apiVersion": "2025-03-01",
Expand All @@ -7,9 +7,10 @@
"kind": "RestApiPoller",
"properties": {
"auth": {
"type": "Basic",
"UserName": "[[parameters('username')]",
"Password": "[[parameters('password')]"
"type": "APIKey",
"ApiKey": "[[base64(concat(parameters('apiToken'), ':'))]",
"ApiKeyName": "Authorization",
"ApiKeyIdentifier": "Basic"
},
"request": {
"apiEndpoint": "[[concat(parameters('bitSightApiUrl'), '/v2/alerts/')]",
Expand Down Expand Up @@ -66,9 +67,10 @@
"kind": "RestApiPoller",
"properties": {
"auth": {
"type": "Basic",
"UserName": "[[parameters('username')]",
"Password": "[[parameters('password')]"
"type": "APIKey",
"ApiKey": "[[base64(concat(parameters('apiToken'), ':'))]",
"ApiKeyName": "Authorization",
"ApiKeyIdentifier": "Basic"
},
"request": {
"apiEndpoint": "[[concat(parameters('bitSightApiUrl'), '/ratings/v2/portfolio')]",
Expand Down Expand Up @@ -167,9 +169,10 @@
"kind": "RestApiPoller",
"properties": {
"auth": {
"type": "Basic",
"UserName": "[[parameters('username')]",
"Password": "[[parameters('password')]"
"type": "APIKey",
"ApiKey": "[[base64(concat(parameters('apiToken'), ':'))]",
"ApiKeyName": "Authorization",
"ApiKeyIdentifier": "Basic"
},
"request": {
"apiEndpoint": "[[concat(parameters('bitSightApiUrl'), '/ratings/v2/portfolio')]",
Expand Down Expand Up @@ -271,9 +274,10 @@
"kind": "RestApiPoller",
"properties": {
"auth": {
"type": "Basic",
"UserName": "[[parameters('username')]",
"Password": "[[parameters('password')]"
"type": "APIKey",
"ApiKey": "[[base64(concat(parameters('apiToken'), ':'))]",
"ApiKeyName": "Authorization",
"ApiKeyIdentifier": "Basic"
},
"request": {
"apiEndpoint": "[[concat(parameters('bitSightApiUrl'), '/ratings/v2/portfolio')]",
Expand Down Expand Up @@ -375,9 +379,10 @@
"kind": "RestApiPoller",
"properties": {
"auth": {
"type": "Basic",
"UserName": "[[parameters('username')]",
"Password": "[[parameters('password')]"
"type": "APIKey",
"ApiKey": "[[base64(concat(parameters('apiToken'), ':'))]",
"ApiKeyName": "Authorization",
"ApiKeyIdentifier": "Basic"
},
"request": {
"apiEndpoint": "[[concat(parameters('bitSightApiUrl'), '/ratings/v2/portfolio')]",
Expand Down
2 changes: 1 addition & 1 deletion Solutions/BitSight/Data/Solution_BitSight.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@
"Data Connectors/BitSight_CCF/ConnectorDefinition.json"
],
"BasePath": "C:\\GitHub\\Azure-Sentinel\\Solutions\\BitSight",
"Version": "3.2.0",
"Version": "3.2.1",
"Metadata": "SolutionMetadata.json",
"TemplateSpec": true,
"Is1PConnector": false
Expand Down
Binary file added Solutions/BitSight/Package/3.2.1.zip
View file
Open in desktop
Binary file not shown.
1,409 changes: 998 additions & 411 deletions Solutions/BitSight/Package/mainTemplate.json
View file
Open in desktop

Large diffs are not rendered by default.

5 changes: 3 additions & 2 deletions Solutions/BitSight/Parsers/BitSightDiligenceStatistics.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
id: 3900f610-56b5-4f8f-84f4-291d9b41dfe2
Function:
Title: Parser for BitSightDiligenceStatistics
Version: '1.1.0'
LastUpdated: '2026-05-25'
Version: '1.2.0'
LastUpdated: '2026-07-30'
Category: Microsoft Sentinel Parser
FunctionName: BitSightDiligenceStatistics
FunctionAlias: BitSightDiligenceStatistics
Expand Down Expand Up @@ -46,6 +46,7 @@ FunctionQuery: |
),
(
BitSightDiligenceStatistics_CL
| summarize arg_max(TimeGenerated, *) by CompanyGuid, RiskVector
| extend
EventVendor = "BitSight",
EventProduct = "DiligenceStatistics"
Expand Down
5 changes: 3 additions & 2 deletions Solutions/BitSight/Parsers/BitSightIndustrialStatistics.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
id: 44c51bc8-1c06-46ca-ba25-0422d0af4015
Function:
Title: Parser for BitSightIndustrialStatistics
Version: '1.1.0'
LastUpdated: '2026-05-25'
Version: '1.2.0'
LastUpdated: '2026-07-30'
Category: Microsoft Sentinel Parser
FunctionName: BitSightIndustrialStatistics
FunctionAlias: BitSightIndustrialStatistics
Expand Down Expand Up @@ -30,6 +30,7 @@ FunctionQuery: |
),
(
BitsightIndustrialStatistics_CL
| summarize arg_max(TimeGenerated, *) by CompanyGuid, RiskVector
| extend
EventVendor = "BitSight",
EventProduct = "IndustrialStatistics"
Expand Down
5 changes: 3 additions & 2 deletions Solutions/BitSight/Parsers/BitSightObservationStatistics.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
id: 144a9fa0-98a3-40f7-b39e-9c7ff0229290
Function:
Title: Parser for BitSightObservationStatistics
Version: '1.1.0'
LastUpdated: '2026-05-25'
Version: '1.2.0'
LastUpdated: '2026-07-30'
Category: Microsoft Sentinel Parser
FunctionName: BitSightObservationStatistics
FunctionAlias: BitSightObservationStatistics
Expand Down Expand Up @@ -30,6 +30,7 @@ FunctionQuery: |
),
(
BitSightObservationStatistics_CL
| summarize arg_max(TimeGenerated, *) by CompanyGuid, RiskVector
| extend
EventVendor = "BitSight",
EventProduct = "ObservationStatistics"
Expand Down
1 change: 1 addition & 0 deletions Solutions/BitSight/ReleaseNotes.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
| **Version** | **Date Modified (DD-MM-YYYY)** | **Change History** |
|-------------|--------------------------------|--------------------------------------------------------------------|
| 3.2.1 | 30-07-2026 | Added arg_max(TimeGenerated, *) de-duplication across the BitSight statistics parsers so snapshot tables return only the latest record per entity, and simplified connector authentication to a single API Token field (HTTP Basic authentication with the token as the username) for both the BitSight Security Statistics and BitSight Security Events connectors. |
| 3.2.0 | 15-06-2026 | Replaced legacy Function App connector with two **Codeless Connector Framework (CCF)** connectors: **BitSight Security Events** (Alerts, Breaches, Findings) and **BitSight Security Statistics** (CompanyDetails, CompanyRatingDetails, DiligenceHistoricalStatistics, DiligenceStatistics, ObservationStatistics, IndustrialStatistics, VulnerabilitiesFindingsSummary, FindingsSummary). Added parsers for **BitSightCompanyRatingDetails** and **BitSightVulnerabilitiesFindingsSummary**. Updated Function App UI page for Log Ingestion API. |
| 3.1.1 | 22-04-2026 | Updated **Solution Package** with the fix of solutionId |
| 3.1.0 | 31-03-2026 | Updated the python runtime version to 3.12. Added support for Log Ingestion API and updated parsers accordingly. <br> Reverted the solution id to fix the BitSight Solution publishing issue. |
Expand Down