Add Office365 Source Connector by alparish · Pull Request #5713 · opensearch-project/data-prepper

alparish · 2025-05-21T23:17:52Z

Description

This change implements an Office 365 source plugin for Data Prepper. The plugin interacts with the Office 365 Management API to retrieve audit logs. Key components include:

Office365RestClient: Handles REST API calls to Office 365 Management API.
Office365Service: Manages the business logic for fetching and processing audit logs.
Office365CrawlerClient: Implements the CrawlerClient interface for Data Prepper integration.
Office365Iterator: Manages the iteration over audit log items.

This PR also includes renaming the existing state objects to be more generic and reuse the same for Atlassian as well as Office365

Renamed AtlassianLeaderProgressState to PaginationCrawlerLeaderProgressState
Renamed AtlassianWorkerProgressState to PaginationCrawlerWorkerProgressState

Note: This is an initial PR for review. Unit tests will be added while awaiting review feedback.

Issues Resolved

Resolves #[Issue number to be closed when this PR is merged]

Check List

New functionality includes testing.
New functionality has a documentation issue. Please link to it in this PR.
- New functionality has javadoc added
Commits are signed with a real name per the DCO

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Savit Aluri <savaluri@amazon.com>

…bi-toolbag/content/workspace-summary.md Delete wasabi testing files.

…bi-toolbag/.metadata/bootstrap-time.txt Delete Wasabi toolbox used for testing.

feat: Office365 Source Initial Commit

Signed-off-by: Alekhya Parisha <aparisha@amazon.com>

…ce365-updates Rename Atlassian state classes to PaginationCrawler

Alekhya Parisha <aparisha@amazon.com>

Signed-off-by: Alekhya Parisha <aparisha@amazon.com>

Signed-off-by: Aparna Parisha <aparisha@amazon.com>

Signed-off-by: Alekhya Parisha <aparisha@amazon.com>

…-processor Add OCSF transformation processor for Office 365 events

Signed-off-by: Alekhya Parisha <aparisha@amazon.com>

…ce365plugin-updates Add pagination support and fix time window handling

dlvenable

Thank you @alparish for this change. I left some general comments and suggestions for the configurations. We also need unit tests.

dlvenable · 2025-05-22T19:35:09Z

+    implementation 'org.projectlombok:lombok:1.18.30'
+    annotationProcessor 'org.projectlombok:lombok:1.18.30'
+
+    testImplementation 'com.fasterxml.jackson.dataformat:jackson-dataformat-yaml:2.13.4'


Do not specify Jackson versions. They are inherited.

Removed Jackson versions

dlvenable · 2025-05-22T19:35:14Z

+
+    testImplementation 'com.fasterxml.jackson.dataformat:jackson-dataformat-yaml:2.13.4'
+    testImplementation project(path: ':data-prepper-test-common')
+    testImplementation 'com.fasterxml.jackson.datatype:jackson-datatype-jsr310:2.13.0'


Do not specify Jackson versions. They are inherited.

Removed Jackson versions

dlvenable · 2025-05-22T19:35:23Z

+    implementation(libs.spring.web)
+}
+
+test {


You don't need these lines.

dlvenable · 2025-05-22T19:37:29Z

+ * The type Constants.
+ */
+public class Constants {
+    public static final String PLUGIN_NAME = "office365";


Microsoft Office 365 is a massive product. The name of this source should reflect what part of Office 365 this is.

I understand this is what this is:

https://learn.microsoft.com/en-us/office/office-365-management-api/office-365-management-activity-api-reference

So call it something like office365_management_activity to use the name of this feature within Office 365.

Renamed the source to ‘microsoft-office365’ as this is approved in blueprint.

dlvenable · 2025-05-22T19:38:21Z

+ * The type Constants.
+ */
+public class Constants {
+    public static final String PLUGIN_NAME = "office365";


Also, you only use this in the Office365Source class. Move it there and make it package protected.

We need to remain PLUGIN_NAME public as it's used by Office365Source class in a different package for the @DataPrepperPlugin annotation

dlvenable · 2025-05-22T19:44:31Z

+package org.opensearch.dataprepper.plugins.source.office365.configuration;
+
+public class MetadataKeyAttributes {
+    public static final String CONTENT_TYPE = "contentType";


Move this into Office365CrawlerClient and make it private.

dlvenable · 2025-05-22T19:45:07Z

+
+    @Override
+    public void initCredentials() {
+        log.info("Initializing credentials...");


Suggested change

log.info("Initializing credentials...");

log.info("Initializing credentials.");

Do not use ellipses in logs. It is confusing to users.

dlvenable · 2025-05-22T19:51:39Z

+                        },
+                        authConfig::renewCredentials
+                );
+            } catch (Exception e) {


Find expected exceptions and handle differently from unexpected exceptions. Then use the stack trace for the unexpected only.

} catch (SomeClientException e) { log.error(NOISY, "Received {} response while fetching audit logs for content type {}", e.getSomeWayToGetErrorCodeAndMessage(), contentType); searchRequestsFailedCounter.increment(); throw new RuntimeException("Failed to fetch audit logs", e); } catch (Exception e) { log.error(NOISY, "Error while fetching audit logs for content type {}", contentType, e); searchRequestsFailedCounter.increment(); throw new RuntimeException("Failed to fetch audit logs", e); }

Here is some similar code:

data-prepper/data-prepper-plugins/parse-json-processor/src/main/java/org/opensearch/dataprepper/plugins/processor/parse/json/ParseJsonProcessor.java

Lines 61 to 73 in ec8e4be

} catch (JsonProcessingException e) {

if (handleFailedEventsOption.shouldLog()) {

LOG.error(SENSITIVE, "An exception occurred due to invalid JSON while parsing [{}] due to {}", message, e.getMessage());

}

parseErrorsCounter.increment();

return Optional.empty();

} catch (Exception e) {

if (handleFailedEventsOption.shouldLog()) {

LOG.error(SENSITIVE, "An exception occurred while using the parse_json processor while parsing [{}]", message, e);

}

processingFailuresCounter.increment();

return Optional.empty();

}

It catches parsing exceptions which are expected and handles them without the stack trace.

dlvenable · 2025-05-22T19:53:07Z

+ * Office 365 Connector main entry point.
+ * This class extends CrawlerSourcePlugin to provide Office 365 specific functionality.
+ */
+@DataPrepperPlugin(name = PLUGIN_NAME,


Add the @Experimental annotation on this until it is ready to reach a stable state.

Added @experimental annotation

dlvenable · 2025-05-22T19:54:34Z

+ */
+@Slf4j
+@Named
+public class Office365CrawlerClient implements CrawlerClient<PaginationCrawlerWorkerProgressState> {


All of these classes will need unit tests.

Added unit tests for Office365RestClient. Will be adding unit tests in the next PR

dlvenable · 2025-05-22T19:57:24Z

+    private static final String SEARCH_REQUESTS_FAILED = "searchRequestsFailed";
+
+    private final RestTemplate restTemplate = new RestTemplate();
+    private final Office365AuthenticationProvider authConfig;


Shouldn't this be the interface? Office365AuthenticationInterface

dlvenable · 2025-05-22T19:57:35Z

+public class Office365Source extends CrawlerSourcePlugin {
+    private static final Logger LOG = LoggerFactory.getLogger(Office365Source.class);
+    private final Office365SourceConfig office365SourceConfig;
+    private final Office365AuthenticationProvider office365AuthProvider;


Shouldn't this be the interface? Office365AuthenticationInterface

Savit Aluri and others added 17 commits May 1, 2025 10:01

feat: Office365 Source Initial Commit

34759ee

Signed-off-by: Savit Aluri <savaluri@amazon.com>

feat: Office365 Source Initial Commit

6afc378

Signed-off-by: Savit Aluri <savaluri@amazon.com>

Delete data-prepper-plugins/saas-source-plugins/office365-source/wasa…

62f2575

…bi-toolbag/content/workspace-summary.md Delete wasabi testing files.

Delete data-prepper-plugins/saas-source-plugins/office365-source/wasa…

154ff8c

…bi-toolbag/.metadata/bootstrap-time.txt Delete Wasabi toolbox used for testing.

Merge pull request #1 from savit-aluri/office365-source

63f7741

feat: Office365 Source Initial Commit

Merge remote-tracking branch 'upstream/main'

b126295

Rename Atlassian state classes to PaginationCrawler

fa1c1fa

Signed-off-by: Alekhya Parisha <aparisha@amazon.com>

Merge pull request opensearch-project#6 from savit-aluri/feature/offi…

f092694

…ce365-updates Rename Atlassian state classes to PaginationCrawler

Add pagination support and fix time window handling

a6d441c

Alekhya Parisha <aparisha@amazon.com>

Add OCSF processor

1294643

Signed-off-by: Alekhya Parisha <aparisha@amazon.com>

Address review comments

3b78765

Signed-off-by: Aparna Parisha <aparisha@amazon.com>

Update OCSF processor to load schema file based on schema_type

383849e

Signed-off-by: Alekhya Parisha <aparisha@amazon.com>

Merge pull request opensearch-project#2 from savit-aluri/feature/ocsf…

73c0b34

…-processor Add OCSF transformation processor for Office 365 events

Address review comments

b04bfd7

Signed-off-by: Alekhya Parisha <aparisha@amazon.com>

Merge pull request opensearch-project#7 from savit-aluri/feature/offi…

1f228b4

…ce365plugin-updates Add pagination support and fix time window handling

Merge remote-tracking branch 'upstream/main'

1698a97

Add Office365 Source Connector

3ead5d0

alparish requested review from KarstenSchnitter, chenqi0805, dinujoh, dlvenable, engechas, graytaylor0, kkondaka, oeyh, san81, sb2k16 and srikanthjg as code owners May 21, 2025 23:17

dlvenable requested changes May 22, 2025

View reviewed changes

dlvenable reviewed May 22, 2025

View reviewed changes

dlvenable requested changes May 22, 2025

View reviewed changes

	log.info("Initializing credentials...");
	log.info("Initializing credentials.");

	} catch (JsonProcessingException e) {
	if (handleFailedEventsOption.shouldLog()) {
	LOG.error(SENSITIVE, "An exception occurred due to invalid JSON while parsing [{}] due to {}", message, e.getMessage());
	}
	parseErrorsCounter.increment();
	return Optional.empty();
	} catch (Exception e) {
	if (handleFailedEventsOption.shouldLog()) {
	LOG.error(SENSITIVE, "An exception occurred while using the parse_json processor while parsing [{}]", message, e);
	}
	processingFailuresCounter.increment();
	return Optional.empty();
	}

Uh oh!

Conversation

alparish commented May 21, 2025

Description

Issues Resolved

Check List

Uh oh!

dlvenable left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants