Add minute range support to Dimensional TimeSlice Source Crawler framework#6368
Conversation
enuraju
requested review from
KarstenSchnitter,
chenqi0805,
dinujoh,
dlvenable,
engechas,
graytaylor0,
kkondaka,
oeyh,
san81,
sb2k16 and
srikanthjg
as code owners
| DimensionalTimeSliceLeaderProgressState leaderProgressState = | ||
| (DimensionalTimeSliceLeaderProgressState) leaderPartition.getProgressState().get(); | ||
| int remainingHours = leaderProgressState.getRemainingHours(); | ||
| long remainingMinutes = leaderProgressState.getRemainingMinutes(); |
There was a problem hiding this comment.
Could have better to use Instant type instead of long
There was a problem hiding this comment.
Changed the type from long to Instant.
| * @deprecated Use {@link #getLookBackMinutes()} for minute-level granularity support | ||
| */ | ||
| @Deprecated | ||
| public int getLookBackHours() { |
There was a problem hiding this comment.
I would suggest to remove this method instead of marking as @Deprecated if it has no references at all.
There was a problem hiding this comment.
I see that some of the state objects still holds hours and you kept this for backward compatibility 👍
There was a problem hiding this comment.
I have removed this method, as it is not required in the connector.
enuraju
force-pushed
the
feature/dimensional-time-slice-crawler-range-fix
branch
from
ad368f2 to
d45c911
Compare
|
|
||
| @JsonProperty("remaining_hours") | ||
| private int remainingHours; | ||
| @JsonProperty("remaining_duration") |
There was a problem hiding this comment.
This is not backward compatible change. It might break existing pipelines. Are we good with this?
There was a problem hiding this comment.
Existing connectors will continue to use the constructor below. Based on the provided remainingHours, we calculate and update the remainingDuration accordingly. This logic ensures backward compatibility, as connectors relying on the remainingHours based behavior will continue to function without any changes.
public DimensionalTimeSliceLeaderProgressState(final Instant lastPollTime, int remainingHours) {
this.lastPollTime = lastPollTime;
long minutes = remainingHours * MINUTES_PER_HOUR;
if (minutes > 0) {
this.remainingDuration = lastPollTime.minus(Duration.ofMinutes(minutes));
} else {
this.remainingDuration = lastPollTime;
}
}
|
Has this been tested with the lowest granularity (e.g., 1-minute intervals) for an extended period? If this change only supports minute-level granularity for lookback, please conduct load testing with the following configuration:
|
|
Could you help me understand the specific use case driving this change? From my understanding, hour-level granularity should still capture all data. What scenario requires minute-level precision for the lookback window? |
@chrisale000 This change supports up to minute-level granularity; it supports days, hours, and hours with minutes as well with backward compatibility. |
@chrisale000 As suggested by Wenjie, we have done this change. |
|
@enuraju |
@chrisale000,
We also have tested with lowest possible intervals like PT1M, PT5M, PT10, PT15M. Please let us know if we are missing any other scenarios. |
| * | ||
| * @param remainingHours the remaining hours (will be converted to remainingDuration) | ||
| */ | ||
| @JsonSetter("remaining_hours") |
There was a problem hiding this comment.
@enuraju , Please create a GitHub issue to remove support for this deprecated field. We will want to remove this on a future breaking change release.
There was a problem hiding this comment.
Sure, We will create it.
| public void setRemainingHours(int remainingHours) { | ||
| if (this.remainingDuration == null || this.remainingDuration.equals(this.lastPollTime)) { | ||
| // Only set if remainingDuration hasn't been set yet (prefer remainingDuration to hours) | ||
| long minutes = remainingHours * MINUTES_PER_HOUR; |
There was a problem hiding this comment.
This logic looks similar to that in the constructor. Can you consolidate the code here to avoid split logic?
There was a problem hiding this comment.
Created a separate method and reused it in both the places to avoid duplicate code.
| private static final String WORKER_PARTITION_WAIT_TIME = "workerPartitionWaitTime"; | ||
| private static final String WORKER_PARTITION_PROCESS_LATENCY = "workerPartitionProcessLatency"; | ||
| private static final Duration HOUR_DURATION = Duration.ofHours(1); | ||
| private static final long MINUTES_PER_HOUR = 60; |
There was a problem hiding this comment.
You can use Duration for these constants and then use that in the code below. It should simplify the code some. And the durations will be clearer using ofHours(1) and ofMinutes(5).
There was a problem hiding this comment.
Made the changes as you suggested.
|
|
||
| assertNotNull(latest); | ||
|
|
||
| int expectedPartitions = 2 * LOG_TYPES.size(); |
There was a problem hiding this comment.
This 2 is a magic number. Make in a final int with a useful name. I believe it represents the expected hourly partitions per log type.
There was a problem hiding this comment.
Declared a variable with value and reused it.
| Instant latest = crawler.crawl(leaderPartition, coordinator); | ||
|
|
||
| assertNotNull(latest); | ||
| int expectedPartitions = 4 * LOG_TYPES.size(); |
There was a problem hiding this comment.
Similar magic number.
Also, why is it four instead of three?
There was a problem hiding this comment.
Declared a variable with value and reused it.
| } | ||
|
|
||
| @Test | ||
| void testCrawl_withHistoricalSync_3Hours() { |
There was a problem hiding this comment.
Update all the test names with some expectation as well. And you don't need test in the name.
Something like this perhaps:
crawl_withHistoricalSync_with3HourLookbackDuration_shouldCreatePartionForEachHour
There was a problem hiding this comment.
Changed the method names as per your comment
| void testCrawl_withHistoricalSync_3Hours() { | ||
| Instant latestHour = Instant.now().truncatedTo(ChronoUnit.HOURS); | ||
| Instant initialTime = latestHour.plusSeconds(WAIT_SECONDS_BEFORE_PARTITION_CREATION + 1); | ||
| Instant lookbackDuration = initialTime.minus(Duration.ofMinutes(180)); |
There was a problem hiding this comment.
Duration.ofHours(3) would be clearer.
There was a problem hiding this comment.
As you suggested, converted Duration.ofMinutes(180) to Duration.ofHours(3)
| @JsonProperty("remaining_hours") | ||
| private int remainingHours; | ||
| @JsonProperty("remaining_duration") | ||
| private Instant remainingDuration; |
There was a problem hiding this comment.
How is this being serialized to DynamoDB and how do we ensure it remains in a consistent format in the DDB table?
There was a problem hiding this comment.
The entire LeaderProgressState is serialized as a single JSON string using a centrally configured ObjectMapper
(with JavaTimeModule), and stored as a String attribute in DynamoDB. All Instant fields are written in ISO-8601 format, and @JsonProperty ensures stable field names. The same mapper configuration is used for deserialization, ensuring a consistent and predictable format in DDB. 
enuraju
force-pushed
the
feature/dimensional-time-slice-crawler-range-fix
branch
from
d45c911 to
7e2426d
Compare
| Instant lookBackHoursAgo = Instant.now().minus(Duration.ofHours(office365SourceConfig.getLookBackHours())); | ||
| if (startTime.isBefore(lookBackHoursAgo) && lookBackHoursAgo.isBefore(endTime)) { | ||
| adjustedStartTime = lookBackHoursAgo; | ||
| Instant lookBackDuration = office365SourceConfig.getLookBackDuration(Instant.now()); |
There was a problem hiding this comment.
Dumb question: Will we be extending these changes to other connectors? If so, I think we should consider abstracting this code block to be used by other connectors too (e.g getAdjustedStartTime(startTime, sourceConfig.getLookBackDuration())
There was a problem hiding this comment.
Correct @vecheka, will take care in the next PR update.
There was a problem hiding this comment.
What do you think if we do it in this PR? So, you don't have to wait for another PR approval.
Non-blocking though but if you need to make a new revision, would highly recommend doing this.
There was a problem hiding this comment.
I’ve updated this PR to abstract the logic so it can be reused by other connectors. This should avoid repeating the same implementation later. Please take a look and let me know if any further adjustments are needed.
|
@enuraju Build is failing, please take a look. |
Signed-off-by: enugraju <enugraju@amazon.com>
Signed-off-by: enugraju <enugraju@amazon.com>
Signed-off-by: enugraju <enugraju@amazon.com>
…dStartTime Signed-off-by: enugraju <enugraju@amazon.com>
enuraju
force-pushed
the
feature/dimensional-time-slice-crawler-range-fix
branch
from
7e2426d to
70581f2
Compare
This build issue got resolved. |
| private static final String WORKER_PARTITION_WAIT_TIME = "workerPartitionWaitTime"; | ||
| private static final String WORKER_PARTITION_PROCESS_LATENCY = "workerPartitionProcessLatency"; | ||
| private static final Duration HOUR_DURATION = Duration.ofHours(1); | ||
| static final Duration WAIT_BEFORE_PARTITION_CREATION = Duration.ofMinutes(5); |
There was a problem hiding this comment.
super-nit (non-blocking): Keep it consistent by using "private static or protected static..."
oeyh
left a comment
oeyh
left a comment
There was a problem hiding this comment.
Approving with one comment on a test. Please check.
| assertEquals(45, actualLookback.toMinutes(), | ||
| "Adjusted start time should be exactly 30 minutes ago"); |
There was a problem hiding this comment.
The assertion checks for 45 minutes but the message says 30 minutes. Which one is correct?
7 participants
Description
This change enhances the Dimensional Crawler framework to support minute-level time ranges for historical data ingestion.
Previously, the crawler relied on hour-based granularity when determining whether to run historical or incremental syncs. As a result, sub-hour ranges such as PT15M or PT30M were rounded down to zero, incorrectly triggering incremental sync and skipping historical data pulls.
This update replaces hour-based tracking with minute-based tracking across the framework and the Office365 source plugin, ensuring correct historical ingestion for any ISO-8601 duration expressed in minutes or hours.
How
Framework Updates
Office365 Plugin Updates
Is this change backward compatible?
Yes.
Testing
Unit / Functional Validation
Integration Verification
Local pipeline run succeeded:
range: PT1M
> Task :data-prepper-main:org.opensearch.dataprepper.DataPrepperExecute.main() 2026-02-04T12:29:31,726 [main] INFO org.opensearch.dataprepper.DataPrepperExecute - Data Prepper 2.14 2026-02-04T12:29:33,225 [main] INFO org.opensearch.dataprepper.core.DataPrepperArgumentConfiguration - Command line args: C:\connectors\amazon-data-prepper\data-prepper\pipelines,C:\connectors\amazon-data-prepper\data-prepper\config\data-prepper-config.yaml 2026-02-04T12:29:33,225 [main] INFO org.opensearch.dataprepper.core.DataPrepperArgs - Using C:\connectors\amazon-data-prepper\data-prepper\pipelines configuration file 2026-02-04T12:29:33,613 [main] INFO org.opensearch.dataprepper.pipeline.parser.transformer.DynamicConfigTransformer - No transformation needed 2026-02-04T12:29:34,738 [main] INFO org.opensearch.dataprepper.plugin.ExtensionsApplier - Loaded 6 extensions: [org.opensearch.dataprepper.plugins.aws.AwsPlugin@39652a30, org.opensearch.dataprepper.plugins.aws.AwsSecretPlugin@5763a655, org.opensearch.dataprepper.plugins.kafka.extension.KafkaClusterConfigExtension@40c8067, org.opensearch.dataprepper.plugins.encryption.EncryptionPlugin@46bb0bdf, org.opensearch.dataprepper.plugins.kinesis.extension.KinesisLeaseConfigExtension@561f9d92, org.opensearch.dataprepper.plugins.geoip.extension.GeoIpConfigExtension@1c84d80a] 2026-02-04T12:29:38,896 [main] INFO org.opensearch.dataprepper.plugins.kafka.extension.KafkaClusterConfigExtension - Applying Kafka Cluster Config Extension. 2026-02-04T12:29:39,174 [main] INFO org.opensearch.dataprepper.core.parser.PipelineTransformer - Building pipeline [pipeline] from provided configuration 2026-02-04T12:29:39,174 [main] INFO org.opensearch.dataprepper.core.parser.PipelineTransformer - Building [microsoft_office365] as source component for the pipeline [pipeline] 2026-02-04T12:29:39,825 [main] INFO org.opensearch.dataprepper.plugins.source.microsoft_office365.Office365Source - Creating Office365 Source Plugin 2026-02-04T12:29:39,825 [main] INFO org.opensearch.dataprepper.core.parser.PipelineTransformer - Building buffer for the pipeline [pipeline] 2026-02-04T12:29:39,834 [main] INFO org.opensearch.dataprepper.core.parser.PipelineTransformer - Building processors for the pipeline [pipeline] 2026-02-04T12:29:39,834 [main] INFO org.opensearch.dataprepper.core.parser.PipelineTransformer - Building sinks for the pipeline [pipeline] 2026-02-04T12:29:39,835 [main] INFO org.opensearch.dataprepper.core.parser.PipelineTransformer - Building [stdout] as sink component 2026-02-04T12:29:39,838 [main] INFO org.opensearch.dataprepper.core.parser.PipelineTransformer - Constructing MultiBufferDecorator with [0] secondary buffers for pipeline [pipeline] 2026-02-04T12:29:39,953 [main] WARN org.opensearch.dataprepper.core.pipeline.server.config.DataPrepperServerConfiguration - Creating data prepper server without authentication. This is not secure. 2026-02-04T12:29:39,954 [main] WARN org.opensearch.dataprepper.core.pipeline.server.config.DataPrepperServerConfiguration - In order to set up Http Basic authentication for the data prepper server, go here: https://github.com/opensearch-project/data-prepper/blob/main/docs/core_apis.md#authentication 2026-02-04T12:29:40,046 [main] INFO org.opensearch.dataprepper.core.pipeline.Pipeline - Pipeline [pipeline] - Initiating pipeline execution 2026-02-04T12:29:40,049 [main] INFO org.opensearch.dataprepper.core.sourcecoordination.SourceCoordinatorFactory - Creating EnhancedLeaseBasedSourceCoordinator with coordination store in_memory for sub-pipeline pipeline 2026-02-04T12:29:40,052 [main] WARN org.opensearch.dataprepper.plugins.sourcecoordinator.inmemory.InMemorySourceCoordinationStore - The in_memory source coordination store is not recommended for production workloads. It is only effective in single node environments of Data Prepper, and can run into memory limitations over time if the number of partitions is too great. 2026-02-04T12:29:40,055 [pipeline-sink-worker-2-thread-1] INFO org.opensearch.dataprepper.core.pipeline.Pipeline - Pipeline [pipeline] Sink is ready, starting source... 2026-02-04T12:29:40,056 [pipeline-sink-worker-2-thread-1] INFO org.opensearch.dataprepper.plugins.source.microsoft_office365.Office365Source - Starting Office365 Source Plugin... 2026-02-04T12:29:40,056 [pipeline-sink-worker-2-thread-1] INFO org.opensearch.dataprepper.plugins.source.source_crawler.auth.AuthenticationInterface - Initializing credentials. 2026-02-04T12:29:40,056 [pipeline-sink-worker-2-thread-1] INFO org.opensearch.dataprepper.plugins.source.microsoft_office365.auth.Office365AuthenticationProvider - Getting new access token for Office 365 Management API 2026-02-04T12:29:40,062 [pipeline-sink-worker-2-thread-1] INFO org.opensearch.dataprepper.plugins.aws.AwsSecretsSupplier - Retrieving latest secrets in aws:secrets:m365_secret. 2026-02-04T12:29:40,072 [main] WARN org.opensearch.dataprepper.core.pipeline.server.HttpServerProvider - Creating Data Prepper server without TLS. This is not secure. 2026-02-04T12:29:40,072 [main] WARN org.opensearch.dataprepper.core.pipeline.server.HttpServerProvider - In order to set up TLS for the Data Prepper server, go here: https://github.com/opensearch-project/data-prepper/blob/main/docs/configuration.md#server-configuration 2026-02-04T12:29:40,111 [main] INFO org.opensearch.dataprepper.core.pipeline.server.DataPrepperServer - Data Prepper server running at :4903 2026-02-04T12:29:40,313 [pipeline-sink-worker-2-thread-1] INFO org.opensearch.dataprepper.plugins.aws.AwsSecretsSupplier - Finished retrieving latest secret in aws:secrets:m365_secret. 2026-02-04T12:29:40,313 [pipeline-sink-worker-2-thread-1] INFO org.opensearch.dataprepper.plugins.aws.AwsSecretsSupplier - Retrieving latest secrets in aws:secrets:m365_secret. 2026-02-04T12:29:40,552 [pipeline-sink-worker-2-thread-1] INFO org.opensearch.dataprepper.plugins.aws.AwsSecretsSupplier - Finished retrieving latest secret in aws:secrets:m365_secret. 2026-02-04T12:29:41,027 [pipeline-sink-worker-2-thread-1] INFO org.opensearch.dataprepper.plugins.source.microsoft_office365.auth.Office365AuthenticationProvider - Received new access token. Expires in 3599 seconds 2026-02-04T12:29:41,027 [pipeline-sink-worker-2-thread-1] INFO org.opensearch.dataprepper.plugins.source.source_crawler.auth.AuthenticationInterface - Credentials initialized successfully 2026-02-04T12:29:41,027 [pipeline-sink-worker-2-thread-1] INFO org.opensearch.dataprepper.plugins.source.microsoft_office365.Office365RestClient - Starting Office 365 subscriptions for audit logs 2026-02-04T12:29:41,796 [pipeline-sink-worker-2-thread-1] INFO org.opensearch.dataprepper.plugins.source.source_crawler.base.CrawlerSourcePlugin - Starting microsoft_office365 Source Plugin 2026-02-04T12:29:41,810 [pool-7-thread-2] INFO org.opensearch.dataprepper.plugins.source.source_crawler.coordination.scheduler.WorkerScheduler - Worker thread started 2026-02-04T12:29:41,810 [pool-7-thread-4] INFO org.opensearch.dataprepper.plugins.source.source_crawler.coordination.scheduler.WorkerScheduler - Worker thread started 2026-02-04T12:29:41,810 [pipeline-sink-worker-2-thread-1] INFO org.opensearch.dataprepper.core.pipeline.Pipeline - Pipeline [pipeline] - Submitting request to initiate the pipeline processing 2026-02-04T12:29:41,810 [pool-7-thread-3] INFO org.opensearch.dataprepper.plugins.source.source_crawler.coordination.scheduler.WorkerScheduler - Worker thread started 2026-02-04T12:29:41,811 [pool-7-thread-4] INFO org.opensearch.dataprepper.plugins.source.source_crawler.coordination.scheduler.WorkerScheduler - Processing Partitions 2026-02-04T12:29:41,811 [pool-7-thread-3] INFO org.opensearch.dataprepper.plugins.source.source_crawler.coordination.scheduler.WorkerScheduler - Processing Partitions 2026-02-04T12:29:41,811 [pool-7-thread-5] INFO org.opensearch.dataprepper.plugins.source.source_crawler.coordination.scheduler.WorkerScheduler - Worker thread started 2026-02-04T12:29:41,811 [pool-7-thread-2] INFO org.opensearch.dataprepper.plugins.source.source_crawler.coordination.scheduler.WorkerScheduler - Processing Partitions 2026-02-04T12:29:41,811 [pool-7-thread-5] INFO org.opensearch.dataprepper.plugins.source.source_crawler.coordination.scheduler.WorkerScheduler - Processing Partitions 2026-02-04T12:29:41,814 [pool-7-thread-1] INFO org.opensearch.dataprepper.plugins.source.source_crawler.coordination.scheduler.LeaderScheduler - Running as a LEADER node 2026-02-04T12:29:41,815 [pool-7-thread-1] INFO org.opensearch.dataprepper.plugins.source.source_crawler.base.DimensionalTimeSliceCrawler - Creating partition for sub-hour historical pull: 1 minutes 2026-02-04T12:29:41,819 [pool-7-thread-1] INFO org.opensearch.dataprepper.plugins.source.source_crawler.base.DimensionalTimeSliceCrawler - Total partitions created in this crawl: 5.0 2026-02-04T12:29:51,821 [pool-7-thread-4] INFO org.opensearch.dataprepper.plugins.source.source_crawler.base.DimensionalTimeSliceCrawler - Processing partition - DimensionType: Audit.AzureActiveDirectory, TimeRange: 2026-02-04T06:58:41.798165Z to 2026-02-04T06:59:41.798165Z 2026-02-04T12:29:51,821 [pool-7-thread-5] INFO org.opensearch.dataprepper.plugins.source.source_crawler.base.DimensionalTimeSliceCrawler - Processing partition - DimensionType: Audit.General, TimeRange: 2026-02-04T06:58:41.798165Z to 2026-02-04T06:59:41.798165Z 2026-02-04T12:29:51,821 [pool-7-thread-3] INFO org.opensearch.dataprepper.plugins.source.source_crawler.base.DimensionalTimeSliceCrawler - Processing partition - DimensionType: Audit.SharePoint, TimeRange: 2026-02-04T06:58:41.798165Z to 2026-02-04T06:59:41.798165Z 2026-02-04T12:29:51,824 [pool-7-thread-2] INFO org.opensearch.dataprepper.plugins.source.source_crawler.base.DimensionalTimeSliceCrawler - Processing partition - DimensionType: DLP.All, TimeRange: 2026-02-04T06:58:41.798165Z to 2026-02-04T06:59:41.798165Z 2026-02-04T12:29:52,017 [pool-7-thread-2] INFO org.opensearch.dataprepper.plugins.source.source_crawler.base.DimensionalTimeSliceCrawler - Processing partition - DimensionType: Audit.Exchange, TimeRange: 2026-02-04T06:58:41.798165Z to 2026-02-04T06:59:41.798165Zrange:PT5M
> Task :data-prepper-main:org.opensearch.dataprepper.DataPrepperExecute.main() 2026-02-04T12:31:34,421 [main] INFO org.opensearch.dataprepper.DataPrepperExecute - Data Prepper 2.14 2026-02-04T12:31:36,430 [main] INFO org.opensearch.dataprepper.core.DataPrepperArgumentConfiguration - Command line args: C:\connectors\amazon-data-prepper\data-prepper\pipelines,C:\connectors\amazon-data-prepper\data-prepper\config\data-prepper-config.yaml 2026-02-04T12:31:36,441 [main] INFO org.opensearch.dataprepper.core.DataPrepperArgs - Using C:\connectors\amazon-data-prepper\data-prepper\pipelines configuration file 2026-02-04T12:31:36,960 [main] INFO org.opensearch.dataprepper.pipeline.parser.transformer.DynamicConfigTransformer - No transformation needed 2026-02-04T12:31:38,588 [main] INFO org.opensearch.dataprepper.plugin.ExtensionsApplier - Loaded 6 extensions: [org.opensearch.dataprepper.plugins.aws.AwsPlugin@13f40d71, org.opensearch.dataprepper.plugins.aws.AwsSecretPlugin@73a845cb, org.opensearch.dataprepper.plugins.kafka.extension.KafkaClusterConfigExtension@18a98913, org.opensearch.dataprepper.plugins.encryption.EncryptionPlugin@7dbae40, org.opensearch.dataprepper.plugins.kinesis.extension.KinesisLeaseConfigExtension@34f7b44f, org.opensearch.dataprepper.plugins.geoip.extension.GeoIpConfigExtension@5403907] 2026-02-04T12:31:43,647 [main] INFO org.opensearch.dataprepper.plugins.kafka.extension.KafkaClusterConfigExtension - Applying Kafka Cluster Config Extension. 2026-02-04T12:31:43,969 [main] INFO org.opensearch.dataprepper.core.parser.PipelineTransformer - Building pipeline [pipeline] from provided configuration 2026-02-04T12:31:43,969 [main] INFO org.opensearch.dataprepper.core.parser.PipelineTransformer - Building [microsoft_office365] as source component for the pipeline [pipeline] 2026-02-04T12:31:44,660 [main] INFO org.opensearch.dataprepper.plugins.source.microsoft_office365.Office365Source - Creating Office365 Source Plugin 2026-02-04T12:31:44,660 [main] INFO org.opensearch.dataprepper.core.parser.PipelineTransformer - Building buffer for the pipeline [pipeline] 2026-02-04T12:31:44,672 [main] INFO org.opensearch.dataprepper.core.parser.PipelineTransformer - Building processors for the pipeline [pipeline] 2026-02-04T12:31:44,673 [main] INFO org.opensearch.dataprepper.core.parser.PipelineTransformer - Building sinks for the pipeline [pipeline] 2026-02-04T12:31:44,674 [main] INFO org.opensearch.dataprepper.core.parser.PipelineTransformer - Building [stdout] as sink component 2026-02-04T12:31:44,679 [main] INFO org.opensearch.dataprepper.core.parser.PipelineTransformer - Constructing MultiBufferDecorator with [0] secondary buffers for pipeline [pipeline] 2026-02-04T12:31:44,865 [main] WARN org.opensearch.dataprepper.core.pipeline.server.config.DataPrepperServerConfiguration - Creating data prepper server without authentication. This is not secure. 2026-02-04T12:31:44,865 [main] WARN org.opensearch.dataprepper.core.pipeline.server.config.DataPrepperServerConfiguration - In order to set up Http Basic authentication for the data prepper server, go here: https://github.com/opensearch-project/data-prepper/blob/main/docs/core_apis.md#authentication 2026-02-04T12:31:45,020 [main] INFO org.opensearch.dataprepper.core.pipeline.Pipeline - Pipeline [pipeline] - Initiating pipeline execution 2026-02-04T12:31:45,029 [main] INFO org.opensearch.dataprepper.core.sourcecoordination.SourceCoordinatorFactory - Creating EnhancedLeaseBasedSourceCoordinator with coordination store in_memory for sub-pipeline pipeline 2026-02-04T12:31:45,035 [main] WARN org.opensearch.dataprepper.plugins.sourcecoordinator.inmemory.InMemorySourceCoordinationStore - The in_memory source coordination store is not recommended for production workloads. It is only effective in single node environments of Data Prepper, and can run into memory limitations over time if the number of partitions is too great. 2026-02-04T12:31:45,040 [pipeline-sink-worker-2-thread-1] INFO org.opensearch.dataprepper.core.pipeline.Pipeline - Pipeline [pipeline] Sink is ready, starting source... 2026-02-04T12:31:45,041 [pipeline-sink-worker-2-thread-1] INFO org.opensearch.dataprepper.plugins.source.microsoft_office365.Office365Source - Starting Office365 Source Plugin... 2026-02-04T12:31:45,041 [pipeline-sink-worker-2-thread-1] INFO org.opensearch.dataprepper.plugins.source.source_crawler.auth.AuthenticationInterface - Initializing credentials. 2026-02-04T12:31:45,041 [pipeline-sink-worker-2-thread-1] INFO org.opensearch.dataprepper.plugins.source.microsoft_office365.auth.Office365AuthenticationProvider - Getting new access token for Office 365 Management API 2026-02-04T12:31:45,046 [pipeline-sink-worker-2-thread-1] INFO org.opensearch.dataprepper.plugins.aws.AwsSecretsSupplier - Retrieving latest secrets in aws:secrets:m365_secret. 2026-02-04T12:31:45,060 [main] WARN org.opensearch.dataprepper.core.pipeline.server.HttpServerProvider - Creating Data Prepper server without TLS. This is not secure. 2026-02-04T12:31:45,060 [main] WARN org.opensearch.dataprepper.core.pipeline.server.HttpServerProvider - In order to set up TLS for the Data Prepper server, go here: https://github.com/opensearch-project/data-prepper/blob/main/docs/configuration.md#server-configuration 2026-02-04T12:31:45,117 [main] INFO org.opensearch.dataprepper.core.pipeline.server.DataPrepperServer - Data Prepper server running at :4903 2026-02-04T12:31:45,288 [pipeline-sink-worker-2-thread-1] INFO org.opensearch.dataprepper.plugins.aws.AwsSecretsSupplier - Finished retrieving latest secret in aws:secrets:m365_secret. 2026-02-04T12:31:45,289 [pipeline-sink-worker-2-thread-1] INFO org.opensearch.dataprepper.plugins.aws.AwsSecretsSupplier - Retrieving latest secrets in aws:secrets:m365_secret. 2026-02-04T12:31:45,522 [pipeline-sink-worker-2-thread-1] INFO org.opensearch.dataprepper.plugins.aws.AwsSecretsSupplier - Finished retrieving latest secret in aws:secrets:m365_secret. 2026-02-04T12:31:45,882 [pipeline-sink-worker-2-thread-1] INFO org.opensearch.dataprepper.plugins.source.microsoft_office365.auth.Office365AuthenticationProvider - Received new access token. Expires in 3599 seconds 2026-02-04T12:31:45,882 [pipeline-sink-worker-2-thread-1] INFO org.opensearch.dataprepper.plugins.source.source_crawler.auth.AuthenticationInterface - Credentials initialized successfully 2026-02-04T12:31:45,882 [pipeline-sink-worker-2-thread-1] INFO org.opensearch.dataprepper.plugins.source.microsoft_office365.Office365RestClient - Starting Office 365 subscriptions for audit logs 2026-02-04T12:31:46,482 [pipeline-sink-worker-2-thread-1] INFO org.opensearch.dataprepper.plugins.source.source_crawler.base.CrawlerSourcePlugin - Starting microsoft_office365 Source Plugin 2026-02-04T12:31:46,494 [pipeline-sink-worker-2-thread-1] INFO org.opensearch.dataprepper.core.pipeline.Pipeline - Pipeline [pipeline] - Submitting request to initiate the pipeline processing 2026-02-04T12:31:46,494 [pool-7-thread-4] INFO org.opensearch.dataprepper.plugins.source.source_crawler.coordination.scheduler.WorkerScheduler - Worker thread started 2026-02-04T12:31:46,494 [pool-7-thread-2] INFO org.opensearch.dataprepper.plugins.source.source_crawler.coordination.scheduler.WorkerScheduler - Worker thread started 2026-02-04T12:31:46,494 [pool-7-thread-3] INFO org.opensearch.dataprepper.plugins.source.source_crawler.coordination.scheduler.WorkerScheduler - Worker thread started 2026-02-04T12:31:46,494 [pool-7-thread-4] INFO org.opensearch.dataprepper.plugins.source.source_crawler.coordination.scheduler.WorkerScheduler - Processing Partitions 2026-02-04T12:31:46,494 [pool-7-thread-5] INFO org.opensearch.dataprepper.plugins.source.source_crawler.coordination.scheduler.WorkerScheduler - Worker thread started 2026-02-04T12:31:46,494 [pool-7-thread-2] INFO org.opensearch.dataprepper.plugins.source.source_crawler.coordination.scheduler.WorkerScheduler - Processing Partitions 2026-02-04T12:31:46,494 [pool-7-thread-3] INFO org.opensearch.dataprepper.plugins.source.source_crawler.coordination.scheduler.WorkerScheduler - Processing Partitions 2026-02-04T12:31:46,494 [pool-7-thread-5] INFO org.opensearch.dataprepper.plugins.source.source_crawler.coordination.scheduler.WorkerScheduler - Processing Partitions 2026-02-04T12:31:46,498 [pool-7-thread-1] INFO org.opensearch.dataprepper.plugins.source.source_crawler.coordination.scheduler.LeaderScheduler - Running as a LEADER node 2026-02-04T12:31:46,498 [pool-7-thread-1] INFO org.opensearch.dataprepper.plugins.source.source_crawler.base.DimensionalTimeSliceCrawler - Creating partition for sub-hour historical pull: 5 minutes 2026-02-04T12:31:46,502 [pool-7-thread-1] INFO org.opensearch.dataprepper.plugins.source.source_crawler.base.DimensionalTimeSliceCrawler - Total partitions created in this crawl: 5.0 2026-02-04T12:31:56,508 [pool-7-thread-3] INFO org.opensearch.dataprepper.plugins.source.source_crawler.base.DimensionalTimeSliceCrawler - Processing partition - DimensionType: Audit.SharePoint, TimeRange: 2026-02-04T06:56:46.483159Z to 2026-02-04T07:01:46.483159Z 2026-02-04T12:31:56,508 [pool-7-thread-5] INFO org.opensearch.dataprepper.plugins.source.source_crawler.base.DimensionalTimeSliceCrawler - Processing partition - DimensionType: Audit.Exchange, TimeRange: 2026-02-04T06:56:46.483159Z to 2026-02-04T07:01:46.483159Z 2026-02-04T12:31:56,508 [pool-7-thread-4] INFO org.opensearch.dataprepper.plugins.source.source_crawler.base.DimensionalTimeSliceCrawler - Processing partition - DimensionType: Audit.General, TimeRange: 2026-02-04T06:56:46.483159Z to 2026-02-04T07:01:46.483159Z 2026-02-04T12:31:56,510 [pool-7-thread-2] INFO org.opensearch.dataprepper.plugins.source.source_crawler.base.DimensionalTimeSliceCrawler - Processing partition - DimensionType: Audit.AzureActiveDirectory, TimeRange: 2026-02-04T06:56:46.483159Z to 2026-02-04T07:01:46.483159Z 2026-02-04T12:31:56,691 [pool-7-thread-3] INFO org.opensearch.dataprepper.plugins.source.source_crawler.base.DimensionalTimeSliceCrawler - Processing partition - DimensionType: DLP.All, TimeRange: 2026-02-04T06:56:46.483159Z to 2026-02-04T07:01:46.483159Zrange: PT10M
> Task :data-prepper-main:org.opensearch.dataprepper.DataPrepperExecute.main() 2026-02-04T12:33:06,815 [main] INFO org.opensearch.dataprepper.DataPrepperExecute - Data Prepper 2.14 2026-02-04T12:33:08,144 [main] INFO org.opensearch.dataprepper.core.DataPrepperArgumentConfiguration - Command line args: C:\connectors\amazon-data-prepper\data-prepper\pipelines,C:\connectors\amazon-data-prepper\data-prepper\config\data-prepper-config.yaml 2026-02-04T12:33:08,144 [main] INFO org.opensearch.dataprepper.core.DataPrepperArgs - Using C:\connectors\amazon-data-prepper\data-prepper\pipelines configuration file 2026-02-04T12:33:08,443 [main] INFO org.opensearch.dataprepper.pipeline.parser.transformer.DynamicConfigTransformer - No transformation needed 2026-02-04T12:33:09,398 [main] INFO org.opensearch.dataprepper.plugin.ExtensionsApplier - Loaded 6 extensions: [org.opensearch.dataprepper.plugins.aws.AwsPlugin@5c448ef, org.opensearch.dataprepper.plugins.aws.AwsSecretPlugin@703e8050, org.opensearch.dataprepper.plugins.kafka.extension.KafkaClusterConfigExtension@49c1e294, org.opensearch.dataprepper.plugins.encryption.EncryptionPlugin@7741ae1b, org.opensearch.dataprepper.plugins.kinesis.extension.KinesisLeaseConfigExtension@50e5032c, org.opensearch.dataprepper.plugins.geoip.extension.GeoIpConfigExtension@545d2560] 2026-02-04T12:33:13,185 [main] INFO org.opensearch.dataprepper.plugins.kafka.extension.KafkaClusterConfigExtension - Applying Kafka Cluster Config Extension. 2026-02-04T12:33:13,449 [main] INFO org.opensearch.dataprepper.core.parser.PipelineTransformer - Building pipeline [pipeline] from provided configuration 2026-02-04T12:33:13,450 [main] INFO org.opensearch.dataprepper.core.parser.PipelineTransformer - Building [microsoft_office365] as source component for the pipeline [pipeline] 2026-02-04T12:33:14,062 [main] INFO org.opensearch.dataprepper.plugins.source.microsoft_office365.Office365Source - Creating Office365 Source Plugin 2026-02-04T12:33:14,063 [main] INFO org.opensearch.dataprepper.core.parser.PipelineTransformer - Building buffer for the pipeline [pipeline] 2026-02-04T12:33:14,072 [main] INFO org.opensearch.dataprepper.core.parser.PipelineTransformer - Building processors for the pipeline [pipeline] 2026-02-04T12:33:14,073 [main] INFO org.opensearch.dataprepper.core.parser.PipelineTransformer - Building sinks for the pipeline [pipeline] 2026-02-04T12:33:14,073 [main] INFO org.opensearch.dataprepper.core.parser.PipelineTransformer - Building [stdout] as sink component 2026-02-04T12:33:14,077 [main] INFO org.opensearch.dataprepper.core.parser.PipelineTransformer - Constructing MultiBufferDecorator with [0] secondary buffers for pipeline [pipeline] 2026-02-04T12:33:14,175 [main] WARN org.opensearch.dataprepper.core.pipeline.server.config.DataPrepperServerConfiguration - Creating data prepper server without authentication. This is not secure. 2026-02-04T12:33:14,175 [main] WARN org.opensearch.dataprepper.core.pipeline.server.config.DataPrepperServerConfiguration - In order to set up Http Basic authentication for the data prepper server, go here: https://github.com/opensearch-project/data-prepper/blob/main/docs/core_apis.md#authentication 2026-02-04T12:33:14,254 [main] INFO org.opensearch.dataprepper.core.pipeline.Pipeline - Pipeline [pipeline] - Initiating pipeline execution 2026-02-04T12:33:14,257 [main] INFO org.opensearch.dataprepper.core.sourcecoordination.SourceCoordinatorFactory - Creating EnhancedLeaseBasedSourceCoordinator with coordination store in_memory for sub-pipeline pipeline 2026-02-04T12:33:14,260 [main] WARN org.opensearch.dataprepper.plugins.sourcecoordinator.inmemory.InMemorySourceCoordinationStore - The in_memory source coordination store is not recommended for production workloads. It is only effective in single node environments of Data Prepper, and can run into memory limitations over time if the number of partitions is too great. 2026-02-04T12:33:14,265 [pipeline-sink-worker-2-thread-1] INFO org.opensearch.dataprepper.core.pipeline.Pipeline - Pipeline [pipeline] Sink is ready, starting source... 2026-02-04T12:33:14,266 [pipeline-sink-worker-2-thread-1] INFO org.opensearch.dataprepper.plugins.source.microsoft_office365.Office365Source - Starting Office365 Source Plugin... 2026-02-04T12:33:14,266 [pipeline-sink-worker-2-thread-1] INFO org.opensearch.dataprepper.plugins.source.source_crawler.auth.AuthenticationInterface - Initializing credentials. 2026-02-04T12:33:14,266 [pipeline-sink-worker-2-thread-1] INFO org.opensearch.dataprepper.plugins.source.microsoft_office365.auth.Office365AuthenticationProvider - Getting new access token for Office 365 Management API 2026-02-04T12:33:14,269 [pipeline-sink-worker-2-thread-1] INFO org.opensearch.dataprepper.plugins.aws.AwsSecretsSupplier - Retrieving latest secrets in aws:secrets:m365_secret. 2026-02-04T12:33:14,277 [main] WARN org.opensearch.dataprepper.core.pipeline.server.HttpServerProvider - Creating Data Prepper server without TLS. This is not secure. 2026-02-04T12:33:14,278 [main] WARN org.opensearch.dataprepper.core.pipeline.server.HttpServerProvider - In order to set up TLS for the Data Prepper server, go here: https://github.com/opensearch-project/data-prepper/blob/main/docs/configuration.md#server-configuration 2026-02-04T12:33:14,302 [main] INFO org.opensearch.dataprepper.core.pipeline.server.DataPrepperServer - Data Prepper server running at :4903 2026-02-04T12:33:14,512 [pipeline-sink-worker-2-thread-1] INFO org.opensearch.dataprepper.plugins.aws.AwsSecretsSupplier - Finished retrieving latest secret in aws:secrets:m365_secret. 2026-02-04T12:33:14,512 [pipeline-sink-worker-2-thread-1] INFO org.opensearch.dataprepper.plugins.aws.AwsSecretsSupplier - Retrieving latest secrets in aws:secrets:m365_secret. 2026-02-04T12:33:14,788 [pipeline-sink-worker-2-thread-1] INFO org.opensearch.dataprepper.plugins.aws.AwsSecretsSupplier - Finished retrieving latest secret in aws:secrets:m365_secret. 2026-02-04T12:33:15,176 [pipeline-sink-worker-2-thread-1] INFO org.opensearch.dataprepper.plugins.source.microsoft_office365.auth.Office365AuthenticationProvider - Received new access token. Expires in 3599 seconds 2026-02-04T12:33:15,176 [pipeline-sink-worker-2-thread-1] INFO org.opensearch.dataprepper.plugins.source.source_crawler.auth.AuthenticationInterface - Credentials initialized successfully 2026-02-04T12:33:15,176 [pipeline-sink-worker-2-thread-1] INFO org.opensearch.dataprepper.plugins.source.microsoft_office365.Office365RestClient - Starting Office 365 subscriptions for audit logs 2026-02-04T12:33:15,743 [pipeline-sink-worker-2-thread-1] INFO org.opensearch.dataprepper.plugins.source.source_crawler.base.CrawlerSourcePlugin - Starting microsoft_office365 Source Plugin 2026-02-04T12:33:15,758 [pipeline-sink-worker-2-thread-1] INFO org.opensearch.dataprepper.core.pipeline.Pipeline - Pipeline [pipeline] - Submitting request to initiate the pipeline processing 2026-02-04T12:33:15,758 [pool-7-thread-3] INFO org.opensearch.dataprepper.plugins.source.source_crawler.coordination.scheduler.WorkerScheduler - Worker thread started 2026-02-04T12:33:15,758 [pool-7-thread-4] INFO org.opensearch.dataprepper.plugins.source.source_crawler.coordination.scheduler.WorkerScheduler - Worker thread started 2026-02-04T12:33:15,759 [pool-7-thread-3] INFO org.opensearch.dataprepper.plugins.source.source_crawler.coordination.scheduler.WorkerScheduler - Processing Partitions 2026-02-04T12:33:15,759 [pool-7-thread-4] INFO org.opensearch.dataprepper.plugins.source.source_crawler.coordination.scheduler.WorkerScheduler - Processing Partitions 2026-02-04T12:33:15,758 [pool-7-thread-2] INFO org.opensearch.dataprepper.plugins.source.source_crawler.coordination.scheduler.WorkerScheduler - Worker thread started 2026-02-04T12:33:15,759 [pool-7-thread-2] INFO org.opensearch.dataprepper.plugins.source.source_crawler.coordination.scheduler.WorkerScheduler - Processing Partitions 2026-02-04T12:33:15,759 [pool-7-thread-5] INFO org.opensearch.dataprepper.plugins.source.source_crawler.coordination.scheduler.WorkerScheduler - Worker thread started 2026-02-04T12:33:15,759 [pool-7-thread-5] INFO org.opensearch.dataprepper.plugins.source.source_crawler.coordination.scheduler.WorkerScheduler - Processing Partitions 2026-02-04T12:33:15,763 [pool-7-thread-1] INFO org.opensearch.dataprepper.plugins.source.source_crawler.coordination.scheduler.LeaderScheduler - Running as a LEADER node 2026-02-04T12:33:15,764 [pool-7-thread-1] INFO org.opensearch.dataprepper.plugins.source.source_crawler.base.DimensionalTimeSliceCrawler - Creating partition for sub-hour historical pull: 10 minutes 2026-02-04T12:33:15,767 [pool-7-thread-1] INFO org.opensearch.dataprepper.plugins.source.source_crawler.base.DimensionalTimeSliceCrawler - Total partitions created in this crawl: 5.0 2026-02-04T12:33:25,771 [pool-7-thread-3] INFO org.opensearch.dataprepper.plugins.source.source_crawler.base.DimensionalTimeSliceCrawler - Processing partition - DimensionType: Audit.SharePoint, TimeRange: 2026-02-04T06:53:15.745665200Z to 2026-02-04T06:58:15.745665200Z 2026-02-04T12:33:25,771 [pool-7-thread-5] INFO org.opensearch.dataprepper.plugins.source.source_crawler.base.DimensionalTimeSliceCrawler - Processing partition - DimensionType: Audit.Exchange, TimeRange: 2026-02-04T06:53:15.745665200Z to 2026-02-04T06:58:15.745665200Z 2026-02-04T12:33:25,771 [pool-7-thread-2] INFO org.opensearch.dataprepper.plugins.source.source_crawler.base.DimensionalTimeSliceCrawler - Processing partition - DimensionType: Audit.General, TimeRange: 2026-02-04T06:53:15.745665200Z to 2026-02-04T06:58:15.745665200Z 2026-02-04T12:33:25,771 [pool-7-thread-4] INFO org.opensearch.dataprepper.plugins.source.source_crawler.base.DimensionalTimeSliceCrawler - Processing partition - DimensionType: Audit.AzureActiveDirectory, TimeRange: 2026-02-04T06:53:15.745665200Z to 2026-02-04T06:58:15.745665200Z 2026-02-04T12:33:25,966 [pool-7-thread-2] INFO org.opensearch.dataprepper.plugins.source.source_crawler.base.DimensionalTimeSliceCrawler - Processing partition - DimensionType: DLP.All, TimeRange: 2026-02-04T06:53:15.745665200Z to 2026-02-04T06:58:15.745665200Z {"CreationTime":"2026-02-04T06:53:06","Id":"401c13f3-d544-4896-a24b-de46265ff4b0","Operation":"Update user.","OrganizationId":"e822651b-5027-4253-83f5-904854601a3b","RecordType":8,"ResultStatus":"Success","UserKey":"Not Available","UserType":4,"Version":1,"Workload":"AzureActiveDirectory","ObjectId":"demo.m3connector@trianzazuresb.onmicrosoft.com","UserId":"ServicePrincipal_3616d279-e97d-48d3-af3e-74ed7de78faf","AzureActiveDirectoryEventType":1,"ExtendedProperties":[{"Name":"additionalDetails","Value":"{\"UserType\":\"Member\",\"User-Agent\":\"Apache-HttpClient/4.5.13 (Java/17.0.17)\"}"},{"Name":"extendedAuditEventCategory","Value":"User"}],"ModifiedProperties":[{"Name":"JobTitle","NewValue":"[\r\n \"Updated by Canary test at 2026-02-04T06:53:04.971539353Z\"\r\n]","OldValue":"[\r\n \"Updated by Canary test at 2026-02-04T06:22:06.062468072Z\"\r\n]"},{"Name":"Included Updated Properties","NewValue":"JobTitle","OldValue":""},{"Name":"TargetId.UserType","NewValue":"Member","OldValue":""},{"Name":"ActorId.ServicePrincipalNames","NewValue":"fb6b0f13-8f1e-4a28-a772-d32d3133da23","OldValue":""},{"Name":"SPN","NewValue":"fb6b0f13-8f1e-4a28-a772-d32d3133da23","OldValue":""}],"Actor":[{"ID":"entraId_app","Type":1},{"ID":"fb6b0f13-8f1e-4a28-a772-d32d3133da23","Type":2},{"ID":"ServicePrincipal_3616d279-e97d-48d3-af3e-74ed7de78faf","Type":2},{"ID":"3616d279-e97d-48d3-af3e-74ed7de78faf","Type":2},{"ID":"ServicePrincipal","Type":2}],"ActorContextId":"e822651b-5027-4253-83f5-904854601a3b","InterSystemsId":"5860d0fa-ed35-4a8a-80c4-54c332c52456","IntraSystemId":"c733dcb0-12e3-4504-97da-1c75b33bcbcf","SupportTicketId":"","Target":[{"ID":"User_0fba2a0b-2680-45c1-9ae6-d20b74edb3ec","Type":2},{"ID":"0fba2a0b-2680-45c1-9ae6-d20b74edb3ec","Type":2},{"ID":"User","Type":2},{"ID":"demo.m3connector@trianzazuresb.onmicrosoft.com","Type":5},{"ID":"1003200511619FF9","Type":3}],"TargetContextId":"e822651b-5027-4253-83f5-904854601a3b"}range: PT15M
> Task :data-prepper-main:org.opensearch.dataprepper.DataPrepperExecute.main() 2026-02-04T12:34:45,640 [main] INFO org.opensearch.dataprepper.DataPrepperExecute - Data Prepper 2.14 2026-02-04T12:34:46,866 [main] INFO org.opensearch.dataprepper.core.DataPrepperArgumentConfiguration - Command line args: C:\connectors\amazon-data-prepper\data-prepper\pipelines,C:\connectors\amazon-data-prepper\data-prepper\config\data-prepper-config.yaml 2026-02-04T12:34:46,867 [main] INFO org.opensearch.dataprepper.core.DataPrepperArgs - Using C:\connectors\amazon-data-prepper\data-prepper\pipelines configuration file 2026-02-04T12:34:47,129 [main] INFO org.opensearch.dataprepper.pipeline.parser.transformer.DynamicConfigTransformer - No transformation needed 2026-02-04T12:34:48,073 [main] INFO org.opensearch.dataprepper.plugin.ExtensionsApplier - Loaded 6 extensions: [org.opensearch.dataprepper.plugins.aws.AwsPlugin@17ec335a, org.opensearch.dataprepper.plugins.aws.AwsSecretPlugin@284990de, org.opensearch.dataprepper.plugins.kafka.extension.KafkaClusterConfigExtension@565d7d2f, org.opensearch.dataprepper.plugins.encryption.EncryptionPlugin@6aea99e7, org.opensearch.dataprepper.plugins.kinesis.extension.KinesisLeaseConfigExtension@1c6a0103, org.opensearch.dataprepper.plugins.geoip.extension.GeoIpConfigExtension@7841bd30] 2026-02-04T12:34:51,671 [main] INFO org.opensearch.dataprepper.plugins.kafka.extension.KafkaClusterConfigExtension - Applying Kafka Cluster Config Extension. 2026-02-04T12:34:51,935 [main] INFO org.opensearch.dataprepper.core.parser.PipelineTransformer - Building pipeline [pipeline] from provided configuration 2026-02-04T12:34:51,936 [main] INFO org.opensearch.dataprepper.core.parser.PipelineTransformer - Building [microsoft_office365] as source component for the pipeline [pipeline] 2026-02-04T12:34:52,461 [main] INFO org.opensearch.dataprepper.plugins.source.microsoft_office365.Office365Source - Creating Office365 Source Plugin 2026-02-04T12:34:52,461 [main] INFO org.opensearch.dataprepper.core.parser.PipelineTransformer - Building buffer for the pipeline [pipeline] 2026-02-04T12:34:52,469 [main] INFO org.opensearch.dataprepper.core.parser.PipelineTransformer - Building processors for the pipeline [pipeline] 2026-02-04T12:34:52,469 [main] INFO org.opensearch.dataprepper.core.parser.PipelineTransformer - Building sinks for the pipeline [pipeline] 2026-02-04T12:34:52,469 [main] INFO org.opensearch.dataprepper.core.parser.PipelineTransformer - Building [stdout] as sink component 2026-02-04T12:34:52,471 [main] INFO org.opensearch.dataprepper.core.parser.PipelineTransformer - Constructing MultiBufferDecorator with [0] secondary buffers for pipeline [pipeline] 2026-02-04T12:34:52,568 [main] WARN org.opensearch.dataprepper.core.pipeline.server.config.DataPrepperServerConfiguration - Creating data prepper server without authentication. This is not secure. 2026-02-04T12:34:52,568 [main] WARN org.opensearch.dataprepper.core.pipeline.server.config.DataPrepperServerConfiguration - In order to set up Http Basic authentication for the data prepper server, go here: https://github.com/opensearch-project/data-prepper/blob/main/docs/core_apis.md#authentication 2026-02-04T12:34:52,663 [main] INFO org.opensearch.dataprepper.core.pipeline.Pipeline - Pipeline [pipeline] - Initiating pipeline execution 2026-02-04T12:34:52,668 [main] INFO org.opensearch.dataprepper.core.sourcecoordination.SourceCoordinatorFactory - Creating EnhancedLeaseBasedSourceCoordinator with coordination store in_memory for sub-pipeline pipeline 2026-02-04T12:34:52,671 [main] WARN org.opensearch.dataprepper.plugins.sourcecoordinator.inmemory.InMemorySourceCoordinationStore - The in_memory source coordination store is not recommended for production workloads. It is only effective in single node environments of Data Prepper, and can run into memory limitations over time if the number of partitions is too great. 2026-02-04T12:34:52,675 [pipeline-sink-worker-2-thread-1] INFO org.opensearch.dataprepper.core.pipeline.Pipeline - Pipeline [pipeline] Sink is ready, starting source... 2026-02-04T12:34:52,676 [pipeline-sink-worker-2-thread-1] INFO org.opensearch.dataprepper.plugins.source.microsoft_office365.Office365Source - Starting Office365 Source Plugin... 2026-02-04T12:34:52,676 [pipeline-sink-worker-2-thread-1] INFO org.opensearch.dataprepper.plugins.source.source_crawler.auth.AuthenticationInterface - Initializing credentials. 2026-02-04T12:34:52,676 [pipeline-sink-worker-2-thread-1] INFO org.opensearch.dataprepper.plugins.source.microsoft_office365.auth.Office365AuthenticationProvider - Getting new access token for Office 365 Management API 2026-02-04T12:34:52,680 [pipeline-sink-worker-2-thread-1] INFO org.opensearch.dataprepper.plugins.aws.AwsSecretsSupplier - Retrieving latest secrets in aws:secrets:m365_secret. 2026-02-04T12:34:52,695 [main] WARN org.opensearch.dataprepper.core.pipeline.server.HttpServerProvider - Creating Data Prepper server without TLS. This is not secure. 2026-02-04T12:34:52,695 [main] WARN org.opensearch.dataprepper.core.pipeline.server.HttpServerProvider - In order to set up TLS for the Data Prepper server, go here: https://github.com/opensearch-project/data-prepper/blob/main/docs/configuration.md#server-configuration 2026-02-04T12:34:52,729 [main] INFO org.opensearch.dataprepper.core.pipeline.server.DataPrepperServer - Data Prepper server running at :4903 2026-02-04T12:34:52,950 [pipeline-sink-worker-2-thread-1] INFO org.opensearch.dataprepper.plugins.aws.AwsSecretsSupplier - Finished retrieving latest secret in aws:secrets:m365_secret. 2026-02-04T12:34:52,950 [pipeline-sink-worker-2-thread-1] INFO org.opensearch.dataprepper.plugins.aws.AwsSecretsSupplier - Retrieving latest secrets in aws:secrets:m365_secret. 2026-02-04T12:34:53,202 [pipeline-sink-worker-2-thread-1] INFO org.opensearch.dataprepper.plugins.aws.AwsSecretsSupplier - Finished retrieving latest secret in aws:secrets:m365_secret. 2026-02-04T12:34:53,536 [pipeline-sink-worker-2-thread-1] INFO org.opensearch.dataprepper.plugins.source.microsoft_office365.auth.Office365AuthenticationProvider - Received new access token. Expires in 3599 seconds 2026-02-04T12:34:53,536 [pipeline-sink-worker-2-thread-1] INFO org.opensearch.dataprepper.plugins.source.source_crawler.auth.AuthenticationInterface - Credentials initialized successfully 2026-02-04T12:34:53,536 [pipeline-sink-worker-2-thread-1] INFO org.opensearch.dataprepper.plugins.source.microsoft_office365.Office365RestClient - Starting Office 365 subscriptions for audit logs 2026-02-04T12:34:54,147 [pipeline-sink-worker-2-thread-1] INFO org.opensearch.dataprepper.plugins.source.source_crawler.base.CrawlerSourcePlugin - Starting microsoft_office365 Source Plugin 2026-02-04T12:34:54,166 [pipeline-sink-worker-2-thread-1] INFO org.opensearch.dataprepper.core.pipeline.Pipeline - Pipeline [pipeline] - Submitting request to initiate the pipeline processing 2026-02-04T12:34:54,166 [pool-7-thread-2] INFO org.opensearch.dataprepper.plugins.source.source_crawler.coordination.scheduler.WorkerScheduler - Worker thread started 2026-02-04T12:34:54,166 [pool-7-thread-4] INFO org.opensearch.dataprepper.plugins.source.source_crawler.coordination.scheduler.WorkerScheduler - Worker thread started 2026-02-04T12:34:54,166 [pool-7-thread-3] INFO org.opensearch.dataprepper.plugins.source.source_crawler.coordination.scheduler.WorkerScheduler - Worker thread started 2026-02-04T12:34:54,166 [pool-7-thread-4] INFO org.opensearch.dataprepper.plugins.source.source_crawler.coordination.scheduler.WorkerScheduler - Processing Partitions 2026-02-04T12:34:54,166 [pool-7-thread-2] INFO org.opensearch.dataprepper.plugins.source.source_crawler.coordination.scheduler.WorkerScheduler - Processing Partitions 2026-02-04T12:34:54,166 [pool-7-thread-3] INFO org.opensearch.dataprepper.plugins.source.source_crawler.coordination.scheduler.WorkerScheduler - Processing Partitions 2026-02-04T12:34:54,166 [pool-7-thread-5] INFO org.opensearch.dataprepper.plugins.source.source_crawler.coordination.scheduler.WorkerScheduler - Worker thread started 2026-02-04T12:34:54,166 [pool-7-thread-5] INFO org.opensearch.dataprepper.plugins.source.source_crawler.coordination.scheduler.WorkerScheduler - Processing Partitions 2026-02-04T12:34:54,169 [pool-7-thread-1] INFO org.opensearch.dataprepper.plugins.source.source_crawler.coordination.scheduler.LeaderScheduler - Running as a LEADER node 2026-02-04T12:34:54,170 [pool-7-thread-1] INFO org.opensearch.dataprepper.plugins.source.source_crawler.base.DimensionalTimeSliceCrawler - Creating partition for sub-hour historical pull: 15 minutes 2026-02-04T12:34:54,174 [pool-7-thread-1] INFO org.opensearch.dataprepper.plugins.source.source_crawler.base.DimensionalTimeSliceCrawler - Total partitions created in this crawl: 5.0 2026-02-04T12:35:04,180 [pool-7-thread-2] INFO org.opensearch.dataprepper.plugins.source.source_crawler.base.DimensionalTimeSliceCrawler - Processing partition - DimensionType: Audit.SharePoint, TimeRange: 2026-02-04T06:49:54.151117700Z to 2026-02-04T06:59:54.151117700Z 2026-02-04T12:35:04,180 [pool-7-thread-3] INFO org.opensearch.dataprepper.plugins.source.source_crawler.base.DimensionalTimeSliceCrawler - Processing partition - DimensionType: DLP.All, TimeRange: 2026-02-04T06:49:54.151117700Z to 2026-02-04T06:59:54.151117700Z 2026-02-04T12:35:04,180 [pool-7-thread-5] INFO org.opensearch.dataprepper.plugins.source.source_crawler.base.DimensionalTimeSliceCrawler - Processing partition - DimensionType: Audit.AzureActiveDirectory, TimeRange: 2026-02-04T06:49:54.151117700Z to 2026-02-04T06:59:54.151117700Z 2026-02-04T12:35:04,180 [pool-7-thread-4] INFO org.opensearch.dataprepper.plugins.source.source_crawler.base.DimensionalTimeSliceCrawler - Processing partition - DimensionType: Audit.General, TimeRange: 2026-02-04T06:49:54.151117700Z to 2026-02-04T06:59:54.151117700Z 2026-02-04T12:35:04,370 [pool-7-thread-4] INFO org.opensearch.dataprepper.plugins.source.source_crawler.base.DimensionalTimeSliceCrawler - Processing partition - DimensionType: Audit.Exchange, TimeRange: 2026-02-04T06:49:54.151117700Z to 2026-02-04T06:59:54.151117700Z {"CreationTime":"2026-02-04T06:53:06","Id":"401c13f3-d544-4896-a24b-de46265ff4b0","Operation":"Update user.","OrganizationId":"e822651b-5027-4253-83f5-904854601a3b","RecordType":8,"ResultStatus":"Success","UserKey":"Not Available","UserType":4,"Version":1,"Workload":"AzureActiveDirectory","ObjectId":"demo.m3connector@trianzazuresb.onmicrosoft.com","UserId":"ServicePrincipal_3616d279-e97d-48d3-af3e-74ed7de78faf","AzureActiveDirectoryEventType":1,"ExtendedProperties":[{"Name":"additionalDetails","Value":"{\"UserType\":\"Member\",\"User-Agent\":\"Apache-HttpClient/4.5.13 (Java/17.0.17)\"}"},{"Name":"extendedAuditEventCategory","Value":"User"}],"ModifiedProperties":[{"Name":"JobTitle","NewValue":"[\r\n \"Updated by Canary test at 2026-02-04T06:53:04.971539353Z\"\r\n]","OldValue":"[\r\n \"Updated by Canary test at 2026-02-04T06:22:06.062468072Z\"\r\n]"},{"Name":"Included Updated Properties","NewValue":"JobTitle","OldValue":""},{"Name":"TargetId.UserType","NewValue":"Member","OldValue":""},{"Name":"ActorId.ServicePrincipalNames","NewValue":"fb6b0f13-8f1e-4a28-a772-d32d3133da23","OldValue":""},{"Name":"SPN","NewValue":"fb6b0f13-8f1e-4a28-a772-d32d3133da23","OldValue":""}],"Actor":[{"ID":"entraId_app","Type":1},{"ID":"fb6b0f13-8f1e-4a28-a772-d32d3133da23","Type":2},{"ID":"ServicePrincipal_3616d279-e97d-48d3-af3e-74ed7de78faf","Type":2},{"ID":"3616d279-e97d-48d3-af3e-74ed7de78faf","Type":2},{"ID":"ServicePrincipal","Type":2}],"ActorContextId":"e822651b-5027-4253-83f5-904854601a3b","InterSystemsId":"5860d0fa-ed35-4a8a-80c4-54c332c52456","IntraSystemId":"c733dcb0-12e3-4504-97da-1c75b33bcbcf","SupportTicketId":"","Target":[{"ID":"User_0fba2a0b-2680-45c1-9ae6-d20b74edb3ec","Type":2},{"ID":"0fba2a0b-2680-45c1-9ae6-d20b74edb3ec","Type":2},{"ID":"User","Type":2},{"ID":"demo.m3connector@trianzazuresb.onmicrosoft.com","Type":5},{"ID":"1003200511619FF9","Type":3}],"TargetContextId":"e822651b-5027-4253-83f5-904854601a3b"}range: PT30
2026-01-21T18:41:37,517 [pool-7-thread-2] INFO org.opensearch.dataprepper.plugins.source.source_crawler.base.DimensionalTimeSliceCrawler - Processing partition - DimensionType: Audit.SharePoint, TimeRange: 2026-01-21T12:41:27.494124500Z to 2026-01-21T13:06:27.494124500Z 2026-01-21T18:41:44,702 [pool-7-thread-4] INFO org.opensearch.dataprepper.plugins.source.source_crawler.base.DimensionalTimeSliceCrawler - Processing partition - DimensionType: DLP.All, TimeRange: 2026-01-21T12:41:27.494124500Z to 2026-01-21T13:06:27.494124500Z {"CreationTime":"2026-01-21T12:37:19","Id":"991b2de8-6271-4c69-a8aa-d6093eda1c00","Operation":"UserLoggedIn","OrganizationId":"e822651b-5027-4253-83f5-904854601a3b","RecordType":15,"ResultStatus":"Success","UserKey":"b5166e08-948e-4ee7-a3a8-71132bb1eda6","UserType":0,"Version":1,"Workload":"AzureActiveDirectory","ClientIP":"136.226.231.85","ObjectId":"00000003-0000-0000-c000-000000000000","UserId":"adarsh.verma@trianz.com","AzureActiveDirectoryEventType":1,"ExtendedProperties":[{"Name":"ResultStatusDetail","Value":"Success"},{"Name":"KeepMeSignedIn","Value":"true"},{"Name":"UserAgent","Value":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/144.0.0.0 Safari/537.36"},{"Name":"UserAuthenticationMethod","Value":"16"},{"Name":"RequestType","Value":"Login:login"}],"ModifiedProperties":[],"Actor":[{"ID":"b5166e08-948e-4ee7-a3a8-71132bb1eda6","Type":0},{"ID":"adarsh.verma@trianz.com","Type":5}],"ActorContextId":"a27f6b6b-dc28-48c2-a138-ddf0f11455f1","ActorIpAddress":"136.226.231.85","InterSystemsId":"5ae0a675-6b57-4738-9847-ab7c81e126ba","IntraSystemId":"991b2de8-6271-4c69-a8aa-d6093eda1c00","SupportTicketId":"","Target":[{"ID":"00000003-0000-0000-c000-000000000000","Type":0}],"TargetContextId":"e822651b-5027-4253-83f5-904854601a3b","ApplicationId":"1e92066f-7ee4-493b-9d96-5af35ba80990","ErrorNumber":"50140"} 2026-01-21T18:42:27,532 [pool-7-thread-1] INFO org.opensearch.dataprepper.plugins.source.source_crawler.base.DimensionalTimeSliceCrawler - Total partitions created in this crawl: 5.0 2026-01-21T18:42:34,766 [pool-7-thread-3] INFO org.opensearch.dataprepper.plugins.source.source_crawler.base.DimensionalTimeSliceCrawler - Processing partition - DimensionType: DLP.All, TimeRange: 2026-01-21T13:06:27.494124500Z to 2026-01-21T13:07:27.530240300Zrange: PT2H
2026-01-21T18:49:31,885 [pool-7-thread-4] INFO org.opensearch.dataprepper.plugins.source.source_crawler.base.DimensionalTimeSliceCrawler - Processing partition - DimensionType: DLP.All, TimeRange: 2026-01-21T13:00:00Z to 2026-01-21T13:14:21.379582900Z 2026-01-21T18:49:31,910 [pool-7-thread-2] INFO org.opensearch.dataprepper.plugins.source.source_crawler.base.DimensionalTimeSliceCrawler - Processing partition - DimensionType: Audit.SharePoint, TimeRange: 2026-01-21T13:00:00Z to 2026-01-21T13:14:21.379582900Z {"CreationTime":"2026-01-21T11:22:47","Id":"8c616f61-ad25-452a-b8dd-0c5c41e212e3","Operation":"Update user.","OrganizationId":"e822651b-5027-4253-83f5-904854601a3b","RecordType":8,"ResultStatus":"Success","UserKey":"Not Available","UserType":4,"Version":1,"Workload":"AzureActiveDirectory","ObjectId":"demo.m3connector@trianzazuresb.onmicrosoft.com","UserId":"ServicePrincipal_3616d279-e97d-48d3-af3e-74ed7de78faf","AzureActiveDirectoryEventType":1,"ExtendedProperties":[{"Name":"additionalDetails","Value":"{\"UserType\":\"Member\",\"User-Agent\":\"Apache-HttpClient/4.5.13 (Java/17.0.17)\"}"},{"Name":"extendedAuditEventCategory","Value":"User"}],"ModifiedProperties":[{"Name":"JobTitle","NewValue":"[\r\n \"Updated by Canary test at 2026-01-21T11:22:46.484860378Z\"\r\n]","OldValue":"[\r\n \"Updated by Canary test at 2026-01-21T10:22:46.227890652Z\"\r\n]"},{"Name":"Included Updated Properties","NewValue":"JobTitle","OldValue":""},{"Name":"TargetId.UserType","NewValue":"Member","OldValue":""},{"Name":"ActorId.ServicePrincipalNames","NewValue":"fb6b0f13-8f1e-4a28-a772-d32d3133da23","OldValue":""},{"Name":"SPN","NewValue":"fb6b0f13-8f1e-4a28-a772-d32d3133da23","OldValue":""}],"Actor":[{"ID":"entraId_app","Type":1},{"ID":"fb6b0f13-8f1e-4a28-a772-d32d3133da23","Type":2},{"ID":"ServicePrincipal_3616d279-e97d-48d3-af3e-74ed7de78faf","Type":2},{"ID":"3616d279-e97d-48d3-af3e-74ed7de78faf","Type":2},{"ID":"ServicePrincipal","Type":2}],"ActorContextId":"e822651b-5027-4253-83f5-904854601a3b","InterSystemsId":"906e4a6b-1d86-43fe-b83b-7fd4a6010e3d","IntraSystemId":"c1cc46cc-844a-42c3-8057-588466f360d0","SupportTicketId":"","Target":[{"ID":"User_0fba2a0b-2680-45c1-9ae6-d20b74edb3ec","Type":2},{"ID":"0fba2a0b-2680-45c1-9ae6-d20b74edb3ec","Type":2},{"ID":"User","Type":2},{"ID":"demo.m3connector@trianzazuresb.onmicrosoft.com","Type":5},{"ID":"1003200511619FF9","Type":3}],"TargetContextId":"e822651b-5027-4253-83f5-904854601a3b"} {"AppAccessContext":{},"CreationTime":"2026-01-21T12:29:41","Id":"4d05e15b-3cae-4c13-a745-08de58e8c08b","Operation":"Set-ConditionalAccessPolicy","OrganizationId":"e822651b-5027-4253-83f5-904854601a3b","RecordType":1,"ResultStatus":"True","UserKey":"NT SERVICE\\MSExchangeServiceHostNetCore (Microsoft.Exchange.ServiceHost)","UserType":3,"Version":1,"Workload":"Exchange","ObjectId":"trianzazuresb.onmicrosoft.com\\2492c29a-0491-4f65-82d2-12528d9506b1","UserId":"NT SERVICE\\MSExchangeServiceHostNetCore (Microsoft.Exchange.ServiceHost)","AppId":"","AppPoolName":"MSExchangeServiceHostNetCore","ClientAppId":"","CorrelationID":"","ExternalAccess":true,"OrganizationName":"trianzazuresb.onmicrosoft.com","OriginatingServer":"MA1PR01MB3305 (15.20.9542.003)","Parameters":[{"Name":"Identity","Value":"trianzazuresb.onmicrosoft.com\\2492c29a-0491-4f65-82d2-12528d9506b1"},{"Name":"PolicyDetails","Value":"{\"Version\":1,\"CreatedDateTime\":\"2025-06-24T06:55:35.6182976Z\",\"State\":\"Reporting\",\"Conditions\":{\"Applications\":{\"Include\":[{\"Applications\":[\"f569b9c7-be15-4e87-86f7-87d30d02090b\"]}]},\"Users\":{\"Include\":[{\"Users\":[\"None\"]}]}},\"Controls\":[{\"Control\":[\"Mfa\"]}],\"EnforceAllPoliciesForEas\":true,\"IncludeOtherLegacyClientTypeForEvaluation\":true}"},{"Name":"PolicyLastUpdatedTime","Value":"1/21/2026 12:29:41 PM"},{"Name":"TenantDefaultPolicy","Value":""},{"Name":"DisplayName","Value":"test demo"},{"Name":"PolicyIdentifierString","Value":""}],"RequestId":"69c4596c-3749-41dd-960b-edabac158084"} {"CreationTime":"2026-01-21T12:57:32","Id":"b082b17a-041f-4cbb-9041-d02590731f00","Operation":"UserLoggedIn","OrganizationId":"e822651b-5027-4253-83f5-904854601a3b","RecordType":15,"ResultStatus":"Success","UserKey":"b5166e08-948e-4ee7-a3a8-71132bb1eda6","UserType":0,"Version":1,"Workload":"AzureActiveDirectory","ClientIP":"117.255.45.51","ObjectId":"00000003-0000-0000-c000-000000000000","UserId":"adarsh.verma@trianz.com","AzureActiveDirectoryEventType":1,"ExtendedProperties":[{"Name":"ResultStatusDetail","Value":"Redirect"},{"Name":"UserAgent","Value":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/144.0.0.0 Safari/537.36"},{"Name":"RequestType","Value":"Kmsi:kmsi"}],"ModifiedProperties":[],"Actor":[{"ID":"b5166e08-948e-4ee7-a3a8-71132bb1eda6","Type":0},{"ID":"adarsh.verma@trianz.com","Type":5}],"ActorContextId":"a27f6b6b-dc28-48c2-a138-ddf0f11455f1","ActorIpAddress":"117.255.45.51","InterSystemsId":"4d02603f-954b-4432-99c1-01ab43262367","IntraSystemId":"b082b17a-041f-4cbb-9041-d02590731f00","SupportTicketId":"","Target":[{"ID":"00000003-0000-0000-c000-000000000000","Type":0}],"TargetContextId":"e822651b-5027-4253-83f5-904854601a3b","ApplicationId":"1e92066f-7ee4-493b-9d96-5af35ba80990","ErrorNumber":"0"} {"CreationTime":"2026-01-21T12:21:23","Id":"e8aa7269-9fcc-4683-b1a4-76fa18551f00","Operation":"UserLoggedIn","OrganizationId":"e822651b-5027-4253-83f5-904854601a3b","RecordType":15,"ResultStatus":"Success","UserKey":"b5166e08-948e-4ee7-a3a8-71132bb1eda6","UserType":0,"Version":1,"Workload":"AzureActiveDirectory","ClientIP":"117.255.45.51","ObjectId":"00000003-0000-0000-c000-000000000000","UserId":"adarsh.verma@trianz.com","AzureActiveDirectoryEventType":1,"ExtendedProperties":[{"Name":"ResultStatusDetail","Value":"Success"},{"Name":"KeepMeSignedIn","Value":"true"},{"Name":"UserAgent","Value":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/144.0.0.0 Safari/537.36"},{"Name":"UserAuthenticationMethod","Value":"16"},{"Name":"RequestType","Value":"Login:login"}],"ModifiedProperties":[],"Actor":[{"ID":"b5166e08-948e-4ee7-a3a8-71132bb1eda6","Type":0},{"ID":"adarsh.verma@trianz.com","Type":5}],"ActorContextId":"a27f6b6b-dc28-48c2-a138-ddf0f11455f1","ActorIpAddress":"117.255.45.51","InterSystemsId":"01469061-44cd-4240-9728-6116ec963ea9","IntraSystemId":"e8aa7269-9fcc-4683-b1a4-76fa18551f00","SupportTicketId":"","Target":[{"ID":"00000003-0000-0000-c000-000000000000","Type":0}],"TargetContextId":"e822651b-5027-4253-83f5-904854601a3b","ApplicationId":"1e92066f-7ee4-493b-9d96-5af35ba80990","ErrorNumber":"50140"} {"CreationTime":"2026-01-21T12:56:58","Id":"1625f761-0d98-4495-a788-f14958871600","Operation":"UserLoggedIn","OrganizationId":"e822651b-5027-4253-83f5-904854601a3b","RecordType":15,"ResultStatus":"Success","UserKey":"3520dc3a-c13d-4b70-928d-e2327c7410ad","UserType":0,"Version":1,"Workload":"AzureActiveDirectory","ClientIP":"103.168.212.55","ObjectId":"00000003-0000-0000-c000-000000000000","UserId":"sreekanth.gude@trianz.com","AzureActiveDirectoryEventType":1,"ExtendedProperties":[{"Name":"ResultStatusDetail","Value":"Redirect"},{"Name":"UserAgent","Value":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/143.0.0.0 Safari/537.36"},{"Name":"RequestType","Value":"OAuth2:Authorize"}],"ModifiedProperties":[],"Actor":[{"ID":"3520dc3a-c13d-4b70-928d-e2327c7410ad","Type":0},{"ID":"sreekanth.gude@trianz.com","Type":5}],"ActorContextId":"a27f6b6b-dc28-48c2-a138-ddf0f11455f1","ActorIpAddress":"103.168.212.55","InterSystemsId":"81d38838-fcc6-4686-8b94-5b5ee41ce717","IntraSystemId":"1625f761-0d98-4495-a788-f14958871600","SupportTicketId":"","Target":[{"ID":"00000003-0000-0000-c000-000000000000","Type":0}],"TargetContextId":"e822651b-5027-4253-83f5-904854601a3b","ApplicationId":"1e92066f-7ee4-493b-9d96-5af35ba80990","ErrorNumber":"0"} 2026-01-21T18:50:21,416 [pool-7-thread-1] INFO org.opensearch.dataprepper.plugins.source.source_crawler.base.DimensionalTimeSliceCrawler - Total partitions created in this crawl: 5.0 2026-01-21T18:50:21,998 [pool-7-thread-5] INFO org.opensearch.dataprepper.plugins.source.source_crawler.base.DimensionalTimeSliceCrawler - Processing partition - DimensionType: DLP.All, TimeRange: 2026-01-21T13:14:21.379582900Z to 2026-01-21T13:15:21.415955Zrange: PT2H30M
2026-01-21T19:50:51,957 [pool-7-thread-5] INFO org.opensearch.dataprepper.plugins.source.source_crawler.base.DimensionalTimeSliceCrawler - Processing partition - DimensionType: Audit.AzureActiveDirectory, TimeRange: 2026-01-21T14:00:00Z to 2026-01-21T14:15:41.377085600Z 2026-01-21T19:50:51,960 [pool-7-thread-4] INFO org.opensearch.dataprepper.plugins.source.source_crawler.base.DimensionalTimeSliceCrawler - Processing partition - DimensionType: Audit.Exchange, TimeRange: 2026-01-21T14:00:00Z to 2026-01-21T14:15:41.377085600Z 2026-01-21T19:50:51,965 [pool-7-thread-2] INFO org.opensearch.dataprepper.plugins.source.source_crawler.base.DimensionalTimeSliceCrawler - Processing partition - DimensionType: DLP.All, TimeRange: 2026-01-21T14:00:00Z to 2026-01-21T14:15:41.377085600Z {"AppAccessContext":{},"CreationTime":"2026-01-21T12:29:41","Id":"4d05e15b-3cae-4c13-a745-08de58e8c08b","Operation":"Set-ConditionalAccessPolicy","OrganizationId":"e822651b-5027-4253-83f5-904854601a3b","RecordType":1,"ResultStatus":"True","UserKey":"NT SERVICE\\MSExchangeServiceHostNetCore (Microsoft.Exchange.ServiceHost)","UserType":3,"Version":1,"Workload":"Exchange","ObjectId":"trianzazuresb.onmicrosoft.com\\2492c29a-0491-4f65-82d2-12528d9506b1","UserId":"NT SERVICE\\MSExchangeServiceHostNetCore (Microsoft.Exchange.ServiceHost)","AppId":"","AppPoolName":"MSExchangeServiceHostNetCore","ClientAppId":"","CorrelationID":"","ExternalAccess":true,"OrganizationName":"trianzazuresb.onmicrosoft.com","OriginatingServer":"MA1PR01MB3305 (15.20.9542.003)","Parameters":[{"Name":"Identity","Value":"trianzazuresb.onmicrosoft.com\\2492c29a-0491-4f65-82d2-12528d9506b1"},{"Name":"PolicyDetails","Value":"{\"Version\":1,\"CreatedDateTime\":\"2025-06-24T06:55:35.6182976Z\",\"State\":\"Reporting\",\"Conditions\":{\"Applications\":{\"Include\":[{\"Applications\":[\"f569b9c7-be15-4e87-86f7-87d30d02090b\"]}]},\"Users\":{\"Include\":[{\"Users\":[\"None\"]}]}},\"Controls\":[{\"Control\":[\"Mfa\"]}],\"EnforceAllPoliciesForEas\":true,\"IncludeOtherLegacyClientTypeForEvaluation\":true}"},{"Name":"PolicyLastUpdatedTime","Value":"1/21/2026 12:29:41 PM"},{"Name":"TenantDefaultPolicy","Value":""},{"Name":"DisplayName","Value":"test demo"},{"Name":"PolicyIdentifierString","Value":""}],"RequestId":"69c4596c-3749-41dd-960b-edabac158084"} {"CreationTime":"2026-01-21T11:50:48","Id":"7d499ac4-6ad1-450f-9583-984c22581a00","Operation":"UserLoggedIn","OrganizationId":"e822651b-5027-4253-83f5-904854601a3b","RecordType":15,"ResultStatus":"Success","UserKey":"b5166e08-948e-4ee7-a3a8-71132bb1eda6","UserType":0,"Version":1,"Workload":"AzureActiveDirectory","ClientIP":"117.235.177.161","ObjectId":"00000003-0000-0000-c000-000000000000","UserId":"adarsh.verma@trianz.com","AzureActiveDirectoryEventType":1,"ExtendedProperties":[{"Name":"ResultStatusDetail","Value":"Redirect"},{"Name":"UserAgent","Value":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/144.0.0.0 Safari/537.36"},{"Name":"RequestType","Value":"Login:reprocess"}],"ModifiedProperties":[],"Actor":[{"ID":"b5166e08-948e-4ee7-a3a8-71132bb1eda6","Type":0},{"ID":"adarsh.verma@trianz.com","Type":5}],"ActorContextId":"a27f6b6b-dc28-48c2-a138-ddf0f11455f1","ActorIpAddress":"117.235.177.161","InterSystemsId":"4f134a12-198a-4c53-8a4f-8b9005fd2882","IntraSystemId":"7d499ac4-6ad1-450f-9583-984c22581a00","SupportTicketId":"","Target":[{"ID":"00000003-0000-0000-c000-000000000000","Type":0}],"TargetContextId":"e822651b-5027-4253-83f5-904854601a3b","ApplicationId":"1e92066f-7ee4-493b-9d96-5af35ba80990","ErrorNumber":"0"} {"CreationTime":"2026-01-21T12:55:41","Id":"18d8475a-30d9-4c25-8c89-aec365f60200","Operation":"UserLoggedIn","OrganizationId":"e822651b-5027-4253-83f5-904854601a3b","RecordType":15,"ResultStatus":"Success","UserKey":"3520dc3a-c13d-4b70-928d-e2327c7410ad","UserType":0,"Version":1,"Workload":"AzureActiveDirectory","ClientIP":"103.168.212.55","ObjectId":"00000003-0000-0000-c000-000000000000","UserId":"sreekanth.gude@trianz.com","AzureActiveDirectoryEventType":1,"ExtendedProperties":[{"Name":"ResultStatusDetail","Value":"Redirect"},{"Name":"UserAgent","Value":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/143.0.0.0 Safari/537.36"},{"Name":"RequestType","Value":"OAuth2:Authorize"}],"ModifiedProperties":[],"Actor":[{"ID":"3520dc3a-c13d-4b70-928d-e2327c7410ad","Type":0},{"ID":"sreekanth.gude@trianz.com","Type":5}],"ActorContextId":"a27f6b6b-dc28-48c2-a138-ddf0f11455f1","ActorIpAddress":"103.168.212.55","InterSystemsId":"c05e679d-231e-4b24-b06f-cfc66118da92","IntraSystemId":"18d8475a-30d9-4c25-8c89-aec365f60200","SupportTicketId":"","Target":[{"ID":"00000003-0000-0000-c000-000000000000","Type":0}],"TargetContextId":"e822651b-5027-4253-83f5-904854601a3b","ApplicationId":"1e92066f-7ee4-493b-9d96-5af35ba80990","ErrorNumber":"0"} {"CreationTime":"2026-01-21T12:57:32","Id":"b082b17a-041f-4cbb-9041-d02590731f00","Operation":"UserLoggedIn","OrganizationId":"e822651b-5027-4253-83f5-904854601a3b","RecordType":15,"ResultStatus":"Success","UserKey":"b5166e08-948e-4ee7-a3a8-71132bb1eda6","UserType":0,"Version":1,"Workload":"AzureActiveDirectory","ClientIP":"117.255.45.51","ObjectId":"00000003-0000-0000-c000-000000000000","UserId":"adarsh.verma@trianz.com","AzureActiveDirectoryEventType":1,"ExtendedProperties":[{"Name":"ResultStatusDetail","Value":"Redirect"},{"Name":"UserAgent","Value":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/144.0.0.0 Safari/537.36"},{"Name":"RequestType","Value":"Kmsi:kmsi"}],"ModifiedProperties":[],"Actor":[{"ID":"b5166e08-948e-4ee7-a3a8-71132bb1eda6","Type":0},{"ID":"adarsh.verma@trianz.com","Type":5}],"ActorContextId":"a27f6b6b-dc28-48c2-a138-ddf0f11455f1","ActorIpAddress":"117.255.45.51","InterSystemsId":"4d02603f-954b-4432-99c1-01ab43262367","IntraSystemId":"b082b17a-041f-4cbb-9041-d02590731f00","SupportTicketId":"","Target":[{"ID":"00000003-0000-0000-c000-000000000000","Type":0}],"TargetContextId":"e822651b-5027-4253-83f5-904854601a3b","ApplicationId":"1e92066f-7ee4-493b-9d96-5af35ba80990","ErrorNumber":"0"}range: P1D
2026-01-21T18:51:48,505 [pool-7-thread-3] INFO org.opensearch.dataprepper.plugins.source.source_crawler.base.DimensionalTimeSliceCrawler - Processing partition - DimensionType: Audit.AzureActiveDirectory, TimeRange: 2026-01-21T13:00:00Z to 2026-01-21T13:16:35.438390800Z 2026-01-21T18:51:48,520 [pool-7-thread-2] INFO org.opensearch.dataprepper.plugins.source.source_crawler.base.DimensionalTimeSliceCrawler - Processing partition - DimensionType: Audit.Exchange, TimeRange: 2026-01-21T13:00:00Z to 2026-01-21T13:16:35.438390800Z 2026-01-21T18:51:48,570 [pool-7-thread-5] INFO org.opensearch.dataprepper.plugins.source.source_crawler.base.DimensionalTimeSliceCrawler - Processing partition - DimensionType: Audit.General, TimeRange: 2026-01-21T13:00:00Z to 2026-01-21T13:16:35.438390800Z {"CreationTime":"2026-01-21T04:58:57","Id":"8874128c-ce82-491d-8b26-12428c8d0b00","Operation":"UserLoggedIn","OrganizationId":"e822651b-5027-4253-83f5-904854601a3b","RecordType":15,"ResultStatus":"Success","UserKey":"f0472c3d-43e3-456a-89d1-ad8c29992dbd","UserType":0,"Version":1,"Workload":"AzureActiveDirectory","ClientIP":"106.51.198.137","ObjectId":"00000003-0000-0000-c000-000000000000","UserId":"sumit.kumar@trianz.com","AzureActiveDirectoryEventType":1,"ExtendedProperties":[{"Name":"ResultStatusDetail","Value":"Redirect"},{"Name":"UserAgent","Value":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/144.0.0.0 Safari/537.36"},{"Name":"RequestType","Value":"OAuth2:Authorize"}],"ModifiedProperties":[],"Actor":[{"ID":"f0472c3d-43e3-456a-89d1-ad8c29992dbd","Type":0},{"ID":"sumit.kumar@trianz.com","Type":5}],"ActorContextId":"a27f6b6b-dc28-48c2-a138-ddf0f11455f1","ActorIpAddress":"106.51.198.137","InterSystemsId":"361d9f56-96e4-4984-88e2-ed3d0b806e7f","IntraSystemId":"8874128c-ce82-491d-8b26-12428c8d0b00","SupportTicketId":"","Target":[{"ID":"00000003-0000-0000-c000-000000000000","Type":0}],"TargetContextId":"e822651b-5027-4253-83f5-904854601a3b","ApplicationId":"1e92066f-7ee4-493b-9d96-5af35ba80990","ErrorNumber":"0"} {"CreationTime":"2026-01-21T06:01:49","Id":"20831b51-2906-4c3e-a5cd-88f8f1ce0c00","Operation":"UserLoggedIn","OrganizationId":"e822651b-5027-4253-83f5-904854601a3b","RecordType":15,"ResultStatus":"Success","UserKey":"a76224d3-123b-433d-ac87-cd0a0e4a6f2c","UserType":0,"Version":1,"Workload":"AzureActiveDirectory","ClientIP":"136.226.251.32","ObjectId":"00000003-0000-0000-c000-000000000000","UserId":"raghav.gupta@trianz.com","AzureActiveDirectoryEventType":1,"ExtendedProperties":[{"Name":"ResultStatusDetail","Value":"Redirect"},{"Name":"UserAgent","Value":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/144.0.0.0 Safari/537.36 Edg/144.0.0.0 OS/10.0.26100"},{"Name":"RequestType","Value":"OAuth2:Authorize"}],"ModifiedProperties":[],"Actor":[{"ID":"a76224d3-123b-433d-ac87-cd0a0e4a6f2c","Type":0},{"ID":"raghav.gupta@trianz.com","Type":5}],"ActorContextId":"a27f6b6b-dc28-48c2-a138-ddf0f11455f1","ActorIpAddress":"136.226.251.32","InterSystemsId":"f293eef7-4ec3-4827-9e8f-a5718b8515ae","IntraSystemId":"20831b51-2906-4c3e-a5cd-88f8f1ce0c00","SupportTicketId":"","Target":[{"ID":"00000003-0000-0000-c000-000000000000","Type":0}],"TargetContextId":"e822651b-5027-4253-83f5-904854601a3b","ApplicationId":"1e92066f-7ee4-493b-9d96-5af35ba80990","ErrorNumber":"0"} {"CreationTime":"2026-01-21T07:19:15","Id":"37048176-6a96-4a5a-bf3f-1c2c6ece1300","Operation":"UserLoggedIn","OrganizationId":"e822651b-5027-4253-83f5-904854601a3b","RecordType":15,"ResultStatus":"Success","UserKey":"39afb8e4-955b-4f1f-896f-a8eaf41ed2aa","UserType":0,"Version":1,"Workload":"AzureActiveDirectory","ClientIP":"103.160.27.17","ObjectId":"00000003-0000-0000-c000-000000000000","UserId":"devang.jain@trianz.com","AzureActiveDirectoryEventType":1,"ExtendedProperties":[{"Name":"ResultStatusDetail","Value":"Redirect"},{"Name":"UserAgent","Value":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/143.0.0.0 Safari/537.36"},{"Name":"RequestType","Value":"OAuth2:Authorize"}],"ModifiedProperties":[],"Actor":[{"ID":"39afb8e4-955b-4f1f-896f-a8eaf41ed2aa","Type":0},{"ID":"devang.jain@trianz.com","Type":5}],"ActorContextId":"a27f6b6b-dc28-48c2-a138-ddf0f11455f1","ActorIpAddress":"103.160.27.17","InterSystemsId":"c0c61512-267d-4c88-a307-d1bbe605a2e4","IntraSystemId":"37048176-6a96-4a5a-bf3f-1c2c6ece1300","SupportTicketId":"","Target":[{"ID":"00000003-0000-0000-c000-000000000000","Type":0}],"TargetContextId":"e822651b-5027-4253-83f5-904854601a3b","ApplicationId":"1e92066f-7ee4-493b-9d96-5af35ba80990","ErrorNumber":"0"}range: P7D
2026-01-21T19:05:35,423 [pool-7-thread-5] INFO org.opensearch.dataprepper.plugins.source.source_crawler.base.DimensionalTimeSliceCrawler - Processing partition - DimensionType: DLP.All, TimeRange: 2026-01-21T13:00:00Z to 2026-01-21T13:30:05.065079Z 2026-01-21T19:05:35,487 [pool-7-thread-4] INFO org.opensearch.dataprepper.plugins.source.source_crawler.base.DimensionalTimeSliceCrawler - Processing partition - DimensionType: Audit.AzureActiveDirectory, TimeRange: 2026-01-21T13:00:00Z to 2026-01-21T13:30:05.065079Z 2026-01-21T19:05:35,491 [pool-7-thread-5] INFO org.opensearch.dataprepper.plugins.source.source_crawler.base.DimensionalTimeSliceCrawler - Processing partition - DimensionType: DLP.All, TimeRange: 2026-01-21T12:00:00Z to 2026-01-21T13:00:00Z 2026-01-21T19:05:35,491 [pool-7-thread-3] INFO org.opensearch.dataprepper.plugins.source.source_crawler.base.DimensionalTimeSliceCrawler - Processing partition - DimensionType: Audit.Exchange, TimeRange: 2026-01-21T13:00:00Z to 2026-01-21T13:30:05.065079Z {"CreationTime":"2026-01-20T09:09:54","Id":"a9c6ff1e-e961-4065-a032-6772ce6f9d00","Operation":"UserLoggedIn","OrganizationId":"e822651b-5027-4253-83f5-904854601a3b","RecordType":15,"ResultStatus":"Success","UserKey":"b5166e08-948e-4ee7-a3a8-71132bb1eda6","UserType":0,"Version":1,"Workload":"AzureActiveDirectory","ClientIP":"136.226.251.35","ObjectId":"00000003-0000-0000-c000-000000000000","UserId":"adarsh.verma@trianz.com","AzureActiveDirectoryEventType":1,"ExtendedProperties":[{"Name":"ResultStatusDetail","Value":"Redirect"},{"Name":"UserAgent","Value":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/144.0.0.0 Safari/537.36 Edg/144.0.0.0 OS/10.0.22631"},{"Name":"RequestType","Value":"Login:reprocess"}],"ModifiedProperties":[],"Actor":[{"ID":"b5166e08-948e-4ee7-a3a8-71132bb1eda6","Type":0},{"ID":"adarsh.verma@trianz.com","Type":5}],"ActorContextId":"a27f6b6b-dc28-48c2-a138-ddf0f11455f1","ActorIpAddress":"136.226.251.35","InterSystemsId":"18476da6-4d06-4b9c-ba14-e47118e2bc55","IntraSystemId":"a9c6ff1e-e961-4065-a032-6772ce6f9d00","SupportTicketId":"","Target":[{"ID":"00000003-0000-0000-c000-000000000000","Type":0}],"TargetContextId":"e822651b-5027-4253-83f5-904854601a3b","ApplicationId":"1e92066f-7ee4-493b-9d96-5af35ba80990","ErrorNumber":"0"} {"AppAccessContext":{},"CreationTime":"2026-01-20T12:06:36","Id":"f2c7157e-e4f2-4b09-447d-08de581c5c2b","Operation":"Set-ConditionalAccessPolicy","OrganizationId":"e822651b-5027-4253-83f5-904854601a3b","RecordType":1,"ResultStatus":"True","UserKey":"NT SERVICE\\MSExchangeServiceHostNetCore (Microsoft.Exchange.ServiceHost)","UserType":3,"Version":1,"Workload":"Exchange","ObjectId":"trianzazuresb.onmicrosoft.com\\058b600e-79d2-4c34-8ac8-95fe8ce13fc0","UserId":"NT SERVICE\\MSExchangeServiceHostNetCore (Microsoft.Exchange.ServiceHost)","AppId":"","AppPoolName":"MSExchangeServiceHostNetCore","ClientAppId":"","CorrelationID":"","ExternalAccess":true,"OrganizationName":"trianzazuresb.onmicrosoft.com","OriginatingServer":"MA1PR01MB3305 (15.20.9520.006)","Parameters":[{"Name":"Identity","Value":"trianzazuresb.onmicrosoft.com\\058b600e-79d2-4c34-8ac8-95fe8ce13fc0"},{"Name":"PolicyDetails","Value":"{\"Version\":0,\"State\":\"Disabled\"}"},{"Name":"PolicyLastUpdatedTime","Value":"1/20/2026 12:06:35 PM"},{"Name":"TenantDefaultPolicy","Value":"18"},{"Name":"DisplayName","Value":"Default Policy"},{"Name":"PolicyIdentifierString","Value":"6/24/2025 6:55:41 AM"}],"RequestId":"7d1c0a2f-6e39-4aff-be80-d7496257393b"} {"CreationTime":"2026-01-20T10:00:38","Id":"b00c382b-643d-4075-b9a6-338986329000","Operation":"UserLoggedIn","OrganizationId":"e822651b-5027-4253-83f5-904854601a3b","RecordType":15,"ResultStatus":"Success","UserKey":"718a7271-4457-4698-8154-052bea6a5ca0","UserType":0,"Version":1,"Workload":"AzureActiveDirectory","ClientIP":"61.95.132.2","ObjectId":"797f4846-ba00-4fd7-ba43-dac1f8f63013","UserId":"718a7271-4457-4698-8154-052bea6a5ca0","AzureActiveDirectoryEventType":1,"ExtendedProperties":[{"Name":"ResultStatusDetail","Value":"Redirect"},{"Name":"UserAgent","Value":"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:147.0) Gecko/20100101 Firefox/147.0"},{"Name":"RequestType","Value":"Login:reprocess"}],"ModifiedProperties":[],"Actor":[{"ID":"718a7271-4457-4698-8154-052bea6a5ca0","Type":0},{"ID":"kumar.thota@trianzmscsp.onmicrosoft.com","Type":5}],"ActorContextId":"0e3bf84a-aeee-49d3-b1de-4d2af071895b","ActorIpAddress":"61.95.132.2","InterSystemsId":"019bdad9-55e1-7d1d-a36a-fad697401f03","IntraSystemId":"b00c382b-643d-4075-b9a6-338986329000","SupportTicketId":"","Target":[{"ID":"797f4846-ba00-4fd7-ba43-dac1f8f63013","Type":0}],"TargetContextId":"e822651b-5027-4253-83f5-904854601a3b","ApplicationId":"c44b4083-3bb0-49c1-b47d-974e53cbdf3c","ErrorNumber":"0"}range: P30D
2026-01-21T19:06:40,834 [main] ERROR org.opensearch.dataprepper.core.validation.LoggingPluginErrorsHandler - 1. pipeline.source.microsoft_office365: caused by: Plugin microsoft_office365 in pipeline pipeline is configured incorrectly: range Range cannot exceed 7 days due to Office 365 API limitation 2026-01-21T19:06:40,835 [main] ERROR org.opensearch.dataprepper.core.parser.PipelineTransformer - Construction of pipeline components failed, skipping building of pipeline [pipeline] and its connected pipelines org.opensearch.dataprepper.model.plugin.InvalidPluginConfigurationException: One or more plugins are not configured correctly in the pipeline: pipeline. at org.opensearch.dataprepper.core.parser.PipelineTransformer.buildPipelineFromConfiguration(PipelineTransformer.java:216) at org.opensearch.dataprepper.core.parser.PipelineTransformer.transformConfiguration(PipelineTransformer.java:122) at org.opensearch.dataprepper.core.DataPrepper.<init>(DataPrepper.java:74) at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)without range:
2026-01-21T19:08:07,981 [pool-7-thread-1] INFO org.opensearch.dataprepper.plugins.source.source_crawler.base.DimensionalTimeSliceCrawler - Total partitions created in this crawl: 0.0 2026-01-21T19:09:07,987 [pool-7-thread-1] INFO org.opensearch.dataprepper.plugins.source.source_crawler.base.DimensionalTimeSliceCrawler - Total partitions created in this crawl: 0.0 2026-01-21T19:10:07,990 [pool-7-thread-1] INFO org.opensearch.dataprepper.plugins.source.source_crawler.base.DimensionalTimeSliceCrawler - Total partitions created in this crawl: 0.0 2026-01-21T19:11:08,004 [pool-7-thread-1] INFO org.opensearch.dataprepper.plugins.source.source_crawler.base.DimensionalTimeSliceCrawler - Total partitions created in this crawl: 0.0 2026-01-21T19:12:08,019 [pool-7-thread-1] INFO org.opensearch.dataprepper.plugins.source.source_crawler.base.DimensionalTimeSliceCrawler - Total partitions created in this crawl: 0.0 2026-01-21T19:13:08,039 [pool-7-thread-1] INFO org.opensearch.dataprepper.plugins.source.source_crawler.base.DimensionalTimeSliceCrawler - Total partitions created in this crawl: 5.0 2026-01-21T19:13:08,209 [pool-7-thread-2] INFO org.opensearch.dataprepper.plugins.source.source_crawler.base.DimensionalTimeSliceCrawler - Processing partition - DimensionType: Audit.AzureActiveDirectory, TimeRange: 2026-01-21T13:38:07.966287400Z to 2026-01-21T13:38:08.032300900Z 2026-01-21T19:13:08,210 [pool-7-thread-3] INFO org.opensearch.dataprepper.plugins.source.source_crawler.base.DimensionalTimeSliceCrawler - Processing partition - DimensionType: DLP.All, TimeRange: 2026-01-21T13:38:07.966287400Z to 2026-01-21T13:38:08.032300900Z 2026-01-21T19:13:08,237 [pool-7-thread-5] INFO org.opensearch.dataprepper.plugins.source.source_crawler.base.DimensionalTimeSliceCrawler - Processing partition - DimensionType: Audit.SharePoint, TimeRange: 2026-01-21T13:38:07.966287400Z to 2026-01-21T13:38:08.032300900Z 2026-01-21T19:13:08,237 [pool-7-thread-4] INFO org.opensearch.dataprepper.plugins.source.source_crawler.base.DimensionalTimeSliceCrawler - Processing partition - DimensionType: Audit.General, TimeRange: 2026-01-21T13:38:07.966287400Z to 2026-01-21T13:38:08.032300900Z 2026-01-21T19:13:08,396 [pool-7-thread-3] INFO org.opensearch.dataprepper.plugins.source.source_crawler.base.DimensionalTimeSliceCrawler - Processing partition - DimensionType: Audit.Exchange, TimeRange: 2026-01-21T13:38:07.966287400Z to 2026-01-21T13:38:08.032300900ZCheck List
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.