add partition timing metrics to LeaderOnlyTokenCrawler

[wandna-amazon]

add missing partition metrics to LeaderOnlyTokenCrawler

aligns LeaderOnlyTokenCrawler metrics with TokenPaginationCrawler as part of the connectors metrics audit (see details)

changes

• added WorkerPartitionWaitTime metric - tracks time partitions wait before processing
• added WorkerPartitionProcessLatency metric - measures actual partition processing time
• updated executePartition() method to record both metrics
• added comprehensive test coverage following existing patterns

benefits

• consistent metrics across both crawler implementations
• better visibility into retry partition performance
• improved observability for partition processing bottlenecks

test results

./gradlew :data-prepper-plugins:saas-source-plugins:source-crawler:test --tests "*testExecutePartitionMetrics*"

OpenJDK 64-Bit Server VM warning: Sharing is only supported for boot loader classes because bootstrap classpath has been appended

Deprecated Gradle features were used in this build, making it incompatible with Gradle 9.0.

You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.

For more on this, please refer to https://docs.gradle.org/8.8/userguide/command_line_interface.html#sec:command_line_warnings in the Gradle documentation.

BUILD SUCCESSFUL in 7s
51 actionable tasks: 4 executed, 47 up-to-date

BUILD SUCCESSFUL in 7s 51 actionable tasks: 4 executed, 47 up-to-date

revision

• rebased
• refactored metrics to use camel case.

revision 2

• fixed tests and enforced camel case

./gradlew :data-prepper-plugins:saas-source-plugins:source-crawler:test

> Task :data-prepper-plugins:saas-source-plugins:source-crawler:compileTestJava
Note: Some input files use unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.
OpenJDK 64-Bit Server VM warning: Sharing is only supported for boot loader classes because bootstrap classpath has been appended

Deprecated Gradle features were used in this build, making it incompatible with Gradle 9.0.

You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.

For more on this, please refer to https://docs.gradle.org/8.8/userguide/command_line_interface.html#sec:command_line_warnings in the Gradle documentation.

BUILD SUCCESSFUL in 39s
51 actionable tasks: 2 executed, 49 up-to-date

###revision 3

Error regarding new RetryHandler code change, quick fix on the test case

./gradlew build
OpenJDK 64-Bit Server VM warning: Sharing is only supported for boot loader classes because bootstrap classpath has been appended
OpenJDK 64-Bit Server VM warning: Sharing is only supported for boot loader classes because bootstrap classpath has been appended

> Task :data-prepper-plugins:encryption-plugin:compileTestJava
Note: /Users/wandna/data-prepper/data-prepper-plugins/encryption-plugin/src/test/java/org/opensearch/dataprepper/plugins/encryption/EncryptionPluginTest.java uses unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.
OpenJDK 64-Bit Server VM warning: Sharing is only supported for boot loader classes because bootstrap classpath has been appended

> Task :data-prepper-plugins:event-json-codecs:compileTestJava
Note: Some input files use unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.
OpenJDK 64-Bit Server VM warning: Sharing is only supported for boot loader classes because bootstrap classpath has been appended

. . .

BUILD SUCCESSFUL in 19m 14s
1637 actionable tasks: 701 executed, 936 up-to-date

[vecheka]

We should keep the naming consistent with the other metrics (lowerCamelCase in this case).

[wandna-amazon]

Yes, I agree however the metric name used in DimensionalTimeSliceCrawler is uppercase naming: link so we can either keep as is, or also make a change to do camel case on DimensionalTimeSliceCrawler metric names.

[vecheka]

Ahh interesting. Not a big blocker but I think we should move these metricNames to be shared global constants then.

[chrisale000]

We should probably update both.

[wandna-amazon]

updated in the last commit