Add cache for single measurement validation#17899
Conversation
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #17899 +/- ##
============================================
+ Coverage 40.80% 40.83% +0.02%
+ Complexity 2612 318 -2294
============================================
Files 5200 5245 +45
Lines 353687 363239 +9552
Branches 45252 46782 +1530
============================================
+ Hits 144326 148332 +4006
- Misses 209361 214907 +5546 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
I think this part has a correctness issue: checkTsFileResourceExists currently warns for every .tsfile because it only checks the filename suffix and never checks whether the corresponding resource file exists. For a normal snapshot containing both xxx.tsfile and xxx.tsfile.resource, this still logs The associated resource file ... is not found in the snapshot, which is a false warning. Please check |
|
* Add cache * Add cache config * Add a log when resource is missing in snapshot * Ignore measurement cache performance IT * Only warn when resource not exist in the snapshot * remove unused exception * add import * spotless



Description
This PR adds a configurable cache for
PathUtils#checkAndReturnSingleMeasurementso repeated measurement validation can reuse previous check results. The cache stores both valid and invalid validation results and can be disabled by setting the cache size to0.It also adds configuration plumbing for integration tests and an ignored performance IT that can be run manually to compare end-to-end write performance with the cache disabled and enabled.
Test Report
Local verification:
Performance averages, 3 repeats per group, 80,000 rows per experiment: