|
| 1 | +.. Copyright 2026-present Alibaba Inc. |
| 2 | +
|
| 3 | +.. Licensed under the Apache License, Version 2.0 (the "License"); |
| 4 | +.. you may not use this file except in compliance with the License. |
| 5 | +.. You may obtain a copy of the License at |
| 6 | +
|
| 7 | +.. http://www.apache.org/licenses/LICENSE-2.0 |
| 8 | +
|
| 9 | +.. Unless required by applicable law or agreed to in writing, software |
| 10 | +.. distributed under the License is distributed on an "AS IS" BASIS, |
| 11 | +.. WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| 12 | +.. See the License for the specific language governing permissions and |
| 13 | +.. limitations under the License. |
| 14 | +
|
| 15 | +Parquet Metadata Cache |
| 16 | +====================== |
| 17 | + |
| 18 | +Overview |
| 19 | +-------- |
| 20 | + |
| 21 | +paimon-cpp can cache serialized Parquet metadata footer bytes for Parquet data files. |
| 22 | +The cache is used by ``ParquetReaderBuilder`` before opening the Arrow Parquet |
| 23 | +reader. On a cache miss, paimon-cpp loads the Parquet file metadata, serializes |
| 24 | +it as a complete metadata footer, and stores those bytes in the public |
| 25 | +``Cache`` abstraction. On a cache hit, paimon-cpp parses the cached footer bytes into |
| 26 | +``parquet::FileMetaData`` and passes the metadata to the Parquet reader. |
| 27 | + |
| 28 | +The cache stores serialized metadata footer bytes instead of caching a |
| 29 | +``parquet::FileMetaData`` instance. This keeps the cache value compact and |
| 30 | +similar to manifest cache values: the cache weight follows the actual cached |
| 31 | +bytes, while the Parquet library still owns metadata parsing and validation. |
| 32 | + |
| 33 | +This optimization is useful when the same Parquet files are opened repeatedly |
| 34 | +in the same process, for example repeated ``get`` or ``scan`` requests over the |
| 35 | +same snapshot. On a cache hit, the read path avoids reading the Parquet footer |
| 36 | +bytes from the filesystem again. paimon-cpp still parses the cached footer bytes |
| 37 | +into ``parquet::FileMetaData`` for each reader open. Data pages, page indexes, |
| 38 | +and column chunks are still read from the file as usual. |
| 39 | + |
| 40 | +Configuration |
| 41 | +------------- |
| 42 | + |
| 43 | +Parquet metadata caching is disabled by default. Embedding applications that |
| 44 | +need it can provide a custom ``Cache`` implementation and inject it through |
| 45 | +``ScanContextBuilder`` or ``ReadContextBuilder``. Parquet reader builders |
| 46 | +receive the cache from the read context and create cache keys with |
| 47 | +``CacheKind::DATA_FILE_FOOTER`` internally. |
| 48 | + |
| 49 | +The cache key represents the file footer and is created from the file URI with |
| 50 | +position ``-1`` and length ``-1``. Callers do not need to construct this key |
| 51 | +directly; they only need to route ``CacheKind::DATA_FILE_FOOTER`` entries to an |
| 52 | +appropriate cache backend. |
| 53 | + |
| 54 | +Example: |
| 55 | + |
| 56 | +.. code-block:: cpp |
| 57 | +
|
| 58 | + class RoutingCache : public paimon::Cache { |
| 59 | + public: |
| 60 | + RoutingCache(std::shared_ptr<paimon::Cache> default_cache, |
| 61 | + std::shared_ptr<paimon::Cache> parquet_metadata_cache) |
| 62 | + : default_cache_(std::move(default_cache)), |
| 63 | + parquet_metadata_cache_(std::move(parquet_metadata_cache)) {} |
| 64 | +
|
| 65 | + paimon::Result<std::shared_ptr<paimon::CacheValue>> Get( |
| 66 | + const std::shared_ptr<paimon::CacheKey>& key, |
| 67 | + std::function<paimon::Result<std::shared_ptr<paimon::CacheValue>>( |
| 68 | + const std::shared_ptr<paimon::CacheKey>&)> supplier) override { |
| 69 | + return Select(key)->Get(key, std::move(supplier)); |
| 70 | + } |
| 71 | +
|
| 72 | + // Put(), Invalidate(), InvalidateAll(), and Size() route in the same way. |
| 73 | +
|
| 74 | + private: |
| 75 | + std::shared_ptr<paimon::Cache> Select( |
| 76 | + const std::shared_ptr<paimon::CacheKey>& key) const { |
| 77 | + return key && key->GetKind() == paimon::CacheKind::DATA_FILE_FOOTER |
| 78 | + ? parquet_metadata_cache_ |
| 79 | + : default_cache_; |
| 80 | + } |
| 81 | +
|
| 82 | + std::shared_ptr<paimon::Cache> default_cache_; |
| 83 | + std::shared_ptr<paimon::Cache> parquet_metadata_cache_; |
| 84 | + }; |
| 85 | +
|
| 86 | + auto cache = std::make_shared<RoutingCache>( |
| 87 | + std::make_shared<MyDefaultCache>(), |
| 88 | + std::make_shared<MyParquetMetadataCache>()); |
| 89 | +
|
| 90 | + paimon::ScanContextBuilder scan_builder(table_path); |
| 91 | + scan_builder.WithCache(cache); |
| 92 | +
|
| 93 | + paimon::ReadContextBuilder read_builder(table_path); |
| 94 | + read_builder.WithCache(cache); |
| 95 | +
|
| 96 | +Passing ``nullptr`` or omitting ``WithCache()`` leaves Parquet metadata caching |
| 97 | +disabled. If a file URI cannot be obtained, paimon-cpp also bypasses the cache |
| 98 | +and opens the Parquet file normally. |
| 99 | + |
| 100 | +Future Optimizations |
| 101 | +-------------------- |
| 102 | + |
| 103 | +- Add hit, miss, bypass, and eviction metrics for Parquet metadata cache. |
| 104 | +- Add single-flight loading for high-concurrency misses on the same Parquet |
| 105 | + file. |
| 106 | +- Evaluate sharing cached metadata footer bytes with page-index prefetch logic when |
| 107 | + those read paths can use the same cache abstraction. |
0 commit comments