Skip to content

Commit 41c6640

Browse files
pdillingerjaykorean
authored andcommitted
Auto-tune manifest file size (facebook#14076)
Summary: Adds auto-tuning of manifest file size to avoid the need to scale `max_manifest_file_size` in proportion to things like number of SST files to properly balance (a) manifest file write amp and new file creation, vs. (b) manifest file space amp and replay time, including non-incremental space usage in backups. (Manifest file write amp comes from re-writing a "live" record when the manifest file is re-created, or "compacted"; space amp is usage beyond what would be used by a compacted manifest file.) In more detail, * Add new option `max_manifest_space_amp_pct` with default value of 500, which defaults to 0.2 write amp and up to roughly 5.0 space amp, except `max_manifest_file_size` is treated as the "minimum" size before re-creating ("compacting") the manifest file. * `max_manifest_file_size` in a way means the same thing, with the same default of 1GB, but in a way has taken on a new role. What is the same is that we do not re-create the manifest file before reaching this size (except for DB re-open), and so users are very unlikely to see a change in default behavior (auto-tuning only kicking in if auto-tuning would exceed 1GB for effective max size for the current manifest file). The new role is as a file size lower bound before auto-tuning kicks in, to minimize churn in files considered "negligibly small." We recommend a new setting of around 1MB or even smaller like 64KB, and expect something like this to become the default soon. * These two options along with `manifest_preallocation_size` are now mutable with SetDBOptions. The effect is nearly immediate, affecting the next write to the current manifest file. Also in this PR: * Refactoring of VersionSet to allow it to get (more) settings from MutableDBOptions. This touches a number of files in not very interesting ways, but notably we have to be careful about thread-safe access to MutableDBOptions fields, and even fields within VersionSet. I have decided to save copies of relevant fields from MutableDBOptions to simplify testing, etc. by not saving a reference to MutableDBOptions but getting notified of updates. * Updated some logging in VersionSet to provide some basic data about final and compacted manifest sizes (effects of auto-tuning), making sure to avoid I/O while holding DB mutex. * Added db_etc3_test.cc which is intended as a successor to db_test and db_test2, but having "test.cc" in its name for easier exclusion of test files when using `git grep`. Intended follow-up: rename db_test2 to db_etc2_test * Moved+updated `ManifestRollOver` test to the new file to be closer to other manifest file rollover testing. Pull Request resolved: facebook#14076 Test Plan: As for correctness, new unit test AutoTuneManifestSize is pretty thorough. Some other unit tests updated appropriately. Manual tests in the performance section were also audited for expected behavior based on the new logging in the DB LOG. Example LOG data with -max_manifest_file_size=2048 -max_manifest_space_amp_pct=500: ``` 2025/10/24-11:12:48.979472 2150678 [/version_set.cc:5927] Created manifest 5, compacted+appended from 52 to 116 2025/10/24-11:12:49.626441 2150682 [/version_set.cc:5927] Created manifest 24, compacted+appended from 2169 to 1801 2025/10/24-11:12:52.194592 2150682 [/version_set.cc:5927] Created manifest 91, compacted+appended from 10913 to 8707 2025/10/24-11:13:02.969944 2150682 [/version_set.cc:5927] Created manifest 362, compacted+appended from 52259 to 13321 2025/10/24-11:13:18.815120 2150681 [/version_set.cc:5927] Created manifest 765, compacted+appended from 80064 to 13304 2025/10/24-11:13:35.590905 2150681 [/version_set.cc:5927] Created manifest 1167, compacted+appended from 79863 to 13304 ``` As you can see, it only took a few iterations of ramp-up to settle on the auto-tuned max manifest size for tracking ~122 live SST files, around 80KB and compacting down to about 13KB. (13KB * (500 + 100) / 100 = 78KB). With the default large setting for max_manifest_file_size, we end up with a 232KB manifest, which is more than 90% wasted space. (A long-running DB would be much worse.) As for performance, we don't expect a difference, even with TransactionDB because actual writing of the manifest is done without holding the DB mutex. I was not able to see a performance regression using db_bench with FIFO compaction and >1000 ~10MB SST files, including settings of -max_manifest_file_size=2048 -max_manifest_space_amp_pct={500,10,0}. No "hiccups" visible with -histogram either. I also tried seeding a 1 second delay in writing new manifest files (other than the first). This had no significant effect at -max_manifest_space_amp_pct=500 but at 100 started causing write stalls in my test. In many ways this is kind of a worst case scenario and out-of-proportion test, but gives me more confidence that a higher number like 500 is probably the best balance in general. Reviewed By: xingbowang Differential Revision: D85445178 Pulled By: pdillinger fbshipit-source-id: 1e6e07e89c586762dd65c65bb7cb2b8b719513f9
1 parent f324397 commit 41c6640

39 files changed

Lines changed: 519 additions & 172 deletions

BUCK

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4844,6 +4844,12 @@ cpp_unittest_wrapper(name="db_encryption_test",
48444844
extra_compiler_flags=[])
48454845

48464846

4847+
cpp_unittest_wrapper(name="db_etc3_test",
4848+
srcs=["db/db_etc3_test.cc"],
4849+
deps=[":rocksdb_test_lib"],
4850+
extra_compiler_flags=[])
4851+
4852+
48474853
cpp_unittest_wrapper(name="db_flush_test",
48484854
srcs=["db/db_flush_test.cc"],
48494855
deps=[":rocksdb_test_lib"],

CMakeLists.txt

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1359,6 +1359,7 @@ if(WITH_TESTS)
13591359
db/db_clip_test.cc
13601360
db/db_dynamic_level_test.cc
13611361
db/db_encryption_test.cc
1362+
db/db_etc3_test.cc
13621363
db/db_flush_test.cc
13631364
db/db_inplace_update_test.cc
13641365
db/db_io_failure_test.cc

Makefile

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1496,6 +1496,9 @@ db_test: $(OBJ_DIR)/db/db_test.o $(TEST_LIBRARY) $(LIBRARY)
14961496
db_test2: $(OBJ_DIR)/db/db_test2.o $(TEST_LIBRARY) $(LIBRARY)
14971497
$(AM_LINK)
14981498

1499+
db_etc3_test: $(OBJ_DIR)/db/db_etc3_test.o $(TEST_LIBRARY) $(LIBRARY)
1500+
$(AM_LINK)
1501+
14991502
compression_test: $(OBJ_DIR)/util/compression_test.o $(TEST_LIBRARY) $(LIBRARY)
15001503
$(AM_LINK)
15011504

db/compaction/compaction_job_test.cc

Lines changed: 9 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -211,8 +211,8 @@ class CompactionJobTestBase : public testing::Test {
211211
table_cache_(NewLRUCache(50000, 16)),
212212
write_buffer_manager_(db_options_.db_write_buffer_size),
213213
versions_(new VersionSet(
214-
dbname_, &db_options_, env_options_, table_cache_.get(),
215-
&write_buffer_manager_, &write_controller_,
214+
dbname_, &db_options_, mutable_db_options_, env_options_,
215+
table_cache_.get(), &write_buffer_manager_, &write_controller_,
216216
/*block_cache_tracer=*/nullptr,
217217
/*io_tracer=*/nullptr, /*db_id=*/"", /*db_session_id=*/"",
218218
/*daily_offpeak_time_utc=*/"",
@@ -546,13 +546,13 @@ class CompactionJobTestBase : public testing::Test {
546546
ASSERT_OK(s);
547547
db_options_.info_log = info_log;
548548

549-
versions_.reset(
550-
new VersionSet(dbname_, &db_options_, env_options_, table_cache_.get(),
551-
&write_buffer_manager_, &write_controller_,
552-
/*block_cache_tracer=*/nullptr, /*io_tracer=*/nullptr,
553-
test::kUnitTestDbId, /*db_session_id=*/"",
554-
/*daily_offpeak_time_utc=*/"",
555-
/*error_handler=*/nullptr, /*unchanging=*/false));
549+
versions_.reset(new VersionSet(
550+
dbname_, &db_options_, mutable_db_options_, env_options_,
551+
table_cache_.get(), &write_buffer_manager_, &write_controller_,
552+
/*block_cache_tracer=*/nullptr, /*io_tracer=*/nullptr,
553+
test::kUnitTestDbId, /*db_session_id=*/"",
554+
/*daily_offpeak_time_utc=*/"",
555+
/*error_handler=*/nullptr, /*unchanging=*/false));
556556
compaction_job_stats_.Reset();
557557

558558
VersionEdit new_db;

db/db_basic_test.cc

Lines changed: 0 additions & 24 deletions
Original file line numberDiff line numberDiff line change
@@ -675,30 +675,6 @@ TEST_F(DBBasicTest, Flush) {
675675
} while (ChangeCompactOptions());
676676
}
677677

678-
TEST_F(DBBasicTest, ManifestRollOver) {
679-
do {
680-
Options options;
681-
options.max_manifest_file_size = 10; // 10 bytes
682-
options = CurrentOptions(options);
683-
CreateAndReopenWithCF({"pikachu"}, options);
684-
{
685-
ASSERT_OK(Put(1, "manifest_key1", std::string(1000, '1')));
686-
ASSERT_OK(Put(1, "manifest_key2", std::string(1000, '2')));
687-
ASSERT_OK(Put(1, "manifest_key3", std::string(1000, '3')));
688-
uint64_t manifest_before_flush = dbfull()->TEST_Current_Manifest_FileNo();
689-
ASSERT_OK(Flush(1)); // This should trigger LogAndApply.
690-
uint64_t manifest_after_flush = dbfull()->TEST_Current_Manifest_FileNo();
691-
ASSERT_GT(manifest_after_flush, manifest_before_flush);
692-
ReopenWithColumnFamilies({"default", "pikachu"}, options);
693-
ASSERT_GT(dbfull()->TEST_Current_Manifest_FileNo(), manifest_after_flush);
694-
// check if a new manifest file got inserted or not.
695-
ASSERT_EQ(std::string(1000, '1'), Get(1, "manifest_key1"));
696-
ASSERT_EQ(std::string(1000, '2'), Get(1, "manifest_key2"));
697-
ASSERT_EQ(std::string(1000, '3'), Get(1, "manifest_key3"));
698-
}
699-
} while (ChangeCompactOptions());
700-
}
701-
702678
TEST_F(DBBasicTest, IdentityAcrossRestarts) {
703679
constexpr size_t kMinIdSize = 10;
704680
do {

db/db_etc3_test.cc

Lines changed: 161 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,161 @@
1+
// Copyright (c) 2011-present, Facebook, Inc. All rights reserved.
2+
// This source code is licensed under both the GPLv2 (found in the
3+
// COPYING file in the root directory) and Apache 2.0 License
4+
// (found in the LICENSE.Apache file in the root directory).
5+
6+
#include "db/db_test_util.h"
7+
8+
namespace ROCKSDB_NAMESPACE {
9+
10+
class DBEtc3Test : public DBTestBase {
11+
public:
12+
DBEtc3Test() : DBTestBase("db_etc3_test", /*env_do_fsync=*/true) {}
13+
};
14+
15+
TEST_F(DBEtc3Test, ManifestRollOver) {
16+
do {
17+
Options options;
18+
// Force new manifest on each manifest write
19+
options.max_manifest_file_size = 0;
20+
options.max_manifest_space_amp_pct = 0;
21+
options = CurrentOptions(options);
22+
CreateAndReopenWithCF({"pikachu"}, options);
23+
{
24+
ASSERT_OK(Put(1, "key1", std::string(1000, '1')));
25+
ASSERT_OK(Put(1, "key2", std::string(1000, '2')));
26+
ASSERT_OK(Put(1, "key3", std::string(1000, '3')));
27+
uint64_t manifest_before_flush = dbfull()->TEST_Current_Manifest_FileNo();
28+
ASSERT_OK(Flush(1)); // This should trigger LogAndApply.
29+
uint64_t manifest_after_flush = dbfull()->TEST_Current_Manifest_FileNo();
30+
ASSERT_GT(manifest_after_flush, manifest_before_flush);
31+
// Re-open should always re-create manifest file
32+
ReopenWithColumnFamilies({"default", "pikachu"}, options);
33+
ASSERT_GT(dbfull()->TEST_Current_Manifest_FileNo(), manifest_after_flush);
34+
ASSERT_EQ(std::string(1000, '1'), Get(1, "key1"));
35+
ASSERT_EQ(std::string(1000, '2'), Get(1, "key2"));
36+
ASSERT_EQ(std::string(1000, '3'), Get(1, "key3"));
37+
}
38+
} while (ChangeCompactOptions());
39+
}
40+
41+
TEST_F(DBEtc3Test, AutoTuneManifestSize) {
42+
// Ensure we have auto-tuning beyond max_manifest_file_size by default
43+
ASSERT_EQ(DBOptions{}.max_manifest_space_amp_pct, 500);
44+
45+
Options options = CurrentOptions();
46+
ASSERT_OK(db_->SetOptions({{"level0_file_num_compaction_trigger", "20"}}));
47+
48+
// Use large column family names to essentially control the amount of payload
49+
// data needed for the manifest file. Drop manifest entries don't include the
50+
// CF name so are small.
51+
uint64_t prev_manifest_num = 0, cur_manifest_num = 0;
52+
std::deque<ColumnFamilyHandle*> handles;
53+
int counter = 5;
54+
auto AddCfFn = [&]() {
55+
std::string name = "cf" + std::to_string(counter++);
56+
name.resize(1000, 'a');
57+
ASSERT_OK(db_->CreateColumnFamily(options, name, &handles.emplace_back()));
58+
prev_manifest_num = cur_manifest_num;
59+
cur_manifest_num = dbfull()->TEST_Current_Manifest_FileNo();
60+
};
61+
auto DropCfFn = [&]() {
62+
ASSERT_OK(db_->DropColumnFamily(handles.front()));
63+
ASSERT_OK(db_->DestroyColumnFamilyHandle(handles.front()));
64+
handles.pop_front();
65+
prev_manifest_num = cur_manifest_num;
66+
cur_manifest_num = dbfull()->TEST_Current_Manifest_FileNo();
67+
};
68+
auto TrivialManifestWriteFn = [&]() {
69+
ASSERT_OK(Put("x", std::to_string(counter++)));
70+
ASSERT_OK(Flush());
71+
prev_manifest_num = cur_manifest_num;
72+
cur_manifest_num = dbfull()->TEST_Current_Manifest_FileNo();
73+
};
74+
75+
options.max_manifest_file_size = 1000000;
76+
options.max_manifest_space_amp_pct = 0; // no auto-tuning yet
77+
DestroyAndReopen(options);
78+
79+
// With the generous (minimum) maximum manifest size, should not be rotated
80+
AddCfFn();
81+
AddCfFn();
82+
AddCfFn();
83+
ASSERT_EQ(prev_manifest_num, cur_manifest_num);
84+
85+
// Change options for small max and (still) no auto-tuning
86+
ASSERT_OK(db_->SetDBOptions({{"max_manifest_file_size", "3000"}}));
87+
88+
// Takes effect on the next manifest write
89+
TrivialManifestWriteFn();
90+
ASSERT_LT(prev_manifest_num, cur_manifest_num);
91+
92+
// Now we have to rewrite the whole manifest on each write because the
93+
// compacted size exceeds the "max" size.
94+
AddCfFn();
95+
ASSERT_LT(prev_manifest_num, cur_manifest_num);
96+
DropCfFn();
97+
ASSERT_LT(prev_manifest_num, cur_manifest_num);
98+
AddCfFn();
99+
ASSERT_LT(prev_manifest_num, cur_manifest_num);
100+
TrivialManifestWriteFn();
101+
ASSERT_LT(prev_manifest_num, cur_manifest_num);
102+
103+
// Enabling auto-tuning should fix this, immediately for next manifest writes.
104+
// This will allow up to double-ish the size of the compacted manifest,
105+
// which last should have been 4000 + some bytes.
106+
ASSERT_EQ(handles.size(), 4U);
107+
ASSERT_OK(db_->SetDBOptions({{"max_manifest_space_amp_pct", "105"}}));
108+
109+
// After 9 CF names should be enough to rotate the manifest
110+
for (int i = 1; i <= 5; ++i) {
111+
if ((i % 2) == 1) {
112+
DropCfFn();
113+
}
114+
AddCfFn();
115+
ASSERT_EQ(prev_manifest_num, cur_manifest_num);
116+
}
117+
TrivialManifestWriteFn();
118+
ASSERT_LT(prev_manifest_num, cur_manifest_num);
119+
120+
// We now have a different last compacted manifest size, should be
121+
// able to go beyond 9 CFs named in manifest this time.
122+
ASSERT_EQ(handles.size(), 6U);
123+
124+
DropCfFn();
125+
DropCfFn();
126+
for (int i = 1; i <= 4; ++i) {
127+
DropCfFn();
128+
AddCfFn();
129+
ASSERT_EQ(prev_manifest_num, cur_manifest_num);
130+
}
131+
// We've written 10 named CFs to the manifest. We should be able to
132+
// dynamically change the auto-tuning still based on the last "compacted"
133+
// manifest size of 7000 + some bytes.
134+
ASSERT_OK(db_->SetDBOptions({{"max_manifest_space_amp_pct", "51"}}));
135+
TrivialManifestWriteFn();
136+
ASSERT_LT(prev_manifest_num, cur_manifest_num);
137+
// And the "compacted" manifest size has reset again, so should be changed
138+
// again sooner.
139+
ASSERT_EQ(handles.size(), 4U);
140+
for (int i = 1; i <= 2; ++i) {
141+
AddCfFn();
142+
ASSERT_EQ(prev_manifest_num, cur_manifest_num);
143+
}
144+
// Enough for manifest change
145+
AddCfFn();
146+
ASSERT_LT(prev_manifest_num, cur_manifest_num);
147+
148+
// Wrap up
149+
while (!handles.empty()) {
150+
DropCfFn();
151+
}
152+
}
153+
154+
} // namespace ROCKSDB_NAMESPACE
155+
156+
int main(int argc, char** argv) {
157+
ROCKSDB_NAMESPACE::port::InstallStackTraceHandler();
158+
::testing::InitGoogleTest(&argc, argv);
159+
RegisterCustomObjects(argc, argv);
160+
return RUN_ALL_TESTS();
161+
}

db/db_impl/db_impl.cc

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -258,10 +258,10 @@ DBImpl::DBImpl(const DBOptions& options, const std::string& dbname,
258258
[this]() { this->TriggerPeriodicCompaction(); });
259259

260260
versions_.reset(new VersionSet(
261-
dbname_, &immutable_db_options_, file_options_, table_cache_.get(),
262-
write_buffer_manager_, &write_controller_, &block_cache_tracer_,
263-
io_tracer_, db_id_, db_session_id_, options.daily_offpeak_time_utc,
264-
&error_handler_, read_only));
261+
dbname_, &immutable_db_options_, mutable_db_options_, file_options_,
262+
table_cache_.get(), write_buffer_manager_, &write_controller_,
263+
&block_cache_tracer_, io_tracer_, db_id_, db_session_id_,
264+
options.daily_offpeak_time_utc, &error_handler_, read_only));
265265
column_family_memtables_.reset(
266266
new ColumnFamilyMemTablesImpl(versions_->GetColumnFamilySet()));
267267

@@ -1412,7 +1412,7 @@ Status DBImpl::SetDBOptions(
14121412
file_options_for_compaction_ = FileOptions(new_db_options);
14131413
file_options_for_compaction_ = fs_->OptimizeForCompactionTableWrite(
14141414
file_options_for_compaction_, immutable_db_options_);
1415-
versions_->ChangeFileOptions(mutable_db_options_);
1415+
versions_->UpdatedMutableDbOptions(mutable_db_options_, &mutex_);
14161416
// TODO(xiez): clarify why apply optimize for read to write options
14171417
file_options_for_compaction_ = fs_->OptimizeForCompactionTableRead(
14181418
file_options_for_compaction_, immutable_db_options_);

db/db_impl/db_impl_follower.cc

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -293,9 +293,9 @@ Status DB::OpenAsFollower(
293293
DBImplFollower* impl =
294294
new DBImplFollower(tmp_opts, std::move(new_env), dbname, src_path);
295295
impl->versions_.reset(new ReactiveVersionSet(
296-
dbname, &impl->immutable_db_options_, impl->file_options_,
297-
impl->table_cache_.get(), impl->write_buffer_manager_,
298-
&impl->write_controller_, impl->io_tracer_));
296+
dbname, &impl->immutable_db_options_, impl->mutable_db_options_,
297+
impl->file_options_, impl->table_cache_.get(),
298+
impl->write_buffer_manager_, &impl->write_controller_, impl->io_tracer_));
299299
impl->column_family_memtables_.reset(
300300
new ColumnFamilyMemTablesImpl(impl->versions_->GetColumnFamilySet()));
301301
impl->wal_in_db_path_ = impl->immutable_db_options_.IsWalDirSameAsDBPath();

db/db_impl/db_impl_open.cc

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -329,7 +329,7 @@ Status DBImpl::NewDB(std::vector<std::string>* new_filenames) {
329329
}
330330
FileTypeSet tmp_set = immutable_db_options_.checksum_handoff_file_types;
331331
file->SetPreallocationBlockSize(
332-
immutable_db_options_.manifest_preallocation_size);
332+
mutable_db_options_.manifest_preallocation_size);
333333
std::unique_ptr<WritableFileWriter> file_writer(new WritableFileWriter(
334334
std::move(file), manifest, file_options, immutable_db_options_.clock,
335335
io_tracer_, nullptr /* stats */,

db/db_impl/db_impl_secondary.cc

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -783,9 +783,9 @@ Status DB::OpenAsSecondary(
783783
handles->clear();
784784
DBImplSecondary* impl = new DBImplSecondary(tmp_opts, dbname, secondary_path);
785785
impl->versions_.reset(new ReactiveVersionSet(
786-
dbname, &impl->immutable_db_options_, impl->file_options_,
787-
impl->table_cache_.get(), impl->write_buffer_manager_,
788-
&impl->write_controller_, impl->io_tracer_));
786+
dbname, &impl->immutable_db_options_, impl->mutable_db_options_,
787+
impl->file_options_, impl->table_cache_.get(),
788+
impl->write_buffer_manager_, &impl->write_controller_, impl->io_tracer_));
789789
impl->column_family_memtables_.reset(
790790
new ColumnFamilyMemTablesImpl(impl->versions_->GetColumnFamilySet()));
791791
impl->wal_in_db_path_ = impl->immutable_db_options_.IsWalDirSameAsDBPath();

0 commit comments

Comments
 (0)