Skip to content

Commit a95e814

Browse files
jermpclaude
andauthored
fix pthash memory estimate (#97)
* Account for input hash vector in construction memory estimate estimate_num_bytes_for_construction returned max(map_bytes, search_bytes), but the input hash vector lives across both phases. The partitioned external builder pushes the per-partition hashes (num_keys * sizeof(hash_type)) into in_memory_partitions and only adds the build estimate to its running 'bytes' counter, so the bytes < config.ram gate underestimated true residency by num_keys * sizeof(hash_type) per partition. For hash128 (16B) plus the search-phase peak (~25B/key), peak per- partition residency is ~41B/key while the gate believed ~25B/key, so config.ram=2GiB actually allowed ~3.2GB resident. Adding the input hashes outside the max makes config.ram match observed RSS. * Drop the redundant in-search hashes term and the comment The previous commit added the input hashes outside the max with the correct sizeof(hash_type), so the pre-existing num_keys * sizeof(uint64_t) term inside num_bytes_for_search was double-counting. --------- Co-authored-by: Claude <noreply@anthropic.com>
1 parent e04a192 commit a95e814

1 file changed

Lines changed: 2 additions & 2 deletions

File tree

include/builders/internal_memory_builder_single_phf.hpp

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -211,10 +211,10 @@ struct internal_memory_builder_single_phf {
211211
num_buckets * sizeof(uint64_t) // pilots
212212
+ num_buckets * sizeof(uint64_t) // buckets
213213
+ (config.minimal ? (table_size - num_keys) * sizeof(uint64_t) : 0) // free_slots
214-
+ num_keys * sizeof(uint64_t) // hashes
215214
+ table_size / 8; // bitmap taken
216215

217-
return std::max<uint64_t>(num_bytes_for_map, num_bytes_for_search);
216+
return std::max<uint64_t>(num_bytes_for_map, num_bytes_for_search) +
217+
num_keys * sizeof(typename hasher_type::hash_type);
218218
}
219219

220220
private:

0 commit comments

Comments
 (0)