Skip to content

optimize: Optimize rockdb batch query performance #2982

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
lokidundun wants to merge 15 commits into
base: master
Choose a base branch
from
Open

optimize: Optimize rockdb batch query performance #2982

Show file tree
Hide file tree
Changes from 6 commits
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
5ba6d4a
optimize: Optimize RocksDB batch query performance
lokidundun Mar 29, 2026
f5405a0
Refactor getByIds to queryByIds in RocksDBTable
lokidundun Mar 31, 2026
939ace0
Modify queryByIds to use super method temporarily
lokidundun Mar 31, 2026
502c7df
Refactor queryByIds to getByIds with HashSet
lokidundun Apr 2, 2026
352f66b
Update RocksDBTables.java
lokidundun Apr 2, 2026
0d9052d
fix: fix ci
lokidundun Apr 6, 2026
45298f9
optimize: optimize the batch query
lokidundun Apr 7, 2026
223fb28
optimize: optimize rockDb query
lokidundun Apr 16, 2026
ce4e2cb
fix: fix ci test
lokidundun Apr 16, 2026
6fe08a7
fix: preserve input id multiplicity in RocksDB multi-get path
Copilot Apr 17, 2026
0391069
fix: fallback getByIds when rocksdb session has pending changes
Copilot Apr 17, 2026
ce802b9
revert: restore rocksdb query files to ce4e2cb state
Copilot Apr 17, 2026
c06225c
fix: fix wrong usage
lokidundun Apr 17, 2026
a99491b
Merge branch 'improvequery' of https://github.com/lokidundun/incubato…
lokidundun Apr 17, 2026
10cb0a4
test: add test for new multi-get path
lokidundun Apr 17, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 21 additions & 7 deletions ...server/hugegraph-core/src/main/java/org/apache/hugegraph/backend/tx/GraphTransaction.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -784,7 +784,8 @@ protected Iterator<Vertex> queryVerticesByIds(Object[] vertexIds, boolean adjace
List<Id> ids = InsertionOrderUtil.newList();
Map<Id, HugeVertex> vertices = new HashMap<>(vertexIds.length);

IdQuery query = new IdQuery(type);
List<Id> backendIds = InsertionOrderUtil.newList();

for (Object vertexId : vertexIds) {
HugeVertex vertex;
Id id = HugeVertex.getIdValue(vertexId);
Expand All @@ -799,17 +800,30 @@ protected Iterator<Vertex> queryVerticesByIds(Object[] vertexIds, boolean adjace
// Found from local tx
vertices.put(vertex.id(), vertex);
} else {
// Prepare to query from backend store
query.query(id);
// store the IDs queried from backend
backendIds.add(id);
}
ids.add(id);
}

if (!query.empty()) {
if (!backendIds.isEmpty()) {
// Query from backend store
query.mustSortByInput(false);
Iterator<HugeVertex> it = this.queryVerticesFromBackend(query);
QueryResults.fillMap(it, vertices);
final int batch = this.batchSize > 0 ? this.batchSize : backendIds.size();
Copy link

Copilot AI Apr 6, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

final int batch = this.batchSize > 0 ? this.batchSize : backendIds.size(); is effectively always this.batchSize because query.batch_size is configured with a minimum of 1 (see CoreOptions.QUERY_BATCH_SIZE). Removing the dead fallback branch will simplify the code.

Suggested change
final int batch = this.batchSize > 0 ? this.batchSize : backendIds.size();
final int batch = this.batchSize;

Copilot uses AI. Check for mistakes.
for (int i = 0; i < backendIds.size(); i += batch) {
int end = Math.min(i + batch, backendIds.size());
IdQuery query = new IdQuery(type);
Copy link

Copilot AI Apr 6, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The PR title says this is a RocksDB optimization, but the functional change here is backend-agnostic batching in GraphTransaction (affecting all stores, especially RPC-based ones). Please align the PR title (or add RocksDB-specific changes) so the intent matches what’s actually being modified.

Copilot uses AI. Check for mistakes.
for (int j = i; j < end; j++) {
Id id = backendIds.get(j);
query.query(id);
}
Copy link

Copilot AI Apr 6, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With batching, duplicated ids that fall into different batches will trigger repeated backend reads/RPCs for the same id. You can keep the output behavior (duplicates preserved via ids) while deduplicating backend fetches (e.g., track a seen-set for backendIds or build per-batch unique ids) to avoid redundant backend queries.

Copilot uses AI. Check for mistakes.
// Single batch capacity check
Query.checkForceCapacity(query.idsSize());

// Query from backend store
query.mustSortByInput(false);
Iterator<HugeVertex> it = this.queryVerticesFromBackend(query);
QueryResults.fillMap(it, vertices);
}
Copy link

Copilot AI Apr 6, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The new multi-batch path isn’t covered by tests. Please add a unit/integration test that exercises queryVerticesByIds() with vertexIds.length > query.batch_size, including (1) duplicates across a batch boundary and (2) mixed local-tx + backend ids, to ensure results and NotFoundException behavior remain unchanged.

Copilot uses AI. Check for mistakes.
}

return new MapperIterator<>(ids.iterator(), id -> {
Expand Down
2 changes: 1 addition & 1 deletion ...egraph-rocksdb/src/main/java/org/apache/hugegraph/backend/store/rocksdb/RocksDBTable.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -309,7 +309,7 @@ protected static BackendEntryIterator newEntryIterator(BackendColumnIterator col
}

protected static BackendEntryIterator newEntryIteratorOlap(
BackendColumnIterator cols, Query query, boolean isOlap) {
BackendColumnIterator cols, Query query, boolean isOlap) {
return new BinaryEntryIterator<>(cols, query, (entry, col) -> {
if (entry == null || !entry.belongToMe(col)) {
HugeType type = query.resultType();
Expand Down
Loading