Skip to content

Refactor social network data load script to use just 20 queries#41

Merged
alexjpwalker merged 1 commit into
typedb:masterfrom
alexjpwalker:social-network-few-queries
Apr 24, 2026
Merged

Refactor social network data load script to use just 20 queries#41
alexjpwalker merged 1 commit into
typedb:masterfrom
alexjpwalker:social-network-few-queries

Conversation

@alexjpwalker
Copy link
Copy Markdown
Member

Product change and motivation

The social-network sample dataset now loads its data using very few queries (20, down from about 4800).

This allows HTTP-based clients to load it very quickly without getting bottlenecked hard by network latency.

Implementation

By making the match stages much chunkier we can drastically cut down on the total number of insert stages required to insert all data. For some types, we go from N queries to 1 for insertion of all instances of said type.

Note that due to a quirk of TypeDB server at the time of writing the chunky match stages are themselves chunked into pipelines that look like match-match-match .... match-insert, sometimes with as many as 100 match stages. This drastically improves performance.

@alexjpwalker alexjpwalker merged commit 59d569f into typedb:master Apr 24, 2026
7 checks passed
@alexjpwalker alexjpwalker deleted the social-network-few-queries branch April 24, 2026 13:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant