Skip to content

Commit 9e4ccf8

Browse files
authored
Micro-optimize transitive depencency hash calculation (#20798)
In small incrementals runs targeting a very large code base, the `transitive_dep_hash` function used up to 9% of total CPU. Micro-optimize it by avoiding json dump, dict construction and `hex()` calls. I'll verify that this helps after this has been merged, but the improved performance (esp. without orjson) seems very likely.
1 parent a7e66a6 commit 9e4ccf8

1 file changed

Lines changed: 15 additions & 11 deletions

File tree

mypy/build.py

Lines changed: 15 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -48,6 +48,7 @@
4848
read_str as read_str_bare,
4949
read_tag,
5050
write_bool,
51+
write_bytes as write_bytes_bare,
5152
write_int as write_int_bare,
5253
write_str as write_str_bare,
5354
write_tag,
@@ -4367,17 +4368,20 @@ def deps_filtered(graph: Graph, vertices: AbstractSet[str], id: str, pri_max: in
43674368

43684369
def transitive_dep_hash(scc: SCC, graph: Graph) -> bytes:
43694370
"""Compute stable snapshot of transitive import structure for given SCC."""
4370-
all_direct_deps = {
4371-
dep
4372-
for id in scc.mod_ids
4373-
for dep in graph[id].dependencies
4374-
if graph[id].priorities.get(dep) != PRI_INDIRECT
4375-
}
4376-
trans_dep_hash_map = {
4377-
dep_id: "" if dep_id in scc.mod_ids else graph[dep_id].trans_dep_hash.hex()
4378-
for dep_id in all_direct_deps
4379-
}
4380-
return hash_digest_bytes(json_dumps(trans_dep_hash_map))
4371+
all_direct_deps = sorted(
4372+
{
4373+
dep
4374+
for id in scc.mod_ids
4375+
for dep in graph[id].dependencies
4376+
if graph[id].priorities.get(dep) != PRI_INDIRECT
4377+
}
4378+
)
4379+
buf = WriteBuffer()
4380+
for dep_id in all_direct_deps:
4381+
write_str_bare(buf, dep_id)
4382+
if dep_id not in scc.mod_ids:
4383+
write_bytes_bare(buf, graph[dep_id].trans_dep_hash)
4384+
return hash_digest_bytes(buf.getvalue())
43814385

43824386

43834387
def missing_stubs_file(cache_dir: str) -> str:

0 commit comments

Comments
 (0)