Comparing codegraph quality vs agent explore tooling?

[leblancdaniel]

Hello, and thank you for this work - it's really interesting and the efficiency/cost story is very clear.

I was wondering if there was any benchmark comparison on the quality of the results using codegraph vs the standard tooling? I would imagine this largely varies on the task, codebase, etc. but it would be really helpful to understand the implication on agent response quality and if there is a real tradeoff there.

Thanks again!
-Daniel

[FulanXisen]

Here is a very simple test i used:

Ask this question to agent:

You answer the same question using Codegraph and without using Codegraph: What is the startup process of [XYZ]? Then please tell me if Codegraph was helpful to you and how much it was helpful.

[colbymchenry]

Fair question. The published benchmarks (in the README) measure efficiency — cost, total tokens, time, and tool-call count — across 7 real repos, comparing an agent answering an architecture question with vs without CodeGraph. There isn't a formal answer-quality A/B score.

Two honest notes on quality:

• CodeGraph returns deterministic, AST-derived facts — exact symbol locations, real call/impact edges, trace paths — not summaries or guesses. That gives the agent precise structural ground truth instead of grep approximations, and in the runs the with-CodeGraph arm reaches the answer with far fewer Read/grep dead-ends, which tends to reduce wrong turns.
• But that's a mechanism argument, not a measured quality delta — I don't want to overclaim a quality win we haven't formally scored.

So: efficiency is benchmarked and consistent; quality isn't formally A/B'd, but the design is "exact facts, not lossy retrieval," so it shouldn't trade quality for the efficiency gains. Closing as answered — appreciate the thoughtful question.