Cost and Factorization Analysis Utility#577
Open
ajay-mk wants to merge 16 commits into
Open
Conversation
A standalone utility that estimates the cost of evaluating a serialized tensor equation. Reads a JSON driver (space sizes + equation files), strips the (anti)symmetrizer, flattens, then per summand optimizes and binarizes once, reusing that tree for the intermediate catalog, cache simulation, and --dump_tree, mirroring MPQC's SeQuantEngine. Emits a Markdown report on largest intermediates, most expensive contractions, shape census, peak storage, FLOPs, and a gated cache simulation. Includes a README and CCSD/DF example drivers.
Diff generated reports against frozen *.md.expected via run_tests.cmake (volatile lines stripped): a spin-orbital CCSD R2 (a Sum of terms, with a --dump_tree check) and a single density-fitted product (the non-Sum path).
Krzmbrzl
reviewed
Jul 9, 2026
Contributor
There was a problem hiding this comment.
Pull request overview
This PR introduces a new standalone cost_analysis utility under utilities/ that reads a JSON driver, optimizes/binarizes SeQuant equations, and emits a Markdown report (plus optional tree dumps) along with reference-style CTest coverage.
Changes:
- Added the
cost_analysisexecutable with JSON-driven configuration, equation parsing, optimization/binarization, cataloging, and optional cache simulation. - Added Markdown report generation and accompanying documentation/examples with frozen
*.md.expectedfixtures. - Integrated the tool into the build and added CTest-based golden-file regression tests.
Reviewed changes
Copilot reviewed 15 out of 15 changed files in this pull request and generated 6 comments.
Show a summary per file
| File | Description |
|---|---|
| utilities/cost_analysis/run_tests.cmake | CMake script to run the utility and compare generated reports to expected fixtures (with volatile lines stripped). |
| utilities/cost_analysis/report.hpp | Declares report-writing and tree-string serialization helpers. |
| utilities/cost_analysis/report.cpp | Implements Markdown report formatting, table generation, and tree stringification. |
| utilities/cost_analysis/README.md | Documents the driver schema, workflow, report contents, and tests. |
| utilities/cost_analysis/examples/df_r1.md.expected | Golden expected report output for DF R1 example. |
| utilities/cost_analysis/examples/df_r1.json | JSON driver for DF R1 example. |
| utilities/cost_analysis/examples/df_r1.inp | Serialized equation input for DF R1 example. |
| utilities/cost_analysis/examples/ccsd_r2.md.expected | Golden expected report output for CCSD R2 example. |
| utilities/cost_analysis/examples/ccsd_r2.json | JSON driver for CCSD R2 example (includes dump_tree). |
| utilities/cost_analysis/examples/ccsd_r2.inp | Serialized equation input for CCSD R2 example. |
| utilities/cost_analysis/examples/.gitignore | Ignores generated reports, filtered files, and dump-tree outputs. |
| utilities/cost_analysis/cost_analysis.hpp | Defines shared config/result data structures between analysis and reporting. |
| utilities/cost_analysis/cost_analysis.cpp | Implements the CLI entrypoint, JSON parsing, context setup, analysis pipeline, cache simulation, and report emission. |
| utilities/cost_analysis/CMakeLists.txt | Adds build target and CTest registrations for the new utility. |
| utilities/CMakeLists.txt | Wires utilities/cost_analysis into the utilities build. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
The generated timestamp changed every run; only the volatile-line stripping in run_tests.cmake kept the reference tests deterministic. Remove it (and the now-unused <chrono> include). Keep the SeQuant version/hash line, still stripped before diffing. Reference fixtures now carry a <version> placeholder instead of a frozen timestamp+hash.
ajay-mk
marked this pull request as ready for review
July 10, 2026 21:10
The input list is the equations to analyze; 'results' collided with the tool's computed output. Rename the JSON key, EquationSpec, and Config field so input equations read distinctly from output results.
Mirrors the existing min_repeats guard: a negative top_n wraps to a huge size_t, and a negative max_footprint silently disables the cache budget.
The driver chdir's into its own parent, so a relative output.path resolves there rather than the invocation directory; print the absolute path in the confirmation line and document the resolution rule. Also document the registry-default fallback for unlisted sizes, correct the report layout description (Cache is one pooled section, not per-result), and trim the CSV/PNO note.
Krzmbrzl
approved these changes
Jul 15, 2026
Krzmbrzl
left a comment
Krzmbrzl
left a comment
Collaborator
There was a problem hiding this comment.
I'm wondering whether there is potential for merging with the external interface. I am currently in the process of rewriting its core design to be processing-step oriented and in this design creating this cost analysis could be a useful thing to be capable of.
Plus, this would have the advantage of a unified JSON driver input syntax.
That's something for the future though and shouldn't hold up this PR.
Member
Author
I think that is a good idea. I initially considered doing that, but wasn't sure if we would want to merge this utility into SeQuant. I wrote it mainly to see how factorization changes when we apply DF/THC to the equations, but turned out to be quite useful. |
Collaborator
|
Yeah, I think it wouldn't really fit with the current structure of the external interface but with the refactor it should be relatively easy to pull something like this off 👍 |
Replace the run_tests.cmake -P wrapper with a generate, sed (strip volatile line), diff multistep process.
Krzmbrzl
reviewed
Jul 15, 2026
Add --omit-revision to drop the volatile git-revision line, and end the report with a single trailing newline (leading section separators instead of trailing ones) so it is byte-stable. The tests then use plain cmake -E compare_files instead of sed/diff, making them portable. The tree dump is diffed against a new R2.tree.txt.expected fixture.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Cost and Factorization Analysis Utility
Adds
utilities/cost_analysis, a standalone tool that estimates the cost of evaluating a tensor equation without running it. It reads a serialized equation, optimizes and binarizes it, then writes a Markdown report on the largest intermediates, the most expensive contractions, peak storage, operation counts, and cache reuse.Inspiration is from MPQC's
SeQuantEngineclass and the several knobs it exposes.What it does
<head> = <rhs>in SeQuant serialization V1), plus context (spin basis, field, index-space sizes, factorization spaces).deserialize→ strip leading (anti)symmetrizer →flatten→optimize+binarize→ catalog the tree's internal nodes → cache simulation across all results.contractions, a Shape census by O/V/X signature, and a Cache table.
Usage
All driver knobs and their defaults are documented in the utility's
README.md.Tests
Two reference tests (
ctest -R cost_analysis_) run the tool onexamples/and diff against frozen*.md.expected.Not supported right now
AsyCostis not proto-index aware; this utility should work onceAsyCostis extended.Example report:
EOM-CCSD(2h2p) R2 (53 terms; i=10, a=38)
Terms: 53; distinct intermediates: 94; reused: 9; largest: 8.3728 MB; peak storage: 40.1953 MB.
Total operations (symbolic): 4i^2a^4 + 18i^3a^3 + 26i^4a^2 + 2ia^4 + 30i^2a^3 + 104i^3a^2 + 26i^4a + 4ia^3 + 55i^2a^2 + 4i^3a + 10*i^2a
Largest intermediates
Most expensive contractions
Shape census
Cache (gated, volatile leaf = "R", min_repeats = 2)