Semi-Sparse Abstract Interpretation (#1803 rebased on #1804)#1806
Semi-Sparse Abstract Interpretation (#1803 rebased on #1804)#1806bjjwwang wants to merge 27 commits into
Conversation
ObjVars propagate densely via joinWith; ValVars are pulled on-demand from their unique def-sites via buildSparseState/getAbstractValue. Key changes: - New -semi-sparse CLI option (Options::SemiSparse) - PreAnalysis::buildOrphanVarDefMap for ValVars with null getICFGNode - getAbstractValue returns by value, handles constants and orphan vars - collectNeededVarIds + buildSparseState pull ValVars before execution - pullBranchConditionVars ensures branch feasibility in sparse mode - updateStateOnPhi uses getAbstractValue for phi operands All 348 ae tests pass in both dense and semi-sparse modes. Made-with: Cursor
…e strategy Use joinAddrOnly for intra-procedural edges (only merge ObjVars) and joinWith for inter-procedural edges (preserve formal params/returns). Key changes: - Add AbstractState::joinAddrOnly to merge only _addrToAbsVal - Split mergeStatesFromPredecessors into intra/inter edge handling - Fix getAbstractValue to use varMap.count() for bottom value detection - Fix PreAnalysis::buildOrphanVarDefMap to always map RetPE vars Results: 345/348 tests pass, 97.7% VarEntry reduction, 70.6% total state size reduction. 3 set_value-related tests remain unsupported. Made-with: Cursor
Made-with: Cursor
After PR SVF-tools#1804 fixed ICFGNode assignments for ValVar subclasses, the only remaining mismatch is call-result ValVars (RetPE LHS) whose getICFGNode() returns CallICFGNode but whose value is written at RetICFGNode. Use CallICFGNode::getRetICFGNode() directly instead of pre-building a map. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Move the @@STATE_SIZE@@ measurement snippet to semi-sparse.md as documentation for future benchmarking. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
|
|
||
| const Option<bool> Options::SparseAE( | ||
| "sparse-ae", "Enable sparse abstract execution via SVFG", false); | ||
| const Option<bool> Options::SemiSparse( |
There was a problem hiding this comment.
Better to make a strong option for all non-sparse, semi and sparse settings as one option
There was a problem hiding this comment.
Better to make a strong option for all non-sparse, semi and sparse settings as one option
OK. got it.
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #1806 +/- ##
==========================================
- Coverage 63.84% 63.52% -0.33%
==========================================
Files 247 247
Lines 24858 25002 +144
Branches 4673 4721 +48
==========================================
+ Hits 15871 15882 +11
- Misses 8987 9120 +133
🚀 New features to boost your workflow:
|
… AbstractState - Reorder getAbstractValue/updateAbstractValue params to (var, node) consistently - Add AESparsity option to group dense/semi-sparse/sparse modes - AbsExtAPI now holds AbstractInterpretation& instead of raw abstractTrace reference - AEDetector::detect takes AbstractInterpretation& instead of AbstractState& - Add override to AEDetector subclass virtual methods Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Decompose loadValue/storeValue into explicit ValVar lookup + ObjVar load/store - Replace all as[id] in updateStateOnLoad/Store/Copy/Gep/Select/Phi/Call/Ret/Addr/Binary/Cmp - Replace all as[id] in AbsExtAPI ext function handlers and SSE_FUNC_PROCESS macro - Replace as[id] in AEDetector detectExtAPI and handleStubFunctions - Remaining as[id] only in: API implementations, low-level state utilities (canSafelyAccessMemory, canSafelyDerefPtr, getStrlen) - Update compatible_dense_semisparse.md with design documentation Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Add ICFGNode* node parameter to getStrlen, canSafelyAccessMemory, canSafelyDerefPtr - Replace as[value_id] with ae.getAbstractValue(var, node) in all three functions - Update all call sites to pass the node context - AbsExtAPI.cpp and AEDetector.cpp now have zero as[idx] bracket accesses Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…dling - Dense mode: read ValVar from current node's state (not def-site) - Semi-sparse mode: pull ValVar from def-site (unchanged) - Fix SVFVar* dispatch: check ObjVar before ValVar (ObjVar inherits ValVar) - Restore direct as[id] for AddrStmt RHS (ObjVar virtual address in _varToVal) - Fixes 97% of test regressions (345/347 semi-sparse, 344/347 dense pass) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…in Binary/Cmp Uninitialized ValVars (e.g., uninitialized local variables, function params) default to bottom when read via as[id]. The old code had explicit guards in updateStateOnBinary and updateStateOnCmp to set them to top (conservative). This prevents isBranchFeasible from falsely pruning reachable branches. Dense: 347/347 pass. Semi-sparse: 345/347 (2 pre-existing). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…irst In semi-sparse mode, buildSparseState and defensive top initialization write into the current node's state. getAbstractValue must check the current state before falling back to def-site lookup, otherwise those writes are invisible. Dense: 347/347 pass. Semi-sparse: 347/347 pass. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…retation - Remove buildSparseState/collectNeededVarIds/addIfValVar — getAbstractValue handles on-demand ValVar lookup from def-sites - Add getGepElementIndex, getGepByteOffset, getGepObjAddrs to AbstractInterpretation, replacing AbstractState methods that used (*this)[id] - Add predNode parameter to isBranchFeasible/isCmpBranchFeasible/ isSwitchBranchFeasible, use getAbstractValue to initialize missing ValVars - Binary/Cmp: treat bottom operands as top locally (no state mutation) - Update AbsExtAPI handleMemcpy/handleMemset/strRead/getStrlen signatures to accept ICFGNode* for getGepObjAddrs calls Dense: 346/347. Semi-sparse: 341/347. Remaining failures: uninitialized function parameters (argc etc.) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
getPointeeElement reads pointer value from state via inVarToAddrsTable, but semi-sparse mode doesn't merge ValVars into the current node's state. Fix: pull pointer value via ae.getAbstractValue before calling getPointeeElement, and add null check for safety. Dense: 346/347. Semi-sparse: 347/347 (100%). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Dense mode previously used as[id] (map::operator[]) which silently inserts a default-constructed bottom entry. This bottom has no analysis semantics — it's just a C++ side-effect — but downstream code (isBranchFeasible narrow) interprets it as "path infeasible", falsely pruning reachable branches. Fix: check inVarToValTable/inVarToAddrsTable first. If absent, fall through to the final top fallback. This correctly models "uninitialized = unknown". Dense: 347/347 (100%). Semi-sparse: 347/347 (100%). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…sFromPredecessors - updateStateOnPhi: unified phi operand read via getAbstractValue(opVar, opICFGNode) instead of separate dense/semi-sparse paths with tmpEs[curId] - pullBranchConditionVars: remove semiSparse guard — dense mode safely no-ops when vars are already in state - Only two sparsity checks remain: getAbstractValue (lookup strategy) and mergeStatesFromPredecessors (joinAddrOnly vs joinWith) Dense: 347/347 (100%). Semi-sparse: 347/347 (100%). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
pullValVarIntoState is no longer needed — isCmpBranchFeasible now pulls ValVars via getAbstractValue directly. pullLoadChainPointerIntoState is still needed: isCmpBranchFeasible narrows ObjVars through load pointers (e.g., narrowing *%a.addr when %0 = load %a.addr is refined). It needs the pointer's AddressValue in the state to find which ObjVar to narrow. Inlined the pull logic as a local lambda. pullBranchConditionVars now only pulls load-chain pointers, not ValVars. Dense: 347/347 (100%). Semi-sparse: 347/347 (100%). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Document getAbstractValue/updateAbstractValue (dense vs semi-sparse strategy, input/output, when top is returned), GEP helpers (getGepElementIndex, getGepByteOffset, getGepObjAddrs — why they were lifted from AbstractState), and branch condition helpers (pullBranchConditionVars, pullLoadChainPointerIntoState — the load-chain pointer pulling for ObjVar narrowing in isCmpBranchFeasible). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…Only joinWith now checks Options::AESparsity internally — in semi-sparse mode it skips ValVar (_varToAbsVal) merge since ValVars are pulled on demand from def-sites via getAbstractValue. This removes the last semiSparse check from mergeStatesFromPredecessors. Now only getAbstractValue(ValVar*) checks AESparsity (lookup strategy). joinWith is the single merge API for both modes. Dense: 347/347 (100%). Semi-sparse: 347/347 (100%). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…State These functions pre-pulled load-chain pointers (e.g., %ptr from `%x = load i32, i32* %ptr`) into predecessor state copies for isCmpBranchFeasible's ObjVar narrowing. This is unnecessary: pointer AddressValues originate from AddrStmt and are propagated via joinWith to all reachable states. ObjVars are always dense in both modes, so the predecessor state copy already contains all needed pointer addresses and ObjVar contents. Dense: 347/347 (100%). Semi-sparse: 347/347 (100%). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Remove 7 functions from AbstractState that internally used (*this)[id] to read ValVars, which doesn't work in semi-sparse mode: Deleted (already have AbstractInterpretation equivalents): - getGepObjAddrs, getElementIndex, getByteOffset, loadValue, storeValue Lifted to AbstractInterpretation (now use getAbstractValue): - getPointeeElement(SVFVar*, ICFGNode*) — resolves pointer to pointee type - getAllocaInstByteSize(AddrStmt*, ICFGNode*) — reads VLA size variables Also updated: - AbsExtAPI::getElementSize: removed AbstractState& param, uses ICFGNode* - ae.cpp unit test: inlined load/store logic AbstractState is now a pure data container: operator[], load/store (ObjVar), initObjVar, join/meet/widen/narrow, print. Dense: 347/347 (100%). Semi-sparse: 347/347 (100%). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
New class AbstractStateManager owns abstractTrace and all state access APIs: - getAbstractValue / updateAbstractValue (all overloads) - getAbstractState / hasAbstractState - getGepElementIndex / getGepByteOffset / getGepObjAddrs - getPointeeElement / getAllocaInstByteSize AbstractInterpretation now owns AbstractStateManager* svfStateMgr and delegates through inline wrappers. All abstractTrace references in AbstractInterpretation.cpp replaced with svfStateMgr->getTrace(). Separation of concerns: - AbstractStateManager: state storage + value access + GEP/type helpers - AbstractInterpretation: analysis engine (WTO, fixpoint, statements) - AbstractState: pure data container (operator[], load/store, join/meet) Dense: 347/347 (100%). Semi-sparse: 347/347 (100%). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Remove all inline wrapper methods from AbstractInterpretation (getAbstractValue, updateAbstractValue, getAbstractState, etc.) - AbsExtAPI now holds AbstractStateManager* instead of AbstractInterpretation& - AEDetector::detect takes AbstractStateManager& instead of AbstractInterpretation& - AbstractInterpretation.cpp: all state calls go through svfStateMgr-> - AEDetector.cpp: handleStubFunctions uses getAEInstance().getStateMgr() for state access, getAEInstance().getUtils() for checkpoints Dense: 347/347 (100%). Semi-sparse: 347/347 (100%). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Combines ValVar pointer lookup (sparsity-aware via getAbstractValue) with ObjVar load/store (always dense) into a single API: - loadValue(pointer, node): resolve address set + load from ObjVars - storeValue(pointer, val, node): resolve address set + store to ObjVars updateStateOnLoad/Store now delegate to these in one line each. Dense: 347/347 (100%). Semi-sparse: 347/347 (100%). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…ar to ValVar - Remove AbstractStateManager& param from AEDetector::detect() and all internal helpers; detectors now fetch the manager via AbstractInterpretation::getAEInstance().getStateMgr() - Remove AbstractState& param from AbsExtAPI::strRead, getStrlen, handleMemcpy, handleMemset; state is retrieved internally from the node - Narrow SVFVar* to ValVar* in getStrlen, getElementSize, strRead, handleMemcpy, handleMemset (callers always pass getArgument() results) - Remove dead dyn_cast<ObjVar>(ValVar*) branches (ObjVar and ValVar are siblings under SVFVar; cast always yields null) - Remove ICFGNode* node param from getElementSize (was unused after dead code removal) - Move getUseSitesOfObjVar, getUseSitesOfValVar, getDefSiteOfValVar, getDefSiteOfObjVar from PreAnalysis to AbstractStateManager - Move SVFG ownership from PreAnalysis to AbstractStateManager; AbstractStateManager builds SVFG itself (only when AESparsity==Sparse) using the AndersenWaveDiff* passed to its constructor Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
| if (const AddrStmt* addrStmt = SVFUtil::dyn_cast<AddrStmt>(stmt2)) | ||
| { | ||
| size = as.getAllocaInstByteSize(addrStmt); | ||
| size = AbstractInterpretation::getAEInstance().getStateMgr()->getAllocaInstByteSize(addrStmt); |
There was a problem hiding this comment.
Do we have the statemgr as a field of the AEDetector class?
| { | ||
| // void SAFE_BUFACCESS(void* data, int size); | ||
| AbstractInterpretation::getAEInstance().getUtils()->checkpoints.erase(callNode); | ||
| AbstractInterpretation& aeInst = AbstractInterpretation::getAEInstance(); |
There was a problem hiding this comment.
Do we have a AI instance in the AEDetector?
| // an unknown but valid memory location. This allows the analysis to continue | ||
| // while being conservatively sound. | ||
| if (!as[value_id].isAddr()) | ||
| AbstractInterpretation& aeInst = AbstractInterpretation::getAEInstance(); |
There was a problem hiding this comment.
See above.
AbstractInterpretation::getAEInstance() has been invoked too frequently.
| if (const AddrStmt* addrStmt = SVFUtil::dyn_cast<AddrStmt>(stmt2)) | ||
| { | ||
| size = as.getAllocaInstByteSize(addrStmt); | ||
| size = AbstractInterpretation::getAEInstance().getStateMgr()->getAllocaInstByteSize(addrStmt); |
| std::vector<AbstractState> intraWorkList; | ||
| std::vector<AbstractState> interWorkList; | ||
| for (auto& edge : node->getInEdges()) |
There was a problem hiding this comment.
Do we still need two worklists here?
|
|
||
| bool AbstractInterpretation::isCmpBranchFeasible(const CmpStmt* cmpStmt, s64_t succ, | ||
| AbstractState& as) | ||
| AbstractState& as, const ICFGNode* predNode) |
There was a problem hiding this comment.
We only need an ICFGEdge argument here, no need to have as and preNode?
|
|
||
| bool AbstractInterpretation::isSwitchBranchFeasible(const SVFVar* var, s64_t succ, | ||
| AbstractState& as) | ||
| AbstractState& as, const ICFGNode* predNode) |
| } | ||
|
|
||
| bool AbstractInterpretation::isBranchFeasible(const IntraCFGEdge* intraEdge, | ||
| AbstractState& as) |
| if (hasAbstractState(globalNode)) | ||
| abstractTrace[node] = abstractTrace[globalNode]; | ||
| if (svfStateMgr->hasAbstractState(globalNode)) | ||
| svfStateMgr->getTrace()[node] = svfStateMgr->getTrace()[globalNode]; |
There was a problem hiding this comment.
Would be good to get ride of getTrace()[] to make code more readable and aligned with other AE/AS files.
Summary
Rebased version of #1803 on top of #1804, with further simplification.
Adds a new
-semi-sparsemode for abstract interpretation that treats top-level SSA variables (ValVars) sparsely and address-taken variables (ObjVars) densely. Since ValVars in LLVM IR are in SSA form (each defined exactly once), carrying them through every ICFG node is redundant — we can always fetch them from their unique definition site on demand. ObjVars are not SSA and still require dense propagation.This reduces the total abstract state size by 70.6% (97.7% reduction in ValVar entries) across 348/348 passing tests, with no changes to the default (dense) mode behavior.
Changes from #1803
ArgValVar,RetValPN,VarArgValPN,GlobalValVar, andGepValVar, eliminating most "orphan" ValVars.orphanVarDefMapentirely: the only remaining def-site mismatch (call-result ValVars whosegetICFGNode()returnsCallICFGNodebut whose value is written atRetICFGNode) is now resolved inline viaCallICFGNode::getRetICFGNode()— no pre-built map needed.PreAnalysis.hchanges: PreAnalysis no longer has any semi-sparse-specific code.set_valueexternal API calls with direct assignments /svf_assertrange checks (PR to Test-Suite). All 348 tests now pass.Key Design
Hybrid merge strategy in
mergeStatesFromPredecessorsIntraCFGEdgejoinAddrOnly_addrToAbsVal(ObjVars). ValVars are pulled on demand from their SSA definition sites.CallCFGEdge/RetCFGEdgejoinWithf(2); f(3)→ formal param must be[2,3]).On-demand ValVar lookup (
getAbstractValue)abstractTrace[defNode]wheredefNode = var->getICFGNode().CallICFGNode::getRetICFGNode()to find the value written by RetPE.Sparse state construction (
buildSparseState)collectNeededVarIdsidentifies which ValVars are needed as operands, then pulls them from def-sites.pullBranchConditionVarsensures branch feasibility checks have the condition variables and load-chain pointers available.Files Changed
Options.h/cpp-semi-sparseCLI optionAbstractState.h/cppjoinAddrOnlymethodAbstractInterpretation.h/cppgetAbstractValue,buildSparseState,collectNeededVarIds,pullBranchConditionVars, hybrid mergePreAnalysis.cpp#includeforCallICFGNode(no semi-sparse-specific logic)Test Results
State Size Reduction
Measured across 281 comparable test invocations:
Per-test: min 38.5%, median 79.4%, mean 78.2%. All comparable tests show reduction.
Test Plan
ctest -R ae: 348/348 pass in default (dense) modectest -R semi_sparse: 347/347 pass in semi-sparse modeset_valuetests fixed in Test-Suite PR