Skip to content

Incorporate codeanalyzer-python v1.0.0 #272

Description

@rahlk

A new codeanalyzer-python release (v1.0.0) is published.
Update the codeanalyzer-python pin in this repo and adapt the
integration to the changes below (see PyCodeanalyzer._run_analyzer).

[1.0.0] - 2026-07-14

Added

  • Four analysis levels (-a 1|2|3|4): L1 is symbol table and Jedi call graph; L2 adds
    PyCG call-graph edges (--pycg-shard scales to large apps); L3 adds intraprocedural dataflow
    (CFG/CDG/DDG, syntactic, prov:["ssa"]); L4 adds interprocedural SDG with synthetic
    parameter-in/out and summary edges, alias-aware DDG (prov:["points-to"]). Dataflow graphs are
    built in-process from the stdlib ast (exceptional CFGs with synthetic ENTRY/EXIT and
    exception/yield/await edges; Ferrante–Ottenstein–Warren control dependence; reaching-definitions
    data dependence over k-limited access paths; a Horwitz–Reps–Binkley SDG with parameter and
    bottom-up summary edges over the Tarjan SCC condensation). They are emitted inline on each
    callable as body/cfg/cdg/ddg under the v2 application tree — not a separate section.
  • --graphs and --graph-field-depth flags (level 3+): --graphs cfg,dfg,pdg,sdg scopes the
    emitted program-graph sections (strict validation — unknown values or use below -a 3 exit
    non-zero; sdg requires -a 4); --graph-field-depth sets the access-path k-limit (default 3),
    the bound that guarantees the interprocedural fixpoint terminates.
  • Context-sensitive backward slicing as an SDG query (codeanalyzer.dataflow.slicing,
    HRB two-phase traversal). Taint is deliberately left to the CLDK SDK — post-SDG it is
    language-independent labeled reachability.
  • Scalpel points-to oracle for L4 alias analysis (optional python-scalpel dependency:
    pip install "codeanalyzer-python[scalpel]"). Automatic type-based fallback when absent.
  • Neo4j schema v2.0.0 — the property graph is re-keyed onto canonical can:// node IDs.
    New PY_PARAM_IN, PY_PARAM_OUT, PY_SUMMARY edges. PY_DDG carries prov property
    to distinguish syntactic (ssa) from semantic (points-to) dependence.
  • Neo4j CPG overlay at levels 3–4 (part of schema v2.0.0): PyCFGNode nodes plus the
    PY_HAS_CFG_NODE/PY_CFG_NEXT/PY_CDG/PY_DDG edges project each callable's control-flow and
    dependence graph, PY_-namespaced like every other row family so a multi-language database never
    mingles analyzers' dependence edges.

Changed

  • BREAKING: canonical schema v2 (analysis.json). Structure is now a single additive CPG
    tree under an Analysis envelope (schema_version, language, max_level,
    analyzer{name,version}, k_limit at L3+, application). The old flat symbol_table +
    call_graph + separate program_graphs is replaced: consumers must read
    application.symbol_table, application.call_graph (now nested under application), and
    inline body/cfg/cdg/ddg on each callable. Nodes carry canonical can:// identifiers.
    Module source is stored once with byte-offset spans; per-node code is dropped. Upgrade:
    pin codeanalyzer-python==1.0.0 and update any code that read top-level
    symbol_table/call_graph to go through application.
  • BREAKING: keystone conformance sweep (Extend python-sdk to surface Java EE CRUD operations and queries #98, part of the schema-v2 change above; key-for-key
    parity with the canonical CPG keystone shared by every analyzer):
    • Containment uses the shared canonical names: modules nest types/functions, classes nest
      callables/types, callables nest callables/types. The historical
      classes/methods/inner_classes/inner_callables keys are gone.
    • call_graph edges are {src, dst, prov, weight} — the list name is the edge type, so the
      type: "CALL_DEP" field and the old source/target/provenance keys are gone.
    • No dangling edge endpoints: imported/builtin call targets are homed as
      can://<lang>/<app>/@external/<module>/<name> entries in application.external_symbols
      (keyed by that id, kind:"external"); first-party endpoints always resolve to their tree id.
    • Envelope carries analyzer: {name, version}; k_limit is emitted only at L3+ (it is a
      dataflow bound and was meaningless at L1/L2).
    • Neo4j projection follows: :PyExternal ghosts merge on id (the @external id, not the
      dotted signature) and PY_CALLS carries prov (was provenance).

Fixed

  • Neo4j edge identity no longer collapses distinct edges between the same endpoint pair:
    PY_DDG merges per (var, prov) and PY_CFG_NEXT per kind via an internal _k
    relationship discriminant. Previously a plain endpoint-pair MERGE silently dropped
    per-variable data dependences (and could collapse a conditional's true/false CFG pair),
    so a live Bolt push materialized fewer relationships than the projection produced.
  • Full-run orphan prune now removes the L3/L4 CPG overlay of a vanished module: the
    containment closure used for pruning was missing PY_HAS_CFG_NODE, stranding every
    PyCFGNode (and its dependence edges) in the database after its module was deleted.
  • Emitted analysis.json round-trips through the Analysis model again.
    PyVariableDeclaration.type was required-but-nullable; exclude_none emission drops the
    key when Jedi cannot infer a variable's type, so re-validating the analyzer's own output
    failed with "Field required" (43k errors on an Odoo-scale run). The field now defaults to
    None; a round-trip gate rides test_v2_keystone.py.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions