Skip to content

IDA Pro: Hex-Rays decompiler plugin via microcode API #11

@kyle-elliott-tob

Description

@kyle-elliott-tob

Summary

Build a native C++ Hex-Rays decompiler plugin that simplifies MBA expressions during decompilation using CoBRA's core library. This provides CoBRA's full simplification pipeline (semilinear, MixedRewrite, decomposition, full-width verification) directly in IDA's microcode optimization pass.

Architecture

Model after gooMBA's integration pattern (HexRaysSA/goomba):

lib/ida/
  CobraIDA.cpp          — Plugin entry, hexrays callback, action registration
  MicrocodeDetector.cpp — MBA detection via minsn_t tree walking
  MicrocodeDetector.h
  MicrocodeReconstructor.cpp — Simplified Expr → minsn_t reconstruction
  MicrocodeReconstructor.h

Hook Point

  • Register hexrays_callback for hxe_glbopt event (global optimization phase)
  • Walk top-level microcode instructions via mba->for_all_topinsns()
  • For each instruction, recursively attempt MBA simplification
  • Return MERR_LOOP to restart optimization if any simplifications were made

Microcode → Expr Bridge

Map Hex-Rays microcode operations to CoBRA Expr kinds:

Hex-Rays op CoBRA Expr
m_add Expr::Add
m_sub Expr::Add + Expr::Negate
m_mul Expr::Mul
m_and Expr::BitwiseAnd
m_or Expr::BitwiseOr
m_xor Expr::BitwiseXor
m_bnot Expr::BitwiseNot
m_neg Expr::Negate
m_shr Expr::LogicalShr (constant shift)

Operand types (mop_t): registers → variables, mop_n (numbers) → constants, sub-instructions → recursive descent.

Evaluator

Build evaluator lambda from microcode — assign values to leaf mop_t operands and evaluate the minsn_t tree (similar to goomba's mcode_emu.hpp approach). This enables CoBRA's full-width verification.

Detection Heuristic

Same approach as goomba's is_mba(): count boolean ops (m_and, m_or, m_xor, m_bnot) and arithmetic ops (m_add, m_sub, m_mul). Require both categories present and minimum tree size.

Verification

CoBRA's built-in full-width verification (evaluator + spot checks) replaces goomba's Z3 equivalence proofs. Significantly faster — no Z3 dependency needed for the plugin itself.

Cost Gate

Use ComputeCost/IsBetter to ensure simplified form is actually simpler, same as the LLVM pass.

Build System

  • COBRA_BUILD_IDA_PLUGIN CMake option
  • Requires IDA SDK path (IDA_SDK_DIR) — not shipped, user must have IDA license
  • Links against cobra-core (static)
  • Produces cobra_ida.so / cobra_ida.dylib / cobra_ida.dll plugin

Task List

  • CMake infrastructure for IDA plugin (SDK detection, plugin target)
  • Plugin entry point: hexrays callback, action registration, popup menu
  • Microcode → Expr bridge (BuildExprFromMinsn)
  • Microcode evaluator (EvaluateMinsn)
  • MBA detection heuristic for microcode
  • Expr → microcode reconstruction (ReconstructMinsn)
  • Cost gate integration
  • Variable remapping for aux var elimination
  • Testing strategy (IDB-based or standalone microcode tests)

References

  • gooMBA — Hex-Rays' official MBA simplifier
  • gooMBA hooks hxe_glbopt, walks via for_all_topinsns, uses minsn_visitor_t
  • gooMBA returns MERR_LOOP after successful simplification to restart decompiler optimization

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions