Skip to content

Type-Aware SCALE-to-JSON Serializer #41

@TarikGul

Description

@TarikGul

Background

Currently, the API uses heuristics to identify which SCALE-encoded values should be transformed during JSON
serialization:

Extrinsics: Field-name pattern matching (~20 patterns)

// Generic approach: "dest", "who", "target", "beneficiary", etc.
if is_account_field(field.name()) {
    // Convert AccountId32 → SS58 address
}

Events: Position-based mappings (~100+ event-specific mappings)
match (pallet, event) {
    ("balances", "Transfer") => vec![0, 1],  // positions 0 and 1 are accounts
    ("treasury", "Awarded") => vec![1],       // position 1 is beneficiary
    // ... 100+ more mappings
}

Problem

These heuristics have several limitations:

  1. Maintenance burden: Every new event type requires manual position mapping updates
  2. False positives: Field names like "data" or "hash" might accidentally match patterns
  3. False negatives: Unusual field names for accounts might be missed
  4. Type ambiguity: Can't distinguish AccountId32 from H256 (both 32 bytes)
  5. Limited scope: Only handles account transformations, but other types need similar treatment (e.g., Balance →
    decimal formatting)

Desired Solution

A type-aware serializer that uses Substrate's type metadata to make transformation decisions:

// Pseudocode - what we want:
for field in value.fields() {
let type_id = field.type_id();

  match registry.resolve(type_id) {
      TypeDef::Composite(AccountId32) => serialize_as_ss58(field),
      TypeDef::Composite(Balance) => serialize_as_decimal(field),
      TypeDef::Composite(H256) => serialize_as_hex(field),
      // ... other type transformations
  }

}

This would work for any type transformation, not just accounts:

  • AccountId32 → SS58 address
  • Balance → decimal string with proper precision
  • Hash → 0x-prefixed hex
  • Custom types → domain-specific formatting

Blocker: Metadata Version Incompatibility

This approach is blocked until subxt-historic v0.50 ships.

Why: Historic blocks use different metadata versions with incompatible type identification systems:

  • Legacy metadata (V8-V13): Uses LookupName for type IDs
  • Modern metadata (V14+): Uses u32 for type IDs

There's no uniform way to resolve types across all metadata versions until subxt-historic v0.50, which will
translate all historic metadata to a modern, uniform format.

Current Status

The heuristic approach (field patterns + position mappings) is the pragmatic solution given these constraints. It
works well for common cases and can be incrementally improved by:

  1. Adding more field name patterns as we discover them
  2. Expanding event position mappings for new pallets
  3. Documenting edge cases and limitations

Future Work

Once subxt-historic v0.50 ships with metadata translation:

  1. Design a type-aware serialization system using PortableRegistry
  2. Create transformation rules based on type definitions (not field names/positions)
  3. Support arbitrary type transformations (not just accounts)
  4. Remove the hardcoded heuristics
  5. Reduce maintenance burden and improve correctness

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type
No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions