py-libp2p CID Status and py-cid Integration Opportunities

[acul71]

py-libp2p CID Status and py-cid Integration Opportunities

Executive Summary

This document analyzes the current CID (Content Identifier) implementation in py-libp2p and identifies opportunities to leverage the full-featured py-cid library to improve code quality, maintainability, and standards compliance.

Current State: py-libp2p uses a simplified, custom CID implementation in libp2p/bitswap/cid.py that handles basic CID operations but lacks many features available in the standard py-cid library.

Recommendation: Migrate to py-cid to gain proper CID encoding/decoding, multibase support, builder patterns, and other advanced features that would significantly improve the codebase.

---

Table of Contents

1. Current CID Implementation in py-libp2p
2. py-cid Features Overview
3. Integration Opportunities
4. Migration Strategy
5. Benefits of Migration

---

Current CID Implementation

Location: libp2p/bitswap/cid.py

The current implementation provides basic CID functionality:

Features:

• CIDv0 and CIDv1 computation (simplified)
• CID verification
• CID prefix extraction (for Bitswap v1.1.0+)
• CID reconstruction from prefix and data
• Version parsing
• Hex string conversion

Limitations:

• Only supports SHA-256 hashing
• Limited codec support (only CODEC_DAG_PB = 0x70 and CODEC_RAW = 0x55)
• No proper base58 encoding for CIDv0
• No multibase encoding support for CIDv1
• Returns raw bytes instead of proper CID objects
• No support for CID string parsing
• Manual hex string conversion throughout codebase
• No CID version conversion capabilities
• No JSON/IPLD format support

Current Usage Pattern:

# Current: Returns raw bytes
cid_bytes = compute_cid_v1(data, codec=CODEC_RAW)
cid_hex = cid_bytes.hex()  # Manual hex conversion

# Used as dictionary keys
self._blocks: dict[bytes, bytes] = {}  # CID bytes -> block data

---

py-cid Features Overview

Core Features

1. Full CIDv0 and CIDv1 Support

   • Proper base58 encoding for CIDv0
   • Multibase encoding for CIDv1 (base32, base64, etc.)
   • Automatic version detection

2. Builder Pattern

   • V0Builder() and V1Builder(codec, mh_type) for fluent CID creation
   • Chainable codec changes

3. Prefix Operations

   • Prefix class for CID metadata (version, codec, multihash type/length)
   • Serialization/deserialization
   • Create CIDs from data using prefix

4. CIDSet

   • Specialized set for managing unique CIDs
   • Hashable CID objects
   • Efficient membership testing

5. JSON/IPLD Format Support

   • to_json_dict() / from_json_dict() for IPLD JSON format
   • CIDJSONEncoder for JSON serialization

6. Advanced Parsing

   • /ipfs/ path parsing (extracts CID from URLs/paths)
   • from_reader() for stream parsing
   • from_bytes_strict() with trailing bytes validation
   • extract_encoding() to detect multibase encoding

7. Version Conversion

   • cidv0.to_v1() and cidv1.to_v0() (when codec allows)

8. Utility Methods

   • to_bytes(), to_text(), from_text() for marshaling
   • key_string() for use as dictionary keys
   • loggable() for logging
   • defined() to check CID validity

---

Integration Opportunities

1. Replace Custom CID Implementation

Location: libp2p/bitswap/cid.py

Current Issues:

• Simplified implementation that doesn't follow CID spec fully
• Comment says "use a proper CID library like py-cid or multiformats"
• Limited functionality compared to py-cid

Improvement:
Replace the custom implementation with py-cid wrapper functions that maintain backward compatibility while providing full functionality.

Example Migration:

# Before (current)
from libp2p.bitswap.cid import compute_cid_v1, verify_cid

cid_bytes = compute_cid_v1(data, codec=CODEC_RAW)
is_valid = verify_cid(cid_bytes, data)

# After (with py-cid)
from cid import V1Builder, make_cid

builder = V1Builder(codec="raw", mh_type="sha2-256")
cid = builder.sum(data)
cid_bytes = cid.buffer  # Get raw bytes if needed
is_valid = cid.multihash == compute_multihash(data)

Files Affected:

• libp2p/bitswap/cid.py - Complete rewrite
• libp2p/bitswap/__init__.py - Update exports
• All files importing from bitswap.cid

---

2. Improve CID String Representation

Location: Throughout libp2p/bitswap/

Current Issue:
CIDs are represented as hex strings using .hex() method, which is not standard CID format.

Current Pattern:

# Found in: client.py, dag.py, examples/bitswap/bitswap.py
logger.info(f"Root CID: {root_cid.hex()}")
logger.info(f"Block {cid.hex()[:16]}...")

Improvement:
Use proper CID string encoding (base58 for v0, multibase for v1).

Example:

# Before
cid_hex = cid_bytes.hex()
logger.info(f"CID: {cid_hex}")

# After
from cid import make_cid

cid = make_cid(cid_bytes)
logger.info(f"CID: {cid}")  # Properly encoded string
# Or for logging
logger.info(f"CID: {cid.loggable()['cid']}")

Files Affected:

• libp2p/bitswap/client.py - ~20 occurrences
• libp2p/bitswap/dag.py - ~15 occurrences
• libp2p/bitswap/dag_pb.py - ~2 occurrences
• examples/bitswap/bitswap.py - ~10 occurrences

---

3. Enhance Block Store with CID Objects

Location: libp2p/bitswap/block_store.py

Current Issue:
Block store uses raw bytes as keys, which makes CID operations difficult.

Current Implementation:

class MemoryBlockStore(BlockStore):
    def __init__(self):
        self._blocks: dict[bytes, bytes] = {}  # CID bytes -> block data

Improvement:
Use CID objects as keys (they're hashable) or add CID-aware methods.

Example:

from cid import CIDv0, CIDv1, make_cid

class MemoryBlockStore(BlockStore):
    def __init__(self):
        self._blocks: dict[CIDv0 | CIDv1, bytes] = {}
    
    async def get_block(self, cid: bytes | str | CIDv0 | CIDv1) -> bytes | None:
        # Accept multiple input types
        if isinstance(cid, (bytes, str)):
            cid = make_cid(cid)
        return self._blocks.get(cid)
    
    async def put_block(self, cid: bytes | str | CIDv0 | CIDv1, data: bytes) -> None:
        if isinstance(cid, (bytes, str)):
            cid = make_cid(cid)
        self._blocks[cid] = data

Benefits:

• Type safety
• Automatic CID validation
• Better error messages
• Support for CID strings in addition to bytes

---

4. Add CIDSet for Wantlist Management

Location: libp2p/bitswap/client.py

Current Issue:
Wantlist uses dictionary with bytes keys, which could benefit from CIDSet for deduplication and efficient operations.

Current Implementation:

self._wantlist: dict[bytes, dict[str, Any]] = {}  # CID -> {priority, want_type, send_dont_have}

Improvement:
Use CIDSet for tracking unique CIDs, with separate metadata storage.

Example:

from cid import CIDSet

class BitswapClient:
    def __init__(self, ...):
        self._wantlist_cids = CIDSet()  # Track unique CIDs
        self._wantlist_metadata: dict[CIDv0 | CIDv1, dict[str, Any]] = {}
    
    def add_to_wantlist(self, cid: bytes, priority: int = DEFAULT_PRIORITY):
        cid_obj = make_cid(cid)
        if cid_obj not in self._wantlist_cids:
            self._wantlist_cids.add(cid_obj)
            self._wantlist_metadata[cid_obj] = {
                'priority': priority,
                'want_type': 1,
                'send_dont_have': False
            }

Benefits:

• Efficient duplicate detection
• Cleaner API
• Better type safety

---

5. Implement Proper CID Parsing from Strings

Location: examples/bitswap/bitswap.py

Current Issue:
Example code manually parses hex strings, which doesn't support standard CID string formats.

Current Pattern:

root_cid = bytes.fromhex(root_cid_hex)  # Only supports hex

Improvement:
Use py-cid's from_string() which supports:

• Base58-encoded CIDv0 (e.g., Qm...)
• Multibase-encoded CIDv1 (e.g., bafy...)
• /ipfs/ paths (e.g., /ipfs/Qm...)
• URLs (e.g., https://ipfs.io/ipfs/Qm...)

Example:

from cid import from_string

# Before
root_cid = bytes.fromhex(root_cid_hex)

# After
cid = from_string(root_cid_hex)  # Supports all formats
root_cid = cid.buffer  # Get bytes if needed
# Or use CID object directly

Files Affected:

• examples/bitswap/bitswap.py - Command-line argument parsing
• Any code that accepts CID strings from users

---

6. Add Builder Pattern for CID Creation

Location: libp2p/bitswap/dag.py, libp2p/bitswap/cid.py

Current Issue:
CID creation is done with low-level functions, making it error-prone.

Current Pattern:

chunk_cid = compute_cid_v1(chunk_data, codec=CODEC_RAW)

Improvement:
Use builder pattern for clearer, more maintainable code.

Example:

from cid import V1Builder

# Before
chunk_cid = compute_cid_v1(chunk_data, codec=CODEC_RAW)

# After
builder = V1Builder(codec="raw", mh_type="sha2-256")
chunk_cid = builder.sum(chunk_data)
# Or chain codec changes
dag_pb_builder = builder.with_codec("dag-pb")

Benefits:

• More readable
• Type-safe codec names
• Easier to extend with different hash types

---

7. Add Prefix Support for Bitswap v1.1.0+

Location: libp2p/bitswap/cid.py

Current Issue:
Custom prefix extraction doesn't leverage py-cid's Prefix class.

Current Implementation:

def get_cid_prefix(cid: bytes) -> bytes:
    # Manual prefix extraction
    if cid[0] == CID_V1:
        return cid[:4]  # Simplified

Improvement:
Use py-cid's Prefix class for proper prefix operations.

Example:

from cid import make_cid

def get_cid_prefix(cid: bytes) -> bytes:
    cid_obj = make_cid(cid)
    prefix = cid_obj.prefix()
    return prefix.to_bytes()  # Proper serialization

Benefits:

• Standards-compliant prefix format
• Proper serialization
• Can reconstruct CID from prefix + data

---

8. Add JSON/IPLD Format Support

Location: API endpoints, serialization code

Current Issue:
No support for IPLD JSON format ({"/": "<cid-string>"}).

Improvement:
Add JSON serialization support for APIs and data exchange.

Example:

from cid import CIDJSONEncoder, make_cid
import json

# Serialize CID to IPLD JSON
cid = make_cid(cid_bytes)
json_str = json.dumps(cid, cls=CIDJSONEncoder)
# Result: '{"\\/": "Qm..."}'

# Deserialize from IPLD JSON
data = json.loads(json_str)
cid = CIDv0.from_json_dict(data)

Use Cases:

• REST API responses
• Configuration files
• Data exchange formats

---

9. Improve CID Validation and Error Handling

Location: Throughout bitswap module

Current Issue:
Limited validation, unclear error messages.

Improvement:
Use py-cid's validation and error handling.

Example:

from cid import make_cid, is_cid

# Validate before processing
if not is_cid(cid_string):
    raise InvalidCIDError(f"Invalid CID format: {cid_string}")

# Parse with proper error handling
try:
    cid = make_cid(cid_bytes)
except ValueError as e:
    raise InvalidCIDError(f"Failed to parse CID: {e}") from e

Benefits:

• Better error messages
• Consistent validation
• Standards compliance

---

10. Add Stream Parsing Support

Location: Protocol message parsing

Current Issue:
No support for parsing CIDs from streams.

Improvement:
Use from_reader() for efficient stream parsing.

Example:

from cid import from_reader

# Parse CID from stream
bytes_read, cid = from_reader(stream)
# Know exactly how many bytes were consumed

Use Cases:

• Protocol buffer parsing
• Network message parsing
• File format parsing

---

Migration Strategy

Phase 1: Add py-cid as Dependency

1. Add py-cid to pyproject.toml:

   dependencies = [
       ...
       "py-cid>=0.4.0",
   ]

2. Update imports to include py-cid alongside current implementation

Phase 2: Create Compatibility Layer

1. Create wrapper functions in libp2p/bitswap/cid.py that:

   • Use py-cid internally
   • Maintain current API for backward compatibility
   • Add new functions for enhanced features

2. Example wrapper:

   from cid import make_cid, V1Builder as PyCIDV1Builder
   
   # Backward compatible
   def compute_cid_v1(data: bytes, codec: int = CODEC_RAW) -> bytes:
       codec_name = codec_to_name(codec)  # Map int to string
       builder = PyCIDV1Builder(codec=codec_name, mh_type="sha2-256")
       cid = builder.sum(data)
       return cid.buffer
   
   # New enhanced API
   def compute_cid_v1_obj(data: bytes, codec: str = "raw") -> CIDv1:
       builder = PyCIDV1Builder(codec=codec, mh_type="sha2-256")
       return builder.sum(data)

Phase 3: Gradual Migration

1. Start with new code: Use py-cid directly in new features
2. Migrate high-impact areas:
   • Block store (improves type safety)
   • Examples (improves user experience)
   • Logging (improves readability)
3. Update tests: Ensure compatibility
4. Deprecate old functions: Mark for removal in future version

Phase 4: Complete Migration

1. Remove custom CID implementation
2. Update all internal code to use py-cid
3. Update documentation
4. Release breaking change version

---

Benefits of Migration

1. Standards Compliance

• Full CID specification support
• Proper encoding/decoding
• Interoperability with other IPFS/libp2p implementations

2. Code Quality

• Reduced maintenance burden (no custom implementation to maintain)
• Better error handling
• Type safety with CID objects

3. Developer Experience

• Better API (builder pattern, fluent interface)
• Support for standard CID string formats
• Better error messages

4. Features

• Support for multiple hash algorithms (not just SHA-256)
• Support for all codecs (not just DAG-PB and RAW)
• JSON/IPLD format support
• Path parsing (/ipfs/...)

5. Performance

• Optimized implementation in py-cid
• Efficient CIDSet for deduplication
• Proper caching of encoded strings

6. Future-Proofing

• py-cid is actively maintained
• New CID features automatically available
• Community support and testing

---

Specific Code Locations for Improvement

High Priority

1. libp2p/bitswap/cid.py (236 lines)

   • Complete rewrite using py-cid
   • Maintain backward compatibility initially

2. libp2p/bitswap/block_store.py (109 lines)

   • Use CID objects as keys
   • Add CID string support

3. libp2p/bitswap/client.py (859 lines)

   • Replace hex string logging with proper CID strings
   • Use CIDSet for wantlist management
   • Improve CID validation

4. libp2p/bitswap/dag.py (585 lines)

   • Use builder pattern for CID creation
   • Replace hex string logging
   • Use CID objects throughout

5. examples/bitswap/bitswap.py (400+ lines)

   • Support standard CID string formats
   • Use CID objects for better UX

Medium Priority

6. libp2p/bitswap/dag_pb.py (269 lines)

   • Use CID objects in Link dataclass
   • Improve CID string representation

7. libp2p/bitswap/__init__.py (64 lines)

   • Update exports to include py-cid functions

8. Tests (tests/core/bitswap/test_cid.py)

   • Update tests to use py-cid
   • Add tests for new features

Low Priority

9. Documentation

   • Update examples to show py-cid usage
   • Document migration path

10. Type Hints

    • Add CID type hints throughout
    • Improve type safety

---

Example: Complete Migration of One Function

Before (Current Implementation)

# libp2p/bitswap/cid.py
def compute_cid_v1(data: bytes, codec: int = CODEC_RAW) -> bytes:
    """Compute a CIDv1 for data (simplified version)."""
    digest = hashlib.sha256(data).digest()
    multihash = bytes([HASH_SHA256, len(digest)]) + digest
    cid = bytes([CID_V1, codec]) + multihash
    return cid

# Usage
cid_bytes = compute_cid_v1(data, codec=CODEC_RAW)
logger.info(f"CID: {cid_bytes.hex()}")

After (With py-cid)

# libp2p/bitswap/cid.py
from cid import V1Builder, make_cid
from typing import Union

# Backward compatible function
def compute_cid_v1(data: bytes, codec: int = CODEC_RAW) -> bytes:
    """Compute a CIDv1 for data (backward compatible)."""
    codec_name = _codec_int_to_name(codec)
    builder = V1Builder(codec=codec_name, mh_type="sha2-256")
    cid = builder.sum(data)
    return cid.buffer

# New enhanced function
def compute_cid_v1_obj(
    data: bytes, 
    codec: str = "raw",
    mh_type: str = "sha2-256"
) -> CIDv1:
    """Compute a CIDv1 object with full py-cid support."""
    builder = V1Builder(codec=codec, mh_type=mh_type)
    return builder.sum(data)

# Usage (backward compatible)
cid_bytes = compute_cid_v1(data, codec=CODEC_RAW)
logger.info(f"CID: {cid_bytes.hex()}")

# Usage (enhanced)
cid = compute_cid_v1_obj(data, codec="raw")
logger.info(f"CID: {cid}")  # Properly encoded string
logger.info(f"CID: {cid.loggable()['cid']}")  # For structured logging

---

Conclusion

Migrating py-libp2p to use py-cid would provide significant benefits:

1. Standards Compliance: Full CID specification support
2. Better Developer Experience: Cleaner API, better error messages
3. More Features: JSON support, path parsing, multiple hash types
4. Reduced Maintenance: No need to maintain custom CID implementation
5. Future-Proofing: Active library with community support

The migration can be done gradually with a compatibility layer, minimizing breaking changes while providing immediate benefits for new code.

Recommended Next Steps:

1. Add py-cid as dependency
2. Create compatibility layer in libp2p/bitswap/cid.py
3. Migrate examples and new code first
4. Gradually migrate internal code
5. Remove custom implementation in future major version

---

References

• py-cid repository: https://github.com/ipld/py-cid
• py-cid documentation: https://py-cid.readthedocs.io
• CID specification: https://github.com/ipld/cid
• Current py-libp2p CID implementation: libp2p/bitswap/cid.py

[gerceboss]

I can take this up after #1194 is merged , as some modifications already done there @acul71 @yashksaini-coder

[ItshMoh]

Phase Description for Original Bitswap py-cid Migration Issue

This document describes the full 4-phase plan for completing the original Bitswap py-cid migration work.

Phase 1: Dependency + Compatibility Layer

What we will do

• Add py-cid>=0.5.0 as a project dependency.
• Introduce shared CID compatibility helpers in libp2p/bitswap/cid.py:
  • parse_cid(...)
  • cid_to_bytes(...)
  • cid_to_text(...)
• Keep existing byte-based APIs unchanged so current callers continue to work.

Phase 2: py-cid Core Refactor in bitswap/cid.py

What we will do

• Re-implement CID generation with py-cid builders:
  • compute_cid_v0() via V0Builder
  • compute_cid_v1() via V1Builder
• Re-implement verification/parsing/prefix operations with CID objects and Prefix.
• Keep byte-returning APIs for compatibility.
• Add object-returning APIs for new call sites:
  • compute_cid_v0_obj(...)
  • compute_cid_v1_obj(...)
  • compute_cid_obj(...)
• Remove migration-only legacy helpers that are redundant in the new architecture.

Phase 3: Module-Wide Integration (Non-Breaking)

What we will do

• Apply shared CID helpers across Bitswap modules:
  • libp2p/bitswap/client.py
  • libp2p/bitswap/dag.py
  • libp2p/bitswap/dag_pb.py
  • libp2p/bitswap/block_store.py
  • libp2p/bitswap/messages.py
  • examples/bitswap/bitswap.py
• Replace scattered .hex() CID display patterns with canonical CID text where appropriate.
• Normalize CID inputs at boundaries (bytes/string/CID object) while keeping wire/storage compatibility.
• Update tests for mixed input types and integration behavior.

Phase 4: Full Adoption + Cleanup (Breaking-Change Window)

What we will do

• Shift internal CID identity to object-first where beneficial (store/wantlist internals).
• Remove remaining deprecated compatibility shims that are no longer needed.
• Finalize public API recommendations (preferred object APIs vs legacy byte APIs).
• Update docs/examples and release notes with final migration guidance.

Summary

• Phase 1 + 2 established and complete CID core migration safely.
• Phase 3 integrates migration across operational Bitswap modules.
• Phase 4 finalizes cleanup and long-term architecture.

[gerceboss]

@ItshMoh let us connect on discord about this
Migration

[acul71]

Current status:

• Phase 1+2 are done: PR feat: Bitswap py-cid Migration Phase 1 + Phase 2 #1235 is merged, issue Bitswap py-cid Migration - Phase 1 + Phase 2 #1234 is closed. The py-cid dependency, shared helpers (parse_cid, cid_to_bytes, cid_to_text), and the migration of CID creation/verification/parsing to py-cid (V0Builder/V1Builder, CID objects, Prefix) are in main. Byte-returning APIs are preserved; object-returning APIs were added; migration-only helpers from add multicodec #1194 were removed.
• Next steps: Phase 3 (module-wide integration: client, dag, block_store, messages, examples) and Phase 4 (full adoption and cleanup) are the remaining work. See issue Migrate Bitswap CID handling to py-cid #1181 for tracking.

Contributions for Phase 3 or Phase 4 are welcome — please coordinate on issue #1181 or this discussion to avoid overlapping PRs.