feat: add llama-index-callbacks-hdp and hdp-llamaindex packages

Three-layer integration providing cryptographic authorization provenance
for LlamaIndex agents and RAG pipelines:

- Layer 1: HdpInstrumentationHandler — modern dispatcher integration
  (LlamaIndex >=0.10.20), hooks QueryStartEvent, AgentToolCallEvent,
  LLMChatStartEvent, QueryEndEvent via root dispatcher
- Layer 2: HdpCallbackHandler — legacy CallbackManager integration,
  hooks start_trace, end_trace, on_event_start/end for all FUNCTION_CALL
  and LLM events
- Layer 3: HdpNodePostprocessor — inline retrieval hop recording and
  data classification enforcement in the RAG pipeline

Shared ContextVar session state isolates tokens across concurrent async
tasks. All crypto primitives bundled inline (Ed25519 + RFC 8785 JCS),
consistent with all other HDP Python packages.

hdp-llamaindex is a thin metapackage re-exporting from the llama_index
namespace for users who discover HDP first.

43 tests passing across session isolation, chain verification, callback
handler, and postprocessor.

Author: asiridalugoda

packages/hdp-llamaindex/README.md

@@ -0,0 +1,11 @@
+# hdp-llamaindex
+
+Metapackage for users who discover HDP first.
+
+```bash
+pip install hdp-llamaindex
+```
+
+This installs `llama-index-callbacks-hdp` and re-exports all classes from the `hdp_llamaindex` namespace.
+
+For full documentation see [llama-index-callbacks-hdp](../llama-index-callbacks-hdp/README.md).

packages/hdp-llamaindex/pyproject.toml

@@ -0,0 +1,21 @@
+[build-system]
+requires = ["hatchling"]
+build-backend = "hatchling.build"
+
+[project]
+name = "hdp-llamaindex"
+version = "0.1.0"
+description = "HDP (Human Delegation Provenance) integration for LlamaIndex — metapackage"
+readme = "README.md"
+license = { text = "CC-BY-4.0" }
+requires-python = ">=3.10"
+dependencies = [
+    "llama-index-callbacks-hdp>=0.1.0",
+]
+
+[project.urls]
+Homepage = "https://github.com/Helixar-AI/HDP"
+Repository = "https://github.com/Helixar-AI/HDP"
+
+[tool.hatch.build.targets.wheel]
+packages = ["src/hdp_llamaindex"]

packages/hdp-llamaindex/src/hdp_llamaindex/__init__.py

@@ -0,0 +1,43 @@
+"""hdp-llamaindex — convenience re-export of the HDP LlamaIndex integration.
+
+Install via `pip install hdp-llamaindex` if you discover HDP first.
+All classes are importable from here or from `llama_index.callbacks.hdp`.
+"""
+
+from llama_index.callbacks.hdp import (
+    DataClassification,
+    HdpCallbackHandler,
+    HdpInstrumentationHandler,
+    HdpNodePostprocessor,
+    HdpPrincipal,
+    HdpScope,
+    HdpToken,
+    HDPScopeViolationError,
+    HopRecord,
+    HopVerification,
+    ScopePolicy,
+    VerificationResult,
+    clear_token,
+    get_token,
+    set_token,
+    verify_chain,
+)
+
+__all__ = [
+    "DataClassification",
+    "HdpCallbackHandler",
+    "HdpInstrumentationHandler",
+    "HdpNodePostprocessor",
+    "HdpPrincipal",
+    "HdpScope",
+    "HdpToken",
+    "HDPScopeViolationError",
+    "HopRecord",
+    "HopVerification",
+    "ScopePolicy",
+    "VerificationResult",
+    "clear_token",
+    "get_token",
+    "set_token",
+    "verify_chain",
+]

packages/llama-index-callbacks-hdp/README.md

@@ -0,0 +1,85 @@
+# llama-index-callbacks-hdp
+
+HDP (Human Delegation Provenance) integration for LlamaIndex — cryptographic authorization provenance for agents and RAG pipelines.
+
+HDP answers the question that observability tools like Arize Phoenix and Langfuse cannot: **who authorized this agent run, under what scope, and can you prove it offline?**
+
+Every tool call, retrieval step, and LLM invocation is recorded in a tamper-evident, cryptographically signed delegation chain. The chain is fully verifiable offline — no network calls, no central registry.
+
+## Installation
+
+```bash
+pip install llama-index-callbacks-hdp
+```
+
+## Usage
+
+### Option 1 — Modern instrumentation dispatcher (LlamaIndex ≥0.10.20)
+
+```python
+from llama_index.callbacks.hdp import HdpInstrumentationHandler, HdpPrincipal, ScopePolicy
+
+HdpInstrumentationHandler.init(
+    signing_key=ed25519_private_key_bytes,
+    principal=HdpPrincipal(id="alice@corp.com", id_type="email"),
+    scope=ScopePolicy(
+        intent="Research pipeline",
+        authorized_tools=["web_search", "retriever"],
+        max_hops=10,
+    ),
+    on_token_ready=lambda token: print(token["header"]["token_id"]),
+)
+```
+
+### Option 2 — Legacy CallbackManager
+
+```python
+from llama_index.callbacks.hdp import HdpCallbackHandler, HdpPrincipal, ScopePolicy
+from llama_index.core import Settings
+from llama_index.core.callbacks import CallbackManager
+
+handler = HdpCallbackHandler(
+    signing_key=ed25519_private_key_bytes,
+    principal=HdpPrincipal(id="alice@corp.com", id_type="email"),
+    scope=ScopePolicy(intent="Research pipeline"),
+)
+Settings.callback_manager = CallbackManager([handler])
+```
+
+### Option 3 — Node postprocessor (inline retrieval enforcement)
+
+```python
+from llama_index.callbacks.hdp import HdpNodePostprocessor
+
+postprocessor = HdpNodePostprocessor(
+    signing_key=ed25519_private_key_bytes,
+    strict=False,
+    check_data_classification=True,
+)
+query_engine = index.as_query_engine(node_postprocessors=[postprocessor])
+```
+
+### Verifying a token
+
+```python
+from llama_index.callbacks.hdp import verify_chain
+
+result = verify_chain(token_dict, public_key_bytes)
+if result.valid:
+    print(f"Chain verified: {result.hop_count} hops")
+```
+
+## What makes HDP different from Arize/Langfuse?
+
+| Capability | Arize / Langfuse | HDP |
+|---|---|---|
+| Records what happened | ✓ | ✓ |
+| Records who authorized it | ✗ | ✓ |
+| Cryptographically signed | ✗ | ✓ |
+| Verifiable offline | ✗ | ✓ |
+| Scope enforcement | ✗ | ✓ |
+| No central registry | n/a | ✓ |
+
+## License
+
+CC-BY-4.0

packages/llama-index-callbacks-hdp/llama_index/callbacks/hdp/__init__.py

@@ -0,0 +1,32 @@
+"""HDP integration for LlamaIndex — cryptographic authorization provenance."""
+
+from ._types import DataClassification, HdpPrincipal, HdpScope, HdpToken, HopRecord
+from .callbacks import HdpCallbackHandler, HDPScopeViolationError, ScopePolicy
+from .instrumentation import HdpInstrumentationHandler
+from .postprocessor import HdpNodePostprocessor
+from .session import clear_token, get_token, set_token
+from .verify import HopVerification, VerificationResult, verify_chain
+
+__all__ = [
+    # Core types
+    "DataClassification",
+    "HdpPrincipal",
+    "HdpScope",
+    "HdpToken",
+    "HopRecord",
+    # Policy
+    "ScopePolicy",
+    "HDPScopeViolationError",
+    # Integration layers
+    "HdpCallbackHandler",
+    "HdpInstrumentationHandler",
+    "HdpNodePostprocessor",
+    # Session
+    "get_token",
+    "set_token",
+    "clear_token",
+    # Verification
+    "verify_chain",
+    "VerificationResult",
+    "HopVerification",
+]

packages/llama-index-callbacks-hdp/llama_index/callbacks/hdp/_crypto.py

@@ -0,0 +1,71 @@
+"""Cryptographic primitives for HDP — Ed25519 signing/verification with RFC 8785 canonical JSON.
+
+Matches the signing scheme in the TypeScript SDK (src/crypto/sign.ts + src/crypto/verify.ts):
+  - Root: canonicalize({header, principal, scope}) → Ed25519 → base64url
+  - Hop:  canonicalize({chain: [...], root_sig: <value>}) → Ed25519 → base64url
+"""
+
+from __future__ import annotations
+
+import base64
+from typing import Any
+
+import jcs
+from cryptography.exceptions import InvalidSignature
+from cryptography.hazmat.primitives.asymmetric.ed25519 import Ed25519PrivateKey, Ed25519PublicKey
+
+
+def _b64url(sig_bytes: bytes) -> str:
+    return base64.urlsafe_b64encode(sig_bytes).rstrip(b"=").decode()
+
+
+def _canonicalize(obj: Any) -> bytes:
+    return jcs.canonicalize(obj)
+
+
+def sign_root(unsigned_token: dict, private_key_bytes: bytes, kid: str) -> dict:
+    subset = {f: unsigned_token[f] for f in ["header", "principal", "scope"] if f in unsigned_token}
+    message = _canonicalize(subset)
+    key = Ed25519PrivateKey.from_private_bytes(private_key_bytes)
+    sig_bytes = key.sign(message)
+    return {
+        "alg": "Ed25519",
+        "kid": kid,
+        "value": _b64url(sig_bytes),
+        "signed_fields": ["header", "principal", "scope"],
+    }
+
+
+def sign_hop(cumulative_chain: list[dict], root_sig_value: str, private_key_bytes: bytes) -> str:
+    payload = {"chain": cumulative_chain, "root_sig": root_sig_value}
+    message = _canonicalize(payload)
+    key = Ed25519PrivateKey.from_private_bytes(private_key_bytes)
+    sig_bytes = key.sign(message)
+    return _b64url(sig_bytes)
+
+
+def _b64url_decode(s: str) -> bytes:
+    padding = 4 - len(s) % 4
+    return base64.urlsafe_b64decode(s + "=" * padding)
+
+
+def verify_root(token: dict, public_key: Ed25519PublicKey) -> bool:
+    try:
+        subset = {f: token[f] for f in ["header", "principal", "scope"] if f in token}
+        message = _canonicalize(subset)
+        sig_bytes = _b64url_decode(token["signature"]["value"])
+        public_key.verify(sig_bytes, message)
+        return True
+    except (InvalidSignature, KeyError, Exception):
+        return False
+
+
+def verify_hop(cumulative_chain: list[dict], root_sig_value: str, hop_signature: str, public_key: Ed25519PublicKey) -> bool:
+    try:
+        payload = {"chain": cumulative_chain, "root_sig": root_sig_value}
+        message = _canonicalize(payload)
+        sig_bytes = _b64url_decode(hop_signature)
+        public_key.verify(sig_bytes, message)
+        return True
+    except (InvalidSignature, Exception):
+        return False

packages/llama-index-callbacks-hdp/llama_index/callbacks/hdp/_types.py

@@ -0,0 +1,69 @@
+"""Python types mirroring the HDP TypeScript SDK schema."""
+
+from __future__ import annotations
+
+from dataclasses import dataclass, field
+from typing import Any, Literal, Optional
+
+DataClassification = Literal["public", "internal", "confidential", "restricted"]
+AgentType = Literal["orchestrator", "sub-agent", "tool-executor", "custom"]
+PrincipalIdType = Literal["email", "uuid", "did", "poh", "opaque"]
+
+
+@dataclass
+class HdpHeader:
+    token_id: str
+    issued_at: int
+    expires_at: int
+    session_id: str
+    version: str = "0.1"
+    parent_token_id: Optional[str] = None
+
+
+@dataclass
+class HdpPrincipal:
+    id: str
+    id_type: PrincipalIdType
+    display_name: Optional[str] = None
+    metadata: Optional[dict[str, Any]] = None
+
+
+@dataclass
+class HdpScope:
+    intent: str
+    data_classification: DataClassification
+    network_egress: bool
+    persistence: bool
+    authorized_tools: Optional[list[str]] = None
+    authorized_resources: Optional[list[str]] = None
+    max_hops: Optional[int] = None
+
+
+@dataclass
+class HdpSignature:
+    alg: str
+    kid: str
+    value: str
+    signed_fields: list[str] = field(default_factory=lambda: ["header", "principal", "scope"])
+
+
+@dataclass
+class HopRecord:
+    seq: int
+    agent_id: str
+    agent_type: AgentType
+    timestamp: int
+    action_summary: str
+    parent_hop: int
+    hop_signature: str
+    agent_fingerprint: Optional[str] = None
+
+
+@dataclass
+class HdpToken:
+    hdp: str
+    header: HdpHeader
+    principal: HdpPrincipal
+    scope: HdpScope
+    chain: list[HopRecord]
+    signature: HdpSignature