feat: add DA visualization HTTP endpoints

README.md

@@ -11,6 +11,7 @@ Ev-node is the basis of the Evolve Stack. For more in-depth information about Ev
 ## Using Evolve
 
 Evolve supports multiple sync modes:
+
 - **Hybrid sync**: Sync from both DA layer and P2P network (default when peers are configured)
 - **DA-only sync**: Sync exclusively from DA layer by leaving P2P peers empty (see [Configuration Guide](docs/learn/config.md#da-only-sync-mode))
 - **P2P-priority sync**: Prioritize P2P with DA as fallback

apps/grpc/single/README.md

@@ -55,8 +55,7 @@ Start the Evolve node with:
   --root-dir ~/.grpc-single \
   --grpc-executor-url http://localhost:50051 \
   --da.address http://localhost:7980 \
-  --da.auth-token your-da-token \
-  --chain-id your-chain-id
+  --da.auth-token your-da-token
 ```
 
 ## Command-Line Flags
@@ -93,12 +92,11 @@ Start the Evolve node with:
 3. Initialize and run the node:
 
     ```bash
-    ./grpc-single init --root-dir ~/.grpc-single
+    ./grpc-single init --root-dir ~/.grpc-single --chain-id test-chain
     ./grpc-single start \
       --root-dir ~/.grpc-single \
       --grpc-executor-url http://localhost:50051 \
-      --da.address http://localhost:7980 \
-      --chain-id test-chain
+      --da.address http://localhost:7980
     ```
 
 ## Architecture

block/CLAUDE.md

@@ -9,6 +9,7 @@ The block package is the core of ev-node's block management system. It handles b
 ## Core Components
 
 ### Manager (`manager.go`)
+
 - **Purpose**: Central orchestrator for all block operations
 - **Key Responsibilities**:
   - Transaction aggregation into blocks
@@ -18,6 +19,7 @@ The block package is the core of ev-node's block management system. It handles b
   - P2P block/header gossiping
 
 ### Aggregation (`aggregation.go`, `lazy_aggregation_test.go`)
+
 - **Purpose**: Collects transactions from mempool and creates blocks
 - **Modes**:
   - **Normal Mode**: Produces blocks at regular intervals (BlockTime)
@@ -28,6 +30,7 @@ The block package is the core of ev-node's block management system. It handles b
   - `normalAggregationLoop`: Regular block production
 
 ### Synchronization (`sync.go`, `sync_test.go`)
+
 - **Purpose**: Keeps the node synchronized with the network
 - **Key Functions**:
   - `SyncLoop`: Main synchronization loop
@@ -36,6 +39,7 @@ The block package is the core of ev-node's block management system. It handles b
   - Handles header and data caching
 
 ### Data Availability (`da_includer.go`, `submitter.go`, `retriever.go`)
+
 - **DA Includer**: Manages DA blob inclusion proofs and validation
 - **Submitter**: Handles block submission to the DA layer with retry logic
 - **Retriever**: Fetches blocks from the DA layer
@@ -45,6 +49,7 @@ The block package is the core of ev-node's block management system. It handles b
   - Batch submission optimization
 
 ### Storage (`store.go`, `store_test.go`)
+
 - **Purpose**: Persistent storage for blocks and state
 - **Key Features**:
   - Block height tracking
@@ -53,6 +58,7 @@ The block package is the core of ev-node's block management system. It handles b
   - Migration support for namespace changes
 
 ### Pending Blocks (`pending_base.go`, `pending_headers.go`, `pending_data.go`)
+
 - **Purpose**: Manages blocks awaiting DA inclusion or validation
 - **Components**:
   - **PendingBase**: Base structure for pending blocks
@@ -64,6 +70,7 @@ The block package is the core of ev-node's block management system. It handles b
   - Memory-efficient caching
 
 ### Metrics (`metrics.go`, `metrics_helpers.go`)
+
 - **Purpose**: Performance monitoring and observability
 - **Key Metrics**:
   - Block production times
@@ -74,20 +81,23 @@ The block package is the core of ev-node's block management system. It handles b
 ## Key Workflows
 
 ### Block Production Flow
+
 1. Transactions collected from mempool
 2. Block created with proper header and data
 3. Block executed through executor
 4. Block submitted to DA layer
 5. Block gossiped to P2P network
 
 ### Synchronization Flow
+
 1. Headers received from P2P network
 2. Headers validated and cached
 3. Block data retrieved from DA layer
 4. Blocks applied to state
 5. Sync progress updated
 
 ### DA Submission Flow
+
 1. Block prepared for submission
 2. Blob created with block data
 3. Submission attempted with retries
@@ -97,47 +107,55 @@ The block package is the core of ev-node's block management system. It handles b
 ## Configuration
 
 ### Time Parameters
+
 - `BlockTime`: Target time between blocks (default: 1s)
 - `DABlockTime`: DA layer block time (default: 6s)
 - `LazyBlockTime`: Max time between blocks in lazy mode (default: 60s)
 
 ### Limits
+
 - `maxSubmitAttempts`: Max DA submission retries (30)
 - `defaultMempoolTTL`: Blocks until tx dropped (25)
 
 ## Testing Strategy
 
 ### Unit Tests
+
 - Test individual components in isolation
 - Mock external dependencies (DA, executor, sequencer)
 - Focus on edge cases and error conditions
 
 ### Integration Tests
+
 - Test component interactions
 - Verify block flow from creation to storage
 - Test synchronization scenarios
 
 ### Performance Tests (`da_speed_test.go`)
+
 - Measure DA submission performance
 - Test batch processing efficiency
 - Validate metrics accuracy
 
 ## Common Development Tasks
 
 ### Adding a New DA Feature
+
 1. Update DA interfaces in `core/da`
 2. Modify `da_includer.go` for inclusion logic
 3. Update `submitter.go` for submission flow
 4. Add retrieval logic in `retriever.go`
 5. Update tests and metrics
 
 ### Modifying Block Production
+
 1. Update aggregation logic in `aggregation.go`
 2. Adjust timing in Manager configuration
 3. Update metrics collection
 4. Test both normal and lazy modes
 
 ### Implementing New Sync Strategy
+
 1. Modify `SyncLoop` in `sync.go`
 2. Update pending block handling
 3. Adjust cache strategies
@@ -182,4 +200,4 @@ The block package is the core of ev-node's block management system. It handles b
 - Log with structured fields
 - Return errors with context
 - Use metrics for observability
-- Test error conditions thoroughly
+- Test error conditions thoroughly

block/manager.go

@@ -25,6 +25,7 @@ import (
 	"github.com/evstack/ev-node/pkg/cache"
 	"github.com/evstack/ev-node/pkg/config"
 	"github.com/evstack/ev-node/pkg/genesis"
+	"github.com/evstack/ev-node/pkg/rpc/server"
 	"github.com/evstack/ev-node/pkg/signer"
 	storepkg "github.com/evstack/ev-node/pkg/store"
 	"github.com/evstack/ev-node/types"
@@ -421,6 +422,16 @@ func NewManager(
 		return nil, fmt.Errorf("failed to load cache: %w", err)
 	}
 
+	// Initialize DA visualization server if enabled
+	if config.RPC.EnableDAVisualization {
+		daVisualizationServer := server.NewDAVisualizationServer(da, logger.With().Str("module", "da_visualization").Logger(), config.Node.Aggregator)
+		server.SetDAVisualizationServer(daVisualizationServer)
+		logger.Info().Msg("DA visualization server enabled")
+	} else {
+		// Ensure the global server is nil when disabled
+		server.SetDAVisualizationServer(nil)
+	}
+
 	return m, nil
 }
 

block/submitter.go

@@ -6,6 +6,7 @@ import (
 	"time"
 
 	coreda "github.com/evstack/ev-node/core/da"
+	"github.com/evstack/ev-node/pkg/rpc/server"
 	"github.com/evstack/ev-node/types"
 	"google.golang.org/protobuf/proto"
 )
@@ -300,6 +301,12 @@ func handleSubmissionResult[T any](
 
 	case coreda.StatusContextCanceled:
 		m.logger.Info().Int("attempt", retryStrategy.attempt).Msg("DA layer submission canceled due to context cancellation")
+
+		// Record canceled submission in DA visualization server
+		if daVisualizationServer := server.GetDAVisualizationServer(); daVisualizationServer != nil {
+			daVisualizationServer.RecordSubmission(&res, retryStrategy.gasPrice, uint64(len(remaining)))
+		}
+
 		return submissionOutcome[T]{
 			RemainingItems:   remaining,
 			RemainingMarshal: marshaled,
@@ -328,6 +335,11 @@ func handleSuccessfulSubmission[T any](
 	remLen := len(remaining)
 	allSubmitted := res.SubmittedCount == uint64(remLen)
 
+	// Record submission in DA visualization server
+	if daVisualizationServer := server.GetDAVisualizationServer(); daVisualizationServer != nil {
+		daVisualizationServer.RecordSubmission(res, retryStrategy.gasPrice, res.SubmittedCount)
+	}
+
 	m.logger.Info().Str("itemType", itemType).Float64("gasPrice", retryStrategy.gasPrice).Uint64("count", res.SubmittedCount).Msg("successfully submitted items to DA layer")
 
 	submitted := remaining[:res.SubmittedCount]
@@ -359,6 +371,12 @@ func handleMempoolFailure[T any](
 	m.logger.Error().Str("error", res.Message).Int("attempt", attempt).Msg("DA layer submission failed")
 
 	m.recordDAMetrics("submission", DAModeFail)
+
+	// Record failed submission in DA visualization server
+	if daVisualizationServer := server.GetDAVisualizationServer(); daVisualizationServer != nil {
+		daVisualizationServer.RecordSubmission(res, retryStrategy.gasPrice, uint64(len(remaining)))
+	}
+
 	retryStrategy.BackoffOnMempool(int(m.config.DA.MempoolTTL), m.config.DA.BlockTime.Duration, m.gasMultiplier)
 
 	m.logger.Info().Dur("backoff", retryStrategy.backoff).Float64("gasPrice", retryStrategy.gasPrice).Msg("retrying DA layer submission with")
@@ -385,6 +403,17 @@ func handleTooBigError[T any](
 
 	m.recordDAMetrics("submission", DAModeFail)
 
+	// Record failed submission in DA visualization server (create a result for TooBig error)
+	if daVisualizationServer := server.GetDAVisualizationServer(); daVisualizationServer != nil {
+		tooBigResult := &coreda.ResultSubmit{
+			BaseResult: coreda.BaseResult{
+				Code:    coreda.StatusTooBig,
+				Message: "blob too big",
+			},
+		}
+		daVisualizationServer.RecordSubmission(tooBigResult, retryStrategy.gasPrice, uint64(len(remaining)))
+	}
+
 	if len(remaining) > 1 {
 		totalSubmitted, err := submitWithRecursiveSplitting(m, ctx, remaining, marshaled, retryStrategy.gasPrice, postSubmit, itemType, namespace)
 		if err != nil {
@@ -437,6 +466,12 @@ func handleGenericFailure[T any](
 	m.logger.Error().Str("error", res.Message).Int("attempt", attempt).Msg("DA layer submission failed")
 
 	m.recordDAMetrics("submission", DAModeFail)
+
+	// Record failed submission in DA visualization server
+	if daVisualizationServer := server.GetDAVisualizationServer(); daVisualizationServer != nil {
+		daVisualizationServer.RecordSubmission(res, retryStrategy.gasPrice, uint64(len(remaining)))
+	}
+
 	retryStrategy.BackoffOnFailure()
 
 	return submissionOutcome[T]{
@@ -637,12 +672,22 @@ func processBatch[T any](
 		postSubmit(submitted, &batchRes, gasPrice)
 		m.logger.Info().Int("batchSize", len(batch.Items)).Uint64("submittedCount", batchRes.SubmittedCount).Msg("successfully submitted batch to DA layer")
 
+		// Record successful submission in DA visualization server
+		if daVisualizationServer := server.GetDAVisualizationServer(); daVisualizationServer != nil {
+			daVisualizationServer.RecordSubmission(&batchRes, gasPrice, batchRes.SubmittedCount)
+		}
+
 		return batchResult[T]{
 			action:         batchActionSubmitted,
 			submittedCount: int(batchRes.SubmittedCount),
 		}
 	}
 
+	// Record failed submission in DA visualization server for all error cases
+	if daVisualizationServer := server.GetDAVisualizationServer(); daVisualizationServer != nil {
+		daVisualizationServer.RecordSubmission(&batchRes, gasPrice, uint64(len(batch.Items)))
+	}
+
 	if batchRes.Code == coreda.StatusTooBig && len(batch.Items) > 1 {
 		// Batch is too big - let the caller handle splitting
 		m.logger.Info().Int("batchSize", len(batch.Items)).Msg("batch too big, returning to caller for splitting")