Skip to content

Commit 5f56eab

Browse files
authored
*: Expose subgraph logs via subgraph GraphQL (#6278)
* graph: Add LogStore trait and core types Introduces the foundation for the log store system with: - LogStore trait for querying logs from backends - LogLevel enum with FromStr trait implementation - LogEntry and LogQuery types for structured log data - LogStoreFactory for creating backend instances - NoOpLogStore as default (disabled) implementation * graph: Add LogStore backend implementations Implements three log storage backends for querying logs: - FileLogStore: Streams JSON Lines files with bounded memory usage - ElasticsearchLogStore: Queries Elasticsearch indices with full-text search - LokiLogStore: Queries Grafana Loki using LogQL All backends implement the LogStore trait and support: - Filtering by log level, timestamp range, and text search - Pagination via first/skip parameters - Returning structured LogEntry objects Dependencies added: reqwest, serde_json for HTTP clients. * graph: Add log drains for writing to backends Implements slog drains for capturing and writing logs: - FileDrain: Writes logs to JSON Lines files (one file per subgraph) - LokiDrain: Writes logs to Grafana Loki via HTTP push API Both drains: - Capture structured log entries with metadata (module, line, column) - Format logs with timestamp, level, text, and arguments - Use efficient serialization with custom KVSerializers * graph,node: Add log store configuration system Adds a configuration layer for selecting and configuring log backends: - LogStoreConfig enum with variants: Disabled, File, Elasticsearch, Loki - LogConfigProvider for loading config from environment variables and CLI args - Unified GRAPH_LOG_STORE_* environment variable naming - CLI arguments with --log-store-backend and backend-specific options - Configuration precedence: CLI args > env vars > defaults - Deprecation warnings for old config variables Supported configuration: - Backend selection (disabled, file, elasticsearch, loki) - File: directory, max size, retention days - Elasticsearch: endpoint, credentials, index, timeout - Loki: endpoint, tenant ID * graph,node: Refactor LoggerFactory for multi-backend support Refactors LoggerFactory to use LogStoreConfig instead of elastic-only: - Replaced elastic_config with log_store_config parameter - Build ElasticLoggingConfig on-demand from LogStoreConfig::Elasticsearch - Support all log drain types (File, Loki, Elasticsearch) - Maintain backward compatibility with existing elastic configuration This enables the factory to create drains for any configured backend while preserving the existing component logger patterns. * graphql,graph: Add GraphQL schema and resolver for _logs query Adds GraphQL API for querying subgraph logs: Schema types: - LogLevel enum (CRITICAL, ERROR, WARNING, INFO, DEBUG) - _Log_ type with id, timestamp, level, text, arguments, meta - _LogArgument_ type for structured key-value pairs - _LogMeta_ type for source location (module, line, column) Query field (_logs) with filters: - level: Filter by log level - from/to: Timestamp range (ISO 8601) - search: Text search in log messages - first/skip: Pagination (max 1000, skip max 10000) * graphql,node,server,store: Wire up _logs query in execution Integrates _logs query into the GraphQL execution pipeline: Execution layer: - Execute _logs queries via log_store.query_logs() - Convert LogEntry results to GraphQL response objects - Handle log store errors gracefully Query parsing: - Recognize _logs as special query field - Build LogQuery from GraphQL arguments - Pass log_store to execution context Service wiring: - Create log store from configuration in launcher - Provide log store to GraphQL runner - Use NoOpLogStore in test environments This completes the read path from GraphQL query to log storage backend. * tests: Add integration tests for log querying Adds comprehensive integration test for _logs query: Test implementation: - Deploys logs-query subgraph and waits for sync - Triggers contract events to generate logs - Queries _logs field with various filters - Verifies log entries are returned correctly - Tests filtering by level and text search * docs: Add log store documentation * graphql,tests: Prevent combining _logs with entity queries * graph: Refactor log drains to share common code - Create graph/src/log/common.rs for common log drain functionality - SimpleKVSerializer: Concatenates KV pairs to strings - VecKVSerializer: Collects KV pairs into Vec<(String, String)> - HashMapKVSerializer: Collects KV pairs into HashMap - LogMeta: Shared metadata structure (module, line, column) - LogEntryBuilder: Builder for common log entry fields - level_to_str(): Converts slog::Level to string - create_async_logger(): Consistent async logger creation - Updated FileDrain, LokiDrain, and ElasticDrain to use the log common utilities * graphql, tests: Ensure logs graphql queries work on failed subgraphs - include _logs in the set of special fields that bypass indexing error shortcutting when subgraph failed - add integration test to ensure _log queries return logs after subgraph failed * graph: Add Default impls for log serializers * graphql, tests: _logs queries only return fields selected in query * graph, node: Cleanup FileLogStore files on startup - Keep logs within retention_hours of now, skipping cleanup if --log-store-retention-hours=0 * CLAUDE.md: Add integration test hang handling and port override guidance * graph, graphql, tests: Support specifying sort order in _logs queries * node, graph, tests: Configure log store via graph-node.toml - Add [log_store] section to graph-node.toml as the sole configuration path for log store backends - Remove env var (GRAPH_LOG_STORE_*) and CLI arg (--log-store-*) config paths, LogStoreConfigProvider, and env var helper utilities * graph, node: Add basic auth support for Loki log store * tests: Add archive and traces features to integration test config The --ethereum-rpc CLI mode defaults to archive,traces features, but the generated TOML config had features = []. This caused subgraphs that depend on those capabilities to fail during indexing.
1 parent 9a31c6d commit 5f56eab

46 files changed

Lines changed: 5483 additions & 737 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

CLAUDE.md

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -98,6 +98,9 @@ TEST_CASE=grafted just test-integration
9898

9999
# (Optional) Use graph-cli instead of gnd for compatibility testing
100100
GRAPH_CLI=node_modules/.bin/graph just test-integration
101+
102+
# Override ports if using different service ports (e.g., for local development)
103+
POSTGRES_TEST_PORT=5432 ETHEREUM_TEST_PORT=8545 IPFS_TEST_PORT=5001 just test-integration
101104
```
102105

103106
**⚠️ Test Verification Requirements:**
@@ -111,6 +114,8 @@ GRAPH_CLI=node_modules/.bin/graph just test-integration
111114
- Integration tests take significant time (several minutes)
112115
- Tests automatically reset the database between runs
113116
- Logs are written to `tests/integration-tests/graph-node.log`
117+
- **If a test hangs for >10 minutes**, it's likely stuck - kill with `pkill -9 integration_tests` and check logs
118+
- CI uses the default ports (3011, 3021, 3001) - local development can override with environment variables
114119

115120
### Code Quality
116121

0 commit comments

Comments
 (0)