Commit 5f56eab
authored
*: Expose subgraph logs via subgraph GraphQL (#6278)
* graph: Add LogStore trait and core types
Introduces the foundation for the log store system with:
- LogStore trait for querying logs from backends
- LogLevel enum with FromStr trait implementation
- LogEntry and LogQuery types for structured log data
- LogStoreFactory for creating backend instances
- NoOpLogStore as default (disabled) implementation
* graph: Add LogStore backend implementations
Implements three log storage backends for querying logs:
- FileLogStore: Streams JSON Lines files with bounded memory usage
- ElasticsearchLogStore: Queries Elasticsearch indices with full-text search
- LokiLogStore: Queries Grafana Loki using LogQL
All backends implement the LogStore trait and support:
- Filtering by log level, timestamp range, and text search
- Pagination via first/skip parameters
- Returning structured LogEntry objects
Dependencies added: reqwest, serde_json for HTTP clients.
* graph: Add log drains for writing to backends
Implements slog drains for capturing and writing logs:
- FileDrain: Writes logs to JSON Lines files (one file per subgraph)
- LokiDrain: Writes logs to Grafana Loki via HTTP push API
Both drains:
- Capture structured log entries with metadata (module, line, column)
- Format logs with timestamp, level, text, and arguments
- Use efficient serialization with custom KVSerializers
* graph,node: Add log store configuration system
Adds a configuration layer for selecting and configuring log backends:
- LogStoreConfig enum with variants: Disabled, File, Elasticsearch, Loki
- LogConfigProvider for loading config from environment variables and CLI args
- Unified GRAPH_LOG_STORE_* environment variable naming
- CLI arguments with --log-store-backend and backend-specific options
- Configuration precedence: CLI args > env vars > defaults
- Deprecation warnings for old config variables
Supported configuration:
- Backend selection (disabled, file, elasticsearch, loki)
- File: directory, max size, retention days
- Elasticsearch: endpoint, credentials, index, timeout
- Loki: endpoint, tenant ID
* graph,node: Refactor LoggerFactory for multi-backend support
Refactors LoggerFactory to use LogStoreConfig instead of elastic-only:
- Replaced elastic_config with log_store_config parameter
- Build ElasticLoggingConfig on-demand from LogStoreConfig::Elasticsearch
- Support all log drain types (File, Loki, Elasticsearch)
- Maintain backward compatibility with existing elastic configuration
This enables the factory to create drains for any configured backend
while preserving the existing component logger patterns.
* graphql,graph: Add GraphQL schema and resolver for _logs query
Adds GraphQL API for querying subgraph logs:
Schema types:
- LogLevel enum (CRITICAL, ERROR, WARNING, INFO, DEBUG)
- _Log_ type with id, timestamp, level, text, arguments, meta
- _LogArgument_ type for structured key-value pairs
- _LogMeta_ type for source location (module, line, column)
Query field (_logs) with filters:
- level: Filter by log level
- from/to: Timestamp range (ISO 8601)
- search: Text search in log messages
- first/skip: Pagination (max 1000, skip max 10000)
* graphql,node,server,store: Wire up _logs query in execution
Integrates _logs query into the GraphQL execution pipeline:
Execution layer:
- Execute _logs queries via log_store.query_logs()
- Convert LogEntry results to GraphQL response objects
- Handle log store errors gracefully
Query parsing:
- Recognize _logs as special query field
- Build LogQuery from GraphQL arguments
- Pass log_store to execution context
Service wiring:
- Create log store from configuration in launcher
- Provide log store to GraphQL runner
- Use NoOpLogStore in test environments
This completes the read path from GraphQL query to log storage backend.
* tests: Add integration tests for log querying
Adds comprehensive integration test for _logs query:
Test implementation:
- Deploys logs-query subgraph and waits for sync
- Triggers contract events to generate logs
- Queries _logs field with various filters
- Verifies log entries are returned correctly
- Tests filtering by level and text search
* docs: Add log store documentation
* graphql,tests: Prevent combining _logs with entity queries
* graph: Refactor log drains to share common code
- Create graph/src/log/common.rs for common log drain functionality
- SimpleKVSerializer: Concatenates KV pairs to strings
- VecKVSerializer: Collects KV pairs into Vec<(String, String)>
- HashMapKVSerializer: Collects KV pairs into HashMap
- LogMeta: Shared metadata structure (module, line, column)
- LogEntryBuilder: Builder for common log entry fields
- level_to_str(): Converts slog::Level to string
- create_async_logger(): Consistent async logger creation
- Updated FileDrain, LokiDrain, and ElasticDrain to use the log common
utilities
* graphql, tests: Ensure logs graphql queries work on failed subgraphs
- include _logs in the set of special fields that bypass indexing error
shortcutting when subgraph failed
- add integration test to ensure _log queries return logs after subgraph
failed
* graph: Add Default impls for log serializers
* graphql, tests: _logs queries only return fields selected in query
* graph, node: Cleanup FileLogStore files on startup
- Keep logs within retention_hours of now, skipping cleanup if
--log-store-retention-hours=0
* CLAUDE.md: Add integration test hang handling and port override guidance
* graph, graphql, tests: Support specifying sort order in _logs queries
* node, graph, tests: Configure log store via graph-node.toml
- Add [log_store] section to graph-node.toml as the sole configuration
path for log store backends
- Remove env var (GRAPH_LOG_STORE_*) and CLI arg (--log-store-*)
config paths, LogStoreConfigProvider, and env var helper utilities
* graph, node: Add basic auth support for Loki log store
* tests: Add archive and traces features to integration test config
The --ethereum-rpc CLI mode defaults to archive,traces features, but
the generated TOML config had features = []. This caused subgraphs
that depend on those capabilities to fail during indexing.1 parent 9a31c6d commit 5f56eab
46 files changed
Lines changed: 5483 additions & 737 deletions
File tree
- docs
- gnd/src/commands/test
- graphql/src
- execution
- query
- store
- graph
- src
- components
- log_store
- log
- schema
- node
- resources/tests
- src
- bin
- server/index-node/src
- store/test-store
- src
- tests/graphql
- tests
- integration-tests/logs-query
- abis
- src
- src
- fixture
- tests
Some content is hidden
Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
98 | 98 | | |
99 | 99 | | |
100 | 100 | | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
101 | 104 | | |
102 | 105 | | |
103 | 106 | | |
| |||
111 | 114 | | |
112 | 115 | | |
113 | 116 | | |
| 117 | + | |
| 118 | + | |
114 | 119 | | |
115 | 120 | | |
116 | 121 | | |
| |||
0 commit comments