feat(monitoring): deploy Prometheus + Grafana stack alongside nodes#185
Merged
Conversation
|
Gave this a hands-on test locally (built the branch, deployed a private single-SR network with
Two things before this can merge:
Minor, non-blocking follow-ups (could be separate issues): document the monitoring workflow in |
barbatos2011
added a commit
to barbatos2011/tron-deployment
that referenced
this pull request
Jun 13, 2026
Replace the equivalence gate's per-store byte copy with a copy-on-write clone. The gate materialises 8 fixture stores twice per run (scratchGo + scratchJava); on a full RocksDB fixture that copy dominates wall-clock once the snapshot is cached. New internal/fsclone package (designed via /plan-eng-review tronprotocol#185): - CloneDir(src,dst)->(method,err): clonefile(2) on darwin (recursive in one syscall), FICLONE per-file on linux, byte-copy fallback elsewhere. Build-tagged clone_{darwin,linux,fallback}.go + clone_unix.go errno classifier. golang.org/x/sys promoted to a direct dep. - Safety: stages into a primitive-created temp dir and atomically renames into place; refuses a pre-existing dst; on failure removes only its own temp, never caller data. Falls back ONLY on CoW-unsupported errnos (EXDEV/EOPNOTSUPP/ENOTSUP/ENOTTY/EINVAL/ENOSYS); real errors propagate instead of being masked as a slow copy. - Drops the manual copy_file_range layer: io.Copy already uses it for *os.File on linux. Equivalence test: - Scratch dirs now created ADJACENT to the fixture (same filesystem) instead of t.TempDir(); t.TempDir() lands under os.TempDir(), often a different filesystem, so the clone would EXDEV and silently fall back to a full copy -- the optimization would fire nowhere. (Codex finding.) - Calls fsclone.CloneDir; test-local cloneDir/copyDir/copyFile and their round-trip tests deleted (single clone+copy impl now lives in fsclone). Tests (internal/fsclone): round-trip + independence, a seam that forces the byte-copy fallback deterministically on any host, real-error propagation, dst-exists refusal, missing-src error, recursiveCopy round-trip. CI: dbfork-equivalence.yml gains a non-blocking CoW probe logging fixture vs scratch device IDs + a cp --reflink=always result, so we can see whether the clone actually reflinks on the runner or falls back. TODOS.md: capture the deferred public 'trond snapshot clone' verb.
Add intent-driven monitoring deployment with --monitor / --no-monitor CLI flags. The monitoring stack is deployed as an independent docker-compose project alongside java-tron nodes. Reuses tron-docker's 5 Grafana dashboard JSONs via //go:embed. Phase 1-6: schema, HOCON metrics injection, renderers, MonitoringRuntime, apply integration, remove cleanup
Clean up Prometheus + Grafana containers when the node is removed. Best-effort cleanup — failure to remove monitoring does not block node removal.
Grafana dashboards from tron-docker use $group template variable. Without the group label, dashboards showed no data even when metrics were being scraped correctly.
The Grafana dashboards from tron-docker have hardcoded defaults for
instance ("tron-docker-dev-001") and group ("java-tron") template
variables. These don't match the dynamically generated labels in
tron-deployment, causing panels to show no data.
Now both query-type and datasource-type variables are normalized:
datasource defaults to "prometheus", query defaults to empty (Grafana
auto-selects the first available label value).
Deploy a single Prometheus + Grafana stack for the entire private network. All nodes are scraped via the shared trond-<name> docker network using container names. Monitoring deployment is non-fatal.
Remove Prometheus + Grafana containers when destroying a private network. Best-effort cleanup — failure does not block destroy.
Unlike trond apply which follows intent's monitoring.enabled, network create requires explicit --monitor flag. This avoids deploying a heavy monitoring stack for every multi-node network unintentionally.
…onitoring usage" This reverts commit 54e9591.
Align trond apply with network create: monitoring is opt-in via --monitor flag. Intent's monitoring.enabled is ignored unless --monitor is passed. Also fix noChangeResult to include monitoring endpoints from existing state, and export ApplyMonitoringDefaults for CLI override use.
When a new node is added to a network that has monitoring deployed, regenerate prometheus.yml with all nodes and SIGHUP the prometheus container to pick up the new scrape target.
Under trond partition, the single Prometheus instance loses visibility into nodes in other groups. This is an intended trade-off.
Both trond status <node> and trond network status now include monitoring stack health (prometheus/grafana container status). Best-effort probe - returns "unknown" when unreachable.
Document --monitor flag, available dashboards, limitations.
Moving the monitoring setup before RenderHOCON ensures the metrics config is injected into the rendered HOCON. Previously it was applied after, so private network nodes started without metrics enabled.
- fakeTarget: add PutFile, Sha256IfExists, CommandExists to match upstream target.Target interface changes - schema: bump version baseline to 1.5.0 after rebase - spec: document verified partition behavior with monitoring
network add now checks whether the network already has a monitoring stack deployed (deployments/<name>-monitoring/docker-compose.yaml) and, if so, flips parsed.Monitoring.Enabled=true on the new node before RenderHOCON. Without this the prometheus scrape config (now reloaded on add) would point at a node whose metrics endpoint is disabled, leaving the target permanently DOWN.
- Change monitoring.prometheus.port default from 9090 to 0
(0 means do not expose prometheus on the host)
- Grafana stays exposed on 3000 by default
- Compose template conditionally renders prometheus ports section
only when PrometheusPort > 0
- Output URLs only include prometheus_url when port is explicitly set
- Users can still expose prometheus via intent:
monitoring:
prometheus:
port: 9091
- examples/monitoring.yaml: remove explicit prometheus.port, add comments explaining the new default (prometheus unexposed, grafana on 3000) - cmd/network/add.go: remove hardcoded "7d" retention in reload; pass empty string since retention is managed by docker-compose command args, not prometheus.yml - internal/render/monitoring.go: document that RenderPrometheusConfig's retention parameter is currently unused
- gofmt all touched files and drop two redundant fmt.Sprintf calls (gosimple S1039) - align prometheus default-port test with the unexposed (port 0) default - skip monitoring teardown when no compose file exists, so `network destroy` no longer prints a stderr warning that pollutes its JSON output and breaks the e2e lifecycle test
…trics fix Upstream c355bca added ensureMetricsEnabled() for features.metrics=true, deliberately a SAFE NO-OP on templates without a prometheus block (nile/private — tracked in upstream issue tronprotocol#167) so it never synthesises into those templates. Our monitoring path needs the opposite behaviour: when monitoring.enabled is true, we must actively synthesise the prometheus block on nile/private (otherwise java-tron never exposes 9527 and Grafana stays empty on private networks — the most common monitoring use-case). Rename our function to ensureMetricsForMonitoring() so both behaviours coexist without changing upstream's contract. Once upstream resolves tronprotocol#167 by adding a prometheus block to nile/private templates, the two functions can be merged in a follow-up.
warku123
force-pushed
the
feat/monitoring
branch
from
June 15, 2026 07:32
550fa0a to
91d9af7
Compare
Author
Rebase update — function rename to coexist with upstream
|
| Function | Trigger | Behaviour on missing prometheus block |
|---|---|---|
ensureMetricsEnabled (upstream, c355bca) |
features.metrics: true |
NO-OP (deliberate, per #167 scope) |
ensureMetricsForMonitoring (this PR) |
monitoring.enabled: true |
Synthesise the block (user explicitly opted into the monitoring stack) |
This avoids the C-style "monitor on private/nile silently produces empty dashboards" failure mode while keeping c355bca's template-non-pollution guarantee intact for the features.metrics path.
Follow-up after #167 lands: once upstream adds a real prometheus block to private_net_config.conf and test_net_config.conf, the two functions can be merged in a small cleanup PR — our synthesis branch will simply become unreachable (case 1: flip existing enable).
CI was green before the rebase and the local go test ./... passes (one flaky TestProbe_StaleWhenOnlyOldBackupServes in internal/snapshot looks pre-existing on develop, unrelated to this PR).
Author
Update — flaky test fix rolled back; two unrelated upstream CI issues to trackReverted commit Two unrelated upstream CI issues observed on 1. Flaky
2.
key: nile-fixture-${{ steps.weekkey.outputs.yyyyww }}
restore-keys:
- nile-fixture-When the weekly key rolls (e.g. ISO week 24 → 25), exact-key cache misses but Both are unrelated to monitoring and should be tracked / fixed by upstream. Happy to file issues or send small fix PRs separately. |
warku123
force-pushed
the
feat/monitoring
branch
from
June 15, 2026 08:37
5971a6a to
91d9af7
Compare
barbatos2011
added a commit
to barbatos2011/tron-deployment
that referenced
this pull request
Jun 15, 2026
The equivalence gate failed with 'destination ... already has a database; pass --force' (exit 10) whenever the weekly fixture cache key rolled. actions/cache reports cache-hit='true' only on an EXACT key match; the 'restore-keys: nile-fixture-' prefix fallback still restores last week's fixture into ./nile-fixture but reports cache-hit=false. The download step was gated 'if: cache-hit != true', so it re-ran into a populated dir and snapshot download (correctly) refused to clobber. Guard on whether the fixture DB directory actually exists instead of on cache-hit: present (exact OR restore-keys hit) → use it; absent (cold miss) → download + prune. Removes the step-level cache-hit gate. Any restored snapshot is valid for equivalence since both Go and Java diff the same copy. Surfaced reviewing tronprotocol#188 (the probe de-flake); affects every PR touching internal/snapshot/** or internal/dbfork/** on a week-key roll, including the monitoring PR tronprotocol#185.
barbatos2011
added a commit
to barbatos2011/tron-deployment
that referenced
this pull request
Jun 15, 2026
The MCP section had drifted from the registered surface, so an agent reading AGENTS.md would miss real capabilities: - Said "20 tools"; trond actually registers 23. Added the two missing ones to the breakdown: verify_config (config drift, read-only) and auto_heal (remediation, destructive, dry_run-able). - The 4 MCP prompts (deploy_fullnode, setup_private_network, diagnose_failing_node, recover_failed_upgrade) registered in prompts.go were undocumented — added a Prompts subsection. - Noted snapshot job control (stop/logs) is CLI-only, and added 'snapshot stop <job-id>' to the detached-download workflow. Counts verified against internal/mcp (23 AddTool, 4 AddPrompt wired via registerPrompts in server.go). Monitoring (PR tronprotocol#185) stays undocumented here until it merges — flagged on that PR.
barbatos2011
added a commit
to barbatos2011/tron-deployment
that referenced
this pull request
Jun 15, 2026
Replace the equivalence gate's per-store byte copy with a copy-on-write clone. The gate materialises 8 fixture stores twice per run (scratchGo + scratchJava); on a full RocksDB fixture that copy dominates wall-clock once the snapshot is cached. New internal/fsclone package (designed via /plan-eng-review tronprotocol#185): - CloneDir(src,dst)->(method,err): clonefile(2) on darwin (recursive in one syscall), FICLONE per-file on linux, byte-copy fallback elsewhere. Build-tagged clone_{darwin,linux,fallback}.go + clone_unix.go errno classifier. golang.org/x/sys promoted to a direct dep. - Safety: stages into a primitive-created temp dir and atomically renames into place; refuses a pre-existing dst; on failure removes only its own temp, never caller data. Falls back ONLY on CoW-unsupported errnos (EXDEV/EOPNOTSUPP/ENOTSUP/ENOTTY/EINVAL/ENOSYS); real errors propagate instead of being masked as a slow copy. - Drops the manual copy_file_range layer: io.Copy already uses it for *os.File on linux. Equivalence test: - Scratch dirs now created ADJACENT to the fixture (same filesystem) instead of t.TempDir(); t.TempDir() lands under os.TempDir(), often a different filesystem, so the clone would EXDEV and silently fall back to a full copy -- the optimization would fire nowhere. (Codex finding.) - Calls fsclone.CloneDir; test-local cloneDir/copyDir/copyFile and their round-trip tests deleted (single clone+copy impl now lives in fsclone). Tests (internal/fsclone): round-trip + independence, a seam that forces the byte-copy fallback deterministically on any host, real-error propagation, dst-exists refusal, missing-src error, recursiveCopy round-trip. CI: dbfork-equivalence.yml gains a non-blocking CoW probe logging fixture vs scratch device IDs + a cp --reflink=always result, so we can see whether the clone actually reflinks on the runner or falls back. TODOS.md: capture the deferred public 'trond snapshot clone' verb.
barbatos2011
added a commit
to barbatos2011/tron-deployment
that referenced
this pull request
Jun 15, 2026
The MCP section had drifted from the registered surface, so an agent reading AGENTS.md would miss real capabilities: - Said "20 tools"; trond actually registers 23. Added the two missing ones to the breakdown: verify_config (config drift, read-only) and auto_heal (remediation, destructive, dry_run-able). - The 4 MCP prompts (deploy_fullnode, setup_private_network, diagnose_failing_node, recover_failed_upgrade) registered in prompts.go were undocumented — added a Prompts subsection. - Noted snapshot job control (stop/logs) is CLI-only, and added 'snapshot stop <job-id>' to the detached-download workflow. Counts verified against internal/mcp (23 AddTool, 4 AddPrompt wired via registerPrompts in server.go). Monitoring (PR tronprotocol#185) stays undocumented here until it merges — flagged on that PR.
barbatos2011
added a commit
to barbatos2011/tron-deployment
that referenced
this pull request
Jun 15, 2026
Replace the equivalence gate's per-store byte copy with a copy-on-write clone. The gate materialises 8 fixture stores twice per run (scratchGo + scratchJava); on a full RocksDB fixture that copy dominates wall-clock once the snapshot is cached. New internal/fsclone package (designed via /plan-eng-review tronprotocol#185): - CloneDir(src,dst)->(method,err): clonefile(2) on darwin (recursive in one syscall), FICLONE per-file on linux, byte-copy fallback elsewhere. Build-tagged clone_{darwin,linux,fallback}.go + clone_unix.go errno classifier. golang.org/x/sys promoted to a direct dep. - Safety: stages into a primitive-created temp dir and atomically renames into place; refuses a pre-existing dst; on failure removes only its own temp, never caller data. Falls back ONLY on CoW-unsupported errnos (EXDEV/EOPNOTSUPP/ENOTSUP/ENOTTY/EINVAL/ENOSYS); real errors propagate instead of being masked as a slow copy. - Drops the manual copy_file_range layer: io.Copy already uses it for *os.File on linux. Equivalence test: - Scratch dirs now created ADJACENT to the fixture (same filesystem) instead of t.TempDir(); t.TempDir() lands under os.TempDir(), often a different filesystem, so the clone would EXDEV and silently fall back to a full copy -- the optimization would fire nowhere. (Codex finding.) - Calls fsclone.CloneDir; test-local cloneDir/copyDir/copyFile and their round-trip tests deleted (single clone+copy impl now lives in fsclone). Tests (internal/fsclone): round-trip + independence, a seam that forces the byte-copy fallback deterministically on any host, real-error propagation, dst-exists refusal, missing-src error, recursiveCopy round-trip. CI: dbfork-equivalence.yml gains a non-blocking CoW probe logging fixture vs scratch device IDs + a cp --reflink=always result, so we can see whether the clone actually reflinks on the runner or falls back. TODOS.md: capture the deferred public 'trond snapshot clone' verb.
3 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Adds an optional, intent-driven monitoring stack (Prometheus + Grafana) that trond can deploy and manage alongside java-tron nodes.
Why are these changes required?
tron-deployment currently has excellent CLI diagnostics (diagnose, health, logs) but no graphical monitoring dashboards. tron-docker already has production-tested Grafana dashboards + Prometheus configs.
This PR embeds those assets and deploys them alongside nodes via an independent docker-compose project.
Key changes:
monitoring:block withenabled,prometheus.port/retention,grafana.port.monitoring.enabled=true,node.metrics.prometheus.enableis automatically injected regardless offeatures.metrics.//go:embed(java-tron-server, java-tron-api, java-tron-api-statistic, java-tron-mechanism, node-exporter-full).trond apply --monitor,trond network create --monitor,trond network add(auto-enables metrics + reloads Prometheus),trond remove/trond network destroy(cleans up monitoring).monitoring.prometheus.port.SchemaVersion1.4.0 → 1.5.0.Follow up:
trond network statuscould surface monitoring container health.Extra details:
Depends on PR #181 (shared docker network + active_peers auto-population), which has already been merged into
developand is included in this rebase.