Monitor performance of tunnels

[geanceretta-inobram]

Often we get tickets from our users complaining a degradation on the remote access speed. Do we have any monitoring tool allowing us to know if the performance of the tunnels are nominal or something weird is happening?

KPIs like number of active tunnels, latencies, etc, if possible, would help a lot.

KPIs for ShellHub Tunnel Monitoring (Server-Side)

Availability & Reliability

KPI	Description
Tunnel uptime (%)	Time a tunnel is active and responsive vs. requested duration
Tunnel establishment success rate	% of tunnel creation requests that succeed vs. fail
Mean time to establish tunnel	Latency from tunnel request to first byte forwarded
Agent connection uptime	% of time the agent↔server WebSocket is alive per device
Reconnection frequency	How often agents drop and reconnect — signals network instability

Performance

KPI	Description
Tunnel latency (p50, p95, p99)	Round-trip time through the tunnel (server→agent→app→agent→server)
Throughput per tunnel	Bytes/sec forwarded through each tunnel
Aggregate throughput	Total bandwidth consumed by all active tunnels
Time to first byte (TTFB)	Time from incoming request to first response byte, measured at the server edge
SSH channel setup time	Time to open a new direct-tcpip channel on an existing connection

Capacity & Utilization

KPI	Description
Active tunnels (concurrent)	Number of tunnels open at any given moment
Active tunnels per device	Identifies devices with unusually high tunnel usage
Active tunnels per namespace/tenant	For multi-tenant: fair-use and capacity planning
Server CPU utilization	CPU consumed by tunnel proxy operations
Server memory utilization	Memory used for buffering tunnel data
Network I/O saturation	% of server network bandwidth consumed by tunnel traffic
WebSocket connection count	Total persistent agent connections the server is managing

Usage & Adoption

KPI	Description
Tunnels created per day/week	Trend of feature adoption
Unique devices using tunnels	How many devices actively use the feature
Unique users creating tunnels	Who's using it and how often
Tunnel duration distribution	Short-lived (debug) vs. long-lived (persistent access) tunnels
Most tunneled ports	Which services are being accessed (80, 443, 3000, 8080, etc.)
Peak concurrent tunnels	Capacity planning — max simultaneous tunnels observed

Errors & Failures

KPI	Description
Tunnel creation failure rate	% of requests that fail (agent unreachable, auth denied, port refused)
Tunnel drop rate	Unexpected tunnel terminations (not user-initiated)
Mean time to recovery (MTTR)	Average time to re-establish a dropped tunnel
Connection refused errors	Agent can reach device but target port isn't listening
Auth/RBAC denial rate	Tunnel requests blocked by policy — security signal
Timeout rate	Tunnels that timeout during establishment or data transfer

Security & Audit

KPI	Description
Tunnel requests per user (rate)	Detect abuse or compromised accounts
Unauthorized tunnel attempts	Attempts to tunnel to restricted devices/ports
Tunnels to sensitive ports	Access to ports like 22, 3306, 5432, 6379 — flag for review
Session duration anomalies	Unusually long or short sessions vs. baseline
Data volume anomalies	Unexpectedly high data transfer through a tunnel

SLA-Oriented Composite KPIs

KPI	Formula
Tunnel availability SLA	(total_tunnel_uptime / total_tunnel_requested_time) × 100
Tunnel success SLA	(successful_creations / total_creation_requests) × 100
P95 tunnel latency budget	Track against a defined threshold (e.g., < 200ms added latency)
Error budget burn rate	How fast you're consuming your allowed failure budget over a period

---

Recommended Alerting Thresholds

• Tunnel creation failure rate > 5% over 5 min
• P99 latency > 500ms (or whatever your baseline + buffer is)
• Agent reconnection rate > 3x/hour per device
• Server CPU > 80% sustained for 10 min
• Concurrent tunnels approaching 80% of tested capacity
• Any tunnel to a sensitive port from an unexpected user/namespace

[gustavosbarreto]

Hi @geanceretta-inobram,

Thanks for the detailed write-up. Before we look into this, could you help us understand how you'd ideally consume this data?

1. Where do you want to see it, inside ShellHub or exported to your own tooling?
2. Who's the audience, your support team during incidents or ops for capacity planning?
3. Visibility only, or alerts as well?

That helps us scope it correctly.

[geanceretta-inobram]

Hi @gustavosbarreto,

Those are AI suggestions that came up when investigating it, but may be useful if they make sense to you.

Being able to both see it on Shellhub and consume it via API would be great. So our R&D team can initially look up on Shellhub's UI but also further via API integrate it in our own backoffice tools as part of a "status of services" dashboard, that could aggregate other non-shellhub related services that we use.

As a MVP anything that could give us a clue of a surge of tunnels creation, usage or latency would be a great start.

Thanks a lot!