Node failure during compute job execution end time tracking causes incorrect billing

[bogdanfazakas]

When a node fails (crashes/restarts) during an active compute job, the job's end timestamp (algoStopTimestamp) is set to the moment the node restarts rather than the moment the algorithm actually stopped running. Since payment is calculated as algoStopTimestamp - algoStartTimestamp, the consumer is billed for the full window including the node's downtime time during which no compute was actually happening.
This is distinct from algorithm failure, where billing is intentional (compute resources were allocated and used). In a node failure scenario, the node operator is incorrectly compensated for time they were offline.
Billing should only cover the time the algorithm was verifiably running. The duration between node crash and node restart (T2 → T3) should not be billed.

[hamidhamid09186-crypto]

Important Consideration regarding Fix #1331: Addressing Potential Edge Cases in Billing Accuracy
While the proposed fix using container.finishedAt in #1331 is a significant step forward, it might still be susceptible to certain catastrophic failure modes where container metadata becomes unavailable or corrupted.
Identified Risks:
Metadata Loss: If the Docker/K8s daemon crashes or performs aggressive garbage collection before the node restarts, finishedAt data might be lost, reverting the system to incorrect billing.
Incomplete Persistence: In cases of hard host crashes or disk I/O errors during the crash, the container lifecycle events might not be reliably recorded.
Proposed Robust Alternative: Hybrid Heartbeat & Orchestrator-Level Tracking
To achieve 100% billing accuracy and protect consumer trust, I suggest a more resilient approach:
Implementation of a Heartbeat Mechanism: Decouple billing from ephemeral container logs. The orchestrator should track periodic state transitions.
State Persistence: Store these transitions in a durable, external registry (Ocean’s job registry) to survive node restarts or container purges.
Fallback Logic: Use a hybrid strategy: billingTime = min(container.finishedAt, lastRecordedHeartbeat).
This ensures that even in "worst-case" scenarios, node operators are not overcompensated for downtime, and users are billed only for verifiable compute time.
I’d be happy to provide a pseudocode implementation for this heartbeat logic if the team finds this direction valuable.