More robust handling of missing uptime events during farmerbot wakeups

[scottyeager]

In the current flow of minting, nodes managed by the farmerbot must submit at least one uptime report during their wake ups, in order for the minting algorithm to detect and register a wake up. The problem is that nodes sometimes fail to submit an uptime report, even though they have booted up as requested and even set their power state to Up. When this happens, the state tracked by minting regarding the node diverges from reality and the normal sequence of events is eventually interpreted as a violation.

Here's an example of such a sequence of events:

# Node is entering standby normally
power_managed: None power_manage_boot: None
PowerStateChanged(state='Down', timestamp=1721033508)
power_managed: 1721033508 power_manage_boot: None
NodeUptimeReported(uptime=2304, timestamp=1721033514)
power_managed: 1721033508 power_manage_boot: None

# Wake up is triggered
PowerTargetChanged(target='Up', timestamp=1721054940)
power_managed: 1721033508 power_manage_boot: 1721054940
# Here's where the uptime event is missing
PowerStateChanged(state='Up', timestamp=1721055780)
power_managed: 1721033508 power_manage_boot: 1721054940
PowerTargetChanged(target='Down', timestamp=1721057664)
power_managed: 1721033508 power_manage_boot: 1721054940
PowerStateChanged(state='Down', timestamp=1721057670)
power_managed: 1721033508 power_manage_boot: 1721054940
NodeUptimeReported(uptime=2166, timestamp=1721057676)
power_managed: None power_manage_boot: None
# Node is now standby, but power_managed is not set

# Next wake up
PowerTargetChanged(target='Up', timestamp=1721117088)
power_managed: None power_manage_boot: 1721117088
# Uptime event missing again
PowerStateChanged(state='Up', timestamp=1721118108)
power_managed: None power_manage_boot: 1721117088
PowerTargetChanged(target='Down', timestamp=1721120118)
power_managed: None power_manage_boot: 1721117088
PowerStateChanged(state='Down', timestamp=1721120124)
power_managed: 1721120124 power_manage_boot: 1721117088
NodeUptimeReported(uptime=2469, timestamp=1721120130)
power_managed: 1721120124 power_manage_boot: 1721117088
# Boot request was not cleared, so it looks like node is still booting up when actually it's shutting down
# This node boots again sometime after 30 minutes from the current power_manage_boot and is assigned a violation

Of course Zos should be responsible for generating the proper sequence of events here, but the fact is that sometimes it doesn't. Summary of that issue is that new health checks implemented in Zos delayed uptime reports from being sent when farmerbot controlled nodes woke up, but most importantly, the implementation caused many nodes to generate the invalid sequence of events I showed an example of above.

Beyond that, it's also the case that transactions submitted to tfchain are sometimes simply not present in the finalized chain (that's actually first how I observed the sequence of events presented above, but we are now seeing many more examples due to the new issue in Zos).

So my thinking here is that we can make minting better able to handle cases like this by also registering a wake up when the node changes its power state to Up.

[scottyeager]

Since the health checks were disabled in Zos shortly after I wrote this issue, we mostly have a large bulk of events for July to address.

What I'm hoping we can do is issue a fixup payment with the next round of minting.

[LeeSmet]

To address the immediate issue, a fixup payment is something which requires a GEP vote for approval. I'm very much not a fan of that, but that's a discussion for another place.

The bigger problem, on a technical level, is that we can't simply ignore events, because the relevant events are missing. Simply using the power up event is not really a solution since it doesn't carry uptime info, and so we don't know when the node booted. This will lead to situations where a node which boots correctly might get a violation for exceeding the 30 minute boot window, or not getting uptime anyway because it exceeds the 24 hour window. But in the case where this still corrects the problem, we now create a situation where a farmerbot managed node gets an additional chance to register a wakeup, which is not the case for "regular" nodes (and these are also affected and can drop uptime). In essence, sleeping nodes get an advantage, which is very much against what was initially discussed for farmerbot

[scottyeager]

I appreciate the points here. I do see the case for not issuing a fixup, since this is a Zos issue and not a minting issue pe se, but indeed that's a conversation to have elsewhere.

Simply using the power up event is not really a solution since it doesn't carry uptime info, and so we don't know when the node booted. This will lead to situations where a node which boots correctly might get a violation for exceeding the 30 minute boot window, or not getting uptime anyway because it exceeds the 24 hour window.

This is true. To be more precise, what I had in mind is an either/or implementation where both uptime events and power state changes to Up could be considered as a wakeup.

After some further consideration, I agree it's not great to try to compensate for the misbehavior of the node or the loss of events from tfchain by adding another condition for granting the node a wakeup.

That said, I still think there's something we can try to fix here. The fact that these nodes are later assigned a violation on their next boot cycle is an implementation detail. To me there should be a clear distinction—either the correct sequence of events occurred and the node is considered to have woken up, or it didn't. In the latter case, a violation should be assigned at that time.

So if the goal instead is to correctly attach the violation to the wakeup sequence where the issue occurred, an approach could be to only register a wakeup when the power target is set to Up. Then the final uptime report sent by the node before going to standby could not be mistaken for a wakeup.

[scottyeager]

I found another approach that I think addresses your concerns @LeeSmet and also makes the minting algorithm more robust.

Here's the outline:

1. When registering a wakeup (ie, uptime event is processed while both power_managed and power_managed_boot are set) check if the power target is Up or Down
2. If the power target is Up, proceed as usual. This is the expected sequence of events
3. If the power target is Down, set power_managed to the timestamp of the uptime event, instead of setting it to None

If it's a concern that power_managed would generally be six seconds greater in the case of the dropped uptime event, it could be set to the time of the last change of the power state to Down. I don't think giving a node an extra six seconds to hit the 24 hour limit on wakeups is a big deal though.

You could argue that these nodes shouldn't be getting a wakeup registered when they have produced an invalid sequence of events. Let's remember though that it's the status quo that the node gets a wakeup when it is shutting down, if it skipped the initial uptime event on boot. The farmer can actually avoid getting a violation later by performing a manual intervention involving boot cycling the node before the next wakeup, such that the state that minting tracks for the node is reset to a valid combination.

As for giving farmerbot nodes an extra chance, versus regular nodes that can also drop uptime events, I think it's not unreasonable because the stakes are higher for farmerbot nodes on each event. A regular node loses 40 minutes of uptime when an event is dropped, which means for a DIY node 54 events can be dropped before the node drops below the uptime threshold solely for that reason.

[scottyeager]

I wanted to detail what I meant regarding the manual intervention that can save nodes affected by this. The farmer can, only by performing normal operations of setting power target events on chain, clear the invalid state and return the node to a normal event sequence from the perspective of minting.

Here's an example of how that would look, based on the example event sequence from my first post on this issue:

NodeUptimeReported(uptime=u0, timestamp=t0)
power_managed: None power_manage_boot: None
# Node is now standby, but power_managed is not set

# Farmer triggers first manual wakeup
PowerTargetChanged(target='Up', timestamp=t1)
power_managed: None power_manage_boot: t1
NodeUptimeReported(uptime=u1, timestamp=t2)
# power_manage_boot is not cleared here because power_managed is not set
power_managed: None power_manage_boot: t1
PowerStateChanged(state='Up', timestamp=t3)
power_managed: None power_manage_boot: t1
PowerTargetChanged(target='Down', timestamp=t4)
power_managed: None power_manage_boot: t1
PowerStateChanged(state='Down', timestamp=t5)
power_managed: t5 power_manage_boot: t1
NodeUptimeReported(uptime=u2, timestamp=t6)
# Because t6 - u2 < t5, this doesn't register as a wakeup
power_managed: t5 power_manage_boot: t1

# From here, the farmer has 30 minutes from t1 to boot the node up again

# Farmer triggers second manual wakeup
PowerTargetChanged(target='Up', timestamp=t7)
# Second boot request is ignored
power_managed: t5 power_manage_boot: t1
NodeUptimeReported(uptime=u3, timestamp=t8)
power_managed: None power_manage_boot: None
# Now a boot has been detected and we are back in a normal state

The thing I've just realized now though, is that this only works if the node submits an uptime event normally at boot on the second manual wakeup (for the first it doesn't actually matter). So for the situation where Zos is consistently preventing the node from creating the correct sequence of events, this workaround is actually not relevant. It can definitely work in the case of isolated incidents, such as when events are dropped from tfchain.