Main firmware needs to be robust to any accelerometer thread failures

[lauralindzey]

This issue is meant to document the implemented error handling and how I've tested it. The goal is to convince myself that the accel thread can't bring down the main control thread, regardless of what the expansion board does or doesn't do.

There are a few ways I've found for a misbehaving sensor thread to derail the control thread:

• if one of the calls to tx_thread_resume fails because the thread has already been terminated, and the control thread winds up in the Error_Handler() loop.
• if somehow the accelerometer thread neither reports failure nor success, leading to the iridium message never being sent

For each failure, list how it would appear to the control thread, what should happen, and how to test.

• initialization failure (while control thread is waiting for all init_flags to be set)

  • manifestation: ACCELEROMETER_INIT_SUCCESS not in controller_self->init_flags and tx_wait_ticks is 0.p
  • handling: accelerometer thread disabled for duty cycle; everything else runs as normal
    • disabling it is done by setting controller_self->thread_status.accelerometer_complete = true, which is what would be set if the accelerometer wasn't enabled in the configuration step (I'm not sure this is ideal; maybe simply setting ACCELEROMETER_INIT_FAILED is the right way to do it? Need to implement accel error handling in controller.)
  • testing: unplug board or flash with firmware that doesn't respond to the self-test message; confirm message printed to LOG and that rest of threads run through to iridium transmission.

• Accelerometer thread times out (via its own timer)

• Accelerometer thread reports failure (invalid response from the accel board?)

• Accelerometer thread never completes

  • The current setup only forces a reset when the overall duty cycle has finished, rather than with a margin that allows the iridium thread to make an attempt. I don't love this because it means that if there's a pervasive bug in a sensor thread, we'll never see data.

[lauralindzey]

Before deploying micros in the field, we tested:

• power on microswift with accelerometer disconnected -- it fails self test, and is not started or waited for.
• power off accelerometer while main board is expecting data from it -- the UART read times out reporting failure, and the rest of the sensors queue messages

I did not add an additional timer in the accel thread; the timeout on the uart read covers that well enough.