Skip to content

Work around flakiness by periodically issuing redundant write commands to fan #35

@Krellan

Description

@Krellan

To avoid flooding the D-Bus interface with traffic, there is an optimization within phosphor-pid-control that avoids issuing a write command to the fan, if the commanded PWM is the same value as it was from earlier. In other words, it tries to avoid writing duplicate PWM that would have no effect.

This is a good optimization, as without it D-Bus can be slowed down so much as to be unusable. However, it needs to be overridden in certain situations, such as when returning to automatic PID control from manual, and thus needing all the fans to receive a new PWM command to be sure they return to what the PID algorithm believes they should be at. This was implemented a while ago: https://gerrit.openbmc.org/c/openbmc/phosphor-pid-control/+/36695

However, it appears this optimization might be too aggressive. It would be worth adding another situation in which it should be overridden. Sometimes, the write path is not 100% reliable, and a fan write command can become lost. When this happens, the fan PWM will not receive the command, so it will remain "stuck" at an old PWM, without changing to the new PWM as it should have been commanded to do.

Various OEM have came up with their own patches to allow for this possibility, to avoid the problem with a fan getting "stuck" at the wrong PWM, by allowing fan writes to happen, regardless if they are redundant or not. Here's an example: https://gerrit.openbmc.org/c/openbmc/phosphor-pid-control/+/37357

I am thinking that a reasonable compromise would be to allow a redundant PWM write only once every X cycles through the main loop, or every Y seconds of elapsed time, whatever would be the most reasonable to implement. This would provide enough opportunities to get the PWM command through, without bogging down the rest of the system by spamming D-Bus needlessly.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions