[202405] Fix race condition between featured daemon-reload and docker container start on chassis

[arista-setu]

Issue:

This PR proposes a fix for an open issue sonic-net/sonic-buildimage#20988 on 202405 branch for chassis

The underlying issue is seen only on chassis but it is rarely seen. It has been seen couple of times only and been not reproducible. I was not able unable to on our chassis as well. However, it seems like there exists a race condition between featured daemon-reload and docker container start process.

Based on current code review, following procedure during boot could have causes this issue:

The featured daemon loops through every feature and for each one writes a systemd config file, calls systemctl daemon-reload (which temporarily tears down cgroup directories)
Then, docker container starts
Here when daemon-reload fires for the next feature in loop ( here note that the container started in the previous iteration might still be initializing), it destroys the group path that the previous container's run process is trying to write its PID to, resulting in the "no such device" error.
https://github.com/sonic-net/sonic-host-services/blob/202405/scripts/featured#L227 > tiggers daemon-reload https://github.com/sonic-net/sonic-host-services/blob/202405/scripts/featured#L367

As a result, the docker start could fail as follow:

ERR container: docker cmd: start for eventd failed with 500 Server Error for http+docker://localhost/v1.43/containers/e6c2c0c3006061fd86d1736f8372ca4a10a7d8709d9a4c30c1da6d996576a2f0/start:
Internal Server Error ("failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: unable to apply cgroup configuration: failed to write 19294: write /sys/fs/cgroup/blkio/docker/e6c2c0c3006061fd86d1736f8372ca4a10a7d8709d9a4c30c1da6d996576a2f0/cgroup.procs: no such device: unknown")

Proposed Fix:

The proposed fix will slightly change docker startup procedure. We could use 2 phase approach in sync_state_field : https://github.com/sonic-net/sonic-host-services/blob/202405/scripts/featured#L210) to avoid race condition between systemctl daemon-reload and Docker container startup:

Phase 1 - write all systemd config files and trigger a single daemon-reload after all config is written
Phase 2 - start all the containers

Testing:

The issue was not reproduced before this change during testing.

However, I've attempted 5-6 reboots on chassis with 202405 image with this particular change. The underlying issue was still not reproduces and no other side-effects of this change were seen. All docker containers started as expected.

Backport request:

A PR with same changes is create against master : #387
Since that PR cannot be backported cleanly, this PR is open to include the change in 202405
If/when this PR merges, I request sonic-host-services submodule bump-up at https://github.com/Azure/sonic-buildimage-msft/tree/202405/src

[mssonicbld]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[mssonicbld]

Hi, there are workflow run(s) waiting for approval, you may be first-time contributor. I will notify maintainers to help approve once PR is approved. Thanks!

_{---Powered by SONiC BuildBot}