Improve race condition handling in featured daemon-reload and docker container startup

[arista-setu]

Issue:

This PR proposes an improvement for an open issue sonic-net/sonic-buildimage#20988.

The underlying issue has been observed on chassis but is rarely seen and has not been reproducible. However, thorough code review suggests a potential race condition may exist between featured daemon-reload and docker container start process which is addressed by this PR

Based on current code review, following procedure during boot could have causes this issue:

• The featured daemon loops through every feature and for each one writes a systemd config file, calls systemctl daemon-reload (which temporarily tears down cgroup directories)
• Then, docker container starts
• Here when daemon-reload fires for the next feature in loop ( here note that the container started in the previous iteration might still be initializing), it destroys the group path that the previous container's run process is trying to write its PID to, resulting in the "no such device" error.

https://github.com/sonic-net/sonic-host-services/blob/202405/scripts/featured#L227 > tiggers daemon-reload https://github.com/sonic-net/sonic-host-services/blob/202405/scripts/featured#L367

As a result, the docker start could fail as follow:

ERR container: docker cmd: start for eventd failed with 500 Server Error for http+docker://localhost/v1.43/containers/e6c2c0c3006061fd86d1736f8372ca4a10a7d8709d9a4c30c1da6d996576a2f0/start:
Internal Server Error ("failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: unable to apply cgroup configuration: failed to write 19294: write /sys/fs/cgroup/blkio/docker/e6c2c0c3006061fd86d1736f8372ca4a10a7d8709d9a4c30c1da6d996576a2f0/cgroup.procs: no such device: unknown")

Proposed Improvement:

The proposed fix will slightly change docker startup procedure. We could use 2 phase approach in sync_state_field : https://github.com/sonic-net/sonic-host-services/blob/202405/scripts/featured#L210) to avoid race condition between systemctl daemon-reload and Docker container startup:

• Phase 1 - write all systemd config files and trigger a single daemon-reload after all config is written
• Phase 2 - start all the containers

Testing:

The issue was not reproduced before this change during testing.

However, I've attempted 5-6 reboots on chassis with 202405 image with this particular change. The underlying issue was still not reproduces and no other side-effects of this change were seen. All docker containers started as expected.

NOTE : Since that issue was never reliably reproduced, and other systemd units (started by sonic.target) actually may or may not cause the same race condition. The proposed improvement in featured is just a defensive approach to eliminate possibility that it race condition is cause by featured

Backport request:

I would like to request to be included in 202405. A direct backport will not apply to 202405 branch and hence a separate PR against the 202405 branch is create : #388

[mssonicbld]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[mssonicbld]

Hi, there are workflow run(s) waiting for approval, you may be first-time contributor. I will notify maintainers to help approve once PR is approved. Thanks!

_{---Powered by SONiC BuildBot}

[anamehra]

we remove daemon-reload from this path but could there be additional tests added to validate the changed flow?