[docker-sonic-vs] Restore ASAN VS image as a healthy sonic-swss vstest base (libasan8, supervisor contract, VS orchagent shims) by securely1g · Pull Request #28108 · sonic-net/sonic-buildimage

securely1g · 2026-06-26T09:08:35Z

Why I did it

This PR restores docker-sonic-vs as a healthy base image for the sonic-swss DVS (vstest) suite. It fixes the ASAN-linked syncd runtime regression and a set of related VS composition/startup/config issues that surfaced once the VS image was rebuilt from the current master sources. Grouped by area:

1. ASAN syncd runtime is missing (libasan.so.8)

The VS container (docker-sonic-vs) base is Debian bookworm (gcc-12), whose AddressSanitizer runtime is provided by libasan8 (libasan.so.8). platform/vs/docker-sonic-vs/Dockerfile.j2 installed libasan6 (gcc-10 / buster–bullseye era), which does not exist in bookworm repos, so the apt-get install silently failed to provide the ASAN runtime.

This was latent until the VS syncd became ASAN-linked. With ENABLE_ASAN=y, the syncd binary requires libasan.so.8 at runtime. Because only libasan6 was requested, the image shipped no libasan.so.8:

# ldd /usr/bin/syncd
        libasan.so.8 => not found

syncd exits 127 and supervisord drives it to FATAL after retries. With syncd down there is no ASIC_DB, which breaks the entire sonic-swss DVS (vstest) suite that runs against docker-sonic-vs. (This is the regression that forced sonic-swss to pin its vstest base image; see sonic-net/sonic-swss commit 8259676a.)

Every other bookworm container in this repo already pins libasan8==12.2.0-14+deb12u1 (e.g. platform/vs/docker-syncd-vs/buildinfo/versions/versions-deb); only this hand-written literal was stale.

2. ASAN image not actually produced / cache-poisoned as ASAN

The PR VS pipeline (azure-pipelines.yml, BuildVS stage) invoked the build template with an inline vs jobGroup that carried no variables, so asan_image kept the top-level default no and the ENABLE_ASAN=y branch was skipped. Nightly (official-build.yml) falls through to the template default where vs sets asan_image: yes, which is why only nightly produced docker-sonic-vs-asan.gz. Now the PR vs jobGroup sets asan_image: yes too.
syncd-vs/libsairedis (and derived libsaivs/libsaimetadata/dbgsym) are cache-keyed via *_DEP_FLAGS without $(ENABLE_ASAN), and ENABLE_ASAN is not in SONIC_COMMON_FLAGS_LIST, so toggling it did not invalidate their rcache entries. An ASAN-built syncd-vs deb could then be restored into a non-ASAN build, shipping an ASAN-linked syncd on a rootfs with no libasan.so.8. Fixed by appending $(ENABLE_ASAN) to the cache-key DEP_FLAGS of syncd-vs and libsairedis, mirroring rules/syncd.dep.

3. VS supervisor / startup contract for vstest

start.sh contract: the sonic-swss vstest harness (tests/conftest.py) requires supervisor program start.sh to reach EXITED. The recomposed VS image used [program:start]; this restores the start.sh supervisor contract and updates dependent-startup waits to start.sh:exited.
portsyncd ordering: portsyncd was inherited from the shared docker-orchagent template with dependent_startup_wait_for=rsyslogd:running, which is too weak for the composed VS image — it could start before ConfigDB PORT data was loaded (seen as DPU port-init instability / HOSTIF readiness failures in DPU-2P/DASH vstests). Parameterized the shared SWSS template (portsyncd_dependent_startup_wait_for, default rsyslogd:running) and overrode it to syncd:running only in the VS image.
gearsyncd gating: on non-gearbox HWSKUs gearsyncd SIGABRTs with no gearbox_config, and the core dump trips the chassis crash gate. Added a gearsyncd_dependent_startup guard (default true) in the shared template and set it false only for the VS image, so VS does not auto-start gearsyncd. Non-VS dockers are unchanged.

4. VS ASIC_VENDOR / orchagent path selection for vstest

ASIC_VENDOR=vs: set once via image ENV so every daemon (buffermgrd, orchagent/neighorch) and any supervisorctl restart sees it under the DVS/vstest harness (which starts docker-sonic-vs directly, not via docker_image_ctl.sh). Also dropped the local buffermgrd.sh, now provided by docker-orchagent via the shared build-context include (the local copy had diverged and was missing peripheral_table.json handling).
DASH northbound ZMQ off on VS: orchagent.sh unconditionally appended -q tcp://127.0.0.1 for non-SmartSwitch platforms (from [Orchagent] Enable ZMQ on orchagent route table. #23028), routing DASH orchs to ZMQ instead of redis DPU_APPL_DB (db 15). The swss DASH DVS tests feed config via ProducerStateTable into db 15, so those writes were never ingested (SAI_OBJECT_TYPE_VIP_ENTRY never created). Skip -q when asic_type==vs; real SmartSwitch/DPU hardware still enables ZMQ.
orchagent.sh VS platform shims: VS-only, all no-ops on real hardware — (a) restore the voq empty-asic_type → platform=vs fallback so VS-only checks fire; (b) HWSKU=Mellanox-SN2700 → platform=mellanox for the mirror_ipv6_separate vstest (separate v4 MIRROR + v6 MIRRORV6 tables); (c) honor DVS_ENV ASIC_TYPE=broadcom-dnx and PFC_DLR_INIT_ENABLE so orchagent selects the broadcom-dnx shared-egress-ACL path and PfcWdAclHandler. Shims (b)/(c) are guarded by if [ "$platform" == "vs" ].

How I did it

platform/vs/docker-sonic-vs/Dockerfile.j2: libasan6 → libasan8 (gated on ENABLE_ASAN == "y"); set ASIC_VENDOR=vs via ENV.
platform/vs/docker-sonic-vs/supervisord.conf.j2: restore start.sh program/contract; override portsyncd_dependent_startup_wait_for=syncd:running; set gearsyncd_dependent_startup=false.
dockers/docker-orchagent/supervisord.conf.common.j2: parameterize portsyncd_dependent_startup_wait_for and add gearsyncd_dependent_startup guard (defaults keep standalone docker-orchagent unchanged).
orchagent.sh: skip northbound DASH -q on asic_type==vs; add VS-only voq/Mellanox/broadcom-dnx/PFC_DLR platform shims.
Drop redundant local platform/vs/docker-sonic-vs/buffermgrd.sh (provided by docker-orchagent include).
Build/cache: append $(ENABLE_ASAN) to syncd-vs/libsairedis cache-key DEP_FLAGS; set asan_image: yes on the PR vs jobGroup in azure-pipelines.yml.

How to verify it

ASAN syncd runtime, on the affected master docker-sonic-vs ASAN image:

Before: ldd /usr/bin/syncd → libasan.so.8 => not found; supervisorctl status syncd → FATAL (exit 127); whole vstest suite aborts (no ASIC_DB).
After: ldd resolves, supervisorctl start syncd → RUNNING and stays up, ASIC_DB initializes.
A/B against the last known-good master base (build 1141167) confirms it ships libasan8 and syncd is RUNNING there.

Local DVS validation (2026-06-26), Ubuntu 22.04 runner container with docker.sock, /lib/modules, and /var/run/redis-vs mounted, host team module loaded:

BROKEN bb0715a3f:     syncd=FATAL, start=EXITED
libasan8 only:        syncd=RUNNING, start=EXITED, but vstest waits for start.sh
libasan8+start.sh:    syncd=RUNNING, start.sh=EXITED, required daemons RUNNING

Focused DVS smoke against the combined runtime delta:

python3 -m pytest -sv --imgname=docker-sonic-vs:bb0715a3f-libasan8-startsh-waits \
  test_vlan.py::TestVlan::test_VlanAddRemove

======================== 1 passed, 5 warnings in 30.30s ========================

DASH DVS, on a 1152313-based image with libasan8 present: with the ZMQ change dash/test_dash_acl.py goes from 1 passed / 12 ERROR to 13 passed; the running DUT shows orchagent started without -q and ASIC VIP_ENTRY=1.

The from-source local image build is still blocked in the dirty csonic2 workspace by an unrelated scapy/python-wheel versions-lock issue (Invalid requirement: =). To avoid conflating that with this fix, the runtime deltas were validated by constructing images from the broken official image and applying only the package/template deltas above.

Once a fixed ASAN image build is available, sonic-swss can revert its vstest pin (runId: 1141167) back to latestFromBranch.

Description for the changelog

[docker-sonic-vs] Restore the ASAN VS image as a healthy sonic-swss vstest base: install libasan8 (bookworm) so the ASAN-linked syncd has its runtime and stops FATAL-ing (exit 127); include ENABLE_ASAN in syncd-vs/sairedis cache keys and enable the ASAN VS image in the PR pipeline; preserve the start.sh supervisor contract; order portsyncd after syncd and gate gearsyncd off for VS; set ASIC_VENDOR=vs; and add VS-only orchagent.sh shims (DASH-on-redis, voq/Mellanox/broadcom-dnx/PFC_DLR) for the DVS suite. All shared-template and orchagent changes are no-ops on real hardware / standalone docker-orchagent.

mssonicbld · 2026-06-26T09:08:43Z

/azp run Azure.sonic-buildimage

azure-pipelines · 2026-06-26T09:08:55Z

Azure Pipelines successfully started running 1 pipeline(s).

mssonicbld · 2026-06-26T10:10:11Z

/azp run Azure.sonic-buildimage

azure-pipelines · 2026-06-26T10:10:24Z

Azure Pipelines successfully started running 1 pipeline(s).

mssonicbld · 2026-06-27T01:36:51Z

/azp run Azure.sonic-buildimage

azure-pipelines · 2026-06-27T01:37:05Z

Azure Pipelines successfully started running 1 pipeline(s).

mssonicbld · 2026-06-27T18:32:57Z

/azp run Azure.sonic-buildimage

azure-pipelines · 2026-06-27T18:33:08Z

Azure Pipelines successfully started running 1 pipeline(s).

buffermgrd.sh is now copied from docker-orchagent via the shared build-context include (COPY --from=orchagent). The local copy in docker-sonic-vs was stale (diverged from orchagent's version, missing peripheral_table.json handling) and is no longer used, so remove it. Addresses review feedback on PR sonic-net#28108. Signed-off-by: securely1g <securely1g@users.noreply.github.com>

mssonicbld · 2026-06-27T20:10:58Z

/azp run Azure.sonic-buildimage

azure-pipelines · 2026-06-27T20:11:10Z

Azure Pipelines successfully started running 1 pipeline(s).

lguohan · 2026-06-28T02:59:12Z

merge last two commits.

mssonicbld · 2026-06-28T03:02:11Z

/azp run Azure.sonic-buildimage

securely1g · 2026-06-28T03:02:18Z

Squashed the last two commits (ASIC_VENDOR env + buffermgrd.sh removal) into a single commit 4d4b83663, and dropped the now-redundant inline comment as requested. Branch is force-pushed (force-with-lease).

azure-pipelines · 2026-06-28T03:02:25Z

Azure Pipelines successfully started running 1 pipeline(s).

securely1g · 2026-06-28T03:02:26Z

/azpw retry

mssonicbld · 2026-06-28T03:03:25Z

Retrying failed(or canceled) jobs...

mssonicbld · 2026-06-28T03:03:28Z

No failed(or canceled) stages or jobs found in the most recent build 1150942.

lguohan · 2026-06-28T19:10:40Z

/azpw retry

mssonicbld · 2026-06-28T19:10:42Z

Retrying failed(or canceled) jobs...

mssonicbld · 2026-07-01T07:15:10Z

/azp run Azure.sonic-buildimage

azure-pipelines · 2026-07-01T07:15:22Z

Azure Pipelines successfully started running 1 pipeline(s).

mssonicbld · 2026-07-01T07:25:20Z

/azp run Azure.sonic-buildimage

azure-pipelines · 2026-07-01T07:25:32Z

Azure Pipelines successfully started running 1 pipeline(s).

mssonicbld · 2026-07-01T07:39:57Z

/azp run Azure.sonic-buildimage

azure-pipelines · 2026-07-01T07:40:11Z

Azure Pipelines successfully started running 1 pipeline(s).

mssonicbld · 2026-07-01T07:42:21Z

/azp run Azure.sonic-buildimage

azure-pipelines · 2026-07-01T07:42:33Z

Azure Pipelines successfully started running 1 pipeline(s).

orchagent.sh VS-only platform overrides so sonic-swss vstests select the right ASIC path on the virtual switch; all no-ops on real hardware. Guard all of these shims on ASIC_VENDOR=vs, which is set once via the docker-sonic-vs image ENV. That is a stable "this is the Virtual Switch" identity even when .asic_type from swss_vars is empty (e.g. voq linecard roles), and it is never set on real hardware, so none of the overrides below can fire off-VS. Inside that block, platform is pinned to vs first (restoring the legacy orchagent.sh fallback for the empty-asic_type voq role) and may then be reassigned per test: 1. HWSKU=Mellanox-SN2700 -> platform=mellanox: the swss mirror_ipv6_separate vstest pins HWSKU=Mellanox-SN2700 and expects the Mellanox split-table mirror layout (separate v4 MIRROR + v6 MIRRORV6 tables), which aclorch selects from platform=mellanox. 2. broadcom-dnx / PFC_DLR VS env shims: honor DVS_ENV ASIC_TYPE=broadcom-dnx (export platform=broadcom, sub_platform=broadcom-dnx) and PFC_DLR_INIT_ENABLE (export pfcDlrInitEnable) so orchagent selects the broadcom-dnx shared-egress-ACL path and PfcWdAclHandler on the VS switch (tests/test_pfcwd_shared_egress_acl_table.py). Because the single guard keys off ASIC_VENDOR (not the mutated platform), the SN2700/broadcom-dnx reassignments cannot leak the shims onto non-VS platforms even if the triggering HWSKU/env vars happen to be present. Signed-off-by: securely1g <securely1g@users.noreply.github.com>

Add a gearsyncd_dependent_startup guard (default true) in docker-orchagent/supervisord.conf.common.j2 wrapping only gearsyncd's dependent_startup lines, and set it false in platform/vs/docker-sonic-vs/supervisord.conf.j2 before the swss include, so the VS image does not auto-start gearsyncd via the dependent-startup plugin. On non-gearbox HWSKUs gearsyncd SIGABRTs with no gearbox_config, and the resulting core dump trips the chassis crash gate. Non-VS dockers keep the default (true), so their gearsyncd startup behavior is unchanged. Signed-off-by: securely1g <securely1g@users.noreply.github.com>

The VS syncd deb (syncd-vs) and libsairedis (and its derived SAI libs: libsaivs, libsaimetadata, ...) are cache-keyed via *_DEP_FLAGS without $(ENABLE_ASAN). ENABLE_ASAN is not part of SONIC_COMMON_FLAGS_LIST, so toggling it does not change the GIT_CONTENT_SHA cache key for these packages and does not invalidate their rcache entries. As a result, a syncd-vs/libsairedis deb built once with ENABLE_ASAN=y is restored from cache into a later ENABLE_ASAN=n build. The shipped docker-sonic-vs then contains an ASAN-instrumented syncd (NEEDED libasan.so.8, __asan_* symbols) while the rootfs is a non-ASAN image that does not ship libasan.so.8, so syncd fails to start: /usr/bin/syncd: error while loading shared libraries: libasan.so.8: cannot open shared object file: No such file or directory Observed on official build 1153324 (ENABLE_ASAN=n banner, yet [ cached ] libsairedis and an ASAN-linked syncd-vs deb). Fix: append $(ENABLE_ASAN) to the cache-key DEP_FLAGS of syncd-vs and libsairedis, mirroring rules/syncd.dep. Setting it on the LIBSAIREDIS parent covers its derived packages (libsaivs, libsaimetadata, dbgsym) which share the same source build. Signed-off-by: securely1g <securely1g@users.noreply.github.com>

The PR pipeline (azure-pipelines.yml, BuildVS stage) invokes azure-pipelines-build.yml with an inline jobGroups entry for 'vs' that carried no variables, so asan_image kept the top-level default 'no' and the ENABLE_ASAN=y branch in the 'Build sonic image' step was skipped. The nightly VS pipeline (official-build.yml) passes no jobGroups and therefore falls through to the template's default jobGroups block, where the vs group sets asan_image: yes -- which is why nightly produces target/docker-sonic-vs-asan.gz but PR builds did not. Set asan_image: yes on the vs jobGroup so PR VS builds also produce docker-sonic-vs-asan.gz (published via the existing 'mv target/*' step). Signed-off-by: securely1g <securely1g@users.noreply.github.com>

mssonicbld · 2026-07-01T18:53:32Z

/azp run Azure.sonic-buildimage

azure-pipelines · 2026-07-01T18:53:46Z

Azure Pipelines successfully started running 1 pipeline(s).

securely1g requested a review from lguohan as a code owner June 26, 2026 09:08

securely1g changed the title ~~[docker-sonic-vs] Fix ASAN build: install libasan8 (bookworm) instead of libasan6~~ [docker-sonic-vs] Fix ASAN syncd runtime and preserve start.sh contract Jun 26, 2026

securely1g mentioned this pull request Jun 26, 2026

Revert "[ci]: Pin sonic-buildimage artifact to build 1141167 (#4700)" sonic-net/sonic-swss#4712

Draft

securely1g force-pushed the fix/vs-syncd-libasan8 branch from d6b1820 to fde71a9 Compare June 27, 2026 18:32

lguohan reviewed Jun 27, 2026

View reviewed changes

Comment thread platform/vs/docker-sonic-vs/buffermgrd.sh

securely1g mentioned this pull request Jun 27, 2026

[ci] Use sonic-buildimage build 1154037 for docker-sonic-vs in vstest sonic-net/sonic-swss#4713

Closed

lguohan reviewed Jun 27, 2026

View reviewed changes

Comment thread platform/vs/docker-sonic-vs/Dockerfile.j2 Outdated

lguohan reviewed Jun 28, 2026

View reviewed changes

Comment thread platform/vs/docker-sonic-vs/Dockerfile.j2 Outdated

securely1g force-pushed the fix/vs-syncd-libasan8 branch from aeb134d to 4d4b836 Compare June 28, 2026 03:02

lguohan previously approved these changes Jun 28, 2026

View reviewed changes

securely1g force-pushed the fix/vs-syncd-libasan8 branch from 9c5f938 to f5b6d4d Compare July 1, 2026 07:15

securely1g force-pushed the fix/vs-syncd-libasan8 branch from f5b6d4d to 6e915a6 Compare July 1, 2026 07:25

securely1g force-pushed the fix/vs-syncd-libasan8 branch from 6e915a6 to c4de16f Compare July 1, 2026 07:39

lguohan previously approved these changes Jul 1, 2026

View reviewed changes

theasianpianist previously approved these changes Jul 1, 2026

View reviewed changes

securely1g changed the title ~~[docker-sonic-vs] Fix ASAN syncd runtime and preserve start.sh contract~~ [docker-sonic-vs] Restore ASAN VS image as a healthy sonic-swss vstest base (libasan8, supervisor contract, VS orchagent shims) Jul 1, 2026

securely1g added 4 commits July 1, 2026 11:53

securely1g dismissed stale reviews from theasianpianist and lguohan via 52d6953 July 1, 2026 18:53

securely1g force-pushed the fix/vs-syncd-libasan8 branch from d003f07 to 52d6953 Compare July 1, 2026 18:53

lguohan approved these changes Jul 1, 2026

View reviewed changes

deepak-singhal0408 mentioned this pull request Jul 1, 2026

Add HSET support to AsyncDBUpdater sonic-net/sonic-swss-common#1217

Merged

lguohan merged commit e583550 into sonic-net:master Jul 2, 2026
29 checks passed

Uh oh!

Conversation

securely1g commented Jun 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why I did it

How I did it

How to verify it

Description for the changelog

Uh oh!

mssonicbld commented Jun 26, 2026

Uh oh!

azure-pipelines Bot commented Jun 26, 2026

Uh oh!

mssonicbld commented Jun 26, 2026

Uh oh!

azure-pipelines Bot commented Jun 26, 2026

Uh oh!

mssonicbld commented Jun 27, 2026

Uh oh!

azure-pipelines Bot commented Jun 27, 2026

Uh oh!

mssonicbld commented Jun 27, 2026

Uh oh!

azure-pipelines Bot commented Jun 27, 2026

Uh oh!

Uh oh!

mssonicbld commented Jun 27, 2026

Uh oh!

azure-pipelines Bot commented Jun 27, 2026

Uh oh!

Uh oh!

Uh oh!

lguohan commented Jun 28, 2026

Uh oh!

mssonicbld commented Jun 28, 2026

Uh oh!

securely1g commented Jun 28, 2026

Uh oh!

azure-pipelines Bot commented Jun 28, 2026

Uh oh!

securely1g commented Jun 28, 2026

Uh oh!

mssonicbld commented Jun 28, 2026

Uh oh!

mssonicbld commented Jun 28, 2026

Uh oh!

lguohan commented Jun 28, 2026

Uh oh!

mssonicbld commented Jun 28, 2026

Uh oh!

mssonicbld commented Jul 1, 2026

Uh oh!

azure-pipelines Bot commented Jul 1, 2026

Uh oh!

mssonicbld commented Jul 1, 2026

Uh oh!

azure-pipelines Bot commented Jul 1, 2026

Uh oh!

mssonicbld commented Jul 1, 2026

Uh oh!

azure-pipelines Bot commented Jul 1, 2026

Uh oh!

mssonicbld commented Jul 1, 2026

Uh oh!

azure-pipelines Bot commented Jul 1, 2026

Uh oh!

mssonicbld commented Jul 1, 2026

Uh oh!

azure-pipelines Bot commented Jul 1, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

securely1g commented Jun 26, 2026 •

edited

Loading