test: create integration workspace with bitbake-setup by gcomneno · Pull Request #535 · yoctoproject/vscode-bitbake

gcomneno · 2026-06-20T17:23:20Z

Summary

Create the Poky integration workspace with bitbake-setup using the standard layout under integration-tests/project-folder.
Keep the generated setup close to the bitbake-setup defaults:
- integration-tests/project-folder/build
- integration-tests/project-folder/layers
- integration-tests/project-folder/config
Keep the BitBake bootstrap checkout under resources/bitbake for future bitbake-setup tests.
Add pinned bitbake-setup source overrides for the fetched layers.
Keep the build-crops wrapper/config as dedicated fixtures for the container-mounted command-wrapper test.

This follows the direction discussed in #530: first update the integration test workspace to use a bitbake-setup generated layout, without adding any new UX.

Testing

npm run fetch:poky
bash -n scripts/fetch-poky.sh
bash -n scripts/update-ref.sh
npm run compile
npm run lint
git diff --check

gcomneno · 2026-06-21T12:10:26Z

Follow-up: the latest commits fix the integration-test fallout from the bitbake-setup layout switch.

The CI failures were caused by a few remaining assumptions from the old Poky layout:

command-wrapper test now checks the generated output under sources/poky/build-crops
test recipe import/removal now targets layers/openembedded-core/meta
bash embedded-language support resolves global helper classes from both legacy Poky and bitbake-setup layouts
resources/site.conf now avoids host-specific absolute paths so the containerized tests can use the workspace-mounted paths

Both CI jobs are now green on 30e5b052.

deribaucourt

Great job, there were a few technical problems I see, but you found a solution for them all! This first iteration should have thus already addressed all the pain points. I added some comments to simplify the implementation and align it even further with what should be the standard bitbake-setup practices.

I'm also discovering bitbake-setup, so feel free to come up with suggestions of your own :)

I'll be on summer vacation next week 😎. Please accept my apologies if I only respond later.

deribaucourt · 2026-06-26T14:20:54Z


-const pathToBuildFolder = path.join(__dirname, '../../../../../integration-tests/project-folder/build')
-const pathToEnvScript = path.join(__dirname, '../../../../../integration-tests/project-folder/sources/poky/oe-init-build-env')
+const pathToBuildFolder = path.join(__dirname, '../../../../../integration-tests/project-folder/sources/poky/build')


build folder should not be under sources. We will use the sources directory only for our custom layers (meta-fixture, ...). Expected standard bitbake-setup directory structure is:

integration-tests/project-folder/build

integration-tests/project-folder/layers

Can you use this bitbake-setup argument to initialize the build folder in the following old path (integration-tests/project-folder/build)?

bitbake-setup --setting default top-dir-prefix integration-tests/project-folder --setting default top-dir-name integration-tests

I know it may be hard to do, but I would prefer if you rebase with the review fixes instead of pushing additional fixup commits on top.

deribaucourt · 2026-06-26T14:34:11Z

-    git clone --depth 1 --branch "$tag" "$url" "$destination"
-    git -C "$destination" checkout --detach "$commit"
+cleanup() {
+    rm -rf "$BOOTSTRAP_DIR"


No need to remove the bitbake bootstrap at the end of this script, npm run clean used to handle cleaning up anything downloaded with npm run fetch. We will be needing it for testing future bitbake-setup features.

deribaucourt · 2026-06-26T14:34:36Z

-YOCTO_DIR="resources/poky"
+RESOURCES_DIR="resources"
+YOCTO_DIR="$RESOURCES_DIR/poky"
+BOOTSTRAP_DIR="$RESOURCES_DIR/.bitbake-setup-bootstrap"


Let's use a simpler folder: resources/bitbake.

deribaucourt · 2026-06-26T14:42:02Z

+POKY_DISTRO="distro/poky"
+POKY_MACHINE="machine/qemux86-64"
+
+rm -rf "$YOCTO_DIR" "$BOOTSTRAP_DIR"


Similarly, cleanup is through npm run clean. Let's not touch this workflow for this PR.

deribaucourt · 2026-06-26T14:43:21Z

-
+trap cleanup EXIT
+
+POKY_CONFIGURATION="poky-wrynose"


We need a commit-pinned revision for the layers. We don't want to accidentally download an updated version and break our pipelines. We may need to define our own bitbake-setup json with pinned versions.

Handled with a pinned source-overrides.json fixture under integration-tests/fixtures/bitbake-setup/. The integration workspace now passes it to bitbake-setup with --source-overrides, so the fetched sources are pinned to explicit commits instead of branch heads.

deribaucourt · 2026-06-26T14:53:42Z


-YOCTO_DIR="resources/poky"
+RESOURCES_DIR="resources"
+YOCTO_DIR="$RESOURCES_DIR/poky"


I don't think Yocto layers should be downloaded to resources/poky anymore. The bitbake-setup standard setup is to put them under <top_dir>/layers. let's keep it as close to possible to standard.

Feel free to correct me if I'm wrong, I'm still new with bitbake-setup.

deribaucourt · 2026-06-26T14:55:14Z

 import fs from 'fs'

-export const integrationBitbakeFolder = path.join(__dirname, '../../../../integration-tests/project-folder/sources/poky/bitbake')
+export const integrationBitbakeFolder = path.join(__dirname, '../../../../integration-tests/project-folder/sources/poky/layers/bitbake')


There is a dedicated commit to update the test path expectations. These changes should be moved to "
test: update integration expectations for bitbake-setup layout"

deribaucourt · 2026-06-26T15:01:26Z

+# integration tests also run inside a container where the repository is mounted
+# at a different path, so keep these locations relative to the active build
+# directory.
+DL_DIR ?= "${TOPDIR}/../../.bitbake-setup-downloads"


This should be generated by bitbake-setup, can't it?

Note that in CI, the DL, HASHSERVE and SSTATE caches are in different locations that are provided through main.yml:

env: # NFS Yocto cache directories on the self-hosted runner DL_DIR: /var/cache/yocto/yocto-dldir SSTATE_DIR: /var/cache/yocto/yocto-sstate BB_ENV_PASSTHROUGH_ADDITIONS: "DL_DIR SSTATE_DIR"

BTW, can you add a commit to add BB_SIGNATURE_HANDLER=OEBasicHash?
This will fix this Wrynose CI warning:

[9502](https://github.com/yoctoproject/vscode-bitbake/actions/runs/28238811952/job/83660175776#step:13:19503) 12:50:40.466 [Debug] WARNING: Sstate directory is shared between several builds (it is set via SSTATE_DIR to /var/cache/yocto/yocto-sstate), but hash equivalency database is inside this particular build directory /var/github/github-runner/actions-runner/vscodeBitbake/_work/vscode-bitbake/vscode-bitbake/integration-tests/project-folder/build. This will prevent sstate reuse, and it is recommended to set the location for the database to a common path via BB_HASHSERVE_DB_DIR, for example: BB_HASHSERVE_DB_DIR = "${SSTATE_DIR}"

I did not add BB_SIGNATURE_HANDLER=OEBasicHash in this rework because the main build configuration is now generated by bitbake-setup, and I wanted to avoid reintroducing manual config edits in this PR. I can handle that as a focused follow-up if you prefer.

deribaucourt · 2026-06-26T15:02:35Z

+
+mkdir -p "$YOCTO_DIR/build-crops"
+cp -R "$YOCTO_DIR/build/conf" "$YOCTO_DIR/build-crops/conf"
+cat > "$YOCTO_DIR/build-crops/conf/bblayers.conf" <<'EOF'


Same remark as native tests environment: should be provided as a bitbake-setup config file.
Also avoid embedding files in scripts when avoidable. They are harder to maintain and develop.

deribaucourt · 2026-06-26T15:22:02Z

  insertTextIntoEmbeddedLanguageDoc(embeddedLanguageDoc, 0, 0, getBashHeader(embeddedLanguageDoc.originalUri))
 }

+const getOpenEmbeddedCoreMetaFolder = (pokyFolder: string): string => {


The extension always made the assumption that poky/ contained bitbake/. This is no longer standard with bitbake-setup. However, this was how most existing projects used to do it, and we don't want to break compatibility with those!

I propose to keep the old behavior as it was. We'll need a new mechanism for bitbake-setup layer structures. I could think of:

adding a new extension setting: pathToLayers. Even then, bitbake-setup exotic configs might allow defining different layer structures?

trying to automatically detect the setup based on the settings we have

Could you take a look at what bitbake-setup could result in, and what would be the safe and simple solution to support both the new and the old structures?

I kept the legacy behavior as the fallback and added a narrow detection path for the standard bitbake-setup layout. The helper class lookup now checks the old poky/meta structure first, then layers/openembedded-core/meta, and only uses a candidate if the expected global classes exist. This avoids introducing a new setting in this PR while keeping compatibility with existing projects.

gcomneno · 2026-06-26T15:37:10Z

Thanks a lot for the detailed review! I’ll keep an eye on the fort while you’re away.
No worries at all about the response time. Enjoy your summer vacation, and I hope you find a place with a merciful breeze!

gcomneno · 2026-07-01T19:55:33Z

I reworked the branch to make the integration workspace follow the standard bitbake-setup layout more closely:

integration-tests/project-folder/build
integration-tests/project-folder/layers
integration-tests/project-folder/config

The BitBake bootstrap is now kept under resources/bitbake instead of being removed by fetch-poky.sh, and generated files are cleaned through npm run clean.

I also added a pinned source-overrides.json fixture for the bitbake-setup sources and updated scripts/update-ref.sh so it refreshes those pinned revisions as well.

The main build configuration is now generated by bitbake-setup; only the build-crops wrapper/config is kept as a small fixture because that test runs through the container-mounted /workdir path.

Local checks pass:

bash -n scripts/fetch-poky.sh
bash -n scripts/update-ref.sh
npm run compile
npm run lint
git diff --check

gcomneno force-pushed the contrib/issue-530-bitbake-setup-workspace branch 3 times, most recently from 8b50cb3 to 2aa92a8 Compare June 21, 2026 10:03

deribaucourt requested changes Jun 26, 2026

View reviewed changes

test: create integration workspace with bitbake-setup

e959cb4

gcomneno force-pushed the contrib/issue-530-bitbake-setup-workspace branch from 30e5b05 to 4a87bbe Compare July 1, 2026 19:35

test: update integration expectations for bitbake-setup layout

7a9fb49

gcomneno force-pushed the contrib/issue-530-bitbake-setup-workspace branch 2 times, most recently from 8c5fce1 to c59af33 Compare July 1, 2026 19:40

gcomneno force-pushed the contrib/issue-530-bitbake-setup-workspace branch 2 times, most recently from 638a14d to a073798 Compare July 2, 2026 07:11

gcomneno added 3 commits July 2, 2026 11:12

test: align command wrapper output path with bitbake-setup layout

38f6069

test: import recipes into bitbake-setup openembedded-core layer

97ae719

server: resolve bash helper classes in bitbake-setup layout

e591fb6

gcomneno force-pushed the contrib/issue-530-bitbake-setup-workspace branch from a073798 to e591fb6 Compare July 2, 2026 09:12


		trap cleanup EXIT

		POKY_CONFIGURATION="poky-wrynose"

Uh oh!

Conversation

gcomneno commented Jun 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Testing

Uh oh!

gcomneno commented Jun 21, 2026

Uh oh!

deribaucourt left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gcomneno commented Jun 26, 2026

Uh oh!

gcomneno commented Jul 1, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

gcomneno commented Jun 20, 2026 •

edited

Loading