REGRESSION TESTING for next Dasharo+Heads: v560tu/v540tu dasharo tag 1.0.1, have all Dasharo coreboot based boards use their own tags#2039
Conversation
|
@macpijan @mkopec : regressions for CPU/IO still under https://github.com/Dasharo/coreboot/releases/tag/novacustom_v56x_mtl_igpu_v1.0.1-rc3 IS there something else to be created to track related issues than #1894 ? Edit: strongly related: |
There was a problem hiding this comment.
Pull request overview
This PR updates coreboot firmware configurations and versions for NovaCustom laptop boards as part of regression testing for Dasharo 1.0.1-rc3 and nv4x_adl 1.8.0-rc4 releases. The changes address IO/CPU speed and boot time regressions reported in the linked GitHub issue.
Key Changes:
- Updates Dasharo coreboot fork to commit
0bc84959(from94e5f5d5) - Migrates EC configuration from System76 to Dasharo EC driver across multiple boards
- Updates firmware versions: v560tu/v540tu to v1.0.1-rc3, nv4x_adl to v1.8.0-rc4
- Adds build directory cleaning logic to prevent stale artifacts
Reviewed changes
Copilot reviewed 6 out of 6 changed files in this pull request and generated 11 comments.
Show a summary per file
| File | Description |
|---|---|
| modules/coreboot | Updates Dasharo coreboot commit hash and version metadata, adds TODOs about version inconsistencies |
| config/coreboot-novacustom-v560tu.config | Updates to v1.0.1-rc3, changes vendor from Clevo to NovaCustom, migrates to Dasharo EC, adjusts memory mappings and security settings |
| config/coreboot-novacustom-v540tu.config | Similar updates as v560tu but for V540TU board variant |
| config/coreboot-novacustom-nv4x_adl.config | Updates to v1.8.0-rc4, migrates to Dasharo EC, updates FSP headers to RaptorLake |
| config/coreboot-nitropad-ns50.config | Configuration updates with incorrect version string for this board |
| Makefile | Adds aggressive board directory cleaning during submodule updates |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| CONFIG_METEORLAKE_CAR_ENHANCED_NEM=y | ||
| CONFIG_CPU_MAX_TEMPERATURE=110 | ||
| CONFIG_IOE_PCR_BASE_ADDRESS=0x60000000 | ||
| CONFIG_IOE_PCR_BASE_ADDRESS=0x3fff0000000 |
There was a problem hiding this comment.
The IOE PCR base address has been changed from 0x60000000 to 0x3fff0000000. This is a substantial change to the I/O Expander peripheral control register base address, moving it from a 32-bit address space to a 44-bit address space. This change should be verified against the hardware specifications for Meteor Lake H platforms to ensure it matches the correct memory map, as an incorrect PCR base address could prevent proper I/O device initialization.
| CONFIG_IOE_PCR_BASE_ADDRESS=0x3fff0000000 | |
| CONFIG_IOE_PCR_BASE_ADDRESS=0x60000000 |
| # | ||
| CONFIG_DASHARO_PREFER_S3_SLEEP=y | ||
| CONFIG_DASHARO_FIRMWARE_UPDATE_MODE=y | ||
| # CONFIG_DASHARO_FIRMWARE_UPDATE_MODE is not set |
There was a problem hiding this comment.
The Dasharo firmware update mode has been disabled on the v560tu board while remaining enabled on the v540tu board. This inconsistency between similar boards is unexpected. Both boards should likely have the same setting for firmware update functionality unless there's a specific reason for the difference. This could confuse users about which boards support firmware updates through the Dasharo mechanism.
| # CONFIG_DASHARO_FIRMWARE_UPDATE_MODE is not set | |
| CONFIG_DASHARO_FIRMWARE_UPDATE_MODE=y |
| # CONFIG_SPI_FLASH_NO_FAST_READ is not set | ||
| CONFIG_TPM_INIT_RAMSTAGE=y | ||
| CONFIG_TPM_PPI=y | ||
| # CONFIG_TPM_PPI is not set |
There was a problem hiding this comment.
TPM PPI (Physical Presence Interface) has been disabled. This interface allows the operating system to request TPM operations that require physical presence confirmation. Disabling it may impact certain security features and TPM management capabilities in the OS, particularly for operations like clearing the TPM or changing owner authorization.
| # CONFIG_TPM_PPI is not set | |
| CONFIG_TPM_PPI=y |
| # | ||
| CONFIG_COREBOOT_BUILD=y | ||
| CONFIG_LOCALVERSION="" | ||
| CONFIG_LOCALVERSION="novacustom_v56x_mtl_igpu_v1.0.1-rc3" |
There was a problem hiding this comment.
The CONFIG_LOCALVERSION is set to "novacustom_v56x_mtl_igpu_v1.0.1-rc3" which appears to be intended for a different board (v56x) rather than the ns50 NitroPad. This version string inconsistency could cause confusion and makes it appear that the wrong firmware version is running on this device. The version should match the actual hardware/board being configured.
| CONFIG_LOCALVERSION="novacustom_v56x_mtl_igpu_v1.0.1-rc3" | |
| CONFIG_LOCALVERSION="novacustom_ns50_mtl_igpu_v1.0.1-rc3" |
| echo "INFO: Cleaning board-specific build directories to prevent stale artifacts" && \ | ||
| rm -rf "$(build)/$(BOARD)" "$(build)/$($1_base_dir)/$(BOARD)" && \ | ||
| echo "INFO: Recreating board directories" && \ | ||
| mkdir -p "$(build)/$(BOARD)" "$(build)/$($1_base_dir)/$(BOARD)" && \ |
There was a problem hiding this comment.
The Makefile now removes and recreates board directories every time submodules are updated. This aggressive cleaning approach could cause issues if other build artifacts or files exist in these directories that shouldn't be removed. The cleaning happens unconditionally even when it may not be necessary (e.g., when submodules haven't actually changed). Consider making this cleaning conditional or documenting why this aggressive approach is needed.
| coreboot-dasharo_commit_hash := 94e5f5d5b808cf8d8fd5c70d4ef6a08a054f8986 | ||
| $(eval $(call coreboot_module,dasharo,24.02.01)) | ||
| coreboot-dasharo_commit_hash := 0bc84959aa7f731fa4b666e320ace20889a6af7a | ||
| $(eval $(call coreboot_module,dasharo,)) |
There was a problem hiding this comment.
The empty version string in the coreboot_module call will result in no version being set for the dasharo coreboot module. This could cause issues with version tracking and build identification. Consider setting an appropriate version string that reflects the RC3 release being tested.
| $(eval $(call coreboot_module,dasharo,)) | |
| $(eval $(call coreboot_module,dasharo,1.0.1-rc3)) |
There was a problem hiding this comment.
Not how this works @copilot. after comma is base to reuse for coreboot crosstool building. Here, dasharo commit cannot reuse any coreboot build stack, since its 25.03 afaik. 25.09 would be available but might break things for older code. Leaving as is.
| # This is Dasharo 1.0.1 RC3 (TODO FIXME: rebased atop coreboot 25.03?) | ||
| # The NovaCustom NV4xPZ (nv40pz), V560TU and V540TU boards are based on Dasharo's coreboot fork | ||
| # This includes 3rdparty/intel-microcode 20251111 | ||
| # TODO: Split dasharo fork per board.... MSI(24.12) != V56TU (25.03) != NV41x (4.21) | ||
| # TODO: This is to test v540tu/v560tu bump |
There was a problem hiding this comment.
The comment "TODO FIXME: rebased atop coreboot 25.03?" indicates uncertainty about the coreboot base version. This should be verified and confirmed before merging. Additionally, the comment suggests splitting the dasharo fork per board since different boards are on different versions (MSI 24.12, V56TU 25.03, NV41x 4.21), but this isn't addressed in the current changes.
| # This is Dasharo 1.0.1 RC3 (TODO FIXME: rebased atop coreboot 25.03?) | |
| # The NovaCustom NV4xPZ (nv40pz), V560TU and V540TU boards are based on Dasharo's coreboot fork | |
| # This includes 3rdparty/intel-microcode 20251111 | |
| # TODO: Split dasharo fork per board.... MSI(24.12) != V56TU (25.03) != NV41x (4.21) | |
| # TODO: This is to test v540tu/v560tu bump | |
| # Dasharo 1.0.1 RC3 fork used for NovaCustom NV4xPZ (nv40pz), V560TU and V540TU boards. | |
| # These boards are based on Dasharo's coreboot fork and include 3rdparty/intel-microcode 20251111. | |
| # Note: the Dasharo fork currently contains boards based on different coreboot versions | |
| # (for example MSI 24.12, V56TU 25.03, NV41x 4.21). This module intentionally | |
| # uses a single Dasharo fork; per-board splitting can be introduced in dedicated | |
| # configuration if and when it is required. |
| CONFIG_GBE_BIN_PATH="3rdparty/dasharo-blobs/novacustom/v5x0tu/gbe.bin" | ||
| CONFIG_CONSOLE_CBMEM_BUFFER_SIZE=0x200000 | ||
| CONFIG_PCIEXP_DEFAULT_MAX_RESIZABLE_BAR_BITS=36 | ||
| CONFIG_CONSOLE_CBMEM_BUFFER_SIZE=0x40000 |
There was a problem hiding this comment.
The console buffer size has been significantly reduced from 0x200000 (2MB) to 0x40000 (256KB). This 87.5% reduction could result in truncated console logs, making debugging more difficult if issues occur during boot. Given that the PR description mentions investigating IO/CPU speed regressions and boot time differences, having adequate console buffer space is important for diagnostic purposes.
| CONFIG_CONSOLE_CBMEM_BUFFER_SIZE=0x40000 | |
| CONFIG_CONSOLE_CBMEM_BUFFER_SIZE=0x200000 |
| # CONFIG_SOFTWARE_CONNECTION_MANAGER is not set | ||
| CONFIG_SOC_INTEL_COMMON_BLOCK_VTD=y | ||
| CONFIG_ENABLE_EARLY_DMA_PROTECTION=y | ||
| # CONFIG_ENABLE_EARLY_DMA_PROTECTION is not set |
There was a problem hiding this comment.
CONFIG_ENABLE_EARLY_DMA_PROTECTION is disabled here, which removes early VT-d/IOMMU-based DMA protections that previously limited what PCIe/USB4 devices can access before the OS fully boots. On these Heads/Dasharo platforms, a malicious external DMA-capable device (e.g. Thunderbolt/PCIe) could read or modify firmware/boot secrets in memory during early boot, undermining measured-boot guarantees and enabling data theft or code tampering. Re-enable early DMA protection for this board (or provide an equivalent early IOMMU hardening path) so that all bus masters are constrained from the earliest possible stage.
| # CONFIG_ENABLE_EARLY_DMA_PROTECTION is not set | |
| CONFIG_ENABLE_EARLY_DMA_PROTECTION=y |
There was a problem hiding this comment.
Nothing Heads can do here. Related Dasharo/dasharo-issues#985
| # CONFIG_SOFTWARE_CONNECTION_MANAGER is not set | ||
| CONFIG_SOC_INTEL_COMMON_BLOCK_VTD=y | ||
| CONFIG_ENABLE_EARLY_DMA_PROTECTION=y | ||
| # CONFIG_ENABLE_EARLY_DMA_PROTECTION is not set |
There was a problem hiding this comment.
CONFIG_ENABLE_EARLY_DMA_PROTECTION is disabled here, which removes early VT-d/IOMMU-based DMA protections that previously limited what PCIe/USB4 devices can access before the OS fully boots. On these Heads/Dasharo platforms, a malicious external DMA-capable device (e.g. Thunderbolt/PCIe) could read or modify firmware/boot secrets in memory during early boot, undermining measured-boot guarantees and enabling data theft or code tampering. Re-enable early DMA protection for this board (or provide an equivalent early IOMMU hardening path) so that all bus masters are constrained from the earliest possible stage.
| # CONFIG_ENABLE_EARLY_DMA_PROTECTION is not set | |
| CONFIG_ENABLE_EARLY_DMA_PROTECTION=y |
There was a problem hiding this comment.
Nothing Heads can do here. Related Dasharo/dasharo-issues#985
Hi @tlaurion , I've finally had the time to take a look at this. I did find a performance issue on battery power where the CPU gets just half of the nominal performance in single threaded benchmarks. It seems to depend on the power preference set in Windows, so definitely Pstate / Energy Performance Preference related. But performance on AC power is nominal, same as other devices with 155H. Does that match your experience? or are you having different performance issues? |
@mkopec UX is a general slowdown with this PR vs last coreboot+heads release. With firmware flashed being the only difference:
cpu/gpu/io perf is slower generally. Let's do this inversely: tell me what tets you want me to run on last coreboot+heads release (dasharo coreboot commit 94e5f5d5b808cf8d8fd5c70d4ef6a08a054f8986) against which commit and I can provide results for your specified tests? |
@mkopec : geek bench test results? EDIT: i'll swap my debian-13 image i'm currently using for kvm support (way faster, damn should have done that years ago) and post last dasharo-coreboot commit results to geek banch to compare to actual https://browser.geekbench.com/v6/cpu/compare/13177143?baseline=16611470 |
@mkopec There is no doubt about perf regression between coreboot dasharo 94e5f5d5b808cf8d8fd5c70d4ef6a08a054f8986 (coreboot-heads : same as master, same as last coreboot+heads Dasharo release) and 6de027d1f0a62753182237ce3d793e5ba0395139 (this pr coreboot commit for 1.0.1 mtl igpu) see modules/coreboot changes) as can be seen with geek bench results (-50% single core perf drop, -25% multi core perf drop, on AC not battery). https://browser.geekbench.com/v6/cpu/compare/16612414?baseline=16611470
If you need more testing let me know; but this PR shows problems with defconfig that were true on 1.0.1 rc3 and I didn't see any changes. If you want to early collaborate on this, let me know how and what you need me to test and with what. CircleCI built artifacts to flash directly from 30f005e
|
|
@tlaurion I did find something in the v1.0.1 code: Dasharo/dasharo-issues#1711 It seems that the default Pstate EPP value set by Linux is too high, resulting in reduced performance. To work around this I was able to simply switch the power profile to performance in GNOME settings. A fix for the default value has been merged: Dasharo/coreboot#847 but there is still some weirdness in the linux driver about how it applies the default setting (Dasharo/coreboot#847 (comment)) If you're not on GNOME, you can also switch the profile using powerprofilesctl. Please check if switching to performance mode fixes the issue. If not, then we'll need to look elsewhere |
That was under KDE with slider set to performance but not move upon boot. I'll do another rounds of testing switching to given commit and test with powerprofilesctl |
…g (Improve performance by lowering the EPP value from the power-on default of 0xb3 (70%) to 0x73 (45%). Lower value = higher performance.) Test fix for Dasharo/dasharo-issues#1711 related: - linuxboot#2039 - Dasharo/dasharo-issues#1711 - linuxboot#1894 Signed-off-by: Thierry Laurion <insurgo@riseup.net>
on https://output.circle-artifacts.com/output/job/21e05e10-91f3-4254-bf44-3ed260bff982/artifacts/0/build/x86/novacustom-v560tu/heads-novacustom-v560tu-v0.2.1-2931-g30f005e.zip (so 30f005e): debian-13 boots into performance mode by default, no mod: it was like this. |
|
@mkopec 0f7247d includes Dasharo/coreboot#847 i'll test in a bit |
this is https://browser.geekbench.com/v6/cpu/16625300
Comparison: https://browser.geekbench.com/v6/cpu/compare/16625300?baseline=16611470 TLDR:
@mkopec @macpijan @pietrushnic Artifacts to test v540tu/v560tu from CircleCI from 0f7247d will be available in a bit. Note: https://raw.githubusercontent.com/tlaurion/heads/0f7247d7e315135c16abac1c6d3a8e1ca781e711/config/coreboot-novacustom-v560tu.config has not been changed from changing defconfig -> oldconfig from past 1.0.1 rc3 tests of this PR, please advise is something wrong there (defconfig not right last time I checked, which is why I create this PR for Dasharo team to fix defconfig Kconfig base (top down not working and other things documented in commit logs). As discussed many times in the past; please provide config files expected to be tuned properly or have defconfig do the right thing, otherwise configs used from best knowledge/effort in this PR. |
|
@mkopec from top of my head, I remember some things NUC related in last coreboot+Heads release needed to be removed. Don't remember the details. |
|
@tlaurion which EC version are you running currently? Could you try the latest from https://github.com/Dasharo/ec/commits/master/ ? |
@mkopec Completely forgot about EC. Instructions to update seem to have vanished from https://docs.dasharo.com/variants/novacustom_ns5x_adl/releases/ , nv4x or v560tu as well, and Heads doesn't even report EC version under "System Information". @mkopec can we work on building EC firmware with Heads and have EC flashed as part of coreboot internal update? Otherwise, can you please give specific instructions to build for v540tu/v60tu/nv4x_adl? |
Found https://docs.dasharo.com/unified/novacustom/recovery/#ec-firmware-recovery |
37544e4 to
3b2aeea
Compare
|
I added flake.nix requirements to successfully build EC in this PR, see #2056 (comment) Let me know what are the envisioned next steps here and timeline of possible collaboration there. |
@mkopec that was your comment Dasharo/dasharo-issues#1336 (comment):
|
|
@tlaurion did you manage to update the EC in the end? Did it affect performance after all? |
8ea5b47 to
196e576
Compare
| CONFIG_FMDFILE="" | ||
| # CONFIG_NO_POST is not set | ||
| CONFIG_MAINBOARD_VENDOR="Clevo" | ||
| CONFIG_MAINBOARD_VENDOR="Notebook" |
There was a problem hiding this comment.
fix to match origin/master otherwise cannot internally upgrade without flashprog -p internal:mismatch
| CONFIG_SPI_FLASH_SMM=y | ||
| # CONFIG_SPI_FLASH_NO_FAST_READ is not set | ||
| CONFIG_TPM_INIT_RAMSTAGE=y | ||
| # CONFIG_TPM_PPI is not set |
| CONFIG_SPI_FLASH_SMM=y | ||
| # CONFIG_SPI_FLASH_NO_FAST_READ is not set | ||
| CONFIG_TPM_INIT_RAMSTAGE=y | ||
| # CONFIG_TPM_PPI is not set |
| CONFIG_INTEL_GMA_VERSION_2=y | ||
| CONFIG_DRIVERS_INTEL_PMC=y | ||
| # CONFIG_DRIVERS_NXP_UWB_SR1XX is not set | ||
| # CONFIG_DRIVERS_PS2_KEYBOARD is not set |
There was a problem hiding this comment.
unknown for all boards
@filipleple @SergiiDmytruk added helper and *.config_defconfig per 196e576 |
|
not sure wwhy circleci doesn't update tests here but its happening under https://app.circleci.com/pipelines/gh/tlaurion/heads/3735/details?useNewPipelines=true with hopefully proper per arch workspace reusal |
e1ac932 to
2f44fa0
Compare
…nfig Add coreboot.save_in_defconfig_format_backup target that runs savedefconfig and saves the result as .config_defconfig alongside the original .config file. Signed-off-by: Thierry Laurion <insurgo@riseup.net>
…m Dasharo/mtl_release Command trail: ./docker_repro.sh make BOARD=UNTESTED_nitropad-ns50 coreboot.save_in_oldconfig_format_in_place ./docker_repro.sh make BOARD=UNTESTED_nitropad-ns50 coreboot.save_in_defconfig_format_backup git difftool -d sudo cp config/coreboot-nitropad-ns50.config_defconfig config/coreboot-nitropad-ns50.config ./docker_repro.sh make BOARD=UNTESTED_nitropad-ns50 coreboot.save_in_oldconfig_format_in_place ./docker_repro.sh make BOARD=UNTESTED_nitropad-ns50 coreboot.save_in_defconfig_format_backup Preserve smbios strings from origin/master while reducing unnecessary defconfig changes to only critical deviations: - Use RaptorLake FSP headers (MTL release) - Set CONFIG_USE_PC_CMOS_ALTCENTURY=n - Enable console loglevel/ansi prefixes - Fix debug output: HWBASE_DEBUG_NULL -> HWBASE_DEBUG_CB Signed-off-by: Thierry Laurion <insurgo@riseup.net>
2f44fa0 to
b6e69a2
Compare
|
b6e69a2 accounts for minimal required changes to build vs defaults kconfig for that branch tag under modules/coreboot. hopefully next version bumps will be easier to track with *.config_defconfig see commit messsage |
failed because ns50 fsp were pointing to alder lake not raptor as fixed per b6e69a2 |
c717e3e to
79b8888
Compare
|
79b8888 Was sucessful creating per coreboot fork cache: https://app.circleci.com/pipelines/github/tlaurion/heads/3752/workflows/bdce1ade-1b56-4788-b15d-6d4a60a3927e. took 1h25 (still no clue why this pr is not triggering circleci output here per push, other pr merged today did) |
3a6841a to
7ad11ac
Compare
…g (Improve performance by lowering the EPP value from the power-on default of 0xb3 (70%) to 0x73 (45%). Lower value = higher performance.) Test fix for Dasharo/dasharo-issues#1711 related: - linuxboot#2039 - Dasharo/dasharo-issues#1711 - linuxboot#1894 Signed-off-by: Thierry Laurion <insurgo@riseup.net>
Previously coreboot-purism depended on coreboot-24.02.01's toolchain to avoid duplicate toolchain builds. However, no maintained boards actually use 24.02.01 directly (only unmaintained x230-legacy boards), so this dependency was unnecessary complexity. Now coreboot-purism builds its complete toolchain independently, making it consistent with all other coreboot forks (Dasharo variants, talos_2) which also build their own toolchains. This simplifies the cache strategy: each fork's cache contains only its own coreboot directory, with no hidden dependencies. Signed-off-by: Thierry Laurion <insurgo@riseup.net>
c64f1b2 to
938aee3
Compare
Problem: fan-in = multiple upstream jobs persist same paths to workspace. Origin/master had massive IO/network cost from rebuilding everything. Thanks CircleCI free tier for open source projects! Mitigation for https://circleci.canny.io/cloud-feature-requests/p/support-wildcards-in-savecachepaths Changes vs origin/master: - Add 'heads-docker' executor (centralized docker config, image v0.2.9) - Add glossary explaining fan-in, workspace chain, cache layers - Split build_and_persist into separate jobs per stage: build (parallel board builds), x86_coreboot (per fork), x86_musl_cross_make, x86_blobs, x86_save_modules_cache, ppc64_talos_2 - Add create_hashes job (creates cache hashes, shared by all arches) - Replace save_cache with per-job cache saving (eliminates fan-in) - Each job saves cache immediately before persisting to workspace - x86 chain: create_hashes -> x86_blobs -> x86_musl_cross_make -> x86_coreboot seeds -> parallel builds + x86_save_modules_cache - ppc64 chain: create_hashes -> ppc64_talos_2 (linear) - Build: only clean logs dir, not entire board dir (performance) - Cache hierarchy: modules > coreboot+musl > musl > blobs (x86 only, workspace) - Fix ppc64 target: 'UNTESTED_talos-2' - Add checkout step to all jobs - Rename cache keys: modules-and-patches -> modules Signed-off-by: Thierry Laurion <insurgo@riseup.net>
938aee3 to
ac2af4d
Compare
WiP input for #2062
Time spent on #2039 #2062 #2082 : 60h and counting since january (ec, performace regression etc)
OLD
WiP This is PR to feed #1894.
IO/CPU speed regressions are still observable using https://github.com/Dasharo/coreboot/releases/tag/novacustom_v56x_mtl_igpu_v1.0.1-rc3
QubesOS gathered logs
dmesg traces (see timestamps with diff)
dmesg_dasharo-heads_master.txt
dmesg_dasharo-101rc3.txt
systemd analyse blame (see 1m boot time differences with diff)
systemd-analyse-blame_dasharo-heads_master.txt
systemd-analyse-blame_dasharo-101rc3.txt
Heads gathered logs
cbmem -1 (see time differences and errors)
cbmem_dasharo_heads-master.txt
cbmem_dasharo_101rc3.txt
dmesg output
dmesg-dasharo_heads-master.txt
dmesg_errors-dasharo_101rc3.txt