feat: migrate away from Arduino#438
Draft
hhvrc wants to merge 96 commits into
Draft
Conversation
I'd prefer to be able to distinguish between the different types easily in support chat.
…o feat/arduino-3.0
Merge develop into feat/arduino-3.0. Replace remaining tcb::span with std::span, add missing WiFi.h include. Build not yet clean — remaining issues: Serial namespace rename (SerialCmds), USBSerial API, and CaptivePortalConfig struct changes. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Mark the Arduino -> native ESP-IDF rework as a major (breaking) change.
Under parallel SCons jobs the ESP-IDF bootloader link intermittently fails with 'cannot open linker script file bootloader.ld' because bootloader.elf can link before the linker-script preprocessing step writes bootloader.ld. The failing board set varied run to run, confirming a race rather than a source problem. Build the bootloader with a single job first so the script is generated deterministically, then run the full parallel build which reuses the up-to-date bootloader.
The previous serial -t bootloader pre-pass was wrong: the single-bootloader-target code path skips linker-script generation entirely, so it failed deterministically. Revert to a plain parallel build and instead retry it. The real failure is a timing race in the upstream ESP-IDF bootloader link (bootloader.elf occasionally links before its generated bootloader.ld is visible). A different board set fails each run. After the first attempt the script is on disk, so an incremental retry re-links only the bootloader and succeeds; retry up to 3 times.
…h audit api.openshock.app / api.openshock.dev present a chain topped by the cross-signed GTS Root R4, whose issuer is GlobalSign Root CA R1. esp_crt_bundle does no path building -- it matches the issuer of the topmost presented cert against the trust store -- and Mozilla/curl retired R1, so verification failed with "No matching trusted root certificate found". Pin R1 so the chain verifies. - Split the store into two tracked files: cacert-curl.pem (verbatim curl/Mozilla extract, sha256 matches curl's published hash) and cacert-merged.pem (base + pinned_certs/*.pem, de-duplicated, with an OpenShock banner). The merged file is what the firmware packs (CONFIG_MBEDTLS_CUSTOM_CERTIFICATE_BUNDLE_PATH). - cert_bundle.py gains an `audit` subcommand: probes the live endpoints to retire pins no chain still needs (opens a PR) and alerts before a still-needed pin expires. Generation de-duplicates inputs and rejects expired/duplicate pins. - Add the pinned-cert-audit CI workflow (weekly + on push + manual).
Move FirmwareBootType, ShockerCommandType, and ShockerModelType off the old const char* + strcasecmp converters onto the StringIEquals / std::string_view pattern already used by OtaUpdateChannel/OtaUpdateStep. The two *FromString functions are renamed to the TryParse<Name> convention with (out, string_view) argument order. Call sites in JsonAPI/JsonSerial drop the now-unnecessary std::string(...).c_str() heap copies and pass views directly; test_enums is updated to match. Behavior is preserved: StringIEquals is length-aware case-insensitive ASCII equality, matching the prior full-string strcasecmp semantics, and safe on non-NUL-terminated views straight from the jsmn parser.
Replace the PlatformIO build with a native idf.py build driven by scripts/build.py: per-board sdkconfig fragments under boards/*.defaults layered onto sdkconfig.defaults, an env header generated by scripts/gen_env_header.py, and the captive-portal static0 image packed by scripts/gen_staticfs.py. Vendor the flatbuffers runtime as a submodule component. Rename chips/ -> partitions/ (ota_4mb.csv, single_4mb.csv) now that it holds only partition tables. Remove platformio.ini, the .env files, the PlatformIO extra_scripts and merge tooling, and the lib/test scaffolding.
Use the Kconfig-generated CONFIG_OPENSHOCK_* symbols directly for GPIO pins, API/CDN domains and buffer sizes now that the PlatformIO -D build-flag path is gone. Bridge the pin/LED macros from CONFIG_* in the shared headers, defaulting an omitted option to INVALID/none. Fix the WS2812B R/G channel swap: it tested an un-prefixed macro that the native build never defines, so the swap silently never triggered on the boards that enable it (Waveshare S3-Zero, Wemos S3-Mini). Read the CONFIG_ symbol directly.
Build every board with scripts/build.py through install-esp-idf-action (new setup-esp-idf action), emitting the OTA partition images (app.bin, staticfs.bin, bootloader, partition-table) and a merge-bin full-flash binary in one job - folding in the old build-staticfs and merge-partitions steps. Derive the board matrix from boards/*.defaults via a Python get-vars.py port, and point cpp-linter/CodeQL at the native compile_commands.json. Retire the PlatformIO CI: pio-cache, build-staticfs and merge-partitions actions, and the get-vars.js/node toolchain.
Remove the committed .claude/ settings and hook, .mcp.json, and the BEFORE_MERGE_WORK migration notes. Stop tracking .vscode/settings.json now that .gitignore covers .vscode/ (the local file stays on disk).
Restore the proven parallel-then-link pipeline shape. build-firmware now compiles only the code partitions (app, bootloader, partition-table) with no frontend/static0 coupling, so it runs concurrently with build-staticfs, which packs the board-independent static0 image once. A new merge-partitions job links each board's partitions with the shared staticfs into the single flashable binary via scripts/merge_image.py (esptool merge-bin at the ota_4mb.csv offsets; bootloader 0x0/0x1000 by chip) - no full ESP-IDF needed. Restores the firmware_staticfs artifact and the CDN's per-board staticfs copy.
install-esp-idf-action only sets IDF_PATH; scripts/build.py then invoked idf.py
with the system Python, which idf.py rejects ("not spawned within an ESP-IDF
shell environment"), failing every board build. Source export.sh in setup-esp-idf
and persist the venv-activated PATH/IDF_PYTHON_ENV_PATH to GITHUB_ENV so python
and idf.py resolve to the ESP-IDF venv in later steps. Install littlefs-python
into that venv for gen_staticfs.py (full compiledb build); leave esptool to
idf.py's bundled copy. Keep the ~/.espressif tools cache (it also captures the
venv, so the littlefs-python install is a no-op on a hit).
Sourcing export.sh failed under the action's EIM-managed install ("tool
esp-rom-elfs has no installed versions"): EIM manages tools in a way the
classic export.sh does not see. The action's intended entry point is the
idf.py wrapper it puts on PATH, which activates the EIM tool + venv
environment itself. Make scripts/build.py prefer that wrapper on POSIX and
fall back to `python tools/idf.py` for an already-active shell (Windows local).
setup-esp-idf drops the export.sh/venv-activation dance (just installs +
caches, broadened to /tmp/esp where the action actually installs).
build-compilationdb builds only `app` (still emits compile_commands.json and
a CodeQL compile pass) so it needs no staticfs/frontend/littlefs. Build steps
call python3 since setup-python no longer provides `python`.
The cpp-linter job crashed with KeyError: 'patch'. GitHub returns 406 for this PR's ~27k-line diff, dropping cpp-linter into a paginated fallback that dereferences file['patch'] on files GitHub omits a patch for. Scan the source tree (files-changed-only: false) instead of the diff API to sidestep the diff-size limit. Disable clang-tidy (tidy-checks: '-*'): the native ESP-IDF compile DB is generated by xtensa-esp32s3-elf-gcc, whose flags LLVM clang-tidy rejects, and standalone headers have no compile command so their ESP-IDF includes don't resolve - both surface as spurious clang-diagnostic-errors. clang-format still gates style and CodeQL covers static analysis; the compile DB build stays so tidy can be re-enabled once it is clang-toolchain compatible. Ignore build/ so the tree walk skips generated sources (CMake compiler probes, sdkconfig.h), and wrap the hand-aligned DNS answer byte grid in clang-format off/on so it passes clang-format.
Move the application out of the leftover PlatformIO src/ + include/ split into auto-discovered components: - main/ holds only main.cpp (the app_main entry). - components/core/ holds the interconnected subsystem tangle (config, serialization, wifi, captive portal, gateway, ota, http/JsonAPI, message handlers, radio, visual, estop, command handler). These form a single dependency cycle (an SCC), so they cannot be separate components. - components/serial/ holds the serial console + command handlers, the one cleanly separable piece (nothing in core depends on it). Drop the now-dead EXTRA_COMPONENT_DIRS/PlatformIO scaffolding from the root CMakeLists, move the esp_websocket_client managed dep and Kconfig.projbuild into core, and repoint generated flatbuffer headers to components/core/include/serialization/_fbs. Builds clean on Wemos-D1-Mini-ESP32 and Seeed-Xiao-ESP32C3 (IDF 6.0.2).
Remove src/LanguageVersionCheck.cpp (a translation unit of #error directives asserting a C++23 compiler) and its registration from the common component.
Phase 0 encapsulation cleanups (no behavior change): - Delete dead includes: estop/EStopManager.h from VisualStateManager.h (header uses no EStop symbol), config/Config.h from WSGateway.cpp (no Config:: use), and esp_wifi_types.h from WSLocal.h (wifi_auth_mode_t reaches consumers via wifi/WiFiNetwork.h). - Demote .cpp-only deps to PRIV_REQUIRES so they stop propagating to consumers: concurrency (logging), dns_server (util/logging/lwip; declare freertos), and rfc8908 (logging/osjson; drop unused common). Public REQUIRES now list only the deps that actually appear in each component's public headers.
Split the serial subsystem into a foundation-layer transport and the app console, and route logging through raw serial writes instead of C stdio: - components/serial (NEW, foundation): OpenShock::Serial owns the raw console byte I/O (UART0 / USB-Serial-JTAG). Init/Read as before, plus Write(). Depends only on the IDF console drivers, so it must not use OS_LOG (Init reports failure via its return value). Write() falls back to the ROM serial output before the driver is installed, so early-boot logs survive. - components/serial_console (renamed from the old app-level `serial`; `console` is a reserved IDF component name): SerialInputHandler + command handlers, now PRIV_REQUIRES core + serial. - logging: openshock_log_printf formats the line and writes the bytes via Serial::Write instead of vprintf/stdout. Logging.h drops its unused <esp_log.h>, so it carries no hardware include (matching its own doc comment). Removing <esp_log.h> from Logging.h dropped a transitive <string.h>; add the <cstring> that HTTPRequestManager, PartitionUtils, CommandHandler, WiFiCredentials and RFTransmitter actually rely on. Builds on Wemos-D1-Mini-ESP32 (UART) and Seeed-Xiao-ESP32C3 (USB-Serial-JTAG).
Add a linux-target Unity host_test for the RF protocol encoders — the highest-value pure-logic pocket in the firmware, previously with zero off-hardware coverage. It compiles the encoder sources directly with a stub <driver/rmt_types.h> (just rmt_symbol_word_t) and no-op Logging.h, so it needs neither esp_driver_rmt nor the serial transport. Coverage (11 cases): EncodeBits<N> MSB-first bit packing; CaiXianlin frame layout decoded back from the RMT symbols (id/channel/type/intensity/checksum), intensity clamping, and the Sound-mutes-intensity quirk; Sequence model dispatch across all five shocker models. Run: cd components/protocols/host_test && idf.py --preview set-target linux && idf.py build then execute build/protocols_host_test.elf (all 11 pass).
Phase 0.5: stop core's captive-portal and gateway internals from leaking heavy IDF headers into its public surface. - Move CaptivePortalInstance.h (pulled esp_http_server / dns_server / fs) and GatewayClient.h (pulled esp_websocket_client / esp_event) out of include/ into src/, alongside their .cpp — both are consumed only within core/src. Same for the message_handlers internals (ShockerCommandList.h, impl/WSGateway.h, impl/WSLocal.h); only WebSocket.h stays public. - Add PRIV_INCLUDE_DIRS "src" so those internal headers still resolve. - Demote esp_http_server, fs, dns_server, rfc8908 and esp_websocket_client to PRIV_REQUIRES now that no public header pulls them, so they no longer propagate to core's consumers (main, serial_console). Public core headers now carry only lightweight includes. Builds on Wemos-D1-Mini-ESP32 and Seeed-Xiao-ESP32C3.
- Annotate set_failed as NoReturn.
- Collapse repeated separators in sanitize() so names don't yield doubled dashes.
- Parse the branch from a non-PR ref with split('/', 2) so refs containing extra
slashes keep their full tail.
hhvrc
commented
Jul 10, 2026
Contributor
Author
There was a problem hiding this comment.
-DDARDUINO_USB_CDC_ON_BOOT=1 was dropped here, need to investigate this.
hhvrc
commented
Jul 10, 2026
Contributor
Author
There was a problem hiding this comment.
-DDARDUINO_USB_CDC_ON_BOOT=1 was dropped here, need to investigate this.
Add coverage for three untested pure-logic primitives (P1.2): Checksum::Sum8 over spans/integrals/structs plus the ReverseNibble lookup tables, TinyVec growth/ zero-fill/assign/move/span semantics, and the FnProxy member-function trampoline. Also unbreak the util host_test on IDF 6.0: hal/gpio_types.h moved from `hal` to `esp_hal_gpio`, so Convert.h no longer resolved (REQUIRES fixed). And drop the OS_PANIC/OS_PANIC_OTA defines from the stub Logging.h — they also live in the stub Panic.h, and TinyVec (the new test) is the first source to pull both, which double-defined them under -Werror. 42 tests pass (was 30) via idf.py --preview set-target linux.
Hoist the OtaUpdateStep -> FirmwareBootType mapping out of OtaUpdateManager's boot path into a pure constexpr OpenShock::InferFirmwareBootType(OtaUpdateStep) in the common enum header, and host-test every step's mapping (P1.3). OtaUpdateManager now calls it instead of the inline switch. This does not remove GatewayClient's dependency on OtaUpdateManager (it still needs the boot-time cached value from GetFirmwareBootType()); the win here is host-testable boot-type logic. common host_test: 14 pass. Firmware builds on Wemos-D1-Mini-ESP32.
…component Move core/include/serialization/_fbs/*_generated.h into a new low-level `serialization` component (REQUIRES only flatbuffers) and point scripts/generate_schemas.py at its include tree. The include prefix (serialization/_fbs/) is unchanged, so no consumer #include edits — only core's REQUIRES gains `serialization`. These generated wire types are shared across config, wifi, the message codecs and the handlers; pulling them out of core into a single flatbuffers-only layer that everything depends on one-way is the prerequisite for extracting config (and later the codecs themselves) from core without a schema dependency cycle. The component is named `serialization` because it is the start of that layer — the codecs (JsonAPI/JsonSerial/WSGateway/WSLocal) move here in a later step. Builds on Wemos-D1-Mini-ESP32.
Move config/ out of core into components/config. Now that the generated schemas are their own `serialization` component, config depends only on low-layer components — common, osjson, util, serialization (HubConfig fbs), flatbuffers, esp_wifi (wifi_auth_mode_t) and esp_hal_gpio in its public headers; chipset, fs, logging, panic and concurrency privately — and nothing in core. core REQUIRES config one-way. No forward-declare/invert was needed: extracting the schemas already broke the config<->serialization cycle, so this is a clean move + CMakeLists. Public config headers still expose the HubConfig flatbuffer (tightening that to a forward-declare is a separate encapsulation step). This unblocks the device_control extraction, whose only upstream-core edge was config. Builds on Wemos-D1-Mini-ESP32 and Seeed-Xiao-ESP32C3.
…rom core Move the hardware-facing actuators and inputs — EStopManager, RFTransmitter, VisualStateManager and the shocker CommandHandler — out of core into a new components/device_control. Their only upstream-core dependency was config, now its own low-layer component, so device_control depends only on config + the foundation/ driver layer and nothing else in core. Wiring: core PRIV_REQUIRES device_control (estop/CommandHandler/visual used only in core .cpp — captiveportal/message_handlers/gateway); main and serial_console REQUIRE it directly (they include those headers). No core public header referenced them, so it's a one-way edge with no cycle. Public headers need common/esp_hal_gpio/esp_driver_rmt/freertos; everything else (config, chipset, events, led_drivers, protocols, esp_wifi/event/netif, ...) is private. Builds on Wemos-D1-Mini-ESP32 and Seeed-Xiao-ESP32C3.
FnProxy: replace the four hand-written cv/noexcept partial specializations with a single shared Body::call trampoline plus four thin Deduce specializations that only pattern-match the qualifier combination. Same capability (every non-ref-qualified member function), body written once, no macros. FnProxy<&C::m> is still a plain R(*)(void*, Args...) function pointer for C callback APIs. DigitCounter: constrain Digits10Count with std::integral and tidy the magnitude/ sign handling. Kept pure arithmetic - no buffer, no allocation. util host_test 42/42 (incl. a const-noexcept case); builds on Wemos-D1-Mini-ESP32 and Seeed-Xiao-ESP32C3.
Body gains a bool NX parameter and declares call() noexcept(NX), so a noexcept member function now yields a noexcept function pointer (it still converts to a plain one for C callback APIs). The const specializations bind C as `const C` so the cast matches the method's constness rather than casting const away. host_test adds static_asserts pinning the deduced pointer types (a plain method stays plain, a const-noexcept method gains noexcept). 42/42; builds on Wemos-D1-Mini-ESP32 and Seeed-Xiao-ESP32C3.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.