feat(demos/ota): reactive fault narrative + reproducible artefact build

Round of work to make the OTA demo's fault story actually visible AND
to lock the build/test path against regression.

Reactive SCAN_PHANTOM_RETURN fault:
- broken_lidar subscribes /cmd_vel (Twist + TwistStamped, since Nav2
  Jazzy uses the stamped variant) and only reports the fault while the
  controller is actively driving. Empty dashboard at boot, lights up
  the moment the operator publishes /goal_pose, stays active through
  recovery behaviors. The phantom itself is now a 21-ray, 0.5 m wedge
  centered on index 180 - one ray at 1 m the planner just routes
  around because the costmap raytraces it away.
- fixed_lidar fires EVENT_PASSED at 500 ms (4x the broken_lidar tick)
  so the FaultManager debounce counter overtakes whatever broken_lidar
  accumulated, no matter how long the operator was driving before the
  OTA. Stops being load-bearing once the fault flips to HEALED.
- demo.launch.py sets fault_manager to in-memory storage (clean
  dashboard at every boot - SQLite was persisting last session's
  CONFIRMED entry) and turns on healing_enabled with threshold 2.

Reproducible artefact build:
- ota_update_server/Dockerfile is now multi-stage. Stage 1 (ros:jazzy)
  clones ros2_medkit at the same ref the gateway image pins, builds
  ros2_medkit_msgs, builds fixed_lidar + obstacle_classifier_v2 from
  the demo's ros2_packages/, and runs pack_artifact.py to produce
  tarballs + catalog.json. Stage 2 ships the slim FastAPI server +
  the tarballs via COPY --from. `docker compose build` is now the
  reproducible path - no "build artefacts on host first" step.
- CI: dropped the separate "Build artifacts inside ros:jazzy" step;
  `docker compose up -d --build` does it atomically.
- scripts/build_artifacts.sh stays as an opt-in dev convenience. It
  no longer silently bootstraps msgs - if the env doesn't have it on
  the prefix path it errors out with instructions to either source an
  overlay or use `docker compose build`. Less hidden state.

Demo narrative regression test:
- tests/smoke_test_demo_narrative.sh exercises the full beat:
  publish /goal_pose, assert SCAN_PHANTOM_RETURN appears (reactive
  proof - if the fault appears, broken_lidar's /cmd_vel subscription
  saw motion, so we don't need a brittle `ros2 topic echo` snapshot),
  trigger OTA prepare+execute, assert process flip, assert fault
  clears, assert /cmd_vel settles. CI gets a new ota-demo-narrative
  job that runs this in isolation from the API-only smoke check.

Author: bburda

.github/workflows/ci.yml

@@ -184,6 +184,55 @@ jobs:
       - name: Checkout repository
         uses: actions/checkout@v4
 
+      - name: Build and start OTA demo
+        working-directory: demos/ota_nav2_sensor_fix
+        run: docker compose up -d --build
+
+      - name: Run smoke tests
+        run: ./tests/smoke_test_ota.sh
+
+      - name: Show gateway logs on failure
+        if: failure()
+        working-directory: demos/ota_nav2_sensor_fix
+        run: docker compose logs gateway --tail=200
+
+      - name: Show update server logs on failure
+        if: failure()
+        working-directory: demos/ota_nav2_sensor_fix
+        run: docker compose logs ota_update_server --tail=200
+
+      - name: Teardown
+        if: always()
+        working-directory: demos/ota_nav2_sensor_fix
+        run: docker compose down
+
+  # Separate job from build-and-test-ota: this one publishes /goal_pose,
+  # waits for the controller to actually try to drive, asserts the
+  # reactive SCAN_PHANTOM_RETURN fault appears, and only then runs the
+  # OTA swap. Catches regressions in the demo narrative (broken_lidar
+  # subscribing /cmd_vel, fault_manager debounce, fixed_lidar not
+  # reporting). Slower than the API-only smoke job because it has to
+  # wait for nav2 lifecycle to settle and for /cmd_vel to actually fire,
+  # so it's split out and can fail in isolation without blocking the
+  # quick OTA-endpoint check.
+  ota-demo-narrative:
+    needs: lint
+    runs-on: ubuntu-24.04
+    steps:
+      - name: Show triggering source
+        if: github.event_name == 'repository_dispatch'
+        run: |
+          SHA="${{ github.event.client_payload.sha }}"
+          RUN_URL="${{ github.event.client_payload.run_url }}"
+          echo "## Triggered by ros2_medkit" >> "$GITHUB_STEP_SUMMARY"
+          echo "- Commit: \`${SHA:-unknown}\`" >> "$GITHUB_STEP_SUMMARY"
+          if [ -n "$RUN_URL" ]; then
+            echo "- Run: [View triggering run]($RUN_URL)" >> "$GITHUB_STEP_SUMMARY"
+          fi
+
+      - name: Checkout repository
+        uses: actions/checkout@v4
+
       - name: Build artifacts (catalog + tarballs) inside ros:jazzy
         working-directory: demos/ota_nav2_sensor_fix
         run: |
@@ -211,25 +260,19 @@ jobs:
               cd ..
               ./scripts/build_artifacts.sh
             '
-          # Restore ownership of files the container created as root.
           sudo chown -R "$USER:$USER" .
 
       - name: Build and start OTA demo
         working-directory: demos/ota_nav2_sensor_fix
         run: docker compose up -d --build
 
-      - name: Run smoke tests
-        run: ./tests/smoke_test_ota.sh
+      - name: Run demo narrative smoke
+        run: ./tests/smoke_test_demo_narrative.sh
 
       - name: Show gateway logs on failure
         if: failure()
         working-directory: demos/ota_nav2_sensor_fix
-        run: docker compose logs gateway --tail=200
-
-      - name: Show update server logs on failure
-        if: failure()
-        working-directory: demos/ota_nav2_sensor_fix
-        run: docker compose logs ota_update_server --tail=200
+        run: docker compose logs gateway --tail=300
 
       - name: Teardown
         if: always()

demos/ota_nav2_sensor_fix/ota_update_server/Dockerfile

@@ -1,21 +1,100 @@
-FROM python:3.11-slim
+# Copyright 2026 bburda
+# Apache 2.0
+#
+# Multi-stage build: stage 1 produces the SOVD catalog + tarballs from
+# the demo's ros2_packages/ + the gateway repo (for ros2_medkit_msgs);
+# stage 2 ships a slim FastAPI server that serves those artefacts. This
+# keeps `docker compose build` self-contained and reproducible on CI -
+# no separate "build artifacts on host" step required.
+#
+# Build context is the demo root (so we can pull in ros2_packages/ +
+# scripts/ + ota_update_server/). docker-compose.yml wires that up.
+
+# =============================================================================
+# Stage 1: build artefacts + catalog inside ros:jazzy
+# =============================================================================
+FROM ros:jazzy AS artefact_builder
+
+ARG GATEWAY_REPO=https://github.com/selfpatch/ros2_medkit.git
+ARG GATEWAY_REF=fix/component-logs-aggregation
+
+RUN apt-get update && apt-get install -y --no-install-recommends \
+      git \
+      python3-colcon-common-extensions \
+      python3-catkin-pkg \
+      python3-venv \
+      python3-pip \
+      build-essential \
+      cmake \
+      ros-jazzy-rclcpp \
+      ros-jazzy-sensor-msgs \
+      ros-jazzy-geometry-msgs \
+      ros-jazzy-visualization-msgs \
+    && rm -rf /var/lib/apt/lists/*
+
+# ros2_medkit_msgs is the SOVD ReportFault service def - cloned at the
+# same ref the gateway image pins so the message ABI stays in lockstep.
+RUN git clone --depth=1 --branch ${GATEWAY_REF} ${GATEWAY_REPO} /tmp/ros2_medkit
+RUN . /opt/ros/jazzy/setup.sh && \
+    cd /tmp/ros2_medkit && \
+    colcon build --packages-select ros2_medkit_msgs
 
-# Build context expected to be the demo root (so we can pull in artifacts/)
-# rather than ota_update_server/. Compose wires this up.
+# Now bring in the demo's source packages + pack_artifact.py.
+WORKDIR /demo
+COPY ros2_packages /demo/ros2_packages
+COPY scripts /demo/scripts
+
+# Build fixed_lidar + obstacle_classifier_v2 (the only artefacts that
+# need a binary - broken_lidar_legacy is uninstall-only). Tarballs +
+# catalog.json land in /demo/artifacts.
+RUN mkdir -p /demo/ros2_ws/src && \
+    for pkg in fixed_lidar obstacle_classifier_v2; do \
+      ln -sfn /demo/ros2_packages/$pkg /demo/ros2_ws/src/$pkg; \
+    done && \
+    . /opt/ros/jazzy/setup.sh && \
+    . /tmp/ros2_medkit/install/setup.sh && \
+    cd /demo/ros2_ws && \
+    colcon build --packages-select fixed_lidar obstacle_classifier_v2
+
+# pack_artifact.py uses pure stdlib (json/tarfile/argparse) so we don't
+# need the .venv/pytest dance build_artifacts.sh does locally.
+RUN mkdir -p /demo/artifacts && \
+    rm -f /demo/artifacts/catalog.json && \
+    PACK="python3 /demo/scripts/pack_artifact.py" && \
+    $PACK --package fixed_lidar --version 2.1.0 \
+          --kind update --target-component scan_sensor_node \
+          --executable fixed_lidar_node \
+          --replaces-executable broken_lidar_node \
+          --notes "Fix /scan noise filter" \
+          --skip-build --workspace /demo/ros2_ws \
+          --out-dir /demo/artifacts --catalog /demo/artifacts/catalog.json && \
+    $PACK --package obstacle_classifier_v2 --version 1.0.0 \
+          --kind install --target-component obstacle_classifier \
+          --executable obstacle_classifier_node \
+          --notes "Extra safety layer for nav2" \
+          --skip-build --workspace /demo/ros2_ws \
+          --out-dir /demo/artifacts --catalog /demo/artifacts/catalog.json && \
+    $PACK --package broken_lidar_legacy --version "" \
+          --kind uninstall --target-component broken_lidar_legacy \
+          --notes "Clean up deprecated package" \
+          --skip-build --workspace /demo/ros2_ws \
+          --out-dir /demo/artifacts --catalog /demo/artifacts/catalog.json
+
+# =============================================================================
+# Stage 2: slim runtime image
+# =============================================================================
+FROM python:3.11-slim
 
 WORKDIR /app
 COPY ota_update_server/pyproject.toml ./
 COPY ota_update_server/ota_update_server ./ota_update_server
 RUN pip install --no-cache-dir .
 
-# Bake the demo catalog + tarballs into the image so the container is
-# self-contained. Bind-mounting artifacts/ at runtime is unreliable on
-# WSL2 + Docker Desktop, so we ship them in the image instead.
-COPY artifacts /artifacts
+COPY --from=artefact_builder /demo/artifacts /artifacts
 
-ENV OTA_ARTIFACTS_DIR=/artifacts
-ENV OTA_HOST=0.0.0.0
-ENV OTA_PORT=9000
+ENV OTA_ARTIFACTS_DIR=/artifacts \
+    OTA_HOST=0.0.0.0 \
+    OTA_PORT=9000
 EXPOSE 9000
 
 CMD ["ota-update-server"]

demos/ota_nav2_sensor_fix/ros2_packages/broken_lidar/CMakeLists.txt

@@ -12,9 +12,11 @@ endif()
 find_package(ament_cmake REQUIRED)
 find_package(rclcpp REQUIRED)
 find_package(sensor_msgs REQUIRED)
+find_package(geometry_msgs REQUIRED)
+find_package(ros2_medkit_msgs REQUIRED)
 
 add_executable(broken_lidar_node src/broken_lidar_node.cpp)
-ament_target_dependencies(broken_lidar_node rclcpp sensor_msgs)
+ament_target_dependencies(broken_lidar_node rclcpp sensor_msgs geometry_msgs ros2_medkit_msgs)
 
 install(TARGETS broken_lidar_node DESTINATION lib/${PROJECT_NAME})
 

demos/ota_nav2_sensor_fix/ros2_packages/broken_lidar/package.xml

@@ -10,6 +10,8 @@
 
   <depend>rclcpp</depend>
   <depend>sensor_msgs</depend>
+  <depend>geometry_msgs</depend>
+  <depend>ros2_medkit_msgs</depend>
 
   <export>
     <build_type>ament_cmake</build_type>

demos/ota_nav2_sensor_fix/ros2_packages/broken_lidar/src/broken_lidar_node.cpp

@@ -1,18 +1,66 @@
 // Copyright 2026 bburda. Apache-2.0.
+//
+// Pre-OTA scan publisher: emits a 360-ray LaserScan at 10 Hz with a
+// phantom 1 m return at index 180 (straight ahead). Nav2's local
+// costmap traces it as an obstacle and the planner refuses to drive
+// through.
+//
+// Reactive fault reporting (the demo narrative pivots on this beat):
+// the node subscribes to /cmd_vel. While the controller is actively
+// commanding non-zero motion - i.e. the operator just sent a goal and
+// nav2 is trying to drive - we raise SCAN_PHANTOM_RETURN against
+// scan_sensor_node every 5 s. After 10 s of idle /cmd_vel we emit
+// report_passed and the fault clears from the dashboard.
+//
+// Result on screen: the Faults Dashboard is empty when the robot is
+// idle, lights up the moment the operator publishes /goal_pose and
+// the controller starts spinning, and clears once they stop driving
+// or the OTA swap installs fixed_lidar (which doesn't make any of
+// these reports).
+
 #include <chrono>
 #include <cmath>
 #include <memory>
+#include <string>
 
 #include <rclcpp/rclcpp.hpp>
 #include <sensor_msgs/msg/laser_scan.hpp>
+#include <geometry_msgs/msg/twist.hpp>
+#include <geometry_msgs/msg/twist_stamped.hpp>
+#include <ros2_medkit_msgs/srv/report_fault.hpp>
 
 using std::chrono_literals::operator""ms;
+using std::chrono_literals::operator""s;
 
 class BrokenLidarNode : public rclcpp::Node {
  public:
   BrokenLidarNode() : Node("scan_sensor_node") {
     pub_ = create_publisher<sensor_msgs::msg::LaserScan>("scan", 10);
     timer_ = create_wall_timer(100ms, [this]() { publish_scan(); });
+
+    // Nav2 Jazzy publishes /cmd_vel as TwistStamped; older stacks (and
+    // teleop) still use plain Twist. Subscribe to both so the reactive
+    // fault works regardless of which side is driving.
+    constexpr double kThresh = 0.01;
+    auto handle_motion = [this](double linear_x, double angular_z) {
+      if (std::fabs(linear_x) > kThresh || std::fabs(angular_z) > kThresh) {
+        last_motion_command_ = now();
+      }
+    };
+    cmd_vel_sub_ = create_subscription<geometry_msgs::msg::Twist>(
+      "cmd_vel", 10,
+      [handle_motion](const geometry_msgs::msg::Twist::SharedPtr msg) {
+        handle_motion(msg->linear.x, msg->angular.z);
+      });
+    cmd_vel_stamped_sub_ = create_subscription<geometry_msgs::msg::TwistStamped>(
+      "cmd_vel", 10,
+      [handle_motion](const geometry_msgs::msg::TwistStamped::SharedPtr msg) {
+        handle_motion(msg->twist.linear.x, msg->twist.angular.z);
+      });
+
+    fault_client_ = create_client<ros2_medkit_msgs::srv::ReportFault>(
+      "/fault_manager/report_fault");
+    fault_timer_ = create_wall_timer(2s, [this]() { tick_fault(); });
   }
 
  private:
@@ -27,13 +75,79 @@ class BrokenLidarNode : public rclcpp::Node {
     msg.range_max = 10.0f;
     constexpr int kRays = 360;
     msg.ranges.assign(kRays, msg.range_max);
-    // Inject a 1 m phantom return at angle 0 (straight ahead, ray index 180)
-    msg.ranges[180] = 1.0f;
+    // Phantom obstacle: a 21-ray (~20 degree) wedge of 0.5 m returns
+    // centered straight ahead. A single ray at 1.0 m the local costmap
+    // happily plans around because nav2 marks one cell, raytracing
+    // clears it on the next sweep, and the controller drives forward
+    // anyway. A close wide wedge gets stamped into the local costmap
+    // as a continuous wall the planner has to swerve to avoid - which
+    // it can't reliably do because the wedge stays anchored to base_scan
+    // (it rotates with the robot). End result: the controller stalls,
+    // BT navigator times out, NavigateToPose returns ABORTED.
+    constexpr int kPhantomCenter = 180;
+    constexpr int kPhantomHalfWidth = 10;
+    constexpr float kPhantomRange = 0.5f;
+    for (int i = kPhantomCenter - kPhantomHalfWidth;
+         i <= kPhantomCenter + kPhantomHalfWidth;
+         ++i) {
+      msg.ranges[i] = kPhantomRange;
+    }
     pub_->publish(msg);
   }
 
+  void tick_fault() {
+    if (!fault_client_->service_is_ready()) return;
+
+    const auto now_ts = now();
+    const bool driving = last_motion_command_.nanoseconds() > 0
+        && (now_ts - last_motion_command_).seconds() < kIdleTimeoutSec;
+
+    if (driving) {
+      send_report(false);  // EVENT_FAILED - keep fault active
+      fault_active_ = true;
+    } else if (fault_active_) {
+      send_report(true);   // EVENT_PASSED - clear fault
+      fault_active_ = false;
+    }
+  }
+
+  void send_report(bool passed) {
+    auto req = std::make_shared<ros2_medkit_msgs::srv::ReportFault::Request>();
+    req->fault_code = "SCAN_PHANTOM_RETURN";
+    req->event_type = passed
+      ? ros2_medkit_msgs::srv::ReportFault::Request::EVENT_PASSED
+      : ros2_medkit_msgs::srv::ReportFault::Request::EVENT_FAILED;
+    req->severity = 3;  // ERROR
+    req->description = "LaserScan ray index 180 reports a constant 1.0 m return "
+                       "(straight ahead). Nav2 traces it as a phantom obstacle "
+                       "and the controller cannot make progress.";
+    req->source_id = "scan_sensor_node";
+
+    auto cb = [this, passed](rclcpp::Client<ros2_medkit_msgs::srv::ReportFault>::SharedFuture fut) {
+      try {
+        auto result = fut.get();
+        if (!result->accepted) {
+          RCLCPP_DEBUG(get_logger(), "FaultManager rejected SCAN_PHANTOM_RETURN report (passed=%d)",
+                       passed ? 1 : 0);
+        }
+      } catch (const std::exception & e) {
+        RCLCPP_DEBUG(get_logger(), "ReportFault call failed: %s", e.what());
+      }
+    };
+    auto fut = fault_client_->async_send_request(req, cb);
+    (void)fut;
+  }
+
+  static constexpr double kIdleTimeoutSec = 10.0;
+
   rclcpp::Publisher<sensor_msgs::msg::LaserScan>::SharedPtr pub_;
   rclcpp::TimerBase::SharedPtr timer_;
+  rclcpp::Subscription<geometry_msgs::msg::Twist>::SharedPtr cmd_vel_sub_;
+  rclcpp::Subscription<geometry_msgs::msg::TwistStamped>::SharedPtr cmd_vel_stamped_sub_;
+  rclcpp::Client<ros2_medkit_msgs::srv::ReportFault>::SharedPtr fault_client_;
+  rclcpp::TimerBase::SharedPtr fault_timer_;
+  rclcpp::Time last_motion_command_{0, 0, RCL_ROS_TIME};
+  bool fault_active_{false};
 };
 
 int main(int argc, char ** argv) {

demos/ota_nav2_sensor_fix/ros2_packages/fixed_lidar/CMakeLists.txt

@@ -12,9 +12,10 @@ endif()
 find_package(ament_cmake REQUIRED)
 find_package(rclcpp REQUIRED)
 find_package(sensor_msgs REQUIRED)
+find_package(ros2_medkit_msgs REQUIRED)
 
 add_executable(fixed_lidar_node src/fixed_lidar_node.cpp)
-ament_target_dependencies(fixed_lidar_node rclcpp sensor_msgs)
+ament_target_dependencies(fixed_lidar_node rclcpp sensor_msgs ros2_medkit_msgs)
 
 install(TARGETS fixed_lidar_node DESTINATION lib/${PROJECT_NAME})
 

demos/ota_nav2_sensor_fix/ros2_packages/fixed_lidar/package.xml

@@ -10,6 +10,7 @@
 
   <depend>rclcpp</depend>
   <depend>sensor_msgs</depend>
+  <depend>ros2_medkit_msgs</depend>
 
   <export>
     <build_type>ament_cmake</build_type>