Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
85 changes: 60 additions & 25 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,29 @@ Supported targets in this repository:
- `grisp2`
- `kontron-albl-imx8mm`

**Table of contents**

- [Overview](#overview)
- [Getting Started](#getting-started)
- [Prerequisites (Linux)](#prerequisites-linux)
- [Prerequisites (macOS)](#prerequisites-macos)
- [Vagrant Variables (Useful Overrides)](#vagrant-variables-useful-overrides)
- [Optional: NFS for `artefacts/` (VirtualBox)](#optional-nfs-for-artefacts-virtualbox)
- [First Successful Build](#first-successful-build)
- [Build Pipeline](#build-pipeline)
- [1. Build Toolchain](#1-build-toolchain)
- [2. Build SDK](#2-build-sdk)
- [3. Build Project Artefact](#3-build-project-artefact)
- [4. Build Firmware](#4-build-firmware)
- [Disk Images & Deployment](#disk-images-deployment)
- [Generate firmware + disk images](#generate-firmware-disk-images)
- [Write firmware to device with `fwup`](#write-firmware-to-device-with-fwup)
- [Convert `.fw` to `.img`](#convert-fw-to-img)
- [Inspect image partitions](#inspect-image-partitions)
- [Troubleshooting](#troubleshooting)
- [Advanced Use Cases](#advanced-use-cases)
- [Platform Source-of-Truth Notes](#platform-source-of-truth-notes)

## Getting Started

### Prerequisites (Linux)
Expand All @@ -58,16 +81,46 @@ macOS is supported via Vagrant VM execution:

```sh
brew install vagrant qemu
brew install --cask virtualbox
```

The scripts start the VM automatically when needed. Use `-P` to reprovision.

**Troubleshooting (macOS / VirtualBox):** On Ruby 3.2+, run **`./scripts/vagrant/patch-vagrant-vbguest-ruby3.sh`**
on the host once (after **`vagrant plugin install vagrant-vbguest`**) so the gem stops using the
removed **`File.exists?`** API. For a quick environment snapshot, run **`./scripts/vagrant/vagrant-diagnose.sh --compact`**;
for a full report (for example when opening an issue), run **`./scripts/vagrant/vagrant-diagnose.sh`**.
The `Vagrantfile` sets `vagrant-vbguest` **`auto_update` to false** by default on Ruby 3.2+ so an
unpatched gem does not hit the broken path; after patching, you can enable Guest Additions updates
with **`VAGRANT_VBGUEST_AUTO_UPDATE=1 vagrant up`**.

Vagrant-only helpers live under **`scripts/vagrant/`**. Builds invoked via `vagrant exec` use
**`GLB_VAGRANT_REPO_ROOT`** (default **`/vagrant`** on VirtualBox with rsync), so inside the VM
the same scripts are at **`/vagrant/scripts/vagrant/`** (for example
`/vagrant/scripts/vagrant/vagrant-diagnose.sh`). Use that path when you need the tree that stays
in sync with the host; the provisioned copy at `/home/vagrant/scripts/` updates when you run
`vagrant provision` or `./build-*.sh -P`.

### Vagrant Variables (Useful Overrides)

You can control `Vagrantfile` behavior with environment variables:

- `VAGRANT_DEFAULT_PROVIDER`: force provider (for example `virtualbox`)
- `VAGRANT_VB_CACHE_STORAGectl`, `VAGRANT_VB_CACHE_PORT`, `VAGRANT_VB_CACHE_DEVICE`: VirtualBox
cache-disk attachment. If **`VAGRANT_VB_CACHE_STORAGectl`** is unset, the name comes from the box
**OVF** under **`~/.vagrant.d/boxes/`**, then from **`storagecontrollername0`** in **`showvminfo`**.
On **`vagrant up`** / **`reload`**, if the box is not cached yet, the **`Vagrantfile` runs
**`vagrant box add`** once so the OVF exists (disable with **`VAGRANT_SKIP_BOX_PREFETCH=1`**). You
can still set **`VAGRANT_VB_CACHE_STORAGectl`** to override. Port / device default to **`1`** /
**`0`**.
- `VAGRANT_DISABLE_NFS=1`: disable NFS path for VirtualBox and use default shared folders
- `GLB_VAGRANT_REPO_ROOT`: repo path inside the guest for `vagrant exec` (default **`/vagrant`**;
use **`/home/vagrant`** on VMware if there is no `/vagrant` mount)
- `VAGRANT_VBGUEST_AUTO_UPDATE=1`: turn `vagrant-vbguest` automatic Guest Additions updates back on
- `VAGRANT_VIRTUALBOX_SYNC_TYPE`: `virtualbox` (vboxsf) or `rsync` (on **Apple Silicon** Macs,
VirtualBox often defaults to **rsync**; Intel-based Macs may use either); with rsync, macOS syncs
`./artefacts/` via **`scripts/common.sh`**
- `VAGRANT_USE_NFS`, `VAGRANT_DISABLE_NFS`: NFS for `artefacts/` on VirtualBox (see subsection below)
- `VM_MEMORY`: VM RAM in MB (default `16384`)
- `VM_CORES`: VM CPU cores (default `8`)
- `VM_PRIMARY_DISK_SIZE`: primary VM disk size (for example `96GB`)
Expand All @@ -76,37 +129,19 @@ You can control `Vagrantfile` behavior with environment variables:
Examples:

```sh
# VirtualBox on Linux: `VAGRANT_DISABLE_NFS=1` skips the NFS-backed `artefacts/` sync (no host-only adapter).
VAGRANT_DEFAULT_PROVIDER=virtualbox VAGRANT_DISABLE_NFS=1 ./build-toolchain.sh grisp2
# macOS + VirtualBox: omit `VAGRANT_DISABLE_NFS=1` (typical on Apple Silicon and Intel; see NFS/rsync above).
VM_MEMORY=8192 VM_CORES=4 ./build-sdk.sh grisp2
VM_PRIMARY_DISK_SIZE=96GB VM_CACHE_DISK_SIZE=20480 ./build-toolchain.sh -P grisp2
```

### Optional: Enable NFS (VirtualBox)

NFS can improve shared-folder performance, but VirtualBox needs a host-only network.

Requirements:

- A VirtualBox host-only adapter exists (with DHCP or static IP).
- `VAGRANT_DISABLE_NFS` is not set to `1`.

Run with NFS:

```sh
VAGRANT_DEFAULT_PROVIDER=virtualbox ./build-toolchain.sh -P grisp2
```

If you get:

```text
NFS requires a host-only network to be created.
```

create a host-only adapter in VirtualBox and reprovision (`-P`), or use:
### Optional: NFS for `artefacts/` (VirtualBox)

```sh
VAGRANT_DEFAULT_PROVIDER=virtualbox VAGRANT_DISABLE_NFS=1 ./build-toolchain.sh grisp2
```
NFS can be faster than `vboxsf` but needs a **host-only network** in VirtualBox. Set
`VAGRANT_USE_NFS=1` (and do not force `rsync` sync). If you see `NFS requires a host-only network`,
either add that adapter in VirtualBox or stay on the default **rsync** (common on **Apple Silicon**
Macs) or **VirtualBox shared folders** (other hosts, including Intel-based Macs).

### First Successful Build

Expand Down
115 changes: 92 additions & 23 deletions Vagrantfile
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,10 @@
################################################################################

require 'rbconfig'
require 'fileutils'

# Host path for rsync / VirtualBox shared folder (must exist on the host).
FileUtils.mkdir_p(File.expand_path('artefacts', __dir__))

VM_PRIMARY_DISK_SIZE = (ENV['VM_PRIMARY_DISK_SIZE']&.strip)
VALID_DISK_SIZE = /\A\d+\s*(KB|MB|GB|TB)\z/i
Expand Down Expand Up @@ -35,6 +39,26 @@ def host_platform
end
end

# True on Apple Silicon Macs, including a Terminal/iTerm running under Rosetta (host_cpu
# may report x86_64 but sysctl still shows e.g. "Apple M3 Pro").
def macos_apple_silicon_for_vbox_rsync?
return false unless host_platform == :macos
return true if is_arm64?

brand = `sysctl -n machdep.cpu.brand_string 2>/dev/null`.to_s
brand.match?(/Apple M\d/)
end

# VirtualBox: `virtualbox` sync needs Guest Additions (vboxsf). On Apple Silicon, vboxsf
# often fails until GA match the guest kernel. `rsync` avoids vboxsf. Override with
# VAGRANT_VIRTUALBOX_SYNC_TYPE=virtualbox or rsync.
def virtualbox_sync_type
v = ENV['VAGRANT_VIRTUALBOX_SYNC_TYPE'].to_s.strip
return v if %w[rsync virtualbox].include?(v)

macos_apple_silicon_for_vbox_rsync? ? 'rsync' : 'virtualbox'
end

def ensure_vagrant_cache_disk
cache_path = File.expand_path("#{CACHE_DISK_BASENAME}.vmdk", __dir__)
cache_size = "#{CACHE_DISK_SIZE}M"
Expand Down Expand Up @@ -87,25 +111,58 @@ def ensure_vagrant_cache_disk
cache_path
end

def register_vmdk_for_virtualbox(path)
unless File.exist?(path)
raise "VMDK file #{path} not found on disk"
end
if system("VBoxManage showmediuminfo '#{path}' > /dev/null 2>&1")
return
end
puts "Registering #{path} as VirtualBox medium..."
unless system("VBoxManage openmedium disk '#{path}' --format VMDK")
raise "Failed to register VMDK disk with VirtualBox (see VBoxManage output above)"
# VirtualBox cache VMDK: register medium; storagectl from env / box OVF / showvminfo (README).
def register_vmdk_for_virtualbox_if_needed(path)
return unless ENV['VAGRANT_DEFAULT_PROVIDER'] == 'virtualbox' ||
ARGV.any? { |arg| arg.include?('virtualbox') }
raise "VMDK file #{path} not found on disk" unless File.exist?(path)
return if system("VBoxManage showmediuminfo '#{path}' > /dev/null 2>&1")

puts "Registering #{path} as VirtualBox medium..."
return if system("VBoxManage openmedium disk '#{path}' --format VMDK")

raise 'Failed to register VMDK disk with VirtualBox (see VBoxManage output above)'
end

def virtualbox_cache_storagectl_name(repo_dir, box_name)
manual = ENV['VAGRANT_VB_CACHE_STORAGectl'].to_s.strip
return manual unless manual.empty?

strict = %w[up reload].include?(ARGV[0])
boxes = File.expand_path('~/.vagrant.d/boxes')
slug = box_name.gsub('/', '-VAGRANTSLASH-')
glob = File.join(boxes, slug, '**', 'virtualbox', '*.ovf')
ovf = Dir.exist?(boxes) ? Dir.glob(glob).max_by { |f| File.mtime(f) } : nil
if strict && ENV['VAGRANT_SKIP_BOX_PREFETCH'] != '1' && !ovf
warn "grisp_alloy: downloading base box #{box_name} (OVF for cache disk)…"
unless system('vagrant', 'box', 'add', box_name, '--provider', 'virtualbox')
raise "grisp_alloy: vagrant box add #{box_name} failed; run it manually."
end
ovf = Dir.glob(glob).max_by { |f| File.mtime(f) }
end
if ovf && File.file?(ovf)
m = File.read(ovf).match(/<StorageController\s+[^>]*\bname="([^"]*)"/i)
return m[1] if m
end
idf = File.expand_path('.vagrant/machines/default/virtualbox/id', repo_dir)
if File.exist?(idf)
out = `VBoxManage showvminfo "#{File.read(idf).strip}" --machinereadable 2>/dev/null`
m = out.match(/^storagecontrollername0="([^"]*)"/m)
return m[1] if m
end
return 'SATA Controller' unless strict

raise 'Cannot resolve VirtualBox storagectl for cache VMDK. Set VAGRANT_VB_CACHE_STORAGectl (README).'
end

CACHE_DISK_PATH = ensure_vagrant_cache_disk

VM_BOX = 'bento/ubuntu-24.04'

Vagrant.configure('2') do |config|

required_plugins = %w( vagrant-scp vagrant-exec )
config.vm.box = "bento/ubuntu-24.04"
config.vm.box = VM_BOX

if VM_PRIMARY_DISK_SIZE
config.vm.disk :disk, size: VM_PRIMARY_DISK_SIZE, primary: true
Expand Down Expand Up @@ -151,14 +208,12 @@ Vagrant.configure('2') do |config|
v.cpus = VM_CORES
required_plugins = %w( vagrant-vbguest )

if ENV['VAGRANT_DEFAULT_PROVIDER'] == 'virtualbox' || ARGV.any? { |arg| arg.include?('virtualbox') }
register_vmdk_for_virtualbox(CACHE_DISK_PATH)
end

# Base boxes already ship with a SATA controller; re-adding it fails.
# Attach the cache disk on the existing controller.
v.customize ['storageattach', :id, '--storagectl', 'SATA Controller',
'--port', 1, '--device', 0, '--type', 'hdd',
register_vmdk_for_virtualbox_if_needed(CACHE_DISK_PATH)
vb_ctl = virtualbox_cache_storagectl_name(__dir__, VM_BOX)
vb_port = Integer(ENV.fetch('VAGRANT_VB_CACHE_PORT', '1'))
vb_dev = Integer(ENV.fetch('VAGRANT_VB_CACHE_DEVICE', '0'))
v.customize ['storageattach', :id, '--storagectl', vb_ctl,
'--port', vb_port, '--device', vb_dev, '--type', 'hdd',
'--medium', CACHE_DISK_PATH]
end

Expand Down Expand Up @@ -288,11 +343,25 @@ Vagrant.configure('2') do |config|
end

config.vm.provider :virtualbox do |v, override|
if ENV['VAGRANT_DISABLE_NFS'] == '1'
# Use VirtualBox shared folders when NFS is explicitly disabled.
override.vm.synced_folder "artefacts/", "/home/vagrant/artefacts", create: true
vt = virtualbox_sync_type
if vt == 'rsync'
# No vboxsf: host must have `rsync` (macOS/Linux do). After builds, run
# Guest→host sync uses `rsync` over SSH (see `vagrant_sync_artefacts_from_guest` in
# scripts/common.sh); `vagrant rsync` only pushes host→guest.
rsync_excl = [
'.git/', '.vagrant/', 'output/', '_build/', '_cache/', '.cursor/',
'.vagrant.cache.vmdk', '.vagrant.cache.raw'
]
override.vm.synced_folder '.', '/vagrant', type: 'rsync',
rsync__exclude: rsync_excl, rsync__auto: true
override.vm.synced_folder 'artefacts/', '/home/vagrant/artefacts', create: true,
type: 'rsync', rsync__auto: true
elsif ENV['VAGRANT_USE_NFS'] == '1' && ENV['VAGRANT_DISABLE_NFS'] != '1'
override.vm.synced_folder 'artefacts/', '/home/vagrant/artefacts', create: true,
type: 'nfs', nfs_version: 3, nfs_udp: false,
mount_options: ['vers=3,tcp']
else
override.vm.synced_folder "artefacts/", "/home/vagrant/artefacts", create: true, type: "nfs", nfs_version: 3, nfs_udp: false, mount_options: ['vers=3,tcp']
override.vm.synced_folder 'artefacts/', '/home/vagrant/artefacts', create: true
end
end

Expand Down
8 changes: 6 additions & 2 deletions build-firmware.sh
Original file line number Diff line number Diff line change
Expand Up @@ -355,8 +355,12 @@ if [[ $ARG_FORCE_VAGRANT == true ]] || [[ $HOST_OS != "linux" ]]; then
trap "vagrant exec rm -rf '$GLB_VAGRANT_FIRMWARE_BUILD_DIR/secpack'" EXIT
fi
cd "$GLB_TOP_DIR"
vagrant exec -- "${GLB_VAGRANT_TOP_DIR}/build-firmware.sh" "${NEW_ARGS[@]}"
exit $?
vagrant exec -- "${GLB_VAGRANT_REPO_ROOT}/build-firmware.sh" "${NEW_ARGS[@]}"
rc=$?
if [[ "$HOST_OS" == "darwin" ]]; then
vagrant_sync_artefacts_from_guest || true
fi
exit "$rc"
fi

# NATIVE LINUX EXECUTION STARTS HERE
Expand Down
8 changes: 6 additions & 2 deletions build-project.sh
Original file line number Diff line number Diff line change
Expand Up @@ -147,8 +147,12 @@ if [[ $ARG_FORCE_VAGRANT == true ]] || [[ $HOST_OS != "linux" ]]; then
trap "cd '$GLB_TOP_DIR'; vagrant halt" EXIT
fi
cd "$GLB_TOP_DIR"
vagrant exec "${GLB_VAGRANT_TOP_DIR}/build-project.sh" "${NEW_ARGS[@]}"
exit $?
vagrant exec "${GLB_VAGRANT_REPO_ROOT}/build-project.sh" "${NEW_ARGS[@]}"
rc=$?
if [[ "$HOST_OS" == "darwin" ]]; then
vagrant_sync_artefacts_from_guest || true
fi
exit "$rc"
fi

# NATIVE LINUX EXECUTION STARTS HERE
Expand Down
8 changes: 6 additions & 2 deletions build-sdk.sh
Original file line number Diff line number Diff line change
Expand Up @@ -119,8 +119,12 @@ if [[ $ARG_FORCE_VAGRANT == true ]] || [[ $HOST_OS != "linux" ]]; then
if [[ $ARG_KEEP_VAGRANT == false ]]; then
trap "cd '$GLB_TOP_DIR'; vagrant halt" EXIT
fi
vagrant exec "${GLB_VAGRANT_TOP_DIR}/build-sdk.sh" "${NEW_ARGS[@]}"
exit $?
vagrant exec "${GLB_VAGRANT_REPO_ROOT}/build-sdk.sh" "${NEW_ARGS[@]}"
rc=$?
if [[ "$HOST_OS" == "darwin" ]]; then
vagrant_sync_artefacts_from_guest || true
fi
exit "$rc"
fi

# NATIVE LINUX EXECUTION STARTS HERE
Expand Down
11 changes: 9 additions & 2 deletions build-toolchain.sh
Original file line number Diff line number Diff line change
Expand Up @@ -103,8 +103,15 @@ if [[ $ARG_FORCE_VAGRANT = true ]] || [[ $HOST_OS != "linux" ]]; then
if [[ $ARG_KEEP_VAGRANT == false ]]; then
trap "cd '$GLB_TOP_DIR'; vagrant halt" EXIT
fi
vagrant exec "${GLB_VAGRANT_TOP_DIR}/build-toolchain.sh" "${NEW_ARGS[@]}"
exit $?
vagrant exec "${GLB_VAGRANT_REPO_ROOT}/build-toolchain.sh" "${NEW_ARGS[@]}"
rc=$?
# VirtualBox rsync (default on Apple Silicon) syncs host→guest only; the toolchain writes
# to artefacts/ inside the VM. Sync guest→host so ./artefacts/ on the Mac is updated.
# Linux native / vboxsf do not need this.
if [[ "$HOST_OS" == "darwin" ]]; then
vagrant_sync_artefacts_from_guest || true
fi
exit "$rc"
fi

# NATIVE LINUX EXECUTION STARTS HERE
Expand Down
Loading