Skip to content

Commit 9b1bcd8

Browse files
madeline-underwoodmadeline-underwood
committed
Updates
1 parent 889903d commit 9b1bcd8

2 files changed

Lines changed: 314 additions & 0 deletions

File tree

‎content/learning-paths/servers-and-cloud-computing/cca-device-attach/3.bounce_buffers.md‎

Lines changed: 312 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,312 @@
1+
---
2+
title: "Explore Bounce Buffers"
3+
weight: 4
4+
5+
### FIXED, DO NOT MODIFY
6+
layout: learningpathall
7+
---
8+
9+
## Overview
10+
11+
This section introduces VirtIO and Bounce Buffers in the context of CCA Realms, and explains how they enable secure data exchange between a Realm and the untrusted external world.
12+
13+
A Realm must eventually use physical devices to interact with the external world. The easiest way to achieve this is by using VirtIO, which provides a fast, high-level emulation layer. This is considered the first level of device attach, where access is mediated by the hypervisor using paravirtualized interfaces.
14+
15+
More advanced device attach features can be enabled by hardware security features like PCIe-TDISP (**T**EE **D**evice **I**nterface **S**ecurity **P**rotocol) and PCIe-IDE (**I**ntegrity
16+
and **D**ata **E**ncryption), where the host OS assigns a physical device to a Realm. The Realm can then make security measurements on the physical device and include those in its attestation base.
17+
18+
### What is VirtIO?
19+
20+
VirtIO provides an efficient, paravirtualized I/O interface between Realms and host devices. It is an abstraction layer for virtual devices in virtualized environments. It provides standardized and efficient interfaces between guest virtual machines (VMs) and host devices, making it easier to develop paravirtualized drivers.
21+
22+
Paravirtualized means that the guest OS is aware it’s running in a virtualized environment and can use optimized drivers (VirtIO) to communicate with virtual hardware. Emulating physical hardware devices (like NICs or disks) for VMs is slow and inefficient. VirtIO allows VMs to bypass full device emulation and use streamlined drivers.
23+
24+
VirtIO is most commonly used with KVM/QEMU virtualization.
25+
26+
Example drivers include:
27+
28+
- `virtio-net`: Paravirtualized networking
29+
- `virtio-blk`: Block device (disk) access
30+
- `virtio-fs`: File sharing (host ↔ guest)
31+
- `virtio-balloon`: Dynamic memory management
32+
- `virtio-rng`: Random number source
33+
- `virtio-console`: Simple console interface
34+
35+
### How does VirtIO work in VMs?
36+
37+
Here is an overview of how VirtIO works in Virtual Machines:
38+
39+
1. The Host Hypervisor (for example, QEMU/KVM) exposes VirtIO backend devices.
40+
2. The guest OS loads VirtIO _frontend_ drivers (for example, `virtio_net`,
41+
`virtio_blk`) that communicate using the VirtIO protocol.
42+
3. Communication happens via shared memory (`virtqueues`) for I/O operations,
43+
avoiding full device emulation.
44+
4. Devices are exposed over the PCI or MMIO bus to the guest.
45+
46+
For example, instead of emulating an Intel e1000 NIC, the host exposes a `virtio-net` interface, and the guest OS uses the `virtio-net` driver to send and receive packets via shared buffers.
47+
48+
## Bounce buffers
49+
50+
### What are bounce buffers?
51+
52+
Bounce buffers are temporary memory buffers used when Direct Memory Access (DMA) cannot be performed directly on the original data buffer. This might be because:
53+
54+
1. The original buffer is not physically contiguous
55+
2. The buffer is in high memory or not accessible to the device
56+
3. The buffer does not meet alignment or boundary constraints required by the device
57+
58+
## Why use bounce buffers?
59+
60+
Data _bounces_ between:
61+
- The original buffer (in user or kernel space) and
62+
- The DMA-capable bounce buffer (used for I/O with the device)
63+
64+
This ensures that data transfer is possible even when the original memory isn’t suitable for DMA.
65+
66+
## CCA Realms, VirtIO and bounce buffers
67+
68+
The defining feature of a Realm is that its memory (called *Realm memory*) is cryptographically isolated from both the Normal and Secure Worlds.
69+
70+
This means that:
71+
- Realm memory is encrypted with unique keys
72+
- Non-Realm entities (host OS, hypervisor) cannot directly read or
73+
write to Realm memory
74+
- Even Direct Memory Access (DMA) from peripherals or untrusted drivers cannot
75+
access Realm data
76+
77+
This design ensures confidentiality but presents a challenge: how can a Realm securely interact with untrusted components, such as network stacks in the host OS, storage subsystems, or I/O devices managed by untrusted drivers? To exchange data securely with untrusted components (for example, network stacks, storage subsystems), Realms use *bounce buffers* as intermediaries.
78+
79+
### How bounce buffers are used with RME
80+
81+
1. Exporting Data:
82+
- A Realm application prepares some data (for example, the results of computation)
83+
- It copies this data from protected Realm memory into a bounce buffer.
84+
- The Realm notifies the untrusted host or hypervisor that the data is ready.
85+
- The host retrieves the data from the bounce buffer.
86+
87+
2. Importing Data:
88+
- The host places data (for example, input from a file or device) into a bounce buffer.
89+
- The Realm is notified and validates the source.
90+
- The Realm copies the data from the bounce buffer into its protected memory.
91+
92+
This pattern preserves confidentiality and integrity of Realm data, since:
93+
- The Realm never allows direct access to its memory.
94+
- It can validate and sanitize any data received via bounce buffers.
95+
- No sensitive data is exposed without explicit copying.
96+
97+
### Is confidentiality preserved with bounce buffers?
98+
99+
In the previous section, it was mentioned that bounce buffers preserve confidentiality. Lets dive a little deeper into that. Bounce buffers are nothing more than an explicitly shared temporary area between the Realm world and the outside world. This does indeed preserve the confidentiality of the rest of the Realm data. On the other hand, for the data being transferred, it is leaving the Realm world and will only remain confidential if it is encrypted in some way, for example, for network traffic, TLS should be used.
100+
101+
## Seeing a Realm's bounce buffers at work
102+
103+
Let's put this to work and check for ourselves that bounce buffers are used. The steps in this section will build on the Key Broker demo that was used in the [CCA
104+
Essentials learning path](/learning-paths/servers-and-cloud-computing/cca-essentials/example/),
105+
demonstrating an end-to-end attestation.
106+
107+
### Start the Key Broker Server (KBS)
108+
109+
First, pull the docker container image with the pre-built KBS, and then run the container:
110+
111+
```bash
112+
docker pull armswdev/cca-learning-path:cca-key-broker-v2
113+
docker run --rm -it armswdev/cca-learning-path:cca-key-broker-v2
114+
```
115+
116+
Now within your running docker container, get a list of network interfaces:
117+
118+
```bash
119+
ip -c a
120+
```
121+
122+
The output should look like:
123+
124+
```output
125+
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
126+
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
127+
inet 127.0.0.1/8 scope host lo
128+
valid_lft forever preferred_lft forever
129+
inet6 ::1/128 scope host
130+
valid_lft forever preferred_lft forever
131+
20: eth0@if21: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
132+
link/ether 02:42:ac:11:00:02 brd ff:ff:ff:ff:ff:ff link-netnsid 0
133+
inet 172.17.0.2/16 brd 172.17.255.255 scope global eth0
134+
valid_lft forever preferred_lft forever
135+
```
136+
137+
Start the KBS on the `eth0` network interface, and replace 172.17.0.2 shown in
138+
the command below with the IP address corresponding to eth0 in the output of `ip
139+
-c a` above.
140+
141+
```bash
142+
./keybroker-server -v --addr 172.17.0.2
143+
```
144+
145+
The output should look like:
146+
147+
```output
148+
INFO starting 16 workers
149+
INFO Actix runtime found; starting in Actix runtime
150+
INFO starting service: "actix-web-service-172.17.0.2:8088", workers: 16, listening on: 172.17.0.2:8088
151+
```
152+
153+
### Get into a Realm
154+
155+
With the Key Broker Server running in one terminal, open up a new terminal in
156+
which you will run the Key Broker Client (KBC). The intent is to
157+
observe that the data transmitted over the network (thru `virtio_net`) are
158+
indeed using bounce buffers.
159+
160+
Pull the docker container image with the pre-built KBC, and then run the container:
161+
162+
```bash
163+
docker pull armswdev/cca-learning-path:cca-simulation-v2
164+
docker run --rm -it armswdev/cca-learning-path:cca-simulation-v2
165+
```
166+
167+
Within the running container, launch the `run-cca-fvp.sh` script to run the Arm
168+
CCA pre-built binaries on the FVP:
169+
170+
```bash
171+
./run-cca-fvp.sh
172+
```
173+
The `run-cca-fvp.sh` script uses the screen command to connect to the different
174+
UARTs in the FVP.
175+
176+
You should see the host Linux kernel boot on your terminal and you will be
177+
prompted to log in to the host. Enter root as the username:
178+
179+
```output
180+
[ 4.169458] Run /sbin/init as init process
181+
[ 4.273748] EXT4-fs (vda): re-mounted 64d1bcff-5d03-412c-83c6-48ec4253590e r/w. Quota mode: none.
182+
Starting syslogd: OK
183+
Starting klogd: OK
184+
Running sysctl: OK
185+
Starting network: [ 5.254843] smc91x 1a000000.ethernet eth0: link up, 10Mbps, half-duplex, lpa 0x0000
186+
udhcpc: started, v1.36.1
187+
udhcpc: broadcasting discover
188+
udhcpc: broadcasting select for 172.20.51.1, server 172.20.51.254
189+
udhcpc: lease of 172.20.51.1 obtained from 172.20.51.254, lease time 86400
190+
deleting routers
191+
adding dns 172.20.51.254
192+
OK
193+
194+
Welcome to the CCA host
195+
host login: root
196+
(host) #
197+
```
198+
199+
Change directory to `/cca` and use `lkvm` to launch a guest Linux in a Realm:
200+
```bash
201+
cd /cca
202+
./lkvm run --realm --disable-sve --irqchip=gicv3-its --firmware KVMTOOL_EFI.fd -c 1 -m 512 --no-pvtime --disk guest-disk.img --restricted_mem --virtio-transport pci --pmu --network mode=user
203+
```
204+
205+
You should see the realm boot. Note that `lkvm` is invoked with `--network
206+
mode=user`, which makes the guest see the network through a VirtIO device.
207+
208+
After boot up, which might take some time, you will be prompted to log in at the
209+
guest Linux prompt. Use root again as the username:
210+
211+
```output
212+
Starting syslogd: OK
213+
Starting klogd: OK
214+
Running sysctl: OK
215+
Starting network: udhcpc: started, v1.36.1
216+
udhcpc: broadcasting discover
217+
udhcpc: broadcasting select for 192.168.33.15, server 192.168.33.1
218+
udhcpc: lease of 192.168.33.15 obtained from 192.168.33.1, lease time 14400
219+
deleting routers
220+
adding dns 172.20.51.254
221+
OK
222+
223+
Welcome to the CCA realm
224+
realm login: root
225+
(realm) #
226+
```
227+
228+
### Observe bounce buffer usage in the realm
229+
230+
First, check that the Linux kernel has tracing support:
231+
232+
```bash { output_lines="2-46" }
233+
ls /sys/kernel/debug/tracing/events/
234+
9p i2c_slave qcom_glink
235+
alarmtimer icmp qcom_smp2p
236+
asoc initcall qdisc
237+
block interconnect ras
238+
bpf_test_run io_uring raw_syscalls
239+
bpf_trace iomap rcu
240+
bridge iommu regmap
241+
capability ipi regulator
242+
cgroup irq rpcgss
243+
chipidea jbd2 rpm
244+
clk kmem rpmh
245+
cma ksm rseq
246+
compaction kvm rtc
247+
cpuhp kyber sched
248+
cros_ec libata scmi
249+
csd lock scsi
250+
dev lockd signal
251+
devfreq maple_tree skb
252+
devlink mdio smbus
253+
dma memcg sock
254+
dma_fence migrate spi
255+
dpaa2_eth mmap spmi
256+
dpaa_eth mmap_lock sunrpc
257+
dwc3 mmc swiotlb
258+
e1000e_trace module task
259+
enable mtu3 tcp
260+
error_report musb tegra_apb_dma
261+
ext4 napi thermal
262+
fib neigh thermal_power_allocator
263+
filelock net thp
264+
filemap netfs timer
265+
fsl_edma netlink timer_migration
266+
ftrace nfs timestamp
267+
gadget nfs4 tlb
268+
gpio notifier udp
269+
gpu_mem oom ufs
270+
handshake optee vmalloc
271+
header_event page_isolation vmscan
272+
header_page page_pool watchdog
273+
hns3 pagemap workqueue
274+
huge_memory percpu writeback
275+
hugetlbfs power xdp
276+
hw_pressure printk xhci-hcd
277+
hwmon pwm
278+
i2c qcom_aoss
279+
```
280+
281+
As shown above, you should see a list of the available trace
282+
points.
283+
284+
Now, enable the kernel tracing infrastructure together with bounce buffer
285+
tracing, read the trace in the background (filtering on `keybroker-app-`) and
286+
run the Key Broker Client application in the realm, using the endpoint address
287+
that the Key Broker Server is listening on (from the other terminal):
288+
289+
```bash
290+
echo 1 > /sys/kernel/debug/tracing/tracing_on
291+
echo 1 > /sys/kernel/debug/tracing/events/swiotlb/enable
292+
grep keybroker-app- /sys/kernel/debug/tracing/trace_pipe &
293+
keybroker-app -v --endpoint http://172.17.0.2:8088 skywalker
294+
```
295+
296+
In the `keybroker-app`command above, `skywalker` is the key name that is
297+
requested from the KBS.
298+
299+
The output should look like:
300+
301+
```output
302+
INFO Requesting key named 'skywalker' from the keybroker server with URL http://172.17.0.2:8088/keys/v1/key/skywalker
303+
INFO Challenge (64 bytes) = [5c, ec, 1e, f5, 93, 54, 4a, 8a, ee, 2e, 46, a0, 50, 0d, 41, dd, d4, 60, b0, 58, 5b, 51, 71, 76, d1, 66, d3, b7, 38, e8, af, ae, 0a, 07, 4e, c5, 60, dc, 4a, c0, b8, 73, 98, d9, bd, af, 41, 96, 99, 6d, 74, cc, 19, 70, 24, c4, c9, 5c, 21, 61, 1a, cb, 76, 75]
304+
INFO Submitting evidence to URL http://172.17.0.2:8088/keys/v1/evidence/1928844131
305+
INFO Attestation success :-) ! The key returned from the keybroker is 'May the force be with you.'
306+
keybroker-app-143 [000] b..2. 1772.607321: swiotlb_bounced: dev_name: 0000:00:00.0 dma_mask=ffffffffffffffff dev_addr=80b6717e size=66 FORCE
307+
keybroker-app-143 [000] b..2. 1772.644478: swiotlb_bounced: dev_name: 0000:00:00.0 dma_mask=ffffffffffffffff dev_addr=80b6717e size=66 FORCE
308+
```
309+
310+
Note that the interleaving of the trace messages and KBC messages might differ
311+
from one run to another. With the `switlb_bounced` messages above you can successfully observe that the bounce buffers are being used in the realm.
312+

‎content/learning-paths/servers-and-cloud-computing/cca-device-attach/_index.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,8 +6,10 @@ minutes_to_complete: 90
66
who_is_this_for: This is an advanced topic for developers who want to understand how Arm CCA Realms interact with I/O devices using VirtIO, bounce buffers, and secure device attach mechanisms.
77

88
learning_objectives:
9+
- Explain the two levels of device attach for Realms
910
- Understand the role of VirtIO and Bounce Buffers in CCA Realms
1011
- Explain what device attach means and how it applies to different types of devices
12+
- Describe how VirtIO enables paravirtualized I/O without full device emulation
1113

1214
prerequisites:
1315
- An AArch64 or x86_64 computer running Linux or macOS. You can also use a cloud instance from one of these [Arm cloud service providers](/learning-paths/servers-and-cloud-computing/csp/).

0 commit comments

Comments
 (0)