🤖 Embodied Multimedia

When Multimedia Meets Embodied Intelligence

Wei Zuo^*1 · Yang Liu^*2 · Weniang Yang¹ · Feng Wu¹ · Wei Zhou³ · Jing Liu^†4 · Peng Sun⁵ · Jing Cheng^†6 · Dingkang Yang^†1 · Xinhua Zeng^†1

¹Fudan University  |  ²Tongji University  |  ³Cardiff University  |  ⁴University of British Columbia  |  ⁵Duke Kunshan University  |  ⁶East China Normal University

^*Equal contribution    ^†Corresponding authors

---

Introduction • Motivation • Architecture • Key Technologies • Evaluation • Challenges • Citation

---

📌 Introduction

The explosive rise of Embodied Intelligence — AI that perceives, reasons, and acts within the physical world — demands a fundamental rethink of how we create, transmit, and evaluate multimedia data.

Traditional multimedia was built for human observers: compressing what the eye cannot see, streaming what the ear can tolerate, and measuring quality by human perceptual standards. But when an embodied agent — a robot arm, an autonomous vehicle, a household assistant — relies on that same data to make real-time physical decisions, the entire pipeline breaks down.

"Multimedia data is no longer merely content for human consumption; it serves as the essential sensory foundation that bridges an agent's cognition and physical execution."

This survey formally introduces Embodied Multimedia: a unified paradigm that re-engineers the entire multimedia stack — from data acquisition to communication to cognition to evaluation — to serve not human audiences, but embodied intelligent agents.

Figure 1: The four-layer Embodied Multimedia architecture and survey organization.

---

🔍 Why Embodied Multimedia?

Modern multimedia systems are fundamentally misaligned with the requirements of embodied agents. The mismatch spans every layer of the pipeline:

Dimension	🖥️ Traditional Multimedia	🤖 Embodied Multimedia
Target User	Human observers	Embodied agents & robots
Core Goal	Optimize perceptual experience	Optimize task execution
Data Acquisition	Passive, fixed viewpoints	Active, dynamic, semantic-aware
Data Processing	Discard imperceptible details	Retain task-critical features
Communication	Bit-level reliable transmission	Task-oriented semantic communication
Retrieval	Static feature matching	Dynamic agent-based reasoning
Generative Models	Aesthetically pleasing content	Physically grounded, actionable outputs
Evaluation	PSNR, SSIM, FID	Task success rate, interaction safety

🚨 The Core Problems

• Data Acquisition Gap — Traditional compression standards (JPEG, HEVC) discard high-frequency texture and edge information that human eyes miss but machine vision critically needs. Fixed-viewpoint passive capture cannot resolve occlusions that block scene understanding.

• Communication Bottleneck — Shannon-theoretic bit-level transmission paradigms cannot meet millisecond-level spatio-temporal synchronization demands of real-time physical interaction. Centralized cloud processing introduces prohibitive latency.

• Retrieval Limitation — Static cross-modal retrieval lacks closed-loop reasoning, making it unable to support the ambiguous, long-horizon, knowledge-intensive tasks embodied agents routinely face.

• Generative Grounding Failure — GANs and diffusion models trained on aesthetic objectives lack physical law understanding. Embodied agents need models that generate actionable, physics-consistent outputs.

• Evaluation Blindspot — PSNR and SSIM measure pixel fidelity, not task utility. Visually degraded data may still perfectly support a successful robotic operation if semantic features are preserved — traditional metrics miss this entirely.

---

🏗️ Architecture

We propose a unified four-layer Embodied Multimedia architecture that covers the complete information processing workflow from sensory input to physical action:

┌─────────────────────────────────────────────────────────────┐
│                    EMBODIED ENVIRONMENT                     │
│              (Home · Hospital · Road · Factory)             │
└──────────────────────────┬──────────────────────────────────┘
                           │ Physical Interaction
        ┌──────────────────▼──────────────────┐
        │          EVALUATION LAYER           │  ← §7
        │  Perception · Cognition · Action    │
        └──────────────────┬──────────────────┘
                           │ Feedback
        ┌──────────────────▼──────────────────┐
        │          COGNITIVE LAYER            │  ← §6
        │  Retrieval · World Model · VLN · VLA│
        └──────────────────┬──────────────────┘
                           │ Semantic Features
        ┌──────────────────▼──────────────────┐
        │        COMMUNICATION LAYER          │  ← §5
        │   Semantic Comm. · Edge-Cloud ECC   │
        └──────────────────┬──────────────────┘
                           │ Raw/Compressed Data
        ┌──────────────────▼──────────────────┐
        │            DATA LAYER               │  ← §4
        │  Acquisition · Enhancement · Codec  │
        └─────────────────────────────────────┘
                           ▲
                    Multimodal Sensors
               (Camera · LiDAR · Touch · IMU)

📊 Layer-by-Layer Summary

🗃️ Data Layer

The perceptual foundation of the entire system. Covers:

• Adaptive Acquisition — Active perception theory; event cameras; semantic curiosity-driven exploration
• Tactile Sensing — Piezoresistive, capacitive, and vision-based tactile sensors for cross-modal fusion
• Digital Human Motion Generation — MDM, ReMoDiffuse, InterGen, T2M-GPT, MotionGPT, MoMask
• Semantic-Guided Enhancement — SAM-/CLIP-/Diffusion-prior guided restoration (DA-CLIP, CoSeR, SeeSR)
• End-to-End Compression — VAE-based image codecs, DVC, DCVC, MobileNVC for on-device deployment

📡 Communication Layer

The neural pathway connecting edge perception to cloud cognition. Covers:

• Semantic Communication — DeepSC (text), DeepSC-S (audio), GAN/NTSCC (image), MU-DeepSC (multimodal)
• Goal-Oriented Communication — GO-COM / GOS-VAE for task-driven transmission
• Edge-Cloud Computing — EECC architecture; "Big Cloud + Small Edge" synergy; federated foundation models

🧠 Cognitive Layer

The decision-making core. Covers:

• Agent-based Retrieval — Four-phase closed loop: Task Analysis → Retrieval Execution → Result Optimization → Strategy Iteration
• World Models — Generative (Genie, WorldDreamer, Drive-WM, Navigation World Models) and Representation-Predictive (V-JEPA 2, DreamerV3, OccWorld)
• Vision-Language-Navigation (VLN) — Environment model-based (DUET, MapGPT, VER, NavMorph) and reasoning-based (NavGPT, NaVid, NaviLLM)
• Vision-Language-Action (VLA) — Action token (RT-1, RT-2, π₀, FAST, OpenVLA); Affordance/trajectory (SayCan, VoxPoser); Code/reasoning (Code as Policies, Inner Monologue)

📏 Evaluation Layer

A multi-dimensional assessment framework. Covers:

• Perception Evaluation — Machine Preference Database (2.25M samples); RA-MIQA for region-aware quality
• Cognitive Evaluation — EQA, ScanQA, SQA3D, PhysBench, EmbodiedEval, IS-Bench, AGENTSAFE
• Action Evaluation — R2R, ALFRED, BEHAVIOR, OVMM, RLBench, ManiSkill, CALVIN, Open X-Embodiment

---

🔬 Key Technologies

Agent-based Retrieval

Moving beyond static cross-modal embedding matching (CCA, VSE, CLIP) toward dynamic retrieval agents with four key phases:

1. 🧩 Task Analysis — query decomposition, sub-query routing, multimodal alignment
2. 🔍 Retrieval Execution — sparse (BM25) + dense (DPR) hybrid; multi-granularity tree retrieval (RAPTOR); ReACT/CoT dynamic triggering
3. ✅ Result Optimization — reranking, FILCO/FiD-Light context filtering, feedback-driven adaptation
4. 🔄 Strategy Iteration — CRAG dynamic switching; continuous alignment between retrieval and generation

World Models

Two major paradigms for equipping agents with the ability to imagine the future:

Paradigm	Representative Works	Key Strength
Generative	Genie, WorldDreamer, Drive-WM, DriveDreamer2, Navigation World Models	High-fidelity video generation, physical law internalization
Representation-Predictive	V-JEPA 2, TD-MPC2, DreamerV3, DayDreamer, OccWorld	Computational efficiency, noise robustness, latent planning

Vision-Language-Action (VLA)

End-to-end mapping from multimodal perception to physical control:

• Action Tokenization — RT-1 → RT-2 → π₀ (Flow Matching) → FAST (DCT tokenization) → OpenVLA-OFT
• Affordance & Trajectory — SayCan, CLIPort, VoxPoser, RT-Trajectory, DriveVLM
• Code & Reasoning — Code as Policies, ProgPrompt, Inner Monologue

---

📐 Evaluation

A New Paradigm: Machine Preference

Traditional metrics (PSNR, SSIM, FID) measure how good data looks to humans. We review a new paradigm measuring how well data serves machines:

• Machine Preference Database (MPD) — 2.25M annotated samples showing the divergence between human and machine quality preferences
• RA-MIQA — Region-Aware Machine Image Quality Assessment; identifies machine sensitivity to specific degraded regions (e.g., background blur)

Action Benchmarks at a Glance

Benchmark	Year	Environment	Task Type
R2R	2018	Matterport3D	Navigation
ALFRED	2020	AI2-THOR	Navigation + Manipulation
RLBench	2020	CoppeliaSim	Manipulation
BEHAVIOR	2022	iGibson	Navigation + Manipulation
CALVIN	2022	PyBullet	Long-horizon Manipulation
OVMM	2023	Habitat	Open-vocab Manipulation
Open X-Embodiment	2024	—	Cross-platform Manipulation

---

🚧 Open Challenges

Research Challenges

Challenge	Description
🎯 Multimodal Spatiotemporal Alignment	Millisecond-level alignment across RGB, LiDAR, tactile, IMU with vastly different sampling rates
🔩 Physical Semantic Gaps	Most datasets lack task-relevant physical attributes: friction, mass, material compliance
📦 Data Scarcity	Interaction-rich datasets with causal signals and long-tail scenarios are prohibitively costly
⚡ On-Device Inference	VLA and world models demand compute that conflicts with robot power/latency budgets
📶 Network Unreliability	Edge-cloud offloading is fragile under bandwidth fluctuations and packet loss
🌉 Sim-to-Real Transfer	Physics simulation fidelity gaps severely constrain physical generalization

Future Directions

• 🔭 Multimodal Active Sensing & Generation — Physics-informed synthetic data with tactile and dynamic properties
• 🪶 Lightweight Embodied Foundation Models — Model compression, parameter-efficient fine-tuning for on-device deployment
• 🔀 Adaptive Edge-Cloud Collaboration — Context-aware routing between lightweight on-device inference and large cloud models
• ⚙️ Physics-Aware World Models — Latent representations capturing geometry, material, and temporal dynamics
• 🧩 Causal Reasoning in VLA — Distinguishing correlation from physical causation for sim-to-real generalization
• 📊 Task- and Safety-Centric Evaluation — Standardized benchmarks explicitly covering interaction safety and hardware transferability

---

📚 Survey Coverage

Domain	Scope	Multimodal Perception	Communication	Cognition & Learning	Physical Action	Unified Architecture
Embodied AI	PCB + LLMs	✅	—	✅	—	—
Embodied AI	Morphology + Action	✅	—	✅	✅	—
Multimedia	CBIR, generative, video	✅	✅	✅	—	—
Ours (Cross-domain)	Full 4-layer pipeline	✅	✅	✅	✅	✅

---

📖 Citation

If you find this survey useful for your research, please consider citing:

@misc{zuo2026embodied,
  author    = {Zuo, Wei and Liu, Yang and Yang, Weniang and Wu, Feng and Zhou, Wei and Liu, Jing and Sun, Peng and Cheng, Jing and Yang, Dingkang and Zeng, Xinhua},
  title     = {Embodied Multimedia: When Multimedia Meets Embodied Intelligence},
  year      = {2026},
  month     = {March},
  note      = {Available at SSRN: \url{https://ssrn.com/abstract=6456578}},
  doi       = {10.2139/ssrn.6456578},
  url       = {https://ssrn.com/abstract=6456578}
}

---

📬 Contact

For questions, discussions, or collaboration inquiries, please reach out to the authors:

• Wei Zuo — wzuo25@m.fudan.edu.cn — Fudan University
• Yang Liu — yangliu25@tongji.edu.cn — Tongji University
• Xinhua Zeng — xhzeng@fudan.edu.cn — Fudan University

Or open a GitHub Issue for technical discussions and suggestions.

---

⭐ Star this repo if you find it helpful — it helps us reach more researchers in the community!