Merge pull request #3208 from annietllnd/data-capture-ue

New LP on Data capture with Unreal Engine

Author: pareenaverma

content/learning-paths/mobile-graphics-and-gaming/neural-graphics-data-capture-unreal/01-why-ngdc.md

@@ -0,0 +1,39 @@
+---
+title: Benefits of Neural Graphics Data Capture for game developers
+weight: 2
+
+### FIXED, DO NOT MODIFY
+layout: learningpathall
+---
+
+## Overview
+
+The Neural Graphics Data Capture plugin helps you create structured datasets directly from Unreal Engine gameplay. This is useful when you want to train or evaluate neural graphics workloads, such as [Neural Super Sampling (NSS)](/learning-paths/mobile-graphics-and-gaming/nss-unreal/) and other temporal upscalers, with real scene motion, geometry, and camera behavior from your own content.
+
+## Benefits of capturing data from your game content
+
+Capturing from your own gameplay is useful when you want to:
+
+- Fine-tune NSS on your specific game content to improve image quality.
+- Evaluate NSS output quality on your own scenes, motion, materials, and camera behavior.
+- Train NSS variants, or your own model architecture from scratch, using representative project data.
+
+With this setup, you can create capture data inside Unreal Engine 5.5 without extra graphics API environment configuration. For teams iterating on model experiments, this lowers setup friction and shortens time-to-first-dataset.
+
+## Common workflow for neural graphics data capture
+
+A common workflow is:
+
+- Capture representative gameplay clips from your project.
+- Export frames and metadata to a dataset folder.
+- Use those datasets as input to Model Gym's [NSS data generation step](https://github.com/arm/neural-graphics-model-gym/blob/main/docs/nss/nss_data_generation.md), which converts captured data into `safetensors` for training.
+- Tune rendering/capture settings (for example supersampling, upscaling ratio, fixed frame rate) and capture again.
+
+## Learning Path objectives
+
+You will:
+
+- Install the plugin into a UE 5.5 C++ project.
+- Enable and compile the module in Visual Studio.
+- Add Level Blueprint nodes to start capture (`C`) and stop capture (`V`).
+- Run in Standalone Game mode and verify output in `Saved/NeuralGraphicsDataset`.

content/learning-paths/mobile-graphics-and-gaming/neural-graphics-data-capture-unreal/02-install-and-enable-plugin.md

@@ -0,0 +1,52 @@
+---
+title: Install and enable the plugin
+weight: 3
+
+### FIXED, DO NOT MODIFY
+layout: learningpathall
+---
+
+## Before you begin
+
+This plugin currently supports **Unreal Engine 5.5**.
+
+Use a **C++ Unreal project** so the plugin module can compile through Visual Studio.
+
+## Clone the repository
+
+The plugin lives in a GitHub repository. Start by cloning it:
+
+```
+git clone https://github.com/arm/neural-graphics-data-capture-for-unreal
+```
+
+## Add plugin files to your project
+
+- Open your Unreal project folder in File Explorer.
+- Create a `Plugins` folder at the root of the project if it doesn't already exist.
+- Copy the `NeuralGraphicsDataCapture` plugin folder from the `Source` directory into your new `Plugins/` directory. 
+
+![Screenshot of Windows File Explorer showing the project root directory with a Plugins folder containing the NeuralGraphicsDataCapture folder. This confirms the plugin is in the correct location.#center](./images/project-plugins-folder.png "Project root with Plugins directory")
+
+## Reopen project and build module
+
+- Reopen the `.uproject` file.
+- If prompted about missing modules, select **Yes** to rebuild.
+
+![Dialog box showing that the NeuralGraphicsDataCapture module is missing and asking whether to rebuild it. Select Yes to continue.#center](./images/missing-modules-rebuild-prompt.png "Missing module rebuild prompt")
+
+- Build the project in Visual Studio.
+
+{{% notice %}}If Unreal doesn't detect the plugin after copying files, regenerate project files and rebuild from Visual Studio.{{% /notice %}}
+
+## Verify plugin is enabled
+
+In Unreal Editor:
+
+- Go to **Edit > Plugins**.
+- Search for `data cap`.
+- Confirm **Neural Graphics Data Capture Plugin for Unreal Engine** is enabled.
+
+![Screenshot of the Unreal Engine Plugins window with the search results showing Neural Graphics Data Capture Plugin for Unreal Engine with the enabled checkbox selected. This confirms the plugin is active.#center](./images/plugin-enabled-in-editor.png "Plugin enabled in Unreal Editor")
+
+Next, add the capture graph to your Level Blueprint.

content/learning-paths/mobile-graphics-and-gaming/neural-graphics-data-capture-unreal/03-configure-level-blueprint.md

@@ -0,0 +1,59 @@
+---
+title: Configure Level Blueprint capture controls
+weight: 4
+
+### FIXED, DO NOT MODIFY
+layout: learningpathall
+---
+
+## Open the Level Blueprint
+
+In your level:
+
+- Open the **Blueprints** drop-down.
+- Select **Open Level Blueprint**.
+
+![Screenshot of the Unreal Engine toolbar showing the Blueprints menu expanded with Open Level Blueprint option highlighted. Select this option to open the blueprint editor.#center](./images/open-level-blueprint-menu.png "Open Level Blueprint menu")
+
+## Fast path: download and paste the ready-made blueprint snippet
+
+On Windows, use PowerShell to download the prepared Level Blueprint snippet from this Learning Path repository:
+
+```powershell
+wget "https://raw.githubusercontent.com/ArmDeveloperEcosystem/arm-learning-paths/main/content/learning-paths/mobile-graphics-and-gaming/neural-graphics-data-capture-unreal/assets/level_blueprint.txt" -OutFile ".\level_blueprint.txt"
+```
+
+Then in Unreal Editor:
+
+- Open `level_blueprint.txt` in a text editor and copy all text.
+- Select empty space in the Level Blueprint Event Graph.
+- Paste to create the full node graph.
+- Select **Compile** and **Save**.
+
+{{% notice %}}
+If you use Git Bash with GNU `wget`, use `wget -O level_blueprint.txt "https://raw.githubusercontent.com/ArmDeveloperEcosystem/arm-learning-paths/main/content/learning-paths/mobile-graphics-and-gaming/neural-graphics-data-capture-unreal/assets/level_blueprint.txt"` instead.
+{{% /notice %}}
+
+## Blueprint capture flow configuration
+
+The snippet wires the following capture flow:
+
+- Gets the **Neural Graphics Data Capture Subsystem**.
+- Builds `NGDCCaptureSettings` from:
+  - `NGDCRenderingSettings`
+  - `NGDCExportSettings`
+- Calls **Begin Capture** on key `C`.
+- Calls **End Capture** on key `V`.
+
+It also sets these export defaults to match this tutorial:
+
+- `Dataset Dir`: `NeuralGraphicsDataset`
+- `Capture Name`: `0000`
+
+![Close-up screenshot of the Level Blueprint showing the capture settings nodes and the Neural Graphics Data Capture Subsystem being wired to Begin Capture and End Capture nodes. This shows how the settings are configured.#center](./images/blueprint-capture-subsystem-closeup.png "Capture settings and subsystem wiring")
+
+The full event graph should look similar to this:
+
+![Screenshot of the complete Level Blueprint event graph showing keyboard input C connected to Begin Capture and keyboard input V connected to End Capture, with all necessary settings nodes connected. This is the expected final blueprint configuration.#center](./images/level-blueprint-full-graph.png "Complete blueprint graph")
+
+Continue to run the level in Standalone mode and test capture.

content/learning-paths/mobile-graphics-and-gaming/neural-graphics-data-capture-unreal/04-run-capture-and-verify-output.md

@@ -0,0 +1,47 @@
+---
+title: Run capture and verify outputs
+weight: 5
+
+### FIXED, DO NOT MODIFY
+layout: learningpathall
+---
+
+## Run in Standalone Game mode
+
+Use **Standalone Game** for capture:
+
+- Open the play mode menu next to the **Play** button.
+- Select **Standalone Game**.
+
+![Screenshot of the Unreal Engine Play mode dropdown menu with Standalone Game option highlighted. Select this option instead of Play in Editor to ensure correct frame dimensions.#center](./images/play-standalone-mode.png "Standalone Game mode selection")
+
+{{% notice %}}
+If you use **New Editor Window (PIE)**, captured frame dimensions can differ from expected output sizes.
+{{% /notice %}}
+
+## Start and stop capture
+
+- Press **Play**.
+- Press `C` to begin capture.
+- Move through the level to capture frames.
+- Press `V` to stop capture.
+
+![Screenshot of the Standalone Game window during active capture showing the game viewport with the character visible. This shows what you'll see while frames are being captured.#center](./images/standalone-capture-running.png "Capture running in Standalone mode")
+
+After stopping, you should see a completion notification in the bottom right corner. 
+
+## Locate the dataset
+
+Captured output is written under:
+
+```
+<YourProject>/Saved/NeuralGraphicsDataset/
+```
+
+With the defaults from this tutorial, see:
+
+```
+<YourProject>/Saved/NeuralGraphicsDataset/0000
+```
+
+Proceed to the next section to tune settings and troubleshoot common issues.

content/learning-paths/mobile-graphics-and-gaming/neural-graphics-data-capture-unreal/05-settings-and-troubleshooting.md

@@ -0,0 +1,54 @@
+---
+title: Capture settings and troubleshooting
+weight: 6
+
+### FIXED, DO NOT MODIFY
+layout: learningpathall
+---
+
+## Available capture settings
+
+You can tune these in `NGDCRenderingSettings` and `NGDCExportSettings`:
+
+- `UpscalingRatio`: Sets the ratio between jittered input and ground-truth output. Fractional values are supported.
+- `SupersamplingRatio`: Controls quality/performance tradeoff while capturing.
+- `FixedFrameRate`: Locks capture frame rate when greater than `0`.
+- `CameraCutTranslationThreshold` and `CameraCutRotationThresholdDegrees`: Heuristics for marking camera cuts when needed.
+- `DatasetDir` and `CaptureName`: Control output folder location and capture name.
+
+## Recommended system specification
+
+For smooth out-of-box capture, a representative spec is:
+
+- RAM: 64 GB
+- GPU: NVIDIA GeForce RTX 4080 (driver 572.96 or newer)
+- OS: 64-bit Windows or Linux
+
+## Common issues and solutions
+
+**Plugin not detected:**
+- Confirm plugin is under your project `Plugins/` directory.
+- Regenerate project files and rebuild.
+
+**Hotkeys don't work:**
+- Verify Level Blueprint has `C` wired to **Begin Capture** and `V` wired to **End Capture**.
+
+**Dataset folder is empty:**
+- Confirm capture started and that export path is writable.
+
+**Stutter or instability:**
+- Lower `SupersamplingRatio` and test again.
+
+**Unexpected output size:**
+- Use **Standalone Game** mode instead of **New Editor Window (PIE)**.
+
+## What you've learned
+
+You've successfully set up a workflow to capture neural graphics datasets directly from Unreal Engine 5.5 gameplay. You can now:
+
+- Generate training data from your own game content without graphics API configuration
+- Capture representative gameplay sequences with real motion and camera behavior
+- Export frame datasets ready for use with Model Gym's NSS data generation pipeline
+- Tune capture settings to match your specific quality and performance requirements
+
+Use this repeatable workflow for model experimentation and evaluation in your neural graphics pipeline.

content/learning-paths/mobile-graphics-and-gaming/neural-graphics-data-capture-unreal/_index.md

@@ -0,0 +1,68 @@
+---
+title: Generate neural graphics datasets with Neural Graphics Data Capture in Unreal Engine
+description: Learn how to capture high-quality frame datasets from Unreal Engine 5.5 gameplay for training and evaluating neural graphics models like Neural Super Sampling.
+
+minutes_to_complete: 30
+
+who_is_this_for: This Learning Path is for Unreal Engine developers who want to generate high-quality frame datasets for training and evaluating neural graphics models.
+
+learning_objectives:
+    - Understand why Neural Graphics Data Capture is useful in a neural graphics workflow.
+    - Install and enable the Neural Graphics Data Capture plugin in Unreal Engine 5.5.
+    - Configure a Level Blueprint to start and stop capture with hotkeys.
+    - Run Standalone capture and verify exported dataset outputs.
+
+prerequisites:
+    - Windows 11
+    - Unreal Engine 5.5 installed
+    - Visual Studio with C++ game development tools
+    - A C++ Unreal project (such as the Third Person template)
+
+author: 
+- Annie Tallund
+- Richard Burton
+
+### Tags
+skilllevels: Introductory
+subjects: Graphics
+armips:
+    - Mali
+    - Immortalis
+tools_software_languages:
+    - Unreal Engine
+    - Visual Studio
+operatingsystems:
+    - Windows
+
+
+
+further_reading:
+    - resource:
+        title: Neural Graphics Data Capture Plugin for Unreal Engine
+        link: https://github.com/arm/neural-graphics-data-capture-for-unreal
+        type: website
+    - resource:
+        title: Neural Graphics Development Kit
+        link: https://developer.arm.com/mobile-graphics-and-gaming/neural-graphics
+        type: website
+    - resource:
+        title: Neural Super Sampling in Unreal Engine
+        link: /learning-paths/mobile-graphics-and-gaming/nss-unreal/
+        type: documentation
+    - resource:
+        title: Generate Unreal Engine project files for your IDE
+        link: https://dev.epicgames.com/documentation/en-us/unreal-engine/how-to-generate-unreal-engine-project-files-for-your-ide
+        type: documentation
+    - resource:
+        title: Neural Graphics Model Gym - NSS Data Generation
+        link: https://github.com/arm/neural-graphics-model-gym/blob/main/docs/nss/nss_data_generation.md
+        type: website
+
+
+
+### FIXED, DO NOT MODIFY
+# ================================================================================
+weight: 1                       # _index.md always has weight of 1 to order correctly
+layout: "learningpathall"       # All files under learning paths have this same wrapper
+learning_path_main_page: "yes"  # This should be surfaced when looking for related content. Only set for _index.md of learning path content.
+---

content/learning-paths/mobile-graphics-and-gaming/neural-graphics-data-capture-unreal/_next-steps.md

@@ -0,0 +1,8 @@
+---
+# ================================================================================
+#       FIXED, DO NOT MODIFY THIS FILE
+# ================================================================================
+weight: 21                  # Set to always be larger than the content in this path to be at the end of the navigation.
+title: "Next Steps"         # Always the same, html page title.
+layout: "learningpathall"   # All files under learning paths have this same wrapper for Hugo processing.
+---