Merge pull request #3379 from pareenaverma/public-contrib/neural-graphics-playbook

Public contrib/neural graphics playbook

Author: pareenaverma

content/learning-paths/mobile-graphics-and-gaming/neural-graphics-playbook-evaluate/01-from-neural-graphics-to-megalights.md

@@ -0,0 +1,58 @@
+---
+title: "From Neural Graphics to MegaLights, and beyond: could it be for me?"
+weight: 2
+
+### FIXED, DO NOT MODIFY
+layout: learningpathall
+---
+
+## The opportunity: why neural graphics, why now
+
+When we first started talking about neural graphics at Arm, the idea was pretty simple: if some parts of rendering can be done more efficiently with machine learning, that frees up budget for everything else. At SIGGRAPH 2025, we announced that [Arm GPUs will have dedicated neural accelerators (NX) in 2026](https://newsroom.arm.com/news/arm-announces-arm-neural-technology). Since then, we’ve worked hard to build the tools that developers will need to seamlessly evaluate and integrate neural graphics use cases.
+
+On mobile, every rendering decision is a tradeoff. Resolution, lighting, effects, frame rate, thermals, battery life all compete for the same limited resources. Ultimately, those technical tradeoffs affect something even more important: the player experience. Visual quality matters, but so does responsiveness, stable performance, and how long a player can stay immersed in the game without their device heating up or throttling.
+
+[Neural Super Sampling (NSS)](https://developer.arm.com/community/arm-community-blogs/b/mobile-graphics-and-gaming-blog/posts/how-to-access-arm-neural-super-sampling) was our first attempt at shifting that balance. Upscaling isn’t new. Shader-based upscalers are already widely used across game engines and mobile games because rendering at lower resolution is one of the most effective ways to save performance. What interested us was whether machine learning could push that idea further.
+
+In practice, NSS allowed us to work with more aggressive upscale ratios, such as rendering at 540p and reconstructing to 1080p, while maintaining image quality that would be difficult to achieve with more traditional approaches. It also opened up better NX utilization in Arm GPUs, while reducing pressure on bandwidth and energy consumption, which are often just as important to mobile developers as raw frame rate.
+
+On mobile, bandwidth is expensive. Moving less data around the system can have a major impact on power efficiency and sustained performance. In many cases, that becomes just as valuable as the rendering speedup itself.
+
+NSS was really a first step in exploring a bigger question: what happens if machine learning becomes a normal part of the rendering pipeline?
+
+## From a single technique to something you can build on
+
+Exploring this question has resulted in the Neural Graphics Development Kit - a set of Vulkan based tools to support early experimentation of different neural graphics use cases that are production ready and efficient for mobile.
+
+| Resource |
+|----------|
+| [Neural Graphics landing page](https://developer.arm.com/mobile-graphics-and-gaming/neural-graphics) |
+| [Enable Neural Super Sampling in Unreal Engine with ML Extensions](/learning-paths/mobile-graphics-and-gaming/nss-unreal/) |
+| [ML SDK for Vulkan](https://github.com/arm/ai-ml-sdk-for-vulkan) |
+| [Neural Super Sampling (NSS) model](https://huggingface.co/Arm/neural-super-sampling) |
+| [ML Emulation Layer for Vulkan](https://github.com/arm/ai-ml-emulation-layer-for-vulkan) |
+| [Neural Graphics Model Gym](https://github.com/arm/neural-graphics-model-gym) |
+
+The development kit, the Unreal plugin, and sample content are provided so you can get started with minimal effort and validate how it behaves with your own content. And that’s really the point. It’s one thing to show a clean before-and-after in a controlled scene, and something else entirely when you’re dealing with dynamic lighting, fast camera movement, complex materials, and all the edge cases that come with real production conditions. That’s where things either hold up or start to break, and where you actually learn if a technique is useful.
+
+Since then, we’ve been expanding neural graphics use cases beyond NSS into applications of frame generation and ray denoising, while also making sure it can be integrated in real projects. This playbook focuses on the techniques we’ve worked on since then, but NSS remains relevant as the most straightforward way to start evaluating neural graphics in a real project.
+
+## Why this matters now
+
+As engines such as Unreal adopt more advanced techniques such as ray tracing, new neural graphics use cases start to make sense in a practical way. Features such as MegaLights make it possible to work with far more dynamic lights than before, without the cost scaling directly with the number of lights. The challenge is that these approaches rely on very low sample counts, which quickly introduce noise, especially on mobile.
+
+Traditional denoisers and shader-based upscalers already help solve part of that problem. But as rendering pushes toward lower resolutions, fewer samples, and more dynamic lighting, reconstruction quality becomes harder to maintain. We’ve found that neural reconstruction allows us to push more aggressive upscale ratios, work with noisier inputs, and still produce stable, high-quality output. It also helps reduce bandwidth and energy pressure, which matters just as much as raw frame rate on mobile devices. More importantly, it starts making some of these newer rendering approaches practical on mobile in the first place.
+
+## Enter Neural Dawn
+
+Neural Dawn is how we validate that the use cases are production ready.
+
+It’s a mobile game created with Unreal Engine using Megalights, combined with the neural graphics work we’ve been developing. The goal was to build a real game and see what actually happens when you try to use this end to end. Where the integration feels natural, where it doesn’t, what kind of gains you get, and what trade-offs show up along the way.
+
+## Why we’re sharing this playbook
+
+This playbook is a reflection of that process. We wanted to share a record of the journey behind Neural Dawn—what worked, what didn’t, what surprised us, and what we’d approach differently next time.
+
+There’s still a gap between seeing a technique and knowing whether it fits your game. That’s the gap we’re trying to close.
+
+Some projects will benefit a lot from this kind of approach, others might not, and that’s fine. It depends on what you’re building, what constraints you’re working under, and artistic direction and features you care about in your game. The goal is to give you enough context and real-world experience to answer a more useful question: is this something worth exploring for your game?

content/learning-paths/mobile-graphics-and-gaming/neural-graphics-playbook-evaluate/02-neural-dawn.md

@@ -0,0 +1,81 @@
+---
+title: About Neural Dawn
+weight: 3
+
+### FIXED, DO NOT MODIFY
+layout: learningpathall
+---
+
+## A game designed around neural graphics
+
+Neural Dawn is a mobile game we built together with Sumo Digital. They focused on the game itself - design, gameplay, art - while we focused on the rendering pipeline and neural graphics systems underneath it.
+
+The goal was to answer a simple question: _What can a game look like if you design it around neural graphics from the start?_
+
+We had three things we wanted to hit at the same time.
+
+First, it had to look good, like _really_ good. The lighting fidelity and scene complexity you would expect from a PC or console game, but on mobile.
+
+Second, it had to run efficiently. We targeted 60 FPS to meet modern expectations for smooth, responsive mobile gameplay.
+
+And third, it had to show what becomes possible when neural graphics and Arm NX are used together with approaches such as MegaLights, on mobile.
+
+Instead of optimizing a traditional graphics pipeline, we built one that depends on neural reconstruction. Neural Dawn is built around MegaLights. We render at low resolution tracing a low amount of rays per pixel, then rely on:
+
+- Neural Super Sampling Denoising (NSSD) to denoise and upscale the image
+- Neural Frame Rate Upscaling (NFRU) to generate intermediate frames
+
+So, the final result looks like a clean, high-resolution, smooth experience, even though we rendered only about one-eighth of the pixels directly.
+
+Ultimately, Neural Dawn shows that neural graphics on mobile are production ready.
+
+TODO: trailer link
+
+## Enabled by neural technology
+
+NFRU and NSSD both help you reach the same end goal. The difference is in what you get out of that.
+
+### NFRU
+
+NFRU gives you extra frames without having to render them. By taking two real frames and generating the one in between, NFRU allows you to go from something like 30 FPS to 60 FPS. You still need to hit your base frame rate target consistently, and there is additional cost associated with NFRU itself, but it can significantly reduce the amount of rendering work needed to reach higher output frame rates.
+
+Some teams will use that added budget to hit a higher frame rate. Others will keep the same target and spend elsewhere: more effects, better lighting, higher quality settings. The important part is that it doesn’t force you to change how your content is built. You’re still rendering the same frames, just fewer of them.
+
+It also stacks well with other techniques. Overall, NFRU is a fairly predictable trade. If you’re looking for something you can try quickly and get value from, this is it.
+
+{{< youtube-nocookie id="YQK1QATQHtI" title="NFRU splitscreen" >}}
+
+### NSSD
+
+NSSD is about making noisy, low-cost rendering usable. If you’ve worked with ray tracing, you’ve seen the issue. At real-time budgets, you can only afford a few rays per pixel. That gives you a very noisy image. Traditional shader- based denoisers help, but can struggle in challenging scenarios (for example, translucent particles such as fire or mist).
+
+Unlike shader-based approaches, which often process diffuse, specular, shadows, and other signals separately, NSSD uses a neural network to reconstruct the final lighting result directly. Doing so allows NSSD to perform denoising and upscaling together in a single pass, reducing the need for multiple specialized denoising stages.
+
+In practical terms, with NSSD, you can achieve high-quality rendering with Raytracing. We validated NSSD with Project Dawn with the following setup:
+
+Rendering at 540p and upscaling to 1080p, tracing 1 ray per pixel with MegaLights.
+
+Normally, lighting cost scales with scene complexity. More lights = more cost. With stochastic techniques, you can keep the cost roughly constant, but the image gets noisy. NSSD cleans up that noise after the fact and makes the content usable.
+
+That’s what enables things such as:
+
+- lots of dynamic lights in a scene
+- soft shadows from all lights
+- more complex lighting setups without baking
+
+NSSD lets you build scenes that you wouldn’t normally even try to render in real-time (on any platform – not just mobile).
+
+{{< youtube-nocookie id="CzBwmQNBGAA" title="NFRU and NSSD splitscreen" >}}
+
+So, what is the bottom line?
+
+NFRU and NSSD don’t compete—they sit at different levels:
+
+- NFRU helps you stretch your existing budget
+- NSSD lets you spend that budget in different ways
+
+You can use NFRU without touching your content.
+
+You can’t really use NSSD that way — it depends on how your game looks and how much you’re willing to invest in getting good results.
+
+That’s why most teams will start with NSS and NFRU, and treat NSSD as something to explore when they’re ready to go deeper. In the next section, you'll learn what is really behind these technologies, and how we enable them through tooling.

content/learning-paths/mobile-graphics-and-gaming/neural-graphics-playbook-evaluate/03-building-blocks.md

@@ -0,0 +1,60 @@
+---
+title: The building blocks
+weight: 4
+
+### FIXED, DO NOT MODIFY
+layout: learningpathall
+---
+
+Up to this point, we’ve focused on the results of what we’ve built. The next step is understanding what’s actually involved in using these techniques. Starting with the core building blocks, the aim with this section is to make the technology easier to reason about in the context of your own game.
+
+Arm provides the foundation: the Unreal Engine plugins, the SDK, prebuilt models, and tooling to run and evaluate them with today’s runtime path centered on Vulkan on Arm platforms. You also get reference implementations and sample content to help you get started quickly, all packaged in the Neural Graphics Development Kit mentioned in the introduction.
+
+What you own is everything that makes this work in your game. That includes integrating the technology into your rendering pipeline, ensuring that required inputs such as motion vectors are correct, and evaluating whether the output meets your quality bar. You're also responsible for making tradeoffs between performance, quality, and latency, and for handling edge cases that arise from your specific content.
+
+In some cases, you might also choose to customize or retrain models. More on that later.
+
+The important takeaway is that neural graphics is not a feature you simply enable. It's a system you integrate, evaluate, and tune. Understanding that upfront will make the rest of the playbook (and your evaluation process) much more effective.
+
+## Technical overview
+
+### NFRU
+
+Neural Frame Rate Upscaling (NFRU) is the part of the system that makes motion feel smooth. The way it works is by looking at two consecutive frames and estimating how every pixel moves between them. This is done using optical flow, which operates in image space and captures not just object motion, but also changes in lighting, shading, and other effects that show up in the final image.
+
+After you have that motion, both frames are warped toward the intermediate timestep. That gives you two different guesses for what the in-between frame should look like — one coming forward from the previous frame, and one coming backward from the next.
+
+![Animation showing two consecutive frames being warped toward an intermediate timestep using optical flow, then combined by the NFRU neural network into a single coherent in-between frame#center](assets/nfru.gif "NFRU warping two frames toward an intermediate timestep")
+
+Those guesses don’t line up perfectly. Some areas are missing, some disagree, and some are just wrong due to motion estimation errors. The neural network’s role is to resolve that. It looks at the warped inputs and the motion field, and learns how to combine them into a single, coherent frame. That includes deciding which input to trust, reconstructing newly visible regions, and smoothing out artifacts so the result is stable over time.
+
+So rather than interpolating, NFRU is really doing a reconstruction of motion. It also deliberately sits at the very end of the pipeline. By that point, the image has already been denoised and upscaled, so the model is working with relatively clean inputs, which makes the problem much easier to solve.
+
+### NSSD
+
+Neural Super Sampling and Denoising (NSSD) is where most of the image quality is actually recovered. It takes a low-resolution, noisy render and turns it into something that looks like a clean, high-resolution frame.
+
+The important thing to understand is that the input here is intentionally very rough. We’re rendering at 540p, and we’re taking only a few lighting samples per pixel. With stochastic rendering techniques such as ReSTIR and MegaLights, that means the image is dominated by noise.
+
+The pipeline is often described as a “compute, neural, compute” sandwich, which is actually a useful way to think about it. It starts with a preprocessing pass that gathers all the information the network needs. This includes not just the noisy radiance buffer, but also depth, normals, motion vectors, and material properties such as albedo and specular.
+
+That context allows the network to understand where surfaces begin and end, how they’re oriented, and how they move over time — which is exactly what you need to separate noise from real signal.
+
+The neural network itself then processes this data. But instead of directly outputting a final image, it produces a set of intermediate signals that guide the rest of the pipeline.
+
+From there, a series of compute passes reconstruct the image. One of the key ideas is the use of a multi-resolution pyramid. The image is represented at several levels of detail, from high-frequency (sharp but noisy) to low-frequency (smooth but blurred). The network predicts how to combine these levels for each pixel.
+
+In practice, that means flat regions can be aggressively smoothed by pulling from lower-resolution representations, while edges and fine details stay sharp by relying more on high-resolution data. Instead of applying a fixed denoising filter, the system is adapting its behavior across the image.
+
+On top of that, there’s a temporal component. Previous frames are reprojected using motion vectors and combined with the current frame, and a feedback tensor is fed back into the network on the next frame. That feedback loop gives the model a kind of memory. It allows the model to learn when to trust historical data and when to rely on the current frame, which is what keeps the result stable instead of flickering.
+
+By the end of the pipeline, you have a frame that is denoised, temporally consistent, and upscaled — for example from 540p to 1080p — even though it was never rendered that way directly.
+
+
+## Putting NFRU and NSSD together
+
+After you put NSSD and NFRU together, the pipeline starts to look very different from a traditional renderer. If you look at the full system, you end up rendering on the order of one eighth of the total pixels across space and time.
+
+That’s really the key takeaway. The renderer is no longer responsible for producing the final image. Its job is to produce just enough signal for the neural system to reconstruct everything else.
+
+![Diagram of the full neural graphics rendering pipeline showing how NSSD denoises and upscales low-resolution frames while NFRU generates intermediate frames, resulting in roughly one eighth of pixels rendered directly#center](assets/pipeline.png "Full neural graphics rendering pipeline combining NSSD and NFRU")