Skip to content

Commit f793bb4

Browse files
MikhailGorobetsMikhailGorobets
committed
SuperResolution: Add documentation
1 parent b42776a commit f793bb4

1 file changed

Lines changed: 282 additions & 0 deletions

File tree

‎Graphics/SuperResolution/readme.md‎

Lines changed: 282 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,2 +1,284 @@
11
# Super Resolution
22

3+
A unified super resolution upscaling library that abstracts multiple hardware-accelerated and software-based
4+
backends behind two interfaces: `ISuperResolutionFactory` (enumeration, configuration, and creation) and
5+
`ISuperResolution` (per-frame execution). The module automatically discovers available upscaler implementations
6+
at factory creation time based on the render device type.
7+
8+
## Supported Backends
9+
10+
| Variant | Type | Graphics API | Description |
11+
|----------------------|----------|---------------------|------------------------------------------------------------------|
12+
| NVIDIA DLSS | Temporal | D3D11, D3D12, Vulkan| Deep-learning based temporal upscaler via NVIDIA NGX SDK |
13+
| Microsoft DirectSR | Temporal | D3D12 | Windows built-in temporal upscaler via DirectSR API |
14+
| AMD FSR | Spatial | All | Shader-based spatial upscaler (Edge Adaptive Upsampling + Contrast Adaptive Sharpening) |
15+
| Apple MetalFX Spatial| Spatial | Metal | Hardware-accelerated spatial upscaler via MetalFX framework |
16+
| Apple MetalFX Temporal| Temporal| Metal | Hardware-accelerated temporal upscaler via MetalFX framework |
17+
18+
## Spatial vs Temporal Upscaling
19+
20+
**Spatial** upscaling operates on a single frame. It requires only the low-resolution color texture as input and
21+
produces an upscaled image using edge-aware filtering and optional sharpening. No motion vectors, depth buffer,
22+
or jitter pattern is needed.
23+
24+
**Temporal** upscaling accumulates information from multiple frames. In addition to the color texture it requires:
25+
26+
- **Depth buffer** — for reprojection and disocclusion detection
27+
- **Motion vectors** — per-pixel 2D motion in pixel space
28+
- **Jitter offset** — sub-pixel offset applied to the projection matrix each frame (typically a Halton sequence)
29+
30+
Optional temporal inputs include:
31+
32+
- **Exposure texture** — a 1x1 texture containing the exposure scale value in the R channel.
33+
Ignored when `SUPER_RESOLUTION_FLAG_AUTO_EXPOSURE` is set.
34+
- **Reactive mask** — per-pixel value in [0, 1] controlling how much the current frame
35+
influences the final result. A value of 0.0 uses normal temporal accumulation; values closer
36+
to 1.0 reduce reliance on history. Useful for alpha-blended objects, particles, or areas
37+
with inaccurate motion vectors. It is recommended to clamp the maximum reactive value to
38+
around 0.9, as values very close to 1.0 rarely produce good results.
39+
- **Ignore history mask** — a binary per-pixel mask where non-zero values
40+
indicate that temporal history should be completely discarded for that pixel (e.g. newly
41+
revealed areas after disocclusion).
42+
43+
## Jitter
44+
45+
Temporal upscalers rely on sub-pixel jitter applied to the projection matrix each frame to reconstruct detail
46+
above the input resolution. The upscaler provides a recommended jitter pattern (Halton 2,3 sequence by default)
47+
via `ISuperResolution::GetJitterOffset()`. The returned values are in **pixel space** (typically in the (-0.5, 0.5) range)
48+
and must be converted to **clip space** before being added to the projection matrix:
49+
50+
```cpp
51+
float JitterX = 0, JitterY = 0;
52+
pSR->GetJitterOffset(FrameIndex, JitterX, JitterY);
53+
54+
// Convert from pixel space to clip space
55+
float JitterClipX = +JitterX / (0.5f * InputWidth);
56+
float JitterClipY = -JitterY / (0.5f * InputHeight);
57+
58+
// Apply to projection matrix (offset the X and Y translation components)
59+
ProjMatrix[2][0] += JitterClipX;
60+
ProjMatrix[2][1] += JitterClipY;
61+
```
62+
63+
The Y component is negated because the pixel-space Y axis points downward while the clip-space Y axis points upward.
64+
The same `JitterX` / `JitterY` values in pixel space must also be passed to `ExecuteSuperResolutionAttribs`
65+
so the upscaler can undo the jitter during reprojection.
66+
67+
For spatial upscaling, `GetJitterOffset()` returns zero for both components and jitter is not needed.
68+
69+
## Texture MIP Bias
70+
71+
When rendering at a lower resolution for upscaling, the GPU selects coarser mipmap levels because screen-space
72+
derivatives are larger relative to the texture coordinate range. To preserve texture detail that the upscaler
73+
will reconstruct, apply a negative MIP LOD bias to texture samplers:
74+
75+
$$
76+
\text{MipBias} = \log_2\!\left(\frac{\text{InputWidth}}{\text{OutputWidth}}\right)
77+
$$
78+
79+
The bias should be applied to all material texture samplers (albedo, normal, roughness, etc.) via
80+
`SamplerDesc::MipLODBias`. This compensates for the lower render resolution and prevents the upscaled
81+
image from looking blurry.
82+
83+
## Motion Vectors
84+
85+
The API expects per-pixel 2D motion vectors in **pixel space** using the `Previous − Current` convention
86+
(pointing from the current position to the previous one).
87+
88+
Use `MotionVectorScaleX` / `MotionVectorScaleY` to convert motion vectors from their native space
89+
and adjust the sign convention at execution time. For example, if the shader computes
90+
`NDC_Current − NDC_Previous`:
91+
92+
```cpp
93+
// Negate direction and convert NDC to pixel space
94+
ExecAttribs.MotionVectorScaleX = -0.5f * static_cast<float>(InputWidth);
95+
ExecAttribs.MotionVectorScaleY = +0.5f * static_cast<float>(InputHeight);
96+
```
97+
98+
X is negative to flip direction; Y is positive because the direction flip and the NDC-to-pixel Y axis flip cancel out.
99+
If motion vectors are already in the `Previous − Current` convention, use `+0.5` for X and `-0.5` for Y.
100+
101+
Motion vectors must use the same resolution as the source color image.
102+
103+
## Optimization Types
104+
105+
`SUPER_RESOLUTION_OPTIMIZATION_TYPE` controls the quality/performance trade-off and determines the recommended
106+
input resolution relative to the output. Default scale factors used when the backend does not provide its own:
107+
108+
| Optimization Type | Scale Factor | Render Resolution (% of output) |
109+
|---------------------------|:------------:|:-------------------------------:|
110+
| `MAX_QUALITY` | 0.75 | 75% |
111+
| `HIGH_QUALITY` | 0.69 | 69% |
112+
| `BALANCED` | 0.56 | 56% |
113+
| `HIGH_PERFORMANCE` | 0.50 | 50% |
114+
| `MAX_PERFORMANCE` | 0.34 | 34% |
115+
116+
Use `ISuperResolutionFactory::GetSourceSettings()` to query the exact optimal input resolution for a given
117+
backend, output resolution, and optimization type.
118+
119+
## Usage
120+
121+
### Creating the Factory
122+
123+
The factory is created per render device. On Windows, the module can be loaded as a shared library:
124+
125+
```cpp
126+
#include "SuperResolutionFactoryLoader.h"
127+
128+
RefCntAutoPtr<ISuperResolutionFactory> pSRFactory;
129+
LoadAndCreateSuperResolutionFactory(pDevice, &pSRFactory);
130+
```
131+
132+
### Enumerating Available Variants
133+
134+
Query the list of upscaler variants supported by the current device:
135+
136+
```cpp
137+
Uint32 NumVariants = 0;
138+
pSRFactory->EnumerateVariants(NumVariants, nullptr);
139+
140+
std::vector<SuperResolutionInfo> Variants(NumVariants);
141+
pSRFactory->EnumerateVariants(NumVariants, Variants.data());
142+
143+
for (const auto& Variant : Variants)
144+
{
145+
// Variant.Name - human-readable name (e.g. "DLSS", "FSR")
146+
// Variant.VariantId - unique identifier for creation
147+
// Variant.Type - SUPER_RESOLUTION_TYPE_SPATIAL or SUPER_RESOLUTION_TYPE_TEMPORAL
148+
}
149+
```
150+
151+
### Querying Optimal Render Resolution
152+
153+
Before creating the upscaler, query the recommended input resolution:
154+
155+
```cpp
156+
SuperResolutionSourceSettingsAttribs SourceAttribs;
157+
SourceAttribs.VariantId = SelectedVariant.VariantId;
158+
SourceAttribs.OutputWidth = SCDesc.Width;
159+
SourceAttribs.OutputHeight = SCDesc.Height;
160+
SourceAttribs.OutputFormat = TEX_FORMAT_R11G11B10_FLOAT
161+
SourceAttribs.Flags = SUPER_RESOLUTION_FLAG_AUTO_EXPOSURE;
162+
SourceAttribs.OptimizationType = SUPER_RESOLUTION_OPTIMIZATION_TYPE_BALANCED;
163+
164+
SuperResolutionSourceSettings SourceSettings;
165+
pSRFactory->GetSourceSettings(SourceAttribs, SourceSettings);
166+
```
167+
168+
### Creating the Upscaler
169+
170+
The upscaler must be recreated when the variant, input resolution, or output resolution changes:
171+
172+
```cpp
173+
SuperResolutionDesc SRDesc;
174+
SRDesc.VariantId = SelectedVariant.VariantId;
175+
SRDesc.InputWidth = SourceSettings.OptimalInputWidth;
176+
SRDesc.InputHeight = SourceSettings.OptimalInputHeight;
177+
SRDesc.OutputWidth = SCDesc.Width;
178+
SRDesc.OutputHeight = SCDesc.Height;
179+
SRDesc.Flags = SUPER_RESOLUTION_FLAG_AUTO_EXPOSURE;
180+
181+
if (SupportsSharpness)
182+
SRDesc.Flags = SRDesc.Flags | SUPER_RESOLUTION_FLAG_ENABLE_SHARPENING;
183+
184+
if (IsTemporalUpscaling)
185+
{
186+
SRDesc.ColorFormat = TEX_FORMAT_R11G11B10_FLOAT;
187+
SRDesc.OutputFormat = TEX_FORMAT_R11G11B10_FLOAT;
188+
SRDesc.DepthFormat = TEX_FORMAT_R32_FLOAT;
189+
SRDesc.MotionFormat = TEX_FORMAT_RG16_FLOAT;
190+
}
191+
else
192+
{
193+
SRDesc.ColorFormat = TEX_FORMAT_RGBA8_UNORM_SRGB;
194+
SRDesc.OutputFormat = TEX_FORMAT_RGBA8_UNORM_SRGB;
195+
}
196+
197+
RefCntAutoPtr<ISuperResolution> pSR;
198+
pSRFactory->CreateSuperResolution(SRDesc, &pSR);
199+
```
200+
201+
### Per-Frame Execution (Temporal)
202+
203+
For temporal upscaling, apply the jitter offset to the projection matrix before rendering the scene
204+
(see [Jitter](#jitter) for details), then execute the upscaler on the **pre-tone-mapped HDR** color buffer:
205+
206+
```cpp
207+
float2 Jitter = {};
208+
pSR->GetJitterOffset(FrameIndex, Jitter.x, Jitter.y);
209+
210+
ExecuteSuperResolutionAttribs ExecAttribs;
211+
ExecAttribs.pContext = pContext;
212+
ExecAttribs.pColorTextureSRV = pRadianceSRV; // HDR pre-tone-mapped color
213+
ExecAttribs.pDepthTextureSRV = pDepthSRV;
214+
ExecAttribs.pMotionVectorsSRV = pMotionVectorsSRV;
215+
ExecAttribs.pOutputTextureView = pOutputUAV;
216+
ExecAttribs.JitterX = Jitter.x;
217+
ExecAttribs.JitterY = Jitter.y;
218+
ExecAttribs.MotionVectorScaleX = -0.5f * static_cast<float>(InputWidth);
219+
ExecAttribs.MotionVectorScaleY = +0.5f * static_cast<float>(InputHeight);
220+
ExecAttribs.CameraNear = ZNear;
221+
ExecAttribs.CameraFar = ZFar;
222+
ExecAttribs.CameraFovAngleVert = YFov;
223+
ExecAttribs.TimeDeltaInSeconds = ElapsedTime;
224+
ExecAttribs.Sharpness = Sharpness;
225+
ExecAttribs.ResetHistory = ResetHistory;
226+
227+
pSR->Execute(ExecAttribs);
228+
```
229+
230+
**`ResetHistory`** should be set to `True` when temporal history is no longer valid:
231+
232+
- First frame after creating or recreating the upscaler
233+
- Camera cut or teleportation (abrupt camera position change)
234+
- Switching between upscaler variants
235+
- Any event that invalidates the correspondence between the current and previous frames
236+
237+
When history is reset, the upscaler discards accumulated temporal data and produces output
238+
based solely on the current frame, which may temporarily reduce quality.
239+
240+
**Depth and camera notes:**
241+
242+
- `DepthFormat` in `SuperResolutionDesc` must be the **SRV-compatible format** (e.g. `TEX_FORMAT_R32_FLOAT`),
243+
not the depth-stencil format (e.g. `TEX_FORMAT_D32_FLOAT`). Use the format of the depth texture's
244+
shader resource view.
245+
- `CameraNear` and `CameraFar` assume depth Z values go from 0 at the near plane to 1 at the far plane.
246+
If using **reverse Z**, swap the two values so that `CameraNear` contains the far plane distance and
247+
`CameraFar` contains the near plane distance.
248+
249+
### Per-Frame Execution (Spatial)
250+
251+
For spatial upscaling, only the color texture and output are required. Execute after tone mapping:
252+
253+
```cpp
254+
ExecuteSuperResolutionAttribs ExecAttribs;
255+
ExecAttribs.pContext = pContext;
256+
ExecAttribs.pColorTextureSRV = pToneMappedSRV; // LDR tone-mapped color
257+
ExecAttribs.pOutputTextureView = pOutputRTV;
258+
ExecAttribs.Sharpness = Sharpness;
259+
260+
pSR->Execute(ExecAttribs);
261+
```
262+
263+
### Render Pipeline Order
264+
265+
The position of the super resolution pass in the rendering pipeline depends on the upscaling type:
266+
267+
**Temporal upscaling** (operates on HDR data, replaces TAA):
268+
269+
```
270+
G-Buffer → Lighting → Super Resolution → Bloom → Tone Mapping → Gamma Correction
271+
```
272+
273+
**Spatial upscaling** (operates on LDR data, after tone mapping):
274+
275+
```
276+
G-Buffer → Lighting → TAA → Bloom → Tone Mapping → Super Resolution → Gamma Correction
277+
```
278+
279+
## References
280+
281+
- [NVIDIA DLSS Programming Guide](https://github.com/NVIDIA/DLSS/blob/main/doc/DLSS_Programming_Guide_Release.pdf)
282+
- [Microsoft DirectSR Specification](https://github.com/microsoft/DirectX-Specs/blob/master/DirectSR/DirectSR.md)
283+
- [AMD FidelityFX Super Resolution](https://github.com/GPUOpen-Effects/FidelityFX-FSR/blob/master/docs/FidelityFX-FSR-Overview-Integration.pdf)
284+
- [Apple MetalFX Documentation](https://developer.apple.com/documentation/metalfx)

0 commit comments

Comments
 (0)