Skip to content

Commit d85662e

Browse files
MikhailGorobetsMikhailGorobets
committed
SuperResolution: Add documentation
1 parent b42776a commit d85662e

File tree

1 file changed

+281
-0
lines changed

1 file changed

+281
-0
lines changed

‎Graphics/SuperResolution/readme.md‎

Lines changed: 281 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,2 +1,283 @@
11
# Super Resolution
22

3+
A unified super resolution upscaling library that abstracts multiple hardware-accelerated and software-based
4+
backends behind two interfaces: `ISuperResolutionFactory` (enumeration, configuration, and creation) and
5+
`ISuperResolution` (per-frame execution). The module automatically discovers available upscaler implementations
6+
at factory creation time based on the render device type.
7+
8+
## Supported Backends
9+
10+
| Variant | Type | Graphics API | Description |
11+
|----------------------|----------|---------------------|------------------------------------------------------------------|
12+
| NVIDIA DLSS | Temporal | D3D11, D3D12, Vulkan| Deep-learning based temporal upscaler via NVIDIA NGX SDK |
13+
| Microsoft DirectSR | Temporal | D3D12 | Windows built-in temporal upscaler via DirectSR API |
14+
| AMD FSR | Spatial | All | Shader-based spatial upscaler (Edge Adaptive Upsampling + Contrast Adaptive Sharpening) |
15+
| Apple MetalFX Spatial| Spatial | Metal | Hardware-accelerated spatial upscaler via MetalFX framework |
16+
| Apple MetalFX Temporal| Temporal| Metal | Hardware-accelerated temporal upscaler via MetalFX framework |
17+
18+
## Spatial vs Temporal Upscaling
19+
20+
**Spatial** upscaling operates on a single frame. It requires only the low-resolution color texture as input and
21+
produces an upscaled image using edge-aware filtering and optional sharpening. No motion vectors, depth buffer,
22+
or jitter pattern is needed.
23+
24+
**Temporal** upscaling accumulates information from multiple frames. In addition to the color texture it requires:
25+
26+
- **Depth buffer** — for reprojection and disocclusion detection
27+
- **Motion vectors** — per-pixel 2D motion in pixel space
28+
- **Jitter offset** — sub-pixel offset applied to the projection matrix each frame (typically a Halton sequence)
29+
30+
Optional temporal inputs include:
31+
32+
- **Exposure texture** — a 1x1 texture containing the exposure scale value in the R channel.
33+
Ignored when `SUPER_RESOLUTION_FLAG_AUTO_EXPOSURE` is set.
34+
- **Reactive mask** — per-pixel value in [0, 1] controlling how much the current frame
35+
influences the final result. A value of 0.0 uses normal temporal accumulation; values closer
36+
to 1.0 reduce reliance on history. Useful for alpha-blended objects, particles, or areas
37+
with inaccurate motion vectors. It is recommended to clamp the maximum reactive value to
38+
around 0.9, as values very close to 1.0 rarely produce good results.
39+
- **Ignore history mask** — a binary per-pixel mask where non-zero values
40+
indicate that temporal history should be completely discarded for that pixel (e.g. newly
41+
revealed areas after disocclusion).
42+
43+
## Jitter
44+
45+
Temporal upscalers rely on sub-pixel jitter applied to the projection matrix each frame to reconstruct detail
46+
above the input resolution. The upscaler provides a recommended jitter pattern (Halton 2,3 sequence by default)
47+
via `ISuperResolution::GetJitterOffset()`. The returned values are in **pixel space** (typically in the (-0.5, 0.5) range)
48+
and must be converted to **clip space** before being added to the projection matrix:
49+
50+
```cpp
51+
float JitterX = 0, JitterY = 0;
52+
pSR->GetJitterOffset(FrameIndex, JitterX, JitterY);
53+
54+
// Convert from pixel space to clip space
55+
float JitterClipX = +JitterX / (0.5f * InputWidth);
56+
float JitterClipY = -JitterY / (0.5f * InputHeight);
57+
58+
// Apply to projection matrix (offset the X and Y translation components)
59+
ProjMatrix[2][0] += JitterClipX;
60+
ProjMatrix[2][1] += JitterClipY;
61+
```
62+
63+
The Y component is negated because the pixel-space Y axis points downward while the clip-space Y axis points upward.
64+
The same `JitterX` / `JitterY` values in pixel space must also be passed to `ExecuteSuperResolutionAttribs`
65+
so the upscaler can undo the jitter during reprojection.
66+
67+
For spatial upscaling, `GetJitterOffset()` returns zero for both components and jitter is not needed.
68+
69+
## Texture MIP Bias
70+
71+
When rendering at a lower resolution for upscaling, the GPU selects coarser mipmap levels because screen-space
72+
derivatives are larger relative to the texture coordinate range. To preserve texture detail that the upscaler
73+
will reconstruct, apply a negative MIP LOD bias to texture samplers:
74+
75+
$$
76+
\text{MipBias} = \log_2\left(\frac{\text{InputWidth}}{\text{OutputWidth}}\right)
77+
$$
78+
79+
The bias should be applied to all material texture samplers (albedo, normal, roughness, etc.) via
80+
`SamplerDesc::MipLODBias`. This compensates for the lower render resolution and prevents the upscaled
81+
image from looking blurry.
82+
83+
## Motion Vectors
84+
85+
The API expects per-pixel 2D motion vectors in **pixel space** using the `Previous − Current` convention.
86+
87+
Use `MotionVectorScaleX` / `MotionVectorScaleY` to convert motion vectors from their native space
88+
and adjust the sign convention at execution time. For example, if the shader computes
89+
`NDC_Current − NDC_Previous`:
90+
91+
```cpp
92+
// Negate direction and convert NDC to pixel space
93+
ExecAttribs.MotionVectorScaleX = -0.5f * static_cast<float>(InputWidth);
94+
ExecAttribs.MotionVectorScaleY = +0.5f * static_cast<float>(InputHeight);
95+
```
96+
97+
X is negative to flip direction; Y is positive because the direction flip and the NDC-to-pixel Y axis flip cancel out.
98+
If motion vectors are already in the `Previous − Current` convention, use `+0.5` for X and `-0.5` for Y.
99+
100+
Motion vectors must use the same resolution as the source color image.
101+
102+
## Optimization Types
103+
104+
`SUPER_RESOLUTION_OPTIMIZATION_TYPE` controls the quality/performance trade-off and determines the recommended
105+
input resolution relative to the output. Default scale factors used when the backend does not provide its own:
106+
107+
| Optimization Type | Scale Factor | Render Resolution (% of output) |
108+
|---------------------------|:------------:|:-------------------------------:|
109+
| `MAX_QUALITY` | 0.75 | 75% |
110+
| `HIGH_QUALITY` | 0.69 | 69% |
111+
| `BALANCED` | 0.56 | 56% |
112+
| `HIGH_PERFORMANCE` | 0.50 | 50% |
113+
| `MAX_PERFORMANCE` | 0.34 | 34% |
114+
115+
Use `ISuperResolutionFactory::GetSourceSettings()` to query the exact optimal input resolution for a given
116+
backend, output resolution, and optimization type.
117+
118+
## Usage
119+
120+
### Creating the Factory
121+
122+
The factory is created per render device. On Windows, the module can be loaded as a shared library:
123+
124+
```cpp
125+
#include "SuperResolutionFactoryLoader.h"
126+
127+
RefCntAutoPtr<ISuperResolutionFactory> pSRFactory;
128+
LoadAndCreateSuperResolutionFactory(pDevice, &pSRFactory);
129+
```
130+
131+
### Enumerating Available Variants
132+
133+
Query the list of upscaler variants supported by the current device:
134+
135+
```cpp
136+
Uint32 NumVariants = 0;
137+
pSRFactory->EnumerateVariants(NumVariants, nullptr);
138+
139+
std::vector<SuperResolutionInfo> Variants(NumVariants);
140+
pSRFactory->EnumerateVariants(NumVariants, Variants.data());
141+
142+
for (const auto& Variant : Variants)
143+
{
144+
// Variant.Name - human-readable name (e.g. "DLSS", "FSR")
145+
// Variant.VariantId - unique identifier for creation
146+
// Variant.Type - SUPER_RESOLUTION_TYPE_SPATIAL or SUPER_RESOLUTION_TYPE_TEMPORAL
147+
}
148+
```
149+
150+
### Querying Optimal Render Resolution
151+
152+
Before creating the upscaler, query the recommended input resolution:
153+
154+
```cpp
155+
SuperResolutionSourceSettingsAttribs SourceAttribs;
156+
SourceAttribs.VariantId = SelectedVariant.VariantId;
157+
SourceAttribs.OutputWidth = SCDesc.Width;
158+
SourceAttribs.OutputHeight = SCDesc.Height;
159+
SourceAttribs.OutputFormat = TEX_FORMAT_R11G11B10_FLOAT
160+
SourceAttribs.Flags = SUPER_RESOLUTION_FLAG_AUTO_EXPOSURE;
161+
SourceAttribs.OptimizationType = SUPER_RESOLUTION_OPTIMIZATION_TYPE_BALANCED;
162+
163+
SuperResolutionSourceSettings SourceSettings;
164+
pSRFactory->GetSourceSettings(SourceAttribs, SourceSettings);
165+
```
166+
167+
### Creating the Upscaler
168+
169+
The upscaler must be recreated when the variant, input resolution, or output resolution changes:
170+
171+
```cpp
172+
SuperResolutionDesc SRDesc;
173+
SRDesc.VariantId = SelectedVariant.VariantId;
174+
SRDesc.InputWidth = SourceSettings.OptimalInputWidth;
175+
SRDesc.InputHeight = SourceSettings.OptimalInputHeight;
176+
SRDesc.OutputWidth = SCDesc.Width;
177+
SRDesc.OutputHeight = SCDesc.Height;
178+
SRDesc.Flags = SUPER_RESOLUTION_FLAG_AUTO_EXPOSURE;
179+
180+
if (SupportsSharpness)
181+
SRDesc.Flags = SRDesc.Flags | SUPER_RESOLUTION_FLAG_ENABLE_SHARPENING;
182+
183+
if (IsTemporalUpscaling)
184+
{
185+
SRDesc.ColorFormat = TEX_FORMAT_R11G11B10_FLOAT;
186+
SRDesc.OutputFormat = TEX_FORMAT_R11G11B10_FLOAT;
187+
SRDesc.DepthFormat = TEX_FORMAT_R32_FLOAT;
188+
SRDesc.MotionFormat = TEX_FORMAT_RG16_FLOAT;
189+
}
190+
else
191+
{
192+
SRDesc.ColorFormat = TEX_FORMAT_RGBA8_UNORM_SRGB;
193+
SRDesc.OutputFormat = TEX_FORMAT_RGBA8_UNORM_SRGB;
194+
}
195+
196+
RefCntAutoPtr<ISuperResolution> pSR;
197+
pSRFactory->CreateSuperResolution(SRDesc, &pSR);
198+
```
199+
200+
### Per-Frame Execution (Temporal)
201+
202+
For temporal upscaling, apply the jitter offset to the projection matrix before rendering the scene
203+
(see [Jitter](#jitter) for details), then execute the upscaler on the **pre-tone-mapped HDR** color buffer:
204+
205+
```cpp
206+
float2 Jitter = {};
207+
pSR->GetJitterOffset(FrameIndex, Jitter.x, Jitter.y);
208+
209+
ExecuteSuperResolutionAttribs ExecAttribs;
210+
ExecAttribs.pContext = pContext;
211+
ExecAttribs.pColorTextureSRV = pRadianceSRV; // HDR pre-tone-mapped color
212+
ExecAttribs.pDepthTextureSRV = pDepthSRV;
213+
ExecAttribs.pMotionVectorsSRV = pMotionVectorsSRV;
214+
ExecAttribs.pOutputTextureView = pOutputUAV;
215+
ExecAttribs.JitterX = Jitter.x;
216+
ExecAttribs.JitterY = Jitter.y;
217+
ExecAttribs.MotionVectorScaleX = -0.5f * static_cast<float>(InputWidth);
218+
ExecAttribs.MotionVectorScaleY = +0.5f * static_cast<float>(InputHeight);
219+
ExecAttribs.CameraNear = ZNear;
220+
ExecAttribs.CameraFar = ZFar;
221+
ExecAttribs.CameraFovAngleVert = YFov;
222+
ExecAttribs.TimeDeltaInSeconds = ElapsedTime;
223+
ExecAttribs.Sharpness = Sharpness;
224+
ExecAttribs.ResetHistory = ResetHistory;
225+
226+
pSR->Execute(ExecAttribs);
227+
```
228+
229+
**`ResetHistory`** should be set to `True` when temporal history is no longer valid:
230+
231+
- First frame after creating or recreating the upscaler
232+
- Camera cut or teleportation (abrupt camera position change)
233+
- Switching between upscaler variants
234+
- Any event that invalidates the correspondence between the current and previous frames
235+
236+
When history is reset, the upscaler discards accumulated temporal data and produces output
237+
based solely on the current frame, which may temporarily reduce quality.
238+
239+
**Depth and camera notes:**
240+
241+
- `DepthFormat` in `SuperResolutionDesc` must be the **SRV-compatible format** (e.g. `TEX_FORMAT_R32_FLOAT`),
242+
not the depth-stencil format (e.g. `TEX_FORMAT_D32_FLOAT`). Use the format of the depth texture's
243+
shader resource view.
244+
- `CameraNear` and `CameraFar` assume depth Z values go from 0 at the near plane to 1 at the far plane.
245+
If using **reverse Z**, swap the two values so that `CameraNear` contains the far plane distance and
246+
`CameraFar` contains the near plane distance.
247+
248+
### Per-Frame Execution (Spatial)
249+
250+
For spatial upscaling, only the color texture and output are required. Execute after tone mapping:
251+
252+
```cpp
253+
ExecuteSuperResolutionAttribs ExecAttribs;
254+
ExecAttribs.pContext = pContext;
255+
ExecAttribs.pColorTextureSRV = pToneMappedSRV; // LDR tone-mapped color
256+
ExecAttribs.pOutputTextureView = pOutputRTV;
257+
ExecAttribs.Sharpness = Sharpness;
258+
259+
pSR->Execute(ExecAttribs);
260+
```
261+
262+
### Render Pipeline Order
263+
264+
The position of the super resolution pass in the rendering pipeline depends on the upscaling type:
265+
266+
**Temporal upscaling** (operates on HDR data, replaces TAA):
267+
268+
```
269+
G-Buffer → Lighting → Super Resolution → Bloom → Tone Mapping → Gamma Correction
270+
```
271+
272+
**Spatial upscaling** (operates on LDR data, after tone mapping):
273+
274+
```
275+
G-Buffer → Lighting → TAA → Bloom → Tone Mapping → Super Resolution → Gamma Correction
276+
```
277+
278+
## References
279+
280+
- [NVIDIA DLSS Programming Guide](https://github.com/NVIDIA/DLSS/blob/main/doc/DLSS_Programming_Guide_Release.pdf)
281+
- [Microsoft DirectSR Specification](https://github.com/microsoft/DirectX-Specs/blob/master/DirectSR/DirectSR.md)
282+
- [AMD FidelityFX Super Resolution](https://github.com/GPUOpen-Effects/FidelityFX-FSR/blob/master/docs/FidelityFX-FSR-Overview-Integration.pdf)
283+
- [Apple MetalFX Documentation](https://developer.apple.com/documentation/metalfx)

0 commit comments

Comments
 (0)