You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: Documentation/articles/FrameRenderTimings.md
+96-36Lines changed: 96 additions & 36 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -17,9 +17,9 @@ Well then, here is a quick introduction to the pain that is figuring out that in
17
17
## But isn't it super simple?
18
18
19
19
- Can't I just log when I change the GameObject in Unity?
20
-
- The short answer is: no.
21
-
- The long answer is: Why the hell is this so complicated!!!
22
-
- More seriously, unless you use GSync or FreeSync (which we will talk about later) then the image on screen only changes X times a second. The 'X; is your monitor refresh rate (like a 60Hz, 120Hz, or 144Hz monitor) and it determines how many times a second your screen changes what it is showing. The time that you change the item in unity is NOT the time that the item actually changes on the screen. VSync helps to solve this issue.
20
+
- The short answer is: no.
21
+
- The long answer is: Why the hell is this so complicated!!!
22
+
- More seriously, unless you use GSync or FreeSync (which we will talk about much later) then the image on screen only changes X times a second. The 'X; is your monitor refresh rate (like a 60Hz, 120Hz, or 144Hz monitor) and it determines how many times a second your screen changes what it is showing. The time that you change the item in unity is NOT the time that the item actually changes on the screen. VSync helps to solve this issue.
23
23
- What is that VSync thing that you mentioned?
24
24
- VSync is a tool that syncs your game with the monitor refresh rate. In other words, your game will only tell the screen to show new things at the same rate the monitor shows new things. This actually sets how often the Update() function is called in Unity (assuming your game isn't lagging).
25
25
- If you have VSync off, you can actually show parts of multiple frames at one time. For example, the top of your screen may show one frame and the bottom of the screen shows the next frame. These lines between different frames being shown at the same time are called "tear lines" (tear as in ripping, not like crying). The general term is "screen tearing".
@@ -34,22 +34,21 @@ Well then, here is a quick introduction to the pain that is figuring out that in
34
34
- If you have a 60Hz monitor and a 16.67ms accuracy is good enough for you, then you're all set!
35
35
- If you have a 120Hz monitor and an 8.33ms accuracy is good enough for you, then you're all set!
36
36
- The pattern is to caculate: $1 / refresh\_rate$
37
-
- Well that's not good enough for you...
37
+
- Is it really that simple?
38
+
- Heck no!
39
+
- There are plenty more things that throw those theoretical numbers off:
40
+
1. Lag while playing the game
41
+
1. OS or graphics card level double/tripple buffering
42
+
1. Delay between the graphics card vsync pulse and writing the data to the monitor
43
+
1. Monitor pixel change times
44
+
- So you decided that you want more accurate timing...
38
45
- Then onto the next section you go.
39
46
40
-
## Can I just use a photodiode sync pulse?
41
-
42
-
- Absolutely! And it is better than the above! But it only get's you halfway there...
43
-
- As a matter of fact, it only gets you as far as the last section ended, just more accurately.
44
-
- Technically it gets you a TOUCH more accurate timings because there is a difference when the frame ends (what PsyForge is logging) and when the graphics card actually gives the sync signal for the next frame data.
45
-
- Don't forget when using a photodiode though to take into account how long it takes to actually SEND the data from the photodiode controller to your app and timestamp it. That delay there can make a difference (usually a few milliseconds).
46
-
- Another downside of a photodiode: you won't be informed when the screen does NOT change (and you can't actually change it every frame). So this will only work for specific special events that matter the most, not for every frame that is logged like the "frameDisplayed" event. On example where this could be problematic is trying to align a constant stream of eyetracking data in a 3D task.
47
-
48
-
## So your "frameDisplayed" event isn't even exact?!? (Part 1)
47
+
## So your "frameDisplayed" event isn't even exact?!?
49
48
50
49
- You are right. Time to learn about monitors.
51
50
52
-
## An Intro to Monitors (Part 1)
51
+
## An Intro to Monitors (Part 1): The Basics
53
52
54
53
- There are a 2 main steps in a monitor's output.
55
54
1. Active Display - when the screen is actually changing
@@ -61,14 +60,14 @@ Well then, here is a quick introduction to the pain that is figuring out that in
61
60
## So why don't we just log when the Active Display starts or VBlank ends?
62
61
63
62
- The simple answer, because I haven't implemented it yet. I can't wait until I can delete this section.
64
-
- This longer answer is that it is way more difficult than it seems because you can't do it in Unity. Not only is it different for every Operating System (Linux, Windows, MacOS), but it differs with what graphics card you are using (Nvidia, AMD). It even changes based on which display server protocol you are using in Linux (X11, Wayland). And to top it all off, sometimes it even works differently between monitors for specific operating systems (looking at you MacOS).
63
+
- Temporary note: This is literally in progress right now.
64
+
- This longer answer is that it is way more difficult than it seems because you can't do it in Unity. Not only is it different for every Operating System (Linux, Windows, MacOS, Android, iOS, VR, etc.), but it differs with what graphics card you are using (Nvidia, AMD, integrated graphics). It even changes based on which display server protocol you are using in Linux (X11, Wayland). And to top it all off, sometimes it even works differently between monitors for specific operating systems (looking at you MacOS).
65
+
- But, you guessed it, that's STILL not enough!
65
66
66
-
## An Intro to Monitors (Part 2)
67
-
68
-
### Pixels don't change right away?!?
67
+
## An Intro to Monitors (Part 2): Pixels don't change right away?!?
69
68
70
69
- That is correct, the real world is messy and pixels do not change color right away.
71
-
- There is still a delay that occurs as the pixels try to change colors. This can either be measured in black-to-white (BTW) or grey-to-grey (GTG) response times.
70
+
- There is still a slight delay that occurs as the pixels try to change colors. This can either be measured in black-to-white (BTW) or grey-to-grey (GTG) response times.
72
71
- Different types of monitors have different black-to-white response times.
73
72
- LCD is about 10ms BTW and 1-10ms GTG
74
73
- LED is about 10ms BTW and 1-10ms GTG
@@ -80,11 +79,21 @@ Well then, here is a quick introduction to the pain that is figuring out that in
80
79
- I recommend you look these values up before buying a monitor. The two best websites are [RTINGS](rtings.com) and [TFT](tftcentral.co.uk)
81
80
1. Check that the Response Time graphs show low values (especially in the top row and the left column)
82
81
1. Input Lag section shows a small value
83
-
- These are both relatively constant for each monitor, so it is easy enough to account for.
82
+
- These are both relatively constant for each monitor, so it is easy enough to account for with a constant temporal offset.
84
83
- You can either find the values online (RTINGS - input lag) or use a camera to find it for your specific experiment (or even the specific part of the experiment)
85
84
- Then add the delay to your display timestamps
86
85
87
-
### Not every pixel shows up at the same time?!?
86
+
## Can I just use a photodiode sync pulse?
87
+
88
+
- Absolutely! And it is WAY better than the above! It tells you exactly when the pixels under your photodiode change.
89
+
- Side notes about photodiodes
90
+
- Don't forget when using a photodiode to take into account how long it takes to actually SEND the data from the photodiode controller to your app and timestamp it. That delay could make a difference (usually this is negligible and is less than 1ms).
91
+
- Downsides of photodiodes
92
+
1. You can't actually change it every frame. So this will only work for specific events that don't happen too rapidly in succession, not for every frame that is logged like the "frameDisplayed" event. So if you have some things changing on screen very fast, consider your photodiode setup carefully.
93
+
1. You can't use it in VR headsets. Although you may be able to use it to figure out what your constant offset from your log events should be and then hope the game doesn't lag.
94
+
- But, yet again, it still doesn't get you all the way there... Not all the pixels on a monitor change at the same time.
95
+
96
+
## An Intro to Monitors (Part 3): Not every pixel shows up at the same time?!?
88
97
89
98
- Right again! Not every pixel on a monitor is updated at the same time. As a matter of fact, they only update one at a time.
90
99
- Monitors actually place them from left to right and top to bottom (like you are typing a word document).
@@ -97,16 +106,75 @@ Well then, here is a quick introduction to the pain that is figuring out that in
97
106
- What horizontal and vertical pixel you care about on the screen
98
107
- If you don't know how to do that then follow these steps: download [GIMP](https://www.gimp.org/), open the file, mouse over where you care about, and then look at the bottom left where it shows your X (horizontal) and Y (vertical) coordinates.
99
108
- Your monitor refresh rate (ex: 120Hz)
100
-
109
+
- We are going to assume your photodiode is in the top left corner for the following exammple. This means that the photodiode pixels turn on first, at the start of the frame.
101
110
- To find when the pixel will display after the start of the frame, it would be:
102
111
$$\frac{horizontal\_pixels * Y + X}{horizonal\_pixels * vertical\_pixels * refresh\_rate}$$
103
-
- As an example, if the pixel coordinates were $X = 960$ (halfway across the screen) and $Y = 540$ (halfway down the screen), then the pixel you cared about would display 4.17ms after the start time of the frame.
112
+
- Example
113
+
- Input values
114
+
- pixel coordinate $X = 960$ (halfway across the screen)
115
+
- pixel coordinate $Y = 540$ (halfway down the screen)
116
+
- $refresh\_rate = 120Hz$
117
+
- $horizontal\_pixels = 1920$
118
+
- $vertical\_pixels = 1080$
119
+
- Then the pixel you cared about would display 4.17ms after the start time of the frame.
- We will call this this 'pixel location timing offset'
122
+
123
+
## So how do I account for all of this stuff?
124
+
125
+
- No matter what, make sure VSync is on.
126
+
127
+
### Photo Diode Method
128
+
129
+
- This is definitely the easiest. Just account for the pixel location timing offset.
130
+
- To be extra precise, try to account for the delay of the photodiode logging as well.
131
+
132
+
### Frame Displayed Method
133
+
134
+
- This obviously has a lot more things to account for.
135
+
1. pixel location timing offset
136
+
1. OS level double/tripple buffering
137
+
- Check in your OS settings
138
+
- If you use MacOS, see the [MacOS forced tripple buffering](#macos-forced-tripple-buffering) section
139
+
1. Graphics card double/tripple buffering
140
+
- See this in your graphics card software settings
141
+
1. Unity double/tripple buffering
142
+
- Check for the [QualitySettings.maxQueuedFrames](https://docs.unity3d.com/6000.2/Documentation/ScriptReference/QualitySettings-maxQueuedFrames.html) in your code and set it to how many buffered frames you want. By default, the value is 2, meaning there is 1 extra buffered frame. This would create a delay of 1 frame's time between your "frameDisplayed" event timestamp and the when the gpu actually got the data to write to screen.
143
+
1. Monitor pixel change times
144
+
- See the [Intro to Monitors (Part 2)](#an-intro-to-monitors-part-2-pixels-dont-change-right-away) section.
145
+
- How do I actually account for all of them? What is the total offset?
146
+
1. We make the assumption that there is not much lag between the end of vblank and when the graphics card writes to the screen.
147
+
1. When processing your log file, for every event, find the next "frameDisplayed" event, add all of the above sources of delay to the "frameDisplayed" event timestamp, and then replace the event timestamp with that value.
148
+
- Simple verification method
149
+
- Required supplies:
150
+
- A phone with 8x slowmo recording (240fps) or faster
151
+
- A little LED light that you can turn on from some C# code
152
+
- Process:
153
+
1. Add something to the experiment that displays to the top left screen.
154
+
1. Add a line to turn on the LED for 100ms in the *PsyForge/Runtime/Scripts/DataMangement/EventReporter.cs* file directly after the *LogTS("frameDisplayed", ...)* line.
155
+
1. Run your game while recording both the screen and the LED with you camera.
156
+
1. Open the video and go frame by frame until you see the thing appear in the top left of the screen.
157
+
1. Now count how many frames you have to go back until the LED initially turned on.
158
+
1. Calculate your total offset using the info you gathered above about your monitor and computer.
159
+
1. Calculate your real offset $number\_of\_camera\_frames * camera\_fps$
160
+
1. Compare your total offset and your real offset. If your real offset is consistently greater/smaller than your total offset by $1/camera\_fps$ then you calculated your total offset wrong.
161
+
- This will tell you, within $1/camera\_fps$ seconds, how accurate your logging is. So if your camera shoots at 240fps, then this will tell you your accuracy within 4ms.
162
+
163
+
## Extra facts that may make you sad/annoyed
105
164
106
-
##So your "frameDisplayed" event isn't even exact?!? (Part 2)
165
+
### MacOS forced tripple buffering
107
166
108
-
- So if you want the exact time you will have to account for all of that stuff. So let's make an see some equations and write an example.
109
-
- TODO
167
+
MacOS versions 10.13 and later have a forced triple buffering by the OS (you can't turn it off), which means there is always a 1 frame delay between when your task "says" it did a thing and when it actually does get displayed to the screen. This doesn't impact durations or relative timings within the task, but it does impact timings relative to externally collected data (ex: EEG). Thank you PsychoPy for actually discussing this issue publicly [in a basic form here](https://discourse.psychopy.org/t/correcting-for-mac-triple-buffering/15391) and [more in-depth here](https://github.com/psychopy/psychopy/issues/2250).
168
+
169
+
## What is GSync and FreeSync? Should I use them?
170
+
171
+
- GSync and FreeSync are the next great innovation in monitor hardware.
172
+
- GSync and FreeSync are the same thing, but GSync is for NVidia graphics cards and FreeSync is for AMD cards. For future bullet points I will just call it GSync, but know that everything applies to both GSync and Freesync.
173
+
- Remember with VSync how the monitor waits for each frame and displays the whole frame all at once (doesn't write part of a frame to screen), well that comes with a downside. If you do NOT get the frame to ready in time then the next time the monitor can write to the screen is on the next VSync. This effectively halves your monitor refresh rate.
174
+
- GSync actually allows the monitor to refresh the screen with the new frame as SOON as your new frame is completely ready. This means that there isn't screen tearing, while avoiding the delays associated with VSync.
175
+
- The main benefit is smoother gameplay when there is a slight lag in your computer.
176
+
- When calulating the pixel location timing offset with GSync, you still use the normal refresh rate of the monitor for the calculation, because that is how fast the pixels will change.
177
+
- NOTE: This only works on GSync and FreeSync certified monitors since this is hardware feature of the new monitors.
110
178
111
179
## Maybe consider a CRT
112
180
@@ -117,15 +185,7 @@ Well then, here is a quick introduction to the pain that is figuring out that in
117
185
- Since these things are huge, I recommend buying locally. Most people from your local Retro-Gaming groups will probably have extra CRTs they would be willing to sell.
118
186
- Yes, it is possible to get 1080p 60Hz CRTs, but it may be tricky to find.
119
187
120
-
## Extra facts that may make you sad/annoyed
121
-
122
-
- MacOS versions 10.13 and later have a forced triple buffering by the OS (you can't turn it off), which means there is always a 1 frame delay between when your task "says" it did a thing and when it actually does get displayed to the screen. This doesn't impact durations or relative timings within the task, but it does impact timings relative to externally collected data (ex: EEG). Thank you PsychoPy for actually discussing this issue publicly [in a basic form here](https://discourse.psychopy.org/t/correcting-for-mac-triple-buffering/15391) and [more in-depth here](https://github.com/psychopy/psychopy/issues/2250).
0 commit comments