|
| 1 | +<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd"> |
| 2 | +<html> |
| 3 | +<!-- Created by GNU Texinfo 6.7, http://www.gnu.org/software/texinfo/ --> |
| 4 | + <head> |
| 5 | + <meta charset="utf-8"> |
| 6 | + <title> |
| 7 | + FFmpeg Automated Testing Environment |
| 8 | + </title> |
| 9 | + <meta name="viewport" content="width=device-width,initial-scale=1.0"> |
| 10 | + <link rel="stylesheet" type="text/css" href="bootstrap.min.css"> |
| 11 | + <link rel="stylesheet" type="text/css" href="style.min.css"> |
| 12 | + </head> |
| 13 | + <body> |
| 14 | + <div class="container"> |
| 15 | + <h1> |
| 16 | + FFmpeg Automated Testing Environment |
| 17 | + </h1> |
| 18 | +<div align="center"> |
| 19 | +</div> |
| 20 | + |
| 21 | + |
| 22 | +<a name="Top"></a> |
| 23 | +<a name="SEC_Top"></a> |
| 24 | + |
| 25 | +<span id="SEC_Contents"></span> |
| 26 | +<h2 class="contents-heading">Table of Contents</h2> |
| 27 | + |
| 28 | +<div class="contents"> |
| 29 | + |
| 30 | +<ul class="no-bullet"> |
| 31 | + <li><a id="toc-Introduction" href="#Introduction">1 Introduction</a></li> |
| 32 | + <li><a id="toc-Using-FATE-from-your-FFmpeg-source-directory" href="#Using-FATE-from-your-FFmpeg-source-directory">2 Using FATE from your FFmpeg source directory</a></li> |
| 33 | + <li><a id="toc-Submitting-the-results-to-the-FFmpeg-result-aggregation-server" href="#Submitting-the-results-to-the-FFmpeg-result-aggregation-server">3 Submitting the results to the FFmpeg result aggregation server</a></li> |
| 34 | + <li><a id="toc-Uploading-new-samples-to-the-fate-suite" href="#Uploading-new-samples-to-the-fate-suite">4 Uploading new samples to the fate suite</a></li> |
| 35 | + <li><a id="toc-FATE-makefile-targets-and-variables" href="#FATE-makefile-targets-and-variables">5 FATE makefile targets and variables</a> |
| 36 | + <ul class="no-bullet"> |
| 37 | + <li><a id="toc-Makefile-targets" href="#Makefile-targets">5.1 Makefile targets</a></li> |
| 38 | + <li><a id="toc-Makefile-variables" href="#Makefile-variables">5.2 Makefile variables</a></li> |
| 39 | + <li><a id="toc-Examples" href="#Examples">5.3 Examples</a></li> |
| 40 | + </ul></li> |
| 41 | +</ul> |
| 42 | +</div> |
| 43 | + |
| 44 | + |
| 45 | +<a name="Introduction"></a> |
| 46 | +<h2 class="chapter">1 Introduction<span class="pull-right"><a class="anchor hidden-xs" href="#Introduction" aria-hidden="true">#</a> <a class="anchor hidden-xs"href="#toc-Introduction" aria-hidden="true">TOC</a></span></h2> |
| 47 | + |
| 48 | +<p>FATE is an extended regression suite on the client-side and a means |
| 49 | +for results aggregation and presentation on the server-side. |
| 50 | +</p> |
| 51 | +<p>The first part of this document explains how you can use FATE from |
| 52 | +your FFmpeg source directory to test your ffmpeg binary. The second |
| 53 | +part describes how you can run FATE to submit the results to FFmpeg’s |
| 54 | +FATE server. |
| 55 | +</p> |
| 56 | +<p>In any way you can have a look at the publicly viewable FATE results |
| 57 | +by visiting this website: |
| 58 | +</p> |
| 59 | +<p><a href="http://fate.ffmpeg.org/">http://fate.ffmpeg.org/</a> |
| 60 | +</p> |
| 61 | +<p>This is especially recommended for all people contributing source |
| 62 | +code to FFmpeg, as it can be seen if some test on some platform broke |
| 63 | +with their recent contribution. This usually happens on the platforms |
| 64 | +the developers could not test on. |
| 65 | +</p> |
| 66 | +<p>The second part of this document describes how you can run FATE to |
| 67 | +submit your results to FFmpeg’s FATE server. If you want to submit your |
| 68 | +results be sure to check that your combination of CPU, OS and compiler |
| 69 | +is not already listed on the above mentioned website. |
| 70 | +</p> |
| 71 | +<p>In the third part you can find a comprehensive listing of FATE makefile |
| 72 | +targets and variables. |
| 73 | +</p> |
| 74 | + |
| 75 | +<a name="Using-FATE-from-your-FFmpeg-source-directory"></a> |
| 76 | +<h2 class="chapter">2 Using FATE from your FFmpeg source directory<span class="pull-right"><a class="anchor hidden-xs" href="#Using-FATE-from-your-FFmpeg-source-directory" aria-hidden="true">#</a> <a class="anchor hidden-xs"href="#toc-Using-FATE-from-your-FFmpeg-source-directory" aria-hidden="true">TOC</a></span></h2> |
| 77 | + |
| 78 | +<p>If you want to run FATE on your machine you need to have the samples |
| 79 | +in place. You can get the samples via the build target fate-rsync. |
| 80 | +Use this command from the top-level source directory: |
| 81 | +</p> |
| 82 | +<div class="example"> |
| 83 | +<pre class="example">make fate-rsync SAMPLES=fate-suite/ |
| 84 | +make fate SAMPLES=fate-suite/ |
| 85 | +</pre></div> |
| 86 | + |
| 87 | +<p>The above commands set the samples location by passing a makefile |
| 88 | +variable via command line. It is also possible to set the samples |
| 89 | +location at source configuration time by invoking configure with |
| 90 | +<samp>--samples=<path to the samples directory></samp>. Afterwards you can |
| 91 | +invoke the makefile targets without setting the <var>SAMPLES</var> makefile |
| 92 | +variable. This is illustrated by the following commands: |
| 93 | +</p> |
| 94 | +<div class="example"> |
| 95 | +<pre class="example">./configure --samples=fate-suite/ |
| 96 | +make fate-rsync |
| 97 | +make fate |
| 98 | +</pre></div> |
| 99 | + |
| 100 | +<p>Yet another way to tell FATE about the location of the sample |
| 101 | +directory is by making sure the environment variable FATE_SAMPLES |
| 102 | +contains the path to your samples directory. This can be achieved |
| 103 | +by e.g. putting that variable in your shell profile or by setting |
| 104 | +it in your interactive session. |
| 105 | +</p> |
| 106 | +<div class="example"> |
| 107 | +<pre class="example">FATE_SAMPLES=fate-suite/ make fate |
| 108 | +</pre></div> |
| 109 | + |
| 110 | +<div class="info"> |
| 111 | +<p>Do not put a ’~’ character in the samples path to indicate a home |
| 112 | +directory. Because of shell nuances, this will cause FATE to fail. |
| 113 | +</p></div> |
| 114 | +<p>To use a custom wrapper to run the test, pass <samp>--target-exec</samp> to |
| 115 | +<code>configure</code> or set the <var>TARGET_EXEC</var> Make variable. |
| 116 | +</p> |
| 117 | + |
| 118 | +<a name="Submitting-the-results-to-the-FFmpeg-result-aggregation-server"></a> |
| 119 | +<h2 class="chapter">3 Submitting the results to the FFmpeg result aggregation server<span class="pull-right"><a class="anchor hidden-xs" href="#Submitting-the-results-to-the-FFmpeg-result-aggregation-server" aria-hidden="true">#</a> <a class="anchor hidden-xs"href="#toc-Submitting-the-results-to-the-FFmpeg-result-aggregation-server" aria-hidden="true">TOC</a></span></h2> |
| 120 | + |
| 121 | +<p>To submit your results to the server you should run fate through the |
| 122 | +shell script <samp>tests/fate.sh</samp> from the FFmpeg sources. This script needs |
| 123 | +to be invoked with a configuration file as its first argument. |
| 124 | +</p> |
| 125 | +<div class="example"> |
| 126 | +<pre class="example">tests/fate.sh /path/to/fate_config |
| 127 | +</pre></div> |
| 128 | + |
| 129 | +<p>A configuration file template with comments describing the individual |
| 130 | +configuration variables can be found at <samp>doc/fate_config.sh.template</samp>. |
| 131 | +</p> |
| 132 | +<p>The mentioned configuration template is also available here: |
| 133 | +</p><pre class="verbatim">slot= # some unique identifier |
| 134 | +repo=git://source.ffmpeg.org/ffmpeg.git # the source repository |
| 135 | +#branch=release/2.6 # the branch to test |
| 136 | +samples= # path to samples directory |
| 137 | +workdir= # directory in which to do all the work |
| 138 | +#fate_recv="ssh -T fate@fate.ffmpeg.org" # command to submit report |
| 139 | +comment= # optional description |
| 140 | +build_only= # set to "yes" for a compile-only instance that skips tests |
| 141 | +ignore_tests= |
| 142 | + |
| 143 | +# the following are optional and map to configure options |
| 144 | +arch= |
| 145 | +cpu= |
| 146 | +cross_prefix= |
| 147 | +as= |
| 148 | +cc= |
| 149 | +ld= |
| 150 | +target_os= |
| 151 | +sysroot= |
| 152 | +target_exec= |
| 153 | +target_path= |
| 154 | +target_samples= |
| 155 | +extra_cflags= |
| 156 | +extra_ldflags= |
| 157 | +extra_libs= |
| 158 | +extra_conf= # extra configure options not covered above |
| 159 | + |
| 160 | +#make= # name of GNU make if not 'make' |
| 161 | +makeopts= # extra options passed to 'make' |
| 162 | +#makeopts_fate= # extra options passed to 'make' when running tests, |
| 163 | + # defaulting to makeopts above if this is not set |
| 164 | +#tar= # command to create a tar archive from its arguments on stdout, |
| 165 | + # defaults to 'tar c' |
| 166 | +</pre> |
| 167 | +<p>Create a configuration that suits your needs, based on the configuration |
| 168 | +template. The <code>slot</code> configuration variable can be any string that is not |
| 169 | +yet used, but it is suggested that you name it adhering to the following |
| 170 | +pattern ‘<samp><var>arch</var>-<var>os</var>-<var>compiler</var>-<var>compiler version</var></samp>’. The |
| 171 | +configuration file itself will be sourced in a shell script, therefore all |
| 172 | +shell features may be used. This enables you to setup the environment as you |
| 173 | +need it for your build. |
| 174 | +</p> |
| 175 | +<p>For your first test runs the <code>fate_recv</code> variable should be empty or |
| 176 | +commented out. This will run everything as normal except that it will omit |
| 177 | +the submission of the results to the server. The following files should be |
| 178 | +present in $workdir as specified in the configuration file: |
| 179 | +</p> |
| 180 | +<ul> |
| 181 | +<li> configure.log |
| 182 | + </li><li> compile.log |
| 183 | + </li><li> test.log |
| 184 | + </li><li> report |
| 185 | + </li><li> version |
| 186 | +</li></ul> |
| 187 | + |
| 188 | +<p>When you have everything working properly you can create an SSH key pair |
| 189 | +and send the public key to the FATE server administrator who can be contacted |
| 190 | +at the email address <a href="mailto:fate-admin@ffmpeg.org">fate-admin@ffmpeg.org</a>. |
| 191 | +</p> |
| 192 | +<p>Configure your SSH client to use public key authentication with that key |
| 193 | +when connecting to the FATE server. Also do not forget to check the identity |
| 194 | +of the server and to accept its host key. This can usually be achieved by |
| 195 | +running your SSH client manually and killing it after you accepted the key. |
| 196 | +The FATE server’s fingerprint is: |
| 197 | +</p> |
| 198 | +<dl compact="compact"> |
| 199 | +<dt>‘<samp>RSA</samp>’</dt> |
| 200 | +<dd><p>d3:f1:83:97:a4:75:2b:a6:fb:d6:e8:aa:81:93:97:51 |
| 201 | +</p></dd> |
| 202 | +<dt>‘<samp>ECDSA</samp>’</dt> |
| 203 | +<dd><p>76:9f:68:32:04:1e:d5:d4:ec:47:3f:dc:fc:18:17:86 |
| 204 | +</p></dd> |
| 205 | +</dl> |
| 206 | + |
| 207 | +<p>If you have problems connecting to the FATE server, it may help to try out |
| 208 | +the <code>ssh</code> command with one or more <samp>-v</samp> options. You should |
| 209 | +get detailed output concerning your SSH configuration and the authentication |
| 210 | +process. |
| 211 | +</p> |
| 212 | +<p>The only thing left is to automate the execution of the fate.sh script and |
| 213 | +the synchronisation of the samples directory. |
| 214 | +</p> |
| 215 | +<a name="Uploading-new-samples-to-the-fate-suite"></a> |
| 216 | +<h2 class="chapter">4 Uploading new samples to the fate suite<span class="pull-right"><a class="anchor hidden-xs" href="#Uploading-new-samples-to-the-fate-suite" aria-hidden="true">#</a> <a class="anchor hidden-xs"href="#toc-Uploading-new-samples-to-the-fate-suite" aria-hidden="true">TOC</a></span></h2> |
| 217 | + |
| 218 | +<p>If you need a sample uploaded send a mail to samples-request. |
| 219 | +</p> |
| 220 | +<p>This is for developers who have an account on the fate suite server. |
| 221 | +If you upload new samples, please make sure they are as small as possible, |
| 222 | +space on each client, network bandwidth and so on benefit from smaller test cases. |
| 223 | +Also keep in mind older checkouts use existing sample files, that means in |
| 224 | +practice generally do not replace, remove or overwrite files as it likely would |
| 225 | +break older checkouts or releases. |
| 226 | +Also all needed samples for a commit should be uploaded, ideally 24 |
| 227 | +hours, before the push. |
| 228 | +If you need an account for frequently uploading samples or you wish to help |
| 229 | +others by doing that send a mail to ffmpeg-devel. |
| 230 | +</p> |
| 231 | +<div class="example"> |
| 232 | +<pre class="example">#First update your local samples copy: |
| 233 | +rsync -vauL --chmod=Dg+s,Duo+x,ug+rw,o+r,o-w,+X fate-suite.ffmpeg.org:/home/samples/fate-suite/ ~/fate-suite |
| 234 | + |
| 235 | +#Then do a dry run checking what would be uploaded: |
| 236 | +rsync -vanL --no-g --chmod=Dg+s,Duo+x,ug+rw,o+r,o-w,+X ~/fate-suite/ fate-suite.ffmpeg.org:/home/samples/fate-suite |
| 237 | + |
| 238 | +#Upload the files: |
| 239 | +rsync -vaL --no-g --chmod=Dg+s,Duo+x,ug+rw,o+r,o-w,+X ~/fate-suite/ fate-suite.ffmpeg.org:/home/samples/fate-suite |
| 240 | +</pre></div> |
| 241 | + |
| 242 | + |
| 243 | +<a name="FATE-makefile-targets-and-variables"></a> |
| 244 | +<h2 class="chapter">5 FATE makefile targets and variables<span class="pull-right"><a class="anchor hidden-xs" href="#FATE-makefile-targets-and-variables" aria-hidden="true">#</a> <a class="anchor hidden-xs"href="#toc-FATE-makefile-targets-and-variables" aria-hidden="true">TOC</a></span></h2> |
| 245 | + |
| 246 | +<a name="Makefile-targets"></a> |
| 247 | +<h3 class="section">5.1 Makefile targets<span class="pull-right"><a class="anchor hidden-xs" href="#Makefile-targets" aria-hidden="true">#</a> <a class="anchor hidden-xs"href="#toc-Makefile-targets" aria-hidden="true">TOC</a></span></h3> |
| 248 | + |
| 249 | +<dl compact="compact"> |
| 250 | +<dt><samp>fate-rsync</samp></dt> |
| 251 | +<dd><p>Download/synchronize sample files to the configured samples directory. |
| 252 | +</p> |
| 253 | +</dd> |
| 254 | +<dt><samp>fate-list</samp></dt> |
| 255 | +<dd><p>Will list all fate/regression test targets. |
| 256 | +</p> |
| 257 | +</dd> |
| 258 | +<dt><samp>fate</samp></dt> |
| 259 | +<dd><p>Run the FATE test suite (requires the fate-suite dataset). |
| 260 | +</p></dd> |
| 261 | +</dl> |
| 262 | + |
| 263 | +<a name="Makefile-variables"></a> |
| 264 | +<h3 class="section">5.2 Makefile variables<span class="pull-right"><a class="anchor hidden-xs" href="#Makefile-variables" aria-hidden="true">#</a> <a class="anchor hidden-xs"href="#toc-Makefile-variables" aria-hidden="true">TOC</a></span></h3> |
| 265 | + |
| 266 | +<dl compact="compact"> |
| 267 | +<dt><code>V</code></dt> |
| 268 | +<dd><p>Verbosity level, can be set to 0, 1 or 2. |
| 269 | + </p><ul> |
| 270 | +<li> 0: show just the test arguments |
| 271 | + </li><li> 1: show just the command used in the test |
| 272 | + </li><li> 2: show everything |
| 273 | + </li></ul> |
| 274 | + |
| 275 | +</dd> |
| 276 | +<dt><code>SAMPLES</code></dt> |
| 277 | +<dd><p>Specify or override the path to the FATE samples at make time, it has a |
| 278 | +meaning only while running the regression tests. |
| 279 | +</p> |
| 280 | +</dd> |
| 281 | +<dt><code>THREADS</code></dt> |
| 282 | +<dd><p>Specify how many threads to use while running regression tests, it is |
| 283 | +quite useful to detect thread-related regressions. |
| 284 | +</p> |
| 285 | +</dd> |
| 286 | +<dt><code>THREAD_TYPE</code></dt> |
| 287 | +<dd><p>Specify which threading strategy test, either ‘<samp>slice</samp>’ or ‘<samp>frame</samp>’, |
| 288 | +by default ‘<samp>slice+frame</samp>’ |
| 289 | +</p> |
| 290 | +</dd> |
| 291 | +<dt><code>CPUFLAGS</code></dt> |
| 292 | +<dd><p>Specify CPU flags. |
| 293 | +</p> |
| 294 | +</dd> |
| 295 | +<dt><code>TARGET_EXEC</code></dt> |
| 296 | +<dd><p>Specify or override the wrapper used to run the tests. |
| 297 | +The <code>TARGET_EXEC</code> option provides a way to run FATE wrapped in |
| 298 | +<code>valgrind</code>, <code>qemu-user</code> or <code>wine</code> or on remote targets |
| 299 | +through <code>ssh</code>. |
| 300 | +</p> |
| 301 | +</dd> |
| 302 | +<dt><code>GEN</code></dt> |
| 303 | +<dd><p>Set to ‘<samp>1</samp>’ to generate the missing or mismatched references. |
| 304 | +</p> |
| 305 | +</dd> |
| 306 | +<dt><code>HWACCEL</code></dt> |
| 307 | +<dd><p>Specify which hardware acceleration to use while running regression tests, |
| 308 | +by default ‘<samp>none</samp>’ is used. |
| 309 | +</p> |
| 310 | +</dd> |
| 311 | +<dt><code>KEEP</code></dt> |
| 312 | +<dd><p>Set to ‘<samp>1</samp>’ to keep temp files generated by fate test(s) when test is successful. |
| 313 | +Default is ‘<samp>0</samp>’, which removes these files. Files are always kept when a test |
| 314 | +fails. |
| 315 | +</p> |
| 316 | +</dd> |
| 317 | +</dl> |
| 318 | + |
| 319 | +<a name="Examples"></a> |
| 320 | +<h3 class="section">5.3 Examples<span class="pull-right"><a class="anchor hidden-xs" href="#Examples" aria-hidden="true">#</a> <a class="anchor hidden-xs"href="#toc-Examples" aria-hidden="true">TOC</a></span></h3> |
| 321 | + |
| 322 | +<div class="example"> |
| 323 | +<pre class="example">make V=1 SAMPLES=/var/fate/samples THREADS=2 CPUFLAGS=mmx fate |
| 324 | +</pre></div> |
| 325 | + |
| 326 | + <p style="font-size: small;"> |
| 327 | + This document was generated using <a href="http://www.gnu.org/software/texinfo/"><em>makeinfo</em></a>. |
| 328 | + </p> |
| 329 | + </div> |
| 330 | + </body> |
| 331 | +</html> |
0 commit comments