|
| 1 | +# Practical-RIFE Accuracy Flow |
| 2 | + |
| 3 | +`qat_loop.py` measures Practical-RIFE eager, PTQ, and QAT accuracy on real |
| 4 | +frame triples. It is intended as the accuracy companion to VGF performance |
| 5 | +measurements, so each reported artifact can be tied back to a reproducible |
| 6 | +model, ExecuTorch revision, and export command. |
| 7 | + |
| 8 | +## Inputs |
| 9 | + |
| 10 | +Use a standard Practical-RIFE checkout with this layout: |
| 11 | + |
| 12 | +```text |
| 13 | +<model-root>/train_log/IFNet_HDv3.py |
| 14 | +<model-root>/train_log/flownet.pkl |
| 15 | +``` |
| 16 | + |
| 17 | +The triples file must identify frame interpolation samples in this order: |
| 18 | + |
| 19 | +```text |
| 20 | +name,input0,input1,target |
| 21 | +sample_00000,/path/frame_0.png,/path/frame_2.png,/path/frame_1.png |
| 22 | +``` |
| 23 | + |
| 24 | +The script treats `input0` and `input1` as the two model inputs and `target` |
| 25 | +as the expected middle frame. |
| 26 | + |
| 27 | +## Real Accuracy Run |
| 28 | + |
| 29 | +Use real frame triples for accuracy reporting. This example compares eager, |
| 30 | +PTQ, and QAT at the default 768x384 shape while preserving uint8 image IO: |
| 31 | + |
| 32 | +```bash |
| 33 | +python examples/arm/QAT_example/qat_loop.py \ |
| 34 | + --triples-list <path-to-real-triples.csv> \ |
| 35 | + --model-root <path-to-Practical-RIFE> \ |
| 36 | + --height 768 \ |
| 37 | + --width 384 \ |
| 38 | + --max-triples 10 \ |
| 39 | + --calibration-samples 8 \ |
| 40 | + --mode all \ |
| 41 | + --io-quantization uint8 \ |
| 42 | + --qat-samples 8 \ |
| 43 | + --qat-steps 50 \ |
| 44 | + --qat-lr 1e-5 \ |
| 45 | + --output-dir out/rife_accuracy_10_real_triples_qat50 |
| 46 | +``` |
| 47 | + |
| 48 | +The output directory contains: |
| 49 | + |
| 50 | +- `report.md`: human-readable aggregate metrics. |
| 51 | +- `metrics.json`: aggregate and per-sample metrics. |
| 52 | +- `metrics.csv`: tabular per-sample metrics. |
| 53 | +- `quantization_coverage/`: PTQ and QAT graph coverage details. |
| 54 | + |
| 55 | +## Smoke Runs |
| 56 | + |
| 57 | +Small random or short QAT runs are useful only for checking that the flow runs. |
| 58 | +Do not use them as model-quality evidence. For example, two samples and three |
| 59 | +QAT steps can produce very low PSNR while still proving that eager, PTQ, and |
| 60 | +QAT execution is wired correctly. |
| 61 | + |
| 62 | +## Reference Samples |
| 63 | + |
| 64 | +Keep reference images outside this repository and pass them through |
| 65 | +`--triples-list`. For internal comparisons, use a fixed triples list and keep |
| 66 | +the image bundle unchanged across graph and quantization experiments. |
| 67 | + |
| 68 | +The triples list is the reproducibility contract. It should use repository- or |
| 69 | +bundle-relative paths where possible, so another user can unpack the same image |
| 70 | +bundle and run the same command without editing absolute paths. |
| 71 | + |
| 72 | +## Reporting Checklist |
| 73 | + |
| 74 | +When sharing accuracy or performance data, include: |
| 75 | + |
| 76 | +- Practical-RIFE commit hash. |
| 77 | +- ExecuTorch commit hash. |
| 78 | +- `qat_loop.py` command line and flags. |
| 79 | +- Dataset or triples-list identity. |
| 80 | +- Input shape and preprocessing mode. |
| 81 | +- IO quantization mode. |
| 82 | +- PTQ calibration sample count. |
| 83 | +- QAT sample count, step count, and learning rate. |
| 84 | +- Reference image bundle identity. |
| 85 | +- Whether a VGF or PTE was exported from the same run. |
| 86 | + |
| 87 | +This keeps accuracy numbers comparable with board performance results and |
| 88 | +avoids mixing smoke-test numbers with real-data validation. |
0 commit comments