Update inference.html

Author: zhushangwen

docs/TheMatrixDocs/inference.html

@@ -456,45 +456,93 @@ <h2>1.Inference with the_matrix.py<a class="headerlink" href="#inference-with-th
 </section>
 <section id="inference-with-run-interactive-sh">
 <h2>2. Inference with run_interactive.sh<a class="headerlink" href="#inference-with-run-interactive-sh" title="Permalink to this heading">#</a></h2>
-<p>The <cite>run_interactive.sh</cite> script orchestrates a multi-stage pipeline using Ray, DIT and VAE processes. It performs the following steps:</p>
+<section id="summary">
+<h3>Summary<a class="headerlink" href="#summary" title="Permalink to this heading">#</a></h3>
+<p>run_interactive.sh launches a fully parallelized, low-latency pipeline that generates video at <strong>16 FPS</strong> end-to-end (i.e. real-time). This script leverages our 8-GPU DiT &amp; VAE parallel inference, stream consistency models, and fused data training to reduce a single-GPU baseline’s 32 s per 4 s video down to 4 s—a <strong>8× speedup</strong>—while maintaining infinite-horizon stability.</p>
+</section>
+<section id="highlights">
+<h3>Highlights<a class="headerlink" href="#highlights" title="Permalink to this heading">#</a></h3>
+<ul class="simple">
+<li><p><strong>8-GPU Parallel Inference</strong>
+DiT and VAE stages each slice work across 8 GPUs for a <strong>6–8× speedup</strong> vs. single-GPU.</p></li>
+<li><p><strong>Stream Consistency Models</strong>
+Novel consistency losses yield <strong>7–10× higher throughput</strong> over naïve frame-by-frame generation.</p></li>
+<li><p><strong>Real-Time Feedback Loop</strong>
+Sustains a continuous <strong>16 FPS</strong> generation/playback cycle with <strong>&lt; 50 ms</strong> input-to-output latency.</p></li>
+</ul>
+</section>
+<section id="two-inference-modes">
+<h3>Two Inference Modes<a class="headerlink" href="#two-inference-modes" title="Permalink to this heading">#</a></h3>
 <ol class="arabic simple">
-<li><p>Stop any existing Ray cluster</p></li>
-<li><p>Compute <cite>CUDA_VISIBLE_DEVICES</cite> based on configured GPU counts</p></li>
-<li><p>Start Ray head node</p></li>
-<li><p>Launch, in order (some in background):
-- <cite>create_ray_pipe.py</cite>
-- <cite>main.py</cite>
-- <cite>start_dit.sh</cite> (DIT inference)
-- <cite>start_decoding_daemon.py</cite> (VAE decoding daemon)</p></li>
+<li><p><strong>API-Driven (`the_matrix.py`)</strong>
+- Use when embedding generation inside your Python app.
+- Offers interactive control via <cite>the_matrix.generate(…)</cite> calls.
+- Suitable for few-shot or ad-hoc video snippets.</p></li>
+<li><p><strong>Scripted Pipeline (`run_interactive.sh`)</strong>
+- End-to-end shell script for bulk or real-time production.
+- Spins up a Ray cluster, runs all stages in parallel, and tears down automatically.
+- Ideal for continuous/live deployments or performance benchmarking.</p></li>
 </ol>
+</section>
+<section id="performance-comparison">
+<h3>Performance Comparison<a class="headerlink" href="#performance-comparison" title="Permalink to this heading">#</a></h3>
+<table class="table" id="id1">
+<caption><span class="caption-text">Inference throughput comparison for a 4 s video</span><a class="headerlink" href="#id1" title="Permalink to this table">#</a></caption>
+<colgroup>
+<col style="width: 25.0%" />
+<col style="width: 25.0%" />
+<col style="width: 25.0%" />
+<col style="width: 25.0%" />
+</colgroup>
+<thead>
+<tr class="row-odd"><th class="head"><p>Mode</p></th>
+<th class="head"><p>GPUs used</p></th>
+<th class="head"><p>FPS achieved</p></th>
+<th class="head"><p>Total latency</p></th>
+</tr>
+</thead>
+<tbody>
+<tr class="row-even"><td><p>Baseline API</p></td>
+<td><p>1</p></td>
+<td><p>~2</p></td>
+<td><p>~32 s</p></td>
+</tr>
+<tr class="row-odd"><td><p>Interactive</p></td>
+<td><p>8</p></td>
+<td><p>16</p></td>
+<td><p>~4 s</p></td>
+</tr>
+</tbody>
+</table>
+</section>
 <section id="configuration">
 <h3>Configuration<a class="headerlink" href="#configuration" title="Permalink to this heading">#</a></h3>
-<p>At the top of <cite>run_interactive.sh</cite>, set the following variables:</p>
-<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span><span class="c1"># Number of GPUs for DIT stage</span>
+<p>At the top of <cite>run_interactive.sh</cite>, set:</p>
+<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span><span class="c1"># GPUs for DiT stage (must sum to 8)</span>
 <span class="nv">NUM_GPUS_DIT</span><span class="o">=</span><span class="m">1</span>
 
-<span class="c1"># Number of GPUs for VAE stage</span>
-<span class="nv">NUM_GPUS_VAE</span><span class="o">=</span><span class="m">3</span>
+<span class="c1"># GPUs for VAE stage (NUM_GPUS_DIT + NUM_GPUS_VAE = 8)</span>
+<span class="nv">NUM_GPUS_VAE</span><span class="o">=</span><span class="m">7</span>
 
 <span class="c1"># Path to stage4 model weights</span>
 <span class="nv">MODEL_PATH</span><span class="o">=</span><span class="s2">&quot;../models/stage4&quot;</span>
 </pre></div>
 </div>
-<p>The script will assemble:</p>
+<p>The script computes:</p>
 <ul class="simple">
-<li><p><strong>GPU_IDS</strong>: a comma-separated list <cite>NUM_GPUS_DIT,NUM_GPUS_DIT+1,…</cite></p></li>
-<li><p><strong>CUDA_VISIBLE_DEVICES</strong>: exported before Ray and Python processes</p></li>
+<li><p><strong>GPU_IDS</strong>: comma-separated list <cite>NUM_GPUS_DIT,…,NUM_GPUS_DIT+NUM_GPUS_VAE-1</cite></p></li>
+<li><p><strong>CUDA_VISIBLE_DEVICES</strong>: exported for Ray &amp; all Python processes</p></li>
 </ul>
 </section>
 <section id="usage">
 <h3>Usage<a class="headerlink" href="#usage" title="Permalink to this heading">#</a></h3>
-<p>Run the entire pipeline with:</p>
+<p>Run the full pipeline:</p>
 <div class="highlight-bash notranslate"><div class="highlight"><pre><span></span>bash<span class="w"> </span>run_interactive.sh
 </pre></div>
 </div>
-<p>Alternatively, export the three variables as environment variables:</p>
-<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span><span class="nb">export</span><span class="w"> </span><span class="nv">NUM_GPUS_DIT</span><span class="o">=</span><span class="m">1</span>
-<span class="nb">export</span><span class="w"> </span><span class="nv">NUM_GPUS_VAE</span><span class="o">=</span><span class="m">3</span>
+<p>Or override via environment:</p>
+<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span><span class="nb">export</span><span class="w"> </span><span class="nv">NUM_GPUS_DIT</span><span class="o">=</span><span class="m">2</span>
+<span class="nb">export</span><span class="w"> </span><span class="nv">NUM_GPUS_VAE</span><span class="o">=</span><span class="m">6</span>
 <span class="nb">export</span><span class="w"> </span><span class="nv">MODEL_PATH</span><span class="o">=</span><span class="s2">&quot;../models/stage4&quot;</span>
 bash<span class="w"> </span>run_interactive.sh
 </pre></div>
@@ -507,20 +555,20 @@ <h3>Sub-script: start_dit.sh<a class="headerlink" href="#sub-script-start-dit-sh
 </div>
 <dl class="field-list simple">
 <dt class="field-odd">NUM_GPUS_DIT<span class="colon">:</span></dt>
-<dd class="field-odd"><p>Number of GPUs to allocate for the DIT process.</p>
+<dd class="field-odd"><p>Number of GPUs allocated to DiT.</p>
 </dd>
 <dt class="field-even">MODEL_PATH<span class="colon">:</span></dt>
-<dd class="field-even"><p>Path to the directory or prefix of stage4 model checkpoint files.</p>
+<dd class="field-even"><p>Directory or prefix of stage4 checkpoint files.</p>
 </dd>
 </dl>
 </section>
 <section id="environment-variables">
 <h3>Environment Variables<a class="headerlink" href="#environment-variables" title="Permalink to this heading">#</a></h3>
 <ul class="simple">
 <li><p><strong>CUDA_VISIBLE_DEVICES</strong>
-Computed by the script as a comma-separated list to assign GPUs.</p></li>
+List of GPU indices assigned to Ray head, DiT, VAE, etc.</p></li>
 <li><p><strong>PYTORCH_CUDA_ALLOC_CONF</strong>
-Set to <cite>expandable_segments:True</cite> to configure PyTorch allocator.</p></li>
+Set to <cite>expandable_segments:True</cite> to optimize CUDA allocator behavior.</p></li>
 </ul>
 </section>
 </section>
@@ -572,6 +620,10 @@ <h3>Environment Variables<a class="headerlink" href="#environment-variables" tit
     <ul class="visible nav section-nav flex-column">
 <li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#inference-with-the-matrix-py">1.Inference with the_matrix.py</a></li>
 <li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#inference-with-run-interactive-sh">2. Inference with run_interactive.sh</a><ul class="nav section-nav flex-column">
+<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#summary">Summary</a></li>
+<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#highlights">Highlights</a></li>
+<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#two-inference-modes">Two Inference Modes</a></li>
+<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#performance-comparison">Performance Comparison</a></li>
 <li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#configuration">Configuration</a></li>
 <li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#usage">Usage</a></li>
 <li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#sub-script-start-dit-sh">Sub-script: start_dit.sh</a></li>
@@ -584,7 +636,7 @@ <h3>Environment Variables<a class="headerlink" href="#environment-variables" tit
   <div class="sidebar-secondary-item">
 
   <div class="tocsection sourcelink">
-    <a href="source/inference.rst.txt">
+    <a href="_sources/inference.rst.txt">
       <i class="fa-solid fa-file-lines"></i> Show Source
     </a>
   </div>
@@ -646,4 +698,4 @@ <h3>Environment Variables<a class="headerlink" href="#environment-variables" tit
 
   </footer>
   </body>
-</html>
+</html>