[Misc] cxs: update param and docs

Signed-off-by: Sohei Koyama <skoyama@ddn.com>

Author: KoyamaSohei

real-multi-round-qa/README.md

@@ -2,70 +2,74 @@
 
 ## Overview
 
-This benchmark is designed to identify **the maximum harmonic mean of user sessions $(C,S)$ that can be kept active while maintaining a steady-state TTFT ≤ 2 s (95-th percentile)**. By sweeping the concurrency ($C$) and sequential ($S$) independently, it isolates whether compute capacity or KV-cache pressure is the first limiting factor.
-
+This benchmark is designed to explore how TTFT changes across different $(C, S)$ combinations by sweeping concurrency ($C$) and session depth ($S$) independently. This helps isolate whether compute capacity or KV-cache pressure is the primary limiting factor.
 
 We highly recommend monitoring vLLM/LMCache/GPU/storage metrics at the same time.
 
 This benchmark feeds full‑length novels to your LLM server and asks many follow‑up questions, just like a book critic. It is handy for testing long‑context handling and KV‑cache tools such as LMCache.
 
-The benchmark is called CxS (pronounced six for simplicity), referring to the product of Concurrent $\times$ Sequential users.
-
-### Definition
-
-Let us define the set of candidate pairs:
-
-$$
-\mathcal{D} = \{ (C_i, S_i) \mid \mathrm{TTFT}_{95}^{(i)} \leq 2 \}
-$$
-
-### Objective
-
-More precisely, we aim to find the pair that maximizes the harmonic mean among all candidates in $\mathcal{D}$:
-
-
-$$
-\underset{(C_i, S_i) \in \mathcal{D}}{\arg\max} \left( \frac{2 C_i S_i}{C_i + S_i} \right)
-$$
-
-We use the harmonic mean to compare scores.  
-As a business metric, we report the product, CxS.  
-For example, we say "Our system can keep up to {C×S} user sessions active!"
+The benchmark is called CxS (pronounced six for simplicity), referring to the product of Concurrent $\times$ Session Depth.
 
 ## Two simple knobs
 
 | Option | What it means |
 | ---- | ---- |
-| `--num-users-concurrent` (C) | How many threads run in parallel.  |
-| `--num-users-sequential` (S) | How many users each thread serves in turn. |
+| `--concurrent` (C) | How many threads run in parallel.  |
+| `--session-depth` (S) | How many sessions each thread serves in turn. |
 
 You can:
-* raise concurrent to test compute-side capability (higher GPU utilization; total KV footprint also rises).
-* raise sequential to test KV-cache pressure (larger resident KV per GPU, little change in instantaneous GPU utilization).
+* raise $C$ to test compute-side capability (higher GPU utilization; total KV footprint also rises).
+* raise $S$ to test KV-cache pressure (larger resident KV per GPU, little change in instantaneous GPU utilization).
 
 ## Execution model
 
 ```
-Concurrent USER: {A,B}
-Sequential USER: {X,Y}
-All USER: {AX,AY,BX,BY}
+Concurrent: {A,B}
+Session Depth: {X,Y}
+All Session: {AX,AY,BX,BY}
 
 Timeline
 -------------------------------------------------
 Thread A:
-  Turn 0 → UserAX: Q1 "Read and summarize this novel. {AX novel contents}" → Get Response
-  Turn 0 → UserAY: Q1 "Read and summarize this novel. {AY novel contents}" → Get Response
-  Turn 1 → UserAX: Q2 "Write down the author's feelings." → Get Response
-  Turn 1 → UserAY: Q2 "Write down the author's feelings." → Get Response
+  Turn 0 → SessionAX: Q1 "Read and summarize this novel. {AX novel contents}" → Get Response
+  Turn 0 → SessionAY: Q1 "Read and summarize this novel. {AY novel contents}" → Get Response
+  Turn 1 → SessionAX: Q2 "Write down the author's feelings." → Get Response
+  Turn 1 → SessionAY: Q2 "Write down the author's feelings." → Get Response
   ...
 Thread B:
-  Turn 0 → UserBX: Q1 "Read and summarize this novel. {BX novel contents}" → Get Response
-  Turn 0 → UserBY: Q1 "Read and summarize this novel. {BY novel contents}" → Get Response
-  Turn 1 → UserBX: Q2 "Write down the author's feelings." → Get Response
-  Turn 1 → UserBY: Q2 "Write down the author's feelings." → Get Response
+  Turn 0 → SessionBX: Q1 "Read and summarize this novel. {BX novel contents}" → Get Response
+  Turn 0 → SessionBY: Q1 "Read and summarize this novel. {BY novel contents}" → Get Response
+  Turn 1 → SessionBX: Q2 "Write down the author's feelings." → Get Response
+  Turn 1 → SessionBY: Q2 "Write down the author's feelings." → Get Response
   ...
 ```
 
+## For system competition
+
+The CxS benchmark provides a scalar score to encourage healthy competition, but its use is not mandatory.
+
+### Definition
+
+Let us define the set of candidate pairs:
+
+$$
+\mathcal{D} = {\{ (C_i, S_i) \mid \mathrm{TTFT}_{95}^{(i)} \leq 2 \}}
+$$
+
+### Objective
+
+More precisely, we aim to find the pair that maximizes the harmonic mean among all candidates in $\mathcal{D}$:
+
+
+$$
+\underset{(C_i, S_i) \in \mathcal{D}}{\arg\max} \left( \frac{2 C_i S_i}{C_i + S_i} \right)
+$$
+
+## For business metric
+
+As a business metric, we report the product, CxS.  
+For example, we say "Our system can keep up to {C×S} user sessions active!"
+
 ## Getting Started
 
 ```bash
@@ -86,52 +90,52 @@ for c in {1..4}; do # You can change c and s to any value you like.
   for s in {1..4}; do
     TIMESTAMP=$(date +%s)
     OUTPUT_FILE="${OUTPUT_DIR}/bench_c${c}_s${s}_${TIMESTAMP}.json"
-    echo "Running benchmark: concurrent=${c}, sequential=${s}"
-    python multi-round-qa.py --num-users-concurrent "$c" --num-users-sequential "$s" --num-rounds "$NUM_ROUNDS" --model "$MODEL" --base-url "$BASE_URL" --output "$OUTPUT_FILE" --src-dir "$SRC_DIR"
+    echo "Running benchmark: C=${c}, S=${s}"
+    python multi-round-qa.py -c "$c" -s "$s" --num-rounds "$NUM_ROUNDS" --model "$MODEL" --base-url "$BASE_URL" --output "$OUTPUT_FILE" --src-dir "$SRC_DIR"
   done
 done
 ```
 
 ```bash
 # Plot and Show Result
 $ python plot.py ./bench_dir_vllm vllm.png
-    num_users_concurrent  num_users_sequential    ttft_95
-0                      4                     2   0.498404
-1                      4                     4  33.565437
-2                      4                     3   0.794144
-3                      1                     4   0.311046
-4                      2                     2   0.406148
-5                      2                     4   0.459704
-6                      2                     3   0.326396
-7                      1                     2   0.411317
-8                      3                     3   0.378674
-9                      2                     1   0.445499
-10                     3                     4  42.531053
-11                     1                     3   0.455651
-12                     4                     1   0.504505
-13                     3                     2   0.393902
-14                     3                     1   0.364927
-15                     1                     1   0.379049
+    c  s    ttft_95
+0   4  2   0.498404
+1   4  4  33.565437
+2   4  3   0.794144
+3   1  4   0.311046
+4   2  2   0.406148
+5   2  4   0.459704
+6   2  3   0.326396
+7   1  2   0.411317
+8   3  3   0.378674
+9   2  1   0.445499
+10  3  4  42.531053
+11  1  3   0.455651
+12  4  1   0.504505
+13  3  2   0.393902
+14  3  1   0.364927
+15  1  1   0.379049
 Max harmonic mean (C,S) where TTFT_95 <= 2s: 3.43
   => C=4.0, S=3.0, CxS=12.0
 $ python plot.py ./bench_dir_lmcache lmcache.png
-    num_users_concurrent  num_users_sequential   ttft_95
-0                      1                     1  0.524989
-1                      3                     2  0.592148
-2                      4                     4  1.202544
-3                      3                     4  1.286755
-4                      2                     1  0.477370
-5                      3                     3  0.586793
-6                      2                     3  0.627655
-7                      4                     1  0.575724
-8                      4                     3  1.251918
-9                      2                     4  0.446477
-10                     1                     4  0.460711
-11                     3                     1  0.495073
-12                     1                     3  0.329389
-13                     4                     2  0.586223
-14                     1                     2  0.477946
-15                     2                     2  0.457463
+    c  s   ttft_95
+0   1  1  0.524989
+1   3  2  0.592148
+2   4  4  1.202544
+3   3  4  1.286755
+4   2  1  0.477370
+5   3  3  0.586793
+6   2  3  0.627655
+7   4  1  0.575724
+8   4  3  1.251918
+9   2  4  0.446477
+10  1  4  0.460711
+11  3  1  0.495073
+12  1  3  0.329389
+13  4  2  0.586223
+14  1  2  0.477946
+15  2  2  0.457463
 Max harmonic mean (C,S) where TTFT_95 <= 2s: 4.00
   => C=4.0, S=4.0, CxS=16.0
 ```

real-multi-round-qa/lmcache.png

@@ -151,7 +151,7 @@ async def run_turn(session: ChatSession, client: openai.AsyncOpenAI) -> Result:
 
 async def run_group(args) -> List[Result]:
     client = openai.AsyncOpenAI(base_url=args.base_url, api_key="EMPTY")
-    sessions = [ChatSession(args) for _ in range(args.num_users_sequential)]
+    sessions = [ChatSession(args) for _ in range(args.session_depth)]
     results = []
 
     while any(not s.is_finished() for s in sessions):
@@ -164,14 +164,14 @@ async def run_group(args) -> List[Result]:
     return results
 
 async def run_all_concurrent(args):
-    tasks = [run_group(args) for _ in range(args.num_users_concurrent)]
+    tasks = [run_group(args) for _ in range(args.concurrent)]
     all_results = await asyncio.gather(*tasks)
     return [asdict(r) for group in all_results for r in group]
 
 def parse_args():
     parser = argparse.ArgumentParser()
-    parser.add_argument("--num-users-concurrent", type=int, required=True)
-    parser.add_argument("--num-users-sequential", type=int, required=True)
+    parser.add_argument("-c", "--concurrent", type=int, required=True)
+    parser.add_argument("-s", "--session-depth", type=int, required=True)
     parser.add_argument("--model", type=str, required=True)
     parser.add_argument("--base-url", type=str, required=True)
     parser.add_argument("--num-rounds", type=int, default=10)

real-multi-round-qa/multi-round-qa.py

@@ -5,11 +5,12 @@
 import pandas as pd
 import numpy as np
 import matplotlib.pyplot as plt
+from matplotlib.colors import LogNorm
 
 def main():
     parser = argparse.ArgumentParser(description="Analyze number of user sessions from benchmark results.")
     parser.add_argument("input_dir", help="Directory containing JSON files")
-    parser.add_argument("output", help="Output path for the 3D bar plot image")
+    parser.add_argument("output", help="Output path for the contour plot image")
     args = parser.parse_args()
 
     json_files = glob.glob(f"{args.input_dir}/*.json")
@@ -26,7 +27,7 @@ def main():
             results = data["results"]
 
             params_fixed = {k: v for k, v in params.items()
-                            if k not in ["num_users_concurrent", "num_users_sequential", "output"]}
+                            if k not in ["concurrent", "session_depth", "output"]}
             all_params.append(params_fixed)
 
             df = pd.DataFrame(results)
@@ -36,8 +37,8 @@ def main():
 
             ttft_95 = df["ttft"].quantile(0.95)
             summary_records.append({
-                "num_users_concurrent": params["num_users_concurrent"],
-                "num_users_sequential": params["num_users_sequential"],
+                "c": params["concurrent"],
+                "s": params["session_depth"],
                 "ttft_95": ttft_95
             })
 
@@ -52,38 +53,46 @@ def main():
         print("No valid TTFT data to visualize.")
         return
 
-    summary_df_sorted = summary_df.sort_values(by=['num_users_concurrent', 'num_users_sequential'])
+    C_vals = summary_df["c"].values
+    S_vals = summary_df["s"].values
+    TTFT_95_vals = summary_df["ttft_95"].values
 
-    fig = plt.figure()
-    ax = fig.add_subplot(111, projection='3d')
+    plt.figure(figsize=(9, 7))
+    contour = plt.tricontourf(
+        C_vals, S_vals, TTFT_95_vals,
+        levels=np.logspace(np.log10(0.01), np.log10(100), 30),
+        cmap='plasma',
+        norm=LogNorm(vmin=0.01, vmax=100)
+    )
 
-    x = summary_df_sorted['num_users_concurrent']
-    y = summary_df_sorted['num_users_sequential']
-    z = np.zeros_like(x)
-    dz = summary_df_sorted['ttft_95']
+    cbar = plt.colorbar(contour)
+    ticks = [0.01, 0.1, 1, 2, 4, 8, 16, 32, 64, 100]
+    cbar.set_ticks(ticks)
+    cbar.set_ticklabels([str(t) for t in ticks])
+    cbar.set_label('TTFT_95 (s)', fontsize=12)
 
-    colors = ['blue' if v <= 2 else 'red' for v in dz]
+    plt.tricontour(C_vals, S_vals, TTFT_95_vals, levels=[2.0], colors='white', linewidths=2, linestyles='dashed')
 
-    ax.bar3d(x, y, z, dx=0.5, dy=0.5, dz=dz, color=colors, shade=True)
-    ax.set_xlabel('Concurrent Users (C)')
-    ax.set_ylabel('Sequential Users (S)')
-    ax.set_zlabel('95% Tail TTFT (s)')
-    plt.title('TTFT 95% Tail vs Concurrent/Sequential Users')
-    ax.invert_xaxis()
-    plt.savefig(args.output)
+    plt.xlabel('Concurrent (C)', fontsize=12)
+    plt.ylabel('Session Depth(S)', fontsize=12)
+    plt.title('TTFT_95 Contour across (C, S)', fontsize=14)
+    plt.grid(True, which='both', linestyle='--', alpha=0.3)
 
-    # Max harmonic mean under 2s TTFT
     summary_under_2s = summary_df[summary_df["ttft_95"] <= 2].copy()
     if not summary_under_2s.empty:
-        summary_under_2s["harmonic_mean"] = 2 * summary_under_2s["num_users_concurrent"] * summary_under_2s["num_users_sequential"] / (
-            summary_under_2s["num_users_concurrent"] + summary_under_2s["num_users_sequential"]
+        summary_under_2s["harmonic_mean"] = 2 * summary_under_2s["c"] * summary_under_2s["s"] / (
+            summary_under_2s["c"] + summary_under_2s["s"]
         )
         best_row = summary_under_2s.sort_values("harmonic_mean", ascending=False).iloc[0]
-        product = best_row["num_users_concurrent"] * best_row["num_users_sequential"]
+        product = best_row["c"] * best_row["s"]
         print(f"Max harmonic mean (C,S) where TTFT_95 <= 2s: {best_row['harmonic_mean']:.2f}")
-        print(f"  => C={best_row['num_users_concurrent']}, S={best_row['num_users_sequential']}, CxS={product}")
+        print(f"  => C={best_row['c']}, S={best_row['s']}, CxS={product}")
+        plt.scatter([best_row['c']], [best_row['s']], c='cyan', edgecolors='black', s=120, marker='*', label='Best (C,S)')
+        plt.legend()
+        plt.autoscale()
     else:
         print("No data points with TTFT_95 <= 2s.")
+    plt.savefig(args.output)
 
 if __name__ == "__main__":
     main()