TEST_SCENARIOS.md

Position Accuracy Evaluation - Test Scenarios

This document provides recommended test scenarios to evaluate the impact of different network conditions and features on movement accuracy.

Test Methodology

Each test should run for at least 1-2 minutes to collect meaningful statistics (~6-12 metric logs at 10-second intervals).

The position error metric measures distance in game units between:

• Ideal position: Where player would be with instant local input
• Actual position: Where player is with networked updates

Baseline Tests (No Simulation)

Test 1: Default Settings

python main.py

Expected result: Very low errors (avg < 0.1)

• Shows effectiveness of prediction+reconciliation+interpolation
• Baseline for all other comparisons

Test 2: Prediction Only

python main.py --no-reconciliation --no-self-interpolation --no-other-interpolation

Expected result: Low errors with smooth movement

• Demonstrates prediction effectiveness
• Shows why reconciliation is important

Test 3: No Prediction (Server Authoritative)

python main.py --no-prediction --no-reconciliation

Expected result: Errors increase significantly

• Shows impact of prediction on local feedback
• Most conservative approach

Test 4: Reconciliation Only

python main.py --no-prediction --no-self-interpolation --no-other-interpolation

Expected result: Delayed but accurate movement

• Server corrections applied
• No client prediction

---

Network Latency Tests

Test 5: Mild Latency (50ms)

python main.py --input-delay 50

Expected result: avg error ~0.2-0.4

• Noticeable but playable
• Still relatively smooth with interpolation

Test 6: Moderate Latency (100ms)

python main.py --input-delay 100

Expected result: avg error ~0.4-0.8

• More noticeable drift
• Interpolation becomes important
• Movement feels slightly sluggish

Test 7: High Latency (200ms)

python main.py --input-delay 200

Expected result: avg error ~0.8-1.5

• Significant position drift
• Prediction helps but not perfect
• Noticeable gameplay impact

Test 8: Very High Latency (500ms)

python main.py --input-delay 500

Expected result: avg error ~2.0-4.0+

• Extreme drift visible in position
• Movement feels very delayed
• This is typical of intercontinental play without optimization

Test 9: Latency - Prediction Disabled

python main.py --input-delay 100 --no-prediction

Expected result: avg error increases by ~2-3x compared to test 6

• Shows prediction is crucial for latency mitigation
• Movement appears delayed/laggy

Test 10: Latency - Reconciliation Disabled

python main.py --input-delay 100 --no-reconciliation

Expected result: Similar to test 6

• Reconciliation has less immediate impact than prediction
• May see position "snaps" less frequently

---

Packet Loss Tests

Test 11: Light Packet Loss (5%)

python main.py --packet-loss 5

Expected result: avg error ~0.1-0.3

• Minimal impact with redundancy
• Occasional small jumps
• Mostly imperceptible

Test 12: Moderate Packet Loss (10%)

python main.py --packet-loss 10

Expected result: avg error ~0.3-0.6

• More frequent small corrections
• Visible but not game-breaking

Test 13: Heavy Packet Loss (20%)

python main.py --packet-loss 20

Expected result: avg error ~0.8-1.5+

• Frequent corrections visible
• Movement becomes unreliable
• Significant gameplay impact

Test 14: Packet Loss - Prediction Disabled

python main.py --packet-loss 10 --no-prediction

Expected result: avg error increases significantly

• Every lost packet causes visible jump
• Movement very jerky

---

Combined Network Conditions

Test 15: Realistic Mobile (100ms delay + 5% loss)

python main.py --input-delay 100 --packet-loss 5

Expected result: avg error ~0.4-0.8

• Typical mobile network scenario
• Prediction+reconciliation both important

Test 16: Poor WiFi (200ms delay + 10% loss)

python main.py --input-delay 200 --packet-loss 10

Expected result: avg error ~1.2-2.0

• Poor WiFi conditions
• Notable gameplay impact
• Shows need for optimization

Test 17: Intercontinental (300ms delay + 3% loss)

python main.py --input-delay 300 --packet-loss 3

Expected result: avg error ~1.5-2.5

• Intercontinental play
• High latency dominant factor
• Shows why regional servers help

Test 18: Worst Case (500ms + 10% loss)

python main.py --input-delay 500 --packet-loss 10

Expected result: avg error 3.0+

• Unplayable scenario
• Shows limits of client prediction
• Would require different architecture

---

Feature Isolation Tests

Test 19: Interpolation Impact (with 100ms delay)

Compare:

# With interpolation
python main.py --input-delay 100

# Without interpolation
python main.py --input-delay 100 --no-self-interpolation

Observation: Interpolation makes movement smoother, reduces perception of discrete ticks

Test 20: Prediction Impact (with 100ms delay)

Compare:

# With prediction
python main.py --input-delay 100

# Without prediction
python main.py --input-delay 100 --no-prediction

Observation: Prediction eliminates perceptible latency in movement

Test 21: Reconciliation Impact (with 100ms delay)

Compare:

# With reconciliation
python main.py --input-delay 100

# Without reconciliation
python main.py --input-delay 100 --no-reconciliation

Observation: Reconciliation handles position corrections from server

---

Analysis Guide

Error Interpretation

Avg Error	P95	Playability	Notes
< 0.1	< 0.2	Excellent	Imperceptible latency
0.1-0.3	0.3-0.6	Very Good	Smooth, responsive
0.3-0.5	0.6-1.0	Good	Noticeable but playable
0.5-1.0	1.0-1.5	Fair	Noticeable drift
1.0-2.0	1.5-3.0	Poor	Significant drift
> 2.0	> 3.0	Unplayable	Major issues

What to Look For

1. Average Error: Overall accuracy

   • Lower is better
   • Increases with latency
   • Reduced by prediction/reconciliation

2. Standard Deviation: Consistency

   • Lower = more consistent behavior
   • Packet loss causes spikes (higher stddev)
   • Good interpolation reduces stddev

3. P95/P99: Worst-case accuracy

   • Shows impact of network variability
   • High values = occasional large jumps
   • Important for consistency perception

4. Max Error: Peak drift

   • Shows worst frame
   • Can indicate lost packet effects
   • Prediction prevents extreme peaks

---

Comparison Template

Use this table to track results across different conditions:

Condition                          | Avg Error | StdDev | P95 | Max | Notes
-----------------------------------|-----------|--------|-----|-----|-------
No simulation (baseline)           |           |        |     |     |
Pred only                         |           |        |     |     |
No prediction                     |           |        |     |     |
100ms delay                       |           |        |     |     |
100ms delay, no prediction        |           |        |     |     |
200ms delay                       |           |        |     |     |
5% packet loss                    |           |        |     |     |
10% packet loss                   |           |        |     |     |
100ms delay + 5% loss             |           |        |     |     |

---

Performance Expectations

Client CPU/Memory Impact

• Minimal: ~0.1% CPU for position tracking
• Memory: ~800 bytes per logging interval (cleared after log)
• No FPS impact: Background thread handles networking

Server Impact

• No server changes: Metrics collected only on client
• No bandwidth increase: Just stores position error locally

---

Experimental Variations

Test Protocol Effects

1. Compare UDP vs WebSocket mode:

   # UDP (default)
   python main.py --input-delay 100
   
   # WebSocket
   python main.py --protocol websocket --input-delay 100

2. Compare with other players nearby:

   • Run multiple clients
   • See if network congestion affects accuracy
   • More players = more updates = more network traffic

3. Test with different server locations:

   # Local server (fast)
   python main.py --server localhost
   
   # Remote server (slow)
   python main.py --server distant-server.com

---

Recording Results

For each test scenario:

1. Note the configuration:

   • Feature flags enabled/disabled
   • Network simulation parameters
   • Protocol (UDP/WebSocket)

2. Observe the metrics output:

   • Average error
   • All percentiles
   • Standard deviation

3. Subjective observation:

   • Does movement feel responsive?
   • Visible stutters or jumps?
   • Can you play effectively?

4. Document patterns:

   • Which features have most impact?
   • Where is the playability threshold?
   • What's your "good enough" error level?

---

Next Steps

After gathering baseline metrics:

1. Identify critical thresholds: At what error does gameplay suffer?
2. Test optimization ideas: Experiment with different parameters
3. Compare with remote play: See real network impact
4. Export for analysis: Consider CSV export for detailed comparison

See IDEAL_POSITION_TRACKER.md for data analysis tools and future enhancements.