[LEADS-348] evaluation latency

[xmican10]

Description

Depends on #230
Further changes for LEADS-348:

• result.execution_time was renamed to result.evaluation_latency which represents better that it is measuring the evaluation
• rounding of result.evaluation_latency was removed for consistency, since other timing metrics are not being rounded
• result.execution_time was reintroduced (for backward compatibility), but currently measures the whole api execution and evaluation per turn
• test_evaluation.py was refactored - redundant mocking was moved to pytest fixture (~500 lines reduced)

Type of change

• Refactor
• New feature
• Bug fix
• CVE fix
• Optimization
• Documentation Update
• Configuration Update
• Bump-up service version
• Bump-up dependent library
• Bump-up library or tool used for development (does not change the final image)
• CI configuration change
• Unit tests improvement

Tools used to create PR

Identify any AI code assistants used in this PR (for transparency and review context)

• Assisted-by: Claude (e.g., Claude, CodeRabbit, Ollama, etc., N/A if not used)
• Generated by: (e.g., tool name and version; N/A if not used)

Related Tickets & Documents

• Related Issue #
• Closes #

Checklist before requesting a review

• I have performed a self-review of my code.
• PR has passed all pre-merge test jobs.
• If it is a core feature, I have added thorough tests.

Testing

• Please provide detailed steps to perform tests related to this code change.
• How were the fix/results from this change verified? Please provide relevant screenshots or results.

Summary by CodeRabbit

Release Notes

• New Features

  • Added evaluation latency metric tracking, now recorded separately from agent latency for improved performance visibility in CSV exports and system outputs.

• Tests

  • Enhanced test coverage and fixtures to validate evaluation latency measurement and result accuracy across all output formats.

[coderabbitai]

Warning

Rate limit exceeded

@xmican10 has exceeded the limit for the number of commits that can be reviewed per hour. Please wait 30 minutes and 47 seconds before requesting another review.

You’ve run out of usage credits. Purchase more in the billing tab.

⌛ How to resolve this issue?

After the wait time has elapsed, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

We recommend that you space out your commits to avoid hitting the rate limit.

🚦 How do rate limits work?

CodeRabbit enforces hourly rate limits for each developer per organization.

Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout.

Please see our FAQ for further information.

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: fef976af-2b02-48df-ac77-69e54946a53d

📥 Commits

Reviewing files that changed from the base of the PR and between e9c4b17 and 457af2f.

📒 Files selected for processing (15)

• config/system.yaml
• src/lightspeed_evaluation/core/constants.py
• src/lightspeed_evaluation/core/models/data.py
• src/lightspeed_evaluation/core/output/generator.py
• src/lightspeed_evaluation/core/storage/sql_storage.py
• src/lightspeed_evaluation/pipeline/evaluation/evaluator.py
• tests/script/conftest.py
• tests/unit/core/models/test_data.py
• tests/unit/core/models/test_summary.py
• tests/unit/core/output/test_generator.py
• tests/unit/core/storage/test_sql_storage.py
• tests/unit/core/system/test_loader.py
• tests/unit/pipeline/evaluation/conftest.py
• tests/unit/pipeline/evaluation/test_errors.py
• tests/unit/pipeline/evaluation/test_evaluator.py

Walkthrough

This PR refactors evaluation result timing from a single execution_time field to a decomposed model where evaluation_latency and agent_latency are measured separately and execution_time equals their sum. The change spans data model, evaluator logic, storage, output, and comprehensive test updates.

Changes

Timing Model Refactoring

Layer / File(s)	Summary
Schema and configuration updates config/system.yaml, src/lightspeed_evaluation/core/constants.py, src/lightspeed_evaluation/core/models/data.py	Configuration and data model fields updated to support evaluation_latency as a CSV column with float defaults and clarified descriptions for turn-level timing fields.
Evaluator timing measurement and computation src/lightspeed_evaluation/pipeline/evaluation/evaluator.py	New _measure_latency() helper computes elapsed time via perf_counter(). evaluate_metric() captures start_time, computes evaluation_latency, sets execution_time = evaluation_latency + agent_latency. Error paths refactored to pass start_time to _create_error_result() instead of precomputed elapsed time.
Storage and output serialization src/lightspeed_evaluation/core/storage/sql_storage.py, src/lightspeed_evaluation/core/output/generator.py	Database schema adds nullable evaluation_latency column with ORM mapping updated accordingly. JSON and CSV output generation emit evaluation_latency in place of execution_time.
Test infrastructure and fixtures tests/unit/pipeline/evaluation/conftest.py, tests/unit/core/system/test_loader.py, tests/script/conftest.py	New evaluator pytest fixture provides pre-mocked evaluator with wired dependencies. System config loader test validates default YAML loads without errors. Script test fixtures updated to use evaluation_latency.
Data model and storage validation tests tests/unit/core/models/test_data.py, tests/unit/core/models/test_summary.py, tests/unit/core/output/test_generator.py, tests/unit/core/storage/test_sql_storage.py	Default value assertions and validation tests switched from execution_time to evaluation_latency. Test fixtures and storage/output tests updated to assert correct evaluation_latency persistence and serialization.
Error handling and evaluator behavior tests tests/unit/pipeline/evaluation/test_evaluator.py, tests/unit/pipeline/evaluation/test_errors.py	Comprehensive test refactor: removed repeated per-test mocking by using injected evaluator fixture. Tests now validate execution_time = evaluation_latency + agent_latency computation, error result field preservation (query, response, contexts), expected-response evaluation paths, and token accumulation correctness including no-double-counting across multi-path and error scenarios.

🎯 3 (Moderate) | ⏱️ ~25 minutes

Suggested reviewers

• asamal4
• VladimirKadlec

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title '[LEADS-348] evaluation latency' directly describes the main change: introducing evaluation_latency as a new timing metric throughout the codebase.
Docstring Coverage	✅ Passed	Docstring coverage is 96.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

Actionable comments posted: 4

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@src/lightspeed_evaluation/core/output/generator.py`:
- Line 817: The JSON summary currently adds "evaluation_latency":
r.evaluation_latency but removed the older "execution_time" field; restore
compatibility by including "execution_time" in the emitted dict (e.g., set
"execution_time": r.execution_time or, if r has no execution_time, set it to the
same value as r.evaluation_latency) alongside "evaluation_latency" in the output
constructed where the dict containing "evaluation_latency" is created so older
consumers still receive execution_time.

In `@src/lightspeed_evaluation/core/storage/sql_storage.py`:
- Line 60: The SQL schema and persistence code currently define/persist only
evaluation_latency, dropping the previously stored execution_time; add back
execution_time = Column(Float, nullable=True) alongside evaluation_latency in
the model definition and update the persistence logic that currently writes
evaluation_latency (e.g., in the function that inserts/updates evaluation
records—look for code referencing evaluation_latency around the save/insert
method) to also set execution_time from the same source value so existing
consumers continue receiving execution_time.

In `@tests/unit/pipeline/evaluation/test_errors.py`:
- Line 160: The test currently asserts results[0].evaluation_latency but misses
verifying the backward-compatible execution_time field on error results; update
the test in tests/unit/pipeline/evaluation/test_errors.py to also assert that
results[0].execution_time equals 0.0 alongside the existing evaluation_latency
assertion so error results explicitly include the legacy execution_time field.

In `@tests/unit/pipeline/evaluation/test_evaluator.py`:
- Around line 583-605: The test
test_execution_time_conversation_level_averages_agent_latency incorrectly
asserts result.agent_latency == 4.0; update the assertion to expect the average
agent latency of the conversation (2.0) since TurnData instances have
agent_latency 1.0 and 3.0 and EvaluationRequest.for_conversation should compute
the mean; change the assertion referencing result.agent_latency in this test to
assert result.agent_latency == 2.0 and keep the other
execution_time/evaluation_latency checks unchanged.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 56132854-62fe-4236-8dc5-3380d17e64a8

📥 Commits

Reviewing files that changed from the base of the PR and between 377a66b and e9c4b17.

📒 Files selected for processing (15)

• config/system.yaml
• src/lightspeed_evaluation/core/constants.py
• src/lightspeed_evaluation/core/models/data.py
• src/lightspeed_evaluation/core/output/generator.py
• src/lightspeed_evaluation/core/storage/sql_storage.py
• src/lightspeed_evaluation/pipeline/evaluation/evaluator.py
• tests/script/conftest.py
• tests/unit/core/models/test_data.py
• tests/unit/core/models/test_summary.py
• tests/unit/core/output/test_generator.py
• tests/unit/core/storage/test_sql_storage.py
• tests/unit/core/system/test_loader.py
• tests/unit/pipeline/evaluation/conftest.py
• tests/unit/pipeline/evaluation/test_errors.py
• tests/unit/pipeline/evaluation/test_evaluator.py