Followup task for LEADS-246 - Renaming api cache to agent by xmican10 · Pull Request #241 · lightspeed-core/lightspeed-evaluation

xmican10 · 2026-05-20T11:51:18Z

Description

Followup task for LEADS-246 - cache default folder renaming from api to agent
E2E test fix

Type of change

Tools used to create PR

Identify any AI code assistants used in this PR (for transparency and review context)

Assisted-by: (e.g., Claude, CodeRabbit, Ollama, etc., N/A if not used)
Generated by: (e.g., tool name and version; N/A if not used)

Related Tickets & Documents

Related Issue #
Closes #

Checklist before requesting a review

I have performed a self-review of my code.
PR has passed all pre-merge test jobs.
If it is a core feature, I have added thorough tests.

Testing

Please provide detailed steps to perform tests related to this code change.
How were the fix/results from this change verified? Please provide relevant screenshots or results.

Summary by CodeRabbit

Chores
- Reorganized cached evaluation artifacts: cache subfolders now use "agent" and "llm" namespaces instead of the legacy "api" location.
Documentation
- Added global caching controls and deprecated component-level cache options; updated examples and tables to reflect new core cache settings.
Tests
- Adjusted test expectations to align with the new cache directory layout.

coderabbitai · 2026-05-20T11:51:53Z

Warning

Rate limit exceeded

@xmican10 has exceeded the limit for the number of commits that can be reviewed per hour. Please wait 47 minutes and 18 seconds before requesting another review.

You’ve run out of usage credits. Purchase more in the billing tab.

⌛ How to resolve this issue?

After the wait time has elapsed, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

We recommend that you space out your commits to avoid hitting the rate limit.

🚦 How do rate limits work?

CodeRabbit enforces hourly rate limits for each developer per organization.

Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout.

Please see our FAQ for further information.

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 6bf03f8d-cb53-4b9b-9d48-fa7f6af3e41d

📥 Commits

Reviewing files that changed from the base of the PR and between fa0e879 and b4195d9.

📒 Files selected for processing (8)

config/system.yaml
docs/configuration.md
src/lightspeed_evaluation/core/constants.py
src/lightspeed_evaluation/core/models/llm.py
src/lightspeed_evaluation/core/models/system.py
tests/integration/system-config-query.yaml
tests/integration/system-config-streaming.yaml
tests/unit/core/models/test_system.py

Walkthrough

Renames DEFAULT_API_CACHE_SUBDIR to DEFAULT_AGENT_CACHE_SUBDIR and updates constants, SystemConfig global cache wiring, unit tests, documentation, and embedding config references to use the new agent subdirectory and core cache settings.

Changes

Cache Subdirectory Rename from API to Agent

Layer / File(s)	Summary
Constants and SystemConfig integration `src/lightspeed_evaluation/core/constants.py`, `src/lightspeed_evaluation/core/models/system.py`	Introduce `DEFAULT_AGENT_CACHE_SUBDIR` and use it in `SystemConfig.global_cache_setup` when computing the legacy `api_cache_path`.
Unit test updates `tests/unit/core/models/test_system.py`	Update expected `config.api.cache_dir` paths across legacy LLM default, override, judge panel, and cache-off scenarios to use the `agent` subdirectory.
Docs and top-level config comment `docs/configuration.md`, `config/system.yaml`	Add `core.cache_enabled`/`core.cache_base_dir`, mark component `cache_dir`/`cache_enabled` as deprecated, and update `core.cache_base_dir` inline comment to show `/llm` and `/agent` layout.
Embedding config field and integration examples `src/lightspeed_evaluation/core/models/llm.py`, `tests/integration/system-config-query.yaml`, `tests/integration/system-config-streaming.yaml`	Add deprecated optional `EmbeddingConfig.cache_dir` and include `embedding.cache_dir` values in integration YAML examples.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

Possibly related PRs

lightspeed-core/lightspeed-evaluation#221: Overlaps on cache-subdirectory constant and SystemConfig cache-path logic.

Suggested reviewers

asamal4

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title accurately reflects the main objective: renaming the api cache subdirectory to agent across the codebase, affecting constants, configuration models, tests, and documentation.
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

coderabbitai

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@config/system.yaml`:
- Line 9: The inline comment for the cache_base_dir setting is inaccurate: it
refers to "/agents" but the runtime uses ".caches/agent" (singular). Update the
comment next to the cache_base_dir YAML key to reflect the implemented subpaths
(e.g., ".caches/llm and .caches/agent") so the comment matches the actual
runtime directories used by the code that reads cache_base_dir.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

Push a commit to this branch (recommended)
Create a new PR with the fixes

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: f2b707ab-8169-4fbb-a73d-86077ec46a29

📥 Commits

Reviewing files that changed from the base of the PR and between 8bc9d6c and 8a617a3.

📒 Files selected for processing (4)

config/system.yaml
src/lightspeed_evaluation/core/constants.py
src/lightspeed_evaluation/core/models/system.py
tests/unit/core/models/test_system.py

asamal4

Thanks.. LGTM !!

coderabbitai

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)

src/lightspeed_evaluation/core/models/system.py (1)
359-376: ⚠️ Potential issue | 🟡 Minor | ⚡ Quick win

Include embedding.cache_dir in deprecation warning check.

The deprecation warning logic checks api.cache_dir (line 359) and llm_pool.defaults.cache_dir (line 363) but only checks embedding.cache_enabled at line 360. Now that EmbeddingConfig.cache_dir is being added in llm.py, this check should also verify self.embedding.cache_dir for consistency.
🔧 Suggested fix
-        has_embedding_cache_config = self.embedding.cache_enabled is False
+        has_embedding_cache_config = (
+            self.embedding.cache_enabled is False or self.embedding.cache_dir
+        )
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/lightspeed_evaluation/core/models/system.py` around lines 359 - 376, The
deprecation-check misses embedding.cache_dir: update the embedding check
(variable has_embedding_cache_config) to mirror the others by considering either
self.embedding.cache_enabled is False or self.embedding.cache_dir is set; i.e.,
change the condition that currently only tests embedding.cache_enabled to also
check self.embedding.cache_dir so the deprecation warning triggers when an
embedding cache directory is provided (refer to self.embedding.cache_enabled and
self.embedding.cache_dir in your change).

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Outside diff comments:
In `@src/lightspeed_evaluation/core/models/system.py`:
- Around line 359-376: The deprecation-check misses embedding.cache_dir: update
the embedding check (variable has_embedding_cache_config) to mirror the others
by considering either self.embedding.cache_enabled is False or
self.embedding.cache_dir is set; i.e., change the condition that currently only
tests embedding.cache_enabled to also check self.embedding.cache_dir so the
deprecation warning triggers when an embedding cache directory is provided
(refer to self.embedding.cache_enabled and self.embedding.cache_dir in your
change).

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: f4eba70e-1a4f-45ae-96f4-a7573d4c4fd1

📥 Commits

Reviewing files that changed from the base of the PR and between 8a617a3 and fa0e879.

📒 Files selected for processing (8)

config/system.yaml
docs/configuration.md
src/lightspeed_evaluation/core/constants.py
src/lightspeed_evaluation/core/models/llm.py
src/lightspeed_evaluation/core/models/system.py
tests/integration/system-config-query.yaml
tests/integration/system-config-streaming.yaml
tests/unit/core/models/test_system.py

✅ Files skipped from review due to trivial changes (3)

tests/integration/system-config-streaming.yaml
tests/integration/system-config-query.yaml
config/system.yaml

asamal4

Thanks. LGTM

coderabbitai Bot reviewed May 20, 2026

View reviewed changes

Comment thread config/system.yaml Outdated

xmican10 force-pushed the rename-api-cache-to-agents branch 3 times, most recently from e4fc103 to 305d811 Compare May 20, 2026 12:23

asamal4 previously approved these changes May 20, 2026

View reviewed changes

Comment thread tests/integration/system-config-agents-query.yaml

xmican10 dismissed asamal4’s stale review via fa0e879 May 20, 2026 14:50

xmican10 force-pushed the rename-api-cache-to-agents branch from 305d811 to fa0e879 Compare May 20, 2026 14:50

coderabbitai Bot reviewed May 20, 2026

View reviewed changes

Renaming api cache to agents

b4195d9

xmican10 force-pushed the rename-api-cache-to-agents branch from fa0e879 to b4195d9 Compare May 20, 2026 15:02

asamal4 approved these changes May 20, 2026

View reviewed changes

asamal4 merged commit 53f91bc into lightspeed-core:main May 20, 2026
16 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Followup task for LEADS-246 - Renaming api cache to agent#241

Followup task for LEADS-246 - Renaming api cache to agent#241
asamal4 merged 1 commit into
lightspeed-core:mainfrom
xmican10:rename-api-cache-to-agents

xmican10 commented May 20, 2026 •

edited by coderabbitai Bot

Loading

Uh oh!

coderabbitai Bot commented May 20, 2026 •

edited

Loading

Rate limit exceeded

Uh oh!

coderabbitai Bot left a comment

Uh oh!

Uh oh!

asamal4 left a comment

Uh oh!

Uh oh!

coderabbitai Bot left a comment

Uh oh!

asamal4 left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

xmican10 commented May 20, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Type of change

Tools used to create PR

Related Tickets & Documents

Checklist before requesting a review

Testing

Summary by CodeRabbit

Uh oh!

coderabbitai Bot commented May 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Rate limit exceeded

Walkthrough

Changes

Estimated code review effort

Possibly related PRs

Suggested reviewers

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

asamal4 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

asamal4 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

xmican10 commented May 20, 2026 •

edited by coderabbitai Bot

Loading

coderabbitai Bot commented May 20, 2026 •

edited

Loading