Context
During skill generation, the version lookup system queries npm/pip for latest stable versions. These are provided to the LLM in the prompt with instructions to use them.
Problem
The programmatic lookup returned correct versions (e.g., next@16.1.6), but the LLM's review flagged this as incorrect:
Next.js version 16.1.6 doesn't exist. Latest stable is 15.x
The LLM incorrectly "corrected" the version based on its training data, which is outdated compared to the real-time npm query.
Root Cause
The LLM's training data has older version information and it overrules the accurate programmatic lookup.
Suggested Fix
- Strengthen prompt language - Make it clearer that the programmatic lookup is authoritative and should not be second-guessed
- Re-query on doubt - If the LLM questions a version, instruct it to re-run the version lookup rather than relying on training data
- Review prompt update - The review checklist should not flag versions that match the provided lookup values
Current Prompt Language
The generate prompt says:
Use these EXACT versions (queried from package managers at generation time):
This may need to be stronger, e.g.:
CRITICAL: These versions were queried from npm/pip at generation time and are AUTHORITATIVE. Do not override based on training data. If uncertain, re-query the package manager.
Related
Context
During skill generation, the version lookup system queries npm/pip for latest stable versions. These are provided to the LLM in the prompt with instructions to use them.
Problem
The programmatic lookup returned correct versions (e.g.,
next@16.1.6), but the LLM's review flagged this as incorrect:The LLM incorrectly "corrected" the version based on its training data, which is outdated compared to the real-time npm query.
Root Cause
The LLM's training data has older version information and it overrules the accurate programmatic lookup.
Suggested Fix
Current Prompt Language
The generate prompt says:
This may need to be stronger, e.g.:
Related