Skip to content

Proposal: introduce evals to 12 factor app #65

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
Viktor286 wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Proposal: introduce evals to 12 factor app #65

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 8 additions & 3 deletions content/factor-03-own-your-context-window.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,15 +9,20 @@ You don't necessarily need to use standard message-based formats for conveying c
<!-- todo syntax highlighting -->
<!-- ![130-own-your-context-building](https://github.com/humanlayer/12-factor-agents/blob/main/img/130-own-your-context-building.png) -->

Everything is context engineering. [LLMs are stateless functions](https://thedataexchange.media/baml-revolution-in-ai-engineering/) that turn inputs into outputs. To get the best outputs, you need to give them the best inputs.
Everything is context engineering. [LLMs are stateless functions](https://thedataexchange.media/baml-revolution-in-ai-engineering/) that turn inputs into outputs. This can be seen as increasing the specialization of your model instance to perform better in your specific use case.

While large language models (LLMs) represent general intelligence, each instance must operate within the rules of your system in order to significantly increase the likelihood of success, despite their inherently non-deterministic nature. This also means the model should be aware of any unique features of your system. These nudges help guide the model toward producing high-quality, specialized results.

Evaluation tests help determine whether the agent is aligned with the system's rules and performs well enough to meet the efficiency threshold.

Creating great context means:

- The prompt and instructions you give to the model
- Any documents or external data you retrieve (e.g. RAG)
- Any past state, tool calls, results, or other history
- Any past state, tool calls, results, or other history
- Any past messages or events from related but separate histories/conversations (Memory)
- Instructions about what sorts of structured data to output
- Instructions about what sorts of structured data to output (Schema)
- Evaluation tests (evals) to establish efficiency benchmarks (Alignment testing, Regression checks)

![image](https://github.com/user-attachments/assets/0f1f193f-8e94-4044-a276-576bd7764fd0)

Expand Down