feat: add evals for fork-specific features (v0.1.2)#79
Merged
Conversation
Add test coverage for agentic-sdlc preset functionality: - Mission Brief test suite: 4 tests for completeness, quality, constraint extraction, approval flow - Fork spec section tests: 3 tests for Goal, Demo Sentence, Boundary Map, Constraints - New graders: check_mission_brief_completeness, check_mission_brief_quality, check_fork_spec_sections - spec-prompt.txt updated with fork-specific sections - Command pattern grader now accepts both speckit.* and adlc.* Evals: 29 LLM tests + 39 unit tests across 7 suites (was 22+39 / 6)
Collaborator
|
LGTM, but you have conflicts that need to be addressed |
Member
Author
|
@kfinkels merge it, but we do need to do src evals as well |
kanfil
added a commit
that referenced
this pull request
Mar 18, 2026
Merge Strategy: - Reset to pre-merge commit 2f0852a (last clean tikalk state) - Re-applied PR #79 evals-refactor changes - Merged upstream/main with careful conflict resolution Tikalk-specific code preserved: - Config management (get_global_config_path, load_config, save_config, etc.) - Architecture config (get_architecture_diagram_format, get_adr_heuristic, etc.) - Skills config (get_skills_config, set_skills_config) - Skill subcommand app with all skill commands (search, install, update, etc.) - show_skills_banner function - Orange theme colors (ACCENT_COLOR, BANNER_COLORS) - Agentic SDLC branding (TAGLINE, show_banner extensions display) - _run_git_command and sync_team_ai_directives functions - install_bundled_extensions and install_bundled_presets - _ensure_commands_for_agent function - _validate_ai_assistant and _validate_ai_commands_dir callbacks Upstream features merged: - New agents: kimi, trae, pi, bob, vibe, tabnine, etc. - Updated preset system with new PresetCatalog features - Test improvements and new test files - Extension system enhancements - Various bug fixes and improvements Conflict Resolution: - Kept tikalk versions: README, pyproject.toml, docs, bash scripts - Kept upstream + tikalk additions: src/specify_cli/__init__.py - Kept upstream versions: agents.py, presets.py, tests All 444 tests pass.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Add test coverage for agentic-sdlc preset functionality that was missing from the upstream evals framework.
Changes
New Tests (7 total)
Mission Brief suite (4 tests): Tests the new Mission Brief enforcement in
/adlc.spec.specifyFork spec sections (3 tests): Tests fork-specific spec template sections
New Graders
check_mission_brief_completeness()- validates Mission Brief has all required elementscheck_mission_brief_quality()- validates quality of each Mission Brief elementcheck_fork_spec_sections()- validates fork-specific spec sectionsFixes
check_extension_manifest()now accepts bothspeckit.*andadlc.*command patternsFiles Changed
evals/prompts/spec-prompt.txt- added fork sections (Goal, Demo Sentence, Boundary Map)evals/prompts/mission-brief-prompt.txt- new prompt for Mission Brief testsevals/configs/promptfooconfig-mission-brief.js- new config for Mission Brief suiteevals/configs/promptfooconfig.js- added 4 Mission Brief testsevals/configs/promptfooconfig-spec.js- added 3 fork section testsevals/graders/custom_graders.py- added 3 new graders, fixed command pattern regexevals/README.md- updated test counts and documentationpyproject.toml- version bump to 0.1.2CHANGELOG.md- added v0.1.2 entryTest Results
Related
/adlc.spec.specify