Scosman/doc skills

[scosman]

I present "Doc Skills". Basically a combination of Leonard's RAG infra and Sam's skill infra: generate a skill from a set of PDFs. Uses the extraction and chunking pipelines we already have, but simplier to deploy than RAG (no embedding/vector search). Probably better for reasonable sized datasets (RAG still wins on huge ones).

Target branch is chat-integration, but that's not right. Just that for now because I branched off of it.

doc_skills.mp4

[github-actions]

📊 Coverage Report

Overall Coverage: 92%

Diff: origin/leonard/chat-integration...HEAD

• app/desktop/desktop_server.py (100%)
• app/desktop/studio_server/doc_skill_api.py (88.4%): Missing lines 72,92,96-100,104-106,120,128,135,142-143,174-175,181-182
• app/desktop/studio_server/doc_skill_pipeline.py (100%)
• app/desktop/studio_server/doc_skill_skill_builder.py (99.3%): Missing lines 13
• app/desktop/studio_server/skill_api.py (80.6%): Missing lines 86,221-222,225-226,248
• libs/core/kiln_ai/adapters/rag/rag_runners.py (100%)
• libs/core/kiln_ai/datamodel/init.py (100%)
• libs/core/kiln_ai/datamodel/document_skill.py (97.1%): Missing lines 9
• libs/core/kiln_ai/datamodel/project.py (66.7%): Missing lines 77

Summary

• Total: 467 lines
• Missing: 28 lines
• Coverage: 94%

Line-by-line

View line-by-line diff coverage

app/desktop/studio_server/doc_skill_api.py

Lines 68-76

  68 def _get_filtered_documents(project: Project, tags: list[str] | None) -> list[Document]:
  69     all_docs = project.documents(readonly=True)
  70     if tags is None:
  71         return all_docs
! 72     return filter_documents_by_tags(all_docs, tags)
  73 
  74 
  75 def compute_doc_skill_progress(
  76     project: Project, doc_skill: DocumentSkill

---

Lines 88-110

   88     extracted = 0
   89     chunked = 0
   90 
   91     for doc in docs:
!  92         has_extraction = any(
   93             ext.extractor_config_id == doc_skill.extractor_config_id
   94             for ext in doc.extractions()
   95         )
!  96         if has_extraction:
!  97             extracted += 1
!  98             for ext in doc.extractions():
!  99                 if ext.extractor_config_id == doc_skill.extractor_config_id:
! 100                     has_chunks = any(
  101                         cd.chunker_config_id == doc_skill.chunker_config_id
  102                         for cd in ext.chunked_documents()
  103                     )
! 104                     if has_chunks:
! 105                         chunked += 1
! 106                     break
  107 
  108     return DocSkillProgress(
  109         total_document_count=len(docs),
  110         total_document_extracted_count=extracted,

---

Lines 116-124

  116 async def _build_workflow_runner(
  117     project: Project, doc_skill: DocumentSkill
  118 ) -> DocSkillWorkflowRunner:
  119     if not doc_skill.extractor_config_id:
! 120         raise HTTPException(status_code=422, detail="Extractor config not found.")
  121     extractor_config = ExtractorConfig.from_id_and_parent_path(
  122         doc_skill.extractor_config_id, project.path
  123     )
  124     if extractor_config is None:

---

Lines 124-132

  124     if extractor_config is None:
  125         raise HTTPException(status_code=422, detail="Extractor config not found.")
  126 
  127     if not doc_skill.chunker_config_id:
! 128         raise HTTPException(status_code=422, detail="Chunker config not found.")
  129     chunker_config = ChunkerConfig.from_id_and_parent_path(
  130         doc_skill.chunker_config_id, project.path
  131     )
  132     if chunker_config is None:

---

Lines 131-139

  131     )
  132     if chunker_config is None:
  133         raise HTTPException(status_code=422, detail="Chunker config not found.")
  134 
! 135     config = DocSkillWorkflowRunnerConfig(
  136         doc_skill=doc_skill,
  137         project=project,
  138         extractor_config=extractor_config,
  139         chunker_config=chunker_config,

---

Lines 138-147

  138         extractor_config=extractor_config,
  139         chunker_config=chunker_config,
  140     )
  141 
! 142     initial_progress = compute_doc_skill_progress(project, doc_skill)
! 143     return DocSkillWorkflowRunner(config, initial_progress)
  144 
  145 
  146 def _serialize_progress(progress: DocSkillProgress) -> dict:
  147     return {

---

Lines 170-179

  170                 latest_progress = progress.model_copy()
  171                 data = _serialize_progress(progress)
  172                 yield f"data: {json.dumps(data, ensure_ascii=False)}\n\n"
  173         except asyncio.TimeoutError:
! 174             logger.info("Doc skill workflow runner timed out waiting for lock")
! 175             latest_progress.logs = [
  176                 LogMessage(
  177                     level="error",
  178                     message="Timed out after waiting for the lock to be acquired. This may be due to a concurrent pipeline running. You may retry in a few minutes.",
  179                 )

---

Lines 177-186

  177                     level="error",
  178                     message="Timed out after waiting for the lock to be acquired. This may be due to a concurrent pipeline running. You may retry in a few minutes.",
  179                 )
  180             ]
! 181             data = _serialize_progress(latest_progress)
! 182             yield f"data: {json.dumps(data, ensure_ascii=False)}\n\n"
  183         except Exception as e:
  184             logger.error(
  185                 f"Unexpected server error running doc skill workflow: {e}",
  186                 exc_info=True,

---

app/desktop/studio_server/doc_skill_skill_builder.py

Lines 9-17

   9 from kiln_ai.datamodel.extraction import Document, OutputFormat
  10 from kiln_ai.datamodel.skill import Skill
  11 
  12 if TYPE_CHECKING:
! 13     from .doc_skill_pipeline import DocSkillWorkflowRunnerConfig
  14 
  15 
  16 class SkillBuilder:
  17     def __init__(

---

app/desktop/studio_server/skill_api.py

Lines 82-90

  82 
  83 
  84 def _count_files_recursive(directory: "pathlib.Path") -> int:
  85     if not directory.exists():
! 86         return 0
  87     return sum(1 for f in directory.rglob("*") if f.is_file())
  88 
  89 
  90 def skill_to_response(skill: Skill) -> SkillResponse:

---

Lines 217-230

  217     ) -> SkillFileCountsResponse:
  218         skill = _get_skill(project_id, skill_id)
  219         try:
  220             reference_count = _count_files_recursive(skill.references_dir())
! 221         except ValueError:
! 222             reference_count = 0
  223         try:
  224             asset_count = _count_files_recursive(skill.assets_dir())
! 225         except ValueError:
! 226             asset_count = 0
  227         return SkillFileCountsResponse(
  228             reference_count=reference_count,
  229             asset_count=asset_count,
  230         )

---

Lines 244-252

  244         ],
  245     ) -> OpenFolderResponse:
  246         skill = _get_skill(project_id, skill_id)
  247         if not skill.path:
! 248             raise HTTPException(status_code=500, detail="Skill path not found")
  249         skill_dir = skill.path.parent
  250         # open_folder expects a file path (it calls os.path.dirname internally)
  251         open_folder(str(skill_dir / "SKILL.md"))
  252         return OpenFolderResponse(path=str(skill_dir))

---

libs/core/kiln_ai/datamodel/document_skill.py

Lines 5-13

   5 from kiln_ai.datamodel.basemodel import ID_TYPE, FilenameString, KilnParentedModel
   6 from kiln_ai.utils.validation import SkillNameString
   7 
   8 if TYPE_CHECKING:
!  9     from kiln_ai.datamodel.project import Project
  10 
  11 
  12 class DocumentSkill(KilnParentedModel):
  13     """Configuration for generating a Skill from project documents.

---

libs/core/kiln_ai/datamodel/project.py

Lines 73-78

  73     def skills(self, readonly: bool = False) -> list[Skill]:
  74         return super().skills(readonly=readonly)  # type: ignore
  75 
  76     def document_skills(self, readonly: bool = False) -> list[DocumentSkill]:
! 77         return super().document_skills(readonly=readonly)  # type: ignore

---

---

• 📊 HTML Coverage Report - Interactive coverage report
• 📈 Diff Coverage Report - Detailed diff analysis
• Github Actions Run - View the full coverage report

[gemini-code-assist]

Code Review

This pull request introduces 'Doc Skills', a new feature that allows users to convert project documents into agent skills. It includes a new DocumentSkill data model, a pipeline runner for extraction and chunking, and a set of API endpoints for managing and running these skills. The frontend is updated with a new entry point, creation templates, and detail pages. I have provided feedback on optimizing the progress computation logic in the API layer and suggested leveraging the newly generated OpenAPI types for the skill file count and folder opening endpoints.

[gemini-code-assist]

The comment on this line states that the new endpoints aren't in the generated OpenAPI types. However, with the schema generation updates in this PR, get_skill_file_counts and open_skill_folder are now included in api_schema.d.ts. You can remove this comment and use the typed client for these API calls for better type safety.

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 4939942f-e930-483b-8be7-1c0ef5cfc51a

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch scosman/doc_skills

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}