Skip to content

feat: file upload to object storage and then async file verification#207

Open
heldtogether wants to merge 1 commit into
doublewordai:mainfrom
heldtogether:feat/async-file-verification
Open

feat: file upload to object storage and then async file verification#207
heldtogether wants to merge 1 commit into
doublewordai:mainfrom
heldtogether:feat/async-file-verification

Conversation

@heldtogether
Copy link
Copy Markdown

Summary

This removes direct dwctl writes to fusillade.files and fusillade.request_templates in the object-store file upload flow.

dwctl still owns object-store ingestion orchestration and file_ingest_jobs, but file placeholder creation and template persistence now go through new fusillade storage APIs instead of raw SQL.

Changes

  • Added fusillade::Storage methods for:
    • creating a file placeholder with a caller-supplied FileId
    • populating templates for an existing file
  • Added FilePlaceholderInput to fusillade
  • Implemented the new APIs in PostgresRequestManager
  • Reused fusillade’s existing batched template insert path for background ingest
  • Updated the dwctl object-store upload path to create the placeholder via request_manager
  • Updated the ingest job to populate templates via request_manager
  • Switched dwctl to the local fusillade path dependency so it can use the new API
  • Added focused fusillade tests for placeholder creation and template population

Why

The object-store upload path had started embedding raw SQL for fusillade tables inside dwctl, which breaks the intended ownership boundary. This change moves that persistence logic back
into fusillade while preserving the current async ingest behavior and immediate file ID response.

Notes

  • file_ingest_jobs remains in dwctl
  • JSONL parsing and validation remain in dwctl
  • Placeholder files still use the current behavior of being created as processed immediately

@heldtogether heldtogether marked this pull request as ready for review April 7, 2026 13:50
@heldtogether heldtogether changed the title File upload to object storage and then async file verification feat: file upload to object storage and then async file verification Apr 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant