Skip to content

Refine data pipeline modularization report#66

Open
piotrnowakowski wants to merge 1 commit into
mainfrom
ON-5659-modularize-URBIND
Open

Refine data pipeline modularization report#66
piotrnowakowski wants to merge 1 commit into
mainfrom
ON-5659-modularize-URBIND

Conversation

@piotrnowakowski

Copy link
Copy Markdown
Collaborator

a full overview of what we have and should do for easier picking apart of the code implemented by agents from this repo into another projects

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think i introduced it in previous PR but ignore the diff as the document was not fully ready before it had much to long context that was not usefull for example

@mircorudolph mircorudolph left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree with the report.

I think the most important steps are:

  • Add missing benchmarks where operational quality gaps are largest
    -> this aligns strongly with writing detailed artifacts. Each step should create clearly named artifacts with inputs, outputs, and relevant intermediate values. Those should always be written as opposed to on-demand benchmarks
  • Add explicit result models for stage outputs where tuple or loose-dict
    contracts are doing too much work. -> makes a lot of sense to have clear hand-off contracts.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants