Skip to content

Commit 1dd0464

Browse files
EngHabuCopilotmachichimaiaroslav-ciupinWangWang0226
authored
Flyte 2: Cut over to Flyte 2 new backend implementation (#6583)
* Use Docker images locally and in CI (#6671) * Use Docker images locally and in CI Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * fix branch prefix tag Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * use github.head_ref instead Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * fix tag in comment Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * make local docker experience Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * try Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * wip Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * Faster python install? Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * wait for image build wf Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * Optimize image Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * cleanups Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * wip Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * regen Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * cleanup not needed commands Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * Update .github/workflows/regenerate-on-comment.yml Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * test changes Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * PR Comments Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * PR Comments Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * Trying with :v2 image Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * Fix WF Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * Update guides Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * Update .github/workflows/regenerate-on-comment.yml Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * Update .github/workflows/check-generate.yml Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * Update .github/workflows/regenerate-on-comment.yml Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * Update DOCKER_QUICK_START.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * PR Comments Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * Update README.md Co-authored-by: Nary Yeh <60069744+machichima@users.noreply.github.com> Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * Update CONTRIBUTING.md Co-authored-by: Nary Yeh <60069744+machichima@users.noreply.github.com> Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * PR Comments Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> --------- Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-authored-by: Nary Yeh <60069744+machichima@users.noreply.github.com> * Don't publish SHA image when merging to v2 or main (#6684) * Don't publish SHA image when merging to v2 or main Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * regen Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> --------- Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * Fix trigger automation type enum (#6691) Signed-off-by: Iaroslav Ciupin <iaroslav@union.ai> * Pytorch proto v2 migration (#6689) * Pytorch proto v2 migration Signed-off-by: WangWang0226 <eeha8834@gmail.com> * Fix archived link and rebase branch to update Cargo.lock version change Signed-off-by: WangWang0226 <eeha8834@gmail.com> * Remove useless line in pytorch.proto Signed-off-by: WangWang0226 <eeha8834@gmail.com> --------- Signed-off-by: WangWang0226 <eeha8834@gmail.com> * Add metadata to list task response proto (#6683) * feat: add metadata to list tash response proto Signed-off-by: machichima <nary12321@gmail.com> * build: make gen Signed-off-by: machichima <nary12321@gmail.com> * build: make gen Signed-off-by: machichima <nary12321@gmail.com> --------- Signed-off-by: machichima <nary12321@gmail.com> * Extend task triggers summary (#6696) Signed-off-by: Iaroslav Ciupin <iaroslav@union.ai> * [FEAT] add List versions request and response (#6687) * feat: add List versions request and response Signed-off-by: machichima <nary12321@gmail.com> * feat: add ListVersions RPC Signed-off-by: machichima <nary12321@gmail.com> * fix: make gen Signed-off-by: machichima <nary12321@gmail.com> * fix: make gen Signed-off-by: machichima <nary12321@gmail.com> * feat: add common list request to list versions request Signed-off-by: machichima <nary12321@gmail.com> --------- Signed-off-by: machichima <nary12321@gmail.com> * Add apps protos (#6693) Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * Add Connector proto (#6659) Signed-off-by: Kevin Su <pingsutw@apache.org> * fix npm publish Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * Fix rust cargo publish Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * Generate missing imports in typescript (#6713) * Generate missing imports in typescript Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * Add dry-run targets Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * the package.json source Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> --------- Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * Trigger cron expression timezone (#6716) Signed-off-by: Iaroslav Ciupin <iaroslav@union.ai> * Fix trigger cron expression timezone (#6720) Signed-off-by: Iaroslav Ciupin <iaroslav@union.ai> * v2 backend - stubs (#6697) * Initial implementation of Queue/State/Runs/Executor services Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * Rename CRD from flyte.org.flyte.org to flyte.org Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * Update display columns for the CRD Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * gen Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * Update stateService interface Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * Lint fix and common go targets Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> --------- Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * Add CacheConfig to RunSpec (#6723) * Add CacheConfig to RunSpec Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * descriptions Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> --------- Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * Nary/watch groups (#6712) * feat: add watch group rpc/request/response proto Signed-off-by: machichima <nary12321@gmail.com> * feat: add TaskGroup Signed-off-by: machichima <nary12321@gmail.com> * build: make gen Signed-off-by: machichima <nary12321@gmail.com> * build: make gen Signed-off-by: machichima <nary12321@gmail.com> --------- Signed-off-by: machichima <nary12321@gmail.com> * Mild cleanup and compilation fix (#6732) * [Fix] Downgrade otel to 1.37 (#6729) * build: otel to 1.37 Signed-off-by: machichima <nary12321@gmail.com> * build: make gen Signed-off-by: machichima <nary12321@gmail.com> --------- Signed-off-by: machichima <nary12321@gmail.com> * Generate mocks for app package (#6734) Signed-off-by: Iaroslav Ciupin <iaroslav@union.ai> * Add list runs proto fields (#6709) * update proto Signed-off-by: Alex Wu <c.alexwu@gmail.com> * update proto Signed-off-by: Alex Wu <c.alexwu@gmail.com> * update proto Signed-off-by: Alex Wu <c.alexwu@gmail.com> * re generate Signed-off-by: Alex Wu <c.alexwu@gmail.com> * update proto Signed-off-by: Alex Wu <c.alexwu@gmail.com> * re generate Signed-off-by: Alex Wu <c.alexwu@gmail.com> --------- Signed-off-by: Alex Wu <c.alexwu@gmail.com> * [Proto] add created_by to task group (#6736) * feat: add created_by to task group proto Signed-off-by: machichima <nary12321@gmail.com> * fix: make gen Signed-off-by: machichima <nary12321@gmail.com> --------- Signed-off-by: machichima <nary12321@gmail.com> * [Proto] Change created by to repeat EnrichedIdentity (#6739) * feat: created by to repeat EnrichedIdentity Signed-off-by: machichima <nary12321@gmail.com> * refactor: update docstring Signed-off-by: machichima <nary12321@gmail.com> * build: make gen Signed-off-by: machichima <nary12321@gmail.com> --------- Signed-off-by: machichima <nary12321@gmail.com> * fix(runs/docker): enable CGO for go-sqlite3 and include gen/go in bui… (#6730) * fix(runs/docker): enable CGO for go-sqlite3 and include gen/go in build context Signed-off-by: WangWang0226 <eeha8834@gmail.com> * fix(runs/docker): improve docker build path and remove unnecessary tag Signed-off-by: WangWang0226 <eeha8834@gmail.com> * fix: Postgres connectivity + compose v2 migration Signed-off-by: WangWang0226 <eeha8834@gmail.com> * commit change of make gen Signed-off-by: WangWang0226 <eeha8834@gmail.com> --------- Signed-off-by: WangWang0226 <eeha8834@gmail.com> * Add description field in task and trigger protos (#6722) * update proto Signed-off-by: Alex Wu <c.alexwu@gmail.com> * update protos Signed-off-by: Alex Wu <c.alexwu@gmail.com> * update proto Signed-off-by: Alex Wu <c.alexwu@gmail.com> * add fields in sourcecode proto Signed-off-by: Alex Wu <c.alexwu@gmail.com> * regenerate idl Signed-off-by: Alex Wu <c.alexwu@gmail.com> * update proto Signed-off-by: Alex Wu <c.alexwu@gmail.com> * generate idls Signed-off-by: Alex Wu <c.alexwu@gmail.com> * generate idl Signed-off-by: Alex Wu <c.alexwu@gmail.com> * re generate Signed-off-by: Alex Wu <c.alexwu@gmail.com> --------- Signed-off-by: Alex Wu <c.alexwu@gmail.com> * [Proto] Watch group add known sort fields (#6760) * feat: add KnownSortField in watch group request Signed-off-by: machichima <nary12321@gmail.com> * fix: latest_created_at to created_at Signed-off-by: machichima <nary12321@gmail.com> * build: make gen Signed-off-by: machichima <nary12321@gmail.com> --------- Signed-off-by: machichima <nary12321@gmail.com> * [Proto] Move action phase from workflow to common proto (#6765) * feat: move action Phase from workflow to common Signed-off-by: machichima <nary12321@gmail.com> * docs: add docstring to the ActionPhase Signed-off-by: machichima <nary12321@gmail.com> * build: make gen with docstring Signed-off-by: machichima <nary12321@gmail.com> * build: make gen Signed-off-by: machichima <nary12321@gmail.com> --------- Signed-off-by: machichima <nary12321@gmail.com> * [Proto] Add latest run to list task response (#6761) * feat: add LatestRunSummary to Task in ListTaskResponse Signed-off-by: machichima <nary12321@gmail.com> * build: make gen Signed-off-by: machichima <nary12321@gmail.com> * feat: update task proto for latest run Signed-off-by: machichima <nary12321@gmail.com> --------- Signed-off-by: machichima <nary12321@gmail.com> * fix proto (#6685) * fix proto Signed-off-by: Alex Wu <c.alexwu@gmail.com> * update protos Signed-off-by: Alex Wu <c.alexwu@gmail.com> * re generate idl Signed-off-by: Alex Wu <c.alexwu@gmail.com> --------- Signed-off-by: Alex Wu <c.alexwu@gmail.com> * Copy v2 plugins (#6718) * copy over flyteplugins Signed-off-by: Paul Dittamo <pvdittamo@gmail.com> * remove secret manager Signed-off-by: Paul Dittamo <pvdittamo@gmail.com> * remove array plugin Signed-off-by: Paul Dittamo <pvdittamo@gmail.com> * delete hive Signed-off-by: Paul Dittamo <pvdittamo@gmail.com> * remove athena, databricks, snowflake, big query Signed-off-by: Paul Dittamo <pvdittamo@gmail.com> * remove presto Signed-off-by: Paul Dittamo <pvdittamo@gmail.com> * copy over coreutils Signed-off-by: Paul Dittamo <pvdittamo@gmail.com> * temp delete webapi Signed-off-by: Paul Dittamo <pvdittamo@gmail.com> * delete auto refresh cache Signed-off-by: Paul Dittamo <pvdittamo@gmail.com> * Revert "delete auto refresh cache" This reverts commit fe08f12c25755447c11c0926d7e7fd25446160b6. * delete kfoperators Signed-off-by: Paul Dittamo <pvdittamo@gmail.com> * delete echo Signed-off-by: Paul Dittamo <pvdittamo@gmail.com> * copy over plugins Signed-off-by: Paul Dittamo <pvdittamo@gmail.com> * gen mocks Signed-off-by: Paul Dittamo <pvdittamo@gmail.com> * simplify modules Signed-off-by: Paul Dittamo <pvdittamo@gmail.com> * copy over flytestdlib Signed-off-by: Paul Dittamo <pvdittamo@gmail.com> * copy over event and cache proto + point flyteplugin and flytestdlib imports to v2 Signed-off-by: Paul Dittamo <pvdittamo@gmail.com> * add RawOutputDataConfig Signed-off-by: Paul Dittamo <pvdittamo@gmail.com> * copy over kubeflow operators + non connector/agent web api plugins Signed-off-by: Paul Dittamo <pvdittamo@gmail.com> * Revert "add RawOutputDataConfig" This reverts commit e260772b6b993ed3169c848a1d0179c6a479ca69. * make gen Signed-off-by: Paul Dittamo <pvdittamo@gmail.com> * clean up Signed-off-by: Paul Dittamo <pvdittamo@gmail.com> * gen Signed-off-by: Paul Dittamo <pvdittamo@gmail.com> * remove web api Signed-off-by: Paul Dittamo <pvdittamo@gmail.com> * gen Signed-off-by: Paul Dittamo <pvdittamo@gmail.com> --------- Signed-off-by: Paul Dittamo <pvdittamo@gmail.com> * [Proto] add short name to watch group task group response (#6781) * feat: add short name to watch group task group response Signed-off-by: machichima <nary12321@gmail.com> * build: make gen Signed-off-by: machichima <nary12321@gmail.com> --------- Signed-off-by: machichima <nary12321@gmail.com> * build: downgrade otel (#6787) Signed-off-by: machichima <nary12321@gmail.com> * flyteidl crate remove extension module (#6789) flyteidl crate remove extension module Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> * add CODEOWNERS (#6794) Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * Add queue dockerfile (#6767) * feat(queue): add Dockerfile and K8s deployment for Queue service Signed-off-by: WangWang0226 <eeha8834@gmail.com> * refactor(queue/k8s): migrate manifests to Kustomize layout and enable kubectl apply -k Signed-off-by: WangWang0226 <eeha8834@gmail.com> * chore(queue): drop Dockerfile healthcheck and align RBAC with executor style Signed-off-by: WangWang0226 <eeha8834@gmail.com> --------- Signed-off-by: WangWang0226 <eeha8834@gmail.com> * Store run source in ActionMetadata (#6793) Signed-off-by: Iaroslav Ciupin <iaroslav@union.ai> * Move the connector proto to the connector folder (#6797) Signed-off-by: Kevin Su <pingsutw@apache.org> * Add created_at field to TriggerRevision (#6803) Signed-off-by: Iaroslav Ciupin <iaroslav@union.ai> * Add tasklog in the TaskMetadata Signed-off-by: Kevin Su <pingsutw@apache.org> * Revert "Add tasklog in the TaskMetadata" This reverts commit ab2f36b9c330f0f1a7241389e984447eef485ac8. * Add tasklog in the TaskMetadata (#6807) Signed-off-by: Kevin Su <pingsutw@apache.org> * [Run Service] Refactor runservice (#6810) * refactor: split out action and general interfaces Signed-off-by: machichima <nary12321@gmail.com> * refactor: update state service Signed-off-by: machichima <nary12321@gmail.com> * chore: remove cmd in gitignore Signed-off-by: machichima <nary12321@gmail.com> * fix: add repository.go as entry point for repository inerface Signed-off-by: machichima <nary12321@gmail.com> * build: make gen Signed-off-by: machichima <nary12321@gmail.com> --------- Signed-off-by: machichima <nary12321@gmail.com> * feat: add error counts for task group proto (#6808) Signed-off-by: machichima <nary12321@gmail.com> * Mirgate Copilot (#6804) * Migrate CoPilot Signed-off-by: Barry Wu <a0987818905@gmail.com> * Update: Add cmd/ and delete go.mod and go.sum in copilot Signed-off-by: Barry Wu <a0987818905@gmail.com> * Fix CI error Signed-off-by: Barry Wu <a0987818905@gmail.com> * Update make gen changes Signed-off-by: Barry Wu <a0987818905@gmail.com> --------- Signed-off-by: Barry Wu <a0987818905@gmail.com> * Add task service (#6813) * feat: add task model and use db&gorm tag together Signed-off-by: machichima <nary12321@gmail.com> * docs: update development guidance Signed-off-by: machichima <nary12321@gmail.com> * feat: init moving task service Signed-off-by: machichima <nary12321@gmail.com> * feat: add filter and sort for task service Signed-off-by: machichima <nary12321@gmail.com> * feat: mount task service Signed-off-by: machichima <nary12321@gmail.com> * refactor: clean up Signed-off-by: machichima <nary12321@gmail.com> * docs: add task service api endpoint docs Signed-off-by: machichima <nary12321@gmail.com> * test: add unit test for repository/impl Signed-off-by: machichima <nary12321@gmail.com> * test: unit test for task transformers Signed-off-by: machichima <nary12321@gmail.com> * feat: add mocks for repo interfaces Signed-off-by: machichima <nary12321@gmail.com> * test: unit test for runs/service Signed-off-by: machichima <nary12321@gmail.com> * build: make gen Signed-off-by: machichima <nary12321@gmail.com> * fix: add migration for task and task spec Signed-off-by: machichima <nary12321@gmail.com> * refactor: format Signed-off-by: machichima <nary12321@gmail.com> * docs: add example script for create and list task Signed-off-by: machichima <nary12321@gmail.com> * fix: remove unused impl/constant.go Signed-off-by: machichima <nary12321@gmail.com> * fix: prevent overwrite filter Signed-off-by: machichima <nary12321@gmail.com> * refactor: move To... function to transformers Signed-off-by: machichima <nary12321@gmail.com> * build: make gen Signed-off-by: machichima <nary12321@gmail.com> --------- Signed-off-by: machichima <nary12321@gmail.com> * [Test] Add Task service api test structure and for DeployTask (#6823) * refactor: rename tests/ to test/ Signed-off-by: machichima <nary12321@gmail.com> * feat: add test setup and cleanp Signed-off-by: machichima <nary12321@gmail.com> * test: add deploy task api test Signed-off-by: machichima <nary12321@gmail.com> * docs: update make commands and docs Signed-off-by: machichima <nary12321@gmail.com> * build: make gen Signed-off-by: machichima <nary12321@gmail.com> --------- Signed-off-by: machichima <nary12321@gmail.com> * Fix: Add timestamps to task creation (#6825) * added timestamps Signed-off-by: “Alex <alexchien130@gmail.com> * added test and match CI Signed-off-by: “Alex <alexchien130@gmail.com> * cleaner testing Signed-off-by: “Alex <alexchien130@gmail.com> * better readability Signed-off-by: “Alex <alexchien130@gmail.com> * remove empty line Signed-off-by: “Alex <alexchien130@gmail.com> --------- Signed-off-by: “Alex <alexchien130@gmail.com> * [Executor] Update taskaction cr (#6826) * feat: remove phase and message Signed-off-by: machichima <nary12321@gmail.com> * feat: add condition type and reason constant Signed-off-by: machichima <nary12321@gmail.com> * feat: use condition instead of phase Signed-off-by: machichima <nary12321@gmail.com> * feat: add read task spec Signed-off-by: machichima <nary12321@gmail.com> * fix: update go mod to build executor correctly Signed-off-by: machichima <nary12321@gmail.com> * feat: add printcolumn for taskaction crd Signed-off-by: machichima <nary12321@gmail.com> * build: update dockerfile and makefile to build successfully Signed-off-by: machichima <nary12321@gmail.com> * docs: add development guide and readme Signed-off-by: machichima <nary12321@gmail.com> * feat: add mockery for workflowconnect Signed-off-by: machichima <nary12321@gmail.com> * test: update taskaction controller test to make it pass Signed-off-by: machichima <nary12321@gmail.com> * feat: add validation for ParentActionName Signed-off-by: machichima <nary12321@gmail.com> * fix: revert changes in kustomization yaml Signed-off-by: machichima <nary12321@gmail.com> * docs: update typo Signed-off-by: machichima <nary12321@gmail.com> * build: make gen Signed-off-by: machichima <nary12321@gmail.com> --------- Signed-off-by: machichima <nary12321@gmail.com> * Support executor (#6792) * add RawOutputDataConfig Signed-off-by: Paul Dittamo <pvdittamo@gmail.com> * add ExecutionEnv idl Signed-off-by: Paul Dittamo <pvdittamo@gmail.com> * add Node related idl Signed-off-by: Paul Dittamo <pvdittamo@gmail.com> * remove datacatalog Signed-off-by: Paul Dittamo <pvdittamo@gmail.com> * re-add mocks Signed-off-by: Paul Dittamo <pvdittamo@gmail.com> * remove dynamic job spec Signed-off-by: Paul Dittamo <pvdittamo@gmail.com> * update cache client Signed-off-by: Paul Dittamo <pvdittamo@gmail.com> * gen storage metadata mock Signed-off-by: Paul Dittamo <pvdittamo@gmail.com> * set v2 scope for prom request metrics Signed-off-by: Paul Dittamo <pvdittamo@gmail.com> * copy over grpcutils metrics Signed-off-by: Paul Dittamo <pvdittamo@gmail.com> * move secret manager impl to executor Signed-off-by: Paul Dittamo <pvdittamo@gmail.com> * copy over garbage collectable changes Signed-off-by: Paul Dittamo <pvdittamo@gmail.com> * gen Signed-off-by: Paul Dittamo <pvdittamo@gmail.com> * remove v1 proto defs Signed-off-by: Paul Dittamo <pvdittamo@gmail.com> * remove execution_envs def Signed-off-by: Paul Dittamo <pvdittamo@gmail.com> * tidy Signed-off-by: Paul Dittamo <pvdittamo@gmail.com> * gen Signed-off-by: Paul Dittamo <pvdittamo@gmail.com> --------- Signed-off-by: Paul Dittamo <pvdittamo@gmail.com> * Add test for TaskService.ListVersions and test script (#6833) * feat: Add test for TaskService.ListVersions and test script Signed-off-by: Kevin Liao <q85292542000@gmail.com> * Update generated files Signed-off-by: Kevin Liao <q85292542000@gmail.com> * Revert "Update generated files" This reverts commit 4e7bb31728c8ebe8e081c40601615cc8e1462da9. Signed-off-by: Kevin Liao <q85292542000@gmail.com> * make gen Signed-off-by: Kevin Liao <q85292542000@gmail.com> * Revert "make gen" This reverts commit e590fed3fe2467d62a02a9b8a25c29ae8aff958f. Signed-off-by: Kevin Liao <q85292542000@gmail.com> * make gen Signed-off-by: Kevin Liao <q85292542000@gmail.com> * Remove cleanup db Signed-off-by: Kevin Liao <q85292542000@gmail.com> * update uv.lock Signed-off-by: Kevin Liao <q85292542000@gmail.com> --------- Signed-off-by: Kevin Liao <q85292542000@gmail.com> * [Test] Task service api test fix review (#6830) * refactor: remove unused function Signed-off-by: machichima <nary12321@gmail.com> * feat: fail directly if test server failed to start Signed-off-by: machichima <nary12321@gmail.com> * refactor: fix typo Signed-off-by: machichima <nary12321@gmail.com> * test: loop through model slice for cleanup Signed-off-by: machichima <nary12321@gmail.com> * test: clean up to defer block Signed-off-by: machichima <nary12321@gmail.com> --------- Signed-off-by: machichima <nary12321@gmail.com> * [v2] Refactor: Define all mockery in .mockery.yaml (#6829) * refactor: define all mockery in .mockery.yaml - Centralize mockery configuration in .mockery.yaml - Remove //go:generate mockery comments from interface files - Add configurations for flyteplugins and flytestdlib packages - Add global filename, inpackage, and outpkg configurations - Remove all 'case: underscore' settings (use global filename template instead) This change consolidates all mock generation configuration into a single location, making it easier to maintain and ensuring consistency across the codebase. This ensures all mock files: - Are named in snake_case via filename template - Are placed in mocks/ directories with package mocks - Follow consistent naming conventions across the codebase Fixes #6815 Signed-off-by: majiayu000 <1835304752@qq.com> * Run make gen Signed-off-by: majiayu000 <1835304752@qq.com> * Update tests for new mockery output Signed-off-by: majiayu000 <1835304752@qq.com> * Regenerate mocks after rebase Signed-off-by: majiayu000 <1835304752@qq.com> * build: move with-expecter to global settings Signed-off-by: machichima <nary12321@gmail.com> * fix: build error in flyteplugins Signed-off-by: machichima <nary12321@gmail.com> * test: convert On to EXPECT in flyteplugins tests Signed-off-by: machichima <nary12321@gmail.com> * test: more On to EXPECT in flyteplugins Signed-off-by: machichima <nary12321@gmail.com> * test: more On to EXPECT in flyteplugins Signed-off-by: machichima <nary12321@gmail.com> * test: On to EXPECT in executor/ Signed-off-by: machichima <nary12321@gmail.com> --------- Signed-off-by: majiayu000 <1835304752@qq.com> Signed-off-by: machichima <nary12321@gmail.com> Co-authored-by: machichima <nary12321@gmail.com> * test: add API test for ListTasks endpoint (#6842) * test: add API test for ListTasks endpoint Signed-off-by: Ryan Lo <wenchih@apache.org> * make gen Signed-off-by: Ryan Lo <wenchih@apache.org> * Revert "make gen" This reverts commit 9ae4c78e246f7e68a9670b771ad2f8fa3b7d1ae6. Signed-off-by: Ryan Lo <wenchih@apache.org> * chore: make gen & add test tasks Signed-off-by: Ryan Lo <wenchih@apache.org> --------- Signed-off-by: Ryan Lo <wenchih@apache.org> * [Fix] Ignore cargo lock (#6844) * build: ignore Cargo.lock Signed-off-by: machichima <nary12321@gmail.com> * build: update Makefile Signed-off-by: machichima <nary12321@gmail.com> * fix: ignore Cargo.lock in dockerignore Signed-off-by: machichima <nary12321@gmail.com> * build: git rm cargo lock Signed-off-by: machichima <nary12321@gmail.com> --------- Signed-off-by: machichima <nary12321@gmail.com> * [Feat][v2] Add data proxy (#6851) * feat: add data proxy proto for CreateUploadLocationRequest Signed-off-by: machichima <nary12321@gmail.com> * feat: add skeleton of data proxy service Signed-off-by: machichima <nary12321@gmail.com> * feat: update file exists and create storage path logic Signed-off-by: machichima <nary12321@gmail.com> * build: gen mockery for metadata Signed-off-by: machichima <nary12321@gmail.com> * feat: move out validation function Signed-off-by: machichima <nary12321@gmail.com> * test: add unit test Signed-off-by: machichima <nary12321@gmail.com> * build: add main.go to run dataproxy Signed-off-by: machichima <nary12321@gmail.com> * docs: add config example for data proxy and data storage Signed-off-by: machichima <nary12321@gmail.com> * docs: add development/readme and minio setup for develop locally Signed-off-by: machichima <nary12321@gmail.com> * docs: add example script Signed-off-by: machichima <nary12321@gmail.com> * docs: update config example YAML Signed-off-by: machichima <nary12321@gmail.com> * build: make gen Signed-off-by: machichima <nary12321@gmail.com> * fix: add AddContentMd5Metadata back and make compatible with flyteidl Signed-off-by: machichima <nary12321@gmail.com> * fix: AddContentMD5Metadata and base64 encode ContentMD5 Signed-off-by: machichima <nary12321@gmail.com> * feat: run all service (include dataproxy) in same port Signed-off-by: machichima <nary12321@gmail.com> * build: make gen Signed-off-by: machichima <nary12321@gmail.com> * fix: update mockery Signed-off-by: machichima <nary12321@gmail.com> --------- Signed-off-by: machichima <nary12321@gmail.com> * apply mockery in queue_service_test (#6859) Signed-off-by: WangWang0226 <eeha8834@gmail.com> * [V2] Dataproxy add gateway (#6866) * feat: add data proxy proto for CreateUploadLocationRequest Signed-off-by: machichima <nary12321@gmail.com> * feat: add skeleton of data proxy service Signed-off-by: machichima <nary12321@gmail.com> * feat: update file exists and create storage path logic Signed-off-by: machichima <nary12321@gmail.com> * build: gen mockery for metadata Signed-off-by: machichima <nary12321@gmail.com> * feat: move out validation function Signed-off-by: machichima <nary12321@gmail.com> * test: add unit test Signed-off-by: machichima <nary12321@gmail.com> * build: add main.go to run dataproxy Signed-off-by: machichima <nary12321@gmail.com> * docs: add config example for data proxy and data storage Signed-off-by: machichima <nary12321@gmail.com> * docs: add development/readme and minio setup for develop locally Signed-off-by: machichima <nary12321@gmail.com> * docs: add example script Signed-off-by: machichima <nary12321@gmail.com> * docs: update config example YAML Signed-off-by: machichima <nary12321@gmail.com> * build: make gen Signed-off-by: machichima <nary12321@gmail.com> * fix: add AddContentMd5Metadata back and make compatible with flyteidl Signed-off-by: machichima <nary12321@gmail.com> * fix: AddContentMD5Metadata and base64 encode ContentMD5 Signed-off-by: machichima <nary12321@gmail.com> * feat: run all service (include dataproxy) in same port Signed-off-by: machichima <nary12321@gmail.com> * build: make gen Signed-off-by: machichima <nary12321@gmail.com> * fix: update mockery Signed-off-by: machichima <nary12321@gmail.com> * feat: add gateway Signed-off-by: machichima <nary12321@gmail.com> --------- Signed-off-by: machichima <nary12321@gmail.com> * Add test script for GetTaskDetails (#6841) * Add test script for GetTaskDetails Signed-off-by: aniket2405 <aniketshaha2001@gmail.com> * Remove Cargo.lock Signed-off-by: aniket2405 <aniketshaha2001@gmail.com> * build: make gen Signed-off-by: machichima <nary12321@gmail.com> --------- Signed-off-by: aniket2405 <aniketshaha2001@gmail.com> Signed-off-by: machichima <nary12321@gmail.com> Co-authored-by: machichima <nary12321@gmail.com> * feat: add phase counts (#6862) Signed-off-by: machichima <nary12321@gmail.com> * build: pin openapiv2 version to prevent panic (#6870) Signed-off-by: machichima <nary12321@gmail.com> * feat: include run name in recent statuses (#6871) Signed-off-by: machichima <nary12321@gmail.com> * Add auth and identity proto (#6850) Signed-off-by: Kevin Su <pingsutw@apache.org> * Add connector plugin (#6861) Signed-off-by: Kevin Su <pingsutw@apache.org> * Always set enableServiceLinks to false (#6884) Signed-off-by: Kevin Su <pingsutw@apache.org> * JsonValuesToLiterals (#6878) Signed-off-by: Carina Ursu <carina@union.ai> * Build SubmitterPodTemplate with default toleration and disable EnableServiceLinks (#6892) Signed-off-by: Kevin Su <pingsutw@apache.org> * Stable sort ImagePullSecrets to avoid spec change errors (#6886) Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * Typed interface migrate (#6721) Signed-off-by: machichima <nary12321@gmail.com> Signed-off-by: Kevin Su <pingsutw@apache.org> Co-authored-by: Kevin Su <pingsutw@apache.org> * update log context (#6894) Signed-off-by: Kevin Su <pingsutw@apache.org> * Fix RayJob submitter pod affinity to use default config (#6897) Signed-off-by: Kevin Su <pingsutw@apache.org> Signed-off-by: Kevin Su <kevin@union.ai> * V2 - auth and project service crate updates (#6907) Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> * feat(workflow): add ActionsService proto for action state APIs (#6899) * feat(workflow): add ActionsService proto for action state APIs Introduce a new flyteidl2/workflow/actions_service.proto that defines an ActionsService consolidating state and queue functionality for actions. Add RPCs to update and retrieve action state (UpdateActionStatus, GetActionState), plus EnqueueAction, AbortQueuedAction, and Watch to support queuing and observation. Define request/response messages for UpdateActionStatus and GetActionState, including validation rules and fields for action identifiers, attempt number, status, and serialized NodeStatus state. This centralizes action state/queue operations into a single service intended to replace parts of the existing StateService and QueueService, simplifies APIs for workers, and provides at-least-once semantics for state watches. Signed-off-by: Sergey Vilgelm <sergey@union.ai> * feat(proto): validate action attempt is greater than zero and regen pb - Add buf.validate rule to require attempt > 0 for action request and response in actions_service.proto. This enforces that action attempts start from 1 and prevents zero-valued attempts from being accepted. - Regenerate Go protobuf bindings (actions_service.pb.go) to reflect the added validation annotations and updated descriptor bytes. Signed-off-by: Sergey Vilgelm <sergey@union.ai> * feat(actions): rename ActionsService RPCs and messages Rename several RPCs and corresponding request/response messages to simplify the ActionsService API and provide clearer, shorter names. - Rename EnqueueAction -> Enqueue for queueing new actions. - Rename Watch -> WatchForUpdates to clarify streaming semantics. - Rename UpdateActionStatus -> Update and its request/response types to UpdateRequest/UpdateResponse. - Rename GetActionState -> GetLatestState and its request/response types to GetLatestStateRequest/GetLatestStateResponse. - Rename AbortQueuedAction -> Abort while keeping the original abort request/response payload types. - Update comments to match the new names and keep deprecation notes. These changes make the service surface more concise and consistent, improving readability and aligning RPC names with their intent. Signed-off-by: Sergey Vilgelm <sergey@union.ai> * feat(actions): extract ActionsService and add action RPC msgs Move ActionsService into a dedicated flyteidl2.workflow.actions package and adjust Go package path accordingly. Replace fully-qualified RPC types with local message names for Enqueue, WatchForUpdates, and Abort RPCs to simplify imports and proto usage. Introduce new request/response messages for Enqueue: - EnqueueRequest with action_id, optional parent_action_name, run_spec, input/run output paths, group, subject and a required oneof spec for Task/Trace/Condition actions. - EnqueueResponse as an empty ack. Add WatchForUpdatesRequest/WatchForUpdatesResponse messages to support streaming updates with filtering by parent_action_id and payloads that carry either ActionUpdate or ControlMessage. Minor changes: - Import flyteidl2/task/run.proto for RunSpec. - Add buf.validate annotations and field constraints for required fields. These changes centralize action-related APIs, improve type locality in the proto, and prepare the ActionsService to replace previous StateService and QueueService behaviors. Signed-off-by: Sergey Vilgelm <sergey@union.ai> * feat(generated): add gRPC ActionsService stubs and servicer Add generated gRPC Python bindings for the ActionsService into the workflow actions package. This creates ActionsServiceStub with unary and streaming RPC methods (Enqueue, GetLatestState, WatchForUpdates, Update, Abort) wired to the protobuf serializers/deserializers, and an ActionsServiceServicer skeleton that returns UNIMPLEMENTED for each method. These generated files provide the client and server class templates required to implement and call the new ActionsService, which consolidates state and execution management for actions (replacing previous StateService and QueueService patterns). Keeping the generated stubs in tree enables service integration and local development until the service implementations are provided. Signed-off-by: Sergey Vilgelm <sergey@union.ai> * chore(docker): bump Rust base image to 1.85.0 Update the Dockerfile build ARG for Rust from 1.84.0 to 1.85.0. This brings the base Rust toolchain in line with the latest stable patch release, addressing compatibility and security fixes in the compiler and standard library. It ensures reproducible builds use the newer toolchain without changing other language versions. Signed-off-by: Sergey Vilgelm <sergey@union.ai> * refactor(actions): move actions proto package and go_package Change the proto package from flyteidl2.workflow.actions to flyteidl2.actions and update the Go package option accordingly. This reduces unnecessary nesting, aligns the proto package with the repository layout, and ensures generated Go code is placed under gen/go/flyteidl2/actions rather than gen/go/flyteidl2/workflow/actions. These changes simplify imports and reflect that the service belongs to the actions domain instead of the workflow subpackage. Signed-off-by: Sergey Vilgelm <sergey@union.ai> * fix(actions): update grpc imports and service paths for ActionsService Update generated gRPC Python stubs to reference the correct module path and fully-qualified service names for ActionsService. Replace incorrect imports and symbol names under flyteidl2.workflow.actions with the canonical flyteidl2.actions module. Adjust all RPC method paths and (de)serializers to use flyteidl2_dot_actions_dot_actions__service__pb2. This fixes mismatches between generated stubs and the actual package layout so gRPC clients and servers import the right protobuf symbols and register the correct RPC endpoints. Signed-off-by: Sergey Vilgelm <sergey@union.ai> --------- Signed-off-by: Sergey Vilgelm <sergey@union.ai> * [Protio] Watch group add run phases avg time delta in response (#6889) * add avg time to run phase field in task group Signed-off-by: machichima <nary12321@gmail.com> * build: make gen Signed-off-by: machichima <nary12321@gmail.com> * build: make gen Signed-off-by: machichima <nary12321@gmail.com> --------- Signed-off-by: machichima <nary12321@gmail.com> * Add app crate (#6913) Signed-off-by: Kevin Su <pingsutw@apache.org> * rename the module from auth to app (#6915) * Add app crate Signed-off-by: Kevin Su <pingsutw@apache.org> * rename the module from auth to app Signed-off-by: Kevin Su <pingsutw@apache.org> * fix Signed-off-by: Kevin Su <pingsutw@apache.org> --------- Signed-off-by: Kevin Su <pingsutw@apache.org> * feat(idl/actions): Add Actions message and change EnqueueRequest to use it (#6916) * feat(actions): introduce generic Action message Replace the previous Run-centric enqueue types with a generic Action message that represents units of work (task, trace, or condition). Reorder and renumber fields for the new Action message, adding clear inline comments for task/trace/condition specs and moving validation constraints accordingly. Add a compact EnqueueRequest that wraps Action and restores the optional RunSpec for propagation to downstream actions. Improve WatchForUpdatesResponse comments to clarify action_update vs control_message and fix minor formatting. These changes enable more flexible action types beyond runs and improve API clarity for enqueuing and update streaming. Signed-off-by: Sergey Vilgelm <sergey@union.ai> * feat(proto): update actions_service generated files Action message and refactor Enqueue/Update descriptors - Replace previous serialized descriptor blob for actions_service.proto with an updated serialized file that introduces a new top-level Action message and shifts several fields/indices (action_id, parent_action_name, input_uri, run_output_base, group, subject, task, trace, condition, spec). This reflects a proto schema change in actions_service.proto. - Adjust Enqueue, Update, GetLatestState, WatchForUpdates and Abort service descriptors to align with the regenerated proto bytes. - Regenerate the generated Python module gen/python/flyteidl2/actions/ actions_service_pb2.py to incorporate the new descriptor contents, ensuring Python bindings match the updated proto definitions. Why: - The proto schema for Flyte actions changed (new Action message / reorganized fields). Regenerating the Python protobuf ensures code uses the correct wire format and field layout for serialization / deserialization and stays in sync with the updated .proto. Signed-off-by: Sergey Vilgelm <sergey@union.ai> * fix(proto): simplify Action message comment and correct wording Clarify the Action protobuf comment to be more concise and correct the wording. Replace the longer explanation with a shorter line that lists the kinds of things an Action can represent (task executions, traces, and conditions). This improves readability and avoids an awkward sentence construction. Signed-off-by: Sergey Vilgelm <sergey@union.ai> * fix(protocol): rename WatchForUpdatesResponse comment Update the proto comment for WatchForUpdatesResponse to correctly describe the message as a response for watching action state updates. This fixes a misleading comment that previously named it WatchForUpdatesRequest, improving clarity and aligning documentation with the message name. Signed-off-by: Sergey Vilgelm <sergey@union.ai> * generate Update generated protobuf/TS/Rust code comments to clarify and fix inaccurate message descriptions introduced by code generation. - Change Action comment to a concise description: "Action represents a unit of work to be executed. Theses can be task executions, traces, or conditions." to avoid the misleading "more generic version of the Run concept" phrasing. - Rename WatchForUpdatesRequest comment to WatchForUpdatesResponse in generated Go and TS to match the actual message type and purpose. - Trim a truncated Rust doc comment for Action to reflect the corrected description. These edits are limited to generated files' comments to improve readability and correctness of the inline documentation without changing runtime behavior. Signed-off-by: Sergey Vilgelm <sergey@union.ai> * docs(proto): clarify Action differs from workflow Action Add a clarifying comment to the Action message in the actions_service.proto file to explicitly note that this Action is distinct from the Action protobuf defined in flyteidl2/workflow/run_definition.proto. This prevents confusion for developers reading the proto files and helps maintainers avoid accidental mixing of the two conceptually similar but separate message types. Signed-off-by: Sergey Vilgelm <sergey@union.ai> * geenrate Add a note to generated TypeScript, Go, and Rust protobuf files clarifying that the flyteidl2.actions.Action type is distinct from the Action protobuf declared in flyteidl2/workflow/run_definition.proto. This change prevents confusion when developers read generated code by explicitly calling out the difference. It also updates the Rust file's embedded FILE_DESCRIPTOR_SET bytes (regenerated content) to reflect the new generated output. Signed-off-by: Sergey Vilgelm <sergey@union.ai> --------- Signed-off-by: Sergey Vilgelm <sergey@union.ai> * Add permission action proto (#6898) * change proto Signed-off-by: Alex Wu <c.alexwu@gmail.com> * fix proto Signed-off-by: Alex Wu <c.alexwu@gmail.com> * re generate idl Signed-off-by: Alex Wu <c.alexwu@gmail.com> --------- Signed-off-by: Alex Wu <c.alexwu@gmail.com> * Fix vet issues (#6901) * Fix vet issues Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * Add git-town tracking Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * gen Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * fix: use GetRunAs with nil checker Signed-off-by: machichima <nary12321@gmail.com> * fix: remove Cargo.lock Signed-off-by: machichima <nary12321@gmail.com> * fix: local fix cherry pick state etcd Signed-off-by: machichima <nary12321@gmail.com> --------- Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> Signed-off-by: machichima <nary12321@gmail.com> Co-authored-by: machichima <nary12321@gmail.com> * [V2] Migrate action service (#6949) Signed-off-by: machichima <nary12321@gmail.com> Signed-off-by: Kevin Su <pingsutw@apache.org> Co-authored-by: Kevin Su <pingsutw@apache.org> * fix: handle task execution errors in k8s plugin resource phase check Read error outputs from the output reader and translate them into appropriate phase transitions (retryable or permanent failure) before writing outputs, preventing silent error swallowing. Signed-off-by: Kevin Su <pingsutw@gmail.com> Signed-off-by: Kevin Su <pingsutw@apache.org> * Add source_dir to UVProject proto (#6939) Signed-off-by: Kevin Su <pingsutw@apache.org> * feat(flyteidl2): add image_build_url field to TaskMetadata proto (#6951) * image build url added Signed-off-by: M. Adil Fayyaz <62440954+AdilFayyaz@users.noreply.github.com> * update Signed-off-by: M. Adil Fayyaz <62440954+AdilFayyaz@users.noreply.github.com> * fix Signed-off-by: M. Adil Fayyaz <62440954+AdilFayyaz@users.noreply.github.com> * fix: typo Signed-off-by: M. Adil Fayyaz <62440954+AdilFayyaz@users.noreply.github.com> * gen incl Signed-off-by: M. Adil Fayyaz <62440954+AdilFayyaz@users.noreply.github.com> --------- Signed-off-by: M. Adil Fayyaz <62440954+AdilFayyaz@users.noreply.github.com> * Fix duplicate VolumeMounts when merging pod templates (#6958) Signed-off-by: Kevin Su <pingsutw@apache.org> * Add label for gpu partitions used (#6921) * improved readme for 2.0 Signed-off-by: Ketan Umare <kumare3@users.noreply.github.com> * [flyteidl2] Generate mocks for Actions service (#6962) * gen actions mocks Signed-off-by: Paul Dittamo <pvdittamo@gmail.com> * gen action service mocks Signed-off-by: Paul Dittamo <pvdittamo@gmail.com> * clean up mockery yaml Signed-off-by: Paul Dittamo <pvdittamo@gmail.com> --------- Signed-off-by: Paul Dittamo <pvdittamo@gmail.com> * Bump OTel SDK to v1.40.0 and fix semconv schema conflict (#6965) * Upgrade OpenTelemetry SDK to v1.40.0 Update OTel SDK packages from v1.37.0 to v1.40.0 and semconv from v1.24.0 to v1.39.0 to address CVE-2025-22870 in transitive dependencies. Updates both direct dependencies and replace directives. Signed-off-by: J.J. Montgomery <monty@union.ai> * make gen Signed-off-by: J.J. Montgomery <monty@union.ai> --------- Signed-off-by: J.J. Montgomery <monty@union.ai> * [V2] Add Internal run service (#6956) * feat: add internal run service Signed-off-by: machichima <nary12321@gmail.com> * fix: store end time in action table Signed-off-by: machichima <nary12321@gmail.com> * feat: init internal run service Signed-off-by: machichima <nary12321@gmail.com> * fix: direclty use ActionPhase enum and store int in DB Signed-off-by: machichima <nary12321@gmail.com> * fix: go through ToK8sResourceRequirements to get empty resources rather than early return, which cause error as Resources is nil, otherwise we will fail on GetResources() == nil check in container_helper.go Signed-off-by: machichima <nary12321@gmail.com> * feat: add ActinEvent table & InsertEvent method Signed-off-by: machichima <nary12321@gmail.com> * fix: UpdateActionPhase use ActinPhase enum as args Signed-off-by: machichima <nary12321@gmail.com> * feat: call run service internal from action client Signed-off-by: machichima <nary12321@gmail.com> * fix: run notifyRunService in goroutine and add TODO Signed-off-by: machichima <nary12321@gmail.com> * feat: run service watch/update DB only Signed-off-by: machichima <nary12321@gmail.com> * fix: call RecordAction when new CR created Signed-off-by: machichima <nary12321@gmail.com> * fix: watch actoin update do not filter out root action (no parent action name) Signed-off-by: machichima <nary12321@gmail.com> * fix: watch before list Signed-off-by: machichima <nary12321@gmail.com> * fix: send existing actions from all pages Signed-off-by: machichima <nary12321@gmail.com> --------- Signed-off-by: machichima <nary12321@gmail.com> * Revert "Bump OTel SDK to v1.40.0 and fix semconv schema conflict (#6965)" (#6967) This reverts commit 7dee1fbc4ede90e6c65c937e8cc4c1e64679d774. Signed-off-by: J.J. Montgomery <monty@union.ai> * Add proto files for domain service (#6891) Signed-off-by: Adam Brock <adam@union.ai> Co-authored-by: Adam Brock <adam@union.ai> * Add OppoBloomFilter to deduplicate RecordAction calls (#6987) Signed-off-by: Kevin Su <pingsutw@apache.org> * fix: correct TaskAction CR/pod names and namespaces, improve run ID generation (#6986) * update Signed-off-by: M. Adil Fayyaz <62440954+AdilFayyaz@users.noreply.github.com> * tests Signed-off-by: M. Adil Fayyaz <62440954+AdilFayyaz@users.noreply.github.com> * fix Signed-off-by: M. Adil Fayyaz <62440954+AdilFayyaz@users.noreply.github.com> * remove lock file Signed-off-by: Kevin Su <pingsutw@apache.org> --------- Signed-off-by: M. Adil Fayyaz <62440954+AdilFayyaz@users.noreply.github.com> Signed-off-by: Kevin Su <pingsutw@apache.org> Co-authored-by: Kevin Su <pingsutw@apache.org> * fix: set explicit 5-minute timeout on TaskAction watch (#6996) Signed-off-by: Kevin Su <pingsutw@apache.org> * [V2]Sandbox for flyte v2 (#6955) * add helm chart Signed-off-by: Alex Wu <c.alexwu@gmail.com> * add docker folder for running k3s cluster Signed-off-by: Alex Wu <c.alexwu@gmail.com> * delete queue state service add actions service Signed-off-by: Alex Wu <c.alexwu@gmail.com> * delete queue state service add actions service Signed-off-by: Alex Wu <c.alexwu@gmail.com> * patch k8s secret config Signed-off-by: Alex Wu <c.alexwu@gmail.com> * fix unimplement rpc issue Signed-off-by: Alex Wu <c.alexwu@gmail.com> * fix k8s dashboard conflict issue Signed-off-by: Alex Wu <c.alexwu@gmail.com> * add normalize path function to std lib Signed-off-by: Alex Wu <c.alexwu@gmail.com> * add sandbox-stop command and fix sandbox command name Signed-off-by: Alex Wu <c.alexwu@gmail.com> * remove cargo file in gen folder Signed-off-by: Alex Wu <c.alexwu@gmail.com> * fix doc flyte version Signed-off-by: Alex Wu <c.alexwu@gmail.com> * delete connector config in configmap Signed-off-by: Alex Wu <c.alexwu@gmail.com> * add sqlite config Signed-off-by: Alex Wu <c.alexwu@gmail.com> * delete service annotation config Signed-off-by: Alex Wu <c.alexwu@gmail.com> * fix ingress config Signed-off-by: Alex Wu <c.alexwu@gmail.com> * regen manifests Signed-off-by: Alex Wu <c.alexwu@gmail.com> * fix some docker script error Signed-off-by: Alex Wu <c.alexwu@gmail.com> * regenerate manifests Signed-off-by: Alex Wu <c.alexwu@gmail.com> * fix dicker image name Signed-off-by: Alex Wu <c.alexwu@gmail.com> * add kube folder to gitignore Signed-off-by: Alex Wu <c.alexwu@gmail.com> * remove test make script under sandbox bundle folder Signed-off-by: Alex Wu <c.alexwu@gmail.com> * fix test cases Signed-off-by: Alex Wu <c.alexwu@gmail.com> * delete cargo.lock Signed-off-by: Alex Wu <c.alexwu@gmail.com> * regenerate proto Signed-off-by: Alex Wu <c.alexwu@gmail.com> * fix gen diff check rule Signed-off-by: Alex Wu <c.alexwu@gmail.com> * fix gen check rule to exclude Cargo Signed-off-by: Alex Wu <c.alexwu@gmail.com> * delete complete-connector folder Signed-off-by: Alex Wu <c.alexwu@gmail.com> * fix sandbox cluster name Signed-off-by: Alex Wu <c.alexwu@gmail.com> --------- Signed-off-by: Alex Wu <c.alexwu@gmail.com> * [V2][Feat] Add events service (#6970) * feat: add EventsService proto Signed-off-by: machichima <nary12321@gmail.com> * feat: add event background worker Signed-off-by: machichima <nary12321@gmail.com> * feat: add events service Signed-off-by: machichima <nary12321@gmail.com> * feat: add to main.go Signed-off-by: machichima <nary12321@gmail.com> * refactor: cleanup executor cmd main.go Signed-off-by: machichima <nary12321@gmail.com> * feat: call eventsClient.Record in executor when having update Signed-off-by: machichima <nary12321@gmail.com> * docs: update development guide Signed-off-by: machichima <nary12321@gmail.com> * feat: add scripts Signed-off-by: machichima <nary12321@gmail.com> * feat: events record use direct RPC req & remove background worker Signed-off-by: machichima <nary12321@gmail.com> * refactor: rename to events proxy service proto Signed-off-by: machichima <nary12321@gmail.com> * refactor: rename to events proxy service Signed-off-by: machichima <nary12321@gmail.com> * refactor: StateServiceURL to EventsServiceURL Signed-off-by: machichima <nary12321@gmail.com> * fix: raise error when fail to persist action event Signed-off-by: machichima <nary12321@gmail.com> * fix: remove redundant if condition Signed-off-by: machichima <nary12321@gmail.com> * fix: update status after persisted Signed-off-by: machichima <nary12321@gmail.com> * fix: remove unused code Signed-off-by: machichima <nary12321@gmail.com> * fix: bug + remove cluster command Signed-off-by: machichima <nary12321@gmail.com> --------- Signed-off-by: machichima <nary12321@gmail.com> * Add IdentityService (UserInfo) stub to runs service (#6997) Signed-off-by: Kevin Su <pingsutw@apache.org> * [V2] Add GitHub Actions workflow for Go unit tests (#6999) Signed-off-by: Kevin Su <pingsutw@apache.org> * [V2] Add project service (#7000) Signed-off-by: machichima <nary12321@gmail.com> * [V2] Add TaskAction garbage collector for terminal CRDs (#6994) Signed-off-by: Kevin Su <pingsutw@apache.org> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> * Add include_system_secrets to secrets apis (#7007) * Add TaskAction validation before finalizer registration (#7006) * add: validation Signed-off-by: M. Adil Fayyaz <62440954+AdilFayyaz@users.noreply.github.com> * fix Signed-off-by: M. Adil Fayyaz <62440954+AdilFayyaz@users.noreply.github.com> * [V2] Add GitHub Actions workflow for Go unit tests (#6999) Signed-off-by: Kevin Su <pingsutw@apache.org> * add: validation Signed-off-by: M. Adil Fayyaz <62440954+AdilFayyaz@users.noreply.github.com> * fix Signed-off-by: M. Adil Fayyaz <62440954+AdilFayyaz@users.noreply.github.com> * minor fix Signed-off-by: M. Adil Fayyaz <62440954+AdilFayyaz@users.noreply.github.com> * fix: tests Signed-off-by: M. Adil Fayyaz <62440954+AdilFayyaz@users.noreply.github.com> --------- Signed-off-by: M. Adil Fayyaz <62440954+AdilFayyaz@users.noreply.github.com> Signed-off-by: Kevin Su <pingsutw@apache.org> Co-authored-by: Kevin Su <pingsutw@apache.org> * Notifications IDL (#6946) * Notifications IDL Signed-off-by: Iaroslav Ciupin <iaroslav@union.ai> * Fix Docker build for CGO-dependent sqlite3 package (#7014) * Fix Docker build for CGO-dependent sqlite3 package The build was failing because `go-sqlite3` requires CGO, but the Dockerfile used cross-compilation (`--platform=${BUILDPLATFORM}`) which disables CGO. Additionally, the .dockerignore negation pattern for `gen/go` was unreliable when the parent `gen/` directory was excluded. - Remove cross-compilation, enable CGO_ENABLED=1 for native build - Copy only gen/go instead of full gen directory - Replace .dockerignore negation with explicit subdirectory exclusions Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Kevin Su <pingsutw@apache.org> * test Signed-off-by: Kevin Su <pingsutw@apache.org> --------- Signed-off-by: Kevin Su <pingsutw@apache.org> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> * notification event type IDL (#7021) Signed-off-by: Iaroslav Ciupin <iaroslav@union.ai> * Add DisableConfigEndpoint option to profutils Config to allow disabling the /config HTTP endpoint. Add pflags-generated config files and fix import paths to use non-versioned module. (#7015) * Add DisableConfigEndpoint option to profutils Config to allow disabling the /config HTTP endpoint. Add pflags-generated config files and fix import paths to use non-versioned module. Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * fix imports Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * Fix tests Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * Fix config registration Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * Fix unit tests Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> --------- Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * [Sandbox][RunService]Switch database from sqlite to postgresql (#7018) * [V2] Run Service - Watch action cache (#7017) * feat: simple state manager to store actions Signed-off-by: machichima <nary12321@gmail.com> * feat: run state manager record node tree Signed-off-by: machichima <nary12321@gmail.com> * fix: also include root action in ListActions Signed-off-by: machichima <nary12321@gmail.com> * fix: prevent error when create default project on startup Signed-off-by: machichima <nary12321@gmail.com> * feat: add watch_actions.sh test script Signed-off-by: machichima <nary12321@gmail.com> * fix: continue for nil enrichedAction for defense Signed-off-by: machichima <nary12321@gmail.com> * fix: raise error in CreateProject when already exists Signed-off-by: machichima <nary12321@gmail.com> --------- Signed-off-by: machichima <nary12321@gmail.com> * Add new role ROLE_TYPE_SYSTEM_PROVISIONED_ACCESS for default system behavior (#7028) Signed-off-by: Katrina Rogan <katroganGH@gmail.com> * minio patch (#7027) Signed-off-by: M. Adil Fayyaz <62440954+AdilFayyaz@users.noreply.github.com> * Add log links support to task template (#7030) Signed-off-by: Kevin Su <pingsutw@apache.org> * feat(runs): wire AbortRun and AbortAction through to ActionsService (#7031) * abort - add Signed-off-by: M. Adil Fayyaz <62440954+AdilFayyaz@users.noreply.github.com> * update Signed-off-by: M. Adil Fayyaz <62440954+AdilFayyaz@users.noreply.github.com> * update test Signed-off-by: M. Adil Fayyaz <62440954+AdilFayyaz@users.noreply.github.com> --------- Signed-off-by: M. Adil Fayyaz <62440954+AdilFayyaz@users.noreply.github.com> * Fix problems including missing charts and the kubeconfig permission (#7033) * fix problems about missing charts and kubeconfig permission Signed-off-by: yuteng <a08h0283@gmail.com> * make dep_build including updating helm-repo and build charts Signed-off-by: yuteng <a08h0283@gmail.com> * deleted unused operator Signed-off-by: yuteng <a08h0283@gmail.com> --------- Signed-off-by: yuteng <a08h0283@gmail.com> * Adjust notifications IDL to support notifications per phases (#7025) * adjust notifications IDL to support notifications per phases * drop delivery config id & rule id * email idl adjustments Signed-off-by: Iaroslav Ciupin <iaroslav@union.ai> * Add node_name to the container selector (#7036) To facilitate getting logs just for a single node, add an optional field to the ContainerSelector message. Signed-off-by: Eugene Yakubovich <eyakubovich@gmail.com> * Migrate #6785 in v1 to v2 (#6918) Signed-off-by: Barry Wu <a0987818905@gmail.com> * feat+test: write empty input Literal if no input (#7055) Signed-off-by: machichima <nary12321@gmail.com> * Ensure GetActionDetails works expected (#7022) Signed-off-by: Kevin Su <pingsutw@apache.org> * Fix: [Run Service] Ensure GetActionData works expected (#6991) Signed-off-by: SharpKoi <sharpkoi1222@gmail.com> Signed-off-by: Kevin Su <pingsutw@apache.org> * [Flyte 6973] Attached action details to `ListRuns` response (#7026) Signed-off-by: yuteng <a08h0283@gmail.com> Signed-off-by: Kevin Su <pingsutw@apache.org> Co-authored-by: Kevin Su <pingsutw@apache.org> * [V2]Sandbox webhook (#7034) Signed-off-by: Alex Wu <c.alexwu@gmail.com> * [V2]Add secret service (#7039) * implement secret service Signed-off-by: Alex Wu <c.alexwu@gmail.com> * insert cluster status into secret response Signed-off-by: Alex Wu <c.alexwu@gmail.com> * minor fix Signed-off-by: Alex Wu <c.alexwu@gmail.com> * use project-domain as secret namespace Signed-off-by: Alex Wu <c.alexwu@gmail.com> * regenerate helm Signed-off-by: Alex Wu <c.alexwu@gmail.com> * rebuild Signed-off-by: Alex Wu <c.alexwu@gmail.com> --------- Signed-off-by: Alex Wu <c.alexwu@gmail.com> * fix: add ProjectService to envoy proxy (#7059) Signed-off-by: machichima <nary12321@gmail.com> * Fix CreateRun response, output URI, and GetActionData (#7064) Signed-off-by: Kevin Su <pingsutw@apache.org> * [V2][Bug] Fix sandbox dev mode dir permission error (#7061) Signed-off-by: Alex Wu <c.alexwu@gmail.com> * Add IsEntryPoint and CodeBundlePath to taskTemplate metadata (#7071) * Add IsEntryPoint flag to TaskTemplate metadata Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * Entrypoint Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * Add CodeBundle to tasks.proto Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * Rename code_bundle_uri Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * Show generation times for make gen Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> --------- Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * [V2][Bug] Fix WatchActionDetails to return full action details in streamed updates (#7070) Signed-off-by: Kevin Su <pingsutw@apache.org> * Fix manager hanging on shutdown (#7057) Signed-off-by: Kevin Su <pingsutw@apache.org> * [V2] add dummy trigger and app service (#7077) * feat: add dummy trigger service Signed-off-by: machichima <nary12321@gmail.com> * feat: add dummy app service Signed-off-by: machichima <nary12321@gmail.com> --------- Signed-off-by: machichima <nary12321@gmail.com> * settings proto definitions (#7049) generated files and lint fixes Signed-off-by: Adam Brock <adam@union.ai> Co-authored-by: Adam Brock <adam@union.ai> * [V2] Add attempt and cache status (#7078) * feat: add attempts and cache status to CR Signed-off-by: machichima <nary12321@gmail.com> * feat: get attempts/cache status from object Signed-off-by: machichima <nary12321@gmail.com> * feat: store attempts/cache status to CR and send through action event Signed-off-by: machichima <nary12321@gmail.com> * feat: consider retry count in generated name Signed-off-by: machichima <nary12321@gmail.com> * feat: get max attempt from task template Signed-off-by: machichima <nary12321@gmail.com> * feat: run service save attempts/cache status to DB Signed-off-by: machichima <nary12321@gmail.com> * build: make sandbox-build Signed-off-by: machichima <nary12321@gmail.com> --------- Signed-off-by: machichima <nary12321@gmail.com> * [V2][Sandbox]Add console to sandbox (#7074) Signed-off-by: Alex Wu <c.alexwu@gmail.com> * [V2] Add Cache service (#7035) * feat: add CacheKey to TaskAction spec Signed-off-by: machichima <nary12321@gmail.com> * feat: add cache service skeleton Signed-off-by: machichima <nary12321@gmail.com> * feat: add cache manager and repository Signed-off-by: machichima <nary12321@gmail.com> * feat: add cache service client Signed-off-by: machichima <nary12321@gmail.com> * feat: add max cache age Signed-off-by: machichima <nary12321@gmail.com> * feat: add cache to task action reconcile loop Signed-off-by: machichima <nary12321@gmail.com> * fix: better reservation cleanup Signed-off-by: machichima <nary12321@gmail.com> * fix: calculate hash key always go through hash formula Signed-off-by: machichima <nary12321@gmail.com> * feat: start cache service in manager/ Signed-off-by: machichima <nary12321@gmail.com> * feat: copy cache service in root Dockerfile Signed-off-by: machichima <nary12321@gmail.com> * fix: use promutils.NewScope Signed-off-by: machichima <nary12321@gmail.com> * fix: remove getStringValue() Signed-off-by: machichima <nary12321@gmail.com> * fix: correcct serialize logic Signed-off-by: machichima <nary12321@gmail.com> * refactor: update function/variable name Signed-off-by: machichima <nary12321@gmail.com> * fix: set metricKey only once Signed-off-by: machichima <nary12321@gmail.com> * fix: heartbeatGracePeriodMultiplier to int type Signed-off-by: machichima <nary12321@gmail.com> * docs: update comment to make them more clear Signed-off-by: machichima <nary12321@gmail.com> * fix: get newest data again before return Signed-off-by: machichima <nary12321@gmail.com> * fix: construct new struct rather than deep copy Signed-off-by: machichima <nary12321@gmail.com> --------- Signed-off-by: machichima <nary12321@gmail.com> * [V2] Run Service - Add the field that GetRunDetails expected (#6977) (#7038) * feat: Add the field that GetRunDetails works expected Signed-off-by: SZL741023 <shengzhe.lin@icloud.com> * fix: fixup after merge v2 Signed-off-by: machichima <nary12321@gmail.com> * feat: add get run details test script Signed-off-by: machichima <nary12321@gmail.com> * feat: add the cache status for run details Signed-off-by: SZL741023 <shengzhe.lin@icloud.com> * fix: GetRunDetails call buildActionDetails Signed-off-by: machichima <nary12321@gmail.com> * fix: fallback to action table spec & use event as source of truth Signed-off-by: machichima <nary12321@gmail.com> * test: set Attempts to 1 Signed-off-by: machichima <nary12321@gmail.com> * fix: get cache status from DB Signed-off-by: machichima <nary12321@gmail.com> * test: set default org to localhost Signed-off-by: machichima <nary12321@gmail.com> * test: mock match error Signed-off-by: machichima <nary12321@gmail.com> * refactor: specJSON to specProto Signed-off-by: machichima <nary12321@gmail.com> --------- Signed-off-by: SZL741023 <shengzhe.lin@icloud.com> Signed-off-by: machichima <nary12321@gmail.com> Co-authored-by: machichima <nary12321@gmail.com> * [V2]Add sandbox image auto iterate CI (#7063) Signed-off-by: Alex Wu <c.alexwu@gmail.com> * feat: add start/end time + duration + metadata (#7081) Signed-off-by: machichima <nary12321@gmail.com> * feat: add app and trigger service to proxy (#7082) Signed-off-by: machichima <nary12321@gmail.com> * feat: update gc and log level config (#7083) Signed-off-by: machichima <nary12321@gmail.com> * [V2] Read user run context (#7084) Signed-off-by: machichima <nary12321@gmail.com> Signed-off-by: Kevin Su <pingsutw@apache.org> Co-authored-by: Kevin Su <pingsutw@apache.org> * [V2] Add RunLogsService for str…
1 parent f0cf961 commit 1dd0464

1,765 files changed

Lines changed: 487617 additions & 1 deletion

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

.dockerignore

Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,39 @@
1+
# Git
2+
.git
3+
.gitignore
4+
.gitattributes
5+
6+
# CI/CD
7+
.github
8+
9+
# Generated files
10+
gen/python/
11+
gen/rust/
12+
gen/ts/
13+
14+
# Dependencies
15+
node_modules/
16+
target/
17+
**/Cargo.lock
18+
*.pyc
19+
__pycache__/
20+
.kube/
21+
22+
# IDE
23+
.vscode/
24+
.idea/
25+
*.swp
26+
*.swo
27+
*~
28+
29+
# OS
30+
.DS_Store
31+
Thumbs.db
32+
33+
# Logs
34+
*.log
35+
36+
# Test coverage
37+
coverage/
38+
*.coverage
39+
.pytest_cache/

.gitattributes

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
# Mark generated protobuf code so GitHub hides it in diffs by default
2+
gen/** linguist-generated=true
3+
4+
# Mark generated mocks so GitHub hides them in diffs by default
5+
**/mocks/mocks.go linguist-generated=true

.github/PULL_REQUEST_TEMPLATE.md

Lines changed: 71 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,71 @@
1+
## Tracking issue
2+
<!--
3+
If your PR fixes an open issue, use `Closes #999` to link your PR with the issue.
4+
Example: Closes #999
5+
6+
If your PR is related to an issue or PR, use `Related to #999` to link your PR.
7+
Example: Related to #999
8+
9+
Remove this section if not applicable
10+
-->
11+
12+
## Why are the changes needed?
13+
14+
<!--
15+
Please clarify why the changes are needed. For instance,
16+
1. If you propose a new API, clarify the use case for a new API.
17+
2. If you fix a bug, you can clarify why it is a bug.
18+
-->
19+
20+
## What changes were proposed in this pull request?
21+
22+
<!--
23+
Please clarify what changes you are proposing. The purpose of this section is to outline the changes and how this PR fixes the issue.
24+
If possible, please consider writing useful notes for better and faster reviews in your PR. See the examples below.
25+
1. If you refactor some codes with changing classes, showing the class hierarchy will help reviewers.
26+
2. If there is design documentation, please add the link.
27+
-->
28+
29+
## How was this patch tested?
30+
31+
<!--
32+
If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
33+
If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
34+
If tests were not added, please describe why they were not added and/or why it was difficult to add.
35+
-->
36+
### Labels
37+
38+
Please add one or more of the following labels to categorize your PR:
39+
- **added**: For new features.
40+
- **changed**: For changes in existing functionality.
41+
- **deprecated**: For soon-to-be-removed features.
42+
- **removed**: For features being removed.
43+
- **fixed**: For any bug fixed.
44+
- **security**: In case of vulnerabilities
45+
46+
This is important to improve the readability of release notes.
47+
48+
### Setup process
49+
50+
### Screenshots
51+
52+
## Check all the applicable boxes <!-- Follow the above conventions to check the box -->
53+
54+
- [ ] I updated the documentation accordingly.
55+
- [ ] All new and existing tests passed.
56+
- [ ] All commits are signed-off.
57+
58+
## Related PRs
59+
60+
<!-- Add related pull requests for reviewers to check -->
61+
62+
## Stack
63+
64+
If you do use [`git town`](https://www.git-town.com/introduction) to manage PR Stacks, the stack relevant to this PR
65+
will show below. Otherwise, you can ignore this section.
66+
67+
<!-- branch-stack -->
68+
69+
## Docs link
70+
71+
<!-- Add documentation link built by CI jobs here, and specify the changed place -->
Lines changed: 47 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,47 @@
1+
name: "Setup Python Environment"
2+
description: "Set up Python environment for the given Python version"
3+
4+
inputs:
5+
python-version:
6+
description: "Python version to use"
7+
required: true
8+
default: "3.12"
9+
uv-version:
10+
description: "uv version to use"
11+
required: true
12+
default: "0.8.4"
13+
working-directory:
14+
description: "Default working directory for all steps"
15+
required: false
16+
default: ""
17+
18+
runs:
19+
using: "composite"
20+
steps:
21+
- uses: actions/setup-python@v5
22+
with:
23+
python-version: ${{ inputs.python-version }}
24+
25+
- name: Install uv
26+
uses: astral-sh/setup-uv@v6
27+
id: setup-uv
28+
with:
29+
version: ${{ inputs.uv-version }}
30+
enable-cache: 'true'
31+
cache-suffix: ${{ inputs.python-version }}
32+
cache-dependency-glob: |
33+
pyproject.toml
34+
uv.lock
35+
working-directory: ${{ inputs.working-directory }}
36+
37+
- name: Install Python dependencies
38+
if: steps.setup-uv.outputs.cache-hit == 'false'
39+
working-directory: ${{ inputs.working-directory }}
40+
run: uv sync --all-groups --frozen
41+
shell: bash
42+
43+
- name: List Python dependencies
44+
if: steps.setup-uv.outputs.cache-hit == 'true'
45+
working-directory: ${{ inputs.working-directory }}
46+
run: uv pip list
47+
shell: bash

.github/dco.yml

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
require:
2+
members: false
3+
allowRemediationCommits:
4+
individual: true
5+
thirdParty: true
Lines changed: 158 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,158 @@
1+
name: Build and Publish CI Docker Image
2+
3+
on:
4+
push:
5+
branches:
6+
- main
7+
- v2
8+
paths:
9+
- 'gen.Dockerfile'
10+
- '.github/workflows/build-ci-image.yml'
11+
pull_request:
12+
paths:
13+
- 'gen.Dockerfile'
14+
- '.github/workflows/build-ci-image.yml'
15+
workflow_dispatch:
16+
inputs:
17+
force_rebuild:
18+
description: 'Force rebuild of the image'
19+
required: false
20+
default: 'false'
21+
22+
env:
23+
REGISTRY: ghcr.io
24+
IMAGE_NAME: ${{ github.repository }}/ci
25+
26+
jobs:
27+
build:
28+
runs-on: ubuntu-latest
29+
permissions:
30+
contents: read
31+
packages: write
32+
pull-requests: write
33+
34+
outputs:
35+
image-tag: ${{ steps.set-tag.outputs.tag }}
36+
37+
steps:
38+
- name: Checkout code
39+
uses: actions/checkout@v4
40+
41+
- name: Set up Docker Buildx
42+
uses: docker/setup-buildx-action@v3
43+
44+
- name: Log in to GitHub Container Registry
45+
uses: docker/login-action@v3
46+
with:
47+
registry: ${{ env.REGISTRY }}
48+
username: ${{ github.actor }}
49+
password: ${{ secrets.GITHUB_TOKEN }}
50+
51+
- name: Extract metadata (tags, labels)
52+
id: meta
53+
uses: docker/metadata-action@v5
54+
with:
55+
images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}
56+
tags: |
57+
# Use branch name for branch pushes
58+
type=ref,event=branch
59+
# Use PR number for pull requests
60+
type=ref,event=pr
61+
# Use 'latest' tag for main branch
62+
type=raw,value=latest,enable=${{ github.ref == 'refs/heads/main' }}
63+
# Use 'v2' tag for v2 branch
64+
type=raw,value=v2,enable=${{ github.ref == 'refs/heads/v2' }}
65+
# Add git sha as tag
66+
type=sha,prefix=${{ github.head_ref }}-,enable=${{ github.ref != 'refs/heads/v2' && github.ref != 'refs/heads/main' }}
67+
68+
- name: Set image tag output
69+
id: set-tag
70+
run: |
71+
if [ "${{ github.event_name }}" == "pull_request" ]; then
72+
echo "tag=pr-${{ github.event.pull_request.number }}" >> $GITHUB_OUTPUT
73+
elif [ "${{ github.ref }}" == "refs/heads/v2" ]; then
74+
echo "tag=v2" >> $GITHUB_OUTPUT
75+
elif [ "${{ github.ref }}" == "refs/heads/main" ]; then
76+
echo "tag=latest" >> $GITHUB_OUTPUT
77+
else
78+
echo "tag=$(echo ${{ github.ref }} | sed 's/refs\/heads\///')" >> $GITHUB_OUTPUT
79+
fi
80+
81+
- name: Build and push Docker image
82+
uses: docker/build-push-action@v6
83+
with:
84+
context: .
85+
file: ./gen.Dockerfile
86+
push: true
87+
tags: ${{ steps.meta.outputs.tags }}
88+
labels: ${{ steps.meta.outputs.labels }}
89+
platforms: linux/amd64,linux/arm64
90+
# Multi-layer caching strategy for speed
91+
cache-from: |
92+
type=registry,ref=${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:buildcache
93+
type=registry,ref=${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:${{ steps.set-tag.outputs.tag }}
94+
type=registry,ref=${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:v2
95+
type=gha
96+
cache-to: |
97+
type=registry,ref=${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:buildcache,mode=max
98+
type=gha,mode=max
99+
# Enable BuildKit features for better caching
100+
build-args: |
101+
BUILDKIT_INLINE_CACHE=1
102+
103+
- name: Image digest
104+
run: echo "Image built with digest ${{ steps.meta.outputs.digest }}"
105+
106+
- name: Comment on PR with image tag
107+
if: github.event_name == 'pull_request'
108+
uses: actions/github-script@v7
109+
with:
110+
script: |
111+
const tag = '${{ steps.set-tag.outputs.tag }}';
112+
const registry = '${{ env.REGISTRY }}';
113+
const imageName = '${{ env.IMAGE_NAME }}';
114+
const fullImage = `${registry}/${imageName}:${tag}`;
115+
116+
const comment = `## 🐳 Docker CI Image Built
117+
118+
The CI Docker image has been built and pushed for this PR!
119+
120+
**Image:** \`${fullImage}\`
121+
122+
This image will be automatically used by CI workflows in this PR.
123+
124+
To test locally:
125+
\`\`\`bash
126+
make gen DOCKER_CI_IMAGE=${fullImage}
127+
\`\`\`
128+
`;
129+
130+
// Find existing comment
131+
const { data: comments } = await github.rest.issues.listComments({
132+
owner: context.repo.owner,
133+
repo: context.repo.repo,
134+
issue_number: context.issue.number,
135+
});
136+
137+
const botComment = comments.find(comment =>
138+
comment.user.type === 'Bot' &&
139+
comment.body.includes('🐳 Docker CI Image Built')
140+
);
141+
142+
if (botComment) {
143+
// Update existing comment
144+
await github.rest.issues.updateComment({
145+
owner: context.repo.owner,
146+
repo: context.repo.repo,
147+
comment_id: botComment.id,
148+
body: comment
149+
});
150+
} else {
151+
// Create new comment
152+
await github.rest.issues.createComment({
153+
owner: context.repo.owner,
154+
repo: context.repo.repo,
155+
issue_number: context.issue.number,
156+
body: comment
157+
});
158+
}

0 commit comments

Comments
 (0)