We've got a little problem when using v2 sync with GitHub sync, as users are now able to do (see #4634)
I unfortunately forgot during development that v2 sync is only suitable for pulling one project per branch from lightning
This is because there's only one workflows folder supported.
So if I set up a v2 sync, and push to github from the app, it'll update workflows on the target branch.
But if I create a sandbox, and sync that to main, then when the sandbox syncs back to github it'll override the main project. If I then sync main, it'll override again to look like main. It doesn't work, basically.
This stuff works great if you're using two branches on github to maintain two versions of the project. Your main and dev are on different branches, and you can diff the workflows folders to understand what's different. It's super hoopy.
It also works great if I have a one-way snyc (which we don't support). Let's say I have on project on main, and when I push changes to the GH repo, I want to deploy the updates to 3 other projects. Those pull-only syncs work great - they use the same version of workflows to update the destination project.
So solutions here:
- As a first pass we should probably just show a carefully worded warning to the user when creating the sync
- We might be able to detect the issue on init and show a warning (we'd have to know there's a sync already set on the same branch - which we can tell from the DB right?)
- We can press on with the V2 github sync, which is about syncing a repo to a workspace and automatically works around this problem
- We can update the v2 sync protocol to support multiple
workflows folders (which means mutiple openfn.yaml files or multiple projects tracked inside a single openfn.yaml file). I'm really not keen on this - it's awkward and ugly and doesn't work well with git.
Another possible consideration, perhaps more for GH sync v2, is monorepos. What if I have one GH repo with multiple different projects synced inside it? Well that's OK, you just have an opennf.yaml inside each folder. The sync would need to take a path to openfn.yaml (rather than default to root) but that's not a big deal
We've got a little problem when using v2 sync with GitHub sync, as users are now able to do (see #4634)
I unfortunately forgot during development that v2 sync is only suitable for pulling one project per branch from lightning
This is because there's only one
workflowsfolder supported.So if I set up a v2 sync, and push to github from the app, it'll update workflows on the target branch.
But if I create a sandbox, and sync that to
main, then when the sandbox syncs back to github it'll override the main project. If I then sync main, it'll override again to look like main. It doesn't work, basically.This stuff works great if you're using two branches on github to maintain two versions of the project. Your main and dev are on different branches, and you can diff the
workflowsfolders to understand what's different. It's super hoopy.It also works great if I have a one-way snyc (which we don't support). Let's say I have on project on main, and when I push changes to the GH repo, I want to deploy the updates to 3 other projects. Those pull-only syncs work great - they use the same version of workflows to update the destination project.
So solutions here:
workflowsfolders (which means mutiple openfn.yaml files or multiple projects tracked inside a single openfn.yaml file). I'm really not keen on this - it's awkward and ugly and doesn't work well with git.Another possible consideration, perhaps more for GH sync v2, is monorepos. What if I have one GH repo with multiple different projects synced inside it? Well that's OK, you just have an opennf.yaml inside each folder. The sync would need to take a path to openfn.yaml (rather than default to root) but that's not a big deal