pushPackageDirectoriesSequentially in sf project deploy commands

[mshanemc]

We've gotten a few people asking why sf project deploy start intentionally doesn't support pushPackageDirectoriesSequentially.

First, make sure you've read the migration docs.

Long ago, the force:source:push would deploy pkgDirs sequentially, 1 deployment for each folder. Deploys are slow, so for the vast majority of users, it's better to do merge all the pkgDirs into 1 deployment. It really sucked if you changed a file in 2 different package directories and pushed it (you just did 2 deployments for no reason!)

When the CLI moved to using the much faster SDR library to match what VSCode Extensions did, the default behavior became "merge the pkgDirs and deploy them in one go". There were a few people who that didn't work for, so we created the pushPackageDirectoriesSequentially property in sfdx-project.json to opt-in where the old behavior was required--props to @kjonescertinia for that idea

1. your project is so big you hit the limit for deployment size. Sequential deploys would work when each pkgDir was under the limit
2. you duplicate metadata across pkgDirs ex: #2191
3. you needed some things to deploy before other things because of metadata API bugs/gaps such that they don't work in the same deployment

When we created the new project deploy commands, we wanted this to be simpler. If you really need multiple deployments in a specific order, you can call the deploy command however you need and that's decoupled from your folder structure.

Use case 1

You have lots of pkgDirs because things are modular. A is large but under the limit, B/C/D aren't as big, but A+B+C+D combined are above the limit. When you want to deploy the entire project to a new folder, you could say (and put in your init script)

sf project deploy start -d folderA
sf project deploy start -d folderB -d folderC -d folderD

Now you've only got 2 deploys instead of 4! Metadata deployments are slow enough so why do extra ones?

Bonus: you couldn't run the same commands to deploy to a Full Sandbox or Prod or Packaging Org (something push could never do).

And if you change a file in A and another file in C, then sf project deploy start only has to do 1 deployment.

Use case 3

Metadata type X has to go before the other types.

sf project deploy start -m X
sf project deploy start (that deploys the rest of the changes. You could use -d or -x if you want something that will deploy to any org)

Now you can organize your project logically instead of based on metadata API quirks.

---

Other design considerations

"I was doing a push and it timed out and now I can't push because tracking is hosed."
"I right-clicked to retrieve something in VSCode and now tracking is hosed."
"I didn't want to deploy/retrieve all the changes, just one, so I use force:source:deploy/retrieve -m and now tracking is hosed."

We heard that a lot.

At a high level, we didn't like (and y'all found confusing) the weird gaps between source:push and source:deploy.

1. push only works with tracking orgs. The source:deploy you have to opt into tracking or it breaks tracking.
2. only source:deploy could specify [metadata types, files, manifest]
3. push is only synchronous, and if it times out or gets disconnected, you're in a broken state.
4. deploy could be async, and has a sibling report command

By unifying the functionality of source:push and source:deploy, we got

1. tracking is used if the org supports it (unless you opt out) so you can use the same commands on any org
2. you can deploy changes (like push) but also have async and report and resume. resume will get the deploy results and update your tracking files corretly

Those features are really, really challenging support across multiple deployments. We were already taking multiple DeployResults from the metadata API response and trying to make it "feel" like one. It's hard to get exactly right.

Now imagine there's a timeout in push after deploy 1 of 5 pkgDir. That's hard to resume or report details from. You would never be able to do --async because the cli would need to do one deploy and then initiate the next.

---

OK, you read all that and still think you can't live without pushPackageDirectoriesSequentially?

Vote this up, and explain why, and what you'd like to see.

[mshanemc]

related proposal
[group pkgDirs into deployments using the sfdx-project.json file instead of doing multiple commands] #2381

[gsbasso]

Hi @mshanemc ,

Firstly thank you so much for your detailed explanation above. It actually helps a lot.

I think, at least in theory, I could just have one deploy command of the form project:deploy:start -d folder1 -d folder2 -d folder3 ...

I'm not clear why, for Use Case 1, where a file is changed in A and one in C, there would only be one deployment if you have two deploy commands (one for A, and one for B, C and D)? Can you clarify?

Can you also clarify, again for Use Case 1, that the second deploy would only start after the first deploy completes (i.e. there is nothing asynchronous going on)?

Finally, as an observation, the concept of tracking is not really relevant if your CI/CD process is creating a new scratch org (as does ours). So one change I do like is the idea of disabling tracking, so time is not wasted keeping this updated (again, if one is always deploying to a new scratch org).

[mshanemc]

I'm not clear why, for Use Case 1, where a file is changed in A and one in C, there would only be one deployment if you have two deploy commands (one for A, and one for B, C and D)? Can you clarify?

I'm saying that you only needed PPDS the very first time (because too many components) and that after that, as you're doing push/pull dev work, it's still splitting up each deployment unnecessarily. In the new model, you need 2 commands for the very first time, but after that you can just say sf project deploy start.

[mshanemc]

Can you also clarify, again for Use Case 1, that the second deploy would only start after the first deploy completes (i.e. there is nothing asynchronous going on)?

Correct. The CLI polls for a deployment to finish, then kicks off the next one. Even if we sent them all at once, the mdapi would just enqueue them even if we did. But if the one fails, we don't want to deploy the next, so we wait for the result.

[mshanemc]

Finally, as an observation, the concept of tracking is not really relevant if your CI/CD process is creating a new scratch org (as does ours). So one change I do like is the idea of disabling tracking, so time is not wasted keeping this updated (again, if one is always deploying to a new scratch org).

Exactly! so org create scratch ... --no-track-source. That's gonna be faster than push with PPDS if tracking doesn't matter.

[mshanemc]

Exactly! so org create scratch ... --no-track-source. That's gonna be faster than push with PPDS if tracking doesn't matter.

and org disable tracking if you CI/CD is not using scratch orgs

[daveespo]

We have worked around the lack of ppds for now. We were part of Use Case 2 (#2191) and worked around the issue by just removing the duplicate metadat. We were lucky that we were able to do so since it was the standardValueSet on Opportunity.Stage that was our primary use case.

The thing we'd really like is to have the globbing behavior of the merging of package directories documented. That is, which directory (first, or last, or alphabetically?) wins when two package directories contain the same metadata?

[mshanemc]

we don't imperatively tell it which order to do things in.

currently, SDR will use the first version of a component it receives, and then NOT "add" another component with the same name/type.

The question is, "hey, nested node stream, which asynchronously read fs file descriptor stream are you going to get to first while creating a zip file in memory?" I don't think that's going to be consistent.

I thin the choices for Use Case 2 become

1. change your metadata (either do a full, no-conflict duplication or don't duplicate)
2. deploy separately

[daveespo]

I'd be happy with sequentially iterating through the packageDirectories array in sfdx-project.json and either have the first metadata component with that name win or the last .. but right now it's unclear if there's any order that it goes through the packageDIrectories array.

And just a reminder of our use case from #2191 -- our package contains a standardValueSet metadata file for Opportunity.Stage -- but we "add a few more" stages in our Scratch Orgs to represent real-world user customization of the Stages on Opp. So we want the "last" directory to win in our case. It's my belief that most people that are doing package-based development have their "package" metadata in the first directory in sfdx-project.json and then add additional metadata in the other package directories for development purposes. After all, you can't build a package if it depends on metadata outside the package directory.

[kjonescertinia]

Part of my interest in this issue is in it being totally clear how third party tools should interpret metadata workspaces. As @daveespo mentions the problem here is this has not been documented anywhere that I have found. This is clearly very bad for such tools but its also pretty terrible state in general because we can't ever be sure how the metadata might end up and org, it's entirely dependent on the push/deploy version and the features/bugs contained within. To be fair the same is actually true of the MDAPI, but the difference is there are years of convention behind how MDAPI works and its been very very stable.

To deal with this in the tools I work on we flag any sfdx-project.json that does not set PPDS with an error since we don't know how to accurately interpret the metadata it contains. If you don't want to continue to support PPDS I think that is OK as long as you error if it is set when you are going to do something that would silently ignore it. Obviously it's fairly easy for us to script our own deploy handling for when PPDS is set so we get the consistent behaviour we need. Another option that might be worth looking at is not supporting source tracking with PPDS. I would guess it's very likely that if you care about PPDS you really don't care about source tracking.

[mshanemc]

I would guess it's very likely that if you care about PPDS you really don't care about source tracking

interesting, wouldn't have assumed that since PPDS was only ever implemented for force:source:push. For the Use Case 1, it would be push my giant project, then I'm gonna do some dev work where I'd want tracking on.

[kjonescertinia]

Sorry I was not that clear, I mean we don't care about changes on the org which it sounded like where some of the difficulty with PPDS is. Mostly doing 'push -f' is the right default for our developers, if only to avoid having to deal with the false positive conflicts being detected.

[jefersonchaves]

Hello @mshanemc,

Thanks for bringing it to my attention.

To be honest, the solution would be a series of fixes on the underlying Metadata API; most of the other arguments are workaround on existing limitations - except for the merge of contents under different folders.

The usual use case is 3, and the usual workaround is to create either a bash script or something similar (but again, if MDAPI - Metadata API - had not required ordering some metadata then we wouldn't have this problem, which would be the ideal scenario).

For ISVs, I tend to assume they can leverage CumulusCI instead of rolling their own (that is what I would probably do if I get there again); for customers is a little trickier but some may end up relying on vendor solutions anyway.

[mshanemc]

the solution would be a series of fixes on the underlying Metadata API
💯 . That won't happen anytime soon. That team is very focused on perf (improving deploy times)

[jefersonchaves]

Look, I agree; that probably makes sense.

In this case, we should clarify that these are workarounds and/or duct tape solutions instead of actual features to avoid future confusion (especially if these workarounds remain for quite some time).

But I believe that the merge on different folders can make sense (e.g. ValueSets for picklists), but maybe guidance is also a short-term workaround, even though I feel it can get quite annoying.

[AllanOricil]

Hack
Create a proper package manger such, as mvn and npm, that can resolve a dependency graph and install packages in a certain order.

[AllanOricil]

@mshanemc @iowillhoit @cromwellryan I don't know if ChatGPT 4 is lying or if he has already analyzed a hidden package you guys are creating. Look what he suggested to me

"Oops" hahahaha

[mshanemc]

100% hallucination

[AllanOricil]

Seems like me. Looking forward to seeing how gpt 5 will behave

[ImJohnMDaniel]

LOL. Looks like it somehow got a hold of @FabienTaillon's command and changed the top most topic. 🤣

[FabienTaillon]

Looks like you're right @ImJohnMDaniel :D

[tscottcertinia]

If you are using an onion design pattern then you want to ensure noone is developing code that references code outside of their layer, only inward. Using directories and pushPackageDirectoriesSequentially for your onion layers you can validate that. Upvoted.