[DISCUSS] DataFusion less frequent major / breaking releases (ease using multiple third-party extensions (like delta, or iceberg) )

[alamb]

Is your feature request related to a problem or challenge?

One of the dreams of the composable data ecosystem is to quickly assemble a system from various components (DataFusion, data formats

DataFusion still releases once a month, which allows code to quickly flow but also causes at least 2 challenges:

1. Takes non trivial work required to upgrade downstream projects, as mentioned in [Discuss] Release cadence / patch releases / Long Term Supported (lts) minor releases #5269
2. Make upgrading and using downstream third-party extensions hard

Third party extensions like delta-rs and iceberg provide TableProviders for DataFusion, which is really nice. However, to use those packages the versions of DataFusion must match exactly.

This means for an application that relies on multiple downstream packages must wait until ALL of them have upgraded to the new version in order to upgrade DataFusion. If there is any delay in the downstream libraries updating, it delays.

For example, an application that wants to use delta-rs, iceberg, and the table-providers crate, there is a race after each upgrade of DataFusion

Let's take a release timeline for

1. +0 days: DataFusion version X released
2. +7 days: New delta-rs releases upgraded to DataFusion X
3. +11 days: new iceberg crate released upgraded to DataFusion X
4. +12 days: new table-providers version is released
5. +13-30 days: End user app can upgrade DataFusion and delta, and icerberg
6. +31 days: New DataFusion is released again

Describe the solution you'd like

I would like downstream libraries to have more time and schedule flexibility when upgrading DataFusion and other dependent crates, so that it is easier to construct a system from different components

Describe alternatives you've considered

Option 1: Switch to major/minor release cadence

We could follow the model of arrow-rs which does releases monthly, but breaking releases only quarterly. Here is how it works in arrow-rs: https://github.com/apache/arrow-rs?tab=readme-ov-file#release-versioning-and-schedule

This would mean continuing to release every month, but only allowing breaking API changes every 3rd release (or some other cadence)

The major cost here is that maintainers and contributors would have to be diligent about not merging breaking API changes until a major release

This is possible to automate somewhat:

• Semver-checks for all crate on merge and push #16078 from @logan-keede
• Adds script to detect breaking API changes/ semver #16541 from @lic

Option 2: LTS and feature branch

-Keep (at least) two branches going: LTS and main, as proposed by @andygrove in #5269

In this model we would likely backport changes to the LTS branch and make releases from there. The downside of this approach is that there is extra work to backport changes to LTS.

Additional context

No response

[findepi]

I would like downstream libraries to have more time and schedule flexibility when upgrading DataFusion and other dependent crates, so that it is easier to construct a system from different components

That's a noble goal.

At the same time we should avoid policies that slow down DataFusion development.

Perhaps part of a solution could be to invert dependencies between the project. For example, Iceberg-DataFusion and Delta-DataFusion integrations could live in this repo.

[alamb]

Perhaps part of a solution could be to invert dependencies between the project. For example, Iceberg-DataFusion and Delta-DataFusion integrations could live in this repo.

That would make running tests in the DataFusion repo harder, perhaps (certainly there would be more of them).

Also, while it could work for icerberg and delta-rs potentially, I am not sure we could plausibly bring all the integrations into the same repo (though I will admit it seems to work for trino 🤔 )

I am not opposed necessairly, just trying to think of the trade offs

Another approach maybe could be to break the datafusion repo into separate repos with less stuff that can move slower and keep as much as possible in other repos that can move faster 🤔

[milenkovicm]

IMHO, option 1 may be better, it is not as frequent as current release cycle but still frequent enough to take breaking changes in relatively small and digestible chunks.

Not sure if option 2 would force downstream project to have same branching strategy. Otherwise it would bring "big update" every time LTS was released.

[alamb]

And to be clear, in my mind option 1 still has us doing monthly releases, we just restrict which releases have breaking API changes

[timsaucer]

I tend to favor option 1. I am currently in this same position where each version of DF requires updates to Lance and datafusion-python. In the last update, within a week of getting my repo upgraded to DF47, DF48 was released.

The part that is unclear to me, but maybe arrow-rs has already solved, is how we go about going PR reviews on work that does introduce breaking changes. Consider the timeline where a feature is developed and ready for review two weeks after a major release. That feature contains breaking API changes. Do we then require that feature to sit in a long living PR for another 6-8 weeks until we merge it to main? Those can be extremely difficult to maintain.

[findepi]

If we look at the problem through the lens of project consumers, the expectation to never break anything is natural. If we look at the problem of feature implementers, the exception to be able to introduce changes, even those that are unfortunately breaking, is natural.

arrow-rs has 2.6x less code and conceptually much simpler API surface. DataFusion is a library for building query engines and data processing tools. It is in active development and is not yet in "mostly complete, mostly maintenance" stage, so internal API changes are often unavoidable. DataFusion being a library, it has enormous API surface, so most of "internal API changes" are actually potentially breaking changes for downstream consumers.

Being able to do such changes once a quarter (4 times a year), looks too rare to me.

Also, while it could work for icerberg and delta-rs potentially, I am not sure we could plausibly bring all the integrations into the same repo

Agreed. At the same time, this issue is not on context of any integrations. It's in a context of real-life problems we (some of us) have today. What are these problems, and what integrations they pertain?

[alamb]

This is a pretty sweet idea from @jonmmease about making upgrades easier (use LLM agents): #13648 (comment)

(it is somewhat orthogonal to the other goals of this ticket, but I thought it was so good I wanted to share. Maybe like all great ideas it seems obvious in retrospect)

[andygrove]

I favor option 2 (as already pointed out in the original issue). I don't think that we should artificially slow down development against the main branch.

We already have a documented process for backporting PRs to release branches (e.g. branch-48) and creating future 48.x.x releases without breaking changes. We always release from the release branches, not the main branch. This is the same model that Apache Spark uses. We just haven't been proactive in backporting PRs to the release branches most of the time.

The idea of holding off on merging PRs to main sounds like a nightmare to me. In Comet, we often temporarily point to a fixed revision of DataFusion's main branch so that we can gradually upgrade to the next version, testing all recently merged PRs. If breaking change PRs are all sitting in a queue until just before the next major DataFusion release, then I'm not sure how we can gradually upgrade.

[andygrove]

The responsibility for creating the PRs to backport fixes to the release branch should fall to the downstream users who are waiting on those fixes. There is additional work for DataFusion maintainers in terms of PR reviews though.

[comphead]

More inclining to version 2 with slight modifications. Similar to Rustc flow https://web.mit.edu/rust-lang_v1.25/arch/amd64_ubuntu1404/share/doc/rust/html/book/second-edition/ch01-03-how-rust-is-made-and-nightly-rust.html

so it would be a main branch representing latest update(keep in mind main can be broken by bad commits, performance issue, etc)

a nightly release branch where DF releases artifacts per night or per week, this branch should be protected by extended tests, performance tests, etc

[alamb]

So for Version 2 (a LTS branch) do we have any proposal for the cadence of releases?

Like would we do 2 releases each month now?

1. LTS release
2. Major release from main

Or would we perhaps keep one release a month and release on alternate months 🤔