Add TPC-DS workflow

[milenkovicm]

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

#1688 adds TPCH workflow, it would be great if we could replicate same behaviour and get TPC-DS (SF1 or SF10) workflow.

Describe the solution you'd like

copy current approach in TPCH workflow

Describe alternatives you've considered

Additional context

[Piyush-08-bot]

hey @milenkovicm, would like to give this a shot — mind assigning it to me?

before diving in, poked around the codebase a bit and TPC-DS doesn't seem to be implemented anywhere yet (only TPC-H exists - tpch.rs, queries/q1-q22 etc). no tpcds binary, no query files, no data gen for it.

so wanna check with you first - is this PR also supposed to add TPC-DS support (binary + queries + data gen), or is that happening separately and this issue is just for the CI workflow once that's ready?

lemme know, thanks!

[milenkovicm]

you're right @Piyush-08-bot that has to be implemented as well
code at https://github.com/apache/datafusion-benchmarks can help i believe

[Piyush-08-bot]

Thanks for confirming @milenkovicm! I checked apache/datafusion-benchmarks and found that TPC-DS is actually mostly ready there:

• All 99 query files (q1.sql-q99.sql) already exist and can be copied over directly
• There's a generic Rust runner pattern (datafusion-rust/main.rs) that registers tables and runs queries by number, similar in spirit to tpch.rs

The main gap is data generation - TPC-H uses tpchgen-cli which is a simple cargo-installable binary, but TPC-DS data generation in datafusion-benchmarks requires downloading tpc-ds-tool.zip from TPC.org manually + building dsdgen via Docker. That's heavier and harder to fully automate in CI the same way as the TPC-H workflow.

Given that, would it be reasonable to scope this PR as:

• Add the TPC-DS query files + a benchmark binary (adapted from tpch.rs pattern) to run against a Ballista cluster
• Start with a smaller subset of queries to keep things manageable
• For the CI workflow, either use a pre-generated small dataset checked into CI cache, or document a manual data-gen step for now, and we can fully automate the data-gen part in a follow-up once we figure out a lighter-weight TPC-DS generator (similar to tpchgen-cli)?

Let me know if this scope works or if you'd prefer a different approach.

[milenkovicm]

clflushopt/tpchgen-rs#218 it would make this issue easier to implement

[alamb]

clflushopt/tpchgen-rs#218 it would make this issue easier to implement

We are close...