Skip to content

feat: model TM1 12.6.1 Arrow/Parquet/Flight TI datasources#1429

Closed
skido-cw wants to merge 1 commit into
cubewise-code:masterfrom
skido-cw:feature/arrow-parquet-flight-datasource
Closed

feat: model TM1 12.6.1 Arrow/Parquet/Flight TI datasources#1429
skido-cw wants to merge 1 commit into
cubewise-code:masterfrom
skido-cw:feature/arrow-parquet-flight-datasource

Conversation

@skido-cw

Copy link
Copy Markdown
Contributor

Summary

Adds first-class modelling for the three new TurboIntegrator input datasource types introduced in TM1 v12, build 12.6.1 — Apache Arrow IPC/Feather (ARROW), Apache Parquet (PARQUET), and Apache Arrow Flight (FLIGHT). This is pure object modelling of the Process DataSource block; no transport coupling and no new service methods. Previously these types fell through to an empty "DataSource": {}.

Changes

  • Process: four new datasource_flight_* scalars (constructor, from_dict, properties) and two new _construct_body_as_dict branches:
    • ARROW/PARQUET — file name or http(s) URL via dataSourceNameForServer/dataSourceNameForClient, plus optional jsonRootPointer / jsonVariableMapping for nested (struct/list/map) columns; no ascii* keys.
    • FLIGHTflightLocation / flightDescriptorType / flightDescriptor / flightAuth.
    • Both branches match Type case-insensitively (see "Verified" below).
  • ProcessService.get / get_all: select the four flight fields.
  • datasource_type stays a free string (no Type whitelist) and no version gating is added — ProcessDataSource is an OData open type whose Type is validated by the engine at execute time.

Verified against a live PA 12.6.1 server

End-to-end create → read-back confirmed two behaviors worth flagging:

  • The Arrow Flight wire fields are flightLocation / flightDescriptorType / flightDescriptor / flightAuth (a dataSourceFlight* payload is silently dropped).
  • The server canonicalizes Type to title-case on read-back (Arrow / Parquet / Flight), which is why the body branches match case-insensitively — otherwise from_dict(get(...)) would round-trip to an empty DataSource.

Testing

  • Offline unit tests (no server) for the body shapes, from_dictbody_as_dict round-trips, and the title-case Type round-trip — 17 tests, all pass.
  • Optional live round-trip test gated to TM1 ≥ 12.6.1 (@skip_if_version_lower_than), self-cleaning — passes against a live 12.6.1 server.
  • black --check . and ruff check . pass.

Backward compatibility

No changes to the existing datasource branches (ASCII / None / ODBC / TM1CubeView / TM1DimensionSubset / JSON); the public API stays byte-compatible for every existing datasource type.

🤖 Generated with Claude Code

Add first-class modelling for the three new TurboIntegrator input
datasource types introduced in TM1 v12 build 12.6.1 (Apache Arrow
IPC/Feather, Parquet, and Arrow Flight), verified end-to-end against a
live PA 12.6.1 server.

Process:
- four new datasource_flight_* scalars (constructor, from_dict, properties)
- ARROW/PARQUET branch reuses dataSourceNameForServer/Client and optional
  jsonRootPointer / jsonVariableMapping (no ascii* fields)
- FLIGHT branch emits the server's verified wire fields: flightLocation /
  flightDescriptorType / flightDescriptor / flightAuth
- columnar/Flight branches match Type case-insensitively, because the
  server canonicalizes Type to title-case on read-back ("Arrow"/"Parquet"/
  "Flight"); this keeps from_dict(server_response) round-trips from
  collapsing to an empty DataSource block

ProcessService.get/get_all select the four flight fields.

datasource_type stays a free string (no Type whitelist) and no version
gating is added, since ProcessDataSource is an OData open type validated by
the engine at execute time.

Tests: offline unit tests for the body shapes, from_dict round-trips, and
the title-case Type round-trip (17 total), plus an optional live round-trip
test gated to TM1 >= 12.6.1. Offline suite and the live 12.6.1 round-trip
both pass.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@skido-cw

Copy link
Copy Markdown
Contributor Author

Superseded by #1431, which contains this datasource-modelling commit plus the new write_dataframe(use_parquet=/use_arrow=) columnar blob-write path. Closing in favor of the combined PR.

@skido-cw skido-cw closed this Jun 29, 2026
@skido-cw skido-cw deleted the feature/arrow-parquet-flight-datasource branch June 29, 2026 16:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant