feat: model TM1 12.6.1 Arrow/Parquet/Flight TI datasources#1429
Closed
skido-cw wants to merge 1 commit into
Closed
feat: model TM1 12.6.1 Arrow/Parquet/Flight TI datasources#1429skido-cw wants to merge 1 commit into
skido-cw wants to merge 1 commit into
Conversation
Add first-class modelling for the three new TurboIntegrator input
datasource types introduced in TM1 v12 build 12.6.1 (Apache Arrow
IPC/Feather, Parquet, and Arrow Flight), verified end-to-end against a
live PA 12.6.1 server.
Process:
- four new datasource_flight_* scalars (constructor, from_dict, properties)
- ARROW/PARQUET branch reuses dataSourceNameForServer/Client and optional
jsonRootPointer / jsonVariableMapping (no ascii* fields)
- FLIGHT branch emits the server's verified wire fields: flightLocation /
flightDescriptorType / flightDescriptor / flightAuth
- columnar/Flight branches match Type case-insensitively, because the
server canonicalizes Type to title-case on read-back ("Arrow"/"Parquet"/
"Flight"); this keeps from_dict(server_response) round-trips from
collapsing to an empty DataSource block
ProcessService.get/get_all select the four flight fields.
datasource_type stays a free string (no Type whitelist) and no version
gating is added, since ProcessDataSource is an OData open type validated by
the engine at execute time.
Tests: offline unit tests for the body shapes, from_dict round-trips, and
the title-case Type round-trip (17 total), plus an optional live round-trip
test gated to TM1 >= 12.6.1. Offline suite and the live 12.6.1 round-trip
both pass.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Contributor
Author
|
Superseded by #1431, which contains this datasource-modelling commit plus the new |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds first-class modelling for the three new TurboIntegrator input datasource types introduced in TM1 v12, build 12.6.1 — Apache Arrow IPC/Feather (
ARROW), Apache Parquet (PARQUET), and Apache Arrow Flight (FLIGHT). This is pure object modelling of theProcessDataSourceblock; no transport coupling and no new service methods. Previously these types fell through to an empty"DataSource": {}.Changes
Process: four newdatasource_flight_*scalars (constructor,from_dict, properties) and two new_construct_body_as_dictbranches:ARROW/PARQUET— file name or http(s) URL viadataSourceNameForServer/dataSourceNameForClient, plus optionaljsonRootPointer/jsonVariableMappingfor nested (struct/list/map) columns; noascii*keys.FLIGHT—flightLocation/flightDescriptorType/flightDescriptor/flightAuth.Typecase-insensitively (see "Verified" below).ProcessService.get/get_all: select the four flight fields.datasource_typestays a free string (noTypewhitelist) and no version gating is added —ProcessDataSourceis an OData open type whoseTypeis validated by the engine at execute time.Verified against a live PA 12.6.1 server
End-to-end create → read-back confirmed two behaviors worth flagging:
flightLocation/flightDescriptorType/flightDescriptor/flightAuth(adataSourceFlight*payload is silently dropped).Typeto title-case on read-back (Arrow/Parquet/Flight), which is why the body branches match case-insensitively — otherwisefrom_dict(get(...))would round-trip to an emptyDataSource.Testing
from_dict⇄body_as_dictround-trips, and the title-caseTyperound-trip — 17 tests, all pass.@skip_if_version_lower_than), self-cleaning — passes against a live 12.6.1 server.black --check .andruff check .pass.Backward compatibility
No changes to the existing datasource branches (
ASCII/None/ODBC/TM1CubeView/TM1DimensionSubset/JSON); the public API stays byte-compatible for every existing datasource type.🤖 Generated with Claude Code