Skip to content

Commit ada154d

Browse files
committed
Forward pipeline context, add other_context to JoinTransform, drop SerializeAsAny
- NarwhalsPipelineModel.__call__ now forwards the caller's context to source(context) rather than passing NullContext, so context-keyed sources (e.g. TPCHDataGenerator with TPCHTableContext) can be used directly without per-table adapter wrappers. - JoinTransform gains an other_context field so a single context-keyed source can be reused across multiple joins (e.g. one TPCHDataGenerator instance serving customer/orders/lineitem). Validator now checks isinstance(other_context, other.context_type) at construction. - Drop SerializeAsAny from source and other fields. ccflow's BaseModel metaclass already wraps bare BaseModel-typed fields, so the explicit annotation was redundant. Verified subclass info still survives JSON round-trip in tests. - Loosen SequenceTransform.transforms to accept the same Union[NarwhalsFrameTransform, Callable] as NarwhalsPipelineModel.transforms, for consistency. Beef up the comment on NarwhalsFrameTransformOrCallable to spell out why both branches matter (BaseModel branch enables type_-based round-trip; Callable branch is a runtime escape hatch that doesn't serialize). - Notebook polish: drop hardcoded TOC, drop graph-awareness subsection, drop trailing Pointers section, fix ascii alignment in section 7 diagram, and remove the TPCHTableProvider adapter (and per-table providers) by leaning on context passthrough -- pipelines now use the generator directly. Combine the AggregateByReturnStatus and SortByReturnStatus transforms into a single SummarizeByReturnStatus (group keys = sort keys, naturally one operation). Add a new section 4 'Aside' that frames NarwhalsFrameTransform as an opt-in convention rather than a requirement, with a tradeoffs table for plain function vs plain BaseModel vs NarwhalsFrameTransform. - Tests: add coverage for plain callable + plain ccflow.BaseModel inside SequenceTransform (round-trips via type_), context forwarding to source, other_context flow-through, and other_context type-mismatch rejection (38 tests total, full suite 687 passed). Signed-off-by: Pascal Tomecek <pascal.tomecek@cubistsystematic.com>
1 parent 4de7717 commit ada154d

4 files changed

Lines changed: 504 additions & 1326 deletions

File tree

0 commit comments

Comments
 (0)