Commit ada154d
committed
Forward pipeline context, add other_context to JoinTransform, drop SerializeAsAny
- NarwhalsPipelineModel.__call__ now forwards the caller's context to
source(context) rather than passing NullContext, so context-keyed sources
(e.g. TPCHDataGenerator with TPCHTableContext) can be used directly without
per-table adapter wrappers.
- JoinTransform gains an other_context field so a single context-keyed source
can be reused across multiple joins (e.g. one TPCHDataGenerator instance
serving customer/orders/lineitem). Validator now checks
isinstance(other_context, other.context_type) at construction.
- Drop SerializeAsAny from source and other fields. ccflow's BaseModel
metaclass already wraps bare BaseModel-typed fields, so the explicit
annotation was redundant. Verified subclass info still survives JSON
round-trip in tests.
- Loosen SequenceTransform.transforms to accept the same
Union[NarwhalsFrameTransform, Callable] as NarwhalsPipelineModel.transforms,
for consistency. Beef up the comment on NarwhalsFrameTransformOrCallable to
spell out why both branches matter (BaseModel branch enables type_-based
round-trip; Callable branch is a runtime escape hatch that doesn't
serialize).
- Notebook polish: drop hardcoded TOC, drop graph-awareness subsection, drop
trailing Pointers section, fix ascii alignment in section 7 diagram, and
remove the TPCHTableProvider adapter (and per-table providers) by leaning
on context passthrough -- pipelines now use the generator directly. Combine
the AggregateByReturnStatus and SortByReturnStatus transforms into a single
SummarizeByReturnStatus (group keys = sort keys, naturally one operation).
Add a new section 4 'Aside' that frames NarwhalsFrameTransform as an
opt-in convention rather than a requirement, with a tradeoffs table for
plain function vs plain BaseModel vs NarwhalsFrameTransform.
- Tests: add coverage for plain callable + plain ccflow.BaseModel inside
SequenceTransform (round-trips via type_), context forwarding to source,
other_context flow-through, and other_context type-mismatch rejection
(38 tests total, full suite 687 passed).
Signed-off-by: Pascal Tomecek <pascal.tomecek@cubistsystematic.com>1 parent 4de7717 commit ada154d
4 files changed
Lines changed: 504 additions & 1326 deletions
File tree
- ccflow
- examples
- models
- tests/models
0 commit comments