Fix: forward aggregate report_type to TriggerReconAggregateService#2401
Fix: forward aggregate report_type to TriggerReconAggregateService#2401moomindani wants to merge 3 commits intodatabrickslabs:mainfrom
Conversation
TriggerReconService.trigger_recon validates that report_type is in
{"schema","data","row","all","aggregate"} via _RECON_REPORT_TYPES, so
"aggregate" is accepted at the entry point. However, _do_recon_one only
branches on {"schema","all"} and {"data","row","all"}; the "aggregate"
case falls through, no comparison is performed, and recon_capture.start()
records the run with status=true. Callers see a successful reconciliation
even when source and target differ.
The aggregate path lives in TriggerReconAggregateService.
trigger_recon_aggregates. CLI dispatch in execute.py already routes the
AGG_RECONCILE_OPERATION_NAME to that service, but programmatic callers
of TriggerReconService.trigger_recon (e.g. notebooks built around the
public Reconcile API) hit the silent no-op.
Forward report_type=="aggregate" to TriggerReconAggregateService at the
top of trigger_recon. The import is deferred because
trigger_recon_aggregate_service already imports trigger_recon_service.
Co-authored-by: Isaac
Cover the routing introduced in this branch:
- report_type='aggregate' is forwarded to
TriggerReconAggregateService.trigger_recon_aggregates without going
through create_recon_dependencies of the data path
- The aggregate-specific kwargs (table_recon, reconcile_config,
local_test_run) are passed through unchanged
- All other report_types ('schema', 'data', 'row', 'all') keep the
existing path and never call the aggregate service
Without the dispatch, the existing _do_recon_one short-circuits silently
for report_type='aggregate' and the run is recorded as successful. The
first test reproduces the original behavior (no aggregate forward) and
fails on origin/main.
Co-authored-by: Isaac
m-abulazm
left a comment
There was a problem hiding this comment.
why do we need this? notebook/python users should use TriggerReconAggregateService.trigger_recon_aggregates
|
Fair question —
We've actually seen users get caught by this in practice — they followed the same Happy to take an alternate fix if you'd prefer — e.g., narrowing |
…ispatch-in-trigger-recon # Conflicts: # src/databricks/labs/lakebridge/reconcile/trigger_recon_service.py
Changes
What does this PR do?
Forwards
report_type=="aggregate"fromTriggerReconService.trigger_recontoTriggerReconAggregateService.trigger_recon_aggregatesso programmatic callers that pass an aggregate report type get the actual aggregate comparison instead of a silent no-op.Relevant implementation details
TriggerReconService.trigger_reconvalidatesreport_typeagainst_RECON_REPORT_TYPES = {"schema","data","row","all","aggregate"}, so"aggregate"is accepted at entry. However,_do_recon_oneonly branches on{"schema","all"}and{"data","row","all"}. The aggregate case falls through, no comparison runs, andrecon_capture.start()records the run withstatus=true. Callers see a successful reconciliation even when source and target differ.The CLI dispatch in
execute.pyalready routesAGG_RECONCILE_OPERATION_NAMEtoTriggerReconAggregateService.trigger_recon_aggregates, but programmatic users of the public Reconcile API (notebooks, integrations) hit the silent no-op without this change.This PR adds an early dispatch at the top of
trigger_recon. The import is deferred becausetrigger_recon_aggregate_servicealready importstrigger_recon_serviceand a top-level import would be circular.Caveats/things to watch out for when reviewing:
execute.py's existing dispatchschema,data,row,all) keep the original pathLinked issues
None
Functionality
TriggerReconService.trigger_reconTests
Manual end-to-end on Databricks Serverless with two tables of different row counts:
aggregate=None,status=true, no rows inrecon.aggregate_*aggregate=False,mismatch=1, rows present inrecon.aggregate_metricsUnit tests added in
tests/unit/reconcile/test_verify_reconcile.py:test_trigger_recon_dispatches_aggregate_to_aggregate_service: forwarding works and the data-path setup is skippedtest_trigger_recon_does_not_dispatch_non_aggregate:schema/data/row/allkeep the original path