Commit f8d9685
Extend pre-delete dep-guard to triggers (#221)
* feat: pre-delete dependency guard for DROP TABLE
Refuses a DROP TABLE while an active Pipeline still references the
resource (as either source or sink), so dropping the underlying Kafka
topic / Venice store / MySQL table can't silently orphan a downstream
pipeline.
Validator framework, made Connection-aware:
- Validated.validate(Issues, Connection) (was: validate(Issues))
- ValidatorProvider.validators(T, Connection) (was: validators(T))
- ValidationService.validate(T, Issues, Connection)
- ValidationService.validateOrThrow(T, Connection)
- ValidationService.validateOrThrow(Collection<T>, Connection)
- ValidationService.validators(T, Connection)
PendingDelete<T> wrapper (hoptimator-api):
- Explicit "this is being deleted" signal so unrelated callers of
validateOrThrow(source, connection) don't accidentally trigger
pre-delete checks.
- Carries an optional selfOwnerUid so cascade-deleted children can be
excluded from the dependent set.
K8s indexed lookup:
- PipelineDependencyLabels stamps `depends-on-<slug>` labels on every
Pipeline CRD at create time, naming each source/sink. The slug is a
16-char SHA-256 prefix of `<database>_<dot-joined-path>`; an
annotation lists the full identifiers so a slug collision can be
detected at check time.
- PipelineDependencyChecker uses a server-indexed label-selector list
+ annotation cross-check + selfOwnerUid filter.
- K8sPipelineDeployer threads sources/sink through and calls
PipelineDependencyLabels.labelsFor / annotationFor at toK8sObject().
K8sPipelineBundle and K8sMaterializedViewDeployer pass the data
through.
Dispatch:
- K8sValidatorProvider returns a K8sPipelineDependencyValidator for
PendingDelete<Source>; registered via
META-INF/services/com.linkedin.hoptimator.ValidatorProvider.
- K8sPipelineDependencyValidator wraps PipelineDependencyChecker as a
Validator.
DROP TABLE wiring:
- HoptimatorDdlExecutor calls
ValidationService.validateOrThrow(new PendingDelete<>(source),
connection) before DeploymentService.delete in the table branch.
HoptimatorDdlUtils.removeTableFromSchema() is the symmetric inverse
of registerTemporaryTableInSchema() for cleanup.
Implementor side-effects (no behavior change):
- KafkaDeployer / VeniceDeployer / MySqlDeployer no longer need a
declarative DependencyGuarded marker — the guard fires from the
validator framework before delete() is reached.
- All existing Validated implementors (DefaultValidator,
CompatibilityValidatorBase, AvroTableValidator, K8sViewTable) and
ValidatorProvider implementors (DefaultValidatorProvider,
CompatibilityValidatorProvider, AvroValidatorProvider) updated to
the new signatures.
Tests: PipelineDependencyLabelsTest, PipelineDependencyCheckerTest,
K8sPipelineDeployerTest assertions for stamping, validator-framework
test updates throughout.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat: support DROP TABLE for logical tables
LogicalTableDeployer.delete() previously threw SQLFeatureNotSupported.
Now implemented end-to-end as a per-tier sequence that mirrors what
running DROP TABLE on each tier independently would do, plus the
LogicalTable CRD removal at the top.
Flow:
1. Per-tier pre-flight via the validator framework:
ValidationService.validateOrThrow(new PendingDelete<>(tierSource,
logicalTableUid), connection) — refuses the drop if any active
external pipeline still references a tier resource. The
selfOwnerUid is the LogicalTable CRD's UID so the implicit
inter-tier pipelines (owned by the CRD, cascade-deleted with it)
don't self-block.
2. Delete the LogicalTable CRD. K8s owner-ref cascade removes its
owned Pipeline and TableTrigger CRDs.
3. Best-effort physical cleanup of each tier resource (Kafka topic,
Venice store, ...). A failed tier delete logs a warning but does
not abort — a stranded tier is recoverable; aborting mid-DROP
isn't.
4. Per-tier schema cleanup: deregister the TemporaryTable entry in
each tier schema only when its physical delete succeeded.
Tests:
- LogicalTableDeployerTest deleteRemovesCrdAndCleansUpTierResources,
deletePropagatesCrdDeletionFailure, deleteSwallowsTierCleanupFailures.
- logical-ddl.id integration test: DROP TABLE LOGICAL.testevent now
succeeds and cascades the implicit nearline-to-online pipeline.
- logical-offline-ddl.id companion update.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* test: integration scenarios + cleanup test warnings
kafka-ddl-create-table.id: cross-driver dependency-guard scenarios
exercising the new pre-delete check end-to-end through the kafka
driver — drop-table-while-pipeline-depends-on-it (source side and
partial-view sink side).
The bulk of the file count is mechanical noise reduction across
existing test files: dropped unused imports, tightened generics on
@SuppressWarnings, etc. — fallout from the warning_cleanup pass.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* update integration test
* comment cleanups and add @nullable annotation
* Migrate UID lookup to kind+name
* Fix issue to preserve existing annotations
* Add more test coverage
* Split depends-on annotation into directional sources/sink
Pipelines previously stamped a single `depends-on` annotation listing
every source and sink undifferentiated. The dep-guard collision check
worked on this, but it loses the source/sink direction information
needed for visualization.
Replace the unified annotation with two directional annotations:
hoptimator.linkedin.com/depends-on-sources: <s1>,<s2>,...
hoptimator.linkedin.com/depends-on-sink: <single sink>
The dep-guard's annotationConfirms now reads sources or sink as the
collision-guard set — same correctness guarantee, no semantic change
for the dep-guard. The split unlocks directional rendering for the
upcoming pipeline visualizer.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Add typed Source/Sink to Trigger and stamp depends-on labels
Two related changes that bring triggers into the depends-on label
index so the pre-delete dep-guard can find them.
Trigger API:
- Replace the old `(String database, List<String> path)` pair with
a typed `Source source` field (existing hoptimator-api type).
Drop the convenience getters database() / path() / table() /
schema() — callers go through trigger.source().X() for symmetry
with the new Sink.
- Add an optional `Sink sink` field for bridging-tier triggers
(LogicalTableDeployer's offline → online reverse-ETL flow), so
the deployer can stamp a depends-on-sink annotation in addition
to the source side.
- Source is nullable for DROP / PAUSE / RESUME paths that only
need the trigger name.
K8sTriggerDeployer stamping:
- On both the toK8sObject and the partial-update paths, stamp:
depends-on-<sourceSlug>: <sourceIdentifier>
annotation depends-on-sources: <sourceIdentifier>
and when sink is set:
depends-on-<sinkSlug>: <sinkIdentifier>
annotation depends-on-sink: <sinkIdentifier>
LogicalTableDeployer wiring:
- Pass `offlineSource` directly as the trigger's Source and a Sink
derived from `onlineSource` (when present) as the trigger's Sink.
HoptimatorDdlExecutor:
- Resolve the target table's database name the same way DROP TABLE
resolves it (HoptimatorJdbcTable / TemporaryTable unwrap), so
user-created `CREATE TRIGGER ... ON <schema>.<table>` triggers
participate in the dep-guard. Without this, Trigger.source was
null and K8sTriggerDeployer skipped label stamping — a trigger
could outlive its source silently.
Tests:
- K8sTriggerDeployerTest gains updateStampsSinkLabelWhenTriggerCarriesASink
pinning that a Trigger carrying a Sink stamps both source-side and
sink-side depends-on labels on the partial-update path.
- TriggerTest covers the new accessor shape (source(), sink(), null
source for the bare-name case).
- LogicalTableDeployerTest asserts trigger.source() / trigger.sink()
instead of the removed convenience getters.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Extend pre-delete dependency guard to TableTrigger CRDs
Triggers carry the same depends-on-<slug> labels Pipelines do (stamped
by K8sTriggerDeployer in the previous commit), but the dep-guard's
PipelineDependencyChecker only consulted the Pipeline CRD list. That
left a hole: a user could DROP TABLE on a source still referenced by
a live trigger.
PipelineDependencyChecker now runs the same label-selector +
annotation-confirmation logic against TableTrigger CRDs as well. The
inner loop is genericized over KubernetesObject; each blocker is
tagged with its CRD kind (pipeline/foo, trigger/bar) so the error
message points the user at what to unhook. Self-owner exclusion still
applies — LogicalTable-owned triggers don't block their parent's
cascade-delete.
Coverage:
- PipelineDependencyCheckerTest gets paired blocksOnExternalTrigger,
skipsSelfOwnedTrigger, and errorMessageListsAllBlockersAcrossKinds
cases that prove triggers participate alongside pipelines.
- New k8s-trigger-validation.id integration scenario:
CREATE TABLE → CREATE TRIGGER → DROP TABLE blocked → DROP TRIGGER →
DROP TABLE succeeds. Mirrors the MV pattern the existing
k8s-validation.id scenarios use.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Generalize dependency labels/checker beyond pipelines
PipelineDependency{Labels,Checker} and K8sPipelineDependencyValidator
were specific to Pipeline CRDs in name only — TableTrigger CRDs now
wear the same depends-on labels and annotations. Renamed to
DependencyLabels, DependencyChecker, and K8sDependencyValidator.
Collapsed the labels API: DependencyLabels now exposes a single
stamp(V1ObjectMeta, Collection<Source>, Sink) that writes both the
depends-on labels and the directional annotations. K8sTriggerDeployer
had reimplemented this inline in two places; both call sites now
collapse to one stamp() call.
Also dropped the 5-arg Trigger constructor; callers must pass a sink
(or null) explicitly.
* refactoring
* Allow multiple sources/sinks (future proofing)
* Fix checkstyle
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: ryannedolan <1387539+ryannedolan@users.noreply.github.com>1 parent f8e29a2 commit f8d9685
27 files changed
Lines changed: 975 additions & 672 deletions
File tree
- hoptimator-api/src
- main/java/com/linkedin/hoptimator
- test/java/com/linkedin/hoptimator
- hoptimator-jdbc/src/main/java/com/linkedin/hoptimator/jdbc
- hoptimator-k8s/src
- main/java/com/linkedin/hoptimator/k8s
- test
- java/com/linkedin/hoptimator/k8s
- resources
- hoptimator-kafka/src/test/resources
- hoptimator-logical/src
- main/java/com/linkedin/hoptimator/logical
- test
- java/com/linkedin/hoptimator/logical
- resources
Lines changed: 24 additions & 23 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1 | 1 | | |
2 | 2 | | |
3 | | - | |
| 3 | + | |
4 | 4 | | |
5 | 5 | | |
6 | 6 | | |
7 | 7 | | |
8 | 8 | | |
9 | 9 | | |
| 10 | + | |
10 | 11 | | |
11 | 12 | | |
12 | | - | |
13 | 13 | | |
14 | 14 | | |
| 15 | + | |
| 16 | + | |
15 | 17 | | |
16 | | - | |
17 | | - | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
18 | 25 | | |
19 | 26 | | |
20 | | - | |
21 | 27 | | |
22 | 28 | | |
| 29 | + | |
| 30 | + | |
23 | 31 | | |
24 | 32 | | |
25 | 33 | | |
26 | 34 | | |
27 | 35 | | |
28 | 36 | | |
29 | | - | |
30 | | - | |
31 | | - | |
32 | | - | |
33 | 37 | | |
34 | 38 | | |
35 | 39 | | |
| |||
38 | 42 | | |
39 | 43 | | |
40 | 44 | | |
41 | | - | |
42 | | - | |
43 | | - | |
44 | | - | |
45 | | - | |
46 | | - | |
47 | | - | |
48 | | - | |
49 | | - | |
50 | | - | |
51 | | - | |
52 | 45 | | |
53 | 46 | | |
54 | 47 | | |
55 | 48 | | |
56 | | - | |
57 | | - | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
58 | 58 | | |
59 | 59 | | |
60 | 60 | | |
61 | 61 | | |
62 | | - | |
| 62 | + | |
| 63 | + | |
63 | 64 | | |
64 | 65 | | |
Lines changed: 1 addition & 1 deletion
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
67 | 67 | | |
68 | 68 | | |
69 | 69 | | |
70 | | - | |
| 70 | + | |
71 | 71 | | |
72 | 72 | | |
73 | 73 | | |
| |||
Lines changed: 21 additions & 8 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
2 | 2 | | |
3 | 3 | | |
4 | 4 | | |
| 5 | + | |
5 | 6 | | |
6 | 7 | | |
7 | 8 | | |
8 | 9 | | |
| 10 | + | |
9 | 11 | | |
10 | 12 | | |
11 | 13 | | |
| |||
15 | 17 | | |
16 | 18 | | |
17 | 19 | | |
18 | | - | |
| 20 | + | |
| 21 | + | |
19 | 22 | | |
20 | 23 | | |
21 | 24 | | |
22 | | - | |
23 | 25 | | |
24 | 26 | | |
25 | | - | |
26 | | - | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
27 | 32 | | |
28 | 33 | | |
29 | 34 | | |
30 | | - | |
31 | | - | |
32 | | - | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
33 | 39 | | |
34 | 40 | | |
35 | 41 | | |
36 | 42 | | |
37 | | - | |
| 43 | + | |
| 44 | + | |
38 | 45 | | |
39 | 46 | | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
40 | 53 | | |
Lines changed: 21 additions & 3 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
229 | 229 | | |
230 | 230 | | |
231 | 231 | | |
232 | | - | |
| 232 | + | |
| 233 | + | |
233 | 234 | | |
234 | 235 | | |
235 | 236 | | |
| |||
255 | 256 | | |
256 | 257 | | |
257 | 258 | | |
| 259 | + | |
| 260 | + | |
| 261 | + | |
| 262 | + | |
| 263 | + | |
| 264 | + | |
| 265 | + | |
| 266 | + | |
| 267 | + | |
| 268 | + | |
| 269 | + | |
| 270 | + | |
| 271 | + | |
| 272 | + | |
| 273 | + | |
| 274 | + | |
| 275 | + | |
258 | 276 | | |
259 | 277 | | |
260 | 278 | | |
| |||
307 | 325 | | |
308 | 326 | | |
309 | 327 | | |
310 | | - | |
| 328 | + | |
311 | 329 | | |
312 | 330 | | |
313 | 331 | | |
| |||
344 | 362 | | |
345 | 363 | | |
346 | 364 | | |
347 | | - | |
| 365 | + | |
348 | 366 | | |
349 | 367 | | |
350 | 368 | | |
| |||
Lines changed: 141 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
0 commit comments