Skip to content

Commit 2dded63

Browse files
dieppadieppa
authored
docs: audit store vs target system conceptual description (#67)
1 parent 0c7b807 commit 2dded63

2 files changed

Lines changed: 226 additions & 207 deletions

File tree

‎docs/flamingock-library-config/changeunits-deep-dive.md‎

Lines changed: 110 additions & 125 deletions
Original file line numberDiff line numberDiff line change
@@ -3,173 +3,158 @@ title: ChangeUnits Deep Dive
33
sidebar_position: 3
44
---
55

6-
## Clarifying changeUnits
7-
8-
A **ChangeUnit** is the atomic, versioned unit of change in Flamingock. It encapsulates logic to modify an external system (the [**target system**](../overview/audit-store-vs-target-system.md)) and provides metadata and rollback capability. ChangeUnits are discovered and executed in a defined order to ensure deterministic, auditable changes.
9-
10-
### What a changeUnit Is
11-
- **Self-contained change**
12-
Each ChangeUnit includes:
13-
- A unique `id` (unique across the entire application)
14-
- An `order` determining execution sequence
15-
- An optional `author` and `description`
16-
- An `@Execution` method (or template) with the change logic
17-
- A `@RollbackExecution` method (or template) with compensating logic
18-
- A `transactional` flag (default `true`) indicating if Flamingock will attempt to wrap execution and audit in a single transaction
19-
20-
- **Versioned and Auditable**
21-
ChangeUnits live in your source code or resources, and their execution is recorded in the [**audit store**](../overview/audit-store-vs-target-system.md) to:
22-
- Prevent duplicate executions
23-
- Track history (who ran which change and when)
24-
- Drive rollbacks and “undo” operations
25-
26-
### What a changeUnit is not
27-
- **Not a long-running job**
28-
ChangeUnits should complete promptly. Flamingock needs to know the result (success or failure) before proceeding. Long-running or asynchronous operations can lead to unexpected behavior or retries.
29-
- **Not a general-purpose script**
30-
While ChangeUnits run code, they are not intended for arbitrary scripting. Their role is to apply deterministic, idempotent changes that evolve your target systems in sync with your application.
6+
# ChangeUnits Deep Dive
317

32-
---
8+
## 1. Introduction: Understanding ChangeUnits
339

34-
## ChangeUnit properties
10+
A **ChangeUnit** is the atomic, versioned, self-contained unit of change in Flamingock.
11+
It encapsulates logic to evolve [**target systems**](../overview/audit-store-vs-target-system.md) safely, deterministically, and with complete auditability.
3512

36-
Every ChangeUnit must define:
37-
- `id` (String): Unique across all ChangeUnits in the application.
38-
- `order` (String or numeric): Defines execution order (evaluated lexicographically or numerically).
39-
- `author` (String): Who is responsible for the change.
40-
- `description` (String, optional): Brief explanation of the change.
41-
- `transactional` (boolean, default `true`): Whether Flamingock will attempt to wrap the change and audit insert in one transaction (if the target system and audit store support transactions).
13+
**Key characteristics:**
14+
- Executed in sequence based on their `order`
15+
- Recorded in the audit store to prevent duplicate execution
16+
- Safe by default: if Flamingock is uncertain about a change's outcome, it stops and requires manual intervention
17+
- Each ChangeUnit runs exactly once per system
4218

4319
---
4420

45-
## Types of changeUnits
21+
## 2. Structure of a ChangeUnit
22+
23+
### Required Properties
24+
- **`id`**: Unique identifier across all ChangeUnits in the application
25+
- **`order`**: Execution sequence (must use zero-padded format like `0001`, `0002`, `_0001_ChangeName`)
26+
- **`author`**: Who is responsible for this change
27+
28+
### Optional Properties
29+
- **`description`**: Brief explanation of what the change does
30+
- **`transactional`** (default `true`): Only relevant if the target system supports transactions. Has no effect on non-transactional systems like S3 or Kafka.
31+
32+
### Required Annotations and Methods
33+
- **`@TargetSystem`**: Specifies which system this change affects
34+
- **`@ChangeUnit`**: Marks the class as a ChangeUnit
35+
- **`@Execution`**: The method containing your change logic
36+
- **`@RollbackExecution`**: The method to undo the change (required for safety and governance)
37+
38+
> **Note:** Rollback is important because in **non-transactional systems**, it's be used to revert changes if execution fails. In **all systems**, rollback is essential for undo operations (via CLI or UI).
4639
47-
ChangeUnits can be defined based on two approaches: code-based and [template-based](../templates/templates-introduction.md)
40+
## 3. Types of ChangeUnits
4841

49-
### Code-based changeUnits
50-
Code-based ChangeUnits are written in Java (or Kotlin/Groovy) with annotations:
42+
### Code-based ChangeUnits
43+
Written in Java (or Kotlin/Groovy) with annotations. Best for **specific jobs** or when you need a **flexibility window** that isn’t covered by an existing template.
44+
45+
This approach gives you full programmatic control, making it the fallback option when no reusable template exists for your use case.
5146

5247
```java
53-
@ChangeUnit(
54-
id = "create_s3_bucket",
55-
order = "0001",
56-
author = "dev-team",
57-
transactional = false,
58-
description = "Create my-app-bucket S3 bucket")
59-
public class _0001_CreateS3BucketChange {
60-
61-
@Execution
62-
public void execute(S3Client s3Client) {
63-
s3Client.createBucket("my-app-bucket");
64-
}
65-
66-
@RollbackExecution
67-
public void rollback(S3Client s3Client) {
68-
s3Client.deleteBucket("my-app-bucket");
69-
}
48+
@TargetSystem("user-database")
49+
@ChangeUnit(id = "add-user-status", order = "0001", author = "dev-team")
50+
public class _0001_AddUserStatus {
51+
52+
@Execution
53+
public void execute(MongoDatabase database) {
54+
database.getCollection("users")
55+
.updateMany(new Document(),
56+
new Document("$set", new Document("status", "active")));
57+
}
58+
59+
@RollbackExecution
60+
public void rollback(MongoDatabase database) {
61+
database.getCollection("users")
62+
.updateMany(new Document(),
63+
new Document("$unset", new Document("status", "")));
64+
}
7065
}
7166
```
7267

73-
#### Discoverability & execution
74-
- **Location**: Files must reside in a source package scanned by Flamingock (default: `src/main/java`).
75-
- **Naming**: Class names should match `_ORDER_name` (e.g., `_0001_CreateS3BucketChange`) to simplify ordering and visibility.
76-
- **Dependencies**: Flamingock injects dependencies (e.g., `S3Client`, `MongoClient`) via Spring or builder-based DI.
68+
### Template-based ChangeUnits
69+
Template-based ChangeUnits use YAML or JSON definitions. They are especially useful for **repetitive or parameterized operations**, where the same logic can to be applied multiple times with different configurations.
70+
71+
- The execution logic is encapsulated in a **template** (provided by Flamingock, a contributor, or created by you).
72+
- Each ChangeUnit then supplies its own configuration to apply that logic consistently.
73+
- This approach ensures **immutability** (the YAML/JSON file itself represents the change) and makes it easier to **reuse proven patterns**.
7774

78-
### Template-based changeUnits
79-
Template-based ChangeUnits use YAML or JSON definitions. Example (SQL DDL):
8075

8176
```yaml
82-
# /src/main/resources/_0003_add_status_column.yml
77+
# File: _0002_add_status_column.yml
8378
id: add_status_column
84-
order: 0003
79+
order: "0002"
8580
author: "db-team"
86-
description: "Add 'status' column to 'orders' table"
81+
description: "Add status column to orders table"
8782
templateName: sql-template
8883
templateConfiguration:
8984
executionSql: |
90-
ALTER TABLE orders ADD COLUMN status VARCHAR(20);
85+
ALTER TABLE orders ADD COLUMN status VARCHAR(20) DEFAULT 'pending';
9186
rollbackSql: |
9287
ALTER TABLE orders DROP COLUMN status;
9388
```
9489
95-
#### Discoverability & execution
96-
- **Location**: While Flamingock will scan src/main/resources by default, we **strongly recommend** placing template files in the same code‐package/directory as your code‐based ChangeUnits.
97-
- This ensures that both code‐based and template‐based ChangeUnits live side by side for visibility and immutability.
98-
- **Naming**: File names should follow `_ORDER_name.yml` or `_ORDER_name.json`.
99-
- **Advantages**:
100-
- Easier immutability: The YAML/JSON file itself represents the change, avoiding modifications in code.
101-
- Better for simple, repeatable tasks (e.g., SQL DDL).
90+
Both types follow the same execution model and provide the same safety guarantees.
10291
103-
---
104-
105-
[//]: # (## How ChangeUnits Are Discovered & Executed)
92+
## 4. Naming & Discoverability
10693
107-
[//]: # ()
108-
[//]: # (Flamingock uses **classpath scanning** to locate ChangeUnits:)
94+
### Enforced Naming Convention
95+
All ChangeUnit files (both code and templates) **must** follow this pattern:
10996
110-
[//]: # ()
111-
[//]: # (- **Code-based**: Scans specified packages for classes annotated with `@ChangeUnit` or `@Change`.)
97+
- **Format**: `_XXXX_DescriptiveName`
98+
- **Order**: Must be at least 4 digits, zero-padded (e.g., `0001`, `0002`, `0100`)
99+
- **Examples**:
100+
- Code: `_0001_CreateUserIndexes.java`
101+
- Template: `_0002_AddStatusColumn.yml`
112102

113-
[//]: # (- **Template-based**: Scans specified resource directories for YAML/JSON files matching pattern `_.*\.(yml|json)`.)
103+
### Why This Convention?
104+
- **Visibility**: Easy to see execution order at a glance
105+
- **Immutability**: Clear versioning prevents accidental modifications
106+
- **Deterministic ordering**: Ensures consistent execution across environments
114107

115-
[//]: # ()
116-
[//]: # (Flamingock builds a **pipeline** of stages and executes ChangeUnits in ascending order based on `order`.)
108+
### File Locations
109+
- **Code-based**: Place in packages scanned by Flamingock (default: `src/main/java`)
110+
- **Template-based**: Place in `src/main/resources` or preferably alongside code-based ChangeUnits
111+
- **Recommendation**: Keep all ChangeUnits (code and templates) in the same package/directory for better organization
117112

118-
[//]: # ()
119-
[//]: # (---)
113+
## 5. Transactional Behavior
120114

121-
## Considerations
115+
- **Transactional target systems** (e.g., MongoDB, PostgreSQL): operations run within a transaction **unless you explicitly set `transactional = false`**.
116+
- **Non-transactional target systems** (e.g., S3, Kafka): the `transactional` flag has no effect — operations are applied without transactional guarantees.
122117

123-
### Transactional behavior
124-
- **Transactional changes (default)**: When the target system and audit store share a transactional context (e.g., MongoDB CE), Flamingock wraps both in a single transaction.
125-
- **Non-transactional changes**: `transactional = false`. Flamingock executes `@Execution` and, upon success, writes to the audit store. If `@Execution` fails, Flamingock invokes `@RollbackExecution`. See [transactions page](../flamingock-library-config/transactions.md)
126-
118+
Some operations may require setting `transactional = false` even in databases:
119+
- DDL operations (e.g., CREATE INDEX, ALTER TABLE)
120+
- Large bulk operations that exceed transaction limits
121+
- Cross-system changes spanning multiple databases
127122

128-
### Immutability
129-
- **Code-based**: Once committed and possibly executed, avoid modifying the class. Instead, introduce a new ChangeUnit for evolution.
130-
- **Template-based**: Treat the file as immutable. Modifying an existing template breaks history ordering — use new template files for new changes.
123+
➡️ To understand how to define and configure **target systems**, see [Target System Configuration](../overview/audit-store-vs-target-system.md)
131124

132-
### Audit store constraints
133-
- **Single audit store per application**: All ChangeUnits in one application write to the same audit store.
134-
- **Audit store integrity**: Do not manually modify audit records in the audit store; this can corrupt Flamingock’s state. Use CLI/UI for supported modifications.
125+
## 6. Default Safety & Recovery
135126

136-
### Idempotency
137-
- ChangeUnits should be idempotent or safe to re-run. Flamingock retries failed ChangeUnits on next startup. If a non-transactional ChangeUnit partially succeeded, ensure it can handle multiple executions or include appropriate guards.
138-
139-
---
127+
**Flamingock's core principle**: If a ChangeUnit execution result is uncertain, Flamingock stops and requires manual intervention. This prevents silent data corruption.
140128

141-
## Best practices
129+
**What this means:**
130+
- If a change fails, Flamingock halts execution
131+
- The issue is recorded in the audit store
132+
- Manual investigation and resolution is required via CLI (or Cloud UI in Cloud Edition)
142133

143-
**Name and location conventions***
144-
- Place both code-based and template-based ChangeUnits in the same source package/directory for visibility and immutability.
145-
- Use filenames or class names prefixed with the zero-padded order (e.g., `_0001_create_s3_bucket.java` or `_0001_create_s3_bucket.yaml`).
134+
➡️ **For advanced recovery strategies**, see [Recovery Strategies](../recovery-and-safety/recovery-strategies.md)
146135

147-
- **Always provide rollback**
148-
Even for transactional ChangeUnits, implement `@RollbackExecution` so CLI “undo” operations work smoothly.
136+
## 7. Best Practices
149137

150-
- **Template-based changeUnits for simplicity and immutability**
151-
Favor templated ChangeUnits (YAML/JSON) for routine, repeatable tasks—such as SQL DDL, configuration toggles, or small API calls. Templates are inherently immutable (being a static file), making it easier to adhere to versioning best practices.
138+
### Core Principles
139+
- **Treat ChangeUnits as immutable**: Once deployed, never modify existing ChangeUnits. Create new ones for corrections.
140+
- **Always provide @RollbackExecution**: Important for CLI undo operations and recovery scenarios.
141+
- **Keep scope focused**: One ChangeUnit should address one logical change.
152142

153-
- **Use Flamingock’s batching feature for long-running operations**(coming soon)
154-
For ChangeUnits that process large workloads (e.g., migrating millions of rows), leverage Flamingock’s built-in batching mechanism. Define a single ChangeUnit that iterates through data in batches; Flamingock will mark it as complete only when all batches succeed, and will resume from the last processed batch on retry.
143+
### Technical Guidelines
144+
- **Make operations idempotent when possible**: Try to design changes that can be safely re-run.
145+
- **Test both execution and rollback**: Include ChangeUnit testing in your CI/CD pipeline.
146+
- **Follow naming conventions**: Use the `_XXXX_DescriptiveName` pattern consistently.
155147

156-
- **Inject minimal dependencies**
157-
Only inject what you need (e.g., clients, DAOs). Avoid injecting large application contexts within ChangeUnits.
148+
### Organizational Best Practices
149+
- **Clear authorship**: Always specify the `author` for accountability.
150+
- **Version control discipline**: Review ChangeUnits in pull requests like any critical code.
151+
- **Document complex changes**: Use the `description` field to explain non-obvious logic.
152+
- **Maintain change logs**: Keep a high-level record of what changes were made when.
158153

159-
- **Write clear descriptions**
160-
Use the `description` property to explain the purpose and impact of each ChangeUnit.
161154

162-
- **Implement idempotency**
163-
For non-transactional operations (e.g., deleting an S3 bucket), wrap calls in checks (e.g., “if exists”) to handle re-runs gracefully.
164-
165-
- **Immutable operations**
166-
Once a ChangeUnit is applied to any environment, treat its code or template as immutable. For corrections, create new ChangeUnits rather than editing old ones.
167-
168-
- **Explicit ordering**
169-
Declare a clear, numeric `order` for each ChangeUnit. Relying on implicit or alphabetical ordering can introduce hidden dependencies and make debugging deployment issues difficult.
170-
171-
- **Audit store hygiene**
172-
Never manually edit or delete records in the audit store. Direct modifications can corrupt Flamingock’s internal state and lead to unpredictable behavior or data loss. If you need to correct audit data, use Flamingock’s supported operations (CLI or UI) or follow documented recovery procedures.
155+
---
173156

174-
- **Documentation and metadata**
175-
Use the `author` and `description` (if available) fields to document the intent of each ChangeUnit. This metadata helps teams understand why a change was made and by whom—critical for code reviews and compliance audits.
157+
**Next Steps:**
158+
- Learn about [dependency injection](./changeunit-dependency-injection.md) in ChangeUnits
159+
- Explore [template-based ChangeUnits](../templates/templates-introduction.md) for declarative changes
160+
- Understand [advanced recovery strategies](../recovery-and-safety/recovery-strategies.md) for production scenarios

0 commit comments

Comments
 (0)