Skip to content

Commit 94c8790

Browse files
CornerstoneCodeCornerstoneCode
committed
updated docs
1 parent a662da8 commit 94c8790

2 files changed

Lines changed: 260 additions & 24 deletions

File tree

‎EntityFrameworkCore.Sqlite.Concurrency/EFCore.Sqlite.Concurrency.csproj‎

Lines changed: 34 additions & 24 deletions
Original file line numberDiff line numberDiff line change
@@ -11,7 +11,7 @@
1111

1212
<!-- Core Package Identity -->
1313
<PackageId>EntityFrameworkCore.Sqlite.Concurrency</PackageId>
14-
<Version>10.0.2</Version>
14+
<Version>10.0.3</Version>
1515

1616
<!-- Core Metadata for Trust & Recognition -->
1717
<Authors>Mike Gotfryd</Authors>
@@ -34,29 +34,39 @@
3434
<!-- 3. STRUCTURED RELEASE NOTES -->
3535
<PackageReleaseNotes>
3636
<![CDATA[
37-
🚀 **v10.0.2 - Initial Stable Release: Production-Ready SQLite Concurrency & Performance**
38-
39-
This first major release transforms SQLite into a robust database for concurrent .NET applications by fixing core limitations of the standard provider.
40-
41-
**✅ SOLVES: Concurrency & Locking Errors**
42-
• **Eliminates `SQLITE_BUSY` / "database is locked" errors** with automatic, application-level write serialization.
43-
• **Guarantees 100% write reliability** under any multi-threaded load.
44-
45-
**⚡ DELIVERS: Exceptional Performance**
46-
• **Achieves up to 10x faster bulk inserts** vs. standard `SaveChanges()` through intelligent batching.
47-
• **Enables true parallel read scaling** with non-blocking connections.
48-
• **Optimizes all interactions** (connections, transactions, WAL mode) for maximum throughput.
49-
50-
**🧩 PROVIDES: Seamless Developer Experience**
51-
• **Drop-in replacement** – change `UseSqlite()` to `UseSqliteWithConcurrency()`.
52-
• **Full EF Core compatibility** – all existing DbContexts, models, and LINQ queries work unchanged.
53-
• **Simplifies complex logic** – abstracts retry patterns, lock management, and connection pooling.
54-
55-
**🏗️ ENSURES: Enterprise-Grade Robustness**
56-
• Built-in production resilience with exponential backoff retry and crash-safe transactions.
57-
• Targets the modern .NET ecosystem with first-class support for **.NET 10** and **Entity Framework Core 10**.
58-
59-
Get started in one line. Stop compromising on SQLite reliability and speed.
37+
v10.0.3 — SQLITE_BUSY_SNAPSHOT fix, IDbContextFactory support, structured logging
38+
39+
BUGS FIXED
40+
• SQLITE_BUSY_SNAPSHOT (extended code 517) now correctly restarts the full operation lambda
41+
instead of retrying the same statement — the only correct fix for a stale WAL read snapshot.
42+
• Exponential backoff now uses full jitter ([baseDelay, 2×baseDelay]) to prevent thundering
43+
herd when multiple threads contend simultaneously.
44+
• Cache=Shared in the connection string now throws ArgumentException at startup — it silently
45+
broke WAL mode semantics in prior versions.
46+
• Invalid SqliteConcurrencyOptions values (MaxRetryAttempts ≤ 0, negative BusyTimeout, etc.)
47+
now throw ArgumentOutOfRangeException at startup instead of silently misbehaving.
48+
49+
NEW FEATURES
50+
• AddConcurrentSqliteDbContextFactory<T> — registers IDbContextFactory<T> with all concurrency
51+
settings. Use this for Task.WhenAll, background services, Channel<T> consumers, and any
52+
workload that creates concurrent database operations. DbContext is not thread-safe; the factory
53+
pattern gives each concurrent flow its own independent instance.
54+
• Structured logging: pass ILoggerFactory (or let DI resolve it) to get Warning logs for
55+
SQLITE_BUSY/SQLITE_BUSY_SNAPSHOT events, Error logs for SQLITE_LOCKED, and Debug logs for
56+
BEGIN IMMEDIATE upgrades — all through your existing logging pipeline.
57+
• GetWalCheckpointStatusAsync — runs PRAGMA wal_checkpoint(PASSIVE) and returns a typed
58+
WalCheckpointStatus with IsBusy, TotalWalFrames, CheckpointedFrames, and CheckpointProgress.
59+
Call periodically to detect long-running readers blocking WAL reclamation before it degrades
60+
read performance.
61+
• TryReleaseMigrationLockAsync — detects and optionally clears a stale __EFMigrationsLock
62+
row left behind by a crashed migration process. Prevents indefinite blocking on Database.Migrate()
63+
in multi-instance deployments.
64+
• SynchronousMode option — configures PRAGMA synchronous (Off / Normal / Full / Extra).
65+
Default remains Normal (recommended for WAL: safe after app crash, fast writes).
66+
• UpgradeTransactionsToImmediate option — opt out of the BEGIN → BEGIN IMMEDIATE rewrite
67+
if you manage write transactions explicitly yourself. Default remains true.
68+
69+
NO BREAKING CHANGES — all existing call sites compile and behave correctly without modification.
6070
]]>
6171
</PackageReleaseNotes>
6272
<!-- =============================================== -->

‎EntityFrameworkCore.Sqlite.Concurrency/doc/v10_0_3.md‎

Lines changed: 226 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,226 @@
1+
# EntityFrameworkCore.Sqlite.Concurrency — v10.0.3 Release Notes
2+
3+
## Fix `SQLITE_BUSY`, `SQLITE_BUSY_SNAPSHOT`, and EF Core Thread-Safety in One Update
4+
5+
This release closes the remaining correctness gaps in how the library handles SQLite locking errors, adds the `IDbContextFactory<T>` registration pattern that EF Core recommends for concurrent workloads, and introduces structured diagnostics for production observability.
6+
7+
If you have hit any of the following in your .NET / EF Core application, this update directly addresses them:
8+
9+
- `Microsoft.Data.Sqlite.SqliteException: SQLite Error 5: 'database is locked'`
10+
- `SQLITE_BUSY_SNAPSHOT` causing unexpected failures mid-transaction
11+
- `InvalidOperationException: A second operation was started on this context` when using `Task.WhenAll`
12+
- Retry storms where all threads wake simultaneously and re-contend for the write lock
13+
- Silent misconfiguration (invalid `MaxRetryAttempts`, `Cache=Shared` + WAL conflicts)
14+
- No visibility into why busy errors occur in production
15+
16+
---
17+
18+
## Bug Fixes
19+
20+
### SQLITE_BUSY_SNAPSHOT now correctly restarts the operation (not just retries)
21+
22+
**Error extended code:** `517` (`SQLITE_BUSY | (2 << 8)`)
23+
24+
This was the most impactful correctness bug. All `SQLITE_BUSY` variants were previously retried the same way — waiting a backoff delay and calling the same statement again. `SQLITE_BUSY_SNAPSHOT` has fundamentally different semantics: the connection's WAL read snapshot became stale after another writer committed. Re-executing the same statement produces the same error. The only correct fix is to roll back the entire transaction and **restart the operation from scratch** so that any data read inside it is re-queried against the current snapshot.
25+
26+
`ExecuteWriteAsync` and `ExecuteWithRetryAsync` now correctly distinguish this case using `SqliteException.SqliteExtendedErrorCode` and restart the full operation lambda.
27+
28+
### Full jitter added to exponential backoff (thundering herd prevention)
29+
30+
Pure exponential backoff (`100ms × 2^n`) causes every contending thread to wake at approximately the same time and immediately re-contend for the write lock — a classic thundering herd. Retry delays are now randomized in `[baseDelay, 2×baseDelay]` so threads spread out naturally without coordination.
31+
32+
### `Cache=Shared` incompatibility with WAL detected at startup
33+
34+
`Cache=Shared` in a SQLite connection string enables a shared page cache across connections. This conflicts with WAL mode's snapshot isolation model, where each connection must independently track read snapshot boundaries. The combination silently corrupts WAL semantics and was previously accepted without warning. It now throws a descriptive `ArgumentException` at startup:
35+
36+
```
37+
Cache=Shared is incompatible with WAL mode and cannot be used with ThreadSafeEFCore.SQLite.
38+
Remove 'Cache=Shared' from your connection string. Connection pooling (Pooling=true) is
39+
enabled automatically and provides efficient connection reuse without the WAL incompatibility.
40+
```
41+
42+
### Startup validation for `SqliteConcurrencyOptions`
43+
44+
Invalid option values (e.g. `MaxRetryAttempts = 0`, negative `BusyTimeout`) previously produced silent incorrect behavior. `Validate()` is now called during `UseSqliteWithConcurrency` and throws `ArgumentOutOfRangeException` with a descriptive message at startup.
45+
46+
---
47+
48+
## New Features
49+
50+
### `AddConcurrentSqliteDbContextFactory<T>` — correct EF Core pattern for concurrent workloads
51+
52+
A `DbContext` instance is not thread-safe and must not be shared across concurrent operations. EF Core's recommended pattern for concurrent workloads — background services, `Task.WhenAll`, `Parallel.ForEachAsync`, `Channel<T>` consumers — is `IDbContextFactory<T>`, which creates an independent context per concurrent flow.
53+
54+
The new registration method wires this up with all concurrency settings and auto-injects `ILoggerFactory`:
55+
56+
```csharp
57+
// Program.cs — for concurrent workloads (hosted services, background queues, Task.WhenAll)
58+
builder.Services.AddConcurrentSqliteDbContextFactory<AppDbContext>("Data Source=app.db");
59+
60+
// The existing method remains for request-scoped use (controllers, Razor Pages, Blazor Server)
61+
builder.Services.AddConcurrentSqliteDbContext<AppDbContext>("Data Source=app.db");
62+
```
63+
64+
Inject `IDbContextFactory<AppDbContext>` and call `CreateDbContext()` per concurrent operation:
65+
66+
```csharp
67+
public class ReportGenerationService
68+
{
69+
private readonly IDbContextFactory<AppDbContext> _factory;
70+
71+
public ReportGenerationService(IDbContextFactory<AppDbContext> factory)
72+
=> _factory = factory;
73+
74+
public async Task GenerateAllReportsAsync(IEnumerable<int> reportIds, CancellationToken ct)
75+
{
76+
// Each task gets its own context — no EF thread-safety violation,
77+
// and ThreadSafeEFCore.SQLite serializes the writes at the SQLite level.
78+
var tasks = reportIds.Select(async id =>
79+
{
80+
await using var db = _factory.CreateDbContext();
81+
var data = await db.ReportData.Where(r => r.ReportId == id).ToListAsync(ct);
82+
var result = ComputeReport(data);
83+
db.Reports.Add(result);
84+
await db.SaveChangesAsync(ct);
85+
});
86+
87+
await Task.WhenAll(tasks);
88+
}
89+
}
90+
```
91+
92+
### Structured logging for `SQLITE_BUSY*` events
93+
94+
Pass an `ILoggerFactory` (or let DI resolve it automatically through `AddConcurrentSqliteDbContext`/`AddConcurrentSqliteDbContextFactory`) and the interceptor emits structured log entries:
95+
96+
| Event | Log Level | Message |
97+
|---|---|---|
98+
| `SQLITE_BUSY` / `SQLITE_BUSY_RECOVERY` | `Warning` | Includes command text and retry attempt number |
99+
| `SQLITE_BUSY_SNAPSHOT` | `Warning` | Identifies stale snapshot, includes command text |
100+
| `SQLITE_LOCKED` | `Error` | Same-connection conflict — indicates an application bug |
101+
| `BEGIN IMMEDIATE` upgrade | `Debug` | Logged when a deferred `BEGIN` is rewritten |
102+
103+
Manual wiring (non-DI scenario):
104+
105+
```csharp
106+
options.UseSqliteWithConcurrency("Data Source=app.db", o =>
107+
{
108+
o.LoggerFactory = loggerFactory;
109+
});
110+
```
111+
112+
### WAL checkpoint health monitoring (`GetWalCheckpointStatusAsync`)
113+
114+
Long-running read transactions block WAL checkpoint completion, causing the WAL file to grow unboundedly and degrade read performance. Call this periodically to detect pressure before it becomes a problem:
115+
116+
```csharp
117+
var status = await SqliteConnectionEnhancer.GetWalCheckpointStatusAsync(connection);
118+
119+
if (status.IsBusy && status.TotalWalFrames > 5000)
120+
logger.LogWarning(
121+
"WAL checkpoint blocked. {Total} frames, {Checkpointed} checkpointed ({Progress:F1}%). " +
122+
"A long-running read transaction may be preventing WAL reclamation.",
123+
status.TotalWalFrames, status.CheckpointedFrames, status.CheckpointProgress);
124+
```
125+
126+
### Migration lock recovery (`TryReleaseMigrationLockAsync`)
127+
128+
EF Core uses a `__EFMigrationsLock` table to serialize concurrent migrations. If a migration process crashes after acquiring the lock, subsequent calls to `Database.Migrate()` block indefinitely. The new helper detects and clears stale locks:
129+
130+
```csharp
131+
// Run once at startup before Database.MigrateAsync()
132+
var connection = db.Database.GetDbConnection();
133+
await connection.OpenAsync();
134+
135+
var wasStale = await SqliteConnectionEnhancer.TryReleaseMigrationLockAsync(connection);
136+
if (wasStale)
137+
logger.LogWarning("Stale EF migration lock detected and cleared. Proceeding with migration.");
138+
139+
await db.Database.MigrateAsync();
140+
```
141+
142+
Pass `release: false` to check without modifying the database (diagnostics only).
143+
144+
### Configurable `SynchronousMode` option
145+
146+
`PRAGMA synchronous` is now configurable instead of hardcoded. Controls the durability vs. write-speed trade-off:
147+
148+
| Mode | Durability | Use case |
149+
|---|---|---|
150+
| `Off` | Lowest — data loss on OS crash | Bulk import scratch databases |
151+
| `Normal` | **Default** — safe after app crash; last commit may be lost on power loss | Most production apps with WAL |
152+
| `Full` | Safe after power loss — extra `fsync` on every commit | Financial records, audit logs |
153+
| `Extra` | Strongest — guards against certain filesystem clock skew bugs | High-compliance environments |
154+
155+
```csharp
156+
options.UseSqliteWithConcurrency("Data Source=app.db", o =>
157+
{
158+
o.SynchronousMode = SqliteSynchronousMode.Full; // power-loss safe
159+
});
160+
```
161+
162+
### `UpgradeTransactionsToImmediate` opt-out
163+
164+
The interceptor rewrites deferred `BEGIN` to `BEGIN IMMEDIATE` by default, which prevents `SQLITE_BUSY_SNAPSHOT` mid-transaction. Power users who manage write transactions explicitly can disable this:
165+
166+
```csharp
167+
options.UseSqliteWithConcurrency("Data Source=app.db", o =>
168+
{
169+
o.UpgradeTransactionsToImmediate = false; // you manage BEGIN IMMEDIATE yourself
170+
});
171+
```
172+
173+
---
174+
175+
## Breaking Changes
176+
177+
None. This release is fully backwards-compatible:
178+
179+
- All existing `UseSqliteWithConcurrency`, `AddConcurrentSqliteDbContext`, and `ExecuteWithRetryAsync` call sites compile and behave correctly without modification.
180+
- `AddConcurrentSqliteDbContextFactory` is additive.
181+
- `Cache=Shared` rejection is technically a new startup error, but `Cache=Shared` + WAL was already producing incorrect behavior silently — this makes the failure explicit and actionable.
182+
- `SynchronousMode` defaults to `Normal`, which was the hardcoded value in prior versions.
183+
- `UpgradeTransactionsToImmediate` defaults to `true`, preserving the prior behavior.
184+
185+
---
186+
187+
## Configuration Reference
188+
189+
| Option | Default | Description |
190+
|--------|---------|-------------|
191+
| `BusyTimeout` | 30 s | `PRAGMA busy_timeout` — SQLite retries lock acquisition internally for this duration before surfacing `SQLITE_BUSY` to the application. |
192+
| `MaxRetryAttempts` | 3 | Application-level retry attempts after `SQLITE_BUSY*`, with exponential backoff and full jitter. |
193+
| `CommandTimeout` | 300 s | EF Core SQL command timeout. |
194+
| `WalAutoCheckpoint` | 1000 pages | `PRAGMA wal_autocheckpoint` — triggers an automatic passive checkpoint after this many WAL frames (~4 MB at the default 4 096-byte page size). Set to `0` to disable. |
195+
| `SynchronousMode` | `Normal` | `PRAGMA synchronous` — durability vs. write-speed trade-off. `Normal` is recommended for WAL mode. |
196+
| `UpgradeTransactionsToImmediate` | `true` | Rewrites `BEGIN`/`BEGIN TRANSACTION` to `BEGIN IMMEDIATE` to prevent `SQLITE_BUSY_SNAPSHOT` mid-transaction. |
197+
| `LoggerFactory` | `null` | Resolved automatically from DI when using `AddConcurrentSqliteDbContext` / `AddConcurrentSqliteDbContextFactory`. |
198+
199+
---
200+
201+
## Upgrade Guide
202+
203+
```csharp
204+
// 1. For request-scoped use (controllers, Razor Pages) — no change needed:
205+
builder.Services.AddConcurrentSqliteDbContext<AppDbContext>("Data Source=app.db");
206+
207+
// 2. For concurrent workloads — switch to the factory:
208+
// Before:
209+
builder.Services.AddConcurrentSqliteDbContext<AppDbContext>("Data Source=app.db");
210+
// After:
211+
builder.Services.AddConcurrentSqliteDbContextFactory<AppDbContext>("Data Source=app.db");
212+
// Then inject IDbContextFactory<AppDbContext> and call CreateDbContext() per task.
213+
214+
// 3. Remove Cache=Shared from any connection string that has it.
215+
216+
// 4. That's it — no other changes required.
217+
```
218+
219+
---
220+
221+
## System Requirements
222+
223+
- .NET 10.0+
224+
- Entity Framework Core 10.0+
225+
- Microsoft.Data.Sqlite 10.0+
226+
- SQLite 3.35.0+ (WAL2 and `SQLITE_BUSY_SNAPSHOT` require 3.37.0+)

0 commit comments

Comments
 (0)