Skip to content

Commit 7123c93

Browse files
committed
Large object pages are now reclaimed when conditions make it possible.
1 parent 6eee435 commit 7123c93

16 files changed

Lines changed: 570 additions & 73 deletions

CLAUDE.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -119,7 +119,7 @@ Three entry points share one per-connection undo log: implicit (statement-level
119119
- **Statement-level atomicity**: a single mutation throwing mid-execution rolls back its partial writes. Multi-row INSERT failing on row 3 leaves zero rows.
120120
- **Explicit txs**: `BEGIN TRAN` increments `TranCount`; only outermost `COMMIT` actually commits; `ROLLBACK` zeroes `TranCount` and walks the entire log. `SAVE TRAN <name>` + `ROLLBACK TRAN <name>` is the EF SaveChanges path inside an explicit tx. Parallel `BeginTransaction``InvalidOperationException`. `COMMIT`/`ROLLBACK` with no active tx → Msg 3902/3903.
121121
- `@@TRANCOUNT` reads connection depth as int.
122-
- **Identity counters and the database-scoped rowversion counter bypass the log** — both keep advancing through rollback. Orphaned LOB chains for rolled-back inserts also leak.
122+
- **Identity counters and the database-scoped rowversion counter bypass the log** — both keep advancing through rollback. (A rolled-back INSERT's off-row LOB chain is reclaimed — see the LOB-reclamation quirk — but the dead heap-page row-payload bytes are not.)
123123
- **Temp-table CREATE/DROP participates in the log** via `TempTableCreation` / `TempTableRemoval` `UndoEntry` subtypes (rollback removes from / restores into the connection's `TempTables` dict). Regular CREATE/DROP TABLE is NOT logged — see the corresponding quirk.
124124
- Locking + MVCC: full 8-mode matrix, row-X writers + row-mode readers per hints/iso, RR/SER/UPDLOCK/XLOCK/TABLOCK/HOLDLOCK/REPEATABLEREAD/NOLOCK/READPAST hints, escalation at 5000 row-locks, Msg 1205 deadlock detection, Msg 1222 timeouts, SNAPSHOT + RCSI with version chains + GC + DMVs. See [`docs/claude/locking.md`](docs/claude/locking.md).
125125

@@ -214,7 +214,7 @@ Per-feature deep-dives live under `docs/claude/`. Each entry below is a trigger:
214214
- `float` text formatting: .NET `G15`/`G7` rather than SQL Server's `1e+015`-style scientific.
215215
- Auto-generated constraint names: PK / UNIQUE shape `PK__<table8>__<16hex>` / `UQ__<table8>__<16hex>` (16-hex 64-bit FNV-1a); CK / FK / DF shape `CK__<table8>__[<col8>__]<8hex>` (8-hex 32-bit FNV-1a). Both are deterministic across runs, distinct from SQL Server's object-id-derived hex (so won't byte-match).
216216
- **DELETE / forwarding UPDATE leak page space**: deleted row payload bytes stay in their original page until process exit; only the slot is tombstoned. Slot directory entries also never reused. An UPDATE whose new payload exceeds the slot's existing extent forwards (single-level pointer matching SQL Server's heap-update behavior) and similarly leaves the original slot's payload bytes dead. An UPDATE that fits in place reuses the slot's bytes and adds no leak.
217-
- **UPDATE leaks LOB chains**: a UPDATE-replaced row's LOB chains stay in `Heap.LobPages`; the row encoder allocates fresh chains for the new payload regardless of whether the UPDATE rewrites in place or forwards. DELETE leaks similarly. Other rows reference LOB pages by stable index, so list compaction would corrupt them.
217+
- **LOB chains are reclaimed; dead heap-page bytes still leak**: a superseded off-row LOB chain (UPDATE-replaced or DELETEd row, or a rolled-back INSERT's chain) is returned to a per-`Heap` free-list (`Heap.FreeLobChain` → `ConcurrentStack<int>`) and reused by the next `AllocateLobChain`, so `LobPages` is bounded by the high-water set of concurrently-live (+ version-pinned) chains rather than total mutation count. Pages keep their stable index (reuse never renumbers), so other rows' `NextPageIndex` links stay valid — no list compaction. Reclamation ownership: an unversioned superseded chain is freed when its undo entry commits (`UndoLog.Commit` / autocommit `RunMutation`); under SNAPSHOT/RCSI the chain is pinned by its `HistoricalVersion` and freed instead when `VersionStore` GC trims it (the two are mutually exclusive via `VersionStore.WillCaptureVersions`, so never double-freed); a rolled-back new chain is freed by the undo entry's `Undo`. Versioning-on **autocommit** has no GC cadence, so those chains still leak (pre-existing, unchanged). Note the row-payload-byte / slot-directory leak below is a *separate* leak (Stage 2) that this does **not** address — dead heap-page bytes still grow monotonically.
218218
- **`GetBytes`/`GetChars` materialize, don't stream**: each call decodes the full column value via `RowDecoder` and slices into the caller's buffer. Behavior matches per-call observation; the streaming-memory guarantee doesn't.
219219
- **`SELECT INTO` string `+` reads as nullable**: real SQL Server projects `cs + 'x'` (both NOT NULL) as NOT NULL; the simulator can't statically distinguish string-concat from integer-add at projection-schema time (the dispatch happens runtime on operand types), so all `Add` results read as nullable. Conservative; no test reliance on string-`+`-non-null.
220220
- **`SELECT INTO` from a CTE drops identity + nullability**: CTE bindings synthesize their wrapper `HeapColumn` entries with `nullable: true` and no identity, so the analyzer treats CTE sources as derived plans. Real SQL Server preserves both through simple single-source CTEs. Fix requires propagating column metadata through CTE bindings.
Lines changed: 157 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,157 @@
1+
using static Microsoft.VisualStudio.TestTools.UnitTesting.Assert;
2+
3+
namespace SqlServerSimulator.Storage;
4+
5+
/// <summary>
6+
/// Guards LOB-chain reclamation (Stage 1 of the heap space-reclamation work):
7+
/// an UPDATE / DELETE that supersedes an off-row <c>nvarchar(MAX)</c> value
8+
/// must return the old chain's pages to the heap's free-list rather than
9+
/// orphaning them in <see cref="Heap.LobPages"/>. Without reclamation
10+
/// a logical dataset of constant size grows <see cref="Heap.LobPages"/>
11+
/// without bound under mutation churn — the leak that made long-running
12+
/// simulations a memory problem. These tests assert the page count stays a
13+
/// small multiple of the live working set, not a function of the churn count.
14+
/// </summary>
15+
[TestClass]
16+
public sealed class LobReclamationTests
17+
{
18+
private static Heap HeapFor(SimulatedDbConnection conn, string table) =>
19+
conn.CurrentDatabase.Schemas[Database.DefaultSchemaName].HeapTables[table].Heap;
20+
21+
private static void Exec(SimulatedDbConnection conn, string sql)
22+
{
23+
using var cmd = conn.CreateCommand();
24+
cmd.CommandText = sql;
25+
_ = cmd.ExecuteNonQuery();
26+
}
27+
28+
// One logical row, repeatedly rewritten with a fresh off-row value. The
29+
// working set is a single chain (one page for the ~200-byte payload); the
30+
// free-list must keep total page count bounded regardless of update count.
31+
[TestMethod]
32+
public void RepeatedUpdate_OfMaxValue_KeepsLobPagesBounded()
33+
{
34+
var conn = new Simulation().CreateDbConnection();
35+
conn.Open();
36+
Exec(conn, "create table t (id int not null primary key, v nvarchar(max) not null)");
37+
Exec(conn, "insert t values (1, replicate(N'a', 100))");
38+
39+
for (var i = 0; i < 100; i++)
40+
Exec(conn, $"update t set v = replicate(N'b', 100) where id = 1");
41+
42+
// Live set is one chain (one page). Allow generous slack for the
43+
// in-flight old+new overlap during a single statement.
44+
IsLessThanOrEqualTo(4, HeapFor(conn, "t").LobPages.Count, "LobPages should stay near the one-chain working set; the free-list bounds repeated-UPDATE churn.");
45+
}
46+
47+
// Delete then re-insert the same logical row many times. Each DELETE
48+
// supersedes a chain; the free-list must reclaim it for the next INSERT.
49+
[TestMethod]
50+
public void DeleteInsertChurn_KeepsLobPagesBounded()
51+
{
52+
var conn = new Simulation().CreateDbConnection();
53+
conn.Open();
54+
Exec(conn, "create table t (id int not null primary key, v nvarchar(max) not null)");
55+
56+
for (var i = 0; i < 100; i++)
57+
{
58+
Exec(conn, "insert t values (1, replicate(N'a', 100))");
59+
Exec(conn, "delete from t where id = 1");
60+
}
61+
62+
IsLessThanOrEqualTo(4, HeapFor(conn, "t").LobPages.Count, "DELETE should free the chain for the next INSERT to reuse.");
63+
}
64+
65+
// A rolled-back INSERT's chain must not leak (closes the legacy
66+
// "orphaned LOB chains for rolled-back inserts also leak" quirk).
67+
[TestMethod]
68+
public void RolledBackInsert_FreesItsChain()
69+
{
70+
var conn = new Simulation().CreateDbConnection();
71+
conn.Open();
72+
Exec(conn, "create table t (id int not null primary key, v nvarchar(max) not null)");
73+
74+
for (var i = 0; i < 100; i++)
75+
{
76+
Exec(conn, "begin tran; insert t values (1, replicate(N'a', 100)); rollback");
77+
}
78+
79+
IsLessThanOrEqualTo(4, HeapFor(conn, "t").LobPages.Count, "Rolled-back INSERT chains should be reclaimed on rollback.");
80+
}
81+
82+
// A rolled-back UPDATE's freshly-allocated new chain must be reclaimed on
83+
// rollback, while the restored old value's chain stays live.
84+
[TestMethod]
85+
public void RolledBackUpdate_FreesNewChain()
86+
{
87+
var conn = new Simulation().CreateDbConnection();
88+
conn.Open();
89+
Exec(conn, "create table t (id int not null primary key, v nvarchar(max) not null)");
90+
Exec(conn, "insert t values (1, replicate(N'a', 100))");
91+
92+
for (var i = 0; i < 100; i++)
93+
Exec(conn, "begin tran; update t set v = replicate(N'b', 100) where id = 1; rollback");
94+
95+
IsLessThanOrEqualTo(4, HeapFor(conn, "t").LobPages.Count, "A rolled-back UPDATE should reclaim its new chain and keep only the restored original.");
96+
}
97+
98+
// Inside one explicit transaction the superseded chains can't be reclaimed
99+
// mid-flight (a rollback might still need them), so LobPages grows to the
100+
// tx's peak. The win is reuse: after the tx commits those chains are free,
101+
// so a second equally-long transaction draws from the free-list instead of
102+
// growing the page list further.
103+
[TestMethod]
104+
public void ReclaimedChains_AreReusedAcrossTransactions()
105+
{
106+
var conn = new Simulation().CreateDbConnection();
107+
conn.Open();
108+
Exec(conn, "create table t (id int not null primary key, v nvarchar(max) not null)");
109+
Exec(conn, "insert t values (1, replicate(N'a', 100))");
110+
111+
const string churnTx = "begin tran; declare @i int = 0; while @i < 100 begin update t set v = replicate(N'b', 100) where id = 1; set @i += 1; end; commit";
112+
Exec(conn, churnTx);
113+
var afterFirst = HeapFor(conn, "t").LobPages.Count;
114+
Exec(conn, churnTx);
115+
var afterSecond = HeapFor(conn, "t").LobPages.Count;
116+
117+
IsLessThanOrEqualTo(afterFirst, afterSecond, $"Second churn tx (peak {afterSecond}) should reuse the first's freed pages (peak {afterFirst}), not grow the list.");
118+
}
119+
120+
// Under READ_COMMITTED_SNAPSHOT a superseded chain is pinned by its history
121+
// entry until version-store GC (which runs at each tx commit) trims it; with
122+
// no concurrent snapshot reader that happens immediately, so churn stays
123+
// bounded across committed transactions.
124+
[TestMethod]
125+
public void VersionedUpdate_ReclaimsViaGarbageCollection()
126+
{
127+
var conn = new Simulation().CreateDbConnection();
128+
conn.Open();
129+
Exec(conn, "alter database simulated set read_committed_snapshot on");
130+
Exec(conn, "create table t (id int not null primary key, v nvarchar(max) not null)");
131+
Exec(conn, "insert t values (1, replicate(N'a', 100))");
132+
133+
for (var i = 0; i < 100; i++)
134+
Exec(conn, "begin tran; update t set v = replicate(N'b', 100) where id = 1; commit");
135+
136+
IsLessThanOrEqualTo(8, HeapFor(conn, "t").LobPages.Count, "Version-store GC should trim each committed tx's history entry and free its chain when no snapshot needs it.");
137+
}
138+
139+
// Reclamation must not corrupt a surviving value: the live row reads back
140+
// correctly after extensive churn around it.
141+
[TestMethod]
142+
public void ReclamationPreservesLiveValue()
143+
{
144+
var conn = new Simulation().CreateDbConnection();
145+
conn.Open();
146+
Exec(conn, "create table t (id int not null primary key, v nvarchar(max) not null)");
147+
Exec(conn, "insert t values (1, N'keep-me')");
148+
Exec(conn, "insert t values (2, replicate(N'x', 200))");
149+
150+
for (var i = 0; i < 50; i++)
151+
Exec(conn, $"update t set v = replicate(N'y', 200) where id = 2");
152+
153+
using var cmd = conn.CreateCommand();
154+
cmd.CommandText = "select v from t where id = 1";
155+
AreEqual("keep-me", (string)cmd.ExecuteScalar()!);
156+
}
157+
}

SqlServerSimulator/Parser/Expressions/DateTimeAdjustments.cs

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -100,10 +100,10 @@ internal sealed class SwitchOffset : Expression
100100
public SwitchOffset(ParserContext context)
101101
{
102102
this.dtoArg = Parse(context);
103-
if (context.Token is not Tokens.Operator { Character: ',' })
103+
if (context.Token is not Operator { Character: ',' })
104104
throw SimulatedSqlException.SyntaxErrorNear(context);
105105
this.offsetArg = Parse(context.MoveNextRequiredReturnSelf());
106-
if (context.Token is not Tokens.Operator { Character: ')' })
106+
if (context.Token is not Operator { Character: ')' })
107107
throw SimulatedSqlException.SyntaxErrorNear(context);
108108
}
109109

@@ -169,10 +169,10 @@ internal sealed class ToDateTimeOffset : Expression
169169
public ToDateTimeOffset(ParserContext context)
170170
{
171171
this.dtArg = Parse(context);
172-
if (context.Token is not Tokens.Operator { Character: ',' })
172+
if (context.Token is not Operator { Character: ',' })
173173
throw SimulatedSqlException.SyntaxErrorNear(context);
174174
this.offsetArg = Parse(context.MoveNextRequiredReturnSelf());
175-
if (context.Token is not Tokens.Operator { Character: ')' })
175+
if (context.Token is not Operator { Character: ')' })
176176
throw SimulatedSqlException.SyntaxErrorNear(context);
177177
}
178178

SqlServerSimulator/Parser/Expressions/IndexLookup.cs

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,7 @@ namespace SqlServerSimulator.Parser.Expressions;
66
/// <summary>
77
/// Shared lookup machinery for the <c>INDEX_COL</c> /
88
/// <c>INDEXKEY_PROPERTY</c> / <c>STATS_DATE</c> functions: resolves an
9-
/// <c>index_id</c> back to the corresponding <see cref="Storage.Index"/> or
9+
/// <c>index_id</c> back to the corresponding <see cref="Index"/> or
1010
/// <see cref="KeyConstraint"/> on a <see cref="HeapTable"/>, mirroring the
1111
/// emission order used by <c>sys.indexes</c> / <c>sys.index_columns</c>.
1212
/// </summary>
@@ -18,15 +18,15 @@ namespace SqlServerSimulator.Parser.Expressions;
1818
/// has no key columns (this function returns <c>null</c> for that case).</description></item>
1919
/// <item><description><c>index_id ≥ 2</c> (or ≥ 1 on a heap): remaining
2020
/// <see cref="KeyConstraint"/>s (UNIQUE) plus user
21-
/// <see cref="Storage.Index"/>es, sorted by <c>ObjectId</c>.</description></item>
21+
/// <see cref="Index"/>es, sorted by <c>ObjectId</c>.</description></item>
2222
/// </list>
2323
/// </remarks>
2424
internal static class IndexLookup
2525
{
2626
/// <summary>
2727
/// Resolves <paramref name="indexId"/> against <paramref name="table"/>.
2828
/// Returns either a <see cref="KeyConstraint"/> (PK / UQ) or a
29-
/// <see cref="Storage.Index"/>, never both. Returns <c>null</c> when the
29+
/// <see cref="Index"/>, never both. Returns <c>null</c> when the
3030
/// id is out of range or refers to the heap row.
3131
/// </summary>
3232
public static (KeyConstraint? Constraint, Index? Index)? ResolveByIndexId(HeapTable table, int indexId)

SqlServerSimulator/SimulatedDbTransaction.cs

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -166,7 +166,12 @@ public override void Commit()
166166
return;
167167
var db = this.Connection.CurrentDatabase;
168168
Storage.VersionStore.FinalizePendingEntries(this.PendingVersionEntries, db);
169-
this.UndoLog.Clear();
169+
// Commit() (vs the former discard-only Clear) reclaims the off-row LOB
170+
// chains superseded by this tx's committed UPDATE/DELETEs in the
171+
// unversioned case; under SNAPSHOT/RCSI those chains are pinned by the
172+
// history entries FinalizePendingEntries just stamped and are reclaimed
173+
// instead by RunGarbageCollection below once no snapshot needs them.
174+
this.UndoLog.Commit();
170175
ReleaseAllLocks();
171176
UnregisterActiveSnapshot(db);
172177
Storage.VersionStore.RunGarbageCollection(db);

SqlServerSimulator/Simulation/Simulation.Delete.cs

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -325,7 +325,7 @@ private static SimulatedStatementOutcome CommitDelete(
325325
if (lockableTable)
326326
context.Batch.AcquireRowLockTxScoped(table, pageIndex, slotIndex, LockMode.Exclusive);
327327
var oldBytes = captureVersions ? table.Heap.ReadSlotBytes(pageIndex, slotIndex) : null;
328-
table.Heap.DeleteAt(pageIndex, slotIndex, undoLog);
328+
table.Heap.DeleteAt(pageIndex, slotIndex, undoLog, ReclaimSuperseded(table, context));
329329
if (oldBytes is not null)
330330
Storage.VersionStore.CaptureWrite(context.Batch, table, (pageIndex, slotIndex), (pageIndex, slotIndex), oldBytes, Storage.VersionWriteKind.Delete);
331331
// Row-lock dict cleanup: the slot is tombstoned and slot ids

SqlServerSimulator/Simulation/Simulation.ForeignKeys.cs

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -246,7 +246,7 @@ private static void CascadeDeleteChildRows(
246246
var childTable = fk.ChildTable;
247247
var undoLog = childTable.IsTableVariable ? context.Batch.CurrentTableVarUndoLog : context.Batch.CurrentUndoLog;
248248
foreach (var (pageIndex, slotIndex, _) in matchingChildRows)
249-
childTable.Heap.DeleteAt(pageIndex, slotIndex, undoLog);
249+
childTable.Heap.DeleteAt(pageIndex, slotIndex, undoLog, ReclaimSuperseded(childTable, context));
250250
// Recurse: the just-deleted child rows may themselves be parents of
251251
// further FKs pointing at this child table.
252252
var oldRows = new List<SqlValue[]>(matchingChildRows.Count);
@@ -377,7 +377,7 @@ private static void RewriteChildFkColumns(
377377
}
378378
if (IsLockableTable(childTable))
379379
context.Batch.AcquireRowLockTxScoped(childTable, pageIndex, slotIndex, LockMode.Exclusive);
380-
childTable.Heap.UpdateAt(pageIndex, slotIndex, RowEncoder.EncodeRow(childTable.StoredColumns, ProjectStoredValues(childTable, newRow), childTable.Heap), undoLog);
380+
childTable.Heap.UpdateAt(pageIndex, slotIndex, RowEncoder.EncodeRow(childTable.StoredColumns, ProjectStoredValues(childTable, newRow), childTable.Heap), undoLog, ReclaimSuperseded(childTable, context));
381381
newPairs.Add((oldClone, newRow));
382382
}
383383
// Recurse: the child rows just got their FK columns rewritten — if
@@ -409,7 +409,7 @@ private static void CascadeSetChildKeysToValue(
409409
}
410410
if (IsLockableTable(childTable))
411411
context.Batch.AcquireRowLockTxScoped(childTable, pageIndex, slotIndex, LockMode.Exclusive);
412-
childTable.Heap.UpdateAt(pageIndex, slotIndex, RowEncoder.EncodeRow(childTable.StoredColumns, ProjectStoredValues(childTable, newRow), childTable.Heap), undoLog);
412+
childTable.Heap.UpdateAt(pageIndex, slotIndex, RowEncoder.EncodeRow(childTable.StoredColumns, ProjectStoredValues(childTable, newRow), childTable.Heap), undoLog, ReclaimSuperseded(childTable, context));
413413
newPairs.Add((oldClone, newRow));
414414
}
415415
// For SET NULL / SET DEFAULT under a DELETE on parent, the recursion

0 commit comments

Comments
 (0)