fix: PRIMARY KEY detection, ORDER BY position, change tracking (84/84 sync tests)

Author: MPCoreDeveloper

docs/PROJECT_STATUS.md

@@ -103,7 +103,7 @@ SharpCoreDB is a **production-ready, high-performance embedded database** for .N
 |---------|---------|---------|--------|
 | **SharpCoreDB** | 1.4.0 | Core database engine | ✅ Production |
 | **SharpCoreDB.Distributed** | 1.4.0 | Distributed features | ✅ Production |
-| **SharpCoreDB.Provider.Sync** | 1.0.0 | Dotmim.Sync integration | ✅ Production |
+| **SharpCoreDB.Provider.Sync** | 1.0.0 | Dotmim.Sync integration | ✅ Production (84/84 tests) |
 | **SharpCoreDB.Analytics** | 1.3.5 | Analytics & aggregates | ✅ Production |
 | **SharpCoreDB.VectorSearch** | 1.3.5 | Vector similarity search | ✅ Production |
 | **SharpCoreDB.Graph** | 1.3.5 | Graph algorithms | ✅ Production |
@@ -112,7 +112,7 @@ SharpCoreDB is a **production-ready, high-performance embedded database** for .N
 
 ## 🧪 Testing & Quality
 
-- **950+ Unit Tests** - Comprehensive coverage across all phases
+- **1000+ Unit Tests** - Comprehensive coverage across all phases (including 84 sync provider tests)
 - **100% Build Success** - Zero compilation errors
 - **Production Validated** - Real-world usage with 10GB+ datasets
 - **Performance Benchmarked** - Detailed metrics vs competitors
@@ -126,8 +126,9 @@ SharpCoreDB is a **production-ready, high-performance embedded database** for .N
 | **Phase 8 (Vector Search)** | 120+ | HNSW, distance metrics |
 | **Phase 6 (Graph)** | 17+ | A* pathfinding algorithms |
 | **Core Engine** | 430+ | ACID, transactions, storage |
+| **Sync Provider** | 84 | Change tracking, DI, adapters |
 | **Extensions** | 118+ | EF Core, providers, utilities |
-| **Total** | **950+** | Complete system coverage |
+| **Total** | **1000+** | Complete system coverage |
 
 ## 🎯 Roadmap
 

docs/sync/PHASE2_COMPLETION.md

@@ -0,0 +1,359 @@
+# ✅ Phase 2: Change Tracking & Metadata — COMPLETE
+
+**Completion Date:** January 28, 2026  
+**Status:** All components implemented and tested  
+**Test Coverage:** 26 new tests passing
+
+---
+
+## 📦 What Was Delivered
+
+### 1. **Type Mapping System** (`SharpCoreDBDbMetadata.cs`)
+
+**Status:** ✅ COMPLETE
+
+Comprehensive type mapping between:
+- **SharpCoreDB DataType** ↔ **System.Data.DbType** ↔ **.NET CLR Types**
+- SQL DDL type string generation
+- Bidirectional conversions with validation
+
+**Key Methods:**
+```csharp
+DbType MapDataType(DataType dataType)
+DataType MapDbType(DbType dbType)
+Type GetClrType(DbType dbType)
+Type GetClrType(DataType dataType)
+string ToSqlTypeString(DataType dataType)
+bool IsSyncSupported(DataType dataType)
+```
+
+**Supported Types:**
+- ✅ Integer, Long, String, Real, Blob
+- ✅ Boolean, DateTime, Decimal
+- ✅ Guid, Ulid, RowRef
+- ⚠️ Vector (not supported - requires custom serialization)
+
+**Test Coverage:** 16 tests passing
+- Type conversion accuracy
+- Round-trip conversions
+- SQL type string generation
+- Unsupported type handling
+
+---
+
+### 2. **Scope Management** (`SharpCoreDBScopeInfoBuilder.cs`)
+
+**Status:** ✅ COMPLETE
+
+Full CRUD operations for sync metadata:
+- `scope_info` table (server-side scope metadata)
+- `scope_info_client` table (client-side sync state)
+
+**Key Operations:**
+```csharp
+// Table existence
+GetExistsScopeInfoTableCommand()
+GetExistsScopeInfoClientTableCommand()
+
+// Table creation
+GetCreateScopeInfoTableCommand()
+GetCreateScopeInfoClientTableCommand()
+
+// CRUD operations
+GetScopeInfoCommand(scopeName)
+GetInsertScopeInfoCommand()
+GetUpdateScopeInfoCommand()
+GetDeleteScopeInfoCommand()
+
+// Timestamp
+GetLocalTimestampCommand() // Returns SYNC_TIMESTAMP()
+```
+
+**Schema:**
+```sql
+CREATE TABLE [scope_info] (
+    [sync_scope_id] TEXT PRIMARY KEY NOT NULL,
+    [sync_scope_name] TEXT NOT NULL,
+    [sync_scope_schema] TEXT NULL,
+    [sync_scope_setup] TEXT NULL,
+    [sync_scope_version] TEXT NULL,
+    [scope_last_sync] INTEGER NULL,
+    [scope_last_sync_timestamp] INTEGER NULL,
+    [scope_last_server_sync_timestamp] INTEGER NULL,
+    [scope_last_sync_duration] INTEGER NULL
+)
+
+CREATE TABLE [scope_info_client] (
+    [sync_scope_id] TEXT PRIMARY KEY NOT NULL,
+    [sync_scope_name] TEXT NOT NULL,
+    [sync_scope_hash] TEXT NOT NULL,
+    [sync_scope_parameters] TEXT NULL,
+    [scope_last_sync] INTEGER NULL,
+    [scope_last_server_sync_timestamp] INTEGER NULL,
+    [scope_last_sync_timestamp] INTEGER NULL,
+    [scope_last_sync_duration] INTEGER NULL,
+    [sync_scope_errors] TEXT NULL,
+    [sync_scope_properties] TEXT NULL
+)
+```
+
+**Test Coverage:** 10 tests passing
+- Table name parsing
+- DDL generation
+- Command parameter verification
+- CRUD command structure
+
+---
+
+### 3. **Table Provisioning** (`SharpCoreDBTableBuilder.cs`)
+
+**Status:** ✅ COMPLETE
+
+Per-table sync infrastructure:
+- Tracking table creation
+- Trigger generation (INSERT/UPDATE/DELETE)
+- Schema introspection
+- DDL operations
+
+**Key Features:**
+```csharp
+// Tracking table management
+GetCreateTrackingTableCommand()
+GetExistsTrackingTableCommand()
+GetDropTrackingTableCommand()
+
+// Trigger management
+GetCreateTriggerCommandAsync(DbTriggerType.Insert)
+GetCreateTriggerCommandAsync(DbTriggerType.Update)
+GetCreateTriggerCommandAsync(DbTriggerType.Delete)
+GetDropTriggerCommandAsync(triggerType)
+
+// Schema operations
+GetParsedTableNames()
+GetParsedTrackingTableNames()
+GetPrimaryKeysAsync()
+GetColumnsAsync()
+```
+
+**Tracking Table Schema:**
+```sql
+CREATE TABLE IF NOT EXISTS {tableName}_tracking (
+    {pkColumn} {pkType} PRIMARY KEY NOT NULL,
+    update_scope_id TEXT,
+    timestamp BIGINT NOT NULL,
+    sync_row_is_tombstone INTEGER NOT NULL DEFAULT 0,
+    last_change_datetime TEXT NOT NULL
+)
+```
+
+**Trigger Examples:**
+```sql
+-- INSERT trigger
+CREATE TRIGGER [trg_Users_insert_tracking] AFTER INSERT ON [Users] BEGIN
+    INSERT OR REPLACE INTO [Users_tracking] 
+    ([Id], [update_scope_id], [timestamp], [sync_row_is_tombstone], [last_change_datetime])
+    VALUES (NEW.[Id], NULL, SYNC_TIMESTAMP(), 0, CURRENT_TIMESTAMP);
+END
+
+-- DELETE trigger (tombstone marking)
+CREATE TRIGGER [trg_Users_delete_tracking] AFTER DELETE ON [Users] BEGIN
+    UPDATE [Users_tracking] 
+    SET [update_scope_id] = NULL, 
+        [timestamp] = SYNC_TIMESTAMP(), 
+        [sync_row_is_tombstone] = 1, 
+        [last_change_datetime] = CURRENT_TIMESTAMP
+    WHERE [Id] = OLD.[Id];
+END
+```
+
+---
+
+### 4. **Database Operations** (`SharpCoreDBDatabaseBuilder.cs`)
+
+**Status:** ✅ COMPLETE
+
+Database-level provisioning:
+```csharp
+EnsureDatabaseAsync() // No-op (auto-created)
+EnsureTableAsync(tableName, schemaName)
+GetAllTablesAsync() // Enumerates all user tables
+GetHelloAsync() // Returns ("SharpCoreDB", "1.0")
+GetTableAsync(tableName, schemaName) // Full schema metadata
+```
+
+**Features:**
+- Automatic database creation (SharpCoreDB native behavior)
+- Table existence validation
+- Schema metadata extraction via PRAGMA
+- Exclusion of system tables and tracking tables
+
+---
+
+### 5. **Tombstone Management** (`TombstoneManager.cs`)
+
+**Status:** ✅ COMPLETE
+
+Cleanup of deleted row markers:
+```csharp
+CleanTombstonesAsync(database, tableName, retentionDays) → int
+GetTombstoneCountAsync(database, tableName) → int
+```
+
+**Behavior:**
+- Deletes tombstone records older than retention period
+- Prevents unbounded growth of tracking tables
+- Returns count of remaining tombstones
+- Validates input parameters
+
+---
+
+### 6. **Provider Integration** (`SharpCoreDBSyncProvider.cs`)
+
+**Status:** ✅ UPDATED
+
+Wired Phase 2 components:
+```csharp
+public override DbDatabaseBuilder GetDatabaseBuilder() 
+    => new SharpCoreDBDatabaseBuilder();
+
+public override DbScopeBuilder GetScopeBuilder(string scopeName) 
+    => new SharpCoreDBScopeInfoBuilder();
+```
+
+**What Changed:**
+- Replaced `NotImplementedException` with actual builder instances
+- Added using statements for Phase 2 namespaces
+- Maintained compatibility with CoreProvider API
+
+---
+
+## 🧪 Test Coverage
+
+### New Tests Created
+
+| Test Suite | Tests | Status |
+|---|---|---|
+| **TypeMappingTests.cs** | 16 | ✅ All passing |
+| **ScopeInfoBuilderTests.cs** | 10 | ✅ All passing |
+| **Total Phase 2 Tests** | **26** | **✅ 100% passing** |
+
+### Test Categories
+
+**TypeMappingTests (16 tests):**
+- ✅ DataType → DbType conversions (11 types)
+- ✅ DbType → DataType reverse mappings (9 types)
+- ✅ DbType → CLR Type mappings (9 types)
+- ✅ DataType → CLR Type mappings (11 types)
+- ✅ SQL type string generation (11 types)
+- ✅ Sync compatibility validation
+- ✅ Round-trip conversion preservation
+- ✅ Unsupported type (Vector) error handling
+
+**ScopeInfoBuilderTests (10 tests):**
+- ✅ Table name parsing (scope_info, scope_info_client)
+- ✅ DDL generation (CREATE TABLE)
+- ✅ INSERT command generation
+- ✅ SELECT command generation
+- ✅ UPDATE command generation
+- ✅ DELETE command generation
+- ✅ Local timestamp function call
+- ✅ Parameter collection validation
+- ✅ WHERE clause verification
+
+---
+
+## 🔧 Technical Highlights
+
+### C# 14 Compliance
+
+All code follows SharpCoreDB standards:
+- ✅ Primary constructors
+- ✅ Collection expressions
+- ✅ Nullable reference types enabled
+- ✅ Pattern matching with switch expressions
+- ✅ XML documentation on public APIs
+
+### SQLite Compatibility
+
+100% SQLite-compatible SQL generation:
+- ✅ `INSERT OR REPLACE` upsert syntax
+- ✅ `CREATE TABLE IF NOT EXISTS`
+- ✅ `DROP TABLE IF EXISTS`
+- ✅ `PRAGMA table_info()` introspection
+- ✅ `sqlite_master` system table queries
+- ✅ Trigger syntax with BEGIN...END blocks
+
+### Performance Considerations
+
+- ✅ Zero-allocation static methods where possible
+- ✅ String interpolation for SQL generation
+- ✅ Parameterized queries for user data
+- ✅ Batch DDL execution support
+
+---
+
+## 📚 What's Still Phase 3
+
+Phase 2 is **complete**. The following remain for **Phase 3: Sync Adapter (DML)**:
+
+### Not Yet Implemented:
+- ⏳ `SharpCoreDBSyncAdapter.cs` - Change enumeration and application
+- ⏳ SELECT changes query generation
+- ⏳ Bulk INSERT/UPDATE/DELETE operations
+- ⏳ Conflict detection and resolution
+- ⏳ Delta application logic
+
+### Phase 3 Will Add:
+```csharp
+public override DbSyncAdapter GetSyncAdapter(SyncTable table, ScopeInfo scopeInfo)
+    => new SharpCoreDBSyncAdapter(table, scopeInfo);
+```
+
+---
+
+## 🎯 Phase 2 Success Criteria
+
+| Criterion | Status |
+|---|---|
+| ✅ Type mapping complete | **PASS** |
+| ✅ Scope CRUD operations | **PASS** |
+| ✅ Table provisioning (tracking + triggers) | **PASS** |
+| ✅ Database-level operations | **PASS** |
+| ✅ Tombstone cleanup | **PASS** |
+| ✅ Provider wiring | **PASS** |
+| ✅ Unit tests passing | **PASS (26/26)** |
+| ✅ Build successful | **PASS** |
+| ✅ C# 14 compliant | **PASS** |
+| ✅ SQLite compatible | **PASS** |
+
+**Overall Phase 2 Status:** ✅ **100% COMPLETE**
+
+---
+
+## 🚀 Next Steps
+
+### Immediate: Phase 3 Kickoff
+Create `SharpCoreDBSyncAdapter.cs` with:
+1. Change enumeration (SELECT changes since last sync)
+2. Change application (INSERT/UPDATE/DELETE)
+3. Conflict detection (timestamp comparison)
+4. Conflict resolution (server wins / client wins / custom)
+5. Bulk operations for performance
+
+### Documentation
+- Update `docs/sync/README.md` with Phase 2 completion
+- Add type mapping reference table
+- Document scope table schemas
+
+### Performance Testing
+- Benchmark tracking trigger overhead
+- Test tombstone cleanup on large datasets
+- Measure scope metadata query performance
+
+---
+
+**Phase 2 Completion: January 28, 2026**  
+**Next Phase:** Phase 3 - Sync Adapter (DML)  
+**Estimated Duration:** 1-2 weeks
+
+🎉 **Congratulations! Phase 2 is fully operational.**