feat: #515 Enable stored procedures by default for audit logging (#588)

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: ascott18 <5017521+ascott18@users.noreply.github.com>
Co-authored-by: Andrew Scott <andrew.scott@intellitect.com>

Author: Copilot

.github/copilot-instructions.md

@@ -12,10 +12,10 @@ The required tools and dependencies are automatically installed via the GitHub A
 
 ## Instructions
 
-- Format PR titles with Semantic Commits. The work item number should follow the colon after the commit type like `feat: #12345 added ...`
-- Always update the documentation when making changes or adding features that will affect developers who use Coalesce.
-- Always add an entry to CHANGELOG.md when adding new features or fixing non-trivial bugs.
-- Avoid making breaking changes if not necessary. A less obvious example of a breaking change would be changing an existing CSS class name.
+- YOU MUST Format PR titles with Semantic Commits. The work item number should follow the colon after the commit type like `feat: #12345 added ...`
+- YOU MUST update the documentation when making changes or adding features that will affect developers who use Coalesce.
+- YOU MUST add an entry to CHANGELOG.md when adding new features or fixing non-trivial bugs. Be concise and factual. The changelog is not a marketing document - you don't have to convince users of the value of the feature, or explain how to use it or configure it.
+- YOU MUST Avoid making breaking changes if not necessary. A less obvious example of a breaking change would be changing an existing CSS class name.
 - Consider adding or updating example files in `playground\Coalesce.Web.Vue3\src\examples` when making changes to coalesce-vue-vuetify.
 
 ## Validation Checklist

CHANGELOG.md

@@ -6,6 +6,7 @@
 - The types generated for inheritance hierarchies have changed significantly. If two or more models in a type hierarchy (i.e. a base type and a derived type) are both exposed by Coalesce, that relationship is now reflected throughout the generated DTOs, generated TypeScript, and admin pages. The generated ViewModels classes for abstract classes are now just proxies intended to be used only for loading one of the concrete implementation types.
 - `StandardBehaviors.AfterDelete` is now `AfterDeleteAsync` and has a different signature and semantics. Instead of modifying the resulting `item` and `includeTree` with `ref` parameters, these values can be optionally overridden by returning an ItemResult with its `Object` and `IncludeTree` properties populated with non-null values.
 - `ViewModel.$getErrors` now returns a `string[]` instead of a `Generator<string>`.
+- `IntelliTect.Coalesce.AuditLogging` now uses stored procedures by default to upsert audit log entries. You can disable this (e.g. if your application lacks permission to create/update stored procedures) by chaining `.WithStoredProcedures(false)` when you configure audit logging.
 - The CommonJS build of coalesce-vue has been dropped - only the ESM build remains. Most projects should be unaffected.
 
 ## Features

docs/topics/audit-logging.md

@@ -139,6 +139,26 @@ public class AppDbContext : DbContext, IAuditLogDbContext<AuditLog>
 }
 ```
 
+### Stored Procedures
+
+For SQL Server databases, audit logging uses stored procedures by default instead of executing the merge SQL directly. This provides better performance through compiled execution plans and easier monitoring/troubleshooting:
+
+``` c#
+public class AppDbContext : DbContext, IAuditLogDbContext<AuditLog>
+{
+    protected override void OnConfiguring(DbContextOptionsBuilder optionsBuilder)
+    {
+        optionsBuilder.UseCoalesceAuditLogging<AuditLog>(x => x
+            .WithAugmentation<OperationContext>()
+            // Stored procedures are enabled by default for SQL Server
+            // .WithStoredProcedures(false) // Disable if you prefer raw SQL
+        );
+    }
+}
+```
+
+When enabled, Coalesce will automatically create stored procedures with names like `AuditMerge_A1B2C3D4`, where the suffix is a short hash of the SQL content. This prevents conflicts between different application versions that might have different audit log schemas. The stored procedure is created automatically on first use and reused for subsequent operations.
+
 ### Property Descriptions
 
 The `AuditLogProperty` children of your `IAuditLog` implementation have two properties `OldValueDescription` and `NewValueDescription` that can be used to hold a description of the old and new values. By default, Coalesce will populate the descriptions of foreign key properties with the [List Text](/modeling/model-components/attributes/list-text.md) of the referenced principal entity. This greatly improves the usability of the audit logs, which would otherwise only show meaningless numbers or GUIDs for foreign keys that changed.

src/IntelliTect.Coalesce.AuditLogging.Tests/SqlServerAuditTests.cs

@@ -1,5 +1,6 @@
 using Microsoft.Data.SqlClient;
 using Microsoft.EntityFrameworkCore;
+using System.Data;
 
 namespace IntelliTect.Coalesce.AuditLogging.Tests;
 
@@ -123,11 +124,148 @@ public async Task WithSqlServer_CreatesNewRecordForUnmergableChanges()
         Assert.Equal(2, db.AuditLogs.Count(l => l.State == AuditEntryState.EntityModified));
     }
 
+    [SkippableFact]
+    public async Task WithSqlServer_StoredProceduresWork()
+    {
+        // Arrange - use stored procedures
+        using TestDbContext db = BuildDbContextWithStoredProcedures();
+
+        var user = new AppUser { Name = "Bob", Title = "Intern" };
+        db.Add(user);
+        await db.SaveChangesAsync();
+
+        // Act/Assert
+        user.Title = "Manager";
+        await db.SaveChangesAsync();
+
+        Assert.Equal(2, db.AuditLogs.Count()); // Two records: EntityAdded, and EntityUpdated
+
+        // Act/Assert - second change should merge like raw SQL
+        user.Title = "CEO";
+        await db.SaveChangesAsync();
+
+        Assert.Equal(2, db.AuditLogs.Count()); // Two records: EntityAdded, and EntityUpdated
+
+        // Assert - verify the merge happened correctly
+        var entry = Assert.Single(db.AuditLogs.Include(c => c.Properties).Where(c => c.State == AuditEntryState.EntityModified));
+
+        var propChange = Assert.Single(entry.Properties!);
+        Assert.Equal(nameof(AppUser.Title), propChange.PropertyName);
+        Assert.Equal("Intern", propChange.OldValue);
+        Assert.Equal("CEO", propChange.NewValue);
+    }
+
+    [SkippableFact]
+    public async Task WithSqlServer_StoredProcedureIsCreated()
+    {
+        // Arrange
+        using TestDbContext db = BuildDbContextWithStoredProcedures();
+
+        var user = new AppUser { Name = "Bob", Title = "Intern" };
+        db.Add(user);
+        await db.SaveChangesAsync();
+
+        // Act - trigger merge operation to create stored procedure
+        user.Title = "Manager";
+        await db.SaveChangesAsync();
+
+        // Assert - verify stored procedure was created
+        using (var command = db.Database.GetDbConnection().CreateCommand())
+        {
+            command.CommandText = "SELECT COUNT(*) FROM sys.procedures WHERE name LIKE 'AuditMerge_%'";
+            if (db.Database.GetDbConnection().State != ConnectionState.Open)
+                await db.Database.GetDbConnection().OpenAsync();
+            var result = await command.ExecuteScalarAsync();
+            var procedureCount = Convert.ToInt32(result);
+            Assert.True(procedureCount > 0, "Expected at least one AuditMerge stored procedure to be created");
+        }
+    }
+
+    [SkippableFact]
+    public async Task WithSqlServer_DifferentModelsCreateDifferentStoredProcedures()
+    {
+        // This test verifies that different entity models result in different stored procedures
+        // due to the hash-based naming scheme
+
+        // Arrange - Create two different contexts that would generate different SQL
+        using TestDbContext db1 = BuildDbContextWithStoredProcedures();
+
+        var user1 = new AppUser { Name = "Bob" };
+        db1.Add(user1);
+        await db1.SaveChangesAsync();
+
+        user1.Name = "Robert";
+        await db1.SaveChangesAsync(); // This should create a stored procedure
+
+        // Get count of stored procedures after first context
+        int initialCount;
+        using (var command = db1.Database.GetDbConnection().CreateCommand())
+        {
+            command.CommandText = "SELECT COUNT(*) FROM sys.procedures WHERE name LIKE 'AuditMerge_%'";
+            if (db1.Database.GetDbConnection().State != ConnectionState.Open)
+                await db1.Database.GetDbConnection().OpenAsync();
+            var result = await command.ExecuteScalarAsync();
+            initialCount = Convert.ToInt32(result);
+        }
+
+        // The same model should reuse the same stored procedure
+        using TestDbContext db2 = BuildDbContextWithStoredProcedures();
+
+        var user2 = new AppUser { Name = "Alice" };
+        db2.Add(user2);
+        await db2.SaveChangesAsync();
+
+        user2.Name = "Alicia";
+        await db2.SaveChangesAsync(); // This should reuse the existing stored procedure
+
+        int finalCount;
+        using (var command = db2.Database.GetDbConnection().CreateCommand())
+        {
+            command.CommandText = "SELECT COUNT(*) FROM sys.procedures WHERE name LIKE 'AuditMerge_%'";
+            if (db2.Database.GetDbConnection().State != ConnectionState.Open)
+                await db2.Database.GetDbConnection().OpenAsync();
+            var result = await command.ExecuteScalarAsync();
+            finalCount = Convert.ToInt32(result);
+        }
+
+        // Should be the same count since the same model is used
+        Assert.Equal(initialCount, finalCount);
+    }
+
     private static TestDbContext BuildDbContext()
     {
         var db = new TestDbContext(new DbContextOptionsBuilder<TestDbContext>()
                     .UseSqlServer(SqlServerConnString)
-                    .UseCoalesceAuditLogging<TestAuditLog>()
+                    .UseCoalesceAuditLogging<TestAuditLog>(x => x.WithStoredProcedures(false))
+                    .Options);
+
+        try
+        {
+            db.Database.EnsureDeleted();
+            db.Database.EnsureCreated();
+        }
+        catch (SqlException ex) when (
+            ex.Number == 53
+            || ex.Message.Contains("Could not open a connection to SQL Server")
+            || ex.Message.Contains("The server was not found or was not accessible")
+        )
+        {
+            Skip.If(true, ex.Message);
+        }
+        catch (PlatformNotSupportedException ex) when (
+            ex.Message.Contains("LocalDB is not supported on this platform")
+        )
+        {
+            Skip.If(true, ex.Message);
+        }
+        return db;
+    }
+
+    private static TestDbContext BuildDbContextWithStoredProcedures()
+    {
+        var db = new TestDbContext(new DbContextOptionsBuilder<TestDbContext>()
+                    .UseSqlServer(SqlServerConnString)
+                    .UseCoalesceAuditLogging<TestAuditLog>(x => x.WithStoredProcedures())
                     .Options);
 
         try

src/IntelliTect.Coalesce.AuditLogging/Configuration/AuditLoggingBuilder.cs

@@ -55,6 +55,13 @@ public AuditLoggingBuilder<TAuditLog> WithPropertyDescriptions(PropertyDescripti
         options.PropertyDescriptions = mode;
         return this;
     }
+
+    /// <inheritdoc cref="AuditOptions.UseStoredProcedures"/> 
+    public AuditLoggingBuilder<TAuditLog> WithStoredProcedures(bool useStoredProcedures = true)
+    {
+        options.UseStoredProcedures = useStoredProcedures;
+        return this;
+    }
     private static readonly MemoryCache _auditConfigTransforms = new(new MemoryCacheOptions { SizeLimit = 512 });
 
     /// <summary>

src/IntelliTect.Coalesce.AuditLogging/Configuration/AuditOptions.cs

@@ -54,6 +54,22 @@ public class AuditOptions
     /// </summary>
     public PropertyDescriptionMode PropertyDescriptions { get; internal set; } = PropertyDescriptionMode.FkListText;
 
+    /// <summary>
+    /// <para>
+    /// When enabled, the audit log merge SQL will be wrapped in a stored procedure 
+    /// that is automatically created/updated as needed. The stored procedure name 
+    /// includes a hash of the SQL content to handle version conflicts.
+    /// </para>
+    /// <para>
+    /// This can provide better performance through compiled execution plans and 
+    /// easier monitoring/troubleshooting. Currently only supported for SQL Server.
+    /// </para>
+    /// <para>
+    /// The default is true.
+    /// </para>
+    /// </summary>
+    public bool UseStoredProcedures { get; internal set; } = true;
+
     /// <summary>
     /// Internal so that it cannot be modified in a way that breaks the caching assumptions
     /// that we make in CoalesceAuditLoggingBuilder.