Feature/tenant lifecycle automation (#1150)

* Add Tenant Lifecycle Automation design document

Expanded `tenant-lifecycle-automation.md` with a detailed framework for automating tenant provisioning, activation, and health verification.

- Defined goals, scope, and non-goals for the automation process.
- Outlined personas (Platform Admin, Tenant Admin, SRE/DevOps).
- Documented high-level workflow for provisioning steps.
- Specified functional, operational, and security requirements.
- Added acceptance criteria for successful provisioning.
- Included failure/recovery criteria for error handling and retries.

This update ensures a robust, secure, and observable tenant lifecycle management process.

* Enhance multitenancy and auditing modules

- Introduced a tenant provisioning workflow with support for migrations, seeding, and cache warming.
- Added `TenantProvisioningService` and related entities to manage provisioning steps and statuses.
- Updated `ITenantService` with methods for tenant migration and seeding.
- Added endpoints for retrieving and retrying tenant provisioning statuses.
- Integrated Hangfire for background job execution and improved its configuration.
- Refactored logging in `LogJobFilter` to use `Logger.DebugFormat`.
- Updated `TenantDbContext` and added migrations for multitenancy and auditing.
- Adjusted service lifetimes for audit and tenant provisioning components.
- Updated `appsettings.json` for database connection and Serilog configuration.
- Removed redundant code and improved maintainability with modern C# features.

* Automate tenant provisioning and improve workflows

Implemented automated tenant provisioning with persisted status
and retry support. Added background provisioning via Hangfire
with inline fallback for dev environments. Introduced startup
hosted services for tenant catalog migration/seeding and
optional auto-provision enqueue. Enhanced `TenantDbContextFactory`
to support PostgreSQL via appsettings. Fixed audit pipeline to
include tenant/user stamps and write per-tenant log batches.

* update packages to 10

Author: iammukeshm

docs/stories/tenant-lifecycle-automation.md

@@ -0,0 +1,88 @@
+# Tenant Lifecycle Automation
+
+Goal: automate tenant provisioning, activation, and health verification so new tenants are production-ready with minimal manual steps while preserving multi-tenant safety, auditing, and observability.
+
+## Scope (In)
+- Create/activate tenant triggers background provisioning workflow.
+- Per-tenant database creation (or schema), migrations, and seed data.
+- Default identity bootstrap (admin user/roles/permissions) tied to tenant.
+- Health verification and status reporting.
+- Idempotent, retryable orchestration with audit and telemetry.
+- Admin endpoints/UX to view workflow state and retry/re-run steps.
+
+## Non-Goals (Out)
+- Full-feature feature-flag platform.
+- Billing/usage metering.
+- Cross-cloud infrastructure automation (K8s, DNS, CDN).
+
+## Personas
+- Platform Admin: initiates tenant creation, monitors status, retries failed steps.
+- Tenant Admin: receives bootstrap credentials, validates app access post-provision.
+- SRE/DevOps: monitors health, investigates failed jobs, tunes resilience.
+
+## High-Level Flow
+1) Admin issues `CreateTenant` (or activates an existing tenant).
+2) System enqueues a provisioning job (Hangfire) keyed by TenantId + correlation.
+3) Workflow steps (all idempotent):
+   - Validate tenant metadata (provider, connection string template, validity).
+   - Create tenant database/schema (or ensure exists) using provider-specific strategy.
+   - Apply EF Core migrations for each enabled module (Multitenancy, Identity, Auditing, etc.).
+   - Seed baseline data (roles, permissions, admin user with reset token, root tenant data if applicable).
+   - Warm caches if enabled (e.g., permissions).
+   - Emit audit + telemetry events for each step.
+4) Mark tenant as `Active` when all steps succeed; surface status via API.
+5) On failure: capture error, mark status `Failed`, allow retry/resume from failed step.
+
+## Functional Requirements
+- Provisioning job:
+  - Runs as Hangfire background job; supports manual trigger and automatic trigger on create/activate.
+  - Stores per-step status, timestamps, and error messages (persisted per tenant).
+  - Uses correlation/trace IDs; logs to OpenTelemetry.
+  - Supports cancellation and exponential backoff retries.
+- Database orchestration:
+  - Provider-aware strategies (PostgreSQL initial target; hooks for SQL Server).
+  - Option to create database if missing; else validate connectivity.
+  - Runs module migrations in deterministic order; stops on first failure.
+- Seeding:
+  - Seeds Identity admin user, default roles/permissions, and tenant metadata.
+  - Issues one-time admin credential or password reset token for Tenant Admin.
+  - Seeds demo data optionally (flag).
+- Status surface:
+  - API to fetch provisioning status history per tenant.
+  - Health check should include tenant provisioning status (ready/degraded/failed).
+- Safety & idempotency:
+  - All steps re-runnable without corrupting state (check-before-create).
+  - Guard against concurrent provisioning for same tenant.
+  - Respect tenant validity/activation flags.
+
+## Operational/Observability Requirements
+- Emit structured logs with TenantId, correlationId, step name, duration, outcome.
+- Create OpenTelemetry spans for each step (db create, migrate, seed, cache warm).
+- Publish audit events for lifecycle changes (Requested, Started, StepFailed, Completed).
+- Expose metrics: provision_duration_seconds, provision_step_failures_total, active_tenants.
+
+## Security Requirements
+- No secrets in logs/audits; hash/scrub credentials.
+- Bootstrap credentials delivered via secure channel (email with reset token or out-of-band).
+- Enforce tenant isolation during provisioning (context scopes, connection string guards).
+- Authorization: only platform admins can trigger or retry provisioning.
+
+## Acceptance Criteria (Happy Path)
+- Creating a tenant triggers a job that:
+  - Creates/validates DB, applies migrations for all enabled modules, seeds identity/admin, warms caches.
+  - Marks tenant Active and Ready; status endpoint shows completed steps with durations.
+- Audit trail shows Requested -> Started -> Completed with TenantId and correlationId.
+- Metrics and traces include the provisioning spans and surface in health checks.
+
+## Failure/Recovery Criteria
+- If migrations fail, status is Failed with error details; job can be retried and resumes idempotently.
+- Double-submit provisioning for same tenant does not run concurrent workflows (dedupe/lock).
+- Partial seeds are safe to re-run (no duplicate roles/users; admin user upsert).
+- Health check reports degraded for tenants with failed provisioning; improves after successful retry.
+
+## Progress Update (Current State)
+- Provisioning workflow implemented with persisted status/steps and 202 responses on tenant creation; retry endpoint available.
+- Background provisioning via Hangfire, with inline fallback when Hangfire/storage is unavailable (dev-friendly).
+- Startup hosted services: tenant catalog migrate/seed (root tenant) and optional auto-provision enqueue.
+- Provider-aware TenantDbContextFactory to select PostgreSQL via appsettings.
+- Audit pipeline fixed to stamp tenant/user on events; audit sink writes per-tenant batches.

src/BuildingBlocks/Jobs/Extensions.cs

@@ -13,6 +13,9 @@ public static class Extensions
 {
     public static IServiceCollection AddHeroJobs(this IServiceCollection services)
     {
+        services.AddOptions<HangfireOptions>()
+            .BindConfiguration(nameof(HangfireOptions));
+
         services.AddHangfireServer(options =>
         {
             options.HeartbeatInterval = TimeSpan.FromSeconds(30);
@@ -23,10 +26,9 @@ public static IServiceCollection AddHeroJobs(this IServiceCollection services)
 
         services.AddHangfire((provider, config) =>
         {
-            var dbOptions = provider
-                .GetRequiredService<IConfiguration>()
-                .GetSection(nameof(DatabaseOptions))
-                .Get<DatabaseOptions>() ?? throw new CustomException("Database options not found");
+            var configuration = provider.GetRequiredService<IConfiguration>();
+            var dbOptions = configuration.GetSection(nameof(DatabaseOptions)).Get<DatabaseOptions>()
+                ?? throw new CustomException("Database options not found");
 
             switch (dbOptions.Provider.ToUpperInvariant())
             {

src/BuildingBlocks/Jobs/FshJobActivator.cs

@@ -38,10 +38,7 @@ private void ReceiveParameters()
             if (tenantInfo is not null)
             {
                 _scope.ServiceProvider.GetRequiredService<IMultiTenantContextSetter>()
-                    .MultiTenantContext = new MultiTenantContext<AppTenantInfo>
-                    {
-                        TenantInfo = tenantInfo
-                    };
+                    .MultiTenantContext = new MultiTenantContext<AppTenantInfo>(tenantInfo);
             }
 
             string userId = _context.GetJobParameter<string>(QueryStringKeys.UserId);
@@ -60,4 +57,4 @@ public override object Resolve(Type type) =>
                 ? _context
                 : _scope.ServiceProvider.GetService(serviceType);
     }
-}
+}

src/BuildingBlocks/Jobs/HangfireOptions.cs

@@ -1,8 +1,8 @@
-namespace FSH.Framework.Jobs;
+namespace FSH.Framework.Jobs;
 
 public class HangfireOptions
 {
     public string UserName { get; set; } = "admin";
     public string Password { get; set; } = "Secure1234!Me";
     public string Route { get; set; } = "/jobs";
-}
+}

src/BuildingBlocks/Jobs/LogJobFilter.cs

@@ -19,7 +19,7 @@ public void OnCreating(CreatingContext context)
         var job = context.Job;
         var jobName = GetJobName(job);
 
-        Logger.InfoFormat(
+        Logger.DebugFormat(
             "Creating job for {0}.", jobName);
     }
 
@@ -30,7 +30,7 @@ public void OnCreated(CreatedContext context)
         var jobId = context.BackgroundJob?.Id ?? "<unknown>";
         var recurringJobId = context.Parameters.TryGetValue("RecurringJobId", out var r) ? r : null;
 
-        Logger.InfoFormat(
+        Logger.DebugFormat(
             "Job created: Id={0}, Name={1}, RecurringJobId={2}",
             jobId,
             jobName,
@@ -45,7 +45,7 @@ public void OnPerforming(PerformingContext context)
         var recurringJobId = context.GetJobParameter<string>("RecurringJobId") ?? "<none>";
         var args = FormatArguments(job.Args);
 
-        Logger.InfoFormat(
+        Logger.DebugFormat(
             "Starting job: Id={0}, Name={1}, RecurringJobId={2}, Queue={3}, Args={4}",
             backgroundJob.Id,
             jobName,
@@ -60,7 +60,7 @@ public void OnPerformed(PerformedContext context)
         var job = backgroundJob.Job;
         var jobName = GetJobName(job);
 
-        Logger.InfoFormat(
+        Logger.DebugFormat(
             "Job completed: Id={0}, Name={1}, Succeeded={2}",
             backgroundJob.Id,
             jobName,
@@ -81,7 +81,7 @@ public void OnStateElection(ElectStateContext context)
 
     public void OnStateApplied(ApplyStateContext context, IWriteOnlyTransaction transaction)
     {
-        Logger.InfoFormat(
+        Logger.DebugFormat(
             "Job state changed: Id={0}, Name={1}, OldState={2}, NewState={3}",
             context.BackgroundJob.Id,
             GetJobName(context.BackgroundJob.Job),
@@ -91,7 +91,7 @@ public void OnStateApplied(ApplyStateContext context, IWriteOnlyTransaction tran
 
     public void OnStateUnapplied(ApplyStateContext context, IWriteOnlyTransaction transaction)
     {
-        Logger.InfoFormat(
+        Logger.DebugFormat(
             "Job state unapplied: Id={0}, Name={1}, OldState={2}",
             context.BackgroundJob.Id,
             GetJobName(context.BackgroundJob.Job),

src/BuildingBlocks/Shared/Multitenancy/AppTenantInfo.cs

@@ -1,18 +1,18 @@
-using Finbuckle.MultiTenant.Abstractions;
+using Finbuckle.MultiTenant.Abstractions;
 
 namespace FSH.Framework.Shared.Multitenancy;
 
-public class AppTenantInfo : ITenantInfo, IAppTenantInfo
+public record AppTenantInfo(string Id, string Identifier, string? Name = null)
+    : TenantInfo(Id, Identifier, Name), IAppTenantInfo
 {
-    public AppTenantInfo()
+    // Parameterless constructor for tooling/EF.
+    public AppTenantInfo() : this(string.Empty, string.Empty)
     {
     }
 
     public AppTenantInfo(string id, string name, string? connectionString, string adminEmail, string? issuer = null)
+        : this(id, id, name)
     {
-        Id = id;
-        Identifier = id;
-        Name = name;
         ConnectionString = connectionString ?? string.Empty;
         AdminEmail = adminEmail;
         IsActive = true;
@@ -21,12 +21,8 @@ public AppTenantInfo(string id, string name, string? connectionString, string ad
         // Add Default 1 Month Validity for all new tenants. Something like a DEMO period for tenants.
         ValidUpto = DateTime.UtcNow.AddMonths(1);
     }
-    public string Id { get; set; } = default!;
-    public string Identifier { get; set; } = default!;
-
-    public string Name { get; set; } = default!;
-    public string ConnectionString { get; set; } = default!;
 
+    public string ConnectionString { get; set; } = string.Empty;
     public string AdminEmail { get; set; } = default!;
     public bool IsActive { get; set; }
     public DateTime ValidUpto { get; set; }
@@ -35,10 +31,13 @@ public AppTenantInfo(string id, string name, string? connectionString, string ad
     public void AddValidity(int months) =>
         ValidUpto = ValidUpto.AddMonths(months);
 
-    public void SetValidity(in DateTime validTill) =>
-        ValidUpto = ValidUpto < validTill
-            ? validTill
+    public void SetValidity(in DateTime validTill)
+    {
+        var normalized = validTill;
+        ValidUpto = ValidUpto < normalized
+            ? normalized
             : throw new InvalidOperationException("Subscription cannot be backdated.");
+    }
 
     public void Activate()
     {
@@ -59,8 +58,10 @@ public void Deactivate()
 
         IsActive = false;
     }
-    string? ITenantInfo.Id { get => Id; set => Id = value ?? throw new InvalidOperationException("Id can't be null."); }
-    string? ITenantInfo.Identifier { get => Identifier; set => Identifier = value ?? throw new InvalidOperationException("Identifier can't be null."); }
-    string? ITenantInfo.Name { get => Name; set => Name = value ?? throw new InvalidOperationException("Name can't be null."); }
-    string? IAppTenantInfo.ConnectionString { get => ConnectionString; set => ConnectionString = value ?? throw new InvalidOperationException("ConnectionString can't be null."); }
-}
+
+    string? IAppTenantInfo.ConnectionString
+    {
+        get => ConnectionString;
+        set => ConnectionString = value ?? throw new InvalidOperationException("ConnectionString can't be null.");
+    }
+}

src/BuildingBlocks/Shared/Multitenancy/MultitenancyConstants.cs

@@ -13,6 +13,7 @@ public static class Root
 
     public const string DefaultPassword = "123Pa$$word!";
     public const string Identifier = "tenant";
+    public const string Schema = "tenant";
 
     public static class Permissions
     {

src/BuildingBlocks/Shared/Shared.csproj

@@ -5,9 +5,10 @@
 		<AssemblyName>FSH.Framework.Shared</AssemblyName>
 	</PropertyGroup>
 	<ItemGroup>
-		<FrameworkReference Include="Microsoft.AspNetCore.App" />
+	  <FrameworkReference Include="Microsoft.AspNetCore.App" />
 	</ItemGroup>
 	<ItemGroup>
+	  <PackageReference Include="Finbuckle.MultiTenant.Abstractions" />
 	  <PackageReference Include="Finbuckle.MultiTenant" />
 	</ItemGroup>
 

src/Directory.Packages.props

@@ -9,15 +9,15 @@
     <IsPackable>true</IsPackable>
   </PropertyGroup>
   <ItemGroup Label="Aspire">
-    <PackageVersion Include="Aspire.Hosting.PostgreSQL" Version="13.0.0" />
-    <PackageVersion Include="Aspire.Hosting.Redis" Version="13.0.0" />
-    <PackageVersion Include="FluentValidation.DependencyInjectionExtensions" Version="12.1.0" />
+    <PackageVersion Include="Aspire.Hosting.PostgreSQL" Version="13.0.1" />
+    <PackageVersion Include="Aspire.Hosting.Redis" Version="13.0.1" />
+    <PackageVersion Include="FluentValidation.DependencyInjectionExtensions" Version="12.1.1" />
     <PackageVersion Include="Microsoft.AspNetCore.Components.Authorization" Version="10.0.0" />
     <PackageVersion Include="Microsoft.Extensions.Http.Resilience" Version="10.0.0" />
     <PackageVersion Include="Microsoft.Extensions.ServiceDiscovery" Version="10.0.0" />
-    <PackageVersion Include="MudBlazor" Version="8.14.0" />
+    <PackageVersion Include="MudBlazor" Version="8.15.0" />
     <PackageVersion Include="MudBlazor.ThemeManager" Version="3.0.0" />
-    <PackageVersion Include="Npgsql.OpenTelemetry" Version="10.0.0-rc.1" />
+    <PackageVersion Include="Npgsql.OpenTelemetry" Version="10.0.0" />
     <PackageVersion Include="OpenTelemetry.Exporter.OpenTelemetryProtocol" Version="1.14.0" />
     <PackageVersion Include="OpenTelemetry.Extensions.Hosting" Version="1.14.0" />
     <PackageVersion Include="OpenTelemetry.Instrumentation.AspNetCore" Version="1.14.0" />
@@ -36,10 +36,12 @@
     <PackageVersion Include="Asp.Versioning.Http" Version="8.1.0" />
     <PackageVersion Include="Asp.Versioning.Mvc" Version="8.1.0" />
     <PackageVersion Include="Asp.Versioning.Mvc.ApiExplorer" Version="8.1.0" />
-    <PackageVersion Include="Finbuckle.MultiTenant" Version="9.4.2" />
-    <PackageVersion Include="Finbuckle.MultiTenant.AspNetCore" Version="9.4.2" />
-    <PackageVersion Include="Finbuckle.MultiTenant.EntityFrameworkCore" Version="9.4.2" />
-    <PackageVersion Include="FluentValidation" Version="12.1.0" />
+    <PackageVersion Include="Finbuckle.MultiTenant" Version="10.0.0" />
+    <PackageVersion Include="Finbuckle.MultiTenant.AspNetCore" Version="10.0.0" />
+    <PackageVersion Include="Finbuckle.MultiTenant.EntityFrameworkCore" Version="10.0.0" />
+    <PackageVersion Include="Finbuckle.MultiTenant.Abstractions" Version="10.0.0" />
+    <PackageVersion Include="Finbuckle.MultiTenant.Identity.EntityFrameworkCore" Version="10.0.0" />
+    <PackageVersion Include="FluentValidation" Version="12.1.1" />
     <PackageVersion Include="Hangfire" Version="1.8.22" />
     <PackageVersion Include="Hangfire.Core" Version="1.8.22" />
     <PackageVersion Include="Hangfire.InMemory" Version="1.0.0" />
@@ -74,12 +76,12 @@
     <PackageVersion Include="MimeKit" Version="4.14.0" />
     <PackageVersion Include="NetArchTest.Rules" Version="1.3.2" />
     <PackageVersion Include="Newtonsoft.Json" Version="13.0.4" />
-    <PackageVersion Include="Npgsql.EntityFrameworkCore.PostgreSQL" Version="10.0.0-rc.2" />
-    <PackageVersion Include="Scalar.AspNetCore" Version="2.10.3" />
+    <PackageVersion Include="Npgsql.EntityFrameworkCore.PostgreSQL" Version="10.0.0" />
+    <PackageVersion Include="Scalar.AspNetCore" Version="2.11.0" />
     <PackageVersion Include="Serilog" Version="4.3.1-dev-02395" />
-    <PackageVersion Include="Serilog.AspNetCore" Version="9.0.0" />
+    <PackageVersion Include="Serilog.AspNetCore" Version="10.0.0" />
     <PackageVersion Include="Serilog.Enrichers.CorrelationId" Version="3.0.1" />
-    <PackageVersion Include="SonarAnalyzer.CSharp" Version="10.15.0.120848" />
+    <PackageVersion Include="SonarAnalyzer.CSharp" Version="10.16.1.129956" />
     <PackageVersion Include="Microsoft.AspNetCore.OpenApi" Version="10.0.0" />
     <PackageVersion Include="Microsoft.NET.Test.Sdk" Version="18.0.1" />
     <PackageVersion Include="System.IdentityModel.Tokens.Jwt" Version="8.14.0" />

src/Modules/Auditing/Modules.Auditing/AuditingModule.cs

@@ -40,7 +40,7 @@ public void ConfigureServices(IHostApplicationBuilder builder)
         // Enrichers used by Audit.Configure (scoped, run on request thread)
         builder.Services.AddScoped<IAuditMaskingService, JsonMaskingService>();
         builder.Services.AddHostedService<AuditingConfigurator>();
-        builder.Services.AddSingleton<IAuditScope, HttpAuditScope>();
+        builder.Services.AddScoped<IAuditScope, HttpAuditScope>();
 
         builder.Services.AddSingleton<ChannelAuditPublisher>();
         builder.Services.AddSingleton<IAuditPublisher>(sp => sp.GetRequiredService<ChannelAuditPublisher>());