Refactor server logs into diagnostics structured logs

[sfmskywalker]

Summary

Refactors the server logs feature into the diagnostics structured logs module:

• Renames Elsa.ServerLogs to Elsa.Diagnostics.StructuredLogs
• Renames public APIs, options, features, shell feature, permissions, routes, docs, tests, and sample host wiring
• Moves the feature contract toward structured/semantic logs instead of raw console streaming
• Adds message template capture from {OriginalFormat}
• Adds active logging scope capture via ISupportExternalScope
• Adds span filtering and broader text matching across templates, properties, and scopes

Why

We decided to split diagnostics into separate modules:

• Structured Logs: semantic ILogger records
• Console Logs: future raw stdout/stderr streaming
• OpenTelemetry: future trace/metric explorer

This PR makes the current implementation accurately represent the Structured Logs module before adding the other diagnostics surfaces.

Validation

• dotnet build src/modules/Elsa.Diagnostics.StructuredLogs/Elsa.Diagnostics.StructuredLogs.csproj
• dotnet test test/unit/Elsa.Diagnostics.StructuredLogs.UnitTests/Elsa.Diagnostics.StructuredLogs.UnitTests.csproj
• dotnet test test/integration/Elsa.Diagnostics.StructuredLogs.IntegrationTests/Elsa.Diagnostics.StructuredLogs.IntegrationTests.csproj
• rg "Elsa\\.ServerLogs|ServerLogStreaming|server-logs|read:server-logs" src test

Residual warnings are existing repo warnings, including NU1903, ILLink analyzer exceptions, and unrelated nullable/unread-parameter warnings.

[gitguardian]

⚠️ GitGuardian has uncovered 1 secret following the scan of your pull request.

Please consider investigating the findings and remediating the incidents. Failure to do so may lead to compromising the associated services or software components.

🔎 Detected hardcoded secret in your pull request

GitGuardian id	GitGuardian status	Secret	Commit	Filename
32583717	Triggered	Generic Password	ea9ed76	test/unit/Elsa.Diagnostics.StructuredLogs.UnitTests/Logging/StructuredLogLoggerProviderTests.cs	View secret

🛠 Guidelines to remediate hardcoded secrets

1. Understand the implications of revoking this secret by investigating where it is used in your code.
2. Replace and store your secret safely. Learn here the best practices.
3. Revoke and rotate this secret.
4. If possible, rewrite git history. Rewriting git history is not a trivial act. You might completely break other contributing developers' workflow and you risk accidentally deleting legitimate data.

To avoid such incidents in the future consider

• following these best practices for managing and storing secrets including API keys and other credentials
• install secret detection on pre-commit to catch secret before it leaves your machine and ease remediation.

---

^{🦉 GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.}

[greptile-apps]

Greptile Summary

This PR renames Elsa.ServerLogs to Elsa.Diagnostics.StructuredLogs, updating all public APIs, routes, permissions, tests, and wiring while adding three new capabilities: {OriginalFormat} message-template capture, active logging scope extraction via ISupportExternalScope, and broader text search across templates, properties, and scopes.

• Core logger (StructuredLogLogger) now extracts the Serilog-style {OriginalFormat} key from the log state, serializes all structured properties and active scopes into StructuredLogEvent, then passes the event through StructuredLogRedactor before publishing it fire-and-forget to the provider.
• In-memory provider uses a lock-protected RingBuffer for recent-log storage and per-subscriber channel capacity enforcement with dropped-event summaries; the StructuredLogSubscriptionManager uses ConcurrentDictionary with atomic key+value removal to safely handle concurrent subscribe/unsubscribe cycles.
• Authorization gap between the hub ([Authorize]) and the REST endpoints (read:diagnostics:structured-logs permission) is documented as intentional in the README but means any authenticated user can stream live logs via SignalR regardless of permission grants.

Confidence Score: 4/5

Safe to merge; the rename is thorough and the new capabilities are well-implemented with sound concurrency.

The core ring buffer, subscriber channel management, redactor, and filter evaluator are all solid. The two open points — reflection in TryAddKeyValueScope and the fire-and-forget PublishAsync discard — are contained to the logger and do not affect correctness for the current in-memory provider. The hub's weaker [Authorize] guard is a documented design choice carried over from Elsa.ServerLogs.

src/modules/Elsa.Diagnostics.StructuredLogs/Logging/StructuredLogLogger.cs and src/modules/Elsa.Diagnostics.StructuredLogs/RealTime/StructuredLogsHub.cs

Security Review

• Authorization split between REST and SignalR (StructuredLogsHub): the hub is decorated with [Authorize] (authentication only) while both REST endpoints enforce read:diagnostics:structured-logs. Any authenticated principal that lacks that permission can still open a SignalR connection and receive the full live log stream, including redacted-but-structured log events, category names, trace/span IDs, tenant IDs, and source metadata.
• Sensitive data in scope/property extraction: StructuredLogRedactor correctly applies name-based and pattern-based redaction to all properties and scopes before events are buffered or streamed. No raw credential leakage was identified in the current pipeline.
• No secrets, hardcoded credentials, or SQL injection surfaces were found in the changed files.

Important Files Changed

Filename	Overview
src/modules/Elsa.Diagnostics.StructuredLogs/Logging/StructuredLogLogger.cs	New logger implementation; reflection used in TryAddKeyValueScope hot path and PublishAsync ValueTask is discarded without error handling.
src/modules/Elsa.Diagnostics.StructuredLogs/RealTime/StructuredLogsHub.cs	SignalR hub uses plain [Authorize] while REST endpoints enforce the specific read:diagnostics:structured-logs permission — intentional per README but creates an authorization gap for the streaming path.
src/modules/Elsa.Diagnostics.StructuredLogs/Providers/InMemory/InMemoryStructuredLogProvider.cs	Solid ring-buffer + subscriber channel implementation; manual capacity enforcement via _pendingItemCount is correct; dropped-event summary re-queuing logic is sound.
src/modules/Elsa.Diagnostics.StructuredLogs/RealTime/StructuredLogSubscriptionManager.cs	Concurrent subscription lifecycle management is correctly handled via ConcurrentDictionary and ICollection.Remove atomic key+value comparison; fire-and-forget StreamAsync exceptions are properly caught.
src/modules/Elsa.Diagnostics.StructuredLogs/Services/StructuredLogFilterEvaluator.cs	Clean static evaluator covering all filter dimensions including MessageTemplate, Properties, and Scopes for text search; no issues found.
src/modules/Elsa.Diagnostics.StructuredLogs/Services/StructuredLogRedactor.cs	Pre-compiled regex patterns and HashSet-based sensitive name lookup are correctly applied to messages, templates, exceptions, properties, and scopes.
src/modules/Elsa.Diagnostics.StructuredLogs/Providers/InMemory/RingBuffer.cs	Thread-safe bounded ring buffer using Queue + lock; Snapshot returns a copy; DroppedCount incremented atomically within the lock.
src/modules/Elsa.Diagnostics.StructuredLogs/ShellFeatures/StructuredLogsFeature.cs	Shell feature correctly copies all bindable option properties; implements both IFastEndpointsShellFeature and IWebShellFeature as expected.

Sequence Diagram

sequenceDiagram
    participant App as Application Code
    participant SLL as StructuredLogLogger
    participant Red as StructuredLogRedactor
    participant IMLP as InMemoryStructuredLogProvider
    participant RB as RingBuffer
    participant Sub as StructuredLogSubscriber
    participant SM as SubscriptionManager
    participant Client as Studio Client

    App->>SLL: ILogger.Log(level, state, exception)
    SLL->>SLL: ExtractProperties + ExtractScopes
    SLL->>Red: Redact(logEvent)
    Red-->>SLL: redacted logEvent
    SLL->>IMLP: PublishAsync(redacted) [fire-and-forget]
    IMLP->>RB: Add(logEvent)
    IMLP->>Sub: TryWrite(logEvent, capacity)
    Sub-->>IMLP: item enqueued / dropped summary queued

    Client->>SM: SubscribeAsync(filter) via Hub
    SM->>IMLP: SubscribeWithDroppedEventsAsync(filter)
    IMLP-->>SM: IAsyncEnumerable stream
    SM->>Client: ReceiveLogEventAsync / ReceiveDroppedEventsAsync

    Client->>IMLP: GET /diagnostics/structured-logs/recent
    IMLP->>RB: Snapshot()
    RB-->>IMLP: filtered + ordered list
    IMLP-->>Client: RecentStructuredLogsResult

Loading

Prompt To Fix All With AI

Fix the following 3 code review issues. Work through them one at a time, proposing concise fixes.

---

### Issue 1 of 3
src/modules/Elsa.Diagnostics.StructuredLogs/Logging/StructuredLogLogger.cs:106-138
**Reflection on the hot logging path**

`TryAddKeyValueScope` iterates over the scope as a bare `IEnumerable`, then uses `GetType().GetProperty("Key").GetValue(value)` to extract each `KeyValuePair`'s key and value. This reflection runs for every log entry that carries a structured scope (e.g. `BeginScope(new Dictionary<string, object?>{...})`). The common path — a `Dictionary<string, object?>` scope — produces `KeyValuePair<string, object?>` elements; a direct pattern-match to `IEnumerable<KeyValuePair<string, object?>>` at the scope level (before iterating) would cover that case without any reflection. The current approach also silently produces null for `KeyValuePair<string, string>` scopes because the value cast `as string` fails for non-`object?` value types rather than calling `ToString`.

### Issue 2 of 3
src/modules/Elsa.Diagnostics.StructuredLogs/Logging/StructuredLogLogger.cs:59-60
**`PublishAsync` ValueTask is silently discarded**

`ILogger.Log` is synchronous so fire-and-forget publishing is unavoidable here, but the discard (`_ = logProvider.PublishAsync(...)`) swallows any exception the provider might throw — including synchronous throws. For the current `InMemoryStructuredLogProvider` this is safe (it always completes synchronously), but a future provider that performs async work or throws on the synchronous path would fail silently.

```suggestion
        var redacted = redactor.Redact(logEvent);
        var publishTask = logProvider.PublishAsync(redacted);
        if (!publishTask.IsCompletedSuccessfully)
            _ = publishTask.AsTask().ContinueWith(
                static t => { _ = t.Exception; },
                TaskContinuationOptions.OnlyOnFaulted | TaskContinuationOptions.ExecuteSynchronously);
```

### Issue 3 of 3
src/modules/Elsa.Diagnostics.StructuredLogs/RealTime/StructuredLogsHub.cs:8-9
**Hub authorization does not enforce the `read:diagnostics:structured-logs` permission**

The hub uses `[Authorize]` (any authenticated user) while both REST endpoints use `ConfigurePermissions(StructuredLogsPermissions.Read)` which enforces the `read:diagnostics:structured-logs` permission. Any authenticated user — including those explicitly denied that permission — can open a SignalR connection and receive the full live log stream. The README explicitly documents this as an intentional match to the existing Elsa workflow hub pattern, but it does mean the REST permission grant has no effect on real-time streaming. If the intent is that `read:diagnostics:structured-logs` controls access to all log data, the hub attribute should be updated to require the same policy.

_{Reviews (1): Last reviewed commit: "Switch from PostgreSQL to Sqlite for wor..." | Re-trigger Greptile}

[greptile-apps]

Reflection on the hot logging path

TryAddKeyValueScope iterates over the scope as a bare IEnumerable, then uses GetType().GetProperty("Key").GetValue(value) to extract each KeyValuePair's key and value. This reflection runs for every log entry that carries a structured scope (e.g. BeginScope(new Dictionary<string, object?>{...})). The common path — a Dictionary<string, object?> scope — produces KeyValuePair<string, object?> elements; a direct pattern-match to IEnumerable<KeyValuePair<string, object?>> at the scope level (before iterating) would cover that case without any reflection. The current approach also silently produces null for KeyValuePair<string, string> scopes because the value cast as string fails for non-object? value types rather than calling ToString.

Prompt To Fix With AI

This is a comment left during a code review.
Path: src/modules/Elsa.Diagnostics.StructuredLogs/Logging/StructuredLogLogger.cs
Line: 106-138

Comment:
**Reflection on the hot logging path**

`TryAddKeyValueScope` iterates over the scope as a bare `IEnumerable`, then uses `GetType().GetProperty("Key").GetValue(value)` to extract each `KeyValuePair`'s key and value. This reflection runs for every log entry that carries a structured scope (e.g. `BeginScope(new Dictionary<string, object?>{...})`). The common path — a `Dictionary<string, object?>` scope — produces `KeyValuePair<string, object?>` elements; a direct pattern-match to `IEnumerable<KeyValuePair<string, object?>>` at the scope level (before iterating) would cover that case without any reflection. The current approach also silently produces null for `KeyValuePair<string, string>` scopes because the value cast `as string` fails for non-`object?` value types rather than calling `ToString`.

How can I resolve this? If you propose a fix, please make it concise.

[greptile-apps]

PublishAsync ValueTask is silently discarded

ILogger.Log is synchronous so fire-and-forget publishing is unavoidable here, but the discard (_ = logProvider.PublishAsync(...)) swallows any exception the provider might throw — including synchronous throws. For the current InMemoryStructuredLogProvider this is safe (it always completes synchronously), but a future provider that performs async work or throws on the synchronous path would fail silently.

Suggested change

	var redacted = redactor.Redact(logEvent);
	_ = logProvider.PublishAsync(redacted);
	var redacted = redactor.Redact(logEvent);
	var publishTask = logProvider.PublishAsync(redacted);
	if (!publishTask.IsCompletedSuccessfully)
	_ = publishTask.AsTask().ContinueWith(
	static t => { _ = t.Exception; },
	TaskContinuationOptions.OnlyOnFaulted | TaskContinuationOptions.ExecuteSynchronously);

Prompt To Fix With AI

This is a comment left during a code review.
Path: src/modules/Elsa.Diagnostics.StructuredLogs/Logging/StructuredLogLogger.cs
Line: 59-60

Comment:
**`PublishAsync` ValueTask is silently discarded**

`ILogger.Log` is synchronous so fire-and-forget publishing is unavoidable here, but the discard (`_ = logProvider.PublishAsync(...)`) swallows any exception the provider might throw — including synchronous throws. For the current `InMemoryStructuredLogProvider` this is safe (it always completes synchronously), but a future provider that performs async work or throws on the synchronous path would fail silently.

```suggestion
        var redacted = redactor.Redact(logEvent);
        var publishTask = logProvider.PublishAsync(redacted);
        if (!publishTask.IsCompletedSuccessfully)
            _ = publishTask.AsTask().ContinueWith(
                static t => { _ = t.Exception; },
                TaskContinuationOptions.OnlyOnFaulted | TaskContinuationOptions.ExecuteSynchronously);
```

How can I resolve this? If you propose a fix, please make it concise.

[greptile-apps]

Hub authorization does not enforce the read:diagnostics:structured-logs permission

The hub uses [Authorize] (any authenticated user) while both REST endpoints use ConfigurePermissions(StructuredLogsPermissions.Read) which enforces the read:diagnostics:structured-logs permission. Any authenticated user — including those explicitly denied that permission — can open a SignalR connection and receive the full live log stream. The README explicitly documents this as an intentional match to the existing Elsa workflow hub pattern, but it does mean the REST permission grant has no effect on real-time streaming. If the intent is that read:diagnostics:structured-logs controls access to all log data, the hub attribute should be updated to require the same policy.

Prompt To Fix With AI

This is a comment left during a code review.
Path: src/modules/Elsa.Diagnostics.StructuredLogs/RealTime/StructuredLogsHub.cs
Line: 8-9

Comment:
**Hub authorization does not enforce the `read:diagnostics:structured-logs` permission**

The hub uses `[Authorize]` (any authenticated user) while both REST endpoints use `ConfigurePermissions(StructuredLogsPermissions.Read)` which enforces the `read:diagnostics:structured-logs` permission. Any authenticated user — including those explicitly denied that permission — can open a SignalR connection and receive the full live log stream. The README explicitly documents this as an intentional match to the existing Elsa workflow hub pattern, but it does mean the REST permission grant has no effect on real-time streaming. If the intent is that `read:diagnostics:structured-logs` controls access to all log data, the hub attribute should be updated to require the same policy.

How can I resolve this? If you propose a fix, please make it concise.