Skip to content
Merged
Changes from 2 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
78 changes: 78 additions & 0 deletions src/content/docs/identityserver/troubleshooting.md
Original file line number Diff line number Diff line change
Expand Up @@ -166,3 +166,81 @@ version used.
<PackageReference Include="System.IdentityModel.Tokens.Jwt" Version="7.4.0"/>
</ItemGroup>
```

## Performance Issues

In some installations, upgrading .NET and IdentityServer has caused performance issues. Since the IdentityServer and
.NET version upgrades typically are done at the same time, it is sometimes hard to tell what the root cause is
for the performance degradation. When working with installations to find the root cause, there are some dependencies
that have been found to cause issues in specific verisons.

### PostgreSQL Pooling

There are issues with some versions of the PostgreSQL client library that gives large memory consumption. Enabling
pooling on the operational store has solved this in the past:

```csharp
.AddOperationalStore(options =>
{
// Enable pooling:
options.EnablePooling = true;

// More settings....
})
```

### Entity Framework Core & Microsoft SQL OPENJSON

Entity Framework Core version 8 introduced a new behaviour when creating `WHERE IN()` sql clauses. Previously the
Comment thread
maartenba marked this conversation as resolved.
Outdated
possible values were supplied as parameters, which meant that the query text was dependent on the number of items
in the collection. This was solved by sending the parameters as a JSON object and using `OPENJSON` to read the parameters.
While this enabled query plan caching, it unfortunately caused Microsoft SQL Server to generate bad query execution plans.

Please see [this EF Core GitHub Issue](https://github.com/dotnet/efcore/issues/32394#issuecomment-2266634632) for information
and possible mitigations.

### Microsoft Azure

The `Azure.Core` package versions `1.41.0` and prior had an issue that caused delays when accessing Azure resources.
This could be Azure blob storage or key vault for data protection or Azure SQL Server for stores, especially if managed
identities are used. This package is typically not referenced directly but brought in as a transient dependency
through other packages. Ensure to use version `1.42.0` or later if you are hosting on Azure.

### Entity Framework Core, Microsoft.Data.SqlClient, and SqlServerRetryingExecutionStrategy

As more developers migrate their database-powered application to the cloud,
they will need to handle intermittent connection failures. In most cases, these transient connection failures occur and resolve in a short period of time, allowing the application to self-correct and continue processing requests. The strategy is known as **connection resiliency**.

In recent versions of Entity Framework Core and `Microsoft.Data.SqlClient`, you can enable this retry strategy explicitly, but in the case of `Microsoft.Data.SqlClient`, when operating in a cloud environment, this strategy is enabled by default or defined in the connection string.

```csharp
protected override void OnConfiguring(DbContextOptionsBuilder optionsBuilder)
{
optionsBuilder
.UseSqlServer(
@"Server=(localdb)\mssqllocaldb;Database=EFMiscellanous.ConnectionResiliency;Trusted_Connection=True;ConnectRetryCount=0",
options => options.EnableRetryOnFailure());
}
```

In most cases, this is a _good feature to have enabled_ but there are drawbacks that can cause severe system degradation.

- Enabling retry on failure causes Entity Framework Core to buffer the result set. Significantly increasing memory requirements and causing garbage collection pauses.
- Some versions of `Microsoft.Data.SqlClient` call `Thread.Sleep` that can lock threads for up to **_10 seconds_**. This can lead to thread exhaustion and server unresponsiveness. We've isolated this issue to versions.

| Microsoft.EntityFrameworkCore.SqlServer | Microsoft.Data.SqlClient | Status |
|:----------------------------------------|:-------------------------|:-----------|
| `8.0.0` | `>=5.1.1` | ✅ Good |
| `8.0.3` | `>=5.1.5` | ❌ Affected |
| `8.0.4` | `>=5.1.5` | ❌ Affected |
| `8.0.6` | `>=5.1.5` | ❌ Affected |
| `8.0.11` | `>=5.1.6` | ✅ Good |
| `9.0.1` | `>=5.1.6` | ✅ Good |
| `>9.0.1` | `>=6.0.0` | ❌ Affected |

Architectural issues that may be causing connection resiliency issues you may want to investigate:

- Lack of caching in a high-load production environment.
- Under-provisioned database instance with limited resources or connections available.
- Datacenter networking issues caused by incorrect zoning choices.
- Under-provisioned application host with limited cores/threads.