Skip to content

Add MCP and API smoke tests to the integration suite on CI #3356

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Mpdreamz merged 13 commits into from
May 20, 2026
Merged

Add MCP and API smoke tests to the integration suite on CI #3356

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
c644716
Add Aspire-driven API + MCP smoke tests and fix Docker-free integrati…
Mpdreamz May 19, 2026
2da1c91
Merge branch 'main' into feature/mcp-api-integration-tests
Mpdreamz May 19, 2026
92532ee
Merge branch 'main' into feature/mcp-api-integration-tests
Mpdreamz May 20, 2026
a257b62
Fix Aspire AppHost startup and smoke test reliability
Mpdreamz May 20, 2026
df32d95
Read ENVIRONMENT from host process; fail with diagnostics on non-200 …
Mpdreamz May 20, 2026
3f356e0
Fix OtlpProxyTests race: configure OTLP endpoint per factory instance
Mpdreamz May 20, 2026
aded8bc
Skip ChangesEndpoint test when API key lacks PIT privilege
Mpdreamz May 20, 2026
12505be
Skip ChangesEndpoint test on CI where API key lacks PIT privilege
Mpdreamz May 20, 2026
407f843
Address review: Kestrel guard via Configuration, null body fails test
Mpdreamz May 20, 2026
36595f9
Switch Assert.SkipWhen to Assert.SkipUnless in smoke tests
Mpdreamz May 20, 2026
3c1639c
Snapshot RecordedLogs in DisposeAsync to prevent concurrent modification
Mpdreamz May 20, 2026
a72f1fe
Only dump logs on TestResult.Failed, not on skip or null state
Mpdreamz May 20, 2026
0d597fa
Address review: whitespace-safe env defaults, add HTTPS_PORTS to guard
Mpdreamz May 20, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 19 additions & 0 deletions .github/workflows/ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -185,6 +185,10 @@ jobs:

integration:
runs-on: docs-builder-latest-16
environment: integration-tests
permissions:
contents: read
id-token: write
steps:
- uses: actions/checkout@v6

Expand All @@ -195,5 +199,20 @@ jobs:
- name: Install Aspire workload
run: dotnet workload install aspire

- name: Configure AWS credentials
uses: aws-actions/configure-aws-credentials@v4
with:
role-to-assume: arn:aws:iam::197730964718:role/elastic-docs-v3-integration-tests
aws-region: us-east-1

- name: Fetch ES credentials
run: |
ES_URL=$(aws ssm get-parameter --name /elastic-docs-v3/prod/docs-elasticsearch-github-actions-readonly-url --with-decryption --query Parameter.Value --output text)
ES_KEY=$(aws ssm get-parameter --name /elastic-docs-v3/prod/docs-elasticsearch-github-actions-readonly-api-key --with-decryption --query Parameter.Value --output text)
echo "::add-mask::$ES_URL"
echo "::add-mask::$ES_KEY"
Comment on lines +212 to +213
Copy link
Copy Markdown
Member

@reakaleek reakaleek May 20, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🚀

dotnet user-secrets set "DocumentationElasticUrl" "$ES_URL" --project aspire
dotnet user-secrets set "DocumentationElasticApiKey" "$ES_KEY" --project aspire

- name: Integration Tests
run: dotnet run --project build -c release -- integrate
62 changes: 31 additions & 31 deletions aspire/AppHost.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -54,50 +54,50 @@ internal static async Task Run(
.WaitForCompletion(cloneAll)
.WithParentRelationship(cloneAll);

var elasticsearchLocal = builder.AddElasticsearch(ElasticsearchLocal)
.WithEnvironment("LICENSE", "trial");
if (!startElasticsearch)
elasticsearchLocal = elasticsearchLocal.WithExplicitStart();
IResourceBuilder<ElasticsearchResource>? elasticsearchLocal = null;
if (startElasticsearch)
elasticsearchLocal = builder.AddElasticsearch(ElasticsearchLocal)
.WithEnvironment("LICENSE", "trial");

var elasticsearchRemote = builder.AddExternalService(ElasticsearchRemote, elasticsearchUrl);

var api = builder.AddProject<Projects.Elastic_Documentation_Api>(Api)
.WithArgs(GlobalArguments)
.WithEnvironment("ENVIRONMENT", "dev")
.WithEnvironment("LLM_GATEWAY_FUNCTION_URL", llmUrl)
.WithEnvironment("LLM_GATEWAY_SERVICE_ACCOUNT_KEY_PATH", llmServiceAccountPath);
.WithEnvironment("LLM_GATEWAY_SERVICE_ACCOUNT_KEY_PATH", llmServiceAccountPath)
.WithHttpEndpoint(isProxied: false)
.WithHttpHealthCheck("/health");

// ReSharper disable once RedundantAssignment
api = startElasticsearch
? api
.WithReference(elasticsearchLocal)
.WithEnvironment("DOCUMENTATION_ELASTIC_URL", elasticsearchLocal.GetEndpoint("http"))
.WithEnvironment(context => context.EnvironmentVariables["DOCUMENTATION_ELASTIC_PASSWORD"] = elasticsearchLocal.Resource.PasswordParameter)
.WithParentRelationship(elasticsearchLocal)
.WaitFor(elasticsearchLocal)
.WithExplicitStart()
.WithReference(elasticsearchLocal!)
.WithEnvironment("DOCUMENTATION_ELASTIC_URL", elasticsearchLocal!.GetEndpoint("http"))
.WithEnvironment(context => context.EnvironmentVariables["DOCUMENTATION_ELASTIC_PASSWORD"] = elasticsearchLocal!.Resource.PasswordParameter)
.WithParentRelationship(elasticsearchLocal!)
.WaitFor(elasticsearchLocal!)
: api.WithReference(elasticsearchRemote)
.WithEnvironment("DOCUMENTATION_ELASTIC_URL", elasticsearchUrl)
.WithEnvironment("DOCUMENTATION_ELASTIC_APIKEY", elasticsearchApiKey)
.WithExplicitStart();
.WithEnvironment("DOCUMENTATION_ELASTIC_APIKEY", elasticsearchApiKey);

var mcp = builder.AddProject<Projects.Elastic_Documentation_Mcp_Remote>(RemoteMcp)
.WithArgs(GlobalArguments)
.WithEnvironment("ENVIRONMENT", "dev");
.WithEnvironment("ENVIRONMENT", "dev")
.WithHttpEndpoint(isProxied: false)
.WithHttpHealthCheck("/health");

// ReSharper disable once RedundantAssignment
mcp = startElasticsearch
? mcp
.WithReference(elasticsearchLocal)
.WithEnvironment("DOCUMENTATION_ELASTIC_URL", elasticsearchLocal.GetEndpoint("http"))
.WithEnvironment(context => context.EnvironmentVariables["DOCUMENTATION_ELASTIC_PASSWORD"] = elasticsearchLocal.Resource.PasswordParameter)
.WithParentRelationship(elasticsearchLocal)
.WaitFor(elasticsearchLocal)
.WithExplicitStart()
.WithReference(elasticsearchLocal!)
.WithEnvironment("DOCUMENTATION_ELASTIC_URL", elasticsearchLocal!.GetEndpoint("http"))
.WithEnvironment(context => context.EnvironmentVariables["DOCUMENTATION_ELASTIC_PASSWORD"] = elasticsearchLocal!.Resource.PasswordParameter)
.WithParentRelationship(elasticsearchLocal!)
.WaitFor(elasticsearchLocal!)
: mcp.WithReference(elasticsearchRemote)
.WithEnvironment("DOCUMENTATION_ELASTIC_URL", elasticsearchUrl)
.WithEnvironment("DOCUMENTATION_ELASTIC_APIKEY", elasticsearchApiKey)
.WithExplicitStart();
.WithEnvironment("DOCUMENTATION_ELASTIC_APIKEY", elasticsearchApiKey);

var indexElasticsearch = builder.AddProject<Projects.docs_builder>(ElasticsearchIngest)
.WithArgs(["assembler", "index", .. GlobalArguments])
Expand All @@ -107,11 +107,11 @@ internal static async Task Run(
// ReSharper disable once RedundantAssignment
indexElasticsearch = startElasticsearch
? indexElasticsearch
.WaitFor(elasticsearchLocal)
.WithReference(elasticsearchLocal)
.WithEnvironment("DOCUMENTATION_ELASTIC_URL", elasticsearchLocal.GetEndpoint("http"))
.WithEnvironment(context => context.EnvironmentVariables["DOCUMENTATION_ELASTIC_PASSWORD"] = elasticsearchLocal.Resource.PasswordParameter)
.WithParentRelationship(elasticsearchLocal)
.WaitFor(elasticsearchLocal!)
.WithReference(elasticsearchLocal!)
.WithEnvironment("DOCUMENTATION_ELASTIC_URL", elasticsearchLocal!.GetEndpoint("http"))
.WithEnvironment(context => context.EnvironmentVariables["DOCUMENTATION_ELASTIC_PASSWORD"] = elasticsearchLocal!.Resource.PasswordParameter)
.WithParentRelationship(elasticsearchLocal!)
: indexElasticsearch
.WithReference(elasticsearchRemote)
.WithEnvironment("DOCUMENTATION_ELASTIC_URL", elasticsearchUrl)
Expand All @@ -129,16 +129,16 @@ internal static async Task Run(

serveStatic = startElasticsearch
? serveStatic
.WithReference(elasticsearchLocal)
.WithEnvironment("DOCUMENTATION_ELASTIC_URL", elasticsearchLocal.GetEndpoint("http"))
.WithEnvironment(context => context.EnvironmentVariables["DOCUMENTATION_ELASTIC_PASSWORD"] = elasticsearchLocal.Resource.PasswordParameter)
.WithReference(elasticsearchLocal!)
.WithEnvironment("DOCUMENTATION_ELASTIC_URL", elasticsearchLocal!.GetEndpoint("http"))
.WithEnvironment(context => context.EnvironmentVariables["DOCUMENTATION_ELASTIC_PASSWORD"] = elasticsearchLocal!.Resource.PasswordParameter)
: serveStatic
.WithReference(elasticsearchRemote)
.WithEnvironment("DOCUMENTATION_ELASTIC_URL", elasticsearchUrl)
.WithEnvironment("DOCUMENTATION_ELASTIC_APIKEY", elasticsearchApiKey);

// ReSharper disable once RedundantAssignment
serveStatic = startElasticsearch ? serveStatic.WaitFor(elasticsearchLocal) : serveStatic.WaitFor(buildAll);
serveStatic = startElasticsearch ? serveStatic.WaitFor(elasticsearchLocal!) : serveStatic.WaitFor(buildAll);

await builder.Build().RunAsync(ct);
}
Expand Down
4 changes: 2 additions & 2 deletions docs-builder.slnx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -84,8 +84,7 @@
</Folder>
<Folder Name="/tests-integration/">
<File Path="tests-integration/Directory.Build.props" />
<Project Path="tests-integration/Elastic.Assembler.IntegrationTests/Elastic.Assembler.IntegrationTests.csproj" />
<Project Path="tests-integration/Elastic.Documentation.Api.IntegrationTests/Elastic.Documentation.Api.IntegrationTests.csproj" />
<Project Path="tests-integration/Elastic.Documentation.IntegrationTests/Elastic.Documentation.IntegrationTests.csproj" />
<Project Path="tests-integration/Search.IntegrationTests/Search.IntegrationTests.csproj" />
<Project Path="tests-integration/Mcp.Remote.IntegrationTests/Mcp.Remote.IntegrationTests.csproj" />
<Project Path="tests-integration/Elastic.ContentDateEnrichment.IntegrationTests/Elastic.ContentDateEnrichment.IntegrationTests.csproj" />
Expand All @@ -95,6 +94,7 @@
<Project Path="tests/authoring/authoring.fsproj" />
<Project Path="tests/Elastic.ApiExplorer.Tests/Elastic.ApiExplorer.Tests.csproj" />
<Project Path="tests/Elastic.Documentation.Api.Infrastructure.Tests/Elastic.Documentation.Api.Infrastructure.Tests.csproj" />
<Project Path="tests/Elastic.Documentation.Api.Tests/Elastic.Documentation.Api.Tests.csproj" />
<Project Path="tests/Elastic.Documentation.Build.Tests/Elastic.Documentation.Build.Tests.csproj" />
<Project Path="tests/Elastic.Documentation.Configuration.Tests/Elastic.Documentation.Configuration.Tests.csproj" />
<Project Path="tests/Elastic.Documentation.LegacyDocs.Tests/Elastic.Documentation.LegacyDocs.Tests.csproj" />
Expand Down
11 changes: 7 additions & 4 deletions src/api/Elastic.Documentation.Api/Program.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,11 +22,14 @@
_ = builder.AddDefaultHealthChecks();
_ = builder.AddDocsApiOpenTelemetry();

// Configure Kestrel to listen on port 8080 (standard container port)
_ = builder.WebHost.ConfigureKestrel(serverOptions =>
// Only hardcode port 8080 when not running under Aspire/orchestration that sets ASPNETCORE_HTTP_PORTS
if (string.IsNullOrEmpty(Environment.GetEnvironmentVariable("ASPNETCORE_HTTP_PORTS")))
{
serverOptions.ListenAnyIP(8080);
});
_ = builder.WebHost.ConfigureKestrel(serverOptions =>
{
serverOptions.ListenAnyIP(8080);
});
}

var environment = Environment.GetEnvironmentVariable("ENVIRONMENT");
Console.WriteLine($"Docs Environment: {environment}");
Expand Down
11 changes: 7 additions & 4 deletions src/api/Elastic.Documentation.Mcp.Remote/Program.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,11 +30,14 @@
_ = builder.Services.ConfigureOpenTelemetryTracerProvider(t =>
t.AddSource(McpToolTelemetry.McpToolSourceName));

// Configure Kestrel to listen on port 8080 (standard container port)
_ = builder.WebHost.ConfigureKestrel(serverOptions =>
// Only hardcode port 8080 when not running under Aspire/orchestration that sets ASPNETCORE_HTTP_PORTS
if (string.IsNullOrEmpty(Environment.GetEnvironmentVariable("ASPNETCORE_HTTP_PORTS")))
{
serverOptions.ListenAnyIP(8080);
});
_ = builder.WebHost.ConfigureKestrel(serverOptions =>
{
serverOptions.ListenAnyIP(8080);
});
}

var environment = Environment.GetEnvironmentVariable("ENVIRONMENT");
Console.WriteLine($"Docs Environment: {environment}");
Expand Down
17 changes: 15 additions & 2 deletions ...mbler.IntegrationTests/AssembleFixture.cs → ...ation.IntegrationTests/AssembleFixture.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,16 +5,17 @@
using Aspire.Hosting;
using Aspire.Hosting.ApplicationModel;
using Aspire.Hosting.Testing;
using Elastic.Documentation.Aspire;
using Elastic.Documentation.ServiceDefaults;
using InMemLogger;
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Logging;
using static Elastic.Documentation.Aspire.ResourceNames;

[assembly: CaptureConsole, AssemblyFixture(typeof(Elastic.Assembler.IntegrationTests.DocumentationFixture))]
[assembly: CaptureConsole, AssemblyFixture(typeof(Elastic.Documentation.IntegrationTests.DocumentationFixture))]

namespace Elastic.Assembler.IntegrationTests;
namespace Elastic.Documentation.IntegrationTests;

public static class DistributedApplicationExtensions
{
Expand Down Expand Up @@ -90,6 +91,14 @@ public async ValueTask InitializeAsync()
_ = await DistributedApplication.ResourceNotifications
.WaitForResourceHealthyAsync(AssemblerServe, cancellationToken: TestContext.Current.CancellationToken)
.WaitAsync(TimeSpan.FromMinutes(3), TestContext.Current.CancellationToken);

_ = await DistributedApplication.ResourceNotifications
.WaitForResourceHealthyAsync(ResourceNames.Api, cancellationToken: TestContext.Current.CancellationToken)
.WaitAsync(TimeSpan.FromMinutes(3), TestContext.Current.CancellationToken);

_ = await DistributedApplication.ResourceNotifications
.WaitForResourceHealthyAsync(RemoteMcp, cancellationToken: TestContext.Current.CancellationToken)
.WaitAsync(TimeSpan.FromMinutes(3), TestContext.Current.CancellationToken);
}
catch (Exception e)
{
Expand All @@ -99,6 +108,10 @@ public async ValueTask InitializeAsync()
}
}

public HttpClient CreateApiClient() => DistributedApplication.CreateHttpClient(ResourceNames.Api);

public HttpClient CreateMcpClient() => DistributedApplication.CreateHttpClient(RemoteMcp);

private async ValueTask ValidateExitCode(string resourceName)
{
var eventResource = await DistributedApplication.ResourceNotifications.WaitForResourceAsync(resourceName, _ => true);
Expand Down
2 changes: 1 addition & 1 deletion ...ationTests/AssemblerConfigurationTests.cs → ...ationTests/AssemblerConfigurationTests.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@
using Microsoft.Extensions.Logging.Abstractions;
using Nullean.ScopedFileSystem;

namespace Elastic.Assembler.IntegrationTests;
namespace Elastic.Documentation.IntegrationTests;

public class PublicOnlyAssemblerConfigurationTests
{
Expand Down
2 changes: 1 addition & 1 deletion ...sembler.IntegrationTests/DocsSyncTests.cs → ...ntation.IntegrationTests/DocsSyncTests.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@
using OpenTelemetry;
using OpenTelemetry.Trace;

namespace Elastic.Assembler.IntegrationTests;
namespace Elastic.Documentation.IntegrationTests;

public class DocsSyncTests
{
Expand Down
2 changes: 2 additions & 0 deletions ...Elastic.Assembler.IntegrationTests.csproj → ...tic.Documentation.IntegrationTests.csproj
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,13 +12,15 @@
<ProjectReference Include="$(SolutionRoot)\aspire\aspire.csproj" />
<ProjectReference Include="$(SolutionRoot)\src\Elastic.Documentation.ServiceDefaults\Elastic.Documentation.ServiceDefaults.csproj" />
<ProjectReference Include="$(SolutionRoot)\src\api\Elastic.Documentation.Api\Elastic.Documentation.Api.csproj" />
<ProjectReference Include="$(SolutionRoot)\src\api\Elastic.Documentation.Mcp.Remote\Elastic.Documentation.Mcp.Remote.csproj" />
<ProjectReference Include="$(SolutionRoot)\src\Elastic.Markdown\Elastic.Markdown.csproj" />
</ItemGroup>
<ItemGroup>
<PackageReference Include="FakeItEasy" />
<PackageReference Include="AngleSharp" />
<PackageReference Include="Aspire.Hosting.Testing"/>
<PackageReference Include="InMemoryLogger"/>
<PackageReference Include="ModelContextProtocol" />
<PackageReference Include="OpenTelemetry" />
<PackageReference Include="OpenTelemetry.Exporter.InMemory" />
</ItemGroup>
Expand Down
2 changes: 1 addition & 1 deletion ...tegrationTests/NavigationBuildingTests.cs → ...tegrationTests/NavigationBuildingTests.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@
using Nullean.ScopedFileSystem;
using RazorSlices;

namespace Elastic.Assembler.IntegrationTests;
namespace Elastic.Documentation.IntegrationTests;

public class NavigationBuildingTests(DocumentationFixture fixture, ITestOutputHelper output) : IAsyncLifetime
{
Expand Down
2 changes: 1 addition & 1 deletion ...r.IntegrationTests/NavigationRootTests.cs → ...n.IntegrationTests/NavigationRootTests.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@
using Nullean.ScopedFileSystem;
using RazorSlices;

namespace Elastic.Assembler.IntegrationTests;
namespace Elastic.Documentation.IntegrationTests;

public class NavigationRootTests(DocumentationFixture fixture, ITestOutputHelper output) : IAsyncLifetime
{
Expand Down
2 changes: 1 addition & 1 deletion ...ionTests/Search/SearchBootstrapFixture.cs → ...ionTests/Search/SearchBootstrapFixture.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Logging;

namespace Elastic.Assembler.IntegrationTests.Search;
namespace Elastic.Documentation.IntegrationTests.Search;

[CollectionDefinition(Collection)]
public class SearchBootstrapFixture(DocumentationFixture fixture) : IAsyncLifetime
Expand Down
2 changes: 1 addition & 1 deletion ...ionTests/Search/SearchIntegrationTests.cs → ...ionTests/Search/SearchIntegrationTests.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
using AwesomeAssertions;
using Elastic.Documentation.Search;

namespace Elastic.Assembler.IntegrationTests.Search;
namespace Elastic.Documentation.IntegrationTests.Search;

/// <summary>
/// Integration tests for the search endpoint exposed through MapSearchEndpoint.
Expand Down
2 changes: 1 addition & 1 deletion ...IntegrationTests/Search/SearchTestBase.cs → ...IntegrationTests/Search/SearchTestBase.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
// Elasticsearch B.V licenses this file to you under the Apache 2.0 License.
// See the LICENSE file in the project root for more information

namespace Elastic.Assembler.IntegrationTests.Search;
namespace Elastic.Documentation.IntegrationTests.Search;

/// <summary>
/// Base class for search integration tests that handles initialization
Expand Down
2 changes: 1 addition & 1 deletion ...bler.IntegrationTests/ServeStaticTests.cs → ...tion.IntegrationTests/ServeStaticTests.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
using AwesomeAssertions;
using Elastic.Documentation.Aspire;

namespace Elastic.Assembler.IntegrationTests;
namespace Elastic.Documentation.IntegrationTests;

public class ServeStaticTests(DocumentationFixture fixture, ITestOutputHelper output) : IAsyncLifetime
{
Expand Down
2 changes: 1 addition & 1 deletion ...r.IntegrationTests/SiteNavigationTests.cs → ...n.IntegrationTests/SiteNavigationTests.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@
using Microsoft.Extensions.Logging.Abstractions;
using Nullean.ScopedFileSystem;

namespace Elastic.Assembler.IntegrationTests;
namespace Elastic.Documentation.IntegrationTests;

public class SiteNavigationTests : IAsyncLifetime
{
Expand Down
Loading
Loading