Add async enumerable benchmark support (#3124)

* Add `IAsyncEnumerable` benchmark support.

* Add validator and analyzer rules.

Author: timcassell

docs/articles/features/async-enumerable.md

@@ -0,0 +1,105 @@
+---
+uid: docs.async-enumerable
+name: IAsyncEnumerable Benchmarks
+---
+
+# IAsyncEnumerable Benchmarks
+
+Benchmark methods can return `IAsyncEnumerable<T>` (or any type that satisfies the C# `await foreach` pattern). The framework drives the iteration to completion on every benchmark invocation, the same way a `await foreach` consumer would, and the elapsed time covers everything from the factory call through the final `MoveNextAsync` that observes end-of-stream.
+
+```csharp
+public class AsyncEnumerableBenchmarks
+{
+    [Benchmark]
+    public async IAsyncEnumerable<int> Produce()
+    {
+        for (int i = 0; i < 1000; i++)
+        {
+            await Task.Yield();
+            yield return i;
+        }
+    }
+}
+```
+
+## What gets measured
+
+For each invocation, the framework:
+
+1. Calls the workload method to obtain the enumerable (compiler-generated state machines do no work here — the body runs lazily).
+2. Calls `GetAsyncEnumerator()` and drives `MoveNextAsync()` / `Current` until the stream ends.
+
+Disposal (`DisposeAsync`) is invoked when the enumerator implements `IAsyncDisposable` or exposes a public `DisposeAsync()` matching the C# pattern, mirroring `await foreach` semantics.
+
+## Custom enumerable types
+
+`IsAsyncEnumerable` detection mirrors the C# `await foreach` resolution rules: it accepts any type that **is** `IAsyncEnumerable<T>`, has a public `GetAsyncEnumerator` method matching the pattern (with all parameters optional, including `CancellationToken`), or implements the interface explicitly. The element type comes from `Current` so it tracks what the compiler binds to — even when a type defines a public pattern method whose `Current` differs from the explicitly-implemented `IAsyncEnumerable<U>.GetAsyncEnumerator`'s `U`. This means hand-written enumerables, ref-struct enumerables, and `ConfiguredCancelableAsyncEnumerable<T>` all work without special configuration.
+
+## Cancellation
+
+The framework does not implicitly call `.WithCancellation(token)` on the returned enumerable. Two reasons:
+
+1. It would make the no-cancellation path unmeasurable.
+2. For types that implement `IAsyncEnumerable<T>` *and* define a public pattern `GetAsyncEnumerator`, `WithCancellation` flips dispatch from the pattern to the interface — silently changing what gets measured.
+
+You have two opt-in routes:
+
+**Class-level field** — works for any benchmark, no return-type changes needed. The iterator can read the token directly:
+
+```csharp
+public class CancellableProducer
+{
+    [BenchmarkCancellation]
+    public CancellationToken CancellationToken;
+
+    [Benchmark]
+    public async IAsyncEnumerable<int> Produce()
+    {
+        for (int i = 0; i < 1000; i++)
+        {
+            CancellationToken.ThrowIfCancellationRequested();
+            await Task.Yield();
+            yield return i;
+        }
+    }
+}
+```
+
+**`[EnumeratorCancellation]` propagation** — return `ConfiguredCancelableAsyncEnumerable<T>` directly from the benchmark to opt the iterator into the C#-idiomatic pattern. The framework's `await foreach` over the returned configured enumerable will propagate the token to a parameter marked `[EnumeratorCancellation]` inside your iterator:
+
+```csharp
+public class EnumeratorCancellationProducer
+{
+    [BenchmarkCancellation]
+    public CancellationToken CancellationToken;
+
+    [Benchmark]
+    public ConfiguredCancelableAsyncEnumerable<int> Produce()
+        => Inner().WithCancellation(CancellationToken);
+
+    private static async IAsyncEnumerable<int> Inner([EnumeratorCancellation] CancellationToken ct = default)
+    {
+        for (int i = 0; i < 1000; i++)
+        {
+            ct.ThrowIfCancellationRequested();
+            await Task.Yield();
+            yield return i;
+        }
+    }
+}
+```
+
+## Setup and cleanup
+
+`[GlobalSetup]`, `[GlobalCleanup]`, `[IterationSetup]`, and `[IterationCleanup]` methods may **not** return `IAsyncEnumerable<T>`. The framework only awaits awaitables in those positions; it would call the iterator factory and discard the enumerable without ever running the body. A mandatory validator rejects this at startup with a clear error — return `void`, `Task`, `ValueTask`, or any awaitable type instead.
+
+## DisassemblyDiagnoser
+
+`[DisassemblyDiagnoser]` walks the benchmark's call graph from a synthetic entry method that just invokes the workload and discards the result. For `IAsyncEnumerable<T>` benchmarks that means the iteration body — `GetAsyncEnumerator`, `MoveNextAsync`, `Current` — is never reached from the entry, regardless of whether the return type is the interface or a custom struct/sealed type. Compiler-generated async iterators only kick off their state machine on the first `MoveNextAsync`, so the disassembler sees nothing of substance from the workload call alone.
+
+To inspect the iterator body, use the diagnoser's `filters` parameter:
+
+```csharp
+[DisassemblyDiagnoser(filters: ["*MoveNextAsync*"])]
+public class MyBenchmarks { /* ... */ }
+```

docs/articles/features/toc.yml

@@ -4,6 +4,8 @@
   href: baselines.md
 - name: Setup And Cleanup
   href: setup-and-cleanup.md
+- name: IAsyncEnumerable Benchmarks
+  href: async-enumerable.md
 - name: Statistics
   href: statistics.md
 - name: Disassembler

src/BenchmarkDotNet.Analyzers/AnalyzerReleases.Unshipped.md

@@ -9,6 +9,8 @@ BDN1602  |  Usage   | Error    | Properties annotated with [BenchmarkCancellatio
 BDN1603  |  Usage   | Error    | [BenchmarkCancellation] attribute is not valid on readonly fields
 BDN1604  |  Usage   | Error    | Properties annotated with [BenchmarkCancellation] must have a public setter
 BDN1605  |  Usage   | Info     | Async benchmarks should have a [BenchmarkCancellation] property for cancellation support
+BDN1700  |  Usage   | Error    | [GlobalSetup]/[GlobalCleanup]/[IterationSetup]/[IterationCleanup] method must not return an async enumerable
+BDN1701  |  Usage   | Warning  | Benchmark/setup/cleanup return type is both awaitable and an async enumerable; the iterator is never enumerated
 
 
 ### Removed Rules

src/BenchmarkDotNet.Analyzers/AsyncTypeShapes.cs

@@ -0,0 +1,116 @@
+using Microsoft.CodeAnalysis;
+
+namespace BenchmarkDotNet.Analyzers;
+
+/// <summary>
+/// Static-analysis counterparts to <c>ReflectionExtensions.IsAsyncEnumerable</c> and
+/// <c>ReflectionExtensions.IsAwaitable</c> on the runtime side. Used by analyzers that need to mirror
+/// the framework's `await foreach` / `await` binding shape rules at compile time.
+/// </summary>
+internal static class AsyncTypeShapes
+{
+    /// <summary>
+    /// Returns true when <paramref name="type"/> would bind as an async enumerable under the C# compiler's
+    /// `await foreach` rules: exact <c>IAsyncEnumerable&lt;T&gt;</c> short-circuit → public-instance
+    /// <c>GetAsyncEnumerator</c> pattern with all-optional parameters whose return has public-instance
+    /// <c>MoveNextAsync</c> (all-optional params) and a public <c>Current</c> property → interface fallback
+    /// via <see cref="ITypeSymbol.AllInterfaces"/>.
+    /// </summary>
+    public static bool IsAsyncEnumerable(ITypeSymbol type, INamedTypeSymbol? asyncEnumerableInterfaceSymbol)
+    {
+        if (asyncEnumerableInterfaceSymbol != null
+            && type is INamedTypeSymbol named
+            && SymbolEqualityComparer.Default.Equals(named.OriginalDefinition, asyncEnumerableInterfaceSymbol))
+        {
+            return true;
+        }
+
+        if (TryFindPatternGetAsyncEnumerator(type) is { } enumeratorType
+            && HasPatternMoveNextAsync(enumeratorType)
+            && HasPublicInstanceProperty(enumeratorType, "Current"))
+        {
+            return true;
+        }
+
+        if (asyncEnumerableInterfaceSymbol != null)
+        {
+            foreach (var implemented in type.AllInterfaces)
+            {
+                if (SymbolEqualityComparer.Default.Equals(implemented.OriginalDefinition, asyncEnumerableInterfaceSymbol))
+                {
+                    return true;
+                }
+            }
+        }
+
+        return false;
+    }
+
+    /// <summary>
+    /// Returns true when <paramref name="type"/> exposes a public parameterless <c>GetAwaiter</c> method —
+    /// the necessary precondition for the C# compiler's <c>await</c> binding. The analyzer doesn't drill
+    /// into the awaiter's <c>IsCompleted</c>/<c>GetResult</c>/<c>OnCompleted</c> shape; the framework's
+    /// runtime <c>IsAwaitable</c> check does that more thoroughly when needed.
+    /// </summary>
+    public static bool IsAwaitable(ITypeSymbol type)
+    {
+        foreach (var member in type.GetMembers("GetAwaiter"))
+        {
+            if (member is IMethodSymbol { DeclaredAccessibility: Accessibility.Public, IsStatic: false, Parameters.Length: 0 })
+            {
+                return true;
+            }
+        }
+        return false;
+    }
+
+    private static ITypeSymbol? TryFindPatternGetAsyncEnumerator(ITypeSymbol type)
+    {
+        foreach (var member in type.GetMembers("GetAsyncEnumerator"))
+        {
+            if (member is IMethodSymbol { DeclaredAccessibility: Accessibility.Public, IsStatic: false } method
+                && AllParametersOptional(method))
+            {
+                return method.ReturnType;
+            }
+        }
+        return null;
+    }
+
+    private static bool HasPatternMoveNextAsync(ITypeSymbol enumeratorType)
+    {
+        foreach (var member in enumeratorType.GetMembers("MoveNextAsync"))
+        {
+            if (member is IMethodSymbol { DeclaredAccessibility: Accessibility.Public, IsStatic: false } method
+                && AllParametersOptional(method))
+            {
+                return true;
+            }
+        }
+        return false;
+    }
+
+    private static bool HasPublicInstanceProperty(ITypeSymbol type, string name)
+    {
+        foreach (var member in type.GetMembers(name))
+        {
+            if (member is IPropertySymbol { DeclaredAccessibility: Accessibility.Public, IsStatic: false })
+            {
+                return true;
+            }
+        }
+        return false;
+    }
+
+    private static bool AllParametersOptional(IMethodSymbol method)
+    {
+        foreach (var parameter in method.Parameters)
+        {
+            if (!parameter.IsOptional)
+            {
+                return false;
+            }
+        }
+        return true;
+    }
+}

src/BenchmarkDotNet.Analyzers/Attributes/SetupCleanupAsyncEnumerableAnalyzer.cs

@@ -0,0 +1,135 @@
+using Microsoft.CodeAnalysis;
+using Microsoft.CodeAnalysis.CSharp;
+using Microsoft.CodeAnalysis.CSharp.Syntax;
+using Microsoft.CodeAnalysis.Diagnostics;
+using System.Collections.Immutable;
+
+namespace BenchmarkDotNet.Analyzers.Attributes;
+
+/// <summary>
+/// Static counterpart to <c>SetupCleanupValidator.ValidateReturnType</c>: a method annotated with
+/// <c>[GlobalSetup]</c>, <c>[GlobalCleanup]</c>, <c>[IterationSetup]</c>, or <c>[IterationCleanup]</c>
+/// must not return an async enumerable. BenchmarkDotNet awaits awaitable returns from those methods but
+/// does not enumerate async enumerables, so the iterator body would silently never run.
+/// </summary>
+[DiagnosticAnalyzer(LanguageNames.CSharp)]
+public class SetupCleanupAsyncEnumerableAnalyzer : DiagnosticAnalyzer
+{
+    internal static readonly DiagnosticDescriptor MustNotReturnAsyncEnumerableRule = new(
+        DiagnosticIds.Attributes_SetupCleanup_MustNotReturnAsyncEnumerable,
+        AnalyzerHelper.GetResourceString(nameof(BenchmarkDotNetAnalyzerResources.Attributes_SetupCleanup_MustNotReturnAsyncEnumerable_Title)),
+        AnalyzerHelper.GetResourceString(nameof(BenchmarkDotNetAnalyzerResources.Attributes_SetupCleanup_MustNotReturnAsyncEnumerable_MessageFormat)),
+        "Usage",
+        DiagnosticSeverity.Error,
+        isEnabledByDefault: true,
+        description: AnalyzerHelper.GetResourceString(nameof(BenchmarkDotNetAnalyzerResources.Attributes_SetupCleanup_MustNotReturnAsyncEnumerable_Description)));
+
+    public override ImmutableArray<DiagnosticDescriptor> SupportedDiagnostics => new DiagnosticDescriptor[]
+    {
+        MustNotReturnAsyncEnumerableRule,
+    }.ToImmutableArray();
+
+    private static readonly string[] SetupCleanupAttributeMetadataNames =
+    [
+        "BenchmarkDotNet.Attributes.GlobalSetupAttribute",
+        "BenchmarkDotNet.Attributes.GlobalCleanupAttribute",
+        "BenchmarkDotNet.Attributes.IterationSetupAttribute",
+        "BenchmarkDotNet.Attributes.IterationCleanupAttribute",
+    ];
+
+    public override void Initialize(AnalysisContext analysisContext)
+    {
+        analysisContext.EnableConcurrentExecution();
+        analysisContext.ConfigureGeneratedCodeAnalysis(GeneratedCodeAnalysisFlags.None);
+
+        analysisContext.RegisterCompilationStartAction(ctx =>
+        {
+            // Only run if BenchmarkDotNet.Annotations is referenced.
+            if (AnalyzerHelper.GetBenchmarkAttributeTypeSymbol(ctx.Compilation) == null)
+            {
+                return;
+            }
+
+            var attributeSymbols = ImmutableArray.CreateBuilder<INamedTypeSymbol>(SetupCleanupAttributeMetadataNames.Length);
+            foreach (var metadataName in SetupCleanupAttributeMetadataNames)
+            {
+                var symbol = ctx.Compilation.GetTypeByMetadataName(metadataName);
+                if (symbol != null)
+                {
+                    attributeSymbols.Add(symbol);
+                }
+            }
+
+            if (attributeSymbols.Count == 0)
+            {
+                return;
+            }
+
+            var asyncEnumerableInterfaceSymbol = ctx.Compilation.GetTypeByMetadataName("System.Collections.Generic.IAsyncEnumerable`1");
+            var captured = (attributeSymbols.ToImmutable(), asyncEnumerableInterfaceSymbol);
+            ctx.RegisterSyntaxNodeAction(c => AnalyzeMethod(c, captured), SyntaxKind.MethodDeclaration);
+        });
+    }
+
+    private static void AnalyzeMethod(
+        SyntaxNodeAnalysisContext context,
+        (ImmutableArray<INamedTypeSymbol> AttributeSymbols, INamedTypeSymbol? AsyncEnumerableInterfaceSymbol) captured)
+    {
+        if (context.Node is not MethodDeclarationSyntax methodDeclarationSyntax)
+        {
+            return;
+        }
+
+        if (context.SemanticModel.GetDeclaredSymbol(methodDeclarationSyntax) is not IMethodSymbol methodSymbol)
+        {
+            return;
+        }
+
+        // Find which (if any) setup/cleanup attribute is applied.
+        INamedTypeSymbol? matchedAttribute = null;
+        foreach (var attributeData in methodSymbol.GetAttributes())
+        {
+            foreach (var candidate in captured.AttributeSymbols)
+            {
+                if (SymbolEqualityComparer.Default.Equals(attributeData.AttributeClass, candidate))
+                {
+                    matchedAttribute = candidate;
+                    break;
+                }
+            }
+            if (matchedAttribute != null) break;
+        }
+
+        if (matchedAttribute == null)
+        {
+            return;
+        }
+
+        // Mirrors ReflectionExtensions.IsAsyncEnumerable: exact IAsyncEnumerable<T> short-circuit, then
+        // public-instance GetAsyncEnumerator pattern, then interface fallback. The runtime validator
+        // also explicitly skips awaitable types — if the return type happens to be both awaitable AND
+        // an async enumerable, BenchmarkDotNet awaits it instead of rejecting it (BDN1701 covers that
+        // ambiguity separately).
+        var returnType = methodSymbol.ReturnType;
+        if (!AsyncTypeShapes.IsAsyncEnumerable(returnType, captured.AsyncEnumerableInterfaceSymbol))
+        {
+            return;
+        }
+
+        if (AsyncTypeShapes.IsAwaitable(returnType))
+        {
+            return;
+        }
+
+        var attributeShortName = matchedAttribute.Name.EndsWith("Attribute")
+            ? matchedAttribute.Name.Substring(0, matchedAttribute.Name.Length - "Attribute".Length)
+            : matchedAttribute.Name;
+
+        context.ReportDiagnostic(Diagnostic.Create(
+            MustNotReturnAsyncEnumerableRule,
+            methodDeclarationSyntax.ReturnType.GetLocation(),
+            attributeShortName,
+            methodSymbol.Name));
+    }
+
+}