Merge branch 'main' into jupinzer/promote_container_detectors_default

Author: jpinz

README.md

@@ -33,21 +33,34 @@ Component Detection can also be used as a library to detect dependencies in your
 
 ## Features
 
-Component Detection supports detecting libraries from the following ecosystem:
+Component Detection supports detecting libraries from the following ecosystems:
 
 | Ecosystem                                                                        | Scanning                                        | Graph Creation |
 | -------------------------------------------------------------------------------- | ----------------------------------------------- | -------------- |
-| CocoaPods                                                                        | ✔                                               | ✔              |
-| [Go](docs/detectors/go.md)                                                       | ✔                                               | ❌              |
+| [CocoaPods](docs/detectors/cocoapods.md)                                         | ✔                                               | ✔              |
+| [Conan](docs/detectors/conan.md)                                                 | ✔                                               | ❌              |
+| [Conda (Python)](docs/detectors/conda.md)                                        | ✔                                               | ✔              |
+| [Docker Compose](docs/detectors/dockercompose.md)                                | ✔                                               | ❌              |
+| [Dockerfile](docs/detectors/dockerfile.md)                                       | ✔                                               | ❌              |
+| [DotNet SDK](docs/detectors/dotnet.md)                                           | ✔                                               | ❌              |
+| [Go](docs/detectors/go.md)                                                       | ✔                                               | ✔ (with Go 1.11+) |
 | [Gradle (lockfiles only)](docs/detectors/gradle.md)                              | ✔                                               | ❌              |
-| [Linux (Debian, Alpine, Rhel, Centos, Fedora, Ubuntu)](docs/detectors//linux.md) | ✔ (via [syft](https://github.com/anchore/syft)) | ❌              |
+| [Helm](docs/detectors/helm.md)                                                   | ✔                                               | ❌              |
+| [Ivy](docs/detectors/ivy.md)                                                     | ✔                                               | ✔              |
+| [Linux (Debian, Alpine, Rhel, Centos, Fedora, Ubuntu)](docs/detectors/linux.md)  | ✔ (via [syft](https://github.com/anchore/syft)) | ❌              |
 | [Maven](docs/detectors/maven.md)                                                 | ✔                                               | ✔              |
 | [NPM (including Yarn, Pnpm)](docs/detectors/npm.md)                              | ✔                                               | ✔              |
 | [NuGet (including Paket)](docs/detectors/nuget.md)                               | ✔                                               | ✔              |
 | [Pip (Python)](docs/detectors/pip.md)                                            | ✔                                               | ✔              |
 | [Poetry (Python, lockfiles only)](docs/detectors/poetry.md)                      | ✔                                               | ❌              |
-| Ruby                                                                             | ✔                                               | ✔              |
-| Rust                                                                             | ✔                                               | ✔              |
+| [Ruby](docs/detectors/ruby.md)                                                   | ✔                                               | ✔              |
+| [Rust (Cargo)](docs/detectors/rust.md)                                           | ✔                                               | ✔              |
+| [SPDX SBOM](docs/detectors/spdx.md)                                              | ✔                                               | ❌              |
+| [Swift](docs/detectors/swift.md)                                                 | ✔                                               | ❌              |
+| [Uv (Python)](docs/detectors/uv.md)                                              | ✔                                               | ✔              |
+| [Vcpkg](docs/detectors/vcpkg.md)                                                 | ✔                                               | ❌              |
+
+See the [detectors directory](docs/detectors/README.md) for the current status (Stable, Experimental, or DefaultOff) of each individual detector.
 
 For a complete feature overview refer to [feature-overview.md](docs/feature-overview.md)
 

docs/detectors/README.md

@@ -87,7 +87,7 @@
 | NuGetComponentDetector                           | Stable     |
 | NuGetPackagesConfigDetector                      | Stable     |
 | NuGetProjectModelProjectCentricComponentDetector | Stable     |
-| MSBuildBinaryLogComponentDetector                | DefaultOff |
+| MSBuildBinaryLogComponentDetector                | Experimental |
 
 - [Pip](pip.md)
 

docs/detectors/nuget.md

@@ -38,7 +38,9 @@ The `NuGetPackagesConfig` detector raises NuGet components referenced by project
 
 ## MSBuildBinaryLog
 
-The `MSBuildBinaryLog` detector is a **DefaultOff** detector intended to eventually replace both the `NuGetProjectCentric` and `DotNet` detectors. It combines MSBuild binary log (binlog) information with `project.assets.json` to provide enhanced component detection with project-level classifications.
+The `MSBuildBinaryLog` detector is an **Experimental** detector intended to eventually replace both the `NuGetProjectCentric` and `DotNet` detectors. It combines MSBuild binary log (binlog) information with `project.assets.json` to provide enhanced component detection with project-level classifications.
+
+As an experimental detector, it runs automatically whenever a scan is performed, but its results are not reported as part of the normal scan output. Instead, the results are compared against the existing `NuGetProjectCentric` and `DotNet` detectors and recorded as telemetry so maintainers can evaluate parity before promoting the detector to default.
 
 It looks for `project.assets.json` files and separately discovers `*.binlog` files. The binlog provides build-time context that isn't available from `project.assets.json` alone.
 

docs/feature-overview.md

@@ -11,7 +11,7 @@
 | NPM | <ul><li>package.json</li><li>package-lock.json</li><li>npm-shrinkwrap.json</li><li>lerna.json</li></ul> | - | ✔ (dev-dependencies in package.json, dev flag in package-lock.json) | ✔ |
 | Yarn (v1, v2) | <ul><li>package.json</li><li>yarn.lock</li></ul> | - | ✔ (dev-dependencies in package.json) | ✔ |
 | Pnpm | <ul><li>shrinkwrap.yaml</li><li>pnpm-lock.yaml</li></ul> | - | ✔ (packages/{package}/dev flag) | ✔ |
-| NuGet | <ul><li>project.assets.json</li><li>*.nupkg</li><li>*.nuspec</li><li>packages.config</li><li>nuget.config</li><li>*.binlog (DefaultOff)</li></ul> | - | - | ✔ (required project.assets.json) |
+| NuGet | <ul><li>project.assets.json</li><li>*.nupkg</li><li>*.nuspec</li><li>packages.config</li><li>nuget.config</li><li>*.binlog (Experimental)</li></ul> | - | - | ✔ (required project.assets.json) |
 | Pip (Python) | <ul><li>setup.py</li><li>requirements.txt</li><li>*setup=distutils.core.run_setup({setup.py}); setup.install_requires*</li><li>dist package METADATA file</li></ul> | <ul><li>Python 2 or Python 3</li><li>Internet connection</li></ul> | ❌ | ✔ |
 | Poetry (Python) | <ul><li>poetry.lock</li><ul> | - | ✔ | ❌ |
 | Ruby | <ul><li>gemfile.lock</li></ul> | - | ❌ | ✔ |

src/Microsoft.ComponentDetection.Common/DockerReference/DockerReferenceUtility.cs

@@ -27,7 +27,10 @@
 namespace Microsoft.ComponentDetection.Common;
 
 using System;
+using System.Buffers;
+using System.Collections.Generic;
 using System.Diagnostics.CodeAnalysis;
+using System.Text.RegularExpressions;
 using Microsoft.ComponentDetection.Contracts;
 using Microsoft.Extensions.Logging;
 
@@ -39,18 +42,55 @@ public static class DockerReferenceUtility
     private const string LEGACYDEFAULTDOMAIN = "index.docker.io";
     private const string OFFICIALREPOSITORYNAME = "library";
 
+    // Delimiters that only appear in an image reference as part of an unresolved templating
+    // token: '$', '{' and '}' cover shell / Helm / Go-template placeholders (e.g. ${VAR},
+    // {{ .Values.tag }}). These are recognized templating syntaxes expected in un-rendered manifests,
+    // so TryParseImageReference skips them (logging a warning) rather than treating them as invalid.
+    // A token wrapped in matching '#' or '!' (handled by DelimiterWrappedTokenRegex) is treated the same way.
+    // When no templating token is present, stray invalid characters (e.g. a single '#' or '!') are reported
+    // via GetInvalidReferenceCharacters.
+    private static readonly char[] TemplateDelimiters = ['$', '{', '}'];
+
+    // Matches token-replacement placeholders that wrap an identifier in double underscores,
+    // e.g. __IMAGE_TAG__ or __MCR_ENDPOINT__. Without this they parse as an uppercase repository
+    // name and surface as a noisy parse failure instead of being skipped as a templated value.
+    private static readonly Regex DoubleUnderscoreTokenRegex = new(@"__\w+__");
+
+    // Matches token-replacement placeholders wrapped in a matching '#' or '!', e.g. #imageTag#,
+    // #cs_containerRegistryLoginServerUrl#, or !imageTag!. A string surrounded by the same '#' or
+    // '!' delimiter is almost always an unsubstituted template variable (Azure DevOps token
+    // replacement and similar), so it is skipped (and may be logged as a warning) instead of
+    // surfacing as a misleading docker-reference parse failure. The backreference requires the closing delimiter to match
+    // the opening one, so a mismatched stray '#' or '!' is left to GetInvalidReferenceCharacters.
+    private static readonly Regex DelimiterWrappedTokenRegex = new(@"([#!])[^#!]+\1");
+
+    // Every character permitted anywhere in a docker reference per the grammar at the top of this
+    // file: alphanumerics, the separators '.', '_' and '-', the path separator '/', the tag/port
+    // and digest separators ':' and '@', and the digest-algorithm separator '+'. Anything else
+    // (e.g. '#', '!') comes from unsubstituted template tokens and is reported as invalid.
+    private static readonly SearchValues<char> ValidReferenceChars = SearchValues.Create(
+        "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789._-/:@+");
+
     /// <summary>
-    /// Returns true if the reference contains unresolved variable placeholders (e.g., ${VAR}, {{ .Values.tag }}).
-    /// Such references should be skipped before calling <see cref="ParseFamiliarName"/> or <see cref="ParseQualifiedName"/>.
+    /// Returns true if the reference contains unresolved variable or templating placeholders,
+    /// e.g. <c>${VAR}</c>, <c>{{ .Values.tag }}</c>, <c>__IMAGE_TAG__</c>, <c>#imageTag#</c>, or
+    /// <c>!imageTag!</c>.
+    /// Such references are not real, resolvable images, so they should be skipped before calling
+    /// <see cref="ParseFamiliarName"/> or <see cref="ParseQualifiedName"/> and treated as
+    /// unresolved values rather than reported as parse failures.
     /// </summary>
     /// <param name="reference">The image reference string to check.</param>
     /// <returns><c>true</c> if the reference contains variable placeholder characters; otherwise <c>false</c>.</returns>
     public static bool HasUnresolvedVariables(string reference) =>
-        reference.IndexOfAny(['$', '{', '}']) >= 0;
+        reference.IndexOfAny(TemplateDelimiters) >= 0 ||
+        DoubleUnderscoreTokenRegex.IsMatch(reference) ||
+        DelimiterWrappedTokenRegex.IsMatch(reference);
 
     /// <summary>
     /// Attempts to parse an image reference string into a <see cref="DockerReference"/>.
-    /// Returns <c>null</c> if the reference contains unresolved variables or cannot be parsed.
+    /// Returns <c>null</c> if the reference contains unresolved variables, contains characters that
+    /// are not valid in a docker reference, or otherwise cannot be parsed. A warning is logged in
+    /// every skip/failure case so that references which are not scanned remain visible in logs.
     /// </summary>
     /// <param name="imageReference">The image reference string to parse.</param>
     /// <param name="logger">Optional logger for recording parse failures.</param>
@@ -59,6 +99,19 @@ public static bool HasUnresolvedVariables(string reference) =>
     {
         if (HasUnresolvedVariables(imageReference))
         {
+            logger?.LogWarning(
+                "Skipping image reference '{ImageReference}' because it contains one or more unresolved template tokens or variable placeholders.",
+                imageReference);
+            return null;
+        }
+
+        var invalidCharacters = GetInvalidReferenceCharacters(imageReference);
+        if (invalidCharacters.Length > 0)
+        {
+            logger?.LogWarning(
+                "Skipping image reference '{ImageReference}' because it contains character(s) that are not valid in a docker reference: {InvalidCharacters}",
+                imageReference,
+                invalidCharacters);
             return null;
         }
 
@@ -76,7 +129,7 @@ public static bool HasUnresolvedVariables(string reference) =>
     /// <summary>
     /// Parses an image reference and registers it with the recorder if valid.
     /// Skips references with unresolved variables or that cannot be parsed,
-    /// logging a warning for parse failures so that remaining entries continue to be processed.
+    /// logging a warning in each skipped case so that remaining entries continue to be processed.
     /// </summary>
     /// <param name="imageReference">The image reference string to parse.</param>
     /// <param name="recorder">The component recorder to register the image with.</param>
@@ -228,6 +281,44 @@ public static DockerReference ParseAll(string name)
         return ParseFamiliarName(name);
     }
 
+    /// <summary>
+    /// Returns the distinct characters in <paramref name="reference"/> that are not valid in any
+    /// part of a docker reference (domain, repository, tag, or digest) as a comma-separated string,
+    /// or an empty string when every character is valid. Characters such as <c>#</c> and <c>!</c>
+    /// commonly appear in unsubstituted template tokens and otherwise surface as misleading
+    /// "must be lowercase" or "invalid reference format" parse errors.
+    /// </summary>
+    /// <param name="reference">The image reference string to inspect.</param>
+    /// <returns>A comma-separated list of invalid characters, or an empty string if there are none.</returns>
+    private static string GetInvalidReferenceCharacters(string reference)
+    {
+        // Vectorized happy-path check: the overwhelmingly common case is an all-valid reference,
+        // for which this returns without allocating. Only gather the offending characters when
+        // at least one is present.
+        var span = reference.AsSpan();
+        if (!span.ContainsAnyExcept(ValidReferenceChars))
+        {
+            return string.Empty;
+        }
+
+        SortedSet<char> invalid = [];
+        foreach (var c in span)
+        {
+            if (!ValidReferenceChars.Contains(c))
+            {
+                invalid.Add(c);
+            }
+        }
+
+        var invalidStrings = new List<string>(invalid.Count);
+        foreach (var c in invalid)
+        {
+            invalidStrings.Add($"'{c}'");
+        }
+
+        return string.Join(", ", invalidStrings);
+    }
+
     private static DockerReference CreateDockerReference(Reference options)
     {
         return DockerReference.CreateDockerReference(options.Repository, options.Domain, options.Digest, options.Tag);