graph

Author: KSemenenko

AGENTS.md

@@ -148,9 +148,12 @@ If no new rule is detected -> do not update the file.
 - Implement code and tests together for every behavior change.
 - Keep the gateway reusable as a NuGet library, not as an app-specific host.
 - Preserve one public execution surface for local `AITool` instances and MCP tools.
-- Preserve one searchable catalog that supports vector ranking when embeddings are available and lexical fallback when they are not.
+- Preserve one searchable catalog that uses Markdown-LD graph ranking by default and supports vector ranking only when embeddings are explicitly selected.
+- Tool search must support sparse high-confidence selection plus an explicit related/next-step expansion path; do not make consumers pass the full tool catalog when a smaller capability set can answer the request.
 - For multilingual or noisy search inputs, prefer a generic English-normalization step before ranking when an AI/query-rewrite component is available, because the user wants the searchable representation to converge to English instead of relying only on language-specific token overlap.
 - Keep meta-tools available through `McpGatewayToolSet` and `IMcpGateway.CreateMetaTools(...)`.
+- When Markdown-LD graph search is selected, startup or explicit index initialization must build and validate the tool graph before search/tool discovery so LLM-facing MCP tool selection is based on the correct focused graph.
+- Markdown-LD graph search must support both startup-generated graphs and filesystem-provided graph files; tests for file-backed graph mode must generate the graph fixture through the package flow rather than relying on a hand-authored static artifact.
 - If a user adds or corrects a persistent workflow rule, update `AGENTS.md` first and only then continue with the task.
 
 ### Repository Layout
@@ -209,7 +212,7 @@ If no new rule is detected -> do not update the file.
   - local tool indexing and invocation
   - MCP tool indexing and invocation
   - vector search behavior
-  - lexical fallback behavior
+  - Markdown-LD graph search and vector-to-graph fallback behavior
 - Keep embedding-based search covered with deterministic local tests by using a fake or test-only embedding generator.
 - Keep request context behavior covered when search or invocation consumes contextual inputs.
 - Do not remove tests to get green builds.
@@ -252,7 +255,8 @@ If no new rule is detected -> do not update the file.
 - Prefer direct generic DI registrations such as `services.TryAddSingleton<IService, Implementation>()` over lambda alias registrations when wiring package services, because the lambda style has already been called out as unreadable and error-prone in this repository.
 - Keep runtime services DI-native from their public/internal constructors; types such as `McpGatewayRegistry` must be creatable through `IOptions<McpGatewayOptions>` and other DI-managed dependencies rather than ad-hoc state-only constructors, because the package design requires services to live fully inside the container.
 - When emitting package identity to external protocols such as MCP client info, never hardcode a fake version string; use the actual assembly/build version so runtime metadata stays aligned with the package being shipped.
-- For search-quality improvements, prefer mathematical or statistical ranking changes over hardcoded phrase lists or ad-hoc query text hacks, because the user explicitly wants tokenizer search to improve through general scoring behavior rather than manual exceptions.
+- For search-quality improvements, prefer mathematical, statistical, or graph-ranking changes over hardcoded phrase lists or ad-hoc query text hacks, because the user explicitly wants token-distance search to improve through general scoring behavior rather than manual exceptions.
+- Do not keep a separate local tokenizer search path when `ManagedCode.MarkdownLd.Kb` already provides token-based graph search; route tokenizer-backed retrieval through Markdown-LD so the package does not carry duplicate ranking implementations.
 - Prefer framework-provided in-memory caching primitives such as `IMemoryCache` over custom process-local storage implementations when they cover the lifecycle and lookup needs, because self-rolled memory stores age poorly and make scaling/concurrency behavior harder to trust.
 - Never keep legacy compatibility shims, obsolete paths, or lingering documentation references to removed implementations when a replacement is accepted, because this repository should converge on the current design instead of carrying dead historical baggage.
 - Never leave `ManagedCode`-prefixed DI/setup extension method names such as `AddManagedCodeMcpGateway(...)` in the public API once concise `McpGateway` naming is available, because these branded leftovers make the package surface inconsistent and read like stale legacy.

Directory.Build.props

@@ -11,7 +11,7 @@
     <AnalysisLevel>latest-recommended</AnalysisLevel>
     <TreatWarningsAsErrors>true</TreatWarningsAsErrors>
     <NoWarn>$(NoWarn);CS1591;CA1707;CA1848;CA1859;CA1873</NoWarn>
-    <Version>0.3.1</Version>
+    <Version>0.3.2</Version>
     <PackageVersion>$(Version)</PackageVersion>
   </PropertyGroup>
 

Directory.Packages.props

@@ -4,20 +4,19 @@
   </PropertyGroup>
   <ItemGroup>
     <PackageVersion Include="DotNet.ReproducibleBuilds" Version="2.0.2" />
-    <PackageVersion Include="Microsoft.Agents.AI" Version="1.0.0-rc3" />
-    <PackageVersion Include="Microsoft.Extensions.AI" Version="10.3.0" />
-    <PackageVersion Include="Microsoft.Extensions.Caching.Memory" Version="10.0.3" />
-    <PackageVersion Include="Microsoft.Extensions.DependencyInjection" Version="10.0.3" />
-    <PackageVersion Include="Microsoft.Extensions.DependencyInjection.Abstractions" Version="10.0.3" />
-    <PackageVersion Include="Microsoft.Extensions.Hosting.Abstractions" Version="10.0.3" />
-    <PackageVersion Include="Microsoft.Extensions.Logging" Version="10.0.3" />
-    <PackageVersion Include="Microsoft.Extensions.Logging.Abstractions" Version="10.0.3" />
-    <PackageVersion Include="Microsoft.ML.Tokenizers" Version="2.0.0" />
-    <PackageVersion Include="Microsoft.ML.Tokenizers.Data.O200kBase" Version="2.0.0" />
-    <PackageVersion Include="Microsoft.Extensions.Options" Version="10.0.3" />
-    <PackageVersion Include="Microsoft.NET.Test.Sdk" Version="18.3.0" />
-    <PackageVersion Include="Microsoft.SourceLink.GitHub" Version="10.0.103" />
-    <PackageVersion Include="ModelContextProtocol" Version="1.1.0" />
-    <PackageVersion Include="TUnit" Version="1.19.0" />
+    <PackageVersion Include="ManagedCode.MarkdownLd.Kb" Version="0.1.1" />
+    <PackageVersion Include="Microsoft.Agents.AI" Version="1.1.0" />
+    <PackageVersion Include="Microsoft.Extensions.AI" Version="10.5.0" />
+    <PackageVersion Include="Microsoft.Extensions.Caching.Memory" Version="10.0.6" />
+    <PackageVersion Include="Microsoft.Extensions.DependencyInjection" Version="10.0.6" />
+    <PackageVersion Include="Microsoft.Extensions.DependencyInjection.Abstractions" Version="10.0.6" />
+    <PackageVersion Include="Microsoft.Extensions.Hosting.Abstractions" Version="10.0.6" />
+    <PackageVersion Include="Microsoft.Extensions.Logging" Version="10.0.6" />
+    <PackageVersion Include="Microsoft.Extensions.Logging.Abstractions" Version="10.0.6" />
+    <PackageVersion Include="Microsoft.Extensions.Options" Version="10.0.6" />
+    <PackageVersion Include="Microsoft.NET.Test.Sdk" Version="18.4.0" />
+    <PackageVersion Include="Microsoft.SourceLink.GitHub" Version="10.0.202" />
+    <PackageVersion Include="ModelContextProtocol" Version="1.2.0" />
+    <PackageVersion Include="TUnit" Version="1.34.0" />
   </ItemGroup>
 </Project>

README.md

@@ -22,7 +22,7 @@ dotnet add package ManagedCode.MCPGateway
 ## What It Gives You
 
 - one gateway for local `AITool` instances and MCP tools
-- one search surface with vector ranking when embeddings are available and lexical fallback when they are not
+- one search surface with default Markdown-LD graph ranking and opt-in vector ranking
 - one invoke surface for both local tools and MCP tools
 - runtime registration through `IMcpGatewayRegistry`
 - reusable gateway meta-tools for chat clients and agents
@@ -75,8 +75,9 @@ var invoke = await gateway.InvokeAsync(new McpGatewayInvokeRequest(
 
 Important defaults:
 
-- search is `Auto` by default
-- `Auto` uses embeddings when available and lexical fallback otherwise
+- search is `Graph` by default
+- graph search uses `ManagedCode.MarkdownLd.Kb` and does not require embeddings
+- embeddings are opt-in through `McpGatewaySearchStrategy.Embeddings` or `McpGatewaySearchStrategy.Auto`
 - the default result size is `5`
 - the maximum result size is `15`
 - the index is built lazily on first list, search, or invoke
@@ -350,7 +351,7 @@ var response = await agent.RunAsync(
 
 ## Optional Warmup
 
-The gateway works without explicit initialization, but you can warm the index eagerly when you want startup validation or a pre-built cache.
+The gateway works without explicit initialization, but you can warm the index eagerly when you want startup validation or a pre-built cache. When Markdown-LD graph search is selected, warmup builds the graph during startup instead of waiting for the first search.
 
 Manual warmup:
 
@@ -393,6 +394,8 @@ services.AddKeyedSingleton<IEmbeddingGenerator<string, Embedding<float>>, MyEmbe
 
 services.AddMcpGateway(options =>
 {
+    options.SearchStrategy = McpGatewaySearchStrategy.Embeddings;
+
     options.AddTool(
         "local",
         AIFunctionFactory.Create(
@@ -405,7 +408,7 @@ services.AddMcpGateway(options =>
 });
 ```
 
-If no embedding generator is registered, the same gateway still works and falls back to lexical search automatically.
+If vector search cannot run for a request, the gateway falls back to the same Markdown-LD graph index used by the default mode and reports a diagnostic. If you register an embedding generator but leave the default `Graph` strategy in place, the generator is not used.
 
 ## Optional Query Normalization
 
@@ -420,8 +423,17 @@ services.AddKeyedSingleton<IChatClient>(
 
 services.AddMcpGateway(options =>
 {
-    options.SearchStrategy = McpGatewaySearchStrategy.Auto;
     options.SearchQueryNormalization = McpGatewaySearchQueryNormalization.TranslateToEnglishWhenAvailable;
+
+    options.AddTool(
+        "local",
+        AIFunctionFactory.Create(
+            static (string query) => $"github:{query}",
+            new AIFunctionFactoryOptions
+            {
+                Name = "github_search_repositories",
+                Description = "Search GitHub repositories by user query."
+            }));
 });
 ```
 
@@ -435,6 +447,20 @@ For process-local caching, use the built-in `IMemoryCache`-backed store:
 services.AddKeyedSingleton<IEmbeddingGenerator<string, Embedding<float>>, MyEmbeddingGenerator>(
     McpGatewayServiceKeys.EmbeddingGenerator);
 services.AddMcpGatewayInMemoryToolEmbeddingStore();
+services.AddMcpGateway(options =>
+{
+    options.SearchStrategy = McpGatewaySearchStrategy.Embeddings;
+
+    options.AddTool(
+        "local",
+        AIFunctionFactory.Create(
+            static (string query) => $"github:{query}",
+            new AIFunctionFactoryOptions
+            {
+                Name = "github_search_repositories",
+                Description = "Search GitHub repositories by user query."
+            }));
+});
 ```
 
 This built-in store reuses the application's shared `IMemoryCache` and only caches embeddings inside the current process. It is useful for local reuse, but it is not durable and does not synchronize across replicas.
@@ -447,25 +473,117 @@ For multi-instance or durable caching, register your own `IMcpGatewayToolEmbeddi
 services.AddKeyedSingleton<IEmbeddingGenerator<string, Embedding<float>>, MyEmbeddingGenerator>(
     McpGatewayServiceKeys.EmbeddingGenerator);
 services.AddSingleton<IMcpGatewayToolEmbeddingStore, MyToolEmbeddingStore>();
+services.AddMcpGateway(options =>
+{
+    options.SearchStrategy = McpGatewaySearchStrategy.Embeddings;
+
+    options.AddTool(
+        "local",
+        AIFunctionFactory.Create(
+            static (string query) => $"github:{query}",
+            new AIFunctionFactoryOptions
+            {
+                Name = "github_search_repositories",
+                Description = "Search GitHub repositories by user query."
+            }));
+});
+```
+
+## Markdown-LD Graph Sources
+
+By default the gateway generates Markdown-LD tool documents from the current local `AITool` and MCP catalog during index build:
+
+```csharp
+services.AddMcpGateway(options =>
+{
+    options.SearchStrategy = McpGatewaySearchStrategy.Graph;
+    options.UseGeneratedMarkdownLdGraph();
+
+    options.AddTool(
+        "local",
+        AIFunctionFactory.Create(
+            static (string query) => $"github:{query}",
+            new AIFunctionFactoryOptions
+            {
+                Name = "github_search_repositories",
+                Description = "Search GitHub repositories by user query."
+            }));
+});
+```
+
+You can also build the same Markdown-LD source documents ahead of time and point the gateway at a file or directory. This is useful when the graph should be generated in a separate step and loaded by the runtime:
+
+```csharp
+var authoringServices = new ServiceCollection();
+authoringServices.AddMcpGateway(options =>
+{
+    options.AddTool(
+        "local",
+        AIFunctionFactory.Create(
+            static (string query) => $"github:{query}",
+            new AIFunctionFactoryOptions
+            {
+                Name = "github_search_repositories",
+                Description = "Search GitHub repositories by user query."
+            }));
+});
+
+await using (var authoringProvider = authoringServices.BuildServiceProvider())
+{
+    var authoringGateway = authoringProvider.GetRequiredService<IMcpGateway>();
+    var descriptors = await authoringGateway.ListToolsAsync();
+    var documents = McpGatewayMarkdownLdGraphFile.CreateDocuments(descriptors);
+
+    await McpGatewayMarkdownLdGraphFile.WriteAsync(
+        "artifacts/mcp-tools.graph.json",
+        documents);
+}
+
+var runtimeServices = new ServiceCollection();
+runtimeServices.AddMcpGateway(options =>
+{
+    options.SearchStrategy = McpGatewaySearchStrategy.Graph;
+    options.UseMarkdownLdGraphFile("artifacts/mcp-tools.graph.json");
+
+    options.AddTool(
+        "local",
+        AIFunctionFactory.Create(
+            static (string query) => $"github:{query}",
+            new AIFunctionFactoryOptions
+            {
+                Name = "github_search_repositories",
+                Description = "Search GitHub repositories by user query."
+            }));
+});
 ```
 
+`UseMarkdownLdGraphFile(...)` accepts:
+
+- a gateway graph bundle JSON file created by `McpGatewayMarkdownLdGraphFile.WriteAsync(...)`
+- a directory containing Markdown-LD source documents
+- a single Markdown-LD source file supported by `ManagedCode.MarkdownLd.Kb`
+
+The bundle is a portable set of Markdown-LD source documents, not a serialized RDF store. The runtime still builds the in-memory `ManagedCode.MarkdownLd.Kb` graph from those documents so focused graph search, related matches, and next-step matches behave the same way as generated startup mode.
+
 ## Search Modes
 
-`McpGatewaySearchStrategy.Auto` is the default and usually the right choice:
+`McpGatewaySearchStrategy.Graph` is the default and usually the right choice for zero-cost local retrieval:
 
-- use vector ranking when embeddings are available
-- fall back to lexical ranking when they are not
+- build or load a Markdown-LD graph during index build
+- use deterministic token-distance search from `ManagedCode.MarkdownLd.Kb`
+- return primary matches, related matches, next-step matches, and focused graph counts
+- keep invocation on the same `ToolId` flow
 
-You can also force a mode:
+You can force graph mode explicitly:
 
 ```csharp
 services.AddMcpGateway(options =>
 {
-    options.SearchStrategy = McpGatewaySearchStrategy.Tokenizer;
+    options.SearchStrategy = McpGatewaySearchStrategy.Graph;
 });
 ```
 
-Or:
+Use embedding mode when the host has an embedding generator and wants vector ranking first:
 
 ```csharp
 services.AddMcpGateway(options =>
@@ -474,13 +592,28 @@ services.AddMcpGateway(options =>
 });
 ```
 
+Use `Auto` only when the host wants a policy mode that can use embeddings when the graph is unavailable and otherwise prefer the graph path:
+
+```csharp
+services.AddMcpGateway(options =>
+{
+    options.SearchStrategy = McpGatewaySearchStrategy.Auto;
+});
+```
+
+Graph mode uses `ManagedCode.MarkdownLd.Kb` to convert every local `AITool` and MCP tool descriptor into an in-memory Markdown-LD knowledge graph. Each tool becomes a Markdown document with structured front matter, source metadata, required arguments, input schema text, graph groups, related-tool hints, and next-step hints. Search uses the graph's deterministic Tiktoken token-distance focused search to rank tool documents and returns normal `McpGatewaySearchMatch` results, so invocation still uses the same `ToolId` flow.
+
+The old separate local tokenizer strategy is intentionally not exposed. Token-based search is provided by `ManagedCode.MarkdownLd.Kb` inside the Markdown-LD graph path.
+
 `McpGatewaySearchResult.RankingMode` reports:
 
 - `vector`
-- `lexical`
+- `graph`
 - `browse`
 - `empty`
 
+`McpGatewayIndexBuildResult` also reports graph index state through `IsGraphSearchEnabled`, `GraphNodeCount`, and `GraphEdgeCount`. These values are useful for startup validation and tests when a host requires graph-backed search to be available.
+
 ## Deeper Docs
 
 Use these when you need design details rather than package onboarding:
@@ -489,6 +622,7 @@ Use these when you need design details rather than package onboarding:
 - [ADR-0001: Runtime boundaries and index lifecycle](docs/ADR/ADR-0001-runtime-boundaries-and-index-lifecycle.md)
 - [ADR-0002: Search ranking and query normalization](docs/ADR/ADR-0002-search-ranking-and-query-normalization.md)
 - [ADR-0003: Reusable chat-client and agent auto-discovery modules](docs/ADR/ADR-0003-reusable-chat-client-and-agent-tool-modules.md)
+- [ADR-0005: Markdown-LD graph search for tool retrieval](docs/ADR/ADR-0005-markdown-ld-graph-search-for-tool-retrieval.md)
 - [Feature spec: Search query normalization and ranking](docs/Features/SearchQueryNormalizationAndRanking.md)
 
 ## Local Development