Skip to content

Commit 7f6ddd6

Browse files
ajtiwari07CopilotJerryNixonrobertopc1Copilot
authored
Enable internal text embedding API (#3441)
**Summary** This PR adds configurable text chunking capabilities to the embeddings API, enabling automatic text segmentation before embedding generation. This feature supports both single-text and multi-document batch processing with runtime configuration and query parameter overrides. **Changes** Configuration Added EmbeddingsChunkingOptions.cs - Configuration model for chunking behavior Enabled (bool) - Enable/disable chunking SizeChars (int) - Chunk size in characters (default: 800) OverlapChars (int) - Overlap between chunks (default: 100) EffectiveSizeChars property ensures minimum valid chunk size Modified EmbeddingsOptions.cs - Added Chunking property and IsChunkingEnabled helper Removed EmbeddingsCacheOptions.cs - Simplified configuration by removing unused cache feature **API Enhancements** Modified Controllers/EmbeddingController.cs Auto-detects request type (single text vs. document array) Implements overlapping text chunking algorithm Supports query parameter overrides: $chunking.enabled, $chunking.size-chars, $chunking.overlap-chars Returns multiple embeddings per document when chunking is enabled Added Models/EmbedDocumentRequest.cs - Request model for document arrays Added Models/EmbedDocumentResponse.cs - Response model with chunked embeddings Schema ([schemas](vscode-file://vscode-app/c:/Users/ajtiwari/AppData/Local/Programs/Microsoft%20VS%20Code/07ff9d6178/resources/app/out/vs/code/electron-browser/workbench/workbench.html)) Modified dab.draft.schema.json - Added chunking configuration schema with validation rules Testing ([UnitTests](vscode-file://vscode-app/c:/Users/ajtiwari/AppData/Local/Programs/Microsoft%20VS%20Code/07ff9d6178/resources/app/out/vs/code/electron-browser/workbench/workbench.html)) Added EmbeddingsChunkingOptionsTests.cs (13 tests) - Configuration validation Added ChunkTextTests.cs (21 tests) - Chunking algorithm validation including edge cases Modified EmbeddingControllerTests.cs (+18 tests) - API endpoint tests for chunking and document arrays Total Test Coverage: 72 tests (48 existing + 24 new) - All passing## Why make this change? **Testing** All 72 unit tests passing Edge cases covered: empty text, very small chunks, overlap larger than chunk size, Unicode text Query parameter parsing validated Backward compatibility verified **Breaking Changes** None - This is a backward-compatible addition. Existing single-text requests continue to work without modification. --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: JerryNixon <1749983+JerryNixon@users.noreply.github.com> Co-authored-by: roberto.perez <robertopc@gmail.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-authored-by: Souvik Ghosh <souvikofficial04@gmail.com> Co-authored-by: Aniruddh Munde <anmunde@microsoft.com> Co-authored-by: sayalikudale <68876274+sayalikudale@users.noreply.github.com> Co-authored-by: RubenCerna2079 <32799214+RubenCerna2079@users.noreply.github.com> Co-authored-by: aaronburtle <93220300+aaronburtle@users.noreply.github.com> Co-authored-by: Anusha Kolan <anushakolan10@gmail.com> Co-authored-by: Sayali Kudale <sayalikudale@microsoft.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Jim Roberts <JimRoberts-MS@users.noreply.github.com>
1 parent c54a01e commit 7f6ddd6

160 files changed

Lines changed: 10338 additions & 577 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎schemas/dab.draft.schema.json‎

Lines changed: 165 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -770,6 +770,171 @@
770770
"default": 4
771771
}
772772
}
773+
},
774+
"embeddings": {
775+
"type": "object",
776+
"description": "Configuration for text embedding/vectorization service. Supports OpenAI and Azure OpenAI providers.",
777+
"additionalProperties": false,
778+
"properties": {
779+
"enabled": {
780+
"type": "boolean",
781+
"description": "Whether the embedding service is enabled. Defaults to true.",
782+
"default": true
783+
},
784+
"provider": {
785+
"type": "string",
786+
"description": "The embedding provider type.",
787+
"enum": ["azure-openai", "openai"]
788+
},
789+
"base-url": {
790+
"type": "string",
791+
"description": "The provider base URL. For Azure OpenAI, use the Azure resource endpoint. For OpenAI, use https://api.openai.com."
792+
},
793+
"api-key": {
794+
"type": "string",
795+
"description": "The API key for authentication. Supports environment variable substitution with @env('VAR_NAME')."
796+
},
797+
"model": {
798+
"type": "string",
799+
"description": "The model or deployment name. Required for Azure OpenAI (deployment name). For OpenAI, defaults to 'text-embedding-3-small' if not specified."
800+
},
801+
"api-version": {
802+
"type": "string",
803+
"description": "Azure API version. Only used for Azure OpenAI provider.",
804+
"default": "2023-05-15"
805+
},
806+
"dimensions": {
807+
"type": "integer",
808+
"description": "Output vector dimensions. Defaults to 1536 if not specified. Useful for Redis schema alignment.",
809+
"default": 1536,
810+
"minimum": 1
811+
},
812+
"timeout-ms": {
813+
"type": "integer",
814+
"description": "Request timeout in milliseconds.",
815+
"default": 30000,
816+
"minimum": 1,
817+
"maximum": 300000
818+
},
819+
"endpoint": {
820+
"type": "object",
821+
"description": "REST endpoint configuration for the embedding service.",
822+
"additionalProperties": false,
823+
"properties": {
824+
"enabled": {
825+
"type": "boolean",
826+
"description": "Whether the /embed REST endpoint is enabled. Defaults to false.",
827+
"default": false
828+
},
829+
"path": {
830+
"type": "string",
831+
"description": "The URL path for the embedding endpoint. Defaults to '/embed'.",
832+
"default": "/embed"
833+
},
834+
"roles": {
835+
"type": "array",
836+
"description": "The roles allowed to access the embedding endpoint. Defaults to ['authenticated'].",
837+
"default": ["authenticated"],
838+
"items": {
839+
"type": "string"
840+
}
841+
}
842+
}
843+
},
844+
"health": {
845+
"type": "object",
846+
"description": "Health check configuration for the embedding service.",
847+
"additionalProperties": false,
848+
"properties": {
849+
"enabled": {
850+
"type": "boolean",
851+
"description": "Whether health checks are enabled for embeddings. Defaults to false.",
852+
"default": false
853+
},
854+
"threshold-ms": {
855+
"type": "integer",
856+
"description": "The maximum response time in milliseconds to be considered healthy.",
857+
"default": 1000,
858+
"minimum": 1,
859+
"maximum": 300000
860+
},
861+
"test-text": {
862+
"type": "string",
863+
"description": "The text to use for health check validation.",
864+
"default": "health check"
865+
},
866+
"expected-dimensions": {
867+
"type": "integer",
868+
"description": "The expected number of dimensions in the embedding result. If specified, dimension validation is performed.",
869+
"minimum": 1
870+
}
871+
}
872+
},
873+
"chunking": {
874+
"type": "object",
875+
"description": "Chunking configuration for text processing before embedding. Used to split large text inputs into smaller chunks.",
876+
"additionalProperties": false,
877+
"properties": {
878+
"enabled": {
879+
"type": "boolean",
880+
"description": "Whether chunking is enabled. Defaults to true.",
881+
"default": true
882+
},
883+
"size-chars": {
884+
"type": "integer",
885+
"description": "The size of each chunk in characters.",
886+
"default": 800,
887+
"minimum": 1
888+
},
889+
"overlap-chars": {
890+
"type": "integer",
891+
"description": "The number of characters to overlap between consecutive chunks. Overlap helps maintain context across chunk boundaries.",
892+
"default": 100,
893+
"minimum": 0
894+
}
895+
}
896+
}
897+
},
898+
"required": ["provider", "base-url", "api-key"],
899+
"allOf": [
900+
{
901+
"$comment": "Azure OpenAI requires the model (deployment name) to be specified.",
902+
"if": {
903+
"properties": {
904+
"provider": {
905+
"const": "azure-openai"
906+
}
907+
},
908+
"required": ["provider"]
909+
},
910+
"then": {
911+
"required": ["model"],
912+
"properties": {
913+
"api-version": {
914+
"type": "string",
915+
"description": "Azure API version. Required for Azure OpenAI provider.",
916+
"default": "2023-05-15"
917+
}
918+
}
919+
}
920+
},
921+
{
922+
"$comment": "OpenAI does not require model (defaults to text-embedding-3-small) and does not use api-version.",
923+
"if": {
924+
"properties": {
925+
"provider": {
926+
"const": "openai"
927+
}
928+
},
929+
"required": ["provider"]
930+
},
931+
"then": {
932+
"properties": {
933+
"api-version": false
934+
}
935+
}
936+
}
937+
]
773938
}
774939
}
775940
},

0 commit comments

Comments
 (0)