Add grammar fields to GenerationConfig for constrained decoding#19616
Conversation
Summary: Add grammar (string) and grammar_type (string) fields to GenerationConfig. Empty-string defaults, no behavior change when empty. Prepares for grammar-constrained decoding integration (e.g. JSON schema, regex). Differential Revision: D105277261
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/19616
Note: Links to docs will display an error until the docs builds have been completed. ❗ 1 Active SEVsThere are 1 currently active SEVs. If your PR is affected, please view them below: This comment was automatically generated by Dr. CI and updates every 15 minutes. |
|
@kirklandsign has exported this pull request. If you are a Meta employee, you can view the originating Diff in D105277261. |
This PR needs a
|
There was a problem hiding this comment.
Pull request overview
Adds grammar-related fields to the LLM runner GenerationConfig as groundwork for future constrained decoding support.
Changes:
- Adds
grammarfor constrained decoding definitions. - Adds
grammar_typeto identify the grammar format.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| std::string grammar; | ||
|
|
||
| // Grammar format: "json_schema", "regex", "lark", or "gbnf". | ||
| // Only used when grammar is non-empty. | ||
| std::string grammar_type; |
| std::string grammar; | ||
|
|
||
| // Grammar format: "json_schema", "regex", "lark", or "gbnf". | ||
| // Only used when grammar is non-empty. | ||
| std::string grammar_type; |
Summary:
Add grammar (string) and grammar_type (string) fields to GenerationConfig.
Empty-string defaults, no behavior change when empty. Prepares for
grammar-constrained decoding integration (e.g. JSON schema, regex).
Differential Revision: D105277261