Skip to content

Commit b9a7fef

Browse files
aviavissaraviavissar
committed
feat: Extend model name Filter-189
1 parent c94887e commit b9a7fef

4 files changed

Lines changed: 382 additions & 92 deletions

File tree

‎docs/plugins.md‎

Lines changed: 18 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -182,18 +182,29 @@ framework returns an error to the client.
182182

183183
#### `model-group-name-filter`
184184

185-
Restricts the candidate models based on the request body's `model` field. Supports explicit model
186-
selection by name, `auto` (or an absent/empty field) to keep all candidates, and `auto/<group-name>`
187-
to select the models belonging to a named group.
185+
Restricts the candidate models based on a configurable request body field (`fieldName`, default `model`).
186+
Supports explicit selection matched against each candidate's name (by default) or one of its attributes,
187+
`auto` (or an absent/empty field) to keep all candidates, and `auto/<group-name>` to select the models
188+
belonging to a named group.
188189

189190
- Absent, empty, or exactly `auto`: all incoming candidates are kept.
190191
- `auto/<group-name>`: each candidate model is checked for `<group-name>` in its own
191-
`modelgroups.GroupsAttributeKey` attribute; only candidates that list that group are kept.
192-
- Any other non-empty string: the single candidate whose name matches that string is kept.
193-
- An unknown group, an empty intersection, or a malformed (non-string) `model` field: no candidates
192+
`modelgroups.GroupsAttributeKey` attribute; only candidates that list that group are kept. This is
193+
always matched by the candidate's name, regardless of `optionalValues.byField` below.
194+
- Any other non-empty string: the candidate(s) whose comparison value — the candidate's name by
195+
default, or the attribute named by `optionalValues.byField` when configured — matches that string
196+
are kept.
197+
- An unknown group, an empty intersection, or a malformed (non-string) field value: no candidates
194198
are returned, and the pipeline rejects the request with HTTP 429.
195199

196-
**Parameters:** None.
200+
Group membership itself is not configured on this filter: it is resolved at filter time from each
201+
candidate model's `modelgroups.GroupsAttributeKey` attribute, populated by the
202+
[`model-config-datasource`][src-modelconfig] plugin from the `groups` section of its config file.
203+
204+
| Parameter | Type | Required | Default | Description |
205+
| --- | --- | --- | --- | --- |
206+
| `fieldName` | string | no | `model` | Request-body field to read the requested value from. |
207+
| `optionalValues.byField` | string | no | (candidate's name) | Candidate attribute compared against `fieldName`'s value for explicit (non-`auto`) matches. |
197208

198209
```yaml
199210
plugins:

‎pkg/framework/plugins/modelselector/filter/modelgroup/README.md‎

Lines changed: 67 additions & 27 deletions
Original file line numberDiff line numberDiff line change
@@ -1,44 +1,51 @@
11
# `model-group` Name Filter
22

3-
Filter the candidate model names based on the request body mode name field.
4-
Support explicit model selection by specifying a model name in the `model` field.
3+
Filter the candidate models based on a configurable request body field.
4+
Support explicit selection by specifying a value in that field, matched against each
5+
candidate's name (by default) or one of its attributes.
56
Support using `auto` or `auto/group-name` for selecting more than one model candidate.
6-
`auto` (or no model name field) enables using all candidate models (no filtering), while using the pattern `auto/group-name` selects
7+
`auto` (or no value in the field) enables using all candidate models (no filtering), while using the pattern `auto/group-name` selects
78
the models that belong to the group `group-name`.
89

9-
It is registered as type `model-group-name-filter` and runs as a modelselector filter. The filter takes no
10-
plugin parameters: group membership is not configured on the filter itself. Instead it is resolved at filter
11-
time from each candidate model's `modelgroups.GroupsAttributeKey` attribute in the datalayer, which is
12-
populated by the `model-config-datasource` plugin (`pkg/framework/plugins/datalayer/modelconfigcollector`)
13-
from the `groups` section of its config file. This lets group membership be updated dynamically (e.g. via a
14-
Kubernetes ConfigMap) without restarting or reconfiguring this filter.
10+
It is registered as type `model-group-name-filter` and runs as a modelselector filter. Group membership is
11+
not configured on the filter itself: it is resolved at filter time from each candidate model's
12+
`modelgroups.GroupsAttributeKey` attribute in the datalayer, which is populated by the
13+
`model-config-datasource` plugin (`pkg/framework/plugins/datalayer/modelconfigcollector`) from the `groups`
14+
section of its config file. This lets group membership be updated dynamically (e.g. via a Kubernetes
15+
ConfigMap) without restarting or reconfiguring this filter.
1516

16-
When the model name is set to `auto/group-name` the filter matches the requested `group-name` against each
17-
candidate model's group attribute and keeps only the candidates that belong to that group.
17+
When the request value is set to `auto/group-name` the filter matches the requested `group-name` against
18+
each candidate model's group attribute and keeps only the candidates that belong to that group. Group
19+
membership is always matched by the candidate's name, regardless of the `optionalValues.byField` setting
20+
below.
1821

1922
## What it does
2023

21-
1. Reads the `model` field from the request body.
24+
1. Reads the configured field (`fieldName`, default `model`) from the request body.
2225
2. If the field is absent, an empty string or `auto`, all incoming candidates are kept.
23-
3. If the `model` field is formatted as `auto/group-name`, with the prefix `auto` and separator `/`, it extracts `group-name` and keeps the candidate models whose `modelgroups.GroupsAttributeKey` attribute lists that group name.
24-
4. If the `model` field is a non-empty string that is not `auto` and does not start with `auto/`, it is treated as an explicit model name and the single matching candidate is kept.
26+
3. If the field's value is formatted as `auto/group-name`, with the prefix `auto` and separator `/`, it extracts `group-name` and keeps the candidate models whose `modelgroups.GroupsAttributeKey` attribute lists that group name.
27+
4. If the field's value is a valid non-empty string that does not start with the prefix `auto`, it is compared against each candidate's comparison value — the candidate's name by default, or the attribute named by `optionalValues.byField` when configured — and the matching candidate(s) are kept.
2528
5. If the intersection is empty or the field is malformed (not a string), the filter returns no candidates and the pipeline rejects the request with HTTP 429.
2629

27-
## Inputs consumed
28-
29-
- The `model` field of the request body.
30-
- The candidate model list passed in by the pipeline, with each model's `modelgroups.GroupsAttributeKey` attribute (if any), as populated by the `model-config-datasource` plugin.
30+
## Configuration
3131

32-
## Example configuration
32+
The parameters are a flat JSON object with two optional keys:
3333

34-
```yaml
35-
plugins:
36-
- type: model-group-name-filter
34+
```json
35+
{
36+
"fieldName": "model",
37+
"optionalValues": {
38+
"byField": "modelId"
39+
}
40+
}
3741
```
3842

43+
- `fieldName` (optional, default `"model"`): the request-body field to read the requested value from.
44+
- `optionalValues.byField` (optional): the name of the candidate attribute compared against `fieldName`'s value for explicit (non-`auto`) matches. When omitted, the candidate's name (`Model.GetName()`) is used. When set, the candidate must carry that attribute as a `StringAttribute` (a small `Cloneable` wrapper around `string`); candidates missing it never match. This setting does not affect `auto/<group-name>` matching, which is always by name.
45+
3946
Group membership itself is configured on the `model-config-datasource` plugin's config file, not here.
40-
That plugin's config file (pointed to by its `modelsPath` parameter) lists models (omitted in the example) and,
41-
alongside them, a `groups` section mapping a group name to the model names that belong to it:
47+
That plugin's config file (pointed to by its `modelsPath` parameter) lists models and, alongside them, a
48+
`groups` section mapping a group name to the model names that belong to it:
4249

4350
```json
4451
{
@@ -49,7 +56,40 @@ alongside them, a `groups` section mapping a group name to the model names that
4956
}
5057
```
5158

52-
With this configuration snippet, a request with `model` set to `auto/fast` is filtered to the candidates `qwen3-8b` and
53-
`gpt-oss-20b`; `auto/planning` is filtered to `gpt-oss-120b` and `gemma4`. A model can appear in more than
54-
one group's `models` list. A group with an empty `name` or an empty `models` list is skipped (and logged) by `model-config-datasource`, as is a group entry naming a model not present in the top-level
59+
With this configuration snippet, a request with `model` set to `auto/fast` is filtered to the candidates
60+
`qwen3-8b` and `gpt-oss-20b`; `auto/planning` is filtered to `gpt-oss-120b` and `gemma4`. A model can appear
61+
in more than one group's `models` list. A group with an empty `name` or an empty `models` list is skipped
62+
(and logged) by `model-config-datasource`, as is a group entry naming a model not present in the top-level
5563
`models` list.
64+
65+
## Inputs consumed
66+
67+
- The `fieldName` field of the request body.
68+
- The candidate model list passed in by the pipeline, including each candidate's name, its
69+
`modelgroups.GroupsAttributeKey` attribute (if any) as populated by the `model-config-datasource` plugin,
70+
and (when `optionalValues.byField` is set) its other attributes.
71+
72+
## Example configurations
73+
74+
### Defaults (the field defaults to `model`, matched by name)
75+
76+
```yaml
77+
plugins:
78+
- type: model-group-name-filter
79+
```
80+
81+
Group membership (for `auto/<group-name>`) is configured separately, on the `model-config-datasource`
82+
plugin, as shown above.
83+
84+
### Matching by a different field/attribute
85+
86+
```yaml
87+
plugins:
88+
- type: model-group-name-filter
89+
parameters:
90+
fieldName: anthropic_version
91+
optionalValues:
92+
byField: anthropicVersion
93+
```
94+
95+
Matches `request.Body["anthropic_version"]` (e.g. `"bedrock-2023-05-31"`) against the `anthropicVersion` attribute of each candidate, instead of the candidate's name.

‎pkg/framework/plugins/modelselector/filter/modelgroup/filter.go‎

Lines changed: 126 additions & 28 deletions
Original file line numberDiff line numberDiff line change
@@ -15,16 +15,18 @@ limitations under the License.
1515
*/
1616

1717
// Package modelgroup implements a modelselector filter that restricts the
18-
// candidate models based on the request body model name field, supporting
19-
// explicit model selection as well as "auto" and "auto/<group-name>" selection
20-
// of a configured group of models.
18+
// candidate models based on a configurable request body field, supporting
19+
// explicit selection (by the candidate's name, or optionally by one of its
20+
// attributes) as well as "auto" and "auto/<group-name>" selection of a
21+
// configured group of models.
2122
//
2223
// For detailed behavioral intent and configuration, see the package README.
2324
package modelgroup
2425

2526
import (
2627
"context"
2728
"encoding/json"
29+
"fmt"
2830
"strings"
2931

3032
"sigs.k8s.io/controller-runtime/pkg/log"
@@ -41,8 +43,13 @@ const (
4143
// ModelGroupFilterType is the registered name of the model-group filter plugin.
4244
ModelGroupFilterType = "model-group-name-filter"
4345

44-
// requestModelField is the request-body field holding the requested model name.
45-
requestModelField = "model"
46+
// defaultFieldName is the request-body field used when the configuration
47+
// omits fieldName.
48+
defaultFieldName = "model"
49+
50+
// defaultByFieldValue is the sentinel meaning "no byField override": the
51+
// candidate value is the model's name (Model.GetName()).
52+
defaultByFieldValue = ""
4653

4754
// autoPrefix is the model name value (or prefix) that triggers "all candidates"
4855
// or group-based filtering.
@@ -55,27 +62,93 @@ const (
5562
// compile-time type validation
5663
var _ modelselector.Filter = &ModelGroupFilter{}
5764

58-
// ModelGroupFilterFactory returns a configured ModelGroupFilter. The filter takes
59-
// no parameters: group membership is resolved at filter time from each candidate
60-
// model's modelgroups.GroupsAttributeKey attribute, populated by the
65+
// compile-time interface validation
66+
var _ datalayer.Cloneable = StringAttribute("")
67+
68+
// StringAttribute is a Cloneable wrapper around a plain string. It lets a
69+
// simple string value be stored on a candidate's AttributeMap, so that
70+
// "optionalValues.byField" can point at it for comparison.
71+
type StringAttribute string
72+
73+
// Clone implements datalayer.Cloneable. Strings are immutable, so cloning
74+
// returns the same value.
75+
func (s StringAttribute) Clone() datalayer.Cloneable {
76+
return s
77+
}
78+
79+
// OptionalValues holds the optional settings of ModelGroupFilter.
80+
type OptionalValues struct {
81+
// ByField names the candidate attribute compared against the request field's
82+
// value for explicit (non-"auto") matches. Empty (the default) means "use
83+
// Model.GetName()". Group membership ("auto/<group-name>") always matches by
84+
// the candidate's name, regardless of this setting.
85+
ByField string `json:"byField,omitempty"`
86+
}
87+
88+
// ModelGroupFilterConfig is the JSON parameter shape for ModelGroupFilter. Group
89+
// membership is NOT configured here: it is resolved at filter time from each
90+
// candidate model's modelgroups.GroupsAttributeKey attribute, populated by the
6191
// model-config-datasource plugin from the shared config file's "groups" list.
62-
func ModelGroupFilterFactory(name string, _ json.RawMessage, _ plugin.Handle) (plugin.Plugin, error) {
63-
return NewModelGroupFilter().WithName(name), nil
92+
type ModelGroupFilterConfig struct {
93+
// FieldName is the request-body field to read the requested value from.
94+
// Defaults to "model" when omitted.
95+
FieldName string `json:"fieldName,omitempty"`
96+
// OptionalValues holds optional settings; nil/omitted uses all defaults.
97+
OptionalValues *OptionalValues `json:"optionalValues,omitempty"`
98+
}
99+
100+
// ModelGroupFilterFactory parses the plugin parameters (fieldName,
101+
// optionalValues) and returns a configured ModelGroupFilter. Group membership
102+
// is resolved at filter time from each candidate model's
103+
// modelgroups.GroupsAttributeKey attribute, populated by the
104+
// model-config-datasource plugin from the shared config file's "groups" list;
105+
// it is not configured here.
106+
func ModelGroupFilterFactory(name string, params json.RawMessage, _ plugin.Handle) (plugin.Plugin, error) {
107+
var cfg ModelGroupFilterConfig
108+
if len(params) > 0 {
109+
if err := json.Unmarshal(params, &cfg); err != nil {
110+
return nil, fmt.Errorf("failed to parse parameters of '%s': %w", ModelGroupFilterType, err)
111+
}
112+
}
113+
114+
fieldName := cfg.FieldName
115+
if fieldName == "" {
116+
fieldName = defaultFieldName
117+
}
118+
byField := defaultByFieldValue
119+
if cfg.OptionalValues != nil {
120+
byField = cfg.OptionalValues.ByField
121+
}
122+
123+
return newModelGroupFilter(fieldName, byField).WithName(name), nil
64124
}
65125

66-
// NewModelGroupFilter initializes a new ModelGroupFilter.
126+
// NewModelGroupFilter initializes a new ModelGroupFilter with the default field
127+
// name ("model") and the default candidate value (the model's name).
67128
func NewModelGroupFilter() *ModelGroupFilter {
129+
return newModelGroupFilter(defaultFieldName, defaultByFieldValue)
130+
}
131+
132+
// newModelGroupFilter builds a ModelGroupFilter that reads requests from
133+
// fieldName and compares explicit (non-auto) matches via byField (empty means
134+
// the candidate's name).
135+
func newModelGroupFilter(fieldName, byField string) *ModelGroupFilter {
68136
return &ModelGroupFilter{
69137
typedName: plugin.TypedName{Type: ModelGroupFilterType, Name: ModelGroupFilterType},
138+
fieldName: fieldName,
139+
byField: byField,
70140
}
71141
}
72142

73-
// ModelGroupFilter restricts the candidate models based on the request body's
74-
// "model" field: an exact model name, "auto"/absent/empty for all candidates,
75-
// or "auto/<group-name>" for the candidates whose modelgroups.GroupsAttributeKey
76-
// attribute lists that group name.
143+
// ModelGroupFilter restricts the candidate models based on a configurable
144+
// request body field: an explicit value (matched by the candidate's name, or
145+
// by an attribute when byField is set), "auto"/absent/empty for all
146+
// candidates, or "auto/<group-name>" for the candidates whose
147+
// modelgroups.GroupsAttributeKey attribute lists that group name.
77148
type ModelGroupFilter struct {
78149
typedName plugin.TypedName
150+
fieldName string
151+
byField string
79152
}
80153

81154
// TypedName returns the type and name tuple of this plugin instance.
@@ -89,38 +162,49 @@ func (f *ModelGroupFilter) WithName(name string) *ModelGroupFilter {
89162
return f
90163
}
91164

92-
// Filter returns the candidate models based on the "model" field of the request:
165+
// Filter returns the candidate models based on the configured request field:
93166
// - absent, empty, or exactly "auto": all candidates pass through.
94-
// - "auto/<group-name>": candidates whose name appears in the named group.
95-
// - a plain non-"auto"-prefixed string: the single candidate matching that name.
96-
// - "auto/" (empty group name), unknown group, unmatched name, or non-string
167+
// - "auto/<group-name>": candidates whose modelgroups.GroupsAttributeKey
168+
// attribute lists that group name.
169+
// - a plain non-"auto"-prefixed string: the candidate(s) whose comparison
170+
// value (the name, or the byField attribute when configured) matches it.
171+
// - "auto/" (empty group name), unknown group, unmatched value, or non-string
97172
// type: no candidates (pipeline rejects with 429).
98173
func (f *ModelGroupFilter) Filter(ctx context.Context, _ *plugin.CycleState, request *requesthandling.InferenceRequest, models []datalayer.Model) []datalayer.Model {
99174
logger := log.FromContext(ctx)
100175

101-
raw := request.Body[requestModelField]
176+
raw := request.Body[f.fieldName]
102177
requested, ok := raw.(string)
103178
if !ok && raw != nil {
104-
logger.V(logutil.VERBOSE).Info("malformed model field in request body, no available model candidates", "field", requestModelField)
179+
logger.V(logutil.VERBOSE).Info("malformed request field, no available model candidates", "field", f.fieldName)
105180
return []datalayer.Model{}
106181
}
107182

108183
if requested == "" || requested == autoPrefix {
109-
logger.V(logutil.VERBOSE).Info("no model or auto in request body, all candidates kept", "field", requestModelField)
184+
logger.V(logutil.VERBOSE).Info("no value or auto in request field, all candidates kept", "field", f.fieldName)
110185
return models
111186
}
112187

113188
autoGroupPrefix := autoPrefix + autoGroupSeparator
114189
if !strings.HasPrefix(requested, autoGroupPrefix) {
115-
// Not an "auto/..." selector: treat it as an explicit model name request.
190+
// Not an "auto/..." selector: treat it as an explicit request, matched
191+
// by each candidate's comparison value (name, or byField attribute).
192+
var result []datalayer.Model
116193
for _, model := range models {
117-
if model.GetName() == requested {
118-
logger.V(logutil.DEBUG).Info("model-group filter applied explicit match", "requested", requested)
119-
return []datalayer.Model{model}
194+
value, found := f.candidateValue(model)
195+
if !found {
196+
continue
197+
}
198+
if value == requested {
199+
result = append(result, model)
120200
}
121201
}
122-
logger.V(logutil.VERBOSE).Info("request body model is not configured", "requested", requested)
123-
return []datalayer.Model{}
202+
if len(result) == 0 {
203+
logger.V(logutil.VERBOSE).Info("request field value is not configured", "requested", requested)
204+
return []datalayer.Model{}
205+
}
206+
logger.V(logutil.DEBUG).Info("model-group filter applied explicit match", "requested", requested, "candidates", len(result))
207+
return result
124208
}
125209

126210
groupName := strings.TrimPrefix(requested, autoGroupPrefix)
@@ -148,3 +232,17 @@ func (f *ModelGroupFilter) Filter(ctx context.Context, _ *plugin.CycleState, req
148232
logger.V(logutil.DEBUG).Info("model-group filter applied group match", "group", groupName, "candidates", len(result))
149233
return result
150234
}
235+
236+
// candidateValue resolves the comparison value for a candidate, for explicit
237+
// (non-"auto") matches: the model's name by default, or a string attribute
238+
// named by byField when set.
239+
func (f *ModelGroupFilter) candidateValue(model datalayer.Model) (string, bool) {
240+
if f.byField == defaultByFieldValue {
241+
return model.GetName(), true
242+
}
243+
value, err := datalayer.ReadAttributeKey[StringAttribute](model.GetAttributes(), f.byField)
244+
if err != nil {
245+
return "", false
246+
}
247+
return string(value), true
248+
}

0 commit comments

Comments
 (0)