Skip to content

Add support for selecting few-shot examples by id#1284

Open
ErenAta16 wants to merge 1 commit into
huggingface:mainfrom
ErenAta16:add-few-shot-id-list-selection
Open

Add support for selecting few-shot examples by id#1284
ErenAta16 wants to merge 1 commit into
huggingface:mainfrom
ErenAta16:add-few-shot-id-list-selection

Conversation

@ErenAta16

Copy link
Copy Markdown

Custom tasks currently choose few-shot examples through few_shots_select (sequential, random, or balanced), which always picks from the pool programmatically. There was no way to pin a task to a specific, reproducible set of few-shot examples the way the LM-Evaluation-Harness id_sampler does, which matters when comparing against a published harness run or when a paper specifies exact few-shot examples by id.

LightevalTaskConfig gains two new fields:

  • few_shots_id_column: name of a dataset column to use as a stable identifier for few-shot rows. When set, this value becomes the id of the corresponding few-shot Doc instead of the row's position in the split. Evaluation docs are unaffected and keep using the row position, so this is opt-in and backward compatible.
  • few_shots_id_list: an explicit list of ids (matching few_shots_id_column, or the row position if it is not set) to use as few-shot examples, in the given order, instead of applying few_shots_select.

FewShotSampler picks up few_shots_id_list ahead of the existing sorting strategies and raises a clear error if any requested id is missing from the few-shot pool, rather than failing silently or picking something else instead.

Fixes #634

Custom tasks currently choose few-shot examples through
few_shots_select (sequential, random, or balanced), which always
picks from the pool programmatically. There was no way to pin a task
to a specific, reproducible set of few-shot examples the way the
LM-Evaluation-Harness id_sampler does, which matters when comparing
against a published harness run or when a paper specifies exact
few-shot examples by id.

LightevalTaskConfig gains two new fields:

- few_shots_id_column: name of a dataset column to use as a stable
  identifier for few-shot rows. When set, this value becomes the id
  of the corresponding few-shot Doc instead of the row's position in
  the split. Evaluation docs are unaffected and keep using the row
  position, so this is opt-in and backward compatible.
- few_shots_id_list: an explicit list of ids (matching
  few_shots_id_column, or the row position if it is not set) to use
  as few-shot examples, in the given order, instead of applying
  few_shots_select.

FewShotSampler picks up few_shots_id_list ahead of the existing
sorting strategies and raises a clear error if any requested id is
missing from the few-shot pool, rather than failing silently or
picking something else instead.

Fixes huggingface#634
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[FT] Build in a way to specify specific IDs/Lines in Dataset to use as few-shot examples in the same split

1 participant