Skip to content

data-model.mdx

Latest commit

 

History

History
261 lines (196 loc) · 12.2 KB

data-model.mdx

File metadata and controls

261 lines (196 loc) · 12.2 KB
Raw
title Experiments Data Model
description Data model for Datasets, DatasetItems, DatasetRuns, DatasetRunItems, and function definitions for Tasks and Evaluators.
sidebarTitle Data Model

Experiments Data Model

This page describes the data model for experiment-related objects in Langfuse. For an overview of how these objects work together, see the Concepts page. For score and score config objects, see the Scores data model.

For detailed reference please refer to

Objects

Datasets [#datasets]

Datasets are a collection of inputs and, optionally, expected outputs that can be used during Dataset runs.

Datasets are a collection of DatasetItems.

classDiagram
direction LR
    class Dataset {
        name
        description
        metadata
    }

    class DatasetItem {
        datasetName
        input
        expectedOutput
        metadata
        sourceTraceId
        sourceObservationId
        id
        status
    }

    Dataset "1" --> "n" DatasetItem
Loading

Dataset object [#dataset-object]

Attribute Type Required Description
id string Yes Unique identifier for the dataset
name string Yes Name of the dataset
description string No Description of the dataset
metadata object No Additional metadata for the dataset
remoteExperimentUrl string No Webhook endpoint for triggering experiments
remoteExperimentPayload object No Payload for triggering experiments

DatasetItem object [#datasetitem-object]

Attribute Type Required Description
id string Yes Unique identifier for the dataset item. Dataset items are upserted on their id. Id needs to be unique (project-level) and cannot be reused across datasets.
datasetId string Yes ID of the dataset this item belongs to
input object No Input data for the dataset item
expectedOutput object No Expected output data for the dataset item
metadata object No Additional metadata for the dataset item
mediaReferences object[] No Resolved media references found in input, expectedOutput, and metadata. Included on SDK dataset fetches and API responses that include resolved dataset media.
sourceTraceId string No ID of the source trace to link this dataset item to
sourceObservationId string No ID of the source observation to link this dataset item to
status DatasetStatus No Status of the dataset item. Defaults to ACTIVE for newly created items. Possible values: ACTIVE, ARCHIVED

DatasetItemMediaReference object [#datasetitemmediareference-object]

Dataset item media references point from a stored media token in input, expectedOutput, or metadata to a signed media download URL.

Attribute Type Required Description
field string Yes Field enum for the dataset item property containing the reference. One of input, expected_output (for expectedOutput), or metadata.
referenceString string Yes Original Langfuse media reference string stored in the dataset item.
jsonPath string Yes JSONPath of the string holding the reference inside the field, for example $['image'].
media object Yes (nullable) Resolved media metadata. null if the referenced media does not exist or has not been uploaded successfully.

The nested media object contains mediaId, contentType, contentLength, url, and urlExpiry. The url is a signed download URL and should be used before its expiration date. To refresh the signed URL, refetch the dataset.

DatasetRun (Experiment Run) [#datasetrun-experiment-run]

Dataset runs are used to run a dataset through your LLM application and optionally apply evaluation methods to the results. This is often referred to as Experiment run.


classDiagram
direction LR
    class DatasetRun {
        id
        name
        description
        metadata
        datasetId
    }

    DatasetRun "1" --> "n" DatasetRunItem

    class DatasetRunItem {
        id
        datasetRunId
        datasetItemId
        traceId
        observationId
    }
Loading

DatasetRun object [#datasetrun-object]

Attribute Type Required Description
id string Yes Unique identifier for the dataset run
name string Yes Name of the dataset run
description string No Description of the dataset run
metadata object No Additional metadata for the dataset run
datasetId string Yes ID of the dataset this run belongs to

DatasetRunItem object [#datasetrunitem-object]

Attribute Type Required Description
id string Yes Unique identifier for the dataset run item
datasetRunId string Yes ID of the dataset run this item belongs to
datasetItemId string Yes ID of the dataset item to link to this run
traceId string Yes ID of the trace to link to this run
observationId string No ID of the observation to link to this run
Most of the time, we recommend that DatasetRunItems reference TraceIDs directly. The reference to ObservationID exists for backwards compatibility with older SDK versions.

End to End Data Relations [#end-to-end-data-relations]

An experiment can combine a few Langfuse objects:

  • DatasetRuns (or Experiment runs) are created by looping through all or selected DatasetItems of a Dataset with your LLM application.
  • For each DatasetItem passed into the LLM application as an Input a DatasetRunItem & a Trace are created.
  • Optionally Scores can be added to the Traces to evaluate the output of the LLM application during the DatasetRun.

classDiagram
direction LR
	namespace Datasets {
        class Dataset {
        }
        class DatasetItem {
        }
    }
    namespace DatasetRuns {
        class DatasetRun {
        }
        class DatasetRunItem {
        }
    }
    namespace Observability {
        class Trace {
        }
        class Observation {
        }
        }
    namespace Evals {
        class Score {
        }
    }


    class DatasetRun {
    }

    class DatasetRunItem {
    }

    class Dataset {
    }

    class DatasetItem {
    }

    class Trace {
	    input
	    output
    }

    class Observation {
        input
	    output
    }

    class Score {
	    name
	    value
	    comment
    }

    Dataset "1" --> "n" DatasetItem
    Dataset "1" --> "n" DatasetRun
    DatasetRun "1" --> "n" DatasetRunItem
    DatasetRunItem "1" --> "1" DatasetItem
    Trace "1" --> "n" Observation
    DatasetRunItem "1" --> "1" Trace
    DatasetRunItem "1" --> "0..1" Observation
    Observation "1" --> "n" Score
    Trace "1" --> "n" Score
Loading

See the Concepts page for more information on how these objects work together conceptually. See the observability core concepts page for more details on traces and observations. See the Scores data model for more details on score and score config objects.

Function Definitions [#function-definitions]

When running experiments via the SDK, you define task and evaluator functions. These are user-defined functions that the experiment runner calls for each dataset item. For more information on how experiments work conceptually, see the Concepts page.

Task [#task]

A task is a function that takes a dataset item and returns an output during an experiment run.

See SDK references for function signatures and parameters:

Evaluator [#evaluator]

An evaluator is a function that scores the output of a task for a single dataset item. Evaluators receive the input, output, expected output, and metadata, and return an Evaluation object that becomes a Score in Langfuse.

See SDK references for function signatures and parameters:

Run Evaluator [#run-evaluator]

A run evaluator is a function that assesses the full experiment results and computes aggregate metrics. When run on Langfuse datasets, the resulting scores are attached to the dataset run.

See SDK references for function signatures and parameters:

For detailed usage examples of tasks and evaluators, see [Experiments via SDK](/docs/evaluation/experiments/experiments-via-sdk).

Local Datasets [#local-datasets]

Currently, if an Experiment via SDK is used to run experiments on local datasets, only traces are created in Langfuse - no dataset runs are generated. Each task execution creates an individual trace for observability and debugging.

We have improvements on our roadmap to support similar functionality such as run overviews, comparison views, and more for experiments on local datasets as for Langfuse datasets.