Skip to content

Config redesign in FastEval - replace boolean flags with lists #2619

Description

@iluise

Describe the task. It can be a feature, documentation, etc.

Every new plot type requires adding a boolean flag to the YAML,
a validation check, and an if branch in plot_orchestration.py. Today we have:

# Per-stream plotting flags (in streams.ERA5.plotting):
plot_maps: true
plot_bias: false
plot_target: false
plot_histograms: true       # or "per-sample" / "across-samples"
plot_animations: true
plot_timeseries: false

# Global evaluation/scoring flags (in evaluation):
summary_plots: true
ratio_plots: false
heat_maps: false
score_cards: false
bar_plots: false
plot_score_maps: false
plot_score_animations: false
plot_score_init_timeseries: true

That's 14 booleans — and growing.

Proposed replacement: use explicit lists of enabled plot/score types.

evaluation:
  metrics: [rmse, mae, psd]

  # Which score-level visualisations to produce:
  score_plots: [summary, psd, timeseries]    # replaces summary_plots, plot_score_init_timeseries, etc.
  # Supported values: summary, ratio, heatmap, scorecard, bar, psd, qq, rank_histogram, score_map, score_animation, timeseries

default_streams:
  ERA5:
    plotting:
      sample: [1, 3]
      forecast_step: [1, 3, 2]
      ensemble: [0, 2, 5]

      # Which data plots to produce per sample:
      data_plots: [maps, histograms, timeseries]   # replaces plot_maps, plot_histograms, plot_timeseries, plot_animations
      # Supported values: maps, bias, target, histograms, animations, timeseries

      # Histogram mode (only relevant if "histograms" is in data_plots):
      histogram_mode: "all"   # "per-sample", "across-samples", or "all" (both)

Hedgedoc URL, if you are keeping notes, plots, logs in hedgedoc.

No response

Area

  • datasets, data readers, data preparation and transfer
  • model
  • science
  • infrastructure and engineering
  • evaluation, export and visualization
  • documentation
  • performance

Metadata

Metadata

Assignees

Labels

evalanything related to the model evaluation pipeline

Type

No type

Fields

No fields configured for issues without a type.

Projects

Status
No status

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions