Add eCDF plot to SBC plots by MolinAlexei · Pull Request #1853 · sbi-dev/sbi

MolinAlexei · 2026-04-13T10:37:01Z

I added an option to do eCDF plotting in the sbc_plot function.
This was apparently already done in pull request #1739, but has not been merged with the main branch since.

I used the same formatting as the existing functions used in sbc_plots. The way I compute the ellipse may be wrong, as I just transform the shaded area from the CDF to an eCDF.

MolinAlexei · 2026-04-13T11:45:02Z

PS : Here is a look of the output plots

janfb · 2026-04-16T13:38:28Z

Thanks a lot for you PR @MolinAlexei !

We will have will have a look at it soon.

codecov · 2026-04-16T13:44:55Z

Codecov Report

❌ Patch coverage is 95.55556% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 87.96%. Comparing base (ea71482) to head (fe4ff88).
⚠️ Report is 8 commits behind head on main.

Files with missing lines	Patch %	Lines
sbi/analysis/plot.py	95.55%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1853      +/-   ##
==========================================
+ Coverage   87.94%   87.96%   +0.02%     
==========================================
  Files         140      140              
  Lines       12845    13192     +347     
==========================================
+ Hits        11297    11605     +308     
- Misses       1548     1587      +39

Flag	Coverage Δ
fast	`82.71% <95.55%> (?)`

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
sbi/analysis/plot.py	`77.96% <95.55%> (+0.77%)`	⬆️

... and 5 files with indirect coverage changes

janfb · 2026-04-18T19:36:49Z

Code review

Found 2 issues:

Docstring for _plot_ranks_as_ecdf lists two parameters (show_legend, legend_kwargs) that are not in the function signature, and describes show_ylabel as toggling a "counts" y-label while the function actually sets plt.ylabel("empirical CDF - expected CDF"). These look copy-pasted from _plot_ranks_as_hist/_plot_ranks_as_cdf and should be cleaned up.

sbi/sbi/analysis/plot.py

Lines 2039 to 2056 in 9f1a6f1

    
           ) -> None: 
        
               """Plot ranks as a delta of the empirical CDFs to the expected CDF 
        
               Args: 
        
                   ranks: SBC ranks in shape (num_sbc_runs, ) 
        
                   num_bins: number of bins for the histogram, recommendation is num_sbc_runs / 20. 
        
                   num_repeats: number of repeats of each CDF step, i.e., resolution of the eCDF. 
        
                   ranks_label: label for the ranks, e.g., when comparing ranks of different 
        
                       methods. 
        
                   xlabel: label for the current parameter 
        
                   color: line color for the cdf. 
        
                   alpha: line transparency. 
        
                   show_ylabel: whether to show y-label "counts". 
        
                   show_legend: whether to show the legend, e.g., when comparing multiple ranks. 
        
                   num_ticks: number of ticks on the x-axis. 
        
                   legend_kwargs: kwargs for the legend. 
        
               """

The new "ecdf" plot type adds ~60 lines of plotting logic (_plot_ranks_as_ecdf, _plot_ecdf_region_expected_under_uniformity) with no corresponding test. Existing test_sbc_rank_plot parametrizes over ("hist", "cdf") and is not extended here.

sbi/sbi/analysis/plot.py

Lines 2028 to 2087 in 9f1a6f1

    
           def _plot_ranks_as_ecdf( 
        
               ranks: np.ndarray, 
        
               num_bins: int, 
        
               num_repeats: int, 
        
               num_sbc_runs: int, 
        
               ranks_label: Optional[str] = None, 
        
               xlabel: Optional[str] = None, 
        
               color: Optional[str] = None, 
        
               alpha: float = 0.8, 
        
               show_ylabel: bool = True, 
        
               num_ticks: int = 3, 
        
           ) -> None: 
        
               """Plot ranks as a delta of the empirical CDFs to the expected CDF 
        
               Args: 
        
                   ranks: SBC ranks in shape (num_sbc_runs, ) 
        
                   num_bins: number of bins for the histogram, recommendation is num_sbc_runs / 20. 
        
                   num_repeats: number of repeats of each CDF step, i.e., resolution of the eCDF. 
        
                   ranks_label: label for the ranks, e.g., when comparing ranks of different 
        
                       methods. 
        
                   xlabel: label for the current parameter 
        
                   color: line color for the cdf. 
        
                   alpha: line transparency. 
        
                   show_ylabel: whether to show y-label "counts". 
        
                   show_legend: whether to show the legend, e.g., when comparing multiple ranks. 
        
                   num_ticks: number of ticks on the x-axis. 
        
                   legend_kwargs: kwargs for the legend. 
        
               """ 
        
               # Construct uniform histogram. 
        
               uni_bins = binom(num_sbc_runs, p=1 / (num_bins)).ppf(0.5) * np.ones(num_bins) 
        
               uni_bins_cdf = uni_bins.cumsum() / uni_bins.sum() 
        
               # Compute the mean to substract to all cdfs 
        
               means = [binom(num_sbc_runs, p=p).ppf(0.5) for p in uni_bins_cdf] 
        
               means_norm = means / np.max(means) 
        
               # Generate histogram of ranks. 
        
               hist, *_ = np.histogram(ranks, bins=num_bins, density=False) 
        
               # Construct empirical CDF, don't include last bin because it is 1 by default 
        
               histcs = hist.cumsum() 
        
               histcs_norm = histcs / histcs.max() 
        
               # Plot cdf and repeat each stair step 
        
               plt.plot( 
        
                   np.linspace(0, 1, num_repeats * (num_bins - 1)), 
        
                   np.repeat(histcs_norm[:-1] - means_norm[:-1], num_repeats), 
        
                   label=ranks_label, 
        
                   color=color, 
        
                   alpha=alpha, 
        
               ) 
        
               if show_ylabel: 
        
                   plt.ylabel("empirical CDF - expected CDF") 
        
               plt.xlim(0, 1) 
        
               plt.xticks(np.linspace(0, 1, num_ticks)) 
        
               plt.xlabel("posterior rank" if xlabel is None else xlabel)

🤖 Generated with Claude Code

_{- If this code review was useful, please react with 👍. Otherwise, react with 👎.}

janfb

looks good, thanks @MolinAlexei!

Please check the automated PR comment above, plus a couple of suggestions inline below, e.g., I suggest to add a ylim as the current default view is a bit too close.

janfb · 2026-04-18T19:55:58Z

+    plot_types = ["hist", "cdf", "ecdf"]
    assert plot_type in plot_types, (
        "plot type {plot_type} not implemented, use one in {plot_types}."
    )


ecdf as third option is a bit misleading, I suggest calling it cdf-diff.

Also, can you please change the assert into an if-else that raises a ValueError if the plot_type is wrong?

janfb · 2026-04-18T20:20:44Z

    colors: Optional[List[str]] = None,
    fig: Optional[Figure] = None,
    ax: Optional[Axes] = None,
    figsize: Optional[tuple] = None,


add ylim kwarg here (see comment below), set the default: ylim: Optional[tuple[float, float]] = (-0.1, 0.1)

please add the ylim arg here explicitly, that's the user facing level. in the docstring say that it applies to the cdf-diff only

here is where we set the default. the downstream functions then require the ylim and have no defaut.

…e cdf-diff plot, and corrected a mistake in the docstring for the plotting function

MolinAlexei · 2026-04-20T14:33:46Z

Hi, I implemented all requested changes in a new commit, let me know if that works for you

janfb

thanks for the updates @MolinAlexei !

I had another look and realized it would be better to set the default at the user-facing function and remove the default in the low-lever private functions. Can you please accept my suggestions and implement the comments? Thanks 🙏

janfb · 2026-04-25T14:09:30Z

+    if ylim is not None:
+        plt.ylim(ylim)
+    else:
+        plt.ylim(-0.125, 0.125)
+
+    plt.xlim(0, 1)


Suggested change

if ylim is not None:

plt.ylim(ylim)

else:

plt.ylim(-0.125, 0.125)

plt.xlim(0, 1)

plt.ylim(*ylim)

plt.xlim(0, 1)

janfb · 2026-04-25T14:15:47Z

            obtained from different methods.
        num_bins: number of bins used for binning the ranks, default is
            num_sbc_runs / 20.
        plot_type: type of SBC plot, histograms ("hist") or empirical cdfs ("cdf").


please adapt the docstring here as well. please check for other locations that might need updating.

janfb · 2026-04-25T14:16:57Z

+    num_ticks: int = 3,
+    ylim: Optional[tuple[float, float]] = None,
+) -> None:
+    """Plot ranks as a delta of the empirical CDFs to the expected CDF


Suggested change

"""Plot ranks as a delta of the empirical CDFs to the expected CDF

"""Plot ranks as a difference of the empirical to the expected CDF

janfb · 2026-04-25T14:17:24Z

    plt.xlabel("posterior rank" if xlabel is None else xlabel)


+def _plot_ranks_as_ecdf(


I suggest to rename this helper funcation to _plot_ranks_as_cdf_diff

janfb · 2026-04-25T14:18:34Z

    num_cols: int = 4,
    params_in_subplots: bool = False,
    show_ylabel: bool = False,
+    ylim: Optional[Tuple[float, float]] = (-0.125, 0.125),


we should actually make this required here and have the default at the top, user facing level:

Suggested change

ylim: Optional[Tuple[float, float]] = (-0.125, 0.125),

ylim: Tuple[float, float],

janfb · 2026-04-25T14:28:49Z

+    alpha: float = 0.8,
+    show_ylabel: bool = True,
+    num_ticks: int = 3,
+    ylim: Optional[tuple[float, float]] = None,


we should actually make this required here and have the default at the top, user facing level:

Suggested change

ylim: Optional[tuple[float, float]] = None,

ylim: Tuple[float, float],

then we don't need the if-else anymore, see below.

janfb · 2026-04-25T14:30:20Z

        params_in_subplots: whether to show each parameter in a separate subplot, or
            all in one.
        show_ylabel: whether to show ylabels and ticks.
+        ylim: limits on the y-axis


add, that this applies to the cdf-diff case only

janfb · 2026-04-25T14:31:09Z

    colors: Optional[List[str]] = None,
    fig: Optional[Figure] = None,
    ax: Optional[Axes] = None,
    figsize: Optional[tuple] = None,


please add the ylim arg here explicitly, that's the user facing level. in the docstring say that it applies to the cdf-diff only

janfb · 2026-04-25T14:31:49Z

    colors: Optional[List[str]] = None,
    fig: Optional[Figure] = None,
    ax: Optional[Axes] = None,
    figsize: Optional[tuple] = None,


here is where we set the default. the downstream functions then require the ylim and have no defaut.

Add eCDF plot to SBC plots

9f1a6f1

janfb requested changes Apr 18, 2026

View reviewed changes

Added option to put an ylim on the cdf-diff plot, added a test for th…

fe4ff88

…e cdf-diff plot, and corrected a mistake in the docstring for the plotting function

MolinAlexei requested a review from janfb April 20, 2026 14:33

janfb requested changes Apr 25, 2026

View reviewed changes

	"""Plot ranks as a delta of the empirical CDFs to the expected CDF
	"""Plot ranks as a difference of the empirical to the expected CDF

		plt.xlabel("posterior rank" if xlabel is None else xlabel)


		def _plot_ranks_as_ecdf(

	ylim: Optional[Tuple[float, float]] = (-0.125, 0.125),
	ylim: Tuple[float, float],

	ylim: Optional[tuple[float, float]] = None,
	ylim: Tuple[float, float],

Conversation

MolinAlexei commented Apr 13, 2026

Uh oh!

MolinAlexei commented Apr 13, 2026

Uh oh!

janfb commented Apr 16, 2026

Uh oh!

codecov Bot commented Apr 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

janfb commented Apr 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Code review

Uh oh!

janfb left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

MolinAlexei commented Apr 20, 2026

Uh oh!

janfb left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

codecov Bot commented Apr 16, 2026 •

edited

Loading

janfb commented Apr 18, 2026 •

edited

Loading