Skip to content

Improve ergonomics for regenerating snapshots#7

Merged
comnik merged 6 commits into
RelationalAI-oss:mainfrom
talf301:tf-auto-regenerate
Jan 21, 2026
Merged

Improve ergonomics for regenerating snapshots#7
comnik merged 6 commits into
RelationalAI-oss:mainfrom
talf301:tf-auto-regenerate

Conversation

@talf301

@talf301 talf301 commented Dec 18, 2025

Copy link
Copy Markdown
Collaborator

Currently when we use SnapshotTesting.jl we need to maintain a global variable indicating whether or not we're regenerating snapshots, and then manually set this variable back and forth when we make apparently breaking changes. This PR adopts some of the ergonomics from ReferenceTests.jl, getting rid of this global variable in each test suite and instead providing 2 options:

  • An environment variable JULIA_SNAPSHOTTESTS_UPDATE indicating whether or not we force updates (defaults to false)
  • When run in an interactive terminal, the test will behave in an interactive way. If the test fails, the user is shown the diff and asked if this is correct and they would like to update accordingly

We get rid of the regenerate flag from the API, so this is a breaking change and requires a major version bump.

@comnik comnik requested a review from Copilot January 5, 2026 18:53

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR removes the regenerate parameter from the snapshot testing API and replaces it with two more ergonomic update mechanisms: an environment variable (JULIA_SNAPSHOTTESTS_UPDATE) for automated updates and interactive prompting when running in a terminal.

Key changes:

  • Removed the regenerate parameter from test_snapshot function signature
  • Added force_update() to check the JULIA_SNAPSHOTTESTS_UPDATE environment variable
  • Implemented interactive update prompting via input_bool() when tests fail in interactive mode
  • Auto-creation of missing snapshot directories with informational messages

Reviewed changes

Copilot reviewed 3 out of 4 changed files in this pull request and generated 2 comments.

File Description
src/snapshots.jl Core implementation: removed regenerate parameter, added auto-creation logic, interactive update prompts, and helper functions force_update() and input_bool()
test/test_update_modes.jl New comprehensive test suite covering environment variable behavior, interactive input simulation, auto-creation, and successful snapshot scenarios
test/runtests.jl Added new test file to the test suite execution

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread src/snapshots.jl
Comment thread test/test_update_modes.jl
Comment on lines +38 to +48
# Create a temporary file with the input
mktempdir() do tmpdir
input_file = joinpath(tmpdir, "input.txt")
write(input_file, input_string)

open(input_file, "r") do input_io
redirect_stdin(input_io) do
redirect_stdout(devnull) do
SnapshotTesting.input_bool("Test prompt")
end
end

Copilot AI Jan 5, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The test_input helper function creates a temporary directory and file for each test case, which adds unnecessary overhead. Consider using IOBuffer for simpler and more efficient input simulation: redirect_stdin(IOBuffer(input_string)) would eliminate the need for temporary files.

Suggested change
# Create a temporary file with the input
mktempdir() do tmpdir
input_file = joinpath(tmpdir, "input.txt")
write(input_file, input_string)
open(input_file, "r") do input_io
redirect_stdin(input_io) do
redirect_stdout(devnull) do
SnapshotTesting.input_bool("Test prompt")
end
end
# Use an in-memory IOBuffer to simulate stdin
io = IOBuffer(input_string)
return redirect_stdin(io) do
redirect_stdout(devnull) do
SnapshotTesting.input_bool("Test prompt")

Copilot uses AI. Check for mistakes.

@comnik comnik left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @talf301 ! Sorry for the delay

@talf301

talf301 commented Jan 20, 2026

Copy link
Copy Markdown
Collaborator Author

@NHDaly do i need some extra permissions to merge?

@comnik comnik merged commit aac9fb4 into RelationalAI-oss:main Jan 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants