Skip to content

perf: speed up modified file lookup in large repositories #544

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
cjappl wants to merge 1 commit into
base: main
Choose a base branch
from
+123 −25
Open

perf: speed up modified file lookup in large repositories #544

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
85 changes: 60 additions & 25 deletions bin/git-forgit
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -222,6 +222,61 @@ _forgit_list_files() {
git ls-files -z "$@" "$rootdir" | tr '\0' '\n' | uniq
}

_forgit_rewrite_repo_paths_for_cwd() {
local cwd mode rootdir separator
rootdir=$1
mode=${2:-paths}
separator=${3-}

@sandr01d sandr01d Jul 11, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is the more common version to do this because it also expands to an empty string if the variable is declared but is null. At least that's what I found when looking it up here.

Suggested change
separator=${3-}
separator=${3:-}

cwd=$(pwd -P)
Comment on lines +226 to +230

@sandr01d sandr01d Jul 11, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: I'd put the definition and assignment of the local variables in the same order. I find it easier to read this way.

Suggested change
local cwd mode rootdir separator
rootdir=$1
mode=${2:-paths}
separator=${3-}
cwd=$(pwd -P)
local rootdir mode separator cwd
rootdir=$1
mode=${2:-paths}
separator=${3-}
cwd=$(pwd -P)


perl -MCwd=realpath -MFile::Spec -e '
use strict;
use warnings;

my ($rootdir, $cwd, $mode, $separator) = @ARGV;
my $normalized_rootdir = realpath($rootdir);
my $normalized_cwd = realpath($cwd);

while (my $line = <STDIN>) {
chomp $line;

my ($status, $repo_path);
if ($mode eq q{status_entries}) {
next unless $line =~ /^(..[^[:space:]]*)( )(.*)$/;
($status, $repo_path) = ($1, $3);
} else {
next if $line eq q{};
$repo_path = $line;
}

my $absolute_path = "$normalized_rootdir/$repo_path";
my $display_path = File::Spec->abs2rel(realpath($absolute_path) // $absolute_path, $normalized_cwd);
$display_path = q{.} if $display_path eq q{};

if ($mode eq q{status_entries}) {
print "[$status] ${display_path}${separator}${normalized_rootdir}/${repo_path}\n";
} else {
print "$display_path\n";
}
}
' "$rootdir" "$cwd" "$mode" "$separator"
}

@cjappl cjappl Jul 9, 2026

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This change was largely just pulled from _forgit_build_status_entries below, with a few ergonomic tweaks to make it work for both situations. The new tests should hopefully make sure this is stable


_forgit_list_modified_files() {
# on large repos with many untracked files, git ls-files
# is substantially slower than git diff
# Wherever possible (when we don't care about untracked files)
# use this command over _forgit_list_files

# See note on _forgit_list_files() about -z and other options
local rootdir
rootdir=$(git rev-parse --show-toplevel)
git -C "$rootdir" diff --name-only -z |
tr '\0' '\n' |
uniq |
_forgit_rewrite_repo_paths_for_cwd "$rootdir"
}

_forgit_list_staged_files() {
local up
up="$(git rev-parse --show-cdup)"
Expand Down Expand Up @@ -291,32 +346,12 @@ _forgit_restore_untracked_color() {
# 1. the human-readable status line shown in fzf
# 2. the absolute-path payload used by preview/edit/add actions
_forgit_build_status_entries() {
local cwd rootdir
local rootdir
rootdir=$1
cwd=$(pwd -P)

# Use a single Perl process so path normalization stays portable while the
# full add-list transformation still runs as one batch pipeline stage.
perl -MCwd=realpath -MFile::Spec -e '
use strict;
use warnings;

my ($rootdir, $cwd, $separator) = @ARGV;
my $normalized_rootdir = realpath($rootdir);
my $normalized_cwd = realpath($cwd);

while (my $line = <STDIN>) {
chomp $line;
next unless $line =~ /^(..[^[:space:]]*)( )(.*)$/;

my ($status, $repo_path) = ($1, $3);
my $absolute_path = "$normalized_rootdir/$repo_path";
my $display_path = File::Spec->abs2rel(realpath($absolute_path) // $absolute_path, $normalized_cwd);
$display_path = "." if $display_path eq q{};

print "[$status] ${display_path}${separator}${normalized_rootdir}/${repo_path}\n";
}
' "$rootdir" "$cwd" "$_ffsep"
_forgit_rewrite_repo_paths_for_cwd "$rootdir" status_entries "$_ffsep"
}

_forgit_is_submodule() {
Expand Down Expand Up @@ -739,7 +774,7 @@ _forgit_restore() {
if [[ $worktree == true || $staged != true ]]; then
while IFS='' read -r file; do
candidates+=("$file")
done < <(_forgit_list_files --modified)
done < <(_forgit_list_modified_files)
fi

[[ ${#candidates[@]} -eq 0 ]] && echo "Nothing to restore." && return 1
Expand Down Expand Up @@ -1093,7 +1128,7 @@ _forgit_checkout_file() {
_forgit_git_checkout_file "$@"
return $?
}
[[ $(_forgit_list_files --modified | wc -l) -eq 0 ]] && echo 'Nothing to checkout.' && return 1
[[ $(_forgit_list_modified_files | wc -l) -eq 0 ]] && echo 'Nothing to checkout.' && return 1
opts="
$FORGIT_FZF_DEFAULT_OPTS
-m -0
Expand All @@ -1103,7 +1138,7 @@ _forgit_checkout_file() {
files=()
while IFS='' read -r file; do
files+=("$file")
done < <(_forgit_list_files --modified |
done < <(_forgit_list_modified_files |
FZF_DEFAULT_OPTS="$opts" fzf)
[[ ${#files[@]} -gt 0 ]] && _forgit_git_checkout_file "$@" "${files[@]}"
}
Expand Down
63 changes: 63 additions & 0 deletions tests/working-tree-changes.test.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,28 @@ function test_forgit_worktree_changes_contains_untracked() {
assert_contains "untracked_file.txt" "$output"
}

function test_forgit_list_files_matches_modified_files_until_untracked_files_exist() {
local list_files_output modified_files_output

list_files_output=$(_forgit_list_files --modified)
modified_files_output=$(_forgit_list_modified_files)

assert_same "$modified_files_output" "$list_files_output"
assert_contains "modified file.txt" "$list_files_output"

touch new_untracked_file.txt

list_files_output=$(_forgit_list_files --exclude-standard --modified --others)
modified_files_output=$(_forgit_list_modified_files)

assert_contains "modified file.txt" "$list_files_output"
assert_contains "new_untracked_file.txt" "$list_files_output"
assert_contains "modified file.txt" "$modified_files_output"
assert_not_contains "new_untracked_file.txt" "$modified_files_output"

rm new_untracked_file.txt
}

function test_forgit_worktree_changes_excludes_staged() {
local output

Expand Down Expand Up @@ -114,6 +136,47 @@ function test_forgit_build_status_entries_uses_cwd_relative_display_paths_for_si
assert_contains "../dir1/file.txt" "$output"
}

function test_forgit_list_modified_files_uses_cwd_relative_paths_for_sibling_entries() {
local output

mkdir -p dir1 dir2
touch dir1/file.txt
git add .
git commit -m "add dirs" --quiet
echo modified >dir1/file.txt
cd dir2 || return 1

output=$(_forgit_list_modified_files)

assert_same "../dir1/file.txt" "$output"
}

function test_forgit_rewrite_repo_paths_for_cwd_rewrites_repo_relative_paths() {
local output rootdir

mkdir -p dir1 dir2
touch dir1/file.txt
rootdir=$(git rev-parse --show-toplevel)
cd dir2 || return 1

output=$(printf 'dir1/file.txt\n' | _forgit_rewrite_repo_paths_for_cwd "$rootdir")

assert_same "../dir1/file.txt" "$output"
}

function test_forgit_rewrite_repo_paths_for_cwd_formats_status_entries() {
local output rootdir

mkdir -p dir1 dir2
touch dir1/file.txt
rootdir=$(git rev-parse --show-toplevel)
cd dir2 || return 1

output=$(printf ' M dir1/file.txt\n' | _forgit_rewrite_repo_paths_for_cwd "$rootdir" status_entries "$_ffsep")

assert_same "[ M] ../dir1/file.txt${_ffsep}${rootdir}/dir1/file.txt" "$output"
}

function test_forgit_build_status_entries_uses_cwd_relative_display_paths_from_logical_symlink_paths() {
local output sandbox

Expand Down
Loading