Skip to content

chore/monorepos #244

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
bwalsh wants to merge 2 commits into
base: development
Choose a base branch
from
Open

chore/monorepos #244

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
6b7fe0a
adds target to generate 100s of files
bwalsh Jul 2, 2026
21db2c2
removes lfs dependencies
bwalsh Jul 2, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 8 additions & 3 deletions tests/monorepos/Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ GEN := generate-fixtures
GF_SRC := ./generate-fixtures.go
OUTDIR := fixtures

.PHONY: test-monorepos build-generate-fixtures prepare-repo generate-fixtures-run finalize clean
.PHONY: test-monorepos build-generate-fixtures prepare-repo generate-fixtures-run finalize clean generate-large-fixtures-run

test-monorepos: build-generate-fixtures prepare-repo generate-fixtures-run finalize
@echo "test-monorepos complete: $(OUTDIR)"
Expand All @@ -24,12 +24,17 @@ prepare-repo:
# @echo " OK"

generate-fixtures-run:
@echo "Generating fixtures into $(OUTDIR)"
@echo "Generating small set of fixtures into $(OUTDIR)"
# Provide TCGA list; change or pipe your own list to the binary as needed.
# @cd $(OUTDIR); printf "TARGET-ALL-P2\nTCGA-GBM\nTCGA-HNSC\nTCGA-KICH\nTCGA-KIRC\nTCGA-KIRP\nTCGA-LAML\nTCGA-LGG\nTCGA-LIHC\nTCGA-LUAD\nTCGA-LUSC\nTCGA-MESO\nTCGA-OV\nTCGA-PAAD\nTCGA-PCPG\nTCGA-PRAD\nTCGA-READ\nTCGA-SARC\nTCGA-SKCM\nTCGA-STAD\nTCGA-TGCT\nTCGA-THCA\nTCGA-THYM\nTCGA-UVM\nWCDT-MCRPC\n" | ../$(GEN) --number-of-subdirectories=3 --number-of-files=100 -- > /dev/null
@cd $(OUTDIR); printf "TARGET-ALL-P2\n" | ../$(GEN) --number-of-subdirectories=1 --number-of-files=2 -- > /dev/null
@echo " OK"

generate-large-fixtures-run:
@echo "Generating large set of fixtures into $(OUTDIR)"
# Provide TCGA list; change or pipe your own list to the binary as needed.
@cd $(OUTDIR); printf "TARGET-ALL-P2\nTCGA-GBM\nTCGA-HNSC\nTCGA-KICH\nTCGA-KIRC\nTCGA-KIRP\nTCGA-LAML\nTCGA-LGG\nTCGA-LIHC\nTCGA-LUAD\nTCGA-LUSC\nTCGA-MESO\nTCGA-OV\nTCGA-PAAD\nTCGA-PCPG\nTCGA-PRAD\nTCGA-READ\nTCGA-SARC\nTCGA-SKCM\nTCGA-STAD\nTCGA-TGCT\nTCGA-THCA\nTCGA-THYM\nTCGA-UVM\nWCDT-MCRPC\n" | ../$(GEN) --number-of-subdirectories=3 --number-of-files=100 -- > /dev/null
@echo " OK"

finalize:
@echo " OK Fixtures created at $(OUTDIR)"

Expand Down
59 changes: 15 additions & 44 deletions tests/monorepos/run-test.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -135,31 +135,6 @@ if [ ! -d "fixtures" ]; then
exit 1
fi

# ensure git-lfs is installed
if ! command -v git-lfs >/dev/null 2>&1; then
echo "error: git-lfs is not installed; please install it to proceed" >&2
# Example install command:
if [ "$(uname -s)" = "Darwin" ]; then
if ! command -v brew >/dev/null 2>&1; then
echo "error: Homebrew is not installed. Please install Homebrew first:" >&2
echo " https://brew.sh" >&2
exit 1
fi
echo "installing git-lfs via Homebrew on macOS" >&2
if ! brew install git-lfs; then
echo "error: failed to install git-lfs via Homebrew" >&2
exit 1
fi
if ! git lfs install --skip-smudge; then
echo "error: failed to initialize git-lfs" >&2
exit 1
fi
else
echo "See installation instructions for your platform:" >&2
echo " https://github.com/git-lfs/git-lfs/wiki/Installation" >&2
exit 1
fi
fi

# ensure git-drs is running from this project's build
# run `which git-drs` and check if it's in the build directory
Expand All @@ -177,7 +152,7 @@ echo "Using git-drs from: $(which git-drs)" >&2


# ensure a gen3 project exists
calypr_admin projects ls --profile "$PROFILE" | grep "/programs/$PROGRAM/projects/$PROJECT" >/dev/null 2>&1 || {
calypr-cli auth --profile=$PROFILE --all | grep "/programs/$PROGRAM/projects/$PROJECT" >/dev/null 2>&1 || {
echo "error: /programs/$PROGRAM/projects/$PROJECT does not exist; please create it first" >&2
exit 1
}
Expand All @@ -199,7 +174,7 @@ if [ "$CLONE" = "true" ]; then
cd ../clone
echo "Cloning remote repository into ../clone" >&2
# clone into current directory
if ! git clone "$GIT_REMOTE" .; then
if ! GIT_LFS_SKIP_SMUDGE=1 git clone --branch main "$GIT_REMOTE" .; then
echo "error: git clone failed" >&2
exit 1
fi
Expand All @@ -211,8 +186,8 @@ if [ "$CLONE" = "true" ]; then
fi
echo "Pulling LFS objects from remote" >&2
git drs init
git drs remote add gen3 "$PROFILE" "$PROGRAM/$PROJECT" --cred "$CREDENTIALS_PATH"
git lfs pull origin main
git drs remote add gen3 origin "$PROGRAM/$PROJECT" --cred "$CREDENTIALS_PATH"
git drs pull origin main

@chatgpt-codex-connector chatgpt-codex-connector Bot Jul 2, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Drop the branch argument from git drs pull

When --clone is used, this always exits before hydration because git drs pull does not accept the branch argument carried over from git lfs pull; I checked cmd/pull/main.go, which rejects more than one arg, and docs/commands.md documents git drs pull as hydration-only. Since this script has set -e, git drs pull origin main aborts the clone validation path instead of hydrating the pointers.

Useful? React with 👍 / 👎.

@bwalsh bwalsh Jul 2, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@matthewpeterkort Can you confirm this behavior?

if grep -q 'https://git-lfs.github.com/spec/v1' ./TARGET-ALL-P2/sub-directory-1/*file-0001.dat; then
echo "error: LFS pointer resolved and data in `TARGET-ALL-P2/sub-directory-1/file-0001.dat`" >&2
exit 1
Expand All @@ -234,8 +209,8 @@ else
git remote add origin "$GIT_REMOTE"

# Initialize drs configuration for this repo
git drs init -t 16
git drs remote add gen3 "$PROFILE" "$PROGRAM/$PROJECT" --cred "$CREDENTIALS_PATH"
git drs init -t 16 --enable-data-client-logs
git drs remote add gen3 origin "$PROGRAM/$PROJECT" --cred "$CREDENTIALS_PATH"
# Set multipart-threshold to 10 (MB) for testing purposes
# Using a smaller threshold to force a multipart upload for testing
# default is 500 (MB)
Expand All @@ -256,8 +231,8 @@ else
fi

# Ensure enable-data-client-logs is present in git config
if ! git config --list | grep -q -- 'enable-data-client-logs'; then
echo "error: git config key 'enable-data-client-logs' not found; please set it before running tests" >&2
if ! git config --local --get drs.enable-data-client-logs >/dev/null; then
echo "error: git config key 'drs.enable-data-client-logs' not found after git drs init" >&2
exit 1
fi

Expand Down Expand Up @@ -303,7 +278,7 @@ for dir in */ ; do
continue
fi
# $dir has trailing slash; don't need trailing slash in track
git lfs track "$dir**"
git drs track "$dir**"
git add "$dir"
git commit -am "Add $dir" 2>&1 | tee commit.log
cat commit.log >> commit-aggregate.log
Expand All @@ -322,10 +297,6 @@ for dir in */ ; do
echo "# Last 3 lines of .drs/lfs/objects tree:" >> lfs-console.log
tree .drs/lfs/objects | tail -3 >> lfs-console.log
fi
echo "# git lfs status:" >> lfs-console.log
git lfs status >> lfs-console.log
echo "# Number of LFS files to be pushed in dry-run:" >> lfs-console.log
git lfs push --dry-run origin main | wc -l >> lfs-console.log
echo "##########################################" >> lfs-console.log
cat lfs-console.log >> lfs-console-aggregate.log

Expand All @@ -334,15 +305,15 @@ for dir in */ ; do
#

# use the first file found in the directory for testing, that will be a single part file
target_file=$(find "$dir" -type f -name '*file-0001.dat')
target_file=$(find "$dir" -type f -name '*file-0001.dat' | head -n 1)
# strip double slashes from path if any
target_file=${target_file//\/\//\/}
if [ -z "$target_file" ]; then
echo "error: no files found in $dir to test content/path changes" >&2
exit 1
fi

original_oid=$(git lfs ls-files -l | awk -v path="$target_file" '$0 ~ (" " path "$") {print $1; exit}')
original_oid=$(git drs ls-files -l | awk -v path="$target_file" '$0 ~ (" " path "$") {print $1; exit}')
if [ -z "$original_oid" ]; then
echo "error: unable to find LFS OID for $target_file" >&2
exit 1
Expand All @@ -353,7 +324,7 @@ for dir in */ ; do
git add "$target_file"
git commit -m "Update content for $target_file"

updated_oid=$(git lfs ls-files -l | awk -v path="$target_file" '$0 ~ (" " path "$") {print $1; exit}')
updated_oid=$(git drs ls-files -l | awk -v path="$target_file" '$0 ~ (" " path "$") {print $1; exit}')
if [ -z "$updated_oid" ]; then
echo "error: unable to find updated LFS OID for $target_file" >&2
exit 1
Expand All @@ -371,7 +342,7 @@ for dir in */ ; do
git mv "$target_file" "$renamed_path"
git commit -m "Rename $target_file to $renamed_path"

renamed_oid=$(git lfs ls-files -l | awk -v path="$renamed_path" '$0 ~ (" " path "$") {print $1; exit}')
renamed_oid=$(git drs ls-files -l | awk -v path="$renamed_path" '$0 ~ (" " path "$") {print $1; exit}')
if [ -z "$renamed_oid" ]; then
echo "error: unable to find LFS OID for renamed path $renamed_path" >&2
exit 1
Expand All @@ -380,12 +351,12 @@ for dir in */ ; do
echo "error: expected same OID after rename for $renamed_path" >&2
exit 1
fi
if git lfs ls-files -l | grep -Fq " $target_file"; then
if git drs ls-files -l | grep -Fq " $target_file"; then
echo "error: expected old path $target_file to be absent after rename" >&2
exit 1
fi

git push origin main 2>&1 | tee -a lfs-console.log
break # uncomment for one directory at a time testing
# break # comment for one directory at a time testing
fi
done
Loading