Skip to content

chore(ci): optimize kokoro system tests with concurrency #17444

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
ohmayr wants to merge 3 commits into
base: main
Choose a base branch
from
Open

chore(ci): optimize kokoro system tests with concurrency #17444

Changes from 1 commit
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
4f00dae
chore(ci): optimize kokoro system tests with concurrency
ohmayr Jun 12, 2026
c20eedc
fix filename issue
ohmayr Jun 12, 2026
5ffa37d
chore(ci): optimize parallel system test execution and fix race condi…
chalmerlowe Jul 16, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 18 additions & 2 deletions .kokoro/system.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -140,10 +140,26 @@ for path in `find 'packages' \
set -e

if [[ "${package_modified}" -gt 0 || "$KOKORO_BUILD_ARTIFACTS_SUBDIR" == *"continuous"* ]]; then
# Call the function - its internal exports won't affect the next loop
run_package_test "$package_name" || RETVAL=$?
PACKAGES_TO_TEST="${PACKAGES_TO_TEST} ${package_name}"
else
echo "No changes in ${package_name} and not a continuous build, skipping."
fi
done

if [ -n "$PACKAGES_TO_TEST" ]; then
mkdir -p .logs
export -f run_package_test
export system_test_script PROJECT_ROOT KOKORO_GFILE_DIR

echo "$PACKAGES_TO_TEST" | xargs -n 1 -P 8 -I {} bash -c 'run_package_test "{}" > ".logs/{}.log" 2>&1 || touch ".logs/{}.failed"'

@gemini-code-assist gemini-code-assist Bot Jun 12, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

Using xargs -I {} changes the delimiter to newline only, meaning it ignores spaces and treats the entire line as a single item. Since echo "$PACKAGES_TO_TEST" outputs all packages on a single space-separated line, xargs will execute the command only once, passing the entire string of packages as a single argument (e.g., run_package_test "pkg1 pkg2 pkg3"). This will cause the tests to fail.

To fix this, output each package on a new line using printf "%s\n" $PACKAGES_TO_TEST (unquoted so that it splits on whitespace) and remove the redundant -n 1 flag.

Suggested change
echo "$PACKAGES_TO_TEST" | xargs -n 1 -P 8 -I {} bash -c 'run_package_test "{}" > ".logs/{}.log" 2>&1 || touch ".logs/{}.failed"'
printf "%s\n" $PACKAGES_TO_TEST | xargs -P 8 -I {} bash -c 'run_package_test "{}" > ".logs/{}.log" 2>&1 || touch ".logs/{}.failed"'


for failed in .logs/*.failed; do
if [ -f "$failed" ]; then
echo "--- FAILED: ${failed%.failed} ---"
cat "${failed%.failed}.log"
RETVAL=1
fi
done
fi

exit ${RETVAL}
Loading