SNOW-2380741: Use all logical cores in snowpark tests. (#3824)

Formerly, we had limited the test parallelism to not overwhelm our Snowflake deployments, but now that we are using multi-cluster warehouses, we should be able to use all 64 cores.

This change speeds up Modin CI jobs according to a sample size of 1. Other long-running CI jobs seem to take about as long before and after. [before](https://github.com/snowflakedb/snowpark-python/actions/runs/18150423863) vs [after](https://github.com/snowflakedb/snowpark-python/actions/runs/18150450772?pr=3824):

<table>
<tr>
 <td> test
 <td> before (minutes)
 <td> after (minutes)
<tr>
 <td> modin-macos-latest-3.9-aws
 <td> 30
 <td> 28
<tr>
 <td>  modin-ubuntu-latest-64-cores-3.10-azure
 <td> 42
 <td> 31
<tr>
 <td>  modin-ubuntu-latest-64-cores-3.12-aws
 <td> 48
 <td> 30
<tr>
 <td>  modin-windows-latest-64-cores-3.11-gcp
 <td> 71
 <td> 42
</table>

Signed-off-by: sfc-gh-mvashishtha <mahesh.vashishtha@snowflake.com>

Author: sfc-gh-mvashishtha

tox.ini

@@ -62,8 +62,14 @@ setenv =
     COVERAGE_FILE = {env:COVERAGE_FILE:{toxworkdir}/.coverage.{envname}}
     ci: SNOWFLAKE_PYTEST_VERBOSITY = -vvv
     # Do not run doctests in parallel so coverage works
-    # Snowpark uses 36 workers to accelerate testing in merge gate
-    !doctest: SNOWFLAKE_PYTEST_PARALLELISM = -n 36
+    # Snowpark uses all available logical cores to accelerate testing in merge gate.
+    # Note that using all available physical cores with `-n auto` seems to
+    # spawn fewer than 64 pytest processes on the 64-core github runners.
+    # For AST tests, use just 36 cores, because using more cores sometimes
+    # causes the test job to fail with exit code 143, which supposedly means
+    # that it uses too many machine resources.
+    !doctest-!ast: SNOWFLAKE_PYTEST_PARALLELISM = -n logical
+    ast: SNOWFLAKE_PYTEST_PARALLELISM = -n 36
     # Snowpark uses 4 workers for daily testing since some of its test jobs use weak MacOS instances.
     !doctest: SNOWFLAKE_PYTEST_DAILY_PARALLELISM = -n 6
     # Set test type, either notset, unit, integ, or both
@@ -129,7 +135,7 @@ commands =
     # is difficult.
     snowparkpandasnotdoctest: bash scripts/run_snowparkpandasnotdoctest.sh {env:SNOW_1314507_WORKAROUND_RERUN_FLAGS} {posargs:}
     # This one only run doctest but we still need to include the tests folder to let tests/conftest.py to mark the doctest files for us
-    snowparkpandasdoctest: {env:MODIN_PYTEST_CMD} --durations=20 -m "{env:SNOWFLAKE_TEST_TYPE}" {posargs:} src/snowflake/snowpark/modin/ tests/unit/modin
+    snowparkpandasdoctest: {env:MODIN_PYTEST_CMD} --durations=20 -m "{env:SNOWFLAKE_TEST_TYPE}" {posargs:} {env:SNOW_1314507_WORKAROUND_RERUN_FLAGS} src/snowflake/snowpark/modin/ tests/unit/modin
     # This one is used by daily_modin_precommit_py39_py310.yml
     snowparkpandasdailynotdoctest: {env:MODIN_PYTEST_DAILY_CMD} --durations=20 -m "{env:SNOWFLAKE_TEST_TYPE}" {posargs:} {env:SNOW_1314507_WORKAROUND_RERUN_FLAGS} tests/unit/modin tests/integ/modin tests/integ/test_df_to_snowpark_pandas.py
     # This one is only called by jenkins job and the only difference from `snowparkpandasnotdoctest` is that it uses