Move from tests/ to tests/integration/ for integration tests (#1240)

We noticed that our Linux builds are regularly hitting "out of disk
space" issues, with CI failures like

```
fatal error: ld terminated with signal 7 [Bus error], core dumped
```

and

```
rustc-LLVM ERROR: IO failure on output stream: No space left on device
```

After some investigation, it turns out that a few things are true:

- GitHub Actions gives us at minimum 20gb of storage space to work with
on a runner (this may become 15gb in the future IIUC)
- `Run Tests` in debug builds, i.e. `cargo nextest run` rather than
`cargo nextest run --release`, required 15gb of storage on the runner
for the test binaries

Combining these two facts together results in us very regularly hitting
the storage limit and getting killed runners.

Here you can see this with some extra debug steps on the runner. Notice
the available storage in `/dev/root/` drop 15gb after `Run Tests` (this
is with a lucky runner that got more then 20gb of storage for its run).

<img width="662" height="630" alt="Screenshot 2026-05-28 at 10 12 52 AM"
src="https://github.com/user-attachments/assets/4504ae7f-5bb0-4757-aa25-e0ba74768b14"
/>

I proved that it was only the debug test binaries causing much issue by
comparing to our one build that runs nextest with `--release`, which did
not show this dramatic reduction in available space

<img width="663" height="670" alt="Screenshot 2026-05-28 at 10 13 35 AM"
src="https://github.com/user-attachments/assets/b884c2e2-f0c4-47dd-b536-91be267560b3"
/>

With more investigation, I was able to determine that all of the size
was coming from our Rust _integration tests_, i.e. the ones in
`ark/tests/` and `amalthea/tests/` and `oak_semantic/tests/`.

**By default in Rust, each individual integration file gets its own test
binary, which is fully linked against its crate (i.e. ark) and all
transitive dependencies (i.e. salsa, biome, tree-sitter, zmq, etc). We
have ~80 integration test files, and on my Mac each one was ~80mb. Holy
shit.**

Luckily, there is an extremely simple solution to this that [matklad
uses](https://matklad.github.io/2021/02/27/delete-cargo-integration-tests.html#Implications).

Rather than this:

```
tests/
  foo.rs
  bar.rs
```

We change to this:

```
tests/
  integration/
    main.rs
    foo.rs
    bar.rs
```

That creates exactly 1 test binary for the crate (i.e. for ark) called
`integration`, collapsing all of this redundancy entirely.

matklad reported

> When a refactoring along these lines was applied to Cargo itself, the
effects were substantial
([numbers](rust-lang/cargo#5022 (comment))).
The time to compile the test suite decreased 3x. The size of on-disk
artifacts decreased 5x.

And that seems to also be useful for us! On my Mac, doing a fresh `just
test` followed by `cargo clean` gives me

Compilation time:

```
# Before
Finished in 1m 28s

# After
Finished in 52.17s
```

On disk storage:

```
# Before
cargo clean
     Removed 21277 files, 8.1GiB total

# After
cargo clean
     Removed 13563 files, 3.5GiB total
```

And on Linux it must be much more substantial!

nextest can handle this single binary just fine, because the
`integration` test binary still exposes a cli that lets you run all of
the tests in there individually in separate processes.

So this whole PR just becomes a rename with no behavior changes, and now
we only get a 2gb increase after `Run Tests` on CI:

<img width="644" height="625" alt="Screenshot 2026-05-28 at 10 55 58 AM"
src="https://github.com/user-attachments/assets/50205e3d-7d38-422b-9bb5-65d30611cfa2"
/>

---

In my investigation I used a `util_free_space.sh` utility from Arrow
that cleans up unused disk space on the runner, giving you more than the
20gb that GHA gives you by default. I've included the modified
`test-linux.yml` here for future us in case we need it again, along with
`scripts/util_free_space.sh`.

<details>

```bash
#!/usr/bin/env bash
#
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements.  See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership.  The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License.  You may obtain a copy of the License at
#
#   http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied.  See the License for the
# specific language governing permissions and limitations
# under the License.

# Pulled from
# https://github.com/apache/arrow/blob/main/ci/scripts/util_free_space.sh

set -eux

df -h
echo "::group::/usr/local/*"
du -hsc /usr/local/*
echo "::endgroup::"
# ~1GB
sudo rm -rf \
  /usr/local/aws-sam-cli \
  /usr/local/julia* || :
echo "::group::/usr/local/bin/*"
du -hsc /usr/local/bin/*
echo "::endgroup::"
# ~1GB (From 1.2GB to 214MB)
sudo rm -rf \
  /usr/local/bin/aliyun \
  /usr/local/bin/azcopy \
  /usr/local/bin/bicep \
  /usr/local/bin/cmake-gui \
  /usr/local/bin/cpack \
  /usr/local/bin/helm \
  /usr/local/bin/hub \
  /usr/local/bin/kubectl \
  /usr/local/bin/minikube \
  /usr/local/bin/node \
  /usr/local/bin/packer \
  /usr/local/bin/pulumi* \
  /usr/local/bin/sam \
  /usr/local/bin/stack \
  /usr/local/bin/terraform || :
# 142M
sudo rm -rf /usr/local/bin/oc || : \
echo "::group::/usr/local/share/*"
du -hsc /usr/local/share/*
echo "::endgroup::"
# 506MB
sudo rm -rf /usr/local/share/chromium || :
# 1.3GB
sudo rm -rf /usr/local/share/powershell || :
echo "::group::/usr/local/lib/*"
du -hsc /usr/local/lib/*
echo "::endgroup::"
# 15GB
sudo rm -rf /usr/local/lib/android || :
# 341MB
sudo rm -rf /usr/local/lib/heroku || :
# 1.2GB
sudo rm -rf /usr/local/lib/node_modules || :
echo "::group::/opt/*"
du -hsc /opt/*
echo "::endgroup::"
# 679MB
sudo rm -rf /opt/az || :
echo "::group::/opt/microsoft/*"
du -hsc /opt/microsoft/*
echo "::endgroup::"
# 197MB
sudo rm -rf /opt/microsoft/powershell || :
echo "::group::/opt/hostedtoolcache/*"
du -hsc /opt/hostedtoolcache/*
echo "::endgroup::"
# 5.3GB
sudo rm -rf /opt/hostedtoolcache/CodeQL || :
# 1.4GB
sudo rm -rf /opt/hostedtoolcache/go || :
# 489MB
sudo rm -rf /opt/hostedtoolcache/PyPy || :
# 376MB
sudo rm -rf /opt/hostedtoolcache/node || :
# Remove Web browser packages
sudo apt purge -y firefox
# google-chrome-stable isn't installed on arm64 image.
sudo apt purge -y google-chrome-stable || :
# microsoft-edge-stable isn't installed on arm64 image.
sudo apt purge -y microsoft-edge-stable || :
df -h
```

```yaml
name: "Test Ark - Linux"

on:
  workflow_call:
  workflow_dispatch:
    inputs:
      ssh:
        description: 'Set up an SSH session before running `cargo nextest run`?'
        type: boolean
        required: true
        default: false

jobs:
  linux:
    runs-on: ubuntu-latest
    name: "Rust: ${{ matrix.config.rust }}${{ matrix.config.flavor == 'release' && ' + release' || '' }}, R: ${{ matrix.config.r }}"
    strategy:
      fail-fast: false
      matrix:
        config:
          # Test on all supported R versions
          # (Particularly important to do this on at least one OS for
          # version dependent R hook override checks, like `utils::View()`)
          - { rust: 'stable',  r: 'release', flavor: 'release' }
          - { rust: 'stable',  r: '4.5' }
          - { rust: 'stable',  r: '4.4' }
          - { rust: 'stable',  r: '4.3' }
          - { rust: 'stable',  r: '4.2' }

          # Builds with `workspace.package.rust-version` (the MSRV) to make sure
          # the project is compatible with the declared minimal Rust version.
          # The stable rows use `rust-toolchain.toml`'s dev toolchain, which can
          # drift ahead, so they wouldn't flag incorrect use of newer features
          # past the MSRV.
          - { rust: 'msrv', r: 'release' }

          # Early warning signal for upstream changes,
          # both in Rust and R-devel
          - { rust: 'nightly',  r: 'devel' }
    timeout-minutes: 40
    env:
      GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
    steps:
      - uses: actions/checkout@v4

      - name: Disk Space After Checkout
        if: always()
        run: df -h

      - name: Update Build Environment
        run: |
          sudo apt-get update

      - name: Disk Space After Update Build Environment
        if: always()
        run: df -h

      - name: Free Up Disk Space
        run: |
          scripts/util_free_space.sh

      - name: Disk Space After Free Up Disk Space
        if: always()
        run: df -h

      - name: Install Nightly Rust
        uses: dtolnay/rust-toolchain@nightly
        if: matrix.config.rust == 'nightly'

      - name: Disk Space After Install Nightly Rust
        if: always()
        run: df -h

      - name: Install MSRV Rust
        if: matrix.config.rust == 'msrv'
        run: |
          msrv=$(python3 ./scripts/print-rust-version.py)
          echo "Using MSRV toolchain: $msrv"
          rustup toolchain install "$msrv" --profile minimal
          echo "RUSTUP_TOOLCHAIN=$msrv" >> "$GITHUB_ENV"

      - name: Disk Space After Install MSRV Rust
        if: always()
        run: df -h

      - name: Report Rust Toolchain
        run: rustup show

      - name: Disk Space After Report Rust Toolchain
        if: always()
        run: df -h

      - name: Rust Cache
        uses: Swatinem/rust-cache@v2
        # Cache isn't useful on nightly, it would be thrown away every day
        if: matrix.config.rust != 'nightly'

      - name: Disk Space After Rust Cache
        if: always()
        run: df -h

      - name: Install nextest
        uses: taiki-e/install-action@v2
        with:
          tool: cargo-nextest

      - name: Disk Space After Install nextest
        if: always()
        run: df -h

      - name: Install R
        uses: r-lib/actions/setup-r@v2
        with:
          r-version: ${{ matrix.config.r }}
          use-public-rspm: true

      - name: Disk Space After Install R
        if: always()
        run: df -h

      - name: Install R Packages Required For Tests
        uses: r-lib/actions/setup-r-dependencies@v2
        with:
          dependencies: '"hard"'
          packages:
            data.table
            dplyr
            haven
            R6
            ragg
            readr
            rstudioapi
            survival
            tibble

      - name: Disk Space After Install R Packages
        if: always()
        run: df -h

      - name: Set LD_LIBRARY_PATH
        run: |
          echo "LD_LIBRARY_PATH=$(R RHOME)/lib" >> $GITHUB_ENV

      - name: Disk Space After Set LD_LIBRARY_PATH
        if: always()
        run: df -h

      - name: Setup SSH access
        uses: mxschmitt/action-tmate@v3
        if: ${{ inputs.ssh }}
        timeout-minutes: 30

      - name: Disk Space After Setup SSH access
        if: always()
        run: df -h

      - name: Build
        run: |
          cargo build ${{ matrix.config.flavor == 'release' && '--release' || '' }}

      - name: Disk Space After Build
        if: always()
        run: df -h

      - name: Run Tests
        env:
          NEXTEST_PROFILE: "ci"
        run: |
          cargo nextest run ${{ matrix.config.flavor == 'release' && '--release' || '' }}

      - name: Disk Space After Run Tests
        if: always()
        run: df -h

```

</details>

Author: DavisVaughan