Skip to content

yt-cluster-integration.md

Latest commit

 

History

History
78 lines (50 loc) · 4.45 KB

yt-cluster-integration.md

File metadata and controls

78 lines (50 loc) · 4.45 KB

Real cluster integration tests

These tests call a live YTsaurus cell through YTProdClient and yt.wrapper. They live under tests/integration/yt_cluster/ and use the pytest marker yt_cluster.

The same collection skip rules apply to tests/integration/examples_cluster/ (subprocess checks for prod-oriented demos). Both packages are skipped when CI=true or when YT_PROXY/YT_TOKEN are not available from the usual cluster test env files.

When pytest collects them

pytest loads those directories only if both YT_PROXY and YT_TOKEN are set after this lookup:

  1. YT_FRAMEWORK_CLUSTER_TEST_ENV — path to a KEY=value file (same format as configs/secrets.env).
  2. Otherwise, if yt-cluster-test.env exists at the repository root, that file is read.
  3. Values from the process environment override the file for the keys listed in the example file.

If the pair is missing, those packages are skipped at collection time. On CI hosts, CI=true also forces that skip. GitHub Actions additionally runs pytest -m "not yt_cluster", so real-cluster tests never run in CI even if credentials were present by mistake.

Setup

Copy the template and fill in real values (the real file is gitignored):

cp yt-cluster-test.example.env yt-cluster-test.env

Never commit yt-cluster-test.env. You can instead export YT_PROXY and YT_TOKEN in the shell or in CI secrets.

Run

From the repo root, using the project Conda env:

conda run -n yt-framework -- pytest -m yt_cluster -xvs

Narrower:

conda run -n yt-framework -- pytest tests/integration/yt_cluster/test_yql.py -xvs

(example-pipelines-cluster)=

Example pipelines on a real cell

Optional subprocess tests under tests/integration/examples_cluster/ exercise the prod-mode demos 06_s3_integration and 07_custom_docker. They reuse the same yt-cluster-test.env (or YT_FRAMEWORK_CLUSTER_TEST_ENV) as the library’s cluster tests: credentials are merged into a temp copy of the example so we never require committing secrets.env inside examples/.

S3 listing (06_s3_integration)

  1. Set YT_FRAMEWORK_EXAMPLE_S3=1.
  2. Put S3_ENDPOINT, S3_DOWNLOAD_ACCESS_KEY, and S3_DOWNLOAD_SECRET_KEY in yt-cluster-test.env and/or examples/06_s3_integration/configs/secrets.env.
  3. Fill client.input_bucket and client.input_prefix in examples/06_s3_integration/stages/list_s3/config.yaml for a bucket you may read.

Custom Docker (07_custom_docker)

  1. Set YT_FRAMEWORK_EXAMPLE_DOCKER=1.
  2. Ensure the image referenced in that example’s stage config is pullable on your cell.

Run only those tests:

conda run -n yt-framework -- pytest tests/integration/examples_cluster/ -m yt_cluster -xvs

Details on env files: Example pipelines.

What the tests assume

  • Cypress namespace: //tmp/yt-framework/testing/<session_id>/… for tables and files. The session fixture creates that node and deletes it recursively after the run.
  • Host scratch: /tmp/yt-framework/testing/<session_id>/… for local files used in upload_file / upload_directory. It is removed when the session ends.
  • Docker: map, map-reduce, reduce, and vanilla jobs do not pass docker_image in OperationResources, so the cell’s default job image is used. The scripts call python3 and trivial shell (true). Secure-env command wrapping also relies on python3 and bash. If a job fails with “command not found”, your default image may differ; adjust the command in the test to match the image, not the framework.
  • YQL helpers on the client call run_yql with pool name default inside the library. Map/sort operations use YT_TEST_POOL (default default) when the test passes op_resources.pool.
  • Optional knobs in the env file: YT_TEST_POOL, YT_TEST_POOL_TREE, YT_TEST_MEMORY_GB, YT_TEST_CPU_LIMIT, YT_TEST_JOB_COUNT.
  • Map extras (test_map_operations.py): mapper env propagation, a second run_map with append=True, the job= alias (no command), two Cypress file dependencies, forwarded options such as title, max_row_weight, and max_failed_jobs, and a check that sensitive env keys are not present in the operation’s plain mapper.environment while jobs still read them at runtime (via secure_vault + promotion).

See also