Skip to content

api.md

Latest commit

 

History

History
290 lines (218 loc) · 6.18 KB

api.md

File metadata and controls

290 lines (218 loc) · 6.18 KB

API reference

yt_framework symbols below are generated from Python docstrings. Job-side entrypoints (ytjobs) have a separate page.

**Navigation**

Use the sidebar headings for modules. Anything not listed is still in the source tree; use `help(module)` locally.

Layout

  • Core — pipeline, stage, registry, discovery, self.deps injection
  • Contracts — StageDependencies and StageContext shared by core and operation drivers (prefer yt_framework.contracts; operations.stage_contracts is a thin re-export)
  • Operations — map, vanilla, map-reduce/reduce, S3 helpers, table helpers, checkpoint upload, sort, tokenizer artifact wiring
  • Typed jobs — StageBootstrapTypedJob
  • YT — yt.support (shared runtime helpers), yt.clients (public client API and mixins), and yt entry (factory, package exports)
  • Utils — env files, logging setup, ignore patterns
  • Packaging — upload helpers shared by operations
  • ytjobs — Job-side reference
**Narrative docs**

How-to guides under `docs/operations`, `docs/configuration`, and `docs/advanced` spell out YAML and stage patterns; this page is the raw API surface.

Core Modules

Pipeline

.. automodule:: yt_framework.core.pipeline
   :members:
   :undoc-members:
   :show-inheritance:

Pipeline config helpers

.. automodule:: yt_framework.core.pipeline_config
   :members:
   :undoc-members:
   :show-inheritance:

Pipeline CLI helpers

.. automodule:: yt_framework.core.pipeline_cli
   :members:
   :undoc-members:
   :show-inheritance:

Stage

.. automodule:: yt_framework.core.stage
   :members:
   :exclude-members: StageContext
   :undoc-members:
   :show-inheritance:

Registry

.. automodule:: yt_framework.core.registry
   :members:
   :undoc-members:
   :show-inheritance:

Discovery

.. automodule:: yt_framework.core.discovery
   :members:
   :undoc-members:
   :show-inheritance:

Core injection (self.deps)

.. automodule:: yt_framework.core.dependencies
   :members:
   :undoc-members:
   :show-inheritance:

Contracts

Stage injection types

.. automodule:: yt_framework.contracts.stage
   :members:
   :undoc-members:
   :show-inheritance:

Import path note

yt_framework.operations.stage_contracts re-exports :class:~yt_framework.contracts.stage.StageContext and :class:~yt_framework.contracts.stage.StageDependencies only. Prefer from yt_framework.contracts import ….

Operations

Map Operations

.. automodule:: yt_framework.operations.command_ops.map
   :members:
   :undoc-members:
   :show-inheritance:

Vanilla Operations

.. automodule:: yt_framework.operations.command_ops.vanilla
   :members:
   :undoc-members:
   :show-inheritance:

Map-reduce and reduce

.. automodule:: yt_framework.operations.command_ops.map_reduce
   :members:
   :undoc-members:
   :show-inheritance:

YQL Operations

YQL operations are *_request methods on the YT client, each taking a frozen request type from yt_framework.yt.clients.yql.yql_requests (for example JoinTablesRequest). See :doc:../operations/yql.

**YQL Operations Location**

YQL operations are implemented as methods on `BaseYTClient` and its subclasses (`YTDevClient` and `YTProdClient`). They are not in a separate operations module.

S3 Operations

.. automodule:: yt_framework.operations.s3
   :members:
   :undoc-members:
   :show-inheritance:

Table operations

.. automodule:: yt_framework.operations.table
   :members:
   :undoc-members:
   :show-inheritance:

Checkpoint Operations

.. automodule:: yt_framework.operations.checkpoint
   :members:
   :undoc-members:
   :show-inheritance:

Sort operations

.. automodule:: yt_framework.operations.command_ops.sort
   :members:
   :undoc-members:
   :show-inheritance:

Tokenizer artifact

.. automodule:: yt_framework.operations._internal.tokenizer_artifact
   :members:
   :undoc-members:
   :show-inheritance:
   :exclude-members: _tar_directory, _prepare_local_archive

Typed jobs

.. automodule:: yt_framework.typed_jobs
   :members:
   :undoc-members:
   :show-inheritance:

Utilities

Environment

.. automodule:: yt_framework.utils.env
   :members:
   :undoc-members:
   :show-inheritance:

Logging

.. automodule:: yt_framework.utils.logging
   :members:
   :undoc-members:
   :show-inheritance:

Ignore Patterns

.. automodule:: yt_framework.utils.ignore
   :members:
   :undoc-members:
   :show-inheritance:

Packaging and operation helpers (advanced)

These modules support code upload, file dependencies, and environment wiring. Most pipelines rely on defaults; use the API when extending the framework.

Upload and local build

.. automodule:: yt_framework.operations.upload
   :members: upload_all_code, build_code_locally, create_code_archive, upload_code_archive
   :show-inheritance:

Stage dependency file lists

.. automodule:: yt_framework.operations.dependencies
   :members: build_stage_dependencies, build_ytjobs_dependencies, build_map_dependencies, build_vanilla_dependencies, add_checkpoint
   :show-inheritance:

Shared operation utilities

.. automodule:: yt_framework.operations.common
   :members: build_environment, prepare_docker_auth, extract_operation_resources, collect_passthrough_kwargs, build_operation_environment, docker_auth_from_op_config, extract_max_failed_jobs
   :show-inheritance:

YT Client

Client Factory

.. automodule:: yt_framework.yt.factory
   :members:
   :undoc-members:
   :show-inheritance:

Dev Client

.. automodule:: yt_framework.yt.clients.client_dev
   :members:
   :undoc-members:
   :show-inheritance:

Production Client

.. automodule:: yt_framework.yt.clients.client_prod
   :members:
   :undoc-members:
   :show-inheritance: