Skip to content

configuration.md

Latest commit

 

History

History
165 lines (109 loc) · 2.89 KB

configuration.md

File metadata and controls

165 lines (109 loc) · 2.89 KB

Configuration Issues

Issues specific to Dev mode and Prod mode configuration.

Dev Mode Issues

Tables not found in dev mode

Error:

FileNotFoundError: .dev/table.jsonl not found

Solution:

  • Check table was created in previous stage
  • Verify table path is correct
  • Check .dev/ directory exists
  • Review previous stage logs

Operation fails in dev mode

Error:

RuntimeError: Map operation failed

Solution:

  • Check mapper script syntax
  • Verify input table exists
  • Review .dev/ directory for logs
  • Check mapper script has if __name__ == "__main__": main() block

Sandbox directory issues

Error:

PermissionError: Cannot create sandbox directory

Solution:

  • Check file permissions on pipeline directory
  • Verify disk space available
  • Check .dev/ directory permissions

DuckDB errors

Error:

Error: DuckDB query failed

Solution:

  • Check YQL to SQL conversion
  • Verify table schemas
  • Review query syntax
  • Check DuckDB version compatibility

Prod Mode Issues

YT credentials not found

Error:

ValueError: secrets are required for prod mode

Solution:

  • Create configs/secrets.env file
  • Add YT_PROXY and YT_TOKEN
  • Verify credentials are correct
  • Check file permissions

Build folder not found

Error:

ValueError: build_folder not found in [pipeline] config section.

Solution:

  • Add build_folder to pipeline config
  • Verify YT path is correct
  • Check YT permissions for build folder
  • Ensure build folder path exists or can be created

Code upload fails

Error:

Error: Failed to upload code to YT

Solution:

  • Check YT credentials and permissions
  • Verify build folder path is correct
  • Check network connectivity
  • Review upload logs

Operation fails on cluster

Error:

RuntimeError: Operation failed on YT cluster

Solution:

  • Check YT web UI for operation details
  • Review operation logs in YT
  • Verify resource limits are sufficient
  • Check operation configuration

YT connection errors

Error:

ConnectionError: Cannot connect to YT cluster

Solution:

  • Verify YT_PROXY URL is correct
  • Check network connectivity
  • Review firewall settings
  • Test YT connection manually

Mode Switching Issues

Dev to Prod migration

Common Issues:

  • Table paths may need adjustment
  • Resource limits may need tuning
  • Checkpoint paths may differ
  • Environment variables may be required

Solution:

  • Test thoroughly in dev mode first
  • Gradually migrate stages
  • Review Dev vs Prod guide
  • Check configuration differences

See Also