Skip to content

Commit ca5a417

Browse files
authored
feat(infra): Introduce a variable to control dlt write disposition behaviour for elt tasks (#159)
### Summary Previously the elt_task cron script could only use the default write disposition of the extract_and_load script. This changes introduces an optional environment variable to allow the flag to be overridden. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * ELT tasks now support configurable write disposition through environment variables, enabling control over data writing behaviour during extraction and loading operations. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
1 parent c7be0cb commit ca5a417

1 file changed

Lines changed: 8 additions & 1 deletion

File tree

infra/ansible-docker/playbooks/elt/templates/cron/elt_task.sh.j2

Lines changed: 8 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,7 @@ set -euo pipefail
44
# Constants
55
ELT_SECRETS=${ELT_SECRETS:?}
66
ELT_GIT_REF=${ELT_GIT_REF:-main}
7+
ELT_WRITE_DISPOSITION=${ELT_WRITE_DISPOSITION:-}
78
EXTRACT_AND_LOAD_SCRIPT=extract_and_load.py
89
LOG_LEVEL=DEBUG
910
ON_PIPELINE_FAILURE=log_and_continue
@@ -51,11 +52,17 @@ set -o allexport; source $ELT_SECRETS; set +o allexport
5152
# Extraction
5253
ensure_elt_sources_exist $git_url $git_clone_dir
5354
cd $git_clone_dir/$extract_and_load_dir
55+
dlt_write_disposition=
56+
if [ -n "$ELT_WRITE_DISPOSITION" ]; then
57+
dlt_write_disposition="--write-disposition $ELT_WRITE_DISPOSITION"
58+
fi
5459
uv run \
5560
--reinstall \
5661
--script $EXTRACT_AND_LOAD_SCRIPT \
5762
--log-level $LOG_LEVEL \
58-
--on-pipeline-step-failure $ON_PIPELINE_FAILURE
63+
--on-pipeline-step-failure $ON_PIPELINE_FAILURE \
64+
$dlt_write_disposition
65+
5966

6067
# Transform
6168
if [ -n "$dbt_project_dir" ]; then

0 commit comments

Comments
 (0)