Skip to content

[WORKAROUND] Traditional schedule for e assets, eager for h* assets#1060

Merged
damonbayer merged 7 commits into
mainfrom
jk-schedule-workaround
May 12, 2026
Merged

[WORKAROUND] Traditional schedule for e assets, eager for h* assets#1060
damonbayer merged 7 commits into
mainfrom
jk-schedule-workaround

Conversation

@jkislin
Copy link
Copy Markdown
Contributor

@jkislin jkislin commented May 11, 2026

This creates a schedule that submits run requests 30 minutes after the upstream cfa-nssp-etl asset is materializing for the initial e assets. This temporarily disables their "automation conditions".
It also sets the initial h* assets' automation conditions to eager(), as the upstream nhsn_hrd_prelim is already gated to Wednesdays.

  • This is a temporary workaround to enable fully automated production this week.
  • We are testing this locally via previewed tick results and letting it run Tuesday.
  • We are still working on a longer term solution that allows us to fully use automation conditions.
  • We will revert this once we have a full solution.
image

@codecov
Copy link
Copy Markdown

codecov Bot commented May 11, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 60.80%. Comparing base (2b84164) to head (f94ce3e).

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1060      +/-   ##
==========================================
+ Coverage   58.44%   60.80%   +2.35%     
==========================================
  Files          28       32       +4     
  Lines        1918     2064     +146     
==========================================
+ Hits         1121     1255     +134     
- Misses        797      809      +12     
Flag Coverage Δ
epiautogp 91.78% <ø> (?)
hewr 76.98% <ø> (ø)
pipelines 55.80% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR introduces a temporary Dagster workaround to keep weekly forecast assets running automatically while custom automation-condition behavior is being debugged. It replaces the prior “upstream” automation-condition approach for the initial e assets with a fixed-time schedule, and sets h* assets to run eagerly under an automation-condition sensor.

Changes:

  • Commented out the custom IsWeekday automation condition and the WeeklyForecastUpstream automation sensor.
  • Added a weekly_forecast_e_schedule to trigger the initial e-asset materializations on a cron.
  • Set pyrenew_h / pyrenew_he automation conditions to eager() and added an automation-condition sensor for the WeeklyForecastH group.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread dagster_defs.py Outdated
Comment thread dagster_defs.py
Comment thread dagster_defs.py
@jkislin jkislin marked this pull request as ready for review May 12, 2026 15:07
Copy link
Copy Markdown
Collaborator

@giomrella giomrella left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good besides the change to EpiAutoGP, which seems like an accidental inclusion

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems unintentional

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's a pre-commit change.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

working on getting julia added to pre-commit.ci so this doesn't happen pre-commit-ci/runner-image#332

@damonbayer damonbayer merged commit a45a915 into main May 12, 2026
69 of 71 checks passed
@damonbayer damonbayer deleted the jk-schedule-workaround branch May 12, 2026 20:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants