Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -361,7 +361,7 @@ select
as capstone_indicator
, date_diff(
'day'
, final_combined_programs.programcertificate_created_on_date
, final_combined_programs.first_courserun_start_on_date
, final_combined_programs.programcertificate_created_on_date
) as program_completion_days
from final_combined_programs
Original file line number Diff line number Diff line change
Expand Up @@ -33,12 +33,6 @@ with enrollment_detail as (
else 0
end
) as capstone_sum
, min(
case
when course_enrollment_detail.courserunenrollment_enrollment_mode = 'verified'
then cast(substring(course_enrollment_detail.courserun_start_on, 1, 10) as date)
end
) as first_courserun_start_on_date
from courses_in_program
inner join course_enrollment_detail
on courses_in_program.course_readable_id = course_enrollment_detail.course_readable_id
Expand Down Expand Up @@ -97,11 +91,7 @@ select
combined_users.user_address_postal_code, combined_users2.user_address_postal_code
) as user_address_postal_code
, case when courses_detail.capstone_sum > 0 then 'Y' else 'N' end as capstone_ind
, date_diff(
'day'
, courses_detail.first_courserun_start_on_date
, cast(substring(enrollment_detail.programcertificate_created_on, 1, 10) as date)
) as program_complete_days
, enrollment_detail.program_completion_days as program_complete_days
from enrollment_detail
left join combined_users
on enrollment_detail.platform_name = '{{ var("edxorg") }}'
Comment on lines 91 to 97

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bug: The calculation for first_courserun_start_on_date now incorrectly includes unverified enrollments, which alters the program_completion_days metric by using an earlier start date for some users.
Severity: MEDIUM

Suggested Fix

The reporting model should revert to its previous logic for calculating first_courserun_start_on_date. Instead of consuming the field from the mart, re-implement the logic that filters for courserunenrollment_enrollment_mode = 'verified' when determining the minimum course run start date. This will restore the original definition of the program_completion_days metric.

Prompt for AI Agent
Review the code at the location below. A potential bug has been identified by an AI
agent. Verify if this is a real issue. If it is, propose a fix; if not, explain why it's
not valid.

Location: src/ol_dbt/models/reporting/program_enrollment_with_user_report.sql#L91-L97

Potential issue: The reporting model `program_enrollment_with_user_report.sql` now
sources the `first_courserun_start_on_date` from a mart model. The previous
implementation calculated this date using only 'verified' enrollments. The mart model,
however, calculates this date based on all enrollment types, including unverified ones.
This change in logic means that for any user with an unverified enrollment that started
before their first verified one, the `program_completion_days` metric will be calculated
from this earlier date, leading to a significant and unintended change in the report's
output.

Did we get this right? 👍 / 👎 to inform future reviews.

Expand Down
22 changes: 18 additions & 4 deletions src/ol_dbt_cli/ol_dbt_cli/lib/sql_parser.py
Original file line number Diff line number Diff line change
Expand Up @@ -197,15 +197,29 @@ def _render_jinja(sql: str) -> tuple[str, list[str], list[str], dict[str, str],
# , __jinja__) as array (varchar)), ', ' <- orphaned tail (array(varchar) is a cast)
# ) as course_level <- real alias, must be last token on its line
#
# The regex uses a tempered greedy token `(?:(?!<next_jinja_col>)[^\n]*\n)*?` to
# avoid consuming past the next broken-column placeholder.
# The regex uses a tempered greedy token `(?:(?!<next_jinja_col>)[^\n]*\n){0,6}?` to
# avoid consuming past the next broken-column placeholder. The repeat count is
# capped (rather than unbounded `*?`) because a *bare* placeholder that never
# gets an alias at all — e.g. the same macro call repeated, unaliased, inside a
# GROUP BY list — has no real terminal to find. Left unbounded, the lazy search
# happily scans past the GROUP BY, the CTE's closing paren, and even a `select`
# keyword to seize on some unrelated downstream `) as alias`, corrupting the
# SQL by deleting everything in between. Real broken-macro tails (see the two
# test cases below) resolve within 1-4 lines, so a small bound safely covers
# genuine cases while making runaway cross-CTE matches structurally impossible:
# past the bound the regex simply fails to match and the bare placeholder is
# left as-is (a harmless unresolved identifier) instead of eating the file.
# Matches both ``__jinja__`` (regex path) and ``__macro__`` (Jinja2 path) placeholders.
_BROKEN_COL_PLACEHOLDER = r"(?:__jinja__|__macro__|__undefined__)"
_BROKEN_COL_RE = re.compile(
rf"([ \t]*,[ \t]*){_BROKEN_COL_PLACEHOLDER}(?![ \t]+as\b)" # column-sep + placeholder, no direct alias
r"[^\n]*\n" # rest of the first (broken) line
rf"(?:(?![ \t]*,[ \t]*{_BROKEN_COL_PLACEHOLDER})[^\n]*\n)*?" # optional continuation lines
r"[ \t]*\)[ \t]+as[ \t]+(\w+)[ \t]*(?:--[^\n]*)?(?:\n|$)", # ) as alias — must be last token on line
rf"(?:(?![ \t]*,[ \t]*{_BROKEN_COL_PLACEHOLDER})[^\n]*\n){{0,6}}?" # bounded continuation lines
r"[ \t]*\)?[ \t]*as[ \t]+(\w+)[ \t]*(?:--[^\n]*)?(?:\n|$)", # [)] as alias — must be last token on line.
# The ')' is optional: a plain macro call (e.g. a UDF-style {{ macro(...) }})
# followed by `as alias` alone on the next line has no stray tokens to
# collapse, so the lazy loop must terminate there instead of overrunning
# into an unrelated later `) as ...` several columns/CTEs downstream.
)


Expand Down
34 changes: 34 additions & 0 deletions src/ol_dbt_cli/tests/test_sql_parser.py
Original file line number Diff line number Diff line change
Expand Up @@ -270,6 +270,40 @@ def test_broken_column_macro_multiline_case(self) -> None:
assert "item_id" in result.output_columns
assert "item_types" in result.output_columns

def test_broken_column_regex_stops_at_plain_multiline_alias(self) -> None:
"""A plain macro call aliased on the next line must not swallow later columns.

Unlike the orphaned-tail cases above, ``{{ macro(...) }}`` here has no stray
tokens trailing it — the very next line is just ``as alias``. The collapse
regex must terminate there instead of lazily overrunning into unrelated
later columns (and even later CTEs) looking for the next ``) as ...``.
"""
sql = (
"with combined as (\n"
" select\n"
" combined.platform_name\n"
" , {{ generate_hash_id('cast(combined.user_id as varchar)') }}\n"
" as user_hashed_id\n"
" , combined.user_id\n"
" , count(distinct combined.course_id) as unique_courses\n"
" from combined\n"
" group by\n"
" combined.platform_name\n"
" , {{ generate_hash_id('cast(combined.user_id as varchar)') }}\n"
" , combined.user_id\n"
")\n"
"select\n"
" combined.platform_name\n"
" , combined.user_hashed_id\n"
" , combined.unique_courses\n"
"from combined"
)
result = parse_model_sql("my_model", sql)
assert result.parse_error is None
assert "platform_name" in result.output_columns
assert "user_hashed_id" in result.output_columns
assert "unique_courses" in result.output_columns

def test_var_in_where_clause_parseable(self) -> None:
"""A model with '{{ var(...) }}' in WHERE must parse without error."""
sql = (
Expand Down
Loading