feat: add edr artifacts commands for warehouse artifact queries#2193
feat: add edr artifacts commands for warehouse artifact queries#2193
edr artifacts commands for warehouse artifact queries#2193Conversation
Adds a new `edr artifacts` subcommand group that lets agents and scripts
query Elementary's warehouse artifact tables directly. Twelve commands
across six entities (test-results, run-results, invocations, models,
sources, tests) with list and get-by-id variants.
Output is a stable JSON envelope on stdout (with `-o table` for humans),
errors are JSON on stderr with `{error, code, details}`, and exit codes
follow 0/1/2 for success/user-error/system-error. List commands paginate
via `--limit` and report `has_more`. Logs are redirected to stderr when
running `edr artifacts ...` so stdout stays pipe-safe.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|
👋 @oravi |
📝 WalkthroughWalkthroughThis PR introduces a new Changes
Sequence Diagram(s)sequenceDiagram
actor User
participant CLI as edr artifacts<br/>(Command)
participant Config as Config<br/>(build_config)
participant Runner as dbt Runner<br/>(create_artifacts_runner)
participant Macro as dbt Macro<br/>(run_macro)
participant Warehouse as Warehouse<br/>(query execution)
participant Output as Output<br/>(emit_json/table)
User->>CLI: edr artifacts <subcommand> [filters]
CLI->>Config: build_config(paths, profile)
Config-->>CLI: Config instance
CLI->>Runner: create_artifacts_runner(config, profile)
Runner-->>CLI: dbt CommandLineDbtRunner<br/>(with profile override if set)
CLI->>Macro: run_macro(runner, macro_name, filters)
Macro->>Warehouse: execute SQL query
Warehouse-->>Macro: result rows
Macro-->>CLI: JSON or error
alt Success
CLI->>CLI: parse_json_field (hydrate data)
CLI->>CLI: apply_pagination (trim + has_more)
CLI->>Output: emit_json or emit_table
Output-->>User: formatted output to stdout
else ArtifactFetchError
CLI->>Output: emit_error(code, details)
Output-->>User: JSON error to stderr
else Other Exception
CLI->>Output: emit_error(INTERNAL_ERROR)
Output-->>User: JSON error to stderr
end
Estimated code review effort🎯 4 (Complex) | ⏱️ ~50 minutes This PR spans 15+ new Python files and 13+ SQL macros with heterogeneous logic: CLI commands with varied filter semantics and pagination patterns, SQL macros with conditional filter composition, Python utilities for error translation and runner patching, and integration points in the main CLI and logging systems. Each entity command exhibits distinct complexity (invocations, test_results, and tests are notably dense), requiring separate reasoning for filters, field hydration, output formatting, and error paths. The total scope (~1600 lines) and architectural breadth demand methodical verification of configuration propagation, macro parameter binding, error handling uniformity, and output structure consistency. Poem
🚥 Pre-merge checks | ✅ 2 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (2 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Actionable comments posted: 6
🧹 Nitpick comments (8)
elementary/monitor/dbt_project/macros/artifacts/get_dbt_run_result.sql (1)
7-28: Duplicated projection definition withget_dbt_run_results.sql.The
heavy_cols/base_colsblocks are byte-identical to the ones inget_dbt_run_results.sql. If a schema column is added/removed, both macros must be edited in lockstep — easy to miss. Consider extracting into a tiny shared macro (e.g.,_dbt_run_results_projection(include_heavy)) in_filter_helpers.sqlor a new_projections.sql, and using it from both call sites.🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@elementary/monitor/dbt_project/macros/artifacts/get_dbt_run_result.sql` around lines 7 - 28, The projection logic in get_dbt_run_result.sql duplicates the identical heavy_cols and base_cols definitions from get_dbt_run_results.sql, which risks drift when schema changes; extract these into a single reusable macro (e.g., _dbt_run_results_projection(include_heavy)) placed in a shared file like _filter_helpers.sql or _projections.sql, then replace the base_cols/heavy_cols and projection construction in both get_dbt_run_result.sql and get_dbt_run_results.sql to call _dbt_run_results_projection(include_heavy) so both macros use the same canonical column list.elementary/monitor/dbt_project/macros/artifacts/get_dbt_models.sql (1)
40-45: Consider a two-column LIKE helper.The
name/aliasblock is duplicated verbatim inget_dbt_tests.sql(which addsshort_name). If you want to trim duplication, a small_like_ci_any_filter(columns, value)helper in_filter_helpers.sqlthat ORs across a list of columns would collapse both sites. Optional.🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@elementary/monitor/dbt_project/macros/artifacts/get_dbt_models.sql` around lines 40 - 45, Create a reusable two-column case-insensitive LIKE helper to remove the duplicated name/alias logic: add a macro named _like_ci_any_filter(columns, value) in _filter_helpers.sql that accepts a list of column names and the value, escapes the value via elementary_cli._sql_escape(value), and returns an OR-clause like lower(col) LIKE lower('%...%') for each column (joined with OR). Replace the duplicated block in get_dbt_models.sql (the name/alias predicate) and the similar block in get_dbt_tests.sql (which also checks short_name) by calling _like_ci_any_filter(['name','alias'] , name) or _like_ci_any_filter(['name','alias','short_name'], name) as appropriate. Ensure the macro returns a properly parenthesized expression and is only rendered when value is not none to preserve existing conditional behavior.elementary/artifacts/entities/models.py (1)
108-117: Consolidate repeated error handling across entity commands.This
except ArtifactFetchError / except Exceptionblock is duplicated verbatim inmodels.model(L154-163) and acrosstest_results.py,run_results.py(and presumably the other three entities). Consider extracting a small helper or decorator (or reusingcli._handle_fetch_errorplus a generic internal-error helper) so every entity command shares one implementation. This also addresses Ruff BLE001 (blindexcept Exception) in one place instead of six.♻️ Example helper
# elementary/artifacts/common.py (or output.py) from functools import wraps def handle_cli_errors(func): `@wraps`(func) def wrapper(*args, **kwargs): try: return func(*args, **kwargs) except ArtifactFetchError as exc: sys.exit(emit_error(str(exc), exc.code, exc.details)) except Exception as exc: # noqa: BLE001 sys.exit(emit_error( f"Unexpected error: {exc}", ErrorCode.INTERNAL_ERROR, {"type": type(exc).__name__}, )) return wrapperThen decorate each entity command with
@handle_cli_errorsand drop the inline try/except.🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@elementary/artifacts/entities/models.py` around lines 108 - 117, Extract the duplicated try/except into a single reusable helper (e.g., handle_cli_errors) in a shared module (like elementary.artifacts.common or output) that catches ArtifactFetchError and generic Exception and calls emit_error with the same payloads; alternatively, compose/extend the existing cli._handle_fetch_error by adding a small internal-error helper that handles the broad Exception case (using ErrorCode.INTERNAL_ERROR and type(exc).__name__) so callers can reuse it. Then remove the repeated except blocks in models.model and the commands in test_results.py, run_results.py, etc., and either decorate those command functions with `@handle_cli_errors` or call the shared handler wrapper so all entity commands use one implementation and Ruff BLE001 is isolated in one place.elementary/artifacts/entities/run_results.py (1)
133-133: Inconsistent macro parameter naming for the same user-facing flag.The same
--include-compiled-codeCLI flag is forwarded to the list macro aslightweight=not include_compiled_code(L133) and to the single-get macro asinclude_compiled_code=include_compiled_code(L182). The inverted sense + different parameter names betweenget_dbt_run_resultsandget_dbt_run_resultmakes future maintenance error-prone (easy to flip the boolean). Consider aligning both SQL macros on one parameter name (e.g.,include_compiled_code) and keeping the boolean polarity identical in both call sites.Also applies to: 182-182
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@elementary/artifacts/entities/run_results.py` at line 133, The list macro call in get_dbt_run_results uses the parameter name lightweight with inverted polarity ("lightweight = not include_compiled_code") while get_dbt_run_result passes include_compiled_code directly; standardize both to the same parameter name and polarity to avoid confusion—update the call in get_dbt_run_results to pass "include_compiled_code = include_compiled_code" (or, alternatively, change get_dbt_run_result to use "lightweight = not include_compiled_code") so both macros use the identical parameter name (include_compiled_code) and the same boolean sense; adjust any downstream macro signatures if necessary to accept include_compiled_code.elementary/artifacts/cli.py (1)
27-34: Remove unused helpers and fix implicitOptionaltyping.The functions
_handle_fetch_errorand_handle_bad_argumentare never called within this module or elsewhere. Meanwhile, all entity modules (models, test_results, run_results, sources, tests, invocations) inline the samesys.exit(emit_error(...))pattern directly. Either adopt these helpers consistently across entities to eliminate duplication, or remove them. Additionally, thedetailsparameter in_handle_bad_argumentshould explicitly typeNoneasOptional[Dict[str, Any]]to match the signature ofemit_errorand avoid Ruff RUF013 violations.♻️ Proposed refactor
-def _handle_fetch_error(exc: ArtifactFetchError) -> None: - code = emit_error(str(exc), exc.code, exc.details) - sys.exit(code) - - -def _handle_bad_argument(message: str, details: dict = None) -> None: - code = emit_error(message, ErrorCode.BAD_ARGUMENT, details or {}) - sys.exit(code) +def _handle_fetch_error(exc: ArtifactFetchError) -> None: + sys.exit(emit_error(str(exc), exc.code, exc.details)) + + +def _handle_bad_argument(message: str, details: Optional[Dict[str, Any]] = None) -> None: + sys.exit(emit_error(message, ErrorCode.BAD_ARGUMENT, details))If adopting these helpers, add
from typing import Any, Dict, Optionaland refactor entity modules to use them instead of inlining the pattern.🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@elementary/artifacts/cli.py` around lines 27 - 34, The two helpers _handle_fetch_error and _handle_bad_argument are unused and _handle_bad_argument has an implicit Optional typing for details; either remove both helpers or consistently use them across entity modules instead of inlining sys.exit(emit_error(...)), and if you keep _handle_bad_argument update its signature to explicitly type details as Optional[Dict[str, Any]] and add the required imports (Any, Dict, Optional) so the parameter matches emit_error and avoids RUF013; ensure references to _handle_fetch_error (ArtifactFetchError) and _handle_bad_argument are wired into callers or deleted to eliminate dead code.elementary/artifacts/entities/invocations.py (2)
203-221: Redundantdata.lengthfield and_emit_singletable uses list-only columns.Two small output concerns:
- Line 212:
"data": {"length": len(rows)}is redundant with the top-level"count"already emitted on line 209. If this isn't load-bearing for a documented contract it's dead weight in the envelope and risks becoming a stale parallel counter._emit_singlerenders the detail record throughLIST_COLUMNS, soedr artifacts invocation <id> -o tablehides fields that the JSON path exposes (e.g.job_id,job_run_id, timing breakdowns). The JSON envelope already uses a distinct key ("invocation"vs"invocations"); a separateDETAIL_COLUMNS(or pivoted key/value rendering for a single row) would be more useful and mirror whattests.pydoes withDETAIL_JSON_FIELDS.♻️ Suggested cleanup for the redundant field
emit_json( { "count": len(rows), "has_more": has_more, "invocations": rows, - "data": {"length": len(rows)}, } )🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@elementary/artifacts/entities/invocations.py` around lines 203 - 221, The _emit_list function emits a redundant nested data.length that duplicates the top-level count—remove the "data": {"length": ...} entry so _emit_list only returns count, has_more, and invocations via emit_json; and update _emit_single to render table output using a richer column set instead of LIST_COLUMNS (introduce or use DETAIL_COLUMNS or a pivoted key/value rendering) so emit_table([row], DETAIL_COLUMNS) (or equivalent key/value display) shows all detail fields like job_id, job_run_id, and timing breakdowns while JSON output remains emit_json({"invocation": row}).
192-200: Consider validating user-supplied--started-after/--started-beforelocally.When the user passes these flags, the strings are forwarded verbatim to the macro without parsing. A typo like
2024-13-01will travel all the way to the warehouse and surface as aWAREHOUSE_ERROR, which is a noisy failure mode for a client-side mistake. Parsing withdatetime.fromisoformat(orclick.DateTime) here would produce a cleanUSER_ERRORwith a clear message and also let you normalize to UTC consistently with the defaults.🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@elementary/artifacts/entities/invocations.py` around lines 192 - 200, The _resolve_time_window function currently forwards user-supplied started_after/started_before strings verbatim; parse each non-None value with datetime.fromisoformat (or equivalent) inside _resolve_time_window, catch ValueError and propagate a clear USER_ERROR (or raise a ValueError with a descriptive message) so typos like "2024-13-01" fail locally, normalize parsed datetimes to UTC (use astimezone(timezone.utc)) and then return their .isoformat() strings; keep the same return signature and treat None by substituting the existing defaults before formatting.elementary/artifacts/entities/tests.py (1)
209-227: Same redundantdata.lengthfield; detail table ignoresDETAIL_JSON_FIELDS.
- Line 218:
"data": {"length": len(rows)}duplicates the top-level"count"on line 215 — same observation as ininvocations.py. Recommend dropping unless it's part of a documented stable contract._emit_singlerenders the single test viaLIST_COLUMNS, which intentionally omits the richer fields hydrated byDETAIL_JSON_FIELDS(test_params,meta,depends_on_*, etc.). Foredr artifacts test <unique_id> -o tablethat means the table output is strictly less informative than the JSON output, which is surprising given the detail hydration above it. A dedicatedDETAIL_COLUMNS(or a two-column key/value rendering for single-record output) would make the hydration actually observable.♻️ Suggested cleanup for the redundant field
emit_json( { "count": len(rows), "has_more": has_more, "tests": rows, - "data": {"length": len(rows)}, } )🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@elementary/artifacts/entities/tests.py` around lines 209 - 227, Remove the redundant "data": {"length": ...} in _emit_list (it duplicates "count") and stop emitting it unless it's a documented contract; in _emit_single, change the table rendering to show the detailed fields instead of LIST_COLUMNS so the hydrated DETAIL_JSON_FIELDS are visible — either use a new DETAIL_COLUMNS constant (or a two-column key/value renderer) when calling emit_table([row], ...) in _emit_single, or branch to a dedicated detail-table helper; keep emit_json behavior unchanged.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@elementary/artifacts/entities/test_results.py`:
- Line 2: The import list in elementary.artifacts.entities.test_results
currently includes Optional but it's unused; remove Optional from the from
typing import Any, Dict, List, Optional statement (leaving Any, Dict, List) to
satisfy flake8 F401 and avoid unused imports in this module (check the top-level
import line in test_results.py).
In `@elementary/artifacts/fetching.py`:
- Around line 45-54: The json.loads call in fetch logic can raise TypeError for
non-string elements (e.g. None/dict/bytes), so update the error handling in the
block around json.loads(response[0]) (in elementary.artifacts.fetching.py) to
also catch TypeError (and you can drop IndexError because of the preceding if
not response guard); ensure ArtifactFetchError is raised with the same
ErrorCode.MALFORMED_ARTIFACTS and include the raw response[0] in the payload as
currently done so `[None]` or other non-string values produce the intended
ArtifactFetchError instead of an unhandled TypeError.
In `@elementary/artifacts/runner.py`:
- Around line 33-41: The wrapper _inject_profile_override currently calls the
saved original as original(command_args=new_args, *args, **kwargs) which both
violates the star-arg-after-keyword rule and can raise TypeError if positional
args are passed; change the call in patched to pass new_args positionally (e.g.,
original(new_args, *args, **kwargs)) so command_args is supplied as a positional
argument and remove the keyword form; keep references to patched, original and
runner._run_command when updating. Also consider the optional durability
improvement of avoiding monkey-patching runner._run_command by subclassing
CommandLineDbtRunner or threading the profile through create_dbt_runner instead
of patching the private method.
In `@elementary/monitor/dbt_project/macros/artifacts/_filter_helpers.sql`:
- Around line 23-27: The _like_ci_filter macro uses
elementary_cli._sql_escape(value) but does not escape SQL LIKE wildcards (%) and
(_) so user input like "foo_bar" matches unexpectedly; update _like_ci_filter to
additionally replace '%' and '_' in the escaped value with '\%' and '\_'
(ensuring proper escaping of the backslash in the SQL literal) and append an
"ESCAPE '\'" clause to the LIKE expression so the backslashes are honored; keep
the case-insensitive lower(...) wrappers and perform the wildcard-escaping on
the result of elementary_cli._sql_escape(value) before embedding it into the
pattern.
- Around line 57-67: The numeric filter macros (_gt_number_filter and
_lt_number_filter) currently inline raw {{ value }} into SQL; coerce and
validate the value before emitting it to prevent injection. Update both macros
to convert value to a numeric type (e.g., use Jinja's int or float filter) and
only render the "and {{ column }} > ..." / "< ..." clause when the coercion
succeeds (keep the existing "if value is not none" guard but replace raw
insertion with the coerced numeric), so the emitted SQL always contains a
validated numeric literal; reference the macros _gt_number_filter and
_lt_number_filter when making this change.
In `@elementary/monitor/dbt_project/macros/artifacts/get_dbt_run_results.sql`:
- Around line 57-58: The ORDER BY in get_dbt_run_results.sql uses only
execute_started_at which is not a total order and can cause pagination
instability; update the ORDER BY clause (the line containing "order by
execute_started_at desc") to add a deterministic tiebreaker using a unique
column (for example run_id or the table primary key column) so it becomes a
total order (e.g., ORDER BY execute_started_at DESC, <unique_column> DESC) while
keeping the existing LIMIT handling intact.
---
Nitpick comments:
In `@elementary/artifacts/cli.py`:
- Around line 27-34: The two helpers _handle_fetch_error and
_handle_bad_argument are unused and _handle_bad_argument has an implicit
Optional typing for details; either remove both helpers or consistently use them
across entity modules instead of inlining sys.exit(emit_error(...)), and if you
keep _handle_bad_argument update its signature to explicitly type details as
Optional[Dict[str, Any]] and add the required imports (Any, Dict, Optional) so
the parameter matches emit_error and avoids RUF013; ensure references to
_handle_fetch_error (ArtifactFetchError) and _handle_bad_argument are wired into
callers or deleted to eliminate dead code.
In `@elementary/artifacts/entities/invocations.py`:
- Around line 203-221: The _emit_list function emits a redundant nested
data.length that duplicates the top-level count—remove the "data": {"length":
...} entry so _emit_list only returns count, has_more, and invocations via
emit_json; and update _emit_single to render table output using a richer column
set instead of LIST_COLUMNS (introduce or use DETAIL_COLUMNS or a pivoted
key/value rendering) so emit_table([row], DETAIL_COLUMNS) (or equivalent
key/value display) shows all detail fields like job_id, job_run_id, and timing
breakdowns while JSON output remains emit_json({"invocation": row}).
- Around line 192-200: The _resolve_time_window function currently forwards
user-supplied started_after/started_before strings verbatim; parse each non-None
value with datetime.fromisoformat (or equivalent) inside _resolve_time_window,
catch ValueError and propagate a clear USER_ERROR (or raise a ValueError with a
descriptive message) so typos like "2024-13-01" fail locally, normalize parsed
datetimes to UTC (use astimezone(timezone.utc)) and then return their
.isoformat() strings; keep the same return signature and treat None by
substituting the existing defaults before formatting.
In `@elementary/artifacts/entities/models.py`:
- Around line 108-117: Extract the duplicated try/except into a single reusable
helper (e.g., handle_cli_errors) in a shared module (like
elementary.artifacts.common or output) that catches ArtifactFetchError and
generic Exception and calls emit_error with the same payloads; alternatively,
compose/extend the existing cli._handle_fetch_error by adding a small
internal-error helper that handles the broad Exception case (using
ErrorCode.INTERNAL_ERROR and type(exc).__name__) so callers can reuse it. Then
remove the repeated except blocks in models.model and the commands in
test_results.py, run_results.py, etc., and either decorate those command
functions with `@handle_cli_errors` or call the shared handler wrapper so all
entity commands use one implementation and Ruff BLE001 is isolated in one place.
In `@elementary/artifacts/entities/run_results.py`:
- Line 133: The list macro call in get_dbt_run_results uses the parameter name
lightweight with inverted polarity ("lightweight = not include_compiled_code")
while get_dbt_run_result passes include_compiled_code directly; standardize both
to the same parameter name and polarity to avoid confusion—update the call in
get_dbt_run_results to pass "include_compiled_code = include_compiled_code" (or,
alternatively, change get_dbt_run_result to use "lightweight = not
include_compiled_code") so both macros use the identical parameter name
(include_compiled_code) and the same boolean sense; adjust any downstream macro
signatures if necessary to accept include_compiled_code.
In `@elementary/artifacts/entities/tests.py`:
- Around line 209-227: Remove the redundant "data": {"length": ...} in
_emit_list (it duplicates "count") and stop emitting it unless it's a documented
contract; in _emit_single, change the table rendering to show the detailed
fields instead of LIST_COLUMNS so the hydrated DETAIL_JSON_FIELDS are visible —
either use a new DETAIL_COLUMNS constant (or a two-column key/value renderer)
when calling emit_table([row], ...) in _emit_single, or branch to a dedicated
detail-table helper; keep emit_json behavior unchanged.
In `@elementary/monitor/dbt_project/macros/artifacts/get_dbt_models.sql`:
- Around line 40-45: Create a reusable two-column case-insensitive LIKE helper
to remove the duplicated name/alias logic: add a macro named
_like_ci_any_filter(columns, value) in _filter_helpers.sql that accepts a list
of column names and the value, escapes the value via
elementary_cli._sql_escape(value), and returns an OR-clause like lower(col) LIKE
lower('%...%') for each column (joined with OR). Replace the duplicated block in
get_dbt_models.sql (the name/alias predicate) and the similar block in
get_dbt_tests.sql (which also checks short_name) by calling
_like_ci_any_filter(['name','alias'] , name) or
_like_ci_any_filter(['name','alias','short_name'], name) as appropriate. Ensure
the macro returns a properly parenthesized expression and is only rendered when
value is not none to preserve existing conditional behavior.
In `@elementary/monitor/dbt_project/macros/artifacts/get_dbt_run_result.sql`:
- Around line 7-28: The projection logic in get_dbt_run_result.sql duplicates
the identical heavy_cols and base_cols definitions from get_dbt_run_results.sql,
which risks drift when schema changes; extract these into a single reusable
macro (e.g., _dbt_run_results_projection(include_heavy)) placed in a shared file
like _filter_helpers.sql or _projections.sql, then replace the
base_cols/heavy_cols and projection construction in both get_dbt_run_result.sql
and get_dbt_run_results.sql to call _dbt_run_results_projection(include_heavy)
so both macros use the same canonical column list.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: a4a3fbd7-a217-479e-83a6-6c54794d53ed
📒 Files selected for processing (29)
elementary/artifacts/README.mdelementary/artifacts/__init__.pyelementary/artifacts/cli.pyelementary/artifacts/common.pyelementary/artifacts/entities/__init__.pyelementary/artifacts/entities/invocations.pyelementary/artifacts/entities/models.pyelementary/artifacts/entities/run_results.pyelementary/artifacts/entities/sources.pyelementary/artifacts/entities/test_results.pyelementary/artifacts/entities/tests.pyelementary/artifacts/fetching.pyelementary/artifacts/output.pyelementary/artifacts/runner.pyelementary/cli/cli.pyelementary/monitor/dbt_project/macros/artifacts/_filter_helpers.sqlelementary/monitor/dbt_project/macros/artifacts/get_dbt_invocation.sqlelementary/monitor/dbt_project/macros/artifacts/get_dbt_invocations.sqlelementary/monitor/dbt_project/macros/artifacts/get_dbt_model.sqlelementary/monitor/dbt_project/macros/artifacts/get_dbt_models.sqlelementary/monitor/dbt_project/macros/artifacts/get_dbt_run_result.sqlelementary/monitor/dbt_project/macros/artifacts/get_dbt_run_results.sqlelementary/monitor/dbt_project/macros/artifacts/get_dbt_source.sqlelementary/monitor/dbt_project/macros/artifacts/get_dbt_sources.sqlelementary/monitor/dbt_project/macros/artifacts/get_dbt_test.sqlelementary/monitor/dbt_project/macros/artifacts/get_dbt_tests.sqlelementary/monitor/dbt_project/macros/artifacts/get_elementary_test_result.sqlelementary/monitor/dbt_project/macros/artifacts/get_elementary_test_results.sqlelementary/utils/log.py
| @@ -0,0 +1,214 @@ | |||
| import sys | |||
| from typing import Any, Dict, List, Optional | |||
There was a problem hiding this comment.
Remove unused Optional import.
Flake8 F401: Optional is imported but not used in this module.
-from typing import Any, Dict, List, Optional
+from typing import Any, Dict, List📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| from typing import Any, Dict, List, Optional | |
| from typing import Any, Dict, List |
🧰 Tools
🪛 Flake8 (7.3.0)
[error] 2-2: 'typing.Optional' imported but unused
(F401)
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@elementary/artifacts/entities/test_results.py` at line 2, The import list in
elementary.artifacts.entities.test_results currently includes Optional but it's
unused; remove Optional from the from typing import Any, Dict, List, Optional
statement (leaving Any, Dict, List) to satisfy flake8 F401 and avoid unused
imports in this module (check the top-level import line in test_results.py).
| if not response: | ||
| return None | ||
| try: | ||
| return json.loads(response[0]) | ||
| except (ValueError, IndexError) as exc: | ||
| raise ArtifactFetchError( | ||
| f"Malformed response from {macro_name}.", | ||
| ErrorCode.MALFORMED_ARTIFACTS, | ||
| {"raw": response[0] if response else None}, | ||
| ) from exc |
There was a problem hiding this comment.
json.loads can raise TypeError, which escapes the MALFORMED_ARTIFACTS path.
json.loads(response[0]) only raises ValueError/IndexError when response[0] is a string or the sequence is empty. If the dbt runner ever returns [None] or a non-string element (e.g. a dict, or a bytes payload on some adapters), json.loads raises TypeError, which is not in the caught tuple on line 49. The exception then propagates past ArtifactFetchError and gets reported by the entity commands as a generic INTERNAL_ERROR with an ugly message instead of the intended stable MALFORMED_ARTIFACTS envelope — undermining the "stable error contract" the PR is specifically trying to establish.
Separately, line 46's if not response: return None treats an empty list the same as no response at all, which seems fine, but note that [None] is truthy and will fall through to json.loads(None) — that's the concrete case that hits the TypeError gap above.
🛡️ Proposed fix
if not response:
return None
+ payload = response[0]
+ if payload is None:
+ return None
try:
- return json.loads(response[0])
- except (ValueError, IndexError) as exc:
+ return json.loads(payload)
+ except (TypeError, ValueError) as exc:
raise ArtifactFetchError(
f"Malformed response from {macro_name}.",
ErrorCode.MALFORMED_ARTIFACTS,
- {"raw": response[0] if response else None},
+ {"raw": payload},
) from exc(IndexError is no longer reachable after the if not response guard, so it can be dropped.)
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@elementary/artifacts/fetching.py` around lines 45 - 54, The json.loads call
in fetch logic can raise TypeError for non-string elements (e.g.
None/dict/bytes), so update the error handling in the block around
json.loads(response[0]) (in elementary.artifacts.fetching.py) to also catch
TypeError (and you can drop IndexError because of the preceding if not response
guard); ensure ArtifactFetchError is raised with the same
ErrorCode.MALFORMED_ARTIFACTS and include the raw response[0] in the payload as
currently done so `[None]` or other non-string values produce the intended
ArtifactFetchError instead of an unhandled TypeError.
| original = runner._run_command | ||
|
|
||
| def patched(self, command_args, *args, **kwargs): | ||
| new_args = list(command_args) | ||
| if new_args: | ||
| new_args = [new_args[0], "--profile", profile] + new_args[1:] | ||
| return original(command_args=new_args, *args, **kwargs) | ||
|
|
||
| runner._run_command = types.MethodType(patched, runner) |
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
# Locate the definition and all call sites of _run_command on the dbt runner.
ast-grep --pattern 'def _run_command($$$): $$$'
rg -nP --type=py -C2 '\b_run_command\s*\('Repository: elementary-data/elementary
Length of output: 5172
🏁 Script executed:
# Get the full _run_command signature
cat -n elementary/clients/dbt/command_line_dbt_runner.py | sed -n '170,190p'Repository: elementary-data/elementary
Length of output: 974
🏁 Script executed:
# Examine the _inject_profile_override function in the file under review
cat -n elementary/artifacts/runner.py | head -50Repository: elementary-data/elementary
Length of output: 1862
🏁 Script executed:
# Check what parameters _run_command actually accepts
rg -A 15 'def _run_command\(' elementary/clients/dbt/command_line_dbt_runner.pyRepository: elementary-data/elementary
Length of output: 674
Fix star-arg after keyword violation and latent TypeError in _run_command wrapper.
Two concerns in _inject_profile_override:
-
Line 39 uses
original(command_args=new_args, *args, **kwargs), which violates Ruff B026 (star-arg after keyword). More critically, if any element ever appears in*args, Python evaluates it positionally forcommand_args, causingTypeError: got multiple values for keyword argument 'command_args'. While no current caller passes extra positional args, this is a trivial footgun to remove: passnew_argspositionally instead. -
Wrapping the private
_run_commandcouples this module to an internal method ofCommandLineDbtRunner. If the base class renames or changes the signature of this hook,--profileinjection silently breaks. A subclass override or threadingprofilethroughcreate_dbt_runnerwould be more durable. Optional, but worth considering.
🛠️ Proposed fix for (1)
def patched(self, command_args, *args, **kwargs):
new_args = list(command_args)
if new_args:
- new_args = [new_args[0], "--profile", profile] + new_args[1:]
- return original(command_args=new_args, *args, **kwargs)
+ new_args = [new_args[0], "--profile", profile, *new_args[1:]]
+ return original(new_args, *args, **kwargs)🧰 Tools
🪛 Ruff (0.15.10)
[warning] 38-38: Consider [new_args[0], "--profile", profile, *new_args[1:]] instead of concatenation
Replace with [new_args[0], "--profile", profile, *new_args[1:]]
(RUF005)
[warning] 39-39: Star-arg unpacking after a keyword argument is strongly discouraged
(B026)
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@elementary/artifacts/runner.py` around lines 33 - 41, The wrapper
_inject_profile_override currently calls the saved original as
original(command_args=new_args, *args, **kwargs) which both violates the
star-arg-after-keyword rule and can raise TypeError if positional args are
passed; change the call in patched to pass new_args positionally (e.g.,
original(new_args, *args, **kwargs)) so command_args is supplied as a positional
argument and remove the keyword form; keep references to patched, original and
runner._run_command when updating. Also consider the optional durability
improvement of avoiding monkey-patching runner._run_command by subclassing
CommandLineDbtRunner or threading the profile through create_dbt_runner instead
of patching the private method.
| {%- macro _like_ci_filter(column, value) -%} | ||
| {%- if value is not none -%} | ||
| and lower({{ column }}) like lower('%{{ elementary_cli._sql_escape(value) }}%') | ||
| {%- endif -%} | ||
| {%- endmacro -%} |
There was a problem hiding this comment.
_like_ci_filter does not escape LIKE wildcards.
% and _ in user input are treated as wildcards, so a search like --name "foo_bar" will unexpectedly match fooXbar. Since the only escaping done is for single quotes, users cannot search for literal %/_. Not a correctness blocker, but worth documenting or escaping (replace('%','\\%') / replace('_','\\_') plus an ESCAPE '\' clause) if literal matching is desired. Minor UX nit given the CLI context.
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@elementary/monitor/dbt_project/macros/artifacts/_filter_helpers.sql` around
lines 23 - 27, The _like_ci_filter macro uses elementary_cli._sql_escape(value)
but does not escape SQL LIKE wildcards (%) and (_) so user input like "foo_bar"
matches unexpectedly; update _like_ci_filter to additionally replace '%' and '_'
in the escaped value with '\%' and '\_' (ensuring proper escaping of the
backslash in the SQL literal) and append an "ESCAPE '\'" clause to the LIKE
expression so the backslashes are honored; keep the case-insensitive lower(...)
wrappers and perform the wildcard-escaping on the result of
elementary_cli._sql_escape(value) before embedding it into the pattern.
| {%- macro _gt_number_filter(column, value) -%} | ||
| {%- if value is not none -%} | ||
| and {{ column }} > {{ value }} | ||
| {%- endif -%} | ||
| {%- endmacro -%} | ||
|
|
||
| {%- macro _lt_number_filter(column, value) -%} | ||
| {%- if value is not none -%} | ||
| and {{ column }} < {{ value }} | ||
| {%- endif -%} | ||
| {%- endmacro -%} |
There was a problem hiding this comment.
Numeric filters embed raw value without coercion/validation.
_gt_number_filter / _lt_number_filter inline {{ value }} directly into SQL. Unlike the string filters (which go through _sql_escape), these have no hygiene at all — any non-numeric input (e.g., user passing 1; drop table x) would be emitted verbatim. Even though the CLI is the only caller today, a cheap coercion keeps parity with the rest of this module and protects against future misuse.
🛡️ Proposed fix
{%- macro _gt_number_filter(column, value) -%}
{%- if value is not none -%}
- and {{ column }} > {{ value }}
+ and {{ column }} > {{ value | float }}
{%- endif -%}
{%- endmacro -%}
{%- macro _lt_number_filter(column, value) -%}
{%- if value is not none -%}
- and {{ column }} < {{ value }}
+ and {{ column }} < {{ value | float }}
{%- endif -%}
{%- endmacro -%}📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| {%- macro _gt_number_filter(column, value) -%} | |
| {%- if value is not none -%} | |
| and {{ column }} > {{ value }} | |
| {%- endif -%} | |
| {%- endmacro -%} | |
| {%- macro _lt_number_filter(column, value) -%} | |
| {%- if value is not none -%} | |
| and {{ column }} < {{ value }} | |
| {%- endif -%} | |
| {%- endmacro -%} | |
| {%- macro _gt_number_filter(column, value) -%} | |
| {%- if value is not none -%} | |
| and {{ column }} > {{ value | float }} | |
| {%- endif -%} | |
| {%- endmacro -%} | |
| {%- macro _lt_number_filter(column, value) -%} | |
| {%- if value is not none -%} | |
| and {{ column }} < {{ value | float }} | |
| {%- endif -%} | |
| {%- endmacro -%} |
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@elementary/monitor/dbt_project/macros/artifacts/_filter_helpers.sql` around
lines 57 - 67, The numeric filter macros (_gt_number_filter and
_lt_number_filter) currently inline raw {{ value }} into SQL; coerce and
validate the value before emitting it to prevent injection. Update both macros
to convert value to a numeric type (e.g., use Jinja's int or float filter) and
only render the "and {{ column }} > ..." / "< ..." clause when the coercion
succeeds (keep the existing "if value is not none" guard but replace raw
insertion with the coerced numeric), so the emitted SQL always contains a
validated numeric literal; reference the macros _gt_number_filter and
_lt_number_filter when making this change.
| order by execute_started_at desc | ||
| limit {{ limit }} |
There was a problem hiding this comment.
Pagination ordering may be non-deterministic on ties.
order by execute_started_at desc alone is not a total order — rows sharing the same execute_started_at can shuffle between pages under limit/limit+1 pagination, causing duplicates or skips when the CLI stitches pages. Adding a tiebreaker on a unique column keeps pagination stable.
🛡️ Proposed fix
- order by execute_started_at desc
+ order by execute_started_at desc, model_execution_id desc📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| order by execute_started_at desc | |
| limit {{ limit }} | |
| order by execute_started_at desc, model_execution_id desc | |
| limit {{ limit }} |
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@elementary/monitor/dbt_project/macros/artifacts/get_dbt_run_results.sql`
around lines 57 - 58, The ORDER BY in get_dbt_run_results.sql uses only
execute_started_at which is not a total order and can cause pagination
instability; update the ORDER BY clause (the line containing "order by
execute_started_at desc") to add a deterministic tiebreaker using a unique
column (for example run_id or the table primary key column) so it becomes a
total order (e.g., ORDER BY execute_started_at DESC, <unique_column> DESC) while
keeping the existing LIMIT handling intact.
Summary
edr artifactssubcommand group so agents and scripts can query Elementary's warehouse artifact tables directly. Twelve commands across six entities (test-results,run-results,invocations,models,sources,tests) with list and get-by-id variants, backed by dedicated dbt macros underelementary_cli.-o tablefor humans), errors as{error, code, details}on stderr, exit codes0/1/2for success/user-error/system-error, and pagination via--limitwithhas_more. Adds a--profileflag so edr can point at a profile other thanelementary.edr artifacts ..., keeping stdout pipe-safe.Test plan
edr artifacts {entity}list + single-get verified end-to-end against a live Snowflake warehouse for all six entity pairs (test-results,run-results,invocations,models,sources,tests).has_moreon list truncation.{"error":..., "code":"NOT_FOUND", ...}on stderr with exit code1.-o tablerenders the documented list columns.edr artifacts --helpand spot-check one entity against their own warehouse.🤖 Generated with Claude Code
Summary by CodeRabbit
Release Notes
New Features
edr artifactscommand group for querying Elementary warehouse artifact tables