fix: handle NUL byte in output_dataclip on step:complete#4910
Conversation
Detect an unstorable NUL byte (0x00) in the decoded output_dataclip body
before inserting the Dataclip and return an {:error, changeset} instead of
letting Repo.insert raise an unhandled Postgrex.Error (22P05). PostgreSQL
cannot store 0x00 in text/jsonb, so the prior code crashed the whole
step:complete reply. Detecting before insert avoids poisoning the
surrounding Repo.transact transaction.
Closes OpenFn#4893
|
Hey @mvanhorn thank you so much for your contribution! Our team is a little swamped at the moment but we'll check out your PR as soon as we can. |
|
heya @mvanhorn, great catch. We've actually just spotted this on our end too. Could you tell us a bit about the user story / scenario that caused the issue for you? We're seeing it when working with spreadsheet. Curious to learn if other behaviours might trigger too. |
|
Honest answer: I came at this from the Sentry error (LIGHTNING-13C) rather than a specific workflow I was debugging, so I don't have one clean repro to point at. The trigger is anything that lands a raw |
|
This is being blocked by OpenFn/kit#1478 . Lets wait for the worker implementation |
|
@theroinaochieng does this still require my review or will this be handled by Frank? Seems quite Elixir focused. Happy to steer a Claude review as an extra pair of eyes but not sure if @doc-han might be able to add more value. |
Description
This PR fixes a crash on
step:complete. When a worker reports anoutput_dataclipwhose decoded JSON contains a NUL byte (0x00),Lightning.Runs.Handlers.CompleteStepraised an unhandledPostgrex.Error(code
22P05,untranslatable_character) while inserting theDataclip,because PostgreSQL cannot store
0x00intext/jsonb. The raise propagatedout of
maybe_save_dataclip/2and failed the whole reply, leaving the step/runwithout a clean completion (Sentry LIGHTNING-13C).
The fix detects the unstorable NUL byte before the insert in the third
maybe_save_dataclip/2clause and returns{:error, changeset}instead ofletting
Repo.insert/1raise. Detecting before the insert (rather than rescuingthe
Postgrex.Error) keeps the surroundingRepo.transact/1transaction frombeing poisoned, and
CompleteStep.call/2's existingwithalready propagatesthe error tuple to the caller. Worker-side handling of the error response lives
in a separate repo and is out of scope.
Two details worth calling out:
Dataclip.new/1strips the unpersisted top-level
configurationkey viaremove_configuration/1, so the guard readsEcto.Changeset.get_field(changeset, :body)after that stripping. A NUL bytethat lives only under
configurationnever reaches PostgreSQL and is allowedthrough.
LightningWeb.ChannelHelpers.reply_with/2sendsinspect(error)to Sentry,so a changeset carrying
changes.bodywould leak the decoded job output intoSentry. The error changeset carries only the
:bodyerror message.Closes #4893
Validation steps
mix test test/lightning/runs_test.exs- three new tests in thedescribe "complete_step/2"block cover: a NUL byte at the top level returns{:error, %Ecto.Changeset{}}(no raise), a NUL byte nested in an arrayelement is also caught, and a NUL byte present only under top-level
configurationstill completes with{:ok, _step}(since that key is notpersisted).
complete_step/2happy-path tests still pass unchanged (normaloutput dataclips insert and complete as before).
Additional notes for the reviewer
contains_null_byte?/1) walks decoded JSON maps, lists,and binaries, checking both keys and values, so it catches NUL bytes at any
depth.
than mutate user data). The worker not yet respecting an error response on
step:completeis tracked separately.AI Usage
Please disclose whether you've used AI anywhere in this PR (it's cool, we just
want to know!):
You can read more details in our
Responsible AI Policy
Pre-submission checklist
/reviewwith Claude Code)
(e.g.,
:owner,:admin,:editor,:viewer) - n/a, this is aninternal data-persistence guard with no authorization surface