[pull] main from firecrawl:main#210
Merged
Merged
Conversation
Detect when a monitor's scrapeOptions.formats requests JSON extraction
(either {type:"json"} or {type:"changeTracking", modes:["json"]}) and
compute the diff against the previous scrape's doc.json instead of the
markdown text. Persist the JSON field diff + a snapshot of current
values to GCS under a kind-discriminated artifact, alongside the
existing markdown diff path.
When a monitor asks for both ["json","git-diff"] modes, compute both
diffs and attach the markdown unified diff as a sidecar on the JSON
artifact so a page whose markdown changed but JSON didn't (or vice
versa) reports status:"changed". Previously this combination silently
returned "same".
Shared compute-and-persist logic extracted into diff-orchestrator.ts
so the runner and the async result recorder go through one path.
Existing markdown-only monitors are unaffected; historical artifacts
without a "kind" field read back as markdown.
Adds three E2E snip tests: JSON happy path, changeTracking-json
normalization, mixed-mode acceptance.
The /v2/monitor/:id/checks/:checkId response now returns a structured
diff object (text / json) plus a snapshot.json for JSON-mode monitors.
Type both fields in the JS and Python SDKs so callers can branch on
the artifact kind instead of inspecting `unknown`.
Additive only: no method signatures change. Existing consumers that
ignored `diff` continue to work; new consumers can read
page.diff.{text,json} and page.snapshot.json.
# Conflicts: # apps/js-sdk/firecrawl/src/v2/types.ts
- diff.ts: replace JSON.stringify object comparison with a recursive deep-equal walker. Prevents spurious "changed" verdicts when the LLM returns the same fields in a different order between runs. - diff-orchestrator.ts: treat empty markdown strings as a valid scrape result (only null/undefined trigger the missing-input fallback). A monitor of a page that legitimately renders no text now reports "same" on subsequent runs instead of a perpetual "changed". - gcs-monitoring.ts: validate the parsed artifact shape before reading `parsed.kind`. Corrupt JSON or unexpected payload types (arrays, primitives, null) now surface as "no diff" instead of throwing and breaking the entire check-detail response.
* feat(monitoring): JSON-mode diff path with snapshot and markdown sidecar
Detect when a monitor's scrapeOptions.formats requests JSON extraction
(either {type:"json"} or {type:"changeTracking", modes:["json"]}) and
compute the diff against the previous scrape's doc.json instead of the
markdown text. Persist the JSON field diff + a snapshot of current
values to GCS under a kind-discriminated artifact, alongside the
existing markdown diff path.
When a monitor asks for both ["json","git-diff"] modes, compute both
diffs and attach the markdown unified diff as a sidecar on the JSON
artifact so a page whose markdown changed but JSON didn't (or vice
versa) reports status:"changed". Previously this combination silently
returned "same".
Shared compute-and-persist logic extracted into diff-orchestrator.ts
so the runner and the async result recorder go through one path.
Existing markdown-only monitors are unaffected; historical artifacts
without a "kind" field read back as markdown.
Adds three E2E snip tests: JSON happy path, changeTracking-json
normalization, mixed-mode acceptance.
* feat(sdk): type MonitorCheckPage.diff and add snapshot field
The /v2/monitor/:id/checks/:checkId response now returns a structured
diff object (text / json) plus a snapshot.json for JSON-mode monitors.
Type both fields in the JS and Python SDKs so callers can branch on
the artifact kind instead of inspecting `unknown`.
Additive only: no method signatures change. Existing consumers that
ignored `diff` continue to work; new consumers can read
page.diff.{text,json} and page.snapshot.json.
* fix(monitoring): address PR review feedback on diff path
- diff.ts: replace JSON.stringify object comparison with a recursive
deep-equal walker. Prevents spurious "changed" verdicts when the LLM
returns the same fields in a different order between runs.
- diff-orchestrator.ts: treat empty markdown strings as a valid scrape
result (only null/undefined trigger the missing-input fallback). A
monitor of a page that legitimately renders no text now reports "same"
on subsequent runs instead of a perpetual "changed".
- gcs-monitoring.ts: validate the parsed artifact shape before reading
`parsed.kind`. Corrupt JSON or unexpected payload types (arrays,
primitives, null) now surface as "no diff" instead of throwing and
breaking the entire check-detail response.
* Nick: other sdks
---------
Co-authored-by: Nicolas <20311743+nickscamara@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
See Commits and Changes for more details.
Created by
pull[bot] (v2.0.0-alpha.4)
Can you help keep this open source service alive? 💖 Please sponsor : )