[DCP - Ingestion]Adds the provenance URL#510
Conversation
Not up to standards ⛔🔴 Issues
|
| Category | Results |
|---|---|
| Security | 1 medium |
| CodeStyle | 4 minor |
| Complexity | 1 medium |
🟢 Metrics 0 complexity · 0 duplication
Metric Results Complexity 0 Duplication 0
NEW Get contextual insights on your PRs based on Codacy's metrics, along with PR and Jira context, without leaving GitHub. Enable AI reviewer
TIP This summary will be updated as you push new changes.
There was a problem hiding this comment.
Code Review
This pull request adds provenance URLs to the JSON-LD export by fetching them from the database and including them in the observation graph. Review feedback identifies a significant performance issue where the provenance URL query is executed redundantly for every data chunk, potentially causing full table scans; it is recommended to fetch this data once in the main process. Additionally, the reviewer suggests using expand_id for the URL values and adding a check for empty strings to ensure consistency and data integrity.
…ovenance URL on GraphRead.
This PR modifies the JSONLD exporter to also add the provenance URL on observation nodes, and then reads back the provenance URL from the MCFGraph, to write it in the Spanner Observation Mutation.
I published a new dataflow test template and successfully loaded the undata import in a test spanner DB.