Skip to content

fix(GAT-9219): publisher raw-data + pid normalisation on write - #1703

Closed
calmacx wants to merge 11 commits into
devfrom
fix/GAT-9018-publisher-fix
Closed

fix(GAT-9219): publisher raw-data + pid normalisation on write#1703
calmacx wants to merge 11 commits into
devfrom
fix/GAT-9018-publisher-fix

Conversation

@calmacx

@calmacx calmacx commented Jul 15, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Screenshots (if relevant)

N/A.

Describe your changes

Fix PR — stacked on PR2 (feat/GAT-9018-2). PR #1703.

summary.publisher is now driven by the raw (post-TRASER) payload rather than synthesised from the requesting team, while its gateway identifier is normalised to a team pid — enabling publish-on-behalf and stopping id/pid corruption.

  • GwdmMetadataHandler — add resolveTeamFromGatewayId() (id-or-pid, public static) and resolvePublisherWithKeys(); buildPublisher() becomes the fallback-only builder; add abstract resolvePublisher().
  • Gwdm2xHandler / Gwdm1xHandler — resolvePublisher() (raw + normalise gatewayId/publisherId to pid + fall back to requesting team); prepareMetadata() uses it.
  • Legacy V1 inline write paths (MetadataOnboard trait, IntegrationDatasetController) apply the same normalisation via MetadataOnboard::normalisePublisher().
  • app:normalise-publisher-gateway-id backfill command — fixes snapshot envelopes and delta patch ops (gatewayId + legacy publisherId), leaves original_metadata untouched; registered in RunPostMigrations before the datasets reindex.
  • FormHydrationController — dataCustodian identifier default → team pid (+ test).

Issue ticket link

https://hdruk.atlassian.net/browse/GAT-9219

Environment / Configuration changes (if applicable)

None.

Requires migrations being run?

No schema migration. Adds a data backfill command (app:normalise-publisher-gateway-id) wired into RunPostMigrations (runs before the datasets reindex).

If not using the pre-push hook. Confirm tests pass:

  • New: PublisherResolutionTest, NormalisePublisherGatewayIdTest.

Checklist before requesting a review

  • I have performed a self-review of my code
  • I have added appropriate unit tests
  • I have created mocks for unit tests (where appropriate)
  • I have added appropriate Behat tests to confirm AC (if applicable) — N/A
  • I have added Swagger annotations for new endpoints (if applicable) — no new endpoints
  • I have added audit logs for new operation logic (if applicable) — N/A
  • I have added new environment variables to the .env.example file (if applicable) — N/A
  • I have added new environment variables to terraform repository (if applicable) — N/A

calmacx and others added 11 commits July 14, 2026 12:04
@calmacx @claude
feat(GAT-9018): add queryable gwdm_version column to dataset_versions
cb0f57d 
Adds an indexed gwdm_version column (default '2.0') to dataset_versions so
the GWDM schema version is queryable rather than only living inside the JSON
metadata envelope (absent on delta rows, unindexed on all rows). Updates the
DatasetVersion model $fillable and switches the reduced-relation selectRaw to
read the title column directly. Also widens the constrained dataset() relation
select to include team_id (consumed by team-scoped checkAccess in the V2/V3
controllers that land in later PRs of this stack).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@calmacx @claude
refactor(GAT-9018): move gwdm_version back-fill into a deployment step
4bea651 
The migration now only adds the gwdm_version column (default '2.0') + index —
no data mutation. The back-fill of existing rows from metadata->gwdmVersion moves
to database/deployment-steps/2026_07_14_140156_backfill_gwdm_version.php, run by
`php artisan deploy:run` after migrations on deploy (docker/start.sh). Run-once,
forward-only; MySQL-guarded (no-op on SQLite where the column default suffices).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@calmacx
update for lint and removing team_id
d0a6897
@calmacx
reapply
bb8609d
@calmacx @claude
refactor(GAT-9018): move gwdm_version back-fill into a deployment step
0d60476 
The migration now only adds the gwdm_version column (default '2.0') + index —
no data mutation. The back-fill of existing rows from metadata->gwdmVersion moves
to database/deployment-steps/2026_07_14_140156_backfill_gwdm_version.php, run by
`php artisan deploy:run` after migrations on deploy (docker/start.sh). Run-once,
forward-only; MySQL-guarded (no-op on SQLite where the column default suffices).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@calmacx @claude
feat(GAT-9018): GWDM version context + handler hierarchy + DatasetSer…
4d871c8 
…vice refactor (2.x)

Introduces the GWDM version-handler abstraction that the metadata write/read
paths pivot on, and refactors DatasetService to delegate all version-specific
logic to handlers resolved via GwdmHandlerFactory (the single point of GWDM
version branching — replaces scattered version_compare calls).

Behaviour-preserving for GWDM 2.x. The 3.0 handler arm is intentionally NOT
enabled here (added in a later PR of the stack, once its handler + SQL tables
exist) — SUPPORTED_VERSIONS is ['2.0','2.1'] in this PR.

Includes the review fixes:
- C1: ES index read sites reconstruct the full envelope (fixes blank search
  docs when a delta row is the latest version).
- C2: GwdmVersionContext::requestedVersion() returns null on absent header.
- H1: linkage extraction reconstructs GWDM (2.x delta rows no longer wipe linkage).
- H2: version-row insert + afterStore wrapped in a single DB::transaction().
- H3: V3 update re-enables dispatchJobs (Linkage + Term extraction).
- H4: schema-version change between versions forces a full snapshot.

Adds the unresolved-linkage columns migration + nullable-FK pivots so the
junction tables are the complete source of truth for linkage reconstruction.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@calmacx @claude
fix(GAT-9018): address PR2 review — linkage read/extract + coverage +…
7dbbc69 
… double-encode

- H-a: getLinkages() now selects raw_url/raw_title so unresolved (free-text)
  linkages render their title/url instead of null.
- H-b: getReconstructedMetadataEnvelope() gains an $applySupplementary flag;
  Gwdm2xHandler::extractLinkages() calls it with false so the junction-table
  writer no longer reads linkage back from the tables it is about to rewrite
  (prevents clobbering freshly-authored linkage on re-extraction).
- :203: assign the array metadata shape (not a json_encode string) to the
  array-cast column — fixes double-encoded metadata in the translate branch
  and clears the phpstan assign.propertyType error.
- M5: spatial coverage is now scoped to the latest version on read
  (Dataset::getAllSpatialCoveragesAttribute) and mapCoverage() prunes stale
  pivot rows per version, so editing coverage down removes entries rather than
  accumulating across versions. Resolved the DISCUSSION POINT comment.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@calmacx @claude
refactor(GAT-9018): clear phpstan level-3 findings in PR2 files
531dc18 
- getMetadataOnly(): widen @return to array{gwdmVersion?, metadata?} (the
  no-versions path returns []).
- getMetadataVersions counts: pluck+countBy instead of a count(*) aggregate
  alias, so no undefined $count property access on the Eloquent model.
- DatasetVersionHasDatasetVersion: declare @property-read dataset_id/pid, the
  attributes populated by Gwdm2xHandler::afterRead()'s join select.

No behaviour change. PR2 is now phpstan-clean.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@calmacx @claude
fix(GAT-9018-2): address branch review comments vs GAT-9018-1
2a505d0 
- IndexElastic/Gwdm2xHandler: replace inline app(DatasetService::class)
  service-locator calls with a single named datasetService() accessor
- IndexElastic: read title/shortTitle from the already-populated
  DatasetVersion columns instead of hardcoded envelope paths, falling
  back to the envelope only for legacy null rows; drops several
  envelope reconstructions that existed only to read a title
- Dataset::latestVersionID(): reformat raw SQL for readability
- Document the cross-team publish attribution gap on buildPublisher()
  (no behaviour change; investigation tracked separately)
- Gwdm21Handler now extends Gwdm20Handler so 2.0 gateway-side fixes
  propagate to 2.1 by default
- config/partners.php: PRUK version is an explicit 0.0.0 placeholder;
  HDRUK version now reads from HDRUK_CURRENT_VERSION
- Widen raw_title to text() — external linkage titles are unbounded,
  unlike the app's own enforced title columns
- Split DatasetLinkageCoverageFixTest into DatasetVersionLinkageTest
  and DatasetSpatialCoverageTest by behaviour; drop PR-review-ID
  naming from these and DatasetVersionSchemaChangeTest/
  GwdmVersionContextTest docblocks
- Remove misplaced deploy-step idempotency test from
  DatasetVersionGwdmVersionColumnTest

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YXoyJ2WdfquHtsmoTydx3x
@calmacx @claude
refactor(GAT-9018): move onboarding default-value paths into GWDM han…
d2d88f7 
…dlers

DEFAULTS_PATHS lived in FormHydrationController keyed by GWDM version
string, duplicating version branching that GwdmHandlerFactory already
owns. Move the paths to GwdmMetadataHandler::defaultValuePaths() so
the handler hierarchy is the single source of truth for version-shaped
metadata lookups, and the controller resolves them via the factory.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YXoyJ2WdfquHtsmoTydx3x
@calmacx @claude
fix(GAT-9018): publisher raw-data + pid normalisation on write, with …
0a12f9a 
…backfill

summary.publisher is now driven by the raw (post-TRASER) payload rather than
synthesised from the requesting team, while its gateway identifier is normalised
to a team pid — enabling publish-on-behalf and stopping id/pid corruption.

- GwdmMetadataHandler: add resolveTeamFromGatewayId() (id-or-pid, public static)
  and resolvePublisherWithKeys(); buildPublisher() becomes the fallback-only
  builder; add abstract resolvePublisher().
- Gwdm2xHandler / Gwdm1xHandler: resolvePublisher() (raw + normalise gatewayId/
  publisherId to pid + fall back to requesting team); prepareMetadata() uses it.
- Legacy V1 inline write paths (MetadataOnboard trait, IntegrationDatasetController)
  apply the same normalisation via MetadataOnboard::normalisePublisher().
- app:normalise-publisher-gateway-id backfill command: fixes snapshot envelopes
  and delta patch ops (gatewayId + legacy publisherId), leaves original_metadata
  untouched; registered in RunPostMigrations before the datasets reindex.
- FormHydrationController: dataCustodian identifier default -> team pid (+ test).
- Tests: PublisherResolutionTest, NormalisePublisherGatewayIdTest.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@gh-actions-pipelines-app

gh-actions-pipelines-app Bot commented Jul 15, 2026

Copy link
Copy Markdown

🎉 Great job! Your PR title follows the correct format. 🚀

@calmacx calmacx mentioned this pull request Jul 15, 2026
Merged
8 tasks
@calmacx calmacx changed the title fix(GAT-9018): publisher raw-data + pid normalisation on write, with … fix(GAT-9219): publisher raw-data + pid normalisation on write Jul 15, 2026
Base automatically changed from feat/GAT-9018-2 to dev July 20, 2026 10:47
@calmacx calmacx closed this Jul 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@calmacx