Skip to content

fix(GAT-9018): free-text linkage fallback + IndexElastic delta-row fixes - #1705

Closed
calmacx wants to merge 3 commits into
devfrom
feat/GAT-9018-2-2
Closed

fix(GAT-9018): free-text linkage fallback + IndexElastic delta-row fixes#1705
calmacx wants to merge 3 commits into
devfrom
feat/GAT-9018-2-2

Conversation

@calmacx

@calmacx calmacx commented Jul 16, 2026

Copy link
Copy Markdown
Collaborator

Stacked on feat/GAT-9018-2. Reinstates the fixes/features that were split out of PR2 because they're independent of the version-handler/DatasetService rearrangement — genuinely pre-existing bugs or new functionality, not caused by the refactor:

  • Free-text linkage fallback: dataset/publication linkages that can't be resolved to a real row are now stored with their raw external url/pid/title/doi instead of being silently dropped. New raw_url/raw_pid/raw_title columns on dataset_version_has_dataset_version, raw_doi on publication_has_dataset_version, both FKs made nullable. Gwdm2xHandler::processDatasetLinkages()/processPublicationLinkages() write the unresolved fallback; afterRead() and DatasetService::getLinkages() read it back.
  • IndexElastic delta-row fix: delta dataset_version rows store an empty metadata column by design, but ES indexing and the tool/publication extraction jobs read it directly, crashing or silently no-oping on delta rows. They now reconstruct the full envelope via DatasetService::getReconstructedMetadataEnvelope(). Adds GwdmMetadataHandler::supportsElasticIndexing() so datasets on a non-indexable GWDM version get deindexed rather than partially indexed, and moves Typesense field extraction into the handler hierarchy via toSearchableFields().
  • Spatial-coverage pivot pruning fix: mapCoverage() and Dataset::getSpatialCoverageViaVersions() now scope to the latest version and prune stale pivot rows, so editing coverage down actually removes entries instead of accumulating across versions forever.

Screenshots (if relevant)

Describe your changes

Issue ticket link

Environment / Configuration changes (if applicable)

Requires migrations being run?

If not using the pre-push hook. Confirm tests pass:

Checklist before requesting a review

  • I have performed a self-review of my code
  • I have added appropriate unit tests
  • I have created mocks for unit tests (where appropriate)
  • I have added appropriate Behat tests to confirm AC (if applicable)
  • I have added Swagger annotations for new endpoints (if applicable)
  • I have added audit logs for new operation logic (if applicable)
  • I have added new environment variables to the .env.example file (if applicable)
  • I have added new environment variables to terraform repository (if applicable)

@gh-actions-pipelines-app

Copy link
Copy Markdown

🎉 Great job! Your PR title follows the correct format. 🚀

Base automatically changed from feat/GAT-9018-2 to dev July 20, 2026 10:47
@calmacx
calmacx force-pushed the feat/GAT-9018-2-2 branch from 5562615 to b0f433c Compare July 21, 2026 08:37
calmacx and others added 3 commits July 21, 2026 10:31
Fix async metadata extraction and Elasticsearch indexing on delta (patch)
version rows, whose raw `metadata` column is empty:

- Extract{Publications,Tools}FromMetadata: reconstruct the GWDM envelope via
  DatasetService::getReconstructedMetadataEnvelope() rather than reading the
  (empty) metadata column directly.
- IndexElastic / DatasetVersion: reconstruct the latest envelope for indexing
  and gate indexing on GwdmMetadataHandler::supportsElasticIndexing().
- GwdmMetadataHandler: add supportsElasticIndexing() + toSearchableFields().
- V2 DatasetController: add ?view=metadataOnly on showActive.

Split out from the combined GAT-9018 branch; the unresolved-linkage fallback
now lives on fix/unresolved-linkages. The validateAllVersions() experiment is
dropped.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
indexElasticPublication, indexElasticDur and indexElasticCollections issued a
Dataset::where(['id' => $id])->first() + latestVersion() query per linked
dataset (and, for publications, a PublicationHasDatasetVersion lookup per row).

Preload all datasets once via whereIn with their versions eager-loaded, resolve
each latest version in memory (Dataset::latestVersion() ignores eager-loaded
relations, so it can't be used here), and batch the publication link-type
lookup with a single whereIn. Also adds a null-guard in indexElasticDur, which
previously fataled on a missing dataset/version.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
getMaterialTypes() was duplicated in IndexElastic and DatasetHydrator, each
branching on the GWDM version with an inline version_compare() — against the
convention that only GwdmHandlerFactory may switch on schema version.

- GwdmMetadataHandler: add getMaterialTypes() (base returns null).
- Gwdm1xHandler: null (legacy has no tissuesSampleCollection).
- Gwdm2xHandler: read + dedupe tissuesSampleCollection.
- IndexElastic / DatasetHydrator: resolve the handler and delegate; the inline
  version_compare() branches (and IndexElastic's dead <2.0 $containsTissue path)
  are gone.

Effective boundary shifts from 2.0 to 1.1 (the factory's split point): versions
in [1.1, 2.0) now use the modern path. Locked in by GwdmHandlerMaterialTypesTest.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant