feat(GAT-9143): Implements the first migration tests to typesense migration. Datasets and Tools. by loki-sinclair-hdruk · Pull Request #1697 · HDRUK/gateway-api

loki-sinclair-hdruk · 2026-07-13T14:30:28Z

Screenshots (if relevant)

Describe your changes

Begins the migration to Typesense. Currently only:

Datasets
Tools

Governed by two individual feature flags. First being:

V2_SearchAggregation
TypesenseSearch

The first defines whether or not the Gateway uses the new Search Provider layer to funnel off searches. The second determines the search method to use, between Elastic (the fallback) and Typesense.

Issue ticket link

Environment / Configuration changes (if applicable)

See .env.example

Requires migrations being run?

No.

If not using the pre-push hook. Confirm tests pass:

Checklist before requesting a review

I have performed a self-review of my code
I have added appropriate unit tests
I have created mocks for unit tests (where appropriate)
I have added appropriate Behat tests to confirm AC (if applicable)
I have added Swagger annotations for new endpoints (if applicable)
I have added audit logs for new operation logic (if applicable)
I have added new environment variables to the .env.example file (if applicable)
I have added new environment variables to terraform repository (if applicable)

…rch. Datasets and Tools

gh-actions-pipelines-app · 2026-07-13T14:30:39Z

🎉 Great job! Your PR title follows the correct format. 🚀

…I, thus filtering would overwrite existing filters with the one selected only

gh-actions-pipelines-app · 2026-07-14T08:23:15Z

🚨 Invalid PR title format!

Your PR title must follow one of these formats:

feat(GAT-1234): Your title
fix!(GAT-5678): Breaking change
RELEASE: vX.Y.Z

Please update your PR title accordingly. 😊

…here are additional filters left, that require computed values, which is more complex

gh-actions-pipelines-app · 2026-07-14T10:36:46Z

🚨 Invalid PR title format!

Your PR title must follow one of these formats:

feat(GAT-1234): Your title
fix!(GAT-5678): Breaking change
RELEASE: vX.Y.Z

Please update your PR title accordingly. 😊

calmacx · 2026-07-14T10:53:00Z

+    /**
+     * Material types present on the dataset (GWDM >= 2.0's
+     * metadata.tissuesSampleCollection[].materialType, excluding the
+     * "no samples" sentinel value), or null if the section is absent.
+     * Backs both containsBioSamples (boolean) and sampleAvailability
+     * (the material type list itself) — same source, two views.
+     */
+    private function materialTypes(array $meta): ?array
+    {
+        $tissues = data_get($meta, 'metadata.tissuesSampleCollection', null);
+        if (!is_array($tissues)) {
+            return null;
+        }
+
+        $types = array_values(array_filter(array_map(
+            fn ($item) => ($item['materialType'] ?? null),
+            $tissues
+        ), fn ($type) => $type !== null && $type !== 'None/not available'));
+
+        return $types === [] ? null : array_values(array_unique($types));
+    }


You might want to hold off on more changes like this until i'm done with my changes to the GWDM management system.

This is very specific to GWDM 2.0 as your comment says with a hard-coded path

as per my comment above, this would be better suited in the new GwdmHandler to retrieve materialTypes

calmacx · 2026-07-14T10:54:24Z

            'geographicLocation'            => $this->spatialCoverage->pluck('region')->all(),
            'datasetDOI'                    => data_get($meta, 'metadata.summary.doiName', ''),
            'conformsTo'                    => array_values(array_filter(explode(';,;', $conformsTo))),
+            'accessService'                 => data_get($meta, 'metadata.accessibility.access.accessServiceCategory', '') ?? '',


As in my other comment - shall we get in these changes and then put a freeze on it until mine are done?

My changes are going to have an impact on data_get --> this will all be re-written and will need to be context aware (aware of the GWDM version)

…m via typesense

gh-actions-pipelines-app · 2026-07-14T15:19:00Z

🚨 Invalid PR title format!

Your PR title must follow one of these formats:

feat(GAT-1234): Your title
fix!(GAT-5678): Breaking change
RELEASE: vX.Y.Z

Please update your PR title accordingly. 😊

gh-actions-pipelines-app · 2026-07-14T15:57:31Z

🚨 Invalid PR title format!

Your PR title must follow one of these formats:

feat(GAT-1234): Your title
fix!(GAT-5678): Breaking change
RELEASE: vX.Y.Z

Please update your PR title accordingly. 😊

gh-actions-pipelines-app · 2026-07-15T13:07:48Z

🚨 Invalid PR title format!

Your PR title must follow one of these formats:

feat(GAT-1234): Your title
fix!(GAT-5678): Breaking change
RELEASE: vX.Y.Z

Please update your PR title accordingly. 😊

gh-actions-pipelines-app · 2026-07-15T14:28:10Z

🚨 Invalid PR title format!

Your PR title must follow one of these formats:

feat(GAT-1234): Your title
fix!(GAT-5678): Breaking change
RELEASE: vX.Y.Z

Please update your PR title accordingly. 😊

gh-actions-pipelines-app · 2026-07-16T11:55:13Z

🚨 Invalid PR title format!

Your PR title must follow one of these formats:

feat(GAT-1234): Your title
fix!(GAT-5678): Breaking change
RELEASE: vX.Y.Z

Please update your PR title accordingly. 😊

… edge where a team was soft-deleted, but its associated assets hadnt been, causing the BE to crash

gh-actions-pipelines-app · 2026-07-20T12:28:46Z

🚨 Invalid PR title format!

Your PR title must follow one of these formats:

feat(GAT-1234): Your title
fix!(GAT-5678): Breaking change
RELEASE: vX.Y.Z

Please update your PR title accordingly. 😊

gh-actions-pipelines-app · 2026-07-20T13:51:51Z

🚨 Invalid PR title format!

Your PR title must follow one of these formats:

feat(GAT-1234): Your title
fix!(GAT-5678): Breaking change
RELEASE: vX.Y.Z

Please update your PR title accordingly. 😊

…s in a few NTH parts, such as admin panel for Typesense models and indexing from GW UI and the reindexing job. Also Stephen blew the scope for this and asked for a DataCustodianNetwork creation and management portal. Wedged that into the new admin portal in the FE too.

gh-actions-pipelines-app · 2026-07-20T16:26:42Z

🚨 Invalid PR title format!

Your PR title must follow one of these formats:

feat(GAT-1234): Your title
fix!(GAT-5678): Breaking change
RELEASE: vX.Y.Z

Please update your PR title accordingly. 😊

…y API call - slight improvements to knock off ~2s per call. Also adds a new command to reindex typesense easily

gh-actions-pipelines-app · 2026-07-21T09:25:13Z

🚨 Invalid PR title format!

Your PR title must follow one of these formats:

feat(GAT-1234): Your title
fix!(GAT-5678): Breaking change
RELEASE: vX.Y.Z

Please update your PR title accordingly. 😊

…ning

gh-actions-pipelines-app · 2026-07-22T12:44:49Z

🚨 Invalid PR title format!

Your PR title must follow one of these formats:

feat(GAT-1234): Your title
fix!(GAT-5678): Breaking change
RELEASE: vX.Y.Z

Please update your PR title accordingly. 😊

calmacx · 2026-07-22T13:02:17Z

+    private function facetDataTypes(): array
+    {
+        return $this->activeDatasetVersions()
+            ->flatMap(fn ($version) => explode(';,;', data_get($version->metadata, 'metadata.summary.datasetType', '') ?? ''))


very specific for GWDM - would break if this changes - will be better in the handlers

gh-actions-pipelines-app · 2026-07-22T13:02:33Z

🚨 Invalid PR title format!

Your PR title must follow one of these formats:

feat(GAT-1234): Your title
fix!(GAT-5678): Breaking change
RELEASE: vX.Y.Z

Please update your PR title accordingly. 😊

equinoxmatt · 2026-07-22T15:05:14Z

+    public function reindex(Request $request): JsonResponse
+    {
+        try {
+            $validator = Validator::make($request->all(), [


Would be nice if we validated this in a custom request object, to avoid the validation logic being in the controller.

Will discuss offline for pattern to use going forward, as per slack.

equinoxmatt · 2026-07-22T15:25:44Z

+            $modelClass = HDRUK::typesenseModelMap()[$this->entity] ?? null;
+
+            if ($modelClass === null) {
+                throw new Exception("Unknown Typesense entity '{$this->entity}'");


The failure is deterministic. If it doesn't work now, it won't work on a retry.

Consider using:

$this->fail(new Exception("Unknown Typesense entity '{$this->entity}'")); return;

equinoxmatt · 2026-07-22T15:39:51Z

+
+            $typesense->createCollectionFromModel($modelClass);
+
+            Artisan::call('scout:import', ['model' => $modelClass]);


Suggest to use $modelClass::makeAllSearchable(); - same effect as scout:import and no need to invoke the CLI.

loki-sinclair-hdruk added 3 commits July 7, 2026 16:00

initial implementation of introducing typesense into search

454a5ae

feat(GAT-9143): Implements the first migration tests to typesense sea…

4f5c010

…rch. Datasets and Tools

adds additional flags to .env.example for typesense

cd67619

loki-sinclair-hdruk requested review from a team as code owners July 13, 2026 14:30

loki-sinclair-hdruk temporarily deployed to dev July 13, 2026 14:30 — with GitHub Actions Inactive

the payload for the filtered requests wasnt properly formed in the AP…

fcb09fa

…I, thus filtering would overwrite existing filters with the one selected only

loki-sinclair-hdruk temporarily deployed to dev July 13, 2026 16:29 — with GitHub Actions Inactive

loki-sinclair-hdruk changed the title ~~feat(GAT-9143): Implements the first migration tests to typesense migration. Datasets and Tools.~~ WIP: feat(GAT-9143): Implements the first migration tests to typesense migration. Datasets and Tools. Jul 14, 2026

includes additional facets for dataset filtering - the simple ones. T…

5e84035

…here are additional filters left, that require computed values, which is more complex

loki-sinclair-hdruk temporarily deployed to dev July 14, 2026 10:36 — with GitHub Actions Inactive

calmacx reviewed Jul 14, 2026

View reviewed changes

calmacx approved these changes Jul 14, 2026

View reviewed changes

implements dataProviderColl (data custodian networks) as a search ter…

b62a2f3

…m via typesense

loki-sinclair-hdruk temporarily deployed to dev July 14, 2026 15:18 — with GitHub Actions Inactive

noticed that the checkbox filters werent working for datasets. fixed

a15c33f

loki-sinclair-hdruk temporarily deployed to dev July 14, 2026 15:57 — with GitHub Actions Inactive

abstracts the configurable config out of the core class

6dea660

loki-sinclair-hdruk temporarily deployed to dev July 15, 2026 13:07 — with GitHub Actions Inactive

adds publications to typesense collection indexing

d92d818

loki-sinclair-hdruk temporarily deployed to dev July 15, 2026 14:28 — with GitHub Actions Inactive

implements Teams (data_custodians) to typesense search

f867b17

loki-sinclair-hdruk temporarily deployed to dev July 16, 2026 11:55 — with GitHub Actions Inactive

fixes a couple of filter/facet bugs with collections - also solves an…

b1e1978

… edge where a team was soft-deleted, but its associated assets hadnt been, causing the BE to crash

loki-sinclair-hdruk temporarily deployed to dev July 20, 2026 12:28 — with GitHub Actions Inactive

applies typesense search to DURs

b43f753

loki-sinclair-hdruk temporarily deployed to dev July 20, 2026 13:51 — with GitHub Actions Inactive

loki-sinclair-hdruk temporarily deployed to dev July 20, 2026 16:26 — with GitHub Actions Inactive

saw that LogRequestResponse was actually really expensive and on ever…

c08cb0b

…y API call - slight improvements to knock off ~2s per call. Also adds a new command to reindex typesense easily

loki-sinclair-hdruk temporarily deployed to dev July 21, 2026 09:25 — with GitHub Actions Inactive

starts to tune the amount of results that can be checked before retur…

09c24d0

…ning

loki-sinclair-hdruk had a problem deploying to dev July 22, 2026 12:44 — with GitHub Actions Failure

loki-sinclair-hdruk had a problem deploying to dev July 22, 2026 12:48 — with GitHub Actions Failure

forgot to update the test when tuning the token drop off

f6c6b95

calmacx reviewed Jul 22, 2026

View reviewed changes

loki-sinclair-hdruk temporarily deployed to dev July 22, 2026 13:02 — with GitHub Actions Inactive

equinoxmatt reviewed Jul 22, 2026

View reviewed changes

equinoxmatt approved these changes Jul 22, 2026

View reviewed changes

loki-sinclair-hdruk changed the title ~~WIP: feat(GAT-9143): Implements the first migration tests to typesense migration. Datasets and Tools.~~ feat(GAT-9143): Implements the first migration tests to typesense migration. Datasets and Tools. Jul 23, 2026

loki-sinclair-hdruk merged commit 2bb2389 into dev Jul 24, 2026
2 of 3 checks passed

loki-sinclair-hdruk deleted the feat/GAT-9143 branch July 24, 2026 15:05


		$typesense->createCollectionFromModel($modelClass);

		Artisan::call('scout:import', ['model' => $modelClass]);

Uh oh!

Conversation

loki-sinclair-hdruk commented Jul 13, 2026

Screenshots (if relevant)

Describe your changes

Issue ticket link

Environment / Configuration changes (if applicable)

Requires migrations being run?

If not using the pre-push hook. Confirm tests pass:

Checklist before requesting a review

Uh oh!

gh-actions-pipelines-app Bot commented Jul 13, 2026

Uh oh!

gh-actions-pipelines-app Bot commented Jul 14, 2026

Uh oh!

gh-actions-pipelines-app Bot commented Jul 14, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gh-actions-pipelines-app Bot commented Jul 14, 2026

Uh oh!

gh-actions-pipelines-app Bot commented Jul 14, 2026

Uh oh!

gh-actions-pipelines-app Bot commented Jul 15, 2026

Uh oh!

gh-actions-pipelines-app Bot commented Jul 15, 2026

Uh oh!

gh-actions-pipelines-app Bot commented Jul 16, 2026

Uh oh!

gh-actions-pipelines-app Bot commented Jul 20, 2026

Uh oh!

gh-actions-pipelines-app Bot commented Jul 20, 2026

Uh oh!

gh-actions-pipelines-app Bot commented Jul 20, 2026

Uh oh!

gh-actions-pipelines-app Bot commented Jul 21, 2026

Uh oh!

gh-actions-pipelines-app Bot commented Jul 22, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gh-actions-pipelines-app Bot commented Jul 22, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants