HTM-1960: Custom extract functionality for layer attributes by mprins · Pull Request #1672 · Tailormap/tailormap-api

mprins · 2026-04-02T11:44:39Z

This is the feature branch / PR to collect all subtasks for the new /extract API

blocks:

HTM-2003: Remove unused GeoServer extension dependencies tailormap-data#131
HTM-1960: Custom extract functionality for layer attributes tailormap-viewer#1274
- HTM-1993 | HTM-1981: Implement new /extract API, cleanup old /export API tailormap-viewer#1312

github-actions · 2026-04-02T11:44:55Z

Dependency Review

The following issues were found:

✅ 0 vulnerable package(s)
✅ 0 package(s) with incompatible licenses
✅ 0 package(s) with invalid SPDX license definitions
⚠️ 5 package(s) with unknown licenses.

See the Details below.

Snapshot Warnings

⚠️: No snapshots were found for the head SHA 8f5ec0a.

Ensure that dependencies are being submitted on PR branches and consider enabling retry-on-snapshot-warnings. See the documentation for more information and troubleshooting advice.

License Issues

pom.xml

Package	Version	License	Issue Type
org.geotools:gt-csv		Null	Unknown License
org.geotools:gt-excel-writer	35-RC	Null	Unknown License
org.geotools:gt-geojson-store		Null	Unknown License
org.geotools:gt-geopkg		Null	Unknown License
org.geotools:gt-shapefile		Null	Unknown License

Allowed Licenses:
MIT, Apache-2.0, BSD-2-Clause, BSD-3-Clause, ISC, MPL-2.0, LGPL-2.1-only, EPL-2.0, EPL-1.0

Excluded from license check:
pkg:maven/org.eclipse.angus/jakarta.mail@2.0.5?type=jar, pkg:maven/jakarta.ws.rs:jakarta.ws.rs-api@4.0.0?type=jar

OpenSSF Scorecard

Package	Version	Score	Details
maven/org.geotools:gt-csv		Unknown	Unknown
maven/org.geotools:gt-excel-writer	35-RC	Unknown	Unknown
maven/org.geotools:gt-geojson-store		Unknown	Unknown
maven/org.geotools:gt-geopkg		Unknown	Unknown
maven/org.geotools:gt-shapefile		Unknown	Unknown

Scanned Files

pom.xml

codecov · 2026-04-02T12:09:01Z

❌ 1 Tests Failed:

Tests completed	Failed	Passed	Skipped
543	1	542	2

View the full list of 1 ❄️ flaky test(s)

org.tailormap.api.controller.admin.TaskAdminControllerIntegrationTest::details_of_task
Flake rate in main: 45.00% (Passed 22 times, Failed 18 times)
Stack Traces | 0.019s run time
Status expected:<200> but was:<500>

To view more test analytics, go to the Test Analytics Dashboard
_{📋 Got 3 mins? Take this short survey to help us improve Test Analytics.}

github-actions · 2026-04-02T12:09:22Z

Test Results

486 tests +7 483 ✅ +6 10m 30s ⏱️ + 5m 24s
58 suites +4 2 💤 +1
58 files +4 1 ❌ ±0

For more details on these failures, see this check.

Results for commit 8f5ec0a. ± Comparison against base commit cd7dd9f.

This pull request removes 12 and adds 19 tests. Note that renamed tests count towards both.

org.tailormap.api.controller.LayerExportControllerIntegrationTest ‑ invalid_output_format_not_accepted
org.tailormap.api.controller.LayerExportControllerIntegrationTest ‑ should_export_geo_json
org.tailormap.api.controller.LayerExportControllerIntegrationTest ‑ should_export_geo_json_with_filter
org.tailormap.api.controller.LayerExportControllerIntegrationTest ‑ should_export_geo_json_with_filter_and_sort
org.tailormap.api.controller.LayerExportControllerIntegrationTest ‑ should_export_geo_package
org.tailormap.api.controller.LayerExportControllerIntegrationTest ‑ should_export_large_filter_using_post
org.tailormap.api.controller.LayerExportControllerIntegrationTest ‑ should_export_large_filter_using_post_and_plus_in_output_format
org.tailormap.api.controller.LayerExportControllerIntegrationTest ‑ should_not_export_hidden_attributes_in_geo_json
org.tailormap.api.controller.LayerExportControllerIntegrationTest ‑ should_not_export_hidden_attributes_in_geo_json_when_requested
org.tailormap.api.controller.LayerExportControllerIntegrationTest ‑ should_return_export_capabilities_with_jdbc_feature_source
…

org.tailormap.api.controller.LayerExtractControllerIntegrationTest ‑ invalid_filter_should_fail_on_extract
org.tailormap.api.controller.LayerExtractControllerIntegrationTest ‑ invalid_filter_should_return_bad_request_on_extract_request
org.tailormap.api.controller.LayerExtractControllerIntegrationTest ‑ should_export_large_filter_to_csv
org.tailormap.api.controller.LayerExtractControllerIntegrationTest ‑ should_export_large_filter_to_excel
org.tailormap.api.controller.LayerExtractControllerIntegrationTest ‑ should_export_large_filter_to_geojson
org.tailormap.api.controller.LayerExtractControllerIntegrationTest ‑ should_export_large_filter_to_shape
org.tailormap.api.controller.LayerExtractControllerIntegrationTest ‑ should_export_large_output_to_csv
org.tailormap.api.controller.LayerExtractControllerIntegrationTest ‑ should_export_to_geopackage
org.tailormap.api.controller.LayerExtractControllerIntegrationTest ‑ should_export_wfs_to_csv_with_authentication
org.tailormap.api.controller.LayerExtractControllerLargeExcelIntegrationTest ‑ should_export_large_dataset_to_excel
…

♻️ This comment has been updated with latest results.

+
+  @GetMapping("/formats")
+  public ResponseEntity<?> formats(
+      @Valid @ModelAttribute GeoServiceLayer layer,


+  @GetMapping("/formats")
+  public ResponseEntity<?> formats(
+      @Valid @ModelAttribute GeoServiceLayer layer,
+      @ModelAttribute GeoService service,


+  public ResponseEntity<?> formats(
+      @Valid @ModelAttribute GeoServiceLayer layer,
+      @ModelAttribute GeoService service,
+      @ModelAttribute Application application,


+      @Valid @ModelAttribute GeoServiceLayer layer,
+      @ModelAttribute GeoService service,
+      @ModelAttribute Application application,
+      @ModelAttribute AppTreeLayerNode appTreeLayerNode) {


+  @GetMapping(path = "/download/{downloadId}")
+  @Counted(value = "tailormap_api_extract_download", description = "Count of layer extract downloads")
+  public ResponseEntity<?> download(
+      @ModelAttribute GeoService service,


+  @Counted(value = "tailormap_api_extract_download", description = "Count of layer extract downloads")
+  public ResponseEntity<?> download(
+      @ModelAttribute GeoService service,
+      @ModelAttribute GeoServiceLayer layer,


+  public ResponseEntity<?> download(
+      @ModelAttribute GeoService service,
+      @ModelAttribute GeoServiceLayer layer,
+      @ModelAttribute Application application,


+      @ModelAttribute GeoService service,
+      @ModelAttribute GeoServiceLayer layer,
+      @ModelAttribute Application application,
+      @ModelAttribute AppTreeLayerNode appTreeLayerNode,


outdated

Copilot

Pull request overview

This PR continues the migration from the legacy layer /export workflow to the new /extract workflow for exporting layer attributes, introducing viewer SSE support, server-side extract generation (CSV/GeoJSON/XLSX/Shapefile/GeoPackage), and updated OpenAPI/configuration plus extensive integration tests.

Changes:

Add viewer SSE endpoint (/events/{clientId}) and a dedicated viewerSseEventBus to isolate viewer SSE traffic from admin broadcasts.
Implement server-side extract creation pipeline with progress reporting and download endpoint, including new utilities (UUIDv7) and GeoTools-backed writers.
Remove legacy /export API + tests, update OpenAPI schemas/specs, configuration properties, build/deps, and add new unit/integration tests for /extract.

Reviewed changes

Copilot reviewed 31 out of 32 changed files in this pull request and generated 10 comments.

Show a summary per file

File	Description
src/test/resources/application.properties	Updates test configuration to new `tailormap-api.extract.*` settings.
src/test/java/org/tailormap/api/util/UUIDv7Test.java	Adds unit tests for UUIDv7 generation/parsing/ordering.
src/test/java/org/tailormap/api/geotools/collection/ProgressReportingFeatureIteratorTest.java	Adds unit test for iterator input validation.
src/test/java/org/tailormap/api/geotools/collection/ProgressReportingFeatureCollectionTest.java	Adds tests validating progress callbacks for decorated feature collections.
src/test/java/org/tailormap/api/controller/TestUrls.java	Removes URL constant no longer needed after export removal.
src/test/java/org/tailormap/api/controller/SseParsingUtils.java	Adds test utility helpers for parsing/counting SSE messages.
src/test/java/org/tailormap/api/controller/ServerSentEventsControllerInvalidInputIntegrationTest.java	Adds integration test for SSE clientId validation.
src/test/java/org/tailormap/api/controller/ServerSentEventsControllerIntegrationTest.java	Adds integration tests for viewer/admin SSE separation + keep-alive behavior.
src/test/java/org/tailormap/api/controller/LayerExtractControllerRestrictedFormatsIntegrationTest.java	Adds integration tests for allowed/disallowed extract formats and input validation.
src/test/java/org/tailormap/api/controller/LayerExtractControllerLargeExcelIntegrationTest.java	Adds large XLSX extract integration test and validates output.
src/test/java/org/tailormap/api/controller/LayerExtractControllerIntegrationTest.java	Adds end-to-end integration tests for multiple extract formats and error paths.
src/test/java/org/tailormap/api/controller/LayerExportControllerIntegrationTest.java	Removes legacy `/export` endpoint tests.
src/main/resources/openapi/viewer-schemas.yaml	Cleans up schema refs and removes legacy export schema.
src/main/resources/openapi/viewer-api.yaml	Removes `/export` paths and adds `/events` + `/extract` endpoints to the spec.
src/main/resources/openapi/status-responses.yaml	Adds `ServerSentEventResponse` schema for SSE payloads.
src/main/resources/application.properties	Switches from `tailormap-api.export.` to `tailormap-api.extract.` defaults and documentation.
src/main/java/org/tailormap/api/util/UUIDv7.java	Adds UUIDv7 utility for time-sortable event IDs.
src/main/java/org/tailormap/api/service/CreateLayerExtractService.java	Implements extract generation, progress/error SSE emission, and cleanup scheduling.
src/main/java/org/tailormap/api/geotools/collection/ProgressReportingFeatureIterator.java	Adds iterator wrapper to emit progress callbacks.
src/main/java/org/tailormap/api/geotools/collection/ProgressReportingFeatureCollection.java	Adds feature collection wrapper that produces progress-reporting iterators.
src/main/java/org/tailormap/api/controller/ServerSentEventsController.java	Adds viewer SSE controller and keep-alive broadcaster using UUIDv7.
src/main/java/org/tailormap/api/controller/LayerExtractController.java	Adds `/extract` controller (formats, extract start, download).
src/main/java/org/tailormap/api/controller/LayerExportController.java	Removes legacy `/export` controller implementation.
src/main/java/org/tailormap/api/controller/admin/ServerSentEventsAdminController.java	Ensures admin SSE JSON is single-line (consistent SSE framing).
src/main/java/org/tailormap/api/configuration/TailormapConfig.java	Adds `viewerSseEventBus` bean with defaults matching eventbus auto-config.
src/main/java/org/tailormap/api/configuration/base/WebMvcConfig.java	Registers enum converter for `ExtractOutputFormat`.
src/main/java/org/tailormap/api/configuration/AsyncConfig.java	Adds async executor for extract tasks.
README.md	Documents having separate admin vs viewer SSE buses/streams.
pom.xml	Updates GeoTools version/deps/repos and buildpack JVM options for native access.
build/qa/PMD-ruleset_for_TM.xml	Updates PMD config to recognize Awaitility `untilAsserted` as an assertion.
build/ci/docker-compose.yml	Minor formatting change.
.mvn/jvm.config	Enables native access for the build JVM.

…t endpoint we still need some, now obsoleted, generated code to keep the compiler happy so that is moved to a temporary schema file

…ith CSV output format

…are exceeded

For now use GeoTools 35-SNAPSHOT to make geojson export work, see https://osgeo-org.atlassian.net/browse/GEOT-7894

…ge record sets

…s don't get aministrative braodcasts

…s more robust Add --enable-native-access=ALL-UNNAMED for the native sqlite/geopackage driver

hopefully around 2026-06-01

mprins · 2026-05-12T09:31:07Z

test failure org.tailormap.api.controller.admin.TaskAdminControllerIntegrationTest::details_of_task is known to be flaky

mprins self-assigned this Apr 2, 2026

mprins added enhancement New feature or request removed breaking possibly breaking changes that need extra attention during upgrade labels Apr 2, 2026

mprins force-pushed the HTM-1960_Custom_extract_functionality_for_layer_attributes branch from 3d68eef to 4a7dc05 Compare April 2, 2026 11:48

mprins force-pushed the HTM-1960_Custom_extract_functionality_for_layer_attributes branch from 4a7dc05 to 4fd0fc0 Compare April 3, 2026 08:40

This comment was marked as resolved.

Sign in to view

mprins force-pushed the HTM-1960_Custom_extract_functionality_for_layer_attributes branch from 8c7e56f to faf5c2f Compare April 10, 2026 13:26

mprins mentioned this pull request Apr 10, 2026

HTM-1961| HTM-1962: Initial implemententation of /events and /extract endpoints with CSV output format #1677

Merged

github-code-quality Bot found potential problems Apr 14, 2026

View reviewed changes

mprins force-pushed the HTM-1960_Custom_extract_functionality_for_layer_attributes branch 2 times, most recently from b9bbb48 to e1cb81b Compare April 17, 2026 13:00

github-code-quality Bot found potential problems Apr 23, 2026

View reviewed changes

mprins mentioned this pull request Apr 23, 2026

HTM-2003: Remove unused GeoServer extension dependencies Tailormap/tailormap-data#131

Merged

1 task

mprins force-pushed the HTM-1960_Custom_extract_functionality_for_layer_attributes branch 4 times, most recently from c35d02a to 17ff46d Compare April 28, 2026 15:03

github-code-quality Bot found potential problems Apr 28, 2026

View reviewed changes

mprins force-pushed the HTM-1960_Custom_extract_functionality_for_layer_attributes branch from 3e74069 to 3ab9dea Compare April 29, 2026 12:01

mprins mentioned this pull request May 1, 2026

HTM-1993 | HTM-1981: Implement new /extract API, cleanup old /export API Tailormap/tailormap-viewer#1312

Merged

2 tasks

mprins force-pushed the HTM-1960_Custom_extract_functionality_for_layer_attributes branch 2 times, most recently from 3ef963f to 8ded6a3 Compare May 5, 2026 15:31

This was referenced May 7, 2026

HTM-2016: Create a separate SseEventBus bean for the viewer so viewers don't get administrative broadcasts #1728

Merged

HTM-1960: Custom extract functionality for layer attributes Tailormap/tailormap-viewer#1274

Merged

mprins force-pushed the HTM-1960_Custom_extract_functionality_for_layer_attributes branch from d9c823f to ee0c726 Compare May 8, 2026 19:50

mprins enabled auto-merge (rebase) May 8, 2026 19:51

Copilot AI reviewed May 8, 2026

View reviewed changes

mprins added 15 commits May 11, 2026 12:25

HTM-1975 | HTM-1976: Remove OpenAPI export endpoint and define extrac…

52b7137

…t endpoint we still need some, now obsoleted, generated code to keep the compiler happy so that is moved to a temporary schema file

HTM-1962: Add GeoTools CSV datastore

a52ae71

HTM-1961: Update /extract API, introduce and implement SSE /events API

50fbeac

HTM-1961 | HTM-1962: Initial implemententation of /extract endpoint w…

8cd7ea9

…ith CSV output format

HTM-1961 | HTM-1962: apply code review comments

8293783

HTM-1963: Introduce org.geotools:gt-excel-writer

d833e98

HTM-1963: Add Excel extract output and integration tests

1b5ab8e

HTM-1963: As defined in requirements: block export when Excel limits …

46c2c82

…are exceeded

HTM-1964: Add GeoJSON extract format (#1702)

5b148c5

For now use GeoTools 35-SNAPSHOT to make geojson export work, see https://osgeo-org.atlassian.net/browse/GEOT-7894

HTM-1965: Implement zipped shapefile extract

dafc286

HTM-1981: cleanup deprecated /export endpoint

a37e5f1

HTM-2017: Improve Excel extract by omitting column auto-sizing on lar…

c7d166c

…ge record sets

HTM-2016: Create a separate SseEventBus bean for the viewer so viewer…

19928c0

…s don't get aministrative braodcasts

HTM-1977: implement geopackage extract

c38e822

HTM-1977: Implement GeoPackage export using low-level API, which seem…

3563960

…s more robust Add --enable-native-access=ALL-UNNAMED for the native sqlite/geopackage driver

mprins force-pushed the HTM-1960_Custom_extract_functionality_for_layer_attributes branch from 99397c7 to e46618e Compare May 11, 2026 10:26

Address code review

8b3efdc

mprins force-pushed the HTM-1960_Custom_extract_functionality_for_layer_attributes branch from e46618e to 8b3efdc Compare May 11, 2026 11:21

mprins requested review from a team, KJLammers, mvdstruijk and steinkobben May 11, 2026 16:39

HTM-1964: disable GeoJSON extracts until we have a GeoTools 35.0 release

8f5ec0a

hopefully around 2026-06-01

mprins changed the title ~~[WIP] HTM-1960: Custom extract functionality for layer attributes~~ HTM-1960: Custom extract functionality for layer attributes May 12, 2026

mprins disabled auto-merge May 12, 2026 09:31

mprins merged commit 6a24203 into main May 12, 2026
29 of 33 checks passed

mprins deleted the HTM-1960_Custom_extract_functionality_for_layer_attributes branch May 12, 2026 09:31

Conversation

mprins commented Apr 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Apr 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Dependency Review

Snapshot Warnings

License Issues

pom.xml

OpenSSF Scorecard

Scanned Files

Uh oh!

codecov Bot commented Apr 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

❌ 1 Tests Failed:

Uh oh!

github-actions Bot commented Apr 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Test Results

Uh oh!

This comment was marked as resolved.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mprins commented May 12, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

mprins commented Apr 2, 2026 •

edited

Loading

github-actions Bot commented Apr 2, 2026 •

edited

Loading

codecov Bot commented Apr 2, 2026 •

edited

Loading

github-actions Bot commented Apr 2, 2026 •

edited

Loading