Skip to content

HTM-1961| HTM-1962: Initial implemententation of /events and /extract endpoints with CSV output format#1677

Merged
mprins merged 4 commits into
HTM-1960_Custom_extract_functionality_for_layer_attributesfrom
HTM-1961_initial_extract_endpoint
Apr 14, 2026
Merged

HTM-1961| HTM-1962: Initial implemententation of /events and /extract endpoints with CSV output format#1677
mprins merged 4 commits into
HTM-1960_Custom_extract_functionality_for_layer_attributesfrom
HTM-1961_initial_extract_endpoint

Conversation

@mprins
Copy link
Copy Markdown
Contributor

@mprins mprins commented Apr 7, 2026

HTM-1961 Powered by Pull Request Badge
HTM-1962

  • HTM-1961: Definition and initial implemententation of /events endpoint
  • HTM-1961: Initial implemententation of /extract endpoint
  • HTM-1962: CSV output format

Note: this will merge to the feature branch/PR, not the main branch

@mprins mprins self-assigned this Apr 7, 2026
@mprins mprins added the enhancement New feature or request label Apr 7, 2026
@github-actions
Copy link
Copy Markdown

github-actions Bot commented Apr 7, 2026

Dependency Review

The following issues were found:
  • ✅ 0 vulnerable package(s)
  • ✅ 0 package(s) with incompatible licenses
  • ✅ 0 package(s) with invalid SPDX license definitions
  • ⚠️ 1 package(s) with unknown licenses.
See the Details below.

Snapshot Warnings

⚠️: No snapshots were found for the head SHA 36c89fa.
Ensure that dependencies are being submitted on PR branches and consider enabling retry-on-snapshot-warnings. See the documentation for more information and troubleshooting advice.

License Issues

pom.xml

PackageVersionLicenseIssue Type
org.geotools:gt-csvNullUnknown License
Allowed Licenses: MIT, Apache-2.0, BSD-2-Clause, BSD-3-Clause, ISC, MPL-2.0, LGPL-2.1-only, EPL-2.0, EPL-1.0
Excluded from license check: pkg:maven/org.eclipse.angus/jakarta.mail@2.0.5?type=jar, pkg:maven/jakarta.ws.rs:jakarta.ws.rs-api@4.0.0?type=jar

OpenSSF Scorecard

PackageVersionScoreDetails
maven/org.geotools:gt-csv UnknownUnknown

Scanned Files

  • pom.xml

@github-actions
Copy link
Copy Markdown

github-actions Bot commented Apr 7, 2026

Test Results

489 tests  +10   488 ✅ +11   6m 44s ⏱️ + 1m 20s
 58 suites + 4     1 💤 ± 0 
 58 files   + 4     0 ❌  -  1 

Results for commit 36c89fa. ± Comparison against base commit faf5c2f.

♻️ This comment has been updated with latest results.

@mprins mprins force-pushed the HTM-1961_initial_extract_endpoint branch 3 times, most recently from 54eb6dd to 3300b1e Compare April 10, 2026 12:25
@mprins mprins force-pushed the HTM-1960_Custom_extract_functionality_for_layer_attributes branch from 8c7e56f to faf5c2f Compare April 10, 2026 13:26
@mprins mprins force-pushed the HTM-1961_initial_extract_endpoint branch from 3300b1e to 69eb8de Compare April 10, 2026 13:31
@mprins mprins changed the title [WIP] HTM-1961| HTM-1962: Initial implemententation of /extract endpoint with CSV output format HTM-1961| HTM-1962: Initial implemententation of /extract endpoint with CSV output format Apr 10, 2026
@Tailormap Tailormap deleted a comment from codecov Bot Apr 10, 2026
@mprins mprins force-pushed the HTM-1961_initial_extract_endpoint branch from 69eb8de to fc288fa Compare April 10, 2026 14:19
@mprins mprins marked this pull request as ready for review April 10, 2026 14:19
Copilot AI review requested due to automatic review settings April 10, 2026 14:19
@mprins mprins requested a review from a team April 10, 2026 14:20
@mprins mprins changed the title HTM-1961| HTM-1962: Initial implemententation of /extract endpoint with CSV output format HTM-1961| HTM-1962: Initial implemententation of /events and /extract endpoints with CSV output format Apr 10, 2026
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Implements initial Server-Sent Events (/events) and layer attribute extraction (/extract) flow, with asynchronous extract job execution and CSV output support, plus accompanying OpenAPI and integration test coverage.

Changes:

  • Added /events/{clientId} SSE subscription endpoint with scheduled keep-alives and clientId validation.
  • Added /extract endpoints for listing allowed formats, starting extracts asynchronously, and downloading generated files.
  • Introduced UUIDv7 utility for time-sortable IDs, extract file creation/cleanup logic, and updated OpenAPI + tests.

Reviewed changes

Copilot reviewed 19 out of 19 changed files in this pull request and generated 12 comments.

Show a summary per file
File Description
src/test/resources/application.properties Adds extract-related test properties and defaults.
src/test/resources/application-postgresql.properties Tunes Hikari pool size for PostgreSQL tests.
src/test/java/org/tailormap/api/util/UUIDv7Test.java Adds unit tests for UUIDv7 generation/parsing/timestamps.
src/test/java/org/tailormap/api/controller/ServerSentEventsControllerInvalidInputIntegrationTest.java Integration test for SSE invalid clientId handling.
src/test/java/org/tailormap/api/controller/ServerSentEventsControllerIntegrationTest.java Integration test for SSE keep-alive delivery.
src/test/java/org/tailormap/api/controller/LayerExtractControllerRestrictedFormatsIntegrationTest.java Integration tests for extract format restriction and invalid inputs.
src/test/java/org/tailormap/api/controller/LayerExtractControllerIntegrationTest.java End-to-end extract-to-download integration tests (CSV).
src/main/resources/openapi/viewer-api.yaml Adds/updates OpenAPI definitions for /events and /extract.
src/main/resources/openapi/status-responses.yaml Updates SSE response schema (eventType enum/details).
src/main/resources/application.properties Adds extract configuration defaults and comments.
src/main/java/org/tailormap/api/util/UUIDv7.java Adds UUIDv7 generator + timestamp extraction/parsing utility.
src/main/java/org/tailormap/api/service/CreateLayerExtractService.java Implements async extract creation, CSV datastore writing, SSE progress/error events, and cleanup job.
src/main/java/org/tailormap/api/controller/ServerSentEventsController.java Adds SSE controller and scheduled keep-alive publisher.
src/main/java/org/tailormap/api/controller/LayerExtractController.java Adds extract start/formats/download endpoints and request validation.
src/main/java/org/tailormap/api/configuration/base/WebMvcConfig.java Registers enum converter for extract output format.
src/main/java/org/tailormap/api/configuration/AsyncConfig.java Adds dedicated async executor for extract jobs.
pom.xml Adds GeoTools CSV module dependency.
build/qa/PMD-ruleset_for_TM.xml Extends PMD assertion detection to include Awaitility untilAsserted.
build/ci/docker-compose.yml Adjusts CI Postgres max_connections configuration.
Comments suppressed due to low confidence (2)

src/main/resources/openapi/viewer-api.yaml:1914

  • The OpenAPI spec defines the extract request body as application/json, but the controller uses @RequestParam (and tests send application/x-www-form-urlencoded). Generated clients will send JSON and fail to bind. Align the OpenAPI requestBody content type/shape with the actual controller contract (or switch the controller to @RequestBody).
      requestBody:
        required: true
        content:
          application/json:
            schema:
              type: object

src/main/resources/openapi/viewer-api.yaml:1925

  • The outputFormat example is still an old MIME type (application/geo+json), while the API (controller + allowed formats properties) expects values like csv, geojson, xlsx, shape. Update the example (and ideally document the allowed values explicitly) to prevent client confusion.
                outputFormat:
                  description: 'Output format.
                    The allowed formats are configured per instance using `tailormap-api.extract.allowed-outputformats`.
                    The (default) geometry is included even if not requested.'
                  type: string
                  nullable: false
                  example: 'application/geo+json'
                attributes:

Comment thread src/main/resources/application.properties Outdated
Comment thread src/test/resources/application.properties Outdated
Comment thread src/main/resources/openapi/viewer-api.yaml
Comment thread src/main/resources/openapi/viewer-api.yaml Outdated
Comment thread src/main/resources/openapi/status-responses.yaml
Comment thread src/main/resources/openapi/viewer-api.yaml Outdated
@mprins mprins force-pushed the HTM-1961_initial_extract_endpoint branch from e40ee52 to 7197946 Compare April 13, 2026 08:01
@mprins mprins requested a review from matthijsln April 13, 2026 08:07
@mprins mprins force-pushed the HTM-1961_initial_extract_endpoint branch from 7197946 to 242037a Compare April 13, 2026 08:18
@mprins mprins force-pushed the HTM-1961_initial_extract_endpoint branch from 242037a to 36c89fa Compare April 13, 2026 08:47
@codecov
Copy link
Copy Markdown

codecov Bot commented Apr 13, 2026

Codecov Report

❌ Patch coverage is 65.62500% with 99 lines in your changes missing coverage. Please review.
✅ All tests successful. No failed tests found.

Files with missing lines Patch % Lines
...ilormap/api/service/CreateLayerExtractService.java 57% 56 Missing and 9 partials ⚠️
...ilormap/api/controller/LayerExtractController.java 73% 12 Missing and 10 partials ⚠️
src/main/java/org/tailormap/api/util/UUIDv7.java 63% 7 Missing and 5 partials ⚠️

Impacted file tree graph

@@                                      Coverage Diff                                       @@
##             HTM-1960_Custom_extract_functionality_for_layer_attributes   #1677     +/-   ##
==============================================================================================
+ Coverage                                                             5%     77%    +73%     
- Complexity                                                          130     136      +6     
==============================================================================================
  Files                                                               157     161      +4     
  Lines                                                              7799    8087    +288     
  Branches                                                            715     751     +36     
==============================================================================================
+ Hits                                                                351    6220   +5869     
+ Misses                                                             7420    1427   -5993     
- Partials                                                             28     440    +412     
Files with missing lines Coverage Δ Complexity Δ
...a/org/tailormap/api/configuration/AsyncConfig.java 100% <100%> (+100%) 0 <0> (ø)
...tailormap/api/configuration/base/WebMvcConfig.java 100% <100%> (+100%) 0 <0> (ø)
...map/api/controller/ServerSentEventsController.java 100% <100%> (ø) 0 <0> (?)
src/main/java/org/tailormap/api/util/UUIDv7.java 63% <63%> (ø) 6 <6> (?)
...ilormap/api/controller/LayerExtractController.java 73% <73%> (ø) 0 <0> (?)
...ilormap/api/service/CreateLayerExtractService.java 57% <57%> (ø) 0 <0> (?)

... and 138 files with indirect coverage changes

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@mprins mprins merged commit 9bdc8a0 into HTM-1960_Custom_extract_functionality_for_layer_attributes Apr 14, 2026
21 checks passed
@mprins mprins deleted the HTM-1961_initial_extract_endpoint branch April 14, 2026 10:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants