Skip to content

README.md

Latest commit

 

History

History
187 lines (137 loc) · 7.23 KB

README.md

File metadata and controls

187 lines (137 loc) · 7.23 KB
Raw

OpenFlights

Back to Microservices

An OpenFlights-based ingestion system that reads .dat datasets, publishes typed Kafka messages, consumes them, and stores the normalized result in PostgreSQL.

Stack: Java 25, Spring Boot, Spring Kafka, Spring Data JPA, PostgreSQL, Swagger/OpenAPI, Cucumber, Testcontainers

Contents

  1. Quick Start
  2. Applications
  3. Import Flow
  4. Modules
  5. Testing Notes

1. Quick Start

Back to top

1.1 Local infrastructure

Start PostgreSQL and Kafka for local development with:

docker compose -f openflights/compose.yaml up -d

Stop them with:

docker compose -f openflights/compose.yaml down

If you want a clean local reset of persisted Docker data:

docker compose -f openflights/compose.yaml down -v

This stack exposes:

Service URL / Port Notes
PostgreSQL localhost:5432 database/user/password are all openflights
Kafka localhost:9092 single-node KRaft broker

1.2 Start the applications

The OpenFlights runtime is split into three Spring Boot applications:

mvn -f openflights/openflights-kafka-producer/pom.xml spring-boot:run
mvn -f openflights/openflights-kafka-consumer/pom.xml spring-boot:run
mvn -f openflights/openflights-app/pom.xml spring-boot:run

2. Applications

Back to top

Module Port Main purpose Swagger / API docs
openflights-kafka-producer 8051 reads OpenFlights .dat files and publishes typed Kafka records http://localhost:8051/swagger-ui.html / http://localhost:8051/api-docs
openflights-kafka-consumer 8052 consumes typed Kafka records and persists them into PostgreSQL no Swagger surface
openflights-app 8050 operational/admin HTTP endpoints for persisted data http://localhost:8050/swagger-ui.html / http://localhost:8050/api-docs

Important runtime responsibilities:

  • OpenFlightsDataConfiguration wires the framework-light openflights-data readers/parsers into Spring beans and provides the dedicated import executor.
  • OpenFlightsFileImportService reads one dataset at a time and publishes records in bounded parallel chunks.
  • KafkaMessageProducer routes each record type to the correct Kafka topic and derives stable Kafka keys.
  • OpenFlightsPersistenceService owns consumer-side normalization, idempotent persistence, placeholder reference handling, and out-of-order route ingestion recovery.
  • OpenFlightsPlaceholderWriteService writes temporary airline/airport/country placeholders in separate transactions so a concurrent duplicate placeholder insert does not abort the outer route transaction.
  • AdminDataController exposes the PostgreSQL cleanup endpoint from openflights-app.
  • SqlController / SqlService / SqlRepository power the admin SQL console in openflights-app, where the backend returns structured query data and the browser renders the result table locally.

openflights-kafka-consumer and openflights-app both reuse the same datasource/JPA defaults from openflights-jpa via classpath:openflights-jpa-defaults.yaml, so the Postgres connection settings stay centralized.

3. Import Flow

Back to top

Import data through the producer endpoints in this order:

  1. countries
  2. airlines
  3. airports
  4. planes
  5. routes

Endpoints:

Dataset Endpoint
countries POST /api/v1/openflights/import/countries
airlines POST /api/v1/openflights/import/airlines
airports POST /api/v1/openflights/import/airports
planes POST /api/v1/openflights/import/planes
routes POST /api/v1/openflights/import/routes

The preferred runtime order is still the list above, even though route ingestion can now recover from missing airline and airport references by creating temporary placeholders and later overwriting them with real reference data.

Kafka topic layout:

Topic Default partitions Notes
openflights.country 1 reference data
openflights.airline 1 reference data
openflights.airport 1 reference data
openflights.plane 1 reference data
openflights.route 8 higher-throughput route ingestion

The route consumer also uses 8 listener threads by default. Those defaults can be adjusted in:

Admin cleanup

openflights-app exposes:

  • DELETE /api/v1/openflights/admin/data

That endpoint deletes persisted PostgreSQL rows only. It does not reset Kafka topics, consumer offsets, or any in-flight retry state.

Admin SQL console

openflights-app also exposes a small SQL console at:

  • http://localhost:8050/index.html

Current design notes:

  • the editor uses CodeMirror with SQL highlighting
  • the browser calls POST /api/v1/openflights/admin/sql
  • the backend returns structured JSON, not pre-rendered HTML
  • the browser renders the result table, toolbar, paging controls, and messages
  • paging is database-backed for row-returning queries, so PostgreSQL is asked for the current page window instead of returning the full result set to the app

That split keeps repositories focused on SQL execution and keeps presentation logic in the frontend where it belongs.

4. Modules

Back to top

Module Responsibility
openflights-api shared records, Kafka topic constants, low-level OpenFlights value parsing
openflights-data framework-light .dat readers and parsers
openflights-jpa JPA entities, repositories, mappers, normalizers, shared datasource defaults
openflights-kafka-producer import endpoints, file-import orchestration, Kafka publishing
openflights-kafka-consumer typed Kafka listeners, persistence orchestration, placeholder repair
openflights-app admin/operational HTTP endpoints for persisted data
openflights-testing Cucumber end-to-end tests with Testcontainers

For the full runtime and data-model breakdown, see ARCHITECTURE.md.

5. Testing Notes

Back to top

openflights-testing runs end-to-end Cucumber scenarios with Testcontainers and uses the same route topic partition count and route consumer concurrency as the normal runtime configuration.

The full-ingestion Cucumber scenario currently expects the final PostgreSQL dataset to reach:

  • countries: 260
  • airlines: 6162
  • airports: 7810
  • planes: 246
  • routes: 67663
  • routeEquipmentCodes: 93231

Those values are asserted directly in OpenFlightsIngestion.feature and cross-checked by the calculator test support.

Per-type and total ingestion timings reported by the Cucumber suite are observational only. They vary from machine to machine depending on CPU, Docker load, and local PostgreSQL/Kafka performance, and they measure when PostgreSQL reaches the expected dataset state rather than raw producer HTTP response time.