Skip to content

Commit 8a74159

Browse files
ThorodanBromPrajwal-Amaravatitanvi029
authored
Add provider for MVT tile generation from Postgres (#1979)
* Initial commit for adding MVT Postgres provider - `/tiles` and `/tiles/{TMS}` APIs are working * Added get_tile method for MVT-postgres for on-the-fly tile generation - Supports properties - Supports both WebMercatorQuad, WorldCRS84Quad Co-authored-by: PRAJWAL S <praju18299@gmail.com> Co-authored-by: Tanvi Prasad <tanvi.prasad@datakaveri.org> * Added documentation for the MVTPostgresProvider class * Updates to MVT postgres provider - Fixed flake8 issues - Corrected tile indices in `ST_TileEnvelope` - Previous indices worked when tile URL was `/z/x/y`. Current ones work for `/z/y/x`, which is OGC-compliant - Used `ST_CurveToLine` when querying for features during tile generation to potentially speed up queries - Martin also uses it (https://maplibre.org/martin/sources-pg-functions.html#simple-function) - Used EWKT representation of `ST_MakeEnvelope(-180,-90,180,90,4326)` for WorldCRS84Quad tiles * Fix flake8 issues in `base_mvt.py` * Fix MVT-postgres documentation - fix example config * Changed name of MVT-postgres plugin to MVT-postgresql - Updated all necessary files to change postgres -> postgresql related to the plugin - Updated documentation - Added authors to plugin file and organized imports * MVT-postgresql : Add check if the indices are not in limits of the TMS, return ProviderTileNotFound error - Otherwise error was thrown from PostgreSQL `ST_MakeTileEnvelope` function * Fix flake8 errors --------- Co-authored-by: PRAJWAL S <praju18299@gmail.com> Co-authored-by: Tanvi Prasad <tanvi.prasad@datakaveri.org>
1 parent 1eb24cb commit 8a74159

4 files changed

Lines changed: 309 additions & 1 deletion

File tree

docs/source/data-publishing/ogcapi-tiles.rst

Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,7 @@ pygeoapi core tile providers are listed below, along with supported features.
2222
`MVT-elastic`_,✅,✅,✅,❌,❌,✅
2323
`MVT-proxy`_,❓,❓,❓,❓,❌,✅
2424
`WMTSFacade`_,✅,❌,✅,✅,✅,❌
25+
`MVT-postgresql`_,✅,✅,✅,✅,❌,✅
2526

2627
Below are specific connection examples based on supported providers.
2728

@@ -130,6 +131,47 @@ Following code block shows how to configure pygeoapi to read Mapbox vector tiles
130131
name: pbf
131132
mimetype: application/vnd.mapbox-vector-tile
132133
134+
MVT-postgresql
135+
^^^^^^^^^^^^^^
136+
137+
.. note::
138+
Requires Python packages sqlalchemy, geoalchemy2 and psycopg2-binary
139+
140+
.. note::
141+
Must have PostGIS installed with protobuf-c support
142+
143+
.. note::
144+
Geometry must be using EPSG:4326
145+
146+
This provider gives support to serving tiles generated using `PostgreSQL <https://www.postgresql.org/>`_ with `PostGIS <https://postgis.net/>`_.
147+
The tiles are rendered on-the-fly using `ST_AsMVT <https://postgis.net/docs/ST_AsMVT.html>`_ and related methods.
148+
149+
This code block shows how to configure pygeoapi to render Mapbox vector tiles from a PostGIS table.
150+
151+
.. code-block:: yaml
152+
153+
providers:
154+
- type: tile
155+
name: MVT-postgresql
156+
data:
157+
host: 127.0.0.1
158+
port: 3010 # Default 5432 if not provided
159+
dbname: test
160+
user: postgres
161+
password: postgres
162+
search_path: [osm, public]
163+
id_field: osm_id
164+
table: hotosm_bdi_waterways
165+
geom_field: foo_geom
166+
options:
167+
zoom:
168+
min: 0
169+
max: 15
170+
format:
171+
name: pbf
172+
mimetype: application/vnd.mapbox-vector-tile
173+
174+
PostgreSQL-related connection options can also be added to `options`. Please refer to the :ref:`PostgreSQL OGC Features Provider<PostgreSQL>` documentation for more information.
133175

134176
WMTSFacade
135177
^^^^^^^^^^

pygeoapi/plugin.py

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -53,6 +53,7 @@
5353
'MVT-tippecanoe': 'pygeoapi.provider.mvt_tippecanoe.MVTTippecanoeProvider', # noqa: E501
5454
'MVT-elastic': 'pygeoapi.provider.mvt_elastic.MVTElasticProvider',
5555
'MVT-proxy': 'pygeoapi.provider.mvt_proxy.MVTProxyProvider',
56+
'MVT-postgresql': 'pygeoapi.provider.mvt_postgresql.MVTPostgreSQLProvider', # noqa: E501
5657
'OracleDB': 'pygeoapi.provider.oracle.OracleProvider',
5758
'OGR': 'pygeoapi.provider.ogr.OGRProvider',
5859
'OpenSearch': 'pygeoapi.provider.opensearch_.OpenSearchProvider',

pygeoapi/provider/base_mvt.py

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -95,7 +95,8 @@ def get_tiles_service(self, baseurl=None, servicepath=None,
9595
:returns: `dict` of item tile service
9696
"""
9797

98-
url = urlparse(self.data)
98+
# self.data will be a dict when using MVTPostgresProvider
99+
url = urlparse(self.data) if isinstance(self.data, str) else urlparse('/') # noqa
99100
baseurl = baseurl or f'{url.scheme}://{url.netloc}'
100101
# @TODO: support multiple types
101102
tile_type = tile_type or self.format_type
Lines changed: 264 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,264 @@
1+
# =================================================================
2+
#
3+
# Authors: Prajwal Amaravati <prajwal.s@satsure.co>
4+
# Tanvi Prasad <tanvi.prasad@cdpg.org.in>
5+
# Bryan Robert <bryan.robert@cdpg.org.in>
6+
#
7+
# Copyright (c) 2025 Prajwal Amaravati
8+
# Copyright (c) 2025 Tanvi Prasad
9+
# Copyright (c) 2025 Bryan Robert
10+
#
11+
# Permission is hereby granted, free of charge, to any person
12+
# obtaining a copy of this software and associated documentation
13+
# files (the "Software"), to deal in the Software without
14+
# restriction, including without limitation the rights to use,
15+
# copy, modify, merge, publish, distribute, sublicense, and/or sell
16+
# copies of the Software, and to permit persons to whom the
17+
# Software is furnished to do so, subject to the following
18+
# conditions:
19+
#
20+
# The above copyright notice and this permission notice shall be
21+
# included in all copies or substantial portions of the Software.
22+
#
23+
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
24+
# EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES
25+
# OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
26+
# NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT
27+
# HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
28+
# WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
29+
# FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
30+
# OTHER DEALINGS IN THE SOFTWARE.
31+
#
32+
# =================================================================
33+
34+
from copy import deepcopy
35+
import logging
36+
37+
from sqlalchemy.sql import text
38+
39+
from pygeoapi.models.provider.base import (
40+
TileSetMetadata, TileMatrixSetEnum, LinkType)
41+
from pygeoapi.provider.base import ProviderConnectionError
42+
from pygeoapi.provider.base_mvt import BaseMVTProvider
43+
from pygeoapi.provider.postgresql import PostgreSQLProvider
44+
from pygeoapi.provider.tile import ProviderTileNotFoundError
45+
from pygeoapi.util import url_join
46+
47+
LOGGER = logging.getLogger(__name__)
48+
49+
50+
class MVTPostgreSQLProvider(BaseMVTProvider):
51+
"""
52+
MVT PostgreSQL Provider
53+
Provider for serving tiles rendered on-the-fly from
54+
feature tables in PostgreSQL
55+
"""
56+
57+
def __init__(self, provider_def):
58+
"""
59+
Initialize object
60+
61+
:param provider_def: provider definition
62+
63+
:returns: pygeoapi.provider.MVT.MVTPostgreSQLProvider
64+
"""
65+
66+
super().__init__(provider_def)
67+
68+
pg_def = deepcopy(provider_def)
69+
# delete the zoom option before initializing the PostgreSQL provider
70+
# that provider breaks otherwise
71+
del pg_def["options"]["zoom"]
72+
self.postgres = PostgreSQLProvider(pg_def)
73+
74+
self.layer_name = provider_def["table"]
75+
self.table = provider_def['table']
76+
self.id_field = provider_def['id_field']
77+
self.geom = provider_def.get('geom_field', 'geom')
78+
79+
LOGGER.debug(f'DB connection: {repr(self.postgres._engine.url)}')
80+
81+
def __repr__(self):
82+
return f'<MVTPostgreSQLProvider> {self.data}'
83+
84+
@property
85+
def service_url(self):
86+
return self._service_url
87+
88+
@property
89+
def service_metadata_url(self):
90+
return self._service_metadata_url
91+
92+
def get_layer(self):
93+
"""
94+
Extracts layer name from url
95+
96+
:returns: layer name
97+
"""
98+
99+
return self.layer_name
100+
101+
def get_tiling_schemes(self):
102+
103+
return [
104+
TileMatrixSetEnum.WEBMERCATORQUAD.value,
105+
TileMatrixSetEnum.WORLDCRS84QUAD.value
106+
]
107+
108+
def get_tiles_service(self, baseurl=None, servicepath=None,
109+
dirpath=None, tile_type=None):
110+
"""
111+
Gets mvt service description
112+
113+
:param baseurl: base URL of endpoint
114+
:param servicepath: base path of URL
115+
:param dirpath: directory basepath (equivalent of URL)
116+
:param tile_type: tile format type
117+
118+
:returns: `dict` of item tile service
119+
"""
120+
121+
super().get_tiles_service(baseurl, servicepath,
122+
dirpath, tile_type)
123+
124+
self._service_url = servicepath
125+
return self.get_tms_links()
126+
127+
def get_tiles(self, layer=None, tileset=None,
128+
z=None, y=None, x=None, format_=None):
129+
"""
130+
Gets tile
131+
132+
:param layer: mvt tile layer
133+
:param tileset: mvt tileset
134+
:param z: z index
135+
:param y: y index
136+
:param x: x index
137+
:param format_: tile format
138+
139+
:returns: an encoded mvt tile
140+
"""
141+
if format_ == 'mvt':
142+
format_ = self.format_type
143+
144+
fields_arr = self.postgres.get_fields().keys()
145+
fields = ', '.join(['"' + f + '"' for f in fields_arr])
146+
if len(fields) != 0:
147+
fields = ',' + fields
148+
149+
query = ''
150+
if tileset == TileMatrixSetEnum.WEBMERCATORQUAD.value.tileMatrixSet:
151+
if not self.is_in_limits(TileMatrixSetEnum.WEBMERCATORQUAD.value, z, x, y): # noqa
152+
raise ProviderTileNotFoundError
153+
154+
query = text("""
155+
WITH
156+
bounds AS (
157+
SELECT ST_TileEnvelope(:z, :x, :y) AS boundgeom
158+
),
159+
mvtgeom AS (
160+
SELECT ST_AsMVTGeom(ST_Transform(ST_CurveToLine({geom}), 3857), bounds.boundgeom) AS geom {fields}
161+
FROM "{table}", bounds
162+
WHERE ST_Intersects({geom}, ST_Transform(bounds.boundgeom, 4326))
163+
)
164+
SELECT ST_AsMVT(mvtgeom, 'default') FROM mvtgeom;
165+
""".format(geom=self.geom, table=self.table, fields=fields)) # noqa
166+
167+
if tileset == TileMatrixSetEnum.WORLDCRS84QUAD.value.tileMatrixSet:
168+
if not self.is_in_limits(TileMatrixSetEnum.WORLDCRS84QUAD.value, z, x, y): # noqa
169+
raise ProviderTileNotFoundError
170+
171+
query = text("""
172+
WITH
173+
bounds AS (
174+
SELECT ST_TileEnvelope(:z, :x, :y,
175+
'SRID=4326;POLYGON((-180 -90,-180 90,180 90,180 -90,-180 -90))'::geometry) AS boundgeom
176+
),
177+
mvtgeom AS (
178+
SELECT ST_AsMVTGeom(ST_CurveToLine({geom}), bounds.boundgeom) AS geom {fields}
179+
FROM "{table}", bounds
180+
WHERE ST_Intersects({geom}, bounds.boundgeom)
181+
)
182+
SELECT ST_AsMVT(mvtgeom, 'default') FROM mvtgeom;
183+
""".format(geom=self.geom, table=self.table, fields=fields)) # noqa
184+
185+
with self.postgres._engine.connect() as session:
186+
result = session.execute(query, {
187+
'z': z,
188+
'y': y,
189+
'x': x
190+
}).fetchone()
191+
192+
if len(bytes(result[0])) == 0:
193+
return None
194+
return bytes(result[0])
195+
196+
def get_html_metadata(self, dataset, server_url, layer, tileset,
197+
title, description, keywords, **kwargs):
198+
199+
service_url = url_join(
200+
server_url,
201+
f'collections/{dataset}/tiles/{tileset}/{{tileMatrix}}/{{tileRow}}/{{tileCol}}?f=mvt') # noqa
202+
metadata_url = url_join(
203+
server_url,
204+
f'collections/{dataset}/tiles/{tileset}/metadata')
205+
206+
metadata = dict()
207+
metadata['id'] = dataset
208+
metadata['title'] = title
209+
metadata['tileset'] = tileset
210+
metadata['collections_path'] = service_url
211+
metadata['json_url'] = f'{metadata_url}?f=json'
212+
213+
return metadata
214+
215+
def get_default_metadata(self, dataset, server_url, layer, tileset,
216+
title, description, keywords, **kwargs):
217+
218+
service_url = url_join(
219+
server_url,
220+
f'collections/{dataset}/tiles/{tileset}/{{tileMatrix}}/{{tileRow}}/{{tileCol}}?f=mvt') # noqa
221+
222+
content = {}
223+
tiling_schemes = self.get_tiling_schemes()
224+
# Default values
225+
tileMatrixSetURI = tiling_schemes[0].tileMatrixSetURI
226+
crs = tiling_schemes[0].crs
227+
# Checking the selected matrix in configured tiling_schemes
228+
for schema in tiling_schemes:
229+
if (schema.tileMatrixSet == tileset):
230+
crs = schema.crs
231+
tileMatrixSetURI = schema.tileMatrixSetURI
232+
233+
tiling_scheme_url = url_join(
234+
server_url, f'/TileMatrixSets/{schema.tileMatrixSet}')
235+
tiling_scheme_url_type = "application/json"
236+
tiling_scheme_url_title = f'{schema.tileMatrixSet} tile matrix set definition' # noqa
237+
238+
tiling_scheme = LinkType(href=tiling_scheme_url,
239+
rel="http://www.opengis.net/def/rel/ogc/1.0/tiling-scheme", # noqa
240+
type_=tiling_scheme_url_type,
241+
title=tiling_scheme_url_title)
242+
243+
if tiling_scheme is None:
244+
msg = f'Could not identify a valid tiling schema' # noqa
245+
LOGGER.error(msg)
246+
raise ProviderConnectionError(msg)
247+
248+
content = TileSetMetadata(title=title, description=description,
249+
keywords=keywords, crs=crs,
250+
tileMatrixSetURI=tileMatrixSetURI)
251+
252+
links = []
253+
service_url_link_type = "application/vnd.mapbox-vector-tile"
254+
service_url_link_title = f'{tileset} vector tiles for {layer}'
255+
service_url_link = LinkType(href=service_url, rel="item",
256+
type_=service_url_link_type,
257+
title=service_url_link_title)
258+
259+
links.append(tiling_scheme)
260+
links.append(service_url_link)
261+
262+
content.links = links
263+
264+
return content.dict(exclude_none=True)

0 commit comments

Comments
 (0)