Tilezen loves contributions from community members like you! Contributions come in many different shapes and sizes. In this file we provide guidance around two of the most common types of contributions: opening issues and opening pull requests.
We ask that you are respectful when contributing to Tilezen or engaging with our community. As a community, we appreciate the fact that contributors might be approaching the project from a different perspective and background. We hope that beginners as well as advanced users will be able to use and contribute back to Tilezen. We want to encourage contributions and feedback from all over the world, which means that English might not be a contributor's native language, and sometimes we may encounter cultural differences. Constructive disagreements can be essential to moving a project forward, but disrespectful language or behavior will not be tolerated.
Above all, be patient, be respectful, and be kind!
Most issues for Tilezen are housed in the Tilezen/vector-datasource repo. Before opening an issue, be sure to search the repository to see if someone else has asked your question before. If not, go ahead and open a new issue.
When submitting bug reports, please be sure to give us as much context as possible so that we can reproduce the error you encountered. Be sure to include:
- System conditions (operating system, browser, etc), if you're running from source
- Steps to reproduce
- Expected outcome
- Actual outcome
- Screenshots, if applicable
- Code that exposes the bug, if you have it (such as a failing test or a basic script)
It's important to get feedback about the quality of local tile results. Your local knowledge will make it easier for us to understand the problem. When submitting issues be sure to include details like:
- Where in the world you were looking:
- A placename and zoom, or
- Zoom, latitude, and longitude, or
- Tile URL, e.g.: https://tile.mapzen.com/mapzen/vector/v1/all/10/518/352.topojson, with the feature ID and/or tile layer, if available
- Your expected result!
- Your actual result :(
Tilezen has several miscellaneous standards:
- we follow PEP8 coding style for Python and use Flake8 to enforce those conventions
- we love tests, check them out
- we use CircleCI for continuous integration testing
- we use semver for package versioning
All unit tests in a project will be automatically invoked when you commit to an existing project; make sure they pass!
We'll gladly invite active contributors to become members of the Tilezen organization. New members will gain direct write permissions, and with great power comes great responsibility.
Generally speaking there are three aspects of developing vector tiles.
- Configuring project setup, see wiki page
- Updating database properties (can be done ahead of time or at runtime)
- Changing how features are selected from the database (requires tileserver restart)
Map database in Postgres stores data from OpenStreetMap and other projects like Natural Earth and Who's On First.
When data is loaded, database triggers calculate if a feature is included in which layer(s), at what "minimum zoom", and other Mapzen specific "mz" properties.
When modifying the logic below, we'll need to update our Postgres functions, migrate the data, and cut new tiles.
- Vector tiles (or just "tiles") allow bite sized access to large geographic databases with "raw" vector geometries and properties instead of rendered raster images.
- Map features (or just "feature") are individual map features, commonly with a name, geometry, and additonal properties like unique ID and source. Example include individual roads, landuse polygons, or business icons.
Tile layers are configured in a root queries.yaml file. This file specifies which jinja template to use per layer, and also specifies post-processing via Python transforms. Individual database features are "filtered" into tiles per layer based on yaml files.
Typical tile content changes occur at the yaml level, but there are 4 levels total:
- yaml files determine which features are included in a tile layer by specifying a series of source data filters and property value rules, the most import of which are a feature's
kindandmin_zoom. - jinja templates These filter & property rules get folded into sql functions, which are generated via layer templates. Some sql functions are run in the database before a tile is requested (for properties like
min_zoom) and others are run as a tile is requested (likekind). - Python post-processing occurs per feature and across layers once a set of features has been returned for a given tile, which is useful for more involved logic.
- layers are specified in the root
queries.yaml. This file specifies which jinja template to use per layer, and also specifies per layer post-processing Python transforms.
The yaml configuration files establish which features are included per layer, and the jinja templates are better suited for rules that apply to all features in a layer.
To recap, with examples:
- yaml files are located in the
yaml/directory. Example: pois.yaml - jinja files are located in the
queries/directory. Example: pois.jinja2. - Python files are located in the
vectordatasource/directory. Example: transform.py. - layers are specified in queries.yaml.
Listens for API requests on localhost, which are in the format of layer/z/x/y.ext.
When tileserver hears a request it asks Postgres for "the stuff" inside that tile's bounding box, configured via the queries.yaml layers file, jinja2 templates, Python transforms, and per feature sql functions generated from the yaml filter files.
Other considerations
- yaml updates don't require restart (but do require a database migration, see below).
- jinja updates don't require restarting tileserver; they are re-read on request during development.
- Python updates don't require restarting tileserver; they are re-read on request during development.
DATABASE MIGRATION: Changes to layer yaml files will require at a minimum reloading the sql functions. This is sufficient if only the kind or any output properties have changed. But for min_zoom changes the affected features will need to be recalculated, probably via a data migration. This topic is covered in further detail below.
We'll cover the following topics in the next sections:
- Choose an issue to work on
- Do your work in a local branch
- Create a new test
- Edit database &/or query logic
- Verify the new logic by running the test
- Perform any modifications, as necessary
- Update data migrations
- Update documentation
- Push your local branch to the server
- Submit a Pull Request (PR)
1. Install prerequisite tool pre-commit package manager on your machine
Since the repo right now uses python2.7, you need to install pre-commit using python2.7.
Then enable it by doing pre-commit install.
We have a backlog of issues, but they are also grouped into milestones and tracked with Waffle board.
When picking an issue from the Ready column for the active milestone, self assign it to let other people know you'll be working on it and move it to the In Progress column.
If you propose to work on an issue in the Backlog but what to confirm some details add a comment to the issue or ask about it in Slack.
Ensure you're on the master branch to establish a clean history:
git checkout masterEnsure your master branch is up-to-date with the server:
git pullYou will need a descriptive name for your new branch, and one way to do that is concatenate your user name, any relevant issue number(s) and a brief description, for example olga/875-camp-ground-zoom. Then create a new branch using git checkout -b <branchname>, like so:
git checkout -b olga/875-camp-ground-zoomCongratulations, now you have a local branch!
We're going to push our work to the server eventually (so other people can see your work, and so you have a backup), so let's make sure that works now:
git pushNOTE: Your first push for a branch might require additional details:
git push --set-upstream origin olga/875-camp-ground-zoomCreate a new test for the issue in integration-test dir. Sometimes it's helpful to look thru the existing tests to find one that is a close match to the pattern and start there.
- Create new test file
- You'll need a specific OpenStreetMap feature ID to test against
- You'll need the coordinates (z/x/y) of a map tile containing that feature
- Run the test
The unit tests are written using the unittest framework, which has been subclassed in FixtureTest to provide some useful methods. This means that each test starts with the import of OsmFixtureTest, and defines a test class. These make it slightly harder to see what's going on, but are necessary to fit into the way unittest structures tests.
from . import FixtureTest
class CampGroundZoom(FixtureTest):
def test_camp_ground_in_landuse_layer(self):
self.load_fixtures([
'http://www.openstreetmap.org/way/431725967',
])
self.assert_has_feature(
16, 10959, 25337, 'landuse',
{'kind': 'camp_site'})Example test run:
In the vector-datasource directory in your first terminal window, run your new test to make sure it fails using the existing config:
python integration-test/__init__.py integration-test/875-camp-grounds-zoom.pyOnce it fails, we'll update our logic in step 4 below so it passes.
Now the gory details...
There are two options to identify test features:
- Query local database using psql on the command line or PGAdmin app.
- Query remote OpenStreetMap database using Overpass Turbo.
Confused about which tags to use? Read up on the OSM wiki (example) and confirm actual usage in TagInfo.
To find an example feature in OpenStreetMap search overpass-turbo for specific tags. Here's a sample query (assuming you've zoomed the map to an interesting area like the greater San Francisco metropolitan area):
/*
This has been generated by the overpass-turbo wizard.
The original search was:
“highway=rest_area”
*/
[out:json][timeout:25];
// gather results
(
// query part for: “highway=rest_area”
node["highway"="rest_area"]({{bbox}});
way["highway"="rest_area"]({{bbox}});
relation["highway"="rest_area"]({{bbox}});
);
// print results
out body;
>;
out skel qt;
Once you find a result you like, click on it's map marker to pull up the info window. Following the link from Overpass take you to a page like:
On that new web page, zoom the map out to the desired min zoom of the feature (it's usually specified in the Issue description in Github), then right click on the map near the marker (but not on the marker!). Then you'll use your web browsers debug tools to "Inspect element" and look for the leaflet-map-pane and follow that down till you find the named raster tile file which encodes the tile coordinate.
Use one of the Mapzen house styles, like Bubble Wrap, to determine the tile:
Click on a feature to "view more", then click "view tile data".
If you're modifying a feature, it can be helpful to search in the JSON response for the thing you want to change to confirm it's the right tile. If you're adding a new feature, you could search for something you know should be in the tile already to confirm you got the right one.
Specific map tile to test with:
But the tests require this to be formatted like:
7, 20, 49
The FixtureTest class provides several useful tests (called using self.):
assert_has_featureassert_no_matching_featureassert_at_least_n_featuresassert_less_than_n_featuresassert_feature_geom_type
Edit the YAML file corresponding to the layer. In this case we're modifying the landuse.yaml to add a new filter that looks for OpenStreetMap feature tagged tourism=camp_site and assigns them a min_zoom based on the feature area of at least 16 but up to zoom 13 depending on the feature's area and assigning a Tilezen kind of camp_site.
- filter: {tourism: camp_site}
min_zoom: GREATEST(LEAST(zoom, 16), 13)
output: {kind: camp_site}Run the test, hopefully it passes now! You'll need to run the test from the project's root directory, you may need to cd ../../ to get back there after step 4 above.
python integration-test/__init__.py integration-test/875-camp-grounds-zoom.pyExample output:
python integration-test/__init__.py integration-test/875-camp-grounds-zoom.py
..
----------------------------------------------------------------------
Ran 2 tests in 2.236s
OKIf the test failed like so:
python integration-test/__init__.py integration-test/875-camp-grounds-zoom.py
.F
======================================================================
FAIL: test_small (integration-test.875-camp-grounds-zoom.CampGroundsZoom)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/matt/Programming/Mapzen/vector-datasource/integration-test/875-camp-grounds-zoom.py", line 34, in test_small
{'kind': 'camp_site', 'sort_rank': 92, 'foo': 'bar'})
File "/home/matt/Programming/Mapzen/vector-datasource/integration-test/__init__.py", line 1212, in assert_has_feature
self.assertions.assert_has_feature(z, x, y, layer, props)
File "/home/matt/Programming/Mapzen/vector-datasource/integration-test/__init__.py", line 1079, in assert_has_feature
(properties, closest['properties'], misses))
AssertionError: Did not find feature including properties {'sort_rank': 92, 'kind': 'camp_site', 'foo': 'bar'}. The closest match was {'kind': 'camp_site', 'area': 29753, 'sort_rank': 92, 'source': 'openstreetmap.org', 'min_zoom': 13.0, 'id': 417405356}: missed {'foo': "None != 'bar'"}.
----------------------------------------------------------------------
Ran 2 tests in 2.214s
FAILED (failures=1)Rinse and repeat, rewrite your code. Don't be afraid to ask for help!
Once you've finished testing your new database logic in step 4 above you need to record that that same SQL in modified form in data/migrations/ to ensure someone with an earlier database configuration can catch up with you. (Migrations are reset for each Tilezen release.)
Continuing the camp_site example, edit the following in the data/migrations/v1.0.0-polygon.sql file:
UPDATE
planet_osm_polygon
SET mz_poi_min_zoom = mz_calculate_min_zoom_pois(planet_osm_polygon.*)
WHERE
(barrier = 'toll_booth' OR
highway IN ('services', 'rest_area') OR
tourism = 'camp_site')
AND COALESCE(mz_poi_min_zoom, 999) <> COALESCE(mz_calculate_min_zoom_pois(planet_osm_polygon.*), 999);
UPDATE
planet_osm_polygon
SET mz_landuse_min_zoom = mz_calculate_min_zoom_landuse(planet_osm_polygon.*)
WHERE
(highway IN ('services', 'rest_area') OR
barrier IN ('city_wall', 'retaining_wall', 'fence') OR
historic = 'citywalls' OR
man_made = 'snow_fence' OR
waterway = 'dam' OR
tourism = 'camp_site' OR
"natural" IN ('forest', 'park'))
AND COALESCE(mz_landuse_min_zoom, 999) <> COALESCE(mz_calculate_min_zoom_landuse(planet_osm_polygon.*), 999);Running the migration script will have the effect of building and re-installing the SQL functions which are responsible for calculating the min zoom columns used to determine if a feature is visible at all. If you find that your tests are passing, but your live database doesn't show a feature in tiles (or shows it at the wrong zoom), then you may need to run a migration again.
Migrations should be idempotent: Running them multiple times should result in the same changes. This is important, as the migrations may be run several times on the development database and many, many times against a local database. This can lead to some complex code to ensure that actions aren't performed twice. Some actions, such as UPDATEs, may be safe to perform multiple times. Even in those cases, it's best to add a restrictive WHERE clause to avoid making unnecessary writes, as these can hugely slow down the migration process.
OpenStreetMap related migrations are recorded in the following files:
v1.0.0-point.sqlv1.0.0-line.sqlv1.0.0-polygon.sql
Migrations for other data sources like Natural Earth and Who's On First go in:
v1.0.0-other-tables.sql
Here's an example out of the v1.0.0-point.sql file:
Updating a simple point feature:
UPDATE planet_osm_point
SET mz_poi_min_zoom = mz_calculate_min_zoom_pois(planet_osm_point.*)
WHERE shop IN ('outdoor');A more complicated point example:
UPDATE
planet_osm_point
SET mz_poi_min_zoom = mz_calculate_min_zoom_pois(planet_osm_point.*)
WHERE
(barrier = 'toll_booth' OR
highway IN ('services', 'rest_area'))
AND COALESCE(mz_poi_min_zoom, 999) <> COALESCE(mz_calculate_min_zoom_pois(planet_osm_point.*), 999);Updating a simple line feature:
UPDATE planet_osm_line
SET mz_boundary_min_zoom = mz_calculate_min_zoom_boundaries(planet_osm_line.*)
WHERE
waterway = 'dam';Updating a simple polygon feature:
UPDATE planet_osm_polygon
SET mz_poi_min_zoom = mz_calculate_min_zoom_pois(planet_osm_polygon.*)
WHERE shop IN ('outdoor');When we calculate both the POIs and the landuse min zoom:
UPDATE planet_osm_polygon
SET mz_poi_min_zoom = mz_calculate_min_zoom_pois(planet_osm_polygon.*),
mz_poi_min_zoom = mz_calculate_min_zoom_landuse(planet_osm_polygon.*)
WHERE shop IN ('outdoor');Everything good? time to update the docs! Generally this is in the docs/layers.md file in the various layer sections to specify new properties and new kind values.
Since camp_site was already in the pois layer, we only need to document it's addition to the alphabetical list of landuse kinds:
* `bridge`
* `camp_site`
* `caravan_site`
First let's commit our changes. Let's confirm which files changed:
git statusYou can also do a git diff on each file to determine if you meant to change or insert logic. Once you've confirmed the changes...
For each, commit using a specific commit message. The first should use the "Connects to #issuenum" format to link up the PR to the original issue in Waffle.io.
git commit -m 'Connects to #875 to add camp_site polygons' filenameNOTE: Subsequent commit messages can be more generic. Also, pre-commit hooks check the code styles at this point by default, many linters should auto fix the issues they identified so you would just git commit again and those errors would be gone, however certain linters require your manual effort to fix them. If you don't fix them your pull request will fail the style check later.
Make sure you have a clean merge by pulling down the latest master by checking out master:
git checkout masterFetch latest changes from the server:
git pull origin masterGo back to your branch:
git checkout olga/875-camp-ground-zoomRebase (compare) it with master:
git rebase masterAnd resolve any funk, as necessary.
Then push to the server so other people can see your work. (If this is a large change over multiple days, please push the server once a day so your work is backed up.)
git pushNOTE: Your first push for a branch might require additional details:
git push --set-upstream origin olga/875-camp-ground-zoomBack on Github.com load the project page and notice there's a button suggested you create a PR for your active branch. Press that green button. Need help? Github docs have you covered.
In the PR form, give it a good title that ties in with the original Issue title. In the comment section summarize the work you did to resolve the issue and indicate you added tests, data migrations, and updated the documentation.
A Tilezen team member will review the PR for you, either merging it right away or following up with questions.
If the review leads to code modifications those should be done in same branch and the PR will automatically update with subsequent commits to the branch.