Determine Where to Continue Hosting the Map Layer Spreadsheets

[allella]

The map layer spreadsheets were moved recently to an OpenCollective owned Google Workspace that's associated with Code With the Carolinas

These files were previously in a Code For Greenville Google Workspace and Code For America killed that in June 2023 and we agreed to merge into Code For / With the Carolinas as a Shared Drive in the meantime. That Shared Drive is now empty and the files still exist and are shared with some people, but as read-only.

I'm not sure it's worth asking them to setup permissions when could just move these under a RefactorGVL Shared Drive if Code With The Carolinas doesn't see a way those layers fit into things they are doing.

[allella]

It was mentioned at the HG Labs meeting that we could consider putting the demo map under hackgreenville.com.

We probably need a broader strategy with what to do with the map layers before we put much effort into anything maps related.

[allella]

@oliviasculley @ThorntonMatthewD @pamelawoodbrowne
I'm posting this in GitHub to summarize much of the Slack conversation and so the highlights aren't lost of the Slack conversation stalls out.

Migrating Off data.openupstate.org

We're having a conversation in #hg-labs channel on the HackGreenville Slack about migrating off of data.openupstate.org since that web server is at end-of-life and Code With The Carolina's split from OpenCollective's Google Apps support means we need to either move our map spreadsheets again, or else find another solution.

Below are some rough page view stats for the map layer pages (not the map previews) for the last 12 months. The SRT layers are the most used, along with breweries, coworking, and then the long-tail.

Doing Away with Most / All Spreadsheets

We're talking about possibly doing away with the spreadsheets as the default and having any contributors create a pull request to edit specific GeoJSON files under this GitHub repository, or a new HackGreenville GitHub repo if we don't want to mix the demo map repo with the GeoJSON layers.

It's still possible that we could keep select layers in a Google Sheet and use the current technique to grab a CSV link that gets convered to a GeoJSON file, and then commit the result to the GitHub repo through a Pull Request. Though, unless / until we have more contributors, it seems I'm one of the only people using the spreadsheets to edit data (breweries, coworking), and edit a GeoJSON file in not much more time.

Swamp Rabbit Trail Layers and OSM

We're talking about how to get some of our data, like Mile Markers, into OpenStreetMap (OSM), and how to potentially use OSM as a "primary" / source of truth" for some of the Swamp Rabbit Trail (SRT) layers.

If we treated OSM as the main "source of truth", and assuming we can stay within their licensing, then we'd probably occasionally export specific GeoJSON point data and putting that GeoJSON file into the GitHub repo to make it easy for apps, like our demo map and Mike's map site, to keep linking to syndicated GeoJSON URLs that are relatively fresh (enough) based on community curated data from OSM's database.

I played around with OSM's Overpass query language and did a basic test of isolating "Water Fountains Around the Swamp Rabbit Trail". You can export directly to GeoJSON and it even provided a raw GeoJSON link to the Overpass API.

Redirecting Old Pages to New, Releveant Domains

We're also talking about redirecting the individual map layer pages (ex. data.openupstate.org/map/swamp-rabbit-trail-water-fountains) to sunset the old Drupal site and data.openupstate.org.

Mike has a long-running map domain that's focused on the SRT. The thought is to redirect our SRT preview pages to Mike's domain and he would possibly / probably add some of the GitHub hosted GeoJSON point data map layers to his SRT map. That would give people the visual OpenStreetMap map tiles, which have a lot of detail about SRT features (in large part due to Mike's contributions to OSM), plus the ability to toggle on clickable point data for specific layers. This would be particular useful if people want to see "all the mile markers" or "all the parking" along the SRT, rather than a localized set of features based on what's rendered into the OSM tiles.

We could 301 redirect the GeoJSON URLs to an equivalent in GitHub, but it might actually be good for both known and unknown users of our URLs to update to the new GitHub URLs.

For semi-popular layers, like co-working and breweries, we may find local domains to host a layer. For example, perhaps OpenWorks hosts a copy of the coworking layer.

Discarding Unpopular, Stale, Incomplete Layers

We'll likely discard some older map layers from SC Codes students, particularly if they are have one or more strikes against them (stale, incomplete, inactive maintainer, low/no traffic / interest).

GCGIS iMap Syndicated Layers

We may continue to syndicate some of the GCGIS iMap layers from the iMAP. I already have a cron job that pulls the GeoJSON and saves a copy, so I could very well run the job every so often and do a Git commit with any changes. The only issue I've noted is a couple layers have points with no lat/long, which i think breaks something in Leaflet. I contacted the maintainers years ago and they never replied. Also, I suspect the iMap project is either not actively maintained, or will be dumped soon enough. So, I'm not sure how much effort to put into syndicating things from GCGIS.

I have some other thoughts regarding how to keep things in sync between the various "sources of truth", but I'll create a separate thread for them.

[allella]

The Swamp Rabbit Trail (SRT) has a number of potential / conflicting "sources of truth" for some of the layers. While we can't control all of them, we can at least try to not make matters worse by competing with sources that are likely to be more accurate.

Wth the GCGIS data that we syndicated from iMap, we stopped maintaining our own (student built) "recycling center" layer and just trusted that the County had a trustworthy dataset that we could syndicate.

It's probably worth considering each layer and which source is most likely to be well maintained. Then, for the GeoJSON point data, we can syndicate using cron jobs, or some such, that pull the latest data, convert it to GeoJSON (if needed), and push changes to the GitHub repo.

SRT Examples

As an example with the SRT, we have things like water fountains in

• OpenStreetMap tiles and the Overpass API
• Greenville City's GIS (map / data )
• (old) SC Codes student water fountain point data

Now, the City of Greenville may not have a complete list if they were only putting features within the City, so perhaps OSM would be the most complete "source of truth" for such things.

Similarly, the SRT Mile Markers are listed

• GC GIS ArcGIS system (lists the path and mile markers)
• SC Codes student point data
• I'm assuming Mike will add the Mile Markers to OSM, if he hasn't already.

Asides

GCGIS has other data, some of which we've already / currently syndicated from the iMap layer using a trick to get all data points

• https://www.gcgis.org/website/AppHelp/ExportLayers/ExportableLayers.html
• https://www.gcgis.org/arcgis/rest/services

This is from 2020, but someone was talking about importing City GIS into OSM). I don't think they were trying to automate since they said " "stare and compare". Plus, OSM has a policy on automated import.

[allella]

Saving this Overpass Turbo example of finding all water fountains within 100 meters of the SRT.

[out:json][timeout:25];

// fetch area "Greenville" to search in
{{geocodeArea:Greenville}}->.searchArea;

node[amenity=drinking_water](area.searchArea)->.water;

relation["name"="Swamp Rabbit Trail"](area.searchArea)->.srt;

// water fountains within 100 meters of the SRT ways
node.water(around.srt:100)->.fountainsNearSRT;

// print the union of nearby fountains and the SRT paths
(.fountainsNearSRT; .srt;);

out body;
>;
out skel qt;

Or, using relationship ID

[out:json][timeout:25];

// fetch area “Greenville” to search in
{{geocodeArea:Greenville}}->.searchArea;

node[amenity=drinking_water](area.searchArea)->.water;

// relationship id 167019 is for the various SRT "ways"
// https://www.openstreetmap.org/relation/167019#map=12/34.8555/-82.3569&layers=G
rel(167019)->.swampRabbitTrail;

// water fountains within 100 meters of the SRT ways
node.water(around.swampRabbitTrail:100)->.fountainsNearSRT;

(.fountainsNearSRT; .swampRabbitTrail;);

out body;
>;
out skel qt;

[allella]

I connected with Bennett Meares today and CC'ed folks on that email, but I've copied it below.

Bennett,
It was nice catching up this evening at the Alt.computing meetup at OpenWorks.
Here's a summary on GitHub where we were considering migrating the more popular layers to specific applications and sunsetting others.  The main motivation is we promote the availability of 70+ map layers on a deprecated Drupal site and server that needs to be sunset before long.  

This site includes a "Map Preview Link" under each layer page that the search engines have indexed and still sends traffic to, but it was also meant to be the place where people could learn how to contribute or use our GeoJSON links in their own applications.  In practice, I'm not aware of anybody using our GeoJSON links in the real world besides the demo map from Olivia / HG Labs, or things I've done (ex. coworking, breweries, TreesUpstate planting map on their site was previously from our syndication of their own plantings spreadsheet).

We talked about doing away with our Google spreadsheet interfaces for the maps and just putting things in a free Github repo and a Github Pages site to host static GeoJSON for the layers.  We could always activate a spreadsheet for a custom datasource if / as needed, using our mature Google Sheets CSV to GeoJSON logic. In practice, I may have been the only person regularly using the custom sheets / datasets to make edits and I could just as easily update GeoJSON files in a GitHub repo or in some other interface you guys have.  

Olivia, CC'ed, was advocating for not killing off the layers that are very low traffic, incomplete, or out of date (ex. quilting guilds, dog waste bags), but we can figure that out as we talk through things.  I included traffic stats in the previously referenced Github comment. 
The data site lists the map layers, the URL to contribute changes via a spreadsheet, the GeoJSON links, and the raw real-time CSV of the Google Sheet.   1/3 of these layers are syndicated from the Greenville County GIS hosted iMAP project (United Way, Furman, Prisma).  We haven't heard from that group in years, so we're not sure if that data is actively maintained.  The remaining 2/3 of the layers are original content mostly from the pilot SC Codes class projects years ago when Pam was running SC Codes.

(side note: since we both had to work around the GCGIS goofiness, here's how I found to use a * in the query string to get all results and fields  https://www.gcgis.org/arcgis/rest/services/imap/imap/MapServer/8/query?where=OBJECTID%3E0&f=geojson&outFields=*&geometryType=esriGeometryPoint

Olivia built a demo map in Leaflet  (repo) to allow for toggling on any/all layers that we provide as GeoJSON as a preview of what's possible.  These layers are all point data besides the Swamp Rabbit Trail and spurs path data that Mike Nice maintains in OpenStreetMaps and which he shared with us and we have attribution in the path file. 

Some of the SRT point data was in the OpenStreetMaps tiles, but some was not in there as point data.  Mike Nice may have since added that to the OSM geo database, so it's possible most or all of the SRT stuff can be syndicated from OSM using their Overpass query tool, assuming there are no issues with intellectual property.  We also talked about possibly redirecting some of our SRT traffic to Mike's SRT map.

In any case, I wanted to give you a history and dump of what we have on our end and thoughts on migrating, merging, or sunsetting some of the bits and maintaining others.

We'd be glad to explore what you and the Geek Squad have built and figure out how to proceed.

Best,
Jim