Mapchete

parallelized batch geoprocessing using Python

Created by Joachim Ungar / @ungarjo

EOX IT Services / eox.at


Creative Commons License This work is licensed under a Creative Commons Attribution 4.0 International License.

http://maps.eox.at/

GDAL command line tools


  • gdaldem hillshade srtm.vrt hillshade.tif
  • gdal_contour srtm.vrt contours.shp


  1. not viable (1,296,001 x 417,601 pixels)
  2. no customization (image filters, clipping, ...)

Tile Pyramids

source: wikipedia

tiled data → CartoCSS/mapfile → map tile


tiled data → process → process output tile

Mapchete

  • Python (scipy, numpy, shapely, ...)
  • chops data into tiles
  • applies user-defined process per tile
  • rasterio & Fiona for reading raster and vector data
  • spherical Mercator or geodetic (WGS84) tile pyramids

Command line utilities

  1. mapchete_serve
  2. mapchete_execute

mapchete_serve

  • starts web server (flask)
  • localhost:5000 → OpenLayers
  • processes tiles on demand
  • always return freshly generated tiles: --overwrite

Command line utilities

  1. mapchete_serve
  2. mapchete_execute

mapchete_execute

  • processes full pyramid
  • ... or just certain zoom levels:

    --zoom 0 8

  • ... or a specific tile (zoom, row, column):

    --tile 5 6 7

  • ... or a specific region:

    --bounds 10 40 12 50

Configuration

your_process.mapchete


%YAML 1.2
---
process_file: # path to process (python file)
input_files:
    file1: # path to file 1
    file2: # path to file 2
output:
    path: # absolute output path
    format: # PNG, PNG_hillshade, GTiff, GeoJSON, PostGIS or NumPy
    type: # geodetic or mercator
    dtype: # bool, uint8, uint16, int16, uint32, int32, float32, float64
    bands: # number of output bands
    nodata: # nodata value

Configuration

your_process.mapchete


process_minzoom: # minimum zoom level this process is valid
process_maxzoom: # maximum zoom level this process is valid
process_bounds: # left bottom right top in "type" CRS
metatiling: # (default is 1); has to be one of 2, 4, 6, 8 or 16

light_azimuth: 315 # user defined parameters
exaggeration: # user defined parameters dependending on zoom level
	zoom<5: 10
	zoom>=5: 5

Process

your_process.py


#!/usr/bin/env python
from mapchete import MapcheteProcess
class Process(MapcheteProcess):
    """Main process class"""
    def __init__(self, **kwargs):
        """Process initialization"""
        MapcheteProcess.__init__(
            self,
            identifier = "my_process_id",
            title="My long process title",
            version = "0.1",
            abstract="short description on what my process does",
        )

    def execute(self):
        """User defined process"""
        # insert magic here

Read data

your_process.py


with self.open(
    self.params["input_files"]["raster_file"], # object containing process config
    resampling="bilinear" # rasterio resampling
    ) as my_raster_rgb_file:

    if my_raster_rgb_file.is_empty(): # if empty, quit tile process
        return "empty"

    r, g, b = my_raster_rgb_file.read() # read RGB bands

Write data

your_process.py


self.write((r, g, b)) # tuple of bands
						

self.write(hillshade_array) # single band as array
						

self.write(features) # GeoJSON-like vector data

GeoTIFF, PNG, PNG_hillshade, GeoJSON, NumPy, PostGIS

Common Functions

  1. hillshade()
  2. contours()
  3. clip()

Merge DEMs

OpenTerrain

EOX hillshade

Multicolor contour lines

Sentinel-2 Cloudless

Sentinel-2 Cloudless

Availability

PyPi: https://pypi.python.org/pypi/mapchete

GitHub: https://github.com/ungarj/mapchete

twitter: @mapchete

Thank You!

Joachim Ungar @ungarjo

EOX IT Services eox.at


Slides: https://github.com/ungarj/foss4g_2016_mapchete