MapServer MapCache: Fast and Featureful Tile Serving

@tbonfort

What's a Tiling Server

  • Precomputed 256x256 images for fast access to static data
  • Grid (projection, extent, resolutions)
  • Source (WMS, MapServer, GDAL)
  • Cache
  • Tileset
  • Service

What's MapCache?

  • Tiling Library with Apache, FastCGI, Nginx and Node frontends
  • Versatile
    • Multiple Cache Backends
    • Multiple Protocols
    • Tile Management Features: seeding, recompression, interpolation
  • Native code: Fast and Efficient
  • Ease deployment: Demo interface

History

  • Started as an independant project in 2010
  • Integrated under MapServer governance in 2012

Protocols

  • XYZ etc...
  • KML
  • WMS

Standard Tile Addressing

  • TMS
  • WMTS (RestFull + KVP)
  • VirtualEarth QuadKeys
  • MapGuide
  • Google x,y,z

WMS

  • WMS-Tiled
  • Responds to untiled requests by assembling tiles
  • Can mirror a full WMS service: layer-by-layer, layer lists
  • Forwards GetFeatureInfo requests

(OGC) Proxy

  • Intercepts requests that can be served from cached tiles
  • Forwards other requests to other upstream server(s)
  • Use Case

Miscellaneous

  • Caching HTTP Headers
  • "Auto" Expiry of old tiles
  • Error Reporting
    • Error Message
    • Empty Image
    • Status Code
  • Meta-Tiling
  • Watermarking
  • Max-Cached-Zoom: Upsample lower level tiles

Image Operations

  • Image data can be recompressed/optimized before being stored into a tile
    • Avoid double JPEG compression, i.e. request PNG from source WMS
    • Be agressive on compression when storing a tile (e.g. quantized PNG with max compression)
    • CPU time for dynamic recompressions can be lowered, at the cost of more bandwidth

Image Formats

  • PNG
    • Compression level
    • Quantization (color reduction to 8bit palette)
  • JPEG
    • Compression level
    • RGB/YCbCr
  • Mixed (JPEG + PNG)
    • Use for raster coverages on limited areas
    • PNG for tiles with transparency
    • JPEG otherwise

Handling Empty/Uniform Tiles

  • Strategy dependent on the cache backend
    • Symbolic linking
    • Reconstruct Paletted 1bit PNG from uniform color
    • For fully seeded caches, treat absent tile as fully transparent

Grids

  • Multiple Grids per Tileset (e.g. WGS84 + Mercator)
  • Handles Grid Aliasing
    • EPSG:3857/EPSG:900913
    • Non Standard Codes
  • Grid Subsets
    • min/max resolution
    • Restricted extent (with tolerance!)

Caches

  • Backend able to store tile data for given x,y,z
  • Extensible: Exists(), Get(), Put(), Delete()
  • Some backend specific hacks: symlinking, blank detection

Disk Cache

  • Mainly for dev, testing, or small tilesets
  • Pros
    • Simple, just provide a base directory
    • Fast
    • Supports blank tile detection
    • (Re-)Use custom layouts: TileCache, ArcGis, Template
  • Cons
    • Difficult to manage large number of files
    • Hits File System Limits
    • Wasted space due to File Sytem block size

Sqlite Cache

  • Stores tile data as blobs in an sqlite file
  • Pros
    • Single File
    • Extensible to support any schema (MBtiles implemented)
    • Efficient Space (blank tile detection possible)
  • Cons
    • Some tweaking needed for very large caches (more than 1 Tb)
    • Concurrent Insertions

MemCache

  • Ideal for Temporary Data: Forecasts, Sensors, ...
  • Pros
    • Easily distributed
    • Automatic pruning
  • Cons
    • Limited Storage

(Geo)TIFF Cache

  • Specialized storage for satellite imagery basemaps
  • JPEG tiles are read directly from a constrained (Geo)TIFF file hierarchy
  • Configurable number of tiles per TIFF file
  • Templated access to file name
  • Pros
    • Efficient on disk usage (TIFF container)
    • TIFF files can be shared with other applications
  • Cons
    • Limited to JPEG data (no transparency)
    • No Concurrent writing to a single TIFF file
    • Updates/deletion of individual tiles does not reclaim storage space
  • Use-Case

Seeder: Pre-generating tiles

  • Multi thread/process seeding
  • "Drill-Down" mode to take advantage of caching on source WMS
  • Seed particular subset
    • Given Dimension
    • Regenerate old tiles
    • Restricted min/max zoom and/or extent
    • Restrict to geometry (e.g. seed to level 18 all areas where population density is over a given value)
  • Pruning mode

Dimensions

  • Multiple Dimensions per Tileset
  • E.g. Forecast Time, Elevation
  • Multiple Types
    • List of Values
    • Intervals
    • Regular Expression
  • Use-Case: BoM MetEye

Time

  • Each timestamp corresponds to a limited extent
  • Dynamic Mosaic given a requested time interval
  • c.f. ESA ngEO Presentation at 15:00

What's coming next

  • TMS/GDAL source
  • Native
  • S3, Riak Caches
  • Vector Tiles (UTFGrid)

Thank you for your Attention