Architecture

Core algorithm

Plan (TLDR):

• Start with Python in Phase 1, get the core algorithm right, just write a script that uses rasterio/fiona with a config file, no fancy CLI or rio plugin yet.
• Pursue path with Javascript for core later. Test feasibility and compare performance.
• Python should only be used for the core algorithm - read features -> create heatmap raster. Javascript can be used for any services built around the core.

Some findings:

• Maldives with a cell size of 100m (the default) and 32-bit floating point values, produces a raster of ~450MB. Much smaller than the 10m and 5m maldives datasets we’re working with for coral, habitat, etc.
• A 100m raster covering the EEZ of Norway would be about 9.6GB, substantially larger but perhaps workable (Lambda has 10GB memory limit)
• geotiff.js is capable of writing geotiff arrays (beta), though uncompressed. It looks like it might be able to write geographic metadata as well and possibly a nodata value or establish nodata value. If geotiff can’t do it all, then node-gdal-next or gdal-js may need to be used.
• gdal-js is a web assembly port of GDAL. Can possibly be run in Node environment but designed for web workers in browser. Has rasterize function built-in.

Heatmap Core - Javascript only path:

• read features with shapefile-js
• create georaster in memory and fill
• use modified bresenham from geoblaze for burn-in, cell values will be summed
  • alternative - use gdal-js port with rasterize, should return an array instead of writing a file?
• geotiff.js write to array buffer, with geo metadata, save as .tif or maybe stream to s3?
  • alternative - use gdal-js to write geotiff

Heatmap Core - Python only path:

• Use fiona to read shapefile
• rasterio to create raster and rasterize and output raster

Heatmap service (Python core with Javascript wrap)

• Python core heatmap function
• JS API handler, handles Task management, calls out to Python heatmap command, pushes raster up to S3.

If Python is the only feasible path forward for the core algorithm, then run with it

CLI:

• Use rio rasterize CLI command for inspiration - https://github.com/mapbox/rasterio/blob/8cb216ca83e57284f8a56bafbe8eda4334a34db6/rasterio/rio/rasterize.py

Docker environment:

• Lightweight multi-stage Docker build supporting local use, debug, and production
• Production container can be deployed to Lambda, behind a web service.

https://docs.docker.com/develop/develop-images/multistage-build/ https://github.com/perrygeo/docker-gdal-base https://github.com/marcel-dempers/docker-development-youtube-series/tree/master/python https://hackernoon.com/how-to-deploy-aws-lambda-with-docker-containers-e51j3141