OHDM MapnikTileServer

The OpenHistoricalDataMap MapnikTileServer is an OpenStreetMap time sensitive fullstack tile server. This means you can go back in time on a OpenStreetMap Map and see how your city changed since you was a child or you can go much more back in time, it's your choice :)

The project is build with Django Cookiecutter and it comes with docker support, it is design to work out of the box with Docker.

The current version of this project based on a fork of openstreetmap-carto.

Features

• documentation: https://mapniktileserver.readthedocs.io/en/latest/
• work out of the box, no special configuration is need
• work Linux, MacOS, BSD & also should work Windows (Windows not tested), just need Docker & Docker-Compose to work
• OSM based tile server with time sensitive tiles
• a development & production configuration
• task-queue for rendering tile with celery
• caching tiles by date range in redis
• SSL with Let's Encrypt included
• generate development database from osm / ohdm
• include sentry.io in production for error tracking

Dependencies

Tile Server

• a custom fork of openstreetmap-carto for tile styles and SQL
• https://mapnik.org/
• python-mapnik
• tile rendering code by wiki.openstreetmap.org
• Project boilerplate by cookiecutter-django

Front-End

• Bootstrap 4 Theme
• OpenLayers for map view

Frontend repo: https://github.com/linuxluigi/ohdm-angular-frontend

minimum server requirements for developing

• 8 GB of RAM
• 100 GB of free disk space (better using a SSD drive instead of HDD)

If you can, use beefy hardware!

Project Structure

OHDM MapnikTileServer
│   .coveragerc                                  # https://coverage.readthedocs.io/en/v4.5.x/config.html
│   .dockerignore                                # https://docs.docker.com/engine/reference/builder/#dockerignore-file
│   .editorconfig                                # https://editorconfig.org/
│   .gitignore                                   # https://git-scm.com/docs/gitignore
│   .pylintrc                                    # https://www.pylint.org/
│   .travis.yml                                  # travis bionic image https://docs.travis-ci.com/user/reference/bionic/
│   LICENSE
│   local.yml                                    # docker-compose file for developing https://cookiecutter-django.readthedocs.io/en/latest/developing-locally-docker.html
│   manage.py                                    # Django start file
│   merge_production_dotenvs_in_dotenv.py        # https://cookiecutter-django.readthedocs.io/en/latest/developing-locally-docker.html?highlight=merge_production_dotenvs_in_dotenv#configuring-the-environment
│   production.yml                               # docker-compose file for production https://cookiecutter-django.readthedocs.io/en/latest/deployment-with-docker.html
│   pytest.ini                                   # https://pytest.org/en/latest/
│   README.md
│   requirements.txt                             # python production requirements (mostly needed for heroku)
│   setup.cfg                                    # https://docs.pytest.org/en/latest/customize.html
│   LICENSE
│
└───.envs
│   └───.local
│   │   │   .django                              # default enviroment vars for django
│   │   │   .postgres                            # default enviroment vars for postgres
│   └───.production                              # need to create, not there by default
│   │   │   .django                              # custom enviroment vars for django
│   │   │   .postgres                            # custom enviroment vars for django
│
└───compose                                      # docker build files
│
└───config                                       # django settings
│
└───locale                                       # https://docs.djangoproject.com/en/2.2/topics/i18n/translation/
│
└───ohdm_django_mapnik                           # main project folder
│   └───contrib                                  # http://cookiecutter-django.readthedocs.io/en/latest/faq.html#why-is-there-a-django-contrib-sites-directory-in-cookiecutter-django
│   └───ohdm                                     # ohdm app for django
│   │   admin.py                                 # model admin interface designer https://docs.djangoproject.com/en/2.2/ref/contrib/admin/
│   │   apps.py                                  # app config https://docs.djangoproject.com/en/2.2/ref/applications/#configuring-applications
│   │   converters.py                            # regex converter
│   │   models.py                                # django models
│   │   tasks.py                                 # celery task https://docs.celeryproject.org/en/latest/userguide/tasks.html
│   │   tests.py                                 # test file
│   │   tile.py                                  # tile producer logic
│   │   urls.py                                  # app url logic
│   │   views.py                                 # app view logic
│   │   └───management                           # app commands folder
│   │   │   │   date_template_importer.py        # convert openstreetmap-carto/project.mml to a ohdm version (not working right now)
│   │   └───migrations                           # Django model migrations (no not edit manually) https://docs.djangoproject.com/en/2.2/topics/migrations/
│
└───requirements                                 # python requirements https://pip.pypa.io/en/stable/user_guide/#requirements-files
│   │   base.txt                                 # requirements for dev & productions
│   │   local.txt                                # requirements for dev
│   │   production.txt                           # requirements for productions

URL Structure

URL's are setup in config/urls.py and ohdm_django_mapnik/ohdm/urls.py.

# admin panel
/admin

# tile url
/tile/<int:year>/<int:month>/<int:day>/<int:zoom>/<float:x_pixel>/<float:y_pixel>/tile.png

# only in development mode enabled

# tile url with
/tile/<int:year>/<int:month>/<int:day>/<int:zoom>/<float:x_pixel>/<float:y_pixel>/reload-style-xml/tile.pngreload style.xml
# tile url with reload project.mml & style.xml
/tile/<int:year>/<int:month>/<int:day>/<int:zoom>/<float:x_pixel>/<float:y_pixel>/reload-project-mml/tile.png
# tile url with default openstreetmap-carto (no time sensitivity)
/tile/<int:zoom>/<float:x_pixel>/<float:y_pixel>/tile.png

Tile example link (in Berlin): http://localhost:8000/tile/2010/02/16/13/4398/2685/tile.png

Setup

Install Docker & Docker-Compose

For installing Docker just follow the instructions on Docker Docs

To install Docker-Compose use pip

$ sudo pip install docker-compose

Setup Development Server

1. download sourcecode

$ mkdir ohdm
$ git clone https://github.com/OpenHistoricalDataMap/MapnikTileServer.git
$ git clone https://github.com/linuxluigi/openstreetmap-carto.git

2. build images

Building for the first time could take some time, after run the command go and get a coffee.

$ cd MapnikTileServer
$ docker-compose -f local.yml build

3. init database

$ docker-compose -f local.yml run --rm django python manage.py migrate

4. create test database (optional)

$ docker-compose -f local.yml up test-database
$ docker-compose -f local.yml run --rm django python manage.py ohdm2mapnik

5. download shapefiles & generate style.xml

$ docker-compose -f local.yml run --rm django /get-shapefiles.sh
$ docker-compose -f local.yml run --rm django python manage.py create_style_xml

6. start test server

$ docker-compose -f local.yml up django celeryworker celerybeat

Or start in background.

$ docker-compose -f local.yml up -d django celeryworker celerybeat

If it works fine you should able to visit the Website via http://example.com:8000/

5. use Flower to monitor tile producing task

With Flower it's possible to monitor Celery task.

To start the Flower server run:

$ docker-compose -f local.yml up -d flower

Then you can via http://example.com:5555/ watch the Flower monitor. The login data stored in .envs/.local/.django

6. create admin user

$ docker-compose -f local.yml run --rm django python manage.py createsuperuser

To visit the admin panel go to http://example.com:8000/admin/

For more infos got to https://cookiecutter-django.readthedocs.io/en/latest/developing-locally-docker.html

7. stop server

$ docker-compose -f local.yml stop

Setup Production Server

Please read also the doc on cookiecutter-django for more details.

1. download sourcecode

$ mkdir ohdm
$ git clone https://github.com/OpenHistoricalDataMap/MapnikTileServer.git

2. build images

Building for the first time could take some time, after run the command go and get coffee.

$ cd MapnikTileServer
$ docker-compose -f production.yml build

3. setup environment files

Create 2 environment files

The first is .envs/.production/.django, below is an example how to fill.

# General
# ------------------------------------------------------------------------------
# DJANGO_READ_DOT_ENV_FILE=True
DJANGO_SETTINGS_MODULE=config.settings.production
DJANGO_SECRET_KEY=kt5UXs5t8QEMR6zpa13DNgb0cHJ3u1m3F4yfOO8ggtDq4xp06MjxI1MbUNYPl4Px
DJANGO_ADMIN_URL=geiHZUuftfcy8ZQCsr8qRNG1tXbs9hL2/
DJANGO_ALLOWED_HOSTS=ohdm.net

# Security
# ------------------------------------------------------------------------------
# TIP: better off using DNS, however, redirect is OK too
DJANGO_SECURE_SSL_REDIRECT=False

# Email
# ------------------------------------------------------------------------------
MAILGUN_API_KEY=
DJANGO_SERVER_EMAIL=
MAILGUN_DOMAIN=

# django-allauth
# ------------------------------------------------------------------------------
DJANGO_ACCOUNT_ALLOW_REGISTRATION=True

# django-compressor
# ------------------------------------------------------------------------------
COMPRESS_ENABLED=

# Gunicorn
# ------------------------------------------------------------------------------
WEB_CONCURRENCY=4

# Sentry
# ------------------------------------------------------------------------------
SENTRY_DSN=


# Redis
# ------------------------------------------------------------------------------
REDIS_URL=redis://redis:6379/0

# Celery
# ------------------------------------------------------------------------------

# Flower
CELERY_FLOWER_USER=msshnhBNlGfVLiDTErFKxFWpBOrcZNVp
CELERY_FLOWER_PASSWORD=YG9IuF8qdJaf2Wm2OI1AWdKLv4ddOixmgyKC7y3Kf1PLEfNK3DaQwlpJRe8eh6pL

# ohdm
# ------------------------------------------------------------------------------
CARTO_STYLE_PATH=/opt/openstreetmap-carto
CARTO_STYLE_PATH_DEBUG=/opt/openstreetmap-carto-debug
OSM_CARTO_VERSION=v4.22.0
TILE_GENERATOR_SOFT_TIMEOUT=240
TILE_GENERATOR_HARD_TIMEOUT=360

The second is .envs/.production/.postgres, also below is an example how to fill it.

# PostgreSQL
# ------------------------------------------------------------------------------
POSTGRES_HOST=postgres
POSTGRES_PORT=5432
POSTGRES_DB=gis
POSTGRES_USER=docker
POSTGRES_PASSWORD=ljloyAbi9HjIuN0AbLC4qaYAPKeFSrMvfUKxbfil1f2OCeZvZBxFZVGVQAkTAXey
POSTGRES_MULTIPLE_EXTENSIONS=postgis,hstore,postgis_topology

Please make sure to change at least:

• DJANGO_SECRET_KEY
• DJANGO_ADMIN_URL
• CELERY_FLOWER_USER
• CELERY_FLOWER_PASSWORD
• POSTGRES_PASSWORD

Be careful to change POSTGRES_USER & POSTGRES_USER it can may broke the tile server.

4. start production server

$ docker-compose -f production.yml up -d django celeryworker celerybeat traefik

The server start on https://ohdm.net/

Also it's possible to start the celery monitor Flower with:

$ docker-compose -f production.yml up -d flower

Flower start on https://ohdm.net:5555/

5. create Database

$ docker-compose -f production.yml run --rm django python manage.py migrate

6. create admin user

$ docker-compose -f production.yml run --rm django python manage.py createsuperuser

To visit the admin panel go to http://ohdm.net/geiHZUuftfcy8ZQCsr8qRNG1tXbs9hL2/

The admin panel url depends on the environment var DJANGO_ADMIN_URL in .envs/.production/.django

7. Backup

Go to cookiecutter-django.readthedocs.io to read how to back up the database.

8. stop server

$ docker-compose -f production.yml stop

Scale-Up for production

A tile server need a lot of resources and a big database. So for a better performance use a beefy server or use docker swarm / kubernetes for scaling.

To scale up use the comments below for the django and celeryworker containers.

$ docker-compose -f production.yml scale django=4
$ docker-compose -f production.yml scale celeryworker=2

Don’t try to scale postgres, celerybeat, or traefik!

Testing

To read how to run test go to cookiecutter-django

Or for running test manual use docker-compose -f local.yml run --rm django pytest

Also the project include a working travis test.

Docs

To create the docs use:

$ docker-compose -f local.yml run --rm django make --directory docs html