7.6 KiB
Overview
This project contains useful info/tools for processing OSM maps into the mapwriter file format (.map). It was developed to help generating offline map files for Locus on Android.
Dependencies / Prereqs
To run this program you'll need the following
- A Linux environment
- Python 3.x installed
- Java 1.8 or higher installed (OpenJDK works)
- wget installed
- gdal-bin
- python3-gdal (Ubuntu) installed
Installation
Head over to releases tab and download the latest. Extract the file and you're good to go. Everything is self-contained minus the above dependencies.
Running
To run the program, cd to the directory where you extracted the release and run './process_maps.py' with at least one '--map-list /path/to/list.txt parameter'. See the lists directory for examples on how to format the map list files.
Program Usage
usage: process_maps.py [-h] --map-list MAP_LIST [--use-ram]
[--max-heap-space MAX_HEAP_SPACE]
[--output-map-name OUTPUT_MAP_NAME] --maps-dir MAPS_DIR
[--no-map-download] [--tag-mapping TAG_MAPPING]
[--tag-transform TAG_TRANSFORM]
optional arguments:
-h, --help show this help message and exit
--map-list MAP_LIST a text file with one map URL per line, can be
specified more than once
--use-ram use RAM for mapsforge processing -- WARNING mapsforge
uses 10x the map size in RAM for processing (ie. 100Mb
map = 1Gb RAM usage), you want a LOT of RAM for this
option
--max-heap-space MAX_HEAP_SPACE
set the max heap space for the JVM, use standard -Xmx
values, default (1g) should be fine if not using
--use-ram argument
--output-map-name OUTPUT_MAP_NAME
set the output .map and .poi file names
--maps-dir MAPS_DIR Where downloaded maps will be stored/read from
--no-map-download Do NOT download maps, re-use maps from maps-dir
--tag-mapping TAG_MAPPING
Specify a custom tag mapping xml file for use with
mapsforge processing
--tag-transform TAG_TRANSFORM
Specify a tag transform file for use PRIOR to
mapsforge processing (this is DIFFERENT than the
mapsforge mapping xml file)
Examples
./process_maps.py --tag-transform ./openandromaps/tag-transform.xml --tag-mapping ./openandromaps/tag-mapping.min.xml --use-ram --max-heap-space 12g --maps-dir cache --no-map-download --output-map-name custom --map-list lists/custom.txt
./process_maps.py --tag-transform ./openandromaps/tag-transform.xml --tag-mapping ./openandromaps/tag-mapping.min.xml --use-ram --max-heap-space 16g --maps-dir cache --no-map-download --output-map-name pennsylvania --map-list lists/pennsylvania.txt
./process_maps.py --tag-transform ./openandromaps/tag-transform.xml --tag-mapping ./openandromaps/tag-mapping.min.xml --use-ram --max-heap-space 12g --maps-dir cache --no-map-download --output-map-name idaho --map-list lists/idaho.txt
./process_maps.py --tag-transform ./openandromaps/tag-transform.xml --tag-mapping ./openandromaps/tag-mapping.min.xml --use-ram --max-heap-space 24g --maps-dir cache --no-map-download --output-map-name australia --map-list lists/australia.txt
./process_maps.py --use-ram --max-heap-space 24g --maps-dir cache --no-map-download --output-map-name canada_ontario --map-list lists/ontario.txt
./process_maps.py --tag-transform ./openandromaps/tag-transform.xml --tag-mapping ./openandromaps/tag-mapping.min.xml --max-heap-space 8g --maps-dir cache --no-map-download --output-map-name midwest-2 --map-list lists/midwest.txt --map-list lists/ontario.txt
./process_maps.py --max-heap-space 8g --maps-dir cache --no-map-download --output-map-name usa_northeast --map-list lists/usa_northeast.txt
./process_maps.py --max-heap-space 8g --maps-dir cache --no-map-download --output-map-name usa_pacific --map-list lists/usa_pacific.txt
./process_maps.py --max-heap-space 8g --maps-dir cache --no-map-download --output-map-name usa_south --map-list lists/usa_south.txt
./process_maps.py --use-ram --max-heap-space 16g --maps-dir cache --no-map-download --output-map-name mexico --map-list lists/mexico.txt
./process_maps.py --use-ram --max-heap-space 24g --maps-dir cache --no-map-download --output-map-name mexico_central_america --map-list lists/mexico.txt --map-list lists/central_america.txt
./process_maps.py --use-ram --max-heap-space 12g --maps-dir cache --no-map-download --output-map-name central_america --map-list lists/central_america.txt
./process_maps.py --max-heap-space 24g --maps-dir cache --no-map-download --output-map-name south_america --map-list lists/south_america.txt
./process_maps.py --max-heap-space 8g --maps-dir cache --no-map-download --output-map-name netherlands --map-list lists/netherlands.txt
./process_maps.py --use-ram --max-heap-space 24g --maps-dir cache --no-map-download --output-map-name spain --map-list lists/spain.txt
./process_maps.py --max-heap-space 8g --maps-dir cache --no-map-download --output-map-name france --map-list lists/france.txt
./process_maps.py --use-ram --max-heap-space 16g --maps-dir cache --no-map-download --output-map-name belgium --map-list lists/belgium.txt
./process_maps.py --use-ram --max-heap-space 12g --maps-dir cache --no-map-download --output-map-name luxembourg --map-list lists/luxembourg.txt
./process_maps.py --use-ram --max-heap-space 12g --maps-dir cache --no-map-download --output-map-name portugal --map-list lists/portugal.txt
Caching maps for re-use / re-processing
Included in the repo there is a download_maps.py
script that is meant to help with downloading maps on a schedule or for caching. It uses the same logic as the main script but for download only.
Program usage
usage: download_maps.py [-h] --map-list MAP_LIST --map-dir MAP_DIR
optional arguments:
-h, --help show this help message and exit
--map-list MAP_LIST a text file with one map URL per line, can be specified
more than once
--map-dir MAP_DIR The directory where maps should be downloaded
Example Usage
./download_maps.py --map-dir ./cache \
--map-list lists/central_america.txt
./download_maps.py --map-dir ./cache \
--map-list lists/midwest.txt
./download_maps.py --map-dir ./cache \
--map-list lists/south_america.txt
Performance
This deserves an expansion but for now: notes on how to improve performance on 'small' systems that may not have >64Gb of RAM to work with for larger maps.
- tmpfs works well for source/tmp/output in disk mode for speedups
- zram /should/ work well if disk mode eats a little too much RAM when backed by tmpfs
- type=ram uses closer to 15-20x source map size in practice
- stacking maps from multiple sources increases memory footprint beyond the rough rules of thumb that are called out by the mapsforge devs
- threads=n>1 is a fast way to ruin a heap space (don't do this with type=ram as a general rule of thumb)
- there are choke points that are not multi-threaded once the main osmosis 'bits' are done running, deal
Look Into
If you're concerned about Java heap usage, the ZGC announced at the 2018 fosdem conference may be useful. I have no had success building a jre/jdk from sources but you may find the following link useful.
Licencing
All code is licensed Apache 2.0 and all non-code is licensed Creative Commons CC-BY-SA-3.0