You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
 
 
 
 
 
 
Mike C b131e95e5c Updated polygons, basic elevation processing notes, info 2 years ago
bin Updated polygons, basic elevation processing notes, info 2 years ago
custom_polys Updated polygons, basic elevation processing notes, info 2 years ago
lists Updated polygons, basic elevation processing notes, info 2 years ago
openandromaps Add way to handle custom polys, add notes on how to re-create openandromaps styles, etc, start basics of elevation handling 2 years ago
pymapsforge Updated polygons, basic elevation processing notes, info 2 years ago
.gitignore Rework lists, update to match main mapsforge docs (land/sea polys in particular), overhaul README for better examples 2 years ago
LICENSE-APACHE-2.0.txt Added licencing 4 years ago
LICENSE-CC-BY-SA-3.0.html Added licencing 4 years ago
README.md Moar example tweaks 2 years ago
azure_notes.md Rework lists, update to match main mapsforge docs (land/sea polys in particular), overhaul README for better examples 2 years ago
custom_polys_and_extracts.md Updated polygons, basic elevation processing notes, info 2 years ago
download_maps.py Rework lists, update to match main mapsforge docs (land/sea polys in particular), overhaul README for better examples 2 years ago
elevation.md Updated polygons, basic elevation processing notes, info 2 years ago
poi-mapping.xml Updated to latest mapsforge poi mappings 2 years ago
process_maps.py Updated polygons, basic elevation processing notes, info 2 years ago
sea.osm Rework lists, update to match main mapsforge docs (land/sea polys in particular), overhaul README for better examples 2 years ago
shape2osm.py Rework lists, update to match main mapsforge docs (land/sea polys in particular), overhaul README for better examples 2 years ago

README.md

Overview

This project contains useful info/tools for processing OSM maps into the mapwriter file format (.map). It was developed to help generating offline map files for Locus on Android.

Dependencies / Prereqs

To run this program you'll need the following

  • A Linux environment
  • Python 3.x installed
  • Java 1.8 or higher installed (OpenJDK works)
  • wget installed
  • gdal-bin
  • python3-gdal (Ubuntu) installed

Installation

Head over to releases tab and download the latest. Extract the file and you're good to go. Everything is self-contained minus the above dependencies.

Running

To run the program, cd to the directory where you extracted the release and run './process_maps.py' with at least one '--map-list /path/to/list.txt parameter'. See the lists directory for examples on how to format the map list files.

Program Usage

usage: process_maps.py [-h] --map-list MAP_LIST [--use-ram]
                       [--max-heap-space MAX_HEAP_SPACE]
                       [--output-map-name OUTPUT_MAP_NAME] --maps-dir MAPS_DIR
                       [--no-map-download] [--tag-mapping TAG_MAPPING]
                       [--tag-transform TAG_TRANSFORM]

optional arguments:
  -h, --help            show this help message and exit
  --map-list MAP_LIST   a text file with one map URL per line, can be
                        specified more than once
  --use-ram             use RAM for mapsforge processing -- WARNING mapsforge
                        uses 10x the map size in RAM for processing (ie. 100Mb
                        map = 1Gb RAM usage), you want a LOT of RAM for this
                        option
  --max-heap-space MAX_HEAP_SPACE
                        set the max heap space for the JVM, use standard -Xmx
                        values, default (1g) should be fine if not using
                        --use-ram argument
  --output-map-name OUTPUT_MAP_NAME
                        set the output .map and .poi file names
  --maps-dir MAPS_DIR   Where downloaded maps will be stored/read from
  --no-map-download     Do NOT download maps, re-use maps from maps-dir
  --tag-mapping TAG_MAPPING
                        Specify a custom tag mapping xml file for use with
                        mapsforge processing
  --tag-transform TAG_TRANSFORM
                        Specify a tag transform file for use PRIOR to
                        mapsforge processing (this is DIFFERENT than the
                        mapsforge mapping xml file)

Examples

./process_maps.py --tag-transform ./openandromaps/tag-transform.xml --tag-mapping ./openandromaps/tag-mapping.min.xml --use-ram --max-heap-space 12g --maps-dir cache --no-map-download --output-map-name nicaragua-costa-rica-panama --map-list lists/nicaragua-costa-rica-panama
./process_maps.py --tag-transform ./openandromaps/tag-transform.xml --tag-mapping ./openandromaps/tag-mapping.min.xml --use-ram --max-heap-space 12g --maps-dir cache --no-map-download --output-map-name au-east --map-list lists/au-east.txt
./process_maps.py --tag-transform ./openandromaps/tag-transform.xml --tag-mapping ./openandromaps/tag-mapping.min.xml --use-ram --max-heap-space 16g --maps-dir cache --no-map-download --output-map-name pennsylvania --map-list lists/pennsylvania.txt
./process_maps.py --tag-transform ./openandromaps/tag-transform.xml --tag-mapping ./openandromaps/tag-mapping.min.xml --use-ram --max-heap-space 12g --maps-dir cache --no-map-download --output-map-name idaho --map-list lists/idaho.txt
./process_maps.py --tag-transform ./openandromaps/tag-transform.xml --tag-mapping ./openandromaps/tag-mapping.min.xml --use-ram --max-heap-space 24g --maps-dir cache --no-map-download --output-map-name australia --map-list lists/australia.txt
./process_maps.py --use-ram --max-heap-space 24g --maps-dir cache --no-map-download --output-map-name canada_ontario --map-list lists/ontario.txt
./process_maps.py --tag-transform ./openandromaps/tag-transform.xml --tag-mapping ./openandromaps/tag-mapping.min.xml --max-heap-space 8g --maps-dir cache --no-map-download --output-map-name midwest --map-list lists/midwest.txt --map-list lists/ontario.txt
./process_maps.py --max-heap-space 8g --maps-dir cache --no-map-download --output-map-name usa_northeast --map-list lists/usa_northeast.txt
./process_maps.py --max-heap-space 8g --maps-dir cache --no-map-download --output-map-name usa_pacific --map-list lists/usa_pacific.txt
./process_maps.py --max-heap-space 8g --maps-dir cache --no-map-download --output-map-name usa_south --map-list lists/usa_south.txt
./process_maps.py --use-ram --max-heap-space 16g --maps-dir cache --no-map-download --output-map-name mexico --map-list lists/mexico.txt
./process_maps.py --use-ram --max-heap-space 24g --maps-dir cache --no-map-download --output-map-name mexico_central_america --map-list lists/mexico.txt --map-list lists/central_america.txt
./process_maps.py --use-ram --max-heap-space 12g --maps-dir cache --no-map-download --output-map-name central_america --map-list lists/central_america.txt
./process_maps.py --max-heap-space 24g --maps-dir cache --no-map-download --output-map-name south_america --map-list lists/south_america.txt
./process_maps.py --max-heap-space 8g --maps-dir cache --no-map-download --output-map-name netherlands --map-list lists/netherlands.txt
./process_maps.py --use-ram --max-heap-space 24g --maps-dir cache --no-map-download --output-map-name spain --map-list lists/spain.txt
./process_maps.py --max-heap-space 8g --maps-dir cache --no-map-download --output-map-name france --map-list lists/france.txt
./process_maps.py --use-ram --max-heap-space 16g --maps-dir cache --no-map-download --output-map-name belgium --map-list lists/belgium.txt
./process_maps.py --use-ram --max-heap-space 12g --maps-dir cache --no-map-download --output-map-name luxembourg --map-list lists/luxembourg.txt
./process_maps.py --use-ram --max-heap-space 12g --maps-dir cache --no-map-download --output-map-name portugal --map-list lists/portugal.txt

Caching maps for re-use / re-processing

Included in the repo there is a download_maps.py script that is meant to help with downloading maps on a schedule or for caching. It uses the same logic as the main script but for download only.

Program usage

usage: download_maps.py [-h] --map-list MAP_LIST --map-dir MAP_DIR

optional arguments:
  -h, --help           show this help message and exit
  --map-list MAP_LIST  a text file with one map URL per line, can be specified
                       more than once
  --map-dir MAP_DIR    The directory where maps should be downloaded

Example Usage

./download_maps.py --map-dir ./cache \
    --map-list lists/central_america.txt

./download_maps.py --map-dir ./cache \
    --map-list lists/midwest.txt

./download_maps.py --map-dir ./cache \
    --map-list lists/south_america.txt

Performance

This deserves an expansion but for now: notes on how to improve performance on 'small' systems that may not have >64Gb of RAM to work with for larger maps.

  • tmpfs works well for source/tmp/output in disk mode for speedups
  • zram /should/ work well if disk mode eats a little too much RAM when backed by tmpfs
  • type=ram uses closer to 15-20x source map size in practice
  • stacking maps from multiple sources increases memory footprint beyond the rough rules of thumb that are called out by the mapsforge devs
  • threads=n>1 is a fast way to ruin a heap space (don't do this with type=ram as a general rule of thumb)
  • there are choke points that are not multi-threaded once the main osmosis 'bits' are done running, deal

Look Into

If you're concerned about Java heap usage, the ZGC announced at the 2018 fosdem conference may be useful. I have no had success building a jre/jdk from sources but you may find the following link useful.

Licencing

All code is licensed Apache 2.0 and all non-code is licensed Creative Commons CC-BY-SA-3.0