osm_map_processing/azure_notes.md

Azure Run Notes

Tested on

• Azure Virtual Machine
• E8S_V3 Standard Sized VM
  • 8 vCPUs
  • 64Gb RAM
  • 128Gb local SSD
• Azure blob storage for map data

Post install / first login

wget https://packages.microsoft.com/config/ubuntu/16.04/packages-microsoft-prod.deb
sudo dpkg -i packages-microsoft-prod.deb
sudo apt update
sudo apt install git tmux vim openjdk-8-jre python3 htop iotop nload
sudo apt upgrade
sudo systemctl reboot

Setup Azure blob storage

How to blob storage @ CLI

https://docs.microsoft.com/en-us/azure/storage/blobs/storage-how-to-mount-container-linux

Setup VM

wget https://packages.microsoft.com/config/ubuntu/16.04/packages-microsoft-prod.deb
sudo dpkg -i packages-microsoft-prod.deb
sudo apt-get update
sudo apt-get install blobfuse

Configure blob storage in VM

vim ~/blob_fuse_connection.cfg
accountName myaccount
accountKey myaccesskey==
containerName mycontainer

Setup various filesystem dependencies / zones

mkdir /home/ubuntu/osmmapdata
sudo chown: /home/ubuntu/osmmapdata
sudo mkdir /mnt/resource/blobfusetmp
sudo chown ubuntu: /mnt/resource/blobfusetmp

Mount blob storage for use

blobfuse /home/ubuntu/osmmapdata \
    --tmp-path=/mnt/resource/blobfusetmp \
    --config-file=/home/ubuntu/blob_connection.cfg \
    -o attr_timeout=240 \
    -o entry_timeout=240 \
    -o negative_timeout=120 \
    --file-cache-timeout-in-seconds=0 \
    -o allow_root

View on Windows Desktop

https://azure.microsoft.com/en-us/features/storage-explorer/

Run OSM Map Processing

Prep session

tmux new -s maps
git clone https://github.com/mcrosson/osm_map_processing.git
cd osm_map_processing

Download maps

Download maps using lists to local cache (use a blob storage endpoint, these can get very large)

# Example, reformat to your needs/desires
./download_maps.py --map-list lists/test.txt --map-dir ./cache

Midwest USA

./process_maps.py --use-ram --max-heap-space 50g --maps-dir cache --no-map-download --output-map-name midwest --map-list lists/midwest.txt --map-list lists/ontario.txt

South America

./process_maps.py --use-ram --max-heap-space 50g --maps-dir cache --no-map-download --output-map-name south_america --map-list lists/south_america.txt

Central America

NOTE: This CAN NOT fit into 64Gb RAM, you'll need to setup a 128Gb SSD/Disk at ~/osm_map_processing/tmp for this to not enounter heap space errors

./process_maps.py --max-heap-space 50g --maps-dir cache --no-map-download --output-map-name central_america --map-list lists/mexico.txt --map-list lists/central_america.txt