Add notes on how to run this with azure vms

2018-02-25 16:16:33 -05:00 · 2018-02-25 16:16:33 -05:00 · 0f6d71f1c0
parent 612f125219
commit 0f6d71f1c0
1 changed files with 109 additions and 0 deletions
--- a/azure_notes.md
+++ b/azure_notes.md
@ -0,0 +1,109 @@
+# Azure Run Notes
+
+### Tested on
+
+* Azure Virtual Machine
+* E8S_V3 Standard Sized VM
+  * 8 vCPUs
+  * 64Gb RAM
+  * 128Gb local SSD
+* Azure blob storage for map data
+
+# Post install / first login
+
+```
+wget https://packages.microsoft.com/config/ubuntu/16.04/packages-microsoft-prod.deb
+sudo dpkg -i packages-microsoft-prod.deb
+sudo apt update
+sudo apt install git tmux vim openjdk-8-jre python3 htop iotop nload
+sudo apt upgrade
+sudo systemctl reboot
+```
+
+# Setup Azure blob storage
+
+### How to blob storage @ CLI
+https://docs.microsoft.com/en-us/azure/storage/blobs/storage-how-to-mount-container-linux
+
+### Setup VM
+
+```
+wget https://packages.microsoft.com/config/ubuntu/16.04/packages-microsoft-prod.deb
+sudo dpkg -i packages-microsoft-prod.deb
+sudo apt-get update
+sudo apt-get install blobfuse
+```
+
+### Configure blob storage in VM
+
+```
+vim ~/blob_fuse_connection.cfg
+accountName myaccount
+accountKey myaccesskey==
+containerName mycontainer
+```
+
+### Setup various filesystem dependencies / zones
+
+```
+mkdir /home/ubuntu/osmmapdata
+sudo chown: /home/ubuntu/osmmapdata
+sudo mkdir /mnt/resource/blobfusetmp
+sudo chown ubuntu: /mnt/resource/blobfusetmp
+```
+
+### Mount blob storage for use
+
+```
+blobfuse /home/ubuntu/osmmapdata \
+    --tmp-path=/mnt/resource/blobfusetmp \
+    --config-file=/home/ubuntu/blob_connection.cfg \
+    -o attr_timeout=240 \
+    -o entry_timeout=240 \
+    -o negative_timeout=120 \
+    --file-cache-timeout-in-seconds=0 \
+    -o allow_root
+```
+
+### View on Windows Desktop
+https://azure.microsoft.com/en-us/features/storage-explorer/
+
+# Run OSM Map Processing
+
+### Prep session
+```
+tmux new -s maps
+git clone https://github.com/mcrosson/osm_map_processing.git
+cd osm_map_processing
+```
+
+### Download maps
+```
+# Download relevant maps using upcoming download util to ~/osmmapdata/cache/blah/date-time
+```
+
+### Midwest USA
+
+```
+./process_maps.py --use-ram --max-heap-space 48g \
+    --output-map-name midwest \
+    --cached-maps-dir ~/osmmapdata/cache/midwest/20180225-1429
+```
+
+### South America
+```
+./process_maps.py --use-ram --max-heap-space 48g \
+    --output-map-name south_america \
+    --cached-maps-dir ~/osmmapdata/cache/south_america/20180225-1429
+```
+
+### Central America
+
+*NOTE: This CAN NOT fit into 64Gb RAM, you'll need to setup a 128Gb SSD/Disk at ~/osm_map_processing/tmp for this to not enounter heap space errors* 
+
+```
+./process_maps.py --max-heap-space 48g \
+    --output-map-name central_america \
+    --cached-maps-dir ~/osmmapdata/cache/central_america/20180225-1429
+```
+