Add notes on how to run this with azure vms
This commit is contained in:
parent
612f125219
commit
0f6d71f1c0
109
azure_notes.md
Normal file
109
azure_notes.md
Normal file
|
@ -0,0 +1,109 @@
|
||||||
|
# Azure Run Notes
|
||||||
|
|
||||||
|
### Tested on
|
||||||
|
|
||||||
|
* Azure Virtual Machine
|
||||||
|
* E8S_V3 Standard Sized VM
|
||||||
|
* 8 vCPUs
|
||||||
|
* 64Gb RAM
|
||||||
|
* 128Gb local SSD
|
||||||
|
* Azure blob storage for map data
|
||||||
|
|
||||||
|
# Post install / first login
|
||||||
|
|
||||||
|
```
|
||||||
|
wget https://packages.microsoft.com/config/ubuntu/16.04/packages-microsoft-prod.deb
|
||||||
|
sudo dpkg -i packages-microsoft-prod.deb
|
||||||
|
sudo apt update
|
||||||
|
sudo apt install git tmux vim openjdk-8-jre python3 htop iotop nload
|
||||||
|
sudo apt upgrade
|
||||||
|
sudo systemctl reboot
|
||||||
|
```
|
||||||
|
|
||||||
|
# Setup Azure blob storage
|
||||||
|
|
||||||
|
### How to blob storage @ CLI
|
||||||
|
https://docs.microsoft.com/en-us/azure/storage/blobs/storage-how-to-mount-container-linux
|
||||||
|
|
||||||
|
### Setup VM
|
||||||
|
|
||||||
|
```
|
||||||
|
wget https://packages.microsoft.com/config/ubuntu/16.04/packages-microsoft-prod.deb
|
||||||
|
sudo dpkg -i packages-microsoft-prod.deb
|
||||||
|
sudo apt-get update
|
||||||
|
sudo apt-get install blobfuse
|
||||||
|
```
|
||||||
|
|
||||||
|
### Configure blob storage in VM
|
||||||
|
|
||||||
|
```
|
||||||
|
vim ~/blob_fuse_connection.cfg
|
||||||
|
accountName myaccount
|
||||||
|
accountKey myaccesskey==
|
||||||
|
containerName mycontainer
|
||||||
|
```
|
||||||
|
|
||||||
|
### Setup various filesystem dependencies / zones
|
||||||
|
|
||||||
|
```
|
||||||
|
mkdir /home/ubuntu/osmmapdata
|
||||||
|
sudo chown: /home/ubuntu/osmmapdata
|
||||||
|
sudo mkdir /mnt/resource/blobfusetmp
|
||||||
|
sudo chown ubuntu: /mnt/resource/blobfusetmp
|
||||||
|
```
|
||||||
|
|
||||||
|
### Mount blob storage for use
|
||||||
|
|
||||||
|
```
|
||||||
|
blobfuse /home/ubuntu/osmmapdata \
|
||||||
|
--tmp-path=/mnt/resource/blobfusetmp \
|
||||||
|
--config-file=/home/ubuntu/blob_connection.cfg \
|
||||||
|
-o attr_timeout=240 \
|
||||||
|
-o entry_timeout=240 \
|
||||||
|
-o negative_timeout=120 \
|
||||||
|
--file-cache-timeout-in-seconds=0 \
|
||||||
|
-o allow_root
|
||||||
|
```
|
||||||
|
|
||||||
|
### View on Windows Desktop
|
||||||
|
https://azure.microsoft.com/en-us/features/storage-explorer/
|
||||||
|
|
||||||
|
# Run OSM Map Processing
|
||||||
|
|
||||||
|
### Prep session
|
||||||
|
```
|
||||||
|
tmux new -s maps
|
||||||
|
git clone https://github.com/mcrosson/osm_map_processing.git
|
||||||
|
cd osm_map_processing
|
||||||
|
```
|
||||||
|
|
||||||
|
### Download maps
|
||||||
|
```
|
||||||
|
# Download relevant maps using upcoming download util to ~/osmmapdata/cache/blah/date-time
|
||||||
|
```
|
||||||
|
|
||||||
|
### Midwest USA
|
||||||
|
|
||||||
|
```
|
||||||
|
./process_maps.py --use-ram --max-heap-space 48g \
|
||||||
|
--output-map-name midwest \
|
||||||
|
--cached-maps-dir ~/osmmapdata/cache/midwest/20180225-1429
|
||||||
|
```
|
||||||
|
|
||||||
|
### South America
|
||||||
|
```
|
||||||
|
./process_maps.py --use-ram --max-heap-space 48g \
|
||||||
|
--output-map-name south_america \
|
||||||
|
--cached-maps-dir ~/osmmapdata/cache/south_america/20180225-1429
|
||||||
|
```
|
||||||
|
|
||||||
|
### Central America
|
||||||
|
|
||||||
|
*NOTE: This CAN NOT fit into 64Gb RAM, you'll need to setup a 128Gb SSD/Disk at ~/osm_map_processing/tmp for this to not enounter heap space errors*
|
||||||
|
|
||||||
|
```
|
||||||
|
./process_maps.py --max-heap-space 48g \
|
||||||
|
--output-map-name central_america \
|
||||||
|
--cached-maps-dir ~/osmmapdata/cache/central_america/20180225-1429
|
||||||
|
```
|
||||||
|
|
Reference in a new issue