lollipopcloud/advanced/virtualization.md

# Virtualization

Some notes on setting up Docker build hosts using emulation. Specifically using qemu to setup arm32v7 and arm64v8 environments for Docker builds.

Handy if you need are using an SBC that's too 'small' for building searx/gogs but *can* run the final Docker image.

## Inspiration / Further Reading

- [https://resin.io/blog/building-arm-containers-on-any-x86-machine-even-dockerhub/](https://resin.io/blog/building-arm-containers-on-any-x86-machine-even-dockerhub/)
- [https://github.com/petrosagg/armv7hf-python-dockerhub/blob/master/Dockerfile](https://github.com/petrosagg/armv7hf-python-dockerhub/blob/master/Dockerfile)
- [https://github.com/ViViDboarder/docker-rpi-homebridge/blob/master/Dockerfile](https://github.com/ViViDboarder/docker-rpi-homebridge/blob/master/Dockerfile)
  - This BREAKS when not cross building so you can't easily run docker build / up on a docker file w/ these in it
- [https://wiki.qemu.org/Documentation/Platforms/ARM](https://wiki.qemu.org/Documentation/Platforms/ARM)
- [https://translatedcode.wordpress.com/2016/11/03/installing-debian-on-qemus-32-bit-arm-virt-board/](https://translatedcode.wordpress.com/2016/11/03/installing-debian-on-qemus-32-bit-arm-virt-board/)
- [https://translatedcode.wordpress.com/2017/07/24/installing-debian-on-qemus-64-bit-arm-virt-board/](https://translatedcode.wordpress.com/2017/07/24/installing-debian-on-qemus-64-bit-arm-virt-board/)
- [https://www.raymii.org/s/articles/virt-install_introduction_and_copy_paste_distro_install_commands.html](https://www.raymii.org/s/articles/virt-install_introduction_and_copy_paste_distro_install_commands.html)
- [http://ftp.nl.debian.org/debian/dists/stretch/main/](http://ftp.nl.debian.org/debian/dists/stretch/main/)
- [https://www.collabora.com/news-and-blog/blog/2017/06/20/debian-armhf-vm-on-arm64-server/](https://www.collabora.com/news-and-blog/blog/2017/06/20/debian-armhf-vm-on-arm64-server/)
- [http://blog.system76.com/post/139138591598/howto-qemu-w-ubuntu-xenial-host-uefi-guest](http://blog.system76.com/post/139138591598/howto-qemu-w-ubuntu-xenial-host-uefi-guest)

## Minor notes

- Use arm64 for arm64v8
- Use armhf for arm32v

## Prep

``` bash

apt install qemu-system-arm libguestfs-tools qemu-efi qemu-efi-aarch64 qemu-efi-arm

```

## Plain QEMU

These processes will let you bootstrap a virtual machine for later import into libvirt management engines.

### 64bit arm v8

``` bash

# Probably should have this in a dedicated dir, no?
mkdir arm64v8
cd arm64v8
# Pull installer
wget http://ports.ubuntu.com/ubuntu-ports/dists/xenial/main/installer-arm64/current/images/netboot/mini.iso
mv mini.iso net-install.iso
# Create storage for VM
qemu-img create -f qcow2 arm64v8-xenial.qcow2 16G
# Prep EFI boot
cp /usr/share/AAVMF/AAVMF_VARS.fd AAVMF_VARS.fd
# Install
#     WARNING: YOU MUST TWEAK THE GRUB KERNEL CONSOLE LINE TO HAVE 'console=ttyAMA0' IMMEDIATELY FOLLOWING THE KERNEL IMAGE OR YOU WON'T GET OUTPUT
#     NOTE: Use CTRL-H for backspace and CTRL-X to boot
#     NOTE: virtio-blk-device and virtio-net-device are necessary as the mini.iso (renamed to net-install.iso) doesn't support PCI (fixed post-install)
qemu-system-aarch64 -M virt -m 2048 -smp 4 -cpu cortex-a53 \
  -drive if=pflash,format=raw,readonly,file=/usr/share/AAVMF/AAVMF_CODE.fd \
  -drive if=pflash,format=raw,file=AAVMF_VARS.fd \
  -cdrom net-install.iso \
  -drive if=none,file=arm64v8-xenial.qcow2,format=qcow2,id=hd \
  -device virtio-blk-device,drive=hd \
  -netdev user,id=arm0 -device virtio-net-device,netdev=arm0 \
  -nographic -no-reboot
# Backup base install (just in case)
cp arm64v8-xenial.qcow2 arm64v8-xenial-base-install.qcow2
# Run the VM with the necessary parms/infos
qemu-system-aarch64 -M virt -m 2048 -smp 4 -cpu cortex-a53 \
  -drive if=pflash,format=raw,readonly,file=/usr/share/AAVMF/AAVMF_CODE.fd \
  -drive if=pflash,format=raw,file=AAVMF_VARS.fd \
  -drive if=none,file=arm64v8-xenial.qcow2,format=qcow2,id=hd \
  -device virtio-blk-device,drive=hd \
  -netdev user,id=arm0 -device virtio-net-device,netdev=arm0 \
  -nographic -no-reboot

```

### 32bit arm v7

``` bash

# Probably should have this in a dedicated dir, no?
mkdir arm32v7
cd arm32v7
# Get netbook kernel/initrd
wget -O installer-vmlinuz http://ports.ubuntu.com/ubuntu-ports/dists/xenial/main/installer-armhf/current/images/generic-lpae/netboot/vmlinuz
wget -O installer-initrd.gz http://ports.ubuntu.com/ubuntu-ports/dists/xenial/main/installer-armhf/current/images/generic-lpae/netboot/initrd.gz
# Create storage for VM
qemu-img create -f qcow2 arm32v7-xenial.qcow2 16G
# Prep EFI boot
cp /usr/share/AAVMF/AAVMF32_VARS.fd AAVMF32_VARS.fd
# Install
qemu-system-arm -M virt -m 2048 -smp 4 -cpu cortex-a15 \
  -kernel installer-vmlinuz \
  -initrd installer-initrd.gz \
  -drive if=none,file=arm32v7-xenial.qcow2,format=qcow2,id=hd \
  -device virtio-blk-pci,drive=hd \
  -netdev user,id=arm0 -device virtio-net-pci,netdev=arm0 \
  -nographic -no-reboot
# Backup base install (just in case)
cp arm32v7-xenial.qcow2 arm32v7-xenial-base-install.qcow2
# Copy vmlinuz/initrd from root filesystem for final boot
virt-filesystems -a arm32v7-xenial.qcow2
virt-ls -a arm32v7-xenial.qcow2 /boot/
virt-copy-out -a arm32v7-xenial.qcow2 /boot/vmlinuz-4.4.0-127-generic-lpae /boot/initrd.img-4.4.0-127-generic-lpae .
# Boot!
qemu-system-arm -M virt -m 2048 -smp 4 -cpu cortex-a15 \
  -kernel vmlinuz-4.4.0-127-generic-lpae \
  -initrd initrd.img-4.4.0-127-generic-lpae \
  -drive if=none,file=arm32v7-xenial.qcow2,format=qcow2,id=hd \
  -device virtio-blk-pci,drive=hd \
  -netdev user,id=arm0 -device virtio-net-pci,netdev=arm0 \
  -nographic -no-reboot

```

## virsh/virt-install

This section is very much a work in progress. Based on further research / reading you'll need to bootstrap the VM with raw qemu and then import to virsh.

``` bash

virt-install --name ubuntu-16.04-arm64v8 \
  --import \
  --virt-type qemu \
  --arch aarch64 \
  --machine virt \
  --memory 2048 \
  --vcpus 4 \
  --noacpi \
  --os-type linux \
  --os-variant ubuntu16.04 \
  --disk path=template-ubuntu-16.04-arm64v8.qcow2,size=25,bus=virtio \
  --network default \
  --graphics none \

```

## Useful virsh commands

For post install usage

``` bash

virsh list --all # List all VMs
virsh dumpxml --domain VM_NAME # Dump the XML for the VM
virsh destroy --domain VM_NAME # Force shutdown VM
virsh undefine --domain VM_NAME # Remove/Delete VM
virsh console VM_NAME
virsh edit VM_NAME
rm DISK_IMAGE_PATH

```

## First Boot

``` bash

vim /etc/default/grub
update-grub
apt update && apt install sudo vim tmux screen wget curl htop git nano rsync iotop nload unzip
ufw allow ssh
ufw enable

```

## rclone

``` bash

curl https://rclone.org/install.sh | sudo bash

```

## Docker

``` bash

apt remove docker docker-engine docker.io
apt install apt-transport-https ca-certificates curl gnupg2 software-properties-common

echo "deb [arch=arm64] https://download.docker.com/linux/ubuntu \
     $(lsb_release -cs) stable" | \
    sudo tee /etc/apt/sources.list.d/docker.list

echo "deb [arch=armhf] https://download.docker.com/linux/ubuntu \
     $(lsb_release -cs) stable" | \
    sudo tee /etc/apt/sources.list.d/docker.list

curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
apt update
apt install docker-ce python3 python3-pip
pip3 install docker-compose
poweroff

```

## Export Docker Container Images

``` bash

for repo in `docker image ls --format "{{.Repository}}"`; do
    outfile=`echo $repo | sed 's+/+_+g'`
    echo "$repo -> $outfile"
    docker save -o ${outfile}.tar $repo
done

```