209 lines
7.4 KiB
Markdown
209 lines
7.4 KiB
Markdown
|
# Virtualization
|
||
|
|
||
|
Some notes on setting up Docker build hosts using emulation. Specifically using qemu to setup arm32v7 and arm64v8 environments for Docker builds.
|
||
|
|
||
|
Handy if you need are using an SBC that's too 'small' for building searx/gogs but *can* run the final Docker image.
|
||
|
|
||
|
## Inspiration / Further Reading
|
||
|
|
||
|
- [https://resin.io/blog/building-arm-containers-on-any-x86-machine-even-dockerhub/](https://resin.io/blog/building-arm-containers-on-any-x86-machine-even-dockerhub/)
|
||
|
- [https://github.com/petrosagg/armv7hf-python-dockerhub/blob/master/Dockerfile](https://github.com/petrosagg/armv7hf-python-dockerhub/blob/master/Dockerfile)
|
||
|
- [https://github.com/ViViDboarder/docker-rpi-homebridge/blob/master/Dockerfile](https://github.com/ViViDboarder/docker-rpi-homebridge/blob/master/Dockerfile)
|
||
|
- This BREAKS when not cross building so you can't easily run docker build / up on a docker file w/ these in it
|
||
|
- [https://wiki.qemu.org/Documentation/Platforms/ARM](https://wiki.qemu.org/Documentation/Platforms/ARM)
|
||
|
- [https://translatedcode.wordpress.com/2016/11/03/installing-debian-on-qemus-32-bit-arm-virt-board/](https://translatedcode.wordpress.com/2016/11/03/installing-debian-on-qemus-32-bit-arm-virt-board/)
|
||
|
- [https://translatedcode.wordpress.com/2017/07/24/installing-debian-on-qemus-64-bit-arm-virt-board/](https://translatedcode.wordpress.com/2017/07/24/installing-debian-on-qemus-64-bit-arm-virt-board/)
|
||
|
- [https://www.raymii.org/s/articles/virt-install_introduction_and_copy_paste_distro_install_commands.html](https://www.raymii.org/s/articles/virt-install_introduction_and_copy_paste_distro_install_commands.html)
|
||
|
- [http://ftp.nl.debian.org/debian/dists/stretch/main/](http://ftp.nl.debian.org/debian/dists/stretch/main/)
|
||
|
- [https://www.collabora.com/news-and-blog/blog/2017/06/20/debian-armhf-vm-on-arm64-server/](https://www.collabora.com/news-and-blog/blog/2017/06/20/debian-armhf-vm-on-arm64-server/)
|
||
|
- [http://blog.system76.com/post/139138591598/howto-qemu-w-ubuntu-xenial-host-uefi-guest](http://blog.system76.com/post/139138591598/howto-qemu-w-ubuntu-xenial-host-uefi-guest)
|
||
|
|
||
|
## Minor notes
|
||
|
|
||
|
- Use arm64 for arm64v8
|
||
|
- Use armhf for arm32v
|
||
|
|
||
|
## Prep
|
||
|
|
||
|
``` bash
|
||
|
|
||
|
apt install qemu-system-arm libguestfs-tools qemu-efi qemu-efi-aarch64 qemu-efi-arm
|
||
|
|
||
|
```
|
||
|
|
||
|
## Plain QEMU
|
||
|
|
||
|
These processes will let you bootstrap a virtual machine for later import into libvirt management engines.
|
||
|
|
||
|
### 64bit arm v8
|
||
|
|
||
|
``` bash
|
||
|
|
||
|
# Probably should have this in a dedicated dir, no?
|
||
|
mkdir arm64v8
|
||
|
cd arm64v8
|
||
|
# Pull installer
|
||
|
wget http://ports.ubuntu.com/ubuntu-ports/dists/xenial/main/installer-arm64/current/images/netboot/mini.iso
|
||
|
mv mini.iso net-install.iso
|
||
|
# Create storage for VM
|
||
|
qemu-img create -f qcow2 arm64v8-xenial.qcow2 16G
|
||
|
# Prep EFI boot
|
||
|
cp /usr/share/AAVMF/AAVMF_VARS.fd AAVMF_VARS.fd
|
||
|
# Install
|
||
|
# WARNING: YOU MUST TWEAK THE GRUB KERNEL CONSOLE LINE TO HAVE 'console=ttyAMA0' IMMEDIATELY FOLLOWING THE KERNEL IMAGE OR YOU WON'T GET OUTPUT
|
||
|
# NOTE: Use CTRL-H for backspace and CTRL-X to boot
|
||
|
# NOTE: virtio-blk-device and virtio-net-device are necessary as the mini.iso (renamed to net-install.iso) doesn't support PCI (fixed post-install)
|
||
|
qemu-system-aarch64 -M virt -m 2048 -smp 4 -cpu cortex-a53 \
|
||
|
-drive if=pflash,format=raw,readonly,file=/usr/share/AAVMF/AAVMF_CODE.fd \
|
||
|
-drive if=pflash,format=raw,file=AAVMF_VARS.fd \
|
||
|
-cdrom net-install.iso \
|
||
|
-drive if=none,file=arm64v8-xenial.qcow2,format=qcow2,id=hd \
|
||
|
-device virtio-blk-device,drive=hd \
|
||
|
-netdev user,id=arm0 -device virtio-net-device,netdev=arm0 \
|
||
|
-nographic -no-reboot
|
||
|
# Backup base install (just in case)
|
||
|
cp arm64v8-xenial.qcow2 arm64v8-xenial-base-install.qcow2
|
||
|
# Run the VM with the necessary parms/infos
|
||
|
qemu-system-aarch64 -M virt -m 2048 -smp 4 -cpu cortex-a53 \
|
||
|
-drive if=pflash,format=raw,readonly,file=/usr/share/AAVMF/AAVMF_CODE.fd \
|
||
|
-drive if=pflash,format=raw,file=AAVMF_VARS.fd \
|
||
|
-drive if=none,file=arm64v8-xenial.qcow2,format=qcow2,id=hd \
|
||
|
-device virtio-blk-device,drive=hd \
|
||
|
-netdev user,id=arm0 -device virtio-net-device,netdev=arm0 \
|
||
|
-nographic -no-reboot
|
||
|
|
||
|
```
|
||
|
|
||
|
### 32bit arm v7
|
||
|
|
||
|
``` bash
|
||
|
|
||
|
# Probably should have this in a dedicated dir, no?
|
||
|
mkdir arm32v7
|
||
|
cd arm32v7
|
||
|
# Get netbook kernel/initrd
|
||
|
wget -O installer-vmlinuz http://ports.ubuntu.com/ubuntu-ports/dists/xenial/main/installer-armhf/current/images/generic-lpae/netboot/vmlinuz
|
||
|
wget -O installer-initrd.gz http://ports.ubuntu.com/ubuntu-ports/dists/xenial/main/installer-armhf/current/images/generic-lpae/netboot/initrd.gz
|
||
|
# Create storage for VM
|
||
|
qemu-img create -f qcow2 arm32v7-xenial.qcow2 16G
|
||
|
# Prep EFI boot
|
||
|
cp /usr/share/AAVMF/AAVMF32_VARS.fd AAVMF32_VARS.fd
|
||
|
# Install
|
||
|
qemu-system-arm -M virt -m 2048 -smp 4 -cpu cortex-a15 \
|
||
|
-kernel installer-vmlinuz \
|
||
|
-initrd installer-initrd.gz \
|
||
|
-drive if=none,file=arm32v7-xenial.qcow2,format=qcow2,id=hd \
|
||
|
-device virtio-blk-pci,drive=hd \
|
||
|
-netdev user,id=arm0 -device virtio-net-pci,netdev=arm0 \
|
||
|
-nographic -no-reboot
|
||
|
# Backup base install (just in case)
|
||
|
cp arm32v7-xenial.qcow2 arm32v7-xenial-base-install.qcow2
|
||
|
# Copy vmlinuz/initrd from root filesystem for final boot
|
||
|
virt-filesystems -a arm32v7-xenial.qcow2
|
||
|
virt-ls -a arm32v7-xenial.qcow2 /boot/
|
||
|
virt-copy-out -a arm32v7-xenial.qcow2 /boot/vmlinuz-4.4.0-127-generic-lpae /boot/initrd.img-4.4.0-127-generic-lpae .
|
||
|
# Boot!
|
||
|
qemu-system-arm -M virt -m 2048 -smp 4 -cpu cortex-a15 \
|
||
|
-kernel vmlinuz-4.4.0-127-generic-lpae \
|
||
|
-initrd initrd.img-4.4.0-127-generic-lpae \
|
||
|
-drive if=none,file=arm32v7-xenial.qcow2,format=qcow2,id=hd \
|
||
|
-device virtio-blk-pci,drive=hd \
|
||
|
-netdev user,id=arm0 -device virtio-net-pci,netdev=arm0 \
|
||
|
-nographic -no-reboot
|
||
|
|
||
|
```
|
||
|
|
||
|
## virsh/virt-install
|
||
|
|
||
|
This section is very much a work in progress. Based on further research / reading you'll need to bootstrap the VM with raw qemu and then import to virsh.
|
||
|
|
||
|
``` bash
|
||
|
|
||
|
virt-install --name ubuntu-16.04-arm64v8 \
|
||
|
--import \
|
||
|
--virt-type qemu \
|
||
|
--arch aarch64 \
|
||
|
--machine virt \
|
||
|
--memory 2048 \
|
||
|
--vcpus 4 \
|
||
|
--noacpi \
|
||
|
--os-type linux \
|
||
|
--os-variant ubuntu16.04 \
|
||
|
--disk path=template-ubuntu-16.04-arm64v8.qcow2,size=25,bus=virtio \
|
||
|
--network default \
|
||
|
--graphics none \
|
||
|
|
||
|
```
|
||
|
|
||
|
## Useful virsh commands
|
||
|
|
||
|
For post install usage
|
||
|
|
||
|
``` bash
|
||
|
|
||
|
virsh list --all # List all VMs
|
||
|
virsh dumpxml --domain VM_NAME # Dump the XML for the VM
|
||
|
virsh destroy --domain VM_NAME # Force shutdown VM
|
||
|
virsh undefine --domain VM_NAME # Remove/Delete VM
|
||
|
virsh console VM_NAME
|
||
|
virsh edit VM_NAME
|
||
|
rm DISK_IMAGE_PATH
|
||
|
|
||
|
```
|
||
|
|
||
|
## First Boot
|
||
|
|
||
|
``` bash
|
||
|
|
||
|
vim /etc/default/grub
|
||
|
update-grub
|
||
|
apt update && apt install sudo vim tmux screen wget curl htop git nano rsync iotop nload unzip
|
||
|
ufw allow ssh
|
||
|
ufw enable
|
||
|
|
||
|
```
|
||
|
|
||
|
## rclone
|
||
|
|
||
|
``` bash
|
||
|
|
||
|
curl https://rclone.org/install.sh | sudo bash
|
||
|
|
||
|
```
|
||
|
|
||
|
## Docker
|
||
|
|
||
|
``` bash
|
||
|
|
||
|
apt remove docker docker-engine docker.io
|
||
|
apt install apt-transport-https ca-certificates curl gnupg2 software-properties-common
|
||
|
|
||
|
echo "deb [arch=arm64] https://download.docker.com/linux/ubuntu \
|
||
|
$(lsb_release -cs) stable" | \
|
||
|
sudo tee /etc/apt/sources.list.d/docker.list
|
||
|
|
||
|
echo "deb [arch=armhf] https://download.docker.com/linux/ubuntu \
|
||
|
$(lsb_release -cs) stable" | \
|
||
|
sudo tee /etc/apt/sources.list.d/docker.list
|
||
|
|
||
|
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
|
||
|
apt update
|
||
|
apt install docker-ce python3 python3-pip
|
||
|
pip3 install docker-compose
|
||
|
poweroff
|
||
|
|
||
|
```
|
||
|
|
||
|
## Export Docker Container Images
|
||
|
|
||
|
``` bash
|
||
|
|
||
|
for repo in `docker image ls --format "{{.Repository}}"`; do
|
||
|
outfile=`echo $repo | sed 's+/+_+g'`
|
||
|
echo "$repo -> $outfile"
|
||
|
docker save -o ${outfile}.tar $repo
|
||
|
done
|
||
|
|
||
|
```
|
||
|
|