Create a script and instructions to create an aarch64 sysroot via qemu (#35420)
* Prepare making a sysroot via qemu, so we do not need a raspberry pi ready (even though this is MUCH slower)
* Update gitignore
* Rename file for clearer naming
* Start adding some documentation on sysroot generation
* Restyle and update the image to 22.04 since this is what we use today
* Restyle
* vm is a common word
* Add helper info for mkpasswd existence
* More help messages
* More help and have auto-install because it is convenient
* Restyle
* More convenient text and help
* More updates for friendlyness
* Fix cloudinit schema
* Allocate more virtual CPUs as some things run slightly faster (a bit over 10% faster boot for me)
* Created a script to upload the cipd package
* Script cleanup and readme update
* User friendly progress and output
* Updated gitignore and script for nicer content
* Script cleanup for symlinks (goes from 7GB to 1.1GB)
* More updates to the rsync execution options
* Single rsync invokation, so that safe symlinks works better
* Restyle
* Update tag to be unique, since I was missing a symlink before and that seemed more broken
* Update integrations/docker/images/stage-1/chip-build-crosscompile/start-sysroot-vm.sh
Co-authored-by: Kiel Oleson <kielo@apple.com>
* Restyle
* Update start-sysroot-vm.sh
Update text about very weak password.
---------
Co-authored-by: Andrei Litvin <andreilitvin@google.com>
Co-authored-by: Kiel Oleson <kielo@apple.com>
diff --git a/.github/.wordlist.txt b/.github/.wordlist.txt
index 9cb24bc..c84d403 100644
--- a/.github/.wordlist.txt
+++ b/.github/.wordlist.txt
@@ -1532,6 +1532,7 @@
visualstudio
vlatest
VLEDs
+vm
vn
vnc
vous
diff --git a/integrations/docker/images/stage-1/chip-build-crosscompile/.gitignore b/integrations/docker/images/stage-1/chip-build-crosscompile/.gitignore
index 37e0c7a..e4e6d2a 100644
--- a/integrations/docker/images/stage-1/chip-build-crosscompile/.gitignore
+++ b/integrations/docker/images/stage-1/chip-build-crosscompile/.gitignore
@@ -1,3 +1,10 @@
.cipd
+cidata.iso
+cipd.yaml
ensure_file.txt
+meta-data
+sysrootsrv.img
ubuntu-21.04-aarch64-sysroot.tar.xz
+ubuntu-24.04-aarch64-sysroot
+ubuntu-24.04-server-cloudimg-arm64.img
+user-data
diff --git a/integrations/docker/images/stage-1/chip-build-crosscompile/README.md b/integrations/docker/images/stage-1/chip-build-crosscompile/README.md
index d27fc2f..4d97a7c 100644
--- a/integrations/docker/images/stage-1/chip-build-crosscompile/README.md
+++ b/integrations/docker/images/stage-1/chip-build-crosscompile/README.md
@@ -3,11 +3,55 @@
This image adds cross-compilation support files (a suitable sysroot) on top of
the regular linux chip-build image.
-The build assumes a file named `ubuntu-21.04-aarch64-sysroot.tar.xz` exists in
+The build assumes a file named `ubuntu-22.04-aarch64-sysroot.tar.xz` exists in
the current directory. This can generally be manually built using an existing
Raspberry Pi device (or equivalent qemu) and a convenience copy was created in
CIPD
+#### Creating a Sysroot (qemu virtual emulator)
+
+NOTE: this approach is slower due to emulation usage, however has the advantage
+of not requiring separate hardware.
+
+Ensure `qemu` and `virt-install` prerequisites are met:
+
+```
+apt-get install qemu-system-arm virtinst libvirt-daemon
+```
+
+Start up the sysroot virtual machine. This will be named `sysrootsrv`:
+
+```
+./start-sysroot-vm.sh
+```
+
+Connect to a console to monitor progress. The VM is configured to pre-install
+packages based on CHIP BUILDING.md documentation plus any additional packages.
+
+```
+virsh console sysrootsrv
+```
+
+Once installation completed, you can also login as `ubuntu/1234` or use
+`ssh ubuntu@localhost -p 5555`
+
+The current image is based on Ubuntu 24.04, so you can create an image via:
+
+```
+rsync -avL -e 'ssh -p 5555' ubuntu@localhost:/lib ubuntu-24.04-aarch64-sysroot
+rsync -avL -e 'ssh -p 5555' ubuntu@localhost:/usr/lib ubuntu-24.04-aarch64-sysroot/usr
+rsync -avL -e 'ssh -p 5555' ubuntu@localhost:/usr/include ubuntu-24.04-aarch64-sysroot/usr
+```
+
+At this point you have a sysroot in `ubuntu-24.04-aarch64-sysroot`
+
+To create/upload a CIPD package tagged with the current date, a script is
+provided:
+
+```
+upload-cipd-package.sh
+```
+
#### Creating a Sysroot
Start with a fresh Raspberry PI image:
@@ -15,16 +59,16 @@
- Use the Raspberry pi imager from https://www.raspberrypi.org/software/
- Follow installation instructions from the CHIP BUILDING.md document.
Generally this includes:
- - Use Ubuntu 21.04 server 64 bit server image
+ - Use Ubuntu 22.04 server 64 bit server image
- Install required compile dependencies via `apt-get`. You may skip
installing git and python3 dependencies to save some image size
Generate a sysroot. You can do this locally (slow due to SD performance):
```
-rsync -avL /lib ubuntu-21.04-aarch64-sysroot/
-rsync -avL /usr/lib ubuntu-21.04-aarch64-sysroot/usr
-rsync -avL /usr/include ubuntu-21.04-aarch64-sysroot/usr
+rsync -avL /lib ubuntu-22.04-aarch64-sysroot/
+rsync -avL /usr/lib ubuntu-22.04-aarch64-sysroot/usr
+rsync -avL /usr/include ubuntu-22.04-aarch64-sysroot/usr
```
or you can copy directly to your host machine via
@@ -33,9 +77,9 @@
export PI="ip-of-raspberry-pi"
ssh-copy-id ubuntu@$PI
-rsync -avL ubuntu@$PI:/lib ubuntu-21.04-aarch64-sysroot
-rsync -avL ubuntu@$PI:/usr/lib ubuntu-21.04-aarch64-sysroot/usr
-rsync -avL ubuntu@$PI:/include/lib ubuntu-21.04-aarch64-sysroot/usr
+rsync -avL ubuntu@$PI:/lib ubuntu-22.04-aarch64-sysroot
+rsync -avL ubuntu@$PI:/usr/lib ubuntu-22.04-aarch64-sysroot/usr
+rsync -avL ubuntu@$PI:/usr/include ubuntu-22.04-aarch64-sysroot/usr
```
NOTE: in the future, if creating a 32-bit image (not covered by this docker
@@ -47,7 +91,7 @@
Once the sysroot is on the host machine, create the corresponding `tar.xz` file:
```
-tar cvfJ ubuntu-21.04-aarch64-sysroot.tar.xz ubuntu-21.04-aarch64-sysroot
+tar cvfJ ubuntu-22.04-aarch64-sysroot.tar.xz ubuntu-22.04-aarch64-sysroot
```
#### CIPD image
@@ -55,12 +99,12 @@
CIPD image is provided as a convenience to avoid the need to spend time
rebuilding a sysroot. It is located at:
-https://chrome-infra-packages.appspot.com/p/experimental/matter/sysroot/ubuntu-21.04-aarch64/+/
+https://chrome-infra-packages.appspot.com/p/experimental/matter/sysroot/ubuntu-22.04-aarch64/+/
and can be downloaded using the `cipd` script from
[depot_tools](https://dev.chromium.org/developers/how-tos/depottools):
```
-echo 'experimental/matter/sysroot/ubuntu-21.04-aarch64 latest' > ensure_file.txt
+echo 'experimental/matter/sysroot/ubuntu-22.04-aarch64 latest' > ensure_file.txt
cipd ensure -ensure-file ensure_file.txt -root ./
```
diff --git a/integrations/docker/images/stage-1/chip-build-crosscompile/start-sysroot-vm.sh b/integrations/docker/images/stage-1/chip-build-crosscompile/start-sysroot-vm.sh
new file mode 100755
index 0000000..fb56c43
--- /dev/null
+++ b/integrations/docker/images/stage-1/chip-build-crosscompile/start-sysroot-vm.sh
@@ -0,0 +1,217 @@
+#!/usr/bin/env bash
+
+# This script is intended to build a SYSROOT that can be used as a base to cross-compile
+# CHIP for aarch64 (i.e. that runs on a RaspberryPi) from a amd64 linux machine
+
+set -e
+
+start_console=1
+
+while getopts "hin" opt; do
+ case $opt in
+ h)
+ echo "Usage: $0 [-h] [-i]"
+ echo " -h Displays this help message"
+ echo " -i Attempt to 'sudo apt-get install' required packages"
+ echo " -n Do not start a console after vm is created"
+ exit 0
+ ;;
+ i)
+ echo "Installing required packages ..."
+ sudo apt-get install \
+ genisoimage \
+ libvirt-daemon \
+ qemu-system-arm \
+ qemu-utils \
+ virtinst \
+ whois \
+ ;
+ ;;
+ n)
+ echo "Console will not be started after virtual machine startup."
+ start_console=0
+ ;;
+ esac
+done
+
+if ! which qemu-system-aarch64 >/dev/null; then
+ echo "Cannot find 'qemu-system-aarch64'. Did you 'sudo apt-get install qemu-system-arm' ?"
+ exit 1
+fi
+
+if ! which virt-install >/dev/null; then
+ echo "Cannot find 'virt-install'. Did you 'sudo apt-get install virtinst' ?"
+ exit 1
+fi
+
+if ! [ -f /usr/sbin/libvirtd ]; then
+ echo "Cannot find 'virt-install'. Did you 'sudo apt-get install libvirt-daemon' ?"
+ exit 1
+fi
+
+if ! which mkpasswd >/dev/null; then
+ echo "Cannot find 'mkpasswd'. Did you 'sudo apt-get install whois' ?"
+ exit 1
+fi
+
+if ! which qemu-img >/dev/null; then
+ echo "Cannot find 'qemu-img'. Did you 'sudo apt-get install qemu-utis' ?"
+ exit 1
+fi
+
+if ! which genisoimage >/dev/null; then
+ echo "Cannot find 'genisoimage'. Did you 'sudo apt-get install genisoimage' ?"
+ exit 1
+fi
+
+CLOUD_IMAGE_URL="https://cloud-images.ubuntu.com/releases/24.04/release-20240821/ubuntu-24.04-server-cloudimg-arm64.img"
+CLOUD_IMAGE_FILE=$(basename "$CLOUD_IMAGE_URL")
+
+if ! [ -f "$CLOUD_IMAGE_FILE" ]; then
+ echo "Image $CLOUD_IMAGE_FILE does not exist. Downloading ..."
+ wget "$CLOUD_IMAGE_URL"
+else
+ echo "Using existing $CLOUD_IMAGE_FILE ..."
+fi
+
+rm -f sysrootsrv.img
+qemu-img create -b "$CLOUD_IMAGE_FILE" -f qcow2 -F qcow2 sysrootsrv.img 10G
+
+# local user allowed to SSH in to the host directly without a password
+SSH_KEY=$(cat ~/.ssh/id_rsa.pub)
+
+# set a login password.
+# NOTE: this is a VERY WEAK password, however this should be ok because
+# this is a local and temporary virtual machine that is NOT accessible
+# anywhere except through the redirected port 5555 on localhost or
+# via "virsh console" (which is the primary purpose for it: debugging)
+#
+# This is NOT meant as a persistent vm and should be deleted after use.
+PASSWORD_HASH=$(mkpasswd 1234)
+
+# Create cloud init files
+cat >user-data <<TEXTEND
+#cloud-config
+
+users:
+ - default
+ - name: ubuntu
+ shell: /bin/bash
+ ssh_authorized_keys:
+ - $SSH_KEY
+ sudo: "ALL=(ALL) NOPASSWD:ALL"
+ groups: sudo,admin,users
+ shell: /bin/bash
+ lock_passwd: false
+ passwd: $PASSWORD_HASH
+
+hostname: sysrootsrv
+
+package_update: true
+package_upgrade: true
+packages:
+ - g++
+ - gcc
+ - libavahi-client-dev
+ - libcairo2-dev
+ - libdbus-1-dev
+ - libgirepository1.0-dev
+ - libglib2.0-dev
+ - libreadline-dev
+ - libsdl2-dev
+ - libssl-dev
+ - ninja-build
+ - pkg-config
+ - unzip
+
+runcmd:
+ # Ubuntu cloud-init merges with a locked ubuntu, so the "lock_passwd" option does
+ # not actually work. Unlock it here to make console work
+ - [ passwd, --unlock, ubuntu ]
+ # A marker saying that initial install is done. Installing all the packages above is VERY slow
+ - [ echo, Initial installation DONE ]
+
+TEXTEND
+
+truncate -s 0 meta-data
+echo 'instance-id: sysrootsrv' >>meta-data
+echo 'local-hostname: sysrootsrv' >>meta-data
+
+genisoimage -output cidata.iso -V cidata -r -J user-data meta-data
+
+# NOTE: direct qemu-system-aarch64 can also be used, however that requires
+# having QEMU_EFI.fd available (and location for that varies). Using
+# virt-install uses extra packages/complexity however also allows
+# the use of virsh for some operations.
+#
+# truncate -s 64m varstore.img
+# truncate -s 64m efi.img
+# dd if=/usr/share/qemu-efi-aarch64/QEMU_EFI.fd of=efi.img conv=notrunc
+#
+# # QEMU console exit is `CTRL + a, x`
+# qemu-system-aarch64 \
+# -m 8192 \
+# -M virt \
+# -cpu max \
+# -nographic \
+# -drive if=pflash,format=raw,file=efi.img,readonly=on \
+# -drive if=pflash,format=raw,file=varstore.img \
+# -drive if=none,file=sysrootsrv.img,id=hd0 \
+# -device virtio-blk-device,drive=hd0 \
+# -cdrom cidata.iso \
+# -netdev user,id=netdev0,hostfwd=tcp::5555-:22 \
+# -device virtio-net-pci,netdev=netdev0 \
+# ; # last line
+
+# EXAMPLE COMMANDS:
+#
+# View generated image:
+# virsh dumpxml sysrootsrv
+#
+# Get a console and login (exit with `CTRL + ]`)
+# virsh console sysrootsrv
+#
+# Delete:
+# virsh destroy sysrootsrv && virsh undefine --nvram sysrootsrv
+virt-install \
+ --name sysrootsrv \
+ --memory 8192 \
+ --arch aarch64 \
+ --vcpus 8 \
+ --import \
+ --disk path=sysrootsrv.img,format=qcow2 \
+ --disk path=cidata.iso,device=cdrom \
+ --os-variant=ubuntu24.04 \
+ --network=user \
+ --xml "xpath.delete=./devices/input[@bus = 'usb']" \
+ --noautoconsole \
+ ;# last-line
+
+# Allow ssh via:
+# ssh ubuntu@localhost -p 5555
+virsh qemu-monitor-command --hmp sysrootsrv 'hostfwd_add ::5555-:22'
+
+echo -e "\e[32mDONE\e[0m"
+echo ""
+echo "To delete the VM use:"
+echo ""
+echo " virsh destroy sysrootsrv && virsh undefine --nvram sysrootsrv"
+echo ""
+echo "VM will auto-install packages via cloud-init. "
+echo "Look out for a message of 'Initial installation DONE' in the console. "
+echo ""
+
+if [ "$start_console" -eq 1 ]; then
+ echo "Delaying console start to have final message visible."
+ echo "Don't worry, vm startup takes a while anyway..."
+ sleep 10
+ echo "Starting console ..."
+ virsh console sysrootsrv
+else
+ echo "Console is not auto-started."
+ echo "To monitor VM startup/status you can open a console using: "
+ echo ""
+ echo " virsh console sysrootsrv"
+ echo ""
+ echo "And you can exit that console using 'CTRL + ]'"
+fi
diff --git a/integrations/docker/images/stage-1/chip-build-crosscompile/upload-cidp-package.sh b/integrations/docker/images/stage-1/chip-build-crosscompile/upload-cidp-package.sh
new file mode 100755
index 0000000..eb668bb
--- /dev/null
+++ b/integrations/docker/images/stage-1/chip-build-crosscompile/upload-cidp-package.sh
@@ -0,0 +1,61 @@
+#!/usr/bin/env bash
+
+# This script is intended to be run post-install (NOTE: install takes a LONG time)
+# of start-sysroot-vm.
+
+set -e
+
+OUTPUT_DIR=ubuntu-24.04-aarch64-sysroot
+PACKAGE=experimental/matter/sysroot/ubuntu-24.04-aarch64
+
+echo "Copying a sysroot in $OUTPUT_DIR"
+mkdir -p "$OUTPUT_DIR"
+
+RSYNC_RSH="ssh -p 5555"
+
+echo "COPYING from VM (ubuntu@localhost on port 5555) ..."
+rsync \
+ --archive \
+ --human-readable \
+ --links \
+ --delete-during \
+ --delete-excluded \
+ --safe-links \
+ --info=progress2 \
+ --info=name0 \
+ --exclude='lib/aarch64-linux-gnu/dri' \
+ --exclude='lib/firmware' \
+ --exclude='lib/git-core' \
+ --exclude='lib/modules' \
+ --exclude='lib/ssl/private' \
+ --exclude='lib/systemd' \
+ ubuntu@localhost:/usr/include \
+ ubuntu@localhost:/usr/lib \
+ "$OUTPUT_DIR/usr"
+
+# Ubuntu 24.04 lib is just a symlink
+rm -f "$OUTPUT_DIR/lib"
+ln -s usr/lib "$OUTPUT_DIR/lib"
+
+echo "Creating cipd definition"
+cat >cipd.yaml <<TEXTEND
+package: $PACKAGE
+description: AArch64 sysroot for cross-compiling matter SDK examples
+install_mode: copy
+data:
+ - dir: "$OUTPUT_DIR"
+TEXTEND
+
+TODAY=$(date '+%Y.%m.%d')
+PACKAGE_VERSION="build-$TODAY"
+
+# NOTE: sufficient permissions are required to be able to upload this.
+# Login is a prerequisite:
+#
+# cipd auth-login
+#
+echo "Uploading package tagged '$PACKAGE_VERSION' to $PACKAGE ..."
+cipd create -pkg-def=cipd.yaml -tag "version:$PACKAGE_VERSION"
+
+echo "Updating version to latest"
+cipd set-ref "$PACKAGE" -version "version:$PACKAGE_VERSION" -ref latest
