We should be able to do this with GitHub Actions via Self-Hosted Runners: https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners/about-self-hosted-runners.

I'd say that the biggest requirement here is, maintainers should be able to choose which devices on the device farm that the code is tested on via some mechanism such as applying labels to the PR. This should prevent the huge amount of device farm load which would happen if we just tested every PR on every device.

Additionally, the actions system needs to enforce, somehow, that the correct jobs go to the correct device farm machines. For example, if I tag my job with NUCLEO_L4, it needs to go to a device farm box that has a STM32L4 board attached and ready to run jobs.

The most naive solution would be to create a self-hosted runner for each device attached to each device farm machine, and tag the runner with a label matching one of the devices attached to it. Then, the job would specify that it can only run on machines which that Mbed device attached.

This is nice because it means we can get parallelism by adding multiple device farmers with the same boards attached, but it does mean that in a worst case scenario every device could be building at once on a machine. I worry that that could cause it to run out of RAM, even if each job uses ninja -j1