Automation of embedded development


I am wondering if there is a standard solution to a problem that I am facing. Say you are developing an embedded Debian Linux device. You want to have a "test farm" - a bunch of copies of your target hardware running a lot of tests, while the development is ongoing. For this to work automatically, your automation setup needs to have a way to fully re-flash the device, even if the image previously flashed to it does not boot. How would that be usually achieved?

I'd imagine some sort of option in the initial bootloader that would look at some hardware switch (that your test host could trip programmatically) and if that is set, then boot into a very minimal and very stable "emergency" Linux system, then you could ssh into that, mount the target partitions and rewrite their contents with the new image to be tested.

Are there ready-made solutions that do such a thing? Generically or even just for some specific development boards? Do people solve this problem in a completely different way? Was unable to find any good info online.

Currently unrated


Mikko 3 weeks, 6 days ago

Many ARM boards and SoCs have a ROM which probes for USB connection and if found, an emergency kernel and initramfs can be loaded directly to memory for execution. This functionality is often disabled by burning fuses from production variants which then have similar functionality with integrity checks in bootloader which may prohibit bootloader updates but enables kernel and userspace flashing.

Link | Reply

Neil Williams 3 weeks, 6 days ago

If you want a piece of a CI solution, look at LAVA which is available in Debian (jessie and later). or
However, most embedded devices are not capable of safely having the entire firmware/bootloader replaced automatically - the device simply lacks any way to automate recovery from a bad deployment. Even with relays etc., the device typically does not provide useful prompts, is not unique and has no real error handling in any "recovery mode" that may be available. It is a limitation of the hardware. Devices which support some kind of BMC can be automated - that's a lesson that embedded engineers have yet to learn from enterprise systems.
I'm willing to talk about this further if you want, we have a lot of experience of automating a range of embedded devices.

Link | Reply

aigarius 3 weeks, 6 days ago

Looking at LAVA architecture I am interesting in the interaction between the dispatcher on the worker and the device-under-test, specifically how to go about safely deploying a new full-system image to DUT in a repeatable and automated way.

I am also looking for recommendations on what specific hardware (be it Raspberry or something else) capable of running Debian would be easiest for a hobbyist to automate in this way, at least from the hardware perspective. I am assuming that I will be needing an Arduino in addition to the DUT to be able to pull down pins and switch between normal and recovery boot

Link | Reply

aigarius 3 weeks, 6 days ago

So far the best I have found is using Raspberry boot flow ( ) trigerred by pulling down some GPIO pins (using a separate Arduino device) into booting from a recovery image fed by the test host via TFTP.

Link | Reply

Raymond Burkholder 3 weeks, 2 days ago

Depending upon how isolated your device is, some things to consider:

* use pxeboot to boot an installer from elsewhere, and with Debian, you can then use preseed post-install rules to build the final image

* use overlayfs to create a locked-in copy and install a test image over-top for testing, after testing, release the over-top and return to the the under-neath.

Link | Reply

New Comment


required (not published)


Recent Posts






RSS / Atom