Automation of embedded development

(4 comments)

I am wondering if there is a standard solution to a problem that I am facing. Say you are developing an embedded Debian Linux device. You want to have a "test farm" - a bunch of copies of your target hardware running a lot of tests, while the development is ongoing. For this to work automatically, your automation setup needs to have a way to fully re-flash the device, even if the image previously flashed to it does not boot. How would that be usually achieved?

I'd imagine some sort of option in the initial bootloader that would look at some hardware switch (that your test host could trip programmatically) and if that is set, then boot into a very minimal and very stable "emergency" Linux system, then you could ssh into that, mount the target partitions and rewrite their contents with the new image to be tested.

Are there ready-made solutions that do such a thing? Generically or even just for some specific development boards? Do people solve this problem in a completely different way? Was unable to find any good info online.

Current rating: 5

Comments

Mikko 4 months, 4 weeks ago

Many ARM boards and SoCs have a ROM which probes for USB connection and if found, an emergency kernel and initramfs can be loaded directly to memory for execution. This functionality is often disabled by burning fuses from production variants which then have similar functionality with integrity checks in bootloader which may prohibit bootloader updates but enables kernel and userspace flashing.

Link | Reply

Neil Williams 4 months, 4 weeks ago

If you want a piece of a CI solution, look at LAVA which is available in Debian (jessie and later).
staging.validation.linaro.org or lava.codehelp.co.uk
However, most embedded devices are not capable of safely having the entire firmware/bootloader replaced automatically - the device simply lacks any way to automate recovery from a bad deployment. Even with relays etc., the device typically does not provide useful prompts, is not unique and has no real error handling in any "recovery mode" that may be available. It is a limitation of the hardware. Devices which support some kind of BMC can be automated - that's a lesson that embedded engineers have yet to learn from enterprise systems.
I'm willing to talk about this further if you want, we have a lot of experience of automating a range of embedded devices.

Link | Reply

aigarius 4 months, 4 weeks ago

Looking at LAVA architecture I am interesting in the interaction between the dispatcher on the worker and the device-under-test, specifically how to go about safely deploying a new full-system image to DUT in a repeatable and automated way.

I am also looking for recommendations on what specific hardware (be it Raspberry or something else) capable of running Debian would be easiest for a hobbyist to automate in this way, at least from the hardware perspective. I am assuming that I will be needing an Arduino in addition to the DUT to be able to pull down pins and switch between normal and recovery boot

Link | Reply

aigarius 4 months, 4 weeks ago

So far the best I have found is using Raspberry boot flow ( https://github.com/raspberrypi/documentation/blob/master/hardware/raspberrypi/bootmodes/README.md ) trigerred by pulling down some GPIO pins (using a separate Arduino device) into booting from a recovery image fed by the test host via TFTP.

Link | Reply

New Comment

required

required (not published)

optional

Recent Posts

Archive

2018
2017
2016
2015
2014
2013
2012
2011
2010
2009
2008
2007
2006
2005

Categories

Authors

Feeds

RSS / Atom