Provide a sample project with continuous integration about arduinounit

[wmacevoy]

The build & upload from command line seems very useful, thanks! I'm also happy to report building outside the embedded environment is now supported, so you can do both en vivo (target) and en vitro (dev env) testing with the same set environment, even the same tests.

[ianfixes]

Hello,

For your consideration, I've implemented this "unit tests on CI" by borrowing heavily from your unit testing framework.
It's available as a ruby gem: https://github.com/ifreecarve/arduino_ci

Example of the arduino_ci_remote.rb script running against a small Arduino library can be found in this Travis CI job (lines 757 on; before that are tests against the gem itself): https://travis-ci.org/ifreecarve/arduino_ci/builds/333078476#L757

This is in alpha.

[bxparks]

I have a somewhat strong disagreement with that Don't Run Unit Tests on the Arduino Device or Emulator stackoverflow article.

The issue is not whether the unit test code runs in the embedded device or on a separate environment (e.g. a desktop PC or cloud computer). It's whether the code itself is structured to be testable. If the code is testable, then it really doesn't matter where it runs, and there's a slight advantage to running the test in the target embedded environment to flush out platform specific issues like integer sizes or endianness.

In the example given above, smartLightswitchSerialHandler() is not testable because its external dependencies are not injectable. The external dependencies are: the global Serial instance and the global digitalWrite() method.

The solution should be relatively straightforward:

1. If the smartLightswitchSerialHandler() is part of a class, then replace the explicit references to Serial and digitalWrite() with overridable getSerial() and writePin() methods respectively:

class MyClass {
  ...
  virtual void Stream* getSerial() = 0;
  virtual void writePin(uint8_t pin, uint8_t value) = 0;
  ...
};

void MyClass::smartLightswitchSerialHandler(int pin) {
  Stream* serial = getSerial();
  if (serial->available() > 0) {
    int incomingByte = serial->read();
    int val = incomingByte == '0' ? LOW : HIGH;    // character '0' means 'off', all others 'on'
    serial->print("Ack ");
    writePin(pin, val);
    serial->print(String(pin));
    serial->print(" ");
    serial->print((char)incomingByte);
  }
}

Then in the unit test, stub out the getSerial() and writePin() methods.

2. If the smartLightswitchSerialHandler() method is a global method, then we are forced to do dependency injection directly into the method:

typedef void (*PinWriter)(uint8_t, uint8_t);

void smartLightswitchSerialHandler(int pin, PinWriter pinWriter, Stream* serial) {
  if (serial->available() > 0) {
    int incomingByte = serial->read();
    int val = incomingByte == '0' ? LOW : HIGH;    // character '0' means 'off', all others 'on'
    serial->print("Ack ");
    pinWriter(pin, val);
    serial->print(String(pin));
    serial->print(" ");
    serial->print((char)incomingByte);
  }
}

In the unit test, we would provide the stubbed versions of PinWriter and Stream.

3. If the signature of smartLightswitchSerialHandler() cannot be changed, because it is a callback function, passed as a pointer to something else, then we are forced to use an out-of-band context object, something like this:

class SmartLightswitchContext {
  virtual void Stream* getSerial() { return &Serial; }
  virtual void writePin(uint8_t pin, uint8_t value) { digitalWrite(pin, value); }

  static SmartLightswitchContext* getContext();
};

void smartLightswitchSerialHandler(int pin) {
  Stream* serial = getContext()->getSerial();
  if (serial->available() > 0) {
    int incomingByte = serial->read();
    int val = incomingByte == '0' ? LOW : HIGH;    // character '0' means 'off', all others 'on'
    serial->print("Ack ");
    getContext()->writePin(pin, val);
    serial->print(String(pin));
    serial->print(" ");
    serial->print((char)incomingByte);
  }
}

Then in your unit test, you clobber the default SmartLightswitchContext with your test stubbing subclass of SmartLightswitchContext.

I guess my point is that if a piece of code is structured to be testable, it can run in both the embedded environment directly, or in a separate environment (with suitable Arduino.h stubs). Therefore, there is a huge value to a unit testing framework like ArduinoUnit which runs directly on the embedded environment, because it requires no additional development overhead, just the Arduino IDE and the target embedded environment.

In one of my unit tests, ArduinoUnit (more accurately my rewrite of ArduinoUnit called AUnit, since ArduinoUnit does not compile under ESP8266) caught a bug caused by the difference in integer size between an AVR (sizeof(int) == 2) and ARM/ESP8266 (sizeof(int) == 4). The bug would not have been caught if it had been run in a separate desktop environment with no variation in integer sizes.

[ianfixes]

Disrespect away :) my library's inability to catch problems related to sizeof(int) is a very valid criticism, unless you know of some compiler magic I could take advantage of.

I wrote my library because PlatformIO turned out not to be Free software and I wanted to do unit tests of Arduino libraries on CI. Whether it's the best option for your project in particular isn't my judgement to make.

[wmacevoy]

@ianfixes - there is now a "vitro" example in v3.0 of arduinounit. Your arduino_ci is much better at mocking (and obviously ci). Can you make an example that uses the advanced mocking features you have from _ci and still run AU tests? Notice I have an au2ju script that (hopefully) makes junit versions of the (much more readable) ArduinoUnit output.

[wmacevoy]

This is such a beautiful idea my eyes glistened. If we created that repository with you as a contributor can you populate it with an example?

[scls19fr]

not in a short time frame (unfortunately)

https://github.com/pololu/dual-vnh5019-motor-shield/blob/6604394006b5c2f6a31a0407826771a96ac1782f/.travis.yml
could be a first attempt

I think it should be a good idea to normalize how/where tests files are stored in an Arduino project.

in a tests directory with test_... .ino filename (like many Python projects).

Autodiscovering tests files may also be interesting (see pytest https://docs.pytest.org/en/latest/goodpractices.html )

[ivankravets]

Take a look at our docs and examples:

• http://docs.platformio.org/en/latest/ci/travis.html

[wmacevoy]

Looks like there needs to be an integration of the CI. What is missing from ArduinoUnit that prevents it from being used more directly? I really would like a CI example that worked off of the ArduinoUnit code base. If there is something fundamental missing, then that would be a reason for a new version.

[ianfixes]

A few things are missing from ArduinoUnit that prevent it from being used for CI. (Although, for each of the following, it's possible I'm missing something obvious and would greatly appreciate being corrected.)

ArduinoUnit

• Runs on a physical microcontroller
• Therefore, must be compiled for a microcontroller target (with avr-gcc, which the Arduino IDE wraps)
• Uses headers provided by the Arduino library which contain microcontroller-specific functionality (especially math and String handling)
• Uses external (community-provided) Arduino libraries that are managed (for compilation) by the IDE
• Sends output to serial port because there is no STDOUT
• Must work within the bounds of setup() and loop()
• May need to rely on user input for interrupt-based functions
• Uses an IDE that is assumed to be available on the machine running the tests
• Only has the concept of testing against the one board that you have plugged into it at any given time

ArduinoCI

• Runs on a virtual machine or your PC - not a microcontroller
• Therefore, must be compiled for an x86-64 target (with g++)
• Must use headers that provide mocks for all microcontroller-specific functionality that the Arduino library would have, such that the Arduino library being tested doesn't notice the difference
• Uses external libraries that must be installed on-demand on the remote CI machine, and included in compilation without the aid of the Arduino IDE (since we are not using it to compile the test code)
• Sends output to STDOUT because there is no serial port
• May need to install the Arduino IDE from scratch (e.g. Travis CI does not come with it installed)
• Has no concept of setup() and loop(); you just run the tests
• Has no concept of interrupts; you just call the functions you want in the order you want
• Can (and should) test compilation against each and any supported Arduino platform supported by the Arduino IDE -- a simple command switches the compilation target

[wmacevoy]

Ok, that's a lot. I still say the API should merge; tests should behave the same on the device as off. I still want tests that execute on the device to be part of a continuous integration model.

[ianfixes]

That's your call. I'm of the opinion (shared by others) that unit tests shouldn't run in the target environment, because at that point they cease to be unit tests -- they're tests of both your code and Arduino's execution model / interrupts / environment / etc.

To say it another way, how can I test the following using arduinounit?

// read from serial port, set a pin, write to serial port
void smartLightswitchSerialHandler(int pin) {
  if (Serial.available() > 0) {
    int incomingByte = Serial.read();
    int val = incomingByte == '0' ? LOW : HIGH;    // character '0' means 'off', all others 'on'
    Serial.print("Ack ");
    digitalWrite(pin, val);
    Serial.print(String(pin));
    Serial.print(" ");
    Serial.print((char)incomingByte);
  }
}

In my framework, I did it like this:

unittest(does_nothing_if_no_data)
{
    // configure initial state
    GodmodeState* state = GODMODE();
    int myPin = 3;
    state->serialPort[0].dataIn = "";
    state->serialPort[0].dataOut = "";
    state->digitalPin[myPin] = LOW;

    // execute action
    smartLightswitchSerialHandler(myPin);

    // assess final state
    assertEqual(LOW, state->digitalPin[myPin]);
    assertEqual("", state->serialPort[0].dataIn);
    assertEqual("", state->serialPort[0].dataOut);
}

unittest(two_flips)
{
    GodmodeState* state = GODMODE();
    int myPin = 3;
    state->serialPort[0].dataIn = "10junk";
    state->serialPort[0].dataOut = "";
    state->digitalPin[myPin] = LOW;
    smartLightswitchSerialHandler(myPin);
    assertEqual(HIGH, state->digitalPin[myPin]);
    assertEqual("0junk", state->serialPort[0].dataIn);
    assertEqual("Ack 3 1", state->serialPort[0].dataOut);

    state->serialPort[0].dataOut = "";
    smartLightswitchSerialHandler(myPin);
    assertEqual(LOW, state->digitalPin[myPin]);
    assertEqual("junk", state->serialPort[0].dataIn);
    assertEqual("Ack 3 0", state->serialPort[0].dataOut);
}

[wmacevoy]

I agree with the philosophy that unit tests should run anywhere, but that's a fairytale and tools like ArduinoUnit and AUnit allow for practical tests, even unit tests, on the target hardware. There is value in building code that can be tested in an architecture-agnostic way, but that is forcing a complexity model on code that means most embedded code would not be tested at all.

[ianfixes]

AUnit does not "provide a sample project with continuous integration". arduino_ci does.

You and I agree on structuring classes for testability, but that's not the point of my example. arduino_ci's unit testing capability is not damaged by calls to global functions, and the example demonstrates that -- for better or worse, enabling tests on existing libraries without restructuring them.

[wmacevoy]

Sorry, don't want to disrespect arduino_ci either. I have gotten a lot of complex code to work in an embedded environment by first building and testing in a non-embedded one. Having continuous testing in that environment is great. I just wish it could be done in the embedded one too...

[bxparks]

Hi, Just to be clear, I wasn't making any judgments about arduino_ci. And I wasn't claiming that AUnit provides continuous integration. I was just explaining why I disagree with the StackOverflow article that you referenced which states rather strongly that unit tests should never run on the target embedded environment.

I agree on the usefulness of continuous integration. I've seen it used for 20-25 years. Any serious project must have it. I'm new to the Arduino world though, so forgive my newbie question: Isn't there a scriptable way of compiling Arduino sketches without using the Arduino IDE (Platform IO? It's on my TODO list to look at.) Once the compile and deployment is scriptable, why can't ArduinoUnit be used to write the unit tests, which sends its test results over Serial, and the host computer can validate its output?

I've seen things like this done, for example, a rack of 100-200 Android phones running continuous integration tests which can't run on the Robolectric Android emulator due to hardware dependencies that isn't supported by the emulator. I can imagine a bank of dozens of embedded microcontrollers connected to a USB hub, all running continuous integration tests on something like ArduinoUnit, driven by a host Linux machine.

[wmacevoy]

Actually that is exactly what I imagined building this summer for testing ArduinoUnit. I am a CS professor and we have a room to host such a system. I'm trying to decide if I can multiplex a single system with a usb hub, or just hook one per server. I like the cheapness of the USB option, and the flexibility of the multi-system option.

[bxparks]

I have 4 embedded chips connected to my 4-port USB hub on my Mac running Arduino IDE. They seem to work perfectly fine.

[wmacevoy]

Do you have a scripted multi-target build? I would love to run a test suite across multiple OS/IDE/HW configurations.

[bxparks]

No... I don't know how to build Arduino sketches on the command line. I cycle through them by hand right now. (See comment about me being an Arduino newbie. :-))

[ianfixes]

Documentation of the CLI isn't ranked very high on Google search, for whatever reason. The guide I eventually found was on GitHub:
https://github.com/arduino/Arduino/blob/master/build/shared/manpage.adoc

The command you're looking for is:

$ arduino --verify /path/to/sketch/sketch.ino

This will check compilation. You can also --upload. These operations will pop up a splash screen for whatever reason, which can get to be very annoying. On OSX, you can work around that as follows:

$ java -cp /Applications/Arduino.app/Contents/Java/* \
  -DAPP_DIR=/Applications/Arduino.app/Contents/Java \
  -Dfile.encoding=UTF-8 -Dapple.awt.UIElement=true \
  -Xms128M -Xmx512M processing.app.Base \
  --verify /path/to/sketch/sketch.ino

[ianfixes]

I also think that stackoverflow answer comes off a little too strong. My main takeaway from it was the point that

Unit tests should never test the functionality of factors outside of your control.

And in light of that, the "target environment" of Arduino-in-particular should be avoided, specifically because it doesn't offer you such control. I was thinking of the serial port when I wrote that comment.

[ianfixes]

Can you link me to the example you're talking about? Also, I'm not clear on what my example would be showing.

[wmacevoy]

I hope better access to mocking. Like your serial port & pin controls.

https://github.com/mmurdoch/arduinounit/tree/master/examples/vitro

[ianfixes]

OK, if I understand this right you've created sort of an emulator for what is already instrumented code. Your original implementation instruments the setup/loop functions so that your test macros can function. This vitro code takes it a step further by emulating setup/loop, such that you can compile the sketch on the host machine and run the same kinds of tests in that environment.

Putting arduino_ci mocks on top of that should be straightforward, but since Arduino's setup/loop paradigm lacks a sense of having "completed", I'm not sure how/where it would be appropriate to ask the mock library how many times things were called -- I'm not sure how to reliably assert a state. Also, since the tests must (by design) all run in parallel within the loop, I'm not sure how to prevent a shared "godmode" state (which would mean that changes to one test might break the expectations for the other tests). Am I missing something here?

[wmacevoy]

In the basic mock main, I loop until all tests have completed or a timeout occurs. If your mocking library depends on the time or loop count you could add calls to the state advancement there; or run it in a different thread. The tests can have dependencies (or not if you are using the unit test point of view). The mocking should not care; either they pass/fail as an inter-related (or independent, depending on the designer's intentions) set of tests or not. No?

[ianfixes]

I've given this more thought, and I have to respectfully disagree and decline. I'm not sold on the idea of unit tests that are tied to the model of Arduino sketches.

The dominant effort in arduino_ci was to overcome the compiler (across platforms), and more than a few times I considered whether things would be easier/faster if I simply contributed my edits to one of the already-existing projects. My assessment was that in each of the existing projects I'd be working around a design goal that conflicted with my own. That's still the case here.

I'd be glad to be proven wrong on this, but I feel like this would be non-trivial development effort for an end product that (to me) doesn't provide the right experience for tests. That said, if you decide to port my mocks over yourself, I'd be glad to answer any questions you run into.

I'm sorry not to be more helpful but there is still a lot of work left to do on arduino_ci and a host of my other personal projects.

[wmacevoy]

Thanks for the consideration. I think there is a logical factoring of mocking from the testing framework. Right now there is a "no subset" problem where a mocking library can be used independently from the testing framework. This makes the mocking library everybody's problem instead of a unified one.

[wmacevoy]

I have built some scripts to support the idea of automated testing (independent of what framework might be employed), but it has been hard to decide which CI framework to support first. Travis is compelling, but I don't like the restrictions and connections it has with your repository. Instead I am going with Jenkins as my first CI server, which seems more agnostic to what kind of project you might be developing. Does anyone out there have experience with building Jenkins plugins?

[scls19fr]

No Jenkins experience on my side. Sorry.

[bxparks]

Hi,
I recently wrote a set of scripts (bash and python) and built a CI framework for Arduino boards using a locally hosted Jenkins instance. Details here: https://github.com/bxparks/AUniter

It runs on a Linux box, and supports AVR, ESP8266, and ESP32 (Teensyduino has some bugs which prevents it from working). I didn't need to write any custom Jenkins plugins.

For the component that validates the unit test results, the script currently looks for the output generated by AUnit. But it ought to be straightforward to validate the output of ArduinoUnit (at least v2.2, I haven't looked at the most recent versions of ArduinoUnit). Let me know if you are interested in adding that functionality.