Add unit testing framework and skeleton test project about infinitime

[JF002]

Thanks for this suggestion @joelghill ! Adding unit tests is definitely a good idea and will help improving the general quality of the code.

However, I think adding unit test in the current code base won't be easy : most of the modules and classes depend on a code that is related to the hardware : device drivers, NimBLE (BLE stack), FreeRTOS,...
If you want to write "true" unit tests (that do not depend on anything else than the class that you want to test), you'll have to stub/mock most of the low level code. That's not impossible, since we've already done that for InfiniSim, but it's not easy either.

I'm currently experimenting with a new architecture for the project, that split the code in 2 parts : the "core" which contains everything that makes InfiniTime : SystemTask, DisplayApp, user applications, settings,... and the "ports", which implement the hardware abstraction. This is not ready yet, but it looks promising to me ;-)
With this new structure, the code in "core" will be very easily unit testable, since it does not depend on any hardware detail, it only depends on "interfaces" (C++ concepts). And we'll be able to run those unit tests on anything (the target hardware or a desktop computer, for example).

You can have a look at this proof of concept here : https://codeberg.org/JF002/InfiniTime-multi-target

[NeroBurner]

I personally prefer Catch2 as it has much nicer syntax.

Do you have some thoughts about testing strategy? What should be tested? What shouldn't?

What do you mean with a skeleton project? Could you explain further what you mean and what it should accomplish?

[joelghill]

I personally prefer Catch2 as it has much nicer syntax.

I'm very open to suggestions on the framework. Google Test appears to have a nice mocking framework which would be really nice, but I know close to nothing about Catch2 so I will defer to people with more experience than me.

Do you have some thoughts about testing strategy? What should be tested? What shouldn't?

In my mind, and in a perfect world, a project would have >80% coverage, and that coverage would be "useful". Unit testing doesn't help when people add nonsense tests that pass and cover code for the sake of simply improving the coverage metric.

For this project though, I think the goal of unit testing should help developers debug and verify their code works without having to connect to a device or run code in a simulator. I would not suggest we enforce coverage metrics, but rather put something in place that is quick and easy to use and is a helpful tool, not a time consuming burden. It would also be helpful to be able to create tests that are able to reproduce bugs. If users report add behavior we could debug with a device to identify troublesome code and then write focused tests that reproduce that edge case and fail state.

As for what specifically should be tested, the BLE controllers, services, and managers all look like good candidates for tests if we are able to mock their dependencies. We could verify how they interpret event payloads, that they properly update whatever internal states they have, and that they make correct calls to other classes/functions. I'm thinking of tacking the Apple Notification Service implementation and unit testing the implementation of the specification would be really helpful in development.

What do you mean with a skeleton project? Could you explain further what you mean and what it should accomplish?

I'm imagining a mostly empty project under a root tests project that would mirror the source code directories. For the scope of this issue I would want to supply enough boiler-plate code and documentation to make it easy for other developers to jump in and add tests easily. I'm open to suggestions here too on how much code and documentation would be appropriate.

[joelghill]

Thanks for this suggestion @joelghill ! Adding unit tests is definitely a good idea and will help improving the general quality of the code.

However, I think adding unit test in the current code base won't be easy : most of the modules and classes depend on a code that is related to the hardware : device drivers, NimBLE (BLE stack), FreeRTOS,... If you want to write "true" unit tests (that do not depend on anything else than the class that you want to test), you'll have to stub/mock most of the low level code. That's not impossible, since we've already done that for InfiniSim, but it's not easy either.

I'm currently experimenting with a new architecture for the project, that split the code in 2 parts : the "core" which contains everything that makes InfiniTime : SystemTask, DisplayApp, user applications, settings,... and the "ports", which implement the hardware abstraction. This is not ready yet, but it looks promising to me ;-) With this new structure, the code in "core" will be very easily unit testable, since it does not depend on any hardware detail, it only depends on "interfaces" (C++ concepts). And we'll be able to run those unit tests on anything (the target hardware or a desktop computer, for example).

You can have a look at this proof of concept here : https://codeberg.org/JF002/InfiniTime-multi-target

The new architecture sounds great and I love the idea of moving the source code in that direction!

In your opinion, is it worth starting unit testing now given that there is a major refactor on the way? I lean towards moving ahead with some testing anyway. It would not be perfect, but some tests are better than none and it would give contributors time to form opinions about the testing framework. It's also just a nice development tool to have available.