ponomarevda / libparams

C library for persistent storing Cyphal/DroneCAN parameters

libparams

libparams is a C-written hardware abstract library that consist of 2 things:

• Parameters Storage - an interface under ROM that allows to store integers and strings with O(1) access complexity.
• Abstract ROM Driver - an interface under your flash memory that allows to write and read sequence of bytes.

It also has an example of flash memory driver implementation for stm32f103 (128 KBytes of flash memory), stm32f103g0 (512 KBytes) based on Stm32CubeIDE HAL and simple Ubuntu flash memory emulation (for SITL usege).

Purpose

The library is intended for real-time embedded applications with small memory such as stm32f103 where persistent storage is required. It is suitable for Cyphal, DroneCAN and other applications as Register interface.

Minimum technical requirements

The codebase is implemented in C99/C11.

The library was tested on stm32f103 (128 Kbytes) and stm32g0 (512 Kbytes). These hardware can be considered as minimum required.

1. DESIGN

The design of the library can be illustrated as shown below:

+-------+----------------+---------------+-------+
|       Parameters (storage.c, storage.h)        |
+-------+----------------+---------------+-------+
                         |
+-------+----------------+---------------+-------+
|       Abstract ROM driver (rom.c, rom.h)       |
+-------+----------------+-----------------------+
        |                |                |
+-------+------+ +-------+------+ +-------+------+
|    STM32f1   | |    STM32g0   | |    Ubuntu    |
| Flash Driver | | Flash Driver | | SITL Driver  |
+--------------+ +--------------+ +--------------+

1.1. High level interface. Parameters

There are 4 types of parameters that we may want to store (they are defined in ParamType_t enum):

• PARAM_TYPE_INTEGER (int32)
• PARAM_TYPE_REAL (float32)
• PARAM_TYPE_BOOLEAN (uint8)
• PARAM_TYPE_STRING (uint8[56])

Each parameter has at least the following fields:

• value
• default value (immutable, that means in can't be changed in real time)
• flags: mutability, required, persistence, visability.

The parameter properties were inspired by the register properties in Cyphal specification. The following properties can be configured by the design of the library:

Property	Meaning
Mutability	Mutability defines the write access. If the parameter is mutable, it can be written by a user for example via Cyphal register interface. Immutable paramters are not allowed to be written by such services, but that doesn't imply that their values are constant (unchanging), because internally the application still can do it.
Required	Required means that the parameter's default value is not essential. Practically, it means that the paramter will not be reset during the paramsResetToDefault().
Persistence (not yet*)	Persistence means that the parameter retains its value permanently across power cycles or any other changes in the state of the server, until it is explicitly overwritten.
Visability (not yet*)	If the parameter is visable, it means it can be accesed from the outside of the application. If the parameter is hidden, it is expected to use it for interal purposes only. The library itself doesn't rely on this property. It is reserved for higher level purposes.

At the moment, all parameters are always persistent and visible. The only property you can configure is mutability and required.

Beside the properties above the Integer and Real parameters have the following additional fields:

• min (const)
• max (const)

A parameter of any type is divided into 2 arrays: *ParamValue_t (actual values) and *Desc_t (auxillary information such as parameter name, default, min and max values) for each parameter type. These arrays expected to be allocated by a user outside the library.

The library allows to have a read/write access to these parameters by their index or name, reset them to default values and other features.

Access to a parameter can be done by index. Since paramters are stored as an array, the access complexity is O(1).

Writing or reading from the external application should be done by name of the parameter. These operations have O(n) complexity, where n - is the total number of the parameters.

Look at libparams/storage.h to get full API and libparams/storage.c for the implementation details.

1.2. Middle level interface. Abstract ROM driver

ROM driver simply allows you to write and read sequence of bytes. Mainly, it consist of 3 operations.

1. Initialization. It is necessary to call romInit() to configure the driver. Storage driver do it automatically.

2. Read operation. You just need to call romRead with corresponded arguments.

3. Write operation. Writing requires to unlock and lock ROM, so it might be done by calling romBeginWrite(), romWrite and romEndWrite() one by one.

Look at rom.h to get full API and rom.c for the implementation details.

1.3. Low level interface. Hardware specific flash driver

Although storage and rom drivers are hardware abstract, they still need a hardware related flash driver. Just for an example, here are a few hardware specific flash driver implementations:

1. STM32F103 (128 Kbytes) flash driver
2. STM32G0B1xx (512 Kbytes flash driver)
3. Ubuntu flash memory emulation (using a file)

For implementation details please refer to the corresponded folder.

New drivers might be added in future.

2. USAGE

It is expected either to add the library into your project as submodule or just copy the required folders. The library doesn't have external dependencies.

You can add pathes to libparams folder, the folder with parameters and to platform specific flash driver. The application examples in tests folder are based on CMakeLists.txt.

If it is SITL mode, you need additionally to specify path to yaml files with parameters values LIBPARAMS_PARAMS_DIR. The flash memory is divided to pages, therefore the base name of file with initial parameters LIBPARAMS_PARAMS_BASE_NAME needs to be specified. The file named <LIBPARAMS_PARAMS_BASE_NAME>_0.yaml will be read as a default parameters values container.

You can create params.c and params.h files with the following content:

// params.h
#pragma once
#include "storage.h"

enum IntParamsIndexes : ParamIndex_t {
    PARAM_NODE_ID,
    PARAM_MAGNETOMETER_ID,
    INTEGER_PARAMS_AMOUNT
};

#define NUM_OF_STR_PARAMS 2
enum class StrParamsIndexes {
    PARAM_SYSTEM_NAME,
    PARAM_MAGNETOMETER_TYPE,
    STRING_PARAMS_AMOUNT
};

// params.c
IntegerDesc_t integer_desc_pool[] = {
    {"uavcan.node.id",      0, 127,     50,     MUTABLE},
    {"uavcan.pub.mag.id",   0, 65535,   65535,  MUTABLE},
};
IntegerParamValue_t integer_values_pool[sizeof(integer_desc_pool) / sizeof(IntegerDesc_t)];

StringDesc_t string_desc_pool[] = {
    {"system.name", "", MUTABLE},
    {"uavcan.pub.mag.type", "uavcan.si.sample.magnetic_field_strength.Vector3", IMMUTABLE},

};
StringParamValue_t string_values_pool[sizeof(string_desc_pool) / sizeof(StringDesc_t)];

Alternatively, you can define your parameters in yaml file and call transpiler script to generate the files above:

# params.yaml
uavcan.node.id:
  type: Integer
  enum: PARAM_NODE_ID
  flags: mutable
  default: 50
  min: 0
  max: 127

system.name:
  type: String
  enum: PARAM_SYSTEM_NAME
  flags: mutable
  default: ""

uavcan.pub.mag:
  type: Port
  data_type: uavcan.si.sample.magnetic_field_strength.Vector3
  enum_base: PARAM_MAGNETOMETER

The initialization of the application can be as shown below:

#include "params.h"

void application_example() {
    paramsInit(IntParamsIndexes::INTEGER_PARAMS_AMOUNT, StrParamsIndexes::STRING_PARAMS_AMOUNT, -1, 1);
    paramsLoad();
}

You can create yaml file to specify initial parameters:

# init_params.yaml
uavcan.node.id: 50

system.name: ""

Please, refer to the libparams/storage.h for the high level usage details because it is self-documented.

3. USAGE EXAMPLES

In tests folder you can find a few examples about how to use the library:

Example	Brief descrtiption	How to try
params_generator/c	An example how to generate C-parameters with python script	make c_generator
params_generator/cpp	An example how to generate C++ parameters with python script	make cpp_generator
platform_specific/stm32f103	Just to test that the library can be build without Warnings and Errors	make stm32f103
platform_specific/stm32g0b1	Just to test that the library can be build without Warnings and Errors	make stm32g0b1
platform_specific/ubuntu	Just to test that the library can be build without Warnings and Errors	make ubuntu

A few real external applications based on this library:

• DroneCAN RL mini v2 example
• Cyphal RL mini v2 example
• Cyphal ubuntu minimal example
• Cyphal ubuntu publisher example
• Cyphal UAV HITL communicator ubuntu

4. CONTRIBUTING

Please, follow the CONTRIBUTING.md guide.

How to run SonarCloud Analysis manually

export SONAR_TOKEN=<...>
~/Downloads/build-wrapper-linux-x86/build-wrapper-linux-x86-64 --out-dir bw-output make coverage
~/Downloads/sonar-scanner-cli-5.0.1.3006-linux/sonar-scanner-5.0.1.3006-linux/bin/sonar-scanner

5. LICENSE

The project is distributed under term of MPL v2.0 license.

6. ACKNOWLEDGEMENTS

This project has been supported by funds from The Foundation for Assistance to Small Innovative Enterprises (FASIE). Moreover, we are honored to be distinguished as laureates of the "Digital Technologies Code" competition, under the federal project "Digital Technologies". We express our profound gratitude for their invaluable support and endorsement.

libparams's Issues

improve ubuntu flash driver emulator

Problem Description

Redundant storage pages feature #33 should be tested in SITL and covered with unit tests first. At the moment ubuntu flash driver emulator can't save parameters and can't handle multiple flash pages.

Current implementation in details

Now stm32 flash memory is emulated with a file in a human-readable format - a simple version of yaml.
Here is an example of valid default.yaml:

uavcan.node.id:         39
pwm.cmd_ttl_ms:         0
pwm.frequency:          50

system.name:            co.rl.mini

Page size is 2048 bytes.
The first part of the file has integer parameters, each integer is 4 bytes long.
At the end there should be strings with 56 bytes size. They are started from the end of the allocated ROM.
All missed parameters are parsed as 0.
This is basically how the parameters are stored in a real stm32 node.
It is enough implementation for the initial emulation purposes and it is easy for a user to manage default parameters at the same time.

Limitations

• while it can load parameters from a file, it was not considered how to save them back
• the driver expects only a single page, so it doesn't ready for redundant pages feature

Suggestion

Let's continue using the same yaml format, but slightly change the logic and make fixes mainly in ubuntu/flash_driver.cpp.

• 1. #35
  Add generate_memory_layout.py (or generate_default_params.py) script. It should generate page_layout.yaml (or default.yaml). As an input it can have either yaml files with parameters or params.cpp file. Let's assume that 4*int_params + 56*str_params < 2048 for a while, so we need only a single page. This step is essential because most of the nodes have outdated default.yaml and we can't continue without updating it.
• 2. Slightly refactor to simplify next steps:
  • Simplify flash driver interface. Instead of flashWriteU64 let's use size_t flashWrite(size_t dest_addr, const uint8_t* src, size_t bytes_to_write)
  • Add a new .cpp and .hpp files to keep all the yaml logic together. It can have a YamlParameters class with 2 methods: read_from_file, write_to_file or similar
• 3. Assuming that page_layout.yaml is always correct, let's update it each time when flashWrite is called.
• 4. Add multiple pages support:
  • Increase flashGetNumberOfPages() from 1 to 2.
  • Extend flashErase and flashWrite to handle multiple pages.
  • Extend generate_memory_layout.py should generate more than 1 page_layout.yaml file to handle multiple pages
  • Extend saving parameters back with multiple pages support

How to write your own yaml file

Hello,

I started writing own yaml file, but I have doubts about some options.

#1 A device node id
#2 Set a system name
#3 Define a publisher is used to report data collected by sensors, using uavcan natural8 type

Questions:
1） Once I want to use a standard data type, I should specify type as "port", right?
2) The Port type will create two data structures, integer and string at the same time. Can I specify default, min, and max for the port type (I set it, but it will not generate these contents)

`uavcan.node.id:
type: Integer
enum: PARAM_NODE_ID
flags: immutable
default: 50
min: 0
max: 127

system.name:
type: String
enum: PARAM_SYSTEM_NAME
flags: immutable
default: "STM32x"

uavcan.pub.det:
type: Port
data_type: uavcan.primitive.scalar.natural8
enum_base: PARAM_DET_SCALAR
flags: mutable
default: 0
min: 0
max: 255`

Some questions about this library

Question #1:
int8_t res;
if (0 == romWrite(0, (uint8_t*)integer_values_pool, INT_VAL_POOL_SIZE)) {
res = LIBPARAMS_UNKNOWN_ERROR;
}
//Why do you use "else if" instead of "if" here? It should always try to save both.
**else if (*0 == romWrite(paramsGetStringMemoryPoolAddress(),
(uint8_t)string_values_pool,
STR_VAL_POOL_SIZE)) {
res = LIBPARAMS_UNKNOWN_ERROR;
} else {
res = LIBPARAMS_OK;
}

Questions #2:
I understand that interger_values_pool and string_values_pool are stored sequentially, so storing string_values_pool should be offset+ INT_VAL_POOL_SIZE, so paramsGetStringMemoryPoolAddress(), I don't understand the intention of this function.

void paramsLoadFromFlash() {
romRead(0, (uint8_t*)integer_values_pool, INT_VAL_POOL_SIZE);
romRead(paramsGetStringMemoryPoolAddress(), (uint8_t*)&string_values_pool, STR_VAL_POOL_SIZE);

for (uint_fast8_t idx = 0; idx < integer_params_amount; idx++) {
    IntegerParamValue_t val = integer_values_pool[idx];
    if (val < integer_desc_pool[idx].min || val > integer_desc_pool[idx].max) {
        integer_values_pool[idx] = integer_desc_pool[idx].def;
    }
}

}

static uint32_t paramsGetStringMemoryPoolAddress() {
return romGetAvailableMemory() - MAX_STRING_LENGTH * string_params_amount;
}

Add redundant storage pages

Problem Description

When saving parameters to the persistent storage, we should initially erase the flash page and then write to it.
There is a chance to lose all the parameters if we turn off the power supply during the paramsSave() call.

Notes:

• The single erase page takes ~20-40 ms for stm32f103

Proxy page approach

We can use proxy (temporary/intermediate) pages:

• During the boot we can check the proxy pages. If they are not erased (not empty), let's try to restore data from them to the main page. Then erase the proxy pages.
• During the paramsSave call we can initially save parameters to the proxy pages. It is expected that the proxy pages arealready erased before this call, so it should be fast.

The approach is simple, but it requires using erase twice for each paramsSave call.

Redundant page approach

Instead of using proxy pages, we can switch main pages after each paramsSave call.

• During the boot we check which pages are erased and which are not. If first group is erased and second is not, let's consider second as main and first as redundant. If both of them are erased or written, let's use any group (it means that we lost the latest params, but at least can restore the previous version)
• During the paramsSave call, we initially save to the redundant group, then erase the main group and switch main to redundant.

This approach is slightly more complex, but it is more efficient.

Notes

• Before testing on a real device, it is better to implement it in SITL and cover with unit tests because the number of erase operations is limited
• Usually we use 1 page for the storage, but for some tasks we need 2 or more pages.
• For compatibility with current API, instead of changing the interface it is better to extend it. For example, this feature can be enabled only if paramsInitRedundantPage(first_idx) is called or similar.