blackbirdhq / defmt-persist Goto Github PK

Currently the COBS encoding decoding is done "manually", but should be possible to get working as a postcard flavor

This should be updated to use https://crates.io/crates/defmt

The general documentation state of the crate is rather low:

• Readme
• Docs.rs
• Cargo.toml
• Examples
• Repository description

I think there is a slight improvement potential in changing from regular cobs to a reverse zero-compressing COBS encoding (https://github.com/Dirbaio/rzcobs)

Main issue currently is the missing ability to specify sentinel character in the current rzCOBS implementation

Driver modules needing access to storage driver:

• MQTT Persistence layer (buffering) [R/W]
• Global logger (This repo) [R/W]
• Wifi HTML Captive Portal [R]

currently the framing will not work for long log statements.

Blocked by: knurling-rs/defmt#185

It would be nice to be able to configure defmt-persist to be either the n first log frames, or the n latest log frames, where n depends on the storage available.

Hello @MathiasKoch! Thanks for this crate, I find it useful and this is exactly what I had in mind for my project for quite some time!

I'm currently trying to plug this into my project that uses an STM32 MCU that has a typical FLASH controller that requires the page to be erased to be written to.

From my understanding, this requirement poses a challenge, that it isn't possible to periodically call drain_storage to flush records into the FLASH as it will fail on consequent calls due to the page with the header not being erased.

Currently, I can imagine some solutions to this:

1. Keep the header in a separate FLASH page and erase it every time the header needs to be updated. This solution is pretty lame and wasteful of the FLASH resource but could work in the short term, or
2. Do not use the header and rely on the fact that "empty" data in FLASH are always 0xFFs. Probably, the log records could have some magic number at the beginning (and/or the end) and during the write, we will skim through the log area to find an empty spot that is only 0xFFs and doesn't have that magic number at the beginning, or
3. Use some more advanced data structure that is more suitable for a FLASH memory, probably similar to those used in filesystems

Hope it makes sense. What are your thoughts on this?

It could be useful to run the firmware with probe-run, see some logs there that it's running OK, and then remove the probe or deploy the device in the field while still be able to read logs, but later and from storage. Both achieved by the same firmware.

This would have to happen behind an opt-in feature flag.

This is our company RFC for the feature, copied here to give people an insight into the thoughts and usecase behind this crate.

Motivation

Currently all logging have been changed to use defmt (https://defmt.ferrous-systems.com/), which makes for a fast debug logging framework.

We should be able to use the two facts about defmt that it is transfer-agnostic and small sized on device to implement a global logger that writes to our external flash instead of RTT.

Background knowledge

NOTE This is slightly outdated, as the format of defmt has been changed to use tags and JSON, but for this project, the description is valid nevertheless.

The way defmtworks is:
Say we have a log statement defmt::info!("This would be awesome to log, but contains a lot of characters that is expensive to include in my binary! This is a variable: {:?}", 5), it would store the string This would be awesome to log, but contains a lot of characters that is expensive to include in my binary! This is a variable: {:?} in a debug section of the ELF, that would not be flashed onto the device, at a known index (in this example index 1).

The log statement on the device would then be changed to send

let bytes = [
    1,  // index
    2,  // timestamp
    5,  // argument
];

through whatever global logger transport is enabled on the device (eg. defmt-rtt)

The timestamp is generated from a function on the device, and could be an actual timestamp (eg. UTC), or just an incrementing number (eg. 1, 2, 3).

A decoder would then be needed to decode these bytes, based on the original ELF containing the aforementioned debug section.

This means that the logging itself is completely transport agnostic (could be serial based, network based, stored and read or whatever. As long as the ELF on the device matches the one used by the decoder)

Proposed Solution

The proposed solution involves writing a defmt-persist crate, that initializes a global logger to write to a given flash area.
The flash area should have an MQTT based API to read the log messages, eg. write to <DeviceType>/log/get/<UUID> and the device will start publishing log statements to <DeviceType>/log/<UUID> or something similar.
Furthermore we need some tooling to read logs and decode them based on the matching elf file from S3.

Implementation Steps

Changes needed to implement the feature:

• Implement defmt-persist. Polished version would be able to rotate logs, so it wouldn't run out of memory. See defmt-rtt and defmt-semihosting for transport layers (https://github.com/knurling-rs/defmt/tree/main/firmware)
• Implement MQTT based device API to read and clear logs. Gold version would be able to filter based on log level and timestamps
• Implement tooling to read and decode messages (https://github.com/knurling-rs/defmt/tree/main/decoder). See (https://github.com/eupn/defmt-print) for a host-side example on how this can be used.

Follow-up task would be to have devices send the full log to MQTT in case of a panic, with a lambda/IoT rule to log it to cloudwatch automatically, so we always persist logs leading up to a panic

Considerations around storage format?

• Perhaps just zero-terminate every message?

Document the overhead of

• CRC
• COBS

in terms of storage size & wire size.

No reason to do extra heavy-lifting when it is already done by https://github.com/jamesmunns/defmt-bbq