sblantipodi / arduino_bootstrapper

Utility classes for bootstrapping Arduino projects with Wifi management, OTA upload management, memory management, MQTT and queue management. (ESP8266/ESP32 ready)

Arduino Bootstrapper

Utility classes for bootstrapping Arduino projects.
In few minutes you will be able to read and write to storage (in JSON format), read and send messages to an MQTT queue, connect to WiFi without hardcoding passwords and more...
Arduino Bootstrapper wants to be a starting point for creating good projects without code duplications.

Espressif ESP8266/ESP32 are the default platforms but you can easily add support for other platforms and boards.
Written for Arduino IDE and PlatformIO.

If you like Arduino Bootstrapper, give it a star, or fork it and contribute!

Credits

• Davide Perini

How to get help if you are stuck?

Open an issue here on Github, your questions could be useful to other users.

How To

There is two way to bootstrap your software using this utilities.

1. Bootstrap a project from scratch
2. Import those utilities to your existing project

There are other projects that uses this utility, you can explore their sources here:

Smart Thermostat, Solar Station, Glow Worm Luciferin, Smart Watch Winder

1) Bootstrap a project from scratch

Clone the bootstrapper

git clone [email protected]:sblantipodi/arduino_bootstrapper.git

Simply edit those file as per description, few minutes required

.
├── ...
├── examples                 # Project src folder
│   ├── ChangeName.cpp       # Main file with loop() and setup() function, rename and copy it into your src folder  
│   ├── ChangeName.h         # Main header file, rename and copy it into your include folder  
│   └── ...      
├── src                      # Folder for core files, edit those files and contribute!
│   ├── BootstrapManager.h   # Core header file with utility classes for bootstrapping
│   ├── QueueManager.h       # Core header file with utility classes for Queue and MQTT management
│   ├── WifiManager.h        # Core header file with utility classes for Wifi and OTA upload management
│   ├── Helpers.h            # Core header file with helper classes 
│   └── ...       
├── platformio.ini           # Configure all the required info (ex: Wifi device name, DNS gateway, ecc.)
├── secrets.ini.template     # Configure password and rename the file in secrets.ini 
└── ...

NOTE:
You should implement those functions that are passed by *pointer to the main bootstrap functions:

class BootstrapManager {
  ...
  public:
    void bootstrapSetup(
        void (*manageDisconnectionFunction)(), 
        void (*manageHardwareButton)(), 
        void (*callback)(char*, byte*, unsigned int)
    );
    void bootstrapLoop(
        void (*manageDisconnectionFunction)(), 
        void (*manageQueueSubscription)(), 
        void (*manageHardwareButton)()
    );  
    ...
};

• manageDisconnections() # OPTIONAL put the logic you need in case your microcontroller is disconnected from the network
• manageHardwareButton() # OPTIONAL put special instruction for hardware button management during network disconnections
• manageQueueSubscription() # subscribe to the desired mqtt topics
• callback() # callback function called when a message arrives from the queue

2) Import those utilities to your existing project

You can import Arduino Bootstrapper into your existing projects in two way:

1. Import via public registries (easyest way to import)

// For PlatformIO
Add `lib_deps` to your `platformio.ini`
lib_deps = ArduinoBootstrapper

// For ArduinoIDE
Simply import the Bootstrapper library from the library manager

2. Import via git submodules (faster updates to latest releases)

git submodule add https://github.com/sblantipodi/arduino_bootstrapper.git arduino_bootstrapper

For both importing method you should then add extra dirs to your platformio.ini

lib_extra_dirs = arduino_bootstrapper

Copy and configure ~/arduino_bootstrapper/secrets.ini.template into secrets.ini.template

Please include BootrapManager.h into your main header file:

#include "BootstrapManager.h"

and initialize the BootstrapManager class:

BootstrapManager bootstrapManager;

In your setup() function add the Wifi, MQTT and OTA bootstrapper

bootstrapManager.bootstrapSetup(manageDisconnections, manageHardwareButton, callback);

In your loop() function add the bootstrap manager function

bootstrapManager.bootstrapLoop(manageDisconnections, manageQueueSubscription, manageHardwareButton);

Please follow the Bootstrap a project from scratch instructions without the initial git clone part.

Enable symlinks in GIT for Windows

This project uses symlinks, Windows does not enable symlinks by default, to enable it, run this cmd from an admin console:

export MSYS=winsymlinks:nativestrict

Continuous Integrations

This project supports CI via GitHub Actions. In the .github/workflows folder there are two workflows

• one for automatic release that is triggered when a git tag is pushed
• one for building the project and creating the artifact (binary firmware)

If you use this syntax:

git tag -a v1.0.0 -m "your commit msg";
git push origin --tags;

text in the commit message will be the description of your release.

Access Point frontend

Arduino Bootstrapper will search for a secrets.ini, if you don't configure it, access point is started.
You can connect to the AP with your mobile and go to http://192.168.4.1 to access the gui
that will let you enter all the passwords without the needs of hardcoding them.

License

This program is licensed under MIT License

Thanks To

Thanks	For
	For the CLion IDE licenses.

arduino_bootstrapper's Issues

Web server for OTA configuration

Hello,
When the Web server has been activated once, it is never possible to activate it again for any modification, excepted by flashing again the device.
Then, on my local version of bootstrapper library, I have added a boolean parameter "forceWebServer" that force the OTA configuration when set.
I use this feature to activate OTA configuration when the device is switched ON/OFF twice quickly.
I use a json file to store power ON information that allows specific ON/OFF sequence detection.
If you feel this could be useful for a future version of your library, I can provide additional information on the code.
Also, it would be useful to add the gateway IP in the config.json file.
I was not able at the beginning to set a static IP address for the controller device because the gateway IP was not set correctly.
Regards.
Patrick

BootstrapManager::readSPIFFS does not work correctly when called several times

Hi Davide,

I am using writeToSPIFFS and readSPIFFS to manage a json file for context saving (containing information such as total running time).
Initial reading at startup was working fine, as it was for the setup file setup.json.
Then during testing I have noticed that the repeated reading of the context file every 10 mins for debugging purposes was returning a json doc containing null information.

You have raised a similar issue in 2020 for the ArduinoJson library but this issue has been closed with the following comment:

I am trying to read/write SPIFFS in the ESP8266, it seems that closing the file before returning the DynamicJsonDocument solved the problem when reading.

After reviewing your code, I noticed that for the specific code for the ESP32, the file.close statement was missing as well as the initial declaration DynamicJsonDocument jsonDoc(1024);

However, adding these 2 statements did not fix the issue.
After additional research, I have found this issue that looks similar.

The final comment for this issue is:

bblanchon commented on 21 Jul 2018
By passing a char[] to deserializeJson(), you triggered the zero-copy mode of ArduinoJson.
In this mode, ArduinoJson stores pointers to the input buffer instead of making copies.
To fix this program, you need to disable the zero-copy mode by passing a read-only input; to do that, simply cast the input to a const char*: DeserializationError error = deserializeJson(_InputDoc, (const char*)json);
``

I then applied this solution to the ESP32 code:

DeserializationError deserializeError = deserializeJson(jsonDoc, (const char*)buf.get());

and now the function is working as expected.

Regards
Patrick

BootstrapManager::readSPIFFS: missing jsonDoc initialization and configFile.close

Hello,
Instruction DynamicJsonDocument jsonDoc(1024); and configFile.close();are missing in readSPIFFS but are part of readLittleFS function.
This was mentioned in issue #12 but not identified as the root cause of this issue.
During testing of the release 1.12.6 with an ESP32, reading a SPIFFS file leads in one occasion to deserialization errors and these missing instructions might be the cause.
These missing instructions should preferably be included in a future version.

Extract of locally modified code:

// read json file from storage
#if defined(ESP32)
DynamicJsonDocument BootstrapManager::readSPIFFS(String filename)
{

  // Helpers classes
  Helpers helper;
  DynamicJsonDocument jsonDoc(1024); // added by Pronoe on 02/03/2022

and

        DeserializationError deserializeError = deserializeJson(jsonDoc, (const char *)buf.get());
        Serial.println("\nReading " + filename);
        if (filename != "setup.json")
          serializeJsonPretty(jsonDoc, Serial);
        
        configFile.close();   // added by Pronoe on 02/03/2022
        if (!deserializeError)
        {
          helper.smartPrintln(F("JSON parsed"));
        }

Suspected bug with writing setup file with ESP32

Hellow,
in WiFiManager.cpp, when writing the setup file, in the case of ESP32, the current code is:
File configFile = SPIFFS.open("/setup.json", "w");
if (!configFile)
{
Serial.println("Failed to open [setup.json] file for writing");
}
serializeJsonPretty(doc, Serial);
serializeJson(doc, configFile);
configFile.close();
Serial.println("[setup.json] written correctly");
The else block that exists for the ESP8296 is missing.
The code should be:
File configFile = SPIFFS.open("/setup.json", "w");
if (!configFile)
{
Serial.println("Failed to open [setup.json] file for writing");
}
else
{
serializeJsonPretty(doc, Serial);
serializeJson(doc, configFile);
configFile.close();
Serial.println("[setup.json] written correctly");

Is this correct ?
The same apply for BootstrapManager::writeToSPIFFS

PingESP utility issue

Hello,
Thank you for your bootstrapper library.
I am using it to build my project of pool filtering pump management based on an ESP32.
I use also some pieces of code from your Smartostat.
I found an issue with PingESP utility because it does not seem to be ESP32 ready.
After including ESP32 definition, the compiler goes on but I still have an issue with the "extern "C" {#include <ping.h>}" declaration.
As I understand, this call for a pre-compiled ping C code.
Thank you for your help.
Patrick

Justification of the SPIFFS.format() instruction when initializing WiFi credentials with OTA web server ?

Hello David,
in the case of WiFi credentials and other parameters being initialized by Web server, before writing these parameters in the setup.json file, you have an instruction SPIFFS.format() in the case of the ESP32 and nothing similar for the ESP8266 (see line 528 in WifiManager.cpp).
What is the aim of this instruction and why isn't it implemented for the ESP8266 ?
Also this format instruction is a bit annoying for me, because it erases my other configuration files when I want to change credentials through Web .
I would prefer a solution with a dedicated BootstrapManager.eraseFileSystem() instruction that could be called if required by the application code rather than automatic formatting.
However to avoid any change to existing code, a setup option parameter extern bool autoEraseFileSystem could be added in Helpers.h with its value set by default to true in Helpers.cpp (in the same way as extern bool fastDisconnectionManagement.
I have already added in this way a parameter extern bool forceWebServer to manage activation on demand of the Web server as a response to issue #9.
I will provide soon after some testing, a pull request with these modifications, if you agree.

DNS configuration issue with ESP32 platform

Hello,
in the WifiManager::setupWiFi routine, there is a call to WiFi.config(microcontrollerIP, IP_gateway, IP_DNS) for both ESP8266 and ESP32.
However, in the time.h library associated to the ESP32, the correct parameters are :
WiFi.config(microcontrollerIP, IP_gateway, Subnet_mask, IP_DNS1, IP_DNS2), with IP_DNS1 & 2 optionnal.
This result to the DNS IP being interpreted as a subnet mask value.
Associated with the use of a static IP for the microcontroller, this prevents access with domain name such as NTP servers for example.
After implementing the correct parameters list, all is working fine.
I think the time.h library is slightly different for the ESP8266 but I cannot check this, because this platform is not installed on my system.
Regards.
Patrick

Always running webserver for Improv branch

I'm experimenting with the improv branch of this library, and it seems to be almost perfect for what I need it to do. The only thing missing is that the webserver which allows the user to configure MQTT (and other params) isn't always running, therefore the "Visit Device" button in the Improv web UI doesn't work. It also doesn't pre populate the saved values in the webform - I assume because the device config was originally intended to be one time use only.

What are your thoughts on this? Do you plan to continue the improv development and include it in the main branch? Do you want me to submit a pull request that enables the web server when improv is enabled?

Either way, nice work. This is library is going to work really well for my project.

AP page shown on mobile is different from docu

Mobile gives a lot more entries like IP. What to input there for DHCP?

[Feature-Request]Add WiFi list to improv

Thanks very much for creating this project. I have not found any others that have implemented the improv function in an easy to use frame work for creating your own projects. In your WiFi manager code , you have the function to generate a list of available WiFi networks. Would it be possible to incorporate this into the improv function in a similar way to this?
RPC Command: Request scanned Wi-Fi networks
I think this would be the equivilent from tasmota

Bad result from BootstrapManager::readValueFromFile for float values

In the frame of my functional tests for the branch https://github.com/Pronoe/arduino_bootstrapper/tree/fix-missing-init-and-file-close-ESP32 I have identified an unattended result from the BootstrapManager::readValueFromFile for float value.
The test json file is:

{
  "STRING_VALUE": "test readValueFromFile",
  "INT_VALUE": 50,
  "FLOAT_VALUE": 200.3
}

The read file is the same and is correct.
Test result is:

intValue: 50 read value: 50 OK
stringValue: test readValueFromFile read value: test readValueFromFile OK
floatValue: 200.30 read value: 200 KO

The involved code in BootstrapManager::readValueFromFile is:

JsonVariant answer = jsonDoc[paramName];
        if (answer.is<char*>()) {
          returnStr = answer.as<String>();
        } else {
          auto returnVal = answer.as<int>();
          returnStr = String(returnVal);
        }

and does not provide the processing of float values.
Actually, this is not a showstopper for my own application because I don't use this function but only readSPIFFS but it could be for some users.

uMQTT library

I think this is perfect bootstrap library for ESP projects. Is it possible to include uMQTT or TinyMqtt broker linked below ?

https://github.com/martin-ger/uMQTTBroker

or

https://github.com/hsaturn/TinyMqtt

Thanks.

Long blocking delays inside readSPIFFS and readLittleFS functions

Hi Davide,
sorry for opening again an issue, I have another question.
In BootstrapManager::readSPIFFS you have a delay of 4 seconds at the end of the function and in BootstrapManager::readLittleFS you have a delay of 2 seconds.
What is the justification of such long delays ?
Probably this is to give time to your smart display to refresh, but if this is the case, you should better displace this delay inside the Helpers::smartDisplay() function.

At the moment, the main loop of my code reads a context file every 10 minutes, and this 4 s delay blocks any other operation during 4 s . For example, the blinking of the built-in Led is stopped during 4 s every 10 minutes.
This is not really serious for my project but this could be an issue for some other time critical applications for other users.

Wrong json document returned by BootstrapManager::parseQueueMsg when the payload is a numerical value

Hello,

for simple MQTT messages with a numerical value such as: Topic: SET/TEMP Payload: 21.5, the parseQueueMsg function returns a json document which contains only the string 21.5 and not as expected {"value":21.5}.
I have tested different MQTT messages:
Topic: CMD Payload: ON returns {"value":"ON"} - correct
Topic: SET Payload: {"TEMP":21.5} returns {"TEMP":21.5} - correct
Topic: SET Payload: 21.5 returns 21.5 - failed
Topic: SET Payload: t1t1 returns true - failed, and I noticed that deserializeJson function was returning no error
This strange behavior is coming from the deserializeJson function which returns no error when the payload is a numeric value or a string such as 'toto', 't1t1', 'true', 'false' ... but delivers a json object not formatted as expected.
So I have modified the code as below (see modifications in 3rd line):

 DeserializationError error = deserializeJson(jsonDoc, (const byte *)payload, length);
  // non json msg
  if (error || ((char)payload[0] != '{'))   // instead of if(error) - tests if the payload looks like a json structure
  {
    JsonObject root = jsonDoc.to<JsonObject>();
    root[VALUE] = message;
    if (DEBUG_QUEUE_MSG)
    {
      String msg = root[VALUE];
      Serial.println(msg);
    }
    return jsonDoc;
  }
  else
  { // return json doc
    if (DEBUG_QUEUE_MSG)
    {
      serializeJsonPretty(jsonDoc, Serial);
      Serial.println();
    }
    return jsonDoc;
  }

Then for the example of Topic: SET Payload: 21.5 returns 21.5, I get the result {"value":21.5} as expected.
Then I suggest to implement this modifications in a future version.
This apply also for the BootstrapManager::parseHttpMsg function.

How to set specific constant overriding those of the Configuration.h file

Let's take as an example, the gateway IP.
In the Configuration.h file of the library, one can find:
// GATEWAY IP
#ifndef GATEWAY_IP
#define GATEWAY_IP "192.168.1.1"
#endif
const char* const IP_GATEWAY = GATEWAY_IP;
As I don't want to modify the original library files, I have included my own file MyConfiguration.h which contains:
#define GATEWAY_IP "192.168.0.254"
Thanks to the #ifndef GATEWAY_IP directive, I expected that my own definition would be taken into account.
But this is not the case, probably due to the fact that the library Bootsrapper is compiled before my sketch.
So at the moment, I had to modify the original file inside the library folder.
What do you suggest to solve for this problem ?

Examples needs an ino file

The examples folder has a cpp and .h file, but there is no ino file.

Memory footprint?

I've used this library together with the serial example for ESP32 project (only serial communication). Before adding it sketch size was 257kB, now it is 968kB. I am not sure the memory left is enough for the devices I want to add.

1.13.0 ESP8266 compile error: PingESP.cpp:84:5: error: 'esp_schedule' was not declared in this scope

I have not yet rooted out the source of this problem, but I just started a new project in Platformio and pulled the latest version 1.13.0 of bootstrapper in from the library manager.

When compiling, esp_schedule can't be found:

.pio/libdeps/d1_mini_lite/ArduinoBootstrapper/src/PingESP.cpp: In static member function 'static void PingESP::receivePingCallback(void*, void*)':
.pio/libdeps/d1_mini_lite/ArduinoBootstrapper/src/PingESP.cpp:84:5: error: 'esp_schedule' was not declared in this scope
   84 |     esp_schedule();
      |     ^~~~~~~~~~~~

Rolling back to version 1.12.12 works fine.

I see that esp_schedule() is still called by PingESP in the older version of Bootstrapper.

Superfluous display reference

Hi there,
nice project. But could you please remove the hardcoded reference to the display?

• LED_BUILTIN: undeclared identifier used in nonblokingBlink

Name conflict with other libraries for const String ERROR

Bootstrapper library uses a const STRING ERROR which is a very common name (définitions in Helpers.h, set value in Helspers.cpp, used in BootstrapManager.cpp and may be in the application code).
I am using also the ezTime library which use the same name ERROR in an enum list.
The compiler raises an error related to the reuse of an existing name, and this, whatever is the order of the #include directives of the 2 libraries.
I solved provisionally this issue by renaming ERROR in the Bootstrapper library but this make testing of the committed modifications more difficult.
I have now taken the option to modify the ERROR name in the ezTime library.
In both case, this is not clean.
Could you recommend a better solution as using namespace or anything else ?