FR: Telnet/Serial console about esp8266-react

[rjwats]

Hey Proddy, Interesting requirement, and certainly not an easy thing to implement elegantly. Some variation of the event or registry pattern (perhaps hierarchical) is the way I try to go in terms of connecting things up. For settings service in particular you could implement a "settings bus". Where interested classes could register accessor and mutator callbacks against their group id "wifi" "ap" etc. Mutators functions would have to accept string or char* and have their return code describe whether: - The setting did not exist - The provided setting value was unparseable or invalid - The setting was applied successfully Accessors could take a pointer or possibly return a string: I'm not sure if you'd need to use the a return code, but you might want to differentiate between: - The setting did not exist - The setting had no value - The setting had a particular value The "settings bus", if done well, could be re-used for anything that wants to modify settings - MQTT for example could hook into it. *Sidenote*: I'm thinking it's time for some singletons in this project, I was trying to avoid them for the sake of purity but the constructors are getting rather argument heavy and adding in

…

On Tue, Dec 3, 2019 at 2:48 PM Proddy ***@***.***> wrote: (from the list in #57 <#57>) I've started implemented a multi-user Serial and Telnet console where using a Serial monitor to the USB or Telnet client to esp8266-react with the plan where you can issue commands to set parameters and monitor traffic. But got stuck finding an optimal way to read and set each of the Services parameters that are stored in the individual JSON files. For example I would like 'set wifi ssid ' to set the SSID. Also in other services I'd like to be able to read certain system settings, e.g. hostname. Can you think of any idea to achieve this? Now I've mangled up my code into esp8266-react's and took over SettingsPersistence to store all settings centrally but its messy. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#66?email_source=notifications&email_token=AAKE4VBL7DQ3T6EHMPRTKVDQWZWUJA5CNFSM4JUZHW42YY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4H5WILFQ>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAKE4VER4NS5AYIK5SBAFH3QWZWUJANCNFSM4JUZHW4Q> .

[rjwats]

Sorry I've been quiet... busy Sunday and weeks are always busy for me. I'll try and find time to push up the formatting changes and baseline clang file tonight. So you can open up a PR for some collaboration - early feedback is the best way to go :)

…

On Tue, Dec 3, 2019 at 4:20 PM Rick Watson ***@***.***> wrote: Hey Proddy, Interesting requirement, and certainly not an easy thing to implement elegantly. Some variation of the event or registry pattern (perhaps hierarchical) is the way I try to go in terms of connecting things up. For settings service in particular you could implement a "settings bus". Where interested classes could register accessor and mutator callbacks against their group id "wifi" "ap" etc. Mutators functions would have to accept string or char* and have their return code describe whether: - The setting did not exist - The provided setting value was unparseable or invalid - The setting was applied successfully Accessors could take a pointer or possibly return a string: I'm not sure if you'd need to use the a return code, but you might want to differentiate between: - The setting did not exist - The setting had no value - The setting had a particular value The "settings bus", if done well, could be re-used for anything that wants to modify settings - MQTT for example could hook into it. *Sidenote*: I'm thinking it's time for some singletons in this project, I was trying to avoid them for the sake of purity but the constructors are getting rather argument heavy and adding in On Tue, Dec 3, 2019 at 2:48 PM Proddy ***@***.***> wrote: > (from the list in #57 <#57>) > I've started implemented a multi-user Serial and Telnet console where using > a Serial monitor to the USB or Telnet client to esp8266-react with the plan > where you can issue commands to set parameters and monitor traffic. But got > stuck finding an optimal way to read and set each of the Services > parameters that are stored in the individual JSON files. For example I > would like 'set wifi ssid ' to set the SSID. Also in other services I'd > like to be able to read certain system settings, e.g. hostname. > > Can you think of any idea to achieve this? Now I've mangled up my code > into esp8266-react's and took over SettingsPersistence to store all > settings centrally but its messy. > > — > You are receiving this because you are subscribed to this thread. > Reply to this email directly, view it on GitHub > <#66?email_source=notifications&email_token=AAKE4VBL7DQ3T6EHMPRTKVDQWZWUJA5CNFSM4JUZHW42YY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4H5WILFQ>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/AAKE4VER4NS5AYIK5SBAFH3QWZWUJANCNFSM4JUZHW4Q> > . >

[proddy]

I have a first implementation of this, but had to mangle up a lot of the inner core so not sure it's for everyone.

Basically it creates a UNIX shell like instance where multiple clients can connect via Serial/tty or Telnet (port 23). Basic commands show the status of the system and after 'su'ing and using the JWT password you can change all the system parameters, e.g. 'set wifi ssid '. Command line validation and default values also supported.

To make this work across all the services I extracted the current way settings are stored in separate SPIFFS files to a single JSON messagepack file and a Config class to read/write settings. Everything is in flash memory and uses std::string. It works great in the framework code but not so nice when building your own services such as the 'demoProject'. I wanted to avoid having so many read/write operations on the Filesystem.

Anyway, I'll share the implementation soon in my fork when I finished ironing out the remaining bugs.

[proddy]

I finished the code this morning have the basic shell & mqtt up and running. Except now the web server crashes due to lack of memory. When I connect to the web with a browser I see the available free heap (on an ESP8266) jump from 4200 bytes down to 800. AsyncWebServer and ArduinoJSON don't have enough mem to deserialize so it stays in a hang state. Need to figure out what is happening.

[proddy]

ok, so checked the console code it into my fork. Good news is that the shell console, telnet and mqtt works. Bad news is that will all these services enabled it eats about 6kb of ram leaving nothing for the web server, so web doesn't work which sucks. I'm still trying to figure out what's using up all the heap.

Anyway take a look and there may be something useful for your project. I put some notes in the README.

https://github.com/proddy/esp8266-react

[rjwats]

cool, i'll take a look

…

On Sat, Dec 28, 2019 at 7:57 PM Proddy ***@***.***> wrote: ok, so checked the console code it into my fork. Good news is that the shell console, telnet and mqtt works. Bad news is that will all these services enabled it eats about 6kb of ram leaving nothing for the web server, so web doesn't work which sucks. I'm still trying to figure out what's using up all the heap. Anyway take a look and there may be something useful for your project. I put some notes in the README. https://github.com/proddy/esp8266-react — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#66?email_source=notifications&email_token=AAKE4VGJZZPF34TGKTWIROTQ26VTRA5CNFSM4JUZHW42YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEHYRCBI#issuecomment-569446661>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAKE4VF2OPBGKK4FVJCWGS3Q26VTRANCNFSM4JUZHW4Q> .

[rjwats]

I've had a flick through the diff, lots of interesting stuff - thanks for sharing it! It's a bit too much to take in in one quick pass :) There's definitely stuff which should come across. I like the improved NTP utilising lwip2 and the MQTT support. I like the shell library you discovered, how did you come about it? It appears to be well written allows for the development of a very modular shell. Moving the strings to flash was a good idea too. I'm sure that will make a big difference to my efforts to bundle the project as a library.

…

On Sat, Dec 28, 2019 at 8:04 PM Rick Watson ***@***.***> wrote: cool, i'll take a look On Sat, Dec 28, 2019 at 7:57 PM Proddy ***@***.***> wrote: > ok, so checked the console code it into my fork. Good news is that the > shell console, telnet and mqtt works. Bad news is that will all these > services enabled it eats about 6kb of ram leaving nothing for the web > server, so web doesn't work which sucks. I'm still trying to figure out > what's using up all the heap. > > Anyway take a look and there may be something useful for your project. I > put some notes in the README. > > https://github.com/proddy/esp8266-react > > — > You are receiving this because you commented. > Reply to this email directly, view it on GitHub > <#66?email_source=notifications&email_token=AAKE4VGJZZPF34TGKTWIROTQ26VTRA5CNFSM4JUZHW42YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEHYRCBI#issuecomment-569446661>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/AAKE4VF2OPBGKK4FVJCWGS3Q26VTRANCNFSM4JUZHW4Q> > . >

[proddy]

The shell library was built by Simon Arlott (https://github.com/nomis) and I met Simon after he had contributed to one of my earlier GitHub projects. I've been slowly learning C++ concepts since like the new std constructors and taking time to study his code. For example, in his UUID libraries (console, telnet, log) he had used a design principle everywhere called Static Order Initialization to make everything as asynchronous as possible so that no one operation should starve any other operation of time to execute for too long.

You probably saw some of the other minor changes I had been experimenting with to squeeze out more memory such as

• moving the vtables to IRAM
• running the linker script at the end that removes the printf/scanf floating point libraries
• Using PIO_FRAMEWORK_ARDUINO_LWIP2_IPV6_LOW_MEMORY
• Adding further compiler optimization flags and deeper linting (e.g. -Wall -Wpedantic)

Anyway even with the flash optimization I still can't get my new services working with along with your web code, as each service takes too much heap space. When I comment out and exclude one service (e.g. OTA or NTP) in ESP8266React.cpp/.h and fire it up, go to the console and type show memory I see a gain of roughly 3500bytes in free heap. Maybe its the way the ESP Async WebServer handlers are initialized? So I had to turn them on and off individually to test all the new code with the web interface, annoying!

[proddy]

Also since you're using BearSSL if you add the build flag -DBEARSSL_SSL_BASIC it will build with a limited set of ciphers:

• TLS_RSA_WITH_AES_128_CBC_SHA256 / AES128-SHA256
• TLS_RSA_WITH_AES_256_CBC_SHA256 / AES256-SHA256
• TLS_RSA_WITH_AES_128_CBC_SHA / AES128-SHA
• TLS_RSA_WITH_AES_256_CBC_SHA / AES256-SHA

which reduces the OTA size with around 50KB.

[proddy]

to help find where all the memory is going I added a very rough feature toggle so at build time the modules (MQTT, LOG, NTP etc) can be included. And then a python memory analyzer which I stole from another project to see what's going on. Here's the output:

$ ./scripts/memanalyzer.py ALL
Selected environment "esp12e" with modules: LOG MQTT NTP OTA

Analyzing DEFAULT configuration

Module              |Cache IRAM     |Init RAM       |R.O. RAM       |Uninit RAM     |Available RAM  |Flash ROM      |Binary size
                    |.text + .text1 |.data          |.rodata        |.bss           |heap + stack   |.irom0.text    |
--------------------|---------------|---------------|---------------|---------------|---------------|---------------|---------------
DEFAULT             |29528          |1556           |25300          |31256          |23824          |750700         |811232
LOG                 |0              |0              |0              |96             |-96            |160            |160
MQTT                |0              |0              |104            |80             |-192           |1080           |1184
NTP                 |0              |0              |0              |112            |-112           |160            |160
OTA                 |0              |0              |0              |104            |-112           |160            |160
--------------------|---------------|---------------|---------------|---------------|---------------|---------------|---------------
ALL MODULES         |0              |0              |104            |400            |-512           |1096           |1200
TOTAL               |29528          |1556           |25404          |31656          |23312          |751796         |812432

next step is to run the code, with the web enabled and use show memory to see how the heap is used when each module is activated.

[proddy]

BTW I moved my code to https://github.com/proddy/MyESP2 so I can work on your fork separately

[rjwats]

Sounds like a good idea - will allow for more granular PRs.

…

On Sun, Dec 29, 2019 at 2:06 PM Proddy ***@***.***> wrote: BTW I moved my code to https://github.com/proddy/MyESP2 so I can work on your fork separately — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#66?email_source=notifications&email_token=AAKE4VEYLTEACA6YA2HKRLTQ3CVIBA5CNFSM4JUZHW42YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEHZAHLY#issuecomment-569508783>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAKE4VAQQD4BBUVFZASFVP3Q3CVIBANCNFSM4JUZHW4Q> .

[proddy]

been off for a while now coming back to this. In my branch (which I moved to https://github.com/proddy/esp8266-react/tree/proddy) is kinda in holding pattern while I figure out what is eating up all the heap memory. With all the modules loaded it leaves around 8k left on the available heap which is not enough to serve the web pages, so renders it pretty useless even for a blinking LED prototype. My previous framework which used the same libraries (asyncweb, asynctcp, ntp, ota, mqtt etc) plus the main app which was massive left 13K free so something is off and I don't know what. I put some notes in the README

Anyways I'm following the TypeScript conversion - great work. I'm learning a few things too. Thanks for still supporting this great project.

[rjwats]

Cheers Proddy!

I've spotted a bunch of tidy-ups now that I've properly re-visited the UI again. Some of the what was written at time time has been somewhat outdated by newer APIs and techniques, so it's worthwhile! I'm working on a straight forward conversion at the moment then later this week I'll get on modernising the code.

Hope to have something to review later this week, it would be great to get some feedback when the time comes!

[proddy]

@rjwats forgetting the console idea for a minute, would it be possible to extrapolate out all the debug messages you send to the serial port (Serial.print....)? The reason is that in some of my projects I can't use Serial and I need to go through all the framework code removing the lines. Perhaps wrap them in #ifdef or make a small logger class like in #85.

[rjwats]

Yes a simple abstraction for logging would be a sensible change! I'm messing around with MQTT at the moment to see what can be supported, need to find time to finish the "feature service" stuff so I can comfortably add features with the knowledge they can be turned off by others when not required.

…

On Mon, Feb 24, 2020 at 9:24 AM Proddy ***@***.***> wrote: @rjwats <https://github.com/rjwats> forgetting the console idea for a minute, would it be possible to extrapolate out all the debug messages you send to the serial port (Serial.print....)? The reason is that in some of my projects I can't use Serial and I need to go through all the framework code removing the lines. Perhaps wrap them in #ifdef or make a small logger class like in #85 <#85>. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#66?email_source=notifications&email_token=AAKE4VHOZQOYSB3YF5SDV5TREOG4HA5CNFSM4JUZHW42YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEMXDMTI#issuecomment-590231117>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAKE4VGTCMJNUXPAD3L44G3REOG4HANCNFSM4JUZHW4Q> .

[proddy]

closing this feature. It's a bit of special use-case and probably not something you want to add this project, with the risk of becoming bloated and having yet-another espurna/tasmota/homie that does everything

[rjwats]

I agree with that assessment. It's a really neat feature that might be a little too niche.

…

On Mon, Jun 22, 2020 at 3:07 PM Proddy ***@***.***> wrote: Closed #66 <#66>. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#66 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAKE4VHJ36SBV2PNZQAMETTRX5QSBANCNFSM4JUZHW4Q> .