DSMR Meter about feature-requests

[Mynasru]

I'm working on this, already made a working custom sensor. Next month I'll try to integrate it into Esphomelib.

[rspaargaren]

Hi I am currently working on a dsmr component for esphome. So we do not have to use the custom code. I will release a beta version soon for testing

[nldroid]

I created a custom component but didn't test it (because i don't have a cable yet)

https://github.com/nldroid/DsmrP1CustomSensor

[rspaargaren]

@rspaargaren I would like to help make a readme, maybe with pictures?. You say its easy to google, but i found 3 different wirings so its not "that easy"!!

So if i can make your coding work (wich looks really clean !) with a BC547 transistor i would be very happy!

Hi I worked on the code together with NLdroid. His code is quite similar to mine and has a better readme.
I have used the following scheme to read the values:

I am using the 5V supply from the meter so I can plug-in the ESP stand alone.

The complete scheme for connecting:

The source of the information I have used is:
https://willem.aandewiel.nl/index.php/2019/04/09/dsmr-logger-v4-slimme-meter-uitlezer/
(In dutch language)

[olivierfaber]

@Mynasru

I'm working on this, already made a working custom sensor. Next month I'll try to integrate it into Esphomelib.

Could you share your custom sensor config?

[geelenbert]

I would like this feature too.

can you share the custom sensor config? Maybee it is something we or someone else can work with.

[rspaargaren]

A working custom DSMR sensor for ESPHOME can be found at:
https://github.com/rspaargaren/DSMR_ESPHOME
This library is based on the dsmr.h -> https://github.com/matthijskooijman/arduino-dsmr/blob/master/src/dsmr.h. So please read this page for more info on how to connect. My code is a bit WIP.

[Pimmetje]

@Mynasru If u need help testing or something else please let me know.

[Mynasru]

@Pimmetje I got the hardware working, but do not yet get reliable results from the software. I plan to port this code to ESPHome (if allowed by the licence). I'll put the code on my GitHub and contact you for testing.

[GraafG]

I like to add some existing libraries:
https://github.com/romix123/ESPEasyDSMR5/blob/master/_P110_P1WifiGateway.ino
https://github.com/arendst/Sonoff-Tasmota/wiki/Wemos-D1-Mini-and-P1-Energy-Meter

I'm using D1 Mini with espeasy firmware at this moment.
D1 and this module: http://www.esp8266thingies.nl/wp/ (I have some boards for d1 mini for sale by the way)

[olivierfaber]

@Pimmetje I got the hardware working, but do not yet get reliable results from the software. I plan to port this code to ESPHome (if allowed by the licence). I'll put the code on my GitHub and contact you for testing.

Any news? Is there anything I can do?

[Pimmetje]

Note that the code does not yet support DSMR 3 matthijskooijman/arduino-dsmr#22 (that what my meter is using).

[Theoreet]

there's some tasmota coding as you can read : http://www.esp8266thingies.nl/wp/software/
+1 from me with this feature request

[Theoreet]

I'm working on this, already made a working custom sensor. Next month I'll try to integrate it into Esphomelib.

could you maybe tell us something about it, of give us an update ?

[geelenbert]

I'm working on this, already made a working custom sensor. Next month I'll try to integrate it into Esphomelib.

yes, i'm interested in this too. Anything we can do to help?

[Taigar2015]

Also interested.
I have ESMR5.
Any news/updates?

[rspaargaren]

I am working on a custom library for the DSMR reader. Based on nldroid his input but not based on softwareserial. In the Yaml I am initiating the UART:

uart:
id: uart_bus
rx_pin: GPIO3
baud_rate: 115200

And within the custom library I am reading the buffer line or line:

void update() override {
ESP_LOGD("DmsrCustom","Updating..");
digitalWrite(D5,HIGH); //Data is only submitted if pin is HIGH
if (available())
{
ESP_LOGD("DmsrCustom","Data ready..start if loop");
memset(telegram, 0, sizeof(telegram));
while (available())
{
ESP.wdtDisable();
int len = readBytesUntil('\n', telegram, P1_MAXLINELENGTH);
ESP.wdtEnable(1);
telegram[len] = '\n';
telegram[len + 1] = 0;
yield();
ESP_LOGD("DmsrCustom","*%s",telegram);
bool result = decode_telegram(len + 1);

etc... etc...

If I run the custom library without a polling interval I can read the full message but as soon as I activate pollinginterval I can only read the second half of the message. Based on my limited guess is that without the pollinginterval the routine is fast enough to emply the uart buffer line for line and even wait for the new line to arrive. Within the polling interval the UART buffer is not updated within the custom routine and only the last part can be read what was still in the buffer. (FIFO) so the first part is already out..?

[mcflexus]

trying to compile it for esp32 but it won't because D5 is not defined.
D5 is called from setup and update functions in dsmr_p1_sensor.h
is this D5 pin the rts pin?

[bramvdpol]

I ran rspaargaren's code on a wemos d1, however I was not getting any power readings. I suspect this may be due to way I wired the p1 port, with RTS permanently high. I commented the logging part back in, while the "READING READY" messages got through, no data was logged. Was running this with few issue previously.

However I read back a bit and noticed @nldroid updated his version yesterday , which works fine after fixing the include name (dsmr_p1_uart.h -> dsmr_p1_sensor.h) in the .yaml. Thanks!

[JoeyPoortman]

@bramvdpol How did you wire it up?? I used a 4 wire rj11 connector and put a 1kOhm resistor between rts and vin (wich gives ~5v). And the only thing i'm getting in espHome is : Reading Ready

[bramvdpol]

You probably need to invert the serial signal coming from the meter, depending on your specific model. There are a few where this is not necesary but for most meters it is. Mine is a XS210 ESMR5 meter which needs inverting. I searched a bit but couldn't find the exact schematic I used, but it is something similar to http://compuron.nl/IoT/meter/index.html. I also needed to add a 1K pull up resistor between data and 5V but this may not be needed for your meter.

Good luck

[sandervandegeijn]

Would like this as a standard component to discard my own custom solution :)

[nldroid]

It's not that hard to use the custom component. See https://github.com/nldroid/CustomP1UartComponent and open read_p1.yaml

[JoeyPoortman]

@bramvdpol Thanks for the quick reply!

I used this script: https://github.com/nldroid/CustomP1UartComponent
And used the hardware inverter you linked. Wired it up but no bueno. I must be doing something wrong with the hardware inverter or the wires from the cable i guess!

D5 is hooked up to pin 2 without a resistor.
D2 is hooked up to the collector pin on the bc547
Pin 5 is hooked up to the base on the bc547
Pin 3 is connected to ground aswell as the emitter of the bc547

This is exactly like the picture with the 74c04 but i'm not using that ofcourse so i guess that my issue is somewhere there right?

[nldroid]

I had problems with transistors too (probably my fault) so i got the hardware inverter. Worked immediatly.

Here's more info about other types of inverters:

http://domoticx.com/p1-poort-slimme-meter-uitlezen-hardware/

[JoeyPoortman]

@nldroid Probably going to order a module and pick this up when i get it.. Its such a waste of time. 3 different esphome setups i've found all use different pinouts or hardware and none have some sort of plan to show you what you should do. This one for example:
https://github.com/rspaargaren/DSMR_ESPHOME Utterly useless without any form of instructions :(

[rspaargaren]

@JoeyPoortman A Readme was on the to do list.... but if you google a bit on the DSMR meters it is quite obvious on the pinout side. Most of the meters require an inverter for the communication. And the read out pin can either be contineously high or connected to and IO on the ESP.

[Mynasru]

I created a custom component but didn't test it (because i don't have a cable yet)

https://github.com/nldroid/DsmrP1CustomSensor

It works for me with a Wemos D1 mini and a simple transistor inverter. I was struggling to get it to work at first. However, I remembered that there was a message request signal, and I did not configure that pin (it is set as D5 in the .h file). So my mistake. I will let it run for a while and check how it is reporting.

[JoeyPoortman]

@rspaargaren I would like to help make a readme, maybe with pictures?. You say its easy to google, but i found 3 different wirings so its not "that easy"!!

So if i can make your coding work (wich looks really clean !) with a BC547 transistor i would be very happy!

[nldroid]

The 'problem' is that there is more than one way to Rome. There's no 'best' way. You can use the one that suits you best, For me it was using a hardware inverter.

Ik you look on this page:

http://domoticx.com/p1-poort-slimme-meter-uitlezen-hardware/

You'll find an example with BC547 inverter.

[JoeyPoortman]

@nldroid Thanks you again! That one uses a BC547 indeed, but it also doesn't use a trigger like mentioned above. So i should in that diagram wire 5v from the arduino, instead from D5 into the BC547 and that should be it right? (that didn't work earlier but i'll try on a breadboard again! )

[JoeyPoortman]

Still doesn't work after spending another hour on it. I do have a Landis+Gyr E360 meter. Maybe its different from the widely used E350?

[rspaargaren]

Still doesn't work after spending another hour on it. I do have a Landis+Gyr E360 meter. Maybe its different from the widely used E350?

If you look on the following page:
http://domoticx.com/p1-poort-slimme-meter-hardware/
You can see that the Gyr E360 has an EMSR5.0 and the Gyr E350 the DSMR4.0. That might make a difference.

My meter is an ISKRA with the EMSR5.0 port. As far as I know the protocol is the same for the two versions. The EMSR5.0 port is able to supply a higher Power output. So you can use it for powering the ESP.

[JoeyPoortman]

I caved and bought a board from Esp8266thingies. Worked right out of the "box". Now only thing is to link it to influxdb to store the data!

I uploaded the same sketch to my old setup and it also worked right away so idk what i did wrong but something wasn't right software wise!

[dynasticorpheus]

@JoeyPoortman Are you now using ESPHOME on the board purchased from Esp8266thingies? Would like to move as well hence wondering if you have a working setup or I should simply stick to the software it was supplied with.

[JoeyPoortman]

@dynasticorpheus I am using the default software the board was delivered with. But i can't get it to work with HA, there is just nothing published. So i'm tempted to just flash esphome on it! The seller has a full mailbox so i can't ask for help anymore :(

[GraafG]

FYI I have some esp8266thingies boards for sale on: https://tweakers.net/aanbod/2306796/p1-shield-voor-wemos.html

They are tested and working, and would be willing to sponser to devs working to get this working in ESPhome.

[rspaargaren]

FYI I have some esp8266thingies boards for sale on: https://tweakers.net/aanbod/2306796/p1-shield-voor-wemos.html

They are tested and working, and would be willing to sponser to devs working to get this working in ESPhome.

Nice piece of hardware! Would not be to hard to adapt the scheme for esphome. However our libraries are working on custom library basis

[JoeyPoortman]

@GraafG Do you use the same software as esp8266thingies delivers them with??

I think if you just drop the code from above onto it, it works straight away. I don't have the original firmware that is on the board so if things go sideways i can't reflash it, that is my only problem atm..

It works like a charm!

[dynasticorpheus]

Latest binary can be downloaded here http://www.esp8266thingies.nl/wp/wp-content/uploads/2019/05/ESPEasyDSMR5v4.bin_.zip

Source: https://github.com/romix123/ESPEasyDSMR5

[dynasticorpheus]

@JoeyPoortman I actually found a solution to get the data fed into my own home assistant instance despite the lack of mqtt support whilst using the software it's shipped with. Not necessarily very straightforward but rock stable though :)

enable remote logging --> syslog-ng outputting json formatted log file --> read by hass file sensor

[JoeyPoortman]

@dynasticorpheus I can't find the option "enable remote logging".

I did just install esphome on it, it works after changing the pin number for reading. It is really slow and goes offline for a second to update. But atleast i have something in HA right away!

Romix said in a email that the tutorial to hook the board up to HA is on his website, i can't find any details as to what you have to setup on the device itself! There are no mqtt messages from the device with any setting (it does however connect to the mqtt broker running in HA).

I would like to have the speed of the espeasydsmr5v4 output directly into influxdb, then i can setup anything i want to display.

The small gaps are the time the device is offline.

[dynasticorpheus]

@JoeyPoortman I think the ESP EASY has built in MQTT support hence it connects however the P1 specific code is not utilizing this.

For my approach you need to set below options which you can find in the menu via Tools --> Advanced

Syslog IP: 192.168.x.x (= home assistant instance with syslog-ng running)
Syslog Level: 2
UDP port: 514

[JoeyPoortman]

@dynasticorpheus Thanks you for the example. I'm running right into the next wall, Syslog-ng isn't a package i can easily install. Is it best to install it using Portainer you think?

[ESPmonster]

@JoeyPoortman I also have/had a lot of trouble getting the ESP8266thingies board to work with HA. The method on Romix' site works (http://www.esp8266thingies.nl/wp/p1-wifi-gateway-parasitic/) --> i.e., setup (generic) HTTP as controller in ESPeasy and add DSMR, IP and port in configuration.yaml (and meter_readings as a group). (Note that I could only find the DSMR integration after looking for 'ignored integrations'!) However, in my case this method only works for ~20 min until the data flow stops and then I have to restart HA (without errors in DSMR debugging log on HA or ESPeasy)!

On the same website you can find a Node-red script that could connect to the port and publish the data as separate MQTT topics but I fail to establish the TCP connection.

For ESPhome, did you change the reading pin to RX or D6?

@dynasticorpheus I might also try your solution but I'm running hassio so Syslog-ng doesn't seem an easy thing to do.

[JoeyPoortman]

I gave up on the software side from Esp8266thingies, its unstable and clunky (sorry).

I went with the DSMR Reader solution, it has to run in its own docker container outside of HA. But its buttery smooth, its been running since March without any glitches! Pm me if you want to get some details in Discord :)

[ESPmonster]

@JoeyPoortman Yes please! My Discord is ***

[dynasticorpheus]

@ESPmonster I actually moved away from Esp8266thingies software in full and flashed below esphome based solution which is rock stable and (auto) creates required sensors in hass by its own without the need for anything else.

https://github.com/tbrasser/CustomP1UartComponent

The only thing that I needed to change in read_p1.yaml next to the WIFI AP credentials was GPIO21 into GPIO3:

uart:
  - rx_pin: GPIO3
    baud_rate: 115200
    id: uart_bus

edit: If you want updates happen faster than once a minute you can change the DELAY_MS variable in dsmr_p1_uart.h

[ESPmonster]

@dynasticorpheus Excellent! I'll try this firmware this week.

In order to make this info more easily find-able for others I'll add some tags: ESP8266thingies Wemos D1 mini lite home assistant DSMR sensor P1 gateway reader MQTT kwh

[ESPmonster]

@dynasticorpheus I installed the P1UARTcomponent firmware. Works OK. For the esp8266thingies board (on a Wemos D1 mini lite) I also had to change the UART pin to:

logger: 
  level: DEBUG
  hardware_uart: UART1

What are your settings in dsmr_p1_sensor.h/dsmr_p1_uart.h?
I have very frequent home assistant api disconnections (~3x/5 min) even though signal strength is optimal and I never had this problem with this or any other ESP devices. My current settings are:

#define P1_MAXTELEGRAMLENGTH 1700
#define DELAY_MS 29000 // Delay in miliseconds before reading another telegram
#define WAIT_FOR_DATA_MS 20000

Edit: also, unfortunately this firmware is missing some data/sensors (N long/short power failures)

[dynasticorpheus]

@ESPmonster Forget to mention I also changed platform and board settings, perhaps this is the culprit? I pushed all my changes to below fork:

https://github.com/dynasticorpheus/CustomP1UartComponent/commit/e790cf353cf6dc3ed76ddbb2875ce9923360ab45

[rspaargaren]

There are some delays in the custom component which are blocking the progress in the code. This is disturbing the progress in overall code and WiFi reception can drop. I have the same issue with the custom code.

[dynasticorpheus]

@ESPmonster @rspaargaren Perhaps because I reduced update frequency from 60 to 3 seconds this is not a problem on my end. Anyway good to know there will be an official component soon, great news.

[ESPmonster]

@dynasticorpheus Thanks for the info and forking! I also adjusted board settings and further played with the setting to no avail. In fact, my other ESPhome devices are suddenly also affected. Turning off the P1 wemos only helps a little so perhaps it has something to do with my router.
@rspaargaren looking forward to your beta, thanks