Homie future? about convention

[schaze]

Hi guys,

it was a very interesting conversation tonight and very nice to meet you "in person"!

short recap of the call today, so I do not forget.

Participants:

• @ogghst
• @Tieske
• @schaze

(@Thalhammer could not attend)

Brief:

• We discussed between revolutionary vs evolutionary approach for the new protocol - in the meeting we leaned more towards evolutionary but nothing is finally decided yet
• We more or less agreed on moving the device description to a single topic per device
• existing datatypes should be extended (complex formats like e.g. json/protobuf/...). At the same time we should think about removing existing complex types like color and just let them be modeled with the generic complex datatype approach (e.g. json/protobuf/...)
• we could not reach a real consensus about json vs protobuf -- we agreed that we would need some more discussions including Thalhammer with this to ensure all pros/cons are considered enough. Should be part of the next discussion
• the convention should also support connected state for properties in a way to reach eventual consistency (proposed solution from Thieske was having a /target topic as an 'in-flight' state, e.g. for when the requested state (with /set) cannot be reached instantly)
• Profiles or Capabilities should be defined on top of the protocol itself to ensure certain structure and behavior

Action items:

• @ogghst : will complete a proof of concept (sparkplugb decorator?) using protobuf and present it to the rest as a revolutionary approach, also including the one topic for the device description change discussed today
• @Tieske : will look for some version of chat channel (slack, irc, ... ) where we can discuss better and more dynamically than in this github issue
• @schaze : will setup a repo with documents where we can start drafting out the device profiles. It will be prefilled with my existing profiles and their descriptions

@ all: please comment if I forgot anything, then I will update this post to ensure we have one location very a complete information can be found.

regards,
schaze

[schaze]

Hi all,

as I have been mentioned in @Tieske's first comment I received mail notifications about this topic and I have been reading your comments with a lot of interest.
If you don´t mind I would like to contribute to the discussion. In the spirit of this issue I will make it a long one as well :).

I actually did have the exact same idea in mind as @Tieske regarding the design of a HA solution. I wanted it split up in independant seperate parts connected by an open service bus. In addition I also wanted each component to be more or less configurable from the startup by simple environment or file input. No manual ui configuration with data stored in some kind of database with the fear of loosing the ever growing solution nurtured over months by some failed upgrade or db inconsistency. Simply place a file(s) along the component and start it - done.

I was a FHEM user at the very beginning years ago, then at some point switched to openHAB. At anypoint I always had my own UI implemented and attached to the solutions. I have tried HomeAssisant several times - I think it is an impressive solution and has gained even more momentum in the last year or two - but I cannot get warm with it - even though it was the only solution that could compete with the UI extectations I had. I always used NodeRed for some edge cases and used mqtt as a bus between different systems.

That led me to search for an alternative which brought me to Homie about 1-1.5 years ago. Unfortunatley already then it seemed like it had not really taken off and there were no real software solutions available to build a full HA stack with.
As I needed a new hobby project anyways I though why not go an implement it on my own (foolish idea as it is a looot of work).
My idea was to get something running and proove it can work, the come here and post about it to get your opinions on it.
It has been running at my home now for about a year and is absolutely reliable. Beginning of the year I started to open source everthing on github split into the base library in nodejs https://github.com/schaze/node-homie that implements the homie convention and actually offers a bit more so it is almost like a framework of sorts.
Of course as you also mentioned in the posts above without device profiles there is no way to make a HA solution. So my idea was to have a project called homecontrol that would define such profiles and implement solutions that would revolve around them to implement a full HA stack with everything needed: https://github.com/homie-homecontrol
But writing code is fun - documentation is not. In addition I did have some other side projects and family that did not let me finish publishing everything so far.
I started with the base libs and nodered as I thought these would provide already some usage for others on their own. (looks like I was right with that :) )
Then followed the deconz (hue) and homematic controllers that publish devices on the bus using the device profiles, afterwards the influxdb logger that logs data to an influxdb for graphing.
What I have still open (but fully working at home) currently is:

• hc-dashboard: Dashboard UI - here a screenshot:
  
• hc-homie-automation-controller: Logik engine including hosting of virtual devices
• hc-alexa-homie - alexa skill and local daemon to integrate homie devices into alexa (still requires manual skill deployment in aws)
• hc-pushnotify - will publish notifications via pushover or gotify sent to a property exposed via homie
• hc-homie-presence - presence detection via network ping
• hc-homie-manager - simple web UI (purely browser based - no backend) to inspect homie mqtt devices and wipe or interact with the (still alpha version so far)
  
• hc-yamaha2homie - connected Yamaha AVRs to homie
• hc-homie-alexa-remote-ctrl - control your amazon echo devices via homie (also exposed timers and musicplayer details)
• vscode-homecontrol - vscode plugin to help with yaml config file for all the above components and also offers homie device discovery within vscode as well as code completion for homie devices:
  

start EDIT: All components can use plain homie devices, using devices that follow the existing profiles makes usage however more simple. end EDIT

I have a few more components on my todo list and the existing ones need some code improvement and partial redesigns (it is just to much stuff doing it alone) but whatever I cannot support currently I can connect via the automation controller (it has JS extension possibilities) and nodeRED.

I have learned a lot by implementing all these and also struggled a lot with the downsides (mentioned by guys above) of the homie convention. I based the whole implementation of rxjs so everthing is more or less stream based wich eliminates some issues regarding them. I especially have a issue with the verbosity of the convention regarding mqtt topics. I have about 150 devices published over all which leads to about 10.690 published topics! In the beginning before I started streamlining the device structures and removed a lot of metadata I was easily above 35.000 - this is simply too much. I also creates a very high load in the beginning for the discovery.

So since I put considerable work into this I am actually happy to see that there is still some interest or at least some people engaged in it.
I would gladly like to participate to work on a new version that will address these issues - I also hope I can bring some first hand experience with me regarding practical issues of the current spec.
We can use and takeover anything I already did regarding device profiles - or throw it away and do it completly different. The current profiles work well enough for me currently but I already have a lot of improvement ideas for them and also notice some wrong turns I took.

TLDR;
Wrote HA solution on homie protocol, started publishing but not finished. Agree with most of the critique and praise above. Would like to contribute any way I can.

Sorry for the looong post.

• schaze

[Thalhammer]

I would be very interested to learn more about your solution!

Theres not really much to learn tbh. It's basically a big C++20 program that basically serves as a plugin management and message broker and loads in sandboxed components which can communicate with the management and other plugins (assuming they have permission to do so) using rpc as well as expose services for other plugins to consume. My "it can not not work" strategy is fairly simple: I have a docker host (no cluster) where everything (including GIT/CI) is running on and it just makes a backup before deploying a version of it and I have a script to quickly revert if something really did break. This and solid unit testing has so far saved me from "the lights dont work anymore". In addition due to the plugins being sandboxed they can't really take down the whole system regardless of what they do, worst case a couple devices won't work. The grpc api on the management daemon is consumed by the tablets I have for control, as well as the webui which is used for setup and such. Might not be the perfect solution, but its one I liked the most. Plus I have a "scripting" interface based on blockly which even allows non technical users to get some basic automation going. Regarding data: Everything configuration related is in a sqlite3 database and graphs currently live in a external influxdb, though I am in the process of porting whisper to c++ and use that instead (its basically rrdtool, but better), because I don't need the flexibility and scale influxdb provides and having it inside makes stuff easier.

Back when I started with homeautomation people still used to use wires for controlling stuff, which is the reason, why most of my stuff isn't actually wifi/ethernet based but uses a custom can based protocol with separate wires to each lightswitch/outlet (The house was built in 2007 and I had literally multiple km of this cable, 4 wire solid shilded, laying around so it was a no brainer). The old house is way behind in automation simply cause I don't have this infrastructure. Back around 2009-2011 when I started with it FHEM was the state of the art and neither openHAB nor Homeassistent even existed, esp* was unthinkable and zwave/zigbee where in their infancy with pretty much no devices available. I hope this explains why I have a completely custom solution for pretty much everything. If I where to do everything new and start from scratch I'd probably use a standard solution, but it would be a waste to scrap my existing code (despite it being a small mess after ~14 years of development) and more importantly people are used to it, which kinda wipes the idea of changing it at its core.

it is just not as easily digestible as JSON IMHO

Its a binary format (with a not quite straight forward encode especially for integers), but there are multiple tools and gateways to convert back and forth between json and protobuf.

Also i would make inspecting the data in mqtt via tools like "MQTT Explorer" a lot harder for debugging or human readability.

Not really. Protobuf can be embedded into js/typescript projects and you can load protobuf files at runtime, without having to generate code for it before hand, meaning the tool could convert between the encoded message and json/some other readable format fairly easily. In addition Protobuf can emit the protobuf description as a protobuf message, meaning you can in theory have selfdescribeing messages, though it's fairly seldom used for obvious reasons.

So the more I think about it the more I like it.

I've worked with protobuf in a number of ways (all the way from grpc usage, over custom rpc on microcontrollers, to generating entire data mappings between sql and code) and while its probably not perfect in every aspect, its imho the best overall package. Though I might be a bit biased here.

I am willing to do so, yes.

I second this.

I think the question is how to we get organized and how much time do people have?

The most obvious option would be to add everyone interested to the involved team (or create a new one to keep the "legacy" protocol separate) and then start writing the new protocol in a separate branch while taking the general structure and ideas from the main branch. While at it I'd also like to do some maintenance work by merging convention-website and homieiot.github.io into a single repository as its usually done and change over to github actions for the deployment.

I am a project manager in my real life :) and I am used to wrap up and assign tasks everyday.

Well I am not, but I have some experience with Thalhammer/jwt-cpp. Feel free to tell me if I screw it up. Judging by the interactions here its only gonna be the four of us (@ogghst, @Tieske, @schaze and me) that mostly work on it, so it should be fairly manageable.

readable: there must be a simple way to read data from mqtt messages

Thats the only real pain point of protobuf. By default its not human readable, you need some kind parser (though it can be as simple as CyberChef and we could probably host a variation of @schaze 's hc-homie-manager along the website to allow people connect directly to their mqtt servers and inspect things. Another thing is that you can always parse a protobuf message regardless of if you have the definitions, as the actual data type of a field is part of the blob, you just wont have a human readable name for the field.

I suggest you all to look at that protocol as seems to me we are really close to it.

I did skim over it and it seems like we are indeed closesish to it. However Sparkplug is really oriented a lot more towards SCADA and Industrial. One important thing I could not find there which would differenciate them is the description of the payload. They treat the protobuf basically as a binary json replacement where every vendor/device can put any number of properties and names it likes to. Homie would provide a fixed subset of commands for things like OTA Updates, device profiles like light switches, etc and a general request/reply mechanism that would support streaming results. Not to mention that homie should really be focused on HA and thus accept contributions a lot easier.

I will investigate in using protobuf in C as it is THE microcontroller language

I have done that before. You need a couple kB of flash to make it work, but there are multiple projects that provide (de-) serialization for embedded targets. The most common choice would probably be nanopb but its also fairly easy to others. A c++ embedded one would be EmbeddedProto (though its sadly payed for non opensource), but its also fairly easy to do your own. I'd argue that once you do more than like 2-3 messages protobuf will outperform json on all platforms both performance and space wise.

[Thalhammer]

Really sorry I couldn't attend, but a personal event stole my timeslot. I don't really disagree with any of the points you guys agreed to though, so its probably not that big of an issue. I really hope I can make it next time.

will setup a repo with documents where we can start drafting out the device profiles

@schaze I'll steal this half of your todo if you don't mind and setup the repo as a private one inside the homieiot org (pretty sure that should be possible to do) and give you 3 permissions for it.

EDIT: You three should all have gotten an invitation.

[Thalhammer]

What paid software have you in your mind?

I think he meant that he's not familiar as to what products opensource projects usually use for working together. Not that he wants to use a paid software to do so.
For another project we have a dedicated discord server, but mostly because it started out as people meeting already in discord. Apart from that: Github already has a fair share of stuff built in (Discussions, Projects and Wiki). Apart from that theres obviously google docs if needed. If we need something else and its opensource I can host it no problem.

[Tieske]

I just added a tab to the spreadsheet by @ogghst ; https://docs.google.com/spreadsheets/d/1zwcJ54LA31pcJZ_Hc8ANlXqmqGOfrOHupxsWKEjduic/edit#gid=2045563054

To try and give it a start to define the design goals. Feel free to comment on it (content as well as structure), and please add your own ideas on design goals.

[Thalhammer]

I added my two cents to the docs.

like expanding custom data structure definitions upon needs

Protobuf actually sortof supports this, just not in the normal way. The usually way this is done is by either embedding the extended fields in a field of the parent, or (my favorit way) by concatenating messages. Unlike something like JSON you can simply concat protobuf messages. Now if you want to extend a message you do something like this:

message Parent {
    required string field1 = 0;
    optional string field2 = 1;
    reserved 1000 to 10000; // Make sure these are only used by extensions
}
message Child {
    required int extension_field1 = 1000;
    required string extension_field2 = 1001;
}

You then encode the parent and Child and simply put them behind each other in the same buffer. The buffer can now be successfully be parsed using both the code generated for Parent as well as Child.

For official extensions we could also do something like this:

message Parent {
    required string field1 = 0;
    optional string field2 = 1;
    optional Extension1Info extension1= 1000;
}

Because of how protobuf is structured you dont need to know about the existense of the field in order to parse the message.

I am not a friend of {J,B,P}SON cause it will put the field names into every single message which is simply useless overhead, since the structure of the message is specified by the convention and thus static. If you dont know the convention for a specific feature you can't use the data regardless if its a named field or a number .

[Thalhammer]

One thing that crossed my mind lately is that protobuf and json doesn't need to be an exclusive choice. We can actually support both fairly easily.
Protobuf generated code (on desktop) can do JSON parsing/serialization out of the box anyway and its 1:1 mappable. So we could allow a device to use either and only a controller would have to understand both, which they easily can. What format to expect could be decided based on the format the device description is in. The opening of a json object '{' is 7b, which conveniently is no longer a valid protobuf tag (it used to be a group start for id 15, but groups have been deprecated ages ago and no major library supports them). So the logic on a controller would be fairly simple as well:

if(description_raw[0] == 0x7b) parse_json(description_raw);
else parse_protobuf(description_raw);

This would give us both the benefits of json (easy integration, wide support, readable) as well as the benefits of protobuf (smaller, faster, safer, codegen) on more advanced devices and well designed libraries.

[Tieske]

@schaze @ThomDietrich @jamesmyatt
None of you added any comments to the sheet so far; https://docs.google.com/spreadsheets/d/1zwcJ54LA31pcJZ_Hc8ANlXqmqGOfrOHupxsWKEjduic/edit#gid=2045563054

would you mind adding your input? (or let us know if you do not intend to, which is obviously fine as well)

Once we have the base input, I'd like to set up a zoom call to discuss, and try and wrap that up. I prefer a call since there are quite some different ideas floating around. Hope every body is fine with that.

[Tieske]

I added a description of the "eventual consistency" as requested by @schaze to a new tab in the sheet

[Tieske]

Since we now have a proposal for Homie 5, I think we can close this now.
Note that the collected design goals (in the spreadsheet) have been merged in the readme.

Please reopen if deemed necessary.

[ThomDietrich]

Great summary. I wouldn't be so hard on Home Assistants automation logic (I actually got quite used to it) and the system is generally very open to other components integrated with it. You can link Node Red to it the same way someone could build a Homie integration for it (I never tested this one). The part you might have undersold in your description is that Home Assistant has hundreds of integrations for proprietary technologies and ecosystems, which are unavoidable in the home automation world. Therefore, I believe that you need something like Home Assistant irrespective of how you integrate your self-build or open source sensor/hardware/software.

I am not utterly proud to admit it but the Home Assistant MQTT Discovery "convention" is a mature alternative to Homie by now, did you have a look at it?

description takes way to many topics, should be a single document probably.

There where many discussions about this. This had its original reasons and discussions over the years did not change that. I can't remember any limitations or issues stemming out of it.
Rather than changing the fundamental way how the protocol is structured, I would suggest to concentrate on smaller improvements to the convention.

[Tieske]

Don't get me wrong; Home Assistant is amazing at what it does. Phenomenal hardware support (same for OpenHAB). But the components should have been be decoupled. Afaik the MQTT support only is inbound, the HASS devices are not exposed in its MQTT format. (They just made the same mistake everyone else made...)

Also I find the format to be limited and not very flexible.

[jamesmyatt]

The main issue is to map-out the Homie ecosystem, look at what it does and how it works, compare them to the other open-source projects out there, work out which bits are important and then decide which bits are worth taking forwards and how to make it happen for the benefit of the entire open-source HA universe. We need to avoid creating something 100% new when the alternatives only seem to do 90% of what we want, and then dropping it when it's not interesting anymore. It can be fun, but it's not productive and will ultimately leave you disappointed.

I'm not an expert but I think that you can make HA and NodeRED communicate directly using components such as https://flows.nodered.org/node/node-red-contrib-home-assistant-websocket. I haven't watched it yet, but I think this video might help: https://www.youtube.com/watch?v=wBUAUdLkYJM.
But maybe it would be enough to add a Home Assistant integration or blueprint to publish selected HA entities to MQTT (I'm pretty sure you can do this via automation and the mqtt.publish service).

You can also use ESPHome without Home Assistant and just use MQTT instead, if you want. I think there are several videos on YouTube about this. Maybe we can just create a component that publishes discovery information to MQTT in Homie format.

Also this month's HA release is supposed to make the Automation UI much better, and you get use to it anyway.

[Thalhammer]

I'll copy paste my original internal post in here, so some might not be directly a response to this post:
======== Copy Paste Start ========

Whats broken

No improvements

As of writing this, there are 26 open issues, most of them feature requests/questions, 1 open pull request and the last mayor commit was 1 1/2 years ago with the last release more than 2 years ago.

It feels like the project is kindof dead. I'd like to change that, but I don't think it is possible to do so with the constraints set by the project as it is. In particular the fact that we split all the data and information into tons of different mqtt topics as well as the fact that we rely on behaviour that isn't guaranted by the underlying technology (message ordering) is quite limiting. Many of the issues where already mentioned by davidgraeff in #199 and nothing has changed in that regard.

Design errors

Another thing is that homie is entirely state based, while the real world simply isn't. Some actions do not really have a state (e.g. water my plant, ) or do not make much sense as a property (e.g. magnetic door latches). While there is a broadcast channel for events, it is a single one for all devices and homie does not define any format, making it practically useless. Homie also assumes that the controller is practically the only source of commands and that practically speaking there is only one. MQTT does not guarantee any message order at all (practically) and rellying on it (as we do) is fundamentally broken. The topology we require is quite limiting and often doesn't represent the real world, causing needlessly complicated (e.g. for a lightbulb the node level is useless) or artificially flattened structures.

And this is only the tip of the iceberg.

Don't get me wrong, davidgraeff did an amazing job at the time, but homie has evolved past that.

How do we fix it ?

Most importantly I'd like to move away from the one topic per attribute. I am not saying we should only have a single topic per x but the state we have now is definitly to much. Instead I'd like to use a small number of topics together with a structured format. As for what structured format it should be?
I am open to suggestions, but I am heavily leaning towards protobuf for a number of reasons, the most important being simplicity, library availability and strict typing/structure. It is also a binary format which should allow for way smaller messages and less overhead. Due to how its designed, it is trivial to stay backwards/forwards compatible as well as allowing people to add custom additions.
The other alternative is obviously json, which while harder to parse, more overhead and not as well defined and typed has the benefit of even broader library support and being human readable/writable.

I'd like to add the option for doing rpc style requests.

To be perfectly honest I'd even wondering whether it makes sense to stay with mqtt at all or use something else, or abuse mqtt with a custom server.

======== Copy Paste End ========

The actual response starts here:

Devices connecting to one service-bus are not able to talk to another service bus. Android/iPhone apps are always specific to a single backend.

At least for a commercial product thats kinda the intention. You want to make sure people are somewhat locked into your ecosystem so you can keep selling your stuff. If everyone could use your (obviously way superior) app with 3rd party stuff you effectively loose money. I don't like but thats simply how it is. As for why OS projects never agreed on a common api subset: I can only speculate but I guess they simply never had any intention/reason to do so.

a service bus

Homie has this in the form of MQTT, however the way homie makes use of it is fundamentally broken. MQTT makes exactly one guarantee, which is that messages on the same topic and the same QOS level (both sub & pub side) will arive in order. As soon as theres multiple topics (and boy does homie use a lot of them) all guarantees are gone. Common brokers might deliver them in order simply because of how they are implemented but thats not guaranteed by anything and might change at any time. Homie however relies on this, e.g. by requiring state to be updated last (which is useless by design).

a protocol/convention

Homie does this pretty well, though imho a bit needlessly verbose.

is "state" based, not RPC

I disagree. While the majority of stuff is indeed purely state based, not everything is and there should be some sort of capability of rpc. Not in the usual way, e.g. like http, but instead as a queue of actions to do somewhen in the future (each of which has a uuid) and a queue/topic where the device posts a reply or status update for the action. A prime example for this would be firmware updates.

device profiles that describe the same functionality

Homie kinda has the base for that in form of the node type. However theres no definition of valid types in the convention and the fact that its restricted to one is kindof restricting. Ideally I'd like to see something like a set of profiles a device can implement, each of which describes a fixed set of topics and fields (more later) that the device needs to implement. Examples would be device types like switch/light/motor, as well as features like ota update, positioning, etc.. (thought I think those could also be split into two separate lists).

description takes way to many topics, should be a single document probably.

Exactly my point. Its made even worse by the lack of ordering guarantees.

how do nodes and properties map onto implementations; a homie device can be a single device (1 light), or it can be a gateway

This is really up to the implementation. In my personal HA I simply built the controller around the split homies has (e.g. I have devices, each of which has a number of nodes, which has properties). On top of that is a driver (e.g. homie, hue, snmp, etc..) but thats not relevant for this. This works quite well with the single issue being that homie relies on state being updated by the last will, which means you can't simulate multiple devices using a single mqtt connection. Depending on your hub you could either use a single device with multiple nodes for each attached device or simulate multiple devices and risk not representing state correctly in case of a sudden disconnect.

interestingly @schaze did some work on models/profiles; https://github.com/homie-homecontrol/hc-node-homie-smarthome#node-documenation It would be nice to see if we could reuse some of that and bring it into the convention or as separate project under the same org.

I didn't actually know about that project, pretty sure we can come up with something like it.

TLDR: In an ideal world, I would switch to a protobuf based system (which is a really powerful language especially if paired with custom options/codegen) to describe both the topics and messages sent within. This would solve a number of issues, as it makes the things well documented, typesafe, allows for complex types and messages and is extensible while maintaining both forward as well as backwards compatibility. If you put some sane restrictions on things like device name length you can even implement (de-)serialization in little code space and without dynamic memory allocation, both of which are kinda relevant in embedded devices.

I'd also attach a message id to every message which creates an implicit ordering inside the protocol, solving the mqtt ordering issue. E.g. if a controller receives both a device description and a state update and the state updates id is lower than the device description ones, it considers the device not ready even if the state says otherwise, because it knows the state is a stale one.
I'd also like to merge all the description (device, node, properties) into a single protobuf message thats published at boot time.
Another competting idea would be to also merge the state into the description (thus on a single topic) and provide some way to do partial updates. This works without the message idea cause messages on a single topic are ordered, so the updates would always arrive after the initial send. I'd also like to add the action queue outlined above.

EDIT: Damn that turned into a pretty long post. The really short TLDR: Protobuf description + better topic layout

[Tieske]

That carries a lot of good points.

I guess the million dollar question is; are we willing to work on a new version?

with follow up questions like;

• who is willing to contribute?
• what are the design goals?
• under the Homie label, or a new project?

[Thalhammer]

So since I put considerable work into this

Damn I can see how much work you did, my home solution is about on par, though with different design choices (e.g. a "monolith" with container based plugins) and I know how much time I put into that. Great job you did there, ngl.

I would gladly like to participate to work on a new version that will address these issues

Help is always wellcome, especially one with experience actually using it on a larger scale (and in this case your scale is even larger than mine 😮 ).

We can use and takeover anything I already did regarding device profiles

Jup I am pretty sure we will shamelessly steal them (in some variation).

@schaze Since you seem to have a pretty large codebase as well, what are your thoughts about moving to protobuf (or something similar) and thus involving codegen on the devices and controller sides. I mainly see the benefits (typesafety, better documentation, easy codegen for tons of languages thus less maintenance), but I can also see how the additional work might be bad for implementors. In my case this would be pretty much no effort cause I already use grpc/protobuf for the api/gui side so the move to protobuf would actually reduce work there, but this is probably not the norm.

Might be a bit much to ask, but could you maybe describe the perfect changes you'd wanna see (ignoring what others and me have outlined above) ?

[Thalhammer]

@ogghst Pinging you in here simply cause you seem to have implemented a new homie project right now. Feel free to drop your thoughts or ignore the ping if you don't want to ;)

[ogghst]

Well, compared to you guys I am a total newcomer, I am just dedicating my very small freetime to play with esp32 (esp-idf, NOT arduino!) and the idea I have to use technology to produce food. Long story.

I will not spend words on the role of communication protocol in a HA ecosystem, but basically I share your vision. This has to be a separate and independent component in the whole stack.

My idea of messaging in a IOT environment is

1 - simple: tried sparkplugb but simply doesn't fit in a open, low code and low performance scenario
2 - self descriptive: the protocol should describe the structure (node, sensor, stats) and the behavior
3 - extensible: the protocol shall be focused on well formed syntax and NOT on huge amount of sensors and states (see homeassistant). My sensors and actuators like PH meters and water conductivity simply cannot be described in predefined protocols
4 - independent: not tied in any way to a framework (see homeassistant)

This is why I came to homie. To me seems the best focused for my real need. I am now planning to propose an OTA extension.

I have no idea on why homie is not widely used, but maybe because i'm just scratching the surface, but I was able to implement this protocol on my framework in days. Simple and effective.

So I am available to help you in moving forward but as said to me the homie philosophy is correct.

[ogghst]

Just ended up reading all the long discussion. I see now the limits of homie and i agree the basic idea is very good, matching lot of requirements, but the implementation has to be revised from the basics.

I do also agree on the use of protobuf as payload.

What I like here is that seems you guys are very open to 'kill' homie, not becoming fanatic, and do not reinvent the wheel but reuse your experience. This is something rare.

Let me think about some concrete ideas on how to move forward, merging the pros of several approaches. Please do the same and keep posting here.

[schaze]

@Thalhammer wrote:

my home solution is about on par, though with different design choices (e.g. a "monolith" with container based plugins) and I know how much time I put into that. Great job you did there, ngl.

Thanks! I would be very interested to learn more about your solution! I wrote mine to be easily deployable in my kubernetes cluster with as less state-fullness as possible. Except for the mqtt and influxdb part everything is more or less working out of the box directly after deployment. All configs etc are kept in ENV vars or configmaps/secrets and are managed by gitops via argocd. That means no backups needed and no risk of any corruption or upgrade fears (which is a major issue when your wife expects the kitchen lights to work no matter your upgrade troubles on OpenHAB....)

I have been reading up on protobuf (I am aware of it but never really used it anywhere yet) and did some thinking in between (also re-reading all posts again).

@Thalhammer wrote:

Might be a bit much to ask, but could you maybe describe the perfect changes you'd wanna see (ignoring what others and me have outlined above) ?

There is not much that I would add to the already proposed changes here. One noteworthy change (which also is somewhat mentioned by you and @Tieske ) is that there is a need for more complex data message formats. Simple primitive types (numbers, strings,...) are not enough for some of the functionality that is needed in HA. Some examples are:

• scheduling plans for heating/air conditioning
• list of active timers in an echo device
• calendar entries
• playlists in music players
• messages to be sent to push notification services
• ...

I opted for using JSON in this case in a string field for homie and added a JSON schema definition into the format field. Works quite well and would be an easy non-breaking extension to the convention. This also ensure type safety, and there is codegens for JSON schemas for multiple languages, however support is nowhere near what protobuf has.

How to move forward?

I think simply improving the current convention in an evolutionary way is not possible any more, the points outlined in the comments above make it pretty clear to me that it needs a more radical change.

protobuf (or message encoding/format)

Even though I am not very familiar with protobuf I would agree to use it for payloads as well.
I do have concerns regarding it being a entry hurdle for new developers to adjust (it is just not as easily digestible as JSON IMHO) and I am not a big fan of code generation. Also i would make inspecting the data in mqtt via tools like "MQTT Explorer" a lot harder for debugging or human readability. But this could probably be solved by writing an explorer that can decode protobuf messages - or make a similar PR to the MQTT Explorer project...
But in the end having the possibility to easily use the convention in a wide variety of language will certainly outweigh this. Also it would make it easier to write plugins/extension for the existing ¨giants" like HASS, OpenHAB and the likes to get them connected to the convention and by this enable a wide hardware support.

So the more I think about it the more I like it.

Protocol/Convention vs Homeautomation profiles

Just like I tried to define device profiles based on the homie convention I would also suggest also to keep this split up as well. The base protocol/convention should be more or less agnostic of the usage scenario.
I think @ogghst's comment is very good reminder (at least for sometimes single minded persons like me) that the main focus of the convention is HA. From what I understand your post ogghst you are trying to build something more on an industrial scale level, right? This is something to keep in mind for design IMHO.

Basically I agree with the summary in @Tieske first comment about the high level design aspects:

• have a service bus
• define convention/protocol
• define device profiles

I am not sure about the "state only" approach and not having any RPC options at all. There could be advanced functions/capability you can not design very cleanly with state only approach or you might not want to do (e.g. devices with features that cannot be reflected as a state change, multi step interactions that require context, bigger sets of metadata that are only required rarely could be on a request only basis)

@Tieske wrote:

I guess the million dollar question is; are we willing to work on a new version?

I am willing to do so, yes. I think the question is how to we get organized and how much time do people have?

[ogghst]

@schaze I am a project manager in my real life :) and I am used to wrap up and assign tasks everyday.

So here this is my take: need is a protocol that

• is derived from other 'pieces'. Mqtt + protobuf/json is probably the best match. This to make easy to integrate in existing systems
• has strong but expandable data typing. I can see standard ones (int, string) but possible to define complex ones (hvac state, calendar, media controls..). This is the why of protobuf/json payloads and typing
• pluggable: like homie extension, one can define his own types structures, validators, processes like firmware upgrade. There should be a mechanism like json schema to retrieve from web the way to marshall/unmarshall data
• readable: there must be a simple way to read data from mqtt messages

Feel free to revise and add/remove requirements, and maybe could be a good idea to write down a list in some organized form, but if you are in agreement with that the first question is: are we talking about sparkplugb? I suggest you all to look at that protocol as seems to me we are really close to it.

As said before, I will investigate in using protobuf in C as it is THE microcontroller language but the implementation is not directly supported. This is the task I am assigning to myself. Your turn :)

[ogghst]

Homie would provide a fixed subset of commands for things like OTA Updates, device profiles like light switches, etc and a general request/reply mechanism that would support streaming results. Not to mention that homie should really be focused on HA and thus accept contributions a lot easier.

Agree. I'll investigate better on sparkplugb as I was involved a couple of years ago on it. My idea is starting from there to see how it can be expanded/modified to suit HA needs. Main concern is how it can be expandable with custom data format and item hierarchy. I'll check it with some use cases, like describing a whole room with several sensors or a weird device like vacuum robot. I suggest to challenge ourselves in making sure we are not doing something already possible with some other open protocols.

I have done that before. You need a couple kB of flash to make it work, but there are multiple projects that provide (de-) serialization for embedded targets.

Thanks. I have forgotten this implementation!
.

[ogghst]

I have started a personal notebook where I am starting to collect requirements. feel free to view/edit as you want. I usually start from use cases and try to avoid removing technical aspects as much as I can here, while still exploring protobuf. This is my personal 'top-cross-bottom' approach :D
https://docs.google.com/spreadsheets/d/1zwcJ54LA31pcJZ_Hc8ANlXqmqGOfrOHupxsWKEjduic/edit?usp=sharing

[Tieske]

So that was a lot of reading... 😄

Had a quick look at sparkplugb, looks indeed similar, but lacks device profiles (didn't see anything related). I also think that sparkplugb is large scale industrial, where Homie is small scale entry-level.

That immediately brings me to protobuf; I really dislike binary formats, exactly for all the reasons mentioned above. If Homie is to be entry-level, and easy to adopt, then the logical choice would be json imho. Easy to read and debug, no additional tooling required, and support is everywhere.

The only downside on json is the size, I have no ESP experience, but if the devices are able to work with current number of Homie topics according to 3.x/4.x spec, then I don't think this will be a problem? If you're Google scale, then the bytes saved add up, but for mere mortals like us (at least me) the added complexity makes no sense.
(assuming json to only be used for complex structured data, or device descriptions, which I assume ESP's are not carrying a lot of)

[Thalhammer]

The only downside on json is the size

Imho there are multiple issues with json, size of the document being one of the smallest.

• Documentation:
  Json has Json Schema, however it is (IMHO) way to cumbersome and not really readable. Consider the following comparison as an example, both of which encode exactly the same message. This is obviously subjective, but I'll just let the example speak for it self.

{
  "$id": "https://example.com/person.schema.json",
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "title": "Person",
  "type": "object",
  "properties": {
    "firstName": {
      "type": "string",
      "description": "The person's first name."
    },
    "lastName": {
      "type": "string",
      "description": "The person's last name."
    },
    "age": {
      "description": "Age in years which must be equal to or greater than zero.",
      "type": "integer",
      "minimum": 0
    }
  }
}

vs

message Person {
     optional string firstName = 1; // The person's first name.
     optional string lastName = 2; // The person's last name.
     optional uint32 age = 3; // Age in years
}

• Codegen
  Making the spec/types machine readable is a fairly important thing. We have this problem rn (the online validator is handcoded and incredibly incomplete because the spec is not machine readable) and implementing custom codegen for json schema is fairly hard in my experience whereas its trivial for protobuf.

• Efficiency
  Even this minimal example protobuf beats the minified json by a factor of 3.5x (46bytes vs 13 bytes, 20 if you base64 encode the protobuf). Not that the important thing here is not the capability of an mcu being able to handle it, but rather the bandwith cost. Cellular IOT cards typically have a very small data allowance (e.g. 1nce provides 500M for 10 years (or roughly 144Kb per day). A factor of 3.5x can be major in those cases (though I agree that in the main use cases of homie its not an issue).

{
  "firstName": "John",
  "lastName": "Doe",
  "age": 21
}

• Performance
  Protobuf encoding/decoding is a lot faster than json, to the point where it beats the native json support in browser using js. This can be a considerable factor for controller implementations as well.

• Free Forwards / Backwards compatibility
  Protobuf is both forwards and backwards compatible meaning you can add and remove and rename most fields without breaking any cosumers. In addition there are options to reserve ranges of ID's e.g. for official features and similar things.

• Service description
  Might come in handy if we plan to support a general request/response scheme.

Thats just my subjective view though, but imho theres very little benefit of json (apart from the readability, which is trivial to solve). Its biggest benefit is probably the thing you dislike the most: A very powerful codegen. That said you shouldn't ever come in touch with the codegen yourself, it should be handled by client libraries and tools.

EDIT: As a side note, I merged the two website repos, switched to github's CI and fixed a bunch of issues that where brought up over the years, so we should be good to go when needed on this front.

[Tieske]

let's save the format discussion for later. I think we have some 'process' to set up first.

I've seen @ogghst list. That sort of seems to start from scratch. Not saying we shouldn't, just wondering. Hence my proposal would be;

1. specify the design goals; who is this for, what problems does it solve, (what problems won't it solve)
2. create a list of current issues we see, as well as current key-elements to retain
3. at some point decide on how to move forward; incremental or from scratch

wdyt?

[Tieske]

As a side note, I merged the two website repos, switched to github's CI and fixed a bunch of issues that where brought up over the years, so we should be good to go when needed on this front.

👍 👍 👍 👍 👍

[ogghst]

let's save the format discussion for later. I think we have some 'process' to set up first.

I've seen @ogghst list. That sort of seems to start from scratch. Not saying we shouldn't, just wondering. Hence my proposal would be;

1. specify the design goals; who is this for, what problems does it solve, (what problems won't it solve)
2. create a list of current issues we see, as well as current key-elements to retain
3. at some point decide on how to move forward; incremental or from scratch

wdyt?

Agree. My list is the point of view of a guy that just joined here. I usually start from use cases when defining a solution. You guys are way more experienced and feel free to not use it. I just suggest to use something more structured than this topic, in order to not waste time and effort in reading tons of discussion, that are needed but a sort of wrap up doc it is as well. I am open for whatever you are used to, and open for a step back if you think i'm just adding noise :)

[schaze]

let's save the format discussion for later. I think we have some 'process' to set up first.

I've seen @ogghst list. That sort of seems to start from scratch. Not saying we shouldn't, just wondering. Hence my proposal would be;

1. specify the design goals; who is this for, what problems does it solve, (what problems won't it solve)
2. create a list of current issues we see, as well as current key-elements to retain
3. at some point decide on how to move forward; incremental or from scratch

wdyt?

I agree as well.
Let's follow this list. I do however think that a set of usecases for point 1 would improve the discussion and ensure the same understanding in context between us. We might need some collaboration space, google docs can work but is maybe not ideal. I have no experience in the open source world what works here...
I will have a pretty full week so I probably won't be able to be very active this week - but I will use the free time to do some thinking and maybe write down all my use cases I currently have and some design goals I have in my head.

[ogghst]

Let's follow this list. I do however think that a set of usecases for point 1 would improve the discussion and ensure the same understanding in context between us. We might need some collaboration space, google docs can work but is maybe not ideal. I have no experience in the open source world what works here...

What paid software have you in your mind? Usually there is an open source counterpart that does 80% of the job :)

I will have a pretty full week so I probably won't be able to be very active this week - but I will use the free time to do some thinking and maybe write down all my use cases I currently have and some design goals I have in my head.

That's why this draft helps, I am in Chicago for work this week (I am Italian) and I do not have a lot of time as well. Updating on my freetime.

[ogghst]

I think he meant that he's not familiar as to what products opensource projects usually use for working together. Not that he wants to use a paid software to do so.

Revised question: if you are used with some paid software (eg teams, jira etc) let us know so maybe we can find the equivalent approach in opensource world. That's it :)

[Tieske]

no real experience with it, but github projects is close by I guess. For quick comms I have a preference for Slack, since I use that a lot already.

[ogghst]

Guys, I'm back at home, looking for a possible serialization protocol as I found protobuf lacking some of the features I would like to implement - like expanding custom data structure definitions upon needs. I jump into this interesting article that seems to cover pros and cons of different solutions: https://www.researchgate.net/publication/352969331_PSON_A_Serialization_Format_for_IoT_Sensor_Networks

I would like to explore PSON and Avro in next days. In parallel I am still collecting requirements and usecases in my offline notebook, I will publish it soon.

[ogghst]

thanks, my feedback is added.

to collect all info and ideas I've migrated use cases and all verbose stuff in a google doc. address is in the first sheet of the spreadsheet to avoid public links and limit the editors to who is really interested on :)

I'm working in several actions in parallel, reviewing multiple protocols and libraries that can be considered similar to this. I need to make a step back and reconsider Sparkplug B, as per today is the one that fits most of the requirements and is well recognized by industry. It has a big drawback: it need code generation and most of its payloads have to be converted to be human readable. Both ones can be overcomed with a cost of little overhead for a beginner user.

Why not directly using Sparkplug B? because it 'merely' describes devices and metrics. Assumption is that users knows devices and what to do with their data. In Smart Home and in general in loose coupled system this is not true. Hubs like Home Assistant or Google Home needs to know that a device is a HVAC system or a light to classify it and adapt their behavior.

So a possible strategy is to 'extend' Sparkplug B protocol to add Smart Home features. Seems to be possible at a first sight. Idea is to develop specific messages in addition to the standard ones. Sparkplug B ecosystem will still see Homie devices, but Homie enabled devices will also be capable of managing aspects like OTA update, Time Sync, context-aware operations (like turning on all lights in the house) without almost no user configuration.

[ogghst]

Unlike something like JSON you can simply concat protobuf messages.

Awesome. that would be my preferred choice, if the idea of 'wrapping' sparkplugb protocol works. I also see that sparkplugb already provide an extension mechanism like (extract from their .proto file)

        message Row {
            repeated DataSetValue elements  = 1;
            extensions                      2 to max;   // For third party extensions
        }

For official extensions we could also do something like this:

This could be the choice for our messages, but if the above works, it's still my preferred, although I don't clearly see pros and cons in generated code.

I am not a friend of {J,B,P}SON cause it will put the field names into every single message which is simply useless overhead, since the structure of the message is specified by the convention and thus static. If you dont know the convention for a specific feature you can't use the data regardless if its a named field or a number .

PSON from what I see is less demanding in term of code footprint and resource utilization. But seems to lack a serious implementation in microcontrollers, where protobuf is very focused. Need to take some time for a couple of tests.

[ogghst]

The more i describe this new spec the more I am convinced that with sparkplugb we can have more or less everything already set. Just define extensions and smart home concepts into properties and metrics and everything should work without even touching proto files.

If that works it should be really straightforward to implement libraries for home controllers and devices in several languages.

[ogghst]

I have put some notes in the specification over a sparkplug-based implementation and starting to put down some ideas (for now just little bit more than initial project declaration) on a java implementation here: sparkplugHA

[Thalhammer]

Hope every body is fine with that.

Fine probably yes, though finding a timeslot that fits everyone might get interesting ;)

[ogghst]

I am usually available from 7pm italian time. Just a couple of days of notice is enough.

[schaze]

Hi guys, sorry for the long pause - I have been traveling in most of my free time the last two weeks. We have a long weekend coming up this week (monday is a bank holiday) - I will add my input to the sheet above then.

I also think a call would make the most sense to get a better discussion going. I have had some time think about the proposed solution approaches and tbh I am torn between doing some simple improvements on the existing protocol (e.g. move device/node/property definitions to a single topic, extend on existing datatypes, having one device status field and maybe think about the node/property structure in general) and going for a complete change like @ogghst is going for with the "adopted" sparkplugb approach...

[ogghst]

Just one update, still working on a draft of my sparkplugb 'decorator' convention, working now on a test case of the features like ota, that is basically the core of this convention. I think I will have something working in one week.

I see huge advantages on this approach, 90% of the specs will be covered and mantained by spb, and every spb controller can still read sensor data.

Also, I think the approach I am using is quite similar to what you did for Homie. Just extending an existing protocol maintaining compatibility. But in order to add semantic flexibility I have just moved the extension level from MQTT topics to MQTT payload.

[schaze]

I have added my comments in the spread sheet. I though a long while but there are no real design goals I would add. IMHO you already put in everything relevant for the start.
@ogghst : I read through your document as well - it approaches the topic quite differently like I had in my mind coming from current homie currently - but to get a better understanding I think will need to read more sparkplugb docs first I believe :).

@Tieske @Thalhammer @ogghst @ThomDietrich @jamesmyatt (@ whomever I forgot to tag):
I think we do need a call to talk and get to a common understanding.
As we have a bank holiday on Monday, I simply propose Monday October 03rd, 20:00 CET - from what I can assume of the participants list, timezones should be close enough (UK, Italy, Denmark, Germany)

[ogghst]

Ok for me on monday oct 3. I have a 5 month old child so I cannot spend too much time here :( 30-45 min max.

On my side I'll try to explain the reasons behind my choice and a few insights on the convention I am testing.

[Thalhammer]

I simply propose Monday October 03rd, 20:00 CET

Sounds good for me.
Poor @ whomever :( Getting pinged on random issues.

@schaze Do you want to do the honors of setting up a zoom/jitsi/teams room ?

[schaze]

As I neither have zoom not teams privately, I choose jitsi (no account needed):
This is the participant link: https://meet.jit.si/moderated/5c852733f7367bc19bc53d9bd72f435237416a20ef4c6c9c74a250cc3b346e21

[Tieske]

Time works for me

[Tieske]

Had a look at some options for more interactive chats. Slack has a free plan, but that has only 90 days retention. So useless.

alternatives would be https://gitter.im or https://app.element.io/ preferences?

[schaze]

Had a look at some options for more interactive chats. Slack has a free plan, but that has only 90 days retention. So useless.

alternatives would be https://gitter.im or https://app.element.io/ preferences?

I know neither so I have no preference...
I just saw that github does also have a discussion feature in the teams section (Thalhammer sent me an invite), maybe we can simply use that? No idea however if it is any good:

[Thalhammer]

alternatives would be https://gitter.im/ or https://app.element.io/ preferences?

I'd prefer app.element.io (Which uses the matrix network) cause I already have an matrix account, however as long as there's a web client I don't really care.

I just saw that github does also have a discussion feature in the teams section

Assuming you want something faster I don't think its an option, cause its pretty much just a global issue board, (i.e. not "realtime"). That said there are lots of other networks that are low to now effort for registration.

[Tieske]

So seems there already is a Gitter one; https://gitter.im/homie-iot/convention (web client)

Let's use that for now, if it doesn't work satisfactory, then we switch to Element. Agree with Thalhammer that the github discussions are very "issue" like.

[Tieske]

I invited @Thalhammer and @ogghst , but couldn't add @schaze for some reason, so please sign up yourself (can use oauth2 with Github)