Settable property indication about convention

[ThomDietrich]

I do agree, that announcing unit, datatype and range of one property should be standardized in the homie convention. In fact I feel like this would be one of the important details of a convention thriving to become a discoverable IoTs-on-MQTT convention.
I would choose another approach than defining a mixed string, which is both hard to parse and easy to mistake and on top of that does not use what MQTT has to offer.

My proposal:

• Currently all device properties of one device are announced using the topics
  homie/device-id/$...
• Similarly all node properties of one node are announced using the topics
  homie/device-id/node-id/$...
• Consequently all "property properties" of one property could be announced using the topics
  homie/device-id/node-id/property-id/$...

Side note: Using the word "properties" in this context might have been a bad idea...

Property properties	Description
$name	pretty name (optional)
$settable	true or false. true: Property can be set using the .../property-id/set topic
$unit	A string representing the unit of the property, e.g. "degree Celsius", "kW", ""
$datatype	integer, float, boolean, string, enum, ....
$range	from:to or ON,OFF,UNDEF or regex: ^(?:[0-9]{1,3}\.){3}[0-9]{1,3}$
$valid	default: "true" or empty. Set to "false" to indicate that this property is not updated currently. (Might be important because of retained messages. Just an example for further possibilities)

The table above is just an example and open for discussion.

---

Additionally I would suggest to introduce for completeness:

• homie/device-id/$nodes
• homie/device-id/node-id/$properties (already defined)
• homie/device-id/node-id/property-id/$attributes

[marvinroger]

I would even replace the [] with (), to differentiate ranges and data types:

homie/686f6d6965/ledstrip/$properties → led[1-3]:settable(enum:true,false)

[marvinroger]

Okay, let's recap with a sample environment device.

Sample

Device properties

homie/686f6d6965/$homie → 2.1.0
homie/686f6d6965/$online → true
homie/686f6d6965/$name → Bedroom environment device
homie/686f6d6965/$localip → 192.168.0.10
homie/686f6d6965/$mac → A1:B2:C3:D4:E5:F6
homie/686f6d6965/$nodes → temperature,weather
homie/686f6d6965/$stats/uptime → 12
homie/686f6d6965/$stats/signal → 72
homie/686f6d6965/$stats/interval → 6
homie/686f6d6965/$fw/name → Firmware
homie/686f6d6965/$fw/version → 1.0.0
homie/686f6d6965/$fw/checksum → 6057f13c496ecf7fd777ceb9e79ae285
homie/686f6d6965/$implementation → esp8266
homie/686f6d6965/$implementation/foo → bar

Node properties (and property properties)

temperature node

homie/686f6d6965/temperature/$type → temperature
homie/686f6d6965/temperature/$properties → degrees

homie/686f6d6965/temperature/degrees/$settable → false
homie/686f6d6965/temperature/degrees/$unit → Celcius
homie/686f6d6965/temperature/degrees/$datatype → float
homie/686f6d6965/temperature/degrees/$range → -10:50
homie/686f6d6965/temperature/degrees/$valid (to be clarified)

weather node

homie/686f6d6965/temperature/$type →  weather
homie/686f6d6965/temperature/$properties → condition

homie/686f6d6965/temperature/condition/$settable → false
homie/686f6d6965/temperature/condition/$unit →
homie/686f6d6965/temperature/condition/$datatype → enum
homie/686f6d6965/temperature/condition/$range → sunny,cloudy,windy,rainy,stormy,foggy,showy
homie/686f6d6965/temperature/condition/$valid (to be clarified)

I like this approach.

Range for each datatypes

• integer, float: from:to e.g. -23:45
• boolean: none
• enum: a,b,c and , escaped by doubling ,,
• string: either none or size:min-max (e.g. size:1-3) or regex:^[A-Fa-f0-9]{32}$

---

I am not sure about the $valid and homie/device-id/node-id/property-id/$attributes topics. Can you clarify their purpose?

[ThomDietrich]

Hey @marvinroger great to get in touch with you ;)

Your examples look great to me!

After going through all possible use cases, I realized that the $valid property is not meaningful. A property not valid will simply not be announced via $properties and a retained message can be deleted in that case.

You can also ignore homie/device-id/node-id/property-id/$attributes. It was supposed the be the analogy of $nodes but a property doesn't have sub-elements like the node has properties.

---

The discovery process

The whole idea behind this definition is to make a device discoverable. How would the discovery actually be implemented?

1. The discovery service doesn't know the device-id of new devices
   -> Will it need to subscribe to homie/+/$online or similar?
2. Next it will subscribe to homie/device-id/$..., including $nodes
3. Next it will subscribe to homie/device-id/node-id/$... for all nodes, including $properties
4. Lastly it will subscribe to homie/device-id/node-id/property/$... for all properties

It might be beneficial to provide a better way to solve step 1.

Steps 2,3,4 are only possible if all messages are retained messages. Otherwise, this will create a race condition.

Would you agree with this description? Is anything missing? @Kwave ?

---

If you want to go forward with this approach, maybe it would be good to choose a new word besides "properties". Topics starting with "$" could be called "attributes" for example.

Best Regards!

[marvinroger]

@ThomDietrich it's a pleasure too, sorry for the long response time, I was busy with school!

Great then. For everyone who read this, please use the +1 and -1 GitHub reactions so we can get your feedback. And of course, feel free to comment if you would like to enter the discussion!

---

For the discovery, I've taken a different approach for one of my projects (see https://github.com/INTECH-RGB/homie-dashboard/blob/master/server/lib/bridges/mqtt-infrastructure.js).

Let's imagine you have the representation of your infrastructure in a structure (JSON for the example):

{}

Basically, we subscribe to homie/#. The problem is we receive the messages with no specific order. Imagine we receive a homie/686f6d6965/temperature/degrees/$datatype → float. We would parse the topic and know that we are receiving the datatype attribute of the degrees property from the temperature node of the 686f6d6965 device. We see the device does not exist in the representation, so we would add it:

{
  "686f6d6965": {
    "nodes": {}
  }
}

Then, we would see that the node does not exist, that the property does not exist, and so on... To end up with:

{
  "686f6d6965": {
    "nodes": {
      "temperature": {
        "properties": {
          "degrees": {
            "datatype": "float"
          }
        }
      }
    }
  }
}

And a device is considered discovered when all required information are populated in the representation. I hope you get the idea?

[marvinroger]

I do agree!

Your suggestion handles the following:

• Enums
• Integer ranges

What about strings (which can be anything, not only enums), floats, more specific types like colors (255,255,255) etc.?

I would rather define data types. Each data types could have optional parameters (like min, max...). Something like:

homie/686f6d6965/switch/$properties → on:settable[enum:true,false]
homie/686f6d6965/temperature/$properties → temperature:settable[integer:60,100]

What do you think?

[sd-zhang]

Good thinking with data types. I completely agree with what you suggested.

Strings should be relatively simple to handle. For example, setting a color using predefined names can be done like this:
homie/686f6d6965/ledstrip/$properties → led[1-3]:settable(string)
homie/686f6d6965/ledstrip/led_1/set ← red

Multivariable settings are slightly tougher, but I think using a separator should be able to handle it, like so:
homie/686f6d6965/ledstrip/$properties → led[1-3]:settable(integer: 0,255; 0, 255; 0,255)

This would indicate that there are three settable integers for this property, each within the range [0, 255]. The color can be set like this:
homie/686f6d6965/ledstrip/led_1/set ← 0, 127, 255

[gorootde]

I've implemented the Openhab 2 Homie binding, that is why I also came across this point @ThomDietrich. It's rather difficult to map the properties to datatypes - or even a meaningful UI representation. Your suggestion would allow to do so.

[ThomDietrich]

@Kwave I've taken a quick look at your feature branch. Would you agree, that it would be beneficial if another "Property property" would be $category holding one of the defined ESH channel categories. $settable already covers Accessible Mode and the exact ESH Item type to use can probably be derived from $unit+$datatype+$range.
What do you think?

[gorootde]

@ThomDietrich I'am not sure if data representation should be part of the homie specification. It depends on how you define the term "Node":

1. If "Homie-Node" is defined as a sensor that is connected to a "Homie-Device", we definitely need the $category topic filled with the ESH channel category. We even need much more topics to indicate whether the sensor is "online" (works as expected) and so on.
2. If "Homie-Node" is defined as a property of the "Homie-Device" with some additional meta information (e.g. the temperature example in the convention), you don't need any additional topics at all. The $type topic may contain the ESH category, whereas the unit and other meta-information are transported via subtopics.

However I'am much more attracted by the $-meta information solution than overloading the $properties topic with complicated syntax (which is not very mqtt'ish). Adding [] or () to the property list introduces a structure to the message, instead of using the already present mechanism to structure topics.

[ThomDietrich]

Hello @Kwave,
My suggestion was not directly aimed at ESH but rather follows along the idea to make the Homie Convention base for fully discoverable devices. Defining the "category" of a property might help discovering systems like ESH/OH to configure and present properties (i.e. items in ESH/OH) in the desired way. Hence it might be beneficial to additionally/optionally add "$category" to the "Property properties" example.

I'am not sure if data representation should be part of the homie specification.

If the abstraction is (as in this case) high enough, I do not see any issue with that. Remember, that I asked you if your OH binding would benefit from the category. If that is not the case, we can just as well leave it out.

It would be great to hear some thoughts from @marvinroger. How do you feel about the presented approach and how realistic is an implementation in that direction?

Best Regards! Thomas

[ThomDietrich]

Get it, like it ;)
Reminds me of the approach taken here.

Makes me speculate if it might be easier to streamline everything a bit to begin with. How would you feel about all the above (all $attributes), but serialized in one message?

home/$announce ← {"686f6d6965": {"$online": "true", "$nodes": {"temperature", "weather"}, ...}}

[marvinroger]

It would not help. A discovery service must be able to discover devices even after the devices are powered up. So "one-shot" announces on homie/$announce would not work, the messages would need to be retained on homie/{deviceid}/$announce topics. But then, the problem remains: the retained messages might arrive out-of-order, so nothing guarantees that the announces will arrive before every other messages, making announces useless.

[ThomDietrich]

Oh, you are totally right with the first part! Of course each device needs it's own retained topic (I would suggest to turn it around).

I did not get the second part of your argument though. The one retained message on homie/$announce/686f6d6965 would contain all "attributes" needed to instantly fully "discover" one device. No need to listen to any other topic for that. The discovery service simply needs to subscribe to homie/$announce/+. It will get to know all previously registered as well as new devices instantly.

All the above assumes, that the goal here is to make all variables and capabilities of on device known through an announcement process upon mqtt connection rather than step by step while its operation...

[marvinroger]

Oh okay, I thought you wanted to keep the current structure of the messages, only announcing through $announce the nodes and properties (not the attributes), so a discovery service could map a device knowing all its nodes and properties, before populating with actual Homie attributes.

You said earlier, and I totally agree with that:

I would choose another approach than defining a mixed string, which is both hard to parse and easy to mistake and on top of that does not use what MQTT has to offer

Your suggestion was basically to exploit the potential of MQTT by making use of semantic topics rather that one "fit-it-all" topic. Your JSON based announce does not follow this approach. Moreover, JSON is pretty costly in terms of memory needed on micro-controllers (ESP8266 for example) which is why we avoided any JSON in the payloads.

[ThomDietrich]

JSON is pretty costly in terms of memory needed on micro-controllers

Excellent point.
Yes I'm still convinced of the MQTT hierarchy approach. However, for transparency I thought it would only be fair to mention the other possibilities. At the end of the day the goal should be to allow discovery services to have full knowledge about the capabilities of a device in a deterministic way (I believe we do agree? 😉). One full JSON message would accomplish that, so would a set of nested topics messages as we've discussed before.

I thought you wanted to keep the current structure of the messages, only announcing through $announce the nodes and properties (not the attributes), so a discovery service could map a device knowing all its nodes and properties, before populating with actual Homie attributes.

Not necessarily. This is a mixture of both. It's not a full description and needs further messages to make a full discovery possible. Do you see any benefits in this?

This is a very interesting discussion! -Thomas

[ThomDietrich]

@marvinroger you want me to recap and post an updated definition? I think we've settled all the open questions.

[marvinroger]

@ThomDietrich my bad once again, I am currently looking for an internship in Paris and it takes a lot of time!

Yes, please go ahead! And if anybody has other advices, let's discuss. 😉

[gorootde]

@marvinroger @ThomDietrich I've created a Pull Request that sums up the discussed items. Please comment in the PR if I did get anything wrong.

[marvinroger]

As the @Kwave's PR implements a lot of what was being discussed here, let's continue in #28