Extensibility about esphome

[OttoWinter]

@yawor No this is not limited to the sensor component. There will be custom platforms for all components eventually. But it will have a separate config schema for each component type. This is because the whole reason behind this integration is to have better interop between yaml and

For example for sensors, this will allow you to write the exciting and more complicated bits in C++ and then register this sensor into esphomeyaml and use all the triggers/configuration variables. For example like this:

sensor:
- platform: custom
  lambda: |-
    auto my_sensor = new MyCustomSensor();
    App.register_component(my_sensor);
    return {my_sensor};

  sensors:
  - name: "My Custom Sensor"
    on_value:
      - do something
    icon: mdi:thermometer
    unit_of_measurement: cd
    # ...

---

For other use cases there will also be a custom top-level section where users can put all their custom components. For them, esphomeyaml won't do anything other than embedding the component into the code.

Also, there will be a CustomMQTTComponent for use-cases where you want to communicate with MQTT but the base types don't handle your needs (for example covers with positions). The syntax there will be a bit like this:

class MyCustomComp : public CustomMQTTComponent {
 public:
  void setup() {
    subscribe("the/topic", &MyCustomComp::on_message);
    subscribe_json("the/topic", &MyCustomComp::on_json_message);
  }

  void on_message(const std::string &topic, const std::string &payload) {
    publish("the/other/topic", "hello world");
  }

  void on_json_message(const std::string &topic, JsonObject &root) {
    int value = root["int"];
  } 
};

I've uploaded a bunch of the changes now (most under branch names like custom-sensor etc). These branches more or less reflect my current dev environment (so not production code!).

[OttoWinter]

Also, for jinja2: I mean in the template you gave there there's no real reason to use jinja2 I can see. If it's only two static blocks that are changed then we could just as well have a template like this:

// Auto generated code by esphomeyaml
#include "esphomelib/application.h"
// AUTO-GENERATED INCLUDE BEGIN
// AUTO-GENERATED INCLUDE END

using namespace esphomelib;

void setup() {
  // AUTO-GENERATED SETUP BEGIN
  // AUTO-GENERATED SETUP END
  App.setup();
}

void loop() {
  App.loop();
}

The user could then modify this template file for their needs. The difference to the current system would be that esphomeyaml re-reads the template in each time instead, of modifying the "template" in place as it does right now. But either way I don't think this will be as much of a problem in 1.10 to re-do this to get to this type of templating.

Also, I personally really don't like jinja2 that much. Yes it's concise but I end up having to look up how to do something every single time.

The other big red flag for me is that any C++ syntax highlighters would pretty much freak out at this. I don't know about you, but I definitely prefer coding in powerful IDEs with syntax highlighting with source lookup etc.

[OttoWinter]

So the reason the ID concept exists in esphomeyaml is because of the order of code in the generated output.

For example, if you have three outputs in your YAML and want to create an RGB light out of it, esphomeyaml must declare the output variables first, otherwise the C++ compiler would create a "usage before declaration" error.

"Only log a warning, instead of raising an error, when an ID isn't found. " is not really possible in esphomeyaml either because there's more information stored in an ID object in esphomeyaml (for example whether it's a pointer, what type it has and so on). These things are populated automatically in the generated output

Allow registering custom components to be used in YAML, just like the built-in lights, outputs, sensors etc.

That sounds better to me. For example like this:

sensor:
  - platform: custom
    construct: >-
      return new MySensor("My Sensor Name", my_arg);
    id: id_of_my_custom_sensor

[TuurDutoit]

Exactly, that would make it a lot easier.
This would require importing the right libraries (or your own header files) in the main.cpp file, right? I think that would be a clean and simple solution.

[yawor]

Modifying platformio.ini or main.cpp files to add custom components is not a great idea. If I would like to share my custom component or even just keep it in git, then I would need to remove all my passwords and other personal info (like ssid, mqtt params etc) before committing code into git. And keeping files in git which are generated is a bad idea in general.

I like the idea of providing platform custom. It would be also nice to have the ability to define extra dependencies and includes directly in yaml config, instead of changing the platformio.ini file after first compilation.
For example:

esphomeyaml:
  dependencies:
    - extradependency1
    - extradependency2
  includes:
    - someinclude1.h
    - someinclude2.h

The dependencies would be added to lib_deps in the platformio.ini and includes would be added as #include directive in main.cpp.
This way we could add another github link with our own custom components and then use the custom sensor platform to register it.
This also helps with keeping everything in git. All secrets are stored in git-ignored secrets.yaml and everything needed is in the yaml file under git.

[OttoWinter]

@yawor Yes I've spent the last couple of days since the 1.9.0 release implementing something similar to that. And most of it is pretty straightforward except for one part: "includes would be added as #include directive in main.cpp"

After this change I want to have main.cpp remain editable. The problem with main.cpp is that there is currently only one place esphomeyaml can change the code in the setup function (inside the AUTO GENERATED CODE STARTS HERE comments).

So having a new section at the top where #include s are added would be a breaking change. I'm working on a migration tool to make this at least not a breaking change for most users, but it's getting pretty ugly.

And for the #include s, I've settled on just copying them into the main.cpp file instead of adding #include lines because then relative paths wouldn't work. In the end, that's all an include does anyway: copying the file contents as-is

[yawor]

Well, backward compatibility is always a two-edge sword. It's nice when it works but can really complicate things. Regarding the autogenerated code, I like what STMicro have done in their CubeMX software. Instead of marking areas in the file where content is autogenerated, they've marked areas where user can place their code (they have them a lot). This approach allows you to insert new user editable areas in the future by just generating a new empty user editable area if it wasn't present previously.
So maybe it would be good to adopt this approach if you're thinking about writing a migration tool.

Like I mentioned in my previous comment, my main issue with adding stuff manually into main.cpp is that main.cpp has all the secrets in clear text so it's unsuitable for committing into git, especially into public repositories.

[yawor]

To extend on my previous comment, here's how the CubeMX's generated code marks user areas:

/* USER CODE BEGIN Includes */
/* USER CODE END Includes */

Every area is marked with USER CODE BEGIN and USER CODE END with a unique identifier of the area. I think this also makes generating code easier. If file exists, fetch all user code into a dict with keys being area identifiers, then start generating the code and insert user codes where appropriate.

--edit

If possible, I would also move all private information for WiFi and MQTT into a separate header file. Maybe secrets.h? It could be generated from the secrets.yaml and then used where !secret name is used in the main yaml file. That way the main.cpp file could be safely committed even into a public repository.
Or, if possible, define each secret only during compilation by dynamically passing -DSECRET_nameofthesecret=xxxx to compiler.

[yawor]

Sorry for being so stubborn about this :). I have an idea on how to resolve the issue without the need to write a migration tool for main.cpp.
My idea is to introduce a file versioning by adding a marker comment on top of the main.cpp file. Something like (just an example):

/* esphomeyaml main.cpp 2.0 */

When compiling a yaml config, the esphomeyaml would need to check for the version marker. No marker would mean a legacy file (current format). It would just regenerate in the same format without any new features, with a warning to the user that the format has changed and maybe a link to more information on that subject and how to migrate. If a user has modified main.cpp it means that they knew what they were doing so it shouldn't be that hard for the user to migrate their code. If user uses an attribute in the yaml file which requires new format but they have a legacy main.cpp, the program could exit with an error with appropriate information.
I would propose to use the user code marking format which I've mentioned in my previous comments as a new format, with multiple user code areas from the start. For example:

• Includes - right after generated includes
• Declarations - function prototypes and variables declarations - after Includes - could be merged with Includes, but I'd leave it as another section, in case you'd want to add something from the generator in between
• Pre-Setup - at the beginning of the setup function, before generated code
• Post-Setup - at the end of the setup function, after generated code
• Functions - user functions implementations - between setup and loop functions
• Loop - a section inside the loop function - after App.loop(), but before delay(16).

I know it could increase the complexity of the code generator because of the need to support two file formats, but it may be worth it. You keep the backward compatibility to some degree and if users want to use new features then they need to migrate their code themselves - they probably know the best how to restructure their own code.

BTW have you thought about using some python template engine for code generation? Maybe jinja2? It's not strictly for generating HTML. I've used it myself to generate code before and HA uses jinja2 as a configuration template engine too. Having main.cpp template as a jinja2 template could also give more possibilities like template override from the yaml file. The built-in template could even expose some overridable blocks. User could then extend the original template and use blocks to insert their own code.

--edit

I think that template + override option could even be better than modifying main.cpp. Just expose the areas I've mentioned above as empty blocks in the template and let user extend the template and put their code in these blocks.

[OttoWinter]

Yes that's kind of how I've implemented it too:

For each file (for example livingroom.yaml in folder /config/), esphomeyaml will create a JSON file in a directory .esphomeyaml (for example /config/.esphomeyaml/livingroom.yaml.json) with some basic metadata about the node. This JSON currently looks like this:

{
  "build_path": "/config/livingroom", 
  "esphomelib_version": {
    "repository": "https://github.com/OttoWinter/esphomelib.git",
    "branch": "dev",
  }, 
  "name": "livingroom", 
  "esp_platform": "ESP32", 
  "address": "192.168.178.207", 
  "firmware_bin_path": "/config/livingroom/.pioenvs/livingroom/firmware.bin", 
  "arduino_version": "https://github.com/platformio/platform-espressif32.git#feature/stage", 
  "src_version": 1, 
  "storage_version": 1, 
  "board": "nodemcu-32s"
}

(might still change until I release it). The src_version property here describes what version the main.cpp file is in. The reason the source version is not directly in the main.cpp file is because I needed to create this metadata storage anyhow, so that the dashboard interface can do things like showing status of the node using pings etc.

The auto-migrated cpp will look like this:

// Auto generated code by esphomeyaml
// ========== AUTO GENERATED INCLUDE BLOCK BEGIN ===========
#include "esphomelib/application.h"
using namespace esphomelib;

...
// ========== AUTO GENERATED INCLUDE BLOCK END ===========

void setup() {
  // ===== DO NOT EDIT ANYTHING BELOW THIS LINE =====
  // ========== AUTO GENERATED CODE BEGIN ===========
  ...
  // =========== AUTO GENERATED CODE END ============
  // ========= YOU CAN EDIT AFTER THIS LINE =========

  App.setup();
}

void loop() {
  App.loop();
}

Re CubeMX and reversing the whole editable areas: I've never heard of that product before but it certainly sounds interesting. For the user editable areas: I think the two sections, one for includes and top-level definitions and one for setup is enough. At least I don't see a reason why esphomeyaml would need to write in any other area. So a) the addition of an include block makes migration code simpler and b) it leaves more freedom to the user.

If possible, I would also move all private information for WiFi and MQTT into a separate header file. Maybe secrets.h? It could be generated from the secrets.yaml and then used where !secret name is used in the main yaml file. That way the main.cpp file could be safely committed even into a public repository.

Well this would be kind of hard using the current C++ generator. I'm assuming you want to edit main.cpp but not have it checked out in git? Well there's a simple fix for that (until 1.10.0 lands):

in main.cpp:

#include "esphomelib/application.h"
#include "my_custom_component.h"

void setup() {
  //

I know it could increase the complexity of the code generator because of the need to support two file formats

To make this clear: I'm not planning on supporting both formats. esphomyaml will just auto-migrate the last main.cpp source to the new format. So the new code will just need to support the new format.

BTW have you thought about using some python template engine for code generation?

You mean having the main.cpp file based on a template like this?

// Auto generated code by esphomeyaml
{{ include_code }}

void setup() {
  {{ setup_code }}
  App.setup();
}

void loop() {
  App.loop();
}

I mean it would be an option but I think with the features discussed above this would become unnecessary. Also, I don't like the overhead of including a full templating engine for something that can easily be solved otherwise - in the end the "AUTO GENERATED CODE START" / END blocks are also just what a templating engine would do.

[yawor]

Keeping metadata in a JSON file is nice.

Regarding CubeMX, I've found the code generated by it really easy to use. The sections in the files for user code are placed in many different locations so most users can place the code where it needs to be.

Regarding a jinja2 template engine, I've been rather thinking about something like this:

// Auto generated code by esphomeyaml
#include "esphomelib/application.h"
{% for inc in extra_includes %}
# include "{{ inc }}"
{% endfor %}
{% block Includes %}{% endblock %}

using namespace esphomelib;

void setup() {
  {{ generated_setup_code }}
  {% block Setup %}{% endblock %}
  App.setup();
}

void loop() {
  App.loop();
  {% block Loop %}{% endblock %}
  delay(16);
}

and then user can provide a template like this:

{% extends "default.cpp.tmpl" %}

{% block Includes %}
#include "somecustomheader.h"
{% endblock %}

{% block Setup %}
  // some user setup code
  // more code
{% endblock %}

Anyway, I think that if it works, it works :). Waiting patiently for 1.10.

[yawor]

@OttoWinter I have a question regarding your proposed custom sensor platform. Are you doing this platform only for the sensor component type? Will we be able to instantiate a component class which inherits some other base component types, for example Cover, Switch or Light? Do you plan to add the custom platform to other types too?
Maybe custom shouldn't be a platform for specific type, but a type of its own with just a single platform (could also be named custom) set by default for that type, as the component instantiation is going to be a lambda/template anyway, so it can return anything. A quick example:

custom:
  - id: some_custom_component
    construct: |-
      return SomeCustomComponent("Some custom component", args);

[OttoWinter]

Done

[yawor]

@OttoWinter sorry for reanimating this issue, but I'd like to discuss some things related to extensibility and this plase seems to be most appropriate (no reason to open new issue, yet).

I've been looking through the code and thinking about how to make it even more extensible. I would like to know what do you think about plugins for esphome.
My idea is that a plugin would be a python package installed in the same python environment as esphome. A plugin would use a named entry point in its setup, which can be then discovered by esphome using pkg_resources.

Example code in setup.py in some python package:

setup(
    #...
    entry_points="""
        [esphome.plugins]
        output.myoutput = myplugin.myoutput
        input.myinput = myplugin.myinput
    """,
    #...
)

A code for component discovery in plugins:

def get_component_from_plugins(domain):
    for entry_point in pkg_resources.iter_entry_points('esphome.plugins'):
        if entry_point.name == domain:
            return entry_point.load()
    return None

A get_component function in esphomeyaml.config could then first try to get a component from built-in components and if it failed, call above function. Or it could call it first - this would allow overriding built-in components, but could potentially cause more issues.

In this example, a user would be able to define myoutput as any other output platform in the esphome:

output:
  - platform: myoutput
    id: foo
    some_config: true

As you see a preliminary support for plugins is not that complicated and doesn't involve that much changes in the esphome code.

[OttoWinter]

@yawor Yes... that would certainly be possible.

Though my question would be: why???
Sure we can create an ecosystem of packages, but why do we need a plugin system?

• For new features I'm happy to accept PRs (as long as code quality is good) - and I want to follow the Home Assistant principle for accepting new features.

• Also, the python code is only one part of the equation. Simply including a python module does allow the use of code generation, but somehow the user would still need to include the C++ too.

• While the entry point system is nice, it won't work for a lot of users. Specifically it will only work for pip-install users, which are a clear minority AFAIK. Docker users would need to create their own dockerfiles and for Hass.io addon users (I think the largest group, have no analytics though) will never be able to benefit from it.

  • Also plugins can never be high-quality - ESPHome doesn't even have a stable (nor documented) API now. The API is far from stable, so I want to have the option to change it as I see fit - it's only for internal use anyway (kind of like HA). Of course a stable API can be created at some point, but it will take some time.

• Also, I think Home Assistant's custom_component approach is much better than entry points: In your config directory you can have a custom_components directory where you put your custom files. See https://github.com/home-assistant/home-assistant/blob/dev/homeassistant/loader.py#L112

But my main question is still why.

[yawor]

Custom components are really good, but they require some level of C++ knowledge from a user. Of course a user can always just copy&paste a code from some tutorial, but it's more complicated than just entering a yaml syntax.

I know that python is only a part of the equation, but you already have everything in place: a component can define LIB_DEPS, which informs platformio what libraries and from where it has to download.

Regarding Docker/Hass.io users, I've been thinking about that too. One way would be to add plugins option in core config, which would take a list of strings passed to pip inside the docker container when compiling. It could take any supported pip package specification.

Regarding the quality, yes, I acknowledge this as a valid issue. This is probably something every framework struggles when it comes to the plugins support. I think the best way would be for plugins to check esphome version themselves and raise an exception if installed version is not supported. The check would need to be done when loading a component module from the plugin, so it won't get loaded in that case. It could throw a specific exception class so esphome would know that the issue is the version incompatibility.

Anyway, this is just my preliminary proposition. I haven't started playing around the code and making pull request because I don't know if this is even desired or not and some discussion is probably needed before any potential work.

[OttoWinter]

Custom components are really good, but they require some level of C++ knowledge from a user. Of course a user can always just copy&paste a code from some tutorial, but it's more complicated than just entering a yaml syntax.

I agree. Though I still don't understand why this should be preferred over just creating a PR to the ESPHome repos. That ensures code quality, compatibility for future versions (as it is in main code base), discoverability (how should people find plugins? I won't add them to the docs because not officially maintained).

Of course the PR cycle takes its time but I think publishing a library too is quite time consuming. For example you would need to publish platformio library (not so easy) and then publish on pypi too - and for both set up the whole build tools too (like setuptools etc).

I think the best way would be for plugins to check esphome version themselves and raise an exception if installed version is not supported. The check would need to be done when loading a component module from the plugin, so it won't get loaded in that case. It could throw a specific exception class so esphome would know that the issue is the version incompatibility.

That is just ugly. I do not want people using old versions of ESPHome just because some plugin hasn't been updated yet (and eventually I want to move to a faster release cycle, which would make things even worse). Also I do not want ESPHome to become a fragmented system, Home Assistant (once again) is a perfect example how to do it IMO.

Custom components are really good, but they require some level of C++ knowledge from a user. Of course a user can always just copy&paste a code from some tutorial, but it's more complicated than just entering a yaml syntax.

I'm not talking about C++ custom components. Please see how Home Assistant handles custom components. Basically, the user would just need to copy a directory to their config.