Accept manual input for rate plan ID (workaround for possible bug in API responses) about ha-openei

[Tuxrug]

The API doesn't allow for zip code lookups.
https://openei.org/services/doc/rest/util_rates/?version=7

I'll look into adding a "Manual" setup to allow input of the plan directly.

I got it to return results by passing my zip code by itself as the Address parameter, but I'll check again when I get home to make sure it wasn't a cached response based on a prior query or anything weird like that. I was also using version=8 so it might have used different search behavior.

Update: I tried with ?api_key=xxxxx&version=7&address=90210 and got utilities for Los Angeles, so the address parameter doesn't require a full address or lat/long pair. It also provided expected results for address=Madison,%20WI confirming city-state pairs work as well.

[Tuxrug]

Looks good! I'm not finding any odd behavior or discrepancies against what the config flow says it'll do.

[firstof9]

The API doesn't allow for zip code lookups.
https://openei.org/services/doc/rest/util_rates/?version=7

I'll look into adding a "Manual" setup to allow input of the plan directly.

[firstof9]

Excellent, thank you for checking, I think a City, State/Zip Code should be fine enough.

[firstof9]

Give 0.1.5-b0 a try.

[Tuxrug]

The following happens before and after an uninstall, restart homeassistant conatiner, re-install, restart.

This is allowing me to search by zip code, but my utility company still isn't showing and I'm unable to find where to manually enter a rateplan ID.

I also noticed that when leaving location blank in configuration, it returns a very large number of utilities when it returned only three options before. I'm not sure how to debug to check whether latitude/longitude is still passing when the location box is cleared since I'm new to GitHub, Python, and Home Assistant.

I am also getting "Retrying setup: None" again after selecting any rateplan with the following log entries caught in the container's log output:

2021-09-10 19:12:33 ERROR (MainThread) [custom_components.openei] Error fetching openei data: ,
2021-09-10 19:12:33 WARNING (MainThread) [homeassistant.config_entries] Config entry 'Southern Iowa Elec Coop, Inc' for openei integration not ready yet: None; Retrying in background,
2021-09-10 19:12:38 ERROR (MainThread) [custom_components.openei] Error fetching openei data: ,
2021-09-10 19:12:48 ERROR (MainThread) [custom_components.openei] Error fetching openei data: ,
2021-09-10 19:13:08 ERROR (MainThread) [custom_components.openei] Error fetching openei data: ,
2021-09-10 19:13:48 ERROR (MainThread) [custom_components.openei] Error fetching openei data: ,

I tried running through the config again I get "User input malformed" on Step 3 both when accepting the remembered values from last flow and selecting new values.

I'm noticing a bunch of weird log entries I haven't noticed before prior to Home Assistant loading custom integrations, so I'll uninstall this integration, recreate the container with a fresh image pull, then run a configuration validation in Home Assistant before I try installing again to make sure something weird in my environment isn't poisoning these results.

[Tuxrug]

After recreating the homeassistant container and re-installing 0.1.5-b0 omitting location still returns utilities from all over the country and selecting a rateplan still gets stuck with log entries identical to above:

2021-09-10 19:35:10 ERROR (MainThread) [custom_components.openei] Error fetching openei data: ,
2021-09-10 19:35:10 WARNING (MainThread) [homeassistant.config_entries] Config entry 'City of Le Sueur, Minnesota (Utility Company)' for openei integration not ready yet: None; Retrying in background,
2021-09-10 19:35:15 ERROR (MainThread) [custom_components.openei] Error fetching openei data: ,
2021-09-10 19:35:25 ERROR (MainThread) [custom_components.openei] Error fetching openei data: ,
2021-09-10 19:35:45 ERROR (MainThread) [custom_components.openei] Error fetching openei data: ,
2021-09-10 19:36:25 ERROR (MainThread) [custom_components.openei] Error fetching openei data: ,
2021-09-10 19:37:45 ERROR (MainThread) [custom_components.openei] Error fetching openei data: ,

Reconfiguration selecting different values still gives "User input malformed" does not save. The weird log entries I mentioned appearing prior to custom components seem to just be triggered by restarting Home Assistant from its own GUI and are not present when restarting via docker, so it doesn't appear to be related.

[Tuxrug]

I found the cause for the "User input malformed" - when repeating the flow, the energy meter option blanks out. Re-selecting an option (in my case None) allows me to safe the re-configuration but it still produces the same errors as above.

[firstof9]

Feel free to roll back, I'll take a look at it in the morning.

Thank you for testing.

[firstof9]

There we go, give 0.1.5-b1 a go

[Tuxrug]

It holds the meter setting on re- now but still returns a large, nationwide list when leaving location blank and gets stuck at "Retrying setup: None". I still can't get the API itself to return the correct utility (it only appears at the web search, but I've never gotten it to appear via API response) or enter a rateplan ID manually.

I'm at work right now but I can try to pull more log information when I get home.

[firstof9]

This new beta allows the the manual input of the plan ID. It should also fix the setup not firing as well.

[Tuxrug]

I'm getting "Unknown error occurred" on Step 1 of configuration now, with the following log entry showing up:

This error originated from a custom integration.

Logger: aiohttp.server
Source: custom_components/openei/config_flow.py:312
Integration: OpenEI Utility Rates (documentation, issues)
First occurred: 6:21:31 PM (3 occurrences)
Last logged: 6:21:44 PM

Error handling request
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/aiohttp/web_protocol.py", line 422, in _handle_request
    resp = await self._request_handler(request)
  File "/usr/local/lib/python3.9/site-packages/aiohttp/web_app.py", line 499, in _handle
    resp = await handler(request)
  File "/usr/local/lib/python3.9/site-packages/aiohttp/web_middlewares.py", line 119, in impl
    return await handler(request)
  File "/usr/src/homeassistant/homeassistant/components/http/security_filter.py", line 60, in security_filter_middleware
    return await handler(request)
  File "/usr/src/homeassistant/homeassistant/components/http/forwarded.py", line 211, in forwarded_middleware
    return await handler(request)
  File "/usr/src/homeassistant/homeassistant/components/http/request_context.py", line 24, in request_context_middleware
    return await handler(request)
  File "/usr/src/homeassistant/homeassistant/components/http/ban.py", line 78, in ban_middleware
    return await handler(request)
  File "/usr/src/homeassistant/homeassistant/components/http/auth.py", line 144, in auth_middleware
    return await handler(request)
  File "/usr/src/homeassistant/homeassistant/components/http/view.py", line 135, in handle
    result = await result
  File "/usr/src/homeassistant/homeassistant/components/config/config_entries.py", line 155, in post
    return await super().post(request, flow_id)
  File "/usr/src/homeassistant/homeassistant/components/http/data_validator.py", line 63, in wrapper
    result = await method(view, request, *args, **kwargs)
  File "/usr/src/homeassistant/homeassistant/helpers/data_entry_flow.py", line 109, in post
    result = await self._flow_mgr.async_configure(flow_id, data)
  File "/usr/src/homeassistant/homeassistant/data_entry_flow.py", line 202, in async_configure
    result = await self._async_handle_step(flow, cur_step["step_id"], user_input)
  File "/usr/src/homeassistant/homeassistant/data_entry_flow.py", line 260, in _async_handle_step
    result: FlowResult = await getattr(flow, method)(user_input)
  File "/config/custom_components/openei/config_flow.py", line 45, in async_step_user
    return await self.async_step_user_2()
  File "/config/custom_components/openei/config_flow.py", line 57, in async_step_user_2
    return await self._show_config_form_2(user_input)
  File "/config/custom_components/openei/config_flow.py", line 88, in _show_config_form_2
    utility_list = await _get_utility_list(self.hass, self._data)
  File "/config/custom_components/openei/config_flow.py", line 273, in _get_utility_list
    plans = await hass.async_add_executor_job(_lookup_plans, plans)
  File "/usr/local/lib/python3.9/concurrent/futures/thread.py", line 52, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/config/custom_components/openei/config_flow.py", line 312, in _lookup_plans
    response.insert(0, "Not Listed")
AttributeError: 'dict' object has no attribute 'insert'

[firstof9]

Ok I'm working on this, feel free to revert to a previous version for now.

[firstof9]

Ok latest beta should clear it all up.

[Tuxrug]

Latest beta let me set my rate plan ID under Not Listed, but leaving location blank is still returning a huge list. I'm not sure how to check if it's still including lat/long by default or if something else is causing the huge list, but the zip code search is working too.

[firstof9]

You can enable debugging on the library, it should output the URL it's using:

service: logger.set_level
data:
  openeihttp: debug

Also if the location box isn't empty entering "" will clear it out and utilize the lat/lon.

[Tuxrug]

It looks like leaving location blank is leaving it blank in the request instead of the lat/long pair, and entering "" passed literally on new configs but is sending the lat/long pair when used on re-configuring after first-time configuration.

Leaving radius and location totally blank on a new config is doing this:
2021-09-14 19:43:35 DEBUG (SyncWorker_5) [openeihttp] Looking up plans via URL: https://api.openei.org/utility_rates?version=latest&format=json&api_key={xxxxxxxx}&sector=Residential&radius=&address=

Entering "" instead of radius and location on a new config generates "Unknown Error Occurred" in Step1 and appears to be passing the double-quotes as-is:

2021-09-14 19:47:04 DEBUG (SyncWorker_3) [openeihttp] Looking up plans via URL: https://api.openei.org/utility_rates?version=latest&format=json&api_key={xxxxxxxx}&sector=Residential&radius=""&address=""
2021-09-14 19:47:04 ERROR (MainThread) [aiohttp.server] Error handling request
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/aiohttp/web_protocol.py", line 422, in _handle_request
resp = await self._request_handler(request)
File "/usr/local/lib/python3.9/site-packages/aiohttp/web_app.py", line 499, in _handle
resp = await handler(request)
File "/usr/local/lib/python3.9/site-packages/aiohttp/web_middlewares.py", line 119, in impl
return await handler(request)
File "/usr/src/homeassistant/homeassistant/components/http/security_filter.py", line 60, in security_filter_middleware
return await handler(request)
File "/usr/src/homeassistant/homeassistant/components/http/forwarded.py", line 211, in forwarded_middleware
return await handler(request)
File "/usr/src/homeassistant/homeassistant/components/http/request_context.py", line 24, in request_context_middleware
return await handler(request)
File "/usr/src/homeassistant/homeassistant/components/http/ban.py", line 78, in ban_middleware
return await handler(request)
File "/usr/src/homeassistant/homeassistant/components/http/auth.py", line 144, in auth_middleware
return await handler(request)
File "/usr/src/homeassistant/homeassistant/components/http/view.py", line 135, in handle
result = await result
File "/usr/src/homeassistant/homeassistant/components/config/config_entries.py", line 155, in post
return await super().post(request, flow_id)
File "/usr/src/homeassistant/homeassistant/components/http/data_validator.py", line 63, in wrapper
result = await method(view, request, *args, **kwargs)
File "/usr/src/homeassistant/homeassistant/helpers/data_entry_flow.py", line 109, in post
result = await self._flow_mgr.async_configure(flow_id, data)
File "/usr/src/homeassistant/homeassistant/data_entry_flow.py", line 202, in async_configure
result = await self._async_handle_step(flow, cur_step["step_id"], user_input)
File "/usr/src/homeassistant/homeassistant/data_entry_flow.py", line 260, in _async_handle_step
result: FlowResult = await getattr(flow, method)(user_input)
File "/config/custom_components/openei/config_flow.py", line 45, in async_step_user
return await self.async_step_user_2()
File "/config/custom_components/openei/config_flow.py", line 57, in async_step_user_2
return await self._show_config_form_2(user_input)
File "/config/custom_components/openei/config_flow.py", line 88, in _show_config_form_2
utility_list = await _get_utility_list(self.hass, self._data)
File "/config/custom_components/openei/config_flow.py", line 286, in _get_utility_list
plans = await hass.async_add_executor_job(_lookup_plans, plans)
File "/usr/local/lib/python3.9/concurrent/futures/thread.py", line 52, in run
result = self.fn(*self.args, **self.kwargs)
File "/config/custom_components/openei/config_flow.py", line 324, in _lookup_plans
response = handler.lookup_plans()
File "/usr/local/lib/python3.9/site-packages/openeihttp/__init__.py", line 79, in lookup_plans
if "error" in result.json().keys():
File "/usr/local/lib/python3.9/site-packages/requests/models.py", line 900, in json
return complexjson.loads(self.text, **kwargs)
File "/usr/local/lib/python3.9/site-packages/simplejson/__init__.py", line 525, in loads
return _default_decoder.decode(s)
File "/usr/local/lib/python3.9/site-packages/simplejson/decoder.py", line 370, in decode
obj, end = self.raw_decode(s)
File "/usr/local/lib/python3.9/site-packages/simplejson/decoder.py", line 400, in raw_decode
return self.scan_once(s, idx=_w(s, idx).end())
simplejson.errors.JSONDecodeError: Expecting ',' delimiter or ']': line 1 column 32 (char 31)
2021-09-14 19:47:09 DEBUG (MainThread) [custom_components.openei] Finished fetching openei data in 0.000 seconds (success: True)

This also happens on a new config with "" only in the location:
2021-09-14 19:53:54 DEBUG (SyncWorker_3) [openeihttp] Looking up plans via URL: https://api.openei.org/utility_rates?version=latest&format=json&api_key={xxxxxxxx}&sector=Residential&radius=&address=""

Repeating an existing configuration and replacing a previous location entry with "" does pass latitude and longitude in the request URL.

[firstof9]

Thanks for checking totally forgot to test a new config situation.

[firstof9]

ok I think I have it all sorted now in beta 4

[Tuxrug]

New configuration flow with blank location and radius is now pulling location-based results again.

The only issue I see is that on re-configure it seems to be treating radius as a mandatory field (giving "User input malformed" if radius is blank or "", only allowing reconfigure if a number is entered). Entering "" as the location is behaving as described as well (tested using a California zip code then reconfiguring and it found me in Colorado).

[firstof9]

excellent, I'll double check on radius

[firstof9]

I changed the radius to only take numbers and gave it a range with a default of 0. That should resolve the radius issue.

[Tuxrug]

The radius is fixed but now it's requiring "" as location to use lat/long on new configurations when it took blank on initial configuration before. It's called out on the setup screen, but it does say that omitting it does the same thing right before it.

Also when doing a reconfigure and selecting a new plan (whether it's manually-specified or picked from list on the previous configuration) it still shows the previously-selected plan ID below the one selected from the list. Unless the user clears it manually on re-configure it's unclear which ID it's actually going to use. Can choosing a new one from the dropdown automatically clear the manual input field?

[firstof9]

I'm trying to find a balance, the UI will freak out if the value is None.

[firstof9]

I believe I've found a suitable work around to the config flow limits in the latest beta.
I've updated the instructions accordingly per config flow as well.

[Tuxrug]

I think the changes to the text on the config flow make it a lot clearer. For some reason though now on reconfigure, "" is passed literally as the address parameter resulting in no results returned

2021-09-17 19:27:42 DEBUG (MainThread) [custom_components.openei.config_flow] Step 1: {'api_key': '{xxxxxxx}', 'location': '""', 'radius': 0}
2021-09-17 19:27:42 DEBUG (MainThread) [custom_components.openei.config_flow] data: {'api_key': '{xxxxxxx}', 'radius': 0, 'utility': 'Not Listed', 'rate_plan': '{xxxxxxx}', 'location': '""'}
2021-09-17 19:27:42 DEBUG (SyncWorker_0) [openeihttp] Looking up plans via URL: https://api.openei.org/utility_rates?version=latest&format=json&api_key={xxxxxxx}&sector=Residential&radius=0&address=""

[firstof9]

Shoot I thought I had that filtered out, it tested it on my dev instance, I'll take a look.

[firstof9]

All right, it looks properly sorted in b7 now.

[Tuxrug]

At first I let HACS upgrade me directly to b8 and I could not configure or reconfigure an instance (any input I tried gave me "Unknown error occurred") but then I downgraded to b7 and it works perfectly. After upgrading to b8 again I get "Unknown error occurred" when trying to configure or re-configure. The already-configured instance from b7 does continue to work in b8.

This error originated from a custom integration.

Logger: aiohttp.server
Source: custom_components/openei/config_flow.py:318
Integration: OpenEI Utility Rates (documentation, issues)
First occurred: 10:58:27 PM (3 occurrences)
Last logged: 10:58:55 PM

Error handling request
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/aiohttp/web_protocol.py", line 422, in _handle_request
    resp = await self._request_handler(request)
  File "/usr/local/lib/python3.9/site-packages/aiohttp/web_app.py", line 499, in _handle
    resp = await handler(request)
  File "/usr/local/lib/python3.9/site-packages/aiohttp/web_middlewares.py", line 119, in impl
    return await handler(request)
  File "/usr/src/homeassistant/homeassistant/components/http/security_filter.py", line 60, in security_filter_middleware
    return await handler(request)
  File "/usr/src/homeassistant/homeassistant/components/http/forwarded.py", line 211, in forwarded_middleware
    return await handler(request)
  File "/usr/src/homeassistant/homeassistant/components/http/request_context.py", line 24, in request_context_middleware
    return await handler(request)
  File "/usr/src/homeassistant/homeassistant/components/http/ban.py", line 78, in ban_middleware
    return await handler(request)
  File "/usr/src/homeassistant/homeassistant/components/http/auth.py", line 144, in auth_middleware
    return await handler(request)
  File "/usr/src/homeassistant/homeassistant/components/http/view.py", line 135, in handle
    result = await result
  File "/usr/src/homeassistant/homeassistant/components/config/config_entries.py", line 213, in post
    return await super().post(request, flow_id)
  File "/usr/src/homeassistant/homeassistant/components/http/data_validator.py", line 63, in wrapper
    result = await method(view, request, *args, **kwargs)
  File "/usr/src/homeassistant/homeassistant/helpers/data_entry_flow.py", line 109, in post
    result = await self._flow_mgr.async_configure(flow_id, data)
  File "/usr/src/homeassistant/homeassistant/data_entry_flow.py", line 202, in async_configure
    result = await self._async_handle_step(flow, cur_step["step_id"], user_input)
  File "/usr/src/homeassistant/homeassistant/data_entry_flow.py", line 260, in _async_handle_step
    result: FlowResult = await getattr(flow, method)(user_input)
  File "/config/custom_components/openei/config_flow.py", line 126, in async_step_user
    return await self.async_step_user_2()
  File "/config/custom_components/openei/config_flow.py", line 137, in async_step_user_2
    return await self._show_config_form_2(user_input)
  File "/config/custom_components/openei/config_flow.py", line 158, in _show_config_form_2
    utility_list = await _get_utility_list(self.hass, self._data)
  File "/config/custom_components/openei/config_flow.py", line 279, in _get_utility_list
    plans = await hass.async_add_executor_job(_lookup_plans, plans)
  File "/usr/local/lib/python3.9/concurrent/futures/thread.py", line 52, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/config/custom_components/openei/config_flow.py", line 318, in _lookup_plans
    response["Not Listed"] = [{"name": "Not Listed", "label": "Not Listed"}]
TypeError: 'method' object does not support item assignment

[firstof9]

Thanks, I'll move that little tweak into the library so this shouldn't happen, all should be well in b9

[Tuxrug]

b9 is still not allowing configure or reconfigure with a "Unknown error occurred", with different radii, blank location, "" location, or zip code entere.

This error originated from a custom integration.

Logger: aiohttp.server
Source: custom_components/openei/config_flow.py:282
Integration: OpenEI Utility Rates (documentation, issues)
First occurred: 3:44:46 PM (8 occurrences)
Last logged: 3:45:26 PM

Error handling request
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/aiohttp/web_protocol.py", line 422, in _handle_request
    resp = await self._request_handler(request)
  File "/usr/local/lib/python3.9/site-packages/aiohttp/web_app.py", line 499, in _handle
    resp = await handler(request)
  File "/usr/local/lib/python3.9/site-packages/aiohttp/web_middlewares.py", line 119, in impl
    return await handler(request)
  File "/usr/src/homeassistant/homeassistant/components/http/security_filter.py", line 60, in security_filter_middleware
    return await handler(request)
  File "/usr/src/homeassistant/homeassistant/components/http/forwarded.py", line 211, in forwarded_middleware
    return await handler(request)
  File "/usr/src/homeassistant/homeassistant/components/http/request_context.py", line 24, in request_context_middleware
    return await handler(request)
  File "/usr/src/homeassistant/homeassistant/components/http/ban.py", line 78, in ban_middleware
    return await handler(request)
  File "/usr/src/homeassistant/homeassistant/components/http/auth.py", line 144, in auth_middleware
    return await handler(request)
  File "/usr/src/homeassistant/homeassistant/components/http/view.py", line 135, in handle
    result = await result
  File "/usr/src/homeassistant/homeassistant/components/config/config_entries.py", line 213, in post
    return await super().post(request, flow_id)
  File "/usr/src/homeassistant/homeassistant/components/http/data_validator.py", line 63, in wrapper
    result = await method(view, request, *args, **kwargs)
  File "/usr/src/homeassistant/homeassistant/helpers/data_entry_flow.py", line 109, in post
    result = await self._flow_mgr.async_configure(flow_id, data)
  File "/usr/src/homeassistant/homeassistant/data_entry_flow.py", line 202, in async_configure
    result = await self._async_handle_step(flow, cur_step["step_id"], user_input)
  File "/usr/src/homeassistant/homeassistant/data_entry_flow.py", line 260, in _async_handle_step
    result: FlowResult = await getattr(flow, method)(user_input)
  File "/config/custom_components/openei/config_flow.py", line 126, in async_step_user
    return await self.async_step_user_2()
  File "/config/custom_components/openei/config_flow.py", line 137, in async_step_user_2
    return await self._show_config_form_2(user_input)
  File "/config/custom_components/openei/config_flow.py", line 158, in _show_config_form_2
    utility_list = await _get_utility_list(self.hass, self._data)
  File "/config/custom_components/openei/config_flow.py", line 282, in _get_utility_list
    for utility in plans:
TypeError: 'method' object is not iterable

[firstof9]

Ok I'll take a look again.

[firstof9]

All sorted, I verified it in my dev instance of HA as well.
beta10 should be good, let me know if you find anything weird.