jphutchins / open-energy-view Goto Github PK

View resource consumption trends, history, analysis, and insights.

Home Page: https://www.openenergyview.com

License: Other

Python 23.72% Shell 1.13% HTML 0.29% JavaScript 63.63% CSS 11.23%

climate-change electricity electricity-consumption electricity-consumption-analysis electricity-consumption-forecasting electricity-meter energy energy-consumption energy-data energy-monitor home-automation iot

open-energy-view's People

Contributors

Stargazers

Watchers

Forkers

nnkostov ianwinsemius gbdavidx aveedibya ase-jabra sjiampojamarn aaronjwood cipher12 riahtu yuanzhongqiao

open-energy-view's Issues

Make development more platform agnostic with docker

It would be nice if all of the services could be spun up simply to test things out. I think docker compose could be a useful tool for this. I'm going to take a crack at writing up a docker compose

Persistent Data Structures

Research viability of PDS to minimize bandwidth necessary for incremental updates.

Problem: User has a large database which is different to the structure in their HTML 5 local storage by perhaps 1%

Goal: Send only this 1% of data to update the user's structure rather than sending the entire structure again.

PG&E electricity view stops abruptly after October 12, 2021

While I was successful in getting past the #37 issue, electrical data only updated through October 12, 2021. No data past that date are shown despite no changes on the utility service side.

Dashboard Date defaults to December 31, 1969

Thanks for inviting me to try out Open Energy View!

Steps I've taken:
I've signed up using my Google account. After, I connected it to my PG&E Account.

Issue:
I saw that the graphs and charts weren't populated and thought it would take a some time for the data to populate. I checked a few days later and the graphs were still blank. I realized that the dates were for the year 1969 (I wasn't a customer at that point in time lol)

Additionally, when I go to change the date range, I only can select the year 1969:

Any thoughts?

Database schema

This project is becoming more complex since the user should be greeted with a flexible graph of their data.

User should be able to choose the date range and the interval displayed should adjust intelligently. This means that sums need to be calculated (add up all hourly intervals for a day sum, add up days for weeks, months, etc.) The question is whether this should be calculated in advance and if so what is the appropriate key to store it under in the database? The current database is keyed by a start time that is hourly - should there be other databases keyed by daily, weekly, and monthly intervals?
Analytics need to be implemented quickly (before Alpha) so that the user is presented with useful information that can impact a reasonable change in habits with measurable outcome.

"Simple" analytics

Goal - provide user a "baseline usage" that helps them to find "phantom power" suckers in their home. Baseline will generally be the lowest Wh recorded recently.

Daily averages.
Weekly averages.
Daily lows.
Daily highs.

More complex:
Usage characteristics:
Sleep, daytime, appliance, evening? Rely on user definition?

Validation of data received every 24 hours?

It would be nice to verify that data has been received and then request it directly after some timout if for some reason it was not received.

Stuck on spinning circle

I believe I'm having a similar issue to #46 - the same symptoms anyway. I've attached a screenshot, I'm getting a 500 error it seems.

Registration RI problem

Remove the =None s in the constructor.
Instead take a keyword method=get_auth_file argument which is the function that will provide the credentials. RI is that client_id, client_secret, cert_crt_path, cert_key_path are not None so constructor will fail of the the supplied function does not fill these class attributes. Validation can wait for later if at all.

Keep customized time partitions after logout

Customization of daily time periods seems now to work until logout. When logging back in, previous customizations are forgotten.

Website appears to be down

Beta website appears to be down right now.

Another 502 Bad Gateway Problem

Not sure if this should really be a new issue since it seems to be a recurring thing, but as best I can tell things seemed to have been working for others ~a week ago? I can authorize permissions with PG&E but then it gets stuck loading when it returns to the app, and if I check the console and try to look at what's going on there's a GET request receiving a 502.

502 bad gateway

After going through the PG&E authorization process without error, I am receiving a 502 bad gateway error and cannot proceed any further on Open Energy View. If I go back to my PG&E authorizations page, I do not see Open Energy View on the list.

Support gas

OEV currently does not parse gas records even though it receives them. This issue is closed when gas records are supported.

Document development environment setup

Create a new dev environment and document procedure. This should provide sample account data for 3600/900 electrical as well as gas so that the gas API can be started.

Timouts and retries

Need to add a timeout and retry mechanism on requests?

Website shows date of Dec 31, 1969 (still)

Is this issue supposed to be resolved? I just registered and it shows a date of Dec 31, 1969.

Algorithm to identify point of change in energy usage.

A user's baseline usage is given by averaging the 3 lowest readings each day. Imagine graphing this daily value over a long period of time. One is likely to see a plateaus in this graph representing periods of time where 24/7 devices remained relatively consistent. Now imagine a user has changed or added a new appliance at some point. This point should be identifiable as a change from one plateau to another. In other words, a section of the graph that has positive or negative slope relative to the sections of no slope around it. Once the point of change is identified an analysis can be delivered to the user to assess the impact on their Wh.

Gaps in data

    It's working now, although there are some data gaps:

Originally posted by @ChrisMD123 in #38 (comment)

Frontend does not parse 900 second intervals

I lazily hardcoded the frontend to only ask the DB for intervals that match 3600 duration (1 hour). PGE has made more meters support 900 duration.

Easy fix: DB translates higher resolution intervals to 1 hour (not cool)

Better fix: Add "interval agnostic" raw data interval to the frontend and force the frontend to parse all standardized intervals from it.

open-energy-view/open_energy_view/frontend/js/api/DatabaseService.js

Lines 48 to 62 in e4265dc

 export const getHourlyData = (source) => { 

 const email = cookie.load("logged_in"); 

 const localStorageString = getLocalStorage(email, source); 

 const lastUpdate = localStorageString 

 ? sliceLastUpdateFromResponse(localStorageString) 

 : null; 

 return attemptP(() => 

 axios.post( 

 "/api/web/data/hours", 

 { source, lastUpdate }, 

 AuthService.getAuthHeader() 

 ) 

 ); 

 };

open-energy-view/open_energy_view/resources.py

Lines 527 to 571 in e4265dc

 class GetHourlyData(Resource): 

 @jwt_required 

 def post(self): 

 data = get_data_parser.parse_args() 

 if not data["source"] or data["source"] == "None": 

 return 

 user = db.session.query(models.User).filter_by(id=get_jwt_identity()).first() 

 source = ( 

 db.session.query(models.Source) 

 .filter_by(friendly_name=data["source"]) 

 .with_parent(user) 

 .first() 

 ) 

 if (data["lastUpdate"]) and (int(data["lastUpdate"]) == source.last_update): 

 return ( 

 { 

 "useLocalStorage": True, 

 "email": user.email, 

 "friendlyName": source.friendly_name, 

 }, 

 200, 

 ) 

 # TODO: currently entries are coming in reversed - research ORDER BY performance 

 hours = ( 

 db.session.query(models.Espi) 

 .filter_by(source_id=source.id, duration=3600) 

 .order_by(models.Espi.start) 

 .all() 

 ) 

 database = ",".join( 

 [f"{entry.start//3600}{entry.watt_hours}" for entry in hours] 

 ) 

 response = { 

 "useLocalStorage": False, 

 "utility": "pge", 

 "interval": 3600, 

 "email": user.email, 

 "friendlyName": data["source"], 

 "lastUpdate": source.last_update, 

 "partitionOptions": source.partition_options, 

 "database": database, 

 } 

 return response, 200

Home Assistant Integration

Your excellent OpenEnergyView project is one of the easiest ways to get programmatic access to PG&E data, especially once the gas meter support is added! PG&E's StreamMyData provides access to electricity readings (after the purchase of compatible hardware) but there seems to be no other access to the gas meter information.

It would be wonderful to have the cumulative readings available in HomeAssistant's energy dashboard (from this feature request thread).

Isolate "server"

The server (currently just server.py) should be isolated from the rest of the program and serve only one purpose :

Listen on 443 for the XML from PGE and upon receiving
open a socket to the main process and send the XML
and keep some of the latest XMLs so they can be retrieved RESTfully

This will allow to port the server to:

a WAR so that people that already have Tomcat servers can use a more secure system - ideally the webserver is on the same computer as the PGESMD service...
and a server in the cloud that can take in PGEs asynchronous XMLs and serve them back to users "synchronously" ... as long as they wait for PGE...

Dashboard appears to be interpreting solar production as the only consumption

I recently installed solar. As soon as PG&E started reporting net energy use rather than just zeroing out the hours I was producing, it seems as if Open Energy View is interpreting my solar generation as my only consumption. See before-and-after screenshots.

	export const getHourlyData = (source) => {
	const email = cookie.load("logged_in");
	const localStorageString = getLocalStorage(email, source);
	const lastUpdate = localStorageString
	? sliceLastUpdateFromResponse(localStorageString)
	: null;

	return attemptP(() =>
	axios.post(
	"/api/web/data/hours",
	{ source, lastUpdate },
	AuthService.getAuthHeader()
	)
	);
	};

	class GetHourlyData(Resource):
	@jwt_required
	def post(self):
	data = get_data_parser.parse_args()
	if not data["source"] or data["source"] == "None":
	return

	user = db.session.query(models.User).filter_by(id=get_jwt_identity()).first()
	source = (
	db.session.query(models.Source)
	.filter_by(friendly_name=data["source"])
	.with_parent(user)
	.first()
	)

	if (data["lastUpdate"]) and (int(data["lastUpdate"]) == source.last_update):
	return (
	{
	"useLocalStorage": True,
	"email": user.email,
	"friendlyName": source.friendly_name,
	},
	200,
	)
	# TODO: currently entries are coming in reversed - research ORDER BY performance
	hours = (
	db.session.query(models.Espi)
	.filter_by(source_id=source.id, duration=3600)
	.order_by(models.Espi.start)
	.all()
	)
	database = ",".join(
	[f"{entry.start//3600}{entry.watt_hours}" for entry in hours]
	)
	response = {
	"useLocalStorage": False,
	"utility": "pge",
	"interval": 3600,
	"email": user.email,
	"friendlyName": data["source"],
	"lastUpdate": source.last_update,
	"partitionOptions": source.partition_options,
	"database": database,
	}
	return response, 200

jphutchins / open-energy-view Goto Github PK

open-energy-view's People

Contributors

Stargazers

Watchers

Forkers

open-energy-view's Issues

Recommend Projects

Recommend Topics

Recommend Org

Jobs