emoncms / graph

Versatile graphing module for emoncms

Graph

A versatile graphing module for emoncms. This module is designed to replace the default data viewer in emoncms and is loaded when a feed view button is clicked.

Install

1. Install the Graph module

And ensure you are runing latest version and Emoncms

cd /var/www/emoncms
git pull
cd /var/www/emoncms/Modules
git clone https://github.com/emoncms/graph

Check for Database upates in Admin

2. Make Emoncms use graph module as default feed viewer

Add (or change) $feedviewpath = "graph/"; in settings.php

Change Log

V1

• Multigraph viewer
• One click add remove feed left/right axis control
• Stats on feed quality, min, max, diff (max-min) mean & stdev.
• View a histogram of any feed, time at value or kWh at power.
• Smooth a noisy feed to see what is going on, select between 0 and 10 point moving average.
• Export data as CSV, select time format and whether to show, remove or use the last value when data is missing.

Histogram

Smoothing

CSV Output

Further development

• Ability to save and load saved graph's
• Import graph built using graph into dashboards
• Tooltips
• Datapoint editor
• Selectable line or bar graph form on a per feed basis
• Selectable line color and fill.

Saved graphs produced with this module could replace vis module visualisations: realtime, rawdata, bargraph, graph, multigraph. Inbuilt data editing could replace edit daily and edit realtime. The histogram calculation on the fly can replace the need for the histogram input processor, feed engine and visualisations and on the fly averaging could replace the need for the PHPFiwa feed engine.

graph's Issues

TypeError: item.data is undefined

I get the following error when I try to open a graph after I did a git pull today.
The installation was done a couple of days ago, then it worked fine...

EmonCMS Error

Message: TypeError: item.data is undefined
URL: http://172.16.4.109/emoncms/Modules/graph/graph.js?v=1
Line: 622
Column: 17

Reverting back to commit 0922dae makes it work again...

graph buttons not responding in embed

More info on the community forum: https://community.openenergymonitor.org/t/graph-id-missing-buttons-not-working-v10-0-0/10953/3

thanks neil and @borpin

graphs don't handle phptimeseries properly

I've noticed various errors in the presentation of phptimeseries data by the graph module. They're described in a forum thread at https://community.openenergymonitor.org/t/graph-peculiarities-with-phptimeseries/9819

It seems clear these are bugs as no one has suggested any other explanation.

Graph module dependency on the config module for vue.js?

Shouldn't vue.js be moved to the main emoncms repo if multiple modules depend on it?

graph/view.php

Line 50 in 866bd60

<script src="<?php echo $path; ?>Modules/config/vue.js"></script>

This is the error message displayed when navigating to the graph page (or refreshing it) if the config module is not installed.

Legend breaks the boundaries of the plot area

Use of units in tag:feed names and "Undefined" units.

Just updated emoncms and graphs module on my live server and now have dozens of account each with dozens of dashboards and graphs etc, all now displaying an ugly "undefined" after every graph legend entry and in multiple places in the graph tooltips.

There are 2 issues here.

First and foremost is the over use of the units field when displaying feed names. A unit is part of the value not the value name. I previously mentioned this in the "added feed unit to tooltip and legend #31" PR comments.

This is an issue even if the unit is defined, for example we have key names used for the apps module that could result in a graph legend like so

emontx: solar_kwh kWh
emontx: export_kwh kWh
emontx: use_kwh kWh

Secondly, it shouldn't display "undefined" even in the correct locations, for this feature of displaying units (when defined) to be backwards compatible it must not display or just display an empty string if the unit is undefined as there are too many pre-existing feeds without units defined.

I am using latest emoncms/master and graph/master at time of writing. db updated and browser cache cleared.

How this relates to Modules\vis\visualisations\graph

Is this a replacement for \Modules\vis\visualisations\graph ?
Visualization graphs are modular also with the bonus that they are usable inside a dashboard, this module alone cant.
What is the idea?

issue downloading csv

when attempting to download a CSV of a graph's data I get this error in the downloaded file:

Warning: Invalid argument supplied for foreach() in /var/www/emoncms/Modules/graph/graph_controller.php on line 97

Warning: file_get_contents(http://localhost/emonmcs//feed/aget.json?id=12): failed to open stream: HTTP request failed! HTTP/1.0 406 Not Acceptable
in /var/www/emoncms/Modules/graph/graph_controller.php on line 130

"Date-time string",": ()"

I was able to overcome the error by marking the feed as public before attempting to re-download.

⚠ WARNING: SELECT ALL...

If you Select All-> Edit -> Make Feed Public then all the feeds come under a single node

Loading the feed/data.json api endpoint used in the graph controller works when logged in:
https://github.com/emoncms/graph/blob/master/graph_controller.php#L89

Seems to be an issue with php downloading the data from /feed before returning it as CSV - without being logged in ?

UI improvement: "Show Statistics" button should be more visible

The "Show Statistics" tag needs to be more prominent.

• Make text darker (black?)
• Make tab action more prominent

Extension of #35

Possibility for a 'stacked' bar graph?

maybe this is already possible and I didnt notice it, but I would like to display a bar graph where each data set is added onto one large bar in different colors rather than the superimposed bars in different colors now. (Similar than the graphs in 'sustainable energy without the hot air')
Will take a look at where in the code this could be achieved.

Second feed in legend shows "undefined" for tag and label

When I select two feeds (one left checkbox one right), the second feed label comes up as "undefined : undefined" in the graph legend. Looks like the handler wasn't adding the name and tag elements to the feedarray object. I made the push into a function to share the code between the two left and right checkbox so the two calls stay in sync.

diff --git a/graph.js b/graph.js
index 19f8ba6..6b91b4c 100644
--- a/graph.js
+++ b/graph.js
@@ -215,10 +215,7 @@ function graph_init_editor()
            }
         }

-        if (loaded==false && checked) {
-            var index = getfeedindex(feedid);
-            feedlist.push({id:feedid, name:feeds[index].name, tag:feeds[index].tag, yaxis:1, fill
-        }
+        if (loaded==false && checked) pushfeedlist(feedid, 1);
         graph_reloaddraw();
     });

@@ -239,7 +236,7 @@ function graph_init_editor()
            }
         }

-        if (loaded==false && checked) feedlist.push({id:feedid, yaxis:2, fill:0, scale: 1.0, delt
+        if (loaded==false && checked) pushfeedlist(feedid, 2);
         graph_reloaddraw();
     });

@@ -344,6 +341,11 @@ function graph_init_editor()
     });
 }

+function pushfeedlist(feedid, yaxis) {
+    var index = getfeedindex(feedid);
+    feedlist.push({id:feedid, name:feeds[index].name, tag:feeds[index].tag, yaxis:yaxis, fill:0,
+}
+
 function graph_reloaddraw() {
     graph_reload();
     graph_draw();

Cannot read property 1 of undefined error

While zooming into power view graph using a MyElectric 2 dashboards:

Display seconds in hover-over

It would be useful to have the seconds included in the hover-over details box, for example these 2 data points in the images below appear to be the same time when they are actually 30s apart.

Since sub-minute intervals are very common, the time indicated is too vague, most likely wrong more often that it's right if the time is taken at face value and not immediately understood to be to the nearest whole minute under (ie truncated not rounded).

Indicating left or right x-axis in the graph legends

Please see "Indicating left or right x-axis in the graph legends" on the OEM forum.

also opened similar request on emoncms/emoncms#678

Show statistics verses show options formatting

I have noticed a couple of minor items with the switching of views that may need addressing.

Firstly the feed names differ on each page as the unit is added (without any spacing) to the options page and not the stats page. (please take a look at #32 as the "no unit" stats page is better IMO), I would prefer to see the unit removed from the options page, but there should at least be a space, probably brackets?

I think it would look marginally better when switching views if the feed names appeared in the same place both horizontally (ie relocate the reorder arrows in the options page or indent the feed names in the stats page) and vertically, I understand the line sizes are different due to the dropdowns in one view and not the other, but would it be possible to force the lines to be bigger when no dropdowns are used? When you have longer lists, this vertical alignment would make it easier to focus on a single line when switching views rather than looking up the feed name again.

Related to the item directly above (so may change with any fix to the above) the views seem to be anchored at the bottom rather than the top, so when clicking between views, the whole page moves to keep the bottom of the stats and options boxes aligned. If you look closely at the pictures above and note the position relative to the lefthand menu items you can see the bottom is static, but the top has moved, this also moves the button in that header bar, so when switching back and forth you need to move the pointer up and down to find the button each time.

The combination of the 2 items above mean the users eye have to keep flitting about to see the button position and to re-lookup each feed line (with longer lists) rather than holding the mouse static and clicking to and fro whilst concentrating on any one line or working though the list one after the other.

Errors when zooming out too far on some graphs

Notice: Undefined offset: 1 in /var/www/emoncms/Modules/feed/feed_model.php on line 607 [{"feedid":"1","data":{"success":false,"message":"request end time before start time"}}], Notice: Undefined offset: 0 in /var/www/emoncms/Modules/feed/feed_model.php on line 607

Display feed units in hover over overlay and key and feeds table below graph

Daily average show tooltips aligned to 12am?

May be related to #39

Embedded graphs do not honor using an apikey to view

When viewing a dashboard with a single embedded graph (created by graph module) the data is only present whilst fully logged in or by setting the feeds to public.

When fully logged in this is the dashboard seen with the graph populated with data

When not logged in and the read (or the write) apikey is included eg https://myserver/emoncms/dashboard/view/occupancy&apikey=abc123etc we see the page, the graph but no data

however that same data can be seen on a different dashboard when logged out and using just the read apikey in the url such as https://myserver/emoncms/dashboard/view/main&apikey=abc123etc

The only way I seem to be able to get the data to show in the graph is by setting all the feeds concerned to public which is not desirable nor needed for other dashboards.

CSV time output format

In the forum thread https://community.openenergymonitor.org/t/timestamp-issue-when-importing-csv-exported-data-from-emoncms-org-feeds-list/7863/19 we discussed time stamp formats for the CSV output in the graphs module.

Do we need to amend the available output format?

The output format is defined here https://github.com/emoncms/graph/blob/master/graph.js#L837

I’ve been using “2017-03-14 00:00:00” format for a long time, Im not entirely sure why TBH, perhaps because it contains everything you need in a condensed format?

I usually convert Unix time stamp to this format with:

$datestr = date("Y-m-d H:i:s",time());

luckily php’s strtotime converts a large variety of time formats back to unix time :slight_smile: I dont really have a strong opinion on this either way

Use new multi feed/data & feed/average api for improved performance

See graph_reload

https://github.com/emoncms/graph/blob/master/graph.js#L554

https://github.com/emoncms/graph/blob/master/graph.js#L575

Download CSV file button

Following on from #22 and ref forum thread https://community.openenergymonitor.org/t/exporting-multiple-feeds-to-csv/8084?u=pb66

It would also be a great addition to be able to download the CSV output from the graphs page to a CSV file, I know it is possible to copy'n'paste (or use the new "copy to clipboard" button I assume?) and create your own local csv file, but it would be nicer to have emoncms do it for you.

What would also be of great use would be to make that download a user accessible api eg

https://emoncms.org/graph/download.csv?id=12345&apikey=1234abcd

or by graph name

https://emoncms.org/graph/download.csv?name=mygraph&apikey=1234abcd

that way an automated download of a certain set of data feeds and dates can be set up by simply creating and saving a graph.

(It may need an optional "userid" in the request to allow public downloads without an apikey.)

The emoncms/graph/readme.md's changelog does actually say V1 will

Export data as CSV, select time format and whether to show, remove or use the last value when data is missing.

but I suspect that may mean in-directly using copy'n'paste as I can't recall seeing download button.

The "cherry" in this feature request would be an optional "download" button displayed with the embedded graph control buttons so that users can download what they see as a csv file direct from emoncms dashboards. (showing or not showing the download button should be a graph setting like the show tags or show legend etc)

[edit] Essentially the "download" api call in graph would provide a sort of pre-defined multiple feeds download to a (potentially) set timespan eg last month's solar and use, whilst also giving it a short name ie using the graph name rather than recalling all the feedids start time and end time etc etc to create a full feed request api.

Whilst I prefer CSV and that is partially already implemented, it might be worth considering a "download JSON" api too perhaps in time?

https://emoncms.org/graph/download.json?id=12345&apikey=1234abcd

which replies with a list of feed name:values for each listed timestamp.

Error displaying values when mouse over with the mouse

in the Graph module, only half of the values are displayed when moving the mouse over the graphs.
The reason is the superimposed DIV "#legend".
I have set the width from "100%" to "auto".
Thus the designation of the graphs does not overlap the entire width and height and I can display all values again.
Is it possible to copy this workarround into the master source texts?
Thank you

display realtime power as kw in dashboard

related to emoncms/dashboard#230

Graph view and saved dashboards are not available publicly

graph id not shown

Statistics missing

The statistics (mean etc) of the old visualisation are missing in graph Module.

New split legend alignment issue

The right aligned legend for the RH axis seems to over hang the graph axis

each entry seems to be left aligned within the legend as a longer feed name moves the LH edge of the legend to the left

this is running the latest emoncms and graph master branches

as seen in latest Chrome browser

Version 72.0.3626.119 (Official Build) (64-bit)

[Is this to do with the additional unit being added to the feed name? Please reconsider issue #32]

Vue not defined

@emrys

This has hapend in the last 48 hours

Server Information

Server Information

Services

• feedwriter :- Active Running - sleep 60s loading...
• redis-server :- Active Running

Emoncms

• Version :- 10.1.6
• Modules :- Administration | EmonHub Config v2.0.4 | Dashboard v2.0.4 | Device v2.0.1 | EventProcesses | Feed | Graph v2.0.4 | Graph v2.0.4 | Group v2.0.0 | Input | CoreProcess | Schedule | Time | User | Visualisation
• Git :-
  • URL :- https://github.com/emoncms/emoncms.git
  • Branch :- * master
  • Describe :- 10.1.6-55-gdc2e870f

Server

• OS :- Linux 4.9.0-8-amd64
• Host :- vps640695 | vps640695.ovh.net | (51.38.69.188)
• Date :- 2019-09-18 17:46:48 BST
• Uptime :- 17:46:48 up 231 days, 6:36, 2 users, load average: 0.30, 0.40, 0.38

Memory

• RAM :- Used: 11.34%
  • Total :- 7.62 GB
  • Used :- 883.93 MB
  • Free :- 6.75 GB

Disk

• / :- Used: 12.78%
  • Total :- 78.74 GB
  • Used :- 10.06 GB
  • Free :- 65.45 GB

HTTP

• Server :- Apache/2.4.25 (Debian) HTTP/1.1 CGI/1.1 80

MySQL

• Version :- 5.5.5-10.1.37-MariaDB-0+deb9u1
• Host :- localhost:6379 (127.0.0.1)
• Date :- 2019-09-18 17:46:48 (UTC 01:00‌​)
• Stats :- Uptime: 19982195 Threads: 1 Questions: 239248992 Slow queries: 0 Opens: 420 Flush tables: 1 Open tables: 65 Queries per second avg: 11.973

Redis

• Version :- 3.2.6
• Host :- localhost:6379
• Size :- 9030 keys (2.72M)
• Uptime :- 231 days

PHP

• Version :- 7.0.33-0+deb9u1 (Zend Version 3.0.0)
• Modules :- apache2handler | calendar v7.0.33-0+deb9u1 | Core v7.0.33-0+deb9u1 | ctype v7.0.33-0+deb9u1 | curl v7.0.33-0+deb9u1 | date v7.0.33-0+deb9u1 | dom v20031129 | exif v7.0.33-0+deb9u1 | fileinfo v1.0.5 | filter v7.0.33-0+deb9u1 | ftp v7.0.33-0+deb9u1 | gettext v7.0.33-0+deb9u1 | hash v1.0 | iconv v7.0.33-0+deb9u1 | json v1.4.0 | libxml v7.0.33-0+deb9u1 | mbstring v7.0.33-0+deb9u1 | mcrypt v7.0.33-0+deb9u1 | mysqli v7.0.33-0+deb9u1 | mysqlnd vmysqlnd 5.0.12-dev - 20150407 - $Id: b5c5906d452ec590732a93b051f3827e02749b83 $ | openssl v7.0.33-0+deb9u1 | pcre v7.0.33-0+deb9u1 | PDO v7.0.33-0+deb9u1 | pdo_mysql v7.0.33-0+deb9u1 | Phar v2.0.2 | posix v7.0.33-0+deb9u1 | readline v7.0.33-0+deb9u1 | redis v4.2.0 | Reflection v7.0.33-0+deb9u1 | session v7.0.33-0+deb9u1 | shmop v7.0.33-0+deb9u1 | SimpleXML v7.0.33-0+deb9u1 | sockets v7.0.33-0+deb9u1 | SPL v7.0.33-0+deb9u1 | standard v7.0.33-0+deb9u1 | sysvmsg v7.0.33-0+deb9u1 | sysvsem v7.0.33-0+deb9u1 | sysvshm v7.0.33-0+deb9u1 | tokenizer v7.0.33-0+deb9u1 | wddx v7.0.33-0+deb9u1 | xml v7.0.33-0+deb9u1 | xmlreader v7.0.33-0+deb9u1 | xmlwriter v7.0.33-0+deb9u1 | xsl v7.0.33-0+deb9u1 | Zend OPcache v7.0.33-0+deb9u1 | zlib v7.0.33-0+deb9u1

Client Information

Client Information

HTTP

• Browser :- Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.132 Safari/537.36
• Language :- en-US,en;q=0.9

Window

• Size :- 1903 x 969

Screen

• Resolution :- 1920 x 1080

js error when user logged out

this occurs when visiting /emoncms/graph if you're logged out

Scrollbar on sidebar overlaps the right-hand tickbox

Change background colour when embedded

See: https://community.openenergymonitor.org/t/emoncms-to-replace-owl-udp-and-multicast/13166

Call to a member function fetch_array() on boolean

on my self-hosted emoncms after an upgrade to emoncms 9.7.3 I get the following on dashboard view /emoncms/dashboard/view/elec

Fatal error: Call to a member function fetch_array() on boolean in /var/www/arthur/emoncms/Modules/graph/graph_model.php on line 91

moving graph out of modules fixes the bug and the dashboards appear ok.

When graph is there the "graph" section seems to work fine.

Sidebar branch: sidebar not populating when opening from Feed List

When clicking on a feed in the Feed List graph viewer opens and the sidebar is blank.

If I open graph viewer manually by selecting 'Graphs' from the sidebar then the sidebar correctly populates:

The sidebar should populate correctly if a feed is opened to view from the Feed List

Collapsing lefthand panel and maximizing graph widths

I would really like to be able to collapse the lefthand panel after selecting feeds or a saved graph to give me more screen for the graph itself.

I get why it's there by default, but I would say I spend 99% of my time playing with the graphing and ignoring the lefthand panel, the panel already hide's with small screens, on medium sized screens the graph sizes are reduced by the lefthand panel and even with much larger screens the graph size is restricted.

Can we please make the lefthand panel permanently user collapsible and let it default to closed on smaller screens?

And also lift the maximum graph width when $fullwidth = true; is set in settings.php.

Accessing graphs directly with apikey

It would be very useful to be able to access any saved graph via a url and apikey eg

http://server.com/emoncms/graph/somegraph?apikey=abc123

This would make frequently visited graphs bookmarkable and save logging in.

I'm aware that graphs can be embedded in dashboards, but 1) the dashboards do not populate when a dash is viewed by apikey (see #8) and 2) only the graph is displayed, not the stats and/or the ability to download csv etc.

error while changing series colour in legend

This was raised in the emoncms repo emoncms/emoncms#1228

Bug. Embed graphs right y axis ignore the manual min and max.

When configuring manual scale on the right hand y axis the embed.php doesn't pass the scale values to the graph and as a result the right yaxis always auto scales when embedded in a dashboard.

Copy to clipboard button

I think it would be a useful feature to have a "copy to clipboard" button (like for the server information on the emoncms/admin page) for the CSV output, perhaps placed at the end of the line for show/hide and other CSV options.

I currently use crtl-a ctrl-c without an issue but a button would be a nice improvement.

feedlist[0].data undefined

https://community.openenergymonitor.org/t/emoncms-error-message/11294

position of tooltip and feeds panel

The tooltip showing an individual data value and time is very useful. When the browser window is narrow enough that the feeds panel hides and the graph fills the full width of the browser window, it also sensibly adjusts its position (moves right relatively) so it is not clipped by the left edge of the browser window when the data point is near the left side of the graph.

BUT ... when the feeds panel is displayed, either because the browser is wide or because the user has chosen to overlay it on the graph, the tooltip is displayed BEHIND the feeds panel where it cannot be read.

The tootltip should either be moved right so it is completely visible (best) or displayed in front of the feeds panel instead of behind it.

Collapsible stats table

When working with a large number of feeds the page becomes very long and awkward to use. for example the page below is one that I typically encounter and want to zoom in on a point and then download the csv, that's a lot of scrolling up and down due to the stats table.

It would be useful if the stats table was collapsible, (perhaps even collapsed by default so the graphing area could be bigger).

sidebar branch - when in mobile view the sidebar open on first click of burger icon (in the graph view)

multi-page tabbed views and opening in modal from embedded graphs

This suggestion combines several existing issues/topics so rather than contaminating any one issue, I started this new issue.

There is a lot going on on the graphs page and there is potential for a lot more functionality if screen space permitted it.

@emrysr suggestion of a tabbed approach to improve the "show statistics" buttons visibility (#36) is IMO a good one that could be extended further and include the graph itself as the first (default) tab/page and the download csv as the last tab/page (#29).

| Graph | Statistics | Options | Download |

This way lots of info can be displayed in the same space without lots of scrolling or overloading the user with an extremely busy page. And with saved graphs potentially being made directly accessible (#27) embedded graph could have a new button so that from any (dashboard) embedded graph a user can open that graph in a modal view with all tabbed views available and the lefthand graphs nav panel totally disabled in modal view only (related to #25).

I know @TrystanLea is not keen on the tabbed suggestion @emrysr suggested to resolve the issue raised by @glynhudson so I will not dwell on the format or mechanics of it, but IMO having 4 (or more) pages occupy the same screen space and group functionality/feature so as not to overwhelm the user or distract from the task at hand, whilst all the other functions are only a click away could amalgamate all the linked design "issues" into one compact space, whether it be tabs or some other method.

The "tabbed" (or similar) layout would also work elsewhere in emoncms eg the admin page is getting a bit busy and things like "admin/users" and "check and update database" etc could be on separate pages (tabs?), likewise emonhub config could serve multiple emonhubs (remote devices or emonesps too?), one under each page/tab.

I personally like the tabbed approach as it permanently shows what sub-pages are available and which one I'm on currently, all in the current window. no hidden dropdowns, no new browser windows, no page refreshing/resetting/redirecting.

perhaps this might also allow an easy way to add in app help for specific pages by having a "help" tab too eventually?

Scale both Y Axis

related to issue on emoncms repo - emoncms/emoncms#1132

time labels are offset

The time labels are sometimes placed in the wrong place. I don't know under what conditions it occurs, but I've attached an example. The vertical lines mark the day boundaries and the labels are ambiguously centred below them. It would make more sense if they had their left hand edges aligned with the vertical lines, or if they were centred in the interval that represents the day the label describes.

Hmm, github won't let me attach a FF screenshot png showing the problem, sorry. 'Something went really wrong, and we can't process that file'. Anyway it's just a weekly graph of any data item - it's always the same.

Dashboard is ignoring graph's legend and tag flags

(relocated here from emoncms/emoncms#638)

I've configure a graph with multiple bars, a fixed interval, etc. but I don't want to see the legend info so I've unchecked 'legend' and 'feed tag' flags. I've save the graph.

When I add that graph on a dashboard, the legend is shown, but if I go again to the graphs page and open the saved graph, the flag is unchecked and I don't see the legend.

Maybe it's a problem related to the dashboard module, but I don't know.

adjust for timezones in graph view

related to emoncms issue: emoncms/emoncms#1060

Right colomumn with checkboxes doesnt appear

Dear developers, faced with problem that choosing second axis unavailible with long feed's names. Scrolling of frame doesnt helps

In day or month view, show consumption up to present time

https://community.openenergymonitor.org/t/monthly-cumulative-costs-graph-not-displaying/13194