cfsotago / greengriddata Goto Github PK

Would it be possible to add the 'ethnicity' variable from the EC2 survey to the safe data package?

For household rf_31, hot water electricity has been recording all zeros since 26 Feb 2016.

collected in surveys, needs to be added to help explain heat pump use

Where can I find the actual answers to survey questions around hot water cylinders specifications and occupant usage behaviour as answered by occupants for each house in the dataset?

It is possible that where a clamp goes around a bundle of wires one is running the opposite direction to give a negative value. There is no real way to determine when this might be the case.

Multiple phases in one house could also cause issues if they were put through the same clamp. The contracted installers should have been aware of this but...

(h/t Jefferson)

Seems to have been omitted. NB - this is just the gender of the respondent so does not indicate much about the household...

Not a requirement but we would be very interested to know what users are (planning to) do with the data (get it from http://reshare.ukdataservice.ac.uk/853334/). Links to any github repos, projects or publications based on the data would be very welcome :-)

The answers provided in ggHouseholdAttributesSafe.csv don't always make sense in the context of the questions given in Table 9.1 of NZ_GREEN_GridHouseholdElectricityDemandStudy_HouseholdAttributeData_v1.0.pdf. An example is provided in the images that seems to imply everyone has a heated swimming pool, 3 toasters, either 1 or three top-loader washing machines AND either 1 or 3 front-loaders, etc. Am I interpreting something wrong?

@jkmair notes:

There are duplicated (or missing) observations if you use NZT in the DST break hours. e.g.:

• The hours 02:00-02:59 NZ time are repeated at the end of DST in April.
  r_dateTime NZT
  2015-04-04T13:59:00Z 5/04/15 2:59
  2015-04-04T14:00:00Z 5/04/15 2:00
  ... ...
  2016-04-02T13:59:00Z 3/04/16 2:59
  2016-04-02T14:00:00Z 3/04/16 2:00

• There seems to consistently be a one hour period removed at
  11:00-11:59UTC on the date of DST ending. However, this does not lineup
  with the NZ time of DST. Not sure if this is a gridSpy issue or data
  processing.
  r_dateTime NZT
  2015-04-05T10:59:00Z 5/04/15 22:59
  2015-04-05T12:00:00Z 6/04/15 0:00
  ... ...
  2016-04-03T10:59:00Z 3/04/16 22:59
  2016-04-03T12:00:00Z 4/04/16 0:00

• A one hour period corresponding to the start of DST, 14:00-14:59 UTC,
  is removed. This results in a 2 hour gap in NZ time.
  r_dateTime NZT
  2015-09-26T13:59:00Z 27/09/15 1:59
  2015-09-26T15:00:00Z 27/09/15 4:00
  ... ...
  2016-09-24T13:58:00Z 25/09/16 1:58
  2016-09-24T15:00:00Z 25/09/16 4:00

-> For some reason the duplicate entries have a TZ_orig of "date NZ", while all of the
non-duplicate entries are "date UTC".

How is (or should) DST be handled?

Hi, I am trying to access the dataset on the UK data service website, however, I am struggling to access the data because my institute, Stellenbosch University in South Africa, is an 'Unusable Identity Provider'. Could you please help me out with accessing the data?

Thanks in advance!

We split 15 & 17 into 15a/15b & 17a/17b due to re-use of monitor. Users should remove:

• rf_15_all_1min_data.csv.gz
• rf_17_all_1min_data.csv.gz

from the data archive if they have them.

We will remove it from future releases.

This was noticed with hot water as -1kw for a few days instead of what was usually +2kw. We consider this was probably a gridspy error due to the current clamp being set to a different phase temporarily. It can be rectified it by multiplying any negative values by -2. (h/t Jefferson)

Note that this correction has not been implemented in the data. The user should deal with this as they see fit.

Basically the input dates are a mess. See https://github.com/dataknut/nzGREENGridDataR/blob/a70d9d4fc7a4ee8406cda2c8bb458bd324ff6f43/R/gridSpyData.R#L309

extractCleanGridSpyCircuit is having problems with rf_15

Error in rbindlist(l, use.names, fill, idcol) :
Item 2 has 4 columns, inconsistent with item 1 which has 5 columns. To fill missing columns use fill=TRUE.
In addition: Warning message:
The following named parsers don't match the column names: linkID

When I delete rf_15 as per instructions here #19 it also fails with error message:

Error: '~/HWC-bookdown/Masters/powerData/rf_15_all_1min_data.csv.gz' does not exist.

Running git clone gives a folder around 234M in size although the files present total 10M. This is due to images from checkPlots/ and various derived HTML files in the history. It should be straightforward to rewrite some of that git history, but that doesn't mean it's worth doing (I'm happy to look into it, but if so we should rewrite before publication since the commit IDs will change).

The hot water circuit on rf_33 looks suspiciously like it has changed houses partway through. An alternative explanation would be that during a period which no data was being collected, a significant change to the element or usage patterns of the household took place.

Two Grid Spy monitoring kits were re-used during the project. These were:

• rf_15
• rf_17

The household attribute data provides information on the dates these were switched to new households (see the household attribute report). The relevant checkPlots also give you a clue.

Inspection of the raw data shows that rf_46 has a lot of different circuit labels:

• some have the form xx_Imag$nnn - these may be 'imaginary power'
• one is voltage (it says so!)
• some have the form xx1$nnn or xx2$nnn - these may or may not be power, imaginary power or voltage
• some are 'normal'

rf_46 was not re-used and the similarity of the label-sets seems to indicate typos.

I'm getting a non-zero exit status when trying to install onto a Windows machine. I have not had this issue when previously installing on Linux.

NOTE: I am not using the most recent versions of digest or data.table. The most recent versions of these two packages require compiling from source which is proving difficult. I have the most recent binary versions of these packages installed. I don't think this would be the problem but it may be.

My guess is that when the required data files are called using the the read.table() function the 'slashes' in the directory path are facing the 'wrong' way, i.e. for Unix filesystems.

I have pasted the Console output below. I'll keep working on a fix and report back once successful, but in the meantime let me know if you have any ideas.

devtools::install_github("CfSOtago/GREENGridData")
Downloading GitHub repo CfSOtago/GREENGridData@HEAD
These packages have more recent versions available.
It is recommended to update all of them.
Which would you like to update?

1: All
2: CRAN packages only
3: None
4: digest (0.6.25 -> 0.6.26) [CRAN]
5: data.table (1.13.0 -> 1.13.2) [CRAN]

Enter one or more numbers, or an empty line to skip updates:3
√ checking for file 'C:\Users\ParkerR\AppData\Local\Temp\RtmpSeTpqr\remotes53407a2343e8\CfSOtago-GREENGridData-3417279/DESCRIPTION' ...

• preparing 'GREENGridData': (5.6s)
  √ checking DESCRIPTION meta-information ...
• excluding invalid files
  Subdirectory 'R' contains invalid file names:
  'README.md'
  Subdirectory 'man' contains invalid file names:
  'README.md'
• checking for LF line-endings in source and make files and shell scripts
• checking for empty or unneeded directories
• building 'GREENGridData_1.0.tar.gz'

Installing package into ‘C:/Users/ParkerR/Anaconda3/envs/r/Rpackages’
(as ‘lib’ is unspecified)
'\aklfs02pi\home1$\ParkerR\Documents'
CMD.EXE was started with the above path as the current directory.
UNC paths are not supported. Defaulting to Windows directory.

• installing source package 'GREENGridData' ...
  ** using staged installation
  ** R
  ** data
  *** moving datasets to lazyload DB
  Warning in read.table(...) :
  incomplete final line found by readTableHeader on 'C:\Users\ParkerR\Anaconda3\envs\r\Rpackages\00LOCK-GREENGridData\00new\GREENGridData/data/circuitsToSum_v1.0.csv'
  Warning in read.table(...) :
  incomplete final line found by readTableHeader on 'C:\Users\ParkerR\Anaconda3\envs\r\Rpackages\00LOCK-GREENGridData\00new\GREENGridData/data/circuitsToSum_v1.0.csv'
  Warning in read.table(...) :
  incomplete final line found by readTableHeader on 'C:\Users\ParkerR\Anaconda3\envs\r\Rpackages\00LOCK-GREENGridData\00new\GREENGridData/data/circuitsToSum_v1.1.csv'
  Warning in read.table(...) :
  incomplete final line found by readTableHeader on 'C:\Users\ParkerR\Anaconda3\envs\r\Rpackages\00LOCK-GREENGridData\00new\GREENGridData/data/circuitsToSum_v1.1.csv'
  Warning in read.table(...) :
  invalid input found on input connection 'C:\Users\ParkerR\Anaconda3\envs\r\Rpackages\00LOCK-GREENGridData\00new\GREENGridData/data/dstNZDates.csv'
  Error in read.table(...) : no lines available in input
  ERROR: lazydata failed for package 'GREENGridData'
• removing 'C:/Users/ParkerR/Anaconda3/envs/r/Rpackages/GREENGridData'
  Error: Failed to install 'GREENGridData' from GitHub:
  (converted from warning) installation of package ‘C:/Users/ParkerR/AppData/Local/Temp/RtmpSeTpqr/file53404cd9fd0/GREENGridData_1.0.tar.gz’ had non-zero exit status

using... https://github.com/CfSOtago/GREENGridData/blob/master/examples/code/extract_per_house_totals.r (which needs moving to the dataProcessing folder)

Most of the question numbers in the household attributes file are not self-explanatory. The survey labels files in /publicData should help. We are trying to locate a transcript of the original household survey.

To create data archive version 2.0

Can be checked by seeing if hot water was ever larger than incomer. (h/t Jefferson)

On the https://cfsotago.github.io/GREENGridData/ page under "Publications to date" the link to "A minimal simulation..." https://www.sciencedirect.com/science/article/pii/S0306261917316197 is broken.
I would have fixed it myself but can't find the right file.