jufemaiz / aemo

AEMO Gem for managing Australian Energy Market data formats

Hi there 👋

• 🔭 I’m currently working on Enosi's Powertracer platform
• 🌱 I’m always learning
• 💬 Ask me about the intersection of technology and energy – I've spent nearly twenty years working on the tough challenges in this space!
• 📫 How to reach me: reach out on 🐦 @joelmcourtney or 🐘 @[email protected]
• 😄 Pronouns: he/him/his
• ⚡ Fun fact: I'm an avid rider of bicycles - even taking my bike with me on a work trip to Japan!

aemo's Issues

AEMO::Region does not reference which AEMO Node it falls in

ACT is within NSW in terms of the market, but is a separate Region.

AEMO::NMI#tni_values should return Array of Hashes not an Array of Arrays of Hashes

https://github.com/jufemaiz/aemo/blob/master/lib/aemo/nmi.rb#L716

Change:

possible_values.map { |x| x['mlf_data']['loss_factors'] }

to

possible_values.map { |x| x['mlf_data']['loss_factors'] }.flatten

Drop support for Ruby versions prior to 2.3

Lonely operator support was introduced to Ruby in 2.3.

Ruby 2.2.8 is currently at EOL support with stable releases at 2.3.5 and 2.4.2 (with 2.5.0-preview1 available for review). Src: https://www.ruby-lang.org/en/downloads/

If there is no opposition to dropping 2.2.8 support I will be dropping support by EOY.

Parse NEM12 raises runtime error if file has a blank meter configuration

Test file:

100,NEM12,201710271425,COMPA,COMPB
200,NEEE000010,,001,E1,N1,7416444,KWH,30

Expected result

raise ArgumentError, 'NMIConfiguration is not valid'

Actual result

undefined method `match' for nil:NilClass

JSON dependency open-ended

WARNING:  open-ended dependency on json (>= 1.7.5) is not recommended
  if json is semantically versioned, use:
    add_runtime_dependency 'json', '~> 1.7', '>= 1.7.5'
WARNING:  See http://guides.rubygems.org/specification-reference/ for help

AEMO::NEM12 » 300 row » IntervalValue1 . . . IntervalValueN with nil value not raising an error

NEM12 file was parsed raising:

NoMethodError: undefined method `match' for nil:NilClass Did you mean? catch

This was caused due to a 300 row with the following representative data:

300,20180529,0.001,0.0,0.0,0.001,0.0,0.0,0.001,0.0,0.001,0.0,0.0,0.001,0.005,0.01,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.001,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,,,,,,,,,,,,,,,,,A,,,20180529160239,

The empty values should not have been empty but rather 0, with a flag of N raised for them. This is due to NEM12 export functionality being used on a dataset that was for the present day.

What should happen is that an exception is raised during the parsing of the dataset as the value present is not valid.

A valid dataset would look like:

300,20180529,0.001,0.0,0.0,0.001,0.0,0.0,0.001,0.0,0.001,0.0,0.0,0.001,0.005,0.01,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.001,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,A,,,20180529160239,
400,1,32,A,,
400,33,48,N,,

This would signify no value for the last 16 elements of the 48 provided here (30 minute interval data).

Likewise, 0 could be a valid value and be without the subsequent 400 rows (but not for the example above)

Timezone needs to be set to NEMTIME

Time parsing needs to be undertaken throughout in UTC+10:00 (aka NEMTIME)

AEMO::NMI::NETWORKS have no AEMO::Region represenation

Changes to method flags to support 5 min settlement and same day churn

Additional method flags (from 1 May 2022):

Type	EST and/or SUB	Descriptor
Type 20	SUB	Churn Correction (Like Day)
Type 21	SUB	Five-minute No Historical Data
Type 59	SUB, EST	Five-minute No Historical Data
Type 69	SUB	Linear extrapolation

https://aemo.com.au/-/media/files/electricity/nem/retail_and_metering/market_settlement_and_transfer_solutions/2022/metrology-procedure-part-b-73.pdf?la=en

CVE on rubyzip v1.1.7

https://gemnasium.com/github.com/jufemaiz/aemo
https://gemnasium.com/gems/rubyzip/versions/1.1.7

Investigate the use of csvreader

Ref:

• https://github.com/csv11/csvreader
• https://rubygems.org/gems/csvreader

Further reading:

• https://github.com/csv11/docs/blob/master/why-the-csv-stdlib-is-broken.md

Update Nokogiri

Ref: https://snyk.io/test/github/jufemaiz/aemo