openwateranalytics / epanet-rtx Goto Github PK

Issue by jamesuber
Tuesday May 13, 2014 at 04:25 GMT
Originally opened as https://github.com/CitiLogics/epanet-rtx/issues/12

Something weird is happening when resampling valid range TS, causing the resampler to hang. Have not looked at this in code as yet; just in configurator. occurs in r21 and r22. The valid range is ValidRange::drop and neither has a PR.

I think that

if (! (time >= start && time <= end) ) {

should be

if (! (time >= start && time < end) ) {

based on what is in Clock::timeValuesInRange -- which indicates that the points method should be returning point in the left-closed interval [t1,t2)

Issue by jamesuber
Friday Apr 18, 2014 at 18:39 GMT
Originally opened as https://github.com/CitiLogics/epanet-rtx/issues/10

try feeding a constant with units of minutes and a flow timeseries with units of gpm into a multiplier. units should be volume, but remain in flow units.

Issue by jamesuber
Tuesday Aug 05, 2014 at 17:19 GMT
Originally opened as https://github.com/CitiLogics/epanet-rtx/issues/15

having min and max over the previous clock cycle would allow, for example, easy construction of minimum nightly flow status when applied to DMA demand TS.

Issue by jamesuber
Friday Aug 16, 2013 at 16:11 GMT
Originally opened as https://github.com/CitiLogics/epanet-rtx/issues/1

If units were saved in the db, we wouldn't have to remember what they were. For example when persisting rtx timeseries data for the purpose of reusing it later, somewhere.

I think that the behavior would be that db data would automatically be converted to the units specified for the timeseries object. And if that timeseries object didn't specify any units, then they would be inherited from the db, with no conversion needed.

reset() does "TRUNCATE TABLE points" which erases all the points in the PointRecord, ignoring the series_id (or, the timeseries_meta information).

This coarse strategy seems to prohibit, for example, sequentially adding new ModularTimeSeries to the same PointRecord, because that PointRecord is truncated every time a new ModularTimeSeries is created and connected to a source (ModularTimeSeries::setSource and ModularTimeSeries::setUnits both logically use reset() ).

Is it logical for reset() to do a more complicated DELETE FROM points WHERE series_id = ? That would seem to fix any use of resetCache by a ModularTimeSeries to only affect the relevant data in the table.

Issue by jamesuber
Monday May 19, 2014 at 14:30 GMT
Originally opened as https://github.com/CitiLogics/epanet-rtx/issues/13

I believe that in PointRecordTime.h the statement

#include <sqltypes.h>

is a dependency on iodbc. only mentioning it if we want the sqlite-only build to truly be sqlite only.

found this out after a clean install of mavericks; now I have nothing.

Issue by jamesuber
Wednesday Feb 05, 2014 at 22:51 GMT
Originally opened as https://github.com/CitiLogics/epanet-rtx/issues/9

we should have one

There is one issue in Linux where when you compile the epanet-rtx the code is looking for "junction.h", but the file is actually "Junction.h", and the compiler throws multiple errors because Linux is case sensitive. A simple fix, but one I thought you might want to know about. This happens on multiple files.

Chris

Issue by jamesuber
Tuesday Aug 05, 2014 at 16:45 GMT
Originally opened as https://github.com/CitiLogics/epanet-rtx/issues/14

stats should compute the stat for all points in the defined backward looking time window, for whatever time that terminates the window -- whether that time is on a regular or irregular clock.

need to have an epanet repo here on GH

Issue by jamesuber
Tuesday Aug 05, 2014 at 18:32 GMT
Originally opened as https://github.com/CitiLogics/epanet-rtx/issues/16

Start with hourly source ts data, followed by lag by 1/2 hour, followed by resampling. The resampler does not bridge across the two points on either side of midnight.

Issue by jamesuber
Monday Apr 21, 2014 at 14:29 GMT
Originally opened as https://github.com/CitiLogics/epanet-rtx/issues/11

A multiplier with one TS in minutes and the other in gpm will correctly list the units options, but they aren't applied successfully. (i.e. change to mvd; open props again, still gpm.) The calcs seem correct, but plotting the data generates an "unknown units" label.

Issue by jamesuber
Monday Oct 06, 2014 at 00:02 GMT
Originally opened as https://github.com/CitiLogics/epanet-rtx/issues/18

Fascinating that even the computation of the median of a series, when approached at an expert level and an eye to efficiency, is not straightforward: https://svn.boost.org/trac/boost/ticket/6992

I think this explains some things I've been seeing using the median filter that just didn't look right. I did confirm that using boost median with three data points always just gives you the last one.

Bottom line is that we shouldn't use the median unless we think there will be at least 5 points.

In the timeseries_demo application, it is illustrating the change in units using the MovingAverage->setUnits method,

movingAverage->setUnits(RTX_GALLON_PER_MINUTE);
printPoints( movingAverage->points(start, start+240) );

but the points method is returning points (and printPoints is printing them out) in the original units before the change.

When these same points are persisted, however, the units change is in effect.

Under gcc the following typical lines of code will fail:

friend std::ostream& operator << (std::ostream & out, TimeSeries &ts);

This is because gcc (2010) will not allow you to forward declare across namespace boundaries.

Also, if your inp file has no reservoirs,an "abort" will be raised in setSimulationParameters. An easy fix is to add a single reservoir to your .inp file.

Hello ,I am using Epanet-RTX to develop my program . But now, I have a question ,I don't konw the relationship between OdbcPointRecord and OdbcPreparedPointRecord? still the OdbcDirectPointRecord? How to Use them? Thanks for your reply.

When ConfigFactory creates modular time series objects, and then connects sources together, it does so in (probably) alphabetic order. it should instead connect from the source-most series, and go through the chain iteratively. that way, units and clocks are set by default rather than having to specify them in the .cfg

Issue by jamesuber
Thursday Oct 17, 2013 at 16:46 GMT
Originally opened as https://github.com/CitiLogics/epanet-rtx/issues/5

could be useful. would calculate a variance (more likely a standard deviation, I think) averaged over a time window, where the mean over that window was defined by a suitable moving average.

One design could require specifying the function to use as the mean, requiring only a compatible clock or resampling capability. that would allow flexibility in specifying, say, a constant (resulting in a running measure of the signal variability), or a moving average with an independent averaging window from the moving variance.

I wish the ConfigFactory to understand it.

Issue by jamesuber
Sunday Aug 18, 2013 at 14:27 GMT
Originally opened as https://github.com/CitiLogics/epanet-rtx/issues/3

See the comparison grab from the configurator, for NKWD.

Isn't the moving average of the moving average of the sum supposed to equal the sum of the moving average of the moving average??!

In response to

  vector< Point > somePoints;
  somePoints = someTimeSeries->points(1222869600,1222870200);

where someTimeSeries is backed by a MysqlPointRecord, following is the record of db queries, from the log file. somePoints contains exactly what I expected, but I was confused about why two queries are being conducted for each time, both a 'time >' and a 'time ='. Could this be a possible optimization opportunity.

130126 12:46:19 648 Execute SELECT time, value FROM points INNER JOIN timeseries_meta USING (series_id) WHERE name = 'L ReservoirCheckValve flow' AND time = 1222869600
648 Execute SELECT time, value FROM points INNER JOIN timeseries_meta USING (series_id) WHERE name = 'L ReservoirCheckValve flow' AND time > 1222869600 LIMIT 1
648 Execute SELECT time, value FROM points INNER JOIN timeseries_meta USING (series_id) WHERE name = 'L ReservoirCheckValve flow' AND time > 1222869660 LIMIT 1
648 Execute SELECT time, value FROM points INNER JOIN timeseries_meta USING (series_id) WHERE name = 'L ReservoirCheckValve flow' AND time > 1222869720 LIMIT 1
648 Execute SELECT time, value FROM points INNER JOIN timeseries_meta USING (series_id) WHERE name = 'L ReservoirCheckValve flow' AND time > 1222869780 LIMIT 1
648 Execute SELECT time, value FROM points INNER JOIN timeseries_meta USING (series_id) WHERE name = 'L ReservoirCheckValve flow' AND time > 1222869840 LIMIT 1
648 Execute SELECT time, value FROM points INNER JOIN timeseries_meta USING (series_id) WHERE name = 'L ReservoirCheckValve flow' AND time > 1222869900 LIMIT 1
648 Execute SELECT time, value FROM points INNER JOIN timeseries_meta USING (series_id) WHERE name = 'L ReservoirCheckValve flow' AND time > 1222869960 LIMIT 1
648 Execute SELECT time, value FROM points INNER JOIN timeseries_meta USING (series_id) WHERE name = 'L ReservoirCheckValve flow' AND time > 1222870020 LIMIT 1
648 Execute SELECT time, value FROM points INNER JOIN timeseries_meta USING (series_id) WHERE name = 'L ReservoirCheckValve flow' AND time > 1222870080 LIMIT 1
648 Execute SELECT time, value FROM points INNER JOIN timeseries_meta USING (series_id) WHERE name = 'L ReservoirCheckValve flow' AND time > 1222870140 LIMIT 1
648 Execute SELECT time, value FROM points INNER JOIN timeseries_meta USING (series_id) WHERE name = 'L ReservoirCheckValve flow' AND time = 1222869660
648 Execute SELECT time, value FROM points INNER JOIN timeseries_meta USING (series_id) WHERE name = 'L ReservoirCheckValve flow' AND time = 1222869720
648 Execute SELECT time, value FROM points INNER JOIN timeseries_meta USING (series_id) WHERE name = 'L ReservoirCheckValve flow' AND time = 1222869780
648 Execute SELECT time, value FROM points INNER JOIN timeseries_meta USING (series_id) WHERE name = 'L ReservoirCheckValve flow' AND time = 1222869840
648 Execute SELECT time, value FROM points INNER JOIN timeseries_meta USING (series_id) WHERE name = 'L ReservoirCheckValve flow' AND time = 1222869900
648 Execute SELECT time, value FROM points INNER JOIN timeseries_meta USING (series_id) WHERE name = 'L ReservoirCheckValve flow' AND time = 1222869960
648 Execute SELECT time, value FROM points INNER JOIN timeseries_meta USING (series_id) WHERE name = 'L ReservoirCheckValve flow' AND time = 1222870020
648 Execute SELECT time, value FROM points INNER JOIN timeseries_meta USING (series_id) WHERE name = 'L ReservoirCheckValve flow' AND time = 1222870080
648 Execute SELECT time, value FROM points INNER JOIN timeseries_meta USING (series_id) WHERE name = 'L ReservoirCheckValve flow' AND time = 1222870140

Issue by jamesuber
Tuesday Sep 02, 2014 at 06:16 GMT
Originally opened as https://github.com/CitiLogics/epanet-rtx/pull/17

sam appreciate you having a look; thought the pull request was the cleanest way to do that? a couple of particular things--

• take note of the changes to resampler.h and modulartimeseries::filteredpoints. small but not certain there are no side effects.
• the aggregator will need to reimplement pointBefore and pointAfter, to find the times from all of the aggregator sources (and not just the first one) when there isn't a clock. I'll do that if you think this is ok.
• includes the bug fix to the sqlite project file that you id'd a while ago. also includes mods I've made to the lame class for processing the runtime data (for nkwd).

jamesuber included the following code: https://github.com/CitiLogics/epanet-rtx/pull/17/commits

... need to flatten the recursive algorithm.

Hi. I'm a beginner.
I have modified the makefile (# *** Files) with new code files.
//
RTX_HEADERS = AggregatorTimeSeries.h BufferPointRecord.h Clock.h ConfigFactory.h ConstantSeries.h CurveFunction.h DbPointRecord.h DequePointRecord.h Element.h EpanetModel.h EpanetSyntheticModel.h FirstDerivative.h IrregularClock.h Junction.h Link.h MapPointRecord.h Model.h ModularTimeSeries.h MovingAverage.h MysqlPointRecord.h Node.h OdbcPointRecord.h OffsetTimeSeries.h Pipe.h Point.h PointRecord.h Pump.h Resampler.h Reservoir.h SineTimeSeries.h Tank.h TimeSeries.h Units.h Valve.h Zone.h rtxExceptions.h rtxMacros.h

RTX_SRC = AggregatorTimeSeries.cpp BufferPointRecord.cpp Clock.cpp ConfigFactory.cpp ConstantSeries.cpp CurveFunction.cpp DbPointRecord.cpp DequePointRecord.cpp Element.cpp EpanetModel.cpp EpanetSyntheticModel.cpp FirstDerivative.cpp IrregularClock.cpp Junction.cpp Link.cpp MapPointRecord.cpp Model.cpp ModularTimeSeries.cpp MovingAverage.cpp MysqlPointRecord.cpp Node.cpp OdbcPointRecord.cpp OffsetTimeSeries.cpp Pipe.cpp Point.cpp PointRecord.cpp Pump.cpp Resampler.cpp Reservoir.cpp SineTimeSeries.cpp Tank.cpp TimeSeries.cpp Units.cpp Valve.cpp Zone.cpp

RTX_OBJS = AggregatorTimeSeries.o BufferPointRecord.o Clock.o ConfigFactory.o ConstantSeries.o CurveFunction.o DbPointRecord.o DequePointRecord.o Element.o EpanetModel.o EpanetSyntheticModel.o FirstDerivative.o IrregularClock.o Junction.o Link.o MapPointRecord.o Model.o ModularTimeSeries.o MovingAverage.o MysqlPointRecord.o Node.o OdbcPointRecord.o OffsetTimeSeries.o Pipe.o Point.o PointRecord.o Pump.o Resampler.o Reservoir.o SineTimeSeries.o Tank.o TimeSeries.o Units.o Valve.o Zone.o
//

I compile code with clang in Ubuntu. It comes the error.
Can you tell me how to solve it?

../../src/OdbcPointRecord.cpp:405:32: error: expected a class or namespace
Point::Qual_t q = Point::Qual_t::good; // todo -- map to rtx quality types

Issue by jamesuber
Thursday Oct 17, 2013 at 02:37 GMT
Originally opened as https://github.com/CitiLogics/epanet-rtx/issues/4

it seems that movingaverage can compute the average over window+1, window, or window-1 points, depending on whether the window is even or odd, and whether the averaged series point is centered on a source point. It's minor in the large, but perhaps should be cleaned up.

resampler source should be

source = "Scada_tag_1_flowrate";

to avoid confusion. Also, the derivative timeseries source should be

source = "Tag 2 volume"

I'm sorry, I have a problem that why not have folder:“epanet” in the folder “src”？i find some functions have been changed in epanet.c,such as the added function “ENgetdemandpattern()” , but i can not find this from the source code files。

Issue by jamesuber
Tuesday Oct 07, 2014 at 16:43 GMT
Originally opened as https://github.com/CitiLogics/epanet-rtx/issues/19

have a project file with a relative path to a record. But in rtx code when the project is opened the record never connects to the db.

Issue by jamesuber
Friday Aug 16, 2013 at 19:54 GMT
Originally opened as https://github.com/CitiLogics/epanet-rtx/issues/2

Solve this case:

The runtimestatus performed really well for all of the NKWD timeseries except for Richardson Road Pump 2, as measured by cumulative pump runtime over a 2 month period, from 11/3/2012 to 12/29/2012.

There is an extraneous 0 status inserted at about Noon on 12/15/12 when the pump is obviously on, causing the loss of about 27 hours of runtime.

This might be due to RR #2 running continuously for long times - in this stretch exceeding a week. We are using the points method with one week time chunks, so maybe a boundary issue lurking somewhere there.

Let's define a pressure measure to go along with the head measure. The junction class would then convert any pressure measures to head for the user (e.g. through an aggregator and offset timeseries chain).

Hello,my name is caihuaqiang，When I try to apply RTX to a real water supply network ，I find that the values of the pipes flow in the scada db all are postive， that is to say，they dont't have direction，how to fix it to adapt to RTX？ it's a significant problem to me，so thanks very much for replying to me as soon as possible！

It is useful to specify flow measures and yet not have them be included in zone delineation or demand calculation. One reason is that a flow measure data might not be good enough for demand estimation, but still useful for model comparision. Another is that you might want to break a zone boundary on purpose, because of some undesirable property of that zone, or even just for sensitivity studies.

enhance the flow_measure to include an explicit indication of its assumed positive direction. Now that is assumed to be specified by the model link upstream/downstream nodes, which is logical, but also a potential trap.

One way would (optionally) require specifying upstream_node.

Don't really know if this is a potential bug. But this noop. addpoint() method is being used. I noticed that and printed out some debug info -

BufferPointRecord::addPoint() -- no point added for TimeSeries Dudley 1080 Station Flow (ranged)
BufferPointRecord::addPoint() -- no point added for TimeSeries Dudley 1080 Station Flow (ranged)
BufferPointRecord::addPoint() -- no point added for TimeSeries Dudley 1080 Station Flow (resampled)
BufferPointRecord::addPoint() -- no point added for TimeSeries Dudley 1080 Station Flow (resampled)

etc...

resampler sometimes drops points from consideration (1.1_preproduction)

Need Linux makefile or CMake file.

Issue by jamesuber
Friday Nov 01, 2013 at 17:39 GMT
Originally opened as https://github.com/CitiLogics/epanet-rtx/issues/7

Revisiting some DMA flow processing involving calculated tank flows and pump flows... if we really want to process these data in a refined manner, we'd start by admitting there should be first derivative discontinuities in the resulting series, due to discrete pump on/off decisions and the resulting impacts on pump and tank flow.

This comes out of an annoyance about the degree of smoothing needed to avoid negative DMA demands. You can see what is going on in the raw data, but there's no way to retain the discrete changes after smoothing the data with a MA. Possibly something like a MA filter with discontinuity detection?

... quite inefficient for large record sets. update the mechanism to use bulk-insertion.

Need better error state handling. Lots of cleanup. Etc.

Issue by jamesuber
Monday Oct 28, 2013 at 20:25 GMT
Originally opened as https://github.com/CitiLogics/epanet-rtx/issues/6

A particular instance noticed where processing of a timeseries pipeline depended on modulartimeseries that were persisted, and the total processing time depended hugely (like, 10 minutes compared to 12 hours) on whether those modulartimeseries were persisted first (as in, asked for their points) - or if they were processed as part of the total timeseries pipeline.

I know that's confusing but it's hard to explain with precision, especially since I'm not certain how generally this will be seen.

For reference, I've got a cfg with a NKWD test case in my copy/code directory, corresponding to processing of the US 27 pump station flows.

Good afternoon, my name es Sergio Anaya, and I want you to help me in my work. I'm new to programming in Epanet-RTX toolkit, and would like an explanation step by step what I have to do to run a hydraulic network with SCADA system, for example.

How do I call the libraries?
How scada link tags of a hydraulic model element?
How the simulation performed?

thanks, I appreciate your help

if you have any general, open-ended questions, feel free to create a new issue (tag it "question").

The mysqlpointrecord destructor hit some sort of snag when calling the driver threadEnd method. Here's

a screen shot.

Issue by jamesuber
Tuesday Jan 21, 2014 at 20:47 GMT
Originally opened as https://github.com/CitiLogics/epanet-rtx/issues/8

convenient to add units strings for MGD/[time]

should be able to trigger a tank re-set externally. create threadsafe Tank::setNeedsReset(bool) and Model::setNeedsTanksReset(bool) methods.

the pointAfter methods for the ModularTimeSeries and derived classes work reliably when it is given a time that is valid. But when the clock is regular you can get an invalid point if you ask for one that is outside the time range of validity. I think that point validity is ensured if the time t >= tstart where tstart is the time of the first DB entry.

This much is clear from the cerr reminders to "check point availability first" - but it will be useful sometimes to know how to determine a valid time, such as when you want to do processing on an entire timeseries and you don't know much about it other than it's in the DB. A method that only checks for point validity and iterates isn't going to be much fun (and possibly never ending).

One can solve this by getting down to the underlying TimeSeries and PointRecord, using the boost dynamic pointer cast to iteratively check upstream objects for whether or not it is a valid ModularTimeSeries. Sam pointed this out to me and provided the following code snippet that iterates beginning from a ModularTimeSeries (thanks, Sam):

  // find a valid future time, else go on to the next TimeSeries
  TimeSeries::sharedPointer candidate = aTimeSeries;
  ModularTimeSeries::sharedPointer castTS;
  while ((castTS = boost::dynamic_pointer_cast<ModularTimeSeries>(candidate))) {
    candidate = castTS->source();
  }
  // not a valid modular series
  Point aPoint = candidate->pointAfter(0);

Well, that works great. Should it be in RTX as an enhancement?

One possibility is to provide a ModularTimeSeries method that allows one to check for a valid range of times, like ModularTimeSeries::validTimeRange() that would return, somehow, the first time and the last time of the PointRecord.

The documentation lists GSL as a prerequisite (http://openwateranalytics.github.io/epanet-rtx/ ). However, I cannot find any references in the sources. Does the documentation need an update?