sandialabs / pecos Goto Github PK
View Code? Open in Web Editor NEWPython package for performance monitoring of time series data
License: Other
Python package for performance monitoring of time series data
License: Other
If you are working with timezone aware dataframes, there is an issue retaining the timezone across the entire pecos analysis chain. The timezone is not stored properly in the testing results (pm.test_results['State Date'] and pm.test_results['End Date']). It looks like this is a known pandas bug. In pecos, this can lead to a mismatch in analysis that use the test results. I'm working on a fix and should have that committed soon, along with some stronger tests for analysis that use timezones.
Hello @kaklise ,
I am trying to understand the intent and actual functioning of the check_delta
function in the monitoring module. Here is an example that I put together:
import pandas as pd
from pecos import monitoring
# Constants
UPPER_LIMIT_RANGE_FILTER = 1.2
LOWER_LIMIT_RANGE_FILTER = 0.5
# Create a random data frame
df = pd.DataFrame(data=[["2015-01-01 00:00:00", -0.76, 2, 2, 1.2],
["2015-01-01 01:00:00", -0.73, 2, 4, 1.1],
["2015-01-01 02:00:00", -0.71, 2, 4, 1.1],
["2015-01-01 03:00:00", -0.68, 2, 32, 1.1],
["2015-01-01 04:00:00", -0.65, 2, 2, 1.0]],
columns=['DateTime', 'column1', 'column2', 'column3', 'column4'])
# Set the index
df["DateTime"] = pd.to_datetime(df["DateTime"])
df.set_index("DateTime", inplace=True)
# Run delta flag
flags = pd.DataFrame()
for station in df.columns:
rating = 2.5
dead_jump_flag_check = monitoring.check_delta(
data=df[[station]],
bound=[LOWER_LIMIT_RANGE_FILTER * rating,
UPPER_LIMIT_RANGE_FILTER * rating],
key=None,
window=3600,
direction=None,
min_failures=1
)
if flags.empty:
flags = dead_jump_flag_check['mask']
else:
flags = pd.merge(left=flags, right=dead_jump_flag_check['mask'], on="DateTime")
print(flags)
My interpretation of the existing functionality here doesn't line up with what I am seeing in the example. The function should be flagging stuck data or abrupt changes. In the description of the function, the difference between max and min values within a rolling window is used for flagging it.
In the above example, I am using window=3600 and min_failures=1, so even if one timestamp fails the condition, the flag should be triggered. I would appreciate if you can explain the following:
column1 column2 column3 column4
DateTime
2015-01-01 00:00:00 True True True True
2015-01-01 01:00:00 False False False False
2015-01-01 02:00:00 False False False False
2015-01-01 03:00:00 False False False False
2015-01-01 04:00:00 False False False False
Why is row 1 flagged ?
Why are column 2 and column 3 not being flagged despite having stuck values and values that jump around ?
Would appreciate some clarity..
Best
Uday
Some data logging systems record data at irregular time stamps. It would be helpful to me if check_timestamp
could handle this case. I'm not sure what the best way to implement it would be. Maybe something like this?
In [36]: index = pd.DatetimeIndex([pd.Timestamp('20161017 00:05:00'), pd.Timestamp('20161017 02:03:00'), pd.Timestamp('20161017 02:50:00')])
In [37]: s = pd.Series([0, 2, 3], index=index)
In [43]: s.resample('3600s').count() == 0
Out[43]:
2016-10-17 00:00:00 False
2016-10-17 01:00:00 True
2016-10-17 02:00:00 False
Freq: 3600S, dtype: bool
Create ci test for python 3.4 and 3.5
Two requirements issues:
Thanks!
See page example at:
The example files all use jpg output files, but there is apparently a bug in matplotlib handling of jpg on macosx. Can I change output files to png to make the examples run on every platform? Is this something Travis CI should catch if there were plots in the test files?
The PerformanceMonitoring class defines _append_test_results() to Append QC results to the PerformanceMonitoring object. This function is called throughout the class relies on pd.Dataframe.append() which is now deprecated.
I just read the presentation at
https://pvpmc.sandia.gov/download/5289/
Questions:
As example, we could generate expected values based on a TMY input and a simple system modelled with pvlib. Then, the performance is validated against that value.
Example misses some templates:
(pecos_trial) C:\Users\%USER%\AppData\Local\Continuum\Anaconda3\envs\pecos_trial\pecos\examples\pv>python pv_example.py
Reading Campbell Scientific CSV file Baseline6kW_2015_11_11.dat
INFO:pecos.io:Reading Campbell Scientific CSV file Baseline6kW_2015_11_11.dat
Reading Campbell Scientific CSV file MET_2015_11_11.dat
INFO:pecos.io:Reading Campbell Scientific CSV file MET_2015_11_11.dat
Check timestamp
[...]
INFO:pecos.io:Writing test results csv file Results\Baseline_System_test_results.csv
Writing HTML report
INFO:pecos.io:Writing HTML report
Traceback (most recent call last):
File "pv_example.py", line 128, in <module>
'Baseline System, Performance Monitoring Report', config)
File "C:\Users\%USER%\AppData\Local\Continuum\Anaconda3\envs\pecos_trial\lib\site-packages\pecos-0.1.2-py3.5.egg\pecos\io.py", line 240, in write_monitoring_report
html_string = _html_template_monitoring_report(content, title, logo, encode)
File "C:\Users\%USER%\AppData\Local\Continuum\Anaconda3\envs\pecos_trial\lib\site-packages\pecos-0.1.2-py3.5.egg\pecos\io.py", line 337, in _html_template_monitoring_report
template = env.get_template('monitoring_report.html')
File "C:\Users\%USER%\AppData\Local\Continuum\Anaconda3\envs\pecos_trial\lib\site-packages\jinja2\environment.py", line 812, in get_template
return self._load_template(name, self.make_globals(globals))
File "C:\Users\%USER%\AppData\Local\Continuum\Anaconda3\envs\pecos_trial\lib\site-packages\jinja2\environment.py", line 774, in _load_template
cache_key = self.loader.get_source(self, name)[1]
File "C:\Users\%USER%\AppData\Local\Continuum\Anaconda3\envs\pecos_trial\lib\site-packages\jinja2\loaders.py", line 235, in get_source
raise TemplateNotFound(template)
jinja2.exceptions.TemplateNotFound: monitoring_report.html
(pecos_trial) C:\Users\%USER%\AppData\Local\Continuum\Anaconda3\envs\pecos_trial\pecos\examples\pv>
Hey!
I've been setting my time filter without using this method, because it tends to be pretty slow (especially for large data). For the simple example timefilter the timeit on my system is:
%timeit 298ms per loop
My work around is to set my time filter using the datetime library and df.index.time. I could do a pull-request illustrating this in an example, if you are interested. My timeit for the simple example timefilter is:
%timeit 6.55ms per loop
Volker
I'm working on changes that will drop 'system name' from PerformanceMonitoring objects. The user can currently supply a system name when adding a DataFrame or a translation dictionary to the analysis. This has created some unnecessary complexity and assumptions about column names in the code. If the user wants to add DataFrames from multiple systems where 1) repeated column names are a concern or 2) they just want to keep a record of the system, I suggest appending the system name to column names before adding it to the PerformanceMonitoring object. System name will also be dropped from the test results table.
'is_monotonic' is deprecated since Pandas 1.5.0
When trying to apply the check_timestamp function to high frequency data (i.e. 50kHz which equals 0.00002s) the function produces a zero division error. I have uploaded a sample of data so you can try and reproduce the error if you need. Let me know if there is any way I can help de-bug this issue. Thanks!
Test
As pointed out in #20, documentation should include instructions on how to setup pecos in a conda environment.
Current use of strings to build HTML is fragile and redundant in some cases. Use of existing mature robust HTML python templates will make management development and collaboration easier.
I recommend jinja2
http://jinja.pocoo.org/
https://pypi.python.org/pypi/Jinja2
https://github.com/pallets/jinja
In a nutshell HTML templates are rendered by jinja2 which replaces jinja2 markup with desired content. Jinja2 markup is embedded directly in the HTML and is a modified subset of python. Python variables are passed to the Jinja2 renderer along with the template and expanded in the output according to the mark-up in the template. Base templates can be "included" to remove redundant content from multiple pages, such as headers and footers, scripts and CSS.
An easy improvement to make the tables look nicer is to use datatables.net javascript plugin in the html templatate dashboad by inserting the following patch:
--- c:\users\mmikof~1\appdata\local\temp\meld-tmp4qxlpp
+++ C:\Users\mmikofski\Documents\Projects\pecos\pecos\io.py
@@ -320,6 +320,7 @@
<title>$title</title>
<meta charset="UTF-8" />
</head>
+ <body>
<table border="0" width="100%">
<col style="width:70%">
<col style="width:30%">
@@ -396,6 +397,7 @@
datestr = date.strftime('%m/%d/%Y')
template = template + pecos.__version__ + ", " + datestr
template = template + """
+ </body>
</html>"""
template = Template(template)
@@ -415,7 +417,10 @@
template = template + """
</title>
<meta charset="UTF-8" />
+ <!-- datatables.net -->
+ <link rel="stylesheet" type="text/css" href="https://cdn.datatables.net/1.10.11/css/jquery.dataTables.css">
</head>
+ <body>
<table border="0" width="100%">
<col style="width:70%">
<col style="width:30%">
@@ -437,7 +442,7 @@
template = template + title
template = template + """
</H2>
- <table border="1" class="dataframe">
+ <table id="myTable" border="1" class="dataframe display">
<thead>
<tr>
<th></th>"""
@@ -496,9 +501,15 @@
datestr = date.strftime('%m/%d/%Y')
template = template + pecos.__version__ + ", " + datestr
template = template + """
+ <!-- jQuery (necessary for datatables.net) -->
+ <script src="https://code.jquery.com/jquery-1.12.3.min.js"></script>
+ <script type="text/javascript" charset="utf8" src="https://cdn.datatables.net/1.10.11/js/jquery.dataTables.js"></script>
+ <script>
+ $(document).ready(function(){
+ $('#myTable').DataTable();
+ });
+ </script>
+ </body>
</html>"""
- template = template + """
- </html>"""
-
return template
@kaklise :
Katherine, Hope you are doing well. Can you kindly post an example for how to use the check_timestamp function. Here is what I tried :
import pandas as pd
from random import random
from pecos import monitoring
# Build a random data set
ts_index = pd.date_range('1/1/2000', periods=1000, freq='T')
v1 = [random() for i in range(1000)]
v2 = [random() for i in range(1000)]
v3 = [random() for i in range(1000)]
ts_df = pd.DataFrame({'v1':v1,'v2':v2,'v3':v3},index=ts_index)
# Test for timestamps
t = monitoring.check_timestamp(
data=ts_df,
frequency=3600
)
print(t["mask"])
Since the index is the time-series for the entire data frame:
monotonicity (since I am using the default, it should reindex)
missing (should fill in any missing timestamps)
duplicates (drop any duplicates, by retaining the first occurrence)
After performing these actions, I am not sure how to interpret the booleans that it returns which are the size of the data frame. Could you kindly provide an example clarifying this ?
Pecos seems to be a very intersting & useful tool for develping a meteo data QC workflow.
I would like to test iot on some historical data but have some questions on use:
The pv example config contains Baseline_config.yml several limit values.
Just for clarification:
Maybe the reason for my question that I cannot see the output due to #20.
I'm not sure if anyone is using the pecos function 'send_email' to send reports, but I plan to update the function to use more flexible python packages (smtplib and email). This will make the function compatible with a wider range of email servers and operating systems. The updated function will no longer support the Outlook option as it is written, let me know if that's a problem.
Updates to the check_delta test require the latest version of Pandas (0.23.0), which are not backward compatible. Pecos will no longer be tested against Python 3.4
A handful of pecos functions/methods use mutable default arguments. This can lead to unexpected behavior, though I don't think the way that pecos is currently using them would cause a problem. Just something I noticed while trying to become more familiar with the library.
http://docs.python-guide.org/en/latest/writing/gotchas/#mutable-default-arguments
check_range, check_increment, evaluate_string, write_monitoring_report, plot_colorblock
Another reason to consider changing them is to set a good example for less experienced people.
Trying to import pecos
fails after pip install pecos
with traceback:
>>> import pecos
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "C:\ProgramData\anaconda3\envs\pecos_pip_install_test\Lib\site-packages\pecos\__init__.py", line 3, in <module>
from pecos import io
File "C:\ProgramData\anaconda3\envs\pecos_pip_install_test\Lib\site-packages\pecos\io.py", line 10, in <module>
import pecos.graphics
File "C:\ProgramData\anaconda3\envs\pecos_pip_install_test\Lib\site-packages\pecos\graphics.py", line 19, in <module>
import pytest
ModuleNotFoundError: No module named 'pytest'
This line in pecos/pecos/graphics.py seems to be the source of the issue.
Custom checks are applied to the entire dataset despite a key being specified. If I replace the following section of the check_custom_static function :
# Function that operates on the entire dataset and returns a mask and
# metadata for the entire dataset
mask, metadata = quality_control_func(self.df)
by this :
# Function that operates on the entire dataset and returns a mask and
# metadata for the entire dataset
mask, metadata = quality_control_func(df)
with df being defined as : df = self._setup_data(key)
it seems to be working.
I see that pecos.pv.basic_pvlib_performance_model
uses older function signatures for pvlib's sapm
and singlediode
. It would be straightforward to update these for pvlib's 0.4 api change, possibly with a version check to retain compatibility. However, it seems to me that pvlib 0.4's ModelChain already accomplishes what this function sets out to do and maybe the function can be deprecated? I ran into the issue when trying to run the pv_example.py
script.
On a related note, I love the yaml files in the pecos examples and I would like to see pvlib gain the ability to make a Location
and PVSystem
from a yaml file. That might make it slightly easier to integrate the pvlib changes into the pv_example.py
script.
Version (0.1.2) should be incremented as it includes changes not present in release 0.1.2 (timezone stuff).
**Update: I noticed that the Pip version is 0.1.7 while the stable version in GitHub and ReadTheDocs is 0.1.8 **
The functional approach as documented in the "Framework" of the documentation shows the following:
results = pecos.monitoring.check_range(data, [-3,3])
Here is my attempt at doing the exact same thing in multiple different ways:
** ATTEMPT 1 **
# Import required packages
import numpy as np
import pandas as pd
import re
from pandas.util import testing as tm
import pecos
# Set Seed and generate a dataframe for testing
tm.N, tm.K = 15, 3
np.random.seed(256)
ts_df = tm.makeTimeDataFrame(freq='D')
# Reset Index
ts_df = ts_df.reset_index()
# Call check_range as shown in the documentation
check_range_test = pecos.monitoring.check_range(ts_df,[-1,1])
It generates the following exception:
module 'pecos.monitoring' has no attribute 'check_range'
** ATTEMPT 2 **
# Import required packages
import numpy as np
import pandas as pd
import re
from pandas.util import testing as tm
from pecos import monitoring
# Set Seed and generate a dataframe for testing
tm.N, tm.K = 15, 3
np.random.seed(256)
ts_df = tm.makeTimeDataFrame(freq='D')
# Reset Index
ts_df = ts_df.reset_index()
# Call check_range as shown in the documentation
check_range_test = monitoring.check_range(ts_df,[-1,1])
It generates the following exception:
module 'pecos.monitoring' has no attribute 'check_range'
Can you kindly point out to what I am doing wrong ? Additionally, do you have any further documentation on the functional approach beyond what is on the ReadTheDocs ?
i use original code "simple_example.py" ,original file "simple.xlsx" in example.
environment in pandas 0.24
system show:
TypeError: Invalid comparison between dtype=float64 and Timedelta
i change pandas to 0.23
it's solved.
For people who have encountered the same mistake
The 'check_delta' QC test checks the difference between the max and min within a rolling window. This test is not very efficient for large data sets (>100000 pts) because it uses df.rolling().apply() to find the location of the min and max.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.