misken / hillmaker Goto Github PK
View Code? Open in Web Editor NEWOccupancy analysis by time of day and day of week, with Python
License: MIT License
Occupancy analysis by time of day and day of week, with Python
License: MIT License
Something like the one in pydantic which can be used like this:
python -c "import pydantic.utils; print(pydantic.utils.version_info())":
If the optional category field is not specified, then the plots fail with:
---------------------------------------------------------------------------
KeyError Traceback (most recent call last)
Cell In[11], line 1
----> 1 results_ex1 = hm.make_hills(scenario, stops_df, in_field, out_field,
2 start_date, end_date,
3 bin_size_minutes=bin_size_mins,
4 output_path=output_path,
5 export_dow_png=True,
6 export_week_png=True,
7 xlabel='xlabel',
8 ylabel='ylabel')
File ~\Anaconda3\envs\aap\Lib\site-packages\hillmaker\hills.py:252, in make_hills(scenario_name, stops_df, in_field, out_field, start_analysis_dt, end_analysis_dt, cat_field, bin_size_minutes, percentiles, cats_to_exclude, occ_weight_field, totals, cap, nonstationary_stats, stationary_stats, no_censored_departures, export_bydatetime_csv, export_summaries_csv, export_dow_png, export_week_png, xlabel, ylabel, output_path, edge_bins, verbosity)
250 if export_week_png:
251 with HillTimer() as t:
--> 252 for metric in summary_dfs['nonstationary']['dow_binofday']:
253 fullwk_df = summary_dfs['nonstationary']['dow_binofday'][metric]
254 fullwk_df = fullwk_df.reset_index()
KeyError: 'dow_binofday'
Only key parameters should be included in CLI.
In make_bydatetime
, even if edge_bins=2
is specified, edge_bins
is being set to 1 and only fractional bins are computed for occupancy in arrival and exit bins. This will be fixed in next release.
Use same logging library as in mwts project.
Oops, need to add tomli>=1.1.0 to setup.py install_requires
list
For now, just pip install tomli into virtual env containing hillmaker
Need a consistent vocabulary for referring to and talking about these things in the user docs, code comments and code itself.
Hard coded "InRoomTS" and "OutRoomTS" in make_bydatetime
. Variables supplied by user (infield, outfield) should be used instead.
Check all path related stuff and convert to using pathlib
Implement same as VBA version.
Currently output_path
serves both purposes but I can see wanting to route the CSV and plots to different destination directories.
Possible outputs
As I am working through some plots today, the full opacity of the bar color clashes with the ggplot plot style. Realized that there is not many bar kwargs exposed, so feel that adding alpha as a param is necessary.
Let user pass in list of fields, in order, to group by.
Not sure what form this should take.
Config files are needed to making automation easier and to serve as scenario documentation.
Include tests for each rectype. Include tests on bin boundaries.
I did something like this already in a Jupyter Notebook, just need to decide the best way to integrate it into the package. Ideally, I would like it to be something as simple as by_category=True
in the existing functions/methods (like in the other OO functions), but the code may be too verbose and might require a separate, make_stacked_hills
function or something.
Create some combination of plots that can automatically be generated and also some sort of "plot template" (i.e. Python code) that makes it easy to create customized plots.
Could also explore the broader landscape of data viz tools to see how they might be used with hillmaker outputs.
Current looping-centric approach with repeated use of .ix
is order of magnitude slower than VBA version. Looking into new multi-index slicers added in pandas 0.14.0.
Due to stops_df
index not being sequential. Can happen if user filters df before feeding into make_hills
.
scenario_name = 'sstest_60'
in_fld_name = 'InRoomTS'
out_fld_name = 'OutRoomTS'
cat_fld_name = 'PatType'
start_analysis = '1/1/1996'
end_analysis = '3/30/1996 23:45'
bin_size_mins = 60
Right now it's just a TOML dump
Plan is to host it as subdomain within hselab.org
Will improve the documentation
Target reader is analyst using hillmaker to crunch data.
This will just make the plot too busy and confusing.
This gives full power of the new plot options to both the OO API and the legacy function API. At this point the CLI still has a limited number of input parameters.
Right now no timing messages in summarize.py
sortlevel was deprecated in 0.20.0. See https://stackoverflow.com/questions/44123874/dataframe-object-has-no-attribute-sort.
This is related to PR #39, but retroactively adding an issue for documentation purposes. The current PR will likely need heavy refactoring based on all of the API changes made recently.
The searchsorted is not catching first bin if arrival at time 0.
Does it make sense to add a validation check to make_hills
that determines the length of warmup and horizon effects based on a rule?
The rule could be based on some combination of LOS and the number of days in the supplied date range. As an example, I am imagining it would compute the 75th percentile of LOS and then automatically adjust the start and end date based on this number. If the date range is quite large, the adjustment can be multiplied by a factor of 5 or something.
The main driver behind this is that casual users of the application might not fully grasp the implication of these effects and providing that adjustment for them could avoid misleading data. Of course, users who fully understand the concepts of these effects could opt to adjust the date range manually and could turn off this functionality with a simple validate_warmup_horizon_effects=False
.
Datetimes in csv are text. Conversion to appropriate data type (pandas Timestamp) should happen at read time. However, what to do about the start and end analysis values used in arguments to hillmaker() main function? They need to be converted to pandas Timestamps before use in comparisons with other Timestamps. Do we force arguments to be Timestamps or try to convert them?
Right now, the summary dataframe has a multi-index:
day_of_week | dow_name | bin_of_day | bin_of_day_str | ย
This makes it painful to select specific rows (e.g. Monday). If we flatten the index and stick the levels as regular columns, it will be much easier to select some subset of rows. In the plotting code, we flatten the index to make it easy to make the plot.
What do you think @jwnorm ?
Forgot to do this for v0.1.0
The hmlib is currently imported using *.
Check if:
By replacing the stops_df and stop_data_csv parameters with a single data parameter, we can move towards more general handling of different kinds of data inputs. Also makes it easier to process inputs since won't have to worry about what to do in case both parameters are used simultaneous (i.e. who gets precedence?).
The CLI should be just as powerful as the other APIs.
If pandas MultiIndex is output, the category field gets exported twice and the day of week and bin of day fields in the index contain many blank values. No need to output the MultiIndex.
The src layout has gained momentum as the recommended approach. I've used it on other projects with good results. Also need to move tests folder to project root.
Is this a bottleneck? Can approach I used in daily averages blog post be of help?
Figure out how to use the Warnings module
It might make sense to store application settings such as some of the key parameter defaults in a file instead of them being scattered throughout the code.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.