misken / hillmaker Goto Github PK

Something like the one in pydantic which can be used like this:

python -c "import pydantic.utils; print(pydantic.utils.version_info())":

If the optional category field is not specified, then the plots fail with:

---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
Cell In[11], line 1
----> 1 results_ex1 = hm.make_hills(scenario, stops_df, in_field, out_field,
      2                             start_date, end_date,
      3                             bin_size_minutes=bin_size_mins,
      4                             output_path=output_path,
      5                             export_dow_png=True,
      6                             export_week_png=True,
      7                             xlabel='xlabel',
      8                             ylabel='ylabel')

File ~\Anaconda3\envs\aap\Lib\site-packages\hillmaker\hills.py:252, in make_hills(scenario_name, stops_df, in_field, out_field, start_analysis_dt, end_analysis_dt, cat_field, bin_size_minutes, percentiles, cats_to_exclude, occ_weight_field, totals, cap, nonstationary_stats, stationary_stats, no_censored_departures, export_bydatetime_csv, export_summaries_csv, export_dow_png, export_week_png, xlabel, ylabel, output_path, edge_bins, verbosity)
    250 if export_week_png:
    251     with HillTimer() as t:
--> 252         for metric in summary_dfs['nonstationary']['dow_binofday']:
    253             fullwk_df = summary_dfs['nonstationary']['dow_binofday'][metric]
    254             fullwk_df = fullwk_df.reset_index()

KeyError: 'dow_binofday'

Only key parameters should be included in CLI.

In make_bydatetime, even if edge_bins=2 is specified, edge_bins is being set to 1 and only fractional bins are computed for occupancy in arrival and exit bins. This will be fixed in next release.

Use same logging library as in mwts project.

Oops, need to add tomli>=1.1.0 to setup.py install_requires list

For now, just pip install tomli into virtual env containing hillmaker

Need a consistent vocabulary for referring to and talking about these things in the user docs, code comments and code itself.

Hard coded "InRoomTS" and "OutRoomTS" in make_bydatetime. Variables supplied by user (infield, outfield) should be used instead.

Check all path related stuff and convert to using pathlib

Implement same as VBA version.

• default: fraction of bin
• full: treat as 1 if fraction > full_threshold

Currently output_path serves both purposes but I can see wanting to route the CSV and plots to different destination directories.

Possible outputs

• frequency of gap size
• time series plot

As I am working through some plots today, the full opacity of the bar color clashes with the ggplot plot style. Realized that there is not many bar kwargs exposed, so feel that adding alpha as a param is necessary.

Let user pass in list of fields, in order, to group by.

Not sure what form this should take.

Config files are needed to making automation easier and to serve as scenario documentation.

Include tests for each rectype. Include tests on bin boundaries.

I did something like this already in a Jupyter Notebook, just need to decide the best way to integrate it into the package. Ideally, I would like it to be something as simple as by_category=True in the existing functions/methods (like in the other OO functions), but the code may be too verbose and might require a separate, make_stacked_hills function or something.

Create some combination of plots that can automatically be generated and also some sort of "plot template" (i.e. Python code) that makes it easy to create customized plots.

Could also explore the broader landscape of data viz tools to see how they might be used with hillmaker outputs.

Current looping-centric approach with repeated use of .ix is order of magnitude slower than VBA version. Looking into new multi-index slicers added in pandas 0.14.0.

Due to stops_df index not being sequential. Can happen if user filters df before feeding into make_hills.

scenario_name = 'sstest_60'
in_fld_name = 'InRoomTS'
out_fld_name = 'OutRoomTS'
cat_fld_name = 'PatType'
start_analysis = '1/1/1996'
end_analysis = '3/30/1996 23:45'
bin_size_mins = 60

Right now it's just a TOML dump

Plan is to host it as subdomain within hselab.org

Will improve the documentation

Target reader is analyst using hillmaker to crunch data.

This will just make the plot too busy and confusing.

This gives full power of the new plot options to both the OO API and the legacy function API. At this point the CLI still has a limited number of input parameters.

Right now no timing messages in summarize.py

sortlevel was deprecated in 0.20.0. See https://stackoverflow.com/questions/44123874/dataframe-object-has-no-attribute-sort.

This is related to PR #39, but retroactively adding an issue for documentation purposes. The current PR will likely need heavy refactoring based on all of the API changes made recently.

The searchsorted is not catching first bin if arrival at time 0.

Does it make sense to add a validation check to make_hills that determines the length of warmup and horizon effects based on a rule?

The rule could be based on some combination of LOS and the number of days in the supplied date range. As an example, I am imagining it would compute the 75th percentile of LOS and then automatically adjust the start and end date based on this number. If the date range is quite large, the adjustment can be multiplied by a factor of 5 or something.

The main driver behind this is that casual users of the application might not fully grasp the implication of these effects and providing that adjustment for them could avoid misleading data. Of course, users who fully understand the concepts of these effects could opt to adjust the date range manually and could turn off this functionality with a simple validate_warmup_horizon_effects=False.

Datetimes in csv are text. Conversion to appropriate data type (pandas Timestamp) should happen at read time. However, what to do about the start and end analysis values used in arguments to hillmaker() main function? They need to be converted to pandas Timestamps before use in comparisons with other Timestamps. Do we force arguments to be Timestamps or try to convert them?

Right now, the summary dataframe has a multi-index:

day_of_week | dow_name | bin_of_day | bin_of_day_str |  

This makes it painful to select specific rows (e.g. Monday). If we flatten the index and stick the levels as regular columns, it will be much easier to select some subset of rows. In the plotting code, we flatten the index to make it easy to make the plot.

What do you think @jwnorm ?

Forgot to do this for v0.1.0

The hmlib is currently imported using *.

Check if:

• in and out cols are part of df
• end analysis date is after start analysis date
• cat field is part of df

By replacing the stops_df and stop_data_csv parameters with a single data parameter, we can move towards more general handling of different kinds of data inputs. Also makes it easier to process inputs since won't have to worry about what to do in case both parameters are used simultaneous (i.e. who gets precedence?).

The CLI should be just as powerful as the other APIs.

If pandas MultiIndex is output, the category field gets exported twice and the day of week and bin of day fields in the index contain many blank values. No need to output the MultiIndex.

The src layout has gained momentum as the recommended approach. I've used it on other projects with good results. Also need to move tests folder to project root.

Is this a bottleneck? Can approach I used in daily averages blog post be of help?

Figure out how to use the Warnings module

It might make sense to store application settings such as some of the key parameter defaults in a file instead of them being scattered throughout the code.