mr8nd / tc-prediction-bands

Hi there 👋

I am now a senior AI Researcher at JPMorgan AI Research in New York. Right before that I was a PhD student in the Statistics & Data Science department at Carnegie Mellon University, where I worked in statistical machine learning methodologies for uncertainty quantification.

• 💬 If you are interested in any of the code, feel free to fork and start playing with it!
• 📫 You can reach me on Twitter (Mr8ND) or on LinkedIn (Nic Dalmasso)
• 😄 Pronouns: He/His

tc-prediction-bands's Issues

JASA document prep

Complete commentary for JASA submission on reproducibility and data access https://docs.google.com/document/d/14tPBrHcBKk5VuAtKA5SAc4OD-CM_LCM3GqhKVVRVUg8/edit?usp=sharing

Main script cleanup (figures and analysis code)

For scripts in the main folder:

• 1. we should rename the files such that the order is obvious (following Hadley Wickam's style guide ) in the following manner: @RobinMDunn (First pass done by @benjaminleroy)
  • 1. @RobinMDunn: is 2-save_train_test.R reproducible? Why is "new_data" just an empty list?

0-download.R
1-parse.R
2-explore.R

• 2. we should create a README.md for this directory that tells people how to run the files, e.g. @Mr8ND

Rscript main/credible_interval_pipeline.R a b

with detailed information on what parameters a, b should be. Additional comments about "parallelization" would also be useful.

• 3. When done with our analysis we should drop the "Sys.Date()" approach to make the pipeline completely reproducible @Mr8ND

• 4. Clean up Table and Images output: @RobinMDunn @benjaminleroy

  • 4.a. Make all images and tables end up in report/images/ and report/tables respectively. Note that tables can be created with xtable like done in main/acc_vs_area_results.R.
  • 4.b. Push most recent code for creating tables. (@benjaminleroy)
  • 4.c. In tables, change Auto to AR and change Non-Auto to Non-AR.
  • 4.d. make graphics and table use of CB types and Sim types consistent. Possibly follow approach inmain/acc_vs_area_results.R (approx line 85) - or that in main/cb_validation_results.R - they are different.

• 4.5. Move a clean version of latex document into report folder. @RobinMDunn

  • 4.5.1. Clean up Latex (remove images that are not used, put sections in as section folder, etc) on sharelatex. @RobinMDunn

Visualization functions:

• write visualization functions to take in either the raw objection or the desired data frame (@benjaminleroy)

Writing Test Down

@Mr8ND - this needs be done after lat/lon correction, and other debugging. Functions can't be correctly written before this (ie. Issue #33) must be done before this issue. ~ @benjaminleroy

The following needs to be done to include tests in the repository

• 1. Understand where and how tests for R package should be written @Mr8ND
• 2.0. write list of functions that need tests @Mr8ND
• 2.1. Identify basic and mechanical functions to be included in the tests
• 3. Obtain a sample of 1/2 TCs to be used as "sanity-check" for the confidence band pipeline - all the 4 confidence bands should be obtained so that we can re-run the test anytime we change something about any of the 4
• 5. (Optional) Investigate whether TRAVIS integration might be quick and useful for a repository

For distance calculation between TCs

For TC's it might be desirable to include bearing and speed (or a 2d derivative) to the function space (making the function space 4d) - it probably would highlight differences between TCs better. (This is decently common to use "derivatives" in FDA).

Test names don't match Test_Sims_350.Rdata

Created test names in save_train_test_names.R do not match old version of Test_Sims_350.Rdata. This effect work down the line in the results and figures where we select a specific TC to examine.

Write a working demo in RMarkdown/Jupyter Notebook

Writing some code to show off:

• Simulation mechanism
• By passing in a pre-calculated simulated curves, we can compute the 4 confidence bands and visualize them

Add a citation?

citation("TCpredictionbands")

Correct names of training and test TCs

Created test names in save_train_test_names.R do not match old version of Test_Sims_350.Rdata. This effect work down the line in the results and figures where we select a specific TC to examine.

Delta Ball Error

There is an error in the ggvis_delta_ball function that may affect all delta ball functions. The fed in data appears to have lat & long vectors reversed. Problematically this probably affects all associated functions.

Clean-up needed

R code should have

• "library(blah)" instead of "require(blah)"
• make sure all files can be run from the project directory and that important files get copied into the final_scripts directory

On markdown file (after Ben updates it)

• add a basic description for each r library included
• add a basic description for each file that is included in the final_scripts directory
  @Mr8ND @RobinMDunn

Code Cleanup (inner function related)

We need to standardize the following:

• 1. change all lon variables to long (as of 6 September 2018)
• 2. make the input parameters lon = 1, lat = 2 into the common c_position = 1:2 (just make a decision on which should be done) Ruling: now use position
• 3. change the names of things from credible to prediction
• 4. Change bubble_CI notation to spherical
• 5. correct spherical and delta ball ordering of lat and lon (incorrect)
  • 5.1 spherical functions: bubble_ci_from_tclist -> ggvis_bubble_data
  • 5.2 delta ball functions: delta_ball_wrapper -> ggvis_delta_ball_contour
• 6. For PBs should use parameter "alpha" not alpha_level to express the associated alpha level (to make 1-alpha PBs). Also alpha_ci_level should then be "alpha_ci".