healthrex / cdss Goto Github PK

Probably need AWS setup to do multiple
After getting flowsheet extraction

[ ] Setup authoring worksheet and example (GI Bleed) so clinical collaborators can follow suit

E.g., Some highly protocolized (e.g., neutropenic fever) vs. some vague diagnostic cases (e.g., altered mental status)

See Isabel evaluation for example setup, though looks largely idiosyncratic custom authoring.
https://bmcmedinformdecismak.biomedcentral.com/articles/10.1186/1472-6947-6-22

Otherwise do more web searching for differing simulation cases (e.g., Medscape education simulation cases or NEJM clinical case reviews, etc.)

12/2017 - Amit interfacing with Lee? on ~22M text pages, but narrowing this down to a reasonable (and approvable) subset.

The editor/reviewer here seems to dislike the idea more than the study, with order lists that are too broad and not specific enough. This is about some framing about how to use decision support. If it’s a fixed automatic protocol / cookbook, then that should be used a different way. This fits with a different kind of existing order set design, which is more like an information retrieval / search results perspective providing an enriched menu of options to consider, with the expectation that the user will only pick out the ones they want for their individual patient. Maybe we drop the Altered Mental Status example that he’s hung up on, since that specifically was one with less clear guidelines, but we tried to include to show the sensitivity of the approach to different levels of specificity.

Anyway, I think we try to come up with some reframing edits and responses to these criticisms back to JAMIA, including a reference to our prior JAMIA publications (e.g., OrderRex) and others (e.g., COWPath) that use the same idea to point out that these are valid areas of investigation, and further back that up with the IMIA Yearbook naming the OrderRex study as a Best Paper in Secondary Use of Clinical Data.

If the editors still won’t send out to review, then we’ll try somewhere else (JBI or IJMI?)

Asked Jason Hom if he is up for any writing parts, while team puts together technical results

When I run TestFeatureMatrixFactory, it still leaves behind a few temp files:

fmf.patient_episodes.tsv
fmf.patient_list.tsv
patient_list.tsv

Some are from the application code, while I think the last is from the test code.
I recommend doing the temp file "cleanup" in the tearDown function of the unit test (will always be run, even if the test doesn't pass) for the latter.

For temp files generated by the application FeatureMatrixFactory class, you may consider putting the temp file cleanup in a destructor (del) function, to make sure it gets done by the time of program completion.

Figure out from Clinical Experience course, what paperwork is necessary to get students/staff eligible for shadowing through clinical environment (i.e., hospital or clinic)

Resubmit the IRB forms / waivers to access the Population Health Sciences claims data sources (E.g., Truven + Optum). Previously held up because my official appointment as faculty was pending, but that should be resolved now.

@jonc101 – looks like there's a limit on the # of people we can add to a private repository, and we hit that limit when trying to add Ron. We can pay for additional slots, but that's $10/person/month.

Creating this task to (a) decide whether we actually want to make the repo public, (b) figure out what we'd need to take care of before doing that, and (c) actually handling those prerequisites.

It's very unlikely anyone would discover the repo on their own, but here's a few things we should probably still take care of:
– Scrub for PHI.
– Scrub for credentials and other security vulnerabilities.
– Add a license indicating terms for using the code (probably just MIT license).
– Update README with instructions on how to build, test, and contribute.

Something in SVN revision 1652 must have broken something.

Module:

• Input: Feature matrix data file for a target (lab) test to predict and list of features/columns/covariates to include or exclude from consideration.
• Output: Collection of standard ML models (e.g., logistic regression, LASSO, random forest, XGBoost, AdaBoost, SVM, neural net? though hard to use "off the shelf") across a credible range of hyperparameters (e.g., lambda coefficient, tree depth, etc.)

This isn't urgent, but we know we'll need to do it eventually and it'll be painful =)

• Norms for writing and utilizing unit/regression tests
• Norms for communicating in email vs. slack
• Norms for a code review-ish process (developing on own branch vs. pushing to master)

Comments received 12/14/2017
Resubmission due 1/9/2018

Shared "lab" email address and AWS account to facilitate centralized admin + billing.

Start with just conversion of current webpage
http://web.stanford.edu/~jonc101

Will see if can get admin assistant support to maintain a site through an online templating tool

12/14/2017 - Spoke with assistant service to look for someone who can help

[ ] Update InsertData.py for Order Data

[ ] Use more recent data - Update normalization script
[...] User Table
Include fill in blanks for those without SID, etc. info
Lookup SIDs for atendings??? Maybe later
[...] Rotation / Assignment Table
[ ] Define schema
[ ] Relation melt resident schedules
[ ] Transform Attending, Resident + Student into importable form (maybe do raw import/copy first, then make it easier to manipulate from there)

• User Rotation / Assignment Schedule Table
  • prov_id (Epic SID)
  • assignment_id ->
    assignment_table, name (Med, or just ICU, ER, outpatient, etc.). Don't break down by Team A,B,C
    Maybe annotate with "isInpatient" or not "isConsult"
  • start_date
  • end_date
  • Role
    MS3, MS4, R1, R2, R3, Attending
  • Team
    A,B,C, or null

[ ] Desktop repopulate
[ ] Repopulate normalized ADT and TTeam tables
[ ] Populate stride_pager_cover data
[ ] Create Indexes

[ ] Itemize types of activity / interactions
[ ] Bracket out by access / workstation type (remote logins)

• Input results of issue #35, have a battery of predictive models / classifiers.
• Calculate standard evaluation metrics such as ROC AUC, AU Precision-Recall Curve, calibration curve, Precision at top X(%) with respective total report
• Allow sorting/selection of models by best performance by given metric

• Example of join without subqueries
• Example of medication orders

Given we're likely to load the database in more environments, we should probably figure out some simple tests for validating both the schema and the actual data.

Jonathan and I thought we both had the same data, but then it turned out I was missing a bunch of clinical_item_category rows, and it's still not entirely clear why.

In the simplest case, we should at least validate the # of records.

Transfer / copy over web medicine "tools" from SVN repo to GitHub under CDSS/medinfo/web

Go see other issue with list of patches to make

I'm somehow using the logging functionality incorrectly by assigning subpackage app names for each Util.log, which seems to result in lots of redundant logging. Try to get rid of this, probably easiest way is just change all apps and tests to use one reference name?

Previous steps enable learning of some standard regression model for a predicted outcome (e.g., logistic regression or LASSO for binary outcomes, Cox proportional hazards for survival time).
[ ] Pull out the coefficients from an above model, round them so they are all multiples of the smallest -> Translate into a simple integer scoring system for the target outcome based on the component features.
[ ] For numerical features, may require binning of values into smaller buckets to really make a simple score with only binary/nominal distinctions (e.g., heart rate binned into 3 categories: <60 or 60-100 or >100, rather than using raw number).
[ ] Translate numerical scores into respective probabilities or mean survival times for the outcomes either analytically from model, or better yet empirically from cross-validation data (in likely case of poor calibration).
[ ] Use above to either generate a numerical model to calculate the score, or could just be lookup tables of the components to hard code at this point.

For individual applications, will then want to empirically evaluate the efficacy of the risk score (vs. full model vs. general ML models vs. baseline prevalence) at predicting outcomes in a held out test set of future data (eval discrimination by ROC AUC, precision-recall curve, and 10 split calibration).

Goal is to get it so that a new developer is ready to work in <1 hour.
For the code, this should be an easy matter of cloning github and installing libraries.

For the data, there is one step (currently).
• Convert raw CSV files to proprietary postgres DB format (takes multiple days)
• Create snapshot of postgres DB (this is what's set up on RDS)
• Load snapshot on local machine

In the future, we could potentially:
• Convert raw CVS files to raw postgres DB format (should be quick)
• Convert raw postgres DB format to proprietary postgres DB format (may still take a while)
• Create snapshot of proprietary postgres DB (this is what's set up on RDS)
• Load snapshot on local machine

It's fine if it's a long one-time script, but just try to avoid lots of individual commands.

Try dumping the database that we already have and then loading that back into PostgreSQL. How long would it take to dump and restore the full database?

Right now generates recommended/associated list based on accumulated clinical data as query.
Try instead break down by different individual query items (e.g., admission diagnosis, uncommon lab results or orders) and allow showing of what is triggering each set of recommendations

Older data uses ICD9, but post-2014 data uses ICD10 codes, so will need to figure out a way to transform and work with both. This will likely be lossy because the mapping is many:many

Double check the source (PDF) data to confirm interactive web page is producing same results:

• Probably had some column shift issue when generating the data tables, as the current version is showing MSSA Penicillin sensitivity at 1454, which doesn’t make sense, since it’s supposed to be a percentage. That’s supposed to be the number tested. Similarly, the sensitivity for clindamycin should be 80, but instead it has 72, which looks like a misplaced copy over from the value for erythromycin.

Other issues

• Don’t add (anaerobes) to all of the Gram Positive Cocci categories. Most such bacteria are not anaerobes. There is already a subset of bugs I named “Gram Positive Cocci (anaerobes)” that you can use for those categories.
• Trimethorprim-Sulfamethoxazole is the same as TMP-SMX. Don’t need to add another entry.
• I think you might have missed the entire “front page” of the PDF. For example, including sensitivities for Enterococcus and Candida species.
  o http://lane.stanford.edu/biomed-resources/antibiograms-shc.html

Should include the following:
• Ways to connect to the database
• List of 5-10 queries of increasing complexity answering clinical questions

Test creating issue

Given a feature matrix with a designated target column and candidate input / feature / independent variable columns (inclusion and/or exclusion list), sort or select the candidate variables by those with the most explanatory power for the target column.

sklearn has a feature_selection API that can facilitate much of this already. Make API usage general enough that can test multiple approaches, but some simple/common ones:

• Filter by univariate metrics (e.g., mutual information, correlation)
• Wrappers (e.g., Greedy Forward) with Cross-Validation: Build prediction model iteratively, adding one feature at a time, but only keep it if it significantly improved cross-validated accuracy. More comprehensive: Try adding each feature alone, and accept only the one that improves CV accuracy the most (assuming adds enough at all). Then recursively continue on the remaining features that were not yet accepted.
• Embedded: LASSO or Random Forest to naturally prune variables as try to build prediction model

Maybe easisest to try first: Just train a random forest against the target outcome, which naturally selects/sorts the features that are most effective at discriminating for the target outcome.

Add JavaScript or similar layer of tracking so can record user

• Time on task
• Number of button clicks and keystrokes
• Number of orders entered over...
• Number of orders considered
  (How many presented through direct search, through recommended lists, through conventional order sets)

Explanatory comment: https://github.com/HealthRex/CDSS/blob/master/medinfo/dataconversion/STRIDEOrderMedConversion.py#L270

At the moment, it includes calls to a bunch of medinfo/ libraries (e.g. TestDBUtil), which make it hard for us to just pass off to other researchers who want access to the data.

For simulated cases developed, work through with "clinical expert panel" to assign scores (positive or negative) for the most common and likely medical decisions / clinical orders.

Too laborious to score 2,000 possible choices, but do the most common up front and then go back and re-review after initial test runs to score any extra steps people took that were not expected.
Probably need scoring per phase/state of each simulation.

TBD: Define a scoring rubric so different experts can aim for some kind of consensus?

• Allow record of each case and each expert and each decision, what points to assign for each
  • This should allow for reconciling or averaging multiple votes.
• Allow grading of points for a choice no matter which state a patient is in. Only if patient in a given state or timing / sequence of choice. For example, ordering CT Head for meningitis not necessarily wrong, but if wait until result to start antibiotics, then wrong.
• Only getting points once if multiple satisfactory choices (e.g., multiple antibiotic options).

Design such that can be stored in a spreadsheet, or better yet, a new "sim_" table in SQL database, so can then just run some simple application code to auto-grade all cases.

Look for the 1-line python command in setupNotes.txt

Largely done already in shared Box folder for MedicineResidentAccessLog2017, including text paging contents, hospital summary information, ADT (admit-discharge-transfer), Treatment Team involvement, etc.

In 12/2017, got data for individual clinical order resolution, but still needs to be loaded / added to insertData.py script, and possibly go through a "clinical item" conversion process as per the medinfo/dataconversion/STRIDE*Conversion.py scripts.