nivlab / sciops

Code and data for "Inattentive responding can induce spurious associations between task behavior and symptom measures"

nivlab.github.io

The main hub for Niv lab's technical resource sites.

Development

Modifying the portfolio

For step-by-step instructions for adding to or modifying the project portfolio, see Portfolio Theme README.

Modifying the library

Edit the library.md file as you see fit :)

Previewing the demos page locally

If you'd like to preview the site locally (for example, in the process of proposing a change):

1. Clone down the project's repository (git clone https://github.com/nivlab/nivlab.github.io)
2. cd into the project's directory
3. Run bundle install to install the necessary dependencies
4. Run bundle exec jekyll serve to start the preview server
5. Visit localhost:4000 in your browser to preview the project

This starts a Jekyll server using your theme. Add pages, documents, data, etc. like normal to test your theme's contents. As you make modifications to your theme and to your content, your site will regenerate and you should see the changes in the browser after a refresh, just like normal.

When the theme is released, only the files in _layouts, _includes, and _sass tracked with Git will be released.

Acknowledgments

This resource was made possible thanks to Github Pages and the Portfolio Theme for Jekyll.

sciops's Issues

Main Figures

Figure 1 (simulation)

• Add correlation, p-value for corresponding data

Figure 2a (score distributions)

• Update sizing

Figure 2b (metric correlations)

• Add arrows(?), or something to denote correlation clusters

Figure 3a (spurious correlations)

• Finalize color scheme + annotations

Figure 3b (spurious correlation by percentage corrupted)

• Finalize choice of survey/correlate pairs

Figure 3c (softmax regression)

• Finalize HDIs

next round of analyses

• Under Q2: What is the correspondence of different metrics on rejecting participants under different thresholds? Look into generalizations of the Dice coefficient?
• Under Q3: Is there are correlation between asymmetric learning rates, questionnaire sum scores, and low-effort responding?
• Under Q3: Dig into hypomania correlations?
• Under Q3: Is there some combination of metrics that can be used to predict low-effort responding via infrequency items assuming they were not collected? Look into decision tree literature?
• Under Q3: Simulations

Outstanding analysis items

• Include PSWQ?
• Combine BAS-D / BAS-R?
• Fix RSTD model?
• Softmax parameterization?
• 7u/7d cutoffs
• Journal?

pre-launch discussion points

1. Sufficient infrequency items?
2. Long-string analysis?

progress 2020-07-11

Some points for discussion after looking through the data today:

• Infrequency thresholds: we may want to think about the consistency between the different infrequency items, which is lower than what would be expected under pure random responding. Obviously pure random responding an unrealistic assumption, but I'm wondering if there's anything else to say about those items (e.g. all-endorse items are somehow less discriminative?).

• Additional survey metrics: there are some recommended survey quality metrics I have not yet implemented as they are somewhat challenging for our dataset. A metric like internal (split-half) consistency is possibly less robust in our case where we have few items per subscale. Similarly, it's not clear if we have enough items to compute consistency via "psychometric synonyms/antonyms". It doesn't seem crucial to me to compute all of these survey metrics as they're not the crux of the paper -- that said, if there's an easy way to compute these it'd be interesting to compare them to behavioral metrics (re: Major Point #2, behavior =/= survey thresholding).

• Thresholding non-behavior metrics: it is somewhat more clear what the anchor points are for thresholding behavior (i.e. chance). It's somewhat less clear for other metrics (total experiment duration, entropy, Mahalanobis D). It's possible the literature may have some recommendations. Short of that, we'll want to think about a sensible rule.

BAS updates

requires re-running of 1pl logistic

data collection (pt 2)

• program surveys
• program attention checks
• program two-step task
• test experiment
• collect data