BUG: Lotteries shown in "description" task do not necessarily correspond to those in the "active" task about sp_experiment

[sappelhoff]

How many trials are unusable?

• We have 40 subjects, each having done 100 trials in the "active" part of the experiment --> 4000 trials

• Each of these trials needs to be translated to a "description" format

• These are two ways in which a trial in description can be considered "unusable" (i.e., not reflecting the "active" counterpart):

  • 1. When a subject samples only one option, there is no "experienced" description for the other option available. In order not to display an empty description, these trials were replaced in the experiment with the "true" underlying description. this was planned from the beginning and the intention was to simply remove these (predicted to be few) trials from analysis.
  • 2. When the bug described in the comment above took place

• The number of dropout trials due to 1) can simply be counted.

• The number of dropout trials due to 2) can be calculated by comparing the output of this function:

  sp_experiment/sp_experiment/utils.py

  Lines 169 to 249 in 490d34d

  	def _get_payoff_setting(df, trial, experienced=False):
  	"""Get the payoff setting.
  	Get the setting and reformat it to fit with internal usage in
  	`define_payoff_settings.py`. That is, the columns are organized as:
  	outcome 1.1, outcome 1.2, probability 1.1, probability 1.2, outcome 2.1,
  	outcome 2.2, probability 2.1, probability 2.2.
  	Parameters
  	----------
  	df : pandas.DataFrame
  	The data from which to obtain payoff settings.
  	trial : int
  	The trial within the data for which to retrieve the payoff setting.
  	experienced : bool
  	if True, get the experienced payoff_setting instead of the true one
  	Returns
  	-------
  	setting : ndarray, shape(1, 8)
  	The payoff setting, first dimension is empty.
  	Notes
  	-----
  	Always take the last available setting, because the previous ones (if there
  	are any) have been dropped due to an error (if there was one)
  	"""
  	trigger_dict = provide_trigger_dict()
  	error_trig = ord(trigger_dict['trig_error'])
  	df = remove_error_rows(df, error_trig)
  	tmp_df = df[(df['trial'] == trial)]
  	if not experienced:
  	tmp_df = tmp_df.loc[:, 'mag0_1':'prob1_2'].dropna()
  	# wrong format is mag, prob, mag, prob, ...
  	wrong_format_setting = tmp_df.iloc[-1].tolist()
  	elif experienced:
  	# outcome is always 2 events after sample action
  	sample_idx = tmp_df[tmp_df['action_type'] == 'sample'].index
  	outcome_idx = sample_idx + 2
  	sides = tmp_df.loc[sample_idx, 'action'].values
  	outcomes = tmp_df.loc[outcome_idx, 'outcome'].values
  	# experienced expected values for left and right option
  	payoff_dict = {side: list(outcomes[sides == side]) for side in [0, 1]}
  	# Calculate probabilities for each outcome
  	wrong_format_setting = list()
  	for side in [0, 1]:
  	distr = payoff_dict[side]
  	# Special case 1 if only one distr sampled: add two 99s with
  	# each p=99
  	if len(distr) == 0:
  	for i in [99, 99, 99, 99]:
  	wrong_format_setting.append(i)
  	continue
  	# Special case 2 if only one outcome sampled: add a 98 with p=0
  	outcomes = np.unique(distr)
  	if len(outcomes) < 2:
  	outcomes = np.append(outcomes, np.array(98))
  	# Go though outcomes
  	for out_i in outcomes:
  	wrong_format_setting.append(out_i)
  	# Round it to one decimal
  	p = np.round(distr.count(out_i) / len(distr), 2)
  	wrong_format_setting.append(p)
  	# Correct format from mag, prob, mag, prob ... to mag, mag, prob, prob ...
  	setting = np.array(wrong_format_setting)[[0, 2, 1, 3, 4, 6, 5, 7]]
  	setting = np.expand_dims(setting, 0)
  	# quick sanity check that we have proper roundings, for example 0.3 instead
  	# of 0.29999999 ... 1., 2., 3., etc. would be fine (as magnitudes)
  	for entry in setting[0]:
  	# 2 for magnitudes, 3 or 4 for probs ... or special entry "99"
  	# (experienced)
  	assert (len(str(entry)) in [2, 3, 4]) or entry in [98, 99]
  	return setting

• namely: What are the return values under parameter experienced=True compared to experienced=False

• if the returned magnitudes (ignoring probabilities and "98" special magnitudes) are in order regardless of the experienced=True/False parameter value, the trial is usable ... else, it is not.

We did such an analysis, and the overall result is:

A proportion of 0.84 (3352/4000) of all trials in the Description task is not mirroring what happened in the Active task.

Thus a huge proportion of trials is not usable. This data is thus not analyzed at all.