CA: CAs and vCAs rewarding system does not incentivize higher assessments quality (CAs) or higher number of assessment reviews (vCAs). about catalyst-prioritized-problems

[victorcorcino]

Yes, the solution remains in 'Prototyping' while it's being tested. Once it generates results, then we move it to 'Validation' and assess them, so we can decide to integrate the solution to the Catalyst process, or place it in the lessons learned column. Maybe 'Prototyping/Testing' would be more accurate than just 'Prototyping' as the column name.

[kenricnelson]

K Nelson, D Ribar, and T Saks met with Phil Lazos, IOG Researcher, on Jan. 11, 2022 to review research goals regarding CA rewards. Communicated two issues:

1. The lottery system for the allocation of rewards does not appear to be helping the program. The lottery should either be dropped or evidence should be provided regarding its merits.
2. The bonus to CAs based on which proposals are funded needs to be dropped.
3. Phil is going to investigate the possibility of a reward bonus based on the ranking match with the voter results. That way if an assessment gives a negative review and the voters agree by downvoting, that would be rewarded. In contrast, if an assessment is negative but the vote is in favor of the project, then a bonus would not be rewarded. This way the quality review on assessments (and the compensation) would have two components: a) the quality (filter/good/excellent) by the vCAs and b) the ranking match with the voters' opinion.
4. Phil is going to investigate a method to assess the original content of an assessment. If this is possible, correlated information such as copying and pasting between assessments could be detected, and assessments that provide voters with new and valuable information would be rewarded.

[kenricnelson]

This is Miro board that is being used to track discussions with the community and IOG regarding the changes to assessment process. https://miro.com/app/board/uXjVOXVhuVk=/

[stephen-rowan]

CA: CAs and vCAs rewarding system does not incentivize higher assessments quality (CAs) or higher number of assessment reviews (vCAs).

[stephen-rowan]

CA: CAs and vCAs rewarding system does not incentivize higher assessments quality (CAs) or higher number of assessment reviews (vCAs).

[manonthemat]

Saw this GitHub issue referenced on the Trello board under Prototyping (In Progress). With the changes to the CA/vCA reward system in Fund6, I was expecting this card to be in the Validation (Testing) column.

[stephen-rowan]

Saw this GitHub issue referenced on the Trello board under Prototyping (In Progress). With the changes to the CA/vCA reward system in Fund6, I was expecting this card to be in the Validation (Testing) column.

Hi Matthias, Circle Meeting 6 left it at "Prototyping (In Progress)" - because I think validation would require some confirmation that assessment quality has improved. But refer to @victorcorcino for confirmation. S

[manonthemat]

Thanks for the clarification.

[stephen-rowan]

Raw input from the community regarding improvements for the CA process:
https://docs.google.com/document/d/1ZCgivRkkwEe4VHPLIa1D7nHhwnXB3_9g_vwwQN8NcSo/edit#heading=h.g5p6c42acklc
Spreadsheet dropbox managed by Dan Ribar to consolidate and track ideas for improving the CA process:
https://docs.google.com/spreadsheets/d/120HylsMq-2oXeop7m8vBoaf7eb2D8viJazJs_EPWU2I/edit#gid=0

[nadiahopkins]

A new lottery system was implemented for F8 as a result of these discussions. That can be found here:

https://docs.google.com/document/d/1l4nI2SGlRoRHTv21VjVWN2oT7Kbloka0V7PrqSwh8fQ/edit

Kenric and I met on Monday, 3/21 to review the CA prioritized problems and identified this as a resolved issue the that had been assessed, improved and can be closed until further instances related to the lottery arise. Thank you!