Comments (7)
Overall these basic endpoints make sense, but I want to clarify a bit on the design.
I don't know where we'd use "feedback"
for approve, and we don't really have metadata about qualifications (like if it's a block) at the moment. On granted-qualifications
in general, we tend to use grant
and revoke
, and we separate these out from approving/rejecting units.
Some key endpoints will be /approve
, /soft-reject
, and /reject
, and they should be able to take multiple unit ids as an argument such that we can do approve-all/soft-reject-all/reject-all queries. Units can also end up having large quantities of data, so we'd likely only want to load the data for a specific unit being actively viewed (though we can pull metadata for chunks).
An aside on the expected review flow:
When going through review, the expectation is that we'll be able to review all of the units in one pass, ideally batched by worker. This behavior in the mock operates through the following steps.
- Aggregate all of the units for the given
task_name
. - Group these units by worker, and compute previous approval statistics for reviewed work while bucketing the unreviewed work. We then shuffle the unreviewed work to lower the chance of missing quality degradation or ramp up phases if only sampling a few units per worker.
- Show the work to the reviewer in worker-grouped batches, with additional previous approval statistics as well as overall counts (how many workers and total units are left to be reviewed).
- Reviewer is able to look at as many per worker as they like, individually actioning. Then, they can bulk apply for the rest.
- Run until all units are reviewed.
This type of setup requires constructing and maintaining some kind of local state on the server, though the setup I've used so far is really not the right way to do it.
from mephisto.
Review flow
- we get list of available tasks from
GET /tasks
- User selects a task
- we get list of available qualifications from
GET /qualifications
- (optional) User selects "approve" and "reject" qualifications
- We pull all unit ids from
GET /tasks/{id}/worker-units-ids
- Due to the need to randomly shuffle units grouped by a worker (to mitigate reviewers bias, etc) we're implementing client-side pagination - client gets full list of all ids, creates a page of unit ids, and then pulls data for those specific units.
- We group units by worker, sort workers by number of their units, and pick them for review one-by-one
- For each worker:
- we pull worker stats from
GET /workers/{id}/stats
- we pull units by ids from
GET /task-units?unit_ids=[...]
- we sort units by
creation_date
and pick them for review one-by-one- for each reviewed unit we pull its details from
GET /task-units-details?unit_ids=[...]
- User can choose to reject/accept unit, grant/revoke qualification, and block the worker
- for each reviewed unit we pull its details from
- we pull worker stats from
API v2
GET /tasks
Get all available tasks (to select one for review)
{
"tasks": [
{
"id": <int>,
"name": <str>,
"is_reviewed": <bool>,
"unit_count": <int>,
"created_at": <datetime>
},
... // more tasks
]
}
GET /qualifications
Get all available qualifications (to select "approve" and "reject" qualifications)
{
"qualifications": [
{
"id": <int>,
"name": <str>,
},
... // more qualifications
]
}
POST /qualifications
Create a new qualification
{
"name": <str>,
}
GET /tasks/{id}/worker-units-ids
Get full, unpaginated list of unit IDs within a task (for subsequent client-side grouping by worker_id and GET /task-units
pagination)
{
"worker_units_ids": [
{
"worker_id": <int>,
"unit_id": <int>,
},
... // more ids
]
}
GET /task-units?{task_id=}{unit_ids=}{qualification_id=}{worker_id=}
Get workers' results (filtered by task_id and/or unit_ids, etc) - without full details of input/output
{
"units": [
{
"id": <int>,
"worker_id": <int>,
"task_id": <int>,
"pay_amount": <int>,
"status": <str>,
"creation_date": <int>,
"results": {
"start": <timestamp>,
"end": <timestamp>,
"input_preview": <json str>, // optional
"output_preview": <json str>, // optional
},
"review": {
"tips": <int>,
"feedback": <str>,
}
},
... // more units
]
}
GET /task-units-details?{unit_ids=}
Get full input for specified workers results (units_ids
is mandatory)
{
"units": [
{
"id": <int>,
"input": <json str>, // instructions for worker
"output": <json str>, // resposne from worker
},
... // more units
]
}
POST /task-units/reject
Reject worker's input
{
"unit_ids": [<int>, ...],
"feedback": <str>, // optional
}
POST /task-units/soft-reject
Soft-reject worker's input
{
"unit_ids": [<int>, ...],
"feedback": <str>, // optional
}
POST /task-units/approve
Approve worker's input
{
"unit_ids": [<int>, ...],
"feedback": <str>, // optional
"tip": <int>, // optional
}
GET /granted-qualifications?{task_id=}{qualification_id=}{worker_id=}{limit=}{offset=}
Get all granted qualifications (filtered by task_id and/or qualification_id, etc)
{
"granted_qualifications": [
{
"id": <int>,
"worker_id": <int>,
"qualification_id": <int>,
"value": <int>,
"creation_date": <int>,
},
... // more granted qualifications
]
}
POST /granted-qualifications/grant
Grant qualification to worker
{
"qualification_id": <int>,
"worker_id": <int>,
"value": <int>,
}
POST /granted-qualifications/revoke
Revoke qualification from worker
{
"qualification_id": <int>,
"worker_id": <int>,
"block_permanently": <bool>,
}
POST /workers/{id}/block
Permanently block a worker
{
"feedback": <str>,
}
GET /workers/{id}/stats?{task_id=}{requester_id=}{since=}{limit=}
Get stats of recent work for the worker
"worker_stats": [
{
"id": <int>,
"stats": {
"total_count": <int>, // within the scope of the filters
"approved_count": <int>,
"rejected_count": <int>,
"soft_rejected_count": <int>,
},
},
]
from mephisto.
API v0
GET /tasks
Get all available tasks (to select one for review)
{
"tasks": [
{
"id": <int>,
"name": <str>,
"is_reviewed": <bool>,
"unit_count": <int>,
"created_at": <datetime>
},
... // more tasks
]
}
GET /qualifications
Get all available qualifications (to select "approve" and "reject" qualifications)
{
"qualifications": [
{
"id": <int>,
"name": <str>,
},
... // more qualifications
]
}
POST /qualifications
Create a new qualification
{
"name": <str>,
}
GET /task-units?{task_id=}{qualification_id=}{worker_id=}{limit=}{offset=}
Get all workers results (filtered by task_id and/or qualification_id, etc)
{
"units": [
{
"id": <int>,
"worker_id": <int>,
"task_id": <int>,
"pay_amount": <int>,
"status": <str>,
"creation_date": <int>,
"results": {
"start": <timestamp>,
"end": <timestamp>,
"input": <json str>,
"tips": <int>, // optional
"feedback": <str>, // optional
}
},
... // more task units
]
}
GET /granted-qualifications?{task_id=}{qualification_id=}{worker_id=}{limit=}{offset=}
Get all granted qualifications (filtered by task_id and/or qualification_id, etc)
{
"granted_qualifications": [
{
"id": <int>,
"worker_id": <int>,
"qualification_id": <int>,
"value": <int>,
// "feedback": <str>,
"creation_date": <int>,
},
... // more granted qualifications
]
}
POST /granted-qualifications/reject
Reject worker's input
{
"qualification_id": <int>,
"worker_id": <int>,
"block_permanently": <bool>,
"feedback": <str>, // optional
}
POST /granted-qualifications/approve
Approve worker's input
{
"qualification_id": <int>,
"worker_id": <int>,
"value": <int>,
"feedback": <str>, // optional
"tip": <int>, // optional
}
from mephisto.
API v1
GET /tasks
Get all available tasks (to select one for review)
{
"tasks": [
{
"id": <int>,
"name": <str>,
"is_reviewed": <bool>,
"unit_count": <int>,
"created_at": <datetime>
},
... // more tasks
]
}
GET /qualifications
Get all available qualifications (to select "approve" and "reject" qualifications)
{
"qualifications": [
{
"id": <int>,
"name": <str>,
},
... // more qualifications
]
}
POST /qualifications
Create a new qualification
{
"name": <str>,
}
GET /task-units?{task_id=}{qualification_id=}{worker_id=}{limit=}{offset=}
Get all workers results (filtered by task_id and/or qualification_id, etc) - without the details of worker's output
{
"workers": [
"id": <int>,
"stats": <json str>,
"units": [
{
"id": <int>,
"worker_id": <int>,
"task_id": <int>,
"pay_amount": <int>,
"status": <str>,
"creation_date": <int>,
"results": {
"start": <timestamp>,
"end": <timestamp>,
"input_preview": <json str>, // optional
"output_preview": <json str>, // optional
},
"review": {
"tips": <int>,
"feedback": <str>,
}
},
... // more units
]
],
... // more workers
}
GET /task-units-details?{unit_ids=}
Get full input for specified workers results
{
"units": [
{
"id": <int>,
"input": <json str>, // instructions for worker
"output": <json str>, // resposne from worker
},
... // more units
]
}
POST /task-units/reject
Reject worker's input
{
"unit_ids": [<int>, ...],
"feedback": <str>, // optional
}
POST /task-units/soft-reject
Soft-reject worker's input
{
"unit_ids": [<int>, ...],
"feedback": <str>, // optional
}
POST /task-units/approve
Approve worker's input
{
"unit_ids": [<int>, ...],
"feedback": <str>, // optional
"tip": <int>, // optional
}
GET /granted-qualifications?{task_id=}{qualification_id=}{worker_id=}{limit=}{offset=}
Get all granted qualifications (filtered by task_id and/or qualification_id, etc)
{
"granted_qualifications": [
{
"id": <int>,
"worker_id": <int>,
"qualification_id": <int>,
"value": <int>,
"creation_date": <int>,
},
... // more granted qualifications
]
}
POST /granted-qualifications/grant
Grant qualification to worker
{
"qualification_id": <int>,
"worker_id": <int>,
"value": <int>,
}
POST /granted-qualifications/revoke
Revoke qualification from worker
{
"qualification_id": <int>,
"worker_id": <int>,
"block_permanently": <bool>,
}
POST /workers/{id}/block
Permanently block a worker
{
"feedback": <str>,
}
from mephisto.
Note, /task-units-input
should be getting the full task data including outputs, not just inputs. We'd be feeding this directly to the review component.
Also, I imagine it makes sense to just move block
to its own endpoint.
from mephisto.
Makes sense, I've just edited API v1.
from mephisto.
This has been implemented in v1.2.0
from mephisto.
Related Issues (20)
- Socket logged error: no close frame received or sent HOT 7
- Unit status not properly synced with long-running generators HOT 1
- Move `eventEmitter` out of `wrap_crowd_source.js` into `mephisto-task`
- Outline release roadmap HOT 2
- __heroku_app_url empty when using heroku HOT 1
- How can I programmatically give worker bonuses during task run? HOT 3
- Frontend install guide - docusaurus v3, and possibly outdated? HOT 2
- Prolific could not retreive a Submission HOT 10
- Docs for integrating Mephisto with Prolific not searchable in docs HOT 2
- Enable worker bonusing and feedback in TaskReview App HOT 1
- Go to next assingment after submit
- docker fails to run HOT 1
- 10-minute Quickstart Docker Build Failing HOT 3
- raise InvalidRegionError(region_name=region_name)
- Enable data consolidation when running Mephisto on a remote server
- Ability for the Task logic to declare itself completed
- Investigate concurrent Task Runs with Mephisto
- Error on initial example HOT 3
- Updating stale ownership of the Docker testing Github Action by disabling and re-enabling the workflow HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from mephisto.