Keeping track of states about natural-instructions

[eshaanpathak]

I can take this on. How many tasks should there be? I was thinking 4 based off of the directories, but there are different task types and question types in the directories. Please let me know what you think!

Pinging @swarooprm on this to get his thoughts too.

[swarooprm]

Yes, there can be 4 tasks at least. I agree that there can be more tasks based on the question types and task types, but that might be slightly tricky to find out.
Please follow these rules whenever you plan to break a dataset into multiple tasks.

• The difference between two tasks can not be just based on the difference in the question asked. E.g. in a typical QA dataset say SQuAD there are multiple questions, but they all belong to the same task. However, it can be possible that questions are in different formats (QA, NLI or completion) or require different skills to answer (common-sense knowledge, maths knowledge etc.), in those cases, you can create multiple tasks for each format and skill.
• In order to explain two tasks to anyone say your friend, the definition has to be sufficiently distinct. E.g. question generation for a dataset is a different task than answering it.

Feel free to ask any other question you might have.