Spacing/tokenization issues about natural-instructions

[danyaljj]

As far as we know, these sentences were directly taken from a different dataset. I don't think we changed the tokenization; they came in like this.

[sanderland]

As far as we know, these sentences were directly taken from a different dataset. I don't think we changed the tokenization; they came in like this.

There is no tokenization to change, that is just speculation on the origin.

Regardless of the origin and cause, you do not consider mis-formatted strings and "UNK" strings in your training data to be an issue?

[danyaljj]

I think the segment you mention is from this file which was introduced in this PR: #284 The PR does not mention the origin of the data (only cites the relevant paper).

My opinion is that, these would a concern is they are incomprehensible to humans (which I think they are not, unless I am missing something).

[sanderland]

Some examples with unreadable outputs:

https://github.com/allenai/natural-instructions/blob/master/tasks/task1579_gigaword_incorrect_summarization.json

    "Positive Examples": [
        {
            "input": "four east timorese youths who scaled the french embassy 's fence here thursday , left the embassy on their way to portugal friday .",
            "output": "UNK latest east javanese asylum seekers leave for peru",
            "explanation": "The example is correct, as the location names are different from the passage"
        },
        {
            "input": "bosnian croat forces have begun torching homes in parts of western bosnia captured during a summer offensive but due to return to serbian control under the dayton peace agreement , un officials said friday .",
            "output": "croats build homes in areas for development",
            "explanation": "The example is correct, as it incorrectly summarizes the passage"
        }
    ],
    "Negative Examples": [
        {
            "input": "five east timorese youths who scaled the french embassy 's fence here thursday , left the embassy on their way to portugal friday .",
            "output": "UNK latest east timorese asylum seekers leave for portugal",
            "explanation": "The example is incorrect, as it correctly summarizes the passage"
        },
        {
            "input": "bosnian croat forces have begun torching homes in parts of western bosnia captured during a summer offensive but due to return to serb control under the dayton peace agreement , un officials said friday .",
            "output": "croats torch homes in areas due to return to serbs",
            "explanation": "The example is incorrect, as it correctly summarizes the passage"
        }
    ],
    "Instances": [
        {
            "id": "task1579-7f515582fdd9457cb5c481c3f89aa349",
            "input": "up to ## afghans have been killed and hundreds injured by a massive explosion at an ammunition depot in the eastern provincial capital jalalabad , kabul red cross officials said thursday .",
            "output": [
                "mobutu notes parliament 's decision to sack government"
            ]
        },
        {
            "id": "task1579-d4478904d95e41d699d0a984530c1fd9",
            "input": "the independent political groups known as ###s should be required to disclose their funding sources while election campaigns are in progress , three republican house members said wednesday .",
            "output": [
                "vanity of UNK : the conductor as composer as entrepreneur"
            ]
        },

        {
            "id": "task1579-60c744c284634dc39cc668edd6e1237b",
            "input": "an e-mail sent to paul harris , a member of the virginia house of delegates , elicits an automatic response : `` i will be out of the office from UNK until UNK .",
            "output": [
                "national semiconductor 's loss is smaller than expected"
            ]
        },

Overall having weird artifacts in inputs seems ok, but having them in outputs seems not: you don't want a model to learn this and start inserting spaces between apostrophes in answers.