Final test corpus which consists of all manually evaluated corpora is: manual-annotations/multilingual-genre-annotated-test-set.json
It is a JSON file with the following languages as keys. Each language item then consists of the following values:
- accuracy, micro_f1, macro_f1: the evaluation results in accuracy, micro F1 and macro F1 metric
- label_scores: per label F1 scores
- dataset: the test dataset. The dataset can be opened as a pandas dataframe: pd.DataFrame(json_dict["lang"]["dataset"])
. It includes:
- the automatically predicted labels (y_pred),
- the manually-evaluated labels (y_true),
- text,
- translation (the translation to English which was used for manual evaluation),
- text_id (the same as in the original MaCoCu or CLASSLA-web corpora)
- metadata
- list of tokens (tokens) and their token ids (token_ids)
- token_overlap: includes the following keys:
- overlap_percentage: percentage of overlap with the train dataset (X-GENRE-train)
- token_list: list of all tokens from the dataset
- overlap_token_list: list of all tokens that appear both in train dataset and test set
- label_level_token_lists: list of all tokens from the dataset for each of the labels
For automatic annotation, we only annotate texts that are longer (or the same size) than 75 words. Furthermore, due to model's limitations, we only take the first 512 words of each text (because the max_sequence_length that model can take is 512 tokens).
See the spreadsheet CLASSLA genres for all sizes and genre distributions.
We use the code 1-select_for_xgenre.py
and 2-predict_extended.py
in /cache/tajak/macocu-mt/
. The automatically annotated datasets are saved in /cache/tajak/macocu-mt/datasets/annotated
.
See the code in copied-code-from-cache
and the README.md file in the same folder for all documentation on how the corpora were annotated and samples were prepared. After annotation the following steps were done:
- Evaluate the sample after annotation:
/home/tajak/Crosslingual-Genre-Bias-Analysis/evaluation-of-annotation.ipynb
- If needed, annotate additional instances: see section
Add additional instances to the sample to achieve 10 instances per label
in/cache/tajak/macocu-mt/analyze-entire-file-prepare-sample.ipynb
to prepare additional instances; and sectionAdd additionally annotated instances
in/home/tajak/Crosslingual-Genre-Bias-Analysis/evaluation-of-annotation.ipynb
to merge them with initial sample and get final evaluations.
Output:
manual-annotations/MaCoCu-{lang}-genre-sample-evaluated-complete-sample.jsonl
- manually evaluated samples (orMaCoCu-{lang}-genre-sample-evaluated-complete-sample-run2.jsonl
in case there were two rounds of adding additional instances)- initial samples + additional instances (so that each label was evaluated on 10 instances); Multiple texts and Other texts are marked, but included in the sample - use the code in/home/tajak/Crosslingual-Genre-Bias-Analysis/evaluation-of-annotation.ipynb
to discard them and evaluate them.
The genre samples are prepared in such way that we take 10 random instances of each genre from the entire corpus. We do not include the genres "Other" and "Mix", as they are not informative - the final sample size is 80 instances. If the manual evalutation shows that some instances had to be discarded, because they were "Multiple texts", "Incomprehensible" or would belong to the category "Other", we annotated additional instances so that each label was annotated on 10 instances.
Then we translate the source text in the sample to English using Google Translate. Google Translate provides models for all of our languages, except for Montenegrin, for which we used the Serbian MT model.
Dataset | Macro F1 | Micro F1 |
---|---|---|
MaCoCu-uk | 0.948 | 0.950 |
CLASSLA.web-sl | 0.936 | 0.938 |
CLASSLA.web-mk | 0.932 | 0.925 |
MaCoCu-tr | 0.899 | 0.9 |
CLASSLA.web-hr | 0.883 | 0.887 |
MaCoCu-sq | 0.854 | 0.850 |
MaCoCu-el | 0.844 | 0.850 |
MaCoCu-ca | 0.827 | 0.825 |
MaCoCu-is | 0.795 | 0.8 |
MaCoCu-mt | 0.488 | 0.557 |
mt | el | tr | sq | is | uk | ca | mk | hr | sl | average | average-wth-mt | |
---|---|---|---|---|---|---|---|---|---|---|---|---|
News | 0.69 | 0.9 | 0.95 | 0.89 | 0.73 | 1 | 0.82 | 0.91 | 0.95 | 0.9 | 0.87 | 0.89 |
Opinion/Argumentation | 0.33 | 0.87 | 0.82 | 0.67 | 0.82 | 0.91 | 0.84 | 0.78 | 0.78 | 0.82 | 0.76 | 0.81 |
Instruction | 0.69 | 0.71 | 0.9 | 0.95 | 0.78 | 0.95 | 0.75 | 1 | 0.75 | 1 | 0.85 | 0.86 |
Information/Explanation | 0.52 | 0.7 | 0.82 | 0.8 | 0.5 | 1 | 0.72 | 0.84 | 0.95 | 0.9 | 0.78 | 0.8 |
Promotion | 0.82 | 0.62 | 0.86 | 0.95 | 0.75 | 0.78 | 0.78 | 0.95 | 0.87 | 1 | 0.84 | 0.84 |
Forum | 0.18 | 0.95 | 0.89 | 0.78 | 0.95 | 0.95 | 0.84 | 1 | 0.91 | 1 | 0.84 | 0.91 |
Prose/Lyrical | 0.18 | 1 | 0.95 | 0.86 | 1 | 1 | 0.91 | 0.95 | 0.86 | 0.91 | 0.86 | 0.93 |
Legal | nan | 1 | 1 | 0.95 | 0.84 | 1 | 0.95 | 0.95 | 1 | 0.95 | 0.96 | 0.96 |
Macro F1 | 0.49 | 0.84 | 0.9 | 0.85 | 0.8 | 0.95 | 0.83 | 0.92 | 0.88 | 0.94 | 0.84 | 0.87 |
Macro f1: 0.936, Micro f1: 0.938, Accuracy: 0.938
precision | recall | f1-score | support | |
---|---|---|---|---|
Forum | 1 | 1 | 1 | 10 |
Information/Explanation | 0.9 | 0.9 | 0.9 | 10 |
Instruction | 1 | 1 | 1 | 10 |
Legal | 1 | 0.909091 | 0.952381 | 11 |
News | 0.9 | 0.9 | 0.9 | 10 |
Opinion/Argumentation | 0.7 | 1 | 0.823529 | 7 |
Promotion | 1 | 1 | 1 | 10 |
Prose/Lyrical | 1 | 0.833333 | 0.909091 | 12 |
Macro f1: 0.883, Micro f1: 0.887, Accuracy: 0.887
precision | recall | f1-score | support | |
---|---|---|---|---|
Forum | 1 | 0.833333 | 0.909091 | 12 |
Information/Explanation | 1 | 0.909091 | 0.952381 | 11 |
Instruction | 0.6 | 1 | 0.75 | 6 |
Legal | 1 | 1 | 1 | 10 |
News | 0.9 | 1 | 0.947368 | 9 |
Opinion/Argumentation | 0.7 | 0.875 | 0.777778 | 8 |
Promotion | 1 | 0.769231 | 0.869565 | 13 |
Prose/Lyrical | 0.9 | 0.818182 | 0.857143 | 11 |
Macro f1: 0.923, Micro f1: 0.925, Accuracy: 0.925
precision | recall | f1-score | support | |
---|---|---|---|---|
Forum | 1 | 1 | 1 | 10 |
Information/Explanation | 0.8 | 0.888889 | 0.842105 | 9 |
Instruction | 1 | 1 | 1 | 10 |
Legal | 0.9 | 1 | 0.947368 | 9 |
News | 1 | 0.833333 | 0.909091 | 12 |
Opinion/Argumentation | 0.7 | 0.875 | 0.777778 | 8 |
Promotion | 1 | 0.909091 | 0.952381 | 11 |
Prose/Lyrical | 1 | 0.909091 | 0.952381 | 11 |
Macro f1: 0.488, Micro f1: 0.557 Accuracy: 0.557
precision | recall | f1-score | support | |
---|---|---|---|---|
Forum | 0.1 | 1 | 0.181818 | 1 |
Information/Explanation | 0.6 | 0.461538 | 0.521739 | 13 |
Instruction | 1 | 0.526316 | 0.689655 | 19 |
News | 0.9 | 0.5625 | 0.692308 | 16 |
Opinion/Argumentation | 0.3 | 0.375 | 0.333333 | 8 |
Promotion | 0.9 | 0.75 | 0.818182 | 12 |
Prose/Lyrical | 0.1 | 1 | 0.181818 | 1 |
Macro f1: 0.854, Micro f1: 0.85 Accuracy: 0.85
precision | recall | f1-score | support | |
---|---|---|---|---|
Forum | 0.7 | 0.875 | 0.777778 | 8 |
Information/Explanation | 1 | 0.666667 | 0.8 | 15 |
Instruction | 0.9 | 1 | 0.947368 | 9 |
Legal | 0.9 | 1 | 0.947368 | 9 |
News | 0.8 | 1 | 0.888889 | 8 |
Opinion/Argumentation | 0.7 | 0.636364 | 0.666667 | 11 |
Promotion | 0.9 | 1 | 0.947368 | 9 |
Prose/Lyrical | 0.9 | 0.818182 | 0.857143 | 11 |
Macro f1: 0.795, Micro f1: 0.8 Accuracy: 0.8
precision | recall | f1-score | support | |
---|---|---|---|---|
Forum | 0.9 | 1 | 0.947368 | 9 |
Information/Explanation | 0.4 | 0.666667 | 0.5 | 6 |
Instruction | 0.7 | 0.875 | 0.777778 | 8 |
Legal | 0.8 | 0.888889 | 0.842105 | 9 |
News | 0.8 | 0.666667 | 0.727273 | 12 |
Opinion/Argumentation | 0.9 | 0.75 | 0.818182 | 12 |
Promotion | 0.9 | 0.642857 | 0.75 | 14 |
Prose/Lyrical | 1 | 1 | 1 | 10 |
Macro f1: 0.844, Micro f1: 0.85, Accuracy: 0.85
precision | recall | f1-score | support | |
---|---|---|---|---|
Forum | 1 | 0.909091 | 0.952381 | 11 |
Information/Explanation | 0.8 | 0.615385 | 0.695652 | 13 |
Instruction | 0.6 | 0.857143 | 0.705882 | 7 |
Legal | 1 | 1 | 1 | 10 |
News | 0.9 | 0.9 | 0.9 | 10 |
Opinion/Argumentation | 1 | 0.769231 | 0.869565 | 13 |
Promotion | 0.5 | 0.833333 | 0.625 | 6 |
Prose/Lyrical | 1 | 1 | 1 | 10 |
Macro f1: 0.827, Micro f1: 0.825 Accuracy: 0.825
precision | recall | f1-score | support | |
---|---|---|---|---|
Forum | 0.8 | 0.888889 | 0.842105 | 9 |
Information/Explanation | 0.9 | 0.6 | 0.72 | 15 |
Instruction | 0.6 | 1 | 0.75 | 6 |
Legal | 0.9 | 1 | 0.947368 | 9 |
News | 0.7 | 1 | 0.823529 | 7 |
Opinion/Argumentation | 0.8 | 0.888889 | 0.842105 | 9 |
Promotion | 0.9 | 0.692308 | 0.782609 | 13 |
Prose/Lyrical | 1 | 0.833333 | 0.909091 | 12 |
Macro f1: 0.899, Micro f1: 0.9 Accuracy: 0.9
precision | recall | f1-score | support | |
---|---|---|---|---|
Forum | 0.8 | 1 | 0.888889 | 8 |
Information/Explanation | 0.7 | 1 | 0.823529 | 7 |
Instruction | 0.9 | 0.9 | 0.9 | 10 |
Legal | 1 | 1 | 1 | 10 |
News | 1 | 0.909091 | 0.952381 | 11 |
Opinion/Argumentation | 0.9 | 0.75 | 0.818182 | 12 |
Promotion | 0.9 | 0.818182 | 0.857143 | 11 |
Prose/Lyrical | 1 | 0.909091 | 0.952381 | 11 |
Macro f1: 0.948, Micro f1: 0.95 Accuracy: 0.95
precision | recall | f1-score | support | |
---|---|---|---|---|
Forum | 0.9 | 1 | 0.947368 | 9 |
Information/Explanation | 1 | 1 | 1 | 10 |
Instruction | 1 | 0.909091 | 0.952381 | 11 |
Legal | 1 | 1 | 1 | 10 |
News | 1 | 1 | 1 | 10 |
Opinion/Argumentation | 1 | 0.833333 | 0.909091 | 12 |
Promotion | 0.7 | 0.875 | 0.777778 | 8 |
Prose/Lyrical | 1 | 1 | 1 | 10 |
For the sample, I randomly sampled 10 instances of each of the genre classes from the first 100.000 texts in the corpora -> 90 instances per corpus. I included "Other" as the label in the sample. However, as this label is mostly used so that the classifier can use it for harder examples, when doing manual annotation, I tried to identify the actual label of these texts, so most of texts, labelled Other, were manually annotated as something else.
I evaluated three corpora: CLASSLA-web.sl, CLASSLA-web.hr and CLASSLA-web.mk. After the two rounds of evaluation of these three corpora, I also evaluated the Albanian corpus: MaCoCu-sq.
The sample was prepared in the same way, except for the fact that we randomly sampled the texts from the entire corpus (not from the first 100.000 as in the first batch). Second difference is that we did not include the "Other" label in the sample, because this is used as a "throw-away" category to be used when the classifier doesn't recognize the text to be of any other, more concrete genres.
This is the distribution after the additionally evaluated instances were added.
For calculating the metrics of classifier's performance, I remove "Other" texts and "Multiple texts" (also "Incomprehensible" in case of Albanian) texts from the sample. Thus, we compare only the predictions of 8 labels, each having 10 instances.
Initial distribution (before post-processing) - OLD RESULTS (before cleaning non-target languages):
MaCoCu-mt:
y_true | count |
---|---|
Instruction | 19 |
News | 16 |
Information/Explanation | 13 |
Promotion | 12 |
Legal | 10 |
Opinion/Argumentation | 8 |
Multiple texts (3%) | 2 |
Forum | 1 |
Prose/Lyrical | 1 |
MaCoCu-el:
y_true | count |
---|---|
Opinion/Argumentation | 13 |
Information/Explanation | 13 |
Forum | 11 |
Prose/Lyrical | 10 |
Legal | 10 |
News | 10 |
Instruction | 7 |
Promotion | 6 |
Multiple texts (7%) | 6 |
MaCoCu-tr:
y_true | count |
---|---|
Opinion/Argumentation | 12 |
News | 11 |
Prose/Lyrical | 11 |
Promotion | 11 |
Instruction | 10 |
Legal | 10 |
Forum | 8 |
Information/Explanation | 7 |
Multiple texts (4.7%) | 4 |
Other (1.17%) | 1 |
MaCoCu-sq:
y_true | count |
---|---|
Information/Explanation | 17 |
Opinion/Argumentation | 12 |
Forum | 12 |
Prose/Lyrical | 12 |
Legal | 9 |
Promotion | 9 |
Instruction | 9 |
News | 8 |
Other (4.12%) | 4 |
Multiple texts (3.1%) | 3 |
Incomprehensible (2.1%) | 2 |
MaCoCu-is:
y_true | count |
---|---|
Promotion | 13 |
Opinion/Argumentation | 12 |
News | 12 |
Prose/Lyrical | 10 |
Legal | 9 |
Forum | 9 |
Instruction | 8 |
Information/Explanation | 7 |
Multiple texts (7.95%) | 7 |
Incomprehensible (1.14%) | 1 |
MaCoCu-ca:
y_true | count |
---|---|
Information/Explanation | 15 |
Promotion | 13 |
Prose/Lyrical | 12 |
Forum | 9 |
Legal | 9 |
Opinion/Argumentation | 9 |
News | 7 |
Instruction | 6 |
Multiple texts (3.5%) | 3 |
Incomprehensible (2.35%) | 2 |
MaCoCu-uk:
y_true | count |
---|---|
Opinion/Argumentation | 12 |
Instruction | 11 |
Prose/Lyrical | 10 |
Legal | 10 |
News | 10 |
Information/Explanation | 10 |
Forum | 9 |
Promotion | 8 |
Multiple texts (9%) | 8 |
In CLASSLA-web corpora, we initially also annotated the "Other" labels, that is why they are present here more than in other corpora.
CLASSLA-mk:
y_true | count |
---|---|
Promotion | 13 |
News | 12 |
Prose/Lyrical | 11 |
Opinion/Argumentation | 11 |
Information/Explanation | 11 |
Forum | 10 |
Instruction | 10 |
Legal | 9 |
Multiple texts (8.3%) | 8 |
Other (1%) | 1 |
CLASSLA-hr:
y_true | count |
---|---|
Promotion | 16 |
Prose/Lyrical | 12 |
Forum | 12 |
Information/Explanation | 11 |
Legal | 10 |
News | 9 |
Opinion/Argumentation | 8 |
Multiple texts (7.2%) | 7 |
Other (6.19%) | 6 |
Instruction | 6 |
y_true | count |
---|---|
Promotion | 13 |
Prose/Lyrical | 13 |
Legal | 11 |
Information/Explanation | 11 |
News | 10 |
Instruction | 10 |
Forum | 10 |
Multiple texts (8.25%) | 8 |
Opinion/Argumentation | 8 |
Other (3.1%) | 3 |
Number of texts, annotated as problematic ("multiple texts") - mostly, they were not a coherent text (just a list of summaries, multiple texts concatenated):
- Slovenian, Croatian, Macedonian, Icelandic, Greek, Ukrainian: 6-9%
- Albanian, Maltese, Turkish, Catalan: 3-5% - there were less problematic texts. However, in Albanian sample, there were also some incomprehensible texts - probably due to bad machine translation - 2% of texts.
I re-did the evaluation on the same dataset, but after some improvements: the text was clearly separated into paragraphs -> texts are much more comprehensible and easier to understand; I used the doccano annotation platform -> easier and (slightly) faster annotation.
When I compared my annotations from the two runs, there were disagreements between the labels in 40 instances - 15% of instances.
The reasons for the differences:
reason | frequency (# instances) |
---|---|
improved second run (text annotated with a label, instead of "problematic") | 8 |
detected "multiple texts" that I didn't in the first run | 9 |
inter-annotator disagreement | 23 |
So we can see that the improved methodology (better shown texts, using doccano and paragraph structure) improved annotation of 17 cases - 6% of all texts (either we were able to annotate texts that were previously incomprehensible, or we detected problematic texts that we previously didn't due to the lacking text structure).
In 23 instances (9%), there was inter-annotator disagreement, which shows the level of difficulty of this task. However, 8 of these cases appeared when annotating texts that were predicted as "Other", which we decided that we won't include in the annotation campaign anyway. If we disregard these cases, disagreement happened in 15 instances - 6% of texts.