clarin-pl / embeddings Goto Github PK

• https://huggingface.co/clarin-pl

• dataset, input_column_name, target_column_name, type_of_dataset [tagging, classification] columns in the table

• general readme

Update README:

• Add example with NER/POS
• Add examples for use with predefined pipelines not only custom one

• table in readme

https://huggingface.co/docs/datasets/package_reference/main_classes.html#datasets.DatasetInfo

• main to MT

• how do we want to load data from local drives
• do we want to prepare special loaders for data - tensorflow datasets, spacy datasets

https://huggingface.co/sentence-transformers/average_word_embeddings_glove.6B.300d/tree/main check how to create similar for polish embeddings and how they word with our pipelines

TODO List:

Hypersearch Pipeline:

• Add other pipelines
• Add tests

Other:

• #91
• #95
  • Sequence Labelling
  • Text Classification
  • Pair Text Classification
• Update README

Done:

• #97
• Add suport for custom optuna search space configuration
• Add retraining task after selection
• Add optuna to pyproject.toml
• Add logging of best found parameters configuration
• @ktagowski Create Pipeline for Hyperameter search
• @djaniak Decide which library to use (optuna/hyperopt/ray):
• #90
• Add option to save optuna dataframe log
• @djaniak Run libraries examples
• @ktagowski Prepare proof-of-concept (notebook)
• Add persister for best found parameters

Abandoned
- [ ] Add support for parsing Pipelines parameters from Yaml

• Library is missing description on Pypi
• Fill community profile checklist (https://github.com/CLARIN-PL/embeddings/community)

Part I: KPWr

• Add NER dataset to HF
• Add NER dataset to library
• Adapt appropriate configuration for NER tag evaluation
• Add NER sequence tagging test to pipeline
• Add NER example

Part II: Poleval 2018 (Nested ner tags)

• Add Poleval NER dataset to HF
• Add Nested NER dataset to library
• Adapt appropriate configuration for Nested NER tag evaluation
• Add Nested NER example

TODO:

• Add possibility to ignore test subset
• Create dev if needed
• Add possibility for evaluation on dev instead test
• Make persisting of model results optional

• readme
• HF

• Rename poorly named methods e.g. Model -> model()
• Rename poorly named parameters / variables

(??: https://huggingface.co/hf-internal-testing)

• Check whether metrics are calculated correctly for POS tags
• Check whether tags names are correctly parsed via seqeval library and add this case for tests
  - [ ] Add NER dataset for sequence tagging

Update HuggingFaceClassificationDataset:

• Change paths to pathlib
• Auto-detect splits in dataset

https://huggingface.co/datasets/clarin-pl/cst-wikinews

• add flair pair classification model

As yet, the dev and test data is automatically sampled in CorpusTransformation if it is set to None. This is because flair Corpus has a sample_missing_splits argument which is set to true by default. We may want to change that behavior and do not automatically sample dev and test data.

==============================herbert-base=================================

2021-08-09 23:57:17,425 loading file herbert-base/best-model.pt
2021-08-09 23:57:31,239     0.728
2021-08-09 23:57:31,239 
Results:
- F-score (micro) 0.728
- F-score (macro) 0.6648
- Accuracy 0.728

By class:
              precision    recall  f1-score   support

       minus     0.7716    0.8171    0.7937       339
        plus     0.6137    0.8678    0.7190       227
        zero     0.9813    0.8898    0.9333       118
         amb     0.5455    0.1324    0.2130       136

   micro avg     0.7280    0.7280    0.7280       820
   macro avg     0.7280    0.6768    0.6648       820
weighted avg     0.7206    0.7280    0.6968       820
 samples avg     0.7280    0.7280    0.7280       820


==============================herbert-kgr10=================================

2021-08-10 00:03:07,082 loading file herbert-kgr10/best-model.pt
2021-08-10 00:03:34,942         0.7671
2021-08-10 00:03:34,943
Results:
- F-score (micro) 0.7671
- F-score (macro) 0.6484
- Accuracy 0.7671

By class:
              precision    recall  f1-score   support

       minus     0.7429    0.9292    0.8257       339
        plus     0.7143    0.9031    0.7977       227
        zero     1.0000    0.9153    0.9558       118
         amb     1.0000    0.0074    0.0146       136

   micro avg     0.7671    0.7671    0.7671       820
   macro avg     0.8643    0.6887    0.6484       820
weighted avg     0.8146    0.7671    0.7021       820
 samples avg     0.7671    0.7671    0.7671       820

tested based on clarin-pl/polemo2-official

Labeling is the American spelling, labelling is the British spelling.

In our code and filenames, we use both spellings interchangeably.
Sequence labeling is more common in the literature so we could use this one.

• propose how it could look alike with examples
• add example sample_x.json

HuggingFace

• Add versioning, plWordNet may have different version, add metadata
  - [ ] How to add graph-data to HF (seperate dataset, or merged with WSD dataset)

TODO (only Text-based pipeline):

• Add WSD dataset to HF
• Add WSD dataset to library
• Add WSD to pipeline
• Add WSD example

Graph (HF)

• Test HF proof of concept.:
  • Feeded texts with ID and TEXT
  • Graph data as a HF dataset attribute

Data in conll format is saved to a file with .csv extension while it probably should be saved to .tsv file as it is tab separated data.

column_format = self._save_to_conll( hf_datadict[subset_name], output_path.joinpath(f"{subset_name}.csv") )

• readme
• HF
• paper