dsfsi / gov-za-multilingual Goto Github PK

We are still having problems with Sentence alignment failures? Are we timing out?

so csv file for each language, use the language codes

e.g. govza-cabinet-statements-en.csv for english

From the output of the action:

I have a hunch in a recent update of panda's they deprecated the append function.
I think it must be pd.concat() now.

Use this as an example https://github.com/dsfsi/za-isizulu-siswati-news-2022

https://sites.google.com/uw.edu/data-statements-for-nlp/sample-data-statements?authuser=0

Description: Github action Fetch cabinet statement failing

Traceback (most recent call last):
  File "main.py", line 6, in <module>
    cabinent_statements = helper_functions.get_cabinent_statements_urls(date) #read cabinet statements until date
  File "/home/runner/work/gov-za-multilingual/gov-za-multilingual/src/gov_cab_statements_scrape/helper_functions.py", line 60, in get_cabinent_statements_urls
    statement['title'] = row.find('td').find('a').text
AttributeError: 'int' object has no attribute 'find

Looks like a simple fix, maybe they changed something in the website structure

https://github.com/dsfsi/gov-za-multilingual/blob/master/notebooks/cabinet_speech_script.ipynb

• Load the already saved JSON file
• Only get speeches that are newer than the youngest from the JSON file
• Update json with new speeches
• Save/Commit

Notes: Use of github actions+ jupyter notebooks (example from covid19za - https://github.com/dsfsi/covid19za/blob/master/.github/workflows/rtlive-model-za.yml )

Pandas keeps updating their functions! >:

Description

Use LASER encodings to perform Sentence alignment on the govza-cabinet-statements.json and output in the data/processed folder.

Tasks

• Download LASER models
• Data cleanup & sentence tokenisation
• Script to align sentences for 2 languages outputting to a csv
• (Additional) Script to align sentences for all languages
• Github action to create new aligned csvs on existing update action commit