guytevet / diversity-eval Goto Github PK

I am trying to find these two datasets if they are still available.
con_test_200_with_hds_resp_gen.csv
dec_test_200_with_hds_resp_gen.csv

When attempting to use run_metrics.py, the program would crash due to non-unicode characters in multiple files in .\data\raw\.

For example:
file: dec_test_1000_no_hds_prompt_gen.csv we have:
sample_id: test.enc.prompt.txt::output.cmdc.uni.sample10.temp0.89::00121
resp_0: little've?" ―Mendez and Puni [src] K!

On Windows 10, I get the following error:
UnicodeEncodeError: 'charmap' codec can't encode character '\u2015' in position 98: character maps to <undefined> csv DictWriter

Potential Solution (the one I've been using to work around this issue):

Change this line from:

output_csv_f = open(param_dict['out_path'], 'w')

to:

output_csv_f = open(param_dict['out_path'], 'w', encoding='utf-8')

Another option is to resave the data files with UTF-8 encoding by default.

By the way, thanks for the interesting paper and making this code available for the research community.