State name change in RKI truth data about covid19-forecast-hub-de

[seabbs]

Thanks @jbracher 👍

Yes it does (in that it assumes the character string uniquely map to locations which they currently do not). We had previously reviewed your code processing and so knew that the strings were coming from a data dictionary (this lets us graph based on place name more easily) rather than maintaining our own data dictionary.

We will refactor our code to not rely on place name remaining stable for the future (i.e use our own data linkage). This might mean our model based submission is late or incomplete this week.

[dwolffram]

Hi, I fixed the files - they all use " Free State of Thuringia" now (839cb94).

[seabbs]

The issue is not in the raw RKI data and appears to be driven by this commit ef0643f which would need to be applied retroactively to all processed data to fix this issue (or another alternative like a note saying names have been updated and no longer uniquely map to FIPs codes).

[seabbs]

Looking at the processing code (

covid19-forecast-hub-de/code/auto_download/rki_update.py

Line 64 in e2a585c

fips_english = pd.read_csv('../../template/base_germany.csv')

) the issue is due to a concatenation of old and new data.

If I were fixing I might refactor so that the processed snapshots stay as truth but the overall "truth" csvs are compiled freshly from the daily snapshots on each update rather than concatenated as this would allow you to change names etc without causing downstream errors or changing truth data.

[jbracher]

Hi Sam, thanks for pointing this out. When I merged the PR in ef0643f I had misunderstood what was being done. Just to be sure: Does this break anything in your codes? We recommend to always use location rather than location_name precisely because there are so many different ways of writing the German names out in English.
We will make this uniform in any case (likely without the umlaut as that seems more correct to me). @dwolffram wrote the respective codes and will take a look.

[jbracher]

Okay, thanks. We'd prefer late to incomplete ;-) @dwolffram will fix the truth data set so there will again be a one to one correspondence. So then everything should work again in your codes, but maybe still better to get rid of this problem. On a side note: The location_name column is optional, so you're not even obliged to have it at all.