Missing log message for failed COVIDcast CSV files about delphi-epidata

[krivard]

The file in question:
20200828_state_deaths_7dav_cumulative_prop.csv.zip

[benjaminysmith]

I can take a look at this -- can you point me at the code where this error occurs?

[dfarrow0]

here's where the correct handling example is logged:

delphi-epidata/src/acquisition/covidcast/csv_importer.py

Line 302 in 801f272

print(' invalid value for %s (%s)' % (str(row), error))

it's unexpected that a file would be archived as failed without any corresponding log message describing the failure reason.

[benjaminysmith]

Thanks David -- that looks like enough to start debugging the issue. To understand more broadly, where does this code get called from? In one of the automation steps?

[dfarrow0]

yep, that's right. here's how it happens:

• task#33 runs step#63 every hour
• step#63 causes the steps in flow#7 to run sequentially
• the first step in flow#7 is step#60, which runs the python file csv_to_database.py
• csv_to_database imports and calls CsvImporter from file csv_importer.py

[benjaminysmith]

Still debugging, adding some findings:

1. The rows seem to parse fine (i.e., cc_rows = CovidcastRow.fromCsvRows(csv_rows, source, signal, time_type, geo_type, time_value, issue, lag, is_wip) returns a list of rows from the file:

ak 4.7801364393158705 None None
al 40.11095411888978 None None
ar 23.64538679687046 None None
az 66.50701137105816 None None
...

2. The version of the code that executed is prior to this patch, which is why the error message shows the deaths_7dav_cumulative_prop False line -- this is the print(signal, is_wip) from line 80.

[benjaminysmith]

I suspect that the root cause of the problem may be that there is an infinite value in the data: pr inf None None -- my best guess right now is that this causes the mysql insert to fail, causing a return value of 0 rows. In the case where 0 is returned this falls through to archive_as_failed which would match the error messages in this issue.

[krivard]

Fixed in PR.