Socrata Pub Fails with "Illegal field name" Error about atd-data-publishing

[johnclary]

From Socrata support:

Thanks for sending those! Here's a quick explanation of what's going on: This behavior appears to be an unintended side-effect of some back-end storage configuration updates on your domain (part of an ongoing platform-wide project you can read about here). I sincerely apologize for the disruption to your workflow! The gist of this error is that the platform is no longer ignoring extra columns in your upload - it used to be that you could upsert records with additional columns and they would just be ignored, but now the platform returns an error calling out the extra columns by name. We are working on making this less disruptive to customers, given that it is a change in expected behavior.

I ran these specific datasets over to engineering and they enabled an "ignore nonexisting fields" setting on them, so you should be able to proceed with your existing jobs now. Would you like to retry, and let me know if you still get any errors?

[johnclary]

Thanks @mateoclarke and @dianamartin for your help on this! The latest:

signal_request_evals_weekly

Per suggestion I corrected a lat/lon with whitespace in the Data Tracker. This did not resolve the issue, but need further debugging to see if we're getting a new error message.

timed corridors

Socrata says that the issue is that we're publishing a boolean value to a text field. This was never an issue in the past, so this looks like another breaking API change that we weren't notified about. They suggested that we update the column in the socrata dataset to be boolean instead of text, though I'm worried that might break the dashboard. Holding off for now.

traffic_study_locations

Resolved!

bcycle_kiosks

Resolved!

[johnclary]

I modified the time corridors column as request. Dashboard did not break. That one's done.

R.e. signal_request_evals, here's my latest email to support:

{
"code" : "invalid-request",
"error" : true,
"message" : "Invalid request: Expected number type for column eval_rank, but got value """,
"data" : { }
}

The issue is that we're sending an empty string for null values in this column. This was not an issue before, so I'm assuming this was another new validation that is being enforced?

Can you confirm?

[johnclary]

We have resolved issues with B-Cycle Kiosks, Signal Timing, and Traffic Counts. The Signal Request data is still failing due to Socrata's API change. Will have the patch finished by next week. Lots of code needs to be refactored to fix this API change make sure other datasets don't break.

[johnclary]

Socrata put out some docs on their new field validations:
https://support.socrata.com/hc/en-us/articles/360024040594-Dataset-Validation-for-Column-Data-Type-Values

[johnclary]

Confirmed that this is working and will be resolved with once we deploy new utilities in prod (#191)