cityofaustin / atd-data-publishing

Python scripts for Austin Transportation's ETL tasks

transportation-data-publishing

This repo houses ETL scripts for Austin Transportation's data integration projects. They're written in Python.

Quick Start

1. Clone this repository to your host: git clone https://github.com/cityofaustin/transportation-data-publishing

2. Create your secrets.py and drop it into transportation-data-publishing/config following the template in fake-secrets.py

3. If setting up ESB inegration add certificates to transportation-data-publishing/config/esb

4. Run scripts as needed, or deploy to a Docker host with transportation-data-deploy

About the Repo Structure

bcycle

These scripts load B-Cycle tripe data from an Austin B-Cycle Dropbox folder to data.austintexas.gov.

config

Config holds configuration files needed for the various scripts. secrets.py belongs here -- see fake_secrets.py as a reference.

data_tracker

These scripts modify data in our Data Tracker application, and support its integration with other applications.

open_data

These scripts publish transportation data to data.austintexas.gov and the City's ArcGIS Online organization site.

traffic_study

These are the dedicated files for publishing traffic study data, as described in the wiki.

Contributing

Public contributions are welcome! Assign pull requests to @johnclary.

Deployment Pipeline

We are currently using CircleCI to automatically rebuild the container on every update to this repo, and autonomously deploy that image to Docker-hub. When working in the production or master branches, there are some considerations to keep in mind:

• You will need someone to review your changes for Production or Master branches.
• The production branch is tagged as latest in dockerhub, which means it will be the default image and "source of truth".
• Ideally you will want to merge to master, which will cause CircleCI to create a master tag in dockerhub, which can be used for additional testing or troubleshooting.
• Any code merged to production, should be production-worthy already. Test thoroughly in master or any other branch before merging in any additional code.

Development

Feel free to create a new branch, and commit/push as many times as you need. The pipeline will create a new docker image (if it does not exist), or update the existing image in dockerhub. Your branch name will be used as the identifying tag in docker hub.

For example:

Say you create a branch with the name 123-atd-updatedcode, as soon as the branch is created CircleCI will begin building the docker image, and tag it as atddocker/atd-data-publishing:123-atd-updatedcode , then it will upload it to Docker hub. If there is already an image with that tag, it will simply update it.

License

As a work of the City of Austin, this project is in the public domain within the United States.

Additionally, we waive copyright and related rights in the work worldwide through the CC0 1.0 Universal public domain dedication.

atd-data-publishing's Issues

retiming data pusher - check for new data

currently just a complete refresh

need to add modified dates in database

signal status: intersections missing lat/lon

resolve by querying intersection database for lat/lon

signal retiming: signal IDs are floats

https://data.austintexas.gov/resource/efct-8fs9.json

fieldname / schema changes

mirror field names in intersection database, esp atd_intersection_id

signal retiming: ETL is failing

local variable 'new_key' referenced before assignment

intersection database: nightly backup

Radar Count Publisher is Failing

It's failing at this line with the error: '<' not supported between instances of 'str' and 'int'.

Appears to be related to Socrata API changes.

signal status - don't upload data if KITS is stale

duh

Street Segment Updater is Failing to Match Known Segments

There are a number of segments in the Data Tracker that have not been updated by street_seg_updater.py. Hence, they are missing street name attributes from the AGOL layer:
2006688, 2010328, 2012287, 2012828, 2012833, 2012841, 2022803, 2023254, 2031897, 2037765, 2040196, 2041973, 2042707, 5280279, 5418521

I spot checked a few and they do exist in the AGOL layer, so there's something else going on here.

email alerts: attach logfile

reduce github cache time

signal status: previous record id

add previous_record_id field to live and historical signal status datasets

publication log - dataset url

add url to dataset affected dataset

Signal Retiming: system name coming through as system id

https://data.austintexas.gov/dataset/Synchronized-Traffic-Signal-Corridors/efct-8fs9

signal retiming - system name coming through as coded value

Inconsistent code structure under __name__ = __main__

Function calling and argument processing parts after name = main are different from one script to another. This part should only takes care of calling existing functions and catching exceptions. As the result, it would be ideal if they maintain a same structure.

ranking issues

• clear old value from recent nulls
• only rank 'completed'--?
• verify in database that scores will reset to 0 if values are cleared from the evaluation

Update Knack and PostgREST Configs to Point to PostgREST Load Balanacer

With the loss of austintexas.io, we need to point our scripts to our postgREST load balancer instead of transportation-data.austintexas.io.

Affected jobs:

• csr_flex_notes
• traffic_reports
• dockless_pt_poly
• dockless_load_bird
• dockless_load_lyft
• dockless_load_spin
• dockless_load_lime

signal status: records missing from kits

Add logging for Socrata upsert functions

Right now it looks like that UpsertData function fails silently in certain situations. In particular, there was one such case when Socrata returned the following error message for a manual upsert command for a single row:

{'message': 'Cannot update a synced dataset manually'}

but a script using UpsertData continued to run without giving any indication that the data wasn't being added to the data set.

dataset metadata

• signalized intersection status
• publishing logs

Auto-Assign Asset Records to 311 SRs

When an SR includes a lat/lon an asset type, automatically fetch and assign the nearest asset record. TMC users and on-call technicians will still need to review the asset attachment, but this should significantly reduce time spent on this task.

Signal PM > Knack Integration is Failing

Job signal_pms_postgre_to_knack is failing with error: '{"choice_values": ["[email protected]", "[email protected]", "[email protected]"], "other_values": []}' since August 12th. Hmm...

signal status: add status duration to historical dataset

signals_on_flash: location fields

write lat/lon as location in socrata dataset. it's currently leaving the field blank

Street Segment Updater is Failing due to KeyError

device_status_gridsmart is failing due to a keyError related to MODIFIED_DATE.

atd-data-publishing: Update Documentation

There needs to be some brief notes on how the pipeline works and how it builds/deploys images to docker-hub.

GRIDSMART Comm Status Checks Are Failing

We are currently failing to check comm status for GRIDSMART due to some apparent data quality issues. Reached out to Joey and Joshil:

I see a few issues. First, we have duplicate choices/typos in the Gridsmart Status menu:

Adv Replacement
Bad
Cage
Good
Needs Investigating
Needs Investigation
No
RMA
Repairs
Unknown
Unkonwn

Second, what do these status mean? And what to they mean in relation to the detector statuses:

OK
BROKEN
UNKNOWN
REMOVED

Lastly, we need every Gridsmart detector to have Detector Status populated. There are ~20 that do not have Detector Status. Please review and add a status, and/or let's discuss a new if needed. I'm going to make it required now in the data tracker.

Also, we have 14 units with a status of OK that do not have port numbers. This is an issue because we cannot check comm status without port.

Migrated to atd-data-tech #1688

github caching = fail

github cdn seems to cache files for 5-10 minutes. what this means is that, although data may be published at very small intervals (e.g. 1 min) to the open data portal, these only 1 in 10ish of these events will be logged on github, which means the 'last updated' date in our application will not be accurate if it is pointing to the github logs.

options:

1. publish logs on socrata rather than github
2. do a complete refresh of dataset on socrata instead of an upsert
3. leave as is and understand that the 'last updated' date in our application is +/- 10 minutes

create tests

signal status: whacky stale data deltas

very stumped on why stale data alerts are fractions of minutes passed

Sig_stat_pub.py is failing due to missing Socrautil Methods

As noted here. Socratautil is missing needed methods, causing this script to fail.

Once socrautil is patched, we simply need to rebuild the /tdp docker image and deploy.

socrata change detection issues

if a field in the source db is changed to Null, data_helpers.DetectChanges will not detect a change vis a vis the destination db!

rolling logfile truncate

e.g. remove older than 3 months?

signals on flash: rename to signal status

CSR Flex Note Archiver Schedule is Wonky

The cron schedule for the csr_flext_note publisher is running hourly every saturday. it should be running once every saturday.

Socrata Pub Fails with "Illegal field name" Error

We recently started receiving an unfamiliar error message from the publisher API. It is a 400 error with message "Illegal field name sent: [field name]".

We occasionally POST records to the publisher endpoint whose payload may include fields that do not exist in the destination dataset. In the past these fields were ignored, but it seems there may be new validation rules in place.

Affected datasets:

• https://data.austintexas.gov/Transportation-and-Mobility/Traffic-and-Pedestrian-Signal-Requests/f6qu-b7zb
• https://data.austintexas.gov/Transportation-and-Mobility/Traffic-Studies-Locations-BETA-/jqhg-imb3
• https://data.austintexas.gov/Transportation-and-Mobility/Synchronized-Traffic-Signal-Corridors/efct-8fs9
• https://data.austintexas.gov/Transportation-and-Mobility/Austin-B-Cycle-Kiosk-Locations/qd73-bsdg

B-Cycle Trip Data is Failing

B-Cycle Trip publisher is failing with a TypeError and has not been updated since November 2018.

After investigation, it appears that new columns were updated to the November data, and furthermore there are no current trip records. Will reach out to BCycle about this.

signal status - nix database id, plan id

siganl status: dont upload if no data

signals on flash - check for stale statuses in KITS

Socrata Publication is failing with `illegal field name` error.

Originally reported by Kenny M., the cameras_socrata publisher is failing with a 502 error. Investigation needed.

Updated title to reflect impact across datasets.

signal status: query kits for status. source all other data from intersection db

signal status - humanize coded values

Publish Signs Work Orders to ArcGIS Online (AGOL)

AGOL feature service is here.

rename branch?

transportation-data-pushers
transportation-data-publishers
transportation-data-publishing

signal status: create street name columns without prefix/suffix

signal status: separate log entries for 'live' and historical commits

signal retiming: missing locations

there are no lat/lon fields in the sync systems dataset:
https://data.austintexas.gov/resource/efct-8fs9.json

backup data: needs a home

nightly backup is dumping to the KITS app server.

we can move the data:

to a network drive
to another access web app database (hmm)
github (hmm)