luceydav / ct_state_open_data

Load, clean and explore available open data

ct_state_open_data

Despite its successful economic history, Connecticut's fortunes have fallen to rank among the the most financially vulnerable. The reasons for this are complex, and offer fodder for both sides of the political spectrum. In 2014, Connecticut passed open data laws and have proceeded to make much public data available. Datasets are disclosed in arbitrary periods, the same dataset often has different fields from one disclosure to the next, different locations have conflicting data and some data which is disclosed subsequently disappears.

Although Connecticut has a highly educated population and success in financial services, manufacturing and home to one of the great research universities, it ranked #32 for data science community http://www2.datainnovation.org/2017-best-states-data.pdf. There is a sense that the legacy of other industries may have crowded out the acquistion of these new skills over the last ten years. This repository was built by a former financial services professional in hopes of beginning to repay the great debt to the open source community over the last few years. It is hoped that collaboration takes hold, that this dataset will grow and improve, leading to insight and greater understanding of the State's challenges and possible solutions.

Though the Data Innovation report linked above ranks Connecticut pretty highly, there are still many challenges to gather and clean data in a form which can be used for analysis. This repository is a first attempt to make the following cleaned up datasets available in R data.tables for the data science community.

The following datasets are currently available:

1. All payments by the state from 2010-2017

2. All payroll from 2010-2018 since 2010 with 5 fields. Fringe benefits was included in this data, but was recently removed. It may be added in a future update of this repository.

3. A second conflicting payroll dataset 2015-2018 including fringe benefits and 38 fields

4. Federal Grant data 2012-2019

Data containing all pension payments from 2010-2017 was previously disclosed but has subsequently been removed. This may be included in a future update of this repository.

Other data which will be included is the Municipal Fiscal Indicators reports for all 169 towns since 2008, IRS federal tax paid by zip code including income ranges 2009-2016, and state sales, real estate conveyance and income tax by town over the last five years.

Please be respectful of the costs of downloading data to the states. Choose how many years are needed, and use the save.image to store an .Rdata file at the end and load when resuming instead of repeatedly loading each time.