codeforkansascity / neighborhood-dashboard

We're working to help neighborhoods help themselves by giving them easy access to useful data to allow them to identify and track problem areas in their neighborhood.

Welcome to the KCNeighborhoodStat app project!

We're working to help neighborhoods help themselves by giving them easy access to useful civic open data. We're ultimately hoping to develop an app that- by providing easier access to civic open data- lets neighborhood organization and neighbors:

• Better understand their neighborhood through data What is my neighborhood- what are its boundaries? How many people live there? How much crime happens here?

• More easily track development and others projects in their neighborhoods Is a new business opening down the street? Is a building being torn down?

• Make better decisions as a community Do we have a lot of senior citizens that need help with minor home repair?

• Become more effective advocates for their community We need more police patrols and codes inspections- here's the data

Civic data: the opportunity

These days, we're seeing a greater availability of civic open data in cities across the country- Kansas City included! We believe this trend towards more open data is especially good news for the neighborhoods that make up the fabric of the city. Rich,up-to-date information about crime, development, and demographics is what neighborhoods need to make better decisions and to advocate for themselves more effectively with government agencies and other organizations.

Civic data: the problem

Unfortunately, even though the data are out there, its potential power to help neighborhoods isn’t yet realized. Accessing civic data might mean hours of wading through spreadsheets, then trying to interpret confusing terminology, or having to go through a website that is not easy to use for the average resident or neighborhood leader. Meanwhile, civic data is usually not presented at the neighborhood-level geography. Right now, for instance, it’s not possible to see the population of a neighborhood or the number of crimes that have occurred there recently.

That’s why we created KCNeighborhoodStat

We’re making an application that displays relevant civic open data at the neighborhood level.

To help realize the full potential of this data in benefitting our city’s neighborhoods, we want this tool to be:

• User-friendly/visually appealing
• Customizable (users can select which datapoints to display for a particular neighborhood)
• Capable of displaying data over time (so that users can understand trends, put data in context)

How it works

Users of the site will be prompted to select a neighborhood on the homepage. This will direct them to a "neighborhood page," where they will be able to select a number of datapoints they would like to study.

Those data would be summarized in simple boxes, like in this example from this [civic data dashboard] (https://dashboard.edmonton.ca/) from the City of Edmonton, Canada’s website.

Another aspect of Edmonton’s dashboard we plan incorporate in our app is the ability to view data over time. By clicking on one of the data boxes, a detailed explanation of the data would expand and the data would be displayed on a graph. The graph could be standalone, allowing multiple datasets to be added to it, to allow comparison.

Ideally, data with a specific location would be displayed on the page’s neighborhood map.

Introduction for Developers

Thank you for assisting with this project! We can always use more coders to assist with what we are doing!

Our Application is built with Ruby on Rails as the backend, and React on the Frontend. In order to run theses, you need to have Ruby, Rails, and NodeJS installed on your device. There articles can assist. (Make sure you use node version 6.2.2)

NodeJS - https://nodejs.org/en/download/package-manager/ Rails - http://installrails.com/

If you are not using windows, I recommend installing the node version manager as well

NVM - https://github.com/creationix/nvm

If you are new to these concepts, these tutorials will be of great assistance.

1. Ruby on Rails Tutorial (Chapters 1-6 should be sufficient)
2. React Tutorial (A few examples should be suffice)

In addition, if you need assistance with git, I highly recommend going through this code school class.

Code School GIT Course

When you have cloned the application, you need to install all of the required rails dependencies to start developing.

bundle install --without production

Once you have checkout out this application, run the following command to setup your Database

bundle exec rake db:create db:migrate db:seed

After your database is ready, you now need to install the node modules.

sudo npm install

From here, what's left is running the following command to compile the react code.

sudo npm run webpack

Finally, run the following command to start the server

rails server

When this command is kicked off, a webpack dev server should be running that continually rebuilds your React code. Essentially, if you change and of the code inside app/js, the application should refresh itself. If you don't see that occur, run the following command to manually recompile the React code.

sudo npm run webpack

That's all you need! Now head to http://localhost:3000 to view the application!

Deployment

The first step is to create a container hosting a postgresql database. Create it, and keep in mind the IP address, port, and login credentials of the databasel.

For the app Docker is used to create an nginx and passenger build for the application. To use this, run the following command to get a secret token.

bundle exec rake secret

From there, run the following command to build the container

docker build . --tag=neighborhood-dashboard-test --build-arg database_name=<database_name> --build-arg database_user=postgres --build-arg database_password=<database_password> --build-arg secret_key_base=<secret_key> --build-arg postgres_port_5432_tcp_addr=<database_addr> --build-arg postgres_port_5432_tcp_port=<database_port|5432>

After that is complete, take the container, and run it on the box with the following command.

docker run -d -p 80:80 <docker_container_id_or_name>

Hosting

We are in the process of gaining control of the http://www.kcneighborhoodstat.org/ domain.

Vacancy Indicator Sources

The Vacancy tab currently includes filters which pull data from a variety of sources to display on the map:

• Land Bank Property with Building: KCMO Land Bank Data dataset where property_condition like 'Structure%'
• Land Bank Property - Vacant Lot, No Building: KCMO Land Bank Data dataset where property_condition like 'Vacant lot or land%'
• Land Bank Property with Demo Needed: KCMO Land Bank Data dataset where demo_needed = 'Y'
• Failure to Register as Vacant: KCMO Property Violations dataset where violation_code = 'NSVACANT'
• Vacant Structure: KCMO 311 Call dataset where request_type in ('Animals / Pets-Rat Treatment-Vacant Home','Animals / Pets-Rat Treatment-Vacant Property','Cleaning vacant Land Trust Lots','Mowing / Weeds-Public Property-City Vacant Lot','Nuisance Violations on Private Property Vacant Structure','Vacant Structure Open to Entry')
• 311 Open Cases: KCMO 311 Call dataset where status='OPEN'
• KCMO Dangerous Buildings: KCMO Dangerous Buildings dataset (no additional criteria)
• Vacant and Boarded: KCMO Property Violations dataset where violation_code = 'NSBOARD01'
• Tax Delinquent: CodeForKC Address API which includes county_delinquent_tax_[year] fields for each year

neighborhood-dashboard's Issues

9/21 Meetup Summary

• Jake still working on building out structure
• Amanda confirmed vacancy data sources with Eric Roche
• Amanda is talking with an attorney with HUD about getting US Postal Service vacancy data.
• HUD contact is also finding out more information on which court systems foreclosures have to go through and if there's a way to get that data
• Amanda and Paul (and 3 others) met with Jackson County last Friday to see what kind of data they have and how we can get it. We would have to pay for staff time to pull the data we want. Paul and Eric are figuring out how we can use the County data that the City already gets so we won't have to pay for data.
  (this summary was submitted by Amanda but is logged in as Ben)

Implement filtering for data ranges for crime data

We need to implement the form that will allow for filtering data by start and end dates

Adding Vacancy Date

For vacant lots and vacant structures, can we list how long it's been vacant in the tooltip?

Code Violation Scoring/Weight

We’re trying to find a way to display the “severity” of code violations – the more severe, the more likely that the neighborhood/City could begin legal proceedings. Peter with Legal Aid went through different types of code violations and scored them, from weakest (1) to strongest (3). He adds, “I do think we need to come up with some sort of formula however. Having ten #1's is obviously just as bad as having one #3. Something to think about I suppose.”

Check out Peter's scoring here: https://codeforkc.slack.com/files/benccf/F0A2AK6G4/code_citation_rankings_lawmo.pdf

Let's think about a scoring formula and discuss soon. Feel free to post your ideas. I will be looking into best practices for this kind of thing over the next few weeks.

@amandaCCF

Setup Application for Offline Development

Right now, we have to hit the mapbox service every time we want to initialize a map for development mode.

In addition, we have to local services to do development with. This makes it impossible to test the application against edge cases that data will give us. We need to be able to fully run the application in an offline environment

Acceptance Criteria:

• Mapbox doesn't rely on HTTPRequests
• All of our services have a mock version available for local development

Add Submit Button to the Neighborhood Search.

Without a submit button on the search form, it's impossible for mobile devices to use the site. We need to add a button so that users can search for their neighborhoods.

Develop an Application Style Guide

In order to keep our Frontend in a manageable and consistent state across all sections of the application, we need to create an application style guide that mounts as an Engine in our application. This will contain any frontend and backend widgets that our app needs to use.

Meetup 8/17

Mockup Updates

Heather updated us on her progress on mockups. Here is the latest general layout of a neighborhood page: https://files.slack.com/files-pri/T0423273A-F097L3VMF/mockups_august_17.jpg. Subtabs are back for now. (We had discussed switching to a "cards" format last week -- the tabs, though, might work better for organizing at least certain data with many subcategories.)

Coding: Crime tab

@JakeLaCombe began working from the mockups, adapting the layout. He also said he should be able to complete the Crime tab by the August 31st Project Fair.

We also got some valuable input from @zachflanders via Slack regarding using bounding boxes to select points. This could have an application for pages like the Crime, when crime reports need to associated with a polygonal geography.

Vacancy tab

We also discussed the potential components of the planned Vacancy tab, the next tab we would like developed. Heather will work those into a mock-up.

To further refine the components, we have asked Peter Hoffman of Legal Aid of Western MO to join us next Monday. He should be able to explain how certain data should be displayed.

Staying in contact going forward

I (@BenCCF) will not be attending Monday-night meetups from now on, on account of the semester starting (specifically an evening class Monday nights). I will still kind of be steering things from afar and will be communicating by Slack and GitHub. Likewise, please communicate with me through those channels. @amandaCCF will still be attending Hack Nights, so if you can still attend, please do so!

(@zmon Still trying to get Heather here on Github. )

Adding Address

Can we add a property's address to the tooltip? That way users are certain what property they're looking at. I attached an example below.

Redesign Application

We have been provided with new mockups with how the application should look. Our app frontend code needs to change to accomodate these new mockups.

Define Dashboard Purpose

Is the dashboard for tracking performance on a specific issue, or is it there to provide general context about the neighborhood?

MVP: Refine the markers for the crime map

For our Crime map, we want to modify the markers as followed.

1. Color code the markers based on their crime category.
2. Stack them in a cylinder fashion, similar to what is found here (http://chriswhong.github.io/nycfelonies/)
3. Add a tooltip description the marker when clicked on.

Redesign Neighborhood Visuals

Neighborhood Visuals - Data for Neighborhood Graphs

MVP: Add a Meta Data for filter information

As a neighborhood coordinator
I would like to see metadata related to the various api's provided by KC socrata sites
So that I can know how relevant the data is, and how I can see the raw data the endpoint provides

We will know when we are done when

• There exists an icon next to all the fileters
• Clicking on the tooltips reveals the url, and the last updated date for the given endpoint.

Consider using sparkle charts in the dashboard

Sparkle charts have good information density for displaying data change over time. These are great for viewing and analyzing multiple trends.

Implement stacked markers

We have datasets, such as code violations, where we could have multiple markers at the same point. To account for this behavior, we decided that the application will incorporate stacked markers in a similar fashion as to what appears on this map.

http://chriswhong.github.io/nycfelonies/

Create a SoQL Query Builder object.

As work has progressed towards adding additional filters for the various data tabs, the code between vacancies and crimes is starting to duplicate. In order to resolve this, we need to create a query builder, that will help with the generation of socrata queries.

Change Neighborhood Search to search by Neighborhood or Address

After discussing Usability at the National Day of Civic Hacking, we determined that the homepage search should be updated to allow users to search by neighborhood or address instead of providing a dropdown of neighborhood to choose from.

Thus, we need to update the search to do this.

Address API: Will it account for changing Census boundaries?

Amanda and I have been discussing displaying certain Census data across time, not just for the 2010 count. We'd want to display 2013 estimates (when available), as well as 2000 counts. Census geography boundaries don't change between 2010 and 2013. However, in many cases, boundaries do change between 2010 and 2000.

Is it feasible to associate addresses with 2000 Census boundaries? What would we need to make that happen?

Add Descriptions of the Crime to the Tooltip

Our Crime Map shows all the crime that occurs in a given neighborhood. However, it's not very descriptive as to what crime actually occurred at the given coordinates. We need to provide a description so that it's clear as to what a map marker indicates.

Acceptance Criteria:

• The Tooltips have a description when you click on them for crime data

Angular version mismatch with angular-route and angular-resource

Angular moduls like angular-route and angular-resource are typically the same version as angular itself in any given project. Is there a particular reason why the version of these two dependencies do not match the angular dependency version? Does anyone have a preference on upgrading to angular 1.4.9?

Vacant Lot Registry Tab Info

Some Data.xlsx

This is excel.doc is just some information I copied and pasted off of the http://webfusion.kcmo.org/coldfusionapps/neighborhood/rentalreg/PropList.cfm site. It's the first 27 rows. (actually the first row got chopped off so it's rows 2-27.)

Add Error Messages for when a Neighborhood isn't found or the dataset errors out

There will be instances when a given neighborhood isn't found in the site. We need to throw toasted messages to the user to show them when this happens.

Create a backend Generic Service Client that can be used to help catch errors

Right now, our backend expects all services to work as is. We need to catch instances of non 200 error codes that are returned from the services we use.

MVP: Add Sex Offenders to the Crime Map

The crime map should contain the ability to add and remove sex offenders from the neighborhood.

What we first need to do is determine what dataset we would like to pull from. data.kcmo.org doesn't appear to have any of that data, so I'm not sure if it's something that can be added there, or if we should look for other possible resources. @zmon

Setup Cron Job task for re populating the neighborhood database

Neighborhood data and boundaries does change, but not constantly. We need to determine the best interval of change to account for, and setup a cron job to reload the database when that happens.

Incorporate 311 Data and Filters into the application

As a Neighborhood Developer
I would like to be able to see 311 open cases and Vacant Structure
So that I can get a better representation of potential vacant structures in a given Neighborhood

We will know when we are done when

• We can see three eleven open cases
• We can look at buildings that are vacant structure

Neighborhood api index response

I have noticed that the API result of neighborhood/index and neighborhood/show are the same when using address "18th and Vine". I am curious if the data response from neighborhood/index is redundant. Is there an instance where the response from neighborhood/index could have any other action than to take the user to a specific neighborhood or to leave the user on the homepage? Is it possible for an address to exist in more than one neighborhood within the dataset?

If the answer to these questions is now, I suggest the neighborhood/index API be modified to return only the ID of the neighborhood or maybe the IDs of the neighborhoods found. This results in less data coming across the pipe and a slightly more performant user experience.

Meetup 7/27: Minutes and Week Ahead

Meetup Minutes

• Clarified basic functionality: Determined how the user would navigate within a neighborhood page, between categories and subcategories. A sketch of this layout is provided below, showing the two levels of tabs. Main category tabs -- at this point Safety, Demographics, Housing, Development, and Vacancy -- would break down into 2-4 "subtabs." In the example below, these subtabs would be Crime and Sex Offenders.

• Went through prototype layouts of each category, pictured below. Vacancy tab has not yet been discussed.

Crime tab Safety tab, actually (Crime would be a "subtab" under Safety.)

Demographics tab

Housing tab

Development tab

To Do - The Week Ahead

Heather:

• Clean up prototype, especially for crime

Jake:

• Begin Safety tab, based on wireframes from Heather

Ben/Amanda (CCF):

• Determine datasets for "Vacancy"
• Update spreadsheet with Property Violation data

Meetup 8/10 Update

We didn't meet at the 8/3 event, for the record.

This week, though, we got mock up for a Safety "tab" on to Jake, and he began building. We were able to find a list of NIBRS codes by "Crime Against" category (https://www.fbi.gov/about-us/cjis/ucr/nibrs/nibrs-user-manual, page 14) -- along with the mock-up, this was probably the biggest
piece of the puzzle needed to get Jake building.

H and @BenCCF discussed mock-ups for other pages/tabs. Besides tweaks to individual pages,

A couple of more major issues/conclusion emerged by evening's end:

• First, while we're still moving ahead with a "tabs" scheme (described in the last meeting's summary), we're considering doing away with the subtabs within each main tab. To illustrate, instead of breaking the main "Demographics" tab into separate subtabs for population, race, income, age, etc., each suptopic would be displayed by default at once as separate "cards". H's mock-ups of a few main tabs suggest we could pull this off while keeping all the data legible.
• Second, the Address API is going to be crucial moving forward. Jake was essentially able to associate the addresses of crime reports to ZIP codes. Getting the addresses linked to neighborhoods will require use of the Address API.

For this next week:

• @JakeLaCombe: Whatever progress on the Crime tab possible. Also, maybe get with Paul re: any work needed on the API.
• Heather: Continued progress on mock-ups
• CCF's end: Prepping to reach out to demographic researcher at UMKC to get some answers on how best to deal with/present some of our data

Consider adding side-by-side comparison of neighborhood stats

Create Dynamic Trending Chart

As a Neighborhood Invester
I want to see a trending chart of various neighborhood datasets
So I can evaluate if a neighborhood has been improving within a timeframe

We will know when we are done when

• We have a trending chart with various datasets.
• We have a legend that adds and removes various datasets to the chart

Implement AngularJS on top of rails

Our app is starting to look better as a single page application. As such, it's highly advised we put angular on top of our rails stack.

Namespace all the controllers to an API namespace

Our application uses a combined stack of Ruby on Rails and AngularJS frontend. In order to properly have a solid separation of concerns, we need to ensure that all of the JSON controllers are scoped to and API namespace. This gives us other benefits as well

• This is the starting point of our application providing an API for users to consume
• We can get rid of the # that appears when you are viewing the application in any browser.

Document API

Our application has an api namespace that provides mapping coordinates for crime data and vacancies. Other applications should be able to consume this data just fine, but our application does not provide any documentation for how to make that work. We need to provide an API documentation so that consumers of the API know how it is used.

Acceptance Criteria:

• Our application has documentation with regards to how our API works

Swap Mapbox with Google Maps

Google Maps provides for a more cost effective way of managing the app, as they allow 25,000 hits per day as opposed to 50,000 hits per month that map box offers at its free tier. This alone will be very beneficial to the application.

Data sources to consider for vacant properties

See some of the last pages of this presentation Knowing Your Neighborhood Tackling Vacant Properties by Improving Access to Data This is on the wiki.

Remove Extraneous Filters for the Crime Data

The Crime Data tab contains extraneous unneeded filters. The only filters that should exist there are start and end date.

Neighborhood page wireframe

Below is a sample wireframe I mocked up that takes a little more advantage of the possible real-estate on a desktop. Any comments, positive or negative? Anyone have any suggestions for mobile?

Outline Parcels in the application

While we add markers for all of the parcels in the application, this isn't giving a sufficient view of what a vacant parcel is. We need to display the entire parcel to the user when they view it on the map.

For now, a json dataset has been provided that provides all of the coordinates of every parcel to display in KC. We need to import that into the app, and use it to display information to the users.

Footer on the Main Page?

What do you think about having the footer on the main page (KCMO: Data Pioneer and How it Works) appear on all pages, no matter which tab you click? Amanda's a fan of the idea. How do you feel about it @JakeLaCombe ?

Create Navigation System on Each Neighborhood Page

Create navigation system for selecting each data source and updating interactive graphs or maps on the page with the data.

Import Neighborhood Geometrics from Address-Api

The old API to retrieving geometric coordinates is outdated, and needs to be replaced with what the address api provides.

This is the endpoint to retrieve them from. http://api.codeforkc.org/neighborhoods-geo/V0/99?city=KANSAS%20CITY&state=MO

Write Unit Tests for the Application

Unit Testing, especially Test Driven Development, is considered the pivotal of software development methodologies in the Ruby community. If we wish to get outside developers helping with our code, we need to start writing unit tests using RSpec in the application.

Update documentation to reflect local development

Not all machines have postgres install, and instead just need SQLite. We need to add the following line to the README.md file so that new developers can easily setup the application

bundle install --without production

Switch over to Census Neighborhood Data for the Neighborhood Data sets

Right now, we are using the dataset provided at digital fabric to get all neighborhood data. To keep consistent with data, we need to switch over to the api sets the census provides.

Add Legally Abandoned Properties to the API

We need to add an endpoint to our API that returns the legally abandoned properties for a given neighborhood. The following formula can be used to determine that a building is legally abandoned.

1. The property is tax delinquent
2. The property is Vacant
3. The property has at least one additional code violation other than a vacant one.

Tax Delinquent

(To be added)

Vacant

Data Source - https://data.kcmo.org/resource/nhtf-e75a.json
The property is either cited for failure to register as vacant, or has been boarded for over 150 days (violation codes are NSVACANT or NSBOARD01)

Additonal Code Violations

Data Source - https://data.kcmo.org/resource/nhtf-e75a.json
The property as at least one code violation other than NSVACANT or NSBOARD01 (https://data.kcmo.org/resource/nhtf-e75a.json)

Note: Properties that are owned by gov't entity wont be listed as tax delinquent

Inconsistency in crime data display

@JakeLaCombe @zachflanders @zmon

Possible bug.

I noticed today crime data is displaying inconsistently across neighborhoods. In the photo attached, I've got screenshots of the crime display from three different neighborhoods.

There are two possible issues here. First, many years appear to have zero crime reports. This would maybe be understandable for years farther back since we might not have data for those years, but here, it's the last few years that are missing reports. Second, the point at which reports disappear seems to be different, too, if you compare Quality Hill and West Plaza, for instance.

And maybe another issue -- In the screenshot, Ivanhoe has no crime displayed.

Would one of you have a look so we can try to understand whether this something on our end or an issue with the data? Is this something that would be solved with the Address API?

@amandaCCF

Meetup 8/24/15

Crime Tab
@JakeLaCombe and @zachflanders fixed the Crime API in the application and started architecting the heatmap.

Vacancy/Property Tab
Heather and @amandaCCF worked with Peter Hoffman with Legal Aid of Western Missouri to identify the data most useful for his organization and neighborhoods and how it should be presented. Heather will use that to rework the mockups.

UNI presentation
@amandaCCF and @zmon will present to the Urban Neighborhood Initiative’s Partners Meeting on Tuesday, Aug 25 to tell them about the app, get feedback on their data needs, and hear how they will use the data.

Hack KC Mid-Point Presentation
@BenCCF and @amandaCCF will be putting together a short presentation for the midpoint roundup that will also include a demo of the crime tab.

Incorporate Filters inside the Vacancy Tab.

As a Investor
I would like to filter the vacancy data by legally abandoned and vacant indicator codes
So that I can easily determine what Vacancies work the best for investing in based on my desires and opportunities

We will know when:

• We have a filters button inside the vacancy tab
• The button activates a dropdown of different filters we can apply to the map.

I've included a screenshot that makes it easier to see what the objective of this issue is.