Comments (12)
I'm still getting errors when I try to add to the database, but here's a CSV of (I think all of) the zoning data on http://maps.dcoz.dc.gov/zr16/.
https://www.dropbox.com/s/5ntjt1pxgwy58ul/pud.csv?dl=0
The code that I used to generate it is here: https://gist.github.com/salomoneb/4eaec0e1f578418eb682151bb2e6816f
I scraped the data by querying an endpoint that I found on the map page: https://maps2.dcgis.dc.gov/dcgis/rest/services/DCOZ/Zone_Mapservice/MapServer/25/query?f=json&where=Square ='
+ SQUARE NUMBER + '&returnGeometry=false&spatialRel=esriSpatialRelIntersects&outFields=*
The script constructs URLs using each property square number in the dc_tax
table and requests data from each one, outputting the results in the CSV.
This is the first real Python script I've written and wildly impractical for ongoing purposes, but it does give us a large baseline snapshot. We'll also probably need to clean the data because many of the values seem to be in inconsistent formats.
If you have a contact at the Office of Zoning, getting direct access to the database is probably the way to go.
from housing-insights.
One thing to point out now that I'm looking at this on my work computer with a functional copy of Excel: check out the "case_type_relief" column, which has at least six PUD-related categories.
from housing-insights.
Awesome! Good stuff - congrats on your first 'real' Python script π It's not wildly impractical either - just a few optimizations as well as some things to make it robust if we need to call the endpoint regularly (and a sleep function to keep their servers happy). Other things to look into are whether they also provide a 'latest' endpoint we could tap for just updates/changes, or a bulk endpoint of some sort.
There is a GIS Steering Committee coming up on Thursday at 10am, I'll go to that and see if I can make any contacts in the Zoning office. They have been doing lots of modern GIS stuff, including recently released tools including the maps.dcoz.dc.gov page you cited, so there's likely someone there that would be interested in talking to us about putting their data to use.
What exactly is each row of this? The case_type_relief column looks like it means nearly every row has some sort of zoning issue associated with it, so does this only return results for square numbers that have associated zoning changes/
from housing-insights.
Noting per our discussion last Tuesday that we'll want to refactor the code to include properties that are on the zoning map but don't have any case data associated with them. We talked about putting these in a separate CSV. I'll do this, upload the code to scripts/analysis
, and add the modified data to the database (if anything changes).
from housing-insights.
Sorry for the delay. This is done. Before I submit a PR, can you give me access to the secrets.json
file needed to set up AWS so that I can upload the scraped data?
from housing-insights.
@salomoneb I wanted to check, do you have any documentation on that API endpoint? We should contact the office of zoning to verify we know how up-to-date that endpoint is; I can do this if you let me know what you have/haven't sent to them already.
from housing-insights.
Unfortunately, I don't have any documentation. I found it by looking at the script sources on the zoning map page. http://maps.dcoz.dc.gov/zr16/
Looking at the dateFiled
column of the scraped results, the most recent, legitimate-looking date is Monday, December 12, 2016. There are also some dates that are listed as being filed in 2020, but I don't know why that is.
The map itself is labeled on the map portal page as a 2016 map, so it's possible that it doesn't include 2017 data. I called the zoning office, but wasn't able to reach the map person. I'm going to try again in a bit.
from housing-insights.
Notes from our conversation with Matthew Holden at Office of Zoning:
- API endpoint @salomoneb mentions above is best place to get up to date data from zoning. Data is not completely accurate, with older data more likely to be incorrect or missing - Matt guessed about 70% accurate. We should link to the actual filings themselves when we cite specific zoning info so that people can get real data out of the filing documents when needed.
- Building permit data from opendata.dc.gov is best source; it is updated automatically on a weekly basis from the relevant database at DCRA (although those are a combo of manually/systematically updated so there will be some lag there).
- No programatic source of the number of inclusionary zoning units that are attached to the PUD updates from the map API endpoint. Have to dig into the filings themselves. Key thing to look for is the 'benefits' requirements attached to zoning relief (also includes things like parks, amenities, etc.)
from housing-insights.
Other notes:
- Raw building permit data is compiled manually by DCRA and automatically pulled into opendata.dc.gov weekly.
- In the building permit data, look at the permit_subtype_name field for more info about project.
- People only go through Office of Zoning after theyβve applied for a building permit through DCRA and need a special exception or exemption.
- Using public amenities to get subsidy (?)
- Some zoning information is incorrect. Likely applies more to cases that are over 5 years old.
- Inclusionary zoning - requires certain percentage of residential unit to be set aside for affordable housing. (see https://dhcd.dc.gov/service/inclusionary-zoning-affordable-housing-program). Varies based on zones. New zoning boundaries defined here: http://dcgis.maps.arcgis.com/apps/MapJournal/index.html?appid=ae5830535a384a7f9ae40527f607b32e. 2016 boundaries seem to have replaced ones defined in 1958, which (I think) are what we scraped from the zoning endpoint in the existingZoning field. Weβd need to update the field this field by mapping old boundaries to new ones.
- Zoning changes occur based on map amendments.
from housing-insights.
@salomoneb I'm not sure if it helps, but I did some quick and dirty analysis of the pud.csv file you scraped to check and see whether the PUD case files represented in that csv might include useful details on the Affordable Housing inventory #90 . It's a different topic than what you're covering in this Issue but there may still be some useful context in the notes I drafted. Bottom line is we will probably need to scrape more data from the endpoint mentioned above if we want to figure out how much detail those PUD filings may have on AH inventory.
from housing-insights.
Note the PUD scraper that @salomoneb wrote should be adapated to follow the BaseApiConn approach in the /python/sources
folder to line up with our current code. We'll want to refactor his scraper into this new format once #198 is done (to provide us a with a source of SSL's) or possibly utilizing the same data as #140 (which would be another source of all SSL's - but note that for this scraper we also want PUD's for non-residential stuff as commercial development nearby to an existing affordable housing unit is important)
from housing-insights.
Putting on hold until either #198 or #140 is done. When that's done we probably want to re-write this in a new ticket to clearly specify how to use the new approach to BaseApiConn classes and which specific data we want to pull from this endpoint.
from housing-insights.
Related Issues (20)
- Convert welcome modal to Bootstrap or fix semantic ui sizing issue HOT 1
- Refactor `_populate_zone_facts` to be cleaner
- Calculate percent subsidized by zone
- Add lat/lon to geocode from mar cleaner step
- Populate 'in_prescat', 'in_dhcd', 'in_dchousing' fields
- Troubleshoot why 'Most Recent REAC date' and 'most recent TOPA date' sliders are bunched up on load HOT 1
- Dedupe address match failing due to integer MAR IDs HOT 1
- Add all project fields to the filter api endpoint
- User Feedback? HOT 2
- Persist missing address id in create_project_subsidy_csv() when returned from mar api call
- test_load_data.py failing due to refactored _populate_zone_facts HOT 1
- Bug in Range Breakdown Displayed (Zone Type: Census Tract && Data Choices: Crime Rate: Violent 12 months) HOT 1
- Inaccurate residential_units counts for at least three census tracts HOT 2
- DHCD addresses not found in MAR HOT 2
- Data retry now prevents null lat/lon project view pages from loading HOT 2
- Preservation Catalog November 2017 update HOT 1
- Front-end refactor master issue
- Track client-side JavaScript errors Rollbar.js HOT 6
- Back-end refactor and data validation master issue
- Possible Data or Import issue (Chastleton)
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
π Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. πππ
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google β€οΈ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from housing-insights.