Zoning / PUD application status nearby about housing-insights

[salomoneb]

I'm still getting errors when I try to add to the database, but here's a CSV of (I think all of) the zoning data on http://maps.dcoz.dc.gov/zr16/.

https://www.dropbox.com/s/5ntjt1pxgwy58ul/pud.csv?dl=0

The code that I used to generate it is here: https://gist.github.com/salomoneb/4eaec0e1f578418eb682151bb2e6816f

I scraped the data by querying an endpoint that I found on the map page: https://maps2.dcgis.dc.gov/dcgis/rest/services/DCOZ/Zone_Mapservice/MapServer/25/query?f=json&where=Square =' + SQUARE NUMBER + '&returnGeometry=false&spatialRel=esriSpatialRelIntersects&outFields=*

The script constructs URLs using each property square number in the dc_tax table and requests data from each one, outputting the results in the CSV.

This is the first real Python script I've written and wildly impractical for ongoing purposes, but it does give us a large baseline snapshot. We'll also probably need to clean the data because many of the values seem to be in inconsistent formats.

If you have a contact at the Office of Zoning, getting direct access to the database is probably the way to go.

[salomoneb]

One thing to point out now that I'm looking at this on my work computer with a functional copy of Excel: check out the "case_type_relief" column, which has at least six PUD-related categories.

[NealHumphrey]

Awesome! Good stuff - congrats on your first 'real' Python script 👍 It's not wildly impractical either - just a few optimizations as well as some things to make it robust if we need to call the endpoint regularly (and a sleep function to keep their servers happy). Other things to look into are whether they also provide a 'latest' endpoint we could tap for just updates/changes, or a bulk endpoint of some sort.

There is a GIS Steering Committee coming up on Thursday at 10am, I'll go to that and see if I can make any contacts in the Zoning office. They have been doing lots of modern GIS stuff, including recently released tools including the maps.dcoz.dc.gov page you cited, so there's likely someone there that would be interested in talking to us about putting their data to use.

What exactly is each row of this? The case_type_relief column looks like it means nearly every row has some sort of zoning issue associated with it, so does this only return results for square numbers that have associated zoning changes/

[salomoneb]

Noting per our discussion last Tuesday that we'll want to refactor the code to include properties that are on the zoning map but don't have any case data associated with them. We talked about putting these in a separate CSV. I'll do this, upload the code to scripts/analysis, and add the modified data to the database (if anything changes).

[salomoneb]

Sorry for the delay. This is done. Before I submit a PR, can you give me access to the secrets.json file needed to set up AWS so that I can upload the scraped data?

[NealHumphrey]

@salomoneb I wanted to check, do you have any documentation on that API endpoint? We should contact the office of zoning to verify we know how up-to-date that endpoint is; I can do this if you let me know what you have/haven't sent to them already.

[salomoneb]

Unfortunately, I don't have any documentation. I found it by looking at the script sources on the zoning map page. http://maps.dcoz.dc.gov/zr16/

Looking at the dateFiled column of the scraped results, the most recent, legitimate-looking date is Monday, December 12, 2016. There are also some dates that are listed as being filed in 2020, but I don't know why that is.

The map itself is labeled on the map portal page as a 2016 map, so it's possible that it doesn't include 2017 data. I called the zoning office, but wasn't able to reach the map person. I'm going to try again in a bit.

[NealHumphrey]

Notes from our conversation with Matthew Holden at Office of Zoning:

• API endpoint @salomoneb mentions above is best place to get up to date data from zoning. Data is not completely accurate, with older data more likely to be incorrect or missing - Matt guessed about 70% accurate. We should link to the actual filings themselves when we cite specific zoning info so that people can get real data out of the filing documents when needed.
• Building permit data from opendata.dc.gov is best source; it is updated automatically on a weekly basis from the relevant database at DCRA (although those are a combo of manually/systematically updated so there will be some lag there).
• No programatic source of the number of inclusionary zoning units that are attached to the PUD updates from the map API endpoint. Have to dig into the filings themselves. Key thing to look for is the 'benefits' requirements attached to zoning relief (also includes things like parks, amenities, etc.)

[salomoneb]

Other notes:

• Raw building permit data is compiled manually by DCRA and automatically pulled into opendata.dc.gov weekly.
• In the building permit data, look at the permit_subtype_name field for more info about project.
• People only go through Office of Zoning after they’ve applied for a building permit through DCRA and need a special exception or exemption.
• Using public amenities to get subsidy (?)
• Some zoning information is incorrect. Likely applies more to cases that are over 5 years old.
• Inclusionary zoning - requires certain percentage of residential unit to be set aside for affordable housing. (see https://dhcd.dc.gov/service/inclusionary-zoning-affordable-housing-program). Varies based on zones. New zoning boundaries defined here: http://dcgis.maps.arcgis.com/apps/MapJournal/index.html?appid=ae5830535a384a7f9ae40527f607b32e. 2016 boundaries seem to have replaced ones defined in 1958, which (I think) are what we scraped from the zoning endpoint in the existingZoning field. We’d need to update the field this field by mapping old boundaries to new ones.
• Zoning changes occur based on map amendments.

[emkap01]

@salomoneb I'm not sure if it helps, but I did some quick and dirty analysis of the pud.csv file you scraped to check and see whether the PUD case files represented in that csv might include useful details on the Affordable Housing inventory #90 . It's a different topic than what you're covering in this Issue but there may still be some useful context in the notes I drafted. Bottom line is we will probably need to scrape more data from the endpoint mentioned above if we want to figure out how much detail those PUD filings may have on AH inventory.

[NealHumphrey]

Note the PUD scraper that @salomoneb wrote should be adapated to follow the BaseApiConn approach in the /python/sources folder to line up with our current code. We'll want to refactor his scraper into this new format once #198 is done (to provide us a with a source of SSL's) or possibly utilizing the same data as #140 (which would be another source of all SSL's - but note that for this scraper we also want PUD's for non-residential stuff as commercial development nearby to an existing affordable housing unit is important)

[NealHumphrey]

Putting on hold until either #198 or #140 is done. When that's done we probably want to re-write this in a new ticket to clearly specify how to use the new approach to BaseApiConn classes and which specific data we want to pull from this endpoint.