Comments (13)
Framework for Data Dictionary currently up in Google Doc Format
Plan of action is to:
- Complete doc for all tables on postgres
- Convert to Markdown tables
- Post to housing-insights wiki pages
from housing-insights.
Hi @lizmeister321 and @salomoneb - hope your Thanksgiving was good! What's your plan for working on this? Do you think you'll be able to do some this week? I'd like to make sure we have a good, digestable summary version that we can print and have on hand at our Wednesday hacknight, so that everyone has something to refer to about the data we have available. It doesn't have to be the complete data dictionary, but we should have the core structure.
from housing-insights.
And let me know what help you need!
from housing-insights.
Hey @NealHumphrey - I was planning to dive into this over the next few nights. I'll probably want to defer to/speak with Liz about this since she set up the Google doc, but for the short term I'd like to:
A) Add each column field for each database table in Column A of the corresponding Google sheet;
B) Define each field in Column B;
C) Add in URL references/citations for each table and/or specific data fields where appropriate. We'll be able to determine what this involves more specifically as we move through the data.
@lizmeister321 - If this plan works for you (or if you have other ideas), maybe we can talk on Slack and figure out how to divide everything up?
from housing-insights.
Great, sounds good. Most of the data has data dictionaries you can copy over - but often these are not quite as descriptive as we would like.
I would suggest aiming for a rough version of your 3 steps based on primarily copy-pasting from source dictionaries, and then add a 3rd step of pulling out or summarizing what look like might be the most important fields - it doesn't have to be perfect, just a best first guess. That way people can read just the summary before making mockups, and look at the full list if they need to know if a specific field is available.
from housing-insights.
That sounds good to me. I agree with your idea about calling out the most valuable fields. I'll think about the best way to show that information.
from housing-insights.
@salomoneb Looks like a great start on the data dictionary in the Google Docs file. I just wanted to check in and make sure I can see it all - is this everything you've done so far?
Next steps:
-
I can copy in stuff related to the Preservation Catalog that I have handy on the appropriate tabs.
-
We need a 1-2 page summary of key things available in the data, for people to reference when drawing mockups.
Do you think you'll be able to do anything on 2 tonight or tomorrow? If not, I can plan to work on it to make sure we have materials ready for Wednesday.
from housing-insights.
@NealHumphrey Yeah, that's it for now. Wading through the census data/various census websites and tools took a bit longer than planned. I was going to wrap up the tax info tonight and start on another table. I spoke to @lizmeister321 the other day and she was going to work on the neighborhoodinfodc info.
I don't know if I'll be able to finish the other tables and do a 1-2 page summary before Wednesday (though maybe!), but I can probably do one or the other. What's most helpful/highest priority for you? Seems like the summary is the goal, but we need the definitions first. How about I see how much I can get done tonight and then we can reassess where we are tomorrow night?
from housing-insights.
Great, sounds good. I am teaching tonight, but can do some work on this during the daytime tomorrow. What if you send me an update on wherever you land this evening. I can do a first draft of summary during the day tomorrow, and then maybe you can just do a quick pass and add anything you think is necessary?
from housing-insights.
Sounds good. I'll send something later.
from housing-insights.
Filled in everything except the neighborhoodinfo tables. I wanted to get these as well, but Excel 2011 for Mac apparently can't open XML spreadsheet files. I might be able to add tomorrow when I'm on my PC work computer.
- Just about every cell that's shaded has a comment attached to it.
- Green cells are informational notes.
- Orange cells are issues (not clear on the field definition; data might be inconsistent or missing)
- The overview tab just has noteworthy links for now, but we can flesh this out.
I'm going to think a little bit more about the analysis portion tomorrow.
from housing-insights.
I updated the data dictionary with the census mapping and building permit tables. The building permit data doesn't have field descriptions, but I think it's still important to list the fields. We can come up with our own descriptions later if necessary.
I also created a new wiki page for general Postgres queries. It might seem sort of trivial, but I always appreciate having these things handy. People should feel free to contribute to this.
Also, Lizzie had suggested converting the Google sheets to Markdown tables. Do we still want to do this? I haven't looked super hard, but csvtomd provides a decent solution (and it can handle multiple files!). csvtomd + [YOUR CSV] + | pbcopy
(on mac) and you can just paste into GitHub. My initial concern with moving everything was that we'd need to create lots of pages, which would be confusing, but maybe we could just put all of the tables on one page and anchor link at the top to each one?
from housing-insights.
I'm moving the link to the data dictionary onto the housinginsights.org/resources page. We can keep it up to date as we go. Closing this issue
from housing-insights.
Related Issues (20)
- Convert welcome modal to Bootstrap or fix semantic ui sizing issue HOT 1
- Refactor `_populate_zone_facts` to be cleaner
- Calculate percent subsidized by zone
- Add lat/lon to geocode from mar cleaner step
- Populate 'in_prescat', 'in_dhcd', 'in_dchousing' fields
- Troubleshoot why 'Most Recent REAC date' and 'most recent TOPA date' sliders are bunched up on load HOT 1
- Dedupe address match failing due to integer MAR IDs HOT 1
- Add all project fields to the filter api endpoint
- User Feedback? HOT 2
- Persist missing address id in create_project_subsidy_csv() when returned from mar api call
- test_load_data.py failing due to refactored _populate_zone_facts HOT 1
- Bug in Range Breakdown Displayed (Zone Type: Census Tract && Data Choices: Crime Rate: Violent 12 months) HOT 1
- Inaccurate residential_units counts for at least three census tracts HOT 2
- DHCD addresses not found in MAR HOT 2
- Data retry now prevents null lat/lon project view pages from loading HOT 2
- Preservation Catalog November 2017 update HOT 1
- Front-end refactor master issue
- Track client-side JavaScript errors Rollbar.js HOT 6
- Back-end refactor and data validation master issue
- Possible Data or Import issue (Chastleton)
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from housing-insights.