Data Dictionary about housing-insights

[lizmeister321]

Framework for Data Dictionary currently up in Google Doc Format

Plan of action is to:

1. Complete doc for all tables on postgres
2. Convert to Markdown tables
3. Post to housing-insights wiki pages

[NealHumphrey]

Hi @lizmeister321 and @salomoneb - hope your Thanksgiving was good! What's your plan for working on this? Do you think you'll be able to do some this week? I'd like to make sure we have a good, digestable summary version that we can print and have on hand at our Wednesday hacknight, so that everyone has something to refer to about the data we have available. It doesn't have to be the complete data dictionary, but we should have the core structure.

[NealHumphrey]

And let me know what help you need!

[salomoneb]

Hey @NealHumphrey - I was planning to dive into this over the next few nights. I'll probably want to defer to/speak with Liz about this since she set up the Google doc, but for the short term I'd like to:

A) Add each column field for each database table in Column A of the corresponding Google sheet;
B) Define each field in Column B;
C) Add in URL references/citations for each table and/or specific data fields where appropriate. We'll be able to determine what this involves more specifically as we move through the data.

@lizmeister321 - If this plan works for you (or if you have other ideas), maybe we can talk on Slack and figure out how to divide everything up?

[NealHumphrey]

Great, sounds good. Most of the data has data dictionaries you can copy over - but often these are not quite as descriptive as we would like.

I would suggest aiming for a rough version of your 3 steps based on primarily copy-pasting from source dictionaries, and then add a 3rd step of pulling out or summarizing what look like might be the most important fields - it doesn't have to be perfect, just a best first guess. That way people can read just the summary before making mockups, and look at the full list if they need to know if a specific field is available.

[salomoneb]

That sounds good to me. I agree with your idea about calling out the most valuable fields. I'll think about the best way to show that information.

[NealHumphrey]

@salomoneb Looks like a great start on the data dictionary in the Google Docs file. I just wanted to check in and make sure I can see it all - is this everything you've done so far?

Next steps:

1. I can copy in stuff related to the Preservation Catalog that I have handy on the appropriate tabs.

2. We need a 1-2 page summary of key things available in the data, for people to reference when drawing mockups.

Do you think you'll be able to do anything on 2 tonight or tomorrow? If not, I can plan to work on it to make sure we have materials ready for Wednesday.

[salomoneb]

@NealHumphrey Yeah, that's it for now. Wading through the census data/various census websites and tools took a bit longer than planned. I was going to wrap up the tax info tonight and start on another table. I spoke to @lizmeister321 the other day and she was going to work on the neighborhoodinfodc info.

I don't know if I'll be able to finish the other tables and do a 1-2 page summary before Wednesday (though maybe!), but I can probably do one or the other. What's most helpful/highest priority for you? Seems like the summary is the goal, but we need the definitions first. How about I see how much I can get done tonight and then we can reassess where we are tomorrow night?

[NealHumphrey]

Great, sounds good. I am teaching tonight, but can do some work on this during the daytime tomorrow. What if you send me an update on wherever you land this evening. I can do a first draft of summary during the day tomorrow, and then maybe you can just do a quick pass and add anything you think is necessary?

[salomoneb]

Sounds good. I'll send something later.

[salomoneb]

Filled in everything except the neighborhoodinfo tables. I wanted to get these as well, but Excel 2011 for Mac apparently can't open XML spreadsheet files. I might be able to add tomorrow when I'm on my PC work computer.

• Just about every cell that's shaded has a comment attached to it.
• Green cells are informational notes.
• Orange cells are issues (not clear on the field definition; data might be inconsistent or missing)
• The overview tab just has noteworthy links for now, but we can flesh this out.

I'm going to think a little bit more about the analysis portion tomorrow.

[salomoneb]

I updated the data dictionary with the census mapping and building permit tables. The building permit data doesn't have field descriptions, but I think it's still important to list the fields. We can come up with our own descriptions later if necessary.

I also created a new wiki page for general Postgres queries. It might seem sort of trivial, but I always appreciate having these things handy. People should feel free to contribute to this.

Also, Lizzie had suggested converting the Google sheets to Markdown tables. Do we still want to do this? I haven't looked super hard, but csvtomd provides a decent solution (and it can handle multiple files!). csvtomd + [YOUR CSV] + | pbcopy (on mac) and you can just paste into GitHub. My initial concern with moving everything was that we'd need to create lots of pages, which would be confusing, but maybe we could just put all of the tables on one page and anchor link at the top to each one?

[NealHumphrey]

I'm moving the link to the data dictionary onto the housinginsights.org/resources page. We can keep it up to date as we go. Closing this issue