18f / 2015-foia-hub Goto Github PK

Create an API to receive requests --- post API

• Accommodate agency specific fields.
• validations on the serverside
  • if bad info --> Return 400?; & errors, if possible.
• if success
  • return 201 & unique id

(After this info is published to agency API.)

Data folder: (TODO - add the location)
Loader location: (TODO - add loader file location)

Explore usa.gov data, and figure out how to integrate with our contact data.
Create tickets as necessary to make it happen.

usa.gov data: http://www.usa.gov/About/developer-resources/federal-agency-directory/index.shtml

This is a place holder for the conversations that we are having, but since it is kind of dependency for everything else, I am turning it into a ticket.

What is stored?

• Request
  • All subsets of the request: 18F/2015-foia-design#8
• Response
• Unique request id
• ids to other systems
• status
• agency contacts
• ????

Create a way to cross link ids from different systems that refer to the same request.

What could a schema look like? How would we ingest it? Where would it live?

SO MANY QUESTIONS THAT MUST BE ANSWERED!

Using agency description, keywords, and potentially foia-search data.

Document fields that will be in agency API -- this is for... #27
/cc @vz3

Create agency API for FOIA-Hub to use.
Requires: #16, #9

Learn more about how information is made available now, inconsistencies in contact info, and discuss what information stakeholders want.

The main thing is:

• Make the form work for every agency

And for how the form works and is stored:

• Storing agency-specific fields as a JSON object.
• Make sure API field requirements are what the form indicates they are.
• Expedited processing request, in a way that doesn't beg everyone to always turn it on.
• Requester categories (press, commercial, noncommercial)
• Classifying an agency (OSTP) as "fee-rare" and having UX optimized for that
• Earliest date for records, indicate if possible
• Accommodating non-email contact details, in an unoverwhelming way.
• Review @jackiekazil's notes for other things that came up talking with partners.

This all needs to be reflected in the intake API, of course.

Should be done here in process_agency_csv():
https://github.com/18F/foia-core/blob/master/foia_core/scripts/load_agency_contacts.py

process_agency_csv was written w/ the csvs as priority, but the yaml data was given priority, so feel free to chop this function up. It is not being used at the writing this this ticket.

The goal of this ticket ---
Match the agency slug, match the department slug, then try to match the name.
If you match the name, then add the email to be associated with the name.
Do no match titles, because they do not match.

The main ones that you are looking for are public liaisons and chief foia officers.

Build an endpoint for agency to pull/push FOIA data from core.
What should the end point include?

• All submitted info
• Unique id
• ???

Set up api on FOIA dev server

Descriptions, names, etc. To blend with any released FOIA data we can get in here, for request-oriented search.

Along with #38 and #39, we could use some functional tests that hit the API with different requests and validate the responses, so we can refactor with more confidence going forward.

This is also an opportunity to choose the test manager we want to use for the app generally. @khandelwal and @jackiekazil, you would probably know best what the right call is for Django/Python today. FWIW, my personal preference is for lighter-touch, testunit-style libraries that focus on basic assertions, rather than RSpec-style libraries that introduce a meta-infrastructure and DSL, but whatever you want to do, I'm in.

Set up base structure for project -- includes data storage and set up files.

Add a way to agencies to add custom fields by agency -- example -- some agencies have custom fields that they want to add -- this should be at the time of initial set up.

Related: #23

@jackiekazil wants a settings default so that foia-hub. Jackie will you add in details of how database connections, and secrets are handled?

For the routing of requests and contacts -- Identify the diff between quality of data between FOIA.gov site & csv

More info here on the location of the web content & csv: https://github.com/18F/foia/tree/master/contacts

The question for this ticket -- are there differences in lists?

• web scrape - has descriptions.
• csv - has email addies.

Related #7 - Acquire & store a list all government agencies

Get an ELB and point it at the main instance, so that our IP will stay more stable over time.

Will also require updates to our bookmarks/links/DNS, but we should do it early on so it's less of a pain. Not so urgent for this sprint, though.

Setup foundation for jinja2 templates in Django

Our dev server should reflect the merge, and clear away unused config and app code.

• add clarification on pyenv notes, mac, and ubuntu
• add example path to the python3 location (ie mkvirtualenv --python=/usr/local/bin/python3 foia-core)
• add pip to virtualenv & virtualenvwrapper
• add start postgres instructions to "could not connect to server" instructions.
• add setting django-admin.py to settings OR switch this command to /.manage.py

Out:

• move cloning repo down to not interfere with env setup stuff
  B/c you need to clone the repo to install the requirements.

... so people know what to do this with code

There are many GB downloaded, in a similar format to State's get it loaded, so that we can start exposing it in our prototype.

Identify why usa.gov has additional entities that DOJ dataset doesn't have
Do we need to include the additions for organizational purposes for routing? What if someone want to route to one of the agencies / departments not included?

• USA.gov info: http://www.usa.gov/About/developer-resources/federal-agency-directory/index.shtml
• FOIA.gov info: https://github.com/18F/foia/tree/master/contacts

Related #7 - Acquire & store a list all government agencies

As discussed in our sprint retrospective, I think @vz3 needs to provide some simple comps to help document where the current thinking around fees and the expedited fields is.

• at least one record with lots of contacts
• at least one record that lacks email and / or phone
• at least one denormalized recorded

Research and implement a way to generate randomized tracking numbers.
(p.s. if usa.gov data is integrated, maybe prefix with normalized agency abbreviation. If not, put this in a separate follow up ticket.)

Compare service center & phone data fields in agency data. These are different, but do they have the same information? Do we need both?

When FOIA request is closed, then core gets notification and closes the case -- actually notification will probably be determined by future research on integration.

• Status updates core.
• Triggers closed email: foia-portal#5
• Grabs pointer to any data docs that were provided in the response.
• Grabs official response from the agency.
• ???

This ticket is for the rewrite of https://github.com/18F/foia/tree/master/contacts

In the process, please adjust the following ---

• Fix this bug which was manually updated in this commit -- usdoj/foia.gov@e918343#diff-d41d8cd98f00b204e9800998ecf8427e
• Add the Chief FOIA Office data to the output. It is located under the Agency icon on the right. (I believe this is on the Agency, not on the office, but I am not certain.)
• This record gets cut off
  Farm Credit Administration
  description: "FCA´s mission is to ensure a dependable source of credit for agriculture and rural America. We do this in two ways:",
• Put out put in this folder (https://github.com/18F/foia-core/tree/master/data), unless we want to make a data repo OR put all the data in main repo -- Open for discussion. I am more concerned with consistency.
• Put script here: https://github.com/18F/foia-core/tree/master/foia_core/scripts -- This might move / if it is turned into a management command,... so this might be a temp holding place. This is to keep it with the other script and is open to discussion.
• Is this scrape error or data error? If data error, can you put in a fix for it output correctly in the yaml? usdoj/foia.gov@ee3fcdb
• This too: usdoj/foia.gov@a2f8f59
• This too: usdoj/foia.gov@6b010bd

I think previous thinking has been that we will store contact data in Github. Let's consolidate that thinking, and revisit those assumptions again.

@khandelwal, hoping you can take a stab at a Markdown doc with documentation for our request intake API, as we talked about last week.

I just laid out the state of things in usdoj/foia.gov#25, and some ideas about what to do next, so this should be done in parallel to updating fields, which I'll add in a separate ticket.

... so people know what to do this with code

Figure out how we are handling access / authentication, then implement or create tickets, then implement.

Compare current contact data list to USA Search to determine if we currently have the best data model for the hierarchy/contact list.

I opened a discussion thread on our API (to make sure non-gov't people saw it) at usdoj/foia.gov#25, and included some thoughts on next steps. They're not complete, and I'd encourage discussion and additions to be added to that thread.

As an actionable thing, let's group some of the immediate changes I outlined, assuming they're non-objectionable, on this ticket. We're not building to a committed v1 here -- this is another round of improvements to make this feel more like a proper API.

So this includes:

• Field name changes
• Ensuring optional and required fields in the API match those as indicated on form
• Cleaning up response body

We have 18f.us set up in Amazon Route 53 in an approved manner, and we can create subdomains very easily.

Parse FOIA officer info from xls data file from FOIA.gov. FOIA officer is not in web scrape.
Should we do anything with other titles in the spreadsheet?

Acquire a list all government agencies (& sub agencies) - we are doing to need this. Source - TBD.

One specific use case on the front-end: 18F/2015-foia-design#2

• @konklone for an assist.