cadenceonefive / cleanknit

Take Socrata datasets and intersect them to get clean normalized fragments

CleanKnit

Take Socrata datasets and intersect them to get clean normalized fragments

cleanknit's Issues

demonstrate use of URL generation via SQL to download Socrata resources

-- demonstration of using SQLite database to code-generate the download URLs
-- for the Socrata metadata collection for the resources of each domain
​
create table socrata_domain
(
    domain varchar primary key,
    resource_count integer not null
);
insert into socrata_domain
    (domain, resource_count)
values
    ('data.somervillema.gov', 54),
    ('data.lacity.org', 2819),
    ('data.cityofnewyork.us', 3461),
    ('data.sfgov.org', 1085);
​
SELECT 'curl "https://api.us.socrata.com/api/catalog/v1?domains=' || printf('%s&offset=0&limit=%d" --output %s.json', domain, resource_count,domain)
FROM socrata_domain
WHERE resource_count > 0;
-- curl "https://api.us.socrata.com/api/catalog/v1?domains=data.somervillema.gov&offset=0&limit=54" --output data.somervillema.gov.json
-- curl "https://api.us.socrata.com/api/catalog/v1?domains=data.lacity.org&offset=0&limit=2819" --output data.lacity.org.json
-- curl "https://api.us.socrata.com/api/catalog/v1?domains=data.cityofnewyork.us&offset=0&limit=3461" --output data.cityofnewyork.us.json
-- curl "https://api.us.socrata.com/api/catalog/v1?domains=data.sfgov.org&offset=0&limit=1085" --output data.sfgov.org.json

Excel UDFs for browsing Socrata

Provide some configuration mechanism to indicate the default domain for searches.

=SocrataSearch("pluto")
=SocrataResource(resource_id, column_list, 100)

Command line client for Socrata

small client with some command-line wrapper for doing stuff with Socrata such as:

1. creating and maintaining a metadata database
2. downloading/mirroring resources
3. creating relational databases with subsets/samples of Socrata resource data

SQLite copies of the NYC socrata tabular

SQLite copies of the NYC socrata tabular datasets of interest that have been obtained by

1. shredding the metadata
2. code-generating the delimited file download URLs
3. profile the data
4. manually cleaned up schemata for a subset of the datasets
5. manually written schema with foreign key constraints