socrata / connectors

ETL Tools and Templates

Socrata QGIS Plugin

Created By Peter Moore

Requirements

Due to TLS security issues with older versions, the plugin requires QGIS3 or above.

Usage

The Socrata plugin is added the 'Web' menu.

Enter in the domain. Username and Password are optional fields.

You can use the "Get Maps" button to get a list of all map assets on the domain or conversely enter in the dataset ID if you have it.

By clicking OK the map will then be added to your project.

NOTES:

All maps on Socrata are in WGS84 and are retrieved through the GeoJSON endpoint.

connectors's Issues

This is a test

`Python Exception <KeyError>: 'layerId'` With Shapefile to Socrata Workflow

We have several workflows that use the Shapefile to Socrata Workflow. No changes have been made on our end. Has the Socrata's API for this changed?

FME Log Output

FME_END_PYTHON: evaluating python script from string...
Python Exception <KeyError>: 'layerId'
...

Request Response That Should Have layerId But Doesn't

I inspected the output of response_summary['layers'] that is used to populate the blueprint dictionary inside the resplaceWithShapeScanResults function. The following is the output, which should have a key/value pair for layerId, but doesn't. I replaced the value of the 4x4, but if you need an example affected dataset 4x4, I can provide that:

[   {   u'name': u'fake-data',
        u'referenceSystem': u'NAD_1983_StatePlane_Tennessee_FIPS_4100_Feet'}]

Also, making sure @aplannersguide is looped in.

CSV writer

Suggest removing Shapefile 2 CSV directory from the repo completely.

Update template submodules to refresh header to match dev.socrata.com

Requires updating _layouts and common. You'll also need to to add a link back to dev.socrata.com to your local nav.

Get rid of superfluous `_site` folder

Looks like you checked in a Jekyll generated _site directory: https://github.com/socrata/connectors/tree/gh-pages/templates/_site

Add a "Connector Services" section

For things like the SeeClickFix connector service. Verify its OK to add SCF connector as service in this list (also should be in app store)

geometry types

It looks like the Esri connector only supports point features. Is this intentional or should we also be supporting lines and polygons?

Truncate

The 'Truncate Dataset First' parameter on the writer set to 'Yes' causes the workspace to fail. Looks like it can't truncate/delete and reload the way FME works with other databases.

Test Issue

Default Writer dataset

The Socrata writer defaults to openddata.socrata.com

It may have to be changed to the correct workspace, in my case: soda.demo.socrata.com

empty connector page

no information in the connector Open311

Timeouts

Large datasets cause the writer to time out. In this case ~67k point features.

Differentiate between Safe FME and Kettle templates

Footer contribute link does not work

footer link "Want to contribute?" on connectors gh-pages page does not work.

I think this is related to the common site.contribute variable at https://github.com/socrata/labs-common-jekyll/blob/master/default.html#L101 not being defined in connectors repo

Merge connectors from the master branch into the gh-pages branch

Having two branches is confusing to people, combining them would make it simpler.

Proposal: In gh-pages, each connector gets its own folder, with its code and an index.md describing it for the catalog.