opennorth / represent-canada Goto Github PK

Some addresses contain Unicode spaces.

Uses the hashtag as the form input value and submits.

• http://drupal.org/project/represent
• http://drupal.org/project/webform_represent

We can also add the Environmental Defence logo, and maybe remove Make them work for you and Next Parliament (or move them to a different page).

The beginnings of a little OS ecosystem?

https://github.com/tauberer/represent-maps

And fix the two "wtf" comments.

Electoral calendars:

• http://www.psc-cfp.gc.ca/plac-acpl/leave-conge/ann2-eng.htm
• http://en.wikipedia.org/wiki/Canadian_electoral_calendar
• http://www.muniscope.ca/research/municipal_facts/Elections/

Some provinces publish election dates:

• Prince Edward Island has an API

Twitter accounts:

• electionsbc
• electionsab
• electionssask
• electionsmb
• electionson
• electionsonfr
• electionsquebec
• electionspei
• electionsns
• nlelections
• elections_nu

All BC provincial electoral districts, for example, amount to 1MB of data, which is not very mobile friendly. If increasing the simplification leads to poor results at high zoom levels, we may want to add a "mobile" version of the shapes as well.

• smile analysis

Geocoder.ca is not very reliable. Use GovKit-CA to create a concordance file at the federal level.

Send us e-mail when CPU utilization stays above ~90%

The City of Vancouver recently updated their website and images are now broken.

Link to Mayors page with photo
Link to Councillors page with photos

The site is still up and down. These pages might not work when you try them.

To offset inaccuracies caused by using the postal code centroid. Currently, such queries are very expensive.

Keep track of most popular exceptions to see if API documentation needs improvement.

Here's a wishlist, which may be broken into smaller issues once we start coding the dashboard:

• send alerts when scraper fails
• keep history of changes in scraper output
• send alerts if number or representatives doesn't match expected number
• perform a little validation

• Yukon: Chief Electoral Officer either doesn't have, or doesn't know about, a digital boundary file.
• Northwest Territories: Needs to decide whether they want to work with us. Apparently, it's a "No."
• Nunavut: Sent second request (Feb 15)

I've contacted all parties that currently do not permit distribution and am awaiting responses. The affected jurisdictions are:

• Ontario: forwarded to Larry Forman (Feb 8, Mar 26). Larry called May 8 to explain that the license on the digital boundary file has become a policy decision. It must now go to senior management, and as it is not a high priority in the current context, there is unlikely to be a change in the near future.

Permission granted

• Prince Edward Island: license added
• Alberta: negative reply from AltaLIS, got permission from Elections Alberta
• British Columbia: got permission from Elections BC
• Nova Scotia: got permission from Elections Nova Scotia
• Calgary: license updated

Improved licensing

I asked PEI about putting a license on their website, and they did! Alberta distributes data through AltaLIS, so it would be a lot of work to change its license. Nova Scotia has a license in its geospatial metadata file; convincing them to change it may be too much work.

Next steps

• Push the private repository to GitHub
• Update the install instructions in the README

see old Boundary Service code

Allow filtering on boundary set and representative set, so that you can get only federal-level information, for instance.

Possible name: elus.opennorth.ca

To more easily identify where to focus on improving the database.

• percent of representative sets with email, etc.
• check for any unexpected duplicate information, e.g. same email
• all unique values for elected_office and gender (we should switch "City councillor" to "Councillor")
• check that all required fields are present

Perhaps we can have slugs that are shorter than these full names.

• Legislative Assembly of British Columbia
• Legislative Assembly of Alberta
• Legislative Assembly of Saskatchewan
• Legislative Assembly of Manitoba
• Legislative Assembly of Ontario
• Assemblée nationale du Québec
• Legislative Assembly of New Brunswick
• Legislative Assembly of Prince Edward Island
• House of Assembly of Newfoundland and Labrador
• Nova Scotia House of Assembly
• Yukon Legislative Assembly
• Legislative Assembly of the Northwest Territories
• Nunavut Legislative Assembly

Non-profits and advocacy groups will want to retrieve electoral districts (and possibly other information) for postal codes in their databases. A web or desktop tool to perform these lookups will help get more users for Represent, and make it a more attractive project to fund.

Some users of howdtheyvote.ca's API prefer XML.

• general video about the Represent service
• short tutorial videos showing how to use each module? (WordPress, Drupal, jQuery plugin)

No data sources found yet. We have lists of municipal associations which may have data.

• Ontario
• Nunavut

Pupa (official):

• New Brunswick
• Newfoundland and Labrador (PDF)
• Prince Edward Island (HTML)
• Manitoba (HTML)
• Saskatchewan (PDF)
• Yukon (PDF)

Pupa (unofficial):

• Union of Nova Scotia Municipalities (HTML and PDF)
• Northwest Territories Association of Communities (HTML)

Scraperwiki (official):

• Quebec: scraper
• British Columbia: scraper
• Alberta: scraper (Elected_01_Cities.xls specifies wards for Calgary and Edmonton, but we have city-specific scrapers for those cities)

Existing scrapers are at: https://scraperwiki.com/tags/cdnpoli Existing scrapers are all under 100 lines, so you can easily read one or two to learn how to write a new scraper. Please write your scraper on ScraperWiki using PHP, Python or Ruby, and consult the list of fields to scrape.

Here's the logic for how we link representatives to boundaries. If the scraper sets a boundary_url field (to, for example, '/boundaries/census-subdivisions/3520005/'), we link the rep to the boundary whose url matches. Otherwise, if it sets district_id, we link it to the boundary whose external_id matches. Otherwise, if it sets district_name, we slugify the field's value and link it to the boundary whose slugified name matches.

Once your scraper is ready:

• change its status to "protected"
• share it with the users "jpmckinney" and "rhymeswithcycle"
• set its schedule to "once per week"
• tag it with "cdnpoli"

And you're done! If you have any questions, do not hesitate to ask [email protected].

Top 7 municipalities makes 1/4 of population. 13 for 1/3. 42 for 1/2. 126 for 2/3. 248 for 3/4.

Twitter handles change. To account for that, we may want to write a separate scraper that uses Twitter account IDs to get the latest handle, which other scrapers integrate with. Facebook doesn't let you change your URL, so reps would need to create a new account to invalidate our data (sucks to be them?). Likewise with changing Twitter account ID.

Scrapers may just need a big hash of name to social media accounts.

Yajl is the fastest JSON parser.

We haven't asked Yukon or Northwest Territories, as we don't have boundaries.

Note: If we want to extend to cities, Montreal has a web service that performs a concordance.

Done

• Québec
• Nunavut: received from Elections Nunavut through third party
• Newfoundland and Labrador: received from Elections Newfoundland and Labrador, who will look into posting the data on the website (Feb 20, 2012)
• New Brunswick: received from Elections New Brunswick (Aug 29, 2012)

Not free

• Saskatchewan has concordances from 2003 (the last representation order). The next update will be in 2013. Made using PCCF from Statistics Canada by a retired colleague.
• Ontario: Larry Forman called May 8, 2012, to explain that the license on the postal code concordance file has become a policy decision. Since then, Elections Ontario no longer distributes the concordance file, as it is waiting to hear back from Canada Post about whether it can publish postal code data and in what form (Aug 20, 2012).
• Alberta: through contract with Canada Post. Am still waiting on reply from Canada Post representative (Feb 6, Feb 15 and Mar 25, 2012), but it is unlikely Canada Post will allow distribution.
• British Columbia: through contract with BC Stats, though licensing agreement with DMTI Spatial.

Not available

• Nova Scotia: "Elections Nova Scotia does not generate this file, so I can not provide you with it."
• Manitoba: "Many rural voters receive their mail in small towns some distance from where they live. We have found that there are too many mailing addresses which are outside of the electoral division where the voter resides."
• Prince Edward Island: "Elections P.E.I. does not have a Postal Code File. Many of our Islanders have mail delivered under Rural Routes and the Postal Codes do not reflect any provincial boundaries." PEI can provide a list of addresses for each of their three counties. Each address list includes a number for an electoral district.

Compile static CSV files daily and upload to S3.

Because the API samples are loaded via ajax, if you loaded the page via an anchor link, your position will jump after about a second to somewhere you didn't want to go.

Change the representative name link to a name followed by URL, e-mail, and API icons.

98.6% of the calls to HowdTheyVote.ca's API are postal code lookups, with another 0.9% lat/lng lookups. When the House of Commons is in session, traffic often reaches 10,000 requests. It sometimes jumps to 200,000 requests in a day when a major campaign is underway.

eg http://mapit.mysociety.org/area/2514.html

• Non-developers get confused when they see a page of JSON.
• Better SEO: makethemworkforyou.com primarily gets traffic from searches for politicians' contact info.
• Add a sitemap.xml to make it easier for Google to spider.

The Alberta Ministry of Municipal Affairs and possibly others publish information about their municipalities' mayors and councillors. We want to email them to ask for lists of mayors (and ideally councillors) with contact information, for every municipality in their jurisdiction.

We have contact info for ministries of municipal affairs and municipal associations. Ministry data is preferred over municipal association data. Issue #16 tracks the effort to scrape available info from ministries and associations, and may contain additional relevant info.

A sample letter for requesting ward boundaries can be tailored for this effort.

Feel free to contact [email protected] if you have any questions.

Avoid further forking of Boundary Service.

https://github.com/tauberer/represent-boundaries

e.g. whether a certain location or representative is "hot"

We coordinate our progress around this spreadsheet. Need to check existence of ward boundaries in: MB, NB, NL, ON, QC.

In the "Ward boundaries" column we track progress with the following ad-hoc codes. Feel free to invent your own, as the following don't cover all possible situations:

• "N/A": the municipality doesn't have wards. You're done!
• "Yes": the municipality has ward boundaries. Next step is to email them.
• "Yes, sent email": the municipality has been contacted, and we are awaiting a response.
• "Integrated": the municipality's ward boundaries have been added to the Represent database.

Once we know which municipalities have wards, we email them to:

• get ward boundaries
• ask under what license the boundaries are made available
• ask when the boundaries will be redrawn (boundaries commission schedule), or when they were last redrawn
• get a list of elected officials with contact information in machine-readable format

A sample letter is given in this document. You may make any modifications you wish (including shortening it).

We ought to think about incentives for cities to keep us updated. If possible, a good incentive would be if we were able to provide postal code concordances.

Feel free to contact [email protected] if you have any questions.

http://en.wikipedia.org/wiki/List_of_municipalities_in_Ontario

http://en.wikipedia.org/wiki/County#Canada

In Sarnia, for example, some city councillors also sit on the county council. In Ontario, at least, each division of a county is a census subdivision. The current schema assumes representatives belong to a single boundary. I don't see a problem with creating a "duplicate" representative and associating them to another boundary.

Group provinces, municipalities...