joesalmon1985 / canvas

project to combine canvassing returns, electoral register data and membership information

Canvas

project to combine canvassing returns, electoral register data and membership information

Running the try7.py file should create a new file within a db (named as you choose) which lists the 9 people who live in the ward now, taking into account that 3 people Bob Rimmington, Natisha Morris and Thomas Morris have left.

However one of the residents doesn't get added because of too many apostropihies in there name.

Try it.

canvas's Issues

councilupdate.py can't use a txt file list of csv files to use

this def doesn't work at the moment and gives an unhelpful error message of IOError: [Errno 2] No such file or directory: 'council02.csv \n'

I think I need to strip the '\n' bit from the line, but can't figure out how.

Need python equivaent of .output like in SQLite

So now we can basically do .import, it might be nice to be able to create output in python as well, some nice csv files from the tables in the databases. Maybe a .output is the wrong direction to go in however.

Scripts need to be less fussy over column headings

Currently freshdata.py and councilupdate.py only work with specific file headings for the CSV files they use, down to the correct character. This could be improved massively, maybe with something smart enough to recognise what type of data is in each column i.e. a name, an address, an intention.

need working version of findmembers.sql in python

we need a working version of findmembers.sql in python. This does what create up to date registers does, but also adds in a filed saying who is a member, and what their email address is.

way to convert postcodes from lower to upper case

A way to convert the lower case postcodes sometimes found on the members database to upper case is required. This could be in SQlite or in Python, as long as the solution doens't include any SQlite command line commands.

t_add_postcode_column.py needs explaining text

I would find it very handy to understand what t_*.py programs do.

python eqivalent of .import

I would find it very useful to have a .import stylee way of createing tables from .csv files in Python.

Deal with additions and deletions

On the updates these also contain deletions as well as additions. One fix would be to simply delete these from the update .csv's, however they won't be removed until the recipt of the anual full electoral register.

Can we modify current SQL to pick out postcode or modify this to a Python code which does this?

Can we modify current SQL to pick out postcode or modify this to a Python code which does this? Most urgently createuptodateregister.sql

python version of createuptodateregister

We need a python version of createuptodateregister

importcsvfile needs to handle apostrophies

So ' make sqlite3 very unhappy.

strHeaders = "'" + "','".join( headingstouse ) + "'"
strSomething = "'" + "','".join( something ) + "'" ""

need to put in square brackets and it'll love it.

combine addpostcode.py with other python code.

Tried to use addpostcode.py in branch 22. This doesn't work for some reason.

Street name generation from electoral registers.

Improve creation of street field, currently this is dependent on writing out 100's of street names and prevents scalability.

Try2.py fails if a column is already present it is trying to add.

Traceback (most recent call last):
File "try4.py", line 294, in
cursor.execute( tobedone )
sqlite3.OperationalError: duplicate column name: v12

This is the error message, the problem is the script needs to make sure the existing canvassing data table has all the same column headings as it will expect later. To do this it adds empty columns if these columns don't already exist.

This is fine if they don't, but if they do it ends the program.

Postcode not name to identify street

Joe, please close #5 in favour of this.

We can work on the basis that a postcode is "good enough" to identify a street for knocking, further I suspect that I can easily convert a postcode to a street name. I will write some code in SQL, python or termsql that will add a postcde COLUMN to the "141201 Harewood.csv" file from the Council.

Need to create tests for all def's in councilupdate.py

I should write tests for all def's in the program councilupdate.py

def createcouncilupdatefiles (self, databaseused):   

Done but fails

def findcouncildata (self, databaseused):

Done, but only works if the test hasn't been run before. Maybe need to do something to delete the test.db at the end of the t_councilupdate.py program?

def addtoremove (self, databaseused):

Not done but should be easy

def addingcouncilremoved (self, databaseused):

Not done but should be easy

def addingtocouncilupdated (self, databaseused):

Done and happy with it

def addingpostcodes (self, databaseused):

Done, although could be more robust maybe?

need working version of createuptodateregister in python

We need a working version of createuptodateregister in python

https://github.com/Joesalmon1985/Canvas/blob/master/createuptodateregister.sql

This creates a table with all the addtions added and deletions taken out.

Idiot proof interface

Develop an idiot proof way of users dropping in their own versions of

council*.csv
old.csv
members.csv (optional)

and running the code to provide them with the file
1newdata.csv
as a minimum