arts / 28degrees-export Goto Github PK

View Code? Open in Web Editor NEW

24.0 24.0 8.0 47 KB

Scraping utility for 28degrees Master Card website

Python 100.00%

28degrees-export's People

Contributors

Stargazers

Watchers

Forkers

michaeltyson bltb hussler87 mbertolacci timkhoury elliotthilaire danielmcconville petemod

28degrees-export's Issues

Auto-detect URLs (Transactions, Statements etc)

See https://github.com/ArtS/28degrees-export/pull/8/files#diff-b1364cf4f46e578846ea7470156fd860R191

HTTP Error 400: Bad Request

Thanks for making this.

The last few times I've tried using it I have been getting:

Logging in...
Traceback (most recent call last):
  File "./export.py", line 315, in <module>
    export(**vars(args))
  File "./export.py", line 227, in export
    br = login(creds)
  File "./export.py", line 30, in wrapped
    r = f(*args, **kwargs)
  File "./export.py", line 129, in login
    br.open('https://28degrees-online.latitudefinancial.com.au/')
  File "/usr/local/lib/python2.7/site-packages/mechanize/_mechanize.py", line 203, in open
    return self._mech_open(url, data, timeout=timeout)
  File "/usr/local/lib/python2.7/site-packages/mechanize/_mechanize.py", line 255, in _mech_open
    raise response
mechanize._response.httperror_seek_wrapper: HTTP Error 400: Bad Request

I suspect the site may have changed. If it's not just me I'll dig into it deeper.

Is anyone else getting this error too?

Total of 0 new transactions obtained

I tried to download new transactions today and the script returned returned Total of 0 new transactions obtained even though I had new transactions available.

I logged into the site and I think the visuals of the table has changed slightly (but maybe not).
I'll have some time to look more closely at the problem and a solution in a couple of weeks.

No module named argparse

Hi when i runt the command I get the following error message..

File "export.py", line 8, in
import argparse

Thanks

IndexError: List index out of range

Hi Guys,
Thanks for building this script. I am looking forward to it working for me. I have installed everything as I am supposed to and I am able to login and start the process. It seems to get to a point and then crash. Here is an output of my script. It does the same if I use the '--csv' switch or not.

This is the first time i have used it so I guess it is trying to extract as many transactions as possible. I noticed there have been some changes to the site over the past couple of months.

Lastly is there a way to tell the script the date ranges to extract?

python export.py --csv
Enter your username: 
xxxxxx
Password: 
Logging in...
OK
Opening transactions page...
OK
Got 20 transactions, from 20/09/2014 to 10/09/2014
Opening next page...
Got 20 transactions, from 10/09/2014 to 03/09/2014
Opening next page...
Got 20 transactions, from 03/09/2014 to 28/08/2014
Opening next page...
Got 20 transactions, from 28/08/2014 to 25/08/2014
Opening next page...
Got 20 transactions, from 24/08/2014 to 22/08/2014
Opening next page...
Got 20 transactions, from 22/08/2014 to 18/08/2014
Opening next page...
Got 20 transactions, from 18/08/2014 to 15/08/2014
Opening next page...
Got 20 transactions, from 14/08/2014 to 06/08/2014
Opening next page...
Got 20 transactions, from 06/08/2014 to 02/08/2014
Opening next page...
Got 20 transactions, from 01/08/2014 to 03/07/2014
Opening next page...
Got 20 transactions, from 30/06/2014 to 11/06/2014
Opening next page...
Got 6 transactions, from 11/06/2014 to 01/06/2014
Opening next page...
Traceback (most recent call last):
  File "export.py", line 286, in <module>
    export(**vars(args))
  File "export.py", line 251, in export
    format_tran_date_for_qif(page_trans[0].date),
IndexError: list index out of range

There was no output file generated...

Move all date formatting to dateutil.py

Latitude Service Centre

Hi there,

I've been using this app for quite a few years - it looks like we're all getting forced onto the new Latitude Service Portal - any chance anyone has looked at updating the script yet?

Coles Mastercard

Hi,

Just letting you know that the Coles No Annual Fee Mastercard is also hosted by Latitude and also works fine with this script for exporting QIF.

Not tested with statements.

I've thrown my version on my fork at: https://github.com/jj777/28degrees-export

Site Change

Has something on the site changed again? With latest from git.

Logging in...
OK
Opening transactions page...
OK
Traceback (most recent call last):
  File "./export.py", line 317, in <module>
    export(**vars(args))
  File "./export.py", line 251, in export
    page_trans = fetchTransactions(text)
  File "./export.py", line 65, in fetchTransactions
    date = parse_tran_date(get_node_text(q('div[name="Transaction_TransactionDat
e"]', row)[0]))
  File "/home/xxxxxxx/28degrees-export/dateutil.py", line 6, in parse_tran_date
    return datetime.strptime(dateStr, f_str)
  File "/usr/lib/python2.7/_strptime.py", line 325, in _strptime
    (data_string, format))
ValueError: time data 'Yesterday' does not match format '%d %b %Y'

Script will fail if there is not enough transactions present for a Next button

I've started using my 28 degrees card again after a while. I have enough transactions that manually copying the details is a pain but not enough to overflow onto a second page.

The script fails because it goes looking for the next button which is not present:

NoSuchElementException: Message: no such element: Unable to locate element: {"method":"name","selector":"nextButton"}

A suitable work around to this problem for now is to spend more money and try again later. 🤑

Full trace below:

Traceback (most recent call last):
  File "./export.py", line 255, in <module>
    export(**vars(args))
  File "./export.py", line 179, in export
    driver = login(creds)
  File "./export.py", line 78, in login
    nextBtn = get_next_btn(driver)
  File "./export.py", line 56, in get_next_btn
    return browser.find_element_by_name('nextButton')
  File "/usr/local/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py", line 440, in find_element_by_name
    return self.find_element(by=By.NAME, value=name)
  File "/usr/local/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py", line 858, in find_element
    'value': value})['value']
  File "/usr/local/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py", line 311, in execute
    self.error_handler.check_response(response)
  File "/usr/local/lib/python2.7/site-packages/selenium/webdriver/remote/errorhandler.py", line 237, in check_response
    raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"name","selector":"nextButton"}
  (Session info: chrome=71.0.3578.98)
  (Driver info: chromedriver=2.37.544337 (8c0344a12e552148c185f7d5117db1f28d6c9e85),platform=Mac OS X 10.14.2 x86_64)

Refactor export()

It's too long.

Unable to access new transactions since November

I've been getting 0 new transactions since mid November. I thought it corresponded with the message for a new card in the mail/activate said card etc. however still nothing new.

Output is:

Enter your username:
.....
Password:
Logging in...
OK
Opening transactions page...
OK
Total of 0 new transactions obtained

Using Python 2.7.6 on Ubuntu Server, 3.13.0-71-generic #114-Ubuntu SMP Tue Dec 1 02:34:22 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux.

Currently setting up a seperate system to test with.

Error

Hi,

I'm getting this error every time I run "export.py":

Traceback (most recent call last):
File "export.py", line 239, in
export()
File "export.py", line 185, in export
t_db = db.init_db()
File "/Users/******/Downloads/28degrees-export-master/db.py", line 13, in init_db
db = sqlite3.connect('./export/transactions.db')
sqlite3.OperationalError: unable to open database file

I'm not a n00b, but I'm not a programmer either, so I'm not too sure what the problem is.

I'm running OSX 10.9.0 and have installed the latest versions of Python, Git, Mechanize and Pyquery. I've also tried it on a second machine running Python 2.7.6, but no luck.

Any thoughts? I'd love to be able to get this working :)

Website

Firstly, thank you for your work so far, much appreciated!

Unfortunately, the recent update to the website seems to have broken your script and produces the following error.

Opening transactions page...
Unable to locate link to transactions page
Exiting...

Thanks in advance

Please include requirements.txt

Hi,

Thanks for making this utility! I'm just trying to get it working and I'm having some trouble installing some of the requirements (PyQuery and Mechanize) so I was wondering if you could include a requirements.txt file so that we have the version numbers you used.

Thanks!

arts / 28degrees-export Goto Github PK

28degrees-export's People

Contributors

Stargazers

Watchers

Forkers

28degrees-export's Issues

Recommend Projects

Recommend Topics

Recommend Org

Jobs