hoffie / dkb-visa

DKB transaction download and conversion (CSV, QIF)

DKB VISA QIF Exporter

The German bank DKB does not provide access to VISA transactions using HBCI, which is why this script has been written.

It will get you your DKB VISA transactions in the QIF format.

It has been inspired by Jens Herrmann's web_bank.py, but has been written using some newer tools/frameworks such as BeautifulSoup and mechanize, in the hope that it will be more stable against changes on the DKB web site.

Thanks to Jens Herrmann for the initial work nevertheless!

You may also be interested in the derived project "dkbweb" by Tom which provides additional features such as bank account transaction exporting.

How does it work?

It will log-in as you at DKB's online banking website, will pretend to be you, will use the CSV export feature and will convert the fetched data to the QIF format, which can be imported by several financial tools.

Requirements

You need Python >=2.7.x and mechanize. On current Ubuntu, this should suffice:

apt-get install python-mechanize

Usage

./dkb.py --userid USER --cardid 1234 --output dkb.qif ./dkb.py --userid USER --cardid 1234 --from-date 01.01.2020 --output 1234.qif --cardid 2345 --from-date 02.02.2020 --output 2345.qif

with USER being the name you are using at the regular DKB online banking web site as well, with 1234 being the last four digits of your card (to select the proper card) and with dkb.qif being the output file.

You will be asked for a start date if you do not specify one using --from-date. Optionally, --to-date can be specified to further limit the time frame. If not given, it defaults to the current date.

Then you will be asked for your PIN and should get your dkb.qif file in the current directory after some informational output.

If it does not work, you may have mis-entered some of the required data or there may have been changes on the DKB web site, in which case adjustments to the script will be necessary.

The resulting QIF file can be imported in several financial tools, although I only tested GnuCash.

To avoid entering PIN and doing two factor authentication multiple times, you also can specify multiple cardids with corresponding output files. In this case,

• output and from-date are mandatory and must be specified for each cardid,
• to-date and qif-account are optional and may be specified either for each cardid or not at all,
• userid is mandatory and must be specified exactly once,
• debug and raw and may be specified once or not at all, and
• the first cardid argument is matched with the first output, from-date and possibly to-date and qif-account arguments, the second cardid argument with the second ouput, from-date, ... argument and so on.

If this script is of any help for you, please let me know. :)

dkb-visa's Issues

Add Github topics to the repository

Hi, can you add the following topics to this repository?

• https://github.com/topics/dkb
• https://github.com/topics/banking
• https://github.com/topics/finance

Thanks!

valuta_date empty for soll/haben

An entry like the following one has an empty COL_VALUTA_DATE, because it does not show a
transaction, only the Soll or Haben (that you currently owe or have). So this
should not be one of the transactions.

"Nein";"01.01.2015";"";"Soll";"-00,01";"";

However filtering and matching transactions to accounts should happen later, so
I used this simple but ugly fix:

if len(line[self.COL_VALUTA_DATE].strip()) == 0:
    return re.sub('.*?(\d{1,2})\.(\d{1,2})\.(\d{2,4}).*?', r'\2/\1/\3',
        line[self.COL_DATE])
else:
    return re.sub('.*?(\d{1,2})\.(\d{1,2})\.(\d{2,4}).*?', r'\2/\1/\3',
        line[self.COL_VALUTA_DATE])

Otherwise the generated qif-file won't be importabable

Python 3 support

Hello,

it looks like mechanize is not available for python3.

I'm currently trying to integrate dkb-visa into my own script, which unfortunately is written in python3. Do you have any plans to migrate to python3? How could I help with the migration?

The following StackOverflow comment mentions some mechanize alternatives for python3, which might be helpful: https://stackoverflow.com/a/31774959

Anway, thank you for the awesome script!

option "--qif-account" is not implemented.

It seems the option "qif-account" is never used. I tried to use it to be able to distinguish between two different accounts when merging both into a common gnucash document and saw no effect. Also, I could confirm in the source code that the value written by the ArgumentParser is never used.

Anyway, thanks for the great rewrite after "web_bank.py" does not work anymore. As a workaround I proposed a feature to web_bank.py and used it myself just doing the conversion from manually downloaded csv files to qif: okurz/web_bank@4175542

Script emits Warning: No parser was explicitly specified

Everytime I run this tool it emits the following warning:

/usr/lib/python2.7/dist-packages/bs4/__init__.py:166: UserWarning: No parser was explicitly specified, so I'm using the best available HTML parser for this system ("lxml"). This usually isn't a problem, but if you run this code on another system, or in a different virtual environment, it may use a different parser and behave differently.

To get rid of this warning, change this:

 BeautifulSoup([your markup])

to this:

 BeautifulSoup([your markup], "lxml")

  markup_type=markup_type))

Does anyone else get this too?

Unable to find transaction selection form

Since yesterday, I get this error message:

Start time: 25.08.2017  
PIN: 
Starting login as user xxx...
Navigating to 'Umsätze'...
Selecting transactions in time frame 25.08.2017 - 01.09.2017...
Traceback (most recent call last):
  File "/usr/bin/dkb.py", line 416, in <module>
    fetcher.select_transactions(args.cardid, from_date, args.to_date)
  File "/usr/bin/dkb.py", line 173, in select_transactions
    br.form = form = self._get_transaction_selection_form()
  File "/usr/bin/dkb.py", line 99, in _get_transaction_selection_form
    raise RuntimeError("Unable to find transaction selection form")
RuntimeError: Unable to find transaction selection form

Did not find the time to look into this in detail, but maybe DKB changed the HTML structure.

to/from date takes value date and not booking date

The command line download start date (--to-date) takes the value dates and not the booking dates. Not really sure what the intention is but from a use case perspective, it might not make sense:

1. I run this script automatically every day to download all card transactions that happened yesterday
2. These new transactions are added automatically to GnuCash so GnuCash is updated every night.

Problem:
If a value date is two days ago or from last week or last month (because a supplier books very late or international transactions or whatever) or there is a refund (which is usually the value date of the original booking), I would have to download all transactions from the last X months.

Only solution I can think of, is do download all transactions from last year and then filter the file by booking date. Maybe this behaviour can be enabled by an additional flag.

Let me know what you think. Am happy to provide the fix.

from_date greater than 3 years ago gives initially displayed transactions and does not create an error.

I tested this via dkbweb.py which calls dkb.py, so this should apply to dkb.py as well.

If you specify a from_date like 1.1.1970, you won't get an error from mechanize. But it triggers another issue: The csv export will only contain the initially visible transactions because the date is not submittable. This means that the page won't update and the csv-file won't be generated with the new dates.

I "fixed" this for myself by fetching all the transactions from the last year every time and later I filter the results via sets. See: https://github.com/inktrap/dkb-export/blob/master/dkbweb.py#L215. Btw.: I also checked that BASEURL is an https-url. :)

Edit: Just to clarify: The allowed from_date is set server side and constant.

Unable to find search period radio box

Running dkb.py as usual results in:

Navigating to 'Umsätze'...
Selecting transactions in time frame 03.03.2017 - 28.05.2017...
Traceback (most recent call last):
  File "./dkb.py", line 427, in <module>
    fetcher.select_transactions(args.cardid, from_date, args.to_date)
  File "./dkb.py", line 190, in select_transactions
    self._select_all_transactions_from(form, from_date, to_date)
  File "./dkb.py", line 115, in _select_all_transactions_from
    raise RuntimeError("Unable to find search period radio box")
RuntimeError: Unable to find search period radio box

Unable to find transaction selection form

Last week and today (19.9.2019) I tried to obtain my transactions and got the following error message and stack trace:

richi@yang:~/Coding/dkb-visa$ ./dkb.py --userid XXXX --cardid YYYY --output dkb_visa_card_umsaetze.qif
Start time: 21.06.2019
PIN: 
Starting login as user XXXX...
Navigating to 'Umsätze'...
Selecting transactions in time frame 21.06.2019 - 18.09.2019...
Traceback (most recent call last):
  File "./dkb.py", line 493, in <module>
    fetcher.select_transactions(args.cardid, from_date, args.to_date)
  File "./dkb.py", line 242, in select_transactions
    br.form = form = self._get_transaction_selection_form()
  File "./dkb.py", line 168, in _get_transaction_selection_form
    raise RuntimeError("Unable to find transaction selection form")
RuntimeError: Unable to find transaction selection form

IndexError: list index out of range

I get the following trace when starting:

Traceback (most recent call last):
File "/home/uwe/dkb-banking/dkb-visa/dkb.py", line 413, in
fetcher.login(args.userid, pin)
File "/home/uwe/dkb-banking/dkb-visa/dkb.py", line 61, in login
br.form = list(br.forms())[0]
IndexError: list index out of range

no data entries can be found at all

Since some days (did not use dkb-visa for some months in before) I can not get any data entries at all, anymore. Maybe webpage form changed again.
Running dkb-visa with pdb debugger and checking the form input IMHO the request URL is fine but the response has an empty block like

…
<input name="$$$event_search" type="hidden" value="search"/>
</form>
</div>
</div>
</div>
</div>
</div>
<div class="clearfix richfooter_wrap" id="richfooter_wrap">
…

where I would actually expect some data as I can see them with the firefox inspector. Does anyone still have success on dkb webpage?

html included in csv file

Note that sometimes the original csv-file includes information, formatted via html.
I solved it like this:

    # check that unparseable lines are not included as transactions
    # dkb should really fix their csv export, they should have the
    # money for that, right?
    csv_text = csv_text.splitlines()
    csv_text_filtered = []
    for line in csv_text:
        if "<br />" in line:
            continue
        else:
            csv_text_filtered.append(line)
    csv_text = "\n".join(csv_text_filtered)

This will exclude transactions that have <br /> in their description.