chrisrbe / pp-p2p-parser Goto Github PK

Mintos import with monthly aggregate does not work with current version 1.2.1 and master version.

Error message is parse-account-statements.py: error: unrecognized arguments: --aggregate monthly

With the current implementation, output files are written by the default csv module. This means that any floating numbers are printed with "." as decimal delimiter by default.

To improve the localisation of this script with regards to US and EU decimal delimiters, pandas provides better support for this kind of differentiation. See also pandas.DataFrame.to_csv for the API definition.

decimalstr, default ‘.’
Character recognized as decimal separator. E.g. use ‘,’ for European data.

To be further investigated if pandas or python supports automatic localization detection to make this location based.

Implementing this feature/rework should help resolve also issues reported in #692.

Add support for Estateguru.

https://app.codacy.com/app/chrisrbe/PP-P2P-Parser/issues/index?bid=7182514

Hello,

I've ran the program on a sample file and I get strange results. For example, mintos reports a deposit of 750 and the program translates that into 75,000,000,000

My file is in US notation

Thanks

Trailing spaces

https://codeclimate.com/github/ChrisRBe/PP-P2P-Parser/README.md#issue_5ae765906a8b140001000028

Line too long (84 > 79 characters)

https://codeclimate.com/github/ChrisRBe/PP-P2P-Parser/parse-account-statements.py#issue_5ae7438c0fd7860001000049

Add support for Swaper.

Hallo,

erst einmal vielen Dank für das Schreiben des Parsers, ich bin absolut begeistert! 👍

Mir ist beim Import von Daten von Mintos aufgefallen, dass die berechnete Summe in PP nicht mit der Summe auf der Website übereinstimmt. Ich denke das liegt zum einen daran, dass gewisse Transaktionen nicht richtig klassifiziert werden, und zum anderen verstehe ich noch nicht ganz, wie Mintos die Zahl auf der Website ausrechnet.

Ich habe einige fehlende Transaktionen gefunden und das .yaml File entsprechend angepasst. Leider ist meine Endsumme immer noch falsch. In jedem Fall wollte ich meine geupdateten RegExes hier mal zur Diskussion stellen.

  deposit: "(Deposits)|(^Incoming client.*)|(^Incoming currency exchange.*)|(^Affiliate partner bonus$)"
  withdraw: "(^Withdraw application.*)|(Outgoing currency.*)|(Withdrawal)"
  interest: "(^Delayed interest.*)|(^Late payment.*)|(^Interest income.*)|(^Cashback.*)|(^.*[Ii]nterest received.*)|(^.*late fees received$)"
  fee: "(^FX commission.*)|(.*secondary market fee$)|(.*discount/premium.*)"

Das Problem bei den discount/premium Einträgen ist, dass sie ggf. auch Positiv sein könnten, also eher als Zinsen oder Einlagen gesehen werden könnten, falls auf dem Zweitmarkt etwas mit Gewinn verkauft werden konnte. Bei Rabatten ist der Wert dieser Transaktionen negativ.

• Why not extend PP itself?
• Why not use an/the API instead of downloaded files?

Hi Chris,

I just used your parser to import my account statement from my Mintos account which worked very smooth - thanks a lot for your work!

However, I noticed a small difference after the import into Portfolio Performance: With the mintos default config secondary market transactions seem to be getting ignored.
For my I use case I solved this by adjusting the interest regex to something like this:
interest: "(^Delayed interest.*)|(^Late payment.*)|(^Interest income.*)|(^Cashback.*)|(^Discount.*)"

This worked for me pretty well, because I only had positive (i.e. discounts) in my secondary market transactions. To be honest I am not sure how Portfolio Performance would handle negative interest statements. Do you have any better idea how to handle such secondary market transactions?

If this helps here is a complete example from the mintos export file:
408071234;2018-08-20 16:36:14;Discount/premium for secondary market transaction 409801234 Loan ID: 3993123-01;0.014028056;0.541010512;EUR

Similar blocks of code found in 2 locations. Consider refactoring.

https://codeclimate.com/github/ChrisRBe/PP-P2P-Parser/src/estateguru_parser.py#issue_5ae76edc9c0c63000100001e

What?
I could imagine an optional parameter like --aggregate=month which should sum up transactions per month and booking type. When not given: continue with the current behavior.

Why?
Using auto invest on Mintos I have tens of thousands single transactions in PP already, and they are rapidly growing. I do not care about keeping books on each transaction, but rather overall performance of the P2P portfolio. IMHO interest payments per month are good enough for this purpose.

Am I missing something?

Does not support the new updated structure of Mintos CSV export

Similar blocks of code found in 2 locations. Consider refactoring.

https://codeclimate.com/github/ChrisRBe/PP-P2P-Parser/src/estateguru_parser.py#issue_5ae8932f2ec7a7000100001e

This should serve as an informative issue when one is trying to use the Debitum Network import functionality merged in #227 .

It turns out that the required grouping and subtraction tasks cannot be done easily. This is because the category "REPAYMENT" consists of two different types of payment:

• payments which cleared the full investment. In this case, one can subtract INVESTMENT-REPAYMENT to get the interest for the respective credit. Example: Investment = 10, repayment = 10.04 -> interest = 0,04
• payments which clear only parts of the investment: In this case, INVESTMENT-REPAYMENT does not give the interest because the credit was not yet fully repaid. For example, Investment = 10, repayment = 2 -> interest = -8

Unfortunately there is no information in the account statement which credit was already fully repaid and which only partly. This could solve the issue mentioned above even though manual preprocessing in a grouped manner would still be needed.

I am in contact with Debitum Network to find out how this can be changed / solved in the future.

Thanks for this effort! 👍 Hoping for some P2P automation soon :)

Mintos provides .xlsx files for download.

I converted them to .csv with ; as column separator and , as decimal seperator. To me, the file looks similar to your src/test/mintos.csv.

However, parsing it leads to an output without content, only the header is written.
Is there anything I am doing wrong?

python 3.7.7

Here is a sample file:

Edited with interest lines

Transaction ID;Date;Details;Turnover;Balance;Currency
1861977623;2020-05-05 09:18:08;Deposits; 50,000000000;50,0000000;EUR
1862112425;2020-05-05 11:14:03;Loan 22742301-01 - investment in loan;-10,000000000;40,0000000;EUR
1862112428;2020-05-05 11:14:04;Loan 23125713-01 - investment in loan;-10,000000000;30,0000000;EUR
1862112431;2020-05-05 11:14:05;Loan 30736058-01 - investment in loan;-10,000000000;20,0000000;EUR
1862112457;2020-05-05 11:14:06;Loan 30837110-01 - investment in loan;-10,000000000;10,0000000;EUR
1862112460;2020-05-05 11:14:07;Loan 31320137-01 - investment in loan;-10,000000000; 0,0000000;EUR
1864297148;2020-05-06 02:48:56;Loan 23125713-01 - principal received;  0,278967768; 0,2789678;EUR
1864297091;2020-05-06 02:48:56;Loan 23125713-01 - interest received;  0,002222222; 0,2811900;EUR
1886919551;2020-05-13 00:08:59;Loan 23125713-01 - principal received;  0,280497756; 0,5616877;EUR
1886919491;2020-05-13 00:08:59;Loan 23125713-01 - interest received;  0,015121606; 

Function platform_factory has a Cognitive Complexity of 6 (exceeds 5 allowed). Consider refactoring.

https://codeclimate.com/github/ChrisRBe/PP-P2P-Parser/parse-account-statements.py#issue_5ae8a357d126ee0001000019

Dependency issue when installing via Python 3.8.13 via

pip install -r requirements.txt

results in

The conflict is caused by:
    The user requested pyflakes==2.4.0
    flake8 3.9.1 depends on pyflakes<2.4.0 and >=2.3.0

I have manually fixed the dependecies in https://github.com/cschindlbeck/PP-P2P-Parser/tree/fix-python-3.8.13-dependencies

This could also be related to #490

The CSV I got from estateguru had following header:
"Lening Code","Projektname","Zweitmarkt","Status","Land","Marktursprung","Fnanzierungsdatum","Nächstes Zahlungsdatum","Nächster Zahlungsbetrag","Fälligkeitsdatum","Repaid Date","Zinssatz","LTV","Anfängliche Kreditlaufzeit","Restlaufzeit","Bonus","Ausstehender Hauptbetrag","% der Beteiligungen","Anfänglicher Hauptbetrag","Erhaltener Zins","Erhaltener Bonus","Zurückgezahlter Hauptbetrag","Erhaltene Strafen","Tage verspätet"

but the testdate for estateguru has this header

"UniqueId","Zahlungsdatum","Bestätigungsdatum","Cashflow-Typ","Cashflow-Status","Projektname","Währung","Betrag","Verfügbar für Investitionen"

Did the estateguru format change or did I download the wrong CSV?

Also I got a few errors when importing a mintos csv to PP. I fixed these error by using the ANSI encoding instead of UTF-8 for special characters like ä and ü.

Buchungswährung => Buchungswährung
Gebühren => Gebühren

Current documentation is not user friendly:

• missing proper guidelines to set up a development environment
• how to use
• how to contribute

Function parse_account_statement has a Cognitive Complexity of 14 (exceeds 5 allowed). Consider refactoring.

https://codeclimate.com/github/ChrisRBe/PP-P2P-Parser/src/p2p_account_statement_parser.py#issue_5ae8be2a6c25b90001000016

#525 introduces a new config file for an english version of the estateguru csv dump.

This issue aims to enable multilanguage support from one single config file especially with regards to the csv field names/ headers.

Options:

• Add evaluation for reg ex similar to the already used config entries as in

  PP-P2P-Parser/config/mintos.yml

  Line 3 in 58614ef

  deposit: "(Deposits)|(^Incoming client.*)|(^Incoming currency exchange.*)|(^Affiliate partner bonus$)"

• add support for different language blocks where best guess approach is used to find a matching entry and using the appropriate field names from the best guestimate.

Hi

I install python3 inkl. pip3 on Ubuntu. I clone this git repro and run "pip3 install ruamel.yaml". This is my debug, what's wrong?:

root@docker:/home/psenn/PP-P2P-Parser# python3 parse-account-statements.py --type mintos --debug 20201022-account-statement.csv INFO:Config ini Traceback (most recent call last): File "parse-account-statements.py", line 90, in <module> sys.exit(main(CMD_ARGS.infile, CMD_ARGS.type)) File "parse-account-statements.py", line 61, in main statement_list = platform_parser.parse_account_statement() File "/home/psenn/PP-P2P-Parser/src/p2p_account_statement_parser.py", line 78, in parse_account_statement category = sttmnt.get_category() File "/home/psenn/PP-P2P-Parser/src/Statement.py", line 28, in get_category booking_type = self._statement[self._config.get_booking_type()] KeyError: 'Details'

Mintos export CSV:

rows 2040
Sum: 25,19...€

after parser:
rows 1440
sum: 25.873301€

i think any filter doesnt work. for example TAX, etc?

Just started investing in https://debitum.network/.

Will try adding support and an example csv ASAP.

E:\git_repos\PP-P2P-Parser>parse-account-statements.py --type mintos C:\Users\chris\Downloads\account-statement.csv
Traceback (most recent call last):
File "E:\git_repos\PP-P2P-Parser\parse-account-statements.py", line 70, in
sys.exit(main(args.infile, args.type))
File "E:\git_repos\PP-P2P-Parser\parse-account-statements.py", line 42, in main
statement_list = platform_parser.parse_account_statement()
File "E:\git_repos\PP-P2P-Parser\src\mintos_parser.py", line 43, in parse_account_statement
if self.relevant_income_regex.match(statement['Details']):
KeyError: 'Details'

Dependabot couldn't authenticate with https://pypi.org/simple/.

You can provide authentication details in your Dependabot dashboard by clicking into the account menu (in the top right) and selecting 'Config variables'.

View the update logs.

Add support for Robocash.

Not a python dev and maybe this best practice but I am using /usr/local/bin/python3 on macOS as my default python3 and packages installed via this interpreter are not available for /usr/bin/python3.

Is there a way to force use /usr/local/bin/python3 for execution instead of the one defined within the script?

Beim Test des Parsers ist mir aufgefallen, dass sehr kleine Beträge zu Fehlbuchungen in PP führen.
Eine Buchung über 0,000041275€ führte in PP bspw. zu einer Buchung von 4,13€ als Zinszahlung.

Beispielbuchungen.zip

Similar blocks of code found in 2 locations. Consider refactoring.

https://codeclimate.com/github/ChrisRBe/PP-P2P-Parser/src/mintos_parser.py#issue_5ae76edc9c0c63000100001d

Check if a standalone package could be generated or if deployment on pypi is an option.

Also includes proper release management in a sense.

Just tried to use your script but I got dependency issues on macOS 12

teqqy@teqqBook ~> python3 --version
Python 3.8.9

ERROR: Cannot install flake8==3.9.1 and pycodestyle==2.8.0 because these package versions have conflicting dependencies.

The conflict is caused by:
    The user requested pycodestyle==2.8.0
    flake8 3.9.1 depends on pycodestyle<2.8.0 and >=2.7.0

To fix this you could try to:
1. loosen the range of package versions you've specified
2. remove package versions to allow pip attempt to solve the dependency conflict

ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/user_guide/#fixing-conflicting-dependencies

Cyclomatic complexity is too high in method parse_account_statement. (7)

https://codeclimate.com/github/ChrisRBe/PP-P2P-Parser/src/p2p_account_statement_parser.py#issue_5ae893342ec7a7000100002d

Avoid deeply nested control flow statements.

https://codeclimate.com/github/ChrisRBe/PP-P2P-Parser/src/p2p_account_statement_parser.py#issue_5ae8be2a6c25b90001000018

Initial version does not support multiple p2p sites in a comfortable way

Just started investing in https://estateguru.co/.

Will try adding support and an example csv ASAP.

Similar blocks of code found in 2 locations. Consider refactoring.

https://codeclimate.com/github/ChrisRBe/PP-P2P-Parser/src/mintos_parser.py#issue_5ae8932f2ec7a7000100001d

Function parse_account_statement has 28 lines of code (exceeds 25 allowed). Consider refactoring.

https://codeclimate.com/github/ChrisRBe/PP-P2P-Parser/src/p2p_account_statement_parser.py#issue_5ae8be2a6c25b90001000017