bramhoven / luciferin Goto Github PK
View Code? Open in Web Editor NEWEasy to use importer for the open source personal finance manager Firefly III.
Home Page: https://docs.luciferin.bramhoven.nl/
License: GNU General Public License v3.0
Easy to use importer for the open source personal finance manager Firefly III.
Home Page: https://docs.luciferin.bramhoven.nl/
License: GNU General Public License v3.0
Sometimes for credit card transactions can be first authorised and later captured.
If this occurres on seperate days, this can cause duplicate transactions in the Nordigen API.
These can be detected by using the bankTransactionCode
.
Authorisations: PMNT-MCRD-UPCT
Capture: PMNT-CCRD-POSD
Add a date to the tag object and not just only in the name of the tag
New installs do not have any Nordigen id and key in settings, so we won't be able to load the accounts on the homepage at first.
The settings have to be checked before retrieving bank accounts.
Only store transfers that come in from the bank where the transaction originates
Mail, SMS, Push notification via browser?
Add tag with import date to the imported transactions
Currently saving accounts are not selectable as an bank account through Nordigen. We have to check if it is possible to recognize these accounts through transactions or maybe they are returned in another way.
As a user I would want to be able to login to the importer using my Firefly III account.
When connecting a bank through Nordigen, it redirects back to the website on port 80 while the url is https
To store information regarding imported transactions, Nordigen requisitions and configuration we would need database support. Initially this will my mysql.
Some times transfers between 2 asset accounts slip through transfer filtering resulting in 2 transactions.
The transaction duplication checking is now in a working state, but it is not really unit testable.
My idea was to introduce the strategy pattern to split up each step.
The few steps that I can currently think of are:
Each of these "strategies" can be easily unittested to fully confirm their inner workings.
In Firefly III you can specify many asset accounts. Some of these might not be able to be imported by Luciferin. For example a savings account with SNS cannot be connected through Nordigen.
We should be able to detect transactions from or to those accounts from the accounts we are able to import. We can then directly import these as transfers instead of having to move them with rules within Firefly III. This will take a require a good look at the current transaction creation and import process, and it will probably have to be restructured.
Somthing along these lines:
Nordigen -> Mapping -> Duplicate detection -> Transfer detection -> Importing
System.NullReferenceException: Object reference not set to an instance of an object.
at Luciferin.BusinessLayer.Firefly.Models.FireflyAccount.Equals(Object obj) in /src/Luciferin.BusinessLayer/Firefly/Models/FireflyAccount.cs:line 35
at System.Collections.Generic.ObjectEqualityComparer`1.IndexOf(T[] array, T value, Int32 startIndex, Int32 count)
at System.Array.IndexOf[T](T[] array, T value, Int32 startIndex, Int32 count)
at System.Collections.Generic.List`1.Contains(T item)
at Luciferin.BusinessLayer.Import.ImportManager.<>c__DisplayClass6_0.<RunImport>b__9(FireflyAccount a) in /src/Luciferin.BusinessLayer/Import/ImportManager.cs:line 129
at System.Linq.Enumerable.WhereListIterator`1.ToList()
at Luciferin.BusinessLayer.Import.ImportManager.RunImport(DateTime fromDate, CancellationToken cancellationToken) in /src/Luciferin.BusinessLayer/Import/ImportManager.cs:line 143
at Luciferin.BusinessLayer.Import.ImportManagerBase.StartImport(IServiceScope scope, CancellationToken cancellationToken) in /src/Luciferin.BusinessLayer/Import/ImportManagerBase.cs:line 155
at Luciferin.Website.Classes.Queue.QueuedHostedService.BackgroundProcessing(CancellationToken stoppingToken) in /src/Luciferin.Website/Classes/Queue/QueuedHostedService.cs:line 60
Nordigen has been acquired by GoCardless and has been renamed to GoCardless Bank Account Data.
The transaction processing flow should be restructured into a more flexible and less error prone.
Transactions can be retrieved from Nordigen in batches and processed parallel to that. This alone will use less memory than before.
Besides that we should have a factory which defines a flow to be able to easily turn flow features on and of and maybe make it modifiable by the user. Also we have to remove importing the requisitions from this flow. That should be seperate and happen before the transaction import. We also import the accounts and not the requisitions itself.
Asset account import flow:
Base flow:
Luciferin should be able to send push notification through Ntfy
�[41m�[30mfail�[39m�[22m�[49m: Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware[1]
An unhandled exception has occurred while executing the request.
Flurl.Http.FlurlHttpException: Call failed with status code 400 (Bad Request): GET https://ob.nordigen.com/api/v2/accounts/{account id}/details/
at Flurl.Http.FlurlRequest.HandleExceptionAsync(FlurlCall call, Exception ex, CancellationToken token)
at Flurl.Http.FlurlRequest.SendAsync(HttpMethod verb, HttpContent content, CancellationToken cancellationToken, HttpCompletionOption completionOption)
at Flurl.Http.FlurlRequest.SendAsync(HttpMethod verb, HttpContent content, CancellationToken cancellationToken, HttpCompletionOption completionOption)
at Flurl.Http.ResponseExtensions.ReceiveJson[T](Task`1 response)
at Luciferin.BusinessLayer.Nordigen.Stores.NordigenStore.GetAccountDetails(String accountId, OpenIdToken openIdToken) in /src/Luciferin.BusinessLayer/Nordigen/Stores/NordigenStore.cs:line 122
at Luciferin.BusinessLayer.Nordigen.NordigenManager.GetAccountDetails(String accountId) in /src/Luciferin.BusinessLayer/Nordigen/NordigenManager.cs:line 118
at Luciferin.BusinessLayer.Import.ImportManagerBase.GetRequisitions() in /src/Luciferin.BusinessLayer/Import/ImportManagerBase.cs:line 127
at Luciferin.Website.Controllers.ImportController.Index() in /src/Luciferin.Website/Controllers/ImportController.cs:line 39
at lambda_method7459(Closure , Object )
at Microsoft.AspNetCore.Mvc.Infrastructure.ActionMethodExecutor.TaskOfActionResultExecutor.Execute(IActionResultTypeMapper mapper, ObjectMethodExecutor executor, Object controller, Object[] arguments)
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeActionMethodAsync>g__Awaited|12_0(ControllerActionInvoker invoker, ValueTask`1 actionResultValueTask)
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeNextActionFilterAsync>g__Awaited|10_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Rethrow(ActionExecutedContextSealed context)
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeInnerFilterAsync>g__Awaited|13_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeNextResourceFilter>g__Awaited|25_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Rethrow(ResourceExecutedContextSealed context)
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeFilterPipelineAsync>g__Awaited|20_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Awaited|17_0(ResourceInvoker invoker, Task task, IDisposable scope)
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Awaited|17_0(ResourceInvoker invoker, Task task, IDisposable scope)
at Microsoft.AspNetCore.Routing.EndpointMiddleware.<Invoke>g__AwaitRequestTask|6_0(Endpoint endpoint, Task requestTask, ILogger logger)
at Microsoft.AspNetCore.Authorization.AuthorizationMiddleware.Invoke(HttpContext context)
at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware.<Invoke>g__Awaited|6_0(ExceptionHandlerMiddleware middleware, HttpContext context, Task task)
�[41m�[30mfail�[39m�[22m�[49m: Microsoft.AspNetCore.Server.Kestrel[13]
Connection id "0HML56UDU5433", Request id "0HML56UDU5433:00000002": An unhandled exception was thrown by the application.
System.InvalidOperationException: The exception handler configured on ExceptionHandlerOptions produced a 404 status response. This InvalidOperationException containing the original exception was thrown since this is often due to a misconfigured ExceptionHandlingPath. If the exception handler is expected to return 404 status responses then set AllowStatusCode404Response to true.
---> Flurl.Http.FlurlHttpException: Call failed with status code 400 (Bad Request): GET https://ob.nordigen.com/api/v2/accounts/{account id}/details/
at Flurl.Http.FlurlRequest.HandleExceptionAsync(FlurlCall call, Exception ex, CancellationToken token)
at Flurl.Http.FlurlRequest.SendAsync(HttpMethod verb, HttpContent content, CancellationToken cancellationToken, HttpCompletionOption completionOption)
at Flurl.Http.FlurlRequest.SendAsync(HttpMethod verb, HttpContent content, CancellationToken cancellationToken, HttpCompletionOption completionOption)
at Flurl.Http.ResponseExtensions.ReceiveJson[T](Task`1 response)
at Luciferin.BusinessLayer.Nordigen.Stores.NordigenStore.GetAccountDetails(String accountId, OpenIdToken openIdToken) in /src/Luciferin.BusinessLayer/Nordigen/Stores/NordigenStore.cs:line 122
at Luciferin.BusinessLayer.Nordigen.NordigenManager.GetAccountDetails(String accountId) in /src/Luciferin.BusinessLayer/Nordigen/NordigenManager.cs:line 118
at Luciferin.BusinessLayer.Import.ImportManagerBase.GetRequisitions() in /src/Luciferin.BusinessLayer/Import/ImportManagerBase.cs:line 127
at Luciferin.Website.Controllers.ImportController.Index() in /src/Luciferin.Website/Controllers/ImportController.cs:line 39
at lambda_method7459(Closure , Object )
at Microsoft.AspNetCore.Mvc.Infrastructure.ActionMethodExecutor.TaskOfActionResultExecutor.Execute(IActionResultTypeMapper mapper, ObjectMethodExecutor executor, Object controller, Object[] arguments)
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeActionMethodAsync>g__Awaited|12_0(ControllerActionInvoker invoker, ValueTask`1 actionResultValueTask)
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeNextActionFilterAsync>g__Awaited|10_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Rethrow(ActionExecutedContextSealed context)
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeInnerFilterAsync>g__Awaited|13_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeNextResourceFilter>g__Awaited|25_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Rethrow(ResourceExecutedContextSealed context)
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeFilterPipelineAsync>g__Awaited|20_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Awaited|17_0(ResourceInvoker invoker, Task task, IDisposable scope)
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Awaited|17_0(ResourceInvoker invoker, Task task, IDisposable scope)
at Microsoft.AspNetCore.Routing.EndpointMiddleware.<Invoke>g__AwaitRequestTask|6_0(Endpoint endpoint, Task requestTask, ILogger logger)
at Microsoft.AspNetCore.Authorization.AuthorizationMiddleware.Invoke(HttpContext context)
at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware.<Invoke>g__Awaited|6_0(ExceptionHandlerMiddleware middleware, HttpContext context, Task task)
--- End of inner exception stack trace ---
at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware.HandleException(HttpContext context, ExceptionDispatchInfo edi)
at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware.<Invoke>g__Awaited|6_0(ExceptionHandlerMiddleware middleware, HttpContext context, Task task)
at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequests[TContext](IHttpApplication`1 application)
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.