bramhoven / luciferin

Easy to use importer for the open source personal finance manager Firefly III.

Hi 👋, I'm Bram

A passionate developer from The Netherlands

• 🔭 I’m currently working on Luciferin (A Firefly III Importer)

• 📫 How to reach me [email protected]

Connect with me:

GitHub stats:

Languages

Frameworks

Backend

Frontend

Devops

Other

luciferin's Issues

Implement notification if bank connection has expired

Mail, SMS, Push notification via browser?

[WISH] Automatically check for new transactions

[WISH] Import events on website to see status

[BUG] Transfers result in duplicate transactions

Only store transfers that come in from the bank where the transaction originates

[WISH] Add date to Firefly III tag

Add a date to the tag object and not just only in the name of the tag

Restructure transaction processing flow

The transaction processing flow should be restructured into a more flexible and less error prone.
Transactions can be retrieved from Nordigen in batches and processed parallel to that. This alone will use less memory than before.
Besides that we should have a factory which defines a flow to be able to easily turn flow features on and of and maybe make it modifiable by the user. Also we have to remove importing the requisitions from this flow. That should be seperate and happen before the transaction import. We also import the accounts and not the requisitions itself.

Asset account import flow:

1. Retrieve requisition
2. Retrieve accounts
3. Compare accounts with Firefly III asset account
4. Import the new ones
5. Store accounts in DB
   • New accounts should be marked as new / have a recalculate balances flag

Base flow:

1. Retrieve paginated transactions for period from Nordigen
2. Retrieve transactions within same range of the current page from Firefly III
3. Store a list of hashes (or normalized strings) of the Firefly III transactions for checking against
4. Convert transaction from API model to an object we understand
5. The next steps should be a per transaction flow:
   1. Check for existing transactions against the existing hashes list
   2. Check for duplicates in current import
      • Maintain a list of hashes (or normalized strings) that we already have imported.
      • Whenever we hit a duplicate check if we can extend the currently imported transaction (and do that)
   3. Check whether the account is one of the asset accounts
   4. Check whether it could be a transfer from an internal savings account
   5. Map it to the correct account
   6. Post it to Firefly III
6. Trigger an transaction processed event
7. That event should cause a calculation of balances for accounts marked as new or having a recalculate balances flag

A few requirements

1. Code should be concise and readable (ie. a factory that defines the import flow and another that defines the transaction processing flow)
2. Batches that have been imported should not be references any more and not be held in memory
3. Notifications should be send whenever it is not possible to process a transaction, with clear details about the transaction
4. Logs should be generated for all the steps. For diagnostics purposes and for showing in the import log

[BUG] New installs gives error on Nordigen call

New installs do not have any Nordigen id and key in settings, so we won't be able to load the accounts on the homepage at first.
The settings have to be checked before retrieving bank accounts.

[WISH] OAuth support

As a user I would want to be able to login to the importer using my Firefly III account.

[WISH] Restructure transaction duplication checking

The transaction duplication checking is now in a working state, but it is not really unit testable.
My idea was to introduce the strategy pattern to split up each step.
The few steps that I can currently think of are:

• Transfer detecting
• Transactions that already exist in firefly
• Duplicate transactions that are received from Nordigen (although I have never encountered this)

Each of these "strategies" can be easily unittested to fully confirm their inner workings.

Correctly handle empty name when adding a new bank connection

Recognizing saving accounts

Currently saving accounts are not selectable as an bank account through Nordigen. We have to check if it is possible to recognize these accounts through transactions or maybe they are returned in another way.

Correctly map transfers between asset accounts that are not imported by Luciferin

In Firefly III you can specify many asset accounts. Some of these might not be able to be imported by Luciferin. For example a savings account with SNS cannot be connected through Nordigen.

We should be able to detect transactions from or to those accounts from the accounts we are able to import. We can then directly import these as transfers instead of having to move them with rules within Firefly III. This will take a require a good look at the current transaction creation and import process, and it will probably have to be restructured.

Somthing along these lines:
Nordigen -> Mapping -> Duplicate detection -> Transfer detection -> Importing

[BUG] Nordigen return URL is bugged

When connecting a bank through Nordigen, it redirects back to the website on port 80 while the url is https

[BUG] Sometimes transfer slip through filtering

Some times transfers between 2 asset accounts slip through transfer filtering resulting in 2 transactions.

Error when importing

System.NullReferenceException: Object reference not set to an instance of an object.
         at Luciferin.BusinessLayer.Firefly.Models.FireflyAccount.Equals(Object obj) in /src/Luciferin.BusinessLayer/Firefly/Models/FireflyAccount.cs:line 35
         at System.Collections.Generic.ObjectEqualityComparer`1.IndexOf(T[] array, T value, Int32 startIndex, Int32 count)
         at System.Array.IndexOf[T](T[] array, T value, Int32 startIndex, Int32 count)
         at System.Collections.Generic.List`1.Contains(T item)
         at Luciferin.BusinessLayer.Import.ImportManager.<>c__DisplayClass6_0.<RunImport>b__9(FireflyAccount a) in /src/Luciferin.BusinessLayer/Import/ImportManager.cs:line 129
         at System.Linq.Enumerable.WhereListIterator`1.ToList()
         at Luciferin.BusinessLayer.Import.ImportManager.RunImport(DateTime fromDate, CancellationToken cancellationToken) in /src/Luciferin.BusinessLayer/Import/ImportManager.cs:line 143
         at Luciferin.BusinessLayer.Import.ImportManagerBase.StartImport(IServiceScope scope, CancellationToken cancellationToken) in /src/Luciferin.BusinessLayer/Import/ImportManagerBase.cs:line 155
         at Luciferin.Website.Classes.Queue.QueuedHostedService.BackgroundProcessing(CancellationToken stoppingToken) in /src/Luciferin.Website/Classes/Queue/QueuedHostedService.cs:line 60

[WISH] Add tags to imported transactions

Add tag with import date to the imported transactions

Whenever Nordigen throws exception Luciferin does not handle it correctly and shows a blank screen

�[41m�[30mfail�[39m�[22m�[49m: Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware[1]
      An unhandled exception has occurred while executing the request.
      Flurl.Http.FlurlHttpException: Call failed with status code 400 (Bad Request): GET https://ob.nordigen.com/api/v2/accounts/{account id}/details/
         at Flurl.Http.FlurlRequest.HandleExceptionAsync(FlurlCall call, Exception ex, CancellationToken token)
         at Flurl.Http.FlurlRequest.SendAsync(HttpMethod verb, HttpContent content, CancellationToken cancellationToken, HttpCompletionOption completionOption)
         at Flurl.Http.FlurlRequest.SendAsync(HttpMethod verb, HttpContent content, CancellationToken cancellationToken, HttpCompletionOption completionOption)
         at Flurl.Http.ResponseExtensions.ReceiveJson[T](Task`1 response)
         at Luciferin.BusinessLayer.Nordigen.Stores.NordigenStore.GetAccountDetails(String accountId, OpenIdToken openIdToken) in /src/Luciferin.BusinessLayer/Nordigen/Stores/NordigenStore.cs:line 122
         at Luciferin.BusinessLayer.Nordigen.NordigenManager.GetAccountDetails(String accountId) in /src/Luciferin.BusinessLayer/Nordigen/NordigenManager.cs:line 118
         at Luciferin.BusinessLayer.Import.ImportManagerBase.GetRequisitions() in /src/Luciferin.BusinessLayer/Import/ImportManagerBase.cs:line 127
         at Luciferin.Website.Controllers.ImportController.Index() in /src/Luciferin.Website/Controllers/ImportController.cs:line 39
         at lambda_method7459(Closure , Object )
         at Microsoft.AspNetCore.Mvc.Infrastructure.ActionMethodExecutor.TaskOfActionResultExecutor.Execute(IActionResultTypeMapper mapper, ObjectMethodExecutor executor, Object controller, Object[] arguments)
         at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeActionMethodAsync>g__Awaited|12_0(ControllerActionInvoker invoker, ValueTask`1 actionResultValueTask)
         at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeNextActionFilterAsync>g__Awaited|10_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
         at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Rethrow(ActionExecutedContextSealed context)
         at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
         at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeInnerFilterAsync>g__Awaited|13_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
         at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeNextResourceFilter>g__Awaited|25_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
         at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Rethrow(ResourceExecutedContextSealed context)
         at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
         at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeFilterPipelineAsync>g__Awaited|20_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
         at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Awaited|17_0(ResourceInvoker invoker, Task task, IDisposable scope)
         at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Awaited|17_0(ResourceInvoker invoker, Task task, IDisposable scope)
         at Microsoft.AspNetCore.Routing.EndpointMiddleware.<Invoke>g__AwaitRequestTask|6_0(Endpoint endpoint, Task requestTask, ILogger logger)
         at Microsoft.AspNetCore.Authorization.AuthorizationMiddleware.Invoke(HttpContext context)
         at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware.<Invoke>g__Awaited|6_0(ExceptionHandlerMiddleware middleware, HttpContext context, Task task)
�[41m�[30mfail�[39m�[22m�[49m: Microsoft.AspNetCore.Server.Kestrel[13]
      Connection id "0HML56UDU5433", Request id "0HML56UDU5433:00000002": An unhandled exception was thrown by the application.
      System.InvalidOperationException: The exception handler configured on ExceptionHandlerOptions produced a 404 status response. This InvalidOperationException containing the original exception was thrown since this is often due to a misconfigured ExceptionHandlingPath. If the exception handler is expected to return 404 status responses then set AllowStatusCode404Response to true.
       ---> Flurl.Http.FlurlHttpException: Call failed with status code 400 (Bad Request): GET https://ob.nordigen.com/api/v2/accounts/{account id}/details/
         at Flurl.Http.FlurlRequest.HandleExceptionAsync(FlurlCall call, Exception ex, CancellationToken token)
         at Flurl.Http.FlurlRequest.SendAsync(HttpMethod verb, HttpContent content, CancellationToken cancellationToken, HttpCompletionOption completionOption)
         at Flurl.Http.FlurlRequest.SendAsync(HttpMethod verb, HttpContent content, CancellationToken cancellationToken, HttpCompletionOption completionOption)
         at Flurl.Http.ResponseExtensions.ReceiveJson[T](Task`1 response)
         at Luciferin.BusinessLayer.Nordigen.Stores.NordigenStore.GetAccountDetails(String accountId, OpenIdToken openIdToken) in /src/Luciferin.BusinessLayer/Nordigen/Stores/NordigenStore.cs:line 122
         at Luciferin.BusinessLayer.Nordigen.NordigenManager.GetAccountDetails(String accountId) in /src/Luciferin.BusinessLayer/Nordigen/NordigenManager.cs:line 118
         at Luciferin.BusinessLayer.Import.ImportManagerBase.GetRequisitions() in /src/Luciferin.BusinessLayer/Import/ImportManagerBase.cs:line 127
         at Luciferin.Website.Controllers.ImportController.Index() in /src/Luciferin.Website/Controllers/ImportController.cs:line 39
         at lambda_method7459(Closure , Object )
         at Microsoft.AspNetCore.Mvc.Infrastructure.ActionMethodExecutor.TaskOfActionResultExecutor.Execute(IActionResultTypeMapper mapper, ObjectMethodExecutor executor, Object controller, Object[] arguments)
         at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeActionMethodAsync>g__Awaited|12_0(ControllerActionInvoker invoker, ValueTask`1 actionResultValueTask)
         at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeNextActionFilterAsync>g__Awaited|10_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
         at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Rethrow(ActionExecutedContextSealed context)
         at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
         at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeInnerFilterAsync>g__Awaited|13_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
         at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeNextResourceFilter>g__Awaited|25_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
         at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Rethrow(ResourceExecutedContextSealed context)
         at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
         at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeFilterPipelineAsync>g__Awaited|20_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
         at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Awaited|17_0(ResourceInvoker invoker, Task task, IDisposable scope)
         at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Awaited|17_0(ResourceInvoker invoker, Task task, IDisposable scope)
         at Microsoft.AspNetCore.Routing.EndpointMiddleware.<Invoke>g__AwaitRequestTask|6_0(Endpoint endpoint, Task requestTask, ILogger logger)
         at Microsoft.AspNetCore.Authorization.AuthorizationMiddleware.Invoke(HttpContext context)
         at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware.<Invoke>g__Awaited|6_0(ExceptionHandlerMiddleware middleware, HttpContext context, Task task)
         --- End of inner exception stack trace ---
         at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware.HandleException(HttpContext context, ExceptionDispatchInfo edi)
         at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware.<Invoke>g__Awaited|6_0(ExceptionHandlerMiddleware middleware, HttpContext context, Task task)
         at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequests[TContext](IHttpApplication`1 application)

Imported credit card transactions can cause duplicate transactions

Sometimes for credit card transactions can be first authorised and later captured.
If this occurres on seperate days, this can cause duplicate transactions in the Nordigen API.

These can be detected by using the bankTransactionCode.

Authorisations: PMNT-MCRD-UPCT
Capture: PMNT-CCRD-POSD

Create asset account for import account if they do not exist yet

[WISH] Add database support for storage

To store information regarding imported transactions, Nordigen requisitions and configuration we would need database support. Initially this will my mysql.

Nordigen renamed to GoCardless Bank Account Data

Nordigen has been acquired by GoCardless and has been renamed to GoCardless Bank Account Data.

1. Check if everything still works
2. Rename to new name

Sending push notifications through Ntfy

Luciferin should be able to send push notification through Ntfy

[WISH] Configuration UI on website

[WISH] Dockerization