Rethink query to get plans from redis by using better filtering about middleware

[Artonus]

Rethinking the way of filtering is very close to reviewing how we access and store the data. For example, it may turn out that storing data in another database (relational or document) can be a better idea than keeping it in Redis, as we do at the moment.
Things to consider in the DB are that it should provide us with a way to scale it to multiple replicas.

[Artonus]

We should handle this topic sooner than later.
We will need a separate meeting to discuss this. I would suggest a meeting next week when Radu gets better.
What we have to discuss:

• Can we use Redis for complex queries?
• Can eventual improvements in the Redis queries support the existing data structure that separates data types into the Redis databases?
• Is the RDBMS/NoSQL DB a better option for us to access the data?
• What data could be moved to the new data store, and what should stay in Redis?
• Can the database support replication in the future?
• Migration plan for the new database

[Artonus]

Possible packages to be used:
https://github.com/redis/redis-om-dotnet
https://github.com/redis/NRedisStack

[Artonus]

The biggest migration challenge is to enable backward compatibility with the data. Some partners are using an older version of the Middleware than the latest. The migration can be complicated within a single database. A new data store will be needed.

Possible migration strategy:

1. Create a new database
2. Pump the existing data into a new database
3. Keep 2 databases running in parallel. / shutdown database over the weekend and force everyone to upgrade to the new version (not recommended)
4. ...

[Artonus]

At the meetings, it was decided that we would keep using Redis but would rework the querying system and use different packages, allowing us better querying capabilities.
The following things have been decided:

1. We keep using Redis in Cluster mode. There is a need to migrate the current Redis configuration to the new cluster, @radu-popescu
2. Prepare the Data Transfer Objects layer to transition between the current models, which will be treated as domain objects. The Redis connection will store DTOs, which later will be mapped to domain objects on retrieval from the data store. @adrianLIrobotics
3. Prepare the k8s job or another service that will transfer all existing objects from the current Redis Data Format to the new data format @Artonus

In the future, there will be a situation in which we will have to store historical data. Redis is unsuitable for this. For analytics and historical data, the SQL database will be needed.

Ad.1 When Redis is in cluster mode, the cluster replica will have to be deployed as a part of the Middleware and automatically connect to the existing cluster. Therefore, the cluster should function in the form of Multi-Leader Replication.

Ad.2: The process of switching the data access has to complete a set of steps:

• Data access in the Middleware has to be extracted to the separate class library to enable better portability for applications responsible for data transfer
• A set of DTO objects has to be created and configured for the desired indexability and searchability
• A mapping between current domain objects and DTOs will have to be created. It has to function both ways
• A new base repository that connects to Redis in a new way will have to be created
• Existing repositories will have to be modified to utilise a new base repository

Ad.3 On top of the data migration, the plan for upgrade and migration of the existing Middleware instances has to be prepared to allow the least possible overhead for the end-users.

[adrianLIrobotics]

I am moving this to the "In progress column" as development has started for this one :)