shopwarelabs / storage

Shopware document storage system

This project is a document storage system for Shopware 6. It allows you to store documents inside storage and query them via key or via different criteria. The goal of this system is to provide a read-optimized system for better scalability and performance and support different kinds of document storage engines like opensearch, mongodb, etc.

Feel free to contribute to this project or implement new storages. All implementations of storage should be tested in the same way, to ensure the same behavior independent of the chosen engine.

When implementing a new storage, please also make sure the storage can be setup locally.

The project is still under development and is not suitable for productive systems. As soon as we can ensure a stable version, this repository will be moved to the Shopware organization. This repository is then split into a mono repository and the different engines are split into many repositories. So it is possible to only need the required engine dependencies for each project

storage's Issues

mongodb: Key value storage

Add new mongodb storage for KeyValue implementation.

factory for criteria classes

Filters, sortings, etc should be created via static factory methods

generic: Clean exception handling for filter type validation. Provide path to the filter

opensearch: Full text search support

DTOs instead of arrays

it would be nice, if we could exchange as much arrays as possible with simple objects with typed properties. first thing I saw, were the sorting and filter arrays

Price fallback system

How to allow "fallback-field" logic like translations.

Would be nice if you can define a key from the context and it uses this as fallback logic.

Update schema tests

• Adding and removing properties
• Changing data type?

generic: Sorting tests

Tests for:

• Scalar fields
• Object fields
• Translated fields
• Object+translated fields
• Multi sorting
• Null values

Document encoding/decoding

• We want to support custom documents, but with doc blocks as properties
• Add support for nested objects

generic: Different total implementations

Allow also patterns like cursor / last-id

meilisearch: contains/prefix/suffix filter are not supported

typesense support

https://typesense.org/docs/26.0/api/search.html

generic: Tests for updating documents

opensearch: Missing aggregation support

Following aggregations are not supported by opensearch, and got pre-catched in the storage

• Min/Max/Avg/Sum agg on text and string fields
• Distinct/Count agg on text fields

generic: Each object (filter, sorting, aggregation, etc) should have a name

Sometimes it would be nice to check if a specific filter is set to a criteria.
It also helps consumers to remove this element from the criteria.
It also helps for error reporting

mysql aggregate support

mysql: Fulltext search support

Get mysql running

Get mysql running in unit test

opensearch aggregate support

Boost support + tests

generic: Get/mget support in all storages

max nesting depth config

Don't allow to deep nesting for "additional" fields on a schema (third party). Provide a config

mongodb: Fulltext search support

Naming

Current suggestions:

1. "ASS" - Abstract search storage
2. "ADS" - Abstract document storage
3. "DSE" - Document Search Engine
4. "SES" - Search Engine Storage

setup: mono repository splitting

Each directory inside /src should be an own repository that projects just need to include the different engines they need

Extension testing

Provide tests which ensure that the storages can be extended.

mongodb translated support

custom field scenario testing

How to have an unkown json blob.

meilisearch: does not support string comparison for LT/LTE/GT/GTE

mellisearch for search engine

title for each criteria

To identify the source of each criteria, i want to enforce setting a title

add static analyze

mysql translated support

setup: Docblock documentation

Documentation should be done via php docblocks

Also provide examples in the docblock which are unit tested

Add documentation CI

Test cases for nested object/object-list with translated fields

setup: Add howto in readme

Add how-to guide in the readme for local setups, including dynamo-db

Also add how-to guide regarding implementing new storages

generic: Storage setup support

Each storage should have a setup function which generates the required tables/collections/indices whatever

This function should also support to update the storage, if the schema changed.

generic: More aggregation tests

• With keys in criteria
• Multiple filter tests
• Multiple aggregations at the same time

fix phpstan error for max level

meilisearch umlauts support

Umlauts seems not working. Revert the \Meilisearch\MeilisearchSearchTest::match "Umlauts" case and make it running :)

generic: AND/OR/XOR/NOR test cases

Currently there are just tests for single filters. Operators are not tested yet

add CI

generic: Object field with null value filter

There are some tests in the FilterStorage test base which are disabled (object fields with null value)

Implement storage::clear

To remove all entries in the storage

Implement storage::destroy

To completly destroy the storage

more key value storages?

influx db

keydb

dragonflyDB

garnet

RocksDB

mongodb aggregate support

gerneric: Nested translated field tests (e.g. categories.name)

There are currently no test for translated fields inside objects or object-lists

meilisearch: aggregation support