microstream-one / microstream

High-Performance Java-Native-Persistence. Store and load any Java Object Graph or Subgraphs partially, Relieved of Heavy-weight JPA. Microsecond Response Time. Ultra-High Throughput. Minimum of Latencies. Create Ultra-Fast In-Memory Database Applications & Microservices.

---

MicroStream is now EclipseStore

The MicroStream project has been moved to the Eclipse Foundation.

Active development will continue there.

It has been split into two projects:

Eclipse Serializer

EclipseStore

MicroStream will receive security patches until the end of 2024.

However, we recommend migrating to EclipseStore to take advantage of the new features and support for newer JDKs.

---

High-Performance Java-Native-Persistence

Store and load any Java Object Graph or Subgraphs partially, Relieved of Heavy-weight JPA. Microsecond Response Time. Ultra-High Throughput. Minimum of Latencies. Create Ultra-Fast In-Memory Database Applications & Microservices.

License

MicroStream is available under Eclipse Public License - v 2.0.

Documentation

• Reference manual
• Get started
• Changelog
• Javadocs
• Examples
• Demo project

Build

To build MicroStream you need Java 11 and Maven.

Just run

mvn install

Contribute

If you want to contribute to MicroStream, please read our guidelines.

Links

• Website: https://microstream.one
• Twitter: @microstreamOne
• Stackoverflow: https://stackoverflow.com/tags/microstream

microstream's Issues

Entity transfer bug causes inefficient file cleanup

Once a file is deemed too space-inefficient, its remaining live data is incrementally transferred to the current head file until the file consists only of logical "gaps" and thus can be safely deleted. This transfer is supposed to (and did in the past) scan for chains of consecutive live data entities and then copy the whole chain as one block for best efficiency.

However, the current algorithm in StorageFileManager$Default#transferOneChainToHeadFile obviously has a bug that causes always only one entity to be transferred.

This has two consequences:
1.) Resolving a file takes an absurdly long amount of time if every incremental transfer action only transfers one single entity (out of tpyically tens of thousands or more per file).
2.) The transfer file becomes ridiculously unnecessary large since every transfer creates a new entry in there that occupies a few bytes. If transferring 50-100 bytes causes a 34 bytes entry, this means that the transactions file will grow to a gigantic size, containing 99.99% silly entries.

While this problem is not critical for correctness or stability, it is critrical for efficiency, especially occupied disk space.
The related logic is a very isolated, confined piece of code of around 20 Lines of Code.
The fix should be rather simple to do. Initial inspection showed that there were simply two concepts for the algorithm mixed together, resulting in an inappropriate loop condition.
Hence, it should urgently be fixed.

Confusing implicit order constraints when setting custom handlers

When setting a custom type handler and a custom eager storing field evaluator, the order in which both setters implicitely defines if the custom eager evaluator is used at all.
This is unnecessarily confusing and must be fixed.

The reason behind this is:

• Registering a custom type handler requires the custom type handler registry to be created.
• Creating the custom type handler registry includes creating and registering the native type handlers.
• Creating the native type handlers requires the type handler creator to be created.
• Creating the type handler creator requires the eager storing field evaluator.
• If no eager evaluator is set at that point, the default one is created and unchangeably used
• Any replacing in the foundation afterwards will have no effect on the type handler creator.

EmbeddedStorageManager storage = EmbeddedStorage.Foundation()
	.onConnectionFoundation(f ->
	{
		// implicitely crucial order, dangerous
		f.setReferenceFieldMandatoryEvaluator(new CustomEagerStoringFieldEvaluator());
		f.getCustomTypeHandlerRegistry().registerTypeHandler(new CustomBufferedImageHandler());
	})
;

Even adding a layer of indirection (custom eager storing field evaluator provider) would not fix that, since the type handler creator needs the actual field evaluator upon beginning to create type handlers.

Separating custom type handlers and native type handlers into two registries would ruin the possibility to optionally replace a native type handlers with a custom one in case one wants to use one's own type handling logic instead of the default one.

But there is another, pretty simply solution:

On the foundation level, all custom type handlers are collected in a separate,intermediate type handler map without even creating the actual custom type handler registry.
When the actual custom type handler registry is created, then all native handlers are registered (same as now), but then all the custom type handlers from the intermediate custom type handler map are registered, still potentially replacing any default native type handlers.

Adding that map is a trivial change and thus done right away.

The only danger with this is, that there ist still a "getCustomTypeHandlerRegistry()" providing logic getter that will create and return the internal registry and nothing in the API prevents a user from calling that "too early" instead of the unproblematic "customTypeHandlers()" property accessor getter.
This will have to be covered by the method's JavaDoc.

Incompatibility with jigsaw-configured project

Reported by CK:
When trying to use MicroStream with a simple JavaFX project based on JDK 12 and a jigsaw configuration file, there are class loading errors that cannot be circumvented.

Ticket J561: Thread names with consistent prefix

A suggestion via support ticket was to let all threads created by MicroStream start with a consistent Prefix so that they are easily recognizable and sorted together in monitoring tools.

This is very easily doable by adding a StorageThreadNameProvider which gets passed a Thread provider instance and its suggested thread name. The name provider can then add a prefix or execute any arbitraty other logic.

Communication dynamic type analysis

So far, a mechanism is still missing to create the complete definitive TypeDictionary for using MicroStream as a means for communication.

Currently, in the proof-of-concept example, the TypeDictionary is defined manually via an API call (ComBinary.Foundation().registerEntityTypes(A.class, B.class, ... );).
This works, but is not a practical solution. No one will manually compile a complete definitive list of all required entity types, collection types, enum types, nested whatsoever classes that will ever be encountered by the application's communication logic.

The database usage has a much simpler situation when it comes to defninig relevant (allowed) entity types:
The application is the only participant in the process of persisting and restoring data. Whatever new classes it encounteres during persisting instances, it can be 100% sure that it itself has all its classes. So it is perfectly sufficient to analyze types and extend the type dictionary dynamically while storing data.

In a communication situation, this is not the case. Every new class encountered poses the question or challenge, if the other peer will be able to understand it. Either in the form of a directly suitable class or in the form of a sufficient mapping to some other class. If the type dictionary was extended dynamically here, there might sooner or later be a class (type dictionary entry) that is not handleable by the communication peer. But then, it might already be too late, given that a lot of communication will have happened already until then. Application state, maybe even persistent database state, will have been changed.
To avoid this, it is preferable, if not mandatory, to synchronize the participating classes at the time the connection is established to make sure none of them will turn out to be a problem.

Sadly, there is no good way to collect the necessary classes in a sufficient, efficient and convenient way.

First of all, there is no "iterate all classes" functionality in Java. Classes are loaded from the class path on demand when the first occurance of a non existing class is encountered. So it simply cannot be done to iterate all classes of the application and let some logic (annotation-based or whatever) decide which ones are deemed suitable / desired / allowed / designed for communication.

There can be no "dry run" or such of the application logic that produces an entity graph containing ALL required classes. No one would take the time to develop it and even if someone did, there would be no guarantee it would be really complete.

One half working solution is to iterate all classes in all libraries on the class path and decide on their relevance for communication.
But this brings with it multiple problems:

• it would not be a good idea to just load ALL classes into the JVM just to do the check. Performance-wise, memory-wise and maybe even application-state-consistency-wise (static initializers doing unpredictable stuff etc.)
• so the decision logic would have to be based on the class (the .class file's) full qualified name itself. This might work well for some (e.g. all "com.my.app.entites.*" classes are allowed) but not at all for others (e.g. is com.third.party.library.internal.processing.Processor$Part$Entry relevant for communication or not? No one knows until it actually shows up in a reference...).
• the jigsaw module system might deny access to loading classes.
• dynamic class loading might happen after the type dictionary has been compiled. Then it would have to be dynamically extended yet again.

Some sophisticated tool, maybe in the form of an IDE plugin, would be conceivable, but that still would not really solve the problem of how to efficiently determine a complete definitive list of all relevant classes.

This situation leads to the following conclusion:
The option of having a preemptively complete definitive type dictionary is good (e.g. for security reasons), but it cannot be the only strategy.
Another strategy that dynamically extends a type dictionary, even during an ongoing communication, is required.
In the end, any potential problem (exception) that can arise from that is no different to any other potential exception: Somewhere down the line, something can go wrong and the application logic must be designed robust enough to prevent its internal state getting inconsistent by that.
For communication, that means: Yes, at some unknown point during communication and after sending an arbitrary amount of messages, the communication layer can encounter an incompatibility problem, causing a message to not being processable, maybe even terminating the whole connection.

As a consequence, there must be the following strategies regarding dynamic type analysis during communiction:

1.) None
All viable types are preemptively defined by a (host-side) type dictionary and there can be no extensions.
This might still be the best solution for many applications, e.g. by keeping and using a dynamically created type dictionary during testing.

2.) Host-Only
The host will dynamically extend its type dictionary as needed and notify the client(s) on every new type for a corresponding type synchronization.
The client is only allowed to use types defined in the type dictionary.

3.) All
All participants are allowed to extend the type dictionary upon encountering a previously not contained class.
More precisely, clients request extending the type dictionary by a specific class at the host, which in turn then does validation and analysis and then, on success, sends a type dictionary extension notification to the client, similar to # 2.

The notifying and requesting logic necessary for microstream-one/microstream-private#2 and microstream-one/microstream-private#3 are yet to be implemented.

[GitHub's parsing of "#" drives me crazy] # 3 can be perfectly safe for a trusted environment (e.g. a cluster running a distributed application), but for an untrusted environment (e.g. communicating with another server, a fat client or "app"), the type dictionary extension request would have to be followed by a lot of security validations, checking black lists, white lists and whatnot. This not trivial.

Consider the following example
(Inspired by this talk: https://www.youtube.com/watch?v=m1sH240pEfw)

A server might have a dynamic code interpreter on its class path.
The client requests the interpreter class to be accepted into the type dictionary.
A sent entity graph is special-tailored on the client side to contain an instance of that interpreter, an arbitrary piece of code in plain string form and a way to make that string get passed to the interpreter.
Result: arbitrary code execution.

Of course, a generic type analysis logic would not understand the intention or danger that comes with such a type dictionary extension. As long as the class is technically serializable, everything is fine.
Even some special logic patterns in the application's business logic (or one of its libraries) that make the special-tailored entity graph and code execution possible might not be design flaw in itself. Only when applying sophisticated ingeniuety that exploits certain characteristics of existing logic it all turns into being a security breach.

That might or might not be seen as a security flaw in the serialization layer itself, but surely it has to provide means to prevent that.
Not allowing "too abstract" classes in is one aspect, e.g. anonymous inner classes instances, lambdas, proxies, etc.
Not allowing known system parts (IO handling instances, ClassLoader, Thread, etc.) is another one.
Maintaining and checking black lists, white lists and such will be another aspect that is required.

StorageEntityInitializer does not use StorageFileReader

There is a dedicated type "StorageFileReader" that is meant to encapsulate all reading from Storage files. E.g. to add logging aspects or replace it by another reading algorithm.
However, it is only used in StorageFileManager#loadData (loading entity data from the files on a cache miss), but not during initialization (StorageEntityInitializer).
StorageEntityInitializer has a method #fillBuffer with redundant logic that it uses to initially read the storage data files.
This is not wrong, but for the StorageFileReader to "completely" make sense, it should be the one and only way to read data from data files.

(Not) handling synthetic classes

Synthetic classes do not have an explizit definition in the source code but are instead generated.
Examples:

• Anonymous inner classes (AIC)
• Anonymous-inner-classes-like redefinitions of enum instances

Since there is no class name, the compiler simply uses enumerations as their names in the form of
[enclosingType]$1
[enclosingType]$2
etc ...

That is perfecly sufficient for a singular runtime, where "the universe" starts anew every time and ends with the process and nothing before or after is ever relevant.

When using a database, that is no longer the case.
And when trying to link a persistent form data structure (type description) to a runtime type by its name, that enumation strategy becomes fatal with nothing more than rearranging some parts of sourcecode (e.g. moving AIC # 2 before AIC # 1, effectively switching their type names (!) ).

For enum subtypes, this can be reliably workarounded by replacing the number with the enum constant field's name. For details, see microstream-one/microstream-private#23.

The idea for AICs was to allow an optional mechanism of reliably identifying it by searching for an identifyer annotation on ANY of its members.
However, this is not possible since java.lang.Class offers no possibility to iterate its synthetic member classes. The method is simply not there.
Of course, the class names could be determined by using a brute force algorithm (checking for the existing of $1, $2, etc. until a ClassNotFoundException occurs), but that is a rather hacky and JDK-version-dependant way, it doesn't cover named classes inside of synthetic classes (yes, they are possible) and so on.

This led to the following consideration:

AICs may be technically classes, but that is only a (rather clumsy) workaround for a pure logic construct. Logic constructs should never be required to be persisted in an entity graph with properly identifyable type names.
Should the need arise to persist them (e.g. storing a collection referencing a Comparator with the Comparator being defined as an AIC), the AIC can be refactored into a proper class.
AIC offer nothing that proper classes cannot provide, they are merely a syntax optimization.

Therefore, the PersistenceTypeResolver (which handles Class <-> TypeName mapping and resolving) must be enhanced with checking all type names for enumerated parts ($1 etc) and throws an exception if one is found.

More precisely: the default logic must be enhanced. Should a user really want to store the application's AICs, the default logic can be replaced by a custom one, which automatically shifts the responsibility for the AIC's handling to the user.

BinaryHandlerGenericMap Error: java.lang.ClassCastException: java.util.HashMap cannot be cast to java.util.List

The code below ends with Exception:
java.lang.ClassCastException: java.util.HashMap cannot be cast to java.util.List

class BinaryHandlerGenericMapTest {

    @Test
    void binaryHandlerGenericMapTest() {
        HashMap<Integer, Integer> original = new HashMap<>();
        original.put(1, 100);
        HashMap<Integer, Integer> copy = new HashMap<>();

        EmbeddedStorageManager storage = EmbeddedStorage.Foundation()
                .onConnectionFoundation(f ->
                        f.registerCustomTypeHandler(
                                BinaryHandlerGenericMap.New(
                                        HashMap.class
                                )
                        ))
                .start(original);
        storage.storeRoot();
        storage.shutdown();

        storage = EmbeddedStorage.Foundation()
                .onConnectionFoundation(f ->
                        f.registerCustomTypeHandler(
                                BinaryHandlerGenericMap.New(
                                        HashMap.class
                                )
                        )
                )
                .start(copy);

        assertIterableEquals(original.entrySet(), copy.entrySet());
        assertIterableEquals(original.values(), copy.values());
        storage.shutdown();
    }
}

java.lang.ClassCastException: java.util.HashMap cannot be cast to java.util.List

	at one.microstream.java.util.AbstractBinaryHandlerList.update(AbstractBinaryHandlerList.java:1)
	at one.microstream.persistence.binary.internal.AbstractBinaryHandlerCustom.update(AbstractBinaryHandlerCustom.java:1)
	at one.microstream.persistence.binary.types.BinaryLoader$Default.buildInstances(BinaryLoader.java:346)
	at one.microstream.persistence.binary.types.BinaryLoader$Default.build(BinaryLoader.java:287)
	at one.microstream.persistence.binary.types.BinaryLoader$Default.get(BinaryLoader.java:701)
	at one.microstream.persistence.binary.types.BinaryLoader$Default.loadRoots(BinaryLoader.java:739)
	at one.microstream.storage.types.EmbeddedStorageManager$Default.loadExistingRoots(EmbeddedStorageManager.java:293)
	at one.microstream.storage.types.EmbeddedStorageManager$Default.initialize(EmbeddedStorageManager.java:314)
	at one.microstream.storage.types.EmbeddedStorageManager$Default.start(EmbeddedStorageManager.java:227)
	at one.microstream.storage.types.EmbeddedStorageManager$Default.start(EmbeddedStorageManager.java:1)
	at one.microstream.storage.types.EmbeddedStorageFoundation.start(EmbeddedStorageFoundation.java:198)
	at one.microstream.BinaryHandlerGenericMapTest.binaryHandlerGenericMapTest(BinaryHandlerGenericMapTest.java:29)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:675)
	at org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
	at org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:125)
	at org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:132)
	at org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:124)
	at org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestMethod(TimeoutExtension.java:74)
	at org.junit.jupiter.engine.execution.ExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(ExecutableInvoker.java:115)
	at org.junit.jupiter.engine.execution.ExecutableInvoker.lambda$invoke$0(ExecutableInvoker.java:105)
	at org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:104)
	at org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:62)
	at org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:43)
	at org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:35)
	at org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:104)
	at org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:98)
	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeTestMethod$6(TestMethodTestDescriptor.java:202)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeTestMethod(TestMethodTestDescriptor.java:198)
	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:135)
	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:69)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$5(NodeTestTask.java:135)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$7(NodeTestTask.java:125)
	at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:135)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:123)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:122)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:80)
	at java.util.ArrayList.forEach(ArrayList.java:1257)
	at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:38)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$5(NodeTestTask.java:139)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$7(NodeTestTask.java:125)
	at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:135)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:123)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:122)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:80)
	at java.util.ArrayList.forEach(ArrayList.java:1257)
	at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:38)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$5(NodeTestTask.java:139)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$7(NodeTestTask.java:125)
	at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:135)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:123)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:122)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:80)
	at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.submit(SameThreadHierarchicalTestExecutorService.java:32)
	at org.junit.platform.engine.support.hierarchical.HierarchicalTestExecutor.execute(HierarchicalTestExecutor.java:57)
	at org.junit.platform.engine.support.hierarchical.HierarchicalTestEngine.execute(HierarchicalTestEngine.java:51)
	at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:229)
	at org.junit.platform.launcher.core.DefaultLauncher.lambda$execute$6(DefaultLauncher.java:197)
	at org.junit.platform.launcher.core.DefaultLauncher.withInterceptedStreams(DefaultLauncher.java:211)
	at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:191)
	at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:128)
	at com.intellij.junit5.JUnit5IdeaTestRunner.startRunnerWithArgs(JUnit5IdeaTestRunner.java:69)
	at com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:47)
	at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:242)
	at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70)

Automatic MemoryAccessor setting

To prevent developer getting frustrating because of their ohn negligence, a mechanism should be implemented that recognizes the platform "enough" to decide which MemoryAccessor to set by default.

More specifically, the following two strategies should be used:

1.)
If a JVM is recognized that requires specific treatment, the MemoryAccessor in XMemory should be set accordingly.

2.)
An optional user argument (-D) that causes the Generic MemoryAccessor to be set.
However, it's questionable why a user who can't manage to write the single one line of code should be able to specify a user argument for the JVM. Nevertheless, it would be something that could be done after deployment.
On the other hand: without at least a properly working default instantiator, MicroStream is hardly usable. So it might be kind of useless after all.

FixedList save and reload throw java.lang.RuntimeException: The following root identifiers cannot be resolved: [defaultRoot]

I have a test:
I just create an Object with one class member: FixedList;
I try to save this object and Load it.
Due this i received this exception:
java.lang.RuntimeException: The following root identifiers cannot be resolved: [defaultRoot]

Full call stack is below:

class BinaryHandlerFixedListTest extends AbstractHandlerTest {

    @Test
    void binaryHandlerFixedListTest() {
        FixedListData original = new FixedListData().fillSampleData();
        FixedListData copy = new FixedListData();

        saveAndReload(original, copy);

        assertIterableEquals(original.getValue(), copy.getValue());

    }


    private class FixedListData implements BinaryHandlerTestData {

        private FixedList<Integer> value = new FixedList<>();

        @Override
        public FixedListData fillSampleData() {
            value = new FixedList<>(150, 100, 50, 158, 84546);
            return null;
        }

        public FixedList<Integer> getValue() {
            return value;
        }
    }
}

abstract class AbstractHandlerTest {

    private File workDir;
    private EmbeddedStorageManager storage;

    private String defineClassName() {
        return this.getClass().getSimpleName();
    }

    @BeforeEach
    @AfterEach
    void cleanStorage() throws IOException {
        if (null != storage && !storage.isShutdown()) {
            storage.shutdown();
        }
        workDir =  FileSystemHelper.getTempWorkDirectory(defineClassName());
        Assertions.assertNotNull(workDir, "Unable to find or create a working directory. Aborting!");
        FileUtils.deleteDirectory(workDir);
    }

    <T> T saveAndReload(T original, T loaded) {
        storage = startStorage(original);
        storage.storeRoot();
        storage.shutdown();

        storage = startStorage(loaded);
        return loaded;
    }

    private EmbeddedStorageManager startStorage(Object root) {
        return  EmbeddedStorage.start(root, workDir);
    }
}

java.lang.RuntimeException: The following root identifiers cannot be resolved: [defaultRoot]

at one.microstream.persistence.types.PersistenceRootResolver.resolveRootEntries(PersistenceRootResolver.java:58)
at one.microstream.persistence.types.BinaryHandlerPersistenceRootsDefault.update(BinaryHandlerPersistenceRootsDefault.java:184)
at one.microstream.persistence.types.BinaryHandlerPersistenceRootsDefault.update(BinaryHandlerPersistenceRootsDefault.java:1)
at one.microstream.persistence.binary.internal.AbstractBinaryHandlerCustom.update(AbstractBinaryHandlerCustom.java:1)
at one.microstream.persistence.binary.types.BinaryLoader$Default.buildInstances(BinaryLoader.java:346)
at one.microstream.persistence.binary.types.BinaryLoader$Default.build(BinaryLoader.java:287)
at one.microstream.persistence.binary.types.BinaryLoader$Default.get(BinaryLoader.java:701)
at one.microstream.persistence.binary.types.BinaryLoader$Default.loadRoots(BinaryLoader.java:739)
at one.microstream.storage.types.EmbeddedStorageManager$Default.loadExistingRoots(EmbeddedStorageManager.java:293)
at one.microstream.storage.types.EmbeddedStorageManager$Default.initialize(EmbeddedStorageManager.java:314)
at one.microstream.storage.types.EmbeddedStorageManager$Default.start(EmbeddedStorageManager.java:227)
at one.microstream.storage.types.EmbeddedStorageManager$Default.start(EmbeddedStorageManager.java:1)
at one.microstream.storage.types.EmbeddedStorage.createAndStartStorageManager(EmbeddedStorage.java:544)
at one.microstream.storage.types.EmbeddedStorage.start(EmbeddedStorage.java:428)
at one.microstream.AbstractHandlerTest.startStorage(AbstractHandlerTest.java:44)
at one.microstream.AbstractHandlerTest.saveAndReload(AbstractHandlerTest.java:39)
at one.microstream.BinaryHandlerFixedListTest.binaryHandlerFixedListTest(BinaryHandlerFixedListTest.java:15)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:675)
at org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
at org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:125)
at org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:132)
at org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:124)
at org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestMethod(TimeoutExtension.java:74)
at org.junit.jupiter.engine.execution.ExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(ExecutableInvoker.java:115)
at org.junit.jupiter.engine.execution.ExecutableInvoker.lambda$invoke$0(ExecutableInvoker.java:105)
at org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:104)
at org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:62)
at org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:43)
at org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:35)
at org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:104)
at org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:98)
at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeTestMethod$6(TestMethodTestDescriptor.java:202)
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeTestMethod(TestMethodTestDescriptor.java:198)
at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:135)
at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:69)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$5(NodeTestTask.java:135)
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$7(NodeTestTask.java:125)
at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:135)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:123)
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:122)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:80)
at java.util.ArrayList.forEach(ArrayList.java:1257)
at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:38)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$5(NodeTestTask.java:139)
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$7(NodeTestTask.java:125)
at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:135)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:123)
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:122)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:80)
at java.util.ArrayList.forEach(ArrayList.java:1257)
at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:38)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$5(NodeTestTask.java:139)
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$7(NodeTestTask.java:125)
at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:135)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:123)
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:122)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:80)
at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.submit(SameThreadHierarchicalTestExecutorService.java:32)
at org.junit.platform.engine.support.hierarchical.HierarchicalTestExecutor.execute(HierarchicalTestExecutor.java:57)
at org.junit.platform.engine.support.hierarchical.HierarchicalTestEngine.execute(HierarchicalTestEngine.java:51)
at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:229)
at org.junit.platform.launcher.core.DefaultLauncher.lambda$execute$6(DefaultLauncher.java:197)
at org.junit.platform.launcher.core.DefaultLauncher.withInterceptedStreams(DefaultLauncher.java:211)
at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:191)
at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:128)
at com.intellij.junit5.JUnit5IdeaTestRunner.startRunnerWithArgs(JUnit5IdeaTestRunner.java:69)
at com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:47)
at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:242)
at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70)
Suppressed: java.io.IOException: Unable to delete file: C:\Users\ZDENEK~1\AppData\Local\Temp\BinaryHandlerFixedListTest\channel_0\transactions_0.sft
	at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:2400)
	at org.apache.commons.io.FileUtils.cleanDirectory(FileUtils.java:1721)
	at org.apache.commons.io.FileUtils.deleteDirectory(FileUtils.java:1617)
	at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:2391)
	at org.apache.commons.io.FileUtils.cleanDirectory(FileUtils.java:1721)
	at org.apache.commons.io.FileUtils.deleteDirectory(FileUtils.java:1617)
	at one.microstream.AbstractHandlerTest.cleanStorage(AbstractHandlerTest.java:31)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:675)
	at org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
	at org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:125)
	at org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:132)
	at org.junit.jupiter.engine.extension.TimeoutExtension.interceptLifecycleMethod(TimeoutExtension.java:111)
	at org.junit.jupiter.engine.extension.TimeoutExtension.interceptAfterEachMethod(TimeoutExtension.java:95)
	at org.junit.jupiter.engine.execution.ExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(ExecutableInvoker.java:115)
	at org.junit.jupiter.engine.execution.ExecutableInvoker.lambda$invoke$0(ExecutableInvoker.java:105)
	at org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:104)
	at org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:62)
	at org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:43)
	at org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:35)
	at org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:104)
	at org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:98)
	at org.junit.jupiter.engine.descriptor.ClassBasedTestDescriptor.invokeMethodInExtensionContext(ClassBasedTestDescriptor.java:464)
	at org.junit.jupiter.engine.descriptor.ClassBasedTestDescriptor.lambda$synthesizeAfterEachMethodAdapter$17(ClassBasedTestDescriptor.java:454)
	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeAfterEachMethods$9(TestMethodTestDescriptor.java:228)
	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeAllAfterMethodsOrCallbacks$12(TestMethodTestDescriptor.java:256)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeAllAfterMethodsOrCallbacks$13(TestMethodTestDescriptor.java:256)
	at java.util.ArrayList.forEach(ArrayList.java:1257)
	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeAllAfterMethodsOrCallbacks(TestMethodTestDescriptor.java:255)
	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeAfterEachMethods(TestMethodTestDescriptor.java:226)
	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:139)
	... 41 more

Convenience factory method with conflicting logic

The convenience factory method
EmbeddedStorage#Foundation(File, StorageConfiguration.Builder)
has a conflicting logic that causes a lot of confusion for the user:

On the one hand, the configuration builder can contain a custom StorageFileProvider (e.g. to specify a custom storage channel file prefix), on the other hand, the method replaces the present StorageFileProvider with a new one based on passed File (base directory), discarding any file-related specifications the user might have done.

A check for conflicting specifications could be done, but the actual problem is that the method allows to set file-related values inconsistently in two places: inside the passed configuration and outside, as an exclusive directory value.

The method was created with convenience in mind, i.e. being able to specify just a storage directory and various other configurations via the builder:

EmbeddedStorage.Foundation(
	new File("path\to\my\storageDirectory"),
	Storage.ConfigurationBuilder()
		.set...
		.set...
);

However, the inconsistency and potential confusion is not worth the slightly more comfortable version compared to the perfectly clean alternative

EmbeddedStorage.Foundation(	
	Storage.ConfigurationBuilder()
		.setStorageFileProvider(Storage.FileProvider(new File("path\to\my\storageDirectory")),
		.set...
		.set...
);

Thus, the variant
EmbeddedStorage#Foundation(File, StorageConfiguration.Builder)
will be removed.

BinaryInstantiator is not configurable

A BinaryInstantiator type exists to modularize the instantiating logic for generically handled types.
By default, a "blank" memory instantiation is used that creates a new instance with all-zero values without calling any constructor.
This default logic can be replaced by an arbitrary logic, e.g. to call constructors for certain types.
Or at least at should be.
Because it isn't so far.
Currently, at some point there is a hardcoded reference to the default logic that cannot be replaced.
This has to be made dynamically by managing the BinaryInstantiator via the BinaryFoundation where the default logic is created if needed.

Stackoverflow in PersistenceTypeDictionaryManager

The method

PersistenceTypeDictionaryManager.Immutable#registerRuntimeTypeDefinition(PersistenceTypeDefinition) 

just calls itself, causing a stack overflow error.

Hide constructors of all BinaryTypeHandler implementations

For purely historic reasons, all BinaryTypeHandler implementations still had public constructors. This is bad if the BinaryTypeHandler concept should change in the future in a fashion that requires an additional mandatory reference.
The cleaner concept is to encapsulate actual constructors in a public static pseudo-constructor method. By project convention, that method is always named "New".

Option for non-fatal exceptionhandling

Currently, every exception encountered on the storage-level (loading and storing entity data etc.) causes the storage channel threads to cease working. This is intentional to immediately disable any more changes to an existing database once an inconsistency is detected.

However, microstream-one/microstream-private#35 introduces another usage mode of the storage level. While so far, only valid objectIds could ever be requested and any not resolvable objectId must be a fatal inconsistency, a "viewer" should always be seen as a harmless querying tool and should NEVER have a fatal effect on the actual database.

Example:
The web interface user changes the objectId requested via the URL manually to a non-sensical value. Currently, this causes the storage to shut down "in panic" while actually, it should only return a "not found"-like response.

Even if a web service tool should issue writing tasks in the future, a problem encountered while executing them should only cause a rollback of any changes (already implemented), but then not a shutdown in panic, but a continued processing.

Replacing the StorageExceptionHandler would work, but wouldn't be a good solution.
A) Because it would require the production storage logic to be modified just to allow the usage of a viewer
B) It would suppress actually fatal exceptions in production mode, too.

This means that tasks must be issueable with a kind of "severity" metainformation. Decided by the StorageConnection or the PersistenceManager instance.
All production code would keep issueing high-severity tasks, while a simple viewer would always just issue tasks with ignorable severity.
Of course, all state must still remain concistent, i.e. a failed write must still cause a correct rollback.

Requirement for actual technical deletion

Deletion in MicroStream works in a very indirect, typical garbage-collected-graph way:
There is no actual deletion, only instances becoming unreachable in the graph. This is enough to make them unaccessible for the application's "normal" business logic, i.e. "logically deleted".
At some (undefined) point in the future, the house keeping (garbage collector plus file cleanup) will deem the byte sequences representing the unreachable instances to be no longer needed and will eventually delete the file they are contained in. This is the only point where actual deletion of data occurs.

However, there might be requirements for an actual, guaranteed, technical deletion of data.
Be it in the form of laws forcing to actually delete data instead of just making them "normally" unreachable or in the form of some outdated corporate secrets or whatever that must be made absolutely unstealable by actual technical deletion.

A simple way to force actual deletion would be to make an instance unreachable (remove the last reference to it) and then call the housekeeping mechanisms with arguments specifying a "total" cleanup.
This works, but it has to be called explicitely and it might take a long time, depending on the size of the database and how many "logical gaps" there are since the last total cleanup. If such a total cleanup was executed on a regular basis (say every few minutes), it might only take a couple of seconds or split seconds even for arbitrarily huge databases.

Apart from that, such a requirement raises some follow-up questions:
1.) What about backups?
There is no point in guaranteeing actual deletion in the live database if backups are allowed to remain untouched. That would be a pure exercise in futility.
2.) What about hard drives
Hard drives use a very similar technical solution: data is not deleted instantly, but only made unreachable and will just be overwritten "eventually".
So, again: if the live database is cleaned and all the backups are cleaned, but the hard drives still hold the data, accessible with just the push of a button in a simple tools, then what's the point?

If the requirements only say to make data "unusable / unqueryable / unreachable" without special access and/or knowledge, then the normal "unreachability mechanism" is already sufficient.

Nevertheless, if push comes to shove, it would be conceivable to implement the following mechanism:
For a given instance, represented by its objectId, a deletion logic iterates over all database files and for every occurance of an instance with that objectId (older versions of it), all bytes between the end of the record header and the end of the record are zeroed out.
This is "as hard" a delete as it can get on the software level, leaving only the hardware level aspect.

In short:

• An actual, technical deletion of data is already possible, but potentially inefficient.
• A more efficient way does not yet exist, but would be easily implementable.
• Such a requirement raises a lot of follow-up questions that must be answered before it actually makes practical sense in the first place.

Headfile Cleanup should wait for max file size

If the head file is still pretty tiny (e.g. 2 kB with a 8 MB max file size) but already contains a lot of logical gaps (e.g. the same collection is stored multiple times), it is currently aggressively resolved into a new head file.
This generates an unnecessary large amount of tiny head files.

It would be smarter to let the head file reach its max file size, first, before resolving it.
But then again, that was the point of not cleaning up head files in the first place.
Maybe that has to be changed back to not cleanup up head files or the condition for head files has to be made smarter or something like that.

Not remotely critical, but a little annoying.

Not handling Proxy classes

Similar to synthetic classes (see Issue #3), instances of proxy classes cannot be handled.
On the one hand because their class names are generically enumerated, making it next to impossible to correctly resolve them. On the other hand because a syntheticly created type like that has no business in a proper persistable entity graph.

BigDecimal as root does not load its value

This test failed:

   @Test
    void bigDecimalTest() {
        BigDecimal type = new BigDecimal(456);
        BigDecimal loaded = new BigDecimal(0);
        saveAndReload(type, loaded);
        assertEquals(type, loaded, "BigDecimal");
    }

org.opentest4j.AssertionFailedError: BigDecimal ==> 
Expected :456
Actual   :0

I just create a BigDecimal with value 456. This object is saved as root and then loaded. After load is the value 0.

Other codes:

    <T> T saveAndReload(T original, T loaded) {
        storage = startStorage(original);
        storage.storeRoot();
        storage.shutdown();

        storage = startStorage(loaded);
        return loaded;
    }

    private EmbeddedStorageManager startStorage(Object root) {
        return  EmbeddedStorage.start(root, workDir);
    }

Converter for Lazy<->?

In addition to the primitive and wrapper value converters, one of the most common cases for refactorings will probably be the change to and from a Lazy reference.

So BinaryValueSetters for

• Lazy->?
• ?->Lazy

are necessary.

Monitoring Tool - Renderer for value types

Best way for value types is probably to display a String representation of their values directly.
E.g. for localized date and time values.
A registry of some kind should enable the developer to do exactly that.

Ticket J192: Abstract enums handled incorrectly

Yes, there are abstract enums, even though enums can never be abstract.
One more episode in the "enum crazy shit" story.

Found by a MicroStream user and reported via support ticket.

An enum Type can declare an abstract method IF it has subclass types and all of them implement the abstract method. This turns the enum type into an abstract class, even though it is not abstract.
Since I didn't know that until the support ticket, I did not cover that case.

The bug is relatively trivial:
In PersistenceTypeHandlerCreator#Abstract#createTypeHandler,
the check for abstract types comes before the check for enum types.
So an abstract enum type (which is not abstract :P), will be handled as an abstract type instead of an enum type. Hence the exception when using that crazy case with Microstream.

The fix is to shift the enum type check before the abstract type check.
The concept of crazy shit enum subclasses themselves is already covered.

Add Spring repository function

In Spring Data exists repository concept:
https://docs.spring.io/spring-data/jpa/docs/1.5.0.RELEASE/reference/html/jpa.repositories.html

It allow me to generate Query in DB only just writing one method in one interface.
Some functions are implemented by default. This is very comfort to use for developers.

Sample from Doku: Spring repo ref

public interface PersonRepository extends Repository<User, Long> {

  List<Person> findByEmailAddressAndLastname(EmailAddress emailAddress, String lastname);

  // Enables the distinct flag for the query
  List<Person> findDistinctPeopleByLastnameOrFirstname(String lastname, String firstname);
  List<Person> findPeopleDistinctByLastnameOrFirstname(String lastname, String firstname);

  // Enabling ignoring case for an individual property
  List<Person> findByLastnameIgnoreCase(String lastname);
  // Enabling ignoring case for all suitable properties
  List<Person> findByLastnameAndFirstnameAllIgnoreCase(String lastname, String firstname);

  // Enabling static ORDER BY for a query
  List<Person> findByLastnameOrderByFirstnameAsc(String lastname);
  List<Person> findByLastnameOrderByFirstnameDesc(String lastname);
}

Link to Spring source code JPA Repo

Monitoring Tool - Paging for huge data sets

For large collections or other huge data sets, a paging functionality at the front-end side is needed.

Handle changing class definitions at runtime

Developing support technologies like JRebel (#26) or Quarkus (Ticket #J505) use "hot code replacement", i.e. change classes of a live Java process at runtime.

This causes MicroStream to fail since the metadata derived from the handled classes during initialization do no longer match the classes.

This is no problem at all for any production usage, since classes don't get changed there during runtime.
However, it does cause confusion for developers using MicroStream with a hot code replacement technology. They assume MicroStream handles those cases (or probably that it makes no difference ) and wonder why they get an error.

The solution would be to provide some kind of Legacy Type Mapping at runtime.
Since LTM is already implemented, this shouldn't be too hard to do, albeit the usual rule about the devil being in the details should apply here, too. E.g. too many changes might confuse the mapping logic. Changes during the execution of a binary operation (store/load) might ruin the result, etc.

Such a solution would also require to have access to some kind of "class change callback". Preferably a central one provided by the JVM directly that every hot code replacement technology MUST use because there is no other way to do it.
Apart from that, every hot code replacement technology would require its own MicroStream compatibility plugin that hooks the runtime LTM logic up with the technology's mechanism.
That would be ... messy.

Generic Memory Access BufferRegistry Error

The assumedly unreachable Error case in BufferRegistry#determineFreeIndex has, of course, been reached during tests on Android.

NPE in StorageLockedFile#Default<init> during import

When using StorageConnection#importFiles a NPE is thrown in StorageLockedFile init.

Exception in thread "main" java.lang.NullPointerException
	at one.microstream.storage.types.StorageLockedFile$Default.<init>(StorageLockedFile.java:131)
	at one.microstream.storage.types.StorageInventoryFile$Default.<init>(StorageInventoryFile.java:55)
	at one.microstream.storage.types.StorageRequestTaskImportData$SourceFileSlice.<init>(StorageRequestTaskImportData.java:460)
	at one.microstream.storage.types.StorageRequestTaskImportData$Default.<init>(StorageRequestTaskImportData.java:70)
	at one.microstream.storage.types.StorageRequestTaskCreator$Default.createImportFromFilesTask(StorageRequestTaskCreator.java:238)
	at one.microstream.storage.types.StorageTaskBroker$Default.enqueueImportFromFilesTask(StorageTaskBroker.java:309)
	at one.microstream.storage.types.StorageRequestAcceptor$Default.importFiles(StorageRequestAcceptor.java:240)
	at one.microstream.storage.types.StorageConnection$Default.importFiles(StorageConnection.java:424)

java.util.Locale persisted erroneously

Reloading a persited java.util.Locale instance produces an errnous instance.
The reloaded instance is not initialized correctly.
The member "BaseLocale baseLocale" is null.

The member baseLocale is null because is declared transient, correctly causing it not to be stored.

It seems that this class needs a custom store and load handling.
The the Java default serialization stores fields of the transient member "baseLocale" and
does some initializing during deserialization

BinaryHandlerGenericQueue throw Error: java.lang.RuntimeException: Type Definition mismatch: 56:java.util.PriorityQueue

Code below produce the Error below:

class BinaryHandlerGenericQueueTest {

    @Test
    void binaryHandlerGenericQueueTest() {
        PriorityQueue<Integer> original = new PriorityQueue<>();
        original.add(100);
        PriorityQueue<Integer> copy = new PriorityQueue<>();

        EmbeddedStorageManager storage = EmbeddedStorage.Foundation()
                .onConnectionFoundation(f ->
                        f.registerCustomTypeHandler(
                                BinaryHandlerGenericQueue.New(
                                        PriorityQueue.class
                                )
                        ))
                .start(original);
        storage.storeRoot();
        storage.shutdown();

        storage = EmbeddedStorage.Foundation()
                .onConnectionFoundation(f ->
                        f.registerCustomTypeHandler(
                                BinaryHandlerGenericQueue.New(
                                        PriorityQueue.class
                                )
                        )
                )
                .start(copy);

        assertIterableEquals(original, copy);
        storage.shutdown();
    }
}

java.lang.RuntimeException: Type Definition mismatch: 56:java.util.PriorityQueue

	at one.microstream.persistence.types.PersistenceTypeDictionaryManager.validateTypeDefinition(PersistenceTypeDictionaryManager.java:38)
	at one.microstream.persistence.types.PersistenceTypeDictionaryManager$Abstract.validateTypeDefinition(PersistenceTypeDictionaryManager.java:152)
	at one.microstream.persistence.types.PersistenceTypeDictionaryManager$Abstract.registerRuntimeTypeDefinition(PersistenceTypeDictionaryManager.java:131)
	at one.microstream.persistence.types.PersistenceTypeDictionaryManager$Exporting.registerRuntimeTypeDefinition(PersistenceTypeDictionaryManager.java:284)
	at one.microstream.persistence.types.PersistenceTypeHandlerManager$Default.unvalidatedRegisterTypeHandler(PersistenceTypeHandlerManager.java:562)
	at one.microstream.persistence.types.PersistenceTypeHandlerManager$Default.initialRegisterTypeHandlers(PersistenceTypeHandlerManager.java:578)
	at one.microstream.persistence.types.PersistenceTypeHandlerManager$Default.internalInitialize(PersistenceTypeHandlerManager.java:836)
	at one.microstream.persistence.types.PersistenceTypeHandlerManager$Default.initialize(PersistenceTypeHandlerManager.java:811)
	at one.microstream.storage.types.EmbeddedStorageFoundation$Default.createEmbeddedStorageManager(EmbeddedStorageFoundation.java:587)
	at one.microstream.storage.types.EmbeddedStorageFoundation.createEmbeddedStorageManager(EmbeddedStorageFoundation.java:125)
	at one.microstream.storage.types.EmbeddedStorageFoundation.start(EmbeddedStorageFoundation.java:197)
	at one.microstream.BinaryHandlerGenericQueueTest.binaryHandlerGenericQueueTest(BinaryHandlerGenericQueueTest.java:29)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:675)
	at org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
	at org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:125)
	at org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:132)
	at org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:124)
	at org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestMethod(TimeoutExtension.java:74)
	at org.junit.jupiter.engine.execution.ExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(ExecutableInvoker.java:115)
	at org.junit.jupiter.engine.execution.ExecutableInvoker.lambda$invoke$0(ExecutableInvoker.java:105)
	at org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:104)
	at org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:62)
	at org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:43)
	at org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:35)
	at org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:104)
	at org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:98)
	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeTestMethod$6(TestMethodTestDescriptor.java:202)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeTestMethod(TestMethodTestDescriptor.java:198)
	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:135)
	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:69)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$5(NodeTestTask.java:135)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$7(NodeTestTask.java:125)
	at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:135)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:123)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:122)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:80)
	at java.util.ArrayList.forEach(ArrayList.java:1257)
	at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:38)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$5(NodeTestTask.java:139)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$7(NodeTestTask.java:125)
	at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:135)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:123)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:122)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:80)
	at java.util.ArrayList.forEach(ArrayList.java:1257)
	at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:38)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$5(NodeTestTask.java:139)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$7(NodeTestTask.java:125)
	at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:135)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:123)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:122)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:80)
	at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.submit(SameThreadHierarchicalTestExecutorService.java:32)
	at org.junit.platform.engine.support.hierarchical.HierarchicalTestExecutor.execute(HierarchicalTestExecutor.java:57)
	at org.junit.platform.engine.support.hierarchical.HierarchicalTestEngine.execute(HierarchicalTestEngine.java:51)
	at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:229)
	at org.junit.platform.launcher.core.DefaultLauncher.lambda$execute$6(DefaultLauncher.java:197)
	at org.junit.platform.launcher.core.DefaultLauncher.withInterceptedStreams(DefaultLauncher.java:211)
	at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:191)
	at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:128)
	at com.intellij.junit5.JUnit5IdeaTestRunner.startRunnerWithArgs(JUnit5IdeaTestRunner.java:69)
	at com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:47)
	at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:242)
	at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70)

Unstored Lazy references are clearable

A Lazy reference/instance that has not been stored, yet, can still be cleared by manually calling Lazy#clear, resulting in the data referenced by it being lost forever if the lazy reference was the last/only reference referencing it.
The automatic timeout-based clearing done via the LazyReferenceManager does not do that (by checking for unstored Lazy references), but the explicitly callable method Lazy#clear lacks such a check.

Storage remains in inconsistent state after exception

The problem in #32 was more specifically:

• The storage instance is initialized
• This opens and locks files
• An exception occurs inside the initialization
• The thread aborts before initialization is completed, marking the storage as running

Now the storage instance is in an inconsistent state:
It has file locks, but it can't be shut down since it is not marked running and calling shutdown() just returns.

In generall, exceptions may not leave the storage instance in an inconsistent state.
In this case, an exception during initialization must do a proper cleanup (close all files, reset modified state, etc.)

This is clearly a bug, albeit not a critical one since if the exception causing the inconsistency is fixed/prevented in the first place, there will be no inconsistency at all.
This is some kind of a "secondary" bug.

Ticket J213: Enum not correctly loaded

Bug report from support ticket : #j213

The reported bug is reproduceable using Microstream 02.00.01-MS-GA-SNAPSHOT.

possibly related to: #28

Summary:
During the enum loading in the reported ticket the enum's field "farbe" is overwritten by the enums name.

Output form the tickets code is :
Gebäude: HAUS
Farbe: HAUS
Farbe: null

Expected is:
Gebäude: HAUS
Farbe: ROT
Farbe: ROT

Analysis:
BinaryHandlerGenericEnum.Update(..) calls AbstractBinaryHandlerReflective.Update(..). Here bytes.updateFixedSize(..) is called to update the “farbe” reference member.
Binary.updateFixedSize(..) uses the loadItemEntityContentAddress(..) to determine the source address. This call returns the address of the enumerations name, which is the first content element.

Maybe a dummy setter could be added for the "name" field to take It's offset into account.

Fehler bei Verwendung von Enum in Enum
Hallo zusammen,

sorry, wenn ich direkt noch ein Ticket zum Thema Enum aufbaue. Ich habe schon seit ein paar
Wochen ein Mysterium, das ich jetzt erst lösen und mit Microstream in Verbindung bringen konnte.

Angenommen ich baue mir ein simples Enum auf:

public enum Farbe {
ROT, BLAU;
}

Dies verwende ich nun in einem anderen Enum:

public enum Gebaeude {
HAUS(Farbe.ROT), KIRCHE(Farbe.BLAU);

public Farbe farbe;

private Gebaeude(Farbe farbe) {
this.farbe = farbe;
}
}

(Mein eigentlicher Code ist deutlich komplexer und fachlich anders, ich habe es nur zur Fehlerbeschreibung vereinfacht.)

Nun verwende ich das zweite Enum direkt in meinem Datenobjekt:

public class Data {
public Gebaeude gebaeude;
}

Schließlich erstelle ich einen neuen Storage:

Data root = new Data();
EmbeddedStorageManager storage = EmbeddedStorage.start(root);
root.gebaeude = Gebaeude.HAUS;
storage.store(root);

Wenn ich diesen Storage nun auslese, wird es seltsam - um nicht zu sagen fehlerhaft:

Data root = new Data();
EmbeddedStorageManager storage = EmbeddedStorage.start(root);

  // Ausgabe HAUS - Hier ist noch alles okay.
  System.out.println("Gebäude: " + root.gebaeude);

  // Ausgabe HAUS - Hier wird es seltsam.
  System.out.println("Farbe:   " + root.gebaeude.farbe);

  // Ausgabe null - Das ist absolut unverständlich.
  System.out.println("Farbe:   " + root.gebaeude.farbe.name());

Ich verwende übrigens folgende Dependencies:

one.microstream storage.embedded 02.00.00-MS-GA

Vielleicht liegt der Fehler auch bei mir? Dann vorab Entschuldigung für das Missverständnis.

Schöne Grüße und Danke für's Draufschauen,
christian packenius.

Value Initializer similar to Value Translator

To translate values from a legacy type instance to a current type instance, value translators with arbitrary logic can be defined. However, for newly added fields, no comparable logic can be defined. Currently, they are forced to remain 0 (0 / 0.0 / false / null), even if they are final fields.
One option to initialize their values is to use custom instantiators (see Issue #8).
However, that is not a full-fledged replacement:

• Constructors might contain logic that shall not be executed by the process of restoring previously stored instances.
• Instances might already exist but shall be updated from the persisted record.

Hence, a mechanism of "ValueInitializer"s similar to the existing "ValueTranslator"s is required.

Android memory accessing specifics implementation

For full support of Android, to tiny implementations are required to handle memory access completely:

1.)
A DefaultInstantiator implementation similar to JdkInstantiatorBlank is required.
On android, Unsafe supports # allocateInstance(), so the implementation is virtually identical to that of the JDK.
See https://android.googlesource.com/platform/libcore/+/9edf43dfcc35c761d97eb9156ac4254152ddbc55/libdvm/src/main/java/sun/misc/Unsafe.java#349

2.)
Since the JDK-specific "Cleaner" hack of JdkDirectBufferDeallocator does not work on andoid, another way must be implemented. Luckily, the android developers - in contrast to so other JVM developers ... - implemented a method DirectByteBuffer#free.
However, it is not public, so it must be called via reflection.
Doing this is preferable, anyway, since it allows the android-specific code to be a part of the main project.
See https://android.googlesource.com/platform/libcore/+/f33eae7e84eb6d3b0f4e86b59605bb3de73009f3/luni/src/main/java/java/nio/DirectByteBuffer.java#280

Communication value validation

When using MicroStream as a database solution, the use case is virtually completely that if internally managed data. Meaning all parts of the process can be trusted. Hence, no encyption or consistency/sanity validation of loaded data is required. At least not by default.

For "Serialization" (actually more like "communication"), this is another story.
Encryption (via a modular logic hook) is needed. So is data validation.

Simple example:
A field whose value can never be null (an instance is created directly in the constructor / at field definition) can be easily transferred as a null value from an external source.

So there must be a way to validate data if required/desired.

It is already possible to implement a custom TypeHandler that transforms instances to and from a persistent form with an arbitrary concept and logic, including any and all validation logic one might think of.
However, implementing such a TypeHandler is rather verbose and required perfectly careful explicit binary offset handling.
microstream-one/microstream-private#88 would ease the explicit binary offset handling part, but the verbosity of implementing a whole TypeHandler just to enable validation is still to high.

The ideal (and technically possible) solution would be to have an annotation like the following:

@Validate(Validations::isCapitalizedString)
private String someStringField;

Method references to static methods are a compile time constant and hence would be a possible annotation value.
However, annotations do not allow that by design.
Maybe they realize and fix that in Java 37 or so, but until they do, that ideal solution is not an option.

Enum instance implementing a Predicate would be possible, but that makes it rather verbose for developers to create validation logic and a compatible Annotation.

Defining a simplified checking logic as a plain string (like min/max value etc) would be possible, but putting logic into plain Strings is always one of the worst ideas (refactorings will sooner or later break it and the compiler can't help) and the "simple" logic will be pretty quickly exploded into a validation language.
Sounds good at first, but is pretty dumb in the long run.

The best solution currently conceivable is the following:

A separate class (maybe a static nested class in the entity itself), called "Validations" or "Checks" or such (the exact name is arbitrary and up to the using developer) that contains validation methods with an equal name as the field whose values shall be validated.

Example:

public class Person
{
	@Validated(Check.class) // optional
	String firstname;

	@Validated // optional
	String lastname;
	
	
	
	public void setFirstname(final String firstname)
	{
		// setter uses centralized checking logic for validation
		this.firstname = X.validate(firstname, Check::firstname);
	}
	
	public void setLastname(final String lastname)
	{
		// setter uses centralized checking logic for validation
		this.lastname = X.validate(lastname, Check::lastname);
	}
	
	
	// centralizing validation logic for both class itself and external reconstruction solutions
	@Validations // optional
	static class Check
	{
		static boolean firstname(final String value)
		{
			// arbitrary logic here. Trivial null-check for simplicity's sake of the example
			return value != null;
		}
		
		static boolean lastname(final String value)
		{
			// arbitrary logic here. Trivial null-check for simplicity's sake of the example
			return value != null;
		}
	}
	
}

The validation logic gets centralized for both explicit internal use in the actual class and implicit use by external state reconstruction logic (aka Serialization / Persistence).
The annotations can be optional and handled by a modular analyzer.
Other possibilities could be to search for a nested static class with a centrally configured name (here "Check") and in there for a suitable method (parameter count and type, return type, name).
Annotations could also specifically define the name of the method to be used, however that's programming in plain strings, again.

Of course, the usual (naive) annotations like @NotNull, @UseGetter, @UseSetter (or maybe better @UseAccessControl) and the like could also be supported. And the Enum function approach, too.
Adding even simpler strategies as a variant is never a problem, but there has to be one solution that can cover "everything" (sanely conceivable in the specific topic's context). That would be the "Validations" class.

Side note:
This is also a nice example of OOP-designwise separations of concerns: The entity class itself should only be the data carrier, without validation logic being hardcoded intertwined. When the validation logic is modularized, it can be reused generically and implicitely by other logic.
In the end, this might be an even cleaner and better solution than the "ideal" solution at the beginning ...

JVM Access Violation in PlatformInternals#internalDeallocateDirectBufferByThunk

With the current release candidate, JM gets an access violation in PlatformInternals#internalDeallocateDirectBufferByThunk when the storage is initialized/loaded.

The used JDK version is identical to the one that code was written with.
All the JDK internals used by that method are present.

If a no-op DirectBufferDeallocator is set to override the behavior, the JVM just hangs at that point, waiting forever for a notification despite no wait/notify or synchronized or any thread controlling code is used.

The PlatformInternals code is written extremely carefully and error-safe:

• offsets to internal fields are retrieved. If something fails there, an internal warning is logged
• the logged warnings are empty
• the method itself checks again every time if viable offsets are present, retrieved instances are not null, are of the assumed type, etc.. If not, nothing is done (as the explicit deallocation is not critical, but rather a fallback-fallback solution for very rare cases)

Nevertheless, there's that access violation and it is currently not in the slightest conceivable what the reason for it is.

Replace java.io.File with java.nio.file.Path

As a quick first step for microstream-one/microstream-private#49, all occurances of java.io.File could be replaced by java.nio.file.Path. Of course with adjusted logic and public API methods overloaded for java.io.File where necessary.

The idea ist that the newer Path API allows a level of abstraction where Paths can present not only local file system elements, but any data storing elements. The old java.io.File cannot do that.

Maybe this can be used to completely replace the IO layer abstraction.
Or at least shift it to a later point / lower priority when such an abstraction is needed to additional logic or performance optimizations.

Enum is not correct loaded from Db.

Following code produce the error:

Or Just checkout the project
https://bitbucket.org/microstreamone/microstream-test/
Find this Class, uncomment @test und launch as Junit Test, to produce the error.

class BinaryHandlerGenericEnum extends AbstractHandlerTest {
File workDir =  FileSystemHelper.getTempWorkDirectory(defineClassName());

    @Test
    void binaryHandlerGenericEnum() {

        final String TEXT = "Sometext";

        GenericEnumData original = GenericEnumData.THIRD;
        original.setOtherValue(TEXT);
        GenericEnumData copy = GenericEnumData.THIRD;
        saveAndShutdown(original);

        original.setOtherValue("somethingOthers");
        load(copy);

        assertEquals(TEXT, copy.getOtherValue());
    }


    private enum GenericEnumData {
        FIRST(1), SECOND(2), THIRD(3), FOURTH(4), FIVE(5);

        int value;
        String otherValue;

        GenericEnumData(int value) {
            this.value = value;
        }

        public String getOtherValue() {
            return otherValue;
        }

        public void setOtherValue(String otherValue) {
            this.otherValue = otherValue;
        }

        public int getValue() {
            return value;
        }

        public void setValue(int value) {
            this.value = value;
        }
    }


    <T> T saveAndShutdown(T original) {
        storage = startStorage(original);
        storage.storeRoot();
        storage.shutdown();
        return original;
    }

    <T> T load(T loaded) {
        storage = startStorage(loaded);
        return loaded;
    }

    private EmbeddedStorageManager startStorage(Object root) {
        return  EmbeddedStorage.start(root, workDir);
    }
}

org.opentest4j.AssertionFailedError: 
Expected :Sometext
Actual   :THIRD

Backup thread ignores storage shutdown

When the storage is explicitely shutdown, an existing backup thread will just continue to work through its item queue. However, as a shutdown closes the StorageFiles' FileChannels, it will encounter a a corresponding exception. Not always (race condition), but fairly easily reproducible.

Of course, the backup thread (more precisely: the StorageBackupHandler implementation) has to consider a shutdown of the parent storage.
This will be a bit tricky: even checking for the shutdown state(s) might still end up in the same exception since there would still be a race condition. Probably the proper way of checking would be:

• Check if a source file is closed on every backup action.
• If so, check if the parent storage is shut down.
• If yes, just terminate. Otherwise, throw an exception.

That some enqueued backup items won't be a problem since the backup content is synchronized with the actual storage content during initialization and any missing backups are then performed.
It might only be relevant as a shutdown option: stop all activities but wait with the actual shutdown until the backup item queue is empty. But this cannot be the default/mandatory behavior since waiting for those backups might take considerable time that might not be worth it.
Or if the requirements become stricter in terms that without an exception, the backup has to be guaranteed to the last byte, then it might become mandatory.

As always, the best strategy is to allow both and make it configurable. The only question then will be which behavior to be set as the default, but that is technically trivial.

Inconsistent Use of Refactoring Mapping

The user-defineable refactoring mapping is intended to specify new identifiers for types, fields, constants for vanished old ones (e.g. because of type renaming).
However, it can currently also be used to map constant fields of a still existing class to compatible constant fields of another class.
This causes resolving conflicts that lead to the conflicint root entries being discarded by the root resolving logic.

Such a use has not been covered, yet, by the design of the refactoring possibilities.
It has to be decided if such an "existing -> existing" mapping shall be possible/allowed at all or if the mapping has to be validated to prevent such a use.

The current state of the processinglogic behind the refactoring mapping almost forces the following approach:
1.) Currently, such a mapping must be disallowed by recognizing it and throwing an exception.
2.) It should be discussed and decided if a "existing -> existing" mapping capability shall be introduced as an additional feature in the future, including the processing logic required to handle it correctly.

Potential Inconsistency in Root Resolving

BinaryHandlerPersistenceRootsDefault#update can potentially remove some root entries when resolving resolvableRoots to resolvedRoots. The int[] objectsIds contains the full list of objectIds, the table resolvedRoots only contains the reduced list. After the resolving (potential reduction), however, objectIds[] is used in combination with resolvedRoots and is associated per index.
This is means that every time any entry is removed, all subsequent entries get inconsistent TypeIds assigned.

This must be fixed by introducing a proper objectId->resolvedEntry table instead of "randomly" associating the objectIds array from before resolving with a potentially reduced resolving result.

Check different NIO FileSystemProviders

After #19 is done we can check if various file system providers are sufficiently implemented for our needs.

In memory file systems

• https://github.com/marschall/memoryfilesystem
• https://github.com/google/jimfs

Cloud drive adaptors

• https://github.com/Upplication/Amazon-S3-FileSystem-NIO2
• https://github.com/elerch/Amazon-S3-FileSystem-NIO2
• https://github.com/seqeralabs/nxf-gsfs
• https://github.com/nextflow-io/jdk7-dxfs

Replace switchByteOrder by MemoryAccessor

Before microstream-one/microstream-private#111, a "switchByteOrder" flag had to be passed around at a lot of places to handle translation between the two byte orders. Now, with the MemoryAccessor interface, it might be a good idea to pass or even use directly that instead of a primitive value that only makes sense through its name and context.
This could remove a lot of redundant code because a lot of low-level methods had to be written twice to account for a switched byte order.

BinaryValueFunctions are not specific enough

Storing a primitive value from an instance in the heap to the storing buffer and setting a primitive value from the loading buffer to an instance in the heap.

So far, every primitive value size (1, 2, 4, 8) has only 4 function for each category (store, storeReversed, set, setReversed and skip). Because for low level memory operations (Unsafe), this is sufficient and it keeps helping the file small (4+1 + 3+1 + 4+1 + 3+1 + 4 = 22 functions).

However, when using MemoryAccessGeneric, meaning Unsafe operations are replaced by DirectByteBuffer API and Reflection, this is no longer sufficient. Reflection setting- and getting-methods perform a rigid type check: a 1 byte boolean value cannot be read as a 1 byte byte value and so on.

This came up during a more intensive Android test.
(It's a little strange why this didn't come up on the very first test of MemoryAccessorGeneric because the storers are used already during initialization. But whatever ...)

The solution must be to create type-specific storers and setters instead of just byte-size-specific ones.

Meaning 9 storers, 7 reversed storers, 9 setters, 7 reverses setters. The skipping setters can remain at 4 because they just skip n bytes in the binary form data.
Making a total of 36 instead of 22. Not dramatically more, but then the primitive type handling is strictly correct, even for reflection.

SelfStoring is not called recursively

The "SelfStoring" method is only called on instances that are directly passed to a PersistentStoring instance (e.g. a Storer). If the storing process encounters an instance of a class implementing SelfStoring during the recursion, it's SelfStoring method is ignored and the instance is stored in the normal fashion.

The question is, if that is a bug that has to be fixed (assumedly yes) or if there is a good reason for this behavior. For example: the more this interface is supported, the more it becomes an equivalent /competitor for a custom type handler (e.g. why not a SelfLoading, too? And in next to no time, it's a secondary TypeHandler.). Also, SelfStoring kind of circumventing the TypeHandler logic created specifically for a particular class. (Doesn't make that SelfStoring a fundamental bug in the first place?)

So maybe the better choice might be to remove the SelfStoring altogether and refer to implementing custom type handlers and make them a little more convenient (e.g. microstream-one/microstream-private#88).

Option to create a full backup

The initial, straight forward, simple and not very efficient backup strategy was to simply copy all storage files to a target directory via a StorageTask. The WIP project contains a StorageBackupHelper class for that that contains a consolidated tiny API for what was initially a rather lengthy manual util code. To the best of my knowledge, the StorageBackupHelper class has never been used, however.

On a technical level, this type of backup creation has actually been completely replaced
by the continuous backup functionality (see StorageConfiguration#backupSetup or issue microstream-one/microstream-private#61).
Also see issue microstream-one/microstream-private#17.

However, it turned out that on a psychological level, customers still want a "full backup" functionality because they do not fully trust a continuous backup or it hasn't been properly explained to them or whatever the reason might be. The absence of problems with the continuous backup plays no role in this, they simply want a "backup everything now" command.
So instead of copying only new files as they get created and filled, there's still need for a variant
that copies ALL files, no matter how inefficient, redundant or unnecessary that is.

As a consequence, the class StorageBackupHelper must be tested and afterwards be moved to a productive project, maybe even in the EmbeddedStorageManager type directly. The details for this are yet to be determined.

BinaryHandlerGenericEnum#deriveEnumConstantMembers is oracle-specific

Tests on android showed that Class#getDeclaredFields does not include fields for enum constants there. As a consequence, BinaryHandlerGenericEnum#deriveEnumConstantMembers does not find the enum constants and causes a NPE later on.

Obviously, creating fields for enum constants is an implementation detail of a JVM.

Monitoring Tool - Expose webserver configuration

It is necessary to expose the webserver's configuration in the API. Possible use cases:

• Add authentication
• Change port

Add OSGi bundle headers to released jar file

In order to use MicroStream in OSGi environments it is necessary to ship the OSGi bundle headers within our jar files. This can be done with a maven plugin in the package phase of the build.

https://blog.osoco.de/2016/05/building-osgi-bundles-with-apache-maven/

Custom Initialization API does not release file locks

A very weird problem occured during test writing:

If an EmbeddedStoragemanager is created and started via EmbeddedStorage.start(), all file locks are released upon shutdown() as intended.
However, if it is created via EmbeddedStorage.Foundation().start(), they are kept.
This is especially weird since the convenience start() Method does exactely the same internally.

This has to be investigated in detail.