koculu / zonetree Goto Github PK

Is your feature request related to a problem? Please describe.
ITransactionalZoneTree does not have a way to iterate the items.

Describe the solution you'd like
CreateIterator from IZoneTree supported on ITransactionalZoneTree.

Describe alternatives you've considered
I've considered using IZoneTree instead of ITransactionalZoneTree, however I require transaction support and it does not appear to be possible to have both IZoneTree and ITransactionalZoneTree in use at the same time.

Additional context

Testing X86 (NuGet 1.6.8) and getting this now:

System.PlatformNotSupportedException: Operation is not supported on this platform.
   at System.Runtime.Intrinsics.X86.Sse42.X64.Crc32(UInt64 crc, UInt64 data)
   at Tenray.ZoneTree.WAL.AsyncCompressedFileSystemWriteAheadLog`2.AppendLogEntry(Byte[] keyBytes, Byte[] valueBytes, Int64 opIndex)
   at Tenray.ZoneTree.WAL.AsyncCompressedFileSystemWriteAheadLog`2.ConsumeQueue()
   at Tenray.ZoneTree.WAL.AsyncCompressedFileSystemWriteAheadLog`2.StopWriter(Boolean consumeAll)
   at Tenray.ZoneTree.WAL.AsyncCompressedFileSystemWriteAheadLog`2.<MarkFrozen>b__37_0()

Could be this in Crc32Computer_SSE42_X86?

    public static uint Compute(uint crc, byte[] data)
    {
        var off = 0;
        var len = data.Length;
        while (len >= 4)
        {
-           crc = (uint)Sse42.X64.Crc32(crc, BitConverter.ToUInt32(data, off));
            off += 4;
            len -= 4;
        }

Originally posted by @markhewett in #46 (comment)

Is your feature request related to a problem? Please describe.
ZoneTree does not support ARM arch in this moment. Try to run it on M1 and get PlatformNotSupportedException in Crc32Computer :(

Describe the solution you'd like
Would be great if we can run it on ARM

Additional context
According to the unit tests, it is enough to fix the Crc32Computer, so if you need help, I can make PR in the next few days.

Describe the bug

When upserting any value with the string key consist of single symbol
reopening (dispose -> open or app restarting, no matter) failed with exception

Write ahead log with segment id = N is corrupted.

with inner exception

System.IO.InvalidDataException: Deserilization of log entry failed. Index=0
 ---> System.IndexOutOfRangeException: Index was outside the bounds of the array.
   at Tenray.ZoneTree.Serializers.Utf8StringSerializer.Deserialize(Byte[] bytes)
   at Tenray.ZoneTree.WAL.AsyncCompressedFileSystemWriteAheadLog`2.DeserializeLogEntry(LogEntry& logEntry)
   at Tenray.ZoneTree.WAL.WriteAheadLogEntryReader.ReadLogEntries[TKey,TValue,TLogEntry](ILogger logger, Stream stream, Boolean stopReadOnException, Boolean stopReadOnChecksumFailure, LogEntryRea
derDelegate`1 logEntryReader, LogEntryDeserializerDelegate`3 logEntryDeserializer, Boolean sortByOpIndexes)
   --- End of inner exception stack trace ---

Keys with 2 and more symbols works well.

To Reproduce

• Create simple console application
• Insert this sample into Program.cs

using var db = new ZoneTreeFactory<string, int>()
    .SetDataDirectory("./data")
    .OpenOrCreate();
    
db.Upsert("0", 123);

• Run it
• Then try run again, you should get exception

Expected behavior

Upserting such keys will not corrupt DB :)

please complete the following information:

• Operating System: Windows 11 x64
• ZoneTree Version: 1.5.2.0

In the README it says "hundreds of times faster than SQLite". But I could not see any benchmark for SQLite.

Describe the bug

namespace Tenray.ZoneTree.Serializers;

public sealed class Int64Serializer : ISerializer<long>
{
    public long Deserialize(byte[] bytes)
    {
        return BitConverter.ToInt32(bytes);
    }

    public byte[] Serialize(in long entry)
    {
        return BitConverter.GetBytes(entry);
    }
}

Expected behavior
Deserialize to int64

Describe the bug

ReverseIterator Seek method does not work correctly. It does not move the iterator to the location of the prefix. Instead it just iterate in reverse starting at the very end of the database.

To Reproduce
using var revIterator = zoneTree.CreateReverseIterator();
revIterator.Seek("myPrefix");
count = 0;
while (revIterator.Next())
{
var key = revIterator.CurrentKey;
var value = revIterator.CurrentValue;
Console.WriteLine($"{key} : {value}");
count++;
if (count > 10)
{
break;
}
}

Expected behavior
After running .Seek("myPrefix"), the ReverseIterator should start iterating from the location of the first key with the prefix.

please complete the following information:

• Operating System: Windows 10
• ZoneTree Version: 1.5.8

First, not sure if feature request is the right category, it's possible the feature already exist, and if not but it wouldn't be too complex to implement, I'm ready to lend an hand :)

I'm looking for a transaction log library I could use in my distributed gameserver project to internally store cluster related data and ZoneTree looked interesting, so I'm considering it. However to do that I'll need to be able to integrate the library in our log replication system. We use the raft algorithm, so we need to have some control on when entries are appended or committed to the log in order to synchronize logs between primaries and replica.

In particular we need to:

• Query the log for last appended and last applied log entry ids
• Intercept log entries to be appended for replication.
• Control when entries are committed.
• Manually append log entries.
• Overwrite/delete uncommitted entries.

Is it something that would be doable with ZoneTree ?

Thank you!

What kind of enhancement you would like to have? Please describe.
This is a performance improvement request.

Describe the enhancement you'd like
Crc32Computer checks CPU architecture for every byte. Reduce the unnecessary conditional CPU instructions.

Is your feature request related to a problem? Please describe.
At this time WriteAheadLogCorruptionException has property Dictionary<int, Exception> Exceptions, it is assumed that these are internal exceptions, but they are not rendering in any ways.
So it is a little bit hard to understand whats goes wrong.

https://github.com/koculu/ZoneTree/blob/main/src/ZoneTree/Exceptions/WriteAheadLogCorruptionException.cs

Describe the solution you'd like
It would be nice that inner exception messages includes into main.

Describe alternatives you've considered
Provide aggregate exception to a base class. For example:

public class ImprovedWriteAheadLogCorruptionException : ZoneTreeException
{
    public Dictionary<int, Exception> Exceptions { get; }

    public ImprovedWriteAheadLogCorruptionException(long segmentId,
        Dictionary<int, Exception> exceptions)
        : base($"Write ahead log with segment id = {segmentId} is corrupted.", new AggregateException(exceptions.Values))
    {
        Exceptions = exceptions;
    }
}

Describe the bug
When ValueSerializer type has a mismatch, the error message shows the KeySerializer type.

To Reproduce
Create a database, change the type of the ValueSerializer and reopen. The message in the exception thrown has the wrong types.

Expected behavior
The error message shows the ValueSerializer types that mismatched, not the KeySerializer types.

Screenshots

The issue seems to be with this statement, in ZoneTreeLoader.cs, it is passing the wrong types into the exception constructor.

if (!string.Equals(ZoneTreeMeta.ValueSerializerType, Options.ValueSerializer.GetType().FullName, StringComparison.Ordinal))
    throw new TreeValueSerializerTypeMismatchException(
        ZoneTreeMeta.KeySerializerType,
        Options.KeySerializer.GetType().FullName);

please complete the following information:

• Operating System: Windows
• ZoneTree Version: 1.6.5

It would be nice to have x86 support in the library - I have some other dependencies with native 32-bit libs.

When running on x86 I get this message when disposing the IZoneTree instance.

System.PlatformNotSupportedException: Operation is not supported on this platform.
   at Tenray.ZoneTree.WAL.AsyncCompressedFileSystemWriteAheadLog`2.AppendLogEntry(Byte[] keyBytes, Byte[] valueBytes, Int64 opIndex)
   at Tenray.ZoneTree.WAL.AsyncCompressedFileSystemWriteAheadLog`2.ConsumeQueue()
   at Tenray.ZoneTree.WAL.AsyncCompressedFileSystemWriteAheadLog`2.StopWriter(Boolean consumeAll)
   at Tenray.ZoneTree.WAL.AsyncCompressedFileSystemWriteAheadLog`2.<MarkFrozen>b__37_0()

Looking at the code it seems to be related to the CRC32C calculation using hardware support.

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

Describe the solution you'd like
A clear and concise description of what you want to happen.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

Describe the bug
Under certain circumstances, FileSystem.ReplaceFile can fail:

System.IO.IOException: Unable to remove the file to be replaced.
         at System.IO.FileSystem.ReplaceFile(String sourceFullPath, String destFullPath, String destBackupFullPath, Boolean ignoreMetadataErrors)
         at Tenray.ZoneTree.AbstractFileStream.LocalFileStreamProvider.Replace(String sourceFileName, String destinationFileName, String destinationBackupFileName)
         at Tenray.ZoneTree.AbstractFileStream.DurableFileWriter.WriteAllBytes(String path, Byte[] bytes)
         at Tenray.ZoneTree.Core.ZoneTreeMetaWAL`2.SaveMetaData(ZoneTreeMeta zoneTreeMeta, Int64 mutableSegment, Int64 diskSegment, Int64[] readOnlySegments, Int64[] bottomSegments, Boolean createNew)
         at Tenray.ZoneTree.Core.ZoneTreeLoader`2.LoadZoneTreeMetaWAL()
         at Tenray.ZoneTree.Core.ZoneTreeLoader`2.LoadZoneTree()
         at Tenray.ZoneTree.ZoneTreeFactory`2.OpenOrCreate()

The Internet seems to suggest that this call can fail if there are any FileSystemWatchers applied to the directory, which can happen incidentally due to various programs (VSCode, etc.)

To Reproduce
The Internet suggests this can be replicated by setting up a FileSystemWatcher in another program: https://stackoverflow.com/questions/46824487/unable-to-remove-the-file-to-be-replaced-error-during-file-replace-if-filesys

Expected behavior
ZoneTree should fall back to Delete/MoveFile if ReplaceFile fails.

please complete the following information:

• Operating System: Windows
• ZoneTree Version: 1.6.0

Describe the bug
MultiPartDiskSegment Seek to existing key at the part border is not working .

To Reproduce
Create a ZoneTree database using MultiPartDiskSegment mode.
Add some data.
Merge data to the disk.
Try to seek to an existing key that is at the border of the disk segments.

Expected behavior
Seek should find the existing key.

please complete the following information:

• Operating System: All
• ZoneTree Version: 1.5.9