mochidev / codabledatastore Goto Github PK

Add a proper way to convert between a read-write DiskPersistence to a read-only view of one, backed by the same instance.

Also rename the internal API for it

Mark transactions closures with @_implicitSelfCapture since they always complete before proceeding.

As it says on the tin, when persisting an instance, the old instance should be returned, especially since we have it already. Same for deletes actually (it only does this when deleting ids).

We should utilize an in-process registry to ensure that only one disk persistence for a given URL is used at once, logging to make sure a developer doesn't inadvertently create a second instance.

To better support file moves, use a bookmark as the primary storage in memory. This way a data store can be used as a file the user can move around.

This should have a static method that takes a url, and returns a shared actor for the volume the URL resides on. This way, all synchronous IO can be isolated to a single actor. Additionally, when running with the 5.9 runtime, we should use a custom executor to grab a new thread for that IO rather than use the swift concurrency pool, since it would eat up a slot otherwise.

This protocol should return an agnostic representation of a comparable structure in a format that the datastore natively understands. Take a look at foundation db for inspiration, as their binary key paths could be a good format. However, since the goal is to still have human readable files on disk, the default format could be an array of enum primitives to be sorted accordingly? One benefit to Comparable is the ability to add new sorting capabilities, so some investigation should be placed into supporting that well.

ie. offer to manually migrate from a known old version type to a new one. This can be useful if the version was initially an Int, but became a String down the line. Currently, the only way around this would be to have an initializer for the old representation in the new type than evaluates to some deprecated, low-sorting version.

The Date in the index is encoded with milliseconds precision, but this may not match the in memory representation nor the persisted representation. For not, don't use indexes involving floats.

To fix, a) have a stable index representation as in #176, and b) make sure primary record points back to indexed entries. (This might be catastrophic for entries whose identifier is a float…)

When a persistence is created and a datastore is registered, an empty index is created, but is not tracked in any way.

Currently, when defining migrations using the shorthand $0/1 syntax, the data and decoder is used in the opposite order:

    static func datastore(for persistence: DiskPersistence<ReadWrite>) -> Datastore {
        .JSONStore(
            persistence: persistence,
            key: datastoreKey,
            version: .zero,
            migrations: [
                .zero: { try $1.decode(KeyValue.self, from: $0) }
            ]
        )
    }

This is similar to a load for an identifier, but it instead skips decoding, just returning if an entry exists.

If you attempt to load a value when the datastore has no root object, a DatastoreInterfaceError.datastoreKeyNotFound error will be thrown instead of nil.

It's currently possible to accumulate many manifest files, so they should probably be paged to not bring down file system performance.

The indexes currently use @unchecked Sendable as a workaround.

Specifically in cases when a bundle ID or app support directory is not present.

Given a sorted set of IDs (or index entries), we should be able to efficiently walk the tree gathering records in the process, skipping pages as we go.

A persisted type might be available under multiple index entries, and it might be useful to load them as such, especially when we aren't loading a range of index values, but discrete IDs instead.

String(describing:) unfortunately will change a complex type like AccountStore.Item.ID into just ID — it might be better to use String(reflecting:) instead, which should provide the fully qualified name. We need to make sure this works well with local types and migrations though, since we may hit those quite often.

See: https://forums.swift.org/t/getting-the-more-fully-qualified-name-of-a-type/14375/10

Sometimes we are only ever interested in a single value. That said, perhaps a special index type can help here, which would inform the generics of the accessors.

If T conforms to Hashable, so should Indexed.

It would be much nicer to get a single common instance when getting the default store:

extension DiskPersistence where AccessMode == ReadWrite {
    /// The default persistence for the read-write store of an app.
    public static var `default`: DiskPersistence<AccessMode> = ...
}

extension DiskPersistence where AccessMode == ReadOnly {
    /// The default persistence for the read-only store of an app.
    public static var readOnlyDefault: DiskPersistence<AccessMode> = self.default.readOnly
}

However, the default URL helper cannot through in that case, specifically on Linux, and we must have a way from converting from a read-write store to a read-only one.

Depends on:

• #15
• #16

ie. it should handle sorting automatically with a tiered approach.

#158 introduced some defaults, but it would be ideal if they could be configured as well. Additionally, platforms like watchOS could use smaller defaults to have a smaller footprint out of the box.

I've been getting the following error any time I try to reinitialize a store that was previously created. Creating and using a store the first time always works, but if I try to reuse the same URL when rebuilding it always fails with this error:

CodableDatastoreTest/CodableDatastoreTestApp.swift:18: Fatal error: 'try!' expression unexpectedly raised an error: Error Domain=NSCocoaErrorDomain Code=256 "The file “2023-07-13 20-58-15 29023E6541A4E39A.snapshot” couldn’t be opened." UserInfo={NSFilePath=/Users/_______/Library/Developer/CoreSimulator/Devices/B6EABA41-EBF0-46A4-8A33-0150DDA84DD2/data/Containers/Data/Application/505B0923-E569-441A-862C-351FA816B866/tmp/73B58D72-6462-4C31-B5B9-FF5DCF5C085F-19496-000005FDA764CEDA/Snapshots/2023/07-13/20-58/2023-07-13 20-58-15 29023E6541A4E39A.snapshot, NSUnderlyingError=0x600000c98660 {Error Domain=NSPOSIXErrorDomain Code=21 "Is a directory"}}

I even copied the DiskPersistenceDatastoreTests file and tried to run it twice with a static URL and get the same error

Example func that works the first time and successfully persists and then fails with above error when trying to build and run a second time

func write() {
        let url = FileManager.default.containerURL(forSecurityApplicationGroupIdentifier: "group.com.pnewelljr.CodableDatastoreTest")!.appending(path: "Store", directoryHint: .isDirectory)
        let persistence = try DiskPersistence(readWriteURL: url)
        try await persistence.createPersistenceIfNecessary()

        let datastore = Datastore.JSONStore(
            persistence: persistence,
            key: "test",
            version: Version.zero,
            migrations: [
                .zero: { data, decoder in
                    try decoder.decode(TestStruct.self, from: data)
                }
            ]
        )
       try await datastore.warm()
      
        
        try await datastore.persist(TestStruct(id: "3", value: "My name is Dimitri"))
        try await datastore.persist(TestStruct(id: "1", value: "Hello, World!"))
        try await datastore.persist(TestStruct(id: "2", value: "Twenty Three is Number One"))

        let count = try await datastore.count
}

We should rename load(_ range:) to load(_ idRange:) to make it clear to code completion results that this is not for the regular indexed fields.

Also rename the current Indexed to _Indexed to make the underlying bits work, and figure out how to make key play nice with Codable.

Parameter packs or macros should be able to help here, hopefully…