agisboye / swiftlmdb Goto Github PK

Receive a SIGSYS when run on a real iPod Touch and iPad Mini 2.
Works OK on simulator.
Issue seems to be this: http://www.openldap.org/lists/openldap-technical/201502/msg00038.html
Fix is to Define MDB_USE_POSIX_SEM to avoid build defaulting to MDB_USE_SYSV_SEM (see within mdb.c source file).

Issue also reported to CLMDB project. Not certain where the fix should ultimately be applied.

ubuntu@ubuntu-xenial:~/src/SwiftLMDB$ uname -a
Linux ubuntu-xenial 4.4.0-72-generic #93-Ubuntu SMP Fri Mar 31 14:07:41 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

ubuntu@ubuntu-xenial:~/src/SwiftLMDB$ swift test
Compile CLMDB midl.c
Compile CLMDB mdb.c
/home/ubuntu/src/SwiftLMDB/.build/checkouts/CLMDB.git--4447338618528579073/Sources/mdb.c:10918:46: warning: data argument not used by format string [-Wformat-extra-args]
(int)mr[i].mr_pid, (size_t)mr[i].mr_tid, txnid);
^
1 warning generated.
Linking CLMDB
Compile Swift Module 'SwiftLMDB' (6 sources)
Compile Swift Module 'SwiftLMDBTests' (1 sources)
Linking ./.build/debug/SwiftLMDBPackageTests.xctest
Test Suite 'All tests' started at 21:03:26.125
Test Suite 'debug.xctest' started at 21:03:26.125
Test Suite 'SwiftLMDBTests' started at 21:03:26.125
Test Case 'SwiftLMDBTests.testGetLMDBVersion' started at 21:03:26.125
Test Case 'SwiftLMDBTests.testGetLMDBVersion' passed (0.0 seconds)
Test Case 'SwiftLMDBTests.testCreateEnvironment' started at 21:03:26.125
Test Case 'SwiftLMDBTests.testCreateEnvironment' passed (0.002 seconds)
Test Case 'SwiftLMDBTests.testCreateUnnamedDatabase' started at 21:03:26.127
Test Case 'SwiftLMDBTests.testCreateUnnamedDatabase' passed (0.0 seconds)
Test Case 'SwiftLMDBTests.testHasKey' started at 21:03:26.127
/home/ubuntu/src/SwiftLMDB/Tests/SwiftLMDBTests/SwiftLMDBTests.swift:109: error: SwiftLMDBTests.testHasKey : XCTAssertTrue failed - No value has been set for this key. Result should be false.
Test Case 'SwiftLMDBTests.testHasKey' failed (0.002 seconds)
Test Case 'SwiftLMDBTests.testPutGetString' started at 21:03:26.130
/home/ubuntu/src/SwiftLMDB/Tests/SwiftLMDBTests/SwiftLMDBTests.swift:149: error: SwiftLMDBTests.testPutGetString : XCTAssertTrue failed - No value has been set for this key. Result should be false.
Test Case 'SwiftLMDBTests.testPutGetString' failed (0.001 seconds)
Test Case 'SwiftLMDBTests.testEmptyKey' started at 21:03:26.131
Test Case 'SwiftLMDBTests.testEmptyKey' passed (0.0 seconds)
Test Case 'SwiftLMDBTests.testPutGetDouble' started at 21:03:26.131
/home/ubuntu/src/SwiftLMDB/Tests/SwiftLMDBTests/SwiftLMDBTests.swift:213: error: SwiftLMDBTests.testPutGetDouble : failed - No value was found for the key.
Test Case 'SwiftLMDBTests.testPutGetDouble' failed (0.001 seconds)
Test Case 'SwiftLMDBTests.testPutGetArray' started at 21:03:26.132
/home/ubuntu/src/SwiftLMDB/Tests/SwiftLMDBTests/SwiftLMDBTests.swift:253: error: SwiftLMDBTests.testPutGetArray : failed - No value was found for the key.
Test Case 'SwiftLMDBTests.testPutGetArray' failed (0.001 seconds)
Test Case 'SwiftLMDBTests.testDelete' started at 21:03:26.132
Test Case 'SwiftLMDBTests.testDelete' passed (0.0 seconds)
Test Case 'SwiftLMDBTests.testDropDatabase' started at 21:03:26.133
Test Case 'SwiftLMDBTests.testDropDatabase' passed (0.001 seconds)
Test Case 'SwiftLMDBTests.testEmptyDatabase' started at 21:03:26.133
Test Case 'SwiftLMDBTests.testEmptyDatabase' passed (0.001 seconds)
Test Suite 'SwiftLMDBTests' failed at 21:03:26.134
Executed 11 tests, with 4 failures (0 unexpected) in 0.009 (0.009) seconds
Test Suite 'debug.xctest' failed at 21:03:26.134
Executed 11 tests, with 4 failures (0 unexpected) in 0.009 (0.009) seconds
Test Suite 'All tests' failed at 21:03:26.134
Executed 11 tests, with 4 failures (0 unexpected) in 0.009 (0.009) seconds

Hi :)

Is this only for basic types of is it possible to store a Codable struct in LMDB ? Or some Data maybe ?

As far as I can tell, there's no way to run an LMDB transaction through SwiftLMDB. I need transactions for an app I'm building.

It looks like the code is all there - just the useful Transaction.init function isn't exposed. Please either make it public or provide another API through which I can run transactions.

The way DataConvertible is implemented for Date in SwiftLMDB right now is this:

extension Date: DataConvertible {}

which then effectively ends up implementing this for each:

public extension DataConvertible {

    init?(data: Data) {
        guard data.count == MemoryLayout<Self>.size else { return nil }
        self = data.withUnsafeBytes { $0.pointee }
    }
    
    var data: Data {
        var value = self
        return Data(buffer: UnsafeBufferPointer(start: &value, count: 1))
    }
}

This is incorrect as Date is a complex type, which cannot simply be memcpy-ed.

What you should do instead is something like this:

extension Date: DataConvertible {

    public init?(data: Data) {
        guard let timeInterval = TimeInterval(data: data) else {
            return nil
        }
        self = Date(timeIntervalSinceReferenceDate: timeInterval)
    }

    public var data: Data {
        return self.timeIntervalSinceReferenceDate.data
    }
}

Is there a way to get a listing/array of all the keys on the database?

@agisboye A great library, thanks!

Have you given any thought to adding cursor operations? E.g. to allow iteration of the database, or for setting and fetching keys in ranges?

Hello !

First of all I would like to thank you all for all your work to bring LMDB into Swift.
This is not a issue report but more like a feature request.
Can you please add a used space variable to Database class, or at least a public available variable for MDB_stat() ?
Also I think a public variable for the LMDB version would be useful sometime in the future.

Thank you !

I'm playing around with LMDB and this library and just tried to use it in a bare-bones macOS application that is sandboxed. However, the database is not opening and mdb_env_open is return 1 (EPERM), suggesting there's a permission problem somewhere.

However the path I'm passing it is within my application's Container. The db lock file is getting created but then something is failing. If this works on iOS then I would assume it should work for macOS sandboxed apps as well.

Just curious if there's any insight on that.

To test, just create a new macOS app in Xcode and try to open a database in your application's support directory as provided by FileManager.

The way DataConvertible is implemented for URL in SwiftLMDB right now is this:

extension URL: DataConvertible {}

which then effectively ends up implementing this for each:

public extension DataConvertible {

    init?(data: Data) {
        guard data.count == MemoryLayout<Self>.size else { return nil }
        self = data.withUnsafeBytes { $0.pointee }
    }
    
    var data: Data {
        var value = self
        return Data(buffer: UnsafeBufferPointer(start: &value, count: 1))
    }
}

This is incorrect as URL is a complex type, which cannot simply be memcpy-ed.

What you should do instead is something like this:

extension URL: DataConvertible {

    public init?(data: Data) {
        guard let url = URL(dataRepresentation: data, relativeTo: nil, isAbsolute: true) else {
            return nil
        }
        self = url
    }

    public var data: Data {
        return self.absoluteURL.dataRepresentation
    }
}

Adding this:
.Package(url: "https://github.com/agisboye/SwiftLMDB.git", majorVersion: 0, minor: 0),

Results in this:
ubuntu@ubuntu-xenial:~/src/Multivariator$ swift build
Fetching https://github.com/agisboye/SwiftLMDB.git
error: unsatisfiable

Due to no tagged release version, as explained in:
https://github.com/apple/swift-package-manager/blob/master/Documentation/Usage.md#publish-a-package

Without the tag, usage as a dependency fails (tested under Ubuntu Linux 16.04).

The way DataConvertible is implemented for Float/Double in SwiftLMDB right now is this:

extension Float: DataConvertible {}
extension Double: DataConvertible {}

which then effectively ends up implementing this for each:

public extension DataConvertible {

    init?(data: Data) {
        guard data.count == MemoryLayout<Self>.size else { return nil }
        self = data.withUnsafeBytes { $0.pointee }
    }
    
    var data: Data {
        var value = self
        return Data(buffer: UnsafeBufferPointer(start: &value, count: 1))
    }
}

This is incorrect as the endianness of the bitwise representation of Float/Double is an implementation detail and thus subject to change at will between platforms, breaking your persisted database.

What you should do instead is this:

extension Float: DataConvertible {

    public init?(data: Data) {
        guard data.count == MemoryLayout<UInt32>.size else { return nil }
        let bigEndian: UInt32 = data.withUnsafeBytes { $0.pointee }
        let bitPattern = UInt32(bigEndian: bigEndian)
        self = Float(bitPattern: bitPattern)
    }

    public var data: Data {
        let bitPattern: UInt32 = self.bitPattern
        var bigEndian = bitPattern.bigEndian
        return Data(buffer: UnsafeBufferPointer(start: &bigEndian, count: 1))
    }
}

extension Double: DataConvertible {

    public init?(data: Data) {
        guard data.count == MemoryLayout<UInt64>.size else { return nil }
        let bigEndian: UInt64 = data.withUnsafeBytes { $0.pointee }
        let bitPattern = UInt64(bigEndian: bigEndian)
        self = Double(bitPattern: bitPattern)
    }

    public var data: Data {
        let bitPattern: UInt64 = self.bitPattern
        var bigEndian = bitPattern.bigEndian
        return Data(buffer: UnsafeBufferPointer(start: &bigEndian, count: 1))
    }
}

Environment.deinit always calls mdb_env_close(handle). However, if mdb_env_open fails on line 82, then the guard condition is entered and the handle is closed then an exception is thrown.

That exception might be caught higher up in such a way that the Environment is released and deinit is called, causing the handle to get released a second time and crashing.

In my case, the database failed to open because it did not have sufficient permissions for the target directory. The exception was caught and an error might be presented to the user, but since the Environment is released the application crashes.

Looks like it might be "safe" to just omit releasing the handle in the guard and instead relying on deinit as that seems to be the case for all other exceptions thrown in the initializer.

I create a new project with Swift 4 & XCode Version 9.2.

I try using Carthage and after the build it not show SwiftLMDB.framework anywhere.

Then, I try to use https://swift.org/package-manager/, it get fetched ok but get:

/.build/checkouts/SwiftLMDB.git-2506504943730548242/Sources/DataConvertible.swift:243:33: 'dataRepresentation' is only available on OS X 10.11 or newer
/.build/checkouts/SwiftLMDB.git-2506504943730548242/Sources/DataConvertible.swift:236:25: 'init(dataRepresentation:relativeTo:isAbsolute:)' is only available on OS X 10.11 or newer

The deployment target is set to OS X 10.11 and Base SDK is 10.13

Apologies for the vague title...

Just started playing around with SwiftLMDB and I was curious if Database.get was implemented correctly based on my interpretation of the documentation.

Database.get calls mdb_get inside a Transaction closure. There is a comment above that points to LMDB's documentation referencing the fact that LMDB will manage the returned memory.

The memory pointed to by the returned values is owned by the database. The caller need not dispose of the memory, and may not modify it in any way.

For values returned in a read-only transaction any modification attempts will cause a SIGSEGV.
Values returned from the database are valid only until a subsequent update operation, or the end of the transaction.

I interpret this to imply that once the Transaction closure finishes, then var dataVal should no longer be considered valid. Which means that read it's mv_data property on line 126 might not be correct.

Shouldn't this line be inside the Transaction closure?

let data = Data(bytes: dataVal.mv_data, count: dataVal.mv_size)

Happy to be proven wrong, just something I noticed while scanning through the code and learning more about LMDB. Thanks for the project and contributions.

(I should add that I actually haven't seen any crashes or issues with this.)

The way DataConvertible is implemented for integer types in SwiftLMDB right now is this:

extension Int: DataConvertible {}
extension Int8: DataConvertible {}
extension Int16: DataConvertible {}
extension Int32: DataConvertible {}
extension Int64: DataConvertible {}
extension UInt: DataConvertible {}
extension UInt8: DataConvertible {}
extension UInt16: DataConvertible {}
extension UInt32: DataConvertible {}
extension UInt64: DataConvertible {}

which then effectively ends up implementing this for each:

public extension DataConvertible {

    init?(data: Data) {
        guard data.count == MemoryLayout<Self>.size else { return nil }
        self = data.withUnsafeBytes { $0.pointee }
    }
    
    var data: Data {
        var value = self
        return Data(buffer: UnsafeBufferPointer(start: &value, count: 1))
    }
}

This is incorrect and will break any usage of SwiftLMDB sharing databases across platforms of different endianness.

What you should do instead (for each integer type individually) is this:

extension Int: DataConvertible {

    public init?(data: Data) {
        guard data.count == MemoryLayout<Int>.size else { return nil }
        let bigEndian: Int = data.withUnsafeBytes { $0.pointee }
        self = Int(bigEndian: bigEndian)
    }

    public var data: Data {
        var bigEndian = self.bigEndian
        return Data(buffer: UnsafeBufferPointer(start: &bigEndian, count: 1))
    }
}

Int8 and UInt8 are the odd ones out here, as there is no endianness for single-byte integers, for obvious reasons.

The way DataConvertible is implemented for Bool in SwiftLMDB right now is this:

extension Bool: DataConvertible {}

which then effectively ends up implementing this for each:

public extension DataConvertible {

    init?(data: Data) {
        guard data.count == MemoryLayout<Self>.size else { return nil }
        self = data.withUnsafeBytes { $0.pointee }
    }
    
    var data: Data {
        var value = self
        return Data(buffer: UnsafeBufferPointer(start: &value, count: 1))
    }
}

This is incorrect as the bitwise representation of Bool is an implementation detail and thus subject to change at will between language versions, breaking your persisted database.

What you should do instead is this:

extension Bool: DataConvertible {
    
    public init?(data: Data) {
        guard data.count == MemoryLayout<UInt8>.size else { return nil }
        self = data.withUnsafeBytes { $0.pointee } != 0
    }

    public var data: Data {
        var value: UInt8 = self ? 1 : 0
        return Data(buffer: UnsafeBufferPointer(start: &value, count: 1))
    }
}