I am experiencing lost data and/or file corruption when using unqlite 1.1.8 on Windows. The following code reliably produces this behavior and has been tested on Windows 7 and 10.

The code inserts 200 records into a database and calls unqlite_commit() after each insertion because frequent commits seem to make the problem happen faster. After inserting the records, I count how many were inserted and the value is 200. I then close and reopen the database and recount the records. The new count is substantially lower (I've seen as low as 6).

Things I've noticed:

1. Removing the call to unqlite_commit() results in the correct amount of records being read back from the file.

2. Using a constant seed in srand() results in the same number of missing records each time. Therefore identical keys/records produces the same behavior every run. I have a commented seed for this purpose.

Am I doing anything wrong in this code?

#include <stdlib.h>
#include <stdint.h>
#include <stdio.h>
#include <time.h>

#include "unqlite.h"

void randomBytes(uint8_t* bytes, size_t count)
{
    for (size_t i = 0; i < count; i++)
    {
        bytes[i] = (rand() % 256);
    }
}

int main()
{
    char* dbPath = "test.db";
    srand((unsigned int) time(nullptr));
    //srand(0x44556677);

    // Delete any existing database file
    unlink(dbPath);

    // Open the db
    unqlite* db = nullptr;
    if (unqlite_open(&db, dbPath, UNQLITE_OPEN_CREATE) != UNQLITE_OK)
    {
        printf("Error opening %s\n", dbPath);
        return 1;
    }

    // Insert data
    for (size_t i = 0; i < 200; i++)
    {
        // Create a key and record
        uint8_t key[40];
        uint8_t record[100];
        randomBytes(key, sizeof(key));
        randomBytes(record, sizeof(record));

        // Call unqlite_commit() each time since frequent commits
        // seem to be a cause of the problem
        if ((unqlite_kv_store(db, key, (int) sizeof(key),
                              record, sizeof(record)) != UNQLITE_OK) ||
            (unqlite_commit(db) != UNQLITE_OK))
        {
            printf("Error saving data\n");
            return 1;
        }
    }

    // Count how many records were inserted. The number should be correct.
    unqlite_kv_cursor* cursor = nullptr;
    if (unqlite_kv_cursor_init(db, &cursor) != UNQLITE_OK)
    {
        printf("Error creating cursor\n");
        unqlite_close(db);
        return 1;
    }

    size_t numInserted = 0;
    for (unqlite_kv_cursor_first_entry(cursor);
         unqlite_kv_cursor_valid_entry(cursor);
         unqlite_kv_cursor_next_entry(cursor))
    {
        numInserted++;
    }
    unqlite_kv_cursor_release(db, cursor);

    // Close and reopen the database
    unqlite_close(db);
    if (unqlite_open(&db, dbPath, UNQLITE_OPEN_CREATE) != UNQLITE_OK)
    {
        printf("Error reopening %s\n", dbPath);
        return 1;
    }

    // Count how many records are in file. The number
    // will be less than were inserted.
    cursor = nullptr;
    if (unqlite_kv_cursor_init(db, &cursor) != UNQLITE_OK)
    {
        printf("Error creating cursor\n");
        unqlite_close(db);
        return 1;
    }

    // This sometimes crashes if the db file is really messed up
    size_t numInFile = 0;
    for (unqlite_kv_cursor_first_entry(cursor);
         unqlite_kv_cursor_valid_entry(cursor);
         unqlite_kv_cursor_next_entry(cursor))
    {
        numInFile++;
    }
    unqlite_kv_cursor_release(db, cursor);
    unqlite_close(db);

    printf("Records inserted: %zu\n", numInserted);
    printf("Records in file: %zu\n", numInFile);

    return 0;
}

Hello,

In the unqlite_open documentation page, it is specified that, if UNQLITE_OPEN_READWRITE is used:

Open the database with read+write privileges. If the database does not exists, an error code is returned.

I would expect this behaviour to be true for UNQLITE_OPEN_READONLY too.

Yet, what I observe is that even if the DB does not exists, the return code is UNQLITE_OK. But if you call any funtion using the returned handle (like a store instruction), you will get a UNQLITE_IOERR return code.

What is the real expected behaviour? The one specified is appealing because it allow us to check immediatly if the DB exists instead of waiting for a first instruction to be made.

Hello,

On a windows 64 bit system compiling in C for 64bit architecture, I get:

unqlite.c(29339) : warning C4244: '=' : conversion from '__int64' to 'int', possible loss of data
unqlite.c(29351) : warning C4244: '=' : conversion from '__int64' to 'int', possible loss of data
unqlite.c(29460) : warning C4244: '=' : conversion from '__int64' to 'int', possible loss of data

This makes sense. From what I see, you do pointer math and store the result in an int. In 64bit, pointers are 64bit and int is usually 32bit (on Windows at least). The loss of data could be real, even if really unlikely.

Hello, I could not find a source of information about when to place strategic commits. It appeared to me, that unqlite would not commit anything to the file storage until I explicitly commit or close the database. Is there some resource talking about ways to avoid a huge commit on closing the database?

I embedded unqlite into my c# application, and from my c# call back function, an exception can be thrown that only c# can catch it. so on any exception thrown by my c# callback function I need away to free all acquired resources by the unqlite_context.

I managed to do so by providing a global callback that gives me the context and stack frame and top stack frame and arguments, I modified VmByteCodeExec and added calls to global callback :

xAbortFuncDelegate lxAbortFunc = xAbortFunc;

\\make c# have mapping from ctx->{aArgs,pStack,pTop} to be used to free resources
if (lxAbortFunc) lxAbortFunc(&sCtx, &aArg, pTos, pStack,\*before*\1);

\\the original line that calls the embedded c functions registered from c#
rc = pFunc->xFunc(&sCtx, (int)SySetUsed(&aArg), (jx9_value **)SySetBasePtr(&aArg));

\\to make c# free mapping info from ctx->{aArgs,pStack,pTop} after releasing the resource acquired by the unqlite_context execution
if (lxAbortFunc) lxAbortFunc(&sCtx, &aArg, pTos, pStack, \*after*\0);

and from my callback function if an exception is thrown, I call a c function that free all resources then re-throw the exception :

VmReleaseCallContext(sCtx);
SySetRelease(aArg);
while (pTos >= pStack) {
jx9MemObjRelease(pTos);
pTos--;
}

so what I need is a built in way to do the task.

Hello -- the available doc online is for version 116 -- and in that zip file it notes that there is a version of the doc in a single html or pdf file -- does the doc for 119 exist in this format and if so, where can I find it to download -- thanks

Good day, i try to build unqlite on Linux:
uname -a
Linux debian 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt20-1+deb8u4 (2016-02-29) x86_64 GNU/Linux

error: declaration of ‘pgno unqlite_page::pgno’ [-fpermissive]
pgno pgno;

Hi, is LZ compression availible?

I have tried to compile unqlite into windows runtime component, but some functions do not support:

AreFileApisANSI
CreateFileW
MoveFileW
GetFileSize
GetTempPathA
GetProcessId
GetStdHandle
LockFile
UnlockFile
GetVersionExW 
GetEnvironmentVariableA
GetFileInformationByHandle
GetProcessId
...

I have tried to make some changes to the source code, like this:

/* int (*xRename)(const char *, const char *) */
static int WinVfs_Rename(const char *zOld, const char *zNew)
{
	void *pOld, *pNew;
	BOOL rc = 0;
	pOld = jx9convertUtf8Filename(zOld);
	if( pOld == 0 ){
		return -1;
	}
	pNew = jx9convertUtf8Filename(zNew);
	if( pNew  ){
#ifdef _WINRT_DLL
		_wrename( (wchar_t*)pOld, (wchar_t*)pNew );
#else
		rc = MoveFileW((LPCWSTR)pOld, (LPCWSTR)pNew);
#endif
	}
	HeapFree(GetProcessHeap(), 0, pOld);
	if( pNew ){
		HeapFree(GetProcessHeap(), 0, pNew);
	}
	return rc ? JX9_OK : - 1;
}

although the removal of some compilation errors, but can not completely solve this problem.

Do you consider add windows universal platform (UWP) support?

Build tool: Visual Studio 2017
OS and version: Windows 10
Target OS: Universal Windows
Unqlite version: 1.1.6

When using an in-memory database, a just-initialized cursor is valid:

unqlite *pDb;
unqlite_kv_cursor *pCursor;

const char key[] = "thekey";
const char data[] = "thedata";

unqlite_open(&pDb, ":mem:", UNQLITE_OPEN_CREATE);

unqlite_kv_store(pDb, key, sizeof(key), data, sizeof(data));
unqlite_commit(pDb);

unqlite_kv_cursor_init(pDb, &pCursor);

printf("No sought key, valid=%d\n", unqlite_kv_cursor_valid_entry(pCursor));

This prints valid=1, and loading the key gives you "thekey".

However, if it's an on-disk database instead (unqlite_open(&pDb, "some_file.tmp", UNQLITE_OPEN_CREATE)), then the initial cursor is not valid, and points to no element.

I think the behavior should be consistent, and in both cases the cursor should start off invalid. Either that, or specify explicitly in the docs that a just-initialized cursor may be either valid or invalid, and to get an invalid cursor on purpose you have to do something like:

unqlite_kv_cursor_last_entry(pCursor);
unqlite_kv_cursor_next_entry(pCursor);

(Note this may not even work, see #29).

I am new to unqlite. I don't exact way to store million data in one go . when I am trying to insert million data its taking too much memory until, I am not calling "unqlite_close". Any help or sample code to insert millions data will be helpful for me

Thank you for reviving the UnQLite project.
I want to use latest version of UnQLite.
Could you make tag "1.1.9" on git repository?

I just noticed db_update_record is not included in amalgam.
Is this an error or feature?

I added the missing functions and it works fine (as far as my tests work)... Should I submit a PR with the fix or there is a "standard" way of doing it?

I am reading a portion from a large record and it fails with error UNQLITE_ABORT. However when I examine the buffer, I see the data was actually read.

When doing the same test with smaller records, the partial read succeeds with UNQLITE_OK. The failures seem to start between 4000 and 5000 bytes.

The code below demonstrates the problem. I have tested it on Windows 7 and 10.

#include <memory.h>
#include <stdlib.h>
#include <stdint.h>
#include <stdio.h>
#include <time.h>

#include "unqlite.h"

void randomBytes(uint8_t* bytes, size_t count)
{
    for (size_t i = 0; i < count; i++)
    {
        bytes[i] = (rand() % 256);
    }
}

int main()
{
    char* dbPath = "test.db";
    srand((unsigned int) time(nullptr));

    // Delete any existing database file
    unlink(dbPath);

    // Open the db
    unqlite* db = nullptr;
    if (unqlite_open(&db, dbPath, UNQLITE_OPEN_CREATE) != UNQLITE_OK)
    {
        printf("Error opening %s\n", dbPath);
        return 1;
    }

    // Insert a 10000 byte record
    uint8_t key[40];
    randomBytes(key, sizeof(key));

    size_t recordSize = 10000;
    uint8_t* record = new uint8_t[recordSize];
    randomBytes(record, recordSize);

    if ((unqlite_kv_store(db, key, (int) sizeof(key),
                          record, recordSize) != UNQLITE_OK) ||
        (unqlite_commit(db) != UNQLITE_OK))
    {
        printf("Error saving data\n");
        return 1;
    }

    // Read the whole record (This will work)
    unqlite_int64 bytesToRead = recordSize;
    int ret = unqlite_kv_fetch(db, key, sizeof(key), record, &bytesToRead);
    if (ret == UNQLITE_OK)
    {
        printf("Successfully read entire record\n");
    }
    else
    {
        printf("Failed to read entire record\n");
    }

    // Read just a portion of the record (This will fail with UNQLITE_ABORT)
    uint8_t partialRecord[50];
    bytesToRead = sizeof(partialRecord);

    ret = unqlite_kv_fetch(db, key, sizeof(key), partialRecord, &bytesToRead);
    if (ret == UNQLITE_OK)
    {
        printf("Successfully read a portion of the record\n");
    }
    else
    {
        printf("An error was returned reading a portion of the record\n");
        if (memcmp(partialRecord, record, sizeof(partialRecord)) == 0)
        {
            printf("But the bytes were actually read\n");
        }
    }

    delete[] record;
    unqlite_close(db);

    return 0;
}

How i can fetch collection names from my db?
Unfortunately, i can't find the way in docs.

Do you have an SDK for iOS and Android?

Any plans to add a makefile?

Hello,

The unqlite.h file present under src/ constains a extern "C" while the one at the root does not, and it is apparently the one available in the archive downloadable from the website.

I do not know if this is intended, but this apparently produces compilation issues on NT while trying to build unqlite as a shared lib.

The following code produces the error:

1 Error: JSON Object: Missing closing braces '}

function main() {
    print "Hello, world\n";
    die;
}
main();

Seems a strange issue, given that it isn't a JSON object.

In an on-disk database, unqlite_kv_cursor_prev_entry of an invalid entry gets a valid cursor pointing to the last entry. In an on-memory database, unqlite_kv_cursor_prev_entry of an invalid entry still keeps the cursor invalid.

In a similar vein to #27, I think it should be consistent and/or explicitly documented.

unqlite is considered to be the nosql counterpart of sqlite.
There are however some features available within sqlite which I also would like to see in unqlite.

Feature Request: Attach

Implementation of a attach method. This attach method will take a filepath as argument.
It will attach the provided filepath to the current database connection.

This allows to use a single database connection to interact with multple unqlite files. Also it might allow JX9 selection over collections in multiple unqlite files.

unqlite is considered to be the nosql counterpart of sqlite.
There are however some features available within sqlite which I also would like to see in unqlite.

Feature Request: Backup

Implementation of a backup method. This backup method will take a filepath as argument.
It will take the write lock on the unqlite file so all readers have to wait. Then it will copy the entire unqlite to the provided filepath, as a backup/snapshot.

This will allow to create a backup of a unqlite database without shutting down all connections to release the file.

Hello,

This is a very specific issue related to old compilers, but this is easy to solve.

For some reason, the compiler produces an error if the .h file does not contain a blank line as last line. I would humbly advise to always end your file with newlines for portability.

I would like to point out that identifiers like "_UNQLITE_H_" and "_JX9H_" do not fit to the expected naming convention of the C language standard.
Would you like to adjust your selection for unique names?

The title says it all! Storing raw binary data in a key value store works well, but I'm having trouble figuring out how to store raw bytes in a document.

How would one go about doing this? For what it's worth, I'm using the Python client library.

Thanks!

Hello,

Can it be possible to add a synchronization mechanism, such as CouchDB? Could be useful when the phone goes offline.

When I run the following, the second printf displays 483 rather than 500. Removing the first while() loop that retrieves all of the keys causes the second printf to display 500. Is this expected?

#include "unqlite.h"
#include <stdio.h>

#define N       500
#define KEYLEN  8
#define DATALEN 100

int main(void) {

  unqlite *db;
  unqlite_kv_cursor *cursor;
  char key[KEYLEN];
  char data[DATALEN];
  char key_array[N][KEYLEN];
  int ikey;
  unqlite_int64 idata;
  int i, j, rc;

  unqlite_open(&db, "test.db", UNQLITE_OPEN_CREATE);

  for (i=0; i<N; i++) {
    sprintf(key, "key_%i", i);
    for (j=0; j<DATALEN; j++)
      data[j] = 'x';
    unqlite_kv_store(db, key, sizeof(key), data, sizeof(data));
  }

  unqlite_close(db);

  unqlite_open(&db, "test.db", UNQLITE_OPEN_CREATE);

  unqlite_kv_cursor_init(db, &cursor);
  unqlite_kv_cursor_reset(cursor);

  j = 0;
  while (unqlite_kv_cursor_valid_entry(cursor)) {
    unqlite_kv_cursor_key(cursor, &key_array[j], &ikey);
    unqlite_kv_cursor_next_entry(cursor);
    j += 1;
  }
  unqlite_kv_cursor_release(db, cursor);
  printf("%i\n", j);

  unqlite_kv_cursor_init(db, &cursor);
  unqlite_kv_cursor_reset(cursor);

  j = 0;
  while (unqlite_kv_cursor_valid_entry(cursor)) {
    unqlite_kv_cursor_key(cursor, &key, &ikey);
    unqlite_kv_fetch(db, key, sizeof(key), &data, &idata);
    unqlite_kv_cursor_next_entry(cursor);
    j += 1;
  }
  unqlite_kv_cursor_release(db, cursor);
  printf("%i\n", j);
}

I'm building the above against unqlite 1.1.6 with Apple LLVM 7.3.0 on MacOS 10.11.6.

Hello,

I am back with some good old portability issue. This can be seen using a -pedantic flag with a GCC-G++ compiler.

In the file Jx9Int.h on line 2709, we find:

enum iErrCode
{
    E_ERROR             = 1,   /* Fatal run-time errors. These indicate errors that can not be recovered 
                                * from, such as a memory allocation problem. Execution of the script is
                                * halted.
                                * The only fatal error under JX9 is an out-of-memory. All others erros
                                * even a call to undefined function will not halt script execution.
                                */
    E_WARNING           ,   /* Run-time warnings (non-fatal errors). Execution of the script is not halted.  */
    E_PARSE             ,   /* Compile-time parse errors. Parse errors should only be generated by the parser.*/
    E_NOTICE            ,   /* Run-time notices. Indicate that the script encountered something that could 
                                * indicate an error, but could also happen in the normal course of running a script. 
                                */
};

The trailing comma after E_NOTICE causes troubles with old compilers:

line 2722: Error: Identifier expected instead of "}".

I noticed that your other enums are formatted without trailing comma.

Thank you!

Hello,

According to dmigous@25f2492
please, apply the same patch on unqlite.c.
Perl's community will says thanks: http://search.cpan.org/perldoc?UnQLite

--- a/modules/UnQLite-0.05/unqlite/unqlite.c
+++ b/modules/UnQLite-0.05/unqlite/unqlite.c
@@ -30275,9 +30275,11 @@ static sxi32 SyOSUtilRandomSeed(void *pBuf, sxu32 nLen, void *pUnused)
        fd = open("/dev/urandom", O_RDONLY);
        if (fd >= 0 ){
                if( read(fd, zBuf, nLen) > 0 ){
+                       close(fd);
                        return SXRET_OK;
                }
                /* FALL THRU */
+               close(fd);
        }
        pid = getpid();
        SyMemcpy((const void *)&pid, zBuf, SXMIN(nLen, sizeof(pid_t)));

Thanks!

Hello,

Line 54073 (referring to unqlite.c amalgamated file), we have an allocation:

pUnused = unqlite_malloc(sizeof(*pUnused));

This allocation is missing the (UnixUnusedFd*) cast, resulting in the compilation error:

Error: Cannot assign void* to UnixUnusedFd*.

I am having file handle issue when using unqlite 1.1.9 on Windows. The following code reliably produces this behavior and has been tested on Windows 7 and 10.

Basically the code creates a corrupt file and tries to load a record from it which results in UNQLITE_CORRUPT. It then calls unqlite_close() followed by the Windows routine DeleteFileA() which returns ERROR_SHARING_VIOLATION because the file handle is still open.

In looking at the unqlite code, I think the problem is in unqlitePagerClose().
It does the following check to see if unqliteOsCloseFree() should be called.
if( !pPager->is_mem && pPager->iState > PAGER_OPEN )

Should that be changed to a logical OR?

#include <stdint.h>
#include <stdio.h>
#include <Windows.h>

#include "unqlite.h"

int main()
{
    // Create a corrupt db file
    char* badData = "badData";
    char* dbPath = "test.db";
    FILE* dbFile = fopen(dbPath, "wb");
    fwrite(badData, strlen(badData), 1, dbFile);
    fclose(dbFile);

    // Open the db
    unqlite* db = nullptr;
    if (unqlite_open(&db, dbPath, UNQLITE_OPEN_CREATE) != UNQLITE_OK)
    {
        printf("Error opening %s\n", dbPath);
        return 1;
    }

    uint8_t key[10];
    uint8_t record[100];
    unqlite_int64 recordSize = sizeof(record);

    if (unqlite_kv_fetch(db, key, sizeof(key), record, &recordSize) == UNQLITE_CORRUPT)
    {
        // Closing the database is not closing the handle because
        // unqlitePagerClose() is not calling unqliteOsCloseFree()
        // Should this line be changed to a logical OR?
        // if( !pPager->is_mem && pPager->iState > PAGER_OPEN ){
        unqlite_close(db);

        // Try to delete the file
        if (!DeleteFileA(dbPath) && GetLastError() == ERROR_SHARING_VIOLATION)
        {
            printf("Failed to delete since handle is still open\n");
        }
    }

    return 0;
}

Is there upsert supported and how to use them?

Hello,

Update, these are actually warnings:

Using a GCC-G++ 4.8.1 with MinGW to compile unqlite to be used with C++ code, with the following flags on the unqlite.c amalgamated file:

-Wall -Werror

I get quite a lot of compilation warnings about deprecated conversion from string constant to "char*". I do not know if you target on supporting this kind of issues, but this might lead to portability issues on various systems.

It is to be noted that VS2013 compiles the file fine.

Using string literals to initialize char* have been deprecated, as literals are of type const char*, and converting them to "char*" drops the const-qualifier.

I used C-like casts (char*) to quickly solve the issue, but this is not a really elegant way to fix this. See comment bellow.

Please let me know if you need further information.

I am experiencing a memory leak when using unqlite 1.1.9 on Windows. The following code reliably produces this behavior and has been tested on Windows 7 and 10.

Basically I'm noticing that calling unqlite_commit() less often aggravates the memory leak.

The code first writes records to a database and only calls unqlite_commit() once. The memory usage is not reduced by that call to unqlite_commit().

I then close and rewrite the database while calling unqlite_commit() after each write operation. The memory usage after writing all the records is significantly lower that the first approach.

Your team previously fixed a memory leak that I reported in 1.1.8 by adding a call to pager_release_page() in pager_commit_phase1(). Is there a possibility this isn't freeing all pages?

The comments in the code explain all the results I am observing. Hopefully this code helps you track down the issue. Thanks.

#include <stdlib.h>
#include <stdint.h>
#include <stdio.h>
#include <time.h>

#include "unqlite.h"

void randomBytes(uint8_t* bytes, size_t count)
{
    for (size_t i = 0; i < count; i++)
    {
        bytes[i] = (rand() % 256);
    }
}

int main()
{
    char* dbPath = "test.db";
    srand((unsigned int) time(nullptr));

    // Delete any existing database file
    unlink(dbPath);

    // Open the db
    unqlite* db = nullptr;
    if (unqlite_open(&db, dbPath, UNQLITE_OPEN_CREATE) != UNQLITE_OK)
    {
        printf("Error opening %s\n", dbPath);
        return 1;
    }

    // Insert twenty 1 MB records
    size_t numRecords = 20;
    size_t recordSize = 1024 * 1024;
    uint8_t* record = new uint8_t[recordSize];
    randomBytes(record, recordSize);

    for (size_t i = 0; i < numRecords; i++)
    {
        size_t key = i;
        if ((unqlite_kv_store(db, &key, (int) sizeof(key),
                              record, recordSize) != UNQLITE_OK))
        {
            printf("Error saving data\n");
            return 1;
        }
    }

    // At this point on Windows, this process is using about 47 MB of RAM
    // Call commit which should free most of this RAM
    unqlite_commit(db);

    // Still using about 47 MB after commit
    // Free it with close
    unqlite_close(db);

    // Close freed most of the RAM as expected
    // Recreate the DB and this time commit after each call to unqlite_kv_store()
    unlink(dbPath);
    db = nullptr;
    if (unqlite_open(&db, dbPath, UNQLITE_OPEN_CREATE) != UNQLITE_OK)
    {
        printf("Error opening %s\n", dbPath);
        return 1;
    }

    for (size_t i = 0; i < numRecords; i++)
    {
        size_t key = i;
        if ((unqlite_kv_store(db, &key, (int) sizeof(key),
            record, recordSize) != UNQLITE_OK))
        {
            printf("Error saving data\n");
            return 1;
        }

        // Commit each time
        unqlite_commit(db);
    }

    // Now this process is only using about 9 MB of RAM
    // There seems to be a memory leak when committing less often
    // Is unqlite_commit() actually freeing all pages it commits to disk?
    unqlite_close(db);

    delete[] record;
    return 0;
}

Loading in a large sequential stream into the kv store seems to get unqlite stuck inside lhAllocateSpace where it looks like the link list is a circle. (iNext goes 12 -> 254 -> 1024 -> 12).

I added a bit of code to detect this and set iNext to zero which seems to allow it to recover but this is a bit of a hack.

sxu16 seen[256];
zPrev = (unsigned char *)zPtr;

seenStart = seen;
while (*seenStart) // while we have a valid seen entry
{
  if (iNext == *seenStart) // check to see if we have seen this index before
  {
    iNext = 0; // if we have, lets run cleanup
    break;
  }

  ++seenStart; // check next entry
}

if (!*seenStart) // if we got to the end add the current index
{
  *seenStart = iNext;
}

if( iNext == 0 ){

Can you add link to this repo at http://unqlite.org/downloads.html to ensure that this is an official repo? :)

Enable Travis CI.

Project admin symisc can register on travis-ci.org.
Enable project in Travis, after which every commit and merge request will be build and tested.

Gitlab CI is provided in repository for anyone who runs their own private GitLab.

I was not able to create a standalone case for that yet, but I experience data loss in production. Basically when I write into database, close it, reopen and try to fetch the data I just wrote, sometimes I get UNQLITE_NOTFOUND.

I tracked the issue down to lhPageDefragment called for slave page. It is clearly doesn't make sense to defragment slave page, and lhPageDefragment doesn't expect to receive slave page. I added a guard at the very beginning of lhPageDefragment:

if (pPage->pMaster != pPage)
  return UNQLITE_FULL;

That fixed the immediate issue -- there are no data losses any more. But here could be bigger issue hidden -- it seems to be logical error to call lhPageDefragment for slave page. Probably there should be other way to defragment slave pages.

I found that library does not report error if open nonexistent database without OPEN_CREATE mode

#include <stdio.h>
#include "unqlite.h"

int main(int argc, char **argv) {
    char *filename = "nonexistent.db";
    unqlite *handle;
    int rc;

    printf("Unqlite version: %s\n", UNQLITE_VERSION);
    printf("Unqlite version number: %d\n", UNQLITE_VERSION_NUMBER);

    rc = unqlite_open(&handle, filename, UNQLITE_OPEN_READWRITE);
    if (rc == 0) {
        printf("Invalid behavior: Zero error on nonexistent db file\n");
        return 0;
    }
    printf("Valid behavior: Non-zero error on nonexistent db file\n");
    return 0;
}

I get following output:

Unqlite version: 1.1.6
Unqlite version number: 1001006
Invalid behavior: Zero error on nonexistent db file

I found strange bug, for some keys unqlite lost some key-value pair after commit, I write code for reproduce it:

#include <string>
#include <ctime>
#include <cstdint>

extern "C" {
#include "../UnQLite-kv/unqlite.h"
}

struct TestStruct
{
    double d1;
    double d2;
};

int main(int argc, char **argv)
{
    std::string filename("TEST." + std::to_string(time(nullptr)));
    TestStruct test = { 1.0, 1.1 };

    // Fill database 
    {
        unqlite *db = nullptr;

        int res = unqlite_open(&db, filename.c_str(), UNQLITE_OPEN_CREATE);
        if (res != UNQLITE_OK)
        {
            return res;
        }

        for (int64_t i = 0; i < 165; ++i)
        {
            int res = unqlite_kv_store(db, &i, sizeof(int64_t), &test, sizeof(TestStruct));
            if (res != UNQLITE_OK)
            {
                return res;
            }
        }

        res = unqlite_close(db);
        if (res != UNQLITE_OK)
        {
            return res;
        }
    }

    // Reopen
    {
        unqlite *db = nullptr;

        int res = unqlite_open(&db, filename.c_str(), UNQLITE_OPEN_CREATE);
        if (res != UNQLITE_OK)
        {
            return res;
        }

        // Write pair with key 162 again
        int64_t i_bug = 162;
        res = unqlite_kv_store(db, &i_bug, sizeof(int64_t), &test, sizeof(TestStruct));
        if (res != UNQLITE_OK)
        {
            return res;
        }

        // Force commit
        res = unqlite_commit(db);
        if (res != UNQLITE_OK)
        {
            return res;
        }

        // Test pair with key 164, it lost
        int64_t i = 164;
        TestStruct test1= {};
        unqlite_int64 buf_size = sizeof(TestStruct);
        res = unqlite_kv_fetch(db, &i, sizeof(int64_t), &test1, &buf_size);
        if (res != UNQLITE_OK)
        {
            printf("%d", res); //Yep, print -6 there
            return res;
        }

        res = unqlite_close(db);
        if (res != UNQLITE_OK)
        {
            return res;
        }
    }

    return 0;
}

For this types of key and value, after reopen database and write pair with key 162 and force commit unqlite lost pair with key 164.
I check it with last unqlite, unqlite-kv and vedis sources on msvc2013, clang 3.8.1 and gcc 6.2.0

I would like to know if the project is still alive. Do you plan any update? Was it replaced by something else?

Consider the following program:

extern "C" {
  #include "unqlite/unqlite.h"
}

#include <cstdlib>
#include <cstdio>


int main(int argc, char *argv[])
{
  unqlite *pDb;
  unqlite_kv_cursor *pCursor;
  unqlite_open(&pDb, "somefile.dat", UNQLITE_OPEN_CREATE);

  for (int i=0; i < 200; i++) {
    int data = i * 5;
    unqlite_kv_store(pDb, &i, sizeof(i), &data, sizeof(data));
  }

  unqlite_kv_cursor_init(pDb, &pCursor);
  unqlite_kv_cursor_last_entry(pCursor);
  printf("next_entry == UNQLITE_OK? %d\n", unqlite_kv_cursor_next_entry(pCursor) == UNQLITE_OK);
  printf("next_entry after last is valid? %d\n", unqlite_kv_cursor_valid_entry(pCursor));

  int key;
  int data;
  int nKey = sizeof(key);
  unqlite_int64 nData = sizeof(data);
  unqlite_kv_cursor_key(pCursor, &key, &nKey);
  unqlite_kv_cursor_data(pCursor, &data, &nData);

  printf("next_entry after last key: %d, value: %d\n", key, data);

  unqlite_kv_cursor_release(pDb, pCursor);
  unqlite_close(pDb);


  return 0;
}

The output is:

$ rm somefile.dat
$ ./test_it
next_entry == UNQLITE_OK? 1
next_entry after last is valid? 1
next_entry after last key: 199, value: 995
$ ./test_it
next_entry == UNQLITE_OK? 1
next_entry after last is valid? 1
next_entry after last key: 1, value: 5

This is surprising, since the next entry after last entry should be invalid. Combined with #27 this makes it very difficult to actually get an invalid cursor (needed for the C++ interface I'm writing, which needs to have a ::end() function that returns an invalid iterator).

If only say 20 elements are inserted, instead of 200, then the behavior is as expected (next entry after last entry is indeed invalid).

Would you like to replace more defines for constant values by enumerations to stress their relationships?

• SXERR_…
• JX9_LIB_CONFIG_…

I install unqlite using "pip install unqlite" command. But such codes generate a file named "test.db_unqlite_journal" and data cannot be written into the file "test.db". If the line 'timer.start()' is deleted, then it works well. Cannot unqlite be used in multithread environment?

`from threading import Thread
from unqlite import UnQLite
import time

class Timer(Thread):
def run(self):
while True:
time.sleep(1)

db = UnQLite("test.db")
users = db.collection("users")
users.create()
users.store({'name':'Tom', 'age':23})

timer = Timer()
timer.start()

time.sleep(1000)`

I see you patched what appears to be a couple (serious) bugs that could cause data corruption... but I notice that some of the fixes were made in unqlite.c that are not in the individual source files (jx9_lib).

Which is the correct source to use?

Is there a reason the patches are not applied to both files?

See the following code, which opens a database, inserts 1000 entries with 4-byte keys and 400-byte values, closes the database, opens it again, and then tries to delete them all:

extern "C" {
  #include "unqlite/unqlite.h"
}

#include <cstdlib>
#include <cstdio>
#include <memory.h>


int main(int argc, char *argv[])
{
  unqlite *pDb;
  unqlite_kv_cursor *pCursor;
  unqlite_open(&pDb, "somefile.dat", UNQLITE_OPEN_CREATE);

  for (int i=0; i < 1000; i++) {
    int data[100];
    memset(data, 33, sizeof(data));
    unqlite_kv_store(pDb, &i, sizeof(i), &data[0], sizeof(data));
  }

  unqlite_close(pDb);

  // open & clear the database
  unqlite_open(&pDb, "somefile.dat", UNQLITE_OPEN_CREATE);

  unqlite_kv_cursor_init(pDb, &pCursor);
  while (true) {
    unqlite_kv_cursor_first_entry(pCursor);
    if (!unqlite_kv_cursor_valid_entry(pCursor)) {
      printf("Everything deleted\n");
      break;
    }

    int key;
    int nBytes = sizeof(key);
    unqlite_kv_cursor_key(pCursor, &key, &nBytes);
    printf("Deleting key %d\n", key);

    if (unqlite_kv_cursor_delete_entry(pCursor) != UNQLITE_OK) {
      printf("Failed to delete?\n");
      break;
    }
  }

  printf("Done");

  unqlite_kv_cursor_release(pDb, pCursor);
  unqlite_close(pDb);

  return 0;
}

The output is:

Deleting key 3597
Deleting key 3884
Deleting key 4171
Deleting key 4458
....
Deleting key 9
Deleting key 9

And then an infinite loop. lldb reports this as a sample stack trace:

* thread #1: tid = 0x1134e01, 0x000000010000f8fc test_it`unqlitePagerAcquire [inlined] SyZero(pSrc=<unavailable>, nSize=<unavailable>) + 9 at unqlite.c:27042, queue = 'com.apple.main-thread', stop reason = signal SIGSTOP
    frame #0: 0x000000010000f8fc test_it`unqlitePagerAcquire [inlined] SyZero(pSrc=<unavailable>, nSize=<unavailable>) + 9 at unqlite.c:27042
   27039    #endif
   27040        zEnd = &zSrc[nSize];
   27041        for(;;){
-> 27042            if( zSrc >= zEnd ){break;} zSrc[0] = 0; zSrc++;
   27043            if( zSrc >= zEnd ){break;} zSrc[0] = 0; zSrc++;
   27044            if( zSrc >= zEnd ){break;} zSrc[0] = 0; zSrc++;
   27045            if( zSrc >= zEnd ){break;} zSrc[0] = 0; zSrc++;
(lldb) bt
* thread #1: tid = 0x1134e01, 0x000000010000f8fc test_it`unqlitePagerAcquire [inlined] SyZero(pSrc=<unavailable>, nSize=<unavailable>) + 9 at unqlite.c:27042, queue = 'com.apple.main-thread', stop reason = signal SIGSTOP
  * frame #0: 0x000000010000f8fc test_it`unqlitePagerAcquire [inlined] SyZero(pSrc=<unavailable>, nSize=<unavailable>) + 9 at unqlite.c:27042
    frame #1: 0x000000010000f8f3 test_it`unqlitePagerAcquire [inlined] pager_alloc_page(pPager=0x0000000101808214, num_page=<unavailable>) + 45 at unqlite.c:55626
    frame #2: 0x000000010000f8c6 test_it`unqlitePagerAcquire(noContent=<unavailable>, pPager=<unavailable>, pgno=<unavailable>, ppPage=<unavailable>, fetchOnly=<unavailable>) + 182 at unqlite.c:57481
    frame #3: 0x000000010003f721 test_it`lhCursorDelete [inlined] lhRecordRemove(pCell=0x0000000101822f1c) + 56 at unqlite.c:50227
    frame #4: 0x000000010003f6e9 test_it`lhCursorDelete(pCursor=<unavailable>) + 57 at unqlite.c:51696
    frame #5: 0x00000001000025f8 test_it`main(argc=<unavailable>, argv=<unavailable>) + 280 at main.cpp:40

This doesn't happen with fewer elements (say 100), or with elements with a smaller value size (say 4 bytes), or if the deleting is done with the same handle that the creating was done with. unqlite_commit()ing after every delete also avoids triggering the bug.

With g++ 4.4.7, on a Linux system, including "unqlite.h" surrounded by "extern C" instruction in a CPP file, I get the following error:

[exec] unqlite.h:661: error: declaration of `pgno unqlite_page::pgno'
[exec] unqlite.h:651: error: changes meaning of `pgno' from `typedef sxu64 pgno'

This is due to:

typedef sxu64 pgno;
/*
 * A database disk page is represented by an instance
 * of the follwoing structure.
 */
typedef struct unqlite_page unqlite_page;
struct unqlite_page
{
  unsigned char *zData;       /* Content of this page */
  void *pUserData;            /* Extra content */
  pgno pgno;                  /* Page number for this page */ // <------ This
};

The unqlite lib was compiled with a C compiler, but I have to use a C++ one with the public header.

This is due to the g++ version I believe. I reported this so you know and can decide whether it is something you'd like to handle or not.

Cheers!

On the homepage it says "thread-safe and full(y) reentrant", but if you look at the compiler flags, it seems to indicate that it is not thread-safe by default?

UNQLITE_ENABLE_THREADS

This option controls whether or not code is included in UnQLite to enable it to operate safely in a multithreaded environment. The default is not. All mutexing code is omitted and it is unsafe to use UnQLite in a multithreaded program. When compiled with the UNQLITE_ENABLE_THREADS directive enabled, UnQLite can be used in a multithreaded program and it is safe to share the same virtual machine and database handle between two or more threads.

Could you help me understand the difference?

See symisc/vedis#3. The same code is used in unqlite, and the same bugs exist.

There are 3 fixes to be applied: symisc/vedis#5, symisc/vedis#6, symisc/vedis#7

Here is example program to reproduce the issue (it is the vedis version adopted for unqlite)

#include <unqlite.h>

#include <stdlib.h>
#include <stdio.h>
#include <time.h>

// generate random bytes of variable (important!) length
char *randomBytes(int* len)
{
    *len = rand() % 4 + 1;
    char *res = (char*)malloc(*len);

    int i;
    for (i = 0; i < *len; i++)
    {
        res[i] = rand() % 256;
    }

    return res;
}

int main()
{
    unqlite *db;
    int res = unqlite_open(&db, "test.db", UNQLITE_OPEN_CREATE);
    if (res != UNQLITE_OK)
    {
        printf("Error opening unqlite: %i\n", (int)res);
        return 0;
    }

    srand(time(NULL));

    char buffer[255];
    unqlite_int64 inout;

    int block = 0;
    int count = 0;

    while(1)
    {
        int keyLen;
        int valLen;
        char *key = randomBytes(&keyLen);
        char *val = randomBytes(&valLen);

        // insert random key
        res = unqlite_kv_store(db, key, keyLen, val, valLen);
        if (res != UNQLITE_OK)
        {
            printf("Unable to store: %i\n", (int)res);
            return 0;
        }

        // verify it was stored correctly
        inout = 255;
        res = unqlite_kv_fetch(db, key, keyLen, &buffer[0], &inout);
        if (res != UNQLITE_OK)
        {
            printf("Unable to fetch: %i", (int)res);
            return 0;
        }

        // generate new random key and delete if it exists
        free(key);
        key = randomBytes(&keyLen);

        inout = 255;
        res = unqlite_kv_fetch(db, key, keyLen, &buffer[0], &inout);
        if (res == UNQLITE_OK)
        {
            res = unqlite_kv_delete(db, key, keyLen);
            if (res != UNQLITE_OK)
            {
                printf("Unable to delete: %i\n", (int)res);
                return 0;
            }
        }

        free(key);
        free(val);

        count++;
        if (count > 10000)
        {
            block++;
            count = 0;
            printf("%i 10K iterations\n", block);
        }
    }
    return 0;
}

After merging the provided PR's could you please update the README in the master to add the build badge from Travis. So everyone visiting the repository can see the status of the latest master build.