couchbaselabs / couchstore

couchbase storage file library

Couchbase (.couch) storage file C library

Currently this library can only be built as part of Couchbase Server due to dependencies on the Couchbase Platform Library and Couchbase Server CMake project. For instructions on building Couchbase Server, see the Manifest Repository.

Tests:

1. make test

This will run the native tests, and also the Lua tests if Lua was installed at the time the configure script ran.

Tests use the CMake CTest system, and the ctest command can be used to run invidual tests and print verbose output.

couchstore's Issues

Question

Can multiple programs/processes access the same DB file at the same time using this library?

Wrote out 61M items, could only read back 34M

Using the following scripts:

To populate the data:

local db = couch.open("/tmp/test.couch", true)

local alphabet = "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789"

local large_enough = 4096
local largestring = ""
while #largestring < large_enough do
   local i = math.random(1, #alphabet)
   largestring = largestring .. string.sub(alphabet, i, i, 1)
end

local start = os.time()
for i = 0, 60, 1
do
   print("Doing " .. i)
   local t = {}
   local k = "k." .. i .. "."
   local v = "value" .. i .. " - "
   for j = 0, 1000000, 1
   do
      table.insert(t, {k .. j, v .. j, 1})
   end
   db:save_bulk(t)
   local e1 = os.time()
   db:commit()
   local e2 = os.time()
   print(string.format("Completed in in %d seconds (commit took %d)",
                       os.difftime(e2, start), os.difftime(e2, e1)))
   start = e2
end

...and to read back

local db = couch.open("/tmp/test.couch")

local count = 0
db:changes(0, function(x) count = count + 1 end)
print("Got " .. count .. " items")

Dependancies

Trying to build it errors out asking for v8? Its not listed on the front page, I'm going down the dependency rabbit hole now.

checking for library containing fabs... none required
checking for pod2man... /usr/bin/pod2man
checking snappy-c.h usability... yes
checking snappy-c.h presence... yes
checking for snappy-c.h... yes
checking netinet/in.h usability... yes
checking netinet/in.h presence... yes
checking for netinet/in.h... yes
checking for inttypes.h... (cached) yes
checking for libsnappy... yes
checking for icu-config... /opt/couchbase/bin/icu-config
checking for ICU >= 3.4.1... yes
checking ICU_CFLAGS... -O3 -ggdb3 -Wall -ansi -pedantic -Wshadow -Wpointer-arith -Wmissing-prototypes -Wwrite-strings -Wno-long-long
checking ICU_CXXFLAGS... -O3 -ggdb3 -W -Wall -ansi -pedantic -Wpointer-arith -Wwrite-strings -Wno-long-long
checking ICU_LIBS... -lpthread -ldl -lm -L/opt/couchbase/lib -licui18n -licuuc -licudata -lpthread -ldl -lm
checking v8.h usability... no
checking v8.h presence... no
checking for v8.h... no
configure: error: Could not find the V8 JavaScript engine library.

update an downlode file from couchbase

how update an downlode file from couchbase?

Larger bodies cause really bad problems on OS X

I have a failing test if I can figure out how to get it attached...

Implement save_docs in lua

I'm thinking something that just takes a table of tables where the outer table is an array of inner tables that are arrays similar to the arguments used by the single document save_docs.

build failure

Hi,

I am trying to compile couchstore. After having all the required deps installed, make splits out following error:

root@dd6cd8e69007:/opt/couchstore# make
make  all-am
make[1]: Entering directory `/opt/couchstore'
  CXX    src/libcouchstore_la-arena.lo
In file included from ./config.h:132:0,
                 from src/arena.cc:8:
./config_static.h:59:31: fatal error: platform/platform.h: No such file or directory
compilation terminated.
make[1]: *** [src/libcouchstore_la-arena.lo] Error 1
make[1]: Leaving directory `/opt/couchstore'
make: *** [all] Error 2

Any hint on how to fix this?

couchstore is out?

Hi, I have some confusions. ForestDB has better read and write performance than couchstore.
Why use couchstore for storage under data service for both database engine and for view engine?
why not ForestDB?

is the lib working with current apache couchdb format?

Which version of v8 is required?

Trying to compile couchstore on OS X with V8 3.22.11 I get

$ make
/Applications/Xcode.app/Contents/Developer/usr/bin/make  all-am
  CXX      src/views/mapreduce/libmapreduce_la-mapreduce.lo
src/views/mapreduce/mapreduce.cc:102:33: error: unknown type name 'Arguments'; did you mean 'v8::internal::Arguments'?
static Handle<Value> emit(const Arguments &args);
                                ^~~~~~~~~
                                v8::internal::Arguments
/opt/local/include/v8.h:141:7: note: 'v8::internal::Arguments' declared here
class Arguments;
      ^
src/views/mapreduce/mapreduce.cc:121:21: error: calling a private constructor of class 'v8::HandleScope'
        HandleScope handleScope;
                    ^
/opt/local/include/v8.h:767:13: note: declared private here
  V8_INLINE HandleScope() {}
            ^
src/views/mapreduce/mapreduce.cc:122:24: error: no matching constructor for initialization of 'Context::Scope'
        Context::Scope contextScope(ctx->jsContext);
                       ^            ~~~~~~~~~~~~~~
/opt/local/include/v8.h:5057:9: note: candidate constructor (the implicit copy constructor) not viable: no known conversion from
      'v8::Persistent<v8::Context>' to 'const v8::Context::Scope' for 1st argument
  class Scope {
        ^
/opt/local/include/v8.h:5059:24: note: candidate constructor not viable: no known conversion from 'v8::Persistent<v8::Context>' to
      'Handle<v8::Context>' for 1st argument
    explicit V8_INLINE Scope(Handle<Context> context) : context_(context) {
                       ^
/opt/local/include/v8.h:5064:19: note: candidate constructor not viable: requires 2 arguments, but 1 was provided
        V8_INLINE Scope(Isolate* isolate, Persistent<Context>& context)) // NOLINT
                  ^
/opt/local/include/v8config.h:333:45: note: expanded from macro 'V8_DEPRECATED'
# define V8_DEPRECATED(message, declarator) declarator
                                            ^
src/views/mapreduce/mapreduce.cc:137:21: error: calling a private constructor of class 'v8::HandleScope'
        HandleScope handleScope;
                    ^
/opt/local/include/v8.h:767:13: note: declared private here
  V8_INLINE HandleScope() {}
            ^
src/views/mapreduce/mapreduce.cc:138:24: error: no matching constructor for initialization of 'Context::Scope'
        Context::Scope contextScope(ctx->jsContext);
                       ^            ~~~~~~~~~~~~~~
/opt/local/include/v8.h:5057:9: note: candidate constructor (the implicit copy constructor) not viable: no known conversion from
      'v8::Persistent<v8::Context>' to 'const v8::Context::Scope' for 1st argument
  class Scope {
        ^
/opt/local/include/v8.h:5059:24: note: candidate constructor not viable: no known conversion from 'v8::Persistent<v8::Context>' to
      'Handle<v8::Context>' for 1st argument
    explicit V8_INLINE Scope(Handle<Context> context) : context_(context) {
                       ^
/opt/local/include/v8.h:5064:19: note: candidate constructor not viable: requires 2 arguments, but 1 was provided
        V8_INLINE Scope(Isolate* isolate, Persistent<Context>& context)) // NOLINT
                  ^
/opt/local/include/v8config.h:333:45: note: expanded from macro 'V8_DEPRECATED'
# define V8_DEPRECATED(message, declarator) declarator
                                            ^
src/views/mapreduce/mapreduce.cc:167:17: error: calling a private constructor of class 'v8::HandleScope'
    HandleScope handleScope;
                ^
/opt/local/include/v8.h:767:13: note: declared private here
  V8_INLINE HandleScope() {}
            ^
src/views/mapreduce/mapreduce.cc:170:20: error: no matching constructor for initialization of 'Context::Scope'
    Context::Scope contextScope(ctx->jsContext);
                   ^            ~~~~~~~~~~~~~~
/opt/local/include/v8.h:5057:9: note: candidate constructor (the implicit copy constructor) not viable: no known conversion from
      'v8::Persistent<v8::Context>' to 'const v8::Context::Scope' for 1st argument
  class Scope {
        ^
/opt/local/include/v8.h:5059:24: note: candidate constructor not viable: no known conversion from 'v8::Persistent<v8::Context>' to
      'Handle<v8::Context>' for 1st argument
    explicit V8_INLINE Scope(Handle<Context> context) : context_(context) {
                       ^
/opt/local/include/v8.h:5064:19: note: candidate constructor not viable: requires 2 arguments, but 1 was provided
        V8_INLINE Scope(Isolate* isolate, Persistent<Context>& context)) // NOLINT
                  ^
/opt/local/include/v8config.h:333:45: note: expanded from macro 'V8_DEPRECATED'
# define V8_DEPRECATED(message, declarator) declarator
                                            ^
src/views/mapreduce/mapreduce.cc:172:67: error: member reference type 'v8::Persistent<v8::Context>' is not a pointer
    Handle<Object> jsonObject = Local<Object>::Cast(ctx->jsContext->Global()->Get(String::New("JSON")));
                                                    ~~~~~~~~~~~~~~^
src/views/mapreduce/mapreduce.cc:177:47: error: 'New' is a private member of 'v8::Persistent<v8::Object,
      v8::NonCopyablePersistentTraits<v8::Object> >'
    isoData->jsonObject = Persistent<Object>::New(jsonObject);
                                              ^
/opt/local/include/v8.h:721:23: note: declared private here
  V8_INLINE static T* New(Isolate* isolate, T* that);
                      ^
src/views/mapreduce/mapreduce.cc:177:61: error: too few arguments to function call, expected 2, have 1
    isoData->jsonObject = Persistent<Object>::New(jsonObject);
                          ~~~~~~~~~~~~~~~~~~~~~~~           ^
/opt/local/include/v8.h:721:3: note: 'New' declared here
  V8_INLINE static T* New(Isolate* isolate, T* that);
  ^
/opt/local/include/v8config.h:302:20: note: expanded from macro 'V8_INLINE'
# define V8_INLINE inline __attribute__((always_inline))
                   ^
src/views/mapreduce/mapreduce.cc:178:51: error: 'New' is a private member of 'v8::Persistent<v8::Function,
      v8::NonCopyablePersistentTraits<v8::Function> >'
    isoData->jsonParseFun = Persistent<Function>::New(parseFun);
                                                  ^
/opt/local/include/v8.h:721:23: note: declared private here
  V8_INLINE static T* New(Isolate* isolate, T* that);
                      ^
src/views/mapreduce/mapreduce.cc:178:63: error: too few arguments to function call, expected 2, have 1
    isoData->jsonParseFun = Persistent<Function>::New(parseFun);
                            ~~~~~~~~~~~~~~~~~~~~~~~~~         ^
/opt/local/include/v8.h:721:3: note: 'New' declared here
  V8_INLINE static T* New(Isolate* isolate, T* that);
  ^
/opt/local/include/v8config.h:302:20: note: expanded from macro 'V8_INLINE'
# define V8_INLINE inline __attribute__((always_inline))
                   ^
src/views/mapreduce/mapreduce.cc:179:51: error: 'New' is a private member of 'v8::Persistent<v8::Function,
      v8::NonCopyablePersistentTraits<v8::Function> >'
    isoData->stringifyFun = Persistent<Function>::New(stringifyFun);
                                                  ^
/opt/local/include/v8.h:721:23: note: declared private here
  V8_INLINE static T* New(Isolate* isolate, T* that);
                      ^
src/views/mapreduce/mapreduce.cc:179:67: error: too few arguments to function call, expected 2, have 1
    isoData->stringifyFun = Persistent<Function>::New(stringifyFun);
                            ~~~~~~~~~~~~~~~~~~~~~~~~~             ^
/opt/local/include/v8.h:721:3: note: 'New' declared here
  V8_INLINE static T* New(Isolate* isolate, T* that);
  ^
/opt/local/include/v8config.h:302:20: note: expanded from macro 'V8_INLINE'
# define V8_INLINE inline __attribute__((always_inline))
                   ^
src/views/mapreduce/mapreduce.cc:189:17: error: calling a private constructor of class 'v8::HandleScope'
    HandleScope handleScope;
                ^
/opt/local/include/v8.h:767:13: note: declared private here
  V8_INLINE HandleScope() {}
            ^
src/views/mapreduce/mapreduce.cc:194:54: error: no viable conversion from 'Handle<v8::ObjectTemplate>' to
      'v8::ExtensionConfiguration *'
    Persistent<Context> context = Context::New(NULL, global);
                                                     ^~~~~~
/opt/local/include/v8.h:4948:31: note: passing argument to parameter 'extensions' here
      ExtensionConfiguration* extensions = NULL,
                              ^
src/views/mapreduce/mapreduce.cc:195:20: error: no matching constructor for initialization of 'Context::Scope'
    Context::Scope contextScope(context);
                   ^            ~~~~~~~
/opt/local/include/v8.h:5057:9: note: candidate constructor (the implicit copy constructor) not viable: no known conversion from
      'Persistent<v8::Context>' to 'const v8::Context::Scope' for 1st argument
  class Scope {
        ^
/opt/local/include/v8.h:5059:24: note: candidate constructor not viable: no known conversion from 'Persistent<v8::Context>' to
      'Handle<v8::Context>' for 1st argument
    explicit V8_INLINE Scope(Handle<Context> context) : context_(context) {
                       ^
/opt/local/include/v8.h:5064:19: note: candidate constructor not viable: requires 2 arguments, but 1 was provided
        V8_INLINE Scope(Isolate* isolate, Persistent<Context>& context)) // NOLINT
                  ^
/opt/local/include/v8config.h:333:45: note: expanded from macro 'V8_DEPRECATED'
# define V8_DEPRECATED(message, declarator) declarator
                                            ^
src/views/mapreduce/mapreduce.cc:198:12: error: member reference type 'Persistent<v8::Context>' is not a pointer
    context->Global()->Set(String::New("sum"), sumFun);
    ~~~~~~~^
src/views/mapreduce/mapreduce.cc:201:12: error: member reference type 'Persistent<v8::Context>' is not a pointer
    context->Global()->Set(String::New("decodeBase64"), decodeBase64Fun);
    ~~~~~~~^
fatal error: too many errors emitted, stopping now [-ferror-limit=]
20 errors generated.
make[1]: *** [src/views/mapreduce/libmapreduce_la-mapreduce.lo] Error 1
make: *** [all] Error 2

Write docs independently from b-trees

This is aligned with Damien's "write docs as soon as possible." CouchDB will allocate space in the file and fill it separately. CouchDB is, however, a concurrent system. I'm investigating ways to help us express concurrency in more areas.

couchbase cluster failure

I am using couchbase 4.0. There was sudden restart of couchbase servers. after restart cluster is not functional
Form this https://issues.couchbase.com/browse/MB-16826 seems like bug in 4.0. just wanted to know recovery process.
below is the error
{error_logger,{{2016,3,21},{14,25,7}},"Protocol: ~tp: the name [email protected] seems to be in use by another Erlang node",["inet_tcp"]}
{error_logger,{{2016,3,21},{14,25,7}},crash_report,[[{initial_call,{net_kernel,init,['Argument__1']}},{pid,<0.21.0>},{registered_name,[]},{error_info,{exit,{error,badarg},[{gen_server,init_it,6,[{file,"gen_server.erl"},{line,320}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,239}]}]}},{ancestors,[net_sup,kernel_sup,<0.10.0>]},{messages,[]},{links,[#Port<0.64>,<0.18.0>]},{dictionary,[{longnames,true}]},{trap_exit,true},{status,running},{heap_size,987},{stack_size,27},{reductions,1345}],[]]}
{error_logger,{{2016,3,21},{14,25,7}},supervisor_report,[{supervisor,{local,net_sup}},{errorContext,start_error},{reason,{'EXIT',nodistribution}},{offender,[{pid,undefined},{name,net_kernel},{mfargs,{net_kernel,start_link,[['[email protected]',longnames]]}},{restart_type,permanent},{shutdown,2000},{child_type,worker}]}]}
{error_logger,{{2016,3,21},{14,25,7}},supervisor_report,[{supervisor,{local,kernel_sup}},{errorContext,start_error},{reason,{shutdown,{failed_to_start_child,net_kernel,{'EXIT',nodistribution}}}},{offender,[{pid,undefined},{name,net_sup},{mfargs,{erl_distribution,start_link,[]}},{restart_type,permanent},{shutdown,infinity},{child_type,supervisor}]}]}
{error_logger,{{2016,3,21},{14,25,7}},crash_report,[[{initial_call,{application_master,init,['Argument__1','Argument__2','Argument__3','Argument__4']}},{pid,<0.9.0>},{registered_name,[]},{error_info,{exit,{{shutdown,{failed_to_start_child,net_sup,{shutdown,{failed_to_start_child,net_kernel,{'EXIT',nodistribution}}}}},{kernel,start,[normal,[]]}},[{application_master,init,4,[{file,"application_master.erl"},{line,133}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,239}]}]}},{ancestors,[<0.8.0>]},{messages,[{'EXIT',<0.10.0>,normal}]},{links,[<0.8.0>,<0.7.0>]},{dictionary,[]},{trap_exit,true},{status,running},{heap_size,376},{stack_size,27},{reductions,117}],[]]}
{error_logger,{{2016,3,21},{14,25,7}},std_info,[{application,kernel},{exited,{{shutdown,{failed_to_start_child,net_sup,{shutdown,{failed_to_start_child,net_kernel,{'EXIT',nodistribution}}}}},{kernel,start,[normal,[]]}}},{type,permanent}]}
{"Kernel pid terminated",application_controller,"{application_start_failure,kernel,{{shutdown,{failed_to_start_child,net_sup,{shutdown,{failed_to_start_child,net_kernel,{'EXIT',nodistribution}}}}},{kernel,start,[normal,[]]}}}"}

Crash dump was written to: erl_crash.dump.1458550506.30171.babysitter
Kernel pid terminated (application_controller) ({application_start_failure,kernel,{{shutdown,{failed_to_start_child,net_sup,{shutdown,{failed_to_start_child,net_kernel,{'EXIT',nodistribution}}}}},{k

Style cleanups

I'd like to get the code consistent with other C++ code (ep-engine basically uses cc-mode out of the box).

This is generally minor, but one thing I do think is quite important is adding lots of curly braces. Optional curly braces are as evil as optional semicolons. Once you start putting the opening sem on the same line as the thing it's opening, you don't worry so much about wasting lines with them. :)

I think we can simplify a lot of things with some more C++ idioms as well (primarily around making sure things are guaranteed to be cleaned up).

Report test coverage

We need a coverage report from the tests. Uncovered lines should either be removed or tested.

attachments

It seems there is no way to manage an attachments in couchstore. Or at least I don't find it. Any idea how to do that?

Large files don't work.

The following test fails on my mac. I do not believe it should. (still waiting for my FreeBSD box to return -- taking forever):

package.path = package.path .. ";tests/?.lua"
local testlib = require("testlib")

function bigdb(dbname)
   local db = couch.open(dbname, true)
   db:truncate(5 * 1024 * 1024 * 1024)
   db:close()

   db = couch.open(dbname)
   testlib.assert_no_doc(db, "k")

   db:save("k", "a value", 1)
   db:commit()
   db:close()

   db = couch.open(dbname)
   testlib.check_doc(db, "k", "a value")
end

testlib.run_test("A Big Database", bigdb)

Meta data purge in couchbase

I am using couchbase 4.1. when i delete the document, a new revision of document is getting created with attribute deleted=True. when compaction runs older revisions are getting deleted but am curious to know when the latest document (which is having only meta data) is going to delete from couchbase permanently. same i have posted in forum . but no luck.
Is there any setting i am missing ?. also wanted to know will it delete from replica too.