GithubHelp home page GithubHelp logo

cloudant / bigcouch Goto Github PK

View Code? Open in Web Editor NEW
567.0 567.0 52.0 9.83 MB

Putting the 'C' back in CouchDB

Home Page: http://bigcouch.cloudant.com/

License: Apache License 2.0

C 1.99% Erlang 24.10% Shell 0.27% JavaScript 21.58% Ruby 0.64% Python 48.62% Makefile 0.07% C++ 0.03% HTML 1.95% CSS 0.76%

bigcouch's Introduction

Cloudant

Please use https://github.ibm.com/cloud-docs/Cloudant for contributions. Pushes to https://github.com/ibm-cloud-docs/Cloudant are only allowed by members of documentation team.

Build & CI

Jenkins

Jenkins is used to publish docs to staging and to production. The setup is managed by IBM Cloud Documentation team.

https://wcp-ace-docs-jenkins.swg-devops.com/job/Docs-build/job/Docs-build-Cloudant/

Travis

Travis is used to keep github.com and github.ibm.com repositories in sync. The setup is managed by Cloudant infra team.

https://travis.ibm.com/Bluemix-Docs

Documentation

Please use https://github.ibm.com/cloud-docs/Cloudant for contributions to the documentation. Pushes to https://github.com/ibm-cloud-docs/Cloudant are only allowed by members of the documentation team. See Documentation Update Process for instructions.

If you are not an IBM employee and want to make a documentation contribution, go to the IBM Cloudant documentation and click Feedback on the page where you want to comment.

bigcouch's People

Contributors

benoitc avatar boorad avatar cmlenz avatar davisp avatar fdmanana avatar janl avatar jasondavies avatar jchris avatar joewilliams avatar kocolosk avatar rnewson avatar steadicat avatar tilgovi avatar till avatar xylakant avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

bigcouch's Issues

Build failure (git fetch --tags origin failed with error)

[user1@neptune ~]$ git clone git://github.com/cloudant/bigcouch.git
Cloning into bigcouch...
remote: Counting objects: 15409, done.
remote: Compressing objects: 100% (5423/5423), done.
remote: Total 15409 (delta 11396), reused 13283 (delta 9731)
Receiving objects: 100% (15409/15409), 3.75 MiB | 288 KiB/s, done.
Resolving deltas: 100% (11396/11396), done.
[user1@neptune ~]$ cd bigcouch/
[user1@neptune bigcouch]$ ./configure
==> configuring bigcouch in rel/bigcouch.config
==> oauth-1.0.1 (get-deps)
==> ibrowse-2.1.0 (get-deps)
==> mochiweb-1.4.1 (get-deps)
==> couch (get-deps)
==> rel (get-deps)
==> bigcouch (get-deps)
Pulling rexi from {git,"git://github.com/cloudant/rexi.git","master"}
Cloning into rexi...
Pulling fabric from {git,"git://github.com/cloudant/fabric.git","master"}
Cloning into fabric...
Pulling mem3 from {git,"git://github.com/cloudant/mem3.git","master"}
Cloning into mem3...
Pulling chttpd from {git,"git://github.com/cloudant/chttpd.git","master"}
Cloning into chttpd...
Updating oauth from {git,"git://github.com/cloudant/erlang-oauth.git",
{tag,"CouchDB-1.0.1-rebar"}}
ERROR: git fetch --tags origin failed with error: 1
[user1@neptune bigcouch]$ git version
git version 1.7.3.2

replication with couchdb

replication in couchdb due to the presence of unkown properties in db infos (n, q, ..). This changue was introduced in a recent version of bigcouch.

could not load validation funs

I'm seeing errors when replicating an existing dataset into bigcouch 0.3.1
Bigcouch is throwing errors with the design documents
There doesn't seem to be any problem with the docs on couch 1.0.3 and they do get replicated into bigcouch (I can query them) but these errors appear in the logs.

could not load validation funs {{badmatch, {function_clause, [{fabric,'-design_docs/1-fun-0-', [{error,timeout},

Error in process <0.12889.45> on node '[email protected]' with exit value: {function_clause
,[{fabric,'-design_docs/1-fun-0-',[{error,{noproc,{gen_server,call,[<0.25484.43>,{pread_iolist,77411342},infinity]},[

more log info here: http://pastebin.com/xgX2YR7G

The list of design docs in the stack trace is huge and doesn't seem to show which one is problematic

setup: This is the bigcouch 0.3.1 compiled rpm
On CentOS 5.6 64bit
curl: 7.20.1
erlang: R13B04 (erts-5.7.5)
icu: 4.4.1
js: js 1.7

in a 3 node ring on 3 VMs, replicating from couchdb 1.0.3 on a local network

If R is more than the number of available copies (e.g. 2 > 1) - 500 unknown_error is returned

I have a cluster of 3 nodes with the following config: Q = 1, N = 2, R = 2, W = 2. When one of my nodes is down and I try to access a document, one of replicas of which was on the downed node, I get the following error with http 500 status code:
{"error":"unknown_error","reason":"timeout"}

This error is not informative at all to be handled on the client code. Secondly, I am not sure the read should fail in that case. May be success with a special http status code (not 200), indicating that the read succeeded, but consistency is not guaranteed (e.g. 203 Non-Authoritative Information).

Thanx

infinity loop with _replicator

On the 0.4-rc branch, when a replication with _replicator db fail (for bad json encoding in source document) the replication goes in loop and block other processing.

To reproduce try to replicate https://upondata.com/afgwar :

  $ curl -XPUT localhost:5984/_replicator/afgwar
  $ curl -XPUT -d'{"source": "https://upondata.com/afgwar", "target": "http://localhost:5984/afgwar"}' localhost:5984/_replicator/afgwar

Then wait. Test was done on a machine with haproxy on top of the 3 devs instances generated with make dev

Odd error when running highly concurrent updates on a key

Version 0.3a
checkout "f91eabafccb3a1ebae9ac770d6ad3d64efcf47dc"

Happens when running 200+ concurrent updates on a single key on a 3 node big couch setup.

Updates are issued round-robin to all 3 nodes. Same result with r=w=2 n=3, and r=w=n=3

[Thu, 23 Dec 2010 23:09:03 GMT] [info] [<0.341.125>] [73515be6] 172.0.128.3 cl1.toystudio.net GET /cd/economy_data 404 1

[Thu, 23 Dec 2010 23:09:04 GMT] [error] [<0.327.125>] [aa0b11a3] function_clause error in HTTP request

[Thu, 23 Dec 2010 23:09:04 GMT] [info] [<0.327.125>] [aa0b11a3] Stacktrace: [{lists,nthtail,[5,[]]},
{fabric_util,'-remove_ancestors/2-fun-1-',3},
{lists,dropwhile,2},
{fabric_util,remove_ancestors,2},
{fabric_doc_open,handle_message,3},
{rexi_utils,process_mailbox,6},
{rexi_utils,recv,6},
{fabric_doc_open,go,3}]

couch test suites are not successful

I have set up a small cluster (2 machines).

couchdb test suites when run fail. Should they run successfully on bigcouch?

Also, the nodes will only remain running for a short period of time.

Timeouts accessing shards

I have clustered three nodes locally on my laptop. Before I restarted the computer, all were operating fine. After restart, I can load Futon and the list of databases but when I try to view a database (via Futon or cURL hitting the load balancer), I get 504 Gateway Timeouts. When I try to view the database going directly to a node's interface, I see the following error(s):

[Mon, 09 May 2011 22:39:23 GMT] [error] [<0.3324.0>] [e8d2a9fd] Uncaught error in HTTP request: {error,
{badmatch,
{timeout,
{[{{shard,
<<"shards/00000000-1fffffff/account/b6/4c/0daa3c49e0c0c8e0a4a5d8026fa3.1304483578">>,
'[email protected]',
<<"account/b6/4c/0daa3c49e0c0c8e0a4a5d8026fa3">>,
[0,536870911],
#Ref<0.0.0.10898>},
nil},
{{shard,
<<"shards/00000000-1fffffff/account/b6/4c/0daa3c49e0c0c8e0a4a5d8026fa3.1304483578">>,
'[email protected]',
<<"account/b6/4c/0daa3c49e0c0c8e0a4a5d8026fa3">>,
[0,536870911],
#Ref<0.0.0.10906>},
nil}, .....
and so on, listing each shard in the DB across nodes. This is followed by:
[Mon, 09 May 2011 22:39:23 GMT] [info] [<0.3324.0>] [e8d2a9fd] Stacktrace: [{chttpd_db,db_req,2},
{chttpd,handle_request,1},
{mochiweb_http,headers,5},
{proc_lib,init_p_do_apply,3}]

I am on the latest code, the shard files exist on the local disk, and this was all working prior to the restart.

Creating a new DB via load balancer times out, which going directly to a node results in these errors:

[Tue, 10 May 2011 00:07:39 GMT] [error] [<0.11766.0>] [fceca23c] Uncaught error in HTTP request: {error,
{case_clause,
{timeout,
[{{shard,undefined,
'[email protected]',
undefined,undefined,#Ref<0.0.0.38046>},
ok},
{{shard,undefined,
'[email protected]',
undefined,undefined,#Ref<0.0.0.38047>},
nil},
{{shard,undefined,
'[email protected]',
undefined,undefined,#Ref<0.0.0.38048>},
nil},
{{shard,undefined,
'[email protected]',
undefined,undefined,#Ref<0.0.0.38049>},
nil}]}}}

[Tue, 10 May 2011 00:07:39 GMT] [info] [<0.11766.0>] [fceca23c] Stacktrace: [{fabric_db_create,go,2},
{chttpd_db,create_db_req,2},
{chttpd,handle_request,1},
{mochiweb_http,headers,5},
{proc_lib,init_p_do_apply,3}]

bigcouch is crashing

I'm penetrating my bigcouch with a little nodejs script. I'm using bigcouch on one node with python-views. /opt/bigcouch/bin/bigcouch gives me the following error:

=ERROR REPORT==== 8-Nov-2010::13:35:09 ===
application: mochiweb
"Accept failed error"
"{error,enfile}"
[error] [<0.17733.0>] {error_report,<0.139.0>,
{<0.17733.0>,crash_report,
[[{initial_call,{mochiweb_socket_server,acceptor_loop,['Argument__1']}},
{pid,<0.17733.0>},
{registered_name,[]},
{error_info,
{exit,
{error,accept_failed},
[{mochiweb_socket_server,acceptor_loop,1},
{proc_lib,init_p_do_apply,3}]}},
{ancestors,[chttpd,chttpd_sup,<0.140.0>]},
{messages,[]},
{links,[<0.142.0>]},
{dictionary,[]},
{trap_exit,false},
{status,running},
{heap_size,233},
{stack_size,24},
{reductions,202}],
[]]}}

=CRASH REPORT==== 8-Nov-2010::13:35:09 ===
crasher:
initial call: mochiweb_socket_server:acceptor_loop/1
pid: <0.17733.0>
registered_name: []
exception exit: {error,accept_failed}
in function mochiweb_socket_server:acceptor_loop/1
ancestors: [chttpd,chttpd_sup,<0.140.0>]
messages: []
links: [<0.142.0>]
dictionary: []
trap_exit: false
status: running
heap_size: 233
stack_size: 24
reductions: 202
neighbours:
[error] [<0.142.0>] {error_report,<0.139.0>,
{<0.142.0>,std_error,
{mochiweb_socket_server,225,{acceptor_error,{error,accept_failed}}}}}

in the log-files there's an error logged, but nothing with useful information in my eyes:

[Mon, 08 Nov 2010 11:40:29 GMT] [error] [<0.9713.1>] {error_report,<0.133.0>,
{<0.9713.1>,std_error,
[{rexi_server,
{exit,
{timeout,
{gen_server,call,
[couch_view,
{get_group_server,<<"shards/c0000000-cfffffff/copy">>,
{group,
<<149,109,51,188,116,201,165,49,186,12,141,66,
178,202,103,132>>,
undefined,nil,<<"_design/urls">>,<<"python">>,
[],
[{view,0,
[<<"out">>],
// some views
nil,0,0,nil,nil}}]}}}},
[{gen_server,call,2},{fabric_rpc,map_view,4},{rexi_server,init_p,2}]]}}

Fail to Join Nodes with Amazon EC2

Hi, there:

I am trying to use Amazon EC2 to create a Bigcouch Cluster. I installed and launched BigCouch on several nodes, and tried to use curl -X PUT as suggested on github page to join all other nodes on the first node. And I get the following error:

=CRASH REPORT==== 5-Dec-2010::20:00:29 ===
crasher:
initial call: couch_rep:init/1
pid: <0.295.0>
registered_name: []
exception exit: {node_not_connected,
<<"[email protected]">>}
in function gen_server:init_it/6
ancestors: [couch_rep_sup,couch_primary_services,couch_server_sup,
<0.86.0>]
messages: []
links: [<0.95.0>]
dictionary: []
trap_exit: true
status: running
heap_size: 987
stack_size: 24
reductions: 219
neighbours:

I am pretty sure it is not firewall thing because I enable all relevant tcp ports. And I can ping all the other nodes from the first node. I have also copied Erlang cookie to all other nodes from the first node.

Is there any other reason that may cause this?

Thank you!

configure failed with latest bigcouch download and git 1.5.x

Tried to install the latest bigcouch package ended up with error:

==> configuring bigcouch in rel/bigcouch.config
==> couch (get-deps)
==> etap (get-deps)
==> rel (get-deps)
==> cloudant-bigcouch-292dca2 (get-deps)
Pulling oauth from {git,"git://github.com/cloudant/erlang-oauth.git",
{tag,"CouchDB-1.0.1-rebar"}}
Initialized empty Git repository in /opt/cloudant-bigcouch-292dca2/deps/oauth/.git/
Switched to a new branch "CouchDB-1.0.1-rebar"
Pulling ibrowse from {git,"git://github.com/cloudant/ibrowse.git",
{branch,"couch"}}
Initialized empty Git repository in /opt/cloudant-bigcouch-292dca2/deps/ibrowse/.git/
error: pathspec 'couch' did not match any file(s) known to git.
ERROR: git checkout couch failed with error: 1

Clean node added to cluster won't receive copies of data - 0.3.0

When adding a new node to a cluster, no data is replicated to the new node even if the value of N specifies that it should.

Is this intended behaviour?

Reproduction steps:
Run "make dev"
Start node dev1
Start node dev2
Run "curl -X PUT http://127.0.0.1:25986/nodes/[email protected] -d {}"
Run "curl -X PUT http://127.0.0.1:25984/test"
Run "curl -X PUT http://127.0.0.1:25984/test/document1 -d {}"
Run "curl -X PUT http://127.0.0.1:25984/test/document2 -d {}"
Start node dev3
Run "curl -X PUT http://127.0.0.1:35986/nodes/[email protected] -d {}"

Result:
Although node dev3 is a fully signed up member of the cluster and can service requests as normal, no data will ever be replicated to it. Even with an N value of 3 meaning that documents should be written to all three nodes, new documents will only be replicated to nodes dev1 and dev2.

Executing "curl http://127.0.0.1:35986/_all_dbs" only returns the following:
["_users","dbs","nodes"]

What is the effect of a small write number on a large number of documents

We are currently looking into using bigcouch, and have a test cluster of 6 nodes. All registered correctly and are part of the cluster as evidenced by:

{"all_nodes":["[email protected]","[email protected]","[email protected]","[email protected]","[email protected]","[email protected]"],"cluster_nodes":["[email protected]","[email protected]","[email protected]","[email protected]","[email protected]","[email protected]"]}

The default cluster options are: q=8, r=2, w=2, n=3. I'm noticing that when I put a new document with a w of 2, it gets replicated to all 6 nodes, instead of just the 2 nodes that need to take place. For the current project, we are looking at millions of documents, and would prefer not to have them replicated to all of the nodes in the cluster, but distributed randomly. The documentation is not real clear on if this is the intended behavior or not.

This is using the latest version of bigcouch.

Thanks for your help.

_users DB shuts down after one minute of running

When starting a new node for the first time, I get the following error after about a minute or so (full log at bottom):

[error] [<0.100.0>] [--------] DB _users shutting down - Fd normal
[error] [<0.100.0>] [--------] ** Generic server <0.100.0> terminating

This is without having made any requests to the node or starting any other nodes.

Reproduction steps:

  • Clone from Github
  • Checkout "bigcouch-0.3.0"
  • Run configure script
  • Run make dev
  • Run rel/dev1/bin/bigcouch
  • Wait about a minute

Environment:
Centos 5.5
Erlang R14A
ICU 4.4
Spidermonkey 1.8.0
curl 7.21.1

Is this anything to worry about?

Apache CouchDB 1.0.2 (LogLevel=info) is starting.
Apache CouchDB has started. Time to relax.
[info] [<0.79.0>] [--------] Apache CouchDB has started on http://127.0.0.1:35986/
[error] [<0.100.0>] [--------] DB _users shutting down - Fd normal
[error] [<0.100.0>] [--------] ** Generic server <0.100.0> terminating
** Last message in was {'DOWN',#Ref<0.0.0.128>,process,<0.99.0>,normal}
** When Server state == {db,<0.100.0>,nil,nil,<<"1297872582322583">>,<0.99.0>,
                            #Ref<0.0.0.128>,
                            {db_header,5,1,0,
                                {971,{1,0}},
                                {1062,1},
                                nil,0,nil,nil,1000},
                            1,
                            {btree,<0.99.0>,
                                {971,{1,0}},
                                #Fun,
                                #Fun,
                                undefined,
                                #Fun},
                            {btree,<0.99.0>,
                                {1062,1},
                                #Fun,
                                #Fun,
                                undefined,
                                #Fun},
                            {btree,<0.99.0>,nil,undefined,undefined,
                                undefined,nil},
                            1,<<"_users">>,
                            "/home/adrian/bigcouch-0.3/rel/tmpdata/dev3/_users.couch",
                            [#Fun],
                            [],nil,
                            {user_ctx,null,[],undefined},
                            nil,1000,
                            [before_header,after_header,on_file_open],
                            true}
** Reason for termination ==
** {noproc,{gen_server,call,[<0.99.0>,close,infinity]}}


=ERROR REPORT==== 16-Feb-2011::16:10:42 ===
** Generic server <0.100.0> terminating
** Last message in was {'DOWN',#Ref<0.0.0.128>,process,<0.99.0>,normal}
** When Server state == {db,<0.100.0>,nil,nil,<<"1297872582322583">>,<0.99.0>,
                            #Ref<0.0.0.128>,
                            {db_header,5,1,0,
                                {971,{1,0}},
                                {1062,1},
                                nil,0,nil,nil,1000},
                            1,
                            {btree,<0.99.0>,
                                {971,{1,0}},
                                #Fun,
                                #Fun,
                                undefined,
                                #Fun},
                            {btree,<0.99.0>,
                                {1062,1},
                                #Fun,
                                #Fun,
                                undefined,
                                #Fun},
                            {btree,<0.99.0>,nil,undefined,undefined,
                                undefined,nil},
                            1,<<"_users">>,
                            "/home/adrian/bigcouch-0.3/rel/tmpdata/dev3/_users.couch",
                            [#Fun],
                            [],nil,
                            {user_ctx,null,[],undefined},
                            nil,1000,
                            [before_header,after_header,on_file_open],
                            true}
** Reason for termination ==
** {noproc,{gen_server,call,[<0.99.0>,close,infinity]}}
[error] [<0.100.0>] [--------] {error_report,<0.78.0>,
    {<0.100.0>,crash_report,
     [[{initial_call,{couch_db_updater,init,['Argument__1']}},
       {pid,<0.100.0>},
       {registered_name,[]},
       {error_info,
           {exit,
               {noproc,{gen_server,call,[<0.99.0>,close,infinity]}},
               [{gen_server,terminate,6},{proc_lib,init_p_do_apply,3}]}},
       {ancestors,[<0.98.0>]},
       {messages,[]},
       {links,[<0.86.0>]},
       {dictionary,[]},
       {trap_exit,false},
       {status,running},
       {heap_size,2584},
       {stack_size,24},
       {reductions,3122}],
      []]}}

=CRASH REPORT==== 16-Feb-2011::16:10:42 ===
  crasher:
    initial call: couch_db_updater:init/1
    pid: <0.100.0>
    registered_name: []
    exception exit: {noproc,{gen_server,call,[<0.99.0>,close,infinity]}}
      in function  gen_server:terminate/6
    ancestors: [<0.98.0>]
    messages: []
    links: [<0.86.0>]
    dictionary: []
    trap_exit: false
    status: running
    heap_size: 2584
    stack_size: 24
    reductions: 3122
  neighbours:
[info] [<0.86.0>] [--------] db _users died with reason {noproc,
                               {gen_server,call,[<0.99.0>,close,infinity]}}

Build fails with couch_view_updater.erl:149: type boolean() undefined

Derrick P. reports this build error when compiling on Centos 5.4:

==> couchjs (compile) scons: Reading SConscript files ...
Checking for C library m... (cached) yes
Checking for C library pthread... (cached) yes
Checking for C library curl... (cached) yes
Checking for C header file js/jsapi.h... (cached) no
Checking for C header file mozjs/jsapi.h... (cached) no
Checking for C header file jsapi.h... (cached) yes
Checking for C library mozjs... (cached) no
Checking for C library js... (cached) yes
Checking whether JS_SetOperationCallback is declared... (cached) no
scons: done reading SConscript files.
scons: Building targets ...
scons: `.' is up to date.
scons: done building targets.
==> ibrowse (compile) ==> couch (compile) src/couch_view_updater.erl:112: type boolean() undefined
src/couch_view_updater.erl:149: type boolean() undefined
make: *** [compile] Error 1

Erlang crash due to insufficient memory

How much memory does bigcouch try to use. I have a small cluster that keeps crashing because it runs out of memory, at least that is the erlang crash dump file is indicating. Can I set the maximum memory allocation in vm.args. I started looking at Erlang documentation but what I have read suggests not making changes unless I know exactly what I'm doing, which I don't.

Continous changes feed timing out behind a proxy

I've applied the commit (b11c5b) mentioned in issue #37 to latest fabric in an attempt to get the continuous feed working in my setup but am still getting a timeout.

I have three big couch (0.4a.0) nodes sitting behind an nginx load balancer. When I try to get the continuous changes feed for a database, I get the initial burst of sequences just fine, but once I start waiting for new changes, I get nothing. Eventually a timeout occurs.

The bigcouch nodes are listening on 1000{1,2,3} and nginx is on 5984.

Executing 'curl -v http://localhost:5984/ts/?changes_timeout=infinity&heartbeat=true&feed=continuous' gives me the initial burst:

  • About to connect() to localhost port 5984 (#0)
  • Trying ::1... Connection refused
  • Trying 127.0.0.1... connected
  • Connected to localhost (127.0.0.1) port 5984 (#0)

    GET /ts/_changes?changes_timeout=infinity&heartbeat=true&feed=continuous HTTP/1.1
    User-Agent: curl/7.21.0 (x86_64-pc-linux-gnu) libcurl/7.21.0 OpenSSL/0.9.8o zlib/1.2.3.4 libidn/1.18
    Host: localhost:5984
    Accept: /

    < HTTP/1.1 200 OK
    < Server: nginx/0.8.54
    < Date: Fri, 08 Apr 2011 19:44:55 GMT
    < Content-Type: text/plain;charset=utf-8
    < Transfer-Encoding: chunked
    < Connection: keep-alive
    < X-Couch-Request-ID: c13b3e38
    < Cache-Control: must-revalidate
    <
    {"seq":"1-g1AAAASle...
    ...snip...
    ...a minute or so later...
  • Connection #0 to host localhost left intact
  • Closing connection #0
    {"seq":"487-g1AAAAGzeJzLYW
    james@localhost:~$

My nginx error log reports:
2011/04/08 12:47:07 [error] 30174#0: *7728 upstream timed out (110: Connection timed out) while reading upstream, client: 127.0.0.1, server: localhost, request: "GET /ts/_changes?changes_timeout=infinity&heartbeat=true&feed=continuous HTTP/1.1", upstream: "http://127.0.0.1:10001/ts/_changes?changes_timeout=infinity&heartbeat=true&feed=continuous", host: "localhost:5984"

The relevant bigcouch node reports:
[Fri, 08 Apr 2011 19:48:07 GMT] [error] [<0.4076.0>] [c13b3e38] Uncaught error in HTTP request: {exit,normal}

[Fri, 08 Apr 2011 19:48:07 GMT] [info] [<0.4076.0>] [c13b3e38] Stacktrace: [{mochiweb_request,send,2},
{chttpd,send_chunk,2},
{chttpd_db,changes_callback,2},
{fabric_view_changes,keep_sending_changes,6},
{fabric_view_changes,go,5},
{chttpd,handle_request,1},
{mochiweb_http,headers,5},
{proc_lib,init_p_do_apply,3}]

[Fri, 08 Apr 2011 19:48:07 GMT] [info] [<0.4076.0>] [c13b3e38] undefined - - 'GET' /ts/_changes?changes_timeout=infinity&heartbeat=true&feed=continuous 500

Any help appreciated.

James

view compaction fails on most shards

Because the design document is stored in one shard (and not all shards) and because couchdb needs the design document to exist for view compaction to succeed, it is not currently possible to compact a view on all shards (a {"error":"not_found","reason":"missing"} is returned).

Requesting any view with include_docs=true returns error

Using current HEAD ( de09b07 ) when trying to execute any view following error is returned:

{"total_rows":1,"offset":0,"rows":[

{"code":500,"error":"error","reason":"{{badmatch,null},\n [{fabric_rpc,view_fold,3},\n  {couch_view,fold_fun,4},\n  {couch_btree,stream_kv_node2,8},\n  {couch_btree,fold,4},\n  {couch_view,fold,4},\n  {fabric_rpc,map_view,4},\n  {rexi_server,init_p,2}]}"


Logs show:

[info] [<0.1298.0>] 127.0.0.1 localhost:15984 PUT /bzinga1/_design/brr 201 519
[info] [<0.1374.0>] Resetting group index "_design/brr" in db shards/e0000000-ffffffff/bzinga1
[info] [<0.1382.0>] Resetting group index "_design/brr" in db shards/00000000-1fffffff/bzinga1
[info] [<0.1386.0>] Resetting group index "_design/brr" in db shards/c0000000-dfffffff/bzinga1
[info] [<0.1394.0>] Resetting group index "_design/brr" in db shards/20000000-3fffffff/bzinga1
[info] [<0.1406.0>] Resetting group index "_design/brr" in db shards/a0000000-bfffffff/bzinga1
[info] [<0.1411.0>] Resetting group index "_design/brr" in db shards/80000000-9fffffff/bzinga1
[info] [<0.1419.0>] Resetting group index "_design/brr" in db shards/60000000-7fffffff/bzinga1
[info] [<0.1431.0>] Resetting group index "_design/brr" in db shards/40000000-5fffffff/bzinga1
[info] [<0.1298.0>] 127.0.0.1 localhost:15984 GET /bzinga1/_design/brr/_view/blah?limit=11&descending=true 200 61
[info] [<0.1298.0>] 127.0.0.1 localhost:15984 GET /bzinga1/_design/brr/_view/blah 200 7
[error] [<0.1460.0>] {error_report,<0.131.0>,
              {<0.1460.0>,std_error,
               [{rexi_server,{error,{badmatch,null}}},
                [{fabric_rpc,view_fold,3},
                 {couch_view,fold_fun,4},
                 {couch_btree,stream_kv_node2,8},
                 {couch_btree,fold,4},
                 {couch_view,fold,4},
                 {fabric_rpc,map_view,4},
                 {rexi_server,init_p,2}]]}}

=ERROR REPORT==== 30-Nov-2010::21:28:27 ===
    rexi_server: {error,{badmatch,null}}
    [{fabric_rpc,view_fold,3},
     {couch_view,fold_fun,4},
     {couch_btree,stream_kv_node2,8},
     {couch_btree,fold,4},
     {couch_view,fold,4},
     {fabric_rpc,map_view,4},
     {rexi_server,init_p,2}]
[error] [<0.1298.0>] fabric_view_map rexi_EXIT {{badmatch,null},
                           [{fabric_rpc,view_fold,3},
                            {couch_view,fold_fun,4},
                            {couch_btree,stream_kv_node2,8},
                            {couch_btree,fold,4},
                            {couch_view,fold,4},
                            {fabric_rpc,map_view,4},
                            {rexi_server,init_p,2}]}

Adding/Removing databases when one node is down

Scenario 1 - Databases

  • Start 3 nodes, link them
  • Shut down one of them
  • Add 1 database (operation had difficulties/timeout on the node that this was performed, but the result was visible immediately on 2nd node)
  • Start the 3rd node
  • check if the new database is visible on 3rd node
    • NO!
  • Add another database
  • check if it is visible on 3rd node
    • Yes, and also the previously created one
  • Shut down 3rd node
  • Remove one database (operation had difficulties/timeout on the node that this was performed, but the result was visible immediately on 2nd node)
  • Start 3rd node
  • check if one of the nodes disappeared
    • NO!
  • Add another database
  • Check if it is visible on 3rd node
    • Yes, and the previously deleted one disappeared

Conclusion:

BigCouch does not synchronize information about databases when a node joins the network only when the database list is modified.

Limit and skip values act diffrently than in couchdb

In couchdb:

  • skip is index of document to start with
  • limit is number of documents to return
    In bigcouch:
  • skip is index of document to start with
  • limit is index of last document to return

basicly for couch db to return 2 documents after 2 first documents:
skip=2&limit=2
in bigcouch
skip=2&limit=4

_changes feed and filter

look like _changes fails when I use a filter
i´m getting a "curl: (56) Received problem 2 in the chunky parser"
"http://HOST/DB/_changes?filter=ddoc/filtername"

the server log looks like :

[Sat, 05 Feb 2011 15:22:47 GMT] [info] [<0.29917.1>] [dbd19ac9] Stacktrace: [{erlang,apply,[[],write_chunk,[]]},
{chttpd,send_chunk,2},
{chttpd,send_chunked_error,2},
{fabric_view_changes,handle_message,3},
{rexi_utils,process_mailbox,6},
{fabric_view_changes,send_changes,5},
{fabric_view_changes,go,5},
{chttpd,handle_request,1}]

actually seems the issue is quite old http://www.apacheserver.net/Problem-with-filters-at237528.htm .. any plans on fixing it ?

I use latest version of bigcouch, either pre-packaged and compiled from source.

Broken Build (and possible remedy)

When building the latest bigcouch from source, I encountered this error during make:

~/local/src/bigcouch$ make
==> couchjs (compile)
scons: Reading SConscript files ...
Checking for C library m... no
Could not find required library m
make: *** [compile] Error 1

I verified that libm was installed in /usr/lib/libm.so and was confused. After a 'make clean' and 'make distclean', the error was still around. After some searching, it appears that the SCons was caching some files but that cache had been corrupted (perhaps by Ctrl-C during a previous make execution). To remedy the failed build, I removed couchjs/.sconf_temp/ and couchjs/.sconsign.dblite and retried the build. Success!

My possible remedy is to on 'make clean' and/or 'make distclean', remove those temp files to ensure a clean start or perhaps SCons can be configured to remove them.

exit value: {timeout,[{mem3_rep,rexi_call,2}

I'm getting seemly random timeouts in the log on my bigcouch 0.4-rc nodes, between VMs on the same network, with no replications triggered.

I have 3 nodes, 101, 102 and 107. this was from the 102 error log (which is being queried) and it mentions 107 (which is not being queried) and there are no errors in 107
there are the same errors at the same time in the 101 log, mentioning 102

the _replicator db is empty, but does have some deleted docs (update seq 73)

log of error: http://pastebin.com/Ehm9iMYD

This is 0.4-RC commit 8c9228a
On CentOS 5.6 64bit
curl: 7.20.1
erlang: R13B04 (erts-5.7.5)
icu: 4.4.1
js: libmozjs 1.9.2

shard copies should synchronize automatically

Currently, if a shard copy becomes unavailable for some period of time during which writes are occurring it does not automatically receive the missing updates when it comes back online. Replication between shard copies would fix this.

make install dependancy fail

make builds mochiweb, but make install claims the dependancy is not met.

quentusrex@quentusrex-desktop:~/Downloads/bigcouch$ make
==> couchjs (compile)
scons/scons.py:169: UserWarning: Unbuilt egg for setuptools unknown version
d = pkg_resources.get_distribution('scons')
scons: Reading SConscript files ...
Checking for C library m... (cached) yes
Checking for C library pthread... (cached) yes
Checking for C library curl... (cached) yes
Checking for C header file js/jsapi.h... (cached) yes
Checking for C library mozjs... (cached) yes
Checking whether JS_SetOperationCallback is declared... (cached) yes
scons: done reading SConscript files.
scons: Building targets ...
scons: `.' is up to date.
scons: done building targets.
==> oauth (compile)
==> ibrowse (compile)
==> mochiweb (compile)
==> rexi (compile)
==> fabric (compile)
==> mem3 (compile)
==> chttpd (compile)
==> couch (compile)
==> rel (compile)
==> bigcouch (compile)

quentusrex@quentusrex-desktop:~/Downloads/bigcouch$ sudo make install
==> couchjs (compile)
scons/scons.py:169: UserWarning: Unbuilt egg for setuptools unknown version
d = pkg_resources.get_distribution('scons')
scons: Reading SConscript files ...
Checking for C library m... (cached) yes
Checking for C library pthread... (cached) yes
Checking for C library curl... (cached) yes
Checking for C header file js/jsapi.h... (cached) yes
Checking for C library mozjs... (cached) yes
Checking whether JS_SetOperationCallback is declared... (cached) yes
scons: done reading SConscript files.
scons: Building targets ...
scons: `.' is up to date.
scons: done building targets.
==> oauth (compile)
==> ibrowse (compile)
==> rexi (compile)
==> fabric (compile)
==> mem3 (compile)
==> chttpd (compile)
==> couch (compile)
==> rel (compile)
==> bigcouch (compile)
Dependency not available: mochiweb-.* ({git,
"https://github.com/cloudant/mochiweb.git",
{branch,"couch"}})
make: *** [compile] Error 1

user db should be synchronized like dbs and nodes dbs

Having hard time this weekend to setup an admin in bigcouh or any user. For admipns I had top setup obe on each osts via the 5486 port, and then setup manually user sync and add user in db. Whatu is the right wy to do that?

No compaction possible

I tried to run a database compaction via port 5984 and got the error "The database could not be compacted: Compaction must be triggered on a per-shard basis in BigCouch".

Ok, makes kinda sense so i moved to port 5986 and clicked on one of the shards which results in the error "An error occurred retrieving a list of all documents: no_db_file".
Well, not a real database, so i guessed that just clicking on the compaction button would do it - unfortunately not: "The database could not be compacted: no_db_file"

The wiki is not big of help either, so is this a bug or is there another, non documented, way that i could not see?! ;)

Thx!

read-your-writes consistency for views

View reads use an implicit R=1 quorum. This can cause problems when writes are performed with W < N, as a view read following a write might not see the result of the write reflected in the view.

One way to circumvent this would be to set a cookie in the write response which indicates the shard copies that contributed to the write quorum. A subsequent read request with the cookie set would only consult those shard copies for that particular portion of the ring.

Need to consider how this approach behaves with repeated write requests. Will clients fall over because they're continually asked to set new cookies?

_changes returns "doc:null" w/ include_docs=true

On vanilla Couch the _changes feed with include_docs=true will return something like "doc": {"_id":"7cdc519202e3319a5fb192740500090a","_rev":"2-8e427d1c4515b98cdc07ae86e7243c07","_deleted":true} for deleted docs. BigCouch returns "doc": null.

This is on Cloudant - {"couchdb":"Welcome","version":"1.0.0","cloudant_build":"1.2.13"}

Internal Replication Error

Hi Guys,

the installation was great, the error log here

[Thu, 21 Oct 2010 12:01:18 GMT] [info] [<0.128.0>] starting dbs -> '[email protected]' internal replication

[Thu, 21 Oct 2010 12:01:18 GMT] [error] [<0.6551.0>] {error_report,<0.85.0>,
{<0.6551.0>,crash_report,
[[{initial_call,{couch_rep,init,['Argument__1']}},
{pid,<0.6551.0>},
{registered_name,[]},
{error_info,
{exit,
{node_not_connected,<<"[email protected]">>},
[{gen_server,init_it,6},{proc_lib,init_p_do_apply,3}]}},
{ancestors,
[couch_rep_sup,couch_primary_services,couch_server_sup,<0.86.0>]},
{messages,[]},
{links,[<0.95.0>]},
{dictionary,[]},
{trap_exit,true},
{status,running},
{heap_size,610},
{stack_size,24},
{reductions,208}],
[]]}}

[Thu, 21 Oct 2010 12:01:18 GMT] [info] [<0.128.0>] mem3_sync strange error {'EXIT',
{{case_clause,
{error,
{{node_not_connected,
<<"[email protected]">>},
{child,undefined,
"9e126890c5732d1fdd641d9d10d5b70e",
{couch_rep,start_link,
["9e126890c5732d1fdd641d9d10d5b70e",
{[{<<"source">>,<<"dbs">>},
{<<"target">>,
{[{<<"node">>,'[email protected]'},
{<<"name">>,<<"dbs">>}]}},
{<<"continuous">>,false},
{<<"async">>,true}]},
{user_ctx,<<"replicator">>,
[<<"_admin">>],
undefined}]},
temporary,1,worker,
[couch_rep]}}}},
[{couch_rep,start_replication_server,2},
{couch_rep,replicate,2},
{mem3_sync,start_push_replication,2},
{mem3_sync,handle_cast,2},
{gen_server,handle_msg,5},
{proc_lib,init_p_do_apply,3}]}}

replication error

"Invalid rev" error in replication. Again, I think this was reported some time ago. Did some more digging, and I found these aparrent fixes :

in couch_rep.erl
change from
case couch_rep_httpc:request(Req) of
{[{<<"error">>, _}, {<<"reason">>}} ->
#doc{id=?l2b(DocId)};
...
TO
{[{<<"error">>, _}, {<<"reason">>, }|]} ->
#doc{id=?l2b(DocId)};
...
(the other way around would be to omit the stacktrace term from propagating till this point)

chttp_db.erl

db_req(#httpd{path_parts=[_DbName, <<"_local">>, Name]}=Req, Db) ->

  • try
    db_doc_req(Req, Db, <<"_local/", Name/binary>>)
  • catch
  •  throw:Error ->
    
  •    send_json(Req, 404, [], {[
    
  •    {<<"error">>, <<"not_found">>},
    
  •   {<<"reason">>, <<"missing">>}
    
  •    ]})
    
  • end;

looks like these changes make replication work both ways, with and without filters.

build fails on warnings

Build is expected to fail on warning. But since erlang 14a has a deprecraction warning wit http client:

Warning: http:request/4 is deprecated and will be removed in R15B; use httpc:request/4

All the build fail. I had to edit rebar.config to use it with this Erlang version :

--- a/rebar.config
+++ b/rebar.config
@@ -25,5 +25,5 @@
     "rel"
 ]}.
 {cover_enabled, true}.
-{erl_opts, [debug_info, fail_on_warning]}.
+{erl_opts, [debug_info]}.
 {lib_dirs, ["apps"]}.

Consistency problem even with majority and unanimous quorums

We are running BigCouch in a 5 server pre-production cluster behind haproxy.
We're have an issue (perhaps with our understanding) with consistency.

We are deliberately running a case that makes it as hard as possible for BigCouch to maintain version consistency across the cluster, we have a requirement for sequential, integer identifiers, so that we can integrate with specific legacy systems. (We're trying real hard to keep a SQL solution off the table)

The test configuration has a database with n=5, q=128

We are testing consistency by having a single document, containing an integer value.
We run multiple test threads that read the value, increment it, and write it back, and repeat.
We are expecting contention, so where we get a version conflict after a write attempt, we simply read the new version and try again.

When we successfully make an update we log the the thread that succeeded and the incremented value.
After the test period, typically 3 minutes, we analyse the successful writes, and look for duplicates writes and for missing numbers in the sequence.

We've run the test rig with r=3/w=3 and r=5/w=5 in an attempt to force the cluster to be sure about document versions before accepting the write. (To act as a monlithic couchdb instance)

If we run the test with 1, 2 or 3 threads the cluster behaves as you would expect (looks just like a single couch box, indeed we have tested against a single couchdb instance, we get no duplicate updates, but some missing on high thread counts/concurrency levels and we can live with that. Performance is down due to the node synchronisation, but its good enough)

BUT: What we see is that when we run the tread counts up above 4, and certainly reproducible every time with a thread count of 10, we are getting duplicate updates. That is two threads updating document version 100 to 101 (and the corresponding document integer value), both believing they made the update because BigCouch has not reported a version conflict on the update. ie. Both client processes thinking they did the update from 100 to 101, so both have the SAME 'unique' value.

We also see missing numbers in the sequence. That is; no thread was able to successfully update version 102 to 103, all threads received version conflict responses, yet looking a the documents version history, it's there in the version chain, that version does exist. So our incrementing integer test skips a value.

Whilst we can live with skipping a few identifiers, we cant manage when we dont have unique identifiers.
We were under the impression that unanimous quorum versions would ensure that the behaviour of the cluster was analogous to a single couchDB server/database. We also assumed that a majority quorum would produce the desired behaviour.

I really do apologise if the issue is not an issue at all, and its simply my misunderstanding of the dynamo quorum mechanism.
Is this expected behaviour?

Many thanks in advance

Cant invoke _info on a shard view

curl -H "Content-Type: application/json" -X GET http://server_name:5984/db_name/_design/view_name/_info
returns what you'd expect for the view "view_name".

However,
curl -H "Content-Type: application/json" -X GET http://server_name/shards%2Fshard_id%2Fdb_name.some_number/_design/view_name/_info
returns {"error":"not_found","reason":"missing"}

Unless there is some other way of figuring out this information?

badmatch when attempting to report errors

2010-10-30_23:24:36.57483 {error_info,
2010-10-30_23:24:36.57483 {error,
2010-10-30_23:24:36.57483 {badmatch,{error,dead_shards}},
2010-10-30_23:24:36.57484 [{chttpd,handle_request,1},
2010-10-30_23:24:36.57484 {mochiweb_http,headers,5},
2010-10-30_23:24:36.57485 {proc_lib,init_p_do_apply,3}]}},
exception error: no match of right hand side value {error,dead_shards}
2010-10-30_23:24:36.57740 in function chttpd:handle_request/1
2010-10-30_23:24:36.57741 in call from mochiweb_http:headers/5

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.