$ make build

github.com/youtube/vitess/go/mysql

go/mysql/vtmysql.c: In function ‘vt_simple_command’:
go/mysql/vtmysql.c:153:3: error: implicit declaration of function ‘simple_command’ [-Werror=implicit-function-declaration]
return simple_command(conn->mysql, command, arg, arg_length, skip_check);
^
cc1: some warnings being treated as errors
make: *** [build] Error 2

test_set_insert_id can fail if it's running on a slow disk (i.e., not SSD/ramdisk). This is bad because it's part of our make site_test suite, and presumably should pass even in less resource-heavy environments.

The particular failure mode that I saw was:

-- 2014-10-29 19:50:43,369 update_stream:185 DEBUG Test Service Disabled: Pass
-- 2014-10-29 19:50:43,371 update_stream:202 DEBUG _test_service_enabled starting @ {'MariaDB': '0-62344-10'}
-- 2014-10-29 19:50:43,371 utils:182 DEBUG run: ['mariadb/bin/vtctl', '-log_dir', 'mariadb/vtdataroot/tmp', '-topo_implementation', 'zook
eeper', '-tablet_manager_protocol', 'bson', '-tablet_protocol', 'gorpc', 'ChangeSlaveType', 'test_nj-0000062345', 'replica'] 
-- 2014-10-29 19:50:43,849 update_stream:204 DEBUG sleeping a bit for the replica action to complete
-- 2014-10-29 19:50:53,855 update_stream:218 DEBUG Test Service Enabled: Pass
-- 2014-10-29 19:51:23,858 update_stream:233 DEBUG Testing enable -> disable switch starting @ {'MariaDB': '0-62344-10'}
-- 2014-10-29 19:51:23,892 utils:182 DEBUG run: ['mariadb/bin/vtctl', '-log_dir', 'mariadb/vtdataroot/tmp', '-topo_implementation', 'zook
eeper', '-tablet_manager_protocol', 'bson', '-tablet_protocol', 'gorpc', 'ChangeSlaveType', 'test_nj-0000062345', 'spare'] 
-- 2014-10-29 19:51:24,280 update_stream:247 ERROR Test Service Switch: FAIL
-- 2014-10-29 19:51:24,280 utils:182 DEBUG run: ['mariadb/bin/vtctl', '-log_dir', 'mariadb/vtdataroot/tmp', '-topo_implementation', 'zook
eeper', '-tablet_manager_protocol', 'bson', '-tablet_protocol', 'gorpc', 'ChangeSlaveType', 'test_nj-0000062345', 'replica'] 
-- 2014-10-29 19:51:24,671 utils:50 INFO ===== test_service_switch (__main__.TestUpdateStream) ... ok
-- 2014-10-29 19:51:24,671 utils:50 INFO ===== test_set_insert_id (__main__.TestUpdateStream)
-- 2014-10-29 19:51:24,671 utils:50 INFO ===== test_set_insert_id (__main__.TestUpdateStream) ... 
-- 2014-10-29 19:51:24,897 update_stream:339 DEBUG test_set_insert_id: starting @ {'MariaDB': '0-62344-428'}
-- 2014-10-29 19:51:24,982 utils:50 INFO ===== test_set_insert_id (__main__.TestUpdateStream) ... FAIL
-- 2014-10-29 19:51:24,982 update_stream:126 DEBUG Tearing down the servers and setup
-- 2014-10-29 19:51:24,982 tablet:624 DEBUG killing vttablet: test_nj-0000062344
-- 2014-10-29 19:51:24,982 tablet:624 DEBUG killing vttablet: test_nj-0000062345
-- 2014-10-29 19:51:24,983 vtclient:127 WARNING db connection failed: test_keyspace.0.master :15009, ('retry: Query server is in NOT_SERVING state', 'SqlQuery.GetSessionId', ':15009 master test_keyspace/0>')
-- 2014-10-29 19:51:24,983 vtdb_logger:64 WARNING vtclient_exception for test_keyspace.0.master: ('unable to create vt connection', 'test_keyspace.0.master', ':15009', RetryError
(('retry: Query server is in NOT_SERVING state', 'SqlQuery.GetSessionId', ':15009 master test_keyspace/0>'),))
Exception in thread write_thd:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 763, in run
    self.__target(*self.__args, **self.__kwargs)
  File "test/update_stream.py", line 197, in perform_writes
    self._exec_vt_txn(self._populate_vt_insert_test)
  File "test/update_stream.py", line 269, in _exec_vt_txn
    vtdb_conn = self._vtdb_conn('localhost:%u' % master_tablet.port)
  File "test/update_stream.py", line 263, in _vtdb_conn
    conn.connect()
  File "mariadb/py-vtdb/vtdb/vtclient.py", line 95, in connect
    return self._connect()
  File "mariadb/py-vtdb/vtdb/vtclient.py", line 133, in _connect
    'unable to create vt connection', db_key, host_addr, db_exception)
OperationalError: ('unable to create vt connection', 'test_keyspace.0.master', ':15009', RetryError(('retry: Query server is in NOT_SERVING state', 'SqlQuery.GetSessionId', ':15009 master test_keyspace/0>'),))

Basically, it looks like the previously run test_service_switch put the cluster into a "NOT_SERVING" state as part of what it's testing. It tries to put the cluster back in an enabled state, but doesn't actually wait for state to fully change. This creates a race condition which can cause subsequent tests to fail.

We probably want to add something like a wait_step condition at the end of the test, and block until the state is what we expect it to be.

thanks for this code first!
i have a doubt about this code, can it support to running out a list with sort and pagination by mysql sharding ?

here is my error log

vagrant@vagrant-ubuntu-trusty-64:/vagrant/src/github.com/youtube/vitess/examples/local$ ./vttablet-up.sh 
Starting MySQL for tablet test-0000000100...
E0312 05:02:22.495230    5032 mysqld.go:348] failed starting, check /vagrant/vtdataroot/vt_0000000100/error.log
E0312 05:02:22.503269    5032 mysqlctl.go:219] failed init mysql: /usr/bin/mysqld_safe: deadline exceeded waiting for /vagrant/vtdataroot/vt_0000000100/mysql.sock
vagrant@vagrant-ubuntu-trusty-64:/vagrant/src/github.com/youtube/vitess/examples/local$ more /vagrant/vtdataroot/vt_0000000100/error.log 
150312 05:00:21 mysqld_safe Starting mysqld daemon with databases from /vagrant/vtdataroot/vt_0000000100/data
150312  5:00:21 [Warning] 'THREAD_CONCURRENCY' is deprecated and will be removed in a future release.
150312  5:00:21 [Warning] Changed limits: max_open_files: 1024  max_connections: 100  table_cache: 457
2015-03-12 05:00:21 7f10d9c937c0 InnoDB: Warning: Using innodb_additional_mem_pool_size is DEPRECATED. This option may be removed in future
 releases, together with the option innodb_use_sys_malloc and with the InnoDB's internal memory allocator.
150312  5:00:21 [Note] InnoDB: Using mutexes to ref count buffer pool pages
150312  5:00:21 [Note] InnoDB: The InnoDB memory heap is disabled
150312  5:00:21 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
150312  5:00:21 [Note] InnoDB: Memory barrier is not used
150312  5:00:21 [Note] InnoDB: Compressed tables use zlib 1.2.8
150312  5:00:21 [Note] InnoDB: Using Linux native AIO
150312  5:00:21 [Note] InnoDB: Not using CPU crc32 instructions
150312  5:00:21 [Note] InnoDB: Initializing buffer pool, size = 32.0M
150312  5:00:21 [Note] InnoDB: Completed initialization of buffer pool
150312  5:00:21 [Warning] InnoDB: Failed to set O_DIRECT on file /vagrant/vtdataroot/vt_0000000100/innodb/data/ibdata1: OPEN: Invalid argum
ent, continuing anyway. O_DIRECT is known to result in 'Invalid argument' on Linux on tmpfs, see MySQL Bug#26662.
150312  5:00:21 [Note] InnoDB: Highest supported file format is Barracuda.
150312  5:00:21 [Note] InnoDB: 128 rollback segment(s) are active.
150312  5:00:21 [Note] InnoDB: Waiting for purge to start
150312  5:00:21 [Note] InnoDB:  Percona XtraDB (http://www.percona.com) 5.6.22-72.0 started; log sequence number 1616584
150312  5:00:21 [Note] Plugin 'FEEDBACK' is disabled.
150312  5:00:21 [Note] Server socket created on IP: '::'.
150312  5:00:21 [ERROR] Can't start server : Bind on unix socket: Operation not permitted
150312  5:00:21 [ERROR] Do you already have another mysqld server running on socket: /vagrant/vtdataroot/vt_0000000100/mysql.sock ?
150312  5:00:21 [ERROR] Aborting

150312  5:00:21 [Note] InnoDB: FTS optimize thread exiting.
150312  5:00:21 [Note] InnoDB: Starting shutdown...
150312  5:00:23 [Note] InnoDB: Shutdown completed; log sequence number 1616594
150312  5:00:23 [Note] /usr/sbin/mysqld: Shutdown complete

150312 05:00:23 mysqld_safe mysqld from pid file /vagrant/vtdataroot/vt_0000000100/mysql.pid ended

sqlparser not support the SQL：drop tables. tb1, tb2, tb3.....
DDL struct can not analyze many tables of drop tables.

Does client code need user and password to access vtgate?If not, then any code can access vtgate without any authentication,i.e. client code can access databases behind vttablets directly, then perhaps security problems occur.

The libraries made available through this project are awesome. It's a bit cumbersome to use them, though, due to the size of the repo. For example, importing the cache go library requires cloning the entire 68MB repo for a few KB of code. Splitting off go/* (excepting go/vt) into a separate repo would result in only a single additional repository but provide the bulk of the benefit in terms of size reduction.

Does vitess support some common features for MySQL as follows:
1、rewriteBatchedStatements=true
2、useServerPrepStmts=true
3、correlation sql operations such as:join、paging、sort、subquery
4、BLOB、BINARY、VARBINARY
5、SAVEPOINT operation
6、SET statement
7、update existed column value for sharded table

Even if go is not installed (or is not on the $PATH, which is more likelier), the script goes on and executes completely. Two solutions: 1. Check if go is present, and fail if not. 2. Pass -e flag to the bash script (set -e), which will make the script fail as soon as first error is encountered (but we might not want this)

when use vitess a few times , vttablet will break off.
the log shows : open too many files.

what's wrong for me?

Mycat is an enforced database which is a replacement for MySQL and support transaction and ACID
https://github.com/MyCATApache/Mycat-Server

The following code panics on iteration i == 33
http://play.golang.org/p/xMDjv9I95E

I ran it as a test file in vt/sqlparser.

Sorry if I miss something but I couldn't find anything so I need to ask it here :) The comment for spanFactory says (https://github.com/youtube/vitess/blob/master/go/trace/trace.go#L60):

// spanFactory should be changed by a plugin during init() to a factory that
// creates an actual Span implementation for that plugin's tracing framework.

As I playing with the rpcplus package, the methods of rpcplus.Client is calling the following lines (Go method):

span := trace.NewSpanFromContext(ctx)
span.StartClient(serviceMethod)
defer span.Finish()

I dig into it and saw that it uses a no-op implementation (fakeSpan). I can't change the Span interface. The spanFactory variable is not exported I don't have any access to it. I've searched for the whole git repo and couldn't find any other implementation too.

Is that used? If yes how should we use it? If not can we export it so I have access to it, otherwise it's seems pretty useless for third-party package users.

Building vitess from git source won't success if you have only 512MB memory.
The gcc error log said: can not allocate memory,.

So , the question is : what is the minmal requirement if we want build and install vitess?

any more details about it?

1Ghz or higher cpu
1GB or 2GB memory
30GB hard disk space
1Mbps or more network can access the internet?

Is there some way to see https://fosdem.org/2014/schedule/event/scaling_with_go:_youtubes_vitess/ presentation video? Would love to see it.

The Dockerfile build need so much times to download and install requirements packages.
So i suggest it can be split two part.

Create a container images include all packages need by build vitess

FROM golang:1.4-wheezy

# Install Vitess build dependencies
RUN apt-get update && apt-get install -y --no-install-recommends \
    automake \
    bison \
    bzip2 \
    curl \
    g++ \
    git \
    libssl-dev \
    libtool \
    make \
    memcached \
    mercurial \
    openjdk-7-jre-headless \
    pkg-config \
    python-dev \
    python-mysqldb \
    python-software-properties \
    python-pip \
    python-virtualenv \
    unzip \
    && rm -rf /var/lib/apt/lists/*

# Install MariaDB 10.0.x
RUN apt-key adv --recv-keys --keyserver keyserver.ubuntu.com 0xcbcb082a1bb943db && \
    add-apt-repository 'deb http://sfo1.mirrors.digitalocean.com/mariadb/repo/10.0/debian wheezy main' && \
    apt-get update && apt-get install -y mariadb-server libmariadbclient-dev
CMD ["bash"]

I've cloned vitess in a new clean environment on my Ubuntu 14.04 LTS 64-bit system. Bootstrapping fails of mysql fails:

libtool: compile: gcc -DHAVE_CONFIG_H -I. -I. -I./src -I./src -g2 -O2 -fno-strict-aliasing -fno-omit-frame-pointer -I/home/frederich/vitess/src/github.com/youtube/vitess/third_party/mysql/ncurses/include -I/home/frederich/vitess/src/github.com/youtube/vitess/third_party/mysql/ncurses/include -DHAVE_TERM_H -DUNIV_LINUX -DUNIV_LINUX -MT thread_lister.lo -MD -MP -MF .deps/thread_lister.Tpo -c src/base/thread_lister.c -fPIC -DPIC -o .libs/thread_lister.o
libtool: compile: gcc -DHAVE_CONFIG_H -I. -I. -I./src -I./src -g2 -O2 -fno-strict-aliasing -fno-omit-frame-pointer -I/home/frederich/vitess/src/github.com/youtube/vitess/third_party/mysql/ncurses/include -I/home/frederich/vitess/src/github.com/youtube/vitess/third_party/mysql/ncurses/include -DHAVE_TERM_H -DUNIV_LINUX -DUNIV_LINUX -MT thread_lister.lo -MD -MP -MF .deps/thread_lister.Tpo -c src/base/thread_lister.c -fPIC -DPIC -o thread_lister.o >/dev/null 2>&1
In file included from src/tcmalloc.cc:116:0:
src/packed-cache-inl.h: In constructor ‘PackedCache<kKeybits, T>::PackedCache(PackedCache<kKeybits, T>::V)’:
src/packed-cache-inl.h:148:47: warning: typedef ‘key_size’ locally defined but not used [-Wunused-local-typedefs]
COMPILE_ASSERT(kKeybits <= sizeof(K) * 8, key_size);
^
src/base/basictypes.h:188:39: note: in definition of macro ‘COMPILE_ASSERT’
typedef CompileAssert<(bool(expr))> msg[bool(expr) ? 1 : -1]
^
src/packed-cache-inl.h:149:49: warning: typedef ‘value_size’ locally defined but not used [-Wunused-local-typedefs]
COMPILE_ASSERT(kValuebits <= sizeof(V) * 8, value_size);
^
src/base/basictypes.h:188:39: note: in definition of macro ‘COMPILE_ASSERT’
typedef CompileAssert<(bool(expr))> msg[bool(expr) ? 1 : -1]
^
src/packed-cache-inl.h:150:43: warning: typedef ‘hash_function’ locally defined but not used [-Wunused-local-typedefs]
COMPILE_ASSERT(kHashbits <= kKeybits, hash_function);
^
src/base/basictypes.h:188:39: note: in definition of macro ‘COMPILE_ASSERT’
typedef CompileAssert<(bool(expr))> msg[bool(expr) ? 1 : -1]
^
src/packed-cache-inl.h:152:20: warning: typedef ‘entry_size_must_be_big_enough’ locally defined but not used [-Wunused-local-typedefs]
entry_size_must_be_big_enough);
^
src/base/basictypes.h:188:39: note: in definition of macro ‘COMPILE_ASSERT’
typedef CompileAssert<(bool(expr))> msg[bool(expr) ? 1 : -1]
^
src/sampler.h: In static member function ‘static double tcmalloc::Sampler::FastLog2(const double&)’:
src/sampler.h:168:49: warning: typedef ‘DoubleMustBe64Bits’ locally defined but not used [-Wunused-local-typedefs]
COMPILE_ASSERT(sizeof(d) == sizeof(uint64_t), DoubleMustBe64Bits);
^
src/base/basictypes.h:188:39: note: in definition of macro ‘COMPILE_ASSERT’
typedef CompileAssert<(bool(expr))> msg[bool(expr) ? 1 : -1]
^
libtool: compile: g++ -DHAVE_CONFIG_H -I. -I. -I./src -I./src -pthread -DNDEBUG -Wall -Wwrite-strings -Woverloaded-virtual -Wno-sign-compare -fno-builtin-malloc -fno-builtin-free -fno-builtin-realloc -fno-builtin-calloc -fno-builtin-cfree -fno-builtin-memalign -fno-builtin-posix_memalign -fno-builtin-valloc -fno-builtin-pvalloc -Wno-unused-result -g2 -O2 -fno-strict-aliasing -fno-omit-frame-pointer -I/home/frederich/vitess/src/github.com/youtube/vitess/third_party/mysql/ncurses/include -I/home/frederich/vitess/src/github.com/youtube/vitess/third_party/mysql/ncurses/include -DHAVE_TERM_H -fno-rtti -MT libtcmalloc_la-linuxthreads.lo -MD -MP -MF .deps/libtcmalloc_la-linuxthreads.Tpo -c src/base/linuxthreads.cc -fPIC -DPIC -o .libs/libtcmalloc_la-linuxthreads.o
In file included from src/base/commandlineflags.h:55:0,
from src/debugallocation.cc:68:
src/packed-cache-inl.h: In constructor ‘PackedCache<kKeybits, T>::PackedCache(PackedCache<kKeybits, T>::V)’:
src/packed-cache-inl.h:148:47: warning: typedef ‘key_size’ locally defined but not used [-Wunused-local-typedefs]
COMPILE_ASSERT(kKeybits <= sizeof(K) * 8, key_size);
^
./src/base/basictypes.h:188:39: note: in definition of macro ‘COMPILE_ASSERT’
typedef CompileAssert<(bool(expr))> msg[bool(expr) ? 1 : -1]
^
src/packed-cache-inl.h:149:49: warning: typedef ‘value_size’ locally defined but not used [-Wunused-local-typedefs]
COMPILE_ASSERT(kValuebits <= sizeof(V) * 8, value_size);
^
./src/base/basictypes.h:188:39: note: in definition of macro ‘COMPILE_ASSERT’
typedef CompileAssert<(bool(expr))> msg[bool(expr) ? 1 : -1]
^
src/packed-cache-inl.h:150:43: warning: typedef ‘hash_function’ locally defined but not used [-Wunused-local-typedefs]
COMPILE_ASSERT(kHashbits <= kKeybits, hash_function);
^
./src/base/basictypes.h:188:39: note: in definition of macro ‘COMPILE_ASSERT’
typedef CompileAssert<(bool(expr))> msg[bool(expr) ? 1 : -1]
^
src/packed-cache-inl.h:152:20: warning: typedef ‘entry_size_must_be_big_enough’ locally defined but not used [-Wunused-local-typedefs]
entry_size_must_be_big_enough);
^
./src/base/basictypes.h:188:39: note: in definition of macro ‘COMPILE_ASSERT’
typedef CompileAssert<(bool(expr))> msg[bool(expr) ? 1 : -1]
^
src/sampler.h: In static member function ‘static double tcmalloc::Sampler::FastLog2(const double&)’:
src/sampler.h:168:49: warning: typedef ‘DoubleMustBe64Bits’ locally defined but not used [-Wunused-local-typedefs]
COMPILE_ASSERT(sizeof(d) == sizeof(uint64_t), DoubleMustBe64Bits);
^
./src/base/basictypes.h:188:39: note: in definition of macro ‘COMPILE_ASSERT’
typedef CompileAssert<(bool(expr))> msg[bool(expr) ? 1 : -1]
^
libtool: compile: g++ -DHAVE_CONFIG_H -I. -I. -I./src -I./src -pthread -DNDEBUG -Wall -Wwrite-strings -Woverloaded-virtual -Wno-sign-compare -fno-builtin-malloc -fno-builtin-free -fno-builtin-realloc -fno-builtin-calloc -fno-builtin-cfree -fno-builtin-memalign -fno-builtin-posix_memalign -fno-builtin-valloc -fno-builtin-pvalloc -Wno-unused-result -g2 -O2 -fno-strict-aliasing -fno-omit-frame-pointer -I/home/frederich/vitess/src/github.com/youtube/vitess/third_party/mysql/ncurses/include -I/home/frederich/vitess/src/github.com/youtube/vitess/third_party/mysql/ncurses/include -DHAVE_TERM_H -fno-rtti -MT libtcmalloc_la-heap-checker.lo -MD -MP -MF .deps/libtcmalloc_la-heap-checker.Tpo -c src/heap-checker.cc -fPIC -DPIC -o .libs/libtcmalloc_la-heap-checker.o
src/base/linuxthreads.cc: In function ‘void ListerThread(ListerParams_)’:
src/base/linuxthreads.cc:312:22: error: invalid conversion from ‘void ()(int, siginfo_t, void_)’ to ‘void ()(int, siginfo, void_)’ [-fpermissive]
sa.sa_sigaction_ = SignalHandler;
^
make[1]: ** [libtcmalloc_la-linuxthreads.lo] Error 1
make[1]: *** Waiting for unfinished jobs....
In file included from src/heap-checker.cc:78:0:
src/heap-checker.cc: In static member function ‘static int HeapLeakChecker::IgnoreLiveThreadsLocked(void, int, pid_t_, _va_list_tag)’:
src/heap-checker.cc:1017:22: warning: typedef ‘SP_register_does_not_look_like_a_pointer’ locally defined but not used [-Wunused-local-typedefs]
SP_register_does_not_look_like_a_pointer);
^
src/base/basictypes.h:188:39: note: in definition of macro ‘COMPILE_ASSERT’
typedef CompileAssert<(bool(expr))> msg[bool(expr) ? 1 : -1]
^
libtool: compile: g++ -DHAVE_CONFIG_H -I. -I. -I./src -I./src -pthread -DNDEBUG -Wall -Wwrite-strings -Woverloaded-virtual -Wno-sign-compare -fno-builtin-malloc -fno-builtin-free -fno-builtin-realloc -fno-builtin-calloc -fno-builtin-cfree -fno-builtin-memalign -fno-builtin-posix_memalign -fno-builtin-valloc -fno-builtin-pvalloc -Wno-unused-result -g2 -O2 -fno-strict-aliasing -fno-omit-frame-pointer -I/home/frederich/vitess/src/github.com/youtube/vitess/third_party/mysql/ncurses/include -I/home/frederich/vitess/src/github.com/youtube/vitess/third_party/mysql/ncurses/include -DHAVE_TERM_H -fno-rtti -MT libtcmalloc_la-tcmalloc.lo -MD -MP -MF .deps/libtcmalloc_la-tcmalloc.Tpo -c src/tcmalloc.cc -fPIC -DPIC -o libtcmalloc_la-tcmalloc.o >/dev/null 2>&1
libtool: compile: g++ -DHAVE_CONFIG_H -I. -I. -I./src -I./src -pthread -DNDEBUG -Wall -Wwrite-strings -Woverloaded-virtual -Wno-sign-compare -fno-builtin-malloc -fno-builtin-free -fno-builtin-realloc -fno-builtin-calloc -fno-builtin-cfree -fno-builtin-memalign -fno-builtin-posix_memalign -fno-builtin-valloc -fno-builtin-pvalloc -Wno-unused-result -g2 -O2 -fno-strict-aliasing -fno-omit-frame-pointer -I/home/frederich/vitess/src/github.com/youtube/vitess/third_party/mysql/ncurses/include -I/home/frederich/vitess/src/github.com/youtube/vitess/third_party/mysql/ncurses/include -DHAVE_TERM_H -fno-rtti -MT libtcmalloc_la-heap-checker.lo -MD -MP -MF .deps/libtcmalloc_la-heap-checker.Tpo -c src/heap-checker.cc -fPIC -DPIC -o libtcmalloc_la-heap-checker.o >/dev/null 2>&1
libtool: compile: g++ -DHAVE_CONFIG_H -I. -I. -I./src -I./src -DNO_TCMALLOC_SAMPLES -pthread -DNDEBUG -Wall -Wwrite-strings -Woverloaded-virtual -Wno-sign-compare -fno-builtin-malloc -fno-builtin-free -fno-builtin-realloc -fno-builtin-calloc -fno-builtin-cfree -fno-builtin-memalign -fno-builtin-posix_memalign -fno-builtin-valloc -fno-builtin-pvalloc -Wno-unused-result -DTCMALLOC_FOR_DEBUGALLOCATION -g2 -O2 -fno-strict-aliasing -fno-omit-frame-pointer -I/home/frederich/vitess/src/github.com/youtube/vitess/third_party/mysql/ncurses/include -I/home/frederich/vitess/src/github.com/youtube/vitess/third_party/mysql/ncurses/include -DHAVE_TERM_H -fno-rtti -MT libtcmalloc_minimal_debug_la-debugallocation.lo -MD -MP -MF .deps/libtcmalloc_minimal_debug_la-debugallocation.Tpo -c src/debugallocation.cc -fPIC -DPIC -o libtcmalloc_minimal_debug_la-debugallocation.o >/dev/null 2>&1
make: *_* [all-recursive] Error 1

We probably want to integrate with an open source CI, like Travis (https://travis-ci.org/) and run tests for each commit (at least those unit tests). For now we require submitter to do a manual test run and that may cause bad commit to broke the build.

Paste: http://paste.openstack.org/show/163121/

After restarting the VM, I just sourced dev.env, and ran ./zk-up.sh. Didn't work

Small nit with the k8s example: If you don't run etcd-up.sh from its dir, e.g.:

./etcd-up.sh

you get:
cat: etcd-service-template.yaml: No such file or directory

hi ,
here i hava some questions about how to use with vitess ,
(1) . first i hava create a shard =0 , and in this shard, i create two vttablet instance, then use command "./vtctl ReparentShard" to electing a master ,everything is ok. but when i when to add a new vttablet into this shard=0 , question appeared。 if i do not create table in the new vttablet , it will no get some data from master . when i create table, it will get new data when master update, but can not get old master data. so i don't know how to do in this place?

(2) . second ,i don't kown hao to use shard. can you give me a stronger document in hao to use shard?

I am not sure i post the issue in the right place, but i have to post.

I have noticed there is not volume in the Dockerfile

https://registry.hub.docker.com/u/vitess/base/dockerfile/

How about add one

VOLUME /myvol

The VOLUME is good solution for data backup/mitgration/share.

The /vt is the Root directory about vitess, it contains almost everything about vitess. include the important data file.

I follow the instructions in GetStarted:

when I make build, there are wrongs:

# github.com/youtube/vitess/go/stats/influxdbbackend
go/stats/influxdbbackend/influxdb_backend.go:40: undefined: client.ClientConfig
go/stats/influxdbbackend/influxdb_backend.go:60: undefined: client.Series
go/stats/influxdbbackend/influxdb_backend.go:62: undefined: client.Series
go/stats/influxdbbackend/influxdb_backend.go:73: backend.client.WriteSeries undefined (type *client.Client has no field or method WriteSeries)

it seems like the Influxdb has been making lots of changes lately.

How to set proxy if our environment behind firewall

The getting start guide not tell us the right way to setting proxy, so our building not success.

There are too many branches in vitess repo and most of them are development branches, not release ones. Maybe we can utilize github pull request model better by forking vitess repo and leave these working branch there. In that way, vitess repo only contains necessary branches: master, releases or major feature branches.

In the marster branch that i cloned last week, after running bootstrap.sh, there's no test.sh under vthook folder, this breaks the test script tabletmanager.py.

I'm trying to use the 'pools' package with the following test code:

package main

import (
    "fmt"
    "time"

    "github.com/youtube/vitess/go/pools"
)

type Foo struct {
    Id          int
}

func (f *Foo) Close() {
    fmt.Printf("Closing Foo #%d\n", f.Id)
}

var nextId int

func fooFactory() (pools.Resource,error) {
    fmt.Printf("Creating %d\n",nextId)
    x := Foo{Id:nextId}
    nextId++
    return &x,nil
}

func main() {
    pool := pools.NewResourcePool(fooFactory,1,4,30 * time.Second)
    fmt.Printf("Pool: %v\n",pool)
    x,_ := pool.Get(0);
    fmt.Printf("X: %v\n",x)
    y,_ := pool.Get(0);
    fmt.Printf("Y: %v\n",y)
}

And I get the following error output:

$ ./bin/x 
Pool: &{0xc208036000 0x400cb0 1 30000000000 0 0}
Creating 0
X: &{0}
fatal error: all goroutines are asleep - deadlock!

goroutine 16 [chan receive]:
github.com/youtube/vitess/go/pools.(*ResourcePool).get(0xc208022180, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0)
        /home/jonhall/git/doublecheck/go/_vendor/src/github.com/youtube/vitess/go/pools/resource_pool.go:109 +0x429
github.com/youtube/vitess/go/pools.(*ResourcePool).Get(0xc208022180, 0x0, 0x0, 0x0, 0x0, 0x0)
        /home/jonhall/git/doublecheck/go/_vendor/src/github.com/youtube/vitess/go/pools/resource_pool.go:87 +0x5a
main.main()
        /home/jonhall/git/doublecheck/go/src/x/x.go:32 +0x1c4

goroutine 19 [finalizer wait]:
runtime.park(0x413130, 0x5504b0, 0x54efc9)
        /usr/lib/go/src/pkg/runtime/proc.c:1369 +0x89
runtime.parkunlock(0x5504b0, 0x54efc9)
        /usr/lib/go/src/pkg/runtime/proc.c:1385 +0x3b
runfinq()
        /usr/lib/go/src/pkg/runtime/mgc0.c:2644 +0xcf
runtime.goexit()
        /usr/lib/go/src/pkg/runtime/proc.c:1445

Am I using it wrong, or is there a bug?

hi vitess team,
I'm doing a test on sharding a table by keyrange. I created a keyspace "test_keyspace" and made two shards for it ("-80", "80-"). after all vttablet started up, I inserted 2 test records (id=0x0A, 0x90) using "vtclient2". The tool executed successfully, but i found all data was inserted into fist shard. (the second record with id=0x90 should go to second shard in my expectation)
After some investigation, I noticed that there might be a problem in the comparing logic of keyspaceid.
The KeyspaceId is a golang builtin string which contains hex coded value. In my test, the max key for my first shard is [0x80]. when inserting new record, vtclient2 trys to compare my binded value(0x90) with max key, it should return index 1 since 0x90 > 0x80. however, the function FindShardForValue(in vt/key/shard.go) returns 0 in my case. Further debug shows the binded value(0x90) was transformed to [0 0 0 0 0 0 0 0x90] before comparing to [0x80].
Is it a bug? or did I miss any configuration when creating shard? Should KeyspaceId also be uniformed to [0 0 0 0 0 0 0 0x80] in zk?

//watson

Paste: http://paste.openstack.org/show/163066/

Doesn't affect run of vitess, but it seems the 'cover' module is not installed due to permission denied issue. bootstrap.sh doesn't allow me to run it as root, so don't know of a simple way to fix this..

first I have a live shard in a cell and two vttablets/databases instance in the shard, I create a table in the master database of the shard and insert data into the database through the mysql client(not vtgate), the other database of the shard could auto sync data from master database.Then I add a new vttablet and daabase instance into the shard,but I found that the new database didn't auto sync data from master database of the shard finally, the new database is empty.
Thanks very much!

can't load package: the code.google.com/p/go.tools/cmd/cover command has moved; use golang.org/x/tools/cmd/cover instead.

doesn't repeat on go 1.3.3

The following program crashes with the panic:

package main

import (
    "github.com/youtube/vitess/go/vt/sqlparser"
)

func main() {
    sql := "SET o=t(0=t(0=t(0=t(0=t(0=t(0=t(0=t(0=t(0=t(0=t(0=t(0=t(0=t(0=t(0=t(0=t(0=t(0=t(0=t(0=t(0=t(0=t(0=t(0=t(0=t(0=t(0=t(0=t(0=t(0=t(0=t(0=t(0=t(0=t(0=t(0=t(0=t(0=t(0=t(0=t(0=t(0=t(0=t(0=t(0=t(0=t(0=t(0=t(0"
    sqlparser.Parse(sql)
}

panic: runtime error: index out of range

goroutine 1 [running]:
github.com/youtube/vitess/go/vt/sqlparser.yyParse(0x7fd9d9e3bc08, 0xc208066000, 0x662df8)
    src/github.com/youtube/vitess/go/vt/sqlparser/yaccpar:207 +0xd7c1
github.com/youtube/vitess/go/vt/sqlparser.Parse(0x5cd9f0, 0xc9, 0x0, 0x0, 0x0, 0x0)
    src/github.com/youtube/vitess/go/vt/sqlparser/ast.go:31 +0xa0
main.main()
    sql.go:9 +0x40

on commit

discovered with https://github.com/dvyukov/go-fuzz

Does vitess support distributed transaction when a ACID operation occured across more than two shards?

Bootstrap.sh can not continue once you break the progress
it just skip some step

vagrant@vagrant-ubuntu-trusty-64:/vagrant/src/github.com/youtube/vitess$ ./bootstrap.sh
skipping zookeeper build
skipping protobuf build
skipping gRPC build

Actually, these step not success, it just be skipping.... what the hell.

Hello guys,

Thanks for the work on this. But are they any plans of supporting PostgreSQL? Would it even be possible. Sorry if I have missed any prior discussions on this front.

Regards,
Karan Misra

I have tried the sqlparser library and I get parsing errors on a few sql queries. Some examples:
DELETE FROM term_hierarchy AS th WHERE th.parent = 1015
Error at position 30: as

SELECT s1 FROM t1 WHERE s1 = ANY (SELECT s1 FROM t2)
Error at position 41: select

DELETE FROM t1, t2 USING t1 INNER JOIN t2 INNER JOIN t3 WHERE t1.id=t2.id AND t2.id=t3.id
Error at position 16: ,

I hava start vttablet and vtgate, everything is OK.
but now, need use shard function ,i create two shard that is -64,64- . but i don't know how to copy data from keyspace/0 to keyspace/-64 and keyspace/64- .

can autors give some examples with shell or or introductions about how to use shard function.

thanks a lot.

Why it has not release or or stable version?

Hi, I try to use MariaDB with vitess, but find that the bootstrap.sh can not get it correctly.

pushd third_party
version="10.0.10"
wget https://downloads.mariadb.org/interstitial/mariadb-${version}/kvm-tarbake-jaunty-x86/mariadb-${version}.tar.gz
tar xzf mariadb-${version}.tar.gz
popd
` ``

MariaDB is 10.0.13 now, maybe the bash needs upgrade?

Thank you! 

hi,
I hava been run vttablet ,vtgate ,zk three servers . but I not now how to run a golang client to visit vtgate .

can you tell me ,which golang's package can i use for vtgate client?

think you .

Unzip is needed when compile protobuf

the build error log

make[1]: Leaving directory `/vagrant/src/github.com/youtube/vitess/third_party/zookeeper/zookeeper-3.3.5/src/c'
--2015-03-12 04:08:58--  https://github.com/google/protobuf/archive/v3.0.0-alpha-1.zip
Resolving github.com (github.com)... 192.30.252.131
Connecting to github.com (github.com)|192.30.252.131|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://codeload.github.com/google/protobuf/zip/v3.0.0-alpha-1 [following]
--2015-03-12 04:09:15--  https://codeload.github.com/google/protobuf/zip/v3.0.0-alpha-1
Resolving codeload.github.com (codeload.github.com)... 192.30.252.147
Connecting to codeload.github.com (codeload.github.com)|192.30.252.147|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [application/zip]
Saving to: ‘v3.0.0-alpha-1.zip’

    [                                        <=>                                                       ] 2,286,099    332KB/s   in 11s    

2015-03-12 04:09:39 (212 KB/s) - ‘v3.0.0-alpha-1.zip’ saved [2286099]

./bootstrap.sh: line 59: unzip: command not found
protobuf build failed

see #470

Hi All:
I recently try to play with vitess , but there are no documents basically . we can just read the code ..... can authors write something to introduce more to us ?

thank you very much .

hi ,sougou:

when i use golang's client about vitess to save my data , there is a question appeared.
In the vttablet, the log show as follows:

W0331 16:59:06.763275 12975 sqlquery.go:297] error: missing bind var v3: Sql: "insert into t_login_history(f_account_id, f_ip, f_createtime) values (?,?,?)", BindVars: {v0: 1, v1: "localhost", v2: 1427792377099, #maxLimit: 10001, }

i just only hava 3 params , why it show me , missing bind var v3?

if i use another sql like :
insert into t_login_history(f_account_id, f_ip, f_createtime) values (1, "localhost", 123456), it will be ok.

is there something wrong about my code ?

test_stream_parity can fail if it's running on a slow disk (i.e., not SSD/ramdisk). This is bad because it's part of our make site_test suite, and presumably should pass even in less resource-heavy environments.

The particular failure mode that I saw was:

-- 2014-10-30 10:33:07,052 utils:50 INFO ===== test_set_insert_id (__main__.TestUpdateStream)                                                                                                     [1847/8060]
-- 2014-10-30 10:33:07,052 utils:50 INFO ===== test_set_insert_id (__main__.TestUpdateStream) ... 
-- 2014-10-30 10:33:07,243 update_stream:339 DEBUG test_set_insert_id: starting @ {'MariaDB': '0-62344-526'}
-- 2014-10-30 10:33:07,294 utils:50 INFO ===== test_set_insert_id (__main__.TestUpdateStream) ... ok
-- 2014-10-30 10:33:07,294 utils:50 INFO ===== test_stream_parity (__main__.TestUpdateStream)
-- 2014-10-30 10:33:07,294 utils:50 INFO ===== test_stream_parity (__main__.TestUpdateStream) ... 
-- 2014-10-30 10:33:17,306 update_stream:284 DEBUG run_test_stream_parity starting @ {'MariaDB': '0-62344-652'}
-- 2014-10-30 10:33:18,240 update_stream:316 DEBUG Test Failed - # of records mismatch, master [{'Category': 'DML', 'GTIDField': {}, 'Timestamp': 1414690397L, 'TableName': 'vt_a', 'Sql': '', 'PkRows': [[('
eid', 1L), ('id', 1L)]]}, {'Category': 'DML', 'GTIDField': {}, 'Timestamp': 1414690397L, 'TableName': 'vt_a', 'Sql': '', 'PkRows': [[('eid', 2L), ('id', 2L)]]}, {'Category': 'DML', 'GTIDField': {}, 'Timest
amp': 1414690397L, 'TableName': 'vt_a', 'Sql': '', 'PkRows': [[('eid', 3L), ('id', 3L)]]}, {'Category': 'DML', 'GTIDField': {}, 'Timestamp': 1414690397L, 'TableName': 'vt_a', 'Sql': '', 'PkRows': [[('eid',
 4L), ('id', 4L)]]}, {'Category': 'DML', 'GTIDField': {}, 'Timestamp': 1414690397L, 'TableName': 'vt_a', 'Sql': '', 'PkRows': [[('eid', 5L), ('id', 5L)]]}, {'Category': 'DML', 'GTIDField': {}, 'Timestamp':
 1414690397L, 'TableName': 'vt_a', 'Sql': '', 'PkRows': [[('eid', 6L), ('id', 6L)]]}, {'Category': 'DML', 'GTIDField': {}, 'Timestamp': 1414690397L, 'TableName': 'vt_a', 'Sql': '', 'PkRows': [[('eid', 7L),
 ('id', 7L)]]}, {'Category': 'DML', 'GTIDField': {}, 'Timestamp': 1414690397L, 'TableName': 'vt_a', 'Sql': '', 'PkRows': [[('eid', 8L), ('id', 8L)]]}, {'Category': 'DML', 'GTIDField': {}, 'Timestamp': 1414
690397L, 'TableName': 'vt_a', 'Sql': '', 'PkRows': [[('eid', 9L), ('id', 9L)]]}, {'Category': 'DML', 'GTIDField': {}, 'Timestamp': 1414690397L, 'TableName': 'vt_a', 'Sql': '', 'PkRows': [[('eid', 10L), ('i
d', 10L)]]}, {'Category': 'DML', 'GTIDField': {}, 'Timestamp': 1414690397L, 'TableName': 'vt_a', 'Sql': '', 'PkRows': [[('eid', 11L), ('id', 11L)]]}, {'Category': 'DML', 'GTIDField': {}, 'Timestamp': 14146
90397L, 'TableName': 'vt_a', 'Sql': '', 'PkRows': [[('eid', 12L), ('id', 12L)]]}, {'Category': 'DML', 'GTIDField': {}, 'Timestamp': 1414690397L, 'TableName': 'vt_a', 'Sql': '', 'PkRows': [[('eid', 13L), ('
id', 13L)]]}, {'Category': 'DML', 'GTIDField': {}, 'Timestamp': 1414690397L, 'TableName': 'vt_a', 'Sql': '', 'PkRows': [[('eid', 14L), ('id', 14L)]]}, {'Category': 'DML', 'GTIDField': {}, 'Timestamp': 1414
690397L, 'TableName': 'vt_a', 'Sql': '', 'PkRows': [[('eid', 15L), ('id', 15L)]]}, {'Category': 'POS', 'GTIDField': {'MariaDB': '0-62344-653'}, 'Timestamp': 1414690397L, 'TableName': '', 'Sql': '', 'PkRows
': []}] replica [{'Category': 'DML', 'GTIDField': {}, 'Timestamp': 1414690397L, 'TableName': 'vt_insert_test', 'Sql': '', 'PkRows': [[('id', 1000252L)]]}, {'Category': 'DML', 'GTIDField': {}, 'Timestamp': 
1414690397L, 'TableName': 'vt_insert_test', 'Sql': '', 'PkRows': [[('id', 1000253L)]]}, {'Category': 'DML', 'GTIDField': {}, 'Timestamp': 1414690397L, 'TableName': 'vt_insert_test', 'Sql': '', 'PkRows': [[
('id', 1000254L)]]}, {'Category': 'DML', 'GTIDField': {}, 'Timestamp': 1414690397L, 'TableName': 'vt_insert_test', 'Sql': '', 'PkRows': [[('id', 1000255L)]]}, {'Category': 'POS', 'GTIDField': {'MariaDB': '
0-62344-652'}, 'Timestamp': 1414690397L, 'TableName': '', 'Sql': '', 'PkRows': []}]
-- 2014-10-30 10:33:18,241 utils:50 INFO ===== test_stream_parity (__main__.TestUpdateStream) ... FAIL
-- 2014-10-30 10:33:18,242 update_stream:126 DEBUG Tearing down the servers and setup
-- 2014-10-30 10:33:18,242 tablet:624 DEBUG killing vttablet: test_nj-0000062344
-- 2014-10-30 10:33:18,242 tablet:624 DEBUG killing vttablet: test_nj-0000062345
-- 2014-10-30 10:33:18,282 vtclient:127 WARNING db connection failed: test_keyspace.0.master :15009, (error(111, 'Connection refused'), '')
-- 2014-10-30 10:33:18,283 vtdb_logger:24 INFO Fetched keyspace test_keyspace from topo_client in 0.000878 secs
-- 2014-10-30 10:33:18,283 vtdb_logger:64 WARNING vtclient_exception for test_keyspace.0.master: ('unable to create vt connection', 'test_keyspace.0.master', ':15009', FatalError
((error(111, 'Connection refused'), ':15009 master test_keyspace/0>'),))
Exception in thread write_thd:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 763, in run
    self.__target(*self.__args, **self.__kwargs)
  File "test/update_stream.py", line 197, in perform_writes
    self._exec_vt_txn(self._populate_vt_insert_test)
  File "test/update_stream.py", line 269, in _exec_vt_txn
    vtdb_conn = self._vtdb_conn('localhost:%u' % master_tablet.port)
  File "test/update_stream.py", line 263, in _vtdb_conn
    conn.connect()
  File "mariadb/py-vtdb/vtdb/vtclient.py", line 95, in connect
    return self._connect()
  File "mariadb/py-vtdb/vtdb/vtclient.py", line 133, in _connect
    'unable to create vt connection', db_key, host_addr, db_exception)
OperationalError: ('unable to create vt connection', 'test_keyspace.0.master', ':15009', FatalError((error(111, 'Connection refused'), ''),))

-- 2014-10-30 10:33:18,464 utils:214 DEBUG run: ['mariadb/bin/mysqlctl', '-log_dir', 'mariadb/vtdataroot/tmp', '-tablet_uid', '62344', 't
eardown', '-force'] extra_env={'EXTRA_MY_CNF': 'mariadb/src/github.com/youtube/vitess/config/mycnf/master_mariadb.cnf'}
-- 2014-10-30 10:33:18,466 utils:214 DEBUG run: ['mariadb/bin/mysqlctl', '-log_dir', 'mariadb/vtdataroot/tmp', '-tablet_uid', '62345', 't
eardown', '-force'] extra_env={'EXTRA_MY_CNF': 'mariadb/src/github.com/youtube/vitess/config/mycnf/master_mariadb.cnf'}
-- 2014-10-30 10:33:22,704 utils:50 INFO ===== ======================================================================
-- 2014-10-30 10:33:22,704 utils:50 INFO ===== FAIL: test_stream_parity (__main__.TestUpdateStream)

@enisoc and I hypothesize that this is flaky because of the first two lines of the test:

  def test_stream_parity(self):
    master_start_position = _get_master_current_position()
    replica_start_position = _get_repl_current_position()

It could be that if there is replication lag at the time that the test starts, the master and replica have different current positions, which cause their events streams to not be equal.

I've observed following errors in recent builds that cause unit test fail.

go build testmain: /home/travis/.gimme/versions/go1.4.linux.amd64/pkg/tool/linux_amd64/6l: signal: killed
go build testmain: /home/travis/.gimme/versions/go1.4.linux.amd64/pkg/tool/linux_amd64/6l: signal: killed
go build github.com/youtube/vitess/go/vt/vtgate: /home/travis/.gimme/versions/go1.4.linux.amd64/pkg/tool/linux_amd64/6g: signal: killed
go build testmain: /home/travis/.gimme/versions/go1.4.linux.amd64/pkg/tool/linux_amd64/6l: signal: killed
go build testmain: /home/travis/.gimme/versions/go1.4.linux.amd64/pkg/tool/linux_amd64/6l: signal: killed
go build testmain: /home/travis/.gimme/versions/go1.4.linux.amd64/pkg/tool/linux_amd64/6l: signal: killed
go build testmain: /home/travis/.gimme/versions/go1.4.linux.amd64/pkg/tool/linux_amd64/6l: signal: killed

The 'getting started' page makes it look like Vitess only works with GCE/Kubernetes. I think it would be great if we add a sentence saying something like 'You can still use Vitess in a non-GCE environment, go here for more details'.