qihoo360 / atlas Goto Github PK
View Code? Open in Web Editor NEWA high-performance and stable proxy for MySQL, it is developed by Qihoo's DBA and infrastructure team
License: GNU General Public License v2.0
A high-performance and stable proxy for MySQL, it is developed by Qihoo's DBA and infrastructure team
License: GNU General Public License v2.0
我看了一下你们写出来的架构图。上面就只有一个MASTER-SLAVE。。那单个MASTER不是就只有单台写的功能。。。。这一个若我想两台服务器扩展写的这一个功能那是实现不了咯?
亚马逊的RDS只有url endpoint, 不提供IP
在创建一个表时使用tokudb存储引擎, 但出查询时 存储引擎 被换为innodb?
主从都支持tokudb, 数据库mariadb10
报错信息如下:
ALERT: failed to execute mysql_stmt_execute(): Err1243 Unknown prepared statement handler (11) given to mysqld_stmt_execute
FATAL: database error, exiting...
测试脚本如下:
sysbench --test=oltp --num-threads=128 --max-requests=8000000 --oltp-test-mode=nontrx --db-driver=mysql --mysql-db=my_db --mysql-host=10.105.45.151 --mysql-port=11811 --mysql-user=xm_dba --mysql-password=123 --oltp-nontrx-mode=select --db-ps-mode=auto run
非常感谢
后端两个mysql,一主一从,使用sysbench select测试时,预期应该将请求分配到只读mysql主机上,但是却分配到主库上。
当有多台从机时,
在用phpmyadmin时,在上面执行的INSERT 的SQL语句会变成latin1写入,从而导致乱码
在mysql-proxy和phpmyadmin中,都使用了utf8来连接。
但是如果只有一台主机,没有从机的时候,乱码情况是不会出现的
请问这个是什么原因?
atlas是不是也可以用于pgsql?
环境 centos6.5 x64
安装版本 2.1
安装方式:rpm安装
现象: ./mysql-proxyd test start后一直等待状态,不显示失败 也不显示成功 一直卡在那里
log:
2014-06-06 14:53:11: (message) mysql-proxy 0.8.2 started - instance: test
2014-06-06 14:53:11: (message) proxy listening on port 0.0.0.0:1234
2014-06-06 14:53:11: (message) added read/write backend: 192.168.10.200:3306
2014-06-06 14:53:11: (message) added read-only backend: 192.168.10.202:3306
2014-06-06 14:53:11: (message) added read-only backend: 192.168.10.203:3306
2014-06-06 14:53:11: (message) added read-only backend: 192.168.10.204:3306
2014-06-06 14:53:11: (message) chassis-event-thread.c:235: starting 7 threads
后续操作:重启后 无效。改用源码方式安装 无效。重启mysql集群的mysql服务,无效。杀死进程后重新启动 ,无效。
求解决
出现了这种情况,链接的次数多了以后,会出现随机的链接等待。是释放的方式不对还是本身使用需要注意什么?
使用中已经使用下面代码关闭了链接
python代码:
conn = MySQLdb.connect()
cursor = conn.cursor()
cursor.close()
conn.close()
代理后端主从复制的两个mysql,使用sysbench测试,到512线程时,提示丢失连接
性能与直连mysql 使用oltp的模式测试有接近一万的请求次数差距。
使用select读测试有在开启128线程时,有接近3万的请求量差距
新手提个问题哈。目前只能通过主键分表,查询的时候必须带主键查询,那使用这种类 nosql 的方式来使用 mysql 的意义在哪里?用于垂直切分出来的大表再水平切分的场景?例如文章摘要表用单表,然后文章详情表用多表,是适用这种场景吗。
使用druid作为datasource的配置。
在测试添加数据过程中,atlas报表不存在,但实际上这张表在master是存在的。求解。
jdbc:mysql://19.134.4.216:3308/tpcc?useUnicode=true&characterEncoding=UTF-8&autoReconnect=true
提示:tpcc.t_user1不存在
请问能不能提供DEB的包?我的系统是ubuntu。
#Atlas后端连接的MySQL从库的IP和端口,@后面的数字代表权重,用来作负载均衡,若省略则默认为1,可设置多项,用逗号分隔
proxy-read-only-backend-addresses = bj-db3:5311,bj-db2:4311,bj-db4:6311
生产环境,我们发现某个从库的连接数非常多,而其他的非常少,可能是什么原因
atlas的分表算法是什么。
计划什么时候推出sharding功能?
一直出现这个错误。
不知道是不是BUG。当从出现延迟与复制线程停止时应该全部走master读写而不走slave.希望告知下。
使用Atlas代理MariaDB Galera Cluster集群,查询请求是均衡的,但是写入请求都到集群的第一台数据库上了。需要怎么设置吗?
打包好的 deb 只有2.2的,新版没有呢?
作者,
您好!
经过反复的测试,通过mysql客户端工具(SQLyogEnt、MysqlFront) 连接后指向update create 语句会报错,sql语句可以确定没有问题,因为我直接通过mysqlclient连接执行是会正确的解析执行的。
日志文件里面内容如下:
C:192.168.0.32:65023 S:192.168.0.17:3306 ERR 0.350 "create table s( id int unsigned not null AUTO_INCREMENT, primary key (id)) ENGINE=InnoDB AUTO_INCREMENT=1 DEFAULT CHARSET=utf8;"
我知道,可能不是bug,可能是客户端工具多发了某些指令,导致atlas 调用master出错,但是日志里面看不出问题,请求帮忙。
请问是不是每次在数据库里添加一个新用户,都要重新启动一下mysql-proxyd服务啊?那不就会有短暂的断开情况,请问怎么解决?
如果配置分表是:
person.mt.id.3
下面的SQL将不会在表名后面添加数字后缀:
select * from mt where mt.id=1
打了一下log,发现plugins/proxy/proxy-plugin.c 的
get_column_index函数中table_name是【dbname.table_name】的格式,导致无法识别分表
期待能处理select和where中带表名的sql
select mt.id from mt where mt.id=1
atlas后台连接的是一主一从,普通sql读写均正常,唯独调用存储过程时被atlas自动路由到了从库,请问如何能让atlas调用主库的存储过程?
atlas会造成jdbc url里面设置的socketTimeout不生效吗?
我们的数据库有一定负载的情况下会出现下面的报错,但当时我们的Mysql状态都是可用的。
由于2.2.1还没有deb包,我们使用的是2.2的Atlas
2014-12-26 19:39:08: (critical) proxy-plugin.c.1454: I have no server backend, closing connection
2014-12-26 19:39:08: (critical) network-mysqld.c.1383: plugin_call(CON_STATE_READ_QUERY) failed
2014-12-26 19:39:08: (critical) proxy-plugin.c.1454: I have no server backend, closing connection
2014-12-26 19:39:08: (critical) network-mysqld.c.1383: plugin_call(CON_STATE_READ_QUERY) failed
2014-12-26 20:09:06: (warning) (libevent) event_del: event has no event_base set.
。。。省略很多相同的log 。。。
2014-12-26 20:09:08: (critical) proxy-plugin.c.1454: I have no server backend, closing connection
2014-12-26 20:09:08: (critical) network-mysqld.c.1383: plugin_call(CON_STATE_READ_QUERY) failed
2014-12-26 20:09:08: (critical) proxy-plugin.c.1454: I have no server backend, closing connection
2014-12-26 20:09:08: (critical) network-mysqld.c.1383: plugin_call(CON_STATE_READ_QUERY) failed
2014-12-26 20:39:05: (warning) (libevent) event_del: event has no event_base set.
2014-12-26 20:39:05: (critical) proxy-plugin.c.1454: I have no server backend, closing connection
2014-12-26 20:39:05: (critical) network-mysqld.c.1383: plugin_call(CON_STATE_READ_QUERY) failed
2014-12-26 20:39:05: (critical) proxy-plugin.c.1454: I have no server backend, closing connection
2014-12-26 20:39:05: (critical) network-mysqld.c.1383: plugin_call(CON_STATE_READ_QUERY) failed
。。。省略很多相同的log 。。。
相关的配置参数为:
proxy-backend-addresses = 192.168.1.12:3306
proxy-read-only-backend-addresses = 192.168.1.13:3306,192.168.1.14:3306
keepalive = true
event-threads = 24
再阿里云使用RDS,然后使用atlas做proxy.外部机器链接时偶尔提示无法链接数据库。同时看日志非常多
2015-02-06 09:36:59: (warning) last message repeated 16 times
2015-02-06 09:36:59: (warning) (libevent) event_del: event has no event_base set.
2015-02-06 09:37:55: (warning) last message repeated 17 times
2015-02-06 09:37:55: (warning) (libevent) event_del: event has no event_base set.
2015-02-06 09:38:26: (warning) last message repeated 44 times
2015-02-06 09:38:26: (warning) (libevent) event_del: event has no event_base set.
My computer information is:
┌─[yuchao@localhost] - [/Github/Atlas] - [2014-09-30 01:10:54]1/RELEASE_X86_64 x86_64
└─[0] <git:(master 102f9f✱✈) > uname -a
Darwin localhost 13.4.0 Darwin Kernel Version 13.4.0: Sun Aug 17 19:50:11 PDT 2014; root:xnu-2422.115.4
when I make the Atlas,
It is some errors:
libtool: link: gcc -dynamiclib -o .libs/libmysql-chassis.0.dylib .libs/libmysql_chassis_la-lua-load-factory.o .libs/libmysql_chassis_la-lua-scope.o .libs/libmysql_chassis_la-chassis-plugin.o .libs/libmysql_chassis_la-chassis-log.o .libs/libmysql_chassis_la-chassis-mainloop.o .libs/libmysql_chassis_la-chassis-event-thread.o .libs/libmysql_chassis_la-chassis-keyfile.o .libs/libmysql_chassis_la-chassis-path.o .libs/libmysql_chassis_la-chassis-filemode.o .libs/libmysql_chassis_la-chassis-limits.o .libs/libmysql_chassis_la-chassis-shutdown-hooks.o .libs/libmysql_chassis_la-chassis-stats.o .libs/libmysql_chassis_la-chassis-frontend.o .libs/libmysql_chassis_la-chassis-options.o .libs/libmysql_chassis_la-chassis-unix-daemon.o .libs/libmysql_chassis_la-chassis-win32-service.o /usr/local/lib/libevent.dylib -L/usr/local/Cellar/glib/2.40.0_1/lib -L/usr/local/opt/gettext/lib -L/usr/local/lib -llua -lgmodule-2.0 -lgthread-2.0 ./.libs/libmysql-chassis-timing.dylib /Users/yuchao/Github/Atlas/src/.libs/libmysql-chassis-glibext.dylib ./.libs/libmysql-chassis-glibext.dylib -lm -ldl -lcrypto -ljemalloc -lglib-2.0 -lintl -Wl,-dylib_file -Wl,/usr/local/mysql-proxy/lib/libmysql-chassis-glibext.0.dylib:/Users/yuchao/Github/Atlas/src/.libs/libmysql-chassis-glibext.dylib -install_name /usr/local/mysql-proxy/lib/libmysql-chassis.0.dylib -compatibility_version 1 -current_version 1.0 -Wl,-single_module
Undefined symbols for architecture x86_64:
"_network_backends_free", referenced from:
_chassis_free in libmysql_chassis_la-chassis-mainloop.o
"_network_mysqld_con_free", referenced from:
_chassis_event_thread_free in libmysql_chassis_la-chassis-event-thread.o
"_network_mysqld_con_handle", referenced from:
_chassis_event_handle in libmysql_chassis_la-chassis-event-thread.o
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make[3]: *** [libmysql-chassis.la] Error 1
make[2]: *** [all] Error 2
make[1]: *** [all-recursive] Error 1
make: *** [all] Error 2
how to solu the error?
请教一下哈,Atlas对于多子表的查询语句,支持排序吗?如果支持排序,对于大结果集的排序是怎样优化排序性能的,非常感谢哈
mysql集群支持事务吗?
配制了主从数据库,主库为192.168.1.210,从库为:192.168.1.208
已经测试主从数据的同步没有问题,主库的修改从库都会同步
但是,我在使用360的atlas2.2.1后,配置如下:
proxy-backend-addresses = 192.168.1.210:3306
#Atlas后端连接的MySQL从库的IP和端口,@后面的数字代表权重,用来作负载均衡,若省略则默认为1,可设置多项,用逗号分隔
proxy-read-only-backend-addresses = 192.168.1.208:3306
我在使用atlas2.2.1做为中间件,实现读写分离。但是我连中间件后,写操作是发给主数据库了,主库也修改了。但是从数据库没有从主数据库同步数据。
我要是直接update主数据库,从数据的数据就马上同步过来(说明主从是没有问题的)
有没有遇到同样问题的朋友?
navicat管理数据库 看不到事件,之前用mysql-proxy 用navicat是可以管理事件的
atlas 不支持分表后的联合查询?
select addr from t_addr,t_user where id=userid and userid = ?
配置如下:
tables = tpcc.t_user.id.6,tpcc.t_addr.userid.6
这种查询,直接报表不存在
请问这个问题怎么解决?
I installed XAMPP at first;
I installed atlas and configured it;
I connect atlas remotely through mysql -hwww.xxxxx.com -u root -pxxxxx -P1234 and it works OK;
I run mysql> show databases; but failed with error:
mysql> show databases;
ERROR 2006 (HY000): MySQL server has gone away
No connection. Trying to reconnect...
Connection id: 13
Current database: *** NONE ***
ERROR 2013 (HY000): Lost connection to MySQL server during query
I check test.log of atlas and found:
2015-05-24 16:26:12: (warning) (libevent) event_del: event has no event_base set.
2015-05-24 16:26:12: (critical) proxy-plugin.c.1454: I have no server backend, closing connection
2015-05-24 16:26:12: (critical) network-mysqld.c.1383: plugin_call(CON_STATE_READ_QUERY) failed
2015-05-24 16:26:31: (warning) (libevent) event_del: event has no event_base set.
2015-05-24 16:26:31: (critical) proxy-plugin.c.1454: I have no server backend, closing connection
2015-05-24 16:26:31: (critical) network-mysqld.c.1383: plugin_call(CON_STATE_READ_QUERY) failed
What's wrong? How to fix it?
*[05/08/2014 16:52:24] C:192.168.1.254:29064 S:192.168.1.18:3306 OK 1.357 "SET @p_staff_name='user',@p_password=123456,@p_ip='192.168.1.207',@p_browser='chrome',@p_mac='00-ff-45-ec-67-90',@p_hd='',@p_cpu='GenuineIntel-BFEBFBFF-000206A7-00000000-00000000',@p_bios='NULL',@p_login_id='0',@p_staff_id='0',@p_xy_staff_id='0',@p_entity_id='0',@p_staff_type='',@p_staff_id_list='',@p_functions='',@p_result=''"
*[05/08/2014 16:52:24] C:192.168.1.254:29064 S:192.168.1.15:3306 OK 10.199 "CALL oap_staff_login_ex(@p_staff_name,@p_password,@p_ip,@p_browser,@p_mac,@p_hd,@p_cpu,@p_bios,@p_login_id,@p_staff_id,@p_xy_staff_id,@p_entity_id,@p_staff_type,@p_staff_id_list,@p_functions,@p_result)"
[05/08/2014 16:52:24] C:192.168.1.254:29064 S:192.168.1.18:3306 OK 1.779 "SELECT @p_login_id,@p_staff_id,@p_xy_staff_id,@p_entity_id,@p_staff_type,@p_staff_id_list,@p_functions,@p_result"
php在调用存储过程中: set 定义的参数值去到从库(192.168.1.18)上面执行,call 需要附带这些参数在主库(192.168.1.15)上执行,导致参数传入无效。。
请问是否可以指定SET 一直都在主库上面执行呢?
SET AUTOCOMMIT=0
START TRANSACTION
当我开启事务的时候,这些语句是怎么处理的?是直接发送到Master上面执行吗?
一个主库一个从库,都设置了utf8,不通过中间层实现读写分离的时候没有中文乱码问题,使用atlas客户端连接,则字符集为latin1,已经在conf中设置了 charset=utf8.
atals版本为最新版本。
'm handling something about sharding recently. What I want to do is to add more machines or nodes to my system so that storing more data is possibale, or scale-out.
I found many people add a middle layer between their applications and databases, mysql for example. That seems to be perfect for me at first glance. But then I'm puzzled about the query and join operation.
Q1:
How to handle queries with offset and limit like:
'select * from xxxx where yyyy order by col_a asc limit num_b, num_c'
Just perform the query 'select * from xxxx where yyyy order by col_a asc limit 0, num_b + num_c' on each machine, merge the results and return [num_b, num_b + num_c)?
Q2:
How to handle the join operations within different databases?
Currently, I'm trying to solve the first problem by merging the results on the middle layer. There comes another tough problem. That is to reduce the memory usage. Working on, and suggestions are welcome~
General error: 1243 Unknown prepared statement handler (27) given to mysqld_stmt_execute (SQL: select * from 。。。
很多这种错误。
然后我查看proxy的日志,
里面很多类似错误:
2015-06-11 21:03:03: (warning) last message repeated 101 times
2015-06-11 21:03:03: (warning) (libevent) event_del: event has no event_base set.
2015-06-11 21:03:04: (warning) last message repeated 101 times
2015-06-11 21:03:04: (warning) (libevent) event_del: event has no event_base set.
2015-06-11 21:03:07: (warning) last message repeated 101 times
2015-06-11 21:03:07: (warning) (libevent) event_del: event has no event_base set.
2015-06-11 21:03:08: (warning) last message repeated 101 times
2015-06-11 21:03:08: (warning) (libevent) event_del: event has no event_base set.
2015-06-11 21:03:12: (warning) last message repeated 101 times
2015-06-11 21:03:12: (warning) (libevent) event_del: event has no event_base set.
2015-06-11 21:03:52: (warning) last message repeated 35 times
2015-06-11 21:03:52: (warning) (libevent) event_del: event has no event_base set.
2015-06-11 21:04:23: (warning) last message repeated 9 times
我的网站的PV50000
当时在线1000IP。atlas好像有点卡了。有些返回不正常。
大家有这样的问题吗?
我c3p0中的配置:
<property name="maxPoolSize" value="100" />
可以吗?
When I switched mysqld to port 3360 and Atlas is on 3306 and all requests are going through Atlas I have warnings in mysqld error.log:
[Warning] Unsafe statement written to the binary log using statement format since BINLOG_FORMAT = STATEMENT
But! If I connect to backend mysqld and see BINLOG_FORMAT is still MIXED as it is to be.
My replication was broken.
Any ideas of what is going on?
现在得重启服务才行。
MySQL主从用了Atlas之后经常报1105的错误,如下:
SQL: UPDATE [Table]forums SET todayposts='0'
Error: Proxy Warning - Syntax Forbidden
Errno.: 1105
报这个错的原因是Atlas拦截了没有Where子句的UPDATE语句。
问题:由于此项目中这种语句是正常的逻辑,而且子站比较多,Atlas中能不能设置不检测这种语句呢?如果没有这个设置,能否后期考虑添加此项配置呢?
谢谢解答~
在java jdbc跨分表合并查询中会出现乱码问题
建议功能
一条sql语句想要同步插入到不同分表中(按不同字段分表eg:表1按userID分表,表二按datetime字段分表)
还有。是否有考虑可以支持按库分表
问题1
目前我遇到如下问题,一是慢查,我设置了,可是无法在日志里面打印出来,配置如下
log-path = /usr/local/mysql-proxy/log
sql-log = OFF
sql-log-slow = 1000
问题2
如何把前端业务IP慢查透传到后端数据库服务器上
在管理接口add master时发现不能用域名,只能用IP。
jdbc连接时判断mysql服务器是否大于5.6.4而决定是否保持datetime跟timestamp是否支持fractional seconds , 判断mysql 服务器是否支持小数秒,其实为判断服务器的版本是否大于 5.6.4
导致jdbc连接atlas时判断服务器版本小于5.6.4从而不支持fractional seconds而无法保持timestamp或者datetime的精度
请问是否能配置atlas返回的mysql server version的值?
Atlas 和zookeeper集合完成分库和分布式事务,分布式锁和最终数据一致性。
同时还可以完成分库后数据表的map,和联合查询等等。。
求大神们拍砖。。
Hi,all:
当我执行完configure,在make的时候,报如下错误:
lua-scope.c:44: error: expected ‘)’ before ‘’ token
lua-scope.c:67: warning: ‘g_mutex_new’ is deprecated (declared at /usr/local/include/glib-2.0/glib/deprecated/gthread.h:272)
lua-scope.c: In function ‘lua_scope_free’:
lua-scope.c:91: warning: ‘g_mutex_free’ is deprecated (declared at /usr/local/include/glib-2.0/glib/deprecated/gthread.h:274)
make[2]: ** [libmysql_chassis_la-lua-scope.lo] Error 1
make[2]: Leaving directory /data/altas/Atlas-2.2/src' make[1]: *** [check] Error 2 make[1]: Leaving directory
/data/altas/Atlas-2.2/src'
make: *** [check-recursive] Error 1
这个需要怎么解决呢?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.