We left some binary and dynamic library files in the tools/ lib/. We are considering removing them.

codis安装成功 ，但是不知道如何支持 pika

您好：
目前使用pika中发现了一个问题。正常情况下一个pika节点的客户端连接数在270-300之间，但连接数有时候会有波动，到一个350-400连接数的峰值，监控如图。

在这些峰值的时间点，都会出现检查节点状态时ping无响应，读取错误等故障。查看日志未发现明显问题，请问有没有什么解决思路，以及pika本身时候有什么限制，谢谢。

当前服务状态：

127.0.0.1:6379> info

Server

pika_version:2.1.0
os:Linux 2.6.32-431.el6.centos.plus.x86_64 x86_64
arch_bits:64
process_id:3150
tcp_port:6379
thread_num:8
sync_thread_num:4
uptime_in_seconds:23350
uptime_in_days:1
config_file:/data/pika/conf/pika.conf

Data

db_size:30618M
compression:snappy
used_memory:32105758360
db_memtable_usage:404912908Bytes
db_tablereader_usage:338472679Bytes

Log

log_size:9536M
safety_purge:write2file3122
expire_logs_days:7
expire_logs_nums:10
write2file:3132

Clients

connected_clients:285

Stats

total_connections_received:27792
instantaneous_ops_per_sec:2280
total_commands_processed:58205799
is_bgsaving:No, , 0
is_scaning_keyspace:No
is_compact:No

Replication(MASTER)

role:master
connected_slaves:0

Keyspace

Time:2016-08-18 13:01:52

kv keys:1934565
hash keys:0
list keys:0
zset keys:2253399
set keys:0

https://github.com/Qihoo360/huststore

pika和huststore相比，都是360开发的， 区别是什么？

看了一下， .gitmodules 里面配置的除了nemo 别的都是 [email protected]/xxx
这样更新就会出现如下错误

建议全部改为 https 的

Would you like to wrap any pointer data members with the template class "std::unique_ptr"?

Update candidates:

• PikaBinlogSenderThread
• PikaServer

Would you like to add more error handling for return values from calls of a function like "pthread_rwlock_init"?

碰到写数据时，出现慢查询 —— cost > 100ms。

1. 部署的结构 ：单台机器上一主两从，多台机器交叉主从关系。机器内存 128G、磁盘ssd。
2. 数据情况： value 大小在 10 ~ 20k 左右， 现在每个实例存储了 80G 数据
3. 写并发情况：每个实例 200 W/s (基准测试时可以支持 3.6w W/s)。每个实例，每次平均更新 40w KV。
4. 内存使用情况： 从节点 (VIRT)2046m (RES)543m

一些想法：

1. 限制主从同步速度来降低从节点的写入速度。但是写入压力不大，粗略估计 15M/s 就到顶了。
2. 增大缓存，避免出现慢查询。现在配置 max-cache-files : 5000，默认应该最多可以90G内存。

以上两种办法，自己评估了下好像都问题的根源。对pika了解的不够，希望能给些建议。

现在的配置文件：

# Pika port
port : 30001
# Thread Number
thread-num : 10
# Sync Thread Number
sync-thread-num : 6
# Item count of sync thread queue
sync-buffer-size : 10
# Pika log path
log-path : /usr/local/logs/pika/30001
# Pika glog level: only INFO and ERROR
loglevel : info
# Pika db path
db-path : /data/pika/db/30001
# Pika write-buffer-size
write-buffer-size : 268435456
# Pika timeout
timeout : 60
# Requirepass
requirepass :
# Userpass
userpass :
# User Blacklist
userblacklist :
# Dump Prefix
dump-prefix :
# daemonize  [yes | no]
daemonize : yes
# Dump Path
dump-path : /usr/local/data/pika/dump/30001
# pidfile Path
pidfile : /var/run/pika/30001.pid
# Max Connection
maxclients : 20000
# the per file size of sst to compact, defalut is 2M
target-file-size-base : 20971520
# Expire-logs-days
expire-logs-days : 7
# Expire-logs-nums
expire-logs-nums : 10
# Root-connection-num
root-connection-num : 2
# Slowlog-log-slower-than
slowlog-log-slower-than : 10000
# slave-read-only(yes/no, 1/0)
slave-read-only : 0
# Pika db sync path
db-sync-path : /usr/local/data/pika/dbsync/30001
# db sync speed(MB) max is set to 125MB, min is set to 0, and if below 0 or above 125, the value will be adjust to 125
db-sync-speed : -1
# network interface
 network-interface : eth0
# replication
slaveof : 

###################
## Critical Settings
###################
# binlog file size: default is 100M,  limited in [1K, 2G]
binlog-file-size : 104857600
# Compression
compression : snappy
# max-background-flushes: default is 1, limited in [1, 4]
max-background-flushes : 1
# max-background-compactions: default is 1, limited in [1, 4]
max-background-compactions : 1
# max-cache-files default is 5000
max-cache-files : 5000

我测试了一下快速安装和编译安装，发现安装完成后大小不一样。这两种安装在我们这边有什么区别？是否都可以直接在线上使用？还有就是在我插入数据后，快速安装的数据目录并没有占用多大空间，编译安装的数据目录每个目录都会生成很大的日志文件，有2.7G，这是正常的吗？

我用redis-py连接pika，当zset的member大概超过2M后就会报错，服务器会RST客户端的TCP连接

我pika主库需要重启，但重启后发现从库报错Could not purge 1, since it is already be used， 这个是什么原因

pika系统对内存分配管理有没有做过控制？

一个测试业务在使用pika，发现随着测试使用，进程实际占用内存一直增长（目前约30G），系统会出现内存不足现象；

Data

db_size:59566M
compression:snappy
used_memory:62459617254
db_memtable_usage:151748356Bytes
db_tablereader_usage:156153589Bytes

Pika write-buffer-size

write-buffer-size : 268435456

max-cache-files default is 5000

max-cache-files : 500

分配60多M的内存好多
00007f9f3c000000 65128 26292 26292 rw--- [ anon ]
00007f9f3ff9a000 408 0 0 ----- [ anon ]
00007f9f40000000 49156 100 100 rw--- [ anon ]
00007f9f43001000 16380 0 0 ----- [ anon ]
00007f9f44000000 63492 128 128 rw--- [ anon ]
00007f9f47e01000 2044 0 0 ----- [ anon ]
00007f9f48000000 62672 52440 52440 rw--- [ anon ]
00007f9f4bd34000 2864 0 0 ----- [ anon ]
00007f9f4c000000 63492 220 220 rw--- [ anon ]
00007f9f4fe01000 2044 0 0 ----- [ anon ]
00007f9f50000000 63492 128 128 rw--- [ anon ]
00007f9f53e01000 2044 0 0 ----- [ anon ]
00007f9f58000000 63492 168 168 rw--- [ anon ]

pfadd pfcount pfmerge 这些将来会支持么

环境：Ubuntu 14.04
分支：master，tag/v2.0.2

编译pika_admin.cc时，执行到

g++ -O2 -g -pipe -fPIC -W -DNDEBUG -Wwrite-strings -Wpointer-arith -Wreorder -Wswitch -Wsign-promo -Wredundant-decls -Wformat -Wall -Wno-unused-parameter -D_GNU_SOURCE -D__STDC_FORMAT_MACROS -std=c++11 -gdwarf-2 -Wno-redundant-decls -c src/pika_admin.cc -o src/pika_admin.o -I./include/ -I./src/ -I./third/glog/src/ -I./third/nemo/output/include/ -I./third/slash/output/include/ -I./third/pink/output/include/

报错

In file included from ./third/slash/output/include/base_conf.h:6:0,
                 from ./include/pika_conf.h:8,
                 from src/pika_admin.cc:2:
./third/slash/output/include/slash_define.h:6:16: error: expected identifier before '#'
 #define NUMBER '#'

经检查，是third/slash/output/include/slash_define.h的include顺序早于/usr/include/google/protobuf，导致protobuf使用了其中宏NUMBER的定义，从而造成语法错误。

We can remove Nemo's lock by compiling flags when used for Pika.

I notice that #48 was merged, but README is still not updated?

maintain a nagios, cacti, zabbix monitoring template for pika

我看pika的设计，似乎所有类型都支持ttl。请问下，对hash设置ttl该如何使用？

如题

pika的写操作可靠性怎么样？是同步落盘吗？还是只保证写入缓存？

请问pika现在支持 Unix domain socket吗？

我在配置文件里似乎没有看到这项

谢谢

请问pika支持go语言嘛？
我们现在在做一个区块链项目，数据会一直扩张，请问pika对集群及其可扩展性的支持如何？ 如果数据量达到1pb，还能有效支持嘛？

I noticed

slash::RecordMutex mutex_record_;

in PikaServer structure, how does pika lock every key when parallel write?

And every kind of data stucture use the same lock， for example set/zset/kv/list/hash , do these data structure write operation use the same kind of lock mechanism?

之前使用redis时，内存已经30G+，所以准备改用pika。但是对数据库进行操作时，有时会用keys命令，使用keys命令时，得到如下错误：

_File "/usr/lib/python2.6/site-packages/redis/client.py", line 568, in keys

return self.execute_command('KEYS', pattern)

File "/usr/lib/python2.6/site-packages/redis/client.py", line 330, in execute_command
*_options
File "/usr/lib/python2.6/site-packages/redis/client.py", line 312, in _execute_command
return self.parse_response(command_name, *_options)
File "/usr/lib/python2.6/site-packages/redis/client.py", line 390, in parse_response
response = self._parse_response(command_name, catch_errors)
File "/usr/lib/python2.6/site-packages/redis/client.py", line 349, in parse_response
raise ResponseError(response)
redis.exceptions.ResponseError: buf is too large

已经试过修改源码中REDIS_MAX_MESSAGE的值，可是依旧报相同的错误。
也试过用scan命令进行扫库，但是效率比redis的scan差很多，不符合使用的要求

I expect that exception handling is usually supported by a C++ program. I wonder why your function "main" does not contain corresponding try and catch instructions so far.

How do you think about recommendations by Matthew Wilson in an article?

Would you like to adjust the implementation if you consider effects for uncaught/unhandled exceptions like they are described by Danny Kalev?

做过快照备份的情况，怎么进行数据恢复？

1、master端需要密码认证时，在slave端使用slaveof命令，无法连接到master。 请问如何在master有密码的情况下，slave端能成功连接？
2、在slave端使用slaveof命令时，偶尔会出现以下错误，无法连接到master：
I0811 17:52:15.360584 31455 pika_admin.cc:118] Trysync, Slave ip_port: xxx.xxx.xxx.xxx:9221 filenum: 0 pro_offset: 0 I0811 17:52:15.361500 31455 pika_admin.cc:130] Trysync, dont FindSlave, so AddBinlogSender rsync: read error: Connection reset by peer (104) rsync error: error in rsync protocol data stream (code 12) at io.c(759) [sender=3.0.6] W0811 17:52:15.366693 31487 pika_server.cc:577] rsync send file failed! From: ./dump/20160811/list/000025.sst, To: list/000025.sst, At: xxx.xxx.xxx.xxx:12221, Error: 3072 rsync: read error: Connection reset by peer (104) rsync error: error in rsync protocol data stream (code 12) at io.c(759) [sender=3.0.6] rsync: read error: Connection reset by peer (104) rsync error: error in rsync protocol data stream (code 12) at io.c(759) [sender=3.0.6] rsync: read error: Connection reset by peer (104) rsync error: error in rsync protocol data stream (code 12) at io.c(759) [sender=3.0.6] rsync: read error: Connection reset by peer (104) rsync error: error in rsync protocol data stream (code 12) at io.c(759) [sender=3.0.6] rsync: read error: Connection reset by peer (104) rsync error: error in rsync protocol data stream (code 12) at io.c(759) [sender=3.0.6]

出现的概率很小，但是出现以后，即使执行重启服务、清库等操作，主从也无法连接成功。

I suggest to reuse a higher level build system than your current small make file so that powerful checks for software features will become easier.

• CMake
• Autotools

Pika timeout

timeout : 60

这个timeout选项如何设置才能，pika服务端不关闭不活动的客户端。
测试过设置 0 或 -1 不起作用。

Since many pika user has gave feedback that if their network configure don't has eth0, pika can't run well

Some times we need operations like CAS(compare and set), lua script can make it atomic and also get good performance by reducing number of times of remote call.

write2file这个日志文件主要用途是做什么的，如果太多了，可不可以直接手动删除？还有就是使用默认配置最大数量10个的时候，感觉没起作用，日志文件数量一直在增加

If we have a machine with a relatively large memory, and want pika to make full use of it to improve efficiency, how can we adjust the config file? (It seems that "write-buffer-size" is related to this problem, but pika seems to store "write-buffer-size" in an integer variable, which is at most 2^31-1 (roughly 2GB), and not big enough.)

I would like to point out that an identifier like "__PIKA_CONF_H__" does eventually not fit to the expected naming convention of the C++ language standard.
Would you like to adjust your selection for unique names?

127.0.0.1:29221> slaveof 192.168.6.158 9221
OK
127.0.0.1:29221> info

Replication(SLAVE)

role:slave
master_host:192.168.6.158
master_port:9221
master_link_status:# down
slave_read_only:1
127.0.0.1:29221> info

Replication(MASTER)

role:single
connected_slaves:0

I have try many time .
Just one way can be OK:

127.0.0.1:29221> slaveof 127.0.0.1 9221
OK
127.0.0.1:29221> info

Replication(SLAVE)

role:slave
master_host:0.0.0.0
master_port:9221
master_link_status:up
slave_read_only:1

but statistic is error:
kv keys:0

We have supported codis to scale out pika. Nevertheless, we think the proxy solution is not the perfect solution, and cluster is the future. We want add the support for the redis cluster protocol to support cluster solution.

环境
客户端：php 7
pika_version:2.1.4
os:Linux 2.6.32-573.26.1.el6.x86_64 x86_64

在cli下单进程插入100,000数据到list队列，lpush入rpop出，经测试顺序一致，速度也很快。

然而在web环境下，并发100时，发现队列数据有丢失。跟踪lpush确实插入成功返回值为1，但rpop时会间隔的漏掉1条、3条，不规律，总体情况是100条丢失45条左右。

不知道有么有人遇到过。

后来直接切换到redis的端口，数据正常。因此排除客户端和环境问题。

Since log is the most important tool that DBA used to check the status of pika. So we need to make the log more readable.

at master brench
db_bench db_stress crash

log:

#0 ./db_bench() [0x476442] rocksdb::CompactionJob::ProcessKeyValueCompaction(long*, rocksdb::Iterator*, bool) /home/cltx/tangyi/pika/third/nemo/3rdparty/rocksdb/db/compaction_job.cc:636
#1 ./db_bench() [0x47a1e5] rocksdb::Status::operator=(rocksdb::Status const&) /home/cltx/tangyi/pika/third/nemo/3rdparty/rocksdb/./include/rocksdb/status.h:157
#2 ./db_bench() [0x4946c5] rocksdb::DBImpl::BackgroundCompaction(bool*, rocksdb::JobContext*, rocksdb::LogBuffer*) /home/cltx/tangyi/pika/third/nemo/3rdparty/rocksdb/db/db_impl.cc:2444
#3 ./db_bench() [0x4a011f] rocksdb::DBImpl::BackgroundCallCompaction() /home/cltx/tangyi/pika/third/nemo/3rdparty/rocksdb/db/db_impl.cc:2147
#4 ./db_bench() [0x52fa7f] BGThread /home/cltx/tangyi/pika/third/nemo/3rdparty/rocksdb/util/env_posix.cc:1702

请问有针对pipeline方面的性能测试吗？

目前我们自己测试，如果超过3w的pipeline就会出现一个很大时延，基本服务就不可用了。

Hey, I want to know
1 .Does pika support pipeline mode of redis?
2. If yes, how does pika gurantee the sequence of excution in multi work thread environment?
Thanks.

HI,

我想了解下，如果在pika中使用key-value结构存储数据，对一个key，value对的容量有限制吗？
使用hashmap结构存储时，对单个有容量限制吗？

we can use sync options of rocksdb.

An extra null pointer check is not needed in functions like the following.

• PikaBinlogSenderThread
• PikaServer

不能用配置文件配置主从吗？必须手动执行slaveof命令？

When I query the write_buffer_size using "config get write_buffer_size" in redis-cli, the value returned is different from the value specified in pika.conf. Which one is the real size of the write buffer? Or do they have different units?