andrec10002 / httpsqs Goto Github PK

Attention: Unable to open the database.

What steps will reproduce the problem?
1. 用ab压力测试httpsqs

What is the expected output? What do you see instead?


What version of the product are you using? On what operating system?
httpsqs 1.3
libevent-1.4.13-stable
tokyocabinet-1.4.45
OS:ubuntu 10.10

Please provide any additional information below.
在ab一直put，get压力测试下，我发现硬盘数据一直增大，内存
使用一直增大，如何操作或者设置能让它们降下来？

附件是内存和硬盘的部分变化。操作就是用ab 
put和get压力测试。

Purpose of code changes on this branch:


When reviewing my code changes, please focus on:


After the review, I'll merge this branch into:
/trunk

Hi, I want to know that which are the main configuration which cause the good 
performance. I am also interested to know about the acknowledge of message 
consumed.

What steps will reproduce the problem?
1. $./httpsqs -x ~/
2. the httpsqs_settings_dataname=/home/foo//httpsqs.db


What is the expected output? What do you see instead?
httpsqs_settings_dataname=/home/foo/httpsqs.db
同时判断输入目录合法性

What version of the product are you using? On what operating system?
1.7

How do I know what the current queue. if have another key to get this messaage?

What steps will reproduce the problem?
1. install nginx-echo-module , add "echo_location_async 
"http://192.168.1.12:1218/?name=log&opt=put&data=test";" into 
nginx.conf("location" section)
2. start httpsqs
3. access the url (hosted by nginx) , then httpsqs crashed


What version of the product are you using? On what operating system?
v1.4(CentOS 5.1 x86_64)

Please provide any additional information below.
log in /var/log/messages:
Aug 20 16:45:50 node1 kernel: httpsqs[20496]: segfault at ffffffff00000010 rip 
00002aaaaaad033b rsp 00007fffd41a4880 error 4
Aug 20 17:53:33 node1 kernel: httpsqs[20781]: segfault at 0000000000000021 rip 
00002aaaaaacee7f rsp 00007fff688b7f90 error 6

文档中都是对单个队列进行操作的方法, 
有没有对全部队列进行操作的相关接口呢?

输入：SS%EE
编码后为：SS%25EE
输出：SS�

输入：SS%88EE
编码后为：SS%2588EE
输出：SS�EE

当出现下面的情况会导致数据直接被截断：
输入：321%00123
编码后为：321%2500123
输出：321（后面的字符被截断了）

临时解决办法：
添加数据进队列前如果发现数据中含有%，则先将%替换为%25再
添加进队列，取出来的就是正确的数据了，似乎是因为如果��
�现了%25，会先将%25解析为%，然后又解析了一次，才导致了这
个问题。

具体的代码没仔细研究（不熟悉C），也许是 char *urldecode(char 
*input_str) 函数导致的吧。

line 76:
$body = @fread($httpsqs_socket, $len);

        应该改为如下方式

       if ($len < 0)
        {
            return false;
        }
        $body = '';

        //修复fread大小限制问题
        while (!feof($httpsqs_socket)) {
        $body .= @fread($httpsqs_socket, 1024);
    }


        fclose($httpsqs_socket);
    $result_array["pos"] = (int)$pos_value;
    $result_array["data"] = $body;
        return $result_array;

What steps will reproduce the problem?
1. ab -c1000 -n50000 http://ip:port/?name=x&data=somedata&opt=put
2. 看status发现数据条目数往往不对。
3. 同样，可以做下get的测试，往往也是不对的。

What is the expected output? What do you see instead?


What version of the product are you using? On what operating system?


Please provide any additional information below.

—— 1并发的测试  ———–

队列初始信息：

HTTP Simple Queue Service v1.3.1

——————————

Queue Name: x

Maximum number of queues: 1000000

Put position of queue (1st lap): 0

Get position of queue (1st lap): 0

Number of unread queue: 0

尝试1并发的50000次添加:

~/laoyc/soft/bin/ab -c1 -n50000 
“http://192.168.22.31:1218/?name=x&data=1000000000&opt=put”

全部成功，查看队列状态：

HTTP Simple Queue Service v1.3.1

——————————

Queue Name: x

Maximum number of queues: 1000000

Put position of queue (1st lap): 50000

Get position of queue (1st lap): 0

Number of unread queue: 50000

尝试49999次get:

~/laoyc/soft/bin/ab -c1 -n49999 
“http://192.168.22.31:1218/?name=x&data=1000000000&opt=get”

全部成功，再次查看队列状态：

HTTP Simple Queue Service v1.3.1

——————————

Queue Name: x

Maximum number of queues: 1000000

Put position of queue (1st lap): 50000

Get position of queue (1st lap): 49999

Number of unread queue: 1

正确。

在测试一次：发现正确。

————– 1000 并发的测试  ————-

HTTP Simple Queue Service v1.3.1

——————————

Queue Name: x

Maximum number of queues: 1000000

Put position of queue (1st lap): 0

Get position of queue (1st lap): 0

Number of unread queue: 0

开始测试并发1000放50000次:

~/laoyc/soft/bin/ab -c1000 -n50000 
“http://192.168.22.31:1218/?name=x&data=1000000000&opt=put”

发现结果是：

HTTP Simple Queue Service v1.3.1

——————————

Queue Name: x

Maximum number of queues: 1000000

Put position of queue (1st lap): 50020

Get position of queue (1st lap): 0

Number of unread queue: 50020

看起来并发是有问题，现在不确定是ab还是httpsqs的并发处理��
�还是tc的问题。

所以这个队列目前只能应用在缓存和不重要的任务的情景。

1000并行get49999次：

~/laoyc/soft/bin/ab -c1000 -n49999 
“http://192.168.22.31:1218/?name=x&data=1000000000&opt=get”

HTTP Simple Queue Service v1.3.1

——————————

Queue Name: x

Maximum number of queues: 1000000

Put position of queue (1st lap): 50020

Get position of queue (1st lap): 50020

Number of unread queue: 0

可见，get也是不准确的。

HttpsqsStatus.java line58:
protected static Pattern pattern = Pattern.compile("HTTP Simple Queue Service 
v(.+?)\\s(?:.+?)\\sQueue Name: (.+?)\\sMaximum number of queues: (\\d+)\\sPut 
position of queue \\((\\d+)st lap\\): (\\d+)\\sGet position of queue 
\\((\\d+)st lap\\): (\\d+)\\sNumber of unread queue: (\\d+)");

since the client use (\\d+)st lap to match the first lap, it will cause regex 
matching failed when the unread number goes into lap 2 or latter.

I change the pattern as follows:
protected static Pattern pattern = Pattern.compile("HTTP Simple Queue Service 
v(.+?)\\s(?:.+?)\\sQueue Name: (.+?)\\sMaximum number of queues: (\\d+)\\sPut 
position of queue \\((\\d+)\\w\\w lap\\): (\\d+)\\sGet position of queue 
\\((\\d+)\\w\\w lap\\): (\\d+)\\sNumber of unread queue: (\\d+)");

hope this would help

How do I know what the current queue2. 

查看指定队列位置点的内容时，返回false。经检查发现变量写
错。

       if(pid = fork())
                exit(0);
        else if(pid < 0)
                exit(1);
pid=fork() 
当返回值是-1时条件也是成立的，虽然能正常工作。但不够严
谨。

建议这样处理：

if((pid = fork()) >0)
                exit(0);
        else if(pid < 0)
                exit(1);

What steps will reproduce the problem?
1. Create a queue of length L.
2. View the contents of the specified queue pos < 1.
3. View the contents of the specified queue pos > L.
4. View the contents of the specified queue pos (pos is not numerical).
5. View the contents of the specified queue pos >=1 && pos <=L.

What is the expected output? What do you see instead?

What I saw:
HTTPSQS_ERROR when pos < 1 (Expected)
BLANK when pos > L [(Expected): HTTPSQS_ERROR]
HTTPSQS_ERROR when pos is not numerical (Expected)
Get the right value when pos >=1 && pos <=L (Expected)

What version of the product are you using? On what operating system?

HTTP Simple Queue Service v1.1

Please provide any additional information below.

这份http://blog.s135.com/httpsqs/文档里的
"4、HTTPSQS 服务器使用文档" 小节

请麻烦更新补充以下说明：

如何创建不同的队列？

httpsqs -d -p 1218 -x /usr/local/project/httpsqs/q2
httpsqs -d -p 1218 -x /usr/local/project/httpsqs/q1

q1和q2为队列名字，对应于客户端 
http://x.x.x.x:1218/?name=q2&opt=put..
里的name参数。

方便后来人。

# -*- coding: utf-8 -*-
#!/usr/bin/python
import urllib2
import urllib
import sys
def httpsqs_get(host, port,name,charset='utf-8'):
    data={'charset':charset,'name' : name, 'opt' : 'get'}
    url='http://'+host+':'+port+'/?'
    data=urllib.urlencode(data)
    getString=url+data
    req = urllib2.Request(getString)
    fd = urllib2.urlopen(req)
    str=fd.read()
    if str=='HTTPSQS_GET_END':
        print 'HTTPSQS_GET_END'
    else:
        return str
def httpsqs_put(host, port,name,data,charset='utf-8'):
    data={'charset':charset,'name' : name, 'opt' : 'put','data' : data}
    url='http://'+host+':'+port+'/?'
    data=urllib.urlencode(data)
    getString=url+data
    req = urllib2.Request(getString)
    fd = urllib2.urlopen(req)
    str=fd.read()
    if str=='HTTPSQS_PUT_OK':
        return 1
    else:
        return 0
#下面是调用方式
data=sfsdfasafasdfasdfadfasdfadsfsdf
srr=httpsqs_put('127.0.0.1','1218','test',data)
sss=httpsqs_get('127.0.0.1','1218','test')

保存为py文件即可

如题？

当队列信息出现 2nd 
lap时候，即队列已满，重新从队列头开始覆盖。httpsqs4j 
会出现无法正常创建HttpsqsStatus对象的问题。

例如：
HTTP Simple Queue Service v1.6
------------------------------
Queue Name: xoyo
Maximum number of queues: 1000000
Put position of queue (2nd lap): 4562
Get position of queue (1st lap): 900045
Number of unread queue: 104517


原因是以前是2st，现在是2nd，所以改了下正则，就可以正常��
�用了。另附上patch

What steps will reproduce the problem?
1. Load the queue up to maxlen + 1
2. Queue resets to 0
3. You lose maxlen messages

What is the expected output? What do you see instead?


What version of the product are you using? On what operating system?


Please provide any additional information below.

wget http://httpsqs.googlecode.com/files/tokyocabinet-1.4.47.tar.gz
tar zxvf tokyocabinet-1.4.47.tar.gz
cd tokyocabinet-1.4.47/
./configure --prefix=/usr/local/tokyocabinet-1.4.47/
执行到上一句时，出现以下错误

configure: error: bzlib.h is required

说是需要安装啥啥来着，可是我内网服务器不能上网，网络��
�供的都是YUM安装呀。

更好发挥httpsqs 
keepalive特性，减少请求连接数，将socket链接修改为长连接，��
�失去链接时自动重连

opt=status的时候, 没有Pos值

What steps will reproduce the problem?
1. name=your_queue_name&opt=maxqueue&num=10
2. 5 puts => 0 get(or 1 get) => 6 puts.

What is the expected output? What do you see instead?
#expected output: return HTTPSQS_PUT_END while performing the 11th(or 12th) put
# what i saw: return HTTPSQS_PUT_OK while performing the 11th(or 12th) put

What version of the product are you using? On what operating system?
httpsqs version: httpsqs-1.3
OS: x86_64 GNU/Linux

Please provide any additional information below.

一直没明白 tchdb,tcbdb,tcfdb,tctdb,tcadb!

看了您的源码:
TCBDB *httpsqs_db_tcbdb;

如果说, 我使用的是 TT, 我怎么选择使用哪种结构呢?

What steps will reproduce the problem?
1.
2.
3.

What is the expected output? What do you see instead?


What version of the product are you using? On what operating system?


Please provide any additional information below.

假设在1个Web请求中，有100次入队，HTTP的话需要连接-关闭100��
�。TCP只需连接-关闭1次，发送100次数据即可

一直没明白 tchdb,tcbdb,tcfdb,tctdb,tcadb!

看了您的源码:
TCBDB *httpsqs_db_tcbdb;

如果说, 我使用的是 TT, 我怎么选择使用哪种结构呢?

now this queue only stores text data,
can it store the object data ?

提交了一个PR，移植httpsqs到FreeBSD上：
http://www.freebsd.org/cgi/query-pr.cgi?pr=156595

问题：

1. 编译期的一个warning，wait函数被隐式定义。需要加入头文件
#include <sys/wait.h>
根据编译gcc的参数不同，有些gcc会自己include某些头文件，可�
��看不到这个warning。

2. 使用了 procfs，只是为了获取可执行文件的路径。
BSD* 下 procfs 
是不推荐使用的（默认也不挂载），但也没有简单可靠地获��
�进程路径的方法。我在移植过程中把这个路径用编译时port系
统指定的安装路径替换了。

3. 不支持读配置文件。
这个给编写 rc.d 
脚本（或者init.d脚本）带来了困难。因为这些脚本只控制daemo
n的启动，终止等状态，daemon的功能性配置应该与之分离。

4. 以非daemon方式运行时，键盘Ctrl-c不会kill掉子进程。
考虑一下在非daemon模式下响应SIGINT信号（更简单的做法是绑��
�atexit函数）。

What steps will reproduce the problem?
1. create a queue
2. set the maxqueue a little small, say: 10
3. just enqueue msgs, never dequeue, the 11st msg will overlap the 1st msg

Please provide any additional information below.
原因在于第303行判断队列已满时遗漏一种情况： （0 == 
queue_get_value && queue_put_value > maxqueue_num）。

修改为：
protected static Pattern pattern = Pattern.compile("HTTP Simple Queue Service 
v(.+?)\\s(?:.+?)\\sQueue Name: (.+?)\\sMaximum number of queues: (\\d+)\\sPut 
position of queue \\((\\d+)\\w+ lap\\): (\\d+)\\sGet position of queue 
\\((\\d+)\\w+ lap\\): (\\d+)\\sNumber of unread queue: (\\d+)");

主要当put第二轮的时候\\d+nd 而原来在正则还是匹配 
\\d+st改成 (\\d+)\\w+：
Queue Name: low
Maximum number of queues: 1000000
Put position of queue (2nd lap): 29068
Get position of queue (1st lap): 997815
Number of unread queue: 31253