哈,又一款超级简单的队列(MQ)实现方案来了~

开源的消息队列已经很多了,但大部分很重,实际环境下,很多可能只是使用到了一点功能而已,杀鸡使用牛刀,着实有些浪费了。很多时候,我们只想要一片绿叶,但它们给了我们整个的春天,很难消化。本着DIR精神, 也琢磨了一个超级简单的队列实现。

说是超级简单,嗯,绝对是超级简单,队列的存储采用Redis进行持久化存储,采用Netty提供HTTP方式的队列的出/入。Redis的客户端采用的Jedis。然后呢,然后就没了啊。

一。Redis

Redis内置订阅发布模型(Publish/Subscribe),其缺陷是,不存储,一旦订阅者断线,将无法接收到消息,再次连接上之后,在断线期间发布者发布的消息都是无法获取到的。只能采用list数组实现,采用rpush/lpop组合命令来实现先进先出的队列模型,当然redis也提供了阻塞版本的blpush/brpush/blpop/brpop等,就看我们实际环境下如何使用了。

JAVA客户端使用Jedis,提供接口也很丰富。但要注意的是,需要使用连接池,否则在大数量的情况下,有可能jedis的连接不够用。

private static JedisPool pool;
 static {
  ResourceBundle bundle = ResourceBundle.getBundle("redis");
  if (bundle == null) {
   throw new IllegalArgumentException(
     "cannot find the SignVerProp.properties");
  }
  JedisPoolConfig config = new JedisPoolConfig();
  config.setMaxActive(Integer.valueOf(bundle
    .getString("redis.pool.maxActive")));
  config.setMaxIdle(Integer.valueOf(bundle
    .getString("redis.pool.maxIdle")));
  config.setMaxWait(Integer.valueOf(bundle
    .getString("redis.pool.maxWait")));
  pool = new JedisPool(config, bundle.getString("redis.server"),
    Integer.valueOf(bundle.getString("redis.port")));
 }

二。Netty

很成熟的NIO框架,用它来提供HTTP方式的队列的出入。嗯,目前只提供HTTP方式的入队列,出队列等。
HTTP形式为:http://服务器名称:端口/队列操作原语/队列名称?msg=消息内容 
队列操作原语,只有get和put;get为出队列,put为入队列。
返回为json: {s:0, m:'错误消息/消息内容'} 
s为success的缩写,值为0,意味着false,1对应成功。 m为message的缩写,错误消息/消息内容,添加消息时,会返回插入消息对应的数目 默认采用UTF-8。

入队列:http://localhost:8080/put/demo?msg=消息内容 
出队列:http://localhost:8080/get/demo HTTP方式,特别适合局域网之间,消息数据的推送。

 

三。入口解读

采用了QueueDaemon封装了netty代码: 调用很简单,程序启动的入口: 而HTTP方式队列请求处理器为HttpRequestHandler:

四。队列处理器

这个很简单,直接采用Jedis客户端即可,这里一直占用一个连接,不释放。

五。ab压力测试

本机配置:Pentium(R) Dual-Core CPU E5200 2.50GHz,2.00 GB内存,Windows XP系统,redis-2.4.5-win32 32版本(非linux版本),都在一台机器上运行。 插入 512 bytes 文本消息队列: 1781.24 requests/sec

D:\Apache2.2\bin>ab -c 10 -n 100000 "http://localhost:8080/put/demo?msg=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
Benchmarking localhost (be patient)
Completed 10000 requests
Completed 20000 requests
Completed 30000 requests
Completed 40000 requests
Completed 50000 requests
Completed 60000 requests
Completed 70000 requests
Completed 80000 requests
Completed 90000 requests
Completed 100000 requests
Finished 100000 requests
Server Software:
Server Hostname:        localhost
Server Port:            8080
Document Path:          /put/demo?msg=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
Document Length:        12 bytes
Concurrency Level:      10
Time taken for tests:   56.141 seconds
Complete requests:      100000
Failed requests:        99991
   (Connect: 0, Receive: 0, Length: 99991, Exceptions: 0)
Write errors:           0
Total transferred:      8188895 bytes
HTML transferred:       1588895 bytes
Requests per second:    1781.24 [#/sec] (mean)
Time per request:       5.614 [ms] (mean)
Time per request:       0.561 [ms] (mean, across all concurrent requests)
Transfer rate:          142.45 [Kbytes/sec] received
Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   1.7      0      16
Processing:     0    5  10.7      0     781
Waiting:        0    5  10.4      0     766
Total:          0    6  10.8      0     781
Percentage of the requests served within a certain time (ms)
  50%      0
  66%     16
  75%     16
  80%     16
  90%     16
  95%     16
  98%     16
  99%     16
 100%    781 (longest request)

插入 512 bytes 文本消息队列(添加Keep-Alive支持): 1875.18 requests/sec

D:\Apache2.2\bin>ab -k -c 10 -n 100000 "http://localhost:8080/put/demo?msg=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
Benchmarking localhost (be patient)
Completed 10000 requests
Completed 20000 requests
Completed 30000 requests
Completed 40000 requests
Completed 50000 requests
Completed 60000 requests
Completed 70000 requests
Completed 80000 requests
Completed 90000 requests
Completed 100000 requests
Finished 100000 requests
Server Software:
Server Hostname:        localhost
Server Port:            8080
Document Path:          /put/demo?msg=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
Document Length:        17 bytes
Concurrency Level:      10
Time taken for tests:   53.328 seconds
Complete requests:      100000
Failed requests:        0
Write errors:           0
Keep-Alive requests:    0
Total transferred:      8300000 bytes
HTML transferred:       1700000 bytes
Requests per second:    1875.18 [#/sec] (mean)
Time per request:       5.333 [ms] (mean)
Time per request:       0.533 [ms] (mean, across all concurrent requests)
Transfer rate:          151.99 [Kbytes/sec] received
Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   1.5      0      16
Processing:     0    5   7.8      0     203
Waiting:        0    5   7.8      0     203
Total:          0    5   7.9      0     203
Percentage of the requests served within a certain time (ms)
  50%      0
  66%      0
  75%     16
  80%     16
  90%     16
  95%     16
  98%     16
  99%     16
 100%    203 (longest request)

获取 512 bytes 消息(With Keep-Alive): 1875.73 requests/sec

D:\Apache2.2\bin>ab -k -c 10 -n 100000 "http://localhost:8080/get/demo"
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
Benchmarking localhost (be patient)
Completed 10000 requests
Completed 20000 requests
Completed 30000 requests
Completed 40000 requests
Completed 50000 requests
Completed 60000 requests
Completed 70000 requests
Completed 80000 requests
Completed 90000 requests
Completed 100000 requests
Finished 100000 requests
Server Software:
Server Hostname:        localhost
Server Port:            8080
Document Path:          /get/demo
Document Length:        523 bytes
Concurrency Level:      10
Time taken for tests:   53.313 seconds
Complete requests:      100000
Failed requests:        0
Write errors:           0
Keep-Alive requests:    0
Total transferred:      58900000 bytes
HTML transferred:       52300000 bytes
Requests per second:    1875.73 [#/sec] (mean)
Time per request:       5.331 [ms] (mean)
Time per request:       0.533 [ms] (mean, across all concurrent requests)
Transfer rate:          1078.91 [Kbytes/sec] received
Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   1.9      0      16
Processing:     0    5   7.5      0      94
Waiting:        0    4   6.9      0      94
Total:          0    5   7.6      0      94
Percentage of the requests served within a certain time (ms)
  50%      0
  66%      0
  75%     16
  80%     16
  90%     16
  95%     16
  98%     16
  99%     16
 100%     94 (longest request)

其它问题

  1. 暂时对安全/授权没有支持,这个其实很容易实现
  2. 队列处理器很简单,直接使用jedis即可
  3. 对队列数据进行分片(sharding),写一个QueueService实现即可
  4. 对Redis的内存持久化不放心,采用diskstore存储模式好了,在Redis中配置,不涉及到程序
  5. 对队列分布式存储,写一个QueueService实现,其它可不用变化
  6. 不适合严格意义上的订阅/发布模型,这里适合多个发布者/单个订阅者环境
  7. HTTP请求返回内容为json格式,xml格式,暂时不需要
  8. 局域网环境下,系统之间进行消息的推送/通知,

项目下载地址:http://code.google.com/p/nettyqueue/ 参考资料:

  1. httpsqs

http://www.blogjava.net/yongboy/archive/2012/03/20/372308.html

 

时间: 2016-06-08

哈,又一款超级简单的队列(MQ)实现方案来了~的相关文章

5款超级简单的IP流量监控工具

对于系统管理员来说,最重要的任务之一就是密切关注网络.当有糟糕的事情发生时,起因可能就是某个卑鄙的人.有可能是黑客,被攻破的系统,或者是出故障的硬件,关键是要找出问题所在. 为此,你需要合适的工具.首先你可能需要用到的工具是IP流量监控工具.所幸的是,如今有很多工具可以为你服务.糟糕的是--其中一些工具还是相当复杂的.这就是为什么我要找出最简单的IP流量监控工具,并且其中5款罗列出来,为你监控网络提供帮助. 1.Microsoft Network Monitor Microsoft Networ

一款超级简单php图片水印代码

在php中给图片增加水印有imagecreatefromjpeg imagecreatefrompng imagecopymerge imagejpeg就成了,只要你设置原图与水印图片就成了,下面看实例. */  代码如下 复制代码 header("content-type: image/jpeg"); $filename='temp/www.111cn.net/111cn.net.jpg'; $im=imagecreatefromjpeg($filename); $s=imagecr

91-关于背景图片(超级简单).........

问题描述 关于背景图片(超级简单)......... 问题描述:有一张图片1300 x 200px的图片用做网页背景,怎么样让图片铺满网页?而不是这样呢? 补充:比方网页像素是1366px 要让1300图片的长度变成1366px,而不是自动填,那样图片就不美了 解决方案 <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml

推荐给 JavaScript 开发者十款超级有用的工具

近年来,JavaScript 越来越受到重视,早已不单单只是写写网页特效的脚本语言了,如今广泛用于网站的前端交互.值得一提的是,当下 Web 开发者们追捧的 Node.js 使得 JavaScript能够在更多环境下运行,甚至是服务器端,使得这门语言再次受到大家的关注.今天,本文向 JavaScript 开发者推荐十款超级有用的工具. YepNope YepNope 是一个异步的条件加载框架,速度超快,只为用户加载需要的脚本.使用非常简单,非常有用!点击访问: http://yepnopejs.

推荐给 JavaScript 开发者十款超级有用的工具来提高客户体验

近年来,JavaScript 越来越受到重视,早已不单单只是写写网页特效的脚本语言了,如今广泛用于网站的前端交互.值得一提的是,当下 Web 开发者们追捧的 Node.js 使得 JavaScript能够在更多环境下运行,甚至是服务器端,使得这门语言再次受到大家的关注.今天,本文向 JavaScript 开发者推荐十款超级有用的工具. YepNope  YepNope 是一个异步的条件加载框架,速度超快,只为用户加载需要的脚本.使用非常简单,非常有用! 点击访问: http://yepnopej

网页CSS技巧:闭合浮动元素超级简单的方法

css|浮动|技巧|网页 关于闭合浮动元素(clearing float)的方法现在已经很多了,你还不了解的话去old9的blog看看,有一篇闭合浮动元素. 这些方法我最喜欢就是 使用:after 伪类动态的嵌入一个用于清除浮动的元素,可惜代码量太大了,看着不够简洁.现在我看到有个方法超级简单.赶紧介绍一下. 原理是这样的,外围元素之所以不能很好的延伸,问题出在了overflow上,因为overflow不可见.见W3C的解释:In addition, if the element has any

PHP实例:实现超级简单的MVC结构

下面是一个超级简单的MVC结构实现,甚至连数据源都用了一个内置的固定数组,虽然简单,但其实众多的PHP Framework核心实现的思想应该和这个是差不多的 只不过一些framework提供了更多的方便开发者使用的工具,我也想自己来实现一个PHP的 框架,目前正在着手策划中,也希望自己能够从框架的开发中学习到更多的PHP设计思想和方法. Controller.php include 'Model.php'; include 'View.php'; class Controller {     p

超级简单

当我在页面中使用ViewState ,通常是用一个属性表示,例如: private int ViewState_UserID { get { return (int) ViewState["UserId"]; } set { ViewState["UserId"] = value; } } 写这样一组代码感觉比较麻烦,如果能像下面这样简单地使用就好了. [ViewStateProperty("UserID")] protected int Vie

include 指令-超级简单的问题 求帮助 求帮助

问题描述 超级简单的问题 求帮助 求帮助 谢谢大神们 解决方案 你的问题是什么呢 看下你的html还有编码 解决方案二: 解决方案三: 代码看不全,应该是jsp页面有问题,一点一点删代码.你就能看出来是哪行报错了 解决方案四: 编程是一种美德,是促使一个人不断向上发展的一种原动力.