Pylons 0.9.7 released
Pylons 0.9.7 is released today.
Pylons is my new favorite web application framework. It was codeigniter, webpy and django.
A lot Cloud buzz today
I have used ubuntu 8.1 ec2 release for a while. I am actually very impressed by the beta version ubuntu cloud. It is a very lean version of ubuntu server. The default configuration run very fast with minimal number of services comparing to CentOS or Redhat. It is fast.
Meanwhile stock market indexes fall to 1997 levels, and Nasdaq took the lead.
Generate ShoreWall blacklist from Spamhaus and DShield
I wrote a bash script to automatically generate Shorewall blacklist from Spamhaus drop list and dshield.org’s block list .
Do not run this script automatically if ssh is the only mean you connect to your server, because you can accidentally blacklist yourself. And you may not run it more often then once per hour due to spamhaus limitation.
#!/bin/sh echo "#ADDRESS/SUBNET PROTOCOL PORT" > /tmp/blacklist wget -q -O - http://feeds.dshield.org/block.txt | awk --posix '/^[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.0\t/ { print $1 "/24";}' >> /tmp/blacklist wget -q -O - http://www.spamhaus.org/drop/drop.lasso | awk --posix '/^[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\// { print $1;}' >> /tmp/blacklist echo "#LAST LINE -- ADD YOUR ENTRIES BEFORE THIS ONE -- DO NOT REMOVE" >> /tmp/blacklist mv /tmp/blacklist /etc/shorewall/blacklist shorewall refresh &>/dev/null
I also use fail2ban to generate dynamic shorewall ban list.
UPDATE: And don’t forget enable blacklist option in /etc/shorewall/shorewall.conf
BLACKLIST_DISPOSITION=DROP
Nginx vs Cherokee
I did a micro benchmark for quick comparison between nginx and cherokee on Linode 540 VPS. Result may differ on a dedicate server. static file(100MB)
Cherokee |
---|
Benchmarking localhost (be patient) Completed 100 requests Completed 200 requests Finished 200 requests Server Software: Cherokee/0.99.0 Server Hostname: localhost Server Port: 80 Document Path: /100mb.test Document Length: 104857600 bytes Concurrency Level: 20 Time taken for tests: 30.696 seconds Complete requests: 200 Failed requests: 0 Write errors: 0 Total transferred: 20971557200 bytes HTML transferred: 20971520000 bytes Requests per second: 6.52 [#/sec] (mean) Time per request: 3069.623 [ms] (mean) Time per request: 153.481 [ms] (mean, across all concurrent requests) Transfer rate: 667184.09 [Kbytes/sec] received Connection Times (ms) min mean[+/-sd] median max Connect: 0 1 0.2 1 1 Processing: 3004 3069 73.9 3048 3228 Waiting: 0 1 0.2 1 2 Total: 3004 3069 73.9 3048 3228 Percentage of the requests served within a certain time (ms) 50% 3048 66% 3068 75% 3071 80% 3191 90% 3228 95% 3228 98% 3228 99% 3228 100% 3228 (longest request) |
Nginx |
Benchmarking localhost (be patient) Completed 100 requests Completed 200 requests Finished 200 requests Server Software: nginx/0.6.35 Server Hostname: localhost Server Port: 80 Document Path: /100mb.test Document Length: 104857600 bytes Concurrency Level: 20 Time taken for tests: 30.543 seconds Complete requests: 200 Failed requests: 0 Write errors: 0 Total transferred: 20971571000 bytes HTML transferred: 20971520000 bytes Requests per second: 6.55 [#/sec] (mean) Time per request: 3054.277 [ms] (mean) Time per request: 152.714 [ms] (mean, across all concurrent requests) Transfer rate: 670536.80 [Kbytes/sec] received Connection Times (ms) min mean[+/-sd] median max Connect: 0 1 0.2 0 1 Processing: 3006 3054 32.6 3047 3106 Waiting: 0 1 0.4 1 3 Total: 3006 3054 32.6 3048 3107 ERROR: The median and mean for the initial connection time are more than twice the standard deviation apart. These results are NOT reliable. Percentage of the requests served within a certain time (ms) 50% 3048 66% 3069 75% 3087 80% 3098 90% 3107 95% 3107 98% 3107 99% 3107 100% 3107 (longest request) |
Small html file (468 bytes)
Nginx |
---|
Server Software: nginx/0.6.35 Server Hostname: localhost Server Port: 80 Document Path: / Document Length: 468 bytes Concurrency Level: 200 Time taken for tests: 0.123 seconds Complete requests: 1000 Failed requests: 0 Write errors: 0 Total transferred: 737100 bytes HTML transferred: 491400 bytes Requests per second: 8150.03 [#/sec] (mean) Time per request: 24.540 [ms] (mean) Time per request: 0.123 [ms] (mean, across all concurrent requests) Transfer rate: 5866.59 [Kbytes/sec] received Connection Times (ms) min mean[+/-sd] median max Connect: 3 8 1.7 9 11 Processing: 6 14 3.3 14 24 Waiting: 3 11 3.3 12 23 Total: 14 22 2.6 22 33 Percentage of the requests served within a certain time (ms) 50% 22 66% 23 75% 23 80% 24 90% 25 95% 26 98% 29 99% 30 100% 33 (longest request) |
Cherokee |
Server Software: Cherokee/0.99.0 Server Hostname: localhost Server Port: 80 Document Path: / Document Length: 468 bytes Concurrency Level: 200 Time taken for tests: 0.139 seconds Complete requests: 1000 Failed requests: 0 Write errors: 0 Total transferred: 732224 bytes HTML transferred: 509184 bytes Requests per second: 7203.83 [#/sec] (mean) Time per request: 27.763 [ms] (mean) Time per request: 0.139 [ms] (mean, across all concurrent requests) Transfer rate: 5151.19 [Kbytes/sec] received Connection Times (ms) min mean[+/-sd] median max Connect: 6 11 2.5 12 16 Processing: 7 14 3.9 14 27 Waiting: 3 10 3.9 9 19 Total: 16 25 3.5 25 33 Percentage of the requests served within a certain time (ms) 50% 25 66% 27 75% 27 80% 28 90% 29 95% 31 98% 32 99% 32 100% 33 (longest request) |
Varnish |
Server Software: ---------- Server Hostname: localhost Server Port: 80 Document Path: / Document Length: 468 bytes Concurrency Level: 200 Time taken for tests: 0.141 seconds Complete requests: 1000 Failed requests: 0 Write errors: 0 Total transferred: 879491 bytes HTML transferred: 528372 bytes Requests per second: 7067.44 [#/sec] (mean) Time per request: 28.299 [ms] (mean) Time per request: 0.141 [ms] (mean, across all concurrent requests) Transfer rate: 6070.07 [Kbytes/sec] received Connection Times (ms) min mean[+/-sd] median max Connect: 6 11 2.5 11 16 Processing: 6 14 3.5 13 24 Waiting: 3 10 3.5 9 21 Total: 17 25 3.4 25 38 Percentage of the requests served within a certain time (ms) 50% 25 66% 26 75% 26 80% 27 90% 29 95% 31 98% 33 99% 34 100% 38 (longest request) |
Small image file (23k bytes)
Nginx |
---|
Server Software: nginx/0.6.35 Server Hostname: localhost Server Port: 80 Document Path: /cherokee-logo.png Document Length: 23619 bytes Concurrency Level: 200 Time taken for tests: 0.139 seconds Complete requests: 1000 Failed requests: 0 Write errors: 0 Total transferred: 24624128 bytes HTML transferred: 24379632 bytes Requests per second: 7171.34 [#/sec] (mean) Time per request: 27.889 [ms] (mean) Time per request: 0.139 [ms] (mean, across all concurrent requests) Transfer rate: 172449.16 [Kbytes/sec] received Connection Times (ms) min mean[+/-sd] median max Connect: 0 5 3.6 4 16 Processing: 0 20 6.1 23 29 Waiting: 0 15 5.7 17 26 Total: 6 25 5.5 26 45 Percentage of the requests served within a certain time (ms) 50% 26 66% 27 75% 27 80% 27 90% 28 95% 35 98% 41 99% 43 100% 45 (longest request) |
Cherokee |
Server Software: Cherokee/0.99.0 Server Hostname: localhost Server Port: 80 Document Path: /cherokee-logo.png Document Length: 23619 bytes Concurrency Level: 200 Time taken for tests: 0.180 seconds Complete requests: 1000 Failed requests: 0 Write errors: 0 Total transferred: 26518338 bytes HTML transferred: 26287947 bytes Requests per second: 5564.92 [#/sec] (mean) Time per request: 35.939 [ms] (mean) Time per request: 0.180 [ms] (mean, across all concurrent requests) Transfer rate: 144113.78 [Kbytes/sec] received Connection Times (ms) min mean[+/-sd] median max Connect: 4 11 3.4 11 18 Processing: 11 21 5.0 21 36 Waiting: 3 8 3.7 7 24 Total: 21 32 4.7 33 43 Percentage of the requests served within a certain time (ms) 50% 33 66% 34 75% 35 80% 36 90% 38 95% 40 98% 41 99% 43 100% 43 (longest request) |
Varnish |
Server Software: -------------- Server Hostname: localhost Server Port: 80 Document Path: /cherokee-logo.png Document Length: 23619 bytes Concurrency Level: 200 Time taken for tests: 0.154 seconds Complete requests: 1000 Failed requests: 0 Write errors: 0 Total transferred: 23933000 bytes HTML transferred: 23619000 bytes Requests per second: 6498.44 [#/sec] (mean) Time per request: 30.777 [ms] (mean) Time per request: 0.154 [ms] (mean, across all concurrent requests) Transfer rate: 151882.08 [Kbytes/sec] received Connection Times (ms) min mean[+/-sd] median max Connect: 4 11 3.0 11 17 Processing: 10 18 4.5 17 33 Waiting: 2 6 2.1 5 12 Total: 15 29 4.6 29 48 Percentage of the requests served within a certain time (ms) 50% 29 66% 29 75% 30 80% 31 90% 33 95% 36 98% 41 99% 46 100% 48 (longest request) |
Nginx is the fastest to serve small static file on my VPS. No significant difference in serving larging file.
Speedup Mysql and Webserver with Intel Compiler and tcmalloc
After reading some recent benchmark reguarding tcmalloc performance on mysql. I decide to rebuild my whole webhosting stack with it.
ICC is intel’s c++ compiler, which has faster performance is also memtioned on mysql website.
Most distros should already has google performance tools prepackaged. Installation of ICC is slightly more complicated, you can download it directly from intel’s website which is free download for non-commercial use. Archlinux and Gentoo both have packaged installer. On ubuntu/debian system you probably also need build-essential and apt-build to rebuilt packages. On archlinux you will need base-devel and abs.
For most packages, the following bash script can be used before configuration/make step. Don’t ommit the dot on first line and change the path of iccvars.sh to your installation directory.
. /opt/intel/Compiler/11.0/081/bin/iccvars.sh intel64 CC=icc CFLAGS="-xHOST -O3 -no-prec-div " LD=xild AR=xiar CXX=icpc CXXFLAGS="-xHOST -O3 -no-prec-div " LDFLAGS=-ltcmalloc_minimal export CC CFLAGS LD AR CXX CXXFLAGS LDFLAGS
These setting seems safe for all packages. Here is a summary of package specific cflags setting.
Mysql | Cherokee | Nginx | Varnish | PHP | Memcached | |
---|---|---|---|---|---|---|
-static | No | No | No | No | N/A | No |
-ipo | No | No | Yes | No | N/A | Yes |
LDFLAGS=-ltcmalloc_minimal | Yes | Yes | Yes | Yes | Yes | Yes |
configure option | –disable-shared –with-mysqld-libs=-ltcmalloc_minimal | None | None | –disable-jemalloc | Failed with ICC | None |
This might disappoint you. But the rebuilt software stacks show no improvement whatsoever in my benchmark.