I wrote a bash script to automatically generate Shorewall blacklist from Spamhaus drop list and dshield.org’s block list .
Do not run this script automatically if ssh is the only mean you connect to your server, because you can accidentally blacklist yourself. And you may not run it more often then once per hour due to spamhaus limitation.
#!/bin/sh
echo "#ADDRESS/SUBNET PROTOCOL PORT" > /tmp/blacklist
wget -q -O - http://feeds.dshield.org/block.txt | awk --posix '/^[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.0\t/ { print $1 "/24";}' >> /tmp/blacklist
wget -q -O - http://www.spamhaus.org/drop/drop.lasso | awk --posix '/^[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\// { print $1;}' >> /tmp/blacklist
echo "#LAST LINE -- ADD YOUR ENTRIES BEFORE THIS ONE -- DO NOT REMOVE" >> /tmp/blacklist
mv /tmp/blacklist /etc/shorewall/blacklist
shorewall refresh &>/dev/nullI also use fail2ban to generate dynamic shorewall ban list.
UPDATE: And don’t forget enable blacklist option in /etc/shorewall/shorewall.conf
BLACKLIST_DISPOSITION=DROP
Yejun Linux blacklist, dshield, firewall, security, shorewall, spamhaus
I did a micro benchmark for quick comparison between nginx and cherokee on Linode 540 VPS. Result may differ on a dedicate server. static file(100MB)
| Cherokee |
Benchmarking localhost (be patient)
Completed 100 requests
Completed 200 requests
Finished 200 requests
Server Software: Cherokee/0.99.0
Server Hostname: localhost
Server Port: 80
Document Path: /100mb.test
Document Length: 104857600 bytes
Concurrency Level: 20
Time taken for tests: 30.696 seconds
Complete requests: 200
Failed requests: 0
Write errors: 0
Total transferred: 20971557200 bytes
HTML transferred: 20971520000 bytes
Requests per second: 6.52 [#/sec] (mean)
Time per request: 3069.623 [ms] (mean)
Time per request: 153.481 [ms] (mean, across all concurrent requests)
Transfer rate: 667184.09 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 1 0.2 1 1
Processing: 3004 3069 73.9 3048 3228
Waiting: 0 1 0.2 1 2
Total: 3004 3069 73.9 3048 3228
Percentage of the requests served within a certain time (ms)
50% 3048
66% 3068
75% 3071
80% 3191
90% 3228
95% 3228
98% 3228
99% 3228
100% 3228 (longest request)
|
| Nginx |
Benchmarking localhost (be patient)
Completed 100 requests
Completed 200 requests
Finished 200 requests
Server Software: nginx/0.6.35
Server Hostname: localhost
Server Port: 80
Document Path: /100mb.test
Document Length: 104857600 bytes
Concurrency Level: 20
Time taken for tests: 30.543 seconds
Complete requests: 200
Failed requests: 0
Write errors: 0
Total transferred: 20971571000 bytes
HTML transferred: 20971520000 bytes
Requests per second: 6.55 [#/sec] (mean)
Time per request: 3054.277 [ms] (mean)
Time per request: 152.714 [ms] (mean, across all concurrent requests)
Transfer rate: 670536.80 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 1 0.2 0 1
Processing: 3006 3054 32.6 3047 3106
Waiting: 0 1 0.4 1 3
Total: 3006 3054 32.6 3048 3107
ERROR: The median and mean for the initial connection time are more than twice the standard
deviation apart. These results are NOT reliable.
Percentage of the requests served within a certain time (ms)
50% 3048
66% 3069
75% 3087
80% 3098
90% 3107
95% 3107
98% 3107
99% 3107
100% 3107 (longest request)
|
Small html file (468 bytes)
| Nginx |
Server Software: nginx/0.6.35
Server Hostname: localhost
Server Port: 80
Document Path: /
Document Length: 468 bytes
Concurrency Level: 200
Time taken for tests: 0.123 seconds
Complete requests: 1000
Failed requests: 0
Write errors: 0
Total transferred: 737100 bytes
HTML transferred: 491400 bytes
Requests per second: 8150.03 [#/sec] (mean)
Time per request: 24.540 [ms] (mean)
Time per request: 0.123 [ms] (mean, across all concurrent requests)
Transfer rate: 5866.59 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 3 8 1.7 9 11
Processing: 6 14 3.3 14 24
Waiting: 3 11 3.3 12 23
Total: 14 22 2.6 22 33
Percentage of the requests served within a certain time (ms)
50% 22
66% 23
75% 23
80% 24
90% 25
95% 26
98% 29
99% 30
100% 33 (longest request)
|
| Cherokee |
Server Software: Cherokee/0.99.0
Server Hostname: localhost
Server Port: 80
Document Path: /
Document Length: 468 bytes
Concurrency Level: 200
Time taken for tests: 0.139 seconds
Complete requests: 1000
Failed requests: 0
Write errors: 0
Total transferred: 732224 bytes
HTML transferred: 509184 bytes
Requests per second: 7203.83 [#/sec] (mean)
Time per request: 27.763 [ms] (mean)
Time per request: 0.139 [ms] (mean, across all concurrent requests)
Transfer rate: 5151.19 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 6 11 2.5 12 16
Processing: 7 14 3.9 14 27
Waiting: 3 10 3.9 9 19
Total: 16 25 3.5 25 33
Percentage of the requests served within a certain time (ms)
50% 25
66% 27
75% 27
80% 28
90% 29
95% 31
98% 32
99% 32
100% 33 (longest request)
|
| Varnish |
Server Software: ----------
Server Hostname: localhost
Server Port: 80
Document Path: /
Document Length: 468 bytes
Concurrency Level: 200
Time taken for tests: 0.141 seconds
Complete requests: 1000
Failed requests: 0
Write errors: 0
Total transferred: 879491 bytes
HTML transferred: 528372 bytes
Requests per second: 7067.44 [#/sec] (mean)
Time per request: 28.299 [ms] (mean)
Time per request: 0.141 [ms] (mean, across all concurrent requests)
Transfer rate: 6070.07 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 6 11 2.5 11 16
Processing: 6 14 3.5 13 24
Waiting: 3 10 3.5 9 21
Total: 17 25 3.4 25 38
Percentage of the requests served within a certain time (ms)
50% 25
66% 26
75% 26
80% 27
90% 29
95% 31
98% 33
99% 34
100% 38 (longest request)
|
Small image file (23k bytes)
| Nginx |
Server Software: nginx/0.6.35
Server Hostname: localhost
Server Port: 80
Document Path: /cherokee-logo.png
Document Length: 23619 bytes
Concurrency Level: 200
Time taken for tests: 0.139 seconds
Complete requests: 1000
Failed requests: 0
Write errors: 0
Total transferred: 24624128 bytes
HTML transferred: 24379632 bytes
Requests per second: 7171.34 [#/sec] (mean)
Time per request: 27.889 [ms] (mean)
Time per request: 0.139 [ms] (mean, across all concurrent requests)
Transfer rate: 172449.16 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 5 3.6 4 16
Processing: 0 20 6.1 23 29
Waiting: 0 15 5.7 17 26
Total: 6 25 5.5 26 45
Percentage of the requests served within a certain time (ms)
50% 26
66% 27
75% 27
80% 27
90% 28
95% 35
98% 41
99% 43
100% 45 (longest request)
|
| Cherokee |
Server Software: Cherokee/0.99.0
Server Hostname: localhost
Server Port: 80
Document Path: /cherokee-logo.png
Document Length: 23619 bytes
Concurrency Level: 200
Time taken for tests: 0.180 seconds
Complete requests: 1000
Failed requests: 0
Write errors: 0
Total transferred: 26518338 bytes
HTML transferred: 26287947 bytes
Requests per second: 5564.92 [#/sec] (mean)
Time per request: 35.939 [ms] (mean)
Time per request: 0.180 [ms] (mean, across all concurrent requests)
Transfer rate: 144113.78 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 4 11 3.4 11 18
Processing: 11 21 5.0 21 36
Waiting: 3 8 3.7 7 24
Total: 21 32 4.7 33 43
Percentage of the requests served within a certain time (ms)
50% 33
66% 34
75% 35
80% 36
90% 38
95% 40
98% 41
99% 43
100% 43 (longest request)
|
| Varnish |
Server Software: --------------
Server Hostname: localhost
Server Port: 80
Document Path: /cherokee-logo.png
Document Length: 23619 bytes
Concurrency Level: 200
Time taken for tests: 0.154 seconds
Complete requests: 1000
Failed requests: 0
Write errors: 0
Total transferred: 23933000 bytes
HTML transferred: 23619000 bytes
Requests per second: 6498.44 [#/sec] (mean)
Time per request: 30.777 [ms] (mean)
Time per request: 0.154 [ms] (mean, across all concurrent requests)
Transfer rate: 151882.08 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 4 11 3.0 11 17
Processing: 10 18 4.5 17 33
Waiting: 2 6 2.1 5 12
Total: 15 29 4.6 29 48
Percentage of the requests served within a certain time (ms)
50% 29
66% 29
75% 30
80% 31
90% 33
95% 36
98% 41
99% 46
100% 48 (longest request)
|
Nginx is the fastest to serve small static file on my VPS. No significant difference in serving larging file.
Yejun Linux benchmark, cherokee, nginx
After reading some recent benchmark reguarding tcmalloc performance on mysql. I decide to rebuild my whole webhosting stack with it.
ICC is intel’s c++ compiler, which has faster performance is also memtioned on mysql website.
Most distros should already has google performance tools prepackaged. Installation of ICC is slightly more complicated, you can download it directly from intel’s website which is free download for non-commercial use. Archlinux and Gentoo both have packaged installer. On ubuntu/debian system you probably also need build-essential and apt-build to rebuilt packages. On archlinux you will need base-devel and abs.
For most packages, the following bash script can be used before configuration/make step. Don’t ommit the dot on first line and change the path of iccvars.sh to your installation directory.
. /opt/intel/Compiler/11.0/081/bin/iccvars.sh intel64
CC=icc
CFLAGS="-xHOST -O3 -no-prec-div "
LD=xild
AR=xiar
CXX=icpc
CXXFLAGS="-xHOST -O3 -no-prec-div "
LDFLAGS=-ltcmalloc_minimal
export CC CFLAGS LD AR CXX CXXFLAGS LDFLAGS
These setting seems safe for all packages. Here is a summary of package specific cflags setting.
| |
Mysql |
Cherokee |
Nginx |
Varnish |
PHP |
Memcached |
| -static |
No |
No |
No |
No |
N/A |
No |
| -ipo |
No |
No |
Yes |
No |
N/A |
Yes |
| LDFLAGS=-ltcmalloc_minimal |
Yes |
Yes |
Yes |
Yes |
Yes |
Yes |
| configure option |
–disable-shared –with-mysqld-libs=-ltcmalloc_minimal |
None |
None |
–disable-jemalloc |
Failed with ICC |
None |
This might disappoint you. But the rebuilt software stacks show no improvement whatsoever in my benchmark.
Yejun Linux, Web cherokee, icc, intel, memcached, mysql, nginx, php, tcmalloc, varnish
I was playing HE.net’s IPv6 certification test last night. And I got stucked on last step, adding an AAAA tld glue record for my domain. Unfortuantely the domain registrar I am using, 1&1 AG, does not support AAAA record in their system.
Here is the email I got this afternoon,
Thank you for contacting us. Unfortunately, our system does not support AAAA record. Our system only supports modifying the DNS, A-record & Mx-Records as well as CNAME. If you have any further questions please do not hesitate to contact us.
– Sincerely,
xxxxxxx xxxxxxx
Technical Support
1&1 Internet
The test from he.net is not easy. Here is the list of softwares and services I used last night through trial and error.
- The first test is browsing an IPv6 website from he.net. I tried many methods. On my local Windows machine, I can ping -6 ipv6 address, I can copy and paste ipv6 into browser. But browser just refused to connect to remote host by ipv6 hostname. I tried to build ipv6 enabled Lynx on a linux machine, which also failed. Because none of above worked, I had to cheat on this step by using http://ipv6.he.net.ipv4.sixxs.org/.
- Run a webserver on IPv6 address. Very simple, I added an ipv6 address into my webserver, Cherokee, and everything worked as they should.
- Run a mail server on IPv6 address. Postfix, the mail server I am currently using, supports IPv6 natively. Simply add inet_protocols = all into main.cf, postfix will start listerning on IPv6 socket. First email unfortunately failed due to dkim-milter refused connection from ipv6 address. After disabling, 2nd email passed through.
- Have a RDNS record for my MX record. I am using ipv6 patched tinydns. NS and SOA record for my /48 prefix to tinydns are required to get response for PTR record. It took me a while to figure that out. Patched tinydns has a utility add-host6 to handle record for both forward and reverse dns but ns and soa need to be added manually.
- Run DNS server natively on IPv6. Even though tinydns is listerning on IPv6 address and my local dig show correct response. But he.net did not accept it for unknown reason. So I installed Bind9, and everything just worked in less than 5 minutes.
- Add an AAAA glue record to TLD nameserver. 1&1 AG does not support it as I said at beginning of the post. I am transferring my domain to GKG.net, hope they works, and I will update this post once I finish this step. Here is a list of domain registrars supporting ipv6 glue.
So here is my test result.
I am using dnsmadeeasy for my DNS, but they do not support IPv6 in any means. My VPS providor, Linode LLC, supports AAAA record in nameserver, but no ipv6 PTR record and no native IPv6 connection, no ipv6 dns servers.
Yejun Linux 1and1, dns, domain, ipv6, mailserver, registrar, webserver
The default installation of WordPress use full version of css and javascript library. In order to minify all of these files, I wrote this bash script to minify all of them. There will be no progress bar while minifying, so be patient.
cd wordpress
find -H . -type f -writable \( -name \*.css -o -name \*.js \) \
-exec sh -c "yuicompressor {} -o /tmp/yui.tmp && mv /tmp/yui.tmp {}" \;This will only work under POSIX system.
Findutils should be included in most popular Linux distros.
Yuicompressor can be downloaded from here.
Yejun Linux bash, css, javascript, js, minify, website optimization, wordpress, yui, yuicompressor
Tinydns support wildcards. However I just made a mistake make my own website inaccessible.
I was pointing all subdomains to a single host including this website.
C*.mudy.info:nw.mudy.info:1200
It worked perfect fine, until today I encounter a website which requires myn...@blog.mudy.info to verify ownership. I know it should just working without any further settings, however to be nice I added an additional mx record to my Tinydns to explicitly indicates the mail host of blog.mudy.info
@blog.mudy.info::nw.mudy.info.:5:1200
After that my website become inaccessible. Apparently tinydns will stop looking into data once a matching name found, even though the record type does not match and there are better matching wildcards.
To fix this, I added another explicit A record.
+blog.mudy.info:66.246.138.44:1200
CNAME record will work as well.
Yejun Linux djbdns, dns, mx, tinydns, wildcards
RSS
Follow
