Mudy’s Blog

This time it is real. Google just announced java for app engine.

Updates: And cron job in app engine as well.

Two differences I noticed from python gae. It deploys byte code (war file) instead of source code to cloud. Using xml instead of yaml.

 Tweet
 

Yejun Uncategorized

Recently I read an interesting idea on amazon EC2 forum about Raid0 strip on EBS to improve disk access performance. So I am very curious to know whether this idea actually works. Technically it is also possible to setup a raid system on Linode(referral link) as well, but it will be backed by same hardware (so I didn’t test this idea).

In this test I used bonnie++ 1.03e with direct IO support. These 3 VPS have slightly different configure. Mosso server has 256MB ram with 2.6.24 kernel and 4 AMD virtual cores. Lindoe vps has 360MB ram with custom built 2.6.29 kernel and 4 intel virtual cores. EC2 high-cpu medium instance has 1.7GB ram with 2.6.21 kernel and 2 intel virtual cores.

Here is the raw test result. On each VPS I run bonnie++ 3 times, then use median of 3 tests as the final result. The summary result is unweighted average value of different columns. Due to the memory size difference, I used different test file size. The EBS I used here is 4×10GB raid0.

In this table, -D means that test run with Direct IO option. The best results are highlighted. Direct IO test on EBS taking forever, so I didn’t finish that test.

In this chart, I used logarithm scales and shifted origin in order to show the relative difference between them. So the column value does not reflect the real test results. Higher value is better.

Conclusions: There is no clear winner in this test. Each VPS has the their high score in different category. Only one thing is clear, O_Direct does not work very well on EBS. Due to the nature of VPS, the Disk IO test is very unreliable. The performance I show here is not repeatable and may not reflect the true disk performance.

 Tweet
 

Yejun Linux amazon, aws, bonnie++, disk, ebs, ec2, io, linode, mosso, raid0

On Varnish’s official website, there is a WordPress optimization guide For The Impatient: Preparing Varnish/Wordpress for a Slashdotting in 60 seconds or less….

The problem is that it removes cookie too aggressively. All non admin page will be virtually static. So I made my own vcl to remove cookies for only static files.

Here it is

backend default {
.host = "10.25.0.1";
.port = "80";
}
 
sub vcl_recv {
# Normalize Content-Encoding
    if (req.http.Accept-Encoding) {
        if (req.url ~ "\.(jpg|png|gif|gz|tgz|bz2|lzma|tbz)(\?.*|)$") {
            remove req.http.Accept-Encoding;
        } elsif (req.http.Accept-Encoding ~ "gzip") {
            set req.http.Accept-Encoding = "gzip";
        } elsif (req.http.Accept-Encoding ~ "deflate") {
            set req.http.Accept-Encoding = "deflate";
        } else {
            remove req.http.Accept-Encoding;
        }
    }
# Remove cookies and query string for real static files
    if (req.url ~ "^/[^?]+\.(jpeg|jpg|png|gif|ico|js|css|txt|gz|zip|lzma|bz2|tgz|tbz|html|htm)(\?.*|)$") {
       unset req.http.cookie;
       set req.url = regsub(req.url, "\?.*$", "");
    }
# Remove cookies from front page
    if (req.url ~ "^/$") {
       unset req.http.cookie;
    }
}
sub vcl_fetch {
        if (req.url ~ "^/$") {
                unset obj.http.set-cookie;
        }
}

So all interactive pages will be sent to php backend with correct cookies. All static files and front page will be served by varnish proxy.

 Tweet
 

Yejun Web varnish, vcl, wordpress

I made 2 public Archlinux EC2 AMI.

Important Notes:

• Most instructions on this page are outdated. All necessary packages have been included inside latest AMI
• If you want to build your own AMI, I released this build script on gitub and aur packages.

Update 2011/1/25

Update kernel to 2.6.37 and fix account creation.

Update 2010/8/30

Change static ip to kernel dhcp and remove initrd

Update 2010/8/28

The network configure will be saved when image first time booted. If you want to revert to dhcp in case you need to rebuild or stop. You should run this

sudo /etc/rc.d/ec2 stop

I also changed default cflags, so if you want to recompile packages, you can use srcpac. For example

sudo abs extra/python
sudo srcpac -Sb python

Update 2010/8/21

Add a user arch with the same ssh key as root.

The hostname is now static, if you want to rebundle, make sure change HOSTNAME in rc.conf to myhost and remove last line of /etc/hosts.

Here is the new build script.

Update 2010/7/23:

Updated to BTRFS as root.

Update 2010/7/20:

Updated to pvgrub and EBS.
Here is the updated script to generate an EC2 EBS.
I also made an aur package for kernel26 with patch from gentoo and opensuse.

There is a simple patch for mainline kernel from amazon.

Updates:
10/21/2009: Updated all packages and use ubuntu kernels. Here is the new AMI making script. Those kernels will load some unnecessary modules, you will need to unload them manually. I will update again if I can found more stable kernel.

They are very basic installation with just ssh. If you need tools like ec2-ami-tools or ec2-api-tools, you can find my aur packages here. Or you can add my private repo to your pacman.conf.

[iphash]
Server = http://static.iphash.net/public/i686/

or

[iphash]
Server = http://static.iphash.net/public/x86_64/

Then

pacman -Sy ec2-ami-tools ec2-api-tools

If you want to roll your own image. Here (outdated - see beginning of this post) is the script I used to make these AMIs.

If you wish to set hostname and domainname you can pass following script as instance user-data.

MYHOST=yourhost    #set your real hostname here
MYDOMAIN=yourdomain  #set your domainname here
 
sed -i s/myhost/$MYHOST/ /etc/rc.conf
hostname $MYHOST
 
echo "NISDOMAINNAME=\"$MYDOMAIN\"" >/etc/conf.d/nisdomainname
nisdomainname $MYDOMAIN
 
/etc/rc.d/syslog-ng restart
 
x=$(curl -s http://169.254.169.254/latest/meta-data/local-ipv4)
if [ $(echo $x|grep 404|wc -l) -eq 0 ]; then
cat <<EOF>/etc/hosts
#<ip-address>   <hostname.domain.org>   <hostname>
127.0.0.1               localhost.localdomain   localhost
$x  $MYHOST.$MYDOMAIN  $MYHOST
# End of file
EOF
fi
cp /etc/skel/.bash* /root/

 Tweet
 

Yejun Linux ami, arch, archlinux, cloud, ec2

Just by doing some simple tasks like dig AAAA record or traceroute6. I am on he.net’s TOP 10 IPv6 certs now.

Updates: This daily dig actually remind me those boring daily quests in WOW.

 Tweet
 

Yejun Linux he.net, ipv6