keskiviikko 19. elokuuta 2009

Apache config for aggressive reverse proxy caching of a nearly static grails website, 10x performance gain

I've been running a website built with Grails on a slow virtual server with very low cpu & memory resources. The performance has been intolerable compared to plain Java webapp (servlet/jsp) performance on the same platform.

I was able to gain 10x-20x performance by caching all content in Apache reverse proxy. Each url will be fetched from the Grails app once per minute regardless of the number of requests served by the frontend Apache. If the content is updated, the delay is only 60 seconds.

This type of configuration is safe for websites that don't contain information that changes per user (no personalization) and the URL always identifies the content returned by the url.

In other cases, you can prevent caching if you set the Cache-Control header as "no-cache" in your Grails Controller or Filter. For example, you could easily setup a filter that always adds Cache-Control: no-cache after the user logs in. Please notice that private user pages must have a unique url, they cannot be the same as cached public urls.

Disclaimer: this config might contain some errors since I didn't test the latest version after adding comments and after restructuring the config file. Please report problems as comments to this blog. Test your config in a test env before putting to production. Don't blame me if you get in trouble after modifying your apache config file... :)

# Caching Reverse Proxy Apache config for Apache 2.2 (Ubuntu 9.04 / Apache 2.2.11)
# author: Lari Hotari, http://quest4grail.blogspot.com/ , 2009-08-19
#
# assumptions: Ubuntu 9.04 (or any Debian compatible distro?), Apache 2.2.x
# Grails app deployed in Tomcat running on same machine, AJP enabled on port 8009
# context root for grails app: 'mygrailsapp'
#
# save this file in /etc/apache2/sites-available/grailsproxy
#
# NOTICE: This example forces caching everything coming from grails in the Apache's
# memory cache for 60 seconds. Add "Cache-Control: no-cache" headers in grails for responses
# that shouldn't be cached or comment out the "CacheIgnoreNoLastMod On" directive.
# Make sure that the URL is the unique identifier for the returned content!
#
# activate required apache modules:
# sudo sh -c "for mod in expires proxy proxy_http proxy_ajp cache mem_cache headers ; do a2enmod $mod; done"
# activate this site:
# sudo a2ensite grailsproxy
# restart apache
# sudo /etc/init.d/apache2 restart
#
# modify config to meet your needs. This example uses VirtualHost config to be able to
# have several websites running on one IP address.
#
<VirtualHost *:80>
ServerName grailsproxy.domain
ServerAlias grailsproxy

# Logging config
LogLevel warn
ErrorLog /var/log/apache2/grailsproxy-error.log
CustomLog /var/log/apache2/grailsproxy-access.log combined

# Config for static files served by apache
DocumentRoot /data/www/grailsproxy
<Directory /data/www/grailsproxy>
Options FollowSymLinks
AllowOverride None
Order allow,deny
allow from all
</Directory>

RewriteEngine On
# redirect request if user is using an alias for this server
RewriteCond %{HTTP_HOST} !^grailsproxy.domain$
RewriteRule ^(.*)$ http://grailsproxy.domain$1 [L,R=301]

# Enable reverseproxy
<Proxy *>
AddDefaultCharset off
Order allow,deny
Allow from all
</Proxy>
ProxyRequests Off
ProxyPreserveHost On

# Proxy config for /mygrailsapp
# You can replace '/mygrailsapp' with '/' if you have deployed
# the grails app as ROOT.war in Tomcat.
ProxyPass /mygrailsapp ajp://localhost:8009/mygrailsapp
ProxyPassReverse /mygrailsapp ajp://localhost:8009/mygrailsapp
<LocationMatch "/mygrailsapp/.*\.(ico|jpg|jpeg|png|gif|js|css)$">
# prevent max-age calculation from Last-Modified
# prevent If-Modified-Since requests
# reduces the number of requests that hit the server
Header unset Last-Modified
Header unset ETag
</LocationMatch>

# Caching reverseproxy config
CacheEnable mem /
CacheEnable mem ajp://
CacheEnable mem http://
MCacheSize 8192
MCacheMaxObjectCount 1000
MCacheMinObjectSize 1
MCacheMaxObjectSize 500000
# cache for 60 seconds by default
CacheDefaultExpire 60
# FORCE caching for all documents (without Cache-Control: no-cache)
CacheIgnoreNoLastMod On
# force caching for all requests
# ignore client side Cache-Control header
CacheIgnoreCacheControl On
# don't add Set-Cookie header to cache
CacheIgnoreHeaders Set-Cookie

# Add expires headers for images, css & js files
# reduces the number of requests that hit the server
ExpiresActive On
ExpiresByType image/gif A600
ExpiresByType image/png A600
ExpiresByType image/jpeg A600
ExpiresByType text/css A600
ExpiresByType text/javascript A600
ExpiresByType application/x-javascript A600
ExpiresByType image/x-icon A600
</VirtualHost>

lauantai 15. elokuuta 2009

VisualVM CPU profiling on 64-bit JVM in Linux, solution for JVM crashing problem

I've had problems profiling Java apps (including Grails/Groovy apps) on my development machine which is a Lenovo T500 laptop running Ubuntu Linux 9.04 64-bit.
After turning on CPU profiling for a Java process in VisualVM, the JVM would always crash.

#
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGSEGV (0xb) at pc=0x00007f6479671aa3, pid=13370, tid=140068334713168
#
# JRE version: 6.0_14-b06
# Java VM: Java HotSpot(TM) 64-Bit Server VM (14.0-b15 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# V [libjvm.so+0x6a7aa3]
#
# If you would like to submit a bug report, please visit:
# http://java.sun.com/webapps/bugreport/crash.jsp
#


I was able to get around the JVM crashing problem with these steps:

  1. Turn off CPU frequency scaling on all CPU cores (example for 2 cores):
    sudo cpufreq-selector -c 0 -g performance; sudo cpufreq-selector -c 1 -g performance

    You can also you Gnome's CPU Frequency scaling monitor to select "performance" profile for both cpus (remember to add applets for all cpu cores)

  2. Reset calibration data in VisualVM (Tools->options->reset calibration data) and restart VisualVM.

  3. Start the JVM you want to profile with -XX:+UseParallelGC -Xshare:off options (JVM bug workaround: http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6862295).



I think that CPU frequency scaling makes the JVM crash on Linux 64 bit JVM (Ubuntu 9.04 + Sun JVM 1.6.0_14). Profiling won't give proper results with frequency scaling and that's also a good reason to turn it off while profiling.

Hope this tips helps anyone having problems profiling apps running in Sun 64 bit 1.6.0 JVM on Linux.