Apache config for aggressive reverse proxy caching of a nearly static grails website, 10x performance gain

I've been running a website built with Grails on a slow virtual server with very low cpu & memory resources. The performance has been intolerable compared to plain Java webapp (servlet/jsp) performance on the same platform.

I was able to gain 10x-20x performance by caching all content in Apache reverse proxy. Each url will be fetched from the Grails app once per minute regardless of the number of requests served by the frontend Apache. If the content is updated, the delay is only 60 seconds.

This type of configuration is safe for websites that don't contain information that changes per user (no personalization) and the URL always identifies the content returned by the url.

In other cases, you can prevent caching if you set the Cache-Control header as "no-cache" in your Grails Controller or Filter. For example, you could easily setup a filter that always adds Cache-Control: no-cache after the user logs in. Please notice that private user pages must have a unique url, they cannot be the same as cached public urls.

Disclaimer: this config might contain some errors since I didn't test the latest version after adding comments and after restructuring the config file. Please report problems as comments to this blog. Test your config in a test env before putting to production. Don't blame me if you get in trouble after modifying your apache config file... :)

# Caching Reverse Proxy Apache config for Apache 2.2 (Ubuntu 9.04 / Apache 2.2.11)
# author: Lari Hotari, http://quest4grail.blogspot.com/ , 2009-08-19
# 
# assumptions: Ubuntu 9.04 (or any Debian compatible distro?), Apache 2.2.x
# Grails app deployed in Tomcat running on same machine, AJP enabled on port 8009
# context root for grails app: 'mygrailsapp'
# 
# save this file in /etc/apache2/sites-available/grailsproxy
#
# NOTICE: This example forces caching everything coming from grails in the Apache's
# memory cache for 60 seconds. Add "Cache-Control: no-cache" headers in grails for responses
# that shouldn't be cached or comment out the "CacheIgnoreNoLastMod On" directive.
# Make sure that the URL is the unique identifier for the returned content!
#
# activate required apache modules:
# sudo sh -c "for mod in expires proxy proxy_http proxy_ajp cache mem_cache headers ; do a2enmod $mod; done"
# activate this site:
# sudo a2ensite grailsproxy
# restart apache
# sudo /etc/init.d/apache2 restart
#
# modify config to meet your needs. This example uses VirtualHost config to be able to
# have several websites running on one IP address.
#
<VirtualHost *:80>
 ServerName grailsproxy.domain
 ServerAlias grailsproxy

 # Logging config
 LogLevel warn
 ErrorLog /var/log/apache2/grailsproxy-error.log
 CustomLog /var/log/apache2/grailsproxy-access.log combined

 # Config for static files served by apache
 DocumentRoot /data/www/grailsproxy
 <Directory /data/www/grailsproxy>
  Options FollowSymLinks
  AllowOverride None
  Order allow,deny
  allow from all
 </Directory>
 
 RewriteEngine On
 # redirect request if user is using an alias for this server
 RewriteCond %{HTTP_HOST} !^grailsproxy.domain$
 RewriteRule ^(.*)$    http://grailsproxy.domain$1    [L,R=301]

 # Enable reverseproxy
 <Proxy *>
  AddDefaultCharset off
  Order allow,deny
  Allow from all
 </Proxy>
 ProxyRequests Off
 ProxyPreserveHost On
 
 # Proxy config for /mygrailsapp
 # You can replace '/mygrailsapp' with '/' if you have deployed
 # the grails app as ROOT.war in Tomcat.
 ProxyPass /mygrailsapp ajp://localhost:8009/mygrailsapp
 ProxyPassReverse /mygrailsapp ajp://localhost:8009/mygrailsapp
 <LocationMatch "/mygrailsapp/.*\.(ico|jpg|jpeg|png|gif|js|css)$">
  # prevent max-age calculation from Last-Modified
  # prevent If-Modified-Since requests
  # reduces the number of requests that hit the server
  Header unset Last-Modified
  Header unset ETag
 </LocationMatch>
 
 # Caching reverseproxy config
 CacheEnable mem /
 CacheEnable mem ajp://
 CacheEnable mem http://
 MCacheSize 8192
 MCacheMaxObjectCount 1000
 MCacheMinObjectSize 1
 MCacheMaxObjectSize 500000
 # cache for 60 seconds by default
 CacheDefaultExpire 60
 # FORCE caching for all documents (without Cache-Control: no-cache)
 CacheIgnoreNoLastMod On
# force caching for all requests
# ignore client side Cache-Control header
CacheIgnoreCacheControl On
# don't add Set-Cookie header to cache
CacheIgnoreHeaders Set-Cookie

 # Add expires headers for images, css & js files
 # reduces the number of requests that hit the server
 ExpiresActive On
 ExpiresByType image/gif A600
 ExpiresByType image/png A600
 ExpiresByType image/jpeg A600
 ExpiresByType text/css A600
 ExpiresByType text/javascript A600
 ExpiresByType application/x-javascript A600
 ExpiresByType image/x-icon A600
</VirtualHost>

VisualVM CPU profiling on 64-bit JVM in Linux, solution for JVM crashing problem

I've had problems profiling Java apps (including Grails/Groovy apps) on my development machine which is a Lenovo T500 laptop running Ubuntu Linux 9.04 64-bit.
After turning on CPU profiling for a Java process in VisualVM, the JVM would always crash.

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007f6479671aa3, pid=13370, tid=140068334713168
#
# JRE version: 6.0_14-b06
# Java VM: Java HotSpot(TM) 64-Bit Server VM (14.0-b15 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# V  [libjvm.so+0x6a7aa3]
#
# If you would like to submit a bug report, please visit:
#   http://java.sun.com/webapps/bugreport/crash.jsp
#

I was able to get around the JVM crashing problem with these steps:

1. Turn off CPU frequency scaling on all CPU cores (example for 2 cores):
   

   sudo cpufreq-selector -c 0 -g performance; sudo cpufreq-selector -c 1 -g performance

   
   You can also you Gnome's CPU Frequency scaling monitor to select "performance" profile for both cpus (remember to add applets for all cpu cores)


2. Reset calibration data in VisualVM (Tools->options->reset calibration data) and restart VisualVM.


3. Start the JVM you want to profile with -XX:+UseParallelGC -Xshare:off options (JVM bug workaround: http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6862295).

I think that CPU frequency scaling makes the JVM crash on Linux 64 bit JVM (Ubuntu 9.04 + Sun JVM 1.6.0_14). Profiling won't give proper results with frequency scaling and that's also a good reason to turn it off while profiling.

Hope this tips helps anyone having problems profiling apps running in Sun 64 bit 1.6.0 JVM on Linux.

Improving Grails TDD within Eclipse

Grails 1.1 unit testing has support for mocking domain classes/objects. This feature is very nice.

These are additional instructions to the previous ones.

Additional instructions to run a Grails domain class unit test:
1. Copy the below code to a file called [grails app rootdir]/src/java/GrailsAwareGroovyTestSuite.java .
2. Update the Eclipse Run Configuration for the launcher created by the previously posted instructions.
3. Change the Test Class name to "GrailsAwareGroovyTestSuite".
(previous instructions contain the full procedure)

I've posted this to the Grails JIRA, it's GRAILS-4123.

UPDATE: Check the JIRA issue to get the latest version of this code.

import grails.util.BuildSettings;
import grails.util.BuildSettingsHolder;
import groovy.util.GroovyTestSuite;

import java.io.File;

import junit.framework.Test;

import org.codehaus.groovy.grails.compiler.injection.ClassInjector;
import org.codehaus.groovy.grails.compiler.injection.DefaultGrailsDomainClassInjector;
import org.codehaus.groovy.grails.compiler.injection.GrailsAwareClassLoader;
import org.codehaus.groovy.grails.compiler.support.GrailsResourceLoader;
import org.codehaus.groovy.grails.plugins.GrailsPluginUtils;

/**
 * 
 * Adds support for running Grails JUnit Tests from Eclipse JUnit runner or even
 * from the command line.
 * 
 * Set GRAILS_HOME environment variable before running the test. 
 * Change the working directory to the Grails application's root directory.
 *  
 * 
 * more information: http://quest4grail.blogspot.com/search/label/junit
 * 
 * 
 * @author Lari Hotari
 * 
 */
public class GrailsAwareGroovyTestSuite extends GroovyTestSuite {
 protected final GrailsAwareClassLoader gcl;

 public GrailsAwareGroovyTestSuite() {
  File basedirFile = new File(".").getAbsoluteFile();
  System.setProperty(BuildSettings.APP_BASE_DIR, basedirFile.getPath());
  BuildSettings grailsSettings = new BuildSettings(new File(System.getenv("GRAILS_HOME")), basedirFile);
  BuildSettingsHolder.setSettings(grailsSettings);

  gcl = new GrailsAwareClassLoader(getClass().getClassLoader());
  gcl.setClassInjectors(new ClassInjector[] { new DefaultGrailsDomainClassInjector() });
  gcl.setResourceLoader(new GrailsResourceLoader(GrailsPluginUtils.getArtefactResources(".")));
 }

 public static Test suite() {
  GrailsAwareGroovyTestSuite suite = new GrailsAwareGroovyTestSuite();
  try {
   suite.loadTestSuite();
  } catch (Exception e) {
   throw new RuntimeException((new StringBuilder()).append("Could not create the test suite: ").append(e)
     .toString(), e);
  }
  return suite;
 }

 @Override
 public Class compile(String fileName) throws Exception {
  return gcl.parseClass(new File(fileName));
 }
}

Missing the green bar developing Grails apps in Eclipse?

I've been missing the green bar developing Grails apps in Eclipse.
I found a solution to the problem.

Instructions (tested with Grails 1.1-RC1):


Creating Eclipse Run Configuration

1. Open the Run Configurations dialog (Run -> Run Configurations. Switch to Java perspective first. Make sure you've added the GRAILS_HOME classpath variable for a new Grails project.)

2. Browse to the item "Junit" and double click on it. It will create a new Junit run configuration.

3. Select "Run a single test"

4. Enter "groovy.util.GroovyTestSuite" as the Test class

5. Switch to "Arguments" tab.

6. In "VM arguments" field enter "-Dtest=${resource_loc}"

7. Click close


Creating and running a test

1. In the command line: grails create-unit-test GrailsJunit

2. Refresh Eclipse view

3. Open the Junit test

4. Open the Run Configurations dialog (Run -> Run Configurations)

5. Select the Junit run configuration you've created recently.

6. Click "Run"


Changing the behaviour of the Run button's default action

In Eclipse 3.4, the default behaviour of clicking the run button is "contextual".
It's easier to run the test if you can just click the Run button.

1. Open "Window->Preferences"
2. Write "launching" in the filter text box.
3. Select "Always launch the previously launched application" in the "Launch Operation" section.

Now you can execute the test by opening the file and just click the run button. Cool!


Re-run button in the JUnit panel

You won't be able to re-run the test by clicking the button in the JUnit panel.
If you want it to work, you will have to create separate launchers for each test.
Another solution is to edit the launcher's configuration each time you start developing a new class.
This can be done by setting the "VM arguments" property to something like this:
-Dtest=${resource_loc:/GrailsJunitTutorial/test/unit/GrailsJunitTests.groovy}

Quest for the Grail of Programming using Grails

Every programmer seems to seek the Holy Grail of Programming. This tendency can been seen in the numerous attempts to build the best framework ever, the über framework that will solve all of the concerns of application programming.

I'm starting to gather small notes about my quest for the Grail of Programming using the Grails platform.