In Chapter 6 “Where to go from here” on page 58 Dean references TotallyLazy for those not wanting to take the scala / clojure route.

java -Dlog4j.configuration=file:///some/path/log4j.properties

And that file contains a class that is not on the class path

log4j.appender.myAppender=biz.minaret.log4j.DatedFileAppender
log4j.rootLogger=error, myAppender

Then due to the log4j static initialisers you will not see an error for the class in question but instead:

Caused by: java.lang.ClassNotFoundException: org.apache.log4j.FileAppender
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
at java.lang.ClassLoader.loadClass(ClassLoader.java:248)

Unfortunately for me this was caused by a transitive dependency changing in Maven. Damn you Maven/Log4J

Well lets answer the obvious questions:
“Can this work for real apps or just fun stuff?” Yes real apps old and new.
“Does it need to be greenfield?” No but it takes a long time (months) to get builds down on old codebases.

Lets have some examples then (greenfield means it was built with speed in mind)

1. 5+ year legacy app, down from 45 mins to 3+ minutes (this is the one we are focusing on at the moment – probably taken a year on and off to get this far) .
2. 3+ year legacy app, down from 20 mins to 3+ minutes (this took a couple of months)
3. 1+ year greenfield app, creeping up to 12 seconds
4. 6 month greenfield app/lib (used by number 1), creeping up to 15 seconds (warning lights are going off here)
5. 2 year greenfield library (used by 1,2,3,4), down to 8 seconds having peeked at 15 seconds

Here are my rules of engagement:

1. Nothing is safe: build tool, compiler, testing library, language, web container, messaging library, persistence layer, architecture, OS, code layout, team and process.
2. Making you app / build fast is just like making your code testable, it will make you and your app better.
3. Following from 2: Don’t split up you tests into slow and fast, split up you app into other apps or libs. Splitting the tests is just ignoring the problem plus people will just stop looking at the slow test.
4. Crazy is good

Build tool:

If you are using Maven (the binary) then you have no hope of ever being in the sub 10 seconds cool club but even if you are in a maven organisation you can use the maven structure and produce maven artifacts with your build but use a different tool (Make, sbt, buildr, ant etc). Think maven is an output contract not an implementation contract. You would have thought that maven and ivy would actually be fast at downloading libs as they tend to download the internet all the time but in fact they are incredible slow (they download sequentially which just blows my mind – this is part due to the stupid transitive nature of those tools).

Compiling:

Incremental or Clean? I truly wish incremental actually worked well, Make does a great job with production code but it just can’t have the insight to understand the relationship between production and test code. The second problem with incremental is that I’m always going to do a clean on CI which tends to mean incremental is only good for incremental non-pushing local builds. As I tend to push on most commits it’s little value to me. Finally incremental using javac (without Make doing the smarts) is actually slower that doing a clean compile (and jar) (i think mainly due to the the fact it checks each file and class to see if it’s exists and changed while in clean mode it only checks to see if the file exists – I hoping to fix this soon)

So I use clean.

This leads me to the actual compiler. If the compiling is slow (hey Scala (see foot note) thats you I’m talking 2) then you need to change compiler (fast scala compiler, sbt) but again they don’t work well in CI as I want a clean build always (not incremental). Right now I’m investigating extending the Java 6 compiler to remove the unneeded file exists checks and possible jaring the class files directly in the compiler (the java compiler supports writing directly to jars but it’s not used for some reason)

Pro tip: Always compile and jar in one step: This will actually speed up you build on a number of fronts, by putting it in a jar when you compile your tests it will load the jar into memory so no more disk IO: instead of lots of tiny random reads, just one nice big fat read for the zip. Do the same for your test jar and your tests will load and run quicker as well. I have timed this a number of times on projects and clean, compiler + jar and test is faster than incremental (with javac) compile and test loose files. Try it and see. (NB Large projects this might not hold but that’s another problem)

Testing:

Stop using crappy testing tools that need parse some text and don’t support refactoring. This will kill your build on two fronts:

1. Parsing say Html and invoking Java/C# etc is going to be slow as hell in the short term.
2. Because you can’t refactor the tests they are going to become a big noose around your neck long term. I’m talking to you Fit, Fitness, Concordion …

Stop using slow acceptance testing tools like Selenium, not only are they incredible slow but they are incredibly high maintenance for very little return on investment. (To be fair to Selenium I believe you can run WebDriver with HtmlUnit directly off an InputStream)
UPDATE: Selenium has an in process project that looks very promising https://github.com/aharin/inproctester

Instead use an in-memory (no TCP/HTTP) web testing tools. A lot of trendy web frameworks have support and if yours doesn’t then maybe it’s time to look for one that does. If you can’t convert a String to Request and back again then you are probably not using a web framework that is well written or well tested, let alone fast.

If you have some production constraint that is slow (say Oracle) then separate the oracle integration tests out from your acceptance tests. I create fast production quality implementations (in memory, local disk, h2, hsql, lucene etc) that for-fill the same contract. And by this I mean I have either an abstract contract test that all implementations inherit or parametrise the test that takes a list of implementations. For more confidence you can have your CI run with Oracle and local run with H2, I normally see a factor of 3-4x faster with this setup. I prefer to just use in memory collection backed version as this can give you a factor up to 10x speed up. People always bitch about not using the exact production stack (blah blah blah) but I seem to be one of the few people actually measuring the cost/benifit.

Some fuzzy facts: On e4 (ramped up to 2 teams with maybe 20+ devs in total) we used the hsql local and oracle on CI setup. Build contained maybe 2000 tests (I don’t separate acceptance, unit etc) Local build was 56 seconds on Ubuntu, 1 minute 30 on windows (same hardware), CI was 3 mins 30 ish. We had one production oracle issue caused by the difference between oracle and hsql (empty string == null in oracle) as most where caught at CI. The bug took us max one day to fix and release but I took some heat for it from the devs but I pulled out the calculator. We had been running for 3 months, we had 12 pairs doing 10+ builds a day each, so that’s about 5 man hours a day or about 20 man days over the 3 months. So 1 day cost for 20 days saved, OK there was some reputation risk but I think hard to measure stuff like attitude and mean time to repair (when you don’t have a second team doing the same storry with full stack build) was even better. Graham Brooks did a great presentation for this for one of the Agile conference.

Language / Libraries / Web Containers etc

If it’s slow ditch it.

Yup I ditched Scala because of it’s terrible compile time. I actually re-wrote a couple of libraries/apps so I have side by side with the exact same code in one language and another. It’s scary without incremental compile (sbt) or long running processes (fast compiler) the raw scala compiler is 10x slower than Java. 10x!!! That’s just insane. With incremental it’s the same a full clean Java compile. Not only is scala slow to compile but the byte code is 2x as big as Java for the same functionality, That must translate into a performance hit. 

UPDATE: For byte code size please compare yadic-67.jar (last scala version) and yadic-68.jar (exact same functionality in java)

It’s not just scala mind:

Lucene is going the same way, between version 3.3 and 3.4, if you don’t create create a singleton for the IndexWriters and Readers it’s 30x slower to instantiate. 30x!!! FFS. Now I give you that in production you don’t create a new index 30 times in a second but for testing it’s a nightmare. At the same time Lucene is so much faster than going across the wire to a remote DB for obvious reasons.

Hibernate Session Factory can take seconds to start up, WTF you are just readying some XML or annotations. So either ditch hibernate or remove it from your acceptance tests (in-memory FTW).
UPDATE: Hibernate seems to take about 1.3 seconds to instantiate the SessionFactory class even with less than 200 line of HBM file.

Velocity is the same, if you don’t use their singleton it can takes seconds to start.
UPDATE: I’ve done some more testing and Velocity does appear to be a little better these days (100ms to start). Not great but not terrible either

Tomcat / Jetty take seconds to start, okay you can run Jetty in embedded with helps a lot but nothing, yes nothing is a fast as the the embedded Java 6 HTTP server, 10ms cold JVM, <1ms warm.

GWT is a big time sync on multiple fronts:

1. The compile step is slow.
2. Because you now have so much generated Javascript you feel you have to use tools like Selenium to acceptance test it.
3. You will have so much more Java code than you really need…

Again I have numbers: Application number 2 we ripped out GWT and replaced it with some crappy MVC setup: compile time reduced by 30 seconds, acceptance testing reduced by nearly 20 minutes as we no longer needed to use IE Selenium driver. Java Code reduced by 82%,

Spring lets just leave it at that. UPDATE: 5+ year old project: Using Xml config takes 14 seconds to start (This can be reduced by using the lazy attribute). 1 year old project using annotations takes 1.3 seconds to start.

OS / Hardware

Windows disk IO is just poor, for builds it’s a killer. As I said before we saw a 50% increase in time using Windows (comapred to Ubuntu) with exactly the same hardware, code.

MacOS is faster than Windows for sure but Linux is fastest. We have dual boot with MacOSX and Ubuntu on my team and we see 15-20% improvement for exactly the same code, hardware. (These are bad boy Mac Pro machines not some Hackintosh).

SSD / RAM DISK: I don’t have exact numbers but saw around 10% decrease in time on my laptop with a new SSD.

Divide and Conquer  / Architecture

As I said before don’t split you test code, split your whole project. We have extracted libraries and applications from our legacy applications. What’s great is we actually need less slow acceptance tests because the library is more focussed and tests at the API level. While the app only needs to test that the default config works. These really helps counter another common anti-pattern I see is that people want to retest every thing at the acceptance level.

If you force logic into a library then an acceptance test is actually an API test as there is no UI.

I think that’s most of it dumped.

*Notes*

Some compile times for 8.5K loc project rewriten from Scala to Java

Scala

• ant clean compile test package: 30 seconds
• sbt clean compile test package: 56 seconds
• sbt incremental compile test package: 6 seconds
• buildr (clean compile test package): 37 seconds
• maven2 clean verify: 42 seconds

Java

• ant clean compile test package: 6.9~ seconds
• make clean compile test package: 6.7~ seconds
• make incremental compile test package: 6.4 seconds (changing one prod file and one test file)
• make incremental compile test package: 3.8 seconds (changing one test file only)

You can check this by typing:

xinput list

If you see something like this:

Virtual core pointer                    	id=2	[master pointer  (3)]
↳ Virtual core XTEST pointer              	id=4	[slave  pointer  (2)]
↳ SynPS/2 Synaptics TouchPad              	id=11	[slave  pointer  (2)]
↳ PS/2 Generic Mouse                      	id=12	[slave  pointer  (2)]

As you can see the TrackPoint is not listed and as such you wont see it in any of the configuration screens.

As I haven’t worked out how to get it to detect it as an actual TrackPoint the best solution I have is to change the acceleration mode to more closely represent what you would expect:

xinput set-prop "PS/2 Generic Mouse" "Device Accel Profile" 6

I then just add this to the startup applications for my profile.

Very chilled out…

• Make it easy for people to do the right thing, hard to do the wrong thing.
• Make it valuable (Finds more bugs than false alarms)
• Make it resilient (Allows me to refactor like the true crazy person I am)
• Make it fast (Magnify it’s value rather deminishing it)
• Make it good enough (Give me enough confidence to do more crazy stuff but don’t become dogmatic)
• Allow people to do what they are good at

Next I’ll talk about how the default setup I use to do, in-memory, out of container testing.

• Semantic Html is my golden hammer
• Progressive enhancement is my swiss army knife
• I love JavaScript but try to never write any but if i must I make sure it’s generic. I try very hard to make sure no else writes any either and if they do I just refactor it till it deletes itself.
• I love AHAH and barely tolerate AJAX / JSON
• REST is my natural state not something I read about

All of these forces / values / beliefs drive me to solve the problem of web testing in a very different approach from most people. It is about an alternative holistic approach to web architecture, testing and productivity.

So now I have expressed some of the forces / the whys, I will try cover the hows and whats.

As the “crazy” guy behind the in-memory / out-of-container acceptance testing on a number of java/.net projects, I think it’s important I explain to people the “Why” and the forces / constraints I am trying to balance. But first I want to quickly lay down my beliefs and values:

• I believe in testing as much as possible (UI included)
• I believe tests must add more value than they cost (Measure it!)
• I value tests that are fast and are resilient to change more than tests that take a long time to run and are brittle.
• When refactoring a feature I value acceptances tests and integration test over unit tests.
• When designing/exploring a new interface / object interaction I value unit tests over acceptance tests to help guide me.
• I believe that QA’s are so much better at finding bugs than DEVs but worse at writing code / abstractions