Journey to a Build

Believe it our not, our customers often ask about our build system. Maybe it's because they're all developers themselves.

Build systems are rarely designed properly from the start. Most I've seen are loose assemblages of scripts with an out-dated step-by-step README telling you how to install all the executables and libraries. Ours was.

Even the shortest journey starts with an embarrassing moment.

I knew we had to fix the build system when Brandon first arrived. He watched as Eric fixed an obscure (but for one customer, critical) bug in the public release of Code Reviewer. Eric smashed Control-Shift-B to build. (You'd think Visual Studio would have an easier mnemonic for the most common operation.) NUnit said the unit tests still pass. Then he checked in the fix.

So far so good. Then Eric turned to Teddy The Build Machine for the installer. Brandon asked Eric why he couldn't just build the installer himself. Eric didn't know, sometimes it didn't seem to work. One time the installer didn't link to the right version of .NET. Sometimes one of the sub-libraries isn't built but you can't tell until you've actually installed it.

Brandon looked at me. "I don't know," I said, "I haven't been able to make a proper build since we officially supported .NET 2.0."

Brandon didn't have to say anything. We knew we were in trouble.

Mapping it out

We started with the Java projects (Code Collaborator). We decided that we wanted all of the following "features" in our shiny new build system:

1. All installers are built properly and with the right library versions, even when we give 1-off builds to customers to test a fix, no matter whose workstation built the installers.
   
   
2. The build (compile, test, install) must be able to run completely unattended, e.g. for official builds, nightly builds, continuous builds.


3. Everyone is debugging against exactly the same code that customers will be using.


4. The automated build/test system is testing against exactly the same code that customers will be using.


5. New computers can build and test everything in 5 minutes, and we're confident enough in the output that we could ship the result to a customer.


6. New computers shouldn't have to go to the Internet to download supporting libraries or documentation, possibly resulting in mis-matched library versions.


7. We can build byte-for-byte* identical copies of installers and development environments from any build we've ever released. (*ignoring inevitable, unimportant details such as "last-modified" file timestamps.)


8. It should be easy (even for an automated script) to tell which files are for production, which are for testing, which are data, and each project's external and internal library dependencies.

I would argue that any development group must to be able to do all these things with confidence if they claim to have a stable product.

It's actually not as hard as it sounds to achieve all of this. But before we got there we of course had to make another mistake.

Bunny Trail: Maven

Maven is an open-source build-management system that supposedly fixes all these problems. It has several parts:

1. A well-defined directory structure clearly separating production code from test code, code from data, build results from original source, and source from different languages.


2. A project definition file including dependencies on all other projects (both in-house projects and third-party libraries).


3. A system for versioning each project separately and making dependencies on specific versions.

We thought Maven would be perfect for the job but we could never make it work. To be fair we were using v1.3 or something and v2.0 is supposed to be much better. On the other hand the only case I know of a commercial product that uses Maven successfully has a full-time developer devoted to build maintenance, so I don't know if it's much better.

The Three Rules

If you've worked with a good build system build, you probably already know that all of these "features" are just manifestations and use-cases of three basic rules:

1. Everyone must have an identical development environment.


2. Files and rules for each project must be consistent.


3. Build, test, install, and dependencies must be machine-readable and runnable unattended.

It's actually not that hard to achieve all of this. Our system is a combination of good ideas from Maven without some of the complexity.

Everywhere the same

Every developer already has an identical development environment when it comes to source code. It's called version control. (Of course you use version control!) All version control systems support some kind of "label" that lets you draw a circle around the versions of source code that go together, e.g. for a public release or nightly build.

The small realization here is that it's not just your own source code that should be checked in -- every external library and executable should go in too. This is the only way to know that everyone is using the same pieces, especially after you decide to change external library versions but then have to go back in time and work on an older build.

In the build system, IDE project, etc., refer to these external libraries with relative paths so you're always including the copies from version control and not something laying around on your hard drive. It's easy to double-check that you've done this right -- use a fresh new machine (you have fresh new machines laying around for testing new installations, right?) and make sure you can compile, test, and build installers from scratch.

At Smart Bear we take it one step further. We check in the entire external library with supporting documentation, example code, and whatever else we can find. With our build integration with Eclipse, I can check out Code Collaborator v1.0 from two years ago and not only build the installers correctly but Eclipse even references the right JavaDoc when I press SHIFT-F2 on a method.

This also ensures that when you do decide to change the version on an external library, everyone is on the same page. Every time we change an external library there's a good chance we've broken something. Our XML-RPC saga is a recent example. If each developer (and the build server) might be using different versions of libraries, we'll have trouble reproducing bugs.

Consistency

Every project has the same layout on disk. We wholesale copied the file layout from Maven, partially because we liked it and partially because we figured that maybe someday we could make Maven work, and it would be easier to convert if our files were already in the right places.

You can look at the Maven project yourself for the details, but it's common-sense techniques such as:

1. All objects created by the build (not in version control) are in one directory (with sub-directories). Easy to "make clean" -- just blow away one directory. Easy to decide what needs to be checked into version control -- everything BUT that directory.


2. Main line source code and data is a completely separate directory from test code and data. Makes it simple for ANT to decide which classes to include in production JAR's and which to include for JUnit tests.


3. Files for WEB-INF or the various standard directories in WAR's are always located in the same place, so JAR's and WAR's can be build consistently.


4. Output of class files, junit class files, Eclipse class files, JavaDoc, JAR's, test coverage reports, etc. are each in a consistent and separate sub-directory in the results directory. This makes it easy for e.g. Eclipse and a command-line build to co-exist without affecting each other.

Unattended Build and Test

Having a continuous build system is important at Smart Bear. We run a full build, unit test, static code analysis, installer run, etc. for every version control check-in. Whoever breaks the build gets shot by Roy's USB-controlled Nerf rocket-launcher.

We use CruiseControl to build automatically and continuously. We don't have nightly builds -- no need when we're building all the time. We build the latest development branch and whichever stable branches are still being actively supported.

Just do it

We have thousands of users at hundred of companies; publishing a DOA maintenance release is not an option. Some users have back-level versions of the software that we have to patch every once in a while. Without a sensible build system, we would never be able to deliver quality releases with confidence.

It's not hard to make a good build system. Learn how to use a good make tool (GNU make and ANT are both excellent and cross-platform), build everything from the command-line, and check everything into version control. Then setting up a nightly or continuous build is just another day's work.