Team-Building for the Cold, Dark, and Alone

Teams are for sissies!

"Team building" is a load of bull. It's happy-talk that changes nothing. It's laughable trust exercises. It stifles the best developers to serve mediocrity.

Right?

This essay isn't rah-rah go team go. Suspend your judgment for five minutes.

I'll give you the punch line first. Peer code review not only finds bugs, it creates an environment where developers work together instead of in parallel. It's team-building in the real sense -- people helping each other to achieve something greater than each could achieve alone.

Think you have a team? You don't have team.

I'll bet most of the code in your application has been seen by only one person. Think about that for three seconds. It's a frightening thought. Teams supposedly work on projects together. Do you really have a team?

Developers write code while squirreled away behind doors and headphones. After the architecture and interface discussions are over, code is written in isolation. Does that sound like a team to you?

New hires want to get into code fast and get their hands dirty but they need a safety net to make sure nothing bad happens. How do your developers handle this? Isn't "teaching and supporting" a team activity?

"Two or more draft animals used to pull a vehicle or farm implement." That's how the American Heritage Dictionary defines "team." This is the extent of "teamwork" in most software development organizations too. But being a team doesn't mean we're merely marching in the same direction.

You're not a team just because you know everyone's name and you eat lunch together. That's just being acquainted. Real teams actually work together. Let's see how peer code review can help build a real team.

One is the loneliest number

"It's scary how much code I've written that no one else has ever seen." I was told this by a developer who had just started using our software to do peer reviews. "The worst part is, I know it can't all be right."

Yes, you should be scared. The average professional novelist makes 9 mistakes per page. No surprise; that's why there are editors. In fact the 2000 Census showed that in America we have more professional editors than professional writers!

So why do we expect developers to write reams of technical code without the safety net of an editor? Aren't we guaranteeing errors?

A startling amount of code has a bus number of one. Your "bus number" is the number of people who have to get hit by a bus until no one is left who understands the code. A software process is easily derailed if a significant amount of code has a bus number of one. If that someone is sick, leaves the company, or really is hit by a bus, parts of a project can stall or might be implemented incorrectly.

Code review provides a safety net for both developers and managers. Managers like the project stability that comes from having a bus number of two or more. Developers like the comfort of knowing that someone has their back. In a real team, people look out for one other.

"Those guys suck."

It was a large software company. A small overworked software group in Austin had been asking for additional developers; finally their wish was granted in the form of 20 developers in India.

The group in Austin was pissed off. They were responsible for running the remote group and incorporating their code into the local base. "Those guys suck." (Yes, that's an exact quote.) "They don't know how to write code. And they don't do it the way we ask them to. And they don't write comments. Forget about comments in English, they write no comments at all!"

The Transformation

The Austin group started using our code review software to check everything being done in India. This doesn't just mean looking at code -- the software facilitates actually communicating on individual lines of code and having a two-way conversation like a chat room or a newsgroup.

We visited them four weeks after the reviews started. "They're pretty cool" said the same developer who had told us they "suck" only 29 days earlier. "They're doing a good job now. They just needed a little direction. We're actually getting a lot of good code written."

How is it possible that developers who "suck" suddenly became "pretty cool?" It turns out that the Indian developers were inexperienced. They were smart and wanted to learn; they were just green. Once the two factions had a mechanism for easily talking to each other about code, suddenly the knowledge transfer could begin.

It wasn't enough when they were only communicating through project specs and long emails about how Austin expected them to work. They needed to talk about specific pieces of code and they needed a process that let this happen naturally.

Emergent Behavior

Once the code review process was in place, the rest literally worked itself out. The developers in India became far more productive and the developers in Austin became happy. The entire group produced far more working code without increasing the budget.

The developers organized themselves into a team. All they needed was an environment of communication and a tool to make it easy. They taught each other without managerial pressure or rules or strict process controls.

Working together, teaching and sharing, being happier, all without being told... sounds like a team!

Teamwork, for real though

Software architecture and design is typically coordinated and even invented in meetings. We take it for granted that these activities are enhanced by collaboration.

Writing the software is no different. You may not need a formal meeting or the time investment of pair-programming, but it does make sense to give your developers "editors." Besides the obvious bug-removal benefits of code review, you'll turn your isolated, individualistic developers into a true team.

I hope you teamwork-disbelievers out there will at least give code review a try. There are lots of ways to do it; just pick one that sounds doable and try it for a month. If you don't find lots of defects and you don't see developers teaching each other, let me know.

Journey to a Build

Believe it our not, our customers often ask about our build system. Maybe it's because they're all developers themselves.

Build systems are rarely designed properly from the start. Most I've seen are loose assemblages of scripts with an out-dated step-by-step README telling you how to install all the executables and libraries. Ours was.

Even the shortest journey starts with an embarrassing moment.

I knew we had to fix the build system when Brandon first arrived. He watched as Eric fixed an obscure (but for one customer, critical) bug in the public release of Code Reviewer. Eric smashed Control-Shift-B to build. (You'd think Visual Studio would have an easier mnemonic for the most common operation.) NUnit said the unit tests still pass. Then he checked in the fix.

So far so good. Then Eric turned to Teddy The Build Machine for the installer. Brandon asked Eric why he couldn't just build the installer himself. Eric didn't know, sometimes it didn't seem to work. One time the installer didn't link to the right version of .NET. Sometimes one of the sub-libraries isn't built but you can't tell until you've actually installed it.

Brandon looked at me. "I don't know," I said, "I haven't been able to make a proper build since we officially supported .NET 2.0."

Brandon didn't have to say anything. We knew we were in trouble.

Mapping it out

We started with the Java projects (Code Collaborator). We decided that we wanted all of the following "features" in our shiny new build system:

1. All installers are built properly and with the right library versions, even when we give 1-off builds to customers to test a fix, no matter whose workstation built the installers.
   
   
2. The build (compile, test, install) must be able to run completely unattended, e.g. for official builds, nightly builds, continuous builds.


3. Everyone is debugging against exactly the same code that customers will be using.


4. The automated build/test system is testing against exactly the same code that customers will be using.


5. New computers can build and test everything in 5 minutes, and we're confident enough in the output that we could ship the result to a customer.


6. New computers shouldn't have to go to the Internet to download supporting libraries or documentation, possibly resulting in mis-matched library versions.


7. We can build byte-for-byte* identical copies of installers and development environments from any build we've ever released. (*ignoring inevitable, unimportant details such as "last-modified" file timestamps.)


8. It should be easy (even for an automated script) to tell which files are for production, which are for testing, which are data, and each project's external and internal library dependencies.

I would argue that any development group must to be able to do all these things with confidence if they claim to have a stable product.

It's actually not as hard as it sounds to achieve all of this. But before we got there we of course had to make another mistake.

Bunny Trail: Maven

Maven is an open-source build-management system that supposedly fixes all these problems. It has several parts:

1. A well-defined directory structure clearly separating production code from test code, code from data, build results from original source, and source from different languages.


2. A project definition file including dependencies on all other projects (both in-house projects and third-party libraries).


3. A system for versioning each project separately and making dependencies on specific versions.

We thought Maven would be perfect for the job but we could never make it work. To be fair we were using v1.3 or something and v2.0 is supposed to be much better. On the other hand the only case I know of a commercial product that uses Maven successfully has a full-time developer devoted to build maintenance, so I don't know if it's much better.

The Three Rules

If you've worked with a good build system build, you probably already know that all of these "features" are just manifestations and use-cases of three basic rules:

1. Everyone must have an identical development environment.


2. Files and rules for each project must be consistent.


3. Build, test, install, and dependencies must be machine-readable and runnable unattended.

It's actually not that hard to achieve all of this. Our system is a combination of good ideas from Maven without some of the complexity.

Everywhere the same

Every developer already has an identical development environment when it comes to source code. It's called version control. (Of course you use version control!) All version control systems support some kind of "label" that lets you draw a circle around the versions of source code that go together, e.g. for a public release or nightly build.

The small realization here is that it's not just your own source code that should be checked in -- every external library and executable should go in too. This is the only way to know that everyone is using the same pieces, especially after you decide to change external library versions but then have to go back in time and work on an older build.

In the build system, IDE project, etc., refer to these external libraries with relative paths so you're always including the copies from version control and not something laying around on your hard drive. It's easy to double-check that you've done this right -- use a fresh new machine (you have fresh new machines laying around for testing new installations, right?) and make sure you can compile, test, and build installers from scratch.

At Smart Bear we take it one step further. We check in the entire external library with supporting documentation, example code, and whatever else we can find. With our build integration with Eclipse, I can check out Code Collaborator v1.0 from two years ago and not only build the installers correctly but Eclipse even references the right JavaDoc when I press SHIFT-F2 on a method.

This also ensures that when you do decide to change the version on an external library, everyone is on the same page. Every time we change an external library there's a good chance we've broken something. Our XML-RPC saga is a recent example. If each developer (and the build server) might be using different versions of libraries, we'll have trouble reproducing bugs.

Consistency

Every project has the same layout on disk. We wholesale copied the file layout from Maven, partially because we liked it and partially because we figured that maybe someday we could make Maven work, and it would be easier to convert if our files were already in the right places.

You can look at the Maven project yourself for the details, but it's common-sense techniques such as:

1. All objects created by the build (not in version control) are in one directory (with sub-directories). Easy to "make clean" -- just blow away one directory. Easy to decide what needs to be checked into version control -- everything BUT that directory.


2. Main line source code and data is a completely separate directory from test code and data. Makes it simple for ANT to decide which classes to include in production JAR's and which to include for JUnit tests.


3. Files for WEB-INF or the various standard directories in WAR's are always located in the same place, so JAR's and WAR's can be build consistently.


4. Output of class files, junit class files, Eclipse class files, JavaDoc, JAR's, test coverage reports, etc. are each in a consistent and separate sub-directory in the results directory. This makes it easy for e.g. Eclipse and a command-line build to co-exist without affecting each other.

Unattended Build and Test

Having a continuous build system is important at Smart Bear. We run a full build, unit test, static code analysis, installer run, etc. for every version control check-in. Whoever breaks the build gets shot by Roy's USB-controlled Nerf rocket-launcher.

We use CruiseControl to build automatically and continuously. We don't have nightly builds -- no need when we're building all the time. We build the latest development branch and whichever stable branches are still being actively supported.

Just do it

We have thousands of users at hundred of companies; publishing a DOA maintenance release is not an option. Some users have back-level versions of the software that we have to patch every once in a while. Without a sensible build system, we would never be able to deliver quality releases with confidence.

It's not hard to make a good build system. Learn how to use a good make tool (GNU make and ANT are both excellent and cross-platform), build everything from the command-line, and check everything into version control. Then setting up a nightly or continuous build is just another day's work.

Overlay Spam

There's a new breed of spam that's beating the gmail filter 10% of the time.

The idea is to take some "real" text and make that the body of the email. The text probably comes from sites like Project Gutenberg. Then emit an image positioned over the text with opaque background color. The image contains the real advertisement.

The "good text" makes it past spam filters, then your client overlays that with the image.

UPDATE Sep 5 06: An article on LifeHacker how to filter these out of gmail.

The Case for Peer Code Review

We just wrote an article about why anyone should care about doing code reviews:

http://articles.smartbearsoftware.com/Case-For-Code-Review

This is the first of an 8-part article series on peer code review we're doing for CM Crossroads.

About Jason Cohen

Thanks for visiting my blog about marketing and small business.  Gear-head geeks like me tend to think these things are inscrutable, best left to MBAs and right-brain types.  I'm here to tell you that's not true!

I'm the founder of Smart Bear Software, maker of Code Collaborator, the world's most popular tool for peer code review and recent winner of the Jolt Award.

I took Smart Bear from start to multiple millions in revenue and 50% profit margin without debt or VC.  I was also a founding member of ITWatchdogs, another bootstrapped startup, and other sundries.

I'm the author of Best Kept Secrets of Peer Code Review, the most popular book (27,000 copies) on modern, lightweight methods for doing peer code review effectively without everyone hating life.

I live in Austin, Texas with my wonderful wife.

Email: esmartbear at gmail dot com