Idiot! Buying SmartBear.com

Being slow cost me $2000. IDIOT!

Yesterday I bought smartbear.com.

Smart Bear started as a sole proprietorship. I switched it to a single-member LLC in Feb 2003, mostly just to see what it was like. (Single-member LLC's and sole proprietorships have identical tax consequences -- none.)

At the time Code Historian and Code Reviewer were shareware and the company name didn't seem to matter. After all, I had http://codehistorian.com, and that's better! So I didn't register smartbear.com.

Let me repeat that. I didn't register smartbear.com.

I could tell you I didn't have any money to spend on such things, which was true, but that's not an excuse, it's cheap. I could tell you I didn't think things would go so well, which was true, but that's not an excuse, it would have been cheap insurance. I could tell you I didn't think anyone was paying attention to little ol' me, which was true, but that's not an excuse, the domain trollers watch the chamber of commerce records and grab anything they can find.

There's no excuse.

By the time I came to my senses, a squatter had taken the domain. For a while the page was just broken. Then the page simply said the domain was for sale. $2000. I ended up getting smartbearsoftware.com.

The way you pronounce "smartbearsoftware.com" over the phone is:

"Smart Bear Software. One word. That's "bear" as in the animal. No, Smart Bear Software. You have to have the software. (pause) Yeah, I know, that's taken by a squatter. (pause) 2000 dollars. (pause) Yeah, we really should. It's the principle of the thing though."

And it really was. I offered the guy $1000. Rejected. I sent a lawyer-letter. Ignored. I asked the lawyer how much it would cost to actually pursue this. Minimum $2500. (Ah, that's how the price was set!)

I pointed out the trademark. He responded by saying it wasn't a registered trademark. I started the process of registering all our trademarks, but years later we're in the last stage of the process but still waiting.

I was goaded into this by the other bears. "Just get the domain. We know it sucks. We know it's unfair. But stop making us type 'software' all the time like chumps."

The final straw was a conversation with Chris Boyd, co-founder of the Best Little ISP on the Planet, Midas Networks. I told him about the domain. "Yeah," he said, "you just have to pay those guys." Chris always tells it straight. That's when I know I was sunk.

So I did it.

DiY Projects & Coffee

Ahh, the in-room coffee pot. Yes I'm going downstairs soon to toast an English muffin, but first coffee...

You know the one about why man-hole covers are round? The makers of this coffee pot didn't get the memo.

Water temperature in the shower ranges from arctic to frigid. Where's that coffee?

The second (potential) customer we're visiting today is having an in-house mutiny. Some want to use our software to do their code reviews, others want to write their own.

There are, of course, reasons to build versus buy, but typically in our space the "build" argument is pretty thin. Any time you have integration (code review system with version control, code review system with issue tracking, code review system with diff tool) you have a never-ending string of little problems.

Take integration with Perforce. When Perforce is about to come out with a new release of their software they send it over to us with a version history so we can check it out. We're able to run our complete unit-test and integration suite on multiple platforms for both client and server, including combinations of earlier versions of both our software and Perforce's.

So by the time the customer installs the new version of Perforce (and possibly the new version of Code Collaborator, in case some changes were necessary), everything works.

If you build it yourself, who is going to baby-sit this and the litany of feature requests? Who's going to field tech support for Safari? Typically one or two people get this role. Call it 1.5 annual developer salaries. Out here (Milpitas), ballpark estimate for this is $250k per year once you include all expenses.

That's $250k per year. Instead of 1/10th that if purchased from us. And the in-house tool will never have as many features (even creature-features, not just "stuff we don't need anyway") as we will with our small but determined army of top-notch developers.

The usual counter-argument is "We'll get exactly what we want." More accurately, you get exactly what you build. You know that what users want and what features are required to let them work effectively are two different things. You know that it takes time and iteration to get there. This will be a full-time software project like anything else, otherwise it's not going to be good.

So we'll see how this demo goes. It's always interesting to hear exactly what someone wants. Often it shows the way to great new features. Then everyone's happy.

Out of coffee...

Black Shirts

Always get black shirts for road trips.

The Smart Bear shirts are all black. Why?

• Coffee doesn't show up on black shirts.
• Spaghetti doesn't show up on black shirts.
• Everyone looks good in black.
• Black doesn't show wrinkles as much.
• Black makes you look thinner. (Thanks Penny!)
  

It is true that white dirt sticks to black while black dirt sticks to white, but white food doesn't stain.

We're bringing a bunch of shirts with us on our latest road trip. Giving away shirts is cliche but people still like them, especially developers who aren't allowed to visit tradeshows anymore. Also we give away real shirts with collars. They're actually nice.

Road Trip! Product Planning by Customers

We're going on another road trip. This time we're touring Silicon Valley. How original!

18 meetings in 8 business days. This is going to be fun...
Half of the meetings are with existing customers. We'll be showing off all the cool new Collab v2.0 features and demo'ing a brand new product that we just filed a patent for. But more on that in another post.

It's vital to keep in close contact with your customers. Your sales force will ignore customers as soon as the money is handed over (until a year passes and they want maintenance revenue). This is fine for the sales guys -- after all their job is to find fresh meat -- but it's not OK for customer service or for product development.

It's your existing customers who know what needs to be built next. They know which parts of the software are most annoying. They think of cool new features that would genuinely help their lives, not just put up pretty bullets on data sheets.

Sure they send complaints and ideas to support@. But that's nothing compared to what happens when you meet face-to-face.

I'll be walking down a row of cubicles and our host will causally announce that "The Smart Bears are here." I'll see a name I recognize and pop in. "Oh hey! I've been meaning to tell you guys, it really sucks that the search results page won't show more than N entries. I have to do this thing every release where I .... and it would save me hours if only it would ....."

Why didn't that guy email us? Doesn't matter, the point is he didn't. But when you're there everything's different.

Sure you can't implement every suggestion. And some will confuse more than help. But this is the only way I know to build software that people actually want to use. You can't compile it from stray emails.

Some people swear by Customer Planning Sessions. You can take managerial representatives from 10 companies and fly them in for two days of exciting meetings about what they want to see and what you're planning to deliver next. Seems fine, but my experience is that the folks that show up are rarely the folks that actually use the software. You certainly get great information about what the managers need -- and that's 1/2 of what we focus on with Code Collaborator -- but the other 1/2 are the developers who stare at the screen every day and you have to make them happy too.

It is possible to give managers reports, metrics, dashboards, and all that other process/control/measurement stuff and yet also have a pleasant, helpful, useful environment for ends users too. You just have to listen!

I thought XML-RPC was supposed to be easy

I love XML-RPC. The devil is in the libraries...

XML-RPC is the poor man's Web Services: A simple protocol for sending structured, function-like RPC messages and receiving a structured response. And really simple -- not simple as in Simple Network Management Protocol -- simple as in the spec fits on a couple of sheets of paper. It works on all platforms and languages.

We use XML-RPC for all our client/server communication. We use Apache's Java XML-RPC library on both ends.

The problems started with OS/390. (Surprise.) A customer was trying to run our client on a Linux partition and of course the default character set was EBCDIC. We don't officially support the Mainframe or EBCDIC, but since Java is write-once-debug-everywhere and since IBM is fully committed to both Linux and OS/390, we shouldn't have any problems, right?

Anyway, the problem wasn't with our code -- the communications were properly encoded. The problem was that Apache's XML-RPC library used the system default character set when creating the HTTP header instead of hard-coding ASCII as required by the MIME spec since... well since OS/390 was invented.

Turns out it was a "known issue." OK, it's open-source, we'll just modify it. We did.

That was library v2. Then v3 finally exited alpha with features we wanted, especially "streaming mode." (Yes, in v2 you had to build the entire request/response in memory before it would be transmitted. Yes this gave us problems transporting really large files for review.)

Also v3 supposedly fixed the OS/390 encoding problem! We can go back to a clean library instead of our hacked version.

Not so fast. In v2 we were allowed define functions that return nothing (void). The XML-RPC protocol requires that every function return a value. v2 glossed over this and all was well.
When we upgraded to v3 everything broke. Now it's not legal, and enforced. No problem, we'll just change the interface! Wrong, then old clients break. And new clients can't talk to old servers because the client-side v3 XML-RPC library complains about not having a return value.

The best we could do is make a new XML-RPC namespace (now we're up to "ccollab3") where "void" functions now return a dummy value. New clients use the new namespace; the old namespace is also still available. Old clients work on new servers, but new clients cannot talk to old servers.

And now all changes to the XML-RPC interface get a pass using the XML-RPC spec, just in case.

Team-Building for the Cold, Dark, and Alone

Teams are for sissies!

"Team building" is a load of bull. It's happy-talk that changes nothing. It's laughable trust exercises. It stifles the best developers to serve mediocrity.

Right?

This essay isn't rah-rah go team go. Suspend your judgment for five minutes.

I'll give you the punch line first. Peer code review not only finds bugs, it creates an environment where developers work together instead of in parallel. It's team-building in the real sense -- people helping each other to achieve something greater than each could achieve alone.

Think you have a team? You don't have team.

I'll bet most of the code in your application has been seen by only one person. Think about that for three seconds. It's a frightening thought. Teams supposedly work on projects together. Do you really have a team?

Developers write code while squirreled away behind doors and headphones. After the architecture and interface discussions are over, code is written in isolation. Does that sound like a team to you?

New hires want to get into code fast and get their hands dirty but they need a safety net to make sure nothing bad happens. How do your developers handle this? Isn't "teaching and supporting" a team activity?

"Two or more draft animals used to pull a vehicle or farm implement." That's how the American Heritage Dictionary defines "team." This is the extent of "teamwork" in most software development organizations too. But being a team doesn't mean we're merely marching in the same direction.

You're not a team just because you know everyone's name and you eat lunch together. That's just being acquainted. Real teams actually work together. Let's see how peer code review can help build a real team.

One is the loneliest number

"It's scary how much code I've written that no one else has ever seen." I was told this by a developer who had just started using our software to do peer reviews. "The worst part is, I know it can't all be right."

Yes, you should be scared. The average professional novelist makes 9 mistakes per page. No surprise; that's why there are editors. In fact the 2000 Census showed that in America we have more professional editors than professional writers!

So why do we expect developers to write reams of technical code without the safety net of an editor? Aren't we guaranteeing errors?

A startling amount of code has a bus number of one. Your "bus number" is the number of people who have to get hit by a bus until no one is left who understands the code. A software process is easily derailed if a significant amount of code has a bus number of one. If that someone is sick, leaves the company, or really is hit by a bus, parts of a project can stall or might be implemented incorrectly.

Code review provides a safety net for both developers and managers. Managers like the project stability that comes from having a bus number of two or more. Developers like the comfort of knowing that someone has their back. In a real team, people look out for one other.

"Those guys suck."

It was a large software company. A small overworked software group in Austin had been asking for additional developers; finally their wish was granted in the form of 20 developers in India.

The group in Austin was pissed off. They were responsible for running the remote group and incorporating their code into the local base. "Those guys suck." (Yes, that's an exact quote.) "They don't know how to write code. And they don't do it the way we ask them to. And they don't write comments. Forget about comments in English, they write no comments at all!"

The Transformation

The Austin group started using our code review software to check everything being done in India. This doesn't just mean looking at code -- the software facilitates actually communicating on individual lines of code and having a two-way conversation like a chat room or a newsgroup.

We visited them four weeks after the reviews started. "They're pretty cool" said the same developer who had told us they "suck" only 29 days earlier. "They're doing a good job now. They just needed a little direction. We're actually getting a lot of good code written."

How is it possible that developers who "suck" suddenly became "pretty cool?" It turns out that the Indian developers were inexperienced. They were smart and wanted to learn; they were just green. Once the two factions had a mechanism for easily talking to each other about code, suddenly the knowledge transfer could begin.

It wasn't enough when they were only communicating through project specs and long emails about how Austin expected them to work. They needed to talk about specific pieces of code and they needed a process that let this happen naturally.

Emergent Behavior

Once the code review process was in place, the rest literally worked itself out. The developers in India became far more productive and the developers in Austin became happy. The entire group produced far more working code without increasing the budget.

The developers organized themselves into a team. All they needed was an environment of communication and a tool to make it easy. They taught each other without managerial pressure or rules or strict process controls.

Working together, teaching and sharing, being happier, all without being told... sounds like a team!

Teamwork, for real though

Software architecture and design is typically coordinated and even invented in meetings. We take it for granted that these activities are enhanced by collaboration.

Writing the software is no different. You may not need a formal meeting or the time investment of pair-programming, but it does make sense to give your developers "editors." Besides the obvious bug-removal benefits of code review, you'll turn your isolated, individualistic developers into a true team.

I hope you teamwork-disbelievers out there will at least give code review a try. There are lots of ways to do it; just pick one that sounds doable and try it for a month. If you don't find lots of defects and you don't see developers teaching each other, let me know.

Journey to a Build

Believe it our not, our customers often ask about our build system. Maybe it's because they're all developers themselves.

Build systems are rarely designed properly from the start. Most I've seen are loose assemblages of scripts with an out-dated step-by-step README telling you how to install all the executables and libraries. Ours was.

Even the shortest journey starts with an embarrassing moment.

I knew we had to fix the build system when Brandon first arrived. He watched as Eric fixed an obscure (but for one customer, critical) bug in the public release of Code Reviewer. Eric smashed Control-Shift-B to build. (You'd think Visual Studio would have an easier mnemonic for the most common operation.) NUnit said the unit tests still pass. Then he checked in the fix.

So far so good. Then Eric turned to Teddy The Build Machine for the installer. Brandon asked Eric why he couldn't just build the installer himself. Eric didn't know, sometimes it didn't seem to work. One time the installer didn't link to the right version of .NET. Sometimes one of the sub-libraries isn't built but you can't tell until you've actually installed it.

Brandon looked at me. "I don't know," I said, "I haven't been able to make a proper build since we officially supported .NET 2.0."

Brandon didn't have to say anything. We knew we were in trouble.

Mapping it out

We started with the Java projects (Code Collaborator). We decided that we wanted all of the following "features" in our shiny new build system:

1. All installers are built properly and with the right library versions, even when we give 1-off builds to customers to test a fix, no matter whose workstation built the installers.
   
   
2. The build (compile, test, install) must be able to run completely unattended, e.g. for official builds, nightly builds, continuous builds.


3. Everyone is debugging against exactly the same code that customers will be using.


4. The automated build/test system is testing against exactly the same code that customers will be using.


5. New computers can build and test everything in 5 minutes, and we're confident enough in the output that we could ship the result to a customer.


6. New computers shouldn't have to go to the Internet to download supporting libraries or documentation, possibly resulting in mis-matched library versions.


7. We can build byte-for-byte* identical copies of installers and development environments from any build we've ever released. (*ignoring inevitable, unimportant details such as "last-modified" file timestamps.)


8. It should be easy (even for an automated script) to tell which files are for production, which are for testing, which are data, and each project's external and internal library dependencies.

I would argue that any development group must to be able to do all these things with confidence if they claim to have a stable product.

It's actually not as hard as it sounds to achieve all of this. But before we got there we of course had to make another mistake.

Bunny Trail: Maven

Maven is an open-source build-management system that supposedly fixes all these problems. It has several parts:

1. A well-defined directory structure clearly separating production code from test code, code from data, build results from original source, and source from different languages.


2. A project definition file including dependencies on all other projects (both in-house projects and third-party libraries).


3. A system for versioning each project separately and making dependencies on specific versions.

We thought Maven would be perfect for the job but we could never make it work. To be fair we were using v1.3 or something and v2.0 is supposed to be much better. On the other hand the only case I know of a commercial product that uses Maven successfully has a full-time developer devoted to build maintenance, so I don't know if it's much better.

The Three Rules

If you've worked with a good build system build, you probably already know that all of these "features" are just manifestations and use-cases of three basic rules:

1. Everyone must have an identical development environment.


2. Files and rules for each project must be consistent.


3. Build, test, install, and dependencies must be machine-readable and runnable unattended.

It's actually not that hard to achieve all of this. Our system is a combination of good ideas from Maven without some of the complexity.

Everywhere the same

Every developer already has an identical development environment when it comes to source code. It's called version control. (Of course you use version control!) All version control systems support some kind of "label" that lets you draw a circle around the versions of source code that go together, e.g. for a public release or nightly build.

The small realization here is that it's not just your own source code that should be checked in -- every external library and executable should go in too. This is the only way to know that everyone is using the same pieces, especially after you decide to change external library versions but then have to go back in time and work on an older build.

In the build system, IDE project, etc., refer to these external libraries with relative paths so you're always including the copies from version control and not something laying around on your hard drive. It's easy to double-check that you've done this right -- use a fresh new machine (you have fresh new machines laying around for testing new installations, right?) and make sure you can compile, test, and build installers from scratch.

At Smart Bear we take it one step further. We check in the entire external library with supporting documentation, example code, and whatever else we can find. With our build integration with Eclipse, I can check out Code Collaborator v1.0 from two years ago and not only build the installers correctly but Eclipse even references the right JavaDoc when I press SHIFT-F2 on a method.

This also ensures that when you do decide to change the version on an external library, everyone is on the same page. Every time we change an external library there's a good chance we've broken something. Our XML-RPC saga is a recent example. If each developer (and the build server) might be using different versions of libraries, we'll have trouble reproducing bugs.

Consistency

Every project has the same layout on disk. We wholesale copied the file layout from Maven, partially because we liked it and partially because we figured that maybe someday we could make Maven work, and it would be easier to convert if our files were already in the right places.

You can look at the Maven project yourself for the details, but it's common-sense techniques such as:

1. All objects created by the build (not in version control) are in one directory (with sub-directories). Easy to "make clean" -- just blow away one directory. Easy to decide what needs to be checked into version control -- everything BUT that directory.


2. Main line source code and data is a completely separate directory from test code and data. Makes it simple for ANT to decide which classes to include in production JAR's and which to include for JUnit tests.


3. Files for WEB-INF or the various standard directories in WAR's are always located in the same place, so JAR's and WAR's can be build consistently.


4. Output of class files, junit class files, Eclipse class files, JavaDoc, JAR's, test coverage reports, etc. are each in a consistent and separate sub-directory in the results directory. This makes it easy for e.g. Eclipse and a command-line build to co-exist without affecting each other.

Unattended Build and Test

Having a continuous build system is important at Smart Bear. We run a full build, unit test, static code analysis, installer run, etc. for every version control check-in. Whoever breaks the build gets shot by Roy's USB-controlled Nerf rocket-launcher.

We use CruiseControl to build automatically and continuously. We don't have nightly builds -- no need when we're building all the time. We build the latest development branch and whichever stable branches are still being actively supported.

Just do it

We have thousands of users at hundred of companies; publishing a DOA maintenance release is not an option. Some users have back-level versions of the software that we have to patch every once in a while. Without a sensible build system, we would never be able to deliver quality releases with confidence.

It's not hard to make a good build system. Learn how to use a good make tool (GNU make and ANT are both excellent and cross-platform), build everything from the command-line, and check everything into version control. Then setting up a nightly or continuous build is just another day's work.