I thought XML-RPC was supposed to be easy

I love XML-RPC. The devil is in the libraries...

XML-RPC is the poor man's Web Services: A simple protocol for sending structured, function-like RPC messages and receiving a structured response. And really simple -- not simple as in Simple Network Management Protocol -- simple as in the spec fits on a couple of sheets of paper. It works on all platforms and languages.

We use XML-RPC for all our client/server communication. We use Apache's Java XML-RPC library on both ends.

The problems started with OS/390. (Surprise.) A customer was trying to run our client on a Linux partition and of course the default character set was EBCDIC. We don't officially support the Mainframe or EBCDIC, but since Java is write-once-debug-everywhere and since IBM is fully committed to both Linux and OS/390, we shouldn't have any problems, right?

Anyway, the problem wasn't with our code -- the communications were properly encoded. The problem was that Apache's XML-RPC library used the system default character set when creating the HTTP header instead of hard-coding ASCII as required by the MIME spec since... well since OS/390 was invented.

Turns out it was a "known issue." OK, it's open-source, we'll just modify it. We did.

That was library v2. Then v3 finally exited alpha with features we wanted, especially "streaming mode." (Yes, in v2 you had to build the entire request/response in memory before it would be transmitted. Yes this gave us problems transporting really large files for review.)

Also v3 supposedly fixed the OS/390 encoding problem! We can go back to a clean library instead of our hacked version.

Not so fast. In v2 we were allowed define functions that return nothing (void). The XML-RPC protocol requires that every function return a value. v2 glossed over this and all was well.
When we upgraded to v3 everything broke. Now it's not legal, and enforced. No problem, we'll just change the interface! Wrong, then old clients break. And new clients can't talk to old servers because the client-side v3 XML-RPC library complains about not having a return value.

The best we could do is make a new XML-RPC namespace (now we're up to "ccollab3") where "void" functions now return a dummy value. New clients use the new namespace; the old namespace is also still available. Old clients work on new servers, but new clients cannot talk to old servers.

And now all changes to the XML-RPC interface get a pass using the XML-RPC spec, just in case.