Set My Data Free

The title of this post comes from a recent post by Tim Bray. Tim was discussing "software as a service", and making the point that for outsourced software services, open source code isn't the issue, open data is the issue. You need a guarantee that you can get your data back when you stop using the service, and you also probably want a guarantee that the service won't keep any of your data for its own future use.

In particular, Tim wrote "information outlives software and transcends software and is more valuable than software". That's a principle that I find myself constantly espousing as well. At the start of 2006, I joined a new partnership, Miley Watts LLP. On the home page of our Web site, the thing you see in big letters is

Your data will outlive your applications

That is because we take a very data-focussed approach to software architecture. The need for portability of data is obvious when dealing with software services, but it's just as important for enterprise and institutional applications as well. Unfortunately, a lot of significant applications are still written so that the only access to the data is via one particular application. This works in the short term, but it gets expensive once you outgrow your application (and that tends to happen these days in no more than a couple of years, sometimes less).

Going back to Tim, I remember the closing speech he gave at the XML 2004 conference. He was recounting how on some development projects he had done in the past, he had to make a real effort to get shown the actual data that the new application was required to process. People were very ready to write descriptive documents of what was required, but without seeing the actual data, Tim knew (very wisely) that he wouldn't know whether the data really matched the descriptions. He also wouldn't have a feel for what patterns of usage were most common, and deserving of more attention during development.

There is an interesting cultural difference I find between people who do a lot of work with XML, and people who work primarily with object-oriented programming languages (Java, C#, C++, etc.). When you work with XML, it would be unusual to just write Schemas without creating XML documents to match them. In creating XML documents, you are looking at actual data, not just at a theoretical structure. Database developers also spend a lot of time looking at actual data. [Beware, though, of completely made-up data that fits a Schema, but doesn't reflect your production data. That can have you optimising your code for all the wrong things.]

However, I know from my time as a Java developer that people spend hours and hours writing object-oriented code, but comparatively little time looking at data. There a few reasons for that. Debuggers are the way developers often interact with data, and a debugger only gives you a tiny window in the data that is in memory. There is also a paucity of tools for creating and editing in-memory data for applications (without having to write a database/XML/file layer in your code). This seems to me to have encouraged a style of development that is agnostic to the actual data. Some applications really suffer from this, especially when testing and debugging end up taking vastly longer than was allowed for in the project plan.

Data matters. It shouldn't be an afterthought. It will outlive your applications.