kontrawize

Sean McGrath and Data Checking, part 2

John Cowan asked why I introduced a distinction between "business rules" and "technical rules" in my post "Sean McGrath and Data Checking". The reason is because I've often had to work with business analysts on business rules and the relationship of those rules to what kinds of checking can be applied to XML business messages using schemas.

In my experience, business analysts do not accept that any rule that is checked by a schema is a business rule (which is the definition of "business rule" that some people with more purely technical backgrounds like to use). Rather, they take the opinion (which I don't oppose) that business rules are things that they, as business analysts, decide upon. Any other rules that are imposed on the messages aren't business rules, in their view.

John gave an example, asking whether it's a business or a technical rule that says the web site must be in HTML. Let's look at that question. When you put up a web site, you typically do so to provide information to people with web browsers. So the business rule is that your web site must publish information in a format that can be displayed by web browsers. A further refinement is that you are likely to want to target a certain group of people with web browsers (from few to many to all possible users), and so you need to publish the information in a format that will be sufficiently convenient for most or all of your target audience to read. You may also have business rules around needing to use technologies in which your staff already have sufficient training.

So in some cases, the choice of HTML for your web site is forced upon you by the sum of one or more business rules. That said, we all know that there are web sites based on Flash rather than HTML, or even XML with CSS rather than HTML, so some people will have the freedom to make a technical choice which is not constrained by a business rule. That kind of choice is a technical rule, as I see it.

Another example is the question of how you define the order of content of a complex type in a W3C XML Schema. There are often business rules that tell you what information you need to have, and there may also be business rules around how you name the elements and attributes. However, whether you impose a strict sequence on the elements in that complex type (as is often done) is typically a technical rule for data-oriented XML (i.e. the business analysts don't typically care what order the data is in), but a business rule for document-oriented XML (where order is important).

This is all nomenclature of course, it's all a matter of jargon, so the question isn't one of what's the right or wrong way to classify business/technical rules. Rather, the question is what's the most helpful way to do so, what makes the most sense to the most people. I think that talking about both business and technical rules works better than talking about business rules alone, even though there are always going to be grey (gray) areas in deciding what is a business or technical rule.

You can learn a lot about what your real business rules are when you try to decide which rules are technical only or not. I've had to do that in the past, and it helps you to realise earlier rather than later when something you thought was a technical rule actually impacts your ability to satisfy the actual business requirements (as opposed to just the ones that were listed for you at the start of a project).

Opera browser dies, but is VNC involved?

I wrote earlier in "Opera browser dies many times each day" that recent versions of the Opera web browser and e-mail client have been crashing on me unacceptably frequently. However, one good thing about Opera is that you can buy a premium support licence for it. I have one because I bought Opera back in the days before they started giving it away, so I contacted them to update my details and then wrote to them.

I'm very happy to say they wrote back today (good service!), and asked me if I was using VNC. Funny thing is, I didn't mention to them that I'm using VMware (which uses VNC to connect from the VMware console to individual virtual machines), so they had no way of knowing that I'm effectively using VNC, so it must be a known issue.

I'm testing now whether setting a special environment variable helps or not. In any case, I'm very glad to have some idea now of what's likely to be causing the problem. For most of you, who won't be using VNC with Opera, you probably have nothing to worry about. I'll let you know if and when Opera support comes up with a definite diagnosis and/or solution.

Sean McGrath and Data Checking

Following on from my earlier post "Sean McGrath and Data Pipelines", I mentioned that Sean and I discussed what kinds of processing people need to do with XML (or other data) in practice. For both of us, a key part of this is data validation and business rules checking.

Now, the terms "business rules" is used by different people to mean different things. For some people, a business rule is any rule of any kind that applies to some data in order to decide whether that data is correct, suitable, etc. for a particular use. For other people, a business rule is a rule that comes directly from a business requirement, and this is the definition I will use here. If you use this latter definition, then you also need to introduce the concept of "technical rules", which are rules that enforce particular technical/structural constraints on the data for the purposes of defining a single way of doing things. If a rule is a technical rule, you should be able to change it without having any effect on whether you are satisfying your business requirements or not. If you can't do that, it isn't a technical rule (or at least, not a pure technical rule).

Back to data validation, there is more to XML validation than just checking an XML document against a schema. In the bigger picture, any schema will check some of your business and technical rules, but it's unlikely to check all of them. So how do you check all of them?

Taking one step back, the XML specification doesn't require you to have schemas for your XML documents. You could write your applications to process any arbitrary XML that they are given (and some software is indeed written to work this way). With this approach, all business/technical rules checking is done within your application.

However, what we find in practice is that people usually use a schema when processing XML. Why? Well, I would suggest a few key reasons:

• it allows you to move some of the rules checking out of your application code, which simplifies the application's code, making it easier to code, debug, etc.;
• it makes it straightforward to apply some of your business/technical rules (those a schema can check) to any application without recoding;
• it allows users who are writing or debugging XML documents (e.g. writing test data or debugging production data) to use schema-sensitive tools, like XML editors, which highlight errors and prompt users with what new XML is allowed at each point in the document. This just speeds the data creation and debugging processes as compared to, for example, examining plain text data in a plain text editor.

Once you start doing some of your data validation with a schema, you have introduced a layered approach to data validation. Some validation is done in a layer outside of your application's core code, some is done inside your application's core code. Since most people are already using layered validation for XML data, it's worth asking the question of whether there is a better layering approach than the default one (1 schema layer + application code) that people mostly use now. I'll cover that in a future post.

Opera browser dies many times each day

I've been using the Opera web browser and e-mail client for many years now, and been very happy with it. However, the latest versions, 9.10 and 9.20, have been a nightmare.

Very frequently, when I visit particular Web pages, it dies. It isn't obvious why, although visiting the same page kills it again. Equally, sometimes looking at an e-mail also kills it dead immediately.

What's worse, I can't even check Opera's own forums for information about it, because those pages are killing it too. I don't know what has happened to Opera's quality control in recent times, but these last couple of versions have been a very poor advertisement (to me) for Opera's ability to put out stable, reliable software.

Sean McGrath and Data Pipelines

I had the privilege of spending the morning with Sean McGrath of Propylon. Sean is definitely one of the thought-leaders in the XML world, and has spent a lot of time thinking about XML processing and XML pipelines. [In spite of the XML focus here, many of the same principles apply to data processing and data pipelines in general.]

A few years back Sean gave us XPipe, a pipeline-style approach to XML processing. Today, I was talking to Sean about XProc, the XML pipeline language that the W3C is developing, and pipelines in general.

Why pipelines? Well, as Sean points out, the real questions that we need to be asking are just what processing do people need to do with XML (or other data) in practice, and how can we make that processing scalable, and make the development of the processing code manageable and cost-effective? The idea behind pipelines is that you break down a big processing task into a series of smaller, simpler pipeline steps, which are run in sequence or in parallel as appropriate.

Both Sean and I already do this routinely. Sean's work mostly involves human-readable documents, mine involves data-oriented XML, but many of the processing steps and issues are the same. With any complex data transformation, it can seem attractive to do the transformation in one big step with a complex piece of code. However, such code invariably takes longer to write than you expect, because it takes longer to debug than you expect. Also, complex code can hide subtle bugs that are missed in testing, and can be hard to isolate and resolve when a problem occurs in a production system.

Using a pipeline of simple steps gives you many points at which you can examine the data during the course of the processing, which is a big aid to debugging and quality control. Also, it makes it easier to spread the development work among multiple developers, and it increases the likelihood that you will create some pipeline steps that are both (a) re-usable, and (b) worth re-using [since (b) doesn't always follow from (a)].

In the past, Sean and I both have sometimes used Apache Ant as the framework for constructing pipelines. Ant is a flexible framework for controlling all kinds of processes, not just build processes as it was originally designed for. However, it's focus on software builds means that it lacks a complete set of built-in XML processing operations ("tasks", in Ant jargon).

It will be interesting to see if XProc will make it easier to construct certain types of data processing pipelines. It isn't clear if XProc will work equally well for all types of pipeline processing, or if it will work best for particular categories of processing. For example, Norm Walsh has written about an approach he wants to take for an XProc implementation. That approach involves streaming the information from one pipeline step to the next such that the pipeline steps can run in parallel, reducing the overall end-to-end processing time of the pipeline to a minimum. I think a lot of people will want to use XProc that way a lot of the time. However, it isn't right for every need.

For example, I sometimes have to work with XML files that are 10 to 100 Mbytes in size, and memory usage becomes an issue. For that kind of processing, it is sometimes better if the pipeline steps are run strictly one after the other, not in parallel, with the intermediate XML stored to disk to free up as much memory as possible. Indeed, sometimes the end-to-end processing speed isn't the main issue, if you can run your pipeline in parallel on multiple data sets using multiple machines. With any pipeline approach, it can be helpful if you can control how the pipeline is deployed and how it runs, so that you can tune its operation to your problem and your system to get the best overall result.

I'm looking at XProc at the moment as I want a simple but configurable .NET/Mono XML pipeline solution for a project, and it struck me that implementing a subset of XProc might be better than designing and implementing something completely proprietary from scratch. If it proves successful, I might consider a more complete XProc implementation. We'll see.

How to buy Microsoft Vista

The cartoon version ...

A Selection of Photos, London and Dublin

For your weekend viewing pleasure, here are some photos that I took with my (film) SLR and recently had scanned. Click on any photo to go to a page with a link to the full-size image.

Christmas Tree Lights

It's wonderful what you can do when your camera lets you trigger the flash at the end of the exposure, rather than the beginning.

Sunset on Greenland Dock, London

Swans swimming at sunset.

Waterfall at the Guinness Storehouse, Dublin

This is part of a display on the necessity for pure water in the making of beer.

Document Comments - Microsoft Word beats OpenOffice

I've mentioned that Microsoft Word has its faults, but one place it is ahead of OpenOffice is in document comments. The issue in particular is that MS Word allows you to apply a comment to a region of text, whereas OpenOffice only allows you to insert a comment at a particular position in the text.

It's important to know which region of a document, from beginning to end, a comment applies to. A comment at one point in the text doesn't give the necessary information about just how much of the text is affected.

This is really important if you are doing document reviews. Rather than adding edits directly to the text as part of the review, it often works better to add comments because they are non-destructive, and the document editor can then take a global approach to implementing the comments once they have been decided. Comments often end up being interlinked, they can't just be implemented completely independently.

For this, which happens a lot when working on standards (for example), MS Word is just easier to use. That's frustrating for me since I otherwise prefer to use OpenOffice. However, the next version of the OpenDocument file format, version 1.2, is supposed to support full round-tripping with the Microsoft Office file format, and so I'm hopeful that future versions of OpenOffice will support a better approach to document comments than they do now.

Fingers crossed.

The Post-It Note Jaguar

Imagine this. Your boss drives a Jaguar (and you don't), you have far more Post-It notes in the stationery cupboard than you know what to do with, and you have nothing to keep your idle hands busy for an afternoon.

What happens?

Come see me talk in New York

The New York XML Special Interest Group is being re-launched, at I will be one of the speakers for that re-launch on the 9th May, 7-9pm, 1 New York Plaza.

G. Ken Holman will be speaking from 7-8pm on UBL (Universal Business Language), then I will be speaking from 8-9pm on code lists and why they are never as simple in practice as you expect in theory. I'll be discussing the work of the OASIS Code List Representation TC, which is producing a standardised version of genericode, the XML format for code lists that I first presented at the XML 2004 conference.

I'm the editor for the OASIS genericode specification, and version 1.0 is almost complete, so it will be out for public review shortly. I'll let you know when. Version 1.0 covers how to statically specify code list information, including multiple keys for the same code list and arbitrary metadata for each entry in the list. Version 2.0 is planned to add supporting for dynamically generating new code lists from existing code lists in an automatic, repeatable, and auditable manner. There may also be a version 3.0 after that to add some other features, that's still to be decided.

If you are going to be in NY on the 9th May, I hope you can come along. Come say hello.