kontrawize

Binary XML, Efficient XML, part 2

Following on from my  earlier post on Efficient XML, John Schneider of Agile Delta suggested that I mention that they have a  free demonstration version of their commercial implementation available. I haven't tried it myself, I can't say anything one way or another about it, but it's probably a good way to get a feel for what you can do with Efficient XML.

Opera 9.50 beta, part 2

In my  earlier post on the Opera 9.50 beta, I forgot to mention that I have a major problem with this beta under Windows XP. It won't open HTML files from the file system, although it still opens HTML from Web sites as normal. I don't know what the problem is, and it doesn't occur under openSUSE Linux. Given how big an issue it is, I'm expecting that Opera will have to release new beta soon to fix the problem.

Binary XML, Efficient XML

I have written before about the need for a standard binary compression technique for XML. It's an issue that divides the XML community somewhat. Some very well known names in the XML world, people I get along with very well socially, are strongly and publicly against the use of binary encodings for XML. To be honest, I think that they are  wrong, albeit for some for the right reasons.

As I've mentioned from time to time, there is no such thing as best practices. There are only practices which are good practices in a particular context, in a particular set of circumstances. The problem I find, when people argue against binary encodings for XML, is that they are typically espousing “best practices”, without giving sufficient thought to what the appropriate contexts are.

So, let me start by putting things in context. The fact that XML's default encoding is text (in some character encoding) is a good thing in many contexts. It means that you can look at the contents of an XML file with just a simple text editor, and sometimes that is enough to be able to understand the contents of the file. You don't get that same convenience with binary encodings. Also, some people are concerned (and with good reason) about document archiving, where you need to be able to access the contents of electronic documents decades or centuries into the future. For that context, textual encoding makes sense, as binary encodings require you to have the software to decode them, and to have the operating system that runs the decoding software, and to have the hardware that runs the operating system, etc. You get the point. These are good reasons for encoding XML as text. Indeed, I never advise someone to use a binary encoding of XML unless they had a specific need to do so.

When would you want to use a binary encoding of XML? Well, binary encoding compresses the XML document into a smaller size, sometimes just a few percent of its original size. Occasionally that might be useful for storage, if you deal with enormous data files (for example, large data sets for numerical simulations). More commonly, though, binary encodings are used when transmitting files across networks. Why? Well, size is an obvious reason, to reduce bandwidth. However, another reason is speed of parsing. In probably the majority of cases, the time taken to parse XML documents is not significant in the context of the whole business process they are used within. People often worry too much about the processing overhead of parsing XML, without looking at the bigger picture of their processes and how long they take from end to end. However, there are contexts in which the parsing time is genuinely significant.

When you have large volumes of documents to process, you can sometimes manage that processing by spreading the work out over a “farm” of processors. Each if each document is processed relatively slowly, the overall throughput can be high. However, that only works if you aren't sensitive to the time taken to process each individual document. In the modern finance world, algorithmic (computer) share trading means that a difference of just 20ms (1/50 second) in processing a message can be the difference between getting the shares at the price you wanted, or missing out. This means that the time taken to process each individual message is important. You can make an XML message smaller by using gzip or such, but you still have to unzip it into text and then parse it. With a good binary encoding, you should be able to decode the XML file directly as an event stream (e.g. like  SAX), without ever having to reconstruct the XML document as text. That can make a very significant difference to the process in time-sensitive contexts like the one I've outlined.

Arguably, in these days of XML messaging and Web Services, the majority of XML documents are  short-lived. They don't need to be archived, and the lifetime of the messages is much shorter than the lifetime of the software that processes or encodes them. When this is the type of XML that you work with, there is no need to treat it like archival XML.

I started trying to push XML compression for these types of banking/financial contexts back in 2001, when there was the MPEG-7 specification. This was for digital video, but with compressed XML to provide metadata. It was a pretty good XML compression scheme, but it never took off, in spite of everyone repeatedly complaining about the size and verbosity of XML messages. Why not? In my mind, there were a couple of key reasons. The W3C didn't recommend any kind of compression at that time, and people take the W3C very seriously and religiously as a “brand”.  Also, there wasn't a large choice of compression/decompression software, and consequently large companies were genuinely worried about being locked in to particular software tools that would play a fundamental part in their messaging systems (of course, they put up with this for relational databases, but that doesn't mean they want to repeat the same lock-in for every piece of software that they use). So they took the pain of textually encoded XML, or used alternative binary formats; bandwidth has not grown as quickly as some people thought it would, not as compared to the growth in the volume of information distributed between financial organisations.

Indeed, although some people view this question as “should I binary encode XML or not?”, the real question for users with high-volume and low-latency requirements is “which binary encoding should I use?” The dominant financial protocol for pre-trade and trade activities is the  FIX protocol, which is a binary protocol. There is also a matching XML variant of the messages, FIXml, but it's the binary protocol that is overwhelmingly used, because the XML variant mostly just makes the messages larger and slower to process. The FIXml schemas are generated from the same model (held in a relational database) that the binary message definitions are generated from.

At this point, some people would say “well, that's OK, XML isn't good for everything, sometimes you just need to use binary encodings, that doesn't mean that you need binary encoding for XML”. However, people often use XML because it is quicker to create Schemas than to create binary encoding formats (and prove that those binary formats are completely unambiguous in their encoding). XML is familiar to many developers now, and there is a whole ecosystem of great tools for XML. Why not make the most of those XML tools, even if you are working in a context where it makes sense to go binary for transmission? Why should all of the tools that you use with binary encodings be different to the tools you use for XML? It's not helpful to be forced to use completely different tool sets.

I've been thinking about all of this because last week, I was phoned up by John Schneider, the CTO of  Agile Delta. The  W3C has actually been working on binary XML for a couple of years now, in spite of snide remarks from some of the XML old guard. In particular, they have been  measuring and comparing a number of alternative approaches to compression and binary encoding of XML. The tests measured not only the amount of compression, but also the time taken to decode and parse binary encoded XML documents. There are also  performance results on the Agile Delta site. John contacted me because Agile Delta's “Efficient XML” approach has been chosen as the best all-round approach of those tested, and is now the focus of the W3C's efforts.

I'm really very pleased to see this work progressing. The W3C's support is very welcome, as I noted, because it's such a powerful brand in the Web/XML world. We don't have support for Efficient XML yet, to the best of my knowledge, from that other powerful brand, Apache. However, there are products available right now that implement Efficient XML, and it seems very likely that this number will grow over the next year or two because of the W3C's support, so that there will be sufficient and genuine choice between vendors.

If you have a context that could genuinely benefit from it, particularly a high-volume and low-latency context, you should do yourself a favour and check out Efficient XML.

Opera 9.50 beta

Opera 9.50 beta is available for download and testing. I'm a long-time user of Opera as my Web browser, e-mail client, and Atom/RSS feed reader. I've had some problems with earlier versions of Opera 9.x, which can  crash a lot when run inside a VMware virtual machine running Linux, due to an memory issue related to  VNC. I'm hopeful that 9.50 will fix this issue (and I've been testing it since the alpha releases for that reason).

That aside, the most interesting new feature in Opera 9.50 seems to be  shared bookmarks. One of the things I like about Opera is that it applies  fast text search to everything. So, if you are looking for a bookmark, there is no need to have them carefully categorised (although you can if you must). Instead, just start typing in some text from the name or description, or part of the URL, and Opera will refine the bookmark list with every key press. You very quickly get the short list that you need. It's great, but if you run Opera on different machines or in different environments (and I effectively run Opera in 6 different environments), you still have the issue of maintaining bookmarks in multiple places.

Of course, there are Web sites for storing your bookmarks, but it's much slower to bookmark links that way, and the search facilities that those sites provide typically aren't as quick and easy as Opera's search. With shared bookmarks, Opera synchronises your bookmarks directly from Opera to a service via the  Opera Community site. Each copy of Opera that you configure to login to an Opera account will have its bookmarks synchronised there. For me, it is a  genuine improvement over what I have had to do previously.

Great Flaming Squirrels

Read on, at The Register.

OASIS genericode 1.0 (Code List Representation, XML format) is available for second public review

OASIS has released the second committee draft of genericode 1.0, the OASIS Code List Representation XML format, for the second public review. The review runs from today for 15 days. See the "Genericode News" section at

http://www.genericode.org/

for details. As I've mentioned previously, this is based on the XML code list format that I presented at the XML 2004 conference (there is a link to that paper at  http://www.genericode.org/).

Comments are very welcome, preferably posted to the OASIS Code List Representation TC's comments list (codelist-comment@lists.oasis-open.org). If you have any general questions about genericode, you can  mail them to me.

The Trouble With XML Schema Imports and Includes

Any of you who have written W3C XML Schemas will know that there are two mechanisms for sharing Schema definitions across files.  <xs:include> is used like traditional programming language “include” statements, so that you can split a single large file up into separate, modular pieces.  <xs:import> is used when you need to use Schema definitions from a different XML namespace, as W3C XML Schema doesn't allow a single Schema file to contain definitions for more than a single namespace.

Now, I have been involved with a number of standards groups, and I know that the what comes out of a standards effort depends on what requirements and scope have been given to the group.  So while I wish that the W3C's XML Schema Working Group had been able to give us something better for Schema definition sharing than just import/include statements, I don't think the working group ever had a scope that would have allowed them, for example, to define a standard for repositories of Schema definitions, and for how to compose and generate Schemas flexibly from the definitions in one or more repositories.  Instead, we have import and include, which are about the best you can do when your scope only allows you to deal with schemas, and not with higher-level concepts like repositories.

As a general rule, almost any solution will work if the problem is simple enough and straightforward enough and isn't particularly demanding in any way.  Many uses of import/include statements are simple enough that these built-in mechanisms do what is needed.  However, there are other situations where import/include don't work as you would hope, and I thought I would mention a couple that I have run into in practice.

For example, I already mentioned that <xs:include> is commonly used allow a large single Schema to be broken up into modular pieces.  If you get the modularity right, you should just be able to include just the pieces that you need into your Schema. However, what happens if you are importing Schemas from a separate namespace using <xs:import>?  I was working with some generated Schemas today (i.e. not hand-written) which had multiple <xs:import> statements for different Schema files which share the same namespace.  Two Schema validators told me that some of the definitions I was importing weren't available, and I wondered if I had some kind of subtle namespace error.  It turned out, and this was news to me, that using <xs:import> more than once for the same namespace is a grey area for W3C XML Schemas, and a typical behaviour for Schema validators is to process the first import for a particular namespace, but to ignore any subsequent imports for the same namespace.  This means you end up with missing definitions and invalid Schemas.

So, it seems that the only safe way to import (as opposed to include) definitions from multiple Schemas in a particular namespace is to define an extra Schema for that namespace, one which does nothing but include all of the other Schemas in that namespace.  You can then import that “include only” into other Schemas and work around the aforementioned “one import only” issue.  However, you lose the ability have true modularity of Schema definitions in multiple namespaces if you are effectively forced to import all Schemas from a particular namespace, rather than just the ones you actually need.

I wonder how many Schema generation tools actually make provision for this issue.  I don't know of any, but I guess it's yet another reason why people often give up on using multiple namespaces, and just use a single namespace wherever possible.

Another issue I've had in the past (and I don't know if things are better or worse now) is when the Schema files are distributed over a hierarchy of directories/folders, rather than all being in the same directory/folder.  When you import/include a Schema, you specify a location, and “good practice” would often be to use relative paths for this, not absolute paths, so that you can move the Schemas to a new directory/folder without breaking the path relationships between the Schemas.

However, sometimes relative paths fail to work correctly when you have the Schemas spread among directories (especially is the directories are not at the same level, but some are sub-directories of others).  In a file system, it's usually quite straightforward to work with relative paths.  However, XML is designed to work on the Web as well, and URLs don't have to obey the same rules that file systems (usually) do about relative paths, especially because Web servers are free to do clever things like URL re-writing on the fly.  For this reason, and particularly if your relative paths contain “..” (parent directory reference), you may find that your Schema validator errs on the side of caution and does not make assumptions about whether two relative paths that it has seen actually compute to produce the the same absolute path.

Why is that an issue?  Sometimes many files include the same Schema (perhaps a Schema of common definitions), and so when the validator is processing includes (and includes within includes, etc.), the validator may find the same file included more than one.  If the validator can't determine whether the relative paths referenced the same file, it may assume that it has loaded two different files containing the same definitions, rather than having loaded the same file twice.  Since you aren't allowed to have two definitions of the same thing, you get Schema validation errors, which can be hard to get rid of, but may magically disappear if you put all of the Schemas into a single directory (with the relative paths adjusted appropriately).

It's not the Schema imports and includes are bad, they are often just fine, but they aren't the complete story, they are only a half-way step to being able to share definitions between Schemas in a completely flexible way.  I can only hope the W3C (or even some other group) takes this further at some stage, e.g. by defining a Schema repository and generation specification. Yes, yes, maybe I should do something about this myself, but it may have to wait until I'm finished with some of the various standards groups I'm already working with (life is short, consultants need to do billable hours too, etc.).

ISO 19312 Securities Data Model (and MDDL)

David Winter posted a comment, noting that he had been unable to find the ISO securities data model in the ISO Web site (that's financial securities – stocks/shares/etc.). The reason that it isn't there is that ISO 19312 is still a work in progress, it hasn't yet been approved (indeed, it hasn't yet started reached the ISO voting stage yet). As such, it isn't yet generally available, although you may find that you can get a copy of the current draft of the data model from your local ISO/standards representatives if you are involved in banking and finance.

David is currently looking at MDDL, which is the best place to be looking in the meantime. Indeed, ISO 19312 will only cover a subset (more or less) of what MDDL covers, at least initially. David seems a little perturbed or surprised that MDDL puts trading terminology (snap, time-series) at the top of its document hierarchy, but in my experience most surprises that people have about standards are usually due to people assuming different requirements to those that were used (or otherwise disagreeing with the requirements that were used). MDDL, for example, was originally designed for distributing financial market data (from stock exchanges and such), so it's not surprising that it talks about snaps (snapshots) and time-series, which are what people look at when they look at market data. MDDL isn't just a financial instrument data model, as ISO 19312 is (but note that ISO 19312 also covers corporate actions – changes to the details of securities because companies have merged, split, changed name, or because of other similar reasons). David considers MDDL to be inverted, so I guess the way he would have done the requirements would have been inverted, in some sense, as compared to the way they were done (and there's often no good or bad in these things, just choices that have to be made, one way or the other).

For anyone who is looking at MDDL as a data model, note that the most versions of MDDL have their data model defined in an XML file (e.g. “mddl-2.5-beta.xml”), which is a simple XML format constrained by a Schema (e.g “mddl-2.5-beta-schema.xsd”). The latest version of MDDL, 3.0-beta, comes with the data model in the form of a UML model (and documentation generated from that model is supplied with the release).

If you feel you must reverse-engineer a data model from the MDDL Schema, please use the “lite” or “base” version of the Schema, as the full Schema implements a number of MDDL-specific “ shorthand” structures which don't make sense if directly reverse-engineered as a data model.

Picasa for Linux

I only came across this by chance on a news group, but there is a version of Picasa available for Linux. That fact isn't obvious from the main Picasa home page. People tell me that Picasa isn't bad, but I wasn't interested in it while I thought it was Windows only. What Google has done is to get it working under Wine, rather than recode it for Linux. I've downloaded it, and it certainly has no problem reading the RAW files from my Fujifilm S3 Pro digital SLR. I'll definitely be playing with it to see what it can do. I'm not sure how the performance under Wine will be, that's an open question.

One things I've noticed, though, it that it appears to want to import all of my photos into its own folder, rather than allowing them to be left in place. My RAW files are 24-26MB each, so I don't want multiple copies of them on my drive if I can avoid it. It will depend on whether I think Picasa is good enough to make the sacrifice and do it Google's way.

Hibernating a Linux laptop with VMware Server, part 2

I previously wrote about how I added a script to openSUSE 10.2 to stop VMware Server when I hibernate my laptop, and restart it when I resume. In openSUSE 10.3, the directory where you put the script has changed. It was " /etc/pm/hooks", but now it is “ /usr/lib/pm-utils/sleep.d”. It looks like the same script should work OK, although I'm had intermittent results so far with hibernation under openSUSE 10.3, so I may yet have other pieces of the puzzle to solve.