kontrawize

Generating XML Schemas from ISO 20022 UML models

As I've mentioned in the past, I'm the principal UK expert (yes, the "PUKE") for ISO TC68/WG4, the ISO working group that has produced an updated version of the ISO 20022 standard. ISO 20022 is standard for defining messages based on definitions in a central repository, with a focus on banking and finance. Those of you whose are involved with financial messaging via the SWIFT network will have used ISO 20022 messages whenever you use one of the XML "MX" series messages. WG4 has produced a 2009 version of ISO 20022; it is currently a committee draft which has gone to TC68 for an international vote, so it is not yet an approved version; the current approved version is the original 2004 version.

My particular interest in ISO 20022 is in the part that describes how to generate XML Schemas from UML models. I like the repository based approach where you can share components between related messages, but don't have to implement that sharing at the Schema level, with includes and imports. ISO 20022 has one Schema file per message, and that makes life easier than if you have to get people to set up a hierarchical directory of cross-linked Schemas correctly.

I wrote an XML Schema generator for the current 2004 version of ISO 20022. That generator works with XMI (UML in XML) files produced by Enterprise Architect (from Sparx Systems), and is implemented as a series of XSLT scripts.

For the new 2009 version of ISO 20022, the good people at MagicDraw donated licences for WG4 to use their UML editor, and so MagicDraw is the UML editor that was used to develop the new metamodel and UML profile. For this reason, for my new implementation I decided to target MagicDraw (I might support other UML editors too, later, depends on how many requests I get).

I decided to continue with my previous approach, working from the XMI, this time using a mix of XQuery and XSLT. XMI is a rather generic XML format that can be hard to read and understand, so I consider it error-prone to try and generate XML Schemas from the XMI directly. Instead, I distill the XMI into a much simpler XML format which reflects the structure of the ISO 20022 metamodel, an "XML DSL". I then process that simpler XML file to generate Schemas or anything else. So, the plan was to use an XQuery to generate the simple XML, and then XSLT to generate the Schemas (there are some things that are easier in XSLT than XQuery).

However, it turns out that the latest XMI 2.1 standard is loose around how UML stereotypes (user annotations) are represented. So, after some failed attempts at "reverse engineering" how MagicDraw represents stereotypes in XMI, I decided I needed a different approach. Instead, I am using the MagicDraw Open API to read the contents of an ISO 20022 UML model. This means I need to be using a Java program, or something that runs on a JVM.

Java itself isn't great for XML, in as much as the code tends to end up rather more verbose and unreadable than it need be. I tried Groovy, which has some reasonable support for XML, but moved away from that because the XML support is incomplete. It seems that in a number of languages, the people who implemented the XML library didn't really like XML, so they decided to only implement support for the parts of XML that they thought were worthwhile. For users, that causes nothing but problems.

What I am currently using is Scala, which is convenient because you can just start typing XML as the value of a variable in the middle of a program, and Scala knows what to do. I've now written the code to generate the simple XML format. It hasn't been as easy as it should have been, because Scala support in IDEs is still weak (I use Intellij), and because Scala's integration with Java is good but not as good as it could or should be. For example, Scala avoids the use of null values. It does have some support for them because it needs that to work with Java libraries. However, at times when Java code would give me a null pointer exception, I find my Scala code just hangs, leaving me with a lot of searching to do to find what went wrong and where. Still, the code is more readable than Java, and things can only get better, so I'm planning to stick with Scala for this.

Once I have XML Schema generation done, the next thing to look at is ASN.1. ASN.1 ("abstract syntax notation") is an ISO standard for designing messages that predates XML, and has been most widely used in the telecommunications industry. Note that I wrote in a previous post that ASN.1 was the format used for ISO 15022 messages (more-or-less the predecessor to ISO 20022). That was wrong, I want to point out; somewhere along the line I got the wrong idea, and in spite of my saying it in a number of WG4 meetings, only recently did someone from SWIFT point out that I was mistaken.

Anyway, WG4 has added support in the 2009 version of ISO 20022 for ASN.1 as an alternate message syntax to XML Schema. Now, ASN.1 itself is actually multi-syntax. There are a couple of binary encodings it supports, plus there is a more recent XML encoding. The ISO 20022 profile of ASN.1 has been constructed so that, as closely as possible, an XML message that is valid according to the XML Schemas will also be valid as an XML encoded ASN.1 message, and vice-versa. Since ASN.1 messages also have a binary encoding, this new ASN.1 support means that ISO 20022 will have an official binary encoding as well as the existing XML encoding. It will be interesting to see how the usage of XML versus binary ISO 20022 messages goes over the next few years; banking and finance applications have size and latency challenges for which plain-text XML often isn't suitable.

I should be able to generate the ASN.1 grammars for the messages from the same simple XML that I use to generate the Schemas. Doing so will show that the simple XML genuinely decouples that back-end code generators from the format of the input UML model; that's something I need to have the possibility to support UML editors other than MagicDraw in the future.

If any of this interests you, either ISO 20022 or my work on Schema and other generators for ISO 20022, feel free to get in touch.

Null propagation, part 3

Almost two years ago now (#1, #2), I wrote about the idea of being able to modify the way Java handles the situation where a method is called on a null value. Normally this causes an exception; I was wanting such a call to return a null instead, so that the result is more like what happens with XPaths when a node in a path doesn't exist. For example, the XPath

/document/person/address

doesn't cause a program exception if there is no <person> element, whereas

getDocument().getPerson().getAddress()

will throw an exception if "getDocument().getPerson()" returns null. These exceptions make it painful to use Java (etc.) for working with XML, because you end up with so many try/catch blocks to deal with of the places that an exception could possibly be thrown.

So, it was interesting when I stumbled across the fact that Groovy (a Java-based scripting language) solves this issue! Groovy has a "?." operator, which returns null if called on a null, and executes the method normally if called on an object, i.e. "a?.b" is equivalent to "a == null ? null : a.b". So

getDocument()?.getPerson()?.getAddress()

will return null if "getDocument()?.getPerson()" returns null. That makes Groovy (for this and other reasons) a better way to write your XML processing code in Java, if you are trying to keep your code as simple and maintainable as possible.

Data modelling, metamodelling and MPS

One side of the work I do for my clients is XML and related technologies, the other side is general data modelling. One thing that interests me in data modelling is the use of metamodels. Now, what is a metamodel? Normally, people switch off or break into a cold sweat when they hear "meta-anything" in a sentence. However, there is nothing scary about metamodels. A metamodel simply defines what things are allowed to be in your data models; it's a model of what you are allowed to have in your data models.

Why do we need metamodels? It's because data modelling is usually done using general purpose modelling technologies. Examples are E-R diagrams, UML, and XML Schemas. Any of these general purpose modelling technologies give you more features than you need or want in any single project. It can be a major problem to constrain people to use as few features as possible, which is what you want to make your data models as simple to understand and as cheap to maintain as possible. The role of a metamodel is to define a limited set of features which are the only ones that can be used out of all of the features that the general purpose modelling methodology provides.

For example, it's common to find people using XML Schemas that (a) use elements only for data, not attributes, and (b) use local element definitions only in the XML Schemas, not global element definitions (root elements accepted). This constraining of XML Schema to a simple subset of its features is equivalent to defining a metamodel that is a subset of the full metamodel for XML Schema. In practice, you can implement just such a "metamodel validation" on an XML Schema by applying some kind of schema language to the XML Schema itself. As it happens, XML Schema editors often won't let you apply an XML Schema to another XML Schema, but you can apply both Schematron and/or RELAX NG to XML Schemas in order to implement checks that the XML Schemas conform to your metamodel (i.e. your design rules).

UML is widely used for data models, and metamodels for UML are written in using a thing called MOF. With the latest versions, UML 2 and MOF 2, MOF is now a subset of UML that can be used for writing metamodels which constrain UML models. Sadly, support for validating UML models using MOF metamodels is still poor (at least in any tools I have used), but I live in hope that within a couple of years it will be possible to validate a UML model with a MOF metamodel as easily as you can validate an XML document with an XML schema.

Today I was catching up on some reading, and I stumbled across a mention of JetBrains Meta Programming System (MPS). I tried an early version of this a couple of years ago; it lets you design custom/bespoke mini programming languages (or mini data formats) that are specific to a particular purpose. The MPS tutorial is a good way to get a feel for MPS, what it does, and how it works. There is an obvious analogy between defining a mini-language for a task rather than letting people loose with a general purpose language, and using a metamodel to constrain people's usage of the features of a general purpose modelling methodology. MPS allowed you to define structured data models that have something of the feel of (simple) textual computer program, or a textual data file like a simple properties file. Although they have the look of text files, they are actually structured, and in some ways it's not unlike generating form-based user interfaces or Web pages. MPS is a neat looking tool, and it's free. I like JetBrains (commercial) Intellij IDE for Java, because it's so clever in the degree of analysis it does of your code as your write it. I was amazed how many issues it was able to point out to me as I was writing my code, before I even compiled it. If the JetBrains folks can apply the same cleverness to MPS, it could be a useful tool. That said, it's not the only tool that can generate user interfaces from models. Eclipse has EMF (like a cut-down version of UML) and Visual Studio has support for Domain Specific Languages (DSL). All of these tools are working to the same end, to provide you with a way to allow people to work with only the minimum set of features that are needed for any particular problem, so that they produce the simplest, most consistent, most maintainable solution that they can.

Metamodels and their equivalents let you take a complex software tool or methodology, and constrain it to effectively turn it into a simpler software tool or methodology. It's a bit more effort, but for a project of any size, it can pay real benefits in the end.

Opera bug in handling of Atom feeds

I use the Opera browser (currently version 9.62) not only as my browser, but also as my e-mail and feed reader. I had noticed that, sometimes, the link at the bottom of a feed (blog) post doesn't take me to the correct page. In particular, this was happening with the Atom feed for Tim Bray's 'ongoing' blog. Tim was one of the authors of the Atom specification for blog feeds; I know Tim, and I know his blog feed follows the Atom specification rigorously.

I wrote to Tim and asked him about it, and it appears that this a known bug in Opera's handling of Atom feeds, that it doesn't not correctly process relative URLs in Atom feeds. Apparently Opera has been doing this wrongly for some years now. For example, for Tim's most recent post, Opera shows the link the Web page as being

http://www.tbray.org/ongoing/CL-Rain

but the actual page URL is

http://www.tbray.org/ongoing/When/200x/2008/11/28/CL-Rain

What went wrong? Well, in the Atom feed for 'ongoing',

• the value of "/feed/id" is "http://www.tbray.org/ongoing/"
• the value of the first "/feed/entry/link/@href" is "CL-Rain"

and Opera appears to be concatenating these to produce the URL

http://www.tbray.org/ongoing/CL-Rain

However, that first "/feed/entry" has its "xml:base" set to "When/200x/2008/11/28/". What Opera should do is add this in the middle, to produce the correct URL

http://www.tbray.org/ongoing/When/200x/2008/11/28/CL-Rain

Anyway, I have now posted a bug for Opera (DSK-240409), so we can only wait and see now when it gets fixed. Fingers crossed that it's sooner, rather than later.

XML Spy, Schema Documentation, and Java, part 2

I wrote previously that I had tried using the XML Spy 2008 Java API to generate Schema documentation, and it hadn't worked. I ended up with an exception in the DLL code that the Java API uses to control XML Spy, and I assumed it was a problem with the Java API, perhaps that it wasn't well enough tested. However, Liz Andrews from Altova kindly posted a comment with some sample code for using the Java API. Thanks, Liz! I can see now where I made my mistake; the Java API reflects the style of the XML Spy COM API, so it's slightly unusual for a Java API, and I hadn't understood the correct way to create a "SpySchemaDocumentationDlg" object. However, the sample code shows exactly how it is done. I haven't tested it yet, but it looks good to me, I can see why it would work and why my own code didn't work.

The documentation that comes with the XML Spy Java API is somewhat minimal, so I hope Altova will think of including this example with the documentation, or adding the example to their web site, at it's an example that explains a lot that the documentation doesn't cover.

XML Spy, Schema Documentation, and Java

I've mentioned in the past that my preferred XML editor/IDE is oXygen (although I currently use XXE to write these blog posts in XHTML; it's also good for editing DocBook). However, my clients almost always use XML Spy (which I also have, for that reason). In particular, a lot of people prefer the Schema diagrams that XML Spy produces to those produced by other tools (although Liquid XML also produces attractive Schema diagrams).

Now, one thing I've done for a number of clients is to implement a Schema quality control framework for them. This is a batch-mode tool that does for XML what a build system (like 'make', 'Ant', 'NAnt', etc.) does for Java/C#/etc. You don't compile Schemas, but you can regression test them against examples, you can test that they conform with your design rules (e.g. to have a consistent style, which makes maintenance easier), you can produce dependency documentation, and generally provide reports that help both developers and managers understand what is happening within a large set of Schemas, and help them catch unintended consequences of changes in a new version of the Schemas, so their own clients don't end up impacted by those consequences.

One thing that you do within such a framework is generate Schema documentation. Since so many people like XML Spy documentation, that is what they ask for. The problem is that it hasn't always been easy to get the result you want with XML Spy. In particular, previous versions produced either one Word file or one HTML file for all of the Schemas connected to your main Schema (assuming you want the documentation fully hyperlinked). That one HTML file could easily have thousands of diagrams in it, and it wasn't uncommon to find that you had produced an HTML file that none of your client's PCs could load without the browser crashing. I'm very happy to find that XML Spy 2008 now allows you to generate separate HTML files with the appropriate hyperlinks between the files, which is an enormous improvement.

In terms of running it from the command line, XML Spy has for a long time had a COM interface for running XML Spy from an external program. I noticed that XML Spy 2008 now has a Java wrapper for that API, which seemed like a good way to integrate it with Apache Ant, so I gave that a try. The bad news is that while some of the functions work, it crashed with a memory violation error when I tried to create the dialog object needed to control the settings for Schema generation. Instead, I wrote a short command-line program in C# to call the COM API more directly, and then I used Ant's 'exec' task to run the .NET executable.

The result is that I had to add an extra technology into the mix (always something to avoid if you can), but it works, and I am now generating the documentation that my client wants from an Apache Ant build process. Who knows, if Altova gives that Java API wrapper some more testing, it might turn out to be usable in XML Spy 2009 or XML Spy 2010.

Efficient XML examples (compressed XML) online

I've written before [#1, #2] about the W3C's “Efficient XML” specification which defines a “binary” XML format that

• compresses your XML (with or without a Schema, but it can do a better job with a Schema);

• can be parsed more quickly than the equivalent textual XML can be parsed, i.e. the compressed XML is effectively pre-parsed (and pre-validated if a Schema was used).

Let's be clear, for those of you who don't like the idea of “binary XML”, I think 80% of world is perfectly happy 80% of the time with XML in its normal textual format, and I'm not suggesting the 80% should change. For some XML users (e.g. in banking/finance, where I do most of my work for clients), textual XML is a barrier to some of their current volume requirements and per-transaction processing time requirements. So some people need this, even if you yourself don't.

Anyway, I noticed during the week that numerous reference compressed files have been posted to the Web. While the files themselves are probably most of interest to people working on implementations of Efficient XML, the file sizes are interesting more generally, in my mind. Sadly, they didn't post a table of file sizes, so I downloaded the FpML (Financial Products Markup Language) files for IRD (Interest Rate Derivatives), which is an area of interest to me (there are lots of examples, though, some of which may be more relevant to what you do).

Each of the FpML IRD example XML files was compressed multiple ways using different compression options, using the Agile Delta implementation of Efficient XML. I looked at the best compression for each example, and found that the average was about 85%. The actual compression varied about 70% and 90%, with a graph of compression versus file size suggesting that compression improves for larger files. The largest of the FpML examples was about 20k, and you would really need larger files again to test whether the compression genuinely improves with the uncompressed file size.

The problem, at the moment, in trying to convince people to look at this technology is that Agile Delta's implementation seems to be the only one available. At least, I can't find another; if you have an implementation, or are planning one, please add a comment to this post with the details. I agree with the adage that an open standard with only a single implementation is no better than a proprietary standard, so I really hope we will see some other implementations before too long. Something from Apache/IBM/Microsoft/Sun would be nice, with smaller vendors keeping them honest. Here's hoping, anyway.

Miley Watts XML File Lister for Apache Ant

I have updated the open-source XML file listing task that I have written for Apache Ant. This is useful, in particular, when you want to apply an XSLT stylesheet to a number of files together (not individually). You use the file listing task to create an XML file listing containing the file URLs, then you load the files into your XSLT stylesheet using the URLs and the XSLT “document()” function.

Version 1.0 created an XML file listing with an XML hierarchy that mirrored the directory hierarchy. The new version 1.1 has two other options. You can produce a flat listing without the directory hierarchy, and you can create a “collection” XML listing that works with the “collection()” function in the Saxon XSLT/XQuery engine.

For details and examples, please visit the download site. I hope you will find it useful.

xmlcatalog.net

A few years ago now, I was a (junior) member of the OASIS Entity Resolution TC, which produced the OASIS XML Catalogs specification (HTML, PDF). XML Catalogs are used to specify simple rules for re-writing or re-directing URIs in XML documents. This work was based on the catalogues used for SGML (from which XML is derived), so there was a lot of genuine long-term user experience factored into the design.

The most common use case for XML Catalogs is re-directing an HTTP URL for an XML Schema or DTD reference to a local copy of the file on your own file system. This aids both performance (HTTP is comparatively slow, and the remote HTTP server is usually that much slower again) and security (you don't have to trust that URLs in an incoming XML document are pointed at the correct schema, or the correct version of the schema). It also means that your different users or different systems can have their own catalogue file that controls where they get their schemas from, and there is no need to try and force one single configuration (a setup of HTTP URLs or a set of file locations) to work for all situations.

XML Catalogs is implemented now by a variety of software tools, e.g. Java, oXygen and XML Spy. However, I needed an implementation to use for a .NET application, and couldn't find a suitable implementation, so I have written one, called “xmlcatalog.net”. You can download it from

http://secure.mileywatts.com/confluence/display/CMSW/Miley+Watts+XML+Catalogs+for+.NET+and+Mono+(xmlcatalog.net)

(or alternatively from http://xmlcatalog.net, once the domain re-direction kicks in properly).

As an example, suppose I have the following (trivial) XML document:

<?xml version="1.0" encoding="UTF-8"?>
<schemaTest xmlns="http://www.example.com/schema/test"
   xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
   xsi:schemaLocation="http://www.example.com/schema/test schematest.xsd">
  <a><c/></a>
  <b><d/></b>
</schemaTest>

Note that the “xmi:schemaLocation” attribute indicates that the relative path to the Schema is “schematest.xsd”. This is a common situation that people end up in; not being able to specify a relative or absolute Schema URL that works for all users, they just assume that the Schema is in the same directory as the XML file (i.e. simplest possible relative path), and let end users sort out the problem for themselves.

Alternatively, you can use the following XML Catalog file:

<?xml version="1.0" encoding="UTF-8"?>
<catalog xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
   xmlns="urn:oasis:names:tc:entity:xmlns:xml:catalog">
  <systemSuffix systemIdSuffix="schematest.xsd"
uri="../xsd/schematest.xsd"/>
</catalog>

which contains one rule: if you see a system ID (i.e. a physical schema location) which ends in “schematest.xsd”, then use the file whose path is “../xsd/schematest.xsd”, relative to the catalogue file. So, this one short XML file gives your users the ability to have their schemas in any location that best suits them – if the XML applications that they use are “ catalogue aware”, all they have to do is keep their catalogue file up-to-date.

In the .NET/Mono XML API, absolute and relative URLs that need to be used by parsers and such (e.g. schema locations) are passed to an XmlResolver. The standard version only provides the ability to convert relative URLs to absolute URLs. xmlcatalog.net provides a derived “ XmlCatalogResolver” class that uses your XML Catalog file(s) to re-write or re-direct the URIs that are being resolved. You can re-direct URIs to local files (using “file:///” URIs or relative paths that are relative to the catalogue file), or to alternative URLs. It is straightforward to configure a .NET/Mono XML parser to use XmlCatalogResolver:

using System.Xml;
using System.Xml.Schema;
...
XmlReaderSettings settings = new XmlReaderSettings();


// These two statements are important for W3C XML Schema validation.
settings.ValidationType = ValidationType.Schema;
settings.ValidationFlags =
        XmlSchemaValidationFlags.ProcessSchemaLocation
        | XmlSchemaValidationFlags.ReportValidationWarnings
        | XmlSchemaValidationFlags.AllowXmlAttributes
        | XmlSchemaValidationFlags.ProcessIdentityConstraints;


settings.ValidationEventHandler +=  new ValidationEventHandler(ValidationHandler);
settings.XmlResolver = new XmlCatalogResolver(catalogPath);


XmlReader reader = XmlReader.Create(xmlPath,settings);


// Read (parse) and validate the file.
while (reader.Read());

By the way, my intention is that xmlcatalog.net should support both .NET and Mono, but I'm having some trouble getting NUnit to run under openSUSE 10.3, so I don't have my test suite running. It may already work under Mono (if you try it, let me know); when I get it tested against Mono, I will post an update.

At the download site

http://secure.mileywatts.com/confluence/display/CMSW/Miley+Watts+XML+Catalogs+for+.NET+and+Mono+(xmlcatalog.net)

you can register to get access to our JIRA issue tracking system, if you need to search the existing issues or log new issues (bugs, feature requests, etc.). We are already using xmlcatalog.net as part of a customer project, and it's working well. If you use .NET or Mono for your XML, give it a try and let me know what you think.

Genericode – 3rd public review, and XML 2007

The 3^rd public review of genericode, the OASIS XML format for code lists, began yesterday and runs for 15 days. This is likely to be the last review before genericode 1.0 is published as a committee specification (it will then start the process to be voted as an OASIS specification).

At the  XML 2007 conference, on the 4^th December, there will be an evening of lightning rounds on standards. Each speaker presents 20 slides for 20 seconds each. I will be there for the OASIS Code List Representation Technical Committee (CLRTC), with 10 slides on genericode, and 10 slides on the CLRTC's new work, context value association. This allows you to associate particular elements in an XML document with a genericode code list; it's a piece you need if you are using genericode remove code lists from your XML schemas, so you aren't forced to republish your schemas every time a code list changes.

See the “ Genericode News” section at  genericode.org for more details.