kontrawize

Converting a MagicDraw UML Model into Enterprise Architect format

I like UML for modelling, in spite of it being too focussed on object-oriented design.  However, interoperability between UML file formats is woeful, even though there is a common XML format for UML, called XMI.  Unfortunately, each UML tool vendor has found a different, incompatible way of using XMI, it seems.  It really holds back UML tools; we live in a world where people now expect interoperability.  The interoperability between UML tools is vastly less than, for example, the interoperability between OpenOffice and MS Office.

However, the "300lb Gorilla" for many vendors is Eclipse, which has its own built-in UML editor.  While the Eclipse implementation doesn't have the same degree of development and features as the commercial tools (although making great strides), the commercial vendors can't ignore it because so many organisations use Eclipse now as their primary Java IDE.  This means that many UML tools can import and export Eclipse ".uml" and/or ".uml2" files, so the Eclipse format was recently recommended to me as a better way (than XMI hacking) for moving model between different UML tools.

This morning, I tried this, using the ISO 20022:2009 draft profile as the example.  This contains both the ISO 20022 profile (stereotypes) and the ISO 20022 type library.  From MagicDraw, I opened "unifi-profile.mdxml" and then selected "File | Export To | EMF UML2 (v2.x) XMI File".

 

I saved to an empty directory, where it created 13 ".uml" files, including "unifi-profile.uml".  That's one thing about the Eclipse format, you end up with lots of files, not a single "project" or "model" file like other UML tools.

I then started Enterprise Architect (EA) and created a new, empty model.  In the project browser, I right-clicked on the top level "Model" node and selected "Import Model from XMI...".  This opens the "Import Package from XMI" dialog, which contains a button "Import EMX / UML2 Files".  Pressing that button gets you to an alternative dialog which allows you to import your Eclipse model.

However, when I went to import "unifi-profile.uml", I couldn't because EA expected UML2 files to have a ".uml2" extension, while MagicDraw saves them with a ".uml" extension.  Luckily, the solution was as easy as renaming "unifi-profile.uml" to be "unifi-profile.uml2" (note: without renaming any of the other ".uml" files).

Did it work?  Well, partially.  It did import the type library, which is good.  A couple of bogus double-entries were created, but these were easy to find as they didn't have stereotypes like the actual type library members.  EA didn't import the actual profile (stereotypes), which wasn't a big a surprise to me.  EA doesn't yet support the full range of UML 2.x metaclasses (these represent UML constructs - classes, attributes, associations, etc.), while MagicDraw does.  That's a pity, it means I will have to construct the profile by hand, as step towards providing EA support in the Londata UML Message Generator.

Generating XML With Scala

As I mentioned in "Generating XML Schemas from ISO 20022 UML models", I am using Scala to write an implementation of the ISO 20022 method for generating XML Schemas from a UML model.  The code is in two parts; the first part processes a MagicDraw UML model via the MagicDraw Open API (a Java API) and produces a simple XML representation of the ISO 20022 message information that is contained in the UML model.  That simple XML representation can then be processed to produce resources such as the XML Schemas; in a later iteration, I also plan to support generation of ASN.1 specifications for the same messages.  ASN.1 is the way that ISO 20022 messages will be able to be binary encoded rather than XML encoded, for situations where volumes require something more compact than textual XML. At the moment, I have just finished writing the code for generating the XML Schemas from the simple XML representation (having previously written the code to generate the simple XML representation from a MagicDraw UML model).

Back to Scala and XML, a major reason for using Scala rather than Java for this code is that Scala has better built-in support for XML. In spite of some really top XML people having worked at Sun in the last decade, the XML facilities in Java have never been more than an awkward tacked-on piece that continues to make Java developers resistant to using XML.  Scala goes a long way to addressing that problem, and making XML straightforward to work with.

For example, in Scala I can write

val x = <document><h1>Example of inline XML in Scala code</h1></document>

which creates a value "x" of type "scala.xml.Node". Note that I didn't have to put the XML in quotes to convert it into a string.  Scala knows what XML is, so you can use XML directly.

If you need to insert values into the XML, you can do it like this:

val famousText = "To be or not to be"
val speaker = "Hamlet"
val famousPhrase = <phrase speaker={speaker}>{famousText}</phrase>

and the value of "famousPhrase" is

<phrase speaker="Hamlet">To be or not to be</phrase>

Braces are used to indicate where something in the XML should be intepreted as code.  For attributes, if the attribute value is contained in braces, the enclosing quotes are omitted.  Some of you may think that this looks not unlike ASP or JSP, and in some ways it does, but it is neatly integrated into Scala.  These expressions can appear in any Scala code, they aren't forced into separate files as you typically have with ASP and JSP.  Also, the resultant expressions are understood by the code to be XML, not just text with angle brackets.  I can extract values from the XML using a path notation:

val extractedSpeakerAttr = famousPhrase \ "@speaker"
val extractedSpeaker = extractedSpeakerAttr.text

The result of this is that "extractedSpeakerAttr" is an XML node sequence with one node, an attribute; "extractedSpeaker" is a string with the value "Hamlet".

This is why I like Scala for working with XML. It's a full general purpose programming language, like Java/C#/etc., but the code you write for working with XML is short and to the point, comparable to XQuery. For my XML Schema generator, it seems like the best solution. Brevity is still one of the best ways to keep your code maintainable, and to keep the cost of that maintenance down to a minimum.

By the way, if you want to try out these Scala example, just download and install Scala, and type "scala" at the command line. That starts the Scala interpreter; you can type the above code, and it will print out the results. It's an easy way to get a feel for Scala and what it can do.

Generating XML Schemas from ISO 20022 UML models

As I've mentioned in the past, I'm the principal UK expert (yes, the "PUKE") for ISO TC68/WG4, the ISO working group that has produced an updated version of the ISO 20022 standard. ISO 20022 is standard for defining messages based on definitions in a central repository, with a focus on banking and finance. Those of you whose are involved with financial messaging via the SWIFT network will have used ISO 20022 messages whenever you use one of the XML "MX" series messages. WG4 has produced a 2009 version of ISO 20022; it is currently a committee draft which has gone to TC68 for an international vote, so it is not yet an approved version; the current approved version is the original 2004 version.

My particular interest in ISO 20022 is in the part that describes how to generate XML Schemas from UML models. I like the repository based approach where you can share components between related messages, but don't have to implement that sharing at the Schema level, with includes and imports. ISO 20022 has one Schema file per message, and that makes life easier than if you have to get people to set up a hierarchical directory of cross-linked Schemas correctly.

I wrote an XML Schema generator for the current 2004 version of ISO 20022. That generator works with XMI (UML in XML) files produced by Enterprise Architect (from Sparx Systems), and is implemented as a series of XSLT scripts.

For the new 2009 version of ISO 20022, the good people at MagicDraw donated licences for WG4 to use their UML editor, and so MagicDraw is the UML editor that was used to develop the new metamodel and UML profile. For this reason, for my new implementation I decided to target MagicDraw (I might support other UML editors too, later, depends on how many requests I get).

I decided to continue with my previous approach, working from the XMI, this time using a mix of XQuery and XSLT. XMI is a rather generic XML format that can be hard to read and understand, so I consider it error-prone to try and generate XML Schemas from the XMI directly. Instead, I distill the XMI into a much simpler XML format which reflects the structure of the ISO 20022 metamodel, an "XML DSL". I then process that simpler XML file to generate Schemas or anything else. So, the plan was to use an XQuery to generate the simple XML, and then XSLT to generate the Schemas (there are some things that are easier in XSLT than XQuery).

However, it turns out that the latest XMI 2.1 standard is loose around how UML stereotypes (user annotations) are represented. So, after some failed attempts at "reverse engineering" how MagicDraw represents stereotypes in XMI, I decided I needed a different approach. Instead, I am using the MagicDraw Open API to read the contents of an ISO 20022 UML model. This means I need to be using a Java program, or something that runs on a JVM.

Java itself isn't great for XML, in as much as the code tends to end up rather more verbose and unreadable than it need be. I tried Groovy, which has some reasonable support for XML, but moved away from that because the XML support is incomplete. It seems that in a number of languages, the people who implemented the XML library didn't really like XML, so they decided to only implement support for the parts of XML that they thought were worthwhile. For users, that causes nothing but problems.

What I am currently using is Scala, which is convenient because you can just start typing XML as the value of a variable in the middle of a program, and Scala knows what to do. I've now written the code to generate the simple XML format. It hasn't been as easy as it should have been, because Scala support in IDEs is still weak (I use Intellij), and because Scala's integration with Java is good but not as good as it could or should be. For example, Scala avoids the use of null values. It does have some support for them because it needs that to work with Java libraries. However, at times when Java code would give me a null pointer exception, I find my Scala code just hangs, leaving me with a lot of searching to do to find what went wrong and where. Still, the code is more readable than Java, and things can only get better, so I'm planning to stick with Scala for this.

Once I have XML Schema generation done, the next thing to look at is ASN.1. ASN.1 ("abstract syntax notation") is an ISO standard for designing messages that predates XML, and has been most widely used in the telecommunications industry. Note that I wrote in a previous post that ASN.1 was the format used for ISO 15022 messages (more-or-less the predecessor to ISO 20022). That was wrong, I want to point out; somewhere along the line I got the wrong idea, and in spite of my saying it in a number of WG4 meetings, only recently did someone from SWIFT point out that I was mistaken.

Anyway, WG4 has added support in the 2009 version of ISO 20022 for ASN.1 as an alternate message syntax to XML Schema. Now, ASN.1 itself is actually multi-syntax. There are a couple of binary encodings it supports, plus there is a more recent XML encoding. The ISO 20022 profile of ASN.1 has been constructed so that, as closely as possible, an XML message that is valid according to the XML Schemas will also be valid as an XML encoded ASN.1 message, and vice-versa. Since ASN.1 messages also have a binary encoding, this new ASN.1 support means that ISO 20022 will have an official binary encoding as well as the existing XML encoding. It will be interesting to see how the usage of XML versus binary ISO 20022 messages goes over the next few years; banking and finance applications have size and latency challenges for which plain-text XML often isn't suitable.

I should be able to generate the ASN.1 grammars for the messages from the same simple XML that I use to generate the Schemas. Doing so will show that the simple XML genuinely decouples that back-end code generators from the format of the input UML model; that's something I need to have the possibility to support UML editors other than MagicDraw in the future.

If any of this interests you, either ISO 20022 or my work on Schema and other generators for ISO 20022, feel free to get in touch.

Data modelling, metamodelling and MPS

One side of the work I do for my clients is XML and related technologies, the other side is general data modelling. One thing that interests me in data modelling is the use of metamodels. Now, what is a metamodel? Normally, people switch off or break into a cold sweat when they hear "meta-anything" in a sentence. However, there is nothing scary about metamodels. A metamodel simply defines what things are allowed to be in your data models; it's a model of what you are allowed to have in your data models.

Why do we need metamodels? It's because data modelling is usually done using general purpose modelling technologies. Examples are E-R diagrams, UML, and XML Schemas. Any of these general purpose modelling technologies give you more features than you need or want in any single project. It can be a major problem to constrain people to use as few features as possible, which is what you want to make your data models as simple to understand and as cheap to maintain as possible. The role of a metamodel is to define a limited set of features which are the only ones that can be used out of all of the features that the general purpose modelling methodology provides.

For example, it's common to find people using XML Schemas that (a) use elements only for data, not attributes, and (b) use local element definitions only in the XML Schemas, not global element definitions (root elements accepted). This constraining of XML Schema to a simple subset of its features is equivalent to defining a metamodel that is a subset of the full metamodel for XML Schema. In practice, you can implement just such a "metamodel validation" on an XML Schema by applying some kind of schema language to the XML Schema itself. As it happens, XML Schema editors often won't let you apply an XML Schema to another XML Schema, but you can apply both Schematron and/or RELAX NG to XML Schemas in order to implement checks that the XML Schemas conform to your metamodel (i.e. your design rules).

UML is widely used for data models, and metamodels for UML are written in using a thing called MOF. With the latest versions, UML 2 and MOF 2, MOF is now a subset of UML that can be used for writing metamodels which constrain UML models. Sadly, support for validating UML models using MOF metamodels is still poor (at least in any tools I have used), but I live in hope that within a couple of years it will be possible to validate a UML model with a MOF metamodel as easily as you can validate an XML document with an XML schema.

Today I was catching up on some reading, and I stumbled across a mention of JetBrains Meta Programming System (MPS). I tried an early version of this a couple of years ago; it lets you design custom/bespoke mini programming languages (or mini data formats) that are specific to a particular purpose. The MPS tutorial is a good way to get a feel for MPS, what it does, and how it works. There is an obvious analogy between defining a mini-language for a task rather than letting people loose with a general purpose language, and using a metamodel to constrain people's usage of the features of a general purpose modelling methodology. MPS allowed you to define structured data models that have something of the feel of (simple) textual computer program, or a textual data file like a simple properties file. Although they have the look of text files, they are actually structured, and in some ways it's not unlike generating form-based user interfaces or Web pages. MPS is a neat looking tool, and it's free. I like JetBrains (commercial) Intellij IDE for Java, because it's so clever in the degree of analysis it does of your code as your write it. I was amazed how many issues it was able to point out to me as I was writing my code, before I even compiled it. If the JetBrains folks can apply the same cleverness to MPS, it could be a useful tool. That said, it's not the only tool that can generate user interfaces from models. Eclipse has EMF (like a cut-down version of UML) and Visual Studio has support for Domain Specific Languages (DSL). All of these tools are working to the same end, to provide you with a way to allow people to work with only the minimum set of features that are needed for any particular problem, so that they produce the simplest, most consistent, most maintainable solution that they can.

Metamodels and their equivalents let you take a complex software tool or methodology, and constrain it to effectively turn it into a simpler software tool or methodology. It's a bit more effort, but for a project of any size, it can pay real benefits in the end.

ISO 19312 Securities Data Model (and MDDL)

David Winter posted a comment, noting that he had been unable to find the ISO securities data model in the ISO Web site (that's financial securities – stocks/shares/etc.). The reason that it isn't there is that ISO 19312 is still a work in progress, it hasn't yet been approved (indeed, it hasn't yet started reached the ISO voting stage yet). As such, it isn't yet generally available, although you may find that you can get a copy of the current draft of the data model from your local ISO/standards representatives if you are involved in banking and finance.

David is currently looking at MDDL, which is the best place to be looking in the meantime. Indeed, ISO 19312 will only cover a subset (more or less) of what MDDL covers, at least initially. David seems a little perturbed or surprised that MDDL puts trading terminology (snap, time-series) at the top of its document hierarchy, but in my experience most surprises that people have about standards are usually due to people assuming different requirements to those that were used (or otherwise disagreeing with the requirements that were used). MDDL, for example, was originally designed for distributing financial market data (from stock exchanges and such), so it's not surprising that it talks about snaps (snapshots) and time-series, which are what people look at when they look at market data. MDDL isn't just a financial instrument data model, as ISO 19312 is (but note that ISO 19312 also covers corporate actions – changes to the details of securities because companies have merged, split, changed name, or because of other similar reasons). David considers MDDL to be inverted, so I guess the way he would have done the requirements would have been inverted, in some sense, as compared to the way they were done (and there's often no good or bad in these things, just choices that have to be made, one way or the other).

For anyone who is looking at MDDL as a data model, note that the most versions of MDDL have their data model defined in an XML file (e.g. “mddl-2.5-beta.xml”), which is a simple XML format constrained by a Schema (e.g “mddl-2.5-beta-schema.xsd”). The latest version of MDDL, 3.0-beta, comes with the data model in the form of a UML model (and documentation generated from that model is supplied with the release).

If you feel you must reverse-engineer a data model from the MDDL Schema, please use the “lite” or “base” version of the Schema, as the full Schema implements a number of MDDL-specific “ shorthand” structures which don't make sense if directly reverse-engineered as a data model.

Versioning data models; it's not just the model that counts!

At the risk of repeating things I've covered in previous posts, one things that I'm often involved in is the question of versioning policy for UML data models and XML schemas. Now, the policy "question" is usually about how you can make changes (usually additions) to a model/schema with minimum impact on applications which use the model/schema, and ideally with zero impact on applications which can legitimately ignore the changes.

This all sounds reasonable, but the problem is that when these policy decisions are made, they are often done is a way that is almost sure to fail. Why? Well, the problem is that people often treat this as a model/schema issue only, as though some particular versioning approach in the model/schema alone is going to be the silver bullet that smooths out all of the potential downstream issues related to model/schema changes. However, it just doesn't work that way.

You can't have a workable policy on managing data change unless your applications are part of that same policy. There is no silver bullet that can work regardless of how your applications have been written. To go back to the relational database example that I sometimes use, in the bad old days when code generators wrote code that was tightly bound to each table and column in a database, database changes almost always broke the applications that used the database. The situation with databases is very different now, because most people follow the policy of interacting with their relational database via SQL (often using ODBC/JDBC), and that SQL layer (when properly used) provides sufficient decoupling to help avoid a lot of potential downstream issues, especially when all that has happened is an addition to your data model.

By contrast, with XML, most applications do not routinely access the XML via an XQuery or an XSLT transform, or via some other layer which would provide the same level of indirection and decoupling that SQL does for relational databases. Instead, a lot of use is made of XML schema compilers, which do the same tight binding that the old relational database schema compilers used to do, with all of the same downstream data change problems.

This is why versioning policy has to apply right along the data processing chain. You need to have good versioning policies for your models, but that alone isn't enough. You also need to have application design policies that match your model/schema policies so that the sum of the two is a robust approach to data change management.

After all, there's no point in building a bridge that only goes half-way across a river, is there?

Adding Interface Restrictions to Java and Other Object-Oriented Languages

I've mentioned before that I think the classical object-oriented programming approach works very well in some contexts, but it isn't useful in all contexts (and indeed, can have a negative impact in some contexts).

As someone who works with XML a lot, I often run into problems with the fact that object-oriented languages allow you to derive new classes by extension, but not by restriction. By contrast, XML Schema types (simple and complex) can be derived by restriction, and it often makes perfect sense to do so. So why don't object-oriented languages support derivation by restriction? [Well, most don't support it, although I think Objective-C may support it.]

It's all because of polymorphism. The idea (as most of you will know) is that if ClassB and ClassC are derived by extension from ClassA, then ClassB and ClassC objects have all of the features of A objects, and can be used wherever a ClassA object can be used. This can be very powerful, I don't deny that, it is often very useful indeed.

However, there are many cases where you don't want all of the features of an existing class, just some of them. What happens if ClassR is derived from ClassA by restriction? Well, you can't use a ClassR object in place of a ClassA object, because it doesn't have all of the methods/fields/etc. of a ClassA object. On the other hand, you can use a ClassA object in place of a ClassR object - since ClassR is a restriction of ClassA, ClassA is an extension of ClassR, and extension allows you to do polymorphism.

So, what derivation by restriction gives us, for object-oriented languages, is a way to define new superclasses of a class after the original class has been defined. Normally, only subclasses can be created after the original class has been defined, unless you go and edit the original class (which isn't always possible or appropriate). [In the example above, ClassB and ClassC are subclasses of ClassA, ClassR is a superclass of ClassA.]

In practice, a difficulty in adding support for derivation by restriction is that there can be complex inter-relationships between the methods and fields of a class, and it may not be simple (or possible) to remove something without breaking something else. This certainly applies to the implementations of methods.

What you can restrict, though, are the available non-private (public, protected, package) signatures of methods and fields. That is to say, you can restrict the external "interface" (in the most general sense) that the class presents to other classes. For Java, this works most obviously with interface classes.

Suppose I have some interface:

interface MyInterface {
  void methodA();
  void methodB();
}

As Java is now, I can extend this interface:

interface Extended extends MyInterface {
  void methodC();
}

and Extended is an interface with 3 methods: methodA, methodB, and methodC.

How would restriction work? Well, in XML Schema, it works by explicitly stating the restricted content model of the derived type, and the derived type is then checked to be a proper restriction of the base type. Similarly, it would be straightforward (I believe) to extend Java (etc.) so that in an interface restriction, you list the methods that are in the derived interface, and these have to be a subset of the methods in the base interface. For example

interface Restricted restricts MyInterface {
  void methodA();
}

would be a valid restriction. However,

interface Restricted restricts MyInterface {
  void methodC();
}

would be invalid (because there is no methodC in MyInterface), so the compiler would flag an error.

You could do a similar thing with implementation classes (not just interfaces), by checking that the restricted class has a set of non-private method/field signatures that are a subset of those in the base class. Indeed, you could also derive interfaces by restriction from implementation classes this way, which would sometimes be useful.

Caveats? Well, this style of derivation by restriction means that the derived class/interface can become invalid if the base class/interface is modified. While that may sound like a problem, it isn't a practical problem in XML Schemas, and I don't think it would be in Java (etc.). It's just something you would know can happen when you use derivation by restriction, and that knowledge would help guide you on whether derivation by restriction was appropriate in a particular context, or not. After all, implementation classes can become invalid when interfaces change, and that is just something we've all learned to live with.

I'm interested to know what you think. Do you agree or disagree? Leave a comment if you have an opinion. Would it be worth proposing it as a JSR (feature addition) for Java? What do you think?