kontrawize

Getting your XML into MarkLogic Server (with a focus in FpML), part 2

Earlier posts in the series

1. Getting started developing with MarkLogic Server (semi-structured database and app server)
2. Getting your XML into MarkLogic Server (with a focus in FpML)

(and check out the comments on these posts, there are some great ones)

In the previous post on developing with MarkLogic Server, I had got as far as using the simple 'Load' tab in the admin console web page to load FpML 4.7 XML Schemas into MarkLogic. In this post, I want to discuss how you view the contents of the database. As I mentioned earlier, databases can be all too easy a place to lose data, and that is because the tools focus more in specific queries than they do on general browsing. While it may not be possible to just "browse" a database containing a huge number of documents (i.e. a huge amount of data), a development setup often has much less content, and being able to browse the documents (in the manner of browsing files in a file system) is a very useful thing to be able to do.

A good place to start is by downloading the 'CQ' application from the MarkLogic developer site. This allows you to run XQueries on a MarkLogic database, but it also allows you to view what files are in your database. Note that while 'CQ' is a tool I would always recommend installing for development purposes, it isn't a tool you should leave installed on a live production system.

1. Download 'CQ' from 'http://developer.marklogic.com/code/cq'. Assuming you are running MarkLogic Server 4.1, download 'mark-logic-cq-4.1-1.zip'.
2. We will install 'CQ' into the existing 'Docs' database that is part of the default MarkLogic Server setup. This database uses a filesystem directory. Go to your MarkLogic Server installation directory, and find the 'Docs' subdirectory.
3. Extract the 'CQ' ZIP file into the 'Docs' directory, preserving the directory structure. This should create a 'cq' directory inside the 'Docs' directory.

It's that easy, 'CQ' is now installed. Let's test it. First, note what the URL is for your admin console. It will be port 8001; my admin console URLs start with 'http://localhost:8001/'. You probably won't have 'localhost', as by default MarkLogic uses the machine name. Make a note of your admin console domain name, and then use that domain to replace 'localhost' in this URL: 'http://localhost:8000/cq/'. You should now see the 'CQ' page:

Just above the 'Query results' section is a 'content-source' drop-down list. Select 'FpML-Schemas' as the content source, and the click the 'explore' link near the top of the page.

You should see results for 37 files appear in the lower section of the page. The exact URIs will depend on where the FpML XML Schemas were on your file system when you loaded them.

So, now you have a basic way to examine what documents are in your database. You can also browse the documents by running an XQuery in 'CQ'. In the list of 'Queries' at the right of the window, click on number 1 and replace the query with

xquery version "1.0";
(: list all docs in database :)
for $i in collection() return document-uri($i)

Make sure that the content source is 'FpML-Schemas', then press the 'TEXT' button. The URIs for all of the documents in the 'FpML-Schemas' database are listed.

Another way to browse the documents in your database is to create a WebDAV server for your database in MarkLogic Server. That will be the subject of the next post in this series.

Getting your XML into MarkLogic Server (with a focus in FpML)

In my previous blog post, I wrote about getting set up for doing development with MarkLogic Server, which is a database and app server for semi-structured data like XML. This time, I want to focus on getting content into MarkLogic Server. When you start developing with MarkLogic Server, or any similar database, you are likely to want to pre-load some information into it. In my case, I wanted to load FpML Schemas, validation rules and examples (FpML is the Financial Products Markup Language, http://www.fpml.org/). It's taken me longer than I expected to load these files the way that I want them in the database, as I get used to what approaches do and don't work with MarkLogic Server, so I want to discuss what I found by experience (and would welcome comments on anything I've done that could have been done more easily another way).

One thing about semi-structured databases is that when you load a block of information (e.g. a message or document), the content tends to stay together in the database (with some exceptions for huge documents). So, if you load an XML document into an XML database, you can extract the same document at a later time. This differs from what happens when you put semi-structured information into a relational database. In the case of FpML, if you use a tool to create an equivalent relational schema for you, you end up with a couple of thousand database tables, which is a bit of a nightmare for even experienced database developers to work with. Additionally, while the tools are able to "burst" your FpML message and populate the tables with the information, I'm not aware of any simple way to request that the database return to you all of the information from a particular message. That's sometimes an important thing to do — an FpML message typically represents the details of a trade of an OTC derivative, and there are real business reasons for wanting to retrieve all of those trade details together. The SQL required to do it is a significant and expensive piece of work to write by hand. Indeed, many teams don't bother, they store the original message separately to the database, because they know that the relational database is going to make hard work of it. While that kind of archiving is OK for ad-hoc auditing purposes, it doesn't help if one of your use cases is to be able to present screens showing the details for particular trades. Where there are good reasons to keep particular sets of related data items together and not have them shredded into separate tables, a semi-structured database can save a lot of development effort.

Here is what you get with an FpML release (they are free to download, but you have to register). I'm taking FpML 4.7 (2nd recommendation) as my example):

When unzipped, there are top-level 'documents', 'html', 'pdf' and 'xml' directories. We'll mainly be interested with the 'xml' directory. It contains the FpML XML Schemas, and its sub-directories contain FpML examples, including some invalid examples for negative testing. However, the other interesting directory for use with MarkLogic is 'html/validation-rules'. Here there are XQueries (*.xq) which implement extra validation rules for FpML, extra rules that XML Schemas can't check.

Before you import a large number of files into a database like MarkLogic, it's a good idea to think about how you want to organise things. One of my traditional criticisms of databases, and the way people use them, is that it can be too easy to "lose" documents or data, lose it in a sea of other information. With MarkLogic, documents are stored in 'Forests', and one or more forests can be part of a 'Database'. An app server uses particular databases. So app servers use databases, databases use forests, forests contain documents. However, there are some other things to think about too.

Every document in the database has a URI to identify it. These are sometimes proper URLs that start with 'http://...', but they can also be just paths like '/AAA/BBB/CCC' (so they aren't constrained to be proper URIs, in spite of what they are called in the documentation). Documents are organised in a hierarchy of directories, like a file system, but each document can also be part of any number of 'collections'. Collections allow you to group any set of files together, regardless of where they are located in the database. So, when importing files, it's worth spending a bit of time thinking about how you want the document URIs to be structured (i.e. how you want your database's directories to be structured), and also what collections you should create to group related files to each other (just to make it easy to find those files later, as a related group).

Additionally, there are relationships between databases. For each database, you can specify which database contains the XML Schemas for validating the XML (it can be the same database, by the way). Additionally, for each app server you can specify a database for 'modules' (XQueries), although you can also choose to store your XQueries in the file system where MarkLogic Server is installed. What this means is

• you need to decide whether your XML will be store in one or more forests, and in one or more databases;
• you need to decide whether your XML Schemas will be in the same database as your XML files, or a different one;
• you need to decide whether your modules (XQueries) will be stored in a database, or in the file system.

Let look at doing the following:

• creating a new forest and database for the FpML 4.7 XML Schemas;
• importing the FpML 4.7 Schemas into that database.

(In later posts, we will look at importing the other files, and doing some validation.)

In the MarkLogic admin console (usually port 8001), go to the 'Forests' section:

Click on the 'Create' tab, and create a new forest called 'FpML-Schemas'. You only need to fill in the name (for the purposes of this exercise), then you can press 'ok'.

You will now have an empty forest for the FpML XML Schemas. Now go to the 'Databases' section of the console:

Click on the 'Create' tab, and create a new database called 'FpML-Schemas' (it's OK to have the same name as the forest):

After you press 'ok', you will find that you are prompted to select one or more forests for the database:

Click on the 'Database->Forests' link and select 'FpML-Schemas' as the forest for the database (remember, we gave them both the same name):

You now have a database with a forest, ready for loading your FpML XML Schemas. The quickest way to load the FpML XML Schemas is to go to the admin page for the 'FpML-Schemas' database, and click on the 'Load' tab. Put in the path for your FpML 4.7 'xml' directory, and use '*.xsd' as the filter. This will load all of the XML Schemas in the 'xml' directory (but not in any sub-directories).

When you press 'ok', you are shown a page with the files that will be loaded, and the database URI that will be assigned to each. You should find that the URIs are identical to the file paths. Press 'ok' to start loading the Schemas. In a few moments, the FpML XML Schemas are loaded. Press 'ok' again. The status page for the 'FpML-Schemas' database should now show that you have some documents in it:

How do you see which XML Schemas have been loaded? How do you control the URIs that are assigned to them when they are loaded in the database? How do you put them in collections? These are things we will look at in the next installment.

Getting started developing with MarkLogic Server (semi-structured database and app server)

I spend much of my professional life working with XML.  I often design XML structures that are used to represent financial information as it is transmitted between computers, but I also spend quite a bit of time looking at tools to support this usage of XML.

To be honest, I'm often a bit horrified by how people set up their software to work with XML. An all too common setup for a system to read incoming XML is:

1. Compile the XML Schemas into Java or .NET using a Schema binding tool (problem #1 - the XML is your decoupling mechanism, to help isolate your systems and stop them from being too tightly coupled to each other, but when you compile a Schema, you tightly couple your code to the XML that is supposed to be your decoupling mechanism.  If you use Schema compilers and every change to your XML's structure over time causes you headaches, you might need to reconsider your use of a Schema binding tool);
2. Create a staging relational database, and write code to store the information from your Java/.NET objects into that database (problem #2 - it can be a lot of work to create a relational model of a hierarchical document, like an XML message.  It is also a lot of work to write the code to do the storage. Do you really want to spend that time? Also, people often model all of the XML in the staging database, even if it won't all be used. That means the system ends up coupled to all changes in the XML, even those that would otherwise not have impacted the system - not a good plan.);
3. Write some more code to copy the information from the staging database into a separate application database, often with a different structure (so the code that you have to write isn't trivial).

How would I do it differently?  It is much less effort to

1. Store the XML in an XML database (i.e. in a semi-structured database). In particular, use an XML database that can store arbitrary XML, not one that requires you to have a Schema for your XML;
2. Query information out of the XML database as your application requires it (typically using XQuery, although your XML database may also support JDBC/ODBC views of the information, to simplify integration with existing apps).

Notice the difference - no time spent writing staging database schemas, no time spent writing code to populate the staging database.  You simply write the queries that you need to get the information out of the XML database, and you should pull the minimum amount of data possible when doing that so that you minimise the coupling between the application and the XML.

In fact, it's worth noting that using an XQuery (or XPath) to pull the information from your XML database provides the same level of decoupling that SQL and views provides when working with a relational database; that use of a query to provide decoupling is so common now with relational databases that people think of it as always having been that way, but it wasn't.  There were software tools in the 80s that wrote C++ code that was tightly bound to the database schema, and that code had all the same problems as we have now with code generated from XML Schemas.  The relational database folks learned that you always need a SQL query to providing decoupling from the physical storage structure; the XML world has been a bit slower to realise the same thing, but it's an important lesson - use XQuery/XPath/XSLT to decouple yourself from the physical structure of your XML and you take a lot of pain out of your application development.

It can be hard to get people to move away from the relational database mindset, in spite of the extra costs of development.  However, various people in the finance world seem to be looking seriously now at semi-structured databases (which includes XML databases), and the one that gets mentioned most often is MarkLogic Server.  I should say that I know quite a few people now who work for MarkLogic (they've been hiring of late, and picking up some really top class people).

So, I've decided to set up a proper MarkLogic development environment for myself (I've only toyed with it in the past).  You can download a development version from the MarkLogic developer site, where there is also a lot of developer documentation.  It doesn't run on Macs yet, but it does run under Windows, Linux and Solaris.  I couldn't get it to install under Windows 7 (Vista is what is on the supported list), so I installed it under Ubuntu.  Ubuntu isn't officially supported but my friends at MarkLogic told me how to install it; if you want to know, let me know.  It's pretty easy, and it's working well for me.

With MarkLogic Server, your installation can run multiple app servers (HTTP, WebDAV, or XDBC [direct code connection]).  Each app server uses certain databases, and each databases uses one or more 'forests' of documents.  The documents are stored compressed on disk, so it doesn't waste space like some XML databases that store them on disk as uncompressed text.  The documents are automatically indexed, but you can also manually adjust how they are indexed (like any database, there is an art to doing that well).

The default setup has a 'Docs' app server which used a 'Documents' database, and these are what you should use in your initial experiments. The first thing you will want to do is load some documents and try out some queries. This can be a bit daunting, it isn't as obvious as it should be how you do this. I believe MarkLogic are working on improving this, but also it's because MarkLogic Server is a secure, industrial-strength database (used by the U.S. security services among others), and so you have to learn to negotiate the permissions. In general, the quick answer is that for your initial learning with MarkLogic Server, use your admin login for everything (i.e. while you are just experimenting, and don't have anything too valuable stored in the database). I spent too much time trying to set up a non-admin user for development work, but it isn't a good way to get to know MarkLogic, so don't make my mistake.

The 'Docs' app server uses the 'Documents' database. Get started by reading the admin guide and create a WebDAV server and an XDBC server for the 'Documents' database (it's pretty easy, I won't describe it here). This allows tools to connect to the database. In particular, I have two working now:

• MarkLogic has an Eclipse plugin that sets it up for XQuery development (available from the MarkLogic developer site). It runs with the Galileo 3.5 release of Eclipse, not with the latest Helios release (trust me, I tried, it doesn't). This gives you syntax support for XQuery, including a knowledge of MarkLogic-specific XQuery functions (MarkLogic Server extends XQuery in the same way PL/SQL or TransactSQL extend SQL - to turn it from a query language into a full programming language. However you can also run pure standard XQuery 1.0 if you want). You can run your XQueries from within Eclipse just as you would run a piece of code;
• oXygen's XML editor, a favourite of mine, also does a great job with MarkLogic Server. It can validate your XQueries using MarkLogic's validator (so it picks up knowledge of MarkLogic-specific functions, etc.), and it can run those queries on MarkLogic Server and return the results, just as easily as Eclipse.

Either of these tools is a great way of getting up and running with MarkLogic Server. You can also install 'CQ' (http://developer.marklogic.com/code/cq) which is a neat little Web UI for running XQueries. It remembers your queries, which is nice, but it doesn't have the syntax checking and colouring that Eclipse and oXygen provide. CQ is well worth installing, but a tool like Eclipse or oXygen is really necessary to be productive (and that is what this is all about, after all, being productive in development, avoiding the effort and cost of unnecessary development).

To load files in a database, go to the admin console (port 8001) and look at the details for that database. There is a 'Load' tab that is the simplest way to get content in. However, it doesn't do what I want, to load content from a whole directory tree while filtering on file names. I am working on an XQuery for that, I will post it later when it is working. Otherwise you can also use WebDAV to drag-and-drop files and directories into MarkLogic Server. I'm going with the XQuery because it will allow me to set things like permissions as the files are imported, and I think that will prove useful, if a bit more involved.

I will write more in the future. I have outlined what I'm doing now. Please feel free to post questions (I know I've skimmed over a lot of details). I will try to answer any questions, or I'll ask one of my MarkLogic friends if they have an answer.

Interrogate, Report, Act - a layered approach to validation & checking

Yesterday I attended a standards meeting (ISO TC68/WG4, ISO 20022, message modelling for banking & finance).  After the meeting, some of us went to a nearby pub for a drink and a chat.  While we were talking, we touched on the subject of validation, and I mentioned my personal preferred approach - "Interrogate, Report, Act".  It seems to resonate with the others, so I thought it might be worth mentioning here.

One thing I've done a few times for banking clients is to build systems for quality control and release of large sets of XML Schemas.  Just as you check code before a release - compile it, test it, run style checks over it - you also need to do the same kind of checking for XML Schemas.  You test that they are valid Schemas, you regression test them using XML examples, you run style checks over them, and perhaps also generate code or other artefacts from them.  What you do, whether it's for Java/C# or whether it's for XML Schemas, is to check some kind of IT resource using some set of rules.  You can call it validation, but that's just a long word for checking.

If you are checking XML documents using a set of structural/formatting rules defined by an XML Schema, you call it validation.  People are used to validators that check XML and log errors, just as they are used to compilers that check code and log errors.  It's very familiar, it's a common model for how such tools work.  However, I've found it to be a problem when I'm building XML Schema checking frameworks.  Why?

One reason is that, with checks like style checks, today's disallowed style is tomorrow's allowed style, and vice-versa.  As organisations develop their XML Schemas over time, they adjust their style checks.  Type extension or substitution groups might be banned today, they might be allowed tomorrow.  To build a flexible checking framework, you want to be able to change quickly the interpretation or the severity of a particular feature, of a particular style.  This is why I found it best to break things up into the three stages - "Interrogate, Report, Act".

Interrogate

Understand the resource(s) you are checking, the structure, the formatting, the style, whatever you need to understand. Don't make any judgements at this stage - the judgements are the things you need to be able to change, so you don't want to embed them into the interrogations.

Report

Put the results of your interrogations into a common reporting format. If you have XML Schema validation results, Schematron validation results, ad-hoc XQuery/XSLT validation results, and Schema compiler validation results (as an example), they won't all be in the same format. Some might be XML, others will be text. Get them into a common format for reporting. That's important so that you can slice and dice your interrogation results, and display them in a consistent way that gives developers, testers and managers the most appropriate summaries. It's also important so that you can easily add new results without impact on your presentation and drill-down code.

Act

Once all interrogation results are in a common reporting format, make your judgements and perform any consequent actions. Throw exceptions, log errors or warnings, or choose to ignore particular results if that is what the system's users have configured (there can be good reasons for ignoring particular test results for particular resources - but for sanity, make sure you capture the reasons for doing so in the configuration information).

This is the layering of concerns that has worked well for me. I've used it for XML and XML Schemas, but it is a general approach that can be applied to any kind of validation or checking process. By not making too early a judgement of thumbs up or thumbs down, you will have a more flexible checking framework that is more easily configurable and extensible than it would otherwise have been.

New version of ISO 20022:2009 UML to XML Schema generator available

Londata, my company, has made a new version of the Londata UML Message Generator available, version 1.0-preview-3.  This is (I believe) still the only available software that implements the new ISO 20022:2009 methodology for generating XML Schemas from UML models (as opposed to the older 2004 version of the standard).  ISO 20022 is the ISO standard for XML messages for banking and finance, as used on the SWIFT network.  ISO 20022 XML Schemas aren't hand-edited in an XML Schema editor as happens with some standards.  Rather, they are all generated from UML models so that there is consistency across the XML Schemas, and so that human errors are minimised.  The Londata UML Message Generator allows companies to use the same ISO methodology for their in-house XML Schema development and management.

Updated example UML models are now available - MagicDraw and Enterprise Architect example models, and Londata's conversion of the official MagicDraw UML profile into Enteprise Architect format.  Enterprise Architect isn't supported yet by the Londata UML Message Generator.  The Enterprise Architect examples were done to prove the feasibility of supporting Enterprise Architect for ISO 20022 modelling, MagicDraw having been used during the development of the 2009 version of the standard.

If you have any questions or comments, they are very welcome, please send them to [email protected]

Want to meet up in NY, 19-25 June? I have something to show you.

I haven't been to NY for a while, but I will be visiting from 19-25 June.  I will be attending the SIFMA Financial Services Technology Expo from 22-24 June, and also doing some meetings.

As well as visiting the Expo, I will have the latest version of the Londata UML to XML Schema generator, and would be happy to give demos.  The software implements the new 2009 version of the ISO 20022 process for generating XML Schemas from UML.  ISO 20022 is the XML format that SWIFT is increasingly using on its network, replacing the older ISO 15022 messages.  I'm happy to come and discuss why the methodology that works for SWIFT and ISO can work for your company too.

In addition to Londata's product, I will also be able to demo a companion product from Nomos. Nomos's tool reads business rules from your UML models and converts them into Java code that validates your XML messages.  This is validation that you can't do with XML Schemas, and for many companies it is the missing piece that is needed to catch errors that can otherwise lead to major delays in testing and roll-out of new systems.  I've seen first hand that if systems that send XML messages go into UAT with too many errors in the messages they are sending (and believe me, they all too often do), it has a huge impact on the testing schedule of the receiving systems, and nobody can really afford to have new systems and new business revenues delayed by weeks or months just because of inadequate checking on XML messages during the development/test/UAT cycle.  The Londata and Nomos tools together give companies a chance to address that problem and rein in their schedules.

If you would like to meet up while I am in NY, please feel free to get in touch; my e-mail is [email protected].  If 19-25 June doesn't suit, or you aren't in New York, please get in touch and we can set up an online discussion or demo to suit you.

Tim Bray, Developer Advocate

It's great to read that, having decided not to stay with Oracle after its takeover of Sun, Tim Bray has decided to take a role at Google.  He's "Developer Advocate", with an initial focus on Android, Google's phone OS.  The reason I'm writing about this is because I was so disappointed to see some of the negative comments about Tim having decided to work for Google.

I've known Tim for quite a few years now, a decade or so, and everyone greatly respects Tim for his honesty and for his determination to stand up (quite publically) for what he believes is right.  At the same time, he also listens to those around him, he is never closed to other people's opinions, he never ignores those opinions (albeit he can't agree with everyone else's opinions, and it would be silly to expect him to).

I'm not an Android developer, I don't have an Android phone.  However, if I was an Android developer, I couldn't imagine having a better advocate than Tim.  If anyone is going to help keep Google honest, keep them doing the right thing by developers, it's Tim.  While I don't expect that he will be the one with the final say on all things Android, I'm sure he will influence Android for the better, and you can't ask for more than that.

I used to work for Reuters, which was also sometimes criticised for making decisions that some people felt weren't in the interest of its customers and partners.  My experience from that time is that big companies aren't so much evil as they are sometimes inept or ignorant.  They don't always accurately gauge how the world around them will respond to the decisions that they make on their products and their general corporate behaviour.  I'm not sure that Google is any better or worse than other big companies in this sense, but I hoping that Tim will get every opportunity to help Google make the right decisions first time.  As a great listener, he has a great sense of what people really want.

I wish Tim every success, and I hope that everyone will give him the support to make his time at Google a success that benefits everyone.

Prague at Night (photos)

I'm in Prague for the first time, to speak at the XML Prague 2010 conference.  Last night, I took these photos from the Charles Bridge, near the conference venue in the old part of Prague.

• 
• 
• 
• 
• 
• 
• 
• 

The full-size photos can also be found here: http://public.fotki.com/CoatesFamily/public/tony/prague-at-night/

Videos & photos from Vienna ...

I've just returned from a week in Vienna for a standards meeting (UN/CEFACT ATG/TMG joint face-to-face meeting).  These are some videos and photos from the trip.

First is a building by the Donaukanal (Danube Canal) that puts on a cool light show in the evening.  It starts slowly; it's worth watching to the end.

I had the great fortune to be invited to the annual ball of the Vienna Technical University (Technische Universität Wien).  It's a wonderful, formal occasion (I had to borrow a bow tie to wear), held in the the old emporer's palace.  The following video of the dancing in the ballroom gives you some idea of what it was like, although the sound doesn't do any justice at all to what like to be there.

I took some other videos there, these aren't as good, but might be of interest to other attendees of the UN/CEFACT meeting.

• http://www.youtube.com/watch?v=4p0eg_saBYM
• http://www.youtube.com/watch?v=7WEJuWVDDyU
• http://www.youtube.com/watch?v=Gd0re0qTgR0
• http://www.youtube.com/watch?v=LqzD0ZWqvLw

I also took some still photos in Vienna, mostly at the ball.  Here are some.

• 
• 
• 
• 
• 
• 
• 
• 
• 
• 
• 
• 

The rest can be seen here: http://public.fotki.com/CoatesFamily/public/tony/uncefact-vienna-2010/

Photos from around Surrey Quays

I've posted some photos I took around Surrey Quays in London in December 2009.  They are from the first photos that I took with my Pentax k10d that was a Christmas present.  The k10d isn't the latest model, rather it's now a pretty good deal second hand.  It doesn't lose a lot to the latest models (unless you want your SLR to take video as well).  The full set of photos is here

http://public.fotki.com/CoatesFamily/public/tony/surrey-quays-2009/

and here are some examples:

• 
• 
• 
• 
• 
•