kontrawize

Disk partitioning for Linux, and the Virtual Laptop

I've talked in the past about how much I like OS virtualisation software, like VMware (and I'm keeping an eye on Xen too). I have been successfully using VMware to run Windows XP on my laptop (which had SUSE Linux 9.3 Pro installed).

However, it was time to upgrade to SUSE Linux 10, and so I decided that the time was right to build a virtual laptop. I would have two virtual sessions, one SUSE 10, the other Windows XP. Both would run on top of SUSE 10.

Why do something like this? Well, eventual convenience. If I need to run my laptop's Windows XP installation on my deskside PC (e.g. if there is a problem with my laptop), I can copy it over in about 10 minutes, and it just runs. Using the SUSE backup/restore, it recently took me 10 hours to do a restore, and that was excluding my /home partition, which is bigger than the other partitions combined. Being able to migrate my work quickly and seamlessly from one machine to another is something I've always wanted to do, but was never able to achieve. Software to help you migrate is getting better (the Windows XP migration tool is not bad, and Tim Bray reported similar results for Macs). However, it's still a bit of a chore. Moving virtual sessions is easy - once you get them set up.

Herein lies a long story, and why I haven't posted much in the last week or two. To build my virtual laptop, I had to migrate my SUSE 9.3 laptop installation into a virtual session running SUSE 10. Along the way, I learned a lot, because I did just about every possible wrong thing. I won't try to list them all, but I will try and tell you what works.

Let's start with disk partitioning. I recently wrote about how I partition my disks for Linux, and John Cowan rightly questioned my choices. The biggest failing of my partitioning scheme was that I only had 1G for the root (/) partition, and this turns out to be too little for the /tmp directory (which was part of that partition). That problem was when doing backups, where I was running out of space in the /tmp directory. It seems that I needed at least 1.5G free in /tmp (and that was for a backup that excluded the /home partition).

So, I either needed more space in the root (/) partition, or /tmp in a larger partition (perhaps of its own). Also, I realised that putting /usr and /opt in separate partitions wasn't very helpful, since they are just two sides of the same coin; no point putting aside a lot of space for /opt when most Linux applications seem to install themselves in /usr (RPMs are fine, but they don't give you choices). So, I made big changes to my approach.

Along the way, I trashed the disk structure on my deskside system. There was a problem when I tried plugging in a USB hard disk to a virtual SUSE session in VMware 5 on SUSE 10. It worked, until I tried to log out, when it all came down around my ears. I ending up have to reformat the disk and re-install SUSE 10 on the deskside system. Luckily I don't keep any important stuff on it at the moment. My new partitioning scheme, based on what I had learned, was

1. /boot, 16M, ext filesystem
2. swap partition, 2G
3. /, 15G
4. /home, rest of 300G drive.

Simplicity, less chance of running out of /tmp space, but still separate /boot and /home partitions. So it's a middle ground between where I started and what John Cowan suggested. With this, I installed VMware and then created a SUSE 9.3 virtual session that I would eventually migrate to my laptop. This was a very good idea, to build the new session first before touching my laptop. I got it wrong many times, and was glad that after some failures I could just go back to my untouched and working laptop, and then just start over again later.

One thing to note with VMware is that you can't increase the maximum disk size that you set when you create a virtual session, so you have to choose carefully. With 60G on my laptop disk, I would afford 20G for the SUSE session (my Windows XP session is 20G). 20G is OK for me since I can leave larger, and less frequently used files on the physical filesystem and access them from the virtual session as required. My partitioning approach was

1. /boot, 16M, ext filesystem
2. swap partition, 1G
3. /, 10G
4. /home, rest of 20G virtual drive.

What worked was to do restore a backup of my laptop (minus /home) into the virtual session. I use rsync to backup /home, as it is a faster way to backup the data you need to backup most often (and /home isn't filled with system files that need special handling). So I used rsync in the reverse direction to get my /home files into the virtual session.

Once I tested that my virtual session was working reasonably (and most importantly, that I could read/send e-mail as normal using Opera), I reformatted my laptop disk for SUSE 10:

1. /boot, 16M, ext filesystem
2. swap partition, 2G
3. /, 10G
4. /home, rest of 60G drive.

As well as holding the virtual session files, this base installation will also hold (a) my Subversion version control repository, (b) installation files, and (c) music/video files, since these all need to be shareable between my SUSE Linux and Windows XP sessions. I installed VMware on my laptop, and then copied my two virtual sessions (SUSE 9.3 and Windows XP) across from my deskside PC.

First trick - VMware is tricky to set up for a widescreen display (my laptop is 1280x800). It can be done, check out the VMware forums or send me an e-mail.

Second trick - I wanted to run the SUSE 10 upgrade installer in my SUSE 9.3 session. For VMware, you have to run the installer in text mode, and say no to the "Test Internet Connection" option. I couldn't get the installer into text mode because I couldn't get my SUSE session to boot from the installation DVD! It turns out that to do this, you have to get to the setup screen in the VMware virtual BIOS. You hit the escape key for this, but you only have less than 1 second, so be quick.

Anyway, now it is all running, and I'm using my SUSE 10 virtual session now as I'm typing this. It was a pain getting here, but I think it was worth it. I'll let you know how it works out going forwards.

A Versioning Odyssey, part 5

Previously: part 1, part 2, part 3, part 4

Just a quickie. In part 4, I left out another version component - the build number. This is typically just a number that increments for each build of the software (whether released to customers or not), although I have also seen the "days since first build" used (that was for a C++ project, where more than 1 build a day was out of the question).

So, my previous example "Gruntmaster 6000.3.1.2" would become something like "Gruntmaster 6000.3.1.2 build 587". I myself tend to keep build numbers separate like this, but if you want to append it to the other version components, go ahead. As long as you have some explanation for your users as to what is what in the version components, you can be flexible.

[ Technorati tags: software architecture ]

A Versioning Odyssey, part 4: human-friendly version labelling

Previously: part 1, part 2, part 3

Human-friendly version labelling. So what is that when it's at home? Sounds a bit grandiose, really. It's what I promised at the end of part 2, but what I really wanted to get across is that it is important to consider people when naming versions, rather than just following arbitrary rules.

I already mentioned my dissatisfaction with simple "major.minor" version labelling, e.g. "3.1". The usual rule here is that you change the minor version if a change is "backwards compatible" (in whatever sense that is appropriately judged), and you change the major version otherwise. The problem is that "minor" changes are sometimes significant to users, while "major" changes (like making an optional item mandatory) are sometimes insignificant to users. This means that this simple rule produces version numbers that don't always change in a way that conveys the significance of the change to users.

I have found this to be a real problem over the years. Sometimes people leave an item as optional, rather than make it mandatory, just to avoid upsetting users with a new major version. When the top-most version number changes, users will always be scared of a big transitional cost about to hit them. There's no point being in denial about this, it's a common thing, and that is that.

OK, so here is my human-friendly suggestion. Comments very welcome, by the way. I suggest that you identify software versions, or versions of anything else, with the following version components:

• name
• product/marketing version (1 or more)
• major version
• minor version
• pre-release type or patch version

I mention the name here first because having a name is an important part of disambiguating whatever it is that you are versioning. The Dilbert series about the "Gruntmaster 6000" wouldn't have been half as funny if it had only been called the "6000", and it might have been confused instead with one of Nokia's phones. So the name counts for a lot.

Next are what I call product or marketing version numbers. These are numbers that you change when you think it makes sense to change them, not because of some arbitrary rule. This is how you convey a sense of significance to users in the version numbering. One such number is often enough, but if you want two, go ahead.

Once you are in control of the most significant version numbers yourself, you can add a major version number and a minor version number in the well known fashion. They aren't at the top now, just in a position where they can convey the information they are meant to convey without sending too wrong a signal to users.

Finally, I recommend a pre-release type or a patch number. A pre-release type is an indication like "alpha" or "beta" to tell your users that a particular version is an early access release of some kind, rather than a production release. Add an extra version number if you have multiple alphas or betas.

A patch number is a less common version number, but no less important. Sometimes, in spite of your best efforts, you release something with a bug. For example, you release a new minor version of an XML schema with a bug which means that it is not backwards compatible. It would have been backwards compatible if it wasn't for the bug, but it isn't. What do you do? Well, you could fix it and release yet another minor version, but that sends the wrong message to users. You don't want users to think that the version with the bug was just another minor version, you want them to be aware that it was buggy and should not be used. A patch version is used for a change which may not be backwards compatible with the thing it patches, but it indicates (a) that a change was made to restore the lost backwards compatibility (or whatever else the bug affected), and (b) that all previous patch versions of the item should be discarded. Yes, this breaks some of the golden rules about major/minor version numbers, but it sends a better message to users than just rolling out yet more major/minor version numbers, and that's important.

In all of the above, where I refer to numbers, you can of course use any sequence, such as letters of some alphabet or other. It helps to have an ordered sequence, though, so that users are in no doubt about which of two versions is the later version.

So, where does that leave us? Well, it means that the "Gruntmaster 6000" would end up with versions like "Gruntmaster 6000.3.1.beta.4" or "Gruntmaster 6000.3.1.2" (2 is the patch version in the latter). This actually gets you pretty close to the Oracle 5-version-number scheme (although, as noted before, I don't know what rules Oracles use for their version numbers).

If this seems complex, well, life isn't always as simple as we want to pretend it is. Simple solutions are a great idea for truly simple problems, but failing to model the real complexity of a problem can get you into trouble. I think that what I have suggested is a fair compromise between all things. What do you think?

[ Technorati tags: software architecture ]

A Versioning Odyssey, part 3

Previously: part 1, part 2

This is a special XML 2005 installment of my versioning posts.  Yesterday I attended a presentation by Henry Thompson (of W3C XML Schema fame) on "Names, Namespaces, XML Languages and XML Definition Languages".  Henry is a member of the W3C TAG, and this work is part of the TAG's investigations into how XML and W3C XML Schema can and should support versioning. Henry's approach was get away from things that people have tried in the past, and just to identify what it is that we are really trying to do, what it is that we actually need to get the right result.  His paper should be available generally sometime after the conference, but basically, he had a couple of key conclusions:

• XML namespaces are not objects with structure, they are just strings (URIs) used to disambiguate short names.  The only reality is that URI+shortname pairs are unique.  It's wrong to talk about "a name in a namespace", because that suggests a structured data content to namespaces that just doesn't exist.  The reality of the way namespaces are used is that people use them solely as strings for disambiguation, just as people use Java package names in Java (the XML namespace concept borrowed from the Java package concept);
• The key thing that we identify with namespaces is names that we have given to particular items.  This includes items that exist within a document, such as element/type definitions in a Schema, or any item in any XML document that has an ID or other key.  In the TAG's "Architecture of the World Wide Web", the recommendation is that named items inside an XML document should be addressable using a URI with a optional fragment identifier (e.g. "http://www.example.com/myDocument.xml#myname").

At the end of the talk, I asked Henry if this name-centric approach meant that the TAG was heading in a direction where all named items are versioned, so that the "version" of a document (e.g. a Schema) was really a configuration of versions of the individual named items in the document.  The answer was a definite yes, since this also allows you, simply and easily, to track what the differences are between versions of the document.  People don't just want to have versions, they also want to know what has changed between versions. I'm very happy with all of this, I have to say.  I think Henry and the TAG are really on the right track with this.  I look forwards to hearing more of this - and the sooner the better.

[ Technorati tags: software architecture, xmlconf2005 ]

Toothing

I read in The Register earlier this year about the phenomenon of "toothing", where people have their Bluetooth-enabled phones configured to broadcast a list of their interests to nearby phones. Your phone then alerts you when someone with compatible interests is nearby. It was a social networking (and dating) technique for the third millenium, I guess. Anyway, The Register's article was to say that it was all a hoax, that the original press article about it was faked.

So I was interested today to read a post by Michael Yuan about the Nokia Sensor application which does exactly what the article described - "toothing" is here.

By coincidence, a few months ago I met a guy who told me that he was the author of the "gaydar" application for mobile phones, so it's not that Nokia is the first to do something like this. Perhaps Nokia Sensor is toothing for the rest of us, though. Well, the rest of us with Nokia phones, anyway.

Optical Illusion

This optical illusion link posted by James Tauber is so fascinating, I had to mention it here. Really funky stuff.

Yes, we DO need binary XML

Rick Jelliffe is a good bloke, don't get me wrong. However, I couldn't agree with his latest post about binary XML. I have written a long comment there, which I encourage you to read.

What I think we all can agree on is that most XML users don't need compressed XML most of the time. Unfortunately, some people, including those in positions of authority at the W3C and other places, seem to have interpreted this as meaning that there is no need for a standard binary compression for XML. I have needed exactly this compression for years now in my work in finance, where the data volumes involved are simply more than the available network bandwidth can support. Everyone used to think that network bandwidth was growing so quickly that the need for compression would quickly disappear, but that hasn't happened in the real world. Even if the main network pipe to a company's door is now bigger, that doesn't mean the internal pipes between their sites have kept up.

For years, people have asked me what would be the next big thing that replaces XML. My answer is that there isn't one - XML is as fundamental as TCP/IP or Ethernet, and nobody asks me what are going to replace these. Nobody particularly wants to replace them, for the most part (except we need IPv6 to get more IP addresses). However, if there is something that could drive people to produce a replacement for XML, it would be big enterprises getting together to come up with a format that can make better use of the available network bandwidth for the huge volumes of data that they would like to be able to move around. It will be a real shame if this happens, since a standard binary XML compression would satisfy this need, and still allow everyone to use the same rich set of tools we already have for XML. The people who need binary XML are a minority, but they are a powerful and important minority, which is something we shouldn't be ignoring.

[ Technorati tags: data architecture, software architecture, XML ]

XML 2005 schedule, part 2

I have changed my schedule for XML 2005 in Atlanta next week, as I am chairing a couple of the sessions. My schedule is now:

Tuesday 15 November

• Keynote: From Atoms to OWLs the new ecology of the Semantic Web
• Keynote: XML and Web Services: A Blueprint for Next Generation Applications
• Linking Outside the Box
• Handling Math in Real-World Workflows: Practical Lessons
• XJ: Integrating XML and Java
• Microsoft's Language Integrated Query and XML
• Names for (versions of) languages: A problem statement and a proposal
• A Tool Kit For Implementing XML Schema Naming and Design Rules

Wednesday 16 November

• Keynote: Plugging into the Pervasive XML Infrastructure
• Keynote: The future of XML Information Management
• System Architecture for XML Offload to a Cell Processor-Based Workstation
• Introduction to the NEW Native XML Support in DB2
• Securing XML - case studies from the Financial Services Industry (chairing)
• ACORD Security Profiles: Securing Insurance Web Services (chairing)
• Analysis of XML schema usage
• Creating and Maintaining Large Families of Related Schemas (this is my presentation)

Thursday 17 November

• Extending the Use of XML in the Enterprise (chairing)
• Making the right constraints for usable and accessible user interfaces (chairing)
• Unit Testing in XSLT 2.0
• Automated mass production of XSLT stylesheets
• "Just" Use XML
• A Generalized Grammar for Three-way XML Synchronization
• Functional XML: An new approach to XML processing
• XML Data Binding: Integrating XML and Object-Oriented Technologies
• Keynote: Everyone's using XML, but does anyone care?

Once again, I hope to see some or all of you there!

[ Technorati tags: xmlconf2005 ]

Identify your data

How do you identify your data? It's an important question, but one that people often ignore when writing applications. What do I mean by identifying data? Well, if you have a database, you have lots of information in it. Usually, each table has a unique key for identifying rows in the table, and that unique key is often an integer. So, if you know the unique key, you can locate the row you want. However, the problem is often the reverse, that you know some of the data, and you want to do a search to find the unique key of the matching row (or rows). Your search might involve other tables too, via foreign keys that link a row in one table to a row in another table. In the database world, this is all perfectly normal stuff, nothing special here.

However, when you look at how applications are written in most programming languages, the situation is different. Objects or structures are identified by a pointer or handle (which is usually implemented as an integer). Most everything that is done with live data in the application is done via these pointers/handles.

What happens, though, if you don't have the pointer/handle for the data you need? You need to do a search of data values that identify the data, the "natural keys" for the data. This is a common thing to need to do in a large system, although programming languages don't tend to provide a lot of support for it, not in the efficient indexed way that databases do. In some cases, the objects may not be "live" per se, they may be been persisted directly into a database, in which case the need to perform an efficient search based on data values is even more important (since you now may have far more data than will fit in memory, so a roll-your-own search from beginning to end could be rather slow).

None of this is particularly new, people come across the problem often enough. What is disappointing though is that there still isn't a well-known way to identify natural keys in your objects/structures, no way to identify the data items which can be used (and perhaps should be used) to identify data. If you look at UML diagrams, they tell show which classes are associated with which others, but they don't identify which data fields are the natural keys. By contract, if you look at an ER diagram (entity-relationship) for a database, it identifies keys that can be used by one table to identify a piece of data in another table. Those aren't necessarily the natural keys, but they can be, that is one of your design options.

So my conclusion of late is that UML is missing something, it is missing a standard notation for indicating natural keys. Now, UML is extensible, so identifying natural keys is something that you would expect to do using stereotypes, which are labels that can be applied to anything in a UML diagram to describe some special feature of it, e.g. that it is a natural key. You could easily imagine having a <<key>> stereotype applied to a field in an class to identify it as a natural (or other) key value. Unfortunately, it's trickier than that, since the natural keys might not be directly contained in the fields, but contained within an object in a field. For example:

+ ParentObject
|
+--  key1
|
+--+ ChildObject
   |
   +-- key2

In this example, there are two key fields (key1 and key2) that are needed to identify the parent object, but key2 is contained within a child object. The question is how you identify in your ParentObject class that one of its natural keys is in the ChildObject class. That is more than simple UML stereotype labelling can solve. At least, I can't think of a good solution yet. I do think we need one, though. Any ideas?

[ Technorati tags: data architecture, software architecture, databases, Java ]

Intel-powered Powerbooks

I see in The Register that the first Intel-powered Apple Macs, in the form of Powerbooks, are due in the first quarter of 2006. When I bought my current laptop, I had a long hard look at the Powerbooks, but was put off by a couple of things. One was that the screen resolution was a bit low compared to some of the other laptops around. The other is that the PowerPC processor meant that running Windows on top of it would be slow, because of the need to fully emulate an Intel processor in software. For better or for worse, there are Windows applications I need to be able to run, even if Windows isn't my primary operating system.

I would be tempted by one of these Intel Powerbooks if VMware is ported to MacOS, but I don't know if it will happen. Apparently you can run the Intel version of Mac OSX in VMware, and that would be cool too if I could run it in VMware on my current laptop.

I don't know what VMware's plan's are, but since Microsoft currently provides a competitor (Virtual PC) and is a major investor in Apple, I can see there could be hurdles to a native Mac OSX version of VMware. I live in hope, though. Maybe someone will port Xen to the Intel Macs, when Xen is more mature.