More Concise ODRL and RightsML Expressions in XML using DRY Asset Patterns

DRY

Don't repeat yourself (DRY) is a much-loved principle of software engineering.

Essentially, the DRY principle states that you should strive to express any particular piece of information only once. This simplification typically improves maintainability and understandability. Systems which require you to repeat information are known by DRY-adherents as "WET" systems (for "We Enjoy Typing" or "Write Everything Twice").

DRY ODRL

In ODRL (and hence RightsML) every permission or prohibition refers to a particular asset.

When you have more than one permission or prohibition to express for a particular asset, this means you need to repeatedly refer to the same asset. How do you avoid repeating more than absolutely necessary to identify which asset the permission or prohibition constrains?

XML Linking

One solution for minimizing the amount of XML Markup is to use XML Linking. Here's an example where you identify an asset o:asset/@id=as1 and then use an @idref to refer to it in other permission or prohibition blocks.

<o:Policy xmlns:o="http://www.w3.org/ns/odrl/2/" type="http://www.w3.org/ns/odrl/2/Set" uid="http://example.com/policy:Z1XZ">
  <o:permission>
    <o:asset id="as1" uid="http://example.com/music:1234908" relation="http://www.w3.org/ns/odrl/2/target"/>
    <o:action id="ac1" name="http://www.w3.org/ns/odrl/2/play"/>
    <o:constraint id="c1" name="http://www.w3.org/ns/odrl/2/spatial" operator="http://www.w3.org/ns/odrl/2/eq" rightOperand="http://www.itu.int/tML/tML-ISO-3166:it"/>
    <o:party id="p1" uid="http://example.com/sony:10" function="http://www.w3.org/ns/odrl/2/assigner"/>
    <o:party id= "p2" uid="http://example.com/billie:888" function="http://www.w3.org/ns/odrl/2/assignee"/>
  </o:permission>

  <o:prohibition>
    <o:asset idref="as1"/>
    <o:action idref="ac1"/>
    <o:constraint name="http://www.w3.org/ns/odrl/2/spatial" operator="http://www.w3.org/ns/odrl/2/eq" rightOperand="http://www.itu.int/tML/tML-ISO-3166:fr"/>
    <o:party idref="p1"/>
    <o:party idref="p2"/>
  </o:prohibition>
</o:Policy>

Embed the ODRL inside the Asset Markup

The other thing you can do is to have markup for the asset itself (rnews:Article in the below example) embed the ODRL policy. This means that the ODRL is even more concise, as it only needs to refer to the ID of the asset.

<rnews:Article xml:id="item8HEX">
  <rnews:title>Allies are Split<rnews:title>
  <rnews:description>Rebel fighters take control...<rnews:description>

...

  <o:Policy xmlns:o="http://www.w3.org/ns/odrl/2/" type="http://www.w3.org/ns/odrl/2/Set" uid="http://example.com/policy:ABAABA">
  <o:permission>
    <o:asset uid="#item8HEX"/>
    <o:action name="http://w3.org/ns/odrl/vocab#distribute"/>
    <o:constraint name="http://www.w3.org/ns/odrl/2/dateTime" operator="http://www.w3.org/ns/odrl/2/gteq" rightOperand="2011-11-11"/>
  </o:permission>
</o:policy>

...

</rnews:Article>

AWS, SQS and Poisonous Items

I've been using Amazon Web Services to handle large scale processing of content. One handy AWS is SQS, the Simple Queue Service. This is great, since it allows you to decouple (and hence scale) your processing. (There are other advantages besides). However, I've encountered a problem with queues that I've nicknamed "The Poisonous Item". I thought I would share it, together with a couple of workarounds (but not solutions) for handling it.

Q is for Queue by Darren Tunnicliff
http://www.flickr.com/photos/darrentunnicliff/3717976312/

A Typical Architecture Using SQS

Let's pretend that you have a system that uses SQS to process XML documents and update them to your Amazon S3 storage. The processing of each XML document can take a variable amount of time (e.g. depending on the size or complexity of the document), so you decide you want to make it scale nicely. You therefore create an application that writes the document ids to an SQS queue and then you create a second application that performs the processing on each document and writes the result to S3. Because each document can be handled independently from all of the others, you can therefore run as many instances of the document processing application as you like, in parallel, fed by your SQS queue. This can be pictured as in the diagram below.

Visibility Timeout

The SQS queue has a visibility timeout, with a default of 30 seconds. What this means is that when a service fetches something from the queue, the item is hidden for a period of time, to allow for the service to handle it. If all goes well, then the service deletes the original item from the queue. However, if the service crashes, the item becomes visible on the queue again (because the visibility timeout expires). This is all a good thing, since it means that your service is reliable in the face of problems (like an instance of your XML processing application crashing).

Poison by Thorius
http://www.flickr.com/photos/thorius/288024760/

Poison Items
The “poison” item scenario is as follows: there’s some problem with one particular item, e.g. it is really huge and takes, say, 10 minutes to process. That means that the item will timeout and become available to each of the consuming services in turn, whilst the others are still processing it. (I also call this the “Titanic” effect where a safety measure actually makes something more vulnerable to certain issues).

The problem, of course, is that every single instance of your XML processing application will eventually be "poisoned" by the long-running item. In the best case, one of the applications eventually completes and removes the poison item from the SQS queue. Even in this case, however, your applications are doing a lot of duplicate work.

So, how can you try to cope with this?

I need a timeout by Ruth Tsang
http://www.flickr.com/photos/ruthtsang/7247429542/

A Longer Timeout

The first workaround, of course, is to bump up the visibility timeout to be higher than the default 30 seconds. This is relatively simple to do and can completely eliminate the poison item problem altogether. Exactly what level to pick for your timeout is, of course, very much dependent on your application (if it is very high - in the hours - then maybe you need to break your processing into smaller steps?) But you still need to have a timeout, to cope with the legitimate problem of a crashed system.

One rule of thumb is to estimate two standard deviations for the range of processing times. That way, if your processing times conform to a "bell curve" (more formally, a normal distribution) then your timeout will cover almost 98% of the situations your application will encounter. But you also need to weigh this against having too many items in the queue be invisible, since it might mislead you into thinking your processing is complete, when it isn't. Or, if you're using autoscaling, it might result in winding down servers too quickly (since legitimate items might be invisible too long).

Jude'll Fix It no. 103 by Derek Davalos
http://www.flickr.com/photos/derekdavalos/9203747318/

Fix It!

The other workaround is to try to "fix" the reason for the lengthy processing time. Of course, this is extremely dependent on why the times are variable in the first place. In my situation, my XML application assembles smaller documents into larger documents and runs them through an XSLT. Since the number of subdocuments can vary considerably, the processing time varies just as much (if not more so). So, my "fix" was to limit the total number of subdocuments with a reasonable upper limit.  This kind of limit might not work for you (and is certainly a workaround).

Potatoes-Kipfler-HeatAffectedHarvest-928 8-2040gram

Potatoes-Kipfler-Heat affected harvest 2040gram by graibeard
http://www.flickr.com/photos/graibeard/4121218392/

Suggestions?

What else can you do to work around the "poison item" problem?

I have been playing a lot with Amazon Web Services. For numerous reasons, I principally like to use these key bits of software in the work I do:

Python
lxml
s3cmd

As a consequence, I find myself repeatedly doing the following steps to bring the Amazon Linux up to scratch for what I need. I thought I would document them here, in case anyone else finds these steps useful. But also as an easy way for me to find it again...

To Just Generally Bring the Server Up To Date
sudo yum update

Amongst other things, this brings you to Python 2.6, which is sufficiently up-to-date for what I need. (By the time you or future me reads this, I suppose it might be more up-to-date than that).

Install lxml
lxml is the best library I've found for working with XML in Python. It is compatible with, but offers lots of nice enhancements beyond, the standard elementtree, including better support for XPath and built in support for XSLT processing.

Based on this very handy blog post, I do the following to install lxml

sudo yum install gcc
sudo yum install python26-devel
sudo yum install libxslt
sudo yum install libxslt-devel
sudo yum install libxml2-devel
sudo easy_install libxml


That last step to install libxml can take a few minutes. But the whole thing typically takes perhaps ten minutes.

Install s3cmd
Since I do a lot of work with Amazon s3, it is handy to have a command line interface to list, get and put files to s3 buckets. I tried following the instructions about how to install s3cmd on the site, but it just wouldn't work. So, now I do this and it works like a charm:

sudo yum --enablerepo epel install s3cmd

And you can run s3cmd --configure to set up and test out the s3 configuration, if you like.

XML Namespaces

If you only take away one thing from this blog posting, let it be this:

XML Protip:

If you can’t figure out why your XPath expression isn’t working, check the namespace.

XML Namespaces

It is possible to define XML elements and attributes in different namespaces.
This promotes reuse of existing vocabularies because, rather than copying someone else's elements and attributes into your schema, you can just directly incorporate their XML vocabulary into your instance documents.

Namespaces Look Like URLs

XML Namespaces look like URLs, although there is no requirement that there be anything in particular that you can retrieve at the end of the URL. But URLs are convenient because they are a well-established way to have a federated naming scheme. In other words, if I own an Internet domain, I can make up URLs within that domain without needing to consult any centralized authority. That means I can use URLs to mint new, guaranteed unique namespace identifiers.

Declaring Namespaces

You can use xmlns to declare a default namespace

<newsItem xmlns='http://iptc.org/std/nar/2006-10-01/'>

  <itemMeta>
    <title>Pope Blesses Astronauts</title>

  </itemMeta>
</newsItem>

In the above example, newsItem is in the http://iptc.org/std/nar/2006-10-01 namespace.
itemMeta and title are also in the http://iptc.org/std/nar/2006-10-01/ namespace, even though they don't have any xmlns listed on the elements. That's because child elements inherit the namespace from their parents.

XML Namespace Prefixes

You can use xmlns:prefix to declare a namespace and bind it to a prefix

<nar:newsItem xmlns:nar='http://iptc.org/std/nar/2006-10-01/'>
  <nar:itemMeta>
    <nar:title>Pope Blesses Astronauts</nar:title>
  </nar:itemMeta>
</nar:newsItem>

newsItem is in the http://iptc.org/std/nar/2006-10-01/ namespace. And so are itemMeta and title: to an XML parser, this document and the previous one are identical.

Whether you use a prefix on each namespaced element or you leave them unadorned is a matter of style and personal preference.

Namespace are Useful, but can be Confusing

Although namespaces are important in constructing modular XML documents and avoiding re-inventing the wheel of vocabularies, they can be quite confusing. In particular, the non-prefix syntax (in which an element inherits its namespace from its parent) can catch you out in various ways.

For example, if you were to just copy and paste the "inner" markup from the first document above, you'd wind up with the document below - and lose the namespace!

  <itemMeta>
    <title>Pope Blesses Astronauts</title>

  </itemMeta>

This is bad.

Similarly, it is routine to look at an instance document and construct an XPath such as this to pick out the value of the title element:

itemMeta/title


And spend hours trying to figure out why it isn't working. Of course, you know that it is because we didn't specify that the itemMeta  and  title elements need to be in the http://iptc.org/std/nar/2006-10-01/ namespace.


Hence my number one tip for debugging XML-related code:

XML Protip:

If you can’t figure out why your XPath expression isn’t working, check the namespace.

Updated

Paul Kelly adds his number one XPath tip: check the spelling first.

Adding Foreign Namespace Support to the NITF XML Schema

As previously discussed on this blog, NITF has very limited support for foreign namespaces.  I've been experimenting with ways to remedy this and presented the results at the most recent face-to-face IPTC meeting (in Rome).  I have posted the NITF slides on slideshare.

During the meeting, it was decided to break the NITF 4.0 effort into two:

• NITF 3.6 release, which would directly address the addition of foreign namespaces and could be approved as early as January 2011
• NITF 4.0 which would add support for G2 features (such as qcodes) and will likely require significantly more work to develop and approve

It was decided to take this approach, since foreign namespace support addresses pressing needs (in fact, many people do not realize that it isn't legal already to mix and match NITF with other XML schema).  It was also felt that this change is relatively minor and so doesn't merit a major version number change (not sure I agree with that, but ...).

Therefore, I have now created an NITF 3.6 set of schema files, fixing various bugs in the previous experimental schema and adding expansion points that were missing.  So, compared with NITF 3.5, the experimental schema

• Added any attributes in globalNITFAttributes and commonNITFAttributes
• Added any element into head, body, docdata, body.head, block, enriched text, after body, media, body.end

I've also versioned the ruby files to 3.6 and altered the comments to discuss NITF 3.6, rather than NITF 3.5. Finally, I've created an NITF instance that exercises the various foreign namespace capabilities.

http://groups.yahoo.com/group/nitf/files/schema/nitf-3-6.xsd
http://groups.yahoo.com/group/nitf/files/schema/nitf-3-6-ruby-include.xsd
http://groups.yahoo.com/group/nitf/files/schema/nitfastronautNSspan.xml

These files are also available via the NITF 3.6 directory on the IPTC website.


Comments?  Questions?  Critiques?