Lessons Learnt from JSON Designs I've Worked On
Over the last couple of years, I've worked on a few JSON schema. For example, IPTC's NINJS (for representing news) and W3C GC ODRL's ODRL in JSON (for representing permissions and restrictions). I've also done some work on JSON internal to AP, for various APIs and search systems.
Along the way, I've learnt some lessons about better or worse ways to design the JSON - both about the way to do it and some JSON "style" tips. I've broken this into three posts:
JSON Style
Here are some principles, heuristics and lessons learnt I've gathered whilst working with and on various JSON formats. I'll expand a little on each one, below.
Simpler JSON Means Simpler Code
JSON's built in datatypes - number, string, Boolean, array, object and null - are a natural fit for many programming languages (particularly scripting / high level languages such as Javascript, Ruby and Python). A large part of the attraction of JSON over XML, therefore, is that it is often easier to deal with data expressed in JSON in those languages.
However, crucially, it makes a big difference as to how the JSON is structured. The simpler the JSON structures are, the easier it is to write correct code to deal with them. This is part of why it is so important to prototype some code to work with a structure you are considering, as it gives you a much better sense of how easy it is to work with a given JSON structure. This principle (Simpler JSON Means Simpler Code) underlies and/or is counter-balanced by most of the rest of the following principles.
Choose Wisely: Once You Commit to a Structure, You Can't Change It (Easily)
Once you declare a given property as having a particular structure, you can't change it (easily). For example, let's say that you want to represent a date in your JSON. You decide that you will call it "arrivalDate" and that you will make this a number (which means it is easy to do date arithmetic, for example):
"arrivalDate" : 20140515
Later, however, you realize that you need to indicate in which timezone this date occurs. Sadly, you can't just tack on a timezone offset indicator (such as "+0500" or "-03") since that would make arrivalDate a string and any existing JSON documents that conform to your first definition would now be invalid.
It is annoying to have an array if there's only ever one item
In XML, you might decide you want to represent a headline like this:
<headline>Dog Bites Man</headline>
And that, since a story can only ever have one headline, your XML schema confidently states that there must be exactly one headline. Later, you realize that there are multiple kinds of headline. Since XML was designed with eXtensiblity in mind (it is the "X" in XML), you can alter your XML schema to allow for multiple headline elements:
<headline>Man Bites Dog</headline>
<headline>This is news!</headline>
The good news? Your old XML documents with a single headline are still valid, according to your new schema.
However, once you have a JSON document like this:
"headline" : "Dog Bites Man"
It is incompatible for a document like this - a JSON property can't be both a string and an array of strings.
"headline" : ["Man Bites Dog", "This is news!"]
So, you might be tempted to construct your JSON defensively - to "future proof" it - by making your properties into arrays, so that you can easily have more than one. However, it becomes really tedious to have to access an array of things when there is only ever one item. (Remember: simpler JSON means simpler code).
Avoid making things arrays, "just in case".
Use the headline / headlines "cheat"
One technique I've used when having to switch from single instances to array properties is to name my properties carefully. When a property is an array, I make it a plural. This leaves open the possibility of having a singular property - such as "headline" - and then later adding an array property with the plural name - "headlines".
Flatter is better
In XML, it is natural to have multi-level structures. When XML is pretty-printed, the indentation of enclosing elements helps to make the document structure easier to grasp. It is tempting to do the same when designing your JSON representation. However, in the spirit of "Simpler JSON Means Simpler Code", it is much easier to deal with your JSON if there are an absolute minimum of "grouping" structures in your JSON.
In the early drafts of IPTC's NINJS, we initially grouped different types of metadata together (into administrative metadata, descriptive metadata and so on. This distinction is a useful one and is still reflected in the "data model" diagram for NINJS:
 |
NINJS Data Model
http://dev.iptc.org/ninjs |
However, once we started to create examples, we realized that it was much better to lose those groups in the actual JSON markup. In particular, it is difficult to query JSON using complex criteria. In part, this is due to a lack of standards for how to specify JSON queries. So, the less structure the better.
Use pattern properties to strike a balance between flexibility and interoperability
Most of the JSON formats I've worked on are designed to be used by multiple systems (whether internally to
AP or as a standard to be used by many publishers). To help ensure that multiple implementations wind up using your JSON in compatible ways, you want to restrict the degrees of freedom to interpret the format in different ways. On the other hand, you need to allow for future requirements so that your JSON formats will be adopted and adapted, rather than discarded as being too inflexible.
It is pretty easy to add something unknown into a JSON format - just add another property. Implementations should implement the "
must ignore" pattern so that they don't break when they encounter an unknown property. However, sometimes, you want to guide future changes to the format, so that certain predictable changes are all done in the same way.
I've found it useful in JSON Schema to use "
pattern properties" to strike this balance. For example, in NINJS, we wanted to allow publishers to use whichever geographic geometry JSON they wanted (to represent lat/long shapes for centroids and the like). We therefore added a patternProperty of "
geometry_*" to allow publishers to use a property with a name starting with "geometry_" and then a suffix to indicate which type of geometry they are using.
Start property labels with a lower case letter
Whilst developing IPTC's NINJS, we experimented with various libraries, to make sure what we were producing would work everywhere. One gotcha we tripped across: the
Java Jaskson library chokes on JSON properties with an initial uppercase letter. It was the only library we discovered that had this problem, but why take the risk?
Use a very restricted set of characters when naming properties
Since it is very common to autogenerate code from JSON property names (such as Java or .Net classes), it makes sense to restrict the character set you use. In order to maximize compatibility across libraries and languages, we determined that this is the safest set of characters to use:
[a-zA-Z_0-9]
i.e. upper and lower case alphas, numbers and underscore.
Inline text markup alternatives
One of the areas where both XML and HTML excel is the rich markup of text, particularly via inline markup. (For a couple of examples using IPTC standards, check out this example in
NITF and this one using
Schema.org-compatible rNews).
There are different possible ways to tackle rich text markup in JSON. Three alternatives we identified within NINJS are:
- Strip out all the inline markup and just leave the plain, unmarked text. Probably fine for short bits of text but tedious as soon as you have any structure - such as paragraphs, never mind hyperlinks. Could be useful for things like indexing in a full text search engine, though.
- Keep the marked up text (such as HTML) in a string, escaping as necessary. Particularly good for delivering bits of web-ready text that can be integrated into a larger page.
- Mechanically translate the original markup into JSON structures. JSONML is a nicely-documented example of this approach. However, given that I advise against this mechanical approach in the first place, I would be very careful before adopting something like JSONML for your text markup - for all the same reasons.
Which method you pick needs to be driven by your particular requirements. It isn't a bad idea to consider having the text represented more than one way, though, if you can afford it.
Support APIs with a full / partial representation indicator
JSON has gained traction as a format for use in APIs. Performance is a key factor in most APIs, so you may well want to deliver an API result in JSON with just a key subset of properties. In which case, you should consider adding a property that indicates whether this is a full or partial representation of the given resource - ideally along with a property that lets you retrieve the entire representation.
When we came up with this idea whilst designing NINJS, we toyed with having a way to describing more than just two possibilities (full/partial). However we decided that - in the general case - those are the only two that matter.
JSON Design: A Series
This is the third and final posting in my series on JSON Design. Part one discussed an approach to designing JSON schema. Part two discussed JSON tools and standards.
