Falcor and GraphQL: Querying JSON APIs (Part Five of Three)

What's the best way to query a REST API which returns JSON? I look at two popular libraries - Netflix's Falcor and Facebook's GraphQL - which aim to overcome problems with API performance and "chattiness".

Querying JSON

XML (and related standards such as XSLT and XQuery) benefit from the power of XPath for selecting and querying XML. However, JSON has no direct equivalent to XPath. (Although there are a lot of projects which have named them selves [jJ][Pp]ath!)

I still like the approach taken by JSONiq- it is essentially XQuery for JSON. However, in this post, I want to talk about two libraries - Falcor and GraphQL - which address the problem in a somewhat different way: how to get just the JSON you want from an API?

2009APR101606 by Peter Renshaw
https://flic.kr/p/6dYAsw

Trade Offs: Speed and Complexity

When you write a client for a typical REST API, you have to confront two basic problems: performance and complexity. On the one hand, if a REST API contains more data than you need, then you're paying a penalty for every unnecessary byte (being transferred over the network and parsed by your code). On the other hand, if a given API response doesn't have everything you need, then you will need to make follow-up calls, which adds complexity and, of course, more latency as you choreograph the back-and-forth.

The designer of the REST API should try to anticipate likely uses, so that they can provide just the right information, in the right ways. And, as I've previously recommended, it is a good idea to build in support for full or partial API responses. However, part of what is exciting about APIs is that they unlock innovation. So, if your API is a success, you will - by definition - have hard-to-anticipate uses of your design.

Sit! by Craig Sunter
https://flic.kr/p/rZ2tyS

Sitting in the Middle

Rather than rely on the REST API perfectly fitting your needs (or supporting a powerful query language) why not have an adapter which sits in-between your client and the REST API? Both Netflix's Falcor and Facebook's GraphQL take this approach: they are each implemented as servers which you configure to turn the REST API you have to work with into one that you want to work with. They differ somewhat in their philosophy and power, however.

Falcor

Falcor - All of the Data in One Giant Model

Netflix has open-sourced their Falcor library, which they use to power their UIs. At the time of writing, it is still in "Developer Preview", however, many people outside of Netflix are using Falcor. You can try out the demo Falcor application or read the Falcor documentation for more details.

Falcor adds some capabilities to the standard JSON model - such as "virtual JSON models" and a "JSON Graph" - to make it easier to cache data on the client side. Using Falcor, you can

• eliminate the need for multiple HTTP requests to get all the data you need
• cache the data locally for better performance
• deal with data using graphs, which are more flexible than the standard tree-model used in JSON
• adapt JSON or non-JSON APIs into a JSON model customized for your application

Falcor is a server-side Javascript library run within a nodejs server. You construct a Falcor data model and define how each component maps to the actual APIs you need to use via "paths". Your application then interacts with the Falcor data model you've defined, while the Falcor server takes care of interacting with the APIs to get you the data you need, including handling caching for greater performance - particularly when you have multiple instances of your application querying a single data model.

A nice overview of working with Falcor is provided by Auth0. And you can find a lot more documentation on the Falcor website.

GraphQL

GraphQL - a Schema and Resolve Functions

Facebook has open-sourced their GraphQL library, which they developed to power their mobile and web apps. At the time of writing, Facebook has released a working draft of the GraphQL spec and a reference implementation in Javascript. They have also created an implementation you can actually download and use. Various people have started to build GraphQL tools and implementations, including GraphQL support in Python (one of my favourite languages). Check out the GraphQL documentation for more details.

As a GraphQL client, you send the server a query, which defines what data you want back. For example

{
  user(id: "1") {
    name
  }
}

Which says "give me back the name of the user who has an id=1".

On the GraphQL server side, you need to configure the schema and the resolve functions. The schema defines the data model which may be fetched from the server. The resolve functions map the fields in the schema into the backend services. A GraphQL resolve function therefore contains whatever code is necessary to fetch and transform data from a backend service - such as a REST API, a MongoDB or a SQL RDBMS - into the form promised by the schema.

There's a nice overview of working with GraphQL on RisingStack. And you can find Facebook's full documentation on GraphQL.

"Choice" by Jeremy Brooks
https://flic.kr/p/nyPkd2

Which One Should You Choose?

Falcor is somewhat simpler to learn than GraphQL. In part, this is because GraphQL is more powerful - in particular it has a much a more sophisticated query capability. Both libraries have been implemented in Javascript, but only GraphQL is designed to be implemented in other languages, too.

Finally, it is worth considering whether you want to adopt either one at all: the REST architecture (when implemented correctly) has tremendous support for caching and scalability. So, rather than abandon a REST API altogether, consider whether you have the option of instead tuning it to perform better (tip: look at the granularity of the resources you've defined).

Designing JSON

This is part of my occasional series on designing and working with JSON:

• Part 1: An approach to designing JSON schema
• Part 2: JSON tools and standards
• Part 3: Lessons learnt from the JSON schema I've worked on
• Part 4: Heroku's HTTP API design guide

This post - the fifth in the trilogy - picks up on a topic I discussed in Part 3 - Lessons Learnt - how to select and query the JSON you get back from an API.

JSON Design Principles and Lessons Learnt: An Approach to Designing JSON (Part One of Three)

Lessons Learnt from JSON Designs I've Worked On

Over the last couple of years, I've worked on a few JSON schema. For example, IPTC's NINJS (for representing news) and W3C GC ODRL's ODRL in JSON (for representing permissions and restrictions). I've also done some work on JSON internal to AP, for various APIs and search systems.

Along the way, I've learnt some lessons about better or worse ways to design the JSON - both about the way to do it and some JSON "style" tips. I've broken this into three posts:

• An approach to designing JSON (this one)
• Handy tools and standards
• My thoughts on JSON style

Automagic JSON?

One way to create a JSON schema is to automatically generate one from an XML Schema. For any given domain, there's probably a decent XML Schema available, so why not take advantage of that and use of the many tools that are available to automatically generate the JSON for you?

In fact, there are quite a few different ways you can translate between XML and JSON, depending on what you're trying to achieve. Therefore, each tool can potentially generate quite different JSON for a given XML document. For a good overview of the different approaches and techniques involved, I recommend this survey of ways to map between XML and JSON. (That PDF is IBM's submission to the W3C Workshop on Data and Services Integration).

If you have a large amount of XML you want to convert into JSON, you may well need to implement your own tool to do the conversion. Not only does this let you control the choices made, it also can give you the opportunity to fix the niggling issues that inevitably arise in your XML as you extended your design in unexpected ways.

However, I recommend that you hand craft the design of your JSON representation, to make it as natural as possible.

A JSON Design Process

What I've found it a good way to design a JSON schema is to follow this simple process:

• Identify a list of candidate properties - perhaps by reviewing relevant XML schema for inspiration
• Think of one or two ways to represent each set of related properties in JSON - and research whether anyone else has designed something like it already
• Construct sample JSON documents for each of the alternatives
• Prototype some code to see how they work for your intended use
• Select the best alternative and add it to your schema
• Write down the examples and your rationale for picking that representation (otherwise you will forget)
• Repeat

After a while, you'll see some repeating patterns and you'll need to write fewer prototypes to try things out. But I still recommend writing down your rationale...

Trying out the JSON in code is particularly important if you haven't done a lot of JSON work before. It really gives you a feel for the best, most natural way to work with JSON and can help get you out of your XML Mindset (if that's where you're starting from).

JSON Design: A Series

Part two will discuss JSON tools and standards.

On the Difficulty of Defining Concepts

There are certain concepts whose meanings seem intuitively obvious - until you need to write them down.

I run into the need to craft a definition for a concept quite often in my work as Deputy Director of Schema Standards for the AP and in the process of defining standards for news at the IPTC.

Unidentified Structure by alanenlgish
http://www.flickr.com/photos/alanenglish/5728890490/

Define - Structure - Define
Routinely, we will decide that we need to represent something. So, we start by giving it a name and will generally sketch out a definition, and probably cite some examples. We will figure out the structure and properties of the concept, testing the structure against all the examples we can think of, to ensure we cover any edge cases. That process can be quite difficult in itself. However, once we think we are done, the original name and tentative definition are often called into question. That's because the process of thinking through the structure of a concept and how to represent it will often uncover variants that don't neatly fit within the original definition. Sometimes, this can lead to arguments over whether what we are modeling is a single core concept after all or is better described as a set of similar but distinct concepts.

Avoid by phunk
http://www.flickr.com/photos/phunk/4849730748/

Avoiding Tag Abuse

It is all too tempting to provide a definition that tautologically uses the name of the concept itself. Or - only slightly better - uses a synonym. Names and definitions are important when crafting a standard. They help others to convey information - avoiding the perils of tag abuse - and can highlight when a standard is genuinely missing an important concept.

Source #7 by nostri-imago
http://www.flickr.com/photos/nostri-imago/2872060566/

Lose the Source
Right now, the IPTC is going through that frustrating process once again over the concept of "Source". News standards such as hNews and rNews have the concept of Source, as did NewsML 1, the pre-cursor to the G2 Family of news standards (NewsML-G2, EventsML-G2 and SportsML-G2). And yet, we're debating whether and how to represent the Source concept within G2. The main hold up appears to be the lack of a clear definition: how to clearly define "Source" by saying what it is that all sources hold in common and what makes Source different than other concepts (such as Copyright Holder or Provider or Author)?

Language games by smackbox
http://www.flickr.com/photos/smackbox/70171055/

Language Games
This problem of defining a slippery concept is not new - or restricted simply to the practice of information modeling. Ludwig Wittgenstein dealt with this question in his Philosophical Investigations. In his aphorisms 65-69, Wittgenstein points out that the concept "game" is a useful one and people can intuitively claim they know what things are games. And yet games do not have "one thing in common" but "are related to one another in many different ways". I suggest that there are many useful concepts (like the concept of a "Source" in news) which cannot be reduced to a single essential set of characteristics but are instead a set of different things that resemble each other.

Ampersands by lwr
http://www.flickr.com/photos/lwr/3940977128/

&?
If all this talk of "concepts" is too abstract, you might prefer a more visual illustration. Consider any letter or alphabetical symbol. Such is the inventiveness of typographers - and the adaptability of human pattern recognition - that it is difficult to identify the core set of rules that identify a particular symbol. What makes an ampersand an ampersand?

"A rose by any other name would smell as sweet" by johnkay
http://www.flickr.com/photos/johnkay/3517370850/

What's in a Name?
Wittgenstein was not the first to point out dividing up the world into named and well-defined concepts is hard. The Sorites Paradox deals with the problem - and importance - of vagueness in defining concepts. Sorites derives from the Greek word soros (meaning ‘heap’):
Would you describe a single grain of wheat as a heap? No. Would you describe two grains of wheat as a heap? No. … You must admit the presence of a heap sooner or later, so where do you draw the line?

And, in the Tao Te Ching, Lao Tsu says

Once the whole is divided, the parts need names.
 There are already enough names.
 One must know when to stop.
 Knowing when to stop averts trouble.