Knight News Challenges

The Knight Foundation runs an annual "news challenge":

The Knight News Challenge accelerates media innovation by funding breakthrough ideas in news and information. Winners receive a share of $5 million in funding and support from Knight’s network of influential peers and advisors to help advance their ideas.
http://newschallenge.tumblr.com/

Numbers by Koen Vereeken
http://www.flickr.com/photos/koenvereeken/2088902012

I was recently browsing through the many entries to the current stage of the challenge: "DATA"

We’re looking for new ways of collecting, understanding, visualizing and helping the public use the large amounts of information generated each day.
http://newschallenge.tumblr.com/post/24130238607/knight-news-challenge-data-is-now-open

I encourage you to look through them, too. To whet your appetite, here are a handful that caught my eye:

Satire
This one might be good for a laugh?

Combine satire, open data, and software development to produce an informative, comedy hackathon centered around the 2012 election.

http://newschallenge.tumblr.com/post/25579071683/satire-hackathon-pairing-comedians-and-developers-to

rNews
I've got an interest in rNews. It is nice to see it spread in the wild a little bit.

Enable publishers to automatically augment content with meta data in the rNews format, derived from natural language processing.

http://newschallenge.tumblr.com/post/25579264393/metafy-us-automatically-tagging-content-in-the-rnews

HTTP for Data / Python

And I've got a soft spot for Python, still one of my favourite programming languages of all time.
Improve Python, Numpy, Scipy for data science & big data. Lay groundwork for an open data web (“HTTP for Data”).

http://newschallenge.tumblr.com/post/25579049658/next-generation-numpy-scipy-and-python-for-data

Wikidata
The fledgling Wikidata initiative is very interesting and has generated a lot of excitement in the semantic web world. Perhaps a little ambitious - a repository for all identifiers for everything - but the ambition is laudable.
Turn Wikidata into a common, free identifier repository for tagging, exploring, linking, and visualizing resources, news stories, and data everywhere.
http://newschallenge.tumblr.com/post/25575917516/wikidata-as-a-central-free-repository-of-identifiers

News Event Service

And my name is on this one...
Create a news event registry and a curated event taxonomy based on both a daily crawl of global news and contributions from news providers.

http://newschallenge.tumblr.com/post/25576383721/the-news-event-service

SemTechBiz 2012

SemTechBiz 2012

It has been four years since I last attended SemTech. I notice that it now has a longer name: SemTechBiz. I will be interested to see what else has changed. I'll find out next week (the conference starts on June 3rd).

Audience by thinkmedialabs
http://www.flickr.com/photos/thinkmedialabs/6177365564/

Speaking

I will be speaking at the conference, along with Andreas Gebhard and Evan Sandhaus. We'll be giving an update on the state of rNews, the IPTC standard for embedding news metadata in web pages. Catch our panel session The State of rNews - One Year after its Release on Tuesday, June 5th, at 11.30-12.15 in room Imperial A. If you'd like to learn a bit more about rNews in the meantime, check out http://dev.iptc.org/rNews
And Amy Sweigert, my boss, will also be presenting. She'll be unveiling the details of the AP's new metadata services - a platform for taxonomy and classification in the cloud. Catch her talk on AP Metadata Services on Wednesday, June 6, 2012 12:00 PM - 12:30 PM in room Yosemite A. And, even if you can't attend, you can look here to find out more about the AP's Metadata Services offerings.

learn by heycoach
http://www.flickr.com/photos/heycoach/1197947341/

Learning

As well as presenting, I hope to learn a little something at the conference. It looks like there's a lot of really interesting sessions, covering 17 different tracks, so I should have plenty of opportunity.

Hello my name is by mlebemle
http://www.flickr.com/photos/mlebemle/3511330328/

Say Hi

So, if you're also going to be at SemTechBiz 2012, please say "Hi". And don't forget the hashtag: #semtechbiz.

Liking Links

I like the link.

Links by Clips
by RambergMediaImages

This may seem an uncontroversial position to take. But I'm specifically talking about the HTML link element. It is a nice model for expressing relationships between the current document and other resources. It has been imported into various XML standards (such as ATOM and NewsML-G2). But it isn't the only way to express links. Even in HTML, it is much more common to use

 <a href="http://example.com/">

than

<link href="http://example.com" rel="related">.

Chain Links
by MyTudut

The Linky Landscape
In fact, there are a range of types of links. If you think about the img element within HTML, it includes a reference to a remote object to include in a web page (via the img/@src attribue). Even though the img element uses an HTTP reference, it isn't very linky - it is more about composing a document from multiple resources of different types.

Anchor
 by Leo Reynolds

Anchors Aweigh!
The HTML a (anchor) element is a big step up in linkyness. It is generally used to specify an outbound navigational link from the current HTML page to some other resource (such as another HTML page, but it could also be a video clip, an image or anything else that can be addressed via a URL). But the HTML a element lets you specify how the remote resource relates to the current page: the a/@rel attribute lets you specify the relationship from the current document to the destination resource. Similarly, the a/@rev attribute lets you express the reverse relationship - how the remote resource relates to this document. (Remember, not all relationships are symmetrical). So, the anchor tag has some nice (though, in practice, rarely used) attributes for specifying in a machine-readable, controlled way the relationships between web resources. One limitation of the HTML a element is that it is only for use for "inline" links - such as marking up a bit of text within a paragraph.

link
by Profound Whatever

Link
The link element has all the same attributes as the anchor element but it is designed to be purely semantic - it simply defines a relationship between the current document and the resource referred to in the link/@href attribute. The link element cannot contain content and so it is purely about referencing another addressable resource. As with the HTML a element, the relationship between the document containing the link and the resource being referenced can be specified using the @rel and @rev attributes.

Links
by rubygold

Web Link Types
In an effort to provide some shared semantics about types of links, IANA put together a small, but growing, list of link types - values that you can use in your @rel or @rev attributes. Whenever you are adding a link to your document, check out the existing list of link relations, to see if you can provide some hints as to the relationship between your document and the linked resource:
http://www.iana.org/assignments/link-relations/link-relations.xml

Chain Links
by Eric M Martin

Spreading the Link
In fact, the HTML link element has proven so useful, that it has been adopted into other markup standards, notably IETF's ATOM and IPTC's NewsML-G2. The IETF issued RFC 5988 specifically about linking, not simply in the context of HTML documents.

So, if you need to establish relationships between resources, then consider using the link element, with @rel or @rev values. And if you happen to be designing a markup language, my advice is to make sure you include the full link element. I predict you'll find it useful over and over again.

One Good Thing About Not Being Popular

I suppose one good thing about my blog not being popular: I don't have to deal with comment spam!

selenium + python = web automation ftw!

This past snowy weekend, I decided I wanted to try out automating one of the routine tasks I have to perform each week via a web browser. After a little googling, I decided that Selenium was my best option. And, with a little bit of work, I was able to use it to create a script to automate this chore - even though the web application I'm automating turned out to be quite unwebfriendly.

Before I forget, I thought I should quickly jot down what I learnt in the process, with the hope that this will be useful to others (or, perhaps, my future self).

Selenium Roses by bcostin
http://www.flickr.com/photos/bcostin/155246852/

Selenium
The Selenium website succinctly describes the software: "Selenium automates browsers". Primarily known as a web site testing tool, the seleniumhq site points out that "[b]oring web-based administration tasks can (and should!) also be automated as well".

It turns out that Selenium is, in fact, a suite of tools. There's Selenium IDE, a Firefox plugin that is ideal for quickly prototyping an automation script. And there are various server side components with the core being the WebDriver API. (The WebDriver is also known as "Selenium 2", as it is a merger of the original Selenium with a project started by a tester at Google. See the Selenium project history for more). The WebDriver lets you control various browsers - Internet Explorer, Chrome, Opera, HtmlUnit. It even has ways to simulate iPhone and Android devices. But the browser best-supported by Selenium is Firefox. I had to download a fresh copy, since I've almost entirely switched to Chrome these days. But it was worth it.

Firefox magnet (wallpaper) by flod
http://www.flickr.com/photos/flod/2568092124/

How I Did It
I wound up basing my script very much on the example in the Getting Started section. Along the way, I developed a few patterns that I'd like to pass along, in the hopes that they will help.

As I mentioned earlier, the web application I was trying to automate turned out to be quite unwebfriendly. What I mean is that it used all kinds of complicated tricks to ensure that, for example, the URL doesn't change as you navigate to different pages within the application. Similarly, rather than using anchor tags (<a href>) for links, it used Javascript "onclick" scripts to dynamically construct them. The good news is that Python-connected-to-Selenium-powering-Firefox is perfect to automate this kind of application, as it fully supports Javascript, frames, pop up windows and the like.

Waiting by amchu
http://www.flickr.com/photos/amchu/5261511319/

Waiting
One thing I quickly figured out was that my script needed to wait for the complicated, multi-part pages to finish loading before going onto the next step. The documentation explains how to use either explicit or implicit waiting in Selenium. At first, I followed the lead of the example and looked for a particular web page title. Or I thought I could detect changes using the URL. Except that my web application doesn't change title or URL as the pages change!

However, I hit upon a solution: in each step, my script searches for a particular element by name or id. So, I realized that waiting for that element to be appear was the most effective strategy. This lead me to code like this:

try:
    # we have to wait for the page to load
    WebDriverWait(driver, 10).until(lambda driver : driver.find_element_by_name("login"))
    
except:
    driver.quit()

(You might also notice that I altered my code to use "except:" rather than "finally:", since I don't want my script to always exit, whether or not the page load completes in time).

Use the Source
I also figured out a variation on the try/wait pattern to help me figure out the next step. Often, I wrote a WebDriverWait function that would time out, without succeeding. Eventually, I realized that simply quitting the driver wasn't the most useful way to debug things. Instead, I would use the page_source property to help me see what was really loaded into the browser at that point:

try:
    WebDriverWait(driver, 10).until(lambda driver : driver.find_element_by_name("mainfs"))
except:
    print driver.page_source

IDE Clues
I also found that it was useful to try to figure out how to navigate the web application using the Selenium-IDE plugin. I would record my actions in the IDE, then review the script it generated, to get clues as to how to write my WebDriver script.

I say "clues", because the IDE generates Selenium 1.0 commands, not the 2.0 syntax I needed to use. Generally, it gave me enough information that, with some searching of the API documentation, I could figure out the right equivalents. However, in certain cases, there are capabilities in Selenium 1.0 that have no direct equivalent in the 2.0.

Grandpa by conowithonen
http://www.flickr.com/photos/cmogle/2907198746/

A Problem of Focus
One example: in 1.0, there's a way to do relative navigation between frames. It turned out I needed this, since at one point in the web application I was trying to automate, it leaves the focus inside a subframe. Experimenting with the IDE, I saw that it wriggled out of this problem by executing

selectFrame(relative=up)

The problem is that there is no equivalent relative move between frames in Selenium 2. Eventually, I figured out that I needed to switch the focus to the entire window (by making use of the "current_window_handle" property), which then let me select the particular subframe I was looking for

driver.switch_to_window(driver.current_window_handle)
driver.switch_to_frame("body")

Don't Give Up!
Once I figured out these patterns of working by experimenting with the IDE, and printing the source whenever I got stuck, I found that automating the web application with Selenium was fairly straightforward. It would be nice if the documentation was a bit fuller. And it would be wonderful if the IDE generated Selenium 2 commands. But I think that my small investment in figuring it all out was worthwhile. So, now I'm looking for more things to try with Selenium. For example, could I use it to try out RESTful testing of APIs or Linked Data? Are there other web-based chores that I could (even partially) automate?

Why I Love Twitter

Recently, someone who doesn't use Twitter asked me to explain why I love that service which describes itself as a "real time information network".

learning by SReinwald
http://www.flickr.com/photos/d1sasta/2321821367/

As with most of these kinds of things, it is a little hard for me to explain why I love Twitter so, particularly to someone who has never used it. But maybe I will learn something from trying?

Tweet of Dweams by born1945
http://www.flickr.com/photos/12567713@N00/2369056718/

Perhaps Twitter is like interesting people from around the world texting and sometimes I text back? But it is more than that. I get most of my news through Twitter and it is like instant messaging with various friends and colleagues and heroes and comedians and ... I used to describe Twitter as being like just the status postings from Facebook. But Facebook has changed so much since then that I'm not sure that description still applies. And the network of people you interact with on Facebook and Twitter are quite different (certainly, the way I use those services).

But, really, Twitter isn't like texting or RSS feeds or instant messaging or Facebooking. Or, actually, it is kind of like all of those things and more and less, all at once. 

REST APIs and Errors

What's the "right way" to indicate errors in a RESTful API?

Rest Area? by joeshlabotnik
http://www.flickr.com/photos/joeshlabotnik/419914250/

HTTP Response Codes
As with many aspects of RESTful APIs, it turns out that this has already been worked out and standardized as a fundamental part of the web architecture.

If you're building a RESTful API, you should use the HTTP response code to indicate whether or not a call to the webservice succeeded (http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html). If everything is OK, then return “200 OK”. If something goes wrong, return an appropriate response code in the 3xx, 4xx, or 5xx range. Codes in the 3xx range indicate different types of redirect, e.g. 307 Temporary Redirect. Codes in the 4xx range are conditions that a client can (generally) address, e.g. 400 Bad Request. Codes in the 5xx range indicate types of server error, e.g. 500 Internal Server Error.

The advantages of using this standard mechanism and these standard error codes include:

1. Good documentation of how they work, what they mean, what to do in each case (e.g. http://en.wikipedia.org/wiki/List_of_HTTP_status_codes)

2. There are quite a few existing implementations of this scheme on both the client side (e.g. web browsers) and on the server side (e.g. web servers and web frameworks)

Error Documents

If a REST API returns a response code in one of the error ranges (3xx, 4xx, 5xx), it should be a signal to the client that any response metadata and payload should not be interpreted as a response to the request, but as an explanation of the error. 

I haven't been able to identify any standard for the details of error responses, probably because the details of errors for different APIs can vary so widely. However, I like Amazon’s S3 REST webservice error document schema:

http://docs.amazonwebservices.com/AmazonS3/2006-03-01/dev/index.html?UsingRESTError.html

An example is

<?xml version="1.0" encoding="UTF-8"?>
<Error>
  <Code>NoSuchKey</Code>
  <Message>The resource you requested does not exist</Message>
  <Resource>/mybucket/myfoto.jpg</Resource>
  <RequestId>4442587FB7D0A2F9</RequestId>
</Error>

The schema is that each <Error> document contains a <Code> (a unique id for the error) and a <Message> (an English-language description of the error). Optionally, <Error> documents can contain additional XML structures that are specific to each kind of error.