Road Less Traveled Road Sign

December 27, 2016

It’s been almost a year since my last blog post, so I thought it might be time to provide an update for anyone interested in what’s been happening with the Structured Stories project.

During the first half of 2016 I focused on research work, much of it associated with my fellowship at the Reynolds Journalism Institute at the University of Missouri. Early in the year I used the WordSmith tool from Automated Insights Inc. to demonstrate use of the Structured Story database to drive natural language generation – so-called ‘automated journalism’. Later in the spring I worked with a University of Missouri team to conduct a broad survey of user comprehension of structured stories, using a greatly simplified user interface and a small set of stories tied event-by-event to text articles. Publication of academic work has continued past the end of the fellowship, including delivery of a joint paper at AEJMC in Minneapolis in August, and delivery of an overview paper at the Computing News Storylines workshop in Austin in November, titled “Computable News Ecosystems: Roles for Humans and Machines”. Other papers about aspects of the project, by myself and others, are in progress and by the time the dust settles there should be another 3 or 4 published papers about the project, touching on reporting efficiency, story consumption and automated journalism. More on that to come.

But despite this academic success, and despite the substantial interest in Structured Stories from the structured journalism community, it has become clear to me that the journalism world is not well-suited to long-term research and development (R&D) projects. Journalism has no tradition of reimagining its fundamental components and assumptions, no institutions charged with exploring technology-heavy alternatives and, critically, no appetite for investment in work that might pay off years in the future. Fortunately, however, there is another domain that shares most of the characteristics and challenges of journalism but which does have a long tradition of supporting long-term R&D – the intelligence domain.

The intelligence community’s essential task is to gather information from a wide variety of sources and to refine and contextualize that information into products that help decision-makers understand a complex world. Despite recent controversies about access to communications metadata, the overwhelming majority of information processed by intelligence agencies these days is actually open source intelligence – ‘OSINT’ in the acronym-heavy government vernacular. The intelligence community employs tens of thousands of analysts to find and organize all this information, and relies largely on text documents to synthesise and communicate its work. Its workflows and processes parallel journalism in many ways, and the too-much-text problem is becoming as critical in intelligence circles as it is in news. Unlike the journalism ecosystem, however, the intelligence world has the ability and willingness to fund long-term R&D of technology-based solutions.

Beginning in mid-2016 I have therefore been adapting Structured Stories for use in intelligence applications, funded by a small research grant and aided by a research company with experience in government R&D funding. Although it is still early in this transition, results have been promising and the prospects for further funding for 2017 and beyond appear encouraging. Much of the functionality addressed by this new development work is directly applicable to journalism, and it is my intention to incorporate it into a journalism-facing product sometime in the future. I don’t expect to publish anything about the intelligence applications of Structured Stories until at least late 2017, but the ‘Computable News Ecosystems’ paper mentioned earlier describes the general system pretty well.

Developing an alternative to text articles as units of news is an audacious goal, and will likely take years, and a small but necessary detour along the way may be the quickest way to reach it. I am still deeply committed to offering a data-centric alternative to our rapidly collapsing text-centric news ecosystem. I intend to keep the existing Structured Stories site alive, and hopefully maintained, and I intend to remain active in structured journalism forums and conversations. I invite anyone interested in the project, or interested in helping out, to get in touch at any time.