Open data in the UK: a job half done
Since 2010 the government has officially promoted the publication of open data: on schools' performance, local crime statistics and civil servants' salaries, to name but a few.
The stated aim was to improve transparency, but also to boost economic growth thanks to new startups expected to be created on the back of the newly-released data.
However some, including Shadbolt, worry it has taken its eye off the ball recently.
"It's the routine availability of data that provides a resource to make stuff happen. And we're quite a long way away from that," he told ComputerworldUK in an interview.
Politicians "need to be constantly reminded that this is still an issue. It's not job done. They often have quite a limited attention span, so it's a long haul really", he adds.
In June 2009 Shadbolt and World Wide Web inventor Sir Tim Berners-Lee were appointed as advisors to the UK government.
They led the development of data.gov.uk, a single website for UK public data launched in 2010. The push for public bodies to publish their data continued under the coalition government, with support from Cabinet Office minister Francis Maude.
In 2012 Shadbolt and Berners-Lee set up the Open Data Institute (ODI), which helps to incubate startups based on open data.
The institute has trained 1,500 people in open data, has 300 members, over 25 startups who have gone or are going through its incubator scheme and has influenced policy, for example nudging the government to appoint a chief data officer earlier this year, according to Shadbolt.
"The best example I always give: TfL's transport data in London means you have the best transport apps in the UK. You don't have to pay for them, they're developed by a community, using better data than we've ever had. It's a great example of where you can take a sector and everybody's got more value from it," he says.
Most importantly perhaps, open data has gone from a 'wacky' fringe topic to a mainstream concept within government, and one few would argue against, at least publicly.
But many in the open data and government community say these achievements, although significant, should be viewed as just the first step.
"Open data like all ideas has a currencywe have to prove out the initial euphoria: here is where it makes a difference, here is why it's important. Even though the government is relatively pleased with its position and where it's got to in the UK, there is so much more to do," Shadbolt says.
"I would say there is more work to be done", Ellen Broad, policy lead for the ODI, agrees.
So what data is available now
There are 20,000 datasets on data.gov.uk, Cabinet Office minister Matthew Hancock MP said in a recent speech - an impressive sounding figure.
All Whitehall departments now regularly publish data online in some form or another.
However the information can be "incomplete and difficult to analyse", limited in accuracy as some datasets are not released or are heavily redacted; and constrained as transactions under £25,000 are rarely published, a 2014 Institute for Government report found.
When and how the data gets published - crucial for startups relying on dependable data- can also be sporadic. Just 19 percent of the data published by departments on data.gov.uk is classed as 'up to date'.
Vast swathes of data are unavailable as a result of complex, locked-down outsourcing contracts with private providers. Two-thirds of outsourced deals in Whitehall do not allow the government access to information on profit margins, according to the National Audit Office.
Open data principles are not "embedded in procurement rules", Dr Rufus Pollock, cofounder of nonprofit organisation Open Knowledge, told MPs on the Public Administration Committee last year.
Local authorities are supposed to publish all spending over £500 online each month.
"All of them are now publishing the data in some form", Dr Ben Worthy, politics lecturer at Birkbeck College, says.
However "not all do this regularly", according to procurement data startup Spend Network, which relies on government procurement data to operate.
Ironically spending data for the Cabinet Office, which leads on transparency and open data policies across government, is about a year out of date.
Officials have refused to release the data in response to a Freedom of Information request.
The Information Commissioner's Office has said it could serve an enforcement notice against the department but has previously often proved unwilling to take a tough stance, preferring instead to 'work with' wayward authorities.
Culture barrier
Their case perhaps demonstrates the crux of the issue: how do you force unwilling organisations to publish their data
Spend Network often has to submit Freedom of Information requests to get data out of foot-dragging councils and other bodies, something they say "takes a lot of time".
One issue is the lack of legislation to compel data release.
"Government has in general declined to legislate for open data, i.e. create new information rights," open data campaigner Owen Boswarva says.
However, for Shadbolt, the issue is not technical or legal, but cultural.
"Typically there will be a set of people who haven't got their heads round it, the issues around licensing and so on. We have 350-odd local authorities in this country, and they all have a different view on what they should and shouldn't be releasing," he says.
National Data Infrastructure
One way to push authorities into cleaning up their act would be to force them to rely upon their open data, according to Shadbolt.
"The best way to get government to really ensure that quality data is there to be had is to also insist it uses it itself.
"So when organisations become dependent on its own data for its own processes, it won't take long for them to sort it out. I think there's a lot of opportunity there with the whole 'eat your own dog food' idea," he says.
Another idea that has gained traction in open data circles is for the government to establish a 'National Information Infrastructure', which the ODI's technical director Jeni Tennison likens to a national road network or power grid, but for open data.
The infrastructure would comprise datasets like the Companies House register, all house sales, schools data, procedures carried out in hospitals, to name just a few.
The government kicked off plans to list all the datasets needed for the infrastructure in October 2013.
However it's gone quiet since the election. The official 'National Information Infrastructure' webpage categorises the policy as having been "published under the 2010 to 2015 Conservative and Liberal Democrat coalition government".
That doesn't mean open data campaigners have forgotten about it though: Shadbolt insists it is a vital next step.
"The need for an open data infrastructure is essential. If we don't have the equivalent for data that we have with our roads, power grid, you won't build a 21st century digital state that is any more than little islands of privilege and monopolies," he says.