Friday, 14 September 2012

Releasing public data really works! (updated)

[03 Nov 13 update: "relooped the loop" by testing the corrected Ordnance Survey data]

UK Ordnance Survey released Open Data to the public two and a half years ago. English Parish boundaries have been more or less constant since the Domesday survey in 1087. That allowed me to post Cambridge University HC Darby and Yale University Julie Bowring socio-economic data, by simply adding attribute data to the Ordnance Survey shape files. That onerous, if one-time, task was entirely manual: when 1SpatialCloud (née Socium) launched Online Validation  it seemed only natural to try it out; they actually wrote some simple rules and we thus co-branded as quality assured by 1SpatialCloud Online Validation Service wherever I posted the data. Here are the resulting error shape files:

UPDATE: See here as a live map the top two features, which are the validation points:

In the upper right hand margin of this blog, you can find my private cloud location.

The resulting 25 spikes and kickbacks out of 1900 Parish polygons for East Anglia are, at 1.3%, a ratio I might have envied in my digital data capture days (1982 - 92 here). Whilst they didn't affect the integrity of my project - Parishes are used as anchors for attributes by which wealth is measured, rather than as areas or perimeters per se - it seemed only natural to inform the Ordnance Survey creators of said deviations.

While I as a user can find and point to the errors, only the authors can fix it - indeed the Ordnance Survey's copyright notice only allows to copy the data under fair dealing - but returning to the download page I found no feedback mechanism other than a Customer Service email... But wait! They promote their data not only online but also in open events publicised by @plangfordsmith such as  this one. I asked there how to feed back this information and a few emails later they check Socium's error polygons and opendata@ordnancesurvey.co.uk reply that these will be incorporated in the next update... rimshot

So that loops the loop:
  1. from @pterhaar unofficial announcement at Where2.0 early 2010 (thanks @stevenfeldman for this) to the actual release that April
  2. through augmenting a portion in East Anglia in my project, that lead to QCing late last year upon Socium's launch and uncovering errors
  3. to returning the error files this month to the Ordnance Survey, and finally the update of said data at the next data release cycle
In other words in a small way here, releasing public data really worked, and thanks everyone for their help - Drs Oosthuizen and Willmoth at Cambridge University, @giscloud and @mapsnrocks for helping with giscloud.com and arcgis.com post this data, data.gov.uk, bgs.gov.uk and sharegeo.ac.uk for posting it, @sociumcloud for online validation and @ordnancesurvey for reaching out.