Opening up public sector data - why and how to bring about change

Public sector activity has always generated data. Every population census produces a mass of valuable information about who is living in a country and their daily lives and activities. And, of course, the information generated and collected by public bodies is much broader than this. Patient health records, mapping information, weather data, tax records, crime statistics, exam results – all of this information can be a hugely valuable resource in the digital age.

Historically public sector data has often been tightly controlled and only made available in response to a specific request. But recent initiatives to make public sector data more readily available are bringing about major changes. These are broadly driven by two goals; to enable innovation in the knowledge economy and to support better delivery of public services. 

Clearly some information will always remain closed. Information relating to national security and defence will often be too sensitive for disclosure. But in many cases there is no good reason to keep information out of the public domain, and plenty of reasons to make it available.

An international effort

Opening up public sector data is an international trend. Focus from the United Nations has so far been on open data principles as a vehicle for enabling inclusive and sustainable development. Within the G20, efforts have targeted anti-corruption, with a Compendium of good practices on the publication and reuse of open data for anti-corrunption adopted in April 2017. 

The G8’s 2013 Open Data Charter adopted a much more general approach, setting out five principles as a foundation for access to, and the release and re-use of, public sector data.

Principle 1: Open Data by Default
Principle 2: Quality and Quantity
Principle 3: Usable by All
Principle 4: Releasing Data for Improved Governance
Principle 5: Releasing Data for Innovation

Beyond these well-established international structures, the 2015 Open Data Charter is an independent project built around six principles: 

1. Open By Default - Instead of individuals or businesses asking for the specific information they want, open by default introduces a presumption of publication for all. Governments should instead justify keeping data closed, for example for security or data protection reasons.
2. Timely and Comprehensive - Data should be published quickly and in a comprehensive way, and where possible, in its original, unmodified form.
3. Accessible and Usable - Making data available in a machine readable way, with attention paid to file formats. Data should be made available free of charge, under an open licence, for example, those developed by Creative Commons.
4. Comparable and Interoperable - Quality datasets with commonly-agreed data standard should be offered.
5. For Improved Governance & Citizen Engagement - Transparency can help to improve public services and make it easier to hold governments to account.
6. For Inclusive Development and Innovation - Data can be a tool to promote innovation and development.

The Charter has been adopted by 54 national and local governments, among them the UK, France, Canada and Australia.

In the US, Project Open Data cites the twin objectives of strengthening democracy and promoting government efficiency, and creating economic opportunity and improving lives. 

The US portal, data.gov, currently boasts over 300,000 datasets across a wide range of topics. The 2013 Federal Open Data Policy explains the objectives.

“Information is a valuable national resource and a strategic asset to the Federal Government, its partners, and the public. In order to ensure that the Federal Government is taking full advantage of its information resources, executive departments and agencies (hereafter referred to as “agencies”) must manage information as an asset throughout its life cycle to promote openness and interoperability, and properly safeguard systems and information. Managing government information as an asset will increase operational efficiencies, reduce costs, improve services, support mission needs, safeguard personal information, and increase public access to valuable government information.”

The resources available through this platform increasingly includes APIs.

UK Government embracing the change

The UK is embracing this global trend. The 2012 White Paper Open Data White Paper - Unleashing the Potential set a clear objective – “We are unflinching in our belief that data that can be published should be published.” UK Government’s public sector data sharing programme, Data.gov.uk, is working to make available a vast range of data held by Government bodies and other public sector organisations (local government, the police, the NHS, charities etc). Data is normally shared on the basis of the Open Government Licence. 

March 2018 saw the launch of the new “Find open data” service enabling data users to search for relevant datasets and see the terms on which they are licensed. This facility enables users to locate data that may be held within other public sector bodies. Ofcom, for example, makes available sets of information underpinning its Connected Nations report. CSV files recording mobile network coverage and data usage information aggregated at local authority level are offered on a royalty-free basis.

Another exciting example is the unlocking of Government mapping and location data. Key parts of the OS MasterMap were released in June 2018. Businesses will be able to make use of this vast data resource without charge in order to create innovative digital services. It is intended to be a step on the journey towards even more open geospatial data infrastructure, positioning the UK as a world leader in location data.

European initiatives

At European level, the Public Sector Information re-use Directive 2003/98/EC (or PSI Directive) was introduced in 2003, requiring EU member states to meet minimum standards in making available material held by public sector organisations. It addressed areas like non-discrimination, charging, exclusive arrangements, transparency, licensing and practical tools to facilitate the discovery and re-use of public sector information.

The PSI Directive has seen several amendments, most recently in 2013 with changes introduced to require re-usability of data by default, and machine-readable formats, to introduce marginal cost charging and to bring museums, libraries and archives within its scope. An updated text is available here. 

The legislation is now earmarked for further development. A public consultation and review has identified areas ripe for updating, now under consideration for legislative change. 

The revision project proposes some ambitious changes. It aims to introduce:

• Real-time access to dynamic data through use of Application Programming Interfaces (APIs)
• Open access to research data from publicly funded research
• A prohibition of new forms of exclusive arrangements between public data holders and the private sector, which might limit the number of data users

The proposal would also extend the scope of the PSI Directive to documents held by public sector bodies operating in the water, energy, transport and postal services.

What about personal data?

Obviously, all public sector data sharing must comply with data protection law. Revealing personal data about identifiable individuals could seriously undermine public confidence and trust. 

The Open Data Charter mentioned above acknowledges this imperative. 

“To make this work, citizens must also feel confident that open data will not compromise their right to privacy.”

UK guidance on re-use of public information issued by the National Archives explains that the public sector body making a dataset available must ensure that it complies with data protection law before making data available for re-use.

Of course, European data protection law has recently seen a major overhaul with the introduction of the General Data Protection Regulation, or GDPR, and national legislation like the UK Data Protection Act. This shifted the balance in favour of individual privacy. While the proposed revisions to the European PSI Directive are careful to retain respect for privacy rights, the European privacy watchdog sees room for improvement. It recommends more explicit cross-referencing to data protection law, and a formal link with national privacy regulators when reviewing decisions to grant access to data. The opinion calls for data protection impact assessments where particularly sensitive data are under consideration, and suggests that anonymisation costs might be passed on to re-users.

An open data agenda

Review and amendment of the PSI Directive is likely to take some time – it is at an early stage in its legislative journey. It may well not be finalised before the time the UK leaves the EU, whether under a no-deal scenario in March 2019, or even after a transition period at the end of 2020. That means it is unclear whether the changes will automatically find their way into UK law. What seems certain, however, is that the UK Government alongside many others around the world will push the open data agenda forwards and increasingly encourage re-use of their datasets by innovative businesses.