On working with Open Data

revolucao

From ’13 to ’16 I’ve worked on Portuguese Open Data initiatives, namely acting a Project Lead for the Public Open Data Portal, and representing the country on a series of international bodies and projects. This text is about what open data is, my experiences in the field and some personal considerations on the subject. All opinions are my own.

Around mid-2013, after roughly seven years working in Marketing & Communications, I was offered the opportunity to work on Open Government projects and initiatives, for the Portuguese Government. In practice that meant taking over the product management of the Portuguese Public Open Data portal ā€“ dados.gov (launched two years earlier) and working on a broader strategy for championing transparency and accountability of the Public Administration.

Open Government was a burgeoning field of Public Administration management. It is related to perennial democratic concepts such as Freedom of Information, accountability and transparency of the state, but for the last decade or so it also became closely tied with digital tools and IT policies.

obamaopengov
The Obama Administration embraced Open Government as one of their major priorities back in 2009.

Open Data

Open Data is one of the most visible fields of Open Government initiatives. At its inception, Open Data is any data that you are free to use without any sort of constraints. Itā€™s data you can use to build apps and make money out of them (if you want). To work on academic projects. Or just lovingly gaze at, in the privacy of your home. There are no strings attached to use this data. You donā€™t need to pay, you donā€™t need to register, you just download or access it. In the Public Administration context, we usually talk about Public Open Data (I will be using both terms interchangeably).

Public Open Data is data that is produced or results of government work. Think about the data that is collected in the census. Or the data regarding the physical locations and addresses of government buildings and infra-structure. Public transportation timetables. Official building energy consumption figures. Information about business registrations. And so on. Think of all the data that hundreds of public institutions have to collect, work with and publish. Thatā€™s a lot of data. All of this data is part of what, in Europe, we call Public Sector Information (PSI). PSI includes all of this data, but also historical documents, photographs, public film, and stuff like that. And do you know who owns this data? Hint: not the government. Government is the custodian of this data, but the owners are all of us ā€“ the taxpayers and citizens.

The reason why Public Open Data is the sexy fledgling of Public Sector Information is that Open Data, at its best format, is meant to be machine-readable and therefore easily streamlined to build those before mentioned apps, websites or products.

There is an interesting (or sterile) debate about what really constitutes machine-readableness. In its most common definition, machine readable information is data that can be extracted and processed by a computer program (usually formats like XML or JSON). Some people argue that a scanned pdf is not really a machine-readable format, while others point that pdfs are opened by computers, which are machines (thusā€¦). I donā€™t really feel qualified to participate in that debate, but you know whatā€™s even more terrible? When data is not available at all, or just buried inside some institutional website or closed room.

Of course we are transparent, just take whatever you need!
“Of course we are transparent, just take whatever you need!”

AndĀ speaking of burying data, this is where Open Data Portals come in. Public Open data portals are where people, usually developers (i.e. geeks, coders, or anyone who knows their way around a ā€˜web serviceā€™), can find open data.

Portals are important because they organize, catalog and aggregate information. Thatā€™s what we have been working on at dados.gov for the past years. Dados.gov is the central public open data portal in Portugal.

Itā€™s challenging work, especially when we are doing it with minimal resources in an emerging (and fast-changing) field. The portal was launched in 2011 and took off with around 100 datasets. Five years later we are approaching the 800 dataset mark from over 15 different data suppliers across multiple ministries and organizations.

Who benefits from Open Data?

There are usually two ways of looking at the value of Open Data. One is the obvious value of transparency and accountability. The other is the economic potential of opening up data free of reuse restrictions (i.e. for commercial purposes). Of course they are not mutually exclusive, but at one point the latter became one the main flags of the movement. Itā€™s a great carrot for gathering political support. The problem? It takes time to happen. So around 2013, 2014, cynics started questioning the value of open data. Where were all the trillions promised by the studies on open data economic value? The string of exciting and rich startups built on open data?

Me in Berlin (and a slide that I thought was original but then found someone had already used it)
Me presenting at a SHARE-PSI workshop in Berlin.

tā€™s good to have cynics. I am known to be a particular practitioner of self-questioning and skepticism. But I donā€™t really think that questioning the potential economic value of open data is questioning the value of open data. Besides having an intrinsic value of transparency and accountability, Open Data can also be a driver of innovation and efficiency for the Public Administration itself. This is something that we have found to be very relevant in the Portuguese context, and a strategy that we actively pursued for the past years.

While responsible for the general promotion and the development of Open Data in Portugal as part of its responsibilities in the fields of public administration ICT policies and electronic government, AMA has been promoting the open data portal as a public shared service for all of the Portuguese Public Administration. Building on the importance of sharing information and promoting its reuse, public organizations are encouraged to use dados.gov as a means of making data available on a robust platform that turns tables into web services, can be accessed via an API and is covered by a Creative Commons BY license.

Countries have been following different strategies in accordance to their political and organizational context. In some nations politicians raised open data as a political flag and encouraged (or ordered) ministries and departments into opening data. Some of these countries also moved quickly into putting legislation into place that enforces open data policies or reuse of information standards.  Federal or regional countries often move at different speeds, but are also encouraged by an internal competition spirit. From 2011 to 2016, in Portugal we have been following mostly a bottom-up approach, working open data at the Public Administration level.

Engaging different people

While frustrating at times, this has been an amazing experience, forcing us to engage the public administration at different levels and promoting something that (we believe) is paramount to the greater good and public service. The work has been a mixture of project management, product development, content management, technological evangelism and, at times, cold call selling. There is a portal and a myriad of webservices to maintain and keep working, and a development pipeline to look for in the next few years. There are workshops, conferences and one to one meetings where we try to bring everyone on board. And there are a lot of blank stares, ā€œIā€™ll call you laterā€ and ā€œwhatā€™s in this for meā€ reactions that we get from data suppliers.

Working with little resources has made us adapt and try to make the most with less. We used what we called ā€œGovernment as a Developerā€ strategy, where we identified high profile web / app development projects within the government and then nudged open data as a building block of such projects. We did that with the Citizen Map app and the Local Transparency Portal, opening up most of the data that powers those applications. This means that anyone can access and use that same data to build similar (or better) stuff.

Engaging stakeholders is another classic tenet of open data initiatives. In Portugal we have been focusing mainly on the Public Administration and Civil Society sectors. Besides all the work weā€™ve been doing at government level, we also have made some occasional efforts in reaching out to hacktivist organizations, data journalists, transparency think tanks, and the like. The academic and business sectors also figure on our priority list, but they require a more complex approach and some investment ā€“ so weā€™ve been focusing on the low-hanging fruit.In the past two years we made significant efforts to reach out and listen to the community. We met with most of the people mentioned above and, most importantly, sat them together and discussed the future vision for open data in Portugal. We followed design thinking methodologies and brainstormed the crap out of the past, present and future of opening up data in Portugal. This allowed us to further access the needs from our partners in the Public Administration, which included the need for training and technical support in developing web services, and also the general feeling from civil society that data portals need political and legal enforcement.

At one point you have to wonder if Design Thinking was just a brilliant IDEO strategy to help 3M sell Post-Its
At one point you have to wonder if Design Thinking was just a brilliant IDEO strategy to help 3M sell more Post-Its.

What happens next

I believe that we are at the nexus of a new kicking point for a nation wide Open Data strategy. The PSI Directive will soon be transposed to Portuguese law, the startup scene in Portugal is entering a new maturity stage, and the Public Administration is now showing signs of interest in embracing these principles (both centrally and locally). Moreover, there seems to be an understanding that you can’t Open Data without addressing other fundamental issues regarding ICT management (i.e. interoperability) in government systems.

I’ve been purposely avoiding the “cultural challenge” of open data, since it could warrant another 2,000 words. Yes, it’s one of the major obstacles, specially with an ageing and insulated Public Administration, but one that can be tackled with policy and management. Public servants aren’t the issue here, but the way work is organized. And that can be changed.

While still at an early stage, I think Open Data represents the first step in advancing towards an openness culture in the Public Administration. There are many benefits to going open, such as transparency, economy, but also efficiency. Hopefully we will soon start thinking about Government as a Platform, about opening up APIs for public services, sharing the source code for major government platforms. Maybe government will also start employing some of the brilliant minds that graduate from Portuguese university’s excelente computer engineering programs. It may take a while, but I’m betting that we will all be here to see it, and contribute to it.