Tragedy of the Commons: Open Government Data

This is a re-post from Apogee Spatial.

All opinions are completely my own and do not represent the position of any entity that I am or have been associated with.

Similar to several other areas of the Federal Government, citizens are mostly unaware of the significant investment, financial and otherwise, needed and performed for their benefit, and the benefit of the private sector. This is especially true for freely available and open data from the Federal government.

What exactly are we talking about?

The term “Tragedy of the Commons” dates back to the 1830s [1] and refers to the scenario where individuals act (rationally) in their own self-interest and contrary to the best interests of the whole community that they are a part of; by depleting some common or shared resource [2].

In the classical example, “commons” refers to shared and unregulated resources, such as land, air, oceans, etc.

The concept of the tragedy of the commons is often used in the context of sustainable development; underscoring solutions that need to be in place that marry economic growth and resource protection.

A lot has changed since 1833 and a lot has remained the same.

What does this have to do with Open Government Data?

Data is the currency of our time. It is the new “commons” — collected, managed and curated by the government when it was too expensive for the private sector to even contemplate doing so.

The investment, in gathering and disseminating data to everyone for free, led to a lot of public benefit — new businesses, new insight on life, product feedback, and new reference points for better decision-making.

Demographic analysis would not be possible without data from the US Census Bureau. The global weather industry would not exist without data from the National Oceanic and Atmospheric Administration. Economists and business owners would have no idea about the health of the economy without the data from the Bureau of Economic Analysis.

Of the $9.8 billion dollars requested by the Department of Commerce in its 2016 budget [3], approximately 1 to 10% of it will be spent on the infrastructure, mechanisms and technologies that create this “data commons”.

Contrast this to the 2013 valuation of the economic value of (only) the weather enterprise, which was $31 billion [4], and you get a glimpse into the contribution that the Commerce Department exerts on our economic lives.

The full impact of the Department providing this economic and environmental “data commons” is most likely in the hundreds of billions of dollars to the trillions of dollars in income and wealth creation.

Building this “data commons” for trade, investment, economic growth, and economic opportunity (through the Department’s twelve bureaus) is not an easy task.

However, it is one that the government does as a part of its mission of freely providing open data to all its citizens.

Unfortunately, as with most things in life, there are nuances and complications.

The Price of Data Production

The satellites, radars, surveys and other mechanisms used to produce the data from the Commerce Department’s bureaus:

1. Were once expensive, and
2. Are paid for by requests to the United States Congress.

The initial expense was rationalized as the cost of sparking innovation. This is no different from investing in the mission to the moon, which was deemed as a necessary step in demonstrating the technical prowess of American science.

The Congressional budget process is an active negotiation that normally leads to the disbursement of lower expenditures than initially requested (and than necessary to complete the mission).

This leaves a situation where investment costs are high and the allocated budgets are often insufficient to cover the collection and disseminating of the “data commons”.

The Lack of Re-Investment

Over time, advances in technology have made it possible to:

1. Leverage the open data produced by the Department’s bureaus in realistic timeframes, and
2. Lower the data production costs.

A significant number of multi-billion dollar industries currently owe their existence to these technological developments, which allow the utilization of the Department’s data infrastructure investment.

Unfortunately, the current set of advances create new demands and require a paradigm shift in terms of native collection and dissemination.

If companies contained internal divisions that handled the ETL (extract-transform-load) process, the company executives would re-invest in upgrading the backbone of their business to meet these market demands and stay competitive.

Unfortunately, the same is not true when that backbone is supplied by a government entity. The prevailing perception is that the backbone has already been paid for through taxes.

This perception of “it being paid for” persists despite the knowledge that budget requests are often made in the context of maintaining existing systems and that these requests are usually significantly reduced by the time the negotiation concludes and monies allocated.

These facts should be an indication that these successful companies should re-invest in the infrastructure that they rely on to provide them (and the rest of the country) with reliable, free, high-quality data.

Unfortunately, this does not appear to the case.

Favoritism and Competition

There are two golden rules that are often repeated in polite circles:

1. The government should not play favorites, and
2. The government should not compete with existing businesses.

For a myriad of good reasons, there is a web of regulation that encapsulate the notion that federal entities are a resource for everyone and not a select few of well-connected entities.

Unfortunately, the history of the creation of industries that leverage federally produced data is littered with examples of creative alliances that lock vendors in, create effective monopolies and stymie long-term innovation. These partnerships need to be untangled and re-purposed to serve the mission of fully serving the American people.

My favorite phrase to hear from a budding or seasoned business person is “the government should not be competing with existing companies”.

Two things always pop to mind.

The first is incentives. Businesses have no incentive to serve the under-represented; have no motivation to provide free and open foundational components upon which entrepreneurs can innovate; have no push to altruistically develop market offerings that will not yield a profit (but have a market demand). However, the government does. What happens when the government develops data products that provide equal and undifferentiated access to all and in the process (inadvertently) competes with an existing business; as the offering increases in popularity and scope?

The second is value & viability. There are a few companies that capitalize on a market gap created from a deficiency in the government’s data production and dissemination process; whether it is in the packaging of the data products or in the deciphering of strange data formats. Are we saying that the federal government is stuck doing things the way that they have always done? What happens when they receive funds to correct past under-fundings and start “doing things in better ways”? Should the current “capitalizing” companies be allowed to force the government to stay in a stagnant state in order to protect their business interests?

In a capitalist environment, when a company is in a field, sector or niche that advances, that business is expected to innovate and find new revenue streams or face irrelevance. It makes sense that in an environment where a firm’s value is in correcting an artificial gap, that when that gap is fixed they should innovate and support the greater good of the larger community.

On the other side of the coin, the cost of instrumentation and data collection has dramatically decreased and businesses see value in competing with the federal government [5]. These businesses hope to sell their data to the government; increasing the cost of data production and reducing the probability that the general public will have access to free and open data. The possible downstream effects range from mildly annoying to severely catastrophic.

It is instructive that businesses can see the validity and reasonableness in them competing with the government, but not see the validity or reasonableness in the government competing with them. Either there is validity in everyone competing or there is no validity in businesses and the government competing. Both cannot be simultaneously true in the same world.

Bringing it all together

Right now, there are entities that are currently acting in their own self-interest and contrary to the needs of the broader community by trying to reduce, and not re-invest in, the shared resource of free and open data.

We have to start addressing this tragedy before it is too late.

References

1. Lloyd, William Forster (1833). Two lectures on the checks to population. England: Oxford University.
2. Hardin, Garrett (1968). “The Tragedy of the Commons”. Science 162 (3859): 1243–1248. doi:1126/science.162.3859.1243. PMID 5699198.
3. DOC OPA (2015). “Fact Sheet: FY 2016 US Department of Commerce Budget”: https://www.commerce.gov/news/fact-sheets/2015/02/fact-sheet-fy-2016-us-department-commerce-budget
4. Lazo, Jeffrey (2013). “The Economic Value of the Weather Enterprise”: http://www.ametsoc.org/boardpges/cwce/docs/profiles/LazoJeffreyK/2013-04-AWF.pdf
5. Leone, Dan (2015). “NOAA Cites Data-sharing Obligations in Draft Commercial Weather Policy” http://spacenews.com/noaa-releases-draft-commercial-space-policy/#sthash.uCA4GlIO.dpuf