Why our governments need a big dose of big data

I am a cynic! (Actually, all analysts are cynics!) However, I am not a conspiracist. That said, in the last few weeks, while the whole world experienced violent oscillations in the equity markets, what I've come to realise is that our governments...


I am a cynic! (Actually, all analysts are cynics!) However, I am not a conspiracist. That said, in the last few weeks, while the whole world experienced violent oscillations in the equity markets, what I've come to realise is that our governments need a big dose of Big Data.

Why? What I observed is that, despite the fact that our politicians are supposed to represent their constituents, constituents generally end up voting for the politician's beliefs (or their party's beliefs).

Politicians then hypothesise on these beliefs in the form of "policy" and then let others, such as the Congressional Budget Office or other think tanks prove or disprove their positions. Many will argue that in many cases, independent of geography, some of these beliefs are based on either popularity or the satisfaction of major donors.

However, the question that should be posed though, is why work on hypothesis when there is a perfectly viable, scientific, predictable approach to all this? We do live in the Information Age, right? Why not leverage technologies, Big Data technologies, to create a more predictable, repeatable and scalable understanding of how policy can affect a nation.

Now, civil libertarians are probably jumping out of their skins right now, protesting at the potential violation on privacy and that "Big Brother is watching!" But it must be noted that we, as a society, consistently and contractually give up our civil liberties and privacy on a daily basis.

When we sign up for a mobile phone we deliberately and contractually give up details about whom we call, from where we call, how and for what we search on the internet. A subscription to an ISP or a land line telephone is similarly deliberate and contractual. If you have a credit card, we contractually give up data on what and where we acquire what we acquire. If you have a bank account, the government already tracks what we deposit or withdraw, and we, as a society contractually give up this information when we sign up for our bank account.

In testing these ideas with my other half, she noted that these contracts are made on a proprietary level with the service providers directly, and not with our governments. (That's what I get for being with a Liberal!) She is right, of course. But, we willingly share our social security (or some other national identifiers) with most (if not all) of these providers. While I am not suggesting that service providers give up the privacy that each of us should enjoy, there are many ways in which we can anonomise this data.

So, what we have concluded is:

  • We have an incredible amount of data about individuals, that we have given up contractually;
  • We can anonymise this data; and
  • We have the technology to bring all this data together

The question that is left, is why governments do not take the data that is available to them, and leverage Big Data technologies to predict how its citizens will react to changes in economic conditions. Big Data can also look at the impact of other economic activity such as (technical) default by governments, or a change in the credit rating of its bonds, etc.

Given the availability and accessibility of technologies, our leaders should take advantage of these technologies as it negotiates and debate fiscal and monetary policies. Could the economic crises of Europe and America have been avoided? Perhaps. What we can tell is that, there is a better possibility of predicting the impact of governmental policies if we leveraged Big Data, than if we didn't.

An esteem colleague at IDC once suggested that the global financial crisis (GFC) in 2009 was caused by creative bankers using Big Data-like technologies to invent new financial instruments that were essentially derivatives on derivatives. He also suggested that the brevity of the GFC is a result of the same Big Data-like technologies to negate the said actions. Personally, I don't fully subscribe to this theory; but it certainly has merit.

So, let's bring it back to the beginning. Why are our governments not taking advantage of the technologies that are available to us to assist society? That is a good question, and one that I cannot answer. (The cynic in me suggests that our politicians have enough trouble turning on their laptops and PDAs, let alone of leveraging technologies to their advantage). When it's all said and done, we need to pressure our leaders to get out of the dark (and non-informational) ages and take advantage of what (Big Data) technologies offer. As an IT community, would you vote for this?

Posted by Benjamin S. Woo