RSS FeedBlogs
RSS FeedSubscribe to this blog
About Author
IDC analysts



The Conundrum of Big Data

Panacea or Problem?

Article comments

Big Data may be an overhyped buzz word today, with recent provocative headlines about Big Data from advancing human Genome research to find specific markers for cancer or genetic diseases, to Big Data analytics raising big privacy questions, as well as the problems of our digital footprint haunting us forever. However one recent headline gave me pause, as it asked the hypothetical question of whether Big Data could have prevented the recent Fort Hood shootings.

Many government entities are increasingly using Big Data analytics for decision making and solving some of their toughest problems. One such Big Data program was started with seed funding from DOD's Defense Advanced Research Project Agency (DARPA), a project that analyses communications from Veterans who have opted in.

The purpose of this program, the Durkheim Project, named after Emile Durkheim, a sociologist whose 1897 publication of "Suicide" provided an early text analysis for suicide risk, is to be able to predict the risk of harmful behaviour by Veterans. According to the Department of Veterans Affairs, an alarming rate of 22 Veterans take their lives every day. This project analyses data from Veteran's Social Media and mobile communications for the purpose of real-time monitoring of text content and behavioural patterns that are statistically correlated with tendencies of harmful behaviour, such as suicide.

In a recent article, Could Big Data Have Prevented the Fort Hood Shooting? - Defense One, by Patrick Tucker, he states "Had Army Spec. Ivan Lopez been enrolled in the Durkheim Program, which uses an algorithm that mines social media posts for indicators of suicidal behaviour, it might have picked up clues that a clinician could have missed in time for an intervention. "One of the founders of the Durkheim Project, Chris Poulin, is quoted in this article as saying, "Given the highly agitated state of the shooter, we may have been able to get him help before he acted, had he been in our system." Indeed, Poulin and his team of researchers developed an algorithm based on a repository of text, updated and analysed by artificial intelligence systems, that accurately predicts suicide 70 % of the time, an improvement over average clinical diagnosis, which is about 50% accurate.

While the Durkheim Project may make a major difference in clinical diagnosis, and thus may be seen as a panacea for predicting harmful behaviour among the population that has opted in, a few problems with this approach may exist. How can the results of this study be extended Veterans who show tendencies of harmful behaviour and are not enrolled? Some of the phrases cited that are most closely associated with suicide include "worthlessness" and" frightening." Is it as simple as providing health care providers the key words that are coupled with the behavioural patterns that statistically correlate with tendencies for harmful behaviour?

And while participation in this project is, and should be voluntary, the article points out that the technology to monitor social media exists. At what point do we weigh the need to do more to prevent shootings on bases, in schools, and places of worship against the right of privacy? Another issue raised in this article is the appropriate action and timing for intervention. Tucker asks, "At what point does an intervention opportunity turn into prosecution of thought crime?"

The Durkheim Project is in early phases of illustrating that Big Data can better predict harmful behaviour. Perhaps answers to these provocative questions will be provided as this project matures, as Big Data is more than access to data and technology. IDC describes Big Data as a mix of talent, technology (hardware, software, and services) and processes to economically extract value from very large amounts of multi-structured data to support decision making at all levels - tactical, operational, and strategic.

Organisations that reach the optimised stage of Big Data maturity have enterprise wide documented strategies and policies in place, continuous training to enhance all the necessary expertise, and are culturally ready to use Big Data for decision making, with processes and policies that have appropriate support, staffing, technology, and funding.

This project looks like it's off to a good start. It will be interesting to monitor progress, developing policies, and results as the Durkheim Project maturates.

Adelaide O'Brien, Research Director, Smart Government StrategiesIDC Government Insights


Send to a friend

Email this article to a friend or colleague:

PLEASE NOTE: Your name is used only to let the recipient know who sent the story, and in case of transmission error. Both your name and the recipient's name and address will not be used for any other purpose.

We use cookies to provide you with a better experience. If you continue to use this site, we'll assume you're happy with this. Alternatively, click here to find out how to manage these cookies

hide cookie message
* *