The role of data scientists will be transformed as machine learning techniques become more widely used by businesses, according to Stephen Brobst, CTO of analytics firm Teradata.
While many of the principles behind 'AI' approaches are not new, interest within an enterprise setting has exploded in recent years. And as usage becomes more widespread and sophisticated, the role of data scientists will begin to evolve too, according to Brobst.
He explains that data scientists have typically spent much of their time 'wrangling' data to feed into predictive models. In future, more of this work will be automated and data scientists will instead be more focused on selecting which machine learning or deep learning tools to utilise for specific tasks.
"Instead of the data scientist spending most of their time working with the data itself, they are going to spend most of their time working with the algorithms - so you have to be much more sophisticated in algorithm selection and topology selection in a neural network and so on," he says.
"You still have to understand how the nature of the data influences your algorithm selection, but you are going to be spending less time preparing data, because data scientists today spend more than 60 percent of their time preparing data, beating data over the head, torturing it, shoving it into a hole, and that is not going to be required anymore."
He says that this is already becoming the case with advanced algorithms used for machine learning purposes currently, but as deep learning - a branch of machine learning - become more widely used, data scientists' priorities will shift further. See also: What is deep learning?
"In deep learning there will be fewer requirements for domain knowledge and more requirements for algorithm selection based on the type of data that you have and so on," he says, "so this shift of skill set will be very, very interesting."
Data science skills gap
For many enterprises, however, the challenge is just finding the right data science expertise in the first place. Accessing skills remains a major challenge, despite the efforts of tech companies to provide training courses and resources.
"There is a big [skills] gap right now, because the number of data scientists who are qualified versus the demand for them is way out of whack - and this is particularly true in the UK," Brobst explains. "Partially that is because the UK is relatively aggressive in understanding that data science is important, so [UK organisations] are kind of on the leading part of that."
See also: How to get a job as a data scientist
It is not just a problem faced by the UK. "We are not producing enough [data scientists] worldwide," he says. "There is a worldwide hunt for a good data scientist going on."
One solution is improve training and education. Brobst explains: "You can't just make it a zero-sum game and just take [data science experts] from wherever else, you also have to build the expertise and invest in the education system," he says.
"In the US there is a tonne of money being invested in STEM - science, technology, engineering and mathematics - for kids at a young age, especially females, to get them to be excited about science and math and so on. As a society we do a particularly bad job with females, because females at a young age are often better than males at mathematics for example. But then we - through societal pressures and other bad influences - convince them that math is not a female thing to be doing; that is for boys. And this is, of course, complete horseshit.
"We need to change the way that society thinks about investing in skill sets, not only in females, but even where kids say 'oh maths, that's for the geeks, and that's no good'. There has been a backlash at least in Silicon Valley, now the geeks are cool, but we need to be investing in the education system to build up the science technology and engineering and mathematics."
Creativity is key
Brobst says that the focus on teaching STEM subjects has missed out on a crucial aspect of jobs such as data science roles, namely the ability to think creatively.
"I like STEAM better than STEM – science, technology, engineering, art and mathematics," he says.
"You have got to teach the creativity, because if it is a purely mechanical thing then you are not going to get really interesting breakthroughs. You will get continuous improvement kind of stuff, but you need that artistic, risk-taking, out of the box thinking as well as the math and science. People think that math and science is all about mechanical thinking and it's not.
"The best mathematicians and scientists were out of the box thinkers. Take Alan Turing - clearly an out of the box thinker, but he was doing math stuff. But if he had only been doing the mechanical part of math he wouldn't have the impact that he has on our lives today.
"You can teach people to think creatively. We beat it out of them in many cases, especially in engineering and mathematics - if it is not a completely rational, step by step process, then you are not following the rules. So math and science is not just about the rules - you get to create new rules if you are truly innovating."
Find your next job with computerworld UK jobs