What the consumerization of big data means for data scientists

By Cameron Graham, TechnologyAdvice

No doubt you’ve heard the term “big data” or “data analytics” more than a few times in the past year. Driven in part by a cloud-computing price war between Amazon, Google, and Microsoft, the barrier to big data entry is lower than ever before. Some legacy companies such as IBM are selling off huge (and profitable) chunks of their business in order to focus on cloud computing and software as a service (SaaS). This massive analytics movement has greatly increased demand for so-called “data scientists,” “data engineers,” or “data strategists.” The Harvard Business Review went so far as to call data science the “sexiest job of the 21st century.

However, as the barrier to entry drops price wise, and smaller companies are able to begin utilizing cloud computing services, the technical barrier to entry will surely drop as well. By technical barrier, I mean the skills required to find patterns in, analyze, and draw conclusions from company data sources. There will certainly always be a market among enterprise companies for dedicated data scientists. Companies such as eBay, for instance, can produce over 10 petabytes of data a year. Small to medium businesses however, don’t produce nearly as much data – and, if given the proper tools, may not need nearly as much expertise in order to analyze it.

When previously complicated technology becomes easier for the general public to use, it’s sometimes referred to as consumerization. It’s a trend that’s played out before, even quite recently. Website creation is a good example. Not long ago, most people didn’t possess the  tools or ability to make their own websites. The field was cornered by specialists who knew HTML, CSS, and Java. Today, anyone that can set up a WordPress account can create a fairly attractive site (not to mention even more powerful tools such as Wix). The technical barrier to entry for website creation is almost zero.

You can see the beginnings of the same phenomenon starting to take place with data analysis software. A recent O’Reilly report highlighted six products that give users in-depth analysis tools, yet require very little to no knowledge of programming. Microsoft recently released Power BI, an Excel add-on that layers advanced analytics tools into the familiar spreadsheet interface. It’s likely that other big data applications will follow suit, especially if these initial forays prove popular with small to medium businesses. As such tools become mainstream, companies will gain the ability to easily conduct data analysis in-house, without the need for a dedicated data scientist.

This doesn’t mean that job demand for data scientists is going to vanish, or even decline. Enterprise companies will certainly need more robust solutions than an automated platform can provide. There’s also a current talent-shortage that has yet to be filled. According to a 2013 SAP survey of US and UK companies, 75 per cent said their staff didn’t have the required skills to take advantage of their data output. For current data scientists and aspiring students, this is a welcome finding. It’s worth keeping in mind though that as the industry grows, technical barriers to entry will become progressively lower, and automated platforms will fill at least part of this demand.


TechnologyAdvice helps companies find business technology. We’ve researched and reviewed hundreds of products, from business intelligence platforms, to electronic health records and project management software, in order to find the very best solutions. Users can read product reviews, find the latest industry news, or sort products based on their individual needs. We’ve advised hundreds of businesses, including multiple Fortune 500 companies.

Would you recommend this article?


Thanks for taking the time to let us know what you think of this article!
We'd love to hear your opinion about this or any other story you read in our publication.

Jim Love, Chief Content Officer, IT World Canada

Featured Download

Latest Blogs

ITB in your inbox

Our experienced team of journalists and bloggers bring you engaging in-depth interviews, videos and content targeted to IT professionals and line-of-business executives.