Open source tool Spark real-world big data applications

We often hear about the promise of big data for organizations to improve everything from inventory fulfillment to marketing strategies, but rarely do we get the nitty-gritty details of how it might be done and with what tools. That’s why this article from Ben Lorica, the chief data scientist at O’Reilly Media Inc. is so refreshing, it details the growth of a promising new open source big data engine and how some companies are using it to their advantage.

Apache Spark is an open source cluster computing system that purports to scour through big data at 100 times the speed of alternative engines such as Hadoop MapReduce. It delivers such speed by allowing for in-memory computing rather than a disk-based system and also aims to make developers’ jobs easier by allowing them to code in Scala, Java, and Python. The project is built upon the Berkeley Data Analytics Stack. The first ever Spark Summit is taking place in San Franciso on Dec. 2 and will feature real-world examples of Spark in action from the likes of Yahoo,, and  Cloudera.

Perhaps most importantly for those in line-of-business roles is the trend of developers finding a way to represent the terabytes of data involved in an analysis with visual representation. That means the complex analysis could soon be more accessible to business users through the tools they’re already accustomed to working with for these purposes. Take the example of ClearStory, a startup that built a platform to allow users to bring together several data sources and produce a visual graphic of the analysis.

While Spark is still a young platform that still hasn’t hit version 1.0, it counts 67 contributors in its community and is being deployed at some companies where real-time data analysis is crucial. With a summit coming up that will feature case studies from major Web brands like Yahoo, you can expect this is one platform you’ll be hearing a lot about in 2014.

Brian Jackson
Brian Jackson
Editorial director of IT World Canada. Covering technology as it applies to business users. Multiple COPA award winner and now judge. Paddles a canoe as much as possible.

Would you recommend this article?


Thanks for taking the time to let us know what you think of this article!
We'd love to hear your opinion about this or any other story you read in our publication.

Jim Love, Chief Content Officer, IT World Canada

Featured Download

ITB in your inbox

Our experienced team of journalists and bloggers bring you engaging in-depth interviews, videos and content targeted to IT professionals and line-of-business executives.

More Best of The Web