Infographic: Big Data and Predictive Analytics

Infographic: Big Data and Predictive Analytics.

Infographic: Big Data and Predictive Analytics

I published a blog post titled Big Data and Rise of Predictive Analytics a couple of days back in which I highlighted that I am happy to say that Predictive Analytics (or Advanced Analytics as some would prefer to say) is going main stream in 2013, thanks to Big Data. It is not too difficult to understand why given the three Vs of Big Data, namely Volume, Variety and Velocity. Only way one can derive full benefits out of Big Data is by using predictive analytics and this is forcing large and medium companies to make necessary investments in building analytics infrastructure and reporting capabilities. And this is excellent news for those in Analytics profession or technology companies/service providers who help clients derive insights from mountains of (big) data.

Good to see that major enterprise software vendors have started focusing their attention on predictive analytics. Here’s a very good infographic on Predictive Analytics published by SAP Blog (infographic embedded below):

 

Big Data and Rise of Predictive Analytics

Back in mid 1990s, when I was a Ph.D. student,  one of the professors asked me what my career goal was and I said: “To help clients serve their customers better through use of Information Technology and Analytics“.

After completing my Ph.D. in November of 1998, when the IT revolution in Enterprise Solutions was about to take off and large and medium companies had started investing in ERP or CRM systems in a major way, I thought it was just a matter of time before Predictive Analytics goes main stream.

Back then, Siebel ’98 and Vantive were the ‘hottest‘ new tools in the market and Dot Com boom was on its way. I expected that in a year or two (or may be three), predictive analytics would become part and partial of all enterprise information systems and use of statistical techniques such as Regression Analysis, Factor Analysis and Multi-Dimensional Scaling would be common while analyzing and reporting information collected using ERP or CRM systems.

Looking back, I think I was over optimistic as this did not happen around 2001-2002 time frame as I expected. Most of the ERP and CRM applications had bare bones reporting functionality with just frequency (%) and advanced analytics was not leveraged.

If an application manager wanted anything more than frequency or % information, he/she had to invest in Business Intelligence (BI) or Data Warehouse (DW) solution. But again, BI or DW solutions analyze past information and are not “predictive” in nature the way high end statistical tools can be.

Yes, one could invest in SAS based analytics solution, but that was expensive, time consuming and out of reach of most companies – even Fortune 500 ones. As a result, use of Predictive Analytics was limited to a handful of use cases such as fraud detection in banks or customer churn management in telecom companies for example, where one could justify the investment in terms of time, effort and costs. But for a majority of ERP or CRM applications, data collected was never analyzed using Predictive Analytics and as a result investments in ERP or CRM systems did not deliver expected return on investment (ROI).

Things started to change around 2008-2009 with advent of social media tools. Again, I thought that it was just a matter of 12-18 months before Predictive Analytics goes main stream as large organizations will be required to use analytics tool to engage their customers on social media channels. In a blog post titled Social Media: The New Front End of CRM System published three years ago, I wrote that “the best any marketer can do is to Listen and Learn from what customers are saying and Engage them in meaningful conversations. In other words, treat Social Media channels as the front-end of CRM system, capture all relevant information from Social Media channels in the database and use Predictive Analytics and Knowledge Management tools to derive insights and help in decision making”. Again it turned out to be a case of over optimism as advanced analytics did not go mainstream in 2009-2010 as I expected.

Finally, yes, I say FINALLY after waiting for fifteen years, I am happy to say that Predictive Analytics (or Advanced Analytics as some would prefer to say) is going main stream in 2013, thanks to Big Data. It is not too difficult to understand why given the three Vs of Big Data, namely Volume, Variety and Velocity. Only way one can derive full benefits out of Big Data is by using predictive analytics and this is forcing large and medium companies to make necessary investments in building analytics infrastructure and reporting capabilities. And this is excellent news for those in Analytics profession or technology companies/service providers who help clients derive insights from mountains of (big) data.

What do you think? Do you agree that Predictive Analytics is going main stream in 2013 or is it a case of over optimism? Would love to hear your thoughts:

Organizational Challenge: Where to Start in Big Data?

Have ever wondered what is a good starting point for an organization as it embarks upon its Big Data voyage? If yes, here’s a good YouTube video on the subject by Stacy Leidwinger, Product Manager, IBM Big Data.

Hope you find this video not only interesting but also educative:

 About the author:

View Harish Kotadia, Ph.D.'s LinkedIn profileView Harish Kotadia, Ph.D.’s profile

5 Ways Big Data Are Fundamentally Changing Information Systems

A lot has been said and written lately about whether Big Data revolution is for real or it is one more hype that will die down soon as tech world moves on to the next fad.

In my opinion, Big Data  is a game changing revolution that will fundamentally change how information is collected, stored, managed and consumed thereby transforming the way we work, live and play.

Given below are five reasons why Big Data will change information systems and corporate IT:

1. Move away for traditional RDBMS:

Ever since electronic storage and processing of data began as a centralized corporate function (remember good old EDP or Electronic Data Processing Department!), Relational Database Management System or RDBMS in short is fundamental to most of the computerized corporate information systems. Even today, most of the information systems such as ERP or CRM are supported by RDBMS.

This is about to change in a big way, thanks to three Vs of Big Data namely, Data Volume, Data Variety and Data Velocity. Traditional data storage and retrieval methods, such as RDBMS are no longer going to work and would necessitate NoSQL (short for “Not only SQL”) database instead of RDBMS. Unlike SQL data or RDBMS which places data inside well defined structures or tables using meta data, NoSQL is designed to capture all data without categorizing and parsing upon entry into the system. This will fundamentally change the architecture of corporate information systems.

 2. Unstructured data handling capability:

Capability of handling both, structured and unstructured data is another important way information systems are going to change fundamentally thanks to Big Data. As noted above, Big Data has three defining attributes – Data Volume, Data Variety and Data Velocity and together they constitute a comprehensive definition of Big Data.

Data Variety implies that Big Data is not just about text or numbers (alphanumeric fields), but also unstructured data. Information systems in future will have to be designed with capability of handling both structured and unstructured data.

 3. Real Time Data Processing:

Given the Velocity or speed with which Big Data is being generated, information systems in future will require capability of processing massive volume of data in real time. Even “near real time”, a phrase often used with current generation of information systems, is not good enough.

A good example of real time data processing is the ability to process social media or sensor data as they are being generated and take necessary action immediately, such as responding to a tweet or Facebook posting. Batch processing, nightly or weekly updates and even near real time data processing are not good enough because of high Data Velocity as is the case with Big Data.

4. Predictive analytics and in memory analytics:

If data is being generated in a variety of formats (structured and unstructued), in high volume and at a high velocity, only way it can be used effectively for decision making is through the use of Predictive Analytics and in memory data analytics. Information systems in future will have to be designed keeping this aspect in mind.

 5. Most data are either user or machine  generated:

And lastly but not the least, most of Big Data are either generated by end users/customers (such as social media data) or by machines/sensors outside the confines or firewall of a company. This is unlike in the past, when most of the data were generated within the firewall of a corporation (such as transaction data, inventory data or factory production data) with very little coming from outside. This will fundamentally transform the architecture of information systems in future.

What do you think? Do you agree that Big Data will fundamentally change information systems and corporate IT? Please do share your thoughts:

 About the author:

View Harish Kotadia, Ph.D.'s LinkedIn profileView Harish Kotadia, Ph.D.’s profile

What Are Big Data, Hadoop and HDFS: 3 Must Watch YouTube Videos

I received great response to one of my previous post titled 5 Must Watch YouTube Videos on Big Data and many of you reached out to me requesting for more such blog posts.

Well, your request is granted. Here are three great videos on what is Big Data, Hadoop and HDFS. Hope you enjoy them and do let me know if it is useful.

Based on your comments and feedback, I will share more such great resources on Big Data. So either leave a comment in comments section of this post or better still, tweet your comment, mentioning my twitter ID @HKotadia and I will surely reply to you.

Enjoy and learn from these videos (double click on videos for full screen view).

1. Explaining Big Data by explainingcomputers:

2. Introducing Apache Hadoop: The Modern Data Operating System by Stanford University:

3. Hadoop Tutorial: Intro to HDFS by Marakana TechTV:

Big Data: Where the Opportunities Are!

I am often asked the question as to which industries offer the best potential for application of a Big Data solution and I have always said that Banking and Financial Services, Insurance, Telecom, Media/Information Services and Manufacturing offer the best promise to begin with. And I have based my judgement not only on the potential impact of a Big Data solution on the top and bottom-line numbers but also on “relative” ease with which Big Data can be used for reducing costs or improving productivity.

My thinking on this subject was validated recently when I came across following excellent visualization on opportunities for Big Data across several industries prepared by Brett Sheppard based on Gartner Intelligence. According to Brett, this visualization is based on “Gartner’s comparison of eleven industries across seven dimensions using data from Gartner forecasts, market statistics and input from dozens of vertical industry experts”.

(Source: See this)