Tag Archives: analytics

Predictive Analytics: A Force Multiplier for Big Data

Force Multiplier, a noun, means something that increases effect of the force. In military usage, force multiplication refers to “an attribute or a combination of attributes which make a given force more effective than that same force would be without it” (for more, see this).

Big Data, which is characterized by three Vs, namely Volume, Variety and Velocity can be a major force in running of any large or medium sized businessas as it adds tremendous value by improving quality of decision making. Thanks to Big Data revolution, it is possible to process large volumes of structured and unstructured data in real time and derive insights from large data sets. This by itself is a huge improvement over pre-Big Data era.

What’s even better is that predictive analytics makes it possible not only to analyze the past, but predict the future too with high degree of confidence level. For example, social media data can enrich risk modeling and help auto insurance companies prepare much better risk profile of an individual. Car sensor data can help auto insurance companies better assess risk posed by a driver’s habits (like speeding, fast acceleration or braking) and come up with auto insurance policy tailored to that specific individual with individual level premium (not at a zip code or a city level).

Another good example is assessment of customer life time value (CLV). Using big data, companies can come up with much better assessment of customer life time value. What makes it even better is that predictive modeling can be used on social media or sensor data in arriving at a much better estimation of CLV so that companies can better target customers with high CLV. This has been very effectively used in Travel and Hospitality industry.

Point I want to highlight here is that Big data is a revolution in itself as it enables organizations identify, store, process and analyze data sets from outside the organizations in a way which was not possible thus far. Add predictive analytics to this mix and it pushes Big Data capability to a whole new level – a true force multiplier. Don’t you agree?

Question is how many large and medium sized companies are in a position to take advantage of not only Big Data revolution, but also effectively leverage Predictive Analytics for driving better insights and decision making. Not too many in my opinion. What do you think? Please do share your opinion:


Big Data and Rise of Predictive Analytics

Back in mid 1990s, when I was a Ph.D. student,  one of the professors asked me what my career goal was and I said: “To help clients serve their customers better through use of Information Technology and Analytics“.

After completing my Ph.D. in November of 1998, when the IT revolution in Enterprise Solutions was about to take off and large and medium companies had started investing in ERP or CRM systems in a major way, I thought it was just a matter of time before Predictive Analytics goes main stream.

Back then, Siebel ’98 and Vantive were the ‘hottest‘ new tools in the market and Dot Com boom was on its way. I expected that in a year or two (or may be three), predictive analytics would become part and partial of all enterprise information systems and use of statistical techniques such as Regression Analysis, Factor Analysis and Multi-Dimensional Scaling would be common while analyzing and reporting information collected using ERP or CRM systems.

Looking back, I think I was over optimistic as this did not happen around 2001-2002 time frame as I expected. Most of the ERP and CRM applications had bare bones reporting functionality with just frequency (%) and advanced analytics was not leveraged.

If an application manager wanted anything more than frequency or % information, he/she had to invest in Business Intelligence (BI) or Data Warehouse (DW) solution. But again, BI or DW solutions analyze past information and are not “predictive” in nature the way high end statistical tools can be.

Yes, one could invest in SAS based analytics solution, but that was expensive, time consuming and out of reach of most companies – even Fortune 500 ones. As a result, use of Predictive Analytics was limited to a handful of use cases such as fraud detection in banks or customer churn management in telecom companies for example, where one could justify the investment in terms of time, effort and costs. But for a majority of ERP or CRM applications, data collected was never analyzed using Predictive Analytics and as a result investments in ERP or CRM systems did not deliver expected return on investment (ROI).

Things started to change around 2008-2009 with advent of social media tools. Again, I thought that it was just a matter of 12-18 months before Predictive Analytics goes main stream as large organizations will be required to use analytics tool to engage their customers on social media channels. In a blog post titled Social Media: The New Front End of CRM System published three years ago, I wrote that “the best any marketer can do is to Listen and Learn from what customers are saying and Engage them in meaningful conversations. In other words, treat Social Media channels as the front-end of CRM system, capture all relevant information from Social Media channels in the database and use Predictive Analytics and Knowledge Management tools to derive insights and help in decision making”. Again it turned out to be a case of over optimism as advanced analytics did not go mainstream in 2009-2010 as I expected.

Finally, yes, I say FINALLY after waiting for fifteen years, I am happy to say that Predictive Analytics (or Advanced Analytics as some would prefer to say) is going main stream in 2013, thanks to Big Data. It is not too difficult to understand why given the three Vs of Big Data, namely Volume, Variety and Velocity. Only way one can derive full benefits out of Big Data is by using predictive analytics and this is forcing large and medium companies to make necessary investments in building analytics infrastructure and reporting capabilities. And this is excellent news for those in Analytics profession or technology companies/service providers who help clients derive insights from mountains of (big) data.

What do you think? Do you agree that Predictive Analytics is going main stream in 2013 or is it a case of over optimism? Would love to hear your thoughts:

5 Ways Big Data Are Fundamentally Changing Information Systems

A lot has been said and written lately about whether Big Data revolution is for real or it is one more hype that will die down soon as tech world moves on to the next fad.

In my opinion, Big Data  is a game changing revolution that will fundamentally change how information is collected, stored, managed and consumed thereby transforming the way we work, live and play.

Given below are five reasons why Big Data will change information systems and corporate IT:

1. Move away for traditional RDBMS:

Ever since electronic storage and processing of data began as a centralized corporate function (remember good old EDP or Electronic Data Processing Department!), Relational Database Management System or RDBMS in short is fundamental to most of the computerized corporate information systems. Even today, most of the information systems such as ERP or CRM are supported by RDBMS.

This is about to change in a big way, thanks to three Vs of Big Data namely, Data Volume, Data Variety and Data Velocity. Traditional data storage and retrieval methods, such as RDBMS are no longer going to work and would necessitate NoSQL (short for “Not only SQL”) database instead of RDBMS. Unlike SQL data or RDBMS which places data inside well defined structures or tables using meta data, NoSQL is designed to capture all data without categorizing and parsing upon entry into the system. This will fundamentally change the architecture of corporate information systems.

 2. Unstructured data handling capability:

Capability of handling both, structured and unstructured data is another important way information systems are going to change fundamentally thanks to Big Data. As noted above, Big Data has three defining attributes – Data Volume, Data Variety and Data Velocity and together they constitute a comprehensive definition of Big Data.

Data Variety implies that Big Data is not just about text or numbers (alphanumeric fields), but also unstructured data. Information systems in future will have to be designed with capability of handling both structured and unstructured data.

 3. Real Time Data Processing:

Given the Velocity or speed with which Big Data is being generated, information systems in future will require capability of processing massive volume of data in real time. Even “near real time”, a phrase often used with current generation of information systems, is not good enough.

A good example of real time data processing is the ability to process social media or sensor data as they are being generated and take necessary action immediately, such as responding to a tweet or Facebook posting. Batch processing, nightly or weekly updates and even near real time data processing are not good enough because of high Data Velocity as is the case with Big Data.

4. Predictive analytics and in memory analytics:

If data is being generated in a variety of formats (structured and unstructued), in high volume and at a high velocity, only way it can be used effectively for decision making is through the use of Predictive Analytics and in memory data analytics. Information systems in future will have to be designed keeping this aspect in mind.

 5. Most data are either user or machine  generated:

And lastly but not the least, most of Big Data are either generated by end users/customers (such as social media data) or by machines/sensors outside the confines or firewall of a company. This is unlike in the past, when most of the data were generated within the firewall of a corporation (such as transaction data, inventory data or factory production data) with very little coming from outside. This will fundamentally transform the architecture of information systems in future.

What do you think? Do you agree that Big Data will fundamentally change information systems and corporate IT? Please do share your thoughts:

 About the author:

View Harish Kotadia, Ph.D.'s LinkedIn profileView Harish Kotadia, Ph.D.’s profile

Big Data Ecosystem

Here’s a very good chart of big data eco-system prepared by Matt Turck and Shivon Zilis (see this link for more). Like the way they have classified major big data players across Infrastructure, Analytics, Applications and Data Sources.

My take on this ecosystem is that we are going to see a lot of action on big data analytics front as it is the analytics layer where major value addition happens in Big Data stack – others, namely Infrastructure, Application and Data Sources are geared for collection and storage of information that is used by Analytics layer for deriving insights for decision making.

Data Rich, but Information Poor: Problem or Opportunity?

It is not uncommon to hear the phrase “Data Rich, but Information Poor” in the context of enterprise information systems. This is especially true when it comes to Big Data, given the three Vs of Big Data, namely Volume, Variety and Velocity (read this for more). What seems like a big problem posed by data deluge is in fact a great opportunity if only companies can unlock the potential of Big Data using predictive analytics.

Since Big Data is characterized by not only Volume, but also Velocity and Variety, it is very important that Big Data is used for analysis in real-time to predict the future and take corrective action based on that analysis. The real value is in using insights derived from predictive analytics and taking corrective action before it is too late, rather than just reporting historical information.

Techniques like Multiple-regression analysis coupled with Factor analysis, Cluster analysis and Causal Path analysis can be used very effectively with Big data. For example, rather than analyzing “historic” attrition rates, predictive analytics will make it possible for companies to identify critical incidents leading to customer attrition so that steps can be taken to retain the customer before he/she defects.

So whether Big Data deluge is a problem or an opportunity depends on what you make of it. It can be a great opportunity if advanced statistical tools are used for analyzing big data and derived insights made available to decision makers and front line employees in real time, if not it can be a great problem.

Dear CEOs, CMOs and CIOs, what’s your choice? And what are you waiting for?


Why Big Data Mining / Analytics is the New Gold Rush

Just mention the words “Big Data” to any technology entrepreneur or investor and observe how his/her face lights up with excitement. Given the perceived opportunity in Big Data, tech entrepreneurs and investors want to capitalize on it by starting /investing in a Big Data Management, Mining and Analytics business. Is this perceived opportunity in Big Data for real or is it a bubble that will burst soon?

I think the perceived opportunity in Big Data is real and is here to stay as Big Data Mining/Analytics will fundamentally change the way business is done not only online but also offline. Here’s why: Big Data is a key enabler for Social Business and without Big Data Mining/Analytics, a large or medium sized company can neither make sense of all the user generated content online nor can collaborate with  customers, suppliers and partners effectively on Social Media channels. And both of these activities, namely gaining insights from user generated content online and collaborating with customers and partners on Social Media channels are critical for success in the age of Social Media.

Can you imagine any large or medium sized business without e-Business in the internet age? Similarly, it is impossible to run a large or medium sized business in the age of Social Media without leveraging user generated content or without collaborating with customers and partners. And for this to happen, Big Data Management, Mining and Analytics are critical/key enablers.

Being a key enabler and catalyst in Social Age, Big Data Management, Mining and Analytics companies are going to command a premium valuation and hence Big Data Mining and Analytics has triggered a new ‘Gold Rush‘, not to mine gold this time but something even more valuable – knowledge and insights.

What do you think? Do you agree that Big Data has triggered a new Gold Rush?


Big Data Analytics a Key Enabler for Social CRM – Airlines Case Study

Big Data Analytics is a hot topic of discussion these days. But many feel that it is more of a “hype” and less of substance. In my opinion, Big Data Analytics is the real deal and if used correctly, can deliver great business results at a fraction of cost compared to other alternatives.

Here’s a simple yet very effective example of using analytics for understanding consumer attitudes towards airlines in real time. In this study, Jeffrey Breen has used the R programming language to analyze consumer sentiments about major U.S. airlines expressed on Twitter. For more on the methodology and results, see the following embedded Slideshare presentation (if you cannot see the embedded file, click here to view it on Slideshare website).

As you can see on slide no. 27, the twitter sentiment scores obtained  for many of the airlines are “comparable” to results of The American Customer Satisfaction Index (ACSI). What is important to note here is that by analyzing few tweets using freely available R programming language (which is part of the GNU project), it is possible to achieve results similar to that of an elaborate and expensive market research study such as ACSI at a fraction of a cost, that too in real time. Isn’t that a game changer?

Now imagine using Big Data Analytics to analyze Social Media data – not only text data but also pictures, video, GPS , Social Graph/Social Media linkages and using that information for engaging customers on Social Media channels. That in my opinion is the ‘holy grail’ (if I may use that term) of Social CRM.

In one of my earlier post, I have defined Social CRM as the business strategy of engaging customers through Social Media with goal of building trust and brand loyalty. Big data analytics is a key enabler for engaging customers on social media channels for building trust and loyalty (Social CRM). What do you think?