Dr. Vasant Dhar, Professor and Head of the Information Systems group at NYU Stern School of Business, joined us for an IDFC-U roundtable discussion on "The Commercialisation of Big Data" in an on-the-record discussion on 19th January 2015.
The discussion was based on Dhar’s paper “Data Science and Prediction” that talks about what big data essentially implies and how data can be used to generate useful insights. The Big Data market is forecasted to double from $28.5 million in 2014 to $50.1 million in 2017. Dhar’s primary focus is to measure the significance of big data in the fields of Finance, Healthcare and Sports. He observed that the largest segment of big data revenue could be attributed to the professional services industry with 40% market share.
Dhar suggested that a different outlook is required for observation and orientation of Big Data, and that too with minimal pre programming. He went on to say that it is critical to look at the signal and the noise in the data. Dhar also mentioned that no amount of modelling could help avoid random exogenous shocks that are impossible to predict. It is important to know if observations from the data are a good representation of reality. Asking the right question is key to generating valuable answers that have not just explanatory power but also predictive power.
Dhar discussed that if there are two sets of data – one that has a linear structure but with high bias and more noise, and another that is non-linear but with low bias and better signals - it is advisable to go with the second set of data to create a deterministic and valid model.
Dhar said that the biggest opportunity in the Big Data market is primarily focused on customer-experience enhancement, which is projected to have immense scope and growth. Dhar’s current research involves prediction in financial markets with the use of numerical data as well as unstructured and tagged data. He is also interested in data governance and analytics that answer questions such as what data to keep and why, how to measure risk and return from the data, and what capabilities to develop internally versus outsourcing to the market.
Watch video excerpts of the talk below: