As organizations obsess to become data-driven, what’s the role of intuition in the future of business? And with the emergence of new technologies such as AI, how can machines and humans work better together to make the right choices?
A landmark KPMG survey, which involved over 1,000 CEOs in New Zealand, “found that 74% of CEOs have overlooked insights provided by data analysis or computer-driven modeling because they were contrary to their own experience or intuition.” In other words, they decided to go against what data suggested as the obvious next step.
This is, to an extent, understandable. After all, CEOs have accumulated millions of “personal data points” through their careers. This experience gives them the confidence to act with exceptional foresight in uncertain situations, which often involve a lack of data or an incomplete view of the situation.
It’s Not About Data Versus Intuition
It is a misconception that business leaders should trust either data or their gut. Both provide valuable information that often complements each other. While data is often seen as an “unbiased” source of information, this isn’t necessarily the case. Data processes, data models and algorithms can be impacted by the personal biases of those who build them. What’s more, while data is neutral by nature, it is ultimately an imprint that doesn’t understand the full context to explain the “why” of what it is shown.
This doesn’t mean that businesses can be run purely on instinct or gut feeling. It would be virtually impossible to accurately understand performance, monitor results, or create estimations and projections without the right data. However, people lead businesses, and in a complex business environment, executives need to be able to balance both.
A recent study conducted by the Fortune Knowledge Group about the emotional logic of business decisions showed that executives “rely heavily on ‘intangible’ differentiators (such as culture or reputation) over quantifiable differentiators as a way of judging potential business partners. About two-thirds (65%) of executives agree that an increasingly complex business environment has made it more difficult to base decisions on purely ‘functional’ factors (for example, cost, quality or efficiency). “
Ultimately, it’s not about data or intuition; instead, it’s about how to harness the power of both. Exceptional data can give a clear view of what is happening, and strong personal experience can provide business leaders with instinctive intuition that works alongside data rather than against it.
Use Your Intuition To Ask The Right Questions
Data processes and models are built by people and are, therefore, inherently biased. Your data will only be as good as the questions asked in the first place. While everyone is looking for valuable insights, sometimes asking the wrong questions might end up giving you data that could lead your organization astray.
Knowing what not to look at is one of the most important jobs for any data analyst. Edward Hare, a retired director of strategy and planning for a Fortune 250 company, noted that rather than concentrating on the data, management should focus on “being able to formulate the right questions to ask at precisely the right moment.”
Of course, the baseline for being able to ask all the right questions will be to have access to as many sources of data as possible, both qualitative and quantitative. This is why the industry of data management, and in particular orchestration and automation, is booming. You simply cannot ask for the answers you need if your business or your “data engine” isn’t able to draw from the dozens of data sources required for in-depth analysis. This is even more important as companies venture into emerging technologies for predictive modeling such as AI, which get exponentially better with more data.
A recent Open Data Science article emphasized the importance of intuition in the era of big data. Contrary to popular belief, the reality is that AI is unlikely to leave data analysts and data engineers jobless. These professions will always be key to ask the right questions and create the right hypotheses. While machines can take over areas where humans just aren’t great (which are also often the most tedious tasks for data professionals), they will free up time for humans to do what they do best: solve problems and build relationships.
The Best Of Both Worlds
You can let your intuition lead your data exploration process. However, you can also let your data inform your intuition. For example, analytics company Mixpanel showed how it uses data about its platform usage and existing customer journeys as the basis to inform its design intuition for upcoming product features.
If there is a single takeout for any business leader, it will be to strive toward building a relationship between their data and their gut. More information doesn’t always lead to better or more confident decisions. However, choices that are purely based on gut instinct and intuition can be misguided. Understanding that data and intuitions are allies instead of rivals is the best possible start.