Should Data Be The Only Core To All Decisions?
Should Data Be The Only Core To All Decisions?
The use of advanced data analytics in organisations’ decision-making process has increased tremendously over the recent past.
Businesses are now focusing on making a data-driven decision-making culture a reality across all levels of their organisations. As a result, data-based insights now form a critical part of the day-to-day decisions. It is used to provide both predictive and prescriptive suggestions needed in critical decision making. However, the question of whether organisations can rely entirely on data in their decision-making processes remains unanswered. Here, we examine whether data-driven decisions are the best decision and understand the role that certain factors like empathy, emotional intelligence, and intuition play in the process.
The steady growth of binary logic
The human brain comprises of two sides: the left and the right side. While the left side is analytical, the right one is creative. Technology has continuously proven that the logic/analytical (left) brain can create wonders. Even the latest advancements in technology continue to support this analogy. However; a more in-depth analysis of the human philosophy indicates that nothing is purely rational.
On the contrary, the rationale can only be justified based on the social context that human nature is a bundle of emotions, and the right balance of emotions is the precursor to a robust rationale.
Technological progressions achieved over the years demonstrate that binary logic can create unprecedented advancements. As such, binary logic is one of the essential capabilities of the current generation. It brings out the realisation that what cannot be measured cannot be managed and has developed the need to measure and track everything. Many organisations now believe that they cannot achieve anything that is not measured and tracked. On a broader perspective, the world is an embodiment of human beings that lacks binary logic. Instead, it is mostly built just on contextualized logic.
Organisations create a multitude of systems and processes to build strong foundations and grow big faster. Apart from providing self-service solutions and increasing automation, these systems and processes open a large pool of data. Companies that excel in binary logic put much emphasis on such data. In most cases, the data helps in recognizing patterns and predicting the future based on existing inputs from systems and processes. However, it becomes problematic when senior management and leaders start looking for data in every work irrespective of its applicability. Eventually, data and metrics tend to become the core of such organisations rather than a problem.
Don’t just measure, act…
Using the right information is beneficial to the organisation’s decision-making process. However, giving more attention to only data or intuition can result in more hurdles than progress. For instance, when the HR teams spend an excessive amount of time measuring employee experience, their time for improving the experience becomes limited.
Let us consider three scenarios or decision patterns of the leaders as a part of the study. They include an evaluation of decision making based:
- Purely on data
- Purely on experience/ gut/context (maybe counter-intuitive to what the data indicates)
- A mixture of data and experience/context/ gut
Purely data-based decisions:
Purely data-based decisions are made based on the analysis of the available data without the use of intuition. One of the domains that work entirely on data is the stock exchange. Typically, the rule of thumb in the stock markets is that if you make a long-term investment, you are bound to gain. For you to win in the short-term, you need to know the right times to enter and exit the market.
This begs the question of how does one know when it is the right time to enter or exit? The typical response to this question is that one will know when to enter and exit based on the market mood. This brings forth another question of what is the “market mood” in such a hard-coded data-driven industry.
To explain this paradigm, let us consider the following example:
The market efficiency theory states that if markets are rational, the pricing for all securities will be close to their fundamental value. This value is based on the premise that all investors have equal access to information. Unfortunately, prices of securities/markets do not always behave in the same way.
Stock experts usually use mathematical algorithms to make precise predictions on security prices. The algorithms predict the prices based on past performance, outlook, and current information about the security. However, history is full of instances where even in the presence of accurate market data and mathematical models market experts have failed to predict stock market crashes.
Using the right information is beneficial to the organisation’s decision-making process. However, giving more attention to only data or intuition can result in more hurdles than progress
Although such crashes were primarily associated with market sentiment, among other reasons, variable factors may not be directly related to the crashes. This may suggest that logical projections are not always accurate, even in a field where data is readily available and mathematical models of prediction are commonly used. Apart from the availability of data, other factors like market sentiment or mood also play a crucial role in determining the outcome.
Several stock markets crashes have occurred over the past 22 years, with SENSEX experiencing the biggest crash between 2007 and 2008. An analysis into the event begs the question of whether the crash was purely driven by data (correction of prices) or based exclusively on the market sentiment. After the crash, it took SENSEX almost three years to regain its initial position. Could market sentiments have lasted that long or was there empirical data that sustained the crash?
The same trend was replicated in 2010-11, where the SENSEX dipped but regained its position only after 2-3 years. In 2015, SENSEX declined once more and regained its position within a short time. Does this mean that bullish sentiment is like a wave that pushes up all stock prices when it comes and similarly crashes the market when a bearish wave starts? It also leads to the fundamental question of whether decisions in the stock market are made purely based on data, or is there a much more significant influence of human sentiment?
In an organisational setup, leaders rely on data in making critical decisions and implementing the plan. However, the implementation process requires an understanding of the contextual realities.
The same argument can also be used to analyse the decision-making process in the world of sports. Comparing players purely based on score may not always help identify the better player. Apart from scores, other variables like which team the player was most effective against also matter in choosing players. For instance, a medium score against a commendable opponent may be a better indicator of potential than merely a good score against a medium opponent.
The first step of the decision-making process usually involves gathering of data. When a company is trying to become more efficient, it will first collect data on its expenses and operations. The main challenge may be choosing the overheads to focus on and connecting data collected to the organisational design. Once adequate data has been gathered, the company analyses the data and makes the most feasible decision. Even though data is the starting point of a decision-making process, it does not always enable optimal decision-making when used alone.
Decisions based purely on experience / gut / instinct / context may sometimes appear to be counter-intuitive to what data indicates
Strictly gut-based choices are justified by an instinct that develops in the subconscious mind. After the mind has absorbed all the data and information from past experiences, it moves to the decision-making stage. While such decisions may appear as if they are purely based on instincts, the rationale behind the decision is based on insights. This is usually the case when much information is floating around in the ecosystem.
However, a purely gut/instinct decision is generally made in the absence of perfect knowledge or when the decision-maker cannot access it.
Multiple examples demonstrate how some people have been driven by their instincts to create billion-dollar organisations. When starting, they lacked enough data to prove that their ideas were worth pursuing. They relied on their gut feeling about their thoughts and chose to pursue them. Over time, such purposes grew into successful products taking significant positions in the market.
For instance, when Steve Jobs was the CEO at Apple, he used to say, “People don’t know what they want. You need to create an experience for them.” During his leadership at the company, Steve challenged the concept of market research and past data. This led to a revolutionary way of thinking and subsequent introduction of a wide range of products including iPad, iPhone, MacBook, and more. Many people have also launched businesses and products based purely on their gut feeling. Among them, some have already failed and exited the market, and others are thriving as big brands.
Decisions based on data and supported by experience / context / gut
Google is the front-runner in any discussions where data forms a significant part of the narrative. Since its inception, Google has made data available to the world. The basic premise of the company’s success is that the decision-makers follow their data. A critical look into the company’s operations also reveals that they also use data alongside contextual reality.
This has helped them to transform some of the basic organisational processes like hiring altogether. For instance, a top-ranking college degree or glowing recommendations are not a requirement for one to be considered for a position at Google.
Over the last couple of years, researchers have focused on the different hiring processes and their outcomes for both academic and commercial considerations. On its part, Google has conducted several studies on the topic and designed its own recruitment process. Unlike other recruitment processes, Google does not consider conventional attributes like a college education, experience, or references of candidates before hiring them. In contrast, the application process starts with the evaluation of the CV based on specific guidelines. The focus is on the alignment of skills and experience with the job description. The candidates are required to share details like their role in the previous organisation, measurement of success, and their scope of work. The guidelines help in standardizing the selection process by making it easier to evaluate CVs from varied domains.
By introducing a new employee hiring and selection process, Google has challenged the traditional way of recruiting employees. It has launched an evaluation process that helps to understand the ability of a candidate to perform in a given current context. The Google People Analytics Team conducted a statistical analysis, which deduced that four interviews were enough to predict the suitability of hiring a candidate at Google with 86% confidence. The study also revealed that the brain teaser puzzles were not a good predictor of a candidate’s performance once they came on board.
Amazon also uses a similar approach in making data-based decisions. When Jeff Bezos was evaluating new ventures for DE Shaw, he came across a set of data showing that the usage of the World Wide Web was growing at an unprecedented rate. His speech at Princeton in 2010 stated, “I came across the fact that Web usage was growing at 2,300 per cent per year I’d never seen or heard of anything that grew that fast, and the idea of building an online bookstore with millions of titles was very exciting to me.” Although he collected data about the World Wide Web, online book store never existed at the time. He used the data to start an online book store, which he combined with his instinct about the future of shopping globally. Eventually, this combination led to the creation of a company that is known as Amazon today.
In an organisational setup, leaders rely on data in making critical decisions and implementing the plan. However; the implementation process requires an understanding of the contextual realities. As such, decision-makers need to consider both the available data and contextual facts before it is important to remember that humans can create the best machines, but the best tool cannot create great humans. Therefore, instincts hold the same amount of significance as data in the decision-making process and can make the difference between success and failure implementing a plan.
Although binary logic and data help leaders and organisations to make a predictive decision, they lack the context that the future holds. Therefore, the use of data should be limited only to the extent in which it is meaningful rather than making data the centre of everything. When making a decision, gut and context must be combined with data to implement measures that improve corporate performance. It is also important to remember that humans can create the best machines, but the best tool cannot create great humans. Therefore, instincts hold the same amount of significance as data in the decision-making process and can make the difference between success and failure.
