For more than two decades, Netflix has been obsessed with machine learning models.
In 2006, the company announced a million-dollar prize to anyone who could improve its recommendation algorithm’s accuracy by 10%. Over 40,000 teams participated in the global challenge.
The competition ran for three years, and only two teams managed to exceed the accuracy threshold. Netflix rewarded a winner that delivered a 10.06% accuracy improvement.
But, they dumped the winning algorithm.
Despite its stellar accuracy, the engineering costs and complexity of this algorithm were very high — too high for its accuracy improvement. Instead, Netflix used a lower-ranked — but simpler and less costly — algorithm that delivered just 8.43% improvement in accuracy.
Notably, by the time this 3-year marathon for algorithm accuracy ended, Netflix’s business model had pivoted. The focus had shifted away from DVD rentals to streaming, thereby diminishing the utility of these algorithms.
Today, many organizations pursuing machine learning (ML) are victims of the accuracy fallacy. They get narrowly focused on model accuracy while wasting precious resources in over-optimization. Business leaders have shelved good solutions that don’t deliver great accuracies.
These are massive lost opportunities for businesses. Your AI/ML model could be inaccurate, but it can still transform your business for the better. In this article, you’ll learn four ways to find out if your model can be transformational for your business.
What influences machine learning model accuracy?
a) Are we including all relevant inputs that can influence outcomes?
b) What is the quality of historical data we’ve collected?
c) Does our model capture the true relationships between the inputs and outcome?
d) How different will future scenarios be from what was used to train the model?
You can see that the model accuracy is at the very end of a chain of not-so-perfect connections. Understanding that models can at best produce an approximation of outcomes is crucial to make the most of them.
In a given situation, your data science team must strive for accuracy improvement by optimizing these factors. Once you have a machine learning solution, here’s how you must evaluate its business impact.
1. Benchmark the model accuracies against human performance
“Most people don’t know what they want unless they see it in context,” says Dan Ariely, the author of Predictably Irrational. We humans can’t make sense of a number in isolation.
This is the challenge with model accuracies. When presented as a percentage, one expects it to be as high as possible, as if these were school grades. We end up making a false comparison between a world where machines make mistakes and a world where mistakes never happen, says Berkeley Dietvorst of the Wharton School.
To contextualize model accuracy, compare it with human error rate for the same task within your organization. Let’s say human accuracy for your job is 81%. A model that’s 78% accurate is just 3% short of the benchmark; it’s not off by 22%.
2. Improve outcomes by augmenting models with human intelligence
Organizations often pit AI against humans as if it were a competition. Sure, to understand the state of artificial intelligence, it is useful to benchmark AI model performance against human accuracy. But all comparisons should stop there. Instead of pitting them against each other, why not bring both the players onto the same team?
Augmented Intelligence is a human-centered partnership that brings people and AI together to enhance cognitive performance. We have seen great examples of augmented intelligence during the pandemic.
At a time when patient-to-healthcare provider contact became a key concern, robot-assisted surgery has saved lives. Nature reports how surgeons used systems enabled by Magnetic Navigation Systems (MNS) to perform remote endovascular surgeries. Not only did the system provide reasonable control and flexibility, but it increased procedural safety.
To improve outcomes, you must keep humans in the loop when you design machine learning systems. In scenarios where your model’s accuracy suffers due to lack of sufficient data or high situational variability, humans can step in for support.
When humans and models are combined, the net accuracy achieved is far greater than what can be achieved by either in isolation.
3. Check if the model has scope for continuous improvement
Algorithms are terrific learners. They improve when they see more data. They improve when you give them specific feedback on where they went wrong. They improve with advances in AI research.
When we make decisions on financial investments, we compute the future value of money. But, we don’t do that with algorithm accuracies. Instead, we evaluate them at face value.
The prediction of protein structures has been one of the toughest challenges in science for the past 50 years. In 2018, the CASP Competition (Critical Assessment of Structure Prediction) saw the effective application of AI. AlphaFold, the AI system built by DeepMind, delivered spectacular results by increasing scores by over 10 points from the previous edition. In 2020, AlphaFold solved this grand challenge by drastically improving its score to 92.4, within just two years.
To understand the real potential of your model, factor in the possible benefits you can reap through more data, better feedback, or research advances in AI.
4. Compute the business value of your model’s outcomes
The previous three steps explained how you can contextualize your accuracy, improve it with augmentation, and project a future value. Now, the final and most crucial step is to quantify and measure your business outcomes.
No, your ML model doesn’t have to be perfect to deliver business value. Often the baseline is so low that even a 1% improvement can yield substantial business gains, says Drew Smith, Vice President at International Institute of Analytics.
During his tenure at IKEA, the furniture conglomerate, Smith reports that long queues (or lines) were the number three reason for customers to shop elsewhere. Flow analytics models helped predict queues that would form in a store.
While the algorithm wasn’t perfect, Smith says that the predictions were much better than the manager’s ‘gut.’ This enabled the store team to institute queue-busting practices, which reduced complaints about queues by 30%. This turned out to be an invaluable tool for excellent customer service.
Ask yourself what your machine learning model’s real purpose is.
Should it help you grow revenues or reduce costs? Translate the net accuracy improvements into the business impact that the model will deliver. Quantify the impact, and this will help you shift the conversation from accuracies to business outcomes.
Transform your business with incremental model improvements
To resolve the disconnect with business leaders on AI model accuracy, you must empathize, educate, and engage with them.
Smith advises that “this is an amazing opportunity to discover more about your stakeholders, their challenges, and the opportunities they want to crack. Be curious as to why they are aiming for such precision.”
Educate them on what the purpose of a machine learning model is. Explain what influences accuracy and how you can realistically manage the factors to drive outcomes.
Finally, engage with them. “Evaluate the tradeoffs with your stakeholders transparently and objectively to gain alignment. Pursue the path that maximizes business value”, says Moore.
Smith adds that the team with a mindset focused on improving many business areas quickly and continuously will beat the team that aims to solve one big problem perfectly.
Netflix is an excellent example of an organization that thrives on incremental innovation. Thanks to ongoing, tiny improvements in their AI models over the past 20 years, more than 80% of videos that people watch on Netflix are discovered through their recommendation system.