Risk of “Machine Learning Overkill”

Screenshot 2018-02-16 08.15.46

We are all justifiable excited about the potential of applied machine learning.  However, Venkat Raman points out the dangers of letting that excitement take us into “machine learning overkill”, when we are applying machine learning for its own sake.

‘This is the problem we are facing, tell us what Machine Learning Algorithms can be applied?’ … The newly minted Data Scientists quickly blurt out 2–3 ML algorithms and the enamored company hires him/her . In due course of time the algorithms are implemented. The Data Scientist impresses the company with good accuracy % of the models. The models are put in production. But lo and behold, the model does not net the company the ROI it hoped for … what happened was the Data Scientist did not have business acumen and thought his/her KPI was just building ‘good’ ML models.
— Venkat Raman So, How Many ML Models You Have NOT Built?

Rama Ramakrishnan highlights a best practice for avoiding this risk: Create a Common-Sense Baseline First.

Experienced practitioners do this routinely.  They first think about the data and the problem a bit, develop some intuition about what makes a solution good, and think about what to avoid. They talk to business end-users who may have been solving the problem manually. They will tell you that common-sense baselines are not just simple to implement, but often hard to beat. And even when data science models beat these baselines, they do so by slim margins.
— Rama Ramakrishnan  Create a Common-Sense Baseline First.

Rama goes on to provide three real world examples (direct marketing, product recommendations, retail price optimization) where thinking first about a baseline solution will better inform your decision making and define a benchmark for judging any potential machine learning based solution.

In choosing between the baseline implementation and a black box AI alternative we should consider not just the relative predictive power of each as measured in the lab but also our relative ability to understand, trust and manage each of them in production.

Clearly the goal shouldn’t be “find most interesting machine learning technique to solve my problem” it should be “find the best way to solve my problem and generate real business value.”  As Peadar Coyle points out, we don’t want to end up a trophy data scientist who is just the “smart nerd in the corner” and doesn’t end up “adding value to organisations”.

XAI is focused on improving our ability to understand and manage our models and through those improvements create exactly this sort of connection between model building and value creation.

Venkat also makes a good point about how in judging the applicability of machine learning we should not just consider the accuracy % as measured in the lab, but we should also make a judgement about the likely delta between our model and reality.  This requires judging the quality of data inputs, the casual relevance of features, the generalizability of the training set, etc.

‘All Models are wrong, some are useful.’ In most Machine Learning Algorithms we try to minimize the loss function.  Models are an abstraction of the reality. The word here is abstraction. It is not actual. If you think about it, the process of building Machine Learning Algorithms itself has a larger ‘Loss Function”. That is we differ from the reality.
— Venkat Raman So, How Many ML Models You Have NOT Built?

Implementing “explainable” machine learning techniques puts us in a much better position to judge how large is the delta between our model and reality.  This is part of the “insight benefit of explainability“.

Leave a Reply