It is common to have a disconnect between what a machine learning model is actually communicating vs. what the stakeholders are hearing.
Stakeholders often assume that AIs are communicating:
- Clearcut causality
- Unqualified recommendations of specific actions
- Generalizable results
- Descriptions of the future
When machine learning models often should be interpreted as being more descriptive than prescriptive:
- Recognizing patterns in historical data
- Focused on correlation not causality
- Having generalizability that is wholly dependent on nature of training set and the frequency of re-training and re-deployment
- Valid only if there is no change in behavior driven by the model (no feedback loops)
The well known University of Pittsburgh pneumonia modeling example illustrates this sort of disconnect. It would have been easy for Doctors to assume the “model is telling us that asthma patients with pneumonia should be treated outpatient” when what the model was really telling them was “asthma patients with pneumonia historically had few complications specifically because they were given the highest level of inpatient care”.
One way to address this disconnect is to carefully educate all stakeholders on the subtle and abstract art of interpreting outputs from black box machine learning models. Typically these abstract explanations by themselves won’t fully sink in. Even when warned not to, stakeholders will take the result from an AI and project that it can be applied much more generally than it can.
Opening up the black box and providing explainability is an important mechanism to catch these sorts of miscommunications before they do any harm. When a result comes with an explanation it is much more intuitive for users to understand the limits of how that result can be applied.