Our cognitive biases are well known: we are overconfident in our intuition, we rely too much on our most easily recalled examples, we prefer simple narrative over subtle accuracy, etc. These biases are incredibly sticky, even when we acknowledge they exist in others we resist acting on the knowledge that they also exist in ourselves. Whether the consequences are high or low we hold tight to these ways of thinking.
Having to fight these deeply ingrained biases will slow the adoption of any new technique. Adjusting your technique to accommodate how humans already operate will accelerate adoption. XAI is more accommodating to our common biases than a black-box technique ever will be.
This is why so many are reluctant to gamble on the mysteries of neural networks. When Malioutov presented his accurate but inscrutable neural network model to his own corporate client, he also offered them an alternative, rule-based model whose workings he could communicate in simple terms. This second, interpretable, model was less accurate than the first, but the client decided to use it anyway—despite being a mathematically sophisticated insurance company for which every percentage point of accuracy mattered. “They could relate to [it] more,” Malioutov says. “They really value intuition highly.” – Is Artificial Intelligence Permanently Inscrutable?