The “rise of the machines” trope is firmly embedded in our pop culture because it resonates for so many of us.
Early on Mary Shelley played on our fears that new technologies will enable our creations to turn against us:
You are my creator, but I am your master; Obey!
Now this is a constant point in any discussion of the future of AI:
… often come across the idea that many people are afraid of what AI might bring. It’s perhaps unsurprising, given both history and the entertainment industry …
– What an artificial intelligence researcher fears about AI
This fear is accentuated by the rise of deep learning neural networks that are inscrutable black boxes:
… fear comes from the black-box nature of the underlying AI process …
– What Is Explainable AI?
Demonstrating successful results aren’t enough to make this anxiety disappear especially when the stakes are high:
The algorithm performed surprisingly well … Here is the strange rub of such a deep learning system: It learns, but it cannot tell us why it has learned; it assigns probabilities, but it cannot easily express the reasoning behind the assignment. Like a child who learns to ride a bicycle by trial and error and, asked to articulate the rules that enable bicycle riding, simply shrugs her shoulders and sails away, the algorithm looks vacantly at us when we ask, ‘Why?’ It is, like death, another black box. …. I cannot shake some inherent discomfort with the thought that an algorithm might understand patterns of mortality better than most humans.
– The Dying Algorithm
While few may openly admit that this free-floating anxiety impacts their decisions it surely will change their behaviors.
XAI is a fundamental step in decreasing this barrier to adoption. Transparent processes that allow meaningful human validation will always be easier to accept than a black box.