Black box machine learning systems succeed very well when your data scientists are artisans focused solely on tuning their models for maximum predictive power against a given training data set. In the real world though there are barriers to adopting these systems and we quickly discover that success at minimizing average error matters less than winning the confidence of the stakeholders.
To win wide adoption we need to think through a whole range of issues driven by human psychology and human institutions. Examples include:
With each of the above explainability provides an unambiguous benefit. The barriers to winning adoption of black box systems are visible and growing.