Machine learning systems are being applied to sentencing prisoners, identifying voter suppression, investing our retirement funds, generating flood warnings and much more. As AI driven decisions become more consequential there is growing recognition that they also need to be understandable.
There is now hope that [deep learning] will be able to diagnose deadly diseases, make million-dollar trading decisions, and do countless other things to transform whole industries. But this won’t happen—or shouldn’t happen—unless we find ways of making techniques like deep learning more understandable to their creators and accountable to their users. – The Dark Secret at the Heart of AI