A “horse” is a machine learning system that gives the right answer for the wrong reasons. These systems are named for Clever Hans who appeared to be doing arithmetic when actually he was astutely reading his handler’s body language. Hans was demonstrating real skill and was giving correct answers, but his learning did not generalize.
Like a machine learning system Hans’ skill was built through repetition of examples rather than any transfer of insight. Also like many machine learning systems Hans’ skill was brittle. If you changed his environment (switch to different handler with different body language) so his inputs were different from how he was trained than he would suddenly fail spectacularly. This is true even though his inputs were still valid given our presumption of how he was deriving the answers. It was only through knowing how he actually derived the answer that you could know how the environmental change would impact him.
So the take-away for machine learning: if we know how a system is producing a result we will dramatically improve our ability to project under what conditions that system will continue to produce robust results. Explainable systems show us how the result is produced, black box system do not.