When I think of this issue and those that are in either camp of how an A.I. should work it brings back a childhood memory. I was around 11 years old and my younger brother bought a pet snake, I don’t remember the exact type of snake. My brother was told that he should feed live mice to the snake and so he bought a bag of live mice to feed the snake. We watched in morbid fascination the feasting of the mice by the snake. On one of these feedings something fascinating happened. A poor mouse was thrown into the snake’s aquarium as usual and the mouse quickly took notice of the snake and stood up, trembling with fear. The snake stared at the mouse and the two animals were in this deadlock eye contact. The mouse started to wobble as if hypnotized by the snake’s stare. Then in a blink of an eye the snake pounced onto the poor mouse, opening its mouth ready to swallow it. But the poor mouse in milliseconds woke from its hypnotic state and at the last moment jumped out of the way of the on coming snake. You would think this quick move by the mouse would save it, at least from the initial attack by the snake, but the snake instantly wrapped its body around the mouse and squeezed it to death. The poor mouse’s eyes bulged where then it died.
So what happened? The snake assumed the mouse would simply not move or at least not in time to escape its mouth, and that was the usual case, but that proved to be wrong in this instance! The snake then reacted to the new situation and wrapped its entire body around the mouse! Here is a prefect example of how prediction is likely wrong in an environment that perpetually changes and reacting to change proves to be the better capability than prediction.
With that said: Artificially Intelligent systems can be wrong in their assumptions as they interact in the environment and must be capable of novel reactions to change at a moments notice. This is what natural selection has learned and why the snake proves to be a very adept animal.