It is possible to simulate stupidity not creative mind. No programmer could forecast all future situations in the robot's life.
Consciousness and Logic
Suppose in a shop you've seen a home robot-servant. In the user instruction you read the robot has sensations and emotions, so it may possess consciousness. How could you make certain of it?
I don't mean intellect or rational behavior. I say about first signs of consciousness - subjective sensations, feeling. A robot may properly (correctly, rationally, well) polish floor or wash dishes and the very this time it might experience absolutely no senses.
Perhaps, live communication with the robot will help you to understand who the robot is -
A feeling person or a soulless machine?
Under the impression of such a difficult problem you purchase this "robot-servant" not only as a servant, mostly for solving the puzzle of consciousness - does the machine really possess consciousness like humans do?
Setting up different units and programs makes the robot coming alive. Initially it looks like an awkward machine. Then it shows some abilities in housekeeping. It gets acquainted with you and your family. It begins to participate in your everyday works. Sometimes you watch TV news together and discuss them with your robot-servant.
Your relationship comes quite unofficial, and you ask him a direct question about his consciousness. He says that he is feeling like you are, feels pain, wishes, loves the nature. It happens he is interested not only with cleaning of rooms; it is interesting to speak with him.
You don't know how the robot is made. He is also interested with this question. You choose to communicate more to determine is it right that the science has created a feeling artificial entity, or it is only a sophisticated imitation of such entity.
In a short time you become good friends. You discuss philosophical problems and political news, go fishing, and play chess with alternative success.
At last you understand that your friend doesn't imitate conscious behavior. It is possible to simulate stupidity not capacitive mind.
You both are peer intellectually. Nobody could forecast and program all future situations in the robot's life. Sometimes in new or hard circumstances the robot finds surprisingly good new solutions. You understand him well and sympathize with him. You see what is pleasant to him, what he is trying to avoid. He has wishes, needs, and interests. Now you are sure that a soulless machine would be unable to appear such a sincere friendship.
After some years of this uncommon friendship you occasionally find a little mark "Windows compatible" on the back of you robot-friend. So you conclude the truth is somewhere in the Microsoft Empire. There you discover a laboratory of artificial intellect. They explain you that a computer cannot possess feelings or consciousness. The problem of artificial consciousness has not solved yet. All these ten years they controlled your robot-friend by radio. They claim you spent your feels and years for a "friendship" with a machinery! You refuse to believe that your friend is merely a serial robot-servant, a mechanism, not a sensing entity!
You pass on the entire story to your robot-friend. He is also feeling deceived - his will was controlled by unknown persons! Despairingly he wills to die and cuts his stomach. You see wheels, pipes, wires, microcircuits. What does it say about?
Maybe the Microsoft's programmers laughed on you? And your friend did actually think, feel, love you without an external control.
Why not use an objective (scientific, experimental) way instead of my personal opinion about actual personality of this robot-servant?
It may seem a good idea to check this robot's possible consciousness by more scientific test. However constructive (objective) definition of consciousness is impossible because personal spaces of subjective experiences are not linked by subjective (ideal) channels. They are linked through material channels that result "one cannot sense senses of others". Material and ideal processes are not intersecting, not reducing to each other. So "objective" test on consciousness is also impossible. In a better case some indirect measurements are possible, which may show some plausible (but not proved logically) connections between ideal and physical phenomena. Such "proves and testes" on possessing of consciousness are proposed in my article " Mechanisms of consciousness ". Instead of objective test (a row of consistent measurements) we are forced to use a subjective test.
Subjective prove of consciousness is not logical but only convincing one. Our formal logic is ultimately supported only on our experience, too. So logical proof of consciousness reduces to a vicious circle and is not quite logical one. The logic has limits. In other words we can prove that some entity is conscious not strictly, but with the same accuracy as we believe that humans are conscious. Proof of consciousness should be supported on convincing reasoning directed as to mind as to heart.
Also our burden is convincing a tester that some results of a specific testbed are evidence that the tested object possesses consciousness. It is not very difficult to "convince" a low educated person. Many accept a computer simulated psychiatrist Eliza as a rational, clever or even sensing computer. Many are ready to believe that Deep Blue "had the aim" to win Kasparov. A participant of comp.ai.philosophy made a conclusion that a scorpion "was trying to bite itself" when it was poured with hot wax.
Our proof should be convincing for an educated and skeptical expert. Even if we could build the proof it remains not strict logically. Formal logic axioms are supported only on experience and common sense . Many sure facts cannot be strictly proved. For instance, you are sure that tomorrow the Sun will rise like it does for millions years. As an explanation you might use scientific evidence about the Solar system and laws of mechanics. However - we create Lows of Nature . They are not strict logical corollaries from observations. They are axiomatic claims about the most general repeated patterns of nature and logical corollaries from such claims. If tomorrow the Sun will not rise, the surprising fact is stronger than all our theories. The fact will force us to search new explanation.
So the problem of our robot's consciousness remains in a domain of one's opinion. Happily my opinion about my consciousness is my consciousness per se; The robot's opinion about his consciousness proves (to him) that he is conscious. But my opinion about him remains a working hypothesis (a theory) like the theory explaining why our sun rises.