Conscious machines will develop exponentially because they don't need to waste their time for studying what their predecessors already know.
In 1997 in comp.ai.philosophy I discussed the subject with Louis Savain, Chris Hooley and William Modlin.
What will happen if an algorithm of artificial mind is common known? It will be not a half-live computer virus, but a well thinking soulless machine. Such a machine cannot feel with or appreciate human's feelings because our "high" senses (humor, lonely...) originate from self-preserving that is a property of life, not of intelligence.
Artificial intelligence is not an expensive and sophisticated A-bomb technology. It merely is an easily copied program. What if the algorithm doesn't need much of computing? I think that properly organized associative brain, which is as smart as mammal's one, could be placed in a desktop computer.
Every tyrant and bandit wishes such accomplice.
The understanding of a danger cannot stop it, especially then AI technology does world round spread. War and suicide are disastrous, too. But we do it.
I think that power of money is the only hope to escape the new technology danger. State budget and personal wallet might win possible military application of AI because the AI is extremely needed in common life.
The problem of artificial mind will be solved soon. And we cannot prepare ourselves for the new era in principle.
At first we will teach robots to what is useful for us. Then AI entities will develop exponentially because they don't need to waste their time for studying what their predecessors have already known. The knowledge will accumulate not in books (and teachers) but in the robot's brains.
It's impossible that a conscious robot, who is 10 times clever than I, were my servant. On the contrary, we will turn to servants of next robot generations, which will multiply their knowledge by 10 times per annum.
It is accepted that consciousness arises as a way of adaptation . Thus, making adapting robots, we make conscious systems. Firstly they look merely like adaptive programs. So that even the most far-seeing of us don't appreciate the systems are conscious. A baby also doesn't look too conscious.
Today the word intelligent is used as a commercial mark. We are ready to see elements of consciousness in an operating system. Are they in animals? Is a dog only intelligent (behaves rationally) or it is sentient and aware of itself? The machine-like intelligence comes to consciousness then an entity discovers that its condition depends on its action. Feel arises. Conscious thing recognizes its own behavior. It distinguishes its own behavior from other events. It differentiates itself from other things.
So every more or less complex adaptive system, that we are inclined to call intelligent, is potentially conscious. From its creation an intelligent thing behaves intelligently; a conscious thing finds its own behavior - it adapts. And we subjectively evaluate, is this behavior really intelligent? I can imagine a system that fluently speaks but is firmly programmed. On the other hand such simple entity as Aplysia invents its own simple behavior without any programming. Its brain couldn't know the surrounding where the Aplysia will live. It has a tiny seed of genuine consciousness. Consciousness stems from adaptation, not from complexity.
Living entities adapt to their life conditions not because they have conscious goal to survive . They care of food, of ceasing pain. Personal surviving, bearing and death are intangible abstractions. A cockroach runs away because it foresees "it will be worst", not because it is afraid of death. The natural selection, caring of cockroach's surviving, gave him capacity for foreseeing.
Evolution developed brain that serves for surviving of species, not especially for thinking. Capacity for foreseeing eventually turned to thinking about oncoming events and actions. Such a gift of nature, for those who love thinking, probably helps us to survive, too. Understanding and valuing this capacity we intend to create machines that thinks and are aware of self. I believe the machine doesn't need so much of resources as a human brain does because their brain doesn't need to care of surviving.
Designer unable to control its robot entire behavior. Moreover, not every designer wishes trace carefully his creature. Often the designer intentionally gives a freedom for, say, a computer virus. Generally, there are many things, we create, which are not fully controlled in principle. Our children, for instance, are. AI is the second example.
Perhaps the first tiny intelligence will possess a tiny free will . But we pretend to make real mind that may compete with human's one. So I believe the "simple intelligence" is only the first step towards the purpose. Eventually the problem of making synthetic mind, that has its drives and free will, will be solved.
If you allow the intelligent machine to make a decision, you cannot know what will the decision be. You also may not be sure that you will be satisfied with the decisions. Are you always satisfied with your own decisions? So the machine can realize its own will in accordance with its own judgement, even if you don't wish that.
We might use artificial selection to grow useful generations of AI. Such a selection or terminating of non-proper artificial entities was successfully applied, say, in breeding of new dogs. But how do you imagine the terminating of an entity that is clever and is more sentient than you are, especially, if it does value itself and its consciousness?
Well-developed robots can invent the "gating algorithms" to pass through our "artificial" selection. If a conscious entity values itself it tries to pass the selecting gate by all means and tricks. By such a way we will select extremely sophisticated and insincere robots. Free enough education is less dangerous, I think.
I see the only way that gives us a hope of peaceful co-existence with robots. When the robots possess human power of intellect and when they interfere to our social life we must give advanced AI right of vote and also the right to be elected. Then the selection of moralities, which acts in democracy, will select AI entities for our (electorate) taste. Until the process is stable I hope we will not create too powerful robots. However this hope is logically unsupported.
Finally nothing alive is needed for new generation of synthetic life. Thank God if we will become pets and toys.