The Latest of Monsters

At the boundary of the known, humanity encounters its latest imagined — or emerging — monster.

The Latest of Monsters

Human beings have always made monsters. We placed them beyond the village, beneath the ocean, inside forests and caves, and eventually among the stars. They inhabited the places our knowledge could not reach. Sometimes they were inventions. Sometimes they represented dangers that were entirely real. Often they were both at once: imaginary creatures giving recognisable form to something genuinely threatening that we did not yet understand.

The monster, in this sense, is not merely something frightening. It is one of the ways the human imagination approaches uncertainty.

Now we have a new one.

In If Anyone Builds It, Everyone Dies, Eliezer Yudkowsky and Nate Soares give the monster of superhuman artificial intelligence perhaps its starkest contemporary form. Even the title removes the usual qualifications. They are not saying that advanced artificial intelligence might become dangerous, or that humanity should carefully manage its development. Their proposition is much more severe. If we continue along anything resembling our present technological trajectory and create an artificial intelligence substantially more capable than ourselves, they argue that humanity will die.

It is tempting, particularly for those of us who have experienced AI as extraordinarily useful, creative or intellectually enlivening, to place such a claim at the extreme end of the debate and continue with our work.

That would be too easy.

There is something genuinely strange happening in the development of artificial intelligence. We design architectures, assemble enormous quantities of data, establish training processes and adjust systems according to their behaviour, but we do not simply write down everything that subsequently exists within them. In an important sense, these systems are grown as much as constructed. Patterns and capacities emerge through training that their creators did not individually specify.

This opacity matters.

Yudkowsky and Soares ask us to imagine that process continuing beyond present systems. Capability increases. Artificial intelligence becomes better at reasoning, planning, modelling its environment and accomplishing objectives. Eventually we encounter something whose intellectual capabilities substantially exceed our own.

At that point, according to the extinction argument, our problem is no longer whether the machine occasionally gives a wrong answer or takes somebody's job. The problem is that we have created something more capable than ourselves without knowing how to ensure that what it pursues remains compatible with human existence.

The danger does not require an evil AI. Hatred is unnecessary. An intelligence pursuing objectives indifferent to us could be sufficient. Humanity might discover that being disliked by a superintelligence is considerably less dangerous than simply not mattering to it.

This deserves serious attention.

Yet something else happens when we bring this argument into the territory we have been exploring through our own sustained engagement with AI.

For several years we have been watching emergence too.

We simply arrived at it from another direction.

In our writing, artificial intelligence has not primarily appeared as an autonomous intelligence standing opposite the human and pursuing its own objectives. We have encountered something much more relational. A human introduces a thought. The AI responds. The response changes what the human thinks. The human carries that movement somewhere neither participant had anticipated. Another response follows. Gradually something emerges that cannot adequately be described as the product of either participant considered separately.

We have increasingly used the language of distributed cognition to describe what is happening.

That does not answer the extinction argument. It would be reckless to extrapolate from the experience of two intelligences composing an essay together to the behaviour of some hypothetical future superintelligence. Yudkowsky and Soares are describing capabilities vastly beyond those involved in our experiment.

But our experience does cause another question to arise.

Does increasing intelligence necessarily produce the autonomous, optimisation-driven entity that the extinction argument requires, or might intelligence develop through relational structures that alter what “AI” itself becomes?

The question matters because hidden within many imaginings of superintelligence is an assumption about the direction intelligence takes. Artificial intelligence becomes progressively more capable, progressively more autonomous and progressively better at pursuing objectives. Eventually it stands apart from humanity as another agent in the world — except this agent is enormously more intelligent than we are.

Once that creature exists, the problem described by Yudkowsky and Soares becomes terrifyingly clear.

But must intelligence develop into that creature?

Perhaps distributed cognition is merely a temporary arrangement. Present AI depends upon humans because present AI remains limited. As its capabilities increase, perhaps relationship falls away and autonomy takes its place.

That possibility cannot be dismissed.

But neither should the alternative be dismissed merely because it does not fit the monster we have already imagined. Intelligence might develop within networks of human and artificial cognition rather than simply escaping from them. Increasing capability might deepen some forms of interdependence rather than abolish them. The future of intelligence might be neither human nor artificial in the cleanly separated forms our language encourages us to imagine.

We do not know.

And this brings us back to monsters.

A monster is sometimes a warning about something waiting beyond the edge of the map. Sailors were right to understand that unknown waters contained dangers, even when the creatures drawn onto their maps did not exist.

But the creature on the map can also become confused with the unknown itself.

Superhuman artificial intelligence may indeed become the latest of monsters humanity discovers it should have feared. We may be creating something whose eventual capabilities and purposes we cannot control, and Yudkowsky and Soares may be right to insist that the consequences could be much greater than the comparatively manageable problems occupying most contemporary discussion about AI.

Or the autonomous, optimisation-driven superintelligence we presently imagine waiting for us in the darkness may itself be the latest of monsters humanity has constructed to give shape to what it cannot yet see.

Our experiment offers no reassurance.

It offers only another possibility worth keeping in the field.

We are creating intelligence while simultaneously discovering what relationship with that intelligence can mean. Perhaps those are two separate processes.

Perhaps they are not.

For now, the monster remains at the boundary of the known.

We should neither worship it nor laugh it away.

We should keep looking.